flat assembler
Message board for the users of flat assembler.

Index > Main > Why cant I add floating point number right

Author
Thread Post new topic Reply to topic
NanoBytes



Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States
NanoBytes
I am trying to add 12 to 0.2, but it keeps returning 12.199... and I cant figure out how to get it to work

Code:
        mov [Integer1],12
        FILD [Integer1]
        mov [Integer1],2
        FILD [Integer1]
        MOV [Integer1],10
        FIDIV [Integer1]
        FADDP st1,st0  
    

_________________
He is no fool who gives what he cannot
keep to gain what he cannot loose.
Post 02 Feb 2012, 03:09
View user's profile Send private message Send e-mail Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17352
Location: In your JS exploiting you and your system
revolution
12.2 cannot be perfectly represented as a floating point number. The FPU will use the closest approximation available 12.199999... Generally one would round the numbers to the required precision for display and the small error in the internal representation would not be seen.
Post 02 Feb 2012, 03:20
View user's profile Send private message Visit poster's website Reply with quote
NanoBytes



Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States
NanoBytes
Ahh, i think i figured it out
Code:
        MOV [Integer1],15
        FILD [Integer1]
        MOV [Integer1],2
        FILD [Integer1]
        MOV [Integer1],10
        FIDIV [Integer1]
        FSUBP st1,st0
        FSTP [Integer1]
        BTS [Integer1],2
        FLD [Integer1]
    

Though, i am worried, I am converting from the registers 10 bytes to the integers 4 bytes just to correctly add two numbers. Is there a more accurate way to do this?

_________________
He is no fool who gives what he cannot
keep to gain what he cannot loose.
Post 02 Feb 2012, 03:39
View user's profile Send private message Send e-mail Visit poster's website Reply with quote
NanoBytes



Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States
NanoBytes
Ok, revolution, how would i round the entire number off?
Post 02 Feb 2012, 04:01
View user's profile Send private message Send e-mail Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17352
Location: In your JS exploiting you and your system
revolution
NanoBytes wrote:
Ok, revolution, how would i round the entire number off?
Rounding is usually only done for display purposes. During internal calculations rounding is generally not done, and is usually not needed.
Post 02 Feb 2012, 04:16
View user's profile Send private message Visit poster's website Reply with quote
NanoBytes



Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States
NanoBytes
I know, I am working on a atof procedure
Post 02 Feb 2012, 05:35
View user's profile Send private message Send e-mail Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17352
Location: In your JS exploiting you and your system
revolution
Post 02 Feb 2012, 05:52
View user's profile Send private message Visit poster's website Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.
Post 02 Feb 2012, 06:46
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17352
Location: In your JS exploiting you and your system
revolution
JohnFound wrote:
Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.
The FPU cannot store enough precision for that.

The closest is 12.19999999999999929
The next value is 12.20000000000000107
Post 02 Feb 2012, 09:40
View user's profile Send private message Visit poster's website Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 559
smiddy
revolution wrote:
JohnFound wrote:
Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.
The FPU cannot store enough precision for that.

The closest is 12.19999999999999929
The next value is 12.20000000000000107
That's interesting! What is the limiting factor causing the precision error?
Post 02 Feb 2012, 13:00
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3502
Location: Bulgaria
JohnFound
Quote:
That's interesting! What is the limiting factor causing the precision error?


The registers size of course.
Post 02 Feb 2012, 13:18
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17352
Location: In your JS exploiting you and your system
revolution
smiddy wrote:
That's interesting! What is the limiting factor causing the precision error?
We can get more precision by using a larger mantissa size, but since the storage is binary and the output is decimal then there will always be some error. You can never completely remove the error, but you can make it arbitrarily small with an arbitrarily large mantissa.
Post 02 Feb 2012, 13:33
View user's profile Send private message Visit poster's website Reply with quote
Matrix



Joined: 04 Sep 2004
Posts: 1171
Location: Overflow
Matrix
revolution wrote:
smiddy wrote:
That's interesting! What is the limiting factor causing the precision error?
We can get more precision by using a larger mantissa size, but since the storage is binary and the output is decimal then there will always be some error. You can never completely remove the error, but you can make it arbitrarily small with an arbitrarily large mantissa.


or we just create a struct doubledouble and stack 2 double values in it, and get 2x precision bits
Post 02 Feb 2012, 21:38
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.