Why cant I add floating point number right

Index > Main > Why cant I add floating point number right

Author

Thread

NanoBytes

Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States

NanoBytes

I am trying to add 12 to 0.2, but it keeps returning 12.199... and I cant figure out how to get it to work

Code:

        mov [Integer1],12
        FILD [Integer1]
        mov [Integer1],2
        FILD [Integer1]
        MOV [Integer1],10
        FIDIV [Integer1]
        FADDP st1,st0

_________________
He is no fool who gives what he cannot
keep to gain what he cannot loose.

02 Feb 2012, 03:09

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 18472
Location: In your JS exploiting you and your system

revolution

12.2 cannot be perfectly represented as a floating point number. The FPU will use the closest approximation available 12.199999... Generally one would round the numbers to the required precision for display and the small error in the internal representation would not be seen.

02 Feb 2012, 03:20

NanoBytes

Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States

NanoBytes

Ahh, i think i figured it out

Code:

        MOV [Integer1],15
        FILD [Integer1]
        MOV [Integer1],2
        FILD [Integer1]
        MOV [Integer1],10
        FIDIV [Integer1]
        FSUBP st1,st0
        FSTP [Integer1]
        BTS [Integer1],2
        FLD [Integer1]

Though, i am worried, I am converting from the registers 10 bytes to the integers 4 bytes just to correctly add two numbers. Is there a more accurate way to do this?

_________________
He is no fool who gives what he cannot
keep to gain what he cannot loose.

02 Feb 2012, 03:39

NanoBytes

Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States

NanoBytes

Ok, revolution, how would i round the entire number off?

02 Feb 2012, 04:01

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 18472
Location: In your JS exploiting you and your system

revolution

NanoBytes wrote:

Ok, revolution, how would i round the entire number off?

Rounding is usually only done for display purposes. During internal calculations rounding is generally not done, and is usually not needed.

02 Feb 2012, 04:16

NanoBytes

Joined: 02 Jun 2011
Posts: 57
Location: Iowa, United States

NanoBytes

I know, I am working on a atof procedure

02 Feb 2012, 05:35

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 18472
Location: In your JS exploiting you and your system

revolution

https://en.wikipedia.org/wiki/Rounding

02 Feb 2012, 05:52

JohnFound

Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria

JohnFound

Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.

02 Feb 2012, 06:46

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 18472
Location: In your JS exploiting you and your system

revolution

JohnFound wrote:

Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.

The FPU cannot store enough precision for that.

The closest is 12.19999999999999929
The next value is 12.20000000000000107

02 Feb 2012, 09:40

smiddy

Joined: 31 Oct 2004
Posts: 557

smiddy

revolution wrote:

JohnFound wrote:
Note that from mathematical point of view 12.199(9) = 12.2 and this is exact equation.
The FPU cannot store enough precision for that.

The closest is 12.19999999999999929
The next value is 12.20000000000000107

That's interesting! What is the limiting factor causing the precision error?

02 Feb 2012, 13:00

JohnFound

Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria

JohnFound

Quote:

That's interesting! What is the limiting factor causing the precision error?

The registers size of course.

02 Feb 2012, 13:18

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 18472
Location: In your JS exploiting you and your system

revolution

smiddy wrote:

That's interesting! What is the limiting factor causing the precision error?

We can get more precision by using a larger mantissa size, but since the storage is binary and the output is decimal then there will always be some error. You can never completely remove the error, but you can make it arbitrarily small with an arbitrarily large mantissa.

02 Feb 2012, 13:33

Matrix

Joined: 04 Sep 2004
Posts: 1166
Location: Overflow

Matrix

revolution wrote:

smiddy wrote:
That's interesting! What is the limiting factor causing the precision error?
We can get more precision by using a larger mantissa size, but since the storage is binary and the output is decimal then there will always be some error. You can never completely remove the error, but you can make it arbitrarily small with an arbitrarily large mantissa.

or we just create a struct doubledouble and stack 2 double values in it, and get 2x precision bits

02 Feb 2012, 21:38

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum