flat assembler
Message board for the users of flat assembler.

Index > Heap > Simple maths question? Just ask Google, NOT!

Author
Thread Post new topic Reply to topic
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17278
Location: In your JS exploiting you and your system
revolution
What is 399999999999999-399999999999998 = ?

See what google thinks the answer is

Oops.
Post 25 Sep 2008, 09:19
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
Oh, that's just one significant figure (from the left, of course).
399999999999999-399999999999998=000000000000001

_________________
¯\(°_o)/¯ unlicense.org
Post 30 Sep 2008, 05:40
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
Post 30 Sep 2008, 06:13
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17278
Location: In your JS exploiting you and your system
revolution
It is interesting to analyse this error. The numbers are easily represented in DP format with 4 bits to spare.
Code:
399999999999997.0 --> 0x42F6BCC41E8FFFD0
399999999999998.0 --> 0x42F6BCC41E8FFFE0
399999999999999.0 --> 0x42F6BCC41E8FFFF0    
So it would seem that the conversion from decimal to float is not properly keeping the least significant digits.
Post 30 Sep 2008, 07:12
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
Maybe there is another intermediate step between the browser and conversion to machine type? ...or the lower bits are used as flags internally? (done that before)

I would have thought GMP could be put on the backend (with some resource limitations).

_________________
¯\(°_o)/¯ unlicense.org
Post 30 Sep 2008, 15:20
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17278
Location: In your JS exploiting you and your system
revolution
Curious, since 399999999999999.9375-399999999999998 = 1.9375.

So the least significant digits/bits are all there and it gives the correct answer also.

Confused
Post 30 Sep 2008, 17:20
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE
399 999 999 999 999.9375 - 399 999 999 999 998.1 = 1.8125
399 999 999 999 999.9375 - 399 999 999 999 998.2 = 1.75
399 999 999 999 999.9375 - 399 999 999 999 998.3 = 1.62500
399 999 999 999 999.9375 - 399 999 999 999 998.4 = 1.5625
399 999 999 999 999.9375 - 399 999 999 999 998.5 = 1.4375
399 999 999 999 999.9375 - 399 999 999 999 998.6 = 1.3125
399 999 999 999 999.9375 - 399 999 999 999 998.7 = 1.25
399 999 999 999 999.9375 - 399 999 999 999 998.8 = 0
399 999 999 999 999.9375 - 399 999 999 999 998.9 = 0

should be possible to isolate the exact bits in error

_________________
¯\(°_o)/¯ unlicense.org
Post 30 Sep 2008, 20:23
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
Just use Pari and you're done Razz
Post 30 Sep 2008, 20:54
View user's profile Send private message Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4237
Location: 2018
edfed
it can be interresting to have the source code of this calculator no?
Post 01 Oct 2008, 21:32
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.