flat assembler
Message board for the users of flat assembler.

Index > Compiler Internals > very small floating point errors at the out of range boundry

Author
Thread Post new topic Reply to topic
WereMole



Joined: 28 May 2016
Posts: 5
WereMole 06 Jan 2017, 16:07
I have written a string to ,half float, float, double and extended double, conversion routine as a class project. I was testing it against the floating point converters in flat assembler when I found a small difference in the way flat assembler handles the under flow condition and the way my Intel processor handled it.

If the exponent is zero and the significand is zero flat assembler will issue an out of range error. I think this test is performed before the floating point number is rounded. There are a few cases where rounding will cause the significand to round up to 1, which should prevent the out of range message.

;---------
dw 5.959E-8 ;causes an out of range error

I created a float:

dd 5.959e-8 ;This float was converted to half precision using VCVTPS2PH with
the round code set to 0. One will get a half precision value of 0x0001
;---------

dd 1.4012984220e-45 ;causes an out of range error

I created a double:

dq 1.4012984220e-45 ;this double was converted to a float using CVTPD2PS with the flags set for rounding. The floating point value created by the conversion was 0x00000001
;----------

dq 4.940656458e-324 ;will cause an out of range error. With rounding set it should convert to 0x0000000000000001

;------------------
Post 06 Jan 2017, 16:07
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8376
Location: Kraków, Poland
Tomasz Grysztar 06 Jan 2017, 16:23
Yes, it seems that there is a bug in fasm 1 there. The same instructions assembled with fasmg give the results you expected.
Post 06 Jan 2017, 16:23
View user's profile Send private message Visit poster's website Reply with quote
idle



Joined: 06 Jan 2011
Posts: 440
Location: Ukraine
idle 06 Jan 2017, 18:04
i remember fasm raising errorrs with ZEROeETC, should be fixef too if it is not yet
Post 06 Jan 2017, 18:04
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8376
Location: Kraków, Poland
Tomasz Grysztar 20 Jan 2017, 14:48
This "out of range" error is corrected in fasm 1.71.59.
Post 20 Jan 2017, 14:48
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.