very small floating point errors at the out of range boundry
I have written a string to ,half float, float, double and extended double, conversion routine as a class project. I was testing it against the floating point converters in flat assembler when I found a small difference in the way flat assembler handles the under flow condition and the way my Intel processor handled it.
If the exponent is zero and the significand is zero flat assembler will issue an out of range error. I think this test is performed before the floating point number is rounded. There are a few cases where rounding will cause the significand to round up to 1, which should prevent the out of range message.
dw 5.959E-8 ;causes an out of range error
I created a float:
dd 5.959e-8 ;This float was converted to half precision using VCVTPS2PH with
the round code set to 0. One will get a half precision value of 0x0001
dd 1.4012984220e-45 ;causes an out of range error
I created a double:
dq 1.4012984220e-45 ;this double was converted to a float using CVTPD2PS with the flags set for rounding. The floating point value created by the conversion was 0x00000001
dq 4.940656458e-324 ;will cause an out of range error. With rounding set it should convert to 0x0000000000000001
You cannot post new topics in this forum You cannot reply to topics in this forum You cannot edit your posts in this forum You cannot delete your posts in this forum You cannot vote in polls in this forum You cannot attach files in this forum You can download files in this forum