Index
> Main > FPU: Rounding and Conversion |
| Author |
|
|
LocoDelAssembly 03 Dec 2006, 16:15
AMD64 Architecture Programmer’s Manual Volume 1: Application Programming wrote: Table 6-2. Types of Rounding You have two options, set the RC to the kind of operation you want to do or seting RC to round toward zero and add/subtract 0.5 before frndint I have no info for you second question, sorry |
|||
03 Dec 2006, 16:15 |
|
|
Jack 04 Dec 2006, 01:13
what I did once was to parse a numeric string to form a string that represents the number in scientific format, then character by character convert the mantissa as a bcd integer, then use fbld to load the integer and then divide it by 10^exponent, you can do 18 digits that way if you want that extra digit you will have to do a little more work.
to go the other way, find the approximate exponent of 10 by multiplying the number by log10(2) then multiply your number by 10^exponent, most of the time that works but sometimes you have to additionally multiply again by 10, once you have the integer store it with fbstp and convert the bcd number to string. or you could just use Raymond's fpu lib http://www.ray.masmcode.com/ |
|||
|
04 Dec 2006, 01:13 |
|
|
cod3b453 09 Dec 2006, 20:49
ok got ceiling and floor sorted, turns out RC = 00 works just adding / subracting 0.5.
I'm still looking through Raymond's fpu lib, but hopefully I'll get that working soon thanks guys! |
|||
|
09 Dec 2006, 20:49 |
|
|
Goplat 09 Dec 2006, 21:21
cod3b453 wrote: ok got ceiling and floor sorted, turns out RC = 00 works just adding / subracting 0.5. That will give wrong answers on odd integers. 5 + 0.5 = 5.5 which rounds to 6. 5 - 0.5 = 4.5 which rounds to 4. |
|||
|
09 Dec 2006, 21:21 |
|
|
cod3b453 13 Dec 2006, 23:10
Goplat wrote:
thanks Goplat i must have not tested odd integers  took me a while to work this out but I ended up add an integer check so that all integers are kept the same, other math tricks didn't seem to work.
Code: tst dt 5.0 val dt 0.0 hlf dt 0.5 fld tword [tst] ; st0 = N fld st0 ; st0 = N; st1 = N frndint ; st0 = int[N]; st1 = N fsub st1,st0 ; st0 = int[N]; st1 = N - int[N] fxch st1 ; st0 = N - int[N]; st1 = int[N] fldz ; st0 = 0; st1 = N - int[N]; st2 = int[N] fcomip st0,st1 ; st0 = N - int[N]; st1 = int[N] fwait faddp st1,st0 ; st0 = N jz @f fld tword [hlf] ; st0 = ½; st1 = N fsubp st1,st0 ; st0 = N - ½ frndint ; st0 = int[N - ½] @@: fistp qword [val] now that this seems to work properly i can use the log10(N) trick which uses the floor function for the exponent. |
|||
|
13 Dec 2006, 23:10 |
|
< Last Thread | Next Thread > |
Forum Rules:
|