Having recently been using the assembler for 64 bit calculations (see RSA) I came across a problem that reminded me of the TASM 33 bit values.

Of course TASM is a 32 bit assembler but it always kept values to a precision of 33 bits internally. The reason for this was the sign of the result. Example:

In TASM (32 bit assembler)

(0FFFFh * 0FFFFH)/0FFFFh = 0FFFFh

Internally (0FFFFh * 0FFFFH) = 0FFFE0001h with sign positive and the the division by 0FFFFh gives the expected answer.

In FASM (64 bit assembler):

(0FFFFFFFFh * 0FFFFFFFFh)/0FFFFFFFFh = 0FFFFFFFFFFFFFFFEh

Whoops, what happened? Internally (0FFFFFFFFh * 0FFFFFFFFh) = 0FFFFFFFE00000001h, a negative number. Fasm reports no error when we multiply two positive numbers and get a negative result. Indeed FASM continues on with the divide and makes the result negative to match the sign or the numerator.

To avoid unexpected results perhaps we can have one extra bit that stores the sign of the result for arithmetic calculations? Is this possible to do in FASM without major modifications? Perhaps instead, if adding and extra sign bit is not easily done, FASM can report an overflow error when a multiplication produces and unexpected sign change?

Just thought I should point out that shortly after the message above was written the internal fasm number handling was changed. The new behaviour is that currently fasm does generate an error if two positive numbers are multiplied together and a negative result is encountered.

Things might be changing so the above behaviour might soon no longer be true.