I accidentally came across this little snippet: https://twitter.com/VK_Intel/status/897206671231131649.*
And it made me smile, because it is so reminiscent of the tricks I've been using back in early 2000's when playing with first versions of my 32-bit unreal mode, where the interrupt handler needed to detect if the code segment is 32-bit or 16-bit and deal with it accordingly. You can read about it in my unREAL mode article.

Unlike 16/32, I never needed to do a similar thing to distinguish 32-bit from 64-bit. The only obvious application that comes to my mind would be malware (though the world of code is complex out there, so I do not rule it out that someone might use it for other obscure purpose). But still, this one made me think a little. I wondered if I could make a variant using the same principle as the original 16/32 "distinguisher" I made for my unreal mode.

You see, the one I made for interrupt handler did it without altering any register. The flags are stored automatically on the stack when interrupt is initiated, so I did not have to worry about them. So I used the CMP instruction, though it was not my intention to use the result for a conditional jump. The trick was that CMP used immediate operand, which was decoded as either 16-bit or 32-bit depending on mode, so there was a difference of two bytes in the interpreted instruction length, and two bytes is exactly what is needed to fit the "JMP SHORT" instruction. This jump would be seen in 16-bit mode, but in 32-bit one it would become part of the immediate of the dummy CMP instruction. It was a "conditional unconditional" jump.

We cannot use the exact same trick to distinguish 64-bit from 32-bit, as they both use the same immediate sizes for instructions like CMP. We could try CMP with a memory operand, because 67h prefix does switch to 16-bit addressing in 32-bit mode and to 32-bit addressing in 64-bit mode. The problem is, the memory address could be invalid. I can use LEA to avoid the exception, but this is going to trash a register:

So, this variant uses the same principle, hiding a short jump within a code of dummy instruction, but this time I could not avoid altering a register.

Wait a second, in modern CPUs we have that NOP instruction with a dummy memory operand, which was created to be used in alignments... Now, this is byte longer, but it preserves all the registers and even flags. If we needed an interrupt handler that would work the same in 32-bit and 64-bit mode... Wait, what?!

Have a nice day, everyone!


___
* This is neither the original nor the best reference. Digging deeper leads to this place, among others: https://stackoverflow.com/questions/38063529/x86-32-x86-64-polyglot-machine-code-fragment-that-detects-64bit-mode-at-run-ti/. But I kept the first one just because it mentions fasm.

Nice trick.

This is pretty cool. I think you can also switch between modes (16/32/64) in user mode programs with CS register, but of course you won't be able to use libraries. (also, it depends on the OS)

You can't get all three together though. IIRC only the 16/32 and 32/64 combinations are possible. No 16/64 or 16/32/64.

Doesn't that depend on the OS? Also, shouldn't it be possible 100% all 3 of them? I mean, all 3 modes are valid in Long Mode, so there has to be a way to switch to it (even if, obviously, undocumented and unreliable between OS versions). Assuming it's not during kernel mode, which wouldn't make much sense to me (I mean, does Linux have 16-bit code in kernel? because it can run 16-bit Windows apps with Wine; Windows' limitation of lack of 16-bit in 64-bit comes from its APIs due to HANDLEs, nothing to do with the CPU).

https://en.wikipedia.org/wiki/Long_mode here it says it can run 16-bit protected mode code, I've never actually fiddled with this stuff, so maybe I'm assuming something stupid (maybe Linux does have special cases for Wine?)

Interesting read: https://www.dkia.at/en/node/180 seems you can

I consulted the old manuals to be sure - yes, both AMD and Intel specifications allow D bit to switch to 16-bit code in the compatibility mode. But the Virtual 8086 mode does not work (perhaps because it messes too much with the addressing). So it should be possible to run 16-bit protected mode applications (like the ones for Windows 3), but not DOS programs, as they would require V86 mode.

So, to continue the (pointless?) exercise, the trick can be easily extended to a three-way switch: And if we allowed the code to destroy some registers and flags, there would be many other interesting options, perhaps: Hmm, I'm not sure if I like this one.

This method is also closely related to the problem of having a snippet that executes as different instructions depending on what model of CPU is it run on. I'd been thinking that this was the thing of the past that died with the advent of CPUID and reliably signalled #UD, but recently I was reminded that there are instructions interpreted by Intel 64 differently than by the original AMD's x86-64.

So when consulting that 16-bit thing I went to both manuals, it is better to check twice, sometimes an interesting discovery may be made.

But wasn't push with 16-bit operand supposed to be valid in 64-bit mode? I mean it specifically said you can't encode 32-bit but instead 16-bit pushes.

It was in the original x86-64 specification by AMD. But when Intel made its own version, they made some changes, like removing the 16-bit pushes and jumps. And at the same time they added 10-byte immediate far jump that AMD was missing.

This board has a long history, if you dig deep enough you may find discussion of these details from the time when 64-bit instructions were being implemented into fasm.

My ideas presented here became a basis for my article published in the first issue of Paged Out! magazine.

In the second issue you may find an article by hh86 about techniques used to construct a x86/x64 hybrid shellcode. It contains several other clever tricks (including one based on the opcode for ARPL becoming MOVSXD in 64-bit mode). I highly recommend reading this excellent article (and while you're at it, you can also take a look at another, unrelated one written by me, just two pages earlier).

After I published my article, I got an e-mail that directed my attention to a brilliant snippet by Martin Strimberg, which is a simple routine that returns width of the word in AL register. It works for all three modes: It is in a way beautiful, so I was quite impressed, even though my initial response was just that it is possible do get a similar result with much shorter code: This one requires a functioning stack, though.

You may be aware (certainly if you've been following my discussions of applying 2-adic arithmetic to assembly) that the lowest byte of the difference of two numbers only depends on their lowest bytes. This is why this should always return a correct value in AL - even if the stack pointer register used for calculation is of different size than the one that was affected by PUSH instruction. Note that it is possible to have 32-bit code segment with 16-bit stack (with PUSH using SP), or a 16-bit code segment with 32-bit stack (with PUSH using ESP). But the value in AL is not affected by these differences, as it only depends on the lowest byte of SP/ESP/RSP.