The idea is to use the invalid opcode exception handler to emulate unsupported instructions or to implement custom opcodes. Unsupported opcodes could then be used by emulating them in software. Custom opcodes could be implemented using the ud2 opcode as a prefix, (the purpose of ud2 is to raise such exceptions).

The basic flow would be
1 execute invalid opcode (raises exception)
2 execute exception handler
3 get address of invalid opcode
4 decode the opcode
5 update the return address
6 emulate the opcode in software
7 return to modified return address

Is this possible, and if so what would be required to make it work?

using fs register.
put the handler address in the first 4 bytes of FS and make sure you save it first.
In your handler get the last eip and read the bytes there and dump them.

Can you tell us what the emulated instructions would do? VIA, AMD on Intel and vice versa? Maybe some really exotic instructions?

Would be nice to run some SIMD on a 386.

This looks Win32-specific

> Would be nice to run some SIMD on a 386.

COOL (IIRC there are FPU emulators and MMX emulators out ...)