flat assembler
Message board for the users of flat assembler.
 Home   FAQ   Search   Register 
 Profile   Log in to check your private messages   Log in 
flat assembler > Projects and Ideas > X86 Opcode and Instruction Reference

Goto page Previous  1, 2, 3  Next
Author
Thread Post new topic Reply to topic
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
heh, x86 is CISC, there are hunders of them. And the reference is still far from complete, all MMX and SSE* instructions are missing. They are coming soon though.
Post 07 May 2008, 17:25
View user's profile Send private message Visit poster's website Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
X86 Opcode and Instruction Reference 1.00: Massive update
Massive update has been made to the reference: All SSE, SSE1, SSE2, SSE3, and SSSE3 instructions added. And, by popular demand, I have added corresponding alphabetically sorted editions:

coder32-abc, coder64-abc, coder-abc

geek32-abc, geek64-abc, geek-abc

Store has been improved, and prices discounted.

Thanks to all contributors that helped me to complete this update!
Post 18 Oct 2008, 22:45
View user's profile Send private message Visit poster's website Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4634
Location: Argentina
MazeGen (or anybody else), do you have or planning to have a tool such that takes your XML reference and "attack" an assembler with all the possible combinations of each instruction to verify the correct encoding (and that all variants are accepted by the assembler)?

Tomasz currently is willing to have fasm heavily tested since he is going to release the 1.68 line soon so such tool would be great to have.
Post 25 Mar 2009, 03:45
View user's profile Send private message Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
I'm thinking about a database for disassemblers testing. It would actually be interesting to have something similar for assemblers testing. However, I can't promise I will do it in near future.

Hmm, and I can call for it in the list of possible contributions, someone could try it on his own perhaps.
Post 25 Mar 2009, 08:35
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2146
Location: Estonia
Okay, what I tried yesterday:
1) Take some ready-made XML-parser and make it work with FASM.
It failed because they were too heavy for this simple task
2) Tried writing my own - kind of succeeded, but
3) FASM.DLL is old and in 32-bit, all my code is 64-bit Smile

Now there are two possibilities:
a) I continue with my 64-bit code and hope for a 64-bit FASM release
b) Rewrite it to 32-bit. Now its easy because I haven't done much.
c) Ditch the project Wink
Post 26 Mar 2009, 07:51
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
Madis, Mikae from wasm.ru sent me in private an ELF parser written in fasm for x86reference.xml. It calls libxml2. I'm not sure how it works in detail, but if you think it can help I can ask him if I might publish it. He is also willing to port it to Windows, if it would be needed.
Post 26 Mar 2009, 08:17
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2146
Location: Estonia
That would be of great help. I think I need to fall back to 32-bit still, because I need a real-time compilation of FASM sources, like:

Code:

Starting FASM thorough testbed...
NOP - 90 | FASM=90 - OK!
INT3 - CC | FASM=CC - OK!
ADD EAX,1 - 83C001 | FASM=123456 - FAILED!
...etc...



Every line would be assembled and compared in real-time. Of course its not really necessary because I can make it in many stages, but its WAY COOLER Very Happy

There's always a possibility of using a platformless middleman (COM, pipes, etc.) and make multiple executables that communicate with each other.
Post 26 Mar 2009, 11:18
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
OK, I asked Mikae for the Windows version.

I still didn't get how exactly you're gonna make the test bed. You want to process the XML and generate an instruction along with its correct opcode automatically? That would be pretty difficult.

This is the first idea which came to my mind:

  1. Create an XSL transformation to output all possible valid operands of an instruction. Let's say we would use XML sorted by opcode. It would look like this:

    • add al, cl ; opcode 00
    • add [bx+si], cl ; opcode 00
    • add [eax], cl ; opcode 00
    • add cx, dx ; opcode 01
    • add [bx+di], dx ; opcode 01
    • add [bx+di], edx ; opcode 01
    • add [ecx], dx ; opcode 01
    • add [ecx], edx ; opcode 01
    • ...and so on for all other opcodes

  2. Assemble it with FASM and NASM at least to pure binary code
  3. Bindiff it to see differences
  4. Check the differences which points to FASM or NASM bugs (if they are not just two different valid encodings of the same instruction)
  5. Fix the bugs in binary code. Now we have binary test bed for future fasm releases


And we can't call 32-bit DLL from 64-bit code? I have never tried it, but it should work.
Post 26 Mar 2009, 11:57
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6602
Location: Kraków, Poland

MazeGen wrote:
[*]Check the differences which points to FASM or NASM bugs (if they are not just two different valid encodings of the same instruction)


What if they both have the same bug? Wink
I remember that I used HIEW to test the fasm output when writing first versions - and later I realized HIEW actually had some bugs and wasn't always so good authority to let me know that my output is correct.
Post 26 Mar 2009, 12:29
View user's profile Send private message Visit poster's website Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
I was testing my disassembler's output against OllyDbg and it was the same - there were more bugs in Olly than in my disassembler Smile

That's also why I'm working on a (web-based, at least in its first version) public disassembler, based on the x86reference.xml, which could become an authority.
Post 26 Mar 2009, 13:09
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2146
Location: Estonia
With bindiffs it becomes extremely hard to know WHERE and more precisely HOW the error happened. The FFFFh versus FFEFh are easy, but what if FASM decides to add a 66 prefix somewhere and the file is one byte bigger and all other bytes are shifted.

That even crossed my mind once, but then it wouldn't help us more than say a program that outputs boolean 1 or 0 (1=correct; 0=buggy) Razz
Post 26 Mar 2009, 13:40
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6602
Location: Kraków, Poland
Maybe put "align 32" after each instruction, or something like that. Wink
Post 26 Mar 2009, 13:41
View user's profile Send private message Visit poster's website Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
Madis, I know that bindiff is not optimal. I'm still not getting your way of making the test bed Smile
Post 26 Mar 2009, 13:57
View user's profile Send private message Visit poster's website Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4634
Location: Argentina
The steps I've imagined when I posted the idea:

Code:

foreach instruction in XML
  foreach operands in x86_64_set
    foreach mode in ('use16''use32''use64')
       clear('test.asm')
       delete('test.bin')
       writeln('test.asm'mode)
       writeln('test.asm'instruction + ' ' + operands)
       ; also write a variable of each type here for memory operands testing

       system('fasm test.asm'; Or something more sophisticated that could tell us if fasm crashed

       if !exist('test.bin') && isValid(XMLinstructionoperand)
         sig_error
       else if !isEncodingValid(XMLinstructionoperandmodefile_contents('test.bin'))
         sig_error




The operands set is all the supported operands in the instruction set (so stupid things like "nop xmm0, fp5, 9" would be tested). So it should test every instruction with zero, one, two and three operands and each operand would be tested with every reg available, some immediates (probably 0, 127, -128 and 255), and memory address combination.

Not sure if the XML reference covers enough info to do that but I believe it does (perhaps you need an external source to know ModR/M and SIB encodings?).
Post 26 Mar 2009, 15:20
View user's profile Send private message Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
Again, how would isEncodingValid work? We would need either an assembler based on the XML, or hand-written test bed. Both is doable, but not easy.

And yes, it should cover all possible mnemonics and all possible operand types and their combination.
Post 26 Mar 2009, 15:41
View user's profile Send private message Visit poster's website Reply with quote
Mikae



Joined: 01 Apr 2009
Posts: 1
Hello all.

MazeGen told me about this topic and asked Windows version of table parser, 'tgen'. First of all I have to say that I've written this parser for internal use and didn't intend to publish it. So it is kind of draft, it looks ugly and has more bugs than lines of code. Also, it supports only one byte instructions for now. Well, you are warned :).

tgen uses libxml2.dll (you can take it at http://xmlsoft.org). libxml2.dll in turn uses iconv.dll and zlib1.dll, which was absent on my fresh XP SP2 installation.

I've changed MazeGen's reference a little bit (commented out some redundant instruction definitions). The changed version is attached.

Windows version looks for x86referense.xml in the same folder it was started and has the following syntax: tgen.exe destination_file_name.

tgen doesn't handle errors correctly and uses int3 for debug. Hope, you will forgive me :).

I attach two versions: the first is pure tgen.asm and XML reference, the second contains tgen.asm, XML reference, libxml2.dll, iconv.dll and zlib1.dll. Woops. Max attach size is 700 Kb. tgen + libs is 1.2 Mb. Should I use rapidshare?

--
With Best Regards,
Mikae.


Description: Pure tgen.
Download
Filename: tgen_pure.zip
Filesize: 36.6 KB
Downloaded: 342 Time(s)

Post 01 Apr 2009, 23:37
View user's profile Send private message ICQ Number Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
Welcome on the board, Mikae Smile

As for the DLLs, they can be downloaded so I think you don't need to attach them here:

http://www.zlatkovic.com/pub/libxml/

Thank you for the Windows version, now I'm gonna look at it Smile
Post 02 Apr 2009, 08:15
View user's profile Send private message Visit poster's website Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia
tgen is working for me! That's good because not everyone knows XSLT (which could do the same job as tgen does). Thanks again, Mikae.

As for the testing, I really think that the only way to create the test bed is to generate it by FASM and NASM, bindiff the results, fix it (if needed) and translate it back, for example, to an XML of the following structure:


Code:
<entry><instr>add [eax], al</instr><opcd>0000</opcd></entry>

Post 03 Apr 2009, 13:14
View user's profile Send private message Visit poster's website Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2146
Location: Estonia
The first stage might be:
XML => MNEM DST,SRC
i.e. ADD Eb,Gb
(weather it be XSLT or custom-programmed parser)
The second stage takes all the lines and expands them further:
ADD [eax],al
ADD [0],al
... and other possible variants
(might be FASM macros, XSLT, or a parser, whichever is easier)
Third stage is to assemble it with FASM. The result should be binary-accurate with what XML dictates.

I hope its possible to take the XML as a source and output both *.BIN of all the assembled code and *.ASM code that is compatible with FASM and shall be sent through it to test its compliance with the *.BIN.
Post 04 Apr 2009, 12:56
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
MazeGen



Joined: 06 Oct 2003
Posts: 953
Location: Czechoslovakia

Madis731 wrote:
I hope its possible to take the XML as a source and output both *.BIN of all the assembled code and *.ASM code that is compatible with FASM and shall be sent through it to test its compliance with the *.BIN.


That is possible, however, it is not a task which could be done during one afternoon. Good luck Wink
Post 06 Apr 2009, 13:10
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3  Next

< Last Thread | Next Thread >

Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2005 phpBB Group.

Main index   Download   Documentation   Examples   Message board
Copyright © 2004-2016, Tomasz Grysztar.