flat assembler
Message board for the users of flat assembler.

flat assembler > Examples and Tutorials > Planned file format tutorials


A tutorial for which format would you like me to write first?
MZ, start with the basics
16%
 16%  [ 3 ]
PE, the popular option
44%
 44%  [ 8 ]
ELF, it does not have as many tutorials
16%
 16%  [ 3 ]
COFF object, there is even less written about this one
5%
 5%  [ 1 ]
Mach-O, it is hard to even get a good specification for it
0%
 0%  [ 0 ]
OMF, just so that some support for it is finally written (at least for fasmg)
0%
 0%  [ 0 ]
Other?
16%
 16%  [ 3 ]
Total Votes : 18

Author
Thread Post new topic Reply to topic
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
As a warm-up before perhaps writing a general introduction to assembly language with the assistance of fasmg I consider writing a series of small tutorials explaining various executable and object formats.

The main tool used during the tutorials would be fasmg, for its capability to generate everything we would ever need, including x86 code with relocations. Generators for various formats already exist for fasmg, but the point of the tutorials would be to write them from scratch, learning as much about the format as possible during the process.

I would like to write at least one, to see how well would it be received, and then decide whether to continue with the series. What format would you like to see explained this way?
Post 10 Jul 2018, 21:31
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16106
Location: Squiddler's Patch
Since PE is the most commonly used format, then this is the place to start IMO. Some of the less used parts can be safely ignored, also IMO.

MZ is "simple" of course, but many people can't easily run the 16-bit programs on their 64-bit OSes.
Post 11 Jul 2018, 01:14
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 207
Location: Russian Federation, Sochi
I can`t say anything against PE as first one. So maybe elf be second (in case of elf: libraries are more interest then executables itself). And Mach-O third.
Post 11 Jul 2018, 05:31
View user's profile Send private message Send e-mail Reply with quote
Ali.A



Joined: 08 Jan 2018
Posts: 130
Other: .sys (device driver)
Post 11 Jul 2018, 20:01
View user's profile Send private message Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
Ali.A wrote:
Other: .sys (device driver)
Device drivers nowadays use PE format, nothing else.

Well, I could also put LE (which was used for Win9x device drivers among other things) on the list next to OMF. Both are mostly irrelevant today.
Post 11 Jul 2018, 20:28
View user's profile Send private message Visit poster's website Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 540
Isn't MZ / COFF part of PE structure? You can''t explain one without involving the other(s). I mean there's a COFF header somewhere in a PE format.
Post 21 Jul 2018, 20:48
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
fasmnewbie wrote:
Isn't MZ / COFF part of PE structure? You can''t explain one without involving the other(s). I mean there's a COFF header somewhere in a PE format.
You should definitely take a look some of my recent tweets. Wink

I am already midway through with writing the first chapter of the tutorials, about PE. I use a dummy stub, so I do not need to explain much about MZ. In fact, I could probably get away with explaining next to nothing, but to keep up appearances I at least set up the stub length fields properly, so the stub appears valid even though it has no actual code.

As for the COFF, of course I mention it, as it is the main source of some of strange and/or obsolete values, but PE really does use only a few of the COFF's structures and the linkable object in COFF format is very, very different from an executable PE image. Things like relocations are done completely differently - obviously for linkable object relocations are much more complex. PE does not use a symbol table, which is essential for object COFF, etc. etc. Therefore COFF by itself deserves another chapter. Though I think that when I'm finished with PE I'm going to go for ELF first.
Post 21 Jul 2018, 21:23
View user's profile Send private message Visit poster's website Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 540
It will be a useful read indeed. I am eager to know how, for example, one can change the structure of a PE format and still get away with it without the OS complaining.

For example, GoLink managed to put the "PE" signature at offset 0x60 rather than the usual 0x80 as specified.

Interesting.
Post 21 Jul 2018, 21:42
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
fasmnewbie wrote:
For example, GoLink managed to put the "PE" signature at offset 0x60 rather than the usual 0x80 as specified.
The example constructed in my tutorial has it on offset 0x40. Wink

If the moving offset of PE looked strange to you, please try this: take the source code of FASMW and replace the first line (FORMAT directive) with:
Code:
format PE GUI 4.0 large NX on '..\fasmd\fasmd.exe'    
Before assembling it you need to have assembled FASMD.ASM in its directory, too.
What you get is a dual executable, it should show up as FASMD when run in DOS and as FASMW when run in Windows.
Post 21 Jul 2018, 21:49
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 444
Location: Belarus
Looking forward to read whatever will be the result. Tweet about the source of 0x10B value is the kind of things I love most in such texts: the way history shines through modern abstractions and formats.
Post 21 Jul 2018, 23:00
View user's profile Send private message Visit poster's website Reply with quote
moveax41h



Joined: 18 Feb 2018
Posts: 47
fasmnewbie wrote:
It will be a useful read indeed. I am eager to know how, for example, one can change the structure of a PE format and still get away with it without the OS complaining.

For example, GoLink managed to put the "PE" signature at offset 0x60 rather than the usual 0x80 as specified.

Interesting.


You and a few others here may be interested in some related work I did in this area: https://toddcullumresearch.com/2017/07/16/portable-executable-file-corruption/

_________________
-moveax41h
Post 25 Jul 2018, 17:34
View user's profile Send private message Reply with quote
catafest



Joined: 05 Aug 2010
Posts: 111
I think a good tutorial / tutorials about FASM can be start with this title:
"How to manage a big project with FASM" in order to have/see the flow of good ways of programming with FASM.
Post 10 Aug 2018, 17:14
View user's profile Send private message Visit poster's website Yahoo Messenger Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
I have the direction established already, and it is one I'm quite satisfied with. There is so much to say and it is all about things that I really wanted to write about.

Also, the order of "PE, then ELF" turned out to be an excellent choice, it allows me to introduce features of fasmg quite slowly (my hope is to not intimidate with them if a reader happens to be interested more in the experimentation with binary formats than in fasmg).
Post 10 Aug 2018, 18:43
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 207
Location: Russian Federation, Sochi
I impressed that "#" used so rarely and in most cases dot used for concatenation tokens.
Looks like tutorial about PE for 32bit address machines (no matter 32bit or 64bit they are, alpha was 64bit on PE32, and PE everywhere PE) is done, stayed PE version for machines with 64bit address.
--------------------------------------------------------------------------------------
To catafest: would thou like to start such tutorial thourself?
Any big project is constructor - part 1 project management: thou have to split it on related logical bloccks, some blocks could be portable, some not. And some of such blocks could be already created by other people (includes or external libraries). Than thou can build skelet of thour future program - working executable in thour target OS.
part 2 process management - thou have to determine in what stage thour application could be - with every stage related some piece of data process operated on and data that stayed unchanged for current stage.
part 3 data and memory management - thou have to determine what structures thou needed, how they will be related, how they will be stored (static, stack, heap, registers, or etc.) in stages that thou determine earlier.
something like that...

start thour big project and share ideas or experience on the fly.
What thou want to create? If it will be interest people can join to thou.
Post 10 Aug 2018, 21:54
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
ProMiNick wrote:
Looks like tutorial about PE for 32bit address machines (...) is done, stayed PE version for machines with 64bit address.
I still plan to at least touch the subject of resources before I go to PE+.
Post 10 Aug 2018, 22:16
View user's profile Send private message Visit poster's website Reply with quote
bzt



Joined: 09 Nov 2018
Posts: 9
Hi,
ProMiNick wrote:
stayed PE version for machines with 64bit address.

To my best knowledge there's no such thing. I've written a bootloader for long mode kernels in PE format, and both x86_64 and AArch64 PE/COFF had 32 bit addresses which I had to sign-extend manually to 64 bit addresses (keeping the sign required for higher half only). It was just the code segment that had 64 bit instructions, that's all. I believe that's the reason why the format is called PE32+ and not PE64.

Cheers,
bzt
Post 14 Nov 2018, 14:33
View user's profile Send private message Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 6981
Location: Kraków, Poland
bzt wrote:
ProMiNick wrote:
stayed PE version for machines with 64bit address.

To my best knowledge there's no such thing. I've written a bootloader for long mode kernels in PE format, and both x86_64 and AArch64 PE/COFF had 32 bit addresses which I had to sign-extend manually to 64 bit addresses (keeping the sign required for higher half only). It was just the code segment that had 64 bit instructions, that's all. I believe that's the reason why the format is called PE32+ and not PE64.

PE+ as a format is able to handle 64-bit VAs fine. Only the RVAs are still limited to 32 bits.

I have already written about this in the tutorial, too.
Post 14 Nov 2018, 16:09
View user's profile Send private message Visit poster's website Reply with quote
edfed



Joined: 20 Feb 2006
Posts: 4166
Location: 2018
any file format is ok, i am sure there are common factors between them
Post 16 Nov 2018, 22:24
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2018, Tomasz Grysztar.

Powered by rwasa.