flat assembler
Message board for the users of flat assembler.

Index > OS Construction > RFC: PEtoBIN - a simple file format for 32/64bit OS Dev

Author
Thread Post new topic Reply to topic
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 18 May 2011, 22:24
The simplest PE COFF file is a single segment flat read write executable. With FASM this is easy to create by
preceding the code with:
Code:
format PE    
This will create a 32 bit flat binary executable with the default Windows ORG of 401000h. This file contains
the 512 byte PE header (which includes the MZ stub) and requires no linking.
Code:
format PE64    
Will create a 64 bit executable. The ORG can be specified as follows:
Code:
format PE at 400000h    
Will set the ORG at 4 Mb for example. By stripping off the 512 byte header a flat binary executable will be
produced. This is where it gets interesting. By following the code with:
Code:
section '.reloc' fixups    
Will append a table of relocation information which can be used by the loader to adjust the absolute address
references so that the executable can be located at a location other than the default chosen. At location 0B4h
from the beginning of the header will be found the 32 bit preferred load address which can be appended as a
64 bit number second to last. At location 1B4h from the beginning of the header will be found the 32 bit relative
offset from the start of the header of this relocation table. By appending this value (minus 512) as a 64 bit
number to the end of the file and stripping off the 512 byte header a flat binary executable with relocation
information will be produced. This header-less binary file (either 32 bit or 64 bit) can be loaded and run from the
default ORG as is. Subsequent loads can be easily relocated by using the last 8 bytes of the file to locate the
relocation table. So the file would be:
    512 byte header to be stripped off
    headerless flat binary file
    relocation table
    64 bit preferred location (with no fixups required)
    64 bit offset from start of file to relocation table
A utility to create a PEtoBIN should be straight forward to create to produce a simple executable file format for
OS development.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com


Last edited by Mike Gonta on 21 May 2011, 15:41; edited 2 times in total
Post 18 May 2011, 22:24
View user's profile Send private message Visit poster's website Reply with quote
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 19 May 2011, 09:38
Assemble yourfile.asm as yourfile1.pe
Code:
format PE at 400000h ; code loaded to 4Mb+4Kb - room for "PSP" style in memory header 
; 512 byte header to be stripped off 
; your code here
  times 255 db 90h
section '.reloc' fixups 
; relocation table    
Then to convert to the proposed format, assemble yourfile1.asm as yourfile.bin
Code:
virtual at 0
  file 'yourfile1.pe':0, 512
  load x WORD from 98h ; PE format Magic Number
  if x=10Bh
    load preferred_address DWORD from 0B4h
    load relocation_offset DWORD from 120h
  else ; PE32+
    load preferred_address QWORD from 0B0h
    load relocation_offset DWORD from 130h
  end if
end virtual

  file 'yourfile1.pe':512
  load x DWORD from $-16
  if DWORD x=0
    store QWORD preferred_address at $-16
    store QWORD relocation_offset at $-8
  else
    dq preferred_address
    dq relocation_offset
  end if
checksum=0
repeat 15
  load x BYTE from $-(%+1)
  checksum=(checksum+x) mod 256
end repeat
store BYTE (256-checksum) at ($-1)    
EDIT: Correction as suggested by LocoDelAssembly.
EDIT: Changed back to PE setup with code loaded 4Kb after load address.
EDIT: Added small optimization to store pointers in existing file if last 2 QWORDS are not being used.
EDIT: Added simple byte checksum of the preferred address pointer and relocation table offset.
EDIT: Added "format PE64" (PE32+) conversion code.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com


Last edited by Mike Gonta on 23 May 2011, 23:24; edited 8 times in total
Post 19 May 2011, 09:38
View user's profile Send private message Visit poster's website Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4624
Location: Argentina
LocoDelAssembly 19 May 2011, 16:44
Mike, by using ORG you kill relocations. As a simple test, change "times 255 db 90h" to "times 10000 mov eax, $", then compile with and without the "org PREFERRED" line commented, you should see a huge size difference.
Post 19 May 2011, 16:44
View user's profile Send private message Reply with quote
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 19 May 2011, 22:06
LocoDelAssembly wrote:
Mike, by using ORG you kill relocations. As a simple test, change "times 255 db 90h" to "times 10000 mov eax, $",
then compile with and without the "org PREFERRED" line commented, you should see a huge size difference.
Hi LocoDelAssembly,
Thank you for pointing that out.
The FORMAT directive provides the "ORG" either by default or specified. The other thing that I overlooked is that
the executable is actually loaded and "ORG"ed 4K in.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com
Post 19 May 2011, 22:06
View user's profile Send private message Visit poster's website Reply with quote
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 21 May 2011, 15:41
I changed the title to RFC (Request For Comments).
This is a simple file format, simple to create (very simple with FASM) and can be used as a flat binary as well.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com
Post 21 May 2011, 15:41
View user's profile Send private message Visit poster's website Reply with quote
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 23 May 2011, 11:49
I've added a simple byte checksum of the preferred address pointer and relocation table offset to the last byte
of the file. This FASM runtime calculation is based on similar code from
Pinczakko's Guide to Self-patching Expansion ROM Source Code. The "format PE64" (PE32+) is limited to a 4 Gb
image size, so if using 64bit the upper 8 bits of the relocation table offset can simply be masked before using.
This provides a simple sanity check that the file is indeed a relocatable binary file.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com


Last edited by Mike Gonta on 23 May 2011, 19:36; edited 1 time in total
Post 23 May 2011, 11:49
View user's profile Send private message Visit poster's website Reply with quote
Dex4u



Joined: 08 Feb 2005
Posts: 1601
Location: web
Dex4u 23 May 2011, 17:55
Great work, I use a simple relocatable file format like this.

This is the main file assembled as a obj
Code:
;=========================================================;

; Test                                                    ;

;---------------------------------------------------------;

format COFF

 ;----------------------------------------------------;

 ; Code.                                              ;

 ;----------------------------------------------------;

Start:

 ;----------------------------------------------------;

 ; Data.                                              ;

 ;----------------------------------------------------;

msg  db 'Hello world!',13,0

 ;----------------------------------------------------;

 ; BSS goes here.                                     ;

 ;----------------------------------------------------;

align 4                                         ;-----+

Cut  db '2CUT'                                         ; These must be here
ModEnd:                  ; you would need this as a pointer to end of file in header 
    


You can add a header to the above

Then you run it though a win or linux app called coff2dex
http://www.dex-os.com/DownLoads/Coff2dex.zip

Its written by Tomasz Grysztar and moded by me
It turns the obj file into two files .dri (a bin file) and a .rel (a file with all the relocatable address that need fix in the bin file).

Then you can take the two file and do something like this
Code:
use32



         org    0x0



              db     'DIF1'                ; 4 byte id

              dd     File1                   ; relocatable file
           dd     File2                   ; bin file + relocatable size

            dd     File2Size-File2     ;  size

              dd     File2Size       ; info



File1:

file   'test.rel'                             ; relocatable

File2:

file   'test.dri'                              ; bin
File2Size:

    

And assemble it.

This is just a example, because you would mod coff2dex (coff2bin the original name) to do the above.

Then to load it anywhere in memory you would just load it in too a temp place, move the bin part to the place you want to run it, say 10MB, then using the reloctable part fix the address, by adding 10MB to the address at that offset.
Post 23 May 2011, 17:55
View user's profile Send private message Reply with quote
Mike Gonta



Joined: 26 Dec 2010
Posts: 243
Mike Gonta 23 May 2011, 20:21
Dex4u wrote:
I use a simple relocatable file format like this. This is the main file assembled as a obj
Hi Craig,
I chose PE over COFF for the following reasons:
    1. The relocation table needs no modification.
    2. Some may wish to link the OBJ and the result will be PE.
The PE relocation table entries requires a little extra work to implement in the loader since they are 12 bit
section offsets, but result in a smaller table size.
Dex4u wrote:
You can add a header to the above
This specification has no header (a "footer" serves the same purpose).
Dex4u wrote:
Then to load it anywhere in memory you would just load it in too a temp place, move the bin part to the place you
want to run it, say 10MB, then using the relocatable part fix the address, by adding 10MB to the address at that offset.
In this specification the image can be loaded in place without relocation.
The fix ups if required (if not loaded to the preferred address) can also be done in place.
The main thing is that the file can be loaded directly to the preferred address just as a flat binary.

_________________
Mike Gonta
look and see - many look but few see

https://mikegonta.com
Post 23 May 2011, 20:21
View user's profile Send private message Visit poster's website Reply with quote
bubach



Joined: 17 Sep 2004
Posts: 341
Location: Trollhättan, Sweden
bubach 24 Jan 2012, 15:58
Hi, very nice idea/work!

I've tried your code to make a small test executable and everything works so far, but what do I need to do in my OS to load it? If I for example load the file to 0x56789F what do I do (in my OS loading code) with that number and the table in order to patch up the code?

Also, is there a fixed size on the relocation data? I noticed there was a lot of unused space after my test code, mostly just zeroes.. Some other values in the middle, lots of zeroes again and then som more values before EOF.

Sorry I'm coming back to OS-dev from a years long break, and also have limited knowlegde on relocation in the first place so this might seem like basic questions? Confused

_________________
BOS homepage: http://bos.asmhackers.net/


Last edited by bubach on 24 Jan 2012, 19:53; edited 1 time in total
Post 24 Jan 2012, 15:58
View user's profile Send private message Reply with quote
cod3b453



Joined: 25 Aug 2004
Posts: 618
cod3b453 24 Jan 2012, 18:58
The PE header contains fields that map the file offsets to virtual addresses. If you compile 64bit code, it doesn't matter where you put it in memory (as long as the relative locations are the same) otherwise you have to match these up.

In general, you need to load the binary into memory to some temporary location, parse the header and copy the file blocks to their specified locations. The entry point field can be used to execute the program as normal. (Same for ELF or PE just different headers)
Post 24 Jan 2012, 18:58
View user's profile Send private message Reply with quote
bubach



Joined: 17 Sep 2004
Posts: 341
Location: Trollhättan, Sweden
bubach 24 Jan 2012, 19:52
I'm using this for 32-bit programs, and there's no header left with his method, only the relocation table - which I'm unsure of how to handle/parse.
Post 24 Jan 2012, 19:52
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3175
Location: Denmark
f0dder 24 Jan 2012, 20:31
What's the point?

It makes sense to keep an existing format (PE, ELF, whatever) since you can use existing tools for development and debugging. If you're so size-constrained that the little overhead of a full PE harms you, you should probably be doing direct binary output.

PE is entirely feasible even for your initial kernel image, you don't need your bootstrap PE loader to support the full thing. You can easily load to a fixed physical offset and use some fixed PE field offsets to get to the initial code - if you need the kernel to be relocated, you can load to <whatever> physical offset, and have position-independent code in the kernel do the relocation and other necessary PE fixups, keeping the bootloader small and sweet.
Post 24 Jan 2012, 20:31
View user's profile Send private message Visit poster's website Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 24 Jan 2012, 20:39
bubach wrote:
Hi, very nice idea/work!

I've tried your code to make a small test executable and everything works so far, but what do I need to do in my OS to load it? If I for example load the file to 0x56789F what do I do (in my OS loading code) with that number and the table in order to patch up the code?

Also, is there a fixed size on the relocation data? I noticed there was a lot of unused space after my test code, mostly just zeroes.. Some other values in the middle, lots of zeroes again and then som more values before EOF.

Sorry I'm coming back to OS-dev from a years long break, and also have limited knowlegde on relocation in the first place so this might seem like basic questions? Confused

Welcome back man! Smile
Post 24 Jan 2012, 20:39
View user's profile Send private message Reply with quote
bubach



Joined: 17 Sep 2004
Posts: 341
Location: Trollhättan, Sweden
bubach 24 Jan 2012, 23:03
Well it's kind of obvious I need to read up on the PE format and figure this thing out myself.

But the reason why is simple, I don't want to use someones elses file-format, and there's no need for anything else then the relocation-table. Just for the sake of having my own OS signature/header in all applications it's worth the extra work. Everyone have diffrent goals, mine isn't to follow in everybody elses footsteps or conform to existing formats.

Hi Smiddy, yes it's been far too long! Glad to be back and eager to get a fileformat/fat12 thing working so I can finally port fasm! Smile


Last edited by bubach on 01 Feb 2012, 21:24; edited 1 time in total
Post 24 Jan 2012, 23:03
View user's profile Send private message Reply with quote
smiddy



Joined: 31 Oct 2004
Posts: 557
smiddy 25 Jan 2012, 00:00
Nice! I'll drop by your site in the next day or two.
Post 25 Jan 2012, 00:00
View user's profile Send private message Reply with quote
Dex4u



Joined: 08 Feb 2005
Posts: 1601
Location: web
Dex4u 25 Jan 2012, 00:45
Something simular to your idea as already been code for DexOS, its not compatable with the latest ver.
But it should give you a good start

Its demos loading a PE file format.


Description:
Download
Filename: DEX4URELOC.ZIP
Filesize: 37.75 KB
Downloaded: 551 Time(s)

Post 25 Jan 2012, 00:45
View user's profile Send private message Reply with quote
bubach



Joined: 17 Sep 2004
Posts: 341
Location: Trollhättan, Sweden
bubach 25 Jan 2012, 01:34
Thats cool, thanks! Very Happy
When I get it working I'll patch my fasm-port to output the format directly Wink
Post 25 Jan 2012, 01:34
View user's profile Send private message Reply with quote
Dex4u



Joined: 08 Feb 2005
Posts: 1601
Location: web
Dex4u 25 Jan 2012, 03:37
bubach wrote:
Thats cool, thanks! Very Happy
When I get it working I'll patch my fasm-port to output the format directly Wink

Cool, look forward to testing it Very Happy
Post 25 Jan 2012, 03:37
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.