flat assembler
Message board for the users of flat assembler.

Index > Unix > Mach-O executables made with fasmg

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 10 Aug 2017, 18:27
I have finally been able to write some macros for fasmg that allow creation of Mach-O executables. I initially thought this could be similar to ELF, where creation of executable files is relatively simple and the main complexity is in a preparation of object files for linking. But now I feel that is is actually the opposite - the object Mach-O may be not that different from ELF, so the analogous macros may be not that hard to create. On the other hand, the Mach-O executable files contain a lot of nuance in the structures for loader and dynamic linker. Consider also that some of the structures and requirements were changing from one version of MacOS to another, and this really adds up to an unfriendly environment.

There are some challenges that still require a lot of work. I would welcome any help if any of you would like to experiment with this.

Let me present the macros I have prepared so far. First, a bunch of equates extracted from various header file. We are going to need many of them:
Code:
CPU_TYPE_ANY            = -1
CPU_ARCH_ABI64          = 0x1000000
CPU_TYPE_VAX            = 1
CPU_TYPE_ROMP           = 2
CPU_TYPE_NS32032        = 4
CPU_TYPE_NS32332        = 5
CPU_TYPE_MC680x0        = 6
CPU_TYPE_I386           = 7
CPU_TYPE_X86_64         = CPU_TYPE_I386 or CPU_ARCH_ABI64
CPU_TYPE_MIPS           = 8
CPU_TYPE_NS32532        = 9
CPU_TYPE_HPPA           = 11
CPU_TYPE_ARM            = 12
CPU_TYPE_MC88000        = 13
CPU_TYPE_SPARC          = 14
CPU_TYPE_I860           = 15
CPU_TYPE_I860_LITTLE    = 16
CPU_TYPE_RS6000         = 17
CPU_TYPE_MC98000        = 18
CPU_TYPE_POWERPC        = 18
CPU_TYPE_POWERPC64      = CPU_TYPE_POWERPC or CPU_ARCH_ABI64
CPU_TYPE_VEO            = 255

CPU_SUBTYPE_MASK        = 0xff000000
CPU_SUBTYPE_LIB64       = 0x80000000

CPU_SUBTYPE_I386_ALL    = 3
CPU_SUBTYPE_X86_64_ALL  = CPU_SUBTYPE_I386_ALL
CPU_SUBTYPE_386         = 3
CPU_SUBTYPE_486         = 4
CPU_SUBTYPE_486SX       = 4 + 128
CPU_SUBTYPE_586         = 5
CPU_SUBTYPE_PENT        = 5 + 0 shl 4
CPU_SUBTYPE_PENTPRO     = 6 + 1 shl 4
CPU_SUBTYPE_PENTII_M3   = 6 + 3 shl 4
CPU_SUBTYPE_PENTII_M5   = 6 + 5 shl 4
CPU_SUBTYPE_PENTIUM_4   = 10 + 0 shl 4

MH_OBJECT      = 0x1
MH_EXECUTE     = 0x2
MH_FVMLIB      = 0x3
MH_CORE        = 0x4
MH_PRELOAD     = 0x5
MH_DYLIB       = 0x6
MH_DYLINKER    = 0x7
MH_BUNDLE      = 0x8
MH_DYLIB_STUB  = 0x9
MH_DSYM        = 0xA
MH_KEXT_BUNDLE = 0xB

MH_NOUNDEFS                = 0x1
MH_INCRLINK                = 0x2
MH_DYLDLINK                = 0x4
MH_BINDATLOAD              = 0x8
MH_PREBOUND                = 0x10
MH_SPLIT_SEGS              = 0x20
MH_LAZY_INIT               = 0x40
MH_TWOLEVEL                = 0x80
MH_FORCE_FLAT              = 0x100
MH_NOMULTIDEFS             = 0x200
MH_NOFIXPREBINDING         = 0x400
MH_PREBINDABLE             = 0x800
MH_ALLMODSBOUND            = 0x1000
MH_SUBSECTIONS_VIA_SYMBOLS = 0x2000
MH_CANONICAL               = 0x4000
MH_WEAK_DEFINES            = 0x8000
MH_BINDS_TO_WEAK           = 0x10000
MH_ALLOW_STACK_EXECUTION   = 0x20000
MH_ROOT_SAFE               = 0x40000
MH_SETUID_SAFE             = 0x80000
MH_NO_REEXPORTED_DYLIBS    = 0x100000
MH_PIE                     = 0x200000
MH_DEAD_STRIPPABLE_DYLIB   = 0x400000
MH_HAS_TLV_DESCRIPTORS     = 0x800000
MH_NO_HEAP_EXECUTION       = 0x1000000
MH_APP_EXTENSION_SAFE      = 0x2000000

LC_REQ_DYLD           = 0x80000000
LC_SEGMENT            = 0x1
LC_SYMTAB             = 0x2
LC_SYMSEG             = 0x3
LC_THREAD             = 0x4
LC_UNIXTHREAD         = 0x5
LC_LOADFVMLIB         = 0x6
LC_IDFVMLIB           = 0x7
LC_IDENT              = 0x8
LC_FVMFILE            = 0x9
LC_PREPAGE            = 0xa
LC_DYSYMTAB           = 0xb
LC_LOAD_DYLIB         = 0xc
LC_ID_DYLIB           = 0xd
LC_LOAD_DYLINKER      = 0xe
LC_ID_DYLINKER        = 0xf
LC_PREBOUND_DYLIB     = 0x10
LC_ROUTINES           = 0x11
LC_SUB_FRAMEWORK      = 0x12
LC_SUB_UMBRELLA       = 0x13
LC_SUB_CLIENT         = 0x14
LC_SUB_LIBRARY        = 0x15
LC_TWOLEVEL_HINTS     = 0x16
LC_PREBIND_CKSUM      = 0x17
LC_LOAD_WEAK_DYLIB    = 0x18
LC_SEGMENT_64         = 0x19
LC_ROUTINES_64        = 0x1a
LC_UUID               = 0x1b
LC_RPATH              = 0x1c + LC_REQ_DYLD
LC_CODE_SIGNATURE     = 0x1d
LC_SEGMENT_SPLIT_INFO = 0x1e
LC_REEXPORT_DYLIB     = 0x1f + LC_REQ_DYLD
LC_LAZY_LOAD_DYLIB    = 0x20
LC_ENCRYPTION_INFO    = 0x21
LC_DYLD_INFO          = 0x22
LC_DYLD_INFO_ONLY     = 0x22 + LC_REQ_DYLD

SG_HIGHVM  = 0x1
SG_FVMLIB  = 0x2
SG_NORELOC = 0x4

SECTION_TYPE       = 0x000000ff
SECTION_ATTRIBUTES = 0xffffff00

S_REGULAR                             = 0x0
S_ZEROFILL                            = 0x1
S_CSTRING_LITERALS                    = 0x2
S_4BYTE_LITERALS                      = 0x3
S_8BYTE_LITERALS                      = 0x4
S_LITERAL_POINTERS                    = 0x5
S_NON_LAZY_SYMBOL_POINTERS            = 0x6
S_LAZY_SYMBOL_POINTERS                = 0x7
S_SYMBOL_STUBS                        = 0x8
S_MOD_INIT_FUNC_POINTERS              = 0x9
S_MOD_TERM_FUNC_POINTERS              = 0x0a
S_COALESCED                           = 0x0b
S_GB_ZEROFILL                         = 0x0c
S_INTERPOSING                         = 0x0d
S_16BYTE_LITERALS                     = 0x0e
S_DTRACE_DOF                          = 0x0f
S_LAZY_DYLIB_SYMBOL_POINTERS          = 0x10
S_THREAD_LOCAL_REGULAR                = 0x11
S_THREAD_LOCAL_ZEROFILL               = 0x12
S_THREAD_LOCAL_VARIABLES              = 0x13
S_THREAD_LOCAL_VARIABLE_POINTERS      = 0x14
S_THREAD_LOCAL_INIT_FUNCTION_POINTERS = 0x15

SECTION_ATTRIBUTES_USR     = 0xff000000
S_ATTR_PURE_INSTRUCTIONS   = 0x80000000
S_ATTR_NO_TOC              = 0x40000000
S_ATTR_STRIP_STATIC_SYMS   = 0x20000000
S_ATTR_NO_DEAD_STRIP       = 0x10000000
S_ATTR_LIVE_SUPPORT        = 0x08000000
S_ATTR_SELF_MODIFYING_CODE = 0x04000000
S_ATTR_DEBUG               = 0x02000000

SECTION_ATTRIBUTES_SYS     = 0x00ffff00
S_ATTR_SOME_INSTRUCTIONS   = 0x00000400
S_ATTR_EXT_RELOC           = 0x00000200
S_ATTR_LOC_RELOC           = 0x00000100

VM_PROT_NONE       = 0x00
VM_PROT_READ       = 0x01
VM_PROT_WRITE      = 0x02
VM_PROT_EXECUTE    = 0x04
VM_PROT_DEFAULT    = VM_PROT_READ or VM_PROT_WRITE
VM_PROT_ALL        = VM_PROT_READ or VM_PROT_WRITE or VM_PROT_EXECUTE
VM_PROT_NO_CHANGE  = 0x08
VM_PROT_COPY       = 0x10
VM_PROT_WANTS_COPY = 0x10

x86_THREAD_STATE32    = 1
x86_FLOAT_STATE32     = 2
x86_EXCEPTION_STATE32 = 3
x86_THREAD_STATE64    = 4
x86_FLOAT_STATE64     = 5
x86_EXCEPTION_STATE64 = 6
x86_THREAD_STATE      = 7
x86_FLOAT_STATE       = 8
x86_EXCEPTION_STATE   = 9
x86_DEBUG_STATE32     = 10
x86_DEBUG_STATE64     = 11
x86_DEBUG_STATE       = 12
THREAD_STATE_NONE     = 13

N_STAB = 0xe0
N_PEXT = 0x10
N_TYPE = 0x0e
N_EXT  = 0x01
N_UNDF = 0x0
N_ABS  = 0x2
N_SECT = 0xe
N_PBUD = 0xc
N_INDR = 0xa

NO_SECT  = 0
MAX_SECT = 255

REFERENCE_TYPE                            = 0xf
REFERENCE_FLAG_UNDEFINED_NON_LAZY         = 0
REFERENCE_FLAG_UNDEFINED_LAZY             = 1
REFERENCE_FLAG_DEFINED                    = 2
REFERENCE_FLAG_PRIVATE_DEFINED            = 3
REFERENCE_FLAG_PRIVATE_UNDEFINED_NON_LAZY = 4
REFERENCE_FLAG_PRIVATE_UNDEFINED_LAZY     = 5
REFERENCED_DYNAMICALLY                    = 0x0010    
Then the basic macros for the creation of Mach-O headers:
Code:
; Basic layer: command headers

MachO::

namespace MachO

        if defined Settings.ProcessorType
                CPUTYPE := Settings.ProcessorType
        else
                CPUTYPE := CPU_TYPE_I386
        end if

        if defined Settings.ProcessorSubtype
                CPUSUBTYPE := Settings.ProcessorSubtype
        else
                if CPUTYPE and CPU_ARCH_ABI64
                        CPUSUBTYPE := CPU_SUBTYPE_I386_ALL + CPU_SUBTYPE_LIB64
                else
                        CPUSUBTYPE := CPU_SUBTYPE_I386_ALL
                end if
        end if

        if defined Settings.FileType
                FILETYPE := Settings.FileType
        else
                FILETYPE := MH_EXECUTE
        end if

        if defined Settings.Flags
                FLAGS := Settings.Flags
        else
                FLAGS := MH_NOUNDEFS + MH_DYLDLINK
        end if

        if defined Settings.SegmentAlignment
              SEGMENT_ALIGNMENT := Settings.SegmentAlignment
        else
              SEGMENT_ALIGNMENT := 1000h
        end if

        if defined Settings.FileAlignment
              FILE_ALIGNMENT := Settings.FileAlignment
        else
              FILE_ALIGNMENT := 1000h
        end if

        if CPUTYPE and CPU_ARCH_ABI64
                magic           dd      0xFEEDFACF
        else
                magic           dd      0xFEEDFACE
        end if
                cputype         dd      CPUTYPE
                cpusubtype      dd      CPUSUBTYPE
                filetype        dd      FILETYPE
                ncmds           dd      NUMBER_OF_COMMANDS
                sizeofcmds      dd      SIZE_OF_COMMANDS
                flags           dd      FLAGS
        if CPUTYPE and CPU_ARCH_ABI64
                reserved        dd      ?
        end if

        COMMAND_NUMBER = 0
        COMMAND_POSITION = $

        commands db SIZE_OF_COMMANDS dup 0

        org $

        macro command type
                if MachO.COMMAND_POSITION > 0
                        virtual at MachO.COMMAND_POSITION
                        MachO.COMMAND_POSITION = 0
                end if
                match t, type
                        if MachO.COMMAND_NUMBER > 0
                                repeat 1, p:MachO.COMMAND_NUMBER
                                        MachO.load#p.SIZE := $ - MachO.load#p.cmd
                                end repeat
                        end if
                        MachO.COMMAND_NUMBER = MachO.COMMAND_NUMBER + 1
                        MachO.COMMAND = $
                        repeat 1, n:MachO.COMMAND_NUMBER
                                namespace MachO.load#n
                                        cmd dd t
                                        cmdsize dd SIZE
                                end namespace
                        end repeat
                end match
        end macro

        macro text
                if MachO.COMMAND_POSITION = 0
                        MachO.COMMAND_POSITION = $
                        load MachO.COMMAND_DATA:$-$$ from $$
                        store MachO.COMMAND_DATA:$-$$ at MachO:$$
                        end virtual
                end if
        end macro

end namespace

macro align? boundary,value:?
        db (boundary-1)-($+boundary-1) mod boundary dup value
end macro

postpone
        MachO.text
        namespace MachO
                if COMMAND_NUMBER > 0
                        repeat 1, p:COMMAND_NUMBER
                                load#p.SIZE := COMMAND_POSITION - MachO.load#p.cmd
                        end repeat
                end if
                NUMBER_OF_COMMANDS := COMMAND_NUMBER
                SIZE_OF_COMMANDS := COMMAND_POSITION - commands
        end namespace
end postpone    
The MachO.command and MachO.text macros allow to switch between defining structures in the Mach-O headers and the actual text of program. The MachO.command can have an argument specifying the type of load command to create (it should be one of the LC_* constants), it then automatically creates and fills the "cmd" and "cmdsize" fields for that command. This little automation allows to manually create various structures needed for a functional Mach-O executable in a quite manageable way. Here's an example:
Code:
include '80386.inc'
use32

include 'macho.inc'

MachO.command LC_SEGMENT
        segname        ddq '__TEXT'
        vmaddr         dd 0
        vmsize         dd SEGMENT_SIZE
        fileoff        dd 0
        filesize       dd SEGMENT_SIZE
        maxprot        dd VM_PROT_ALL
        initprot       dd VM_PROT_READ or VM_PROT_EXECUTE
        nsects         dd 2
        flags          dd 0

MachO.command
    section1:
        .sectname       ddq '__text'
        .segname        ddq '__TEXT'
        .addr           dd SECTION1_ADDRESS
        .size           dd SECTION1_SIZE
        .offset         dd SECTION1_OFFSET
        .align          dd 4
        .reloff         dd 0
        .nreloc         dd 0
        .flags          dd 0
        .reserved1      dd ?
        .reserved2      dd ?

MachO.text

    align 16

    SECTION1_ADDRESS := $
    SECTION1_OFFSET := $%

    SYS_exit = 1
    SYS_write = 4

    start:
        push    msg.length
        push    msg
        push    1
        mov     eax,SYS_write
        sub     esp,4
        int     0x80
        add     esp,4*4
        push    0
        mov     eax,SYS_exit
        sub     esp,4
        int     0x80

    SECTION1_SIZE := $ - SECTION1_ADDRESS

MachO.command
    section2:
        .sectname       ddq '__cstring'
        .segname        ddq '__TEXT'
        .addr           dd SECTION2_ADDRESS
        .size           dd SECTION2_SIZE
        .offset         dd SECTION2_OFFSET
        .align          dd 2
        .reloff         dd 0
        .nreloc         dd 0
        .flags          dd 0
        .reserved1      dd ?
        .reserved2      dd ?

MachO.text

    align 4

    SECTION2_ADDRESS := $
    SECTION2_OFFSET := $%

        msg db 'Hello World!',0Ah
        .length = $ - msg

        db 1000h-$ dup 0        ; pad file to a minimum of 4096 bytes required by modern MacOS versions

    SECTION2_SIZE := $ - SECTION2_ADDRESS

    SEGMENT_SIZE := $

MachO.command LC_UNIXTHREAD
    thread_state:
        .flavor dd 1    ; i386_THREAD_STATE
        .count  dd 16   ; i386_THREAD_STATE_COUNT
        .eax    dd 0
        .ebx    dd 0
        .ecx    dd 0
        .edx    dd 0
        .edi    dd 0
        .esi    dd 0
        .ebp    dd 0
        .esp    dd 0
        .ss     dd 0
        .eflags dd 0
        .eip    dd start
        .cs     dd 0
        .ds     dd 0
        .es     dd 0
        .fs     dd 0
        .gs     dd 0
    
The next layer of macro automates the creation of segments, sections and thread initialization structures:
Code:
; Intermediate layer: segments and sections

namespace MachO

        if defined Settings.BaseAddress

                VM_ADDRESS = Settings.BaseAddress

                if CPUTYPE and CPU_ARCH_ABI64
                        MachO.command LC_SEGMENT_64
                else
                        MachO.command LC_SEGMENT
                end if
                namespace MachO.segment1
                        segname        emit 16: '__PAGEZERO'
                if CPUTYPE and CPU_ARCH_ABI64
                        vmaddr         dq       0
                        vmsize         dq       VM_ADDRESS
                        fileoff        dq       0
                        filesize       dq       0
                else
                        vmaddr         dd       0
                        vmsize         dd       VM_ADDRESS
                        fileoff        dd       0
                        filesize       dd       0
                end if
                        maxprot        dd       0
                        initprot       dd       0
                        nsects         dd       0
                        flags          dd       0
                end namespace

                SEGMENT_NUMBER = 1
        else
                VM_ADDRESS = 0
                SEGMENT_NUMBER = 0
        end if

        TEXT_OFFSET = $%

end namespace

macro MachO.segment declaration
        MachO.close_segment

        local NAME,VMADDR,VMSIZE,FILEOFF,FILESIZE,MAXPROT,INITPROT,NSECTS

        INITPROT = VM_PROT_NONE
        MAXPROT = VM_PROT_ALL
        match name attributes, declaration
                NAME = name
                local sequence
                define sequence attributes:
                while 1
                        match attribute tail, sequence
                                match =readable?, attribute
                                        INITPROT = INITPROT or VM_PROT_READ
                                        MAXPROT = MAXPROT or VM_PROT_READ
                                else match =notreadable?, attribute
                                        INITPROT = INITPROT and not VM_PROT_READ
                                        MAXPROT = MAXPROT and not VM_PROT_READ
                                else match =writeable?, attribute
                                        INITPROT = INITPROT or VM_PROT_WRITE
                                        MAXPROT = MAXPROT or VM_PROT_WRITE
                                else match =writable?, attribute
                                        INITPROT = INITPROT or VM_PROT_WRITE
                                        MAXPROT = MAXPROT or VM_PROT_WRITE
                                else match =notwriteable?, attribute
                                        INITPROT = INITPROT and not VM_PROT_WRITE
                                        MAXPROT = MAXPROT and not VM_PROT_WRITE
                                else match =notwritable?, attribute
                                        INITPROT = INITPROT and not VM_PROT_WRITE
                                        MAXPROT = MAXPROT and not VM_PROT_WRITE
                                else match =executable?, attribute
                                        INITPROT = INITPROT or VM_PROT_EXECUTE
                                        MAXPROT = MAXPROT or VM_PROT_EXECUTE
                                else match =notexecutable?, attribute
                                        INITPROT = INITPROT and not VM_PROT_EXECUTE
                                        MAXPROT = MAXPROT and not VM_PROT_EXECUTE
                                else
                                        err 'invalid argument'
                                end match
                                redefine sequence tail
                        else
                                break
                        end match
                end while
        else
                NAME = definition
        end match
        if lengthof NAME > 16
                err 'name to long'
        end if

        VMADDR := MachO.VM_ADDRESS

        if $% = MachO.TEXT_OFFSET & FILESIZE > 0
                MachO.FILE_OFFSET = 0
        else
                MachO.FILE_OFFSET = $%
        end if

        if FILESIZE
                FILEOFF := MachO.FILE_OFFSET
        else
                FILEOFF := 0
        end if

        MachO.SEGMENT_NUMBER = MachO.SEGMENT_NUMBER + 1
        if MachO.CPUTYPE and CPU_ARCH_ABI64
                MachO.command LC_SEGMENT_64
        else
                MachO.command LC_SEGMENT
        end if
        repeat 1, s:MachO.SEGMENT_NUMBER
                namespace MachO.segment#s
                        segname        emit 16: NAME
                if MachO.CPUTYPE and CPU_ARCH_ABI64
                        vmaddr          dq      VMADDR
                        vmsize          dq      VMSIZE
                        fileoff         dq      FILEOFF
                        filesize        dq      FILESIZE
                else
                        vmaddr          dd      VMADDR
                        vmsize          dd      VMSIZE
                        fileoff         dd      FILEOFF
                        filesize        dd      FILESIZE
                end if
                        maxprot         dd      MAXPROT
                        initprot        dd      INITPROT
                        nsects          dd      NSECTS
                        flags           dd      0
                end namespace
                repeat NSECTS
                        namespace MachO.segment#s#_section#%
                                sectname       emit 16: ?
                                segname        emit 16: ?
                        if MachO.CPUTYPE and CPU_ARCH_ABI64
                                addr           dq       ?
                                size           dq       ?
                        else
                                addr           dd       ?
                                size           dd       ?
                        end if
                                offset         dd       ?
                                align.         dd       ?
                                reloff         dd       ?
                                nreloc         dd       ?
                                flags          dd       ?
                                reserved1      dd       ?
                                reserved2      dd       ?
                        if MachO.CPUTYPE and CPU_ARCH_ABI64
                                               dd       ?
                        end if
                        end namespace
                end repeat
        end repeat

        macro MachO.close_segment
                MachO.close_section
                local OFFSET,NEXT,TOP,FILL
                TOP = $%
                NSECTS := MachO.SECTION_NUMBER
                if MachO.FILE_OFFSET >= $%%
                        FILL = MachO.FILE_OFFSET
                else
                        FILL = $%%
                end if
                repeat 1, s:MachO.SEGMENT_NUMBER
                        repeat NSECTS, n:2
                                if % = 1
                                        load OFFSET from MachO:MachO.segment#s#_section#%.offset
                                else
                                        OFFSET = NEXT
                                end if
                                if % < %%
                                        load NEXT from MachO:MachO.segment#s#_section#n.offset
                                else
                                        NEXT = TOP
                                end if
                                if OFFSET >= $%%
                                        store S_ZEROFILL at MachO:MachO.segment#s#_section#%.flags
                                        store 0 at MachO:MachO.segment#s#_section#%.size
                                else
                                        FILL = NEXT
                                end if
                        end repeat
                end repeat
                FILESIZE := FILL - MachO.FILE_OFFSET
                FILL = FILL - $%%
                MachO.SECTION_BYPASS = 1
                section MachO.VM_ADDRESS + FILESIZE - FILL
                restore MachO.SECTION_BYPASS
                db FILL dup 0
                if FILESIZE
                        rb (MachO.FILE_ALIGNMENT-$%) and (MachO.FILE_ALIGNMENT-1)
                end if
                MachO.VM_ADDRESS = MachO.VM_ADDRESS + TOP - MachO.FILE_OFFSET
                MachO.VM_ADDRESS = MachO.VM_ADDRESS + (MachO.SEGMENT_ALIGNMENT-MachO.VM_ADDRESS) and (MachO.SEGMENT_ALIGNMENT-1)
                VMSIZE := MachO.VM_ADDRESS - VMADDR
                purge MachO.close_segment
        end macro

        MachO.text
        org VMADDR + $% - MachO.FILE_OFFSET
        MachO.SECTION_NUMBER = 0

end macro

macro MachO.section declaration
        if MachO.SEGMENT_NUMBER = 0
                MachO.segment '__TEXT' readable writable executable
        end if
        MachO.close_section

        local SECTNAME,SEGNAME,FLAGS,ALIGN,RESERVED1,RESERVED2

        FLAGS = S_REGULAR
        ALIGN = 0
        RESERVED1 = 0
        RESERVED2 = 0

        local sequence
        match segname:sectname tail, declaration:
                SECTNAME = sectname
                SEGNAME = segname
                define sequence tail
        else match name tail, declaration:
                SECTNAME = name
                repeat 1, s:MachO.SEGMENT_NUMBER
                        load SEGNAME from MachO:MachO.segment#s.segname
                end repeat
                define sequence tail
        end match

        while 1
                match :, sequence
                        break
                else match =align? boundary tail, sequence
                        ALIGN = bsf boundary
                        if bsf boundary <> bsr boundary
                                err 'invalid alignment value'
                        end if
                        redefine sequence tail
                else match =flags?(value) tail, sequence
                        FLAGS = value
                        redefine sequence tail
                else match =reserved1?(value) tail, sequence
                        RESERVED1 = value
                        redefine sequence tail
                else match =reserved2?(value) tail, sequence
                        RESERVED2 = value
                        redefine sequence tail
                else
                        err 'invalid arguments'
                end match
        end while

        MachO.text
        if ALIGN
                align 1 shl ALIGN
        end if

        MachO.SECTION_NUMBER = MachO.SECTION_NUMBER + 1

        repeat 1, s:MachO.SEGMENT_NUMBER, t:MachO.SECTION_NUMBER
                if ~ defined MachO.segment#s#_section#t.sectname
                        MachO.command
                        namespace MachO.segment#s#_section#%
                                sectname       emit 16: ?
                                segname        emit 16: ?
                        if MachO.CPUTYPE and CPU_ARCH_ABI64
                                addr           dq       ?
                                size           dq       ?
                        else
                                addr           dd       ?
                                size           dd       ?
                        end if
                                offset         dd       ?
                                align.         dd       ?
                                reloff         dd       ?
                                nreloc         dd       ?
                                flags          dd       ?
                                reserved1      dd       ?
                                reserved2      dd       ?
                        if MachO.CPUTYPE and CPU_ARCH_ABI64
                                               dd       ?
                        end if
                        end namespace
                        MachO.text
                end if
                store SECTNAME at MachO:MachO.segment#s#_section#t.sectname
                store SEGNAME at MachO:MachO.segment#s#_section#t.segname
                store $ at MachO:MachO.segment#s#_section#t.addr
                store $% at MachO:MachO.segment#s#_section#t.offset
                store FLAGS at MachO:MachO.segment#s#_section#t.flags
                store ALIGN at MachO:MachO.segment#s#_section#t.align
                store RESERVED1 at MachO:MachO.segment#s#_section#t.reserved1
                store RESERVED2 at MachO:MachO.segment#s#_section#t.reserved2
        end repeat

end macro

macro MachO.close_section
        MachO.text
        if MachO.SECTION_NUMBER
                local SIZE
                repeat 1, s:MachO.SEGMENT_NUMBER, t:MachO.SECTION_NUMBER
                        load OFFSET from MachO:MachO.segment#s#_section#t.offset
                        store $%-OFFSET at MachO:MachO.segment#s#_section#t.size
                end repeat
        end if
end macro

macro MachO.close_segment
end macro

postpone
        MachO.close_segment
        if $%% < 1000h
                store 0:byte at $-1             ; enforce minimum file size for OS X 10.10.5 and higher
        end if
end postpone

macro segment? args&
        MachO.segment args
end macro

macro section? args&
        if defined MachO.SECTION_BYPASS
                section args
        else
                MachO.section args
        end if
end macro

macro entry? regs&
        iterate reg, regs
                match name:value, reg
                        MachO.thread.name? := value
                else if MachO.CPUTYPE = CPU_TYPE_I386
                        MachO.thread.eip? := reg
                else if MachO.CPUTYPE = CPU_TYPE_X86_64
                        MachO.thread.rip? := reg
                end if
        end iterate
        MachO.command LC_UNIXTHREAD
        if MachO.CPUTYPE = CPU_TYPE_I386
                MachO.thread.flavor dd x86_THREAD_STATE32
                iterate name, eax,ebx,ecx,edx,edi,esi,ebp,esp,ss,eflags,eip,cs,ds,es,fs,gs
                        if % = 1
                                MachO.thread.count dd %%
                        end if
                        if defined MachO.thread.name?
                                dd MachO.thread.name?
                        else
                                dd ?
                        end if
                end iterate
        else if MachO.CPUTYPE = CPU_TYPE_X86_64
                MachO.thread.flavor dd x86_THREAD_STATE64
                iterate name, rax,rbx,rcx,rdx,rdi,rsi,rbp,rsp,r8,r9,r10,r11,r12,r13,r14,r15,rip,rflags,cs,fs,gs
                        if % = 1
                                MachO.thread.count dd %%*2
                        end if
                        if defined MachO.thread.name?
                                dq MachO.thread.name?
                        else
                                dq ?
                        end if
                end iterate
        else
                err 'CPU not supported'
        end if
        MachO.text
end macro    
With these additional macros the earlier example can be greatly simplified:
Code:
include '80386.inc'
use32

include 'macho.inc'

segment '__TEXT' readable executable

  section '__text'

  SYS_exit = 1
  SYS_write = 4

  entry $

        push    msg.length
        push    msg
        push    1
        mov     eax,SYS_write
        sub     esp,4
        int     0x80
        add     esp,4*4
        push    0
        mov     eax,SYS_exit
        sub     esp,4
        int     0x80

  section '__cstring'

        msg db 'Hello World!',0Ah
        .length = $ - msg    
Note that the sections are subdivisions of segments, and the segments define continuous areas in program memory that have specified access permissions. This is analogous to the corresponding structures in ELF format, where segments and sections can co-exist in a similar way, though fasm's formatters never allowed to create both in the same file - because ELF segments are optional (and mostly useless) in object files, and ELF sections are optional in executable file. In case of Mach-O it is not possible to create section without a parent segment. But it is allowed to create segment with no sections - in the above example all the "section" lines could be removed and it would still work.

In addition to traditional fasm's "readable", "writeable" and "executable" permission keywords there are also "notreadable", "notwriteable", and "notexecutable". This is because Mach-O segments have separate fields for initial and maximum allowable permissions. The use of negated keywords causes permissions to be removed even from the "maxprot" field.

There is a convention to create the "__PAGEZERO" segment at address 0 with all permissions disabled, mainly as a protection against accidental use of null pointers. It is possible to prepare such segment manually, like:
Code:
include '80386.inc'
use32

include 'macho.inc'

segment '__PAGEZERO' notreadable notwritable notexecutable

  rb 1000h

segment '__TEXT' readable executable

  entry $

        push    msg.length
        push    msg
        push    1
        mov     eax,4
        sub     esp,4
        int     0x80
        add     esp,4*4
        push    0
        mov     eax,1
        sub     esp,4
        int     0x80

        msg db 'Hello World!',0Ah
        .length = $ - msg    
But it is also possible to have it made automatically, by specifying the "BaseAddress" setting for executable. The macros then create "_PAGEZERO" segment spanning the entire area from 0 up to the given base address:
Code:
include '80386.inc'
use32

MachO.Settings.BaseAddress = 0x1000

include 'macho.inc'

segment '__TEXT' readable executable

  entry $

        push    msg.length
        push    msg
        push    1
        mov     eax,4
        sub     esp,4
        int     0x80
        add     esp,4*4
        push    0
        mov     eax,1
        sub     esp,4
        int     0x80

        msg db 'Hello World!',0Ah
        .length = $ - msg    
And by altering the "ProcessorType" setting it is possible to create 64-bit executables:
Code:
include 'x64.inc'
use64

MachO.Settings.ProcessorType equ CPU_TYPE_X86_64
MachO.Settings.BaseAddress = 0x100000000

include 'macho.inc'

segment '__TEXT' readable executable

        SYSCALL_CLASS_UNIX = 2
        SYSCALL_CLASS_SHIFT = 24
        define SYSCALL_CONSTRUCT_UNIX SYSCALL_CLASS_UNIX shl SYSCALL_CLASS_SHIFT +

        SYS_exit = 1
        SYS_write = 4

        entry $, rdi: 1, rsi: msg

        mov     rdx,msg.length
        mov     rax,SYSCALL_CONSTRUCT_UNIX(SYS_write)
        syscall
        mov     rdi,rax
        mov     rax,SYSCALL_CONSTRUCT_UNIX(SYS_exit)
        syscall

        msg db 'This is a 64-bit test!',0Ah,0
        .length = $ - msg    
This 64-bit example uses the Unix syscall convention for a change, and it also utilizes the LC_UNIXTHREAD command created by the "entry" macro to set up a few more registers beside RIP. For some reason this is not a reliable method in general, it does not work for some of the registers, but for RSI and RDI it seems to be working.


Last edited by Tomasz Grysztar on 11 Aug 2017, 16:26; edited 3 times in total
Post 10 Aug 2017, 18:27
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 10 Aug 2017, 18:48
The further layer of macros is for the dynamic linking. This is a dangerous ground, the structures required for this are numerous and cross-reference each other, and on the top of that, recent versions of MacOS introduced an entirely new load commands (LC_DYLD) for this purpose. I have not yet touched these new commands (which appear to contain some kind of byte code interpreted by dylink), for now I tried to get the legacy ones to work.

First I tried the manual creation of symbol tables and dynamic linking structures:
Code:
include '80386.inc'
use32

MachO.Settings.BaseAddress = 0x1000

include 'macho.inc'

segment '__TEXT' readable executable

  section '__text' align 16

  entry start

  start:
        and     esp,0FFFFFFF0h
        sub     esp,10h

        mov     dword [esp],msg
        call    printf

        and     dword [esp],0
        call    exit

  section '__cstring' align 4

        msg db 'Hello World!',0Ah,0

segment '__IMPORT' readable executable

  section '__nl_symbol_ptr' flags(S_NON_LAZY_SYMBOL_POINTERS) reserved1(0)

    __printf dd ?
    __exit dd ?

  section '__jump_table' flags(S_SYMBOL_STUBS+S_ATTR_SOME_INSTRUCTIONS+S_ATTR_PURE_INSTRUCTIONS) reserved1(0) reserved2(STUB_SIZE)

    printf:
        jmp     dword [__printf]

    STUB_SIZE := $ - printf

    exit:
        jmp     dword [__exit]

segment '__LINKEDIT' readable

    symbols = $%
    ; msg
        dd      str_msg     ; n_strx
        db      N_SECT      ; n_type
        db      2           ; n_sect
        dw      0           ; n_desc
        dd      msg         ; n_value
    ; start
        dd      str_start
        db      N_SECT + N_EXT
        db      1
        dw      0
        dd      start
    ; _printf
        dd      str_printf
        db      N_EXT
        db      0
        dw      REFERENCE_FLAG_UNDEFINED_NON_LAZY
        dd      0
    ; _exit
        dd      str_exit
        db      N_EXT
        db      0
        dw      REFERENCE_FLAG_UNDEFINED_NON_LAZY
        dd      0

    indirect = $%
    ; indexes into symbol table:
        dd      2       ; _printf
        dd      3       ; _exit

    strings = $%
        db      ' ',0
    str_msg = $% - strings
        db      'msg',0
    str_start = $% - strings
        db      'start',0
    str_printf = $% - strings
        db      '_printf',0
    str_exit = $% - strings
        db      '_exit',0
    strings_size = $% - strings

MachO.command LC_LOAD_DYLINKER
    dd dylinker-MachO.COMMAND
    dylinker db '/usr/lib/dyld',0
    align 4

MachO.command LC_LOAD_DYLIB
    dd dylib-MachO.COMMAND
    dd 2                ; timestamp
    dd 0x006F0103       ; currentver
    dd 0x00010000       ; compatver
    dylib db '/usr/lib/libSystem.B.dylib',0
    align 4

MachO.command LC_SYMTAB
    dd symbols          ; symoff
    dd 4                ; nsyms
    dd strings          ; stroff
    dd strings_size     ; strsize

MachO.command LC_DYSYMTAB
; one local symbol (msg):
    dd 0                ; ilocalsym
    dd 1                ; nlocalsym
; one public symbol (start):
    dd 1                ; iextdefsym
    dd 1                ; nextdefsym
; two external symbols (printf, exit):
    dd 2                ; iundefsym
    dd 2                ; nundefsym
    dd 0                ; tocoff
    dd 0                ; ntoc
    dd 0                ; modtaboff
    dd 0                ; nmodtab
    dd 0                ; extrefsymoff
    dd 0                ; nextrefsyms
; all external symbols are linked indirectly:
    dd indirect         ; indirectsymoff
    dd 2                ; nindirectsyms
    dd 0                ; extreloff
    dd 0                ; nextrel
    dd 0                ; locreloff
    dd 0                ; nlocrel    
This seems to work, though I'm still curious what should it look like to use lazy symbol pointers. Apparently this would be a more usual method of linking external functions, but it requires some additional quirks that I are out of my grasp at the moment. And this is still just using the legacy load commands!

The macros I made for dynamic linking use the non-lazy pointers like the above sample:
Code:
; Upper layer: dynamic linking

namespace MachO
        NSYMS = 0
        LIB_NUMBER = 0
end namespace

macro interpreter? path
        MachO.command LC_LOAD_DYLINKER
        namespace MachO.dylinker_command
                lc_str          dd dylinker-MachO.COMMAND
                dylinker        db path,0
                align 4
        end namespace
end macro

macro uses? lib&
        MachO.LIB_NUMBER = MachO.LIB_NUMBER + 1
        MachO.command LC_LOAD_DYLIB
        repeat 1, l:MachO.LIB_NUMBER
        namespace MachO.dylib#l#_command
                lc_str          dd dylib-MachO.COMMAND
                timestamp       dd 2
                match path (a.b.c=,x.y.z), lib
                        current_version         dd (x and 0FFFFh) shl 16 + y shl 8 + z
                        compatibility_version   dd (a and 0FFFFh) shl 16 + b shl 8 + c
                        dylib                   db path,0
                else
                        current_version         dd 10000h
                        compatibility_version   dd 10000h
                        dylib                   db lib,0
                end match
                align 4
        end namespace
        end repeat
end macro

macro import? definitions&
        iterate <name,string>, definitions
                MachO.NSYMS = MachO.NSYMS + 1
                define MachO.symbol name
                name.extrn := 1
                name.type := N_EXT
                name.desc := REFERENCE_FLAG_UNDEFINED_NON_LAZY
                define name.str string
        end iterate
end macro

postpone
if MachO.NSYMS

        segment '__IMPORT' readable writable executable

        section '__nl_symbol_ptr' flags(S_NON_LAZY_SYMBOL_POINTERS) reserved1(0) align 4

                irpv sym, MachO.symbol
                        if sym.extrn
                                if MachO.CPUTYPE and CPU_ARCH_ABI64
                                        sym.ptr dq ?
                                else
                                        sym.ptr dd ?
                                end if
                        end if
                end irpv

        section '__jump_table' flags(S_SYMBOL_STUBS+S_ATTR_SOME_INSTRUCTIONS+S_ATTR_PURE_INSTRUCTIONS) reserved1(0) reserved2(MachO.JUMP_SIZE) align 16

                irpv sym, MachO.symbol
                        if sym.extrn
                                sym: jmp [sym.ptr]
                                if % = 1
                                        MachO.JUMP_SIZE := $ - sym
                                end if
                        end if
                end irpv

        segment '__LINKEDIT' readable

                MachO.SYMOFF := $%
                irpv sym, MachO.symbol
                        namespace MachO.nlist#%
                                n_strx dd sym.strx
                                n_type db sym.type
                                n_sect db 0
                                n_desc dw sym.desc
                        if MachO.CPUTYPE and CPU_ARCH_ABI64
                                n_value dq sym
                        else
                                n_value dd sym
                        end if
                        end namespace
                end irpv

                MachO.INDIRECTSYMOFF := $%
                irpv sym, MachO.symbol
                        if sym.extrn
                                dd %-1
                        end if
                end irpv

                MachO.STROFF := $%
                db 20h,0
                irpv sym, MachO.symbol
                        sym.strx := $% - MachO.STROFF
                        db string sym.str, 0
                end irpv
                MachO.STRSIZE := $% - MachO.STROFF

        MachO.command LC_SYMTAB
        namespace MachO.symtab_command
                symoff  dd SYMOFF
                nsyms   dd NSYMS
                stroff  dd STROFF
                strsize dd STRSIZE
        end namespace

        MachO.command LC_DYSYMTAB
        namespace MachO.dysymtab_command
                ilocalsym       dd 0
                nlocalsym       dd 0
                iextdefsym      dd 0
                nextdefsym      dd 0
                iundefsym       dd 0
                nundefsym       dd NSYMS
                tocoff          dd 0
                ntoc            dd 0
                modtaboff       dd 0
                nmodtab         dd 0
                extrefsymoff    dd 0
                nextrefsyms     dd 0
                indirectsymoff  dd INDIRECTSYMOFF
                nindirectsyms   dd NSYMS
                extreloff       dd 0
                nextrel         dd 0
                locreloff       dd 0
                nlocrel         dd 0
        end namespace

end if
end postpone    
The syntax enabled by these macros is similar to the ones I made for dynamicallly linked ELF executables:
Code:
include '80386.inc'
use32

MachO.Settings.BaseAddress = 0x1000

include 'macho.inc'

interpreter '/usr/lib/dyld'
uses '/usr/lib/libSystem.B.dylib' (1.0.0, 1225.0.0)
import printf,'_printf',\
       exit,'_exit'

segment '__TEXT' readable executable

  section '__text' align 16

  entry start

  start:
        and     esp,0FFFFFFF0h
        sub     esp,10h

        mov     dword [esp],msg
        call    printf

        and     dword [esp],0
        call    exit

  section '__cstring' align 4

        msg db 'Hello World!',0Ah,0    
What was not present in ELF is the inclusion of two version numbers for every linked library. The first one is "compatibility version" and the second one is "current version". You can see their values in an executable with "otool -L" command.
Post 10 Aug 2017, 18:48
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 10 Aug 2017, 19:06
I'm attaching the complete set of macros I have made so far, packaged with a few examples.

I was not able to get working 64-bit executable with dynamic linking, it needs an additional investigation. If you find any way to correct or improve these macros, please let me know.


Description: Mach-O executable formatting macros for fasmg, with examples
Download
Filename: machO.zip
Filesize: 8.88 KB
Downloaded: 1293 Time(s)

Post 10 Aug 2017, 19:06
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 10 Aug 2017, 21:07
Finally, the above macros can be used to assemble fasmg itself as a dynamically linked Mach-O executable. I'm attaching a pre-assembled executable and the source files that need to be added to the standard "source" directory of fasmg.

EDIT: this is now included in the official fasmg packaging.


Last edited by Tomasz Grysztar on 26 Aug 2017, 15:42; edited 2 times in total
Post 10 Aug 2017, 21:07
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 10 Aug 2017, 21:46
fasmg sources not full:
Code:
flat assembler  version g.hvj4j
C:\FASM\SOURCE_G\fasmg_macos\fasmg.asm [66]:
        or al,TRACE_ERROR_STACK
macro or [2] macro parse_operand [10] macro parse_operand_value [31]:
        ns.imm = +op
Processed: @src.imm = +TRACE_ERROR_STACK
Error: symbol 'TRACE_ERROR_STACK' is undefined or out of scope.    


pushes & popes in machO - cancelled? Why so strange ccall?
Post 10 Aug 2017, 21:46
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 10 Aug 2017, 21:57
ProMiNick wrote:
fasmg sources not full:
Code:
flat assembler  version g.hvj4j
C:\FASM\SOURCE_G\fasmg_macos\fasmg.asm [66]:
        or al,TRACE_ERROR_STACK
macro or [2] macro parse_operand [10] macro parse_operand_value [31]:
        ns.imm = +op
Processed: @src.imm = +TRACE_ERROR_STACK
Error: symbol 'TRACE_ERROR_STACK' is undefined or out of scope.    
Please make sure you use the latest fasmg source base. It assembles correctly with version hvvox.
Post 10 Aug 2017, 21:57
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 10 Aug 2017, 22:24
Yes. All compiled OK.


What about ccall? pushes? why moves to [esp...] used?
Post 10 Aug 2017, 22:24
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 11 Aug 2017, 09:47
ProMiNick wrote:
What about ccall? pushes? why moves to [esp...] used?
This is a variant of "ccall" intended only for fasmg sources, because SYSTEM.INC always uses it with simple (register/constant) arguments and memory operands are not supported. The setting up of stack frame is similar to the one of fastcall convention, I have done it this way because it allows to allocate stack space and align it at the same time. The alignment is necessary for MacOS, previous patches also had to provide it in some way.

PS I have updated the macros above with a few small fixes.
Post 11 Aug 2017, 09:47
View user's profile Send private message Visit poster's website Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 11 Aug 2017, 15:32
Oh thank you Tomasz! I just went through the trouble of using objconv.
Some questions:
If I port the code reasonably from fasm to fasmg, will fasmg produce identical binaries? Does fasmg do all of the size optimisations that fasm does?
Post 11 Aug 2017, 15:32
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 11 Aug 2017, 16:13
tthsqe wrote:
If I port the code reasonably from fasm to fasmg, will fasmg produce identical binaries? Does fasmg do all of the size optimisations that fasm does?
The x86 macros that come with the base package reproduce the encoding and optimizations of fasm 1 quite faithfully. For example, self-assembled fasmg is identical to the binary assembled with fasm 1 (the only differences are in the PE headers, where there are things like timestamp).

Of course it is also possible that there may be some alternative macro sets implementing x86 in the future. I think someone on the board worked on one already. A less complex one could be made to allow faster assembly (though there may be not much to gain, implementation of instructions as macros is going to be relatively slow by its very nature). On the opposite end of spectrum, I started working on even more complex macros that demonstrate some of the ideas I had for fasm 2 (before I decided to reduce my fasm 2 project to just fasmg).
Post 11 Aug 2017, 16:13
View user's profile Send private message Visit poster's website Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 11 Aug 2017, 16:32
I will give porting my source from fasm to fasmg a shot then. BTW, I know how much trouble it is to get documentation on apple's suff. Any they change things from version to version.
Post 11 Aug 2017, 16:32
View user's profile Send private message Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 12 Aug 2017, 07:38
I noticed that your mach-o macros don't contain anything resembling relocations. Could I get confirmation/disconfirmation of the following concerning the code at the end?

0. In any format:
The d address in f is encoded as rip-relative since fasm knows the sizes of the sections.

1. In format PE64:
The d address in g cannot be encoded as rip-relative by an architecture constraint. In the dword field of the encoding, fasm puts the absolute address of d since the absolute address of the base of the module is known. If the absolute address of d cannot be encoded as a dword, an error occurs.

2. In format PE64 with fixups:
The d address in g is encoded using an assumed absolute address (p) of the base of the module. If the OS decides to load the module at another address (q), then "data fixups; end data" contains the locations of the dwords/qwords that need to be shifted by q-p. If any of these operations overflows, an error occurs.

3. In format ELF64:
The d address in g is encoded using "dd ?". When the OS loads the module at some address (p), then the relocation section contains the locations of the dwords/qwords that need to modified based on the value of p. If any of these operations overflows, an error occurs.

4. In format MACH-O:
Much the same as "format PE64"? Of course MACH-O has some relocation scheme so that dynamically linked libraries are possible?

Code:
...[some section]...
f:
    mov eax, dword[d]
    ret

g:
    xor eax, eax
    mov eax, dword[d+rax]
    ret

...[another section]...
d:
    dd 42    
Post 12 Aug 2017, 07:38
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 12 Aug 2017, 14:09
tthsqe wrote:
I noticed that your mach-o macros don't contain anything resembling relocations.
I mentioned in the beginning that I started with the executable variant of this format, hoping that it would be quicker and simpler to implement than a linkable object. But, while Mach-O object may be similar to ELF object in implementation, executable turned out much more complex than ELF (as a quick look at the ELFEXE.INC may prove).

tthsqe wrote:
Could I get confirmation/disconfirmation of the following concerning the code at the end?
Yes, but first let me explain the capabilities of different formats in general.

All of the three PE/COFF, ELF, and Mach-O formats have both the executable and object variant (in case of PE/COFF the executable variant is PE and the object variant is COFF).

The executable variant is always an image assembled with a fixed memory layout and an assumed base address. Optionally the image may allow to be loaded at different address, either because it is position-independent or because it has fixups that can be applied to correct the addresses in code when the base shifts. Only PE format has the ability to contain fixups, in case of ELF and Mach-O if the executable is to allow being loaded at different base, in needs to be PIC.

All of the executable formats also allow dynamic linking, which involves resolving some external symbols and placing their addresses in a special data table in the executable image. Most of the time these are the addresses of external functions, and calling these functions can be done either though an indirect call to the field in the dynamic linking table:
Code:
call    [__exit]    
or through a gate:
Code:
        call    exit
; ...
exit:
        jmp     [_exit]    

As for the object formats, they always have true relocations.

tthsqe wrote:
0. In any format:
The d address in f is encoded as rip-relative since fasm knows the sizes of the sections.
First: this is only true for 64-bit code. In 32-bit mode f would use absolute addressing.
Second: even RIP-relative addressing may still require relocation, when the output is a relocatable object and f and d reside in different sections. In the process of linking the sections may end up at different distance from each other and relative addressing still requires relocation. This applies to RIP-relative data addressing, and to the relative jumps/calls as well.

tthsqe wrote:
1. In format PE64:
The d address in g cannot be encoded as rip-relative by an architecture constraint. In the dword field of the encoding, fasm puts the absolute address of d since the absolute address of the base of the module is known. If the absolute address of d cannot be encoded as a dword, an error occurs.
This is true for any executable format with no fixups, so for executable ELF or Mach-O it is the same (as they have no option to contain fixups).

tthsqe wrote:
2. In format PE64 with fixups:
The d address in g is encoded using an assumed absolute address (p) of the base of the module. If the OS decides to load the module at another address (q), then "data fixups; end data" contains the locations of the dwords/qwords that need to be shifted by q-p. If any of these operations overflows, an error occurs.
Yes, and this is unique to PE (I would say it is one of its strong points).

tthsqe wrote:
3. In format ELF64:
The d address in g is encoded using "dd ?". When the OS loads the module at some address (p), then the relocation section contains the locations of the dwords/qwords that need to modified based on the value of p. If any of these operations overflows, an error occurs.
This is what happens in case of object files, and it is the same for kinds of objects - COFF, ELF, or Mach-O.

In general Mach-O format capabilities are similar to ELF - the executable variant allows dynamic linking (this includes things like GOT for PIC) but no fixups, the object variant has true relocations. All the examples I created so far are executables (with MH_EXECUTABLE type in header), but you could use the same low-layer macros to create an object file. You'd need to alter some fields in the header with settings like:
Code:
MachO.Settings.FileType equ MH_OBJECT
MachO.Settings.Flags = 0    
and then perhaps alter the SEGMENT/SECTION macros a little, to use relocatable virtual address (like ELF64.INC) and generate relocation tables for each section.
Post 12 Aug 2017, 14:09
View user's profile Send private message Visit poster's website Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 21 Aug 2017, 08:14
Thomas, your 64 macros don't work for dynamic linking.

I have attached the binary of a working executable as well as some of its info with objconv. Your macros are missing a few sections as well as the trampoline instructions for the dynamic loader. This program calls write then exit.
In the program, "write" and "exit" are calls to jmp instructions in the ._TEXT.__stubs section. Each of these jumps fetches an address from the ._DATA.__la_symbol_ptr section. Initially, these address in the ._DATA.__la_symbol_ptr point to something like
Code:
        push    N
        jmp     ?_004     

where ?_004 is the address of the thing that calls imp_dyld_stub_binder. If you look in the binary, your will see strings "@_exit" and "@_write" in this order. The "N" is the offset of function name from the first name (@_exit). As for how imp_dyld_stub_binder knows where to write the address of the found function to the ._DATA.__la_symbol_ptr section, I have no idea. (I am fairly certain that imp_dyld_stub_binder will overwrite these addresses so that it is not called the next time the imported function with this name is called).

Also, objconv completed with some error, so the human readable form might not be complete. I have been reading http://timetobleed.com/dynamic-linking-elf-vs-mach-o/

Code:
; Disassembly of file: mac_test
; Mode: 64 bits
; Syntax: YASM/NASM
; Instruction set: 8086, x64

default rel

global __mh_execute_header
global _main

extern dyld_stub_binder                                 ; byte
extern _write                                           ; byte
extern _exit                                            ; byte


SECTION ._TEXT._code align=64 execute                   ; section number 1, code

_main:  ; Function begin
        and     rsp, 0FFFFFFFFFFFFFFF0H                 ; 0F40 _ 48: 83. E4, F0
        mov     edi, 1                                  ; 0F44 _ BF, 00000001
        lea     rsi, [rel ?_001]                        ; 0F49 _ 48: 8D. 35, 00000011(rel)
        mov     edx, 13                                 ; 0F50 _ BA, 0000000D
        call    ?_003                                   ; 0F55 _ E8, 0000001A(rel)
        mov     eax, edi                                ; 0F5A _ 89. F8
        call    ?_002                                   ; 0F5C _ E8, 0000000D(rel)
?_001:
; Error: Prefix after REX prefix not allowed
; Note: Prefix bit or byte has no meaning in this context
;       insb                                            ; 0F61 _ 48 65: 6C
        db 48H, 65H, 6CH
;       insb                                            ; 0F64 _ 6C
        db 6CH
;       outsd                                           ; 0F65 _ 6F
        db 6FH
;       and     byte [rdi+6FH], dl                      ; 0F66 _ 20. 57, 6F
        db 20H, 57H, 6FH
;       jc      97H                                     ; 0F69 _ 72, 6C
        db 72H, 6CH
; Note: Function does not end with ret or jmp
;       and     dword [fs:rdx], ecx                     ; 0F6B _ 64: 21. 0A
        db 64H, 21H, 0AH
; _main End of function


SECTION ._TEXT.__stubs align=2 execute                  ; section number 2, code

?_002:  ; Local function
        jmp     near [rel ?_005]                        ; 0F6E _ FF. 25, 0000009C(rel)

?_003:  ; Local function
        jmp     near [rel ?_006]                        ; 0F74 _ FF. 25, 0000009E(rel)


SECTION ._TEXT.__stub_helper align=4 execute            ; section number 3, code

?_004:  ; Local function
        lea     r11, [rel imp__exit]                    ; 0F7C _ 4C: 8D. 1D, 00000085(rel)
        push    r11                                     ; 0F83 _ 41: 53
        jmp     near [rel imp_dyld_stub_binder]         ; 0F85 _ FF. 25, 00000075(rel)

        nop                                             ; 0F8B _ 90
; Note: Immediate operand could be made smaller by sign extension
        push    0                                       ; 0F8C _ 68, 00000000
; Note: Immediate operand could be made smaller by sign extension
        jmp     ?_004                                   ; 0F91 _ E9, FFFFFFE6

; Note: Immediate operand could be made smaller by sign extension
        push    12                                      ; 0F96 _ 68, 0000000C
; Note: Immediate operand could be made smaller by sign extension
        jmp     ?_004                                   ; 0F9B _ E9, FFFFFFDC


SECTION ._TEXT.__unwind_info align=4 noexecute          ; section number 4, data

        db 01H, 00H, 00H, 00H, 1CH, 00H, 00H, 00H       ; 0FA0 _ ........
        db 00H, 00H, 00H, 00H, 1CH, 00H, 00H, 00H       ; 0FA8 _ ........
        db 00H, 00H, 00H, 00H, 1CH, 00H, 00H, 00H       ; 0FB0 _ ........
        db 02H, 00H, 00H, 00H, 40H, 0FH, 00H, 00H       ; 0FB8 _ ....@...
        db 34H, 00H, 00H, 00H, 34H, 00H, 00H, 00H       ; 0FC0 _ 4...4...
        db 6FH, 0FH, 00H, 00H, 00H, 00H, 00H, 00H       ; 0FC8 _ o.......
        db 34H, 00H, 00H, 00H, 03H, 00H, 00H, 00H       ; 0FD0 _ 4.......
        db 0CH, 00H, 01H, 00H, 10H, 00H, 01H, 00H       ; 0FD8 _ ........
        db 00H, 00H, 00H, 00H, 00H, 00H, 00H, 00H       ; 0FE0 _ ........


SECTION ._DATA.__nl_symbol_ptr align=8 noexecute        ; section number 5, data

imp_dyld_stub_binder:                                   ; import from ?
        dq 0000000000000000H                            ; 1000 _ 0000000000000000 

imp__exit:                                              ; dword
        dd 00000000H                                    ; 1008 _ 0 

imp__write:                                             ; dword
        dd 00000000H                                    ; 100C _ 0 


SECTION ._DATA.__la_symbol_ptr align=8 noexecute        ; section number 6, data

?_005:                                                  ; switch/case jump table
        dq 0000000100000F8CH                            ; 1010 _ 0000000100000F8C 

?_006:                                                  ; switch/case jump table
        dq 0000000100000F96H                            ; 1018 _ 0000000100000F96     


Code:
Dump of file: mac_test, type: Mach-O Little Endian64
Dump of Mach-O file mac_test
-----------------------------------------------
File size: 0x2120
File header:
  CPU type: Intel 64 bit, subtype: unknown(0x80000003)
  File type: Mach-O Little Endian - demand paged executable file
  Number of load commands: 15, Size of commands: 0x460, Flags: 200085

Command 1: 64-bit segment, size: 0x48
  Name: __PAGEZERO,
  Memory address 0x0000000000000000, Memory size 0x0000000100000000
  File offset 0x0000000000000000, File size 0x0000000000000000
  Maxprot 0x0, Initprot 0x0
  Number of sections 0, Flags 0x0

Command 2: 64-bit segment, size: 0x188
  Name: __TEXT,
  Memory address 0x0000000100000000, Memory size 0x0000000000001000
  File offset 0x0000000000000000, File size 0x0000000000001000
  Maxprot 0x7, Initprot 0x5
  Number of sections 4, Flags 0x0

Command 3: 64-bit segment, size: 0xE8
  Name: __DATA,
  Memory address 0x0000000100001000, Memory size 0x0000000000001000
  File offset 0x0000000000001000, File size 0x0000000000001000
  Maxprot 0x7, Initprot 0x3
  Number of sections 2, Flags 0x0

Command 4: 64-bit segment, size: 0x48
  Name: __LINKEDIT,
  Memory address 0x0000000100002000, Memory size 0x0000000000001000
  File offset 0x0000000000002000, File size 0x0000000000000120
  Maxprot 0x7, Initprot 0x1
  Number of sections 0, Flags 0x0

Command 5: unknown(0x80000022), size: 0x30

Command 6: Symbol table, size: 0x18
  Symbol table offset 0x2078, number of symbols 5,
  String table offset 0x20E0, String table size 0x40

Command 7: dynamic link-edit symbol table info, size: 0x50
  Index to local symbols 0, number of local symbols 0,
  Index to external symbols 0, number of external symbols 2,
  Index to undefined symbols 2, number of undefined symbols 3,
  File offset to TOC 0x0, number of entries in TOC 0,
  File offset to module table 0x0, Number of module table entries 0,
  Offset to referenced symbol table 0x0, Number of referenced symtab entries 0
  Offset to indirect symbol table 0x20C8, Number of indirect symtab entries 6
  Offset to external relocation entries 0x0, Number of external reloc. entries 0

  Offset to local relocation entries 0x0, Number of local reloc. entries 0

Command 8: load a dynamic linker, size: 0x20

Command 9: uuid, size: 0x18

Command 10: unknown(0x24), size: 0x10

Command 11: unknown(0x2A), size: 0x10

Command 12: unknown(0x80000028), size: 0x18

Command 13: load a dynamicly linked shared library, size: 0x38

Command 14: unknown(0x26), size: 0x10

Command 15: unknown(0x29), size: 0x10

Sections:

Section 1: Name: _code, Segment: __TEXT.
  Memory address 0xF40, Size 0x2E, File offset 0xF40
  Alignment 64, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x80000400, reserved1 0x0, reserved2 0x0

Section 2: Name: __stubs, Segment: __TEXT.
  Memory address 0xF6E, Size 0xC, File offset 0xF6E
  Alignment 2, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x80000408, reserved1 0x0, reserved2 0x6

Section 3: Name: __stub_helper, Segment: __TEXT.
  Memory address 0xF7C, Size 0x24, File offset 0xF7C
  Alignment 4, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x80000400, reserved1 0x0, reserved2 0x0

Section 4: Name: __unwind_info, Segment: __TEXT.
  Memory address 0xFA0, Size 0x48, File offset 0xFA0
  Alignment 4, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x0, reserved1 0x0, reserved2 0x0

Section 5: Name: __nl_symbol_ptr, Segment: __DATA.
  Memory address 0x1000, Size 0x10, File offset 0x1000
  Alignment 8, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x6, reserved1 0x2, reserved2 0x0

Section 6: Name: __la_symbol_ptr, Segment: __DATA.
  Memory address 0x1010, Size 0x10, File offset 0x1010
  Alignment 8, Reloc. ent. offset 0x0, Num reloc. 0
  Flags 0x7, reserved1 0x4, reserved2 0x0

Symbol table:

  Public symbols:
   0 __mh_execute_header, Section 1, Value 0x0
    External, Defined
    Reference type: External non lazy,  Flags: Referenced dynamically,
   1 _main, Section 1, Value 0xF40
    External, Defined
    Reference type: External non lazy,  Flags:

  External symbols:
   2 _exit, Section 0, Value 0x0
    External, Undefined, no section
    Reference type: External non lazy,  Flags:
   3 _write, Section 0, Value 0x0
    External, Undefined, no section
    Reference type: External non lazy,  Flags:
   4 dyld_stub_binder, Section 0, Value 0x0
    External, Undefined, no section
    Reference type: External non lazy,  Flags:

  Indirect symbols:
   _exit, type 0x1, sect 0, desc 0x100, val 0x0
   _write, type 0x1, sect 0, desc 0x100, val 0x0
   dyld_stub_binder, type 0x1, sect 0, desc 0x100, val 0x0
   Unknown(0x40000000)
   _exit, type 0x1, sect 0, desc 0x100, val 0x0
   _write, type 0x1, sect 0, desc 0x100, val 0x0

String table:
    0:
    2: __mh_execute_header
   22: _main
   28: _exit
   34: _write
   41: dyld_stub_binder
   58:
   59:
   60:
   61:
   62:
   63:    


Description: working binary - remove .txt extension
Download
Filename: mac_test.txt
Filesize: 8.28 KB
Downloaded: 1210 Time(s)

Post 21 Aug 2017, 08:14
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 21 Aug 2017, 10:51
tthsqe wrote:
Thomas, your 64 macros don't work for dynamic linking.
Yes, I mentioned that already. The addresses of functions are resolved correctly but jumping to them causes segfault, because dynamic linker does not map the libraries as executable memory. This may have something to do with the expectation that functions should be linked as "lazy" externals, while my example (made in ELF-like fashion) uses non-lazy linking for them. Perhaps the dynamic linker then assumes that these are data symbols instead of functions and does not give them executable rights. For some reason this worked with 32-bit, but perhaps the NX bit is not used there.

The "lazy" externals require additional "trampolines" as you mentioned, but I did not find out what makes them tick (I like to know exactly how things work and then make the absolutely minimal implementation). Each stub (the one that consists only of a single JMP instruction) has to initially jump to the trampoline, and that code in turn has to call the dynamic linker somehow, and provide the parameters (in registers?) telling what function needs to have it address resolved and updated in the stub - this is the part I have not yet figured out. It make take some time before a have a chance to work on it more - if you find out anything in the meantime, please let me know.
Post 21 Aug 2017, 10:51
View user's profile Send private message Visit poster's website Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 21 Aug 2017, 13:54
Ok. This is probably take a long time to work out. I can see right away that for 64bits, your macros are generating jmp instructions in a __jmp_table section. It looks like the way it is done is to put a "dq ?" in the __DATA,__nl_symbol_ptr section.

I'm reading https://www.mikeash.com/pyblog/friday-qa-2012-11-09-dyld-dynamic-linking-on-os-x.html
Post 21 Aug 2017, 13:54
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 430
Location: Australia
redsock 21 Aug 2017, 21:23
I attempted to create a VirtualBox guest out of macOS Sierra yesterday to put up, the VM gets stuck waiting on the root device though so I was unable to finish.

Didn't have a ton of time to research, but if any of you have ideas to get me past that point I can donate another macOS VM to play around with.

_________________
2 Ton Digital - https://2ton.com.au/
Post 21 Aug 2017, 21:23
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 22 Aug 2017, 19:33
tthsqe wrote:
Ok. This is probably take a long time to work out. I can see right away that for 64bits, your macros are generating jmp instructions in a __jmp_table section. It looks like the way it is done is to put a "dq ?" in the __DATA,__nl_symbol_ptr section.
What my macros generate is the non-lazy binding, and it is actually quite similar to dynamic linking of ELF executables or PE imports. In all cases you have a variable that is updated by dynamic linker with the pointer to a function at run-time:
Code:
_exit dq ? ; DD in 32-bit case    
You can then simply call the function by using this pointer directly (and the direct use of pointer requires an indirect call):
Code:
call [_exit]    
and this is in fact the default way fasm's macros are wired for ELF and PE. In all cases, including ELF and PE, you can also define additional function stubs and call them instead:
Code:
call exit
; ...
exit: ; stub
jmp [_exit]    
In ELF and PE this is something you can use optionally, it is not forced. In Mach-O for some reason dynamic linking does not work properly unless you have these stubs.

OK, we now have non-lazy binding covered, not let's talk about the lazy one, which may actually be necessary for the functions in 64-bit Mach-O (though this is just my theory at the moment). In this case dynamic linker does not fill the import variables with pointers to the functions. These variables should be pre-filled at the assembly time with pointers to the "stub helpers":
Code:
_exit dq _exit.helper

_exit.helper:
; some code that calls dynamic linker and tells it to update [_exit] with a pointer to the external function, and then jump there    
The helper gets called only once, the first time the function is called, because then the variable get updated with the resolved pointer to the function. Note that this is independent from whether you call the function through the stub or directly.

Now the detail I was missing when I wanted to change my macros from non-lazy binding to a lazy one is what that stub helper should do, exactly (in both 32-bit and 64-bit variants) and how can it call the dynamic linker.
Post 22 Aug 2017, 19:33
View user's profile Send private message Visit poster's website Reply with quote
Grom PE



Joined: 13 Mar 2008
Posts: 114
Location: i@grompe.org.ru
Grom PE 23 Aug 2017, 05:15
redsock wrote:
I attempted to create a VirtualBox guest out of macOS Sierra yesterday to put up, the VM gets stuck waiting on the root device though so I was unable to finish.

Didn't have a ton of time to research, but if any of you have ideas to get me past that point I can donate another macOS VM to play around with.
I was able to run some preinstalled image of macOS El Capitan on VMWare.

But an image of macOS Sierra didn't want to cooperate — it crashed (rebooted) in minutes after any nontrivial action, and fasmg and related binaries gave an error "Illegal Instruction: 4" on it. I'm not sure if I got a broken image of Sierra or its loader is more strict and something is missing. Or maybe something to do with the fact I'm trying this on AMD processor.
Post 23 Aug 2017, 05:15
View user's profile Send private message Visit poster's website Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 23 Aug 2017, 07:19
Thomasz, there is the source I am using.

Code:
include 'x86/include/x64.inc'
use64

MachO.Settings.ProcessorType equ CPU_TYPE_X86_64
MachO.Settings.BaseAddress = 0x100000000
include 'x86/macinc/macho.inc'
entry Start

interpreter '/usr/lib/dyld'
uses '/usr/lib/libSystem.B.dylib' (1.0.0, 1.0.0)
import _write,'_write'
import _exit,'_exit'


segment '__TEXT' readable executable

Start:
    and rsp, -16
    mov edi, 1
    lea rsi, [msg]
    mov edx, msg_end - msg
    call _write
    mov edi, eax
    call _exit
msg: db 'Hello World!',10
msg_end:
    


your current macros are generating something like this
Code:
segment '__TEXT' readable executable
Start:
    and rsp, -16
    mov edi, 1
    lea rsi, [msg]
    mov edx, msg_end - msg
    call _write
    mov edi, eax
    call _exit
msg: db 'Hello World!',10
msg_end:

segment '__IMPORT' readable writable executable
section '__nl_symbol_ptr' 
    _write2: dq ?
    _exit2: dq ?
section '__jump_table'
    _write: jmp qword[_write2]
    _exit:  jmp qword[_exit2]
    



The problem with this is that dyld is going to have to write the correct addresses
into qword[_write2] and qword[_exit2]. But this segment is marked executable. I though this was a big no-no.
This might explain why your current macros generate exe's that segfault in dyld.


From what I can tell you have to do something like this
Code:
segment '__TEXT' readable executable
Start:
    and rsp, -16
    mov edi, 1
    lea rsi, [msg]
    mov edx, msg_end - msg
    call qword[_write]
    mov edi, eax
    call qword[_exit]
msg: db 'Hello World!',10
msg_end:

segment '__DATA' readable writable
section '__nl_symbol_ptr' 
    _write: dq ?
    _exit: dq ?
    

and direct dyld to fill in qword[_write] and qword[_exit]
Post 23 Aug 2017, 07:19
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.