flat assembler - fasmg - bugs in x86-64 macros

Index > Compiler Internals > fasmg - bugs in x86-64 macros

Author

Thread

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 15 Aug 2017, 21:00

for completeness put a

Code:

include 'x86/include/format/format.inc'
include 'x86/include/avx.inc'

format ELF64 executable
entry Start

segment executable readable
Start:

at the beginning of all of these examples. I also had to change some of the recursive includes go get fasmg working out of the box.

here are some cases where fasmg does not match fasm1

Not smallest encoding

Code:

mov rax, 0xFFFFFFFFFF000000

If I sign extended the imm by subtracting 2^64, then it gives smallest and matches fasm1.

internal error

Code:

mov r8d, dword[rbp+10]  ; ok
mov r8d, dword[rbp+100] ; ok
mov r8d, dword[rbp+1000]; Error: symbol 'index' is undefined or out of scope.

However, on this last example, i have some some macros that do "mov r8d, dword[rbp+foo.bar]". This does compile but it compiles to "mov r8d, dword[rip+foo.bar]". This makes me believe it is a initialisation error in index.

I will keep posting more bugs as I find them.

15 Aug 2017, 21:00

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 15 Aug 2017, 21:52

same problem with lea. The "<" is fasmg and the ">" is fasm1:

Code:

<    15081:     48 8d 35 28 ff ff ff    lea    rsi,[rip+0xffffffffffffff28]
>    15081:     48 8d b5 28 ff ff ff    lea    rsi,[rbp-0xd8]

15 Aug 2017, 21:52

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 15 Aug 2017, 23:00

Out of a 100kb exe, besides the mov rax, ... issue, this are the only differences between fasmg (<) and fasm1 (>). Obviously the output from fasmg is a complete error in these cases. They all use rbp as the base of an address.

Code:

<     baa5:     44 8b 05 d0 01 00 00    mov    r8d,DWORD PTR [rip+0x1d0]        # 0xbc7c
---
>     baa5:     44 8b 85 d0 01 00 00    mov    r8d,DWORD PTR [rbp+0x1d0]
14999c14999
<     bab0:     4c 03 05 00 02 00 00    add    r8,QWORD PTR [rip+0x200]        # 0xbcb7
---
>     bab0:     4c 03 85 00 02 00 00    add    r8,QWORD PTR [rbp+0x200]
15702c15702
<     c394:     89 35 d0 01 00 00       mov    DWORD PTR [rip+0x1d0],esi        # 0xc56a
---
>     c394:     89 b5 d0 01 00 00       mov    DWORD PTR [rbp+0x1d0],esi
18579c18579
<     eca9:     48 03 05 f8 01 00 00    add    rax,QWORD PTR [rip+0x1f8]        # 0xeea8
---
>     eca9:     48 03 85 f8 01 00 00    add    rax,QWORD PTR [rbp+0x1f8]
19132c19132
<     f542:     4c 8b 05 08 02 00 00    mov    r8,QWORD PTR [rip+0x208]        # 0xf751
---
>     f542:     4c 8b 85 08 02 00 00    mov    r8,QWORD PTR [rbp+0x208]
19139c19139
<     f566:     44 8b 05 d0 01 00 00    mov    r8d,DWORD PTR [rip+0x1d0]        # 0xf73d
---
>     f566:     44 8b 85 d0 01 00 00    mov    r8d,DWORD PTR [rbp+0x1d0]
19141c19141
<     f571:     4c 03 05 00 02 00 00    add    r8,QWORD PTR [rip+0x200]        # 0xf778
---
>     f571:     4c 03 85 00 02 00 00    add    r8,QWORD PTR [rbp+0x200]
21190c21190
<    1131b:     4c 8b 05 08 02 00 00    mov    r8,QWORD PTR [rip+0x208]        # 0x1152a
---
>    1131b:     4c 8b 85 08 02 00 00    mov    r8,QWORD PTR [rbp+0x208]
21197c21197
<    1133f:     44 8b 05 d0 01 00 00    mov    r8d,DWORD PTR [rip+0x1d0]        # 0x11516
---
>    1133f:     44 8b 85 d0 01 00 00    mov    r8d,DWORD PTR [rbp+0x1d0]
21199c21199
<    1134a:     4c 03 05 00 02 00 00    add    r8,QWORD PTR [rip+0x200]        # 0x11551
---
>    1134a:     4c 03 85 00 02 00 00    add    r8,QWORD PTR [rbp+0x200]
22280c22280
<    1239c:     4c 8b 05 08 02 00 00    mov    r8,QWORD PTR [rip+0x208]        # 0x125ab
---
>    1239c:     4c 8b 85 08 02 00 00    mov    r8,QWORD PTR [rbp+0x208]
22287c22287
<    123c0:     44 8b 05 d0 01 00 00    mov    r8d,DWORD PTR [rip+0x1d0]        # 0x12597
---
>    123c0:     44 8b 85 d0 01 00 00    mov    r8d,DWORD PTR [rbp+0x1d0]
22289c22289
<    123cb:     4c 03 05 00 02 00 00    add    r8,QWORD PTR [rip+0x200]        # 0x125d2
---
>    123cb:     4c 03 85 00 02 00 00    add    r8,QWORD PTR [rbp+0x200]
24993c24993
<    14a25:     48 8b 1d e0 01 00 00    mov    rbx,QWORD PTR [rip+0x1e0]        # 0x14c0c
---
>    14a25:     48 8b 9d e0 01 00 00    mov    rbx,QWORD PTR [rbp+0x1e0]
25114c25114
<    14bf8:     0f b6 05 d0 01 00 00    movzx  eax,BYTE PTR [rip+0x1d0]        # 0x14dcf
---
>    14bf8:     0f b6 85 d0 01 00 00    movzx  eax,BYTE PTR [rbp+0x1d0]
25168c25168
<    14cab:     44 89 25 dc 01 00 00    mov    DWORD PTR [rip+0x1dc],r12d        # 0x14e8e
---
>    14cab:     44 89 a5 dc 01 00 00    mov    DWORD PTR [rbp+0x1dc],r12d
25246c25246
<    14da7:     89 0d d0 01 00 00       mov    DWORD PTR [rip+0x1d0],ecx        # 0x14f7d
---
>    14da7:     89 8d d0 01 00 00       mov    DWORD PTR [rbp+0x1d0],ecx
25461c25461
<    15081:     48 8d 35 28 ff ff ff    lea    rsi,[rip+0xffffffffffffff28]        # 0x14fb0
---
>    15081:     48 8d b5 28 ff ff ff    lea    rsi,[rbp-0xd8]
25468,25469c25468,25469
<    150a1:     48 8b 35 e8 01 00 00    mov    rsi,QWORD PTR [rip+0x1e8]        # 0x15290
<    150a8:     4c 8b 35 e0 01 00 00    mov    r14,QWORD PTR [rip+0x1e0]        # 0x1528f
---
>    150a1:     48 8b b5 e8 01 00 00    mov    rsi,QWORD PTR [rbp+0x1e8]
>    150a8:     4c 8b b5 e0 01 00 00    mov    r14,QWORD PTR [rbp+0x1e0]

15 Aug 2017, 23:00

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8357
Location: Kraków, Poland

Tomasz Grysztar 16 Aug 2017, 08:24

I made two little corrections in x64.inc, please try it now.

16 Aug 2017, 08:24

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 16 Aug 2017, 11:44

Identical binaries! BTW, fasm1 assembles the source in 0.1 seconds. fasmg used to incorrectly assemble the source in 18 seconds. Now fasmg correctly assembles the source in 21 seconds.
I'm not sure how to get this time down.

Do you know if anyone has started writing macros for arm-v8 (64 bit arm)?

Also, could you put the mach-o formatters in the main distribution?

16 Aug 2017, 11:44

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8357
Location: Kraków, Poland

Tomasz Grysztar 16 Aug 2017, 13:47

tthsqe wrote:

BTW, fasm1 assembles the source in 0.1 seconds. fasmg used to incorrectly assemble the source in 18 seconds. Now fasmg correctly assembles the source in 21 seconds.
I'm not sure how to get this time down.

I have been warning about this is since the beginning - implementing a complex instruction sets like x86 entirely in form of macros is going to be slow no matter what. You could, however, try to optimize some of them, perhaps making some compromises in their feature set, to get the assembly time a little bit down. Reducing a number of lines in any macro should have a positive effect on performance, especially in the macros that get called extensively. For example this one from x64.inc:

Code:

macro x86.parse_operand ns,op
        ns.size = 0
        match prefix value, op
                match :sz, x86.prefix
                        ns.size = sz
                        x86.parse_operand_value ns,value
                else
                        x86.parse_operand_value ns,op
                end match
        else
                x86.parse_operand_value ns,op
        end match
end macro

can be made simpler:

Code:

macro x86.parse_operand ns,op
        ns.size = 0
        match :sz value, x86.op
                ns.size = sz
                x86.parse_operand_value ns,value
        else
                x86.parse_operand_value ns,op
        end match
end macro

and even this reduces the assembly speed a little bit (but it breaks some of the tricks with equates).

tthsqe wrote:

Do you know if anyone has started writing macros for arm-v8 (64 bit arm)?

I was hoping that revolution could try, having so much experience from making FASMARM, but it seems they have not enough spare time to play with fasmg. Honestly, I would prefer to leave this to people more competent in ARM architecture than me, but there have been no volunteers so far.

tthsqe wrote:

Also, could you put the mach-o formatters in the main distribution?

I would like to have them tester and confirmed a bit more before putting them out for general use. But I may include the self-hosted MacOS version of fasmg (which includes these macros) in the main package.

16 Aug 2017, 13:47

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 16 Aug 2017, 15:14

I will try to get a simple hello world working for arm64. As far I can see, this involves getting a working subset of the instruction set as well as modifying your elfexe macros

16 Aug 2017, 15:14

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8357
Location: Kraków, Poland

Tomasz Grysztar 16 Aug 2017, 15:42

tthsqe wrote:

I will try to get a simple hello world working for arm64. As far I can see, this involves getting a working subset of the instruction set as well as modifying your elfexe macros

The ELF.Settings namespace allows to set up some of the values in headers without modifying the ELFEXE.INC. If you'd need some additional modifications, it would be good to make them also in form of settings.

16 Aug 2017, 15:42

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20430
Location: In your JS exploiting you and your system

revolution 16 Aug 2017, 19:35

Tomasz Grysztar wrote:

tthsqe wrote:
Do you know if anyone has started writing macros for arm-v8 (64 bit arm)?
I was hoping that revolution could try, having so much experience from making FASMARM, but it seems they have not enough spare time to play with fasmg. Honestly, I would prefer to leave this to people more competent in ARM architecture than me, but there have been no volunteers so far.

It is a lot of work. I briefly looked into it and discovered that. For the basic 32-bit V1 it might be not too bad in terms of time required, but it quickly gets complicated after that. FWIW I can't yet justify the amplified assembly times compared to fasm1. But if someone wants to give it a go they can use the verifiers I include with fasmarm to check that they are on the right track. That might speed things up a little.

16 Aug 2017, 19:35

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 17 Aug 2017, 11:38

Thomasz, how well is arm syntax going to play with fasmg?
For example, this is valid aarch64 syntax

Code:

ldr x0, [x1, :lo12:a+3]
stp x0, x1, [sp, -16]!

17 Aug 2017, 11:38

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8357
Location: Kraków, Poland

Tomasz Grysztar 17 Aug 2017, 12:10

You can MATCH any combination of special characters and other symbol with no problem, except for the semicolon and backslash at the end of line, as they are processed and stripped by the line reader. Also the "#" has a special meaning in fasmg's identifier syntax, but you can MATCH it away before it goes to the expression evaluator.

My introductory manual for fasmg contains several examples of processing various kind of syntax (though they focus on Z80 architecture, as it was especially fitting to showcase some of the methods). I may add more in the future. In fact, if you encounter any trouble applying these tricks to ARM syntax, please let me know and I may expand the documentation with explained solutions.

Now as I think of it, it might be a good idea to add a section about processing values that contain both relocatable addresses and registers, as it happens when x86 instructions are combined with object output formats.

17 Aug 2017, 12:10

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 17 Aug 2017, 21:37

Hello world is working at https://github.com/tthsqe12/asm/blob/master/armfish.arm
Thomasz, if you have time, could you make sure I'm on the right track with the macros? There are in arm/include/aarch64.inc
EDIT: I simply cannot find good enough information on relocations, so I will be completely ignoring them and only concentrating on the elfexe format.

17 Aug 2017, 21:37

tthsqe

Joined: 20 May 2009
Posts: 767

tthsqe 04 Oct 2017, 07:21

There is another inconsistency between fasm1 and fasmg. With

Code:

format PE64 console

I had to apply the following patch to get fasmg to match fasm1.

Code:

else match =PE64? settings, clause:
                PE.Settings.Magic = 0x20B
                PE.Settings.Machine = IMAGE_FILE_MACHINE_AMD64
 -              PE.Settings.Characteristics = IMAGE_FILE_EXECUTABLE_IMAGE or IMAGE_FILE_32BIT_MACHINE or IMAGE_FILE_LINE_NUMS_STRIPPED or IMAGE_FILE_LOCAL_SYMS_STRIPPED
 +              PE.Settings.Characteristics = IMAGE_FILE_EXECUTABLE_IMAGE or IMAGE_FILE_LARGE_ADDRESS_AWARE or IMAGE_FILE_LINE_NUMS_STRIPPED or IMAGE_FILE_LOCAL_SYMS_STRIPPED
                PE.Settings.DllCharacteristics = 0
                PE.Settings.MajorSubsystemVersion = 5
                PE.Settings.MinorSubsystemVersion = 0

I think fasm1 is right here.

04 Oct 2017, 07:21

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8357
Location: Kraków, Poland

Tomasz Grysztar 04 Oct 2017, 09:04

tthsqe wrote:

There is another inconsistency between fasm1 and fasmg. (...)

Thanks. I'm correcting it both in fasmg's basic examples and in the Windows headers package. They now have different but compatible implementations of "format" macro.

04 Oct 2017, 09:04

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum