flat assembler
Message board for the users of flat assembler.

Index > Compiler Internals > [solved] fasm.x64 with Linux 5.2 - not enough memory

Author
Thread Post new topic Reply to topic
st



Joined: 12 Jul 2019
Posts: 49
Location: Russia
st 12 Jul 2019, 13:35
Hi,

I am using fasm g for some time with Linux.
After I have upgraded the kernel to v5.2.0, fasmg.x64 (version g.il0vt) fails to compile any source with the "Error: not enough memory to complete the assembly" message.

Here are few straces:
Code:
$ strace fasmg.x64 mlvalues.inc 
execve("/usr/local/bin/fasmg.x64", ["fasmg.x64", "mlvalues.inc"], 0x7ffca0ab0158 /* 72 vars */) = 0
brk(NULL)                               = 0x555556a2d000
write(2, "Error: ", 7Error: )                  = 7
write(2, "not enough memory to complete th"..., 42not enough memory to complete the assembly) = 42
write(2, ".\n", 2.
)                      = 2
exit(3)                                 = ?
+++ exited with 3 +++

$ strace fasmg.x64 mlvalues.inc 
execve("/usr/local/bin/fasmg.x64", ["fasmg.x64", "mlvalues.inc"], 0x7ffe22516498 /* 72 vars */) = 0
brk(NULL)                               = 0x5555557b7000
write(2, "Error: ", 7Error: )                  = 7
write(2, "not enough memory to complete th"..., 42not enough memory to complete the assembly) = 42
write(2, ".\n", 2.
)                      = 2
exit(3)                                 = ?
+++ exited with 3 +++
    

It works fine with the norandmaps kernel parameter or under GDB.
Both 5.2.0 and previous (working) kernel have # CONFIG_COMPAT_BRK is not set

I doubt if it is related to the new CONFIG_SHUFFLE_PAGE_ALLOCATOR option, since
Code:
cat /sys/module/page_alloc/parameters/shuffle
N    


Obviously, fails this brk syscall:
Code:
    malloc_brk:
    ; in:
    ;  r14d = requested size
    ; out:
    ;  eax - allocated block, equal to previous value of [malloc_lbrk]
    ;  ecx = allocated size
    ;  edi = what was in r12d
    ;  esi = what was in r13d
    ;  on error jumps to out_of_memory (does not return)
    ; preserves: rbx
    ; note: for internal use only
        cmp     dword [malloc_lbrk],0
        jne     malloc_lbrk_valid
        mov     eax,12                  ; sys_brk
        xor     edi,edi
        syscall
        mov     ecx,eax
        cmp     rcx,rax
        jne     out_of_memory
    
However I am not sure shall brk return 32-bit address or not https://github.com/torvalds/linux/blob/457c89965399115e5cd8bf38f9c597293405703d/mm/mmap.c#L193
(personally I prefer mmap with the predefined in compile-time address in an analogous scenario)

Old good fasm fails too:
Code:
$ fasm.x64 mlvalues.inc 
flat assembler  version 1.73.12error: failed to allocate memory within 32-bit addressing range.
    


Is someone encountering the same behaviour? Or it is just me messed with kernel config.

edit: marked as solved.


Last edited by st on 19 Jul 2019, 07:55; edited 2 times in total
Post 12 Jul 2019, 13:35
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 12 Jul 2019, 14:05
The tricks used to make fasm 1 and fasmg run in long mode on Linux depend on brk point being within range of memory addressable with 32 bits. If your variant of kernel makes it that brk is above 4G, the memory provided by sys_brk is unusable.

There is a possible partial workaround - you could put a large area of reserved data at the end of fasm/fasmg executable, it would then reside in the low memory and you could feed that block to the internal allocator instead of memory provided by sys_brk. But this would be a static allocation of a fixed amount of memory, so it is not something that we could just put into a standard distribution of fasm/fasmg.

Or, as you mentioned, we could try mmap instead. But can you make mmap reliably return memory in the low 4G?
Post 12 Jul 2019, 14:05
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20363
Location: In your JS exploiting you and your system
revolution 12 Jul 2019, 14:19
Linux also has a bug when running 32-bit code on a 64-bit kernel and using sys_brk. It can return memory above 4G but the return value in EAX only shows the lower 32-bits. A debugger shows the full RAX value with the full address above 4G. No error is given to the app, and naturally it crashes when trying to access the bad address.
Post 12 Jul 2019, 14:19
View user's profile Send private message Visit poster's website Reply with quote
st



Joined: 12 Jul 2019
Posts: 49
Location: Russia
st 12 Jul 2019, 15:50
Tomasz Grysztar wrote:

Or, as you mentioned, we could try mmap instead. But can you make mmap reliably return memory in the low 4G?

Well, honestly I did not see any documentation that declares this shall be allways done by mmap. However fasm seems to be not relocatable and therefore depending on mentioned behaviour (because kernel loads executable in such a mapped memory).

Actually I have some code that mmaps pages directly after ones belonging to ELF:
Code:
        lea     rax, [heap_small.unaligned_end + PAGE_SIZE]
        and     rax, not (PAGE_SIZE-1)
        mov     [uncommited], rax
;;;;;;;;;;;;;

add_page:
        mov     rdi, [uncommited]
        mov     esi, HEAP_INCREMENT
        mov     edx, PROT_READ or PROT_WRITE
        mov     r10d, MAP_PRIVATE or MAP_ANONYMOUS
        mov     r8, -1
        zero    r9
        lea     rax, [rdi + rsi]
        mov     [uncommited], rax
        sys.mmap
;;;;;;;;;;;;;;

heap_small:     rb HEAP_INIT_SIZE
.unaligned_end:
    


From the other hand, it is unclear for me, may be this should be considered as some Linux kernel regression. I think I'll test some Fedora and/or Ubuntu with this new kernel.


revolution wrote:
Linux also has a bug when running 32-bit code on a 64-bit kernel and using sys_brk. It can return memory above 4G but the return value in EAX only shows the lower 32-bits. A debugger shows the full RAX value with the full address above 4G. No error is given to the app, and naturally it crashes when trying to access the bad address.
32 bit fasm works still fine on my system. I use it as a workaround. Smile
Post 12 Jul 2019, 15:50
View user's profile Send private message Visit poster's website Reply with quote
st



Joined: 12 Jul 2019
Posts: 49
Location: Russia
st 13 Jul 2019, 09:01
The same failure on Fedora Rawhide, Linux kernel 5.2.0-1.fc31.x86_64. Did not test Ubuntu, since 19.10 (Eoan Ermine) Daily Build 2019-07-12 has some 5.0.x kernel (without such issue).

I have opened a bug https://bugzilla.kernel.org/show_bug.cgi?id=204157
Post 13 Jul 2019, 09:01
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 13 Jul 2019, 09:07
st wrote:
32 bit fasm works still fine on my system. I use it as a workaround. Smile
That tricky x64 version was made for systems that do not support 32-bit executables, you do not gain anything by using it when a regular 32-bit one is an option for you.

Anyway, I may try to add some fail-safe mechanism based on mmap, I just need to find some spare time.
Post 13 Jul 2019, 09:07
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 13 Jul 2019, 16:53
I have prepared a patch for fasmg that tries sys_mmap when sys_brk memory is unusable with 32-bit addressing. Please try it and let me know if the trick works on your system. I could add something similar to fasm 1, too - as long as it turns out to be a working solution.
Post 13 Jul 2019, 16:53
View user's profile Send private message Visit poster's website Reply with quote
st



Joined: 12 Jul 2019
Posts: 49
Location: Russia
st 14 Jul 2019, 03:13
Thanks.

I have compiled this version with 32-bit fasmg:
Code:
~/src/flat assembler, the next generation-dff57f287f/source/linux/x64 $ fasmg fasmg.asm 
flat assembler  version g.il0vt
5 passes, 16.3 seconds, 57012 bytes.    


It works as expected:
Code:
~/src/flat assembler, the next generation-dff57f287f/source/linux/x64 $ ./fasmg fasmg.asm fasmg.x64
flat assembler  version g.il0vt
5 passes, 17.4 seconds, 57012 bytes.    


Both shows version g.il0vt, so just in case I've the same way compiled and tested old one:
Code:
~/src/fasmg/source/linux/x64 $ ./fasmg malloc.inc 
Error: not enough memory to complete the assembly.    


BTW Linux kernel gurus give no answer yet.
Post 14 Jul 2019, 03:13
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 14 Jul 2019, 09:55
Thank you! As it seems to work, I'm releasing fasm 1 with an analogous fail-safe.
Post 14 Jul 2019, 09:55
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.