flat assembler - why the ENTER instruction fails on allocating

Index > Linux > why the ENTER instruction fails on allocating > 1416 byte

Author

Thread

fly

Joined: 27 Apr 2005
Posts: 11
Location: Salzburg, Austria

fly 06 Jun 2005, 15:18

i believed that the enter instruction equals to the push/mov/sub sequence, but when i try to allocate more than 1416 bytes for the stack frame the code below produces a segfault.
first i thought, my litte knowledge in assembler is the reason for that, but i tried to replace enter/leave instructions with the corresponding push/... sequences, and... it works fine!
The question is: Why?

this one will fail:

Code:

format ELF64 executable
section readable executable

entry $
        enter 1417, 0

        leave

mov rbx, 0
mov rax, 1
int 0x80

this one will work:

Code:

format ELF64 executable
section readable executable

entry $
        push rbp
        mov rbp, rsp
        sub rsp, 1417

        mov rsp, rbp
        pop rbp

mov rbx, 0
mov rax, 1
int 0x80

06 Jun 2005, 15:18

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20424
Location: In your JS exploiting you and your system

revolution 07 Jun 2005, 02:00

My guess is that RSP must be aligned by 8. 1417 does not align the stack properly. After subtracting 1417 from RSP then first instruction that uses the stack will also fail.

Last edited by revolution on 07 Jun 2005, 09:48; edited 2 times in total

07 Jun 2005, 02:00

fly

Joined: 27 Apr 2005
Posts: 11
Location: Salzburg, Austria

fly 07 Jun 2005, 08:09

i tried a well-aligned value, but with the same result. interestingly the 32-bit counterpart allows a value below 3005 to work.

Code:

format ELF executable
section readable executable

entry $
        ;enter 3005, 0
        enter 4096, 0

        leave

mov ebx, 0
mov eax, 1
int 0x80

edit: also 32byte alignment (e.g. 32768) will produce a segfault

07 Jun 2005, 08:09

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum