flat assembler
Message board for the users of flat assembler.

Home Search Register
Log in to check your private messages Log in

Index > Linux > What calling convention in Linux?
Author
Thread Post new topic Reply to topic
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 19 Apr 2018, 07:19
What calling convention in Linux?

Code:
if x68family
 if code64
    lincall.reg0 equ rax
    lincall.reg0.lo equ eax
    lincall.reg1 equ rdi
    lincall.reg1.lo equ edi
    lincall.reg2 equ rsi
    lincall.reg2.lo equ esi
    lincall.reg3 equ rdx
    lincall.reg3.lo equ edx
  else
    lincall.reg0 equ eax
    lincall.reg1 equ ebx
    lincall.reg2 equ ecx
    lincall.reg3 equ edx
  end if
 if nolink
    lincall.call equ int 80h
 else
    lincall.call equ call eax ; lincall.call equ call [eax]
 end if
else if armfamily
  if code64
    lincall.reg0 equ x8
    ;lincall.reg0.lo equ eax
    lincall.reg1 equ x0
    ;lincall.reg1.lo equ edi
    lincall.reg2 equ x1
    ;lincall.reg2.lo equ esi
    lincall.reg3 equ x2
    ;lincall.reg3.lo equ edx
  else
    lincall.reg0 equ x8
    lincall.reg1 equ x0
    lincall.reg2 equ x1
    lincall.reg3 equ x2
  end if
  if nolink
    lincall.call equ svc  0
  else
    lincall.call equ blr  x8
  end if
end if
    
    

linuxcall write,stdout,ADDR msg,msgsize
pseudo {
    if 64mode & msgsize<2^32 & defined lincall.reg3.lo
       mov lincall.reg3.lo, msgsize
    else
      mov  lincall.reg3, msgsize
   end if
    match =ADDR addr,ADDR msg
        lea lincall.reg2, [msg] ; for arm: adr lincall.reg2, msg
    else if 64mode & msg<2^32 & defined lincall.reg2.lo
       mov lincall.reg2.lo, msg
    else
      mov  lincall.reg2, msg
   end if
    if 64mode & stdout<2^32 & defined lincall.reg1.lo
       mov lincall.reg1.lo, stdout
    else
      mov  lincall.reg1, stdout
   end if
    if 64mode & write<2^32 & defined lincall.reg0.lo
       mov lincall.reg0.lo, write
    else
      mov  lincall.reg0, write
   end if
  lincall.call }

_________________
I don`t like to refer by "you" to one person.
My soul requires acronim "thou" instead.
Post 19 Apr 2018, 07:19
View user's profile Send private message Send e-mail Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20343
Location: In your JS exploiting you and your system
revolution 19 Apr 2018, 09:25
Can you be more specific about what you are asking?
Post 19 Apr 2018, 09:25
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 19 Apr 2018, 11:10
all linux examles on this forum could be simplificated with macros to this form:
Code:
format ELF executable 3 ; format ELF64 executable 3
entry start
;settings = nolink(int|syscall)|dylink
;include architecture

segment readable executable 
Start:
            stackalign16
            lincall      write, stdout, ADDR msg, msgsize
            lincall      exit, 0

segment readable writeable
            msg db 'Hello World!', 10
            msgsize = $-msg


Code:
format ELF ; format ELF64
public main
;settings = nolink (int|syscall)|dylink
;include architecture
extrn write
extrn exit

section '.code' executable align 64  
main:
            lincall      write, stdout, ADDR msg, msgsize
            lincall      exit, 0

section '.data' writeable align 64 
            msg db 'Hello World!', 10
            msgsize = $-msg


but there is no macroses for linux calls provided. Why? And why so few examples for linux? Even in fasm package for linux? Why there is no GUI linux examples? (because in linux there `re different GUI engines?) so, why not realized one or couple of them?

_________________
I don`t like to refer by "you" to one person.
My soul requires acronim "thou" instead.
Post 19 Apr 2018, 11:10
View user's profile Send private message Send e-mail Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 430
Location: Australia
redsock 19 Apr 2018, 20:02
Page 10 of this excellent resource is highly recommended: http://www.agner.org/optimize/calling_conventions.pdf

IMO, a "lincall" macro is overkill since the register order is pretty easy to remember/recognise/identify.

_________________
2 Ton Digital - https://2ton.com.au/
Post 19 Apr 2018, 20:02
View user's profile Send private message Reply with quote
fasmnewbie



Joined: 01 Mar 2011
Posts: 555
fasmnewbie 20 Apr 2018, 04:19
It's not easy to standardize anything on Linux beyond the kernel. Kernel services int 80h uses 32-bit fastcall but C uses CDECL. The problem: Any descriptions of the API uses C as the main interface, which in turn was wrapped around fastcall int 80h services.

For 64-bit, the problem is with the "red zone". Red zone is not easy to define with high-level macros (it is still possible, though, if people insist). Red zone is implicit or based rather on assumptions of the last RSP movement following a CALL instruction. This alone opens up countless ways of implemention / interpretations by different people or vendors.

If you debug some 64-bit C program, you'll find that C addresses some local data on stack that you did not allocate yourself. That's red zone. Should FASM use this red zone? What would happen if the function itself use the redzone to place its own data? How much stack be allocated off the redzone should there exists LOCALS data beyond 128-byte of such high-level PROC? Now that's the problem: It's too implementation-dependant. You probably need to use many command line switches to deal with PROC's redzone. ELF64 executable don't have such luxury - you need to come up with a fixed PROC setup. That's a nasty portablity problem too, if for example your code is used by others which do / do not observe red zone. Your LOCALS data will suffer due to stack corruptions.

Here's a short discussion on how nightmarish it is to implement a standardized high-level proc on 64-bit Linux:
https://stackoverflow.com/questions/37941779/why-do-we-need-stack-allocation-when-we-have-a-red-zone

So the best and the most practical choice on Linux 64 is to stay low.
Post 20 Apr 2018, 04:19
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 20 Apr 2018, 13:10
It is not all clear...: StackTop is allways located at [rsp+redzone], but times before that redzone was unexisted, so StackTop was allways pointed by [rsp+0], Now in linux redzone not +0, but +128.

In code:
Code:
mov [rsp+1],rbx ; redzone in rsp+0 to rsp+128, for preserving used redzone from rsp+1 to rsp+1+sizeof(rsp), so ...
; ...here we have up to 119 stack bytes for calls, passing parameters, interupts and so on before preserved rbx will be erased
mov rbx,[rsp+1]


If I correct red zone is nice feature. But dangerous for newby programmers.
But it has no meaning to macros for linux. In assemblers (not in HLL) addressing relative to esp was allways responsibility of programmer , not of compiler (or macros in it).

_________________
I don`t like to refer by "you" to one person.
My soul requires acronim "thou" instead.
Post 20 Apr 2018, 13:10
View user's profile Send private message Send e-mail Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20343
Location: In your JS exploiting you and your system
revolution 20 Apr 2018, 13:20
I think the red zone is from [RSP-128] to [RSP-1]. [RSP+0] and up is the normal stack zone.
Post 20 Apr 2018, 13:20
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 20 Apr 2018, 13:33
continue: Am I correct?:
Int 80h - only for linuxes where enabled 32bit elfs?
Int 80h incompatibable with stack in 64 bit?
For 64bit Linux syscall more preferable in all cases? with dissabled support of 32 bit it is only choise?
sysenter is only for 32 bit Linux, unsupported in all 64 linux? obsolete even in 32 bit versions?

And, is there something exept kernel that goes from one linux version to another? I mean some *.so files?
Post 20 Apr 2018, 13:33
View user's profile Send private message Send e-mail Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 20 Apr 2018, 13:55
to Revolution, push,call,int,syscall not overrides [rsp] value. the only way how it can be realized these instructions override [rsp+128] value instead.

Test: where stored value in callee of retaddr in [rsp], or in [rsp+128] for case callee has no rbp frame, has no params, not preserves registers.

redzone cant be located in [rsp-128] because in any place we could use redzone
push rax
mov [rsp],rcx ;rcx redzoned ; rsp points top of redzone not top of stack
...
mov rcx,[rsp] ;rcx restored
pop rax ; original rax will be restored


Last edited by ProMiNick on 20 Apr 2018, 14:04; edited 1 time in total
Post 20 Apr 2018, 13:55
View user's profile Send private message Send e-mail Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20343
Location: In your JS exploiting you and your system
revolution 20 Apr 2018, 14:01
ProMiNick wrote:
to Revolution, push,call,int,syscall not overrides [rsp] value. the only way how it can be realized these instructions override [rsp+128] value instead.
I think that [RSP+128} (or any [RSP+c]) is the normal user stack space. No call or push should ever alter memory above the RSP value else programs will crash pretty quickly.

{RSP-c] is below the normal stack and is where the red zone is defined.
Post 20 Apr 2018, 14:01
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 20 Apr 2018, 14:06
so redzone not hardware feature? it HLL trick?
Post 20 Apr 2018, 14:06
View user's profile Send private message Send e-mail Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 799
Location: Russian Federation, Sochi
ProMiNick 20 Apr 2018, 14:09
what about code from https://stackoverflow.com/questions/37941779/why-do-we-need-stack-allocation-when-we-have-a-red-zone
if I remove sub rsp,32 and add rsp,32 as they not needed for redzone, and remove rbp frame -not used here - left this
Code:
function:
    mov  QWORD PTR [rsp],   rcx   ; copy rcx into our scratch area - this shouldn`t affect ret address
    mov  QWORD PTR [rsp+8], rdx   ; copy rdx into our scratch area

    ; ...do something that clobbers rcx and rdx...

    mov  rcx, [rsp]               ; retrieve original value of rcx from our scratch area
    mov  rdx, [rsp+8]             ; retrieve original value of rdx from our scratch area
    ret
Post 20 Apr 2018, 14:09
View user's profile Send private message Send e-mail Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20343
Location: In your JS exploiting you and your system
revolution 20 Apr 2018, 14:20
ProMiNick wrote:
so redzone not hardware feature? it HLL trick?
It is purely a software defined zone. The hardware will overwrite [RSP-8] on the first push or call. Hence the red zone can't be used across function calls. It can only be used between function calls, or in leaf functions. Otherwise you can adjust RSP (with sub RSP,...) to reserve the space, then you can use [RSP+c] as normal without fear of it being overwritten by calls or pushes.
Post 20 Apr 2018, 14:20
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Main index Download Documentation Examples Message board

Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.