flat assembler
Message board for the users of flat assembler.

Index > Windows > call, stdcall, fastcall, invoke confusion; 64-bit style

Author
Thread Post new topic Reply to topic
farrier



Joined: 26 Aug 2004
Posts: 274
Location: North Central Mississippi
farrier
call, stdcall, fastcall, invoke confusion; 64-bit style

Just tried converting a 32 bit program to 64 bit, and immediately ran into problems with my own proc creation, and calling those procs. The small program shown has one proc called Proc_exp. The main program tries to invoke the Proc_exp but invoke would never work in 64 bit, "operand size not specified". stdcall worked for 32 bit and 64 bit. In Proc_exp all that is done is invoke MessageBox, but in 64 bit the values passed to MessageBox were taken from the stack and not from ecx, edx, r8, and r9 as expected. What is the proper way of calling invoking in a proc in 64 bit world.

Code:
format PE64 GUI
;format PE GUI 5.0
entry start

include 'win64axp.inc'
;include 'win32axp.inc'

section '.text'  code readable executable
 start:                                                                       ;32 bit                                         64 bit
      sub             rsp, 8                                          ;                                               sub rsp,08
  stdcall Proc_exp, 0, mb_text, mb_title, 0                       ;PUSH 0                                         sub rsp,20
                                                                  ;PUSH OFFSET 00402011                           mov rcx,00000000
                                                                    ;PUSH OFFSET 00402011                           mov rdx,00402000
                                                                    ;PUSH 0                                         mov r8,00402011
                                                                     ;                                               mov r9,00000000
                                                                     ;CALL 0040101B                                  call 0000000000401042
                                                                       ;                                               add rsp,20
                                                                  ;                                               sub rsp,20
  invoke  ExitProcess, 0                                          ;PUSH 0                                         mov rcx,00000000
                                                                    ;CALL DWORD PTR DS:[<&KERNEL32.ExitProcess>]      call qword [0000000000403068]
proc   Proc_exp, hWnd, lpText, lpTitle, bType

  invoke  MessageBox, [hWnd], [lpText], [lpTitle], [bType]        ;PUSH EBP                                       push rbp
                                                                    ;MOV EBP,ESP                                    mov rbp,rsp
                                                                 ;                                               sub rsp,20
                                                                  ;PUSH DWORD PTR SS:[ARG.4]                  mov rcx,[rbp+10]
                                                                    ;PUSH DWORD PTR SS:[ARG.3]                  mov rdx,[rbp+18]
                                                                    ;PUSH DWORD PTR SS:[ARG.2]                  mov r8,[rbp+20]
                                                                     ;PUSH DWORD PTR SS:[ARG.1]                  mov r9,[rbp+28]
                                                                     ;CALL DWORD PTR DS:[<&USER32.MessageBoxA>]        call qword [0000000000403098]
                                                                       ;                                               add rsp,20
  ret                                                             ;LEAVE                                          leave
                                                                       ;RETN 10                                        ret
endp

section '.data'  data readable writeable

mb_text    db      "MessageBox Text!", 0
mb_title     db      "MessageBox Title!", 0

section '.idata' import data readable

  library        kernel32,       'KERNEL32.DLL',\
                 user32,         'USER32.DLL',\
                   comctl32,       'COMCTL32.DLL',\
                 comdlg32,       'comdlg32',\
                     shell32,        'SHELL32.DLL'

 include 'api\kernel32.inc'
       include 'api\user32.inc'
 include 'api\comctl32.inc'
       include 'api\comdlg32.inc'
       include 'api\shell32.inc'

    













CPU Disasm
Address Hex dump Command Comments
00401000 /. 6A 00 PUSH 0 ; /Arg4 = 0
00401002 |. 68 11204000 PUSH OFFSET 00402011 ; |Arg3 = ASCII "MessageBox Title!"
00401007 |. 68 00204000 PUSH OFFSET 00402000 ; |Arg2 = ASCII "MessageBox Text!"
0040100C |. 6A 00 PUSH 0 ; |Arg1 = 0
0040100E |. E8 08000000 CALL 0040101B ; \ExpProc.0040101B
00401013 |. 6A 00 PUSH 0 ; /ExitCode = 0
00401015 \. FF15 60304000 CALL DWORD PTR DS:[<&KERNEL32.ExitProces ; \KERNEL32.ExitProcess
0040101B /$ 55 PUSH EBP ; ExpProc.0040101B(guessed Arg1,Arg2,Arg3,Arg4)
0040101C |. 89E5 MOV EBP,ESP
0040101E |. FF75 14 PUSH DWORD PTR SS:[ARG.4] ; /Type => [ARG.4]
00401021 |. FF75 10 PUSH DWORD PTR SS:[ARG.3] ; |Caption => [ARG.3]
00401024 |. FF75 0C PUSH DWORD PTR SS:[ARG.2] ; |Text => [ARG.2]
00401027 |. FF75 08 PUSH DWORD PTR SS:[ARG.1] ; |hOwner => [ARG.1]
0040102A |. FF15 80304000 CALL DWORD PTR DS:[<&USER32.MessageBoxA> ; \USER32.MessageBoxA
00401030 |. C9 LEAVE
00401031 \. C2 1000 RETN 10



0000000000401000 4883EC08 sub rsp,08
0000000000401004 4883EC20 sub rsp,20
0000000000401008 48C7C100000000 mov rcx,00000000
000000000040100F 48C7C200204000 mov rdx,00402000
0000000000401016 49C7C011204000 mov r8,00402011
000000000040101D 49C7C100000000 mov r9,00000000
0000000000401024 E819000000 call 0000000000401042
0000000000401029 4883C420 add rsp,20
000000000040102D 4883EC20 sub rsp,20
0000000000401031 48C7C100000000 mov rcx,00000000
0000000000401038 FF152A200000 call qword [0000000000403068] ; []=00000000779B00C0=ntdll.RtlExitUserProcess
000000000040103E 4883C420 add rsp,20
0000000000401042 55 push rbp
0000000000401043 4889E5 mov rbp,rsp
0000000000401046 4883EC20 sub rsp,20
000000000040104A 488B4D10 mov rcx,[rbp+10]
000000000040104E 488B5518 mov rdx,[rbp+18]
0000000000401052 4C8B4520 mov r8,[rbp+20]
0000000000401056 4C8B4D28 mov r9,[rbp+28]
000000000040105A FF1538200000 call qword [0000000000403098] ; []=000000007792E96C=USER32.MessageBoxA
0000000000401060 4883C420 add rsp,20
0000000000401064 C9 leave
0000000000401065 C3 ret

_________________
Some Assembly Required
It's a good day to code!
U.S.Constitution; Bill of Rights; Amendment 1:
... the right of the people peaceably to assemble, ...
The code is dark, and full of errors!
Post 16 Sep 2012, 09:30
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17254
Location: In your JS exploiting you and your system
revolution
I you want to use paameters from the stack then you have to put them there yourself:
Code:
proc    Proc_exp, hWnd, lpText, lpTitle, bType

       mov     [hWnd],rcx
  mov     [lpText],rdx
        mov     [lpTitle],r8
        mov     [bType],r9

        invoke  MessageBox, [hWnd], [lpText], [lpTitle], [bType]    
If you know that the registers still hold valid entry values then you can use the registers directly:
Code:
proc    Proc_exp, hWnd, lpText, lpTitle, bType

        invoke  MessageBox, rcx, rdx, r8, r9    
Post 16 Sep 2012, 09:58
View user's profile Send private message Visit poster's website Reply with quote
farrier



Joined: 26 Aug 2004
Posts: 274
Location: North Central Mississippi
farrier
Thanks revolution!

So, am I missing something, or is the proc macro not really useful in the 64 bit world since the parameters passed by stdcall or fastcall are not directly accessible in the proc routine. In both examples you provide, the parameters: hWnd; lpText; lpTitle; bType cannot be used directly, and have to be referenced via rcx, rdx, r8, and r9. What is the use of the parameters in the proc macro, other than to reserve space on the stack?

Would it be more useful to have the proc macro reference rcx, rdx, r8, and r9 directly? Or should I just define the proc without parameters, and just "call" the proc after manually pushing the parameters to rcx, rdx, r8, and r9? Then once in the proc, use the registers as needed.

Thanks for any help.

farrier

_________________
Some Assembly Required
It's a good day to code!
U.S.Constitution; Bill of Rights; Amendment 1:
... the right of the people peaceably to assemble, ...
The code is dark, and full of errors!
Post 16 Sep 2012, 19:38
View user's profile Send private message Reply with quote
LocoDelAssembly
Your code has a bug


Joined: 06 May 2005
Posts: 4633
Location: Argentina
LocoDelAssembly
farrier, the first four params are probably allocated in the so-called spill area, which is a space callers set up for callees to move the parameter registers in there. The reason fasm couldn't automatically take the params by name from the registers is that as soon as you call some other proc/function the registers are garbage (perhaps even your own garbage since you used them to pass the parameters in the call). Since the registers are volatile, you need to manually preserve them.

Take a look at http://flatassembler.net/docs.php?article=win32#1.4 . Perhaps you could write your custom prologue to perform the auto-saving of the parameters onto stack (also try to search both the package and the forum, I think someone already did this).
Post 16 Sep 2012, 22:44
View user's profile Send private message Reply with quote
farrier



Joined: 26 Aug 2004
Posts: 274
Location: North Central Mississippi
farrier
Thanks LocoDelAssembly,

I looked thru the docs on the calling conventions for 64 bit, and never saw a requirement that reference to the passed parameters within the called proc had to be thru the stack instead of using the registers directly. It seems as if this was a fasm design decision. I know there is a requirement to reserve stack space corresponding to the number of parameters even if they are not used, but when referencing the parameters passed to a proc, why not use the registers--by default--instead of the stack locations? This is how they are passed to the proc in the first place, and if the registers need to be saved, then they can be saved. Or offer an option to use registers or stack locations!

Till then, I'll use a "personalized" approach! instead of the standard proc macros.

thanks again!,

farrier

_________________
Some Assembly Required
It's a good day to code!
U.S.Constitution; Bill of Rights; Amendment 1:
... the right of the people peaceably to assemble, ...
The code is dark, and full of errors!
Post 17 Sep 2012, 10:00
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17254
Location: In your JS exploiting you and your system
revolution
the problem is here:
Code:
proc    Proc_exp, hWnd, lpText, lpTitle, bType

 xor rcx,rcx ;does rcx now contain proper data for MessageBox?

        invoke  MessageBox, rcx, rdx, r8, r9    
How would the assembler know if rcx still has valid data? Only the programmer can know the contents of the registers. Perhaps the clearing of rcx is deliberate? Perhaps it is a programmer error? The assembler cannot know.
Post 17 Sep 2012, 10:19
View user's profile Send private message Visit poster's website Reply with quote
farrier



Joined: 26 Aug 2004
Posts: 274
Location: North Central Mississippi
farrier
revolution,

I agree that the registers are more volatile than stack variables, but that is a side effect of the fastcall protocol. The user of the fastcall "invoked" proc must--by definition--be aware of the nature of the registers vs. stack variables.

My point is, the values passed to the proc are in the registers and can be used directly and passed to a subsequent proc. If the user of the proc must transfer the register values to the stack, why use fastcall? This seems to defeat the advantage of using the registers to pass parameters. Pass parameters to subsequent procs via the register-defined-parameters and not an assumed stack variable.

Thanks for the advice,

farrier

_________________
Some Assembly Required
It's a good day to code!
U.S.Constitution; Bill of Rights; Amendment 1:
... the right of the people peaceably to assemble, ...
The code is dark, and full of errors!
Post 17 Sep 2012, 15:56
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17254
Location: In your JS exploiting you and your system
revolution
For a leaf procedure using registers can be a win. And usually leaf procedures are the most frequently called in any program.
Post 17 Sep 2012, 16:00
View user's profile Send private message Visit poster's website Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1408
Location: Toronto, Canada
AsmGuru62
I once noticed a cool pattern in parameters passing.
I had a really big procedure for handling of WM_PAINT message for
the syntax-colored editor and there was a lot of locals: rectangles, fonts,
colors, etc. all kind of stuff I needed for painting.

So, instead of passing stuff around - I was just calling the leaf procedures.
The trick is that EBP does not change by a 'naked' CALL instruction.
If leaf procedure IS NOT USING the proc macro, then EBP is always set to
point to that huge locals block with all the data. So, all I had to do is just CALL my leafs.
I had about 15 leaf procedures there and just sometimes
I was passing maybe one or two registers - the rest was assumed to be under EBP.
And in High-Level Language I had to pass the same values further down the
line of leafs (a few times even) - a waste of PUSH-es and stack frame ENTER/LEAVE codes.

If I needed a local variable in one of the leafs - I was adding it into the root WM_PAINT locals area.
Neat trick, however, if locals area gets large (> 127 bytes) - it bloats the generated code,
because an offset gets to be 4 bytes instead of just 1 byte.
Post 17 Sep 2012, 20:59
View user's profile Send private message Send e-mail Reply with quote
farrier



Joined: 26 Aug 2004
Posts: 274
Location: North Central Mississippi
farrier
AsmGuru62, revolution

Thanks for the help!

I guess I just got used to "the good ole" 32-bit proc and invoke!

My take from this:

When calling a Windows function, or non-documented 64-bit function, use the "new improved" calling techniques: fastcall; invoke. When designing my own functions and calling them, I will use my own design and calling rules, The ideas proposed by AsmGuru62 make sense and will come in handy!

Thanks again,

farrier

_________________
Some Assembly Required
It's a good day to code!
U.S.Constitution; Bill of Rights; Amendment 1:
... the right of the people peaceably to assemble, ...
The code is dark, and full of errors!
Post 18 Sep 2012, 05:08
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.