flat assembler
Message board for the users of flat assembler.

Index > Windows > Problems with 64-bit

Author
Thread Post new topic Reply to topic
Hornet52



Joined: 20 Feb 2011
Posts: 11
Location: NULL
Hornet52
Hey,

I had problems before with the Win32 api but I think I have it figured out pretty good so I thought I'd move on to the Win64 api. I unfortunately had trouble with this due to the poor amount of resources available for the Win64 api. Below is my best attempt at understanding it. Currently the CreateWindowEx fails with an "invalid window handle"(error 1400), which I find difficult to understand.

A couple more questions:

i.) A Win64 application starts with a "sub RSP, 8", due to the qword push by an api call (which realigns the stack on 16 byte boundary), correct?

ii.) The Win64 api uses the fastcall convention which needs a 32 byte "shadow stack" plus whatever parameters come before that, correct?

code:
Code:
Main:
   sub RSP, 0x08                                 ;align stack, (for call which subs 0x08)

   mov RCX, 0x00                                 ;Get hInstance for class etc.
   sub RSP, 0x20
   call QWORD [GetModuleHandleA]
   add RSP, 0x20
   mov [WNDCLASSEX.hInstance], RAX

   mov RCX, 0x00
   mov RDX, 0x7F00                               ;Get a cursor for application
   sub RSP, 0x20
   call QWORD [LoadCursorA]
   add RSP, 0x20
   mov [WNDCLASSEX.hCursor], RAX

   mov RCX, 0x00000096                           ;Colour background RGB(150,0,0)
   sub RSP, 0x20
   call QWORD [CreateSolidBrush]
   add RSP, 0x20   
   mov [WNDCLASSEX.hbrBackground], RAX

   lea RCX, [WNDCLASSEX]                         ;Register windows class
   sub RSP, 0x20
   call QWORD [RegisterClassExA]
   add RSP, 0x20

   mov RCX, 0x00
   lea RDX, [String.ClassName]
   lea R8, [String.WindowName]
   mov R9, 0x10CF0000                            ;WS_OVERLAPPEDWINDOW | WS_VISIBLE
   push QWORD 0x64
   push QWORD 0x64
   push QWORD 0x01F4
   push QWORD 0x01F4
   push QWORD 0x00
   push QWORD 0x00
   push QWORD [WNDCLASSEX.hInstance]
   push QWORD 0x00                               ;Create Window
   sub RSP, 0x20
   call QWORD [CreateWindowExA] 
   add RSP, 0x60                                 ;cleanup stack             

   or RAX, RAX
   jz PostError
   
MessageLoop:
   lea RCX, [MSG]
   xor RDX, RDX
   xor R8, R8
   xor R9, R9
   sub RSP, 0x20
   call QWORD [GetMessageA]
   add RSP, 0x20
   
   cmp EAX, 0xFFFFFFFF                           ;check for GetMessage error
   je PostError
   or EAX, EAX
   jz WM_QUIT                                    ;check for quiting

   lea RCX, [MSG]
   sub RSP, 0x20
   call QWORD [TranslateMessage]
   add RSP, 0x20
   lea RCX, [MSG]
   sub RSP, 0x20
   call QWORD [DispatchMessageA]
   add RSP, 0x20
jmp MessageLoop


WndProc:                                                                       
   push RBP                                      ;[RBP+0x28] LParam
   mov RBP, RSP                                  ;[RBP+0x20] WParam
   push RBX                                      ;[RBP+0x18] Message (dword)
   push RSI                                      ;[RBP+0x10] Hwnd
   push RDI                                      ;[RBP+0x08] Callback address
   push R12                                      ;[RBP+0x00] Last RBP
   push R13
   push R14
   push R15
   mov [RBP+0x10], RCX
   mov [RBP+0x18], RDX
   mov [RBP+0x20], R8
   mov [RBP+0x28], R9

   cmp EDX, 0x02
   je WM_DESTROY
   cmp EDX, 0x012 
   je WM_QUIT

.Default:
   sub RSP, 0x20
   call QWORD [DefWindowProcA]                   ;params should be setup from input
   add RSP, 0x20

.Return:
   pop R15
   pop R14
   pop R13
   pop R12
   pop RDI
   pop RSI
   pop RBX
   pop RBP
   ret 

WM_DESTROY:
   mov RCX, 0x00
   sub RSP, 0x20
   call QWORD [PostQuitMessage]
   add RSP, 0x20
   jmp WndProc.Return
   
WM_QUIT:
   mov RCX, [MSG.wParam]
   sub RSP, 0x20
   call QWORD [ExitProcess]
   add RSP, 0x20

PostError:                                       ;Make better error handler later
   mov RCX, 0x00
   lea RDX, [String.ErrorMsg]
   mov R8, 0x00
   mov R9, 0x10
   sub RSP, 0x20
   call QWORD [MessageBoxA]
   add RSP, 0x20
   jmp WM_QUIT
                                                 ;===================.data===================

WNDCLASSEX:
  .cbSize        dd 0x50                         ;80 bytes (same everytime)
  .style         dd 0x03                         ;CS_HREDRAW | CS_VREDRAW, redraw
  .lpfnWndProc   dq WndProc
  .cbClsExtra    dd 0x00
  .cbWndExtra    dd 0x00
  .hInstance     dq 0x00                         ;Will be set in Main
  .hIcon         dq 0x00                         ;No Icon
  .hCursor       dq 0x00                         ;Will be set in Main
  .hbrBackground dq 0x00                         ;Will be set in Main
  .lpszMenuName  dq 0x00             
  .lpszClassName dq String.ClassName             ;"Main Class"
  .hIconSm       dq 0x00   

MSG:
  .hwnd    dq 0x00
  .message dd 0x00
  .padQ1   dd 0x00
  .wParam  dq 0x00
  .lParam  dq 0x00
  .time    dd 0x00
  .ptx     dd 0x00
  .pty     dd 0x00
  .padQ2   dd 0x00

String:
.ClassName:
   db "Main Class", 0x00 
.WindowName:
   db "Basic Window", 0x00
.Hello:
   db "Hello World!", 0x00
.ErrorMsg:
   db "Fatal Error! Application closing", 0x00
    


Thanks
-Hornet
Post 24 Feb 2011, 06:46
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
Your parameters for CreateWindowExA are pushed in reverse order.

Also you should align all the structures you pass to the APIs to at least mod 8.
Post 24 Feb 2011, 07:37
View user's profile Send private message Visit poster's website Reply with quote
Hornet52



Joined: 20 Feb 2011
Posts: 11
Location: NULL
Hornet52
Pushed in reverse? I thought in the fastcall convention additional parameters were pushed on the stack left-to-right?

I was actually sure about the structures and was more worried if the api's would actually recognize that they're padded, thanks for the tip though
Post 24 Feb 2011, 08:08
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17270
Location: In your JS exploiting you and your system
revolution
If you want your code to work as expected then I suggest you push your parameters in the reverse of what you currently have them.

It is important to align your structures. If you don't you can sometimes get unexpected failures during the API calls. Padding of internal parameters of structures is also required, it is not detected by the APIs, the API functions just assume it is done properly.
Post 24 Feb 2011, 13:29
View user's profile Send private message Visit poster's website Reply with quote
Hornet52



Joined: 20 Feb 2011
Posts: 11
Location: NULL
Hornet52
Well I tried pushing the parameters in reverse order (see below) but it doesn't work as I exactly expect. No error is thrown from what I can tell but the process just dies silently.


Code:
Main:
   sub RSP, 0x08                                 ;align stack, (for call which subs 0x08)

   mov RCX, 0x00                                 ;Get hInstance for class etc.
   sub RSP, 0x20
   call QWORD [GetModuleHandleA]
   add RSP, 0x20
   mov [WNDCLASSEX.hInstance], RAX

   mov RCX, 0x00
   mov RDX, 0x7F00                               ;Get a cursor for application
   sub RSP, 0x20
   call QWORD [LoadCursorA]
   add RSP, 0x20
   mov [WNDCLASSEX.hCursor], RAX

   mov RCX, 0x00000096                           ;Colour background RGB(150,0,0)
   sub RSP, 0x20
   call QWORD [CreateSolidBrush]
   add RSP, 0x20   
   mov [WNDCLASSEX.hbrBackground], RAX

   lea RCX, [WNDCLASSEX]                         ;Register windows class
   sub RSP, 0x20
   call QWORD [RegisterClassExA]
   add RSP, 0x20

   mov RCX, 0x00
   lea RDX, [String.ClassName]
   lea R8, [String.WindowName]
   mov R9, 0x10CF0000                            ;WS_OVERLAPPEDWINDOW | WS_VISIBLE
   push QWORD 0x00                               ;Create Window 
   push QWORD [WNDCLASSEX.hInstance]
   push QWORD 0x00
   push QWORD 0x00
   push QWORD 0x01F4
   push QWORD 0x01F4
   push QWORD 0x64
   push QWORD 0x64
   sub RSP, 0x20
   call QWORD [CreateWindowExA] 
   add RSP, 0x60                                 ;cleanup stack             

   or RAX, RAX
   jz PostError
   
MessageLoop:
   lea RCX, [MSG]
   xor RDX, RDX
   xor R8, R8
   xor R9, R9
   sub RSP, 0x20
   call QWORD [GetMessageA]
   add RSP, 0x20
   
   cmp EAX, 0xFFFFFFFF                           ;check for GetMessage error
   je PostError
   or EAX, EAX
   jz WM_QUIT                                    ;check for quiting

   lea RCX, [MSG]
   sub RSP, 0x20
   call QWORD [TranslateMessage]
   add RSP, 0x20
   lea RCX, [MSG]
   sub RSP, 0x20
   call QWORD [DispatchMessageA]
   add RSP, 0x20
jmp MessageLoop


WndProc:                                                                       
   push RBP                                      ;[RBP+0x28] LParam
   mov RBP, RSP                                  ;[RBP+0x20] WParam
   push RBX                                      ;[RBP+0x18] Message (dword)
   push RSI                                      ;[RBP+0x10] Hwnd
   push RDI                                      ;[RBP+0x08] Callback address
   push R12                                      ;[RBP+0x00] Last RBP
   push R13
   push R14
   push R15
   mov [RBP+0x10], RCX
   mov [RBP+0x18], RDX
   mov [RBP+0x20], R8
   mov [RBP+0x28], R9

   cmp EDX, 0x02
   je WM_DESTROY
   cmp EDX, 0x012 
   je WM_QUIT

.Default:
   sub RSP, 0x20
   call QWORD [DefWindowProcA]                   ;params should be setup from input
   add RSP, 0x20

.Return:
   pop R15
   pop R14
   pop R13
   pop R12
   pop RDI
   pop RSI
   pop RBX
   pop RBP
   ret 

WM_DESTROY:
   mov RCX, 0x00
   sub RSP, 0x20
   call QWORD [PostQuitMessage]
   add RSP, 0x20
   jmp WndProc.Return
   
WM_QUIT:
   mov RCX, [MSG.wParam]
   sub RSP, 0x20
   call QWORD [ExitProcess]
   add RSP, 0x20

PostError:                                       ;Make better error handler later
   mov RCX, 0x00
   lea RDX, [String.ErrorMsg]
   mov R8, 0x00
   mov R9, 0x10
   sub RSP, 0x20
   call QWORD [MessageBoxA]
   add RSP, 0x20
   jmp WM_QUIT
                                                 ;===================.data===================

ALIGN 0x08, db 0x00
WNDCLASSEX:
  .cbSize        dd 0x50                         ;80 bytes (same everytime)
  .style         dd 0x03                         ;CS_HREDRAW | CS_VREDRAW, redraw
  .lpfnWndProc   dq WndProc
  .cbClsExtra    dd 0x00
  .cbWndExtra    dd 0x00
  .hInstance     dq 0x00                         ;Will be set in Main
  .hIcon         dq 0x00                         ;No Icon
  .hCursor       dq 0x00                         ;Will be set in Main
  .hbrBackground dq 0x00                         ;Will be set in Main
  .lpszMenuName  dq 0x00             
  .lpszClassName dq String.ClassName             ;"Main Class"
  .hIconSm       dq 0x00   

ALIGN 0x08, db 0x00
MSG:
  .hwnd    dq 0x00
  .message dd 0x00
  .padQ1   dd 0x00
  .wParam  dq 0x00
  .lParam  dq 0x00
  .time    dd 0x00
  .ptx     dd 0x00
  .pty     dd 0x00
  .padQ2   dd 0x00

String:
.ClassName:
   db "Main Class", 0x00 
.WindowName:
   db "Basic Window", 0x00
.Hello:
   db "Hello World!", 0x00
.ErrorMsg:
   db "Fatal Error! Application closing", 0x00
    
Post 24 Feb 2011, 18:44
View user's profile Send private message Reply with quote
b1528932



Joined: 21 May 2010
Posts: 287
b1528932
why dont you make macro for those pushes?

And why use pushes at all.
Subtract sp, and mov. or even rep movs. Keep in ming that if u use guard page you have to limit ammount of arguments 'pushed'.

Code:
   mov RCX, 0x00
   lea RDX, [String.ClassName]
   lea R8, [String.WindowName]
   mov R9, 0x10CF0000                            ;WS_OVERLAPPEDWINDOW | WS_VISIBLE
   push QWORD 0x00                               ;Create Window 
   push QWORD [WNDCLASSEX.hInstance]
   push QWORD 0x00
   push QWORD 0x00
   push QWORD 0x01F4
   push QWORD 0x01F4
   push QWORD 0x64
   push QWORD 0x64
   sub RSP, 0x20
   call QWORD [CreateWindowExA]      

push in reverse order


and use virtual macro to allocate uninitialized data. YOu setup stack frame at the beggining, call bp - XX a WNDCLASS and access it like a struct.
Post 24 Feb 2011, 21:19
View user's profile Send private message Reply with quote
asmhack



Joined: 01 Feb 2008
Posts: 431
asmhack
Post 25 Feb 2011, 01:28
View user's profile Send private message Reply with quote
Hornet52



Joined: 20 Feb 2011
Posts: 11
Location: NULL
Hornet52
@asmhack: Thanks for the link, it actually answered a number of questions for me. A small error still remains (possibly with stack alignment), but I've started to look through some more (albeit poorly written 64bit material) which hopefully should lead to a solution.
Post 25 Feb 2011, 20:01
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.