GetOpenFileName and GetSaveFileName fail on x64

Index > Windows > GetOpenFileName and GetSaveFileName fail on x64

Author

Thread

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 16 May 2014, 15:03

Hi everybody
I have been using the following code to open and save files in my x86 applications for the past few months and everything works fine.

Code:


invoke GetSaveFileNameA,filedialogsave
.if eax = 0
ret
.endif    
;same with open dialog box

data part
title db "ZMH Tech Open and close disk tray",0
filedialogopen OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filename,260,0,0,0,title,OFN_EXPLORER,0,0,0,0,0
filedialogsave OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filenamesave,260,0,0,0,title,OFN_EXPLORER,0,0,0,0,0
filter  db 'All files (*.*)',0,'*.*',0,0

filename rb 260
filenamesave rb 260

I recently tried converting one of my projects to x64 and everything works except for the open and save dialog boxes. when I go to open or save a file the application closes. I read the article at http://stackoverflow.com/questions/4982680/getopenfilename-fails-in-64-bit-but-works-in-32bit and it said something about the alignment or structsize but it did not make sense how to fix this in assembly (since the post is in c++)
does anybody know why this code is failing in x64 and fine in x86, and how to fix it

thanks

16 May 2014, 15:03

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20304
Location: In your JS exploiting you and your system

revolution 16 May 2014, 15:21

For 64-bit code you need to align data structures to multiples of 8 bytes, and the stack to multiples of 16 bytes.

Make sure the filedialogopen is aligned properly by putting "align 8" before it. Your "title" text is probably making it unaligned and you might want to move it in with other byte aligned data.

16 May 2014, 15:21

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 16 May 2014, 15:30

I do have align 8 on my data. here is my complete data section if it helps

Code:

section '.data' data readable writeable

align 8

wc WNDCLASS 0,WindowProc,0,0,NULL,NULL,NULL,COLOR_BTNFACE+1,NULL,title
wc2 WNDCLASS 0,WindowProc2,0,0,NULL,NULL,NULL,COLOR_BTNFACE+1,NULL,form2
wc3 WNDCLASS 0,WindowProc3,0,0,NULL,NULL,NULL,COLOR_BTNFACE+1,NULL,form3

title db "ZMH Tech Open and close disk tray",0
titlefilter db "ZMHTechScreendimmerfilter",0
form2 db "WindowProc2",0
form3 db "WindowProc3",0
exeversion db "1.0.0.0",0
exebitsize db "x64",0
filedialogopen OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filename,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
filedialogsave OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filenamesave,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
filter  db 'All files (*.*)',0,'*.*',0,0
aboutform db 0
aboutformsettings db 0

buffer1 dq ?
buffer2 dq ?
buffer3 dq ?
filename rb 260
filenamesave rb 260
hFile dq ?
Bytes dq ?
hStream dq ?
mathbuffer1 dd ?
mathbuffer2 dd ?
mathbuffer3 dq ?
mathbufferbuffer1 dd ?

DesktopRect RECT ?
msg MSG
client RECT


InetHandle dq ?
UrlHandle dq ?
ReadNext dq ?
DownloadBuffer  rb 7
hthread0 dq ?
thread0status dq ?

I have align 4 on my x86 and I changed it to align 4 as well, but it failed on that as well.

16 May 2014, 15:30

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20304
Location: In your JS exploiting you and your system

revolution 16 May 2014, 16:14

But you don't have align 8 for filedialogopen.

And, yes, of course seeing the entire section helps. One never knows what is important and what is not.

16 May 2014, 16:14

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 16 May 2014, 17:13

Im sorry, but I don't exactly understand where to put the align 8. I tried putting it right above the

Code:

 filedialogopen OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filename,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0

line but this did not work.

is that what you mean?

16 May 2014, 17:13

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 16 May 2014, 17:17

I also thought you could of meant to add align 8 above the actual code of

Code:

align 8
invoke GetSaveFileNameA,filedialogsave
.if eax = 0
ret
.endif

but this did not work either

16 May 2014, 17:17

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20304
Location: In your JS exploiting you and your system

revolution 16 May 2014, 17:44

Make sure your structure internals are aligned properly, and that it starts at mod 8 = 0.

BTW: Code alignment is not important on X86 CPUs.

16 May 2014, 17:44

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 16 May 2014, 18:13

for some reason I can't get this to work out. I am new to asm so I don't know excaty what you mean by structure internals. I put the code in a basic template I use for x64 and it closes the app when I run it. I am posting the template here. could you please help me find the error
thanks

Code:


format PE64 GUI 5.0
entry start

include 'C:\Users\Administrator\Programs\assembly\Fasm\INCLUDE\win64ax.inc'

about = 1000
website = 1001
button1 = 1002
textbox1 = 1003

section '.text' code readable executable

  start:
        sub     rsp,8           ; Make stack dqword aligned

       invoke  GetModuleHandleA,0
        mov     [wc.hInstance],rax
        invoke  LoadIconA,rax,17
        mov     [wc.hIcon],rax
        invoke  LoadCursorA,0,IDC_ARROW
        mov     [wc.hCursor],rax
        invoke  RegisterClassA,wc
        test    rax,rax
        jz      error

; create the form

invoke GetDesktopWindow
invoke GetWindowRect,rax,DesktopRect

;calculate startx

mov rax,qword[DesktopRect.right]
lea rdi, [rax - 800]
shr rdi,1

;calculate starty

mov rax,qword[DesktopRect.bottom]
lea rsi,[rax - 600]
shr rsi,1

        invoke  LoadMenuA,[wc.hInstance],37
        invoke  CreateWindowExA,0,title,title,WS_VISIBLE+WS_OVERLAPPEDWINDOW,rdi,rsi,800,600,NULL,rax,[wc.hInstance],NULL
        test    rax,rax
        jz      error

  msg_loop:
        invoke  GetMessageA,msg,NULL,0,0
        cmp     rax,1
        jb      end_loop
        jne     msg_loop
        invoke  TranslateMessage,msg
        invoke  DispatchMessageA,msg
        jmp     msg_loop

  error:
        invoke  MessageBoxA,NULL,"Startup failed.",NULL,MB_ICONERROR+MB_OK

  end_loop:
        invoke  ExitProcess,[msg.wParam]

proc WindowProc uses rbx rsi rdi, hwnd,wmsg,wparam,lparam

; Note that first four parameters are passed in registers,
; while names given in the declaration of procedure refer to the stack
; space reserved for them - you may store them there to be later accessible
; if the contents of registers gets destroyed. This may look like:
       mov     [hwnd],rcx
       mov     [wmsg],rdx
       mov     [wparam],r8
       mov     [lparam],r9

        cmp     [wmsg],WM_CREATE
        je      .wmcreate
        cmp     [wmsg],WM_SIZE
        je      .wmsize
        cmp     [wmsg],WM_SETFOCUS
        je      .wmsetfocus
        cmp     [wmsg],WM_COMMAND
        je      .wmcommand
        cmp     [wmsg],WM_DESTROY
        je      .wmdestroy
  .defwndproc:
        invoke  DefWindowProcA,[hwnd],[wmsg],[wparam],[lparam]
        jmp     .finish
  .wmcreate:
        invoke  GetClientRect,[hwnd],client

;textbox 1

        invoke  CreateWindowExA,WS_EX_CLIENTEDGE,"EDIT",NULL,WS_TABSTOP+WS_VISIBLE+WS_CHILD,145,400,80,20,[hwnd],textbox1,[wc.hInstance],NULL
        mov     rsi,rax
        invoke  CreateFontA,16,0,0,0,FW_NORMAL,FALSE,FALSE,FALSE,ANSI_CHARSET,OUT_RASTER_PRECIS,CLIP_DEFAULT_PRECIS,DEFAULT_QUALITY,FIXED_PITCH+FF_DONTCARE,NULL
        mov rdi,rax
        invoke  SendMessageA,rsi,WM_SETFONT,rdi,FALSE

;button 1

        invoke  CreateWindowExA,NULL,"BUTTON","button1",WS_TABSTOP+WS_VISIBLE+WS_CHILD+BS_DEFPUSHBUTTON,145,300,80,24,[hwnd],button1,[wc.hInstance],NULL
        mov     rsi,rax
        invoke  CreateFontA,16,0,0,0,FW_NORMAL,FALSE,FALSE,FALSE,ANSI_CHARSET,OUT_RASTER_PRECIS,CLIP_DEFAULT_PRECIS,DEFAULT_QUALITY,FIXED_PITCH+FF_DONTCARE,NULL
        mov rdi,rax
        invoke  SendMessageA,rsi,WM_SETFONT,rdi,FALSE

; form loading

;check for debugger
    
invoke IsDebuggerPresent
test rax, rax
jne .debuggerfound
ret
    
    .debuggerfound:
    invoke ExitProcess,0
  .wmsize:
        invoke  GetClientRect,[hwnd],client
        invoke  MoveWindow,rsi,[client.left],[client.top],[client.right],[client.bottom],TRUE
        xor     rax,rax
        jmp     .finish
  .wmsetfocus:
        invoke  SetFocus,rsi
        xor     rax,rax
        jmp     .finish
  .wmcommand:
        mov     rax,[wparam]
        and     rax,0FFFFh
        cmp     rax,about
        je      .about
        cmp     rax,website
        je      .website
        cmp     rax,button1
        je      .button1
        cmp     rax,textbox1
        je      .textbox1
  .about:
        invoke  MessageBoxA,[hwnd],abouttext,title,MB_OK
        jmp     .finish
  .website:
        invoke  ShellExecuteA,[hwnd],NULL,"http://www.zmhtech.com",NULL,NULL,SW_HIDE
        jmp     .finish
  .button1:
       invoke GetOpenFileNameA,filedialogopen
.if rax = 0
ret
.endif
        jmp     .finish
   .textbox1:
        jmp     .finish
  .wmdestroy:
        invoke  PostQuitMessage,0
        xor     rax,rax
  .finish:
   pop     rdi rsi rbx
   ret

endp

section '.data' data readable writeable

align 8

wc WNDCLASS 0,WindowProc,0,0,NULL,NULL,NULL,COLOR_BTNFACE+1,NULL,title

abouttext         db "ZMH Tech Open and close disk tray, Version 1.0.0.0",13,10
                  db "Copyright © 2013, All rights reserved by ZMH Tech: http://www.zmhtech.com",13,10,0
title db "ZMH Tech Open and close disk tray",0
filedialogopen OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filename,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
filedialogsave OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filenamesave,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
filter  db 'All files (*.*)',0,'*.*',0,0
filename rb 260
filenamesave rb 260

  DesktopRect RECT ?

  msg MSG
  client RECT

section '.idata' import data readable writeable

  library kernel,'KERNEL32.DLL',\
          user,'USER32.DLL',\
          Shell32,'Shell32.dll',\
          Comdlg32,'Comdlg32.dll',\
          gdi32,'GDI32.DLL'

  import kernel,\
         GetModuleHandleA,'GetModuleHandleA',\
         IsDebuggerPresent,"IsDebuggerPresent",\
         ExitProcess,'ExitProcess'

  import user,\
         GetDesktopWindow,'GetDesktopWindow',\
         GetWindowRect,'GetWindowRect',\
         RegisterClassA,'RegisterClassA',\
         CreateWindowExA,'CreateWindowExA',\
         DefWindowProcA,'DefWindowProcA',\
         GetMessageA,'GetMessageA',\
         TranslateMessage,'TranslateMessage',\
         DispatchMessageA,'DispatchMessageA',\
         SendMessageA,'SendMessageA',\
         LoadCursorA,'LoadCursorA',\
         LoadIconA,'LoadIconA',\
         LoadMenuA,'LoadMenuA',\
         GetClientRect,'GetClientRect',\
         MoveWindow,'MoveWindow',\
         SetFocus,'SetFocus',\
         MessageBoxA,'MessageBoxA',\
         PostQuitMessage,'PostQuitMessage'

import Shell32,\
         ShellExecuteA,'ShellExecuteA'

import Comdlg32,\
         GetOpenFileNameA,'GetOpenFileNameA',\
         GetSaveFileNameA,'GetSaveFileNameA'

  import gdi32,\
         CreateFontA,'CreateFontA'

section '.rsrc' resource data readable

  ; resource directory

  directory RT_ICON,icons,\
            RT_GROUP_ICON,group_icons,\
            RT_MANIFEST,manifest,\
            RT_VERSION,versions,\
            RT_MENU,menus

  ; resource subdirectories



  resource icons,\
           1,LANG_NEUTRAL,icon_data

  resource group_icons,\
           17,LANG_NEUTRAL,main_icon

resource manifest,\
         1,LANG_NEUTRAL,manifestxpstyles

 resource versions,\
           1,LANG_NEUTRAL,version

 icon main_icon,icon_data,'ZMH Tech Template.ico'

 resdata manifestxpstyles
         db '<?xml version="1.0" encoding="UTF-8" standalone="yes"?>',13,10
         db '<assembly xmlns="urn:schemas-microsoft-com:asm.v1" manifestVersion="1.0">',13,10
         db '<assemblyIdentity name="x.x.x" processorArchitecture="*" version="5.1.0.0" type="win32"/> ',13,10
         db '<description>no</description>',13,10
         db '<dependency>',13,10
         db '<dependentAssembly>',13,10
         db '<assemblyIdentity type="win32" name="Microsoft.Windows.Common-Controls" version="6.0.0.0" processorArchitecture="*" publicKeyToken="6595b64144ccf1df" language="*" />',13,10
         db '</dependentAssembly>',13,10
         db '</dependency>',13,10
         db '</assembly>',13,10
endres

 versioninfo version,VOS__WINDOWS32,VFT_APP,VFT2_UNKNOWN,LANG_ENGLISH+SUBLANG_DEFAULT,0,\
          'LegalCopyright', 'Copyright © 2013, All rights reserved.',\
          'FileDescription', 'ZMH Tech Open and close disk tray',\
          'FileVersion', '1.0.0.0',\
          'InternalName', 'ZMH Tech Open and close disk tray',\
          'OriginalFilename', 'ZMH Tech Open and close disk tray.exe',\
          'ProductName', 'ZMH Tech Open and close disk tray'

 resource menus,\
           37,LANG_ENGLISH+SUBLANG_DEFAULT,main_menu

menu main_menu
       menuitem 'Help',0,MFR_POPUP + MFR_END
                menuitem 'About',about
                menuseparator
                menuitem 'Website',website,MFR_END

16 May 2014, 18:13

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20304
Location: In your JS exploiting you and your system

revolution 16 May 2014, 23:20

patchariadog wrote:

for some reason I can't get this to work out. I am new to asm so I don't know excaty what you mean by structure internals. I put the code in a basic template I use for x64 and it closes the app when I run it. I am posting the template here. could you please help me find the error
thanks

Here is my suggestion for how to lay out your data section:

Code:

section '.data' data readable writeable

;sections are always aligned to mod 4096 = 0

wc WNDCLASS 0,WindowProc,0,0,NULL,NULL,NULL,COLOR_BTNFACE+1,NULL,title
filedialogopen OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filename,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
filedialogsave OPENFILENAME sizeof.OPENFILENAME,0,0,filter,0,0,0,filenamesave,260,0,0,0,NULL,OFN_EXPLORER,0,0,0,0,0
  DesktopRect RECT ?
  msg MSG
  client RECT

;put the byte aligned text last

abouttext         db "ZMH Tech Open and close disk tray, Version 1.0.0.0",13,10
                  db "Copyright © 2013, All rights reserved by ZMH Tech: http://www.zmhtech.com",13,10,0
title db "ZMH Tech Open and close disk tray",0
filter  db 'All files (*.*)',0,'*.*',0,0
filename rb 260
filenamesave rb 260

16 May 2014, 23:20

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 17 May 2014, 02:39

I am so sorry, but I am really confuesd. I think your talking about the sub rsp,8 is wrong or I am not sure really what to do at this point. I don't know if your trying to have me add something to the code or the data section or both. I read fasm post on 64 stack alignment and I still really don't understand why the invoke GetopenfilenameA is failing?

sorry to be confused and bothersome, but thanks for all of the help

17 May 2014, 02:39

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20304
Location: In your JS exploiting you and your system

revolution 17 May 2014, 02:58

"sub rsp,8" is fine (although "and rsp,-16 would be a little more sure).

If you look at my suggested data section above you will see that I have put all the byte aligned data at the end so that it doesn't make any qword data that follows it to be unaligned.

BTW: I ran your unmodified code in a W7-64 system and it showed the open dialog without a problem. Was I supposed to do something else to make it fail?

Last edited by revolution on 17 May 2014, 15:15; edited 1 time in total

17 May 2014, 02:58

patchariadog

Joined: 24 Mar 2013
Posts: 94

patchariadog 17 May 2014, 15:06

I am running windows 8, so I used a computer that had x64 windows 7 and your right it does work fine. so it must be a windows 8 glitch. actually I am running windows 8.1 but I doubt that makes a difference

thanks, I will have to look into this.

17 May 2014, 15:06

madmatt

Joined: 07 Oct 2003
Posts: 1045
Location: Michigan, USA

madmatt 20 May 2014, 10:50

Hi patchariadog, from what I can tell the fasm's OPENFILENAME structure is missing a few things.

try this one:

Code:

struct OPENFILENAMEA
       lStructSize dd ?,?
       hwndOwner dq ?
       hInstance dq ?
       lpstrFilter dq ?
       lpstrCustomFilter dq ?
       nMaxCustFilter dd ?
       nFilterIndex dd ?
       lpstrFile dq ?
       nMaxFile dd ?,?
       lpstrFileTitle dq ?
       nMaxFileTitle dd ?,?
       lpstrInitialDir dq ?
       lpstrTitle dq ?
       Flags dd ?
       nFileOffset dw ?
       nFileExtension dw ?
       lpstrDefExt dq ?
       lCustData dd ?,?
       lpfnHook dq ?
       lpTemplateName dq ?
       pvReserved dq ?
       dwReserved dd ?
       FlagsEx dd ?
ends

20 May 2014, 10:50

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum