flat assembler
Message board for the users of flat assembler.

Index > Windows > GetSaveFileName dialog and application heap

Author
Thread Post new topic Reply to topic
SokilOff



Joined: 20 Sep 2010
Posts: 15
SokilOff 22 Feb 2016, 21:38
Hello.

Recently I ran into an interesting problem.

There's a small program that allocates different amounts of memory in the heap, then fills it with appropriate values and then calls the standard GetSaveFileName dialog to save the result.

The problem begins after allocating some amounts of heap and calling GetSaveFileName - it crashes somewhere deep inside of COMDLG32.DLL. With some other amounts of allocated heap program doesn't crash and GetSaveFileName works as it should.

Let's say when I allocate and use 160k and then call GetSaveFileName the program crashes, with 180k - no crash, with 320k - crash etc.

I'd think about incorrect flags or incorrectly filled mandatory fields of GetSaveFileName or bad stack alignment, but it works in most cases. Looks like it crashes in some cases depending on "configuration" of allocated heap pieces. I wonder how to trace the reason of it ?
Post 22 Feb 2016, 21:38
View user's profile Send private message Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1694
Location: Toronto, Canada
AsmGuru62 22 Feb 2016, 22:12
Can you provide the crashing code?
Post 22 Feb 2016, 22:12
View user's profile Send private message Send e-mail Reply with quote
SokilOff



Joined: 20 Sep 2010
Posts: 15
SokilOff 22 Feb 2016, 23:32
AsmGuru62 wrote:
Can you provide the crashing code?


Here's the part of the problematic function. Host OS is Win7 64-bit, program is 32-bit, uses unicode functions.
ramBuff - address of previously allocated piece of heap (via HeapAlloc), ramSize - it's size
If ramBuff = 160 kb, the program crashes, if it's = 180 kb, all goes well.

Code:
proc         SaveData   ramBuff, ramSize

             local     hFile:DWORD
             local     BytesWritten:DWORD
             local     ofn:OPENFILENAME

             stdcall   ZeroMem, fileName, fileName.Size           ; simple function that fills RAM with zeroes
             stdcall   ZeroMem, addr ofn, sizeof.OPENFILENAME
             mov       [hFile], 0

             mov       [ofn.lStructSize], sizeof.OPENFILENAME
             mov       eax,[hMainWindow]
             mov       [ofn.hwndOwner], eax
             mov       eax,[hMyApp]
             mov       [ofn.hInstance], eax
             mov       [ofn.nMaxFile], 260
             mov       [ofn.lpstrFile], fileName
             mov       [ofn.lpstrFilter], flFilter
             mov       [ofn.lpstrDefExt], flDefExt
             mov       [ofn.lpstrTitle], flHeader
             mov       [ofn.Flags], OFN_LONGNAMES or OFN_OVERWRITEPROMPT or OFN_HIDEREADONLY or OFN_DONTADDTORECENT
             invoke    GetSaveFileName, addr ofn   ; <---- it crashes here before saving anything from allocated buffer
             test      eax, eax
             jz        .Exit

; ...rest of the procedure...

     .Exit:

             ret
endp 
    


And in the .data section (for unicode GetSaveFileName):
Code:
  flHeader    du       'Save data to...', 0
  flFilter    du       'Data file (*.dat)', 0, '*.dat', 0, 0
  flDefExt    du       'dat', 0

  fileName:      rw       260 ; max.path in t_chars
         .Size    =       $-fileName 
    
Post 22 Feb 2016, 23:32
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20537
Location: In your JS exploiting you and your system
revolution 22 Feb 2016, 23:36
SokilOff: Insufficient information. Please provide a minimal program that exhibits the fault.
Post 22 Feb 2016, 23:36
View user's profile Send private message Visit poster's website Reply with quote
SokilOff



Joined: 20 Sep 2010
Posts: 15
SokilOff 23 Feb 2016, 00:17
revolution: I know, but it's relatively big piece of code and it's not that easy to cut everything "unnecessary". That's why I thought maybe someone has any idea about the nature of this kind of problem (or how to trace down such crashes).
Post 23 Feb 2016, 00:17
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20537
Location: In your JS exploiting you and your system
revolution 23 Feb 2016, 00:21
The problem will be in your code, not the API. We can't help you with just the code you provided. The code you show doesn't even use ramBuff!

My guess would be you have overlapping buffers, or perhaps some buffer is too short. But there is no way to know, you didn't show what you have done.
Post 23 Feb 2016, 00:21
View user's profile Send private message Visit poster's website Reply with quote
SokilOff



Joined: 20 Sep 2010
Posts: 15
SokilOff 23 Feb 2016, 02:02
revolution wrote:
The problem will be in your code, not the API. We can't help you with just the code you provided. The code you show doesn't even use ramBuff!

The most strange thing - crashes happened _before_ something was done with ramBuff.

Quote:
My guess would be you have overlapping buffers, or perhaps some buffer is too short.

I've changed my code and instead of using default heap (via GetProcessHeap) I create a separate heap using HeapCreate. It helped and crashes are gone. So your guess was probably correct. Thanks !
Post 23 Feb 2016, 02:02
View user's profile Send private message Reply with quote
Trinitek



Joined: 06 Nov 2011
Posts: 257
Trinitek 23 Feb 2016, 04:40
If I'm not mistaken, I thought the Microsoft documentation mentioned somewhere that it's safer to create a new heap instead of getting the preexisting heap for the process? I do remember purposefully making the decision to use HeapCreate instead of GetProcessHeap in order to avoid a possible situation like this, at least.
Post 23 Feb 2016, 04:40
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20537
Location: In your JS exploiting you and your system
revolution 23 Feb 2016, 04:47
Trinitek wrote:
If I'm not mistaken, I thought the Microsoft documentation mentioned somewhere that it's safer to create a new heap instead of getting the preexisting heap for the process?
Link? Safer in what way?

I don't see how it would make a difference.
Post 23 Feb 2016, 04:47
View user's profile Send private message Visit poster's website Reply with quote
Trinitek



Joined: 06 Nov 2011
Posts: 257
Trinitek 23 Feb 2016, 05:27
Can't find one. I guess I was mistaken.

The first and only time that I used HeapCreate was for a TSP bruteforcer DLL, and it seemed really nice to be able to have my own private heap and destroy it in one sweep with HeapDestroy. Although it's true, I don't really see why using the process heap wouldn't also work, but since I was doing quite a few things that were new to me, I thought it would be one less point of possible confusion and failure.
Post 23 Feb 2016, 05:27
View user's profile Send private message Reply with quote
SokilOff



Joined: 20 Sep 2010
Posts: 15
SokilOff 23 Feb 2016, 15:49
If I'm not mistaken standard shell dialogs like GetSaveFileName use default program heap. Moreover 3rd party shell extensions (they could contain errors as well) will be also loaded into process' memory. All this can lead to different unexpected results. That's why creating separate heap may be a bit safer way to handle such situations.
Post 23 Feb 2016, 15:49
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.