flat assembler
Message board for the users of flat assembler.

Index > Main > Heap corruption debugging

Author
Thread Post new topic Reply to topic
donn



Joined: 05 Mar 2010
Posts: 178
donn
Any quick pointers on fixing assembly heap corruption errors?

I believe I can reproduce the issue, occurs when I comment out every test (using GoogleTest) except one 'Storage' test that lists files in a directory. The fault occurs after the application exits and is very intermittent.

The GoogleTest test calls retrieveRecordNames, an assembly function exported as MS COFF in Storage.inc. GoogleTest call looks like:
Code:
        List* recordNames = retrieveRecordNames(record);    


Which makes a couple calls to 'string manipulation' (Letters.inc) calls and pattern matching calls (Matching.inc).

There are some WinAPI calls. The calling convention looks correct, stack seems fine, think I just made a typo somewhere, but bit time consuming.

When attaching the VS debugger:

Code:
//
// free_base.cpp
//
//      Copyright (c) Microsoft Corporation. All rights reserved.
…
extern "C" void __declspec(noinline) __cdecl _free_base(void* const block)
{
    if (block == nullptr)
    {
        return;
    }

    if (!HeapFree(select_heap(block), 0, block))
    


Fails on the last line here.

Links:
Azure build (fails in GoogleTest):
https://dev.azure.com/lewissall/hpdporg/_build/results?buildId=58&view=logs&j=12f1170f-54f2-53f3-20dd-22fc7dff55f9&t=12f1170f-54f2-53f3-20dd-22fc7dff55f9
Storage.inc call:
https://github.com/hpdporg/datap/blob/4d810f0ebe9b90c95516d50803cc2d244ea22287/src/main/asm/Storage/Storage.inc#L783
String manipulation calls:
https://github.com/hpdporg/datap/blob/master/src/main/asm/Letters/Letters.inc
Pattern matching call:
https://github.com/hpdporg/datap/blob/4d810f0ebe9b90c95516d50803cc2d244ea22287/src/main/asm/Matching/Matching.inc#L907
Test:
https://github.com/hpdporg/datap/blob/4d810f0ebe9b90c95516d50803cc2d244ea22287/src/test/cpp/StorageTests.cpp#L80

I can figure it out, but was wondering if anyone had any tricks they used, like calling the function many times. I still don't step through the assembly with a debugger, but I can print out debugging messages (I write debug messages into a file).

Take care.
Post 16 Apr 2020, 18:18
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
Maybe I missed something. What is this related to? Is this a bug in fasmg?
Post 16 Apr 2020, 18:55
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Oh no, it's a bug in a project I'm working on. It's the third nasty heap corruption issue I've run into. Just curious if anyone has any best practices or tips they use when they hit these. I'll figure it out if not.
Post 16 Apr 2020, 18:57
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7725
Location: Kraków, Poland
Tomasz Grysztar
Personally I am still using MemProof to debug such things. It is an outdated and no longer maintained tool, so perhaps you might find something more modern.

I'm attaching a screenshot of a quick example I made to demonstrate how you can use such tool. The write past the allocated size has been detected, and the size of a block can be a helpful hint for debugging if the bug is some kind of a buffer overflow. If the bug showed up consistently as a write past some specific memory block, I would then set up (in a debugger) a breakpoint on write to memory addresses at the end of that block, to find the code that does the offending write.


Description: Detection of a heap corruption.
Filesize: 61.87 KB
Viewed: 593 Time(s)

memproof.png


Post 16 Apr 2020, 19:22
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Oh neat, downloaded it and tried running the exe but it said 'Error starting the debug process ... datapTests.exe'

I'll look into it more tomorrow, maybe it's because the exe was from a Release, non-Debug build in Visual Studio, or maybe I just ran it wrong.

Also found this article, seems there's a WinAPI you can dump:
https://docs.microsoft.com/en-us/visualstudio/debugger/finding-memory-leaks-using-the-crt-library?view=vs-2019
Post 17 Apr 2020, 02:00
View user's profile Send private message Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 707
Location: Belarus
DimonSoft
Since when CRT is considered WinAPI?
Post 17 Apr 2020, 07:39
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2915
Location: [RSP+8*5]
bitRAKE

_________________
¯\(°_o)/¯ unlicense.org
Post 17 Apr 2020, 14:59
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
I hadn't read the article in full yet, but interesting. CRT is not WinAPI, but seems Microsoft provides some functionality on Windows that you can't see elsewhere:

GCC equivalent to Microsoft crtdbg
crtdumpmemoryleaks

UCRT is Windows only, but not a pure Windows API: Introducing the Universal CRT since it's an implementation with Windows-only enhancements. It's certainly not a classical Win32 API but check out the Microsoft tools section of this win32 memleak guide, CRT Debug Heap is linked! Yes, UCRT is provided in the Windows SDK now, but I'm getting to used to these lines getting blurred, which is not good. Seems Microsoft is really embracing these blurred lines now, with .net on Linux, Edge on Chrome, they're losing their heritage a bit.
Post 17 Apr 2020, 16:08
View user's profile Send private message Reply with quote
Ali.Z



Joined: 08 Jan 2018
Posts: 344
Ali.Z
IIRC, crt was included since win nt4.0

_________________
Asm For Wise Humans
Post 17 Apr 2020, 23:49
View user's profile Send private message Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 707
Location: Belarus
DimonSoft
Post 18 Apr 2020, 16:58
View user's profile Send private message Visit poster's website Reply with quote
donn



Joined: 05 Mar 2010
Posts: 178
donn
Interesting articles. I finally just fixed the heap corruption error by the way!

The problem was here:

Code:
        sub rsp, 8*8
        mov rcx, 64                                     ; Size
        call allocate.linear
        add rsp, 8*8
        mov [w32FindData], rax    


When I increased the allocation size of the w32_find_data struct, the problem went away. I'm a bit exhausted, so I could be wrong but I ran a ton of tests, built, and the build passes too:

Passing Azure Build

Definitely have to look into this structure again and get the correct size. Quite exhausted so I just padded it with a ton of space as a band-aid. Process I used to pinpoint this was process of elimination (comment out setup and teardown in GoogleTest, each of the matching and string manip calls) and log debug messages of how many times findNextFile is called.

I will have to revisit this to get the correct size, but the other good news is that I had a Hadoop big data project that had the exact same heap corruption error a year ago that I never had the time to solve. Fingers crossed this was it...
Post 21 Apr 2020, 00:09
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.