flat assembler
Message board for the users of flat assembler.

flat assembler > Windows > [famg win64] An auto growing heap example:

Author
Thread Post new topic Reply to topic
bitRAKE



Joined: 21 Jul 2003
Posts: 2732
Location: dank orb
What is it?

This little example compares the time of reading pages which hit guard pages to those that don't.

Why bother?

It's a common problem with little tool programs: we want it to grow gracefully to handle more than first planned.

Why share?

I can think of at least three API documentation issues which are not clear, or completely in error. Without trial and error these things remain a mystery. First is the allocation sizes allowed in 64-bit windows. It's not DWORD size - a single call can reserve massive blocks of address space.

Related is the idea of VirtualQuery/Protect which are typically superfluous. While committing reserved pages the protection can be established at the same time with VirtualAlloc. Which makes this implementation rather concise.

Finally, the SYSTEM_INFO structure: Every conversion I've seen is in error. dwActiveProcessorMask is neither a DWORD or a PTR - WTF MS means by DWORD_PTR is anyone's guess. It's actually a bit mask of 64-bit size. Assembly coders know what a bit mask is - guess it left MS's vocabulary.

Output should look something like:
Code:
Allocation Granularity: 65536
Page Size: 4096
Reserved 8589934592 bytes at address 000000007FFF0000.
Iniitially committing 65536 bytes.

Testing guard pages...

Commitment read took 25640 ms
Flat read took 15657 ms    


Description:
Download
Filename: AutoHeap.zip
Filesize: 5.21 KB
Downloaded: 37 Time(s)


_________________
unlicense.org
Post 14 Jan 2019, 23:13
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16509
Location: M87*
I did a similar thing with fasm in Win32. But I didn't use guard pages since there was no guarantee that the memory is used in any particular order. Instead i used exception handling to allocate the pages as they were accessed. I also used a granularity of 64kB.

It made the assembler slightly slower for large monolithic projects but allowed much more flexibility with running multiple instances at once, so overall I could get more throughput when creating multiple outputs.
Post 15 Jan 2019, 02:33
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2732
Location: dank orb
The time difference seem massive to me. Maybe, because the memory is getting cleared?

_________________
unlicense.org


Last edited by bitRAKE on 15 Jan 2019, 02:53; edited 2 times in total
Post 15 Jan 2019, 02:44
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16509
Location: M87*
bitRAKE wrote:
The time difference seem massive to me. Maybe, because the memory is getting cleared?
Your test is not for the whole application, just the memory. So the results won't equate to what would be seen in the whole application.
Post 15 Jan 2019, 02:48
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2732
Location: dank orb
Yeah, the time would be amortized over the usage.

Some of my laziest algorithms use open addressing, but in those cases I just commit the full block at once. Same if I use a freelist. Don't really have a use case for random access, atm.

There must be a way to call VirtualAlloc that doesn't clear the memory. How else would HeapAlloc be able to do so?
Post 15 Jan 2019, 02:55
View user's profile Send private message Visit poster's website Reply with quote
comrade



Joined: 16 Jun 2003
Posts: 1128
Location: Russian Federation
bitRAKE wrote:
There must be a way to call VirtualAlloc that doesn't clear the memory. How else would HeapAlloc be able to do so?


I don't believe there is. VirtualAlloc is a light wrapper around NtAllocateVirtualMemory which grabs pages from the zero-initialized list.

HeapAlloc gets pages from VirtualAlloc which are zero'ed as well. If you are seeing them initialized to a different value, it could be the effect of the debug heap which is used when the process is started under a debugger.

_________________
comrade (comrade64@live.com; http://comrade.ownz.com/)
Post 20 Jan 2019, 05:50
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 16509
Location: M87*
It is a security thing. All pages mapped into a process are cleared by the OS. That way information leakage is minimised. Since the heap can reallocate data you already put there previously it doesn't clear memory. Only when the heap needs more memory it will get a new page mapped and that will be cleared by the OS. So sometimes new heap allocations will be cleared, and other times it will have previous contents.
Post 20 Jan 2019, 06:06
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 2732
Location: dank orb
I thought as much. Which makes the Heap setting only partially useful. Might be something to spinning a thread pestering the OS for pages as an application initializes -- assuming we know a large buffer is needed. Curious how that would compare to just requesting a couple gigs in the PE header - that might be streamlined in some way? (Too, hopeful.)
Post 20 Jan 2019, 07:16
View user's profile Send private message Visit poster's website Reply with quote
comrade



Joined: 16 Jun 2003
Posts: 1128
Location: Russian Federation
Probably, the only way to get non-zeroed virtual memory is to create a file mapping backed by a pre-allocated file that's been prepped via SetEndOfFile and SetFileValidData. The latter requires a special privilege (SE_MANAGE_VOLUME_NAME) to avoid filling the file contents with zero prior to extending it. I haven't tried it yet though. Obviously, backing an anonymous heap by a named file is less than desirable. (Virtual memory is like an anonymous file mapping: it is backed by the pagefile.)
Post 21 Jan 2019, 06:09
View user's profile Send private message Visit poster's website AIM Address Yahoo Messenger MSN Messenger ICQ Number Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2019, Tomasz Grysztar.

Powered by rwasa.