flat assembler
Message board for the users of flat assembler.

Index > Windows > Memory allocation problem

Author
Thread Post new topic Reply to topic
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
I'm trying to allocate a large amount of memory for an app but keep failing

Code:
invoke  GlobalAlloc, GPTR, 0x7FFFFFFF   ; 2GB - 1 byte
    

fails with err code 8 (not enough mem)

My machine has 3GB ram and the pagefile is set to 10GB min

If I use
Code:
mov     [memInf + MEMORYSTATUS.dwiLength], sizeof.MEMORYSTATUS
invoke  GlobalMemoryStatus, memInf
cinvoke printf, memSzStr, [memInf + MEMORYSTATUS.dwAvailPhys], [memInf + MEMORYSTATUS.dwAvailPageFile], [memInf + MEMORYSTATUS.dwAvailVirtual]
    


I get
Code:
availPhys: 1743282176
availPageFile: 4294967295
availVirtual: 2068615168
    

as the result.

with availPageFile showing 4GB - 1byte I can't understand the problem.
It is the same with all other allocation calls, i.e. malloc, VirtualAlloc etc

Any idea if I'm doing something wrong?

Thanks
Post 06 Aug 2012, 01:17
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17715
Location: In your JS exploiting you and your system
revolution
32-bit Windows loads the system DLLs at the high end of the lower 2GB addresses (i.e. <0x80000000) and reserves the addresses above 0x80000000 for the kernel. Even if you boot in 3GB mode you still won't find a contiguous 2GB range of available address space. You have two options:
  1. Use 64-bit Windows, or
  2. Boot in 3GB mode and break your allocations into smaller sections
Post 06 Aug 2012, 01:24
View user's profile Send private message Visit poster's website Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
Thx revolution for rapid reply Wink

I'm running Win7 64-bit so what do I need to do\use ?

(haven't done any programming for 64-bit)
Post 06 Aug 2012, 01:28
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17715
Location: In your JS exploiting you and your system
revolution
Also the pagefile cannot act as a memory substitute, only as a backing store for swapped process memory. So even though you are using 64-bit you can't pretend that you have 10GB physical memory by increasing the size of the pagefile, you will still have a maximum per-process memory of 3GB less the memory used by the kernel and drivers.
Post 06 Aug 2012, 01:33
View user's profile Send private message Visit poster's website Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
OK, thanks for clearing up my misunderstanding of the pagefile.
I'll have to do some research on x64 programming then.
I currently only have a copy of the old win32.hlp file. Is there a 64-bit version?
Post 06 Aug 2012, 02:30
View user's profile Send private message Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
baldr
magicSqr,

Those who program for Windows witout MSDN should be banned from existence. Wink
Post 06 Aug 2012, 15:22
View user's profile Send private message Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
lol

What can I say, still using win32.hlp, w32dasm, etc.

Just an old-fashioned kind o' guy. Looks like x64 is gonna drag me kicking and screaming into the 18th century :O)

btw, had a look at winDbg(x64) (@_@), shocking, why didn't someone make w64dasm, ggrrrrrr
Post 06 Aug 2012, 17:28
View user's profile Send private message Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
Okley Dokley,

found out about rax, rbx etc. 16 registers, loving it Smile

I can print unsigned 64-bit integers with printf using %llu

Could someone point me to a 64-bit version of the win32 API ref and the same for C (C++) functions

Many thanks
Post 06 Aug 2012, 20:26
View user's profile Send private message Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1419
Location: Toronto, Canada
AsmGuru62
What is shocking about WinDbg64?
Post 07 Aug 2012, 14:17
View user's profile Send private message Send e-mail Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
AsmGuru62 wrote:
What is shocking about WinDbg64?


Nothing really, just a case of old dogs and new tricks Wink
Post 07 Aug 2012, 14:23
View user's profile Send private message Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
baldr
magicSqr wrote:
Could someone point me to a 64-bit version of the win32 API ref and the same for C (C++) functions
Pal, MSDN has it all. Unless you're traffic-wise, Microsoft could be useful (even if you are, it pays off for every bit transmitted).
Post 09 Aug 2012, 03:25
View user's profile Send private message Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
Thanks baldr, I'm trawling my way through it now
Post 09 Aug 2012, 22:23
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Remember to flag the application as "large address space aware" - that'll give you quite a bit of address space, even for a 32bit application, if you're running under 64bit Windows.

You might want to check Mark Russinovich's "Pushing the Limits of Windows" blog series, especially the part on virtual memory.

Also, stay away from Global/LocalAlloc unless you have to use them (clipboard API, AllocateStreamOnHGlobal). HeapAlloc is better for general allocations, and VirtualAlloc is your friend if you deal with huge blocks.
Post 10 Aug 2012, 11:24
View user's profile Send private message Visit poster's website Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
Thanks f0dder, i'll take a look.

Now I'm onto 64-bit I can allocate 2 chunks of 2GB and a chunk ~ 0.75GB which will keep me going.

If I don't find the info I seek within this space, the files I'll be checking get larger and larger so eventually I'd end up in the position of having to use a temporary file anyway to save the 2GB and read more.

As for the local vs global vs virtual, anything I've ever tried never rendered any extra mem, so I normally use global for ease of typing Smile
Post 10 Aug 2012, 22:07
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Are you allocating these large amounts of memory to read full file contents into, then do in-memory searches? If so, you might want to play around a bit with memory-mapped files instead. While you lose some control of exactly how data is read, it can be quite advantageous in some situations - working on (read-only) data directly out of the filesystem cache buffers without copying to your own buffers can be pretty nice in some situations. And especially if your application is 64bit things get easy because you can be lazy and map huge files directly, instead of mapping/unmapping views of the file.

It's not the end-all-be-all of file access that some people seem to believe, but it can definitely be nice for some access patterns Smile
Post 10 Aug 2012, 22:37
View user's profile Send private message Visit poster's website Reply with quote
magicSqr



Joined: 27 Aug 2011
Posts: 105
magicSqr
Many thanks f0dder, file-mapping was exactly what I needed Wink
Post 12 Aug 2012, 10:47
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.