Index
> Compiler Internals > [sug] targeted allocation of memory in Windows fasmGoto page 1, 2, 3 Next |
| Author |
|
|
revolution 11 Jan 2015, 18:12
By using exception handling it is possible to make fasm use only the memory that it needs when assembling.
My motivation for this is fourfold:
Please see further ahead in the thread for updated code with enhancements and bug fixes. In FASM.ASM: We start by making fasm large address aware. This allows us to use up to approximately 2.75GB. Although the code shown here is only able to use 2GB. This can be raised if desired to the maximum of 2.75GB with some changes to the memory reservation code. For now we keep the limit to just under 2GB. Code: - format PE console + format PE console large Code: - GlobalMemoryStatus dd rva _GlobalMemoryStatus + SetUnhandledExceptionFilter dd rva _SetUnhandledExceptionFilter ;... - _GlobalMemoryStatus dw 0 - db 'GlobalMemoryStatus',0 + _SetUnhandledExceptionFilter dw 0 + db 'SetUnhandledExceptionFilter',0 Define some constants for ease of readability. Code: +MEMORY_ALLOCATION_BLOCK_SIZE = 1 shl 16 ;64kB. Higher numbers reduce the number of calls to the allocator. A value of 1 will allocate just the pages where the memory is accessed but will call the allocator more times +MEMORY_ALLOCATION_MAX_SIZE = 0x7ffc0000 ;Windows fails all calls that try to allocate more than this Code: - push buffer - call [GlobalMemoryStatus] - mov eax,dword [buffer+20] - mov edx,dword [buffer+12] - cmp eax,0 - jl large_memory - cmp edx,0 - jl large_memory - shr eax,2 - add eax,edx - jmp allocate_memory - large_memory: - mov eax,80000000h + mov eax,MEMORY_ALLOCATION_MAX_SIZE allocate_memory: Code: - push MEM_COMMIT + push MEM_RESERVE Code: + push memory_allocation_handler + call [SetUnhandledExceptionFilter] Code: - shl eax,1 + lea eax,[eax*3] Code: +memory_allocation_handler: + mov eax,[esp+4] ;get pointer to exception information + mov ecx,[eax] ;EXCEPTION_POINTERS.ExceptionRecord + mov edx,[ecx] ;EXCEPTION_RECORD.ExceptionCode + cmp edx,0xc0000005 ;EXCEPTION_ACCESS_VIOLATION + jnz .fail + mov edx,[ecx+24] ;EXCEPTION_RECORD.ExceptionAddress + cmp edx,[memory_start] + jb .fail + mov eax,[additional_memory_end] + sub eax,edx ;don't allocate more than EAX bytes + jbe .fail + mov ecx,MEMORY_ALLOCATION_BLOCK_SIZE + cmp eax,ecx + jbe .allocation_size_defined + mov eax,ecx ;don't allocate more than ECX bytes + .allocation_size_defined: + push PAGE_READWRITE + push MEM_COMMIT + push eax ;allocate this many bytes + push edx ;at this address + call [VirtualAlloc] + test eax,eax + jz out_of_memory + mov eax,-1 ;EXCEPTION_CONTINUE_EXECUTION + ret 4 + .fail: ;some other exception happened + mov eax,0 ;EXCEPTION_CONTINUE_SEARCH + retn 4 Code: read: + push ebx edx ecx + push PAGE_READWRITE + push MEM_COMMIT + push ecx ;allocate the required number of bytes + push edx ;at this address + call [VirtualAlloc] + pop ecx edx ebx + test eax,eax + jz file_error During my testing I found that fasm assembling itself needs only ~2.3MB of memory. On my test system fasm will reserve 2GB of RAM at startup and only commit 2.3MB during compilation. The Windows dialog box complaining about low memory availability no longer appears. And I can can now have multiple instances all compiling simultaneously without running out of RAM.
Last edited by revolution on 13 Apr 2015, 15:48; edited 1 time in total |
|||||||||||||||||||||||||||||||
11 Jan 2015, 18:12 |
|
|
JohnFound 11 Jan 2015, 19:18
Very good idea! May I use it for Fresh IDE? Also, it would be good to have it for Linux as well.
|
|||
|
11 Jan 2015, 19:18 |
|
|
JohnFound 11 Jan 2015, 20:32
After some though and source reading, Linux version uses sys_brk, that IIRC, works exactly this way - allocates pages only on access. So, no need of update.
|
|||
|
11 Jan 2015, 20:32 |
|
|
l_inc 11 Jan 2015, 20:35
revolution
Quote: fasm can't allocate more than 1GB of RAM if you have 2GB or more RAM available That is just not true. I always have more and I always let it preallocate 1.5 GB. The rest is used for libraries. Quote: Fix a problem with Windows showing an annoying dialog box asking to close fasm because of excessive memory use I can't remember I've ever seen that dialog box. Quote: Change the allocation algorithm to reserve (i.e. not commit) all of the address space. Because later we want to commit pages on demand. In fact, Windows does that for you. Even if you allocate memory with MEM_COMMIT the kernel does nothing but creating the VADs. The page tables remain untouched until you actually try to read or write the allocated memory. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 20:35 |
|
|
revolution 11 Jan 2015, 20:40
l_inc wrote: revolution Plus the commit will starve others apps of memory, Windows will not allow the committed RAM to be used by other apps else where is the guarantee of having committed RAM? |
|||
|
11 Jan 2015, 20:40 |
|
|
revolution 11 Jan 2015, 20:45
When the current code sees more than 2GB it tries to allocate 0x80000000 which will fail due to the 0x7ffc0000 limit, then it halves the amount to 0x40000000 and succeeds in getting 1GB. Always for me, everytime I see "(1048576 kilobytes memory)".
|
|||
|
11 Jan 2015, 20:45 |
|
|
l_inc 11 Jan 2015, 20:49
revolution
Quote: else where is the guarantee of having committed RAM? The guarantee is called "pagefile". If you run out of real RAM by touching too many of the committed pages, the working set of other applications will be reduced, and then you'll run out of your own working set limit and your pages will be swapped out. You can do your own tests and look at the working set performance counter in the process explorer before you allocate memory, after that, but before touching it and after touching the pages. Your working set will increase only after touching the pages. Besides you can read the "Windows Internals" and the WRK. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 20:49 |
|
|
revolution 11 Jan 2015, 20:53
I seem to not see what you see.
Anyhow, I still feel that committing so much RAM is not the best use of the RAM or the pagefile. Commit on demand reduces the load on both. |
|||
|
11 Jan 2015, 20:53 |
|
|
Tomasz Grysztar 11 Jan 2015, 20:55
revolution wrote: [*]Fix an existing bug with the allocation algorithm. Currently, and somewhat paradoxically, fasm can use more memory only when you have less memory available. fasm can't allocate more than 1GB of RAM if you have 2GB or more RAM available. Only when you drop below about 1.5GB will fasm be able to allocate all of it. In case of Linux version I did not implement any eager heuristic, it just has a hard limit of 16M by default, and to allocate more you always need to use "-m" parameter. Turns out it was not that bad decision, as I never got any complaints. 
revolution wrote: We start by making fasm large address aware. Anyway, this is a very interesting modification of fasm, and I think something like this could be suitable for fasmw. |
|||
|
11 Jan 2015, 20:55 |
|
|
revolution 11 Jan 2015, 21:01
Tomasz Grysztar wrote: By the way: the reason why I never marked the console version with "large" keyword (though I did it with fasmw) is that I mantain this version using a very basic set of API and PE features, in order to keep it compatible with environments like Win32s WDOSX/HX, etc. That's why this interface is stuck almost unchanged since early 2000's. |
|||
|
11 Jan 2015, 21:01 |
|
|
Tomasz Grysztar 11 Jan 2015, 21:05
revolution wrote:
 Last edited by Tomasz Grysztar on 11 Jan 2015, 21:06; edited 1 time in total |
|||
|
11 Jan 2015, 21:05 |
|
|
l_inc 11 Jan 2015, 21:06
revolution
Quote: Commit on demand reduces the load on both It does not. The pages are not swapped out unless they've ever been touched, because they just don't exist. Quote: it tries to allocate 0x80000000 which will fail due to the 0x7ffc0000 limit I might have forgotten something, but I can't remember there is such limit. You are free to allocate more than 2GB if you've configured a 3:1 split. Otherwise you shouldn't even try to allocate that much, because there are libraries that take away a significant amount of the virtual address space. Quote: I always get only 1GB allocated, unless I have less than 2GB free then I can get more allocated up to 1.75GB This behaviour would be interesting to investigate, but my guess would be that the libraries occupy less sparse regions of virtual memory in some cases. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 21:06 |
|
|
revolution 11 Jan 2015, 21:08
The 0x7ffc0000 limit exists only per call. In total you can get more but it requires more than one call.
Or, at least, it does for me on Win7-64. Perhaps you see something different there also? |
|||
|
11 Jan 2015, 21:08 |
|
|
revolution 11 Jan 2015, 21:19
l_inc wrote: revolution |
|||
|
11 Jan 2015, 21:19 |
|
|
l_inc 11 Jan 2015, 22:21
revolution
Quote: The 0x7ffc0000 limit exists only per call. In total you can get more but it requires more than one call It would be a bit cumbersome for me to do the test, cause I always use the standard 2:2 split. And reconfiguring it often results in crashes of some applications. I looked up into the wrk and I didn't find such a limitation after superficial reading. I guess your limit is caused by other virtual addressing space reservations (including dlls) that do not allow a virtual address range to be contiguous. Quote: Your pagefile is not of infinite size. At some point Windows will deny commit allocations when the sum of all commits exceeds its capacity. That's totally true. But this rarely happens (don't you btw. have your pagefile disabled? This would explain your annoying low memory message). You should get into low memory conditions just at the moment of compilation, so that another application gets it's memory allocation request denied. Because compilation doesn't normally last for long it is preferable for fasm to commit all the required memory at once, because it avoids additional performance penalties this way. As long as this contributes solely to the commit charge, but not to the process working set, there isn't much of a problem here. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 22:21 |
|
|
revolution 11 Jan 2015, 22:41
l_inc wrote: I guess your limit is caused by other virtual addressing space reservations (including dlls) that do not allow a virtual address range to be contiguous. l_inc wrote: ... (don't you btw. have your pagefile disabled? This would explain your annoying low memory message). You should get into low memory conditions just at the moment of compilation, so that another application gets it's memory allocation request denied. Because compilation doesn't normally last for long it is preferable for fasm to commit all the required memory at once, because it avoids additional performance penalties this way. As long as this contributes solely to the commit charge, but not to the process working set, there isn't much of a problem here. I want to re-quote this part with a highlight: l_inc wrote: ... it is preferable for fasm to commit all the required memory at once ... |
|||
|
11 Jan 2015, 22:41 |
|
|
l_inc 11 Jan 2015, 22:49
revolution
By "required" I meant what the user requested it to allocate with the "-m" switch. 
Quote: the tradeoff is that I am not wasting my memory, I can use my memory more efficiently Again, you are not wasting your memory anyway. You waste the commit charge, which is the system's reservation for the case you actually use the commited memory. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 22:49 |
|
|
revolution 11 Jan 2015, 22:57
Committed memory is memory I can't give to another process. I think of that as a waste if I don't use all of the amount committed.
|
|||
|
11 Jan 2015, 22:57 |
|
|
l_inc 11 Jan 2015, 23:01
revolution
It's not memory. It's just a number. What you waste is a portion of a number. _________________ Faith is a superposition of knowledge and fallacy |
|||
|
11 Jan 2015, 23:01 |
|
| Goto page 1, 2, 3 Next < Last Thread | Next Thread > |
Forum Rules:
|