flat assembler
Message board for the users of flat assembler.

Index > Compiler Internals > Why does FASM use so much memory?

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
If I use a simple file like this:

Code:
file 'example.wav'    
Where the .wav is a mere 500 KB file, I get memory usage during compiling of 500MB Confused

is this a bug or what?

_________________
Previously known as The_Grey_Beast
Post 31 Aug 2009, 22:35
View user's profile Send private message Reply with quote
windwakr



Joined: 30 Jun 2004
Posts: 827
Location: Michigan, USA
windwakr
Hmmm, I tried with a text file filled with 500KB of "applebananacarrot" and only had 15MB of memory used. What version do you use?

Code:
[BLEH] fasm test.asm
flat assembler  version 1.68  (15360 kilobytes memory)
1 passes, 510006 bytes.
    

_________________
----> * <---- My star, won HERE
Post 31 Aug 2009, 22:46
View user's profile Send private message Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
1.68 I guess, let me try download it again and see if it still manifests.

EDIT: still the same Sad

Code:
B:\>D:\Programs\Fasm\FASM.EXE test.asm
flat assembler  version 1.68  (525765 kilobytes memory)
1 passes, 577340 bytes.    


EDIT: do you have the latest version? if not, can you send me yours (upload an attachment, you can delete it later) so I can see if it's because of a new version? Smile

_________________
Previously known as The_Grey_Beast
Post 31 Aug 2009, 23:17
View user's profile Send private message Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
WTF

I tested it with an empty asm file and it still uses that amount of memory. I also tried a 2007 version (1.67.something) from the webarchive, still the same result.

Does FASM preallocate memory depending on your RAM/free memory available?

Considering I have plenty of RAM available that's probably why, but otherwise I'm completely clueless Confused

EDIT: I can use -m switch to make it use less, although I would have preferred if it dynamically allocated memory depending how much it needed, not how much RAM (or free memory) you have.

Anyway sorry for annoying with this thread -- but still if you can consider it Tomasz and if you can make it dynamically expand the allocated memory Smile (in Windows you do this with VirtualAlloc/VirtualReserve)

_________________
Previously known as The_Grey_Beast
Post 31 Aug 2009, 23:36
View user's profile Send private message Reply with quote
windwakr



Joined: 30 Jun 2004
Posts: 827
Location: Michigan, USA
windwakr
I downloaded the latest version for my test, I have no clue why yours uses so much memory.
Post 31 Aug 2009, 23:44
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7737
Location: Kraków, Poland
Tomasz Grysztar
Borsuc wrote:
Anyway sorry for annoying with this thread -- but still if you can consider it Tomasz and if you can make it dynamically expand the allocated memory Smile (in Windows you do this with VirtualAlloc/VirtualReserve)

Not possible with fasm 1.x architecture. Check out my presentation on fasm 2.
Post 31 Aug 2009, 23:45
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
Alright no problem, at least I know how to work around it (using -m) and know what causes it. This is fine now, at least I know it's not a bug.

I'll check the presentation also.
Cheers. Smile
Post 31 Aug 2009, 23:49
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Borsuc wrote:

Code:
B:\>D:\Programs\Fasm\FASM.EXE test.asm
flat assembler  version 1.68  (525765 kilobytes memory)
1 passes, 577340 bytes.    



Are you sure that's the amount used or just available? I highly doubt it actually uses that much.
Post 01 Sep 2009, 03:24
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17349
Location: In your JS exploiting you and your system
revolution
rugxulo wrote:
Are you sure that's the amount used or just available? I highly doubt it actually uses that much.
fasm will preallocate all its memory before even loading the .asm file. Last time I looked the algo was to query for total memory available and allocate 3/4. If that failed then keep reducing to 3/4 and try again until a successful allocation. So the allocator is ignorant of how much memory is needed to assemble the source.
Post 01 Sep 2009, 03:34
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
Quote:
Not possible with fasm 1.x architecture

Are you sure? If I understand fasm1 architectury correctly, you get one huge buffer, and then "allocate" two arrays of structures in that buffer, one from beginning on and other from end backwards.

If that is correct, with little modification in sources, it should possible to only VirtualAlloc(MEM_RESERVE) huge block of memory and then VirtualAlloc(MEM_COMMIT) memory pages from its begging and end when actually needed.
Post 01 Sep 2009, 11:41
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17349
Location: In your JS exploiting you and your system
revolution
vid wrote:
If that is correct, with little modification in sources, it should possible to only VirtualAlloc(MEM_RESERVE) huge block of memory and then VirtualAlloc(MEM_COMMIT) memory pages from its begging and end when actually needed.
But fasm is multi-OS compatible. VirtualAlloc is not available in all OSes.
Post 01 Sep 2009, 12:11
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
Quote:
But fasm is multi-OS compatible. VirtualAlloc is not available in all OSes.

Just add one tiny OS-dependent layer that does commit on win, and nothing on others, and call this on every allocation.
Post 01 Sep 2009, 12:50
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17349
Location: In your JS exploiting you and your system
revolution
vid wrote:
Just add one tiny OS-dependent layer that does commit on win, and nothing on others, and call this on every allocation.
But there is only one allocation stage, and that is at start-up.

But I have to wonder if there really is something that is broken that actually needs fixing? What is the reason to make major alterations to the internal structure of fasm? The code is not exactly friendly to change and I wouldn't like to have a number of new introduced bugs just because Borsuc saw a 500MB allocation momentarily until he discovered the -m switch.
Post 01 Sep 2009, 12:56
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7737
Location: Kraków, Poland
Tomasz Grysztar
Definitely it's better to start writing from scratch than to try fixing this, as it would require changes all over the entire code. And the gain still would be doubtful - I never encountered any problems because of that allocation on Windows. Memory likes to be used. Wink
Post 01 Sep 2009, 13:04
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
Quote:
But there is only one allocation stage, and that is at start-up.

I was talking about custom allocation, eg. about allocation of new element in one those two arrays.

Quote:
efinitely it's better to start writing from scratch than to try fixing this, as it would require changes all over the entire code

At how many places in code you move pointer to next structure? Can't be that many. But I understand your position in not "fixing" this.
Post 01 Sep 2009, 14:08
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7737
Location: Kraków, Poland
Tomasz Grysztar
vid wrote:
At how many places in code you move pointer to next structure? Can't be that many.

Many of the instruction generators (lots of them), the expression calculator (in a few places), display allocator, nested structure allocator, relocation storage, debug line numbers storage, generators of format-specific data (like PE resources or fixups, also quite a few of them), and this is all just in the assembler module - the preprocessor and parser have a lots of their own allocations, too.

Anyway, maybe it could be possible to just reserve the address space in Windows and catch the page fault exception in order to allocate new pages in the middle when needed? But I still doubt it would give any advantage, it would probably only decrease the performance and not help with anything.
Post 01 Sep 2009, 14:15
View user's profile Send private message Visit poster's website Reply with quote
neville



Joined: 13 Jul 2008
Posts: 507
Location: New Zealand
neville
When I adapted fasm (ver 1.48 ) as the resident assembler for my FAMOS IDE it quickly became obvious that it would be quite difficult to radically alter fasm's memory allocations to fit in with the ideal requirements of my memory operating system.

As stated previously, fasm takes memory based on what is available, not on what it actually needs. As also stated previously it then uses memory top-down as well as bottom-up. From my memory, the original source buffer extends down from the top of available memory and the symbol table is written below that. The pre-processor output is written from the bottom of available memory, and the binary output is written above that (pointers memory_start and code_start respectively).

My solution to this problem for the IDE version of FAMOS was to allocate a larger but "finite" (currently 16MB) system memory buffer in which fasm is constrained to place all its output, and allocate specific buffers for fasm within the system buffer.

So currently the pre-processed source and the generated binary code are limited to 4MB each which has been adequate for my purposes so far. I posted the development code here: http://board.flatassembler.net/topic.php?t=9197 And my working IDE in FAMOS is discussed and available here: http://board.flatassembler.net/topic.php?t=9324
Of course every module required changes but system.inc was changed a lot and format.inc was eliminated entirely.

_________________
FAMOS - the first memory operating system
Post 02 Sep 2009, 01:02
View user's profile Send private message Visit poster's website Reply with quote
windwakr



Joined: 30 Jun 2004
Posts: 827
Location: Michigan, USA
windwakr
If FASM takes memory based on what's available, then why did it only take 15MB on computer? I had at least 1.5GB free memory at the time.

_________________
----> * <---- My star, won HERE
Post 02 Sep 2009, 01:21
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7737
Location: Kraków, Poland
Tomasz Grysztar
Only fasm for DOS and for Win32 console use the amount based on what is available by default. The Linux/libc version and Win32 IDE both use 16 MB by default (you can of course adjust it with settings).
Post 02 Sep 2009, 04:58
View user's profile Send private message Visit poster's website Reply with quote
windwakr



Joined: 30 Jun 2004
Posts: 827
Location: Michigan, USA
windwakr
For my test I used a brand new download of "fasm.exe" out of "flat assembler 1.68 for DOS". Wouldn't that fall under the category of "fasm for DOS and for Win32 console"? So, why did it only allocate 15MB?
Post 02 Sep 2009, 05:14
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.