flat assembler
Message board for the users of flat assembler.

Index > Compiler Internals > fasm to use self indulged virtual memory?

Goto page Previous  1, 2
Author
Thread Post new topic Reply to topic
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
revolution wrote:
But I have to ask this: Does anyone here actually have source code that needs more than 2GB of memory to compile? Or is this just an academic discussion? And I am not talking about some trivial source file of repeated nonsense made just to test the system.
"f0dder likes this". If there's no compelling real-world usecase to implement a feature, don't do it. Especially not if it's likely to be complicated, and possibly slowing down the assembler. And from the bits and pieces Tomasz have written about fasm's internal workings, it does sound like it would be a major undertaking to do such a thing. Much better idea to focus effort on other architectural enhancements, IMHO.


baldr wrote:
Simple source with include "Win32AX.Inc" can grow quite fast with each macro used (EXAMPLES\HELLO.ASM is almost 3.5 MiB preprocessed, +1.7k for each extra invoke, +7k for each .if/.while/.repeat).
Ouch, that's a fair amount.

If memory consumption is a real-world problem, I'd rather suggest focusing on decreasing it in other ways, though. Like perhaps moving parts of invoke and .directive handling from macro implementation to assembly... I don't know enough about fasm's internals to know whether that's feasible or how much it would save, but I assume it could shave down on RAM usage and compilation time at the cost of reduced flexibility.

_________________
Image - carpe noctem
Post 22 Oct 2010, 17:34
View user's profile Send private message Visit poster's website Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
baldr
f0dder,

Features come at a cost, simplicity of fasm macro engine brings a heavy toll on memory consumption.

----8<----
Tomasz Gryzstar,

This isn't a critique, I understand (probably) your considerations regarding design of so powerful preprocessor. Is there a whitepaper about second generation of fasm?
Post 22 Oct 2010, 17:47
View user's profile Send private message Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
baldr: Have you seen FASMCON 2009 talk about FASM2?
Post 22 Oct 2010, 23:26
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3045
Location: vpcmipstrm
bitRAKE
I thought the memory problem was solved with OBJ support? Getting familiar with FASM, I initially had the same concern/confusion about its operation. Not just the macros, but simple things like RB taking more memory seems bothersome.

Only one of my projects really have a problem with FASM's memory limit, and a work-around wasn't that difficult. It's common for me to use FASM for data processing - loading a huge data file directly into EXE - the lazy way. Combine that with a large hash table - the lazy way. Throw on some 256-aligned memory structures - the lazy way. Memory goes quick.

(Someone say a x86-64 version of FASM? Very Happy Can I help?)
Post 23 Oct 2010, 01:37
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
vid wrote:
Quote:
Ok, so what's wrong with using file directly (without caching/mapping it into memory)?
There is no reason to do it.


+1 to original suggestion: add optional temp file support to FASM. Why ?

1. This pagefile.sys is hacky and slow as hell (and there is none at all in UNREAL mode)

2. With temp file you have almost 4 GiB for source and another almost 4 GiB for preprocessed stuff. With pagefile.sys you have 4 GiB for both together and additionally all the other crap (AFAIK "only" cca 1 GiB in XP).

> I thought the memory problem was solved with OBJ support?

Shocked
Post 23 Oct 2010, 09:12
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3045
Location: vpcmipstrm
bitRAKE
DOS386 wrote:
> I thought the memory problem was solved with OBJ support?

Shocked
I.E. Work with smaller fragments of program, and let the linker join the parts. Especially when working with deeply complex macros, the totality of available memory can be used for each sub-routine, etc. Or when working in a limited environment (virtual/embedded machine, for example) the project creation is not limited by available memory.

If my project requires 4GB to build then I have limited audience and development choices myself. The downside being slower build times - which are hopefully mitigated by the use of NMAKE or similar.

_________________
¯\(°_o)/¯ unlicense.org
Post 24 Oct 2010, 02:16
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7797
Location: Kraków, Poland
Tomasz Grysztar
DOS386 wrote:
1. This pagefile.sys is hacky and slow as hell
Which swapping implementation are you referring to?

DOS386 wrote:
(and there is none at all in UNREAL mode)

I agree, but making unREAL version use some swapping mechanism would in fact beat the point of using unREAL in the first place (which was that it should work directly on physical memory and be faster than the other variants). So to get more memory it is still simpler and better to use DPMI version with some decent virtual memory implementation.

DOS386 wrote:
2. With temp file you have almost 4 GiB for source and another almost 4 GiB for preprocessed stuff.
You would not really be able to have 4G preprocessed source without some more tweaks in fasm's internals - since fasm in some areas would still need to be able to distinguish pointer into preprocessed source from the pointer into other data. And with preprocessed source stored in file you'd need to rely on OS to cache it well in order to get an acceptable speed - fasm's random accesses to preprocessed source would result in terrible performance if constant disk accesses were necessary.
Post 24 Oct 2010, 14:08
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7797
Location: Kraków, Poland
Tomasz Grysztar
bitRAKE wrote:
Someone say a x86-64 version of FASM? Very Happy Can I help?
Help me decide, whether I should begin with rewriting fasm 1 architecture into x86-64, or with fasm 2 as described on fasmcon 2009. Wink
Post 24 Oct 2010, 14:11
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17667
Location: In your JS exploiting you and your system
revolution
I vote fasm 2. All this talk about 4GB sources is pie-in-the-sky stuff. No one really has sources that need so much memory.
Post 24 Oct 2010, 14:48
View user's profile Send private message Visit poster's website Reply with quote
baldr



Joined: 19 Mar 2008
Posts: 1651
baldr
revolution,

fasm 1 is, by design, memory-hungry. At least when some advanced macros are used. I really should see that FASMCON 2009 presentation. Wink
Post 24 Oct 2010, 14:58
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17667
Location: In your JS exploiting you and your system
revolution
Well I still have a copy, M2U00510.avi, 119124kB, 12m24s.

Although I have no idea how to send to you. Question
Post 24 Oct 2010, 15:06
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
try to use yousendit.com, or something similar, if you have decent enough upload.
Post 24 Oct 2010, 15:33
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Tomasz Grysztar wrote:
bitRAKE wrote:
Someone say a x86-64 version of FASM? Very Happy Can I help?
Help me decide, whether I should begin with rewriting fasm 1 architecture into x86-64, or with fasm 2 as described on fasmcon 2009. Wink
Definitely fasm2!

_________________
Image - carpe noctem
Post 24 Oct 2010, 15:52
View user's profile Send private message Visit poster's website Reply with quote
vid
Verbosity in development


Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia
vid
And write it in some portable language! ... Oh well...
Post 24 Oct 2010, 16:25
View user's profile Send private message Visit poster's website AIM Address MSN Messenger ICQ Number Reply with quote
Alphonso



Joined: 16 Jan 2007
Posts: 294
Alphonso
Ummm.. FASMW with AWE Very Happy
Post 25 Oct 2010, 05:13
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3045
Location: vpcmipstrm
bitRAKE
baldr wrote:
I really should see that FASMCON 2009 presentation.
I'd also like to see this if someone posts it somewhere.

_________________
¯\(°_o)/¯ unlicense.org
Post 25 Oct 2010, 10:31
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.