fasm to use self indulged virtual memory?

Index > Compiler Internals > fasm to use self indulged virtual memory?

Goto page Previous 1, 2

Author

Thread

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 22 Oct 2010, 17:34

revolution wrote:

But I have to ask this: Does anyone here actually have source code that needs more than 2GB of memory to compile? Or is this just an academic discussion? And I am not talking about some trivial source file of repeated nonsense made just to test the system.

"f0dder likes this". If there's no compelling real-world usecase to implement a feature, don't do it. Especially not if it's likely to be complicated, and possibly slowing down the assembler. And from the bits and pieces Tomasz have written about fasm's internal workings, it does sound like it would be a major undertaking to do such a thing. Much better idea to focus effort on other architectural enhancements, IMHO.

baldr wrote:

Simple source with include "Win32AX.Inc" can grow quite fast with each macro used (EXAMPLES\HELLO.ASM is almost 3.5 MiB preprocessed, +1.7k for each extra invoke, +7k for each .if/.while/.repeat).

Ouch, that's a fair amount.

If memory consumption is a real-world problem, I'd rather suggest focusing on decreasing it in other ways, though. Like perhaps moving parts of invoke and .directive handling from macro implementation to assembly... I don't know enough about fasm's internals to know whether that's feasible or how much it would save, but I assume it could shave down on RAM usage and compilation time at the cost of reduced flexibility.

_________________

- carpe noctem

22 Oct 2010, 17:34

baldr

Joined: 19 Mar 2008
Posts: 1651

baldr 22 Oct 2010, 17:47

f0dder,

Features come at a cost, simplicity of fasm macro engine brings a heavy toll on memory consumption.

----8<----
Tomasz Gryzstar,

This isn't a critique, I understand (probably) your considerations regarding design of so powerful preprocessor. Is there a whitepaper about second generation of fasm?

22 Oct 2010, 17:47

vid
Verbosity in development

Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia

vid 22 Oct 2010, 23:26

baldr: Have you seen FASMCON 2009 talk about FASM2?

22 Oct 2010, 23:26

bitRAKE

Joined: 21 Jul 2003
Posts: 4020
Location: vpcmpistri

bitRAKE 23 Oct 2010, 01:37

I thought the memory problem was solved with OBJ support? Getting familiar with FASM, I initially had the same concern/confusion about its operation. Not just the macros, but simple things like RB taking more memory seems bothersome.

Only one of my projects really have a problem with FASM's memory limit, and a work-around wasn't that difficult. It's common for me to use FASM for data processing - loading a huge data file directly into EXE - the lazy way. Combine that with a large hash table - the lazy way. Throw on some 256-aligned memory structures - the lazy way. Memory goes quick.

(Someone say a x86-64 version of FASM? Very Happy

Can I help?)

23 Oct 2010, 01:37

DOS386

Joined: 08 Dec 2006
Posts: 1900

DOS386 23 Oct 2010, 09:12

vid wrote:

Quote:
Ok, so what's wrong with using file directly (without caching/mapping it into memory)?
There is no reason to do it.

+1 to original suggestion: add optional temp file support to FASM. Why ?

1. This pagefile.sys is hacky and slow as hell (and there is none at all in UNREAL mode)

2. With temp file you have almost 4 GiB for source and another almost 4 GiB for preprocessed stuff. With pagefile.sys you have 4 GiB for both together and additionally all the other crap (AFAIK "only" cca 1 GiB in XP).

> I thought the memory problem was solved with OBJ support?

Shocked

23 Oct 2010, 09:12

bitRAKE

Joined: 21 Jul 2003
Posts: 4020
Location: vpcmpistri

bitRAKE 24 Oct 2010, 02:16

DOS386 wrote:

> I thought the memory problem was solved with OBJ support?

I.E. Work with smaller fragments of program, and let the linker join the parts. Especially when working with deeply complex macros, the totality of available memory can be used for each sub-routine, etc. Or when working in a limited environment (virtual/embedded machine, for example) the project creation is not limited by available memory.

If my project requires 4GB to build then I have limited audience and development choices myself. The downside being slower build times - which are hopefully mitigated by the use of NMAKE or similar.

_________________
¯\(°_o)/¯ “languages are not safe - uses can be” Bjarne Stroustrup

24 Oct 2010, 02:16

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 24 Oct 2010, 14:08

DOS386 wrote:

1. This pagefile.sys is hacky and slow as hell

Which swapping implementation are you referring to?

DOS386 wrote:

(and there is none at all in UNREAL mode)

I agree, but making unREAL version use some swapping mechanism would in fact beat the point of using unREAL in the first place (which was that it should work directly on physical memory and be faster than the other variants). So to get more memory it is still simpler and better to use DPMI version with some decent virtual memory implementation.

DOS386 wrote:

2. With temp file you have almost 4 GiB for source and another almost 4 GiB for preprocessed stuff.

You would not really be able to have 4G preprocessed source without some more tweaks in fasm's internals - since fasm in some areas would still need to be able to distinguish pointer into preprocessed source from the pointer into other data. And with preprocessed source stored in file you'd need to rely on OS to cache it well in order to get an acceptable speed - fasm's random accesses to preprocessed source would result in terrible performance if constant disk accesses were necessary.

24 Oct 2010, 14:08

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8351
Location: Kraków, Poland

Tomasz Grysztar 24 Oct 2010, 14:11

bitRAKE wrote:

Someone say a x86-64 version of FASM? Can I help?

Help me decide, whether I should begin with rewriting fasm 1 architecture into x86-64, or with fasm 2 as described on fasmcon 2009. Wink

24 Oct 2010, 14:11

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20300
Location: In your JS exploiting you and your system

revolution 24 Oct 2010, 14:48

I vote fasm 2. All this talk about 4GB sources is pie-in-the-sky stuff. No one really has sources that need so much memory.

24 Oct 2010, 14:48

baldr

Joined: 19 Mar 2008
Posts: 1651

baldr 24 Oct 2010, 14:58

revolution,

fasm 1 is, by design, memory-hungry. At least when some advanced macros are used. I really should see that FASMCON 2009 presentation. Wink

24 Oct 2010, 14:58

revolution
When all else fails, read the source

Joined: 24 Aug 2004
Posts: 20300
Location: In your JS exploiting you and your system

revolution 24 Oct 2010, 15:06

Well I still have a copy, M2U00510.avi, 119124kB, 12m24s.

Although I have no idea how to send to you. Question

24 Oct 2010, 15:06

vid
Verbosity in development

Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia

vid 24 Oct 2010, 15:33

try to use yousendit.com, or something similar, if you have decent enough upload.

24 Oct 2010, 15:33

f0dder

Joined: 19 Feb 2004
Posts: 3175
Location: Denmark

f0dder 24 Oct 2010, 15:52

Tomasz Grysztar wrote:

bitRAKE wrote:
Someone say a x86-64 version of FASM? Can I help?
Help me decide, whether I should begin with rewriting fasm 1 architecture into x86-64, or with fasm 2 as described on fasmcon 2009.

Definitely fasm2!

_________________

- carpe noctem

24 Oct 2010, 15:52

vid
Verbosity in development

Joined: 05 Sep 2003
Posts: 7105
Location: Slovakia

vid 24 Oct 2010, 16:25

And write it in some portable language! ... Oh well...

24 Oct 2010, 16:25

Alphonso

Joined: 16 Jan 2007
Posts: 295

Alphonso 25 Oct 2010, 05:13

Ummm.. FASMW with AWE Very Happy

25 Oct 2010, 05:13

bitRAKE

Joined: 21 Jul 2003
Posts: 4020
Location: vpcmpistri

bitRAKE 25 Oct 2010, 10:31

baldr wrote:

I really should see that FASMCON 2009 presentation.

I'd also like to see this if someone posts it somewhere.

_________________
¯\(°_o)/¯ “languages are not safe - uses can be” Bjarne Stroustrup

25 Oct 2010, 10:31

Goto page Previous 1, 2

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum