Hi there. I have some macros which parse data located in external files. Thus these macros intensively use the file directive. Depending on the size and structure of the external files these macros may make the compilation time indefinitely long (up to five minutes in some practice relevant situations).

So my suggestion is to improve fasm's way to process the file directive. Every time this directive is processed, fasm opens a file, sets a file pointer, reads data and closes the file. Just a simple maintaining of a (hash) table of all opened file handles without closing them (and opening later again) before the compilation ends would drastically improve the performance. Almost 20 times in my tests. There're probably many optimization possibilities of the file directive more obvious to the author.

Best regards.

Well sure there are many memory/file optimisations that could be done. But remember that while fasm can be used as a general purpose data parser that is not its primary purpose.

Seeing as how you have some special parsing requirements for large files then, in the meantime, while waiting for any upgrade, perhaps it would be easiest to write (in assembly of course) a small app to parse and generate the needed data into a single file that you then feed into fasm along with the rest of code.

revolution
Your recommendation to use an external parser is sometimes very hard or even impossible to use, because the macros I am talking about correlate with the code of the compiled executable. I probably need to give an example of such macro to be more clear.

Let's say, I need to compile a proxy dll intercepting calls to a few functions from some other dll. A while ago I've written a macro, which allows to do it easily (attached to the post). And let's say the target is QtGuid4.dll of 160MB size. The following code will intercept some desired functions:


It compiles within 8.5 seconds with a modified version of fasm. Now tell me please, how long does it take for you to compile this code with the official fasm version.

Are you still going to insist this is not the use case of the fasm's primary purpose? Then please show me an external parser which allows to achieve the same.

Yes. You are doing it in such a way as to create havoc.

Here is a possible alternative which I expect will compile in milliseconds.
1) generate a file with all the functions from QtGuid4.DLL. Just a simple text list with a small header will suffice.
2) write a custom export macro that places all the exports conditionally based upon "used".
3) include your file.

And in the 'exports_from_QtGuid4.dll.txt' file you might have this:

revolution

I didn't get the metaphor, but that's actually not as simple as you described it. E.g. you removed the original dll name from the macro parameters, but it's actually necessary to build the forwarded export. Second, an ordinal base of the original dll must also be included into the file generated by an external parser. Third, the order of the function names must be preserved by an external parser according to the AddressOfFunctions table including also functions (and just table holes), that are exported only by ordinal (and it's important to generate some collisionless labels for these functions). Thus the names will have to be extra sorted by the macro. Clearly your suggestion can be extended to a working solution, but not as simple as you say and with one (following) "but".

As you said, that I could "feed into fasm along with the rest of code" some preparsed data, I assumed you were talking about the file directive. In your alternative solution you're just replacing this directive with include. But this is actually always possible (back to the stupid masm and bin2db?). If you are familiar with the fasm's "main purpose", what is then the file directive's purpose? I assume the main point was to get rid of some external tools and make fasm to a single powerful and comfortable tool. At least I always loved the fasm's ability to be a standalone. What you suggest now is to operate on tonsils via anus (which is necessary in masm) even though a better way is provided specifically for this case.

So if the author wanted to make fasm better by means of providing the file directive, my suggestion is to do it good and moreover in a very simple way without any damage to the architecture or backward compatibility.

I did not realize that open/close cycle gave such a bad overhead on Windows systems. In DOS the main problem with "file" was when reading multiple small chunks (just like you do it here), because combined with multiple assembly passes it generated a lot of separate HDD accesses and this was causing a long and very noisy assemblies - but loading some disk cache software like SMARTDRV was helping a lot.
And is the slow open/close time somehow related to DLL being very big in this case? As you don't load the whole file into memory and just do small reads from export table, this size should not have such an effect unless it is causing slow open for some cause.

Anyway, the right way to fix it would be to modify Win32 interfaces to provide file handle caching, like you suggested.

Tomasz Grysztar

Well. Only indirectly. Large dll size <= large export table => many file accesses => low performance. The timing of a single file access does not depend on the number of bytes read or on a file size.

In this macro I tried to minimize the number of processing cycles of the file directive as much as I could. E.g. having this improvement it would be possible to load the whole file (or just the export directory) reducing the number of file loads to a single or a few.

Thank you for your support. I thought you will disregard this topic, because it's just a performance issue rather than a bug.