flat assembler
Message board for the users of flat assembler.

Index > Compiler Internals > Files access optimization. Suggestion.

Author
Thread Post new topic Reply to topic
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 01 Sep 2011, 14:29
Hi there. I have some macros which parse data located in external files. Thus these macros intensively use the file directive. Depending on the size and structure of the external files these macros may make the compilation time indefinitely long (up to five minutes in some practice relevant situations).

So my suggestion is to improve fasm's way to process the file directive. Every time this directive is processed, fasm opens a file, sets a file pointer, reads data and closes the file. Just a simple maintaining of a (hash) table of all opened file handles without closing them (and opening later again) before the compilation ends would drastically improve the performance. Almost 20 times in my tests. There're probably many optimization possibilities of the file directive more obvious to the author.

Best regards.
Post 01 Sep 2011, 14:29
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20421
Location: In your JS exploiting you and your system
revolution 01 Sep 2011, 14:46
Well sure there are many memory/file optimisations that could be done. But remember that while fasm can be used as a general purpose data parser that is not its primary purpose.

Seeing as how you have some special parsing requirements for large files then, in the meantime, while waiting for any upgrade, perhaps it would be easiest to write (in assembly of course) a small app to parse and generate the needed data into a single file that you then feed into fasm along with the rest of code.
Post 01 Sep 2011, 14:46
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 01 Sep 2011, 15:49
revolution
Your recommendation to use an external parser is sometimes very hard or even impossible to use, because the macros I am talking about correlate with the code of the compiled executable. I probably need to give an example of such macro to be more clear.

Let's say, I need to compile a proxy dll intercepting calls to a few functions from some other dll. A while ago I've written a macro, which allows to do it easily (attached to the post). And let's say the target is QtGuid4.dll of 160MB size. The following code will intercept some desired functions:
Code:
format PE GUI 4.0 DLL
entry DllEntryPoint

include 'win32a.inc'
include 'forwardedexport.inc'

section '.text' code readable executable

proc DllEntryPoint hinstDLL,fdwReason,lpvReserved
    mov     eax,TRUE
    ret
endp

FormLayoutInvalidate:
        invoke MessageBox,NULL,text1,title,MB_OK
ret

TabBarKeyPress:
  invoke MessageBox,NULL,text2,title,MB_OK
ret

 title           db 'Title',0
      text1           db 'A call to FormLayoutInvalidate catched',0
     text2           db 'A call to TabBarKeyPress catched',0

section '.idata' import data readable writeable

      library user32,'USER32.DLL'

   import user32,\
            MessageBox,'MessageBoxA'

section '.edata' export data readable

       exportForwarded 'forwexp.dll','QtGuid4.dll',\
          FormLayoutInvalidate,'_ZN11QFormLayout10invalidateEv',\
          TabBarKeyPress,'_ZN7QTabBar13keyPressEventEP9QKeyEvent'

section '.reloc' fixups data discardable    


It compiles within 8.5 seconds with a modified version of fasm. Now tell me please, how long does it take for you to compile this code with the official fasm version.

Are you still going to insist this is not the use case of the fasm's primary purpose? Then please show me an external parser which allows to achieve the same.


Description:
Download
Filename: FORWARDEDEXPORT.zip
Filesize: 2.24 KB
Downloaded: 527 Time(s)

Post 01 Sep 2011, 15:49
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20421
Location: In your JS exploiting you and your system
revolution 01 Sep 2011, 22:57
l_inc wrote:
Are you still going to insist this is not the use case of the fasm's primary purpose?
Yes. You are doing it in such a way as to create havoc.

Here is a possible alternative which I expect will compile in milliseconds.
1) generate a file with all the functions from QtGuid4.DLL. Just a simple text list with a small header will suffice.
2) write a custom export macro that places all the exports conditionally based upon "used".
3) include your file.

Code:
macro forwarded_exports file_name,[label,function]
  forward
  if used label
    ;...
  end if
}

include 'exports_from_QtGuid4.dll.txt'    
And in the 'exports_from_QtGuid4.dll.txt' file you might have this:
Code:
forwarded_exports 'forwexp.dll',\
   FormLayoutInvalidate,'_ZN11QFormLayout10invalidateEv',\
   TabBarKeyPress,'_ZN7QTabBar13keyPressEventEP9QKeyEvent',\
   somelabelname,'someexportedname',\
   ;...    
Post 01 Sep 2011, 22:57
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 02 Sep 2011, 01:29
revolution
Quote:
You are doing it in such a way as to create havoc.

I didn't get the metaphor, but that's actually not as simple as you described it. E.g. you removed the original dll name from the macro parameters, but it's actually necessary to build the forwarded export. Second, an ordinal base of the original dll must also be included into the file generated by an external parser. Third, the order of the function names must be preserved by an external parser according to the AddressOfFunctions table including also functions (and just table holes), that are exported only by ordinal (and it's important to generate some collisionless labels for these functions). Thus the names will have to be extra sorted by the macro. Clearly your suggestion can be extended to a working solution, but not as simple as you say and with one (following) "but".

As you said, that I could "feed into fasm along with the rest of code" some preparsed data, I assumed you were talking about the file directive. In your alternative solution you're just replacing this directive with include. But this is actually always possible (back to the stupid masm and bin2db?). If you are familiar with the fasm's "main purpose", what is then the file directive's purpose? I assume the main point was to get rid of some external tools and make fasm to a single powerful and comfortable tool. At least I always loved the fasm's ability to be a standalone. What you suggest now is to operate on tonsils via anus (which is necessary in masm) even though a better way is provided specifically for this case.

So if the author wanted to make fasm better by means of providing the file directive, my suggestion is to do it good and moreover in a very simple way without any damage to the architecture or backward compatibility.
Post 02 Sep 2011, 01:29
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 02 Sep 2011, 07:59
I did not realize that open/close cycle gave such a bad overhead on Windows systems. In DOS the main problem with "file" was when reading multiple small chunks (just like you do it here), because combined with multiple assembly passes it generated a lot of separate HDD accesses and this was causing a long and very noisy assemblies - but loading some disk cache software like SMARTDRV was helping a lot.
And is the slow open/close time somehow related to DLL being very big in this case? As you don't load the whole file into memory and just do small reads from export table, this size should not have such an effect unless it is causing slow open for some cause.

Anyway, the right way to fix it would be to modify Win32 interfaces to provide file handle caching, like you suggested.
Post 02 Sep 2011, 07:59
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 02 Sep 2011, 15:55
Tomasz Grysztar
Quote:
And is the slow open/close time somehow related to DLL being very big in this case?

Well. Only indirectly. Large dll size <= large export table => many file accesses => low performance. The timing of a single file access does not depend on the number of bytes read or on a file size.

In this macro I tried to minimize the number of processing cycles of the file directive as much as I could. E.g. having this improvement it would be possible to load the whole file (or just the export directory) reducing the number of file loads to a single or a few.

Thank you for your support. I thought you will disregard this topic, because it's just a performance issue rather than a bug.
Post 02 Sep 2011, 15:55
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.