fasmg as a preprocessor

While the macro language of fasmg is much more capable than that of fasm 1, we may still prefer to assemble larger projects with the latter, especially for performance reasons. An interesting question is whether we could use fasmg as a preprocessor for the source before passing it to fasm 1 for the final assembly.

It is easy to process all the lines of source file with "macro ?!" feature of fasmg, but it is normal for assembly programs to span multiple files, joined by the INCLUDE directives. A preprocessor macro-script would need to process them accordingly and collect the contents of all included files for processing.

Relatively recently I discovered a trick that uses some of the obscure features of fasmg to process source text in an environment where none or almost none of the usual instructions/directives are defined. This allows to use a simple "macro ?" to catch all the lines except the ones with INCLUDE, by defining INCLUDE as an unconditional instruction within that environment. And because INCLUDE itself is not intercepted by "macro ?", the latter is then applied to the lines inside the included file.

Here I present a simple template of such preprocessing macro-script:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        db `line,13,10
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

end namespace

It simply passes to the output all the tokenized lines without any further changes. The source becomes solidified into a single file, with comments and excess whitespace removed, as this happens during the tokenization process.

The script requires "source" variable to be defined with a quoted path to the main source file to preprocess. This can be done by adding a definition on top of the script, or by defining it directly from the command line:

Code:

fasmg preprocess.asm -isource='hello.asm' preprocessed.asm

The above works in Windows command line, in Linux console the quotes need to be escaped:

Code:

fasmg preprocess.asm -isource=\'hello.asm\' preprocessed.asm

There are several tricks used here. First, the namespace of a local variable is used as a context for reading and interpreting the source files. This namespace is detached from the main symbol tree, therefore inside of it no defined symbols exist, including no instructions/directives. We define a single macroinstruction inside that namespace - INCLUDE - and make it call the original INCLUDE from the main symbol tree.

This works because of the outer MATCH that assigns the symbolic links to various symbols from the main tree (stored in "symlinks" variable) to parameters that are replaced with these symbolic links everywhere inside that block, which included the definition of "preprocess" macro. Therefore the body of "preprocess" macro contains symbolic links instead of just raw names of directives like INCLUDE or IF, so they are recognized and called correctly even from inside the "empty" namespace. When INCLUDE macro is processed, its body contains a symbolic link to the original INCLUDE.

The DB directive is not among those replaced with symbolic links, but "namespace PREPROCESSOR" block is used to temporarily switch context back to where all the normal instructions are defined. Inside this block we may put some more interesting preprocessing later.

The preprocessing ends when the value of __FILE__ signal that we are back in the file containing the preprocessing script, which means that everything inside the INCLUDE has been processed.

Now that we have a template ready, we can try doing something more interesting with it. I have chosen a simple task - scan all the function labels (assumed to be defined as "function:") and the CALL instructions in the source and then generate EXTRN declarations for any function that is seen to be called but not defined anywhere. A script doing this may look like this:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        match name:, line
                                                                if ~ defined local.name
                                                                        restore local.name          ; force variable status to disable forward-referencing
                                                                        local.name = 1
                                                                end if
                                                        else match =call? name, line
                                                                match any any, name                 ; ignore multi-token arguments
                                                                else
                                                                        if ~ defined called.name
                                                                                restore called.name
                                                                                called.name = 1
                                                                                define called name  ; collect for IRPV
                                                                        end if
                                                                end match
                                                        end match
                                                        db `line,13,10
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

        irpv name, called
                if ~ defined local.name
                        db 13,10,'extrn ',`name
                end if
        end irpv

end namespace

This still does not alter any of the source lines, it just appends some EXTRN definitions at the end.

Now we might prefer to put the EXTRN definition on top of the file. This can be done easily by keeping the preprocessed source temporarily in a VIRTUAL block. Since fasm 1 does not allow EXTRN definitions before the FORMAT directive we need to look for FORMAT and only then start buffering the further lines in a VIRTUAL block so we can later dump them all after the EXTRN definitions are generated:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                VIRTUAL_BLOCK = 0

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        match name:, line
                                                                if ~ defined local.name
                                                                        restore local.name          ; force variable status to disable forward-referencing
                                                                        local.name = 1
                                                                end if
                                                        else match =call? name, line
                                                                match any any, name                 ; ignore multi-token arguments
                                                                else
                                                                        if ~ defined called.name
                                                                                restore called.name
                                                                                called.name = 1
                                                                                define called name  ; collect for IRPV
                                                                        end if
                                                                end match
                                                        end match
                                                        db `line,13,10
                                                        match =format? any, line
                                                                VIRTUAL_BLOCK = 1
                                                                virtual
                                                                VirtualBlock::
                                                        end match
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

        if VIRTUAL_BLOCK
                end virtual
        end if

        irpv name, called
                if ~ defined local.name
                        db 'extrn ',`name,13,10
                end if
        end irpv

        virtual VirtualBlock
                load contents:$-$$ from $$
        end virtual
        db contents

end namespace

That's all for now. I may post more when I prepare another interesting example. If you do something interesting with this template, please let us know!