fasmg as a preprocessor

Index > Macroinstructions > fasmg as a preprocessor

Author

Thread

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8390
Location: Kraków, Poland

Tomasz Grysztar 06 Jul 2018, 18:40

While the macro language of fasmg is much more capable than that of fasm 1, we may still prefer to assemble larger projects with the latter, especially for performance reasons. An interesting question is whether we could use fasmg as a preprocessor for the source before passing it to fasm 1 for the final assembly.

It is easy to process all the lines of source file with "macro ?!" feature of fasmg, but it is normal for assembly programs to span multiple files, joined by the INCLUDE directives. A preprocessor macro-script would need to process them accordingly and collect the contents of all included files for processing.

Relatively recently I discovered a trick that uses some of the obscure features of fasmg to process source text in an environment where none or almost none of the usual instructions/directives are defined. This allows to use a simple "macro ?" to catch all the lines except the ones with INCLUDE, by defining INCLUDE as an unconditional instruction within that environment. And because INCLUDE itself is not intercepted by "macro ?", the latter is then applied to the lines inside the included file.

Here I present a simple template of such preprocessing macro-script:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        db `line,13,10
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

end namespace

It simply passes to the output all the tokenized lines without any further changes. The source becomes solidified into a single file, with comments and excess whitespace removed, as this happens during the tokenization process.

The script requires "source" variable to be defined with a quoted path to the main source file to preprocess. This can be done by adding a definition on top of the script, or by defining it directly from the command line:

Code:

fasmg preprocess.asm -isource='hello.asm' preprocessed.asm

The above works in Windows command line, in Linux console the quotes need to be escaped:

Code:

fasmg preprocess.asm -isource=\'hello.asm\' preprocessed.asm

There are several tricks used here. First, the namespace of a local variable is used as a context for reading and interpreting the source files. This namespace is detached from the main symbol tree, therefore inside of it no defined symbols exist, including no instructions/directives. We define a single macroinstruction inside that namespace - INCLUDE - and make it call the original INCLUDE from the main symbol tree.

This works because of the outer MATCH that assigns the symbolic links to various symbols from the main tree (stored in "symlinks" variable) to parameters that are replaced with these symbolic links everywhere inside that block, which included the definition of "preprocess" macro. Therefore the body of "preprocess" macro contains symbolic links instead of just raw names of directives like INCLUDE or IF, so they are recognized and called correctly even from inside the "empty" namespace. When INCLUDE macro is processed, its body contains a symbolic link to the original INCLUDE.

The DB directive is not among those replaced with symbolic links, but "namespace PREPROCESSOR" block is used to temporarily switch context back to where all the normal instructions are defined. Inside this block we may put some more interesting preprocessing later.

The preprocessing ends when the value of __FILE__ signal that we are back in the file containing the preprocessing script, which means that everything inside the INCLUDE has been processed.

Now that we have a template ready, we can try doing something more interesting with it. I have chosen a simple task - scan all the function labels (assumed to be defined as "function:") and the CALL instructions in the source and then generate EXTRN declarations for any function that is seen to be called but not defined anywhere. A script doing this may look like this:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        match name:, line
                                                                if ~ defined local.name
                                                                        restore local.name          ; force variable status to disable forward-referencing
                                                                        local.name = 1
                                                                end if
                                                        else match =call? name, line
                                                                match any any, name                 ; ignore multi-token arguments
                                                                else
                                                                        if ~ defined called.name
                                                                                restore called.name
                                                                                called.name = 1
                                                                                define called name  ; collect for IRPV
                                                                        end if
                                                                end match
                                                        end match
                                                        db `line,13,10
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

        irpv name, called
                if ~ defined local.name
                        db 13,10,'extrn ',`name
                end if
        end irpv

end namespace

This still does not alter any of the source lines, it just appends some EXTRN definitions at the end.

Now we might prefer to put the EXTRN definition on top of the file. This can be done easily by keeping the preprocessed source temporarily in a VIRTUAL block. Since fasm 1 does not allow EXTRN definitions before the FORMAT directive we need to look for FORMAT and only then start buffering the further lines in a VIRTUAL block so we can later dump them all after the EXTRN definitions are generated:

Code:

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                VIRTUAL_BLOCK = 0

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        match name:, line
                                                                if ~ defined local.name
                                                                        restore local.name          ; force variable status to disable forward-referencing
                                                                        local.name = 1
                                                                end if
                                                        else match =call? name, line
                                                                match any any, name                 ; ignore multi-token arguments
                                                                else
                                                                        if ~ defined called.name
                                                                                restore called.name
                                                                                called.name = 1
                                                                                define called name  ; collect for IRPV
                                                                        end if
                                                                end match
                                                        end match
                                                        db `line,13,10
                                                        match =format? any, line
                                                                VIRTUAL_BLOCK = 1
                                                                virtual
                                                                VirtualBlock::
                                                        end match
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

        if VIRTUAL_BLOCK
                end virtual
        end if

        irpv name, called
                if ~ defined local.name
                        db 'extrn ',`name,13,10
                end if
        end irpv

        virtual VirtualBlock
                load contents:$-$$ from $$
        end virtual
        db contents

end namespace

That's all for now. I may post more when I prepare another interesting example. If you do something interesting with this template, please let us know!

06 Jul 2018, 18:40

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8390
Location: Kraków, Poland

Tomasz Grysztar 07 Jul 2018, 11:33

What if one would prefer to just write a regular macros that would get preprocessed, even if just for clarity? We can make a framework that would allow something like that.

Let's say we have the following macro that we would like to apply in our source:

Code:

macro mov? args&
        match [destaddr]=,[srcaddr], args
                push [srcaddr]
                pop [destaddr]
        else
                mov args
        end match
end macro

A modified preprocessing script allowing to use the above macro with relatively little changes may look like this:

Code:

macro emit? line&
        db `line,13,10
end macro

define MACROS

define symlinks macro:end
match macro:end, symlinks

        macro MACROS.macro?! definition&
                local NAME
                rawmatch name parameters, definition
                        define NAME name
                else
                        define NAME definition
                end rawmatch
                match name, NAME
                define name +name
                macro end?.macro?!
                        esc end macro
                        end match
                        purge end?.macro?
                end macro
                esc macro definition
        end macro

end match

namespace MACROS

        ; The macros to be preprocessed go here

        macro mov? args&
                match [destaddr]=,[srcaddr], args
                        emit push [srcaddr]
                        emit pop [destaddr]
                else
                        emit mov args
                end match
        end macro

end namespace

PREPROCESSOR := __FILE__

namespace PREPROCESSOR

        define symlinks include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR
        match include:macro:purge:if:else:end:namespace:__FILE__:PREPROCESSOR, symlinks

                macro preprocess file
                        local empty
                        macro empty.include?! file
                               include file
                        end macro
                        namespace empty
                                macro ? line&
                                        if __FILE__ = PREPROCESSOR
                                                purge ?
                                                line
                                        else
                                                namespace PREPROCESSOR
                                                        match +command, MACROS.line
                                                                command
                                                        else
                                                                emit line
                                                        end match
                                                end namespace
                                        end if
                                end macro
                                include file
                        end namespace
                end macro

        end match

        preprocess source

end namespace

The EMIT macro is needed to distinguish the lines that would go into preprocessed source from the ones processed locally by the preprocessing script.

07 Jul 2018, 11:33

dosmancer

Joined: 20 Feb 2025
Posts: 17
Location: Kingdom of Sweden

dosmancer 04 Mar 2025, 23:46

I tried this with calminstructions and what's nice is that I don't need to add `emit` to many lines (like for the ordinary macros) since CALM sends data explicitly to the assembler via the `assemble` command. So it was sufficient to override `assemble` to get some calminstructions to work with preprocess.asm without any changes to them:

Code:

namespace MACROS
        macro calminstruction?.assemble? &line&
                stringify line
                emit lengthof line, line
                emit 1, 13
                emit 1, 10
        end macro

        calminstruction @enter
                local tmp
                arrange tmp, =push =bp
                assemble tmp
                arrange tmp, =mov =bp, =sp
                assemble tmp
        end calminstruction

        calminstruction @pushall
                local tmp
                arrange tmp, =pushad
                assemble tmp
                arrange tmp, =push =ds
                assemble tmp
                arrange tmp, =push =es
                assemble tmp
        end calminstruction

        calminstruction @popall
                local tmp
                arrange tmp, =pop =es
                assemble tmp
                arrange tmp, =pop =ds
                assemble tmp
                arrange tmp, =popad
                assemble tmp
        end calminstruction

        iterate <__a, __b>, \
                a, na, \
                b, nb, \
                c, nc, \
                e, ne, \
                g, ng, \
                l, nl, \
                o, no, \
                p, np, \
                s, ns, \
                z, nz

                calminstruction @if#__a line&
                        local tmp
                        arrange tmp, =j=__b =@=f
                        assemble tmp
                        assemble line
                        arrange tmp, =@=@:
                        assemble tmp
                end calminstruction

                calminstruction @if#__b line&
                        local tmp
                        arrange tmp, =j=__a =@=f
                        assemble tmp
                        assemble line
                        arrange tmp, =@=@:
                        assemble tmp
                end calminstruction

                ; Make visible to the preprocessor
                define @if#__a +@if#__a
                define @if#__b +@if#__b
        end iterate

        ; Make visible to the preprocessor
        define @enter +@enter
        define @pushall +@pushall
        define @popall +@popall
end namespace

Gives extra incitement to convert my macros to CALM. Admittedly not the most impressive macros but I like the idea of having a separate preprocess step generating a file so I can see very clearly what effect macros have on the code.

04 Mar 2025, 23:46

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum