[fasmg] Inline macros. Why not?

Index > Programming Language Design > [fasmg] Inline macros. Why not?

Goto page Previous 1, 2, 3 Next

Do you like the idea of inline macros?

Yes, I like

80%

[ 8 ]

No, I don't

10%

[ 1 ]

Don't care

10%

[ 1 ]

Total Votes : 10

Author

Thread

VEG

Joined: 06 Feb 2013
Posts: 80

VEG 01 Jun 2017, 09:05

I've fixed another bug with overwriting results when nesting was used. Also case-insensitive imacros (with "?" as postfix) are acceptable now. And new instruction "ipurge" was added, it purges an imacro Smile

Code:

macro imacro_preprocess resultn, line&

        local buffer,preprocessed,inline,cursor

        define buffer line :
        define preprocessed :
        while 1
                match a <=? b, preprocessed buffer
                        redefine preprocessed a
                        redefine buffer b

                        match x =?> y, buffer
                                redefine cursor x
                                redefine buffer y
                        else
                                err 'missing ?>'
                                break
                        end match

                        redefine inline :
                        while 1
                                match <=? d, cursor
                                        match i, inline
                                                redefine inline i <?
                                        end match
                                        redefine cursor d ?>
                                else match c <=? d, cursor
                                        match i, inline
                                                redefine inline i c <?
                                        end match
                                        redefine cursor d ?>
                                else match i, inline cursor
                                        redefine inline i
                                        break
                                end match
                                match =?> y, buffer
                                        redefine buffer y
                                else match x =?> y, buffer
                                        match c, cursor
                                                redefine cursor c x
                                        end match
                                        redefine buffer y
                                else
                                        err 'missing ?>'
                                        break
                                end match
                        end while

                        repeat 1, n:resultn
                                resultn = resultn + 1
                                local result#n
                                match :command args, inline
                                        imacro_preprocess resultn, command#_imacro result#n, args
                                else match :command, inline
                                        imacro_preprocess resultn, command#_imacro result#n
                                end match
                                match text, preprocessed result#n
                                        redefine preprocessed text
                                end match
                        end repeat

                else match a, preprocessed buffer
                        redefine preprocessed a
                        break
                end match
        end while

        local execute
        match :command:, preprocessed
                macro execute
                        command
                end macro
        end match
        execute

end macro

macro ? line&

        local resultn
        resultn = 0
        imacro_preprocess resultn, line

end macro

macro imacro line&
        local begin
        match iname=? args, line
                macro begin
                        esc macro iname#_imacro? result, args
                end macro
        else match iname=?, line
                macro begin
                        esc macro iname#_imacro? result
                end macro
        else match iname args, line
                macro begin
                        esc macro iname#_imacro result, args
                end macro
        else match iname, line
                macro begin
                        esc macro iname#_imacro result
                end macro
        end match
        begin
end macro

macro end?.imacro!
        esc end macro
end macro

macro ipurge iname
        purge iname#_imacro
end macro

imacro concat first*,rest*&
        result = string first
        iterate s, rest
                result = string result + (s) shl (((bsr result - 1) shr 3 + 1) shl 3)
        end iterate
end imacro

imacro tohex number*,digits:8
        virtual at 0
                repeat digits
                        digit = ((number) shr ((%%-%) shl 2)) and 0Fh
                        if digit < 10
                                db '0'+digit
                        else
                                db 'A'+digit-10
                        end if
                end repeat
                load result:$ from 0
        end virtual
end imacro

And a new problem Sad

It seems that "macro ?" doesn't process lines inside other macros Sad

It works:

Code:

display <? concat '0x', <? tohex 12345 ?>, 10 ?>

It doesn't work:

Code:

macro test
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
end macro
test

Code:

flat assembler  version g.htee8
imacro.asm [157]:
        test
macro ? [4] macro imacro_preprocess [73] macro execute [1] macro test [1]:
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
Processed: display <? concat '0x', <? tohex 12345 ?>, 10 ?>
Error: invalid expression.

My guess was that it is because of conditional/unconditional execution. I've tried to add "!" to the "macro ?" and "macro imacro_preprocess", and it seems that it goes into an endless loop as the result.

Last edited by VEG on 01 Jun 2017, 09:35; edited 1 time in total

01 Jun 2017, 09:05

VEG

Joined: 06 Feb 2013
Posts: 80

VEG 01 Jun 2017, 09:21

It seems that "macro ?!" also doesn't preprocess lines inside macros:

Code:

macro ?! line&
        display `line, 10
        line
end macro

macro test
        db 0
end macro

Outputs:

Code:

flat assembler  version g.htee8

macro test



1 pass, 0 bytes.

And this variant:

Code:

macro ?! line&
        display `line, 10
        line
end macro

macro test
        db 0
end macro
db 1
test

Outputs something crazy:

Code:

flat assembler  version g.htee8

macro test
db 1
test
db 0
end macro

1 pass, 2 bytes.

When I remove ! from the "macro ?", output is more logical (but without macros at all):

Code:

flat assembler  version g.htee8
db 1
test

1 pass, 2 bytes.

Was it done to avoid preprocessing of "macro ?" itself? How does "macro ?!" work? Maybe it will be ok to pass macros bodies to the "macro ?!" at least. It will be ok to ignore just "macro ?" or "macro ?!" bodies themselves for avoiding of preprocessing of own code.

01 Jun 2017, 09:21

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 09:46

VEG wrote:

How does "macro ?!" work?

This one intercept absolutely every line with no exception. It is only really useful when you need to process the syntax completely on your own (possibly inside a section of code, because you can "purge ?" later).

01 Jun 2017, 09:46

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 09:59

VEG wrote:

And a new problem It seems that "macro ?" doesn't process lines inside other macros
It works:
Code:
display <? concat '0x', <? tohex 12345 ?>, 10 ?>    
It doesn't work:
Code:
macro test
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
end macro
test    
Code:
flat assembler  version g.htee8
imacro.asm [157]:
        test
macro ? [4] macro imacro_preprocess [73] macro execute [1] macro test [1]:
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
Processed: display <? concat '0x', <? tohex 12345 ?>, 10 ?>
Error: invalid expression.    
My guess was that it is because of conditional/unconditional execution. I've tried to add "!" to the "macro ?" and "macro imacro_preprocess", and it seems that it goes into an endless loop as the result.

It does not work because "imacro_preprocess" processes the line "test", but it is no longer processed inside the "test" macro itself. To handle this you need to define "test" in such way that "imacro_preprocess" would get called for the lines inside "test" too. There is a couple of ways to do that.

First, you could simply make the "test" macro unconditional:

Code:

macro test!
        display <? concat '0x', <? tohex 12345 ?>, 10 ?> 
end macro 
test

Second, you could define the macro with an analogous interceptor inside:

Code:

macro test
        macro ? line&
                ; ...
        end macro
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
        purge ?
end macro 
test

01 Jun 2017, 09:59

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 11:05

I remade your sample to show how some of the things may be done differently. I think that use of the namespace is simpler that the name suffix. I also added a "pmacro" variant which can be used to define macro that has preprocessing done for its line. We cannot safely do it for every macro, because applying this universally would also affect out own macros while they need to stay "low level".

Code:

macro imacro_preprocess line&
        local buffer,preprocessed,inline,cursor 
        define buffer line :
        define preprocessed : 
        while 1 
                match a <=? b, preprocessed buffer 
                        redefine preprocessed a 
                        redefine buffer b 

                        match x =?> y, buffer 
                                redefine cursor x 
                                redefine buffer y 
                        else 
                                err 'missing ?>' 
                                break 
                        end match 
                        redefine inline : 
                        while 1 
                                match <=? d, cursor 
                                        match i, inline 
                                                redefine inline i <? 
                                        end match 
                                        redefine cursor d ?> 
                                else match c <=? d, cursor 
                                        match i, inline 
                                                redefine inline i c <? 
                                        end match 
                                        redefine cursor d ?> 
                                else match i, inline cursor 
                                        redefine inline i 
                                        break 
                                end match 
                                match =?> y, buffer 
                                        redefine buffer y 
                                else match x =?> y, buffer 
                                        match c, cursor 
                                                redefine cursor c x 
                                        end match 
                                        redefine buffer y 
                                else 
                                        err 'missing ?>' 
                                        break 
                                end match 
                        end while 
                        match :command, inline 
                                imacro_preprocess imacro.command
                                match text, preprocessed $result 
                                        redefine preprocessed text 
                                end match 
                        end match 
                else match a, preprocessed buffer 
                        redefine preprocessed a 
                        break 
                end match 
        end while 
        match :command:, preprocessed
                command
        end match
end macro 

macro ? line& 
        imacro_preprocess line
end macro

define imacro

macro imacro! definition&
        esc macro imacro.definition
end macro

macro end?.imacro! 
        esc end macro  
end macro  

macro ipurge? iname
        purge imacro.iname
end macro

macro pmacro! definition&
        esc macro definition
                macro ? line&
                        imacro_preprocess line
                end macro
end macro

macro end?.pmacro?!
                purge ?
        esc end macro
end macro

macro ppurge? pname
        purge pname
end macro

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

imacro concat first*,rest*&
        local result
        result = string first 
        iterate s, rest 
                result = string result + (s) shl (((bsr result - 1) shr 3 + 1) shl 3) 
        end iterate
        redefine $result result
end imacro 

imacro tohex number*,digits:8
        local result
        virtual at 0 
                repeat digits 
                        digit = ((number) shr ((%%-%) shl 2)) and 0Fh 
                        if digit < 10 
                                db '0'+digit 
                        else 
                                db 'A'+digit-10 
                        end if 
                end repeat 
                load result:$ from 0 
        end virtual
        redefine $result result
end imacro

pmacro test
        display <? concat '0x', <? tohex 12345 ?>, 10 ?> 
end pmacro
test

01 Jun 2017, 11:05

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 11:19

Another side note: this (excerpt from your macro) does not work like you probably expected it to:

Code:

local result#n 
match :command args, inline 
        imacro_preprocess resultn, command#_imacro result#n, args

The LOCAL directive defines a parameter (just like it did in fasm 1) so it works similarly to how the macro parameters work. It means that it is only replaced when a token with such name is encountered, so it would replace "result0" but not "result#0". What you should do it in this case is to just use this one:

Code:

local result

at startup, and then every "result#n" is going to become local, because "result" will be replaced with a symbol in local namespace and attaching additional suffix to it will create an additional symbol in local namespace. If this feels like a undocumented feature, you can use "result.value#n" instead - this defines "value#n" symbol in the namespace of "result" symbol, which is local, so everything then is local.

01 Jun 2017, 11:19

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 12:04

VEG wrote:

Tomasz Grysztar, oh, nice idea! I haven't thought about this use case. It seems that my variant also works with symbolic variables.

What I had in mind there was to allow these inline macros to work like a true preprocessing, for example:

Code:

imacro d mode
        if mode = 16
                $result equ dw
        else
                $result equ dd
        end if
end imacro

<? d 16 ?> -1

01 Jun 2017, 12:04

VEG

Joined: 06 Feb 2013
Posts: 80

VEG 01 Jun 2017, 13:16

Thank you for your examples.

Tomasz Grysztar wrote:

I think that use of the namespace is simpler that the name suffix.

I had used suffix because I thought that it will help to honor current namespace. But it doesn't. Also it doesn't allow to call <? nsname.mname 1,2,3 ?> at all because match gets just the first piece before the dot, so it needs an improvement which will collect all pieces of name with dots. I just thought that it would be nice to preserve behavior of standard macros. It is not a frequent use case of course, so I'll try to do it later.

01 Jun 2017, 13:16

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 13:45

VEG wrote:

I had used suffix because I thought that it will help to honor current namespace. But it doesn't. Currently, it doesn't allow to call <? nsname.mname 1,2,3 ?> at all because match gets just the first piece before the dot, so it needs an improvement which will collect all pieces of name with dots. I just thought that it would be nice to preserve behavior of standard macros. It is not a frequent use case of course, so I'll try to do it later.

Try matching at the separating space:

Code:

        match iname= args, line
                esc macro iname#_imacro result, args

And perhaps even use a sub-namespace instead of a suffix?

Code:

        match iname= args, line
                esc macro iname.imacro result, args

01 Jun 2017, 13:45

VEG

Joined: 06 Feb 2013
Posts: 80

VEG 01 Jun 2017, 15:25

Quote:

Try matching at the separating space:

Nice, I like this simple solution. I think that I had seen it somewhere in the examples, but haven't noticed how useful it could be and forgot about it.

Quote:

And perhaps even use a sub-namespace instead of a suffix?

But an inline macro name can coincide with a real namespace, and it will define an "imacro" macro inside. It seems that "_imacro" postfix doesn't have any visible drawbacks. Maybe it looks not so beautiful (especially in error messages). But it is not a native solution. We don't have a separate class for inline macros, so we need some thing for differentiation. Actually, first thing that I had tried was the "imacro." prefix, because it looks like the most beautiful and simple solution for a general case. And then I remembered about namespaces...

About pmacro. If "macro" instruction will be redefined after enabling the "macro ?" and defining the imacro_preprocess macros, maybe it will not affect them? Hm... Maybe it will be better to redefine the "macro" instruction, but add some flag which will disable preprocessing inside current macro. Or add an "llmacro" (low-level macro) instruction for special cases. It worth to think how to make it better Smile

01 Jun 2017, 15:25

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 15:56

VEG wrote:

About pmacro. If "macro" instruction will be redefined after enabling the "macro ?" and defining the imacro_preprocess macros, maybe it will not affect them?

One of the problems is that even the "imacro" needs to use the "macro" internally.

01 Jun 2017, 15:56

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 16:17

As a curiosity: it also might be possible to redefine "macro" locally by embedding the area of source in a child namespace while using the "macro" instruction from the outer namespace when needed. But this is a very very tricky and fragile construction:

Code:

define original_macro macro

namespace embedded
postpone
        end namespace
end postpone

match _macro, original_macro
        _macro macro?! definition&
                _macro end?.macro?!
                        purge ?
                        esc end macro
                        purge end?.macro?
                end macro
                esc _macro definition
                esc esc _macro ? line&
                        imacro_preprocess line
                esc esc end macro
        end macro
end match

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

macro test
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
end macro

test

01 Jun 2017, 16:17

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 01 Jun 2017, 18:59

Scrap the above, it actually works much better if there is a proxy macro for END too:

Code:

define original_macro macro
define original_end end

namespace embedded
postpone
        end namespace
end postpone

define end?

match _macro:_end, original_macro:original_end
        iterate directive, macro,struc,virtual,namespace,if,match,rmatch,rawmatch,while,repeat,iterate,rept,irp,irpv,postpone
                _macro end?.directive?!
                       esc  _end directive
                _end macro
        end iterate
        _macro end?! directive
                end?.directive
        _end macro
        _macro macro?! definition&
                esc _macro definition
                _macro ? line&
                        imacro_preprocess line
                _end macro
        _end macro
        _macro end?.macro?!
                purge ?
                esc _end macro
        _end macro
end match

;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;

macro test
        display <? concat '0x', <? tohex 12345 ?>, 10 ?>
end macro

test

This one is less tricky (the double ESC that previous one had to use is something that should not have to happen normally) and it works correctly even with nested macros.

Note that to allow "imacro" definitions to be processed correctly in the embedded namespace you have to make them use the original MACRO and END directives in the same way.

01 Jun 2017, 18:59

VEG

Joined: 06 Feb 2013
Posts: 80

VEG 09 Dec 2017, 09:26

Looks frightening. Maybe it would be better to add some construction like "postpone", which will allow to execute some instructions every time when any new block of the code starts. Like this:

Code:

beforemacro
    imacro_init
end beforemacro

It will enable the inline macros for every following macro (or any other kind of block which create new "context" for code).

09 Dec 2017, 09:26

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 09 Dec 2017, 15:56

The problem is: if you try to re-define portions of the same language that you used to build these redefinitions, it is going to create self-interference no matter what. The less troublesome route is to define another separate language on top of the basic one which is necessary make higher layers work. That's why my proposed solution was to separate namespaces of the two languages so that they would not interfere with each other.

09 Dec 2017, 15:56

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 15 Jan 2020, 07:39

With the introduction of CALM this kind of preprocessing becomes much more viable. And especially the new abilities of MATCH, that I have recently been experimenting with, make it relatively easy to do.

I'm presenting here my proof of concept. It is based on a trick similar to the one I used when I needed to emulate NASM's preprocessor, but with CALM it becomes much easier and faster:

Code:

calminstruction inlinemacro?! declaration&
        local   name
        match   name arguments?, declaration
        arrange tmp, =__inline__.name
        publish name, tmp
        arrange tmp, =struc (return?) =inlinemacro.name arguments
        assemble tmp
end calminstruction

macro end?.inlinemacro?!
        end struc
end macro

calminstruction calminstruction?.initsym? var*, val*
        publish var, val
end calminstruction

calminstruction ? text&
        local   head, tail, name, arguments, more
        transform text
        jno     ready
    preprocess:
        match   head? =__inline__.name tail?, text
        jno     ready
        match   (arguments?) tail?, tail
        jno     ready   ; syntax error
    collect:
        match   arguments?, arguments, ()
        jyes    inline
        match   more?) tail?, tail
        jno     ready   ; syntax error
        arrange arguments, arguments) more
        jump    collect
    inline:
        local   tmp, return, i
        initsym i, 0
        compute i, i+1
        arrange return, =__return__.i
        arrange tmp, return =inlinemacro.name arguments
        arrange text, head return tail
        take    text, tmp
        jump    preprocess
    ready:
        assemble text
        take    , text
        take    text, text
        jyes    preprocess
end calminstruction

Code:

inlinemacro test a
        return = (a) xor 1
end inlinemacro


db test(3), test(1+(1+1)*3)

db test(test(0) shl 1)

This is a basic variant, but it should be easily extendable in various directions, depending on specific needs.

I am going to leave it up to you all to do anything more with it. I confess that development of all things related to CALM project has put a lot of strain on me and I probably should take a break.

15 Jan 2020, 07:39

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 15 Jan 2020, 15:37

I added a tweaked version of the above macro into fasmg's add-on repository on GitHub. I used the newly added option of TRANSFORM to reduce interference with other things and after I checked that it works nicely even when included together with Windows headers and listing I decided it might be a worthy addition to that set of packages (which in general are becoming much better since the introduction of CALM, although I also try to keep them simple and basic).

BTW, the content of listing depends on the order in which INLINE.INC and LISTING.INC are included. For instance this snippet I made while testing with DIALOG.ASM example:

Code:

inlinemacro LoHi? lo*,hi*
        return = (lo) + (hi) shl 16
end inlinemacro

        cmp     [wparam],LoHi(IDOK,BN_CLICKED)

gives the following if INLINE.INC is included before LISTING.INC:

Code:

[000000000040108E] 0000048E: 83 7D 10 01              cmp [wparam],LoHi(IDOK,BN_CLICKED)

but when they are swapped, and LISTING.INC is included first, then the listing contains:

Code:

[000000000040108E]                                    __return__.1 inlinemacro.LoHi? IDOK,BN_CLICKED
[000000000040108E] 0000048E: 83 7D 10 01              cmp [wparam], __return__.1

Yeah, it's hard to take previously mentioned break when the things are working so nicely. But I do not plan anything other than little tweaks (and fixes if necessary) for the near future.

15 Jan 2020, 15:37

bitRAKE

Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri

bitRAKE 18 Mar 2020, 08:27

Code:

inlinemacro reg32 reg64*
        local temp
        if 1 metadataof reg64 relativeto x86.r64
                element temp : x86.r32 + (1 metadataof reg64 - x86.r64)
                return = temp
        else
                err '64-bit register expected'
        end if
end inlinemacro

Often in 64-bit programming I write a macro that needs both register sizes for efficient encoding. Being able to inline this conversion greatly increases the readable of code, IMHO.

The above macro doesn't seem to work in general though - I'm missing something. For example,

Code:

macro Freelist__Get item,list
        mov reg32(item),reg32(list)
        mov reg32(list),[list]
end macro

macro Freelist__Put list,item
        mov [list],reg32(item)
        mov reg32(list),reg32(item)
end macro

...will error in the operand parsing.

Thank you for your continued work on this.

_________________
¯\(°_o)/¯ AI may [not] have aided with the above reply.

18 Mar 2020, 08:27

Tomasz Grysztar

Joined: 16 Jun 2003
Posts: 8435
Location: Kraków, Poland

Tomasz Grysztar 18 Mar 2020, 09:05

bitRAKE wrote:

The above macro doesn't seem to work in general though - I'm missing something. For example,
Code:
macro Freelist__Get item,list
        mov reg32(item),reg32(list)
        mov reg32(list),[list]
end macro

macro Freelist__Put list,item
        mov [list],reg32(item)
        mov reg32(list),reg32(item)
end macro    
...will error in the operand parsing.

The lines that define macro are normally not intercepted by this inline macro processor, to preprocess them (and unroll inline macros during the definition of your Freelist__Get/Freelist__Put), you'd need to make the preprocessing interceptor unconditional, in other words change it from:

Code:

calminstruction ? text&

to:

Code:

calminstruction ?! text&

I did not make it default, because unconditional macros sometimes create unwanted interference. For example, you'd need to make sure to not preprocess lines with INLINEMACRO definitions:

Code:

calminstruction ?! text&
        local   head, tail, name, arguments, more
        match   =inlinemacro? tail, text
        jyes    ready
        transform text, inlinemacro
        jno     ready
      ; ...

Also, making this preprocessing unconditional would enable it in constructs like IF, etc. But there is a trap - for example if you use inline macro inside ELSE IF clause:

Code:

else if foo(bar) = 0

then preprocessing is going to turn it into something like:

Code:

__return__.1 inlinemacro.foo bar
else if __return__.1 = 0

and obviously this is not going to work as intended.

18 Mar 2020, 09:05

bitRAKE

Joined: 21 Jul 2003
Posts: 4260
Location: vpcmpistri

bitRAKE 18 Mar 2020, 09:22

Just changing to Freelist__Get!/Freelist__Put! works and the bytes just disappear. Very Happy

(Discussed at the top of the page.) Embarassed

18 Mar 2020, 09:22

Goto page Previous 1, 2, 3 Next

< Last Thread | Next Thread >

Forum Rules:

You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum