I am preparing to finally release the initial version of my new project, developed under the working title "fasm g". Before I do so, I think I should take some time to describe in more detail what is it and how is it related to fasm, in addition to my previous statements.

First of all: this is not fasm 2, and it is not even an x86 assembler. But it does not implement an instruction set for any other specific architecture, either. It is just a bare assembly engine that only has instructions like DB or DD to generate and assemble various data. It is therefore something like a third-generation successor of my old MDAT tool (if one took fasm 1.x and removed all the x86 instructions and formats, leaving just the directives like DB/IF/REPEAT and preprocessor, it could perhaps be called a second-generation MDAT). This means that it probably could be extended to contain an implementation of x86 instructions and become fasm 2, but I'm not going to attempt it unless there is real demand. The new architecture is quite different and while it has its advantages, it also has some weaknesses compared to fasm 1.

Back in 2009, on our fasmcon meeting I gave a short talk about the features I planned for fasm 2. I think that most of what I said then still holds today. In fact, some of the fragments of then hypothetical code I presented can now be successfully assembled not only with fasm g, but even with fasm 1.71 - because during the years of my struggle with ideas for fasm 2 I actually managed to implement some of them (notably the labeled virtual blocks) into fasm 1.x line.

One of the important things I said then was that fasm 2 would be slower than fasm 1, because it would no longer have a separate preprocessor and parser stage, but it would do multiple passes on the source text. This was one of my fundamental assumptions for the new architecture: I wanted to get rid of of fasm's separate layers and separate languages for each layer - but at a cost of performance. Now, when I have fasm g working already, I begin to think that it may actually be possible to design an architecture that would be "the best of both worlds" and use some of the tricks of fasm 1 for better performance while implementing a unified language like fasm g - but this is at the moment just a general vision, like fasm 2 was back in 2009.
And fasm g is in fact exactly what I envisioned back then, it has no separate preprocessor and does multiple passes on pre-tokenized source text, and this is noticeably slower than multiple passes on internally precompiled source that fasm 1 does.

Another consequence of the language unification is that macroinstructions are almost completely incompatible between fasm 1 and fasm g. On the other hand, the language of assembler module has been kept almost unchanged - and it is possible to have sources that are assembled with the same result by both assemblers. Of course such source would have to be limited just to directives like DB, VIRTUAL, REPEAT, LOAD, etc. - but even though this appears to be a very small common subset of languages, one can find many small snippets for fasm 1 that can be assembled with fasm g, and I have been using selected examples taken from this board to test the new assembler.

But probably the most important point remains: how can an assembler that does not implement any actual instruction set be useful at all? And can it even be called an assembler then*?

For the second question my answer is that I use the term "assembler" in a specific meaning, the one I introduced in the Understanding fasm text when I wrote: The defining feature of an assembler how I see it is the "code resolving" ability - the fact that you can forward-reference values of symbols and create self-dependencies that the assembler needs to resolve. It does very literally "assemble" an output which is a solution that needs to comply to all the given constraints. And whether the assembled interdependent pieces of information are the instruction codes for some machine, or data structures, or a graphical data, or sound, it is in my opinion a secondary matter. I am not claiming that my definition of assembler is better or more appropriate than any other out there - but remembering the passionate arguments over what is an assembler that did sometimes happen on the net, I decided I can just as well use my own definition of "assembler" in all my personal work, as long as I state it clearly beforehand (just like in a mathematics book one can use a slightly different definitions of some terms that the ones used by other authors, as long as the definitions are precisely stated before they are used).

Back to the most important question: what can fasm g be used for? The answer may already be obvious to the ones that used fasm 1.x for purposes other than the assembly for x86 architecture. Different instruction sets can be implemented in form of macroinstructions, and some examples of this type already exist for fasm 1. Of course, when these macroinstructions need to be complex, the assembly is going to be much slower than it would be if instruction set was implemented natively into assembler - and this adds to the fact that fasm g in general is slower than fasm 1, so you should certainly not expect an amazing performance. On the other hand, the assembly is often used for small snippets, or programs for machines like microcontrollers that have very little memory - in these cases the sources are small enough that the performance drawbacks can be forgiven, while the instruction sets implemented in form of macroinstructions give great flexibility and can be really fun to play with. And while I do not know whether this new assembler is going to really interest anyone else (especially since it replaces some of the confusing features of fasm 1 with the new ones that may actually still be confusing but in a completely new way), I have a lot of fun with it already. I created the macroinstruction packages that implement the instruction sets for 8086/80186 (including MZ output with relocations), 8051 (with Intel HEX output) and JVM (with .class file generator converted directly from my example for fasm 1) and I plan to include at least these in the initial release.

___
* In our private talk vid suggested to me that I could use the following slogan to advertise my new project: "everything you want from an assembler, without an assembler". I think it is a good one.

Tomasz, all is very interesting, but I would like to see some more details. For example, what will be the syntax of the macro language? How about normal conditional preprocessing? Normal nesting of macro definitions? Normal arithmetic in the preprocessor?

In fact, I want to read the user manual, even more than to download the program.

I have nearly finished the basic manual, which contains the general description of all the features of language, with examples - this will be included in the release. But I also want to write another guide specifically for the people that already know and use the languages of fasm 1 - such "transition guide" I plan to post here, I hope soon.

I think I'm going to make it a short summary of key differences and similarities between fasmg and fasm and leave it there for now. I feel that this project, though a fun thing to play with, consumed way too much of my time recently, a time that maybe would be better spent on implementing AVX512 into fasm. Anyway, I have a basic manual more or less ready, so I can release the thing anytime soon. And then I should probably take a break from it.

This is a brief comparison of fasmg features to their analogs in fasm. It does demonstrate some of the basic commands of fasmg, but it is not a description of this language.

fasmg does have a partial compatibility with fasm, but this applies to just a small subset of fasm's features. fasmg does not implement the x86 architecture instructions or the output formats, any output it generates is through the declarations like DB. It also does not have a separate preprocessor like fasm and thus the language of fasm's preprocessor is also not present. In fasm the preprocessor used a different set of syntax constructions than the assembler to make them distinguishable from each other - in fasmg the language is unified, and features that in case of fasm were implemented in preprocessor (like macros) now use the syntax constructions of fasm's assembler module.

So the only instructions that remain relatively unchanged are the ones that were the directives of assembly module in fasm. Still, it is possible to find some sources that were originally written for fasm but can be assembled with fasmg without any changes. This snippet from the Understanding fasm is one such example: The LOAD and STORE have a new, slightly altered syntax rules, but they also allow the "legacy syntax" identical to the one used by fasm.

While the features of assembly module may work more or less the same, there is a couple of things that were processed by parser module in case of fasm and they are either not present or noticeably modified in case of fasmg. The EQ and IN conditions are not implemented in fasmg as they were inherently not safe to use in general context (they were partially superseded by MATCH in case of fasm and in case of fasmg there are also other possible replacements). The EQTYPE is present in fasmg, but works very differently (as fasm's EQTYPE was actually in part specific to x86 symbols), though it still can be used to detect strings. The anonymous labels are not implemented by fasmg (but they can easily be implemented through macroinstructions, including the variations that could be hard to implement in case of fasm). The symbols starting with dot may appear to behave in the same way as in fasm, but they follow the different set of rules in fasmg (this is related to the fact that fasmg relies heavily on namespaces, and it treats the dot as a special character).

The basic definitions of assembly symbols are the same as in fasm. The labels are defined with ":" or LABEL and can be forward-referenced, the variables are defined with "=" and can be forward-referenced only when they have exactly one value. The forward-referenced values are resolved by fasmg in the same way as fasm did it, so this traditional snippet assembles as it used to: There are also some new syntax variants. With ":=" it is possible to define a constant (it is actually the same as definition with LABEL), while "=:" stacks the values of variable on top of each other, to allow reverting a variable back to its previous value with RESTORE:
When the "=" and "=:" definitions are mixed, the "=" replaces the last value instead of preserving it (but it may still preserve the older ones).

There is still a couple of features of the assembler module of fasm that are not present in fasmg. There is no ALIGN or TIMES - I decided it's better to have them as macros only (ALIGN gains from the macro customization, while TIMES was in fasm only for semi-compatibility with NASM anyway). Also the syntax like "a = byte 1" is not allowed in fasmg (it probably could be added, but the reason for its omitting is similar to why the STORE has a new recommended syntax - fasmg allows sizes to be specified by a numerical expressions and symbols like "byte" simply have the numerical values; placing two numerical expressions next to each other can lead to ill-defined boundaries of said expressions).

The EQU is present in fasmg, even though it was preprocessor's directive in fasm. And as in case of fasm it defines the symbolic value with an effect similar to the simple substitution of text: But there are also some differences, for example the symbolic variables are recognized only in the expressions in the arguments to the instructions.

Since EQU is now a part of the unified language, it is affected by constructions like IF or REPEAT. The symbolic variable can even be forward-referenced, as long as it has a single definition. There is also an additional variant, REEQU, which is to EQU what "=" is to "=:" (it is like RESTORE and EQU combined into one operation).

The REPEAT in fasmg combines the functions of fasm's REPEAT and REPT: The "counter" here is an example of parameter. Any parameter is replaced with its text in the line of source immediately before this line is interpreted - this is the feature of fasmg which the most similar to fasm's preprocessing. But the "#" operator, though gives the similar effect, works quite differently than in fasm - it is never removed from the source text, and it simply causes the adjacent names to be treated as a one long name when looking for an identifier of symbol.
Also "`" now works very differently, though often gives similar results as in fasm: fasmg processes the "`" operator only when it precedes a recognized parameter, and the entire value of parameter is then converted to quoted string.

The macroinstructions in fasmg use the syntax of the assembler: The options like "*", ":" and "&" are present for macro parameters and work like in fasm, but there is no grouping of parameters. Instead, the iteration over parameters now needs to be done with IRP: While there is no directive like REVERSE to invert the order of processing of such list, the order can be altered freely with the new instruction INDX: The %% parameter can be used in addition to % in a block like IRP or REPEAT to get the total number of scheduled repeats. The INDX applies to the current iterator - the block created with IRP or IRPV.

The macros are now the symbols of assembler, and therefore they are handled like the other kinds of such symbols - for example a macro can be forward-referenced when it has a single definition. Such macro can also call itself to create a recursion. But if a macro is redefined and becomes a variable, then when it calls itself it actually calls the previous value - just like in fasm.

Like IRP or MACRO, also the STRUC, MATCH and POSTPONE are implemented in fasmg with the END syntax instead of braces.

The fasm's traditional escaping that was required in nested preprocessing structures is not needed with fasmg constructions - the assembler controls the nesting levels in the same way as fasm always did with IF/WHILE/REPEAT; the "#" character is never removed from text so it is not affected by nesting; and the "`" character is only processed when used with the right name of parameter, so it is enough to use non-conflicting names for the parameters at different levels of nesting. For the rare cases when a macro needs to open another macro without closing it, there is an ESC command.

A few notes about the implementation:

• I have written it as a 32-bit program, using a very basic 386+ instruction set, also without any instructions that do not have the counterparts in the long mode. My idea was that a program that uses just a basic instruction set and a few registers could be more easily converted into other architectures, though I only really had x86-64 in mind. Such simple implementation also means that there are no sophisticated optimizations there, but obviously the processing speed was not my priority here.
  
• The EBP register is left unused by this implementation. I reserved it so that it can be possibly used to address all of the assembler's variables, as such thing would be needed to create a thread-safe library.
  
• The engine is designed in such a way, that it should be relatively easy to make a library out of it that would allow an access to internal API to extend the language with additional symbols (like instruction sets, etc.) even at a run-time. The assemblers for specific architectures (like x86) could be perhaps created in such a modular way.
  
• Currently there are console interfaces for Windows and Linux and the Linux one links to libc, because fasmg requires a well-performing malloc/realloc API. Under Windows it uses HeapAlloc/HeapReAlloc.
  
• The floating-point numbers are not supported yet, though the engine has the reserved places for their implementation. They would be another type of an expression value (recognizable by EQTYPE) in addition to strings and numeric (linear polynomial) values. As I felt that I'm already loosing to much time to this project, I decided I could leave out the floating-point values for now, as I did not need them for any initial examples.
  
• A known bug: the counters created by REPEAT create a special kind of token, which is equivalent to a plain decimal number, but when it is compared with the same decimal number using MATCH, they are not considered identical. I initially left it out in the implementation of MATCH and was too lazy to get back to it.
  

I am preparing the package with the manual, sources and a few examples - I am going to post it here soon.

I like it! It is great to get rid of curly braces, back slashes and differences between preprocessor and assembling stages. On the other hand removing "common", "forward" and "reverse" seems to make the syntax a bit more complex and not so readable. (for example, I didn't understood the example with irp, repeat and indx - are there nested loops, only one loop or two sequential loops?

I also have several questions:
I understand that the speed is not important just now, but still, what is the estimated speed of FASMG related to FASM?

Isn't implementation of the instruction set through macros too slow, especially for such complex CPUs as x86 and x86-64? Or too complex? How about the size optimizations of the instructions?

Are you planning to implement built-in instruction handling in addition to FASMG? Maybe in some modular way that to allow different instruction sets to be switched in easy?

What binary formats are supported?
Are FAS files generated as in FASM? Or maybe in different format?

Sorry for so many questions, but I am really interested, because the syntax seems to be really good, addressing most of FASM flaws.

BTW: If once the preprocessor and the assembler are joined, why not to join the assembly symbols and the preprocessor symbols? Why to keep two different and not compatible entities instead of one. I understand, that it will need to handle two different types of data (numbers and symbols) in one variable, but the type converting is not impossible and may even become great flexible feature. In addition, some directives will become redundant and the syntax cleaner. IMHO.

It is hard to tell, since there are very few constructs that can actually be directly compared. On the source that can be assembled with both assemblers, fasmg is usually a bit slower, though in some cases it was actually faster.

Yes, of course the implementation of x86 as macros is incredibly complex and slow. This is what I wanted to emphasize when I wrote that the slowness of fasmg is further escalated by the fact that the only examples of a "real" assembly must be done with such complex macros, and this may give this engine a "bad publicity". In my tests the processing of any syntax through macros was about one order of magnitude slower than the native implementation. On the other hand, it was really satisfying to get it all working.

The IF directive plus fasm's multi-pass resolving give the same results as with the native implementations of these optimizations. Actually it could be done in the same way with fasm 1, as the examples like DCPU-16 macros demonstrated.

I considered fasmg to be a prototype and perhaps a base engine for fasm2, but whether I ever try to make such fasm2 depends on whether there will be any interest in such project. Because for my personal needs fasm1+fasmg is actually enough for now, and a project like fasm2 would be a huge undertaking for me.

There are no built-in formats, the examples generate everything (including relocations) by themselves.

There are no symbol-dumping and assembly-dumping facilities in this engine at the moment. It's the another fragment of work I decided to set aside, like the floats support.

Here comes the package I prepared. I am a bit tired with this project at the moment, so I have done the final packaging a bit hastily, but I hope this is enough for a first preview.

UPDATE: I have made this package available on the official download page, you can get the latest version there when any new updates are made.

Can macros access a previous decision from a previous pass and declare that a further pass is required?

A nice feature I'm missing in fasm 1. There are some inaccuracies in the manual... Some proofreading might be desirable (I'd like to volunteer ).

revolution

This is possible with fasm 1. It's possible to take values from any specific previous pass and to use those in the following passes or to do smth. only if a condition is true at some specific pass etc. So I guess a mechanism of doing the same wouldn't change.

I mean only for the current instruction. Not from something further back.

revolution
I can't unambiguously decipher your clarification, but it seems like you want to declare a need for another pass by checking if some constraint is satisfied. This is kinda convoluted way of thinking, cause multipass processing is there for satisfying such constraints. So instead of attempting to do some explicit intra-pass specific checking and pass count control you just need to specify the constraint and fasm then applies as many passes as needed to make it true.

What I mean is something like this ARM code: If we don't take into account previous passes and the encodings that were tried then we end up with the message "error: code cannot be generated". We have to have some way to tell if previous passes had tried but failed to find a solution. The only way to solve this is to encode an "inefficient" wide instruction with a small constant value. i.e. Without knowledge of previous passes we get stuck in a loop of alternating encoding narrow and wide instructions over and over.

Note that the alternative narrow encoding is this: But we can't use it because it causes a new pass when "y" has a new value and it then needs to encode "cmp r0,0x100" which only fits in a wide encoding.

The revolution's question is about the "oscillator problem", when some conditional assembly (like when trying to optimize an instruction to a shortest form) causes the initial condition to be changed to its opposite, and this alternates between the consecutive passes. The stream of passes with alternating conditions then never ends and the solution cannot be found.

[Note: the samples I provide below assemble exactly the same with either fasm 1.71 or fasmg pre-release]

Let's have a look at this simplified example: This would be an instruction in some hypothetical architecture, which has a short form for even addresses, but requires a long form when the address is odd. When it is formulated as above, it is in fact a plainly self-contradictory source, so it is nothing surprising that assembler is not able to find a solution. It is just like a classic example of "if not defined, then defined here" antinomy in fasm.

So, to make a source that is not self-contradictory, just like in the example of "if ~defined", we need to modify conditions a bit. This one gets correctly resolved to a long form of instruction: (also, if additional byte was put just before "v", it would also resolve correctly, this time to a short form)

You can look at it from the point of view of tracing the changes of values during the consecutive passes (though that requires some knowledge of fasm's implementation) - the symbol "broken" then simply traces that the optimization to short form failed at least once, and it prevents it from being chosen again (thus preventing the oscillation).
But I prefer to describe the problems like above in the terms of logical contradictions and removing them, because then this is less dependent on the particular implementation, and is more a general feature of language. Of course it is possible that current implementation of fasm or fasmg may not be able to find a solution even in case, when source is not self-contradictory, and then the knowledge of implementation details may help to find the right set of modifications to help the assembler deal with the problem - for this reason in case of fasm I documented some of the details of how the predictions are made. But I still prefer to start with a general formulation for an abstract language, and leave the description of fasm's current implementation for an "appendix" (that is more or less what I did in fasm 1 manual, though in a very compressed form).

revolution asked whether fasmg allows macros to trace the stored decisions across the passes. As l_inc rightly pointed out, this is something that never was a real problem even in case of fasm 1 - because macros are able to define as many local symbols as they need, and they can use these symbols to trace the predictions. The actual problem that applied to fasmarm was with the native instruction handlers, which in case of fasm 1 architecture had much harder way to define any internal symbols or other kind of data storage for this purpose. The macros never had such problem.

Thanks for the explanation.

Also, how does it treat floating point numbers? Specifically: What do SHR, SHL, BSR and BSF do with FP inputs? Are variables and constants tagged in any way as floats? Can we use +, -, *, / etc. to do arithmetic in float space? Can we convert from float to integer and/or integer to float?

A small follow-up in the topic of the problematic "oscillating" optimizations: in my design notes for fasm2 I wrote down an idea for an additional feature that would allow attempts to solve this kind of problems by measuring something like a "stability" of input values. I did not test whether this is viable, but I hoped that the fasmg engine would allow to more easily test various ideas of this kind.

As I wrote a few posts earlier, I did not implement floats in this first version - I decided that this is one the features that could be left out initially, to reduce the amount of work required to get the first version out. But I designed the data structures with the floats in mind - when I add them, they are going to be another type of expression values, like the strings or numeric ones. Every operator is aware of the types of input values and can act accordingly - currently it is usually the conversion of string to numeric value, when a numeric operator is applied (for example (+'a') is a numeric value, while pure ('a') is still a string), and the "string" operator allows the conversion in the opposite direction (like (string 'Hello'+20h)). The "float" operator would allow conversion of standard numeric value to float, but the exact behavior of all the other operators would still need to be decided. Probably some of them would simply disallow input values of float type (and also "float" could disallow input value of string type, as this would still be enforceable with "float +"), but at least the standard arithmetic on floats would be a nice thing to have.

Most useful would be the ability to define the float format. For example for ARM there are various formats used: 1, 2, 4 and 8 bytes. The 4 and 8 byte formats are the usual IEEE754. The 2 byte format has 2 variants. And the 1 byte format for some coprocessor modules.

And further to the 2 byte format one of the variants does not encode any infinity value so the exponent can be all 1's and encode a normal float value.

What I planned was like fasm1 does it: use own high-precision format (at least quad precision) for handling of the floating point value until it is finally used with DD or DQ, or other such directive. Only then the final conversion into selected format would occur. And if I added some additional operators for extraction of fp fields, then one could also create custom formats with macros, in addition to the IEEE ones created with standard data directives.

PS. I have updated the preview package above, I fixed a few small bugs today.

Good work. Say good bye to \{\} forever.

What about the "save block/file" feature?

As for "`", the option to use `UPPERCASE would be nice to auto-create multiple names and texts from one generic "name": NAME, 'name', ID_NAME, TYPE_NAME, name_fp, name_import, 'NAME.BMP', etc. How about ``name or `+name?

Is this redefine for both define/equ? Do if/define/equ work the same together? Have you considered a type of "if symbolic" (ifs)?

What's this .type/.size applied to macro parameters? Does . (ns.size) have a different meaning? I thought the "else match" syntax may be impossible, since there is no way to tell by looking at "else" itself that it corresponds to the match.

Is the "esc" a return from macro?

Any %line, %date, %file/%name tags?