Ever since I released fasmg 5 years ago, I've been mentioning my intention to make some kind of "assembler construction kit" based on this engine. What I initially had in mind was a set of documentation and tools that would aid in writing native instruction handlers working with fasmg core - and I even wrote a short tutorial implementing few examples of simple x86 instructions. Near the end of that tutorial I mentioned one of the problems that actually kept me from trying to create fasm 2 this way. After I started using fasmg to customize output formats (including relocations) through macros, I realized it would be very hard to give up flexibility of this approach. I though I could perhaps make native instruction handlers that would be able to call user-defined macros for things like "emit dword", but I felt it would be clunky and not easy to implement well. On the other hand, my implementation of x86 instructions in form of macros, even though slow, turned out to be not really as slow as I feared. Honestly, it was (if only barely) fast enough that I could use it for medium-sized projects like fasmg and enjoy all the flexibility given by the macros. In addition to that, my vision of fasm 2 had some bold ideas that turned out to be much easier to implement when using fasmg's macros for the task.

In parallel (and since the very beginning) people have been suggesting another idea - to make a separate language that could be compiled into instruction handlers. Such definitions could even be processed and compiled at the time of assembly, so they might be as flexible as macro packages, while offering better performance. I liked the idea in principle, as it had something in common with a bytecode used internally by fasm 1 (which is one of the few features of fasm that were completely missing in fasmg). A potential of bringing advantages of both approaches together sounded really tempting. The problem, however, was that I had no clear idea what that language should look like. While designing the language of fasmg one of my basic principles was to keep as much familiarity as possible, retaining or at least imitating various syntactic structures of fasm 1. But here I needed to invent something distinct and I had no good idea how to make it not look like another completely new language that one needs to learn in addition to the basic language of fasm.

All these ideas have been hanging around in the corners of my mind, waiting for a good moment to get combined into something that would really speak to me. And as I continued to be reminded from time to time that awfully long assembly times with macros may sometimes be a roadblock, I had a very good incentive to keep thinking about it. And this is how I came up with the design of CALM language.

It has several features that made me feel that this is exactly what I needed. First and foremost, it utilizes the existing framework of fasmg engine in such way that it was really easy to implement (which was quite important for me, considering various constraints). As a consequence, it also uses concepts and syntactical structures that should feel familiar to anyone already knowing fasmg, even though it really is a completely new sub-language. For example, there is a MATCH command that in use feels mostly the same as such named directive of fasm and fasmg. On the other hand, there is a completely a different control flow - it uses jumps instead of structural programming (the main rationale for this was to allow for better performance), which makes it feel like a kind of assembly language. And this is where the name came from - Compiled Assembly-Like Macro.

The instructions written in this language are compiled at the time of definition into a code for a specialized VM. Unlike macros, they do not need to allocate a new namespace for local symbols every time they are called, which allows them to not only be faster, but also require much less memory for processing. And for any task that they are not able to perform themselves, they can simply construct a text and pass it to regular assembly - this allows things like x86 instruction handler calling DD macro defined by the relocation-handling macros, which one of the things I so much wanted to have.

The fact that these instructions can be transparently used in place of existing macro, and also can execute other existing macros themselves, makes it possible to write CALM replacements for individual macros without having to rewrite entire packages. For instance, I started by rewriting "x86.parse_operand" macro in 80386.INC package, while leaving all other macros that interact with it unchanged. This change alone increased the speed of assembly noticeably, while requiring substantially less effort that rewriting the whole instruction set.

The command set is very concise, at least for start. I needed just a dozen of commands to re-implement x86 macros, and whatever cannot be done with them I do by simply assembling a regular statement of fasmg's language from inside the CALM instruction. While I have many interesting ideas for additional commands, I prefer to not implement them too hastily, but first find out in practice what is really needed.

What I am especially fond of is that this extension plugs into the fasmg engine in a way that should allow to utilize it to its full potential. While it is still not going to be anywhere as efficient as the specialized and minimalistic approach of fasm 1, it brings together features of both designs into a compromise that may (I hope) be just good enough for most purposes. I am only afraid that by removing the main bottlenecks CALM instructions may expose some portions of the engine that I did not care to optimize, because previously it did not matter. Normally this should simply lead to further development and improving the code, but currently this just adds to the end of terribly long list of additional things to do that I brought onto myself by starting this project.

Also, I would like to thank MaoKo for some early testing of the extension.

Q. Why not compile the existing macros instead?

A: The macros by their very nature are not well-suited to be compiled. Because they are at their core just a substitution of text, the lines they generate may turn out to contain different commands and different syntactical structures every time a macro is called. While this allows for some fun tricks, it also prevents a pre-compilation of a kind that is available to language like CALM.

It is important to note that this basic simplicity of macros is also often an unwanted feature. For example when you write a macro like:
you probably do not really want this macro to be able to execute an IF directive when called like "set IF 0, 1". It is a side-effect of macro being just a substitution of text and to get rid of this unwanted effect and allow the instruction to be pre-compiled, the semantics of the entire process need to change. In the language of CALM all commands have well-defined effects and the data of the arguments passed to the instruction are held and processed in variables accessed by these commands. The above macro rewritten as CALM instruction would look like: and here semantics prevent "var" argument from being anything other than the identifier of a symbol to define.

Q. How fast is it?

A: For x86 assembly, we can get the final verdict only after implementing entirety of instruction set using CALM, although even rewriting just a single key macro in the package can already have a noticeable effect. Please do not expect miracles, though - this is supposed to somewhat counteract ridiculous slowness of implementing everything through macros, not magically make it as fast as a natively implemented assembler.

Currently I have rewritten just a few of the most frequently used macros in the basic x86 package, and this already substantially reduced the time of fasmg's self-hosting.
With 80386.INC:
With 80386.ALM (which still uses 80386.INC under the hood, and just overwrites a couple of key macros): (the memory requirements are also similarly reduced, fasmg's macros are quite memory-hungry).

Rewriting all instructions into CALM should increase the speed even further (although it is going to have diminishing returns). However, this still would not use the potential of CALM to its fullest. I'm rewriting the macros in a way that preserves some compatibility, for example "x86.parse_instruction" can still be used the same way as with macro implementation (it is used by some external packages, like the "fastcall" macro). This requires some additional shuffling of the values between namespaces, etc. I think it might be worth it to later attempt writing another implementation completely from scratch, with architecture perhaps more reminiscent of fasm 1 internals, as it could be even faster.

Update (2021): The current version of x86 package, making full use of CALM capabilities, got down this self-hosting time to 0.6 seconds on the same machine.

Q. Does this make normal macros obsolete?

A: I think of these two ways of defining instructions as complementing each other. Someone may find it easier to write a macro for a simple task where the performance difference does not matter. Some ideas may be better conveyed in form of a macro. Also, macros may be able to achieve some things not possible with CALM and vice versa, but they can cooperate and call each other if needed.

On the other hand, it feels like many of the existing macro packages would simply become much better if rewritten using CALM. This is certainly going to be a continued process and I would expect the content of standard packages to change quite a lot over time. The good news is that all old macros are still going to work as usual, CALM just adds another way of implementing new instructions.

One type of macros that almost make no sense to use when CALM is available are the interceptor macros, used for example to detect and process nonstandard syntax. Because they are usually called for a large percentage of lines in the source text, having them as transparent and fast as possible is crucial. CALM can provide both these things. Let me show you another example that I used for testing: Even though this interceptor is called only for lines starting with a label, it still slows down the assembly process noticeably even on medium-sized sources. However the slowdown nearly disappeared when I tried it with a CALM version of the same interceptor:
On the opposite end, an example of a package that relies on the features of macros that are (at least for now) not directly available to CALM instructions is the one that provides anonymous labels.

Q. What does it mean for the prospect of fasm 2?

A: After the dust settles, I am certainly going to make another attempt at implementing my advanced x86 encoder in form of CALM instructions. I'm probably going to start it from scratch, as mentioned above, to make optimal use of the new techniques. I hope that if I succeed, this may be something deserving the name of fasm 2 - with all the customizability I wanted it to have and perhaps (hopefully!) an acceptable performance.

Q. I would like to try it. Where is it?

A: I'm taking my time to prepare a good initial release. You can, however, access the current shapshot from the public copy of fasmg's Fossil repository.

you are bloating assembler
and confusing self-reliance (or self-indulgence, if you please) with actual tool-kit for assembly purpose

what is needed for that range of flexibility and effectiveness that you seek is a compiler that can change itself yet retaining the possibility to produce for any given platform

I converted more of the x86 instructions, including all the most crucial ones, and got self-hosting down to: (compared to 5.1 seconds with pure macro implementation). This is more or less what I expected. In my various preliminary tests CALM instructions are on average about 6 times faster than equivalent macros.

I'm going to keep working on the new instruction packages, but fasmg itself seems mostly ready to release at this point (I just need to run it through a fuzzer now). If you'd prefer me to release it early, please let me know!

It felt like a discovery to me when I realized that the engine actually allows me to customize commands used to define CALM code, not only in form of macros, but even as CALM instructions themselves. At first I thought it might seem a little crazy, but it turns out that it can be quite pleasant work with: Altering existing commands is possible as well: And, going a step further, an unconditional interceptor gives even more options of customizing syntax (note that only this kind of interceptor works in CALM definitions, as they otherwise use a different assembly mode): For now I'm just playing with it (it is really satisfying when a world of new possibilities emerges from a good design), but this shows how it might be possible to write new modules in multiple layers, allowing to sculpt definitions of an instruction set with a syntax tuned specifically for a given architecture. However, I keep writing x86 instruction handlers as mostly raw CALM code, to get a better feel of it (and see whether it needs any improvements). But when I start writing the next encoder, I may even attempt to write something that would generate CALM instructions out of a more maintainable set of definitions.

There are, in fact, so many interesting things to do with it, that I'm a bit overwhelmed, especially since I have nearly depleted my reserves of spare time for now. This is perhaps an argument for releasing CALM-powered fasmg already, and hoping that there might be others interested in joining the efforts.

Wow this is fantastic!

My vote is definitely a +1 for CALM-powered fasmg release, though I don't know how I feel about retiring my reliance on fasm1

Is this going to become "fasm 2" then?

I was already working on rewriting my ez80 macros in CALM before reading this topic. I have found that even trivial macros are faster when translated to CALM and can be even faster if carefully rewritten. I initially assumed that compute would simplify constant subexpressions as it precompiled, but when I realized that wasn't the case, I now precompute those expressions manually instead and inject the final result with repeat 1. Lately, I have been struggling to figure out a way to create unique label names in a custom CALM command. Your unique macro sounds promising, but it doesn't seem to be useful for anything. For example:

shows that unique only gets called once inside asm, just like how init only gets called once inside unique.

I think my current specific problem could be solved with . or # support in labels like so:

The only workaround I have found so far is passing them explicitly like:

Yes, I made a mistake, UNIQUE should be evaluated at different point. I'm going to rewrite it.

As for your second problem, allowing # in CALM command/label names is certainly possible, just a bit of work. Please be patient, though - at the moment I'm having more ideas than I'm able to process.

I updated the text of my example, it just needed UNIQUE to be assembled at the time when ASM is executed, not when it is defined. Note that I needed INITSYM there in order to preserve the recognition context of "ln".

As for your second problem, all I can offer for now is some kind of pre-defined name pool:

UPDATE: I managed to add concatenation support to the commands/labels identification, which allows for this implementation of "getname" in above sample: Obviously, it no longer needs the name pool then.

BTW, as it seems there are more people using version from repository than I thought, I started generating new version numbers for these snapshots.

Intriguing, you found a middle ground to increase the flexibility even further! I was happy to find the condition flag set by MATCH/TRANSFORM/CHECK persists beyond the following line. Seems like this was by design, but not indicated in the documentation. Lest you'd just include the branch targets on the same line (i.e. CHECK {yes},{no},a=20). Hope it stays that way.

Yes, it is by design. There is this sentence near the current end of section 15: The ASSEMBLE even goes the extra mile in order to preserve this flag.

Can you better explain what COMPUTE creates? Yet, the last example needs to convert the result to decimal and then a string. What does this extra separation offer in terms of the COMPUTE design? Are the symbols evaluated only during ARRANGE and just pre-compiled by COMPUTE? The auto-bracketing of sub-expressions certainly simplifies syntax. Thank you.

This is based on nomenclature that dates back to fasm 1. A numeric result is what you put into a variable when you use "=" in regular assembly: Floats, strings and linear polynomials are all kinds of such result. What is important here is that there is an evaluation of expression involved.

On the other hand, a symbolic value is a pre-tokenized text, this is what you assign with EQU/DEFINE: There is no evaluation of expression here (although EQU does symbolic replacements, in CALM this is made into separate operation with TRANSFORM), this is just a sequence of syntactical tokens "3", "*" and "7".

All values of arguments to CALM instruction are initially symbolic. When COMPUTE sees such value in an expression it calculates, it needs to parse the sequence of tokens and then evaluate it as a sub-expression. On the other hand, if you give COMPUTE an expression on variables that have numeric values, it does not need to do any parsing, because the "outer" expression has been pre-parsed during CALM compilation.

ARRANGE allows to build a new symbolic value by stitching together various sequences of tokens. When you give it a variable that has a numeric value instead of symbolic, it has to somehow make a sequence of tokens out of it. Therefore it converts it to a decimal token if possible.

Finally, when a sequence of tokens is converted into a string with STRINGIFY, it is made into a numeric value again - a kind of string that you can assign using "=" operator.

Bug report:
the 32-bit calm version on fossil crashes under Linux when compiling.
I bisected the branch with the following results:


To test each branch I did the following:

1. Compile the branch using fasmg.is7xq
   
2. chmod +x fasmg
   
3. Move the new fasmg to an opportune location
   
4. Compile the branch again using the new fasmg
   

All the versions run correctly when given no arguments, showing the help string.
With the "BAD" versions when compiling I had an output such as:

With the new bracket detection, parsing the dup syntax manually now seems feasible, but still complicated enough I wanted to share my implementation. This reimplements various data definition macros in terms of single element emits. I am using something similar to this so that duplicated values can be relocated properly.

Also, I forgot, this is using libc version 2.30-8

Marut, since I can't reproduce the crash, here's my latest 32-bit linux executable to see if it still crashes for you.

Your version is working fine on my end, jacobly.

EDIT: Ok, I think I got it.
It was a bootstrapping problem: to compile I need a working compiler, and the public fasmg binaries don't yet support the CALM directives, which are already integral part of the latest development version.
So I had to compile an older version first, one that had support for the new directives without requiring them to run; looks like I picked a botched version.
So the latest version is self-hosting fine.

TL;DR: self-hosting shenanigans, the compiler was the problem, not the compiled.

Ah, other ways to bootstrap are to replace the first line of selfhost.inc with include '../../examples/x86/include/80386.inc' and use the latest fasmg release, or use the latest fasm release.

Great work! I'm amazed to see how some of you are becoming more fluent in CALM than I am, even before the official release. And the bracket detection has only been implemented today (I was not even sure if it's usable enough), so I'm impressed that you already incorporated it so nicely! I'm going to need something similar for my relocating formatters, and I wanted to do it a bit differently, but I like some your tricks.

Yeah, I'm still using fasm 1 for safe bootstrapping myself.

Yeah, at first I wasn't sure if not being able to match "enclosings" was going to be an issue, but it still ended up being pretty obvious how to use it to detect arguments entirely enclosed in matching parentheses like so:

which is much better than before:

which in turn, is way better than the messy recursive solution I had before CALM.

The first official release is out (version "ist2m"). While there remain some x86 instruction set macros that I wanted to convert, I felt that I should not withhold this any longer, as the base package seems already quite stable and way ahead of what was before. And all of you that already started working with CALM, you may now publish your work knowing that the officially downloaded fasmg is going to support it.

At the same time I also updated some of the supplementary macro packages in the GitHub repository. This includes a new XCALM.INC package which provides some of the basic additional commands for CALM that I first showed earlier in this thread. I found that I use them all the time, especially since INITSYM allows to prepare a text equipped with recognition context of CALM instruction, which in turn allows ASM to assemble instructions referring to local symbols.

I certainly should start working on some kind of tutorial for CALM, explaining the tricks like these in depth. On the other hand, I am still learning it myself, even simple things like INITSYM were something that I discovered rather that designed, and I expect there are other interesting techniques waiting to be found (and I'm pretty sure you're going to surprise me with your own!).