This does not work:




I am kind of lost between the numeric value, the decimal token and the string token.

ARRANGE in principle operates on symbolic values, as it composes a tokenized text and only symbolic variables provide values of this kind. There is a single exception: ARRANGE also can convert a plain positive number into a token corresponding to a sequence of decimal characters, which is a kind of value it can put into the produced tokenized text.

For other types of computed values, including negative numbers, floating-point numbers, strings, and linear polynomials, there is no such conversion to text implemented and ARRANGE does not allow them. If you need to pass such value to a directive like DISPLAY, the best way is to refer to a variable holding the value. The manual contains an example that does exactly that - look for the "calminstruction show? text&" example, which uses "initsym" to compose assembled text that refers to a local variable. Pointing to a local variable is only possible with a trick like "initsym", but you could avoid it by using a global variable as a proxy:

I think I am lost because I don't picture the full hierarchy of fasmg object types, then I have a very approximate view of what calm instructions do.

It really follows the principles of fasm 1 quite closely. In fasm 1 you had two layers: preprocessor, operating purely on tokenized text, which was called "symbolic value" (EQU/DEFINE were preprocessor's directives in fasm 1, hence they defined symbolic variables), and assembler, operating on computed binary objects.

With fasmg I retained this classification, but now there is a single engine, so symbolic values and computed values can be assigned to symbols on the same layer. But overall, you still have two worlds. First a symbolic one, where you operate on tokenized text, this is where you use EQU, DEFINE, and also operations like MATCH. In case of CALM it's the ARRANGE and also MATCH (there they are in fact dual to each other).

The second world is all the computed values. All the types that exist in fasmg existed in fasm 1 already and could be compared with EQTYPE - numeric values (which were an early version of fasmg's linear polynomials, allowing for up to two scaled registers and possibly relocatable base, and only 65 bits of precision), floating-point values, strings. In fasm 1 only numeric values could be held in variables, so = could not assign floating-point value or string without converting it to plain numeric value. In case of fasmg variables are capable of holding any of these three types.

In short, EQU/DEFINE is for symbolic values (snippets of source text), = is for all the other ones. This classic example from the manual illustrates the basic idea: Note that this snippet does exactly the same thing in fasm 1 as in fasmg.

In CALM context you have ARRANGE operating almost purely in the world of text (with that single exception of being able to convert plain numeric value to a sequence of decimal digits), analogously to EQU, while COMPUTE is like = and can produce all other kinds of values.

STRINGIFY takes a symbolic value and converts it to string type. This is now in the computed world, you no longer have a snippet of source text, only a string of raw bits. There is no way to get back to source text world from here - again with the single exception of producing a decimal digits sequence, which only works for plan positive integer values. You could actually treat the string as a binary representation of a huge integer, then convert it to decimal representation, and then force DISPLAY to convert that number into binary string once again. But that implies unnecessarily converting back and forth and I do not recommend this technique: To see what happens there, just replace "display" with some nonsense that would cause an error: And the error message shows what is produced: That's why I suggest keeping the string value in a variable and then referring DISPLAY to that variable. But you can use whatever suits you best.

A string is not a token from the tokenized world?

You mean there is a string token and a string binary value?

Yes, the string in tokenized text is a different thing, it is complete with enclosing quotes, it could even be missing the closing quote and be causing an error because of that.

The string as the value in computed context is just the bytes corresponding to the contents between quotes, and nothing else.

You could think of this distinction as: a representation of string in source text vs its run-time form.

Allright! And calm stringify generates a string binary value, not a string token in a "tokenized" value.

Then if I am not mistaken the string operator does cast inplace the bits of a value to a a binary string. It has nothing to do with a "string" token.

This is what I meant, all this is very confusing,"string" can be a token or a binary value. This near-internals information is required for proper understanding of calm programming, and even normal fasmg programming (equ/define,=,etc).

Basically, we would need a "advanced" cheat sheet to know what types of values (tokenized/binary) a command/operator needs and produces.

-- edit

I guess, apart from classic macros, "commands" working and producing tokenized values should be somewhat rare. Then an explicit reference of those commands could be nice, and those transforming for good tokenized values to binary values (since there is no come back).

I generally refer to the former as "quoted string" and I think I more or less consistently did this throughout documentation. It might be worth it to add "(not a quoted string)" remark to the STRINGIFY description.

That could be a way to remove the confusion.

That said, if I picture it well, the calm "assemble" command will expect a variable containing a "tokenized" value. If this "tokenized" value does contain itself a "display" command, the "assemble" command will simply compute the "literals" token (string, numbers) arguments to a binary value, resolve the name token arguments to their corresponding binary values, then call the "display" command with all those binary values as arguments. And the "display" command will refuse any argument which is not a binary value, for instance a "tokenized" value.

Is that not too wrong and/or incomplete?

The way I prefer to describe it, ASSEMBLE command does not compute anything, it just takes the value of a symbolic variable and passes it as a line of text to the assembly queue. I pointed to an artificially induced error message to demonstrate it: what is shown after the "Processed:" header is the raw line of text that got passed to the assembler by the ASSEMBLE command (and this text in our example is produced by ARRANGE). When the text is "display string 8392569455040360033", then the DISPLAY instruction takes over, and it is the handler of this specific instruction that decides what to do further with the text "string 8392569455040360033". In case of DISPLAY, the text that follows it is parsed as an expression, and then the expression is evaluated and the final value displayed as a string or as a single byte value, depending on the resulting type.

I should stress this further: DISPLAY, just like any other instruction or macroinstruction, sees its arguments as a pure text, a portion of the line of source that invoked it. It then parses this text and interprets it according to its own rules. I'm not sure what you had in mind when saying that DISPLAY would refuse any argument which is not a binary value, but it is false in every sense that I can think of. Not only the arguments to DISPLAY are initially just a text, they may also contain identifiers of symbolic variables, that would then be unrolled recursively to find out what the expression is: (By the way, this is another source that assembles the same with fasm 1 as well as fasm g.)

When you said "line of text" , I guess a sequence of lexical tokens.

So in my words:

The command arguments would actually be given to the command code as a sequence of tokens, I guess all the tokens following the command name token on the line.

With your example, the display command is calling some expression evaluation code on a part of the sequence of tokens it was given as argument.

Is calling this expression evaluation code is automatic on each argument sequence of tokens (separated by the ',' token) or the command has to call explicitely this evaluation code on a selected parts of the sequence of argument tokens?

This seems to be a question concerning specific implementation. From the point of view of language design it should not matter whether particular implementation parses the whole line in advance, or in portions just-in-time. Is there some hidden problem behind this question?

Well, it depends, if calling the evaluation code on the argument sequence of tokens is done independently from the command code, that would remove some freedom from the command to decide the format of its arguments, then the language specifications... but it would mean the command is receiving a list of "evaluated" binary values unless there is pure token-only evaluation code?

As I mentioned, an instruction "sees" the arguments as a line of pure text - it could turn out that DISPLAY was overridden by a macro, but it doesn't matter whether the instruction is implemented in form of macro, CALM, or a built-in handler, in every case you have the same text, like "string 8392569455040360033", and it is up to the implementation of the instruction to decide how to interpret it.

I'm not sure what the confusion is. I should stress once more: when ARRANGE produces a "display string 8392569455040360033" text and passes it to the assembly queue, it is the same as if you have simply written "display string 8392569455040360033" as a line in source file, to the letter. The decimal digits as ASCII characters. Under the hood there might be some optimizations that make representation look a bit different, just to save on processing time, but from the point of view of language user there really should be no difference how this line of text was produced.

Talking about the specifics of fasmg's internal implementation would only cloud the matters.

Can we say that arrange is similar to the eval that many other interpreted languages have?

No, fasmg has an actual EVAL and it does something different - it takes the value of a string (possibly a computed one) and interprets it as a new text to assemble: A string value differs from a symbolic value - the former is a binary sequence of characters extracted from between quotes or computed in some other way, the latter is a piece of source text (tokenized, but otherwise preserved, it is what you assign with EQU etc.)

Also you probably meant ASSEMBLE, not ARRANGE. ASSEMBLE takes the value of a symbolic variable and assembles it as a new line, an effect similar to this snippet in base language (not CALM): As for ARRANGE, it is basically the opposite of MATCH. Where MATCH takes a symbolic value and cuts it into pieces, ARRANGE can take such pieces and compose them into a single symbolic value.

Allright, then all commands receive their arguments as a block of tokenized text.

That's why "the reference manual" I did talk about in another thread.

For instance in the case the display command, we would need to know what the command is doing with that argument of tokenized text: splitting based on the ',' token, then "expression evaluation" on each of those arguments until we end up with only "binary" values which bytes will be sent to the output device.

Well, this is the idea.

Yes, a good idea, and this time you described the process quite well.

Anyway, the specific issue that started this thread is going to become less relevant once I finish and release CALM phase 3.

Yep, I think this reference manual is critical for fasmg.

What do you mean by CALM phase 3?

It's the new development branch I linked above.

With a version compiled from that branch you can do this directly in CALM: