Since FASM is a very flexible and powerful tool, I was thinking about using it to generate executable files for exotic operating systems. And it turns out that one of the hard parts here is that fixups might be needed but in a format different from the one used in PE EXE.

Meanwhile section is a directive, not a macro. One could redefine the keyword with a macro but it looks like there’s no way to get the list of all memory references (in fact, any memory references that use labels as part of an expression) within a macro or with assembler directives.

The first thing is natural: preprocessor has no knowledge about labels. But what about giving access to the process of fixups generation to the source code? I feel it’s not quite simple as a concept that might conflict with other pieces, but it would be nice to discuss the possibilities.

Hence fasmg? It truly is superior in every way.

Except for its performance; very slow. And a requirement for many support files instead of just a single exe.

revolution those are opposite ends of an imaginary use case landscape. Build performance is mitigated by parallelism and caching. Environments so space restricted to require a single EXE use case are probably being cross-built for in the first place. But I welcome all examples to the contrary - just haven't seen them.

It isn't about the space requirements for more files, it is the use case of needing to keep them all together when transferring it around. It's not too dissimilar to the DLL hell problem.

Parallelism can't fix the performance problem because it is single threaded. Caching also can't fix the performance problem because it isn't a data transfer bottleneck.

Yep. It looks more like a toy than a mature development tool for everyday use. In spite of some cool ideas behind fasmg, if I needed to slow down build process, I’d switch to C++.

Parallelism as in launching multiple instances. Caching as in only building watch changed.

Parallelism applies equally to fasm also. I do already use it, and it helps a lot. But the basic performance can't be improved this way.

For caching are you suggesting that the program should be broken down into only small pieces and we generate multiple .o files, then use a linker for join them? We would need a make system to control that, but it is doable. Although not efficient IMO. fasm can assemble faster than the linker can link .o files.

I don't disagree, but I do think monolithic development has its limitations. Maybe, I'm just getting old. The moving edge of development needs to fit within the scope of my limited mind.

https://github.com/nidud/asmc has a very simple make program - in assembler. fasmg can also be the linker.

Why monolitic? Modules/units are not exclusively tied to object files. In fact, the whole Pascal ecosystem shows that source level units make experience smoother and decreases build time greatly. Implementing effective Pascal-like units with FASM is a bit tricky but what can easily be done is usually quite enough and has better build times.

Anyway, guys, we seem to have gone off-topic.

Let me try to bring it back on topic, then. Here's a link to a previous discussion of the same thing: https://board.flatassembler.net/topic.php?t=16028

Even though at the time I was attempting some experiments within limits of fasm 1 architecture, after all I was only able to solve this issue with fasmg. In fact, that thread I linked is one of several discussions that took place in 2014 that ultimately led me to create fasmg (I started designing fasmg in October 2014, after another of these discussions, this time the one about fasm's inability to provide true namespaces).

Moreover, the issue of instruction encoders having to interact with customized relocation generators was what pushed me to invent CALM instead of making a native assembler based on fasmg engine. And these capabilities are what made it possible for me to write things like my PE tutorial, complete with manual creation of all structures.

In the first topic linked I see suggestions that closely resemble what I imagine the feature could be implemented like. If I missed nothing the only real reason why something like that hasn’t been implemented since then is that FASM 2 -> fasm g became the new target, and FASM 1 took place of an old boring toy.

<OffTopic>
Even though I can understand the pleasure of playing with fasm g as an example of quite generic tool flexible enough for literally anything, I’d like to somehow encourage you, Tomasz, to continue the work on FASM 1 as well. You may consider this comparison inappropiate but I’ve heard of a guy who invented a programming language. The language got popular really fast and is still alive 50 years later, it became the basis of a modern language with mind-blowing features and has strong support within its community. Meanwhile, the guy was trying to fix some he-thought-so mistakes in the original design of the language, so another language emerged: much more consistent, with very cool and quite compact set of language features but still quite usable. Then 1 or 2 more languages followed. They were even more ideal. But you know what, all of these “later” languages got their community acceptance but none of them got close enough to the success of the first one. That guy was Niklaus Wirth.

FASM 1 really is a great tool. I can’t even imagine how to teach students x86 asm with any other assembler these days: the ease of getting started, the ease of use, the simple and cool idea of “the source code says it all” somewhat contrary to the widely used approaches, and at the same time quite a lot of cool things that can be done after diving deeper. The most advanced of them literally get crazy with FASM and ask me to share more info about cool tricks with it. I even had to give it a try and to write a set of articles in Russian with practical recommendations on how to get started with macros and stuff to make life with bigger FASM projects easier. Some pieces were asked to be moved to my normal lectures.

So, let me insist: FASM 1 might be your personal Pascal. I believe, at some point in time, when the amount of different and often contradictory features gets to the critical volume, there might occur reasons to reimplement it as FASM 2 reviewing the features and ensuring their better consistency. But fasm g is unlikely to be the new widely accepted best assembler, for well-known reasons.
</OffTopic>

But what I didn’t get from the previous discussion is whether such a feature is implementable at all. Are there any complexities that make implementing it a bad idea?

Your remarks sound to me as fitting about 10 years in the past. The point that you mention had already come, after years and years of pushing fasm's core further and further. Note that I first announced need for fasm 2 on fasmcon 2009, as I was already aware that for some of the ideas that we were already discussing back then, the engine of fasm 1 was not going to be enough. And only several years later, when a breaking point came and fasm's engine could no longer handle all the interesting things that I wanted to try, fasm g was born. And even then I keep developing fasm 1, but although I managed to back-port a few of fun and simple features from fasmg, in general capabilities of fasm's core are not enough to implement some of the most useful features of fasmg.

I also think that your analogy is far from perfect: fasmg is not really a new separate language, as it offers quite a lot of compatibility with fasm 1, sometimes going an extra mile just to provide that. You can have fasmg assemble many of existing fasm-based Win32/Win64 projects, with little to no changes. With projects that rely on additional macros, you have to convert them to new syntax, but I also have examples of large projects that could be adapted with relatively little work compared to their scope, for example HeavyThing (and, BTW, entire HeavyThing assembles in about 3 seconds with fasm 1 and 10 seconds with fasmg - not as much difference as one might think, but that's because of a heavy macro usage there). Anyway, in my guides and tutorials I switch between fasm and fasmg when needed, and it is not even a big deal most of the time.

My point was more about the fact that when something becomes too generic/academic/abstract for everyday tasks, it might not become the replacement of the original stuff, from the community point of view.

Still, the question about theoretical implementability of the feature seems to be left unanswered

What if there were a named virtual address space - called say fixups. If one wanted to get the size of that virtual space: ...and then one could read the data within? Seems like an elegant solution, but I haven't looked into what code changes would need to be made. Ideally, it's nice if the address space was read-only and updated on use to exactly match internal relocation data. It makes the minimum assumption about what is needed and would try to reuse the existing code.

I'm not sure if something like this would suit your need?

This is exactly what has been discussed in the topic linked to by Tomasz. But I’ve never seen the evaluation of the amount of changes and possible inconsistencies that may occur.

It only conflicts with same name virtual spaces (not likely). Making it read-only prevents most other kinds of problems, but adds a test to all other virtual spaces (rarely impacts performance?).

l_inc did propose the exact same thing! Now, I'm quite confident this is the way forward. Time to start work on a patch.

I’m more worried about the coexistence of this and the multipass assembly in FASM manner.

I'm in the process of learning more about this problem. So, there are bound to be errors in understanding. Yet, the relocation data is additive. Which means latter data after some point would not be considered. For example, if the processing of the relocation data itself created more relocation data - that additional data would be ignored.

The fixups pseudo-address space could be marked "dirty" if the number of relocations changed -- signaling another pass is needed. As is done with other address spaces. There might be a way to dead-lock this mechanism into a loop -- to be discovered, no doubt. (We relegate this modality to "user error".)

Presently, on the front-end relocations are pregenerated by mark_relocation (formats.inc:411) based on output format: They store 4,5,12 and 12 bytes respectively at [free_additional_memory]. Then there is a post relocation process -- unneeded by the implementation I have in mind. Luckily, they are stored linearly during assembly - this should allow the "virtual fixups" to access this array directly, rather than copy to another address space.

I've decided to change the format directive to further limit the generation of this additional data. "format relocatable binary" In this way, it would not conflict with any existing code. Format prefixes are kind of underutilized.