flat assembler
Message board for the users of flat assembler.

Index > Main > [Suggestion???] Fixups with a macro

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
Since FASM is a very flexible and powerful tool, I was thinking about using it to generate executable files for exotic operating systems. And it turns out that one of the hard parts here is that fixups might be needed but in a format different from the one used in PE EXE.

Meanwhile section is a directive, not a macro. One could redefine the keyword with a macro but it looks like there’s no way to get the list of all memory references (in fact, any memory references that use labels as part of an expression) within a macro or with assembler directives.

The first thing is natural: preprocessor has no knowledge about labels. But what about giving access to the process of fixups generation to the source code? I feel it’s not quite simple as a concept that might conflict with other pieces, but it would be nice to discuss the possibilities.
Post 19 Jul 2020, 21:36
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
Hence fasmg? It truly is superior in every way. Wink

_________________
¯\(°_o)/¯ unlicense.org
Post 19 Jul 2020, 22:08
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17624
Location: In your JS exploiting you and your system
revolution
bitRAKE wrote:
Hence fasmg? It truly is superior in every way.
Except for its performance; very slow. And a requirement for many support files instead of just a single exe.
Post 19 Jul 2020, 22:36
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
revolution those are opposite ends of an imaginary use case landscape. Build performance is mitigated by parallelism and caching. Environments so space restricted to require a single EXE use case are probably being cross-built for in the first place. But I welcome all examples to the contrary - just haven't seen them.

_________________
¯\(°_o)/¯ unlicense.org
Post 19 Jul 2020, 22:58
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17624
Location: In your JS exploiting you and your system
revolution
It isn't about the space requirements for more files, it is the use case of needing to keep them all together when transferring it around. It's not too dissimilar to the DLL hell problem.

Parallelism can't fix the performance problem because it is single threaded. Caching also can't fix the performance problem because it isn't a data transfer bottleneck.
Post 20 Jul 2020, 03:26
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
revolution wrote:
bitRAKE wrote:
Hence fasmg? It truly is superior in every way.
Except for its performance; very slow. And a requirement for many support files instead of just a single exe.

Yep. It looks more like a toy than a mature development tool for everyday use. In spite of some cool ideas behind fasmg, if I needed to slow down build process, I’d switch to C++.
Post 20 Jul 2020, 04:50
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
revolution wrote:
Parallelism can't fix the performance problem because it is single threaded. Caching also can't fix the performance problem because it isn't a data transfer bottleneck.
Parallelism as in launching multiple instances. Caching as in only building watch changed.

_________________
¯\(°_o)/¯ unlicense.org
Post 20 Jul 2020, 08:53
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17624
Location: In your JS exploiting you and your system
revolution
bitRAKE wrote:
Parallelism as in launching multiple instances. Caching as in only building watch changed.
Parallelism applies equally to fasm also. I do already use it, and it helps a lot. But the basic performance can't be improved this way.

For caching are you suggesting that the program should be broken down into only small pieces and we generate multiple .o files, then use a linker for join them? We would need a make system to control that, but it is doable. Although not efficient IMO. fasm can assemble faster than the linker can link .o files.
Post 20 Jul 2020, 09:07
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
I don't disagree, but I do think monolithic development has its limitations. Maybe, I'm just getting old. The moving edge of development needs to fit within the scope of my limited mind.

https://github.com/nidud/asmc has a very simple make program - in assembler. fasmg can also be the linker. Wink

_________________
¯\(°_o)/¯ unlicense.org
Post 20 Jul 2020, 10:15
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
Why monolitic? Modules/units are not exclusively tied to object files. In fact, the whole Pascal ecosystem shows that source level units make experience smoother and decreases build time greatly. Implementing effective Pascal-like units with FASM is a bit tricky but what can easily be done is usually quite enough and has better build times.
Post 20 Jul 2020, 11:20
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
Anyway, guys, we seem to have gone off-topic.
Post 20 Jul 2020, 11:26
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7781
Location: Kraków, Poland
Tomasz Grysztar
DimonSoft wrote:
Anyway, guys, we seem to have gone off-topic.
Let me try to bring it back on topic, then. Here's a link to a previous discussion of the same thing: https://board.flatassembler.net/topic.php?t=16028

Even though at the time I was attempting some experiments within limits of fasm 1 architecture, after all I was only able to solve this issue with fasmg. In fact, that thread I linked is one of several discussions that took place in 2014 that ultimately led me to create fasmg (I started designing fasmg in October 2014, after another of these discussions, this time the one about fasm's inability to provide true namespaces).

Moreover, the issue of instruction encoders having to interact with customized relocation generators was what pushed me to invent CALM instead of making a native assembler based on fasmg engine. And these capabilities are what made it possible for me to write things like my PE tutorial, complete with manual creation of all structures.
Post 20 Jul 2020, 12:01
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
In the first topic linked I see suggestions that closely resemble what I imagine the feature could be implemented like. If I missed nothing the only real reason why something like that hasn’t been implemented since then is that FASM 2 -> fasm g became the new target, and FASM 1 took place of an old boring toy.

<OffTopic>
Even though I can understand the pleasure of playing with fasm g as an example of quite generic tool flexible enough for literally anything, I’d like to somehow encourage you, Tomasz, to continue the work on FASM 1 as well. You may consider this comparison inappropiate but I’ve heard of a guy who invented a programming language. The language got popular really fast and is still alive 50 years later, it became the basis of a modern language with mind-blowing features and has strong support within its community. Meanwhile, the guy was trying to fix some he-thought-so mistakes in the original design of the language, so another language emerged: much more consistent, with very cool and quite compact set of language features but still quite usable. Then 1 or 2 more languages followed. They were even more ideal. But you know what, all of these “later” languages got their community acceptance but none of them got close enough to the success of the first one. That guy was Niklaus Wirth.

FASM 1 really is a great tool. I can’t even imagine how to teach students x86 asm with any other assembler these days: the ease of getting started, the ease of use, the simple and cool idea of “the source code says it all” somewhat contrary to the widely used approaches, and at the same time quite a lot of cool things that can be done after diving deeper. The most advanced of them literally get crazy with FASM and ask me to share more info about cool tricks with it. I even had to give it a try and to write a set of articles in Russian with practical recommendations on how to get started with macros and stuff to make life with bigger FASM projects easier. Some pieces were asked to be moved to my normal lectures.

So, let me insist: FASM 1 might be your personal Pascal. I believe, at some point in time, when the amount of different and often contradictory features gets to the critical volume, there might occur reasons to reimplement it as FASM 2 reviewing the features and ensuring their better consistency. But fasm g is unlikely to be the new widely accepted best assembler, for well-known reasons.
</OffTopic>

But what I didn’t get from the previous discussion is whether such a feature is implementable at all. Are there any complexities that make implementing it a bad idea?
Post 20 Jul 2020, 14:05
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7781
Location: Kraków, Poland
Tomasz Grysztar
DimonSoft wrote:
So, let me insist: FASM 1 might be your personal Pascal. I believe, at some point in time, when the amount of different and often contradictory features gets to the critical volume, there might occur reasons to reimplement it as FASM 2 reviewing the features and ensuring their better consistency.
Your remarks sound to me as fitting about 10 years in the past. The point that you mention had already come, after years and years of pushing fasm's core further and further. Note that I first announced need for fasm 2 on fasmcon 2009, as I was already aware that for some of the ideas that we were already discussing back then, the engine of fasm 1 was not going to be enough. And only several years later, when a breaking point came and fasm's engine could no longer handle all the interesting things that I wanted to try, fasm g was born. And even then I keep developing fasm 1, but although I managed to back-port a few of fun and simple features from fasmg, in general capabilities of fasm's core are not enough to implement some of the most useful features of fasmg.

I also think that your analogy is far from perfect: fasmg is not really a new separate language, as it offers quite a lot of compatibility with fasm 1, sometimes going an extra mile just to provide that. You can have fasmg assemble many of existing fasm-based Win32/Win64 projects, with little to no changes. With projects that rely on additional macros, you have to convert them to new syntax, but I also have examples of large projects that could be adapted with relatively little work compared to their scope, for example HeavyThing (and, BTW, entire HeavyThing assembles in about 3 seconds with fasm 1 and 10 seconds with fasmg - not as much difference as one might think, but that's because of a heavy macro usage there). Anyway, in my guides and tutorials I switch between fasm and fasmg when needed, and it is not even a big deal most of the time.
Post 20 Jul 2020, 16:11
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
My point was more about the fact that when something becomes too generic/academic/abstract for everyday tasks, it might not become the replacement of the original stuff, from the community point of view.

Still, the question about theoretical implementability of the feature seems to be left unanswered Smile
Post 20 Jul 2020, 18:52
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
What if there were a named virtual address space - called say fixups. If one wanted to get the size of that virtual space:
Code:
virtual fixups
    f_length = $
end virtual
i=0
while i < f_length
    load a byte from fixups:i
    i = i + 1
end while    
...and then one could read the data within? Seems like an elegant solution, but I haven't looked into what code changes would need to be made. Ideally, it's nice if the address space was read-only and updated on use to exactly match internal relocation data. It makes the minimum assumption about what is needed and would try to reuse the existing code.

I'm not sure if something like this would suit your need?

_________________
¯\(°_o)/¯ unlicense.org
Post 25 Jul 2020, 02:25
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
This is exactly what has been discussed in the topic linked to by Tomasz. But I’ve never seen the evaluation of the amount of changes and possible inconsistencies that may occur.
Post 25 Jul 2020, 07:46
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
It only conflicts with same name virtual spaces (not likely). Making it read-only prevents most other kinds of problems, but adds a test to all other virtual spaces (rarely impacts performance?).

l_inc did propose the exact same thing! Now, I'm quite confident this is the way forward. Time to start work on a patch.

_________________
¯\(°_o)/¯ unlicense.org
Post 25 Jul 2020, 07:58
View user's profile Send private message Visit poster's website Reply with quote
DimonSoft



Joined: 03 Mar 2010
Posts: 807
Location: Belarus
DimonSoft
I’m more worried about the coexistence of this and the multipass assembly in FASM manner.
Post 25 Jul 2020, 11:39
View user's profile Send private message Visit poster's website Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 3018
Location: vpcmipstrm
bitRAKE
I'm in the process of learning more about this problem. So, there are bound to be errors in understanding. Yet, the relocation data is additive. Which means latter data after some point would not be considered. For example, if the processing of the relocation data itself created more relocation data - that additional data would be ignored.

The fixups pseudo-address space could be marked "dirty" if the number of relocations changed -- signaling another pass is needed. As is done with other address spaces. There might be a way to dead-lock this mechanism into a loop -- to be discovered, no doubt. (We relegate this modality to "user error".)

Presently, on the front-end relocations are pregenerated by mark_relocation (formats.inc:411) based on output format:
Code:
        cmp     [output_format],2
        je      mark_mz_relocation
        cmp     [output_format],3
        je      mark_pe_relocation
        cmp     [output_format],4
        je      mark_coff_relocation
        cmp     [output_format],5
        je      mark_elf_relocation    
They store 4,5,12 and 12 bytes respectively at [free_additional_memory]. Then there is a post relocation process -- unneeded by the implementation I have in mind. Luckily, they are stored linearly during assembly - this should allow the "virtual fixups" to access this array directly, rather than copy to another address space.

I've decided to change the format directive to further limit the generation of this additional data. "format relocatable binary" In this way, it would not conflict with any existing code. Format prefixes are kind of underutilized.

_________________
¯\(°_o)/¯ unlicense.org
Post 25 Jul 2020, 19:48
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.