flat assembler
Message board for the users of flat assembler.

flat assembler > Heap > Ask to Tomasz (x64 porting)

Author
Thread Post new topic Reply to topic
ProMiNick



Joined: 24 Mar 2012
Posts: 321
Location: Russian Federation, Sochi
Why porting to x64 realized so like it is?
I mean why parts of expressions weren`t replaced with parameters. And setup of that parameters not included in file - 1 for every bitness. Instead of that macros are used. (macros has different syntaxes for fasm1 & fasmg, but symbol definition has common syntax)

that
Code:
if ns.address_registers eq esp & x86.mode = 64
        ns.displacement = ns.displacement shl 1
        x86.encode_address_32_64 ns
        ns.mode = 64    
could be realized via parameters in 2 includes (1 for 32 bit) and (1 for 64 bit)
Code:
define _regs.stack esp;rsp in 64 bit include
define _mem.elements (x86.mode/8)* ;in both includes    

and than patch source that
Code:
[esp+4]    
become
Code:
[_regs.stack+_mem.elements(1)]    

of cource in fasmg there is no _mem.elements addends.
By analogy to esp/rsp replasement with _regs.stack other regs could be replaced (in places where such replacement needed) eax/rax as _regs.accum etc.
Some fasmg variables could be altered to qword size in both bitnesses with unused high part in case of 32.

_________________
I don`t like to refer by "you" to one person.
My soul requires acronim "thou" instead.
Post 13 Jan 2019, 00:16
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 7299
Location: Kraków, Poland
What I have done is not "porting", it is adapting 32-bit program to run in long mode with very little alterations, a trick similar to what I have done with fasm 1 before (but now, with instruction set in form of macros we have ability to make some even more interesting tricks - note that I demonstrated we could get rid of all 67h prefixes in case of fasmg). The original source can be maintained as a 32-bit program still, while this set of tricks allows to assemble it at any moment to a 64-bit ELF that you can run on a system not supporting 32-bit executables.

Actual porting would require the original sources to be modified, all pointer fields in structures would have to be updated, many of structures would grow quite much because of that (and because of additional alignment adjustments). Then the code would need to be altered to use Rxx registers where it operates on pointers, but not for other uses of 32-bit values, etc. etc. This would certainly take much longer than the 3 hours I recorded while making the tricky adaptation.

At core, fasmg continues to be 32-bit, as this allows it to run in more places. Also, the growth of data structures because of use of larger pointers would most likely hit performance, while it is very unlikely to need that much memory in any of the usual uses of fasmg (though I'm ready to be surprised).
Post 13 Jan 2019, 07:37
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 7299
Location: Kraków, Poland
Oh, perhaps I misunderstood the question at first. If you're asking why I have not simply done something like
Code:
esp equ rsp    
instead of customizing instruction encoder - then you are right, I could do it this way (and fasm 1 variant of these macros does it exactly like that), but with the macro approach there are more advanced tricks that can be done, for example getting rid of 67h prefixes that I also demonstrated.
Post 13 Jan 2019, 07:52
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 321
Location: Russian Federation, Sochi
_esp equ rsp - not esp equ rsp, I mean make replacement only in places where it needed (in square brackets for example). And field extending to qword only in places were it needed too (In assembler core there is no need to extend structure fields because of 32bit logic of it) i mean extend fields only in places where assembler need to communicate with x86/x64 environment.
Post 13 Jan 2019, 09:36
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 7299
Location: Kraków, Poland
ProMiNick wrote:
_esp equ rsp - not esp equ rsp, I mean make replacement only in places where it needed (in square brackets for example).
This is why fasmg macro modification in instruction encoder is more robust - it is able to modify things only in specific circumstances (like for square-bracketed addresses of specific type) and still apply it all without touching original 32-bit source files.

Because as soon we start modifying the original sources and effectively make a fork, then why not go all the way?
ProMiNick wrote:
And field extending to qword only in places were it needed too (In assembler core there is no need to extend structure fields because of 32bit logic of it) i mean extend fields only in places where assembler need to communicate with x86/x64 environment.
Once you allow 64-bit pointers to sneak in from OS (malloc), you need to modify all the structures that include pointers, and then registers that are used to transfer values to/from these fields - and this becomes quite a substantial modification to fasmg's core.

The idea I had originally was to define things like:
Code:
rbx equ ebx ; for 32-bit version
; for 64-bit version no definition needed    
and
Code:
struc dr? values&
    . dd values ; for 32-bit version
end struc    
or
Code:
struc dr? values&
    . dq values ; for 64-bit version
end struc    
And then adapt all structures to use DR for all fields than contain a pointer, and adapt entire code to use registers like RBX instead of EBX when transferring pointer values (in 32-bit version RBX would end up being assembled as EBX anyway).

But this would be a lot work (compared to the trick I used) for probably a very little gain (in 64-bit version the structures would grow substantially, so cache problems would become worse and overall performance might suffer). And this still would not be taking advantage of other things long mode offers, like additional registers, ability to process data in larger units, etc.
Post 13 Jan 2019, 10:24
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 321
Location: Russian Federation, Sochi
I still trying to maintain fasm&fasmgAllInOne package (trying to collect every ideas & examples related to fasm & fasmg in one place), and evolve it. But some portions stoped to work due to fasm&fasmg updates that are out of concept.
In parallel I trying to add fasmarm to it.
I found a bug in kolibry part of my packege - I fix it, but later I found disadvantage in fasm adapted to menuet & kolibry - they couldn`t work effectively with includes located somewhere (every include is relative to current dir of current source or it has full path, and can`t be relative to variable include dir, because of absence of initialisation of such variable - so this is a bug of my package too.)
I don`t upload package for a long long time because I can`t make everything work and in same time can`t fix all bugs found earlier.
Macro porting is out of my concept - so I have to drop fasmg x64 linux.
Post 13 Jan 2019, 12:07
View user's profile Send private message Send e-mail Reply with quote
Tomasz Grysztar
Assembly Artist


Joined: 16 Jun 2003
Posts: 7299
Location: Kraków, Poland
Are your problems caused by some modifications to fasmg core files? That is, by forking it?

In principle, when porting fasm[g] to another OS, you should not have to modify the core files. That was my general idea that seemed to work well in case of fasm 1 and with fasmg it is better developed, I hope.

I hinted at this when I was showing stripping down to "raw core" of fasmg during my first session. I may dedicate another session to this topic in the future - perhaps port to some less-mainstream OS, or maybe just make a DOS/DPMI port - to demonstrate the idea how separation of "core" and "interface" works in fasmg (which is a refinement of the same idea used by fasm 1).

I could also talk about fedit in relation to this topic - it shows how the idea of OS-independent core may work even in case of something requiring a UI, like text editor.

Note how introduction of a new "interface" (like fasmg.x64 discussed here) does not then affect the other ones, because "core" remains unchanged. All the rules how to interface with core stay as they were and nothing is broken just because we want to make a new interface.

Currently there are only console interfaces for fasmg (and one library interface in form of DLL), though in theory we could make a UI one (analogous to fasmw, perhaps also based on fedit) too. And again, core would remain "as is" - and it could still be developed independently of all the different interfaces, as long as all interfaces followed the documented rules.

During these recorded sessions I was pointing out how important it is to strictly follow all the conventions documented in source (and at the same time, how important it is to document them in detail), for this reason exactly. You should rely only on what is documented and not make any assumptions outside of that.

BTW... (please forgive me all the digressions, there are so many interesting directions to follow here...)
Since I mentioned library interface - in principle it should also be possible to make .so/PIE version of fasmg. When all the variables used by fasmg are given EBP-based addresses, like DLL version already does, the entire core effectively becomes position-independent.
Post 13 Jan 2019, 13:23
View user's profile Send private message Visit poster's website Reply with quote
ProMiNick



Joined: 24 Mar 2012
Posts: 321
Location: Russian Federation, Sochi
Yes I forked core a little (removed format binary as built in) and macros that could be say that they are forked - alterencoding - is mainencoding in my realization - sometimes it conflicts with something that come with update. And I restructured includes.
But all is dosn`t matter: What convinient to me, and what not. With task of increasing frendliness of fasm for kolibry & menuet I will do it self.

Tomasz, all of that what you creating is cool and interesting.
Especialy I glad to read that in plans fasmg .so.
Post 13 Jan 2019, 14:24
View user's profile Send private message Send e-mail Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2019, Tomasz Grysztar.

Powered by rwasa.