flat assembler
Message board for the users of flat assembler.

Index > Projects and Ideas > fasm 2 as a set of scripts and headers for fasmg

Author
Thread Post new topic Reply to topic
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 11 Jul 2023, 22:31
After updating my advanced x86 encoder prototype I decided to finally combine it with the existing set of fasm compatibility headers and demonstrate the vision I had for fasm 2.

This is a self-contained package, which includes the fasmg binary, set of include files and "fasm2" script that launches fasmg with the x86 instruction sets and formats automatically included (and the environment set up with headers like "win32ax.inc" available if sources need them).

This provides a non-trivial level of compatibility with fasm 1 out of the box. As long as the sources do not use the {} syntax of fasm's preprocessor, there is a high chance they can be assembled with this prototype (and in some simple cases it is enough to replace braces with appropriate statements like "end rept" or "end macro", but proceed with caution). In fact, both FASMD.ASM and FASMW.ASM from the standard fasm 1 distribution can be assembled with no changes and no additional setup.

That being said, it is just a prototype that I expect to fail in many ways when confronted with new environments and source bases. It has only Win32 and x64 Linux versions of fasmg binaries, so if you need a different one for your system, you should get the complete fasmg package separately.

The purpose of this preview is to demonstrate what I envisioned fasm 2 could be. I do not promise to keep working on it.

Update: the preview can now be downloaded from the official page.


Last edited by Tomasz Grysztar on 02 Feb 2024, 17:14; edited 1 time in total
Post 11 Jul 2023, 22:31
View user's profile Send private message Visit poster's website Reply with quote
Roman



Joined: 21 Apr 2012
Posts: 1796
Roman 12 Jul 2023, 13:03
fasm 2 this is fasmg or grounded on fasmg ?
What difference fasm 2 and fasmg ?
Post 12 Jul 2023, 13:03
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 12 Jul 2023, 18:37
Roman wrote:
fasm 2 this is fasmg or grounded on fasmg ?
What difference fasm 2 and fasmg ?
This fasm2 package is an x86 assembler written in language of fasmg (mostly in sub-language of CALM). It therefore runs on top of it: fasmg is the engine, fasm2 is a scripted x86 assembler running on that engine.

In this regard it is similar to the previous x86 package for fasmg that provided fasm compatibility, but this one is based on the new x86-2 encoder. It is in fact a little slower than that basic x86 implementation, but is more flexible, allowing to switch several encoding options and choose target CPU extensions on the fly. For example:
Code:
; Restrict to 8086 instructions:

use i86

org 100h
        ; ...
        int     20h    
In addition to USE, it is also possible to chose encoder options and enable CPU features for individual instructions:
Code:
{i386}  add eax,ebx    
By default all the available instruction sets are enabled, for fasm compatibility.

As for the macro sets that emulate fasm's FORMAT directive and Windows headers, they are mostly unchanged from the basic fasm compatibility package, please refer to the other thread for the information on them.
Post 12 Jul 2023, 18:37
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 13 Jul 2023, 20:24
As a demonstration of what a transitional project could look like, I simplified the source code of fasmg into a form that can be assembled with fasm or fasm2 (except for the Mach-O variant, which is unsupported by fasm). Almost everything is directly compatible, except for tricky variants (x64, DLL) that require special macros (there I still used the trick to include either the .INC variant for fasm, or .ALM variant for fasm2).

Changes from the standard source code of fasmg are limited to subdirectories with specific OS interfaces.
Post 13 Jul 2023, 20:24
View user's profile Send private message Visit poster's website Reply with quote
vityacv



Joined: 27 Oct 2012
Posts: 7
vityacv 16 Jul 2023, 08:26
In include file
\include\equates\gdi64.inc
there seems some extra text on structure definition 'packed'
struct BITMAPFILEHEADER, packed
probably compiler should return error on this?

sample:
Code:
format binary 

Main:
ret

align 16
    

error while compiling:
Code:
Processed: align 16
Error: illegal instruction.
    


If you try to create:
Code:
format PE64 as 'bin'
entry Main
Main:
ret
    

compiler returns:
Code:
macro format?.PE64? [54] 
Custom error: invalid argument.
    


For PE64 format shoud we set flag IMAGE_DLLCHARACTERISTICS_HIGH_ENTROPY_VA as default?

What about no reloc data and ASLR:
Code:
data fixups
dd 0,8 <- can it be created automatically?
end data
    


In PE, MajorLinkerVersion\MinorLinkerVersion probably can fill with 2.0 instead of 0's

"Extended implementation of anonymous labels", will be included later?
Post 16 Jul 2023, 08:26
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 16 Jul 2023, 08:43
vityacv wrote:
In include file
\include\equates\gdi64.inc
there seems some extra text on structure definition 'packed'
struct BITMAPFILEHEADER, packed
probably compiler should return error on this?
PACKED is a keyword that allows to disable alignment checking that is done by the new version of STRUCT. As mentioned above, all of the macros outside of the x86-2 encoder are copied from the fasm compatibility package, so the features that were added there are also present here.

vityacv wrote:
What about no reloc data and ASLR:
Code:
data fixups
dd 0,8 <- can it be created automatically?
end data
    
As it stands, fasmg's PE formatter differs from fasm 1 in how it generates the fixups table, so it should not have the same issues. And of course, it is much easier to tweak when needed.

vityacv wrote:
"Extended implementation of anonymous labels", will be included later?
That's an easy addition, done.
Post 16 Jul 2023, 08:43
View user's profile Send private message Visit poster's website Reply with quote
vityacv



Joined: 27 Oct 2012
Posts: 7
vityacv 23 Jul 2023, 06:05
Works perfectly with latest changes, thanks!

There's another sample that doesn't compile
Code:
format binary
use64 

struct IMAGE_DOS_HEADER 
e_magic rw 1
e_cblp rw 1 
e_cp rw 1 
e_crlc rw 1 
e_cparhdr rw 1
e_minalloc rw 1 
e_maxalloc rw 1 
e_ss rw 1 
e_sp rw 1 
e_csum rw 1 
e_ip rw 1 
e_cs rw 1 
e_lfarlc rw 1 
e_ovno rw 1 
e_res rw 4
e_oemid rw 1
e_oeminfo rw 1
e_res2 rw 10
e_lfanew rd 1 
ends

teste IMAGE_DOS_HEADER 'MZ',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,testq.size

testq:
mov rax,1 
ret 
align 4 
testq.size = $-teste
    

If you try to use
Code:
teste IMAGE_DOS_HEADER 'MZ',0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
IMAGE_DOS_HEADER [2] struct?.instantiate [62]
Custom error: value too long to fit in e_magic.
    

if I fill e_magic with 0 but e_lfanew with value
Code:
teste IMAGE_DOS_HEADER 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,testq.size
Error: not enough memory to complete the assembly.
    


if you use 0's as initializer it compiles
Code:
teste IMAGE_DOS_HEADER 0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0,0
    
Post 23 Jul 2023, 06:05
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8354
Location: Kraków, Poland
Tomasz Grysztar 23 Jul 2023, 07:54
The version of STRUCT that is currently included in the compatibility package executes the data directive with values in the arguments. This works with any data types that contain the value in the statement (without knowing what they might be - as opposed to the limited catalogue of supported data statements in fasm'1 version of macro), but cannot work with statements like RW where the argument is the size and not the value.

As discussed in the thread about STRUCT there are various directions we could go when implementing it. I've chosen this variant when building the package because it is the most robust one I have available for fasmg right now.
Post 23 Jul 2023, 07:54
View user's profile Send private message Visit poster's website Reply with quote
vityacv



Joined: 27 Oct 2012
Posts: 7
vityacv 04 May 2024, 18:17
Is it possible to speedup build time in these cases?

Bin file '128mb.bin'
flat assembler version g.kd3c
2 passes, 24.4 seconds, 134547968 bytes.

rq 20000000
db 1
flat assembler version g.kd3c
3 passes, 28.1 seconds, 160001536 bytes.
Post 04 May 2024, 18:17
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4043
Location: vpcmpistri
bitRAKE 04 May 2024, 22:09
vityacv wrote:
Is it possible to speedup build time in these cases?

Bin file '128mb.bin'
flat assembler version g.kd3c
2 passes, 24.4 seconds, 134547968 bytes.

rq 20000000
db 1
flat assembler version g.kd3c
3 passes, 28.1 seconds, 160001536 bytes.
Address spaces are optimized to be used in another manner:
Code:
repeat 1 shl 20
        virtual at 0
        A#%::
                db 'A'
                rb 1 shl 30
        end virtual
end repeat

vbytes = 0
repeat 1 shl 20
        vbytes = vbytes + sizeof A#%
end repeat

repeat 1,V:vbytes
        display `V,' virtual bytes',10
end repeat    
Code:
flat assembler  version g.kd3c
1125899907891200 virtual bytes

1 pass, 2.4 seconds, 0 bytes.    
... i.e. only the tail should be reserved space. As soon as literal data is encoded there is an assumption that the whole address space should be literally present for use by LOAD/STORE/et al.

There are probably several ways to accomplish the larger task - if this is a problem in your work.

It's not just "virtual" address spaces:
Code:
N := 20
lbytes = 0
vbytes = 0
repeat 1 shl N
        A#%: ; could use $$ instead of A#%
        dd %
        rb 1 shl 30
        lbytes = lbytes + $%% - A#%
        vbytes = vbytes + $ - A#% - 4
        section $%% ; new address area
end repeat

repeat 1,L:lbytes,V:vbytes
        display `L,' literal bytes',10
        display `V,' virtual bytes'
end repeat
restartout ; don't write data    

_________________
¯\(°_o)/¯ “languages are not safe - uses can be” Bjarne Stroustrup
Post 04 May 2024, 22:09
View user's profile Send private message Visit poster's website Reply with quote
vityacv



Joined: 27 Oct 2012
Posts: 7
vityacv 06 Jul 2024, 10:01
Is struct align supported?
Code:
include 'win64w.inc'                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                
format pe64                                                                                                                                                                                                                                                                     
entry Main                                                                                                                                                                                                                                                                      
                                                                                                                                                                                                                                                                                
struct tests                                                                                                                                                                                                                                                                    
a1 rd 1                                                                                                                                                                                                                                                                         
rd 1                                                                                                                                                                                                                                                                            
a2 rw 1+1                                                                                                                                                                                                                                                                       
rd 1                                                                                                                                                                                                                                                                            
a3 rd 1                                                                                                                                                                                                                                                                         
rd 1                                                                                                                                                                                                                                                                            
ends                                                                                                                                                                                                                                                                            
                                                                                                                                                                                                                                                                                
;struct tests                                                                                                                                                                                                                                                                   
;a1 rd 1                                                                                                                                                                                                                                                                        
;align 8                                                                                                                                                                                                                                                                        
;a2 rw 1                                                                                                                                                                                                                                                                        
;align 8                                                                                                                                                                                                                                                                        
;a3 rd 1                                                                                                                                                                                                                                                                        
;align 8                                                                                                                                                                                                                                                                        
;ends                                                                                                                                                                                                                                                                           
                                                                                                                                                                                                                                                                                
Main:                                                                                                                                                                                                                                                                           
virtual at 0                                                                                                                                                                                                                                                                    
.e rd 1                                                                                                                                                                                                                                                                         
.Check tests                                                                                                                                                                                                                                                                    
end virtual                                                                                                                                                                                                                                                                     
mov [rsp+.Check.a3],0 ;unexpected, got -4 if align used                                                                                                                                                                                                                              
ret 
    
Post 06 Jul 2024, 10:01
View user's profile Send private message Reply with quote
bitRAKE



Joined: 21 Jul 2003
Posts: 4043
Location: vpcmpistri
bitRAKE 06 Jul 2024, 20:37
Do you want to align relative to the virtual space or relative to the structure start?
Code:
; Note: No error checking, alignment value is assumed to be a power of two >0
; and the base is assumed to support infinite alignment.
calminstruction _align value*,base:$$
        ; directive reserves space when the second parameter is not provided.
        emit (value - (($-base) and (value-1))) and (value-1)
end calminstruction    
... I think the following will produce what you are expecting?
Code:
struct tests
        a1 rd 1
        _align 8,.
        a2 rw 1
        _align 8,.
        a3 rd 1
        _align 8,.
ends    
Explaining why it works is more difficult: The labeled macro instructions have a special use of "." - as the name of the STRUC. The advanced STRUCT macros use labeled macros in the form of:
Code:
calminstruction (instance) name values&
        call struct?.instantiate, instance, sname, values
end calminstruction    
... hope that helps.

_________________
¯\(°_o)/¯ “languages are not safe - uses can be” Bjarne Stroustrup
Post 06 Jul 2024, 20:37
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.