flat assembler
Message board for the users of flat assembler.

Index > Main > flat assembler 1.71.00-01

Goto page 1, 2, 3, 4  Next
Author
Thread Post new topic Reply to topic
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 20 Sep 2012, 23:39
I open the new development line today, with a release that introduces a new feature that breaks another fasm's limitation that has been troubling me for a long time. Initially I planned it only for fasm 2 (and you may have seen it mentioned on my presentation on fasmcon 2009) because I thought it would be too much work to implement it in fasm 1.x. However, after earlier this year I managed to overcome problems with implementing 65-th bit for expression calculations (which was a similar story) I realized that this one is also within my reach. And in fact this one required much less work overall.

So, what is it about? The LOAD and STORE directives. While they allowed to tweak with bytes in assembled code, they could operate only within the boundaries of current addressing space - and this was very limiting (it led to ugly solutions like re-defining the same data in virtual block again and again just to be able to load another chunk from it).

So, the syntactic idea I had (and presented on fasmcon 2009) was to allow to make a special kind of label, which would identify the addressing space - and then use this label with LOAD/STORE to specify, in which addressing space the operation should be performed. Such label is distinguished by the fact that it is followed with double colon instead of single one, and it can only be used in the addressing part of LOAD/STORE syntax, which now can have the "space:address" form. The old plain "address" form is also still allowed - it operates on current addressing space, as always.

This allows, for example, to label the virtual block and load the data from it even from the outside:
Code:
virtual at 10h
  my_space::
    db 'ABCD'
end virtual

load a byte from my_space:11h
display a    

Of course only the code that has already been generated can be accessed this way, so addressing spaces cannot be forward-referenced and when loading from current addressing space, only the bytes generated up to the current point can be accessed.

Other examples may include rewriting some parts of data section in PE format by the operations specified in the later code section.

The fact that virtual blocks can now be accessed from the outside allows to have assembly-time storage buffers, it is even possible to load some external file into virtual block and then access any portion of it at any time.
One of the first places where I utilized this feature were the encoding include files (in INCLUDE/ENCODING directory) - I could get rid of the repeating of definitions, conversion tables and string data can now reside in easily accessible virtual buffers.

The other obvious application is to write back into data section values gathered later in the code. This requires that the data section has already the required amount of bytes reserved, but this can be accomplished with forward-referenced size counter. The STORE directive needs to be placed inside the IF block that ensures we only try to write the data when the allocated space is already of the enough size (STORE would fail with "value out of range" error otherwise). I prepared a simple example of how to do it:
Code:
 __str__section::
 __str__data db __STR__DATA_SIZE dup 0
 __str__current = 0

struc str [data] {
 common local buffer,size,a
  virtual at 0
    buffer:: db data
    size = $
  end virtual
  label . at __str__data + __str__current
  __str__current = __str__current + size
  if __str__current <= __STR__DATA_SIZE
    repeat size
      load a byte from buffer:%-1
      store byte a at __str__section:.+%-1
    end repeat
  end if
}

; ...

some_message str 'Testing one',0

other_message str 'Doing something...',13,10

__STR__DATA_SIZE = __str__current    
The storage area is defined on top (it is "__str__data"). The "__str__section" label is defined to allow these macros to work even when put into different sections. The later string definitions are written into storage area as soon as it is allocated with enough space (and the required size is defined as "__STR__DATA_SIZE", which can be forward-referenced).

This simple solution gathers only strings that are defined after the storage area. With a few preprocessor tricks we can make the set of macros that will allow to define strings anywhere and put the storage area also in any place we like:
Code:
struc str [data] {
 common local ..buffer,size,a
  virtual at 0
    ..buffer:: db data
    size = $
  end virtual
  label . at __str__data + __str__current
  __str__current = __str__current + size
  if __str__current <= __STR__DATA_SIZE
    repeat size
      load a byte from ..buffer:%-1
      store byte a at __str__section:.+%-1
    end repeat
  end if
}

define __str__cnt 0

struc str [data] {
 common
   rept 1 i:__str__cnt+1 \{
     define __str__cnt i
     define __str__\#i . str data
   \}
}

macro __str__previous {}

macro .str {
  __str__section::
  __str__data db __STR__DATA_SIZE dup 0
  __str__current = 0
  restruc str
  rept __str__cnt i \{ match def,__str__\#i \\{ def \\} \}
}

macro .end {
  __STR__DATA_SIZE = __str__current
}

some_message str 'Testing one',0

.str

other_message str 'Doing something...',13,10

.end    
Here the ".str" macro decides where to put the storage area, while "str" definitions can be put anywhere in the source. The ".end" macro must come at the end of source so that the total string data size can get defined correctly.

There are many more interesting applications of these features. The other place where I was missing this feature was when trying to implement some compression algorithms in fasm's assembly-time language - now it should be possible, I may give it a try later.

PS Please remember that this is an early development release, so it may contain some bugs and I do not guarantee that everything that assembled well with 1.70 will be assembled correctly by this one. Though I did my best to ensure that.
Post 20 Sep 2012, 23:39
View user's profile Send private message Visit poster's website Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 20 Sep 2012, 23:45
One more example, which is a refinement of the string gathering macros from above. It implements a Boyer-Moore-Hornspool algorithm to scan the string storage for an occurrence of each new strings it gets to define. If it succeeds, it points it to the existing data, so even repeated strings are defined only once.
Code:
virtual at 0
  __str__character_skips::
  dd 256 dup ?
end virtual

struc str [data] {
 common local ..buffer,size,a,b,c,p,skip,found

  virtual at 0
    ..buffer:: db data
    size = $
  end virtual

  times 256 store dword size at __str__character_skips:(%-1)*4
  repeat size-1
    load a byte from ..buffer:%-1
    store dword size-% at __str__character_skips:a*4
  end repeat

  found = -1
  p = 0
  while p + size <= __str__current & p + size <= __STR__DATA_SIZE
    c = size
    while c > 0
      load a byte from __str__section:__str__data+p+c-1
      load b byte from ..buffer:c-1
      if a<>b
        c = -1
        break
      end if
      c = c - 1
    end while
    if c = 0
      found = p
      break
    else
      load a byte from __str__section:__str__data+p+size-1
      load skip dword from __str__character_skips:a*4
      p = p + skip
    end if
  end while

  if found >= 0
    label . at __str__data + found
  else
    label . at __str__data + __str__current
    __str__current = __str__current + size
    if __str__current <= __STR__DATA_SIZE
      repeat size
        load a byte from ..buffer:%-1
        store byte a at __str__section:.+%-1
      end repeat
    end if
  end if
}

define __str__cnt 0

struc str [data] {
 common
   rept 1 i:__str__cnt+1 \{
     define __str__cnt i
     define __str__\#i . str data
   \}
}

macro __str__previous {}

macro .str {
  __str__section::
  __str__data db __STR__DATA_SIZE dup 0
  __str__current = 0
  restruc str
  rept __str__cnt i \{ match def,__str__\#i \\{ def \\} \}
}

macro .end {
  __STR__DATA_SIZE = __str__current
}


; Usage example:

some_message str 'Testing one',0

.str

other_message str 'Doing something...',13,10

        mov     eax,some_message
        mov     ebx,submessage
        mov     ecx,other_message

submessage str 'one',0

.end    
I think it works, though it may contain some mistakes. It is late already...
Post 20 Sep 2012, 23:45
View user's profile Send private message Visit poster's website Reply with quote
cod3b453



Joined: 25 Aug 2004
Posts: 618
cod3b453 21 Sep 2012, 00:06
Shame I don't have time to play with this now but I have a feeling this is going to both greatly simplify some of my macros and make even more of the ones I was too scared to write possible Laughing.

Thank you, Tomasz Cool
Post 21 Sep 2012, 00:06
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 01:41
Tomasz Grysztar
Thank you very much. The feature is very cool. And special address space labeling is much better then giving a name to an addressing space as I suggested. But I have a couple of comments.

1. Wouldn't it be faster to copy strings with qwords ? In most cases I try to do like this (a fragment of a macro. Not compilable as is):
Code:
repeat sizeof.string/8
 virtual
             db string
           load qw1 qword from $$+(%-1)*8
      end virtual
 store qword qw1 at lbl+(%-1)*8
end repeat
repeat sizeof.string mod 8
  virtual
             db string
           load c1 byte from $$+(sizeof.string-sizeof.string mod 8)+(%-1)
      end virtual
 store byte c1 at lbl+(sizeof.string-sizeof.string mod 8)+(%-1)
end repeat    


2.
Quote:
The ".end" macro must come at the end of source so that the total string data size can get defined correctly.

I don't think there's a need in the ".end" macro. There's a simple trick to detect the first and the last macro invocation:
Code:
macro def_Foo forwardref*, backwardref*
{
      macro Foo [args]
    \{
            \common \local ..forwardref,..backwardref
                 if ~ defined backwardref                                                ;if the previous macro expansion does not exist
                             display 'This is the first Foo invocation',13,10
                  end if
                      if ~ defined ..forwardref                                               ;if the following macro expansion does not exist
                            display 'This is the last Foo invocation',13,10
                   end if
                      label forwardref                                                                ;this one is referenced by the previous macro expansion
                     label ..backwardref                                                             ;this one will be referenced by the following macro expansion
                       purge Foo
                   def_Foo ..forwardref,..backwardref
  \}
}
macro Foo {}
match ,
{
   local ..forwardref,..backwardref                        ;..forwardref will be defined
       def_Foo ..forwardref,..backwardref                      ;..backwardref will referenced
}

display '0',13,10
Foo
display '1',13,10
Foo
display '2',13,10
Foo
display '3',13,10
Foo
display '4',13,10    


3. There's a bug in the new version. You seem to forget to consider non-zero bases of non-virtual addressing spaces:
Code:
org 10h
db 'ABCD'
load a byte from 11h ;Fails
display a,13,10    
Post 21 Sep 2012, 01:41
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20363
Location: In your JS exploiting you and your system
revolution 21 Sep 2012, 02:35
Now one extra thing to consider is the namespace collisions that can (will?) occur when incorporating code from other sources (or even in the same file if one forgets to make a new name).
Code:
;main file
virtual at 0
  buffer::
  dd 256 dup ?
end virtual
include 'some_libray.inc'    
Code:
;inside some_libray.inc
virtual at 0
  buffer::
  db 16 dup ?
end virtual    
Now what happens there?

The same problem exists with JS and function name collisions. Perhaps the "solutions" that were used there can be useful here? I'm not sure.
Post 21 Sep 2012, 02:35
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 03:26
revolution
Nobody canceled anonymous and local labels. Thus the following still works:
Code:
@@::
db 'A'
virtual
        primary::
   db 'B'
    load x byte from @B:0
   display x,13,10
     .virtual::
end virtual
secondary::
load x byte from primary.virtual:1
display x,13,10    


Therefore the addressing space labels do not introduce any more collision problems, than normal labels. Besides normal global labels become prefixes for the local addressing space labels (and vise versa), which makes local addressing space labels kind of local inside procedures. As usual you can always define a unique label by using the local directive.
Post 21 Sep 2012, 03:26
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20363
Location: In your JS exploiting you and your system
revolution 21 Sep 2012, 04:03
l_inc wrote:
As usual you can always define a unique label by using the local directive.
local is only valid inside macro/struc. It won't help for things like addressing across section separation.

But thanks for the info regarding the naming, I'll look into it in more detail later.
Post 21 Sep 2012, 04:03
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 04:32
revolution
Quote:
local is only valid inside macro/struc.

As well as any other macroblock like match, rept, irp, irps.
Quote:
It won't help for things like addressing across section separation.

Why not? The following code perfectly works across section separation, and there is no collision of the ..space label with itself.
Code:
format PE GUI 4.0

include 'win32a.inc'

entry start

struc defdata [data]
{
 common local ..space, buf
   ..space:: .: db data
    .size = $-.
 
    macro .copy
 \{
            repeat .size
                        load buf byte from ..space:.+(%-1)
                      db buf
              end repeat
  \}
}

section '.text' readable executable
     proc start

      ret
 endp
        hello defdata 'Hello '
    world defdata 'world!'
    
section '.data' readable writable
     hello.copy
  world.copy    

Sure it's not the perfect way to create a copy.
Post 21 Sep 2012, 04:32
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria
JohnFound 21 Sep 2012, 05:30
Great new feature! I will use and test it ASAP with FreshLib, because it seems to solve the long waiting problem with unique string definitions and can make the data processing really easy.

Thanks Tomasz!
Post 21 Sep 2012, 05:30
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 21 Sep 2012, 07:34
l_inc wrote:
1. Wouldn't it be faster to copy strings with qwords ?
You are of course right. But these are just simple examples, to show the principle. I stitched them up quickly and tried to keep them as simplified as possible.

l_inc wrote:
There's a bug in the new version. You seem to forget to consider non-zero bases of non-virtual addressing spaces: (...)

The ORG directive was not working at all (well, except when with zero base). I uploaded the updated package, which fixes that among a few other things.
Post 21 Sep 2012, 07:34
View user's profile Send private message Visit poster's website Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3499
Location: Bulgaria
JohnFound 21 Sep 2012, 07:49
I tested FASM 1.71 with Fresh and it works OK.
Post 21 Sep 2012, 07:49
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
hopcode



Joined: 04 Mar 2008
Posts: 563
Location: Germany
hopcode 21 Sep 2012, 09:15
dear Coherence,
because on the manual at
http://flatassembler.net/docs.php?article=manual#2.2.4
it speaks "indipendent addressing space" about virtual
AND
it speaks "expected to appear in memory ...(the following
code itself is not moved in any way)..." about org
AND
it speaks "...from the already assembled code..." about load/store,
directives,
wouldnt it be simply better to consolidate/extend the load/store things
even on not yet assembled code, instead of creating such
new feature that, at least for now, seems to assert just now, completely
and openly again, the weakness of virtual ?

yes, because all of the examples above, AFAIK, are already doable in a more simple manner
on 1.69. also, i am curious now to see who among you will be capable to skip my boring post Wink in the usual way.
and just because
when i say virtual i would like to say "indipendent addressing space"
and when i say org i would like to say "responsibility of programmer to put
the code at correct address at run-time." also when
i say load i would like always to operate on a well known addressing space.
even if i do no generate code for it.
ok, focusing few seconds carefully the following code,
Code:
virtual at 10h
  my_space:
    db 'ABCD'
end virtual

load a byte from my_space:11h
display a
    
and the code posted above again
Code:
virtual at 10h
  my_space::
    db 'ABCD'
end virtual

load a byte from my_space:11h
display a
    

it will be yet harder (quite impossible) to convince me
to update to 1.70. but i will be very tankful to that person
who succeeds, because i would really like that update.

Cheers,

_________________
⠓⠕⠏⠉⠕⠙⠑
Post 21 Sep 2012, 09:15
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 11:42
Tomasz Grysztar
One more bug is related to the register based virtual addressing spaces:
Code:
virtual at ebx
        space::
        db 'ABCD'
        load a byte from ebx+1       ;works
        load b byte from space:$$+1  ;works
        load c byte from space:ebx+1 ;fails
        display a,b,c,13,10
end virtual    

And this one (with forward referencing):
Code:
virtual at ebx+esi
    db 'ABCD'
    load a byte from $$+1       ;works
    load b byte from space:$$+1 ;fails
    space::
end virtual
display a,b,13,10    
Post 21 Sep 2012, 11:42
View user's profile Send private message Reply with quote
shutdownall



Joined: 02 Apr 2010
Posts: 517
Location: Munich
shutdownall 21 Sep 2012, 11:58
I maybe plan to upgrade my Z80 version to this new release.
Just a simple question.
You load only single bytes from virtual block to somewhere.
That's not very convenient.

How can I move the complete block with all data ?
Is it possible ? In a loop ?


Code:

virtual at 2000h
 myspace::

db 'this is a very long text to transfer but could hold even instructions and much more data definitions'

end virtual

org somewhere

load a byte from myspace:0
store byte a at $$
     


And can do this in a loop for every byte defined in the virtual address space ?

Could someone provide a working example ?

This would offer new ways of creating code. Cool
Post 21 Sep 2012, 11:58
View user's profile Send private message Send e-mail Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 12:03
shutdownall
Quote:
Is it possible ? In a loop ?

Yes.
Quote:
Could someone provide a working example ?

This topic is full of examples. Look at the second snippet in the first post.
Post 21 Sep 2012, 12:03
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 8356
Location: Kraków, Poland
Tomasz Grysztar 21 Sep 2012, 12:11
l_inc wrote:
One more bug is related to the register based virtual addressing spaces:
Code:
virtual at ebx
        space::
        db 'ABCD'
        load a byte from ebx+1       ;works
        load b byte from space:$$+1  ;works
        load c byte from space:ebx+1 ;fails
        display a,b,c,13,10
end virtual    

Oh, this is a parsing error. Note that if you write "space:+ebx+1" instead of "space:ebx+1", it works like a charm. This reveals that the problem lies in parsing stage.

l_inc wrote:
And this one (with forward referencing):

The addressing space labels cannot be forward-referenced, I wrote about it already. I will try to make the error handling a bit better here.
Post 21 Sep 2012, 12:11
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 21 Sep 2012, 12:20
Tomasz Grysztar
Quote:
The addressing space labels cannot be forward-referenced, I wrote about it already. I will try to make the error handling a bit better here.

You wrote, that the addressing spaces cannot be forward referenced. Not the labels. Thus referencing the current addressing space with the label declared afterwards does not violate this rule.
But thank you. I could make some better testing to understand this.
Post 21 Sep 2012, 12:20
View user's profile Send private message Reply with quote
hopcode



Joined: 04 Mar 2008
Posts: 563
Location: Germany
hopcode 21 Sep 2012, 12:29
i wait eh ? Very Happy
i count sheep in the meanwhile, ok ? Very Happy

_________________
⠓⠕⠏⠉⠕⠙⠑
Post 21 Sep 2012, 12:29
View user's profile Send private message Visit poster's website Reply with quote
AsmGuru62



Joined: 28 Jan 2004
Posts: 1640
Location: Toronto, Canada
AsmGuru62 21 Sep 2012, 12:31
Amazing work, Tomasz!
Post 21 Sep 2012, 12:31
View user's profile Send private message Send e-mail Reply with quote
shutdownall



Joined: 02 Apr 2010
Posts: 517
Location: Munich
shutdownall 21 Sep 2012, 12:35
l_inc wrote:
shutdownall
This topic is full of examples. Look at the second snippet in the first post.


Thanks. It's not easy to immediately understand some complex examples but I will do my very best later. Very Happy
Post 21 Sep 2012, 12:35
View user's profile Send private message Send e-mail Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2, 3, 4  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.