flat assembler
Message board for the users of flat assembler.

Index > Macroinstructions > Is it possible to pass non-numeric to REPT macro? workaround

Author
Thread Post new topic Reply to topic
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
hi all

I have the following code and the problem is that I cannot pass "m_size_in_dwords" to the REPT cycle since FASM says invalid instruction.
So I limited "cipher" section to approx 40 kilobytes ... and the question is: is there really no way to use REPT macro as:
rept m_size_in_dwords x:0
\{
load d_\#x dword from 4*x
\}

Code:
macro m_start name
{
  name:
  virtual at 0
}

macro m_end
{
 local m_end_internal
 local nonvirt_bytes
  align 4
  m_end_internal:
  db 0 ; fake align-end marker

  m_size_in_dwords EQU m_end_internal/4

  rept 10000 x1:0
  \{
    if x1 < m_size_in_dwords
     load d_\#x1 dword from 4*x1
    end if
  \}

  end virtual

 nonvirt_bytes:
  rept 10000 x2:0
  \{
    if x2 < m_size_in_dwords
     dd d_\#x2
    end if
  \}
}

m_start cipher
db 255, 254, 253, 252, 251
db 9000 dup(0xaa)
m_end

m_start cipher2
db 7, 8, 9, 10, 11, 12, 13, 14, 15
m_end


    


P.S. I checked the code above in OLLY and it worked correctly. The only hardcoded limitation is 10000 cycles per REPT invocation

thanks
Post 22 Nov 2015, 23:15
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
What you do is an old and very inefficient way of getting an unknown (at preprocessing time) amount of data out of a virtual addressing space. But in even in those old days this was required only for implementing algorithms possibly having less output than input data, such as compression or digesting. Other algorithms (such as encryption) could have been implemented with more efficient but often complicated in-place data modification.

Now fasm has addressing space labels (see the last two paragraphs at the end of the section "Addressing spaces"), and none of the above is necessary.

P.S. Btw. you still should use in-place data modification, if your algorithm does not become too complicated. Having addressing space labels in virtual addressing spaces reserves compile-time memory that is not regainable. Thus even if you absolutely need these, you should avoid linear increase of unregainable memory by either creating addressing space labels outside of the virtual addressing spaces only or by having a single common virtual addressing space for all invocations of your macros.

_________________
Faith is a superposition of knowledge and fallacy
Post 22 Nov 2015, 23:29
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
l_inc thanks for the answer, I'm not going to invent some encryption algos on my own. Those RSA macros you refer to assume that encrypted section will reside in the code. My intent is to create open and closing macros to prepare structured data inside and encrypt on demand with ability to reject some, calculate crc32, or whatever Smile) it's not a problem to reuse modified RSA-fasm.inc inside Smile the only problem is that the following does not work, however "m_size_in_dwords" is known value while REPT is invoked:
Code:
rept m_size_in_dwords x:0 
    


йThanks
Post 23 Nov 2015, 00:07
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
Quote:
Those RSA macros you refer to assume that encrypted section will reside in the code.

I wasn't referring to any specific macros. I was talking about general approaches. And it doesn't make any difference if it's code or data to be processed.

Quote:
the only problem is that the following does not work, however "m_size_in_dwords" is known value while REPT is invoked

It doesn't work, because m_size_in_dwords is not known, when rept is invoked. To be more precise m_size_in_dwords is only known as a sequence of tokens, but not as a numeric value. rept is a preprocessor directive and m_end_internal is only resolved after the preprocessing, i.e. at the assembly stage. See my short explanation here.

Quote:
My intent is to create open and closing macros to prepare structured data inside and encrypt on demand with ability to reject some, calculate crc32, or whatever

I don't think, there's any common work that could be done in a generic way covering all these "whatever" use cases. Each of these need specific macros.

_________________
Faith is a superposition of knowledge and fallacy
Post 23 Nov 2015, 00:22
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
l_inc in-place data modification is not acceptable for me because mentioned macro can potentially reject all data between opening and closing section, but in-place modification assumes reserving spare bytes in the code. In my approch I'm able to allocate that datablock wherever I like, even on stack in runtime


Last edited by Devel99 on 23 Nov 2015, 00:42; edited 1 time in total
Post 23 Nov 2015, 00:23
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
Quote:
in-place data modification is not acceptable for me because mentioned macro can potentially reject all data between opening and closing section

The problem is that you are talking about abstract macros that only potentially do something, but you will need to make a decision about the implementation approach for each specific macro individually. Value names generation with rept can and should be avoided in any case. The other two approaches are:
1) In-place data modification. Not always possible, but often most efficient.
2) Addressing space labels. Always possible.

To make it clear: what approach is suitable for you is only possible to decide depending on what specific kind of data processing you need.

Quote:
In my approch I'm able to allocate that datablock whereever I like, even on stack in runtime

Well, this is not possible without knowing the data at compile-time. So it should still be encoded somewhere in the program, which makes in-place data modification again possible.

_________________
Faith is a superposition of knowledge and fallacy
Post 23 Nov 2015, 00:42
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
l_inc
Quote:

Well, this is not possible without knowing the data at compile-time. So it should still be encoded somewhere in the program


Sure, it should be encoded, it resides between opening and closing sections. For example the code below allocates required blocks on stack

Code:
format PE console
include 'win32ax.inc'

macro m_start name
{
  name:
  push eax
  mov eax, esp
  virtual at 0
}

macro m_end
{
 local m_end_internal
 local nonvirt_bytes
  align 4
  m_end_internal:
  db 0 ; fake align-end marker

  m_size_in_dwords EQU m_end_internal/4

  rept 20000 x1:0
  \{
    if x1 < m_size_in_dwords
     load d_\#x1 dword from 4*x1
    end if
  \}

  end virtual

 nonvirt_bytes:
  rept 20000 x2:0
  \{
    if x2 < m_size_in_dwords
     push dword d_\#x2
    end if
  \}

}

start:

m_start cipher
db 0xff, 0xfe, 0xfd, 0xfc, 0xfb
db 10 dup(0xaa)
m_end

 ; use data on stack [esp...eax]
 mov esp, eax
 pop eax

m_start cipher2
db 7, 8, 9, 10, 11, 12, 13, 14, 15
m_end

 ; use data on stack [esp...eax]
 mov esp, eax
 pop eax


start_next:
        invoke  ExitProcess, 0

.end start
    


Description:
Filesize: 24.98 KB
Viewed: 5367 Time(s)

wefewrgergh.png


Post 23 Nov 2015, 01:12
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
l_inc are there any FASM compiler memory limitations? Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun?
Post 23 Nov 2015, 01:20
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17662
Location: In your JS exploiting you and your system
revolution
Eliminate the REPT and replace with REPEAT and use an address space.

REPT cannot ever know the length of the code because the length is obtained only after REPT has long since finished doing its thing. You will need reverse causality to work if you want to use REPT for that.
Post 23 Nov 2015, 01:24
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17662
Location: In your JS exploiting you and your system
revolution
Devel99 wrote:
... are there any FASM compiler memory limitations? Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun?
There is a modification available for the console version to use 3GB.

See here: http://board.flatassembler.net/topic.php?t=17667
Post 23 Nov 2015, 01:26
View user's profile Send private message Visit poster's website Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
revolution cool article. I have 32GB on board, I pray that one day FASM will become a 64-bit application Cool
Post 23 Nov 2015, 01:50
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
Quote:
Sure, it should be encoded, it resides between opening and closing sections.

More specifically it resides inside of the push instructions. Your implementation is ridiculously slow and inefficient. Here's how the macros are supposed to be coded efficiently:
Code:
format PE console
include 'win32ax.inc'

match,
{
    local ..start
    macro m_ \{ ..start: \}

    macro _m
    \{  
        local ..as,cbMem,cbCod,buf
        cbMem = $ - ..start
        rb cbCod-cbMem
        ..as::
        virtual
            repeat cbMem/4+1
               load buf dword from ..as:..start+(%-1)*4
               push buf 
            end repeat
            cbCod = $-$$
            repeat cbCod
               load buf byte from $$+%-1
               store byte buf at ..as:..start+%-1
            end repeat
        end virtual
    \}  
}


start:
   push ebp
   mov ebp,esp
      m_
         db $FF,$FE,$FD,$FC,$FB
         db 10 dup $AA 
      _m
   leave
   ret 
.end start    

The above code is rather a proof of concept for in-place modification than the final version for reasons mentioned below.

I made a couple of simplifications and used both in-place data modification and virtual addressing spaces for educational purposes here. So there's some room for optimization.
It should be noted however that the optimized form of the push instruction can use less space than the data it puts onto the stack, in which case you'd get a compilation error. This would be indeed the data compression case I was talking about before. And a good solution would be to put the data into a virtual block and an addressing space label outside of the virtual block, so that you store the instructions to the outside from within the virtual addressing space (same as shown in the example actually).

Quote:
Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun?

There isn't much more fun you can get with fasmw, cause its address space is polluted with GUI libraries. If you want to push usable memory to the limits you'd need to use the console version of fasm. Anyway my point was about efficient macro coding, not about polluting as much memory as possible with sloppy coded macros.

_________________
Faith is a superposition of knowledge and fallacy
Post 23 Nov 2015, 02:05
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
l_inc Thanks for the answer and for the good sample code. Wink But I don't think your solution is flexible. It's good enough just for proof of concept only. Let's try to comment out your "push" Rolling Eyes and then try to comment out "push" in previously suggested code with a fixed hardcoded rept cycles. Obviously you'll need to track rb XXX size, so after changing code logic you'll have two bound code chunks instead of one. Actually I don't care how sloopy the macro is and how much time and memory it takes to compile that, bacause the only thing which really matters is a byte sequence produced by FASM.
I don't want to discuss pros and cons of in-place modification now Laughing really, but you have to create placeholder first and track its size (in your implementation RB xxx ...) which is not the case with fixed rept implementation
Post 23 Nov 2015, 14:31
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
revolution thank you. I tried REPEAT later and it turned out that REPEAT was not quite suitable for my code logic. Sad I appreciate L_INC's effort in helping me to aviod "out of memory" issue but I can afford to add extra 32GB onboard )))
Post 23 Nov 2015, 14:48
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
Quote:
But I don't think your solution is flexible. [...] Let's try to comment out your "push" Rolling Eyes and then try to comment out "push" in previously suggested code with a fixed hardcoded rept cycles.

I don't understand your point. Why would you be willing to comment out push, if the very purpose of the macro is to push things onto the stack? I mean, no single piece of code in the world is resilient against arbitrary commenting out its parts. If you break the code, it breaks. Sure, but what's the point?
Quote:
It's good enough just for proof of concept only.

It's good for any application that does not do any kind of data compression. That's why it was meant to be a proof of concept in this case.
Quote:
Obviously you'll need to track rb XXX size, so after changing code logic you'll have two bound code chunks instead of one.

Again I don't understand the point. Tracking the size of the reserved region is automatic by design. What kind of logic change do you mean? There are no "two bound code chunks" (whatever that means) to be produced anyway.

Quote:
but you have to create placeholder first and track its size (in your implementation RB xxx ...) which is not the case with fixed rept implementation

rb is a single little statement. It's simpler, it's shorter, it's faster. What's worrying you about it?

Quote:
Actually I don't care how sloopy the macro is and how much time and memory it takes to compile that, bacause the only thing which really matters is a byte sequence produced by FASM.

This is wrong. It's probably not as important for hello world projects, but you are going to run your head against the wall as soon as your project becomes just a little larger will take minutes and hours to compile and eventually will eat up more memory than available, so that you'll be sitting and hunting down those damn macros preventing fasm from producing any sequence of bytes.

Quote:
I don't want to discuss pros and cons of in-place modification now

As I said, in-place modification is not the best way in this case. My code isn't even really in-place modification, because it requires variable amount of additional virtual space (in-place is by definition using constant amount of space), but it shows that in-place would be possible, and it is only a few little modifications away from a perfect off-place solution, which I described after the code. rept is a horrible idea and luckily it's not needed anymore.

Quote:
I appreciate L_INC's effort in helping me to aviod "out of memory" issue but I can afford to add extra 32GB onboard )))

fasm can't use those. It can use 3GB and a bit at most. And even being able to add more memory won't help you to deal with the long compilation times.

_________________
Faith is a superposition of knowledge and fallacy
Post 23 Nov 2015, 15:20
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
revolution l_inc
good morning guys
I played a bit with fixed rept code tonight and actually ran out of memory after declaring 200 blocks and peak FASMW memory load was 2Gb Cool
I have to admit that my idea is not perfect

My next approach is to create private section where a part of code is declared in virt block, fully resides in virt and not reflected to resulting PE executable. That was my first time I tried virt in private, so I stuck with this:
store byte bt111 at as111:st111 + (%-1)
I have address space as111, declared non-virt label (st111) but I get "value out of range"? Why do you think it is possible? Which value is out of range? Shocked

Code:
format PE console

include 'win32ax.inc'

; --- nonvirt addr spc ---
as111::


macro m_start4 name
{
  name:
  mov eax, esp
  virtual at 0
}

macro m_end4 name
{
 local nonvirt_bytes, data_end_internal, code_end_internal

  align 4 

  data_end_internal:
  data_size_in_dwords = data_end_internal/4

  ; prolog template
  push edx
  mov edx, esp
  sub esp, 118

  ; repeating data template
  repeat data_size_in_dwords
    load dt1 dword from (%-1)*4
    push dword dt1
  end repeat

  ; epilog template
  mov esp, edx
  pop edx
  ret

  code_end_internal:
  code_size_in_bytes = code_end_internal - data_end_internal

  repeat code_size_in_bytes
    load bt111 byte from data_end_internal + (%-1)
    store byte bt111 at as111:st111 + (%-1)
  end repeat

  end virtual

; --- nonvirt bytes ---

nonvirt_bytes:
  rb code_size_in_bytes
}


st111:
; --- private section ---
m_start4 cipher1
db 3, 4, 5, 6, 1, 1, 1, 1, 0xfd, 0xfc, 0xfb
db 10 dup(0xaa)
m_end4 cipher1


start:
   call st111

   invoke  ExitProcess, 0

.end start
    
Post 24 Nov 2015, 09:58
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17662
Location: In your JS exploiting you and your system
revolution
Address space labels keep their offsets so your label is not at offset zero but instead at the PE load address in the first section. You can either adjust for that with appropriate +/- values, or shift the label into the virtual block and the offset will follow the block start, or don't make the virtual block at offset zero.
Post 24 Nov 2015, 10:34
View user's profile Send private message Visit poster's website Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc
Devel99
Quote:
My next approach is to create private section where a part of code is declared in virt block

There is no such thing called "private section". And it's unclear what you mean by that, cause the location of your virtual block isn't private or special in any other way.
revolution wrote:
Address space labels keep their offsets so your label is not at offset zero but instead at the PE load address in the first section

This is wrong. Addressing space labels do not keep their offsets. All they keep is sort of identifier of an addressing space they refer to. Hence the fix suggestions aren't suitable.

But the accessible address range for the load/store directives is limited by two values: the base address of the addressing space, which is $$ if you apply it inside of that addressing space, and current address value, which is $ if you apply it at the location of the load/store directive, but inside of that addressing space (meaning: the address right after the last byte in the accessed addressing space stored up until the occurrence of load/store).

The lower bound of your accessible range referenced by as111 equals to st111 (as well as name), so no problem here. But the upper bound of the accessible range is just two bytes larger: right after the mov eax,esp instruction. The reserved range rb code_size_in_bytes comes directly after that and is not accessible anymore.

P.S. Note that you are overwriting the mov eax,esp instruction.

_________________
Faith is a superposition of knowledge and fallacy
Post 24 Nov 2015, 12:23
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
YEAH! Laughing Laughing
you both are the best. Cool

;mov eax, esp
rb code_size_in_bytes
virtual at 0

store byte bt111 at as111:name + (%-1)


annn ... that's it! I'm happy!! and not because of the mess in private Smile
thanks again


Description:
Filesize: 28.79 KB
Viewed: 5286 Time(s)

324t2442h.png


Post 24 Nov 2015, 14:27
View user's profile Send private message Reply with quote
Devel99



Joined: 26 Oct 2015
Posts: 30
Devel99
revolution setting virtual base to zero is more handy at a glance because it's possible to access any encapsulated private section regardless of where it is allocated, in a heap or on stack, for example:
Code:
; --- private section --- 
  m_start4 cipher1
  db 3, 4, 5, 6, 1, 1, 1, 1, 0xfd, 0xfc, 0xfb
    .x1 dd -1.1
    .x2 dd -2.2
    db 10 dup(0xaa) 
  m_end4 cipher1 
.....
 mov ebx, [esp + cipher1.x1]
 call [ebp + cipher1.x2]
    


at least I think so
Post 24 Nov 2015, 14:59
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on GitHub, YouTube, Twitter.

Website powered by rwasa.