flat assembler
Message board for the users of flat assembler.
![]() |
Author |
|
l_inc 22 Nov 2015, 23:29
Devel99
What you do is an old and very inefficient way of getting an unknown (at preprocessing time) amount of data out of a virtual addressing space. But in even in those old days this was required only for implementing algorithms possibly having less output than input data, such as compression or digesting. Other algorithms (such as encryption) could have been implemented with more efficient but often complicated in-place data modification. Now fasm has addressing space labels (see the last two paragraphs at the end of the section "Addressing spaces"), and none of the above is necessary. P.S. Btw. you still should use in-place data modification, if your algorithm does not become too complicated. Having addressing space labels in virtual addressing spaces reserves compile-time memory that is not regainable. Thus even if you absolutely need these, you should avoid linear increase of unregainable memory by either creating addressing space labels outside of the virtual addressing spaces only or by having a single common virtual addressing space for all invocations of your macros. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 23 Nov 2015, 00:07
l_inc thanks for the answer, I'm not going to invent some encryption algos on my own. Those RSA macros you refer to assume that encrypted section will reside in the code. My intent is to create open and closing macros to prepare structured data inside and encrypt on demand with ability to reject some, calculate crc32, or whatever
![]() ![]() Code: rept m_size_in_dwords x:0 йThanks |
|||
![]() |
|
l_inc 23 Nov 2015, 00:22
Devel99
Quote: Those RSA macros you refer to assume that encrypted section will reside in the code. I wasn't referring to any specific macros. I was talking about general approaches. And it doesn't make any difference if it's code or data to be processed. Quote: the only problem is that the following does not work, however "m_size_in_dwords" is known value while REPT is invoked It doesn't work, because m_size_in_dwords is not known, when rept is invoked. To be more precise m_size_in_dwords is only known as a sequence of tokens, but not as a numeric value. rept is a preprocessor directive and m_end_internal is only resolved after the preprocessing, i.e. at the assembly stage. See my short explanation here. Quote: My intent is to create open and closing macros to prepare structured data inside and encrypt on demand with ability to reject some, calculate crc32, or whatever I don't think, there's any common work that could be done in a generic way covering all these "whatever" use cases. Each of these need specific macros. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 23 Nov 2015, 00:23
l_inc in-place data modification is not acceptable for me because mentioned macro can potentially reject all data between opening and closing section, but in-place modification assumes reserving spare bytes in the code. In my approch I'm able to allocate that datablock wherever I like, even on stack in runtime
Last edited by Devel99 on 23 Nov 2015, 00:42; edited 1 time in total |
|||
![]() |
|
l_inc 23 Nov 2015, 00:42
Devel99
Quote: in-place data modification is not acceptable for me because mentioned macro can potentially reject all data between opening and closing section The problem is that you are talking about abstract macros that only potentially do something, but you will need to make a decision about the implementation approach for each specific macro individually. Value names generation with rept can and should be avoided in any case. The other two approaches are: 1) In-place data modification. Not always possible, but often most efficient. 2) Addressing space labels. Always possible. To make it clear: what approach is suitable for you is only possible to decide depending on what specific kind of data processing you need. Quote: In my approch I'm able to allocate that datablock whereever I like, even on stack in runtime Well, this is not possible without knowing the data at compile-time. So it should still be encoded somewhere in the program, which makes in-place data modification again possible. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 23 Nov 2015, 01:12
l_inc
Quote:
Sure, it should be encoded, it resides between opening and closing sections. For example the code below allocates required blocks on stack Code: format PE console include 'win32ax.inc' macro m_start name { name: push eax mov eax, esp virtual at 0 } macro m_end { local m_end_internal local nonvirt_bytes align 4 m_end_internal: db 0 ; fake align-end marker m_size_in_dwords EQU m_end_internal/4 rept 20000 x1:0 \{ if x1 < m_size_in_dwords load d_\#x1 dword from 4*x1 end if \} end virtual nonvirt_bytes: rept 20000 x2:0 \{ if x2 < m_size_in_dwords push dword d_\#x2 end if \} } start: m_start cipher db 0xff, 0xfe, 0xfd, 0xfc, 0xfb db 10 dup(0xaa) m_end ; use data on stack [esp...eax] mov esp, eax pop eax m_start cipher2 db 7, 8, 9, 10, 11, 12, 13, 14, 15 m_end ; use data on stack [esp...eax] mov esp, eax pop eax start_next: invoke ExitProcess, 0 .end start
|
||||||||||
![]() |
|
Devel99 23 Nov 2015, 01:20
l_inc are there any FASM compiler memory limitations? Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun?
|
|||
![]() |
|
revolution 23 Nov 2015, 01:24
Eliminate the REPT and replace with REPEAT and use an address space.
REPT cannot ever know the length of the code because the length is obtained only after REPT has long since finished doing its thing. You will need reverse causality to work if you want to use REPT for that. |
|||
![]() |
|
revolution 23 Nov 2015, 01:26
Devel99 wrote: ... are there any FASM compiler memory limitations? Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun? See here: http://board.flatassembler.net/topic.php?t=17667 |
|||
![]() |
|
Devel99 23 Nov 2015, 01:50
revolution cool article. I have 32GB on board, I pray that one day FASM will become a 64-bit application
![]() |
|||
![]() |
|
l_inc 23 Nov 2015, 02:05
Devel99
Quote: Sure, it should be encoded, it resides between opening and closing sections. More specifically it resides inside of the push instructions. Your implementation is ridiculously slow and inefficient. Here's how the macros are supposed to be coded efficiently: Code: format PE console include 'win32ax.inc' match, { local ..start macro m_ \{ ..start: \} macro _m \{ local ..as,cbMem,cbCod,buf cbMem = $ - ..start rb cbCod-cbMem ..as:: virtual repeat cbMem/4+1 load buf dword from ..as:..start+(%-1)*4 push buf end repeat cbCod = $-$$ repeat cbCod load buf byte from $$+%-1 store byte buf at ..as:..start+%-1 end repeat end virtual \} } start: push ebp mov ebp,esp m_ db $FF,$FE,$FD,$FC,$FB db 10 dup $AA _m leave ret .end start The above code is rather a proof of concept for in-place modification than the final version for reasons mentioned below. I made a couple of simplifications and used both in-place data modification and virtual addressing spaces for educational purposes here. So there's some room for optimization. It should be noted however that the optimized form of the push instruction can use less space than the data it puts onto the stack, in which case you'd get a compilation error. This would be indeed the data compression case I was talking about before. And a good solution would be to put the data into a virtual block and an addressing space label outside of the virtual block, so that you store the instructions to the outside from within the virtual addressing space (same as shown in the example actually). Quote: Just took a quick look at FASMW process and it's a 32-bit process. Why not set FASM compiler memory to 3GB threshold and have fun? There isn't much more fun you can get with fasmw, cause its address space is polluted with GUI libraries. If you want to push usable memory to the limits you'd need to use the console version of fasm. Anyway my point was about efficient macro coding, not about polluting as much memory as possible with sloppy coded macros. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 23 Nov 2015, 14:31
l_inc Thanks for the answer and for the good sample code.
![]() ![]() I don't want to discuss pros and cons of in-place modification now ![]() |
|||
![]() |
|
Devel99 23 Nov 2015, 14:48
revolution thank you. I tried REPEAT later and it turned out that REPEAT was not quite suitable for my code logic.
![]() |
|||
![]() |
|
l_inc 23 Nov 2015, 15:20
Devel99
Quote: But I don't think your solution is flexible. [...] Let's try to comment out your "push" I don't understand your point. Why would you be willing to comment out push, if the very purpose of the macro is to push things onto the stack? I mean, no single piece of code in the world is resilient against arbitrary commenting out its parts. If you break the code, it breaks. Sure, but what's the point? Quote: It's good enough just for proof of concept only. It's good for any application that does not do any kind of data compression. That's why it was meant to be a proof of concept in this case. Quote: Obviously you'll need to track rb XXX size, so after changing code logic you'll have two bound code chunks instead of one. Again I don't understand the point. Tracking the size of the reserved region is automatic by design. What kind of logic change do you mean? There are no "two bound code chunks" (whatever that means) to be produced anyway. Quote: but you have to create placeholder first and track its size (in your implementation RB xxx ...) which is not the case with fixed rept implementation rb is a single little statement. It's simpler, it's shorter, it's faster. What's worrying you about it? Quote: Actually I don't care how sloopy the macro is and how much time and memory it takes to compile that, bacause the only thing which really matters is a byte sequence produced by FASM. This is wrong. It's probably not as important for hello world projects, but you are going to run your head against the wall as soon as your project becomes just a little larger will take minutes and hours to compile and eventually will eat up more memory than available, so that you'll be sitting and hunting down those damn macros preventing fasm from producing any sequence of bytes. Quote: I don't want to discuss pros and cons of in-place modification now As I said, in-place modification is not the best way in this case. My code isn't even really in-place modification, because it requires variable amount of additional virtual space (in-place is by definition using constant amount of space), but it shows that in-place would be possible, and it is only a few little modifications away from a perfect off-place solution, which I described after the code. rept is a horrible idea and luckily it's not needed anymore. Quote: I appreciate L_INC's effort in helping me to aviod "out of memory" issue but I can afford to add extra 32GB onboard ))) fasm can't use those. It can use 3GB and a bit at most. And even being able to add more memory won't help you to deal with the long compilation times. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 24 Nov 2015, 09:58
revolution l_inc
good morning guys I played a bit with fixed rept code tonight and actually ran out of memory after declaring 200 blocks and peak FASMW memory load was 2Gb ![]() I have to admit that my idea is not perfect My next approach is to create private section where a part of code is declared in virt block, fully resides in virt and not reflected to resulting PE executable. That was my first time I tried virt in private, so I stuck with this: store byte bt111 at as111:st111 + (%-1) I have address space as111, declared non-virt label (st111) but I get "value out of range"? Why do you think it is possible? Which value is out of range? ![]() Code: format PE console include 'win32ax.inc' ; --- nonvirt addr spc --- as111:: macro m_start4 name { name: mov eax, esp virtual at 0 } macro m_end4 name { local nonvirt_bytes, data_end_internal, code_end_internal align 4 data_end_internal: data_size_in_dwords = data_end_internal/4 ; prolog template push edx mov edx, esp sub esp, 118 ; repeating data template repeat data_size_in_dwords load dt1 dword from (%-1)*4 push dword dt1 end repeat ; epilog template mov esp, edx pop edx ret code_end_internal: code_size_in_bytes = code_end_internal - data_end_internal repeat code_size_in_bytes load bt111 byte from data_end_internal + (%-1) store byte bt111 at as111:st111 + (%-1) end repeat end virtual ; --- nonvirt bytes --- nonvirt_bytes: rb code_size_in_bytes } st111: ; --- private section --- m_start4 cipher1 db 3, 4, 5, 6, 1, 1, 1, 1, 0xfd, 0xfc, 0xfb db 10 dup(0xaa) m_end4 cipher1 start: call st111 invoke ExitProcess, 0 .end start |
|||
![]() |
|
revolution 24 Nov 2015, 10:34
Address space labels keep their offsets so your label is not at offset zero but instead at the PE load address in the first section. You can either adjust for that with appropriate +/- values, or shift the label into the virtual block and the offset will follow the block start, or don't make the virtual block at offset zero.
|
|||
![]() |
|
l_inc 24 Nov 2015, 12:23
Devel99
Quote: My next approach is to create private section where a part of code is declared in virt block There is no such thing called "private section". And it's unclear what you mean by that, cause the location of your virtual block isn't private or special in any other way. revolution wrote: Address space labels keep their offsets so your label is not at offset zero but instead at the PE load address in the first section This is wrong. Addressing space labels do not keep their offsets. All they keep is sort of identifier of an addressing space they refer to. Hence the fix suggestions aren't suitable. But the accessible address range for the load/store directives is limited by two values: the base address of the addressing space, which is $$ if you apply it inside of that addressing space, and current address value, which is $ if you apply it at the location of the load/store directive, but inside of that addressing space (meaning: the address right after the last byte in the accessed addressing space stored up until the occurrence of load/store). The lower bound of your accessible range referenced by as111 equals to st111 (as well as name), so no problem here. But the upper bound of the accessible range is just two bytes larger: right after the mov eax,esp instruction. The reserved range rb code_size_in_bytes comes directly after that and is not accessible anymore. P.S. Note that you are overwriting the mov eax,esp instruction. _________________ Faith is a superposition of knowledge and fallacy |
|||
![]() |
|
Devel99 24 Nov 2015, 14:27
YEAH!
![]() ![]() you both are the best. ![]() ;mov eax, esp rb code_size_in_bytes virtual at 0 store byte bt111 at as111:name + (%-1) annn ... that's it! I'm happy!! and not because of the mess in private ![]() thanks again
|
||||||||||
![]() |
|
Devel99 24 Nov 2015, 14:59
revolution setting virtual base to zero is more handy at a glance because it's possible to access any encapsulated private section regardless of where it is allocated, in a heap or on stack, for example:
Code: ; --- private section --- m_start4 cipher1 db 3, 4, 5, 6, 1, 1, 1, 1, 0xfd, 0xfc, 0xfb .x1 dd -1.1 .x2 dd -2.2 db 10 dup(0xaa) m_end4 cipher1 ..... mov ebx, [esp + cipher1.x1] call [ebp + cipher1.x2] at least I think so |
|||
![]() |
|
< Last Thread | Next Thread > |
Forum Rules:
|
Copyright © 1999-2025, Tomasz Grysztar. Also on GitHub, YouTube.
Website powered by rwasa.