flat assembler
Message board for the users of flat assembler.

Index > DOS > How to write simple packer?

Goto page Previous  1, 2, 3  Next
Author
Thread Post new topic Reply to topic
TDCNL



Joined: 25 Jan 2006
Posts: 56
TDCNL
Adam Kachwalla wrote:
There is something I might have left out:

Compressing it also adds protection (to a certain extent) from crackers. Also, if the unpacking code is a gigabyte long, of course it will take more memory (That's the sort of packer I would dump). Packers such as UPX will indeed have to take a few more bytes, but that is nothing: esspecially if you are loading a 1MB executable from a slow device. UPX compresses that executable to only 300KB, so at least it will fit on a floppy disk! That is a more obvious example (I hope).


In that case it'd be a good _solution_ to pack, but it doesn't mean executable loading would be faster!

_________________
:: The Dutch Cracker ::
Post 26 Apr 2006, 06:16
View user's profile Send private message Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
Of course you have to make some study - how much (on EVERY computer) is HDD worse than RAM and beyond that - how much CPU cache to use and how much RAM.

Then you can choose between packing ratio and unpacking time. You might have fun trying it, but I don't think that you can't get more then 50% packing with <10KB algorithm that can easily do it in the CPU cache with plenty of room to programs.

For example when HDD on your current computer is 10x slower than RAM, then you can waste 8-9x more time on unpacking than on reading - of course you can hide a bit of HDD loading behind other CPU work, but then again how do you clear HDDs 1MB, 2MB, or even 8MB of cache every time you start your program.

I think the cache are clever enough to optimize the load time of your program enough that you don't need to pack.

When it comes to game data - there is one possibility you might want to use that - this is where you make sure, that you will exceed HDD cache at least 100 times, because execeeding it 2-3 times won't give you any advantage. That is because when you have loaded your first bunch of 8MB, the HDD has fetched in the background another 8MB Razz
Post 26 Apr 2006, 06:54
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Quote:

Compressing it also adds protection (to a certain extent) from crackers.

Not really. Trace OEP, dump, rebuild imports - presto, done.

Quote:

Packers such as UPX will indeed have to take a few more bytes, but that is nothing: esspecially if you are loading a 1MB executable from a slow device.

All pages in your program will be marked 'dirty', and in low-memory situations those pages will have to be written to your paging file to be read back in later. For an uncompressed executable, most pages will be clean, and can thus be discarded instead of flushed to the paging file.

As for 'slow devices', how often do you load files from, say, a floppy drive?

EXE compression is something you shouldn't do in general, and it should almost always be a choice the end-user makes. In a few cases (like where size REALLY does matter), it can be okay though... I do use compression for my fSekrit program.
Post 26 Apr 2006, 12:38
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Some (older?) DJGPP programs packed with (older?) versions of UPX can't even be unpacked correctly (i.e., TDE will unpack, won't run). Of course, it's usually not necessary to unpack anything (maybe change DPMI stub to WDOSX, perhaps?). Nevertheless, it's obviously worth considering whether you need it or not. But I say go ahead and use it unless you have a good reason not to. If you have no reason to waste space, then don't. Razz


    .COM/.EXE packers
  1. aPACK 0.99b -- best for 16-bit DOS programs, free for non-commercial use, no src
  2. 32LiTE 0.02d -- for various 32-bit DOS programs, free for non-commercial use, no src
  3. UPX 2.02 -- win32/pe, djgpp2/coff, dos/com, dos/exe, dos/sys, watcom/le, etc., C++/ASM src, GPLv2
  4. 624 1.1 -- for small .COM files < 25k (C/ASM src for old version only!)
  5. DIET 1.45f -- old DOS .COM/.EXE packer, freeware, no src
  6. LZEXE 0.91 -- another old DOS .EXE packer, freeware, no src


    .COM archivers
  1. PACO 1.2 (in PBATS) -- combine your .COMs into one .COM, freeware, no src
  2. XEQ 1.60 -- combines your .COM files into itself (XEQ.COM), freeware, no src
  3. ARK 1.01 -- combine your .COMs into itself (ARK.EXE), .ASM src, GPLv2


    .COM/.EXE unpackers
  1. IUP 0.67 -- generic .COM/.EXE unpacker, .ASM src, public domain
  2. UNP 4.11 -- specific/generic unpacker, cardware, no src
  3. CUP386 3.4 -- generic unpacker (with debugger?), freeware, no src


P.S. Packers put a stub in your .COM/.EXE, and the stub is probably copyrighted (e.g., UPX allows free redistribution but no modification).

P.P.S. Some DOS extenders have their own means of compression (WDOSX, PMODE/W, DOS/32A).


Last edited by rugxulo on 29 Oct 2006, 00:43; edited 3 times in total
Post 26 Apr 2006, 22:04
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
f0dder: How often do you notice the 'uncompressing' stage which is VERY fast (at least in my experience). If you're from the view that "a few bytes/clock cycles don't matter", do you REALLY see a difference when loading compressed executables?

personally, if I designed my own OS, I would've employed the .exe format to be compressed (not super-heavy compressed, but fast compression). It saves a few bytes and it doesn't make any difference. You can, as well, have slow disk access (like fragmentation, etc.). So believe me, there's no REAL difference, except for the smaller size, of course Wink
Post 01 May 2006, 11:35
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
I don't notice the "uncompressing" stage, but I do notice the memory wasted. Again, "private bytes", dirty vs. clean pages, etc.

Quote:

personally, if I designed my own OS, I would've employed the .exe format to be compressed (not super-heavy compressed, but fast compression). It saves a few bytes and it doesn't make any difference. You can, as well, have slow disk access (like fragmentation, etc.). So believe me, there's no REAL difference, except for the smaller size, of course Wink

Compressing at the exe level is silly - you should compress at filesystem level instead. This is a lot better, since it can be integrated with the memory manager etc., and you can avoid 'dirty pages' this way.
Post 01 May 2006, 12:36
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Let me give an example: WinSCP 3.8.0 build 312, http://winscp.sf.net . It's distributed compressed with UPX. I uncompressed and recompressed with the most recent UPX, to get the best possible ratio.

UNCOMPRESSED:
....exe size: 4.215.296 bytes
....private: 3.224 kb
....virtual: 109.036 kb
....working set: 8.340 kb

COMPRESSED:
....exe size: 1.214.464 bytes
....private: 6.876 kb
....virtual: 109.068 kb
....working set: 10.558 kb

So, for every open instance of WinSCP, you waste 3.652kb of *physical* memory. That quickly adds up if you don't have a high-spec machine. And in the case of a low-memory situation, those "private bytes" will have to be written to your paging file and restored later on. Compare that with the non-private bytes of the working set, which can simply be discarded and re-read from the exe - you skip a COSTLY write-to-disk.

Oh, and as for the decompression phase not wasting a lot of clock cycles... that's not the full truth if you have on-demand virus scanning running. Jørgen "Jibz" Ibsen told me that starting FileZilla on his old box was almost unbearable, until he decompressed the executables.

Remember, not everybody has gigahertz++ computers. And if you're writing assembly programs, shouldn't they already be tiny? Wink
Post 01 May 2006, 12:49
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
f0dder wrote:
Remember, not everybody has gigahertz++ computers. And if you're writing assembly programs, shouldn't they already be tiny? Wink
Very Happy
I was talking about small programs. Those do not require such big amounts of decompression anyway.

"dirty pages" come because "UPX" compression is not integrated in the operating system. Like I said, if I would design my own Windows, I would've add this "compressed" format directly into the kernel (just like adding compression on a filesystem), and that will no longer require dirty pages. And won't require the "stub" in the exe too, since it's integrated directly into the kernel.

I fully agree UPX is not to be used everywhere, but Microsoft claimed compressed exes are bad, but that's because THEY didn't implement this feature in the OS directly (like they implemented NTFS, but a compression specialized for .exe). That's why I started this argument, to comment on their false statements, sorry for misguiding then Smile
Post 01 May 2006, 13:19
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Quote:

I would've add this "compressed" format directly into the kernel (just like adding compression on a filesystem), and that will no longer require dirty pages.

This would be a relatively complex task to achieve, especially if you also want page-sharing across processes. Again, I'd say it's better to compress at filesystem level... you also gain the advantage of being able to compress other stuff than just executables.

Quote:

but Microsoft claimed compressed exes are bad

Huh, when?

Anywya, compressed exes ARE generally kinda bad... again, the on-demand virus scanner argument. The vscan has to unpack the exe before it can scan (and this can require emulation in the case of unknown packer (slow)). With filesystem compression, nothing has to be emulated, and you get almost no speed hit.
Post 01 May 2006, 13:27
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
f0dder wrote:
Anywya, compressed exes ARE generally kinda bad... again, the on-demand virus scanner argument. The vscan has to unpack the exe before it can scan (and this can require emulation in the case of unknown packer (slow)). With filesystem compression, nothing has to be emulated, and you get almost no speed hit.

I don't see why compressing at filesystem level is faster than the integrated .exe compression. Shouldn't they be the same? since they're at the same (low) level.

Oh yeah, i forgot about virus scans.. Very Happy
Post 01 May 2006, 13:33
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
It won't necessarily be faster at the filesystem level, but it will make the memory management less complicated and more efficient (dirty bit, sharing).

The main place where fs-level compression has a *speed* advantage over exe-level is the virus scanners.
Post 01 May 2006, 15:04
View user's profile Send private message Visit poster's website Reply with quote
Cas



Joined: 26 Feb 2004
Posts: 82
Location: Argentina
Cas
OzzY... what about this?:

Most characters in a text file are letters and these are just 26 times 2, that is 26 capital and 26 lowercase letters, which makes up a total of 52 different characters. This can be stored in a minimum of 6bit each, in contrast with the standard 8bit per byte. What you can do is the following:

- Give a number from 0 to 63 to each letter, upper and lower case (52 in total) and number (10 in total) and the space character. And you have one left. So, numbers 0 to 62 represent letters, numbers and the space.
- In order for you to represent another character, you use the code 63 that you have left and then you add a normal 8bit byte with the actual value. This way you will actually save 2/8 on most characters and waste 6/14 on a very little amount of them.
- Now, to store 6bit bytes in a sequence, you can do this, for example:

First 8bit byte = [first 6bit byte][2 lowest bits of fourth byte]
Second 8bit byte = [second 6bit byte][2 middle bits of fourth byte]
Third 8bit byte = [third 6bit byte][2 highest bits of fourth byte]

and so on. It is simple, though it could sound a little complex written that way. If your interested and can give you clearer (and longer) example. Just post me back of drop me a line at lucasmastersid@gmail.com. But I guess the links the guys posted can help you much more if you want to get deep into it. Good luck.
Post 04 May 2006, 05:20
View user's profile Send private message Yahoo Messenger MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Cas, that approach won't gain you very much, you might as well go for Huffman or Arithmetic coding if you're going to use not-whole-bytes encoding.

Or stick with the LZ family, which has decent speed and ratio, is well-known and well-researched with lots of existing implementations.
Post 04 May 2006, 09:35
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
UPX has no memory overhead for most formats. So, maybe using it blindly for everything would be bad, but most medium-sized, non-concurrent Win9x programs or DLLs are okay (see the doc for more details).
Post 05 May 2006, 04:49
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Okay, I forgot to mention, 624 has source code (C/ASM). It's a "small .COM" packer, not a text packer, but it might help. Smile
Post 05 May 2006, 17:15
View user's profile Send private message Visit poster's website Reply with quote
Cas



Joined: 26 Feb 2004
Posts: 82
Location: Argentina
Cas
Of course, f0dder, I agree. But I think maybe OzzY will have less trouble implementing a simpler compression system (specially if it is going to be in ASM) and said he wanted something he could understand while he did it. In addition... aren't procedures such as Huffman compression copyrighted? I know for a fact that LZ is (that is the main reason why graphic format PNG appeared). As I said, I wanted to state something simple. I would've referred to RLE if it weren't because it is useless against text files. (OzzY: RLE is what you suggested at the beggining).
Post 08 May 2006, 06:50
View user's profile Send private message Yahoo Messenger MSN Messenger Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Afaik, only a specific variant of LZ was patented by unisys, which was the reason the GIF format got shunned in favour of PNG - but there's a lot of algorithms in the LZ family. Also, I believe Unisys's patent has expired now.

I don't think there's any problems with the huffman algorithm (which is simple enough), but some of the Arithmetic routines might be covered by patents.
Post 08 May 2006, 10:32
View user's profile Send private message Visit poster's website Reply with quote
bogdanontanu



Joined: 07 Jan 2004
Posts: 403
Location: Sol. Earth. Europe. Romania. Bucuresti
bogdanontanu
LZW and Arithmetic coders are patented.

Do not expect patents to rarely expire as they can be extended undefinitively under the law and with good layers. but maybe they will "let them be"

Huffman, LZ77 (sliding window) and Range coders are not patented AFAIK.

IMHO LZ77 as presented in BriefLZ is pretty easy to understand and implement...be it in ASM or any other programming language.

Again IMHO it is much more usefull and "real" that a simple RLE.
Post 08 May 2006, 10:38
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
BriefLZ: http://www.ibsensoftware.com/download.html , by Jørgen "Jibz" Ibsen, a pretty decent programmer.
Post 08 May 2006, 10:42
View user's profile Send private message Visit poster's website Reply with quote
pierre



Joined: 07 Nov 2004
Posts: 10
pierre
Post 08 May 2006, 11:29
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.