Index
> DOS > How to write simple packer?Goto page Previous 1, 2, 3 Next |
| Author |
|
|
Madis731 26 Apr 2006, 06:54
Of course you have to make some study - how much (on EVERY computer) is HDD worse than RAM and beyond that - how much CPU cache to use and how much RAM.
Then you can choose between packing ratio and unpacking time. You might have fun trying it, but I don't think that you can't get more then 50% packing with <10KB algorithm that can easily do it in the CPU cache with plenty of room to programs. For example when HDD on your current computer is 10x slower than RAM, then you can waste 8-9x more time on unpacking than on reading - of course you can hide a bit of HDD loading behind other CPU work, but then again how do you clear HDDs 1MB, 2MB, or even 8MB of cache every time you start your program. I think the cache are clever enough to optimize the load time of your program enough that you don't need to pack. When it comes to game data - there is one possibility you might want to use that - this is where you make sure, that you will exceed HDD cache at least 100 times, because execeeding it 2-3 times won't give you any advantage. That is because when you have loaded your first bunch of 8MB, the HDD has fetched in the background another 8MB  |
|||
26 Apr 2006, 06:54 |
|
|
f0dder 26 Apr 2006, 12:38
Quote:
Not really. Trace OEP, dump, rebuild imports - presto, done. Quote:
All pages in your program will be marked 'dirty', and in low-memory situations those pages will have to be written to your paging file to be read back in later. For an uncompressed executable, most pages will be clean, and can thus be discarded instead of flushed to the paging file. As for 'slow devices', how often do you load files from, say, a floppy drive? EXE compression is something you shouldn't do in general, and it should almost always be a choice the end-user makes. In a few cases (like where size REALLY does matter), it can be okay though... I do use compression for my fSekrit program. |
|||
|
26 Apr 2006, 12:38 |
|
|
rugxulo 26 Apr 2006, 22:04
Some (older?) DJGPP programs packed with (older?) versions of UPX can't even be unpacked correctly (i.e., TDE will unpack, won't run). Of course, it's usually not necessary to unpack anything (maybe change DPMI stub to WDOSX, perhaps?). Nevertheless, it's obviously worth considering whether you need it or not. But I say go ahead and use it unless you have a good reason not to. If you have no reason to waste space, then don't.
.COM/.EXE packers
.COM archivers
.COM/.EXE unpackers P.S. Packers put a stub in your .COM/.EXE, and the stub is probably copyrighted (e.g., UPX allows free redistribution but no modification). P.P.S. Some DOS extenders have their own means of compression (WDOSX, PMODE/W, DOS/32A). Last edited by rugxulo on 29 Oct 2006, 00:43; edited 3 times in total |
|||
|
26 Apr 2006, 22:04 |
|
|
Borsuc 01 May 2006, 11:35
f0dder: How often do you notice the 'uncompressing' stage which is VERY fast (at least in my experience). If you're from the view that "a few bytes/clock cycles don't matter", do you REALLY see a difference when loading compressed executables?
personally, if I designed my own OS, I would've employed the .exe format to be compressed (not super-heavy compressed, but fast compression). It saves a few bytes and it doesn't make any difference. You can, as well, have slow disk access (like fragmentation, etc.). So believe me, there's no REAL difference, except for the smaller size, of course  |
|||
|
01 May 2006, 11:35 |
|
|
f0dder 01 May 2006, 12:36
I don't notice the "uncompressing" stage, but I do notice the memory wasted. Again, "private bytes", dirty vs. clean pages, etc.
Quote:
Compressing at the exe level is silly - you should compress at filesystem level instead. This is a lot better, since it can be integrated with the memory manager etc., and you can avoid 'dirty pages' this way. |
|||
|
01 May 2006, 12:36 |
|
|
f0dder 01 May 2006, 12:49
Let me give an example: WinSCP 3.8.0 build 312, http://winscp.sf.net . It's distributed compressed with UPX. I uncompressed and recompressed with the most recent UPX, to get the best possible ratio.
UNCOMPRESSED: ....exe size: 4.215.296 bytes ....private: 3.224 kb ....virtual: 109.036 kb ....working set: 8.340 kb COMPRESSED: ....exe size: 1.214.464 bytes ....private: 6.876 kb ....virtual: 109.068 kb ....working set: 10.558 kb So, for every open instance of WinSCP, you waste 3.652kb of *physical* memory. That quickly adds up if you don't have a high-spec machine. And in the case of a low-memory situation, those "private bytes" will have to be written to your paging file and restored later on. Compare that with the non-private bytes of the working set, which can simply be discarded and re-read from the exe - you skip a COSTLY write-to-disk. Oh, and as for the decompression phase not wasting a lot of clock cycles... that's not the full truth if you have on-demand virus scanning running. Jørgen "Jibz" Ibsen told me that starting FileZilla on his old box was almost unbearable, until he decompressed the executables. Remember, not everybody has gigahertz++ computers. And if you're writing assembly programs, shouldn't they already be tiny? |
|||
|
01 May 2006, 12:49 |
|
|
Borsuc 01 May 2006, 13:19
f0dder wrote: Remember, not everybody has gigahertz++ computers. And if you're writing assembly programs, shouldn't they already be tiny? 
I was talking about small programs. Those do not require such big amounts of decompression anyway. "dirty pages" come because "UPX" compression is not integrated in the operating system. Like I said, if I would design my own Windows, I would've add this "compressed" format directly into the kernel (just like adding compression on a filesystem), and that will no longer require dirty pages. And won't require the "stub" in the exe too, since it's integrated directly into the kernel. I fully agree UPX is not to be used everywhere, but Microsoft claimed compressed exes are bad, but that's because THEY didn't implement this feature in the OS directly (like they implemented NTFS, but a compression specialized for .exe). That's why I started this argument, to comment on their false statements, sorry for misguiding then  |
|||
|
01 May 2006, 13:19 |
|
|
f0dder 01 May 2006, 13:27
Quote:
This would be a relatively complex task to achieve, especially if you also want page-sharing across processes. Again, I'd say it's better to compress at filesystem level... you also gain the advantage of being able to compress other stuff than just executables. Quote:
Huh, when? Anywya, compressed exes ARE generally kinda bad... again, the on-demand virus scanner argument. The vscan has to unpack the exe before it can scan (and this can require emulation in the case of unknown packer (slow)). With filesystem compression, nothing has to be emulated, and you get almost no speed hit. |
|||
|
01 May 2006, 13:27 |
|
|
Borsuc 01 May 2006, 13:33
f0dder wrote: Anywya, compressed exes ARE generally kinda bad... again, the on-demand virus scanner argument. The vscan has to unpack the exe before it can scan (and this can require emulation in the case of unknown packer (slow)). With filesystem compression, nothing has to be emulated, and you get almost no speed hit. I don't see why compressing at filesystem level is faster than the integrated .exe compression. Shouldn't they be the same? since they're at the same (low) level. Oh yeah, i forgot about virus scans.. |
|||
|
01 May 2006, 13:33 |
|
|
f0dder 01 May 2006, 15:04
It won't necessarily be faster at the filesystem level, but it will make the memory management less complicated and more efficient (dirty bit, sharing).
The main place where fs-level compression has a *speed* advantage over exe-level is the virus scanners. |
|||
|
01 May 2006, 15:04 |
|
|
Cas 04 May 2006, 05:20
OzzY... what about this?:
Most characters in a text file are letters and these are just 26 times 2, that is 26 capital and 26 lowercase letters, which makes up a total of 52 different characters. This can be stored in a minimum of 6bit each, in contrast with the standard 8bit per byte. What you can do is the following: - Give a number from 0 to 63 to each letter, upper and lower case (52 in total) and number (10 in total) and the space character. And you have one left. So, numbers 0 to 62 represent letters, numbers and the space. - In order for you to represent another character, you use the code 63 that you have left and then you add a normal 8bit byte with the actual value. This way you will actually save 2/8 on most characters and waste 6/14 on a very little amount of them. - Now, to store 6bit bytes in a sequence, you can do this, for example: First 8bit byte = [first 6bit byte][2 lowest bits of fourth byte] Second 8bit byte = [second 6bit byte][2 middle bits of fourth byte] Third 8bit byte = [third 6bit byte][2 highest bits of fourth byte] and so on. It is simple, though it could sound a little complex written that way. If your interested and can give you clearer (and longer) example. Just post me back of drop me a line at lucasmastersid@gmail.com. But I guess the links the guys posted can help you much more if you want to get deep into it. Good luck. |
|||
|
04 May 2006, 05:20 |
|
|
f0dder 04 May 2006, 09:35
Cas, that approach won't gain you very much, you might as well go for Huffman or Arithmetic coding if you're going to use not-whole-bytes encoding.
Or stick with the LZ family, which has decent speed and ratio, is well-known and well-researched with lots of existing implementations. |
|||
|
04 May 2006, 09:35 |
|
|
rugxulo 05 May 2006, 04:49
UPX has no memory overhead for most formats. So, maybe using it blindly for everything would be bad, but most medium-sized, non-concurrent Win9x programs or DLLs are okay (see the doc for more details).
|
|||
|
05 May 2006, 04:49 |
|
|
rugxulo 05 May 2006, 17:15
Okay, I forgot to mention, 624 has source code (C/ASM). It's a "small .COM" packer, not a text packer, but it might help.
|
|||
|
05 May 2006, 17:15 |
|
|
Cas 08 May 2006, 06:50
Of course, f0dder, I agree. But I think maybe OzzY will have less trouble implementing a simpler compression system (specially if it is going to be in ASM) and said he wanted something he could understand while he did it. In addition... aren't procedures such as Huffman compression copyrighted? I know for a fact that LZ is (that is the main reason why graphic format PNG appeared). As I said, I wanted to state something simple. I would've referred to RLE if it weren't because it is useless against text files. (OzzY: RLE is what you suggested at the beggining).
|
|||
|
08 May 2006, 06:50 |
|
|
f0dder 08 May 2006, 10:32
Afaik, only a specific variant of LZ was patented by unisys, which was the reason the GIF format got shunned in favour of PNG - but there's a lot of algorithms in the LZ family. Also, I believe Unisys's patent has expired now.
I don't think there's any problems with the huffman algorithm (which is simple enough), but some of the Arithmetic routines might be covered by patents. |
|||
|
08 May 2006, 10:32 |
|
|
bogdanontanu 08 May 2006, 10:38
LZW and Arithmetic coders are patented.
Do not expect patents to rarely expire as they can be extended undefinitively under the law and with good layers. but maybe they will "let them be" Huffman, LZ77 (sliding window) and Range coders are not patented AFAIK. IMHO LZ77 as presented in BriefLZ is pretty easy to understand and implement...be it in ASM or any other programming language. Again IMHO it is much more usefull and "real" that a simple RLE. |
|||
|
08 May 2006, 10:38 |
|
|
f0dder 08 May 2006, 10:42
BriefLZ: http://www.ibsensoftware.com/download.html , by Jørgen "Jibz" Ibsen, a pretty decent programmer.
|
|||
|
08 May 2006, 10:42 |
|
|
pierre 08 May 2006, 11:29
For text only packing see :
http://www.frontiernet.net/~fys/hugi/compoold.htm#compo2 |
|||
|
08 May 2006, 11:29 |
|
| Goto page Previous 1, 2, 3 Next < Last Thread | Next Thread > |
Forum Rules:
|