flat assembler
Message board for the users of flat assembler.

Index > DOS > How to write simple packer?

Goto page Previous  1, 2, 3
Author
Thread Post new topic Reply to topic
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
I'm no lawyer, but everything I read on the Internet claims Unisys's patent has run out for LZW (a variation of LZ78?) both here in the States and overseas. Supposedly, IBM mistakenly got (and still has) a patent for the same thing and never made a fuss over it. Their patent is widely considered invalid for what we're referring to. Besides, GNU zip (and you know how legally careful they are!) supports unpacking Unix 'compress'-packed files (i.e., .Z), which used a form of LZW. Even Info-ZIP finally put support back into their Unzip for the Shrink method (LZW-based) because of this expiration (circa 2004, I think).

Anyways, I don't personally worry about it (much), but anyways most people use Deflate-based LZ77 compression (in .ZIP or .GZ) or something else (BZip2, UHarc) like BWT or PPM, none of which have presented a problem (thankfully). Deflate is meant to be (and claimed to be) a patentless method. There will always be worries about these types of things. I just hope the world can behave itself and not get too greedy/mean.
Post 09 May 2006, 03:31
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Post 19 May 2006, 12:24
View user's profile Send private message Visit poster's website Reply with quote
Adam Kachwalla



Joined: 01 Apr 2006
Posts: 150
Adam Kachwalla
Basically, if you want to know which is faster, try compressing a file and shoving it onto a floppy disk. Then format the disk and put the uncompressed file on it.
Post 10 Jun 2006, 07:38
View user's profile Send private message Reply with quote
2



Joined: 26 Sep 2006
Posts: 92
2
Cool stuff. See,I kinda wanted to make a compressor that would somehow
remove all spaces in a text file and then somehow put them back in the right places. Though I'm sure nobody here can make a compression
program that beats 7zip.
Post 04 Nov 2006, 22:57
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
PAQ8i and UHarc both beat 7-Zip in lots of stuff. Other programs are better with certain files (e.g., BMF for .BMP, WavPack for .WAV, Stuffit for .JPG, etc.).

See Maximum Compression (or Matt Mahoney's large 1GB text benchmark) for a good comparison of archivers.

P.S. 7-Zip and WavPack (or the old official, real-mode version) have been ported to DOS.
Post 05 Nov 2006, 02:38
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1903
DOS386
OzzY wrote:
Quote:

Hi! I'm thinking about making a text packer.
I have a text like this:
tttttffffaaaaaaabbb
and it gets packed to
t5f4a7b3
What I mean is pack the text based on how much times each char repeats.
I thinks you understand what I mean.


This is RLE and a good start. Many files are inefficient enough to be
compressible this way. Shocked

bogdanontanu wrote:

Quote:
LZW and Arithmetic coders are patented.


LZW84 is no longer Razz, Arith maybe still is ... but there is Range
- almost identical, but unpatented Razz

rugxulo wrote:

Quote:
P.S. 7-Zip and WavPack (or the old official, real-mode version) have been ported to DOS.


Right. Prefer the new DOS/32A based WAVPACK. Razz

FLAC also ported:

http://blairdude.googlepages.com/flac

Finally, 98% of Win32 console packers do work in DOS using HX Razz

DOS no longer has problems with console packers Razz

_________________
Bug Nr.: 12345

Title: Hello World program compiles to 100 KB !!!

Status: Closed: NOT a Bug
Post 05 Jan 2007, 10:18
View user's profile Send private message Reply with quote
ATV



Joined: 31 Aug 2004
Posts: 109
Location: Finland
ATV
Few years ago I was testing some lz algorithms, compression rate is not good as zip/rar/7z
But they are small 2546/2845/1521 bytes
Attachment has lzari/lzhuf/lzss with my asm (TASM) source and Haruhiko Okumura original c source


Description: lzari/lzhuf/lzss com + asm/c source
Download
Filename: LZ_3PAK.ZIP
Filesize: 42.16 KB
Downloaded: 143 Time(s)

Post 08 Jan 2007, 13:08
View user's profile Send private message Reply with quote
MCD



Joined: 21 Aug 2004
Posts: 604
Location: Germany
MCD
you usually have to optimize between 2 situations:
more speed or less size.

UPX-packed executables bring neither significant size changes nor significant speed changes if the executables are small. If the executables are bigger, than UPX can reduce the time to load the exec. files (cause. there will be less to load from media) and reduce the size of the files on media, but it will usually also increase the amount of used RAM (which is not freed by UPX during the whole lifetime of the program, which is more a design flaw of UPX, but which could be changed).

Anyway, in many modern programs, the executable parts of an executable Exclamation Laughing are seldom the cause of big executable files. It's those irresponsible, unthrifty use of resources in executables which makes them so big and bloated, which should be better but in external data files in the first place(which is happily common under Linux), where you can apply better and more specialized compression than with UPX (also maintenance will be easier).

And if the size of the executable part really reach several MBs, then you have probably made some very common mistakes:
1.) you have put your whole big application consisting of several executable parts, static libraries and s.o. into 1file, dude Rolling Eyes
2.) you have probably never heard of code-encapsulating with functions, procedures, methods, classes, also simple jumps...
3.) you have used a HLL-compiler that produces or forces you to produce very redundant code (like that old Pascal/Delphi thing did to me)
4.) You have used a HLL-compiler like in 3.) for embedded-systems or other memory limited systems, which is even worse Razz

so there should be no use for UPX in the first place. Concludingly we can say that UPX is good workaround for some common design flaws Shocked

To speak of encryption: cracking an executable that is simply packed(no additional encryption) is much simpler than cracking an executable that is strongly encrypted (like Rijndael, Serpent...) unless you don't know the packing-algorithm, which in turn is also not completly impossible to retrieve.
So you should not use packers for encryption in the first place (unless those features an additional strong encryption)

My first RLE/dictionary stuff I did was years ago in TASM, but several weeks ago, we repeated that topic in IT-studies, but much more theoritically. The only version of an RLE-algorithm I got right now for you Ozzy is in Java, which probably won't fit your needs.

_________________
MCD - the inevitable return of the Mad Computer Doggy

-||__/
.|+-~
.|| ||
Post 08 Jan 2007, 21:10
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
MCD: keep in mind that windows does "just in time" loading of executables, so you could have a couple hundred megabytes of resources in the .exe and it wouldn't be a problem if you didn't use them. Resources also have the advantage of being sharable across processes.

Once your start exeuctable-packing, these advantages go away. The entire exe WILL be loaded to memory at startup, there will be no page-sharing across multiple process instances, and all pages are dirty and must, in a low-mem situation, be paged out to disk and re-read later, rather than just discarded and re-read from the executable file.

Check out http://f0dder.reteam.org/packandstuff.htm (old and needs probably some fixing)

Quote:

cracking an executable that is simply packed(no additional encryption) is much simpler than cracking an executable that is strongly encrypted

Not unless there's other protection stuff in place - otherwise the task is the same: find OEP, dump, rebuild imports. Encryption, by itself, helps nothing.
Post 08 Jan 2007, 23:36
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
MCD wrote:
so there should be no use for UPX in the first place. Concludingly we can say that UPX is good workaround for some common design flaws Shocked


If you have the .EXE but don't have the source, UPX helps. Also, in low-space situations (e.g., trying to make a boot floppy), it helps too. Some redundancy is because people forget to do simple things (strip the .EXE, compile without debug support, disable certain runtime features like globbing, etc.). Of course, a little elbow grease (see Hugi Compo) can go a LONG way. Wink

P.S. Don't quote me, but I think UPX refuses to compress anything if the amount compressed doesn't equal or surpass 1/8 of the original size (okay, maybe I misunderstood, but I think it's something like that).
Post 09 Jan 2007, 05:40
View user's profile Send private message Visit poster's website Reply with quote
MCD



Joined: 21 Aug 2004
Posts: 604
Location: Germany
MCD
f0dder wrote:
MCD: keep in mind that windows does "just in time" loading of executables, so you could have a couple hundred megabytes of resources in the .exe and it wouldn't be a problem if you didn't use them.

I had forgotten that

f0dder wrote:
Resources also have the advantage of being sharable across processes.
Resources in external files can also be shared across processes, or did I misunderstood you?
Post 11 Jan 2007, 03:37
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
MCD wrote:
f0dder wrote:
Resources also have the advantage of being sharable across processes.
Resources in external files can also be shared across processes, or did I misunderstood you?


Yes, I think you did a bit Smile

The thing I mean by sharing is that the same physical memory is used across different processes; if you want to do something like this yourself with external resources, you need to do stuff with shared memory mapped files Smile

_________________
Image - carpe noctem
Post 11 Jan 2007, 09:39
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar.

Powered by rwasa.