flat assembler
Message board for the users of flat assembler.

Index > Heap > UPX memory @Rugxulo hogginess | BIG compression thread

Goto page 1, 2  Next
Author
Thread Post new topic Reply to topic
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
rugxulo wrote:
UPX uses a lot more than PAQ. I've noticed it swaps pretty easily and without warning. (Blame C++ or even LZMA, I guess.)


Funny claim, but wrong. Did you test ? Idea

Compressing itself (bloat = 1.4 MiB) --lzma --best 24 MiB

Compressing MPLAYER (bloat = 14 MiB) --best 96 MiB NRVR slow as hell, poor result

Compressing MPLAYER (bloat = 14 MiB) --lzma 128 MiB

Compressing MPLAYER (bloat = 14 MiB) --best --lzma 312 MiB Shocked

This is of course negligible compared to P.A.Q. hogging almost 3 GiB Shocked in its insane mode, irrespective how small the source file is Shocked and even worse it needs that much again to decompress it back Shocked

UPX 3.03 DGJPP version (check my deBUGGER about DGJPP quality)

PS: of course I answer here now, preventing further pollution of Tomasz's announcement Idea

EDIT : updated subject


Last edited by DOS386 on 09 Mar 2009, 01:59; edited 1 time in total
Post 20 Feb 2009, 03:49
View user's profile Send private message Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
Interesting, I never heard of PAQ. How does PAQ compare to 7Zip Ultra Compression with solid mode and a lot of memory?

ps: I'm compression freak Razz

And for UPX I use --ultra-brute just for kicks. It's really slow though. Then again just find a use for my processor.
Post 20 Feb 2009, 17:13
View user's profile Send private message Reply with quote
Madis731



Joined: 25 Sep 2003
Posts: 2141
Location: Estonia
Madis731
Yeah, me too - http://www.maximumcompression.com/index.html

but where's the *original* topic. DOS386 started from somewhere middle...
Post 21 Feb 2009, 14:42
View user's profile Send private message Visit poster's website Yahoo Messenger MSN Messenger Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
> Interesting, I never heard of PAQ.

http://www.cs.fit.edu/~mmahoney/compression/

> How does PAQ compare to 7Zip Ultra Compression with solid mode and a lot of memory?

PAQ is noticeably stronger ... but unusably slow + memhoggy + unstable.

> ps: I'm compression freak

A very new one if ever Laughing

Madis731 wrote:

> but where's the *original* topic

I wrote:

> preventing further pollution of Tomasz's announcement

http://board.flatassembler.net/topic.php?t=9770&start=40
Post 24 Feb 2009, 01:06
View user's profile Send private message Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Btw, from that post:
DOS386 wrote:
YES. Try to kick pagefile.sys Idea
You couldn't do that on win2k and below (on win2k it would give you a warning during boot and create a 20meg (or so) temporary pagefile). But since WinXP you can disable pagefile.sys totally. How much physical RAM you'll need to do this successfully depends on your needs - 512meg wasn't enough for games, but ever since I upgraded to 1gig several years back, I've run without swapping and never looked back.
Post 24 Feb 2009, 05:20
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 17287
Location: In your JS exploiting you and your system
revolution
After I upgraded to XP and removed the pagefile.sys file I found it quite enlightening as to which apps handle out of memory situations gracefully and which don't. And it is usually that they handle it as gracefully as a fat man riding downhill with no brakes. So there is a lesson to all you programmers out there, never simply assume there is always enough memory for your app!
Post 24 Feb 2009, 05:30
View user's profile Send private message Visit poster's website Reply with quote
f0dder



Joined: 19 Feb 2004
Posts: 3170
Location: Denmark
f0dder
Hear ye, revolution, hear ye!

There's even some applications out there that will refuse to run at all if the pagefile has been demanded (can't remember the specifics, but iirc there's at least one adobe app that won't).
Post 24 Feb 2009, 06:20
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
> But since WinXP you can disable pagefile.sys totally

OK ... 98 also could and got even more buggy then Laughing

revolution wrote:
After I upgraded to XP and removed the pagefile.sys file I found it quite enlightening as to which apps handle out of memory situations gracefully and which don't. And it is usually that they handle it as gracefully as a fat man riding downhill with no brakes. So there is a lesson to all you programmers out there, never simply assume there is always enough memory for your app!


Very good points Smile
Post 24 Feb 2009, 08:08
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
DOS386 wrote:
rugxulo wrote:
UPX uses a lot more than PAQ. I've noticed it swaps pretty easily and without warning. (Blame C++ or even LZMA, I guess.)


Funny claim, but wrong. Did you test ? Idea


Yes, and it (UPX) ran out of memory quite quickly (even on my recent laptop). I never use more than option "-1" for PAQ (37 MB RAM or 48 MB for .JPG) since it doesn't usually help much plus slows down even more. I would've tweaked PAQ8f if I really wanted low memory (21 MB RAM or 28 MB for .JPG) or even better, just stuck with LPAQ (6 or 9 MB RAM), but I figured PAQ8o8 was latest / greatest, so ....

DOS386 wrote:

Compressing itself (bloat = 1.4 MiB) --lzma --best 24 MiB

Compressing MPLAYER (bloat = 14 MiB) --best 96 MiB NRVR slow as hell, poor result

Compressing MPLAYER (bloat = 14 MiB) --lzma 128 MiB

Compressing MPLAYER (bloat = 14 MiB) --best --lzma 312 MiB Shocked


Yes, UPX uses too much memory perhaps. But you can dial it down by using -1 -2 -3 -4 etc (since it's like -7 or -8 by default). And yes, it can be slow, but try "--best --lzma --all-filters" instead of "--ultra-brute" to save a few years of your life. Laughing

N.B. "--all-filters" is an undocumented, "hidden" option (just as "--small" and "--fileinfo").

DOS386 wrote:

This is of course negligible compared to P.A.Q. hogging almost 3 GiB Shocked in its insane mode, irrespective how small the source file is Shocked and even worse it needs that much again to decompress it back Shocked


Correction: I think you have to be "insane" to use PAQ -8 or -9 for real-world stuff. Like I said, I never use besides -1. (Default is -5 except for my hacked variant.) But it's not "unstable" as you claim since it verifies the compressed output before continuing, which is probably one of the reasons it's so slow, especially under VMs. (Besides, the MMX or SSE2 speedups help a lot, so it could be MUCH worse!!) Hint: recompile it with best compiler you can find (which for me is DJGPP's G++ 4.3.2, GNU/Linux users are "free" to use Intel's for non-commercial use).

DOS386 wrote:

UPX 3.03 DGJPP version (check my deBUGGER about DGJPP quality)


Try my DJGPP compile(s) of UPX-UCL (source) or compile your own via -mtune=native for faster speeds.
Post 24 Feb 2009, 08:16
View user's profile Send private message Visit poster's website Reply with quote
asmfan



Joined: 11 Aug 2006
Posts: 392
Location: Russian
asmfan
Btw where you got to know all this options? Running with -h/-? doesn't give much (no --lzma, etc.). I only found they're mentioned in changelog but nowhere officially else.
Post 24 Feb 2009, 09:40
View user's profile Send private message Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
f0dder wrote:
But since WinXP you can disable pagefile.sys totally. How much physical RAM you'll need to do this successfully depends on your needs - 512meg wasn't enough for games, but ever since I upgraded to 1gig several years back, I've run without swapping and never looked back.
Wait, what's the advantage without a swap file? I thought that if you have enough RAM it won't even swap -- why do you need to implicitly tell it to not use the swap file?

For example, if you have 2 GBs of RAM, and you use 1GB, I thought you won't swap at all. But if you need 2.2 you will swap? Better than not running at all. What's the disadvantage by keeping swapping "just in case" or is Windows stupid to use it even if your RAM isn't full?

rugxulo wrote:
And yes, it can be slow, but try "--best --lzma --all-filters" instead of "--ultra-brute" to save a few years of your life. Laughing
Well you can run it in the background on an unused core. What does "--all-filters" do?

_________________
Previously known as The_Grey_Beast
Post 25 Feb 2009, 19:33
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
asmfan wrote:

Btw where you got to know all this options? Running with -h/-? doesn't give much (no --lzma, etc.). I only found they're mentioned in changelog but nowhere officially else.


The docs aren't quite recent enough to list everything. I think --brute or --ultra-brute automatically test LZMA for you (unlike --best or -1 through -9). The only case where you really have to explicitly tell it to use LZMA is for 16-bit DOS .EXEs, because LZMA can be much slower to unpack on old machines than NRV or UCL (especially for big binaries).

The three undocumented options (which may be removed in future versions) are --all-filters, --small, --fileinfo :

--all-filters can be used with --best --lzma for 99% of the same result as --ultra-brute (assuming LZMA produces smallest) but a million times faster. This is what latest versions of NASM/DOS use. (What does it do? "Tests all filters", I dunno more about it than you guys here.)

--small doesn't seem to do much, and although I haven't exhaustively tested, it only seems to affect DJGPP2/COFF (and not Watcom/LE). It's less than a few hundred bytes savings, so only useful for floppies (if even).

--fileinfo tells, in somewhat cryptic numbers, what methods were used to pack an .EXE.

Borsuc wrote:

Well you can run it in the background on an unused core.


Assuming your cpu has multi-core. My single-core P4 just plains almost freezes if I try to do --ultra-brute, very very resource intensive.
Post 27 Feb 2009, 08:32
View user's profile Send private message Visit poster's website Reply with quote
Borsuc



Joined: 29 Dec 2005
Posts: 2466
Location: Bucharest, Romania
Borsuc
rugxulo wrote:
Assuming your cpu has multi-core. My single-core P4 just plains almost freezes if I try to do --ultra-brute, very very resource intensive.
Yeah well I still have my P4 right now (though plan to get a new quad soon) but it has hyper-threading, set it to run in the background, lowered its priority, and it worked relatively smooth to multi-task (not that I use otherwise intensive applications, so that could be a factor as well).

_________________
Previously known as The_Grey_Beast
Post 27 Feb 2009, 20:58
View user's profile Send private message Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
rugxulo wrote:
Yes, and it (UPX) ran out of memory quite quickly (even on my recent laptop).


COOL, but this doesn't really tell anything ...

Quote:
Correction: I think you have to be "insane" to use PAQ -8 or -9 for real-world stuff.


I'm not. I use just only ZIP and 7-ZIP for real-world stuff, and never PAQ or WingRK Laughing

Quote:
But it's not "unstable" as you claim since it verifies the compressed output before continuing


Good, but missed the point: with "unstable" I didn't mean "buggy" , but "algo changing 1'000'000 times per year" Laughing

Quote:
probably one of the reasons it's so slow


Probably NO, or by factor 2 at most.

Quote:
especially under VMs.


Good, but one day you will have to explain why you prefer to run DGJPP/PAQ/etc under "VM" at all Mad

Quote:
DOS386 wrote:
check my deBUGGER about DGJPP quality
Try my DJGPP compile(s) of


Done! But what is supposed to be different ? They are very same as the "official" version !!!
Post 28 Feb 2009, 12:51
View user's profile Send private message Reply with quote
Tomasz Grysztar



Joined: 16 Jun 2003
Posts: 7725
Location: Kraków, Poland
Tomasz Grysztar
DOS386 wrote:
I'm not. I use just only ZIP and 7-ZIP for real-world stuff, and never PAQ or WingRK Laughing

WinRK? Is this one somehow related to the old RKIVE I've been using in DOS back in the 90s?
Post 28 Feb 2009, 13:03
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386
Tomasz Grysztar wrote:
Quote:
WingRK

WinRK? Is this one somehow related to the old RKIVE I've been using in DOS back in the 90s?


No idea. Where to get RKIVE ? WingRK is Windoze only, GUI only, and very buggy, OTOH forum full of bug reports got deleted recently Laughing
Post 28 Feb 2009, 13:40
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
DOS386 wrote:

Quote:
Correction: I think you have to be "insane" to use PAQ -8 or -9 for real-world stuff.


I'm not. I use just only ZIP and 7-ZIP for real-world stuff, and never PAQ or WingRK Laughing


If you don't need .WAV or .JPG (re)compression, fine. Otherwise, you're gonna have to use PAQ8 or wait for LZMA2 (whenever that will be)! Wink

DOS386 wrote:

Quote:
But it's not "unstable" as you claim since it verifies the compressed output before continuing


Good, but missed the point: with "unstable" I didn't mean "buggy" , but "algo changing 1'000'000 times per year" Laughing


Well, he's working on ZPAQ, which should be a stable base, hopefully.

DOS386 wrote:

Quote:
probably one of the reasons it's so slow


Probably NO, or by factor 2 at most.


It's written in C++ with a very complex neural network context mixer thingamabob. It's not supposed to be fast, just good! Razz

DOS386 wrote:

Quote:
especially under VMs.


Good, but one day you will have to explain why you prefer to run DGJPP/PAQ/etc under "VM" at all Mad


Because it mostly "just works"? Because I don't have drivers for other OSes? Because it came bundled on this laptop with a huge NTFS partition that I haven't backed up or resized yet (which I don't consider fun in the least)?

DOS386 wrote:

Quote:
DOS386 wrote:
check my deBUGGER about DGJPP quality
Try my DJGPP compile(s) of


Done! But what is supposed to be different ? They are very same as the "official" version !!!


No. The UPX-UCL versions are "free software". Plus, by luck the recompile(s) are using a newer G++ and are thus faster (even moreso with specific cpu optimizations instead of "one-size-fits-all"). So license and speed are really the "only" reasons for bothering.
Post 09 Mar 2009, 00:18
View user's profile Send private message Visit poster's website Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
DOS386 wrote:
Tomasz Grysztar wrote:
Quote:
WingRK

WinRK? Is this one somehow related to the old RKIVE I've been using in DOS back in the 90s?


No idea. Where to get RKIVE ? WingRK is Windoze only, GUI only, and very buggy, OTOH forum full of bug reports got deleted recently Laughing


RKIVE v1.4 - File archiver 75546 1996-10-30 00:00:00
RKIVE v1.92 beta 1 - File archiver 133515 1998-04-07 00:00:00

Yes, there was at one time a DOS version of RKIVE (even bundled with his old DJGPP TWS package).

Code:
http://www.maximumcompression.com/data/text.php

001 WinRK 3.0.3     PWCM 912MB Dict 330115  88.95   0.8837
002   PAQAR 4.5       -8      333759  88.83   0.8934
003   PAQ8P   -7      358413  88.01   0.9594
013   LPAQ8   7       427349  85.70   1.1440
024   7-Zip 4.60b     -m0=ppmd:o=20:mem=26    442703  85.19   1.1851
049   UHARC 0.6b      -mx     476675  84.05   1.2760
057   RKIVE 1.92      -p0 -mt3 -t4096 -b60000 -mm1    487233  83.70   1.3043
    


No idea if those are the latest versions, but from all accounts, they are buggy too. In fact, Malcolm Taylor's WinRK only gets such good results due to being heavily inspired by PAQ itself.

http://www.uclc.info/
http://www.maximumcompression.com

P.S. Tomasz, I never really directed you to this due to my own timidity, honestly, but since it's come up, you may weakly be interested (I mean, it does have an optional FASM bit although I use NASM by default since it supports both DJGPP and OpenWatcom):

paq8o8z-Feb28-bin.zip (1749k, bins only, only if you can't build it yourself)
paq8o8z-Feb28-src.zip (392k, srcs only)

IMHO, it's almost as good as any compressor out there (or very close, at least) although not exactly having tons of options like RAR or ARJ. Actually, I think 7-Zip is probably one of the best overall (speed / ratio / license / portability / features).
Post 09 Mar 2009, 00:30
View user's profile Send private message Visit poster's website Reply with quote
DOS386



Joined: 08 Dec 2006
Posts: 1901
DOS386


Thanks. By Malcolm Taylor, so it is related to WingRK.

Quote:
I think 7-Zip is probably one of the best overall (speed / ratio / license / portability / features).


YES.

Quote:
If you don't need .WAV or .JPG (re)compression, fine. Otherwise, you're gonna have to use PAQ8 or wait for LZMA2 (whenever that will be)!


There are better choices for WAV. And NO, I don't "recompress" JPG Very Happy

Quote:
Well, he's working on ZPAQ, which should be a stable base, hopefully.


When it's out I'll check. But not before. Shocked

Quote:
It's not supposed to be fast, just good!


Not "good", just provide very maximal compression at any cost. OR do you know a product that is "supposed to be" shitty ?

> Because it mostly "just works"?

MinGW also would, and better.

> Because I don't have drivers for other OSes?

Does DGJPP need them ?

> Because it came bundled on this laptop with a huge NTFS partition that I haven't backed up or resized yet

Use DBAN. Idea

> UPX-UCL versions are "free software".

Kicked the "NRVR" ? But why didn't you update the texts and the package ? Not even the date is updated Confused How to distinguish UPX-UCL from the "official" release ?

Quote:
license and speed are really the "only" reasons for bothering.


I see. The garbage I/O is definitely the very same.

Oops, forgot the most important thing for Tomasz : KGB archiver by Tomasz Laughing
Post 09 Mar 2009, 01:53
View user's profile Send private message Reply with quote
rugxulo



Joined: 09 Aug 2005
Posts: 2341
Location: Usono (aka, USA)
rugxulo
Well, I can't sleep, so I'll respond to this I guess. Wink

DOS386 wrote:

There are better choices for WAV. And NO, I don't "recompress" JPG Very Happy


Your loss. Less floppies to transport, smaller e-mail attachments, etc. etc. Wink

DOS386 wrote:

Quote:
It's not supposed to be fast, just good!


Not "good", just provide very maximal compression at any cost. OR do you know a product that is "supposed to be" shitty ?


What, you think it goes too far? I agree 7-Zip is better for average / normal files, but otherwise you need something robust! Razz

DOS386 wrote:

> Because it mostly "just works"?

MinGW also would, and better.


MinGW better than DJGPP? (I assume that's what you meant.) No, it's not. In particular, MinGW builds won't work in DOS without MSVCRT.DLL, which needs a separate license. Plus, their tmpfile() needs Admin access due to writing to root dir (!), which doesn't work by default in Vista. In other words, most PAQ Win32 builds are MinGW-based and all have the same problem. I'd feel really really foolish if I started a new CMD as Admin every time just to be able to compress a stupid file. Besides, I've timed it, the DJGPP version is no slower, at least for me.

DOS386 wrote:

> Because I don't have drivers for other OSes?

Does DGJPP need them ?


Not for PAQ, no, but if I ever intend to write you any more online messages, I might actually need a working Internet connection. Laughing

DOS386 wrote:

> Because it came bundled on this laptop with a huge NTFS partition that I haven't backed up or resized yet

Use DBAN. Idea


No thanks. I do wonder, however, if *BSD has better Broadcom support. Oh well, too hard to find a live CD for that. I guess I'm too lazy / busy with other stuff. (Besides, I'm sadly *nix ignorant.)

DOS386 wrote:

Quote:

> UPX-UCL versions are "free software".

Kicked the "NRVR" ? But why didn't you update the texts and the package ? Not even the date is updated Confused How to distinguish UPX-UCL from the "official" release ?

Quote:
license and speed are really the "only" reasons for bothering.


I see. The garbage I/O is definitely the very same.


Code:
[ Vista ] - Mon 03/09/2009 >upx -V
upx 3.03
NRV data compression library 0.84
UCL data compression library 1.03
zlib data compression library 1.2.3
LZMA SDK version 4.43
Copyright (C) 1996-2008 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2008 Laszlo Molnar
Copyright (C) 2000-2008 John F. Reiser
Copyright (C) 2002-2008 Jens Medoch
Copyright (C) 1995-2005 Jean-loup Gailly and Mark Adler
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx -L'.

[ Vista ] - Mon 03/09/2009 >upx-ucl -V
upx 3.03
UCL data compression library 1.03
LZMA SDK version 4.43
Copyright (C) 1996-2008 Markus Franz Xaver Johannes Oberhumer
Copyright (C) 1996-2008 Laszlo Molnar
Copyright (C) 2000-2008 John F. Reiser
Copyright (C) 2002-2008 Jens Medoch
Copyright (C) 1999-2006 Igor Pavlov
UPX comes with ABSOLUTELY NO WARRANTY; for details type 'upx-ucl -L'.

[ Vista ] - Mon 03/09/2009 >scrndump
    



In short, it may not be readily apparent, but UPX-UCL lacks NRV. Besides, the obvious way to tell the difference is file name (since I did NOT name my compiles UPX.EXE). Even the file date should tell you something.

DOS386 wrote:

Oops, forgot the most important thing for Tomasz : KGB archiver by Tomasz Laughing


Which is PAQ6-based, your favorite! Laughing
Post 09 Mar 2009, 08:36
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You can attach files in this forum
You can download files in this forum


Copyright © 1999-2020, Tomasz Grysztar. Also on YouTube, Twitter.

Website powered by rwasa.