flat assembler
Message board for the users of flat assembler.

Index > Projects and Ideas > I wrote fast LZ-type 64-bit compressor

Author
Thread Post new topic Reply to topic
SergeASM



Joined: 13 Nov 2015
Posts: 21
SergeASM 13 Jun 2017, 13:04
Hi all,

I wrote in FASM small OS independed 64-bit procedures for compress/uncompress RAM-RAM "on the fly". The procedures all working data store in registers. Both can be multi-call/thread-safe and can be stored in ROM at any address.

Here you can download my small and simple program for Win64, which compress/uncompress given file:
http://www.cronc.com/lz4i64.zip
The file can have size from 0 to unlimited. Note:
1. Your CPU should support AVX instructions set.
2. For large files you must have ~1Gb free memory for compression, and ~2Gb for uncompression. (Generally speaken its not necessary: the program can be compiled to read smaller pieces of data.)
3. The program does not calculate and store check sum.
4. The program does not save file attributes in archive.
5. The program rewrites files without prompt.
6. After (un)compress the program shows working time in ms by GetTickCount.

Authors of compression programs often uses file enwik8 (100 million bytes length) for testing. Its a piece of Englich Wikipedia in XML format. You can download it from
http://www.mattmahoney.net/dc/textdata

On Intel I3 5005U 2GHz with RAM 1.6 GHz my program compresses enwik8 in 0.8 s., and uncompresses in 0.42 s. Compression ratio ~ 52%.

lz4i64.exe enwik8
ren enwik8 enwik8_1
lz4i64.exe enwik8.lz4i
fc /b enwik8 enwik8_1
No differences encountered.

Wow! Smile

My program is similar to lz4 by Yann Collet in fast (simple hash) mode:
https://github.com/lz4/lz4
But my program uses hash table of 512 K cells 8 bytes each...

Also you can find other similar projects: QuickLZ, Snappy by Google...

Is someone interested in my procedures, or I'm wasting my time?

Sorry for poor English.

_________________
Serge
Post 13 Jun 2017, 13:04
View user's profile Send private message Visit poster's website Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 19726
Location: In your JS exploiting you and your system
revolution 13 Jun 2017, 13:54
SergeASM wrote:
lz4i64.exe enwik8
ren enwik8 enwik8_1
lz4i64.exe enwik8.lz4i
fc /b enwik8 enwik8_1
No differences encountered.
So your code takes enwik8 and creates enwik8.lz4i? And later your code auto-uncompresses the data from enwik8.lz4i? How does your code determine whether to compress or uncompress?
Post 13 Jun 2017, 13:54
View user's profile Send private message Visit poster's website Reply with quote
SergeASM



Joined: 13 Nov 2015
Posts: 21
SergeASM 14 Jun 2017, 07:28
If the given file name ends on .lz4i then my code uncompress the file, otherwise compress it. For simple example to demonstrate my subroutines its enough.
Post 14 Jun 2017, 07:28
View user's profile Send private message Visit poster's website Reply with quote
komoco



Joined: 27 Sep 2017
Posts: 2
komoco 27 Sep 2017, 05:00
And where is the souce code?
Post 27 Sep 2017, 05:00
View user's profile Send private message Reply with quote
SergeASM



Joined: 13 Nov 2015
Posts: 21
SergeASM 15 Oct 2017, 08:16
At this time I'm working on a new fast & quick pure LZ file compressor by my idea. I hope over time I will finalize this LZ4 like compressor and will publish source code.
Post 15 Oct 2017, 08:16
View user's profile Send private message Visit poster's website Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  


< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2023, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.