flat assembler
Message board for the users of flat assembler.

Index > Linux > my HeavyThing general purpose x86_64 linux library

Goto page 1, 2, 3, 4  Next
Author
Thread Post new topic Reply to topic
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 01 Feb 2015, 21:47
I am pleased to announce the public release of my GPL library.

https://2ton.com.au/HeavyThing/

Unfortunately, the library tarball is too large to attach directly to this post.

Lots of topics/functionality covered that I have not personally seen anywhere else, too many to list here. I have included several example programs in addition to my showcase pieces that make use of the library extensively. Among them, an SSH2 "talk" server, a TLS1.2-capable web performance benchmarking tool, and a full-featured webserver that includes assembly language function hooks to facilitate "rapid web application server" development.

To my knowledge, this is the single-largest GPL release of x86_64 assembler. As you might imagine, it also represents many thousands of hours of work and is a result of my 27 years experience as a commercial software developer. I truly hope it provides value to our community.

While the library is designed for x86_64 linux, there are a great many functions that do not rely on the underlying kernel of course, so I hope that the library will be useful to other x86_64 programmers outside the scope of just linux as well.

Cheers to everyone here on the board!

P.S. I have chosen to enable very strong encryption parameters for my webserver (which is running rwasa, a part of my library). Due to my choices here, old versions of Firefox will not negotiate a connection.
Post 01 Feb 2015, 21:47
View user's profile Send private message Reply with quote
HaHaAnonymous



Joined: 02 Dec 2012
Posts: 1178
Location: Unknown
HaHaAnonymous 02 Feb 2015, 19:04
[ Post removed by author. ]


Last edited by HaHaAnonymous on 28 Feb 2015, 17:59; edited 1 time in total
Post 02 Feb 2015, 19:04
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 06 Feb 2015, 06:29
Hey thanks, one or more people turned it loose on reddit/hackernews, so a lot of people got ahold of it over the last few days. Response has been tremendous, really positive. Assembly language is not dead by any means Smile
Post 06 Feb 2015, 06:29
View user's profile Send private message Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 06 Feb 2015, 07:11
Do you have implementations of the pthread functions (mutex, conditional variables, ...)?
Post 06 Feb 2015, 07:11
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 06 Feb 2015, 18:34
tthsqe wrote:
Do you have implementations of the pthread functions (mutex, conditional variables, ...)?
Most of my server applications don't require threading, and instead use fork + socketpair + epoll for IPC (see my epoll_child.inc for how I do that). Main reason I didn't bother to include POSIX-style pthread goods is because between the clone and futex syscall, they are very easy to do in assembler (along with normal processor based atomic ops).
Post 06 Feb 2015, 18:34
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 08 Feb 2015, 18:07
Great work redsock.
Post 08 Feb 2015, 18:07
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 08 Feb 2015, 18:31
tuimatrix demo.

an exception (segmentation fault) at this line;

Code:
mov r8,rsi
add rsi,rdx
mov rax,qword[rsi]    


it says invalid reference to 0x0000000000000002.
Post 08 Feb 2015, 18:31
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 08 Feb 2015, 19:09
Hmm runs okay for me, any chance you could fire it up with gdb and give me the stacktrace as well?
Post 08 Feb 2015, 19:09
View user's profile Send private message Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 09 Feb 2015, 06:20
Would you like some help with some more math functions? i looked an the power function and it looks too complkicated...
Post 09 Feb 2015, 06:20
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 09 Feb 2015, 19:23
tthsqe wrote:
Would you like some help with some more math functions? i looked an the power function and it looks too complkicated...
Haha, the design from Naoki Shibata for that SSE pow function is amazing! Agreed it is complicated, but the speed of it is insane compared to the other "easier to read" varieties.
Post 09 Feb 2015, 19:23
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 10 Feb 2015, 10:18
redsock wrote:
Hmm runs okay for me, any chance you could fire it up with gdb and give me the stacktrace as well?


sorry my friend. i dont talk gdb or anything related. I use Evan's Debugger (EDB) exclusively. It reports something like this;

qword ptr [0000000000000002h] = 0ffffffffffffffffh

The addressing error could suggest a slight portability issue because i tested it on a 64-bit laptop. Maybe u should too.
Post 10 Feb 2015, 10:18
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 10 Feb 2015, 21:51
system error wrote:
tuimatrix demo.

an exception (segmentation fault) at this line;

Code:
mov r8,rsi
add rsi,rdx
mov rax,qword[rsi]    


it says invalid reference to 0x0000000000000002.
Ahh, turns out mmap doesn't actually return a -1 like I thought, and I never bothered to run any of it in a constrained memory environment with no VM/swap.

The latest version should do the deed without complaint/segfault:

https://2ton.com.au/HeavyThing/

Cheers for that

Edit: updated URL so it points to the page with the latest, noticed a few downloads of the older version from this post.


Last edited by redsock on 17 Feb 2015, 01:11; edited 1 time in total
Post 10 Feb 2015, 21:51
View user's profile Send private message Reply with quote
l_inc



Joined: 23 Oct 2009
Posts: 881
l_inc 10 Feb 2015, 23:51
redsock
Although I don't feel like a potential user of the library, it seems to be well-written and includes nice design decisions such as common named settings and on-demand compilation of features. There are some things that I'd do differently, but what really dispirits me are little pieces of negligence in the code such as this:
Quote:
mov eax, dword [r8*4+.tables+3072]
and r10d, 0xff
xor eax, dword [r9*4+.tables+2048]
shr r11d, 24
xor eax, dword [r10*4+.tables+1024]
xor eax, dword [r11*4+.tables]

Lots of unnamed numeric constants and explicit type casting (instead of defining multiple .tableX as dword labels and named constants of kind .sizeof.tableX and .countof.tableX) a priori indicate sloppy coding style for me, which in this case dissonates with both library design and implementation.

_________________
Faith is a superposition of knowledge and fallacy
Post 10 Feb 2015, 23:51
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 11 Feb 2015, 02:38
l_inc wrote:

but what really dispirits me are little pieces of negligence in the code such as this:
l_inc, really appreciate your feedback. I concede that some of my code sections did not receive educational commentary, but mostly where I chose to do so are for well-known and/or common functionality. In the routine you reference for example, there are a great many CRC32 table-based routines in existence, and there aren't very many ways to skin that particular cat. Whether this is sloppy coding practice or not, I am not sure I agree with you entirely. Alas, readability is important.

Someday when I am sufficiently bored I will do an editorial pass over all of it and make such code sections easier to read and follow.

Any and all feedback is very much appreciated, especially constructive criticism like yours.

Cheers
Post 11 Feb 2015, 02:38
View user's profile Send private message Reply with quote
tthsqe



Joined: 20 May 2009
Posts: 767
tthsqe 11 Feb 2015, 08:34
Hello there,
This seems to be quite a library.
I like how things are weighted down the big integers part. I wrote the mandelbrot explorer for the official fasm examples, so I can appreciate your work here.
I got to testing your pow function (to should really say that it operates on packed doubles) against a 'typical implementation' (https://github.com/tthsqe12/src-tt/blob/master/fasm/misc_text_hot.asm#L56)
I ran it a few million times on (1.1,1.2)^(2.3,2.4)
Code:
pow: 296 cycles/call   @ 3120 bytes
math_powdd: 254 cycles/call @ 134 bytes     

however the math_powdd only does one operation at a time, so yours wins the throughput contest. Smile If you avx'ed it up and added some fmadds you could get 4 ops per call and probably get the call time time to the 250 cycles/call mark.
Post 11 Feb 2015, 08:34
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 11 Feb 2015, 08:50
tthsqe wrote:
Hello there,
This seems to be quite a library.
I like how things are weighted down the big integers part. I wrote the mandelbrot explorer for the official fasm examples, so I can appreciate your work here.
I got to testing your pow function (to should really say that it operates on packed doubles) against a 'typical implementation' (https://github.com/tthsqe12/src-tt/blob/master/fasm/misc_text_hot.asm#L56)
I ran it a few million times on (1.1,1.2)^(2.3,2.4)
Code:
pow: 296 cycles/call   @ 3120 bytes
math_powdd: 254 cycles/call @ 134 bytes     

however the math_powdd only does one operation at a time, so yours wins the throughput contest. Smile If you avx'ed it up and added some fmadds you could get 4 ops per call and probably get the call time time to the 250 cycles/call mark.
Legendary! Smile Because the speed for AES was so dramatically reduced with non AESNI, I finally had to add hardware-capability jump tables to it, and that was well worth it. I had no idea the same applied to pow, I will be adding this in the very near future.
Post 11 Feb 2015, 08:50
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 17 Feb 2015, 01:49
revolution, after my initial release a couple of weeks ago, got quite a bit of feedback regarding my choice to use 4096 bit Diffie-Hellman parameters, and as I result, I have wound-back the choices for my site, so you should be able to get to it now, haha.

Let me know if it works at all with no JS enabled.

As for the rest, wow, a little bit of social media and a ton of feedback goes a long way. Been a very busy couple of weeks for me, very happy with the latest release, and so far my clients are also delighted with it.

Cheers everyone
Post 17 Feb 2015, 01:49
View user's profile Send private message Reply with quote
revolution
When all else fails, read the source


Joined: 24 Aug 2004
Posts: 20421
Location: In your JS exploiting you and your system
revolution 17 Feb 2015, 02:53
The basic website loads and displays. But I didn't click on anything or download anything because it appears to be only for Linux and I've never had any success running Linux on any of my computers.

I notice the webpage gives no explanation as to what it actually does. There are many details about how to download it, compile it, etc. but no description to say what one is supposed to use it for. Is it some sort of system monitor? Or a web server? Or a command shell? An assembler? I couldn't figure out from the "Feature Highlights" what its primary function or purpose is. I probably just missed something obvious but even a simple single paragraph description would be immensely useful "HeavyThing is ...".
Post 17 Feb 2015, 02:53
View user's profile Send private message Visit poster's website Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 434
Location: Australia
redsock 17 Feb 2015, 05:12
revolution wrote:
I notice the webpage gives no explanation as to what it actually does. ... I probably just missed something obvious but even a simple single paragraph description would be immensely useful "HeavyThing is ...".
Haha, your point is received. In my head, [insert programming language here] library loosely translate to: box of goodies that does things useful. So my excuse as it were is simply that calling it an x86_64 assembler library and then listing its "features" does say it what is.

I should point out however that my lovely lady says you raise a very valid point. I shall consider adding an introductory paragraph about what it really is. Thanks for your input and consideration.

Cheers
Post 17 Feb 2015, 05:12
View user's profile Send private message Reply with quote
system error



Joined: 01 Sep 2013
Posts: 670
system error 17 Feb 2015, 07:10
revolution wrote:
The basic website loads and displays. But I didn't click on anything or download anything because it appears to be only for Linux and I've never had any success running Linux on any of my computers.

I notice the webpage gives no explanation as to what it actually does. There are many details about how to download it, compile it, etc. but no description to say what one is supposed to use it for. Is it some sort of system monitor? Or a web server? Or a command shell? An assembler? I couldn't figure out from the "Feature Highlights" what its primary function or purpose is. I probably just missed something obvious but even a simple single paragraph description would be immensely useful "HeavyThing is ...".


I think the main highlight is algorithms.
Post 17 Feb 2015, 07:10
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page 1, 2, 3, 4  Next

< Last Thread | Next Thread >
Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Copyright © 1999-2024, Tomasz Grysztar. Also on GitHub, YouTube.

Website powered by rwasa.