flat assembler
Message board for the users of flat assembler.
 Home   FAQ   Search   Register 
 Profile   Log in to check your private messages   Log in 
flat assembler > Linux > my HeavyThing general purpose x86_64 linux library

Goto page Previous  1, 2, 3  Next
Author
Thread Post new topic Reply to topic
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
Hi, redsock. I have a question about configuring rwasa for working with my FastCGI script. I am recently playing with FastCGI forum inplementation in assembly language (Demo: http://board.asm32.info). It runs on Apache, but I though it would be great if we have entirely assembly language web solution and tried to make it work with rwasa. But can't configure it properly.

Usually, the web servers start the FastCGI processes and pass the socket where it have to listen in STDIN handler. Usually they spawn several processes according to the load.

How I should configure rwasa to do so?
Post 27 Mar 2016, 11:17
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
Well, I managed to start my script with rwasa, (this does not cancel the first question Smile ) but have two more questions:

1. How to set it to send all requests to the FastCGI server.
2. When I configure it for some file type, It gives me "506 Unimplemented Error Code" on attempt to load some page. What is wrong?
Post 27 Mar 2016, 12:58
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
I use the "spawn-fcgi" package that is available with most distros.

Code:
spawn-fcgi -s /dev/shm/php.sock /usr/bin/php-cgi

Alternatively, most modern fastcgi programs deal with their own process management (php is no different):

Code:
PHP_FCGI_CHILDREN=20 php-cgi -e -b /dev/shm/php.sock



Whether it is a unix socket (considerably better for localhost) or IPv4 doesn't really matter obviously, the fastcgi process is responsible for dealing with itself independently of the webserver. When I start rwasa with

Code:
./rwasa -bind 80 -sandbox /var/www/html -fastcgi .php /dev/shm/php.sock -indexfiles index.php,index.html -errsyslog -runas root -foreground

and then dump an example wordpress install there I don't get 506's (works as I expect it to).

As for passing every single request backward via FastCGI (an uncommon thing to do in a normal web enviro due to things like static resources, images, CSS, etc) but nevertheless: ~line 2050 of webserver.inc, if you add something like

Code:
    jmp    .fastcgi_doit

it will pass everything to the FastCGI handler though of course I haven't tested this Wink

If you have an example FastCGI handler you can post/send me that is spewing the 506's I'll take a closer look for you but since everything I have here (php and HHVM) both work I am unable to test further.

Cheers

_________________
2 Ton Digital - https://2ton.com.au/
Post 27 Mar 2016, 23:02
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria

redsock wrote:
I use the "spawn-fcgi" package that is available with most distros.

Code:
spawn-fcgi -s /dev/shm/php.sock /usr/bin/php-cgi

Alternatively, most modern fastcgi programs deal with their own process management (php is no different):

Code:
PHP_FCGI_CHILDREN=20 php-cgi -e -b /dev/shm/php.sock




Not so important. I made the script able to run properly and to create its own unix socket on "./engine.sock" (btw, using relative paths is another thing I would like in rwasa for testing purposes).


Quote:
As for passing every single request backward via FastCGI (an uncommon thing to do in a normal web enviro due to things like static resources, images, CSS, etc) but nevertheless: ~line 2050 of webserver.inc, if you add something like

Code:
    jmp    .fastcgi_doit

it will pass everything to the FastCGI handler though of course I haven't tested this Wink



Well, if you want to run only language interpreters (like PHP) it is normal to pass only some kind of files to them. But in my case, the whole site engine is inside the script and all the content is generated from a database (sqlite).


Quote:
If you have an example FastCGI handler you can post/send me that is spewing the 506's I'll take a closer look for you but since everything I have here (php and HHVM) both work I am unable to test further.



The whole system is open source. The repository is at http://asm32.info/fossil/repo/asmbb - use the branch RecursiveTemplate. But in order to compile it you will need the FreshLibDev branch of FreshLib from http://fresh.flatassembler.net/fossil/repo/fresh/index

Here is some snapshot that should work:

FreshLibDev.zip; AsmBB

Set %lib% to point to unpacket FreshLibDev.zip "freshlib" directory and compile "source/index.asm"

The script needs all requests from the server.

Best regards.

_________________
Tox ID: A48DEF727DF44C3B5C2E576B65021F1A45D8FA52E2F8E257F1CAE148BBADB162FDF7820BD1F9
Post 28 Mar 2016, 03:59
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia

JohnFound wrote:
...Here is some snapshot that should work:

FreshLibDev.zip; AsmBB

Set %lib% to point to unpacket FreshLibDev.zip "freshlib" directory and compile "source/index.asm"

The script needs all requests from the server.

Best regards.

Hmm, downloaded both zipfiles, compiling is giving me some grief... from the AsmBoard/source dir:

Code:
lib=../../FreshLibDev/freshlib fasm source/index.asm
flat assembler  version 1.71.39  (16384 kilobytes memory)
errorout of stack space.



A quick strace reveals:

Code:
 (...snip...)
gettimeofday({1459139698509797}, NULL) = 0
open("index.asm"O_RDONLY)             = 3
lseek(30SEEK_END)                   = 3034
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...3034) = 3034
close(3)                                = 0
open("../../FreshLibDev/freshlib/freshlib.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 691
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...691) = 691
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros/allmacros.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 832
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...832) = 832
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros/_uniquelists.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 1509
lseek(30SEEK_SET)                   = 0
read(3"; list - name of the list.\r\n; li"...1509) = 1509
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros/_executable.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 1133
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...1133) = 1133
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros//_executable.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 1133
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...1133) = 1133
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros///_executable.inc"O_RDONLY) = 3
lseek(30SEEK_END)                   = 1133
lseek(30SEEK_SET)                   = 0
read(3"; ______________________________"...1133) = 1133
close(3)                                = 0
open("../../FreshLibDev/freshlib/macros////_executable.inc"O_RDONLY) = 3


And so on... what am I doing wrong?

Cheers

_________________
2 Ton Digital - https://2ton.com.au/
Post 28 Mar 2016, 04:37
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
Sorry, I forgot about the other needed env variable: "TargetOS=Linux". But on my computer it still can't be compiled with console FASM, giving "out of memory". Usually It needs 128MB to compile, but now it refuses with even much more. I should investigate it further...
Post 28 Mar 2016, 05:30
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
Ok so, got it all sorted out.

This is the diff output from my HeavyThing-1.16 version's webserver.inc...

Code:
2090,2092d2089
<       ; JohnFound mod:
<       jmp     .fastcgi_doit
<       ; end JohnFound mod
2747d2743

2901,2911d2896
<       ; JohnFound mod:
<       ; So, the other language-based C/C++ based FastCGI implementations do not send
<       ; a Status: header for no-error-condition (200 OK == no error)
<       ; Since JohnFound's version does indeed send a 200 OK Status header, this was
<       ; passing the error code 200 to the webservercfg$error handler, which indeed
<       ; correctly was reporting a 506 unimplemented ERROR CODE, since 200 is not an
<       ; error code.
<       ; to remedy this:
<       cmp     rsi200
<       je      .do_checkcontenttype
<       ; end JohnFound mod




If you add those lines and recompile rwasa, it will send ALL requests via the FastCGI handler, just make sure to pass it a bogus .endswith on the command line arguments (it still redirects to the first supplied fastcgi handler, but ignores the endswith with the above modification 3 liner.

The longer section is the long-winded explanation of what rwasa was doing.

Let me know if that works on your end or if you want me to upload the modified webserver.inc Smile

Cheers

_________________
2 Ton Digital - https://2ton.com.au/
Post 28 Mar 2016, 05:35
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
Sorry here is a patch file for you instead, easier to read than diff...

Code:
*** HeavyThing-1.16/webserver.inc       2016-01-28 10:29:09.320183337 +1000
--- webserver.inc       2016-03-28 15:30:22.617705132 +1000
***************
*** 2087,2092 ****
--- 2087,2095 ----
        cmp     qword [rbx+_avlofs_next], 0
        je      .dofilesystem
        mov     rbx, [rbx+_avlofs_next]
+       ; JohnFound mod:
+       jmp     .fastcgi_doit
+       ; end JohnFound mod
  calign
  .fastcgiwalk:
        mov     rdir15
***************
*** 2741,2746 ****
--- 2744,2750 ----
        mov     rdxrax
        call    epoll$outbound
        ; if epoll$outbound _failed_ to connect, we get a zero in eax

        test    eaxeax
        jz      .fastcgi_error
        ; we return -1 here so that the webserver layer does _nothing_ with our request
***************
*** 2894,2899 ****
--- 2898,2914 ----
        je      .do206
        mov     rdi, [rbx+webserver_config_ofs]
        pop     rsi
+       ; JohnFound mod:
+       ; So, the other language-based C/C++ based FastCGI implementations do not send
+       ; a Status: header for no-error-condition (200 OK == no error)
+       ; Since JohnFound's version does indeed send a 200 OK Status header, this was
+       ; passing the error code 200 to the webservercfg$error handler, which indeed
+       ; correctly was reporting a 506 unimplemented ERROR CODE, since 200 is not an
+       ; error code.
+       ; to remedy this:
+       cmp     rsi200
+       je      .do_checkcontenttype
+       ; end JohnFound mod
        call    webservercfg$error
        mov     rdir13
        mov     r13rax



_________________
2 Ton Digital - https://2ton.com.au/
Post 28 Mar 2016, 05:43
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia

JohnFound wrote:
Sorry, I forgot about the other needed env variable: "TargetOS=Linux". But on my computer it still can't be compiled with console FASM, giving "out of memory". Usually It needs 128MB to compile, but now it refuses with even much more. I should investigate it further...

Same as with the HeavyThing library, I compile with fasm -m 262144 and it worked ok. I had to comment out a missing dependency (uconfig.inc or something like that).

_________________
2 Ton Digital - https://2ton.com.au/
Post 28 Mar 2016, 05:45
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
Oh, I found it. It is naming problem (I am usually compiling it through WINE/Fresh IDE). Simply rename the file "freshlib/data/uConfig.asm" to "uconfig.asm".

Then the following command compiles it for me (but 3GB of needed memory is still very strange) and places the script where it has to be:

Code:
TargetOS="Linux" lib="../../FreshLibDev/freshlib" fasm -m 3000000 index.asm ../www/engine



In the directory www there are configuration files for apache and lighttpd I use for testing. If you run "engine" in "www" it will create "engine.sock" and will wait on it.
Post 28 Mar 2016, 05:56
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
@redsock - white I typed, you manage to post a solution. Thank I will try it. Smile
Post 28 Mar 2016, 06:11
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria
So, with the patched version of rwasa and with some bugfixes in my code Very Happy it works. Below is a screenshot from the AsmBB running on rwasa. Idea

But, redsock, why not to include these changes in the official release? I mean not permanent redirection of course, but syntax like the following:


Code:
rwasa -fastcgi /some_path <sock address>



Where all requests to /some_path/* are considered FastCGI. (and if you set "/" it means the all requests to this site are considered FastCGI). This option is available for all web servers I know.

Also, all web servers I know allow "Status: 200 OK" to be sent.

Also, notice the script running problem. Now I can run the FastCGI script separately, because I am on my own compute. But imagine it was on shared hosting where I can't just start programs on the server.

Best regards.


Description:
Filesize: 16.91 KB
Viewed: 1136 Time(s)

with_rwasa_small.png



_________________
Tox ID: A48DEF727DF44C3B5C2E576B65021F1A45D8FA52E2F8E257F1CAE148BBADB162FDF7820BD1F9
Post 28 Mar 2016, 07:51
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia

JohnFound wrote:
...But, redsock, why not to include these changes in the official release? I mean not permanent redirection of course, but syntax like the following:


Code:
rwasa -fastcgi /some_path <sock address>



Where all requests to /some_path/* are considered FastCGI. (and if you set "/" it means the all requests to this site are considered FastCGI). This option is available for all web servers I know.

Also, all web servers I know allow "Status: 200 OK" to be sent.

Also, notice the script running problem. Now I can run the FastCGI script separately, because I am on my own compute. But imagine it was on shared hosting where I can't just start programs on the server.

Best regards.

I will include the changes in the next release Smile There are a few items left outstanding before 1.17 comes out but when I get to those, will be sure to include these updates for rwasa.

How does process management for FastCGI backends work with Apache and/or nginx? Fork/exec at startup is certainly doable (even fork/exec during worker process runtime, though more messy)... I have not seen FastCGI coupled to webservers this way in production environments before.

Anyway, glad the patch worked for you.

Cheers!

_________________
2 Ton Digital - https://2ton.com.au/
Post 28 Mar 2016, 21:07
View user's profile Send private message Reply with quote
JohnFound



Joined: 16 Jun 2003
Posts: 3434
Location: Bulgaria

redsock wrote:
How does process management for FastCGI backends work with Apache and/or nginx? Fork/exec at startup is certainly doable (even fork/exec during worker process runtime, though more messy)... I have not seen FastCGI coupled to webservers this way in production environments before.



In both Apache and lighttpd, the FastCGI processes are started depending on the load. If the response from the process start to delay more than acceptable, new one is started and the requests are sent there, until the previous processes finish their work. If the process hangs it is terminated. If there are no requests for long time, even all child processes can be terminated.

You didn't see such use, because usually FastCGI is used only for executing language interpreters where the interpreter is started by the system administrator. This is possible of course only if you own the server. It is not an option for shared hosting web hosting.

_________________
Tox ID: A48DEF727DF44C3B5C2E576B65021F1A45D8FA52E2F8E257F1CAE148BBADB162FDF7820BD1F9
Post 29 Mar 2016, 03:54
View user's profile Send private message Visit poster's website ICQ Number Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
Hey all,

In donating the virtual server for Tomasz and the community here, and in part due to the above discussion with @JohnFound re: rwasa (and that it is now running this board), CGI, FastCGI, et al, I got motivated and did the v1.17 release of my library today. Check out the ChangeLog for the nitty-gritty details: https://2ton.com.au/library_as_html/ChangeLog.html

Added lots of new functionality to the library as well as a decent handful of bugfixes.

Let me know if you have any issues.

Cheers!

_________________
2 Ton Digital - https://2ton.com.au/
Post 10 Aug 2016, 04:42
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
Another update today... v1.18 released... small Fossil related bugfix for @JohnFound and https://fresh.flatassembler.net in dealing with compressed FastCGI redirect responses...

And I included a simple but fast http://dev.maxmind.com/geoip/geoip2/geolite2/ parser and lookup utility (as HTML: https://2ton.com.au/library_as_html/maxmind.inc.html)

Cheers!

_________________
2 Ton Digital - https://2ton.com.au/
Post 30 Aug 2016, 04:48
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia
Another small update today... v1.19 released (https://2ton.com.au/HeavyThing/). A bug I introduced in 1.18 for the way that rwasa was dealing with FastCGI redirection combined with Set-Cookie got fixed up (same one that caused the persistent logins here on the board to malfunction there for a short while, though the board got the 1.19 rwasa straight away I didn't do a new library release til today).

Also included are a few new library examples:

TCP outbound example in simple_socket for both IPv4 and hostname-based epoll sockets.

A https://download.libsodium.org/doc/ HeavyThing example for crypto_box_easy and crypto_box_open_easy.

And last but not least, a SHA3/256 Keccak example similar to the SHA256 one already included.

Cheers as usual! Smile

_________________
2 Ton Digital - https://2ton.com.au/
Post 13 Sep 2016, 00:06
View user's profile Send private message Reply with quote
sleepsleep



Joined: 05 Oct 2006
Posts: 6659
Location: ˛                              ⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣ Posts: 6699
thanks redsock,
/me started using rwasa 1.19

question
- is it possible to allow it to lists files in sandbox directory?
- is it possible to allow it prints client GET, POST etc, when running -foreground?

eg

Code:

[Tue Sep 13 2016 23:47:06 GMT+0000 (GMT)] "GET /" "Mozilla/5.0 (Windows NT 6.1; WOW128; rv:37.0) Gecko/20100101 Firefox/32.0"
[Tue Sep 13 2016 23:47:06 GMT+0000 (GMT)] "GET /css/phonon.css" "Mozilla/5.0 (Windows NT 6.1; WOW128; rv:37.0) Gecko/20100101 Firefox/32.0"
[Tue Sep 13 2016 23:47:06 GMT+0000 (GMT)] "GET /js/phonon.js" "Mozilla/5.0 (Windows NT 6.1; WOW128; rv:37.0) Gecko/20100101 Firefox/32.0"


Post 13 Sep 2016, 15:52
View user's profile Send private message Reply with quote
redsock



Joined: 09 Oct 2009
Posts: 261
Location: Australia

sleepsleep wrote:
thanks redsock,
/me started using rwasa 1.19

question
- is it possible to allow it to lists files in sandbox directory?
- is it possible to allow it prints client GET, POST etc, when running -foreground?

Logs to stdout is an interesting idea, I will add that to the list... in the meantime, there is a define at the top of webserver.inc in the library called webserverdebug. If you set it to 1 and recompile rwasa, you'll get full request headers to stdout.

By "list files in sandbox directory" I assume you mean old-school directory listings when requests arrive for /some/directory/ yeah? I debated originally when I designed rwasa whether or not to include such functionality, and concluded then that since there are HTML, CSS, images, etc required for same, those should be left in the hands of the webops teams. As such, you'd have to use some kind of -indexfiles option to specify your own directory listing handler (which could then reasonably produce a styled directory listing w/ HTML, CSS, image links, etc).

_________________
2 Ton Digital - https://2ton.com.au/
Post 13 Sep 2016, 20:56
View user's profile Send private message Reply with quote
sleepsleep



Joined: 05 Oct 2006
Posts: 6659
Location: ˛                              ⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣⁣ Posts: 6699
thanks redsock,
i reassemble back and got http requests now,


redsock wrote:

I assume you mean old-school directory listings when requests arrive for /some/directory/ yeah?


yes,


redsock wrote:
those should be left in the hands of the webops teams


but how webops team could generate those files listing dynamically?
Post 14 Sep 2016, 04:20
View user's profile Send private message Reply with quote
Display posts from previous:
Post new topic Reply to topic

Jump to:  
Goto page Previous  1, 2, 3  Next

< Last Thread | Next Thread >

Forum Rules:
You cannot post new topics in this forum
You cannot reply to topics in this forum
You cannot edit your posts in this forum
You cannot delete your posts in this forum
You cannot vote in polls in this forum
You cannot attach files in this forum
You can download files in this forum


Powered by phpBB © 2001-2005 phpBB Group.

Main index   Download   Documentation   Examples   Message board
Copyright © 2004-2016, Tomasz Grysztar.