What a fascinating project. The fact that the whole thing is written in assembly has stuck with me after the first time I saw it. I've kept coming back to this as an example of something that breaks your whole paradigm and smacks you over the head to show that some things you know may not be correct.
So we can program websites in assembly language. Who would have thought?
The question I have is where should we focus on learning the AsmBB source? I've downloaded it and begun to comb through it, but a starting point may help me find out more about how you managed to successfully get the whole thing working. I'm more interested in learning assembly, but I've found that jumping headlong into examples does me more good than just writing small bits of code that don't do much.
Well, I can help by shortly describing the structure of the project. But notice that because of using FastCGI protocol, AsmBB is a little bit more complex than usual for beginner assembly programmer.
The simple CGI web application (for example my CMS MiniMagAsm) works by short living CGI processes. They are started by the web server, receive the HTTP request by reading STDIN, write the response (e.g. HTML) to STDOUT and terminate.
The FastCGI application works a little bit different. At first it works all the time and listens for connections on some socket. The web server connects to this socket and sends the request. Then the FastCGI application sends back the response using the same connection. Serving multiply requests on the same connection and multiply connections are possible. Also, the web server can start several instances of the FastCGI application. It is all about a performance.
So, the structure of AsmBB:
Everything starts in engine.asm, line:84 - the label start:. Here the engine initializes its environments and opens the SQLite database with the forum data. Then it calls the procedure Listen (line: 126) that is the main loop of the program. It returns only when the engine terminates. After retuning from Listen, the program closes the database, finalizes its works and exits.
The procedure Listen is defined in the source file fcgi.asm:155. This is the code, that handles FastCGI protocol. It simply listens for connections from the web server and on connection creates a thread that to serve the request and continues to listen. The thread is started in the procedure procServeRequest in fcgi.asm:347.
This thread, receives the request information on the socket and collects it for future processing.
Once the whole information is collected - i.e. the HTTP reques, the POST data (if any) and the environment parameters, the procedure ServeOneRequest is called. It is actually the "business logic" code that makes the application to act as a web forum.
ServeOneRequest returns the response (e.g. HTML code, images, etc.) that is returned to the web server, according to FastCGI protocol. After completing the request, the thread terminates or stays waiting for another request. It depends on how the web server can multiplex the requests on the FastCGI connection.
ServeOneRequest is located in the file commands.asm:49. It analyzes the request URL and the request type (GET, POST) and decides how to serve it. For example, it can return some file directly, or read the information from the database, or store some information in the database.
The URL analyze is important and located on commands.asm:332 - it dispatches the control depending on the URL elements. The addresses of the different procedures are loaded to ECX and later called (label .exec_command, line: 524 or .exec_command2:, line: 541).
Later you can browse these procedures. They are located in different files, serving different aspects of forum engine. For example, the procedure ListThreads (threadlist.asm) creates the list of threads on the front page of the forum.
The procedure ShowThread (showthread.asm) displays one thread. And so on.
Notice, that AsmBB widely uses the library FreshLib. You can read more in FreshLib reference and FreshLib user guide, but unfortunately the documentation is far from perfect.
In order to better browse the big code, scattered across multiply files, I would suggest using Fresh IDE code browsing features. Read more in the following article: Tips and tricks. Especially useful if the feature "Goto definition" (Ctrl+D) that will jump you at the line where some label is defined. The cross reference is also useful (Ctrl+R).
Thanks, That's exactly what I was looking for. I'm going to start to dig in and take a look.
One question, when you are using the term "thread" are you referring to a thread similar to linux pthreads? Or thread as in another line of execution that still runs serially?
A note on FreshLib. I read they use stdcall calling convention. Is that the standard used throughout, not cdecl?
Also the FreshIDE looks awesome, I had already attempted to compile it on Linux, but it appears I needed to set up wine to get that all working. Instead, I've always stuck with Vim and used ctags to generate tags and do the definition lookups, plus using "lv" + "lw" commands in Vim, which gets me pretty far. Any suggestions for someone who is completely indoctrinated into the Vim + ctags + bash workflow?
Thanks again for replying. That is extremely helpful.
Yes, thread is the same as in "pthreads". Some code, executed in parallel but in the same memory environment as the main program.
FreshLib uses some modified stdcall, because it is more native for x86 CPUs, than cdecl. The modifications of the conventions is that FreshLib uses CF for returning boolean values and often returns more than one value in different registers.
Fresh IDE is some kind of hybrid application. It knows about Linux and actively use its features. In fact, it works in Linux better than in Windows. Developing portable and Windows applications in Linux also depends on WINE.
But if you are looking for native application, you should wait for v3.0 of Fresh IDE that is in active, but slow development.
Of course, it is possible. You will need FASM installed. Then the following environment variables set:
lib - The path to the FreshLib directory.
TargetOS - The platform you want to compile for. Currently supported values are "Win32", "Linux" and partially "KolibriOS".
Then compile the main project file. (i.e. engine.asm for AsmBB) with:
Yes, I have those environment variables set. That's the same way I attempted to compile FreshIDE. I had wanted to run a compile on Linux for the FreshIDE, but ran into some snags along the way. Will have to get Wine set up before attempting a compile again.
Ok, I was looking around for some good examples of threading with assembly. Best I can tell, the only way is through the kernel with a system call unless we set up our own scheduler of sorts. Good to find an example.
Have you thought of a thread pool instead of spawning a thread on the fly? I've found that thread creation is fairly expensive and, if I understand what you're saying then it's likely the thread is created through the kernel for each new request. The reason I ask is it's a potential place I could cut my teeth on toying around with the source code. For instance make it where the threading isn't spawned on the fly, rather that the threads are created beforehand based on the CPU cores + hyperthreading and any new job is just a matter of handing that off to the preexisting threads. This might help with extremely high loads. (of course it may be laughable to get 'better' performance out of something that is already operating so damn fast). :)
Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32". Actually I didn't tried, but it should work and if not, it is definitely a bug.
But of course you can run this .exe file only through WINE. I am working on OS independent GUI toolkit in FreshLib, but it is still not powerful enough to provide Fresh IDE functionality.
Well, you are right, while the threads are faster than CGI processes spawning, they actually slower the program, compared to the fastest possible performance.
But notice, that FastCGI protocol allows reusing connections (i.e. threads), this way improving the performance. Of course, this depends on how the web server implements FastCGI protocol. The AsmBB handles this case properly.
Even better performance can be provided by using non blocking sockets, but I don't think it will make the things much better in this very application.
Of course! I don't know why I didn't think of it that way. Going to try this.
Ah, I didn't take into account FastCGI's reuse feature. The new thread is started on the first connection and carried on throughout multiple connections. Got it.
flat assembler version 1.71.54 (150000 kilobytes memory)
freshide/freshlib/equates/Win32/allequates.inc [18]:
include '_kernel32.inc'
error: file not found.
I found that using the Linux version of Fasm doesn't find certain dependencies. One of which are \_kernel32.inc. Compiling the windows version down in fasmsrc/Win32 using
wine fasm Fasm.asm
and then running
wine fasmsrc/Win32/Fasm.exe Fresh.asm
and then
wine Fresh.exe
solved all my issues. Damn those compiles are fast.
Anyways, I'm up and running with Fresh v2.5.1 on Linux.
So it is a bug. Actually it is a heritage from the DOS times. FASM for Windows handles the filenames as case insensitive, while Linux version as case sensitive. For example, the real name of _kernel32.inc is _KERNEL32.INC.
It is fixed now in 06420cf49f2db03e version and FASM for linux compiles everything fine.
The file manifest.uuid is generated by fossil SCM, and contains the current version of the source code. If you downloaded the source as a zip archive this file will miss. Also, fossil settings can stop this file from generation.
In fact having this file is not so important. It is used only for displaying the checkin identifier of the engine down in the footer of the forum:
AsmBB v1.2 (check-in: c6221a60f2d93916)
You can fix the problem by simply creating file with 16 ascii symbols that will be used as a version identifier. The fossil command that will enable generating the file is:
Yes, probably to the manifest of FreshLib checkout. But as I said, it is not so important for the functionality of the program. It is only a string message.