AsmBB

Power
Login Register

Learning the AsmBB source
0

#15164 (ツ) pewm
Created 24.12.2016, read: 22516 times

What a fascinating project. The fact that the whole thing is written in assembly has stuck with me after the first time I saw it. I've kept coming back to this as an example of something that breaks your whole paradigm and smacks you over the head to show that some things you know may not be correct.

So we can program websites in assembly language. Who would have thought?

The question I have is where should we focus on learning the AsmBB source? I've downloaded it and begun to comb through it, but a starting point may help me find out more about how you managed to successfully get the whole thing working. I'm more interested in learning assembly, but I've found that jumping headlong into examples does me more good than just writing small bits of code that don't do much.

#15165 (ツ) johnfound
Created 24.12.2016, read: 22511 times
pewm

The question I have is where should we focus on learning the AsmBB source? I've downloaded it and begun to comb through it, but a starting point may help me find out more about how you managed to successfully get the whole thing working. I'm more interested in learning assembly, but I've found that jumping headlong into examples does me more good than just writing small bits of code that don't do much.

Well, I can help by shortly describing the structure of the project. But notice that because of using FastCGI protocol, AsmBB is a little bit more complex than usual for beginner assembly programmer.

The simple CGI web application (for example my CMS MiniMagAsm) works by short living CGI processes. They are started by the web server, receive the HTTP request by reading STDIN, write the response (e.g. HTML) to STDOUT and terminate.

The FastCGI application works a little bit different. At first it works all the time and listens for connections on some socket. The web server connects to this socket and sends the request. Then the FastCGI application sends back the response using the same connection. Serving multiply requests on the same connection and multiply connections are possible. Also, the web server can start several instances of the FastCGI application. It is all about a performance.

So, the structure of AsmBB:

Everything starts in engine.asm, line:84 - the label start:. Here the engine initializes its environments and opens the SQLite database with the forum data. Then it calls the procedure Listen (line: 126) that is the main loop of the program. It returns only when the engine terminates. After retuning from Listen, the program closes the database, finalizes its works and exits.

The procedure Listen is defined in the source file fcgi.asm:155. This is the code, that handles FastCGI protocol. It simply listens for connections from the web server and on connection creates a thread that to serve the request and continues to listen. The thread is started in the procedure procServeRequest in fcgi.asm:347.

This thread, receives the request information on the socket and collects it for future processing.

Once the whole information is collected - i.e. the HTTP reques, the POST data (if any) and the environment parameters, the procedure ServeOneRequest is called. It is actually the "business logic" code that makes the application to act as a web forum.

ServeOneRequest returns the response (e.g. HTML code, images, etc.) that is returned to the web server, according to FastCGI protocol. After completing the request, the thread terminates or stays waiting for another request. It depends on how the web server can multiplex the requests on the FastCGI connection.

ServeOneRequest is located in the file commands.asm:49. It analyzes the request URL and the request type (GET, POST) and decides how to serve it. For example, it can return some file directly, or read the information from the database, or store some information in the database.

The URL analyze is important and located on commands.asm:332 - it dispatches the control depending on the URL elements. The addresses of the different procedures are loaded to ECX and later called (label .exec_command, line: 524 or .exec_command2:, line: 541).

Later you can browse these procedures. They are located in different files, serving different aspects of forum engine. For example, the procedure ListThreads (threadlist.asm) creates the list of threads on the front page of the forum.

The procedure ShowThread (showthread.asm) displays one thread. And so on.

Notice, that AsmBB widely uses the library FreshLib. You can read more in FreshLib reference and FreshLib user guide, but unfortunately the documentation is far from perfect.

In order to better browse the big code, scattered across multiply files, I would suggest using Fresh IDE code browsing features. Read more in the following article: Tips and tricks. Especially useful if the feature "Goto definition" (Ctrl+D) that will jump you at the line where some label is defined. The cross reference is also useful (Ctrl+R).

FreshLib user guide

#15166 (ツ) pewm
Created 25.12.2016, read: 22492 times

Thanks, That's exactly what I was looking for. I'm going to start to dig in and take a look.

One question, when you are using the term "thread" are you referring to a thread similar to linux pthreads? Or thread as in another line of execution that still runs serially?

A note on FreshLib. I read they use stdcall calling convention. Is that the standard used throughout, not cdecl?

Also the FreshIDE looks awesome, I had already attempted to compile it on Linux, but it appears I needed to set up wine to get that all working. Instead, I've always stuck with Vim and used ctags to generate tags and do the definition lookups, plus using "lv" + "lw" commands in Vim, which gets me pretty far. Any suggestions for someone who is completely indoctrinated into the Vim + ctags + bash workflow?

Thanks again for replying. That is extremely helpful.

#15168 (ツ) johnfound
Created 30.12.2016, read: 22484 times
pewm

One question, when you are using the term "thread" are you referring to a thread similar to linux pthreads? Or thread as in another line of execution that still runs serially?

Yes, thread is the same as in "pthreads". Some code, executed in parallel but in the same memory environment as the main program.

pewm

A note on FreshLib. I read they use stdcall calling convention. Is that the standard used throughout, not cdecl?

FreshLib uses some modified stdcall, because it is more native for x86 CPUs, than cdecl. The modifications of the conventions is that FreshLib uses CF for returning boolean values and often returns more than one value in different registers.

pewm

Also the FreshIDE looks awesome, I had already attempted to compile it on Linux, but it appears I needed to set up wine to get that all working.

Fresh IDE is some kind of hybrid application. It knows about Linux and actively use its features. In fact, it works in Linux better than in Windows. Developing portable and Windows applications in Linux also depends on WINE.

But if you are looking for native application, you should wait for v3.0 of Fresh IDE that is in active, but slow development.

pewm

Instead, I've always stuck with Vim and used ctags to generate tags and do the definition lookups, plus using "lv" + "lw" commands in Vim, which gets me pretty far. Any suggestions for someone who is completely indoctrinated into the Vim + ctags + bash workflow?

Of course, it is possible. You will need FASM installed. Then the following environment variables set:

lib - The path to the FreshLib directory.

TargetOS - The platform you want to compile for. Currently supported values are "Win32", "Linux" and partially "KolibriOS".

Then compile the main project file. (i.e. engine.asm for AsmBB) with:

fasm -m 150000 engine.asm
#15169 (ツ) pewm
Created 25.12.2016, read: 22478 times
johnfound

Of course, it is possible. You will need FASM installed. Then the following environment variables set:

lib - The path to the FreshLib directory.

TargetOS - The platform you want to compile for. Currently supported values are "Win32", "Linux" and partially "KolibriOS".

Then compile the main project file. (i.e. engine.asm for AsmBB) with:

fasm -m 150000 engine.asm

Yes, I have those environment variables set. That's the same way I attempted to compile FreshIDE. I had wanted to run a compile on Linux for the FreshIDE, but ran into some snags along the way. Will have to get Wine set up before attempting a compile again.

#15170 (ツ) pewm
Created 25.12.2016, read: 22473 times
johnfound

Yes, thread is the same as in "pthreads". Some come, executed in parallel but in the same memory environment as the main program.

Ok, I was looking around for some good examples of threading with assembly. Best I can tell, the only way is through the kernel with a system call unless we set up our own scheduler of sorts. Good to find an example.

Have you thought of a thread pool instead of spawning a thread on the fly? I've found that thread creation is fairly expensive and, if I understand what you're saying then it's likely the thread is created through the kernel for each new request. The reason I ask is it's a potential place I could cut my teeth on toying around with the source code. For instance make it where the threading isn't spawned on the fly, rather that the threads are created beforehand based on the CPU cores + hyperthreading and any new job is just a matter of handing that off to the preexisting threads. This might help with extremely high loads. (of course it may be laughable to get 'better' performance out of something that is already operating so damn fast). :)

#15171 (ツ) johnfound
Created 25.12.2016, read: 22468 times
pewm

Yes, I have those environment variables set. That's the same way I attempted to compile FreshIDE. I had wanted to run a compile on Linux for the FreshIDE, but ran into some snags along the way. Will have to get Wine set up before attempting a compile again.

Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32". Actually I didn't tried, but it should work and if not, it is definitely a bug.

But of course you can run this .exe file only through WINE. I am working on OS independent GUI toolkit in FreshLib, but it is still not powerful enough to provide Fresh IDE functionality.

pewm

I've found that thread creation is fairly expensive and, if I understand what you're saying then it's likely the thread is created through the kernel for each new request.

Well, you are right, while the threads are faster than CGI processes spawning, they actually slower the program, compared to the fastest possible performance.

But notice, that FastCGI protocol allows reusing connections (i.e. threads), this way improving the performance. Of course, this depends on how the web server implements FastCGI protocol. The AsmBB handles this case properly.

Even better performance can be provided by using non blocking sockets, but I don't think it will make the things much better in this very application.

#15172 (ツ) pewm
Created 26.12.2016, read: 22463 times
johnfound

Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32".

Of course! I don't know why I didn't think of it that way. Going to try this.

johnfound

But notice, that FastCGI protocol allows reusing connections (i.e. threads), this way improving the performance. Of course, this depends on how the web server implements FastCGI protocol. The AsmBB handles this case properly.

Ah, I didn't take into account FastCGI's reuse feature. The new thread is started on the first connection and carried on throughout multiple connections. Got it.

#15173 (ツ) pewm
Created 26.12.2016, read: 22460 times
johnfound

Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32". Actually I didn't tried, but it should work and if not, it is definitely a bug.

SOLVED

flat assembler  version 1.71.54  (150000 kilobytes memory)
freshide/freshlib/equates/Win32/allequates.inc [18]:
include '_kernel32.inc'
error: file not found.

I found that using the Linux version of Fasm doesn't find certain dependencies. One of which are \_kernel32.inc. Compiling the windows version down in fasmsrc/Win32 using

wine fasm Fasm.asm

and then running

wine fasmsrc/Win32/Fasm.exe Fresh.asm

and then

wine Fresh.exe

solved all my issues. Damn those compiles are fast.

Anyways, I'm up and running with Fresh v2.5.1 on Linux.

By the way, My environment variables for this are

export lib=freshide/freshlib
export TargetOS=Win32

For those following after me trying the same thing.

#15174 (ツ) johnfound
Created 26.12.2016, read: 22452 times
pewm
flat assembler  version 1.71.54  (150000 kilobytes memory)
freshide/freshlib/equates/Win32/allequates.inc [18]:
include '_kernel32.inc'
error: file not found.

I found that using the Linux version of Fasm doesn't find certain dependencies. One of which are \_kernel32.inc.

So it is a bug. Actually it is a heritage from the DOS times. FASM for Windows handles the filenames as case insensitive, while Linux version as case sensitive. For example, the real name of _kernel32.inc is _KERNEL32.INC.

It is fixed now in 06420cf49f2db03e version and FASM for linux compiles everything fine.

#15175 (ツ) pewm
Created 26.12.2016, read: 22445 times
johnfound

It is fixed now in 06420cf49f2db03e version and FASM for linux compiles everything fine.

confirmed this compile works now with Linux version of FASM

#15177 (ツ) pewm
Created 26.12.2016, read: 22433 times
johnfound

Then compile the main project file. (i.e. engine.asm for AsmBB) with:

fasm -m 150000 engine.asm

SOLVED I seemed to get an error with the Linux compile of AsmBB on Linux. The error is

flat assembler  version 1.71.54  (150000 kilobytes memory)
Initialized data address: $804F20C
engine.asm [252]:
endp
freshide/freshlib/macros/Linux/_executable.inc [52] postpone [10]:
        \IncludeAllGlobals
freshide/freshlib/macros/_globals.inc [218] IncludeAllGlobals [6]:
  IncludeIGlobals
freshide/freshlib/macros/_globals.inc [44] IncludeIGlobals [6]:
    I
freshide/freshlib/macros/_globals.inc [44] match [1]:
    I
freshide/freshlib/macros/_globals.inc [40] IGlobals [1]:
    __IGlobalBlock
version.asm [6] __IGlobalBlock [4]:
              file "../manifest.uuid":0,16
error: file not found.

also, compiling with Win32 TargetOS and Wine:

PC:~/asmbb/source$ wine fasmsrc/fasm/source/Win32/fasm.exe -m 150000 engine.asm
flat assembler  version 1.71.58  (150000 kilobytes memory)
Initialized data address: $4083D4
engine.asm [252]:
endp
freshide/freshlib/macros/Win32/_executable.inc [68] postpone [24]:
        \IncludeAllGlobals
freshide/freshlib/macros/_globals.inc [218] IncludeAllGlobals [6]:
  IncludeIGlobals
freshide/freshlib/macros/_globals.inc [44] IncludeIGlobals [6]:
    I
freshide/freshlib/macros/_globals.inc [44] match [1]:
    I
freshide/freshlib/macros/_globals.inc [40] IGlobals [1]:
    __IGlobalBlock
version.asm [6] __IGlobalBlock [4]:
              file "../manifest.uuid":0,16
processed: file '../manifest.uuid':0,16
error: file not found.

I fixed the error with this change to version.asm in AsmBB

PC:~/asmbb/source$ diff version.asm version_fixed.asm
6c6
<               file "../manifest.uuid":0,16
---
>               file "%lib%/../manifest.uuid":0,16
8c8
<               file "../manifest.uuid":0,16
---
>               file "%lib%/../manifest.uuid":0,16

Everything compiles now under Linux.

#15178 (ツ) johnfound
Created 27.12.2016, read: 22419 times
pewm

SOLVED I seemed to get an error with the Linux compile of AsmBB on Linux. The error is

              file "../manifest.uuid":0,16
error: file not found.

The file manifest.uuid is generated by fossil SCM, and contains the current version of the source code. If you downloaded the source as a zip archive this file will miss. Also, fossil settings can stop this file from generation.

In fact having this file is not so important. It is used only for displaying the checkin identifier of the engine down in the footer of the forum:

AsmBB v1.2 (check-in: c6221a60f2d93916)

You can fix the problem by simply creating file with 16 ascii symbols that will be used as a version identifier. The fossil command that will enable generating the file is:

fossil settings manifest on
#15180 (ツ) pewm
Created 27.12.2016, read: 22409 times

Gotcha. In that case I pointed it to the wrong one.

#15181 (ツ) johnfound
Created 27.12.2016, read: 22407 times
pewm

Gotcha. In that case I pointed it to the wrong one.

Yes, probably to the manifest of FreshLib checkout. But as I said, it is not so important for the functionality of the program. It is only a string message.

Learning the AsmBB source
0

AsmBB v3.0 (check-in: a316dab8b98d07d9); SQLite v3.42.0 (check-in: 831d0fb2836b71c9);
©2016..2023 John Found; Licensed under EUPL. Powered by Assembly language Created with Fresh IDE