..:: AsmBB ::..: Learning the AsmBB source
<img src="https://board.asm32.info/images/title.svg" alt="Title img">
<h1>AsmBB is ultrafast web forum, written entirely in assembly language. This site is the official support development forum and demo/test installation.</h1>
tag:board.asm32.info,2018-03-06:Thread642016-12-30T21:08:20Zjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151682016-12-30T21:08:20Z
<blockquote><header>pewm</header><p>One question, when you are using the term "thread" are you referring to a thread similar to linux pthreads? Or thread as in another line of execution that still runs serially?
</p></blockquote>
<p>Yes, thread is the same as in "pthreads". Some code, executed in parallel but in the same memory environment as the main program.
</p>
<blockquote><header>pewm</header><p>A note on FreshLib. I read they use stdcall calling convention. Is that the standard used throughout, not cdecl?
</p></blockquote>
<p>FreshLib uses some modified stdcall, because it is more native for x86 CPUs, than cdecl. The modifications of the conventions is that FreshLib uses CF for returning boolean values and often returns more than one value in different registers.
</p>
<blockquote><header>pewm</header><p>Also the FreshIDE looks awesome, I had already attempted to compile it on Linux, but it appears I needed to set up wine to get that all working.
</p></blockquote>
<p>Fresh IDE is some kind of hybrid application. It knows about Linux and actively use its features. In fact, it works in Linux better than in Windows. Developing portable and Windows applications in Linux also depends on WINE.
</p>
<p>But if you are looking for native application, you should wait for v3.0 of Fresh IDE that is in active, but slow development.
</p>
<blockquote><header>pewm</header><p>Instead, I've always stuck with Vim and used ctags to generate tags and do the definition lookups, plus using "lv" + "lw" commands in Vim, which gets me pretty far. Any suggestions for someone who is completely indoctrinated into the Vim + ctags + bash workflow?
</p></blockquote>
<p>Of course, it is possible. You will need FASM installed. Then the following environment variables set:
</p>
<p><code>lib</code> - The path to the FreshLib directory.
</p>
<p><code>TargetOS</code> - The platform you want to compile for. Currently supported values are "Win32", "Linux" and partially "KolibriOS".
</p>
<p>Then compile the main project file. (i.e. <code>engine.asm</code> for AsmBB) with:
</p>
<pre><code class="">fasm -m 150000 engine.asm
</code></pre>
johnfoundjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151812016-12-27T12:37:12Z
<blockquote><header>pewm</header><p>Gotcha. In that case I pointed it to the wrong one.
</p></blockquote>
<p>Yes, probably to the manifest of FreshLib checkout. But as I said, it is not so important for the functionality of the program. It is only a string message.</p>
johnfoundpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151802016-12-27T12:34:30Z
<p>Gotcha. In that case I pointed it to the wrong one.</p>
pewmjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151782016-12-27T05:59:12Z
<blockquote><header>pewm</header><p>SOLVED
I seemed to get an error with the Linux compile of AsmBB on Linux. The error is
</p>
<pre><code class=""> file "../manifest.uuid":0,16
error: file not found.
</code></pre></blockquote>
<p>The file manifest.uuid is generated by fossil SCM, and contains the current version of the source code. If you downloaded the source as a zip archive this file will miss. Also, fossil settings can stop this file from generation.
</p>
<p>In fact having this file is not so important. It is used only for displaying the checkin identifier of the engine down in the footer of the forum:
</p><pre><code class="">AsmBB v1.2 (check-in: c6221a60f2d93916)
</code></pre><p>You can fix the problem by simply creating file with 16 ascii symbols that will be used as a version identifier. The fossil command that will enable generating the file is:
</p><pre><code class="">fossil settings manifest on
</code></pre>
johnfoundpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151772016-12-26T20:03:14Z
<blockquote><header>johnfound</header>
<p>Then compile the main project file. (i.e. <code>engine.asm</code> for AsmBB) with:
</p>
<pre><code class="">fasm -m 150000 engine.asm
</code></pre></blockquote>
<p>SOLVED
I seemed to get an error with the Linux compile of AsmBB on Linux. The error is
</p>
<pre><code class="">flat assembler version 1.71.54 (150000 kilobytes memory)
Initialized data address: $804F20C
engine.asm [252]:
endp
freshide/freshlib/macros/Linux/_executable.inc [52] postpone [10]:
\IncludeAllGlobals
freshide/freshlib/macros/_globals.inc [218] IncludeAllGlobals [6]:
IncludeIGlobals
freshide/freshlib/macros/_globals.inc [44] IncludeIGlobals [6]:
I
freshide/freshlib/macros/_globals.inc [44] match [1]:
I
freshide/freshlib/macros/_globals.inc [40] IGlobals [1]:
__IGlobalBlock
version.asm [6] __IGlobalBlock [4]:
file "../manifest.uuid":0,16
error: file not found.
</code></pre>
<p>also, compiling with Win32 TargetOS and Wine:
</p><pre><code class="">PC:~/asmbb/source$ wine fasmsrc/fasm/source/Win32/fasm.exe -m 150000 engine.asm
flat assembler version 1.71.58 (150000 kilobytes memory)
Initialized data address: $4083D4
engine.asm [252]:
endp
freshide/freshlib/macros/Win32/_executable.inc [68] postpone [24]:
\IncludeAllGlobals
freshide/freshlib/macros/_globals.inc [218] IncludeAllGlobals [6]:
IncludeIGlobals
freshide/freshlib/macros/_globals.inc [44] IncludeIGlobals [6]:
I
freshide/freshlib/macros/_globals.inc [44] match [1]:
I
freshide/freshlib/macros/_globals.inc [40] IGlobals [1]:
__IGlobalBlock
version.asm [6] __IGlobalBlock [4]:
file "../manifest.uuid":0,16
processed: file '../manifest.uuid':0,16
error: file not found.
</code></pre>
<p>I fixed the error with this change to version.asm in AsmBB
</p>
<pre><code class="">PC:~/asmbb/source$ diff version.asm version_fixed.asm
6c6
< file "../manifest.uuid":0,16
---
> file "%lib%/../manifest.uuid":0,16
8c8
< file "../manifest.uuid":0,16
---
> file "%lib%/../manifest.uuid":0,16
</code></pre>
<p>Everything compiles now under Linux.
</p>
pewmpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151752016-12-26T14:39:12Z
<blockquote><header>johnfound</header><p>It is fixed now in <a href="https://fresh.flatassembler.net/fossil/repo/fresh/info/06420cf49f2db03e">06420cf49f2db03e</a> version and FASM for linux compiles everything fine.
</p>
</blockquote>
<p>confirmed this compile works now with Linux version of FASM</p>
pewmjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151742016-12-26T07:45:14Z
<blockquote><header>pewm</header><pre><code class="">flat assembler version 1.71.54 (150000 kilobytes memory)
freshide/freshlib/equates/Win32/allequates.inc [18]:
include '_kernel32.inc'
error: file not found.
</code></pre>
<p>I found that using the Linux version of Fasm doesn't find certain dependencies. One of which are \_kernel32.inc.
</p></blockquote>
<p>So it is a bug. Actually it is a heritage from the DOS times. FASM for Windows handles the filenames as case insensitive, while Linux version as case sensitive. For example, the real name of <code>_kernel32.inc</code> is <code>_KERNEL32.INC</code>.
</p>
<p>It is fixed now in <a href="https://fresh.flatassembler.net/fossil/repo/fresh/info/06420cf49f2db03e">06420cf49f2db03e</a> version and FASM for linux compiles everything fine.
</p>
johnfoundpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151732016-12-26T03:22:43Z
<blockquote><header>johnfound</header><p>Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32". Actually I didn't tried, but it should work and if not, it is definitely a bug.
</p></blockquote>
<p>SOLVED
</p><pre><code class="">flat assembler version 1.71.54 (150000 kilobytes memory)
freshide/freshlib/equates/Win32/allequates.inc [18]:
include '_kernel32.inc'
error: file not found.
</code></pre>
<p>I found that using the Linux version of Fasm doesn't find certain dependencies. One of which are \_kernel32.inc. Compiling the windows version down in fasmsrc/Win32 using
</p><pre><code class="">wine fasm Fasm.asm
</code></pre>
<p>and then running
</p><pre><code class="">wine fasmsrc/Win32/Fasm.exe Fresh.asm
</code></pre>
<p>and then
</p><pre><code class="">wine Fresh.exe
</code></pre>
<p>solved all my issues. Damn those compiles are fast.
</p>
<p>Anyways, I'm up and running with Fresh v2.5.1 on Linux.
</p>
<p>By the way, My environment variables for this are
</p><pre><code class="">export lib=freshide/freshlib
export TargetOS=Win32
</code></pre>
<p>For those following after me trying the same thing.</p>
pewmpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151722016-12-26T02:35:44Z
<blockquote><header>johnfound</header>
<p>Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows
.exe files. Simply set TargetOS = "Win32".
</p></blockquote>
<p>Of course! I don't know why I didn't think of it that way. Going to try this.
</p>
<blockquote><header>johnfound</header><p>But notice, that FastCGI protocol allows reusing connections (i.e. threads), this way improving the performance. Of course, this depends on how the web server implements FastCGI protocol. The AsmBB handles this case properly.
</p></blockquote>
<p>Ah, I didn't take into account FastCGI's reuse feature. The new thread is started on the first connection and carried on throughout multiple connections. Got it.</p>
pewmjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151712016-12-25T19:49:46Z
<blockquote><header>pewm</header><p>Yes, I have those environment variables set. That's the same way I attempted to compile FreshIDE. I had wanted to run a compile on Linux for the FreshIDE, but ran into some snags along the way. Will have to get Wine set up before attempting a compile again.
</p></blockquote>
<p>Actually it is not a problem to compile FreshIDE in Linux even without WINE, because FASM for Linux can compile windows .exe files. Simply set TargetOS = "Win32". Actually I didn't tried, but it should work and if not, it is definitely a bug.
</p>
<p>But of course you can run this .exe file only through WINE. I am working on OS independent <a href="https://fresh.flatassembler.net/index.cgi?page=content/articles/5_FreshLibGUI.txt">GUI toolkit</a> in FreshLib, but it is still not powerful enough to provide Fresh IDE functionality.
</p>
<blockquote><header>pewm</header><p>I've found that thread creation is fairly expensive and, if I understand what you're saying then it's likely the thread is created through the kernel for each new request.
</p></blockquote>
<p>Well, you are right, while the threads are faster than CGI processes spawning, they actually slower the program, compared to the fastest possible performance.
</p>
<p>But notice, that FastCGI protocol allows reusing connections (i.e. threads), this way improving the performance. Of course, this depends on how the web server implements FastCGI protocol. The AsmBB handles this case properly.
</p>
<p>Even better performance can be provided by using non blocking sockets, but I don't think it will make the things much better in this very application.</p>
johnfoundpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151702016-12-25T15:03:29Z
<blockquote><header>johnfound</header><p>Yes, thread is the same as in "pthreads". Some come, executed in parallel but in the same memory environment as the main program.
</p></blockquote>
<p>Ok, I was looking around for some good examples of threading with assembly. Best I can tell, the only way is through the kernel with a system call unless we set up our own scheduler of sorts. Good to find an example.
</p>
<p>Have you thought of a thread pool instead of spawning a thread on the fly? I've found that thread creation is fairly expensive and, if I understand what you're saying then it's likely the thread is created through the kernel for each new request. The reason I ask is it's a potential place I could cut my teeth on toying around with the source code. For instance make it where the threading isn't spawned on the fly, rather that the threads are created beforehand based on the CPU cores + hyperthreading and any new job is just a matter of handing that off to the preexisting threads. This might help with extremely high loads. (of course it may be laughable to get 'better' performance out of something that is already operating so damn fast). :)</p>
pewmpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151692016-12-25T14:33:43Z
<blockquote><header>johnfound</header><p>Of course, it is possible. You will need FASM installed. Then the following environment variables set:
</p>
<p><code>lib</code> - The path to the FreshLib directory.
</p>
<p><code>TargetOS</code> - The platform you want to compile for. Currently supported values are "Win32", "Linux" and partially "KolibriOS".
</p>
<p>Then compile the main project file. (i.e. <code>engine.asm</code> for AsmBB) with:
</p>
<pre><code class="">fasm -m 150000 engine.asm
</code></pre></blockquote>
<p>Yes, I have those environment variables set. That's the same way I attempted to compile FreshIDE. I had wanted to run a compile on Linux for the FreshIDE, but ran into some snags along the way. Will have to get Wine set up before attempting a compile again.
</p>
pewmpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151662016-12-25T04:40:31Z
<p>Thanks, That's exactly what I was looking for. I'm going to start to dig in and take a look.
</p>
<p>One question, when you are using the term "thread" are you referring to a thread similar to linux pthreads? Or thread as in another line of execution that still runs serially?
</p>
<p>A note on FreshLib. I read they use stdcall calling convention. Is that the standard used throughout, not cdecl?
</p>
<p>Also the FreshIDE looks awesome, I had already attempted to compile it on Linux, but it appears I needed to set up wine to get that all working. Instead, I've always stuck with Vim and used ctags to generate tags and do the definition lookups, plus using "lv" + "lw" commands in Vim, which gets me pretty far. Any suggestions for someone who is completely indoctrinated into the Vim + ctags + bash workflow?
</p>
<p>Thanks again for replying. That is extremely helpful.</p>
pewmjohnfound on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151652016-12-24T10:58:18Z
<blockquote><header>pewm</header><p>The question I have is where should we focus on learning the AsmBB source? I've downloaded it and begun to comb through it, but a starting point may help me find out more about how you managed to successfully get the whole thing working. I'm more interested in learning assembly, but I've found that jumping headlong into examples does me more good than just writing small bits of code that don't do much.
</p></blockquote>
<p>Well, I can help by shortly describing the structure of the project. But notice that because of using FastCGI protocol, AsmBB is a little bit more complex than usual for beginner assembly programmer.
</p>
<p>The simple CGI web application (for example my <a href="http://chiselapp.com/user/johnfound/repository/MiniMagAsm/index">CMS MiniMagAsm</a>) works by short living CGI processes. They are started by the web server, receive the HTTP request by reading STDIN, write the response (e.g. HTML) to STDOUT and terminate.
</p>
<p>The FastCGI application works a little bit different. At first it works all the time and listens for connections on some socket. The web server connects to this socket and sends the request. Then the FastCGI application sends back the response using the same connection. Serving multiply requests on the same connection and multiply connections are possible. Also, the web server can start several instances of the FastCGI application. It is all about a performance.
</p>
<h1>So, the structure of AsmBB:
</h1>
<p>Everything starts in <code>engine.asm</code>, line:84 - the label <code>start:</code>. Here the engine initializes its environments and opens the SQLite database with the forum data. Then it calls the procedure <code>Listen</code> (line: 126) that is the main loop of the program. It returns only when the engine terminates. After retuning from <code>Listen</code>, the program closes the database, finalizes its works and exits.
</p>
<p>The procedure <code>Listen</code> is defined in the source file <code>fcgi.asm:155</code>. This is the code, that handles <a href="https://en.wikipedia.org/wiki/FastCGI">FastCGI</a> protocol. It simply listens for connections from the web server and on connection creates a thread that to serve the request and continues to listen. The thread is started in the procedure <code>procServeRequest</code> in <code>fcgi.asm:347</code>.
</p>
<p>This thread, receives the request information on the socket and collects it for future processing.
</p>
<p>Once the whole information is collected - i.e. the HTTP reques, the POST data (if any) and the environment parameters, the procedure <code>ServeOneRequest</code> is called. It is actually the "business logic" code that makes the application to act as a web forum.
</p>
<p><code>ServeOneRequest</code> returns the response (e.g. HTML code, images, etc.) that is returned to the web server, according to FastCGI protocol. After completing the request, the thread terminates or stays waiting for another request. It depends on how the web server can multiplex the requests on the FastCGI connection.
</p>
<p><code>ServeOneRequest</code> is located in the file <code>commands.asm:49</code>. It analyzes the request URL and the request type (GET, POST) and decides how to serve it. For example, it can return some file directly, or read the information from the database, or store some information in the database.
</p>
<p>The URL analyze is important and located on <code>commands.asm:332</code> - it dispatches the control depending on the URL elements. The addresses of the different procedures are loaded to <code>ECX</code> and later called (label <code>.exec_command</code>, line: 524 or <code>.exec_command2:</code>, line: 541).
</p>
<p>Later you can browse these procedures. They are located in different files, serving different aspects of forum engine. For example, the procedure <code>ListThreads</code> (threadlist.asm) creates the list of threads on the front page of the forum.
</p>
<p>The procedure <code>ShowThread</code> (showthread.asm) displays one thread. And so on.
</p>
<p>Notice, that AsmBB widely uses the library FreshLib. You can read more in <a href="https://fresh.flatassembler.net/index.cgi?page=content/articles/2_FreshLibDoc.txt">FreshLib reference</a> and <a href="https://FreshLib%20user%20guide/">FreshLib user guide</a>, but unfortunately the documentation is far from perfect.
</p>
<p>In order to better browse the big code, scattered across multiply files, I would suggest using <a href="https://fresh.flatassembler.net/">Fresh IDE</a> code browsing features. Read more in the following article: <a href="https://fresh.flatassembler.net/index.cgi?page=content/articles/1_tips.txt#H6.">Tips and tricks</a>. Especially useful if the feature "Goto definition" (Ctrl+D) that will jump you at the line where some label is defined. The cross reference is also useful (Ctrl+R).
</p>
<p><a href="https://FreshLib%20user%20guide/">FreshLib user guide</a>
</p>
johnfoundpewm on Learning the AsmBB sourcetag:board.asm32.info,2018-03-06:Post151642016-12-24T04:03:53Z
<p>What a fascinating project. The fact that the whole thing is written in assembly has stuck with me after the first time I saw it. I've kept coming back to this as an example of something that breaks your whole paradigm and smacks you over the head to show that some things you know may not be correct.
</p>
<p>So we can program websites in assembly language. Who would have thought?
</p>
<p>The question I have is where should we focus on learning the AsmBB source? I've downloaded it and begun to comb through it, but a starting point may help me find out more about how you managed to successfully get the whole thing working. I'm more interested in learning assembly, but I've found that jumping headlong into examples does me more good than just writing small bits of code that don't do much.</p>
pewm