thesis/content/a_tour_of_tfw.tex

\chapter{A Tour of TFW}

The purpose of this chapter is to further detail the built-in components
provided by the framework.
As previously mentioned, these components are implemented as event handlers 
running in the \texttt{solvable} Docker container and frontend
components written in Angular.
For instance the built-in code editor requires a frontend component and an event
handler to function properly, while the frontend component responsible for
drawing out and managing other components implement no
event handler.

In the Tutorial Framework most of the built-ins define APIs, which are TFW messages
that can be used to interact with them.
For example, to inject a command into the terminal one would use a message like this:
\begin{lstlisting}[captionpos=b,caption={An API Message Capable of Writing in the Terminal}]
{ 
    "key": "shell",
    "data":
    {
        "command": "write",
        "value": "echo 'Hello TFW World!'\n"
    }
}
\end{lstlisting}
Notice the \texttt{\textbackslash{}n} at the end of the command.
By including a newline character, we are also capable of executing commands in the
user's terminal.
Were this newline omitted, the command would only be written to the terminal
(but not automatically executed) for users to inspect and potentially execute themselves.

Some components emit or broadcast messages on specific events, for instance the
\texttt{FSMManagingEventHandler} broadcasts the following message on state transitions:
\begin{lstlisting}[captionpos=b,caption={An FSM Update message}]
{
    "key": "fsm_update",
    "data" :
    {
        "current_state": ...string...,
        "valid_transitions": ...[array of {"trigger": ...string...} objects]...,
        "in_accepted_state": ...boolean...,
        "last_event": ...
                      object like
                      {
                          "from_state": ...string...,
                          "to_state": ...string...,
                          "trigger": ...string...,
                          "timestamp": ...unix timestamp...
                      }
                      ...
    }
}
\end{lstlisting}
As you can see this message contains loads of useful information regarding what is
exactly happening in the tutorial at a given point and can be used by client code
to make informed decisions based on this knowledge.

It is not the purpose of this text to provide a complete API documentation, so in the
following I am only going to explain possibilities provided by given components rather
than showcasing actual, real-life API messages.

\section{Messages Component}

The framework must allow content creators to communicate their \emph{message} to the user.
In other words some way must be provided to ``talk'' to users.
This is the responsibility of the \emph{messages} frontend component, which
provides a chatbox-like element on the web application the framework can send
messages to.
The simplest form of communication it accomodates it the insertion of text
into the chatbox through API messages.
Every message has an optional \emph{originator}, which serves signal to the user
on the purpose of the message.
These messages are also timestamped.

\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot Typing in the Messages Component}    

A particularly interesting feature of the messages component is that TFW client code
can queue a bunch of messages for the component to send one by one, separated by
appropriate pauses in time so that the user is capable of conveniently reading through all
off them.
Similarly to a real chat application, some
``jumping dots'' indicate if the bot is still ``typing''.
The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
set by developers according to their specific use cases.
This creates an experience similar to chatting with someone in real time, as the time
it takes for each message to be displayed is depending on the lenght of the previous message.
This illusion is made possible through appropriate \texttt{setTimeout()} calls in
TypeScript and some elementary math to calculate the proper delays in milliseconds based on
message lengths:

\[ charactersPerMinute = wordsPerMinute * 5 \]
\[ charactersPerSeconds = charactersPerMinute / 60 \]
\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
\[ timeoutMilliseconds = timeoutSeconds * 1000 \]

\section{IDE Component}\label{idecomponent}

This is the code editor integrated into the frontend of the framework.
It allows users to select, display and edit files.
Developers can configure which directory on the file system of the \texttt{solvable}
container should the editor list files from.
The editor features the ``Deploy'' button referred to earlier in this paper, which is
capable of restarting processes that might execute a file visible in the editor.

To implement this IDE%
\footnote{Integrated development environment}
I have integrated the open source Monaco editor developed by Microsoft into the
Angular web application TFW uses as a frontend, and added functionality to it
that allows the editor to integrate with the framework.
This involves commnication with an event handler dedicated to this feature,
which is capable of reading and writing files to disk, while sending and receiving
editor content from the frontend component.
The interaction of this event handler and the Monaco editor provides a seamless
editing expirience, featuring autosave at configurable intervals, code completion,
automatic code coloring for several programming languages and more.

Perhaps the most ``magical'' feature of this editor is that if any process
in the Docker container writes a file that is being displayed in the editor,
the contents of that file are automatially refreshed without any user
interaction whatsoever.
Besides that, if a file is created in the directory the editor is configured
to display, that file is automatially displayed on a new tab in the IDE.

This allows for really interesting demo opportunities.
Lets say I create a file using the terminal on the frontend by executing the
command \texttt{touch file.txt}. A new tab on the editor automatically
appears. If I select it I can confirm that I have successfully created an
empty file.
After this let's run a \texttt{while} cycle in the command line which
peroadically appends some text to \texttt{file.txt}:
\begin{lstlisting}[captionpos=b,caption={Bash While Cycle Writing to a File Periodically},
                   language=bash]
while true
do
    echo 'Hey there!' >> file.txt
    sleep 1
done
\end{lstlisting}
The results speak for themselves:
\pic{figures/ide_demo.png}{The Editor Demo Involving Automatic File Refreshing}
As you can see, the file contents are automatially updated as the bash script appends
to the file.
This feature is implemented by using the inotify API%
\footnote{\href{http://man7.org/linux/man-pages/man7/inotify.7.html}
{http://man7.org/linux/man-pages/man7/inotify.7.html}}
provided by the Linux kernel to monitor file system events involving the directory listed by
the editor. The event handler of the editor hooks callbacks to said events which notify the
Tutorial Framework to reload the list of files in the directory and the contents of
the selected files.
The code making this feature possible is reused several times in the framework
for interesting purposes such as monitoring the logs of processes.

The editor also allows content creators to completely control it using API messages.
This involves selecting, reading and writing files as well as changing the
selected directory.
These features allow content creators to ``guide'' a user through code bases
for example, where in each step of a tutorial a file is opened and explained
through messages sent to the chatbox of the messages component.

\section{Terminal Component}

This is a full-fledged xterm terminal emulator running right in the user's browser.
I have partially derived this from my early work I have briefly mentioned
in~\ref{intro:emergence}, but it required lot's of new functionality to work the way
it does today.
The difference this time is that I had to integrate it into an Angular component.
I have used the awesome xterm.js%
\footnote{\href{https://xtermjs.org}{https://xtermjs.org}}
terminal emulator to do so.

This component has a tiny server process which is managed by a TFW event handler.
This small server is responsible for spawning bash sessions and
unix pseudoterminals (or \texttt{pty}s) in the \texttt{solvable} Docker
container.
It is also responsible for connecting the master end of the \texttt{pty} to the
emulator running in your browser and the slave end to the bash session it has
spawned.
This way users are able to work in the shell displayed on the frontend just like
they would on their home machines, which allows for great tutorials
explaining topics that involve the usage of a shell.
Note that this allows coverion an extremely wide variety of topics using TFW: from compiling
shared libraries for development, using cryptographic FUSE filesystems
for enhanced privacy%
\footnote{\href{https://github.com/rfjakob/gocryptfs}{https://github.com/rfjakob/gocryptfs}},
to automating cloud infrastructure using Ansible%
\footnote{\href{https://www.ansible.com}{https://www.ansible.com}}.

This component exposes several functions through TFW message APIs, such as injecting commands
to the terminal, reading command history and registering callbacks that are invoked when
certain command are executed by the user.

\pic{figures/terminal.png}{The Frontend Terminal of TFW Running top}

\section{Console Component}

This component is a simple textbox that can be used to display anything to the user,
from the results of unit test to the output of processes.
The console has no event handler: it is a purely frontend component which exposes a simple
API through TFW messages to write and read it's contents.

It works great when combined with the process management capabilities of the framework:
if configured to do so it can display the output of processes like webservers in real time.
When using this next to the frontend editor of the framework, it allows for a development
experience similar to working in an IDE on your laptop.

\pic{figures/console_and_editor.png}{The Console Displaying Live Process Logs Next to the TFW Code Editor}

\section{Process Management}

The framework includes an event handler capable of managing processes running inside
the \texttt{solvable} Docker container.
It's capabilities include starting, stopping and restarting processes.
It is also capable of emitting the standard out or standard error logs of processes
(by broadcasting TFW messages).
This component can be iteracted with using TFW API messages.

The Tutorial Framework uses supervisor%
\footnote{\href{http://supervisord.org}{http://supervisord.org}}
to run multiple processes inside a Docker container
(whereas usually Docker containers run a single process only).
This is going to be explained further in a later chapter.
All this is possible through using the xmlrpc%
\footnote{\href{https://docs.python.org/3/library/xmlrpc.html}{https://docs.python.org/3/library/xmlrpc.html}}
API exposed by supervisor, which allows the framework to iteract with processes it controls.

It is also possible to find out what files does a process write logs to.
Combining this with the inotify capabilities of TFW explained
briefly in~\ref{idecomponent}, it becomes possible to implement live log monitoring
in the framework.
The features involving the use of inotify were among the most difficult ones implement,
since the sheer number of impossible to debug issues that such
a complex system could come with.

I'll briefly explain such a bug, which I've found to be immersely exciting.
During the initial development of this feature all processes inside the
\texttt{solvable} Docker container were writing their logs to files
in the FHS%
\footnote{The File Hierarchy Standard is a standard maintained by the Linux foundation,
defining a common directory structure for compliant systems. See
\href{https://wiki.linuxfoundation.org/lsb/fhs}{https://wiki.linuxfoundation.org/lsb/fhs}}
location \texttt{/tmp/}.
All logs coming from the container itself were also logged to this location.
This had caused an infinite recursion: when a process would write to \texttt{/tmp/}
inotify would invoke a process that would also log to that location causing the kernel to
emit more inotify events, which in turn would cause more and more new proesses to spawn
and write to \texttt{/tmp/}, causing the whole procedure to repeat again and again.
This continued until my machine would start to run out of memory and stat swapping
pages to disk%
\footnote{When a modern operating system runs out of physical RAM, it is going to swap
virtual memory pages to disk so it can continue to operate --- slowly}
like crazy, causing the whole system to spiral downwards
in a spectacular fashion until the whole thing managed to crash.
It was an event of such chaotic beauty, that I often fondly recall it to this day.
Of course it would take me several hours to identify the exact causes behind this
fascinating phenomenon, but those were \emph{very} fun hours.