thesis/content/a_tour_of_tfw.tex

\chapter{A Tour of TFW}\label{atouroftfw}

The bulk of the functionality provided by the Tutorial Framework
is implemented in pre-made components that are built upon the architecture
described in Chapter~\ref{architecture}.
To create tutorials, developers generally rely on and use TFW provided components to
implement most of the common functionality and then build more exercise-specific logic
based on the TFW architecture themselves.
A \emph{very important} point to keep in mind is that most of this
exercise-specific logic will be implemented in \textbf{FSM callbacks} and
custom \textbf{event handlers}.
The whole framework is built in a way to faciliate this process and developers
who understand this mindset are almost always guaranteed to create great
content using TFW.

The purpose of this chapter is to further detail the built-in components
provided by the framework.
As previously mentioned, these components are implemented as event handlers 
running in the \code{solvable} Docker container and frontend
components written in Angular.
For instance the built-in code editor requires a frontend component and an event
handler to function properly, while the frontend component responsible for
drawing out and managing other components implements no
event handler, so it only exists on the frontend.

In the Tutorial Framework most of the built-ins define APIs, which are TFW messages
that can be used to interact with them.
For example, to inject a command into the terminal one would use a message like this:
\begin{lstlisting}[captionpos=b,caption={An API Message Capable of Writing in the Terminal}]
{ 
    "key": "shell",
    "data":
    {
        "command": "write",
        "value": "echo 'Hello TFW World!'\n"
    }
}
\end{lstlisting}
Notice the \code{\n} at the end of the command.
By including a newline character, we are also capable of executing commands in the
user's terminal.
Were this newline omitted, the command would only be written to the terminal
(but not automatically executed) for users to inspect and potentially execute themselves.

Some components emit or broadcast messages on specific events, for instance the
\code{FSMManagingEventHandler} broadcasts the following message on state transitions:
\begin{lstlisting}[captionpos=b,caption={An FSM Update message}]
{
    "key": "fsm_update",
    "data" :
    {
        "current_state": ...string...,
        "valid_transitions": ...[array of {"trigger": ...string...} objects]...,
        "in_accepted_state": ...boolean...,
        "last_event": ...
                      object like
                      {
                          "from_state": ...string...,
                          "to_state": ...string...,
                          "trigger": ...string...,
                          "timestamp": ...unix timestamp...
                      }
                      ...
    }
}
\end{lstlisting}
As you can see this message contains loads of useful information regarding what is
exactly happening in the tutorial at a given point and can be used by client code
to make informed decisions based on this knowledge.

It is not the purpose of this text to provide a complete API documentation, so in the
following I am only going to explain possibilities provided by given components rather
than showcasing actual, real-life API messages.

\section{Messages Component}

The framework must allow content creators to communicate their \emph{message} to the user.
In other words some way must be provided to ``talk'' to users.
This is the responsibility of the \emph{messages} frontend component, which
provides a chatbox-like element on the web application the framework can send
messages to.
The simplest form of communication it accomodates it the insertion of text
into the chatbox through API messages.
Every message has an optional \emph{originator}, which serves signal to the user
on the purpose of the message.
These messages are also timestamped.

\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot Typing in the Messages Component}    

A particularly interesting feature of the messages component is that TFW client code
can queue a bunch of messages for the component to send one by one, separated by
appropriate pauses in time so that the user is capable of conveniently reading through all
off them.
Similarly to a real chat application, some
``jumping dots'' indicate if the bot is still ``typing''.
The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
set by developers according to their specific use cases.
This creates an experience similar to chatting with someone in real time, as the time
it takes for each message to be displayed is depending on the lenght of the previous message.
This illusion is made possible through appropriate \code{setTimeout()} calls in
TypeScript and some elementary math to calculate the proper delays in milliseconds based on
message lengths:

\[ charactersPerMinute = wordsPerMinute * 5 \]
\[ charactersPerSeconds = charactersPerMinute / 60 \]
\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
\[ timeoutMilliseconds = timeoutSeconds * 1000 \]

\section{IDE Component}\label{idecomponent}

This is the code editor integrated into the frontend of the framework.
It allows users to select, display and edit files.
Developers can configure which directory on the file system of the \code{solvable}
container should the editor list files from.
The editor features the ``Deploy'' button referred to earlier in this paper, which is
capable of restarting processes that might execute a file visible in the editor.

To implement this IDE%
\footnote{Integrated development environment}
I have integrated the open source Monaco editor developed by Microsoft into the
Angular web application TFW uses as a frontend, and added functionality to it
that allows the editor to integrate with the framework.
This involves commnication with an event handler dedicated to this feature,
which is capable of reading and writing files to disk, while sending and receiving
editor content from the frontend component.
The interaction of this event handler and the Monaco editor provides a seamless
editing expirience, featuring autosave at configurable intervals, code completion,
automatic code coloring for several programming languages and more.

Perhaps the most ``magical'' feature of this editor is that if any process
in the Docker container writes a file that is being displayed in the editor,
the contents of that file are automatially refreshed without any user
interaction whatsoever.
Besides that, if a file is created in the directory the editor is configured
to display, that file is automatially displayed on a new tab in the IDE.

This allows for really interesting demo opportunities.
Lets say I create a file using the terminal on the frontend by executing the
command \code{touch file.txt}. A new tab on the editor automatically
appears. If I select it I can confirm that I have successfully created an
empty file.
After this let's run a \code{while} cycle in the command line which
peroadically appends some text to \code{file.txt}:
\begin{lstlisting}[captionpos=b,caption={Bash While Cycle Writing to a File Periodically},
                   language=bash]
while true
do
    echo 'Hey there!' >> file.txt
    sleep 1
done
\end{lstlisting}
The results speak for themselves:
\pic{figures/ide_demo.png}{The Editor Demo Involving Automatic File Refreshing}
As you can see, the file contents are automatially updated as the bash script appends
to the file.
This feature is implemented by using the inotify API%
\footnote{\href{http://man7.org/linux/man-pages/man7/inotify.7.html}
{http://man7.org/linux/man-pages/man7/inotify.7.html}}
provided by the Linux kernel to monitor file system events involving the directory listed by
the editor. The event handler of the editor hooks callbacks to said events which notify the
Tutorial Framework to reload the list of files in the directory and the contents of
the selected files.
The code making this feature possible is reused several times in the framework
for interesting purposes such as monitoring the logs of processes.

The editor also allows content creators to completely control it using API messages.
This involves selecting, reading and writing files as well as changing the
selected directory.
These features allow content creators to ``guide'' a user through code bases
for example, where in each step of a tutorial a file is opened and explained
through messages sent to the chatbox of the messages component.

\section{Terminal Component}

This is a full-fledged xterm terminal emulator running right in the user's browser.
I have partially derived this from my early work I have briefly mentioned
in~\ref{intro:emergence}, but it required lot's of new functionality to work the way
it does today.
The difference this time is that I had to integrate it into an Angular component.
I have used the awesome xterm.js%
\footnote{\href{https://xtermjs.org}{https://xtermjs.org}}
terminal emulator to do so.

This component has a tiny server process which is managed by a TFW event handler.
This small server is responsible for spawning bash sessions and
unix pseudoterminals (or \code{pty}s) in the \code{solvable} Docker
container.
It is also responsible for connecting the master end of the \code{pty} to the
emulator running in your browser and the slave end to the bash session it has
spawned.
This way users are able to work in the shell displayed on the frontend just like
they would on their home machines, which allows for great tutorials
explaining topics that involve the usage of a shell.
Note that this allows coverion an extremely wide variety of topics using TFW: from compiling
shared libraries for development, using cryptographic FUSE filesystems
for enhanced privacy%
\footnote{\href{https://github.com/rfjakob/gocryptfs}{https://github.com/rfjakob/gocryptfs}},
to automating cloud infrastructure using Ansible%
\footnote{\href{https://www.ansible.com}{https://www.ansible.com}}.

This component exposes several functions through TFW message APIs, such as injecting commands
to the terminal, reading command history and registering callbacks that are invoked when
certain command are executed by the user.

\pic{figures/terminal.png}{The Frontend Terminal of TFW Running top}

The implementation of reading command history is quite an exotic one.
The framework needs to be able to detect if the user has executed any command in the
container using an interactive bash session.
This is not an easy thing to accomplish without relying on some sort of heavyweight
monitoring solution such as Sysdig%
\footnote{\href{https://sysdig.comq}{https://sysdig.com}}.
I deemed most simiar systems a huge overkill to implement this functionality, and their
memory footprints are not something we could afford here.
Another way would be to use \code{pam_tty_audit.so} in the PAM%
\footnote{Linux Pluggable Authentication Modules:
\href{http://man7.org/linux/man-pages/man3/pam.3.html}
{http://man7.org/linux/man-pages/man3/pam.3.html}}
configurations responsible for logins, as this allows for various TTY auditing functions,
but I have found an ever simpler approach to the problem in the end.
By using the inotify system built into TFW, I can set up the user's environment in
such a way, that I can enforce and determine the location of the bash \code{HISTFILE}%
\footnote{This environment variable contains the path to the file bash writes command
history to}
of the user.
This way I can monitor changes made to this file and read the commands executed
by the user from it.
It is important to keep in mind that the user is able to ``sabotage'' this method%
\footnote{By unsetting the \code{HISTFILE} envvar for example},
but that should not be an issue as this is not a feature that is intended to be
used in competitive environments (and if the users of a tutorial intentionally
break the system under themselves, well, good for them).

\section{Console Component}

This component is a simple textbox that can be used to display anything to the user,
from the results of unit test to the output of processes.
The console has no event handler: it is a purely frontend component which exposes a simple
API through TFW messages to write and read it's contents.

It works great when combined with the process management capabilities of the framework:
if configured to do so it can display the output of processes like webservers in real time.
When using this next to the frontend editor of the framework, it allows for a development
experience similar to working in an IDE on your laptop.

\pic{figures/console_and_editor.png}{The Console Displaying Live Process Logs Next to the TFW Code Editor}

\section{Process Management}\label{processmanagement}

The framework includes an event handler capable of managing processes running inside
the \code{solvable} Docker container.
It's capabilities include starting, stopping and restarting processes.
It is also capable of emitting the standard out or standard error logs of processes
(by broadcasting TFW messages).
This component can be iteracted with using TFW API messages.

The Tutorial Framework uses supervisor%
\footnote{\href{http://supervisord.org}{http://supervisord.org}}
to run multiple processes inside a Docker container
(whereas usually Docker containers run a single process only).
This is going to be explained further in a later chapter.
All this is possible through using the xmlrpc%
\footnote{\href{https://docs.python.org/3/library/xmlrpc.html}{https://docs.python.org/3/library/xmlrpc.html}}
API exposed by supervisor, which allows the framework to iteract with processes it controls.

It is also possible to find out what files does a process write logs to.
Combining this with the inotify capabilities of TFW explained
briefly in~\ref{idecomponent}, it becomes possible to implement live log monitoring
in the framework.
The features involving the use of inotify were among the most difficult ones implement,
since the sheer number of impossible to debug issues that such
a complex system could come with.

I'll briefly explain such a bug, which I've found to be immersely exciting.
During the initial development of this feature all processes inside the
\code{solvable} Docker container were writing their logs to files
in the FHS%
\footnote{The File Hierarchy Standard is a standard maintained by the Linux foundation,
defining a common directory structure for compliant systems. See
\href{https://wiki.linuxfoundation.org/lsb/fhs}{https://wiki.linuxfoundation.org/lsb/fhs}}
location \code{/tmp/}.
All logs coming from the container itself were also logged to this location.
This had caused an infinite recursion: when a process would write to \code{/tmp/}
inotify would invoke a process that would also log to that location causing the kernel to
emit more inotify events, which in turn would cause more and more new proesses to spawn
and write to \code{/tmp/}, causing the whole procedure to repeat again and again.
This continued until my machine would start to run out of memory and stat swapping
pages to disk%
\footnote{When a modern operating system runs out of physical RAM, it is going to swap
virtual memory pages to disk so it can continue to operate --- slowly}
like crazy, causing the whole system to spiral downwards
in a spectacular fashion until the whole thing managed to crash.
It was an event of such chaotic beauty, that I often fondly recall it to this day.
Of course it would take me several hours to identify the exact causes behind this
fascinating phenomenon, but those were \emph{very} fun hours.

\section{FSM Management}

I have already mentioned the event handler called \code{FSMManagingEventHandler},
which is responsible for managing the framework FSM.
For completeness I chose to include it on this chapter as well.
The API it exposes through TFW messages allows client code to attempt stepping the
state machine.
As previously explained this is something that is considered to be a \emph{privileged}
operation in case authentication is enabled, causing the event handler to accept
digitally signed messages only.
In the event of a successful FSM step, this component is going to broadcast
a message showcased as an example in the beginning of Chapter~\ref{atouroftfw}.

\section{Web Component}

The web component allows developers to configure what web application should
be displayed on the frontend.
There are two options for doing this:
\begin{enumerate}
    \item Supply the URL of a web application served from the
          Docker container to be displayed
    \item Implement an Angular component on the frontend and display it
\end{enumerate} 

In most cases, when developers make tutorials on Python and Java topics for
example, they will implement a web application in the respective programming
language the exercise is about and most developers have no experience in
Angular development.
This is why more often then not the fist option is chosen.
This feature is implemented by embedding a standard HTML iframe%
\footnote{\href{https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe}
{https://developer.mozilla.org/en-US/docs/Web/HTML/Element/iframe}}
inside the frontend, which is controllable by TFW
(as an example it can be reloaded or navigated from the framework).

Watchful readers might now be thinking about how does ``controlled by the framework''
thing work with the Same Origin Policy%
\footnote{\href{https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin_policy}
{https://developer.mozilla.org/en-US/docs/Web/Security/Same-origin\_policy}}
being in effect?
The answer is that developers must use a \emph{relative url}, that is an URL relative
to the entry pont of the TFW frontend itself.
To allow serving several web applications from a single port the framework
supports optional reverse-proxy configurations through the nginx%
\footnote{\href{http://nginx.org}{http://nginx.org}} web server ran by the framework.
More on this in a later chapter.

\section{Various Frontend Features}

The angular frontend of features several different layouts.
These layouts are useful to accomodate different workflows for users,
such as the previous exampe of editig code and being able to view the
result of said code in real time next to the editor.
Another example would be editing Ansible playbooks in the file editor,
and then trying to run them in the terminal.
There are also almost full screen views for each component that makes sense
to be used that way.

The frontend was designed in a way to be fully responsive in windows sizes
that still keep the whole thing usable (i.e.\ it would not be practial to start
solving TFW tutorials on a smart phone, simply because of size limits, so they are
not supported, but the frontend still behaves as expected on small laptops or bigger tablets).
This is not an easy thing to impelent and maintain due to the lots of small
incompatibilites between browsers given the complexity of the frontend.

Just remember that a few years ago the clearfix%
\footnote{\href{https://stackoverflow.com/questions/8554043/what-is-a-clearfix}
{https://stackoverflow.com/questions/8554043/what-is-a-clearfix}}
hack was the industry standard in creating CSS layouts.
The situation has improved \emph{a lot} since then with flexboxes
and grid layouts despite the sheer chaos that is generally involved in web
standardization efforts, but CSS espacially%
\footnote{\href{https://developer.mozilla.org/en-US/docs/Web/CSS/CSS3}
{https://developer.mozilla.org/en-US/docs/Web/CSS/CSS3}}.

The framework frontend is built on grid layout and flexboxes%
\footnote{\href{https://developer.mozilla.org/en-US/docs/Web/CSS/CSS\_Grid\_Layout}
{https://developer.mozilla.org/en-US/docs/Web/CSS/CSS\_Grid\_Layout}},
which gives us the best hopes of being able to maintain it down the line.
It would involve unimaginable horrors to support this multi-layout
frontend on older browsers, so browsers without flex and grid
support are not supported by TFW.
Arguably this is a good thing, as people should keep their browsers up to date to
follow frequent security patches anyway, so let this serve as a reminder to
developers looking to get into IT security that the first step is to
keep your software up to date.

The frontend of the framework exposes some additional APIs.
These include the changing of layouts, selecting the terminal or console
component to be displayed, the possibility of dynamically modifying
frontend configuration values (such as the frequency of autosaving the files in the editor)
and more.