Start reading through text to fix errors
This commit is contained in:
		@@ -10,23 +10,26 @@ A \emph{very important} point to keep in mind is that most of this
 | 
				
			|||||||
exercise-specific logic will be implemented in \textbf{FSM callbacks} and
 | 
					exercise-specific logic will be implemented in \textbf{FSM callbacks} and
 | 
				
			||||||
custom \textbf{event handlers}.
 | 
					custom \textbf{event handlers}.
 | 
				
			||||||
The whole framework is built in a way to faciliate this process and developers
 | 
					The whole framework is built in a way to faciliate this process and developers
 | 
				
			||||||
who understand this mindset are almost always guaranteed to create great
 | 
					who understand this mindset almost always find it a breeze to create great
 | 
				
			||||||
content using TFW.
 | 
					content using TFW\@.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The purpose of this chapter is to further detail the built-in components
 | 
					The purpose of this chapter is to further detail the built-in components
 | 
				
			||||||
provided by the framework.
 | 
					provided by the framework.
 | 
				
			||||||
As previously mentioned, these components are implemented as event handlers 
 | 
					As previously mentioned before, these components are implemented as event handlers 
 | 
				
			||||||
running in the \code{solvable} Docker container and frontend
 | 
					running in the \code{solvable} Docker container which communicate with frontend
 | 
				
			||||||
components written in Angular.
 | 
					components written in Angular.
 | 
				
			||||||
For instance the built-in code editor requires a frontend component and an event
 | 
					Some components might only feature one of these however,
 | 
				
			||||||
handler to function properly, while the frontend component responsible for
 | 
					for instance the built-in code editor requires a frontend component and an event
 | 
				
			||||||
drawing out and managing other components implements no
 | 
					handler to function properly, while the component responsible for
 | 
				
			||||||
event handler, so it only exists on the frontend.
 | 
					drawing out and managing frontend components implements no
 | 
				
			||||||
 | 
					event handler, so it purely exists in Angular.
 | 
				
			||||||
 | 
					An other example of a purely frontend component would be the messages component,
 | 
				
			||||||
 | 
					which is used to display messages to the user.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
In the Tutorial Framework most of the built-ins define APIs, which are TFW messages
 | 
					In the Tutorial Framework most of the built-ins define APIs, which are TFW messages
 | 
				
			||||||
that can be used to interact with them.
 | 
					that can be used to interact with them.
 | 
				
			||||||
For example, to inject a command into the terminal one would use a message like this:
 | 
					For example, to inject a command into the terminal one would use a message like this:
 | 
				
			||||||
\begin{lstlisting}[captionpos=b,caption={An API Message Capable of Writing in the Terminal}]
 | 
					\begin{lstlisting}[captionpos=b,caption={An API message capable of writing to the terminal}]
 | 
				
			||||||
{ 
 | 
					{ 
 | 
				
			||||||
    "key": "shell",
 | 
					    "key": "shell",
 | 
				
			||||||
    "data":
 | 
					    "data":
 | 
				
			||||||
@@ -36,15 +39,15 @@ For example, to inject a command into the terminal one would use a message like
 | 
				
			|||||||
    }
 | 
					    }
 | 
				
			||||||
}
 | 
					}
 | 
				
			||||||
\end{lstlisting}
 | 
					\end{lstlisting}
 | 
				
			||||||
Notice the \code{\n} at the end of the command.
 | 
					Notice the ``\code{\n}'' at the end of the command.
 | 
				
			||||||
By including a newline character, we are also capable of executing commands in the
 | 
					By including a newline character, we are also capable of executing commands directly
 | 
				
			||||||
user's terminal.
 | 
					in the user's terminal (and the user can see this).
 | 
				
			||||||
Were this newline omitted, the command would only be written to the terminal
 | 
					Were this newline omitted, the command would only be written to the terminal
 | 
				
			||||||
(but not automatically executed) for users to inspect and potentially execute themselves.
 | 
					--- but not automatically executed --- for users to inspect and potentially execute themselves.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Some components emit or broadcast messages on specific events, for instance the
 | 
					Some components emit or broadcast messages on specific events, for instance the
 | 
				
			||||||
\code{FSMManagingEventHandler} broadcasts the following message on state transitions:
 | 
					\code{FSMManagingEventHandler} broadcasts the following message on state transitions:
 | 
				
			||||||
\begin{lstlisting}[captionpos=b,caption={An FSM Update message}]
 | 
					\begin{lstlisting}[captionpos=b,caption={An FSM update message}]
 | 
				
			||||||
{
 | 
					{
 | 
				
			||||||
    "key": "fsm_update",
 | 
					    "key": "fsm_update",
 | 
				
			||||||
    "data" :
 | 
					    "data" :
 | 
				
			||||||
@@ -65,38 +68,38 @@ Some components emit or broadcast messages on specific events, for instance the
 | 
				
			|||||||
}
 | 
					}
 | 
				
			||||||
\end{lstlisting}
 | 
					\end{lstlisting}
 | 
				
			||||||
As you can see this message contains loads of useful information regarding what is
 | 
					As you can see this message contains loads of useful information regarding what is
 | 
				
			||||||
exactly happening in the tutorial at a given point and can be used by client code
 | 
					exactly happening in the tutorial at a given point in time and can be used by client code
 | 
				
			||||||
to make informed decisions based on this knowledge.
 | 
					to make informed decisions based on this knowledge.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
It is not the purpose of this text to provide a complete API documentation, so in the
 | 
					It is not the purpose of this text to provide a complete API documentation however, so in the
 | 
				
			||||||
following I am only going to explain possibilities provided by given components rather
 | 
					following I am only going to explain possibilities provided by given components rather
 | 
				
			||||||
than showcasing actual, real-life API messages.
 | 
					than showcasing actual, real-life API messages.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\section{Messages Component}
 | 
					\section{Messages Component}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The framework must allow content creators to communicate their \emph{message} to the user.
 | 
					The framework must allow content creators to communicate their \emph{message} to the user.
 | 
				
			||||||
In other words some way must be provided to ``talk'' to users.
 | 
					In other words, some way must be provided to ``talk'' to users.
 | 
				
			||||||
This is the responsibility of the \emph{messages} frontend component, which
 | 
					This is the responsibility of the \emph{messages} component, which
 | 
				
			||||||
provides a chatbox-like element on the web application the framework can send
 | 
					provides a chatbox-like element on the frontend.
 | 
				
			||||||
messages to.
 | 
					 | 
				
			||||||
The simplest form of communication it accomodates it the insertion of text
 | 
					The simplest form of communication it accomodates it the insertion of text
 | 
				
			||||||
into the chatbox through API messages.
 | 
					into the chatbox through API messages.
 | 
				
			||||||
Every message has an optional \emph{originator}, which serves signal to the user
 | 
					Every message has an optional \emph{originator}, which serves to remind the user
 | 
				
			||||||
on the purpose of the message.
 | 
					of the purpose of the given message.
 | 
				
			||||||
These messages are also timestamped.
 | 
					These messages are also timestamped so that it is easier to navigate through them
 | 
				
			||||||
 | 
					and look back older messages from the user.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot Typing in the Messages Component}    
 | 
					\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot typing in the messages component}    
 | 
				
			||||||
 | 
					
 | 
				
			||||||
A particularly interesting feature of the messages component is that TFW client code
 | 
					A particularly interesting feature of the messages component is that TFW client code
 | 
				
			||||||
can queue a bunch of messages for the component to send one by one, separated by
 | 
					can queue a bunch of messages for the component to display one by one, separated by
 | 
				
			||||||
appropriate pauses in time so that the user is capable of conveniently reading through all
 | 
					appropriate pauses in time so that the user is capable of conveniently reading through all
 | 
				
			||||||
off them.
 | 
					off them.
 | 
				
			||||||
Similarly to a real chat application, some
 | 
					Similarly to a real chat application, some
 | 
				
			||||||
``jumping dots'' indicate if the bot is still ``typing''.
 | 
					``jumping dots'' indicate if the bot is still ``typing'' something.
 | 
				
			||||||
The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
 | 
					The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
 | 
				
			||||||
set by developers according to their specific use cases.
 | 
					set by developers according to their specific requirements.
 | 
				
			||||||
This creates an experience similar to chatting with someone in real time, as the time
 | 
					This creates an experience similar to chatting with someone in real time, as the time
 | 
				
			||||||
it takes for each message to be displayed is depending on the lenght of the previous message.
 | 
					it takes for each message to be displayed depends on the lenght of the previous message.
 | 
				
			||||||
This illusion is made possible through appropriate \code{setTimeout()} calls in
 | 
					This illusion is made possible through appropriate \code{setTimeout()} calls in
 | 
				
			||||||
TypeScript and some elementary math to calculate the proper delays in milliseconds based on
 | 
					TypeScript and some elementary math to calculate the proper delays in milliseconds based on
 | 
				
			||||||
message lengths:
 | 
					message lengths:
 | 
				
			||||||
@@ -106,14 +109,16 @@ message lengths:
 | 
				
			|||||||
\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
 | 
					\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
 | 
				
			||||||
\[ timeoutMilliseconds = timeoutSeconds * 1000 \]
 | 
					\[ timeoutMilliseconds = timeoutSeconds * 1000 \]
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					The value 5 comes from the fact that on average english words are 5
 | 
				
			||||||
 | 
					characters long according to some studies.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\section{IDE Component}\label{idecomponent}
 | 
					\section{IDE Component}\label{idecomponent}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
This is the code editor integrated into the frontend of the framework.
 | 
					This is the code editor integrated into the frontend of the framework.
 | 
				
			||||||
It allows users to select, display and edit files.
 | 
					It allows users to select, display and edit files.
 | 
				
			||||||
Developers can configure which directory on the file system of the \code{solvable}
 | 
					Developers can configure which directory of the file system should the editor list files from.
 | 
				
			||||||
container should the editor list files from.
 | 
					 | 
				
			||||||
The editor features the ``Deploy'' button referred to earlier in this paper, which is
 | 
					The editor features the ``Deploy'' button referred to earlier in this paper, which is
 | 
				
			||||||
capable of restarting processes that might execute a file visible in the editor.
 | 
					capable of restarting processes that might be running code from a file visible in the editor.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
To implement this IDE%
 | 
					To implement this IDE%
 | 
				
			||||||
\footnote{Integrated development environment}
 | 
					\footnote{Integrated development environment}
 | 
				
			||||||
@@ -124,7 +129,7 @@ This involves commnication with an event handler dedicated to this feature,
 | 
				
			|||||||
which is capable of reading and writing files to disk, while sending and receiving
 | 
					which is capable of reading and writing files to disk, while sending and receiving
 | 
				
			||||||
editor content from the frontend component.
 | 
					editor content from the frontend component.
 | 
				
			||||||
The interaction of this event handler and the Monaco editor provides a seamless
 | 
					The interaction of this event handler and the Monaco editor provides a seamless
 | 
				
			||||||
editing expirience, featuring autosave at configurable intervals, code completion,
 | 
					editing experience, featuring autosave at configurable intervals, code completion,
 | 
				
			||||||
automatic code coloring for several programming languages and more.
 | 
					automatic code coloring for several programming languages and more.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Perhaps the most ``magical'' feature of this editor is that if any process
 | 
					Perhaps the most ``magical'' feature of this editor is that if any process
 | 
				
			||||||
@@ -141,7 +146,7 @@ appears. If I select it I can confirm that I have successfully created an
 | 
				
			|||||||
empty file.
 | 
					empty file.
 | 
				
			||||||
After this let's run a \code{while} cycle in the command line which
 | 
					After this let's run a \code{while} cycle in the command line which
 | 
				
			||||||
peroadically appends some text to \code{file.txt}:
 | 
					peroadically appends some text to \code{file.txt}:
 | 
				
			||||||
\begin{lstlisting}[captionpos=b,caption={Bash While Cycle Writing to a File Periodically},
 | 
					\begin{lstlisting}[captionpos=b,caption={Bash while cycle writing to a file periodically},
 | 
				
			||||||
                   language=bash]
 | 
					                   language=bash]
 | 
				
			||||||
while true
 | 
					while true
 | 
				
			||||||
do
 | 
					do
 | 
				
			||||||
@@ -150,7 +155,7 @@ do
 | 
				
			|||||||
done
 | 
					done
 | 
				
			||||||
\end{lstlisting}
 | 
					\end{lstlisting}
 | 
				
			||||||
The results speak for themselves:
 | 
					The results speak for themselves:
 | 
				
			||||||
\pic{figures/ide_demo.png}{The Editor Demo Involving Automatic File Refreshing}
 | 
					\pic{figures/ide_demo.png}{The editor demo involving automatic file refreshing}
 | 
				
			||||||
As you can see, the file contents are automatially updated as the bash script appends
 | 
					As you can see, the file contents are automatially updated as the bash script appends
 | 
				
			||||||
to the file.
 | 
					to the file.
 | 
				
			||||||
This feature is implemented by using the inotify API%
 | 
					This feature is implemented by using the inotify API%
 | 
				
			||||||
@@ -158,13 +163,13 @@ This feature is implemented by using the inotify API%
 | 
				
			|||||||
{http://man7.org/linux/man-pages/man7/inotify.7.html}}
 | 
					{http://man7.org/linux/man-pages/man7/inotify.7.html}}
 | 
				
			||||||
provided by the Linux kernel to monitor file system events involving the directory listed by
 | 
					provided by the Linux kernel to monitor file system events involving the directory listed by
 | 
				
			||||||
the editor. The event handler of the editor hooks callbacks to said events which notify the
 | 
					the editor. The event handler of the editor hooks callbacks to said events which notify the
 | 
				
			||||||
Tutorial Framework to reload the list of files in the directory and the contents of
 | 
					Tutorial Framework to reload the list of files in the directory as well as the contents of
 | 
				
			||||||
the selected files.
 | 
					the selected file.
 | 
				
			||||||
The code making this feature possible is reused several times in the framework
 | 
					The code making this feature possible is reused several times in the framework
 | 
				
			||||||
for interesting purposes such as monitoring the logs of processes.
 | 
					for interesting purposes such as monitoring the logs of processes.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The editor also allows content creators to completely control it using API messages.
 | 
					The editor also allows content creators to completely control it using API messages.
 | 
				
			||||||
This involves selecting, reading and writing files as well as changing the
 | 
					This involves the selecting, reading and writing of files as well as changing the
 | 
				
			||||||
selected directory.
 | 
					selected directory.
 | 
				
			||||||
These features allow content creators to ``guide'' a user through code bases
 | 
					These features allow content creators to ``guide'' a user through code bases
 | 
				
			||||||
for example, where in each step of a tutorial a file is opened and explained
 | 
					for example, where in each step of a tutorial a file is opened and explained
 | 
				
			||||||
@@ -253,6 +258,8 @@ It's capabilities include starting, stopping and restarting processes.
 | 
				
			|||||||
It is also capable of emitting the standard out or standard error logs of processes
 | 
					It is also capable of emitting the standard out or standard error logs of processes
 | 
				
			||||||
(by broadcasting TFW messages).
 | 
					(by broadcasting TFW messages).
 | 
				
			||||||
This component can be iteracted with using TFW API messages.
 | 
					This component can be iteracted with using TFW API messages.
 | 
				
			||||||
 | 
					The ``Deploy'' button on the code editor uses this component to restart
 | 
				
			||||||
 | 
					processes.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The Tutorial Framework uses supervisor%
 | 
					The Tutorial Framework uses supervisor%
 | 
				
			||||||
\footnote{\href{http://supervisord.org}{http://supervisord.org}}
 | 
					\footnote{\href{http://supervisord.org}{http://supervisord.org}}
 | 
				
			||||||
 
 | 
				
			|||||||
@@ -6,10 +6,11 @@ interactive tutorials running inside Docker containers, semi-automatically showc
 | 
				
			|||||||
IT topics in real time. The user is guided through exercises using real environments with
 | 
					IT topics in real time. The user is guided through exercises using real environments with
 | 
				
			||||||
real software, all with the possibility of interaction at any time.
 | 
					real software, all with the possibility of interaction at any time.
 | 
				
			||||||
This technology can supplement/improve the way e-learning is usually done today
 | 
					This technology can supplement/improve the way e-learning is usually done today
 | 
				
			||||||
--- which is mostly articles and learning videos --- and help users get hands-on experience
 | 
					--- which is mostly through articles and learning videos --- and help users get hands-on
 | 
				
			||||||
on their way of acquiring knowledge.
 | 
					experience on their way of acquiring knowledge.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Currently more than 60 learning exercises based on this framework are available on
 | 
					Currently more than 60 learning exercises based on this framework are available on
 | 
				
			||||||
the e-learning platform called Avatao, with more being released every week.
 | 
					the e-learning platform called Avatao, with more being released every week.
 | 
				
			||||||
This text is going to justify the need for such technology, explain the ideas leading
 | 
					This text is going to justify the need for such technology, explain the ideas leading
 | 
				
			||||||
to it, discuss architecture, use-cases and more.
 | 
					to it, discuss use-cases, architecture, the features of the framework and how
 | 
				
			||||||
 | 
					developers can use it to create learning exercises.
 | 
				
			||||||
 
 | 
				
			|||||||
@@ -6,29 +6,35 @@ two Docker images:
 | 
				
			|||||||
\begin{itemize}
 | 
					\begin{itemize}
 | 
				
			||||||
    \item the \code{solvable} image is responsible for running the framework and the client
 | 
					    \item the \code{solvable} image is responsible for running the framework and the client
 | 
				
			||||||
          code depending on it
 | 
					          code depending on it
 | 
				
			||||||
    \item the \code{controller} image is responsible for solution checking (to figure out
 | 
					    \item the \code{controller} image is responsible for solution checking: to figure out
 | 
				
			||||||
          whether the user completed the tutorial or not)
 | 
					          whether the user has successfully completed the tutorial or not
 | 
				
			||||||
\end{itemize}
 | 
					\end{itemize}
 | 
				
			||||||
During most of this capter I am going to be discussing the \code{solvable} Docker image,
 | 
					During most of this chapter I am going to be discussing the \code{solvable} Docker image,
 | 
				
			||||||
with the exception of Section~\ref{solutioncheck}, where I will dive into how the
 | 
					with the exception of Section~\ref{solutioncheck}, where I will dive into how the
 | 
				
			||||||
\code{controller} image is implemented.
 | 
					\code{controller} image is implemented.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The most important feature of the framework is it's messaging system.
 | 
					The most important feature of the framework is it's messaging system.
 | 
				
			||||||
Basically what we need is a system where processes running inside a Docker container
 | 
					Basically what we need is a system where processes running inside a Docker container
 | 
				
			||||||
would be allowed to communicate with eachother. 
 | 
					would be allowed to communicate with eachother. 
 | 
				
			||||||
This is easy with lots of possible solutions (named pipes, sockets or shared memory to name a few).
 | 
					This task is very easy to solve, with lots of possible solutions
 | 
				
			||||||
The hard part is that frontend components running inside a web browser --- which could be
 | 
					(named pipes, sockets or shared memory to name a few).
 | 
				
			||||||
potentially on the other side of the planet --- would also need to partake in said communication.
 | 
					The hard part is that frontend components running inside a web browser --- which could
 | 
				
			||||||
 | 
					potentially be located on the other side of the planet%
 | 
				
			||||||
 | 
					\footnote{Potentially introducing all sorts of issues regarding latency} --- would
 | 
				
			||||||
 | 
					also need to partake in said communication.
 | 
				
			||||||
So what we need to create is something of a hybrid between an IPC system and something
 | 
					So what we need to create is something of a hybrid between an IPC system and something
 | 
				
			||||||
that can communicate with JavaScript running in a browser connected to it.
 | 
					that can communicate with JavaScript running in a browser connected to it.
 | 
				
			||||||
The solution the framework uses is a proxy server, which connects to frontend components
 | 
					The solution the framework uses is a proxy server, which connects to frontend components
 | 
				
			||||||
on one side and handles interprocess communication on the other side. 
 | 
					on one side and handles interprocess communication on the other side. 
 | 
				
			||||||
This way the server is capable of proxying messages between the two sides, enabling
 | 
					This way the server is capable of proxying messages between the two sides, enabling
 | 
				
			||||||
communitaion between them.
 | 
					communitaion between them.
 | 
				
			||||||
Notice that this way what we have is essentially an IPC system in which a web application
 | 
					Notice that this way what we have is essentially an IPC%
 | 
				
			||||||
 | 
					\footnote{Interprocess communication} system in which a web application
 | 
				
			||||||
can ``act like'' it was running on the backend in a sense: it is easily able to
 | 
					can ``act like'' it was running on the backend in a sense: it is easily able to
 | 
				
			||||||
communicate with processes on the backend, while in reality the web application
 | 
					communicate with processes running there, while in reality the web application
 | 
				
			||||||
runs in the browser of the user, on a completely different machine.
 | 
					is running in the browser of the user, on a completely different machine and it uses
 | 
				
			||||||
 | 
					some means of communication that is routed through the public internet to achieve this
 | 
				
			||||||
 | 
					effect.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\begin{note}
 | 
					\begin{note}
 | 
				
			||||||
The core idea and initial implementation of this server comes from Bálint Bokros,
 | 
					The core idea and initial implementation of this server comes from Bálint Bokros,
 | 
				
			||||||
@@ -38,54 +44,65 @@ message authentication, restoration of frontend state, a complete overhaul of th
 | 
				
			|||||||
state tracking system and the possibility for solution checking among other things).
 | 
					state tracking system and the possibility for solution checking among other things).
 | 
				
			||||||
If you are explicitly interested in the differences between the original POC implementation
 | 
					If you are explicitly interested in the differences between the original POC implementation
 | 
				
			||||||
(which is out of scope for this thesis due to lenght constraints) and the current
 | 
					(which is out of scope for this thesis due to lenght constraints) and the current
 | 
				
			||||||
framework please consult Bálint's excellent paper and Bachelor's Thesis on it\cite{BokaThesis}.
 | 
					framework please consult Bálint's excellent paper and Bachelor's thesis on it\cite{BokaThesis}.
 | 
				
			||||||
\end{note}
 | 
					\end{note}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Now let us take a closer look:
 | 
					Now let us take a closer look at the technology used to implement such a server and
 | 
				
			||||||
 | 
					some of the design decisions behind this:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{Connecting to the Frontend}
 | 
					\subsection{Connecting to the Frontend}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The old way of creating dynamic webpages was AJAX polling, which is basically sending
 | 
					The old way of creating dynamic webpages was AJAX%
 | 
				
			||||||
 | 
					\footnote{AJAX stands for Asynchronous JavaScript And XML, despite usually not having
 | 
				
			||||||
 | 
					anything to do with XML in practice}
 | 
				
			||||||
 | 
					polling, which is basically sending
 | 
				
			||||||
HTTP requests to a server at regular intervals from JavaScript to update the contents
 | 
					HTTP requests to a server at regular intervals from JavaScript to update the contents
 | 
				
			||||||
of your website (and as such requiring to go over the whole TCP handshake and the
 | 
					of your website (and as such requiring to go over the whole TCP handshake and the
 | 
				
			||||||
HTTP request-response on each update).
 | 
					HTTP request-response on each update).
 | 
				
			||||||
This has been superseded by WebSockets around 2011, which provide a full-duplex
 | 
					This has been superseded by WebSockets around 2011, which provide a full-duplex
 | 
				
			||||||
communication channel over TCP between your browser and the server.
 | 
					communication channel over TCP between your browser and the server.
 | 
				
			||||||
This is done by initiation a protocol handshake using the \code{Connection: Upgrade}
 | 
					This is done by initiating a protocol handshake using the \code{Connection: Upgrade}
 | 
				
			||||||
HTTP header, which establishes a premanent socket connection between the browser
 | 
					HTTP header, which establishes a premanent socket connection between the browser
 | 
				
			||||||
and the server.
 | 
					and the server.
 | 
				
			||||||
This allows for communication with lower overhead and latency facilitating efficient
 | 
					This allows for communication with lower overhead and latency facilitating efficient
 | 
				
			||||||
real-time applications.
 | 
					real-time applications, which were not always possible to create before due to
 | 
				
			||||||
 | 
					the overheads%
 | 
				
			||||||
 | 
					\footnote{In some applications this overhead could be bigger than the actual data sent,
 | 
				
			||||||
 | 
					such as singaling} introduced by AJAX polling.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The Tutorial Framework uses WebSockets to connect to it's web frontend.
 | 
					The Tutorial Framework uses WebSockets to connect to it's web frontend.
 | 
				
			||||||
The framework proxy server is capable to connecting to an arbirary number of websockets,
 | 
					The TFW proxy server is capable to connecting to an arbirary number of WebSockets,
 | 
				
			||||||
which allows opening different components in separate browser windows and tabs, or even
 | 
					which allows the framework to simultaneously connect to components running in
 | 
				
			||||||
in different browsers at once (such as opening a terminal in Chrome and an IDE in Firefox).
 | 
					separate browser windows and tabs, or even
 | 
				
			||||||
 | 
					in different browsers altogether (such as opening a terminal in Chrome and an IDE in Firefox).
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{Interprocess Communication}
 | 
					\subsection{Interprocess Communication}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
To handle communication with processes running inside the container TFW utilizes
 | 
					To handle communication with processes running inside the container TFW utilizes
 | 
				
			||||||
the asynchronous distributed messaging library ZeroMQ%
 | 
					the asynchronous distributed messaging called library ZeroMQ%
 | 
				
			||||||
\footnote{\href{http://zeromq.org}{http://zeromq.org}} or ZMQ as short.
 | 
					\footnote{\href{http://zeromq.org}{http://zeromq.org}} or ZMQ as short.
 | 
				
			||||||
The rationale behind this is that unlike other messaging systems such as
 | 
					The rationale behind this is that unlike other messaging systems such as
 | 
				
			||||||
RabbitMQ%
 | 
					RabbitMQ%
 | 
				
			||||||
\footnote{\href{https://www.rabbitmq.com}{https://www.rabbitmq.com}} or Redis%
 | 
					\footnote{\href{https://www.rabbitmq.com}{https://www.rabbitmq.com}} or Redis%
 | 
				
			||||||
\footnote{\href{https://redis.io}{https://redis.io}},
 | 
					\footnote{\href{https://redis.io}{https://redis.io}},
 | 
				
			||||||
ZMQ does not require a daemon (message broker process) and as such
 | 
					ZMQ does not require a message broker daemon to be running in the background at all times
 | 
				
			||||||
has a much lower memory footprint while still providing various messaging
 | 
					and as such has a much lower memory footprint while still providing various messaging
 | 
				
			||||||
patterns and bindings for almost any widely used programming language.
 | 
					patterns and bindings for almost any widely used programming language.
 | 
				
			||||||
An other --- yet untilized --- capability of this solution is that since ZMQ is capable
 | 
					An other --- yet untilized --- capability of this solution is that since ZMQ is capable
 | 
				
			||||||
of using simple TCP sockets, we could even communicate with processes running on remote
 | 
					of using simple TCP sockets, we could even communicate with processes running on remote
 | 
				
			||||||
hosts using the framework.
 | 
					hosts using the current architecture of the framework.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
There are various lower level and higher level alternatives for IPC other than
 | 
					There are various lower level and higher level alternatives for IPC other than
 | 
				
			||||||
ZMQ which were also considered during the desing process of the framework at some point.
 | 
					ZMQ which were also considered during the design process of the framework at some point.
 | 
				
			||||||
A few examples of top contenders and reasons for not using them in the end:
 | 
					A few examples of top contenders and reasons for not using them in the end:
 | 
				
			||||||
\begin{itemize}
 | 
					\begin{itemize}
 | 
				
			||||||
    \item The handling of raw TCP sockets would involve lot's of boilerplate logic that
 | 
					    \item The handling of raw TCP sockets would involve lot's of boilerplate logic that
 | 
				
			||||||
    already have quality implementations in messaging libraries: i.e.\ making sure that
 | 
					    already have quality implementations in messaging libraries: i.e.\ making sure that
 | 
				
			||||||
    all bytes are sent or received both require checking the return values of the
 | 
					    all bytes are sent or received both require constantly checking the return values of the
 | 
				
			||||||
    libc \code{send()} and \code{recv()} system calls, while ZMQ takes care of this
 | 
					    libc \code{send()} and \code{recv()} system calls%
 | 
				
			||||||
 | 
					\footnote{Developers forget this very often, resulting in almost untraceable bugs
 | 
				
			||||||
 | 
					that seem to occour randomly},
 | 
				
			||||||
 | 
					    while ZMQ takes care of this
 | 
				
			||||||
    extra logic involved and even provides higher level messaging patterns such as
 | 
					    extra logic involved and even provides higher level messaging patterns such as
 | 
				
			||||||
    subscribe-publish, which would need to be implemented on top of raw sockets again.
 | 
					    subscribe-publish, which would need to be implemented on top of raw sockets again.
 | 
				
			||||||
    \item Using something like gRPC\footnote{\href{https://grpc.io}{https://grpc.io}}
 | 
					    \item Using something like gRPC\footnote{\href{https://grpc.io}{https://grpc.io}}
 | 
				
			||||||
@@ -95,11 +112,15 @@ A few examples of top contenders and reasons for not using them in the end:
 | 
				
			|||||||
    which would make the framework
 | 
					    which would make the framework
 | 
				
			||||||
    less lightweight and flexible: socket communication with or without ZMQ does not
 | 
					    less lightweight and flexible: socket communication with or without ZMQ does not
 | 
				
			||||||
    force you to write synchronous or asynchronous code, whereas common HTTP servers
 | 
					    force you to write synchronous or asynchronous code, whereas common HTTP servers
 | 
				
			||||||
    are either async or pre-fork in nature, which extort certain design choices on code
 | 
					    are either async%
 | 
				
			||||||
 | 
					\footnote{Async servers use the \code{select} or \code{epoll} system calls among others
 | 
				
			||||||
 | 
					to avoid blocking on IO} or pre-fork%
 | 
				
			||||||
 | 
					\footnote{Pre-fork servers spawn multiple processes and threads to handle requests
 | 
				
			||||||
 | 
					simultaneously} in nature, which extorts certain design choices on code
 | 
				
			||||||
    built on them.
 | 
					    built on them.
 | 
				
			||||||
\end{itemize}
 | 
					\end{itemize}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\section{High Level Overview}
 | 
					\section{Architectural Overview}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Now being familiar with the technological basis of the framework we can now
 | 
					Now being familiar with the technological basis of the framework we can now
 | 
				
			||||||
discuss it in more detail.
 | 
					discuss it in more detail.
 | 
				
			||||||
@@ -116,11 +137,11 @@ Architecturally TFW consists of four main components:
 | 
				
			|||||||
          that is implemented as an event handler called \code{FSMManagingEventHandler}
 | 
					          that is implemented as an event handler called \code{FSMManagingEventHandler}
 | 
				
			||||||
\end{itemize}
 | 
					\end{itemize}
 | 
				
			||||||
Note that it is important to keep in mind that as I've mentioned previously,
 | 
					Note that it is important to keep in mind that as I've mentioned previously,
 | 
				
			||||||
the TFW Server and event handlers reside in the \code{solvable} Docker container.
 | 
					the TFW server and event handlers reside in the \code{solvable} Docker container.
 | 
				
			||||||
They all run in separate processes and only communicate using ZeroMQ sockets.
 | 
					They all run in separate processes and only communicate with eachother using ZeroMQ sockets.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
In the following sections I am going to explain each of the main components in
 | 
					In the following sections I am going to explain each of the main components in
 | 
				
			||||||
greater detail, as well as how they interact with each other,
 | 
					greater detail, as well as how they interact with eachother,
 | 
				
			||||||
their respective responsibilities,
 | 
					their respective responsibilities,
 | 
				
			||||||
some of the design choices behind them and more.
 | 
					some of the design choices behind them and more.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -149,7 +170,10 @@ Let's inspect further what a valid TFW message might look like:
 | 
				
			|||||||
 | 
					
 | 
				
			||||||
All valid messages \emph{must} include a \code{key} field as this is used by the
 | 
					All valid messages \emph{must} include a \code{key} field as this is used by the
 | 
				
			||||||
framework for addressing: event handlers and frontend components subscribe to one
 | 
					framework for addressing: event handlers and frontend components subscribe to one
 | 
				
			||||||
or more \code{key}s and only receive messages with \code{key}s they have
 | 
					or more of these \code{key}s and only receive%
 | 
				
			||||||
 | 
					\footnote{In reality they do receive them, just like how network interfaces receive all
 | 
				
			||||||
 | 
					ethernet frames, they just choose ignore the ones not concerning them}
 | 
				
			||||||
 | 
					messages with \code{key}s that they have
 | 
				
			||||||
subscribed to.
 | 
					subscribed to.
 | 
				
			||||||
It is possible to send a message with an empty key, however these messages will not
 | 
					It is possible to send a message with an empty key, however these messages will not
 | 
				
			||||||
be forwarded by the TFW server (but will reach it, so in case the target of a message
 | 
					be forwarded by the TFW server (but will reach it, so in case the target of a message
 | 
				
			||||||
@@ -165,12 +189,12 @@ at a later point in this paper.
 | 
				
			|||||||
The default behaviour of the TFW server is that it forwards all messages from coming from
 | 
					The default behaviour of the TFW server is that it forwards all messages from coming from
 | 
				
			||||||
the frontend to the event handlers and vice versa.
 | 
					the frontend to the event handlers and vice versa.
 | 
				
			||||||
So messages coming from the WebSockets of the frontend are forwarded to event handlers
 | 
					So messages coming from the WebSockets of the frontend are forwarded to event handlers
 | 
				
			||||||
via ZMQ and messages received through ZMQ from event handlers are forwarded to
 | 
					via ZMQ and messages received on ZMQ from event handlers are forwarded to
 | 
				
			||||||
the frontend via WebSockets.
 | 
					the frontend via WebSockets.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The TFW server is also capable of ``reflecting'' messages back to the side they were
 | 
					The TFW server is also capable of ``reflecting'' messages back to the side they were
 | 
				
			||||||
received on (to faciliate event handler to event handler for instance), or broadcast
 | 
					received from (to faciliate event handler to event handler communication for instance),
 | 
				
			||||||
messages to all components.
 | 
					or broadcast messages to all components.
 | 
				
			||||||
This is possible by embedding a whole TFW message in the \code{data} field of
 | 
					This is possible by embedding a whole TFW message in the \code{data} field of
 | 
				
			||||||
an outer wrapper message with a special \code{key} that signals to the TFW server that
 | 
					an outer wrapper message with a special \code{key} that signals to the TFW server that
 | 
				
			||||||
this message requires special attention.
 | 
					this message requires special attention.
 | 
				
			||||||
@@ -181,7 +205,7 @@ An example of this would be:
 | 
				
			|||||||
    "data":
 | 
					    "data":
 | 
				
			||||||
    {
 | 
					    {
 | 
				
			||||||
        ...
 | 
					        ...
 | 
				
			||||||
        The message you want to broadcast or mirror
 | 
					        The whole message you want to broadcast or mirror
 | 
				
			||||||
        (with it's own "key" and "data" fields)
 | 
					        (with it's own "key" and "data" fields)
 | 
				
			||||||
        ...
 | 
					        ...
 | 
				
			||||||
    }
 | 
					    }
 | 
				
			||||||
@@ -198,7 +222,7 @@ As discussed earlier, using ZeroMQ allows developers to implement event handlers
 | 
				
			|||||||
in a wide variety of programming languages.
 | 
					in a wide variety of programming languages.
 | 
				
			||||||
This is very important for the framework, as content creators often create
 | 
					This is very important for the framework, as content creators often create
 | 
				
			||||||
challenges that are very specific to a language, for example the showcasing
 | 
					challenges that are very specific to a language, for example the showcasing
 | 
				
			||||||
of a security vulnerability in an older version of Java.
 | 
					of a security vulnerability in an older version of the Java standard library.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
These event handlers are used to write most of the code developers wish to
 | 
					These event handlers are used to write most of the code developers wish to
 | 
				
			||||||
integrate with the framework.
 | 
					integrate with the framework.
 | 
				
			||||||
@@ -210,11 +234,20 @@ based on this knowledge.
 | 
				
			|||||||
An event handler such as this could be invoked by sending a message to it
 | 
					An event handler such as this could be invoked by sending a message to it
 | 
				
			||||||
at any time when the running of the tests would be required.
 | 
					at any time when the running of the tests would be required.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					An interesting thing to mention is that there \emph{could} be event handlers which
 | 
				
			||||||
 | 
					broadcast messages with a \code{key} that they are also subscribed to.
 | 
				
			||||||
 | 
					This can distrupt their behaviour in weird ways if they are not prepared to
 | 
				
			||||||
 | 
					deal with their own ``echoes''.
 | 
				
			||||||
 | 
					The framework offers a solution for this by providing a special
 | 
				
			||||||
 | 
					event handler type, which is capable of filtering out it's own broadcasts.
 | 
				
			||||||
 | 
					The way they do this is by caching the checksum of every message they broadcast,
 | 
				
			||||||
 | 
					and ignore the first message that comes back with the same checksum.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{Frontend}
 | 
					\subsection{Frontend}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
This is a web application that runs in the browser of the user and uses
 | 
					This is a web application that runs in the browser of the user and uses
 | 
				
			||||||
multiple WebSocket connections to connect to the TFW server.
 | 
					multiple WebSockets to connect to the TFW server.
 | 
				
			||||||
Due to rapidly increasing complexity the original implementation (written in
 | 
					Due to rapidly increasing complexity, the original implementation (written in
 | 
				
			||||||
plain JavaScript with jQuery%
 | 
					plain JavaScript with jQuery%
 | 
				
			||||||
\footnote{\href{https://jquery.com}{https://jquery.com}} and Bootstrap%
 | 
					\footnote{\href{https://jquery.com}{https://jquery.com}} and Bootstrap%
 | 
				
			||||||
\footnote{\href{https://getbootstrap.com}{https://getbootstrap.com}}) was becoming
 | 
					\footnote{\href{https://getbootstrap.com}{https://getbootstrap.com}}) was becoming
 | 
				
			||||||
@@ -234,7 +267,7 @@ Other reasons included that the frontend of the Avatao platform is also written
 | 
				
			|||||||
in Angular (bonus points for experienced team members in the company).
 | 
					in Angular (bonus points for experienced team members in the company).
 | 
				
			||||||
An other good thing going for it is that Angular forces you to use TypeScript%
 | 
					An other good thing going for it is that Angular forces you to use TypeScript%
 | 
				
			||||||
\footnote{\href{https://www.typescriptlang.org}{https://www.typescriptlang.org}}
 | 
					\footnote{\href{https://www.typescriptlang.org}{https://www.typescriptlang.org}}
 | 
				
			||||||
which tries to remedy the issues\cite{JavaScript}
 | 
					which tries to remedy some of the issues\cite{JavaScript}
 | 
				
			||||||
with JavaScript by being a language that transpiles to JavaScript while
 | 
					with JavaScript by being a language that transpiles to JavaScript while
 | 
				
			||||||
strongly encouraging things like static typing or Object Oriented Principles.
 | 
					strongly encouraging things like static typing or Object Oriented Principles.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -244,11 +277,11 @@ strongly encouraging things like static typing or Object Oriented Principles.
 | 
				
			|||||||
 | 
					
 | 
				
			||||||
A good chunk of the framework codebase is a bunch of pre-made, built-in components
 | 
					A good chunk of the framework codebase is a bunch of pre-made, built-in components
 | 
				
			||||||
that implement commonly required functionality for developers to use.
 | 
					that implement commonly required functionality for developers to use.
 | 
				
			||||||
These components usually involve an event handler and an Angular component which
 | 
					These components usually involve an event handler and an Angular component
 | 
				
			||||||
communicates with it to realize some functionality.
 | 
					communicating with eachother to realize some sort of functionality.
 | 
				
			||||||
An example would be the built-in code editor of the framework
 | 
					An example would be the built-in code editor of the framework
 | 
				
			||||||
(visible on the left side of Figure~\ref{figures/tfw_frontend.png}).
 | 
					(visible on the right side of Figure~\ref{figures/tfw_frontend.png}).
 | 
				
			||||||
This code editor is essentially a Monaco editor%
 | 
					This code editor essentially is a Monaco editor%
 | 
				
			||||||
\footnote{\href{https://microsoft.github.io/monaco-editor/}
 | 
					\footnote{\href{https://microsoft.github.io/monaco-editor/}
 | 
				
			||||||
{https://microsoft.github.io/monaco-editor/}}
 | 
					{https://microsoft.github.io/monaco-editor/}}
 | 
				
			||||||
instance integrated into Angular and upgraded with the capability to
 | 
					instance integrated into Angular and upgraded with the capability to
 | 
				
			||||||
@@ -256,21 +289,23 @@ exchanges messages with an event handler to save, read and edit files
 | 
				
			|||||||
that reside in the writeable file system of the \code{solvable}
 | 
					that reside in the writeable file system of the \code{solvable}
 | 
				
			||||||
Docker container.
 | 
					Docker container.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
All of the built-ins come with full API documentation explaining what they do
 | 
					All of the built-ins come with a full API documentation explaining what they do
 | 
				
			||||||
on receiving specific messages, and what messages they emit on different events.
 | 
					on receiving specific messages, and what kind of messages they may emit on different events.
 | 
				
			||||||
This greatly expands the capabilities of the framework, since it allows
 | 
					This greatly expands the capabilities of the framework, since it allows
 | 
				
			||||||
developers to do things including, but not limited to:
 | 
					developers to do things including, but not limited to:
 | 
				
			||||||
\begin{itemize}
 | 
					\begin{itemize}
 | 
				
			||||||
    \item making the code editor automatically appear in sections
 | 
					    \item making the code editor automatically appear in sections
 | 
				
			||||||
          of the tutorial where the user needs to use it
 | 
					          of the tutorial where the user needs to use it, then disappear
 | 
				
			||||||
 | 
					          when it is no longer needed to conserve space
 | 
				
			||||||
    \item inject commands into the user's terminal
 | 
					    \item inject commands into the user's terminal
 | 
				
			||||||
    \item hook into messages emitted from components to detect events, such as
 | 
					    \item hook callbacks to run code on messages emitted from components to
 | 
				
			||||||
 | 
					          detect events, such as
 | 
				
			||||||
          to detect if the user has clicked a button or executed a command
 | 
					          to detect if the user has clicked a button or executed a command
 | 
				
			||||||
          in the terminal
 | 
					          in the terminal
 | 
				
			||||||
    \item monitor the logs (stdout or stderr) of a given process
 | 
					    \item monitor the logs (stdout or stderr) of a given process in real time
 | 
				
			||||||
\end{itemize}
 | 
					\end{itemize}
 | 
				
			||||||
Every pre-made component is designed with the mindset to allow flexible
 | 
					Every pre-made component is designed with the mindset to allow flexible
 | 
				
			||||||
and creative usage by developers, with the possibility of future extensions.
 | 
					and creative usage by developers, with the added possibility of future extensions.
 | 
				
			||||||
Often when developers require certain new features, they open an issue on
 | 
					Often when developers require certain new features, they open an issue on
 | 
				
			||||||
the git repository of the framework for me to review and possibly implement
 | 
					the git repository of the framework for me to review and possibly implement
 | 
				
			||||||
later.
 | 
					later.
 | 
				
			||||||
@@ -279,18 +314,22 @@ One example would be when a developer wanted to automatically advance the tutori
 | 
				
			|||||||
when the user has entered a specific string into a file.
 | 
					when the user has entered a specific string into a file.
 | 
				
			||||||
This one didn't even require a new feature: I recommended him to implement an event
 | 
					This one didn't even require a new feature: I recommended him to implement an event
 | 
				
			||||||
handler listening to the messages of the built-in file editor, filter the messages
 | 
					handler listening to the messages of the built-in file editor, filter the messages
 | 
				
			||||||
which contain file content that is going to be written to disk, and simply
 | 
					which contain file content that is being sent to be written to disk, and simply
 | 
				
			||||||
search these messages for the given string.
 | 
					search these messages for the given string.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The exact capabilities of these built-in components will be explained in greater
 | 
					The exact capabilities of these built-in components will be explained in greater
 | 
				
			||||||
detail in a later chapter.
 | 
					detail in Chapter~\ref{atouroftfw}.
 | 
				
			||||||
 | 
					Developers who are well-aware of these capabilites are able to use the framework in extremely
 | 
				
			||||||
 | 
					creative ways allowing for very interesting functionality, such as the above example.
 | 
				
			||||||
 | 
					The components of TFW can often be combined to work together in unexpected, yet useful
 | 
				
			||||||
 | 
					ways, similarly how command-line utilities on UNIX-like systems do.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{TFW Finite State Machine}
 | 
					\subsection{TFW Finite State Machine}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
An important requirement we have specified during~\ref{requirements} was that
 | 
					An important requirement we have specified during~\ref{requirements} was that
 | 
				
			||||||
the framework must be capable of tracking user progress.
 | 
					the framework must be capable of tracking user progress.
 | 
				
			||||||
TFW allows developers to define a \emph{finite state machine}
 | 
					TFW allows developers to define a \emph{finite state machine}
 | 
				
			||||||
which is capable of describing the desired ``story'' of a tutorial.
 | 
					which is capable of describing the desired ``story'' of a learning exercise.
 | 
				
			||||||
The states of the machine could be certain points in time during the completion of the
 | 
					The states of the machine could be certain points in time during the completion of the
 | 
				
			||||||
tutorial envisioned and transitions could be events that influence the
 | 
					tutorial envisioned and transitions could be events that influence the
 | 
				
			||||||
state, such as the editing of files, execution of commands and so on.
 | 
					state, such as the editing of files, execution of commands and so on.
 | 
				
			||||||
@@ -301,23 +340,25 @@ Take the fixing of a SQL Injection%
 | 
				
			|||||||
vulnerability as an example.
 | 
					vulnerability as an example.
 | 
				
			||||||
Let's assume, that the source code is vulnerable to a SQL injection attack
 | 
					Let's assume, that the source code is vulnerable to a SQL injection attack
 | 
				
			||||||
because it tries to compose a query with string concatenation instead of
 | 
					because it tries to compose a query with string concatenation instead of
 | 
				
			||||||
using a parameterized query provided by the database library.
 | 
					using a prepared statement provided by the database library.
 | 
				
			||||||
A challenge developer could implement an FSM in the framework that looks like this:
 | 
					A challenge developer could implement an FSM in the framework that looks like this:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\pic[width=.6\textwidth]{figures/tfw_fsm.png}{An Example for a Finite State Machine in TFW}
 | 
					\pic[width=.6\textwidth]{figures/tfw_fsm.png}{An example for a finite state machine in TFW}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
In case the source file has been edited, the unit test cases designed to detect
 | 
					In case the source file has been edited, the unit test cases designed to detect
 | 
				
			||||||
whether the code is vulnerable or not are invoked.
 | 
					whether the code is vulnerable or not are invoked.
 | 
				
			||||||
Depending on the results three cases are possible:
 | 
					Depending on the results three cases are possible:
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\begin{description}
 | 
					\begin{description}
 | 
				
			||||||
    \item[All test cases have succeeded:] If all the tests succeeded then the user has managed
 | 
					    \item[All test cases have succeeded:] If all the tests cases have ran successfully,
 | 
				
			||||||
 | 
					    then the user has managed
 | 
				
			||||||
    to fix the code properly and we can display a congratulating message accordingly.
 | 
					    to fix the code properly and we can display a congratulating message accordingly.
 | 
				
			||||||
    \item[All test cases have failed:] In this case the solution is incorrect
 | 
					    \item[All test cases have failed:] In this case the submitted solution is incorrect
 | 
				
			||||||
    and we can offer some hints.
 | 
					    and we should offer some hints, so that the user can try again more effectively,
 | 
				
			||||||
 | 
					    optionally displaying more and more hints with each successive failure.
 | 
				
			||||||
    \item[Some test cases have succeeded:] It is possible that the based on the test cases
 | 
					    \item[Some test cases have succeeded:] It is possible that the based on the test cases
 | 
				
			||||||
    that have succeeded and failed we can determine that the user tried to blacklist
 | 
					    that have succeeded and failed we can determine that the user has tried to blacklist
 | 
				
			||||||
    certain SQL keywords. This is a common, but incorrect solution of fixing a SQL
 | 
					    certain SQL keywords. This is a common, but incorrect ``solution'' of fixing a SQL
 | 
				
			||||||
    injection vulnerability. Now we can explain to users why their solution is wrong,
 | 
					    injection vulnerability. Now we can explain to users why their solution is wrong,
 | 
				
			||||||
    and give them helpful tips.
 | 
					    and give them helpful tips.
 | 
				
			||||||
\end{description}
 | 
					\end{description}
 | 
				
			||||||
@@ -330,10 +371,11 @@ This is a very engaging feature that offers an immersive learning experience for
 | 
				
			|||||||
users, which many solutions for distance education lack so often.
 | 
					users, which many solutions for distance education lack so often.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Developers can use a YAML file or write Python code to implement finite
 | 
					Developers can use a YAML file or write Python code to implement finite
 | 
				
			||||||
state machines.
 | 
					state machines in TFW\@. This is going to be further detailed in
 | 
				
			||||||
In state machine implementations it is possbile to subscribe callbacks to be
 | 
					Chapter~\ref{usingtfw}.
 | 
				
			||||||
 | 
					In the implementation of state machines it is also possbile to subscribe callbacks to be
 | 
				
			||||||
invoked on certain events regarding the machine, such as before and after
 | 
					invoked on certain events regarding the machine, such as before and after
 | 
				
			||||||
state transitions, or onentering and exiting a state.
 | 
					state transitions, or on entering and exiting a state.
 | 
				
			||||||
It is \emph{very} important to be aware of these callbacks, as much of the
 | 
					It is \emph{very} important to be aware of these callbacks, as much of the
 | 
				
			||||||
actual tutorial logic is often going to be implemented in these.
 | 
					actual tutorial logic is often going to be implemented in these.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -351,22 +393,28 @@ The \code{trigger} field of a message can be used to step the framework FSM
 | 
				
			|||||||
if all preconditions are met.
 | 
					if all preconditions are met.
 | 
				
			||||||
The way this works is if the TFW server encounters a message with a
 | 
					The way this works is if the TFW server encounters a message with a
 | 
				
			||||||
\code{trigger} defined, it notifies the event handler managing
 | 
					\code{trigger} defined, it notifies the event handler managing
 | 
				
			||||||
the state machine.
 | 
					the state machine so it can attempt activating said \code{trigger}.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Since messages can come from unauthenticated sources, it is possible to
 | 
					Since messages in the system can come from unauthenticated sources (such as the frontend),
 | 
				
			||||||
 | 
					it is possible to
 | 
				
			||||||
enforce the authentication of privileged messages, such as messages containing a \code{trigger}.
 | 
					enforce the authentication of privileged messages, such as messages containing a \code{trigger}.
 | 
				
			||||||
The framework allows trusted code to access a cryptographic key on the file system, which
 | 
					The framework allows trusted code to access a cryptographic key stored on the file system
 | 
				
			||||||
 | 
					with proper permissions, which
 | 
				
			||||||
can be used to digitally sign messages (this is what the \code{signature} message
 | 
					can be used to digitally sign messages (this is what the \code{signature} message
 | 
				
			||||||
field is designed for).
 | 
					field is designed for) using HMAC%
 | 
				
			||||||
In this case the TFW server will only forward privileged messages that
 | 
					\footnote{Hash-based message authentication code}.
 | 
				
			||||||
have a valid signature.
 | 
					In this case the TFW server will only forward the privileged messages that
 | 
				
			||||||
 | 
					have a valid signature, and the evend handler managing the state machine
 | 
				
			||||||
 | 
					will also validate the signature of messages it receives
 | 
				
			||||||
 | 
					(and sign the updates it broadcasts as well, so that other components can verify that
 | 
				
			||||||
 | 
					they come from a trusted source).
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{Solution checking}\label{solutioncheck}
 | 
					\subsection{Solution checking}\label{solutioncheck}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Traditionally most challenges on the Avatao platform implement a Docker image called
 | 
					Traditionally most challenges on the Avatao platform implement a Docker image called
 | 
				
			||||||
\code{controller}, which is responsible for detecting the successful
 | 
					\code{controller}, which is responsible for detecting the successful
 | 
				
			||||||
solution of a challenge.
 | 
					solution of a challenge.
 | 
				
			||||||
When using the Tutorial Framework a pre-implemented \code{controller}
 | 
					When using the Tutorial Framework, a pre-implemented \code{controller}
 | 
				
			||||||
image is available, which listens to messages emitted by the
 | 
					image is available, which listens to messages emitted by the
 | 
				
			||||||
framework FSM, and detects if the final state defined by developers is reached.
 | 
					framework FSM, and detects if the final state defined by developers is reached.
 | 
				
			||||||
This means that if content creators implement a proper FSM, the solution checking
 | 
					This means that if content creators implement a proper FSM, the solution checking
 | 
				
			||||||
@@ -378,4 +426,5 @@ traditional hacking challenges, such as exercises developed for CTF%
 | 
				
			|||||||
\footnote{A ``capture the flag'' game is a competition designed for professionals
 | 
					\footnote{A ``capture the flag'' game is a competition designed for professionals
 | 
				
			||||||
--- or just people interested in the field ---  to sharpen their skills in IT security.
 | 
					--- or just people interested in the field ---  to sharpen their skills in IT security.
 | 
				
			||||||
Avatao often organises similar events.}
 | 
					Avatao often organises similar events.}
 | 
				
			||||||
events.
 | 
					events, as the controller image is also capable of verifying the authenticity of
 | 
				
			||||||
 | 
					FSM update messages via inspecting their signatures.
 | 
				
			||||||
 
 | 
				
			|||||||
@@ -3,7 +3,7 @@
 | 
				
			|||||||
\section{Project justification}
 | 
					\section{Project justification}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
As the world is being completely engulfed by software, the need for accessible, but
 | 
					As the world is being completely engulfed by software, the need for accessible, but
 | 
				
			||||||
high quality learning materials on software engineering and especially secure software
 | 
					high quality learning materials covering software engineering and especially secure software
 | 
				
			||||||
engineering is on the rise.
 | 
					engineering is on the rise.
 | 
				
			||||||
While we are enjoying the comfort that information technology provides us, we often forget
 | 
					While we are enjoying the comfort that information technology provides us, we often forget
 | 
				
			||||||
about the risks involved in relying so much on software in our everyday lives.
 | 
					about the risks involved in relying so much on software in our everyday lives.
 | 
				
			||||||
@@ -39,14 +39,14 @@ knowledge is something that comes naturally, rather than something we have to st
 | 
				
			|||||||
I believe that this is something that \emph{can} and \emph{should} be applied to
 | 
					I believe that this is something that \emph{can} and \emph{should} be applied to
 | 
				
			||||||
everything we do as a society.
 | 
					everything we do as a society.
 | 
				
			||||||
The only thing we can hope and work for is to become better and better as time
 | 
					The only thing we can hope and work for is to become better and better as time
 | 
				
			||||||
and generations pass.
 | 
					and generations pass by.
 | 
				
			||||||
We \emph{must} do better, and education is the way forward.
 | 
					We \emph{must} do better, and education is the way forward.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
The short term goal of this project --- and the goal of this thesis --- is to provide
 | 
					The short term goal of this project --- and the goal of this thesis --- is to provide
 | 
				
			||||||
a new angle in the education of software engineering, especially secure software
 | 
					a new angle in the education of software engineering, especially secure software
 | 
				
			||||||
engineering based on the aspirations above, with the long term goal of bringing
 | 
					engineering based on the aspirations above, with the long term goal of bringing
 | 
				
			||||||
something new to the table in the matter of IT education as a whole
 | 
					something new to the table in the matter of IT education as a whole
 | 
				
			||||||
(not just developers, but users as well).
 | 
					(not just for developers, but for users as well).
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\section{A Short Introduction to Avatao}
 | 
					\section{A Short Introduction to Avatao}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -96,11 +96,12 @@ things like exercises involving the use of Docker or Windows based challenges.
 | 
				
			|||||||
\section{Emergence}\label{intro:emergence}
 | 
					\section{Emergence}\label{intro:emergence}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
While working as a content creator I have stumbled into the idea of automating the completion
 | 
					While working as a content creator I have stumbled into the idea of automating the completion
 | 
				
			||||||
of challenges for QA\footnote{Quality Assurrance} and demo purposes%
 | 
					of challenges for QA\footnote{Quality Assurrance} and demo purposes.
 | 
				
			||||||
\footnote{I used to record short videos or GIFs to showcase my content to management}.
 | 
					I used to record short videos or GIFs to showcase my content to management.
 | 
				
			||||||
In a certain scenario I was required to integrate a web based terminal emulator in a
 | 
					In a certain scenario I was required to integrate a web based terminal emulator into a
 | 
				
			||||||
frontend application to improve user experience by making it possible to use a shell
 | 
					frontend application to improve user experience by making it possible to use a shell
 | 
				
			||||||
right on the website rather than having to connect through SSH\@.
 | 
					right on the website rather than having to connect through SSH\@.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
After I got this working I was looking into writing hacky bash scripts to automate the steps
 | 
					After I got this working I was looking into writing hacky bash scripts to automate the steps
 | 
				
			||||||
required to complete the challenge in order to make it easier for me to record the solution,
 | 
					required to complete the challenge in order to make it easier for me to record the solution,
 | 
				
			||||||
as I have often found myself recording over and over again for a demo without any mistakes.
 | 
					as I have often found myself recording over and over again for a demo without any mistakes.
 | 
				
			||||||
@@ -109,6 +110,7 @@ to a hidden gem of a project on GitHub called \code{demo-magic}%
 | 
				
			|||||||
\footnote{\href{https://github.com/paxtonhare/demo-magic}{https://github.com/paxtonhare/demo-magic}},
 | 
					\footnote{\href{https://github.com/paxtonhare/demo-magic}{https://github.com/paxtonhare/demo-magic}},
 | 
				
			||||||
which is esentially a bash script that simulates someone typing into a terminal and executing
 | 
					which is esentially a bash script that simulates someone typing into a terminal and executing
 | 
				
			||||||
commands.
 | 
					commands.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
I have created a fork%
 | 
					I have created a fork%
 | 
				
			||||||
\footnote{
 | 
					\footnote{
 | 
				
			||||||
\href{https://git.strongds.hu/mrtoth/demo.sh/src/master/demo.sh}
 | 
					\href{https://git.strongds.hu/mrtoth/demo.sh/src/master/demo.sh}
 | 
				
			||||||
@@ -131,7 +133,7 @@ commands executed during the solution process.
 | 
				
			|||||||
I was quite pleased with myself, no longer having to do the busywork of recording videos,
 | 
					I was quite pleased with myself, no longer having to do the busywork of recording videos,
 | 
				
			||||||
but what I did not know was that I have accidentally
 | 
					but what I did not know was that I have accidentally
 | 
				
			||||||
did something far more than a hacky bash script solving challenges, as this little script
 | 
					did something far more than a hacky bash script solving challenges, as this little script
 | 
				
			||||||
would help formulate the idea of the project \emph{Tutorial Framework} or just \emph{TFW}.
 | 
					would help formulate the idea of the \emph{Tutorial Framework} or just \emph{TFW}.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\section{Vision of the Tutorial Framework}
 | 
					\section{Vision of the Tutorial Framework}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -177,14 +179,20 @@ Meanwhile a console could show the output of the webserver.
 | 
				
			|||||||
For example if the source code the user tried to deploy was invalid, the framework
 | 
					For example if the source code the user tried to deploy was invalid, the framework
 | 
				
			||||||
should report the exact exception raised while running the application.
 | 
					should report the exact exception raised while running the application.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\pic{figures/webapp_and_editor.png}{The Code Editor and Web Application Example In TFW}
 | 
					\pic{figures/webapp_and_editor.png}{The code editor and web application example in TFW}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
Even if we did all this, we would still need a way to integrate this whole thing into 
 | 
					Even if we did all this, we would still need a way to integrate this whole thing into 
 | 
				
			||||||
a web based frontend with a file editor, terminal, chat window and stuff like that.
 | 
					a web based frontend with a file editor, terminal, chat window and stuff like that.
 | 
				
			||||||
Turns out that today all this can be done by writing a few hundred lines of Python
 | 
					Turns out that today all this can be done by writing a few hundred lines of Python
 | 
				
			||||||
code which uses the Tutorial Framework.
 | 
					code which uses the Tutorial Framework.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\pic{figures/webapp_and_editor_err.png}{Invalid Code and Deployment Failure with Process Output}
 | 
					\pic{figures/webapp_and_editor_err.png}{Invalid code and deployment failure with process output}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					Note that it is possible to try out the current version of the Tutorial Framewok
 | 
				
			||||||
 | 
					using a guest account on the Avatao platform on this
 | 
				
			||||||
 | 
					\href{https://platform.avatao.com/paths/d0ccef1f-0389-45bf-9d44-e85b86d66c49/challenges/a7e08c0a-199f-4f8d-aa7e-51b6e9bfcb15}{url}%
 | 
				
			||||||
 | 
					\footnote{\href{https://platform.avatao.com/paths/d0ccef1f-0389-45bf-9d44-e85b86d66c49/challenges/a7e08c0a-199f-4f8d-aa7e-51b6e9bfcb15}
 | 
				
			||||||
 | 
					{https://platform.avatao.com/paths/d0ccef1f-0389-45bf-9d44-e85b86d66c49/challenges/a7e08c0a-199f-4f8d-aa7e-51b6e9bfcb15}}.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
\subsection{Project Requirements}\label{requirements}
 | 
					\subsection{Project Requirements}\label{requirements}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
@@ -199,8 +207,8 @@ To achieve our goals we would need:
 | 
				
			|||||||
    \item a way to keep track of user progress
 | 
					    \item a way to keep track of user progress
 | 
				
			||||||
    \item a way to to handle various events (i.e.\ we can react when
 | 
					    \item a way to to handle various events (i.e.\ we can react when
 | 
				
			||||||
          the user has edited a file, or has executed a command in the terminal)
 | 
					          the user has edited a file, or has executed a command in the terminal)
 | 
				
			||||||
    \item a highly flexible messaging system, in which processes and
 | 
					    \item a highly flexible messaging system, in which processes running on the backend and
 | 
				
			||||||
          frontend components (running in a web browser) could communicate with eachother
 | 
					          frontend components running in a web browser could communicate with eachother
 | 
				
			||||||
    \item a web based frontend with lots of built-in options (terminal, file editor, chat
 | 
					    \item a web based frontend with lots of built-in options (terminal, file editor, chat
 | 
				
			||||||
          window, etc.) that use said messaging system
 | 
					          window, etc.) that use said messaging system
 | 
				
			||||||
    \item stable APIs that can be exposed to content creators to work with (so that
 | 
					    \item stable APIs that can be exposed to content creators to work with (so that
 | 
				
			||||||
@@ -236,14 +244,14 @@ completely rewritten due to an increased focus on code quality,
 | 
				
			|||||||
extensibility and API stability required by new features.
 | 
					extensibility and API stability required by new features.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
It is interesting to note, that when I've mentioned that the project requirements
 | 
					It is interesting to note, that when I've mentioned that the project requirements
 | 
				
			||||||
were kept general on purpose (\ref{requirements}) I had good reason to do so.
 | 
					were kept general on purpose in~\ref{requirements}, I had good reason to do so.
 | 
				
			||||||
When taking a look at the requirements of Bálint's Thesis, much of that
 | 
					When taking a look at the requirements of Bálint's thesis, much of that
 | 
				
			||||||
is completely obsolete by now.
 | 
					is completely obsolete by now.
 | 
				
			||||||
But since the project has followed Agile Methodology%
 | 
					But since the project has followed Agile Methodology%
 | 
				
			||||||
\footnote{Manifesto for Agile Software Development:
 | 
					\footnote{Manifesto for Agile Software Development:
 | 
				
			||||||
\href{https://agilemanifesto.org}{https://agilemanifesto.org}}
 | 
					\href{https://agilemanifesto.org}{https://agilemanifesto.org}}
 | 
				
			||||||
from the start, we were able to adapt to these changes without losing
 | 
					from the start, we were able to adapt to these changes without losing
 | 
				
			||||||
the progess he made in said Thesis. Quoting from the Agile Manifesto: 
 | 
					the progess he made in said thesis. Quoting from the Agile Manifesto: 
 | 
				
			||||||
``Responding to change over following a plan''.
 | 
					``Responding to change over following a plan''.
 | 
				
			||||||
This is a really important takeaway.
 | 
					This is a really important takeaway.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 
 | 
				
			|||||||
@@ -67,17 +67,31 @@ us to do so.
 | 
				
			|||||||
 | 
					
 | 
				
			||||||
\section{Things That I Have Learned}
 | 
					\section{Things That I Have Learned}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
I've spent a long time working on and maintaining this project.
 | 
					Despite being an enthusiast of \LaTeX{} for a few years now, I still managed to learn a great
 | 
				
			||||||
 | 
					deal about it while working on this text.
 | 
				
			||||||
 | 
					This might seem like something unrelated, but most documentation issues with software often
 | 
				
			||||||
 | 
					come from the fact that developers usually dislike writing documentation.
 | 
				
			||||||
 | 
					Since working with \LaTeX{} I \emph{love} writing larger bodies of text such as this,
 | 
				
			||||||
 | 
					as I just simply enjoy admiring quality typography which WYSIWYG%
 | 
				
			||||||
 | 
					\footnote{What You See Is What You Get} editors just seem unable to produce.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
 | 
					I've spent a long time working on and maintaining the Tutorial Framework.
 | 
				
			||||||
While the list of technical things I've learned is long and exciting, I also feel like
 | 
					While the list of technical things I've learned is long and exciting, I also feel like
 | 
				
			||||||
I've learned a lot about supporting other developers, project management and communication.
 | 
					I've learned a lot about supporting other developers, project management and communication.
 | 
				
			||||||
 | 
					
 | 
				
			||||||
A thing that I will always remember as a software engineer and I've learned during this period
 | 
					The most important thing, that I will always remember as a software engineer
 | 
				
			||||||
 | 
					and is something that I've learned during this period
 | 
				
			||||||
is to never, ever lower my expectations regarding code quality.
 | 
					is to never, ever lower my expectations regarding code quality.
 | 
				
			||||||
No matter what anybody tells you about ``but we have to finish until'' and stuff like that,
 | 
					No matter what anybody tells you about things like ``but we have to make haste and finish in time'',
 | 
				
			||||||
in the long run it is always like shooting yourself in the foot.
 | 
					in the long run, making compromises in code quality is always like shooting yourself in the leg.
 | 
				
			||||||
We as professionals must always thrive for excellence, and must always express our
 | 
					We as professionals must always \emph{thrive} for excellence, and must always express our
 | 
				
			||||||
deepest respect towards our craft.
 | 
					deepest respect towards our craft.
 | 
				
			||||||
The only way we can do this is by creating quality software as craftsmen.
 | 
					The only way we can do this is by creating quality software while being a responsible
 | 
				
			||||||
 | 
					\emph{craftsman}.
 | 
				
			||||||
 | 
					It is a thing of great importance, which cannot be stressed enough, that in the software
 | 
				
			||||||
 | 
					field \emph{craftsmanship}%
 | 
				
			||||||
 | 
					\footnote{\href{http://manifesto.softwarecraftsmanship.org}
 | 
				
			||||||
 | 
					{http://manifesto.softwarecraftsmanship.org}} is what matters most.
 | 
				
			||||||
Many developers fail to understand that no matter how insignificant the code you write
 | 
					Many developers fail to understand that no matter how insignificant the code you write
 | 
				
			||||||
today may seem, software is art, and art is something worth pursuing just for the sake
 | 
					today may seem, software is art, and art is something worth pursuing just for the sake
 | 
				
			||||||
of doing art itself.
 | 
					of doing art itself.
 | 
				
			||||||
 
 | 
				
			|||||||
@@ -1,4 +1,4 @@
 | 
				
			|||||||
\chapter{Using the Framework}
 | 
					\chapter{Using the Framework}\label{usingtfw}
 | 
				
			||||||
 | 
					
 | 
				
			||||||
In this section I am going to dive into further detail on how client code is supposed
 | 
					In this section I am going to dive into further detail on how client code is supposed
 | 
				
			||||||
to use the framework, some of the design decisions behind this and how everything is
 | 
					to use the framework, some of the design decisions behind this and how everything is
 | 
				
			||||||
 
 | 
				
			|||||||
										
											Binary file not shown.
										
									
								
							| 
		 Before Width: | Height: | Size: 73 KiB After Width: | Height: | Size: 73 KiB  | 
		Reference in New Issue
	
	Block a user