Continue writing chapter 'A Tour of TFW'

content/a_tour_of_tfw.tex

@@ -23,7 +23,7 @@ For example, to inject a command into the terminal one would use a message like
     }
 }
 \end{lstlisting}
-Notice the \texttt{``\\n''} at the end of the command.
+Notice the \texttt{\textbackslash{}n} at the end of the command.
 By including a newline character, we are also capable of executing commands in the
 user's terminal.
 Were this newline omitted, the command would only be written to the terminal
@@ -59,4 +59,197 @@ It is not the purpose of this text to provide a complete API documentation, so i
 following I am only going to explain possibilities provided by given components rather
 than showcasing actual, real-life API messages.
 
-\section{Messages component}
+\section{Messages Component}
+
+The framework must allow content creators to communicate their \emph{message} to the user.
+In other words some way must be provided to ``talk'' to users.
+This is the responsibility of the \emph{messages} frontend component, which
+provides a chatbox-like element on the web application the framework can send
+messages to.
+The simplest form of communication it accomodates it the insertion of text
+into the chatbox through API messages.
+Every message has an optional \emph{originator}, which serves signal to the user
+on the purpose of the message.
+These messages are also timestamped.
+
+\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot Typing in the Messages Component}    
+
+A particularly interesting feature of the messages component is that TFW client code
+can queue a bunch of messages for the component to send one by one, separated by
+appropriate pauses in time so that the user is capable of conveniently reading through all
+off them.
+Similarly to a real chat application, some
+``jumping dots'' indicate if the bot is still ``typing''.
+The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
+set by developers according to their specific use cases.
+This creates an experience similar to chatting with someone in real time, as the time
+it takes for each message to be displayed is depending on the lenght of the previous message.
+This illusion is made possible through appropriate \texttt{setTimeout()} calls in
+TypeScript and some elementary math to calculate the proper delays in milliseconds based on
+message lengths:
+
+\[ charactersPerMinute = wordsPerMinute * 5 \]
+\[ charactersPerSeconds = charactersPerMinute / 60 \]
+\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
+\[ timeoutMilliseconds = timeoutSeconds * 1000 \]
+
+\section{IDE Component}\label{idecomponent}
+
+This is the code editor integrated into the frontend of the framework.
+It allows users to select, display and edit files.
+Developers can configure which directory on the file system of the \texttt{solvable}
+container should the editor list files from.
+The editor features the ``Deploy'' button referred to earlier in this paper, which is
+capable of restarting processes that might execute a file visible in the editor.
+
+To implement this IDE%
+\footnote{Integrated development environment}
+I have integrated the open source Monaco editor developed by Microsoft into the
+Angular web application TFW uses as a frontend, and added functionality to it
+that allows the editor to integrate with the framework.
+This involves commnication with an event handler dedicated to this feature,
+which is capable of reading and writing files to disk, while sending and receiving
+editor content from the frontend component.
+The interaction of this event handler and the Monaco editor provides a seamless
+editing expirience, featuring autosave at configurable intervals, code completion,
+automatic code coloring for several programming languages and more.
+
+Perhaps the most ``magical'' feature of this editor is that if any process
+in the Docker container writes a file that is being displayed in the editor,
+the contents of that file are automatially refreshed without any user
+interaction whatsoever.
+Besides that, if a file is created in the directory the editor is configured
+to display, that file is automatially displayed on a new tab in the IDE.
+
+This allows for really interesting demo opportunities.
+Lets say I create a file using the terminal on the frontend by executing the
+command \texttt{touch file.txt}. A new tab on the editor automatically
+appears. If I select it I can confirm that I have successfully created an
+empty file.
+After this let's run a \texttt{while} cycle in the command line which
+peroadically appends some text to \texttt{file.txt}:
+\begin{lstlisting}[captionpos=b,caption={Bash While Cycle Writing to a File Periodically},
+                   language=bash]
+while true
+do
+    echo 'Hey there!' >> file.txt
+    sleep 1
+done
+\end{lstlisting}
+The results speak for themselves:
+\pic{figures/ide_demo.png}{The Editor Demo Involving Automatic File Refreshing}
+As you can see, the file contents are automatially updated as the bash script appends
+to the file.
+This feature is implemented by using the inotify API%
+\footnote{\href{http://man7.org/linux/man-pages/man7/inotify.7.html}
+{http://man7.org/linux/man-pages/man7/inotify.7.html}}
+provided by the Linux kernel to monitor file system events involving the directory listed by
+the editor. The event handler of the editor hooks callbacks to said events which notify the
+Tutorial Framework to reload the list of files in the directory and the contents of
+the selected files.
+The code making this feature possible is reused several times in the framework
+for interesting purposes such as monitoring the logs of processes.
+
+The editor also allows content creators to completely control it using API messages.
+This involves selecting, reading and writing files as well as changing the
+selected directory.
+These features allow content creators to ``guide'' a user through code bases
+for example, where in each step of a tutorial a file is opened and explained
+through messages sent to the chatbox of the messages component.
+
+\section{Terminal Component}
+
+This is a full-fledged xterm terminal emulator running right in the user's browser.
+I have partially derived this from my early work I have briefly mentioned
+in~\ref{intro:emergence}, but it required lot's of new functionality to work the way
+it does today.
+The difference this time is that I had to integrate it into an Angular component.
+I have used the awesome xterm.js%
+\footnote{\href{https://xtermjs.org}{https://xtermjs.org}}
+terminal emulator to do so.
+
+This component has a tiny server process which is managed by a TFW event handler.
+This small server is responsible for spawning bash sessions and
+unix pseudoterminals (or \texttt{pty}s) in the \texttt{solvable} Docker
+container.
+It is also responsible for connecting the master end of the \texttt{pty} to the
+emulator running in your browser and the slave end to the bash session it has
+spawned.
+This way users are able to work in the shell displayed on the frontend just like
+they would on their home machines, which allows for great tutorials
+explaining topics that involve the usage of a shell.
+Note that this allows coverion an extremely wide variety of topics using TFW: from compiling
+shared libraries for development, using cryptographic FUSE filesystems
+for enhanced privacy%
+\footnote{\href{https://github.com/rfjakob/gocryptfs}{https://github.com/rfjakob/gocryptfs}},
+to automating cloud infrastructure using Ansible%
+\footnote{\href{https://www.ansible.com}{https://www.ansible.com}}.
+
+This component exposes several functions through TFW message APIs, such as injecting commands
+to the terminal, reading command history and registering callbacks that are invoked when
+certain command are executed by the user.
+
+\pic{figures/terminal.png}{The Frontend Terminal of TFW Running top}
+
+\section{Console Component}
+
+This component is a simple textbox that can be used to display anything to the user,
+from the results of unit test to the output of processes.
+The console has no event handler: it is a purely frontend component which exposes a simple
+API through TFW messages to write and read it's contents.
+
+It works great when combined with the process management capabilities of the framework:
+if configured to do so it can display the output of processes like webservers in real time.
+When using this next to the frontend editor of the framework, it allows for a development
+experience similar to working in an IDE on your laptop.
+
+\pic{figures/console_and_editor.png}{The Console Displaying Live Process Logs Next to the TFW Code Editor}
+
+\section{Process Management}
+
+The framework includes an event handler capable of managing processes running inside
+the \texttt{solvable} Docker container.
+It's capabilities include starting, stopping and restarting processes.
+It is also capable of emitting the standard out or standard error logs of processes
+(by broadcasting TFW messages).
+This component can be iteracted with using TFW API messages.
+
+The Tutorial Framework uses supervisor%
+\footnote{\href{http://supervisord.org}{http://supervisord.org}}
+to run multiple processes inside a Docker container
+(whereas usually Docker containers run a single process only).
+This is going to be explained further in a later chapter.
+All this is possible through using the xmlrpc%
+\footnote{\href{https://docs.python.org/3/library/xmlrpc.html}{https://docs.python.org/3/library/xmlrpc.html}}
+API exposed by supervisor, which allows the framework to iteract with processes it controls.
+
+It is also possible to find out what files does a process write logs to.
+Combining this with the inotify capabilities of TFW explained
+briefly in~\ref{idecomponent}, it becomes possible to implement live log monitoring
+in the framework.
+The features involving the use of inotify were among the most difficult ones implement,
+since the sheer number of impossible to debug issues that such
+a complex system could come with.
+
+I'll briefly explain such a bug, which I've found to be immersely exciting.
+During the initial development of this feature all processes inside the
+\texttt{solvable} Docker container were writing their logs to files
+in the FHS%
+\footnote{The File Hierarchy Standard is a standard maintained by the Linux foundation,
+defining a common directory structure for compliant systems. See
+\href{https://wiki.linuxfoundation.org/lsb/fhs}{https://wiki.linuxfoundation.org/lsb/fhs}}
+location \texttt{/tmp/}.
+All logs coming from the container itself were also logged to this location.
+This had caused an infinite recursion: when a process would write to \texttt{/tmp/}
+inotify would invoke a process that would also log to that location causing the kernel to
+emit more inotify events, which in turn would cause more and more new proesses to spawn
+and write to \texttt{/tmp/}, causing the whole procedure to repeat again and again.
+This continued until my machine would start to run out of memory and stat swapping
+pages to disk%
+\footnote{When a modern operating system runs out of physical RAM, it is going to swap
+virtual memory pages to disk so it can continue to operate --- slowly}
+like crazy, causing the whole system to spiral downwards
+in a spectacular fashion until the whole thing managed to crash.
+It was an event of such chaotic beauty, that I often fondly recall it to this day.
+Of course it would take me several hours to identify the exact causes behind this
+fascinating phenomenon, but those were \emph{very} fun hours.

content/introduction.tex

@@ -93,7 +93,7 @@ In the future we also plan on supporting the use of virtual machines to implemen
 challenges, which could further increase this fexibility by addig the possiblity to do
 things like exercises involving the use of Docker or Windows based challenges.
 
-\section{Emergence}
+\section{Emergence}\label{intro:emergence}
 
 While working as a content creator I have stumbled into the idea of automating the completion
 of challenges for QA\footnote{Quality Assurrance} and demo purposes%
@@ -162,11 +162,29 @@ the file by decrypting it and then checking if the configuration file is valid o
 After all this we would still have to offer \emph{relevant} and helpful assistance if
 something went wrong.
 
+Another scenario we've visioned was the following: Imagine a code editor on the
+right which contains the authentication logic of a website.
+On the left, imagine that the website which the code in the editor
+implements is present. Note that the website is completely real: it is an actual, functional web
+application users can interact with (i.e.\ navigate through the pages, register or log in).
+The code editor has a button titled ``Deploy'' on it.
+If the user changes the source code of the application and clicks this button, the application
+should restart itself with the new code.
+Let's say that the user comments out the part that authenticates a user.
+In this case the application should let anyone log in dummy credentials.
+Meanwhile a console could show the output of the webserver.
+For example if the source code the user tried to deploy was invalid, the framework
+should report the exact exception raised while running the application.
+
+\pic{figures/webapp_and_editor.png}{The Code Editor and Web Application Example In TFW}
+
 Even if we did all this, we would still need a way to integrate this whole thing into 
 a web based frontend with a file editor, terminal, chat window and stuff like that.
 Turns out that today all this can be done by writing a few hundred lines of Python
 code which uses the Tutorial Framework.
 
+\pic{figures/webapp_and_editor_err.png}{Invalid Code and Deployment Failure with Process Output}
+
 \subsection{Project Requirements}\label{requirements}
 
 Based on this it is now more or less possible to define requirements for the project.