Continue writing chapter 'A Tour of TFW'

This commit is contained in:
Kristóf Tóth 2018-12-01 20:24:24 +01:00
parent 791650a8a7
commit c126fe8fff
8 changed files with 214 additions and 3 deletions

View File

@ -23,7 +23,7 @@ For example, to inject a command into the terminal one would use a message like
} }
} }
\end{lstlisting} \end{lstlisting}
Notice the \texttt{``\\n''} at the end of the command. Notice the \texttt{\textbackslash{}n} at the end of the command.
By including a newline character, we are also capable of executing commands in the By including a newline character, we are also capable of executing commands in the
user's terminal. user's terminal.
Were this newline omitted, the command would only be written to the terminal Were this newline omitted, the command would only be written to the terminal
@ -59,4 +59,197 @@ It is not the purpose of this text to provide a complete API documentation, so i
following I am only going to explain possibilities provided by given components rather following I am only going to explain possibilities provided by given components rather
than showcasing actual, real-life API messages. than showcasing actual, real-life API messages.
\section{Messages component} \section{Messages Component}
The framework must allow content creators to communicate their \emph{message} to the user.
In other words some way must be provided to ``talk'' to users.
This is the responsibility of the \emph{messages} frontend component, which
provides a chatbox-like element on the web application the framework can send
messages to.
The simplest form of communication it accomodates it the insertion of text
into the chatbox through API messages.
Every message has an optional \emph{originator}, which serves signal to the user
on the purpose of the message.
These messages are also timestamped.
\pic[width=.5\textwidth]{figures/chatbot.png}{The avataobot Typing in the Messages Component}
A particularly interesting feature of the messages component is that TFW client code
can queue a bunch of messages for the component to send one by one, separated by
appropriate pauses in time so that the user is capable of conveniently reading through all
off them.
Similarly to a real chat application, some
``jumping dots'' indicate if the bot is still ``typing''.
The timing of pauses and messages is based on the \emph{WPM} --- or Words Per Minute ---
set by developers according to their specific use cases.
This creates an experience similar to chatting with someone in real time, as the time
it takes for each message to be displayed is depending on the lenght of the previous message.
This illusion is made possible through appropriate \texttt{setTimeout()} calls in
TypeScript and some elementary math to calculate the proper delays in milliseconds based on
message lengths:
\[ charactersPerMinute = wordsPerMinute * 5 \]
\[ charactersPerSeconds = charactersPerMinute / 60 \]
\[ timeoutSeconds = lastMessageLength / charactersPerSeconds \]
\[ timeoutMilliseconds = timeoutSeconds * 1000 \]
\section{IDE Component}\label{idecomponent}
This is the code editor integrated into the frontend of the framework.
It allows users to select, display and edit files.
Developers can configure which directory on the file system of the \texttt{solvable}
container should the editor list files from.
The editor features the ``Deploy'' button referred to earlier in this paper, which is
capable of restarting processes that might execute a file visible in the editor.
To implement this IDE%
\footnote{Integrated development environment}
I have integrated the open source Monaco editor developed by Microsoft into the
Angular web application TFW uses as a frontend, and added functionality to it
that allows the editor to integrate with the framework.
This involves commnication with an event handler dedicated to this feature,
which is capable of reading and writing files to disk, while sending and receiving
editor content from the frontend component.
The interaction of this event handler and the Monaco editor provides a seamless
editing expirience, featuring autosave at configurable intervals, code completion,
automatic code coloring for several programming languages and more.
Perhaps the most ``magical'' feature of this editor is that if any process
in the Docker container writes a file that is being displayed in the editor,
the contents of that file are automatially refreshed without any user
interaction whatsoever.
Besides that, if a file is created in the directory the editor is configured
to display, that file is automatially displayed on a new tab in the IDE.
This allows for really interesting demo opportunities.
Lets say I create a file using the terminal on the frontend by executing the
command \texttt{touch file.txt}. A new tab on the editor automatically
appears. If I select it I can confirm that I have successfully created an
empty file.
After this let's run a \texttt{while} cycle in the command line which
peroadically appends some text to \texttt{file.txt}:
\begin{lstlisting}[captionpos=b,caption={Bash While Cycle Writing to a File Periodically},
language=bash]
while true
do
echo 'Hey there!' >> file.txt
sleep 1
done
\end{lstlisting}
The results speak for themselves:
\pic{figures/ide_demo.png}{The Editor Demo Involving Automatic File Refreshing}
As you can see, the file contents are automatially updated as the bash script appends
to the file.
This feature is implemented by using the inotify API%
\footnote{\href{http://man7.org/linux/man-pages/man7/inotify.7.html}
{http://man7.org/linux/man-pages/man7/inotify.7.html}}
provided by the Linux kernel to monitor file system events involving the directory listed by
the editor. The event handler of the editor hooks callbacks to said events which notify the
Tutorial Framework to reload the list of files in the directory and the contents of
the selected files.
The code making this feature possible is reused several times in the framework
for interesting purposes such as monitoring the logs of processes.
The editor also allows content creators to completely control it using API messages.
This involves selecting, reading and writing files as well as changing the
selected directory.
These features allow content creators to ``guide'' a user through code bases
for example, where in each step of a tutorial a file is opened and explained
through messages sent to the chatbox of the messages component.
\section{Terminal Component}
This is a full-fledged xterm terminal emulator running right in the user's browser.
I have partially derived this from my early work I have briefly mentioned
in~\ref{intro:emergence}, but it required lot's of new functionality to work the way
it does today.
The difference this time is that I had to integrate it into an Angular component.
I have used the awesome xterm.js%
\footnote{\href{https://xtermjs.org}{https://xtermjs.org}}
terminal emulator to do so.
This component has a tiny server process which is managed by a TFW event handler.
This small server is responsible for spawning bash sessions and
unix pseudoterminals (or \texttt{pty}s) in the \texttt{solvable} Docker
container.
It is also responsible for connecting the master end of the \texttt{pty} to the
emulator running in your browser and the slave end to the bash session it has
spawned.
This way users are able to work in the shell displayed on the frontend just like
they would on their home machines, which allows for great tutorials
explaining topics that involve the usage of a shell.
Note that this allows coverion an extremely wide variety of topics using TFW: from compiling
shared libraries for development, using cryptographic FUSE filesystems
for enhanced privacy%
\footnote{\href{https://github.com/rfjakob/gocryptfs}{https://github.com/rfjakob/gocryptfs}},
to automating cloud infrastructure using Ansible%
\footnote{\href{https://www.ansible.com}{https://www.ansible.com}}.
This component exposes several functions through TFW message APIs, such as injecting commands
to the terminal, reading command history and registering callbacks that are invoked when
certain command are executed by the user.
\pic{figures/terminal.png}{The Frontend Terminal of TFW Running top}
\section{Console Component}
This component is a simple textbox that can be used to display anything to the user,
from the results of unit test to the output of processes.
The console has no event handler: it is a purely frontend component which exposes a simple
API through TFW messages to write and read it's contents.
It works great when combined with the process management capabilities of the framework:
if configured to do so it can display the output of processes like webservers in real time.
When using this next to the frontend editor of the framework, it allows for a development
experience similar to working in an IDE on your laptop.
\pic{figures/console_and_editor.png}{The Console Displaying Live Process Logs Next to the TFW Code Editor}
\section{Process Management}
The framework includes an event handler capable of managing processes running inside
the \texttt{solvable} Docker container.
It's capabilities include starting, stopping and restarting processes.
It is also capable of emitting the standard out or standard error logs of processes
(by broadcasting TFW messages).
This component can be iteracted with using TFW API messages.
The Tutorial Framework uses supervisor%
\footnote{\href{http://supervisord.org}{http://supervisord.org}}
to run multiple processes inside a Docker container
(whereas usually Docker containers run a single process only).
This is going to be explained further in a later chapter.
All this is possible through using the xmlrpc%
\footnote{\href{https://docs.python.org/3/library/xmlrpc.html}{https://docs.python.org/3/library/xmlrpc.html}}
API exposed by supervisor, which allows the framework to iteract with processes it controls.
It is also possible to find out what files does a process write logs to.
Combining this with the inotify capabilities of TFW explained
briefly in~\ref{idecomponent}, it becomes possible to implement live log monitoring
in the framework.
The features involving the use of inotify were among the most difficult ones implement,
since the sheer number of impossible to debug issues that such
a complex system could come with.
I'll briefly explain such a bug, which I've found to be immersely exciting.
During the initial development of this feature all processes inside the
\texttt{solvable} Docker container were writing their logs to files
in the FHS%
\footnote{The File Hierarchy Standard is a standard maintained by the Linux foundation,
defining a common directory structure for compliant systems. See
\href{https://wiki.linuxfoundation.org/lsb/fhs}{https://wiki.linuxfoundation.org/lsb/fhs}}
location \texttt{/tmp/}.
All logs coming from the container itself were also logged to this location.
This had caused an infinite recursion: when a process would write to \texttt{/tmp/}
inotify would invoke a process that would also log to that location causing the kernel to
emit more inotify events, which in turn would cause more and more new proesses to spawn
and write to \texttt{/tmp/}, causing the whole procedure to repeat again and again.
This continued until my machine would start to run out of memory and stat swapping
pages to disk%
\footnote{When a modern operating system runs out of physical RAM, it is going to swap
virtual memory pages to disk so it can continue to operate --- slowly}
like crazy, causing the whole system to spiral downwards
in a spectacular fashion until the whole thing managed to crash.
It was an event of such chaotic beauty, that I often fondly recall it to this day.
Of course it would take me several hours to identify the exact causes behind this
fascinating phenomenon, but those were \emph{very} fun hours.

View File

@ -93,7 +93,7 @@ In the future we also plan on supporting the use of virtual machines to implemen
challenges, which could further increase this fexibility by addig the possiblity to do challenges, which could further increase this fexibility by addig the possiblity to do
things like exercises involving the use of Docker or Windows based challenges. things like exercises involving the use of Docker or Windows based challenges.
\section{Emergence} \section{Emergence}\label{intro:emergence}
While working as a content creator I have stumbled into the idea of automating the completion While working as a content creator I have stumbled into the idea of automating the completion
of challenges for QA\footnote{Quality Assurrance} and demo purposes% of challenges for QA\footnote{Quality Assurrance} and demo purposes%
@ -162,11 +162,29 @@ the file by decrypting it and then checking if the configuration file is valid o
After all this we would still have to offer \emph{relevant} and helpful assistance if After all this we would still have to offer \emph{relevant} and helpful assistance if
something went wrong. something went wrong.
Another scenario we've visioned was the following: Imagine a code editor on the
right which contains the authentication logic of a website.
On the left, imagine that the website which the code in the editor
implements is present. Note that the website is completely real: it is an actual, functional web
application users can interact with (i.e.\ navigate through the pages, register or log in).
The code editor has a button titled ``Deploy'' on it.
If the user changes the source code of the application and clicks this button, the application
should restart itself with the new code.
Let's say that the user comments out the part that authenticates a user.
In this case the application should let anyone log in dummy credentials.
Meanwhile a console could show the output of the webserver.
For example if the source code the user tried to deploy was invalid, the framework
should report the exact exception raised while running the application.
\pic{figures/webapp_and_editor.png}{The Code Editor and Web Application Example In TFW}
Even if we did all this, we would still need a way to integrate this whole thing into Even if we did all this, we would still need a way to integrate this whole thing into
a web based frontend with a file editor, terminal, chat window and stuff like that. a web based frontend with a file editor, terminal, chat window and stuff like that.
Turns out that today all this can be done by writing a few hundred lines of Python Turns out that today all this can be done by writing a few hundred lines of Python
code which uses the Tutorial Framework. code which uses the Tutorial Framework.
\pic{figures/webapp_and_editor_err.png}{Invalid Code and Deployment Failure with Process Output}
\subsection{Project Requirements}\label{requirements} \subsection{Project Requirements}\label{requirements}
Based on this it is now more or less possible to define requirements for the project. Based on this it is now more or less possible to define requirements for the project.

BIN
figures/chatbot.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 75 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 108 KiB

BIN
figures/ide_demo.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 36 KiB

BIN
figures/terminal.png Normal file

Binary file not shown.

After

Width:  |  Height:  |  Size: 301 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 384 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 448 KiB