baseimage-tutorial-framework/README.md

baseimage-tutorial-framework

This is the beating heart of TFW – the Docker baseimage containing the internals of the framework.

Every tutorial-framework based challenge has a solvable Docker image based on this one: their Dockerfiles begin with FROM eu.gcr.io/avatao-challengestore/tutorial-framework. Note that TFW is not avaliable on Docker Hub due to legal reasons and is only accessible through local builds (don't worry, we've got you covered with build scripts in the test-tutorial-framework repo).

This document explains the general concepts of TFW and should be the first thing you read before getting started with development.

For more on building and running you should check the test-tutorial-framework repo.

The framework

The goal of the tutorial-framework is to help content developers in creating interactive tutorials for the Avatao platform.

To make this possible TFW implements a robust messaging system and provides several pre-written components built upon it, such as a file editor and a terminal (both running in your browser).

The foundation of the whole framework is the messaging system connecting the frontend with the backend. Frontend components use websockets to connect to the TFW server, to which you can hook several event handlers defining how to handle specific messages.

Networking details

Event handlers connect to the TFW server using ZMQ. They receive messages on their SUB(scribe) sockets, which are connected to the PUB(lish) socket of the server. Event handlers reply on their PUSH socket, then their messages are received on the PULL socket of the server.

The TFW server is basically just a fancy proxy. It's behaviour is quite simple: it proxies every message received from the fontend to the event handlers and vice versa.

The server is also capable of "mirroring" messages back to their source. This is useful for communication between event handlers or frontend components (event handler to event handler or frontend component to frontend component communication).

Components can also broadcast messages (broadcasted messages are received both by event handlers and the frontend as well).

Event handlers

Imagine event handlers as callbacks that are invoked when TFW receives a specific type of message. For instance, you could send a message to the framework when the user does something of note.

Event handler allow you to define actions triggered on the backend when the user presses a button on the frontend or moves the cursor to a specific area, etc.

Event handlers use ZeroMQ to connect to the framework. Due to this they are as loosely-coupled as possible: usually they are running in separate processes and only communicate with TFW through ZMQ.

Our pre-made event handlers are written in Python3, but you can write event handlers in any language that has ZeroMQ bindings (this means virtually any language).

This makes the framework really flexible: you can demonstrate the concepts you want to in any language while using the same set of tools provided by TFW. Inside Avatao this means that any of the content teams can use the framework with ease.

To implement an event handler in Python3 you should subclass the EventHandlerBase or FSMAwareEventHandler class in tfw.event_handler_base (the first provides a minimal working EventHandler, the second allows you to execute code on FSM events).

FSM

Another unique feature of the framework is the FSM – finite state machine – representing the state of your challenge. This allows you to track users progressing with the tasks you've defined for them to complete.

For instance, you could represent whether the user managed to create a malicious user with a state called user_registered and subscribe callbacks to events regarding that state (like entering or leaving).

You could create challenges that can be completed in several different ways: imagine a state called challenge_complete, which indicates if the challenge is completed. Several series of actions (triggers) could lead to this state.

This enables you to guide your users through the experience you've envisioned with your tutorial. We can provide a whole new level of interactivity in our challenges because we know what the user is doing. This includes context-dependent hints and the automatic typing of commands to a terminal.

Frontend

Note that our frontend implementation is written in Angular. It is maintained and documented in the frontend-tutorial-framework repository.

Messaging format

The framework uses JSON messages internally and in exposed APIs as well. These messages must comply with some rules. Don't worry, we are not too fond of rules around these parts.

The TFW message format:

{
    "key: ...some identifier used for addressing...,
    "data":
    {
        ...
        JSON object carrying anything, preferably cats
        ...
    },
    "trigger": ...FSM action...,
    "signature": ...HMAC signature for authenticated messages...,
    "seq": ...sequence number...
}

• The key field is used by TFW for addressing and every message must have one (it can be an empty string though)
• The data object can contain anything you might want to send
• The trigger key is an optional field that triggers an FSM action with that name from the current state (whatever that might be)
• The signature field is present on authenticated messages (such as fsm_updates)
• The seq key is a counter incremented with each proxied message in the TFW server

To mirror messages back to their sources you can use a special messaging format, in which the message to be mirrored is enveloped inside the data field of the outer message:

    "key": "mirror",
    "data":
    {
        ...
        The message you want to mirror (with it's own "key" and "data" fields)
        ...
    }

Broadcasting messages is possible in a similar manner by using "key": "broadcast" in the outer message.

Where to go next

Most of the components you need have docstrings included (hang on tight, this is work in progress) – refer to them for usage info.

In the docs folder you can find our Sphinx-based documentation, which you can build using the hack/tfw.sh script in the test-tutorial-framework repository.

To get started you should take a look at test-tutorial-framework, which serves as an example project as well.

API

APIs exposed by our pre-witten event handlers are documented here.

IdeEventHandler

This event handler is responsible for reading and writing files shown in the frontend code editor.

You can read the content of the currently selected file like so:

{ 
    "key": "ide",
    "data":
    {
        "command": "read"
    }
}

Use the following message to overwrite the content of the currently selected file:

{ 
    "key": "ide",
    "data":
    {
        "command": "write",
        "content": ...string...
    }
}

To select a file use the following message:

{ 
    "key": "ide",
    "data":
    {
        "command": "select",
        "filename": ...string...
    }
}

You can switch to a new working directory using this message (note that the directory must be in allowed_directories):

{ 
    "key": "ide",
    "data":
    {
        "command": "selectdir",
        "directory": ...string...
    }
}

Overwriting the current list of excluded file patterns is possible with this message:

{ 
    "key": "ide",
    "data":
    {
        "command": "exclude",
        "exclude": ...array of strings...
    }
}

TerminalEventHandler

Event handler responsible for running a backend for xterm.js to connect to (frontend terminal backend).

By default callbacks on terminal history are invoked as soon as a command starts to execute in the terminal (they do not wait for the started command to finish, the callback may even run in paralell with the command).

If you want to wait for them and invoke your callbacks after the command has finished, please set the TFW_DELAY_HISTAPPEND envvar to 1. Practically this can be done by appending an export to the user's .bashrc file from your Dockerfile, like so:

RUN echo "export TFW_DELAY_HISTAPPEND=1" >> /home/${AVATAO_USER}/.bashrc

Writing to the terminal:

{ 
    "key": "shell",
    "data":
    {
        "command": "write",
        "value": ...string...
    }
}

You can read terminal command history like so:

{ 
    "key": "shell",
    "data":
    {
        "command": "read",
        "count": ...number...
    }
}

ProcessManagingEventHandler

This event handler is responsible for managing processes controlled by supervisord.

Starting, stopping and restarting supervisor processes can be done using similar messages (where command is start, stop or restart):

{ 
    "key": "processmanager",
    "data":
    {
        "command": ...string...,
        "process_name": ...string...
    }
}

LogMonitoringEventHandler

Event handler emitting real time logs (stdout and stderr) from supervisord processes.

To change which supervisor process is monitored use this message:

{ 
    "key": "logmonitor",
    "data" :
    {
        "command": "process_name",
        "value": ...string...
    }
}

To set the tail length of logs (the monitor will send back the last value characters of the log):

{ 
    "key": "logmonitor",
    "data" :
    {
        "command": "log_tail",
        "value": ...number...
    }
}

FSMManagingEventHandler

This event handler controls the TFW finite state machine (FSM).

To attempt executing a trigger on the FSM use (this will also generate an FSM update message):

{
    "key": "fsm",
    "data" :
    {
        "command": "trigger",
        "value": ...string...
    }
}

To force the broadcasting of an FSM update you can use this message:

{
    "key": "fsm",
    "data" :
    {
        "command": "update"
    }
}

This event handler broadcasts FSM update messages after handling commands in the following format:

{
    "key": "fsm_update",
    "data" :
    {
        "current_state": ...string...,
        "valid_transitions": ...array of {"trigger": ...string...} objects...
    }
}

DirectorySnapshottingEventHandler

Event handler capable of taking and restoring snapshots of directories (saving and restoring directory contens).

You can take a snapshot of the directories with the following message:

{
    "key": "snapshot",
    "data" :
    {
        "command": "take_snapshot"
    }
}

To restore the state of the files in the directories use:

{
    "key": "snapshot",
    "data" :
    {
        "command": "restore_snapshot",
        "value": ...date string (can parse ISO 8601, unix timestamp, etc.)...
    }
}

It is also possible to exclude files that match given patterns (formatted like lines in .gitignore files):

{
    "key": "snapshot",
    "data" :
    {
        "command": "exclude",
        "value": ...list of patterns to exclude from snapshots...
    }
}