mrtoth/baseimage-tutorial-framework
1
0
Fork 0
mirror of https://github.com/avatao-content/baseimage-tutorial-framework synced 2024-11-25 04:41:31 +00:00
Code Issues Releases Wiki Activity

Update documentation

Kristóf Tóth 2019-12-19 13:47:09 +01:00
parent 6d22a9470f
commit d064512b6c
7 changed files with 248 additions and 49 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

54
Building-everything-manually.md

@ -1,33 +1,55 @@
In case you would like to avoid running anything locally and/or pulling images from [Docker hub](https://hub.docker.com/u/avatao), let's see how we can build the images completely manually. If you are here by accident, you should just use our [tfw.sh](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/tfw.sh-script) script instead. Even if you follow the recommended development process and use our [tfw.sh](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/tfw.sh-script) script, it can still be quite useful to understand the build process (i.e. in case you ever need to troubleshoot something).
Let's see how to build/run a TFW based challenge manually.
### The relationship between images
First of all, let's take a look at the `FROM` statements in our `Dockerfile`s:
### Connection among the images
First of all, let's take a look at the `FROM` statements in each `Dockerfile`:
```dockerfile ```dockerfile
Frontend: # frontend-tutorial-framework:
FROM avatao/debian:buster FROM avatao/debian:buster
Base image: # baseimage-tutorial-framework (multistage build):
FROM avatao/frontend-tutorial-framework:chausie-20190915 as frontend FROM avatao/frontend-tutorial-framework:chausie-20190915 as frontend
FROM avatao/debian:buster FROM avatao/debian:buster
Test challenge: ############## Implementation details above this line ##############
# test-tutorial-framework (or your challenge):
FROM avatao/baseimage-tutorial-framework FROM avatao/baseimage-tutorial-framework
``` ```
As you can see, we use multistage building in the base image, so we should move forward in this exact order. The baseimage-tutorial-framework uses a multistage Docker build to include the necessary build artifacts of the frontend (static `html` and `js` files, which will be served by nginx).
Every baseimage release has a compatible frontend version bundled into it using this mechanism.
### Building the images Your challenge should use the TFW baseimage (`avatao/baseimage-tutorial-framework`), which is available on Docker Hub.
Make sure you have every [dependency](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/Setting-up-a-development-environment) installed, then clone the [baseimage](https://github.com/avatao-content/baseimage-tutorial-framework), [frontend](https://github.com/avatao-content/frontend-tutorial-framework) and [test](https://github.com/avatao-content/test-tutorial-framework) repositories next to each other. Finally, enter these commands to build all of them: Notice the tags in the image names (i.e. `chausie-20190915`).
The `chausie` part before the dash is the major version and the timestamp `20190915` is the minor version.
Updating challenges to new minor versions should be safe and easy, as we try our best not to introduce any breaking changes during the lifetime of a given major release.
A new major indicates that some public API has changed, and the upgrade procedure should be done manually with extra care.
Note that due to limited resources, we only support the latest major version at all times (i.e. no bugfixes and/or new features for older releases).
The latest major `chausie` was intended for long term support and should be the version we use in the foreseeable future (no further API breakages are planned for now).
The example `Dockerfile` in test-tutorial-framework does not specify a tag (therefore it defaults to `latest`).
You should edit this file and pin the latest TFW tag in your `Dockerfile` to keep your builds nice & reproducible.
It should look similar to this:
```dockerfile
# test-tutorial-framework (or your challenge):
FROM avatao/baseimage-tutorial-framework:chausie-20190915
```
You can find available releases [here](https://hub.docker.com/r/avatao/baseimage-tutorial-framework/tags).
Generally speaking you should always try to use the newest one for the best experience.
### Building your challenge
It is practical to bootstrap the creation of your challenge from the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repository, which serves as a nice template project.
Clone this repository and enter these commands to build it:
```bash ```bash
cd frontend-tutorial-framework cd test-tutorial-framework
docker build -t avatao/frontend-tutorial-framework:chausie-20190915 .
cd ../baseimage-tutorial-framework
docker build -t avatao/baseimage-tutorial-framework:chausie-20190915 .
cd ../test-tutorial-framework
docker build -t test-tutorial-framework -f solvable/Dockerfile . docker build -t test-tutorial-framework -f solvable/Dockerfile .
``` ```
Finally, you can run the container using this command:
After all of these, you can start the container any time with this command:
```bash ```bash
docker run --rm -p 8888:8888 -e AVATAO_SECRET=secret test-tutorial-framework docker run --rm -p 8888:8888 -e AVATAO_SECRET=secret test-tutorial-framework
``` ```
By default TFW exercises only expose port `8888` for HTTP and use a reverse proxy inside the container to allow running multiple services.
The `AVATAO_SECRET` envvar is used by a custom PAM module in the `avatao/debian:buster` image to inject secrets at runtime for authentication (for challenges with SSH for instance).

16
Introduction.md

@ -1,11 +1,19 @@
Efficiency in learning can be remarkably improved by presenting the content in a visual context. With **TFW**, you can demonstrate problems in an interactive manner without losing flexibility thanks to the framework's messaging system which allows you to communicate with any kind of process that is able to run in a container. The *Tutorial Framework* (or, for short *TFW*) allows you to create interactive learning environments running inside containers. These environments are capable of automatically guiding users through a set of topics by allowing them to interact with real software through a simple web browser. Users can attack web services, write some code to fix the vulnerabilities they’ve just exploited and use a terminal to deploy their changes. The framework gives you tools to track user progress, run test cases to check their solutions and even offer context dependent help based on the results.
Efficiency in learning can be remarkably improved by presenting the content in a visual context. Using TFW, you can demonstrate problems in an interactive and flexible manner thanks to the framework's messaging system which allows you to communicate with any kind of process that is able to run in a container.
You should also read our [blog post](https://blog.avatao.com/Containerizing-IT-Sec/) on the framework for a short & gentle introduction.
## What kind of features does TFW provide? ## What kind of features does TFW provide?
[[images/frontend.png]] [[images/frontend.png]]
*** ***
In the upper-left part of the page you can see the chat bot, right next to it is a custom web service, in the rightmost part is the web IDE and the remaining one is a fully-featured terminal environment with an optional console tab where you can insert arbitrary text. To manipulate the components, you can send commands on the backend side to the *TFW server*, where event handlers process your request and instruct the frontend through a *WebSocket* connection. In the upper-left part of the page you can see our chat bot, which you can use to send messages to users.
Right next to it is a custom web application (which can be replaced with an app of your choosing).
In the rightmost part is our web editor (based on Monaco), which users can use to edit text files (source code for instance).
The component on the bottom-left is a fully-featured xterm terminal emulator (based on xterm.js), with an optional console tab which you can use to display live process logs or arbitrary text.
To use components, you can send commands to the *TFW server*, which will route it to event handlers capable of processing your request, instructing the frontend to display stuff through a *WebSocket* connection.
### Sequence of events ### Sequence of events
Everything that has a beginning has an end. Generally, you can define the steps of an exercise using a [finite-state machine](https://en.wikipedia.org/wiki/Finite-state_machine). This means that certain events can trigger a transition from one state to another. For instance, you want the user to login in your web service, and if it was successful, you would trigger the next step by notifying the *TFW server*, which will forward your message to the *FSM* instance that evaluates your request, and finally the state change happens, and additional logic is executed to prepare the environment for the next step until the last. Everything that has a beginning has an end. Generally, you can define the steps of an exercise using a [finite-state machine](https://en.wikipedia.org/wiki/Finite-state_machine). This means that certain events can trigger a transition from one state to another. For instance, you want the user to login in your web service. If it was successful, you would trigger the next step by sending a message to the*TFW server*, which will route your message to the *FSM* instance. This instance attempts to execute your request, and finally the state change happens (if it was valid on the model you've defined). You can also execute additional logic on various FSM events (on_enter, on_exit, etc.)
In the third chapter, you will find detailed description regarding each component. In the third chapter, you will find a more detailed description of each component.

27
Key,-intent-and-scope.md

@ -1,10 +1,23 @@
TFW messages have these three essential properties that are described below. You have to add these to the *JSON object* either explicitly, or implicitly by calling the `send_message` method on a server connector. In case of the latter, you only have to provide the key, the default values for the *intent* property is `"control"` and `"zmq"` for *scope*. TFW JSON messages have some essential fields that are described below.
### Key It is important to understand that the `key` field is mandatory for all TFW messages and you will be using it all the time, but the `intent` and `scope` fields are more of implementation details, only used in rare edge cases internally in the framework and are not intended for general usage by challenge developers.
It indicates the domain of the message along with a command like `message.send`. You can define your own ones and send it to the TFW server, but nothing will happen until you create an event handler, which listens to that exact key.
### Intent ### Key (mandatory)
This property has to accepted values: `"control"` and `"event"`. The former used to instruct an event handler to perform an action and the latter emits information about something's state. This distinction is really important, because the lack of intent could lead to infinite recursion in event handlers. This field specifies the topic of a message and is used for addressing.
Parties interested in certain topics can subscribe to a set of keys to receive messages containing them.
It has no default value and it is mandatory (every valid TFW message will have a `key`).
It usually indicates the domain of the message along with a command like `message.send`. You can define your own keys and send messages with them to the TFW server, but nothing will happen until you create event handlers listening to those keys.
Event handlers use prefix matching on the value of the `key` field to determine if they should receive a given message (i.e. an event handler subscribed to `ide` will receive both `ide.write` and `ide.read` messages).
### Scope ### Intent (optional)
This one specifies the direction of the message: `"zmq"`, `"websocket"`, `"broadcast"`. Their names are quite talkative, the first one forwards the message to the event handlers, the second one goes to the frontend and the third one is send to both directions. This field has two valid values: `"control"` and `"event"`.
The former is used to instruct an event handler to perform an action and the latter emits information about something (RPC and event advertisement).
This distinction is really important, because the lack of intent could lead to infinite recursion in some event handlers.
Generally there is no need to explicitly set this field unless you are creating complex event handlers with several available commands and emitted events (most of the time you don't really have to care this exists).
The default value is `"control"` (messages without an `intent` key are treated as control messages).
### Scope (optional)
The TFW server uses this field for routing and it specifies the direction of messages.
It has three valid values: `"zmq"`, `"websocket"`, `"broadcast"`.
Their names are quite descriptive, the first one forwards the message to the event handlers, the second one addresses the frontend, and the third one broadcasts the message to both directions.
Note that messages without a `scope` field are treated as `zmq`, therefore there is usually no need to specify this explicitly.

9
Main-repositories.md

@ -1,11 +1,14 @@
## [baseimage-tutorial-framework](https://github.com/avatao-content/baseimage-tutorial-framework) ## [baseimage-tutorial-framework](https://github.com/avatao-content/baseimage-tutorial-framework)
This is the beating heart of TFW – the Docker baseimage containing the internals of the framework. It's most important element is the robust messaging system that makes it straightforward to implement interactive exercises. This is the beating heart of TFW – the Docker baseimage containing the internals of the framework. It's most important element is the robust messaging system based on ZeroMQ, that makes it straightforward to implement interactive exercises.
## [frontend-tutorial-framework](https://github.com/avatao-content/frontend-tutorial-framework) ## [frontend-tutorial-framework](https://github.com/avatao-content/frontend-tutorial-framework)
This is the Angular frontend of TFW. The main exposed features are our pre-implemented components based on the `src/app/services/websocket.service.ts` service. This is the Angular frontend of TFW.
This service provides an *RxJS* based communication API to the framework backend (TFW server and event handlers). Another useful features are a bunch of pre-designed layouts and dynamic switching between them. Mostly it's just a thin displays layer and complex logic should be implemented in the container instead.
Apart from configuring it, you shouldn't have to mess with it directly, as our built-in event handlers (maintained in the baseimage repository) will do the heavy lifting for you (i.e. instrumenting the frontend with the proper messages and whatnot).
The main exposed features are our pre-implemented components based on the `src/app/services/websocket.service.ts` service.
This service implements a communication API based on RxJS and WebSockets to communicate with the framework backend (TFW server and event handlers). More useful features include a bunch of pre-designed layouts and dynamic switching between them.
# [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) # [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework)

2
Networking.md

@ -1,5 +1,7 @@
The foundation of the whole framework is the messaging system connecting the frontend with the backend. Frontend components use *websockets* to connect to the TFW server, to which you can hook several [event handlers](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/Event-handlers) defining how to handle specific messages. The foundation of the whole framework is the messaging system connecting the frontend with the backend. Frontend components use *websockets* to connect to the TFW server, to which you can hook several [event handlers](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/Event-handlers) defining how to handle specific messages.
Note that a nice overview of the framework networking is included in [this blog post](https://blog.avatao.com/Containerizing-IT-Sec/), which you should read.
## ZMQ ## ZMQ
Event handlers connect to the TFW server using **ZMQ**. They receive messages on their *SUB(scribe)* sockets, which are connected to the *PUB(lish)* socket of the server. Event handlers reply on their *PUSH* socket, then their messages are received on the *PULL* socket of the server. Event handlers connect to the TFW server using **ZMQ**. They receive messages on their *SUB(scribe)* sockets, which are connected to the *PUB(lish)* socket of the server. Event handlers reply on their *PUSH* socket, then their messages are received on the *PULL* socket of the server.

172
Setting-up-a-development-environment.md

@ -1,7 +1,8 @@
To be able to use the framework locally, you may need to install some of the packages listed below depending on your intentions. The framework supports and encourages local challenge development, but you may need to install some of the dependencies listed below (especially if you intend to use our helper scripts instead of building images manually).
### Running challenges ### Building and running challenges
You can run/develop TFW exercises locally with only utilizing **Docker** containers, however, if you don't prefer [building everything manually](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/Building-everything-manually), then it is advisable to use our [tfw.sh](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/tfw.sh-script) script that requires: You can build and run TFW exercises locally with just Docker, however, it is advisable to use our [tfw.sh](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/tfw.sh-script) script for ease of development, unless you prefer [building everything manually](https://github.com/avatao-content/baseimage-tutorial-framework/wiki/Building-everything-manually).
On top of Docker, our `tfw.sh` script requires the following:
* GNU coreutils * GNU coreutils
* bash * bash
* git * git
@ -13,12 +14,167 @@ pip install git+ssh://git@github.com/avatao-content/baseimage-tutorial-framework
# HTTPS: # HTTPS:
pip install git+https://github.com/avatao-content/baseimage-tutorial-framework.git pip install git+https://github.com/avatao-content/baseimage-tutorial-framework.git
``` ```
Please note that we use our Docker baseimage to distribute the TFW codebase (available on (Docker Hub)[https://hub.docker.com/r/avatao/baseimage-tutorial-framework]) and this pip package only serves to make development more comfortable (thereby it is not published on PyPi intentionally).
### Editing the base image ### Developing the baseimage itself
If you install everything mentioned above and clone the base image [repository](https://github.com/avatao-content/baseimage-tutorial-framework), you are covered. Make sure that the repository shares the same parent folder with your challenge, so *tfw.sh* can detect it. If you install everything mentioned above and clone the [baseimage repository](https://github.com/avatao-content/baseimage-tutorial-framework) next to an instance of a challenge repository (preferably [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework)), you should be good to go (the `tfw.sh` script supports developing TFW itself).
Make sure that your clone of the baseimage repository shares the same parent folder with your challenge (test-tfw), so `tfw.sh` can find it.
You might also want to install the TFW pip package in editable mode (`pip install -e <baseimage_path>`), so that changes can instantly propagate to your development environment.
### Editing the frontend ### Developing the TFW frontend
Normally, the frontend is just statically served by *nginx* in the container, but if you wish to modify it, you will need these locally: Normally, the frontend is only present in images as a bunch of build artifacts served by *nginx*, but if you wish to develop the frontend itself, you may need the following dependencies locally:
* yarn * yarn
* Angular CLI * Angular CLI
After that, clone this [repository](https://github.com/avatao-content/frontend-tutorial-framework), and run `yarn install` in the root directory. To begin working on the frontend you should:
1. Clone the [frontend repository](https://github.com/avatao-content/frontend-tutorial-framework)
2. Install it's npm dependencies using `yarn install`
3. Run it using `yarn start`
3. Your frontend should be available on `localhost:4200`
Please note that you will also need to run a TFW challenge locally for the frontend to connect to (preferably test-tfw using `tfw.sh`).
TFW based challenges expose the port `8888`. Our frontend repository comes with a `proxy.conf.json` which will automatically forward the appropriate HTTP requests from the frontend to this port when using `yarn start` or `ng serve`.
It is also possible to treat the frontend as a Docker image for development if you'd rather avoid installing messy JS packages, but expect slower build times in this case.
### Inside test-tutorial-framework
The repository of a tutorial-framework based challenge is quite similar to a regular challenge.
The project root should look something like this:
```
your_repo
├── solvable
│ └── [TFW based Docker image]
├── controller
│ └── [solution checking]
├── metadata
│ └── [challenge descriptions, writeups, etc.]
└── config.yml
```
The only notable difference is that the `solvable` Docker image is a child of our baseimage: `solvable/Dockerfile` begins with `FROM avatao/baseimage-tutorial-framework`.
From now on we are going to focus on the `solvable` image.
#### Basics of a TFW based challenge
Let us take a closer look on `solvable`:
```
solvable
├── Dockerfile
├── nginx webserver configurations
├── supervisor process manager (init replacement)
└── src example source code
```
Note that our baseimage *requires* the `nginx` and `supervisor` folders to be in these **exact** locations and to be used as described below.
This is a contract your image **must** comply.
The `src` directory contains a simple example of using TFW.
#### nginx
All TFW based challenges expose a single port defined in the `TFW_PUBLIC_PORT` envvar which is set to `8888` by default.
This means that in order to run multiple HTTP services we must use a reverse proxy.
Any `.conf` files in `solvable/nginx/` will be automatically included in the nginx configuration.
In case you want to serve a website or service you must proxy it through `TFW_PUBLIC_PORT`.
This is really easy: just create a config file in `solvable/nginx/` similar to this one:
```
location /yoururl {
proxy_pass http://127.0.0.1:3333;
}
```
After this you can access the service running on port `3333` at `http://localhost:8888/yoururl`
It is very important to understand that from now on your application must behave well behind a reverse proxy.
What this means is all `href`s must point the proxied paths (e.g. links should refer to `/yoururl/register` instead of `/register`) on your HTML pages.
You can learn about configuring nginx in [this](https://www.digitalocean.com/community/tutorials/understanding-the-nginx-configuration-file-structure-and-configuration-contexts) handy little tutorial.
#### supervisor
In most Docker conainers there is a single process running (with `PID 1` inside the PID namespace).
When working with TFW you can run as many processes as you want to by using supervisord.
Any `.conf` files in the `solvable/supervisor/` directory will be included in the supervisor configuration.
The programs you define this way are easy to manage (starting/stopping/restarting) using the `supervisorctl` command line tool or our built-in event handler.
You can even configure your processes to start with the container by including `autostart=true` in your configuration file.
To run your own webservice for instance you need to create a config file in `solvable/supervisor/` similar to this one:
```
[program:yourprogram]
user=user
directory=/home/user/example/
command=python3 server.py
autostart=true
```
This starts the `/home/user/example/server.py` script using `python3` after your container entered the running state (because of `autostart=true`, supervisor does not start programs by default).
You can learn more about configuring supervisor [here](http://supervisord.org/configuration.html).
#### src
This folder contains an template setup of our pre-written event handlers and example FSMs.
Note that this is not a part of the framework by any means, these are just simple examples.
```
solvable/src
├── event_handler_main.py event handlers implemented in python
├── frontend_config.yaml YAML configuration of the fontend (parsed by event_handler_main)
├── pipe_io_main.py spawns POSIX named pipes capable of communicating with the TFW server
├── webservice/ an example webserver
├── test_fsm.py example FSM in python
└── test_fsm.yml example FSM in yaml
```
`event_handler_main.py` contains example usage of our pre-defined event handlers written in Python3.
As you can see they run in a separate process (set up in `solvable/supervisor/event_handler_main.conf`).
These event handlers could be implemented in any language that has ZMQ bindings.
Note that you don't have to use all our event handlers.
Should you want to avoid using a feature, you can just delete the appropriate event handler from `event_handler_main.py`.
`pipe_io_main.py` runs proxy event handlers capable of creating and communicating over POSIX named pipes.
These allow you to send/receive messages to/from the TFW server using the `open()`, `write()` and `read()` system calls instead of ZMQ sockets.
For example you could send a message with the command `echo [some JSON] > /tmp/tfw_send` in a terminal.
Or you could use a readline function in any programming language to receive a message.
It also monitors the `/run/tfw/` directory for the creation of new pipes.
`test_fsm.yml` and `test_fsm.py` are the implementations of the same FSM in YAML and Python to provide you examples of creating your own machine.
It is genarally a good idea to separate these files from the rest of the stuff in `solvable`, so it is a good practice to create an `src` directory.
#### FSM
A good state machine is the backbone of a good TFW challenge.
There are two ways to define a state machine:
- Using a YAML configuration file
- Implementing it in Python
The first option allows you to handle FSM callbacks and custom logic in any programming language (not just Python) and is generally really easy to work with (you can execute arbitrary shell commands on events).
You should choose this method unless you have good reason not to.
This involves creating your YAML file (see `test_fsm.yml` for an example) and parsing it using our `YamlFSM` class (see `event_handler_main.py` for an example).
The second option allows you to implement your FSM in Python, using the transitions library.
To do this just subclass our `FSMBase` class or use our `LinearFSM` class for simple machines (see `test_fsm.py` for an example).
In your FSM you can define callbacks for states and transitions.
State callbacks:
- `on_enter`
- `on_exit`
Transition callbacks:
- `before`
- `after`
In your YAML file you can use these in the state and transition objects as keys, then add a shell command to run as a value (again, see `test_fsm.yml` for examples).
It is also possible to add preconditions to transitions.
This is done by adding a `predicates` key with a list of shell commands to run.
If you do this, the transition will only succeed if the return code of all predicates was `0` (as per unix convention for success).
Our `YamlFSM` implementation also supports jinja2 templates inside the `YAML` config file (examples in `test_fsm.yml`).

13
tfw.sh-script.md

@ -1,9 +1,4 @@
This script is located in the `hack` folder of the challenge, and its primary purpose is to orchestrate building and running TFW-based exercises. It operates with the following commands: This script is located in the `hack` folder of challenges (or test-tfw), and its primary purpose is to simplify building and running TFW-based exercises.
Essentially it removes the mental overhead of having to do something like "`cd` to a specific directory and execute `docker build ... && docker run ...`" all the time.
| Command | Description | It also facilitiates workflows for developers of TFW itself, such as releasing a new version of the frontend and/or baseimage.
| ------------ | -------------------------------------------------------- | Run it without any arguments to print usage info.
| `start` | build & run TFW challenge |
| `run` | run TFW challenge |
| `buildtfw` | build TFW baseimage |
| `build` | build TFW baseimage and challenge |
| `releasetfw` | tag TFW baseimage and push to upstream (for maintainers) |