Hotfix supervisor infinite log recursion if no LogMonitor

Console

.pylintrc

@@ -2,4 +2,4 @@
 
 ignored-modules = zmq
 max-line-length = 150
-disable = missing-docstring, too-few-public-methods
+disable = missing-docstring, too-few-public-methods, invalid-name

Dockerfile

@@ -11,7 +11,8 @@ RUN curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -
         libzmq5                                \
         nginx                                  \
         gettext-base                                                                                   &&\
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/*                                                                        &&\
+    ln -sf /bin/bash /bin/sh
 
 COPY requirements.txt /tmp
 RUN pip3 install -r /tmp/requirements.txt
@@ -26,7 +27,7 @@ ENV TFW_PUBLIC_PORT=8888          \
 
 EXPOSE ${TFW_PUBLIC_PORT}
 
-ENV PYTHONPATH="/usr/local/lib/"                                           \
+ENV PYTHONPATH="/usr/local/lib"                                            \
     TFW_SUPERVISOR_HTTP_URI="http://localhost:${TFW_SUPERVISOR_HTTP_PORT}" \
     TFW_SUPERVISORD_CONF="/etc/supervisor/supervisord.conf"                \
     TFW_SUPERVISORD_COMPONENTS="/etc/supervisor/conf"                      \
@@ -40,7 +41,7 @@ ENV PYTHONPATH="/usr/local/lib/"                                           \
     PROMPT_COMMAND="history -a"
 
 COPY bashrc /tmp
-RUN echo "export HISTFILE=${TFW_HISTFILE}\n" >> /tmp/bashrc &&\
+RUN echo "export HISTFILE=${TFW_HISTFILE}" >> /tmp/bashrc &&\
     cat /tmp/bashrc >> /home/${AVATAO_USER}/.bashrc
 
 COPY supervisor/supervisord.conf ${TFW_SUPERVISORD_CONF}
@@ -49,7 +50,7 @@ COPY nginx/default.conf ${TFW_NGINX_DEFAULT}
 COPY nginx/components/ ${TFW_NGINX_COMPONENTS}
 COPY lib LICENSE ${TFW_LIB_DIR}
 
-RUN for dir in "${TFW_LIB_DIR}" "/etc/nginx" "/etc/supervisor"; do \
+RUN for dir in "${TFW_LIB_DIR}"/{tfw,tao,envvars} "/etc/nginx" "/etc/supervisor"; do \
         chown -R root:root "$dir" && chmod -R 700 "$dir";                               \
     done
 

README.md

@@ -3,11 +3,11 @@
 This is the beating heart of TFW – the Docker baseimage containing the internals of the framework.
 
 Every tutorial-framework based challenge has a `solvable` Docker image based on this one: their `Dockerfile`s begin with `FROM eu.gcr.io/avatao-challengestore/tutorial-framework`.
-Note that TFW is not avaliable on Docker Hub due to legal reasons and is only accessible through local builds (dont't worry, we've got you covered with build scripts in the `test-tutorial-framework` repo).
+Note that TFW is not avaliable on Docker Hub due to legal reasons and is only accessible through local builds (don't worry, we've got you covered with build scripts in the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repo).
 
 This document explains the general concepts of TFW and should be the first thing you read before getting started with development.
 
-For more on building and running you should consult the `test-tutorial-framework` repo.
+For more on building and running you should check the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repo.
 
 ## The framework
 
@@ -22,9 +22,9 @@ Frontend components use websockets to connect to the TFW server, to which you ca
 
 ### Event handlers
 
-Imagine event handlers as callbacks that are invoked when TFW receives a specific type of message. For instance you could send a message to the framework when the user does something of note.
+Imagine event handlers as callbacks that are invoked when TFW receives a specific type of message. For instance, you could send a message to the framework when the user does something of note.
 
-This allows you to define actions triggered on the backend when the user presses a button on the frontend, moves the cursor to a specific area or anything like that.
+Event handler allow you to define actions triggered on the backend when the user presses a button on the frontend or moves the cursor to a specific area, etc.
 
 Event handlers use ZeroMQ to connect to the framework. Due to this they are as loosely-coupled as possible: usually they are running in separate processes and only communicate with TFW through ZMQ.
 
@@ -38,9 +38,9 @@ Inside Avatao this means that any of the content teams can use the framework wit
 Another unique feature of the framework is the FSM – finite state machine – representing the state of your challenge.
 This allows you to track users progressing with the tasks you've defined for them to complete.
 
-For instance you could represent whether the user managed to create a malicious user with a state called `user_registered` and subscribe callbacks to events regarding that state (like entering or leaving).
+For instance, you could represent whether the user managed to create a malicious user with a state called `user_registered` and subscribe callbacks to events regarding that state (like entering or leaving).
 
-You could create challenges that can be completed in several different ways: imagine a state called `challenge_complete`, which represents when the challenge is completed. Several series of actions (triggers) could lead to this state.
+You could create challenges that can be completed in several different ways: imagine a state called `challenge_complete`, which indicates if the challenge is completed. Several series of actions (triggers) could lead to this state.
 
 This enables you to guide your users through the experience you've envisioned with your tutorial.
 We can provide a whole new level of interactivity in our challenges because we know what the user is doing.
@@ -48,12 +48,12 @@ This includes context-dependent hints and the automatic typing of commands to a
 
 ### Frontend
 
-Note that our frontend implementation is written in Angular. It is maintained and documented in the `frontend-tutorial-framework` repository.
+Note that our frontend implementation is written in Angular. It is maintained and documented in the [frontend-tutorial-framework](https://github.com/avatao-content/frontend-tutorial-framework) repository.
 
 ### Messaging format
 
 The framework uses JSON messages internally and in exposed APIs as well.
-These messages must comply some rules.
+These messages must comply with some rules.
 Don't worry, we are not too fond of rules around these parts.
 
 The TFW message format:
@@ -80,4 +80,4 @@ The TFW message format:
 
 Most of the components you need have docstrings included (hang on tight, this is work in progress) – refer to them for usage info.
 
-To get started you should take a look at the `test-tutorial-framework` repository, which serves as an example project as well.
+To get started you should take a look at the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repository, which serves as an example project as well.

lib/tfw/components/__init__.py

@@ -3,7 +3,8 @@
 
 from .directory_monitoring_event_handler import DirectoryMonitoringEventHandler
 from .process_managing_event_handler import ProcessManagingEventHandler
-from .terminado_event_handler import TerminadoEventHandler
+from .terminal_event_handler import TerminalEventHandler
 from .ide_event_handler import IdeEventHandler
 from .history_monitor import HistoryMonitor, BashMonitor, GDBMonitor
 from .terminal_commands import TerminalCommands
+from .log_monitoring_event_handler import LogMonitoringEventHandler

lib/tfw/components/directory_monitor.py

@@ -15,10 +15,12 @@ LOG = logging.getLogger(__name__)
 
 
 class DirectoryMonitor(ObserverMixin):
-    def __init__(self, directory):
+    def __init__(self, directories):
         ObserverMixin.__init__(self)
         self.eventhandler = IdeReloadWatchdogEventHandler()
+        for directory in directories:
             self.observer.schedule(self.eventhandler, directory, recursive=True)
+
         self.pause, self.resume = self.eventhandler.pause, self.eventhandler.resume
 
     @property

lib/tfw/components/ide_event_handler.py

@@ -98,22 +98,36 @@ class IdeEventHandler(EventHandlerBase, MonitorManagerMixin):
     By default all files in the directory specified in __init__ are displayed
     on the fontend. Note that this is a stateful component.
 
+    When any file in the selected directory changes they are automatically refreshed
+    on the frontend (this is done by listening to inotify events).
+
     This EventHandler accepts messages that have a data["command"] key specifying
     a command to be executed.
     The API of each command is documented in their respective handlers.
     """
-    def __init__(self, key, directory, allowed_directories, selected_file=None, exclude=None):
+    def __init__(self, key, directory, allowed_directories, selected_file=None, exclude=None,
+                 additional_watched_directories=None):
         """
         :param key: the key this instance should listen to
         :param directory: working directory which the EventHandler should serve files from
         :param allowed_directories: list of directories that can be switched to using the selectdir command
         :param selected_file: file that is selected by default
         :param exclude: list of filenames that should not appear between files (for *.o, *.pyc, etc.)
+        :param additional_watched_directories: refresh the selected file when files change in these directories
+                                               (the working directory is watched by default, this is useful for
+                                               symlinks and such)
         """
         super().__init__(key)
+        try:
             self.filemanager = FileManager(allowed_directories=allowed_directories, working_directory=directory,
                                            selected_file=selected_file, exclude=exclude)
-        MonitorManagerMixin.__init__(self, DirectoryMonitor, self.filemanager.workdir)
+        except IndexError:
+            raise EnvironmentError(f'No file(s) in IdeEventHandler working_directory "{directory}"!')
+
+        self.watched_directories = [self.filemanager.workdir]
+        if additional_watched_directories:
+            self.watched_directories.extend(additional_watched_directories)
+        MonitorManagerMixin.__init__(self, DirectoryMonitor, self.watched_directories)
 
         self.commands = {'read':      self.read,
                          'write':     self.write,

lib/tfw/components/log_monitor.py

@@ -0,0 +1,52 @@
+# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
+# All Rights Reserved. See LICENSE file for details.
+
+import logging
+from os.path import dirname
+
+from watchdog.events import PatternMatchingEventHandler as PatternMatchingWatchdogEventHandler
+
+from tfw.networking.event_handlers import ServerUplinkConnector
+from tfw.decorators import RateLimiter
+from tfw.mixins import ObserverMixin, SupervisorLogMixin
+
+
+class LogMonitor(ObserverMixin):
+    def __init__(self, process_name, log_tail=0):
+        self.prevent_log_recursion()
+        ObserverMixin.__init__(self)
+        event_handler = SendLogWatchdogEventHandler(process_name, log_tail=log_tail)
+        self.observer.schedule(
+            event_handler,
+            event_handler.path
+        )
+
+    @staticmethod
+    def prevent_log_recursion():
+        # This is done to prevent inotify event logs triggering themselves (infinite log recursion)
+        logging.getLogger('watchdog.observers.inotify_buffer').propagate = False
+
+
+class SendLogWatchdogEventHandler(PatternMatchingWatchdogEventHandler, SupervisorLogMixin):
+    def __init__(self, process_name, log_tail=0):
+        self.acquire_own_supervisor_instance()  # This thread-localises the xmlrpc client
+        self.process_name = process_name
+        self.procinfo = self.supervisor.getProcessInfo(self.process_name)
+        super().__init__([self.procinfo['stdout_logfile'], self.procinfo['stderr_logfile']])
+        self.uplink = ServerUplinkConnector()
+        self.log_tail = log_tail
+
+    @property
+    def path(self):
+        return dirname(self.procinfo['stdout_logfile'])
+
+    @RateLimiter(rate_per_second=5)
+    def on_modified(self, event):
+        self.uplink.send({
+            'key': 'processlog',
+            'data': {
+                'command': 'new_log',
+                'stdout': self.read_stdout(self.process_name, tail=self.log_tail),
+                'stderr': self.read_stderr(self.process_name, tail=self.log_tail)
+            }
+        })

lib/tfw/components/log_monitoring_event_handler.py

@@ -0,0 +1,37 @@
+# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
+# All Rights Reserved. See LICENSE file for details.
+
+
+from tfw import EventHandlerBase
+from tfw.mixins import MonitorManagerMixin
+from tfw.config.logs import logging
+from .log_monitor import LogMonitor
+
+LOG = logging.getLogger(__name__)
+
+
+class LogMonitoringEventHandler(EventHandlerBase, MonitorManagerMixin):
+    def __init__(self, key, process_name, log_tail=0):
+        super().__init__(key)
+        self.process_name = process_name
+        self.log_tail = log_tail
+        MonitorManagerMixin.__init__(self, LogMonitor, self.process_name, self.log_tail)
+
+        self.command_handlers = {
+            'process_name': self.handle_process_name,
+            'log_tail':     self.handle_log_tail
+        }
+
+    def handle_event(self, message):
+        try:
+            data = message['data']
+            self.command_handlers[data['command']](data)
+            self.reload_monitor()
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_process_name(self, data):
+        self.set_monitor_args(data['process_name'], self.log_tail)
+
+    def handle_log_tail(self, data):
+        self.set_monitor_args(self.process_name, data['log_tail'])

lib/tfw/components/process_managing_event_handler.py

@@ -4,19 +4,18 @@
 from xmlrpc.client import Fault as SupervisorFault
 
 from tfw import EventHandlerBase
-from tfw.mixins import SupervisorMixin
+from tfw.mixins import SupervisorMixin, SupervisorLogMixin
 from tfw.config.logs import logging
 from .directory_monitor import with_monitor_paused
 
 LOG = logging.getLogger(__name__)
 
 
-class ProcessManager(SupervisorMixin):
+class ProcessManager(SupervisorMixin, SupervisorLogMixin):
     def __init__(self):
         self.commands = {'start':      self.start_process,
                          'stop':       self.stop_process,
-                         'restart': self.restart_process,
+                         'restart':    self.restart_process}
-                         'readlog': self.read_log}
 
     def __call__(self, command, process_name):
         return self.commands[command](process_name)
@@ -35,21 +34,24 @@ class ProcessManagingEventHandler(EventHandlerBase):
     Commands available: start, stop, restart, readlog
     (the names are as self-documenting as it gets)
     """
-    def __init__(self, key, dirmonitor=None):
+    def __init__(self, key, dirmonitor=None, log_tail=0):
         super().__init__(key)
         self.key = key
         self.monitor = dirmonitor
         self.processmanager = ProcessManager()
+        self.log_tail = log_tail
 
     @with_monitor_paused
     def handle_event(self, message):
         try:
             data = message['data']
+            try:
                 self.processmanager(data['command'], data['process_name'])
+            except SupervisorFault as fault:
+                message['data']['error'] = fault.faultString
+            finally:
+                message['data']['stdout'] = self.processmanager.read_stdout(data['process_name'], self.log_tail)
+                message['data']['stderr'] = self.processmanager.read_stderr(data['process_name'], self.log_tail)
             return message
         except KeyError:
             LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
-        except SupervisorFault as fault:
-            message['data']['error'] = fault.faultString
-            message['data']['log'] = self.processmanager.read_log(message['data']['process_name'])
-            return message

lib/tfw/components/terminal_commands.py

@@ -3,6 +3,7 @@
 
 from abc import ABC
 from re import match
+from shlex import split
 
 from tfw.config.logs import logging
 
@@ -57,10 +58,10 @@ class TerminalCommands(ABC):
         return match(self._command_method_regex, string)
 
     def callback(self, history):
-        parts = history[-1].split()
+        parts = split(history[-1])
         command = parts[0]
         if command in self.command_implemetations.keys():
             try:
                 self.command_implemetations[command](*parts[1:])
-            except IndexError:
+            except Exception: # pylint: disable=broad-except
-                LOG.debug('Command "%s" failed!', command)
+                LOG.exception('Command "%s" failed:', command)

lib/tfw/components/terminal_event_handler.py

@@ -10,7 +10,7 @@ from .terminado_mini_server import TerminadoMiniServer
 LOG = logging.getLogger(__name__)
 
 
-class TerminadoEventHandler(EventHandlerBase):
+class TerminalEventHandler(EventHandlerBase):
     """
     Event handler responsible for managing terminal sessions for frontend xterm
     sessions to connect to. You need to instanciate this in order for frontend

lib/tfw/linear_fsm.py

@@ -5,14 +5,18 @@ from .fsm_base import FSMBase
 
 
 class LinearFSM(FSMBase):
+    # pylint: disable=anomalous-backslash-in-string
     """
     This is a state machine for challenges with linear progression, consisting of
-    a number of steps specified in the constructor. It automatically sets up a single
+    a number of steps specified in the constructor. It automatically sets up 2
-    action between states as such:
+    actions (triggers) between states as such:
-    0 ==step_1==> 1 ==step_2==> 2 ==step_3==> 3 ... and so on
+    (0) --  step_1  --> (1) --  step_2  --> (2) --  step_3  --> (3) ... and so on
+        \-step_next-/       \-step_next-/       \-step_next-/
     """
     def __init__(self, number_of_steps):
         self.states = list(map(str, range(number_of_steps)))
-        self.transitions = [{'trigger': f'step_{int(index)+1}', 'source': index, 'dest': str(int(index)+1)}
+        self.transitions = []
-                            for index in self.states[:-1]]
+        for index in self.states[:-1]:
+            self.transitions.append({'trigger': f'step_{int(index)+1}', 'source': index, 'dest': str(int(index)+1)})
+            self.transitions.append({'trigger': 'step_next', 'source': index, 'dest': str(int(index)+1)})
         super(LinearFSM, self).__init__()

lib/tfw/mixins/__init__.py

@@ -1,7 +1,7 @@
 # Copyright (C) 2018 Avatao.com Innovative Learning Kft.
 # All Rights Reserved. See LICENSE file for details.
 
-from .supervisor_mixin import SupervisorMixin
+from .supervisor_mixin import SupervisorMixin, SupervisorLogMixin
 from .callback_mixin import CallbackMixin
 from .observer_mixin import ObserverMixin
 from .monitor_manager_mixin import MonitorManagerMixin

lib/tfw/mixins/monitor_manager_mixin.py

@@ -7,21 +7,24 @@ LOG = logging.getLogger(__name__)
 
 
 class MonitorManagerMixin:
-    def __init__(self, monitor_type, directory):
+    def __init__(self, monitor_type, *monitor_args):
         self._monitor_type = monitor_type
         self._monitor = None
-        self._monitored_directory = directory
+        self.monitor_args = monitor_args
         self.reload_monitor()
 
     @property
     def monitor(self):
         return self._monitor
 
+    def set_monitor_args(self, *monitor_args):
+        self.monitor_args = monitor_args
+
     def reload_monitor(self):
         if self._monitor:
             try:
                 self._monitor.stop()
             except KeyError:
                 LOG.debug('Working directory was removed – ignoring...')
-        self._monitor = self._monitor_type(self._monitored_directory)
+        self._monitor = self._monitor_type(*self.monitor_args)
         self._monitor.watch()   # This runs on a separate thread

lib/tfw/mixins/supervisor_mixin.py

@@ -9,9 +9,21 @@ from os import remove
 from tfw.config import TFWENV
 
 
-class SupervisorMixin:
+def get_supervisor_instance():
-    supervisor = xmlrpc.client.ServerProxy(TFWENV.SUPERVISOR_HTTP_URI).supervisor
+    return xmlrpc.client.ServerProxy(TFWENV.SUPERVISOR_HTTP_URI).supervisor
 
+
+class SupervisorBaseMixin:
+    supervisor = get_supervisor_instance()
+
+    def acquire_own_supervisor_instance(self):
+        """
+        Give this instance non-static, local xmlrpc client
+        """
+        self.supervisor = get_supervisor_instance()
+
+
+class SupervisorMixin(SupervisorBaseMixin):
     def stop_process(self, process_name):
         with suppress(SupervisorFault):
             self.supervisor.stopProcess(process_name)
@@ -19,12 +31,20 @@ class SupervisorMixin:
     def start_process(self, process_name):
         self.supervisor.startProcess(process_name)
 
-    def read_log(self, process_name):
-        logs = self.supervisor.readProcessStderrLog(process_name, 0, 0)
-        remove(self.supervisor.getProcessInfo(process_name)['stderr_logfile'])
-        self.supervisor.clearProcessLogs(process_name)
-        return logs
-
     def restart_process(self, process_name):
         self.stop_process(process_name)
         self.start_process(process_name)
+
+
+class SupervisorLogMixin(SupervisorBaseMixin):
+    def read_stdout(self, process_name, tail=0):
+        return self.supervisor.readProcessStdoutLog(process_name, -tail, 0)
+
+    def read_stderr(self, process_name, tail=0):
+        return self.supervisor.readProcessStderrLog(process_name, -tail, 0)
+
+    def clear_logs(self, process_name):
+        for logfile in ('stdout_logfile', 'stderr_logfile'):
+            with suppress(FileNotFoundError):
+                remove(self.supervisor.getProcessInfo(process_name)[logfile])
+        self.supervisor.clearProcessLogs(process_name)

lib/tfw/networking/event_handlers/server_connector.py

@@ -35,6 +35,9 @@ class ServerUplinkConnector(ZMQConnectorBase):
     def send_to_eventhandler(self, message):
         """
         Send a message to an event handler.
+        This envelopes the desired message in the 'data' field of the message to
+        TFWServer, which will mirror it to event handlers.
+
         :param message: JSON message you want to send
         :param message['key']: key of event handler you want to address
         """
@@ -46,6 +49,7 @@ class ServerUplinkConnector(ZMQConnectorBase):
     def send(self, message):
         """
         Send a message to the TFW server
+
         :param message: JSON message you want to send
         """
         self._zmq_push_socket.send_multipart(serialize_tfw_msg(message))

lib/tfw/networking/serialization.py

@@ -29,10 +29,16 @@ def validate_message(message):
 
 
 def serialize_tfw_msg(message):
+    """
+    Create TFW multipart data from message dict
+    """
     return _serialize_all(message['key'], message)
 
 
 def deserialize_tfw_msg(*args):
+    """
+    Return message from TFW multipart data
+    """
     return _deserialize_all(*args)[1]
 
 
@@ -45,12 +51,20 @@ def _deserialize_all(*args):
 
 
 def _serialize_single(data):
+    """
+    Return input as bytes
+    (serialize input if it is JSON)
+    """
     if not isinstance(data, str):
         data = json.dumps(data)
     return _encode_if_needed(data)
 
 
 def _deserialize_single(data):
+    """
+    Try parsing input as JSON, return it as
+    string if parsing fails.
+    """
     try:
         return json.loads(data)
     except ValueError:
@@ -58,12 +72,20 @@ def _deserialize_single(data):
 
 
 def _encode_if_needed(value):
+    """
+    Return input as bytes
+    (encode if input is string)
+    """
     if isinstance(value, str):
         value = value.encode('utf-8')
     return value
 
 
 def _decode_if_needed(value):
+    """
+    Return input as string
+    (decode if input is bytes)
+    """
     if isinstance(value, (bytes, bytearray)):
         value = value.decode('utf-8')
     return value

nginx/default.conf

@@ -4,6 +4,7 @@ server {
     proxy_connect_timeout 7d;
     proxy_send_timeout 7d;
     proxy_read_timeout 7d;
+    absolute_redirect off;
 
     location = /ws {
         proxy_pass http://127.0.0.1:${TFW_WEB_PORT};

setup.py

@@ -0,0 +1,23 @@
+from os.path import dirname, realpath, join
+
+from setuptools import setup, find_packages
+
+
+here = dirname(realpath(__file__))
+
+with open(join(here, 'VERSION'), 'r') as ifile:
+    version = ifile.read().strip('\n')
+with open(join(here, 'requirements.txt'), 'r') as ifile:
+    requirements = ifile.read().splitlines()
+
+setup(name='tfw',
+      version=version,
+      description='Avatao tutorial-framework',
+      url='https://github.com/avatao-content/baseimage-tutorial-framework',
+      author='Avatao.com Innovative Learning Kft.',
+      author_email='support@avatao.com',
+      license='custom',
+      packages=find_packages('lib'),
+      package_dir={'': 'lib'},
+      install_requires=requirements,
+      zip_safe=False)

supervisor/supervisord.conf

@@ -1,6 +1,7 @@
 [supervisord]
 user=root
-logfile = /tmp/supervisord.log
+logfile=/dev/null
+logfile_maxbytes=0
 loglevel = debug
 pidfile = /tmp/supervisord.pid