Fix message timing on empty queue

Implement inotify based pipe connector

.pyenvrc

@@ -1,4 +0,0 @@
-export PYENV_ROOT="$HOME/.pyenv"
-export PATH="$PYENV_ROOT/bin:$PATH"
-eval "$(pyenv init -)"
-eval "$(pyenv virtualenv-init -)"

.pylintrc

@@ -0,0 +1,12 @@
+[TYPECHECK]
+
+ignored-modules = zmq
+max-line-length = 120
+disable = missing-docstring, too-few-public-methods, invalid-name
+
+[SIMILARITIES]
+
+min-similarity-lines=8
+ignore-comments=yes
+ignore-docstrings=yes
+ignore-imports=yes

COPYING

@@ -0,0 +1,674 @@
+                    GNU GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+                            Preamble
+
+  The GNU General Public License is a free, copyleft license for
+software and other kinds of works.
+
+  The licenses for most software and other practical works are designed
+to take away your freedom to share and change the works.  By contrast,
+the GNU General Public License is intended to guarantee your freedom to
+share and change all versions of a program--to make sure it remains free
+software for all its users.  We, the Free Software Foundation, use the
+GNU General Public License for most of our software; it applies also to
+any other work released this way by its authors.  You can apply it to
+your programs, too.
+
+  When we speak of free software, we are referring to freedom, not
+price.  Our General Public Licenses are designed to make sure that you
+have the freedom to distribute copies of free software (and charge for
+them if you wish), that you receive source code or can get it if you
+want it, that you can change the software or use pieces of it in new
+free programs, and that you know you can do these things.
+
+  To protect your rights, we need to prevent others from denying you
+these rights or asking you to surrender the rights.  Therefore, you have
+certain responsibilities if you distribute copies of the software, or if
+you modify it: responsibilities to respect the freedom of others.
+
+  For example, if you distribute copies of such a program, whether
+gratis or for a fee, you must pass on to the recipients the same
+freedoms that you received.  You must make sure that they, too, receive
+or can get the source code.  And you must show them these terms so they
+know their rights.
+
+  Developers that use the GNU GPL protect your rights with two steps:
+(1) assert copyright on the software, and (2) offer you this License
+giving you legal permission to copy, distribute and/or modify it.
+
+  For the developers' and authors' protection, the GPL clearly explains
+that there is no warranty for this free software.  For both users' and
+authors' sake, the GPL requires that modified versions be marked as
+changed, so that their problems will not be attributed erroneously to
+authors of previous versions.
+
+  Some devices are designed to deny users access to install or run
+modified versions of the software inside them, although the manufacturer
+can do so.  This is fundamentally incompatible with the aim of
+protecting users' freedom to change the software.  The systematic
+pattern of such abuse occurs in the area of products for individuals to
+use, which is precisely where it is most unacceptable.  Therefore, we
+have designed this version of the GPL to prohibit the practice for those
+products.  If such problems arise substantially in other domains, we
+stand ready to extend this provision to those domains in future versions
+of the GPL, as needed to protect the freedom of users.
+
+  Finally, every program is threatened constantly by software patents.
+States should not allow patents to restrict development and use of
+software on general-purpose computers, but in those that do, we wish to
+avoid the special danger that patents applied to a free program could
+make it effectively proprietary.  To prevent this, the GPL assures that
+patents cannot be used to render the program non-free.
+
+  The precise terms and conditions for copying, distribution and
+modification follow.
+
+                       TERMS AND CONDITIONS
+
+  0. Definitions.
+
+  "This License" refers to version 3 of the GNU General Public License.
+
+  "Copyright" also means copyright-like laws that apply to other kinds of
+works, such as semiconductor masks.
+
+  "The Program" refers to any copyrightable work licensed under this
+License.  Each licensee is addressed as "you".  "Licensees" and
+"recipients" may be individuals or organizations.
+
+  To "modify" a work means to copy from or adapt all or part of the work
+in a fashion requiring copyright permission, other than the making of an
+exact copy.  The resulting work is called a "modified version" of the
+earlier work or a work "based on" the earlier work.
+
+  A "covered work" means either the unmodified Program or a work based
+on the Program.
+
+  To "propagate" a work means to do anything with it that, without
+permission, would make you directly or secondarily liable for
+infringement under applicable copyright law, except executing it on a
+computer or modifying a private copy.  Propagation includes copying,
+distribution (with or without modification), making available to the
+public, and in some countries other activities as well.
+
+  To "convey" a work means any kind of propagation that enables other
+parties to make or receive copies.  Mere interaction with a user through
+a computer network, with no transfer of a copy, is not conveying.
+
+  An interactive user interface displays "Appropriate Legal Notices"
+to the extent that it includes a convenient and prominently visible
+feature that (1) displays an appropriate copyright notice, and (2)
+tells the user that there is no warranty for the work (except to the
+extent that warranties are provided), that licensees may convey the
+work under this License, and how to view a copy of this License.  If
+the interface presents a list of user commands or options, such as a
+menu, a prominent item in the list meets this criterion.
+
+  1. Source Code.
+
+  The "source code" for a work means the preferred form of the work
+for making modifications to it.  "Object code" means any non-source
+form of a work.
+
+  A "Standard Interface" means an interface that either is an official
+standard defined by a recognized standards body, or, in the case of
+interfaces specified for a particular programming language, one that
+is widely used among developers working in that language.
+
+  The "System Libraries" of an executable work include anything, other
+than the work as a whole, that (a) is included in the normal form of
+packaging a Major Component, but which is not part of that Major
+Component, and (b) serves only to enable use of the work with that
+Major Component, or to implement a Standard Interface for which an
+implementation is available to the public in source code form.  A
+"Major Component", in this context, means a major essential component
+(kernel, window system, and so on) of the specific operating system
+(if any) on which the executable work runs, or a compiler used to
+produce the work, or an object code interpreter used to run it.
+
+  The "Corresponding Source" for a work in object code form means all
+the source code needed to generate, install, and (for an executable
+work) run the object code and to modify the work, including scripts to
+control those activities.  However, it does not include the work's
+System Libraries, or general-purpose tools or generally available free
+programs which are used unmodified in performing those activities but
+which are not part of the work.  For example, Corresponding Source
+includes interface definition files associated with source files for
+the work, and the source code for shared libraries and dynamically
+linked subprograms that the work is specifically designed to require,
+such as by intimate data communication or control flow between those
+subprograms and other parts of the work.
+
+  The Corresponding Source need not include anything that users
+can regenerate automatically from other parts of the Corresponding
+Source.
+
+  The Corresponding Source for a work in source code form is that
+same work.
+
+  2. Basic Permissions.
+
+  All rights granted under this License are granted for the term of
+copyright on the Program, and are irrevocable provided the stated
+conditions are met.  This License explicitly affirms your unlimited
+permission to run the unmodified Program.  The output from running a
+covered work is covered by this License only if the output, given its
+content, constitutes a covered work.  This License acknowledges your
+rights of fair use or other equivalent, as provided by copyright law.
+
+  You may make, run and propagate covered works that you do not
+convey, without conditions so long as your license otherwise remains
+in force.  You may convey covered works to others for the sole purpose
+of having them make modifications exclusively for you, or provide you
+with facilities for running those works, provided that you comply with
+the terms of this License in conveying all material for which you do
+not control copyright.  Those thus making or running the covered works
+for you must do so exclusively on your behalf, under your direction
+and control, on terms that prohibit them from making any copies of
+your copyrighted material outside their relationship with you.
+
+  Conveying under any other circumstances is permitted solely under
+the conditions stated below.  Sublicensing is not allowed; section 10
+makes it unnecessary.
+
+  3. Protecting Users' Legal Rights From Anti-Circumvention Law.
+
+  No covered work shall be deemed part of an effective technological
+measure under any applicable law fulfilling obligations under article
+11 of the WIPO copyright treaty adopted on 20 December 1996, or
+similar laws prohibiting or restricting circumvention of such
+measures.
+
+  When you convey a covered work, you waive any legal power to forbid
+circumvention of technological measures to the extent such circumvention
+is effected by exercising rights under this License with respect to
+the covered work, and you disclaim any intention to limit operation or
+modification of the work as a means of enforcing, against the work's
+users, your or third parties' legal rights to forbid circumvention of
+technological measures.
+
+  4. Conveying Verbatim Copies.
+
+  You may convey verbatim copies of the Program's source code as you
+receive it, in any medium, provided that you conspicuously and
+appropriately publish on each copy an appropriate copyright notice;
+keep intact all notices stating that this License and any
+non-permissive terms added in accord with section 7 apply to the code;
+keep intact all notices of the absence of any warranty; and give all
+recipients a copy of this License along with the Program.
+
+  You may charge any price or no price for each copy that you convey,
+and you may offer support or warranty protection for a fee.
+
+  5. Conveying Modified Source Versions.
+
+  You may convey a work based on the Program, or the modifications to
+produce it from the Program, in the form of source code under the
+terms of section 4, provided that you also meet all of these conditions:
+
+    a) The work must carry prominent notices stating that you modified
+    it, and giving a relevant date.
+
+    b) The work must carry prominent notices stating that it is
+    released under this License and any conditions added under section
+    7.  This requirement modifies the requirement in section 4 to
+    "keep intact all notices".
+
+    c) You must license the entire work, as a whole, under this
+    License to anyone who comes into possession of a copy.  This
+    License will therefore apply, along with any applicable section 7
+    additional terms, to the whole of the work, and all its parts,
+    regardless of how they are packaged.  This License gives no
+    permission to license the work in any other way, but it does not
+    invalidate such permission if you have separately received it.
+
+    d) If the work has interactive user interfaces, each must display
+    Appropriate Legal Notices; however, if the Program has interactive
+    interfaces that do not display Appropriate Legal Notices, your
+    work need not make them do so.
+
+  A compilation of a covered work with other separate and independent
+works, which are not by their nature extensions of the covered work,
+and which are not combined with it such as to form a larger program,
+in or on a volume of a storage or distribution medium, is called an
+"aggregate" if the compilation and its resulting copyright are not
+used to limit the access or legal rights of the compilation's users
+beyond what the individual works permit.  Inclusion of a covered work
+in an aggregate does not cause this License to apply to the other
+parts of the aggregate.
+
+  6. Conveying Non-Source Forms.
+
+  You may convey a covered work in object code form under the terms
+of sections 4 and 5, provided that you also convey the
+machine-readable Corresponding Source under the terms of this License,
+in one of these ways:
+
+    a) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by the
+    Corresponding Source fixed on a durable physical medium
+    customarily used for software interchange.
+
+    b) Convey the object code in, or embodied in, a physical product
+    (including a physical distribution medium), accompanied by a
+    written offer, valid for at least three years and valid for as
+    long as you offer spare parts or customer support for that product
+    model, to give anyone who possesses the object code either (1) a
+    copy of the Corresponding Source for all the software in the
+    product that is covered by this License, on a durable physical
+    medium customarily used for software interchange, for a price no
+    more than your reasonable cost of physically performing this
+    conveying of source, or (2) access to copy the
+    Corresponding Source from a network server at no charge.
+
+    c) Convey individual copies of the object code with a copy of the
+    written offer to provide the Corresponding Source.  This
+    alternative is allowed only occasionally and noncommercially, and
+    only if you received the object code with such an offer, in accord
+    with subsection 6b.
+
+    d) Convey the object code by offering access from a designated
+    place (gratis or for a charge), and offer equivalent access to the
+    Corresponding Source in the same way through the same place at no
+    further charge.  You need not require recipients to copy the
+    Corresponding Source along with the object code.  If the place to
+    copy the object code is a network server, the Corresponding Source
+    may be on a different server (operated by you or a third party)
+    that supports equivalent copying facilities, provided you maintain
+    clear directions next to the object code saying where to find the
+    Corresponding Source.  Regardless of what server hosts the
+    Corresponding Source, you remain obligated to ensure that it is
+    available for as long as needed to satisfy these requirements.
+
+    e) Convey the object code using peer-to-peer transmission, provided
+    you inform other peers where the object code and Corresponding
+    Source of the work are being offered to the general public at no
+    charge under subsection 6d.
+
+  A separable portion of the object code, whose source code is excluded
+from the Corresponding Source as a System Library, need not be
+included in conveying the object code work.
+
+  A "User Product" is either (1) a "consumer product", which means any
+tangible personal property which is normally used for personal, family,
+or household purposes, or (2) anything designed or sold for incorporation
+into a dwelling.  In determining whether a product is a consumer product,
+doubtful cases shall be resolved in favor of coverage.  For a particular
+product received by a particular user, "normally used" refers to a
+typical or common use of that class of product, regardless of the status
+of the particular user or of the way in which the particular user
+actually uses, or expects or is expected to use, the product.  A product
+is a consumer product regardless of whether the product has substantial
+commercial, industrial or non-consumer uses, unless such uses represent
+the only significant mode of use of the product.
+
+  "Installation Information" for a User Product means any methods,
+procedures, authorization keys, or other information required to install
+and execute modified versions of a covered work in that User Product from
+a modified version of its Corresponding Source.  The information must
+suffice to ensure that the continued functioning of the modified object
+code is in no case prevented or interfered with solely because
+modification has been made.
+
+  If you convey an object code work under this section in, or with, or
+specifically for use in, a User Product, and the conveying occurs as
+part of a transaction in which the right of possession and use of the
+User Product is transferred to the recipient in perpetuity or for a
+fixed term (regardless of how the transaction is characterized), the
+Corresponding Source conveyed under this section must be accompanied
+by the Installation Information.  But this requirement does not apply
+if neither you nor any third party retains the ability to install
+modified object code on the User Product (for example, the work has
+been installed in ROM).
+
+  The requirement to provide Installation Information does not include a
+requirement to continue to provide support service, warranty, or updates
+for a work that has been modified or installed by the recipient, or for
+the User Product in which it has been modified or installed.  Access to a
+network may be denied when the modification itself materially and
+adversely affects the operation of the network or violates the rules and
+protocols for communication across the network.
+
+  Corresponding Source conveyed, and Installation Information provided,
+in accord with this section must be in a format that is publicly
+documented (and with an implementation available to the public in
+source code form), and must require no special password or key for
+unpacking, reading or copying.
+
+  7. Additional Terms.
+
+  "Additional permissions" are terms that supplement the terms of this
+License by making exceptions from one or more of its conditions.
+Additional permissions that are applicable to the entire Program shall
+be treated as though they were included in this License, to the extent
+that they are valid under applicable law.  If additional permissions
+apply only to part of the Program, that part may be used separately
+under those permissions, but the entire Program remains governed by
+this License without regard to the additional permissions.
+
+  When you convey a copy of a covered work, you may at your option
+remove any additional permissions from that copy, or from any part of
+it.  (Additional permissions may be written to require their own
+removal in certain cases when you modify the work.)  You may place
+additional permissions on material, added by you to a covered work,
+for which you have or can give appropriate copyright permission.
+
+  Notwithstanding any other provision of this License, for material you
+add to a covered work, you may (if authorized by the copyright holders of
+that material) supplement the terms of this License with terms:
+
+    a) Disclaiming warranty or limiting liability differently from the
+    terms of sections 15 and 16 of this License; or
+
+    b) Requiring preservation of specified reasonable legal notices or
+    author attributions in that material or in the Appropriate Legal
+    Notices displayed by works containing it; or
+
+    c) Prohibiting misrepresentation of the origin of that material, or
+    requiring that modified versions of such material be marked in
+    reasonable ways as different from the original version; or
+
+    d) Limiting the use for publicity purposes of names of licensors or
+    authors of the material; or
+
+    e) Declining to grant rights under trademark law for use of some
+    trade names, trademarks, or service marks; or
+
+    f) Requiring indemnification of licensors and authors of that
+    material by anyone who conveys the material (or modified versions of
+    it) with contractual assumptions of liability to the recipient, for
+    any liability that these contractual assumptions directly impose on
+    those licensors and authors.
+
+  All other non-permissive additional terms are considered "further
+restrictions" within the meaning of section 10.  If the Program as you
+received it, or any part of it, contains a notice stating that it is
+governed by this License along with a term that is a further
+restriction, you may remove that term.  If a license document contains
+a further restriction but permits relicensing or conveying under this
+License, you may add to a covered work material governed by the terms
+of that license document, provided that the further restriction does
+not survive such relicensing or conveying.
+
+  If you add terms to a covered work in accord with this section, you
+must place, in the relevant source files, a statement of the
+additional terms that apply to those files, or a notice indicating
+where to find the applicable terms.
+
+  Additional terms, permissive or non-permissive, may be stated in the
+form of a separately written license, or stated as exceptions;
+the above requirements apply either way.
+
+  8. Termination.
+
+  You may not propagate or modify a covered work except as expressly
+provided under this License.  Any attempt otherwise to propagate or
+modify it is void, and will automatically terminate your rights under
+this License (including any patent licenses granted under the third
+paragraph of section 11).
+
+  However, if you cease all violation of this License, then your
+license from a particular copyright holder is reinstated (a)
+provisionally, unless and until the copyright holder explicitly and
+finally terminates your license, and (b) permanently, if the copyright
+holder fails to notify you of the violation by some reasonable means
+prior to 60 days after the cessation.
+
+  Moreover, your license from a particular copyright holder is
+reinstated permanently if the copyright holder notifies you of the
+violation by some reasonable means, this is the first time you have
+received notice of violation of this License (for any work) from that
+copyright holder, and you cure the violation prior to 30 days after
+your receipt of the notice.
+
+  Termination of your rights under this section does not terminate the
+licenses of parties who have received copies or rights from you under
+this License.  If your rights have been terminated and not permanently
+reinstated, you do not qualify to receive new licenses for the same
+material under section 10.
+
+  9. Acceptance Not Required for Having Copies.
+
+  You are not required to accept this License in order to receive or
+run a copy of the Program.  Ancillary propagation of a covered work
+occurring solely as a consequence of using peer-to-peer transmission
+to receive a copy likewise does not require acceptance.  However,
+nothing other than this License grants you permission to propagate or
+modify any covered work.  These actions infringe copyright if you do
+not accept this License.  Therefore, by modifying or propagating a
+covered work, you indicate your acceptance of this License to do so.
+
+  10. Automatic Licensing of Downstream Recipients.
+
+  Each time you convey a covered work, the recipient automatically
+receives a license from the original licensors, to run, modify and
+propagate that work, subject to this License.  You are not responsible
+for enforcing compliance by third parties with this License.
+
+  An "entity transaction" is a transaction transferring control of an
+organization, or substantially all assets of one, or subdividing an
+organization, or merging organizations.  If propagation of a covered
+work results from an entity transaction, each party to that
+transaction who receives a copy of the work also receives whatever
+licenses to the work the party's predecessor in interest had or could
+give under the previous paragraph, plus a right to possession of the
+Corresponding Source of the work from the predecessor in interest, if
+the predecessor has it or can get it with reasonable efforts.
+
+  You may not impose any further restrictions on the exercise of the
+rights granted or affirmed under this License.  For example, you may
+not impose a license fee, royalty, or other charge for exercise of
+rights granted under this License, and you may not initiate litigation
+(including a cross-claim or counterclaim in a lawsuit) alleging that
+any patent claim is infringed by making, using, selling, offering for
+sale, or importing the Program or any portion of it.
+
+  11. Patents.
+
+  A "contributor" is a copyright holder who authorizes use under this
+License of the Program or a work on which the Program is based.  The
+work thus licensed is called the contributor's "contributor version".
+
+  A contributor's "essential patent claims" are all patent claims
+owned or controlled by the contributor, whether already acquired or
+hereafter acquired, that would be infringed by some manner, permitted
+by this License, of making, using, or selling its contributor version,
+but do not include claims that would be infringed only as a
+consequence of further modification of the contributor version.  For
+purposes of this definition, "control" includes the right to grant
+patent sublicenses in a manner consistent with the requirements of
+this License.
+
+  Each contributor grants you a non-exclusive, worldwide, royalty-free
+patent license under the contributor's essential patent claims, to
+make, use, sell, offer for sale, import and otherwise run, modify and
+propagate the contents of its contributor version.
+
+  In the following three paragraphs, a "patent license" is any express
+agreement or commitment, however denominated, not to enforce a patent
+(such as an express permission to practice a patent or covenant not to
+sue for patent infringement).  To "grant" such a patent license to a
+party means to make such an agreement or commitment not to enforce a
+patent against the party.
+
+  If you convey a covered work, knowingly relying on a patent license,
+and the Corresponding Source of the work is not available for anyone
+to copy, free of charge and under the terms of this License, through a
+publicly available network server or other readily accessible means,
+then you must either (1) cause the Corresponding Source to be so
+available, or (2) arrange to deprive yourself of the benefit of the
+patent license for this particular work, or (3) arrange, in a manner
+consistent with the requirements of this License, to extend the patent
+license to downstream recipients.  "Knowingly relying" means you have
+actual knowledge that, but for the patent license, your conveying the
+covered work in a country, or your recipient's use of the covered work
+in a country, would infringe one or more identifiable patents in that
+country that you have reason to believe are valid.
+
+  If, pursuant to or in connection with a single transaction or
+arrangement, you convey, or propagate by procuring conveyance of, a
+covered work, and grant a patent license to some of the parties
+receiving the covered work authorizing them to use, propagate, modify
+or convey a specific copy of the covered work, then the patent license
+you grant is automatically extended to all recipients of the covered
+work and works based on it.
+
+  A patent license is "discriminatory" if it does not include within
+the scope of its coverage, prohibits the exercise of, or is
+conditioned on the non-exercise of one or more of the rights that are
+specifically granted under this License.  You may not convey a covered
+work if you are a party to an arrangement with a third party that is
+in the business of distributing software, under which you make payment
+to the third party based on the extent of your activity of conveying
+the work, and under which the third party grants, to any of the
+parties who would receive the covered work from you, a discriminatory
+patent license (a) in connection with copies of the covered work
+conveyed by you (or copies made from those copies), or (b) primarily
+for and in connection with specific products or compilations that
+contain the covered work, unless you entered into that arrangement,
+or that patent license was granted, prior to 28 March 2007.
+
+  Nothing in this License shall be construed as excluding or limiting
+any implied license or other defenses to infringement that may
+otherwise be available to you under applicable patent law.
+
+  12. No Surrender of Others' Freedom.
+
+  If conditions are imposed on you (whether by court order, agreement or
+otherwise) that contradict the conditions of this License, they do not
+excuse you from the conditions of this License.  If you cannot convey a
+covered work so as to satisfy simultaneously your obligations under this
+License and any other pertinent obligations, then as a consequence you may
+not convey it at all.  For example, if you agree to terms that obligate you
+to collect a royalty for further conveying from those to whom you convey
+the Program, the only way you could satisfy both those terms and this
+License would be to refrain entirely from conveying the Program.
+
+  13. Use with the GNU Affero General Public License.
+
+  Notwithstanding any other provision of this License, you have
+permission to link or combine any covered work with a work licensed
+under version 3 of the GNU Affero General Public License into a single
+combined work, and to convey the resulting work.  The terms of this
+License will continue to apply to the part which is the covered work,
+but the special requirements of the GNU Affero General Public License,
+section 13, concerning interaction through a network will apply to the
+combination as such.
+
+  14. Revised Versions of this License.
+
+  The Free Software Foundation may publish revised and/or new versions of
+the GNU General Public License from time to time.  Such new versions will
+be similar in spirit to the present version, but may differ in detail to
+address new problems or concerns.
+
+  Each version is given a distinguishing version number.  If the
+Program specifies that a certain numbered version of the GNU General
+Public License "or any later version" applies to it, you have the
+option of following the terms and conditions either of that numbered
+version or of any later version published by the Free Software
+Foundation.  If the Program does not specify a version number of the
+GNU General Public License, you may choose any version ever published
+by the Free Software Foundation.
+
+  If the Program specifies that a proxy can decide which future
+versions of the GNU General Public License can be used, that proxy's
+public statement of acceptance of a version permanently authorizes you
+to choose that version for the Program.
+
+  Later license versions may give you additional or different
+permissions.  However, no additional obligations are imposed on any
+author or copyright holder as a result of your choosing to follow a
+later version.
+
+  15. Disclaimer of Warranty.
+
+  THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
+APPLICABLE LAW.  EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
+HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
+OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
+THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
+PURPOSE.  THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
+IS WITH YOU.  SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
+ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
+
+  16. Limitation of Liability.
+
+  IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
+WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
+THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
+GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
+USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
+DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
+PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
+EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
+SUCH DAMAGES.
+
+  17. Interpretation of Sections 15 and 16.
+
+  If the disclaimer of warranty and limitation of liability provided
+above cannot be given local legal effect according to their terms,
+reviewing courts shall apply local law that most closely approximates
+an absolute waiver of all civil liability in connection with the
+Program, unless a warranty or assumption of liability accompanies a
+copy of the Program in return for a fee.
+
+                     END OF TERMS AND CONDITIONS
+
+            How to Apply These Terms to Your New Programs
+
+  If you develop a new program, and you want it to be of the greatest
+possible use to the public, the best way to achieve this is to make it
+free software which everyone can redistribute and change under these terms.
+
+  To do so, attach the following notices to the program.  It is safest
+to attach them to the start of each source file to most effectively
+state the exclusion of warranty; and each file should have at least
+the "copyright" line and a pointer to where the full notice is found.
+
+    <one line to give the program's name and a brief idea of what it does.>
+    Copyright (C) <year>  <name of author>
+
+    This program is free software: you can redistribute it and/or modify
+    it under the terms of the GNU General Public License as published by
+    the Free Software Foundation, either version 3 of the License, or
+    (at your option) any later version.
+
+    This program is distributed in the hope that it will be useful,
+    but WITHOUT ANY WARRANTY; without even the implied warranty of
+    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+    GNU General Public License for more details.
+
+    You should have received a copy of the GNU General Public License
+    along with this program.  If not, see <https://www.gnu.org/licenses/>.
+
+Also add information on how to contact you by electronic and paper mail.
+
+  If the program does terminal interaction, make it output a short
+notice like this when it starts in an interactive mode:
+
+    <program>  Copyright (C) <year>  <name of author>
+    This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
+    This is free software, and you are welcome to redistribute it
+    under certain conditions; type `show c' for details.
+
+The hypothetical commands `show w' and `show c' should show the appropriate
+parts of the General Public License.  Of course, your program's commands
+might be different; for a GUI interface, you would use an "about box".
+
+  You should also get your employer (if you work as a programmer) or school,
+if any, to sign a "copyright disclaimer" for the program, if necessary.
+For more information on this, and how to apply and follow the GNU GPL, see
+<https://www.gnu.org/licenses/>.
+
+  The GNU General Public License does not permit incorporating your program
+into proprietary programs.  If your program is a subroutine library, you
+may consider it more useful to permit linking proprietary applications with
+the library.  If this is what you want to do, use the GNU Lesser General
+Public License instead of this License.  But first, please read
+<https://www.gnu.org/licenses/why-not-lgpl.html>.

COPYING.LESSER

@@ -0,0 +1,165 @@
+                   GNU LESSER GENERAL PUBLIC LICENSE
+                       Version 3, 29 June 2007
+
+ Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
+ Everyone is permitted to copy and distribute verbatim copies
+ of this license document, but changing it is not allowed.
+
+
+  This version of the GNU Lesser General Public License incorporates
+the terms and conditions of version 3 of the GNU General Public
+License, supplemented by the additional permissions listed below.
+
+  0. Additional Definitions.
+
+  As used herein, "this License" refers to version 3 of the GNU Lesser
+General Public License, and the "GNU GPL" refers to version 3 of the GNU
+General Public License.
+
+  "The Library" refers to a covered work governed by this License,
+other than an Application or a Combined Work as defined below.
+
+  An "Application" is any work that makes use of an interface provided
+by the Library, but which is not otherwise based on the Library.
+Defining a subclass of a class defined by the Library is deemed a mode
+of using an interface provided by the Library.
+
+  A "Combined Work" is a work produced by combining or linking an
+Application with the Library.  The particular version of the Library
+with which the Combined Work was made is also called the "Linked
+Version".
+
+  The "Minimal Corresponding Source" for a Combined Work means the
+Corresponding Source for the Combined Work, excluding any source code
+for portions of the Combined Work that, considered in isolation, are
+based on the Application, and not on the Linked Version.
+
+  The "Corresponding Application Code" for a Combined Work means the
+object code and/or source code for the Application, including any data
+and utility programs needed for reproducing the Combined Work from the
+Application, but excluding the System Libraries of the Combined Work.
+
+  1. Exception to Section 3 of the GNU GPL.
+
+  You may convey a covered work under sections 3 and 4 of this License
+without being bound by section 3 of the GNU GPL.
+
+  2. Conveying Modified Versions.
+
+  If you modify a copy of the Library, and, in your modifications, a
+facility refers to a function or data to be supplied by an Application
+that uses the facility (other than as an argument passed when the
+facility is invoked), then you may convey a copy of the modified
+version:
+
+   a) under this License, provided that you make a good faith effort to
+   ensure that, in the event an Application does not supply the
+   function or data, the facility still operates, and performs
+   whatever part of its purpose remains meaningful, or
+
+   b) under the GNU GPL, with none of the additional permissions of
+   this License applicable to that copy.
+
+  3. Object Code Incorporating Material from Library Header Files.
+
+  The object code form of an Application may incorporate material from
+a header file that is part of the Library.  You may convey such object
+code under terms of your choice, provided that, if the incorporated
+material is not limited to numerical parameters, data structure
+layouts and accessors, or small macros, inline functions and templates
+(ten or fewer lines in length), you do both of the following:
+
+   a) Give prominent notice with each copy of the object code that the
+   Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the object code with a copy of the GNU GPL and this license
+   document.
+
+  4. Combined Works.
+
+  You may convey a Combined Work under terms of your choice that,
+taken together, effectively do not restrict modification of the
+portions of the Library contained in the Combined Work and reverse
+engineering for debugging such modifications, if you also do each of
+the following:
+
+   a) Give prominent notice with each copy of the Combined Work that
+   the Library is used in it and that the Library and its use are
+   covered by this License.
+
+   b) Accompany the Combined Work with a copy of the GNU GPL and this license
+   document.
+
+   c) For a Combined Work that displays copyright notices during
+   execution, include the copyright notice for the Library among
+   these notices, as well as a reference directing the user to the
+   copies of the GNU GPL and this license document.
+
+   d) Do one of the following:
+
+       0) Convey the Minimal Corresponding Source under the terms of this
+       License, and the Corresponding Application Code in a form
+       suitable for, and under terms that permit, the user to
+       recombine or relink the Application with a modified version of
+       the Linked Version to produce a modified Combined Work, in the
+       manner specified by section 6 of the GNU GPL for conveying
+       Corresponding Source.
+
+       1) Use a suitable shared library mechanism for linking with the
+       Library.  A suitable mechanism is one that (a) uses at run time
+       a copy of the Library already present on the user's computer
+       system, and (b) will operate properly with a modified version
+       of the Library that is interface-compatible with the Linked
+       Version.
+
+   e) Provide Installation Information, but only if you would otherwise
+   be required to provide such information under section 6 of the
+   GNU GPL, and only to the extent that such information is
+   necessary to install and execute a modified version of the
+   Combined Work produced by recombining or relinking the
+   Application with a modified version of the Linked Version. (If
+   you use option 4d0, the Installation Information must accompany
+   the Minimal Corresponding Source and Corresponding Application
+   Code. If you use option 4d1, you must provide the Installation
+   Information in the manner specified by section 6 of the GNU GPL
+   for conveying Corresponding Source.)
+
+  5. Combined Libraries.
+
+  You may place library facilities that are a work based on the
+Library side by side in a single library together with other library
+facilities that are not Applications and are not covered by this
+License, and convey such a combined library under terms of your
+choice, if you do both of the following:
+
+   a) Accompany the combined library with a copy of the same work based
+   on the Library, uncombined with any other library facilities,
+   conveyed under the terms of this License.
+
+   b) Give prominent notice with the combined library that part of it
+   is a work based on the Library, and explaining where to find the
+   accompanying uncombined form of the same work.
+
+  6. Revised Versions of the GNU Lesser General Public License.
+
+  The Free Software Foundation may publish revised and/or new versions
+of the GNU Lesser General Public License from time to time. Such new
+versions will be similar in spirit to the present version, but may
+differ in detail to address new problems or concerns.
+
+  Each version is given a distinguishing version number. If the
+Library as you received it specifies that a certain numbered version
+of the GNU Lesser General Public License "or any later version"
+applies to it, you have the option of following the terms and
+conditions either of that published version or of any later version
+published by the Free Software Foundation. If the Library as you
+received it does not specify a version number of the GNU Lesser
+General Public License, you may choose any version of the GNU Lesser
+General Public License ever published by the Free Software Foundation.
+
+  If the Library as you received it specifies that a proxy can decide
+whether future versions of the GNU Lesser General Public License shall
+apply, that proxy's public statement of acceptance of any version is
+permanent authorization for you to choose that version for the
+Library.

Dockerfile

@@ -1,72 +1,79 @@
-FROM avatao/debian:buster
+ARG FRONTEND_VERSION
+FROM avatao/frontend-tutorial-framework:${FRONTEND_VERSION} as frontend
 
-RUN curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash -                                     &&\
+FROM avatao/debian:buster
-    curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add -                             &&\
+ARG FRONTEND_VERSION
-    echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list &&\
+LABEL tfw.frontend.version=${FRONTEND_VERSION}
-    apt-get update                                                                                     &&\
+
+RUN apt-get update                           &&\
     apt-get install -y --no-install-recommends \
-        nodejs                                 \
-        yarn                                   \
         supervisor                             \
+        ncat                                   \
         libzmq5                                \
         nginx                                  \
+        jq                                     \
         gettext-base                         &&\
-    rm -rf /var/lib/apt/lists/*
+    rm -rf /var/lib/apt/lists/*              &&\
+    ln -sf /bin/bash /bin/sh
 
 COPY requirements.txt /tmp
 RUN pip3 install -r /tmp/requirements.txt
 
+RUN curl -Ls https://github.com/krallin/tini/releases/download/v0.18.0/tini-amd64 --output /bin/init &&\
+    echo "12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855 /bin/init"                 |\
+    sha256sum --check --status                                                                       &&\
+    chmod 755 /bin/init
+
 ENV TFW_PUBLIC_PORT=8888            \
     TFW_WEB_PORT=4242               \
     TFW_LOGIN_APP_PORT=6666         \
-    TFW_TERMINADO_PORT=7878       \
+    TFW_TERMINAL_PORT=7878          \
+    TFW_TERMINAL_FALLBACK_PORT=7879 \
     TFW_SUPERVISOR_HTTP_PORT=9001   \
-    TFW_PUBLISHER_PORT=7654       \
+    TFW_PUB_PORT=7654               \
-    TFW_RECEIVER_PORT=8765
+    TFW_PULL_PORT=8765
 
 EXPOSE ${TFW_PUBLIC_PORT}
 
-ENV PYTHONPATH="/usr/local/lib/"                                           \
+ENV PYTHONPATH="/usr/local/lib"                                            \
     TFW_SUPERVISOR_HTTP_URI="http://localhost:${TFW_SUPERVISOR_HTTP_PORT}" \
     TFW_SUPERVISORD_CONF="/etc/supervisor/supervisord.conf"                \
     TFW_SUPERVISORD_COMPONENTS="/etc/supervisor/conf"                      \
     TFW_NGINX_CONF="/etc/nginx/nginx.conf"                                 \
     TFW_NGINX_DEFAULT="/etc/nginx/sites-enabled/default"                   \
     TFW_NGINX_COMPONENTS="/etc/nginx/components"                           \
-    TFW_LIB_DIR="/usr/local/lib/"                                          \
+    TFW_LIB_DIR="/usr/local/lib"                                           \
-    TFW_TERMINADO_DIR="/tmp/terminado_server"                              \
     TFW_FRONTEND_DIR="/srv/frontend"                                       \
+    TFW_DIR="/.tfw"                                                        \
+    TFW_SERVER_DIR="/.tfw/tfw_server"                                      \
+    TFW_AUTH_KEY="/tmp/tfw-auth.key"                                       \
+    TFW_LOGS_DIR="/var/log/tfw"                                            \
+    TFW_PIPES_DIR="/run/tfw"                                               \
+    TFW_SNAPSHOTS_DIR="/tmp/tfw-snapshots"                                 \
     TFW_HISTFILE="/home/${AVATAO_USER}/.bash_history"                      \
     PROMPT_COMMAND="history -a"
 
-RUN echo "shopt -s cmdhist\n"                 \
+COPY bashrc supervisor/tfw_init.sh /tmp/
-         "shopt -s histappend\n"              \
-         "unset HISTCONTROL\n"                \
-         "export HISTFILE=$TFW_HISTFILE\n"    \
-         "export HISTFILESIZE=1000\n"         \
-         "export HISTSIZE=1000\n"             \
-         'PROMPT_COMMAND="history -a"\n'      \
-    >> /home/${AVATAO_USER}/.bashrc
-
 COPY supervisor/supervisord.conf ${TFW_SUPERVISORD_CONF}
+COPY supervisor/components/ ${TFW_SUPERVISORD_COMPONENTS}
 COPY nginx/nginx.conf ${TFW_NGINX_CONF}
 COPY nginx/default.conf ${TFW_NGINX_DEFAULT}
-COPY lib ${TFW_LIB_DIR}
+COPY nginx/components/ ${TFW_NGINX_COMPONENTS}
+COPY tfw ${TFW_LIB_DIR}/tfw
+COPY supervisor/tfw_server.py ${TFW_SERVER_DIR}/
+COPY --from=frontend /srv/dist ${TFW_FRONTEND_DIR}
 
-ONBUILD ARG BUILD_CONTEXT="."
+VOLUME ["${TFW_LOGS_DIR}", "${TFW_PIPES_DIR}"]
-ONBUILD ARG NOFRONTEND=""
 
-ONBUILD COPY ${BUILD_CONTEXT}/nginx/components/ ${TFW_NGINX_COMPONENTS}
+ONBUILD ARG BUILD_CONTEXT="solvable"
-ONBUILD COPY ${BUILD_CONTEXT}/supervisor/components/ ${TFW_SUPERVISORD_COMPONENTS}
 
-ONBUILD RUN chown -R ${AVATAO_USER} /var/log/nginx /var/lib/nginx                                          &&\
+ONBUILD COPY ${BUILD_CONTEXT}/nginx/ ${TFW_NGINX_COMPONENTS}
-            for f in "${TFW_NGINX_DEFAULT}" ${TFW_NGINX_COMPONENTS}/*.conf; do                               \
+ONBUILD COPY ${BUILD_CONTEXT}/supervisor/ ${TFW_SUPERVISORD_COMPONENTS}
+
+ONBUILD RUN for f in "${TFW_NGINX_DEFAULT}" ${TFW_NGINX_COMPONENTS}/*.conf; do                               \
                 envsubst "$(printenv | cut -d= -f1 | grep TFW_ | sed -e 's/^/$/g')" < $f > $f~ && mv $f~ $f ;\
             done
+ONBUILD VOLUME ["/etc/nginx", "/etc/supervisor", "/var/lib/nginx", "/var/log/nginx", "${TFW_LIB_DIR}/tfw"]
 
-ONBUILD COPY ${BUILD_CONTEXT}/frontend /data/
+ENTRYPOINT ["/bin/init", "--"]
-ONBUILD RUN test -z "${NOFRONTEND}" && cd /data && yarn install --frozen-lockfile || :
+CMD exec supervisord --nodaemon --configuration ${TFW_SUPERVISORD_CONF}
-ONBUILD RUN test -z "${NOFRONTEND}" && cd /data && yarn build --no-progress || :
-ONBUILD RUN test -z "${NOFRONTEND}" && mv /data/dist ${TFW_FRONTEND_DIR} && rm -rf /data || :
-
-CMD exec supervisord --nodaemon

README.md

@@ -1,16 +1,5 @@
-Tutorial Framework
+# baseimage-tutorial-framework
-==================
 
-## Building and running with magic starter
+This is the beating heart of TFW – the Docker baseimage containing the internals of the framework. It's most important element is the robust messaging system that makes it straightforward to implement interactive exercises.
 
-Execute the `tfw_magic_start.sh` script and it will automagically run a backend instance in Docker and serve the Angular frontend locally.
+To learn more about the framework and get started, please consult our [wiki](https://github.com/avatao-content/baseimage-tutorial-framework/wiki).
-
-This requires the `test-tutorial-framework` (Docker image `FROM baseimage-tutorial-framework`) and `frontend-tutorial-framework` repositories to have a common parent folder.
-
-## Building with Docker
-
-Simply issue `docker build -t baseimage-tutorial-framework .` in the project root.
-
-## Creating child images
-
-Documentation in progress...

VERSION

@@ -1 +1 @@
-egyptianmau
+chausie

bashrc

@@ -0,0 +1,20 @@
+_postcmd() 
+{
+    AT_PROMPT=1
+    history -a
+}
+PROMPT_COMMAND="_postcmd"
+
+_precmd()
+{
+    [ -z "$AT_PROMPT" ] && return
+    unset AT_PROMPT
+    [ -z "$TFW_DELAY_HISTAPPEND" ] && history -a
+}
+trap _precmd DEBUG
+
+shopt -s cmdhist
+shopt -s histappend
+unset HISTCONTROL
+export HISTFILESIZE=1000
+export HISTSIZE=1000

docs/Makefile

@@ -0,0 +1,20 @@
+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line.
+SPHINXOPTS    =
+SPHINXBUILD   = sphinx-build
+SPHINXPROJ    = baseimage-tutorial-framework
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

docs/make.bat

@@ -0,0 +1,36 @@
+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+set SPHINXPROJ=baseimage-tutorial-framework
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.http://sphinx-doc.org/
+	exit /b 1
+)
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS%
+
+:end
+popd

docs/source/components/components.rst

@@ -0,0 +1,32 @@
+Components
+----------
+
+.. automodule:: tfw.components
+
+.. autoclass:: IdeEventHandler
+    :members:
+
+.. autoclass:: TerminalEventHandler
+    :members:
+
+.. autoclass:: ProcessManagingEventHandler
+    :members:
+
+.. autoclass:: LogMonitoringEventHandler
+    :members:
+
+.. autoclass:: TerminalCommands
+    :members:
+
+.. autoclass:: HistoryMonitor
+    :members:
+
+.. autoclass:: BashMonitor
+    :members:
+
+.. autoclass:: FSMManagingEventHandler
+    :members:
+
+.. autoclass:: CommandsEqual
+    :members:
+

docs/source/conf.py

@@ -0,0 +1,166 @@
+# -*- coding: utf-8 -*-
+#
+# Configuration file for the Sphinx documentation builder.
+#
+# This file does only contain a selection of the most common options. For a
+# full list see the documentation:
+# http://www.sphinx-doc.org/en/master/config
+
+# -- Path setup --------------------------------------------------------------
+
+# If extensions (or modules to document with autodoc) are in another directory,
+# add these directories to sys.path here. If the directory is relative to the
+# documentation root, use os.path.abspath to make it absolute, like shown here.
+#
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../../lib'))
+
+
+# -- Project information -----------------------------------------------------
+
+project = 'baseimage-tutorial-framework'
+copyright = '2018, Avatao Innovative Learning Kft'
+author = 'Kristóf Tóth'
+
+# The short X.Y version
+version = ''
+# The full version, including alpha/beta/rc tags
+release = 'mainecoon'
+
+
+# -- General configuration ---------------------------------------------------
+
+# If your documentation needs a minimal Sphinx version, state it here.
+#
+# needs_sphinx = '1.0'
+
+# Add any Sphinx extension module names here, as strings. They can be
+# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
+# ones.
+extensions = [
+    'sphinx.ext.autodoc',
+]
+
+# Add any paths that contain templates here, relative to this directory.
+templates_path = ['_templates']
+
+# The suffix(es) of source filenames.
+# You can specify multiple suffix as a list of string:
+#
+# source_suffix = ['.rst', '.md']
+source_suffix = '.rst'
+
+# The master toctree document.
+master_doc = 'index'
+
+# The language for content autogenerated by Sphinx. Refer to documentation
+# for a list of supported languages.
+#
+# This is also used if you do content translation via gettext catalogs.
+# Usually you set "language" from the command line for these cases.
+language = None
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path .
+exclude_patterns = []
+
+# The name of the Pygments (syntax highlighting) style to use.
+pygments_style = 'sphinx'
+
+
+# -- Options for HTML output -------------------------------------------------
+
+# The theme to use for HTML and HTML Help pages.  See the documentation for
+# a list of builtin themes.
+#
+html_theme = 'alabaster'
+
+# Theme options are theme-specific and customize the look and feel of a theme
+# further.  For a list of options available for each theme, see the
+# documentation.
+#
+# html_theme_options = {}
+
+# Add any paths that contain custom static files (such as style sheets) here,
+# relative to this directory. They are copied after the builtin static files,
+# so a file named "default.css" will overwrite the builtin "default.css".
+html_static_path = ['_static']
+
+# Custom sidebar templates, must be a dictionary that maps document names
+# to template names.
+#
+# The default sidebars (for documents that don't match any pattern) are
+# defined by theme itself.  Builtin themes are using these templates by
+# default: ``['localtoc.html', 'relations.html', 'sourcelink.html',
+# 'searchbox.html']``.
+#
+# html_sidebars = {}
+
+
+# -- Options for HTMLHelp output ---------------------------------------------
+
+# Output file base name for HTML help builder.
+htmlhelp_basename = 'baseimage-tutorial-frameworkdoc'
+
+
+# -- Options for LaTeX output ------------------------------------------------
+
+latex_elements = {
+    # The paper size ('letterpaper' or 'a4paper').
+    #
+    # 'papersize': 'letterpaper',
+
+    # The font size ('10pt', '11pt' or '12pt').
+    #
+    # 'pointsize': '10pt',
+
+    # Additional stuff for the LaTeX preamble.
+    #
+    # 'preamble': '',
+
+    # Latex figure (float) alignment
+    #
+    # 'figure_align': 'htbp',
+}
+
+# Grouping the document tree into LaTeX files. List of tuples
+# (source start file, target name, title,
+#  author, documentclass [howto, manual, or own class]).
+latex_documents = [
+    (master_doc, 'baseimage-tutorial-framework.tex', 'baseimage-tutorial-framework Documentation',
+     'Kristóf Tóth', 'manual'),
+]
+
+
+# -- Options for manual page output ------------------------------------------
+
+# One entry per manual page. List of tuples
+# (source start file, name, description, authors, manual section).
+man_pages = [
+    (master_doc, 'baseimage-tutorial-framework', 'baseimage-tutorial-framework Documentation',
+     [author], 1)
+]
+
+
+# -- Options for Texinfo output ----------------------------------------------
+
+# Grouping the document tree into Texinfo files. List of tuples
+# (source start file, target name, title, author,
+#  dir menu entry, description, category)
+texinfo_documents = [
+    (master_doc, 'baseimage-tutorial-framework', 'baseimage-tutorial-framework Documentation',
+     author, 'baseimage-tutorial-framework', 'One line description of project.',
+     'Miscellaneous'),
+]
+
+
+# -- Extension configuration -------------------------------------------------
+def skip(app, what, name, obj, skip, options):
+    if name == "__init__":
+        return False
+    return skip
+
+def setup(app):
+    app.connect("autodoc-skip-member", skip)

docs/source/foundations/eventhandlers.rst

@@ -0,0 +1,15 @@
+Event handler base classes
+--------------------------
+
+Subclass these to create your cusom event handlers.
+
+.. automodule:: tfw.event_handler_base
+
+.. autoclass:: EventHandlerBase
+    :members:
+
+.. autoclass:: FSMAwareEventHandler
+    :members:
+
+.. autoclass:: BroadcastingEventHandler
+    :members:

docs/source/foundations/fsms.rst

@@ -0,0 +1,15 @@
+FSM base classes
+----------------
+
+Subclass these to create an FSM that fits your tutorial/challenge.
+
+.. automodule:: tfw.fsm
+
+.. autoclass:: FSMBase
+    :members:
+
+.. autoclass:: LinearFSM
+    :members:
+
+.. autoclass:: YamlFSM
+    :members:

docs/source/foundations/tfwserver.rst

@@ -0,0 +1,7 @@
+TFWServer
+---------
+
+.. automodule:: tfw.networking
+
+.. autoclass:: TFWServer
+    :members:

docs/source/index.rst

@@ -0,0 +1,55 @@
+.. baseimage-tutorial-framework documentation master file, created by
+   sphinx-quickstart on Fri Jun  1 14:29:07 2018.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to baseimage-tutorial-framework's documentation!
+========================================================
+
+Foundations
+-----------
+
+This part covers the soil which the framework is based on and stuff you will need to develop your own challenges.
+
+.. toctree::
+    :glob:
+
+    foundations/*
+
+Networking
+----------
+
+You can use these to send messages to the frontend or the event handlers through TFWServer.
+
+.. toctree::
+    :glob:
+
+    networking/*
+
+Components
+----------
+
+These are pre-written components for you to use, such as our IDE, terminal or console.
+
+.. toctree::
+    :glob:
+
+    components/*
+
+Utility
+-------
+
+These are useful decorators, mixins and helpers to make common dev tasks easier.
+
+.. toctree::
+    :glob:
+
+    utility/*
+
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

docs/source/networking/networking.rst

@@ -0,0 +1,22 @@
+Networking
+----------
+
+.. automodule:: tfw.networking
+
+.. autoclass:: TFWServerConnector
+    :members:
+
+.. automodule:: tfw.networking.event_handlers.server_connector
+
+.. autoclass:: ServerUplinkConnector
+    :members:
+
+.. automodule:: tfw.networking
+
+.. autoclass:: MessageSender
+    :members:
+
+.. automodule:: tfw.networking.fsm_aware
+
+.. autoclass:: FSMAware
+    :members:

docs/source/utility/decorators.rst

@@ -0,0 +1,10 @@
+TFW decorators
+--------------
+
+.. automodule:: tfw.decorators.rate_limiter
+
+.. autoclass:: RateLimiter
+    :members:
+
+.. autoclass:: AsyncRateLimiter
+    :members:

lib/envvars.py

@@ -1,9 +0,0 @@
-from collections import namedtuple
-from os import environ
-
-
-def prefixed_envvars_to_namedtuple(prefix: str, tuple_name: str):
-    envvars = {envvar.replace(prefix, '', 1): environ.get(envvar)
-               for envvar in environ.keys()
-               if envvar.startswith(prefix)}
-    return namedtuple(tuple_name, envvars)(**envvars)

lib/tao/config/__init__.py

@@ -1 +0,0 @@
-from .envvars import * # pylint: disable=wildcard-import

lib/tao/config/envvars.py

@@ -1,3 +0,0 @@
-from envvars import prefixed_envvars_to_namedtuple
-
-TAOENV = prefixed_envvars_to_namedtuple('AVATAO_', 'taoenvtuple')

lib/tfw/components/decorators/__init__.py

@@ -1 +0,0 @@
-from .rate_limiter import RateLimiter

lib/tfw/components/decorators/rate_limiter.py

@@ -1,22 +0,0 @@
-from functools import wraps
-from time import time, sleep
-
-
-class RateLimiter:
-    def __init__(self, rate_per_second):
-        self.min_interval = 1 / float(rate_per_second)
-        self.last_call = time()
-
-    def __call__(self, fun):
-        @wraps(fun)
-        def wrapper(*args, **kwargs):
-            self._limit_rate()
-            fun(*args, **kwargs)
-        return wrapper
-
-    def _limit_rate(self):
-        since_last_call = time() - self.last_call
-        to_next_call = self.min_interval - since_last_call
-        self.last_call = time()
-        if to_next_call > 0:
-            sleep(to_next_call)

lib/tfw/components/directory_monitor.py

@@ -1,81 +0,0 @@
-from functools import wraps
-
-from watchdog.observers import Observer
-from watchdog.events import FileSystemEventHandler as FileSystemWatchdogEventHandler
-
-from tfw.networking.event_handlers.server_connector import ServerUplinkConnector
-from tfw.components.decorators import RateLimiter
-
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class DirectoryMonitor:
-    def __init__(self, directory):
-        self.observer = Observer()
-        self.eventhandler = WebideReloadWatchdogEventHandler()
-        self.observer.schedule(self.eventhandler, directory, recursive=True)
-        self.pause, self.resume = self.eventhandler.pause, self.eventhandler.resume
-
-    def watch(self):
-        self.observer.start()
-
-    def stop(self):
-        self.observer.stop()
-        self.observer.join()
-
-    @property
-    def ignore(self):
-        return self.eventhandler.ignore
-
-    @ignore.setter
-    def ignore(self, value):
-        self.eventhandler.ignore = value if value >= 0 else 0
-
-    @property
-    def pauser(self):
-        return DirectoryMonitor.Pauser(self)
-
-    class Pauser:
-        def __init__(self, directory_monitor):
-            self.directorymonitor = directory_monitor
-        def __enter__(self):
-            self.directorymonitor.pause()
-        def __exit__(self, exc_type, exc_val, exc_tb):
-            self.directorymonitor.resume()
-
-
-class WebideReloadWatchdogEventHandler(FileSystemWatchdogEventHandler):
-    def __init__(self):
-        super().__init__()
-        self.uplink = ServerUplinkConnector()
-        self._paused = False
-        self.ignore = 0
-
-    def pause(self):
-        self._paused = True
-
-    def resume(self):
-        self._paused = False
-
-    @RateLimiter(rate_per_second=2)
-    def on_modified(self, event):
-        if self._paused:
-            return
-        if self.ignore > 0:
-            self.ignore = self.ignore - 1
-            return
-        LOG.debug(event)
-        key = 'webide'
-        self.uplink.send(key, {'data': {'command': 'reload'}})
-
-
-def with_monitor_paused(fun):
-    @wraps(fun)
-    def wrapper(self, *args, **kwargs):
-        if self.monitor:
-            with self.monitor.pauser:
-                return fun(self, *args, **kwargs)
-        return fun(self, *args, **kwargs)
-    return wrapper

lib/tfw/components/directory_monitoring_event_handler.py

@@ -1,72 +0,0 @@
-from os.path import isdir, exists
-
-from tfw.components.directory_monitor import DirectoryMonitor
-from tfw.event_handler_base import TriggerlessEventHandler
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class DirectoryMonitoringEventHandler(TriggerlessEventHandler):
-    def __init__(self, key, directory):
-        super().__init__(key)
-        self._directory = directory
-        self._monitor = None
-        self.reload_monitor()
-        self.commands = {'pause':     self.pause,
-                         'resume':    self.resume,
-                         'ignore':    self.ignore,
-                         'selectdir': self.selectdir}
-
-    @property
-    def directory(self):
-        return self._directory
-
-    @directory.setter
-    def directory(self, directory):
-        if not exists(directory) or not isdir(directory):
-            raise EnvironmentError('No such directory!')
-        self._directory = directory
-
-    @property
-    def monitor(self):
-        return self._monitor
-
-    def reload_monitor(self):
-        if self._monitor:
-            try:
-                self._monitor.stop()
-            except KeyError:
-                logging.debug('Working directory was removed – ignoring...')
-        self._monitor = DirectoryMonitor(self._directory)
-        self._monitor.watch()   # This runs on a separate thread
-
-    def handle_event(self, key, message):
-        try:
-            message['data'] = self.commands[message['data']['command']](message['data'])
-            return message
-        except KeyError:
-            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
-
-    def pause(self, data):
-        self.monitor.pause()
-        return data
-
-    def resume(self, data):
-        self.monitor.resume()
-        return data
-
-    def ignore(self, data):
-        self.monitor.ignore += data['ignore']
-        return data
-
-    def selectdir(self, data):
-        try:
-            self.directory = data['directory']
-            self.reload_monitor()
-            return data
-        except EnvironmentError:
-            LOG.error('Failed to switch directory!')
-
-    def cleanup(self):
-        self.monitor.stop()

lib/tfw/components/history_monitor.py

@@ -1,80 +0,0 @@
-from os.path import dirname
-from re import findall
-from re import compile as compileregex
-from abc import ABC, abstractmethod
-
-from watchdog.observers import Observer
-from watchdog.events import PatternMatchingEventHandler
-
-from tfw.components.mixins.callback_mixin import CallbackMixin
-from tfw.components.decorators.rate_limiter import RateLimiter
-
-
-class CallbackEventHandler(PatternMatchingEventHandler, ABC):
-    def __init__(self, files, *callbacks):
-        super().__init__(files)
-        self.callbacks = callbacks
-
-    @RateLimiter(rate_per_second=2)
-    def on_modified(self, event):
-        for callback in self.callbacks:
-            callback()
-
-
-class HistoryMonitor(CallbackMixin, ABC):
-    def __init__(self, histfile):
-        CallbackMixin.__init__(self)
-        self.histfile = histfile
-        self._history = []
-        self._last_length = len(self._history)
-        self.observer = Observer()
-        self.observer.schedule(CallbackEventHandler([self.histfile],
-                                                    self._fetch_history,
-                                                    self._invoke_callbacks),
-                               dirname(self.histfile))
-
-    @property
-    def history(self):
-        return self._history
-
-    def _fetch_history(self):
-        self._last_length = len(self._history)
-        with open(self.histfile, 'r') as ifile:
-            pattern = compileregex(self.command_pattern)
-            data = ifile.read()
-            self._history = [self.sanitize_command(command) for command in findall(pattern, data)]
-
-    @property
-    @abstractmethod
-    def command_pattern(self):
-        raise NotImplementedError
-
-    def sanitize_command(self, command):
-        # pylint: disable=no-self-use
-        return command
-
-    def _invoke_callbacks(self):
-        if self._last_length < len(self._history):
-            self._execute_callbacks(self.history)
-
-    def watch(self):
-        self.observer.start()
-
-    def stop(self):
-        self.observer.stop()
-        self.observer.join()
-
-
-class BashMonitor(HistoryMonitor):
-    @property
-    def command_pattern(self):
-        return r'.+'
-
-    def sanitize_command(self, command):
-        return command.strip()
-
-
-class GDBMonitor(HistoryMonitor):
-    @property
-    def command_pattern(self):
-        return r'(?<=\n)\+(.+)\n'

lib/tfw/components/mixins/__init__.py

@@ -1 +0,0 @@
-from .supervisor_mixin import SupervisorMixin

lib/tfw/components/mixins/callback_mixin.py

@@ -1,17 +0,0 @@
-from functools import partial
-
-
-class CallbackMixin:
-    def __init__(self):
-        self._callbacks = []
-
-    def subscribe_callback(self, callback, *args, **kwargs):
-        fun = partial(callback, *args, **kwargs)
-        self._callbacks.append(fun)
-
-    def unsubscribe_callback(self, callback):
-        self._callbacks.remove(callback)
-
-    def _execute_callbacks(self, *args, **kwargs):
-        for callback in self._callbacks:
-            callback(*args, **kwargs)

lib/tfw/components/mixins/supervisor_mixin.py

@@ -1,27 +0,0 @@
-import xmlrpc.client
-from xmlrpc.client import Fault as SupervisorFault
-from contextlib import suppress
-from os import remove
-
-from tfw.config import TFWENV
-
-
-class SupervisorMixin:
-    supervisor = xmlrpc.client.ServerProxy(TFWENV.SUPERVISOR_HTTP_URI).supervisor
-
-    def stop_process(self, process_name):
-        with suppress(SupervisorFault):
-            self.supervisor.stopProcess(process_name)
-
-    def start_process(self, process_name):
-        self.supervisor.startProcess(process_name)
-
-    def read_log(self, process_name):
-        logs = self.supervisor.readProcessStderrLog(process_name, 0, 0)
-        remove(self.supervisor.getProcessInfo(process_name)['stderr_logfile'])
-        self.supervisor.clearProcessLogs(process_name)
-        return logs
-
-    def restart_process(self, process_name):
-        self.stop_process(process_name)
-        self.start_process(process_name)

lib/tfw/components/process_managing_event_handler.py

@@ -1,40 +0,0 @@
-from xmlrpc.client import Fault as SupervisorFault
-
-from tfw.event_handler_base import TriggerlessEventHandler
-from tfw.components.mixins import SupervisorMixin
-from tfw.components.directory_monitor import with_monitor_paused
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class ProcessManager(SupervisorMixin):
-    def __init__(self):
-        self.commands = {'start': self.start_process,
-                         'stop': self.stop_process,
-                         'restart': self.restart_process,
-                         'readlog': self.read_log}
-
-    def __call__(self, command, process_name):
-        return self.commands[command](process_name)
-
-
-class ProcessManagingEventHandler(TriggerlessEventHandler):
-    def __init__(self, key, dirmonitor=None):
-        super().__init__(key)
-        self.key = key
-        self.monitor = dirmonitor
-        self.processmanager = ProcessManager()
-
-    @with_monitor_paused
-    def handle_event(self, key, message):
-        try:
-            data = message['data']
-            self.processmanager(data['command'], data['process_name'])
-            return message
-        except KeyError:
-            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
-        except SupervisorFault as fault:
-            message['data']['error'] = fault.faultString
-            message['data']['log'] = self.processmanager.read_log(message['data']['process_name'])
-            return message

lib/tfw/components/source_code_event_handler.py

@@ -1,163 +0,0 @@
-from os.path import isfile, join, relpath, exists, isdir
-from glob import glob
-from fnmatch import fnmatchcase
-from collections import Iterable
-
-from tfw.event_handler_base import TriggerlessEventHandler
-from tfw.components.directory_monitor import DirectoryMonitor
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class FileManager:
-    def __init__(self, working_directory, selected_file=None, exclude=None):
-        self._exclude, self.exclude = None, exclude
-        self._workdir, self.workdir = None, working_directory
-        self._filename, self.filename = None, selected_file or self.files[0]
-
-    @property
-    def exclude(self):
-        return self._exclude
-
-    @exclude.setter
-    def exclude(self, exclude):
-        if exclude is None:
-            return
-        if not isinstance(exclude, Iterable):
-            raise TypeError('Exclude must be Iterable!')
-        self._exclude = exclude
-
-    @property
-    def workdir(self):
-        return self._workdir
-
-    @workdir.setter
-    def workdir(self, directory):
-        if not exists(directory) or not isdir(directory):
-            raise EnvironmentError('"{}" is not a directory!'.format(directory))
-        self._workdir = directory
-
-    @property
-    def filename(self):
-        return self._filename
-
-    @filename.setter
-    def filename(self, filename):
-        if not filename in self.files:
-            raise EnvironmentError('No such file in workdir!')
-        self._filename = filename
-
-    @property
-    def files(self):
-        return [self._relpath(file) for file in glob(join(self._workdir, '**/*'), recursive=True)
-                if isfile(file) and
-                not any(fnmatchcase(file, blacklisted) for blacklisted in self.exclude)]
-
-    @property
-    def file_contents(self):
-        with open(self._filepath(self.filename), 'r', errors='surrogateescape') as ifile:
-            return ifile.read()
-
-    @file_contents.setter
-    def file_contents(self, value):
-        with open(self._filepath(self.filename), 'w', errors='surrogateescape') as ofile:
-            ofile.write(value)
-
-    def _filepath(self, filename):
-        return join(self._workdir, filename)
-
-    def _relpath(self, filename):
-        return relpath(self._filepath(filename), start=self._workdir)
-
-
-class SourceCodeEventHandler(TriggerlessEventHandler):
-    def __init__(self, key, directory, selected_file=None, exclude=None):
-        super().__init__(key)
-        self.filemanager = FileManager(directory, selected_file=selected_file, exclude=exclude)
-
-        self.commands = {'read':      self.read,
-                         'write':     self.write,
-                         'select':    self.select,
-                         'selectdir': self.select_dir,
-                         'exclude':   self.exclude}
-
-        self._monitor = None
-        self.reload_monitor()
-
-    @property
-    def monitor(self):
-        return self._monitor
-
-    def reload_monitor(self):
-        if self._monitor:
-            try:
-                self._monitor.stop()
-            except KeyError:
-                logging.debug('Working directory was removed – ignoring...')
-        self._monitor = DirectoryMonitor(self.filemanager.workdir)
-        self._monitor.watch()   # This runs on a separate thread
-
-    def read(self, data):
-        try:
-            data['content'] = self.filemanager.file_contents
-        except PermissionError:
-            data['content'] = 'You have no permission to open that file :('
-        except FileNotFoundError:
-            data['content'] = 'This file was removed :('
-        except Exception: # pylint: disable=broad-except
-            data['content'] = 'Failed to read file :('
-        return data
-
-    def write(self, data):
-        self.monitor.ignore = self.monitor.ignore + 1
-        try:
-            self.filemanager.file_contents = data['content']
-        except Exception: # pylint: disable=broad-except
-            LOG.exception('Error writing file!')
-        del data['content']
-        return data
-
-    def select(self, data):
-        try:
-            self.filemanager.filename = data['filename']
-        except EnvironmentError:
-            LOG.exception('Failed to select file "%s"', data['filename'])
-        return data
-
-    def select_dir(self, data):
-        try:
-            self.filemanager.workdir = data['directory']
-            self.reload_monitor()
-            try:
-                self.filemanager.filename = self.filemanager.files[0]
-                self.read(data)
-            except IndexError:
-                data['content'] = 'No files in this directory :('
-        except EnvironmentError:
-            LOG.error('Failed to select directory "%s"', data['directory'])
-        return data
-
-    def exclude(self, data):
-        try:
-            self.filemanager.exclude = list(data['exclude'])
-        except TypeError:
-            LOG.error('Exclude must be Iterable!')
-        return data
-
-    def attach_fileinfo(self, data):
-        data['filename'] = self.filemanager.filename
-        data['files'] = self.filemanager.files
-        data['directory'] = self.filemanager.workdir
-
-    def handle_event(self, key, message):
-        try:
-            data = message['data']
-            message['data'] = self.commands[data['command']](data)
-            self.attach_fileinfo(data)
-            return message
-        except KeyError:
-            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
-
-    def cleanup(self):
-        self.monitor.stop()

lib/tfw/components/terminado_event_handler.py

@@ -1,44 +0,0 @@
-from tfw.components.terminado_mini_server import TerminadoMiniServer
-from tfw.event_handler_base import TriggerlessEventHandler
-from tfw.config import TFWENV
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class TerminadoEventHandler(TriggerlessEventHandler):
-    def __init__(self, key, monitor):
-        super().__init__(key)
-        self.working_directory = TFWENV.TERMINADO_DIR
-        self._historymonitor = monitor
-        self.terminado_server = TerminadoMiniServer('/terminal', TFWENV.TERMINADO_PORT, TFWENV.TERMINADO_WD, ['bash'])
-        self.commands = {'write': self.write,
-                         'read': self.read}
-        if self._historymonitor:
-            self._historymonitor.watch()
-        self.terminado_server.listen()
-
-    @property
-    def historymonitor(self):
-        return self._historymonitor
-
-    def handle_event(self, key, message):
-        LOG.debug('TerminadoEventHandler received event: %s', message)
-        try:
-            message['data'] = self.commands[message['data']['command']](message['data'])
-            return message
-        except KeyError:
-            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
-
-    def write(self, data):
-        self.terminado_server.pty.write(data['shellcmd'])
-
-    def read(self, data):
-        data['count'] = int(data.get('count', 1))
-        if self.historymonitor:
-            data['history'] = self.historymonitor.history[-data['count']:]
-        return data
-
-    def cleanup(self):
-        if self.historymonitor:
-            self.historymonitor.stop()

lib/tfw/components/terminado_mini_server.py

@@ -1,47 +0,0 @@
-from tornado.ioloop import IOLoop
-from tornado.web import Application
-from terminado import TermSocket, SingleTermManager
-
-from tfw.config import TFWENV
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class TerminadoMiniServer:
-    def __init__(self, url, port, workdir, shellcmd):
-        self.port = port
-        self._term_manager = SingleTermManager(shell_command=shellcmd,
-                                               term_settings={'cwd': workdir})
-        self.application = Application(
-            [(
-                url,
-                TerminadoMiniServer.ResetterTermSocket,
-                {'term_manager': self._term_manager}
-            )]
-        )
-
-    @property
-    def term_manager(self):
-        return self._term_manager
-
-    @property
-    def pty(self):
-        return self.term_manager.terminal.ptyproc
-
-    class ResetterTermSocket(TermSocket): # pylint: disable=abstract-method
-        def check_origin(self, origin):
-            return True
-
-        def on_close(self):
-            self.term_manager.terminal = None
-            self.term_manager.get_terminal()
-
-    def listen(self):
-        self.application.listen(self.port)
-
-
-if __name__ == '__main__':
-    LOG.info('Terminado Mini Server listening on %s', TFWENV.TERMINADO_PORT)
-    TerminadoMiniServer('/terminal', TFWENV.TERMINADO_PORT, TFWENV.TERMINADO_WD, ['bash']).listen()
-    IOLoop.instance().start()

lib/tfw/config/__init__.py

@@ -1 +0,0 @@
-from .envvars import *

lib/tfw/config/envvars.py

@@ -1,3 +0,0 @@
-from envvars import prefixed_envvars_to_namedtuple
-
-TFWENV = prefixed_envvars_to_namedtuple('TFW_', 'tfwenvtuple')

lib/tfw/config/logs.py

@@ -1,3 +0,0 @@
-import logging
-
-logging.basicConfig(level=logging.DEBUG)

lib/tfw/event_handler_base.py

@@ -1,85 +0,0 @@
-from abc import ABC, abstractmethod
-
-from tfw.networking.serialization import deserialize_all
-from tfw.networking.event_handlers.server_connector import ServerConnector
-
-
-class EventHandlerBase(ABC):
-    def __init__(self, key):
-        self.server_connector = ServerConnector()
-        self.key = key
-        self.subscriptions = set()
-        self.subscribe(self.key)
-        self.subscribe('reset')
-        self.server_connector.register_callback(self.event_handler_callback)
-
-    def event_handler_callback(self, msg_parts):
-        key, message = deserialize_all(*msg_parts)
-        response = self.dispatch_handling(key, message)
-        if response is None:
-            return
-        self.server_connector.send(key, response)
-
-    @abstractmethod
-    def dispatch_handling(self, key, message):
-        raise NotImplementedError
-
-    def _dispatch_handling(self, key, message):
-        # pylint: disable=no-else-return
-        if key != 'reset':
-            return self.handle_event(key, message)
-        else:
-            return self.handle_reset(message)
-
-    @abstractmethod
-    def handle_event(self, key, message):
-        raise NotImplementedError
-
-    def handle_reset(self, message):
-        # pylint: disable=unused-argument,no-self-use
-        return None
-
-    def cleanup(self):
-        pass
-
-    def message_other(self, key, data):
-        message = {
-            'key': key,
-            'data': data
-        }
-        self.server_connector.send(key, message)
-
-    def subscribe(self, key):
-        if key not in self.subscriptions:
-            self.subscriptions.add(key)
-            self.server_connector.subscribe(key)
-
-    def unsubscribe(self, key):
-        try:
-            self.subscriptions.remove(key)
-            self.server_connector.unsubscribe(key)
-        except KeyError:
-            pass
-
-    def unsubscribe_all(self):
-        for sub in self.subscriptions:
-            self.server_connector.unsubscribe(key=sub)
-        self.subscriptions.clear()
-
-
-class TriggerlessEventHandler(EventHandlerBase, ABC):
-    # pylint: disable=abstract-method
-    def dispatch_handling(self, key, message):
-        return self._dispatch_handling(key, message)
-
-
-class TriggeredEventHandler(EventHandlerBase, ABC):
-    # pylint: disable=abstract-method
-    def __init__(self, key, trigger):
-        super().__init__(key)
-        self.trigger = trigger
-
-    def dispatch_handling(self, key, message):
-        if message.get('trigger') == self.trigger:
-            return self._dispatch_handling(key, message)
-        return None

lib/tfw/fsm_base.py

@@ -1,26 +0,0 @@
-from typing import List
-
-from transitions import Machine
-
-from tfw.components.mixins.callback_mixin import CallbackMixin
-
-
-class FSMBase(CallbackMixin):
-    states, transitions = [], []
-
-    def __init__(self, initial: str = None, accepted_states: List[str] = None):
-        CallbackMixin.__init__(self)
-        self.accepted_states = accepted_states or [self.states[-1]]
-        self.machine = Machine(model=self,
-                               states=self.states,
-                               transitions=self.transitions,
-                               initial=initial or self.states[0],
-                               send_event=True,
-                               ignore_invalid_triggers=True,
-                               after_state_change='execute_callbacks')
-
-    def execute_callbacks(self, event_data):
-        self._execute_callbacks(event_data.kwargs)
-
-    def is_solved(self):
-        return self.state in self.accepted_states # pylint: disable=no-member

lib/tfw/message_sender.py

@@ -1,18 +0,0 @@
-from datetime import datetime
-
-from tfw.networking.event_handlers.server_connector import ServerUplinkConnector
-
-
-class MessageSender:
-    def __init__(self, custom_key: str = None):
-        self.server_connector = ServerUplinkConnector()
-        self.key = custom_key or 'message'
-
-    def send(self, originator, message):
-        data = {
-            'originator': originator,
-            'timestamp': datetime.now().isoformat(),
-            'message': message
-        }
-        response = {'data': data}
-        self.server_connector.send(self.key, response)

lib/tfw/networking/controller_connector.py

@@ -1,15 +0,0 @@
-import zmq
-from zmq.eventloop.zmqstream import ZMQStream
-
-from tfw.config import TFWENV
-from tfw.networking.zmq_connector_base import ZMQConnectorBase
-
-
-class ControllerConnector(ZMQConnectorBase):
-    def __init__(self, zmq_context=None):
-        super(ControllerConnector, self).__init__(zmq_context)
-        self._zmq_rep_socket = self._zmq_context.socket(zmq.REP)
-        self._zmq_rep_socket.connect('tcp://localhost:{}'.format(TFWENV.CONTROLLER_PORT))
-        self._zmq_rep_stream = ZMQStream(self._zmq_rep_socket)
-
-        self.register_callback = self._zmq_rep_stream.on_recv_stream

lib/tfw/networking/event_handlers/server_connector.py

@@ -1,39 +0,0 @@
-from functools import partial
-
-import zmq
-from zmq.eventloop.zmqstream import ZMQStream
-
-from tfw.networking.serialization import serialize_all
-from tfw.config import TFWENV
-from tfw.networking.zmq_connector_base import ZMQConnectorBase
-
-
-class ServerDownlinkConnector(ZMQConnectorBase):
-    def __init__(self, zmq_context=None):
-        super(ServerDownlinkConnector, self).__init__(zmq_context)
-        self._zmq_sub_socket = self._zmq_context.socket(zmq.SUB)
-        self._zmq_sub_socket.connect('tcp://localhost:{}'.format(TFWENV.PUBLISHER_PORT))
-        self._zmq_sub_stream = ZMQStream(self._zmq_sub_socket)
-
-        self.subscribe = partial(self._zmq_sub_socket.setsockopt_string, zmq.SUBSCRIBE)
-        self.unsubscribe = partial(self._zmq_sub_socket.setsockopt_string, zmq.UNSUBSCRIBE)
-        self.register_callback = self._zmq_sub_stream.on_recv
-
-
-class ServerUplinkConnector(ZMQConnectorBase):
-    def __init__(self, zmq_context=None):
-        super(ServerUplinkConnector, self).__init__(zmq_context)
-        self._zmq_push_socket = self._zmq_context.socket(zmq.PUSH)
-        self._zmq_push_socket.connect('tcp://localhost:{}'.format(TFWENV.RECEIVER_PORT))
-
-    def send_to_eventhandler(self, key, message):
-        message['data']['key'] = key
-        self.send('mirror', message)
-
-    def send(self, key, message):
-        message['key'] = key
-        self._zmq_push_socket.send_multipart(serialize_all(key, message))
-
-
-class ServerConnector(ServerUplinkConnector, ServerDownlinkConnector):
-    pass

lib/tfw/networking/serialization.py

@@ -1,21 +0,0 @@
-import json
-
-
-def encode_if_needed(value):
-    if isinstance(value, str):
-        value = value.encode('utf-8')
-    return value
-
-
-def decode_if_needed(value):
-    if isinstance(value, (bytes, bytearray)):
-        value = value.decode('utf-8')
-    return value
-
-
-def serialize_all(key, data):
-    return [encode_if_needed(frame) for frame in (key, json.dumps(data))]
-
-
-def deserialize_all(key, data):
-    return decode_if_needed(key), json.loads(data)

lib/tfw/networking/server/controller_responder.py

@@ -1,36 +0,0 @@
-from tfw.networking.controller_connector import ControllerConnector
-from tfw.networking.serialization import deserialize_all, serialize_all
-
-
-class ControllerResponder:
-    def __init__(self, fsm):
-        self.fsm = fsm
-        self.token = None
-        self.controller_connector = ControllerConnector()
-        self.controller_connector.register_callback(self.handle_controller_request)
-        self.controller_request_handlers = {
-            'solution_check': self.handle_solution_check_request,
-            'test': self.handle_test_request,
-            'token': self.handle_token_request
-        }
-
-    def handle_controller_request(self, stream, msg_parts):
-        key, data = deserialize_all(*msg_parts)
-        response = self.controller_request_handlers[key](data)
-        stream.send_multipart(serialize_all(self.token, response))
-
-    def handle_test_request(self, data):
-        # pylint: disable=unused-argument,no-self-use
-        return 'OK'
-
-    def handle_token_request(self, data):
-        if self.token is None:
-            self.token = data
-        return {'token': self.token}
-
-    def handle_solution_check_request(self, data):
-        # pylint: disable=unused-argument
-        return {
-            'solved': self.fsm.is_solved(),
-            'message': 'solved' if self.fsm.is_solved() else 'not solved'
-        }

lib/tfw/networking/server/event_handler_connector.py

@@ -1,38 +0,0 @@
-import zmq
-from zmq.eventloop.zmqstream import ZMQStream
-
-from tfw.networking.zmq_connector_base import ZMQConnectorBase
-from tfw.networking.serialization import serialize_all
-from tfw.config import TFWENV
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class EventHandlerDownlinkConnector(ZMQConnectorBase):
-    def __init__(self, zmq_context=None):
-        super(EventHandlerDownlinkConnector, self).__init__(zmq_context)
-        self._zmq_pull_socket = self._zmq_context.socket(zmq.PULL)
-        self._zmq_pull_stream = ZMQStream(self._zmq_pull_socket)
-        address = 'tcp://*:{}'.format(TFWENV.RECEIVER_PORT)
-        self._zmq_pull_socket.bind(address)
-        LOG.debug('Pull socket bound to %s', address)
-
-
-class EventHandlerUplinkConnector(ZMQConnectorBase):
-    def __init__(self, zmq_context=None):
-        super(EventHandlerUplinkConnector, self).__init__(zmq_context)
-        self._zmq_pub_socket = self._zmq_context.socket(zmq.PUB)
-        address = 'tcp://*:{}'.format(TFWENV.PUBLISHER_PORT)
-        self._zmq_pub_socket.bind(address)
-        LOG.debug('Pub socket bound to %s', address)
-
-
-class EventHandlerConnector(EventHandlerDownlinkConnector, EventHandlerUplinkConnector):
-    def register_callback(self, callback):
-        self._zmq_pull_stream.on_recv(callback)
-
-    def send_message(self, message: dict, key: str = None):
-        if not key:
-            key = message.get('key', '')
-        self._zmq_pub_socket.send_multipart(serialize_all(key, message))

lib/tfw/networking/server/tfw_server.py

@@ -1,95 +0,0 @@
-from collections import defaultdict
-
-from tornado.web import Application
-
-from tfw.networking.server.zmq_websocket_handler import ZMQWebSocketProxy
-from tfw.networking.event_handlers.server_connector import ServerUplinkConnector
-from tfw.message_sender import MessageSender
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class TFWServer:
-    def __init__(self, fsm_type):
-        self._fsm = fsm_type()
-        self._fsm_updater = FSMUpdater(self._fsm)
-        self._fsm_manager = FSMManager(self._fsm)
-        self._fsm.subscribe_callback(self._fsm_updater.update)
-
-        self.application = Application(
-            [(r'/ws', ZMQWebSocketProxy, {'make_response': self.make_response,
-                                          'proxy_filter': self.proxy_filter})]
-        )
-        #self.controller_responder = ControllerResponder(self.fsm) TODO: add this once controller stuff is resolved
-
-    @property
-    def fsm(self):
-        return self._fsm
-
-    @property
-    def fsm_manager(self):
-        return self._fsm_manager
-
-    def make_response(self, message):
-        trigger = message.get('trigger', '')
-        try:
-            self._fsm_manager.trigger(trigger, message)
-        except AttributeError:
-            LOG.debug('FSM failed to execute nonexistent trigger: "%s"', trigger)
-        return message
-
-    def proxy_filter(self, message):
-        # pylint: disable=unused-argument,no-self-use
-        return True
-
-    def listen(self, port):
-        self.application.listen(port)
-
-
-class FSMManager:
-    def __init__(self, fsm):
-        self._fsm = fsm
-        self.trigger_predicates = defaultdict(list)
-        self.messenge_sender = MessageSender()
-
-    @property
-    def fsm(self):
-        return self._fsm
-
-    def trigger(self, trigger, message):
-        predicate_results = []
-        for predicate in self.trigger_predicates[trigger]:
-            success, message = predicate()
-            predicate_results.append(success)
-            self.messenge_sender.send('FSM', message)
-
-        if all(predicate_results):
-            try:
-                self.fsm.trigger(trigger, message=message)
-            except AttributeError:
-                LOG.debug('FSM failed to execute nonexistent trigger: "%s"', trigger)
-
-    def subscribe_predicate(self, trigger, *predicates):
-        self.trigger_predicates[trigger].extend(predicates)
-
-    def unsubscribe_predicate(self, trigger, *predicates):
-        self.trigger_predicates[trigger] = [predicate for predicate in self.trigger_predicates[trigger]
-                                            not in predicates]
-
-
-class FSMUpdater:
-    def __init__(self, fsm):
-        self.fsm = fsm
-        self.uplink = ServerUplinkConnector()
-
-    def update(self, kwargs_dict):
-        # pylint: disable=unused-argument
-        self.uplink.send(*self.generate_fsm_update())
-
-    def generate_fsm_update(self):
-        key = 'FSMUpdate'
-        response = {'data': {'current_state': self.fsm.state,
-                             'valid_transitions':
-                                 [{'trigger': trigger} for trigger in self.fsm.machine.get_triggers(self.fsm.state)]}}
-        return key, response

lib/tfw/networking/server/zmq_websocket_handler.py

@@ -1,76 +0,0 @@
-import json
-from abc import ABC, abstractmethod
-
-from tornado.websocket import WebSocketHandler
-
-from tfw.networking.serialization import deserialize_all
-from tfw.networking.server.event_handler_connector import EventHandlerConnector
-from tfw.config.logs import logging
-
-LOG = logging.getLogger(__name__)
-
-
-class ZMQWebSocketHandler(WebSocketHandler, ABC):
-    _event_handler_connector = EventHandlerConnector()
-    instances = set()
-
-    def prepare(self):
-        ZMQWebSocketHandler.instances.add(self)
-
-    def on_close(self):
-        ZMQWebSocketHandler.instances.remove(self)
-
-    def open(self, *args, **kwargs):
-        LOG.debug('WebSocket connection initiated')
-        self._event_handler_connector.register_callback(self.zmq_callback)
-
-    @classmethod
-    def zmq_callback(cls, msg_parts):
-        keyhandlers = {'mirror': cls.mirror}
-
-        key, data = deserialize_all(*msg_parts)
-        LOG.debug('Received on pull socket: %s', data)
-        if key not in keyhandlers:
-            for instance in cls.instances:
-                instance.write_message(data)
-        else:
-            try:
-                keyhandlers[key](data['data'])
-            except KeyError:
-                LOG.error('Invalid mirror message format! Ignoring.')
-
-    @classmethod
-    def mirror(cls, data):
-        key = data['key']
-        cls._event_handler_connector.send_message({'data': data}, key)
-
-    def on_message(self, message):
-        LOG.debug('Received on WebSocket: %s', message)
-        self.send_message(self.make_response(message))
-
-    @abstractmethod
-    def make_response(self, message):
-        raise NotImplementedError
-
-    def send_message(self, message: dict, key: str = None):
-        self._event_handler_connector.send_message(message, key)
-
-    # much secure, very cors, wow
-    def check_origin(self, origin):
-        return True
-
-
-class ZMQWebSocketProxy(ZMQWebSocketHandler):
-    # pylint: disable=abstract-method
-    def initialize(self, make_response, proxy_filter):
-        # pylint: disable=arguments-differ
-        self._make_response = make_response
-        self._proxy_filter = proxy_filter
-
-    def on_message(self, message):
-        message = json.loads(message)
-        if self._proxy_filter(message):
-            super().on_message(message)
-
-    def make_response(self, message):
-        return self._make_response(message)

lib/tfw/networking/zmq_connector_base.py

@@ -1,6 +0,0 @@
-import zmq
-
-
-class ZMQConnectorBase:
-    def __init__(self, zmq_context=None):
-        self._zmq_context = zmq_context or zmq.Context.instance()

nginx/components/login.conf

@@ -1,3 +0,0 @@
-location = /login {
-        proxy_pass http://127.0.0.1:${TFW_LOGIN_APP_PORT};
-    }

nginx/components/terminal.conf

@@ -1,5 +1,5 @@
 location = /terminal {
-        proxy_pass http://127.0.0.1:${TFW_TERMINADO_PORT};
+        proxy_pass http://terminal;
         proxy_http_version 1.1;
         proxy_set_header Upgrade $http_upgrade;
         proxy_set_header Connection "upgrade";

nginx/default.conf

@@ -1,9 +1,15 @@
+upstream terminal {
+    server 127.0.0.1:${TFW_TERMINAL_PORT};
+    server 127.0.0.1:${TFW_TERMINAL_FALLBACK_PORT} backup;
+}
+
 server {
     listen ${TFW_PUBLIC_PORT};
     server_name localhost;
     proxy_connect_timeout 7d;
     proxy_send_timeout 7d;
     proxy_read_timeout 7d;
+    absolute_redirect off;
 
     location = /ws {
         proxy_pass http://127.0.0.1:${TFW_WEB_PORT};

nginx/nginx.conf

@@ -1,13 +1,11 @@
 worker_processes auto;
 pid /tmp/nginx.pid;
 
-events
+events {
-{
     worker_connections 1024;
 }
 
-http
+http {
-{
     sendfile on;
     tcp_nopush on;
     tcp_nodelay on;

pytest.ini

@@ -0,0 +1,4 @@
+[pytest]
+filterwarnings =
+    ignore::DeprecationWarning:zmq
+    ignore::DeprecationWarning:watchdog

requirements.txt

@@ -1,5 +1,15 @@
-tornado==5.0
+tornado>=6.0.0,<7.0.0
-pyzmq==17.0.0
+pyzmq>=17.0.0,<18.0.0
-transitions==0.6.4
+transitions>=0.0.0,<1.0.0
-terminado==0.8.1
+terminado>=0.0.0,<1.0.0
-watchdog==0.8.3
+watchdog>=0.0.0,<1.0.0
+PyYAML>=5.0.0,<6.0.0
+Jinja2>=2.0.0,<3.0.0
+cryptography>=2.0.0,<3.0.0
+python-dateutil>=2.0.0,<3.0.0
+SQLAlchemy>=1.0.0,<2.0.0
+python-dateutil>=2.0.0,<3.0.0
+pytest>=5.0.0,<6.0.0
+pylint>=2.0.0,<3.0.0
+rope>=0.0.0,<1.0.0
+pipe-io-server>=1.0.0,<2.0.0

setup.py

@@ -0,0 +1,29 @@
+from os.path import dirname, realpath, join
+
+from setuptools import setup, find_packages
+
+here = dirname(realpath(__file__))
+
+with open(join(here, 'VERSION'), 'r') as ifile:
+    version = ifile.read().strip('\n')
+with open(join(here, 'requirements.txt'), 'r') as ifile:
+    requirements = ifile.read().splitlines()
+
+setup(
+    name='tfw',
+    version=version,
+    description='Avatao tutorial-framework',
+    url='https://github.com/avatao-content/baseimage-tutorial-framework',
+    author='Avatao.com Innovative Learning Kft.',
+    author_email='support@avatao.com',
+    license='custom',
+    packages=find_packages(),
+    package_dir={'tfw': 'tfw'},
+    install_requires=requirements,
+    extras_require={
+        'docs': [
+            'sphinx >= 1.7.0',
+        ],
+    },
+    zip_safe=False,
+)

supervisor/components/terminal_fallback.conf

@@ -0,0 +1,4 @@
+[program:terminal_fallback]
+command=ncat -klp %(ENV_TFW_TERMINAL_FALLBACK_PORT)s
+autostart=true
+autorestart=true

supervisor/components/tfw_init.conf

@@ -0,0 +1,7 @@
+[program:tfw_init]
+user=root
+directory=/tmp
+command=bash tfw_init.sh
+autorestart=false
+startsecs=0
+priority=1

supervisor/components/tfw_server.conf

@@ -0,0 +1,5 @@
+[program:tfw_server]
+user=root
+directory=%(ENV_TFW_SERVER_DIR)s
+command=python3 -u tfw_server.py
+priority=2

supervisor/supervisord.conf

@@ -1,8 +1,10 @@
 [supervisord]
-user=user
+user=root
-logfile = /tmp/supervisord.log
+logfile=/dev/null
+logfile_maxbytes=0
 loglevel = debug
 pidfile = /tmp/supervisord.pid
+strip_ansi=true
 
 [inet_http_server]
 port = 127.0.0.1:%(ENV_TFW_SUPERVISOR_HTTP_PORT)s
@@ -18,9 +20,5 @@ command=/usr/sbin/nginx -g 'daemon off;'
 autostart=true
 autorestart=true
 
-[program:app]
-directory=%(ENV_TFW_APP_DIR)s
-command=python3 app.py
-
 [include]
 files=%(ENV_TFW_SUPERVISORD_COMPONENTS)s/*.conf

supervisor/tfw_init.sh

@@ -0,0 +1,15 @@
+#!/bin/bash
+set -euo pipefail
+
+echo "export HISTFILE=\"${TFW_HISTFILE}\"" >> /tmp/bashrc &&
+  cat /tmp/bashrc >> "/home/${AVATAO_USER}/.bashrc"
+
+if [[ -z "${HOTRELOAD-}" ]]; then
+  for dir in "${TFW_LIB_DIR}/tfw" "/etc/nginx" "/etc/supervisor"; do
+    chown -R root:root "${dir}" && chmod -R 700 "${dir}";
+  done
+fi
+
+chmod 777 "${TFW_PIPES_DIR}"
+
+rm -f bashrc requirements.txt tfw_init.sh

supervisor/tfw_server.py

@@ -0,0 +1,11 @@
+from tornado.ioloop import IOLoop
+
+from tfw.main import TFWServer, setup_logger, setup_signal_handlers
+
+
+if __name__ == '__main__':
+    setup_logger(__file__)
+    TFWServer().listen()
+
+    setup_signal_handlers()
+    IOLoop.instance().start()

tfw/components/frontend/__init__.py

@@ -0,0 +1,6 @@
+from .console_logs_handler import ConsoleLogsHandler
+from .frontend_proxy_handler import FrontendProxyHandler
+from .frontend_ready_handler import FrontendReadyHandler
+from .message_queue_handler import MessageQueueHandler
+from .message_sender import MessageSender
+from .frontend_config_handler import FrontendConfigHandler

tfw/components/frontend/console_logs_handler.py

@@ -0,0 +1,19 @@
+import logging
+
+LOG = logging.getLogger(__name__)
+
+
+class ConsoleLogsHandler:
+    keys = ['process.log.new']
+
+    def __init__(self, *, stream):
+        self.stream = stream
+
+    def handle_event(self, message, connector):
+        try:
+            connector.send_message({
+                'key': 'console.write',
+                'content': message[self.stream]
+            })
+        except KeyError:
+            LOG.error('Invalid %s message received: %s', self.keys, message)

tfw/components/frontend/frontend_config_handler/__init__.py

@@ -0,0 +1 @@
+from .frontend_config_handler import FrontendConfigHandler

tfw/components/frontend/frontend_config_handler/frontend_config_handler.py

@@ -0,0 +1,36 @@
+from yaml import safe_load
+
+from tfw.internals.networking import Scope
+
+
+class FrontendConfigHandler:
+    keys = ['frontend.ready']
+
+    def __init__(self, config_path):
+        self._config_path = config_path
+
+    def handle_event(self, _, connector):
+        # pylint: disable=no-self-use
+        for message in self._config_messages:
+            connector.send_message(message, scope=Scope.WEBSOCKET)
+        connector.send_message({'key': 'frontend.ready'}, scope=Scope.WEBSOCKET)
+
+    @property
+    def _config_messages(self):
+        config = safe_load(self._config_str)
+        return [
+            self._create_config_message(k, v)
+            for k, v in config.items()
+        ]
+
+    @property
+    def _config_str(self):
+        with open(self._config_path, 'r') as ifile:
+            return ifile.read()
+
+    @staticmethod
+    def _create_config_message(key, value):
+        return {
+            'key': f'frontend.{key}',
+            **value
+        }

tfw/components/frontend/frontend_config_handler/test_frontend_config_handler.py

@@ -0,0 +1,40 @@
+from textwrap import dedent
+
+from .frontend_config_handler import FrontendConfigHandler
+
+
+class MockFrontendConfigHandler(FrontendConfigHandler):
+    @property
+    def _config_str(self):
+        return dedent('''
+        cat:
+            someConfigKey: lel
+            otherStuff:    42
+        foo:
+            hey:           yaaay
+        ''')
+
+
+class MockConnector:
+    def __init__(self):
+        self.messages = []
+
+    def send_message(self, message, **_):
+        self.messages.append(message)
+
+
+def test_frontend_config_handler():
+    connector = MockConnector()
+    handler = MockFrontendConfigHandler('')
+
+    handler.handle_event({'key': 'frontend.ready'}, connector)
+    assert connector.messages[0] == {
+        'key': 'frontend.cat',
+        'someConfigKey': 'lel',
+        'otherStuff': 42
+    }
+    assert connector.messages[1] == {
+        'key': 'frontend.foo',
+        'hey': 'yaaay'
+    }
+    assert connector.messages[-1] == {'key': 'frontend.ready'}

tfw/components/frontend/frontend_proxy_handler.py

@@ -0,0 +1,34 @@
+from tfw.internals.networking import Scope
+
+from .message_storage import FrontendMessageStorage
+
+
+class FrontendProxyHandler:
+    keys = ['console', 'dashboard', 'frontend', 'message', 'ide.read', 'deploy.finish']
+    type_id = 'ControlEventHandler'
+
+    def __init__(self):
+        self.connector = None
+        self._frontend_message_storage = FrontendMessageStorage()
+
+    def send_message(self, message):
+        self.connector.send_message(message, scope=Scope.WEBSOCKET)
+
+    def handle_event(self, message, _):
+        self._frontend_message_storage.save_message(message)
+        if message['key'] == 'frontend.ready':
+            self.recover_frontend()
+            return
+        if self._filter_message(message):
+            self.send_message(message)
+
+    @staticmethod
+    def _filter_message(message):
+        return not message['key'].startswith((
+            'ide',
+            'message.queue'
+        ))
+
+    def recover_frontend(self):
+        for message in self._frontend_message_storage.messages:
+            self.send_message(message)

tfw/components/frontend/frontend_ready_handler.py

@@ -0,0 +1,39 @@
+import logging
+
+from tfw.internals.crypto import KeyManager, sign_message
+
+LOG = logging.getLogger(__name__)
+
+
+class FrontendReadyHandler:
+    keys = ['frontend.ready', 'fsm.update']
+
+    def __init__(self, initial_trigger):
+        self.connector = None
+        self._auth_key = KeyManager().auth_key
+        self.initial_trigger = initial_trigger
+
+        self.commands = {
+            'frontend.ready': self.handle_ready,
+            'fsm.update': self.handle_update
+        }
+
+    def handle_event(self, message, _):
+        try:
+            self.commands[message['key']]()
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_ready(self):
+        trigger = {
+            'key': 'fsm.trigger',
+            'transition': self.initial_trigger
+        }
+        sign_message(self._auth_key, trigger)
+        self.connector.send_message(trigger)
+
+    def handle_update(self):
+        self.stop()
+
+    def stop(self):
+        pass

tfw/components/frontend/message_queue_handler/__init__.py

@@ -0,0 +1 @@
+from .message_queue_handler import MessageQueueHandler

tfw/components/frontend/message_queue_handler/message_queue_handler.py

@@ -0,0 +1,91 @@
+import logging
+from time import sleep, time
+from queue import Queue, Empty
+from threading import Thread
+from contextlib import suppress
+
+LOG = logging.getLogger(__name__)
+
+
+class MessageQueueHandler:
+    keys = ['message.queue']
+    type_id = 'ControlEventHandler'
+    avg_word_len = 5
+    drain_poll_freq = 0.2
+
+    def __init__(self, wpm):
+        self.connector = None
+        self.wpm = wpm
+        self._queue = Queue()
+        self._drain_queue = Queue()
+        self._thread = Thread(target=self._dispatch_messages)
+
+        self._commands = {
+            'message.queue': self.handle_queue,
+            'message.queue.drain': self.handle_drain
+        }
+
+    def _dispatch_messages(self):
+        for message in iter(self._queue.get, None):
+            is_queue_empty = self._queue.empty()
+            message['typing'] = not is_queue_empty
+            self.connector.send_message(message)
+            self._sleep(self._get_sleep_time(message, is_queue_empty))
+
+    def _get_sleep_time(self, message, is_queue_empty):
+        words_per_min = message['wpm'] if 'wpm' in message else self.wpm
+        chars_per_min = self.avg_word_len * words_per_min / 60
+        return 0 if is_queue_empty else len(message['message']) / chars_per_min
+
+    def _sleep(self, seconds):
+        poll_freq = self.drain_poll_freq
+        if seconds < poll_freq:
+            poll_freq = seconds
+
+        sleep_until = time() + seconds
+        while time() < sleep_until:
+            sleep(poll_freq)
+            with suppress(Empty):
+                self._drain_queue.get(block=False)
+                self._drain()
+                break
+
+    def _drain(self):
+        with suppress(Empty):
+            while True:
+                message = self._queue.get(block=False)
+                if message is None:
+                    break
+                message['typing'] = False
+                self.connector.send_message(message)
+
+    def handle_event(self, message, _):
+        try:
+            self._commands[message['key']](message)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_queue(self, message):
+        for unpacked in self._generate_messages_from_queue(message):
+            self._queue.put(unpacked)
+
+    @staticmethod
+    def _generate_messages_from_queue(queue_message):
+        for message in queue_message['messages']:
+            yield {
+                'key': 'message.send',
+                **message
+            }
+
+    def handle_drain(self, _):
+        self._drain_queue.put(True)
+
+    def start(self):
+        self._thread.start()
+
+    def cleanup(self):
+        # clearing the queue forces the loop in
+        # _dispatch_messages to block on _queue.get
+        self._queue.queue.clear()
+        self._queue.put(None)
+        self._thread.join()

tfw/components/frontend/message_queue_handler/test_message_queue.py

@@ -0,0 +1,152 @@
+# pylint: disable=redefined-outer-name
+from time import time
+from os import urandom
+from random import randint
+from queue import Queue, Empty
+
+import pytest
+
+from .message_queue_handler import MessageQueueHandler
+
+
+class MockConnector:
+    def __init__(self):
+        self.callback = None
+        self.messages = Queue()
+        self.send_times = Queue()
+
+    def raise_event(self, message):
+        self.callback(message, self)
+
+    def send_message(self, message):
+        self.messages.put(message)
+        self.send_times.put(time())
+
+
+@pytest.fixture
+def handler():
+    class NoSleepMessageQueueHandler(MessageQueueHandler):
+        sleep_start_times = Queue()
+        sleep_seconds = Queue()
+        sleep_end_times = Queue()
+
+        def _sleep(self, seconds):
+            self.sleep_start_times.put(time())
+            self.sleep_seconds.put(seconds)
+            super()._sleep(seconds)
+            self.sleep_end_times.put(time())
+
+    handler = NoSleepMessageQueueHandler(100000)
+    handler.connector = MockConnector()
+    handler.connector.callback = handler.handle_event
+
+    handler.start()
+    yield handler
+    handler.cleanup()
+
+
+def get_message_queue(*, size=None):
+    size = randint(5, 10) if not size else size
+    return {
+        'key': 'message.queue',
+        'messages': [
+            {'originator': urandom(4).hex(), 'message': urandom(randint(10, 20)).hex()}
+            for _ in range(size)
+        ]
+    }
+
+
+def test_order(handler):
+    queue = get_message_queue()
+    handler.connector.raise_event(queue)
+    expected_messages = queue['messages']
+
+    actual_messages = []
+    for _ in expected_messages:
+        actual_messages.append(handler.connector.messages.get())
+
+    assert len(actual_messages) == len(expected_messages)
+    for i in range(len(expected_messages)):  # pylint: disable=consider-using-enumerate
+        message = actual_messages[i]
+        assert message['key'] == 'message.send'
+        assert message['originator'] == expected_messages[i]['originator']
+        assert message['typing'] == (i < len(expected_messages)-1)
+
+
+def test_timing(handler):
+    q1 = get_message_queue(size=2)
+    q2 = get_message_queue(size=2)
+    handler.connector.raise_event(q1)
+    handler.connector.raise_event(q2)
+
+    messages = []
+    send_times = []
+    sleep_start_times = []
+    sleep_seconds = []
+    sleep_end_times = []
+    for _ in range(len(q1['messages']) + len(q2['messages'])):
+        messages.append(handler.connector.messages.get())
+        send_times.append(handler.connector.send_times.get())
+        sleep_start_times.append(handler.sleep_start_times.get())
+        sleep_seconds.append(handler.sleep_seconds.get())
+        sleep_end_times.append(handler.sleep_end_times.get())
+
+    # no sleep before first message
+    assert sleep_start_times[0] > send_times[0]
+    assert messages[0]['typing']
+
+    # at least 'seconds' sleep before sending next messages
+    assert (send_times[0] + sleep_seconds[0]) < sleep_end_times[0]
+    assert (send_times[0] + sleep_seconds[0]) < send_times[1]
+    assert messages[1]['typing']
+    assert (send_times[1] + sleep_seconds[1]) < sleep_end_times[1]
+    assert (send_times[1] + sleep_seconds[1]) < send_times[2]
+    assert messages[2]['typing']
+
+    # at least 'seconds' sleep after last message
+    assert (send_times[2] + sleep_seconds[2]) < sleep_end_times[3]
+    assert not messages[3]['typing']
+
+
+def test_drain(handler):
+    q1 = get_message_queue()
+    q2 = get_message_queue()
+    messages_count = len(q1['messages']) + len(q2['messages'])
+    handler.wpm = 0.01  # very slow, everything will just block
+    handler.connector.raise_event(q1)
+    handler.connector.raise_event(q2)
+
+    handler.connector.messages.get(timeout=0.1)  # no sleep before first message
+    with pytest.raises(Empty):
+        handler.connector.messages.get(timeout=0.1)
+
+    handler.connector.raise_event({'key': 'message.queue.drain'})
+
+    for _ in range(messages_count - 1):
+        handler.connector.messages.get()
+
+
+def test_queue_works_after_drain(handler):
+    handler.connector.raise_event({'key': 'message.queue.drain'})
+
+    q = get_message_queue()
+    expected_msg_count = len(q['messages'])
+    handler.connector.raise_event(q)
+
+    messages = []
+    for _ in range(expected_msg_count):
+        messages.append(handler.connector.messages.get())
+
+    assert len(messages) == expected_msg_count
+    assert handler._thread.is_alive()  # pylint: disable=protected-access
+
+
+def test_queue_can_be_stopped(handler):
+    handler.wpm = 0.01  # very slow, everything will just block
+
+    q = get_message_queue()
+    handler.connector.raise_event(q)
+    handler.cleanup()
+    handler.connector.raise_event({'key': 'message.queue.drain'})
+
+    assert not handler._thread.is_alive()  # pylint: disable=protected-access

tfw/components/frontend/message_sender.py

@@ -0,0 +1,30 @@
+class MessageSender:
+    def __init__(self, uplink):
+        self.uplink = uplink
+
+    def send(self, message, originator=None):
+        message = {
+            'key': 'message.send',
+            'message': message
+        }
+        if originator:
+            message['originator'] = originator
+        self.uplink.send_message(message)
+
+    def queue_messages(self, messages, originator=None):
+        message_queue = {
+            'key': 'message.queue',
+            'messages': []
+        }
+        for message in messages:
+            next_message = {'message': message}
+            if originator:
+                next_message['originator'] = originator
+            message_queue['messages'].append(next_message)
+        self.uplink.send_message(message_queue)
+
+    def set_originator(self, originator):
+        self.uplink.send_message({
+            'key': 'message.config',
+            'originator': originator
+        })

tfw/components/frontend/message_storage.py

@@ -0,0 +1,49 @@
+from abc import ABC, abstractmethod
+from contextlib import suppress
+
+
+class MessageStorage(ABC):
+    def __init__(self):
+        self._messages = []
+
+    def save_message(self, message):
+        with suppress(KeyError, AttributeError):
+            if self._filter_message(message):
+                self._messages.extend(self._transform_message(message))
+
+    @abstractmethod
+    def _filter_message(self, message):
+        raise NotImplementedError
+
+    def _transform_message(self, message):   # pylint: disable=no-self-use
+        yield message
+
+    def clear(self):
+        self._messages.clear()
+
+    @property
+    def messages(self):
+        yield from self._messages
+
+
+class FrontendMessageStorage(MessageStorage):
+    def _filter_message(self, message):
+        return message['key'].startswith((
+            'console.write',
+            'message.send',
+            'ide.read'
+        ))
+
+    def _transform_message(self, message):
+        transformations = {
+            'ide.read': self._delete_ide_content
+        }
+        if message['key'] in transformations:
+            yield from transformations[message['key']](message)
+        else:
+            yield message
+
+    @staticmethod
+    def _delete_ide_content(message):
+        del message['content']
+        yield message

tfw/components/fsm/__init__.py

@@ -0,0 +1 @@
+from .fsm_handler import FSMHandler

tfw/components/fsm/fsm_handler.py

@@ -0,0 +1,42 @@
+import logging
+
+from tfw.internals.crypto import KeyManager, sign_message
+from tfw.internals.networking import Scope, Intent
+
+from .fsm_updater import FSMUpdater
+
+LOG = logging.getLogger(__name__)
+
+
+class FSMHandler:
+    keys = ['fsm']
+    type_id = 'ControlEventHandler'
+
+    def __init__(self, *, fsm_type):
+        self.connector = None
+        self.fsm = fsm_type()
+        self._fsm_updater = FSMUpdater(self.fsm)
+        self.auth_key = KeyManager().auth_key
+
+        self.command_handlers = {
+            'fsm.trigger': self.handle_trigger,
+            'fsm.update' : self.handle_update
+        }
+
+    def handle_event(self, message, _):
+        try:
+            message = self.command_handlers[message['key']](message)
+            if message:
+                fsm_update_message = self._fsm_updater.fsm_update
+                sign_message(self.auth_key, fsm_update_message)
+                self.connector.send_message(fsm_update_message, Scope.BROADCAST, Intent.EVENT)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_trigger(self, message): # pylint: disable=inconsistent-return-statements
+        if self.fsm.step(message['transition']):
+            return message
+
+    def handle_update(self, message):
+        # pylint: disable=no-self-use
+        return message

tfw/components/fsm/fsm_updater.py

@@ -0,0 +1,26 @@
+class FSMUpdater:
+    def __init__(self, fsm):
+        self.fsm = fsm
+
+    @property
+    def fsm_update(self):
+        return {
+            'key': 'fsm.update',
+            **self.fsm_update_data
+        }
+
+    @property
+    def fsm_update_data(self):
+        valid_transitions = [
+            {'trigger': trigger}
+            for trigger in self.fsm.get_triggers(self.fsm.state)
+        ]
+        if not self.fsm.event_log:
+            return None
+        last_fsm_event = self.fsm.event_log[-1]
+        return {
+            'current_state': self.fsm.state,
+            'valid_transitions': valid_transitions,
+            'in_accepted_state': self.fsm.in_accepted_state,
+            'last_event': last_fsm_event
+        }

tfw/components/ide/__init__.py

@@ -0,0 +1,2 @@
+from .ide_handler import IdeHandler
+from .deploy_handler import DeployHandler

tfw/components/ide/deploy_handler.py

@@ -0,0 +1,34 @@
+import logging
+
+LOG = logging.getLogger(__name__)
+
+
+class DeployHandler:
+    keys = ['deploy.start', 'process.restart']
+
+    def __init__(self, process_name='webservice'):
+        self.connector = None
+        self.process_name = process_name
+
+        self.commands = {
+            'deploy.start': self.handle_deploy,
+            'process.restart': self.handle_process
+        }
+
+    def handle_event(self, message, _):
+        try:
+            self.commands[message['key']](message)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_deploy(self, _):
+        self.connector.send_message({
+            'key': 'process.restart',
+            'name': self.process_name
+        })
+
+    def handle_process(self, message):
+        response = {'key': 'deploy.finish'}
+        if 'error' in message:
+            response['error'] = message['error']
+        self.connector.send_message(response)

tfw/components/ide/file_manager/__init__.py

@@ -0,0 +1 @@
+from .file_manager import FileManager

tfw/components/ide/file_manager/file_manager.py

@@ -0,0 +1,66 @@
+import logging
+from functools import wraps
+from glob import glob
+from fnmatch import fnmatchcase
+from os.path import dirname, isdir, isfile, realpath, isabs
+
+LOG = logging.getLogger(__name__)
+
+
+def _with_is_allowed(func):
+    @wraps(func)
+    def wrapper(self, *args, **kwargs):
+        if self.is_allowed(args[0]):
+            return func(self, *args, **kwargs)
+        raise ValueError('Forbidden path.')
+    return wrapper
+
+
+class FileManager: # pylint: disable=too-many-instance-attributes
+    def __init__(self, patterns):
+        self.patterns = patterns
+
+    @property
+    def files(self):
+        return list(set(
+            path
+            for pattern in self.patterns
+            for path in glob(pattern, recursive=True)
+            if isfile(path) and self.is_allowed(path)
+        ))
+
+    @property
+    def parents(self):
+        return list(set(
+            self._find_directory(pattern)
+            for pattern in self.patterns
+        ))
+
+    @staticmethod
+    def _find_directory(pattern):
+        while pattern and not isdir(pattern):
+            pattern = dirname(pattern)
+        return pattern
+
+    def is_allowed(self, filepath):
+        return any(
+            fnmatchcase(realpath(filepath), pattern)
+            for pattern in self.patterns
+        )
+
+    def find_file(self, filename):
+        if not isabs(filename):
+            for filepath in self.files:
+                if filepath.endswith(filename):
+                    return filepath
+        return filename
+
+    @_with_is_allowed
+    def read_file(self, filepath): # pylint: disable=no-self-use
+        with open(filepath, 'rb', buffering=0) as ifile:
+            return ifile.read().decode(errors='surrogateescape')
+
+    @_with_is_allowed
+    def write_file(self, filepath, contents): # pylint: disable=no-self-use
+        with open(filepath, 'wb', buffering=0) as ofile:
+            ofile.write(contents.encode())

tfw/components/ide/file_manager/test_file_manager.py

@@ -0,0 +1,94 @@
+# pylint: disable=redefined-outer-name
+from dataclasses import dataclass
+from secrets import token_urlsafe
+from os import mkdir, symlink
+from os.path import join, realpath, basename
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import pytest
+
+from .file_manager import FileManager
+
+
+@dataclass
+class ManagerContext:
+    workdir: str
+    subdir: str
+    subfile: str
+    manager: FileManager
+
+    def create_random_file(self, dirname, extension):
+        filename = self.join(f'{dirname}/{generate_name()}{extension}')
+        Path(filename).touch()
+        return filename
+
+    def create_random_folder(self, basepath):
+        dirname = self.join(f'{basepath}/{generate_name()}')
+        mkdir(dirname)
+        return dirname
+
+    def create_random_link(self, source, dirname, extension):
+        linkname = self.join(f'{dirname}/{generate_name()}{extension}')
+        symlink(source, linkname)
+        return linkname
+
+    def join(self, path):
+        return join(self.workdir, path)
+
+def generate_name():
+    return token_urlsafe(16)
+
+@pytest.fixture()
+def context():
+    with TemporaryDirectory() as workdir:
+        workdir = realpath(workdir)  # macOS uses a symlinked TMPDIR
+        subdir = join(workdir, generate_name())
+        subfile = join(subdir, generate_name() + '.txt')
+        mkdir(subdir)
+        Path(subfile).touch()
+        manager = FileManager([join(workdir, '**/*.txt')])
+        yield ManagerContext(workdir, subdir, subfile, manager)
+
+def test_matching_files(context):
+    newdir = context.create_random_folder(context.subdir)
+    newfile = context.create_random_file(newdir, '.txt')
+    newlink = context.create_random_link(newfile, newdir, '.txt')
+    assert set(context.manager.files) == {context.subfile, newfile, newlink}
+
+def test_unmatching_files(context):
+    newtxt = context.create_random_file(context.workdir, '.txt')
+    newbin = context.create_random_file(context.subdir, '.bin')
+    context.create_random_link(newtxt, context.subdir, '.txt')
+    context.create_random_link(newbin, context.subdir, '.txt')
+    assert context.manager.files == [context.subfile]
+
+def test_parents(context):
+    newdir = context.create_random_folder(context.workdir)
+    context.manager.patterns += [f'{newdir}/[!/@]*/**/?.c']
+    assert set(context.manager.parents) == {context.workdir, newdir}
+
+def test_read_write_file(context):
+    for _ in range(128):
+        content = token_urlsafe(32)
+        context.manager.write_file(context.subfile, content)
+        assert context.manager.read_file(context.subfile) == content
+        with open(context.subfile, 'r') as ifile:
+            assert ifile.read() == content
+
+def test_regular_ide_actions(context):
+    newfile1 = context.create_random_file(context.subdir, '.txt')
+    newfile2 = context.create_random_file(context.subdir, '.txt')
+    for _ in range(4):
+        context.manager.filename = newfile1
+        content1, content2 = token_urlsafe(32), token_urlsafe(32)
+        context.manager.write_file(newfile1, content1)
+        context.manager.write_file(newfile2, content2)
+        assert context.manager.read_file(newfile1) == content1
+        assert context.manager.read_file(newfile2) == content2
+
+def test_find_file(context):
+    for _ in range(5):
+        file_abs = context.create_random_file(context.subdir, '.txt')
+        file_base = basename(file_abs)
+        assert context.manager.find_file(file_base) == file_abs

tfw/components/ide/ide_handler.py

@@ -0,0 +1,107 @@
+import logging
+
+from tfw.internals.networking import Scope
+from tfw.internals.inotify import InotifyObserver
+
+from .file_manager import FileManager
+
+LOG = logging.getLogger(__name__)
+
+BUILD_ARTIFACTS = (
+    "*.a",
+    "*.class",
+    "*.dll",
+    "*.dylib",
+    "*.elf",
+    "*.exe",
+    "*.jar",
+    "*.ko",
+    "*.la",
+    "*.lib",
+    "*.lo",
+    "*.o",
+    "*.obj",
+    "*.out",
+    "*.py[cod]",
+    "*.so",
+    "*.so.*",
+    "*.tar.gz",
+    "*.zip",
+    "*__pycache__*"
+)
+
+
+class IdeHandler:
+    keys = ['ide']
+    type_id = 'ControlEventHandler'
+
+    def __init__(self, *, patterns, initial_file=None):
+        self.connector = None
+        self.filemanager = FileManager(patterns)
+        self._initial_file = initial_file or ''
+        self._ignore_inotify_src = ''
+
+        self.monitor = InotifyObserver(
+            path=self.filemanager.parents,
+            exclude=BUILD_ARTIFACTS
+        )
+        self.monitor.on_modified = self._reload_frontend
+        self.monitor.start()
+
+        self.commands = {
+            'ide.read'  : self.read,
+            'ide.write' : self.write
+        }
+
+    def _reload_frontend(self, event):  # pylint: disable=unused-argument
+        if self._ignore_inotify_src == event.src_path:
+            self._ignore_inotify_src = ''
+            return
+        self.send_message({'key': 'ide.reload'})
+
+    @property
+    def initial_file(self):
+        if not self.filemanager.is_allowed(self._initial_file):
+            self._initial_file = self.filemanager.files[0]
+        return self._initial_file
+
+    def send_message(self, message):
+        self.connector.send_message(message, scope=Scope.WEBSOCKET)
+
+    def handle_event(self, message, _):
+        try:
+            self.commands[message['key']](message)
+            message['files'] = self.filemanager.files
+            self.send_message(message)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def read(self, message):
+        if 'patterns' in message:
+            self.filemanager.patterns = message['patterns']
+            self.monitor.paths = self.filemanager.parents
+        try:
+            message['filename'] = self.filemanager.find_file(
+                message.get('filename') or self.initial_file
+            )
+            message['content'] = self.filemanager.read_file(message['filename'])
+        except (PermissionError, ValueError):
+            message['content'] = 'You have no permission to open that file :('
+        except FileNotFoundError:
+            message['content'] = 'This file does not exist :('
+        except Exception: # pylint: disable=broad-except
+            message['content'] = 'Failed to read file :('
+            LOG.exception('Error reading file!')
+
+    def write(self, message):
+        try:
+            self._ignore_inotify_src = message['filename']
+            self.filemanager.write_file(message['filename'], message['content'])
+        except KeyError:
+            LOG.error('You must provide a filename to write!')
+        except Exception: # pylint: disable=broad-except
+            LOG.exception('Error writing file!')
+        del message['content']
+
+    def cleanup(self):
+        self.monitor.stop()

tfw/components/pipe_io/__init__.py

@@ -0,0 +1,3 @@
+from .pipe_io_handler import PipeIOHandler, PipeIOHandlerBase
+from .command_handler import CommandHandler
+from .pipe_connector import ProxyPipeConnectorHandler

tfw/components/pipe_io/command_handler.py

@@ -0,0 +1,58 @@
+from subprocess import PIPE, Popen
+from functools import partial
+from os import getpgid, killpg
+from os.path import join
+from signal import SIGTERM
+from secrets import token_urlsafe
+from threading import Thread
+from contextlib import suppress
+
+from pipe_io_server import terminate_process_on_failure
+
+from .pipe_io_handler import PipeIOHandler, DEFAULT_PERMISSIONS
+
+
+class CommandHandler(PipeIOHandler):
+    def __init__(self, command, permissions=DEFAULT_PERMISSIONS):
+        super().__init__(
+            self._generate_tempfilename(),
+            self._generate_tempfilename(),
+            permissions
+        )
+
+        self._proc_stdin = open(self.pipe_io.out_pipe, 'rb')
+        self._proc_stdout = open(self.pipe_io.in_pipe, 'wb')
+        self._proc = Popen(
+            command, shell=True, executable='/bin/bash',
+            stdin=self._proc_stdin, stdout=self._proc_stdout, stderr=PIPE,
+            start_new_session=True
+        )
+        self._monitor_proc_thread = self._start_monitor_proc()
+
+    def _generate_tempfilename(self):
+        # pylint: disable=no-self-use
+        random_filename = partial(token_urlsafe, 10)
+        return join('/tmp', f'{type(self).__name__}.{random_filename()}')
+
+    def _start_monitor_proc(self):
+        thread = Thread(target=self._monitor_proc, daemon=True)
+        thread.start()
+        return thread
+
+    @terminate_process_on_failure
+    def _monitor_proc(self):
+        return_code = self._proc.wait()
+        if return_code == -int(SIGTERM):
+            # supervisord asked the program to terminate, this is fine
+            return
+        if return_code != 0:
+            _, stderr = self._proc.communicate()
+            raise RuntimeError(f'Subprocess failed ({return_code})! Stderr:\n{stderr.decode()}')
+
+    def cleanup(self):
+        with suppress(ProcessLookupError):
+            process_group_id = getpgid(self._proc.pid)
+            killpg(process_group_id, SIGTERM)
+            self._proc_stdin.close()
+            self._proc_stdout.close()
+        super().cleanup()

tfw/components/pipe_io/pipe_connector/__init__.py

@@ -0,0 +1 @@
+from .proxy_pipe_connector_handler import ProxyPipeConnectorHandler

tfw/components/pipe_io/pipe_connector/pipe_connector.py

@@ -0,0 +1,90 @@
+import logging
+from stat import S_ISFIFO
+from os import access, listdir, mkdir, stat, R_OK
+from os.path import exists, join
+
+from pipe_io_server import PipeWriterServer
+
+from tfw.internals.inotify import InotifyObserver, InotifyFileCreatedEvent, InotifyFileDeletedEvent
+
+LOG = logging.getLogger(__name__)
+
+
+class PipeConnector:
+    reader_pattern, writer_pattern = 'send', 'recv'
+
+    def __init__(self, path):
+        self.recv_pipes, self.send_pipes = {}, {}
+        self.observer = self._build_observer(path)
+        self.observer.on_any_event = self._on_any_event
+        self.observer.start()
+        self._connect_existing_pipes(path)
+
+    def _build_observer(self, path): # pylint: disable=no-self-use
+        if not exists(path):
+            mkdir(path)
+        if not access(path, R_OK):
+            raise ValueError('Path does not exist or is not accessible.')
+        return InotifyObserver(path, patterns=[f'*{self.reader_pattern}*', f'*{self.writer_pattern}*'])
+
+    def _connect_existing_pipes(self, path):
+        for node in listdir(path):
+            node_path = join(path, node)
+            if self._is_pipe(node_path):
+                self._create_pipe(node_path)
+                LOG.debug('Connected to existing pipe "%s"', node_path)
+
+    def _on_any_event(self, event):
+        path = event.src_path
+        if self._is_pipe(path) and isinstance(event, InotifyFileCreatedEvent):
+            self._create_pipe(path)
+            LOG.debug('Connected to new pipe "%s"', path)
+        elif isinstance(event, InotifyFileDeletedEvent):
+            self._delete_pipe(path)
+            LOG.debug('Disconnected from deleted pipe "%s"', path)
+
+    @staticmethod
+    def _is_pipe(path):
+        return exists(path) and S_ISFIFO(stat(path).st_mode)
+
+    def _create_pipe(self, path):
+        if self._find_pipe(path):
+            return
+        server = None
+        if self.writer_pattern in path:
+            pipes, server = self.recv_pipes, self.build_writer(path)
+        elif self.reader_pattern in path:
+            pipes, server = self.send_pipes, self.build_reader(path)
+        if server:
+            server.start()
+            pipes[path] = server
+
+    def _find_pipe(self, path):
+        pipes = None
+        if path in self.recv_pipes.keys():
+            pipes = self.recv_pipes
+        if path in self.send_pipes.keys():
+            pipes = self.send_pipes
+        return pipes
+
+    def build_reader(self, path): # pylint: disable=no-self-use
+        raise NotImplementedError()
+
+    def build_writer(self, path): # pylint: disable=no-self-use
+        return PipeWriterServer(path, manage_pipes=False)
+
+    def _delete_pipe(self, path):
+        pipes = self._find_pipe(path)
+        if pipes:
+            pipes[path].stop()
+            del pipes[path]
+
+    def broadcast(self, message):
+        for server in self.recv_pipes.values():
+            server.send_message(message)
+
+    def stop(self):
+        for pipe in self.recv_pipes.values():
+            pipe.stop()
+        for pipe in self.send_pipes.values():
+            pipe.stop()

tfw/components/pipe_io/pipe_connector/proxy_pipe_connector_handler.py

@@ -0,0 +1,46 @@
+import logging
+from threading import Lock
+from json import dumps, loads, JSONDecodeError
+
+from pipe_io_server import PipeReaderServer
+
+from .pipe_connector import PipeConnector
+
+LOG = logging.getLogger(__name__)
+
+
+class ProxyPipeConnectorHandler:
+    keys = ['']
+
+    def __init__(self, path):
+        self.connector, self.pipes = None, None
+        self.path = path
+
+    def start(self):
+        self.pipes = ProxyPipeConnector(self.path, self.connector)
+
+    def handle_event(self, message, _):
+        self.pipes.broadcast(dumps(message).encode())
+
+    def cleanup(self):
+        self.pipes.stop()
+
+
+class ProxyPipeConnector(PipeConnector):
+    def __init__(self, path, connector):
+        self.connector = connector
+        self.mutex = Lock()
+        super().__init__(path)
+
+    def build_reader(self, path):
+        reader = PipeReaderServer(path, manage_pipes=False)
+        reader.handle_message = self._handle_message
+        return reader
+
+    def _handle_message(self, message):
+        try:
+            json_object = loads(message)
+            with self.mutex:
+                self.connector.send_message(json_object)
+        except JSONDecodeError:
+            LOG.error('Received invalid JSON message: %s', message)

tfw/components/pipe_io/pipe_connector/test_proxy_pipe_connector.py

@@ -0,0 +1,157 @@
+# pylint: disable=redefined-outer-name
+from enum import Enum
+from dataclasses import dataclass
+from json import dumps
+from secrets import token_urlsafe
+from os import urandom, mkfifo, mkdir, remove
+from os.path import join
+from pathlib import Path
+from tempfile import TemporaryDirectory
+
+import pytest
+from tfw.internals.inotify import InotifyFileCreatedEvent, InotifyFileDeletedEvent
+
+from .proxy_pipe_connector_handler import ProxyPipeConnector
+
+
+class Action(Enum):
+    SEND = 'send'
+    RECV = 'recv'
+
+
+@dataclass
+class PipeContext:
+    workdir: str
+    pipes: ProxyPipeConnector
+
+    def emit_pipe_creation_event(self, action, inode_creator):
+        filename = self.join(f'{self.generate_name()}_{action.value}')
+        inode_creator(filename)
+        self.pipes.observer.on_any_event(InotifyFileCreatedEvent(filename))
+        return filename
+
+    def join(self, path):
+        return join(self.workdir, path)
+
+    @staticmethod
+    def generate_name():
+        return urandom(4).hex()
+
+
+class MockPipeConnector(ProxyPipeConnector):
+    def __init__(self, path, connector):
+        self.reader_events, self.writer_events = [], []
+        super().__init__(path, connector)
+
+    def _build_observer(self, path):
+        return MockObserver()
+
+    def build_reader(self, path):
+        self.reader_events.append(path)
+        reader = MockPipeServer()
+        reader.handle_message = self._handle_message
+        return reader
+
+    def build_writer(self, path):
+        self.writer_events.append(path)
+        return MockPipeServer()
+
+
+class MockObserver:
+    def start(self):
+        pass
+
+    def on_any_event(self, event):
+        pass
+
+
+class MockPipeServer:
+    def __init__(self):
+        self.messages = []
+
+    def handle_message(self, message):
+        pass
+
+    def send_message(self, message):
+        self.messages.append(message)
+
+    def start(self):
+        pass
+
+    def stop(self):
+        pass
+
+
+class MockConnector: # pylint: disable=too-few-public-methods
+    def __init__(self):
+        self.messages = []
+
+    def send_message(self, message):
+        self.messages.append(message)
+
+
+@pytest.fixture
+def workdir():
+    with TemporaryDirectory() as workdir:
+        yield workdir
+
+
+@pytest.fixture
+def context(workdir):
+    mkfifo(join(workdir, Action.SEND.value))
+    mkfifo(join(workdir, Action.RECV.value))
+    yield PipeContext(workdir, MockPipeConnector(workdir, MockConnector()))
+
+
+def test_existing_pipe_connection(context):
+    assert join(context.workdir, Action.SEND.value) in context.pipes.send_pipes.keys()
+    assert join(context.workdir, Action.RECV.value) in context.pipes.recv_pipes.keys()
+
+
+def test_pipe_creation_deletion(context):
+    cases = [
+        (Action.RECV, context.pipes.recv_pipes, context.pipes.writer_events),
+        (Action.SEND, context.pipes.send_pipes, context.pipes.reader_events)
+    ]
+
+    for action, pipes, events in cases:
+        path = context.emit_pipe_creation_event(action, mkfifo)
+        assert events[-1] == path
+        assert path in pipes.keys()
+        remove(path)
+        context.pipes.observer.on_any_event(InotifyFileDeletedEvent(path))
+        assert path not in pipes.keys()
+
+
+def test_handle_message(context):
+    path = context.emit_pipe_creation_event(Action.SEND, mkfifo)
+    payload = {'key': token_urlsafe(16)}
+    context.pipes.send_pipes[path].handle_message(dumps(payload))
+    assert context.pipes.connector.messages[-1] == payload
+    context.pipes.send_pipes[path].handle_message(token_urlsafe(32))
+    assert len(context.pipes.connector.messages) == 1
+
+
+def test_broadcast(context):
+    paths = [
+        context.emit_pipe_creation_event(Action.RECV, mkfifo)
+        for _ in range(32)
+    ]
+    payload = {'key': token_urlsafe(16)}
+
+    context.pipes.broadcast(payload)
+    for path in paths:
+        assert context.pipes.recv_pipes[path].messages[-1] == payload
+
+def test_inode_types(context):
+    touch = lambda path: Path(path).touch()
+    cases = [
+        (Action.RECV, context.pipes.recv_pipes, mkdir),
+        (Action.SEND, context.pipes.send_pipes, mkdir),
+        (Action.RECV, context.pipes.recv_pipes, touch),
+        (Action.SEND, context.pipes.send_pipes, touch)
+    ]
+
+    for action, pipes, creator in cases:
+        path = context.emit_pipe_creation_event(action, creator)
+        assert path not in pipes.keys()

tfw/components/pipe_io/pipe_io_handler.py

@@ -0,0 +1,47 @@
+import logging
+from abc import abstractmethod
+from json import loads, dumps
+
+from pipe_io_server import PipeIOServer
+
+LOG = logging.getLogger(__name__)
+DEFAULT_PERMISSIONS = 0o600
+
+
+class PipeIOHandlerBase:
+    keys = ['']
+
+    def __init__(self, in_pipe_path, out_pipe_path, permissions=DEFAULT_PERMISSIONS):
+        self.connector = None
+        self.in_pipe = in_pipe_path
+        self.out_pipe = out_pipe_path
+        self.pipe_io = PipeIOServer(
+            in_pipe_path,
+            out_pipe_path,
+            permissions
+        )
+        self.pipe_io.handle_message = self._server_handle_message
+        self.pipe_io.start()
+
+    def _server_handle_message(self, message):
+        try:
+            self.handle_pipe_event(message)
+        except:  # pylint: disable=bare-except
+            LOG.exception('Failed to handle message %s from pipe %s!', message, self.in_pipe)
+
+    @abstractmethod
+    def handle_pipe_event(self, message_bytes):
+        raise NotImplementedError()
+
+    def cleanup(self):
+        self.pipe_io.stop()
+
+
+class PipeIOHandler(PipeIOHandlerBase):
+    def handle_event(self, message, _):
+        json_bytes = dumps(message).encode()
+        self.pipe_io.send_message(json_bytes)
+
+    def handle_pipe_event(self, message_bytes):
+        json = loads(message_bytes)
+        self.connector.send_message(json)

tfw/components/pipe_io/test_pipe_io_handler.py

@@ -0,0 +1,45 @@
+# pylint: disable=redefined-outer-name
+from tempfile import TemporaryDirectory
+from os.path import join
+from queue import Queue
+from json import dumps, loads
+
+import pytest
+
+from .pipe_io_handler import PipeIOHandler
+
+
+@pytest.fixture
+def pipe_io():
+    with TemporaryDirectory() as tmpdir:
+        pipeio = PipeIOHandler(
+            join(tmpdir, 'in'),
+            join(tmpdir, 'out'),
+            permissions=0o600
+        )
+        pipeio.connector = MockConnector()
+        yield pipeio
+        pipeio.cleanup()
+
+
+class MockConnector:
+    def __init__(self):
+        self.messages = Queue()
+
+    def send_message(self, msg):
+        self.messages.put(msg)
+
+
+def test_pipe_io_handler_recv(pipe_io):
+    test_msg = {'key': 'cica'}
+    with open(pipe_io.in_pipe, 'w') as ofile:
+        ofile.write(dumps(test_msg) + '\n')
+
+    assert pipe_io.connector.messages.get() == test_msg
+
+
+def test_pipe_io_handler_send(pipe_io):
+    test_msg = {'key': 'cica'}
+    pipe_io.handle_event(test_msg, None)
+    with open(pipe_io.out_pipe, 'r') as ifile:
+        assert loads(ifile.readline()) == test_msg

tfw/components/process_management/__init__.py

@@ -0,0 +1,2 @@
+from .process_handler import ProcessHandler
+from .process_log_handler import ProcessLogHandler

tfw/components/process_management/log_inotify_observer.py

@@ -0,0 +1,34 @@
+from tfw.internals.networking import Scope, Intent
+from tfw.internals.inotify import InotifyObserver
+
+from .supervisor import ProcessLogManager
+
+
+class LogInotifyObserver(InotifyObserver, ProcessLogManager):
+    def __init__(self, connector, process_name, supervisor_uri, log_tail=0):
+        self._connector = connector
+        self._process_name = process_name
+        self.log_tail = log_tail
+        self._procinfo = None
+        ProcessLogManager.__init__(self, supervisor_uri)
+        InotifyObserver.__init__(self, self._get_logfiles())
+
+    def _get_logfiles(self):
+        self._procinfo = self.supervisor.getProcessInfo(self._process_name)
+        return self._procinfo['stdout_logfile'], self._procinfo['stderr_logfile']
+
+    @property
+    def process_name(self):
+        return self._process_name
+
+    @process_name.setter
+    def process_name(self, process_name):
+        self._process_name = process_name
+        self.paths = self._get_logfiles()
+
+    def on_modified(self, event):
+        self._connector.send_message({
+            'key': 'process.log.new',
+            'stdout': self.read_stdout(self.process_name, tail=self.log_tail),
+            'stderr': self.read_stderr(self.process_name, tail=self.log_tail)
+        }, Scope.BROADCAST, Intent.EVENT)

tfw/components/process_management/process_handler.py

@@ -0,0 +1,33 @@
+import logging
+from xmlrpc.client import Fault as SupervisorFault
+
+from tfw.internals.networking import Scope, Intent
+
+from .supervisor import ProcessManager
+
+LOG = logging.getLogger(__name__)
+
+
+class ProcessHandler(ProcessManager):
+    keys = ['process']
+    type_id = 'ControlEventHandler'
+
+    def __init__(self, *, supervisor_uri):
+        ProcessManager.__init__(self, supervisor_uri)
+
+        self.commands = {
+            'process.start': self.start_process,
+            'process.stop': self.stop_process,
+            'process.restart': self.restart_process
+        }
+
+    def handle_event(self, message, connector):
+        try:
+            try:
+                self.commands[message['key']](message['name'])
+            except SupervisorFault as fault:
+                message['error'] = fault.faultString
+            connector.send_message(message, scope=Scope.BROADCAST, intent=Intent.EVENT)
+        except KeyError:
+            if not message['key'].startswith('process.log'):
+                LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)

tfw/components/process_management/process_log_handler.py

@@ -0,0 +1,58 @@
+import logging
+
+from .log_inotify_observer import LogInotifyObserver
+
+LOG = logging.getLogger(__name__)
+
+
+class ProcessLogHandler:
+    keys = ['process.log']
+    type_id = 'ControlEventHandler'
+
+    def __init__(self, *, process_name, supervisor_uri, log_tail=0):
+        self.connector, self._monitor = None, None
+        self.process_name = process_name
+        self._supervisor_uri = supervisor_uri
+        self._initial_log_tail = log_tail
+
+        self.command_handlers = {
+            'process.log.set': self.handle_set,
+            'process.log.start': self.handle_start,
+            'process.log.stop': self.handle_stop
+        }
+
+    def start(self):
+        if not self._monitor:
+            self._monitor = LogInotifyObserver(
+                connector=self.connector,
+                process_name=self.process_name,
+                supervisor_uri=self._supervisor_uri,
+                log_tail=self._initial_log_tail
+            )
+            self._monitor.start()
+
+    def handle_event(self, message, _):
+        try:
+            self.command_handlers[message['key']](message)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_set(self, data):
+        if data.get('name'):
+            self._monitor.process_name = data['name']
+        if data.get('tail'):
+            self._monitor.log_tail = data['tail']
+
+    def handle_start(self, _):
+        self.start()
+
+    def handle_stop(self, _):
+        self._stop_monitor()
+
+    def _stop_monitor(self):
+        if self._monitor:
+            self._monitor.stop()
+            self._monitor = None
+
+    def cleanup(self):
+        self._stop_monitor()

tfw/components/process_management/supervisor.py

@@ -0,0 +1,36 @@
+from os import remove
+from contextlib import suppress
+import xmlrpc.client
+from xmlrpc.client import Fault as SupervisorFault
+
+
+class SupervisorBase:
+    def __init__(self, supervisor_uri):
+        self.supervisor = xmlrpc.client.ServerProxy(supervisor_uri).supervisor
+
+
+class ProcessManager(SupervisorBase):
+    def stop_process(self, process_name):
+        with suppress(SupervisorFault):
+            self.supervisor.stopProcess(process_name)
+
+    def start_process(self, process_name):
+        self.supervisor.startProcess(process_name)
+
+    def restart_process(self, process_name):
+        self.stop_process(process_name)
+        self.start_process(process_name)
+
+
+class ProcessLogManager(SupervisorBase):
+    def read_stdout(self, process_name, tail=0):
+        return self.supervisor.readProcessStdoutLog(process_name, -tail, 0)
+
+    def read_stderr(self, process_name, tail=0):
+        return self.supervisor.readProcessStderrLog(process_name, -tail, 0)
+
+    def clear_logs(self, process_name):
+        for logfile in ('stdout_logfile', 'stderr_logfile'):
+            with suppress(FileNotFoundError):
+                remove(self.supervisor.getProcessInfo(process_name)[logfile])
+        self.supervisor.clearProcessLogs(process_name)

tfw/components/snapshots/__init__.py

@@ -0,0 +1 @@
+from .snapshot_handler import SnapshotHandler

tfw/components/snapshots/snapshot_handler.py

@@ -0,0 +1,78 @@
+import logging
+from os.path import join as joinpath
+from os.path import basename
+from os import makedirs
+from datetime import datetime
+
+from dateutil import parser as dateparser
+
+from .snapshot_provider import SnapshotProvider
+
+LOG = logging.getLogger(__name__)
+
+
+class SnapshotHandler:
+    keys = ['snapshot']
+
+    def __init__(self, *, directories, snapshots_dir, exclude_unix_patterns=None):
+        self._snapshots_dir = snapshots_dir
+        self.snapshot_providers = {}
+        self._exclude_unix_patterns = exclude_unix_patterns
+        self.init_snapshot_providers(directories)
+
+        self.command_handlers = {
+            'snapshot.take': self.handle_take_snapshot,
+            'snapshot.restore': self.handle_restore_snapshot,
+            'snapshot.exclude': self.handle_exclude
+        }
+
+    def init_snapshot_providers(self, directories):
+        for index, directory in enumerate(directories):
+            git_dir = self.init_git_dir(index, directory)
+            self.snapshot_providers[directory] = SnapshotProvider(
+                directory,
+                git_dir,
+                self._exclude_unix_patterns
+            )
+
+    def init_git_dir(self, index, directory):
+        git_dir = joinpath(
+            self._snapshots_dir,
+            f'{basename(directory)}-{index}'
+        )
+        makedirs(git_dir, exist_ok=True)
+        return git_dir
+
+    def handle_event(self, message, _):
+        try:
+            self.command_handlers[message['key']](message)
+        except KeyError:
+            LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
+
+    def handle_take_snapshot(self, _):
+        LOG.debug('Taking snapshots of directories %s', self.snapshot_providers.keys())
+        for provider in self.snapshot_providers.values():
+            provider.take_snapshot()
+
+    def handle_restore_snapshot(self, message):
+        date = dateparser.parse(
+            message.get(
+                'value',
+                datetime.now().isoformat()
+            )
+        )
+        LOG.debug(
+            'Restoring snapshots (@ %s) of directories %s',
+            date,
+            self.snapshot_providers.keys()
+        )
+        for provider in self.snapshot_providers.values():
+            provider.restore_snapshot(date)
+
+    def handle_exclude(self, message):
+        exclude_unix_patterns = message['value']
+        if not isinstance(exclude_unix_patterns, list):
+            raise KeyError
+
+        for provider in self.snapshot_providers.values():
+            provider.exclude = exclude_unix_patterns