Compare commits

...
Author SHA1 Message Date
Kristóf Tóth 5b95f1d4b4 License project under GNU LGPLv3 2019-09-12 13:43:23 -04:00
Kristóf Tóth 983f02f362 Merge branch 'chausie' 2019-09-12 13:42:07 -04:00
Kristóf Tóth 70cd4c51cb Use new frontend image tag from DockerHub 2019-09-12 13:41:26 -04:00
R. Richard 56620c4b5f Copy frontend as a multistage artifact 2019-09-12 13:41:26 -04:00
Kristóf Tóth 6c5699cff9 Make unused but overridden public method protected 2019-09-12 02:29:10 +02:00
Kristóf Tóth aa1112af0e Refactor connecting existing pipes to a method 2019-09-12 02:29:10 +02:00
R. Richard d6f2eb987f Make PipeConnector stop gracefully and connect to every pipe on init 2019-09-12 02:29:10 +02:00
Kristóf Tóth 5b0f79dbae Move CommandHandler to a separate file 2019-09-05 16:10:46 +02:00
Kristóf Tóth 8890e67b86 Implement test cases for PipeIOHandler 2019-09-05 15:52:34 +02:00
Kristóf Tóth 89b720c439 Add forgotten test case 2019-09-04 18:18:03 +02:00
Kristóf Tóth 613d906b44 Simplify PipeIOHandlerBase by using meghod assignment 2019-09-04 18:07:36 +02:00
Kristóf Tóth b733f53b83 Remove unused but beautiful product of me being with my muse :( 2019-09-04 17:30:06 +02:00
Kristóf Tóth 10cd07973d Implement unit tests for EventHandler subclasses 2019-09-04 17:13:58 +02:00
Kristóf Tóth 620212f00f Accept relative paths in the IdeHandler API's "filename" key 2019-09-04 15:13:40 +02:00
Kristóf Tóth 66c9c5a592 Implement finding files from relative paths FileManager 2019-09-04 15:13:11 +02:00
Kristóf Tóth 777dc9ccfc Remove unnecessary pylint: disable comment 2019-09-02 16:25:04 +02:00
Kristóf Tóth fab582505f Improve IdeHandler JSON API 2019-09-02 16:21:19 +02:00
Kristóf Tóth 701b918e6f Refactor IdeHandler 2019-09-02 15:49:38 +02:00
Kristóf Tóth 1d8de09c2a Make Filemanager.is_allowed() public 2019-09-02 15:49:13 +02:00
R. Richard 57ce35d99d Fix pipe deletion check 2019-09-02 13:55:40 +02:00
Kristóf Tóth 517684e84a Fix TerminalCommandsHandler using old JSON API 2019-08-30 17:47:58 +02:00
Kristóf Tóth a5db2f23a0 Log PipeConnector events 2019-08-30 17:45:54 +02:00
Kristóf Tóth bb5c000b1d Implement builtin handler for configuring frontend 2019-08-30 17:37:43 +02:00
Kristóf Tóth ffc06a64ee Clarify exception message in EHFactoryBase 2019-08-30 15:28:11 +02:00
Kristóf Tóth 9712ebf102 Specify EventHandler types in builtin handlers 2019-08-30 14:45:53 +02:00
Kristóf Tóth 25cf672231 Enable handlers to specify EventHandler type 2019-08-30 14:44:57 +02:00
Kristóf Tóth 11bb1e492a Add get_type() method to TypeIdRegistryMixin 2019-08-30 14:44:26 +02:00
Kristóf Tóth d9bcfb3705 Add metaclass capable of building instances from class name string 2019-08-29 12:03:14 +02:00
Kristóf Tóth 80eca7e322 Fix leftover string from old API 2019-08-28 16:46:05 +02:00
Kristóf Tóth 016130b081 Fix pylint issues 2019-08-28 16:40:23 +02:00
Kristóf Tóth c1df007d6c Fix refactored method name 2019-08-28 16:34:55 +02:00
R. Richard 25f54a71b5 Refactor API message format 2019-08-28 16:31:43 +02:00
Kristóf Tóth 996e8e2af7 Remove unused keys from FrontendMessageStore 2019-08-28 15:29:20 +02:00
Kristóf Tóth 3c84d2f3aa Avoid replaying "frontend.ready" messages 2019-08-28 15:04:58 +02:00
Kristóf Tóth 0295baad47 Fix bug where message scope would be replaced with default scope 2019-08-28 14:56:59 +02:00
Kristóf Tóth 13e247b009 Proxy all keys prefixed with 'frontend' to frontend 2019-08-28 14:56:31 +02:00
R. Richard ddb47d696c Trigger first step in an individual event handler 2019-08-27 13:34:33 +02:00
R. Richard 1f813a2138 Add minimal init system to container 2019-08-27 13:34:33 +02:00
R. Richard 947d0a27b7 History monitor should only emit events 2019-08-26 14:53:52 +02:00
R. Richard 0ea6188a05 Make WPM customization available per-message basis 2019-08-26 14:18:40 +02:00
R. Richard af3781adb3 Fix issues in PR #67 2019-08-26 13:54:26 +02:00
R. Richard 7a14928432 Refactor deploy handler 2019-08-26 13:54:26 +02:00
R. Richard c5afdfb1b2 Create handler for deploying in the IDE 2019-08-26 13:54:26 +02:00
R. Richard 0f379891e3 Move message queue logic to backend 2019-08-26 13:54:26 +02:00
R. Richard 965973a32f Fix issues in PR #64 2019-08-26 11:12:13 +02:00
R. Richard 2e5867cc49 Introduce intent for TFW messages and fix FSM related bugs 2019-08-26 11:12:13 +02:00
Kristóf Tóth f626fef8f8 Open up TFW pipes 2019-08-15 17:41:05 +02:00
Kristóf Tóth 2e9a84bc9b Fix init script without HOTRELOAD 2019-08-15 17:26:23 +02:00
R. Richard 75a9fce47c Modify resources' directory structure 2019-08-15 17:13:36 +02:00
R. Richard 8bde319159 Add underscore to process name 2019-08-15 17:13:06 +02:00
R. Richard e0a915ac55 Move initial FSM trigger to backend 2019-08-15 15:38:40 +02:00
Kristóf Tóth c753b63273 Merge branch 'processlog-rework' into chausie 2019-08-15 14:40:02 +02:00
Kristóf Tóth c84ac99141 Write process logs to console with a separate handler 2019-08-15 14:37:08 +02:00
Kristóf Tóth 552c9f5f93 Rename FrontendHandler -> FrontendProxyHandler 2019-08-15 10:43:30 +02:00
R. Richard 2fb971d317 Fix inode creation issue on MacOS 2019-08-14 16:05:55 +02:00
therealkrispet a97a4f4b12 Merge pull request #63 from avatao-content/pipe-connector
Implement inotify based pipe connector
2019-08-14 15:48:46 +02:00
R. Richard dc0615c11e Fix issues in PR #63 2019-08-14 15:42:11 +02:00
R. Richard f5582f0207 Implement inotify based pipe connector 2019-08-14 14:12:31 +02:00
Kristóf Tóth 0df378fb92 Fix FileManager unit test cases for macOS (symlinked TMPDIR) 2019-08-14 13:49:58 +02:00
Kristóf Tóth 8f7715565e Silence watchdog's internal deprecation warning when using pytest 2019-08-14 13:49:25 +02:00
Kristóf Tóth 9e49a93df3 Fix event handler pytest.raises test cases 2019-08-12 16:25:03 +02:00
R. Richard 0549c5e4c8 Fix message filtering issue 2019-08-12 11:57:54 +02:00
Kristóf Tóth 0b7557f198 Fix frontend message replay broken during refactor 2019-08-09 10:59:27 +02:00
Kristóf Tóth 7ba502277f Remove accidental comment 2019-08-09 10:56:38 +02:00
therealkrispet 81790e18e1 Merge pull request #62 from avatao-content/api-rework
Api rework
2019-08-08 16:32:30 +02:00
Kristóf Tóth b5e53cb946 Use prefix matching in ZMQConnector 2019-08-08 15:05:37 +02:00
R. Richard 35e5b595d1 Rename message keys 2019-08-08 14:45:59 +02:00
R. Richard f5e7d6016a Fix condition in PR #62 2019-08-08 08:27:41 +02:00
R. Richard 13179e59eb Restore generate_messages_from_queue() 2019-08-07 11:24:56 +02:00
R. Richard 09ffe2bdcf Forward relevant messages to the frontend 2019-08-07 09:49:25 +02:00
R. Richard f8ff0bcbb4 Simplify sending bot messages 2019-08-07 09:49:05 +02:00
R. Richard 4b7510e704 Update process- and logmonitor according to the new API 2019-08-07 09:47:57 +02:00
R. Richard e414ea2631 Simplify terminal handler 2019-08-07 09:47:28 +02:00
R. Richard dded7fd65c Update FSM related classes according to the new API 2019-08-07 09:46:58 +02:00
R. Richard d31a850a4e Simplify IDE handler and file manager 2019-08-07 09:44:03 +02:00
R. Richard 1210a30c31 Fix event order problem 2019-08-07 09:42:03 +02:00
Kristóf Tóth e6d2777520 Replace pipe-io-server source with the new pip package 2019-08-06 11:33:26 +02:00
R. Richard e87a16991c Implement signed event handler 2019-08-05 13:47:05 +02:00
R. Richard 0a218c8d6d Fix relative path problem 2019-08-01 15:12:58 +02:00
Kristóf Tóth 38eec06b4d Move ZMQ bind address logging to TFWServer 2019-07-31 17:36:03 +02:00
R. Richard 96c6c9b358 Correct whitespaces and add context manager for file locking 2019-07-31 17:19:12 +02:00
R. Richard 25bd9aa0f3 Implement reference counting mechanism 2019-07-31 17:19:12 +02:00
Kristóf Tóth 911831fdb1 Implement sync API for networking 2019-07-31 17:08:47 +02:00
Kristóf Tóth 78c3a8cf98 Fix socket bind order regarding ZMQStream 2019-07-30 15:23:47 +02:00
Kristóf Tóth 30223fe99f Rename ServerConnector interface to Connector ¯\_(ツ)_/¯ 2019-07-30 15:17:29 +02:00
Kristóf Tóth fd92c443b2 Update PipeIOServer dependency 2019-07-30 14:45:33 +02:00
Kristóf Tóth 9a4da0b36f Implement bootiful unit tests for TFW network stack 2019-07-29 13:02:46 +02:00
Kristóf Tóth 6069ea5089 Merge branch 'eventhandler-rework' into chausie 2019-07-25 12:10:43 +02:00
Kristóf Tóth 3fca555251 Move TFWRouter to separate file 2019-07-24 17:22:08 +02:00
Kristóf Tóth 42f959878f Add signal handling to tfw_server.py 2019-07-24 16:47:03 +02:00
Kristóf Tóth f2b6123d41 Remove unnecessary newlines 2019-07-24 16:32:57 +02:00
Kristóf Tóth 52399f413c Simplify package structure 2019-07-24 15:50:41 +02:00
Kristóf Tóth a23224aced Rework whole package structure (improved dependency handling) 2019-07-24 15:17:40 +02:00
Kristóf Tóth c6e01d294d Inject TFWENV dependencies to builtins 2019-07-23 15:32:50 +02:00
Kristóf Tóth ae69a094c7 Improve names in EventHandlerFactoryBase to improve readability 2019-07-23 11:47:59 +02:00
Kristóf Tóth 9725f805ed Silence unjust pylint warnings 2019-07-15 11:09:57 +02:00
Kristóf Tóth 45a9a753b2 Fix test_event_handler 2019-07-15 11:09:43 +02:00
Kristóf Tóth adf2672cd4 Add forgotten dateutil dependency 2019-07-12 23:38:18 +02:00
Kristóf Tóth 026db9ea84 Avoid duplicate filename (EventHandlerFactory) 2019-07-12 23:32:15 +02:00
Kristóf Tóth de78f48930 Adjust the whole framework to event handler dependency inversion 2019-07-12 23:25:16 +02:00
Kristóf Tóth 42beafcf04 Update dependencies 2019-07-12 23:14:47 +02:00
Kristóf Tóth f73c059429 Improve EventHandlerFactoryBase 2019-07-11 18:52:15 +02:00
Kristóf Tóth f1679ffb50 Add new EventHandler stuff as per interface segregation principle 2019-07-10 14:26:12 +02:00
Kristóf Tóth 84a28a1582 Remove unused rate limiter classes (sad to see you go) 2019-07-09 11:02:45 +02:00
Kristóf Tóth e5dee3a1b3 Remove leftover parentheses 2019-07-09 09:12:21 +02:00
Kristóf Tóth 64ea7d8218 Fix ZMQ socketopt call order 2019-07-08 20:12:12 +02:00
R. Richard 9be21d88ac Create event handler to wrap TerminalCommands 2019-07-08 14:13:58 +02:00
R. Richard 98271a3783 Remove dead code 2019-07-08 14:13:58 +02:00
R. Richard fbe60de968 Separate independent classes from built-in event handlers 2019-07-08 14:13:58 +02:00
Kristóf Tóth f6a369496d Improve color scheme 2019-07-04 17:28:26 +02:00
Kristóf Tóth dc62508a7b Remove message scope during routing 2019-07-04 15:55:23 +02:00
Kristóf Tóth 80e42a4091 Ignore message scope in message signatures 2019-07-04 15:51:55 +02:00
Kristóf Tóth 2c0aba2ffe Fix bug where sign_message wasn't idempotent 2019-07-04 15:19:48 +02:00
Kristóf Tóth 459ae684de Make up for forgotten parentheses 2019-07-03 16:48:56 +02:00
Kristóf Tóth ca697b41b3 Further improve EventHandler <-> EventHandlerBase port situation 2019-07-03 16:48:17 +02:00
Kristóf Tóth 49844c7b58 Fix trying to send None as a TFW message 2019-07-03 15:49:20 +02:00
Kristóf Tóth 81accf94ae Fix inotify unit tests on BSD based systems 2019-07-02 16:39:36 +02:00
R. Richard 882ab60490 Print stack trace during an exception 2019-07-02 15:39:46 +02:00
Kristóf Tóth 3be018eb17 Make EventHandlerBase capable of stopping all it's instances 2019-07-01 16:01:54 +02:00
R. Richard 68a6469d23 Refactor TFW logging 2019-06-28 17:44:49 +02:00
R. Richard f94c5b0c37 Rename server_connector to uplink 2019-06-28 17:44:49 +02:00
Kristóf Tóth aafd145fc5 Delete copyright headers in preparation of going open source ^^ 2019-06-28 17:44:43 +02:00
Kristóf Tóth 018286f397 Improve EventHandler response sending pattern 2019-06-28 16:50:36 +02:00
Kristóf Tóth 481a919606 Improve FSMAware api 2019-06-28 16:49:58 +02:00
Kristóf Tóth 78bd97493a Rework EventHandler situation 2019-06-28 15:11:02 +02:00
Kristóf Tóth 28c3e68b5a Refactor TFWLog 2019-06-27 17:42:41 +02:00
Kristóf Tóth bcee486463 Refactor InotifyObserver 2019-06-27 17:42:41 +02:00
Kristóf Tóth 491eaf6d5e Refactor HistoryMonitor 2019-06-27 17:42:41 +02:00
R. Richard 44a1433d3b Refactor Inotify 2019-06-27 17:42:41 +02:00
R. Richard da19769362 Fix file observer 2019-06-27 17:42:41 +02:00
Kristóf Tóth 6cc3d32097 Refactor LogMonitoringEventHandler 2019-06-27 17:42:41 +02:00
R. Richard c5c5d2c10a Fix logging bug 2019-06-27 17:42:41 +02:00
R. Richard 5f5fc1a8c9 Refactor IdeEventHandler 2019-06-27 17:42:41 +02:00
R. Richard 28f09086a2 Fix pylint warnings 2019-06-27 17:42:41 +02:00
R. Richard 810f6d5297 Remove deprecated argument 2019-06-27 17:42:41 +02:00
R. Richard ffe512776a Make files observable and refactor event handlers 2019-06-27 17:42:41 +02:00
R. Richard 778f155580 Acquire log path from the environment 2019-06-27 17:42:41 +02:00
R. Richard 149dcb0b4b Move MessageSender due to dependencies 2019-06-27 17:42:41 +02:00
R. Richard b9ee2031e2 Delete unused import 2019-06-27 17:42:41 +02:00
R. Richard 5e06f72c35 Rename private methods to be underscored 2019-06-27 17:42:41 +02:00
R. Richard d8e2f8a92a Make file operations unbuffered 2019-06-27 17:42:41 +02:00
R. Richard 494761d2a7 Replace mixins with inotify based observers 2019-06-27 17:42:41 +02:00
R. Richard 597523b3b9 Make observing multiple directories possible 2019-06-27 17:42:41 +02:00
R. Richard 65c8bcee2b Remove DirectoryMonitoringEventHandler 2019-06-27 17:42:41 +02:00
R. Richard d031f481b8 Print exceptions with stack trace 2019-06-27 17:42:41 +02:00
R. Richard cbc62c5a10 Switch import order 2019-06-27 17:42:41 +02:00
R. Richard 13e7470f48 Separate event handlers from independent components 2019-06-27 17:42:41 +02:00
R. Richard c8a0546041 Remove unnecessary imports 2019-06-27 17:42:41 +02:00
R. Richard c6e0b54930 Make Inotify testing faster 2019-06-27 17:42:41 +02:00
R. Richard 8bf18113b2 Implement custom logging logic 2019-06-27 17:42:41 +02:00
R. Richard 85c720127e Make HistoryMonitor utilize our inotify module 2019-06-27 17:42:41 +02:00
Kristóf Tóth 49856cebe2 Improve InotifyEvent.__repr__ 2019-06-18 14:54:54 +02:00
Kristóf Tóth 3cc70c2147 Fix IdeEventHandler broken import 2019-06-17 15:09:42 +02:00
therealkrispet 8a5c7b2a49 Merge pull request #56 from avatao-content/inotify
Create inotify module with unit tests
2019-06-11 13:03:52 +02:00
R. Richard 20bfe3c352 Make recursive watching optional and underscore private attributes/methods 2019-06-11 13:03:02 +02:00
R. Richard d2131682a8 Make recursive watching optional and rename reset() 2019-06-11 11:21:46 +02:00
therealkrispet 8482d7ec2b Merge pull request #55 from avatao-content/file_manager_fixes
File manager fixes
2019-06-11 11:14:24 +02:00
R. Richard 8cbe737d2f Turn context into a dataclass and add new test case 2019-06-11 11:11:29 +02:00
R. Richard b44fd200c6 Make fixtures function scoped and add new tests 2019-06-11 11:11:29 +02:00
R. Richard 9cb8ef0e72 Add unit test for file manager 2019-06-11 11:11:29 +02:00
R. Richard a69031015b Refactor file manager 2019-06-11 11:11:29 +02:00
therealkrispet f495ff2d7c Merge pull request #58 from avatao-content/logging
Import the logging modules directly
2019-06-11 11:08:43 +02:00
R. Richard 8be0105ef6 Remove unnecessary config 2019-06-11 11:07:12 +02:00
therealkrispet 29524b8d20 Merge pull request #59 from avatao-content/connector-rm
Add SQLAlchemy as dependency
2019-06-11 10:36:07 +02:00
R. Richard 105a574d7f Import the logging modules directly 2019-06-10 15:32:45 +02:00
R. Richard 3d00cbeebc Add SQLAlchemy as dependency 2019-06-06 16:57:15 +02:00
R. Richard ca6797b354 Fixing issues in PR #56 2019-06-06 13:41:13 +02:00
R. Richard dfac686bfc Create inotify module with unit tests 2019-06-06 10:30:01 +02:00
Kristóf Tóth bed99c20be Merge branch 'networking-rework' into chausie 2019-06-04 14:45:13 +02:00
Kristóf Tóth c8e98af516 Improve module dependencies by moving port envvars out of tfw.networking 2019-06-04 14:41:51 +02:00
Kristóf Tóth f151ecfbac Improve ZMQ port envvar names 2019-06-04 14:41:51 +02:00
Kristóf Tóth e44a99fa6d Remove ZMQConnectorBase 2019-06-04 14:41:51 +02:00
Kristóf Tóth 82df8a8065 Handle pylint suggestions 2019-06-04 14:41:51 +02:00
Kristóf Tóth 2134d743c3 Improve TFW lib layout 2019-06-04 14:41:51 +02:00
Kristóf Tóth 01d9003501 Rework whole TFW networking model 2019-06-04 14:41:51 +02:00
Kristóf Tóth 613919a5b6 Implement closing EventHandlerConnector 2019-06-04 14:41:50 +02:00
Kristóf Tóth 6431fac9b1 Set ZMQ HWM in EventHandlerConnector to infinite 2019-06-04 14:41:50 +02:00
Kristóf Tóth 6b23b863ed Fix code formatting 2019-06-04 14:41:50 +02:00
Kristóf Tóth 26c6c5d1e6 Refactor EventHandlerConnector family of classes 2019-06-04 14:41:50 +02:00
R. Richard de6afdc84a Make Terminado stoppable 2019-05-31 13:10:33 +02:00
R. Richard 71f05fe92c Fix terminal writing error on challenge startup 2019-05-31 11:13:33 +02:00
R. Richard 7d50ee607a Close ZMQ sockets gracefully 2019-05-20 11:06:57 +02:00
Kristóf Tóth 2a3d29f080 Remove git hooks 2019-05-17 00:04:36 +02:00
Kristóf Tóth 743ff8be4e Update packages 2019-05-17 00:03:46 +02:00
Kristóf Tóth 16eb1ba780 Name new release 2019-05-16 23:44:44 +02:00
Kristóf Tóth 07cd1264f5 Merge branch 'ocicat', the unrealized dream. Ocicat will return... 2019-05-15 17:10:05 +02:00
Kristóf Tóth f43c463692 Merge branch 'pipeio' 2019-05-13 16:31:57 +02:00
Kristóf Tóth 51b5578270 Update builder image in .drone.yml 2019-05-13 15:07:19 +02:00
Kristóf Tóth 3bfe6db036 Avoid treating supervisord's SIGTERM as an error in CommandEventHandler 2019-05-13 14:53:31 +02:00
Kristóf Tóth 94dee63a41 Improve CommandEventHandler subprocess failure error message 2019-05-13 14:52:17 +02:00
Kristóf Tóth 1617761184 Improve CommandEventHandler error detection (avoid polling) 2019-05-13 11:17:30 +02:00
Kristóf Tóth ca5be9d848 Detect errors in the subprocess of CommandEventHandler 2019-05-13 10:09:06 +02:00
Kristóf Tóth 78d70f2f8b Make PipeIOEventHandlerBase handle all input errors (while screaming) 2019-05-13 10:09:06 +02:00
Kristóf Tóth fc5124afb0 Make arbitrary processes capable of being EventHandlers (PipeIO+stdin/stdout) 2019-05-13 10:09:06 +02:00
Kristóf Tóth c4d3319ed9 Refactor TransformerPipeIOEventHandler 2019-05-13 10:09:06 +02:00
Kristóf Tóth 69b3b17724 Implement EventHandler for transforming and filtering pipe messages 2019-05-13 10:09:06 +02:00
Kristóf Tóth 078f8532cc Rename parameter to reflect type in PipeIOEventHandler.handle_pipe_event 2019-05-13 10:09:06 +02:00
Kristóf Tóth 1bfaac0493 Remove potentially harmful 'error handling' 2019-05-13 10:09:06 +02:00
Kristóf Tóth 9e36bde974 Consider PipeIOServer public API from now on 2019-05-13 10:09:06 +02:00
Kristóf Tóth 8f1ae9e286 Add jq as a dependency 2019-05-13 10:09:06 +02:00
Kristóf Tóth 9ad77eaed8 Allow passing an iterable with keys to subscribe to in EventHandlerBase.__init__() 2019-05-13 10:09:06 +02:00
Kristóf Tóth 90b780a5c0 Allow subclassing pipe based event handlers (refactor PipeIOEventHandler to base class and impl) 2019-05-13 10:09:06 +02:00
Kristóf Tóth ddc79c9717 Allow subscribing to all ZMQ topics in EventHandlerBase 2019-05-13 10:09:06 +02:00
Kristóf Tóth bb8e0c7458 Handle JSON serialization errors in PipeIOEventHandler 2019-05-13 10:09:06 +02:00
Kristóf Tóth f94d571d19 Support initializing IO pipes with specific permissions 2019-05-13 10:09:06 +02:00
Kristóf Tóth 065aa56182 Implement EventHandler capable of proxying between TFW and named pipes 2019-05-13 10:09:06 +02:00
Kristóf Tóth 6ea0967a21 Fork PipeIOServer to TFW 2019-05-13 10:08:59 +02:00
Kristóf Tóth 0771a068e0 Implement helper to get all EventHandler instances in a given stack frame 2019-05-02 14:27:48 +02:00
Kristóf Tóth 1d969f0967 Implement lazy_factory 🍰 2018-10-09 11:25:11 +02:00
Kristóf Tóth 031400c0c4 Update pip packages 2018-08-22 14:09:53 +02:00
Kristóf Tóth 806623c80d Extend API documentation 2018-08-08 19:27:58 +02:00
Kristóf Tóth 48c3df621b Add readme info on delaying histfile appending 2018-08-07 17:28:56 +02:00
Kristóf Tóth 21f05ad850 Silence unjust pylint warning 2018-08-06 15:52:01 +02:00
Kristóf Tóth 44bdc96547 Support changing gitignore from event handler API 2018-08-06 15:47:14 +02:00
Kristóf Tóth b7ed4c3d0f Implement gitignore functionality in SnapshotProvider 2018-08-06 15:42:51 +02:00
Kristóf Tóth 59dce4a848 Restore latest snapshot if no date is provided 2018-08-06 14:55:58 +02:00
Kristóf Tóth cbb807dfb4 Implement restore_snapshot latest commit on branch detection 2018-08-06 14:19:18 +02:00
Kristóf Tóth 782df25bee Fix broken init_git_dir logic 🐛 2018-08-06 13:40:16 +02:00
Kristóf Tóth b140550686 Remove debugging log from CommandsEqual 2018-08-06 13:16:20 +02:00
Kristóf Tóth 01e5577890 Document FSMManagingEventHandler 2018-08-04 23:27:18 +02:00
Kristóf Tóth f6d77e1132 Add docstrings to CommandsEqual 2018-08-04 21:49:06 +02:00
Kristóf Tóth 16c936b2cd Add exclude_patterns support for CommandsEqual 2018-08-04 21:12:06 +02:00
Kristóf Tóth 4f881a0ea0 Implement must_contain_patterns CommandsEqual 2018-08-03 16:07:12 +02:00
Kristóf Tóth 8454236bc8 Implement must_begin_similarly constraint CommandsEqual 2018-08-03 15:13:02 +02:00
Kristóf Tóth b6d8f7913f Improve CommandsEqual with fuzzy logic 2018-08-03 15:01:44 +02:00
Kristóf Tóth 7572699e55 Start working on something better than == for history checks 2018-08-03 13:40:34 +02:00
Kristóf Tóth 088a1cefc5 Fix initialization issues with DirectorySnapshottingEH 2018-08-03 11:58:03 +02:00
Kristóf Tóth e383be0149 Fix restore_snapshot() choking on timestamps before initial commit 2018-08-03 11:55:51 +02:00
Kristóf Tóth 3d2e3e7db3 Fix python3.7 incompatibilities in SnapshotProvider 2018-08-03 11:39:55 +02:00
Kristóf Tóth 3fee8fee20 Implement first version of DirectorySnapshottingEventHandler 2018-08-01 17:24:39 +02:00
Kristóf Tóth df0e24319d Fix SnapshotProvider failing on taking_snapshot without changes 2018-08-01 17:19:31 +02:00
Kristóf Tóth 2e97d18340 Fix SnapshotProvider failing depending on python3.7 2018-08-01 17:18:43 +02:00
Kristóf Tóth e3b97ee190 Fix message sequencing not being global 2018-08-01 17:15:42 +02:00
Kristóf Tóth 564c97e66a Remove unused variable from terminado server 2018-08-01 14:17:17 +02:00
Kristóf Tóth aa0fe5d00f Improve Dockerfile 2018-08-01 14:17:02 +02:00
Kristóf Tóth 14a98587a4 Silence pylint false positive 2018-07-31 15:18:35 +02:00
Kristóf Tóth 1ad9e0d0e3 Add sphinx docs on rate limiting 2018-07-31 14:34:28 +02:00
Kristóf Tóth b6d72812c4 Refactor ZMQWSProxy filter and callback initialization 2018-07-31 13:34:00 +02:00
Kristóf Tóth 3bc30ab503 Add info on signature and seq keys to readme 2018-07-31 13:34:00 +02:00
Kristóf Tóth a04b078513 Implement message sequence numbers in ZMQWSProxy 2018-07-31 13:34:00 +02:00
Kristóf Tóth d94bc37d48 Add event logging to FSMAware 2018-07-31 13:34:00 +02:00
Kristóf Tóth 199e1a5d6e Update readme with Python3 event handler info 2018-07-31 13:34:00 +02:00
Kristóf Tóth a2d1531ea4 Fix sphinx docs broken after dependency hell 2018-07-31 13:34:00 +02:00
Kristóf Tóth 732b896d17 Add docstrings to YamlFSM 2018-07-31 13:34:00 +02:00
Kristóf Tóth 8df196f258 Add note on step_next trigger to LinearFSM 2018-07-31 13:34:00 +02:00
Kristóf Tóth 3d3328f835 Fix bad indentation 2018-07-31 13:34:00 +02:00
Kristóf Tóth 3fad818b2b Add pylint pre-push hooks 2018-07-31 13:33:59 +02:00
Kristóf Tóth 7a670f37f2 Resolve Python circular import hell - hopefully forever 2018-07-31 13:33:59 +02:00
Kristóf Tóth 7fb5a37831 Make TFWServer FSMAware 2018-07-31 13:33:59 +02:00
Kristóf Tóth ca09e868f1 Fix ZMQWSProxy not executing filters/callbacks on nested messages 2018-07-31 13:33:59 +02:00
Kristóf Tóth 5715c57ebc Move FSMAware to a separate file in networking 2018-07-31 13:33:59 +02:00
Kristóf Tóth fe79b598a7 Update major version due to API breakage 2018-07-31 13:33:59 +02:00
Kristóf Tóth a6b7fa04ab Rework fsm_update API 2018-07-31 13:33:59 +02:00
Kristóf Tóth d718b6425e Refactor FSMAware part from FSMAwareEH to a separate class 2018-07-31 13:33:59 +02:00
Kristóf Tóth 8c6a14cef5 Move fsm stuff to a separate directory in lib 2018-07-31 13:33:59 +02:00
Kristóf Tóth 52b2adb9c4 Add missing legal boilerplate to YamlFSM 2018-07-31 13:33:59 +02:00
Bokros Bálint 015d8f4355 Merge pull request #38 from avatao-content/rate_limit_magic
Rate limit magic
2018-07-31 13:27:30 +02:00
Kristóf Tóth 3dff144b91 Fix RateLimiter family debounce stuff 2018-07-31 11:48:41 +02:00
Kristóf Tóth 8a0928beca Fix RateLimiter semantics 2018-07-31 09:56:18 +02:00
Kristóf Tóth 3c3012ffe8 Add docstrings to AsyncRateLimiter 2018-07-31 09:23:49 +02:00
Kristóf Tóth 128f48702a Make AsyncRateLimiter thread safe 2018-07-31 09:14:33 +02:00
Kristóf Tóth 4679a3494c Add docstrings to RateLimiter 2018-07-31 05:19:15 +02:00
Kristóf Tóth ee7adb10be Refactor AsyncRateLimiter for ease of use (IOLoop passing) 2018-07-30 18:04:24 +02:00
Kristóf Tóth eeb36b6488 Implement IOLoop based AsyncRateLimiter 2018-07-30 17:55:52 +02:00
Kristóf Tóth 6044f70804 Refactor RateLimiter to allow proper subclassing 2018-07-30 17:54:26 +02:00
Bokros Bálint f58f362e46 Merge pull request #37 from avatao-content/require_signature
Require signature
2018-07-23 17:27:04 +02:00
Kristóf Tóth b3e8af2024 Implement require_signature mode in FSMManagingEH 2018-07-23 17:16:37 +02:00
Kristóf Tóth e846a2b111 Make TFWServer sign commands issued due to signed triggers 2018-07-23 17:15:14 +02:00
Kristóf Tóth 8ba99d8e36 Make FSMBase.step() return an exit code 2018-07-23 17:14:14 +02:00
Kristóf Tóth 1a52ed2c9a Add error handling logic to MessageStorage 2018-07-20 20:47:10 +02:00
Kristóf Tóth bad7297c94 Fix TerminalEventHandler returning none as 'data' 2018-07-20 20:34:18 +02:00
Kristóf Tóth 878b6d3f2a Merge branch 'state_rewind' 2018-07-20 15:19:53 +02:00
Kristóf Tóth cb4ba563e9 Fix LinearFSM not being compatible with controller stuff 2018-07-20 15:03:49 +02:00
Kristóf Tóth 96b4e314a9 Add 'in_accepted_state' to fsm_update messages 2018-07-20 14:40:47 +02:00
Kristóf Tóth e1e0acb3b8 Fix FSMBase default accepted_state 2018-07-20 14:38:26 +02:00
Kristóf Tóth 446d453947 Make TFW lib code location a volume so controller can use it 2018-07-20 14:37:26 +02:00
Kristóf Tóth 3e5dd6e102 Fix typo 2018-07-20 10:27:58 +02:00
Kristóf Tóth bd6cdc5409 Fix stored messages growing infinitely on reloads 2018-07-20 09:34:13 +02:00
Kristóf Tóth 7376504dd2 Make frontend state recovery recover selected file in IDE 2018-07-20 09:34:07 +02:00
Kristóf Tóth a8fdd3b0d8 Refactor SnapshotProvider 2018-07-19 16:54:11 +02:00
Kristóf Tóth 0dd66c33bf Make error handling more robust 2018-07-19 16:54:11 +02:00
Kristóf Tóth a703ee821d Fix SnapshotProvider initialization stuff 2018-07-19 16:54:11 +02:00
Kristóf Tóth 1bb6286d24 Fix timeline property 2018-07-19 16:54:11 +02:00
Kristóf Tóth 889ec92928 Use UUIDs instead of commit hashes for new branch names (git ambigous) 2018-07-19 16:54:11 +02:00
Kristóf Tóth 98e1bbe9d6 Remove unnecessary _last_valid_tag stuff 2018-07-19 16:54:11 +02:00
Kristóf Tóth d8ac0dc311 Reorder SnapshotProvider methods in accordance with Uncle Bob 2018-07-19 16:54:11 +02:00
Kristóf Tóth c6d3b8ad21 Ensure _last_valid_branch consistency 2018-07-19 16:54:11 +02:00
Kristóf Tóth dd056a37bb Make coding style more consistent 2018-07-19 16:54:11 +02:00
Kristóf Tóth 85523ede32 Make SnapshotProvider more in line with reality 2018-07-19 16:54:11 +02:00
Kristóf Tóth fa3ce317f0 Rename stuff to improve code readability 2018-07-19 16:54:11 +02:00
Kristóf Tóth fb2beb470f Make subprocess calls fail if something goes wrong 2018-07-19 16:54:11 +02:00
Kristóf Tóth d306d0e484 Implement properties to get info from SnapshotProvider instance 2018-07-19 16:54:11 +02:00
Kristóf Tóth e80782aa6c Implement checking whether directories passed exsist 2018-07-19 16:54:11 +02:00
Kristóf Tóth 49e0efa4c9 Run git init only if there is no repo present 2018-07-19 16:54:11 +02:00
Kristóf Tóth 2f5e8d11f3 Automatically init git in SnapshotProvider 2018-07-19 16:54:11 +02:00
Kristóf Tóth c279b2517f Implement branching in SnapshotProvider 2018-07-19 16:54:11 +02:00
Kristóf Tóth 8e87025905 Implement git based fs snapshot provider 2018-07-19 16:54:07 +02:00
Kristóf Tóth d5feba7076 Implement HMAC signatures of fsm_update broadcast messages 2018-07-16 14:31:52 +02:00
Kristóf Tóth c658894c12 Implement authentication key management 2018-07-16 14:28:40 +02:00
Kristóf Tóth c28a66fc48 Make forward slash usage more consistent 2018-07-16 13:30:53 +02:00
Kristóf Tóth a79c68515b Remove controller stuff, tidy code based on pylint suggestions 2018-07-16 11:18:27 +02:00
Kristóf Tóth 7c13d31de0 Refactor message signing and verifying logic 2018-07-16 10:29:06 +02:00
Kristóf Tóth 5770e29733 Move message checksum logic to crypto.py 2018-07-15 17:30:19 +02:00
Kristóf Tóth b2cb60ef02 Implement message signing and verification logic 2018-07-15 17:29:16 +02:00
Kristóf Tóth eb2c3a8dd0 Add crypto module with HMAC-SHA256 implementation 2018-07-15 17:27:35 +02:00
Kristóf Tóth 16d98c75ca Add message_bytes() function to serialization module 2018-07-15 17:26:00 +02:00
Kristóf Tóth 646a671ff3 Add FSMAwareEventHandler which keeps track of FSM state 2018-07-14 20:45:56 +02:00
Kristóf Tóth 4236a17b69 Include last trigger in fsm_update messages 2018-07-14 20:45:26 +02:00
Kristóf Tóth 77f14df17b Implement trigger history in FSMBase 2018-07-14 20:44:31 +02:00
Kristóf Tóth d98ce58651 Add proper multikey support for EventHandlers with compatibility 2018-07-14 20:06:34 +02:00
Kristóf Tóth cfa8fc5ab6 Remove deprecated 'reset' key. Sad to see you go :( 2018-07-13 15:42:18 +02:00
Kristóf Tóth 688c615fd7 Implement frontend state recovery backend (partial) 2018-07-13 15:24:45 +02:00
Kristóf Tóth 9c20e94dd8 Implement generating single messages from a queue 2018-07-13 14:09:13 +02:00
Kristóf Tóth d2084b2e51 Add frontend/eventhandler message callback capabilities to ZMQWSProxy 2018-07-13 14:08:30 +02:00
Kristóf Tóth f6cc7ae495 Refactor ZMQWSProxy.__init__ 2018-07-12 16:49:00 +02:00
Kristóf Tóth da98576260 Add note on FSM update messages in API docs 2018-07-12 15:30:10 +02:00
Kristóf Tóth b06472ad19 Reintroduce FSM update messages with minor changes 2018-07-12 15:20:35 +02:00
Kristóf Tóth 17b24377dd Implement a ZMQ prefix-match countermeasure in EHBase 2018-07-12 15:17:39 +02:00
Kristóf Tóth 558e660268 Use ServerUplinkConnetor.broadcast() instead of local helper 2018-07-12 14:58:31 +02:00
Kristóf Tóth 381cd2b11e Simplify ServerUplinkConnector mirror logic, add broadcast support 2018-07-12 14:57:43 +02:00
Kristóf Tóth 9f6ac6d27e Implement optional Jinja2 templating support for YamlFSM 2018-07-11 15:30:52 +02:00
Kristóf Tóth 09bcb7de6b Merge branch 'fsm_as_eventhandler' 2018-07-11 10:34:49 +02:00
Kristóf Tóth 0e8f529726 Optimize FSMBase by using generators for predicate logic 2018-07-10 15:40:10 +02:00
Kristóf Tóth f78f01d6e1 Provide full event handler API documentation 2018-07-10 14:37:20 +02:00
Kristóf Tóth 5262401b18 Update README and explain TFW server behaviour better 2018-07-07 21:30:24 +02:00
Kristóf Tóth 57d2475ebc Fix invalid trigger killing FSMBase 2018-07-06 15:40:27 +01:00
Kristóf Tóth 7cfa63bacf Rename config transformation method for easier understanding 2018-07-06 12:31:25 +01:00
Kristóf Tóth 7f583d8d1f Improve YamlFSM initialization logic 2018-07-06 12:27:26 +01:00
Kristóf Tóth c7ee97f0c6 Simplify predicate finding logic YamlFSM 2018-07-04 21:58:30 +02:00
Kristóf Tóth 1beb419b09 Remove subprocess spawning stuff from YamlFSM for SRP 2018-07-04 18:15:34 +02:00
Kristóf Tóth d71a25e30a Implement subscribing predicates found in yaml 2018-07-04 18:11:42 +02:00
Kristóf Tóth ea76a19595 Refactor YamlFSM moar 2018-07-04 18:00:41 +02:00
Kristóf Tóth bfa1bffbc5 Refactor YamlFSM 2018-07-04 17:48:31 +02:00
Kristóf Tóth 7a92d88b73 Refactor FSMBase to subclass transitions.Machine 2018-07-04 15:48:16 +02:00
Kristóf Tóth 91c257554f Simplify callback monkeypatching logic in YamlFSM 2018-07-03 20:09:47 +02:00
Kristóf Tóth 022a997dc2 Implement monkey patching callbacks in YamlFSM config 2018-07-03 19:06:54 +02:00
Kristóf Tóth f575a80bd4 Implement optional delay in bash history command appending 2018-07-03 15:22:29 +02:00
Kristóf Tóth 5e4303ac06 Add first version of YamlFSM 2018-07-03 15:14:00 +02:00
Kristóf Tóth 7c0e6d49bc Handle starting TFWServer in baseimage 2018-06-29 22:53:44 +02:00
Kristóf Tóth 708c920784 Move FSM handling logic to an event handler 2018-06-29 22:03:19 +02:00
Kristóf Tóth a6563bcd89 Implement event handler base class that broadcasts everything 2018-06-29 22:02:26 +02:00
Kristóf Tóth 196e753fb9 Fix typo in comment 2018-06-29 15:59:03 +02:00
Kristóf Tóth 427694623f Rename file to harmonize with new networking code structure 2018-06-29 15:40:07 +02:00
Kristóf Tóth 7ed0715f4c Reorder stuff to follow the teachings of Uncle Bob 2018-06-29 15:33:45 +02:00
Kristóf Tóth b217ac59c8 Move message validation code to where it belongs 2018-06-29 12:06:08 +02:00
Kristóf Tóth 36a86b0454 Hide ZMQ serialization magic from EHConnector clients 2018-06-29 11:58:05 +02:00
Kristóf Tóth 934f8ec74c Hide ZMQ serialization magic from ServerConnector clients 2018-06-29 11:50:36 +02:00
Kristóf Tóth 1b65bd4d3d Implement message broadcasting 2018-06-29 10:54:08 +02:00
Kristóf Tóth f8233d51a9 Completely rework TFWServer networking 🍰 2018-06-28 17:33:20 +02:00
Kristóf Tóth 3ba56a8096 Implement batch callback subscription in CallbackMixin 2018-06-28 17:31:55 +02:00
158 changed files with 5259 additions and 1598 deletions
+2 -2
View File
@@ -1,6 +1,6 @@
pipeline:
build:
image: eu.gcr.io/avatao-public/docker:el7
image: eu.gcr.io/avatao-challengestore/challenge-builder
volumes:
- /etc/docker:/etc/docker:ro
- /root/.docker:/root/.docker:ro
@@ -10,4 +10,4 @@ pipeline:
- docker push eu.gcr.io/avatao-challengestore/tutorial-framework:${DRONE_TAG}
when:
event: 'tag'
branch: refs/tags/bombay-20*
branch: refs/tags/chausie-20*
+7
View File
@@ -3,3 +3,10 @@
ignored-modules = zmq
max-line-length = 120
disable = missing-docstring, too-few-public-methods, invalid-name
[SIMILARITIES]
min-similarity-lines=8
ignore-comments=yes
ignore-docstrings=yes
ignore-imports=yes
+674
View File
@@ -0,0 +1,674 @@
GNU GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
Preamble
The GNU General Public License is a free, copyleft license for
software and other kinds of works.
The licenses for most software and other practical works are designed
to take away your freedom to share and change the works. By contrast,
the GNU General Public License is intended to guarantee your freedom to
share and change all versions of a program--to make sure it remains free
software for all its users. We, the Free Software Foundation, use the
GNU General Public License for most of our software; it applies also to
any other work released this way by its authors. You can apply it to
your programs, too.
When we speak of free software, we are referring to freedom, not
price. Our General Public Licenses are designed to make sure that you
have the freedom to distribute copies of free software (and charge for
them if you wish), that you receive source code or can get it if you
want it, that you can change the software or use pieces of it in new
free programs, and that you know you can do these things.
To protect your rights, we need to prevent others from denying you
these rights or asking you to surrender the rights. Therefore, you have
certain responsibilities if you distribute copies of the software, or if
you modify it: responsibilities to respect the freedom of others.
For example, if you distribute copies of such a program, whether
gratis or for a fee, you must pass on to the recipients the same
freedoms that you received. You must make sure that they, too, receive
or can get the source code. And you must show them these terms so they
know their rights.
Developers that use the GNU GPL protect your rights with two steps:
(1) assert copyright on the software, and (2) offer you this License
giving you legal permission to copy, distribute and/or modify it.
For the developers' and authors' protection, the GPL clearly explains
that there is no warranty for this free software. For both users' and
authors' sake, the GPL requires that modified versions be marked as
changed, so that their problems will not be attributed erroneously to
authors of previous versions.
Some devices are designed to deny users access to install or run
modified versions of the software inside them, although the manufacturer
can do so. This is fundamentally incompatible with the aim of
protecting users' freedom to change the software. The systematic
pattern of such abuse occurs in the area of products for individuals to
use, which is precisely where it is most unacceptable. Therefore, we
have designed this version of the GPL to prohibit the practice for those
products. If such problems arise substantially in other domains, we
stand ready to extend this provision to those domains in future versions
of the GPL, as needed to protect the freedom of users.
Finally, every program is threatened constantly by software patents.
States should not allow patents to restrict development and use of
software on general-purpose computers, but in those that do, we wish to
avoid the special danger that patents applied to a free program could
make it effectively proprietary. To prevent this, the GPL assures that
patents cannot be used to render the program non-free.
The precise terms and conditions for copying, distribution and
modification follow.
TERMS AND CONDITIONS
0. Definitions.
"This License" refers to version 3 of the GNU General Public License.
"Copyright" also means copyright-like laws that apply to other kinds of
works, such as semiconductor masks.
"The Program" refers to any copyrightable work licensed under this
License. Each licensee is addressed as "you". "Licensees" and
"recipients" may be individuals or organizations.
To "modify" a work means to copy from or adapt all or part of the work
in a fashion requiring copyright permission, other than the making of an
exact copy. The resulting work is called a "modified version" of the
earlier work or a work "based on" the earlier work.
A "covered work" means either the unmodified Program or a work based
on the Program.
To "propagate" a work means to do anything with it that, without
permission, would make you directly or secondarily liable for
infringement under applicable copyright law, except executing it on a
computer or modifying a private copy. Propagation includes copying,
distribution (with or without modification), making available to the
public, and in some countries other activities as well.
To "convey" a work means any kind of propagation that enables other
parties to make or receive copies. Mere interaction with a user through
a computer network, with no transfer of a copy, is not conveying.
An interactive user interface displays "Appropriate Legal Notices"
to the extent that it includes a convenient and prominently visible
feature that (1) displays an appropriate copyright notice, and (2)
tells the user that there is no warranty for the work (except to the
extent that warranties are provided), that licensees may convey the
work under this License, and how to view a copy of this License. If
the interface presents a list of user commands or options, such as a
menu, a prominent item in the list meets this criterion.
1. Source Code.
The "source code" for a work means the preferred form of the work
for making modifications to it. "Object code" means any non-source
form of a work.
A "Standard Interface" means an interface that either is an official
standard defined by a recognized standards body, or, in the case of
interfaces specified for a particular programming language, one that
is widely used among developers working in that language.
The "System Libraries" of an executable work include anything, other
than the work as a whole, that (a) is included in the normal form of
packaging a Major Component, but which is not part of that Major
Component, and (b) serves only to enable use of the work with that
Major Component, or to implement a Standard Interface for which an
implementation is available to the public in source code form. A
"Major Component", in this context, means a major essential component
(kernel, window system, and so on) of the specific operating system
(if any) on which the executable work runs, or a compiler used to
produce the work, or an object code interpreter used to run it.
The "Corresponding Source" for a work in object code form means all
the source code needed to generate, install, and (for an executable
work) run the object code and to modify the work, including scripts to
control those activities. However, it does not include the work's
System Libraries, or general-purpose tools or generally available free
programs which are used unmodified in performing those activities but
which are not part of the work. For example, Corresponding Source
includes interface definition files associated with source files for
the work, and the source code for shared libraries and dynamically
linked subprograms that the work is specifically designed to require,
such as by intimate data communication or control flow between those
subprograms and other parts of the work.
The Corresponding Source need not include anything that users
can regenerate automatically from other parts of the Corresponding
Source.
The Corresponding Source for a work in source code form is that
same work.
2. Basic Permissions.
All rights granted under this License are granted for the term of
copyright on the Program, and are irrevocable provided the stated
conditions are met. This License explicitly affirms your unlimited
permission to run the unmodified Program. The output from running a
covered work is covered by this License only if the output, given its
content, constitutes a covered work. This License acknowledges your
rights of fair use or other equivalent, as provided by copyright law.
You may make, run and propagate covered works that you do not
convey, without conditions so long as your license otherwise remains
in force. You may convey covered works to others for the sole purpose
of having them make modifications exclusively for you, or provide you
with facilities for running those works, provided that you comply with
the terms of this License in conveying all material for which you do
not control copyright. Those thus making or running the covered works
for you must do so exclusively on your behalf, under your direction
and control, on terms that prohibit them from making any copies of
your copyrighted material outside their relationship with you.
Conveying under any other circumstances is permitted solely under
the conditions stated below. Sublicensing is not allowed; section 10
makes it unnecessary.
3. Protecting Users' Legal Rights From Anti-Circumvention Law.
No covered work shall be deemed part of an effective technological
measure under any applicable law fulfilling obligations under article
11 of the WIPO copyright treaty adopted on 20 December 1996, or
similar laws prohibiting or restricting circumvention of such
measures.
When you convey a covered work, you waive any legal power to forbid
circumvention of technological measures to the extent such circumvention
is effected by exercising rights under this License with respect to
the covered work, and you disclaim any intention to limit operation or
modification of the work as a means of enforcing, against the work's
users, your or third parties' legal rights to forbid circumvention of
technological measures.
4. Conveying Verbatim Copies.
You may convey verbatim copies of the Program's source code as you
receive it, in any medium, provided that you conspicuously and
appropriately publish on each copy an appropriate copyright notice;
keep intact all notices stating that this License and any
non-permissive terms added in accord with section 7 apply to the code;
keep intact all notices of the absence of any warranty; and give all
recipients a copy of this License along with the Program.
You may charge any price or no price for each copy that you convey,
and you may offer support or warranty protection for a fee.
5. Conveying Modified Source Versions.
You may convey a work based on the Program, or the modifications to
produce it from the Program, in the form of source code under the
terms of section 4, provided that you also meet all of these conditions:
a) The work must carry prominent notices stating that you modified
it, and giving a relevant date.
b) The work must carry prominent notices stating that it is
released under this License and any conditions added under section
7. This requirement modifies the requirement in section 4 to
"keep intact all notices".
c) You must license the entire work, as a whole, under this
License to anyone who comes into possession of a copy. This
License will therefore apply, along with any applicable section 7
additional terms, to the whole of the work, and all its parts,
regardless of how they are packaged. This License gives no
permission to license the work in any other way, but it does not
invalidate such permission if you have separately received it.
d) If the work has interactive user interfaces, each must display
Appropriate Legal Notices; however, if the Program has interactive
interfaces that do not display Appropriate Legal Notices, your
work need not make them do so.
A compilation of a covered work with other separate and independent
works, which are not by their nature extensions of the covered work,
and which are not combined with it such as to form a larger program,
in or on a volume of a storage or distribution medium, is called an
"aggregate" if the compilation and its resulting copyright are not
used to limit the access or legal rights of the compilation's users
beyond what the individual works permit. Inclusion of a covered work
in an aggregate does not cause this License to apply to the other
parts of the aggregate.
6. Conveying Non-Source Forms.
You may convey a covered work in object code form under the terms
of sections 4 and 5, provided that you also convey the
machine-readable Corresponding Source under the terms of this License,
in one of these ways:
a) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by the
Corresponding Source fixed on a durable physical medium
customarily used for software interchange.
b) Convey the object code in, or embodied in, a physical product
(including a physical distribution medium), accompanied by a
written offer, valid for at least three years and valid for as
long as you offer spare parts or customer support for that product
model, to give anyone who possesses the object code either (1) a
copy of the Corresponding Source for all the software in the
product that is covered by this License, on a durable physical
medium customarily used for software interchange, for a price no
more than your reasonable cost of physically performing this
conveying of source, or (2) access to copy the
Corresponding Source from a network server at no charge.
c) Convey individual copies of the object code with a copy of the
written offer to provide the Corresponding Source. This
alternative is allowed only occasionally and noncommercially, and
only if you received the object code with such an offer, in accord
with subsection 6b.
d) Convey the object code by offering access from a designated
place (gratis or for a charge), and offer equivalent access to the
Corresponding Source in the same way through the same place at no
further charge. You need not require recipients to copy the
Corresponding Source along with the object code. If the place to
copy the object code is a network server, the Corresponding Source
may be on a different server (operated by you or a third party)
that supports equivalent copying facilities, provided you maintain
clear directions next to the object code saying where to find the
Corresponding Source. Regardless of what server hosts the
Corresponding Source, you remain obligated to ensure that it is
available for as long as needed to satisfy these requirements.
e) Convey the object code using peer-to-peer transmission, provided
you inform other peers where the object code and Corresponding
Source of the work are being offered to the general public at no
charge under subsection 6d.
A separable portion of the object code, whose source code is excluded
from the Corresponding Source as a System Library, need not be
included in conveying the object code work.
A "User Product" is either (1) a "consumer product", which means any
tangible personal property which is normally used for personal, family,
or household purposes, or (2) anything designed or sold for incorporation
into a dwelling. In determining whether a product is a consumer product,
doubtful cases shall be resolved in favor of coverage. For a particular
product received by a particular user, "normally used" refers to a
typical or common use of that class of product, regardless of the status
of the particular user or of the way in which the particular user
actually uses, or expects or is expected to use, the product. A product
is a consumer product regardless of whether the product has substantial
commercial, industrial or non-consumer uses, unless such uses represent
the only significant mode of use of the product.
"Installation Information" for a User Product means any methods,
procedures, authorization keys, or other information required to install
and execute modified versions of a covered work in that User Product from
a modified version of its Corresponding Source. The information must
suffice to ensure that the continued functioning of the modified object
code is in no case prevented or interfered with solely because
modification has been made.
If you convey an object code work under this section in, or with, or
specifically for use in, a User Product, and the conveying occurs as
part of a transaction in which the right of possession and use of the
User Product is transferred to the recipient in perpetuity or for a
fixed term (regardless of how the transaction is characterized), the
Corresponding Source conveyed under this section must be accompanied
by the Installation Information. But this requirement does not apply
if neither you nor any third party retains the ability to install
modified object code on the User Product (for example, the work has
been installed in ROM).
The requirement to provide Installation Information does not include a
requirement to continue to provide support service, warranty, or updates
for a work that has been modified or installed by the recipient, or for
the User Product in which it has been modified or installed. Access to a
network may be denied when the modification itself materially and
adversely affects the operation of the network or violates the rules and
protocols for communication across the network.
Corresponding Source conveyed, and Installation Information provided,
in accord with this section must be in a format that is publicly
documented (and with an implementation available to the public in
source code form), and must require no special password or key for
unpacking, reading or copying.
7. Additional Terms.
"Additional permissions" are terms that supplement the terms of this
License by making exceptions from one or more of its conditions.
Additional permissions that are applicable to the entire Program shall
be treated as though they were included in this License, to the extent
that they are valid under applicable law. If additional permissions
apply only to part of the Program, that part may be used separately
under those permissions, but the entire Program remains governed by
this License without regard to the additional permissions.
When you convey a copy of a covered work, you may at your option
remove any additional permissions from that copy, or from any part of
it. (Additional permissions may be written to require their own
removal in certain cases when you modify the work.) You may place
additional permissions on material, added by you to a covered work,
for which you have or can give appropriate copyright permission.
Notwithstanding any other provision of this License, for material you
add to a covered work, you may (if authorized by the copyright holders of
that material) supplement the terms of this License with terms:
a) Disclaiming warranty or limiting liability differently from the
terms of sections 15 and 16 of this License; or
b) Requiring preservation of specified reasonable legal notices or
author attributions in that material or in the Appropriate Legal
Notices displayed by works containing it; or
c) Prohibiting misrepresentation of the origin of that material, or
requiring that modified versions of such material be marked in
reasonable ways as different from the original version; or
d) Limiting the use for publicity purposes of names of licensors or
authors of the material; or
e) Declining to grant rights under trademark law for use of some
trade names, trademarks, or service marks; or
f) Requiring indemnification of licensors and authors of that
material by anyone who conveys the material (or modified versions of
it) with contractual assumptions of liability to the recipient, for
any liability that these contractual assumptions directly impose on
those licensors and authors.
All other non-permissive additional terms are considered "further
restrictions" within the meaning of section 10. If the Program as you
received it, or any part of it, contains a notice stating that it is
governed by this License along with a term that is a further
restriction, you may remove that term. If a license document contains
a further restriction but permits relicensing or conveying under this
License, you may add to a covered work material governed by the terms
of that license document, provided that the further restriction does
not survive such relicensing or conveying.
If you add terms to a covered work in accord with this section, you
must place, in the relevant source files, a statement of the
additional terms that apply to those files, or a notice indicating
where to find the applicable terms.
Additional terms, permissive or non-permissive, may be stated in the
form of a separately written license, or stated as exceptions;
the above requirements apply either way.
8. Termination.
You may not propagate or modify a covered work except as expressly
provided under this License. Any attempt otherwise to propagate or
modify it is void, and will automatically terminate your rights under
this License (including any patent licenses granted under the third
paragraph of section 11).
However, if you cease all violation of this License, then your
license from a particular copyright holder is reinstated (a)
provisionally, unless and until the copyright holder explicitly and
finally terminates your license, and (b) permanently, if the copyright
holder fails to notify you of the violation by some reasonable means
prior to 60 days after the cessation.
Moreover, your license from a particular copyright holder is
reinstated permanently if the copyright holder notifies you of the
violation by some reasonable means, this is the first time you have
received notice of violation of this License (for any work) from that
copyright holder, and you cure the violation prior to 30 days after
your receipt of the notice.
Termination of your rights under this section does not terminate the
licenses of parties who have received copies or rights from you under
this License. If your rights have been terminated and not permanently
reinstated, you do not qualify to receive new licenses for the same
material under section 10.
9. Acceptance Not Required for Having Copies.
You are not required to accept this License in order to receive or
run a copy of the Program. Ancillary propagation of a covered work
occurring solely as a consequence of using peer-to-peer transmission
to receive a copy likewise does not require acceptance. However,
nothing other than this License grants you permission to propagate or
modify any covered work. These actions infringe copyright if you do
not accept this License. Therefore, by modifying or propagating a
covered work, you indicate your acceptance of this License to do so.
10. Automatic Licensing of Downstream Recipients.
Each time you convey a covered work, the recipient automatically
receives a license from the original licensors, to run, modify and
propagate that work, subject to this License. You are not responsible
for enforcing compliance by third parties with this License.
An "entity transaction" is a transaction transferring control of an
organization, or substantially all assets of one, or subdividing an
organization, or merging organizations. If propagation of a covered
work results from an entity transaction, each party to that
transaction who receives a copy of the work also receives whatever
licenses to the work the party's predecessor in interest had or could
give under the previous paragraph, plus a right to possession of the
Corresponding Source of the work from the predecessor in interest, if
the predecessor has it or can get it with reasonable efforts.
You may not impose any further restrictions on the exercise of the
rights granted or affirmed under this License. For example, you may
not impose a license fee, royalty, or other charge for exercise of
rights granted under this License, and you may not initiate litigation
(including a cross-claim or counterclaim in a lawsuit) alleging that
any patent claim is infringed by making, using, selling, offering for
sale, or importing the Program or any portion of it.
11. Patents.
A "contributor" is a copyright holder who authorizes use under this
License of the Program or a work on which the Program is based. The
work thus licensed is called the contributor's "contributor version".
A contributor's "essential patent claims" are all patent claims
owned or controlled by the contributor, whether already acquired or
hereafter acquired, that would be infringed by some manner, permitted
by this License, of making, using, or selling its contributor version,
but do not include claims that would be infringed only as a
consequence of further modification of the contributor version. For
purposes of this definition, "control" includes the right to grant
patent sublicenses in a manner consistent with the requirements of
this License.
Each contributor grants you a non-exclusive, worldwide, royalty-free
patent license under the contributor's essential patent claims, to
make, use, sell, offer for sale, import and otherwise run, modify and
propagate the contents of its contributor version.
In the following three paragraphs, a "patent license" is any express
agreement or commitment, however denominated, not to enforce a patent
(such as an express permission to practice a patent or covenant not to
sue for patent infringement). To "grant" such a patent license to a
party means to make such an agreement or commitment not to enforce a
patent against the party.
If you convey a covered work, knowingly relying on a patent license,
and the Corresponding Source of the work is not available for anyone
to copy, free of charge and under the terms of this License, through a
publicly available network server or other readily accessible means,
then you must either (1) cause the Corresponding Source to be so
available, or (2) arrange to deprive yourself of the benefit of the
patent license for this particular work, or (3) arrange, in a manner
consistent with the requirements of this License, to extend the patent
license to downstream recipients. "Knowingly relying" means you have
actual knowledge that, but for the patent license, your conveying the
covered work in a country, or your recipient's use of the covered work
in a country, would infringe one or more identifiable patents in that
country that you have reason to believe are valid.
If, pursuant to or in connection with a single transaction or
arrangement, you convey, or propagate by procuring conveyance of, a
covered work, and grant a patent license to some of the parties
receiving the covered work authorizing them to use, propagate, modify
or convey a specific copy of the covered work, then the patent license
you grant is automatically extended to all recipients of the covered
work and works based on it.
A patent license is "discriminatory" if it does not include within
the scope of its coverage, prohibits the exercise of, or is
conditioned on the non-exercise of one or more of the rights that are
specifically granted under this License. You may not convey a covered
work if you are a party to an arrangement with a third party that is
in the business of distributing software, under which you make payment
to the third party based on the extent of your activity of conveying
the work, and under which the third party grants, to any of the
parties who would receive the covered work from you, a discriminatory
patent license (a) in connection with copies of the covered work
conveyed by you (or copies made from those copies), or (b) primarily
for and in connection with specific products or compilations that
contain the covered work, unless you entered into that arrangement,
or that patent license was granted, prior to 28 March 2007.
Nothing in this License shall be construed as excluding or limiting
any implied license or other defenses to infringement that may
otherwise be available to you under applicable patent law.
12. No Surrender of Others' Freedom.
If conditions are imposed on you (whether by court order, agreement or
otherwise) that contradict the conditions of this License, they do not
excuse you from the conditions of this License. If you cannot convey a
covered work so as to satisfy simultaneously your obligations under this
License and any other pertinent obligations, then as a consequence you may
not convey it at all. For example, if you agree to terms that obligate you
to collect a royalty for further conveying from those to whom you convey
the Program, the only way you could satisfy both those terms and this
License would be to refrain entirely from conveying the Program.
13. Use with the GNU Affero General Public License.
Notwithstanding any other provision of this License, you have
permission to link or combine any covered work with a work licensed
under version 3 of the GNU Affero General Public License into a single
combined work, and to convey the resulting work. The terms of this
License will continue to apply to the part which is the covered work,
but the special requirements of the GNU Affero General Public License,
section 13, concerning interaction through a network will apply to the
combination as such.
14. Revised Versions of this License.
The Free Software Foundation may publish revised and/or new versions of
the GNU General Public License from time to time. Such new versions will
be similar in spirit to the present version, but may differ in detail to
address new problems or concerns.
Each version is given a distinguishing version number. If the
Program specifies that a certain numbered version of the GNU General
Public License "or any later version" applies to it, you have the
option of following the terms and conditions either of that numbered
version or of any later version published by the Free Software
Foundation. If the Program does not specify a version number of the
GNU General Public License, you may choose any version ever published
by the Free Software Foundation.
If the Program specifies that a proxy can decide which future
versions of the GNU General Public License can be used, that proxy's
public statement of acceptance of a version permanently authorizes you
to choose that version for the Program.
Later license versions may give you additional or different
permissions. However, no additional obligations are imposed on any
author or copyright holder as a result of your choosing to follow a
later version.
15. Disclaimer of Warranty.
THERE IS NO WARRANTY FOR THE PROGRAM, TO THE EXTENT PERMITTED BY
APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT
HOLDERS AND/OR OTHER PARTIES PROVIDE THE PROGRAM "AS IS" WITHOUT WARRANTY
OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO,
THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR
PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE PROGRAM
IS WITH YOU. SHOULD THE PROGRAM PROVE DEFECTIVE, YOU ASSUME THE COST OF
ALL NECESSARY SERVICING, REPAIR OR CORRECTION.
16. Limitation of Liability.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING
WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MODIFIES AND/OR CONVEYS
THE PROGRAM AS PERMITTED ABOVE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY
GENERAL, SPECIAL, INCIDENTAL OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE
USE OR INABILITY TO USE THE PROGRAM (INCLUDING BUT NOT LIMITED TO LOSS OF
DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD
PARTIES OR A FAILURE OF THE PROGRAM TO OPERATE WITH ANY OTHER PROGRAMS),
EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF
SUCH DAMAGES.
17. Interpretation of Sections 15 and 16.
If the disclaimer of warranty and limitation of liability provided
above cannot be given local legal effect according to their terms,
reviewing courts shall apply local law that most closely approximates
an absolute waiver of all civil liability in connection with the
Program, unless a warranty or assumption of liability accompanies a
copy of the Program in return for a fee.
END OF TERMS AND CONDITIONS
How to Apply These Terms to Your New Programs
If you develop a new program, and you want it to be of the greatest
possible use to the public, the best way to achieve this is to make it
free software which everyone can redistribute and change under these terms.
To do so, attach the following notices to the program. It is safest
to attach them to the start of each source file to most effectively
state the exclusion of warranty; and each file should have at least
the "copyright" line and a pointer to where the full notice is found.
<one line to give the program's name and a brief idea of what it does.>
Copyright (C) <year> <name of author>
This program is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License as published by
the Free Software Foundation, either version 3 of the License, or
(at your option) any later version.
This program is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
GNU General Public License for more details.
You should have received a copy of the GNU General Public License
along with this program. If not, see <https://www.gnu.org/licenses/>.
Also add information on how to contact you by electronic and paper mail.
If the program does terminal interaction, make it output a short
notice like this when it starts in an interactive mode:
<program> Copyright (C) <year> <name of author>
This program comes with ABSOLUTELY NO WARRANTY; for details type `show w'.
This is free software, and you are welcome to redistribute it
under certain conditions; type `show c' for details.
The hypothetical commands `show w' and `show c' should show the appropriate
parts of the General Public License. Of course, your program's commands
might be different; for a GUI interface, you would use an "about box".
You should also get your employer (if you work as a programmer) or school,
if any, to sign a "copyright disclaimer" for the program, if necessary.
For more information on this, and how to apply and follow the GNU GPL, see
<https://www.gnu.org/licenses/>.
The GNU General Public License does not permit incorporating your program
into proprietary programs. If your program is a subroutine library, you
may consider it more useful to permit linking proprietary applications with
the library. If this is what you want to do, use the GNU Lesser General
Public License instead of this License. But first, please read
<https://www.gnu.org/licenses/why-not-lgpl.html>.
+165
View File
@@ -0,0 +1,165 @@
GNU LESSER GENERAL PUBLIC LICENSE
Version 3, 29 June 2007
Copyright (C) 2007 Free Software Foundation, Inc. <https://fsf.org/>
Everyone is permitted to copy and distribute verbatim copies
of this license document, but changing it is not allowed.
This version of the GNU Lesser General Public License incorporates
the terms and conditions of version 3 of the GNU General Public
License, supplemented by the additional permissions listed below.
0. Additional Definitions.
As used herein, "this License" refers to version 3 of the GNU Lesser
General Public License, and the "GNU GPL" refers to version 3 of the GNU
General Public License.
"The Library" refers to a covered work governed by this License,
other than an Application or a Combined Work as defined below.
An "Application" is any work that makes use of an interface provided
by the Library, but which is not otherwise based on the Library.
Defining a subclass of a class defined by the Library is deemed a mode
of using an interface provided by the Library.
A "Combined Work" is a work produced by combining or linking an
Application with the Library. The particular version of the Library
with which the Combined Work was made is also called the "Linked
Version".
The "Minimal Corresponding Source" for a Combined Work means the
Corresponding Source for the Combined Work, excluding any source code
for portions of the Combined Work that, considered in isolation, are
based on the Application, and not on the Linked Version.
The "Corresponding Application Code" for a Combined Work means the
object code and/or source code for the Application, including any data
and utility programs needed for reproducing the Combined Work from the
Application, but excluding the System Libraries of the Combined Work.
1. Exception to Section 3 of the GNU GPL.
You may convey a covered work under sections 3 and 4 of this License
without being bound by section 3 of the GNU GPL.
2. Conveying Modified Versions.
If you modify a copy of the Library, and, in your modifications, a
facility refers to a function or data to be supplied by an Application
that uses the facility (other than as an argument passed when the
facility is invoked), then you may convey a copy of the modified
version:
a) under this License, provided that you make a good faith effort to
ensure that, in the event an Application does not supply the
function or data, the facility still operates, and performs
whatever part of its purpose remains meaningful, or
b) under the GNU GPL, with none of the additional permissions of
this License applicable to that copy.
3. Object Code Incorporating Material from Library Header Files.
The object code form of an Application may incorporate material from
a header file that is part of the Library. You may convey such object
code under terms of your choice, provided that, if the incorporated
material is not limited to numerical parameters, data structure
layouts and accessors, or small macros, inline functions and templates
(ten or fewer lines in length), you do both of the following:
a) Give prominent notice with each copy of the object code that the
Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the object code with a copy of the GNU GPL and this license
document.
4. Combined Works.
You may convey a Combined Work under terms of your choice that,
taken together, effectively do not restrict modification of the
portions of the Library contained in the Combined Work and reverse
engineering for debugging such modifications, if you also do each of
the following:
a) Give prominent notice with each copy of the Combined Work that
the Library is used in it and that the Library and its use are
covered by this License.
b) Accompany the Combined Work with a copy of the GNU GPL and this license
document.
c) For a Combined Work that displays copyright notices during
execution, include the copyright notice for the Library among
these notices, as well as a reference directing the user to the
copies of the GNU GPL and this license document.
d) Do one of the following:
0) Convey the Minimal Corresponding Source under the terms of this
License, and the Corresponding Application Code in a form
suitable for, and under terms that permit, the user to
recombine or relink the Application with a modified version of
the Linked Version to produce a modified Combined Work, in the
manner specified by section 6 of the GNU GPL for conveying
Corresponding Source.
1) Use a suitable shared library mechanism for linking with the
Library. A suitable mechanism is one that (a) uses at run time
a copy of the Library already present on the user's computer
system, and (b) will operate properly with a modified version
of the Library that is interface-compatible with the Linked
Version.
e) Provide Installation Information, but only if you would otherwise
be required to provide such information under section 6 of the
GNU GPL, and only to the extent that such information is
necessary to install and execute a modified version of the
Combined Work produced by recombining or relinking the
Application with a modified version of the Linked Version. (If
you use option 4d0, the Installation Information must accompany
the Minimal Corresponding Source and Corresponding Application
Code. If you use option 4d1, you must provide the Installation
Information in the manner specified by section 6 of the GNU GPL
for conveying Corresponding Source.)
5. Combined Libraries.
You may place library facilities that are a work based on the
Library side by side in a single library together with other library
facilities that are not Applications and are not covered by this
License, and convey such a combined library under terms of your
choice, if you do both of the following:
a) Accompany the combined library with a copy of the same work based
on the Library, uncombined with any other library facilities,
conveyed under the terms of this License.
b) Give prominent notice with the combined library that part of it
is a work based on the Library, and explaining where to find the
accompanying uncombined form of the same work.
6. Revised Versions of the GNU Lesser General Public License.
The Free Software Foundation may publish revised and/or new versions
of the GNU Lesser General Public License from time to time. Such new
versions will be similar in spirit to the present version, but may
differ in detail to address new problems or concerns.
Each version is given a distinguishing version number. If the
Library as you received it specifies that a certain numbered version
of the GNU Lesser General Public License "or any later version"
applies to it, you have the option of following the terms and
conditions either of that published version or of any later version
published by the Free Software Foundation. If the Library as you
received it does not specify a version number of the GNU Lesser
General Public License, you may choose any version of the GNU Lesser
General Public License ever published by the Free Software Foundation.
If the Library as you received it specifies that a proxy can decide
whether future versions of the GNU Lesser General Public License shall
apply, that proxy's public statement of acceptance of any version is
permanent authorization for you to choose that version for the
Library.
+28 -28
View File
@@ -1,29 +1,31 @@
FROM avatao/frontend-tutorial-framework:chausie-20190912 as frontend
FROM avatao/debian:buster
RUN curl -sL https://deb.nodesource.com/setup_8.x | sudo -E bash - &&\
curl -sS https://dl.yarnpkg.com/debian/pubkey.gpg | sudo apt-key add - &&\
echo "deb https://dl.yarnpkg.com/debian/ stable main" | sudo tee /etc/apt/sources.list.d/yarn.list &&\
apt-get update &&\
RUN apt-get update &&\
apt-get install -y --no-install-recommends \
nodejs \
yarn \
supervisor \
libzmq5 \
nginx \
gettext-base &&\
rm -rf /var/lib/apt/lists/* &&\
jq \
gettext-base &&\
rm -rf /var/lib/apt/lists/* &&\
ln -sf /bin/bash /bin/sh
COPY requirements.txt /tmp
RUN pip3 install -r /tmp/requirements.txt
RUN curl -Ls https://github.com/krallin/tini/releases/download/v0.18.0/tini-amd64 --output /bin/init &&\
echo "12d20136605531b09a2c2dac02ccee85e1b874eb322ef6baf7561cd93f93c855 /bin/init" |\
sha256sum --check --status &&\
chmod 755 /bin/init
ENV TFW_PUBLIC_PORT=8888 \
TFW_WEB_PORT=4242 \
TFW_LOGIN_APP_PORT=6666 \
TFW_TERMINADO_PORT=7878 \
TFW_SUPERVISOR_HTTP_PORT=9001 \
TFW_PUBLISHER_PORT=7654 \
TFW_RECEIVER_PORT=8765
TFW_PUB_PORT=7654 \
TFW_PULL_PORT=8765
EXPOSE ${TFW_PUBLIC_PORT}
@@ -34,28 +36,30 @@ ENV PYTHONPATH="/usr/local/lib" \
TFW_NGINX_CONF="/etc/nginx/nginx.conf" \
TFW_NGINX_DEFAULT="/etc/nginx/sites-enabled/default" \
TFW_NGINX_COMPONENTS="/etc/nginx/components" \
TFW_LIB_DIR="/usr/local/lib/" \
TFW_TERMINADO_DIR="/tmp/terminado_server" \
TFW_LIB_DIR="/usr/local/lib" \
TFW_FRONTEND_DIR="/srv/frontend" \
TFW_DIR="/.tfw" \
TFW_SERVER_DIR="/.tfw/tfw_server" \
TFW_AUTH_KEY="/tmp/tfw-auth.key" \
TFW_LOGS_DIR="/var/log/tfw" \
TFW_PIPES_DIR="/run/tfw" \
TFW_SNAPSHOTS_DIR="/tmp/tfw-snapshots" \
TFW_HISTFILE="/home/${AVATAO_USER}/.bash_history" \
PROMPT_COMMAND="history -a"
COPY bashrc /tmp
RUN echo "export HISTFILE=${TFW_HISTFILE}" >> /tmp/bashrc &&\
cat /tmp/bashrc >> /home/${AVATAO_USER}/.bashrc
COPY bashrc supervisor/tfw_init.sh /tmp/
COPY supervisor/supervisord.conf ${TFW_SUPERVISORD_CONF}
COPY supervisor/components/ ${TFW_SUPERVISORD_COMPONENTS}
COPY nginx/nginx.conf ${TFW_NGINX_CONF}
COPY nginx/default.conf ${TFW_NGINX_DEFAULT}
COPY nginx/components/ ${TFW_NGINX_COMPONENTS}
COPY lib LICENSE ${TFW_LIB_DIR}
COPY tfw ${TFW_LIB_DIR}/tfw
COPY supervisor/tfw_server.py ${TFW_SERVER_DIR}/
COPY --from=frontend /srv/dist ${TFW_FRONTEND_DIR}
RUN for dir in "${TFW_LIB_DIR}"/{tfw,tao,envvars} "/etc/nginx" "/etc/supervisor"; do \
chown -R root:root "$dir" && chmod -R 700 "$dir"; \
done
VOLUME ["${TFW_LOGS_DIR}", "${TFW_PIPES_DIR}"]
ONBUILD ARG BUILD_CONTEXT="solvable"
ONBUILD ARG NOFRONTEND=""
ONBUILD COPY ${BUILD_CONTEXT}/nginx/ ${TFW_NGINX_COMPONENTS}
ONBUILD COPY ${BUILD_CONTEXT}/supervisor/ ${TFW_SUPERVISORD_COMPONENTS}
@@ -63,11 +67,7 @@ ONBUILD COPY ${BUILD_CONTEXT}/supervisor/ ${TFW_SUPERVISORD_COMPONENTS}
ONBUILD RUN for f in "${TFW_NGINX_DEFAULT}" ${TFW_NGINX_COMPONENTS}/*.conf; do \
envsubst "$(printenv | cut -d= -f1 | grep TFW_ | sed -e 's/^/$/g')" < $f > $f~ && mv $f~ $f ;\
done
ONBUILD VOLUME ["/etc/nginx", "/var/lib/nginx", "/var/log/nginx"]
ONBUILD VOLUME ["/etc/nginx", "/var/lib/nginx", "/var/log/nginx", "${TFW_LIB_DIR}/tfw"]
ONBUILD COPY ${BUILD_CONTEXT}/frontend /data/
ONBUILD RUN test -z "${NOFRONTEND}" && cd /data && yarn install --frozen-lockfile || :
ONBUILD RUN test -z "${NOFRONTEND}" && cd /data && yarn build --no-progress || :
ONBUILD RUN test -z "${NOFRONTEND}" && mv /data/dist ${TFW_FRONTEND_DIR} && rm -rf /data || :
CMD exec supervisord --nodaemon
ENTRYPOINT ["/bin/init", "--"]
CMD exec supervisord --nodaemon --configuration ${TFW_SUPERVISORD_CONF}
-12
View File
@@ -1,12 +0,0 @@
AVATAO CONFIDENTIAL
Copyright (C) 2018 Avatao.com Innovative Learning Kft.
All Rights Reserved.
All source code, configuration files and documentation contained herein
is, and remains the exclusive property of Avatao.com Innovative Learning Kft.
The intellectual and technical concepts contained herein are proprietary
to Avatao.com Innovative Learning Kft. and are protected by trade secret
or copyright law. Dissemination of this information or reproduction of
this material is strictly forbidden unless prior written permission is
obtained from Avatao.com Innovative Learning Kft.
+260 -3
View File
@@ -20,6 +20,20 @@ Frontend components use websockets to connect to the TFW server, to which you ca
![TFW architecture](docs/tfw_architecture.png)
### Networking details
Event handlers connect to the TFW server using ZMQ.
They receive messages on their `SUB`(scribe) sockets, which are connected to the `PUB`(lish) socket of the server.
Event handlers reply on their `PUSH` socket, then their messages are received on the `PULL` socket of the server.
The TFW server is basically just a fancy proxy.
It's behaviour is quite simple: it proxies every message received from the fontend to the event handlers and vice versa.
The server is also capable of "mirroring" messages back to their source.
This is useful for communication between event handlers or frontend components (event handler to event handler or frontend component to frontend component communication).
Components can also broadcast messages (broadcasted messages are received both by event handlers and the frontend as well).
### Event handlers
Imagine event handlers as callbacks that are invoked when TFW receives a specific type of message. For instance, you could send a message to the framework when the user does something of note.
@@ -33,6 +47,8 @@ Our pre-made event handlers are written in Python3, but you can write event hand
This makes the framework really flexible: you can demonstrate the concepts you want to in any language while using the same set of tools provided by TFW.
Inside Avatao this means that any of the content teams can use the framework with ease.
To implement an event handler in Python3 you should subclass the `EventHandlerBase` or `FSMAwareEventHandler` class in `tfw.event_handler_base` (the first provides a minimal working `EventHandler`, the second allows you to execute code on FSM events).
### FSM
Another unique feature of the framework is the FSM finite state machine representing the state of your challenge.
@@ -60,26 +76,267 @@ The TFW message format:
```text
{
"key: "some identifier used for addressing",
"key: ...some identifier used for addressing...,
"data":
{
...
JSON object carrying anything, preferably cats
...
},
"trigger": "FSM action"
"trigger": ...FSM action...,
"signature": ...HMAC signature for authenticated messages...,
"seq": ...sequence number...
}
```
- The `key` field is used by TFW for addressing and every message must have one (it can be an empty string though)
- The `data` object can contain anything you might want to send
- The `trigger` key is an optional field that triggers an FSM action with that name from the current state (whatever that might be)
- The `signature` field is present on authenticated messages (such as `fsm_update`s)
- The `seq` key is a counter incremented with each proxied message in the TFW server
To mirror messages back to their sources you can use a special messaging format, in which the message to be mirrored is enveloped inside the `data` field of the outer message:
```text
"key": "mirror",
"data":
{
...
The message you want to mirror (with it's own "key" and "data" fields)
...
}
```
Broadcasting messages is possible in a similar manner by using `"key": "broadcast"` in the outer message.
## Where to go next
Most of the components you need have docstrings included (hang on tight, this is work in progress) refer to them for usage info.
In the `docs` folder you can find our Sphinx-based API documentation, which you can build using the `hack/tfw.sh` script in the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repository.
In the `docs` folder you can find our Sphinx-based documentation, which you can build using the `hack/tfw.sh` script in the [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework) repository.
To get started you should take a look at [test-tutorial-framework](https://github.com/avatao-content/test-tutorial-framework), which serves as an example project as well.
## API
APIs exposed by our pre-witten event handlers are documented here.
### IdeEventHandler
This event handler is responsible for reading and writing files shown in the frontend code editor.
You can read the content of the currently selected file like so:
```
{
"key": "ide",
"data":
{
"command": "read"
}
}
```
Use the following message to overwrite the content of the currently selected file:
```
{
"key": "ide",
"data":
{
"command": "write",
"content": ...string...
}
}
```
To select a file use the following message:
```
{
"key": "ide",
"data":
{
"command": "select",
"filename": ...string...
}
}
```
You can switch to a new working directory using this message (note that the directory must be in `allowed_directories`):
```
{
"key": "ide",
"data":
{
"command": "selectdir",
"directory": ...string...
}
}
```
Overwriting the current list of excluded file patterns is possible with this message:
```
{
"key": "ide",
"data":
{
"command": "exclude",
"exclude": ...array of strings...
}
}
```
### TerminalEventHandler
Event handler responsible for running a backend for `xterm.js` to connect to (frontend terminal backend).
By default callbacks on terminal history are invoked *as soon as* a command starts to execute in the terminal (they do not wait for the started command to finish, the callback may even run in paralell with the command).
If you want to wait for them and invoke your callbacks *after* the command has finished, please set the `TFW_DELAY_HISTAPPEND` envvar to `1`.
Practically this can be done by appending an `export` to the user's `.bashrc` file from your `Dockerfile`, like so:
`RUN echo "export TFW_DELAY_HISTAPPEND=1" >> /home/${AVATAO_USER}/.bashrc`
Writing to the terminal:
```
{
"key": "shell",
"data":
{
"command": "write",
"value": ...string...
}
}
```
You can read terminal command history like so:
```
{
"key": "shell",
"data":
{
"command": "read",
"count": ...number...
}
}
```
### ProcessManagingEventHandler
This event handler is responsible for managing processes controlled by supervisord.
Starting, stopping and restarting supervisor processes can be done using similar messages (where `command` is `start`, `stop` or `restart`):
```
{
"key": "processmanager",
"data":
{
"command": ...string...,
"process_name": ...string...
}
}
```
### LogMonitoringEventHandler
Event handler emitting real time logs (`stdout` and `stderr`) from supervisord processes.
To change which supervisor process is monitored use this message:
```
{
"key": "logmonitor",
"data" :
{
"command": "process_name",
"value": ...string...
}
}
```
To set the tail length of logs (the monitor will send back the last `value` characters of the log):
```
{
"key": "logmonitor",
"data" :
{
"command": "log_tail",
"value": ...number...
}
}
```
### FSMManagingEventHandler
This event handler controls the TFW finite state machine (FSM).
To attempt executing a trigger on the FSM use (this will also generate an FSM update message):
```
{
"key": "fsm",
"data" :
{
"command": "trigger",
"value": ...string...
}
}
```
To force the broadcasting of an FSM update you can use this message:
```
{
"key": "fsm",
"data" :
{
"command": "update"
}
}
```
This event handler broadcasts FSM update messages after handling commands in the following format:
```
{
"key": "fsm_update",
"data" :
{
"current_state": ...string...,
"valid_transitions": ...array of {"trigger": ...string...} objects...
}
}
```
### DirectorySnapshottingEventHandler
Event handler capable of taking and restoring snapshots of directories (saving and restoring directory contens).
You can take a snapshot of the directories with the following message:
```
{
"key": "snapshot",
"data" :
{
"command": "take_snapshot"
}
}
```
To restore the state of the files in the directories use:
```
{
"key": "snapshot",
"data" :
{
"command": "restore_snapshot",
"value": ...date string (can parse ISO 8601, unix timestamp, etc.)...
}
}
```
It is also possible to exclude files that match given patterns (formatted like lines in `.gitignore` files):
```
{
"key": "snapshot",
"data" :
{
"command": "exclude",
"value": ...list of patterns to exclude from snapshots...
}
}
```
+1 -1
View File
@@ -1 +1 @@
bombay
chausie
+1 -1
View File
@@ -9,7 +9,7 @@ _precmd()
{
[ -z "$AT_PROMPT" ] && return
unset AT_PROMPT
history -a
[ -z "$TFW_DELAY_HISTAPPEND" ] && history -a
}
trap _precmd DEBUG
+7
View File
@@ -23,3 +23,10 @@ Components
.. autoclass:: BashMonitor
:members:
.. autoclass:: FSMManagingEventHandler
:members:
.. autoclass:: CommandsEqual
:members:
+1 -1
View File
@@ -26,7 +26,7 @@ author = 'Kristóf Tóth'
# The short X.Y version
version = ''
# The full version, including alpha/beta/rc tags
release = 'bombay'
release = 'mainecoon'
# -- General configuration ---------------------------------------------------
+5 -2
View File
@@ -3,10 +3,13 @@ Event handler base classes
Subclass these to create your cusom event handlers.
.. automodule:: tfw
.. automodule:: tfw.event_handler_base
.. autoclass:: EventHandlerBase
:members:
.. autoclass:: TriggeredEventHandler
.. autoclass:: FSMAwareEventHandler
:members:
.. autoclass:: BroadcastingEventHandler
:members:
+4 -1
View File
@@ -3,10 +3,13 @@ FSM base classes
Subclass these to create an FSM that fits your tutorial/challenge.
.. automodule:: tfw
.. automodule:: tfw.fsm
.. autoclass:: FSMBase
:members:
.. autoclass:: LinearFSM
:members:
.. autoclass:: YamlFSM
:members:
+10
View File
@@ -36,6 +36,16 @@ These are pre-written components for you to use, such as our IDE, terminal or co
components/*
Utility
-------
These are useful decorators, mixins and helpers to make common dev tasks easier.
.. toctree::
:glob:
utility/*
Indices and tables
==================
+6 -1
View File
@@ -6,7 +6,7 @@ Networking
.. autoclass:: TFWServerConnector
:members:
.. automodule:: tfw.networking.event_handlers
.. automodule:: tfw.networking.event_handlers.server_connector
.. autoclass:: ServerUplinkConnector
:members:
@@ -15,3 +15,8 @@ Networking
.. autoclass:: MessageSender
:members:
.. automodule:: tfw.networking.fsm_aware
.. autoclass:: FSMAware
:members:
+10
View File
@@ -0,0 +1,10 @@
TFW decorators
--------------
.. automodule:: tfw.decorators.rate_limiter
.. autoclass:: RateLimiter
:members:
.. autoclass:: AsyncRateLimiter
:members:
Binary file not shown.

Before

Width:  |  Height:  |  Size: 40 KiB

After

Width:  |  Height:  |  Size: 46 KiB

-2
View File
@@ -1,2 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
-4
View File
@@ -1,4 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .envvars import TAOENV
-6
View File
@@ -1,6 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from envvars import LazyEnvironment
TAOENV = LazyEnvironment('AVATAO_', 'taoenvtuple').environment
-6
View File
@@ -1,6 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .event_handler_base import EventHandlerBase, TriggeredEventHandler
from .fsm_base import FSMBase
from .linear_fsm import LinearFSM
-10
View File
@@ -1,10 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .directory_monitoring_event_handler import DirectoryMonitoringEventHandler
from .process_managing_event_handler import ProcessManagingEventHandler
from .terminal_event_handler import TerminalEventHandler
from .ide_event_handler import IdeEventHandler
from .history_monitor import HistoryMonitor, BashMonitor, GDBMonitor
from .terminal_commands import TerminalCommands
from .log_monitoring_event_handler import LogMonitoringEventHandler
-81
View File
@@ -1,81 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from functools import wraps
from watchdog.events import FileSystemEventHandler as FileSystemWatchdogEventHandler
from tfw.networking.event_handlers import ServerUplinkConnector
from tfw.decorators import RateLimiter
from tfw.mixins import ObserverMixin
from tfw.config.logs import logging
LOG = logging.getLogger(__name__)
class DirectoryMonitor(ObserverMixin):
def __init__(self, ide_key, directories):
self.eventhandler = IdeReloadWatchdogEventHandler(ide_key)
for directory in directories:
self.observer.schedule(self.eventhandler, directory, recursive=True)
self.pause, self.resume = self.eventhandler.pause, self.eventhandler.resume
@property
def ignore(self):
return self.eventhandler.ignore
@ignore.setter
def ignore(self, value):
self.eventhandler.ignore = value if value >= 0 else 0
@property
def pauser(self):
return DirectoryMonitor.Pauser(self)
class Pauser:
def __init__(self, directory_monitor):
self.directorymonitor = directory_monitor
def __enter__(self):
self.directorymonitor.pause()
def __exit__(self, exc_type, exc_val, exc_tb):
self.directorymonitor.resume()
class IdeReloadWatchdogEventHandler(FileSystemWatchdogEventHandler):
def __init__(self, ide_key):
super().__init__()
self.ide_key = ide_key
self.uplink = ServerUplinkConnector()
self._paused = False
self.ignore = 0
def pause(self):
self._paused = True
def resume(self):
self._paused = False
@RateLimiter(rate_per_second=2)
def on_modified(self, event):
if self._paused:
return
if self.ignore > 0:
self.ignore = self.ignore - 1
return
LOG.debug(event)
self.uplink.send({
'key': self.ide_key,
'data': {'command': 'reload'}
})
def with_monitor_paused(fun):
@wraps(fun)
def wrapper(self, *args, **kwargs):
if self.monitor:
with self.monitor.pauser:
return fun(self, *args, **kwargs)
return fun(self, *args, **kwargs)
return wrapper
@@ -1,70 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from os.path import isdir, exists
from tfw import EventHandlerBase
from tfw.config.logs import logging
from tfw.mixins import MonitorManagerMixin
from .directory_monitor import DirectoryMonitor
LOG = logging.getLogger(__name__)
class DirectoryMonitoringEventHandler(EventHandlerBase, MonitorManagerMixin):
def __init__(self, key, directory):
super().__init__(key)
self._directory = directory
MonitorManagerMixin.__init__(
self,
DirectoryMonitor,
key,
self._directory
)
self.commands = {
'pause': self.pause,
'resume': self.resume,
'ignore': self.ignore,
'selectdir': self.selectdir
}
@property
def directory(self):
return self._directory
@directory.setter
def directory(self, directory):
if not exists(directory) or not isdir(directory):
raise EnvironmentError('No such directory!')
self._directory = directory
def handle_event(self, message):
try:
message['data'] = self.commands[message['data']['command']](message['data'])
return message
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def pause(self, data):
self.monitor.pause()
return data
def resume(self, data):
self.monitor.resume()
return data
def ignore(self, data):
self.monitor.ignore += data['ignore']
return data
def selectdir(self, data):
try:
self.directory = data['directory']
self.reload_monitor()
return data
except EnvironmentError:
LOG.error('Failed to switch directory!')
def cleanup(self):
self.monitor.stop()
-254
View File
@@ -1,254 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from os.path import isfile, join, relpath, exists, isdir, realpath
from glob import glob
from fnmatch import fnmatchcase
from collections import Iterable
from tfw import EventHandlerBase
from tfw.mixins import MonitorManagerMixin
from tfw.config.logs import logging
from .directory_monitor import DirectoryMonitor
LOG = logging.getLogger(__name__)
class FileManager: # pylint: disable=too-many-instance-attributes
def __init__(self, working_directory, allowed_directories, selected_file=None, exclude=None):
self._exclude, self.exclude = None, exclude
self._allowed_directories, self.allowed_directories = None, allowed_directories
self._workdir, self.workdir = None, working_directory
self._filename, self.filename = None, selected_file or self.files[0]
@property
def exclude(self):
return self._exclude
@exclude.setter
def exclude(self, exclude):
if exclude is None:
return
if not isinstance(exclude, Iterable):
raise TypeError('Exclude must be Iterable!')
self._exclude = exclude
@property
def workdir(self):
return self._workdir
@workdir.setter
def workdir(self, directory):
if not exists(directory) or not isdir(directory):
raise EnvironmentError(f'"{directory}" is not a directory!')
if not self._is_in_allowed_dir(directory):
raise EnvironmentError(f'Directory "{directory}" is not allowed!')
self._workdir = directory
@property
def allowed_directories(self):
return self._allowed_directories
@allowed_directories.setter
def allowed_directories(self, directories):
self._allowed_directories = directories
@property
def filename(self):
return self._filename
@filename.setter
def filename(self, filename):
if filename not in self.files:
raise EnvironmentError('No such file in workdir!')
self._filename = filename
@property
def files(self):
return [
self._relpath(file)
for file in glob(join(self._workdir, '**/*'), recursive=True)
if isfile(file)
and self._is_in_allowed_dir(file)
and not self._is_blacklisted(file)
]
@property
def file_contents(self):
with open(self._filepath(self.filename), 'r', errors='surrogateescape') as ifile:
return ifile.read()
@file_contents.setter
def file_contents(self, value):
with open(self._filepath(self.filename), 'w', errors='surrogateescape') as ofile:
ofile.write(value)
def _is_in_allowed_dir(self, path):
return any(
realpath(path).startswith(allowed_dir)
for allowed_dir in self.allowed_directories
)
def _is_blacklisted(self, file):
return any(
fnmatchcase(file, blacklisted)
for blacklisted in self.exclude
)
def _filepath(self, filename):
return join(self._workdir, filename)
def _relpath(self, filename):
return relpath(self._filepath(filename), start=self._workdir)
class IdeEventHandler(EventHandlerBase, MonitorManagerMixin):
# pylint: disable=too-many-arguments
"""
Event handler implementing the backend of our browser based IDE.
By default all files in the directory specified in __init__ are displayed
on the fontend. Note that this is a stateful component.
When any file in the selected directory changes they are automatically refreshed
on the frontend (this is done by listening to inotify events).
This EventHandler accepts messages that have a data['command'] key specifying
a command to be executed.
The API of each command is documented in their respective handler.
"""
def __init__(self, key, directory, allowed_directories, selected_file=None, exclude=None):
"""
:param key: the key this instance should listen to
:param directory: working directory which the EventHandler should serve files from
:param allowed_directories: list of directories that can be switched to using selectdir
:param selected_file: file that is selected by default
:param exclude: list of filenames that should not appear between files (for .o, .pyc, etc.)
"""
super().__init__(key)
try:
self.filemanager = FileManager(
allowed_directories=allowed_directories,
working_directory=directory,
selected_file=selected_file,
exclude=exclude
)
except IndexError:
raise EnvironmentError(
f'No file(s) in IdeEventHandler working_directory "{directory}"!'
)
MonitorManagerMixin.__init__(
self,
DirectoryMonitor,
self.key,
self.filemanager.allowed_directories
)
self.commands = {
'read': self.read,
'write': self.write,
'select': self.select,
'selectdir': self.select_dir,
'exclude': self.exclude
}
def read(self, data):
"""
Read the currently selected file.
:return dict: message with the contents of the file in data['content']
"""
try:
data['content'] = self.filemanager.file_contents
except PermissionError:
data['content'] = 'You have no permission to open that file :('
except FileNotFoundError:
data['content'] = 'This file was removed :('
except Exception: # pylint: disable=broad-except
data['content'] = 'Failed to read file :('
return data
def write(self, data):
"""
Overwrites a file with the desired string.
:param data: TFW message data containing key 'content'
(new file content)
"""
self.monitor.ignore = self.monitor.ignore + 1
try:
self.filemanager.file_contents = data['content']
except Exception: # pylint: disable=broad-except
LOG.exception('Error writing file!')
del data['content']
return data
def select(self, data):
"""
Selects a file from the current directory.
:param data: TFW message data containing 'filename'
(name of file to select relative to the current directory)
"""
try:
self.filemanager.filename = data['filename']
except EnvironmentError:
LOG.exception('Failed to select file "%s"', data['filename'])
return data
def select_dir(self, data):
"""
Select a new working directory to display files from.
:param data: TFW message data containing 'directory'
(absolute path of diretory to select.
must be a path whitelisted in
self.allowed_directories)
"""
try:
self.filemanager.workdir = data['directory']
self.reload_monitor()
try:
self.filemanager.filename = self.filemanager.files[0]
self.read(data)
except IndexError:
data['content'] = 'No files in this directory :('
except EnvironmentError as err:
LOG.error('Failed to select directory "%s". Reason: %s', data['directory'], str(err))
return data
def exclude(self, data):
"""
Overwrite list of excluded files
:param data: TFW message data containing 'exclude'
(list of unix-style filename patterns to be excluded,
e.g.: ["\*.pyc", "\*.o")
"""
try:
self.filemanager.exclude = list(data['exclude'])
except TypeError:
LOG.error('Exclude must be Iterable!')
return data
def attach_fileinfo(self, data):
"""
Basic information included in every response to the frontend.
"""
data['filename'] = self.filemanager.filename
data['files'] = self.filemanager.files
data['directory'] = self.filemanager.workdir
def handle_event(self, message):
try:
data = message['data']
message['data'] = self.commands[data['command']](data)
self.attach_fileinfo(data)
return message
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def cleanup(self):
self.monitor.stop()
-56
View File
@@ -1,56 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import logging
from os.path import dirname
from watchdog.events import PatternMatchingEventHandler as PatternMatchingWatchdogEventHandler
from tfw.networking.event_handlers import ServerUplinkConnector
from tfw.decorators import RateLimiter
from tfw.mixins import ObserverMixin, SupervisorLogMixin
class LogMonitor(ObserverMixin):
def __init__(self, process_name, log_tail=0):
self.prevent_log_recursion()
event_handler = SendLogWatchdogEventHandler(
process_name,
log_tail=log_tail
)
self.observer.schedule(
event_handler,
event_handler.path
)
@staticmethod
def prevent_log_recursion():
# This is done to prevent inotify event logs triggering themselves (infinite log recursion)
logging.getLogger('watchdog.observers.inotify_buffer').propagate = False
class SendLogWatchdogEventHandler(PatternMatchingWatchdogEventHandler, SupervisorLogMixin):
def __init__(self, process_name, log_tail=0):
self.process_name = process_name
self.procinfo = self.supervisor.getProcessInfo(self.process_name)
super().__init__([
self.procinfo['stdout_logfile'],
self.procinfo['stderr_logfile']
])
self.uplink = ServerUplinkConnector()
self.log_tail = log_tail
@property
def path(self):
return dirname(self.procinfo['stdout_logfile'])
@RateLimiter(rate_per_second=5)
def on_modified(self, event):
self.uplink.send({
'key': 'processlog',
'data': {
'command': 'new_log',
'stdout': self.read_stdout(self.process_name, tail=self.log_tail),
'stderr': self.read_stderr(self.process_name, tail=self.log_tail)
}
})
@@ -1,67 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from tfw import EventHandlerBase
from tfw.mixins import MonitorManagerMixin
from tfw.config.logs import logging
from .log_monitor import LogMonitor
LOG = logging.getLogger(__name__)
class LogMonitoringEventHandler(EventHandlerBase, MonitorManagerMixin):
"""
Monitors the output of a supervisor process (stdout, stderr) and
sends the results to the frontend.
Accepts messages that have a data['command'] key specifying
a command to be executed.
The API of each command is documented in their respective handler.
"""
def __init__(self, key, process_name, log_tail=0):
super().__init__(key)
self.process_name = process_name
self.log_tail = log_tail
MonitorManagerMixin.__init__(
self,
LogMonitor,
self.process_name,
self.log_tail
)
self.command_handlers = {
'process_name': self.handle_process_name,
'log_tail': self.handle_log_tail
}
def handle_event(self, message):
try:
data = message['data']
self.command_handlers[data['command']](data)
self.reload_monitor()
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_process_name(self, data):
"""
Changes the monitored process.
:param data: TFW message data containing 'value'
(name of the process to monitor)
"""
self.set_monitor_args(data['value'], self.log_tail)
def handle_log_tail(self, data):
"""
Sets tail length of the log the monitor will send
to the frontend (the monitor will send back the last
'value' characters of the log).
:param data: TFW message data containing 'value'
(new tail length)
"""
self.set_monitor_args(self.process_name, data['value'])
def cleanup(self):
self.monitor.stop()
@@ -1,65 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from xmlrpc.client import Fault as SupervisorFault
from tfw import EventHandlerBase
from tfw.mixins import SupervisorMixin, SupervisorLogMixin
from tfw.config.logs import logging
from .directory_monitor import with_monitor_paused
LOG = logging.getLogger(__name__)
class ProcessManager(SupervisorMixin, SupervisorLogMixin):
def __init__(self):
self.commands = {
'start': self.start_process,
'stop': self.stop_process,
'restart': self.restart_process
}
def __call__(self, command, process_name):
return self.commands[command](process_name)
class ProcessManagingEventHandler(EventHandlerBase):
"""
Event handler that can manage processes managed by supervisor.
This EventHandler accepts messages that have a data['command'] key specifying
a command to be executed.
Every message must contain a data['process_name'] field with the name of the
process to manage. This is the name specified in supervisor config files like so:
[program:someprogram]
Commands available: start, stop, restart, readlog
(the names are as self-documenting as it gets)
"""
def __init__(self, key, dirmonitor=None, log_tail=0):
super().__init__(key)
self.key = key
self.monitor = dirmonitor
self.processmanager = ProcessManager()
self.log_tail = log_tail
@with_monitor_paused
def handle_event(self, message):
try:
data = message['data']
try:
self.processmanager(data['command'], data['process_name'])
except SupervisorFault as fault:
message['data']['error'] = fault.faultString
finally:
message['data']['stdout'] = self.processmanager.read_stdout(
data['process_name'],
self.log_tail
)
message['data']['stderr'] = self.processmanager.read_stderr(
data['process_name'],
self.log_tail
)
return message
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
@@ -1,87 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from tfw import EventHandlerBase
from tfw.config import TFWENV
from tfw.config.logs import logging
from tao.config import TAOENV
from .terminado_mini_server import TerminadoMiniServer
LOG = logging.getLogger(__name__)
class TerminalEventHandler(EventHandlerBase):
"""
Event handler responsible for managing terminal sessions for frontend xterm
sessions to connect to. You need to instanciate this in order for frontend
terminals to work.
This EventHandler accepts messages that have a data['command'] key specifying
a command to be executed.
The API of each command is documented in their respective handler.
"""
def __init__(self, key, monitor):
"""
:param key: key this EventHandler listens to
:param monitor: tfw.components.HistoryMonitor instance to read command history from
"""
super().__init__(key)
self.working_directory = TFWENV.TERMINADO_DIR
self._historymonitor = monitor
bash_as_user_cmd = ['sudo', '-u', TAOENV.USER, 'bash']
self.terminado_server = TerminadoMiniServer(
'/terminal',
TFWENV.TERMINADO_PORT,
TFWENV.TERMINADO_WD,
bash_as_user_cmd
)
self.commands = {
'write': self.write,
'read': self.read
}
if self._historymonitor:
self._historymonitor.watch()
self.terminado_server.listen()
@property
def historymonitor(self):
return self._historymonitor
def handle_event(self, message):
LOG.debug('TerminadoEventHandler received event: %s', message)
try:
data = message['data']
message['data'] = self.commands[data['command']](data)
return message
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def write(self, data):
"""
Writes a string to the terminal session (on the pty level).
Useful for pre-typing and executing commands for the user.
:param data: TFW message data containing 'value'
(command to be written to the pty)
"""
self.terminado_server.pty.write(data['value'])
def read(self, data):
"""
Reads the history of commands executed.
:param data: TFW message data containing 'count'
(the number of history elements to return)
:return dict: message with list of commands in data['history']
"""
data['count'] = int(data.get('count', 1))
if self.historymonitor:
data['history'] = self.historymonitor.history[-data['count']:]
return data
def cleanup(self):
if self.historymonitor:
self.historymonitor.stop()
-4
View File
@@ -1,4 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .envvars import TFWENV
-6
View File
@@ -1,6 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from envvars import LazyEnvironment
TFWENV = LazyEnvironment('TFW_', 'tfwenvtuple').environment
-6
View File
@@ -1,6 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import logging
logging.basicConfig(level=logging.DEBUG)
-5
View File
@@ -1,5 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .rate_limiter import RateLimiter
from .lazy_property import lazy_property
-25
View File
@@ -1,25 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from functools import wraps
from time import time, sleep
class RateLimiter:
def __init__(self, rate_per_second):
self.min_interval = 1 / float(rate_per_second)
self.last_call = time()
def __call__(self, fun):
@wraps(fun)
def wrapper(*args, **kwargs):
self._limit_rate()
fun(*args, **kwargs)
return wrapper
def _limit_rate(self):
since_last_call = time() - self.last_call
to_next_call = self.min_interval - since_last_call
self.last_call = time()
if to_next_call > 0:
sleep(to_next_call)
-104
View File
@@ -1,104 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from abc import ABC, abstractmethod
from tfw.networking import deserialize_tfw_msg
from tfw.networking.event_handlers import ServerConnector
class EventHandlerBase(ABC):
"""
Abstract base class for all Python based EventHandlers. Useful implementation template
for other languages.
Derived classes must implement the handle_event() method
"""
def __init__(self, key):
self.server_connector = ServerConnector()
self.key = key
self.subscribe(self.key, 'reset')
self.server_connector.register_callback(self.event_handler_callback)
def event_handler_callback(self, msg_parts):
"""
Callback that is invoked when receiving a message.
Dispatches messages to handler methods and sends
a response back in case the handler returned something.
This is subscribed in __init__().
"""
message = deserialize_tfw_msg(*msg_parts)
response = self.dispatch_handling(message)
if response:
response['key'] = message['key']
self.server_connector.send(response)
def dispatch_handling(self, message):
"""
Used to dispatch messages to their specific handlers.
"""
if message['key'] != 'reset':
return self.handle_event(message)
return self.handle_reset(message)
@abstractmethod
def handle_event(self, message):
"""
Abstract method that implements the handling of messages.
:param message: the message received
"""
raise NotImplementedError
def handle_reset(self, message):
# pylint: disable=unused-argument,no-self-use
"""
Usually 'reset' events receive some sort of special treatment.
:param message: the message received
"""
return None
def subscribe(self, *keys):
"""
Subscribe this EventHandler to receive events for given keys.
Note that you can subscribe to the same key several times in which
case you will need to unsubscribe multiple times in order to stop
receiving events.
:param keys: list of keys to subscribe to
"""
for key in keys:
self.server_connector.subscribe(key)
def unsubscribe(self, *keys):
"""
Unsubscribe this eventhandler from the given keys.
:param keys: list of keys to unsubscribe from
"""
for key in keys:
self.server_connector.unsubscribe(key)
def cleanup(self):
"""
Perform cleanup actions such as releasing database
connections and stuff like that.
"""
pass
class TriggeredEventHandler(EventHandlerBase, ABC):
# pylint: disable=abstract-method
"""
Abstract base class for EventHandlers which are only triggered in case
TFWServer has successfully triggered an FSM step defined in __init__.
"""
def __init__(self, key, trigger):
super().__init__(key)
self.trigger = trigger
def dispatch_handling(self, message):
if message.get('trigger') == self.trigger:
return super().dispatch_handling(message)
return None
-37
View File
@@ -1,37 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from typing import List
from transitions import Machine
from tfw.mixins import CallbackMixin
class FSMBase(CallbackMixin):
"""
A general FSM base class you can inherit from to track user progress.
See linear_fsm.py for an example use-case.
TFW the transitions library for state machines, please refer to their
documentation for more information on creating your own machines:
https://github.com/pytransitions/transitions
"""
states, transitions = [], []
def __init__(self, initial: str = None, accepted_states: List[str] = None):
self.accepted_states = accepted_states or [self.states[-1]]
self.machine = Machine(
model=self,
states=self.states,
transitions=self.transitions,
initial=initial or self.states[0],
send_event=True,
ignore_invalid_triggers=True,
after_state_change='execute_callbacks'
)
def execute_callbacks(self, event_data):
self._execute_callbacks(event_data.kwargs)
def is_solved(self):
return self.state in self.accepted_states # pylint: disable=no-member
-29
View File
@@ -1,29 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .fsm_base import FSMBase
class LinearFSM(FSMBase):
# pylint: disable=anomalous-backslash-in-string
"""
This is a state machine for challenges with linear progression, consisting of
a number of steps specified in the constructor. It automatically sets up 2
actions (triggers) between states as such:
(0) -- step_1 --> (1) -- step_2 --> (2) -- step_3 --> (3) ... and so on
"""
def __init__(self, number_of_steps):
self.states = list(map(str, range(number_of_steps)))
self.transitions = []
for index in self.states[:-1]:
self.transitions.append({
'trigger': f'step_{int(index)+1}',
'source': index,
'dest': str(int(index)+1)
})
self.transitions.append({
'trigger': 'step_next',
'source': index,
'dest': str(int(index)+1)
})
super(LinearFSM, self).__init__()
-7
View File
@@ -1,7 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .supervisor_mixin import SupervisorMixin, SupervisorLogMixin
from .callback_mixin import CallbackMixin
from .observer_mixin import ObserverMixin
from .monitor_manager_mixin import MonitorManagerMixin
-30
View File
@@ -1,30 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from tfw.config.logs import logging
LOG = logging.getLogger(__name__)
class MonitorManagerMixin:
def __init__(self, monitor_type, *monitor_args):
self._monitor_type = monitor_type
self._monitor = None
self.monitor_args = monitor_args
self.reload_monitor()
@property
def monitor(self):
return self._monitor
def set_monitor_args(self, *monitor_args):
self.monitor_args = monitor_args
def reload_monitor(self):
if self._monitor:
try:
self._monitor.stop()
except KeyError:
LOG.debug('Working directory was removed ignoring...')
self._monitor = self._monitor_type(*self.monitor_args)
self._monitor.watch() # This runs on a separate thread
-19
View File
@@ -1,19 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from watchdog.observers import Observer
from tfw.decorators import lazy_property
class ObserverMixin:
@lazy_property
def observer(self):
return Observer()
def watch(self):
self.observer.start()
def stop(self):
self.observer.stop()
self.observer.join()
-9
View File
@@ -1,9 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .serialization import serialize_tfw_msg, deserialize_tfw_msg, validate_message
from .zmq_connector_base import ZMQConnectorBase
# from .controller_connector import ControllerConnector # TODO: readd once controller stuff is resolved
from .message_sender import MessageSender
from .event_handlers.server_connector import ServerUplinkConnector as TFWServerConnector
from .server.tfw_server import TFWServer
@@ -1,18 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import zmq
from zmq.eventloop.zmqstream import ZMQStream
from tfw.config import TFWENV
from tfw.networking import ZMQConnectorBase
class ControllerConnector(ZMQConnectorBase):
def __init__(self, zmq_context=None):
super(ControllerConnector, self).__init__(zmq_context)
self._zmq_rep_socket = self._zmq_context.socket(zmq.REP)
self._zmq_rep_socket.connect(f'tcp://localhost:{TFWENV.CONTROLLER_PORT}')
self._zmq_rep_stream = ZMQStream(self._zmq_rep_socket)
self.register_callback = self._zmq_rep_stream.on_recv_stream
@@ -1,4 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .server_connector import ServerConnector, ServerUplinkConnector, ServerDownlinkConnector
@@ -1,59 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from functools import partial
import zmq
from zmq.eventloop.zmqstream import ZMQStream
from tfw.networking import serialize_tfw_msg
from tfw.networking import ZMQConnectorBase
from tfw.config import TFWENV
class ServerDownlinkConnector(ZMQConnectorBase):
def __init__(self, zmq_context=None):
super(ServerDownlinkConnector, self).__init__(zmq_context)
self._zmq_sub_socket = self._zmq_context.socket(zmq.SUB)
self._zmq_sub_socket.connect(f'tcp://localhost:{TFWENV.PUBLISHER_PORT}')
self._zmq_sub_stream = ZMQStream(self._zmq_sub_socket)
self.subscribe = partial(self._zmq_sub_socket.setsockopt_string, zmq.SUBSCRIBE)
self.unsubscribe = partial(self._zmq_sub_socket.setsockopt_string, zmq.UNSUBSCRIBE)
self.register_callback = self._zmq_sub_stream.on_recv
class ServerUplinkConnector(ZMQConnectorBase):
"""
Class capable of sending messages to the TFW server and event handlers.
"""
def __init__(self, zmq_context=None):
super(ServerUplinkConnector, self).__init__(zmq_context)
self._zmq_push_socket = self._zmq_context.socket(zmq.PUSH)
self._zmq_push_socket.connect(f'tcp://localhost:{TFWENV.RECEIVER_PORT}')
def send_to_eventhandler(self, message):
"""
Send a message to an event handler.
This envelopes the desired message in the 'data' field of the message to
TFWServer, which will mirror it to event handlers.
:param message: JSON message you want to send
:param message['key']: key of event handler you want to address
"""
nested_message = {'key': message['key'], 'data': message.pop('data')}
message['key'] = 'mirror'
message['data'] = nested_message
self.send(message)
def send(self, message):
"""
Send a message to the TFW server
:param message: JSON message you want to send
"""
self._zmq_push_socket.send_multipart(serialize_tfw_msg(message))
class ServerConnector(ServerUplinkConnector, ServerDownlinkConnector):
pass
-46
View File
@@ -1,46 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from tfw.networking.event_handlers import ServerUplinkConnector
class MessageSender:
"""
Provides mechanisms to send messages to our frontend messaging component.
"""
def __init__(self):
self.server_connector = ServerUplinkConnector()
self.key = 'message'
self.queue_key = 'queueMessages'
def send(self, originator, message):
"""
Sends a message.
:param originator: name of sender to be displayed on the frontend
:param message: message to send
"""
data = {
'originator': originator,
'message': message
}
self.server_connector.send({
'key': self.key,
'data': data
})
def queue_messages(self, originator, messages):
"""
Queues a list of messages to be displayed in a chatbot-like manner.
:param originator: name of sender to be displayed on the frontend
:param messages: list of messages to queue
"""
data = {
'messages': [
{'message': message, 'originator': originator}
for message in messages
]
}
self.server_connector.send({
'key': self.queue_key,
'data': data
})
-7
View File
@@ -1,7 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from .event_handler_connector import EventHandlerConnector, EventHandlerUplinkConnector, EventHandlerDownlinkConnector
from .tfw_server import TFWServer
from .zmq_websocket_handler import ZMQWebSocketProxy
# from .controller_responder import ControllerResponder # TODO: readd once controller stuff is resolved
@@ -1,38 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from tfw.networking import deserialize_all, serialize_all, ControllerConnector
class ControllerResponder:
def __init__(self, fsm):
self.fsm = fsm
self.token = None
self.controller_connector = ControllerConnector()
self.controller_connector.register_callback(self.handle_controller_request)
self.controller_request_handlers = {
'solution_check': self.handle_solution_check_request,
'test': self.handle_test_request,
'token': self.handle_token_request
}
def handle_controller_request(self, stream, msg_parts):
key, data = deserialize_all(*msg_parts)
response = self.controller_request_handlers[key](data)
stream.send_multipart(serialize_all(self.token, response))
def handle_test_request(self, data):
# pylint: disable=unused-argument,no-self-use
return 'OK'
def handle_token_request(self, data):
if self.token is None:
self.token = data
return {'token': self.token}
def handle_solution_check_request(self, data):
# pylint: disable=unused-argument
return {
'solved': self.fsm.is_solved(),
'message': 'solved' if self.fsm.is_solved() else 'not solved'
}
@@ -1,38 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import zmq
from zmq.eventloop.zmqstream import ZMQStream
from tfw.networking import ZMQConnectorBase, serialize_tfw_msg
from tfw.config import TFWENV
from tfw.config.logs import logging
LOG = logging.getLogger(__name__)
class EventHandlerDownlinkConnector(ZMQConnectorBase):
def __init__(self, zmq_context=None):
super(EventHandlerDownlinkConnector, self).__init__(zmq_context)
self._zmq_pull_socket = self._zmq_context.socket(zmq.PULL)
self._zmq_pull_stream = ZMQStream(self._zmq_pull_socket)
address = f'tcp://*:{TFWENV.RECEIVER_PORT}'
self._zmq_pull_socket.bind(address)
LOG.debug('Pull socket bound to %s', address)
class EventHandlerUplinkConnector(ZMQConnectorBase):
def __init__(self, zmq_context=None):
super(EventHandlerUplinkConnector, self).__init__(zmq_context)
self._zmq_pub_socket = self._zmq_context.socket(zmq.PUB)
address = f'tcp://*:{TFWENV.PUBLISHER_PORT}'
self._zmq_pub_socket.bind(address)
LOG.debug('Pub socket bound to %s', address)
class EventHandlerConnector(EventHandlerDownlinkConnector, EventHandlerUplinkConnector):
def register_callback(self, callback):
self._zmq_pull_stream.on_recv(callback)
def send_message(self, message: dict):
self._zmq_pub_socket.send_multipart(serialize_tfw_msg(message))
-134
View File
@@ -1,134 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from collections import defaultdict
from tornado.web import Application
from tfw.networking import MessageSender
from tfw.networking.event_handlers import ServerUplinkConnector
from tfw.networking.server import EventHandlerConnector
from tfw.config.logs import logging
from .zmq_websocket_handler import ZMQWebSocketProxy
LOG = logging.getLogger(__name__)
class TFWServer:
"""
This class handles the proxying of messages between the frontend and event handers.
It proxies messages from the "/ws" route to all event handlers subscribed to a ZMQ
SUB socket. It also manages an FSM you can define as a constructor argument.
"""
def __init__(self, fsm_type):
"""
:param fsm_type: the type of FSM you want TFW to use
"""
self._fsm = fsm_type()
self._fsm_updater = FSMUpdater(self._fsm)
self._fsm_manager = FSMManager(self._fsm)
self._fsm.subscribe_callback(self._fsm_updater.update)
self._event_handler_connector = EventHandlerConnector()
self.application = Application([(
r'/ws', ZMQWebSocketProxy,{
'make_eventhandler_message': self.make_eventhandler_message,
'proxy_filter': self.proxy_filter,
'handle_trigger': self.handle_trigger,
'event_handler_connector': self._event_handler_connector
})]
)
# self.controller_responder = ControllerResponder(self.fsm)
# TODO: add this once controller stuff is resolved
@property
def fsm(self):
return self._fsm
@property
def fsm_manager(self):
return self._fsm_manager
def make_eventhandler_message(self, message):
self.trigger_fsm(message)
message['FSMUpdate'] = self._fsm_updater.get_fsm_state_and_transitions()
return message
def handle_trigger(self, message):
LOG.debug('Executing handler for trigger "%s"', message.get('trigger', ''))
self.trigger_fsm(message)
def trigger_fsm(self, message):
trigger = message.get('trigger', '')
try:
self._fsm_manager.trigger(trigger, message)
except AttributeError:
LOG.debug('FSM failed to execute nonexistent trigger: "%s"', trigger)
def proxy_filter(self, message):
# pylint: disable=unused-argument,no-self-use
return True
def listen(self, port):
self.application.listen(port)
class FSMManager:
def __init__(self, fsm):
self._fsm = fsm
self.trigger_predicates = defaultdict(list)
self.messenge_sender = MessageSender()
@property
def fsm(self):
return self._fsm
def trigger(self, trigger, message):
predicate_results = []
for predicate in self.trigger_predicates[trigger]:
success, message = predicate(message)
predicate_results.append(success)
self.messenge_sender.send('FSM', message)
if all(predicate_results):
try:
self.fsm.trigger(trigger, message=message)
except AttributeError:
LOG.debug('FSM failed to execute nonexistent trigger: "%s"', trigger)
def subscribe_predicate(self, trigger, *predicates):
self.trigger_predicates[trigger].extend(predicates)
def unsubscribe_predicate(self, trigger, *predicates):
self.trigger_predicates[trigger] = [
predicate
for predicate in self.trigger_predicates[trigger]
not in predicates
]
class FSMUpdater:
def __init__(self, fsm):
self.fsm = fsm
self.uplink = ServerUplinkConnector()
def update(self, kwargs_dict):
# pylint: disable=unused-argument
self.uplink.send(self.generate_fsm_update())
def generate_fsm_update(self):
return {
'key': 'FSMUpdate',
'data': self.get_fsm_state_and_transitions()
}
def get_fsm_state_and_transitions(self):
state = self.fsm.state
valid_transitions = [
{'trigger': trigger}
for trigger in self.fsm.machine.get_triggers(self.fsm.state)
]
return {
'current_state': state,
'valid_transitions': valid_transitions
}
@@ -1,91 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import json
from abc import ABC, abstractmethod
from tornado.websocket import WebSocketHandler
from tfw.networking import deserialize_tfw_msg, validate_message
from tfw.config.logs import logging
LOG = logging.getLogger(__name__)
class ZMQWebSocketHandler(WebSocketHandler, ABC):
instances = set()
def initialize(self, **kwargs): # pylint: disable=arguments-differ
self._event_handler_connector = kwargs['event_handler_connector']
def prepare(self):
ZMQWebSocketHandler.instances.add(self)
def on_close(self):
ZMQWebSocketHandler.instances.remove(self)
def open(self, *args, **kwargs):
LOG.debug('WebSocket connection initiated')
self._event_handler_connector.register_callback(self.zmq_callback)
def zmq_callback(self, msg_parts):
keyhandlers = {'mirror': self.mirror}
message = deserialize_tfw_msg(*msg_parts)
LOG.debug('Received on pull socket: %s', message)
if not validate_message(message):
return
self.handle_trigger(message)
if message['key'] not in keyhandlers:
for instance in ZMQWebSocketHandler.instances:
instance.write_message(message)
else:
try:
keyhandlers[message['key']](message)
except KeyError:
LOG.error('Invalid mirror message format! Ignoring.')
def mirror(self, message):
message = message['data']
self._event_handler_connector.send_message(message)
def on_message(self, message):
LOG.debug('Received on WebSocket: %s', message)
if validate_message(message):
self.send_message(self.make_eventhandler_message(message))
@abstractmethod
def make_eventhandler_message(self, message):
raise NotImplementedError
def send_message(self, message: dict):
self._event_handler_connector.send_message(message)
@abstractmethod
def handle_trigger(self, message):
raise NotImplementedError
# much secure, very cors, wow
def check_origin(self, origin):
return True
class ZMQWebSocketProxy(ZMQWebSocketHandler):
# pylint: disable=abstract-method
def initialize(self, **kwargs): # pylint: disable=arguments-differ
super(ZMQWebSocketProxy, self).initialize(**kwargs)
self._make_eventhandler_message = kwargs['make_eventhandler_message']
self._proxy_filter = kwargs['proxy_filter']
self._handle_trigger = kwargs['handle_trigger']
def on_message(self, message):
message = json.loads(message)
if self._proxy_filter(message):
super().on_message(message)
def make_eventhandler_message(self, message):
return self._make_eventhandler_message(message)
def handle_trigger(self, message):
self._handle_trigger(message)
-9
View File
@@ -1,9 +0,0 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
import zmq
class ZMQConnectorBase:
def __init__(self, zmq_context=None):
self._zmq_context = zmq_context or zmq.Context.instance()
+4
View File
@@ -0,0 +1,4 @@
[pytest]
filterwarnings =
ignore::DeprecationWarning:zmq
ignore::DeprecationWarning:watchdog
+15 -5
View File
@@ -1,5 +1,15 @@
tornado==5.0
pyzmq==17.0.0
transitions==0.6.4
terminado==0.8.1
watchdog==0.8.3
tornado>=6.0.0,<7.0.0
pyzmq>=17.0.0,<18.0.0
transitions>=0.0.0,<1.0.0
terminado>=0.0.0,<1.0.0
watchdog>=0.0.0,<1.0.0
PyYAML>=5.0.0,<6.0.0
Jinja2>=2.0.0,<3.0.0
cryptography>=2.0.0,<3.0.0
python-dateutil>=2.0.0,<3.0.0
SQLAlchemy>=1.0.0,<2.0.0
python-dateutil>=2.0.0,<3.0.0
pytest>=5.0.0,<6.0.0
pylint>=2.0.0,<3.0.0
rope>=0.0.0,<1.0.0
pipe-io-server>=1.0.0,<2.0.0
+13 -13
View File
@@ -1,6 +1,6 @@
from os.path import dirname, realpath, join
from setuptools import setup, find_packages
from setuptools import setup
here = dirname(realpath(__file__))
@@ -10,20 +10,20 @@ with open(join(here, 'requirements.txt'), 'r') as ifile:
requirements = ifile.read().splitlines()
setup(
name = 'tfw',
version = version,
description = 'Avatao tutorial-framework',
url = 'https://github.com/avatao-content/baseimage-tutorial-framework',
author = 'Avatao.com Innovative Learning Kft.',
author_email = 'support@avatao.com',
license = 'custom',
packages = find_packages('lib'),
package_dir = {'': 'lib'},
install_requires = requirements,
extras_require = {
name='tfw',
version=version,
description='Avatao tutorial-framework',
url='https://github.com/avatao-content/baseimage-tutorial-framework',
author='Avatao.com Innovative Learning Kft.',
author_email='support@avatao.com',
license='custom',
packages=['tfw'],
package_dir={'tfw': 'tfw'},
install_requires=requirements,
extras_require={
'docs': [
'sphinx >= 1.7.0',
],
},
zip_safe = False,
zip_safe=False,
)
+6
View File
@@ -0,0 +1,6 @@
[program:tfw_init]
user=root
directory=/tmp
command=bash tfw_init.sh
autorestart=false
startsecs=0
+4
View File
@@ -0,0 +1,4 @@
[program:tfw_server]
user=root
directory=%(ENV_TFW_SERVER_DIR)s
command=python3 -u tfw_server.py
+15
View File
@@ -0,0 +1,15 @@
#!/bin/bash
set -euo pipefail
echo "export HISTFILE=\"${TFW_HISTFILE}\"" >> /tmp/bashrc &&
cat /tmp/bashrc >> "/home/${AVATAO_USER}/.bashrc"
if [[ -z "${HOTRELOAD-}" ]]; then
for dir in "${TFW_LIB_DIR}/tfw" "/etc/nginx" "/etc/supervisor"; do
chown -R root:root "${dir}" && chmod -R 700 "${dir}";
done
fi
chmod 777 "${TFW_PIPES_DIR}"
rm -f bashrc requirements.txt tfw_init.sh
+11
View File
@@ -0,0 +1,11 @@
from tornado.ioloop import IOLoop
from tfw.main import TFWServer, setup_logger, setup_signal_handlers
if __name__ == '__main__':
setup_logger(__file__)
TFWServer().listen()
setup_signal_handlers()
IOLoop.instance().start()
View File
View File
+6
View File
@@ -0,0 +1,6 @@
from .console_logs_handler import ConsoleLogsHandler
from .frontend_proxy_handler import FrontendProxyHandler
from .frontend_ready_handler import FrontendReadyHandler
from .message_queue_handler import MessageQueueHandler
from .message_sender import MessageSender
from .frontend_config_handler import FrontendConfigHandler
@@ -0,0 +1,19 @@
import logging
LOG = logging.getLogger(__name__)
class ConsoleLogsHandler:
keys = ['process.log.new']
def __init__(self, *, stream):
self.stream = stream
def handle_event(self, message, connector):
try:
connector.send_message({
'key': 'console.write',
'content': message[self.stream]
})
except KeyError:
LOG.error('Invalid %s message received: %s', self.keys, message)
@@ -0,0 +1 @@
from .frontend_config_handler import FrontendConfigHandler
@@ -0,0 +1,36 @@
from yaml import safe_load
from tfw.internals.networking import Scope
class FrontendConfigHandler:
keys = ['frontend.ready']
def __init__(self, config_path):
self._config_path = config_path
def handle_event(self, _, connector):
# pylint: disable=no-self-use
for message in self._config_messages:
connector.send_message(message)
connector.send_message({'key': 'frontend.ready'}, scope=Scope.WEBSOCKET)
@property
def _config_messages(self):
config = safe_load(self._config_str)
return [
self._create_config_message(k, v)
for k, v in config.items()
]
@property
def _config_str(self):
with open(self._config_path, 'r') as ifile:
return ifile.read()
@staticmethod
def _create_config_message(key, value):
return {
'key': f'frontend.{key}',
**value
}
@@ -0,0 +1,40 @@
from textwrap import dedent
from .frontend_config_handler import FrontendConfigHandler
class MockFrontendConfigHandler(FrontendConfigHandler):
@property
def _config_str(self):
return dedent('''
cat:
someConfigKey: lel
otherStuff: 42
foo:
hey: yaaay
''')
class MockConnector:
def __init__(self):
self.messages = []
def send_message(self, message, **_):
self.messages.append(message)
def test_frontend_config_handler():
connector = MockConnector()
handler = MockFrontendConfigHandler('')
handler.handle_event({'key': 'frontend.ready'}, connector)
assert connector.messages[0] == {
'key': 'frontend.cat',
'someConfigKey': 'lel',
'otherStuff': 42
}
assert connector.messages[1] == {
'key': 'frontend.foo',
'hey': 'yaaay'
}
assert connector.messages[-1] == {'key': 'frontend.ready'}
@@ -0,0 +1,34 @@
from tfw.internals.networking import Scope
from .message_storage import FrontendMessageStorage
class FrontendProxyHandler:
keys = ['console', 'dashboard', 'frontend', 'message', 'ide.read', 'deploy.finish']
type_id = 'ControlEventHandler'
def __init__(self):
self.connector = None
self._frontend_message_storage = FrontendMessageStorage()
def send_message(self, message):
self.connector.send_message(message, scope=Scope.WEBSOCKET)
def handle_event(self, message, _):
self._frontend_message_storage.save_message(message)
if message['key'] == 'frontend.ready':
self.recover_frontend()
return
if self._filter_message(message):
self.send_message(message)
@staticmethod
def _filter_message(message):
return not message['key'].startswith((
'ide',
'message.queue'
))
def recover_frontend(self):
for message in self._frontend_message_storage.messages:
self.send_message(message)
@@ -0,0 +1,39 @@
import logging
from tfw.internals.crypto import KeyManager, sign_message
LOG = logging.getLogger(__name__)
class FrontendReadyHandler:
keys = ['frontend.ready', 'fsm.update']
def __init__(self, initial_trigger):
self.connector = None
self._auth_key = KeyManager().auth_key
self.initial_trigger = initial_trigger
self.commands = {
'frontend.ready': self.handle_ready,
'fsm.update': self.handle_update
}
def handle_event(self, message, _):
try:
self.commands[message['key']]()
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_ready(self):
trigger = {
'key': 'fsm.trigger',
'transition': self.initial_trigger
}
sign_message(self._auth_key, trigger)
self.connector.send_message(trigger)
def handle_update(self):
self.stop()
def stop(self):
pass
@@ -0,0 +1 @@
from .message_queue_handler import MessageQueueHandler
@@ -0,0 +1,43 @@
from time import sleep
from queue import Queue
from threading import Thread
class MessageQueueHandler:
keys = ['message.queue']
type_id = 'ControlEventHandler'
def __init__(self, wpm):
self.connector = None
self.wpm = wpm
self._queue = Queue()
self._thread = Thread(target=self._dispatch_messages)
def _dispatch_messages(self):
for message in iter(self._queue.get, None):
wpm = message['wpm'] if 'wpm' in message else self.wpm
cps = 5 * wpm / 60
sleep(len(message['message']) / cps)
self.connector.send_message(message)
def handle_event(self, message, _):
for unpacked in self._generate_messages_from_queue(message):
self._queue.put(unpacked)
@staticmethod
def _generate_messages_from_queue(queue_message):
last = queue_message['messages'][-1]
for message in queue_message['messages']:
yield {
'key': 'message.send',
'typing': message is not last,
**message
}
def start(self):
self._thread.start()
def cleanup(self):
self._queue.queue.clear()
self._queue.put(None)
self._thread.join()
@@ -0,0 +1,67 @@
# pylint: disable=redefined-outer-name
from math import inf
from time import sleep
from os import urandom
from random import randint
import pytest
from .message_queue_handler import MessageQueueHandler
class MockConnector:
def __init__(self):
self.callback = None
self.messages = []
def raise_event(self, message):
self.callback(message, self)
sleep(0.01)
def send_message(self, message):
self.messages.append(message)
@pytest.fixture
def handler():
connector = MockConnector()
handler = MessageQueueHandler(inf)
handler.connector = connector
connector.callback = handler.handle_event
handler.start()
yield handler
handler.cleanup()
@pytest.fixture
def queue():
yield {
'key': 'message.queue',
'messages': [
{'originator': urandom(4).hex(), 'message': urandom(16).hex()}
for _ in range(randint(5, 10))
]
}
def test_message_order(handler, queue):
handler.connector.raise_event(queue)
old_list = queue['messages']
new_list = handler.connector.messages
length = len(old_list)
assert len(new_list) == length
for i in range(length):
unpacked = new_list[i]
assert unpacked['key'] == 'message.send'
assert unpacked['originator'] == old_list[i]['originator']
assert unpacked['typing'] == (i < length-1)
def test_wpm(handler, queue):
handler.wpm = 10000
handler.connector.raise_event(queue)
assert not handler.connector.messages
handler.wpm = 100000000
handler.connector.raise_event(queue)
sleep(0.25)
assert len(handler.connector.messages) == 2*len(queue['messages'])
+30
View File
@@ -0,0 +1,30 @@
class MessageSender:
def __init__(self, uplink):
self.uplink = uplink
def send(self, message, originator=None):
message = {
'key': 'message.send',
'message': message
}
if originator:
message['originator'] = originator
self.uplink.send_message(message)
def queue_messages(self, messages, originator=None):
message_queue = {
'key': 'message.queue',
'messages': []
}
for message in messages:
next_message = {'message': message}
if originator:
next_message['originator'] = originator
message_queue['messages'].append(next_message)
self.uplink.send_message(message_queue)
def set_originator(self, originator):
self.uplink.send_message({
'key': 'message.config',
'originator': originator
})
@@ -0,0 +1,49 @@
from abc import ABC, abstractmethod
from contextlib import suppress
class MessageStorage(ABC):
def __init__(self):
self._messages = []
def save_message(self, message):
with suppress(KeyError, AttributeError):
if self._filter_message(message):
self._messages.extend(self._transform_message(message))
@abstractmethod
def _filter_message(self, message):
raise NotImplementedError
def _transform_message(self, message): # pylint: disable=no-self-use
yield message
def clear(self):
self._messages.clear()
@property
def messages(self):
yield from self._messages
class FrontendMessageStorage(MessageStorage):
def _filter_message(self, message):
return message['key'].startswith((
'console.write',
'message.send',
'ide.read'
))
def _transform_message(self, message):
transformations = {
'ide.read': self._delete_ide_content
}
if message['key'] in transformations:
yield from transformations[message['key']](message)
else:
yield message
@staticmethod
def _delete_ide_content(message):
del message['content']
yield message
+1
View File
@@ -0,0 +1 @@
from .fsm_handler import FSMHandler
+42
View File
@@ -0,0 +1,42 @@
import logging
from tfw.internals.crypto import KeyManager, sign_message
from tfw.internals.networking import Scope, Intent
from .fsm_updater import FSMUpdater
LOG = logging.getLogger(__name__)
class FSMHandler:
keys = ['fsm']
type_id = 'ControlEventHandler'
def __init__(self, *, fsm_type):
self.connector = None
self.fsm = fsm_type()
self._fsm_updater = FSMUpdater(self.fsm)
self.auth_key = KeyManager().auth_key
self.command_handlers = {
'fsm.trigger': self.handle_trigger,
'fsm.update' : self.handle_update
}
def handle_event(self, message, _):
try:
message = self.command_handlers[message['key']](message)
if message:
fsm_update_message = self._fsm_updater.fsm_update
sign_message(self.auth_key, fsm_update_message)
self.connector.send_message(fsm_update_message, Scope.BROADCAST, Intent.EVENT)
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_trigger(self, message): # pylint: disable=inconsistent-return-statements
if self.fsm.step(message['transition']):
return message
def handle_update(self, message):
# pylint: disable=no-self-use
return message
+26
View File
@@ -0,0 +1,26 @@
class FSMUpdater:
def __init__(self, fsm):
self.fsm = fsm
@property
def fsm_update(self):
return {
'key': 'fsm.update',
**self.fsm_update_data
}
@property
def fsm_update_data(self):
valid_transitions = [
{'trigger': trigger}
for trigger in self.fsm.get_triggers(self.fsm.state)
]
if not self.fsm.event_log:
return None
last_fsm_event = self.fsm.event_log[-1]
return {
'current_state': self.fsm.state,
'valid_transitions': valid_transitions,
'in_accepted_state': self.fsm.in_accepted_state,
'last_event': last_fsm_event
}
+2
View File
@@ -0,0 +1,2 @@
from .ide_handler import IdeHandler
from .deploy_handler import DeployHandler
+34
View File
@@ -0,0 +1,34 @@
import logging
LOG = logging.getLogger(__name__)
class DeployHandler:
keys = ['deploy.start', 'process.restart']
def __init__(self, process_name='webservice'):
self.connector = None
self.process_name = process_name
self.commands = {
'deploy.start': self.handle_deploy,
'process.restart': self.handle_process
}
def handle_event(self, message, _):
try:
self.commands[message['key']](message)
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_deploy(self, _):
self.connector.send_message({
'key': 'process.restart',
'name': self.process_name
})
def handle_process(self, message):
response = {'key': 'deploy.finish'}
if 'error' in message:
response['error'] = message['error']
self.connector.send_message(response)
@@ -0,0 +1 @@
from .file_manager import FileManager
@@ -0,0 +1,66 @@
import logging
from functools import wraps
from glob import glob
from fnmatch import fnmatchcase
from os.path import dirname, isdir, isfile, realpath, isabs
LOG = logging.getLogger(__name__)
def _with_is_allowed(func):
@wraps(func)
def wrapper(self, *args, **kwargs):
if self.is_allowed(args[0]):
return func(self, *args, **kwargs)
raise ValueError('Forbidden path.')
return wrapper
class FileManager: # pylint: disable=too-many-instance-attributes
def __init__(self, patterns):
self.patterns = patterns
@property
def files(self):
return list(set(
path
for pattern in self.patterns
for path in glob(pattern, recursive=True)
if isfile(path) and self.is_allowed(path)
))
@property
def parents(self):
return list(set(
self._find_directory(pattern)
for pattern in self.patterns
))
@staticmethod
def _find_directory(pattern):
while pattern and not isdir(pattern):
pattern = dirname(pattern)
return pattern
def is_allowed(self, filepath):
return any(
fnmatchcase(realpath(filepath), pattern)
for pattern in self.patterns
)
def find_file(self, filename):
if not isabs(filename):
for filepath in self.files:
if filepath.endswith(filename):
return filepath
return filename
@_with_is_allowed
def read_file(self, filepath): # pylint: disable=no-self-use
with open(filepath, 'rb', buffering=0) as ifile:
return ifile.read().decode(errors='surrogateescape')
@_with_is_allowed
def write_file(self, filepath, contents): # pylint: disable=no-self-use
with open(filepath, 'wb', buffering=0) as ofile:
ofile.write(contents.encode())
@@ -0,0 +1,94 @@
# pylint: disable=redefined-outer-name
from dataclasses import dataclass
from secrets import token_urlsafe
from os import mkdir, symlink
from os.path import join, realpath, basename
from pathlib import Path
from tempfile import TemporaryDirectory
import pytest
from .file_manager import FileManager
@dataclass
class ManagerContext:
workdir: str
subdir: str
subfile: str
manager: FileManager
def create_random_file(self, dirname, extension):
filename = self.join(f'{dirname}/{generate_name()}{extension}')
Path(filename).touch()
return filename
def create_random_folder(self, basepath):
dirname = self.join(f'{basepath}/{generate_name()}')
mkdir(dirname)
return dirname
def create_random_link(self, source, dirname, extension):
linkname = self.join(f'{dirname}/{generate_name()}{extension}')
symlink(source, linkname)
return linkname
def join(self, path):
return join(self.workdir, path)
def generate_name():
return token_urlsafe(16)
@pytest.fixture()
def context():
with TemporaryDirectory() as workdir:
workdir = realpath(workdir) # macOS uses a symlinked TMPDIR
subdir = join(workdir, generate_name())
subfile = join(subdir, generate_name() + '.txt')
mkdir(subdir)
Path(subfile).touch()
manager = FileManager([join(workdir, '**/*.txt')])
yield ManagerContext(workdir, subdir, subfile, manager)
def test_matching_files(context):
newdir = context.create_random_folder(context.subdir)
newfile = context.create_random_file(newdir, '.txt')
newlink = context.create_random_link(newfile, newdir, '.txt')
assert set(context.manager.files) == {context.subfile, newfile, newlink}
def test_unmatching_files(context):
newtxt = context.create_random_file(context.workdir, '.txt')
newbin = context.create_random_file(context.subdir, '.bin')
context.create_random_link(newtxt, context.subdir, '.txt')
context.create_random_link(newbin, context.subdir, '.txt')
assert context.manager.files == [context.subfile]
def test_parents(context):
newdir = context.create_random_folder(context.workdir)
context.manager.patterns += [f'{newdir}/[!/@]*/**/?.c']
assert set(context.manager.parents) == {context.workdir, newdir}
def test_read_write_file(context):
for _ in range(128):
content = token_urlsafe(32)
context.manager.write_file(context.subfile, content)
assert context.manager.read_file(context.subfile) == content
with open(context.subfile, 'r') as ifile:
assert ifile.read() == content
def test_regular_ide_actions(context):
newfile1 = context.create_random_file(context.subdir, '.txt')
newfile2 = context.create_random_file(context.subdir, '.txt')
for _ in range(4):
context.manager.filename = newfile1
content1, content2 = token_urlsafe(32), token_urlsafe(32)
context.manager.write_file(newfile1, content1)
context.manager.write_file(newfile2, content2)
assert context.manager.read_file(newfile1) == content1
assert context.manager.read_file(newfile2) == content2
def test_find_file(context):
for _ in range(5):
file_abs = context.create_random_file(context.subdir, '.txt')
file_base = basename(file_abs)
assert context.manager.find_file(file_base) == file_abs
+101
View File
@@ -0,0 +1,101 @@
import logging
from tfw.internals.networking import Scope
from tfw.internals.inotify import InotifyObserver
from .file_manager import FileManager
LOG = logging.getLogger(__name__)
BUILD_ARTIFACTS = (
"*.a",
"*.class",
"*.dll",
"*.dylib",
"*.elf",
"*.exe",
"*.jar",
"*.ko",
"*.la",
"*.lib",
"*.lo",
"*.o",
"*.obj",
"*.out",
"*.py[cod]",
"*.so",
"*.so.*",
"*.tar.gz",
"*.zip",
"*__pycache__*"
)
class IdeHandler:
keys = ['ide']
type_id = 'ControlEventHandler'
def __init__(self, *, patterns, initial_file=None):
self.connector = None
self.filemanager = FileManager(patterns)
self._initial_file = initial_file or ''
self.monitor = InotifyObserver(
path=self.filemanager.parents,
exclude=BUILD_ARTIFACTS
)
self.monitor.on_modified = self._reload_frontend
self.monitor.start()
self.commands = {
'ide.read' : self.read,
'ide.write' : self.write
}
def _reload_frontend(self, event): # pylint: disable=unused-argument
self.send_message({'key': 'ide.reload'})
@property
def initial_file(self):
if not self.filemanager.is_allowed(self._initial_file):
self._initial_file = self.filemanager.files[0]
return self._initial_file
def send_message(self, message):
self.connector.send_message(message, scope=Scope.WEBSOCKET)
def handle_event(self, message, _):
try:
self.commands[message['key']](message)
message['files'] = self.filemanager.files
self.send_message(message)
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def read(self, message):
if 'patterns' in message:
self.filemanager.patterns = message['patterns']
try:
message['filename'] = self.filemanager.find_file(
message.get('filename') or self.initial_file
)
message['content'] = self.filemanager.read_file(message['filename'])
except (PermissionError, ValueError):
message['content'] = 'You have no permission to open that file :('
except FileNotFoundError:
message['content'] = 'This file does not exist :('
except Exception: # pylint: disable=broad-except
message['content'] = 'Failed to read file :('
LOG.exception('Error reading file!')
def write(self, message):
try:
self.filemanager.write_file(message['filename'], message['content'])
except KeyError:
LOG.error('You must provide a filename to write!')
except Exception: # pylint: disable=broad-except
LOG.exception('Error writing file!')
del message['content']
def cleanup(self):
self.monitor.stop()
+3
View File
@@ -0,0 +1,3 @@
from .pipe_io_handler import PipeIOHandler, PipeIOHandlerBase
from .command_handler import CommandHandler
from .pipe_connector import ProxyPipeConnectorHandler
+58
View File
@@ -0,0 +1,58 @@
from subprocess import PIPE, Popen
from functools import partial
from os import getpgid, killpg
from os.path import join
from signal import SIGTERM
from secrets import token_urlsafe
from threading import Thread
from contextlib import suppress
from pipe_io_server import terminate_process_on_failure
from .pipe_io_handler import PipeIOHandler, DEFAULT_PERMISSIONS
class CommandHandler(PipeIOHandler):
def __init__(self, command, permissions=DEFAULT_PERMISSIONS):
super().__init__(
self._generate_tempfilename(),
self._generate_tempfilename(),
permissions
)
self._proc_stdin = open(self.pipe_io.out_pipe, 'rb')
self._proc_stdout = open(self.pipe_io.in_pipe, 'wb')
self._proc = Popen(
command, shell=True, executable='/bin/bash',
stdin=self._proc_stdin, stdout=self._proc_stdout, stderr=PIPE,
start_new_session=True
)
self._monitor_proc_thread = self._start_monitor_proc()
def _generate_tempfilename(self):
# pylint: disable=no-self-use
random_filename = partial(token_urlsafe, 10)
return join('/tmp', f'{type(self).__name__}.{random_filename()}')
def _start_monitor_proc(self):
thread = Thread(target=self._monitor_proc, daemon=True)
thread.start()
return thread
@terminate_process_on_failure
def _monitor_proc(self):
return_code = self._proc.wait()
if return_code == -int(SIGTERM):
# supervisord asked the program to terminate, this is fine
return
if return_code != 0:
_, stderr = self._proc.communicate()
raise RuntimeError(f'Subprocess failed ({return_code})! Stderr:\n{stderr.decode()}')
def cleanup(self):
with suppress(ProcessLookupError):
process_group_id = getpgid(self._proc.pid)
killpg(process_group_id, SIGTERM)
self._proc_stdin.close()
self._proc_stdout.close()
super().cleanup()
@@ -0,0 +1 @@
from .proxy_pipe_connector_handler import ProxyPipeConnectorHandler
@@ -0,0 +1,90 @@
import logging
from stat import S_ISFIFO
from os import access, listdir, mkdir, stat, R_OK
from os.path import exists, join
from pipe_io_server import PipeWriterServer
from tfw.internals.inotify import InotifyObserver, InotifyFileCreatedEvent, InotifyFileDeletedEvent
LOG = logging.getLogger(__name__)
class PipeConnector:
reader_pattern, writer_pattern = 'send', 'recv'
def __init__(self, path):
self.recv_pipes, self.send_pipes = {}, {}
self.observer = self._build_observer(path)
self.observer.on_any_event = self._on_any_event
self.observer.start()
self._connect_existing_pipes(path)
def _build_observer(self, path): # pylint: disable=no-self-use
if not exists(path):
mkdir(path)
if not access(path, R_OK):
raise ValueError('Path does not exist or is not accessible.')
return InotifyObserver(path, patterns=[f'*{self.reader_pattern}*', f'*{self.writer_pattern}*'])
def _connect_existing_pipes(self, path):
for node in listdir(path):
node_path = join(path, node)
if self._is_pipe(node_path):
self._create_pipe(node_path)
LOG.debug('Connected to existing pipe "%s"', node_path)
def _on_any_event(self, event):
path = event.src_path
if self._is_pipe(path) and isinstance(event, InotifyFileCreatedEvent):
self._create_pipe(path)
LOG.debug('Connected to new pipe "%s"', path)
elif isinstance(event, InotifyFileDeletedEvent):
self._delete_pipe(path)
LOG.debug('Disconnected from deleted pipe "%s"', path)
@staticmethod
def _is_pipe(path):
return exists(path) and S_ISFIFO(stat(path).st_mode)
def _create_pipe(self, path):
if self._find_pipe(path):
return
server = None
if self.writer_pattern in path:
pipes, server = self.recv_pipes, self.build_writer(path)
elif self.reader_pattern in path:
pipes, server = self.send_pipes, self.build_reader(path)
if server:
server.start()
pipes[path] = server
def _find_pipe(self, path):
pipes = None
if path in self.recv_pipes.keys():
pipes = self.recv_pipes
if path in self.send_pipes.keys():
pipes = self.send_pipes
return pipes
def build_reader(self, path): # pylint: disable=no-self-use
raise NotImplementedError()
def build_writer(self, path): # pylint: disable=no-self-use
return PipeWriterServer(path, manage_pipes=False)
def _delete_pipe(self, path):
pipes = self._find_pipe(path)
if pipes:
pipes[path].stop()
del pipes[path]
def broadcast(self, message):
for server in self.recv_pipes.values():
server.send_message(message)
def stop(self):
for pipe in self.recv_pipes.values():
pipe.stop()
for pipe in self.send_pipes.values():
pipe.stop()
@@ -0,0 +1,46 @@
import logging
from threading import Lock
from json import dumps, loads, JSONDecodeError
from pipe_io_server import PipeReaderServer
from .pipe_connector import PipeConnector
LOG = logging.getLogger(__name__)
class ProxyPipeConnectorHandler:
keys = ['']
def __init__(self, path):
self.connector, self.pipes = None, None
self.path = path
def start(self):
self.pipes = ProxyPipeConnector(self.path, self.connector)
def handle_event(self, message, _):
self.pipes.broadcast(dumps(message).encode())
def cleanup(self):
self.pipes.stop()
class ProxyPipeConnector(PipeConnector):
def __init__(self, path, connector):
self.connector = connector
self.mutex = Lock()
super().__init__(path)
def build_reader(self, path):
reader = PipeReaderServer(path, manage_pipes=False)
reader.handle_message = self._handle_message
return reader
def _handle_message(self, message):
try:
json_object = loads(message)
with self.mutex:
self.connector.send_message(json_object)
except JSONDecodeError:
LOG.error('Received invalid JSON message: %s', message)
@@ -0,0 +1,157 @@
# pylint: disable=redefined-outer-name
from enum import Enum
from dataclasses import dataclass
from json import dumps
from secrets import token_urlsafe
from os import urandom, mkfifo, mkdir, remove
from os.path import join
from pathlib import Path
from tempfile import TemporaryDirectory
import pytest
from tfw.internals.inotify import InotifyFileCreatedEvent, InotifyFileDeletedEvent
from .proxy_pipe_connector_handler import ProxyPipeConnector
class Action(Enum):
SEND = 'send'
RECV = 'recv'
@dataclass
class PipeContext:
workdir: str
pipes: ProxyPipeConnector
def emit_pipe_creation_event(self, action, inode_creator):
filename = self.join(f'{self.generate_name()}_{action.value}')
inode_creator(filename)
self.pipes.observer.on_any_event(InotifyFileCreatedEvent(filename))
return filename
def join(self, path):
return join(self.workdir, path)
@staticmethod
def generate_name():
return urandom(4).hex()
class MockPipeConnector(ProxyPipeConnector):
def __init__(self, path, connector):
self.reader_events, self.writer_events = [], []
super().__init__(path, connector)
def _build_observer(self, path):
return MockObserver()
def build_reader(self, path):
self.reader_events.append(path)
reader = MockPipeServer()
reader.handle_message = self._handle_message
return reader
def build_writer(self, path):
self.writer_events.append(path)
return MockPipeServer()
class MockObserver:
def start(self):
pass
def on_any_event(self, event):
pass
class MockPipeServer:
def __init__(self):
self.messages = []
def handle_message(self, message):
pass
def send_message(self, message):
self.messages.append(message)
def start(self):
pass
def stop(self):
pass
class MockConnector: # pylint: disable=too-few-public-methods
def __init__(self):
self.messages = []
def send_message(self, message):
self.messages.append(message)
@pytest.fixture
def workdir():
with TemporaryDirectory() as workdir:
yield workdir
@pytest.fixture
def context(workdir):
mkfifo(join(workdir, Action.SEND.value))
mkfifo(join(workdir, Action.RECV.value))
yield PipeContext(workdir, MockPipeConnector(workdir, MockConnector()))
def test_existing_pipe_connection(context):
assert join(context.workdir, Action.SEND.value) in context.pipes.send_pipes.keys()
assert join(context.workdir, Action.RECV.value) in context.pipes.recv_pipes.keys()
def test_pipe_creation_deletion(context):
cases = [
(Action.RECV, context.pipes.recv_pipes, context.pipes.writer_events),
(Action.SEND, context.pipes.send_pipes, context.pipes.reader_events)
]
for action, pipes, events in cases:
path = context.emit_pipe_creation_event(action, mkfifo)
assert events[-1] == path
assert path in pipes.keys()
remove(path)
context.pipes.observer.on_any_event(InotifyFileDeletedEvent(path))
assert path not in pipes.keys()
def test_handle_message(context):
path = context.emit_pipe_creation_event(Action.SEND, mkfifo)
payload = {'key': token_urlsafe(16)}
context.pipes.send_pipes[path].handle_message(dumps(payload))
assert context.pipes.connector.messages[-1] == payload
context.pipes.send_pipes[path].handle_message(token_urlsafe(32))
assert len(context.pipes.connector.messages) == 1
def test_broadcast(context):
paths = [
context.emit_pipe_creation_event(Action.RECV, mkfifo)
for _ in range(32)
]
payload = {'key': token_urlsafe(16)}
context.pipes.broadcast(payload)
for path in paths:
assert context.pipes.recv_pipes[path].messages[-1] == payload
def test_inode_types(context):
touch = lambda path: Path(path).touch()
cases = [
(Action.RECV, context.pipes.recv_pipes, mkdir),
(Action.SEND, context.pipes.send_pipes, mkdir),
(Action.RECV, context.pipes.recv_pipes, touch),
(Action.SEND, context.pipes.send_pipes, touch)
]
for action, pipes, creator in cases:
path = context.emit_pipe_creation_event(action, creator)
assert path not in pipes.keys()
+47
View File
@@ -0,0 +1,47 @@
import logging
from abc import abstractmethod
from json import loads, dumps
from pipe_io_server import PipeIOServer
LOG = logging.getLogger(__name__)
DEFAULT_PERMISSIONS = 0o600
class PipeIOHandlerBase:
keys = ['']
def __init__(self, in_pipe_path, out_pipe_path, permissions=DEFAULT_PERMISSIONS):
self.connector = None
self.in_pipe = in_pipe_path
self.out_pipe = out_pipe_path
self.pipe_io = PipeIOServer(
in_pipe_path,
out_pipe_path,
permissions
)
self.pipe_io.handle_message = self._server_handle_message
self.pipe_io.start()
def _server_handle_message(self, message):
try:
self.handle_pipe_event(message)
except: # pylint: disable=bare-except
LOG.exception('Failed to handle message %s from pipe %s!', message, self.in_pipe)
@abstractmethod
def handle_pipe_event(self, message_bytes):
raise NotImplementedError()
def cleanup(self):
self.pipe_io.stop()
class PipeIOHandler(PipeIOHandlerBase):
def handle_event(self, message, _):
json_bytes = dumps(message).encode()
self.pipe_io.send_message(json_bytes)
def handle_pipe_event(self, message_bytes):
json = loads(message_bytes)
self.connector.send_message(json)
@@ -0,0 +1,45 @@
# pylint: disable=redefined-outer-name
from tempfile import TemporaryDirectory
from os.path import join
from queue import Queue
from json import dumps, loads
import pytest
from .pipe_io_handler import PipeIOHandler
@pytest.fixture
def pipe_io():
with TemporaryDirectory() as tmpdir:
pipeio = PipeIOHandler(
join(tmpdir, 'in'),
join(tmpdir, 'out'),
permissions=0o600
)
pipeio.connector = MockConnector()
yield pipeio
pipeio.cleanup()
class MockConnector:
def __init__(self):
self.messages = Queue()
def send_message(self, msg):
self.messages.put(msg)
def test_pipe_io_handler_recv(pipe_io):
test_msg = {'key': 'cica'}
with open(pipe_io.in_pipe, 'w') as ofile:
ofile.write(dumps(test_msg) + '\n')
assert pipe_io.connector.messages.get() == test_msg
def test_pipe_io_handler_send(pipe_io):
test_msg = {'key': 'cica'}
pipe_io.handle_event(test_msg, None)
with open(pipe_io.out_pipe, 'r') as ifile:
assert loads(ifile.readline()) == test_msg
@@ -0,0 +1,2 @@
from .process_handler import ProcessHandler
from .process_log_handler import ProcessLogHandler
@@ -0,0 +1,34 @@
from tfw.internals.networking import Scope, Intent
from tfw.internals.inotify import InotifyObserver
from .supervisor import ProcessLogManager
class LogInotifyObserver(InotifyObserver, ProcessLogManager):
def __init__(self, connector, process_name, supervisor_uri, log_tail=0):
self._connector = connector
self._process_name = process_name
self.log_tail = log_tail
self._procinfo = None
ProcessLogManager.__init__(self, supervisor_uri)
InotifyObserver.__init__(self, self._get_logfiles())
def _get_logfiles(self):
self._procinfo = self.supervisor.getProcessInfo(self._process_name)
return self._procinfo['stdout_logfile'], self._procinfo['stderr_logfile']
@property
def process_name(self):
return self._process_name
@process_name.setter
def process_name(self, process_name):
self._process_name = process_name
self.paths = self._get_logfiles()
def on_modified(self, event):
self._connector.send_message({
'key': 'process.log.new',
'stdout': self.read_stdout(self.process_name, tail=self.log_tail),
'stderr': self.read_stderr(self.process_name, tail=self.log_tail)
}, Scope.BROADCAST, Intent.EVENT)
@@ -0,0 +1,33 @@
import logging
from xmlrpc.client import Fault as SupervisorFault
from tfw.internals.networking import Scope, Intent
from .supervisor import ProcessManager
LOG = logging.getLogger(__name__)
class ProcessHandler(ProcessManager):
keys = ['process']
type_id = 'ControlEventHandler'
def __init__(self, *, supervisor_uri):
ProcessManager.__init__(self, supervisor_uri)
self.commands = {
'process.start': self.start_process,
'process.stop': self.stop_process,
'process.restart': self.restart_process
}
def handle_event(self, message, connector):
try:
try:
self.commands[message['key']](message['name'])
except SupervisorFault as fault:
message['error'] = fault.faultString
connector.send_message(message, scope=Scope.BROADCAST, intent=Intent.EVENT)
except KeyError:
if not message['key'].startswith('process.log'):
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
@@ -0,0 +1,44 @@
import logging
from .log_inotify_observer import LogInotifyObserver
LOG = logging.getLogger(__name__)
class ProcessLogHandler:
keys = ['process.log']
type_id = 'ControlEventHandler'
def __init__(self, *, process_name, supervisor_uri, log_tail=0):
self.connector, self._monitor = None, None
self.process_name = process_name
self._supervisor_uri = supervisor_uri
self._initial_log_tail = log_tail
self.command_handlers = {
'process.log.set': self.handle_set
}
def start(self):
self._monitor = LogInotifyObserver(
connector=self.connector,
process_name=self.process_name,
supervisor_uri=self._supervisor_uri,
log_tail=self._initial_log_tail
)
self._monitor.start()
def handle_event(self, message, _):
try:
self.command_handlers[message['key']](message)
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_set(self, data):
if data.get('name'):
self._monitor.process_name = data['name']
if data.get('tail'):
self._monitor.log_tail = data['tail']
def cleanup(self):
self._monitor.stop()
@@ -1,22 +1,15 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from os import remove
from contextlib import suppress
import xmlrpc.client
from xmlrpc.client import Fault as SupervisorFault
from contextlib import suppress
from os import remove
from tfw.decorators import lazy_property
from tfw.config import TFWENV
class SupervisorBaseMixin:
@lazy_property
def supervisor(self):
return xmlrpc.client.ServerProxy(TFWENV.SUPERVISOR_HTTP_URI).supervisor
class SupervisorBase:
def __init__(self, supervisor_uri):
self.supervisor = xmlrpc.client.ServerProxy(supervisor_uri).supervisor
class SupervisorMixin(SupervisorBaseMixin):
class ProcessManager(SupervisorBase):
def stop_process(self, process_name):
with suppress(SupervisorFault):
self.supervisor.stopProcess(process_name)
@@ -29,7 +22,7 @@ class SupervisorMixin(SupervisorBaseMixin):
self.start_process(process_name)
class SupervisorLogMixin(SupervisorBaseMixin):
class ProcessLogManager(SupervisorBase):
def read_stdout(self, process_name, tail=0):
return self.supervisor.readProcessStdoutLog(process_name, -tail, 0)
+1
View File
@@ -0,0 +1 @@
from .snapshot_handler import SnapshotHandler
@@ -0,0 +1,78 @@
import logging
from os.path import join as joinpath
from os.path import basename
from os import makedirs
from datetime import datetime
from dateutil import parser as dateparser
from .snapshot_provider import SnapshotProvider
LOG = logging.getLogger(__name__)
class SnapshotHandler:
keys = ['snapshot']
def __init__(self, *, directories, snapshots_dir, exclude_unix_patterns=None):
self._snapshots_dir = snapshots_dir
self.snapshot_providers = {}
self._exclude_unix_patterns = exclude_unix_patterns
self.init_snapshot_providers(directories)
self.command_handlers = {
'snapshot.take': self.handle_take_snapshot,
'snapshot.restore': self.handle_restore_snapshot,
'snapshot.exclude': self.handle_exclude
}
def init_snapshot_providers(self, directories):
for index, directory in enumerate(directories):
git_dir = self.init_git_dir(index, directory)
self.snapshot_providers[directory] = SnapshotProvider(
directory,
git_dir,
self._exclude_unix_patterns
)
def init_git_dir(self, index, directory):
git_dir = joinpath(
self._snapshots_dir,
f'{basename(directory)}-{index}'
)
makedirs(git_dir, exist_ok=True)
return git_dir
def handle_event(self, message, _):
try:
self.command_handlers[message['key']](message)
except KeyError:
LOG.error('IGNORING MESSAGE: Invalid message received: %s', message)
def handle_take_snapshot(self, _):
LOG.debug('Taking snapshots of directories %s', self.snapshot_providers.keys())
for provider in self.snapshot_providers.values():
provider.take_snapshot()
def handle_restore_snapshot(self, message):
date = dateparser.parse(
message.get(
'value',
datetime.now().isoformat()
)
)
LOG.debug(
'Restoring snapshots (@ %s) of directories %s',
date,
self.snapshot_providers.keys()
)
for provider in self.snapshot_providers.values():
provider.restore_snapshot(date)
def handle_exclude(self, message):
exclude_unix_patterns = message['value']
if not isinstance(exclude_unix_patterns, list):
raise KeyError
for provider in self.snapshot_providers.values():
provider.exclude = exclude_unix_patterns
@@ -0,0 +1,221 @@
import re
from subprocess import run, CalledProcessError, PIPE
from getpass import getuser
from os.path import isdir
from os.path import join as joinpath
from uuid import uuid4
from dateutil import parser as dateparser
class SnapshotProvider:
def __init__(self, directory, git_dir, exclude_unix_patterns=None):
self._classname = self.__class__.__name__
author = f'{getuser()} via TFW {self._classname}'
self.gitenv = {
'GIT_DIR': git_dir,
'GIT_WORK_TREE': directory,
'GIT_AUTHOR_NAME': author,
'GIT_AUTHOR_EMAIL': '',
'GIT_COMMITTER_NAME': author,
'GIT_COMMITTER_EMAIL': '',
'GIT_PAGER': 'cat'
}
self._init_repo()
self.__last_valid_branch = self._branch
if exclude_unix_patterns:
self.exclude = exclude_unix_patterns
def _init_repo(self):
self._check_environment()
if not self._repo_is_initialized:
self._run(('git', 'init'))
if self._number_of_commits == 0:
try:
self._snapshot()
except CalledProcessError:
raise EnvironmentError(f'{self._classname} cannot init on empty directories!')
self._check_head_not_detached()
def _check_environment(self):
if not isdir(self.gitenv['GIT_DIR']) or not isdir(self.gitenv['GIT_WORK_TREE']):
raise EnvironmentError(f'{self._classname}: "directory" and "git_dir" must exist!')
@property
def _repo_is_initialized(self):
return self._run(
('git', 'status'),
check=False
).returncode == 0
@property
def _number_of_commits(self):
return int(
self._get_stdout((
'git', 'rev-list',
'--all',
'--count'
))
)
def _snapshot(self):
self._run((
'git', 'add',
'-A'
))
try:
self._get_stdout((
'git', 'commit',
'-m', 'Snapshot'
))
except CalledProcessError as err:
if b'nothing to commit, working tree clean' not in err.output:
raise
def _check_head_not_detached(self):
if self._head_detached:
raise EnvironmentError(f'{self._classname} cannot init from detached HEAD state!')
@property
def _head_detached(self):
return self._branch == 'HEAD'
@property
def _branch(self):
return self._get_stdout((
'git', 'rev-parse',
'--abbrev-ref', 'HEAD'
))
def _get_stdout(self, *args, **kwargs):
kwargs['stdout'] = PIPE
kwargs['stderr'] = PIPE
stdout_bytes = self._run(*args, **kwargs).stdout
return stdout_bytes.decode().rstrip('\n')
def _run(self, *args, **kwargs):
if 'check' not in kwargs:
kwargs['check'] = True
if 'env' not in kwargs:
kwargs['env'] = self.gitenv
return run(*args, **kwargs)
@property
def exclude(self):
with open(self._exclude_path, 'r') as ofile:
return ofile.read()
@exclude.setter
def exclude(self, exclude_patterns):
with open(self._exclude_path, 'w') as ifile:
ifile.write('\n'.join(exclude_patterns))
@property
def _exclude_path(self):
return joinpath(
self.gitenv['GIT_DIR'],
'info',
'exclude'
)
def take_snapshot(self):
if self._head_detached:
self._checkout_new_branch_from_head()
self._snapshot()
def _checkout_new_branch_from_head(self):
branch_name = str(uuid4())
self._run((
'git', 'branch',
branch_name
))
self._checkout(branch_name)
def _checkout(self, what):
self._run((
'git', 'checkout',
what
))
def restore_snapshot(self, date):
commit = self._get_commit_from_timestamp(date)
branch = self._last_valid_branch
if commit == self._latest_commit_on_branch(branch):
commit = branch
self._checkout(commit)
def _get_commit_from_timestamp(self, date):
commit = self._get_stdout((
'git', 'rev-list',
'--date=iso',
'-n', '1',
f'--before="{date.isoformat()}"',
self._last_valid_branch
))
if not commit:
commit = self._get_oldest_parent_of_head()
return commit
def _get_oldest_parent_of_head(self):
return self._get_stdout((
'git',
'rev-list',
'--max-parents=0',
'HEAD'
))
@property
def _last_valid_branch(self):
if not self._head_detached:
self.__last_valid_branch = self._branch
return self.__last_valid_branch
def _latest_commit_on_branch(self, branch):
return self._get_stdout((
'git', 'log',
'-n', '1',
'--pretty=format:%H',
branch
))
@property
def all_timelines(self):
return self._branches
@property
def _branches(self):
git_branch_output = self._get_stdout(('git', 'branch'))
regex_pattern = re.compile(r'(?:[^\S\n]|[*])') # matches '*' and non-newline whitespace chars
return re.sub(regex_pattern, '', git_branch_output).splitlines()
@property
def timeline(self):
return self._last_valid_branch
@timeline.setter
def timeline(self, value):
self._checkout(value)
@property
def snapshots(self):
return self._pretty_log_branch()
def _pretty_log_branch(self):
git_log_output = self._get_stdout((
'git', 'log',
'--pretty=%H@%aI'
))
commits = []
for line in git_log_output.splitlines():
commit_hash, timestamp = line.split('@')
commits.append({
'hash': commit_hash,
'timestamp': dateparser.parse(timestamp)
})
return commits
+3
View File
@@ -0,0 +1,3 @@
from .terminal_handler import TerminalHandler
from .terminal_commands_handler import TerminalCommandsHandler
from .commands_equal import CommandsEqual
+107
View File
@@ -0,0 +1,107 @@
from shlex import split
from re import search
from tfw.internals.lazy import lazy_property
class CommandsEqual:
# pylint: disable=too-many-arguments
"""
This class is useful for comparing executed commands with
excepted commands (i.e. when triggering a state change when
the correct command is executed).
Note that in most cases you should test the changes
caused by the commands instead of just checking command history
(stuff can be done in countless ways and preparing for every
single case is impossible). This should only be used when
testing the changes would be very difficult, like when
explaining stuff with cli tools and such.
This class implicitly converts to bool, use it like
if CommandsEqual(...): ...
It tries detecting differing command parameter orders with similar
semantics and provides fuzzy logic options.
The rationale behind this is that a few false positives
are better than only accepting a single version of a command
(i.e. using ==).
"""
def __init__(
self, command_1, command_2,
fuzzyness=1, begin_similarly=True,
include_patterns=None, exclude_patterns=None
):
"""
:param command_1: Compared command 1
:param command_2: Compared command 2
:param fuzzyness: float between 0 and 1.
the percentage of arguments required to
match between commands to result in True.
i.e 1 means 100% - all arguments need to be
present in both commands, while 0.75
would mean 75% - in case of 4 arguments
1 could differ between the commands.
:param begin_similarly: bool, the first word of the commands
must match
:param include_patterns: list of regex patterns the commands
must include
:param exclude_patterns: list of regex patterns the commands
must exclude
"""
self.command_1 = split(command_1)
self.command_2 = split(command_2)
self.fuzzyness = fuzzyness
self.begin_similarly = begin_similarly
self.include_patterns = include_patterns
self.exclude_patterns = exclude_patterns
def __bool__(self):
if self.begin_similarly:
if not self.beginnings_are_equal:
return False
if self.include_patterns is not None:
if not self.commands_contain_include_patterns:
return False
if self.exclude_patterns is not None:
if not self.commands_contain_no_exclude_patterns:
return False
return self.similarity >= self.fuzzyness
@lazy_property
def beginnings_are_equal(self):
return self.command_1[0] == self.command_2[0]
@lazy_property
def commands_contain_include_patterns(self):
return all((
self.contains_regex_patterns(self.command_1, self.include_patterns),
self.contains_regex_patterns(self.command_2, self.include_patterns)
))
@lazy_property
def commands_contain_no_exclude_patterns(self):
return all((
not self.contains_regex_patterns(self.command_1, self.exclude_patterns),
not self.contains_regex_patterns(self.command_2, self.exclude_patterns)
))
@staticmethod
def contains_regex_patterns(command, regex_parts):
command = ' '.join(command)
for pattern in regex_parts:
if not search(pattern, command):
return False
return True
@lazy_property
def similarity(self):
parts_1 = set(self.command_1)
parts_2 = set(self.command_2)
difference = parts_1 - parts_2
deviance = len(difference) / len(max(parts_1, parts_2))
return 1 - deviance
@@ -1,29 +1,12 @@
# Copyright (C) 2018 Avatao.com Innovative Learning Kft.
# All Rights Reserved. See LICENSE file for details.
from os.path import dirname
from re import findall
from re import compile as compileregex
from abc import ABC, abstractmethod
from watchdog.events import PatternMatchingEventHandler
from tfw.mixins import CallbackMixin, ObserverMixin
from tfw.decorators import RateLimiter
from tfw.internals.networking import Intent
from tfw.internals.inotify import InotifyObserver
class CallbackEventHandler(PatternMatchingEventHandler, ABC):
def __init__(self, files, *callbacks):
super().__init__(files)
self.callbacks = callbacks
@RateLimiter(rate_per_second=2)
def on_modified(self, event):
for callback in self.callbacks:
callback()
class HistoryMonitor(CallbackMixin, ObserverMixin, ABC):
class HistoryMonitor(ABC, InotifyObserver):
"""
Abstract class capable of monitoring and parsing a history file such as
bash HISTFILEs. Monitoring means detecting when the file was changed and
@@ -35,29 +18,30 @@ class HistoryMonitor(CallbackMixin, ObserverMixin, ABC):
command pattern property and optionally the sanitize_command method.
See examples below.
"""
def __init__(self, histfile):
def __init__(self, uplink, histfile):
self.histfile = histfile
self._history = []
self._last_length = len(self._history)
self.observer.schedule(
CallbackEventHandler(
[self.histfile],
self._fetch_history,
self._invoke_callbacks
),
dirname(self.histfile)
)
self.history = []
self._last_length = len(self.history)
self.uplink = uplink
super().__init__(self.histfile)
@property
def history(self):
return self._history
@abstractmethod
def domain(self):
raise NotImplementedError()
def on_modified(self, event):
self._fetch_history()
if self._last_length < len(self.history):
for command in self.history[self._last_length:]:
self.send_message(command)
def _fetch_history(self):
self._last_length = len(self._history)
self._last_length = len(self.history)
with open(self.histfile, 'r') as ifile:
pattern = compileregex(self.command_pattern)
data = ifile.read()
self._history = [
self.history = [
self.sanitize_command(command)
for command in findall(pattern, data)
]
@@ -71,9 +55,11 @@ class HistoryMonitor(CallbackMixin, ObserverMixin, ABC):
# pylint: disable=no-self-use
return command
def _invoke_callbacks(self):
if self._last_length < len(self._history):
self._execute_callbacks(self.history)
def send_message(self, command):
self.uplink.send_message({
'key': f'history.{self.domain}',
'command': command
}, intent=Intent.EVENT)
class BashMonitor(HistoryMonitor):
@@ -86,6 +72,10 @@ class BashMonitor(HistoryMonitor):
shopt -s histappend
unset HISTCONTROL
"""
@property
def domain(self):
return 'bash'
@property
def command_pattern(self):
return r'.+'
@@ -99,6 +89,10 @@ class GDBMonitor(HistoryMonitor):
HistoryMonitor to monitor GDB sessions.
For this to work "set trace-commands on" must be set in GDB.
"""
@property
def domain(self):
return 'gdb'
@property
def command_pattern(self):
return r'(?<=\n)\+(.+)\n'

Some files were not shown because too many files have changed in this diff Show More