Remove CLI (#139)

* Remove the CLI

* Massive reorganization of the sources related to the CLI removal

* Fix the compiler test

* Improve MyPy configs

* Fix typing issues in the demo app

* Add new test for the demo in place of the old CLI-based one.

* Fix a python-3.7-specific typing issue in the demo app

* Fix mypy and add coverage

* Windows-specific fix: inherit %SYSTEMROOT% to avoid appveyor/ci#1995

* Naturally, there is no SYSTEMROOT on GNU/Linux

* Pass the PATH as well

* Final typing issues

* Remove obsolete entries from .gitignore

* Sync up the DSDL compiler with recent changes to the CLI tool

* Correct a minor enum misuse

* A weakly related change: ongoing work on CLI revealed that low send timeout values may cause issues depending on the performance of the garbage collector

* Update the demos

* Remove unnecessary copy

* Remove unnecessary runtime typing checks

* Improve error handling in transport session setup: detect and raise errors as early as possible. The old behavior was a legacy from the time when it was possible to change the local node-ID at runtime; it no longer makes sense.

* Update the heartbeat publisher to make it compatible with the case where the transport is anonymous and the output session cannot be created

* Add node test in place of the removed CLI test

* MyPy fixes

* Fix tests

* Add coverage of an important case in RedundantTransport

* Remove deprecated constructs

* Relax type in update_from_builtin()

* Update references to the command line tool per https://forum.uavcan.org/t/migrating-the-pyuavcan-cli-tool-into-a-different-package/1039/5

* Drop unused test setup code

.gitignore

@@ -50,8 +50,3 @@ coverage.xml
 # MS stuff
 *.code-workspace
 .vscode
-
-# DSDL generated namespaces
-/uavcan/*
-/sirius_cyber_corp/*
-/test_dsdl_namespace/*

.gitmodules

@@ -1,3 +1,3 @@
 [submodule "public_regulated_data_types_for_testing"]
-	path = tests/public_regulated_data_types
+	path = demo/public_regulated_data_types
 	url = https://github.com/UAVCAN/public_regulated_data_types

CONTRIBUTING.rst

@@ -16,23 +16,16 @@ When adding new tools and such, please put all their configuration there to keep
 All shippable entities are located exclusively inside the directory ``pyuavcan/``.
 The entirety of the directory is shipped.
 
-The directory ``tests/public_regulated_data_types/`` is needed only for testing and documentation building.
+The submodule ``demo/public_regulated_data_types/`` is needed only for demo, testing, and documentation building.
 It should be kept reasonably up-to-date, but remember that it does not affect the final product in any way.
 We no longer ship DSDL namespaces with code for reasons explained in the user documentation.
 
 Please desist from adding any new VCS submodules or subtrees.
 
-The usage demo scripts that are included in the user documentation are located under ``tests/demo/``.
-This is probably mildly surprising since one would expect to find docs under ``docs/``,
-but it is done this way to facilitate testing and static analysis of the demo scripts.
+The demos that are included in the user documentation are located under ``demo/``.
 Whenever the test suite is run, it tests the demo application as well in order to ensure that it is correct and
 compatible with the current version of the library -- keeping the docs up-to-date is vitally important.
 
-The CLI tool is auto-tested as well, the tests are located under ``tests/cli/``.
-It's somewhat trickier than with the rest of the code because it requires us to
-launch new processes and keep track of their code coverage metrics;
-the details are explained in a dedicated section.
-
 There are major automation scripts located in the source root directory.
 You will need them if you are developing the library; please open them and read the comments inside to understand
 how they work and how to use them.
@@ -45,7 +38,7 @@ Third-party dependencies
 
 The general rule is that external dependencies are to be avoided unless doing so would increase the complexity
 of the codebase considerably.
-There are three kinds of 3rd-party dependencies used by this library:
+There are two kinds of 3rd-party dependencies used by this library:
 
 - **Core dependencies.** Those are absolutely required to use the library.
   The list of core deps contains two libraries: Nunavut and NumPy, and it is probably not going to be extended ever
@@ -59,9 +52,6 @@ There are three kinds of 3rd-party dependencies used by this library:
   conventions in the user documentation and in ``setup.cfg``.
   When developing new transports or media sub-layers, try to avoid adding new dependencies.
 
-- **Other dependencies.** Those are needed for some minor optional components and features of the library,
-  such as the CLI tool.
-
 
 Coding conventions
 ------------------
@@ -187,7 +177,7 @@ The scanner should not be run before the full general test suite since it relies
 
 When writing tests, aim to cover at least 90% of branches, excepting the DSDL generated packages (at least for now)
 (the DSDL test data is synthesized at run time).
-Ensure that your tests do not emit any errors or warnings into the CLI output upon successful execution,
+Ensure that your tests do not emit any errors or warnings into stderr output upon successful execution,
 because that may distract the developer from noticing true abnormalities
 (you may use ``caplog.at_level('CRITICAL')`` to suppress undesirable output).
 
@@ -214,22 +204,6 @@ Certain tests require real-time execution.
 If they appear to be failing with timeout errors and such, consider re-running them on a faster system.
 It is recommended to run the test suite with at least 2 GB of free RAM and an SSD.
 
-Auto-tests may spawn new processes, e.g., to test the CLI tool. In order to keep their code coverage measured,
-we have put the coverage setup code into a special module ``sitecustomize.py``, which is auto-imported
-every time a new Python interpreter is started (as long as the module's path is in ``PYTHONPATH``, of course).
-Hence, every invocation of Python made during testing is coverage-tracked, which is great.
-This is why we don't invoke ``coverage`` manually when running tests.
-After the tests are executed, we end up with some dozen or more of ``.coverage*`` files scattered across the
-source directories.
-The scattered coverage files are then located automatically and combined into one file,
-which is then analyzed by report generators and other tools like SonarQube.
-
-When tests that spawn new processes fail, they may leave their children running in the background,
-which may adversely influence other tests that are executed later,
-so an error in one test may crash a dozen of unrelated ones invoked afterwards.
-You need to be prepared for that and always start analyzing the test report starting with the first failure.
-Ideally, though, this should be fixed by adding robust cleanup logic for each test.
-
 Some of the components of the library and of the test suite require DSDL packages to be generated.
 Those must be dealt with carefully as it needs to be ensured that the code that requires generated
 packages to be available is not executed until they are generated.
@@ -239,7 +213,7 @@ and other higher-level components are tested against them.
 At least the following locations should be checked first:
 
 - ``tests/presentation`` -- generic presentation layer test cases.
-- ``tests/cli`` -- CLI and demo test cases.
+- ``tests/demo`` -- demo test cases.
 - The list may not be exhaustive, please grep the sources to locate all relevant modules.
 
 Many tests rely on the DSDL-generated packages being available for importing.
@@ -265,14 +239,6 @@ Normally, this should be done a few months after a new version of CPython is rel
 If the CI/CD pipelines pass, you are all set.
 
 
-Debugging
----------
-
-When debugging argument parsing issues in the CLI,
-you won't see any stacktrace unless verbose logging is enabled before the argument parser is constructed.
-To work around that, use the environment variable `PYUAVCAN_LOGLEVEL` (see the user docs for details).
-
-
 Releasing
 ---------
 
@@ -300,3 +266,5 @@ Make sure to mark it as a source directory to enable code completion and type an
 (for PyCharm: right click -> Mark Directory As -> Sources Root).
 
 Configure the IDE to remove trailing whitespace on save in the entire file.
+Or, even better, configure a File Watcher to run Black on save
+(make sure to disable running it on external file changes though).

README.md

@@ -13,7 +13,6 @@ Full-featured UAVCAN stack in Python
 
 PyUAVCAN is a full-featured implementation of the UAVCAN protocol stack intended for non-embedded,
 user-facing applications such as GUI software, diagnostic tools, automation scripts, prototypes, and various R&D cases.
-PyUAVCAN consists of a Python library (package) and a simple CLI tool for basic diagnostics and shell script automation.
 
 PyUAVCAN aims to support all features and transport layers of UAVCAN,
 be portable across all major platforms supporting Python,
@@ -31,3 +30,6 @@ The acronym *UAVCAN* stands for ***Uncomplicated Application-level Vehicular Com
 **READ THE DOCS: [pyuavcan.readthedocs.io](https://pyuavcan.readthedocs.io/)**
 
 **Ask questions: [forum.uavcan.org](https://forum.uavcan.org/)**
+
+*See also: [**U-tool**](https://github.com/UAVCAN/u) -- a CLI tool for diagnostics and management of
+UAVCAN networks built on top of PyUAVCAN.*

clean.sh

@@ -2,9 +2,6 @@
 
 rm -rf dist build ./*.egg-info .coverage* htmlcov .*cache .*_generated *.log *.tmp &> /dev/null
 
-# DSDL-generated packages
-rm -rf uavcan sirius_cyber_corp test_dsdl_namespace &> /dev/null
-
 pushd docs || exit 1
 rm -rf _build .coverage* .*_generated
 popd || exit 1

demo/README.md

@@ -0,0 +1,7 @@
+PyUAVCAN demo application
+=========================
+
+This directory contains the demo application.
+It is invoked and verified by the integration test suite along with the main library codebase.
+
+Please refer to the official library documentation for details about this demo.

tests/demo/demo_app.py → demo/demo_app.py

@@ -1,6 +1,6 @@
 #!/usr/bin/env python3
 #
-# PyUAVCAN demo application. This file is included in the user documentation, please keep it tidy.
+# PyUAVCAN demo application.
 #
 # Distributed under CC0 1.0 Universal (CC0 1.0) Public Domain Dedication. To the extent possible under law, the
 # UAVCAN Consortium has waived all copyright and related or neighboring rights to this work.
@@ -45,18 +45,18 @@
     import sirius_cyber_corp  # This is our vendor-specific root namespace. Custom data types.
     import pyuavcan.application  # The application module requires the standard types from the root namespace "uavcan".
 except (ImportError, AttributeError):
-    src_dir = os.path.abspath(os.path.dirname(__file__))
-    # Generate our vendor-specific namespace. It may make use of the standard data types (most namespaces do,
+    src_dir = pathlib.Path(__file__).resolve().parent
+    # Generate our application-specific namespace. It may make use of the standard data types (most namespaces do,
     # because the standard root namespace contains important basic types), so we include it in the lookup path set.
     # The paths are hard-coded here for the sake of conciseness.
     pyuavcan.dsdl.generate_package(
-        root_namespace_directory=os.path.join(src_dir, "../dsdl/namespaces/sirius_cyber_corp/"),
-        lookup_directories=[os.path.join(src_dir, "../public_regulated_data_types/uavcan")],
+        root_namespace_directory=src_dir / "custom_data_types/sirius_cyber_corp",
+        lookup_directories=[src_dir / "public_regulated_data_types/uavcan/"],
         output_directory=dsdl_generated_dir,
     )
     # Generate the standard namespace. The order actually doesn't matter.
     pyuavcan.dsdl.generate_package(
-        root_namespace_directory=os.path.join(src_dir, "../public_regulated_data_types/uavcan"),
+        root_namespace_directory=src_dir / "public_regulated_data_types/uavcan/",
         output_directory=dsdl_generated_dir,
     )
     # Okay, we can try importing again. We need to clear the import cache first because Python's import machinery
@@ -73,7 +73,7 @@
 
 
 class DemoApplication:
-    def __init__(self):
+    def __init__(self) -> None:
         # The interface to run the demo against is selected via the environment variable with a default option provided.
         # Virtual CAN bus is supported only on GNU/Linux, but other interfaces used here should be compatible
         # with at least Windows as well.
@@ -88,6 +88,7 @@ def __init__(self):
         # For example, anonymous node cannot be a server, since without an ID it cannot be addressed.
         # Here, we assign a node-ID statically, because this is a simplified demo.
         # Most applications would need this to be configurable, some may support the PnP node-ID allocation protocol.
+        transport: pyuavcan.transport.Transport
         if interface_kind == "udp" or not interface_kind:  # This is the default.
             # The UDP/IP transport in this example runs on the local loopback interface, so no setup is needed.
             # The UDP transport requires us to specify the IP address; the node-ID equals the value of several least
@@ -122,6 +123,7 @@ def __init__(self):
             media_1 = pyuavcan.transport.can.media.socketcan.SocketCANMedia(f"vcan1", mtu=32)
             media_2 = pyuavcan.transport.can.media.socketcan.SocketCANMedia(f"vcan2", mtu=64)
             # All transports in a redundant group MUST share the same node-ID.
+            assert isinstance(transport, pyuavcan.transport.redundant.RedundantTransport)
             transport.attach_inferior(pyuavcan.transport.can.CANTransport(media_0, local_node_id=42))
             transport.attach_inferior(pyuavcan.transport.can.CANTransport(media_1, local_node_id=42))
             transport.attach_inferior(pyuavcan.transport.can.CANTransport(media_2, local_node_id=42))
@@ -134,6 +136,7 @@ def __init__(self):
             # may be observed with wired+wireless links used concurrently; see https://forum.uavcan.org/t/557.
             # All transports in a redundant group MUST share the same node-ID.
             transport = pyuavcan.transport.redundant.RedundantTransport()
+            assert isinstance(transport, pyuavcan.transport.redundant.RedundantTransport)
             transport.attach_inferior(pyuavcan.transport.udp.UDPTransport("127.0.0.42"))
             transport.attach_inferior(
                 pyuavcan.transport.serial.SerialTransport("socket://localhost:50905", local_node_id=42)
@@ -163,7 +166,7 @@ def __init__(self):
         self._node = pyuavcan.application.Node(presentation, node_info)
 
         # Published heartbeat fields can be configured trivially by assigning them on the heartbeat publisher instance.
-        self._node.heartbeat_publisher.mode = uavcan.node.Mode_1_0.OPERATIONAL
+        self._node.heartbeat_publisher.mode = uavcan.node.Mode_1_0.OPERATIONAL  # type: ignore
         # The vendor-specific status code is the two least significant decimal digits of the local process' PID.
         self._node.heartbeat_publisher.vendor_specific_status_code = os.getpid() % 100
 
@@ -219,10 +222,10 @@ async def _serve_linear_least_squares_request(
         self._pub_diagnostic_record.publish_soon(diagnostic_msg)
 
         # This is just the business logic.
-        sum_x = sum(map(lambda p: p.x, request.points))
-        sum_y = sum(map(lambda p: p.y, request.points))
-        a = sum_x * sum_y - len(request.points) * sum(map(lambda p: p.x * p.y, request.points))
-        b = sum_x * sum_x - len(request.points) * sum(map(lambda p: p.x ** 2, request.points))
+        sum_x = sum(map(lambda p: p.x, request.points))  # type: ignore
+        sum_y = sum(map(lambda p: p.y, request.points))  # type: ignore
+        a = sum_x * sum_y - len(request.points) * sum(map(lambda p: p.x * p.y, request.points))  # type: ignore
+        b = sum_x * sum_x - len(request.points) * sum(map(lambda p: p.x ** 2, request.points))  # type: ignore
         try:
             slope = a / b
             y_intercept = (sum_y - slope * sum_x) / len(request.points)
@@ -241,7 +244,7 @@ async def _serve_linear_least_squares_request(
             self._pub_diagnostic_record.publish_soon(
                 uavcan.diagnostic.Record_1_1(
                     severity=uavcan.diagnostic.Severity_1_0(uavcan.diagnostic.Severity_1_0.INFO),
-                    text=f'Solution for {",".join(f"({p.x},{p.y})" for p in request.points)}: {slope}, {y_intercept}',
+                    text=f"Solution for {','.join(f'({p.x},{p.y})' for p in request.points)}: {slope}, {y_intercept}",
                 )
             )
             return sirius_cyber_corp.PerformLinearLeastSquaresFit_1_0.Response(slope=slope, y_intercept=y_intercept)
@@ -295,9 +298,7 @@ async def _handle_temperature(
             )
         ):
             print(
-                "Diagnostic message could not be sent in",
-                self._pub_diagnostic_record.send_timeout,
-                "seconds",
+                f"Diagnostic publication timed out in {self._pub_diagnostic_record.send_timeout} seconds",
                 file=sys.stderr,
             )
 
@@ -309,14 +310,17 @@ async def _handle_temperature(
     async def list_tasks_periodically() -> None:
         """Print active tasks periodically for demo purposes."""
         while True:
-            print(
-                "\nRunning tasks:\n" + "\n".join(f"{i:4}: {t.get_coro()}" for i, t in enumerate(asyncio.all_tasks())),
-                file=sys.stderr,
-            )
+            if sys.version_info >= (3, 8):  # The task introspection API we use is not available before Python 3.8
+                print(
+                    "\nRunning tasks:\n"
+                    + "\n".join(f"{i:4}: {t.get_coro()}" for i, t in enumerate(asyncio.all_tasks())),
+                    file=sys.stderr,
+                )
+            else:
+                print(f"\nRunning {len(asyncio.all_tasks())} tasks")
             await asyncio.sleep(10)
 
-    if sys.version_info >= (3, 8):  # The task introspection API we use is not available before Python 3.8
-        asyncio.get_event_loop().create_task(list_tasks_periodically())
+    asyncio.get_event_loop().create_task(list_tasks_periodically())
 
     # The node and PyUAVCAN objects have created internal tasks, which we need to run now.
     # In this case we want to automatically stop and exit when no tasks are left to run.

docs/conf.py

@@ -30,7 +30,7 @@
 
 # The generated files are not documented, but they must be importable to import the target package.
 DSDL_GENERATED_ROOT = REPOSITORY_ROOT / ".test_dsdl_generated"
-PUBLIC_REGULATED_DATA_TYPES_ROOT = REPOSITORY_ROOT / "tests" / "public_regulated_data_types"
+PUBLIC_REGULATED_DATA_TYPES_ROOT = REPOSITORY_ROOT / "demo" / "public_regulated_data_types"
 
 sys.path.insert(0, str(REPOSITORY_ROOT))
 sys.path.insert(0, str(DSDL_GENERATED_ROOT))

docs/index.rst

@@ -20,7 +20,6 @@ Contents
 
    pages/architecture
    pages/installation
-   pages/cli
    pages/demo
    pages/faq
    pages/api

docs/pages/architecture.rst

@@ -9,8 +9,6 @@ Overview
 PyUAVCAN is a full-featured implementation of the `UAVCAN protocol stack <https://uavcan.org>`_
 intended for non-embedded, user-facing applications such as GUI software, diagnostic tools,
 automation scripts, prototypes, and various R&D cases.
-PyUAVCAN consists of a Python library (package) and a simple :abbr:`CLI (command line interface)`
-tool for basic diagnostics and shell script automation.
 It is designed to support **GNU/Linux**, **MS Windows**, and **macOS** as first-class target platforms.
 
 The reader should understand the basics of `UAVCAN <https://uavcan.org/>`_ and be familiar with
@@ -329,9 +327,7 @@ For more information, read the API docs for :mod:`pyuavcan.util`.
 Command-line tool
 -----------------
 
-The command-line tool named ``pyuavcan`` (like the library)
-can be installed as described in the :ref:`installation` chapter.
-Run ``pyuavcan --help`` (or ``python -m pyuavcan --help``)
-to see the usage documentation, or read the :ref:`cli` chapter.
-
-The tool also serves as a (somewhat convoluted) library usage demo.
+There is an independent but related project that is built on top of PyUAVCAN:
+`Yakut <https://github.com/UAVCAN/yakut>`_.
+It is a command-line interface utility for diagnostics and management of UAVCAN networks.
+Consider it as an extensive collection of practical usage examples for PyUAVCAN.