From af4c13eee9f771a8f4575131c0a1ed954941ccc4 Mon Sep 17 00:00:00 2001 From: Stefan Kraus Date: Thu, 4 Jan 2024 11:52:25 +0100 Subject: [PATCH] doc: robot: Add docs about robotframework usage This change adds documentation about the usage of the newly implemented robotframework harness. Hereby, general setup and usage, together with added ZephyrLibrary keywords, are described. Signed-off-by: Stefan Kraus --- doc/develop/test/index.rst | 1 + doc/develop/test/robot.rst | 123 +++++++++++++++++++++++++++++++++++++ 2 files changed, 124 insertions(+) create mode 100644 doc/develop/test/robot.rst diff --git a/doc/develop/test/index.rst b/doc/develop/test/index.rst index c8c67d20ef64e45..434fe5a9777633b 100644 --- a/doc/develop/test/index.rst +++ b/doc/develop/test/index.rst @@ -9,6 +9,7 @@ Testing ztest twister pytest + robot coverage BabbleSim ztest_deprecated diff --git a/doc/develop/test/robot.rst b/doc/develop/test/robot.rst new file mode 100644 index 000000000000000..4ec19262ea10aec --- /dev/null +++ b/doc/develop/test/robot.rst @@ -0,0 +1,123 @@ +.. integration-with-robot: + +Integration with robot framework +################################ + +*Please mind that integration of twister with robot is still work in progress. Not every platform +type is supported in robot (yet). If you find any issue with the integration or have an idea for +an improvement, please, let us know about it and open a GitHub issue/enhancement.* + +.. note:: + + This section of the documentation describes the integration of generic Python Robot Framework + library -- currently called `robotframework` -- in twister; in contrast to the specific renode + harness which is called just `robot`. + +Introduction +************ + +*“Robot Framework is a generic open source automation framework. It can be used for test automation +and robotic process automation (RPA).“* (``_). +The focus of Robot Framework is especially on acceptance tests. + +The basic behaviour and implementation is similar to the pytest implementation. + +Integration with twister +************************ + +By default, there is nothing to be done to enable robot support in twister. The plugin is +developed as a part of Zephyr’s tree. To enable install-less operation, twister first extends +``PYTHONPATH`` with path to this plugin. If one prefers to use the installed version of the plugin, +they must add ``--allow-installed-plugin`` flag to twister’s call. + +Robot Framework test are discovered the same way as other twister tests, i.e., by a presence +of testcase.yaml or sample.yaml. Inside, a keyword ``harness`` tells twister how to handle a given test. +In the case of ``harness: robotframework``, most of twister’s workflow (test suites discovery, +parallelization, building and reporting) remains the same as for other harnesses. The change +happens during the execution step. + +If ``harness: robotframework`` is used, twister calls a robot_wrapper entrypoint in `pytest-twister-harness` that +handles proper argument parsing from twister (e.g., baudrate, serial device) and calls the +Python3 Robot Framework library afterwards. +This is needed to not replicate all argument parsing from pytest and robot, but having all configuration +available in the robot execution later. +After test execution, twister searches for a XUnit report file (robot_report.xml) containing test results, and +set its own results accordingly. + + +How to create a robot test +************************** + +An example of a robot test is given at :zephyr_file:`samples/subsys/testsuite/robotframework/shell/robot/shell_module.robot`. +Twister calls robot for each configuration from the .yaml file which uses ``harness: robotframework``. +By default, it points to ``robot`` directory, located next to a directory with binary sources. +A keyword ``robot_test_path`` placed under ``harness_config`` section can be used to point to other +files, directories or subtests. + +One can also pass some extra arguments to robot from yaml file using ``robot_args`` keyword +under ``harness_config``, e.g.: ``robot_args: [‘-t=test_method’, ‘-t=test_method’]``. +There is also an option to pass ``--robot-args`` through Twister command line parameters. +This can be particularly useful when one wants to select a specific testcase from a test suite. +For instance, one can use a command: + +.. code-block:: console + + $ ./scripts/twister -p qemu_cortex_m3 --device-testing \ + -T samples/subsys/testsuite/robotframework/shell/ \ + --robot-args="--SuiteStatLevel 2 --Metadata Version:3" + + +Note that ``--robot-args`` can be passed multiple times to pass several arguments to the robot framework. + +Robot keywords +============== + +Besides some common keywords provided by Robot Framework themselves, there is a +'ZephyrLibrary' that can be loaded and used in the Robot test files. + +.. code-block:: robotframework + *** Settings *** + Library twister_harness.robot_framework.ZephyrLibrary + +This library provides additional, Zephyr specific keywords and phrases that +can be used to handle interaction with the device. + + +Get a Device +------------ + +Give access to a ``DeviceAdapter`` type object, that represents Device Under Test. +This is the core of the following interaction. +It is required to launch and interact with other keywords. +It yields a device prepared according to the requested type +(``native``, ``qemu``, ``hardware``, etc.). All types of devices share the same API. +This allows for writing tests which are device-type-agnostic. + + +Run Device +---------- + +This keyword flashes and launches a device that was previously acquired by 'Get a Device'. +Otherwise, the Device will keep its previous state. + + +Run Command +------------------------ + +This connects to the running device or emulator via a serial console and executes +the given command ```` via shell. Note that the target must be +compiled with ``CONFIG_SHELL=y``. +It waits for a proper response of the ``wait_for_prompt`` method, so it does not +start the scenario until the DUT is ready. + + +Close Device +------------ + +This closes the serial connection to the device. + + +Limitations +*********** + +* Not every platform type is supported in the plugin (yet).