From 0970b0d27f9740368dea92ca155fbacffc6abcf3 Mon Sep 17 00:00:00 2001 From: MengT Date: Thu, 26 Oct 2023 14:18:16 +0800 Subject: [PATCH 1/5] Doc: add job definition and how-to guides --- docs/.wordlist.txt | 6 +++ docs/how-to/cancel-job.rst | 13 ++++++ docs/how-to/change-server.rst | 36 +++++++++++++++ docs/how-to/index.rst | 11 ++++- docs/how-to/install-cli.rst | 33 ++++++++++++++ docs/how-to/submit-job.rst | 21 +++++++++ docs/reference/index.rst | 1 + docs/reference/job-schema.rst | 84 +++++++++++++++++++++++++++++++++++ 8 files changed, 204 insertions(+), 1 deletion(-) create mode 100644 docs/how-to/cancel-job.rst create mode 100644 docs/how-to/change-server.rst create mode 100644 docs/how-to/install-cli.rst create mode 100644 docs/how-to/submit-job.rst create mode 100644 docs/reference/job-schema.rst diff --git a/docs/.wordlist.txt b/docs/.wordlist.txt index 2bd80412..af6a70c8 100644 --- a/docs/.wordlist.txt +++ b/docs/.wordlist.txt @@ -4,6 +4,7 @@ APIs artifact artifacts balancer +BG CharmHub CLI CM3 @@ -32,11 +33,13 @@ Juju Kubeflow Kubernetes Lenovo +lifecycle logfile maas MaaS MAAS Makefile +microservice MUX muxpi MuxPi @@ -67,6 +70,7 @@ runtime SDWire SecureBoot SKU +SSID subdirectories subfolders subtree @@ -79,7 +83,9 @@ Ubuntu UI URI USB +virtualenv VM webhook +WPA xenial yaml diff --git a/docs/how-to/cancel-job.rst b/docs/how-to/cancel-job.rst new file mode 100644 index 00000000..1a87c5e7 --- /dev/null +++ b/docs/how-to/cancel-job.rst @@ -0,0 +1,13 @@ +Cancel a job +=============== + +You can cancel a Testflinger job at any point. However, if the job is in the middle of provisioning, it will finish provisioning before cancelling it completely. This prevents accidentally leaving a system in a bad, partially provisioned state. + +To cancel a job, run Testflinger CLI with the ``cancel`` argument and provide the job ID: + +.. code-block:: shell + + $ testflinger-cli cancel + + +If a job reaches the global timeout or any timeout for a specific operation, such as the output timeout, Testflinger will cancel the job automatically to release resources. In this case, you can resubmit the job or extend the timeout to allow more execution time. diff --git a/docs/how-to/change-server.rst b/docs/how-to/change-server.rst new file mode 100644 index 00000000..79f9d8de --- /dev/null +++ b/docs/how-to/change-server.rst @@ -0,0 +1,36 @@ +Connect to a different server +============================== + +By default, Testflinger CLI uses the default Testflinger server located at ``https://testflinger.canonical.com``. To specify a different server to use, you can either run Testflinger CLI with the ``--server`` parameter, or set an environment variable. + + +Specify the server in command line +--------------------------------------- + +Run the Testflinger CLI command with an ``--server `` argument. The server URI must start with either ``http://`` or ``https://``. For example: + +.. code-block:: shell + + $ testflinger-cli --server https://testflinger.example.com jobs + +However, if the environment variable ``TESTFLINGER_SERVER`` is set, this argument will be ignored. + + +Set an environment variable +----------------------------- + +If you want to use a server URI for all operations, you can set the environment variable ``TESTFLINGER_SERVER``. This variable overwrites the default server address and the string specified by the ``--server`` argument. + +To set an environment variable on Ubuntu: + +.. code-block:: shell + + $ export TESTFLINGER_SERVER="https://testflinger.example.com" + +If you want to change the environment variable permanently, add the export command to a system-wide or user-specific configuration file. + +To verify that the variable has been set, run: + +.. code-block:: shell + + $ printenv TESTFLINGER_SERVER diff --git a/docs/how-to/index.rst b/docs/how-to/index.rst index a5ddc12f..ecea6efa 100644 --- a/docs/how-to/index.rst +++ b/docs/how-to/index.rst @@ -1,6 +1,15 @@ How-to guides -============= +============== These how-to guides cover key operations and processes in Testflinger. +Work with jobs via Testflinger CLI +----------------------------------- +.. toctree:: + :maxdepth: 1 + + install-cli + change-server + submit-job + cancel-job diff --git a/docs/how-to/install-cli.rst b/docs/how-to/install-cli.rst new file mode 100644 index 00000000..978a3be1 --- /dev/null +++ b/docs/how-to/install-cli.rst @@ -0,0 +1,33 @@ +Install Testflinger CLI +======================== + +The ``testflinger-cli`` client is a command line tool used for interacting with Testflinger servers. You can use the client to submit test jobs to the devices under test (DUT), check the job status and get testing results. + +You can either install ``testflinger-cli`` through Snap or check out the code from our GitHub repository and run the tool in a Python virtual environment. + + +Install via Snap +----------------- +The most convenient way to get the CLI tool is via snap: + +.. code-block:: shell + + $ sudo snap install testflinger-cli + + +Install in virtual environment +------------------------------- + +If you are using the CLI from an automated test runner, such as Jenkins, you may want to install the tool in a virtual environment instead. + +To run it from the source code, please make sure that the ``python3-click`` and ``python3-requests`` packages are installed, and then run the following commands: + +.. code-block:: shell + + $ git clone https://github.com/canonical/testflinger + $ cd cli + $ virtualenv -p python3 env + $ . env/bin/activate + $ pip install . + + diff --git a/docs/how-to/submit-job.rst b/docs/how-to/submit-job.rst new file mode 100644 index 00000000..ddb04db1 --- /dev/null +++ b/docs/how-to/submit-job.rst @@ -0,0 +1,21 @@ +Submit a test job +=================== + +Once you have a YAML or JSON file with your :doc:`job definition <../reference/job-schema>`, you can submit the job to the main Testflinger server using the CLI: + +.. code-block:: shell + + $ testflinger-cli submit example-job.yaml + + +If you want to submit the jobs to a Testflinger server other than the default one, see :doc:`change-server`. + +If the job is successful submitted, you will see a ``job_id`` returned by Testflinger: + +.. code-block:: shell + + Job submitted successfully! + job_id: 2bac1457-0000-0000-0000-15f23f69fd39 + + +You can use the ``job_id`` to further monitor and manager the submitted job. diff --git a/docs/reference/index.rst b/docs/reference/index.rst index 83ef8a42..cc32f893 100644 --- a/docs/reference/index.rst +++ b/docs/reference/index.rst @@ -9,6 +9,7 @@ Working with device connectors .. toctree:: :maxdepth: 1 + job-schema device-connector-types device-connector-conf diff --git a/docs/reference/job-schema.rst b/docs/reference/job-schema.rst new file mode 100644 index 00000000..bd6ff572 --- /dev/null +++ b/docs/reference/job-schema.rst @@ -0,0 +1,84 @@ +Test job schema +================= + +Test jobs can be defined in either YAML or JSON format. + +The following table lists the key elements that a job definition file should contain. + +.. list-table:: Test job schema + :header-rows: 1 + + * - Field + - Type + - Description + * - ``job_queue`` + - String + - Name of the job queue to which you want to submit the job. This field is mandatory. + * - ``global_timeout`` + - integer + - | (Optional) Maximum time (in seconds) a job is allowed to run. If the global timeout is reached while a job is still running, Testflinger will close the job immediately. By default, the timeout for each Testflinger agent is 4 hours. + | If your test job will presumably take more than 4 hours to complete, you can set a larger number in the ``global_timeout`` field in the test job. However, if a timeout is specified on a device, the value for this parameter will be overridden by the per-device configurations. + * - ``output_timeout`` + - integer + - (Optional) Maximum time (in seconds) Testflinger should wait in the ``test`` phase to get output from the job. If the timeout is reached before the test phase has any output, Testflinger will cancel the job. This value should be smaller than, or equal to, the ``global_timeout``. The default value is 15 minutes. + * - ``allocation_timeout`` + - integer + - (Optional) Maximum time (in seconds) Testflinger should wait in the ``allocate`` phase for multi-device jobs to reach the ``allocated`` state. If the timeout is reached before all devices are allocated, Testflinger will cancel the job. The default timeout to allocate all devices is 2 hours. + * - ``_data`` + - dictionary + - | (Optional) Sections that define the configurations and commands about how the job should be executed in each of the test phases. If any phase encounters a non-zero exit code, no further phases will be executed. + | Supported test phases include: + | - provision + | - firmware_update + | - test + | - allocate + | - reserve + | For detailed information about how to define the data to include in each test phase, see :doc:`test-phases`. + +Environment variables +---------------------------- + +The following environment variables are supported in the test environment on the HOST system (not on the device under test). You can refer to these variables when defining a job: + +.. list-table:: Environment variables on the HOST system + :header-rows: 1 + + * - Name + - Description + * - ``WPA_BG_SSID`` + - SSID for BG WiFi access point + * - ``WPA_BG_PSK`` + - WPA Key for BG WiFi access point + * - ``WPA_N_SSID`` + - SSID for N WiFi access point + * - ``WPA_N_PSK`` + - WPA Key for N WiFi access point + * - ``WPA_AC_SSID`` + - SSID for AC WiFi access point + * - ``WPA_AC_PSK`` + - WPA Key for AC WiFi access point + * - ``OPEN_BG_SSID`` + - SSID for OPEN BG WiFi access point + * - ``OPEN_N_SSID`` + - SSID for OPEN N WiFi access point + * - ``OPEN_AC_SSID`` + - SSID for OPEN AC WiFi access point + * - ``DEVICE_IP`` + - IP address of the device under test (used for SSH) + + +Example job in YAML +---------------------------- + +The following example YAML file defines a job that provisions the Ubuntu 22.04-jammy system on the device, and print the distribution-specific information on the provisioned device: + +.. code-block:: yaml + + job-queue: example-queue + global-timeout: 28800 + output-timeout: 3600 + provision_data: + distro: jammy + test_data: + test_cmds: | + ssh -t ubuntu@DEVICE_IP lsb_release -a From eb5af02b1dd47007f78e430d906f71560b444d65 Mon Sep 17 00:00:00 2001 From: MengT Date: Thu, 9 Nov 2023 16:20:30 +0200 Subject: [PATCH 2/5] address comments --- docs/reference/job-schema.rst | 53 ++++++++++------------------------- 1 file changed, 15 insertions(+), 38 deletions(-) diff --git a/docs/reference/job-schema.rst b/docs/reference/job-schema.rst index bd6ff572..5a76402d 100644 --- a/docs/reference/job-schema.rst +++ b/docs/reference/job-schema.rst @@ -10,22 +10,31 @@ The following table lists the key elements that a job definition file should con * - Field - Type + - Default - Description * - ``job_queue`` - String + - / - Name of the job queue to which you want to submit the job. This field is mandatory. * - ``global_timeout`` - integer - - | (Optional) Maximum time (in seconds) a job is allowed to run. If the global timeout is reached while a job is still running, Testflinger will close the job immediately. By default, the timeout for each Testflinger agent is 4 hours. - | If your test job will presumably take more than 4 hours to complete, you can set a larger number in the ``global_timeout`` field in the test job. However, if a timeout is specified on a device, the value for this parameter will be overridden by the per-device configurations. + - | 14400 + | (4 hours) + - | (Optional) Maximum time (in seconds) a job is allowed to run. If the global timeout is reached while a job is still running, Testflinger will close the job immediately. + | You can set a global timeout larger than the default value, but the per-device configuration might have a more restrict timeout and overwrite the global value. * - ``output_timeout`` - integer - - (Optional) Maximum time (in seconds) Testflinger should wait in the ``test`` phase to get output from the job. If the timeout is reached before the test phase has any output, Testflinger will cancel the job. This value should be smaller than, or equal to, the ``global_timeout``. The default value is 15 minutes. + - | 900 + | (15 minutes) + - (Optional) Maximum time (in seconds) Testflinger should wait in the ``test`` phase to get output from the job. If the timeout is reached before the test phase has any output, Testflinger will cancel the job. This value should be smaller than, or equal to, the ``global_timeout``. * - ``allocation_timeout`` - integer - - (Optional) Maximum time (in seconds) Testflinger should wait in the ``allocate`` phase for multi-device jobs to reach the ``allocated`` state. If the timeout is reached before all devices are allocated, Testflinger will cancel the job. The default timeout to allocate all devices is 2 hours. + - | 7200 + | (2 hours) + - (Optional) Maximum time (in seconds) Testflinger should wait in the ``allocate`` phase for multi-device jobs to reach the ``allocated`` state. If the timeout is reached before all devices are allocated, Testflinger will cancel the job. * - ``_data`` - dictionary + - / - | (Optional) Sections that define the configurations and commands about how the job should be executed in each of the test phases. If any phase encounters a non-zero exit code, no further phases will be executed. | Supported test phases include: | - provision @@ -35,42 +44,10 @@ The following table lists the key elements that a job definition file should con | - reserve | For detailed information about how to define the data to include in each test phase, see :doc:`test-phases`. -Environment variables ----------------------------- - -The following environment variables are supported in the test environment on the HOST system (not on the device under test). You can refer to these variables when defining a job: - -.. list-table:: Environment variables on the HOST system - :header-rows: 1 - - * - Name - - Description - * - ``WPA_BG_SSID`` - - SSID for BG WiFi access point - * - ``WPA_BG_PSK`` - - WPA Key for BG WiFi access point - * - ``WPA_N_SSID`` - - SSID for N WiFi access point - * - ``WPA_N_PSK`` - - WPA Key for N WiFi access point - * - ``WPA_AC_SSID`` - - SSID for AC WiFi access point - * - ``WPA_AC_PSK`` - - WPA Key for AC WiFi access point - * - ``OPEN_BG_SSID`` - - SSID for OPEN BG WiFi access point - * - ``OPEN_N_SSID`` - - SSID for OPEN N WiFi access point - * - ``OPEN_AC_SSID`` - - SSID for OPEN AC WiFi access point - * - ``DEVICE_IP`` - - IP address of the device under test (used for SSH) - - Example job in YAML ---------------------------- -The following example YAML file defines a job that provisions the Ubuntu 22.04-jammy system on the device, and print the distribution-specific information on the provisioned device: +The following example YAML file defines a job that retrieves the Ubuntu 22.04-jammy system image from the given URL, provisions the image on the device at IP address ``DEVICE_IP``, and prints the distribution-specific information on the provisioned device: .. code-block:: yaml @@ -78,7 +55,7 @@ The following example YAML file defines a job that provisions the Ubuntu 22.04-j global-timeout: 28800 output-timeout: 3600 provision_data: - distro: jammy + url: https://cdimage.ubuntu.com/ubuntu/releases/jammy/release/example.img.xz test_data: test_cmds: | ssh -t ubuntu@DEVICE_IP lsb_release -a From 9b167be79606e9f371d6658c868bda98faf9ba9c Mon Sep 17 00:00:00 2001 From: MengT Date: Fri, 17 Nov 2023 20:32:49 +0800 Subject: [PATCH 3/5] update device type in job example --- docs/.wordlist.txt | 1 + docs/reference/job-schema.rst | 20 ++++++++++++++++---- 2 files changed, 17 insertions(+), 4 deletions(-) diff --git a/docs/.wordlist.txt b/docs/.wordlist.txt index af6a70c8..1b088b60 100644 --- a/docs/.wordlist.txt +++ b/docs/.wordlist.txt @@ -34,6 +34,7 @@ Kubeflow Kubernetes Lenovo lifecycle +LTS logfile maas MaaS diff --git a/docs/reference/job-schema.rst b/docs/reference/job-schema.rst index 5a76402d..0c4b1f2a 100644 --- a/docs/reference/job-schema.rst +++ b/docs/reference/job-schema.rst @@ -44,18 +44,30 @@ The following table lists the key elements that a job definition file should con | - reserve | For detailed information about how to define the data to include in each test phase, see :doc:`test-phases`. -Example job in YAML +Example jobs in YAML ---------------------------- -The following example YAML file defines a job that retrieves the Ubuntu 22.04-jammy system image from the given URL, provisions the image on the device at IP address ``DEVICE_IP``, and prints the distribution-specific information on the provisioned device: +The following example YAML file defines a job that provisions the Ubuntu Core 22 system on a Raspberry Pi 4 device. The job retrieves the image from the given URL, provisions the image on the device at IP address ``DEVICE_IP``, and prints the distribution-specific information on the provisioned device: .. code-block:: yaml - job-queue: example-queue + job_queue: raspi4b global-timeout: 28800 output-timeout: 3600 provision_data: - url: https://cdimage.ubuntu.com/ubuntu/releases/jammy/release/example.img.xz + url: https://cdimage.ubuntu.com/ubuntu-core/22/stable/current/ubuntu-core-22-arm64+raspi.img.xz test_data: test_cmds: | ssh -t ubuntu@DEVICE_IP lsb_release -a + +Data specified in the ``provision_data`` section varies on device types. For example, to provision server images on a MAAS device, the ``distro`` field should be used to indicate the system version. The following YAML file defines a job that provisions the Ubuntu 22.04 LTS (Jammy Jellyfish) server install image on a MAAS device and prints the information about its processors and network interface configurations: + +.. code-block:: yaml + + job_queue: maas-x86-node + provision_data: + distro: jammy + test_data: + test_cmds: | + ssh ubuntu@$DEVICE_IP cat /proc/cpuinfo + ssh ubuntu@$DEVICE_IP ifconfig From ac8f39765a58d0bf02214ea49973607ae06db8fb Mon Sep 17 00:00:00 2001 From: MengT Date: Mon, 27 Nov 2023 21:09:22 +0800 Subject: [PATCH 4/5] update job name in example --- docs/reference/job-schema.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/reference/job-schema.rst b/docs/reference/job-schema.rst index 0c4b1f2a..3633e877 100644 --- a/docs/reference/job-schema.rst +++ b/docs/reference/job-schema.rst @@ -51,7 +51,7 @@ The following example YAML file defines a job that provisions the Ubuntu Core 22 .. code-block:: yaml - job_queue: raspi4b + job_queue: rpi4b global-timeout: 28800 output-timeout: 3600 provision_data: From 24ac5a67f379a090cbfed586205fb0ea5e94c4a6 Mon Sep 17 00:00:00 2001 From: MengT Date: Tue, 28 Nov 2023 09:58:12 +0800 Subject: [PATCH 5/5] resolve conflict --- docs/.wordlist.txt | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/.wordlist.txt b/docs/.wordlist.txt index 1b088b60..7a977a77 100644 --- a/docs/.wordlist.txt +++ b/docs/.wordlist.txt @@ -34,7 +34,6 @@ Kubeflow Kubernetes Lenovo lifecycle -LTS logfile maas MaaS @@ -84,9 +83,11 @@ Ubuntu UI URI USB -virtualenv +UUID VM webhook WPA xenial yaml +LTS +virtualenv