Unified agent host charm (#437)

* Use config options instead of resource files for ssh keys (#328)

* Shared agent code location (#329)

* Move run_with_logged_errors out of charm.py

* Update testflinger source and add action for it

* Add integration tests and docs for the update-testflinger action

* cleanup

* Add update-configs action (#337)

* Write out the supervisord configs for all agents (#351)

* Write out the supervisord configs for all agents

* Update the supervisor services when configs/code is updated

* Update README for the agent charm with instructions for test/debug

* redirect stderr for all supervisord agents

* Fix charm to expect base64 encoded ssh keys as already documented

* important fix - reservations won't work without this

Any script (like ssh-copy-id) which uses $HOME, or ~ or anything like
that won't work without this. Although supervisord execs the process as
the specified user, there's no shell. So we need to set certain vars
like this in the supervisord config.

* Also install the device-connector so that things like reserve work

* Rename ssh config options for better consistency

* Update testflinger-agent-host-charm README

* Update tf-cmd-scripts to work with the new agent-host charm (#355)

* Add a config option with a sane default for the agent host ssh config (#370)

* Add agent-host terraform module (#362)

* Add agent-host terraform module

* Separate variables from main.tf

* Agent host terraform constraints (#387)

* Fix some formatting issues in the README.md

* Add variables to build constraints to the agent-host terraform module

* Update tf-cleanup for this branch to keep current with PR#382 that was just merged (#405)

* Remove vars no longer used from tf-cleanup

* Unit/Integration test fixes for testflinger-agent-host-charm

agent/charms/testflinger-agent-host-charm/README.md

@@ -1,4 +1,6 @@
-# Overview
+# Testflinger Agent Host Charm
+
+## Overview
 
 This charm provides the base system for a host system that will be used for
 testflinger device agents. It installs the base dependencies and provides a
@@ -7,17 +9,138 @@ target for deploying the "testflinger-agent" along with
 in `src/tf-cmd-scripts/` to the host system. The scripts would be used by the
 testflinger-agent to trigger the testflinger-device-connector at each phase.
 
-# Building
+## Building
 To build this charm, first install charmcraft (`sudo snap install --classic
 charmcraft`) then run `charmcraft pack`
 
-# Configuration
+## Testing
+This charm includes both unit and integration tests. The integration tests
+take some time to run and require setting up juju in your test environment
+as described [here](https://juju.is/docs/sdk/dev-setup#heading--manual-set-up-juju).
+
+Once you've done that, you can run all the unit and integration tests by
+going up 2 directories from here to the `agent` directory, and running:
+
+```
+$ tox -e charm
+```
+
+For debugging, it can be useful to keep the model that was deployed so that
+it can be reused. To do this, you can add a few additional arguments to tox:
+```
+$ tox -e charm -v -- --model=testmodel --keep-models
+```
+
+This will create a model called `testmodel` and keep it after the run is
+complete. If you want to reuse it without deploying again, you can
+run the same command again with `--no-deploy` at the end.
+
+## Configuration
 Supported options for this charm are:
 
-  - ssh-priv-key:
+  - ssh-private-key:
       base64 encoded ssh private keyfile
-  - ssh-pub-key:
+  - ssh-public-key:
       base64 encoded ssh public keyfile
+  - ssh-config:
+      config data to write as ~/.ssh/config
+  - config-repo:
+      Git repo containing device agent config data
+  - config-branch:
+      Git branch to pull for the config data
+  - config-dir:
+      Path from the root of the config repo where the directories and configs are located for this agent host
 
 To keep the tf-cmd-scripts files up-to-date, run `juju upgrade-charm
-{testflinger-agent-host-application}`.
+{testflinger-agent-host-application}`.
+
+The config-repo where the Testflinger configs for the agents are stored should
+contain a directory tree. The leaf directories should be named the same as the
+agent-id configured for the agent in the `testflinger-agent.conf` file, and
+this directory should contain the config files for both the agent and the
+device connector.  For example, consider a repo containing the configs for
+multiple locations and agents. It might have a structure like this:
+```
+/
+- lab1/
+  - agent-101/
+    - testflinger-agent.conf
+    - default.yaml
+  - agent-102/
+    - testflinger-agent.conf
+    - default.yaml
+  - ...
+- lab2/
+  - agent-201/
+    - testflinger-agent.conf
+    - default.yaml
+  - ...
+...
+```
+
+In order to make the charm consider only the agents under the `lab1` directory,
+you should set the config-dir to `lab1`. 
+
+## Actions
+The following actions are supported for this charm:
+
+  - update-testflinger:
+      This action is used to update the testflinger-agent and install it to a
+      location shared by all the agents running on this host.
+      This action will trigger all running agents to restart when they are not running a job.
+  - update-configs:
+      This action pulls the git repo set in the charm config to update the
+      agent configs on the agent host.
+      This action will run `supervisorctl update` to force start new agents, and stop any agents
+      that are no longer configured. This *could* also force restart any agents, if the
+      supervisord config for that agent has changed. This should not normally happen. This action
+      will also trigger all running agents to restart when they are not running a job in case the
+      testflinger config has changed.
+
+## Operational Notes
+
+### Using supervisorctl on the agent host to check status
+
+The agent host is configured to use supervisorctl to manage the agents. From the agent host,
+you can run `sudo supervisorctl status` to see the status of all the agents configured on it.
+
+### Viewing the agent logs
+To show the logs for a specific agent, run `sudo supervisorctl tail <agent name>`.
+You can also use the `-f` option to follow the logs.
+To show the logs for supervisorctl itself, to see what it's recently started, stopped, or
+signalled, you can use `supervisorctl maintail`.
+
+### Stopping and restarting agents
+You can use `sudo supervisorctl stop <agent name>` to stop a specific agent.
+Be aware that other actions on the charm such as `update-configs` might later cause this to
+restart the agents. In order to mark it offline so that it will no longer process jobs, you may
+want to either remove it completely from the configs, or set a disable marker file using
+`touch /tmp/TESTFLINGER-DEVICE-OFFLINE-<agent name>` instead.
+
+To signal an agent to safely restart when it's no longer running a job, you can run
+`sudo supervisorctl signal USR1 <agent name>`. 
+
+### Updating agent configs
+
+If the agent configs in the config-repo have changed, you can run
+`juju run testflinger-agent-host-application/0 update-configs` (use `run-action` for juju < 3.x)
+to update the agent configs on the agent host. This will automatically add any new agents,
+remove any agents that are no longer configured, and all agents will be sent a signal to restart
+when they are not running a job.
+
+### Updating the Testflinger repo
+
+When the code for the agent or device connectors changes, you can run
+`juju run testflinger-agent-host-application/0 update-testflinger` to update the Testflinger
+repo on the agent host. This will automatically trigger a safe restart on all the agents after
+installing the new code.
+
+### Changes to the supervisord configs
+
+Be aware that if you change anything in the supervisord configs under /etc/supervisor/conf.d,
+the agents will be **forced** to restart without waiting for running jobs to terminate the next
+time you run the `update-configs` action or perform a configuration change in the charm. This
+is because it runs `supervisorctl update`. This should not normally happen, but if for some
+reason you need to make a change to the charm that causes it to write changes to the supervisord
+configs, then you should take precautions to ensure that the agents are not running jobs when
+`supervisord update` is run the next time.

agent/charms/testflinger-agent-host-charm/actions.yaml

@@ -0,0 +1,4 @@
+update-configs:
+  description: Update Testflinger agent configs
+update-testflinger:
+  description: Update Testflinger agent code

agent/charms/testflinger-agent-host-charm/config.yaml

@@ -0,0 +1,29 @@
+options:
+  config-repo:
+    type: string
+    description: Git repo containing device agent config data
+    default: ""
+  config-branch:
+    type: string
+    description: Git branch to pull for the config data
+    default: "main"
+  config-dir:
+    type: string
+    description: Path from the root of the config repo where the directories and configs are located for this agent host
+    default: ""
+  ssh-private-key:
+    type: string
+    description: ssh private key for connecting to local test devices
+    default: ""
+  ssh-public-key:
+    type: string
+    description: ssh public key for connecting to local test devices
+    default: ""
+  ssh-config:
+    type: string
+    description: ssh config for connecting to local test devices
+    default: |
+      StrictHostKeyChecking no
+      UserKnownHostsFile /dev/null
+      LogLevel QUIET
+      ConnectTimeout 30

agent/charms/testflinger-agent-host-charm/metadata.yaml

@@ -10,12 +10,3 @@ description: |
   This charm provides support for device agents running in the hwcert lab
 summary: |
   Host system for device agents
-resources:
-  ssh_priv_key:
-    type: file
-    filename: id_rsa
-    description: SSH private key
-  ssh_pub_key:
-    type: file
-    filename: id_rsa.pub
-    description: SSH public key

agent/charms/testflinger-agent-host-charm/requirements.txt

@@ -1 +1,3 @@
 ops >= 1.4.0
+GitPython==3.1.43
+Jinja2==3.1.4