[CERTTF-303] Introduce support for file attachments

[boukeas]

Description

This PR introduces functionality that allows a testflinger client (e.g. the testflinger CLI) to submit a job that includes attachments, with these attachments eventually finding their way into the testflinger agent.

Resolved issues

CERTTF-303

• Removes the need to create configuration files as here documents within the test_cmds field; the configuration files can be attached instead.
• Bash scripts can be attached and used in the test_cmds field
• Attached files can also be useful in the provisioning or firmware_update phases
• Will serve as a step for addressing RTW-282

Changes

Server

• Extend the Job submission schema so that attachments fields are allowed under test_data, specifying which local files should be packed into an archive and submitted along with a job. For example:

test_data:
  attachments:
    - local: "/var/log/syslog"
      agent: "logs/syslog"
    - local: "data/config.json"
    - local: "data/scripts/script.sh"
      agent: "test.sh"

• Such fields are also allowed under the provisioning and firmware update data but the current schema doesn't validate against them because too many arbitrary fields are used in these phases, depending on the use case. This validation issue is not relevant to this PR.

• In the implementation of the endpoint for submitting a job, a new derived attachments status field is stored in the job data, taking the values:

  • none when there are no attachments to the job
  • waiting when there are attachments but the corresponding archive file hasn't been submitted yet and
  • complete when there are attachments and the corresponding archive file has been submitted

  Only the values none or complete allow a job to be removed from a queue.

• Introduce two endpoints for submitting and retrieving an archive file that contains the attachments for a specific job. The submission of an attachments archive sets the value of the attachments status field for the job to complete.

• Move database-related code into a separate database module, so that the API implementation is database-agnostic. This has been partly realised: the database interactions that are relevant to this PR have been moved into reusable functions of the database module. If approved, this change can also be applied to the implementation of all API endpoints, eventually removing all references to MongoDB from the API implementation and making the database an injectable dependency into the app.

CLI

The CLI checks if the job submission includes references to attachments in the provisioning, firmware and test phases and, if so, packs the attachments into an archive and follows the job submission with an additional attachments archive submission to the server (using the new endpoint described above)

Agent

The agent checks if the job submission is associated with attachments and, if so, retrieves the attachments archive from the server, unpacks it and optionally moves/renames the attachments. Note that all attachments are placed under an attachments folder in the agents "rundir". They are additionally separated into phase-related folders, e.g. attachments/test for the attachments provided under test_data.

Points for discussion

• The intention is that the current functionality will be later extended to attaching files that are intended to end up in the device (rather than the host).
• Efficiency concerns: how will the server behave when large files are submitted and retrieved? This is the main reason that the job submission still only includes the JSON, without the attachments archive (as a multipart-encoded request). If the job submission was bundled with a large attachments archive then it would take much longer to complete. Attachment archives are submitted and retrieved separately. In the future, this might also allow for asynchronous transfers.
• There is an evident need for tooling modules that are common among the CLI, the server and the agent. There is code that would be (re)usable and is currently isolated and repeated.
• How to test in a non-local staging environment

Documentation

Currently only inline documentation is provided. If the approach is approved the documentation will be extended under a separate issue/PR.

Tests

• The test suite has been extended on the CLI, server and agent sides to provide new unit tests for the new functionality. Some additional tests should be added to test for negative scenarios, i.e. behaviour under unexpected circumstances.
• Manual tests have been performed in a local testflinger deployment, using this job YAML, to verify that the specified files do indeed find their way to the agent, under the specified names.

job_queue: myqueue
provision_data:
  attachments:
    - local: "README.md"
firmware_update_data:
  attachments:
    - local: "CONTRIBUTING.md"
test_data:
  attachments:
    - local: "/var/log/syslog"
      agent: "logs/syslog"
    - local: "data/config.json"
    - local: "data/scripts/script.sh"
      agent: "test.sh"
  test_cmds: |
    ls -alR
    echo "*** These are the contents of the script file"
    cat attachments/test/test.sh
    echo "*** Running the script file"
    echo "(it's supposed to print the contents of the config file)"
    chmod u+x attachments/test/test.sh
    attachments/test/test.sh attachments/test/data/config.json

[nadzyah]

Thanks for your contribution! I’ve left several suggestions to make it more clear

[plars]

I really love some of the improvements you made here, great work! I have a few comments below, let me know what you think.

[thp-canonical]

This is a welcome feature addition, thanks for working on this! See comments.

[plars]

This looks reasonable to me. I haven't had a chance to try it end-to-end yet, but I know you did and we will also want to do it in staging as a final sanity check. However I had one other thing that I wanted to ask below about how we ensure the attachments are removed once they are uploaded?
I did see that you have an uploadDate timestamp on it to use for TTL. Would be nice if we could delete-on-use, but I'm ok with the TTL, but didn't see where it was getting set in the index data. Did I just miss it?
Update: Nevermind! I realized about 10 seconds after sending this that we were already using uploadDate as the TTL index, so this should be just fine.
If you are happy with your local testing, then I am ok with this, pending further testing in staging.

[plars]

One minor suggestion about a test for the secure_filter() update that you just made, otherwise looks good

[plars]

Thanks for adding that! +1

[nadzyah]

Also want to approve it since I don’t have questions that can be potential blockers. Good PR, well done c: