job-runner: testing runs isolated to their own container

[allisonkarlitskaya]

Introduce job-runner, a new asyncio python application designed to replace a good chunk of our backend CI infra:

• tests-invoke
• make-checkout
• s3-streamer
• (eventually, maybe) run-queue

and dramatically simplify others:

• issue-scan
• tests-scan

while making it easier to play with our testing infrastructure locally.

The main new feature vs. the existing stuff is that each testing job is run in its own one-time-use container. The image for that container is under control of the revision being tested, allowing for gating of task container upgrades (avoiding the usual wave of broken pixels, linting changes, etc.).

Instead of running our jobs from the giant blobs of shell pasted together in issue-scan or tests-scan we consume them as the new "Job" json format, already introduced in #5932.

The new approach also makes a conscious effort to reduce the amount of exposure of secrets to test runs (although we have some more work to do here).

TODO:

• Includes workflows: Update to podman 4 in cockpituous workflow #6018 to fix tests
• Needs tasks: Add job-runner.toml configuration cockpituous#586 ; once that lands, revert the "TEMP HACK" commit
• Needs tasks: Bind the host's podman API socket cockpituous#583 deployed to prod (it isn't yet) before landing, so that it will actually work
• Needs tasks: Add integration test for image-refresh via issue-scan cockpituous#588 to ensure we don't break image refreshes
• Add host name to log, like the old s3-streamer
• cidfile crash after successful runs https://paste.centos.org/view/01fcdfc8 with podman-remote; this breaks attachments, thus a blocker not reproducible, ignore

[allisonkarlitskaya]

lolwat

[allisonkarlitskaya]

Random note: while hacking on this, despite all kinds of exceptions and irregular exits for all kinds of reasons (including many programming errors), I haven't seen a single leaked container in over two weeks (when I got the finally: block working properly to remove the container image).

[martinpitt]

First round of review. We need to talk 😁

[martinpitt]

test_pr() failed. The inner bots tests succeeded (and the statuses bubble is green here), but the output check failed:

ERROR: did not find 'Running on:.*cockpituous' in '[test log]`

That used to be done by the old s3-streamer:

s3-streamer:                Running on:    {platform.node()}

While we can of course change the string format, the node where job-runner runs on is vitally important for maintenance. So this needs to be put back. I'll add a TODO entry to the description for this.

[martinpitt]

I adjusted the expected test output in cockpit-project/cockpituous#587 and added a FIXUP commit here which adds back 'Running on:". The integration test passes now.

[martinpitt]

This failure looks very relevant:

Error: statfs /tmp/tmp7wu7vgej/checkout-and-run: no such file or directory

Fortunately it already happens in test_mock_pr(), and it effortlessly reproduces in COCKPIT_BOTS_BRANCH=job-runner tasks/run-local.sh.

This is because we are using podman-remote in that test (and will in prod), so that temp dir will indeed not exist.

[martinpitt]

Thanks! I ignored the design complexity aspect now, and looked at some implementation questions/bugs.

[martinpitt]

Let's do this! ⭐ 🚀 🏁