Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tasks: Add job-runner.toml configuration #586

Merged
merged 2 commits into from
Mar 7, 2024
Merged

tasks: Add job-runner.toml configuration #586

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
11b1591
tests: Don't rebuild container on changes to `install-service`
martinpitt Mar 7, 2024
1daed3e
tasks: Add job-runner.toml configuration
martinpitt Mar 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/tests.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,7 +46,7 @@ jobs:
- name: Check which containers changed
id: containers_changed
run: |
tasks=$(git diff --name-only origin/main..HEAD -- tasks/ | grep -Ev 'run-local.sh|openssl.cnf|README|mock-|.yaml' || true)
tasks=$(git diff --name-only origin/main..HEAD -- tasks/ | grep -Ev 'run-local.sh|openssl.cnf|install-service|README|mock-|.yaml' || true)
# print for debugging
echo "tasks: $tasks"
[ -z "$tasks" ] || echo "tasks=true" >> "$GITHUB_OUTPUT"
Expand Down
48 changes: 48 additions & 0 deletions ansible/roles/tasks-systemd/tasks/main.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,54 @@
maxsockets=3
cafile=/run/secrets/tasks/npm-registry.crt

- name: Create job-runner configuration
copy:
dest: /etc/job-runner.toml
content: |
[logs]
driver='s3'

[forge.github]
token = [{file="/run/secrets/webhook/.config--github-token"}]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(feel free to say no) can you get us an extra --volume for /var/secrets/github-token and use it from that location?


[logs.s3]
# bots lib/stores.py LOG_STORE
url = 'https://cockpit-logs.us-east-1.linodeobjects.com/'
key = [{file="/run/secrets/tasks/s3-keys/cockpit-logs.us-east-1.linodeobjects.com"}]
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ditto for this (ie: /run/secrets/s3-keys/cockpit-logs.us-east-1.linodeobjects.com)


[container]
command = ['podman-remote', '--url=unix:///podman.sock']
run-args = [
# resources
'--device=/dev/kvm',
'--memory=24g',
'--pids-limit=16384',
'--shm-size=1024m',
# qcow overlays on tmpfs
'--tmpfs=/tmp:size=14g',
'--env=TEST_OVERLAY_DIR=/tmp',
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@allisonkarlitskaya FYI, I just added this so that we can eventually drop

cockpituous/tasks/Containerfile

Line 81 in accb1fd

TEST_OVERLAY_DIR=/tmp
which is a bit ugly, too disconnected from the other half (i.e. the --tmpfs), and won't work with other containers.

I also reordered these by purpose instead of primarily on type, which makes it a bit easier to maintain/understand.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I feel like a bit more vertical whitespace would help keeping the sections visually separated from each other...

# local image stores
'--env=COCKPIT_IMAGE_STORES_FILE=/config/image-stores',
'--volume=/var/cache/cockpit-tasks/image-stores:/config/image-stores:ro',
# generic secrets
'--volume=/var/lib/cockpit-secrets/tasks/npm-registry.crt:/run/secrets/tasks/npm-registry.crt:ro',
# various configuration
'--volume=/etc/npmrc:/etc/npmrc:ro',
'--volume=/var/cache/cockpit-tasks/images:/cache/images:rw',
'--env=TEST_JOBS={{ TEST_JOBS | default(8) }}',
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

How about GIT_AUTHOR_NAME and GIT_AUTHOR_EMAIL? I'm not sure if we also need COMMITTER but I assume it'll default to author if it's not set?

]

[container.secrets]
# these are *host* paths, this is podman-remote
image-upload=[
'--volume=/var/lib/cockpit-secrets/tasks/s3-keys/:/run/secrets/s3-keys:ro',
'--env=COCKPIT_S3_KEY_DIR=/run/secrets/s3-keys',
]
github-token=[
'--volume=/var/lib/cockpit-secrets/webhook/.config--github-token:/run/secrets/github-token:ro',
'--env=COCKPIT_GITHUB_TOKEN_FILE=/run/secrets/github-token',
]

- name: Set up systemd service for cockpit/tasks
shell: |
export INSTANCES={{ instances | default(4) }}
Expand Down
2 changes: 2 additions & 0 deletions tasks/install-service
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -65,7 +65,9 @@ ExecStart=/usr/bin/podman run --name=cockpit-tasks-%i --hostname=${CONTAINER_HOS
--volume=\${TEST_SECRETS}/webhook:/run/secrets/webhook:ro \
--volume=${IMAGE_STORES}:/config/image-stores:ro \
--volume=/etc/npmrc:/etc/npmrc:ro \
--volume=/etc/job-runner.toml:/config/job-runner.toml:ro \
--volume=%t/podman/podman.sock:/podman.sock:rw \
--env=JOB_RUNNER_CONFIG=/config/job-runner.toml \
--env=COCKPIT_GITHUB_TOKEN_FILE=/run/secrets/webhook/.config--github-token \
--env=COCKPIT_S3_KEY_DIR=/run/secrets/tasks/s3-keys \
--env=COCKPIT_IMAGE_STORES_FILE=/config/image-stores \
Expand Down
52 changes: 52 additions & 0 deletions tasks/run-local.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,6 +108,54 @@ EOF
fi
}

create_job_runner_config() {
# we never want to push to real GitHub branches in this test
run_args="'--security-opt=label=disable', '--volume=$MYDIR/mock-git-push:/usr/local/bin/git:ro'"

if [ "$1" = "mock" ]; then
forge_opts="api-url = '$GHAPI_URL_POD'"
# needs to run in pod network so that it can access GITHUB_API_POD
run_args="${run_args}, '--pod=cockpituous'"
run_args="${run_args}, '--env=GITHUB_API=$GHAPI_URL_POD', '--env=COCKPIT_IMAGE_UPLOAD_STORE=$S3_URL_POD/images/'"
elif [ "$1" = "real" ]; then
forge_opts=""
else
echo "ERROR: unknown job-runner config $1" >&2
exit 1
fi

cat <<EOF > $SECRETS/tasks/job-runner.toml
[logs]
driver='s3'
[forge.github]
token = [{file="/run/secrets/webhook/.config--github-token"}]
$forge_opts
[logs.s3]
url = '$S3_URL_POD/logs'
ca = [{file='/run/secrets/webhook/ca.pem'}]
key = [{file="/run/secrets/tasks/s3-keys/localhost.localdomain"}]
[container]
command = ['podman-remote', '--url=unix:///podman.sock']
run-args = [$run_args]
[container.secrets]
# these are *host* paths, this is podman-remote
image-upload=[
'--volume=$SECRETS/tasks/s3-keys:/run/secrets/s3-keys:ro',
'--env=COCKPIT_S3_KEY_DIR=/run/secrets/s3-keys',
'--volume=$SECRETS/webhook/ca.pem:/run/secrets/ca.pem:ro',
'--env=COCKPIT_CA_PEM=/run/secrets/ca.pem',
]
github-token=[
'--volume=$SECRETS/webhook/.config--github-token:/run/secrets/github-token:ro',
'--env=COCKPIT_GITHUB_TOKEN_FILE=/run/secrets/github-token',
]
EOF
}

launch_containers() {
cleanup() {
if [ $? -ne 0 ] && [ -z "$INTERACTIVE" ] && [ -t 0 ]; then
Expand Down Expand Up @@ -189,6 +237,7 @@ EOF
--env=COCKPIT_BOTS_REPO=${COCKPIT_BOTS_REPO:-} \
--env=COCKPIT_BOTS_BRANCH=${COCKPIT_BOTS_BRANCH:-} \
--env=COCKPIT_TESTMAP_INJECT=main/unit-tests \
--env=JOB_RUNNER_CONFIG=/run/secrets/tasks/job-runner.toml \
--env=AMQP_SERVER=$AMQP_POD \
--env=S3_LOGS_URL=$S3_URL_POD/logs/ \
--env=COCKPIT_S3_KEY_DIR=/run/secrets/tasks/s3-keys \
Expand Down Expand Up @@ -251,6 +300,7 @@ test_image() {

test_mock_pr() {
podman cp "$MYDIR/mock-github" cockpituous-tasks:/work/bots/mock-github
create_job_runner_config mock
podman exec -i cockpituous-tasks sh -euxc "
cd bots
# test mock PR against our checkout, so that cloning will work
Expand Down Expand Up @@ -292,6 +342,7 @@ test_mock_pr() {
test_pr() {
# need to use real GitHub token for this
[ -z "$TOKEN" ] || cp -fv "$TOKEN" "$SECRETS"/webhook/.config--github-token
create_job_runner_config real

# run the main loop in the background; we could do this with a single run-queue invocation,
# but we want to test the cockpit-tasks script
Expand Down Expand Up @@ -347,6 +398,7 @@ test_pr() {
test_mock_image_refresh() {
podman cp "$MYDIR/mock-github" cockpituous-tasks:/work/bots/mock-github
podman cp "$MYDIR/mock-git-push" cockpituous-tasks:/usr/local/bin/git
create_job_runner_config mock

podman exec -i cockpituous-tasks sh -euxc "
cd bots
Expand Down