switch to new queue runner

[zowoq]

The builders connect to the runner but haven't built anything yet, seems to be some breakage in the UI.

Builds work.

#!/usr/bin/env bash
# https://github.com/NixOS/infra/blob/5d020952f5b870ff323904035347cd768530617d/non-critical-infra/hosts/staging-hydra/genca.sh
set -x

hosts="build03 build04"

O="Nix Community Infra"

newDir="$(date '+%Y-%m-%dT%H:%M')"
mkdir "${newDir}"
cd "${newDir}" || exit

openssl genpkey -algorithm Ed25519 -out ca.key
openssl req -x509 -new -nodes -key ca.key -sha256 -days 18250 -out ca.crt \
  -subj "/O=${O}/CN=hydra-queue-runner-ca"

cat <<EOF >server.cnf
[req]
prompt             = no
x509_extensions    = v3_req
req_extensions     = v3_req
default_md         = sha256
distinguished_name = req_distinguished_name

[req_distinguished_name]
O  = ${O}
CN = queue-runner.hydra.nix-community.org

[v3_req]
basicConstraints = CA:FALSE
keyUsage         = nonRepudiation, digitalSignature, keyEncipherment, keyAgreement
extendedKeyUsage = critical, serverAuth
subjectAltName   = @alt_names

[alt_names]
DNS.1 = queue-runner.hydra.nix-community.org
EOF

openssl genpkey -algorithm Ed25519 -out server.key
openssl req -new -key server.key -out server.csr -config server.cnf
openssl x509 -req -in server.csr -CA ca.crt -CAkey ca.key -CAcreateserial -out server.crt -days 18250 -sha256 -extfile server.cnf -extensions v3_req

for host in ${hosts}; do
  openssl genpkey -algorithm Ed25519 -out "client-${host}.key"
  openssl req -new -key "client-${host}.key" -out "client-${host}.csr" \
    -subj "/O=${O}/CN=hydra-queue-builder-${host}"
  openssl x509 -req -in "client-${host}.csr" -CA ca.crt -CAkey ca.key -CAcreateserial -out "client-${host}.crt" -days 18250 -sha256
done

rm -rf -- *.csr *.srl
rm server.cnf

cd - || exit

[Mic92]

Please let us know what issues you find. cc @Conni2461 @dasJ

[Conni2461]

that was not an issue with the new queue runner but, hydra-www user just was not an allowed user, so was not allowed to actually talk with the nix daemon, that caused the failure above, works now. Will push the changes in a sec to NixOS/infra#790

If you have any questions or feedback, we can also have a call in the next couple of days :)

Edit: following commit NixOS/infra@222c51c

[zowoq]

error: cannot open connection to remote store 'daemon': error: reading from file: Connection reset by peer at /nix/store/fw80w6pjsq5hn8nqx61pf832almq2hhs-hydra-0.1.20250805.79ba8fd/bin/.hydra-update-gc-roots-wrapped line 57.

Fixed by adding hydra to nix allowed-users.

The nix allowed-users default is "*", not having that breaks other stuff.

---

https://hydra.nix-community.org/queue-runner-status
"status" : "down"

https://staging-hydra.nixos.org/queue-runner-status works.

[Conni2461]

There is an updated version which now uses ffi (for some parts): https://github.com/NixOS/infra/compare/222c51c8e346216a0b8cc9df23af6633a99c3cc0..e21d6f595545327913f6664f4b1afcc4d5036e3f

[Mic92]

@zowoq NixOS/hydra#1510 we fixed the xss issues you are showing in the screenshots.

[zowoq]

Might vendor the queue runner modules and packages here.

[Conni2461]

yeah, i tested it with emacs-overlay/master and it worked fine, try settings hydra-queue-runner-v2.settings.queueTriggerTimerInS = 300; this ensures that the queue check run is triggered every 300s. The queue check most likely early exited for you. Early exists exists so we can act on changes like bump to font or new builds but if none were made, no event was sent and the check doesnt trigger. So setting a interval should ensure that the open builds are all processed and scheduled

[zowoq]

This has been working well for the last couple of days so I'll merge and keep it enabled.

Only outstanding issue is the status page not working but not really a problem as everything else seems to be working.

https://hydra.nix-community.org/queue-runner-status
"status" : "down"

[Conni2461]

yeah i know that this is an issue, i dont 100% know how the hydra actually gets the data but if you run sudo -u hydra-queue-runner -g hydra /nix/store/$QUEUE_RUNNER_PATH-hydra-queue-runner/bin/queue-runner --config-path /etc/hydra/queue-runner.toml --status once it works. We will most likely push a fix to the hydra-www (perl codebase), like we did for the machines page, and that endpoint will then just query localhost:8080/status because we now have an endpoint that exposes the data. we no longer need to wait for a notification in some pg queue and then write that results to the database 😅

[dasJ]

https://github.com/NixOS/hydra/blob/master/src/lib/Hydra/Controller/Root.pm#L182