diff --git a/.github/ISSUE_TEMPLATE/create_release_branch.md b/.github/ISSUE_TEMPLATE/create_release_branch.md index 9c5a8268a..c460a90ed 100644 --- a/.github/ISSUE_TEMPLATE/create_release_branch.md +++ b/.github/ISSUE_TEMPLATE/create_release_branch.md @@ -13,16 +13,16 @@ Make sure to follow the steps below and ensure all actions are completed and sig - **K8s version**: 1.xx -- **Owner**: +- **Owner**: `who plans to do the work` -- **Reviewer**: +- **Reviewer**: `who plans to review the work` -- **PR**: -- +- **PR**: https://github.com/canonical/k8s-snap/pull/`` + -- **PR**: +- **PR**: https://github.com/canonical/k8s-snap/pull/`` #### Actions @@ -53,7 +53,7 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Owner**: Create `release-1.xx` branch from latest `master` in k8s-dqlite - `git clone git@github.com:canonical/k8s-dqlite.git ~/tmp/release-1.xx` - `pushd ~/tmp/release-1.xx` - - `git switch main` + - `git switch master` - `git pull` - `git checkout -b release-1.xx` - `git push origin release-1.xx` @@ -89,7 +89,7 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Owner**: Create `release-1.xx` branch from latest `main` in rawfile-localpv - `git clone git@github.com:canonical/rawfile-localpv.git ~/tmp/release-1.xx` - `pushd ~/tmp/release-1.xx` - - `git switch main` + - `git switch rockcraft` - `git pull` - `git checkout -b release-1.xx` - `git push origin release-1.xx` @@ -98,7 +98,6 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] **Reviewer**: Ensure `release-1.xx` branch is based on latest changes on `main` at the time of the release cut. - [ ] **Owner**: Create PR to initialize `release-1.xx` branch: - [ ] Update `KUBERNETES_RELEASE_MARKER` to `stable-1.xx` in [/build-scripts/hack/update-component-versions.py][] - - [ ] Update `master` to `release-1.xx` in [/build-scripts/components/k8s-dqlite/version][] - [ ] Update `"main"` to `"release-1.xx"` in [/build-scripts/hack/generate-sbom.py][] - [ ] `git commit -m 'Release 1.xx'` - [ ] Create PR against `release-1.xx` with the changes and request review from **Reviewer**. Make sure to update the issue `Information` section with a link to the PR. @@ -107,43 +106,22 @@ The steps are to be followed in-order, each task must be completed by the person - [ ] Add `release-1.xx` in [.github/workflows/update-components.yaml][] - [ ] Remove unsupported releases from the list (if applicable, consult with **Reviewer**) - [ ] Create PR against `main` with the changes and request review from **Reviewer**. Make sure to update the issue information with a link to the PR. -- [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-strict` branch is created. -- [ ] **Owner**: Create launchpad builders for `release-1.xx` - - [ ] Go to [lp:k8s][] and do **Import now** to pick up all latest changes. - - [ ] Under **Branches**, select `release-1.xx`, then **Create snap package** - - [ ] Set **Snap recipe name** to `k8s-snap-1.xx` - - [ ] Set **Owner** to `Canonical Kubernetes (containers)` - - [ ] Set **The project that this Snap is associated with** to `k8s` - - [ ] Set **Series** to Infer from snapcraft.yaml - - [ ] Set **Processors** to `AMD x86-64 (amd64)` and `ARM ARMv8 (arm64)` - - [ ] Enable **Automatically build when branch changes** - - [ ] Enable **Automatically upload to store** - - [ ] Set **Registered store name** to `k8s` - - [ ] In **Store Channels**, set **Track** to `1.xx-classic` and **Risk** to `edge`. Leave **Branch** empty - - [ ] Click **Create snap package** at the bottom of the page. -- [ ] **Owner**: Create launchpad builders for `release-1.xx-strict` - - [ ] Return to [lp:k8s][]. - - [ ] Under **Branches**, select `autoupdate/release-1.xx-strict`, then **Create snap package** - - [ ] Set **Snap recipe name** to `k8s-snap-1.xx-strict` - - [ ] Set **Owner** to `Canonical Kubernetes (containers)` - - [ ] Set **The project that this Snap is associated with** to `k8s` - - [ ] Set **Series** to Infer from snapcraft.yaml - - [ ] Set **Processors** to `AMD x86-64 (amd64)` and `ARM ARMv8 (arm64)` - - [ ] Enable **Automatically build when branch changes** - - [ ] Enable **Automatically upload to store** - - [ ] Set **Registered store name** to `k8s` - - [ ] In **Store Channels**, set **Track** to `1.xx` and **Risk** to `edge`. Leave **Branch** empty - - [ ] Click **Create snap package** at the bottom of the page. +- [ ] **Reviewer**: On merge, confirm [Auto-update strict branch] action runs to completion and that the `autoupdate/release-1.xx-*` flavor branches are created. + - [ ] autoupdate/release-1.xx-strict + - [ ] autoupdate/release-1.xx-moonray +- [ ] **Owner**: Create launchpad builders for `release-1.xx` and flavors + - [ ] Run the [Confirm Snap Builds][] Action - [ ] **Reviewer**: Ensure snap recipes are created in [lp:k8s/+snaps][] - - look for `k8s-snap-1.xx` - - look for `k8s-snap-1.xx-strict` + - [ ] look for `k8s-snap-1.xx-classic` + - [ ] look for `k8s-snap-1.xx-strict` + - [ ] look for `k8s-snap-1.xx-moonray` + - [ ] make sure each is "Authorized for Store Upload" #### After release - [ ] **Owner** follows up with the **Reviewer** and team about things to improve around the process. - [ ] **Owner**: After a few weeks of stable CI, update default track to `1.xx/stable` via - On the snap [releases page][], select `Track` > `1.xx` -- [ ] **Reviewer**: Ensure snap recipes are created in [lp:k8s/+snaps][] @@ -161,6 +139,7 @@ The steps are to be followed in-order, each task must be completed by the person [.github/workflows/update-components.yaml]: ../workflows/update-components.yaml [/build-scripts/components/hack/update-component-versions.py]: ../../build-scripts/components/hack/update-component-versions.py [/build-scripts/components/k8s-dqlite/version]: ../../build-scripts/components/k8s-dqlite/version -[/build-scripts/hack/generate-sbom.py]: ../..//build-scripts/hack/generate-sbom.py +[/build-scripts/hack/generate-sbom.py]: ../../build-scripts/hack/generate-sbom.py [lp:k8s]: https://code.launchpad.net/~cdk8s/k8s/+git/k8s-snap [lp:k8s/+snaps]: https://launchpad.net/k8s/+snaps +[Confirm Snap Builds]: https://github.com/canonical/canonical-kubernetes-release-ci/actions/workflows/create-release-branch.yaml diff --git a/.github/workflows/auto-merge-successful-prs.yaml b/.github/workflows/auto-merge-successful-prs.yaml new file mode 100644 index 000000000..e7f4fc096 --- /dev/null +++ b/.github/workflows/auto-merge-successful-prs.yaml @@ -0,0 +1,29 @@ +name: Auto-merge Successful PRs + +on: + workflow_dispatch: + schedule: + - cron: "0 */4 * * *" # Every 4 hours + +permissions: + contents: read + +jobs: + merge-successful-prs: + runs-on: ubuntu-latest + + steps: + - name: Harden Runner + uses: step-security/harden-runner@v2 + with: + egress-policy: audit + - name: Checking out repo + uses: actions/checkout@v4 + - uses: actions/setup-python@v5 + with: + python-version: '3.12' + - name: Auto-merge pull requests if all status checks pass + env: + GH_TOKEN: ${{ secrets.BOT_TOKEN }} + run: | + build-scripts/hack/auto-merge-successful-pr.py diff --git a/.github/workflows/automatic-doc-checks.yml b/.github/workflows/automatic-doc-checks.yml index 5472b1662..d3bba4576 100644 --- a/.github/workflows/automatic-doc-checks.yml +++ b/.github/workflows/automatic-doc-checks.yml @@ -3,13 +3,15 @@ name: Core Documentation Checks on: - workflow_dispatch +permissions: + contents: read + concurrency: group: ${{ github.workflow }}-${{ github.ref }} cancel-in-progress: true jobs: documentation-checks: - uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main - with: - working-directory: 'docs/moonray' - + - uses: canonical/documentation-workflows/.github/workflows/documentation-checks.yaml@main + with: + working-directory: 'docs/moonray' diff --git a/.github/workflows/cron-jobs.yaml b/.github/workflows/cron-jobs.yaml index fc658da51..59a1227a1 100644 --- a/.github/workflows/cron-jobs.yaml +++ b/.github/workflows/cron-jobs.yaml @@ -6,7 +6,7 @@ on: permissions: contents: read - + jobs: TICS: permissions: @@ -27,6 +27,9 @@ jobs: uses: actions/checkout@v4 with: ref: ${{matrix.branch}} + - uses: actions/setup-python@v5 + with: + python-version: '3.12' - name: Install Go uses: actions/setup-go@v5 with: @@ -47,14 +50,14 @@ jobs: # TICS requires us to have the test results in cobertura xml format under the # directory use below - make go.unit + sudo make go.unit go install github.com/boumenot/gocover-cobertura@latest gocover-cobertura < coverage.txt > coverage.xml mkdir .coverage mv ./coverage.xml ./.coverage/ # Install the TICS and staticcheck - go install honnef.co/go/tools/cmd/staticcheck@v0.4.7 + go install honnef.co/go/tools/cmd/staticcheck@v0.5.1 . <(curl --silent --show-error 'https://canonical.tiobe.com/tiobeweb/TICS/api/public/v1/fapi/installtics/Script?cfg=default&platform=linux&url=https://canonical.tiobe.com/tiobeweb/TICS/') # We need to have our project built @@ -62,7 +65,7 @@ jobs: # will try to build parts of the project itself sudo add-apt-repository -y ppa:dqlite/dev sudo apt install dqlite-tools libdqlite-dev -y - make clean + sudo make clean go build -a ./... TICSQServer -project k8s-snap -tmpdir /tmp/tics -branchdir $HOME/work/k8s-snap/k8s-snap/ @@ -79,6 +82,8 @@ jobs: - { branch: main, channel: latest/edge } # Stable branches # Add branches to test here + - { branch: release-1.30, channel: 1.30-classic/edge } + - { branch: release-1.31, channel: 1.31-classic/edge } steps: - name: Harden Runner @@ -103,6 +108,8 @@ jobs: format: "sarif" output: "trivy-k8s-repo-scan--results.sarif" severity: "MEDIUM,HIGH,CRITICAL" + env: + TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db" - name: Gather Trivy repo scan results run: | cp trivy-k8s-repo-scan--results.sarif ./sarifs/ @@ -111,7 +118,10 @@ jobs: snap download k8s --channel ${{ matrix.channel }} mv ./k8s*.snap ./k8s.snap unsquashfs k8s.snap - ./trivy rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif + for var in $(env | grep -o '^TRIVY_[^=]*'); do + unset "$var" + done + ./trivy --db-repository public.ecr.aws/aquasecurity/trivy-db rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif - name: Get HEAD sha run: | SHA="$(git rev-parse HEAD)" diff --git a/.github/workflows/docs-spelling-checks.yml b/.github/workflows/docs-spelling-checks.yml new file mode 100644 index 000000000..913ab75a8 --- /dev/null +++ b/.github/workflows/docs-spelling-checks.yml @@ -0,0 +1,32 @@ +name: Documentation Spelling Check + +on: + workflow_dispatch: + # pull_request: + # paths: + # - 'docs/**' +permissions: + contents: read + +jobs: + spell-check: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - name: Install aspell + run: sudo apt-get install aspell aspell-en + - id: spell-check + name: Spell Check + run: make spelling + working-directory: docs/canonicalk8s + continue-on-error: true + # - if: ${{ github.event_name == 'pull_request' && steps.spell-check.outcome == 'failure' }} + # uses: actions/github-script@v6 + # with: + # script: | + # github.rest.issues.createComment({ + # issue_number: context.issue.number, + # owner: context.repo.owner, + # repo: context.repo.repo, + # body: 'Hi, looks like pyspelling job found some issues, you can check it [here](${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_id }})' + # }) diff --git a/.github/workflows/go.yaml b/.github/workflows/go.yaml index c29585c09..56be690a9 100644 --- a/.github/workflows/go.yaml +++ b/.github/workflows/go.yaml @@ -2,6 +2,8 @@ name: Go on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -19,6 +23,7 @@ jobs: permissions: contents: read # for actions/checkout to fetch code pull-requests: write # for marocchino/sticky-pull-request-comment to create or update PR comment + checks: write # for golangci/golangci-lint-action to checks to allow the action to annotate code in the PR. name: Unit Tests & Code Quality runs-on: ubuntu-latest @@ -67,6 +72,19 @@ jobs: # root ownership so the tests must be run as root: run: sudo make go.unit + - name: dqlite-for-golangci-lint + working-directory: src/k8s + run: | + sudo add-apt-repository ppa:dqlite/dev + sudo apt update + sudo apt install dqlite-tools libdqlite-dev + + - name: golangci-lint + uses: golangci/golangci-lint-action@v6 + with: + version: v1.61 + working-directory: src/k8s + test-binary: name: Binaries runs-on: ubuntu-latest diff --git a/.github/workflows/integration-informing.yaml b/.github/workflows/integration-informing.yaml index 708094078..9ade424d0 100644 --- a/.github/workflows/integration-informing.yaml +++ b/.github/workflows/integration-informing.yaml @@ -2,11 +2,15 @@ name: Informing Integration Tests on: push: + paths-ignore: + - 'docs/**' branches: - main - 'release-[0-9]+.[0-9]+' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -17,7 +21,7 @@ jobs: runs-on: ubuntu-20.04 strategy: matrix: - patch: ["strict", "moonray"] + patch: ["moonray"] fail-fast: false steps: - name: Harden Runner @@ -54,16 +58,16 @@ jobs: strategy: matrix: os: ["ubuntu:20.04"] - patch: ["strict", "moonray"] + patch: ["moonray"] fail-fast: false - runs-on: ubuntu-20.04 + runs-on: ["self-hosted", "Linux", "AMD64", "jammy", "large"] steps: - name: Check out code uses: actions/checkout@v4 - name: Setup Python uses: actions/setup-python@v5 with: - python-version: '3.8' + python-version: '3.10' - name: Install tox run: pip install tox - name: Install lxd @@ -72,29 +76,33 @@ jobs: sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' + sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT + sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - name: Download snap uses: actions/download-artifact@v4 with: name: k8s-${{ matrix.patch }}.snap - path: build + path: ${{ github.workspace }}/build - name: Apply ${{ matrix.patch }} patch run: | ./build-scripts/patches/${{ matrix.patch }}/apply - name: Run end to end tests + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s-${{ matrix.patch }}.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_FLAVOR: ${{ matrix.patch }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports run: | - export TEST_SNAP="$PWD/build/k8s-${{ matrix.patch }}.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE=${{ matrix.os }} - export TEST_INSPECTION_REPORTS_DIR="$HOME/inspection-reports" cd tests/integration && sg lxd -c 'tox -e integration' - name: Prepare inspection reports if: failure() run: | - tar -czvf inspection-reports.tar.gz -C $HOME inspection-reports + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports echo "artifact_name=inspection-reports-${{ matrix.os }}-${{ matrix.patch }}" | sed 's/:/-/g' >> $GITHUB_ENV - name: Upload inspection report artifact if: failure() uses: actions/upload-artifact@v4 with: name: ${{ env.artifact_name }} - path: inspection-reports.tar.gz + path: ${{ github.workspace }}/inspection-reports.tar.gz diff --git a/.github/workflows/integration.yaml b/.github/workflows/integration.yaml index 6a58c1194..2aefbb939 100644 --- a/.github/workflows/integration.yaml +++ b/.github/workflows/integration.yaml @@ -2,6 +2,8 @@ name: Integration Tests on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read @@ -51,6 +55,8 @@ jobs: steps: - name: Check out code uses: actions/checkout@v4 + with: + fetch-depth: 0 - name: Setup Python uses: actions/setup-python@v5 with: @@ -59,7 +65,7 @@ jobs: run: pip install tox - name: Run branch_management tests run: | - tox -c tests/branch_management -e integration + tox -c tests/branch_management -e test test-integration: name: Test ${{ matrix.os }} @@ -67,7 +73,7 @@ jobs: fail-fast: false matrix: os: ["ubuntu:20.04", "ubuntu:22.04", "ubuntu:24.04"] - runs-on: ubuntu-20.04 + runs-on: ["self-hosted", "Linux", "AMD64", "jammy", "large"] needs: build steps: @@ -76,7 +82,7 @@ jobs: - name: Setup Python uses: actions/setup-python@v5 with: - python-version: '3.8' + python-version: '3.10' - name: Install tox run: pip install tox - name: Install lxd @@ -85,29 +91,38 @@ jobs: sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' + sudo iptables -I DOCKER-USER -i lxdbr0 -j ACCEPT + sudo iptables -I DOCKER-USER -o lxdbr0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT - name: Download snap uses: actions/download-artifact@v4 with: name: k8s.snap - path: build + path: ${{ github.workspace }}/build - name: Run end to end tests + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports + # Test the latest (up to) 6 releases for the flavour + # TODO(ben): upgrade nightly to run all flavours + TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic" + # Upgrading from 1.30 is not supported. + TEST_VERSION_UPGRADE_MIN_RELEASE: "1.31" + TEST_MIRROR_LIST: '[{"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}"}, {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io", "username": "", "password": ""}]' run: | - export TEST_SNAP="$PWD/build/k8s.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE=${{ matrix.os }} - export TEST_INSPECTION_REPORTS_DIR="$HOME/inspection-reports" - cd tests/integration && sg lxd -c 'tox -e integration -- -k test_control_plane_nodes' + cd tests/integration && sg lxd -c 'tox -e integration' - name: Prepare inspection reports if: failure() run: | - tar -czvf inspection-reports.tar.gz -C $HOME inspection-reports + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports echo "artifact_name=inspection-reports-${{ matrix.os }}" | sed 's/:/-/g' >> $GITHUB_ENV - name: Upload inspection report artifact if: failure() uses: actions/upload-artifact@v4 with: name: ${{ env.artifact_name }} - path: inspection-reports.tar.gz + path: ${{ github.workspace }}/inspection-reports.tar.gz security-scan: permissions: @@ -121,6 +136,13 @@ jobs: uses: step-security/harden-runner@v2 with: egress-policy: audit + - name: Login to GitHub Container Registry + uses: docker/login-action@v3 + with: + # We run into rate limiting issues if we don't authenticate + registry: ghcr.io + username: ${{ github.actor }} + password: ${{ secrets.GITHUB_TOKEN }} - name: Checking out repo uses: actions/checkout@v4 - name: Fetch snap @@ -130,10 +152,12 @@ jobs: path: build - name: Setup Trivy vulnerability scanner run: | - mkdir -p sarifs + mkdir -p manual-trivy/sarifs + pushd manual-trivy VER=$(curl --silent -qI https://github.com/aquasecurity/trivy/releases/latest | awk -F '/' '/^location/ {print substr($NF, 1, length($NF)-1)}'); wget https://github.com/aquasecurity/trivy/releases/download/${VER}/trivy_${VER#v}_Linux-64bit.tar.gz tar -zxvf ./trivy_${VER#v}_Linux-64bit.tar.gz + popd - name: Run Trivy vulnerability scanner in repo mode uses: aquasecurity/trivy-action@master with: @@ -142,15 +166,20 @@ jobs: format: "sarif" output: "trivy-k8s-repo-scan--results.sarif" severity: "MEDIUM,HIGH,CRITICAL" + env: + TRIVY_DB_REPOSITORY: "public.ecr.aws/aquasecurity/trivy-db" - name: Gather Trivy repo scan results run: | - cp trivy-k8s-repo-scan--results.sarif ./sarifs/ + cp trivy-k8s-repo-scan--results.sarif ./manual-trivy/sarifs/ - name: Run Trivy vulnerability scanner on the snap run: | + for var in $(env | grep -o '^TRIVY_[^=]*'); do + unset "$var" + done cp build/k8s.snap . unsquashfs k8s.snap - ./trivy rootfs ./squashfs-root/ --format sarif > sarifs/snap.sarif + ./manual-trivy/trivy --db-repository public.ecr.aws/aquasecurity/trivy-db rootfs ./squashfs-root/ --format sarif > ./manual-trivy/sarifs/snap.sarif - name: Upload Trivy scan results to GitHub Security tab uses: github/codeql-action/upload-sarif@v3 with: - sarif_file: "sarifs" + sarif_file: "./manual-trivy/sarifs" diff --git a/.github/workflows/nightly-test.yaml b/.github/workflows/nightly-test.yaml index 915d6c3f3..3e7fd8972 100644 --- a/.github/workflows/nightly-test.yaml +++ b/.github/workflows/nightly-test.yaml @@ -2,49 +2,68 @@ name: Nightly Latest/Edge Tests on: schedule: - - cron: '0 0 * * *' # Runs every midnight + - cron: "0 0 * * *" # Runs every midnight permissions: contents: read jobs: test-integration: - name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.releases }} + name: Integration Test ${{ matrix.os }} ${{ matrix.arch }} ${{ matrix.release }} strategy: matrix: os: ["ubuntu:20.04", "ubuntu:22.04", "ubuntu:24.04"] arch: ["amd64", "arm64"] - releases: ["latest/edge"] + release: ["latest/edge"] fail-fast: false # TODO: remove once arm64 works - runs-on: ${{ matrix.arch == 'arm64' && 'Ubuntu_ARM64_4C_16G_01' || 'ubuntu-20.04' }} + runs-on: ${{ matrix.arch == 'arm64' && ["self-hosted", "Linux", "ARM64", "jammy", "large"] || ["self-hosted", "Linux", "AMD64", "jammy", "large"] }} steps: - name: Checking out repo uses: actions/checkout@v4 - - name: Setup Python + - name: Install lxd and tox run: | sudo apt update - sudo apt install -y python3 python3-pip - - name: Install tox - run: | - pip3 install tox==4.13 - - name: Install lxd - run: | - sudo snap refresh lxd --channel 5.19/stable + sudo apt install -y tox + sudo snap refresh lxd --channel 5.21/stable sudo lxd init --auto sudo usermod --append --groups lxd $USER sg lxd -c 'lxc version' - name: Create build directory run: mkdir -p build - - name: Install $${ matrix.releases }} k8s snap + - name: Install ${{ matrix.release }} k8s snap run: | cd build - snap download k8s --channel=${{ matrix.releases }} --basename k8s + snap download k8s --channel=${{ matrix.release }} --basename k8s - name: Run end to end tests # tox path needs to be specified for arm64 + env: + TEST_SNAP: ${{ github.workspace }}/build/k8s.snap + TEST_SUBSTRATE: lxd + TEST_LXD_IMAGE: ${{ matrix.os }} + TEST_INSPECTION_REPORTS_DIR: ${{ github.workspace }}/inspection-reports + # Test the latest (up to) 6 releases for the flavour + # TODO(ben): upgrade nightly to run all flavours + TEST_VERSION_UPGRADE_CHANNELS: "recent 6 classic" + # Upgrading from 1.30 is not supported. + TEST_VERSION_UPGRADE_MIN_RELEASE: "1.31" + TEST_STRICT_INTERFACE_CHANNELS: "recent 6 strict" + TEST_MIRROR_LIST: '[{"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io", "username": "${{ github.actor }}", "password": "${{ secrets.GITHUB_TOKEN }}"}, {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io", "username": "", "password": ""}]' run: | - export TEST_SNAP="$PWD/build/k8s.snap" - export TEST_SUBSTRATE=lxd - export TEST_LXD_IMAGE="${{ matrix.os }}" export PATH="/home/runner/.local/bin:$PATH" - cd tests/integration && sg lxd -c 'tox -e integration' + cd tests/integration && sg lxd -c 'tox -vve integration' + - name: Prepare inspection reports + if: failure() + run: | + tar -czvf inspection-reports.tar.gz -C ${{ github.workspace }} inspection-reports + echo "artifact_name=inspection-reports-${{ matrix.os }}-${{ matrix.arch }}" | sed 's/:/-/g' >> $GITHUB_ENV + - name: Upload inspection report artifact + if: failure() + uses: actions/upload-artifact@v4 + with: + name: ${{ env.artifact_name }} + path: ${{ github.workspace }}/inspection-reports.tar.gz + - name: Tmate debugging session + if: ${{ failure() && github.event_name == 'pull_request' }} + uses: mxschmitt/action-tmate@v3 + timeout-minutes: 10 diff --git a/.github/workflows/python.yaml b/.github/workflows/python.yaml index 2ccb0979f..0c51d8ecf 100644 --- a/.github/workflows/python.yaml +++ b/.github/workflows/python.yaml @@ -2,6 +2,8 @@ name: Python on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read diff --git a/.github/workflows/sbom.yaml b/.github/workflows/sbom.yaml index 846a19e76..2faa27a28 100644 --- a/.github/workflows/sbom.yaml +++ b/.github/workflows/sbom.yaml @@ -2,6 +2,8 @@ name: SBOM on: push: + paths-ignore: + - 'docs/**' branches: - main - autoupdate/strict @@ -10,6 +12,8 @@ on: - 'autoupdate/release-[0-9]+.[0-9]+-strict' - 'autoupdate/sync/**' pull_request: + paths-ignore: + - 'docs/**' permissions: contents: read diff --git a/.github/workflows/sync-images.yaml b/.github/workflows/sync-images.yaml deleted file mode 100644 index 02b143a1b..000000000 --- a/.github/workflows/sync-images.yaml +++ /dev/null @@ -1,23 +0,0 @@ -name: Sync upstream images to ghcr.io - -on: - workflow_dispatch: - -jobs: - publish: - runs-on: ubuntu-latest - steps: - - name: Checkout repository - uses: actions/checkout@v4 - - - name: Install Go - uses: actions/setup-go@v5 - with: - go-version: "1.22" - - - name: Sync images - env: - USERNAME: ${{ github.actor }} - PASSWORD: ${{ secrets.GITHUB_TOKEN }} - run: | - ./build-scripts/hack/sync-images.sh diff --git a/.github/workflows/update-branches.yaml b/.github/workflows/update-branches.yaml index 356bbce5b..b6ed8f38e 100644 --- a/.github/workflows/update-branches.yaml +++ b/.github/workflows/update-branches.yaml @@ -41,7 +41,7 @@ jobs: - name: Sync ${{ github.ref }} to ${{ steps.determine.outputs.branch }} uses: actions/checkout@v4 with: - ssh-key: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} + ssh-key: ${{ secrets.BOT_SSH_KEY }} - name: Apply ${{ matrix.patch }} patch run: | git checkout -b ${{ steps.determine.outputs.branch }} diff --git a/.github/workflows/update-components.yaml b/.github/workflows/update-components.yaml index e46bd55df..23aa952a4 100644 --- a/.github/workflows/update-components.yaml +++ b/.github/workflows/update-components.yaml @@ -21,6 +21,7 @@ jobs: # Keep main branch up to date - main # Supported stable release branches + - release-1.31 - release-1.30 steps: @@ -32,7 +33,7 @@ jobs: uses: actions/checkout@v4 with: ref: ${{ matrix.branch }} - ssh-key: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} + ssh-key: ${{ secrets.BOT_SSH_KEY }} - name: Setup Python uses: actions/setup-python@v5 @@ -50,10 +51,11 @@ jobs: - name: Create pull request uses: peter-evans/create-pull-request@v6 with: - git-token: ${{ secrets.DEPLOY_KEY_TO_UPDATE_STRICT_BRANCH }} commit-message: "[${{ matrix.branch }}] Update component versions" title: "[${{ matrix.branch }}] Update component versions" body: "[${{ matrix.branch }}] Update component versions" branch: "autoupdate/sync/${{ matrix.branch }}" + labels: | + automerge delete-branch: true base: ${{ matrix.branch }} diff --git a/.gitignore b/.gitignore index 8c7172b67..896206d59 100644 --- a/.gitignore +++ b/.gitignore @@ -21,3 +21,5 @@ k8s_*.txt /docs/tools/.sphinx/.wordlist.dic /docs/tools/.sphinx/.doctrees/ /docs/tools/.sphinx/node_modules +/docs/tools/.sphinx/styles/* +/docs/tools/.sphinx/vale.ini diff --git a/.golangci.yml b/.golangci.yml new file mode 100644 index 000000000..ece0c60a6 --- /dev/null +++ b/.golangci.yml @@ -0,0 +1,328 @@ +# This code is licensed under the terms of the MIT license https://opensource.org/license/mit +# Copyright (c) 2021 Marat Reymers + +## Golden config for golangci-lint v1.61.0 +# +# This is the best config for golangci-lint based on my experience and opinion. +# It is very strict, but not extremely strict. +# Feel free to adapt and change it for your needs. +linters: + disable-all: true + enable: + - gosimple + - asasalint + - asciicheck + - bidichk + - bodyclose + - canonicalheader + - copyloopvar + - decorder + - dogsled + - dupword + - durationcheck + # - err113 - TODO(ben): maybe consider later + # - errcheck + # - errchkjson + - errname + - errorlint + - exhaustive + - fatcontext + - forbidigo + - forcetypeassert + # - funlen - TODO(ben): maybe consider later; needs some refactoring + - gci + - ginkgolinter + - gocheckcompilerdirectives + # - gochecknoglobals - TODO(ben): We have some global vars, maybe refactor later as this sounds like a useful linter + - gochecksumtype + # - goconst - TODO(ben): Consider moving consts into separate package + - gocritic + - godot + - gofmt + - gofumpt + + +# TODO(ben): Enable those linters step by step and fix existing issues. +# - goheader +# - goimports +# - gomoddirectives +# - gomodguard +# - goprintffuncname +# - gosec +# - gosmopolitan +# - govet +# - grouper +# - importas +# - inamedparam +# - ineffassign +# - interfacebloat +# - intrange +# - ireturn +# - lll +# - loggercheck +# - maintidx +# - makezero +# - mirror +# - misspell +# - mnd +# - musttag +# - nakedret +# - nestif +# - nilerr +# - nilnil +# - nlreturn +# - noctx +# - nolintlint +# - nonamedreturns +# - nosprintfhostport +# - paralleltest +# - perfsprint +# - prealloc +# - predeclared +# - promlinter +# - protogetter +# - reassign +# - revive +# - rowserrcheck +# - sloglint +# - spancheck +# - sqlclosecheck +# - staticcheck +# - stylecheck +# - tagalign +# - tagliatelle +# - tenv +# - testableexamples +# - testifylint +# - testpackage +# - thelper +# - tparallel +# - unconvert +# - unparam +# - unused +# - usestdlibvars +# - varnamelen +# - wastedassign +# - whitespace +# - wrapcheck +# - wsl +# - zerologlint + +run: + # Timeout for analysis, e.g. 30s, 5m. + # Default: 1m + timeout: 5m + +# This file contains only configs which differ from defaults. +# All possible options can be found here https://github.com/golangci/golangci-lint/blob/master/.golangci.reference.yml +linters-settings: + cyclop: + # The maximal code complexity to report. + # Default: 10 + max-complexity: 30 + # The maximal average package complexity. + # If it's higher than 0.0 (float) the check is enabled + # Default: 0.0 + package-average: 10.0 + + errcheck: + # Report about not checking of errors in type assertions: `a := b.(MyStruct)`. + # Such cases aren't reported by default. + # Default: false + check-type-assertions: true + + exhaustive: + # Program elements to check for exhaustiveness. + # Default: [ switch ] + check: + - switch + - map + + exhaustruct: + # List of regular expressions to exclude struct packages and their names from checks. + # Regular expressions must match complete canonical struct package/name/structname. + # Default: [] + exclude: + # std libs + - "^net/http.Client$" + - "^net/http.Cookie$" + - "^net/http.Request$" + - "^net/http.Response$" + - "^net/http.Server$" + - "^net/http.Transport$" + - "^net/url.URL$" + - "^os/exec.Cmd$" + - "^reflect.StructField$" + # public libs + - "^github.com/Shopify/sarama.Config$" + - "^github.com/Shopify/sarama.ProducerMessage$" + - "^github.com/mitchellh/mapstructure.DecoderConfig$" + - "^github.com/prometheus/client_golang/.+Opts$" + - "^github.com/spf13/cobra.Command$" + - "^github.com/spf13/cobra.CompletionOptions$" + - "^github.com/stretchr/testify/mock.Mock$" + - "^github.com/testcontainers/testcontainers-go.+Request$" + - "^github.com/testcontainers/testcontainers-go.FromDockerfile$" + - "^golang.org/x/tools/go/analysis.Analyzer$" + - "^google.golang.org/protobuf/.+Options$" + - "^gopkg.in/yaml.v3.Node$" + + funlen: + # Checks the number of lines in a function. + # If lower than 0, disable the check. + # Default: 60 + lines: 100 + # Checks the number of statements in a function. + # If lower than 0, disable the check. + # Default: 40 + statements: 50 + # Ignore comments when counting lines. + # Default false + ignore-comments: true + + gocognit: + # Minimal code complexity to report. + # Default: 30 (but we recommend 10-20) + min-complexity: 20 + + gocritic: + # Settings passed to gocritic. + # The settings key is the name of a supported gocritic checker. + # The list of supported checkers can be find in https://go-critic.github.io/overview. + settings: + captLocal: + # Whether to restrict checker to params only. + # Default: true + paramsOnly: false + underef: + # Whether to skip (*x).method() calls where x is a pointer receiver. + # Default: true + skipRecvDeref: false + + gomodguard: + blocked: + # List of blocked modules. + # Default: [] + modules: + - github.com/golang/protobuf: + recommendations: + - google.golang.org/protobuf + reason: "see https://developers.google.com/protocol-buffers/docs/reference/go/faq#modules" + - github.com/satori/go.uuid: + recommendations: + - github.com/google/uuid + reason: "satori's package is not maintained" + - github.com/gofrs/uuid: + recommendations: + - github.com/gofrs/uuid/v5 + reason: "gofrs' package was not go module before v5" + + govet: + # Enable all analyzers. + # Default: false + enable-all: true + # Disable analyzers by name. + # Run `go tool vet help` to see all analyzers. + # Default: [] + disable: + - fieldalignment # too strict + # Settings per analyzer. + settings: + shadow: + # Whether to be strict about shadowing; can be noisy. + # Default: false + strict: true + + inamedparam: + # Skips check for interface methods with only a single parameter. + # Default: false + skip-single-param: true + + mnd: + # List of function patterns to exclude from analysis. + # Values always ignored: `time.Date`, + # `strconv.FormatInt`, `strconv.FormatUint`, `strconv.FormatFloat`, + # `strconv.ParseInt`, `strconv.ParseUint`, `strconv.ParseFloat`. + # Default: [] + ignored-functions: + - args.Error + - flag.Arg + - flag.Duration.* + - flag.Float.* + - flag.Int.* + - flag.Uint.* + - os.Chmod + - os.Mkdir.* + - os.OpenFile + - os.WriteFile + - prometheus.ExponentialBuckets.* + - prometheus.LinearBuckets + + nakedret: + # Make an issue if func has more lines of code than this setting, and it has naked returns. + # Default: 30 + max-func-lines: 0 + + nolintlint: + # Exclude following linters from requiring an explanation. + # Default: [] + allow-no-explanation: [ funlen, gocognit, lll ] + # Enable to require an explanation of nonzero length after each nolint directive. + # Default: false + require-explanation: true + # Enable to require nolint directives to mention the specific linter being suppressed. + # Default: false + require-specific: true + + perfsprint: + # Optimizes into strings concatenation. + # Default: true + strconcat: false + + rowserrcheck: + # database/sql is always checked + # Default: [] + packages: + - github.com/jmoiron/sqlx + + sloglint: + # Enforce not using global loggers. + # Values: + # - "": disabled + # - "all": report all global loggers + # - "default": report only the default slog logger + # https://github.com/go-simpler/sloglint?tab=readme-ov-file#no-global + # Default: "" + no-global: "all" + # Enforce using methods that accept a context. + # Values: + # - "": disabled + # - "all": report all contextless calls + # - "scope": report only if a context exists in the scope of the outermost function + # https://github.com/go-simpler/sloglint?tab=readme-ov-file#context-only + # Default: "" + context: "scope" + + tenv: + # The option `all` will run against whole test files (`_test.go`) regardless of method/function signatures. + # Otherwise, only methods that take `*testing.T`, `*testing.B`, and `testing.TB` as arguments are checked. + # Default: false + all: true + +issues: + # Maximum count of issues with the same text. + # Set to 0 to disable. + # Default: 3 + max-same-issues: 50 + + exclude-rules: + - source: "(noinspection|TODO)" + linters: [ godot ] + - source: "//noinspection" + linters: [ gocritic ] + - path: "_test\\.go" + linters: + - dupword + - err113 + - forcetypeassert + - goconst diff --git a/README.md b/README.md index 587518aa7..0c86511f3 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ # Canonical Kubernetes Snap [![End to End Tests](https://github.com/canonical/k8s-snap/actions/workflows/integration.yaml/badge.svg)](https://github.com/canonical/k8s-snap/actions/workflows/integration.yaml) -![](https://img.shields.io/badge/Kubernetes-1.30-326de6.svg) +![](https://img.shields.io/badge/Kubernetes-1.31-326de6.svg) [![Get it from the Snap Store](https://snapcraft.io/static/images/badges/en/snap-store-black.svg)](https://snapcraft.io/k8s) diff --git a/SECURITY.md b/SECURITY.md new file mode 100644 index 000000000..1a3299a16 --- /dev/null +++ b/SECURITY.md @@ -0,0 +1,15 @@ +# Security Policy + +## Reporting a Vulnerability + +To report a security issue, please follow the steps below: + +Using GitHub, file a [Private Security Report](https://github.com/canonical/k8s-snap/security/advisories/new) with: +- A description of the issue +- Steps to reproduce the issue +- Affected versions of the `k8s-snap` package +- Any known mitigations for the issue + +The [Ubuntu Security disclosure and embargo policy](https://ubuntu.com/security/disclosure-policy) contains more information about what to expect during this process and our requirements for responsible disclosure. + +Thank you for contributing to the security and integrity of `k8s-snap`! diff --git a/build-scripts/hack/auto-merge-successful-pr.py b/build-scripts/hack/auto-merge-successful-pr.py new file mode 100755 index 000000000..ea6c98e6b --- /dev/null +++ b/build-scripts/hack/auto-merge-successful-pr.py @@ -0,0 +1,55 @@ +#!/bin/env python3 + +import shlex +import subprocess +import json + +LABEL = "automerge" +APPROVE_MSG = "All status checks passed for PR #{}." + + +def sh(cmd: str) -> str: + """Run a shell command and return its output.""" + _pipe = subprocess.PIPE + result = subprocess.run(shlex.split(cmd), stdout=_pipe, stderr=_pipe, text=True) + if result.returncode != 0: + raise Exception(f"Error running command: {cmd}\nError: {result.stderr}") + return result.stdout.strip() + + +def get_pull_requests() -> list: + """Fetch open pull requests matching some label.""" + prs_json = sh("gh pr list --state open --json number,labels") + prs = json.loads(prs_json) + return [pr for pr in prs if any(label["name"] == LABEL for label in pr["labels"])] + + +def check_pr_passed(pr_number) -> bool: + """Check if all status checks passed for the given PR.""" + checks_json = sh(f"gh pr checks {pr_number} --json bucket") + checks = json.loads(checks_json) + return all(check["bucket"] == "pass" for check in checks) + + +def approve_and_merge_pr(pr_number) -> None: + """Approve and merge the PR.""" + print(APPROVE_MSG.format(pr_number) + "Proceeding with merge...") + sh(f'gh pr review {pr_number} --comment -b "{APPROVE_MSG.format(pr_number)}"') + sh(f"gh pr merge {pr_number} --admin --squash") + + +def process_pull_requests(): + """Process the PRs and merge if checks have passed.""" + prs = get_pull_requests() + + for pr in prs: + pr_number: int = pr["number"] + + if check_pr_passed(pr_number): + approve_and_merge_pr(pr_number) + else: + print(f"Status checks have not passed for PR #{pr_number}. Skipping merge.") + + +if __name__ == "__main__": + process_pull_requests() diff --git a/build-scripts/hack/generate-sbom.py b/build-scripts/hack/generate-sbom.py index d7644124a..94b550c04 100755 --- a/build-scripts/hack/generate-sbom.py +++ b/build-scripts/hack/generate-sbom.py @@ -139,13 +139,15 @@ def k8s_snap_c_dqlite_components(manifest, extra_files): def rock_cilium(manifest, extra_files): LOG.info("Generating SBOM info for Cilium rocks") + cilium_version = "1.16.3" + with util.git_repo(CILIUM_ROCK_REPO, CILIUM_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "cilium/rockcraft.yaml").read_text() - operator_rockcraft = (d / "cilium-operator-generic/rockcraft.yaml").read_text() + rockcraft = (d / f"{cilium_version}/cilium/rockcraft.yaml").read_text() + operator_rockcraft = (d / f"{cilium_version}/cilium-operator-generic/rockcraft.yaml").read_text() - extra_files["cilium/rockcraft.yaml"] = rockcraft - extra_files["cilium-operator-generic/rockcraft.yaml"] = operator_rockcraft + extra_files[f"{cilium_version}/cilium/rockcraft.yaml"] = rockcraft + extra_files[f"{cilium_version}/cilium-operator-generic/rockcraft.yaml"] = operator_rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["cilium"]["source"] @@ -169,10 +171,10 @@ def rock_cilium(manifest, extra_files): }, "language": "go", "details": [ - "cilium/rockcraft.yaml", + f"{cilium_version}/cilium/rockcraft.yaml", "cilium/go.mod", "cilium/go.sum", - "cilium-operator-generic/rockcraft.yaml", + f"{cilium_version}/cilium-operator-generic/rockcraft.yaml", "cilium-operator-generic/go.mod", "cilium-operator-generic/go.sum", ], @@ -190,9 +192,10 @@ def rock_coredns(manifest, extra_files): with util.git_repo(COREDNS_ROCK_REPO, COREDNS_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "rockcraft.yaml").read_text() + # TODO(ben): This should not be hard coded. + rockcraft = (d / "1.11.3/rockcraft.yaml").read_text() - extra_files["coredns/rockcraft.yaml"] = rockcraft + extra_files["coredns/1.11.3/rockcraft.yaml"] = rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["coredns"]["source"] @@ -211,7 +214,11 @@ def rock_coredns(manifest, extra_files): "revision": rock_repo_commit, }, "language": "go", - "details": ["coredns/rockcraft.yaml", "coredns/go.mod", "coredns/go.sum"], + "details": [ + "coredns/1.11.3/rockcraft.yaml", + "coredns/go.mod", + "coredns/go.sum", + ], "source": { "type": "git", "repo": repo_url, @@ -226,9 +233,10 @@ def rock_metrics_server(manifest, extra_files): with util.git_repo(METRICS_SERVER_ROCK_REPO, METRICS_SERVER_ROCK_TAG) as d: rock_repo_commit = util.parse_output(["git", "rev-parse", "HEAD"], cwd=d) - rockcraft = (d / "rockcraft.yaml").read_text() + # TODO(ben): This should not be hard coded. + rockcraft = (d / "0.7.2/rockcraft.yaml").read_text() - extra_files["metrics-server/rockcraft.yaml"] = rockcraft + extra_files["metrics-server/0.7.2/rockcraft.yaml"] = rockcraft rockcraft_yaml = yaml.safe_load(rockcraft) repo_url = rockcraft_yaml["parts"]["metrics-server"]["source"] @@ -248,7 +256,7 @@ def rock_metrics_server(manifest, extra_files): }, "language": "go", "details": [ - "metrics-server/rockcraft.yaml", + "metrics-server/0.7.2/rockcraft.yaml", "metrics-server/go.mod", "metrics-server/go.sum", ], diff --git a/build-scripts/hack/sync-images.sh b/build-scripts/hack/sync-images.sh deleted file mode 100755 index 5ff80f959..000000000 --- a/build-scripts/hack/sync-images.sh +++ /dev/null @@ -1,11 +0,0 @@ -#!/bin/bash - -# Description: -# Sync images from upstream repositories under ghcr.io/canonical. -# -# Usage: -# $ USERNAME="$username" PASSWORD="$password" ./sync-images.sh - -DIR="$(realpath "$(dirname "${0}")")" - -"${DIR}/../../src/k8s/tools/regsync.sh" once -c "${DIR}/upstream-images.yaml" diff --git a/build-scripts/hack/sync-images.yaml b/build-scripts/hack/sync-images.yaml deleted file mode 100644 index 8f8b68ca4..000000000 --- a/build-scripts/hack/sync-images.yaml +++ /dev/null @@ -1,31 +0,0 @@ -sync: - - source: ghcr.io/canonical/k8s-snap/pause:3.10 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/pause:3.10' - type: image - - source: ghcr.io/canonical/cilium-operator-generic:1.15.2-ck2 - target: '{{ env "MIRROR" }}/canonical/cilium-operator-generic:1.15.2-ck2' - type: image - - source: ghcr.io/canonical/cilium:1.15.2-ck2 - target: '{{ env "MIRROR" }}/canonical/cilium:1.15.2-ck2' - type: image - - source: ghcr.io/canonical/coredns:1.11.1-ck4 - target: '{{ env "MIRROR" }}/canonical/coredns:1.11.1-ck4' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-provisioner:v5.0.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-provisioner:v5.0.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-resizer:v1.11.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-resizer:v1.11.1' - type: image - - source: ghcr.io/canonical/k8s-snap/sig-storage/csi-snapshotter:v8.0.1 - target: '{{ env "MIRROR" }}/canonical/k8s-snap/sig-storage/csi-snapshotter:v8.0.1' - type: image - - source: ghcr.io/canonical/metrics-server:0.7.0-ck1 - target: '{{ env "MIRROR" }}/canonical/metrics-server:0.7.0-ck1' - type: image - - source: ghcr.io/canonical/rawfile-localpv:0.8.0-ck4 - target: '{{ env "MIRROR" }}/canonical/rawfile-localpv:0.8.0-ck4' - type: image diff --git a/build-scripts/hack/update-component-versions.py b/build-scripts/hack/update-component-versions.py index 0d99d79a5..e59905bad 100755 --- a/build-scripts/hack/update-component-versions.py +++ b/build-scripts/hack/update-component-versions.py @@ -15,13 +15,17 @@ import sys import yaml from pathlib import Path +import re import util +import urllib.request + logging.basicConfig(level=logging.INFO) LOG = logging.getLogger(__name__) DIR = Path(__file__).absolute().parent +SNAPCRAFT = DIR.parent.parent / "snap/snapcraft.yaml" COMPONENTS = DIR.parent / "components" CHARTS = DIR.parent.parent / "k8s" / "manifests" / "charts" @@ -44,7 +48,7 @@ # MetalLB Helm repository and chart version METALLB_REPO = "https://metallb.github.io/metallb" -METALLB_CHART_VERSION = "0.14.5" +METALLB_CHART_VERSION = "0.14.8" def get_kubernetes_version() -> str: @@ -125,6 +129,8 @@ def update_component_versions(dry_run: bool): if not dry_run: Path(path).write_text(version.strip() + "\n") + update_go_version(dry_run) + for component, pull_helm_chart in [ ("bitnami/contour", pull_contour_chart), ("metallb", pull_metallb_chart), @@ -134,6 +140,25 @@ def update_component_versions(dry_run: bool): pull_helm_chart() +def update_go_version(dry_run: bool): + k8s_version = (COMPONENTS / "kubernetes/version").read_text().strip() + url = f"https://raw.githubusercontent.com/kubernetes/kubernetes/refs/tags/{k8s_version}/.go-version" + with urllib.request.urlopen(url) as response: + go_version = response.read().decode("utf-8").strip() + + LOG.info("Upstream go version is %s", go_version) + go_snap = f'go/{".".join(go_version.split(".")[:2])}/stable' + snapcraft_yaml = SNAPCRAFT.read_text() + if f"- {go_snap}" in snapcraft_yaml: + LOG.info("snapcraft.yaml already contains go version %s", go_snap) + return + + LOG.info("Update go snap version to %s in %s", go_snap, SNAPCRAFT) + if not dry_run: + updated = re.sub(r"- go/\d+\.\d+/stable", f"- {go_snap}", snapcraft_yaml) + SNAPCRAFT.write_text(updated) + + def main(): parser = argparse.ArgumentParser( "update-component-versions.py", usage=USAGE, description=DESCRIPTION diff --git a/build-scripts/hack/update-coredns-chart.sh b/build-scripts/hack/update-coredns-chart.sh new file mode 100755 index 000000000..04e4550f9 --- /dev/null +++ b/build-scripts/hack/update-coredns-chart.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +VERSION="1.36.0" +DIR=$(realpath $(dirname "${0}")) + +CHARTS_PATH="$DIR/../../k8s/manifests/charts" + +cd "$CHARTS_PATH" + +helm pull --repo https://coredns.github.io/helm coredns --version $VERSION diff --git a/build-scripts/hack/update-gateway-api-chart.sh b/build-scripts/hack/update-gateway-api-chart.sh index eb2b9a91b..40039c7ab 100755 --- a/build-scripts/hack/update-gateway-api-chart.sh +++ b/build-scripts/hack/update-gateway-api-chart.sh @@ -1,6 +1,6 @@ #!/bin/bash -VERSION="v1.0.0" +VERSION="v1.1.0" DIR=`realpath $(dirname "${0}")` CHARTS_PATH="$DIR/../../k8s/components/charts" @@ -16,7 +16,6 @@ rm -rf gateway-api/templates/* rm -rf gateway-api/charts cp gateway-api-src/config/crd/standard/* gateway-api/templates/ cp gateway-api-src/config/crd/experimental/gateway.networking.k8s.io_tlsroutes.yaml gateway-api/templates/ -cp gateway-api-src/config/crd/experimental/gateway.networking.k8s.io_grpcroutes.yaml gateway-api/templates/ sed -i 's/^\(version: \).*$/\1'"${VERSION:1}"'/' gateway-api/Chart.yaml sed -i 's/^\(appVersion: \).*$/\1'"${VERSION:1}"'/' gateway-api/Chart.yaml sed -i 's/^\(description: \).*$/\1'"A Helm Chart containing Gateway API CRDs"'/' gateway-api/Chart.yaml diff --git a/build-scripts/hack/update-metallb-chart.sh b/build-scripts/hack/update-metallb-chart.sh new file mode 100755 index 000000000..5f8feab3e --- /dev/null +++ b/build-scripts/hack/update-metallb-chart.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +VERSION="0.14.8" +DIR=$(realpath $(dirname "${0}")) + +CHARTS_PATH="$DIR/../../k8s/manifests/charts" + +cd "$CHARTS_PATH" + +helm pull --repo https://metallb.github.io/metallb metallb --version $VERSION diff --git a/build-scripts/hack/update-metrics-server-chart.sh b/build-scripts/hack/update-metrics-server-chart.sh index 7d8dc352b..873257a28 100755 --- a/build-scripts/hack/update-metrics-server-chart.sh +++ b/build-scripts/hack/update-metrics-server-chart.sh @@ -1,9 +1,9 @@ #!/bin/bash -VERSION="3.12.0" -DIR=`realpath $(dirname "${0}")` +VERSION="3.12.2" +DIR=$(realpath $(dirname "${0}")) -CHARTS_PATH="$DIR/../../k8s/components/charts" +CHARTS_PATH="$DIR/../../k8s/manifests/charts" cd "$CHARTS_PATH" diff --git a/build-scripts/hack/upstream-images.yaml b/build-scripts/hack/upstream-images.yaml index fbc74a75a..2f8b50f3c 100644 --- a/build-scripts/hack/upstream-images.yaml +++ b/build-scripts/hack/upstream-images.yaml @@ -49,11 +49,11 @@ sync: - source: quay.io/tigera/operator:v1.34.0 target: ghcr.io/canonical/k8s-snap/tigera/operator:v1.34.0 type: image - - source: quay.io/metallb/controller:v0.14.5 - target: ghcr.io/canonical/k8s-snap/metallb/controller:v0.14.5 + - source: quay.io/metallb/controller:v0.14.8 + target: ghcr.io/canonical/k8s-snap/metallb/controller:v0.14.8 type: image - - source: quay.io/metallb/speaker:v0.14.5 - target: ghcr.io/canonical/k8s-snap/metallb/speaker:v0.14.5 + - source: quay.io/metallb/speaker:v0.14.8 + target: ghcr.io/canonical/k8s-snap/metallb/speaker:v0.14.8 type: image - source: quay.io/frrouting/frr:9.0.2 target: ghcr.io/canonical/k8s-snap/frrouting/frr:9.0.2 diff --git a/build-scripts/patches/moonray/apply b/build-scripts/patches/moonray/apply index 1233dae42..32a2f8510 100755 --- a/build-scripts/patches/moonray/apply +++ b/build-scripts/patches/moonray/apply @@ -2,9 +2,12 @@ DIR="$(realpath "$(dirname "${0}")")" -# Configure git author -git config user.email k8s-bot@canonical.com -git config user.name k8s-bot +# Configure git author if unset +git_email=$(git config --default "" user.email) +if [ -z "${git_email}" ]; then + git config user.email k8s-team-ci@canonical.com + git config user.name 'k8s-team-ci (CDK Bot)' +fi # Remove unrelated tests rm "${DIR}/../../../tests/integration/tests/test_cilium_e2e.py" diff --git a/build-scripts/patches/strict/0001-Strict-patch.patch b/build-scripts/patches/strict/0001-Strict-patch.patch index 2ca7c50fa..bed2ed6ff 100644 --- a/build-scripts/patches/strict/0001-Strict-patch.patch +++ b/build-scripts/patches/strict/0001-Strict-patch.patch @@ -1,16 +1,17 @@ -From 3338580f4e22b001615320c40b1c1ad95f8a945e Mon Sep 17 00:00:00 2001 +From 94dadc0e3963e0b01af66e490500c619ec45c019 Mon Sep 17 00:00:00 2001 From: Angelos Kolaitis Date: Fri, 10 May 2024 19:17:55 +0300 Subject: [PATCH] Strict patch --- - k8s/hack/init.sh | 6 +- - k8s/wrappers/services/containerd | 5 - - snap/snapcraft.yaml | 168 ++++++++++++++++++++++++++++++- - 3 files changed, 172 insertions(+), 7 deletions(-) + k8s/hack/init.sh | 6 +- + k8s/wrappers/services/containerd | 5 - + snap/snapcraft.yaml | 171 +++++++++++++++++++++- + tests/integration/tests/test_util/util.py | 38 +++-- + 4 files changed, 198 insertions(+), 22 deletions(-) diff --git a/k8s/hack/init.sh b/k8s/hack/init.sh -index a0b57c7d..d53b528a 100755 +index a0b57c7..d53b528 100755 --- a/k8s/hack/init.sh +++ b/k8s/hack/init.sh @@ -1,3 +1,7 @@ @@ -23,7 +24,7 @@ index a0b57c7d..d53b528a 100755 +"${DIR}/connect-interfaces.sh" +"${DIR}/network-requirements.sh" diff --git a/k8s/wrappers/services/containerd b/k8s/wrappers/services/containerd -index c3f71a01..a82e1c03 100755 +index c3f71a0..a82e1c0 100755 --- a/k8s/wrappers/services/containerd +++ b/k8s/wrappers/services/containerd @@ -21,9 +21,4 @@ You can try to apply the profile manually by running: @@ -37,7 +38,7 @@ index c3f71a01..a82e1c03 100755 - k8s::common::execute_service containerd diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml -index 54b5fc0b..01631684 100644 +index 9d21e55..26f49ad 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -7,7 +7,7 @@ description: |- @@ -49,7 +50,7 @@ index 54b5fc0b..01631684 100644 base: core20 environment: REAL_PATH: $PATH -@@ -216,6 +216,20 @@ parts: +@@ -217,6 +217,20 @@ parts: apps: k8s: command: k8s/wrappers/commands/k8s @@ -70,7 +71,7 @@ index 54b5fc0b..01631684 100644 containerd: command: k8s/wrappers/services/containerd daemon: notify -@@ -226,43 +240,195 @@ apps: +@@ -227,43 +241,198 @@ apps: restart-condition: always start-timeout: 5m before: [kubelet] @@ -263,9 +264,61 @@ index 54b5fc0b..01631684 100644 + plugs: + - network + - network-bind -+ - process-control + - network-control ++ - network-observe ++ - process-control + - firewall-control ++ - system-observe ++ - mount-observe +diff --git a/tests/integration/tests/test_util/util.py b/tests/integration/tests/test_util/util.py +index 3e54d68..295c458 100644 +--- a/tests/integration/tests/test_util/util.py ++++ b/tests/integration/tests/test_util/util.py +@@ -191,21 +191,29 @@ def remove_k8s_snap(instance: harness.Instance): + ["snap", "remove", config.SNAP_NAME, "--purge"] + ) + +- LOG.info("Waiting for shims to go away...") +- stubbornly(retries=20, delay_s=5).on(instance).until( +- lambda p: all( +- x not in p.stdout.decode() +- for x in ["containerd-shim", "cilium", "coredns", "/pause"] +- ) +- ).exec(["ps", "-fea"]) +- +- LOG.info("Waiting for kubelet and containerd mounts to go away...") +- stubbornly(retries=20, delay_s=5).on(instance).until( +- lambda p: all( +- x not in p.stdout.decode() +- for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] +- ) +- ).exec(["mount"]) ++ # NOTE(lpetrut): on "strict", the snap remove hook is unable to: ++ # * terminate processes ++ # * remove network namespaces ++ # * list mounts ++ # ++ # https://paste.ubuntu.com/p/WscCCfnvGH/plain/ ++ # https://paste.ubuntu.com/p/sSnJVvZkrr/plain/ ++ # ++ # LOG.info("Waiting for shims to go away...") ++ # stubbornly(retries=20, delay_s=5).on(instance).until( ++ # lambda p: all( ++ # x not in p.stdout.decode() ++ # for x in ["containerd-shim", "cilium", "coredns", "/pause"] ++ # ) ++ # ).exec(["ps", "-fea"]) ++ # ++ # LOG.info("Waiting for kubelet and containerd mounts to go away...") ++ # stubbornly(retries=20, delay_s=5).on(instance).until( ++ # lambda p: all( ++ # x not in p.stdout.decode() ++ # for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] ++ # ) ++ # ).exec(["mount"]) + + # NOTE(neoaggelos): Temporarily disable this as it fails on strict. + # For details, `snap changes` then `snap change $remove_k8s_snap_change`. -- -2.34.1 +2.43.0 diff --git a/build-scripts/patches/strict/apply b/build-scripts/patches/strict/apply index 1729742e2..3f6f7de14 100755 --- a/build-scripts/patches/strict/apply +++ b/build-scripts/patches/strict/apply @@ -3,8 +3,11 @@ DIR="$(realpath "$(dirname "${0}")")" # Configure git author -git config user.email k8s-bot@canonical.com -git config user.name k8s-bot +git_email=$(git config --default "" user.email) +if [ -z "${git_email}" ]; then + git config user.email k8s-team-ci@canonical.com + git config user.name 'k8s-team-ci (CDK Bot)' +fi # Apply strict patch git am "${DIR}/0001-Strict-patch.patch" diff --git a/docs/README.md b/docs/README.md index 9c51c5e0f..f1af97305 100644 --- a/docs/README.md +++ b/docs/README.md @@ -1,6 +1,7 @@ # K8s snap documentation -This part of the repository contains the tools and the source for generating documentation for the Canonical Kubernetes snap. +This part of the repository contains the tools and the source for generating +documentation for the Canonical Kubernetes snap. The directories are organised like this: @@ -11,16 +12,20 @@ The directories are organised like this: ├── README.md ├── src │ ├──{source files for the docs} -└── tools - ├──{sphinx build tools for creating the docs} +├── canonicalk8s +│ ├──{sphinx build tools for creating the docs for Canonical K8s} +├── moonray +│ ├──{sphinx build tools for creating the docs for Canonical K8s} ``` ## Building the docs -This documentation uses the /tools/Makefile to generate HTML docs from the sources. -This can also run specific local tests such as spelling and linkchecking. +This documentation uses the /canonicalk8s/Makefile to generate HTML docs from +the sources. This can also run specific local tests such as spelling and +linkchecking. ## Contributing to the docs -Contributions to this documentation are welcome. Generally these follow the same -rules and process as other contributions - modify the docs source and submit a PR. \ No newline at end of file +Contributions to this documentation are welcome. Generally these follow the +same rules and process as other contributions - modify the docs source and +submit a PR. diff --git a/docs/canonicalk8s/conf.py-old b/docs/canonicalk8s/conf.py-old new file mode 100644 index 000000000..f4346485e --- /dev/null +++ b/docs/canonicalk8s/conf.py-old @@ -0,0 +1,150 @@ +import sys +import os +import yaml + +sys.path.append('./') +from custom_conf import * + +# Configuration file for the Sphinx documentation builder. +# You should not do any modifications to this file. Put your custom +# configuration into the custom_conf.py file. +# If you need to change this file, contribute the changes upstream. +# +# For the full list of built-in configuration values, see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +############################################################ +### Extensions +############################################################ + +extensions = [ + 'sphinx_design', + 'sphinx_tabs.tabs', + 'sphinx_reredirects', + 'canonical.youtube-links', + 'canonical.related-links', + 'canonical.custom-rst-roles', + 'canonical.terminal-output', + 'sphinx_copybutton', + 'sphinxext.opengraph', + 'myst_parser', + 'sphinxcontrib.jquery', + 'notfound.extension' +] +extensions.extend(custom_extensions) + +### Configuration for extensions + +# Additional MyST syntax +myst_enable_extensions = [ + 'substitution', + 'deflist', + 'linkify' +] +myst_enable_extensions.extend(custom_myst_extensions) + +# Used for related links +if not 'discourse_prefix' in html_context and 'discourse' in html_context: + html_context['discourse_prefix'] = html_context['discourse'] + '/t/' + +# The default for notfound_urls_prefix usually works, but not for +# documentation on documentation.ubuntu.com +if slug: + notfound_urls_prefix = '/' + slug + '/en/latest/' + +notfound_context = { + 'title': 'Page not found', + 'body': '

Page not found

\n\n

Sorry, but the documentation page that you are looking for was not found.

\n

Documentation changes over time, and pages are moved around. We try to redirect you to the updated content where possible, but unfortunately, that didn\'t work this time (maybe because the content you were looking for does not exist in this version of the documentation).

\n

You can try to use the navigation to locate the content you\'re looking for, or search for a similar page.

\n', +} + +# Default image for OGP (to prevent font errors, see +# https://github.com/canonical/sphinx-docs-starter-pack/pull/54 ) +if not 'ogp_image' in locals(): + ogp_image = 'https://assets.ubuntu.com/v1/253da317-image-document-ubuntudocs.svg' + +############################################################ +### General configuration +############################################################ + +exclude_patterns = [ + '_build', + 'Thumbs.db', + '.DS_Store', + '.sphinx', + '_parts' +] +exclude_patterns.extend(custom_excludes) + +rst_epilog = ''' +.. include:: /reuse/links.txt +''' +if 'custom_rst_epilog' in locals(): + rst_epilog = custom_rst_epilog + +source_suffix = { + '.rst': 'restructuredtext', + '.md': 'markdown', +} + +if not 'conf_py_path' in html_context and 'github_folder' in html_context: + html_context['conf_py_path'] = html_context['github_folder'] + +# For ignoring specific links +linkcheck_anchors_ignore_for_url = [ + r'https://github\.com/.*' +] +linkcheck_anchors_ignore_for_url.extend(custom_linkcheck_anchors_ignore_for_url) + +# Tags cannot be added directly in custom_conf.py, so add them here +for tag in custom_tags: + tags.add(tag) + +############################################################ +### Styling +############################################################ + +# Find the current builder +builder = 'dirhtml' +if '-b' in sys.argv: + builder = sys.argv[sys.argv.index('-b')+1] + +# Setting templates_path for epub makes the build fail +if builder == 'dirhtml' or builder == 'html': + templates_path = ['.sphinx/_templates'] + +# Theme configuration +html_theme = 'furo' +html_last_updated_fmt = '' +html_permalinks_icon = '¶' + +if html_title == '': + html_theme_options = { + 'sidebar_hide_name': True + } + +############################################################ +### Additional files +############################################################ + +html_static_path = ['.sphinx/_static'] + +html_css_files = [ + 'custom.css', + 'header.css', + 'github_issue_links.css', + 'furo_colors.css' +] +html_css_files.extend(custom_html_css_files) + +html_js_files = ['header-nav.js'] +if 'github_issues' in html_context and html_context['github_issues'] and not disable_feedback_button: + html_js_files.append('github_issue_links.js') +html_js_files.extend(custom_html_js_files) + + +############################################################ +### Myst configuration +############################################################ +if os.path.exists('./reuse/substitutions.yaml'): + with open('./reuse/substitutions.yaml', 'r') as fd: + myst_substitutions = yaml.safe_load(fd.read()) diff --git a/docs/moonray/.custom_wordlist.txt b/docs/moonray/.custom_wordlist.txt index e69de29bb..40495c3f4 100644 --- a/docs/moonray/.custom_wordlist.txt +++ b/docs/moonray/.custom_wordlist.txt @@ -0,0 +1,266 @@ +adapter's +adapters +allocatable +allocator +AlwaysPullImages +api +apiserver +apparmor +AppArmor +args +ARP +asn +ASN +autostart +autosuspend +aws +backend +backported +balancers +benoitblanchon +bgp +BGP +bootloader +CABPCK +CACPCK +capi +CAPI +CAs +Center +ceph +Ceph +cephcsi +cephx +cgroup +cgroups +cidr +CIDR +cidrs +CIDRs +CK8sControlPlane +CLI +CLIs +CloudFormation +ClusterAPI +clusterctl +ClusterRole +ClusterRoleBinding +CMK +CNI +Commenter +config +configMap +ConfigMap +containerd +CoreDNS +Corosync +CPUs +cpuset +crt +csi +CSI +CSRs +cyclictest +daemonset +DaemonSet +datastore +datastores +dbus +de +deallocation +deployable +discoverable +DMA +dns +DNS +DPDK +DRBD +drv +dqlite +EAL +EasyRSA +enp +enum +etcd +EventRateLimit +failover +gapped +GCP +ghcr +Gi +github +GPLv +Graber +Graber's +grafana +haircommander +Harbor +hostname +hostpath +HPC +html +http +https +HugePage +HugePages +iavf +init +initialise +integrations +io +IOMMU +IOV +ip +IPv +IPv4 +IPv6 +IRQs +Jinja +jitter +juju +Juju's +KMS +kube +kube-apiserver +kube-controller-manager +kube-proxy +kube-scheduler +kube-system +kubeconfig +kubectl +kubelet +kubepods +kubernetes +latencies +Latencies +libcontainer +lifecycle +linux +Lite's +LoadBalancer +localhost +Lookaside +lookups +loopback +LPM +lxc +LxcSecurity +LXD +MAAS +macOS +Maskable +MCE +MetalLB +Microbot +MicroCluster +MicroK +MicroK8s +MinIO +modprobe +Moonray +mq +mtu +MTU +multicast +MULTICAST +Multipass +Multus +nameservers +Netplan +NetworkAttachmentDefinition +NFD +NFV +nginx +NGINX +NIC +NMI +nodeport +nohz +NUMA +numactl +OCI +OOM +OpenStack +OSDs +ParseDuration +passthrough +passwordless +pci +PEM +performant +PID +PMD +PMDs +PPA +proc +programmatically +provisioner +PRs +PV +qdisc +qlen +QoS +RADOS +rbac +RBAC +RBD +rc +RCU +README +regctl +regsync +roadmap +Rockcraft +rollout +runtimes +rw +sandboxed +SANs +scalable +SCHED +sControlPlane +sd +SELinux +ServiceAccount +Snapcraft +snapd +SR-IOV +stackexchange +stgraber +STONITH +StorageClass +sudo +sys +systemd +taskset +Telco +throughs +tickless +TLB +tls +TLS +toml +TSC +TTL +ttyS +ubuntu +unix +unschedulable +unsquashed +Velero +vf +VF +vfio +VFIO +VFs +virtualised +VLAN +VMs +VMware +VNFs +VPCs +VSphere +WIP +www +yaml +YAMLs diff --git a/docs/moonray/.sphinx/requirements.txt b/docs/moonray/.sphinx/requirements.txt index c019f178a..fd19a7b47 100644 --- a/docs/moonray/.sphinx/requirements.txt +++ b/docs/moonray/.sphinx/requirements.txt @@ -1,2 +1,3 @@ git+https://github.com/canonical/canonical-sphinx@main#egg=canonical-sphinx sphinx-autobuild +pyspelling diff --git a/docs/src/.custom_wordlist.txt b/docs/src/.custom_wordlist.txt new file mode 100644 index 000000000..cde8f16bf --- /dev/null +++ b/docs/src/.custom_wordlist.txt @@ -0,0 +1,280 @@ +adapter's +adapters +allocatable +allocator +AlwaysPullImages +api +apiserver +apparmor +AppArmor +args +ARP +asn +ASN +autostart +autosuspend +aws +backend +backported +balancers +benoitblanchon +bgp +BGP +bootloader +BPF +CABPCK +CACPCK +capi +CAPI +CAs +Center +ceph +Ceph +cephcsi +cephx +cgroup +cgroups +cidr +CIDR +cidrs +CIDRs +CIS +CK8sControlPlane +CLI +CLIs +CloudFormation +ClusterAPI +clusterctl +ClusterRole +ClusterRoleBinding +CMK +CNI +Commenter +config +configMap +ConfigMap +containerd +CoreDNS +Corosync +CPUs +cpuset +crt +csi +CSI +CSRs +cyclictest +daemonset +DaemonSet +datastore +datastores +dbus +de +deallocation +deployable +discoverable +DMA +dns +DNS +DPDK +DRBD +drv +dqlite +EAL +EasyRSA +eBPF +enp +enum +etcd +eth +EventRateLimit +ExternalIP +failover +gapped +GCP +ghcr +Gi +github +GPLv +Graber +Graber's +grafana +haircommander +Harbor +hostname +hostpath +HPC +html +http +https +HugePage +HugePages +iavf +init +initialise +integrations +InternalIP +io +IOMMU +IOV +ip +IPIP +IPIPCrossSubnet +IPv +IPv4 +IPv6 +IRQs +Jinja +jitter +juju +Juju's +KMS +kube +kube-apiserver +kube-controller-manager +kube-proxy +kube-scheduler +kube-system +kubeconfig +kubectl +kubelet +kubepods +kubernetes +latencies +Latencies +libcontainer +lifecycle +linux +Lite's +LoadBalancer +localhost +Lookaside +lookups +loopback +LPM +lxc +LxcSecurity +LXD +MAAS +macOS +Maskable +MCE +MetalLB +Microbot +MicroCluster +MicroK +MicroK8s +MinIO +modprobe +Moonray +mq +mtu +MTU +multicast +MULTICAST +Multipass +Multus +nameservers +Netplan +NetworkAttachmentDefinition +NFD +NFV +nginx +NGINX +NIC +NMI +NodeInternalIP +nodeport +nohz +NUMA +numactl +OCI +OOM +OpenStack +OSDs +ParseDuration +passthrough +passwordless +pci +PEM +performant +PID +PMD +PMDs +PPA +proc +programmatically +provisioner +PRs +PV +qdisc +qlen +QoS +RADOS +rbac +RBAC +RBD +rc +RCU +README +regctl +regsync +roadmap +Rockcraft +rollout +Runtime +runtimes +rw +sandboxed +SANs +scalable +SCHED +sControlPlane +sd +SELinux +ServiceAccount +Snapcraft +snapd +SR-IOV +src +stackexchange +stgraber +STONITH +StorageClass +sudo +sys +systemd +taskset +Telco +throughs +tickless +TLB +tls +TLS +toml +TSC +TTL +ttyS +ubuntu +unix +unschedulable +unsquashed +Velero +vf +VF +vfio +VFIO +VFs +virtualised +VLAN +VLANs +VMs +VMware +VNFs +VPCs +VSphere +VXLAN +VXLANCrossSubnet +WIP +www +yaml +YAMLs diff --git a/docs/src/.wordlist.txt b/docs/src/.wordlist.txt new file mode 100644 index 000000000..7a7612432 --- /dev/null +++ b/docs/src/.wordlist.txt @@ -0,0 +1,56 @@ +# This wordlist is from the Sphinx starter pack and should not be +# modified. Add any custom terms to .custom_wordlist.txt instead. +# Leave a blank line at the end to support concatenation. + +addons +API +APIs +balancer +Charmhub +CLI +Diátaxis +Dqlite +dropdown +EBS +EKS +enablement +favicon +Furo +Git +GitHub +Grafana +IAM +installable +JSON +Juju +Kubeflow +Kubernetes +Launchpad +linter +LTS +Makefile +Matrix +Mattermost +MyST +namespace +namespaces +NodePort +Numbat +observability +OEM +OLM +Permalink +pre +Quickstart +ReadMe +reST +reStructuredText +RTD +subdirectories +subfolders +subtree +Ubuntu +UI +UUID +VM +YAML diff --git a/docs/src/_parts/bootstrap_config.md b/docs/src/_parts/bootstrap_config.md index 6594f10ae..b1ed6f477 100644 --- a/docs/src/_parts/bootstrap_config.md +++ b/docs/src/_parts/bootstrap_config.md @@ -189,7 +189,7 @@ If omitted defaults to `true`. Sets the cloud provider to be used by the cluster. When this is set as `external`, node will wait for an external cloud provider to -do cloud specific setup and finish node initialisation. +do cloud specific setup and finish node initialization. Possible values: `external`. diff --git a/docs/src/capi/explanation/capi-ck8s.svg b/docs/src/capi/explanation/capi-ck8s.svg deleted file mode 100644 index d7df80727..000000000 --- a/docs/src/capi/explanation/capi-ck8s.svg +++ /dev/null @@ -1,4 +0,0 @@ - - - -
Canonical Kubernetes Bootstrap Provider (CABCK)
CAPI Machine with Canonical Kubernetes Config
CA
Join Token
kubeconfig
Canonical Kubernetes Control Plane Provider (CACPCK)
Infrastructure Providder
Control Plane
Worker Nodes
VM  #1
VM  #2
VM  #3
VM  #N-2
VM  #N-1
VM  #N
...
Provisioned (Workload) Cluster
User
Cluster EP
clusterctl get config
Bootstrap (Management) Cluster
Bootstrap secret
Deliver cloudinit 
for nodes
User talks to cluster EP
Generate Secrets
- Join Token
- CA
 diff --git a/docs/src/capi/tutorial/getting-started.md b/docs/src/capi/tutorial/getting-started.md index 71a8f823a..fa6aac101 100644 --- a/docs/src/capi/tutorial/getting-started.md +++ b/docs/src/capi/tutorial/getting-started.md @@ -1,4 +1,4 @@ -# Cluster provisioning with CAPI and Canonical K8s +# Cluster provisioning with CAPI and {{product}} This guide covers how to deploy a {{product}} multi-node cluster using Cluster API (CAPI). @@ -16,7 +16,7 @@ curl -L https://github.com/kubernetes-sigs/cluster-api/releases/download/v1.7.3/ sudo install -o root -g root -m 0755 clusterctl /usr/local/bin/clusterctl ``` -### Configure clusterctl +## Configure `clusterctl` `clusterctl` contains a list of default providers. Right now, {{product}} is not yet part of that list. To make `clusterctl` aware of the new @@ -33,10 +33,10 @@ providers: url: "https://github.com/canonical/cluster-api-k8s/releases/latest/control-plane-components.yaml" ``` -### Set up a management cluster +## Set up a management cluster -The management cluster hosts the CAPI providers. You can use Canonical -Kubernetes as a management cluster: +The management cluster hosts the CAPI providers. You can use {{product}} as a +management cluster: ``` sudo snap install k8s --classic --edge @@ -50,7 +50,7 @@ When setting up the management cluster, place its kubeconfig under `~/.kube/config` so other tools such as `clusterctl` can discover and interact with it. -### Prepare the infrastructure provider +## Prepare the infrastructure provider Before generating a cluster, you need to configure the infrastructure provider. Each provider has its own prerequisites. Please follow the instructions @@ -68,7 +68,7 @@ chmod +x clusterawsadm sudo mv clusterawsadm /usr/local/bin ``` -`clusterawsadm` helps you bootstrapping the AWS environment that CAPI will use +`clusterawsadm` helps you bootstrapping the AWS environment that CAPI will use. It will also create the necessary IAM roles for you. Start by setting up environment variables defining the AWS account to use, if @@ -153,7 +153,7 @@ You are now all set to deploy the MAAS CAPI infrastructure provider. ```` ````` -### Initialise the management cluster +## Initialise the management cluster To initialise the management cluster with the latest released version of the providers and the infrastructure of your choice: @@ -162,7 +162,7 @@ providers and the infrastructure of your choice: clusterctl init --bootstrap ck8s --control-plane ck8s -i ``` -### Generate a cluster spec manifest +## Generate a cluster spec manifest Once the bootstrap and control-plane controllers are up and running, you can apply the cluster manifests with the specifications of the cluster you want to @@ -198,7 +198,7 @@ set the cluster’s properties. Review the available options in the respective definitions file and edit the cluster manifest (`cluster.yaml` above) to match your needs. -### Deploy the cluster +## Deploy the cluster To deploy the cluster, run: @@ -237,7 +237,7 @@ You can then see the workload nodes using: KUBECONFIG=./kubeconfig sudo k8s kubectl get node ``` -### Delete the cluster +## Delete the cluster To delete a cluster: diff --git a/docs/src/charm/tutorial/getting-started.md b/docs/src/charm/tutorial/getting-started.md index b0859a5d1..222ba1120 100644 --- a/docs/src/charm/tutorial/getting-started.md +++ b/docs/src/charm/tutorial/getting-started.md @@ -7,7 +7,7 @@ instances and also to integrate other operators to enhance or customise your Kubernetes deployment. This tutorial will take you through installing Kubernetes and some common first steps. -## What you will learn +## What will be covered - How to install {{product}} - Making a cluster @@ -41,7 +41,9 @@ The currently available versions of the charm can be discovered by running: ``` juju info k8s ``` + or + ``` juju info k8s-worker ``` @@ -106,13 +108,14 @@ fetched earlier also includes a list of the relations possible, and from this we can see that the k8s-worker requires "cluster: k8s-cluster". To connect these charms and effectively add the worker to our cluster, we use -the 'integrate' command, adding the interface we wish to connect +the 'integrate' command, adding the interface we wish to connect. ``` juju integrate k8s k8s-worker:cluster ``` -After a short time, the worker node will share information with the control plane and be joined to the cluster. +After a short time, the worker node will share information with the control plane +and be joined to the cluster. ## 4. Scale the cluster (Optional) @@ -168,7 +171,8 @@ config file which will just require a bit of editing: juju run k8s/0 get-kubeconfig >> ~/.kube/config ``` -The output includes the root of the YAML, `kubeconfig: |`, so we can just use an editor to remove that line: +The output includes the root of the YAML, `kubeconfig: |`, so we can just use an +editor to remove that line: ``` nano ~/.kube/config @@ -189,6 +193,7 @@ kubectl config show ``` ...which should output something like this: + ``` apiVersion: v1 clusters: @@ -217,15 +222,7 @@ running a simple command such as : kubectl get pods -A ``` -This should return some pods, confirming the command can reach the cluster: - -``` -NAMESPACE NAME READY STATUS RESTARTS AGE -kube-system cilium-4m5xj 1/1 Running 0 35m -kube-system cilium-operator-5ff9ddcfdb-b6qxm 1/1 Running 0 35m -kube-system coredns-7d4dffcffd-tvs6v 1/1 Running 0 35m -kube-system metrics-server-6f66c6cc48-wdxxk 1/1 Running 0 35m -``` +This should return some pods, confirming the command can reach the cluster. ## Next steps diff --git a/docs/src/index.md b/docs/src/index.md deleted file mode 100644 index 1635cc3f5..000000000 --- a/docs/src/index.md +++ /dev/null @@ -1,108 +0,0 @@ -# {{product}} documentation - -{{product}} is a performant, lightweight, secure and -opinionated distribution of **Kubernetes** which includes everything needed to -create and manage a scalable cluster suitable for all use cases. - -You can find out more about {{product}} on this [overview page] or -see a more detailed explanation in our [architecture documentation]. - -![Illustration depicting working on components and clouds][logo] - -```{toctree} -:hidden: -:titlesonly: -Home -``` - -```{toctree} -:hidden: -:titlesonly: -:maxdepth: 6 -:caption: Deploy from Snap package -Overview -snap/tutorial/index -snap/howto/index -snap/explanation/index -snap/reference/index -``` - -```{toctree} -:hidden: -:caption: Deploy with Juju -:titlesonly: -:glob: -Overview -charm/tutorial/index -charm/howto/index -charm/explanation/index -charm/reference/index -``` - -```{toctree} -:hidden: -:caption: Deploy with Cluster API (WIP) -:titlesonly: -:glob: -Overview -capi/tutorial/index -capi/howto/index -capi/explanation/index -capi/reference/index -``` - ---- - -````{grid} 1 1 2 2 - -```{grid-item-card} -:link: snap/ -### [Install K8s from a snap ›](snap/index) -^^^ -Our tutorials, How To guides and other pages will explain how to install, - configure and use the {{product}} 'k8s' snap. -``` - -```{grid-item-card} -:link: charm/ -### [Deploy K8s using Juju ›](charm/index) -^^^ -Our tutorials, How To guides and other pages will explain how to install, - configure and use the {{product}} 'k8s' charm. -``` - - -```{grid-item-card} -:link: capi/ -### [Deploy K8s using Cluster API ›](capi/index) -^^^ -Our tutorials, guides and explanation pages will explain how to install, - configure and use {{product}} through CAPI. -``` -```` - ---- - -## Project and community - -{{product}} is a member of the Ubuntu family. It's an open source -project which welcomes community involvement, contributions, suggestions, fixes -and constructive feedback. - -- Our [Code of Conduct] -- Our [community] -- How to [contribute] -- Our development [roadmap] - - - -[logo]: https://assets.ubuntu.com/v1/843c77b6-juju-at-a-glace.svg - - - -[Code of Conduct]: https://ubuntu.com/community/ethos/code-of-conduct -[community]: snap/reference/community -[contribute]: snap/howto/contribute -[roadmap]: snap/reference/roadmap -[overview page]: snap/explanation/about -[architecture documentation]: snap/reference/architecture diff --git a/docs/src/snap/howto/install/offline.md b/docs/src/snap/howto/install/offline.md index e522efa50..f133455c9 100644 --- a/docs/src/snap/howto/install/offline.md +++ b/docs/src/snap/howto/install/offline.md @@ -61,13 +61,13 @@ add a dummy default route on the `eth0` interface using the following command: ip route add default dev eth0 ``` -```{note} +```{note} Ensure that `eth0` is the name of the default network interface used for pod-to-pod communication. ``` -The dummy gateway will only be used by the Kubernetes services to -know which interface to use, actual connectivity to the internet is not +The dummy gateway will only be used by the Kubernetes services to +know which interface to use, actual connectivity to the internet is not required. Ensure that the dummy gateway rule survives a node reboot. #### Ensure proxy access @@ -94,7 +94,7 @@ For {{product}}, it is also necessary to fetch the images used by its features (network, DNS, etc.) as well as any images that are needed to run specific workloads. -```{note} +```{note} The image options are presented in the order of increasing complexity of implementation. It may be helpful to combine these options for different scenarios. @@ -167,12 +167,18 @@ any upstream registries (e.g. `docker.io`) and the private mirror. ##### Load images with regsync We recommend using [regsync][regsync] to copy images from the upstream registry -to your private registry. Refer to the [sync-images.yaml][sync-images-yaml] -file that contains the configuration for syncing images from the upstream -registry to the private registry. Using the output from `k8s list-images` -update the images in the [sync-images.yaml][sync-images-yaml] file if -necessary. Update the file with the appropriate mirror, and specify a mirror -for ghcr.io that points to the registry. +to your private registry. +For that, create a `sync-images.yaml` file that maps the output from +`k8s list-images` to the private registry mirror and specify a mirror for +ghcr.io that points to the registry. + +``` +sync: + - source: ghcr.io/canonical/k8s-snap/pause:3.10 + target: '{{ env "MIRROR" }}/canonical/k8s-snap/pause:3.10' + type: image + ... +``` After creating the `sync-images.yaml` file, use [regsync][regsync] to sync the images. Assuming your registry mirror is at http://10.10.10.10:5050, run: @@ -264,7 +270,7 @@ capabilities = ["pull", "resolve"] HTTPS requires the additionally specification of the registry CA certificate. Copy the certificate to `/var/snap/k8s/common/etc/containerd/hosts.d/ghcr.io/ca.crt`. -Then add the configuration in +Then add the configuration in `/var/snap/k8s/common/etc/containerd/hosts.d/ghcr.io/hosts.toml`: ``` diff --git a/docs/src/snap/howto/proxy.md b/docs/src/snap/howto/proxy.md deleted file mode 100644 index ad4186625..000000000 --- a/docs/src/snap/howto/proxy.md +++ /dev/null @@ -1,55 +0,0 @@ -# Configure proxy settings for K8s - -{{product}} packages a number of utilities (eg curl, helm) which need -to fetch resources they expect to find on the internet. In a constrained -network environment, such access is usually controlled through proxies. - -To set up a proxy using squid follow the -[how-to-install-a-squid-server][squid] tutorial. - -## Adding proxy configuration for the k8s snap - -If necessary, create the `snap.k8s.containerd.service.d` directory: - -```bash -sudo mkdir -p /etc/systemd/system/snap.k8s.containerd.service.d -``` - -```{note} It is important to add whatever address ranges are used by the - cluster itself to the `NO_PROXY` and `no_proxy` variables. -``` - -For example, assume we have a proxy running at `http://squid.internal:3128` and -we are using the networks `10.0.0.0/8`,`192.168.0.0/16` and `172.16.0.0/12`. -We would add the configuration to the -(`/etc/systemd/system/snap.k8s.containerd.service.d/http-proxy.conf`) file: - -```bash -# /etc/systemd/system/snap.k8s.containerd.service.d/http-proxy.conf -[Service] -Environment="HTTPS_PROXY=http://squid.internal:3128" -Environment="HTTP_PROXY=http://squid.internal:3128" -Environment="NO_PROXY=10.0.0.0/8,10.152.183.1,192.168.0.0/16,127.0.0.1,172.16.0.0/12" -Environment="https_proxy=http://squid.internal:3128" -Environment="http_proxy=http://squid.internal:3128" -Environment="no_proxy=10.0.0.0/8,10.152.183.1,192.168.0.0/16,127.0.0.1,172.16.0.0/12" -``` - -Note that you may need to restart for these settings to take effect. - -```{note} The **10.152.183.0/24** CIDR needs to be covered in the juju-no-proxy - list as it is the Kubernetes service CIDR. Without this any pods will not be - able to reach the cluster's kubernetes-api. You should also exclude the range - used by pods (which defaults to **10.1.0.0/16**) and any required - local networks. -``` - -## Adding proxy configuration for the k8s charms - -Proxy configuration is handled by Juju when deploying the `k8s` charms. Please -see the [documentation for adding proxy configuration via Juju]. - - - -[documentation for adding proxy configuration via Juju]: /charm/howto/proxy -[squid]: https://ubuntu.com/server/docs/how-to-install-a-squid-server diff --git a/docs/src/snap/reference/annotations.md b/docs/src/snap/reference/annotations.md index 0868cda20..3c4230437 100644 --- a/docs/src/snap/reference/annotations.md +++ b/docs/src/snap/reference/annotations.md @@ -1,7 +1,7 @@ # Annotations This page outlines the annotations that can be configured during cluster -[bootstrap]. To do this, set the cluster-config/annotations parameter in +[bootstrap]. To do this, set the `cluster-config.annotations` parameter in the bootstrap configuration. | Name | Description | Values | diff --git a/docs/src/snap/tutorial/add-remove-nodes.md b/docs/src/snap/tutorial/add-remove-nodes.md index 72ff32988..9065c1a6c 100644 --- a/docs/src/snap/tutorial/add-remove-nodes.md +++ b/docs/src/snap/tutorial/add-remove-nodes.md @@ -1,4 +1,4 @@ -# Adding and Removing Nodes +# Adding and removing nodes Typical production clusters are hosted across multiple data centres and cloud environments, enabling them to leverage geographical distribution for improved @@ -56,14 +56,15 @@ sudo snap install --classic --edge k8s ### 2. Bootstrap your control plane node -Bootstrap the control plane node: +Bootstrap the control plane node with default configuration: ``` sudo k8s bootstrap ``` {{product}} allows you to create two types of nodes: control plane and -worker nodes. In this example, we're creating a worker node. +worker nodes. In this example, we just initialised a control plane node, now +let's create a worker node. Generate the token required for the worker node to join the cluster by executing the following command on the control-plane node: @@ -72,6 +73,9 @@ the following command on the control-plane node: sudo k8s get-join-token worker --worker ``` +`worker` refers to the name of the node we want to join. `--worker` is the type +of node we want to join. + A base64 token will be printed to your terminal. Keep it handy as you will need it for the next step. @@ -81,36 +85,36 @@ it for the next step. ### 3. Join the cluster on the worker node -To join the worker node to the cluster, run: +To join the worker node to the cluster, run on worker node: ``` sudo k8s join-cluster ``` -After a few seconds, you should see: `Joined the cluster.` +After a few seconds, you should see: `Joined the cluster.` ### 4. View the status of your cluster -To see what we've accomplished in this tutorial: +Let's review what we've accomplished in this tutorial. -If you created a control plane node, check that it joined successfully: +To see the control plane node created: ``` sudo k8s status ``` -If you created a worker node, verify with this command: +Verify the worker node joined successfully with this command +on control-plane node: ``` sudo k8s kubectl get nodes ``` -You should see that you've successfully added a worker or control plane node to -your cluster. +You should see that you've successfully added a worker to your cluster. Congratulations! -### 4. Remove Nodes and delete the VMs (Optional) +### 4. Remove nodes and delete the VMs (Optional) It is important to clean-up your nodes before tearing down the VMs. @@ -139,7 +143,7 @@ multipass delete worker multipass purge ``` -## Next Steps +## Next steps - Discover how to enable and configure Ingress resources [Ingress][Ingress] - Learn more about {{product}} with kubectl [How to use diff --git a/docs/src/snap/tutorial/getting-started.md b/docs/src/snap/tutorial/getting-started.md index 69832a87d..c3b033c82 100644 --- a/docs/src/snap/tutorial/getting-started.md +++ b/docs/src/snap/tutorial/getting-started.md @@ -19,22 +19,24 @@ Install the {{product}} snap with: sudo snap install k8s --edge --classic ``` -### 2. Bootstrap a Kubernetes Cluster +### 2. Bootstrap a Kubernetes cluster -Bootstrap a Kubernetes cluster with default configuration using: +The bootstrap command initialises your cluster and configures your host system +as a Kubernetes node. If you would like to bootstrap a Kubernetes cluster with +default configuration run: ``` sudo k8s bootstrap ``` -This command initialises your cluster and configures your host system -as a Kubernetes node. For custom configurations, you can explore additional options using: ``` sudo k8s bootstrap --help ``` +Bootstrapping the cluster can only be done once. + ### 3. Check cluster status To confirm the installation was successful and your node is ready you @@ -44,6 +46,13 @@ should run: sudo k8s status ``` +It may take a few moments for the cluster to be ready. Confirm that {{product}} +has transitioned to the `cluster status ready` state by running: + +``` +sudo k8s status --wait-ready +``` + Run the following command to list all the pods in the `kube-system` namespace: @@ -51,19 +60,13 @@ namespace: sudo k8s kubectl get pods -n kube-system ``` -You will observe at least three pods running: +You will observe at least three pods running. The functions of these three pods +are: - **CoreDNS**: Provides DNS resolution services. - **Network operator**: Manages the life-cycle of the networking solution. - **Network agent**: Facilitates network management. -Confirm that {{product}} has transitioned to the `k8s is ready` state -by running: - -``` -sudo k8s status --wait-ready -``` - ### 5. Access Kubernetes The standard tool for deploying and managing workloads on Kubernetes @@ -124,7 +127,7 @@ running: sudo k8s kubectl get pods ``` -### 8. Enable Local Storage +### 8. Enable local storage In scenarios where you need to preserve application data beyond the life-cycle of the pod, Kubernetes provides persistent volumes. @@ -166,7 +169,7 @@ You can inspect the storage-writer-pod with: sudo k8s kubectl describe pod storage-writer-pod ``` -### 9. Disable Local Storage +### 9. Disable local storage Begin by removing the pod along with the persistent volume claim: @@ -201,14 +204,14 @@ sudo snap remove k8s --purge This option ensures complete removal of the snap and its associated data. -## Next Steps +## Next steps - Learn more about {{product}} with kubectl: [How to use kubectl] - Explore Kubernetes commands with our [Command Reference Guide] - Learn how to set up a multi-node environment [Setting up a K8s cluster] - Configure storage options: [Storage] - Master Kubernetes networking concepts: [Networking] -- Discover how to enable and configure Ingress resources [Ingress] +- Discover how to enable and configure Ingress resources: [Ingress] diff --git a/docs/src/snap/tutorial/kubectl.md b/docs/src/snap/tutorial/kubectl.md index 02b643c18..5bdf04ef9 100644 --- a/docs/src/snap/tutorial/kubectl.md +++ b/docs/src/snap/tutorial/kubectl.md @@ -13,7 +13,7 @@ Before you begin, make sure you have the following: [Getting Started]) - You are using the built-in `kubectl` command from the snap. -### 1. The Kubectl Command +### 1. The kubectl command The `kubectl` command communicates with the [Kubernetes API server][kubernetes-api-server]. @@ -21,17 +21,25 @@ The `kubectl` command communicates with the The `kubectl` command included with {{product}} is built from the original upstream source into the `k8s` snap you have installed. -### 2. How To Use Kubectl +### 2. How to use kubectl -To access `kubectl`, run the following command: +To access `kubectl`, run the following: ``` -sudo k8s kubectl +sudo k8s kubectl ``` +This will display a list of commands possible with `kubectl`. + > **Note**: Only control plane nodes can use the `kubectl` command. Worker > nodes do not have access to this command. +The format of `kubectl` commands are: + +``` +sudo k8s kubectl +``` + ### 3. Configuration In {{product}}, the `kubeconfig` file that is being read to display @@ -49,7 +57,7 @@ Let's review what was created in the [Getting Started] guide. To see what pods were created when we enabled the `network` and `dns` -components: +components during the cluster bootstrap: ``` sudo k8s kubectl get pods -o wide -n kube-system @@ -68,7 +76,7 @@ The `kubernetes` service in the `default` namespace is where the Kubernetes API server resides, and it's the endpoint with which other nodes in your cluster will communicate. -### 5. Creating and Managing Objects +### 5. Creating and managing objects Let's deploy an NGINX server using this command: diff --git a/docs/tools/.sphinx/_static/404.svg b/docs/tools/.sphinx/_static/404.svg new file mode 100644 index 000000000..b353cd339 --- /dev/null +++ b/docs/tools/.sphinx/_static/404.svg @@ -0,0 +1,13 @@ + + + + + + + + + diff --git a/docs/tools/.sphinx/_static/footer.css b/docs/tools/.sphinx/_static/footer.css new file mode 100644 index 000000000..a0a1db454 --- /dev/null +++ b/docs/tools/.sphinx/_static/footer.css @@ -0,0 +1,47 @@ +.display-contributors { + color: var(--color-sidebar-link-text); + cursor: pointer; +} +.all-contributors { + display: none; + z-index: 55; + list-style: none; + position: fixed; + top: 0; + bottom: 0; + left: 0; + right: 0; + width: 200px; + height: 200px; + overflow-y: scroll; + margin: auto; + padding: 0; + background: var(--color-background-primary); + scrollbar-color: var(--color-foreground-border) transparent; + scrollbar-width: thin; +} + +.all-contributors li:hover { + background: var(--color-sidebar-item-background--hover); + width: 100%; +} + +.all-contributors li a{ + color: var(--color-sidebar-link-text); + padding: 1rem; + display: inline-block; +} + +#overlay { + position: fixed; + display: none; + width: 100%; + height: 100%; + top: 0; + left: 0; + right: 0; + bottom: 0; + background-color: rgba(0,0,0,0.5); + z-index: 2; + cursor: pointer; +} diff --git a/docs/tools/.sphinx/_static/footer.js b/docs/tools/.sphinx/_static/footer.js new file mode 100644 index 000000000..9a08b1e99 --- /dev/null +++ b/docs/tools/.sphinx/_static/footer.js @@ -0,0 +1,12 @@ +$(document).ready(function() { + $(document).on("click", function () { + $(".all-contributors").hide(); + $("#overlay").hide(); + }); + + $('.display-contributors').click(function(event) { + $('.all-contributors').toggle(); + $("#overlay").toggle(); + event.stopPropagation(); + }); +}) diff --git a/docs/tools/.sphinx/_templates/404.html b/docs/tools/.sphinx/_templates/404.html new file mode 100644 index 000000000..4cb2d50d3 --- /dev/null +++ b/docs/tools/.sphinx/_templates/404.html @@ -0,0 +1,17 @@ +{% extends "page.html" %} + +{% block content -%} +
+

Page not found

+
+
+
+ {{ body }} +
+
+ Penguin with a question mark +
+
+
+
+{%- endblock content %} diff --git a/docs/tools/.sphinx/build_requirements.py b/docs/tools/.sphinx/build_requirements.py new file mode 100644 index 000000000..df6f149b4 --- /dev/null +++ b/docs/tools/.sphinx/build_requirements.py @@ -0,0 +1,127 @@ +import sys + +sys.path.append('./') +from custom_conf import * + +# The file contains helper functions and the mechanism to build the +# .sphinx/requirements.txt file that is needed to set up the virtual +# environment. + +# You should not do any modifications to this file. Put your custom +# requirements into the custom_required_modules array in the custom_conf.py +# file. If you need to change this file, contribute the changes upstream. + +legacyCanonicalSphinxExtensionNames = [ + "youtube-links", + "related-links", + "custom-rst-roles", + "terminal-output" + ] + +def IsAnyCanonicalSphinxExtensionUsed(): + for extension in custom_extensions: + if (extension.startswith("canonical.") or + extension in legacyCanonicalSphinxExtensionNames): + return True + + return False + +def IsNotFoundExtensionUsed(): + return "notfound.extension" in custom_extensions + +def IsSphinxTabsUsed(): + for extension in custom_extensions: + if extension.startswith("sphinx_tabs."): + return True + + return False + +def AreRedirectsDefined(): + return ("sphinx_reredirects" in custom_extensions) or ( + ("redirects" in globals()) and \ + (redirects is not None) and \ + (len(redirects) > 0)) + +def IsOpenGraphConfigured(): + if "sphinxext.opengraph" in custom_extensions: + return True + + for global_variable_name in list(globals()): + if global_variable_name.startswith("ogp_"): + return True + + return False + +def IsMyStParserUsed(): + return ("myst_parser" in custom_extensions) or \ + ("custom_myst_extensions" in globals()) + +def DeduplicateExtensions(extensionNames: [str]): + extensionNames = dict.fromkeys(extensionNames) + resultList = [] + encounteredCanonicalExtensions = [] + + for extensionName in extensionNames: + if extensionName in legacyCanonicalSphinxExtensionNames: + extensionName = "canonical." + extensionName + + if extensionName.startswith("canonical."): + if extensionName not in encounteredCanonicalExtensions: + encounteredCanonicalExtensions.append(extensionName) + resultList.append(extensionName) + else: + resultList.append(extensionName) + + return resultList + +if __name__ == "__main__": + requirements = [ + "furo", + "pyspelling", + "sphinx", + "sphinx-autobuild", + "sphinx-copybutton", + "sphinx-design", + "sphinxcontrib-jquery", + "watchfiles", + "GitPython" + + ] + + requirements.extend(custom_required_modules) + + if IsAnyCanonicalSphinxExtensionUsed(): + requirements.append("canonical-sphinx-extensions") + + if IsNotFoundExtensionUsed(): + requirements.append("sphinx-notfound-page") + + if IsSphinxTabsUsed(): + requirements.append("sphinx-tabs") + + if AreRedirectsDefined(): + requirements.append("sphinx-reredirects") + + if IsOpenGraphConfigured(): + requirements.append("sphinxext-opengraph") + + if IsMyStParserUsed(): + requirements.append("myst-parser") + requirements.append("linkify-it-py") + + # removes duplicate entries + requirements = list(dict.fromkeys(requirements)) + requirements.sort() + + with open(".sphinx/requirements.txt", 'w') as requirements_file: + requirements_file.write( + "# DO NOT MODIFY THIS FILE DIRECTLY!\n" + "#\n" + "# This file is generated automatically.\n" + "# Add custom requirements to the custom_required_modules\n" + "# array in the custom_conf.py file and run:\n" + "# make clean && make install\n") + + for requirement in requirements: + requirements_file.write(requirement) + requirements_file.write('\n') diff --git a/docs/tools/.sphinx/fonts/Ubuntu-B.ttf b/docs/tools/.sphinx/fonts/Ubuntu-B.ttf new file mode 100644 index 000000000..b173da274 Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-B.ttf differ diff --git a/docs/tools/.sphinx/fonts/Ubuntu-R.ttf b/docs/tools/.sphinx/fonts/Ubuntu-R.ttf new file mode 100644 index 000000000..d748728a2 Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-R.ttf differ diff --git a/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf b/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf new file mode 100644 index 000000000..4f2d2bc7c Binary files /dev/null and b/docs/tools/.sphinx/fonts/Ubuntu-RI.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf new file mode 100644 index 000000000..7bd666576 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-B.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf new file mode 100644 index 000000000..fdd309d71 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-R.ttf differ diff --git a/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf b/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf new file mode 100644 index 000000000..18f81a292 Binary files /dev/null and b/docs/tools/.sphinx/fonts/UbuntuMono-RI.ttf differ diff --git a/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt b/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt new file mode 100644 index 000000000..ae78a8f94 --- /dev/null +++ b/docs/tools/.sphinx/fonts/ubuntu-font-licence-1.0.txt @@ -0,0 +1,96 @@ +------------------------------- +UBUNTU FONT LICENCE Version 1.0 +------------------------------- + +PREAMBLE +This licence allows the licensed fonts to be used, studied, modified and +redistributed freely. The fonts, including any derivative works, can be +bundled, embedded, and redistributed provided the terms of this licence +are met. The fonts and derivatives, however, cannot be released under +any other licence. The requirement for fonts to remain under this +licence does not require any document created using the fonts or their +derivatives to be published under this licence, as long as the primary +purpose of the document is not to be a vehicle for the distribution of +the fonts. + +DEFINITIONS +"Font Software" refers to the set of files released by the Copyright +Holder(s) under this licence and clearly marked as such. This may +include source files, build scripts and documentation. + +"Original Version" refers to the collection of Font Software components +as received under this licence. + +"Modified Version" refers to any derivative made by adding to, deleting, +or substituting -- in part or in whole -- any of the components of the +Original Version, by changing formats or by porting the Font Software to +a new environment. + +"Copyright Holder(s)" refers to all individuals and companies who have a +copyright ownership of the Font Software. + +"Substantially Changed" refers to Modified Versions which can be easily +identified as dissimilar to the Font Software by users of the Font +Software comparing the Original Version with the Modified Version. + +To "Propagate" a work means to do anything with it that, without +permission, would make you directly or secondarily liable for +infringement under applicable copyright law, except executing it on a +computer or modifying a private copy. Propagation includes copying, +distribution (with or without modification and with or without charging +a redistribution fee), making available to the public, and in some +countries other activities as well. + +PERMISSION & CONDITIONS +This licence does not grant any rights under trademark law and all such +rights are reserved. + +Permission is hereby granted, free of charge, to any person obtaining a +copy of the Font Software, to propagate the Font Software, subject to +the below conditions: + +1) Each copy of the Font Software must contain the above copyright +notice and this licence. These can be included either as stand-alone +text files, human-readable headers or in the appropriate machine- +readable metadata fields within text or binary files as long as those +fields can be easily viewed by the user. + +2) The font name complies with the following: +(a) The Original Version must retain its name, unmodified. +(b) Modified Versions which are Substantially Changed must be renamed to +avoid use of the name of the Original Version or similar names entirely. +(c) Modified Versions which are not Substantially Changed must be +renamed to both (i) retain the name of the Original Version and (ii) add +additional naming elements to distinguish the Modified Version from the +Original Version. The name of such Modified Versions must be the name of +the Original Version, with "derivative X" where X represents the name of +the new work, appended to that name. + +3) The name(s) of the Copyright Holder(s) and any contributor to the +Font Software shall not be used to promote, endorse or advertise any +Modified Version, except (i) as required by this licence, (ii) to +acknowledge the contribution(s) of the Copyright Holder(s) or (iii) with +their explicit written permission. + +4) The Font Software, modified or unmodified, in part or in whole, must +be distributed entirely under this licence, and must not be distributed +under any other licence. The requirement for fonts to remain under this +licence does not affect any document created using the Font Software, +except any version of the Font Software extracted from a document +created using the Font Software may only be distributed under this +licence. + +TERMINATION +This licence becomes null and void if any of the above conditions are +not met. + +DISCLAIMER +THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, +EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF +MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OF +COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE +COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, +INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL +DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING +FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM OTHER +DEALINGS IN THE FONT SOFTWARE. diff --git a/docs/tools/.sphinx/get_vale_conf.py b/docs/tools/.sphinx/get_vale_conf.py new file mode 100644 index 000000000..23d890153 --- /dev/null +++ b/docs/tools/.sphinx/get_vale_conf.py @@ -0,0 +1,41 @@ +#! /usr/bin/env python + +import requests +import os + +DIR=os.getcwd() + +def main(): + + if os.path.exists(f"{DIR}/.sphinx/styles/Canonical"): + print("Vale directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + + if os.path.exists(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical"): + print("Vocab directory exists") + else: + os.makedirs(f"{DIR}/.sphinx/styles/config/vocabularies/Canonical") + + url = "https://api.github.com/repos/canonical/praecepta/contents/styles/config/vocabularies/Canonical" + r = requests.get(url) + for item in r.json(): + download = requests.get(item["download_url"]) + file = open(".sphinx/styles/config/vocabularies/Canonical/" + item["name"], "w") + file.write(download.text) + file.close() + config = requests.get("https://raw.githubusercontent.com/canonical/praecepta/main/vale.ini") + file = open(".sphinx/vale.ini", "w") + file.write(config.text) + file.close() + +if __name__ == "__main__": + main() diff --git a/docs/tools/.sphinx/images/Canonical-logo-4x.png b/docs/tools/.sphinx/images/Canonical-logo-4x.png new file mode 100644 index 000000000..fd75696eb Binary files /dev/null and b/docs/tools/.sphinx/images/Canonical-logo-4x.png differ diff --git a/docs/tools/.sphinx/images/front-page-light.pdf b/docs/tools/.sphinx/images/front-page-light.pdf new file mode 100644 index 000000000..bb68cdf8f Binary files /dev/null and b/docs/tools/.sphinx/images/front-page-light.pdf differ diff --git a/docs/tools/.sphinx/images/front-page.png b/docs/tools/.sphinx/images/front-page.png new file mode 100644 index 000000000..c80e84303 Binary files /dev/null and b/docs/tools/.sphinx/images/front-page.png differ diff --git a/docs/tools/.sphinx/images/normal-page-footer.pdf b/docs/tools/.sphinx/images/normal-page-footer.pdf new file mode 100644 index 000000000..dfd73cbc7 Binary files /dev/null and b/docs/tools/.sphinx/images/normal-page-footer.pdf differ diff --git a/docs/tools/.sphinx/latex_elements_template.txt b/docs/tools/.sphinx/latex_elements_template.txt new file mode 100644 index 000000000..2b13b514a --- /dev/null +++ b/docs/tools/.sphinx/latex_elements_template.txt @@ -0,0 +1,119 @@ +{ + 'papersize': 'a4paper', + 'pointsize': '11pt', + 'fncychap': '', + 'preamble': r''' +%\usepackage{charter} +%\usepackage[defaultsans]{lato} +%\usepackage{inconsolata} +\setmainfont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{Ubuntu} +\setmonofont[UprightFont = *-R, BoldFont = *-B, ItalicFont=*-RI, Extension = .ttf]{UbuntuMono} +\usepackage[most]{tcolorbox} +\tcbuselibrary{breakable} +\usepackage{lastpage} +\usepackage{tabto} +\usepackage{ifthen} +\usepackage{etoolbox} +\usepackage{fancyhdr} +\usepackage{graphicx} +\usepackage{titlesec} +\usepackage{fontspec} +\usepackage{tikz} +\usepackage{changepage} +\usepackage{array} +\usepackage{tabularx} +\definecolor{yellowgreen}{RGB}{154, 205, 50} +\definecolor{title}{RGB}{76, 17, 48} +\definecolor{subtitle}{RGB}{116, 27, 71} +\definecolor{label}{RGB}{119, 41, 100} +\definecolor{copyright}{RGB}{174, 167, 159} +\makeatletter +\def\tcb@finalize@environment{% + \color{.}% hack for xelatex + \tcb@layer@dec% +} +\makeatother +\newenvironment{sphinxclassprompt}{\color{yellowgreen}\setmonofont[Color = 9ACD32, UprightFont = *-R, Extension = .ttf]{UbuntuMono}}{} +\tcbset{enhanced jigsaw, colback=black, fontupper=\color{white}} +\newtcolorbox{termbox}{use color stack, breakable, colupper=white, halign=flush left} +\newenvironment{sphinxclassterminal}{\setmonofont[Color = white, UprightFont = *-R, Extension = .ttf]{UbuntuMono}\sphinxsetup{VerbatimColor={black}}\begin{termbox}}{\end{termbox}} +\newcommand{\dimtorightedge}{% + \dimexpr\paperwidth-1in-\hoffset-\oddsidemargin\relax} +\newcommand{\dimtotop}{% + \dimexpr\height-1in-\voffset-\topmargin-\headheight-\headsep\relax} +\newtoggle{tpage} +\AtBeginEnvironment{titlepage}{\global\toggletrue{tpage}} +\fancypagestyle{plain}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{normal}{ + \fancyhf{} + \fancyfoot[R]{\thepage\ of \pageref*{LastPage}} + \renewcommand{\headrulewidth}{0pt} + \renewcommand{\footrulewidth}{0pt} +} +\fancypagestyle{titlepage}{% + \fancyhf{} + \fancyfoot[L]{\footnotesize \textcolor{copyright}{© 2024 Canonical Ltd. All rights reserved.}} +} +\newcommand\sphinxbackoftitlepage{\thispagestyle{titlepage}} +\titleformat{\chapter}[block]{\Huge \color{title} \bfseries\filright}{\thechapter .}{1.5ex}{} +\titlespacing{\chapter}{0pt}{0pt}{0pt} +\titleformat{\section}[block]{\huge \bfseries\filright}{\thesection .}{1.5ex}{} +\titlespacing{\section}{0pt}{0pt}{0pt} +\titleformat{\subsection}[block]{\Large \bfseries\filright}{\thesubsection .}{1.5ex}{} +\titlespacing{\subsection}{0pt}{0pt}{0pt} +\setcounter{tocdepth}{1} +\renewcommand\pagenumbering[1]{} +''', + 'sphinxsetup': 'verbatimwithframe=false, pre_border-radius=0pt, verbatimvisiblespace=\\phantom{}, verbatimcontinued=\\phantom{}', + 'extraclassoptions': 'openany,oneside', + 'maketitle': r''' +\begin{titlepage} +\begin{flushleft} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south east, inner sep=0] at (current page.south east) { + \includegraphics[width=\paperwidth, height=\paperheight]{front-page-light} + }; + \end{tikzpicture} +\end{flushleft} + +\vspace*{3cm} + +\begin{adjustwidth}{8cm}{0pt} +\begin{flushleft} + \huge \textcolor{black}{\textbf{}{\raggedright{$PROJECT}}} +\end{flushleft} +\end{adjustwidth} + +\vfill + +\begin{adjustwidth}{8cm}{0pt} +\begin{tabularx}{0.5\textwidth}{ l l } + \textcolor{lightgray}{© 2024 Canonical Ltd.} & \hspace{3cm} \\ + \textcolor{lightgray}{All rights reserved.} & \hspace{3cm} \\ + & \hspace{3cm} \\ + & \hspace{3cm} \\ + +\end{tabularx} +\end{adjustwidth} + +\end{titlepage} +\RemoveFromHook{shipout/background} +\AddToHook{shipout/background}{ + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=south west, align=left, inner sep=0] at (current page.south west) { + \includegraphics[width=\paperwidth]{normal-page-footer} + }; + \end{tikzpicture} + \begin{tikzpicture}[remember picture,overlay] + \node[anchor=north east, opacity=0.5, inner sep=35] at (current page.north east) { + \includegraphics[width=4cm]{Canonical-logo-4x} + }; + \end{tikzpicture} + } +''', +} \ No newline at end of file diff --git a/docs/tools/.sphinx/pa11y.json b/docs/tools/.sphinx/pa11y.json new file mode 100644 index 000000000..8df0cb9cb --- /dev/null +++ b/docs/tools/.sphinx/pa11y.json @@ -0,0 +1,9 @@ +{ + "chromeLaunchConfig": { + "args": [ + "--no-sandbox" + ] + }, + "reporter": "cli", + "standard": "WCAG2AA" +} \ No newline at end of file diff --git a/docs/tools/Makefile.sp b/docs/tools/Makefile.sp new file mode 100644 index 000000000..0ad2c8b62 --- /dev/null +++ b/docs/tools/Makefile.sp @@ -0,0 +1,156 @@ +# Minimal makefile for Sphinx documentation +# +# `Makefile.sp` is from the Sphinx starter pack and should not be +# modified. +# Add your customisation to `Makefile` instead. + +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXDIR = .sphinx +SPHINXOPTS ?= -c . -d $(SPHINXDIR)/.doctrees -j auto +SPHINXBUILD ?= sphinx-build +SOURCEDIR = ../src +METRICSDIR = $(SOURCEDIR)/metrics +BUILDDIR = ../_build +VENVDIR = $(SPHINXDIR)/venv +PA11Y = $(SPHINXDIR)/node_modules/pa11y/bin/pa11y.js --config $(SPHINXDIR)/pa11y.json +VENV = $(VENVDIR)/bin/activate +TARGET = * +ALLFILES = *.rst **/*.rst +ADDPREREQS ?= +REQPDFPACKS = latexmk fonts-freefont-otf texlive-latex-recommended texlive-latex-extra texlive-fonts-recommended texlive-font-utils texlive-lang-cjk texlive-xetex plantuml xindy tex-gyre dvipng + +.PHONY: sp-full-help sp-woke-install sp-pa11y-install sp-install sp-run sp-html \ + sp-epub sp-serve sp-clean sp-clean-doc sp-spelling sp-spellcheck sp-linkcheck sp-woke \ + sp-allmetrics sp-pa11y sp-pdf-prep-force sp-pdf-prep sp-pdf Makefile.sp sp-vale sp-bash + +sp-full-help: $(VENVDIR) + @. $(VENV); $(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + @echo "\n\033[1;31mNOTE: This help texts shows unsupported targets!\033[0m" + @echo "Run 'make help' to see supported targets." + +# Shouldn't assume that venv is available on Ubuntu by default; discussion here: +# https://bugs.launchpad.net/ubuntu/+source/python3.4/+bug/1290847 +$(SPHINXDIR)/requirements.txt: + @python3 -c "import venv" || \ + (echo "You must install python3-venv before you can build the documentation."; exit 1) + python3 -m venv $(VENVDIR) + @if [ ! -z "$(ADDPREREQS)" ]; then \ + . $(VENV); pip install \ + $(PIPOPTS) --require-virtualenv $(ADDPREREQS); \ + fi + . $(VENV); python3 $(SPHINXDIR)/build_requirements.py + +# If requirements are updated, venv should be rebuilt and timestamped. +$(VENVDIR): $(SPHINXDIR)/requirements.txt + @echo "... setting up virtualenv" + python3 -m venv $(VENVDIR) + . $(VENV); pip install $(PIPOPTS) --require-virtualenv \ + --upgrade -r $(SPHINXDIR)/requirements.txt \ + --log $(VENVDIR)/pip_install.log + @test ! -f $(VENVDIR)/pip_list.txt || \ + mv $(VENVDIR)/pip_list.txt $(VENVDIR)/pip_list.txt.bak + @. $(VENV); pip list --local --format=freeze > $(VENVDIR)/pip_list.txt + @touch $(VENVDIR) + +sp-woke-install: + @type woke >/dev/null 2>&1 || \ + { echo "Installing \"woke\" snap... \n"; sudo snap install woke; } + +sp-pa11y-install: + @type $(PA11Y) >/dev/null 2>&1 || { \ + echo "Installing \"pa11y\" from npm... \n"; \ + mkdir -p $(SPHINXDIR)/node_modules/ ; \ + npm install --prefix $(SPHINXDIR) pa11y; \ + } + +sp-install: $(VENVDIR) + +sp-run: sp-install + . $(VENV); sphinx-autobuild -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + +# Doesn't depend on $(BUILDDIR) to rebuild properly at every run. +sp-html: sp-install + . $(VENV); $(SPHINXBUILD) -W --keep-going -b dirhtml "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-epub: sp-install + . $(VENV); $(SPHINXBUILD) -b epub "$(SOURCEDIR)" "$(BUILDDIR)" -w $(SPHINXDIR)/warnings.txt $(SPHINXOPTS) + +sp-serve: sp-html + cd "$(BUILDDIR)"; python3 -m http.server --bind 127.0.0.1 8000 + +sp-clean: sp-clean-doc + @test ! -e "$(VENVDIR)" -o -d "$(VENVDIR)" -a "$(abspath $(VENVDIR))" != "$(VENVDIR)" + rm -rf $(VENVDIR) + rm -f $(SPHINXDIR)/requirements.txt + rm -rf $(SPHINXDIR)/node_modules/ + rm -rf $(SPHINXDIR)/styles + rm -rf $(SPHINXDIR)/vale.ini + +sp-clean-doc: + git clean -fx "$(BUILDDIR)" + rm -rf $(SPHINXDIR)/.doctrees + +sp-spellcheck: + . $(VENV) ; python3 -m pyspelling -c $(SPHINXDIR)/spellingcheck.yaml -j $(shell nproc) + +sp-spelling: sp-html sp-spellcheck + +sp-linkcheck: sp-install + . $(VENV) ; $(SPHINXBUILD) -b linkcheck "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) || { grep --color -F "[broken]" "$(BUILDDIR)/output.txt"; exit 1; } + exit 0 + +sp-woke: sp-woke-install + woke $(ALLFILES) --exit-1-on-failure \ + -c https://github.com/canonical/Inclusive-naming/raw/main/config.yml + +sp-pa11y: sp-pa11y-install sp-html + find $(BUILDDIR) -name *.html -print0 | xargs -n 1 -0 $(PA11Y) + +sp-vale: sp-install + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + @cat $(SOURCEDIR)/.wordlist.txt $(SOURCEDIR)/.custom_wordlist.txt >> $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt + @echo "" + @echo "Running Vale against $(TARGET). To change target set TARGET= with make command" + @echo "" + @. $(VENV); vale --config "$(SPHINXDIR)/vale.ini" --glob='*.{md,txt,rst}' $(TARGET) || true + @cat $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt > $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept.txt && rm $(SPHINXDIR)/styles/config/vocabularies/Canonical/accept_backup.txt + +sp-pdf-prep: sp-install + @for packageName in $(REQPDFPACKS); do (dpkg-query -W -f='$${Status}' $$packageName 2>/dev/null | \ + grep -c "ok installed" >/dev/null && echo "Package $$packageName is installed") && continue || \ + (echo "\nPDF generation requires the installation of the following packages: $(REQPDFPACKS)" && \ + echo "" && echo "Run sudo make pdf-prep-force to install these packages" && echo "" && echo \ + "Please be aware these packages will be installed to your system") && exit 1 ; done + +sp-pdf-prep-force: + apt-get update + apt-get upgrade -y + apt-get install --no-install-recommends -y $(REQPDFPACKS) \ + +sp-pdf: sp-pdf-prep + @. $(VENV); sphinx-build -M latexpdf "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) + @rm ./$(BUILDDIR)/latex/front-page-light.pdf || true + @rm ./$(BUILDDIR)/latex/normal-page-footer.pdf || true + @find ./$(BUILDDIR)/latex -name "*.pdf" -exec mv -t ./$(BUILDDIR) {} + + @rm -r $(BUILDDIR)/latex + @echo "\nOutput can be found in ./$(BUILDDIR)\n" + +sp-allmetrics: sp-html + @echo "Recording documentation metrics..." + @echo "Checking for existence of vale..." + . $(VENV) + @. $(VENV); test -d $(SPHINXDIR)/venv/lib/python*/site-packages/vale || pip install vale + @. $(VENV); test -f $(SPHINXDIR)/vale.ini || python3 $(SPHINXDIR)/get_vale_conf.py + @. $(VENV); find $(SPHINXDIR)/venv/lib/python*/site-packages/vale/vale_bin -size 195c -exec vale --config "$(SPHINXDIR)/vale.ini" $(TARGET) > /dev/null \; + @eval '$(METRICSDIR)/scripts/source_metrics.sh $(PWD)' + @eval '$(METRICSDIR)/scripts/build_metrics.sh $(PWD) $(METRICSDIR)' + + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile.sp + . $(VENV); $(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/docs/tools/make.bat b/docs/tools/make.bat new file mode 100644 index 000000000..32bb24529 --- /dev/null +++ b/docs/tools/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=. +set BUILDDIR=_build + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.https://www.sphinx-doc.org/ + exit /b 1 +) + +if "%1" == "" goto help + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/docs/tools/metrics/scripts/build_metrics.sh b/docs/tools/metrics/scripts/build_metrics.sh new file mode 100755 index 000000000..b7140a1a9 --- /dev/null +++ b/docs/tools/metrics/scripts/build_metrics.sh @@ -0,0 +1,14 @@ +#!/bin/bash + +links=0 +images=0 + +# count number of links +links=$(find . -type d -path './.sphinx' -prune -o -name '*.html' -exec cat {} + | grep -o "&2 echo -e "$msg" +} + +function get_dqlite_node_id() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.ID' +} + +function get_dqlite_node_addr() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.Address' +} + +function get_dqlite_node_role() { + local infoYamlPath=$1 + sudo cat $infoYamlPath | yq -r '.Role' +} + +function get_dqlite_role_from_cluster_yaml() { + # Note that the cluster.yaml role may not match the info.yaml role. + # In case of a freshly joined node, info.yaml will show it as a "voter" + # while cluster.yaml lists it as a "spare" node. + local clusterYamlPath=$1 + local nodeId=$2 + + # Update the specified node. + sudo cat $clusterYamlPath | \ + yq -r "(.[] | select(.ID == \"$nodeId\") | .Role )" +} + +function set_dqlite_node_role() { + # The yq snap installs in confined mode, so it's unable to access the + # Dqlite config files. + # In order to modify files in-place, we're using sponge. It reads all + # the stdin data before opening the output file. + local infoYamlPath=$1 + local role=$2 + sudo cat $infoYamlPath | \ + yq ".Role = $role" | + sudo sponge $infoYamlPath +} + +# Update cluster.yaml, setting the specified node as voter (role = 0). +# The other nodes will become spares, having the role set to 2. +function set_dqlite_node_as_sole_voter() { + local clusterYamlPath=$1 + local nodeId=$2 + + # Update the specified node. + sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID == \"$nodeId\") | .Role ) = 0" | \ + sudo sponge $clusterYamlPath + + # Update the other nodes. + sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID != \"$nodeId\") | .Role ) = 2" | \ + sudo sponge $clusterYamlPath +} + +function get_dql_peer_ip() { + local clusterYamlPath=$1 + local nodeId=$2 + + local addresses=( $(sudo cat $clusterYamlPath | \ + yq "(.[] | select(.ID != \"$nodeId\") | .Address )") ) + + if [[ ${#addresses[@]} -gt 1 ]]; then + log_message "More than one dql peers found: ${addresses[@]}" + exit 1 + fi + + if [[ ${#addresses[@]} -lt 1 ]]; then + log_message "No dql peers found." + exit 1 + fi + + echo ${addresses[0]} | cut -d ":" -f 1 +} + +# This function moves the Dqlite state directories to the DRBD mount, +# replacing them with symlinks. This ensures that the primary will always use +# the latest DRBD data. +# +# The existing contents are moved to a backup folder, which can be used as +# part of the recovery process. +function move_statedirs() { + sudo mkdir -p $DRBD_MOUNT_DIR/k8s-dqlite + sudo mkdir -p $DRBD_MOUNT_DIR/k8sd + + log_message "Validating Dqlite state directories." + check_statedir $K8S_DQLITE_STATE_DIR $DRBD_MOUNT_DIR/k8s-dqlite + check_statedir $K8SD_STATE_DIR $DRBD_MOUNT_DIR/k8sd + + if [[ ! -L $K8S_DQLITE_STATE_DIR ]] || [[ ! -L $K8SD_STATE_DIR ]]; then + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + + local expRole=`get_expected_dqlite_role` + # For fresh k8s clusters, the info.yaml role may not match the cluster.yaml role. + local k8sDqliteRole=`get_dqlite_role_from_cluster_yaml \ + $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId` + + if [[ $expRole -ne $k8sDqliteRole ]]; then + # TODO: consider automating this. We may move the pacemaker resource + # ourselves and maybe even copy the remote files through scp or ssh. + # However, there's a risk of race conditions. + log_message "DRBD volume mounted on replica, refusing to transfer Dqlite files." + log_message "Move the DRBD volume to the primary node (through the fs_res Pacemaker resource) and try again." + log_message "Example: sudo crm resource move fs_res && sudo crm resource clear fs_res" + exit 1 + fi + fi + + # Ensure that the k8s services are stopped. + log_message "Stopping k8s services." + sudo snap stop k8s + + if [[ ! -L $K8S_DQLITE_STATE_DIR ]]; then + log_message "Not a symlink: $K8S_DQLITE_STATE_DIR, " \ + "transferring to $DRBD_MOUNT_DIR/k8s-dqlite" + sudo cp -r $K8S_DQLITE_STATE_DIR/. $DRBD_MOUNT_DIR/k8s-dqlite + + log_message "Creating k8s-dqlite state dir backup: $K8S_DQLITE_STATE_BKP_DIR" + sudo rm -rf $K8S_DQLITE_STATE_BKP_DIR + sudo mv $K8S_DQLITE_STATE_DIR/ $K8S_DQLITE_STATE_BKP_DIR + + log_message "Creating symlink $K8S_DQLITE_STATE_DIR -> $DRBD_MOUNT_DIR/k8s-dqlite" + sudo ln -sf $DRBD_MOUNT_DIR/k8s-dqlite $K8S_DQLITE_STATE_DIR + else + log_message "Symlink $K8S_DQLITE_STATE_DIR points to $DRBD_MOUNT_DIR/k8s-dqlite" + fi + + if [[ ! -L $K8SD_STATE_DIR ]]; then + log_message "Not a symlink: $K8SD_STATE_DIR, " \ + "transferring to $DRBD_MOUNT_DIR/k8sd" + sudo cp -r $K8SD_STATE_DIR/. $DRBD_MOUNT_DIR/k8sd + + log_message "Creating k8sd state dir backup: $K8SD_STATE_BKP_DIR" + sudo rm -rf $K8SD_STATE_BKP_DIR + sudo mv $K8SD_STATE_DIR/ $K8SD_STATE_BKP_DIR + + log_message "Creating symlink $K8SD_STATE_DIR -> $DRBD_MOUNT_DIR/k8sd" + sudo ln -sf $DRBD_MOUNT_DIR/k8sd $K8SD_STATE_DIR + else + log_message "Symlink $K8SD_STATE_DIR points to $DRBD_MOUNT_DIR/k8sd" + fi +} + +function ensure_mount_rw() { + if ! mount | grep "on $DRBD_MOUNT_DIR type" &> /dev/null; then + log_message "Missing DRBD mount: $DRBD_MOUNT_DIR" + return 1 + fi + + if ! mount | grep "on $DRBD_MOUNT_DIR type" | grep "rw" &> /dev/null; then + log_message "DRBD mount read-only: $DRBD_MOUNT_DIR" + return 1 + fi +} + +function wait_drbd_promoted() { + log_message "Waiting for one of the DRBD nodes to be promoted." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if sudo crm resource status drbd_master_slave | grep Promoted ; then + log_message "DRBD node promoted." + return 0 + else + log_message "No DRBD node promoted yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for primary DRBD node." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +function ensure_drbd_unmounted() { + if mount | grep "on $DRBD_MOUNT_DIR type" &> /dev/null ; then + log_message "DRBD device mounted: $DRBD_MOUNT_DIR" + return 1 + fi +} + +function ensure_drbd_ready() { + ensure_mount_rw + + diskStatus=`sudo drbdadm status r0 | grep disk | head -1 | cut -d ":" -f 2` + if [[ $diskStatus != "UpToDate" ]]; then + log_message "DRBD disk status not ready. Current status: $diskStatus" + return 1 + else + log_message "DRBD disk up to date." + fi +} + +function wait_drbd_primary () { + log_message "Waiting for primary DRBD node to be ready." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if ensure_drbd_ready; then + log_message "Primary DRBD node ready." + return 0 + else + log_message "Primary DRBD node not ready yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for primary DRBD node." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +function wait_for_peer_k8s() { + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Waiting for k8s to start on peer: $peerIp. Timeout: ${PEER_READY_TIMEOUT}s." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $PEER_READY_TIMEOUT ]]; do + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo k8s status &> /dev/null; then + log_message "Peer ready." + return 0 + else + log_message "Peer not ready yet, retrying in ${pollInterval}s." + sleep $pollInterval + fi + done + + log_message "Timed out waiting for k8s services to start on peer." \ + "Waited: ${SECONDS}. Timeout: ${PEER_READY_TIMEOUT}s." + return 1 + +} + +# "drbdadm status" throws the following if our service starts before +# Pacemaker initialized DRBD (even on the secondary). +# +# r0: No such resource +# Command 'drbdsetup-84 status r0' terminated with exit code 10 +function wait_drbd_resource () { + log_message "Waiting for DRBD resource." + + local pollInterval=2 + # Special parameter, no need to increase it ourselves. + SECONDS=0 + + while [[ $SECONDS -lt $DRBD_READY_TIMEOUT ]]; do + if sudo drbdadm status &> /dev/null; then + log_message "DRBD ready." + return 0 + else + log_message "DRBD not ready yet, retrying in ${pollInterval}s" + sleep $pollInterval + fi + done + + log_message "Timed out waiting for DRBD resource." \ + "Waited: ${SECONDS}. Timeout: ${DRBD_READY_TIMEOUT}s." + return 1 +} + +# Based on the DRBD volume state, we decide if this node should be a +# Dqlite voter or a spare. +function get_expected_dqlite_role() { + drbdResRole=`sudo drbdadm status $DRBD_RES_NAME | head -1 | grep role | cut -d ":" -f 2` + + case $drbdResRole in + "Primary") + echo $DQLITE_ROLE_VOTER + ;; + "Secondary") + echo $DQLITE_ROLE_SPARE + ;; + *) + log_message "Unexpected DRBD role: $drbdResRole" + exit 1 + ;; + esac +} + +function validate_drbd_state() { + wait_drbd_promoted + + drbdResRole=`sudo drbdadm status $DRBD_RES_NAME | head -1 | grep role | cut -d ":" -f 2` + + case $drbdResRole in + "Primary") + wait_drbd_primary + ;; + "Secondary") + ensure_drbd_unmounted + ;; + *) + log_message "Unexpected DRBD role: $drbdResRole" + exit 1 + ;; + esac +} + +# After a failover, the state dir points to the shared DRBD volume. +# We need to restore the node certificate and config files. +function restore_dqlite_confs_and_certs() { + log_message "Restoring Dqlite configs and certificates." + + sudo cp $K8S_DQLITE_STATE_BKP_DIR/info.yaml $K8S_DQLITE_STATE_DIR + + sudo cp $K8SD_STATE_BKP_DIR/database/info.yaml $K8SD_STATE_DIR/database/ + sudo cp $K8SD_STATE_BKP_DIR/daemon.yaml $K8SD_STATE_DIR/ + + # restore k8s-dqlite certificates + sudo cp $K8S_DQLITE_STATE_BKP_DIR/cluster.crt $K8S_DQLITE_STATE_DIR + sudo cp $K8S_DQLITE_STATE_BKP_DIR/cluster.key $K8S_DQLITE_STATE_DIR + + # restore k8sd certificates + sudo cp $K8SD_STATE_BKP_DIR/cluster.crt $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/cluster.key $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/server.crt $K8SD_STATE_DIR + sudo cp $K8SD_STATE_BKP_DIR/server.key $K8SD_STATE_DIR +} + +# Promote the current node as primary and prepare the recovery archives. +function promote_as_primary() { + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local k8sdNodeId=`get_dqlite_node_id $K8SD_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Stopping local k8s services." + sudo snap stop k8s + + # After a node crash, there may be a leaked control socket file and + # k8sd will refuse to perform the recovery. We've just stopped the k8s snap, + # it should be safe to remove such stale unix sockets. + log_message "Removing stale control sockets." + sudo rm -f $K8SD_STATE_DIR/control.socket + + local stoppedPeer=0 + log_message "Checking peer k8s services: $peerIp" + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo snap services k8s | grep -v inactive | grep "active"; then + log_message "Attempting to stop peer k8s services." + # Stop the k8s snap directly instead of the wrapper service so that + # we won't cause failures if both nodes start at the same time. + # The secondary will wait for the k8s services to start on the primary. + if ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo snap stop k8s; then + stoppedPeer=1 + log_message "Successfully stopped peer k8s services." + log_message "The stopped services are going to be restarted after the recovery finishes." + else + log_message "Couldn't stop k8s services on the peer node." \ + "Assuming that the peer node is stopped and proceeding with the recovery." + fi + fi + + log_message "Ensuring rw access to DRBD mount." + # Having RW access to the DRBD mount implies that this is the primary node. + ensure_mount_rw + + restore_dqlite_confs_and_certs + + log_message "Updating Dqlite roles." + # Update info.yaml + set_dqlite_node_role $K8S_DQLITE_INFO_YAML $DQLITE_ROLE_VOTER + set_dqlite_node_role $K8SD_INFO_YAML $DQLITE_ROLE_VOTER + + # Update cluster.yaml + set_dqlite_node_as_sole_voter $K8S_DQLITE_CLUSTER_YAML $k8sDqliteNodeId + set_dqlite_node_as_sole_voter $K8SD_CLUSTER_YAML $k8sdNodeId + + log_message "Restoring Dqlite." + sudo $K8SD_PATH cluster-recover \ + --state-dir=$K8SD_STATE_DIR \ + --k8s-dqlite-state-dir=$K8S_DQLITE_STATE_DIR \ + --log-level $K8SD_LOG_LEVEL \ + --non-interactive + + # TODO: consider removing offending segments if the last snapshot is behind + # and then try again. + + log_message "Copying k8sd recovery tarball to $K8SD_RECOVERY_TARBALL_BKP" + sudo cp $K8SD_RECOVERY_TARBALL $K8SD_RECOVERY_TARBALL_BKP + + log_message "Restarting k8s services." + sudo snap start k8s + + # TODO: validate k8s status + + if [[ $stoppedPeer -ne 0 ]]; then + log_message "Restarting peer k8s services: $peerIp" + # It's importand to issue a restart here since we stopped the k8s snap + # directly and the wrapper service doesn't currently monitor it. + ssh $SSH_OPTS $SSH_USERNAME@$peerIp sudo systemctl restart $SYSTEMD_SERVICE_NAME || + log_message "Couldn't start peer k8s services." + fi +} + +function process_recovery_files_on_secondary() { + local peerIp="$1" + + log_message "Ensuring that the DRBD volume is unmounted." + ensure_drbd_unmounted + + log_message "Restoring local Dqlite backup files." + sudo cp -r $K8S_DQLITE_STATE_BKP_DIR/. $DRBD_MOUNT_DIR/k8s-dqlite/ + sudo cp -r $K8SD_STATE_BKP_DIR/. $DRBD_MOUNT_DIR/k8sd/ + + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/00*-* + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/snapshot-* + sudo rm -f $DRBD_MOUNT_DIR/k8s-dqlite/metadata* + + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/00*-* + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/snapshot-* + sudo rm -f $DRBD_MOUNT_DIR/k8sd/database/metadata* + + log_message "Retrieving k8sd recovery tarball." + scp $SSH_OPTS $SSH_USERNAME@$peerIp:$K8SD_RECOVERY_TARBALL_BKP /tmp/ + sudo mv /tmp/`basename $K8SD_RECOVERY_TARBALL_BKP` \ + $K8SD_RECOVERY_TARBALL + + # TODO: do we really need to transfer recovery tarballs in this situation? + # the spare is simply forwarding the requests to the primary, it doesn't really + # hold any data. + lastK8sDqliteRecoveryTarball=`ssh $SSH_USERNAME@$peerIp \ + sudo ls /var/snap/k8s/common/ | \ + grep -P "recovery-k8s-dqlite-.*post-recovery" | \ + tail -1` + if [ -z "$lastK8sDqliteRecoveryTarball" ]; then + log_message "couldn't retrieve latest k8s-dqlite recovery tarball from $peerIp" + exit 1 + fi + + log_message "Retrieving k8s-dqlite recovery tarball." + scp $SSH_USERNAME@$peerIp:/var/snap/k8s/common/$lastK8sDqliteRecoveryTarball /tmp/ + sudo tar -xf /tmp/$lastK8sDqliteRecoveryTarball -C $K8S_DQLITE_STATE_DIR + + log_message "Updating Dqlite roles." + # Update info.yaml + set_dqlite_node_role $K8S_DQLITE_INFO_YAML $DQLITE_ROLE_SPARE + set_dqlite_node_role $K8SD_INFO_YAML $DQLITE_ROLE_SPARE + # We're skipping cluster.yaml, we expect the recovery archives to contain + # updated cluster.yaml files. +} + +# Recover a former primary, now secondary Dqlite node. +# Run "promote_as_primary" on the ther node first. +function rejoin_secondary() { + log_message "Recovering secondary node." + + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Stopping k8s services." + sudo snap stop k8s + + log_message "Adding temporary Pacemaker constraint." + # We need to prevent failovers from happening while restoring secondary + # Dqlite data, otherwise we may end up overriding or deleting the primary + # node data. + # + # TODO: consider reducing the constraint scope (e.g. resource level constraint + # instead of putting the entire node in standby). + sudo crm node standby + if ! process_recovery_files_on_secondary $peerIp; then + log_message "Dqlite recovery filed, removing temporary Pacemaker constraints." + sudo crm node online + exit 1 + fi + + log_message "Restoring Pacemaker state." + sudo crm node online + + log_message "Restarting k8s services" + sudo snap start k8s +} + +function install_packages() { + sudo apt-get update + + sudo DEBIAN_FRONTEND=noninteractive apt-get install \ + python3 python3-netaddr \ + pacemaker resource-agents-extra \ + drbd-utils ntp linux-image-generic snap moreutils -y + sudo modprobe drbd || sudo apt-get install -y linux-modules-extra-$(uname -r) + + sudo snap install jq + sudo snap install yq + sudo snap install install k8s --classic $K8S_SNAP_CHANNEL +} + +function check_statedir() { + local stateDir="$1" + local expLink="$2" + + if [[ ! -e $stateDir ]]; then + log_message "State directory missing: $stateDir" + exit 1 + fi + + target=`readlink -f $stateDir` + if [[ -L "$stateDir" ]] && [[ "$target" != "$expLink" ]]; then + log_message "Unexpected symlink target. " \ + "State directory: $stateDir. " \ + "Expected symlink target: $expLink. " \ + "Actual symlink target: $target." + exit 1 + fi + + if [[ ! -L $stateDir ]] && [[ ! -z "$( ls -A $expLink )" ]]; then + log_message "State directory is not a symlink, however the " \ + "expected link target exists and is not empty. " \ + "We can't know which files to keep, erroring out. " \ + "State directory: $stateDir. " \ + "Expected symlink target: $expLink." + exit 1 + fi +} + +function check_peer_recovery_tarballs() { + log_message "Retrieving k8s-dqlite node id." + local k8sDqliteNodeId=`get_dqlite_node_id $K8S_DQLITE_INFO_BKP_YAML` + if [[ -z $k8sDqliteNodeId ]]; then + log_message "Couldn't retrieve k8s-dqlite node id." + exit 1 + fi + + log_message "Retrieving Dqlite peer ip." + local peerIp=`get_dql_peer_ip $K8S_DQLITE_CLUSTER_BKP_YAML $k8sDqliteNodeId` + if [[ -z $peerIp ]]; then + log_message "Couldn't retrieve Dqlite peer ip." + exit 1 + fi + + log_message "Checking for recovery taballs on $peerIp." + + k8sdRecoveryTarball=`ssh $SSH_OPTS $SSH_USERNAME@$peerIp \ + sudo ls -A "$K8SD_RECOVERY_TARBALL_BKP"` + if [[ -z $k8sdRecoveryTarball ]]; then + log_message "Peer $peerIp doesn't have k8sd recovery tarball." + return 1 + fi + + lastK8sDqliteRecoveryTarball=`ssh $SSH_OPTS $SSH_USERNAME@$peerIp \ + sudo ls /var/snap/k8s/common/ | \ + grep -P "recovery-k8s-dqlite-.*post-recovery"` + if [[ -z $k8sdRecoveryTarball ]]; then + log_message "Peer $peerIp doesn't have k8s-dqlite recovery tarball." + return 1 + fi +} + +function start_service() { + log_message "Initializing node." + + # DRBD is the primary source of truth for the Dqlite role. + # We need to wait for it to become available. + wait_drbd_resource + + # dump the DRBD and pacemaker status for debugging purposes. + sudo drbdadm status + sudo crm status + + validate_drbd_state + + move_statedirs + + local expRole=`get_expected_dqlite_role` + case $expRole in + $DQLITE_ROLE_VOTER) + log_message "Assuming the Dqlite voter role (primary)." + + # We'll assume that if the primary stopped, it needs to go through + # the recovery process. + promote_as_primary + ;; + $DQLITE_ROLE_SPARE) + log_message "Assuming the Dqlite spare role (secondary)." + + wait_for_peer_k8s + + if check_peer_recovery_tarballs; then + log_message "Recovery tarballs found, initiating recovery." + rejoin_secondary + else + # Maybe the primary didn't change and we don't need to go + # through the recovery process. + # TODO: consider comparing the cluster.yaml files from the + # two nodes. + log_message "Recovery tarballs missing, skipping recovery." + log_message "Starting k8s services." + sudo snap k8s start + fi + ;; + *) + log_message "Unexpected Dqlite role: $expRole" + exit 1 + ;; + esac +} + +function clean_recovery_data() { + log_message "Cleaning up Dqlite recovery data." + rm -f $K8SD_RECOVERY_TARBALL + rm -f $K8SD_RECOVERY_TARBALL_BKP + rm -f $K8S_DQLITE_STATE_DIR/recovery-k8s-dqlite* +} + +function purge() { + log_message "Removing the k8s snap and all the associated files." + + sudo snap remove --purge k8s + + if [[ -d $DRBD_MOUNT_DIR ]]; then + log_message "Cleaning up $DRBD_MOUNT_DIR." + sudo rm -rf $DRBD_MOUNT_DIR/k8sd + sudo rm -rf $DRBD_MOUNT_DIR/k8s-dqlite + + if ! ensure_drbd_unmounted; then + log_message "Cleaning up $DRBD_MOUNT_DIR mount point." + + # The replicas use the mount dir directly, without a block device + # attachment. We need to clean up the mount point as well. + # + # We're using another mount with "--bind" to bypass the DRBD mount. + tempdir=`mktemp -d` + # We need to mount the parent dir. + sudo mount --bind `dirname $DRBD_MOUNT_DIR` $tempdir + sudo rm -rf $tempdir/`basename $DRBD_MOUNT_DIR`/k8sd + sudo rm -rf $tempdir/`basename $DRBD_MOUNT_DIR`/k8s-dqlite + sudo umount $tempdir + sudo rm -rf $tempdir + fi + fi +} + +function clear_taints() { + log_message "Clearing tainted Pacemaker resources." + sudo crm resource clear ha_k8s_failover_service + sudo crm resource clear fs_res + sudo crm resource clear drbd_master_slave + + sudo crm resource cleanup ha_k8s_failover_service + sudo crm resource cleanup fs_res + sudo crm resource cleanup drbd_master_slave +} + +function main() { + local command=$1 + + case $command in + "move_statedirs") + move_statedirs + ;; + "install_packages") + install_packages + ;; + "start_service") + start_service + ;; + "clean_recovery_data") + clean_recovery_data + ;; + "purge") + purge + ;; + "clear_taints") + clear_taints + ;; + *) + cat << EOF +Unknown command: $1 + +usage: $0 + +Commands: + move_statedirs Move the Dqlite state directories to the DRBD mount, + replacing them with symlinks. + The existing contents are moved to a backup folder, + which can be used as part of the recovery process. + install_packages Install the packages required by the two-node HA + cluster. + start_service Initialize the k8s services, taking the following + steps: + 1. Based on the DRBD state, decide if this node + should assume the primary (dqlite voter) or + secondary (spare) role. + 2. If this is the first start, transfer the Dqlite + state directories and create backups. + 3. If this node is a primary, promote it and initiate + the Dqlite recovery, creating recovery tarballs. + Otherwise, copy over the recovery files and + join the existing cluster as a spare. + 4. Start the k8s services. + IMPORTANT: ensure that the DRBD volume is attached + to the primary node when running the command for + the first time. + clean_recovery_data Remove database recovery files. Should be called + after the cluster has been fully recovered. + purge Remove the k8s snap and all its associated files. + clear_taints Clear tainted Pacemaker resources. + +EOF + ;; + esac +} + +if [[ $sourced -ne 1 ]]; then + main $@ +fi diff --git a/k8s/lib.sh b/k8s/lib.sh index e3b871cfb..e81b7aefd 100755 --- a/k8s/lib.sh +++ b/k8s/lib.sh @@ -47,6 +47,8 @@ k8s::common::is_strict() { k8s::remove::network() { k8s::common::setup_env + "${SNAP}/bin/kube-proxy" --cleanup || true + k8s::cmd::k8s x-cleanup network || true } @@ -60,17 +62,48 @@ k8s::remove::containers() { # delete cni network namespaces ip netns list | cut -f1 -d' ' | grep -- "^cni-" | xargs -n1 -r -t ip netns delete || true - # unmount NFS volumes forcefully, as unmounting them normally may hang otherwise. + # The PVC loopback devices use container paths, making them tricky to identify. + # We'll rely on the volume mount paths (/var/lib/kubelet/*). + local LOOP_DEVICES=`cat /proc/mounts | grep /var/lib/kubelet/pods | grep /dev/loop | cut -d " " -f 1` + + # unmount Pod NFS volumes forcefully, as unmounting them normally may hang otherwise. cat /proc/mounts | grep /run/containerd/io.containerd. | grep "nfs[34]" | cut -f2 -d' ' | xargs -r -t umount -f || true cat /proc/mounts | grep /var/lib/kubelet/pods | grep "nfs[34]" | cut -f2 -d' ' | xargs -r -t umount -f || true - # unmount volumes + # unmount Pod volumes gracefully. cat /proc/mounts | grep /run/containerd/io.containerd. | cut -f2 -d' ' | xargs -r -t umount || true cat /proc/mounts | grep /var/lib/kubelet/pods | cut -f2 -d' ' | xargs -r -t umount || true - # umount lingering volumes by force, to prevent potential volume leaks. + # unmount lingering Pod volumes by force, to prevent potential volume leaks. cat /proc/mounts | grep /run/containerd/io.containerd. | cut -f2 -d' ' | xargs -r -t umount -f || true cat /proc/mounts | grep /var/lib/kubelet/pods | cut -f2 -d' ' | xargs -r -t umount -f || true + + # unmount various volumes exposed by CSI plugin drivers. + cat /proc/mounts | grep /var/lib/kubelet/plugins | cut -f2 -d' ' | xargs -r -t umount -f || true + + # remove kubelet plugin sockets, as we don't have the containers associated with them anymore, + # so kubelet won't try to access inexistent plugins on reinstallation. + find /var/lib/kubelet/plugins/ -name "*.sock" | xargs rm -f || true + rm /var/lib/kubelet/plugins_registry/*.sock || true + + cat /proc/mounts | grep /var/snap/k8s/common/var/lib/containerd/ | cut -f2 -d' ' | xargs -r -t umount || true + + # cleanup loopback devices + for dev in $LOOP_DEVICES; do + losetup -d $dev + done +} + +k8s::remove::containerd() { + k8s::common::setup_env + + # only remove containerd if the snap was already bootstrapped. + # this is to prevent removing containerd when it is not installed by the snap. + for file in "containerd-socket-path" "containerd-config-dir" "containerd-root-dir" "containerd-cni-bin-dir"; do + if [ -f "$SNAP_COMMON/lock/$file" ]; then + rm -rf $(cat "$SNAP_COMMON/lock/$file") + fi + done } # Run a ctr command against the local containerd socket @@ -169,3 +202,11 @@ k8s::kubelet::ensure_shared_root_dir() { mount -o remount --make-rshared "$SNAP_COMMON/var/lib/kubelet" /var/lib/kubelet fi } + +# Loads the kernel module names given as arguments +# Example: 'k8s::util::load_kernel_modules mod1 mod2 mod3' +k8s::util::load_kernel_modules() { + k8s::common::setup_env + + modprobe $@ +} diff --git a/k8s/manifests/charts/cilium-1.15.2.tgz b/k8s/manifests/charts/cilium-1.15.2.tgz deleted file mode 100644 index 6bf08bd03..000000000 Binary files a/k8s/manifests/charts/cilium-1.15.2.tgz and /dev/null differ diff --git a/k8s/manifests/charts/cilium-1.16.3.tgz b/k8s/manifests/charts/cilium-1.16.3.tgz new file mode 100644 index 000000000..eaca333a4 Binary files /dev/null and b/k8s/manifests/charts/cilium-1.16.3.tgz differ diff --git a/k8s/manifests/charts/coredns-1.29.0.tgz b/k8s/manifests/charts/coredns-1.29.0.tgz deleted file mode 100644 index 47b44f442..000000000 Binary files a/k8s/manifests/charts/coredns-1.29.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/coredns-1.36.0.tgz b/k8s/manifests/charts/coredns-1.36.0.tgz new file mode 100644 index 000000000..7e44977a5 Binary files /dev/null and b/k8s/manifests/charts/coredns-1.36.0.tgz differ diff --git a/k8s/manifests/charts/gateway-api-1.0.0.tgz b/k8s/manifests/charts/gateway-api-1.0.0.tgz deleted file mode 100644 index 7b84d44a5..000000000 Binary files a/k8s/manifests/charts/gateway-api-1.0.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/gateway-api-1.1.0.tgz b/k8s/manifests/charts/gateway-api-1.1.0.tgz new file mode 100644 index 000000000..57ade850a Binary files /dev/null and b/k8s/manifests/charts/gateway-api-1.1.0.tgz differ diff --git a/k8s/manifests/charts/metallb-0.14.5.tgz b/k8s/manifests/charts/metallb-0.14.5.tgz deleted file mode 100644 index c37846191..000000000 Binary files a/k8s/manifests/charts/metallb-0.14.5.tgz and /dev/null differ diff --git a/k8s/manifests/charts/metallb-0.14.8.tgz b/k8s/manifests/charts/metallb-0.14.8.tgz new file mode 100644 index 000000000..a78eb231c Binary files /dev/null and b/k8s/manifests/charts/metallb-0.14.8.tgz differ diff --git a/k8s/manifests/charts/metrics-server-3.12.0.tgz b/k8s/manifests/charts/metrics-server-3.12.0.tgz deleted file mode 100644 index 22f9f8dc2..000000000 Binary files a/k8s/manifests/charts/metrics-server-3.12.0.tgz and /dev/null differ diff --git a/k8s/manifests/charts/metrics-server-3.12.2.tgz b/k8s/manifests/charts/metrics-server-3.12.2.tgz new file mode 100644 index 000000000..4538e8a10 Binary files /dev/null and b/k8s/manifests/charts/metrics-server-3.12.2.tgz differ diff --git a/k8s/scripts/inspect.sh b/k8s/scripts/inspect.sh index a4ae51cc5..9f559451c 100755 --- a/k8s/scripts/inspect.sh +++ b/k8s/scripts/inspect.sh @@ -111,6 +111,17 @@ function collect_service_diagnostics { journalctl -n 100000 -u "snap.$service" &>"$INSPECT_DUMP/$service/journal.log" } +function collect_registry_mirror_logs { + local mirror_units=`systemctl list-unit-files --state=enabled | grep "registry-" | awk '{print $1}'` + if [ -n "$mirror_units" ]; then + mkdir -p "$INSPECT_DUMP/mirrors" + + for mirror_unit in $mirror_units; do + journalctl -n 100000 -u "$mirror_unit" &>"$INSPECT_DUMP/mirrors/$mirror_unit.log" + done + fi +} + function collect_network_diagnostics { log_info "Copy network diagnostics to the final report tarball" ip a &>"$INSPECT_DUMP/ip-a.log" || true @@ -182,6 +193,9 @@ else check_expected_services "${worker_services[@]}" fi +printf -- 'Collecting registry mirror logs\n' +collect_registry_mirror_logs + printf -- 'Collecting service arguments\n' collect_args diff --git a/k8s/wrappers/services/kube-proxy b/k8s/wrappers/services/kube-proxy index 6cd55a999..ef368665a 100755 --- a/k8s/wrappers/services/kube-proxy +++ b/k8s/wrappers/services/kube-proxy @@ -3,4 +3,16 @@ . "$SNAP/k8s/lib.sh" k8s::util::wait_kube_apiserver + +# NOTE: kube-proxy reads some values related to the `nf_conntrack` +# module from procfs on startup, so we must ensure it's loaded: +# https://github.com/canonical/k8s-snap/issues/626 +if [ -f "/proc/sys/net/netfilter/nf_conntrack_max" ]; then + echo "Kernel module nf_conntrack was already loaded before kube-proxy startup." +else + k8s::util::load_kernel_modules nf_conntrack \ + && echo "Successfully modprobed nf_conntrack before kube-proxy startup." \ + || echo "WARN: Failed to 'modprobe nf_conntrack' before kube-proxy startup." +fi + k8s::common::execute_service kube-proxy diff --git a/snap/hooks/remove b/snap/hooks/remove index 29cf572c3..e84c36dd9 100755 --- a/snap/hooks/remove +++ b/snap/hooks/remove @@ -7,3 +7,5 @@ k8s::common::setup_env k8s::remove::containers k8s::remove::network + +k8s::remove::containerd diff --git a/snap/snapcraft.yaml b/snap/snapcraft.yaml index 9d21e55f1..435f40fb2 100644 --- a/snap/snapcraft.yaml +++ b/snap/snapcraft.yaml @@ -164,6 +164,7 @@ parts: - ethtool - hostname - iproute2 + - ipset - kmod - libatm1 - libnss-resolve diff --git a/src/k8s/Makefile b/src/k8s/Makefile index 4e7d33154..8911da0d8 100644 --- a/src/k8s/Makefile +++ b/src/k8s/Makefile @@ -7,6 +7,13 @@ go.fmt: go mod tidy go fmt ./... +go.lint: +ifeq (, $(shell which golangci-lint)) + echo "golangci-lint not found, installing it" + go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.61.0 +endif + golangci-lint run + go.vet: $(DQLITE_BUILD_SCRIPTS_DIR)/static-go-vet.sh ./... @@ -14,7 +21,7 @@ go.unit: $(DQLITE_BUILD_SCRIPTS_DIR)/static-go-test.sh -v ./pkg/... ./cmd/... -coverprofile=coverage.txt --cover go.doc: bin/static/k8s - bin/static/k8s generate-docs --output-dir ../../docs/src/_parts/commands/ + bin/static/k8s generate-docs --output-dir ../../docs/src/_parts/ --project-dir . ## Static Builds static: bin/static/k8s bin/static/k8sd bin/static/k8s-apiserver-proxy diff --git a/src/k8s/cmd/k8s/hooks.go b/src/k8s/cmd/k8s/hooks.go index 481d63df6..77a945580 100644 --- a/src/k8s/cmd/k8s/hooks.go +++ b/src/k8s/cmd/k8s/hooks.go @@ -34,3 +34,36 @@ func hookInitializeFormatter(env cmdutil.ExecutionEnvironment, format *string) f } } } + +// hookCheckLXD verifies the ownership of directories needed for Kubernetes to function. +// If a potential issue is detected, it displays a warning to the user. +func hookCheckLXD() func(*cobra.Command, []string) { + return func(cmd *cobra.Command, args []string) { + // pathsOwnershipCheck paths to validate root is the owner + pathsOwnershipCheck := []string{"/sys", "/proc", "/dev/kmsg"} + inLXD, err := cmdutil.InLXDContainer() + if err != nil { + cmd.PrintErrf("Failed to check if running inside LXD container: %s", err.Error()) + return + } + if inLXD { + var errMsgs []string + for _, pathToCheck := range pathsOwnershipCheck { + if err = cmdutil.ValidateRootOwnership(pathToCheck); err != nil { + errMsgs = append(errMsgs, err.Error()) + } + } + if len(errMsgs) > 0 { + if debug, _ := cmd.Flags().GetBool("debug"); debug { + cmd.PrintErrln("Warning: When validating required resources potential issues found:") + for _, errMsg := range errMsgs { + cmd.PrintErrln("\t", errMsg) + } + } + cmd.PrintErrln("The lxc profile for Canonical Kubernetes might be missing.") + cmd.PrintErrln("For running k8s inside LXD container refer to " + + "https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/install/lxd/") + } + } + } +} diff --git a/src/k8s/cmd/k8s/k8s.go b/src/k8s/cmd/k8s/k8s.go index 45e26d0c8..815faff7e 100644 --- a/src/k8s/cmd/k8s/k8s.go +++ b/src/k8s/cmd/k8s/k8s.go @@ -35,13 +35,11 @@ func addCommands(root *cobra.Command, group *cobra.Group, commands ...*cobra.Com } func NewRootCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { - var ( - opts struct { - logDebug bool - logVerbose bool - stateDir string - } - ) + var opts struct { + logDebug bool + logVerbose bool + stateDir string + } cmd := &cobra.Command{ Use: "k8s", Short: "Canonical Kubernetes CLI", diff --git a/src/k8s/cmd/k8s/k8s_bootstrap.go b/src/k8s/cmd/k8s/k8s_bootstrap.go index b4243d824..9ba6525d3 100644 --- a/src/k8s/cmd/k8s/k8s_bootstrap.go +++ b/src/k8s/cmd/k8s/k8s_bootstrap.go @@ -13,6 +13,7 @@ import ( apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/client/snapd" "github.com/canonical/k8s/pkg/config" "github.com/canonical/k8s/pkg/k8sd/features" "github.com/canonical/k8s/pkg/utils" @@ -45,8 +46,26 @@ func newBootstrapCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { Use: "bootstrap", Short: "Bootstrap a new Kubernetes cluster", Long: "Generate certificates, configure service arguments and start the Kubernetes services.", - PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat)), + PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat), hookCheckLXD()), Run: func(cmd *cobra.Command, args []string) { + snapdClient, err := snapd.NewClient() + if err != nil { + cmd.PrintErrln("Error: failed to create snapd client: %w", err) + env.Exit(1) + return + } + microk8sInfo, err := snapdClient.GetSnapInfo("microk8s") + if err != nil { + cmd.PrintErrln("Error: failed to check if microk8s is installed: %w", err) + env.Exit(1) + return + } + if microk8sInfo.StatusCode == 200 && microk8sInfo.HasInstallDate() { + cmd.PrintErrln("Error: microk8s snap is installed. Please remove it using the following command and try again:\n\n sudo snap remove microk8s") + env.Exit(1) + return + } + if opts.interactive && opts.configFile != "" { cmd.PrintErrln("Error: --interactive and --file flags cannot be set at the same time.") env.Exit(1) diff --git a/src/k8s/cmd/k8s/k8s_bootstrap_test.go b/src/k8s/cmd/k8s/k8s_bootstrap_test.go index 39fa074d8..ca24ef624 100644 --- a/src/k8s/cmd/k8s/k8s_bootstrap_test.go +++ b/src/k8s/cmd/k8s/k8s_bootstrap_test.go @@ -3,11 +3,11 @@ package k8s import ( "bytes" _ "embed" - "os" "path/filepath" "testing" apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" cmdutil "github.com/canonical/k8s/cmd/util" "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" @@ -62,7 +62,10 @@ var testCases = []testCase{ Enabled: utils.Pointer(true), }, CloudProvider: utils.Pointer("external"), - Annotations: map[string]string{apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove: "true"}, + Annotations: map[string]string{ + apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove: "true", + apiv1_annotations.AnnotationSkipStopServicesOnRemove: "true", + }, }, ControlPlaneTaints: []string{"node-role.kubernetes.io/control-plane:NoSchedule"}, PodCIDR: utils.Pointer("10.100.0.0/16"), @@ -105,7 +108,7 @@ var testCases = []testCase{ func mustAddConfigToTestDir(t *testing.T, configPath string, data string) { t.Helper() // Create the cluster bootstrap config file - err := os.WriteFile(configPath, []byte(data), 0644) + err := utils.WriteFile(configPath, []byte(data), 0o644) if err != nil { t.Fatal(err) } diff --git a/src/k8s/cmd/k8s/k8s_generate_docs.go b/src/k8s/cmd/k8s/k8s_generate_docs.go index 10a8ea715..b95bb8b46 100644 --- a/src/k8s/cmd/k8s/k8s_generate_docs.go +++ b/src/k8s/cmd/k8s/k8s_generate_docs.go @@ -1,29 +1,64 @@ package k8s import ( + "path" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/docgen" "github.com/spf13/cobra" "github.com/spf13/cobra/doc" ) func newGenerateDocsCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { var opts struct { - outputDir string + outputDir string + projectDir string } cmd := &cobra.Command{ Use: "generate-docs", Hidden: true, Short: "Generate markdown documentation", Run: func(cmd *cobra.Command, args []string) { - if err := doc.GenMarkdownTree(cmd.Parent(), opts.outputDir); err != nil { + outPath := path.Join(opts.outputDir, "commands") + if err := doc.GenMarkdownTree(cmd.Parent(), outPath); err != nil { cmd.PrintErrf("Error: Failed to generate markdown documentation for k8s command.\n\nThe error was: %v\n", err) env.Exit(1) return } + + outPath = path.Join(opts.outputDir, "bootstrap_config.md") + err := docgen.MarkdownFromJsonStructToFile(apiv1.BootstrapConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for bootstrap configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + + outPath = path.Join(opts.outputDir, "control_plane_join_config.md") + err = docgen.MarkdownFromJsonStructToFile(apiv1.ControlPlaneJoinConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for ctrl plane join configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + + outPath = path.Join(opts.outputDir, "worker_join_config.md") + err = docgen.MarkdownFromJsonStructToFile(apiv1.WorkerJoinConfig{}, outPath, opts.projectDir) + if err != nil { + cmd.PrintErrf("Error: Failed to generate markdown documentation for worker join configuration\n\n") + cmd.PrintErrf("Error: %v", err) + env.Exit(1) + return + } + cmd.Printf("Generated documentation in %s\n", opts.outputDir) }, } cmd.Flags().StringVar(&opts.outputDir, "output-dir", ".", "directory where the markdown docs will be written") + cmd.Flags().StringVar(&opts.projectDir, "project-dir", "../../", "the path to k8s-snap/src/k8s") return cmd } diff --git a/src/k8s/cmd/k8s/k8s_join_cluster.go b/src/k8s/cmd/k8s/k8s_join_cluster.go index 7507fedcb..4cd5bfe6d 100644 --- a/src/k8s/cmd/k8s/k8s_join_cluster.go +++ b/src/k8s/cmd/k8s/k8s_join_cluster.go @@ -32,7 +32,7 @@ func newJoinClusterCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { cmd := &cobra.Command{ Use: "join-cluster ", Short: "Join a cluster using the provided token", - PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat)), + PreRun: chainPreRunHooks(hookRequireRoot(env), hookInitializeFormatter(env, &opts.outputFormat), hookCheckLXD()), Args: cmdutil.ExactArgs(env, 1), Run: func(cmd *cobra.Command, args []string) { token := args[0] diff --git a/src/k8s/cmd/k8s/k8s_set_test.go b/src/k8s/cmd/k8s/k8s_set_test.go index 8c6bc23d1..7431f1b04 100644 --- a/src/k8s/cmd/k8s/k8s_set_test.go +++ b/src/k8s/cmd/k8s/k8s_set_test.go @@ -192,7 +192,7 @@ func Test_updateConfigMapstructure(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cfg).To(SatisfyAll(tc.assertions...)) } }) diff --git a/src/k8s/cmd/k8s/k8s_x_capi.go b/src/k8s/cmd/k8s/k8s_x_capi.go index 2da816bcb..2c4658fd8 100644 --- a/src/k8s/cmd/k8s/k8s_x_capi.go +++ b/src/k8s/cmd/k8s/k8s_x_capi.go @@ -1,10 +1,9 @@ package k8s import ( - "os" - apiv1 "github.com/canonical/k8s-snap-api/api/v1" cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/utils" "github.com/spf13/cobra" ) @@ -48,7 +47,7 @@ func newXCAPICmd(env cmdutil.ExecutionEnvironment) *cobra.Command { return } - if err := os.WriteFile(env.Snap.NodeTokenFile(), []byte(token), 0600); err != nil { + if err := utils.WriteFile(env.Snap.NodeTokenFile(), []byte(token), 0o600); err != nil { cmd.PrintErrf("Error: Failed to write the node token to file.\n\nThe error was: %v\n", err) env.Exit(1) return diff --git a/src/k8s/cmd/k8s/k8s_x_snapd_config.go b/src/k8s/cmd/k8s/k8s_x_snapd_config.go index f950cec57..b0f07f9a1 100644 --- a/src/k8s/cmd/k8s/k8s_x_snapd_config.go +++ b/src/k8s/cmd/k8s/k8s_x_snapd_config.go @@ -64,11 +64,7 @@ func newXSnapdConfigCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { ctx, cancel := context.WithTimeout(cmd.Context(), opts.timeout) defer cancel() if err := control.WaitUntilReady(ctx, func() (bool, error) { - _, partOfCluster, err := client.NodeStatus(cmd.Context()) - if !partOfCluster { - cmd.PrintErrf("Node is not part of a cluster: %v\n", err) - env.Exit(1) - } + _, _, err := client.NodeStatus(cmd.Context()) return err == nil, nil }); err != nil { cmd.PrintErrf("Error: k8sd did not come up in time: %v\n", err) diff --git a/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml b/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml index 79def822f..2fa172ca6 100644 --- a/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml +++ b/src/k8s/cmd/k8s/testdata/bootstrap-config-full.yaml @@ -23,6 +23,7 @@ cluster-config: cloud-provider: external annotations: k8sd/v1alpha/lifecycle/skip-cleanup-kubernetes-node-on-remove: true + k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: true control-plane-taints: - node-role.kubernetes.io/control-plane:NoSchedule pod-cidr: 10.100.0.0/16 diff --git a/src/k8s/cmd/k8sd/k8sd.go b/src/k8s/cmd/k8sd/k8sd.go index dedfafdf7..3fd28264a 100644 --- a/src/k8s/cmd/k8sd/k8sd.go +++ b/src/k8s/cmd/k8sd/k8sd.go @@ -91,7 +91,7 @@ func NewRootCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { addCommands( cmd, &cobra.Group{ID: "cluster", Title: "K8sd clustering commands:"}, - newClusterRecoverCmd(), + newClusterRecoverCmd(env), ) return cmd diff --git a/src/k8s/cmd/k8sd/k8sd_cluster_recover.go b/src/k8s/cmd/k8sd/k8sd_cluster_recover.go old mode 100755 new mode 100644 index 766acc49f..983aa52b2 --- a/src/k8s/cmd/k8sd/k8sd_cluster_recover.go +++ b/src/k8s/cmd/k8sd/k8sd_cluster_recover.go @@ -16,6 +16,9 @@ import ( "github.com/canonical/go-dqlite" "github.com/canonical/go-dqlite/app" "github.com/canonical/go-dqlite/client" + cmdutil "github.com/canonical/k8s/cmd/util" + "github.com/canonical/k8s/pkg/log" + "github.com/canonical/k8s/pkg/utils" "github.com/canonical/lxd/shared" "github.com/canonical/lxd/shared/termios" "github.com/canonical/microcluster/v3/cluster" @@ -23,12 +26,9 @@ import ( "github.com/spf13/cobra" "golang.org/x/sys/unix" "gopkg.in/yaml.v2" - - "github.com/canonical/k8s/pkg/log" - "github.com/canonical/k8s/pkg/utils" ) -const recoveryConfirmation = `You should only run this command if: +const preRecoveryMessage = `You should only run this command if: - A quorum of cluster members is permanently lost - You are *absolutely* sure all k8s daemons are stopped (sudo snap stop k8s) - This instance has the most up to date database @@ -36,8 +36,17 @@ const recoveryConfirmation = `You should only run this command if: Note that before applying any changes, a database backup is created at: * k8sd (microcluster): /var/snap/k8s/common/var/lib/k8sd/state/db_backup..tar.gz * k8s-dqlite: /var/snap/k8s/common/recovery-k8s-dqlite--pre-recovery.tar.gz +` -Do you want to proceed? (yes/no): ` +const recoveryConfirmation = "Do you want to proceed? (yes/no): " + +const nonInteractiveMessage = `Non-interactive mode requested. + +The command will assume that the dqlite configuration files have already been +modified with the updated cluster member roles and addresses. + +Initiating the dqlite database recovery. +` const clusterK8sdYamlRecoveryComment = `# Member roles can be modified. Unrecoverable nodes should be given the role "spare". # @@ -75,6 +84,7 @@ const yamlHelperCommentFooter = "# ------- everything below will be written ---- var clusterRecoverOpts struct { K8sDqliteStateDir string + NonInteractive bool SkipK8sd bool SkipK8sDqlite bool } @@ -87,29 +97,30 @@ func logDebugf(format string, args ...interface{}) { msg := fmt.Sprintf(format, args...) log.L().Info(msg) } - } -func newClusterRecoverCmd() *cobra.Command { +func newClusterRecoverCmd(env cmdutil.ExecutionEnvironment) *cobra.Command { cmd := &cobra.Command{ Use: "cluster-recover", Short: "Recover the cluster from this member if quorum is lost", - RunE: func(cmd *cobra.Command, args []string) error { + Run: func(cmd *cobra.Command, args []string) { log.Configure(log.Options{ LogLevel: rootCmdOpts.logLevel, AddDirHeader: true, }) - if err := recoveryCmdPrechecks(cmd.Context()); err != nil { - return err + if err := recoveryCmdPrechecks(cmd); err != nil { + cmd.PrintErrf("Recovery precheck failed: %v\n", err) + env.Exit(1) } if clusterRecoverOpts.SkipK8sd { - cmd.Printf("Skipping k8sd recovery.") + cmd.Printf("Skipping k8sd recovery.\n") } else { k8sdTarballPath, err := recoverK8sd() if err != nil { - return fmt.Errorf("failed to recover k8sd, error: %w", err) + cmd.PrintErrf("Failed to recover k8sd, error: %v\n", err) + env.Exit(1) } cmd.Printf("K8sd cluster changes applied.\n") cmd.Printf("New database state saved to %s\n", k8sdTarballPath) @@ -120,15 +131,15 @@ func newClusterRecoverCmd() *cobra.Command { } if clusterRecoverOpts.SkipK8sDqlite { - cmd.Printf("Skipping k8s-dqlite recovery.") + cmd.Printf("Skipping k8s-dqlite recovery.\n") } else { k8sDqlitePreRecoveryTarball, k8sDqlitePostRecoveryTarball, err := recoverK8sDqlite() if err != nil { - return fmt.Errorf( - "failed to recover k8s-dqlite, error: %w, "+ - "pre-recovery backup: %s", - err, k8sDqlitePreRecoveryTarball, - ) + cmd.PrintErrf( + "Failed to recover k8s-dqlite, error: %v, "+ + "pre-recovery backup: %s\n", + err, k8sDqlitePreRecoveryTarball) + env.Exit(1) } cmd.Printf("K8s-dqlite cluster changes applied.\n") cmd.Printf("New database state saved to %s\n", @@ -138,13 +149,13 @@ func newClusterRecoverCmd() *cobra.Command { k8sDqlitePostRecoveryTarball, clusterRecoverOpts.K8sDqliteStateDir) cmd.Printf("Pre-recovery database backup: %s\n\n", k8sDqlitePreRecoveryTarball) } - - return nil }, } cmd.Flags().StringVar(&clusterRecoverOpts.K8sDqliteStateDir, "k8s-dqlite-state-dir", "", "k8s-dqlite datastore location") + cmd.Flags().BoolVar(&clusterRecoverOpts.NonInteractive, "non-interactive", + false, "disable interactive prompts, assume that the configs have been updated") cmd.Flags().BoolVar(&clusterRecoverOpts.SkipK8sd, "skip-k8sd", false, "skip k8sd recovery") cmd.Flags().BoolVar(&clusterRecoverOpts.SkipK8sDqlite, "skip-k8s-dqlite", @@ -166,13 +177,13 @@ func removeEmptyLines(content []byte) []byte { return out } -func recoveryCmdPrechecks(ctx context.Context) error { - log := log.FromContext(ctx) +func recoveryCmdPrechecks(cmd *cobra.Command) error { + log := log.FromContext(cmd.Context()) log.V(1).Info("Running prechecks.") - if !termios.IsTerminal(unix.Stdin) { - return fmt.Errorf("this command is meant to be run in an interactive terminal") + if !termios.IsTerminal(unix.Stdin) && !clusterRecoverOpts.NonInteractive { + return fmt.Errorf("interactive mode requested in a non-interactive terminal") } if clusterRecoverOpts.K8sDqliteStateDir == "" { @@ -182,21 +193,31 @@ func recoveryCmdPrechecks(ctx context.Context) error { return fmt.Errorf("k8sd state dir not specified") } - reader := bufio.NewReader(os.Stdin) - fmt.Print(recoveryConfirmation) + cmd.Print(preRecoveryMessage) + cmd.Print("\n") - input, err := reader.ReadString('\n') - if err != nil { - return fmt.Errorf("couldn't read user input, error: %w", err) - } - input = strings.TrimSuffix(input, "\n") + if clusterRecoverOpts.NonInteractive { + cmd.Print(nonInteractiveMessage) + cmd.Print("\n") + } else { + reader := bufio.NewReader(os.Stdin) + cmd.Print(recoveryConfirmation) + + input, err := reader.ReadString('\n') + if err != nil { + return fmt.Errorf("couldn't read user input, error: %w", err) + } + input = strings.TrimSuffix(input, "\n") - if strings.ToLower(input) != "yes" { - return fmt.Errorf("cluster edit aborted; no changes made") + if strings.ToLower(input) != "yes" { + return fmt.Errorf("cluster edit aborted; no changes made") + } + + cmd.Print("\n") } if !clusterRecoverOpts.SkipK8sDqlite { - if err = ensureK8sDqliteMembersStopped(ctx); err != nil { + if err := ensureK8sDqliteMembersStopped(cmd.Context()); err != nil { return err } } @@ -272,7 +293,7 @@ func ensureK8sDqliteMembersStopped(ctx context.Context) error { }(ctx, dial, member.Address) } - for _, _ = range members { + for range members { addr, ok := <-c if !ok { return fmt.Errorf("channel closed unexpectedly") @@ -325,7 +346,7 @@ func yamlEditorGuide( newContent = removeEmptyLines(newContent) if applyChanges { - err = os.WriteFile(path, newContent, os.FileMode(0o644)) + err = utils.WriteFile(path, newContent, os.FileMode(0o644)) if err != nil { return nil, fmt.Errorf("could not write file: %s, error: %w", path, err) } @@ -376,59 +397,64 @@ func recoverK8sd() (string, error) { clusterYamlPath := path.Join(m.FileSystem.DatabaseDir, "cluster.yaml") clusterYamlCommentHeader := fmt.Sprintf("# K8sd cluster configuration\n# (based on the trust store and %s)\n", clusterYamlPath) - clusterYamlContent, err := yamlEditorGuide( - "", - false, - slices.Concat( - []byte(clusterYamlCommentHeader), - []byte("#\n"), - []byte(clusterK8sdYamlRecoveryComment), - []byte(yamlHelperCommentFooter), - []byte("\n"), - oldMembersYaml, - ), - false, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) - } + clusterYamlContent := oldMembersYaml + if !clusterRecoverOpts.NonInteractive { + // Interactive mode requested (default). + // Assist the user in configuring dqlite. + clusterYamlContent, err = yamlEditorGuide( + "", + false, + slices.Concat( + []byte(clusterYamlCommentHeader), + []byte("#\n"), + []byte(clusterK8sdYamlRecoveryComment), + []byte(yamlHelperCommentFooter), + []byte("\n"), + oldMembersYaml, + ), + false, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } - infoYamlPath := path.Join(m.FileSystem.DatabaseDir, "info.yaml") - infoYamlCommentHeader := fmt.Sprintf("# K8sd info.yaml\n# (%s)\n", infoYamlPath) - _, err = yamlEditorGuide( - infoYamlPath, - true, - slices.Concat( - []byte(infoYamlCommentHeader), - []byte("#\n"), - []byte(infoYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) - } + infoYamlPath := path.Join(m.FileSystem.DatabaseDir, "info.yaml") + infoYamlCommentHeader := fmt.Sprintf("# K8sd info.yaml\n# (%s)\n", infoYamlPath) + _, err = yamlEditorGuide( + infoYamlPath, + true, + slices.Concat( + []byte(infoYamlCommentHeader), + []byte("#\n"), + []byte(infoYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } - daemonYamlPath := path.Join(m.FileSystem.StateDir, "daemon.yaml") - daemonYamlCommentHeader := fmt.Sprintf("# K8sd daemon.yaml\n# (%s)\n", daemonYamlPath) - _, err = yamlEditorGuide( - daemonYamlPath, - true, - slices.Concat( - []byte(daemonYamlCommentHeader), - []byte("#\n"), - []byte(daemonYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", fmt.Errorf("interactive text editor failed, error: %w", err) + daemonYamlPath := path.Join(m.FileSystem.StateDir, "daemon.yaml") + daemonYamlCommentHeader := fmt.Sprintf("# K8sd daemon.yaml\n# (%s)\n", daemonYamlPath) + _, err = yamlEditorGuide( + daemonYamlPath, + true, + slices.Concat( + []byte(daemonYamlCommentHeader), + []byte("#\n"), + []byte(daemonYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", fmt.Errorf("interactive text editor failed, error: %w", err) + } } newMembers := []cluster.DqliteMember{} @@ -465,40 +491,53 @@ func recoverK8sd() (string, error) { func recoverK8sDqlite() (string, string, error) { k8sDqliteStateDir := clusterRecoverOpts.K8sDqliteStateDir + var err error + clusterYamlContent := []byte{} clusterYamlPath := path.Join(k8sDqliteStateDir, "cluster.yaml") clusterYamlCommentHeader := fmt.Sprintf("# k8s-dqlite cluster configuration\n# (%s)\n", clusterYamlPath) - clusterYamlContent, err := yamlEditorGuide( - clusterYamlPath, - true, - slices.Concat( - []byte(clusterYamlCommentHeader), - []byte("#\n"), - []byte(clusterK8sDqliteRecoveryComment), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) - } - infoYamlPath := path.Join(k8sDqliteStateDir, "info.yaml") - infoYamlCommentHeader := fmt.Sprintf("# k8s-dqlite info.yaml\n# (%s)\n", infoYamlPath) - _, err = yamlEditorGuide( - infoYamlPath, - true, - slices.Concat( - []byte(infoYamlCommentHeader), - []byte("#\n"), - []byte(infoYamlRecoveryComment), - utils.YamlCommentLines(clusterYamlContent), - []byte("\n"), - []byte(yamlHelperCommentFooter), - ), - true, - ) - if err != nil { - return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + if clusterRecoverOpts.NonInteractive { + clusterYamlContent, err = os.ReadFile(clusterYamlPath) + if err != nil { + return "", "", fmt.Errorf( + "could not read k8s-dqlite cluster.yaml, error: %w", err) + } + } else { + // Interactive mode requested (default). + // Assist the user in configuring dqlite. + clusterYamlContent, err = yamlEditorGuide( + clusterYamlPath, + true, + slices.Concat( + []byte(clusterYamlCommentHeader), + []byte("#\n"), + []byte(clusterK8sDqliteRecoveryComment), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + } + + infoYamlPath := path.Join(k8sDqliteStateDir, "info.yaml") + infoYamlCommentHeader := fmt.Sprintf("# k8s-dqlite info.yaml\n# (%s)\n", infoYamlPath) + _, err = yamlEditorGuide( + infoYamlPath, + true, + slices.Concat( + []byte(infoYamlCommentHeader), + []byte("#\n"), + []byte(infoYamlRecoveryComment), + utils.YamlCommentLines(clusterYamlContent), + []byte("\n"), + []byte(yamlHelperCommentFooter), + ), + true, + ) + if err != nil { + return "", "", fmt.Errorf("interactive text editor failed, error: %w", err) + } } newMembers := []dqlite.NodeInfo{} diff --git a/src/k8s/cmd/main.go b/src/k8s/cmd/main.go index 578449308..5e4b38bf1 100644 --- a/src/k8s/cmd/main.go +++ b/src/k8s/cmd/main.go @@ -30,14 +30,26 @@ func main() { // choose command based on the binary name base := filepath.Base(os.Args[0]) + var err error switch base { case "k8s-apiserver-proxy": - k8s_apiserver_proxy.NewRootCmd(env).ExecuteContext(ctx) + err = k8s_apiserver_proxy.NewRootCmd(env).ExecuteContext(ctx) case "k8sd": - k8sd.NewRootCmd(env).ExecuteContext(ctx) + err = k8sd.NewRootCmd(env).ExecuteContext(ctx) case "k8s": - k8s.NewRootCmd(env).ExecuteContext(ctx) + err = k8s.NewRootCmd(env).ExecuteContext(ctx) default: panic(fmt.Errorf("invalid entrypoint name %q", base)) } + + // Although k8s commands typically use Run instead of RunE and handle + // errors directly within the command execution, this acts as a safeguard in + // case any are overlooked. + // + // Furthermore, the Cobra framework may not invoke the "Run*" entry points + // at all in case of argument parsing errors, in which case we *need* to + // handle the errors here. + if err != nil { + env.Exit(1) + } } diff --git a/src/k8s/cmd/util/hooks.go b/src/k8s/cmd/util/hooks.go new file mode 100644 index 000000000..6069a8f12 --- /dev/null +++ b/src/k8s/cmd/util/hooks.go @@ -0,0 +1,56 @@ +package cmdutil + +import ( + "fmt" + "os" + "strings" + "syscall" +) + +// getFileOwnerAndGroup retrieves the UID and GID of a file. +func getFileOwnerAndGroup(filePath string) (uid, gid uint32, err error) { + // Get file info using os.Stat + fileInfo, err := os.Stat(filePath) + if err != nil { + return 0, 0, fmt.Errorf("error getting file info: %w", err) + } + // Convert the fileInfo.Sys() to syscall.Stat_t to access UID and GID + stat, ok := fileInfo.Sys().(*syscall.Stat_t) + if !ok { + return 0, 0, fmt.Errorf("failed to cast to syscall.Stat_t") + } + // Return the UID and GID + return stat.Uid, stat.Gid, nil +} + +// ValidateRootOwnership checks if the specified path is owned by the root user and root group. +func ValidateRootOwnership(path string) (err error) { + uid, gid, err := getFileOwnerAndGroup(path) + if err != nil { + return err + } + if uid != 0 { + return fmt.Errorf("owner of %s is user with UID %d expected 0", path, uid) + } + if gid != 0 { + return fmt.Errorf("owner of %s is group with GID %d expected 0", path, gid) + } + return nil +} + +// InLXDContainer checks if k8s runs in a lxd container. +func InLXDContainer() (isLXD bool, err error) { + initialProcessEnvironmentVariables := "/proc/1/environ" + content, err := os.ReadFile(initialProcessEnvironmentVariables) + if err != nil { + // if the permission to file is missing we still want to display info about lxd + if os.IsPermission(err) { + return true, fmt.Errorf("cannnot access %s to check if runing in LXD container: %w", initialProcessEnvironmentVariables, err) + } + return false, fmt.Errorf("cannnot read %s to check if runing in LXD container: %w", initialProcessEnvironmentVariables, err) + } + if strings.Contains(string(content), "container=lxc") { + return true, nil + } + return false, nil +} diff --git a/src/k8s/go.mod b/src/k8s/go.mod index 57164d98b..68d3f1cde 100644 --- a/src/k8s/go.mod +++ b/src/k8s/go.mod @@ -5,7 +5,7 @@ go 1.22.6 require ( dario.cat/mergo v1.0.0 github.com/canonical/go-dqlite v1.22.0 - github.com/canonical/k8s-snap-api v1.0.5 + github.com/canonical/k8s-snap-api v1.0.13 github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230 github.com/canonical/microcluster/v3 v3.0.0-20240827143335-f7a4d3984970 github.com/go-logr/logr v1.4.2 @@ -14,6 +14,7 @@ require ( github.com/onsi/gomega v1.32.0 github.com/pelletier/go-toml v1.9.5 github.com/spf13/cobra v1.8.1 + golang.org/x/mod v0.20.0 golang.org/x/net v0.28.0 golang.org/x/sync v0.8.0 golang.org/x/sys v0.24.0 diff --git a/src/k8s/go.sum b/src/k8s/go.sum index 0a695e62d..20f296a62 100644 --- a/src/k8s/go.sum +++ b/src/k8s/go.sum @@ -99,8 +99,8 @@ github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0 h1:nvj0OLI3YqYXe github.com/bugsnag/panicwrap v0.0.0-20151223152923-e2c28503fcd0/go.mod h1:D/8v3kj0zr8ZAKg1AQ6crr+5VwKN5eIywRkfhyM/+dE= github.com/canonical/go-dqlite v1.22.0 h1:DuJmfcREl4gkQJyvZzjl2GHFZROhbPyfdjDRQXpkOyw= github.com/canonical/go-dqlite v1.22.0/go.mod h1:Uvy943N8R4CFUAs59A1NVaziWY9nJ686lScY7ywurfg= -github.com/canonical/k8s-snap-api v1.0.5 h1:49bgi6CGtFjCPweeTz55Sv/waKgCl6ftx4BqXt3RI9k= -github.com/canonical/k8s-snap-api v1.0.5/go.mod h1:LDPoIYCeYnfgOFrwVPJ/4edGU264w7BB7g0GsVi36AY= +github.com/canonical/k8s-snap-api v1.0.13 h1:Z+IW6Knvycu+DrkmH+9qB1UNyYiHfL+rFvT9DtSO2+g= +github.com/canonical/k8s-snap-api v1.0.13/go.mod h1:LDPoIYCeYnfgOFrwVPJ/4edGU264w7BB7g0GsVi36AY= github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230 h1:YOqZ+/14OPZ+/TOXpRHIX3KLT0C+wZVpewKIwlGUmW0= github.com/canonical/lxd v0.0.0-20240822122218-e7b2a7a83230/go.mod h1:YVGI7HStOKsV+cMyXWnJ7RaMPaeWtrkxyIPvGWbgACc= github.com/canonical/microcluster/v3 v3.0.0-20240827143335-f7a4d3984970 h1:UrnpglbXELlxtufdk6DGDytu2JzyzuS3WTsOwPrkQLI= diff --git a/src/k8s/hack/env.sh b/src/k8s/hack/env.sh index 66af208c7..60d0d9865 100755 --- a/src/k8s/hack/env.sh +++ b/src/k8s/hack/env.sh @@ -2,7 +2,7 @@ ## Component repositories REPO_MUSL="https://git.launchpad.net/musl" -REPO_LIBTIRPC="https://salsa.debian.org/debian/libtirpc.git" +REPO_LIBTIRPC="https://git.launchpad.net/libtirpc" REPO_LIBNSL="https://github.com/thkukuk/libnsl.git" REPO_LIBUV="https://github.com/libuv/libuv.git" REPO_LIBLZ4="https://github.com/lz4/lz4.git" diff --git a/src/k8s/pkg/client/dqlite/remove_test.go b/src/k8s/pkg/client/dqlite/remove_test.go index a2f316a65..8252add83 100644 --- a/src/k8s/pkg/client/dqlite/remove_test.go +++ b/src/k8s/pkg/client/dqlite/remove_test.go @@ -16,21 +16,21 @@ func TestRemoveNodeByAddress(t *testing.T) { client, err := dqlite.NewClient(ctx, dqlite.ClientOpts{ ClusterYAML: filepath.Join(dirs[0], "cluster.yaml"), }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(client).NotTo(BeNil()) members, err := client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(2)) memberToRemove := members[0].Address if members[0].Role == dqlite.Voter { memberToRemove = members[1].Address } - g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove)).To(BeNil()) + g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove)).To(Succeed()) members, err = client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(1)) }) }) @@ -41,11 +41,11 @@ func TestRemoveNodeByAddress(t *testing.T) { client, err := dqlite.NewClient(ctx, dqlite.ClientOpts{ ClusterYAML: filepath.Join(dirs[0], "cluster.yaml"), }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(client).NotTo(BeNil()) members, err := client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(2)) memberToRemove := members[0] @@ -61,7 +61,7 @@ func TestRemoveNodeByAddress(t *testing.T) { g.Expect(client.RemoveNodeByAddress(ctx, memberToRemove.Address)).To(Succeed()) members, err = client.ListMembers(ctx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(members).To(HaveLen(1)) g.Expect(members[0].Role).To(Equal(dqlite.Voter)) g.Expect(members[0].Address).ToNot(Equal(memberToRemove.Address)) diff --git a/src/k8s/pkg/client/dqlite/util_test.go b/src/k8s/pkg/client/dqlite/util_test.go index b1d8084f1..83cea9d63 100644 --- a/src/k8s/pkg/client/dqlite/util_test.go +++ b/src/k8s/pkg/client/dqlite/util_test.go @@ -26,7 +26,7 @@ var nextDqlitePort = 37312 // }) // } // -// ``` +// ```. func withDqliteCluster(t *testing.T, size int, f func(ctx context.Context, dirs []string)) { ctx, cancel := context.WithCancel(context.Background()) defer cancel() diff --git a/src/k8s/pkg/client/helm/client.go b/src/k8s/pkg/client/helm/client.go index abe6c5da6..faf2ad031 100644 --- a/src/k8s/pkg/client/helm/client.go +++ b/src/k8s/pkg/client/helm/client.go @@ -4,6 +4,7 @@ import ( "bytes" "context" "encoding/json" + "errors" "fmt" "path/filepath" @@ -57,7 +58,7 @@ func (h *client) Apply(ctx context.Context, c InstallableChart, desired State, v get := action.NewGet(cfg) release, err := get.Run(c.Name) if err != nil { - if err != driver.ErrReleaseNotFound { + if !errors.Is(err, driver.ErrReleaseNotFound) { return false, fmt.Errorf("failed to get status of release %s: %w", c.Name, err) } isInstalled = false @@ -93,7 +94,7 @@ func (h *client) Apply(ctx context.Context, c InstallableChart, desired State, v // there is already a release installed, so we must run an upgrade action upgrade := action.NewUpgrade(cfg) upgrade.Namespace = c.Namespace - upgrade.ReuseValues = true + upgrade.ResetThenReuseValues = true chart, err := loader.Load(filepath.Join(h.manifestsBaseDir, c.ManifestPath)) if err != nil { diff --git a/src/k8s/pkg/client/helm/mock/mock.go b/src/k8s/pkg/client/helm/mock/mock.go index 6eea2d797..52a0f2f45 100644 --- a/src/k8s/pkg/client/helm/mock/mock.go +++ b/src/k8s/pkg/client/helm/mock/mock.go @@ -13,14 +13,14 @@ type MockApplyArguments struct { Values map[string]any } -// Mock is a mock implementation of helm.Client +// Mock is a mock implementation of helm.Client. type Mock struct { ApplyCalledWith []MockApplyArguments ApplyChanged bool ApplyErr error } -// Apply implements helm.Client +// Apply implements helm.Client. func (m *Mock) Apply(ctx context.Context, c helm.InstallableChart, desired helm.State, values map[string]any) (bool, error) { m.ApplyCalledWith = append(m.ApplyCalledWith, MockApplyArguments{Context: ctx, Chart: c, State: desired, Values: values}) return m.ApplyChanged, m.ApplyErr diff --git a/src/k8s/pkg/client/k8sd/mock/mock.go b/src/k8s/pkg/client/k8sd/mock/mock.go index d306e1a95..62915eb5b 100644 --- a/src/k8s/pkg/client/k8sd/mock/mock.go +++ b/src/k8s/pkg/client/k8sd/mock/mock.go @@ -56,14 +56,17 @@ func (m *Mock) BootstrapCluster(_ context.Context, request apiv1.BootstrapCluste m.BootstrapClusterCalledWith = request return m.BootstrapClusterResponse, m.BootstrapClusterErr } + func (m *Mock) GetJoinToken(_ context.Context, request apiv1.GetJoinTokenRequest) (apiv1.GetJoinTokenResponse, error) { m.GetJoinTokenCalledWith = request return m.GetJoinTokenResponse, m.GetJoinTokenErr } + func (m *Mock) JoinCluster(_ context.Context, request apiv1.JoinClusterRequest) error { m.JoinClusterCalledWith = request return m.JoinClusterErr } + func (m *Mock) RemoveNode(_ context.Context, request apiv1.RemoveNodeRequest) error { m.RemoveNodeCalledWith = request return m.RemoveNodeErr @@ -72,6 +75,7 @@ func (m *Mock) RemoveNode(_ context.Context, request apiv1.RemoveNodeRequest) er func (m *Mock) NodeStatus(_ context.Context) (apiv1.NodeStatusResponse, bool, error) { return m.NodeStatusResponse, m.NodeStatusInitialized, m.NodeStatusErr } + func (m *Mock) ClusterStatus(_ context.Context, waitReady bool) (apiv1.ClusterStatusResponse, error) { return m.ClusterStatusResponse, m.ClusterStatusErr } @@ -87,6 +91,7 @@ func (m *Mock) RefreshCertificatesRun(_ context.Context, request apiv1.RefreshCe func (m *Mock) GetClusterConfig(_ context.Context) (apiv1.GetClusterConfigResponse, error) { return m.GetClusterConfigResponse, m.GetClusterConfigErr } + func (m *Mock) SetClusterConfig(_ context.Context, request apiv1.SetClusterConfigRequest) error { m.SetClusterConfigCalledWith = request return m.SetClusterConfigErr diff --git a/src/k8s/pkg/client/kubernetes/configmap_test.go b/src/k8s/pkg/client/kubernetes/configmap_test.go index ea11d6aba..55b8ce713 100644 --- a/src/k8s/pkg/client/kubernetes/configmap_test.go +++ b/src/k8s/pkg/client/kubernetes/configmap_test.go @@ -79,7 +79,6 @@ func TestWatchConfigMap(t *testing.T) { case <-time.After(time.Second): t.Fatal("Timed out waiting for watch to complete") } - }) } } diff --git a/src/k8s/pkg/client/kubernetes/endpoints.go b/src/k8s/pkg/client/kubernetes/endpoints.go index 3d6709800..0840effb9 100644 --- a/src/k8s/pkg/client/kubernetes/endpoints.go +++ b/src/k8s/pkg/client/kubernetes/endpoints.go @@ -5,6 +5,7 @@ import ( "fmt" "sort" + "github.com/canonical/k8s/pkg/utils" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" "k8s.io/client-go/util/retry" @@ -40,7 +41,13 @@ func (c *Client) GetKubeAPIServerEndpoints(ctx context.Context) ([]string, error } for _, addr := range subset.Addresses { if addr.IP != "" { - addresses = append(addresses, fmt.Sprintf("%s:%d", addr.IP, portNumber)) + var address string + if utils.IsIPv4(addr.IP) { + address = addr.IP + } else { + address = fmt.Sprintf("[%s]", addr.IP) + } + addresses = append(addresses, fmt.Sprintf("%s:%d", address, portNumber)) } } } diff --git a/src/k8s/pkg/client/kubernetes/endpoints_test.go b/src/k8s/pkg/client/kubernetes/endpoints_test.go index 238886aa6..d750829a5 100644 --- a/src/k8s/pkg/client/kubernetes/endpoints_test.go +++ b/src/k8s/pkg/client/kubernetes/endpoints_test.go @@ -109,7 +109,7 @@ func TestGetKubeAPIServerEndpoints(t *testing.T) { g.Expect(err).To(HaveOccurred()) g.Expect(servers).To(BeEmpty()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(servers).To(Equal(tc.expectedAddresses)) } }) diff --git a/src/k8s/pkg/client/kubernetes/node_test.go b/src/k8s/pkg/client/kubernetes/node_test.go index 55165e3c4..22ab79a1b 100644 --- a/src/k8s/pkg/client/kubernetes/node_test.go +++ b/src/k8s/pkg/client/kubernetes/node_test.go @@ -28,7 +28,7 @@ func TestDeleteNode(t *testing.T) { }, metav1.CreateOptions{}) err := client.DeleteNode(context.Background(), nodeName) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("node does not exist is successful", func(t *testing.T) { @@ -37,7 +37,7 @@ func TestDeleteNode(t *testing.T) { nodeName := "test-node" err := client.DeleteNode(context.Background(), nodeName) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("node deletion fails", func(t *testing.T) { diff --git a/src/k8s/pkg/client/kubernetes/pods_test.go b/src/k8s/pkg/client/kubernetes/pods_test.go index 5f3922e1f..b10fe76c7 100644 --- a/src/k8s/pkg/client/kubernetes/pods_test.go +++ b/src/k8s/pkg/client/kubernetes/pods_test.go @@ -98,7 +98,7 @@ func TestCheckForReadyPods(t *testing.T) { err := client.CheckForReadyPods(context.Background(), tc.namespace, tc.listOptions) if tc.expectedError == "" { - g.Expect(err).Should(BeNil()) + g.Expect(err).ShouldNot(HaveOccurred()) } else { g.Expect(err).Should(MatchError(tc.expectedError)) } diff --git a/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go b/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go index 55ecc8061..3f88b6b21 100644 --- a/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go +++ b/src/k8s/pkg/client/kubernetes/restart_daemonset_test.go @@ -66,9 +66,9 @@ func TestRestartDaemonset(t *testing.T) { if tc.expectError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ds, err := client.AppsV1().DaemonSets("namespace").Get(context.Background(), "test", metav1.GetOptions{}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ds.Spec.Template.Annotations["kubectl.kubernetes.io/restartedAt"]).NotTo(BeEmpty()) } }) diff --git a/src/k8s/pkg/client/kubernetes/restart_deployment_test.go b/src/k8s/pkg/client/kubernetes/restart_deployment_test.go index 2bc7aa9d7..cdb59608d 100644 --- a/src/k8s/pkg/client/kubernetes/restart_deployment_test.go +++ b/src/k8s/pkg/client/kubernetes/restart_deployment_test.go @@ -66,9 +66,9 @@ func TestRestartDeployment(t *testing.T) { if tc.expectError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) deploy, err := client.AppsV1().Deployments("namespace").Get(context.Background(), "test", metav1.GetOptions{}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(deploy.Spec.Template.Annotations["kubectl.kubernetes.io/restartedAt"]).NotTo(BeEmpty()) } }) diff --git a/src/k8s/pkg/client/kubernetes/server_groups.go b/src/k8s/pkg/client/kubernetes/server_groups.go index cb58b9d0c..a3581d7a8 100644 --- a/src/k8s/pkg/client/kubernetes/server_groups.go +++ b/src/k8s/pkg/client/kubernetes/server_groups.go @@ -6,7 +6,7 @@ import ( v1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) -// ListResourcesForGroupVersion lists the resources for a given group version (e.g. "cilium.io/v2alpha1") +// ListResourcesForGroupVersion lists the resources for a given group version (e.g. "cilium.io/v2alpha1"). func (c *Client) ListResourcesForGroupVersion(groupVersion string) (*v1.APIResourceList, error) { resources, err := c.Discovery().ServerResourcesForGroupVersion(groupVersion) if err != nil { diff --git a/src/k8s/pkg/client/kubernetes/server_groups_test.go b/src/k8s/pkg/client/kubernetes/server_groups_test.go index 3126442a5..028f311e6 100644 --- a/src/k8s/pkg/client/kubernetes/server_groups_test.go +++ b/src/k8s/pkg/client/kubernetes/server_groups_test.go @@ -3,12 +3,11 @@ package kubernetes_test import ( "testing" - fakediscovery "k8s.io/client-go/discovery/fake" - fakeclientset "k8s.io/client-go/kubernetes/fake" - "github.com/canonical/k8s/pkg/client/kubernetes" . "github.com/onsi/gomega" v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + fakeclientset "k8s.io/client-go/kubernetes/fake" ) func TestListResourcesForGroupVersion(t *testing.T) { @@ -59,7 +58,7 @@ func TestListResourcesForGroupVersion(t *testing.T) { if tt.expectedError { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(resources).To(Equal(tt.expectedList)) } }) diff --git a/src/k8s/pkg/client/kubernetes/status.go b/src/k8s/pkg/client/kubernetes/status.go index e98f87778..c89646c43 100644 --- a/src/k8s/pkg/client/kubernetes/status.go +++ b/src/k8s/pkg/client/kubernetes/status.go @@ -34,9 +34,8 @@ func (c *Client) CheckKubernetesEndpoint(ctx context.Context) error { // HasReadyNodes returns true if there is at least one Ready node in the cluster, false otherwise. func (c *Client) HasReadyNodes(ctx context.Context) (bool, error) { nodes, err := c.CoreV1().Nodes().List(ctx, metav1.ListOptions{}) - if err != nil { - return false, fmt.Errorf("failed to list nodes: %v", err) + return false, fmt.Errorf("failed to list nodes: %w", err) } for _, node := range nodes.Items { diff --git a/src/k8s/pkg/client/kubernetes/status_test.go b/src/k8s/pkg/client/kubernetes/status_test.go index 7dae3e115..64b4bc063 100644 --- a/src/k8s/pkg/client/kubernetes/status_test.go +++ b/src/k8s/pkg/client/kubernetes/status_test.go @@ -93,7 +93,7 @@ func TestClusterHasReadyNodes(t *testing.T) { ready, err := client.HasReadyNodes(context.Background()) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ready).To(Equal(tt.expectedReady)) }) } diff --git a/src/k8s/pkg/client/snapd/refresh_status.go b/src/k8s/pkg/client/snapd/refresh_status.go index e2feb37a7..c6fa55581 100644 --- a/src/k8s/pkg/client/snapd/refresh_status.go +++ b/src/k8s/pkg/client/snapd/refresh_status.go @@ -17,15 +17,16 @@ func (c *Client) GetRefreshStatus(changeID string) (*types.RefreshStatus, error) if err != nil { return nil, fmt.Errorf("failed to get snapd change status: %w", err) } + defer resp.Body.Close() resBody, err := io.ReadAll(resp.Body) if err != nil { - return nil, fmt.Errorf("client: could not read response body: %s", err) + return nil, fmt.Errorf("client: could not read response body: %w", err) } var changeResponse snapdChangeResponse if err := json.Unmarshal(resBody, &changeResponse); err != nil { - return nil, fmt.Errorf("client: could not unmarshal response body: %s", err) + return nil, fmt.Errorf("client: could not unmarshal response body: %w", err) } return &changeResponse.Result, nil diff --git a/src/k8s/pkg/client/snapd/snap_info.go b/src/k8s/pkg/client/snapd/snap_info.go new file mode 100644 index 000000000..b09f61ec4 --- /dev/null +++ b/src/k8s/pkg/client/snapd/snap_info.go @@ -0,0 +1,41 @@ +package snapd + +import ( + "encoding/json" + "fmt" + "io" + "time" +) + +type SnapInfoResult struct { + InstallDate time.Time `json:"install-date"` +} + +type SnapInfoResponse struct { + StatusCode int `json:"status-code"` + Result SnapInfoResult `json:"result"` +} + +func (c *Client) GetSnapInfo(snap string) (*SnapInfoResponse, error) { + resp, err := c.client.Get(fmt.Sprintf("http://localhost/v2/snaps/%s", snap)) + if err != nil { + return nil, fmt.Errorf("failed to get snapd snap info: %w", err) + } + defer resp.Body.Close() + + resBody, err := io.ReadAll(resp.Body) + if err != nil { + return nil, fmt.Errorf("client: could not read response body: %w", err) + } + + var snapInfoResponse SnapInfoResponse + if err := json.Unmarshal(resBody, &snapInfoResponse); err != nil { + return nil, fmt.Errorf("client: could not unmarshal response body: %w", err) + } + + return &snapInfoResponse, nil +} + +func (s SnapInfoResponse) HasInstallDate() bool { + return !s.Result.InstallDate.IsZero() +} diff --git a/src/k8s/pkg/docgen/godoc.go b/src/k8s/pkg/docgen/godoc.go new file mode 100755 index 000000000..eaa50c572 --- /dev/null +++ b/src/k8s/pkg/docgen/godoc.go @@ -0,0 +1,130 @@ +package docgen + +import ( + "fmt" + "go/ast" + "go/doc" + "go/parser" + "go/token" + "reflect" +) + +var packageDocCache = make(map[string]*doc.Package) + +func findTypeSpec(decl *ast.GenDecl, symbol string) (*ast.TypeSpec, error) { + for _, spec := range decl.Specs { + typeSpec, ok := spec.(*ast.TypeSpec) + if !ok { + return nil, fmt.Errorf("spec is not *ast.TypeSpec") + } + if symbol == typeSpec.Name.Name { + return typeSpec, nil + } + } + return nil, nil +} + +func getStructTypeFromDoc(packageDoc *doc.Package, structName string) (*ast.StructType, error) { + for _, docType := range packageDoc.Types { + if structName != docType.Name { + continue + } + typeSpec, err := findTypeSpec(docType.Decl, docType.Name) + if err != nil { + return nil, fmt.Errorf("failed to find type spec: %w", err) + } + if typeSpec == nil { + continue + } + structType, ok := typeSpec.Type.(*ast.StructType) + if !ok { + // Not a structure. + continue + } + return structType, nil + } + return nil, nil +} + +func parsePackageDir(packageDir string) (*ast.Package, error) { + fset := token.NewFileSet() + packages, err := parser.ParseDir(fset, packageDir, nil, parser.ParseComments) + if err != nil { + return nil, fmt.Errorf("couldn't parse go package: %s", packageDir) + } + + if len(packages) == 0 { + return nil, fmt.Errorf("no go package found: %s", packageDir) + } + if len(packages) > 1 { + return nil, fmt.Errorf("multiple go package found: %s", packageDir) + } + + // We have a map containing a single entry and we need to return it. + for _, pkg := range packages { + return pkg, nil + } + + // shouldn't really get here. + return nil, fmt.Errorf("failed to parse go package") +} + +func getAstStructField(structType *ast.StructType, fieldName string) (*ast.Field, error) { + for _, field := range structType.Fields.List { + for _, fieldIdent := range field.Names { + if fieldIdent.Name == fieldName { + return field, nil + } + } + } + return nil, nil +} + +func getPackageDoc(packagePath string, projectDir string) (*doc.Package, error) { + packageDoc, found := packageDocCache[packagePath] + if found { + return packageDoc, nil + } + + packageDir, err := getGoPackageDir(packagePath, projectDir) + if err != nil { + return nil, fmt.Errorf("failed to retrieve package dir: %w", err) + } + + pkg, err := parsePackageDir(packageDir) + if err != nil { + return nil, fmt.Errorf("failed to parse package dir: %w", err) + } + + packageDoc = doc.New(pkg, packageDir, doc.AllDecls|doc.PreserveAST) + packageDocCache[packagePath] = packageDoc + + return packageDoc, nil +} + +func getFieldDocstring(i any, field reflect.StructField, projectDir string) (string, error) { + inType := reflect.TypeOf(i) + + packageDoc, err := getPackageDoc(inType.PkgPath(), projectDir) + if err != nil { + return "", fmt.Errorf("failed to retrieve package doc: %w", err) + } + + structType, err := getStructTypeFromDoc(packageDoc, inType.Name()) + if err != nil { + return "", fmt.Errorf("failed to retrieve struct type: %w", err) + } + if structType == nil { + return "", fmt.Errorf("could not find %s structure definition", inType.Name()) + } + + astField, err := getAstStructField(structType, field.Name) + if err != nil { + return "", fmt.Errorf("failed to retrieve struct field: %w", err) + } + if astField == nil { + return "", fmt.Errorf("could not find %s.%s field definition", inType.Name(), field.Name) + } + + return astField.Doc.Text(), nil +} diff --git a/src/k8s/pkg/docgen/gomod.go b/src/k8s/pkg/docgen/gomod.go new file mode 100755 index 000000000..f51ec2e52 --- /dev/null +++ b/src/k8s/pkg/docgen/gomod.go @@ -0,0 +1,100 @@ +package docgen + +import ( + "fmt" + "os" + "path" + "strings" + + "golang.org/x/mod/modfile" + "golang.org/x/mod/module" +) + +func getGoDepModulePath(name string, version string) (string, error) { + cachePath := os.Getenv("GOMODCACHE") + if cachePath == "" { + goPath := os.Getenv("GOPATH") + if goPath == "" { + goPath = path.Join(os.Getenv("HOME"), "/go") + } + cachePath = path.Join(goPath, "pkg", "mod") + } + + escapedPath, err := module.EscapePath(name) + if err != nil { + return "", fmt.Errorf( + "couldn't escape module path %s: %w", name, err) + } + + escapedVersion, err := module.EscapeVersion(version) + if err != nil { + return "", fmt.Errorf( + "couldn't escape module version %s: %w", version, err) + } + + path := path.Join(cachePath, escapedPath+"@"+escapedVersion) + + // Validate the path. + if _, err := os.Stat(path); err != nil { + return "", fmt.Errorf( + "go module path not accessible %s %s %s: %w", + name, version, path, err) + } + + return path, nil +} + +func getDependencyVersionFromGoMod(goModPath string, packageName string, directOnly bool) (string, string, error) { + goModContents, err := os.ReadFile(goModPath) + if err != nil { + return "", "", fmt.Errorf("could not read go.mod file %s: %w", goModPath, err) + } + goModFile, err := modfile.ParseLax(goModPath, goModContents, nil) + if err != nil { + return "", "", fmt.Errorf("could not parse go.mod file %s: %w", goModPath, err) + } + + for _, dep := range goModFile.Require { + if directOnly && dep.Indirect { + continue + } + if strings.HasPrefix(packageName, dep.Mod.Path) { + return dep.Mod.Path, dep.Mod.Version, nil + } + } + + return "", "", fmt.Errorf("could not find dependency %s in %s", packageName, goModPath) +} + +func getGoModPath(projectDir string) (string, error) { + return path.Join(projectDir, "go.mod"), nil +} + +func getGoPackageDir(packageName string, projectDir string) (string, error) { + if packageName == "" { + return "", fmt.Errorf("could not retrieve package dir, no package name specified.") + } + + if strings.HasPrefix(packageName, "github.com/canonical/k8s/") { + return strings.Replace(packageName, "github.com/canonical/k8s", projectDir, 1), nil + } + + // Dependency, need to retrieve its version from go.mod. + goModPath, err := getGoModPath(projectDir) + if err != nil { + return "", err + } + + basePackageName, version, err := getDependencyVersionFromGoMod(goModPath, packageName, false) + if err != nil { + return "", err + } + + basePath, err := getGoDepModulePath(basePackageName, version) + if err != nil { + return "", err + } + + subPath := strings.TrimPrefix(packageName, basePackageName) + return path.Join(basePath, subPath), nil +} diff --git a/src/k8s/pkg/docgen/json_struct.go b/src/k8s/pkg/docgen/json_struct.go new file mode 100644 index 000000000..5dc5e5a67 --- /dev/null +++ b/src/k8s/pkg/docgen/json_struct.go @@ -0,0 +1,139 @@ +package docgen + +import ( + "fmt" + "os" + "reflect" + "strings" + + "github.com/canonical/k8s/pkg/utils" +) + +type JsonTag struct { + Name string + Options []string +} + +type Field struct { + Name string + TypeName string + JsonTag JsonTag + FullJsonPath string + Docstring string +} + +// Generate Markdown documentation for a JSON or YAML based on +// the Go structure definition, parsing field annotations. +func MarkdownFromJsonStruct(i any, projectDir string) (string, error) { + fields, err := ParseStruct(i, projectDir) + if err != nil { + return "", err + } + + entryTemplate := `### %s +**Type:** ` + "`%s`" + `
+ +%s +` + + var out strings.Builder + for _, field := range fields { + outFieldType := strings.ReplaceAll(field.TypeName, "*", "") + entry := fmt.Sprintf(entryTemplate, field.FullJsonPath, outFieldType, field.Docstring) + out.WriteString(entry) + } + + return out.String(), nil +} + +// Generate Markdown documentation for a JSON or YAML based on +// the Go structure definition, parsing field annotations. +// Write the output to the specified file path. +// The project dir is used to parse the source code and identify dependencies +// based on the go.mod file. +func MarkdownFromJsonStructToFile(i any, outFilePath string, projectDir string) error { + content, err := MarkdownFromJsonStruct(i, projectDir) + if err != nil { + return err + } + + err = utils.WriteFile(outFilePath, []byte(content), 0o644) + if err != nil { + return fmt.Errorf("failed to write markdown documentation to %s: %w", outFilePath, err) + } + return nil +} + +func getJsonTag(field reflect.StructField) JsonTag { + jsonTag := JsonTag{} + + jsonTagStr := field.Tag.Get("json") + if jsonTagStr == "" { + // Use yaml tags as fallback, which have the same format. + jsonTagStr = field.Tag.Get("yaml") + } + if jsonTagStr != "" { + jsonTagSlice := strings.Split(jsonTagStr, ",") + if len(jsonTagSlice) > 0 { + jsonTag.Name = jsonTagSlice[0] + } + if len(jsonTagSlice) > 1 { + jsonTag.Options = jsonTagSlice[1:] + } + } + + return jsonTag +} + +func ParseStruct(i any, projectDir string) ([]Field, error) { + inType := reflect.TypeOf(i) + + if inType.Kind() != reflect.Struct { + return nil, fmt.Errorf("structure parsing failed, not a structure: %s", inType.Name()) + } + + outFields := []Field{} + fields := reflect.VisibleFields(inType) + for _, field := range fields { + jsonTag := getJsonTag(field) + docstring, err := getFieldDocstring(i, field, projectDir) + if err != nil { + fmt.Fprintf(os.Stderr, "WARNING: could not retrieve field docstring: %s.%s, error: %v", + inType.Name(), field.Name, err) + } + + if field.Type.Kind() == reflect.Struct { + fieldIface := reflect.ValueOf(i).FieldByName(field.Name).Interface() + nestedFields, err := ParseStruct(fieldIface, projectDir) + if err != nil { + return nil, fmt.Errorf("couldn't parse %s.%s: %w", inType, field.Name, err) + } + + outField := Field{ + Name: field.Name, + TypeName: "object", + JsonTag: jsonTag, + FullJsonPath: jsonTag.Name, + Docstring: docstring, + } + outFields = append(outFields, outField) + + for _, nestedField := range nestedFields { + // Update the json paths of the nested fields based on the field name. + nestedField.FullJsonPath = jsonTag.Name + "." + nestedField.FullJsonPath + outFields = append(outFields, nestedField) + } + } else { + outField := Field{ + Name: field.Name, + TypeName: field.Type.String(), + JsonTag: jsonTag, + FullJsonPath: jsonTag.Name, + Docstring: docstring, + } + outFields = append(outFields, outField) + } + } + + return outFields, nil +} diff --git a/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go b/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go new file mode 100644 index 000000000..51ed19c75 --- /dev/null +++ b/src/k8s/pkg/k8sd/api/capi_certificate_refresh.go @@ -0,0 +1,87 @@ +package api + +import ( + "fmt" + "net/http" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" + "github.com/canonical/k8s/pkg/utils" + "github.com/canonical/lxd/lxd/response" + "github.com/canonical/microcluster/v3/state" + "golang.org/x/sync/errgroup" + certv1 "k8s.io/api/certificates/v1" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" +) + +// postApproveWorkerCSR approves the worker node CSR for the specified seed. +// The certificate approval process follows these steps: +// 1. The CAPI provider calls the /x/capi/refresh-certs/plan endpoint from a +// worker node, which generates a CSR and creates a CertificateSigningRequest +// object in the cluster. +// 2. The CAPI provider then calls the /k8sd/refresh-certs/run endpoint with +// the seed. This endpoint waits until the CSR is approved and the certificate +// is signed. Note that this is a blocking call. +// 3. The CAPI provider calls the /x/capi/refresh-certs/approve endpoint from +// any control plane node to approve the CSR. +// 4. The /x/capi/refresh-certs/run endpoint completes and returns once the +// certificate is approved and signed. +func (e *Endpoints) postApproveWorkerCSR(s state.State, r *http.Request) response.Response { + snap := e.provider.Snap() + + req := apiv1.ClusterAPIApproveWorkerCSRRequest{} + + if err := utils.NewStrictJSONDecoder(r.Body).Decode(&req); err != nil { + return response.BadRequest(fmt.Errorf("failed to parse request: %w", err)) + } + + if err := r.Body.Close(); err != nil { + return response.InternalError(fmt.Errorf("failed to close request body: %w", err)) + } + + client, err := snap.KubernetesClient("") + if err != nil { + return response.InternalError(fmt.Errorf("failed to get Kubernetes client: %w", err)) + } + + g, ctx := errgroup.WithContext(r.Context()) + + // CSR names + csrNames := []string{ + fmt.Sprintf("k8sd-%d-worker-kubelet-serving", req.Seed), + fmt.Sprintf("k8sd-%d-worker-kubelet-client", req.Seed), + fmt.Sprintf("k8sd-%d-worker-kube-proxy-client", req.Seed), + } + + for _, csrName := range csrNames { + g.Go(func() error { + if err := client.WatchCertificateSigningRequest( + ctx, + csrName, + func(request *certv1.CertificateSigningRequest) (bool, error) { + request.Status.Conditions = append(request.Status.Conditions, certv1.CertificateSigningRequestCondition{ + Type: certv1.CertificateApproved, + Status: corev1.ConditionTrue, + Reason: "ApprovedByCK8sCAPI", + Message: "This CSR was approved by the Canonical Kubernetes CAPI Provider", + LastUpdateTime: metav1.Now(), + }) + _, err := client.CertificatesV1().CertificateSigningRequests().UpdateApproval(ctx, csrName, request, metav1.UpdateOptions{}) + if err != nil { + return false, fmt.Errorf("failed to update CSR %s: %w", csrName, err) + } + return true, nil + }, + ); err != nil { + return fmt.Errorf("certificate signing request failed: %w", err) + } + return nil + }) + } + + if err := g.Wait(); err != nil { + return response.InternalError(fmt.Errorf("failed to approve worker node CSR: %w", err)) + } + + return response.SyncResponse(true, apiv1.ClusterAPIApproveWorkerCSRResponse{}) +} diff --git a/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go b/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go new file mode 100644 index 000000000..c6bc96ac8 --- /dev/null +++ b/src/k8s/pkg/k8sd/api/capi_certificates_expiry.go @@ -0,0 +1,50 @@ +package api + +import ( + "fmt" + "net/http" + "time" + + apiv1 "github.com/canonical/k8s-snap-api/api/v1" + databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" + "github.com/canonical/lxd/lxd/response" + "github.com/canonical/microcluster/v3/state" +) + +func (e *Endpoints) postCertificatesExpiry(s state.State, r *http.Request) response.Response { + config, err := databaseutil.GetClusterConfig(r.Context(), s) + if err != nil { + return response.InternalError(fmt.Errorf("failed to get cluster config: %w", err)) + } + + certificates := []string{ + config.Certificates.GetCACert(), + config.Certificates.GetClientCACert(), + config.Certificates.GetAdminClientCert(), + config.Certificates.GetAPIServerKubeletClientCert(), + config.Certificates.GetFrontProxyCACert(), + } + + var earliestExpiry time.Time + // Find the earliest expiry certificate + // They should all be about the same but better double-check this. + for _, cert := range certificates { + if cert == "" { + continue + } + + cert, _, err := pkiutil.LoadCertificate(cert, "") + if err != nil { + return response.InternalError(fmt.Errorf("failed to load certificate: %w", err)) + } + + if earliestExpiry.IsZero() || cert.NotAfter.Before(earliestExpiry) { + earliestExpiry = cert.NotAfter + } + } + + return response.SyncResponse(true, &apiv1.CertificatesExpiryResponse{ + ExpiryDate: earliestExpiry.Format(time.RFC3339), + }) +} diff --git a/src/k8s/pkg/k8sd/api/certificates_refresh.go b/src/k8s/pkg/k8sd/api/certificates_refresh.go index 2d8153dc1..c94f33433 100644 --- a/src/k8s/pkg/k8sd/api/certificates_refresh.go +++ b/src/k8s/pkg/k8sd/api/certificates_refresh.go @@ -1,10 +1,15 @@ package api import ( + "context" + "crypto/rand" + "crypto/rsa" + "crypto/sha256" "crypto/x509/pkix" + "encoding/base64" "fmt" "math" - "math/rand" + "math/big" "net" "net/http" "path/filepath" @@ -28,7 +33,11 @@ import ( ) func (e *Endpoints) postRefreshCertsPlan(s state.State, r *http.Request) response.Response { - seed := rand.Intn(math.MaxInt) + seedBigInt, err := rand.Int(rand.Reader, big.NewInt(math.MaxInt)) + if err != nil { + return response.InternalError(fmt.Errorf("failed to generate seed: %w", err)) + } + seed := int(seedBigInt.Int64()) snap := e.provider.Snap() isWorker, err := snaputil.IsWorker(snap) @@ -49,7 +58,6 @@ func (e *Endpoints) postRefreshCertsPlan(s state.State, r *http.Request) respons return response.SyncResponse(true, apiv1.RefreshCertificatesPlanResponse{ Seed: seed, }) - } func (e *Endpoints) postRefreshCertsRun(s state.State, r *http.Request) response.Response { @@ -66,6 +74,8 @@ func (e *Endpoints) postRefreshCertsRun(s state.State, r *http.Request) response // refreshCertsRunControlPlane refreshes the certificates for a control plane node. func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) response.Response { + log := log.FromContext(r.Context()) + req := apiv1.RefreshCertificatesRunRequest{} if err := utils.NewStrictJSONDecoder(r.Body).Decode(&req); err != nil { return response.BadRequest(fmt.Errorf("failed to parse request: %w", err)) @@ -81,6 +91,13 @@ func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) return response.InternalError(fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname())) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + serviceIPs, err := utils.GetKubernetesServiceIPsFromServiceCIDRs(clusterConfig.Network.GetServiceCIDR()) if err != nil { return response.InternalError(fmt.Errorf("failed to get IP address(es) from ServiceCIDR %q: %w", clusterConfig.Network.GetServiceCIDR(), err)) @@ -119,27 +136,67 @@ func refreshCertsRunControlPlane(s state.State, r *http.Request, snap snap.Snap) return response.InternalError(fmt.Errorf("failed to write control plane certificates: %w", err)) } - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), clusterConfig.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, clusterConfig.APIServer.GetSecurePort(), *certificates); err != nil { return response.InternalError(fmt.Errorf("failed to generate control plane kubeconfigs: %w", err)) } - if err := snaputil.RestartControlPlaneServices(r.Context(), snap); err != nil { - return response.InternalError(fmt.Errorf("failed to restart control plane services: %w", err)) - } + // NOTE: Restart the control plane services in a separate goroutine to avoid + // restarting the API server, which would break the k8sd proxy connection + // and cause missed responses in the proxy side. + readyCh := make(chan error) + go func() { + // NOTE: Create a new context independent of the request context to ensure + // the restart process is not cancelled by the client. + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + select { + case err := <-readyCh: + if err != nil { + log.Error(err, "Failed to refresh certificates") + return + } + case <-ctx.Done(): + log.Error(ctx.Err(), "Timeout waiting for certificates to be refreshed") + return + } - kubeletCert, _, err := pkiutil.LoadCertificate(certificates.KubeletCert, "") + if err := snaputil.RestartControlPlaneServices(ctx, snap); err != nil { + log.Error(err, "Failed to restart control plane services") + } + }() + + apiServerCert, _, err := pkiutil.LoadCertificate(certificates.APIServerCert, "") if err != nil { return response.InternalError(fmt.Errorf("failed to read kubelet certificate: %w", err)) } - expirationTimeUNIX := kubeletCert.NotAfter.Unix() + expirationTimeUNIX := apiServerCert.NotAfter.Unix() + + return response.ManualResponse(func(w http.ResponseWriter) (rerr error) { + defer func() { + readyCh <- rerr + close(readyCh) + }() - return response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ - ExpirationSeconds: int(expirationTimeUNIX), + err := response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ + ExpirationSeconds: int(expirationTimeUNIX), + }).Render(w) + if err != nil { + return fmt.Errorf("failed to render response: %w", err) + } + + f, ok := w.(http.Flusher) + if !ok { + return fmt.Errorf("ResponseWriter is not type http.Flusher") + } + + f.Flush() + return nil }) } -// refreshCertsRunWorker refreshes the certificates for a worker node +// refreshCertsRunWorker refreshes the certificates for a worker node. func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) response.Response { log := log.FromContext(r.Context()) @@ -167,6 +224,18 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo certificates.CACert = clusterConfig.Certificates.GetCACert() certificates.ClientCACert = clusterConfig.Certificates.GetClientCACert() + k8sdPublicKey, err := pkiutil.LoadRSAPublicKey(clusterConfig.Certificates.GetK8sdPublicKey()) + if err != nil { + return response.InternalError(fmt.Errorf("failed to load k8sd public key, error: %w", err)) + } + + hostnames := []string{snap.Hostname()} + ips := []net.IP{net.ParseIP(s.Address().Hostname())} + + extraIPs, extraNames := utils.SplitIPAndDNSSANs(req.ExtraSANs) + hostnames = append(hostnames, extraNames...) + ips = append(ips, extraIPs...) + g, ctx := errgroup.WithContext(r.Context()) for _, csr := range []struct { @@ -185,8 +254,8 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo commonName: fmt.Sprintf("system:node:%s", snap.Hostname()), organization: []string{"system:nodes"}, usages: []certv1.KeyUsage{certv1.UsageDigitalSignature, certv1.UsageKeyEncipherment, certv1.UsageServerAuth}, - hostnames: []string{snap.Hostname()}, - ips: []net.IP{net.ParseIP(s.Address().Hostname())}, + hostnames: hostnames, + ips: ips, signerName: "k8sd.io/kubelet-serving", certificate: &certificates.KubeletCert, key: &certificates.KubeletKey, @@ -209,7 +278,6 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo key: &certificates.KubeProxyClientKey, }, } { - csr := csr g.Go(func() error { csrPEM, keyPEM, err := pkiutil.GenerateCSR( pkix.Name{ @@ -224,14 +292,34 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo return fmt.Errorf("failed to generate CSR for %s: %w", csr.name, err) } + // Obtain the SHA256 sum of the CSR request. + hash := sha256.New() + _, err = hash.Write([]byte(csrPEM)) + if err != nil { + return fmt.Errorf("failed to checksum CSR %s, err: %w", csr.name, err) + } + + signature, err := rsa.EncryptPKCS1v15(rand.Reader, k8sdPublicKey, hash.Sum(nil)) + if err != nil { + return fmt.Errorf("failed to sign CSR %s, err: %w", csr.name, err) + } + signatureB64 := base64.StdEncoding.EncodeToString(signature) + + expirationSeconds := int32(req.ExpirationSeconds) + if _, err = client.CertificatesV1().CertificateSigningRequests().Create(ctx, &certv1.CertificateSigningRequest{ ObjectMeta: metav1.ObjectMeta{ Name: csr.name, + Annotations: map[string]string{ + "k8sd.io/signature": signatureB64, + "k8sd.io/node": snap.Hostname(), + }, }, Spec: certv1.CertificateSigningRequestSpec{ - Request: []byte(csrPEM), - Usages: csr.usages, - SignerName: csr.signerName, + Request: []byte(csrPEM), + ExpirationSeconds: &expirationSeconds, + Usages: csr.usages, + SignerName: csr.signerName, }, }, metav1.CreateOptions{}); err != nil { return fmt.Errorf("failed to create CSR for %s: %w", csr.name, err) @@ -249,9 +337,7 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo } return nil - }) - } if err := g.Wait(); err != nil { @@ -262,21 +348,54 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo return response.InternalError(fmt.Errorf("failed to write worker PKI: %w", err)) } + nodeIP := net.ParseIP(s.Address().Hostname()) + if nodeIP == nil { + return response.InternalError(fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname())) + } + + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Kubeconfigs - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), fmt.Sprintf("%s:%d", localhostAddress, clusterConfig.APIServer.GetSecurePort()), certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { return response.InternalError(fmt.Errorf("failed to generate kubelet kubeconfig: %w", err)) } - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), fmt.Sprintf("%s:%d", localhostAddress, clusterConfig.APIServer.GetSecurePort()), certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { return response.InternalError(fmt.Errorf("failed to generate kube-proxy kubeconfig: %w", err)) } - // Restart the services - if err := snap.RestartService(r.Context(), "kubelet"); err != nil { - return response.InternalError(fmt.Errorf("failed to restart kubelet: %w", err)) - } - if err := snap.RestartService(r.Context(), "kube-proxy"); err != nil { - return response.InternalError(fmt.Errorf("failed to restart kube-proxy: %w", err)) - } + // NOTE: Restart the worker services in a separate goroutine to avoid + // restarting the kube-proxy and kubelet, which would break the + // proxy connection and cause missed responses in the proxy side. + readyCh := make(chan error, 1) + go func() { + // NOTE: Create a new context independent of the request context to ensure + // the restart process is not cancelled by the client. + ctx, cancel := context.WithTimeout(context.Background(), 5*time.Minute) + defer cancel() + + select { + case err := <-readyCh: + if err != nil { + log.Error(err, "Failed to refresh certificates") + return + } + case <-ctx.Done(): + log.Error(ctx.Err(), "Timeout waiting for certificates to be refreshed") + return + } + + if err := snap.RestartService(ctx, "kubelet"); err != nil { + log.Error(err, "Failed to restart kubelet") + } + if err := snap.RestartService(ctx, "kube-proxy"); err != nil { + log.Error(err, "Failed to restart kube-proxy") + } + }() cert, _, err := pkiutil.LoadCertificate(certificates.KubeletCert, "") if err != nil { @@ -284,10 +403,27 @@ func refreshCertsRunWorker(s state.State, r *http.Request, snap snap.Snap) respo } expirationTimeUNIX := cert.NotAfter.Unix() - return response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ - ExpirationSeconds: int(expirationTimeUNIX), - }) + return response.ManualResponse(func(w http.ResponseWriter) (rerr error) { + defer func() { + readyCh <- rerr + close(readyCh) + }() + err := response.SyncResponse(true, apiv1.RefreshCertificatesRunResponse{ + ExpirationSeconds: int(expirationTimeUNIX), + }).Render(w) + if err != nil { + return fmt.Errorf("failed to render response: %w", err) + } + + f, ok := w.(http.Flusher) + if !ok { + return fmt.Errorf("ResponseWriter is not type http.Flusher") + } + + f.Flush() + return nil + }) } // isCertificateSigningRequestApprovedAndIssued checks if the certificate @@ -298,7 +434,6 @@ func isCertificateSigningRequestApprovedAndIssued(csr *certv1.CertificateSigning for _, condition := range csr.Status.Conditions { if condition.Type == certv1.CertificateApproved && condition.Status == corev1.ConditionTrue { return len(csr.Status.Certificate) > 0, nil - } if condition.Type == certv1.CertificateDenied && condition.Status == corev1.ConditionTrue { return false, fmt.Errorf("CSR %s was denied: %s", csr.Name, condition.Reason) diff --git a/src/k8s/pkg/k8sd/api/cluster_remove.go b/src/k8s/pkg/k8sd/api/cluster_remove.go index 6cbc22e62..731389683 100644 --- a/src/k8s/pkg/k8sd/api/cluster_remove.go +++ b/src/k8s/pkg/k8sd/api/cluster_remove.go @@ -8,6 +8,7 @@ import ( "time" apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" "github.com/canonical/k8s/pkg/log" "github.com/canonical/k8s/pkg/utils" @@ -87,7 +88,7 @@ func (e *Endpoints) postClusterRemove(s state.State, r *http.Request) response.R return response.InternalError(fmt.Errorf("failed to get cluster config: %w", err)) } - if _, ok := cfg.Annotations[apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove]; ok { + if _, ok := cfg.Annotations[apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove]; ok { // Explicitly skip removing the node from Kubernetes. log.Info("Skipping Kubernetes worker node removal") return response.SyncResponse(true, nil) diff --git a/src/k8s/pkg/k8sd/api/cluster_tokens.go b/src/k8s/pkg/k8sd/api/cluster_tokens.go index d685ce1b9..1562f4b23 100644 --- a/src/k8s/pkg/k8sd/api/cluster_tokens.go +++ b/src/k8s/pkg/k8sd/api/cluster_tokens.go @@ -10,6 +10,7 @@ import ( apiv1 "github.com/canonical/k8s-snap-api/api/v1" "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/log" "github.com/canonical/k8s/pkg/utils" "github.com/canonical/lxd/lxd/response" "github.com/canonical/microcluster/v3/microcluster" @@ -48,17 +49,19 @@ func (e *Endpoints) postClusterJoinTokens(s state.State, r *http.Request) respon } func getOrCreateJoinToken(ctx context.Context, m *microcluster.MicroCluster, tokenName string, ttl time.Duration) (string, error) { + log := log.FromContext(ctx) + // grab token if it exists and return it records, err := m.ListJoinTokens(ctx) if err != nil { - fmt.Println("Failed to get existing tokens. Trying to create a new token.") + log.V(1).Info("Failed to get existing tokens. Trying to create a new token.") } else { for _, record := range records { if record.Name == tokenName { return record.Token, nil } } - fmt.Println("No token exists yet. Creating a new token.") + log.V(1).Info("No token exists yet. Creating a new token.") } token, err := m.NewJoinToken(ctx, tokenName, ttl) diff --git a/src/k8s/pkg/k8sd/api/endpoints.go b/src/k8s/pkg/k8sd/api/endpoints.go index e7e7af5d0..4ae9946e3 100644 --- a/src/k8s/pkg/k8sd/api/endpoints.go +++ b/src/k8s/pkg/k8sd/api/endpoints.go @@ -85,18 +85,18 @@ func (e *Endpoints) Endpoints() []rest.Endpoint { Post: rest.EndpointAction{ Handler: e.postWorkerInfo, AllowUntrusted: true, - AccessHandler: ValidateWorkerInfoAccessHandler("worker-name", "worker-token"), + AccessHandler: ValidateWorkerInfoAccessHandler("Worker-Name", "Worker-Token"), }, }, // Certificates { Name: "RefreshCerts/Plan", - Path: "k8sd/refresh-certs/plan", + Path: apiv1.RefreshCertificatesPlanRPC, Post: rest.EndpointAction{Handler: e.postRefreshCertsPlan}, }, { Name: "RefreshCerts/Run", - Path: "k8sd/refresh-certs/run", + Path: apiv1.RefreshCertificatesRunRPC, Post: rest.EndpointAction{Handler: e.postRefreshCertsRun}, }, // Kubeconfig @@ -140,6 +140,26 @@ func (e *Endpoints) Endpoints() []rest.Endpoint { Path: apiv1.ClusterAPIRemoveNodeRPC, Post: rest.EndpointAction{Handler: e.postClusterRemove, AccessHandler: ValidateCAPIAuthTokenAccessHandler("capi-auth-token"), AllowUntrusted: true}, }, + { + Name: "ClusterAPI/CertificatesExpiry", + Path: apiv1.ClusterAPICertificatesExpiryRPC, + Post: rest.EndpointAction{Handler: e.postCertificatesExpiry, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Plan", + Path: apiv1.ClusterAPICertificatesPlanRPC, + Post: rest.EndpointAction{Handler: e.postRefreshCertsPlan, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Run", + Path: apiv1.ClusterAPICertificatesRunRPC, + Post: rest.EndpointAction{Handler: e.postRefreshCertsRun, AccessHandler: e.ValidateNodeTokenAccessHandler("node-token"), AllowUntrusted: true}, + }, + { + Name: "ClusterAPI/RefreshCerts/Approve", + Path: apiv1.ClusterAPIApproveWorkerCSRRPC, + Post: rest.EndpointAction{Handler: e.postApproveWorkerCSR, AccessHandler: ValidateCAPIAuthTokenAccessHandler("capi-auth-token"), AllowUntrusted: true}, + }, // Snap refreshes { Name: "Snap/Refresh", diff --git a/src/k8s/pkg/k8sd/api/impl/k8sd.go b/src/k8s/pkg/k8sd/api/impl/k8sd.go index bc8eabdb5..e062a12f5 100644 --- a/src/k8s/pkg/k8sd/api/impl/k8sd.go +++ b/src/k8s/pkg/k8sd/api/impl/k8sd.go @@ -59,5 +59,4 @@ func GetLocalNodeStatus(ctx context.Context, s state.State, snap snap.Snap) (api Address: s.Address().Hostname(), ClusterRole: clusterRole, }, nil - } diff --git a/src/k8s/pkg/k8sd/api/response.go b/src/k8s/pkg/k8sd/api/response.go index 626aadfa8..01c94e373 100644 --- a/src/k8s/pkg/k8sd/api/response.go +++ b/src/k8s/pkg/k8sd/api/response.go @@ -5,9 +5,9 @@ import ( ) const ( - // StatusNodeUnavailable is the Http status code that the API returns if the node isn't in the cluster + // StatusNodeUnavailable is the Http status code that the API returns if the node isn't in the cluster. StatusNodeUnavailable = 520 - // StatusNodeInUse is the Http status code that the API returns if the node is already in the cluster + // StatusNodeInUse is the Http status code that the API returns if the node is already in the cluster. StatusNodeInUse = 521 ) diff --git a/src/k8s/pkg/k8sd/api/worker.go b/src/k8s/pkg/k8sd/api/worker.go index efd750598..d030ba31b 100644 --- a/src/k8s/pkg/k8sd/api/worker.go +++ b/src/k8s/pkg/k8sd/api/worker.go @@ -26,7 +26,7 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp } // Existence of this header is already checked in the access handler. - workerName := r.Header.Get("worker-name") + workerName := r.Header.Get("Worker-Name") nodeIP := net.ParseIP(req.Address) if nodeIP == nil { return response.BadRequest(fmt.Errorf("failed to parse node IP address %s", req.Address)) @@ -63,7 +63,7 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp return response.InternalError(fmt.Errorf("failed to retrieve list of known kube-apiserver endpoints: %w", err)) } - workerToken := r.Header.Get("worker-token") + workerToken := r.Header.Get("Worker-Token") if err := s.Database().Transaction(r.Context(), func(ctx context.Context, tx *sql.Tx) error { return database.DeleteWorkerNodeToken(ctx, tx, workerToken) }); err != nil { @@ -86,5 +86,6 @@ func (e *Endpoints) postWorkerInfo(s state.State, r *http.Request) response.Resp KubeProxyClientCert: workerCertificates.KubeProxyClientCert, KubeProxyClientKey: workerCertificates.KubeProxyClientKey, K8sdPublicKey: cfg.Certificates.GetK8sdPublicKey(), + Annotations: cfg.Annotations, }) } diff --git a/src/k8s/pkg/k8sd/app/app.go b/src/k8s/pkg/k8sd/app/app.go index d6172d124..720401e3f 100644 --- a/src/k8s/pkg/k8sd/app/app.go +++ b/src/k8s/pkg/k8sd/app/app.go @@ -236,7 +236,7 @@ func (a *App) Run(ctx context.Context, customHooks *state.Hooks) error { // markNodeReady will decrement the readyWg counter to signal that the node is ready. // The node is ready if: // - the microcluster database is accessible -// - the kubernetes endpoint is reachable +// - the kubernetes endpoint is reachable. func (a *App) markNodeReady(ctx context.Context, s state.State) error { log := log.FromContext(ctx).WithValues("startup", "waitForReady") diff --git a/src/k8s/pkg/k8sd/app/cluster_util.go b/src/k8s/pkg/k8sd/app/cluster_util.go index 9255be3e4..46b23d366 100644 --- a/src/k8s/pkg/k8sd/app/cluster_util.go +++ b/src/k8s/pkg/k8sd/app/cluster_util.go @@ -3,10 +3,12 @@ package app import ( "context" "fmt" + "net" "github.com/canonical/k8s/pkg/k8sd/setup" "github.com/canonical/k8s/pkg/snap" snaputil "github.com/canonical/k8s/pkg/snap/util" + mctypes "github.com/canonical/microcluster/v3/rest/types" ) func startControlPlaneServices(ctx context.Context, snap snap.Snap, datastore string) error { @@ -58,3 +60,22 @@ func waitApiServerReady(ctx context.Context, snap snap.Snap) error { return nil } + +func DetermineLocalhostAddress(clusterMembers []mctypes.ClusterMember) (string, error) { + // Check if any of the cluster members have an IPv6 address, if so return "::1" + // if one member has an IPv6 address, other members should also have IPv6 interfaces + for _, clusterMember := range clusterMembers { + memberAddress := clusterMember.Address.Addr().String() + nodeIP := net.ParseIP(memberAddress) + if nodeIP == nil { + return "", fmt.Errorf("failed to parse node IP address %q", memberAddress) + } + + if nodeIP.To4() == nil { + return "[::1]", nil + } + } + + // If no IPv6 addresses are found this means the cluster is IPv4 only + return "127.0.0.1", nil +} diff --git a/src/k8s/pkg/k8sd/app/cluster_util_test.go b/src/k8s/pkg/k8sd/app/cluster_util_test.go new file mode 100644 index 000000000..acd6b5e1e --- /dev/null +++ b/src/k8s/pkg/k8sd/app/cluster_util_test.go @@ -0,0 +1,120 @@ +package app_test + +import ( + "net/netip" + "testing" + + "github.com/canonical/k8s/pkg/k8sd/app" + mctypes "github.com/canonical/microcluster/v3/rest/types" + . "github.com/onsi/gomega" +) + +func TestDetermineLocalhostAddress(t *testing.T) { + t.Run("IPv4Only", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.1:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.2:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.3:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("127.0.0.1")) + }) + + t.Run("IPv6Only", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fda1:8e75:b6ef::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fd51:d664:aca3::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fda3:c11d:3cda::]:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("[::1]")) + }) + + t.Run("IPv4_IPv6_Mixed", func(t *testing.T) { + g := NewWithT(t) + + mockMembers := []mctypes.ClusterMember{ + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node1", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.1:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node2", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("[fd51:d664:aca3::]:1234"), + }, + }, + }, + { + ClusterMemberLocal: mctypes.ClusterMemberLocal{ + Name: "node3", + Address: mctypes.AddrPort{ + AddrPort: netip.MustParseAddrPort("10.1.0.3:1234"), + }, + }, + }, + } + + localhostAddress, err := app.DetermineLocalhostAddress(mockMembers) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(localhostAddress).To(Equal("[::1]")) + }) +} diff --git a/src/k8s/pkg/k8sd/app/hooks_bootstrap.go b/src/k8s/pkg/k8sd/app/hooks_bootstrap.go index 4144a0f84..1386edca9 100644 --- a/src/k8s/pkg/k8sd/app/hooks_bootstrap.go +++ b/src/k8s/pkg/k8sd/app/hooks_bootstrap.go @@ -10,6 +10,8 @@ import ( "net" "net/http" "path/filepath" + "strconv" + "strings" "time" apiv1 "github.com/canonical/k8s-snap-api/api/v1" @@ -28,7 +30,6 @@ import ( // onBootstrap is called after we bootstrap the first cluster node. // onBootstrap configures local services then writes the cluster config on the database. func (a *App) onBootstrap(ctx context.Context, s state.State, initConfig map[string]string) error { - // NOTE(neoaggelos): context timeout is passed over configuration, so that hook failures are propagated to the client ctx, cancel := context.WithCancel(ctx) defer cancel() @@ -119,8 +120,8 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT if err != nil { return fmt.Errorf("failed to prepare HTTP request: %w", err) } - httpRequest.Header.Add("worker-name", s.Name()) - httpRequest.Header.Add("worker-token", token.Secret) + httpRequest.Header.Add("Worker-Name", s.Name()) + httpRequest.Header.Add("Worker-Token", token.Secret) httpResponse, err := httpClient.Do(httpRequest) if err != nil { @@ -178,11 +179,29 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT return fmt.Errorf("failed to write worker node certificates: %w", err) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + + port := "6443" + if len(response.APIServers) == 0 { + return fmt.Errorf("no APIServers found in worker node info") + } + // Get the secure port from the first APIServer since they should all be the same. + port = response.APIServers[0][strings.LastIndex(response.APIServers[0], ":")+1:] + securePort, err := strconv.Atoi(port) + if err != nil { + return fmt.Errorf("failed to parse apiserver secure port: %w", err) + } + // Kubeconfigs - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), fmt.Sprintf("%s:%d", localhostAddress, securePort), certificates.CACert, certificates.KubeletClientCert, certificates.KubeletClientKey); err != nil { return fmt.Errorf("failed to generate kubelet kubeconfig: %w", err) } - if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "127.0.0.1:6443", certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { + if err := setup.Kubeconfig(filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), fmt.Sprintf("%s:%d", localhostAddress, securePort), certificates.CACert, certificates.KubeProxyClientCert, certificates.KubeProxyClientKey); err != nil { return fmt.Errorf("failed to generate kube-proxy kubeconfig: %w", err) } @@ -197,6 +216,9 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT // TODO(neoaggelos): We should be explicit here and try to avoid having worker nodes use // or set other cluster configuration keys by accident. cfg := types.ClusterConfig{ + APIServer: types.APIServer{ + SecurePort: utils.Pointer(securePort), + }, Network: types.Network{ PodCIDR: utils.Pointer(response.PodCIDR), ServiceCIDR: utils.Pointer(response.ServiceCIDR), @@ -206,6 +228,7 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT CACert: utils.Pointer(response.CACert), ClientCACert: utils.Pointer(response.ClientCACert), }, + Annotations: response.Annotations, } // Pre-init checks @@ -229,10 +252,10 @@ func (a *App) onBootstrapWorkerNode(ctx context.Context, s state.State, encodedT if err := setup.KubeletWorker(snap, s.Name(), nodeIP, response.ClusterDNS, response.ClusterDomain, response.CloudProvider, joinConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), response.PodCIDR, joinConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), response.PodCIDR, localhostAddress, joinConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } - if err := setup.K8sAPIServerProxy(snap, response.APIServers, joinConfig.ExtraNodeK8sAPIServerProxyArgs); err != nil { + if err := setup.K8sAPIServerProxy(snap, response.APIServers, securePort, joinConfig.ExtraNodeK8sAPIServerProxyArgs); err != nil { return fmt.Errorf("failed to configure k8s-apiserver-proxy: %w", err) } if err := setup.ExtraNodeConfigFiles(snap, joinConfig.ExtraNodeConfigFiles); err != nil { @@ -277,6 +300,13 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst return fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname()) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Create directories if err := setup.EnsureAllDirectories(snap); err != nil { return fmt.Errorf("failed to create directories: %w", err) @@ -296,7 +326,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst // NOTE: Default certificate expiration is set to 20 years. certificates := pki.NewK8sDqlitePKI(pki.K8sDqlitePKIOpts{ Hostname: s.Name(), - IPSANs: []net.IP{{127, 0, 0, 1}}, + IPSANs: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")}, NotBefore: notBefore, NotAfter: notBefore.AddDate(20, 0, 0), AllowSelfSignedCA: true, @@ -395,14 +425,15 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst } // Generate kubeconfigs - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), cfg.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, cfg.APIServer.GetSecurePort(), *certificates); err != nil { return fmt.Errorf("failed to generate kubeconfigs: %w", err) } // Configure datastore switch cfg.Datastore.GetType() { case "k8s-dqlite": - if err := setup.K8sDqlite(snap, fmt.Sprintf("%s:%d", nodeIP.String(), cfg.Datastore.GetK8sDqlitePort()), nil, bootstrapConfig.ExtraNodeK8sDqliteArgs); err != nil { + address := fmt.Sprintf("%s:%d", utils.ToIPString(nodeIP), cfg.Datastore.GetK8sDqlitePort()) + if err := setup.K8sDqlite(snap, address, nil, bootstrapConfig.ExtraNodeK8sDqliteArgs); err != nil { return fmt.Errorf("failed to configure k8s-dqlite: %w", err) } case "external": @@ -417,7 +448,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst if err := setup.KubeletControlPlane(snap, s.Name(), nodeIP, cfg.Kubelet.GetClusterDNS(), cfg.Kubelet.GetClusterDomain(), cfg.Kubelet.GetCloudProvider(), cfg.Kubelet.GetControlPlaneTaints(), bootstrapConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), bootstrapConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), localhostAddress, bootstrapConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } if err := setup.KubeControllerManager(snap, bootstrapConfig.ExtraNodeKubeControllerManagerArgs); err != nil { @@ -426,7 +457,7 @@ func (a *App) onBootstrapControlPlane(ctx context.Context, s state.State, bootst if err := setup.KubeScheduler(snap, bootstrapConfig.ExtraNodeKubeSchedulerArgs); err != nil { return fmt.Errorf("failed to configure kube-scheduler: %w", err) } - if err := setup.KubeAPIServer(snap, nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), bootstrapConfig.ExtraNodeKubeAPIServerArgs); err != nil { + if err := setup.KubeAPIServer(snap, cfg.APIServer.GetSecurePort(), nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), bootstrapConfig.ExtraNodeKubeAPIServerArgs); err != nil { return fmt.Errorf("failed to configure kube-apiserver: %w", err) } diff --git a/src/k8s/pkg/k8sd/app/hooks_join.go b/src/k8s/pkg/k8sd/app/hooks_join.go index 0e164858f..ac67b526f 100644 --- a/src/k8s/pkg/k8sd/app/hooks_join.go +++ b/src/k8s/pkg/k8sd/app/hooks_join.go @@ -48,6 +48,13 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri return fmt.Errorf("failed to parse node IP address %q", s.Address().Hostname()) } + var localhostAddress string + if nodeIP.To4() == nil { + localhostAddress = "[::1]" + } else { + localhostAddress = "127.0.0.1" + } + // Create directories if err := setup.EnsureAllDirectories(snap); err != nil { return fmt.Errorf("failed to create directories: %w", err) @@ -64,7 +71,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri // NOTE: Default certificate expiration is set to 20 years. certificates := pki.NewK8sDqlitePKI(pki.K8sDqlitePKIOpts{ Hostname: s.Name(), - IPSANs: []net.IP{{127, 0, 0, 1}}, + IPSANs: []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1")}, NotBefore: notBefore, NotAfter: notBefore.AddDate(20, 0, 0), }) @@ -140,7 +147,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri return fmt.Errorf("failed to write control plane certificates: %w", err) } - if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), cfg.APIServer.GetSecurePort(), *certificates); err != nil { + if err := setup.SetupControlPlaneKubeconfigs(snap.KubernetesConfigDir(), localhostAddress, cfg.APIServer.GetSecurePort(), *certificates); err != nil { return fmt.Errorf("failed to generate kubeconfigs: %w", err) } @@ -158,10 +165,16 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri } cluster := make([]string, len(members)) for _, member := range members { - cluster = append(cluster, fmt.Sprintf("%s:%d", member.Address.Addr(), cfg.Datastore.GetK8sDqlitePort())) + var address string + if member.Address.Addr().Is6() { + address = fmt.Sprintf("[%s]", member.Address.Addr()) + } else { + address = member.Address.Addr().String() + } + cluster = append(cluster, fmt.Sprintf("%s:%d", address, cfg.Datastore.GetK8sDqlitePort())) } - address := fmt.Sprintf("%s:%d", nodeIP.String(), cfg.Datastore.GetK8sDqlitePort()) + address := fmt.Sprintf("%s:%d", utils.ToIPString(nodeIP), cfg.Datastore.GetK8sDqlitePort()) if err := setup.K8sDqlite(snap, address, cluster, joinConfig.ExtraNodeK8sDqliteArgs); err != nil { return fmt.Errorf("failed to configure k8s-dqlite with address=%s cluster=%v: %w", address, cluster, err) } @@ -177,7 +190,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri if err := setup.KubeletControlPlane(snap, s.Name(), nodeIP, cfg.Kubelet.GetClusterDNS(), cfg.Kubelet.GetClusterDomain(), cfg.Kubelet.GetCloudProvider(), cfg.Kubelet.GetControlPlaneTaints(), joinConfig.ExtraNodeKubeletArgs); err != nil { return fmt.Errorf("failed to configure kubelet: %w", err) } - if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), joinConfig.ExtraNodeKubeProxyArgs); err != nil { + if err := setup.KubeProxy(ctx, snap, s.Name(), cfg.Network.GetPodCIDR(), localhostAddress, joinConfig.ExtraNodeKubeProxyArgs); err != nil { return fmt.Errorf("failed to configure kube-proxy: %w", err) } if err := setup.KubeControllerManager(snap, joinConfig.ExtraNodeKubeControllerManagerArgs); err != nil { @@ -186,7 +199,7 @@ func (a *App) onPostJoin(ctx context.Context, s state.State, initConfig map[stri if err := setup.KubeScheduler(snap, joinConfig.ExtraNodeKubeSchedulerArgs); err != nil { return fmt.Errorf("failed to configure kube-scheduler: %w", err) } - if err := setup.KubeAPIServer(snap, nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), joinConfig.ExtraNodeKubeAPIServerArgs); err != nil { + if err := setup.KubeAPIServer(snap, cfg.APIServer.GetSecurePort(), nodeIP, cfg.Network.GetServiceCIDR(), s.Address().Path("1.0", "kubernetes", "auth", "webhook").String(), true, cfg.Datastore, cfg.APIServer.GetAuthorizationMode(), joinConfig.ExtraNodeKubeAPIServerArgs); err != nil { return fmt.Errorf("failed to configure kube-apiserver: %w", err) } diff --git a/src/k8s/pkg/k8sd/app/hooks_remove.go b/src/k8s/pkg/k8sd/app/hooks_remove.go index 696cc1b46..bb9cc941e 100644 --- a/src/k8s/pkg/k8sd/app/hooks_remove.go +++ b/src/k8s/pkg/k8sd/app/hooks_remove.go @@ -3,11 +3,12 @@ package app import ( "context" "database/sql" + "errors" "fmt" "net" "os" - apiv1 "github.com/canonical/k8s-snap-api/api/v1" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations" databaseutil "github.com/canonical/k8s/pkg/k8sd/database/util" "github.com/canonical/k8s/pkg/k8sd/pki" "github.com/canonical/k8s/pkg/k8sd/setup" @@ -59,8 +60,9 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Error(err, "Failed to wait for node to finish microcluster join before removing. Continuing with the cleanup...") } - if cfg, err := databaseutil.GetClusterConfig(ctx, s); err == nil { - if _, ok := cfg.Annotations[apiv1.AnnotationSkipCleanupKubernetesNodeOnRemove]; !ok { + cfg, err := databaseutil.GetClusterConfig(ctx, s) + if err == nil { + if _, ok := cfg.Annotations.Get(apiv1_annotations.AnnotationSkipCleanupKubernetesNodeOnRemove); !ok { c, err := snap.KubernetesClient("") if err != nil { log.Error(err, "Failed to create Kubernetes client", err) @@ -121,12 +123,9 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Info("Removing worker node mark") if err := snaputil.MarkAsWorkerNode(snap, false); err != nil { - log.Error(err, "Failed to unmark node as worker") - } - - log.Info("Stopping worker services") - if err := snaputil.StopWorkerServices(ctx, snap); err != nil { - log.Error(err, "Failed to stop worker services") + if !errors.Is(err, os.ErrNotExist) { + log.Error(err, "failed to unmark node as worker") + } } log.Info("Cleaning up control plane certificates") @@ -134,9 +133,16 @@ func (a *App) onPreRemove(ctx context.Context, s state.State, force bool) (rerr log.Error(err, "failed to cleanup control plane certificates") } - log.Info("Stopping control plane services") - if err := snaputil.StopControlPlaneServices(ctx, snap); err != nil { - log.Error(err, "Failed to stop control-plane services") + if _, ok := cfg.Annotations.Get(apiv1_annotations.AnnotationSkipStopServicesOnRemove); !ok { + log.Info("Stopping worker services") + if err := snaputil.StopWorkerServices(ctx, snap); err != nil { + log.Error(err, "Failed to stop worker services") + } + + log.Info("Stopping control plane services") + if err := snaputil.StopControlPlaneServices(ctx, snap); err != nil { + log.Error(err, "Failed to stop control-plane services") + } } return nil diff --git a/src/k8s/pkg/k8sd/app/hooks_start.go b/src/k8s/pkg/k8sd/app/hooks_start.go index c86515fbb..1f5cecc3e 100644 --- a/src/k8s/pkg/k8sd/app/hooks_start.go +++ b/src/k8s/pkg/k8sd/app/hooks_start.go @@ -61,6 +61,24 @@ func (a *App) onStart(ctx context.Context, s state.State) error { func(ctx context.Context) (types.ClusterConfig, error) { return databaseutil.GetClusterConfig(ctx, s) }, + func() (string, error) { + c, err := s.Leader() + if err != nil { + return "", fmt.Errorf("failed to get leader client: %w", err) + } + + clusterMembers, err := c.GetClusterMembers(ctx) + if err != nil { + return "", fmt.Errorf("failed to get cluster members: %w", err) + } + + localhostAddress, err := DetermineLocalhostAddress(clusterMembers) + if err != nil { + return "", fmt.Errorf("failed to determine localhost address: %w", err) + } + + return localhostAddress, nil + }, func(ctx context.Context, dnsIP string) error { if err := s.Database().Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { if _, err := database.SetClusterConfig(ctx, tx, types.ClusterConfig{ diff --git a/src/k8s/pkg/k8sd/app/provider.go b/src/k8s/pkg/k8sd/app/provider.go index 0125633aa..5cae366a1 100644 --- a/src/k8s/pkg/k8sd/app/provider.go +++ b/src/k8s/pkg/k8sd/app/provider.go @@ -43,5 +43,5 @@ func (a *App) NotifyFeatureController(network, gateway, ingress, loadBalancer, l } } -// Ensure App implements api.Provider +// Ensure App implements api.Provider. var _ api.Provider = &App{} diff --git a/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go b/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go index 8e29eae83..326267e0d 100644 --- a/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go +++ b/src/k8s/pkg/k8sd/controllers/control_plane_configuration.go @@ -23,7 +23,7 @@ type ControlPlaneConfigurationController struct { } // NewControlPlaneConfigurationController creates a new controller. -// triggerCh is typically a `time.NewTicker().C` +// triggerCh is typically a `time.NewTicker().C`. func NewControlPlaneConfigurationController(snap snap.Snap, waitReady func(), triggerCh <-chan time.Time) *ControlPlaneConfigurationController { return &ControlPlaneConfigurationController{ snap: snap, @@ -35,7 +35,7 @@ func NewControlPlaneConfigurationController(snap snap.Snap, waitReady func(), tr // Run starts the controller. // Run accepts a context to manage the lifecycle of the controller. // Run accepts a function that retrieves the current cluster configuration. -// Run will loop every time the trigger channel is +// Run will loop every time the trigger channel is. func (c *ControlPlaneConfigurationController) Run(ctx context.Context, getClusterConfig func(context.Context) (types.ClusterConfig, error)) { c.waitReady() @@ -71,8 +71,7 @@ func (c *ControlPlaneConfigurationController) Run(ctx context.Context, getCluste func (c *ControlPlaneConfigurationController) reconcile(ctx context.Context, config types.ClusterConfig) error { // kube-apiserver: external datastore - switch config.Datastore.GetType() { - case "external": + if config.Datastore.GetType() == "external" { // certificates certificatesChanged, err := setup.EnsureExtDatastorePKI(c.snap, &pki.ExternalDatastorePKI{ DatastoreCACert: config.Datastore.GetExternalCACert(), diff --git a/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go b/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go index 33ed9d4b3..89bda62a8 100644 --- a/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go +++ b/src/k8s/pkg/k8sd/controllers/control_plane_configuration_test.go @@ -16,7 +16,7 @@ import ( . "github.com/onsi/gomega" ) -// channelSendTimeout is the timeout for pushing to channels for TestControlPlaneConfigController +// channelSendTimeout is the timeout for pushing to channels for TestControlPlaneConfigController. const channelSendTimeout = 100 * time.Millisecond type configProvider struct { @@ -197,7 +197,7 @@ func TestControlPlaneConfigController(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", earg) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(eval)) }) } @@ -209,7 +209,7 @@ func TestControlPlaneConfigController(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-controller-manager", earg) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(eval)) }) } @@ -222,7 +222,7 @@ func TestControlPlaneConfigController(t *testing.T) { _, err := os.Stat(file) if mustExist { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } else { g.Expect(err).To(MatchError(os.ErrNotExist)) } diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/config.go b/src/k8s/pkg/k8sd/controllers/csrsigning/config.go index fa19d3e0b..08e88bda0 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/config.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/config.go @@ -1,6 +1,9 @@ package csrsigning -import "github.com/canonical/k8s/pkg/k8sd/types" +import ( + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/csrsigning" + "github.com/canonical/k8s/pkg/k8sd/types" +) type internalConfig struct { autoApprove bool @@ -8,7 +11,7 @@ type internalConfig struct { func internalConfigFromAnnotations(annotations types.Annotations) internalConfig { var cfg internalConfig - if v, ok := annotations.Get("k8sd/v1alpha1/csrsigning/auto-approve"); ok && v == "true" { + if v, ok := annotations.Get(apiv1_annotations.AnnotationAutoApprove); ok && v == "true" { cfg.autoApprove = true } return cfg diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/const.go b/src/k8s/pkg/k8sd/controllers/csrsigning/const.go index 074066396..a6729df73 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/const.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/const.go @@ -3,9 +3,9 @@ package csrsigning import "time" const ( - // requeueAfterSigningFailure is the time to requeue requests when any step of the signing process failed + // requeueAfterSigningFailure is the time to requeue requests when any step of the signing process failed. requeueAfterSigningFailure = 3 * time.Second - // requeueAfterWaitingForApproved is the amount of time to requeue requests if waiting for CSR to be approved + // requeueAfterWaitingForApproved is the amount of time to requeue requests if waiting for CSR to be approved. requeueAfterWaitingForApproved = 10 * time.Second ) diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go b/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go index 3ecdb7877..03371a9d7 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/controller.go @@ -10,7 +10,6 @@ import ( "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/utils" "k8s.io/client-go/rest" - "sigs.k8s.io/controller-runtime/pkg/cache" ctrllog "sigs.k8s.io/controller-runtime/pkg/log" "sigs.k8s.io/controller-runtime/pkg/manager" diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go index 95ccbbb95..779f10777 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile.go @@ -9,6 +9,7 @@ import ( "fmt" "time" + "github.com/canonical/k8s/pkg/utils" pkiutil "github.com/canonical/k8s/pkg/utils/pki" certv1 "k8s.io/api/certificates/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -96,6 +97,15 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) return ctrl.Result{}, err } + notBefore := time.Now() + var notAfter time.Time + + if obj.Spec.ExpirationSeconds != nil { + notAfter = utils.SecondsToExpirationDate(notBefore, int(*obj.Spec.ExpirationSeconds)) + } else { + notAfter = time.Now().AddDate(10, 0, 0) + } + var crtPEM []byte switch obj.Spec.SignerName { case "k8sd.io/kubelet-serving": @@ -114,8 +124,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) CommonName: obj.Spec.Username, Organization: obj.Spec.Groups, }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, IPAddresses: certRequest.IPAddresses, DNSNames: certRequest.DNSNames, BasicConstraintsValid: true, @@ -149,8 +159,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) CommonName: obj.Spec.Username, Organization: obj.Spec.Groups, }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, BasicConstraintsValid: true, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, @@ -181,8 +191,8 @@ func (r *csrSigningReconciler) Reconcile(ctx context.Context, req ctrl.Request) Subject: pkix.Name{ CommonName: "system:kube-proxy", }, - NotBefore: time.Now(), - NotAfter: time.Now().AddDate(10, 0, 0), // TODO: expiration date from obj, or config + NotBefore: notBefore, + NotAfter: notAfter, BasicConstraintsValid: true, ExtKeyUsage: []x509.ExtKeyUsage{x509.ExtKeyUsageClientAuth}, KeyUsage: x509.KeyUsageKeyEncipherment | x509.KeyUsageDigitalSignature, diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go index c767ce9ed..3c1bcc6a4 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve.go @@ -13,7 +13,8 @@ import ( ) func reconcileAutoApprove(ctx context.Context, log log.Logger, csr *certv1.CertificateSigningRequest, - priv *rsa.PrivateKey, client client.Client) (ctrl.Result, error) { + priv *rsa.PrivateKey, client client.Client, +) (ctrl.Result, error) { var result certv1.RequestConditionType if err := validateCSR(csr, priv); err != nil { diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go index 78cbb819e..9c2a6f574 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_approve_test.go @@ -8,15 +8,14 @@ import ( "errors" "testing" + k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" + "github.com/canonical/k8s/pkg/log" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" . "github.com/onsi/gomega" certv1 "k8s.io/api/certificates/v1" v1 "k8s.io/api/core/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ctrl "sigs.k8s.io/controller-runtime" - - k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" - "github.com/canonical/k8s/pkg/log" - pkiutil "github.com/canonical/k8s/pkg/utils/pki" ) func TestAutoApprove(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go index 04493f83d..f731c82d8 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/reconcile_test.go @@ -8,6 +8,11 @@ import ( "testing" "time" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/csrsigning" + k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" + "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/log" + pkiutil "github.com/canonical/k8s/pkg/utils/pki" . "github.com/onsi/gomega" certv1 "k8s.io/api/certificates/v1" apierrors "k8s.io/apimachinery/pkg/api/errors" @@ -16,11 +21,6 @@ import ( "k8s.io/utils/ptr" ctrl "sigs.k8s.io/controller-runtime" "sigs.k8s.io/controller-runtime/pkg/client" - - k8smock "github.com/canonical/k8s/pkg/k8sd/controllers/csrsigning/test" - "github.com/canonical/k8s/pkg/k8sd/types" - "github.com/canonical/k8s/pkg/log" - pkiutil "github.com/canonical/k8s/pkg/utils/pki" ) func TestCSRNotFound(t *testing.T) { @@ -69,7 +69,7 @@ func TestFailedToGetCSR(t *testing.T) { result, err := reconciler.Reconcile(context.Background(), getDefaultRequest()) g.Expect(result).To(Equal(ctrl.Result{})) - g.Expect(err).To(MatchError(getErr)) + g.Expect(err).To(MatchError(getErr.Error())) } func TestHasSignedCertificate(t *testing.T) { @@ -298,7 +298,7 @@ func TestNotApprovedCSR(t *testing.T) { getClusterConfig: func(context.Context) (types.ClusterConfig, error) { return types.ClusterConfig{ Annotations: map[string]string{ - "k8sd/v1alpha1/csrsigning/auto-approve": "true", + apiv1_annotations.AnnotationAutoApprove: "true", }, Certificates: types.Certificates{ K8sdPrivateKey: ptr.To(priv), diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go b/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go index d3d9df0a9..4e0c9b414 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/validate.go @@ -4,6 +4,7 @@ import ( "crypto/rsa" "crypto/sha256" "crypto/subtle" + "encoding/base64" "fmt" "github.com/canonical/k8s/pkg/utils" @@ -21,7 +22,12 @@ func validateCSR(obj *certv1.CertificateSigningRequest, priv *rsa.PrivateKey) er return fmt.Errorf("failed to parse x509 certificate request: %w", err) } - encryptedSignature := obj.Annotations["k8sd.io/signature"] + encryptedSignatureB64 := obj.Annotations["k8sd.io/signature"] + encryptedSignature, err := base64.StdEncoding.DecodeString(encryptedSignatureB64) + if err != nil { + return fmt.Errorf("failed to decode b64 signature: %w", err) + } + signature, err := rsa.DecryptPKCS1v15(nil, priv, []byte(encryptedSignature)) if err != nil { return fmt.Errorf("failed to decrypt signature: %w", err) diff --git a/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go b/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go index 7c806a484..7e9a919a8 100644 --- a/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go +++ b/src/k8s/pkg/k8sd/controllers/csrsigning/validate_test.go @@ -5,6 +5,7 @@ import ( "crypto/rsa" "crypto/sha256" "crypto/x509/pkix" + "encoding/base64" "testing" pkiutil "github.com/canonical/k8s/pkg/utils/pki" @@ -93,7 +94,7 @@ func TestValidateCSREncryption(t *testing.T) { }, }, expectErr: true, - expectErrMessage: "failed to decrypt signature", + expectErrMessage: "failed to decode b64 signature", }, { name: "Missing Signature", @@ -219,5 +220,5 @@ func mustCreateEncryptedSignature(g Gomega, pub *rsa.PublicKey, csrPEM string) s signature, err := rsa.EncryptPKCS1v15(rand.Reader, pub, hash.Sum(nil)) g.Expect(err).NotTo(HaveOccurred()) - return string(signature) + return base64.StdEncoding.EncodeToString(signature) } diff --git a/src/k8s/pkg/k8sd/controllers/feature.go b/src/k8s/pkg/k8sd/controllers/feature.go index 909a43bc3..33589b88e 100644 --- a/src/k8s/pkg/k8sd/controllers/feature.go +++ b/src/k8s/pkg/k8sd/controllers/feature.go @@ -72,6 +72,7 @@ func NewFeatureController(opts FeatureControllerOpts) *FeatureController { func (c *FeatureController) Run( ctx context.Context, getClusterConfig func(context.Context) (types.ClusterConfig, error), + getLocalhostAddress func() (string, error), notifyDNSChangedIP func(ctx context.Context, dnsIP string) error, setFeatureStatus func(ctx context.Context, name types.FeatureName, featureStatus types.FeatureStatus) error, ) { @@ -79,7 +80,11 @@ func (c *FeatureController) Run( ctx = log.NewContext(ctx, log.FromContext(ctx).WithValues("controller", "feature")) go c.reconcileLoop(ctx, getClusterConfig, setFeatureStatus, features.Network, c.triggerNetworkCh, c.reconciledNetworkCh, func(cfg types.ClusterConfig) (types.FeatureStatus, error) { - return features.Implementation.ApplyNetwork(ctx, c.snap, cfg.Network, cfg.Annotations) + localhostAddress, err := getLocalhostAddress() + if err != nil { + return types.FeatureStatus{Enabled: false, Message: "failed to determine the localhost address"}, fmt.Errorf("failed to get localhost address: %w", err) + } + return features.Implementation.ApplyNetwork(ctx, c.snap, localhostAddress, cfg.APIServer, cfg.Network, cfg.Annotations) }) go c.reconcileLoop(ctx, getClusterConfig, setFeatureStatus, features.Gateway, c.triggerGatewayCh, c.reconciledGatewayCh, func(cfg types.ClusterConfig) (types.FeatureStatus, error) { diff --git a/src/k8s/pkg/k8sd/controllers/node_configuration_test.go b/src/k8s/pkg/k8sd/controllers/node_configuration_test.go index 38e6a3bf8..b6936e201 100644 --- a/src/k8s/pkg/k8sd/controllers/node_configuration_test.go +++ b/src/k8s/pkg/k8sd/controllers/node_configuration_test.go @@ -147,7 +147,7 @@ func TestConfigPropagation(t *testing.T) { for ekey, evalue := range tc.expectArgs { val, err := snaputil.GetServiceArgument(s, "kubelet", ekey) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(evalue)) } diff --git a/src/k8s/pkg/k8sd/database/capi_auth.go b/src/k8s/pkg/k8sd/database/capi_auth.go index f8a4f54fc..498c2eaa7 100644 --- a/src/k8s/pkg/k8sd/database/capi_auth.go +++ b/src/k8s/pkg/k8sd/database/capi_auth.go @@ -8,12 +8,10 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - clusterAPIConfigsStmts = map[string]int{ - "insert-capi-token": MustPrepareStatement("cluster-configs", "insert-capi-token.sql"), - "select-capi-token": MustPrepareStatement("cluster-configs", "select-capi-token.sql"), - } -) +var clusterAPIConfigsStmts = map[string]int{ + "insert-capi-token": MustPrepareStatement("cluster-configs", "insert-capi-token.sql"), + "select-capi-token": MustPrepareStatement("cluster-configs", "select-capi-token.sql"), +} // SetClusterAPIToken stores the ClusterAPI token in the cluster config. func SetClusterAPIToken(ctx context.Context, tx *sql.Tx, token string) error { diff --git a/src/k8s/pkg/k8sd/database/capi_auth_test.go b/src/k8s/pkg/k8sd/database/capi_auth_test.go index eca571532..2ffbcdf46 100644 --- a/src/k8s/pkg/k8sd/database/capi_auth_test.go +++ b/src/k8s/pkg/k8sd/database/capi_auth_test.go @@ -17,10 +17,10 @@ func TestClusterAPIAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { err := database.SetClusterAPIToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("CheckAuthToken", func(t *testing.T) { @@ -28,22 +28,22 @@ func TestClusterAPIAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { valid, err := database.ValidateClusterAPIToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("InvalidToken", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { valid, err := database.ValidateClusterAPIToken(ctx, tx, "invalid-token") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) }) diff --git a/src/k8s/pkg/k8sd/database/cluster_config.go b/src/k8s/pkg/k8sd/database/cluster_config.go index 23e945a48..aab8f5c4b 100644 --- a/src/k8s/pkg/k8sd/database/cluster_config.go +++ b/src/k8s/pkg/k8sd/database/cluster_config.go @@ -10,12 +10,10 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - clusterConfigsStmts = map[string]int{ - "insert-v1alpha2": MustPrepareStatement("cluster-configs", "insert-v1alpha2.sql"), - "select-v1alpha2": MustPrepareStatement("cluster-configs", "select-v1alpha2.sql"), - } -) +var clusterConfigsStmts = map[string]int{ + "insert-v1alpha2": MustPrepareStatement("cluster-configs", "insert-v1alpha2.sql"), + "select-v1alpha2": MustPrepareStatement("cluster-configs", "select-v1alpha2.sql"), +} // SetClusterConfig updates the cluster configuration with any non-empty values that are set. // SetClusterConfig will attempt to merge the existing and new configs, and return an error if any protected fields have changed. diff --git a/src/k8s/pkg/k8sd/database/cluster_config_test.go b/src/k8s/pkg/k8sd/database/cluster_config_test.go index aee438f6f..e4b21d220 100644 --- a/src/k8s/pkg/k8sd/database/cluster_config_test.go +++ b/src/k8s/pkg/k8sd/database/cluster_config_test.go @@ -3,11 +3,11 @@ package database_test import ( "context" "database/sql" - "github.com/canonical/k8s/pkg/utils" "testing" "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/types" + "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" ) @@ -26,19 +26,19 @@ func TestClusterConfig(t *testing.T) { // Write some config to the database err := d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { _, err := database.SetClusterConfig(context.Background(), tx, expectedClusterConfig) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) // Retrieve it and map it to the struct err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("CannotUpdateCA", func(t *testing.T) { @@ -65,11 +65,11 @@ func TestClusterConfig(t *testing.T) { err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("Update", func(t *testing.T) { @@ -104,18 +104,18 @@ func TestClusterConfig(t *testing.T) { }, }) g.Expect(returnedConfig).To(Equal(expectedClusterConfig)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = d.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { clusterConfig, err := database.GetClusterConfig(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(clusterConfig).To(Equal(expectedClusterConfig)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) } diff --git a/src/k8s/pkg/k8sd/database/feature_status_test.go b/src/k8s/pkg/k8sd/database/feature_status_test.go index 61e380f98..200e57f3b 100644 --- a/src/k8s/pkg/k8sd/database/feature_status_test.go +++ b/src/k8s/pkg/k8sd/database/feature_status_test.go @@ -6,11 +6,10 @@ import ( "testing" "time" - . "github.com/onsi/gomega" - "github.com/canonical/k8s/pkg/k8sd/database" "github.com/canonical/k8s/pkg/k8sd/features" "github.com/canonical/k8s/pkg/k8sd/types" + . "github.com/onsi/gomega" ) func TestFeatureStatus(t *testing.T) { @@ -45,21 +44,20 @@ func TestFeatureStatus(t *testing.T) { t.Run("ReturnNothingInitially", func(t *testing.T) { g := NewWithT(t) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(BeEmpty()) - }) t.Run("SettingNewStatus", func(t *testing.T) { g := NewWithT(t) err := database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(HaveLen(2)) g.Expect(ss[features.Network].Enabled).To(Equal(networkStatus.Enabled)) @@ -71,26 +69,25 @@ func TestFeatureStatus(t *testing.T) { g.Expect(ss[features.DNS].Message).To(Equal(dnsStatus.Message)) g.Expect(ss[features.DNS].Version).To(Equal(dnsStatus.Version)) g.Expect(ss[features.DNS].UpdatedAt).To(Equal(dnsStatus.UpdatedAt)) - }) t.Run("UpdatingStatus", func(t *testing.T) { g := NewWithT(t) err := database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) // set and update err = database.SetFeatureStatus(ctx, tx, features.Network, networkStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.DNS, dnsStatus2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = database.SetFeatureStatus(ctx, tx, features.Gateway, gatewayStatus) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) ss, err := database.GetFeatureStatuses(ctx, tx) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ss).To(HaveLen(3)) // network stayed the same diff --git a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go index 22c7b7a2b..73e708ef7 100644 --- a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go +++ b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens.go @@ -13,15 +13,13 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - k8sdTokensStmts = map[string]int{ - "insert-token": MustPrepareStatement("kubernetes-auth-tokens", "insert-token.sql"), - "select-by-token": MustPrepareStatement("kubernetes-auth-tokens", "select-by-token.sql"), - "select-by-username": MustPrepareStatement("kubernetes-auth-tokens", "select-by-username.sql"), - "delete-by-token": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-token.sql"), - "delete-by-username": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-username.sql"), - } -) +var k8sdTokensStmts = map[string]int{ + "insert-token": MustPrepareStatement("kubernetes-auth-tokens", "insert-token.sql"), + "select-by-token": MustPrepareStatement("kubernetes-auth-tokens", "select-by-token.sql"), + "select-by-username": MustPrepareStatement("kubernetes-auth-tokens", "select-by-username.sql"), + "delete-by-token": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-token.sql"), + "delete-by-username": MustPrepareStatement("kubernetes-auth-tokens", "delete-by-username.sql"), +} func groupsToString(inGroups []string) (string, error) { groupMap := make(map[string]struct{}, len(inGroups)) diff --git a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go index 18d73779b..dd7d8ce9e 100644 --- a/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go +++ b/src/k8s/pkg/k8sd/database/kubernetes_auth_tokens_test.go @@ -19,27 +19,27 @@ func TestKubernetesAuthTokens(t *testing.T) { var err error token1, err = database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token1).To(Not(BeEmpty())) token2, err = database.GetOrCreateToken(ctx, tx, "user2", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token2).To(Not(BeEmpty())) g.Expect(token1).To(Not(Equal(token2))) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) t.Run("Existing", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { token, err := database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(Equal(token1)) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) @@ -48,23 +48,23 @@ func TestKubernetesAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { username, groups, err := database.CheckToken(ctx, tx, token1) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(username).To(Equal("user1")) g.Expect(groups).To(ConsistOf("group1", "group2")) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("user2", func(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { username, groups, err := database.CheckToken(ctx, tx, token2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(username).To(Equal("user2")) g.Expect(groups).To(ConsistOf("group1", "group2")) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) @@ -72,15 +72,15 @@ func TestKubernetesAuthTokens(t *testing.T) { g := NewWithT(t) err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { err := database.DeleteToken(ctx, tx, token2) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) username, groups, err := database.CheckToken(ctx, tx, token2) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) g.Expect(username).To(BeEmpty()) g.Expect(groups).To(BeEmpty()) return nil }) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) }) } diff --git a/src/k8s/pkg/k8sd/database/schema.go b/src/k8s/pkg/k8sd/database/schema.go index 44d665629..b96a5559f 100644 --- a/src/k8s/pkg/k8sd/database/schema.go +++ b/src/k8s/pkg/k8sd/database/schema.go @@ -36,7 +36,7 @@ func schemaApplyMigration(migrationPath ...string) schema.Update { path := filepath.Join(append([]string{"sql", "migrations"}, migrationPath...)...) b, err := sqlMigrations.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid migration file %s: %s", path, err)) + panic(fmt.Errorf("invalid migration file %s: %w", path, err)) } return func(ctx context.Context, tx *sql.Tx) error { if _, err := tx.ExecContext(ctx, string(b)); err != nil { @@ -51,7 +51,7 @@ func MustPrepareStatement(queryPath ...string) int { path := filepath.Join(append([]string{"sql", "queries"}, queryPath...)...) b, err := sqlQueries.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid query file %s: %s", path, err)) + panic(fmt.Errorf("invalid query file %s: %w", path, err)) } return cluster.RegisterStmt(string(b)) } diff --git a/src/k8s/pkg/k8sd/database/util_test.go b/src/k8s/pkg/k8sd/database/util_test.go index 265302bfe..e39712b59 100644 --- a/src/k8s/pkg/k8sd/database/util_test.go +++ b/src/k8s/pkg/k8sd/database/util_test.go @@ -12,16 +12,14 @@ import ( ) const ( - // microclusterDatabaseInitTimeout is the timeout for microcluster database initialization operations + // microclusterDatabaseInitTimeout is the timeout for microcluster database initialization operations. microclusterDatabaseInitTimeout = 3 * time.Second - // microclusterDatabaseShutdownTimeout is the timeout for microcluster database shutdown operations + // microclusterDatabaseShutdownTimeout is the timeout for microcluster database shutdown operations. microclusterDatabaseShutdownTimeout = 3 * time.Second ) -var ( - // nextIdx is used to pick different listen ports for each microcluster instance - nextIdx int -) +// nextIdx is used to pick different listen ports for each microcluster instance. +var nextIdx int // DB is an interface for the internal microcluster DB type. type DB interface { @@ -38,13 +36,13 @@ type DB interface { // WithDB(t, func(ctx context.Context, db DB) { // err := db.Transaction(ctx, func(ctx context.Context, tx *sql.Tx) error { // token, err := database.GetOrCreateToken(ctx, tx, "user1", []string{"group1", "group2"}) -// if !g.Expect(err).To(BeNil()) { +// if !g.Expect(err).To(Not(HaveOccurred())) { // return err // } // g.Expect(token).To(Not(BeEmpty())) // return nil // }) -// g.Expect(err).To(BeNil()) +// g.Expect(err).To(Not(HaveOccurred())) // }) // }) // } diff --git a/src/k8s/pkg/k8sd/database/worker.go b/src/k8s/pkg/k8sd/database/worker.go index c97da8267..043231a1a 100644 --- a/src/k8s/pkg/k8sd/database/worker.go +++ b/src/k8s/pkg/k8sd/database/worker.go @@ -12,13 +12,11 @@ import ( "github.com/canonical/microcluster/v3/cluster" ) -var ( - workerStmts = map[string]int{ - "insert-token": MustPrepareStatement("worker-tokens", "insert.sql"), - "select-token": MustPrepareStatement("worker-tokens", "select.sql"), - "delete-token": MustPrepareStatement("worker-tokens", "delete-by-token.sql"), - } -) +var workerStmts = map[string]int{ + "insert-token": MustPrepareStatement("worker-tokens", "insert.sql"), + "select-token": MustPrepareStatement("worker-tokens", "select.sql"), + "delete-token": MustPrepareStatement("worker-tokens", "delete-by-token.sql"), +} // CheckWorkerNodeToken returns true if the specified token can be used to join the specified node on the cluster. // CheckWorkerNodeToken will return true if the token is empty or if the token is associated with the specified node diff --git a/src/k8s/pkg/k8sd/database/worker_test.go b/src/k8s/pkg/k8sd/database/worker_test.go index 61de6e499..ce154fcdb 100644 --- a/src/k8s/pkg/k8sd/database/worker_test.go +++ b/src/k8s/pkg/k8sd/database/worker_test.go @@ -17,39 +17,39 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("Default", func(t *testing.T) { g := NewWithT(t) exists, err := database.CheckWorkerNodeToken(ctx, tx, "somenode", "sometoken") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeFalse()) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "somenode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) othertoken, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "someothernode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(othertoken).To(HaveLen(48)) g.Expect(othertoken).NotTo(Equal(token)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "somenode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) valid, err = database.CheckWorkerNodeToken(ctx, tx, "someothernode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) valid, err = database.CheckWorkerNodeToken(ctx, tx, "someothernode", othertoken) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) err = database.DeleteWorkerNodeToken(ctx, tx, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) valid, err = database.CheckWorkerNodeToken(ctx, tx, "somenode", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) newToken, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "somenode", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(newToken).To(HaveLen(48)) g.Expect(newToken).ToNot(Equal(token)) }) @@ -58,22 +58,22 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("Valid", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "nodeExpiry1", time.Now().Add(time.Hour)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "nodeExpiry1", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) }) t.Run("Expired", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "nodeExpiry2", time.Now().Add(-time.Hour)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) valid, err := database.CheckWorkerNodeToken(ctx, tx, "nodeExpiry2", token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeFalse()) }) }) @@ -81,7 +81,7 @@ func TestWorkerNodeToken(t *testing.T) { t.Run("AnyNodeName", func(t *testing.T) { g := NewWithT(t) token, err := database.GetOrCreateWorkerNodeToken(ctx, tx, "", tokenExpiry) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(token).To(HaveLen(48)) for _, name := range []string{"", "test", "other"} { @@ -89,7 +89,7 @@ func TestWorkerNodeToken(t *testing.T) { g := NewWithT(t) valid, err := database.CheckWorkerNodeToken(ctx, tx, name, token) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(valid).To(BeTrue()) }) } diff --git a/src/k8s/pkg/k8sd/features/calico/internal.go b/src/k8s/pkg/k8sd/features/calico/internal.go index 930bc674a..e3e48443d 100644 --- a/src/k8s/pkg/k8sd/features/calico/internal.go +++ b/src/k8s/pkg/k8sd/features/calico/internal.go @@ -4,27 +4,10 @@ import ( "fmt" "strings" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" "github.com/canonical/k8s/pkg/k8sd/types" ) -const ( - annotationAPIServerEnabled = "k8sd/v1alpha1/calico/apiserver-enabled" - annotationEncapsulationV4 = "k8sd/v1alpha1/calico/encapsulation-v4" - annotationEncapsulationV6 = "k8sd/v1alpha1/calico/encapsulation-v6" - annotationAutodetectionV4FirstFound = "k8sd/v1alpha1/calico/autodetection-v4/firstFound" - annotationAutodetectionV4Kubernetes = "k8sd/v1alpha1/calico/autodetection-v4/kubernetes" - annotationAutodetectionV4Interface = "k8sd/v1alpha1/calico/autodetection-v4/interface" - annotationAutodetectionV4SkipInterface = "k8sd/v1alpha1/calico/autodetection-v4/skipInterface" - annotationAutodetectionV4CanReach = "k8sd/v1alpha1/calico/autodetection-v4/canReach" - annotationAutodetectionV4CIDRs = "k8sd/v1alpha1/calico/autodetection-v4/cidrs" - annotationAutodetectionV6FirstFound = "k8sd/v1alpha1/calico/autodetection-v6/firstFound" - annotationAutodetectionV6Kubernetes = "k8sd/v1alpha1/calico/autodetection-v6/kubernetes" - annotationAutodetectionV6Interface = "k8sd/v1alpha1/calico/autodetection-v6/interface" - annotationAutodetectionV6SkipInterface = "k8sd/v1alpha1/calico/autodetection-v6/skipInterface" - annotationAutodetectionV6CanReach = "k8sd/v1alpha1/calico/autodetection-v6/canReach" - annotationAutodetectionV6CIDRs = "k8sd/v1alpha1/calico/autodetection-v6/cidrs" -) - type config struct { encapsulationV4 string encapsulationV6 string @@ -64,7 +47,11 @@ func parseAutodetectionAnnotations(annotations types.Annotations, autodetectionM case "firstFound": autodetectionValue = autodetectionValue == "true" case "cidrs": - autodetectionValue = strings.Split(autodetectionValue.(string), ",") + if strValue, ok := autodetectionValue.(string); ok { + autodetectionValue = strings.Split(strValue, ",") + } else { + return nil, fmt.Errorf("invalid type for cidrs annotation: %T", autodetectionValue) + } } return map[string]any{ @@ -82,18 +69,18 @@ func internalConfig(annotations types.Annotations) (config, error) { apiServerEnabled: defaultAPIServerEnabled, } - if v, ok := annotations.Get(annotationAPIServerEnabled); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationAPIServerEnabled); ok { c.apiServerEnabled = v == "true" } - if v, ok := annotations.Get(annotationEncapsulationV4); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationEncapsulationV4); ok { if err := checkEncapsulation(v); err != nil { return config{}, fmt.Errorf("invalid encapsulation-v4 annotation: %w", err) } c.encapsulationV4 = v } - if v, ok := annotations.Get(annotationEncapsulationV6); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationEncapsulationV6); ok { if err := checkEncapsulation(v); err != nil { return config{}, fmt.Errorf("invalid encapsulation-v6 annotation: %w", err) } @@ -101,12 +88,12 @@ func internalConfig(annotations types.Annotations) (config, error) { } v4Map := map[string]string{ - annotationAutodetectionV4FirstFound: "firstFound", - annotationAutodetectionV4Kubernetes: "kubernetes", - annotationAutodetectionV4Interface: "interface", - annotationAutodetectionV4SkipInterface: "skipInterface", - annotationAutodetectionV4CanReach: "canReach", - annotationAutodetectionV4CIDRs: "cidrs", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "firstFound", + apiv1_annotations.AnnotationAutodetectionV4Kubernetes: "kubernetes", + apiv1_annotations.AnnotationAutodetectionV4Interface: "interface", + apiv1_annotations.AnnotationAutodetectionV4SkipInterface: "skipInterface", + apiv1_annotations.AnnotationAutodetectionV4CanReach: "canReach", + apiv1_annotations.AnnotationAutodetectionV4CIDRs: "cidrs", } autodetectionV4, err := parseAutodetectionAnnotations(annotations, v4Map) @@ -119,12 +106,12 @@ func internalConfig(annotations types.Annotations) (config, error) { } v6Map := map[string]string{ - annotationAutodetectionV6FirstFound: "firstFound", - annotationAutodetectionV6Kubernetes: "kubernetes", - annotationAutodetectionV6Interface: "interface", - annotationAutodetectionV6SkipInterface: "skipInterface", - annotationAutodetectionV6CanReach: "canReach", - annotationAutodetectionV6CIDRs: "cidrs", + apiv1_annotations.AnnotationAutodetectionV6FirstFound: "firstFound", + apiv1_annotations.AnnotationAutodetectionV6Kubernetes: "kubernetes", + apiv1_annotations.AnnotationAutodetectionV6Interface: "interface", + apiv1_annotations.AnnotationAutodetectionV6SkipInterface: "skipInterface", + apiv1_annotations.AnnotationAutodetectionV6CanReach: "canReach", + apiv1_annotations.AnnotationAutodetectionV6CIDRs: "cidrs", } autodetectionV6, err := parseAutodetectionAnnotations(annotations, v6Map) diff --git a/src/k8s/pkg/k8sd/features/calico/internal_test.go b/src/k8s/pkg/k8sd/features/calico/internal_test.go index dd4c630fb..208198315 100644 --- a/src/k8s/pkg/k8sd/features/calico/internal_test.go +++ b/src/k8s/pkg/k8sd/features/calico/internal_test.go @@ -3,6 +3,7 @@ package calico import ( "testing" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" . "github.com/onsi/gomega" ) @@ -26,8 +27,8 @@ func TestInternalConfig(t *testing.T) { { name: "Valid", annotations: map[string]string{ - annotationAPIServerEnabled: "true", - annotationEncapsulationV4: "IPIP", + apiv1_annotations.AnnotationAPIServerEnabled: "true", + apiv1_annotations.AnnotationEncapsulationV4: "IPIP", }, expectedConfig: config{ apiServerEnabled: true, @@ -39,15 +40,15 @@ func TestInternalConfig(t *testing.T) { { name: "InvalidEncapsulation", annotations: map[string]string{ - annotationEncapsulationV4: "Invalid", + apiv1_annotations.AnnotationEncapsulationV4: "Invalid", }, expectError: true, }, { name: "InvalidAPIServerEnabled", annotations: map[string]string{ - annotationAPIServerEnabled: "invalid", - annotationEncapsulationV4: "VXLAN", + apiv1_annotations.AnnotationAPIServerEnabled: "invalid", + apiv1_annotations.AnnotationEncapsulationV4: "VXLAN", }, expectedConfig: config{ apiServerEnabled: false, @@ -59,15 +60,15 @@ func TestInternalConfig(t *testing.T) { { name: "MultipleAutodetectionV4", annotations: map[string]string{ - annotationAutodetectionV4FirstFound: "true", - annotationAutodetectionV4Kubernetes: "true", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "true", + apiv1_annotations.AnnotationAutodetectionV4Kubernetes: "true", }, expectError: true, }, { name: "ValidAutodetectionCidrs", annotations: map[string]string{ - annotationAutodetectionV4CIDRs: "10.1.0.0/16,2001:0db8::/32", + apiv1_annotations.AnnotationAutodetectionV4CIDRs: "10.1.0.0/16,2001:0db8::/32", }, expectedConfig: config{ apiServerEnabled: false, diff --git a/src/k8s/pkg/k8sd/features/calico/network.go b/src/k8s/pkg/k8sd/features/calico/network.go index 82e98c788..8820e6c8f 100644 --- a/src/k8s/pkg/k8sd/features/calico/network.go +++ b/src/k8s/pkg/k8sd/features/calico/network.go @@ -17,16 +17,16 @@ const ( deleteFailedMsgTmpl = "Failed to delete Calico, the error was: %v" ) -// ApplyNetwork will deploy Calico when cfg.Enabled is true. -// ApplyNetwork will remove Calico when cfg.Enabled is false. +// ApplyNetwork will deploy Calico when network.Enabled is true. +// ApplyNetwork will remove Calico when network.Enabled is false. // ApplyNetwork will always return a FeatureStatus indicating the current status of the // deployment. // ApplyNetwork returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annotations types.Annotations) (types.FeatureStatus, error) { +func ApplyNetwork(ctx context.Context, snap snap.Snap, _ string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() - if !cfg.GetEnabled() { + if !network.GetEnabled() { if _, err := m.Apply(ctx, ChartCalico, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall network: %w", err) return types.FeatureStatus{ @@ -54,7 +54,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annota } podIpPools := []map[string]any{} - ipv4PodCIDR, ipv6PodCIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + ipv4PodCIDR, ipv6PodCIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) if err != nil { err = fmt.Errorf("invalid pod cidr: %w", err) return types.FeatureStatus{ @@ -79,9 +79,9 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annota } serviceCIDRs := []string{} - ipv4ServiceCIDR, ipv6ServiceCIDR, err := utils.ParseCIDRs(cfg.GetServiceCIDR()) + ipv4ServiceCIDR, ipv6ServiceCIDR, err := utils.SplitCIDRStrings(network.GetServiceCIDR()) if err != nil { - err = fmt.Errorf("invalid service cidr: %v", err) + err = fmt.Errorf("invalid service cidr: %w", err) return types.FeatureStatus{ Enabled: false, Version: CalicoTag, diff --git a/src/k8s/pkg/k8sd/features/calico/network_test.go b/src/k8s/pkg/k8sd/features/calico/network_test.go index c0a324028..0a8b4d716 100644 --- a/src/k8s/pkg/k8sd/features/calico/network_test.go +++ b/src/k8s/pkg/k8sd/features/calico/network_test.go @@ -5,25 +5,25 @@ import ( "errors" "testing" - . "github.com/onsi/gomega" - + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/calico" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/k8sd/features/calico" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" "github.com/canonical/k8s/pkg/utils" + . "github.com/onsi/gomega" "k8s.io/utils/ptr" ) // NOTE(hue): status.Message is not checked sometimes to avoid unnecessary complexity var defaultAnnotations = types.Annotations{ - "k8sd/v1alpha1/calico/apiserver-enabled": "true", - "k8sd/v1alpha1/calico/encapsulation-v4": "VXLAN", - "k8sd/v1alpha1/calico/encapsulation-v6": "VXLAN", - "k8sd/v1alpha1/calico/autodetection-v4/firstFound": "true", - "k8sd/v1alpha1/calico/autodetection-v6/firstFound": "true", + apiv1_annotations.AnnotationAPIServerEnabled: "true", + apiv1_annotations.AnnotationEncapsulationV4: "VXLAN", + apiv1_annotations.AnnotationEncapsulationV6: "VXLAN", + apiv1_annotations.AnnotationAutodetectionV4FirstFound: "true", + apiv1_annotations.AnnotationAutodetectionV6FirstFound: "true", } func TestDisabled(t *testing.T) { @@ -39,11 +39,14 @@ func TestDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) @@ -65,11 +68,14 @@ func TestDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) @@ -94,17 +100,20 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) g.Expect(status.Version).To(Equal(calico.CalicoTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) t.Run("InvalidServiceCIDR", func(t *testing.T) { g := NewWithT(t) @@ -115,18 +124,21 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) g.Expect(status.Version).To(Equal(calico.CalicoTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -140,13 +152,16 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) @@ -157,7 +172,7 @@ func TestEnabled(t *testing.T) { callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(calico.ChartCalico)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateValues(t, callArgs.Values, cfg) + validateValues(t, callArgs.Values, network) }) t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -168,13 +183,16 @@ func TestEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), ServiceCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := calico.ApplyNetwork(context.Background(), snapM, cfg, defaultAnnotations) + status, err := calico.ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, defaultAnnotations) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeTrue()) @@ -185,17 +203,17 @@ func TestEnabled(t *testing.T) { callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(calico.ChartCalico)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateValues(t, callArgs.Values, cfg) + validateValues(t, callArgs.Values, network) }) } -func validateValues(t *testing.T, values map[string]any, cfg types.Network) { +func validateValues(t *testing.T, values map[string]any, network types.Network) { g := NewWithT(t) - podIPv4CIDR, podIPv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + podIPv4CIDR, podIPv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) g.Expect(err).ToNot(HaveOccurred()) - svcIPv4CIDR, svcIPv6CIDR, err := utils.ParseCIDRs(cfg.GetServiceCIDR()) + svcIPv4CIDR, svcIPv6CIDR, err := utils.SplitCIDRStrings(network.GetServiceCIDR()) g.Expect(err).ToNot(HaveOccurred()) // calico network @@ -211,10 +229,10 @@ func validateValues(t *testing.T, values map[string]any, cfg types.Network) { "encapsulation": "VXLAN", })) g.Expect(calicoNetwork["ipPools"].([]map[string]any)).To(HaveLen(2)) - g.Expect(calicoNetwork["nodeAddressAutodetectionV4"].(map[string]any)["firstFound"]).To(Equal(true)) - g.Expect(calicoNetwork["nodeAddressAutodetectionV6"].(map[string]any)["firstFound"]).To(Equal(true)) + g.Expect(calicoNetwork["nodeAddressAutodetectionV4"].(map[string]any)["firstFound"]).To(BeTrue()) + g.Expect(calicoNetwork["nodeAddressAutodetectionV6"].(map[string]any)["firstFound"]).To(BeTrue()) - g.Expect(values["apiServer"].(map[string]any)["enabled"]).To(Equal(true)) + g.Expect(values["apiServer"].(map[string]any)["enabled"]).To(BeTrue()) // service CIDRs g.Expect(values["serviceCIDRs"].([]string)).To(ContainElements(svcIPv4CIDR, svcIPv6CIDR)) diff --git a/src/k8s/pkg/k8sd/features/calico/status.go b/src/k8s/pkg/k8sd/features/calico/status.go index 423fe7426..8cafae4d4 100644 --- a/src/k8s/pkg/k8sd/features/calico/status.go +++ b/src/k8s/pkg/k8sd/features/calico/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/chart.go b/src/k8s/pkg/k8sd/features/cilium/chart.go index bcae2b11b..a6e9bc310 100644 --- a/src/k8s/pkg/k8sd/features/cilium/chart.go +++ b/src/k8s/pkg/k8sd/features/cilium/chart.go @@ -11,7 +11,7 @@ var ( ChartCilium = helm.InstallableChart{ Name: "ck-network", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "cilium-1.15.2.tgz"), + ManifestPath: filepath.Join("charts", "cilium-1.16.3.tgz"), } // ChartCiliumLoadBalancer represents manifests to deploy Cilium LoadBalancer resources. @@ -25,10 +25,10 @@ var ( chartGateway = helm.InstallableChart{ Name: "ck-gateway", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "gateway-api-1.0.0.tgz"), + ManifestPath: filepath.Join("charts", "gateway-api-1.1.0.tgz"), } - //chartGatewayClass represents a manifest to deploy a GatewayClass called ck-gateway. + // chartGatewayClass represents a manifest to deploy a GatewayClass called ck-gateway. chartGatewayClass = helm.InstallableChart{ Name: "ck-gateway-class", Namespace: "default", @@ -39,11 +39,11 @@ var ( ciliumAgentImageRepo = "ghcr.io/canonical/cilium" // CiliumAgentImageTag is the tag to use for the cilium-agent image. - CiliumAgentImageTag = "1.15.2-ck2" + CiliumAgentImageTag = "1.16.3-ck0" // ciliumOperatorImageRepo is the image to use for cilium-operator. ciliumOperatorImageRepo = "ghcr.io/canonical/cilium-operator" // ciliumOperatorImageTag is the tag to use for the cilium-operator image. - ciliumOperatorImageTag = "1.15.2-ck2" + ciliumOperatorImageTag = "1.16.3-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/cleanup.go b/src/k8s/pkg/k8sd/features/cilium/cleanup.go index bb97321e8..78d1fd098 100644 --- a/src/k8s/pkg/k8sd/features/cilium/cleanup.go +++ b/src/k8s/pkg/k8sd/features/cilium/cleanup.go @@ -5,6 +5,7 @@ import ( "fmt" "os" "os/exec" + "strings" "github.com/canonical/k8s/pkg/snap" ) @@ -13,10 +14,30 @@ func CleanupNetwork(ctx context.Context, snap snap.Snap) error { os.Remove("/var/run/cilium/cilium.pid") if _, err := os.Stat("/opt/cni/bin/cilium-dbg"); err == nil { - if err := exec.CommandContext(ctx, "/opt/cni/bin/cilium-dbg", "cleanup", "--all-state", "--force").Run(); err != nil { + if err := exec.CommandContext(ctx, "/opt/cni/bin/cilium-dbg", "post-uninstall-cleanup", "--all-state", "--force").Run(); err != nil { return fmt.Errorf("cilium-dbg cleanup failed: %w", err) } } + for _, cmd := range []string{"iptables", "ip6tables", "iptables-legacy", "ip6tables-legacy"} { + out, err := exec.Command(fmt.Sprintf("%s-save", cmd)).Output() + if err != nil { + return fmt.Errorf("failed to read iptables rules: %w", err) + } + + lines := strings.Split(string(out), "\n") + for i, line := range lines { + if strings.Contains(strings.ToLower(line), "cilium") { + lines[i] = "" + } + } + + restore := exec.Command(fmt.Sprintf("%s-restore", cmd)) + restore.Stdin = strings.NewReader(strings.Join(lines, "\n")) + if err := restore.Run(); err != nil { + return fmt.Errorf("failed to restore iptables rules: %w", err) + } + } + return nil } diff --git a/src/k8s/pkg/k8sd/features/cilium/gateway.go b/src/k8s/pkg/k8sd/features/cilium/gateway.go index 8632e660e..44b7f9b63 100644 --- a/src/k8s/pkg/k8sd/features/cilium/gateway.go +++ b/src/k8s/pkg/k8sd/features/cilium/gateway.go @@ -10,8 +10,8 @@ import ( ) const ( - gatewayDeleteFailedMsgTmpl = "Failed to delete Cilium Gateway, the error was %v" - gatewayDeployFailedMsgTmpl = "Failed to deploy Cilium Gateway, the error was %v" + GatewayDeleteFailedMsgTmpl = "Failed to delete Cilium Gateway, the error was %v" + GatewayDeployFailedMsgTmpl = "Failed to deploy Cilium Gateway, the error was %v" ) // ApplyGateway assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -38,7 +38,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -48,7 +48,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -58,7 +58,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -75,7 +75,7 @@ func enableGateway(ctx context.Context, snap snap.Snap) (types.FeatureStatus, er return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -95,7 +95,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -105,7 +105,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -116,7 +116,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } @@ -133,7 +133,7 @@ func disableGateway(ctx context.Context, snap snap.Snap, network types.Network) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } diff --git a/src/k8s/pkg/k8sd/features/cilium/gateway_test.go b/src/k8s/pkg/k8sd/features/cilium/gateway_test.go new file mode 100644 index 000000000..2bcab0b11 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/gateway_test.go @@ -0,0 +1,270 @@ +package cilium_test + +import ( + "context" + "errors" + "fmt" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestGatewayEnabled(t *testing.T) { + t.Run("HelmApplyErr", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("AlreadyDeployed", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: false, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + + helmCiliumArgs := helmM.ApplyCalledWith[2] + g.Expect(helmCiliumArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(helmCiliumArgs.State).To(Equal(helm.StateUpgradeOnly)) + g.Expect(helmCiliumArgs.Values["gatewayAPI"].(map[string]any)["enabled"]).To(BeTrue()) + }) + + t.Run("RolloutFail", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + }) +} + +func TestGatewayDisabled(t *testing.T) { + t.Run("HelmApplyErr", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeleteFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("AlreadyDeleted", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: false, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) + + helmCiliumArgs := helmM.ApplyCalledWith[1] + g.Expect(helmCiliumArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(helmCiliumArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(helmCiliumArgs.Values["gatewayAPI"].(map[string]any)["enabled"]).To(BeFalse()) + }) + + t.Run("RolloutFail", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.GatewayDeployFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := cilium.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) + }) +} diff --git a/src/k8s/pkg/k8sd/features/cilium/ingress.go b/src/k8s/pkg/k8sd/features/cilium/ingress.go index d62f6153f..a80cdd876 100644 --- a/src/k8s/pkg/k8sd/features/cilium/ingress.go +++ b/src/k8s/pkg/k8sd/features/cilium/ingress.go @@ -10,8 +10,8 @@ import ( ) const ( - ingressDeleteFailedMsgTmpl = "Failed to delete Cilium Ingress, the error was: %v" - ingressDeployFailedMsgTmpl = "Failed to deploy Cilium Ingress, the error was: %v" + IngressDeleteFailedMsgTmpl = "Failed to delete Cilium Ingress, the error was: %v" + IngressDeployFailedMsgTmpl = "Failed to deploy Cilium Ingress, the error was: %v" ) // ApplyIngress assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -54,14 +54,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, ne return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } else { err = fmt.Errorf("failed to disable ingress: %w", err) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeleteFailedMsgTmpl, err), }, err } } @@ -95,7 +95,7 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, ne return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } diff --git a/src/k8s/pkg/k8sd/features/cilium/ingress_test.go b/src/k8s/pkg/k8sd/features/cilium/ingress_test.go new file mode 100644 index 000000000..6a8e4977c --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/ingress_test.go @@ -0,0 +1,209 @@ +package cilium_test + +import ( + "context" + "errors" + "fmt" + "testing" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestIngress(t *testing.T) { + applyErr := errors.New("failed to apply") + for _, tc := range []struct { + name string + // given + networkEnabled bool + applyChanged bool + ingressEnabled bool + helmErr error + // then + statusMsg string + statusEnabled bool + }{ + { + name: "HelmFailNetworkEnabled", + networkEnabled: true, + helmErr: applyErr, + statusMsg: fmt.Sprintf(cilium.IngressDeployFailedMsgTmpl, fmt.Errorf("failed to enable ingress: %w", applyErr)), + statusEnabled: false, + }, + { + name: "HelmFailNetworkDisabled", + networkEnabled: false, + statusMsg: fmt.Sprintf(cilium.IngressDeleteFailedMsgTmpl, fmt.Errorf("failed to disable ingress: %w", applyErr)), + statusEnabled: false, + helmErr: applyErr, + }, + { + name: "HelmUnchangedIngressEnabled", + ingressEnabled: true, + statusMsg: cilium.EnabledMsg, + statusEnabled: true, + }, + { + name: "HelmUnchangedIngressDisabled", + ingressEnabled: false, + statusMsg: cilium.DisabledMsg, + statusEnabled: false, + }, + { + name: "HelmChangedIngressDisabled", + applyChanged: true, + ingressEnabled: false, + statusMsg: cilium.DisabledMsg, + statusEnabled: false, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyErr: tc.helmErr, + ApplyChanged: tc.applyChanged, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{ + Enabled: ptr.To(tc.networkEnabled), + } + ingress := types.Ingress{ + Enabled: ptr.To(tc.ingressEnabled), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + if tc.helmErr == nil { + g.Expect(err).To(Not(HaveOccurred())) + } else { + g.Expect(err).To(MatchError(applyErr)) + } + g.Expect(status.Enabled).To(Equal(tc.statusEnabled)) + g.Expect(status.Message).To(Equal(tc.statusMsg)) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) + } +} + +func TestIngressRollout(t *testing.T) { + t.Run("Error", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "dummy", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.IngressDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, + }, + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, + }, + ) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := cilium.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + validateIngressValues(g, callArgs.Values, ingress) + }) +} + +func validateIngressValues(g Gomega, values map[string]any, ingress types.Ingress) { + ingressController, ok := values["ingressController"].(map[string]any) + g.Expect(ok).To(BeTrue()) + if ingress.GetEnabled() { + g.Expect(ingressController["enabled"]).To(BeTrue()) + g.Expect(ingressController["loadbalancerMode"]).To(Equal("shared")) + g.Expect(ingressController["defaultSecretNamespace"]).To(Equal("kube-system")) + g.Expect(ingressController["defaultTLSSecret"]).To(Equal(ingress.GetDefaultTLSSecret())) + g.Expect(ingressController["enableProxyProtocol"]).To(Equal(ingress.GetEnableProxyProtocol())) + } else { + g.Expect(ingressController["enabled"]).To(BeFalse()) + g.Expect(ingressController["defaultSecretNamespace"]).To(Equal("")) + g.Expect(ingressController["defaultSecretName"]).To(Equal("")) + g.Expect(ingressController["enableProxyProtocol"]).To(BeFalse()) + } +} diff --git a/src/k8s/pkg/k8sd/features/cilium/internal.go b/src/k8s/pkg/k8sd/features/cilium/internal.go new file mode 100644 index 000000000..72758019b --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/internal.go @@ -0,0 +1,75 @@ +package cilium + +import ( + "fmt" + "slices" + "strconv" + "strings" + + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" + "github.com/canonical/k8s/pkg/k8sd/types" +) + +const ( + // minVLANIDValue is the minimum valid 802.1Q VLAN ID value. + minVLANIDValue = 0 + // maxVLANIDValue is the maximum valid 802.1Q VLAN ID value. + maxVLANIDValue = 4094 +) + +type config struct { + devices string + directRoutingDevice string + vlanBPFBypass []int +} + +func validateVLANBPFBypass(vlanList string) ([]int, error) { + vlanList = strings.TrimSpace(vlanList) + // Maintain compatibility with the Cilium chart definition + vlanList = strings.Trim(vlanList, "{}") + vlans := strings.Split(vlanList, ",") + + vlanTags := make([]int, 0, len(vlans)) + seenTags := make(map[int]struct{}) + + for _, vlan := range vlans { + vlanID, err := strconv.Atoi(strings.TrimSpace(vlan)) + if err != nil { + return []int{}, fmt.Errorf("failed to parse VLAN tag: %w", err) + } + if vlanID < minVLANIDValue || vlanID > maxVLANIDValue { + return []int{}, fmt.Errorf("VLAN tag must be between 0 and %d", maxVLANIDValue) + } + + if _, ok := seenTags[vlanID]; ok { + continue + } + seenTags[vlanID] = struct{}{} + vlanTags = append(vlanTags, vlanID) + } + + slices.Sort(vlanTags) + return vlanTags, nil +} + +func internalConfig(annotations types.Annotations) (config, error) { + c := config{} + + if v, ok := annotations.Get(apiv1_annotations.AnnotationDevices); ok { + c.devices = v + } + + if v, ok := annotations.Get(apiv1_annotations.AnnotationDirectRoutingDevice); ok { + c.directRoutingDevice = v + } + + if v, ok := annotations[apiv1_annotations.AnnotationVLANBPFBypass]; ok { + vlanTags, err := validateVLANBPFBypass(v) + if err != nil { + return config{}, fmt.Errorf("failed to parse VLAN BPF bypass list: %w", err) + } + c.vlanBPFBypass = vlanTags + } + + return c, nil +} diff --git a/src/k8s/pkg/k8sd/features/cilium/internal_test.go b/src/k8s/pkg/k8sd/features/cilium/internal_test.go new file mode 100644 index 000000000..14af95736 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/internal_test.go @@ -0,0 +1,147 @@ +package cilium + +import ( + "testing" + + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" + . "github.com/onsi/gomega" +) + +func TestInternalConfig(t *testing.T) { + for _, tc := range []struct { + name string + annotations map[string]string + expectedConfig config + expectError bool + }{ + { + name: "Empty", + annotations: map[string]string{}, + expectedConfig: config{ + devices: "", + directRoutingDevice: "", + vlanBPFBypass: nil, + }, + expectError: false, + }, + { + name: "Valid", + annotations: map[string]string{ + apiv1_annotations.AnnotationDevices: "eth+ lxdbr+", + apiv1_annotations.AnnotationDirectRoutingDevice: "eth0", + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,3", + }, + expectedConfig: config{ + devices: "eth+ lxdbr+", + directRoutingDevice: "eth0", + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + { + name: "Single valid VLAN", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1}, + }, + expectError: false, + }, + { + name: "Multiple valid VLANs", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,3,4,5", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3, 4, 5}, + }, + expectError: false, + }, + { + name: "Wildcard VLAN", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "0", + }, + expectedConfig: config{ + vlanBPFBypass: []int{0}, + }, + expectError: false, + }, + { + name: "Invalid VLAN tag format", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "abc", + }, + expectError: true, + }, + { + name: "VLAN tag out of range", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "4095", + }, + expectError: true, + }, + { + name: "VLAN tag negative", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "-1", + }, + expectError: true, + }, + { + name: "Duplicate VLAN tags", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,2,2,3", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + { + name: "Mixed spaces and commas", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: " 1, 2,3 ,4 , 5 ", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3, 4, 5}, + }, + expectError: false, + }, + { + name: "Invalid mixed with valid", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "1,abc,3", + }, + expectError: true, + }, + { + name: "Nil annotations", + annotations: nil, + expectedConfig: config{}, + expectError: false, + }, + { + name: "VLAN with curly braces", + annotations: map[string]string{ + apiv1_annotations.AnnotationVLANBPFBypass: "{1,2,3}", + }, + expectedConfig: config{ + vlanBPFBypass: []int{1, 2, 3}, + }, + expectError: false, + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + parsed, err := internalConfig(tc.annotations) + if tc.expectError { + g.Expect(err).To(HaveOccurred()) + } else { + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(parsed).To(Equal(tc.expectedConfig)) + } + }) + } +} diff --git a/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go b/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go index 54feb61ce..b642be977 100644 --- a/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go +++ b/src/k8s/pkg/k8sd/features/cilium/loadbalancer.go @@ -12,8 +12,8 @@ import ( const ( lbEnabledMsgTmpl = "enabled, %s mode" - lbDeleteFailedMsgTmpl = "Failed to delete Cilium Load Balancer, the error was: %v" - lbDeployFailedMsgTmpl = "Failed to deploy Cilium Load Balancer, the error was: %v" + LbDeleteFailedMsgTmpl = "Failed to delete Cilium Load Balancer, the error was: %v" + LbDeployFailedMsgTmpl = "Failed to deploy Cilium Load Balancer, the error was: %v" ) // ApplyLoadBalancer assumes that the managed Cilium CNI is already installed on the cluster. It will fail if that is not the case. @@ -31,7 +31,7 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(lbDeleteFailedMsgTmpl, err), + Message: fmt.Sprintf(LbDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ @@ -46,23 +46,24 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, - Message: fmt.Sprintf(lbDeployFailedMsgTmpl, err), + Message: fmt.Sprintf(LbDeployFailedMsgTmpl, err), }, err } - if loadbalancer.GetBGPMode() { + switch { + case loadbalancer.GetBGPMode(): return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, Message: fmt.Sprintf(lbEnabledMsgTmpl, "BGP"), }, nil - } else if loadbalancer.GetL2Mode() { + case loadbalancer.GetL2Mode(): return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, Message: fmt.Sprintf(lbEnabledMsgTmpl, "L2"), }, nil - } else { + default: return types.FeatureStatus{ Enabled: true, Version: CiliumAgentImageTag, @@ -198,7 +199,7 @@ func waitForRequiredLoadBalancerCRDs(ctx context.Context, snap snap.Snap, bgpMod requiredCount := len(requiredCRDs) for _, resource := range resources.APIResources { if _, ok := requiredCRDs[resource.Name]; ok { - requiredCount = requiredCount - 1 + requiredCount-- } } return requiredCount == 0, nil diff --git a/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go b/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go index 96f283c98..22aa25e5d 100644 --- a/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go +++ b/src/k8s/pkg/k8sd/features/cilium/loadbalancer_test.go @@ -3,16 +3,16 @@ package cilium_test import ( "context" "errors" + "fmt" "testing" - . "github.com/onsi/gomega" - "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/client/kubernetes" "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" v1 "k8s.io/api/apps/v1" metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" fakediscovery "k8s.io/client-go/discovery/fake" @@ -43,7 +43,7 @@ func TestLoadBalancerDisabled(t *testing.T) { g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.LbDeleteFailedMsgTmpl, err))) g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) @@ -52,6 +52,7 @@ func TestLoadBalancerDisabled(t *testing.T) { g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) + t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -115,111 +116,150 @@ func TestLoadBalancerEnabled(t *testing.T) { g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(Equal(fmt.Sprintf(cilium.LbDeployFailedMsgTmpl, err))) g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) - g.Expect(callArgs.Values["l2announcements"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) - g.Expect(callArgs.Values["bgpControlPlane"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) + l2announcements, ok := callArgs.Values["l2announcements"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(l2announcements["enabled"]).To(Equal(lbCfg.GetL2Mode())) + bgpControlPlane, ok := callArgs.Values["bgpControlPlane"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(bgpControlPlane["enabled"]).To(Equal(lbCfg.GetBGPMode())) }) - t.Run("Success", func(t *testing.T) { - g := NewWithT(t) - helmM := &helmmock.Mock{ - // setting changed == true to check for restart annotation - ApplyChanged: true, - } - clientset := fake.NewSimpleClientset( - &v1.Deployment{ - ObjectMeta: metav1.ObjectMeta{ - Name: "cilium-operator", - Namespace: "kube-system", + for _, tc := range []struct { + name string + l2Mode bool + bGPMode bool + statusMessage string + }{ + { + name: "SuccessL2Mode", + l2Mode: true, + bGPMode: false, + statusMessage: "enabled, L2 mode", + }, + { + name: "SuccessBGPMode", + l2Mode: false, + bGPMode: true, + statusMessage: "enabled, BGP mode", + }, + { + name: "SuccessUnknownMode", + l2Mode: false, + bGPMode: false, + statusMessage: "enabled, Unknown mode", + }, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium-operator", + Namespace: "kube-system", + }, }, - }, - &v1.DaemonSet{ - ObjectMeta: metav1.ObjectMeta{ - Name: "cilium", - Namespace: "kube-system", + &v1.DaemonSet{ + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + }, }, - }, - ) - fd, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) - g.Expect(ok).To(BeTrue()) - fd.Resources = []*metav1.APIResourceList{ - { - GroupVersion: "cilium.io/v2alpha1", - APIResources: []metav1.APIResource{ - {Name: "ciliuml2announcementpolicies"}, - {Name: "ciliumloadbalancerippools"}, - {Name: "ciliumbgppeeringpolicies"}, + ) + fd, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fd.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "cilium.io/v2alpha1", + APIResources: []metav1.APIResource{ + {Name: "ciliuml2announcementpolicies"}, + {Name: "ciliumloadbalancerippools"}, + {Name: "ciliumbgppeeringpolicies"}, + }, }, - }, - } - snapM := &snapmock.Snap{ - Mock: snapmock.Mock{ - HelmClient: helmM, - KubernetesClient: &kubernetes.Client{Interface: clientset}, - }, - } - lbCfg := types.LoadBalancer{ - Enabled: ptr.To(true), - // setting both modes to true for testing purposes - L2Mode: ptr.To(true), - L2Interfaces: ptr.To([]string{"eth0", "eth1"}), - BGPMode: ptr.To(true), - BGPLocalASN: ptr.To(64512), - BGPPeerAddress: ptr.To("10.0.0.1/32"), - BGPPeerASN: ptr.To(64513), - BGPPeerPort: ptr.To(179), - CIDRs: ptr.To([]string{"192.0.2.0/24"}), - IPRanges: ptr.To([]types.LoadBalancer_IPRange{ - {Start: "20.0.20.100", Stop: "20.0.20.200"}, - }), - } - networkCfg := types.Network{ - Enabled: ptr.To(true), - } + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + lbCfg := types.LoadBalancer{ + Enabled: ptr.To(true), + // setting both modes to true for testing purposes + L2Mode: ptr.To(tc.l2Mode), + L2Interfaces: ptr.To([]string{"eth0", "eth1"}), + BGPMode: ptr.To(tc.bGPMode), + BGPLocalASN: ptr.To(64512), + BGPPeerAddress: ptr.To("10.0.0.1/32"), + BGPPeerASN: ptr.To(64513), + BGPPeerPort: ptr.To(179), + CIDRs: ptr.To([]string{"192.0.2.0/24"}), + IPRanges: ptr.To([]types.LoadBalancer_IPRange{ + {Start: "20.0.20.100", Stop: "20.0.20.200"}, + }), + } + networkCfg := types.Network{ + Enabled: ptr.To(true), + } - status, err := cilium.ApplyLoadBalancer(context.Background(), snapM, lbCfg, networkCfg, nil) + status, err := cilium.ApplyLoadBalancer(context.Background(), snapM, lbCfg, networkCfg, nil) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(status.Enabled).To(BeTrue()) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(tc.statusMessage)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) - firstCallArgs := helmM.ApplyCalledWith[0] - g.Expect(firstCallArgs.Chart).To(Equal(cilium.ChartCilium)) - g.Expect(firstCallArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) - g.Expect(firstCallArgs.Values["l2announcements"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) - g.Expect(firstCallArgs.Values["bgpControlPlane"].(map[string]any)["enabled"]).To(Equal(lbCfg.GetL2Mode())) + firstCallArgs := helmM.ApplyCalledWith[0] + g.Expect(firstCallArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(firstCallArgs.State).To(Equal(helm.StateUpgradeOnlyOrDeleted(networkCfg.GetEnabled()))) + l2announcements, ok := firstCallArgs.Values["l2announcements"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(l2announcements["enabled"]).To(Equal(lbCfg.GetL2Mode())) + bgpControlPlane, ok := firstCallArgs.Values["bgpControlPlane"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(bgpControlPlane["enabled"]).To(Equal(lbCfg.GetBGPMode())) - secondCallArgs := helmM.ApplyCalledWith[1] - g.Expect(secondCallArgs.Chart).To(Equal(cilium.ChartCiliumLoadBalancer)) - g.Expect(secondCallArgs.State).To(Equal(helm.StatePresent)) - validateLoadBalancerValues(t, secondCallArgs.Values, lbCfg) + secondCallArgs := helmM.ApplyCalledWith[1] + g.Expect(secondCallArgs.Chart).To(Equal(cilium.ChartCiliumLoadBalancer)) + g.Expect(secondCallArgs.State).To(Equal(helm.StatePresent)) + validateLoadBalancerValues(t, secondCallArgs.Values, lbCfg) - // check if cilium-operator and cilium daemonset are restarted - deployment, err := clientset.AppsV1().Deployments("kube-system").Get(context.Background(), "cilium-operator", metav1.GetOptions{}) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(deployment.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) - daemonSet, err := clientset.AppsV1().DaemonSets("kube-system").Get(context.Background(), "cilium", metav1.GetOptions{}) - g.Expect(err).ToNot(HaveOccurred()) - g.Expect(daemonSet.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) - }) + // check if cilium-operator and cilium daemonset are restarted + deployment, err := clientset.AppsV1().Deployments("kube-system").Get(context.Background(), "cilium-operator", metav1.GetOptions{}) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(deployment.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) + daemonSet, err := clientset.AppsV1().DaemonSets("kube-system").Get(context.Background(), "cilium", metav1.GetOptions{}) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(daemonSet.Spec.Template.Annotations).To(HaveKey("kubectl.kubernetes.io/restartedAt")) + }) + } } func validateLoadBalancerValues(t *testing.T, values map[string]interface{}, lbCfg types.LoadBalancer) { g := NewWithT(t) - l2 := values["l2"].(map[string]any) + l2, ok := values["l2"].(map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(l2["enabled"]).To(Equal(lbCfg.GetL2Mode())) g.Expect(l2["interfaces"]).To(Equal(lbCfg.GetL2Interfaces())) - cidrs := values["ipPool"].(map[string]any)["cidrs"].([]map[string]any) + ipPool, ok := values["ipPool"].(map[string]any) + g.Expect(ok).To(BeTrue()) + cidrs, ok := ipPool["cidrs"].([]map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(cidrs).To(HaveLen(len(lbCfg.GetIPRanges()) + len(lbCfg.GetCIDRs()))) for _, cidr := range lbCfg.GetCIDRs() { g.Expect(cidrs).To(ContainElement(map[string]any{"cidr": cidr})) @@ -228,10 +268,12 @@ func validateLoadBalancerValues(t *testing.T, values map[string]interface{}, lbC g.Expect(cidrs).To(ContainElement(map[string]any{"start": ipRange.Start, "stop": ipRange.Stop})) } - bgp := values["bgp"].(map[string]any) + bgp, ok := values["bgp"].(map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(bgp["enabled"]).To(Equal(lbCfg.GetBGPMode())) g.Expect(bgp["localASN"]).To(Equal(lbCfg.GetBGPLocalASN())) - neighbors := bgp["neighbors"].([]map[string]any) + neighbors, ok := bgp["neighbors"].([]map[string]any) + g.Expect(ok).To(BeTrue()) g.Expect(neighbors).To(HaveLen(1)) g.Expect(neighbors[0]["peerAddress"]).To(Equal(lbCfg.GetBGPPeerAddress())) g.Expect(neighbors[0]["peerASN"]).To(Equal(lbCfg.GetBGPPeerASN())) diff --git a/src/k8s/pkg/k8sd/features/cilium/network.go b/src/k8s/pkg/k8sd/features/cilium/network.go index 4418e8b1d..08e4a62f7 100644 --- a/src/k8s/pkg/k8sd/features/cilium/network.go +++ b/src/k8s/pkg/k8sd/features/cilium/network.go @@ -3,6 +3,7 @@ package cilium import ( "context" "fmt" + "strings" "github.com/canonical/k8s/pkg/client/helm" "github.com/canonical/k8s/pkg/k8sd/types" @@ -17,18 +18,24 @@ const ( networkDeployFailedMsgTmpl = "Failed to deploy Cilium Network, the error was: %v" ) -// ApplyNetwork will deploy Cilium when cfg.Enabled is true. -// ApplyNetwork will remove Cilium when cfg.Enabled is false. +// required for unittests. +var ( + getMountPath = utils.GetMountPath + getMountPropagationType = utils.GetMountPropagationType +) + +// ApplyNetwork will deploy Cilium when network.Enabled is true. +// ApplyNetwork will remove Cilium when network.Enabled is false. // ApplyNetwork requires that bpf and cgroups2 are already mounted and available when running under strict snap confinement. If they are not, it will fail (since Cilium will not have the required permissions to mount them). // ApplyNetwork requires that `/sys` is mounted as a shared mount when running under classic snap confinement. This is to ensure that Cilium will be able to automatically mount bpf and cgroups2 on the pods. // ApplyNetwork will always return a FeatureStatus indicating the current status of the // deployment. // ApplyNetwork returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ types.Annotations) (types.FeatureStatus, error) { +func ApplyNetwork(ctx context.Context, snap snap.Snap, localhostAddress string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() - if !cfg.GetEnabled() { + if !network.GetEnabled() { if _, err := m.Apply(ctx, ChartCilium, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall network: %w", err) return types.FeatureStatus{ @@ -44,9 +51,19 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, nil } - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + config, err := internalConfig(annotations) + if err != nil { + err = fmt.Errorf("failed to parse annotations: %w", err) + return types.FeatureStatus{ + Enabled: false, + Version: CiliumAgentImageTag, + Message: fmt.Sprintf(networkDeployFailedMsgTmpl, err), + }, err + } + + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) if err != nil { - err = fmt.Errorf("invalid kube-proxy --cluster-cidr value: %v", err) + err = fmt.Errorf("invalid kube-proxy --cluster-cidr value: %w", err) return types.FeatureStatus{ Enabled: false, Version: CiliumAgentImageTag, @@ -54,7 +71,23 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, err } + ciliumNodePortValues := map[string]any{ + "enabled": true, + // kube-proxy also binds to the same port for health checks so we need to disable it + "enableHealthCheck": false, + } + + if config.directRoutingDevice != "" { + ciliumNodePortValues["directRoutingDevice"] = config.directRoutingDevice + } + + bpfValues := map[string]any{} + if config.vlanBPFBypass != nil { + bpfValues["vlanBypass"] = config.vlanBPFBypass + } + values := map[string]any{ + "bpf": bpfValues, "image": map[string]any{ "repository": ciliumAgentImageRepo, "tag": CiliumAgentImageTag, @@ -87,14 +120,26 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type "clusterPoolIPv6PodCIDRList": ipv6CIDR, }, }, - "nodePort": map[string]any{ - "enabled": true, + "envoy": map[string]any{ + "enabled": false, // 1.16+ installs envoy as a standalone daemonset by default if not explicitly disabled }, + // https://docs.cilium.io/en/v1.15/network/kubernetes/kubeproxy-free/#kube-proxy-hybrid-modes + "nodePort": ciliumNodePortValues, "disableEnvoyVersionCheck": true, + // socketLB requires an endpoint to the apiserver that's not managed by the kube-proxy + // so we point to the localhost:secureport to talk to either the kube-apiserver or the kube-apiserver-proxy + "k8sServiceHost": strings.Trim(localhostAddress, "[]"), // Cilium already adds the brackets for ipv6 addresses, so we need to remove them + "k8sServicePort": apiserver.GetSecurePort(), + // This flag enables the runtime device detection which is set to true by default in Cilium 1.16+ + "enableRuntimeDeviceDetection": true, + } + + if config.devices != "" { + values["devices"] = config.devices } if snap.Strict() { - bpfMnt, err := utils.GetMountPath("bpf") + bpfMnt, err := getMountPath("bpf") if err != nil { err = fmt.Errorf("failed to get bpf mount path: %w", err) return types.FeatureStatus{ @@ -104,7 +149,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type }, err } - cgrMnt, err := utils.GetMountPath("cgroup2") + cgrMnt, err := getMountPath("cgroup2") if err != nil { err = fmt.Errorf("failed to get cgroup2 mount path: %w", err) return types.FeatureStatus{ @@ -127,7 +172,7 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type "hostRoot": cgrMnt, } } else { - pt, err := utils.GetMountPropagationType("/sys") + pt, err := getMountPropagationType("/sys") if err != nil { err = fmt.Errorf("failed to get mount propagation type for /sys: %w", err) return types.FeatureStatus{ @@ -139,7 +184,8 @@ func ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, _ type if pt == utils.MountPropagationPrivate { onLXD, err := snap.OnLXD(ctx) if err != nil { - log.FromContext(ctx).Error(err, "Failed to check if running on LXD") + logger := log.FromContext(ctx) + logger.Error(err, "Failed to check if running on LXD") } if onLXD { err := fmt.Errorf("/sys is not a shared mount on the LXD container, this might be resolved by updating LXD on the host to version 5.0.2 or newer") diff --git a/src/k8s/pkg/k8sd/features/cilium/network_test.go b/src/k8s/pkg/k8sd/features/cilium/network_test.go index 175cd95a0..eb40caaef 100644 --- a/src/k8s/pkg/k8sd/features/cilium/network_test.go +++ b/src/k8s/pkg/k8sd/features/cilium/network_test.go @@ -1,24 +1,31 @@ -package cilium_test +package cilium import ( "context" "errors" + "fmt" "testing" - . "github.com/onsi/gomega" - + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/cilium" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" - "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/types" "github.com/canonical/k8s/pkg/snap" snapmock "github.com/canonical/k8s/pkg/snap/mock" "github.com/canonical/k8s/pkg/utils" + . "github.com/onsi/gomega" + "k8s.io/klog/v2" + "k8s.io/klog/v2/ktesting" "k8s.io/utils/ptr" ) // NOTE(hue): status.Message is not checked sometimes to avoid unnecessary complexity +var annotations = types.Annotations{ + apiv1_annotations.AnnotationDevices: "eth+ lxdbr+", + apiv1_annotations.AnnotationDirectRoutingDevice: "eth0", +} + func TestNetworkDisabled(t *testing.T) { t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -32,23 +39,27 @@ func TestNetworkDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeleteFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) + t.Run("Success", func(t *testing.T) { g := NewWithT(t) @@ -58,20 +69,23 @@ func TestNetworkDisabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(false), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(Equal(cilium.DisabledMsg)) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(DisabledMsg)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) g.Expect(callArgs.Values).To(BeNil()) }) @@ -87,18 +101,22 @@ func TestNetworkEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("invalid-cidr"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) g.Expect(err).To(HaveOccurred()) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) - g.Expect(helmM.ApplyCalledWith).To(HaveLen(0)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) }) + t.Run("Strict", func(t *testing.T) { g := NewWithT(t) @@ -109,24 +127,28 @@ func TestNetworkEnabled(t *testing.T) { Strict: true, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, annotations) g.Expect(err).ToNot(HaveOccurred()) g.Expect(status.Enabled).To(BeTrue()) - g.Expect(status.Message).To(Equal(cilium.EnabledMsg)) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(EnabledMsg)) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateNetworkValues(t, callArgs.Values, cfg, snapM) + validateNetworkValues(g, callArgs.Values, network, snapM) }) + t.Run("HelmApplyFails", func(t *testing.T) { g := NewWithT(t) @@ -139,31 +161,212 @@ func TestNetworkEnabled(t *testing.T) { HelmClient: helmM, }, } - cfg := types.Network{ + network := types.Network{ Enabled: ptr.To(true), PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - status, err := cilium.ApplyNetwork(context.Background(), snapM, cfg, nil) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, annotations) g.Expect(err).To(MatchError(applyErr)) g.Expect(status.Enabled).To(BeFalse()) - g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) - g.Expect(status.Version).To(Equal(cilium.CiliumAgentImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) callArgs := helmM.ApplyCalledWith[0] - g.Expect(callArgs.Chart).To(Equal(cilium.ChartCilium)) + g.Expect(callArgs.Chart).To(Equal(ChartCilium)) g.Expect(callArgs.State).To(Equal(helm.StatePresent)) - validateNetworkValues(t, callArgs.Values, cfg, snapM) + validateNetworkValues(g, callArgs.Values, network, snapM) }) } -func validateNetworkValues(t *testing.T, values map[string]any, cfg types.Network, snap snap.Snap) { - t.Helper() - g := NewWithT(t) +func TestNetworkMountPath(t *testing.T) { + for _, tc := range []struct { + name string + }{ + {name: "bpf"}, + {name: "cgroup2"}, + } { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + mountPathErr := fmt.Errorf("%s not found", tc.name) + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: true, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + getMountPath = func(fsType string) (string, error) { + if fsType == tc.name { + return "", mountPathErr + } + return tc.name, nil + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(mountPathErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + } +} + +func TestNetworkMountPropagationType(t *testing.T) { + t.Run("failedGetMountSys", func(t *testing.T) { + g := NewWithT(t) + + mountErr := errors.New("/sys not found") + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return "", mountErr + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(mountErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + + t.Run("MountPropagationPrivateOnLXDError", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + OnLXDErr: errors.New("failed to check LXD"), + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + logger := ktesting.NewLogger(t, ktesting.NewConfig(ktesting.BufferLogs(true))) + ctx := klog.NewContext(context.Background(), logger) + + status, err := ApplyNetwork(ctx, snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + testingLogger, ok := logger.GetSink().(ktesting.Underlier) + if !ok { + panic("Should have had a ktesting LogSink!?") + } + g.Expect(testingLogger.GetBuffer().String()).To(ContainSubstring("Failed to check if running on LXD")) + }) + + t.Run("MountPropagationPrivateOnLXD", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(path string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + OnLXD: true, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } + + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) + + t.Run("MountPropagationPrivate", func(t *testing.T) { + g := NewWithT(t) + + getMountPropagationType = func(_ string) (utils.MountPropagationType, error) { + return utils.MountPropagationPrivate, nil + } + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + Strict: false, + }, + } + network := types.Network{ + Enabled: ptr.To(true), + PodCIDR: ptr.To("192.0.2.0/24,2001:db8::/32"), + } + apiserver := types.APIServer{ + SecurePort: ptr.To(6443), + } - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(cfg.GetPodCIDR()) + status, err := ApplyNetwork(context.Background(), snapM, "127.0.0.1", apiserver, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(Equal(fmt.Sprintf(networkDeployFailedMsgTmpl, err))) + + g.Expect(status.Version).To(Equal(CiliumAgentImageTag)) + g.Expect(helmM.ApplyCalledWith).To(BeEmpty()) + }) +} + +func validateNetworkValues(g Gomega, values map[string]any, network types.Network, snap snap.Snap) { + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(network.GetPodCIDR()) g.Expect(err).ToNot(HaveOccurred()) bpfMount, err := utils.GetMountPath("bpf") @@ -177,8 +380,20 @@ func validateNetworkValues(t *testing.T, values map[string]any, cfg types.Networ g.Expect(values["cgroup"].(map[string]any)["hostRoot"]).To(Equal(cgrMount)) } + g.Expect(values["k8sServiceHost"]).To(Equal("127.0.0.1")) + g.Expect(values["k8sServicePort"]).To(Equal(6443)) g.Expect(values["ipam"].(map[string]any)["operator"].(map[string]any)["clusterPoolIPv4PodCIDRList"]).To(Equal(ipv4CIDR)) g.Expect(values["ipam"].(map[string]any)["operator"].(map[string]any)["clusterPoolIPv6PodCIDRList"]).To(Equal(ipv6CIDR)) g.Expect(values["ipv4"].(map[string]any)["enabled"]).To(Equal((ipv4CIDR != ""))) g.Expect(values["ipv6"].(map[string]any)["enabled"]).To(Equal((ipv6CIDR != ""))) + + devices, exists := annotations.Get(apiv1_annotations.AnnotationDevices) + if exists { + g.Expect(values["devices"]).To(Equal(devices)) + } + + directRoutingDevice, exists := annotations.Get(apiv1_annotations.AnnotationDirectRoutingDevice) + if exists { + g.Expect(values["nodePort"].(map[string]any)["directRoutingDevice"]).To(Equal(directRoutingDevice)) + } } diff --git a/src/k8s/pkg/k8sd/features/cilium/status.go b/src/k8s/pkg/k8sd/features/cilium/status.go index 65212848c..37c629abd 100644 --- a/src/k8s/pkg/k8sd/features/cilium/status.go +++ b/src/k8s/pkg/k8sd/features/cilium/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/cilium/status_test.go b/src/k8s/pkg/k8sd/features/cilium/status_test.go new file mode 100644 index 000000000..083844e5d --- /dev/null +++ b/src/k8s/pkg/k8sd/features/cilium/status_test.go @@ -0,0 +1,124 @@ +package cilium_test + +import ( + "context" + "testing" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/cilium" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" +) + +func TestCheckNetwork(t *testing.T) { + t.Run("ciliumOperatorNotReady", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("operatorNoCiliumPods", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset(&corev1.PodList{ + Items: []corev1.Pod{ + { + ObjectMeta: metav1.ObjectMeta{ + Name: "pod1", + Namespace: "kube-system", + Labels: map[string]string{"io.cilium/app": "operator"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + }, + }) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("allPodsPresent", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset(&corev1.PodList{ + Items: []corev1.Pod{ + { + ObjectMeta: metav1.ObjectMeta{ + Name: "operator", + Namespace: "kube-system", + Labels: map[string]string{"io.cilium/app": "operator"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + { + ObjectMeta: metav1.ObjectMeta{ + Name: "cilium", + Namespace: "kube-system", + Labels: map[string]string{"k8s-app": "cilium"}, + }, + Status: corev1.PodStatus{ + Phase: corev1.PodRunning, + Conditions: []corev1.PodCondition{ + {Type: corev1.PodReady, Status: corev1.ConditionTrue}, + }, + }, + }, + }, + }) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + + err := cilium.CheckNetwork(context.Background(), snapM) + g.Expect(err).NotTo(HaveOccurred()) + }) +} diff --git a/src/k8s/pkg/k8sd/features/contour/chart.go b/src/k8s/pkg/k8sd/features/contour/chart.go index ff8fa2016..1291dbe85 100644 --- a/src/k8s/pkg/k8sd/features/contour/chart.go +++ b/src/k8s/pkg/k8sd/features/contour/chart.go @@ -34,29 +34,29 @@ var ( ManifestPath: filepath.Join("charts", "ck-contour-common-1.28.2.tgz"), } - // contourGatewayProvisionerEnvoyImageRepo represents the image to use for envoy in the gateway. - contourGatewayProvisionerEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/envoyproxy/envoy" + // ContourGatewayProvisionerEnvoyImageRepo represents the image to use for envoy in the gateway. + ContourGatewayProvisionerEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/envoyproxy/envoy" // NOTE: The image version is v1.29.2 instead of 1.28.2 // to follow the upstream configuration for the contour gateway provisioner. - // contourGatewayProvisionerEnvoyImageTag is the tag to use for for envoy in the gateway. - contourGatewayProvisionerEnvoyImageTag = "v1.29.2" + // ContourGatewayProvisionerEnvoyImageTag is the tag to use for envoy in the gateway. + ContourGatewayProvisionerEnvoyImageTag = "v1.29.2" - // contourIngressEnvoyImageRepo represents the image to use for the Contour Envoy proxy. - contourIngressEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/envoy" + // ContourIngressEnvoyImageRepo represents the image to use for the Contour Envoy proxy. + ContourIngressEnvoyImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/envoy" - // contourIngressEnvoyImageTag is the tag to use for the Contour Envoy proxy image. - contourIngressEnvoyImageTag = "1.28.2-debian-12-r0" + // ContourIngressEnvoyImageTag is the tag to use for the Contour Envoy proxy image. + ContourIngressEnvoyImageTag = "1.28.2-debian-12-r0" - // contourIngressContourImageRepo represents the image to use for Contour. - contourIngressContourImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/contour" + // ContourIngressContourImageRepo represents the image to use for Contour. + ContourIngressContourImageRepo = "ghcr.io/canonical/k8s-snap/bitnami/contour" - // contourIngressContourImageTag is the tag to use for the Contour image. - contourIngressContourImageTag = "1.28.2-debian-12-r4" + // ContourIngressContourImageTag is the tag to use for the Contour image. + ContourIngressContourImageTag = "1.28.2-debian-12-r4" - // contourGatewayProvisionerContourImageRepo represents the image to use for the Contour Gateway Provisioner. - contourGatewayProvisionerContourImageRepo = "ghcr.io/canonical/k8s-snap/projectcontour/contour" + // ContourGatewayProvisionerContourImageRepo represents the image to use for the Contour Gateway Provisioner. + ContourGatewayProvisionerContourImageRepo = "ghcr.io/canonical/k8s-snap/projectcontour/contour" - // contourGatewayProvisionerContourImageTag is the tag to use for the Contour Gateway Provisioner image. - contourGatewayProvisionerContourImageTag = "v1.28.2" + // ContourGatewayProvisionerContourImageTag is the tag to use for the Contour Gateway Provisioner image. + ContourGatewayProvisionerContourImageTag = "v1.28.2" ) diff --git a/src/k8s/pkg/k8sd/features/contour/gateway.go b/src/k8s/pkg/k8sd/features/contour/gateway.go index d7b203c6b..dd52e9bff 100644 --- a/src/k8s/pkg/k8sd/features/contour/gateway.go +++ b/src/k8s/pkg/k8sd/features/contour/gateway.go @@ -11,10 +11,10 @@ import ( ) const ( - enabledMsg = "enabled" - disabledMsg = "disabled" - gatewayDeployFailedMsgTmpl = "Failed to deploy Contour Gateway, the error was: %v" - gatewayDeleteFailedMsgTmpl = "Failed to delete Contour Gateway, the error was: %v" + EnabledMsg = "enabled" + DisabledMsg = "disabled" + GatewayDeployFailedMsgTmpl = "Failed to deploy Contour Gateway, the error was: %v" + GatewayDeleteFailedMsgTmpl = "Failed to delete Contour Gateway, the error was: %v" ) // ApplyGateway will install a helm chart for contour-gateway-provisioner on the cluster when gateway.Enabled is true. @@ -32,14 +32,14 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to uninstall the contour gateway chart: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeleteFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: disabledMsg, + Version: ContourGatewayProvisionerContourImageTag, + Message: DisabledMsg, }, nil } @@ -48,8 +48,8 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to apply common contour CRDS: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } @@ -57,22 +57,22 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to wait for required contour common CRDs to be available: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } values := map[string]any{ "projectcontour": map[string]any{ "image": map[string]any{ - "repository": contourGatewayProvisionerContourImageRepo, - "tag": contourGatewayProvisionerContourImageTag, + "repository": ContourGatewayProvisionerContourImageRepo, + "tag": ContourGatewayProvisionerContourImageTag, }, }, "envoyproxy": map[string]any{ "image": map[string]any{ - "repository": contourGatewayProvisionerEnvoyImageRepo, - "tag": contourGatewayProvisionerEnvoyImageTag, + "repository": ContourGatewayProvisionerEnvoyImageRepo, + "tag": ContourGatewayProvisionerEnvoyImageTag, }, }, } @@ -81,20 +81,20 @@ func ApplyGateway(ctx context.Context, snap snap.Snap, gateway types.Gateway, ne err = fmt.Errorf("failed to install the contour gateway chart: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourGatewayProvisionerContourImageTag, - Message: fmt.Sprintf(gatewayDeployFailedMsgTmpl, err), + Version: ContourGatewayProvisionerContourImageTag, + Message: fmt.Sprintf(GatewayDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourGatewayProvisionerContourImageTag, - Message: enabledMsg, + Version: ContourGatewayProvisionerContourImageTag, + Message: EnabledMsg, }, nil } // waitForRequiredContourCommonCRDs waits for the required contour CRDs to be available -// by checking the API resources by group version +// by checking the API resources by group version. func waitForRequiredContourCommonCRDs(ctx context.Context, snap snap.Snap) error { client, err := snap.KubernetesClient("") if err != nil { diff --git a/src/k8s/pkg/k8sd/features/contour/gateway_test.go b/src/k8s/pkg/k8sd/features/contour/gateway_test.go new file mode 100644 index 000000000..b4ac8e0b4 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/contour/gateway_test.go @@ -0,0 +1,209 @@ +package contour_test + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/contour" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestGatewayDisabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeleteFailedMsgTmpl, err))) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(contour.DisabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} + +func TestGatewayEnabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*v1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []v1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []v1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyGateway(context.Background(), snapM, gateway, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + + values := helmM.ApplyCalledWith[1].Values + contourValues, ok := values["projectcontour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourImage, ok := contourValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(contourImage["repository"]).To(Equal(contour.ContourGatewayProvisionerContourImageRepo)) + g.Expect(contourImage["tag"]).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + envoyValues, ok := values["envoyproxy"].(map[string]any) + g.Expect(ok).To(BeTrue()) + envoyImage, ok := envoyValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(envoyImage["repository"]).To(Equal(contour.ContourGatewayProvisionerEnvoyImageRepo)) + g.Expect(envoyImage["tag"]).To(Equal(contour.ContourGatewayProvisionerEnvoyImageTag)) + }) + + t.Run("CrdDeploymentFailed", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*v1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []v1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []v1.APIResource{}, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + gateway := types.Gateway{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*2) + defer cancel() + status, err := contour.ApplyGateway(ctx, snapM, gateway, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("failed to wait for required contour common CRDs")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourGatewayProvisionerContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.GatewayDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} diff --git a/src/k8s/pkg/k8sd/features/contour/ingress.go b/src/k8s/pkg/k8sd/features/contour/ingress.go index a43023d1c..ea5a6e030 100644 --- a/src/k8s/pkg/k8sd/features/contour/ingress.go +++ b/src/k8s/pkg/k8sd/features/contour/ingress.go @@ -11,8 +11,8 @@ import ( ) const ( - ingressDeleteFailedMsgTmpl = "Failed to delete Contour Ingress, the error was: %v" - ingressDeployFailedMsgTmpl = "Failed to deploy Contour Ingress, the error was: %v" + IngressDeleteFailedMsgTmpl = "Failed to delete Contour Ingress, the error was: %v" + IngressDeployFailedMsgTmpl = "Failed to deploy Contour Ingress, the error was: %v" ) // ApplyIngress will install the contour helm chart when ingress.Enabled is true. @@ -24,7 +24,7 @@ const ( // deployment. // ApplyIngress returns an error if anything fails. The error is also wrapped in the .Message field of the // returned FeatureStatus. -// Contour CRDS are applied through a ck-contour common chart (Overlap with gateway) +// Contour CRDS are applied through a ck-contour common chart (Overlap with gateway). func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ types.Network, _ types.Annotations) (types.FeatureStatus, error) { m := snap.HelmClient() @@ -33,14 +33,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to uninstall ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeleteFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeleteFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: disabledMsg, + Version: ContourIngressContourImageTag, + Message: DisabledMsg, }, nil } @@ -49,8 +49,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to apply common contour CRDS: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -58,8 +58,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to wait for required contour common CRDs to be available: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -70,8 +70,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ "envoy": map[string]any{ "image": map[string]any{ "registry": "", - "repository": contourIngressEnvoyImageRepo, - "tag": contourIngressEnvoyImageTag, + "repository": ContourIngressEnvoyImageRepo, + "tag": ContourIngressEnvoyImageTag, }, }, "contour": map[string]any{ @@ -83,16 +83,23 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ }, "image": map[string]any{ "registry": "", - "repository": contourIngressContourImageRepo, - "tag": contourIngressContourImageTag, + "repository": ContourIngressContourImageRepo, + "tag": ContourIngressContourImageTag, }, }, } if ingress.GetEnableProxyProtocol() { - contour := values["contour"].(map[string]any) + contour, ok := values["contour"].(map[string]any) + if !ok { + err := fmt.Errorf("unexpected type for contour values") + return types.FeatureStatus{ + Enabled: false, + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), + }, err + } contour["extraArgs"] = []string{"--use-proxy-protocol"} - } changed, err := m.Apply(ctx, chartContour, helm.StatePresent, values) @@ -100,8 +107,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to enable ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } @@ -110,8 +117,8 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to rollout restart contour to apply ingress: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } } @@ -127,14 +134,14 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to install the delegation resource for default TLS secret: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourIngressContourImageTag, - Message: enabledMsg, + Version: ContourIngressContourImageTag, + Message: EnabledMsg, }, nil } @@ -142,16 +149,16 @@ func ApplyIngress(ctx context.Context, snap snap.Snap, ingress types.Ingress, _ err = fmt.Errorf("failed to uninstall the delegation resource for default TLS secret: %w", err) return types.FeatureStatus{ Enabled: false, - Version: contourIngressContourImageTag, - Message: fmt.Sprintf(ingressDeployFailedMsgTmpl, err), + Version: ContourIngressContourImageTag, + Message: fmt.Sprintf(IngressDeployFailedMsgTmpl, err), }, err } return types.FeatureStatus{ Enabled: true, - Version: contourIngressContourImageTag, - Message: enabledMsg, + Version: ContourIngressContourImageTag, + Message: EnabledMsg, }, nil } diff --git a/src/k8s/pkg/k8sd/features/contour/ingress_test.go b/src/k8s/pkg/k8sd/features/contour/ingress_test.go new file mode 100644 index 000000000..9547d2032 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/contour/ingress_test.go @@ -0,0 +1,404 @@ +package contour_test + +import ( + "context" + "errors" + "fmt" + "testing" + "time" + + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/contour" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + v1 "k8s.io/api/apps/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestIngressDisabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeleteFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(false), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.DisabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) +} + +func TestIngressEnabled(t *testing.T) { + t.Run("HelmFailed", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + + status, err := contour.ApplyIngress(context.Background(), snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + }) + + t.Run("SuccessWithEnabledProxyProtocol", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + EnableProxyProtocol: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + }) + + t.Run("SuccessWithDefaultTLSSecret", func(t *testing.T) { + g := NewWithT(t) + defaultTLSSecret := "secret" + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "ck-ingress-contour-contour", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + DefaultTLSSecret: ptr.To(defaultTLSSecret), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).NotTo(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(contour.EnabledMsg)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(3)) + validateIngressValues(g, helmM.ApplyCalledWith[1].Values, ingress) + g.Expect(helmM.ApplyCalledWith[2].Values["defaultTLSSecret"]).To(Equal(defaultTLSSecret)) + }) + + t.Run("NoCR", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset() + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/metav1", + APIResources: []metav1.APIResource{}, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*2) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + }) + + t.Run("NoDeployment", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{ + ApplyChanged: true, + } + clientset := fake.NewSimpleClientset( + &v1.Deployment{ + ObjectMeta: metav1.ObjectMeta{ + Name: "dummy", + Namespace: "projectcontour", + }, + }) + fakeDiscovery, ok := clientset.Discovery().(*fakediscovery.FakeDiscovery) + g.Expect(ok).To(BeTrue()) + fakeDiscovery.Resources = []*metav1.APIResourceList{ + { + GroupVersion: "projectcontour.io/v1alpha1", + APIResources: []metav1.APIResource{ + {Name: "contourconfigurations"}, + {Name: "contourdeployments"}, + {Name: "extensionservices"}, + }, + }, + { + GroupVersion: "projectcontour.io/v1", + APIResources: []metav1.APIResource{ + {Name: "tlscertificatedelegations"}, + {Name: "httpproxies"}, + }, + }, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{ + Interface: clientset, + }, + }, + } + network := types.Network{} + ingress := types.Ingress{ + Enabled: ptr.To(true), + } + ctx, cancel := context.WithTimeout(context.Background(), time.Second*20) + defer cancel() + + status, err := contour.ApplyIngress(ctx, snapM, ingress, network, nil) + + g.Expect(err).To(HaveOccurred()) + g.Expect(err.Error()).To(ContainSubstring("failed to rollout restart contour to apply ingress")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(contour.ContourIngressContourImageTag)) + g.Expect(status.Message).To(Equal(fmt.Sprintf(contour.IngressDeployFailedMsgTmpl, err))) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(2)) + }) +} + +func validateIngressValues(g Gomega, values map[string]interface{}, ingress types.Ingress) { + contourValues, ok := values["contour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourImage, ok := contourValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(contourImage["repository"]).To(Equal(contour.ContourIngressContourImageRepo)) + g.Expect(contourImage["tag"]).To(Equal(contour.ContourIngressContourImageTag)) + envoyValues, ok := values["envoy"].(map[string]any) + g.Expect(ok).To(BeTrue()) + envoyImage, ok := envoyValues["image"].(map[string]any) + g.Expect(ok).To(BeTrue()) + g.Expect(envoyImage["repository"]).To(Equal(contour.ContourIngressEnvoyImageRepo)) + g.Expect(envoyImage["tag"]).To(Equal(contour.ContourIngressEnvoyImageTag)) + + if ingress.GetEnableProxyProtocol() { + conturExtraValues, ok := values["contour"].(map[string]any) + g.Expect(ok).To(BeTrue()) + contourExtraArgs, ok := conturExtraValues["extraArgs"].([]string) + g.Expect(ok).To(BeTrue()) + g.Expect(contourExtraArgs[0]).To(Equal("--use-proxy-protocol")) + } +} diff --git a/src/k8s/pkg/k8sd/features/contour/register.go b/src/k8s/pkg/k8sd/features/contour/register.go index 4c3797032..59cc84792 100644 --- a/src/k8s/pkg/k8sd/features/contour/register.go +++ b/src/k8s/pkg/k8sd/features/contour/register.go @@ -8,9 +8,9 @@ import ( func init() { images.Register( - fmt.Sprintf("%s:%s", contourIngressEnvoyImageRepo, contourIngressEnvoyImageTag), - fmt.Sprintf("%s:%s", contourIngressContourImageRepo, contourIngressContourImageTag), - fmt.Sprintf("%s:%s", contourGatewayProvisionerContourImageRepo, contourGatewayProvisionerContourImageTag), - fmt.Sprintf("%s:%s", contourGatewayProvisionerEnvoyImageRepo, contourGatewayProvisionerEnvoyImageTag), + fmt.Sprintf("%s:%s", ContourIngressEnvoyImageRepo, ContourIngressEnvoyImageTag), + fmt.Sprintf("%s:%s", ContourIngressContourImageRepo, ContourIngressContourImageTag), + fmt.Sprintf("%s:%s", ContourGatewayProvisionerContourImageRepo, ContourGatewayProvisionerContourImageTag), + fmt.Sprintf("%s:%s", ContourGatewayProvisionerEnvoyImageRepo, ContourGatewayProvisionerEnvoyImageTag), ) } diff --git a/src/k8s/pkg/k8sd/features/coredns/chart.go b/src/k8s/pkg/k8sd/features/coredns/chart.go index 72cf75bfb..eaa940c5e 100644 --- a/src/k8s/pkg/k8sd/features/coredns/chart.go +++ b/src/k8s/pkg/k8sd/features/coredns/chart.go @@ -8,15 +8,15 @@ import ( var ( // chartCoreDNS represents manifests to deploy CoreDNS. - chart = helm.InstallableChart{ + Chart = helm.InstallableChart{ Name: "ck-dns", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "coredns-1.29.0.tgz"), + ManifestPath: filepath.Join("charts", "coredns-1.36.0.tgz"), } // imageRepo is the image to use for CoreDNS. imageRepo = "ghcr.io/canonical/coredns" - // imageTag is the tag to use for the CoreDNS image. - imageTag = "1.11.1-ck4" + // ImageTag is the tag to use for the CoreDNS image. + ImageTag = "1.11.3-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/coredns/coredns.go b/src/k8s/pkg/k8sd/features/coredns/coredns.go index 28402e707..79e66d6e6 100644 --- a/src/k8s/pkg/k8sd/features/coredns/coredns.go +++ b/src/k8s/pkg/k8sd/features/coredns/coredns.go @@ -29,17 +29,17 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. m := snap.HelmClient() if !dns.GetEnabled() { - if _, err := m.Apply(ctx, chart, helm.StateDeleted, nil); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StateDeleted, nil); err != nil { err = fmt.Errorf("failed to uninstall coredns: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deleteFailedMsgTmpl, err), }, "", err } return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: disabledMsg, }, "", nil } @@ -47,7 +47,7 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. values := map[string]any{ "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, "service": map[string]any{ "name": "coredns", @@ -82,11 +82,11 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. }, } - if _, err := m.Apply(ctx, chart, helm.StatePresent, values); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StatePresent, values); err != nil { err = fmt.Errorf("failed to apply coredns: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } @@ -96,7 +96,7 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. err = fmt.Errorf("failed to create kubernetes client: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } @@ -105,14 +105,14 @@ func ApplyDNS(ctx context.Context, snap snap.Snap, dns types.DNS, kubelet types. err = fmt.Errorf("failed to retrieve the coredns service: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, "", err } return types.FeatureStatus{ Enabled: true, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(enabledMsgTmpl, dnsIP), }, dnsIP, err } diff --git a/src/k8s/pkg/k8sd/features/coredns/coredns_test.go b/src/k8s/pkg/k8sd/features/coredns/coredns_test.go new file mode 100644 index 000000000..0d10e2bd3 --- /dev/null +++ b/src/k8s/pkg/k8sd/features/coredns/coredns_test.go @@ -0,0 +1,196 @@ +package coredns_test + +import ( + "context" + "errors" + "strings" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/client/kubernetes" + "github.com/canonical/k8s/pkg/k8sd/features/coredns" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + corev1 "k8s.io/api/core/v1" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" +) + +func TestDisabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(false), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring(applyErr.Error()))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(ContainSubstring("failed to uninstall coredns")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(callArgs.Values).To(BeNil()) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(false), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(Equal("disabled")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + g.Expect(callArgs.Values).To(BeNil()) + }) +} + +func TestEnabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring(applyErr.Error()))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Message).To(ContainSubstring("failed to apply coredns")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) + t.Run("HelmApplySuccessServiceFails", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + clientset := fake.NewSimpleClientset() + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(MatchError(ContainSubstring("services \"coredns\" not found"))) + g.Expect(str).To(BeEmpty()) + g.Expect(status.Message).To(ContainSubstring("failed to retrieve the coredns service")) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + clusterIp := "10.96.0.10" + corednsService := &corev1.Service{ + ObjectMeta: metav1.ObjectMeta{ + Name: "coredns", + Namespace: "kube-system", + }, + Spec: corev1.ServiceSpec{ + ClusterIP: clusterIp, + }, + } + clientset := fake.NewSimpleClientset(corednsService) + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + KubernetesClient: &kubernetes.Client{Interface: clientset}, + }, + } + dns := types.DNS{ + Enabled: ptr.To(true), + } + kubelet := types.Kubelet{} + + status, str, err := coredns.ApplyDNS(context.Background(), snapM, dns, kubelet, nil) + + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(str).To(Equal(clusterIp)) + g.Expect(status.Message).To(ContainSubstring("enabled at " + clusterIp)) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(coredns.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(coredns.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + validateValues(g, callArgs.Values, dns, kubelet) + }) +} + +func validateValues(g Gomega, values map[string]any, dns types.DNS, kubelet types.Kubelet) { + service := values["service"].(map[string]any) + g.Expect(service["clusterIP"]).To(Equal(kubelet.GetClusterDNS())) + + servers := values["servers"].([]map[string]any) + plugins := servers[0]["plugins"].([]map[string]any) + g.Expect(plugins[3]["parameters"]).To(ContainSubstring(kubelet.GetClusterDomain())) + g.Expect(plugins[5]["parameters"]).To(ContainSubstring(strings.Join(dns.GetUpstreamNameservers(), " "))) +} diff --git a/src/k8s/pkg/k8sd/features/coredns/register.go b/src/k8s/pkg/k8sd/features/coredns/register.go index 1dc54d943..566d31f09 100644 --- a/src/k8s/pkg/k8sd/features/coredns/register.go +++ b/src/k8s/pkg/k8sd/features/coredns/register.go @@ -8,6 +8,6 @@ import ( func init() { images.Register( - fmt.Sprintf("%s:%s", imageRepo, imageTag), + fmt.Sprintf("%s:%s", imageRepo, ImageTag), ) } diff --git a/src/k8s/pkg/k8sd/features/coredns/status.go b/src/k8s/pkg/k8sd/features/coredns/status.go index 629eabe87..94d3fe66a 100644 --- a/src/k8s/pkg/k8sd/features/coredns/status.go +++ b/src/k8s/pkg/k8sd/features/coredns/status.go @@ -5,7 +5,6 @@ import ( "fmt" "github.com/canonical/k8s/pkg/snap" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" ) diff --git a/src/k8s/pkg/k8sd/features/implementation_default.go b/src/k8s/pkg/k8sd/features/implementation_default.go index b78a67fae..3cad3e3b6 100644 --- a/src/k8s/pkg/k8sd/features/implementation_default.go +++ b/src/k8s/pkg/k8sd/features/implementation_default.go @@ -4,18 +4,20 @@ import ( "github.com/canonical/k8s/pkg/k8sd/features/cilium" "github.com/canonical/k8s/pkg/k8sd/features/coredns" "github.com/canonical/k8s/pkg/k8sd/features/localpv" + "github.com/canonical/k8s/pkg/k8sd/features/metallb" metrics_server "github.com/canonical/k8s/pkg/k8sd/features/metrics-server" ) // Default implements the Canonical Kubernetes built-in features. -// Cilium is used for networking (network + load-balancer + ingress + gateway). +// Cilium is used for networking (network + ingress + gateway). +// MetalLB is used for LoadBalancer. // CoreDNS is used for DNS. // MetricsServer is used for metrics-server. // LocalPV Rawfile CSI is used for local-storage. var Implementation Interface = &implementation{ applyDNS: coredns.ApplyDNS, applyNetwork: cilium.ApplyNetwork, - applyLoadBalancer: cilium.ApplyLoadBalancer, + applyLoadBalancer: metallb.ApplyLoadBalancer, applyIngress: cilium.ApplyIngress, applyGateway: cilium.ApplyGateway, applyMetricsServer: metrics_server.ApplyMetricsServer, diff --git a/src/k8s/pkg/k8sd/features/interface.go b/src/k8s/pkg/k8sd/features/interface.go index 18eb4a978..3753af341 100644 --- a/src/k8s/pkg/k8sd/features/interface.go +++ b/src/k8s/pkg/k8sd/features/interface.go @@ -12,7 +12,7 @@ type Interface interface { // ApplyDNS is used to configure the DNS feature on Canonical Kubernetes. ApplyDNS(context.Context, snap.Snap, types.DNS, types.Kubelet, types.Annotations) (types.FeatureStatus, string, error) // ApplyNetwork is used to configure the network feature on Canonical Kubernetes. - ApplyNetwork(context.Context, snap.Snap, types.Network, types.Annotations) (types.FeatureStatus, error) + ApplyNetwork(context.Context, snap.Snap, string, types.APIServer, types.Network, types.Annotations) (types.FeatureStatus, error) // ApplyLoadBalancer is used to configure the load-balancer feature on Canonical Kubernetes. ApplyLoadBalancer(context.Context, snap.Snap, types.LoadBalancer, types.Network, types.Annotations) (types.FeatureStatus, error) // ApplyIngress is used to configure the ingress controller feature on Canonical Kubernetes. @@ -28,7 +28,7 @@ type Interface interface { // implementation implements Interface. type implementation struct { applyDNS func(context.Context, snap.Snap, types.DNS, types.Kubelet, types.Annotations) (types.FeatureStatus, string, error) - applyNetwork func(context.Context, snap.Snap, types.Network, types.Annotations) (types.FeatureStatus, error) + applyNetwork func(context.Context, snap.Snap, string, types.APIServer, types.Network, types.Annotations) (types.FeatureStatus, error) applyLoadBalancer func(context.Context, snap.Snap, types.LoadBalancer, types.Network, types.Annotations) (types.FeatureStatus, error) applyIngress func(context.Context, snap.Snap, types.Ingress, types.Network, types.Annotations) (types.FeatureStatus, error) applyGateway func(context.Context, snap.Snap, types.Gateway, types.Network, types.Annotations) (types.FeatureStatus, error) @@ -40,8 +40,8 @@ func (i *implementation) ApplyDNS(ctx context.Context, snap snap.Snap, dns types return i.applyDNS(ctx, snap, dns, kubelet, annotations) } -func (i *implementation) ApplyNetwork(ctx context.Context, snap snap.Snap, cfg types.Network, annotations types.Annotations) (types.FeatureStatus, error) { - return i.applyNetwork(ctx, snap, cfg, annotations) +func (i *implementation) ApplyNetwork(ctx context.Context, snap snap.Snap, localhostAddress string, apiserver types.APIServer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { + return i.applyNetwork(ctx, snap, localhostAddress, apiserver, network, annotations) } func (i *implementation) ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.LoadBalancer, network types.Network, annotations types.Annotations) (types.FeatureStatus, error) { diff --git a/src/k8s/pkg/k8sd/features/localpv/chart.go b/src/k8s/pkg/k8sd/features/localpv/chart.go index 5a655fc57..8dd2248af 100644 --- a/src/k8s/pkg/k8sd/features/localpv/chart.go +++ b/src/k8s/pkg/k8sd/features/localpv/chart.go @@ -7,8 +7,8 @@ import ( ) var ( - // chart represents manifests to deploy Rawfile LocalPV CSI. - chart = helm.InstallableChart{ + // Chart represents manifests to deploy Rawfile LocalPV CSI. + Chart = helm.InstallableChart{ Name: "ck-storage", Namespace: "kube-system", ManifestPath: filepath.Join("charts", "rawfile-csi-0.9.0.tgz"), @@ -16,8 +16,8 @@ var ( // imageRepo is the repository to use for Rawfile LocalPV CSI. imageRepo = "ghcr.io/canonical/rawfile-localpv" - // imageTag is the image tag to use for Rawfile LocalPV CSI. - imageTag = "0.8.0-ck4" + // ImageTag is the image tag to use for Rawfile LocalPV CSI. + ImageTag = "0.8.0-ck4" // csiNodeDriverImage is the image to use for the CSI node driver. csiNodeDriverImage = "ghcr.io/canonical/k8s-snap/sig-storage/csi-node-driver-registrar:v2.10.1" diff --git a/src/k8s/pkg/k8sd/features/localpv/localpv.go b/src/k8s/pkg/k8sd/features/localpv/localpv.go index bd812b443..8555ff088 100644 --- a/src/k8s/pkg/k8sd/features/localpv/localpv.go +++ b/src/k8s/pkg/k8sd/features/localpv/localpv.go @@ -38,13 +38,13 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora "csiDriverArgs": []string{"--args", "rawfile", "csi-driver", "--disable-metrics"}, "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, }, "node": map[string]any{ "image": map[string]any{ "repository": imageRepo, - "tag": imageTag, + "tag": ImageTag, }, "storage": map[string]any{ "path": cfg.GetLocalPath(), @@ -58,19 +58,19 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora }, } - if _, err := m.Apply(ctx, chart, helm.StatePresentOrDeleted(cfg.GetEnabled()), values); err != nil { + if _, err := m.Apply(ctx, Chart, helm.StatePresentOrDeleted(cfg.GetEnabled()), values); err != nil { if cfg.GetEnabled() { err = fmt.Errorf("failed to install rawfile-csi helm package: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deployFailedMsgTmpl, err), }, err } else { err = fmt.Errorf("failed to delete rawfile-csi helm package: %w", err) return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(deleteFailedMsgTmpl, err), }, err } @@ -79,13 +79,13 @@ func ApplyLocalStorage(ctx context.Context, snap snap.Snap, cfg types.LocalStora if cfg.GetEnabled() { return types.FeatureStatus{ Enabled: true, - Version: imageTag, + Version: ImageTag, Message: fmt.Sprintf(enabledMsg, cfg.GetLocalPath()), }, nil } else { return types.FeatureStatus{ Enabled: false, - Version: imageTag, + Version: ImageTag, Message: disabledMsg, }, nil } diff --git a/src/k8s/pkg/k8sd/features/localpv/localpv_test.go b/src/k8s/pkg/k8sd/features/localpv/localpv_test.go new file mode 100644 index 000000000..2a8da057c --- /dev/null +++ b/src/k8s/pkg/k8sd/features/localpv/localpv_test.go @@ -0,0 +1,154 @@ +package localpv_test + +import ( + "context" + "errors" + "testing" + + "github.com/canonical/k8s/pkg/client/helm" + helmmock "github.com/canonical/k8s/pkg/client/helm/mock" + "github.com/canonical/k8s/pkg/k8sd/features/localpv" + "github.com/canonical/k8s/pkg/k8sd/types" + snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + "k8s.io/utils/ptr" +) + +func TestDisabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(false), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + + validateValues(g, callArgs.Values, cfg) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(false), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StateDeleted)) + + validateValues(g, callArgs.Values, cfg) + }) +} + +func TestEnabled(t *testing.T) { + t.Run("HelmApplyFails", func(t *testing.T) { + g := NewWithT(t) + + applyErr := errors.New("failed to apply") + helmM := &helmmock.Mock{ + ApplyErr: applyErr, + } + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(true), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).To(MatchError(applyErr)) + g.Expect(status.Enabled).To(BeFalse()) + g.Expect(status.Message).To(ContainSubstring(applyErr.Error())) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + + validateValues(g, callArgs.Values, cfg) + }) + t.Run("Success", func(t *testing.T) { + g := NewWithT(t) + + helmM := &helmmock.Mock{} + snapM := &snapmock.Snap{ + Mock: snapmock.Mock{ + HelmClient: helmM, + }, + } + cfg := types.LocalStorage{ + Enabled: ptr.To(true), + Default: ptr.To(true), + ReclaimPolicy: ptr.To("reclaim-policy"), + LocalPath: ptr.To("local-path"), + } + + status, err := localpv.ApplyLocalStorage(context.Background(), snapM, cfg, nil) + + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(status.Enabled).To(BeTrue()) + g.Expect(status.Version).To(Equal(localpv.ImageTag)) + g.Expect(helmM.ApplyCalledWith).To(HaveLen(1)) + + callArgs := helmM.ApplyCalledWith[0] + g.Expect(callArgs.Chart).To(Equal(localpv.Chart)) + g.Expect(callArgs.State).To(Equal(helm.StatePresent)) + + validateValues(g, callArgs.Values, cfg) + }) +} + +func validateValues(g Gomega, values map[string]any, cfg types.LocalStorage) { + sc := values["storageClass"].(map[string]any) + g.Expect(sc["isDefault"]).To(Equal(cfg.GetDefault())) + g.Expect(sc["reclaimPolicy"]).To(Equal(cfg.GetReclaimPolicy())) + + storage := values["node"].(map[string]any)["storage"].(map[string]any) + g.Expect(storage["path"]).To(Equal(cfg.GetLocalPath())) +} diff --git a/src/k8s/pkg/k8sd/features/localpv/register.go b/src/k8s/pkg/k8sd/features/localpv/register.go index b9f5f644b..084f6a40b 100644 --- a/src/k8s/pkg/k8sd/features/localpv/register.go +++ b/src/k8s/pkg/k8sd/features/localpv/register.go @@ -9,7 +9,7 @@ import ( func init() { images.Register( // Rawfile LocalPV CSI driver images - fmt.Sprintf("%s:%s", imageRepo, imageTag), + fmt.Sprintf("%s:%s", imageRepo, ImageTag), // CSI images csiNodeDriverImage, csiProvisionerImage, diff --git a/src/k8s/pkg/k8sd/features/metallb/chart.go b/src/k8s/pkg/k8sd/features/metallb/chart.go index a7bdffd2a..b3cc9f8f6 100644 --- a/src/k8s/pkg/k8sd/features/metallb/chart.go +++ b/src/k8s/pkg/k8sd/features/metallb/chart.go @@ -11,7 +11,7 @@ var ( ChartMetalLB = helm.InstallableChart{ Name: "metallb", Namespace: "metallb-system", - ManifestPath: filepath.Join("charts", "metallb-0.14.5.tgz"), + ManifestPath: filepath.Join("charts", "metallb-0.14.8.tgz"), } // ChartMetalLBLoadBalancer represents manifests to deploy MetalLB L2 or BGP resources. @@ -22,16 +22,16 @@ var ( } // controllerImageRepo is the image to use for metallb-controller. - controllerImageRepo = "ghcr.io/canonical/k8s-snap/metallb/controller" + controllerImageRepo = "ghcr.io/canonical/metallb-controller" // ControllerImageTag is the tag to use for metallb-controller. - ControllerImageTag = "v0.14.5" + ControllerImageTag = "v0.14.8-ck0" // speakerImageRepo is the image to use for metallb-speaker. - speakerImageRepo = "ghcr.io/canonical/k8s-snap/metallb/speaker" + speakerImageRepo = "ghcr.io/canonical/metallb-speaker" // speakerImageTag is the tag to use for metallb-speaker. - speakerImageTag = "v0.14.5" + speakerImageTag = "v0.14.8-ck0" // frrImageRepo is the image to use for frrouting. frrImageRepo = "ghcr.io/canonical/k8s-snap/frrouting/frr" diff --git a/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go b/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go index ef199d600..6477def8e 100644 --- a/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go +++ b/src/k8s/pkg/k8sd/features/metallb/loadbalancer.go @@ -47,19 +47,20 @@ func ApplyLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types.L }, err } - if loadbalancer.GetBGPMode() { + switch { + case loadbalancer.GetBGPMode(): return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, Message: fmt.Sprintf(enabledMsgTmpl, "BGP"), }, nil - } else if loadbalancer.GetL2Mode() { + case loadbalancer.GetL2Mode(): return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, Message: fmt.Sprintf(enabledMsgTmpl, "L2"), }, nil - } else { + default: return types.FeatureStatus{ Enabled: true, Version: ControllerImageTag, @@ -90,12 +91,14 @@ func enableLoadBalancer(ctx context.Context, snap snap.Snap, loadbalancer types. "repository": controllerImageRepo, "tag": ControllerImageTag, }, + "command": "/controller", }, "speaker": map[string]any{ "image": map[string]any{ "repository": speakerImageRepo, "tag": speakerImageTag, }, + "command": "/speaker", // TODO(neoaggelos): make frr enable/disable configurable through an annotation // We keep it disabled by default "frr": map[string]any{ @@ -170,13 +173,13 @@ func waitForRequiredLoadBalancerCRDs(ctx context.Context, snap snap.Snap, bgpMod return false, nil } - requiredCRDs := map[string]bool{ - "metallb.io/v1beta1:ipaddresspools": true, - "metallb.io/v1beta1:l2advertisements": true, + requiredCRDs := map[string]struct{}{ + "metallb.io/v1beta1:ipaddresspools": {}, + "metallb.io/v1beta1:l2advertisements": {}, } if bgpMode { - requiredCRDs["metallb.io/v1beta2:bgppeers"] = true - requiredCRDs["metallb.io/v1beta1:bgpadvertisements"] = true + requiredCRDs["metallb.io/v1beta2:bgppeers"] = struct{}{} + requiredCRDs["metallb.io/v1beta1:bgpadvertisements"] = struct{}{} } requiredCount := len(requiredCRDs) diff --git a/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go b/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go index 0bc1fb6f1..7ba673c78 100644 --- a/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go +++ b/src/k8s/pkg/k8sd/features/metallb/loadbalancer_test.go @@ -5,18 +5,17 @@ import ( "errors" "testing" - . "github.com/onsi/gomega" - metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" - fakediscovery "k8s.io/client-go/discovery/fake" - "k8s.io/client-go/kubernetes/fake" - "k8s.io/utils/ptr" - "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" "github.com/canonical/k8s/pkg/client/kubernetes" "github.com/canonical/k8s/pkg/k8sd/features/metallb" "github.com/canonical/k8s/pkg/k8sd/types" snapmock "github.com/canonical/k8s/pkg/snap/mock" + . "github.com/onsi/gomega" + metav1 "k8s.io/apimachinery/pkg/apis/meta/v1" + fakediscovery "k8s.io/client-go/discovery/fake" + "k8s.io/client-go/kubernetes/fake" + "k8s.io/utils/ptr" ) func TestDisabled(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/features/metrics-server/chart.go b/src/k8s/pkg/k8sd/features/metrics-server/chart.go index 8c4e46958..0c01aa6ba 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/chart.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/chart.go @@ -11,12 +11,12 @@ var ( chart = helm.InstallableChart{ Name: "metrics-server", Namespace: "kube-system", - ManifestPath: filepath.Join("charts", "metrics-server-3.12.0.tgz"), + ManifestPath: filepath.Join("charts", "metrics-server-3.12.2.tgz"), } // imageRepo is the image to use for metrics-server. imageRepo = "ghcr.io/canonical/metrics-server" // imageTag is the image tag to use for metrics-server. - imageTag = "0.7.0-ck1" + imageTag = "0.7.2-ck0" ) diff --git a/src/k8s/pkg/k8sd/features/metrics-server/internal.go b/src/k8s/pkg/k8sd/features/metrics-server/internal.go index 2300b7139..c927e348d 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/internal.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/internal.go @@ -1,10 +1,8 @@ package metrics_server -import "github.com/canonical/k8s/pkg/k8sd/types" - -const ( - annotationImageRepo = "k8sd/v1alpha1/metrics-server/image-repo" - annotationImageTag = "k8sd/v1alpha1/metrics-server/image-tag" +import ( + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/metrics-server" + "github.com/canonical/k8s/pkg/k8sd/types" ) type config struct { @@ -18,10 +16,10 @@ func internalConfig(annotations types.Annotations) config { imageTag: imageTag, } - if v, ok := annotations.Get(annotationImageRepo); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationImageRepo); ok { config.imageRepo = v } - if v, ok := annotations.Get(annotationImageTag); ok { + if v, ok := annotations.Get(apiv1_annotations.AnnotationImageTag); ok { config.imageTag = v } diff --git a/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go b/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go index ced59104f..d9cdc8370 100644 --- a/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go +++ b/src/k8s/pkg/k8sd/features/metrics-server/metrics_server_test.go @@ -2,8 +2,10 @@ package metrics_server_test import ( "context" + "errors" "testing" + apiv1_annotations "github.com/canonical/k8s-snap-api/api/v1/annotations/metrics-server" "github.com/canonical/k8s/pkg/client/helm" helmmock "github.com/canonical/k8s/pkg/client/helm/mock" metrics_server "github.com/canonical/k8s/pkg/k8sd/features/metrics-server" @@ -14,29 +16,51 @@ import ( ) func TestApplyMetricsServer(t *testing.T) { + helmErr := errors.New("failed to apply") for _, tc := range []struct { name string config types.MetricsServer expectState helm.State + helmError error }{ { - name: "Enable", + name: "EnableWithoutHelmError", config: types.MetricsServer{ Enabled: utils.Pointer(true), }, expectState: helm.StatePresent, + helmError: nil, }, { - name: "Disable", + name: "DisableWithoutHelmError", config: types.MetricsServer{ Enabled: utils.Pointer(false), }, expectState: helm.StateDeleted, + helmError: nil, + }, + { + name: "EnableWithHelmError", + config: types.MetricsServer{ + Enabled: utils.Pointer(true), + }, + expectState: helm.StatePresent, + helmError: helmErr, + }, + { + name: "DisableWithHelmError", + config: types.MetricsServer{ + Enabled: utils.Pointer(false), + }, + expectState: helm.StateDeleted, + helmError: helmErr, }, } { t.Run(tc.name, func(t *testing.T) { g := NewWithT(t) - h := &helmmock.Mock{} + h := &helmmock.Mock{ + ApplyErr: tc.helmError, + } s := &snapmock.Snap{ Mock: snapmock.Mock{ HelmClient: h, @@ -44,16 +68,23 @@ func TestApplyMetricsServer(t *testing.T) { } status, err := metrics_server.ApplyMetricsServer(context.Background(), s, tc.config, nil) - g.Expect(err).ToNot(HaveOccurred()) + if tc.helmError == nil { + g.Expect(err).ToNot(HaveOccurred()) + } else { + g.Expect(err).To(HaveOccurred()) + } g.Expect(h.ApplyCalledWith).To(ConsistOf(SatisfyAll( HaveField("Chart.Name", Equal("metrics-server")), HaveField("Chart.Namespace", Equal("kube-system")), HaveField("State", Equal(tc.expectState)), ))) - if tc.config.GetEnabled() { + switch { + case errors.Is(tc.helmError, helmErr): + g.Expect(status.Message).To(ContainSubstring(helmErr.Error())) + case tc.config.GetEnabled(): g.Expect(status.Message).To(Equal("enabled")) - } else { + default: g.Expect(status.Message).To(Equal("disabled")) } }) @@ -72,12 +103,12 @@ func TestApplyMetricsServer(t *testing.T) { Enabled: utils.Pointer(true), } annotations := types.Annotations{ - "k8sd/v1alpha1/metrics-server/image-repo": "custom-image", - "k8sd/v1alpha1/metrics-server/image-tag": "custom-tag", + apiv1_annotations.AnnotationImageRepo: "custom-image", + apiv1_annotations.AnnotationImageTag: "custom-tag", } status, err := metrics_server.ApplyMetricsServer(context.Background(), s, cfg, annotations) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(h.ApplyCalledWith).To(ConsistOf(HaveField("Values", HaveKeyWithValue("image", SatisfyAll( HaveKeyWithValue("repository", "custom-image"), HaveKeyWithValue("tag", "custom-tag"), diff --git a/src/k8s/pkg/k8sd/pki/k8sdqlite.go b/src/k8s/pkg/k8sd/pki/k8sdqlite.go index b1d74bc37..0e81d823d 100644 --- a/src/k8s/pkg/k8sd/pki/k8sdqlite.go +++ b/src/k8s/pkg/k8sd/pki/k8sdqlite.go @@ -64,7 +64,7 @@ func (c *K8sDqlitePKI) CompleteCertificates() error { return fmt.Errorf("k8s-dqlite certificate not specified and generating self-signed certificates is not allowed") } - template, err := pkiutil.GenerateCertificate(pkix.Name{CommonName: "k8s"}, c.notBefore, c.notAfter, false, append(c.dnsSANs, c.hostname), append(c.ipSANs, net.IP{127, 0, 0, 1})) + template, err := pkiutil.GenerateCertificate(pkix.Name{CommonName: "k8s"}, c.notBefore, c.notAfter, false, append(c.dnsSANs, c.hostname), append(c.ipSANs, net.ParseIP("127.0.0.1"), net.ParseIP("::1"))) if err != nil { return fmt.Errorf("failed to generate k8s-dqlite certificate: %w", err) } diff --git a/src/k8s/pkg/k8sd/pki/worker.go b/src/k8s/pkg/k8sd/pki/worker.go index d945f053f..590b53a99 100644 --- a/src/k8s/pkg/k8sd/pki/worker.go +++ b/src/k8s/pkg/k8sd/pki/worker.go @@ -48,7 +48,7 @@ func (c *ControlPlanePKI) CompleteWorkerNodePKI(hostname string, nodeIP net.IP, c.notAfter, false, []string{hostname}, - []net.IP{{127, 0, 0, 1}, nodeIP}, + []net.IP{net.ParseIP("127.0.0.1"), net.ParseIP("::1"), nodeIP}, ) if err != nil { return nil, fmt.Errorf("failed to generate kubelet certificate for hostname=%s address=%s: %w", hostname, nodeIP.String(), err) diff --git a/src/k8s/pkg/k8sd/pki/worker_test.go b/src/k8s/pkg/k8sd/pki/worker_test.go index 4c9fac5cc..65d85215e 100644 --- a/src/k8s/pkg/k8sd/pki/worker_test.go +++ b/src/k8s/pkg/k8sd/pki/worker_test.go @@ -13,7 +13,6 @@ import ( ) func TestControlPlanePKI_CompleteWorkerNodePKI(t *testing.T) { - g := NewWithT(t) notBefore := time.Now() serverCACert, serverCAKey, err := pkiutil.GenerateSelfSignedCA(pkix.Name{CommonName: "kubernetes-ca"}, notBefore, notBefore.AddDate(1, 0, 0), 2048) diff --git a/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf b/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf new file mode 100644 index 000000000..39273db4a --- /dev/null +++ b/src/k8s/pkg/k8sd/setup/auth-token-webhook.conf @@ -0,0 +1,17 @@ +apiVersion: v1 +kind: Config +clusters: + - name: k8s-token-auth-service + cluster: + certificate-authority: "cluster.crt" + tls-server-name: 127.0.0.1 + server: "https://auth-webhook.url" +current-context: webhook +contexts: +- context: + cluster: k8s-token-auth-service + user: k8s-apiserver + name: webhook +users: + - name: k8s-apiserver + user: {} diff --git a/src/k8s/pkg/k8sd/setup/certificates.go b/src/k8s/pkg/k8sd/setup/certificates.go index 283101e53..c8e67d227 100644 --- a/src/k8s/pkg/k8sd/setup/certificates.go +++ b/src/k8s/pkg/k8sd/setup/certificates.go @@ -8,6 +8,7 @@ import ( "github.com/canonical/k8s/pkg/k8sd/pki" "github.com/canonical/k8s/pkg/snap" + "github.com/canonical/k8s/pkg/utils" ) // ensureFile creates fname with the specified contents, mode and owner bits. @@ -39,7 +40,7 @@ func ensureFile(fname string, contents string, uid, gid int, mode fs.FileMode) ( var contentChanged bool if contents != string(origContent) { - if err := os.WriteFile(fname, []byte(contents), mode); err != nil { + if err := utils.WriteFile(fname, []byte(contents), mode); err != nil { return false, fmt.Errorf("failed to write: %w", err) } contentChanged = true @@ -73,7 +74,7 @@ func ensureFiles(uid, gid int, mode fs.FileMode, files map[string]string) (bool, // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureExtDatastorePKI(snap snap.Snap, certificates *pki.ExternalDatastorePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.EtcdPKIDir(), "ca.crt"): certificates.DatastoreCACert, filepath.Join(snap.EtcdPKIDir(), "client.key"): certificates.DatastoreClientKey, filepath.Join(snap.EtcdPKIDir(), "client.crt"): certificates.DatastoreClientCert, @@ -84,7 +85,7 @@ func EnsureExtDatastorePKI(snap snap.Snap, certificates *pki.ExternalDatastorePK // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureK8sDqlitePKI(snap snap.Snap, certificates *pki.K8sDqlitePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.K8sDqliteStateDir(), "cluster.crt"): certificates.K8sDqliteCert, filepath.Join(snap.K8sDqliteStateDir(), "cluster.key"): certificates.K8sDqliteKey, }) @@ -94,7 +95,7 @@ func EnsureK8sDqlitePKI(snap snap.Snap, certificates *pki.K8sDqlitePKI) (bool, e // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureControlPlanePKI(snap snap.Snap, certificates *pki.ControlPlanePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.KubernetesPKIDir(), "apiserver-kubelet-client.crt"): certificates.APIServerKubeletClientCert, filepath.Join(snap.KubernetesPKIDir(), "apiserver-kubelet-client.key"): certificates.APIServerKubeletClientKey, filepath.Join(snap.KubernetesPKIDir(), "apiserver.crt"): certificates.APIServerCert, @@ -116,7 +117,7 @@ func EnsureControlPlanePKI(snap snap.Snap, certificates *pki.ControlPlanePKI) (b // and have the correct content, permissions and ownership. // It returns true if one or more files were updated and any error that occurred. func EnsureWorkerPKI(snap snap.Snap, certificates *pki.WorkerNodePKI) (bool, error) { - return ensureFiles(snap.UID(), snap.GID(), 0600, map[string]string{ + return ensureFiles(snap.UID(), snap.GID(), 0o600, map[string]string{ filepath.Join(snap.KubernetesPKIDir(), "ca.crt"): certificates.CACert, filepath.Join(snap.KubernetesPKIDir(), "client-ca.crt"): certificates.ClientCACert, filepath.Join(snap.KubernetesPKIDir(), "kubelet.crt"): certificates.KubeletCert, diff --git a/src/k8s/pkg/k8sd/setup/certificates_internal_test.go b/src/k8s/pkg/k8sd/setup/certificates_internal_test.go index 665574dda..9454458d3 100644 --- a/src/k8s/pkg/k8sd/setup/certificates_internal_test.go +++ b/src/k8s/pkg/k8sd/setup/certificates_internal_test.go @@ -14,12 +14,12 @@ func TestEnsureFile(t *testing.T) { tempDir := t.TempDir() fname := filepath.Join(tempDir, "test") - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) createdFile, _ := os.ReadFile(fname) - g.Expect(string(createdFile) == "test").To(BeTrue()) + g.Expect(string(createdFile)).To(Equal("test")) }) t.Run("DeleteFile", func(t *testing.T) { @@ -28,13 +28,13 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // Create a file with some content. - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) // Delete the file. - updated, err = ensureFile(fname, "", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) _, err = os.Stat(fname) @@ -47,26 +47,26 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // Create a file with some content. - updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) // ensureFile with same content should return that the file was not updated. - updated, err = ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "test", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) // Change the content and ensureFile should return that the file was updated. - updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeTrue()) createdFile, _ := os.ReadFile(fname) - g.Expect(string(createdFile) == "new content").To(BeTrue()) + g.Expect(string(createdFile)).To(Equal("new content")) // Change permissions and ensureFile should return that the file was not updated. - updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0666) - g.Expect(err).To(BeNil()) + updated, err = ensureFile(fname, "new content", os.Getuid(), os.Getgid(), 0o666) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) }) @@ -76,8 +76,8 @@ func TestEnsureFile(t *testing.T) { fname := filepath.Join(tempDir, "test") // ensureFile on inexistent file with empty content should return that the file was not updated. - updated, err := ensureFile(fname, "", os.Getuid(), os.Getgid(), 0777) - g.Expect(err).To(BeNil()) + updated, err := ensureFile(fname, "", os.Getuid(), os.Getgid(), 0o777) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(updated).To(BeFalse()) }) } diff --git a/src/k8s/pkg/k8sd/setup/certificates_test.go b/src/k8s/pkg/k8sd/setup/certificates_test.go index 32526faec..8ea74313e 100644 --- a/src/k8s/pkg/k8sd/setup/certificates_test.go +++ b/src/k8s/pkg/k8sd/setup/certificates_test.go @@ -28,7 +28,7 @@ func TestEnsureK8sDqlitePKI(t *testing.T) { } _, err := setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "cluster.crt"), @@ -37,7 +37,7 @@ func TestEnsureK8sDqlitePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -71,7 +71,7 @@ func TestEnsureControlPlanePKI(t *testing.T) { } _, err := setup.EnsureControlPlanePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "apiserver-kubelet-client.crt"), @@ -92,7 +92,7 @@ func TestEnsureControlPlanePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -115,7 +115,7 @@ func TestEnsureWorkerPKI(t *testing.T) { } _, err := setup.EnsureWorkerPKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "ca.crt"), @@ -126,7 +126,7 @@ func TestEnsureWorkerPKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -147,7 +147,7 @@ func TestExtDatastorePKI(t *testing.T) { } _, err := setup.EnsureExtDatastorePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) expectedFiles := []string{ filepath.Join(tempDir, "ca.crt"), @@ -157,7 +157,7 @@ func TestExtDatastorePKI(t *testing.T) { for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } } @@ -186,7 +186,7 @@ func TestEmptyCert(t *testing.T) { // Should create files _, err := setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) certificates = &pki.K8sDqlitePKI{ K8sDqliteCert: "", @@ -195,10 +195,10 @@ func TestEmptyCert(t *testing.T) { // Should delete files _, err = setup.EnsureK8sDqlitePKI(mock, certificates) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) for _, file := range expectedFiles { _, err := os.Stat(file) - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) } } diff --git a/src/k8s/pkg/k8sd/setup/containerd.go b/src/k8s/pkg/k8sd/setup/containerd.go index dbf1545ec..eb9909124 100644 --- a/src/k8s/pkg/k8sd/setup/containerd.go +++ b/src/k8s/pkg/k8sd/setup/containerd.go @@ -108,12 +108,12 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs return fmt.Errorf("failed to render containerd config.toml: %w", err) } - if err := os.WriteFile(filepath.Join(snap.ContainerdConfigDir(), "config.toml"), b, 0600); err != nil { + if err := utils.WriteFile(filepath.Join(snap.ContainerdConfigDir(), "config.toml"), b, 0o600); err != nil { return fmt.Errorf("failed to write config.toml: %w", err) } if _, err := snaputil.UpdateServiceArguments(snap, "containerd", map[string]string{ - "--address": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), + "--address": snap.ContainerdSocketPath(), "--config": filepath.Join(snap.ContainerdConfigDir(), "config.toml"), "--root": snap.ContainerdRootDir(), "--state": snap.ContainerdStateDir(), @@ -131,7 +131,7 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs if err := utils.CopyFile(snap.CNIPluginsBinary(), cniBinary); err != nil { return fmt.Errorf("failed to copy cni plugin binary: %w", err) } - if err := os.Chmod(cniBinary, 0700); err != nil { + if err := os.Chmod(cniBinary, 0o700); err != nil { return fmt.Errorf("failed to chmod cni plugin binary: %w", err) } if err := os.Chown(cniBinary, snap.UID(), snap.GID()); err != nil { @@ -159,6 +159,27 @@ func Containerd(snap snap.Snap, extraContainerdConfig map[string]any, extraArgs } } + if err := saveSnapContainerdPaths(snap); err != nil { + return err + } + + return nil +} + +func saveSnapContainerdPaths(s snap.Snap) error { + // Write the containerd-related paths to files to properly clean-up on removal. + m := map[string]string{ + "containerd-socket-path": s.ContainerdSocketDir(), + "containerd-config-dir": s.ContainerdConfigDir(), + "containerd-root-dir": s.ContainerdRootDir(), + "containerd-cni-bin-dir": s.CNIBinDir(), + } + + for filename, content := range m { + if err := utils.WriteFile(filepath.Join(s.LockFilesDir(), filename), []byte(content), 0o600); err != nil { + return fmt.Errorf("failed to write %s: %w", filename, err) + } + } return nil } diff --git a/src/k8s/pkg/k8sd/setup/containerd_test.go b/src/k8s/pkg/k8sd/setup/containerd_test.go index 4327499e4..8bd56399b 100644 --- a/src/k8s/pkg/k8sd/setup/containerd_test.go +++ b/src/k8s/pkg/k8sd/setup/containerd_test.go @@ -20,13 +20,14 @@ func TestContainerd(t *testing.T) { dir := t.TempDir() - g.Expect(os.WriteFile(filepath.Join(dir, "mockcni"), []byte("echo hi"), 0600)).To(Succeed()) + g.Expect(utils.WriteFile(filepath.Join(dir, "mockcni"), []byte("echo hi"), 0o600)).To(Succeed()) s := &mock.Snap{ Mock: mock.Mock{ ContainerdConfigDir: filepath.Join(dir, "containerd"), ContainerdRootDir: filepath.Join(dir, "containerd-root"), ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + ContainerdSocketPath: filepath.Join(dir, "containerd-run", "containerd.sock"), ContainerdRegistryConfigDir: filepath.Join(dir, "containerd-hosts"), ContainerdStateDir: filepath.Join(dir, "containerd-state"), ContainerdExtraConfigDir: filepath.Join(dir, "containerd-confd"), @@ -53,7 +54,7 @@ func TestContainerd(t *testing.T) { t.Run("Config", func(t *testing.T) { g := NewWithT(t) b, err := os.ReadFile(filepath.Join(dir, "containerd", "config.toml")) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(string(b)).To(SatisfyAll( ContainSubstring(fmt.Sprintf(`imports = ["%s/*.toml", "/custom/imports/*.toml"]`, filepath.Join(dir, "containerd-confd"))), ContainSubstring(fmt.Sprintf(`conf_dir = "%s"`, filepath.Join(dir, "cni-netd"))), @@ -62,8 +63,8 @@ func TestContainerd(t *testing.T) { )) info, err := os.Stat(filepath.Join(dir, "containerd", "config.toml")) - g.Expect(err).To(BeNil()) - g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0600))) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0o600))) switch stat := info.Sys().(type) { case *syscall.Stat_t: @@ -78,13 +79,13 @@ func TestContainerd(t *testing.T) { g := NewWithT(t) for _, plugin := range []string{"plugin1", "plugin2"} { link, err := os.Readlink(filepath.Join(dir, "opt-cni-bin", plugin)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(link).To(Equal("cni")) } info, err := os.Stat(filepath.Join(dir, "opt-cni-bin")) - g.Expect(err).To(BeNil()) - g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0700))) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(info.Mode().Perm()).To(Equal(fs.FileMode(0o700))) switch stat := info.Sys().(type) { case *syscall.Stat_t: @@ -107,7 +108,7 @@ func TestContainerd(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "containerd", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -115,8 +116,24 @@ func TestContainerd(t *testing.T) { t.Run("--address", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "containerd", "--address") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) }) + + t.Run("Lockfiles", func(t *testing.T) { + g := NewWithT(t) + m := map[string]string{ + "containerd-socket-path": s.ContainerdSocketDir(), + "containerd-config-dir": s.ContainerdConfigDir(), + "containerd-root-dir": s.ContainerdRootDir(), + "containerd-cni-bin-dir": s.CNIBinDir(), + } + for filename, content := range m { + + b, err := os.ReadFile(filepath.Join(s.LockFilesDir(), filename)) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(string(b)).To(Equal(content)) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/directories.go b/src/k8s/pkg/k8sd/setup/directories.go index f4b8c4779..7b0a2afc9 100644 --- a/src/k8s/pkg/k8sd/setup/directories.go +++ b/src/k8s/pkg/k8sd/setup/directories.go @@ -33,7 +33,7 @@ func EnsureAllDirectories(snap snap.Snap) error { if dir == "" { continue } - if err := os.MkdirAll(dir, 0700); err != nil { + if err := os.MkdirAll(dir, 0o700); err != nil { return fmt.Errorf("failed to create required directory: %w", err) } } diff --git a/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json b/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json new file mode 100644 index 000000000..5a1c21735 --- /dev/null +++ b/src/k8s/pkg/k8sd/setup/k8s-apiserver-proxy.json @@ -0,0 +1 @@ +{"endpoints":null} \ No newline at end of file diff --git a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go index 8a54bb60a..e288afbfd 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go +++ b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy.go @@ -11,7 +11,7 @@ import ( ) // K8sAPIServerProxy prepares configuration for k8s-apiserver-proxy. -func K8sAPIServerProxy(snap snap.Snap, servers []string, extraArgs map[string]*string) error { +func K8sAPIServerProxy(snap snap.Snap, servers []string, securePort int, extraArgs map[string]*string) error { configFile := filepath.Join(snap.ServiceExtraConfigDir(), "k8s-apiserver-proxy.json") if err := proxy.WriteEndpointsConfig(servers, configFile); err != nil { return fmt.Errorf("failed to write proxy configuration file: %w", err) @@ -20,7 +20,7 @@ func K8sAPIServerProxy(snap snap.Snap, servers []string, extraArgs map[string]*s if _, err := snaputil.UpdateServiceArguments(snap, "k8s-apiserver-proxy", map[string]string{ "--endpoints": configFile, "--kubeconfig": filepath.Join(snap.KubernetesConfigDir(), "kubelet.conf"), - "--listen": "127.0.0.1:6443", + "--listen": fmt.Sprintf(":%d", securePort), }, nil); err != nil { return fmt.Errorf("failed to write arguments file: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go index 3236e464b..8a01ea750 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go +++ b/src/k8s/pkg/k8sd/setup/k8s_apiserver_proxy_test.go @@ -27,7 +27,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).To(Succeed()) tests := []struct { key string @@ -35,20 +35,20 @@ func TestK8sApiServerProxy(t *testing.T) { }{ {key: "--endpoints", expectedVal: filepath.Join(s.Mock.ServiceExtraConfigDir, "k8s-apiserver-proxy.json")}, {key: "--kubeconfig", expectedVal: filepath.Join(s.Mock.KubernetesConfigDir, "kubelet.conf")}, - {key: "--listen", expectedVal: "127.0.0.1:6443"}, + {key: "--listen", expectedVal: ":6443"}, } for _, tc := range tests { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tc.expectedVal).To(Equal(val)) }) } args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-apiserver-proxy")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -61,7 +61,7 @@ func TestK8sApiServerProxy(t *testing.T) { "--listen": nil, // This should trigger a delete "--my-extra-arg": utils.Pointer("my-extra-val"), } - g.Expect(setup.K8sAPIServerProxy(s, nil, extraArgs)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, extraArgs)).To(Succeed()) tests := []struct { key string @@ -75,7 +75,7 @@ func TestK8sApiServerProxy(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tc.expectedVal).To(Equal(val)) }) } @@ -83,13 +83,13 @@ func TestK8sApiServerProxy(t *testing.T) { t.Run("--listen", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", "--listen") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-apiserver-proxy")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("MissingExtraConfigDir", func(t *testing.T) { @@ -98,7 +98,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) s.Mock.ServiceExtraConfigDir = "nonexistent" - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).ToNot(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).ToNot(Succeed()) }) t.Run("MissingServiceArgumentsDir", func(t *testing.T) { @@ -107,7 +107,7 @@ func TestK8sApiServerProxy(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sApiServerMock) s.Mock.ServiceArgumentsDir = "nonexistent" - g.Expect(setup.K8sAPIServerProxy(s, nil, nil)).ToNot(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, nil, 6443, nil)).ToNot(Succeed()) }) t.Run("JSONFileContent", func(t *testing.T) { @@ -118,7 +118,7 @@ func TestK8sApiServerProxy(t *testing.T) { endpoints := []string{"192.168.0.1", "192.168.0.2", "192.168.0.3"} fileName := filepath.Join(s.Mock.ServiceExtraConfigDir, "k8s-apiserver-proxy.json") - g.Expect(setup.K8sAPIServerProxy(s, endpoints, nil)).To(Succeed()) + g.Expect(setup.K8sAPIServerProxy(s, endpoints, 6443, nil)).To(Succeed()) b, err := os.ReadFile(fileName) g.Expect(err).NotTo(HaveOccurred()) @@ -130,4 +130,29 @@ func TestK8sApiServerProxy(t *testing.T) { // Compare the expected endpoints with those in the file g.Expect(config.Endpoints).To(Equal(endpoints)) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.K8sAPIServerProxy(s, nil, 1234, nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--listen", expectedVal: ":1234"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "k8s-apiserver-proxy", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_dqlite.go b/src/k8s/pkg/k8sd/setup/k8s_dqlite.go index ab4b8fbcd..1ee6a5cf9 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_dqlite.go +++ b/src/k8s/pkg/k8sd/setup/k8s_dqlite.go @@ -22,7 +22,7 @@ func K8sDqlite(snap snap.Snap, address string, cluster []string, extraArgs map[s if err := os.RemoveAll(snap.K8sDqliteStateDir()); err != nil { return fmt.Errorf("failed to cleanup not-empty k8s-dqlite directory: %w", err) } - if err := os.MkdirAll(snap.K8sDqliteStateDir(), 0700); err != nil { + if err := os.MkdirAll(snap.K8sDqliteStateDir(), 0o700); err != nil { return fmt.Errorf("failed to create k8s-dqlite state directory: %w", err) } } @@ -32,7 +32,7 @@ func K8sDqlite(snap snap.Snap, address string, cluster []string, extraArgs map[s return fmt.Errorf("failed to create init.yaml file for address=%s cluster=%v: %w", address, cluster, err) } - if err := os.WriteFile(filepath.Join(snap.K8sDqliteStateDir(), "init.yaml"), b, 0600); err != nil { + if err := utils.WriteFile(filepath.Join(snap.K8sDqliteStateDir(), "init.yaml"), b, 0o600); err != nil { return fmt.Errorf("failed to write init.yaml: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go b/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go index 8bfcedbab..391b3b616 100644 --- a/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go +++ b/src/k8s/pkg/k8sd/setup/k8s_dqlite_test.go @@ -28,7 +28,7 @@ func TestK8sDqlite(t *testing.T) { s := mustSetupSnapAndDirectories(t, setK8sDqliteMock) // Call the K8sDqlite setup function with mock arguments - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, nil)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, nil)).To(Succeed()) // Ensure the K8sDqlite arguments file has the expected arguments and values tests := []struct { @@ -42,7 +42,7 @@ func TestK8sDqlite(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -50,7 +50,7 @@ func TestK8sDqlite(t *testing.T) { // Ensure the K8sDqlite arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-dqlite")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -65,7 +65,7 @@ func TestK8sDqlite(t *testing.T) { "--storage-dir": utils.Pointer("overridden-storage-dir"), } // Call the K8sDqlite setup function with mock arguments - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, extraArgs)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", []string{"192.168.0.1:1234"}, extraArgs)).To(Succeed()) // Ensure the K8sDqlite arguments file has the expected arguments and values tests := []struct { @@ -79,7 +79,7 @@ func TestK8sDqlite(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -88,14 +88,14 @@ func TestK8sDqlite(t *testing.T) { t.Run("--listen", func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "k8s-dqlite", "--listen") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeZero()) }) // Ensure the K8sDqlite arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "k8s-dqlite")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("YAMLFileContents", func(t *testing.T) { @@ -112,10 +112,10 @@ func TestK8sDqlite(t *testing.T) { "192.168.0.3:1234", } - g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", cluster, nil)).To(BeNil()) + g.Expect(setup.K8sDqlite(s, "192.168.0.1:1234", cluster, nil)).To(Succeed()) b, err := os.ReadFile(filepath.Join(s.Mock.K8sDqliteStateDir, "init.yaml")) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(string(b)).To(Equal(expectedYaml)) }) diff --git a/src/k8s/pkg/k8sd/setup/kube_apiserver.go b/src/k8s/pkg/k8sd/setup/kube_apiserver.go index 37c1c0192..cfac02e8e 100644 --- a/src/k8s/pkg/k8sd/setup/kube_apiserver.go +++ b/src/k8s/pkg/k8sd/setup/kube_apiserver.go @@ -5,6 +5,7 @@ import ( "net" "os" "path/filepath" + "strconv" "strings" "github.com/canonical/k8s/pkg/k8sd/types" @@ -49,9 +50,9 @@ var ( ) // KubeAPIServer configures kube-apiserver on the local node. -func KubeAPIServer(snap snap.Snap, nodeIP net.IP, serviceCIDR string, authWebhookURL string, enableFrontProxy bool, datastore types.Datastore, authorizationMode string, extraArgs map[string]*string) error { +func KubeAPIServer(snap snap.Snap, securePort int, nodeIP net.IP, serviceCIDR string, authWebhookURL string, enableFrontProxy bool, datastore types.Datastore, authorizationMode string, extraArgs map[string]*string) error { authTokenWebhookConfigFile := filepath.Join(snap.ServiceExtraConfigDir(), "auth-token-webhook.conf") - authTokenWebhookFile, err := os.OpenFile(authTokenWebhookConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0600) + authTokenWebhookFile, err := os.OpenFile(authTokenWebhookConfigFile, os.O_WRONLY|os.O_CREATE|os.O_TRUNC, 0o600) if err != nil { return fmt.Errorf("failed to open auth-token-webhook.conf: %w", err) } @@ -77,13 +78,14 @@ func KubeAPIServer(snap snap.Snap, nodeIP net.IP, serviceCIDR string, authWebhoo "--kubelet-preferred-address-types": "InternalIP,Hostname,InternalDNS,ExternalDNS,ExternalIP", "--profiling": "false", "--request-timeout": "300s", - "--secure-port": "6443", + "--secure-port": strconv.Itoa(securePort), "--service-account-issuer": "https://kubernetes.default.svc", "--service-account-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--service-account-signing-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--service-cluster-ip-range": serviceCIDR, "--tls-cert-file": filepath.Join(snap.KubernetesPKIDir(), "apiserver.crt"), "--tls-cipher-suites": strings.Join(apiserverTLSCipherSuites, ","), + "--tls-min-version": "VersionTLS12", "--tls-private-key-file": filepath.Join(snap.KubernetesPKIDir(), "apiserver.key"), } diff --git a/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go b/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go index 7327a83f0..44677aa73 100644 --- a/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_apiserver_test.go @@ -37,7 +37,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -63,6 +63,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, {key: "--request-timeout", expectedVal: "300s"}, @@ -78,7 +79,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -86,7 +87,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ArgsNoProxy", func(t *testing.T) { @@ -96,7 +97,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -123,6 +124,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, } @@ -130,7 +132,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -138,7 +140,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -153,7 +155,7 @@ func TestKubeAPIServer(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the KubeAPIServer setup function with mock arguments - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", extraArgs)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", true, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", extraArgs)).To(Succeed()) // Ensure the kube-apiserver arguments file has the expected arguments and values tests := []struct { @@ -178,6 +180,7 @@ func TestKubeAPIServer(t *testing.T) { {key: "--service-cluster-ip-range", expectedVal: "10.0.0.0/24"}, {key: "--tls-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.crt")}, {key: "--tls-cipher-suites", expectedVal: apiserverTLSCipherSuites}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--tls-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "apiserver.key")}, {key: "--etcd-servers", expectedVal: fmt.Sprintf("unix://%s", filepath.Join(s.Mock.K8sDqliteStateDir, "k8s-dqlite.sock"))}, {key: "--request-timeout", expectedVal: "300s"}, @@ -194,7 +197,7 @@ func TestKubeAPIServer(t *testing.T) { t.Run(tc.key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(tc.expectedVal)) }) } @@ -206,7 +209,7 @@ func TestKubeAPIServer(t *testing.T) { // Ensure the kube-apiserver arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ArgsDualstack", func(t *testing.T) { g := NewWithT(t) @@ -214,7 +217,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Setup without proxy to simplify argument list - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24,fd01::/64", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24,fd01::/64", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(Succeed()) g.Expect(snaputil.GetServiceArgument(s, "kube-apiserver", "--service-cluster-ip-range")).To(Equal("10.0.0.0/24,fd01::/64")) _, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) @@ -227,7 +230,7 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Setup without proxy to simplify argument list - g.Expect(setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(BeNil()) + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("external"), ExternalServers: utils.Pointer([]string{"datastoreurl1", "datastoreurl2"})}, "Node,RBAC", nil)).To(Succeed()) g.Expect(snaputil.GetServiceArgument(s, "kube-apiserver", "--etcd-servers")).To(Equal("datastoreurl1,datastoreurl2")) _, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-apiserver")) @@ -241,8 +244,34 @@ func TestKubeAPIServer(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeAPIServerMock) // Attempt to configure kube-apiserver with an unsupported datastore - err := setup.KubeAPIServer(s, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("unsupported")}, "Node,RBAC", nil) + err := setup.KubeAPIServer(s, 6443, net.ParseIP("192.168.0.1"), "10.0.0.0/24", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("unsupported")}, "Node,RBAC", nil) g.Expect(err).To(HaveOccurred()) g.Expect(err).To(MatchError(ContainSubstring("unsupported datastore"))) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.KubeAPIServer(s, 6443, net.ParseIP("2001:db8::"), "fd98::/108", "https://auth-webhook.url", false, types.Datastore{Type: utils.Pointer("k8s-dqlite")}, "Node,RBAC", nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--advertise-address", expectedVal: "2001:db8::"}, + {key: "--service-cluster-ip-range", expectedVal: "fd98::/108"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kube-apiserver", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/kube_controller_manager.go b/src/k8s/pkg/k8sd/setup/kube_controller_manager.go index 95eb941eb..18a3c435f 100644 --- a/src/k8s/pkg/k8sd/setup/kube_controller_manager.go +++ b/src/k8s/pkg/k8sd/setup/kube_controller_manager.go @@ -22,6 +22,7 @@ func KubeControllerManager(snap snap.Snap, extraArgs map[string]*string) error { "--root-ca-file": filepath.Join(snap.KubernetesPKIDir(), "ca.crt"), "--service-account-private-key-file": filepath.Join(snap.KubernetesPKIDir(), "serviceaccount.key"), "--terminated-pod-gc-threshold": "12500", + "--tls-min-version": "VersionTLS12", "--use-service-account-credentials": "true", } // enable cluster-signing if certificates are available diff --git a/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go b/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go index 453703095..7c274a6b7 100644 --- a/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_controller_manager_test.go @@ -31,7 +31,7 @@ func TestKubeControllerManager(t *testing.T) { os.Create(filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")) // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, nil)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, nil)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -47,6 +47,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, {key: "--cluster-signing-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--cluster-signing-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")}, @@ -63,7 +64,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) @@ -79,7 +80,7 @@ func TestKubeControllerManager(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeControllerManagerMock) // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, nil)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, nil)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -95,6 +96,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, } for _, tc := range tests { @@ -109,7 +111,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) @@ -133,7 +135,7 @@ func TestKubeControllerManager(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the kube controller manager setup function - g.Expect(setup.KubeControllerManager(s, extraArgs)).To(BeNil()) + g.Expect(setup.KubeControllerManager(s, extraArgs)).To(Succeed()) // Ensure the kube controller manager arguments file has the expected arguments and values tests := []struct { @@ -148,6 +150,7 @@ func TestKubeControllerManager(t *testing.T) { {key: "--root-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--service-account-private-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "serviceaccount.key")}, {key: "--terminated-pod-gc-threshold", expectedVal: "12500"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--use-service-account-credentials", expectedVal: "true"}, {key: "--cluster-signing-cert-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.crt")}, {key: "--cluster-signing-key-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "ca.key")}, @@ -170,7 +173,7 @@ func TestKubeControllerManager(t *testing.T) { // Ensure the kube controller manager arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-controller-manager")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) t.Run("MissingArgsDir", func(t *testing.T) { g := NewWithT(t) diff --git a/src/k8s/pkg/k8sd/setup/kube_proxy.go b/src/k8s/pkg/k8sd/setup/kube_proxy.go index 0e64a60aa..ff29b0443 100644 --- a/src/k8s/pkg/k8sd/setup/kube_proxy.go +++ b/src/k8s/pkg/k8sd/setup/kube_proxy.go @@ -12,10 +12,10 @@ import ( ) // KubeProxy configures kube-proxy on the local node. -func KubeProxy(ctx context.Context, snap snap.Snap, hostname string, podCIDR string, extraArgs map[string]*string) error { +func KubeProxy(ctx context.Context, snap snap.Snap, hostname string, podCIDR string, localhostAddress string, extraArgs map[string]*string) error { serviceArgs := map[string]string{ "--cluster-cidr": podCIDR, - "--healthz-bind-address": "127.0.0.1", + "--healthz-bind-address": fmt.Sprintf("%s:10256", localhostAddress), "--kubeconfig": filepath.Join(snap.KubernetesConfigDir(), "proxy.conf"), "--profiling": "false", } diff --git a/src/k8s/pkg/k8sd/setup/kube_proxy_test.go b/src/k8s/pkg/k8sd/setup/kube_proxy_test.go index 1852c8478..f0eb92cf8 100644 --- a/src/k8s/pkg/k8sd/setup/kube_proxy_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_proxy_test.go @@ -28,10 +28,10 @@ func TestKubeProxy(t *testing.T) { }, } - g.Expect(setup.EnsureAllDirectories(s)).To(BeNil()) + g.Expect(setup.EnsureAllDirectories(s)).To(Succeed()) t.Run("Args", func(t *testing.T) { - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) for key, expectedVal := range map[string]string{ "--cluster-cidr": "10.1.0.0/16", @@ -43,7 +43,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -55,7 +55,7 @@ func TestKubeProxy(t *testing.T) { "--healthz-bind-address": nil, "--my-extra-arg": utils.Pointer("my-extra-val"), } - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", extraArgs)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", extraArgs)).To(Not(HaveOccurred())) for key, expectedVal := range map[string]string{ "--cluster-cidr": "10.1.0.0/16", @@ -68,7 +68,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -80,7 +80,7 @@ func TestKubeProxy(t *testing.T) { s.Mock.OnLXD = true t.Run("ArgsOnLXD", func(t *testing.T) { - g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.KubeProxy(context.Background(), s, "myhostname", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) for key, expectedVal := range map[string]string{ "--conntrack-max-per-core": "0", @@ -88,7 +88,7 @@ func TestKubeProxy(t *testing.T) { t.Run(key, func(t *testing.T) { g := NewWithT(t) val, err := snaputil.GetServiceArgument(s, "kube-proxy", key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(Equal(expectedVal)) }) } @@ -102,11 +102,37 @@ func TestKubeProxy(t *testing.T) { s.Mock.Hostname = "dev" s.Mock.ServiceArgumentsDir = filepath.Join(dir, "k8s") - g.Expect(setup.EnsureAllDirectories(s)).To(BeNil()) - g.Expect(setup.KubeProxy(context.Background(), s, "dev", "10.1.0.0/16", nil)).To(BeNil()) + g.Expect(setup.EnsureAllDirectories(s)).To(Succeed()) + g.Expect(setup.KubeProxy(context.Background(), s, "dev", "10.1.0.0/16", "127.0.0.1", nil)).To(Succeed()) val, err := snaputil.GetServiceArgument(s, "kube-proxy", "--hostname-override") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeEmpty()) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + g.Expect(setup.KubeProxy(context.Background(), s, "dev", "fd98::/108", "[::1]", nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--cluster-cidr", expectedVal: "fd98::/108"}, + {key: "--healthz-bind-address", expectedVal: "[::1]:10256"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kube-proxy", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/kube_scheduler.go b/src/k8s/pkg/k8sd/setup/kube_scheduler.go index 8a68b1689..ab64bfd29 100644 --- a/src/k8s/pkg/k8sd/setup/kube_scheduler.go +++ b/src/k8s/pkg/k8sd/setup/kube_scheduler.go @@ -18,6 +18,7 @@ func KubeScheduler(snap snap.Snap, extraArgs map[string]*string) error { "--leader-elect-lease-duration": "30s", "--leader-elect-renew-deadline": "15s", "--profiling": "false", + "--tls-min-version": "VersionTLS12", }, nil); err != nil { return fmt.Errorf("failed to render arguments file: %w", err) } diff --git a/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go b/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go index 20339734a..ae84ba87f 100644 --- a/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go +++ b/src/k8s/pkg/k8sd/setup/kube_scheduler_test.go @@ -26,7 +26,7 @@ func TestKubeScheduler(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeSchedulerMock) // Call the kube scheduler setup function - g.Expect(setup.KubeScheduler(s, nil)).To(BeNil()) + g.Expect(setup.KubeScheduler(s, nil)).To(Succeed()) // Ensure the kube scheduler arguments file has the expected arguments and values tests := []struct { @@ -39,6 +39,7 @@ func TestKubeScheduler(t *testing.T) { {key: "--leader-elect-lease-duration", expectedVal: "30s"}, {key: "--leader-elect-renew-deadline", expectedVal: "15s"}, {key: "--profiling", expectedVal: "false"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, } for _, tc := range tests { t.Run(tc.key, func(t *testing.T) { @@ -52,8 +53,7 @@ func TestKubeScheduler(t *testing.T) { // Ensure the kube scheduler arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-scheduler")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) - + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WithExtraArgs", func(t *testing.T) { @@ -68,7 +68,7 @@ func TestKubeScheduler(t *testing.T) { "--my-extra-arg": utils.Pointer("my-extra-val"), } // Call the kube scheduler setup function - g.Expect(setup.KubeScheduler(s, extraArgs)).To(BeNil()) + g.Expect(setup.KubeScheduler(s, extraArgs)).To(Succeed()) // Ensure the kube scheduler arguments file has the expected arguments and values tests := []struct { @@ -80,6 +80,7 @@ func TestKubeScheduler(t *testing.T) { {key: "--kubeconfig", expectedVal: filepath.Join(s.Mock.KubernetesConfigDir, "scheduler.conf")}, {key: "--leader-elect-renew-deadline", expectedVal: "15s"}, {key: "--profiling", expectedVal: "true"}, + {key: "--tls-min-version", expectedVal: "VersionTLS12"}, {key: "--my-extra-arg", expectedVal: "my-extra-val"}, } for _, tc := range tests { @@ -99,8 +100,7 @@ func TestKubeScheduler(t *testing.T) { // Ensure the kube scheduler arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kube-scheduler")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) - + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("MissingArgsDir", func(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/setup/kubelet.go b/src/k8s/pkg/k8sd/setup/kubelet.go index ff4cd9e0f..6b387b3d6 100644 --- a/src/k8s/pkg/k8sd/setup/kubelet.go +++ b/src/k8s/pkg/k8sd/setup/kubelet.go @@ -52,8 +52,8 @@ func kubelet(snap snap.Snap, hostname string, nodeIP net.IP, clusterDNS string, "--anonymous-auth": "false", "--authentication-token-webhook": "true", "--client-ca-file": filepath.Join(snap.KubernetesPKIDir(), "client-ca.crt"), - "--container-runtime-endpoint": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), - "--containerd": filepath.Join(snap.ContainerdSocketDir(), "containerd.sock"), + "--container-runtime-endpoint": snap.ContainerdSocketPath(), + "--containerd": snap.ContainerdSocketPath(), "--cgroup-driver": "systemd", "--eviction-hard": "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'", "--fail-swap-on": "false", diff --git a/src/k8s/pkg/k8sd/setup/kubelet_test.go b/src/k8s/pkg/k8sd/setup/kubelet_test.go index 99129e3d5..a0cb91ba3 100644 --- a/src/k8s/pkg/k8sd/setup/kubelet_test.go +++ b/src/k8s/pkg/k8sd/setup/kubelet_test.go @@ -14,8 +14,10 @@ import ( // These values are hard-coded and need to be updated if the // implementation changes. -var expectedControlPlaneLabels = "node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/worker=,k8sd.io/role=control-plane" -var expectedWorkerLabels = "node-role.kubernetes.io/worker=,k8sd.io/role=worker" +var ( + expectedControlPlaneLabels = "node-role.kubernetes.io/control-plane=,node-role.kubernetes.io/worker=,k8sd.io/role=control-plane" + expectedWorkerLabels = "node-role.kubernetes.io/worker=,k8sd.io/role=worker" +) var kubeletTLSCipherSuites = "TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305,TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256,TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384,TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305,TLS_RSA_WITH_AES_128_GCM_SHA256,TLS_RSA_WITH_AES_256_GCM_SHA384" @@ -30,11 +32,12 @@ func mustSetupSnapAndDirectories(t *testing.T, createMock func(*mock.Snap, strin func setKubeletMock(s *mock.Snap, dir string) { s.Mock = mock.Mock{ - KubernetesPKIDir: filepath.Join(dir, "pki"), - KubernetesConfigDir: filepath.Join(dir, "k8s-config"), - KubeletRootDir: filepath.Join(dir, "kubelet-root"), - ServiceArgumentsDir: filepath.Join(dir, "args"), - ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + KubernetesPKIDir: filepath.Join(dir, "pki"), + KubernetesConfigDir: filepath.Join(dir, "k8s-config"), + KubeletRootDir: filepath.Join(dir, "kubelet-root"), + ServiceArgumentsDir: filepath.Join(dir, "args"), + ContainerdSocketDir: filepath.Join(dir, "containerd-run"), + ContainerdSocketPath: filepath.Join(dir, "containerd-run", "containerd.sock"), } } @@ -57,8 +60,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -89,7 +92,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneWithExtraArgs", func(t *testing.T) { @@ -115,8 +118,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -153,7 +156,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneArgsNoOptional", func(t *testing.T) { @@ -163,7 +166,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet control plane setup function - g.Expect(setup.KubeletControlPlane(s, "dev", nil, "", "", "", nil, nil)).To(BeNil()) + g.Expect(setup.KubeletControlPlane(s, "dev", nil, "", "", "", nil, nil)).To(Succeed()) tests := []struct { key string @@ -173,8 +176,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -201,7 +204,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerArgs", func(t *testing.T) { @@ -211,7 +214,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -222,8 +225,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -254,7 +257,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerWithExtraArgs", func(t *testing.T) { @@ -269,7 +272,7 @@ func TestKubelet(t *testing.T) { } // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", extraArgs)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", extraArgs)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -280,8 +283,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -316,7 +319,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("WorkerArgsNoOptional", func(t *testing.T) { @@ -326,7 +329,7 @@ func TestKubelet(t *testing.T) { s := mustSetupSnapAndDirectories(t, setKubeletMock) // Call the kubelet worker setup function - g.Expect(setup.KubeletWorker(s, "dev", nil, "", "", "", nil)).To(BeNil()) + g.Expect(setup.KubeletWorker(s, "dev", nil, "", "", "", nil)).To(Succeed()) // Ensure the kubelet arguments file has the expected arguments and values tests := []struct { @@ -337,8 +340,8 @@ func TestKubelet(t *testing.T) { {key: "--anonymous-auth", expectedVal: "false"}, {key: "--authentication-token-webhook", expectedVal: "true"}, {key: "--client-ca-file", expectedVal: filepath.Join(s.Mock.KubernetesPKIDir, "client-ca.crt")}, - {key: "--container-runtime-endpoint", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, - {key: "--containerd", expectedVal: filepath.Join(s.Mock.ContainerdSocketDir, "containerd.sock")}, + {key: "--container-runtime-endpoint", expectedVal: s.Mock.ContainerdSocketPath}, + {key: "--containerd", expectedVal: s.Mock.ContainerdSocketPath}, {key: "--cgroup-driver", expectedVal: "systemd"}, {key: "--eviction-hard", expectedVal: "'memory.available<100Mi,nodefs.available<1Gi,imagefs.available<1Gi'"}, {key: "--fail-swap-on", expectedVal: "false"}, @@ -365,7 +368,7 @@ func TestKubelet(t *testing.T) { // Ensure the kubelet arguments file has exactly the expected number of arguments args, err := utils.ParseArgumentFile(filepath.Join(s.Mock.ServiceArgumentsDir, "kubelet")) g.Expect(err).ToNot(HaveOccurred()) - g.Expect(len(args)).To(Equal(len(tests))) + g.Expect(args).To(HaveLen(len(tests))) }) t.Run("ControlPlaneNoArgsDir", func(t *testing.T) { @@ -397,7 +400,34 @@ func TestKubelet(t *testing.T) { g.Expect(setup.KubeletControlPlane(s, "dev", net.ParseIP("192.168.0.1"), "10.152.1.1", "test-cluster.local", "provider", nil, nil)).To(Succeed()) val, err := snaputil.GetServiceArgument(s, "kubelet", "--hostname-override") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(val).To(BeEmpty()) }) + + t.Run("IPv6", func(t *testing.T) { + g := NewWithT(t) + + // Create a mock snap + s := mustSetupSnapAndDirectories(t, setKubeletMock) + s.Mock.Hostname = "dev" + + // Call the kubelet control plane setup function + g.Expect(setup.KubeletControlPlane(s, "dev", net.ParseIP("2001:db8::"), "2001:db8::1", "test-cluster.local", "provider", nil, nil)).To(Succeed()) + + tests := []struct { + key string + expectedVal string + }{ + {key: "--cluster-dns", expectedVal: "2001:db8::1"}, + {key: "--node-ip", expectedVal: "2001:db8::"}, + } + for _, tc := range tests { + t.Run(tc.key, func(t *testing.T) { + g := NewWithT(t) + val, err := snaputil.GetServiceArgument(s, "kubelet", tc.key) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(tc.expectedVal).To(Equal(val)) + }) + } + }) } diff --git a/src/k8s/pkg/k8sd/setup/templates.go b/src/k8s/pkg/k8sd/setup/templates.go index 8725c8f6a..4fac815b4 100644 --- a/src/k8s/pkg/k8sd/setup/templates.go +++ b/src/k8s/pkg/k8sd/setup/templates.go @@ -7,16 +7,14 @@ import ( "text/template" ) -var ( - //go:embed embed - templates embed.FS -) +//go:embed embed +var templates embed.FS func mustTemplate(parts ...string) *template.Template { path := filepath.Join(append([]string{"embed"}, parts...)...) b, err := templates.ReadFile(path) if err != nil { - panic(fmt.Errorf("invalid template %s: %s", path, err)) + panic(fmt.Errorf("invalid template %s: %w", path, err)) } return template.Must(template.New(path).Parse(string(b))) } diff --git a/src/k8s/pkg/k8sd/setup/util_extra_files.go b/src/k8s/pkg/k8sd/setup/util_extra_files.go index 31f3cfb57..163562ea5 100644 --- a/src/k8s/pkg/k8sd/setup/util_extra_files.go +++ b/src/k8s/pkg/k8sd/setup/util_extra_files.go @@ -7,6 +7,7 @@ import ( "strings" "github.com/canonical/k8s/pkg/snap" + "github.com/canonical/k8s/pkg/utils" ) // ExtraNodeConfigFiles writes the file contents to the specified filenames in the snap.ExtraFilesDir directory. @@ -20,7 +21,7 @@ func ExtraNodeConfigFiles(snap snap.Snap, files map[string]string) error { filePath := filepath.Join(snap.ServiceExtraConfigDir(), filename) // Write the content to the file - if err := os.WriteFile(filePath, []byte(content), 0400); err != nil { + if err := utils.WriteFile(filePath, []byte(content), 0o400); err != nil { return fmt.Errorf("failed to write to file %s: %w", filePath, err) } diff --git a/src/k8s/pkg/k8sd/setup/util_extra_files_test.go b/src/k8s/pkg/k8sd/setup/util_extra_files_test.go index ad4fa38bb..378810513 100644 --- a/src/k8s/pkg/k8sd/setup/util_extra_files_test.go +++ b/src/k8s/pkg/k8sd/setup/util_extra_files_test.go @@ -60,7 +60,7 @@ func TestExtraNodeConfigFiles(t *testing.T) { // Verify the file exists info, err := os.Stat(filePath) g.Expect(err).ToNot(gomega.HaveOccurred()) - g.Expect(info.Mode().Perm()).To(gomega.Equal(os.FileMode(0400))) + g.Expect(info.Mode().Perm()).To(gomega.Equal(os.FileMode(0o400))) // Verify the file content actualContent, err := os.ReadFile(filePath) diff --git a/src/k8s/pkg/k8sd/setup/util_kubeconfig.go b/src/k8s/pkg/k8sd/setup/util_kubeconfig.go index 3793a97f0..7b1157df1 100644 --- a/src/k8s/pkg/k8sd/setup/util_kubeconfig.go +++ b/src/k8s/pkg/k8sd/setup/util_kubeconfig.go @@ -57,7 +57,7 @@ func KubeconfigString(url string, caPEM string, crtPEM string, keyPEM string) (s } // SetupControlPlaneKubeconfigs writes kubeconfig files for the control plane components. -func SetupControlPlaneKubeconfigs(kubeConfigDir string, securePort int, pki pki.ControlPlanePKI) error { +func SetupControlPlaneKubeconfigs(kubeConfigDir string, localhostAddress string, securePort int, pki pki.ControlPlanePKI) error { for _, kubeconfig := range []struct { file string crt string @@ -69,10 +69,9 @@ func SetupControlPlaneKubeconfigs(kubeConfigDir string, securePort int, pki pki. {file: "scheduler.conf", crt: pki.KubeSchedulerClientCert, key: pki.KubeSchedulerClientKey}, {file: "kubelet.conf", crt: pki.KubeletClientCert, key: pki.KubeletClientKey}, } { - if err := Kubeconfig(filepath.Join(kubeConfigDir, kubeconfig.file), fmt.Sprintf("127.0.0.1:%d", securePort), pki.CACert, kubeconfig.crt, kubeconfig.key); err != nil { + if err := Kubeconfig(filepath.Join(kubeConfigDir, kubeconfig.file), fmt.Sprintf("%s:%d", localhostAddress, securePort), pki.CACert, kubeconfig.crt, kubeconfig.key); err != nil { return fmt.Errorf("failed to write kubeconfig %s: %w", kubeconfig.file, err) } } return nil - } diff --git a/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go b/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go index 6a51e0f4e..230f921b6 100644 --- a/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go +++ b/src/k8s/pkg/k8sd/setup/util_kubeconfig_test.go @@ -34,5 +34,5 @@ users: actual, err := setup.KubeconfigString("server", "ca", "crt", "key") g.Expect(actual).To(Equal(expectedConfig)) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_certificates.go b/src/k8s/pkg/k8sd/types/cluster_config_certificates.go index 9d2b7d066..e7aa1120c 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_certificates.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_certificates.go @@ -25,6 +25,7 @@ func (c Certificates) GetClientCACert() string { } return c.GetCACert() } + func (c Certificates) GetClientCAKey() string { // versions before 1.30.2 were using the same CA for server and client certificates if v := getField(c.ClientCAKey); v != "" { @@ -38,6 +39,7 @@ func (c Certificates) GetServiceAccountKey() string { return getField(c.ServiceA func (c Certificates) GetAPIServerKubeletClientCert() string { return getField(c.APIServerKubeletClientCert) } + func (c Certificates) GetAPIServerKubeletClientKey() string { return getField(c.APIServerKubeletClientKey) } @@ -46,5 +48,5 @@ func (c Certificates) GetAdminClientKey() string { return getField(c.AdminClien func (c Certificates) GetK8sdPublicKey() string { return getField(c.K8sdPublicKey) } func (c Certificates) GetK8sdPrivateKey() string { return getField(c.K8sdPrivateKey) } -// Empty returns true if all Certificates fields are unset +// Empty returns true if all Certificates fields are unset. func (c Certificates) Empty() bool { return c == Certificates{} } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go b/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go index debaa6d20..413ac2390 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_convert_loadbalancer_internal_test.go @@ -71,7 +71,7 @@ func Test_loadBalancerCIDRsFromAPI(t *testing.T) { t.Run("Nil", func(t *testing.T) { g := NewWithT(t) cidrs, ranges, err := loadBalancerCIDRsFromAPI(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cidrs).To(BeNil()) g.Expect(ranges).To(BeNil()) }) @@ -84,7 +84,7 @@ func Test_loadBalancerCIDRsFromAPI(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*cidrs).To(Equal(tc.internalCIDRs)) g.Expect(*ranges).To(Equal(tc.internalRanges)) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go b/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go index 194eedc80..ad58cb846 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_convert_test.go @@ -187,7 +187,7 @@ func TestClusterConfigFromBootstrapConfig(t *testing.T) { g := NewWithT(t) config, err := types.ClusterConfigFromBootstrapConfig(tc.bootstrap) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(config).To(Equal(tc.expectConfig)) }) } @@ -254,6 +254,5 @@ func TestClusterConfigFromBootstrapConfig(t *testing.T) { g.Expect(err).To(HaveOccurred()) }) } - }) } diff --git a/src/k8s/pkg/k8sd/types/cluster_config_datastore.go b/src/k8s/pkg/k8sd/types/cluster_config_datastore.go index 6f0d4dbd0..1f354904b 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_datastore.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_datastore.go @@ -29,7 +29,7 @@ func (c Datastore) GetExternalClientCert() string { return getField(c.ExternalCl func (c Datastore) GetExternalClientKey() string { return getField(c.ExternalClientKey) } func (c Datastore) Empty() bool { return c == Datastore{} } -// DatastorePathsProvider is to avoid circular dependency for snap.Snap in Datastore.ToKubeAPIServerArguments() +// DatastorePathsProvider is to avoid circular dependency for snap.Snap in Datastore.ToKubeAPIServerArguments(). type DatastorePathsProvider interface { K8sDqliteStateDir() string EtcdPKIDir() string diff --git a/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go b/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go index 92c0cd7c9..eb6f6d99a 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_kubelet_test.go @@ -72,7 +72,7 @@ func TestKubelet(t *testing.T) { g := NewWithT(t) cm, err := tc.kubelet.ToConfigMap(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cm).To(Equal(tc.configmap)) }) @@ -80,7 +80,7 @@ func TestKubelet(t *testing.T) { g := NewWithT(t) k, err := types.KubeletFromConfigMap(tc.configmap, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(k).To(Equal(tc.kubelet)) }) }) @@ -90,7 +90,7 @@ func TestKubelet(t *testing.T) { func TestKubeletSign(t *testing.T) { g := NewWithT(t) key, err := rsa.GenerateKey(rand.Reader, 4096) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) kubelet := types.Kubelet{ CloudProvider: utils.Pointer("external"), @@ -99,14 +99,14 @@ func TestKubeletSign(t *testing.T) { } configmap, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap).To(HaveKeyWithValue("k8sd-mac", Not(BeEmpty()))) t.Run("NoSign", func(t *testing.T) { g := NewWithT(t) configmap, err := kubelet.ToConfigMap(nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap).To(Not(HaveKey("k8sd-mac"))) }) @@ -114,7 +114,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) fromKubelet, err := types.KubeletFromConfigMap(configmap, &key.PublicKey) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fromKubelet).To(Equal(kubelet)) }) @@ -122,7 +122,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) configmap2, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(configmap2).To(Equal(configmap)) }) @@ -130,7 +130,7 @@ func TestKubeletSign(t *testing.T) { g := NewWithT(t) wrongKey, err := rsa.GenerateKey(rand.Reader, 2048) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) cm, err := types.KubeletFromConfigMap(configmap, &wrongKey.PublicKey) g.Expect(cm).To(BeZero()) @@ -142,10 +142,10 @@ func TestKubeletSign(t *testing.T) { t.Run(editKey, func(t *testing.T) { g := NewWithT(t) key, err := rsa.GenerateKey(rand.Reader, 2048) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) c, err := kubelet.ToConfigMap(key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(c).To(HaveKeyWithValue("k8sd-mac", Not(BeEmpty()))) t.Run("Manipulated", func(t *testing.T) { diff --git a/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go b/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go index e77f44fcb..532ec3f1c 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_merge_test.go @@ -208,9 +208,9 @@ func TestMergeClusterConfig(t *testing.T) { result, err := types.MergeClusterConfig(tc.old, tc.new) if tc.expectErr { - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(Equal(tc.expectResult)) } }) @@ -408,7 +408,7 @@ func TestMergeClusterConfig_Scenarios(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(merged).To(Equal(tc.expectMerged)) } }) diff --git a/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go b/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go index 251826d1c..24032c9b9 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_merge_util_test.go @@ -28,12 +28,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -60,12 +60,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -94,12 +94,12 @@ func Test_mergeField(t *testing.T) { result, err := mergeField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) @@ -128,12 +128,12 @@ func Test_mergeSliceField(t *testing.T) { result, err := mergeSliceField(tc.old, tc.new, tc.allowChange) switch { case tc.expectErr: - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) case tc.expectVal == nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(result).To(BeNil()) case tc.expectVal != nil: - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(*result).To(Equal(*tc.expectVal)) } }) diff --git a/src/k8s/pkg/k8sd/types/cluster_config_validate.go b/src/k8s/pkg/k8sd/types/cluster_config_validate.go index c673e3173..8ec1b8d50 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_validate.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_validate.go @@ -6,6 +6,8 @@ import ( "net/netip" "net/url" "strings" + + "github.com/canonical/k8s/pkg/utils" ) func validateCIDRs(cidrString string) error { @@ -21,6 +23,65 @@ func validateCIDRs(cidrString string) error { return nil } +// validateCIDROverlap checks for overlap and size constraints between pod and service CIDRs. +// It parses the provided podCIDR and serviceCIDR strings, checks for IPv4 and IPv6 overlaps. +func validateCIDROverlap(podCIDR string, serviceCIDR string) error { + // Parse the CIDRs + podIPv4CIDR, podIPv6CIDR, err := utils.SplitCIDRStrings(podCIDR) + if err != nil { + return fmt.Errorf("failed to parse pod CIDR: %w", err) + } + + svcIPv4CIDR, svcIPv6CIDR, err := utils.SplitCIDRStrings(serviceCIDR) + if err != nil { + return fmt.Errorf("failed to parse service CIDR: %w", err) + } + + // Check for IPv4 overlap + if podIPv4CIDR != "" && svcIPv4CIDR != "" { + if overlap, err := utils.CIDRsOverlap(podIPv4CIDR, svcIPv4CIDR); err != nil { + return fmt.Errorf("failed to check for IPv4 overlap: %w", err) + } else if overlap { + return fmt.Errorf("pod CIDR %q and service CIDR %q overlap", podCIDR, serviceCIDR) + } + } + + // Check for IPv6 overlap + if podIPv6CIDR != "" && svcIPv6CIDR != "" { + if overlap, err := utils.CIDRsOverlap(podIPv6CIDR, svcIPv6CIDR); err != nil { + return fmt.Errorf("failed to check for IPv6 overlap: %w", err) + } else if overlap { + return fmt.Errorf("pod CIDR %q and service CIDR %q overlap", podCIDR, serviceCIDR) + } + } + + return nil +} + +// validateIPv6CIDRSize ensures that the service IPv6 CIDR is not larger than /108. +// Ref: https://documentation.ubuntu.com/canonical-kubernetes/latest/snap/howto/networking/dualstack/#cidr-size-limitations +func validateIPv6CIDRSize(serviceCIDR string) error { + _, svcIPv6CIDR, err := utils.SplitCIDRStrings(serviceCIDR) + if err != nil { + return fmt.Errorf("invalid CIDR: %w", err) + } + + if svcIPv6CIDR == "" { + return nil + } + + _, ipv6Net, err := net.ParseCIDR(svcIPv6CIDR) + if err != nil { + return fmt.Errorf("invalid CIDR: %w", err) + } + + if prefixLength, _ := ipv6Net.Mask.Size(); prefixLength < 108 { + return fmt.Errorf("service CIDR %q cannot be larger than /108", serviceCIDR) + } + + return nil +} + // Validate that a ClusterConfig does not have conflicting or incompatible options. func (c *ClusterConfig) Validate() error { // check: validate that PodCIDR and ServiceCIDR are configured @@ -31,6 +92,14 @@ func (c *ClusterConfig) Validate() error { return fmt.Errorf("invalid service CIDR: %w", err) } + if err := validateCIDROverlap(c.Network.GetPodCIDR(), c.Network.GetServiceCIDR()); err != nil { + return fmt.Errorf("invalid cidr configuration: %w", err) + } + // Can't be an else-if, because default values could already be set. + if err := validateIPv6CIDRSize(c.Network.GetServiceCIDR()); err != nil { + return fmt.Errorf("invalid service CIDR: %w", err) + } + // check: ensure network is enabled if any of ingress, gateway, load-balancer are enabled if !c.Network.GetEnabled() { if c.Gateway.GetEnabled() { diff --git a/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go b/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go index db58934ed..d24ca3fdb 100644 --- a/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go +++ b/src/k8s/pkg/k8sd/types/cluster_config_validate_test.go @@ -10,15 +10,18 @@ import ( func TestValidateCIDR(t *testing.T) { for _, tc := range []struct { - cidr string - expectErr bool + cidr string + expectPodErr bool + expectSvcErr bool }{ - {cidr: "10.1.0.0/16"}, - {cidr: "2001:0db8::/32"}, - {cidr: "10.1.0.0/16,2001:0db8::/32"}, - {cidr: "", expectErr: true}, - {cidr: "bananas", expectErr: true}, - {cidr: "fd01::/64,fd02::/64,fd03::/64", expectErr: true}, + {cidr: "192.168.0.0/16"}, + {cidr: "2001:0db8::/108"}, + {cidr: "10.2.0.0/16,2001:0db8::/108"}, + {cidr: "", expectPodErr: true, expectSvcErr: true}, + {cidr: "bananas", expectPodErr: true, expectSvcErr: true}, + {cidr: "fd01::/108,fd02::/108,fd03::/108", expectPodErr: true, expectSvcErr: true}, + {cidr: "10.1.0.0/32", expectPodErr: true, expectSvcErr: true}, + {cidr: "2001:0db8::/32", expectSvcErr: true}, } { t.Run(tc.cidr, func(t *testing.T) { t.Run("Pod", func(t *testing.T) { @@ -30,10 +33,10 @@ func TestValidateCIDR(t *testing.T) { }, } err := config.Validate() - if tc.expectErr { + if tc.expectPodErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) t.Run("Service", func(t *testing.T) { @@ -45,10 +48,10 @@ func TestValidateCIDR(t *testing.T) { }, } err := config.Validate() - if tc.expectErr { + if tc.expectSvcErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) }) @@ -123,7 +126,7 @@ func TestValidateExternalServers(t *testing.T) { if tc.expectErr { g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } }) } diff --git a/src/k8s/pkg/k8sd/types/refresh.go b/src/k8s/pkg/k8sd/types/refresh.go index 365c9cb30..286826230 100644 --- a/src/k8s/pkg/k8sd/types/refresh.go +++ b/src/k8s/pkg/k8sd/types/refresh.go @@ -17,7 +17,7 @@ type RefreshOpts struct { } func RefreshOptsFromAPI(req apiv1.SnapRefreshRequest) (RefreshOpts, error) { - var optsMap = map[string]string{ + optsMap := map[string]string{ "localPath": req.LocalPath, "channel": req.Channel, "revision": req.Revision, diff --git a/src/k8s/pkg/k8sd/types/worker_test.go b/src/k8s/pkg/k8sd/types/worker_test.go index 586ffa032..cc5692185 100644 --- a/src/k8s/pkg/k8sd/types/worker_test.go +++ b/src/k8s/pkg/k8sd/types/worker_test.go @@ -17,11 +17,11 @@ func TestWorkerTokenEncode(t *testing.T) { g := NewWithT(t) s, err := token.Encode() - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(s).ToNot(BeEmpty()) decoded := &types.InternalWorkerNodeToken{} err = decoded.Decode(s) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(decoded).To(Equal(token)) } diff --git a/src/k8s/pkg/proxy/config.go b/src/k8s/pkg/proxy/config.go index 08c2d6a86..450f2a689 100644 --- a/src/k8s/pkg/proxy/config.go +++ b/src/k8s/pkg/proxy/config.go @@ -5,6 +5,8 @@ import ( "fmt" "os" "sort" + + "github.com/canonical/k8s/pkg/utils" ) // Configuration is the format of the apiserver proxy endpoints config file. @@ -33,7 +35,7 @@ func WriteEndpointsConfig(endpoints []string, file string) error { return fmt.Errorf("failed to marshal configuration: %w", err) } - if err := os.WriteFile(file, b, 0600); err != nil { + if err := utils.WriteFile(file, b, 0o600); err != nil { return fmt.Errorf("failed to write configuration file %s: %w", file, err) } return nil diff --git a/src/k8s/pkg/proxy/userspace.go b/src/k8s/pkg/proxy/userspace.go index 4fa143592..4082d7947 100644 --- a/src/k8s/pkg/proxy/userspace.go +++ b/src/k8s/pkg/proxy/userspace.go @@ -27,6 +27,8 @@ import ( "net" "sync" "time" + + "github.com/canonical/k8s/pkg/utils" ) type remote struct { @@ -78,7 +80,8 @@ func (tp *tcpproxy) Run() error { tp.MonitorInterval = 5 * time.Minute } for _, srv := range tp.Endpoints { - addr := fmt.Sprintf("%s:%d", srv.Target, srv.Port) + ip := net.ParseIP(srv.Target) + addr := fmt.Sprintf("%s:%d", utils.ToIPString(ip), srv.Port) tp.remotes = append(tp.remotes, &remote{srv: srv, addr: addr}) } diff --git a/src/k8s/pkg/snap/interface.go b/src/k8s/pkg/snap/interface.go index 03094679e..f2ea533df 100644 --- a/src/k8s/pkg/snap/interface.go +++ b/src/k8s/pkg/snap/interface.go @@ -39,12 +39,13 @@ type Snap interface { EtcdPKIDir() string // /etc/kubernetes/pki/etcd KubeletRootDir() string // /var/lib/kubelet - ContainerdConfigDir() string // /var/snap/k8s/common/etc/containerd - ContainerdExtraConfigDir() string // /var/snap/k8s/common/etc/containerd/conf.d - ContainerdRegistryConfigDir() string // /var/snap/k8s/common/etc/containerd/hosts.d - ContainerdRootDir() string // /var/snap/k8s/common/var/lib/containerd - ContainerdSocketDir() string // /var/snap/k8s/common/run - ContainerdStateDir() string // /run/containerd + ContainerdConfigDir() string // classic confinement: /etc/containerd, strict confinement: /var/snap/k8s/common/etc/containerd + ContainerdExtraConfigDir() string // classic confinement: /etc/containerd/conf.d, strict confinement: /var/snap/k8s/common/etc/containerd/conf.d + ContainerdRegistryConfigDir() string // classic confinement: /etc/containerd/hosts.d, strict confinement: /var/snap/k8s/common/etc/containerd/hosts.d + ContainerdRootDir() string // classic confinement: /var/lib/containerd, strict confinement: /var/snap/k8s/common/var/lib/containerd + ContainerdSocketDir() string // classic confinement: /run/containerd, strict confinement: /var/snap/k8s/common/run/containerd + ContainerdSocketPath() string // classic confinement: /run/containerd/containerd.sock, strict confinement: /var/snap/k8s/common/run/containerd/containerd.sock + ContainerdStateDir() string // classic confinement: /run/containerd, strict confinement: /var/snap/k8s/common/run/containerd K8sdStateDir() string // /var/snap/k8s/common/var/lib/k8sd/state K8sDqliteStateDir() string // /var/snap/k8s/common/var/lib/k8s-dqlite diff --git a/src/k8s/pkg/snap/mock/mock.go b/src/k8s/pkg/snap/mock/mock.go index fc9261720..21846253d 100644 --- a/src/k8s/pkg/snap/mock/mock.go +++ b/src/k8s/pkg/snap/mock/mock.go @@ -32,6 +32,7 @@ type Mock struct { ContainerdRegistryConfigDir string ContainerdRootDir string ContainerdSocketDir string + ContainerdSocketPath string ContainerdStateDir string K8sdStateDir string K8sDqliteStateDir string @@ -78,6 +79,7 @@ func (s *Snap) StartService(ctx context.Context, name string) error { } return s.StartServiceErr } + func (s *Snap) StopService(ctx context.Context, name string) error { if len(s.StopServiceCalledWith) == 0 { s.StopServiceCalledWith = []string{name} @@ -86,6 +88,7 @@ func (s *Snap) StopService(ctx context.Context, name string) error { } return s.StopServiceErr } + func (s *Snap) RestartService(ctx context.Context, name string) error { if len(s.RestartServiceCalledWith) == 0 { s.RestartServiceCalledWith = []string{name} @@ -94,6 +97,7 @@ func (s *Snap) RestartService(ctx context.Context, name string) error { } return s.RestartServiceErr } + func (s *Snap) Refresh(ctx context.Context, opts types.RefreshOpts) (string, error) { if len(s.RefreshCalledWith) == 0 { s.RefreshCalledWith = []types.RefreshOpts{opts} @@ -102,107 +106,145 @@ func (s *Snap) Refresh(ctx context.Context, opts types.RefreshOpts) (string, err } return "", s.RefreshErr } + func (s *Snap) RefreshStatus(ctx context.Context, changeID string) (*types.RefreshStatus, error) { return nil, nil } + func (s *Snap) Strict() bool { return s.Mock.Strict } + func (s *Snap) OnLXD(context.Context) (bool, error) { return s.Mock.OnLXD, s.Mock.OnLXDErr } + func (s *Snap) UID() int { return s.Mock.UID } + func (s *Snap) GID() int { return s.Mock.GID } + func (s *Snap) Hostname() string { return s.Mock.Hostname } + func (s *Snap) ContainerdConfigDir() string { return s.Mock.ContainerdConfigDir } + func (s *Snap) ContainerdRootDir() string { return s.Mock.ContainerdRootDir } + func (s *Snap) ContainerdStateDir() string { return s.Mock.ContainerdStateDir } + func (s *Snap) ContainerdSocketDir() string { return s.Mock.ContainerdSocketDir } + +func (s *Snap) ContainerdSocketPath() string { + return s.Mock.ContainerdSocketPath +} + func (s *Snap) ContainerdExtraConfigDir() string { return s.Mock.ContainerdExtraConfigDir } + func (s *Snap) ContainerdRegistryConfigDir() string { return s.Mock.ContainerdRegistryConfigDir } + func (s *Snap) KubernetesConfigDir() string { return s.Mock.KubernetesConfigDir } + func (s *Snap) KubernetesPKIDir() string { return s.Mock.KubernetesPKIDir } + func (s *Snap) EtcdPKIDir() string { return s.Mock.EtcdPKIDir } + func (s *Snap) KubeletRootDir() string { return s.Mock.KubeletRootDir } + func (s *Snap) CNIConfDir() string { return s.Mock.CNIConfDir } + func (s *Snap) CNIBinDir() string { return s.Mock.CNIBinDir } + func (s *Snap) CNIPluginsBinary() string { return s.Mock.CNIPluginsBinary } + func (s *Snap) CNIPlugins() []string { return s.Mock.CNIPlugins } + func (s *Snap) K8sdStateDir() string { return s.Mock.K8sdStateDir } + func (s *Snap) K8sDqliteStateDir() string { return s.Mock.K8sDqliteStateDir } + func (s *Snap) ServiceArgumentsDir() string { return s.Mock.ServiceArgumentsDir } + func (s *Snap) ServiceExtraConfigDir() string { return s.Mock.ServiceExtraConfigDir } + func (s *Snap) LockFilesDir() string { return s.Mock.LockFilesDir } + func (s *Snap) NodeTokenFile() string { return s.Mock.NodeTokenFile } + func (s *Snap) KubernetesClient(namespace string) (*kubernetes.Client, error) { return s.Mock.KubernetesClient, nil } + func (s *Snap) KubernetesNodeClient(namespace string) (*kubernetes.Client, error) { return s.Mock.KubernetesNodeClient, nil } + func (s *Snap) HelmClient() helm.Client { return s.Mock.HelmClient } + func (s *Snap) K8sDqliteClient(context.Context) (*dqlite.Client, error) { return s.Mock.K8sDqliteClient, nil } + func (s *Snap) K8sdClient(address string) (k8sd.Client, error) { return s.Mock.K8sdClient, nil } + func (s *Snap) SnapctlGet(ctx context.Context, args ...string) ([]byte, error) { s.SnapctlGetCalledWith = append(s.SnapctlGetCalledWith, args) return s.Mock.SnapctlGet[strings.Join(args, " ")], s.SnapctlGetErr } + func (s *Snap) SnapctlSet(ctx context.Context, args ...string) error { - s.SnapctlSetCalledWith = append(s.SnapctlGetCalledWith, args) + s.SnapctlSetCalledWith = append(s.SnapctlSetCalledWith, args) return s.SnapctlSetErr } + func (s *Snap) PreInitChecks(ctx context.Context, config types.ClusterConfig) error { s.PreInitChecksCalledWith = append(s.PreInitChecksCalledWith, config) return s.PreInitChecksErr diff --git a/src/k8s/pkg/snap/pebble_test.go b/src/k8s/pkg/snap/pebble_test.go index f85491140..e90b21e39 100644 --- a/src/k8s/pkg/snap/pebble_test.go +++ b/src/k8s/pkg/snap/pebble_test.go @@ -7,7 +7,6 @@ import ( "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/snap/mock" - . "github.com/onsi/gomega" ) @@ -22,7 +21,7 @@ func TestPebble(t *testing.T) { }) err := snap.StartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble start test-service")) t.Run("Fail", func(t *testing.T) { @@ -30,7 +29,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -43,7 +42,7 @@ func TestPebble(t *testing.T) { RunCommand: mockRunner.Run, }) err := snap.StopService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble stop test-service")) t.Run("Fail", func(t *testing.T) { @@ -51,7 +50,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -65,7 +64,7 @@ func TestPebble(t *testing.T) { }) err := snap.RestartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("testdir/bin/pebble restart test-service")) t.Run("Fail", func(t *testing.T) { @@ -73,7 +72,7 @@ func TestPebble(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) } diff --git a/src/k8s/pkg/snap/snap.go b/src/k8s/pkg/snap/snap.go index 032f6c988..057d30cde 100644 --- a/src/k8s/pkg/snap/snap.go +++ b/src/k8s/pkg/snap/snap.go @@ -3,6 +3,7 @@ package snap import ( "bytes" "context" + "errors" "fmt" "os" "os/exec" @@ -23,18 +24,20 @@ import ( ) type SnapOpts struct { - SnapInstanceName string - SnapDir string - SnapCommonDir string - RunCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + SnapInstanceName string + SnapDir string + SnapCommonDir string + RunCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + ContainerdBaseDir string } // snap implements the Snap interface. type snap struct { - snapDir string - snapCommonDir string - snapInstanceName string - runCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + snapDir string + snapCommonDir string + snapInstanceName string + runCommand func(ctx context.Context, command []string, opts ...func(c *exec.Cmd)) error + containerdBaseDir string } // NewSnap creates a new interface with the K8s snap. @@ -51,6 +54,15 @@ func NewSnap(opts SnapOpts) *snap { runCommand: runCommand, } + containerdBaseDir := opts.ContainerdBaseDir + if containerdBaseDir == "" { + containerdBaseDir = "/" + if s.Strict() { + containerdBaseDir = opts.SnapCommonDir + } + } + s.containerdBaseDir = containerdBaseDir + return s } @@ -161,19 +173,23 @@ func (s *snap) Hostname() string { } func (s *snap) ContainerdConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd") + return filepath.Join(s.containerdBaseDir, "etc", "containerd") } func (s *snap) ContainerdRootDir() string { - return filepath.Join(s.snapCommonDir, "var", "lib", "containerd") + return filepath.Join(s.containerdBaseDir, "var", "lib", "containerd") } func (s *snap) ContainerdSocketDir() string { - return filepath.Join(s.snapCommonDir, "run") + return filepath.Join(s.containerdBaseDir, "run", "containerd") +} + +func (s *snap) ContainerdSocketPath() string { + return filepath.Join(s.containerdBaseDir, "run", "containerd", "containerd.sock") } func (s *snap) ContainerdStateDir() string { - return "/run/containerd" + return filepath.Join(s.containerdBaseDir, "run", "containerd") } func (s *snap) CNIConfDir() string { @@ -250,11 +266,11 @@ func (s *snap) NodeTokenFile() string { } func (s *snap) ContainerdExtraConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd", "conf.d") + return filepath.Join(s.containerdBaseDir, "etc", "containerd", "conf.d") } func (s *snap) ContainerdRegistryConfigDir() string { - return filepath.Join(s.snapCommonDir, "etc", "containerd", "hosts.d") + return filepath.Join(s.containerdBaseDir, "etc", "containerd", "hosts.d") } func (s *snap) restClientGetter(path string, namespace string) genericclioptions.RESTClientGetter { @@ -323,6 +339,18 @@ func (s *snap) PreInitChecks(ctx context.Context, config types.ClusterConfig) er } } + // check if the containerd path already exists, signaling the fact that another containerd instance + // is already running on this node, which will conflict with the snap. + // Checks the directories instead of the containerd.sock file, since this file does not exist if + // containerd is not running/stopped. + if _, err := os.Stat(s.ContainerdSocketDir()); err == nil { + return fmt.Errorf("The path '%s' required for the containerd socket already exists. "+ + "This may mean that another service is already using that path, and it conflicts with the k8s snap. "+ + "Please make sure that there is no other service installed that uses the same path, and remove the existing directory.", s.ContainerdSocketDir()) + } else if !errors.Is(err, os.ErrNotExist) { + return fmt.Errorf("Encountered an error while checking '%s': %w", s.ContainerdSocketDir(), err) + } + return nil } diff --git a/src/k8s/pkg/snap/snap_test.go b/src/k8s/pkg/snap/snap_test.go index f694a0ef0..e99ff0384 100644 --- a/src/k8s/pkg/snap/snap_test.go +++ b/src/k8s/pkg/snap/snap_test.go @@ -3,15 +3,64 @@ package snap_test import ( "context" "fmt" + "os" + "path/filepath" "testing" + "github.com/canonical/k8s/pkg/k8sd/types" "github.com/canonical/k8s/pkg/snap" "github.com/canonical/k8s/pkg/snap/mock" - . "github.com/onsi/gomega" ) func TestSnap(t *testing.T) { + t.Run("NewSnap", func(t *testing.T) { + t.Run("containerd path with opt", func(t *testing.T) { + g := NewWithT(t) + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + RunCommand: mockRunner.Run, + ContainerdBaseDir: "/foo/lish", + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join("/foo/lish", "run", "containerd", "containerd.sock"))) + }) + + t.Run("containerd path classic", func(t *testing.T) { + g := NewWithT(t) + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + RunCommand: mockRunner.Run, + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join("/", "run", "containerd", "containerd.sock"))) + }) + + t.Run("containerd path strict", func(t *testing.T) { + g := NewWithT(t) + // We're checking if the snap is strict in NewSnap, which checks the snap.yaml file. + tmpDir, err := os.MkdirTemp("", "test-snap-k8s") + g.Expect(err).To(Not(HaveOccurred())) + defer os.RemoveAll(tmpDir) + + err = os.MkdirAll(filepath.Join(tmpDir, "meta"), os.ModeDir) + g.Expect(err).To(Not(HaveOccurred())) + + snapData := []byte("confinement: strict\n") + err = os.WriteFile(filepath.Join(tmpDir, "meta", "snap.yaml"), snapData, 0o644) + g.Expect(err).To(Not(HaveOccurred())) + + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + SnapDir: tmpDir, + SnapCommonDir: tmpDir, + RunCommand: mockRunner.Run, + }) + + g.Expect(snap.ContainerdSocketPath()).To(Equal(filepath.Join(tmpDir, "run", "containerd", "containerd.sock"))) + }) + }) + t.Run("Start", func(t *testing.T) { g := NewWithT(t) mockRunner := &mock.Runner{} @@ -22,7 +71,7 @@ func TestSnap(t *testing.T) { }) err := snap.StartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl start --enable k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -30,7 +79,7 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -43,7 +92,7 @@ func TestSnap(t *testing.T) { RunCommand: mockRunner.Run, }) err := snap.StopService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl stop --disable k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -51,7 +100,7 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "test-service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) }) }) @@ -65,7 +114,7 @@ func TestSnap(t *testing.T) { }) err := snap.RestartService(context.Background(), "test-service") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(mockRunner.CalledWithCommand).To(ConsistOf("snapctl restart k8s.test-service")) t.Run("Fail", func(t *testing.T) { @@ -73,7 +122,54 @@ func TestSnap(t *testing.T) { mockRunner.Err = fmt.Errorf("some error") err := snap.StartService(context.Background(), "service") - g.Expect(err).NotTo(BeNil()) + g.Expect(err).To(HaveOccurred()) + }) + }) + + t.Run("PreInitChecks", func(t *testing.T) { + g := NewWithT(t) + // Replace the ContainerdSocketDir to avoid checking against a real containerd.sock that may be running. + containerdDir, err := os.MkdirTemp("", "test-containerd") + g.Expect(err).To(Not(HaveOccurred())) + defer os.RemoveAll(containerdDir) + + mockRunner := &mock.Runner{} + snap := snap.NewSnap(snap.SnapOpts{ + SnapDir: "testdir", + SnapCommonDir: "testdir", + RunCommand: mockRunner.Run, + ContainerdBaseDir: containerdDir, + }) + conf := types.ClusterConfig{} + + err = snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(Not(HaveOccurred())) + expectedCalls := []string{} + for _, binary := range []string{"kube-apiserver", "kube-controller-manager", "kube-scheduler", "kube-proxy", "kubelet"} { + expectedCalls = append(expectedCalls, fmt.Sprintf("testdir/bin/%s --version", binary)) + } + g.Expect(mockRunner.CalledWithCommand).To(ConsistOf(expectedCalls)) + + t.Run("Fail socket exists", func(t *testing.T) { + g := NewWithT(t) + // Create the containerd.sock file, which should cause the check to fail. + err := os.MkdirAll(snap.ContainerdSocketDir(), os.ModeDir) + g.Expect(err).To(Not(HaveOccurred())) + f, err := os.Create(snap.ContainerdSocketPath()) + g.Expect(err).To(Not(HaveOccurred())) + f.Close() + defer os.Remove(f.Name()) + + err = snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(HaveOccurred()) + }) + + t.Run("Fail run command", func(t *testing.T) { + g := NewWithT(t) + mockRunner.Err = fmt.Errorf("some error") + + err := snap.PreInitChecks(context.Background(), conf) + g.Expect(err).To(HaveOccurred()) }) }) } diff --git a/src/k8s/pkg/snap/util/arguments.go b/src/k8s/pkg/snap/util/arguments.go index 63a5ae1c8..02aad87fb 100644 --- a/src/k8s/pkg/snap/util/arguments.go +++ b/src/k8s/pkg/snap/util/arguments.go @@ -103,8 +103,8 @@ func UpdateServiceArguments(snap snap.Snap, serviceName string, updateMap map[st // sort arguments so that output is consistent sort.Strings(newArguments) - if err := os.WriteFile(argumentsFile, []byte(strings.Join(newArguments, "\n")+"\n"), 0600); err != nil { - return false, fmt.Errorf("failed to write arguments for service %s: %q", serviceName, err) + if err := utils.WriteFile(argumentsFile, []byte(strings.Join(newArguments, "\n")+"\n"), 0o600); err != nil { + return false, fmt.Errorf("failed to write arguments for service %s: %w", serviceName, err) } return changed, nil } diff --git a/src/k8s/pkg/snap/util/arguments_test.go b/src/k8s/pkg/snap/util/arguments_test.go index f2550dd09..0d23ccd6a 100644 --- a/src/k8s/pkg/snap/util/arguments_test.go +++ b/src/k8s/pkg/snap/util/arguments_test.go @@ -2,12 +2,12 @@ package snaputil_test import ( "fmt" - "os" "path/filepath" "testing" "github.com/canonical/k8s/pkg/snap/mock" snaputil "github.com/canonical/k8s/pkg/snap/util" + "github.com/canonical/k8s/pkg/utils" . "github.com/onsi/gomega" ) @@ -32,7 +32,7 @@ func TestGetServiceArgument(t *testing.T) { --key=value-of-service-two `, } { - g.Expect(os.WriteFile(filepath.Join(dir, svc), []byte(args), 0600)).To(BeNil()) + g.Expect(utils.WriteFile(filepath.Join(dir, svc), []byte(args), 0o600)).To(Succeed()) } for _, tc := range []struct { @@ -56,9 +56,9 @@ func TestGetServiceArgument(t *testing.T) { value, err := snaputil.GetServiceArgument(s, tc.service, tc.key) if tc.expectErr { - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) } else { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(value).To(Equal(tc.expectValue)) } }) @@ -75,14 +75,14 @@ func TestUpdateServiceArguments(t *testing.T) { } _, err := snaputil.GetServiceArgument(s, "service", "--key") - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) changed, err := snaputil.UpdateServiceArguments(s, "service", map[string]string{"--key": "value"}, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeTrue()) value, err := snaputil.GetServiceArgument(s, "service", "--key") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(value).To(Equal("value")) }) @@ -183,11 +183,11 @@ func TestUpdateServiceArguments(t *testing.T) { }, } changed, err := snaputil.UpdateServiceArguments(s, "service", initialArguments, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeTrue()) changed, err = snaputil.UpdateServiceArguments(s, "service", tc.update, tc.delete) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(Equal(tc.expectedChange)) for key, expectedValue := range tc.expectedValues { @@ -197,7 +197,7 @@ func TestUpdateServiceArguments(t *testing.T) { t.Run("Reapply", func(t *testing.T) { g := NewWithT(t) changed, err := snaputil.UpdateServiceArguments(s, "service", tc.update, tc.delete) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(changed).To(BeFalse()) }) }) diff --git a/src/k8s/pkg/snap/util/node.go b/src/k8s/pkg/snap/util/node.go index 09dfe8289..d121970bc 100644 --- a/src/k8s/pkg/snap/util/node.go +++ b/src/k8s/pkg/snap/util/node.go @@ -25,7 +25,7 @@ func MarkAsWorkerNode(snap snap.Snap, mark bool) error { if err := os.Chown(fname, snap.UID(), snap.GID()); err != nil { return fmt.Errorf("failed to chown %s: %w", fname, err) } - if err := os.Chmod(fname, 0600); err != nil { + if err := os.Chmod(fname, 0o600); err != nil { return fmt.Errorf("failed to chmod %s: %w", fname, err) } } else { diff --git a/src/k8s/pkg/snap/util/node_test.go b/src/k8s/pkg/snap/util/node_test.go index 88b21fcb4..5529609e9 100644 --- a/src/k8s/pkg/snap/util/node_test.go +++ b/src/k8s/pkg/snap/util/node_test.go @@ -26,7 +26,7 @@ func TestIsWorker(t *testing.T) { lock.Close() exists, err := snaputil.IsWorker(mock) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeTrue()) }) @@ -34,7 +34,7 @@ func TestIsWorker(t *testing.T) { mock.Mock.LockFilesDir = "/non-existent" g := NewWithT(t) exists, err := snaputil.IsWorker(mock) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(exists).To(BeFalse()) }) } @@ -52,24 +52,24 @@ func TestMarkAsWorkerNode(t *testing.T) { t.Run("MarkWorker", func(t *testing.T) { g := NewWithT(t) err := snaputil.MarkAsWorkerNode(mock, true) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) workerFile := filepath.Join(mock.LockFilesDir(), "worker") g.Expect(workerFile).To(BeAnExistingFile()) // Clean up err = os.Remove(workerFile) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) }) t.Run("UnmarkWorker", func(t *testing.T) { g := NewWithT(t) workerFile := filepath.Join(mock.LockFilesDir(), "worker") _, err := os.Create(workerFile) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) err = snaputil.MarkAsWorkerNode(mock, false) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(workerFile).NotTo(BeAnExistingFile()) }) diff --git a/src/k8s/pkg/utils/certificate.go b/src/k8s/pkg/utils/certificate.go index 68067ea55..b817083d8 100644 --- a/src/k8s/pkg/utils/certificate.go +++ b/src/k8s/pkg/utils/certificate.go @@ -10,7 +10,7 @@ import ( ) // SplitIPAndDNSSANs splits a list of SANs into IP and DNS SANs -// Returns a list of IP addresses and a list of DNS names +// Returns a list of IP addresses and a list of DNS names. func SplitIPAndDNSSANs(extraSANs []string) ([]net.IP, []string) { var ipSANs []net.IP var dnsSANs []string @@ -57,7 +57,7 @@ func TLSClientConfigWithTrustedCertificate(remoteCert *x509.Certificate, rootCAs // GetRemoteCertificate retrieves the remote certificate from a given address // The address should be in the format of "hostname:port" -// Returns the remote certificate or an error +// Returns the remote certificate or an error. func GetRemoteCertificate(address string) (*x509.Certificate, error) { // validate address _, _, err := net.SplitHostPort(address) @@ -85,6 +85,7 @@ func GetRemoteCertificate(address string) (*x509.Certificate, error) { if err != nil { return nil, err } + defer resp.Body.Close() // Retrieve the certificate if resp.TLS == nil || len(resp.TLS.PeerCertificates) == 0 { @@ -94,7 +95,7 @@ func GetRemoteCertificate(address string) (*x509.Certificate, error) { return resp.TLS.PeerCertificates[0], nil } -// CertFingerprint returns the SHA256 fingerprint of a certificate +// CertFingerprint returns the SHA256 fingerprint of a certificate. func CertFingerprint(cert *x509.Certificate) string { return fmt.Sprintf("%x", sha256.Sum256(cert.Raw)) } diff --git a/src/k8s/pkg/utils/certificate_test.go b/src/k8s/pkg/utils/certificate_test.go index bd37b4797..c40c3a533 100644 --- a/src/k8s/pkg/utils/certificate_test.go +++ b/src/k8s/pkg/utils/certificate_test.go @@ -9,7 +9,6 @@ import ( "testing" "github.com/canonical/k8s/pkg/utils" - . "github.com/onsi/gomega" ) @@ -41,18 +40,18 @@ func TestTLSClientConfigWithTrustedCertificate(t *testing.T) { tlsConfig, err := utils.TLSClientConfigWithTrustedCertificate(remoteCert, rootCAs) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(tlsConfig.ServerName).To(Equal("bubblegum.com")) g.Expect(tlsConfig.RootCAs.Subjects()).To(ContainElement(remoteCert.RawSubject)) // Test with invalid remote certificate tlsConfig, err = utils.TLSClientConfigWithTrustedCertificate(nil, rootCAs) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) g.Expect(tlsConfig).To(BeNil()) // Test with nil root CAs _, err = utils.TLSClientConfigWithTrustedCertificate(remoteCert, nil) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) } func TestCertFingerprint(t *testing.T) { diff --git a/src/k8s/pkg/utils/cidr.go b/src/k8s/pkg/utils/cidr.go index 124f75aea..015eea3af 100644 --- a/src/k8s/pkg/utils/cidr.go +++ b/src/k8s/pkg/utils/cidr.go @@ -4,6 +4,7 @@ import ( "fmt" "math/big" "net" + "regexp" "strconv" "strings" @@ -11,6 +12,8 @@ import ( ) // findMatchingNodeAddress returns the IP address of a network interface that belongs to the given CIDR. +// It prefers the address with the fewest subnet bits. +// Loopback addresses are ignored. func findMatchingNodeAddress(cidr *net.IPNet) (net.IP, error) { addrs, err := net.InterfaceAddrs() if err != nil { @@ -25,7 +28,7 @@ func findMatchingNodeAddress(cidr *net.IPNet) (net.IP, error) { if !ok { continue } - if cidr.Contains(ipNet.IP) { + if cidr.Contains(ipNet.IP) && !ipNet.IP.IsLoopback() { _, subnetBits := cidr.Mask.Size() if selectedSubnetBits == -1 || subnetBits < selectedSubnetBits { // Prefer the address with the fewest subnet bits @@ -75,6 +78,21 @@ func GetKubernetesServiceIPsFromServiceCIDRs(serviceCIDR string) ([]net.IP, erro // ParseAddressString parses an address string and returns a canonical network address. func ParseAddressString(address string, port int64) (string, error) { + // Matches a CIDR block at the beginning of the address string + // e.g. [2001:db8::/32]:8080 + re := regexp.MustCompile(`^\[?([a-z0-9:.]+\/[0-9]+)\]?`) + cidrMatches := re.FindStringSubmatch(address) + if len(cidrMatches) != 0 { + // Resolve CIDR + if _, ipNet, err := net.ParseCIDR(cidrMatches[1]); err == nil { + matchingIP, err := findMatchingNodeAddress(ipNet) + if err != nil { + return "", fmt.Errorf("failed to find a matching node address for %q: %w", address, err) + } + address = strings.ReplaceAll(address, cidrMatches[1], matchingIP.String()) + } + } + host, hostPort, err := net.SplitHostPort(address) if err == nil { address = host @@ -90,20 +108,67 @@ func ParseAddressString(address string, port int64) (string, error) { if address == "" { address = util.NetworkInterfaceAddress() - } else if _, ipNet, err := net.ParseCIDR(address); err == nil { - matchingIP, err := findMatchingNodeAddress(ipNet) + } + + return net.JoinHostPort(address, fmt.Sprintf("%d", port)), nil +} + +// GetDefaultAddress returns the default IPv4 and IPv6 addresses of the host. +func GetDefaultAddress() (ipv4, ipv6 string, err error) { + // Get a list of network interfaces. + interfaces, err := net.Interfaces() + if err != nil { + return "", "", fmt.Errorf("failed to get network interfaces: %w", err) + } + + // Loop through each network interface + for _, iface := range interfaces { + // Skip interfaces that are down or loopback interfaces + if iface.Flags&net.FlagUp == 0 || iface.Flags&net.FlagLoopback != 0 { + continue + } + + // Get a list of addresses for the current interface. + addrs, err := iface.Addrs() if err != nil { - return "", fmt.Errorf("failed to find a matching node address for %q: %w", address, err) + return "", "", fmt.Errorf("failed to get addresses for interface %q: %w", iface.Name, err) + } + + // Loop through each address + for _, addr := range addrs { + // Parse the address to get the IP + var ip net.IP + switch v := addr.(type) { + case *net.IPNet: + ip = v.IP + case *net.IPAddr: + ip = v.IP + } + + // Check if it's an IPv4 or IPv6 address and not a loopback + if ip.IsLoopback() { + continue + } + + if ip.To4() != nil && ipv4 == "" { + ipv4 = ip.String() + } else if ip.To4() == nil && ipv6 == "" { + ipv6 = ip.String() + } + + // Break early if both IPv4 and IPv6 addresses are found + if ipv4 != "" && ipv6 != "" { + return ipv4, ipv6, nil + } } - address = matchingIP.String() } - return util.CanonicalNetworkAddress(address, port), nil + return ipv4, ipv6, nil } -// ParseCIDRs parses the given CIDR string and returns the respective IPv4 and IPv6 CIDRs. -func ParseCIDRs(CIDRstring string) (string, string, error) { - clusterCIDRs := strings.Split(CIDRstring, ",") +// SplitCIDRStrings parses the given CIDR string and returns the respective IPv4 and IPv6 CIDRs. +func SplitCIDRStrings(cidrString string) (string, string, error) { + clusterCIDRs := strings.Split(cidrString, ",") if v := len(clusterCIDRs); v != 1 && v != 2 { return "", "", fmt.Errorf("invalid CIDR list: %v", clusterCIDRs) } @@ -125,3 +190,42 @@ func ParseCIDRs(CIDRstring string) (string, string, error) { } return ipv4CIDR, ipv6CIDR, nil } + +// IsIPv4 returns true if the address is a valid IPv4 address, false otherwise. +// The address may contain a port number. +func IsIPv4(address string) bool { + ip := strings.Split(address, ":")[0] + parsedIP := net.ParseIP(ip) + return parsedIP != nil && parsedIP.To4() != nil +} + +// ToIPString returns the string representation of an IP address. +// If the IP address is an IPv6 address, it is enclosed in square brackets. +func ToIPString(ip net.IP) string { + if ip.To4() != nil { + return ip.String() + } + return "[" + ip.String() + "]" +} + +// CIDRsOverlap checks if two given CIDR blocks overlap. +// It takes two strings representing the CIDR blocks as input and returns a boolean indicating +// whether they overlap and an error if any of the CIDR blocks are invalid. +func CIDRsOverlap(cidr1, cidr2 string) (bool, error) { + _, ipNet1, err1 := net.ParseCIDR(cidr1) + _, ipNet2, err2 := net.ParseCIDR(cidr2) + + if err1 != nil { + return false, fmt.Errorf("couldn't parse CIDR block %q: %w", cidr1, err1) + } + + if err2 != nil { + return false, fmt.Errorf("couldn't parse CIDR block %q: %w", cidr2, err2) + } + + if ipNet1.Contains(ipNet2.IP) || ipNet2.Contains(ipNet1.IP) { + return true, nil + } + + return false, nil +} diff --git a/src/k8s/pkg/utils/cidr_test.go b/src/k8s/pkg/utils/cidr_test.go index 3dfbe3941..8a9658995 100644 --- a/src/k8s/pkg/utils/cidr_test.go +++ b/src/k8s/pkg/utils/cidr_test.go @@ -1,12 +1,12 @@ package utils_test import ( + "errors" "fmt" "net" "testing" "github.com/canonical/k8s/pkg/utils" - "github.com/canonical/lxd/lxd/util" . "github.com/onsi/gomega" ) @@ -24,7 +24,7 @@ func TestGetFirstIP(t *testing.T) { t.Run(tc.cidr, func(t *testing.T) { g := NewWithT(t) ip, err := utils.GetFirstIP(tc.cidr) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ip.String()).To(Equal(tc.ip)) }) } @@ -49,7 +49,7 @@ func TestGetKubernetesServiceIPsFromServiceCIDRs(t *testing.T) { ips[idx] = v.String() } - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(ips).To(Equal(tc.ips)) }) } @@ -66,7 +66,7 @@ func TestGetKubernetesServiceIPsFromServiceCIDRs(t *testing.T) { g := NewWithT(t) _, err := utils.GetKubernetesServiceIPsFromServiceCIDRs(tc.cidr) - g.Expect(err).ToNot(BeNil()) + g.Expect(err).To(HaveOccurred()) }) } }) @@ -76,12 +76,20 @@ func TestParseAddressString(t *testing.T) { g := NewWithT(t) // Seed the default address - defaultAddress := util.NetworkInterfaceAddress() - ip := net.ParseIP(defaultAddress) - subnetMask := net.CIDRMask(24, 32) - networkAddress := ip.Mask(subnetMask) + defaultIPv4, defaultIPv6, err := utils.GetDefaultAddress() + g.Expect(err).ToNot(HaveOccurred()) + + ip4 := net.ParseIP(defaultIPv4) + subnetMask4 := net.CIDRMask(24, 32) + networkAddress4 := ip4.Mask(subnetMask4) // Infer the CIDR notation - networkAddressCIDR := fmt.Sprintf("%s/24", networkAddress.String()) + networkAddressCIDR := fmt.Sprintf("%s/24", networkAddress4.String()) + + ip6 := net.ParseIP(defaultIPv6) + subnetMask6 := net.CIDRMask(64, 128) + networkAddress6 := ip6.Mask(subnetMask6) + // Infer the CIDR notation + networkAddressCIDR6 := fmt.Sprintf("%s/64", networkAddress6.String()) for _, tc := range []struct { name string @@ -90,18 +98,28 @@ func TestParseAddressString(t *testing.T) { want string wantErr bool }{ - {name: "EmptyAddress", address: "", port: 8080, want: fmt.Sprintf("%s:8080", defaultAddress), wantErr: false}, - {name: "CIDR", address: networkAddressCIDR, port: 8080, want: fmt.Sprintf("%s:8080", defaultAddress), wantErr: false}, - {name: "CIDRAndPort", address: fmt.Sprintf("%s:9090", networkAddressCIDR), port: 8080, want: fmt.Sprintf("%s:9090", defaultAddress), wantErr: false}, + {name: "CIDR6LinkLocalPort", address: "[::/0%eth0]:9090", port: 8080, want: fmt.Sprintf("[%s%%eth0]:9090", defaultIPv6), wantErr: false}, + {name: "CIDRAndPort", address: fmt.Sprintf("%s:9090", networkAddressCIDR), port: 8080, want: fmt.Sprintf("%s:9090", defaultIPv4), wantErr: false}, + {name: "CIDR", address: networkAddressCIDR, port: 8080, want: fmt.Sprintf("%s:8080", defaultIPv4), wantErr: false}, + {name: "EmptyAddress", address: "", port: 8080, want: fmt.Sprintf("%s:8080", defaultIPv4), wantErr: false}, + {name: "CIDR6LinkLocalDefault", address: "::/0%eth0", port: 8080, want: fmt.Sprintf("[%s%%eth0]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6Default", address: "::/0", port: 8080, want: fmt.Sprintf("[%s]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6DefaultPort", address: "[::/0]:9090", port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, + {name: "CIDR6DefaultPort", address: "[::/0]:9090", port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, + {name: "CIDR6", address: networkAddressCIDR6, port: 8080, want: fmt.Sprintf("[%s]:8080", defaultIPv6), wantErr: false}, + {name: "CIDR6AndPort", address: fmt.Sprintf("[%s]:9090", networkAddressCIDR6), port: 8080, want: fmt.Sprintf("[%s]:9090", defaultIPv6), wantErr: false}, {name: "IPv4", address: "10.0.0.10", port: 8080, want: "10.0.0.10:8080", wantErr: false}, {name: "IPv4AndPort", address: "10.0.0.10:9090", port: 8080, want: "10.0.0.10:9090", wantErr: false}, {name: "NonMatchingCIDR", address: "10.10.5.0/24", port: 8080, want: "", wantErr: true}, {name: "IPv6", address: "fe80::1:234", port: 8080, want: "[fe80::1:234]:8080", wantErr: false}, + {name: "IPv6Zone", address: "fe80::1:234%eth0", port: 8080, want: "[fe80::1:234%eth0]:8080", wantErr: false}, + {name: "IPv6ZoneAndPort", address: "[fe80::1:234%eth0]:9090", port: 8080, want: "[fe80::1:234%eth0]:9090", wantErr: false}, {name: "IPv6AndPort", address: "[fe80::1:234]:9090", port: 8080, want: "[fe80::1:234]:9090", wantErr: false}, {name: "InvalidPort", address: "127.0.0.1:invalid-port", port: 0, want: "", wantErr: true}, {name: "PortOutOfBounds", address: "10.0.0.10:70799", port: 8080, want: "", wantErr: true}, } { t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) got, err := utils.ParseAddressString(tc.address, tc.port) if tc.wantErr { g.Expect(err).To(HaveOccurred()) @@ -153,14 +171,87 @@ func TestParseCIDRs(t *testing.T) { for _, tc := range testCases { t.Run(tc.input, func(t *testing.T) { - ipv4CIDR, ipv6CIDR, err := utils.ParseCIDRs(tc.input) + g := NewWithT(t) + ipv4CIDR, ipv6CIDR, err := utils.SplitCIDRStrings(tc.input) if tc.expectedErr { - Expect(err).To(HaveOccurred()) + g.Expect(err).To(HaveOccurred()) } else { - Expect(err).To(BeNil()) - Expect(ipv4CIDR).To(Equal(tc.expectedIPv4)) - Expect(ipv6CIDR).To(Equal(tc.expectedIPv6)) + g.Expect(err).To(Not(HaveOccurred())) + g.Expect(ipv4CIDR).To(Equal(tc.expectedIPv4)) + g.Expect(ipv6CIDR).To(Equal(tc.expectedIPv6)) } }) } } + +func TestIsIPv4(t *testing.T) { + tests := []struct { + address string + expected bool + }{ + {"192.168.1.1:80", true}, + {"127.0.0.1", true}, + {"::1", false}, + {"[fe80::1]:80", false}, + {"256.256.256.256", false}, // Invalid IPv4 address + } + + for _, tc := range tests { + t.Run(tc.address, func(t *testing.T) { + g := NewWithT(t) + result := utils.IsIPv4(tc.address) + g.Expect(result).To(Equal(tc.expected)) + }) + } +} + +func TestToIPString(t *testing.T) { + tests := []struct { + ip net.IP + expected string + }{ + {net.ParseIP("192.168.1.1"), "192.168.1.1"}, + {net.ParseIP("::1"), "[::1]"}, + {net.ParseIP("fe80::1"), "[fe80::1]"}, + {net.ParseIP("127.0.0.1"), "127.0.0.1"}, + } + + for _, tc := range tests { + t.Run(tc.expected, func(t *testing.T) { + g := NewWithT(t) + result := utils.ToIPString(tc.ip) + g.Expect(result).To(Equal(tc.expected)) + }) + } +} + +// getInterfaceNameForIP returns the network interface name associated with the given IP. +func getInterfaceNameForIP(ip net.IP) (string, error) { + interfaces, err := net.Interfaces() + if err != nil { + return "", err + } + + for _, iface := range interfaces { + addrs, err := iface.Addrs() + if err != nil { + return "", err + } + + for _, addr := range addrs { + var ipAddr net.IP + switch v := addr.(type) { + case *net.IPNet: + ipAddr = v.IP + case *net.IPAddr: + ipAddr = v.IP + } + + if ipAddr.Equal(ip) { + return iface.Name, nil + } + } + } + + return "", errors.New("no interface found for the given IP") +} diff --git a/src/k8s/pkg/utils/control/retry_test.go b/src/k8s/pkg/utils/control/retry_test.go index 88ee21e31..1de78cb16 100644 --- a/src/k8s/pkg/utils/control/retry_test.go +++ b/src/k8s/pkg/utils/control/retry_test.go @@ -22,7 +22,6 @@ func TestRetryFor(t *testing.T) { } return nil }) - if err != nil { t.Errorf("Expected nil error, got: %v", err) } @@ -42,7 +41,7 @@ func TestRetryFor(t *testing.T) { return errors.New("failed") }) - if err != context.Canceled { + if !errors.Is(err, context.Canceled) { t.Errorf("Expected context.Canceled error, got: %v", err) } }) diff --git a/src/k8s/pkg/utils/control/wait_test.go b/src/k8s/pkg/utils/control/wait_test.go index 9fb3cd2dc..2b18758e8 100644 --- a/src/k8s/pkg/utils/control/wait_test.go +++ b/src/k8s/pkg/utils/control/wait_test.go @@ -12,11 +12,11 @@ func mockCheckFunc() (bool, error) { return true, nil } -var testError = errors.New("test error") +var errTest = errors.New("test error") // Mock check function that returns an error. func mockErrorCheckFunc() (bool, error) { - return false, testError + return false, errTest } func TestWaitUntilReady(t *testing.T) { @@ -34,7 +34,7 @@ func TestWaitUntilReady(t *testing.T) { cancel2() // Cancel the context immediately err2 := WaitUntilReady(ctx2, mockCheckFunc) - if err2 == nil || err2 != context.Canceled { + if err2 == nil || !errors.Is(err2, context.Canceled) { t.Errorf("Expected context.Canceled error, got: %v", err2) } @@ -52,7 +52,7 @@ func TestWaitUntilReady(t *testing.T) { defer cancel4() err4 := WaitUntilReady(ctx4, mockErrorCheckFunc) - if err4 == nil || !errors.Is(err4, testError) { + if err4 == nil || !errors.Is(err4, errTest) { t.Errorf("Expected test error, got: %v", err4) } } diff --git a/src/k8s/pkg/utils/errors/errors.go b/src/k8s/pkg/utils/errors/errors.go deleted file mode 100644 index a363850cb..000000000 --- a/src/k8s/pkg/utils/errors/errors.go +++ /dev/null @@ -1,16 +0,0 @@ -package errors - -import "errors" - -// DeeplyUnwrapError unwraps an wrapped error. -// DeeplyUnwrapError will return the innermost error for deeply nested errors. -// DeeplyUnwrapError will return the existing error if the error is not wrapped. -func DeeplyUnwrapError(err error) error { - for { - cause := errors.Unwrap(err) - if cause == nil { - return err - } - err = cause - } -} diff --git a/src/k8s/pkg/utils/errors/errors_test.go b/src/k8s/pkg/utils/errors/errors_test.go deleted file mode 100644 index 3562695c5..000000000 --- a/src/k8s/pkg/utils/errors/errors_test.go +++ /dev/null @@ -1,46 +0,0 @@ -package errors - -import ( - "errors" - "fmt" - "testing" - - "github.com/onsi/gomega" -) - -func TestDeeplyUnwrapError(t *testing.T) { - g := gomega.NewWithT(t) - - t.Run("when error is not wrapped", func(t *testing.T) { - err := errors.New("test error") - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(err)) - }) - - t.Run("when error is wrapped once", func(t *testing.T) { - innerErr := errors.New("inner error") - err := fmt.Errorf("outer wrapper: %w", innerErr) - - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(innerErr)) - }) - - t.Run("when error is deeply nested", func(t *testing.T) { - innermostErr := errors.New("innermost error") - innerErr := fmt.Errorf("middle wrapper: %w", innermostErr) - err := fmt.Errorf("outer wrapper: %w", innerErr) - - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.Equal(innermostErr)) - }) - - t.Run("when error is nil", func(t *testing.T) { - var err error - unwrapped := DeeplyUnwrapError(err) - - g.Expect(unwrapped).To(gomega.BeNil()) - }) -} diff --git a/src/k8s/pkg/utils/file.go b/src/k8s/pkg/utils/file.go index c4d1dc79f..63cfad1bf 100644 --- a/src/k8s/pkg/utils/file.go +++ b/src/k8s/pkg/utils/file.go @@ -15,9 +15,8 @@ import ( "sort" "strings" - "github.com/moby/sys/mountinfo" - "github.com/canonical/k8s/pkg/log" + "github.com/moby/sys/mountinfo" ) // ParseArgumentLine parses a command-line argument from a single line. @@ -120,6 +119,7 @@ func CopyFile(srcFile, dstFile string) error { return nil } + func FileExists(path ...string) (bool, error) { if _, err := os.Stat(filepath.Join(path...)); err != nil { if !os.IsNotExist(err) { @@ -258,3 +258,35 @@ func CreateTarball(tarballPath string, rootDir string, walkDir string, excludeFi return nil } + +// WriteFile writes data to a file with the given name and permissions. +// The file is written to a temporary file in the same directory as the target file +// and then renamed to the target file to avoid partial writes in case of a crash. +func WriteFile(name string, data []byte, perm fs.FileMode) error { + dir := filepath.Dir(name) + tmpFile, err := os.CreateTemp(dir, "tmp-*") + if err != nil { + return fmt.Errorf("failed to create temp file: %w", err) + } + defer os.Remove(tmpFile.Name()) + + if _, err := tmpFile.Write(data); err != nil { + tmpFile.Close() + return fmt.Errorf("failed to write to temp file: %w", err) + } + + if err := tmpFile.Chmod(perm); err != nil { + tmpFile.Close() + return fmt.Errorf("failed to set permissions on temp file: %w", err) + } + + if err := tmpFile.Close(); err != nil { + return fmt.Errorf("failed to close temp file: %w", err) + } + + if err := os.Rename(tmpFile.Name(), name); err != nil { + return fmt.Errorf("failed to rename temp file to target file: %w", err) + } + + return nil +} diff --git a/src/k8s/pkg/utils/file_test.go b/src/k8s/pkg/utils/file_test.go index 85af2f79c..db6437036 100644 --- a/src/k8s/pkg/utils/file_test.go +++ b/src/k8s/pkg/utils/file_test.go @@ -4,6 +4,7 @@ import ( "fmt" "os" "path/filepath" + "sync" "testing" "github.com/canonical/k8s/pkg/utils" @@ -88,7 +89,7 @@ func TestParseArgumentFile(t *testing.T) { g := NewWithT(t) filePath := filepath.Join(t.TempDir(), tc.name) - err := os.WriteFile(filePath, []byte(tc.content), 0755) + err := utils.WriteFile(filePath, []byte(tc.content), 0o755) if err != nil { t.Fatalf("Failed to setup testfile: %v", err) } @@ -157,17 +158,17 @@ func TestFileExists(t *testing.T) { testFilePath := fmt.Sprintf("%s/myfile", t.TempDir()) _, err := os.Create(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) fileExists, err := utils.FileExists(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fileExists).To(BeTrue()) err = os.Remove(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) fileExists, err = utils.FileExists(testFilePath) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(fileExists).To(BeFalse()) } @@ -182,3 +183,92 @@ func TestGetMountPropagationType(t *testing.T) { g.Expect(err).ToNot(HaveOccurred()) g.Expect(mountType).To(Equal(utils.MountPropagationShared)) } + +func TestWriteFile(t *testing.T) { + t.Run("PartialWrites", func(t *testing.T) { + g := NewWithT(t) + + name := filepath.Join(t.TempDir(), "testfile") + + const ( + numWriters = 200 + numIterations = 200 + ) + + var wg sync.WaitGroup + wg.Add(numWriters) + + expContent := "key: value" + expPerm := os.FileMode(0o644) + + for i := 0; i < numWriters; i++ { + go func(writerID int) { + defer wg.Done() + + for j := 0; j < numIterations; j++ { + g.Expect(utils.WriteFile(name, []byte(expContent), expPerm)).To(Succeed()) + + content, err := os.ReadFile(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(string(content)).To(Equal(expContent)) + + fileInfo, err := os.Stat(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(fileInfo.Mode().Perm()).To(Equal(expPerm)) + } + }(i) + } + + wg.Wait() + }) + + tcs := []struct { + name string + expContent []byte + expPerm os.FileMode + }{ + { + name: "test1", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o644), + }, + { + name: "test2", + expContent: []byte(""), + expPerm: os.FileMode(0o600), + }, + { + name: "test3", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o755), + }, + { + name: "test4", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o777), + }, + { + name: "test5", + expContent: []byte("key: value"), + expPerm: os.FileMode(0o400), + }, + } + + for _, tc := range tcs { + t.Run(tc.name, func(t *testing.T) { + g := NewWithT(t) + + name := filepath.Join(t.TempDir(), tc.name) + + g.Expect(utils.WriteFile(name, tc.expContent, tc.expPerm)).To(Succeed()) + + content, err := os.ReadFile(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(string(content)).To(Equal(string(tc.expContent))) + + fileInfo, err := os.Stat(name) + g.Expect(err).ToNot(HaveOccurred()) + g.Expect(fileInfo.Mode().Perm()).To(Equal(tc.expPerm)) + }) + } +} diff --git a/src/k8s/pkg/utils/hostname_test.go b/src/k8s/pkg/utils/hostname_test.go index 752fb5125..694c15ec1 100644 --- a/src/k8s/pkg/utils/hostname_test.go +++ b/src/k8s/pkg/utils/hostname_test.go @@ -28,10 +28,10 @@ func TestCleanHostname(t *testing.T) { g := NewWithT(t) hostname, err := utils.CleanHostname(tc.hostname) if tc.expectValid { - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(hostname).To(Equal(tc.expectHostname)) } else { - g.Expect(err).To(Not(BeNil())) + g.Expect(err).To(HaveOccurred()) } }) } diff --git a/src/k8s/pkg/utils/mapstructure.go b/src/k8s/pkg/utils/mapstructure.go index 60e650b45..19fc297b0 100644 --- a/src/k8s/pkg/utils/mapstructure.go +++ b/src/k8s/pkg/utils/mapstructure.go @@ -1,6 +1,7 @@ package utils import ( + "fmt" "reflect" "strings" "unicode" @@ -15,12 +16,16 @@ func YAMLToStringSliceHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) return data, nil } - if data.(string) == "" { + strData, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } + if strData == "" { return data, nil } var result []string - if err := yaml.Unmarshal([]byte(data.(string)), &result); err != nil { + if err := yaml.Unmarshal([]byte(strData), &result); err != nil { return data, nil } @@ -34,7 +39,10 @@ func StringToFieldsSliceHookFunc(r rune) mapstructure.DecodeHookFunc { return data, nil } - raw := data.(string) + raw, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } if raw == "" { return []string{}, nil } @@ -49,12 +57,16 @@ func YAMLToStringMapHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) ( return data, nil } - if data.(string) == "" { + strData, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } + if strData == "" { return map[string]string{}, nil } var result map[string]string - if err := yaml.Unmarshal([]byte(data.(string)), &result); err != nil { + if err := yaml.Unmarshal([]byte(strData), &result); err != nil { return data, nil } @@ -67,14 +79,17 @@ func StringToStringMapHookFunc(f reflect.Kind, t reflect.Kind, data interface{}) return data, nil } - raw := data.(string) + raw, ok := data.(string) + if !ok { + return nil, fmt.Errorf("expected string but got %T", data) + } if raw == "" { return map[string]string{}, nil } fields := strings.FieldsFunc(raw, func(this rune) bool { return this == ',' || unicode.IsSpace(this) }) result := make(map[string]string, len(fields)) - for _, kv := range strings.FieldsFunc(raw, func(this rune) bool { return this == ',' || unicode.IsSpace(this) }) { + for _, kv := range fields { parts := strings.SplitN(kv, "=", 2) if len(parts) < 2 { return data, nil diff --git a/src/k8s/pkg/utils/pki/generate_test.go b/src/k8s/pkg/utils/pki/generate_test.go index e2fe9976c..1ab80c819 100644 --- a/src/k8s/pkg/utils/pki/generate_test.go +++ b/src/k8s/pkg/utils/pki/generate_test.go @@ -14,20 +14,20 @@ func TestGenerateSelfSignedCA(t *testing.T) { cert, key, err := pkiutil.GenerateSelfSignedCA(pkix.Name{CommonName: "test-cert"}, notBefore, notBefore.AddDate(10, 0, 0), 2048) g := NewWithT(t) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cert).ToNot(BeEmpty()) g.Expect(key).ToNot(BeEmpty()) t.Run("Load", func(t *testing.T) { c, k, err := pkiutil.LoadCertificate(cert, key) - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(c).ToNot(BeNil()) g.Expect(k).ToNot(BeNil()) }) t.Run("LoadCertOnly", func(t *testing.T) { cert, key, err := pkiutil.LoadCertificate(cert, "") - g.Expect(err).To(BeNil()) + g.Expect(err).To(Not(HaveOccurred())) g.Expect(cert).ToNot(BeNil()) g.Expect(key).To(BeNil()) }) diff --git a/src/k8s/pkg/utils/pki/load.go b/src/k8s/pkg/utils/pki/load.go index 705efc8ba..46c0d23fe 100644 --- a/src/k8s/pkg/utils/pki/load.go +++ b/src/k8s/pkg/utils/pki/load.go @@ -64,8 +64,7 @@ func LoadRSAPublicKey(keyPEM string) (*rsa.PublicKey, error) { if pb == nil { return nil, fmt.Errorf("failed to parse PEM block") } - switch pb.Type { - case "PUBLIC KEY": + if pb.Type == "PUBLIC KEY" { parsed, err := x509.ParsePKIXPublicKey(pb.Bytes) if err != nil { return nil, fmt.Errorf("failed to parse public key: %w", err) @@ -86,8 +85,7 @@ func LoadCertificateRequest(csrPEM string) (*x509.CertificateRequest, error) { if pb == nil { return nil, fmt.Errorf("failed to parse certificate request PEM") } - switch pb.Type { - case "CERTIFICATE REQUEST": + if pb.Type == "CERTIFICATE REQUEST" { parsed, err := x509.ParseCertificateRequest(pb.Bytes) if err != nil { return nil, fmt.Errorf("failed to parse certificate request: %w", err) diff --git a/src/k8s/pkg/utils/time.go b/src/k8s/pkg/utils/time.go index 19102c1c5..d414e7232 100644 --- a/src/k8s/pkg/utils/time.go +++ b/src/k8s/pkg/utils/time.go @@ -39,7 +39,7 @@ func SecondsToExpirationDate(now time.Time, seconds int) time.Time { // - y: years // - mo: months // - d: days -// - any other unit supported by time.ParseDuration +// - any other unit supported by time.ParseDuration. func TTLToSeconds(ttl string) (int, error) { if len(ttl) < 2 { return 0, fmt.Errorf("invalid TTL length: %s", ttl) diff --git a/src/k8s/pkg/utils/time_test.go b/src/k8s/pkg/utils/time_test.go index dce809058..5b816ebfc 100644 --- a/src/k8s/pkg/utils/time_test.go +++ b/src/k8s/pkg/utils/time_test.go @@ -44,9 +44,7 @@ func TestSecondsToExpirationDate(t *testing.T) { got := utils.SecondsToExpirationDate(now, tt.seconds) g.Expect(got).To(Equal(tt.want)) }) - } - } func TestTTLToSeconds(t *testing.T) { diff --git a/tests/branch_management/tests/conftest.py b/tests/branch_management/tests/conftest.py index f745828fb..85045f683 100644 --- a/tests/branch_management/tests/conftest.py +++ b/tests/branch_management/tests/conftest.py @@ -2,26 +2,68 @@ # Copyright 2024 Canonical, Ltd. # from pathlib import Path +from typing import Optional import pytest import requests import semver +STABLE_URL = "https://dl.k8s.io/release/stable.txt" +RELEASE_URL = "https://dl.k8s.io/release/stable-{}.{}.txt" -@pytest.fixture -def upstream_release() -> semver.VersionInfo: + +def _upstream_release(ver: semver.Version) -> Optional[semver.Version]: + """Semver of the major.minor release if it exists""" + r = requests.get(RELEASE_URL.format(ver.major, ver.minor)) + if r.status_code == 200: + return semver.Version.parse(r.content.decode().lstrip("v")) + + +def _get_max_minor(ver: semver.Version) -> semver.Version: + """ + Get the latest patch release based on the provided major. + + e.g. 1.. could yield 1.31.4 if 1.31 is the latest stable release on that maj channel + e.g. 2.. could yield 2.12.1 if 2.12 is the latest stable release on that maj channel + """ + out = semver.Version(ver.major, 0, 0) + while ver := _upstream_release(ver): + out, ver = ver, semver.Version(ver.major, ver.minor + 1, 0) + return out + + +def _previous_release(ver: semver.Version) -> semver.Version: + """Return the prior release version based on the provided version ignoring patch""" + if ver.minor != 0: + return _upstream_release(semver.Version(ver.major, ver.minor - 1, 0)) + return _get_max_minor(semver.Version(ver.major, 0, 0)) + + +@pytest.fixture(scope="session") +def stable_release() -> semver.Version: """Return the latest stable k8s in the release series""" - release_url = "https://dl.k8s.io/release/stable.txt" - r = requests.get(release_url) + r = requests.get(STABLE_URL) r.raise_for_status() return semver.Version.parse(r.content.decode().lstrip("v")) -@pytest.fixture -def current_release() -> semver.VersionInfo: +@pytest.fixture(scope="session") +def current_release() -> semver.Version: """Return the current branch k8s version""" ver_file = ( Path(__file__).parent / "../../../build-scripts/components/kubernetes/version" ) version = ver_file.read_text().strip() return semver.Version.parse(version.lstrip("v")) + + +@pytest.fixture +def prior_stable_release(stable_release) -> semver.Version: + """Return the prior release to the upstream stable""" + return _previous_release(stable_release) + + +@pytest.fixture +def prior_release(current_release) -> semver.Version: + """Return the prior release to the current release""" + return _previous_release(current_release) diff --git a/tests/branch_management/tests/test_branches.py b/tests/branch_management/tests/test_branches.py index 426fde599..556892c03 100644 --- a/tests/branch_management/tests/test_branches.py +++ b/tests/branch_management/tests/test_branches.py @@ -1,93 +1,127 @@ # # Copyright 2024 Canonical, Ltd. # +import functools +import logging +import subprocess from pathlib import Path -from subprocess import check_output import requests +log = logging.getLogger(__name__) +K8S_GH_REPO = "https://github.com/canonical/k8s-snap.git/" +K8S_LP_REPO = " https://git.launchpad.net/k8s" -def _get_max_minor(major): - """Get the latest minor release of the provided major. - For example if you use 1 as major you will get back X where X gives you latest 1.X release. - """ - minor = 0 - while _upstream_release_exists(major, minor): - minor += 1 - return minor - 1 - - -def _upstream_release_exists(major, minor): - """Return true if the major.minor release exists""" - release_url = "https://dl.k8s.io/release/stable-{}.{}.txt".format(major, minor) - r = requests.get(release_url) - return r.status_code == 200 - -def _confirm_branch_exists(branch): - cmd = f"git ls-remote --heads https://github.com/canonical/k8s-snap.git/ {branch}" - output = check_output(cmd.split()).decode("utf-8") - assert branch in output, f"Branch {branch} does not exist" +def _sh(*args, **kwargs): + default = {"text": True, "stderr": subprocess.PIPE} + try: + return subprocess.check_output(*args, **{**default, **kwargs}) + except subprocess.CalledProcessError as e: + log.error("stdout: %s", e.stdout) + log.error("stderr: %s", e.stderr) + raise e def _branch_flavours(branch: str = None): patch_dir = Path("build-scripts/patches") branch = "HEAD" if branch is None else branch - cmd = f"git ls-tree --full-tree -r --name-only {branch} {patch_dir}" - output = check_output(cmd.split()).decode("utf-8") + cmd = f"git ls-tree --full-tree -r --name-only origin/{branch} {patch_dir}" + output = _sh(cmd.split()) patches = set( Path(f).relative_to(patch_dir).parents[0] for f in output.splitlines() ) return [p.name for p in patches] -def _confirm_recipe(track, flavour): +@functools.lru_cache +def _confirm_branch_exists(repo, branch): + log.info(f"Checking {branch} branch exists in {repo}") + cmd = f"git ls-remote --heads {repo} {branch}" + output = _sh(cmd.split()) + return branch in output + + +def _confirm_all_branches_exist(leader): + assert _confirm_branch_exists( + K8S_GH_REPO, leader + ), f"GH Branch {leader} does not exist" + branches = [leader] + branches += [f"autoupdate/{leader}-{fl}" for fl in _branch_flavours(leader)] + if missing := [b for b in branches if not _confirm_branch_exists(K8S_GH_REPO, b)]: + assert missing, f"GH Branches do not exist {missing}" + if missing := [b for b in branches if not _confirm_branch_exists(K8S_LP_REPO, b)]: + assert missing, f"LP Branches do not exist {missing}" + + +@functools.lru_cache +def _confirm_recipe_exist(track, flavour): recipe = f"https://launchpad.net/~containers/k8s/+snap/k8s-snap-{track}-{flavour}" r = requests.get(recipe) return r.status_code == 200 -def test_branches(upstream_release): - """Ensures git branches exist for prior releases. +def _confirm_all_recipes_exist(track, branch): + log.info(f"Checking {track} recipe exists") + assert _confirm_branch_exists( + K8S_GH_REPO, branch + ), f"GH Branch {branch} does not exist" + flavours = ["classic"] + _branch_flavours(branch) + recipes = {flavour: _confirm_recipe_exist(track, flavour) for flavour in flavours} + if missing := [fl for fl, exists in recipes.items() if not exists]: + assert missing, f"LP Recipes do not exist for {track} {missing}" + - We need to make sure the LP builders pointing to the main github branch are only pushing - to the latest and current k8s edge snap tracks. An indication that this is not enforced is - that we do not have a branch for the k8s release for the previous stable release. Let me - clarify with an example. +def test_prior_branches(prior_stable_release): + """Ensures git branches exist for prior stable releases. - Assuming upstream stable k8s release is v1.12.x, there has to be a 1.11 github branch used - by the respective LP builders for building the v1.11.y. + This is to ensure that the prior release branches exist in the k8s-snap repository + before we can proceed to build the next release. For example, if the current stable + k8s release is v1.31.0, there must be a release-1.30 branch before updating main. """ - if upstream_release.minor != 0: - major = upstream_release.major - minor = upstream_release.minor - 1 - else: - major = int(upstream_release.major) - 1 - minor = _get_max_minor(major) - - prior_branch = f"release-{major}.{minor}" - print(f"Current stable is {upstream_release}") - print(f"Checking {prior_branch} branch exists") - _confirm_branch_exists(prior_branch) - flavours = _branch_flavours(prior_branch) - for flavour in flavours: - prior_branch = f"autoupdate/{prior_branch}-{flavour}" - print(f"Checking {prior_branch} branch exists") - _confirm_branch_exists(prior_branch) - - -def test_launchpad_recipe(current_release): + branch = f"release-{prior_stable_release.major}.{prior_stable_release.minor}" + _confirm_all_branches_exist(branch) + + +def test_prior_recipes(prior_stable_release): + """Ensures the recipes exist for prior stable releases. + + This is to ensure that the prior release recipes exist in launchpad before we can proceed + to build the next release. For example, if the current stable k8s release is v1.31.0, there + must be a k8s-snap-1.30-classic recipe before updating main. + """ + track = f"{prior_stable_release.major}.{prior_stable_release.minor}" + branch = f"release-{track}" + _confirm_all_recipes_exist(track, branch) + + +def test_branches(current_release): + """Ensures the current release has a release branch. + + This is to ensure that the current release branches exist in the k8s-snap repository + before we can proceed to build it. For example, if the current stable + k8s release is v1.31.0, there must be a release-1.31 branch. + """ + branch = f"release-{current_release.major}.{current_release.minor}" + _confirm_all_branches_exist(branch) + + +def test_recipes(current_release): """Ensures the current recipes are available. - We should ensure that a launchpad recipe exists for this release to be build with + We should ensure that a launchpad recipes exist for this release to be build with + + This can fail when a new minor release (e.g. 1.32) is detected and its release branch + is yet to be created from main. """ track = f"{current_release.major}.{current_release.minor}" - print(f"Checking {track} recipe exists") - flavours = ["classic"] + _branch_flavours() - recipe_exists = {flavour: _confirm_recipe(track, flavour) for flavour in flavours} - if missing_recipes := [ - flavour for flavour, exists in recipe_exists.items() if not exists - ]: - assert ( - not missing_recipes - ), f"LP Recipes do not exist for {track} {missing_recipes}" + branch = f"release-{track}" + _confirm_all_recipes_exist(track, branch) + + +def test_tip_recipes(): + """Ensures the tip recipes are available. + + We should ensure that a launchpad recipes always exist for tip to be build with + """ + _confirm_all_recipes_exist("latest", "main") diff --git a/tests/branch_management/tox.ini b/tests/branch_management/tox.ini index 371ad51e4..a7f2cd8de 100644 --- a/tests/branch_management/tox.ini +++ b/tests/branch_management/tox.ini @@ -30,15 +30,16 @@ commands = black {tox_root}/tests --check --diff [testenv:test] -description = Run integration tests +description = Run branch management tests deps = -r {tox_root}/requirements-test.txt commands = pytest -v \ - --maxfail 1 \ --tb native \ --log-cli-level DEBUG \ --disable-warnings \ + --log-format "%(asctime)s %(levelname)s %(message)s" \ + --log-date-format "%Y-%m-%d %H:%M:%S" \ {posargs} \ {tox_root}/tests pass_env = diff --git a/tests/integration/lxd-ipv6-profile.yaml b/tests/integration/lxd-ipv6-profile.yaml new file mode 100644 index 000000000..900edd7d2 --- /dev/null +++ b/tests/integration/lxd-ipv6-profile.yaml @@ -0,0 +1,7 @@ +description: "LXD profile for Canonical Kubernetes with IPv6-only networking" +devices: + eth0: + name: eth0 + nictype: bridged + parent: LXD_IPV6_NETWORK + type: nic diff --git a/tests/integration/lxd-profile.yaml b/tests/integration/lxd-profile.yaml index c6a05f38e..7f5e720dc 100644 --- a/tests/integration/lxd-profile.yaml +++ b/tests/integration/lxd-profile.yaml @@ -1,6 +1,6 @@ description: "LXD profile for Canonical Kubernetes" config: - linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,iptable_raw,netlink_diag,nf_nat,overlay,br_netfilter,xt_socket + linux.kernel_modules: ip_vs,ip_vs_rr,ip_vs_wrr,ip_vs_sh,ip_tables,ip6_tables,iptable_raw,netlink_diag,nf_nat,overlay,br_netfilter,xt_socket,nf_conntrack raw.lxc: | lxc.apparmor.profile=unconfined lxc.mount.auto=proc:rw sys:rw cgroup:rw diff --git a/tests/integration/lxd/setup-image.sh b/tests/integration/lxd/setup-image.sh index 4b587bad1..c5427fc5e 100755 --- a/tests/integration/lxd/setup-image.sh +++ b/tests/integration/lxd/setup-image.sh @@ -69,6 +69,8 @@ case "${BASE_DISTRO}" in # snapd is preinstalled on Ubuntu OSes lxc shell tmp-builder -- bash -c 'snap wait core seed.loaded' lxc shell tmp-builder -- bash -c 'snap install '"${BASE_SNAP}" + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'apt update && apt install -y "linux-modules-$(uname -r)"}' ;; almalinux) # install snapd and ensure /snap/bin is in the environment @@ -77,6 +79,8 @@ case "${BASE_DISTRO}" in lxc shell tmp-builder -- bash -c 'dnf install tar sudo -y' lxc shell tmp-builder -- bash -c 'dnf install fuse squashfuse -y' lxc shell tmp-builder -- bash -c 'dnf install snapd -y' + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'dnf install -y kernel-modules-core' lxc shell tmp-builder -- bash -c 'systemctl enable --now snapd.socket' lxc shell tmp-builder -- bash -c 'ln -s /var/lib/snapd/snap /snap' @@ -92,6 +96,8 @@ case "${BASE_DISTRO}" in lxc shell tmp-builder -- bash -c 'snap install snapd '"${BASE_SNAP}" lxc shell tmp-builder -- bash -c 'echo PATH=$PATH:/snap/bin >> /etc/environment' lxc shell tmp-builder -- bash -c 'apt autoremove; apt clean; apt autoclean; rm -rf /var/lib/apt/lists' + # NOTE(aznashwan): 'nf_conntrack' required by kube-proxy: + lxc shell tmp-builder -- bash -c 'apt update && apt install -y "linux-modules-$(uname -r)"}' # NOTE(neoaggelos): disable apparmor in containerd, as it causes trouble in the default setup lxc shell tmp-builder -- bash -c ' diff --git a/tests/integration/requirements-test.txt b/tests/integration/requirements-test.txt index 91282e09c..0fcd9c093 100644 --- a/tests/integration/requirements-test.txt +++ b/tests/integration/requirements-test.txt @@ -3,3 +3,4 @@ pytest==7.3.1 PyYAML==6.0.1 tenacity==8.2.3 pylint==3.2.5 +cryptography==43.0.3 diff --git a/tests/integration/templates/bootstrap-session.yaml b/tests/integration/templates/bootstrap-all.yaml similarity index 100% rename from tests/integration/templates/bootstrap-session.yaml rename to tests/integration/templates/bootstrap-all.yaml diff --git a/tests/integration/templates/bootstrap-csr-auto-approve.yaml b/tests/integration/templates/bootstrap-csr-auto-approve.yaml new file mode 100644 index 000000000..43fe77c98 --- /dev/null +++ b/tests/integration/templates/bootstrap-csr-auto-approve.yaml @@ -0,0 +1,9 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + metrics-server: + enabled: true + annotations: + k8sd/v1alpha1/csrsigning/auto-approve: true diff --git a/tests/integration/templates/bootstrap-ipv6-only.yaml b/tests/integration/templates/bootstrap-ipv6-only.yaml new file mode 100644 index 000000000..442857805 --- /dev/null +++ b/tests/integration/templates/bootstrap-ipv6-only.yaml @@ -0,0 +1,16 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + cluster-domain: cluster.local + local-storage: + enabled: true + local-path: /storage/path + default: false + gateway: + enabled: true + metrics-server: + enabled: true +pod-cidr: fd01::/108 +service-cidr: fd98::/108 diff --git a/tests/integration/templates/bootstrap-skip-service-stop.yaml b/tests/integration/templates/bootstrap-skip-service-stop.yaml new file mode 100644 index 000000000..13a536cf2 --- /dev/null +++ b/tests/integration/templates/bootstrap-skip-service-stop.yaml @@ -0,0 +1,9 @@ +cluster-config: + network: + enabled: true + dns: + enabled: true + metrics-server: + enabled: true + annotations: + k8sd/v1alpha/lifecycle/skip-stop-services-on-remove: true diff --git a/tests/integration/templates/etcd/etcd-tls.conf b/tests/integration/templates/etcd/etcd-tls.conf index 59243ba98..83cd61a86 100644 --- a/tests/integration/templates/etcd/etcd-tls.conf +++ b/tests/integration/templates/etcd/etcd-tls.conf @@ -18,6 +18,6 @@ [alt_names] IP.1 = 127.0.0.1 - IP.2 = $IP + IP.2 = $IP DNS.1 = localhost DNS.2 = $NAME diff --git a/tests/integration/templates/etcd/etcd.service b/tests/integration/templates/etcd/etcd.service index 0ac4e9933..e770666a0 100644 --- a/tests/integration/templates/etcd/etcd.service +++ b/tests/integration/templates/etcd/etcd.service @@ -11,6 +11,7 @@ LimitNOFILE=40000 TimeoutStartSec=0 + Environment=ETCD_UNSUPPORTED_ARCH=$ARCH ExecStart=/tmp/test-etcd/etcd --name $NAME \ --data-dir /tmp/etcd/s1 \ --listen-client-urls $CLIENT_URL \ diff --git a/tests/integration/templates/nginx-ipv6-only.yaml b/tests/integration/templates/nginx-ipv6-only.yaml new file mode 100644 index 000000000..93a5647a2 --- /dev/null +++ b/tests/integration/templates/nginx-ipv6-only.yaml @@ -0,0 +1,36 @@ +apiVersion: apps/v1 +kind: Deployment +metadata: + name: nginxipv6 +spec: + selector: + matchLabels: + run: nginxipv6 + replicas: 1 + template: + metadata: + labels: + run: nginxipv6 + spec: + containers: + - name: nginxipv6 + image: rocks.canonical.com/cdk/diverdane/nginxdualstack:1.0.0 + ports: + - containerPort: 80 +--- +apiVersion: v1 +kind: Service +metadata: + name: nginx-ipv6 + labels: + run: nginxipv6 +spec: + type: NodePort + ipFamilies: + - IPv6 + ipFamilyPolicy: SingleStack + ports: + - port: 80 + protocol: TCP + selector: + run: nginxipv6 diff --git a/tests/integration/templates/registry/hosts.toml b/tests/integration/templates/registry/hosts.toml new file mode 100644 index 000000000..416c9a642 --- /dev/null +++ b/tests/integration/templates/registry/hosts.toml @@ -0,0 +1,2 @@ +[host."http://$IP:$PORT"] +capabilities = ["pull", "resolve"] diff --git a/tests/integration/templates/registry/registry-config.yaml b/tests/integration/templates/registry/registry-config.yaml new file mode 100644 index 000000000..28610ffbb --- /dev/null +++ b/tests/integration/templates/registry/registry-config.yaml @@ -0,0 +1,22 @@ +version: 0.1 +log: + fields: + service: registry +storage: + cache: + blobdescriptor: inmemory + filesystem: + rootdirectory: /var/lib/registry/$NAME +http: + addr: :$PORT + headers: + X-Content-Type-Options: [nosniff] +health: + storagedriver: + enabled: true + interval: 10s + threshold: 3 +proxy: + remoteurl: $REMOTE + username: $USERNAME + password: $PASSWORD diff --git a/tests/integration/templates/registry/registry.service b/tests/integration/templates/registry/registry.service new file mode 100644 index 000000000..83de843a6 --- /dev/null +++ b/tests/integration/templates/registry/registry.service @@ -0,0 +1,15 @@ + [Unit] + Description=registry-$NAME + Documentation=https://github.com/distribution/distribution + + [Service] + Type=simple + Restart=always + RestartSec=5s + LimitNOFILE=40000 + TimeoutStartSec=0 + + ExecStart=/bin/registry serve /etc/distribution/$NAME.yaml + + [Install] + WantedBy=multi-user.target diff --git a/tests/integration/tests/conftest.py b/tests/integration/tests/conftest.py index eb6fbb07f..7333c2167 100644 --- a/tests/integration/tests/conftest.py +++ b/tests/integration/tests/conftest.py @@ -1,16 +1,22 @@ # # Copyright 2024 Canonical, Ltd. # +import itertools import logging from pathlib import Path -from typing import Generator, List, Union +from typing import Generator, Iterator, List, Optional, Union import pytest from test_util import config, harness, util from test_util.etcd import EtcdCluster +from test_util.registry import Registry LOG = logging.getLogger(__name__) +# The following snaps will be downloaded once per test run and preloaded +# into the harness instances to reduce the number of downloads. +PRELOADED_SNAPS = ["snapd", "core20"] + def _harness_clean(h: harness.Harness): "Clean up created instances within the test harness." @@ -78,14 +84,45 @@ def h() -> harness.Harness: _harness_clean(h) +@pytest.fixture(scope="session") +def registry(h: harness.Harness) -> Optional[Registry]: + if config.USE_LOCAL_MIRROR: + yield Registry(h) + else: + # local image mirror disabled, avoid initializing the + # registry mirror instance. + yield None + + +@pytest.fixture(scope="session", autouse=True) +def snapd_preload() -> None: + if not config.PRELOAD_SNAPS: + LOG.info("Snap preloading disabled, skipping...") + return + + LOG.info(f"Downloading snaps for preloading: {PRELOADED_SNAPS}") + for snap in PRELOADED_SNAPS: + util.run( + [ + "snap", + "download", + snap, + f"--basename={snap}", + "--target-directory=/tmp", + ] + ) + + def pytest_configure(config): config.addinivalue_line( "markers", "bootstrap_config: Provide a custom bootstrap config to the bootstrapping node.\n" "disable_k8s_bootstrapping: By default, the first k8s node is bootstrapped. This marker disables that.\n" - "dualstack: Support dualstack on the instances.\n" + "no_setup: No setup steps (pushing snap, bootstrapping etc.) are performed on any node for this test.\n" + "network_type: Specify network type to use for the infrastructure (IPv4, Dualstack or IPv6).\n" "etcd_count: Mark a test to specify how many etcd instance nodes need to be created (None by default)\n" - "node_count: Mark a test to specify how many instance nodes need to be created\n", + "node_count: Mark a test to specify how many instance nodes need to be created\n" + "snap_versions: Mark a test to specify snap_versions for each node\n", ) @@ -98,11 +135,25 @@ def node_count(request) -> int: return int(node_count_arg) +def snap_versions(request) -> Iterator[Optional[str]]: + """An endless iterable of snap versions for each node in the test.""" + marking = () + if snap_version_marker := request.node.get_closest_marker("snap_versions"): + marking, *_ = snap_version_marker.args + # endlessly repeat of the configured snap version after exhausting the marking + return itertools.chain(marking, itertools.repeat(None)) + + @pytest.fixture(scope="function") def disable_k8s_bootstrapping(request) -> bool: return bool(request.node.get_closest_marker("disable_k8s_bootstrapping")) +@pytest.fixture(scope="function") +def no_setup(request) -> bool: + return bool(request.node.get_closest_marker("no_setup")) + + @pytest.fixture(scope="function") def bootstrap_config(request) -> Union[str, None]: bootstrap_config_marker = request.node.get_closest_marker("bootstrap_config") @@ -113,41 +164,66 @@ def bootstrap_config(request) -> Union[str, None]: @pytest.fixture(scope="function") -def dualstack(request) -> bool: - return bool(request.node.get_closest_marker("dualstack")) +def network_type(request) -> Union[str, None]: + bootstrap_config_marker = request.node.get_closest_marker("network_type") + if not bootstrap_config_marker: + return "IPv4" + network_type, *_ = bootstrap_config_marker.args + return network_type @pytest.fixture(scope="function") def instances( h: harness.Harness, + registry: Registry, node_count: int, tmp_path: Path, disable_k8s_bootstrapping: bool, + no_setup: bool, bootstrap_config: Union[str, None], - dualstack: bool, + request, + network_type: str, ) -> Generator[List[harness.Instance], None, None]: """Construct instances for a cluster. Bootstrap and setup networking on the first instance, if `disable_k8s_bootstrapping` marker is not set. """ - if not config.SNAP: - pytest.fail("Set TEST_SNAP to the path where the snap is") - if node_count <= 0: pytest.xfail("Test requested 0 or fewer instances, skip this test.") - snap_path = (tmp_path / "k8s.snap").as_posix() - LOG.info(f"Creating {node_count} instances") instances: List[harness.Instance] = [] - for _ in range(node_count): + for _, snap in zip(range(node_count), snap_versions(request)): # Create instances and setup the k8s snap in each. - instance = h.new_instance(dualstack=dualstack) + instance = h.new_instance(network_type=network_type) instances.append(instance) - util.setup_k8s_snap(instance, snap_path) - if not disable_k8s_bootstrapping: + if config.PRELOAD_SNAPS: + for preloaded_snap in PRELOADED_SNAPS: + ack_file = f"{preloaded_snap}.assert" + remote_path = (tmp_path / ack_file).as_posix() + instance.send_file( + source=f"/tmp/{ack_file}", + destination=remote_path, + ) + instance.exec(["snap", "ack", remote_path]) + + snap_file = f"{preloaded_snap}.snap" + remote_path = (tmp_path / snap_file).as_posix() + instance.send_file( + source=f"/tmp/{snap_file}", + destination=remote_path, + ) + instance.exec(["snap", "install", remote_path]) + + if not no_setup: + util.setup_k8s_snap(instance, tmp_path, snap) + + if config.USE_LOCAL_MIRROR: + registry.apply_configuration(instance) + + if not disable_k8s_bootstrapping and not no_setup: first_node, *_ = instances if bootstrap_config is not None: @@ -166,42 +242,17 @@ def instances( # Cleanup after each test. # We cannot execute _harness_clean() here as this would also - # remove the session_instance. The harness ensures that everything is cleaned up + # remove session scoped instances. The harness ensures that everything is cleaned up # at the end of the test session. for instance in instances: if config.INSPECTION_REPORTS_DIR is not None: LOG.debug("Generating inspection reports for test instances") _generate_inspection_report(h, instance.id) - h.delete_instance(instance.id) - - -@pytest.fixture(scope="session") -def session_instance( - h: harness.Harness, tmp_path_factory: pytest.TempPathFactory -) -> Generator[harness.Instance, None, None]: - """Constructs and bootstraps an instance that persists over a test session. - - Bootstraps the instance with all k8sd features enabled to reduce testing time. - """ - LOG.info("Setup node and enable all features") - - snap_path = str(tmp_path_factory.mktemp("data") / "k8s.snap") - instance = h.new_instance() - util.setup_k8s_snap(instance, snap_path) - - bootstrap_config_path = "/home/ubuntu/bootstrap-session.yaml" - instance.send_file( - (config.MANIFESTS_DIR / "bootstrap-session.yaml").as_posix(), - bootstrap_config_path, - ) - - instance.exec(["k8s", "bootstrap", "--file", bootstrap_config_path]) - util.wait_until_k8s_ready(instance, [instance]) - util.wait_for_network(instance) - util.wait_for_dns(instance) - - yield instance + try: + util.remove_k8s_snap(instance) + finally: + h.delete_instance(instance.id) @pytest.fixture(scope="function") diff --git a/tests/integration/tests/test_cleanup.py b/tests/integration/tests/test_cleanup.py index e3fa4e37e..784ac6b4a 100644 --- a/tests/integration/tests/test_cleanup.py +++ b/tests/integration/tests/test_cleanup.py @@ -9,6 +9,13 @@ LOG = logging.getLogger(__name__) +CONTAINERD_PATHS = [ + "/etc/containerd", + "/opt/cni/bin", + "/run/containerd", + "/var/lib/containerd", +] + @pytest.mark.node_count(1) def test_node_cleanup(instances: List[harness.Instance]): @@ -16,40 +23,11 @@ def test_node_cleanup(instances: List[harness.Instance]): util.wait_for_dns(instance) util.wait_for_network(instance) - LOG.info("Uninstall k8s...") - instance.exec(["snap", "remove", "k8s", "--purge"]) - - LOG.info("Waiting for shims to go away...") - util.stubbornly(retries=5, delay_s=5).on(instance).until( - lambda p: all( - x not in p.stdout.decode() - for x in ["containerd-shim", "cilium", "coredns", "/pause"] - ) - ).exec(["ps", "-fea"]) - - LOG.info("Waiting for kubelet and containerd mounts to go away...") - util.stubbornly(retries=5, delay_s=5).on(instance).until( - lambda p: all( - x not in p.stdout.decode() - for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] - ) - ).exec(["mount"]) - - # NOTE(neoaggelos): Temporarily disable this as it fails on strict. - # For details, `snap changes` then `snap change $remove_k8s_snap_change`. - # Example output follows: - # - # 2024-02-23T14:10:42Z ERROR ignoring failure in hook "remove": - # ----- - # ... - # ip netns delete cni-UUID1 - # Cannot remove namespace file "/run/netns/cni-UUID1": Device or resource busy - # ip netns delete cni-UUID2 - # Cannot remove namespace file "/run/netns/cni-UUID2": Device or resource busy - # ip netns delete cni-UUID3 - # Cannot remove namespace file "/run/netns/cni-UUID3": Device or resource busy + util.remove_k8s_snap(instance) - # LOG.info("Waiting for CNI network namespaces to go away...") - # util.stubbornly(retries=5, delay_s=5).on(instance).until( - # lambda p: "cni-" not in p.stdout.decode() - # ).exec(["ip", "netns", "list"]) + # Check that the containerd-related folders are removed on snap removal. + process = instance.exec( + ["ls", *CONTAINERD_PATHS], capture_output=True, text=True, check=False + ) + for path in CONTAINERD_PATHS: + assert f"cannot access '{path}': No such file or directory" in process.stderr diff --git a/tests/integration/tests/test_clustering.py b/tests/integration/tests/test_clustering.py index 57b35ef5b..b5a24df31 100644 --- a/tests/integration/tests/test_clustering.py +++ b/tests/integration/tests/test_clustering.py @@ -1,10 +1,16 @@ # # Copyright 2024 Canonical, Ltd. # +import datetime import logging +import os +import subprocess +import tempfile from typing import List import pytest +from cryptography import x509 +from cryptography.hazmat.backends import default_backend from test_util import config, harness, util LOG = logging.getLogger(__name__) @@ -33,6 +39,31 @@ def test_control_plane_nodes(instances: List[harness.Instance]): ), f"only {cluster_node.id} should be left in cluster" +@pytest.mark.node_count(2) +@pytest.mark.snap_versions([util.previous_track(config.SNAP), config.SNAP]) +def test_mixed_version_join(instances: List[harness.Instance]): + """Test n versioned node joining a n-1 versioned cluster.""" + cluster_node = instances[0] # bootstrapped on the previous channel + joining_node = instances[1] # installed with the snap under test + + join_token = util.get_join_token(cluster_node, joining_node) + util.join_cluster(joining_node, join_token) + + util.wait_until_k8s_ready(cluster_node, instances) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "node should have joined cluster" + + assert "control-plane" in util.get_local_node_status(cluster_node) + assert "control-plane" in util.get_local_node_status(joining_node) + + cluster_node.exec(["k8s", "remove-node", joining_node.id]) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 1, "node should have been removed from cluster" + assert ( + nodes[0]["metadata"]["name"] == cluster_node.id + ), f"only {cluster_node.id} should be left in cluster" + + @pytest.mark.node_count(3) def test_worker_nodes(instances: List[harness.Instance]): cluster_node = instances[0] @@ -88,7 +119,19 @@ def test_no_remove(instances: List[harness.Instance]): assert "control-plane" in util.get_local_node_status(joining_cp) assert "worker" in util.get_local_node_status(joining_worker) - cluster_node.exec(["k8s", "remove-node", joining_cp.id]) + # TODO: k8sd sometimes fails when requested to remove nodes immediately + # after bootstrapping the cluster. It seems that it takes a little + # longer for trust store changes to be propagated to all nodes, which + # should probably be fixed on the microcluster side. + # + # For now, we'll perform some retries. + # + # failed to POST /k8sd/cluster/remove: failed to delete cluster member + # k8s-integration-c1aee0-2: No truststore entry found for node with name + # "k8s-integration-c1aee0-2" + util.stubbornly(retries=3, delay_s=5).on(cluster_node).exec( + ["k8s", "remove-node", joining_cp.id] + ) nodes = util.ready_nodes(cluster_node) assert len(nodes) == 3, "cp node should not have been removed from cluster" cluster_node.exec(["k8s", "remove-node", joining_worker.id]) @@ -96,6 +139,62 @@ def test_no_remove(instances: List[harness.Instance]): assert len(nodes) == 3, "worker node should not have been removed from cluster" +@pytest.mark.node_count(3) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-skip-service-stop.yaml").read_text() +) +def test_skip_services_stop_on_remove(instances: List[harness.Instance]): + cluster_node = instances[0] + joining_cp = instances[1] + worker = instances[2] + + join_token = util.get_join_token(cluster_node, joining_cp) + util.join_cluster(joining_cp, join_token) + + join_token_worker = util.get_join_token(cluster_node, worker, "--worker") + util.join_cluster(worker, join_token_worker) + + util.wait_until_k8s_ready(cluster_node, instances) + + # TODO: skip retrying this once the microcluster trust store issue is addressed. + util.stubbornly(retries=3, delay_s=5).on(cluster_node).exec( + ["k8s", "remove-node", joining_cp.id] + ) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "cp node should have been removed from the cluster" + services = joining_cp.exec( + ["snap", "services", "k8s"], capture_output=True, text=True + ).stdout.split("\n")[1:-1] + print(services) + for service in services: + if "k8s-apiserver-proxy" in service: + assert ( + " inactive " in service + ), "apiserver proxy should be inactive on control-plane" + else: + assert " active " in service, "service should be active" + + cluster_node.exec(["k8s", "remove-node", worker.id]) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 1, "worker node should have been removed from the cluster" + services = worker.exec( + ["snap", "services", "k8s"], capture_output=True, text=True + ).stdout.split("\n")[1:-1] + print(services) + for service in services: + for expected_active_service in [ + "containerd", + "k8sd", + "kubelet", + "kube-proxy", + "k8s-apiserver-proxy", + ]: + if expected_active_service in service: + assert ( + " active " in service + ), f"{expected_active_service} should be active on worker" + + @pytest.mark.node_count(3) def test_join_with_custom_token_name(instances: List[harness.Instance]): cluster_node = instances[0] @@ -149,3 +248,85 @@ def test_join_with_custom_token_name(instances: List[harness.Instance]): cluster_node.exec(["k8s", "remove-node", joining_cp_with_hostname.id]) nodes = util.ready_nodes(cluster_node) assert len(nodes) == 1, "cp node with hostname should be removed from the cluster" + + +@pytest.mark.node_count(2) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-csr-auto-approve.yaml").read_text() +) +def test_cert_refresh(instances: List[harness.Instance]): + cluster_node = instances[0] + joining_worker = instances[1] + + join_token_worker = util.get_join_token(cluster_node, joining_worker, "--worker") + util.join_cluster(joining_worker, join_token_worker) + + util.wait_until_k8s_ready(cluster_node, instances) + nodes = util.ready_nodes(cluster_node) + assert len(nodes) == 2, "nodes should have joined cluster" + + assert "control-plane" in util.get_local_node_status(cluster_node) + assert "worker" in util.get_local_node_status(joining_worker) + + extra_san = "test_san.local" + + def _check_cert(instance, cert_fname): + # Ensure that the certificate was refreshed, having the right expiry date + # and extra SAN. + cert_dir = _get_k8s_cert_dir(instance) + cert_path = os.path.join(cert_dir, cert_fname) + + cert = _get_instance_cert(instance, cert_path) + date = datetime.datetime.now() + assert (cert.not_valid_after - date).days in (364, 365) + + san = cert.extensions.get_extension_for_class(x509.SubjectAlternativeName) + san_dns_names = san.value.get_values_for_type(x509.DNSName) + assert extra_san in san_dns_names + + joining_worker.exec( + ["k8s", "refresh-certs", "--expires-in", "1y", "--extra-sans", extra_san] + ) + + _check_cert(joining_worker, "kubelet.crt") + + cluster_node.exec( + ["k8s", "refresh-certs", "--expires-in", "1y", "--extra-sans", extra_san] + ) + + _check_cert(cluster_node, "kubelet.crt") + _check_cert(cluster_node, "apiserver.crt") + + # Ensure that the services come back online after refreshing the certificates. + util.wait_until_k8s_ready(cluster_node, instances) + + +def _get_k8s_cert_dir(instance: harness.Instance): + tested_paths = [ + "/etc/kubernetes/pki/", + "/var/snap/k8s/common/etc/kubernetes/pki/", + ] + for path in tested_paths: + if _instance_path_exists(instance, path): + return path + + raise Exception("Could not find k8s certificates dir.") + + +def _instance_path_exists(instance: harness.Instance, remote_path: str): + try: + instance.exec(["ls", remote_path]) + return True + except subprocess.CalledProcessError: + return False + + +def _get_instance_cert( + instance: harness.Instance, remote_path: str +) -> x509.Certificate: + with tempfile.NamedTemporaryFile() as fp: + instance.pull_file(remote_path, fp.name) + + pem = fp.read() + cert = x509.load_pem_x509_certificate(pem, default_backend()) + return cert diff --git a/tests/integration/tests/test_dns.py b/tests/integration/tests/test_dns.py index e51285d4b..72b13b82a 100644 --- a/tests/integration/tests/test_dns.py +++ b/tests/integration/tests/test_dns.py @@ -2,14 +2,22 @@ # Copyright 2024 Canonical, Ltd. # import logging +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util LOG = logging.getLogger(__name__) -def test_dns(session_instance: harness.Instance): - session_instance.exec( +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_dns(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + + instance.exec( [ "k8s", "kubectl", @@ -23,7 +31,7 @@ def test_dns(session_instance: harness.Instance): ], ) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -37,14 +45,14 @@ def test_dns(session_instance: harness.Instance): ] ) - result = session_instance.exec( + result = instance.exec( ["k8s", "kubectl", "exec", "busybox", "--", "nslookup", "kubernetes.default"], capture_output=True, ) assert "10.152.183.1 kubernetes.default.svc.cluster.local" in result.stdout.decode() - result = session_instance.exec( + result = instance.exec( ["k8s", "kubectl", "exec", "busybox", "--", "nslookup", "canonical.com"], capture_output=True, check=False, diff --git a/tests/integration/tests/test_dualstack.py b/tests/integration/tests/test_dualstack.py deleted file mode 100644 index 53d586504..000000000 --- a/tests/integration/tests/test_dualstack.py +++ /dev/null @@ -1,58 +0,0 @@ -# -# Copyright 2024 Canonical, Ltd. -# -import logging -from ipaddress import IPv4Address, IPv6Address, ip_address -from typing import List - -import pytest -from test_util import config, harness, util - -LOG = logging.getLogger(__name__) - - -@pytest.mark.node_count(1) -@pytest.mark.bootstrap_config( - (config.MANIFESTS_DIR / "bootstrap-dualstack.yaml").read_text() -) -@pytest.mark.dualstack() -def test_dualstack(instances: List[harness.Instance]): - main = instances[0] - dualstack_config = (config.MANIFESTS_DIR / "nginx-dualstack.yaml").read_text() - - # Deploy nginx with dualstack service - main.exec( - ["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(dualstack_config) - ) - addresses = ( - util.stubbornly(retries=5, delay_s=3) - .on(main) - .exec( - [ - "k8s", - "kubectl", - "get", - "svc", - "nginx-dualstack", - "-o", - "jsonpath='{.spec.clusterIPs[*]}'", - ], - text=True, - capture_output=True, - ) - .stdout - ) - - for ip in addresses.split(): - addr = ip_address(ip.strip("'")) - if isinstance(addr, IPv6Address): - address = f"http://[{str(addr)}]" - elif isinstance(addr, IPv4Address): - address = f"http://{str(addr)}" - else: - pytest.fail(f"Unknown IP address type: {addr}") - - # need to shell out otherwise this runs into permission errors - util.stubbornly(retries=3, delay_s=1).on(main).exec( - ["curl", address], shell=True - ) diff --git a/tests/integration/tests/test_gateway.py b/tests/integration/tests/test_gateway.py index 9770af46f..c2116e7cb 100644 --- a/tests/integration/tests/test_gateway.py +++ b/tests/integration/tests/test_gateway.py @@ -3,9 +3,13 @@ # import json import logging +import subprocess +import time from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -36,20 +40,63 @@ def get_gateway_service_node_port(p): return None -def test_gateway(session_instance: harness.Instance): +def get_external_service_ip(instance: harness.Instance) -> str: + try_count = 0 + gateway_ip = None + while gateway_ip is None and try_count < 5: + try_count += 1 + try: + gateway_ip = ( + instance.exec( + [ + "k8s", + "kubectl", + "get", + "gateway", + "my-gateway", + "-o=jsonpath='{.status.addresses[0].value}'", + ], + capture_output=True, + ) + .stdout.decode() + .replace("'", "") + ) + except subprocess.CalledProcessError: + gateway_ip = None + pass + time.sleep(3) + return gateway_ip + + +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_gateway(instances: List[harness.Instance]): + instance = instances[0] + instance_default_ip = util.get_default_ip(instance) + instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) + lb_cidr = util.find_suitable_cidr( + parent_cidr=instance_default_cidr, + excluded_ips=[instance_default_ip], + ) + instance.exec( + ["k8s", "set", f"load-balancer.cidrs={lb_cidr}", "load-balancer.l2-mode=true"] + ) + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + manifest = MANIFESTS_DIR / "gateway-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for nginx pod to show up...") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "my-nginx" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Nginx pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -67,14 +114,21 @@ def test_gateway(session_instance: harness.Instance): gateway_http_port = None result = ( util.stubbornly(retries=7, delay_s=3) - .on(session_instance) + .on(instance) .until(lambda p: get_gateway_service_node_port(p) is not None) .exec(["k8s", "kubectl", "get", "service", "-o", "json"]) ) gateway_http_port = get_gateway_service_node_port(result) - assert gateway_http_port is not None, "No ingress nodePort found." + assert gateway_http_port is not None, "No Gateway nodePort found." - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + # Test the Gateway service via loadbalancer IP. + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", f"localhost:{gateway_http_port}"]) + + gateway_ip = get_external_service_ip(instance) + assert gateway_ip is not None, "No Gateway IP found." + util.stubbornly(retries=5, delay_s=5).on(instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", f"{gateway_ip}", "-H", "Host: foo.bar.com"]) diff --git a/tests/integration/tests/test_ingress.py b/tests/integration/tests/test_ingress.py index f2cccc61c..c39115f83 100644 --- a/tests/integration/tests/test_ingress.py +++ b/tests/integration/tests/test_ingress.py @@ -3,10 +3,13 @@ # import json import logging +import subprocess +import time from pathlib import Path from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -35,11 +38,60 @@ def get_ingress_service_node_port(p): return None -def test_ingress(session_instance: List[harness.Instance]): +def get_external_service_ip(instance: harness.Instance, service_namespace) -> str: + try_count = 0 + ingress_ip = None + while ingress_ip is None and try_count < 5: + try_count += 1 + for svcns in service_namespace: + svc = svcns["service"] + namespace = svcns["namespace"] + try: + ingress_ip = ( + instance.exec( + [ + "k8s", + "kubectl", + "--namespace", + namespace, + "get", + "service", + svc, + "-o=jsonpath='{.status.loadBalancer.ingress[0].ip}'", + ], + capture_output=True, + ) + .stdout.decode() + .replace("'", "") + ) + if ingress_ip is not None: + return ingress_ip + except subprocess.CalledProcessError: + ingress_ip = None + pass + time.sleep(3) + return ingress_ip + + +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_ingress(instances: List[harness.Instance]): + instance = instances[0] + instance_default_ip = util.get_default_ip(instance) + instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) + lb_cidr = util.find_suitable_cidr( + parent_cidr=instance_default_cidr, + excluded_ips=[instance_default_ip], + ) + instance.exec( + ["k8s", "set", f"load-balancer.cidrs={lb_cidr}", "load-balancer.l2-mode=true"] + ) + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) result = ( util.stubbornly(retries=7, delay_s=3) - .on(session_instance) + .on(instance) .until(lambda p: get_ingress_service_node_port(p) is not None) .exec(["k8s", "kubectl", "get", "service", "-A", "-o", "json"]) ) @@ -49,18 +101,18 @@ def test_ingress(session_instance: List[harness.Instance]): assert ingress_http_port is not None, "No ingress nodePort found." manifest = MANIFESTS_DIR / "ingress-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for nginx pod to show up...") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "my-nginx" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Nginx pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -74,6 +126,19 @@ def test_ingress(session_instance: List[harness.Instance]): ] ) - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", f"localhost:{ingress_http_port}", "-H", "Host: foo.bar.com"]) + + # Test the ingress service via loadbalancer IP + ingress_ip = get_external_service_ip( + instance, + [ + {"service": "ck-ingress-contour-envoy", "namespace": "projectcontour"}, + {"service": "cilium-ingress", "namespace": "kube-system"}, + ], + ) + assert ingress_ip is not None, "No ingress IP found." + util.stubbornly(retries=5, delay_s=5).on(instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", f"{ingress_ip}", "-H", "Host: foo.bar.com"]) diff --git a/tests/integration/tests/test_loadbalancer.py b/tests/integration/tests/test_loadbalancer.py index 9f882d7ed..97fb69e88 100644 --- a/tests/integration/tests/test_loadbalancer.py +++ b/tests/integration/tests/test_loadbalancer.py @@ -1,7 +1,6 @@ # # Copyright 2024 Canonical, Ltd. # -import ipaddress import logging from pathlib import Path from typing import List @@ -13,29 +12,6 @@ LOG = logging.getLogger(__name__) -def find_suitable_cidr(parent_cidr: str, excluded_ips: List[str]): - net = ipaddress.IPv4Network(parent_cidr, False) - - # Starting from the first IP address from the parent cidr, - # we search for a /30 cidr block(4 total ips, 2 available) - # that doesn't contain the excluded ips to avoid collisions - # /30 because this is the smallest CIDR cilium hands out IPs from - for i in range(4, 255, 4): - lb_net = ipaddress.IPv4Network(f"{str(net[0]+i)}/30", False) - - contains_excluded = False - for excluded in excluded_ips: - if ipaddress.ip_address(excluded) in lb_net: - contains_excluded = True - break - - if contains_excluded: - continue - - return str(lb_net) - raise RuntimeError("Could not find a suitable CIDR for LoadBalancer services") - - @pytest.mark.node_count(2) def test_loadbalancer(instances: List[harness.Instance]): instance = instances[0] @@ -47,7 +23,7 @@ def test_loadbalancer(instances: List[harness.Instance]): instance_default_cidr = util.get_default_cidr(instance, instance_default_ip) - lb_cidr = find_suitable_cidr( + lb_cidr = util.find_suitable_cidr( parent_cidr=instance_default_cidr, excluded_ips=[instance_default_ip, tester_instance_default_ip], ) @@ -107,9 +83,6 @@ def test_loadbalancer(instances: List[harness.Instance]): ) service_ip = p.stdout.decode().replace("'", "") - p = tester_instance.exec( - ["curl", service_ip], - capture_output=True, - ) - - assert "Welcome to nginx!" in p.stdout.decode() + util.stubbornly(retries=5, delay_s=3).on(tester_instance).until( + lambda p: "Welcome to nginx!" in p.stdout.decode() + ).exec(["curl", service_ip]) diff --git a/tests/integration/tests/test_metrics_server.py b/tests/integration/tests/test_metrics_server.py index 1fa0331c9..0759a41e4 100644 --- a/tests/integration/tests/test_metrics_server.py +++ b/tests/integration/tests/test_metrics_server.py @@ -2,20 +2,28 @@ # Copyright 2024 Canonical, Ltd. # import logging +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util LOG = logging.getLogger(__name__) -def test_metrics_server(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_metrics_server(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + LOG.info("Waiting for metrics-server pod to show up...") - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( + util.stubbornly(retries=15, delay_s=5).on(instance).until( lambda p: "metrics-server" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-n", "kube-system", "-o", "json"]) LOG.info("Metrics-server pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -31,6 +39,6 @@ def test_metrics_server(session_instance: harness.Instance): ] ) - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( - lambda p: session_instance.id in p.stdout.decode() + util.stubbornly(retries=15, delay_s=5).on(instance).until( + lambda p: instance.id in p.stdout.decode() ).exec(["k8s", "kubectl", "top", "node"]) diff --git a/tests/integration/tests/test_network.py b/tests/integration/tests/test_network.py index e4d483b4a..838a5c249 100644 --- a/tests/integration/tests/test_network.py +++ b/tests/integration/tests/test_network.py @@ -4,21 +4,29 @@ import json import logging from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) -def test_network(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_network(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + manifest = MANIFESTS_DIR / "nginx-pod.yaml" - p = session_instance.exec( + p = instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -32,7 +40,7 @@ def test_network(session_instance: harness.Instance): ] ) - p = session_instance.exec( + p = instance.exec( [ "k8s", "kubectl", @@ -51,6 +59,6 @@ def test_network(session_instance: harness.Instance): assert len(out["items"]) > 0, "No NGINX pod found" podIP = out["items"][0]["status"]["podIP"] - util.stubbornly(retries=5, delay_s=5).on(session_instance).until( + util.stubbornly(retries=5, delay_s=5).on(instance).until( lambda p: "Welcome to nginx!" in p.stdout.decode() ).exec(["curl", "-s", f"http://{podIP}"]) diff --git a/tests/integration/tests/test_networking.py b/tests/integration/tests/test_networking.py new file mode 100644 index 000000000..1804ab1fa --- /dev/null +++ b/tests/integration/tests/test_networking.py @@ -0,0 +1,123 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from ipaddress import IPv4Address, IPv6Address, ip_address +from typing import List + +import pytest +from test_util import config, harness, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.bootstrap_config( + (config.MANIFESTS_DIR / "bootstrap-dualstack.yaml").read_text() +) +@pytest.mark.dualstack() +def test_dualstack(instances: List[harness.Instance]): + main = instances[0] + dualstack_config = (config.MANIFESTS_DIR / "nginx-dualstack.yaml").read_text() + + # Deploy nginx with dualstack service + main.exec( + ["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(dualstack_config) + ) + addresses = ( + util.stubbornly(retries=5, delay_s=3) + .on(main) + .exec( + [ + "k8s", + "kubectl", + "get", + "svc", + "nginx-dualstack", + "-o", + "jsonpath='{.spec.clusterIPs[*]}'", + ], + text=True, + capture_output=True, + ) + .stdout + ) + + for ip in addresses.split(): + addr = ip_address(ip.strip("'")) + if isinstance(addr, IPv6Address): + address = f"http://[{str(addr)}]" + elif isinstance(addr, IPv4Address): + address = f"http://{str(addr)}" + else: + pytest.fail(f"Unknown IP address type: {addr}") + + # need to shell out otherwise this runs into permission errors + util.stubbornly(retries=3, delay_s=1).on(main).exec( + ["curl", address], shell=True + ) + + +@pytest.mark.node_count(3) +@pytest.mark.disable_k8s_bootstrapping() +@pytest.mark.network_type("dualstack") +def test_ipv6_only_on_dualstack_infra(instances: List[harness.Instance]): + main = instances[0] + joining_cp = instances[1] + joining_worker = instances[2] + + ipv6_bootstrap_config = ( + config.MANIFESTS_DIR / "bootstrap-ipv6-only.yaml" + ).read_text() + + main.exec( + ["k8s", "bootstrap", "--file", "-", "--address", "::/0"], + input=str.encode(ipv6_bootstrap_config), + ) + + join_token = util.get_join_token(main, joining_cp) + joining_cp.exec(["k8s", "join-cluster", join_token, "--address", "::/0"]) + + join_token_worker = util.get_join_token(main, joining_worker, "--worker") + joining_worker.exec(["k8s", "join-cluster", join_token_worker, "--address", "::/0"]) + + # Deploy nginx with ipv6 service + ipv6_config = (config.MANIFESTS_DIR / "nginx-ipv6-only.yaml").read_text() + main.exec(["k8s", "kubectl", "apply", "-f", "-"], input=str.encode(ipv6_config)) + addresses = ( + util.stubbornly(retries=5, delay_s=3) + .on(main) + .exec( + [ + "k8s", + "kubectl", + "get", + "svc", + "nginx-ipv6", + "-o", + "jsonpath='{.spec.clusterIPs[*]}'", + ], + text=True, + capture_output=True, + ) + .stdout + ) + + for ip in addresses.split(): + addr = ip_address(ip.strip("'")) + if isinstance(addr, IPv6Address): + address = f"http://[{str(addr)}]" + elif isinstance(addr, IPv4Address): + assert False, "IPv4 address found in IPv6-only cluster" + else: + pytest.fail(f"Unknown IP address type: {addr}") + + # need to shell out otherwise this runs into permission errors + util.stubbornly(retries=3, delay_s=1).on(main).exec( + ["curl", address], shell=True + ) + + # This might take a while + util.stubbornly(retries=config.DEFAULT_WAIT_RETRIES, delay_s=20).until( + util.ready_nodes(main) == 3 + ) diff --git a/tests/integration/tests/test_smoke.py b/tests/integration/tests/test_smoke.py index c5dd95c38..ab2ee7552 100644 --- a/tests/integration/tests/test_smoke.py +++ b/tests/integration/tests/test_smoke.py @@ -66,6 +66,7 @@ def test_smoke(instances: List[harness.Instance]): LOG.info("Verify the functionality of the CAPI endpoints.") instance.exec("k8s x-capi set-auth-token my-secret-token".split()) + instance.exec("k8s x-capi set-node-token my-node-token".split()) body = { "name": "my-node", @@ -89,7 +90,6 @@ def test_smoke(instances: List[harness.Instance]): capture_output=True, ) response = json.loads(resp.stdout.decode()) - assert ( response["error_code"] == 0 ), "Failed to generate join token using CAPI endpoints." @@ -101,6 +101,32 @@ def test_smoke(instances: List[harness.Instance]): metadata.get("token") is not None ), "Token not found in the generate-join-token response." + resp = instance.exec( + [ + "curl", + "-XPOST", + "-H", + "Content-Type: application/json", + "-H", + "node-token: my-node-token", + "--unix-socket", + "/var/snap/k8s/common/var/lib/k8sd/state/control.socket", + "http://localhost/1.0/x/capi/certificates-expiry", + ], + capture_output=True, + ) + response = json.loads(resp.stdout.decode()) + assert ( + response["error_code"] == 0 + ), "Failed to get certificate expiry using CAPI endpoints." + metadata = response.get("metadata") + assert ( + metadata is not None + ), "Metadata not found in the certificate expiry response." + assert util.is_valid_rfc3339( + metadata.get("expiry-date") + ), "Token not found in the certificate expiry response." + def status_output_matches(p: subprocess.CompletedProcess) -> bool: result_lines = p.stdout.decode().strip().split("\n") if len(result_lines) != len(STATUS_PATTERNS): diff --git a/tests/integration/tests/test_storage.py b/tests/integration/tests/test_storage.py index 2a8ce2c6f..497e401d9 100644 --- a/tests/integration/tests/test_storage.py +++ b/tests/integration/tests/test_storage.py @@ -5,8 +5,10 @@ import logging import subprocess from pathlib import Path +from typing import List -from test_util import harness, util +import pytest +from test_util import config, harness, util from test_util.config import MANIFESTS_DIR LOG = logging.getLogger(__name__) @@ -20,14 +22,20 @@ def check_pvc_bound(p: subprocess.CompletedProcess) -> bool: return False -def test_storage(session_instance: harness.Instance): +@pytest.mark.bootstrap_config((config.MANIFESTS_DIR / "bootstrap-all.yaml").read_text()) +def test_storage(instances: List[harness.Instance]): + instance = instances[0] + util.wait_until_k8s_ready(instance, [instance]) + util.wait_for_network(instance) + util.wait_for_dns(instance) + LOG.info("Waiting for storage provisioner pod to show up...") - util.stubbornly(retries=15, delay_s=5).on(session_instance).until( + util.stubbornly(retries=15, delay_s=5).on(instance).until( lambda p: "ck-storage" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-n", "kube-system", "-o", "json"]) LOG.info("Storage provisioner pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -44,18 +52,18 @@ def test_storage(session_instance: harness.Instance): ) manifest = MANIFESTS_DIR / "storage-setup.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for storage writer pod to show up...") - util.stubbornly(retries=3, delay_s=10).on(session_instance).until( + util.stubbornly(retries=3, delay_s=10).on(instance).until( lambda p: "storage-writer-pod" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Storage writer pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -70,16 +78,16 @@ def test_storage(session_instance: harness.Instance): ) LOG.info("Waiting for storage to get provisioned...") - util.stubbornly(retries=3, delay_s=1).on(session_instance).until( - check_pvc_bound - ).exec(["k8s", "kubectl", "get", "pvc", "-o", "json"]) + util.stubbornly(retries=3, delay_s=1).on(instance).until(check_pvc_bound).exec( + ["k8s", "kubectl", "get", "pvc", "-o", "json"] + ) LOG.info("Storage got provisioned and pvc is bound.") - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "LOREM IPSUM" in p.stdout.decode() ).exec(["k8s", "kubectl", "logs", "storage-writer-pod"]) - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -92,18 +100,18 @@ def test_storage(session_instance: harness.Instance): ) manifest = MANIFESTS_DIR / "storage-test.yaml" - session_instance.exec( + instance.exec( ["k8s", "kubectl", "apply", "-f", "-"], input=Path(manifest).read_bytes(), ) LOG.info("Waiting for storage reader pod to show up...") - util.stubbornly(retries=3, delay_s=10).on(session_instance).until( + util.stubbornly(retries=3, delay_s=10).on(instance).until( lambda p: "storage-reader-pod" in p.stdout.decode() ).exec(["k8s", "kubectl", "get", "pod", "-o", "json"]) LOG.info("Storage reader pod showed up.") - util.stubbornly(retries=3, delay_s=1).on(session_instance).exec( + util.stubbornly(retries=3, delay_s=1).on(instance).exec( [ "k8s", "kubectl", @@ -117,7 +125,7 @@ def test_storage(session_instance: harness.Instance): ] ) - util.stubbornly(retries=5, delay_s=10).on(session_instance).until( + util.stubbornly(retries=5, delay_s=10).on(instance).until( lambda p: "LOREM IPSUM" in p.stdout.decode() ).exec(["k8s", "kubectl", "logs", "storage-reader-pod"]) diff --git a/tests/integration/tests/test_strict_interfaces.py b/tests/integration/tests/test_strict_interfaces.py new file mode 100644 index 000000000..58d5df75e --- /dev/null +++ b/tests/integration/tests/test_strict_interfaces.py @@ -0,0 +1,75 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from typing import List + +import pytest +from test_util import config, harness, snap, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.no_setup() +@pytest.mark.skipif( + not config.STRICT_INTERFACE_CHANNELS, reason="No strict channels configured" +) +def test_strict_interfaces(instances: List[harness.Instance], tmp_path): + channels = config.STRICT_INTERFACE_CHANNELS + cp = instances[0] + current_channel = channels[0] + + if current_channel.lower() == "recent": + if len(channels) != 3: + pytest.fail( + "'recent' requires the number of releases as second argument and the flavour as third argument" + ) + _, num_channels, flavour = channels + channels = snap.get_channels(int(num_channels), flavour, cp.arch, "edge", True) + + for channel in channels: + util.setup_k8s_snap(cp, tmp_path, channel, connect_interfaces=False) + + # Log the current snap version on the node. + out = cp.exec(["snap", "list", config.SNAP_NAME], capture_output=True) + LOG.info(f"Current snap version: {out.stdout.decode().strip()}") + + check_snap_interfaces(cp, config.SNAP_NAME) + + cp.exec(["snap", "remove", config.SNAP_NAME, "--purge"]) + + +def check_snap_interfaces(cp, snap_name): + """Check the strict snap interfaces.""" + interfaces = [ + "docker-privileged", + "kubernetes-support", + "network", + "network-bind", + "network-control", + "network-observe", + "firewall-control", + "process-control", + "kernel-module-observe", + "cilium-module-load", + "mount-observe", + "hardware-observe", + "system-observe", + "home", + "opengl", + "home-read-all", + "login-session-observe", + "log-observe", + ] + for interface in interfaces: + cp.exec( + [ + "snap", + "run", + "--shell", + snap_name, + "-c", + f"snapctl is-connected {interface}", + ], + ) diff --git a/tests/integration/tests/test_util/config.py b/tests/integration/tests/test_util/config.py index 2778d893f..4ba31b337 100644 --- a/tests/integration/tests/test_util/config.py +++ b/tests/integration/tests/test_util/config.py @@ -1,11 +1,16 @@ # # Copyright 2024 Canonical, Ltd. # +import json import os from pathlib import Path DIR = Path(__file__).absolute().parent +# The following defaults are used to define how long to wait for a condition to be met. +DEFAULT_WAIT_RETRIES = int(os.getenv("TEST_DEFAULT_WAIT_RETRIES") or 120) +DEFAULT_WAIT_DELAY_S = int(os.getenv("TEST_DEFAULT_WAIT_DELAY_S") or 5) + MANIFESTS_DIR = DIR / ".." / ".." / "templates" # ETCD_DIR contains all templates required to setup an etcd database. @@ -15,11 +20,29 @@ ETCD_URL = os.getenv("ETCD_URL") or "https://github.com/etcd-io/etcd/releases/download" # ETCD_VERSION is the version of etcd to use. -ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.3.8" +ETCD_VERSION = os.getenv("ETCD_VERSION") or "v3.4.34" + +# REGISTRY_DIR contains all templates required to setup an registry mirror. +REGISTRY_DIR = MANIFESTS_DIR / "registry" + +# REGISTRY_URL is the url from which the registry binary should be downloaded. +REGISTRY_URL = ( + os.getenv("REGISTRY_URL") + or "https://github.com/distribution/distribution/releases/download" +) + +# REGISTRY_VERSION is the version of registry to use. +REGISTRY_VERSION = os.getenv("REGISTRY_VERSION") or "v2.8.3" + +# FLAVOR is the flavor of the snap to use. +FLAVOR = os.getenv("TEST_FLAVOR") or "" # SNAP is the absolute path to the snap against which we run the integration tests. SNAP = os.getenv("TEST_SNAP") +# SNAP_NAME is the name of the snap under test. +SNAP_NAME = os.getenv("TEST_SNAP_NAME") or "k8s" + # SUBSTRATE is the substrate to use for running the integration tests. # One of 'local' (default), 'lxd', 'juju', or 'multipass'. SUBSTRATE = os.getenv("TEST_SUBSTRATE") or "local" @@ -55,6 +78,20 @@ or (DIR / ".." / ".." / "lxd-dualstack-profile.yaml").read_text() ) +# LXD_IPV6_NETWORK is the network to use for LXD containers with ipv6-only configured. +LXD_IPV6_NETWORK = os.getenv("TEST_LXD_IPV6_NETWORK") or "ipv6-br0" + +# LXD_IPV6_PROFILE_NAME is the profile name to use for LXD containers with ipv6-only configured. +LXD_IPV6_PROFILE_NAME = ( + os.getenv("TEST_LXD_IPV6_PROFILE_NAME") or "k8s-integration-ipv6" +) + +# LXD_IPV6_PROFILE is the profile to use for LXD containers with ipv6-only configured. +LXD_IPV6_PROFILE = ( + os.getenv("TEST_LXD_IPV6_PROFILE") + or (DIR / ".." / ".." / "lxd-ipv6-profile.yaml").read_text() +) + # LXD_IMAGE is the image to use for LXD containers. LXD_IMAGE = os.getenv("TEST_LXD_IMAGE") or "ubuntu:22.04" @@ -88,3 +125,36 @@ # JUJU_MACHINES is a list of existing Juju machines to use. JUJU_MACHINES = os.getenv("TEST_JUJU_MACHINES") or "" + +# A list of space-separated channels for which the upgrade tests should be run in sequential order. +# First entry is the bootstrap channel. Afterwards, upgrades are done in order. +# Alternatively, use 'recent ' to get the latest channels for . +VERSION_UPGRADE_CHANNELS = ( + os.environ.get("TEST_VERSION_UPGRADE_CHANNELS", "").strip().split() +) + +# The minimum Kubernetes release to upgrade from (e.g. "1.31") +# Only relevant when using 'recent' in VERSION_UPGRADE_CHANNELS. +VERSION_UPGRADE_MIN_RELEASE = os.environ.get("TEST_VERSION_UPGRADE_MIN_RELEASE") + +# A list of space-separated channels for which the strict interface tests should be run in sequential order. +# Alternatively, use 'recent strict' to get the latest channels for strict. +STRICT_INTERFACE_CHANNELS = ( + os.environ.get("TEST_STRICT_INTERFACE_CHANNELS", "").strip().split() +) + +# Cache and preload certain snaps such as snapd and core20 to avoid fetching them +# for every test instance. Note that k8s-snap is currently based on core20. +PRELOAD_SNAPS = (os.getenv("TEST_PRELOAD_SNAPS") or "1") == "1" + +# Setup a local image mirror to reduce the number of image pulls. The mirror +# will be configured to run in a session scoped harness instance (e.g. LXD container) +USE_LOCAL_MIRROR = (os.getenv("TEST_USE_LOCAL_MIRROR") or "1") == "1" + +DEFAULT_MIRROR_LIST = [ + {"name": "ghcr.io", "port": 5000, "remote": "https://ghcr.io"}, + {"name": "docker.io", "port": 5001, "remote": "https://registry-1.docker.io"}, +] + +# Local mirror configuration. +MIRROR_LIST = json.loads(os.getenv("TEST_MIRROR_LIST", "{}")) or DEFAULT_MIRROR_LIST diff --git a/tests/integration/tests/test_util/etcd.py b/tests/integration/tests/test_util/etcd.py index 44f8a0eee..d53e8ee66 100644 --- a/tests/integration/tests/test_util/etcd.py +++ b/tests/integration/tests/test_util/etcd.py @@ -90,6 +90,7 @@ def add_node(self): ] substitutes = { + "ARCH": instance.arch, "NAME": instance.id, "IP": ip, "CLIENT_URL": f"https://{ip}:2379", @@ -224,13 +225,14 @@ def add_node(self): input=str.encode(src.substitute(substitutes)), ) + arch = instance.arch instance.exec( [ "curl", "-L", - f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"{self.etcd_url}/{self.etcd_version}/etcd-{self.etcd_version}-linux-{arch}.tar.gz", "-o", - f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz", ] ) instance.exec(["mkdir", "-p", "/tmp/test-etcd"]) @@ -238,7 +240,7 @@ def add_node(self): [ "tar", "xzvf", - f"/tmp/etcd-{self.etcd_version}-linux-amd64.tar.gz", + f"/tmp/etcd-{self.etcd_version}-linux-{arch}.tar.gz", "-C", "/tmp/test-etcd", "--strip-components=1", diff --git a/tests/integration/tests/test_util/harness/base.py b/tests/integration/tests/test_util/harness/base.py index 829d64511..7e01ea04f 100644 --- a/tests/integration/tests/test_util/harness/base.py +++ b/tests/integration/tests/test_util/harness/base.py @@ -2,7 +2,7 @@ # Copyright 2024 Canonical, Ltd. # import subprocess -from functools import partial +from functools import cached_property, partial class HarnessError(Exception): @@ -30,6 +30,13 @@ def __init__(self, h: "Harness", id: str) -> None: def id(self) -> str: return self._id + @cached_property + def arch(self) -> str: + """Return the architecture of the instance""" + return self.exec( + ["dpkg", "--print-architecture"], text=True, capture_output=True + ).stdout.strip() + def __str__(self) -> str: return f"{self._h.name}:{self.id}" @@ -42,7 +49,7 @@ class Harness: name: str - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: """Creates a new instance on the infrastructure and returns an object which can be used to interact with it. diff --git a/tests/integration/tests/test_util/harness/juju.py b/tests/integration/tests/test_util/harness/juju.py index d8e3a694c..ad89e956e 100644 --- a/tests/integration/tests/test_util/harness/juju.py +++ b/tests/integration/tests/test_util/harness/juju.py @@ -53,9 +53,9 @@ def __init__(self): self.constraints, ) - def new_instance(self, dualstack: bool = False) -> Instance: - if dualstack: - raise HarnessError("Dualstack is currently not supported by Juju harness") + def new_instance(self, network_type: str = "IPv4") -> Instance: + if network_type: + raise HarnessError("Currently only IPv4 is supported by Juju harness") for instance_id in self.existing_machines: if not self.existing_machines[instance_id]: diff --git a/tests/integration/tests/test_util/harness/local.py b/tests/integration/tests/test_util/harness/local.py index 2b790c6cf..7c71b2970 100644 --- a/tests/integration/tests/test_util/harness/local.py +++ b/tests/integration/tests/test_util/harness/local.py @@ -27,12 +27,12 @@ def __init__(self): LOG.debug("Configured local substrate") - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: if self.initialized: raise HarnessError("local substrate only supports up to one instance") - if dualstack: - raise HarnessError("Dualstack is currently not supported by Local harness") + if network_type != "IPv4": + raise HarnessError("Currently only IPv4 is supported by Local harness") self.initialized = True LOG.debug("Initializing instance") diff --git a/tests/integration/tests/test_util/harness/lxd.py b/tests/integration/tests/test_util/harness/lxd.py index bc2c3909e..757c17cb6 100644 --- a/tests/integration/tests/test_util/harness/lxd.py +++ b/tests/integration/tests/test_util/harness/lxd.py @@ -52,11 +52,26 @@ def __init__(self): ), ) + self._configure_network( + config.LXD_IPV6_NETWORK, + "ipv4.address=none", + "ipv6.address=auto", + "ipv4.nat=false", + "ipv6.nat=true", + ) + self.ipv6_profile = config.LXD_IPV6_PROFILE_NAME + self._configure_profile( + self.ipv6_profile, + config.LXD_IPV6_PROFILE.replace( + "LXD_IPV6_NETWORK", config.LXD_IPV6_NETWORK + ), + ) + LOG.debug( "Configured LXD substrate (profile %s, image %s)", self.profile, self.image ) - def new_instance(self, dualstack: bool = False) -> Instance: + def new_instance(self, network_type: str = "IPv4") -> Instance: instance_id = f"k8s-integration-{os.urandom(3).hex()}-{self.next_id()}" LOG.debug("Creating instance %s with image %s", instance_id, self.image) @@ -71,9 +86,17 @@ def new_instance(self, dualstack: bool = False) -> Instance: self.profile, ] - if dualstack: + if network_type.lower() not in ["ipv4", "dualstack", "ipv6"]: + raise HarnessError( + f"unknown network type {network_type}, need to be one of 'IPv4', 'IPv6', 'dualstack'" + ) + + if network_type.lower() == "dualstack": launch_lxd_command.extend(["-p", self.dualstack_profile]) + if network_type.lower() == "ipv6": + launch_lxd_command.extend(["-p", self.ipv6_profile]) + try: stubbornly(retries=3, delay_s=1).exec(launch_lxd_command) self.instances.add(instance_id) @@ -205,9 +228,17 @@ def delete_instance(self, instance_id: str): raise HarnessError(f"unknown instance {instance_id}") try: - run(["lxc", "rm", instance_id, "--force"]) + # There are cases where the instance is not deleted properly and this command is stuck. + # A timeout prevents this. + # TODO(ben): This is a workaround for an issue that arises because of our use of + # privileged containers. We eventually move away from this (not supported >24.10) + # which should also fix this issue and make this timeout unnecessary. + run(["lxc", "rm", instance_id, "--force"], timeout=60 * 5) except subprocess.CalledProcessError as e: raise HarnessError(f"failed to delete instance {instance_id}") from e + except subprocess.TimeoutExpired: + LOG.warning("LXC container removal timed out.") + pass self.instances.discard(instance_id) diff --git a/tests/integration/tests/test_util/harness/multipass.py b/tests/integration/tests/test_util/harness/multipass.py index 4cea3a194..218b3eb17 100644 --- a/tests/integration/tests/test_util/harness/multipass.py +++ b/tests/integration/tests/test_util/harness/multipass.py @@ -36,11 +36,9 @@ def __init__(self): LOG.debug("Configured Multipass substrate (image %s)", self.image) - def new_instance(self, dualstack: bool = False) -> Instance: - if dualstack: - raise HarnessError( - "Dualstack is currently not supported by Multipass harness" - ) + def new_instance(self, network_type: str = "IPv4") -> Instance: + if network_type: + raise HarnessError("Currently only IPv4 is supported by Multipass harness") instance_id = f"k8s-integration-{os.urandom(3).hex()}-{self.next_id()}" diff --git a/tests/integration/tests/test_util/registry.py b/tests/integration/tests/test_util/registry.py new file mode 100644 index 000000000..6f2bd0e52 --- /dev/null +++ b/tests/integration/tests/test_util/registry.py @@ -0,0 +1,178 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from string import Template +from typing import List, Optional + +from test_util import config +from test_util.harness import Harness, Instance +from test_util.util import get_default_ip + +LOG = logging.getLogger(__name__) + + +class Mirror: + def __init__( + self, + name: str, + port: int, + remote: str, + username: Optional[str] = None, + password: Optional[str] = None, + ): + """ + Initialize the Mirror object. + + Args: + name (str): The name of the mirror. + port (int): The port of the mirror. + remote (str): The remote URL of the upstream registry. + username (str, optional): Authentication username. + password (str, optional): Authentication password. + """ + self.name = name + self.port = port + self.remote = remote + self.username = username + self.password = password + + +class Registry: + + def __init__(self, h: Harness): + """ + Initialize the Registry object. + + Args: + h (Harness): The test harness object. + """ + self.registry_url = config.REGISTRY_URL + self.registry_version = config.REGISTRY_VERSION + self.instance: Instance = None + self.harness: Harness = h + self._mirrors: List[Mirror] = self.get_configured_mirrors() + self.instance = self.harness.new_instance() + + arch = self.instance.arch + self.instance.exec( + [ + "curl", + "-L", + f"{self.registry_url}/{self.registry_version}/registry_{self.registry_version[1:]}_linux_{arch}.tar.gz", + "-o", + f"/tmp/registry_{self.registry_version}_linux_{arch}.tar.gz", + ] + ) + + self.instance.exec( + [ + "tar", + "xzvf", + f"/tmp/registry_{self.registry_version}_linux_{arch}.tar.gz", + "-C", + "/bin/", + "registry", + ], + ) + + self._ip = get_default_ip(self.instance) + + self.add_mirrors() + + def get_configured_mirrors(self) -> List[Mirror]: + mirrors: List[Mirror] = [] + for mirror_dict in config.MIRROR_LIST: + for field in ["name", "port", "remote"]: + if field not in mirror_dict: + raise Exception( + f"Invalid 'TEST_MIRROR_LIST' configuration. Missing field: {field}" + ) + + mirror = Mirror( + mirror_dict["name"], + mirror_dict["port"], + mirror_dict["remote"], + mirror_dict.get("username"), + mirror_dict.get("password"), + ) + mirrors.append(mirror) + return mirrors + + def add_mirrors(self): + for mirror in self._mirrors: + self.add_mirror(mirror) + + def add_mirror(self, mirror: Mirror): + substitutes = { + "NAME": mirror.name, + "PORT": mirror.port, + "REMOTE": mirror.remote, + "USERNAME": mirror.username or "", + "PASSWORD": mirror.password or "", + } + + self.instance.exec(["mkdir", "-p", "/etc/distribution"]) + self.instance.exec(["mkdir", "-p", f"/var/lib/registry/{mirror.name}"]) + + with open( + config.REGISTRY_DIR / "registry-config.yaml", "r" + ) as registry_template: + src = Template(registry_template.read()) + self.instance.exec( + ["dd", f"of=/etc/distribution/{mirror.name}.yaml"], + sensitive_kwargs=True, + input=str.encode(src.substitute(substitutes)), + ) + + with open(config.REGISTRY_DIR / "registry.service", "r") as registry_template: + src = Template(registry_template.read()) + self.instance.exec( + ["dd", f"of=/etc/systemd/system/registry-{mirror.name}.service"], + sensitive_kwargs=True, + input=str.encode(src.substitute(substitutes)), + ) + + self.instance.exec(["systemctl", "daemon-reload"]) + self.instance.exec(["systemctl", "enable", f"registry-{mirror.name}.service"]) + self.instance.exec(["systemctl", "start", f"registry-{mirror.name}.service"]) + + @property + def mirrors(self) -> List[Mirror]: + """ + Get the list of mirrors in the registry. + + Returns: + List[Mirror]: The list of mirrors. + """ + return self._mirrors + + @property + def ip(self) -> str: + """ + Get the IP address of the registry. + + Returns: + str: The IP address of the registry. + """ + return self._ip + + # Configure the specified instance to use this registry mirror. + def apply_configuration(self, instance): + for mirror in self.mirrors: + substitutes = { + "IP": self.ip, + "PORT": mirror.port, + } + + instance.exec(["mkdir", "-p", f"/etc/containerd/hosts.d/{mirror.name}"]) + + with open(config.REGISTRY_DIR / "hosts.toml", "r") as registry_template: + src = Template(registry_template.read()) + instance.exec( + [ + "dd", + f"of=/etc/containerd/hosts.d/{mirror.name}/hosts.toml", + ], + input=str.encode(src.substitute(substitutes)), + ) diff --git a/tests/integration/tests/test_util/snap.py b/tests/integration/tests/test_util/snap.py new file mode 100644 index 000000000..f64c64cb9 --- /dev/null +++ b/tests/integration/tests/test_util/snap.py @@ -0,0 +1,122 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import json +import logging +import re +import urllib.error +import urllib.request +from typing import List, Optional + +from test_util.util import major_minor + +LOG = logging.getLogger(__name__) + +SNAP_NAME = "k8s" + +# For Snap Store API request +SNAPSTORE_INFO_API = "https://api.snapcraft.io/v2/snaps/info/" +SNAPSTORE_HEADERS = { + "Snap-Device-Series": "16", + "User-Agent": "Mozilla/5.0", +} +RISK_LEVELS = ["stable", "candidate", "beta", "edge"] + + +def get_snap_info(snap_name=SNAP_NAME): + """Get the snap info from the Snap Store API.""" + req = urllib.request.Request( + SNAPSTORE_INFO_API + snap_name, headers=SNAPSTORE_HEADERS + ) + try: + with urllib.request.urlopen(req) as response: # nosec + return json.loads(response.read().decode()) + except urllib.error.HTTPError as e: + LOG.exception("HTTPError ({%s}): {%s} {%s}", req.full_url, e.code, e.reason) + raise + except urllib.error.URLError as e: + LOG.exception("URLError ({%s}): {%s}", req.full_url, e.reason) + raise + + +def filter_arch_and_flavor(channels: List[dict], arch: str, flavor: str) -> List[tuple]: + """Filter available channels by architecture and match them with a given regex pattern + for a flavor.""" + if flavor == "strict": + pattern = re.compile(r"(\d+)\.(\d+)\/(" + "|".join(RISK_LEVELS) + ")") + else: + pattern = re.compile( + r"(\d+)\.(\d+)-" + re.escape(flavor) + r"\/(" + "|".join(RISK_LEVELS) + ")" + ) + + matched_channels = [] + for ch in channels: + if ch["channel"]["architecture"] == arch: + channel_name = ch["channel"]["name"] + match = pattern.match(channel_name) + if match: + major, minor, risk = match.groups() + matched_channels.append((channel_name, int(major), int(minor), risk)) + + return matched_channels + + +def get_most_stable_channels( + num_of_channels: int, + flavor: str, + arch: str, + include_latest: bool = True, + min_release: Optional[str] = None, +) -> List[str]: + """Get an ascending list of latest channels based on the number of channels + flavour and architecture.""" + snap_info = get_snap_info() + + # Extract channel information and filter by architecture and flavor + arch_flavor_channels = filter_arch_and_flavor( + snap_info.get("channel-map", []), arch, flavor + ) + + # Dictionary to store the most stable channels for each version + channel_map = {} + for channel, major, minor, risk in arch_flavor_channels: + version_key = (int(major), int(minor)) + + if min_release is not None: + _min_release = major_minor(min_release) + if _min_release is not None and version_key < _min_release: + continue + + if version_key not in channel_map or RISK_LEVELS.index( + risk + ) < RISK_LEVELS.index(channel_map[version_key][1]): + channel_map[version_key] = (channel, risk) + + # Sort channels by major and minor version (ascending order) + sorted_versions = sorted(channel_map.keys(), key=lambda v: (v[0], v[1])) + + # Extract only the channel names + final_channels = [channel_map[v][0] for v in sorted_versions[:num_of_channels]] + + if include_latest: + final_channels.append(f"latest/edge/{flavor}") + + return final_channels + + +def get_channels( + num_of_channels: int, flavor: str, arch: str, risk_level: str, include_latest=True +) -> List[str]: + """Get channels based on the risk level, architecture and flavour.""" + snap_info = get_snap_info() + arch_flavor_channels = filter_arch_and_flavor( + snap_info.get("channel-map", []), arch, flavor + ) + + matching_channels = [ch[0] for ch in arch_flavor_channels if ch[3] == risk_level] + matching_channels = matching_channels[:num_of_channels] + if include_latest: + latest_channel = f"latest/edge/{flavor}" + matching_channels.append(latest_channel) + + return matching_channels diff --git a/tests/integration/tests/test_util/test_bootstrap.py b/tests/integration/tests/test_util/test_bootstrap.py new file mode 100644 index 000000000..1bcc688dd --- /dev/null +++ b/tests/integration/tests/test_util/test_bootstrap.py @@ -0,0 +1,16 @@ +# +# Copyright 2024 Canonical, Ltd. +# +from typing import List + +import pytest +from test_util import harness + + +@pytest.mark.node_count(1) +@pytest.mark.disable_k8s_bootstrapping() +def test_microk8s_installed(instances: List[harness.Instance]): + instance = instances[0] + instance.exec("snap install microk8s --classic".split()) + result = instance.exec("k8s bootstrap".split(), capture_output=True, check=False) + assert "Error: microk8s snap is installed" in result.stderr.decode() diff --git a/tests/integration/tests/test_util/util.py b/tests/integration/tests/test_util/util.py index 8d9875d18..1c3106178 100644 --- a/tests/integration/tests/test_util/util.py +++ b/tests/integration/tests/test_util/util.py @@ -1,14 +1,19 @@ # # Copyright 2024 Canonical, Ltd. # +import ipaddress import json import logging +import re import shlex import subprocess +import urllib.request +from datetime import datetime from functools import partial from pathlib import Path from typing import Any, Callable, List, Mapping, Optional, Union +import pytest from tenacity import ( RetryCallState, retry, @@ -20,13 +25,21 @@ from test_util import config, harness LOG = logging.getLogger(__name__) +RISKS = ["stable", "candidate", "beta", "edge"] +TRACK_RE = re.compile(r"^(\d+)\.(\d+)(\S*)$") def run(command: list, **kwargs) -> subprocess.CompletedProcess: """Log and run command.""" kwargs.setdefault("check", True) - LOG.debug("Execute command %s (kwargs=%s)", shlex.join(command), kwargs) + sensitive_command = kwargs.pop("sensitive_command", False) + sensitive_kwargs = kwargs.pop("sensitive_kwargs", sensitive_command) + + logged_command = shlex.join(command) if not sensitive_command else "" + logged_kwargs = kwargs if not sensitive_kwargs else "" + + LOG.debug("Execute command %s (kwargs=%s)", logged_command, logged_kwargs) return subprocess.run(command, **kwargs) @@ -126,21 +139,105 @@ def until( return Retriable() -# Installs and setups the k8s snap on the given instance and connects the interfaces. -def setup_k8s_snap(instance: harness.Instance, snap_path: Path): - LOG.info("Install k8s snap") - instance.send_file(config.SNAP, snap_path) - instance.exec(["snap", "install", snap_path, "--classic", "--dangerous"]) +def _as_int(value: Optional[str]) -> Optional[int]: + """Convert a string to an integer.""" + try: + return int(value) + except (TypeError, ValueError): + return None - LOG.info("Ensure k8s interfaces and network requirements") - instance.exec(["/snap/k8s/current/k8s/hack/init.sh"], stdout=subprocess.DEVNULL) + +def setup_k8s_snap( + instance: harness.Instance, + tmp_path: Path, + snap: Optional[str] = None, + connect_interfaces=True, +): + """Installs and sets up the snap on the given instance and connects the interfaces. + + Args: + instance: instance on which to install the snap + tmp_path: path to store the snap on the instance + snap: choice of track, channel, revision, or file path + a snap track to install + a snap channel to install + a snap revision to install + a path to the snap to install + """ + cmd = ["snap", "install", "--classic"] + which_snap = snap or config.SNAP + + if not which_snap: + pytest.fail("Set TEST_SNAP to the channel, revision, or path to the snap") + + if isinstance(which_snap, str) and which_snap.startswith("/"): + LOG.info("Install k8s snap by path") + snap_path = (tmp_path / "k8s.snap").as_posix() + instance.send_file(which_snap, snap_path) + cmd += ["--dangerous", snap_path] + elif snap_revision := _as_int(which_snap): + LOG.info("Install k8s snap by revision") + cmd += [config.SNAP_NAME, "--revision", snap_revision] + elif "/" in which_snap or which_snap in RISKS: + LOG.info("Install k8s snap by specific channel: %s", which_snap) + cmd += [config.SNAP_NAME, "--channel", which_snap] + elif channel := tracks_least_risk(which_snap, instance.arch): + LOG.info("Install k8s snap by least risky channel: %s", channel) + cmd += [config.SNAP_NAME, "--channel", channel] + + instance.exec(cmd) + if connect_interfaces: + LOG.info("Ensure k8s interfaces and network requirements") + instance.exec(["/snap/k8s/current/k8s/hack/init.sh"], stdout=subprocess.DEVNULL) + + +def remove_k8s_snap(instance: harness.Instance): + LOG.info("Uninstall k8s...") + stubbornly(retries=20, delay_s=5).on(instance).exec( + ["snap", "remove", config.SNAP_NAME, "--purge"] + ) + + LOG.info("Waiting for shims to go away...") + stubbornly(retries=20, delay_s=5).on(instance).until( + lambda p: all( + x not in p.stdout.decode() + for x in ["containerd-shim", "cilium", "coredns", "/pause"] + ) + ).exec(["ps", "-fea"]) + + LOG.info("Waiting for kubelet and containerd mounts to go away...") + stubbornly(retries=20, delay_s=5).on(instance).until( + lambda p: all( + x not in p.stdout.decode() + for x in ["/var/lib/kubelet/pods", "/run/containerd/io.containerd"] + ) + ).exec(["mount"]) + + # NOTE(neoaggelos): Temporarily disable this as it fails on strict. + # For details, `snap changes` then `snap change $remove_k8s_snap_change`. + # Example output follows: + # + # 2024-02-23T14:10:42Z ERROR ignoring failure in hook "remove": + # ----- + # ... + # ip netns delete cni-UUID1 + # Cannot remove namespace file "/run/netns/cni-UUID1": Device or resource busy + # ip netns delete cni-UUID2 + # Cannot remove namespace file "/run/netns/cni-UUID2": Device or resource busy + # ip netns delete cni-UUID3 + # Cannot remove namespace file "/run/netns/cni-UUID3": Device or resource busy + + # LOG.info("Waiting for CNI network namespaces to go away...") + # stubbornly(retries=5, delay_s=5).on(instance).until( + # lambda p: "cni-" not in p.stdout.decode() + # ).exec(["ip", "netns", "list"]) def wait_until_k8s_ready( control_node: harness.Instance, instances: List[harness.Instance], - retries: int = 30, - delay_s: int = 5, + retries: int = config.DEFAULT_WAIT_RETRIES, + delay_s: int = config.DEFAULT_WAIT_DELAY_S, node_names: Mapping[str, str] = {}, ): """ @@ -168,12 +265,12 @@ def wait_until_k8s_ready( def wait_for_dns(instance: harness.Instance): LOG.info("Waiting for DNS to be ready") - instance.exec(["k8s", "x-wait-for", "dns"]) + instance.exec(["k8s", "x-wait-for", "dns", "--timeout", "20m"]) def wait_for_network(instance: harness.Instance): LOG.info("Waiting for network to be ready") - instance.exec(["k8s", "x-wait-for", "network"]) + instance.exec(["k8s", "x-wait-for", "network", "--timeout", "20m"]) def hostname(instance: harness.Instance) -> str: @@ -261,3 +358,159 @@ def get_default_ip(instance: harness.Instance): ["ip", "-o", "-4", "route", "show", "to", "default"], capture_output=True ) return p.stdout.decode().split(" ")[8] + + +def get_global_unicast_ipv6(instance: harness.Instance, interface="eth0") -> str: + # --- + # 2: eth0: mtu 1500 qdisc fq_codel state UP group default qlen 1000 + # link/ether 00:16:3e:0f:4d:1e brd ff:ff:ff:ff:ff:ff + # inet + # inet6 fe80::216:3eff:fe0f:4d1e/64 scope link + # --- + # Fetching the global unicast address for the specified interface, e.g. fe80::216:3eff:fe0f:4d1e + result = instance.exec( + ["ip", "-6", "addr", "show", "dev", interface, "scope", "global"], + capture_output=True, + text=True, + ) + output = result.stdout + ipv6_regex = re.compile(r"inet6\s+([a-f0-9:]+)\/[0-9]*\s+scope global") + match = ipv6_regex.search(output) + if match: + return match.group(1) + return None + + +# Checks if a datastring is a valid RFC3339 date. +def is_valid_rfc3339(date_str): + try: + # Attempt to parse the string according to the RFC3339 format + datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%S%z") + return True + except ValueError: + return False + + +def tracks_least_risk(track: str, arch: str) -> str: + """Determine the snap channel with the least risk in the provided track. + + Args: + track: the track to determine the least risk channel for + arch: the architecture to narrow the revision + + Returns: + the channel associated with the least risk + """ + LOG.debug("Determining least risk channel for track: %s on %s", track, arch) + if track == "latest": + return f"latest/edge/{config.FLAVOR or 'classic'}" + + INFO_URL = f"https://api.snapcraft.io/v2/snaps/info/{config.SNAP_NAME}" + HEADERS = { + "Snap-Device-Series": "16", + "User-Agent": "Mozilla/5.0", + } + + req = urllib.request.Request(INFO_URL, headers=HEADERS) + with urllib.request.urlopen(req) as response: + snap_info = json.loads(response.read().decode()) + + risks = [ + channel["channel"]["risk"] + for channel in snap_info["channel-map"] + if channel["channel"]["track"] == track + and channel["channel"]["architecture"] == arch + ] + if not risks: + raise ValueError(f"No risks found for track: {track}") + risk_level = {"stable": 0, "candidate": 1, "beta": 2, "edge": 3} + channel = f"{track}/{min(risks, key=lambda r: risk_level[r])}" + LOG.info("Least risk channel from track %s is %s", track, channel) + return channel + + +def major_minor(version: str) -> Optional[tuple]: + """Determine the major and minor version of a Kubernetes version string. + + Args: + version: the version string to determine the major and minor version for + + Returns: + a tuple containing the major and minor version or None if the version string is invalid + """ + if match := TRACK_RE.match(version): + maj, min, _ = match.groups() + return int(maj), int(min) + return None + + +def previous_track(snap_version: str) -> str: + """Determine the snap track preceding the provided version. + + Args: + snap_version: the snap version to determine the previous track for + + Returns: + the previous track + """ + LOG.debug("Determining previous track for %s", snap_version) + + if not snap_version: + assumed = "latest" + LOG.info( + "Cannot determine previous track for undefined snap -- assume %s", + snap_version, + assumed, + ) + return assumed + + if snap_version.startswith("/") or _as_int(snap_version) is not None: + assumed = "latest" + LOG.info( + "Cannot determine previous track for %s -- assume %s", snap_version, assumed + ) + return assumed + + if maj_min := major_minor(snap_version): + maj, min = maj_min + if min == 0: + with urllib.request.urlopen( + f"https://dl.k8s.io/release/stable-{maj - 1}.txt" + ) as r: + stable = r.read().decode().strip() + maj_min = major_minor(stable) + else: + maj_min = (maj, min - 1) + elif snap_version.startswith("latest") or "/" not in snap_version: + with urllib.request.urlopen("https://dl.k8s.io/release/stable.txt") as r: + stable = r.read().decode().strip() + maj_min = major_minor(stable) + + flavor_track = {"": "classic", "strict": ""}.get(config.FLAVOR, config.FLAVOR) + track = f"{maj_min[0]}.{maj_min[1]}" + (flavor_track and f"-{flavor_track}") + LOG.info("Previous track for %s is from track: %s", snap_version, track) + return track + + +def find_suitable_cidr(parent_cidr: str, excluded_ips: List[str]): + """Find a suitable CIDR for LoadBalancer services""" + net = ipaddress.IPv4Network(parent_cidr, False) + + # Starting from the first IP address from the parent cidr, + # we search for a /30 cidr block(4 total ips, 2 available) + # that doesn't contain the excluded ips to avoid collisions + # /30 because this is the smallest CIDR cilium hands out IPs from + for i in range(4, 255, 4): + lb_net = ipaddress.IPv4Network(f"{str(net[0]+i)}/30", False) + + contains_excluded = False + for excluded in excluded_ips: + if ipaddress.ip_address(excluded) in lb_net: + contains_excluded = True + break + + if contains_excluded: + continue + + return str(lb_net) + raise RuntimeError("Could not find a suitable CIDR for LoadBalancer services") diff --git a/tests/integration/tests/test_version_upgrades.py b/tests/integration/tests/test_version_upgrades.py new file mode 100644 index 000000000..d8798c874 --- /dev/null +++ b/tests/integration/tests/test_version_upgrades.py @@ -0,0 +1,65 @@ +# +# Copyright 2024 Canonical, Ltd. +# +import logging +from typing import List + +import pytest +from test_util import config, harness, snap, util + +LOG = logging.getLogger(__name__) + + +@pytest.mark.node_count(1) +@pytest.mark.no_setup() +@pytest.mark.skipif( + not config.VERSION_UPGRADE_CHANNELS, reason="No upgrade channels configured" +) +def test_version_upgrades(instances: List[harness.Instance], tmp_path): + channels = config.VERSION_UPGRADE_CHANNELS + cp = instances[0] + current_channel = channels[0] + + if current_channel.lower() == "recent": + if len(channels) != 3: + pytest.fail( + "'recent' requires the number of releases as second argument and the flavour as third argument" + ) + _, num_channels, flavour = channels + channels = snap.get_most_stable_channels( + int(num_channels), + flavour, + cp.arch, + min_release=config.VERSION_UPGRADE_MIN_RELEASE, + ) + if len(channels) < 2: + pytest.fail( + f"Need at least 2 channels to upgrade, got {len(channels)} for flavour {flavour}" + ) + current_channel = channels[0] + + LOG.info( + f"Bootstrap node on {current_channel} and upgrade through channels: {channels[1:]}" + ) + + # Setup the k8s snap from the bootstrap channel and setup basic configuration. + util.setup_k8s_snap(cp, tmp_path, current_channel) + cp.exec(["k8s", "bootstrap"]) + + util.wait_until_k8s_ready(cp, instances) + LOG.info(f"Installed {cp.id} on channel {current_channel}") + + for channel in channels[1:]: + LOG.info(f"Upgrading {cp.id} from {current_channel} to channel {channel}") + + # Log the current snap version on the node. + out = cp.exec(["snap", "list", config.SNAP_NAME], capture_output=True) + LOG.info(f"Current snap version: {out.stdout.decode().strip()}") + + # note: the `--classic` flag will be ignored by snapd for strict snaps. + cp.exec( + ["snap", "refresh", config.SNAP_NAME, "--channel", channel, "--classic"] + ) + util.wait_until_k8s_ready(cp, instances) + current_channel = channel + LOG.info(f"Upgraded {cp.id} on channel {channel}") diff --git a/tests/integration/tox.ini b/tests/integration/tox.ini index e2d7296b2..bdd82d029 100644 --- a/tests/integration/tox.ini +++ b/tests/integration/tox.ini @@ -1,52 +1,53 @@ [tox] -no_package = True +skipsdist = True skip_missing_interpreters = True env_list = format, lint, integration -min_version = 4.0.0 [testenv] set_env = PYTHONBREAKPOINT=pdb.set_trace PY_COLORS=1 -pass_env = +passenv = PYTHONPATH [testenv:format] description = Apply coding style standards to code -deps = -r {tox_root}/requirements-dev.txt +deps = -r {toxinidir}/requirements-dev.txt commands = - licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests - isort {tox_root}/tests --profile=black - black {tox_root}/tests + licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests + isort {toxinidir}/tests --profile=black + black {toxinidir}/tests [testenv:lint] description = Check code against coding style standards -deps = -r {tox_root}/requirements-dev.txt +deps = -r {toxinidir}/requirements-dev.txt commands = - codespell {tox_root}/tests - flake8 {tox_root}/tests - licenseheaders -t {tox_root}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {tox_root}/tests --dry - isort {tox_root}/tests --profile=black --check - black {tox_root}/tests --check --diff + codespell {toxinidir}/tests + flake8 {toxinidir}/tests + licenseheaders -t {toxinidir}/.copyright.tmpl -cy -o 'Canonical, Ltd' -d {toxinidir}/tests --dry + isort {toxinidir}/tests --profile=black --check + black {toxinidir}/tests --check --diff [testenv:integration] description = Run integration tests deps = - -r {tox_root}/requirements-test.txt + -r {toxinidir}/requirements-test.txt commands = - pytest -v \ + pytest -vv \ --maxfail 1 \ --tb native \ --log-cli-level DEBUG \ + --log-format "%(asctime)s %(levelname)s %(message)s" \ + --log-date-format "%Y-%m-%d %H:%M:%S" \ --disable-warnings \ {posargs} \ - {tox_root}/tests -pass_env = + {toxinidir}/tests +passenv = TEST_* [flake8] max-line-length = 120 select = E,W,F,C,N -ignore = W503 +ignore = W503,E231,E226 exclude = venv,.git,.tox,.tox_env,.venv,build,dist,*.egg_info show-source = true