Files
cozystack/.github/workflows/e2e-proxmox.yaml
T
Aleksei Sviridkin 048c029e29 fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it (#4971)
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin
kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process
umask and never removed it, so it outlived the run on the self-hosted
runner, world-readable under the usual umask. `dump_diagnostics` then
read whatever sat at that path, so a run that failed before
`fetch_tenant_kubeconfig` reported the previous run's tenant as its own.

- The file now lives under `RUNNER_TEMP`, which the runner exports into
every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it
for a local run, and `TMPDIR` is the last fallback. The workflow sets no
path of its own: the `runner` context is not available in job-level
`env`.
- `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell,
since the workflow's `umask 077` covers only the staging step.
- `apply-tenant`, the suite's first step, removes a leftover before
doing anything else, so diagnostics in this run can only see this run's
file.
- The suite's `finally` removes it, before the `keep_tenant` exit, and
the workflow's "Remove staged credentials" step removes it again for a
run killed before `finally`.

Verification: `sh -n` and shellcheck on the script; actionlint on
`e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported,
as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases
with a stub `kubectl`: the fetched file has no group or other bits under
`umask 022`, `apply-tenant` removes a planted leftover, and the suite's
`finally` (run out of the Chainsaw file on the `keep_tenant` path)
removes the file, so `diagnostics` after it prints no tenant section.
Each goes red with its line (`umask 077`, the leftover `rm`, the
`finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9,
teardown contract 8, step1b 4, talos template 12), also under
`hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have
cannot dispatch the workflow at the moment, so the workflow steps
themselves are unexercised beyond actionlint.

Closes #4806


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Proxmox end-to-end test runs store tenant credentials in a temporary
work directory and create kubeconfig files with restricted permissions.
* Stale tenant kubeconfig files are cleared before a new tenant fixture
is applied. Cleanup removes the file even when tenant data is retained
or a run is interrupted, preventing credentials from appearing in later
diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:31:31 +03:00

168 lines
7.9 KiB
YAML

name: E2E (proxmox)
# The Proxmox substrate's own lane, separate from every other e2e workflow for
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
# the oracle-vm-* runners the other lanes use will never have one.
#
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
# whatever the workflow checks out, and this one holds credentials to a
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
# hand both to its author. Manual dispatch and a schedule are the whole trigger
# surface, and both come from someone who already has write access.
on:
workflow_dispatch:
inputs:
keep_tenant:
description: "Leave the tenant in place after the run, for debugging"
type: boolean
default: false
schedule:
# Nightly, offset from the upstream lanes so a shared hypervisor is not
# asked to do two things at once.
- cron: "40 2 * * *"
permissions:
contents: read
concurrency:
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
# over all three. Never cancel in progress: a killed run leaks VMs.
group: e2e-proxmox
cancel-in-progress: false
jobs:
e2e:
name: Proxmox substrate
# The job needs a self-hosted runner carrying the `proxmox` label. On a
# repository without one a scheduled run would sit in the queue until GitHub
# cancels it, so the schedule fires only where the stand is declared;
# manual dispatch always runs.
if: github.event_name == 'workflow_dispatch' || vars.PROXMOX_NIGHTLY == 'true'
runs-on: [self-hosted, proxmox]
timeout-minutes: 90
env:
COZY_PROXMOX_NS: tenant-e2eproxmox
COZY_PROXMOX_CLUSTER: k8s-e2e
# No fallbacks: an unset variable fails the first step below instead of
# sending a key to a guessed address or probing a storage the stand lacks.
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE }}
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
steps:
- name: Checkout
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
with:
persist-credentials: false
- name: Require the stand's variables
env:
PVE_URL: ${{ vars.PROXMOX_URL }}
PVE_REGION: ${{ vars.PROXMOX_REGION }}
run: |
set -eu
for v in COZY_PROXMOX_STORAGE COZY_PVE_SSH PVE_URL PVE_REGION; do
eval "val=\${$v:-}"
[ -n "$val" ] || { echo "$v is unset: set the PROXMOX_* repository variables for this stand" >&2; exit 1; }
done
- name: Stage credentials
id: stage
env:
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
PVE_URL: ${{ vars.PROXMOX_URL }}
PVE_REGION: ${{ vars.PROXMOX_REGION }}
run: |
set -eu
umask 077
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
# Unless the current context resolves to a server, client-go either
# fails or falls back to its defaults, in-cluster credentials among
# them, and the calls below would reach a cluster other than the
# stand's. Tear down repeats this check.
server=$(kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" config view --minify \
-o jsonpath='{.clusters[0].cluster.server}' 2>"$RUNNER_TEMP/kubeconfig-check.err" || true)
if [ -z "$server" ]; then
echo "COZY_KUBECONFIG does not resolve to a server" >&2
# Empty when the context resolves to a cluster with no server.
cat "$RUNNER_TEMP/kubeconfig-check.err" >&2
exit 1
fi
# OpenSSH refuses a private key whose last line has no newline ("error
# in libcrypto"), and `gh secret set` strips it from a key read from
# stdin. A second newline, when the secret kept its own, is harmless.
printf '%s\n' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
# The credentials the management-side CSI controller and CCM read. A
# Secret rather than values so no token is ever written into a
# rendered HelmRelease.
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
create secret generic proxmox-credentials \
--from-literal=url="$PVE_URL" \
--from-literal=token_id="$PVE_TOKEN_ID" \
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
--from-literal=region="$PVE_REGION" \
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
- name: Enable the suite
# It ships disabled so the normal lanes cannot select it; this is the
# one place that is allowed to turn it on.
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
- name: Run the Proxmox suite
env:
KUBECONFIG: ${{ github.workspace }}/.kubeconfig
run: |
chainsaw test \
--config hack/e2e-chainsaw/.chainsaw.yaml \
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
- name: Tear down
# always(), because the whole point of the teardown is the failing run:
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
# them.
if: always() && !inputs.keep_tenant
env:
STAGE_OUTCOME: ${{ steps.stage.outcome }}
run: |
set -eu
kc="$GITHUB_WORKSPACE/.kubeconfig"
# The suite runs only after Stage succeeds, so without that this run
# created no tenant.
if [ "$STAGE_OUTCOME" != success ]; then
echo "::notice::Tear down skipped: Stage credentials ended $STAGE_OUTCOME, so this run created no tenant"
exit 0
fi
# The check Stage makes. Every call names the file, so none falls back
# to this self-hosted runner's own kubeconfig. Failing it here means
# the file broke after Stage, and the tenant may still be live.
server=$(kubectl --kubeconfig "$kc" config view --minify \
-o jsonpath='{.clusters[0].cluster.server}' 2>"$RUNNER_TEMP/kubeconfig-check.err" || true)
if [ -z "$server" ]; then
echo "::error::Tear down did not run: $kc no longer resolves to a server, so tenant e2eproxmox and its VMs may still be live"
cat "$RUNNER_TEMP/kubeconfig-check.err"
exit 1
fi
# A failed delete must not stop the ones after it: stopping there leaves
# behind exactly the VMs this step exists to remove. The step still
# fails at the end.
rc=0
kubectl --kubeconfig "$kc" delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found || rc=1
kubectl --kubeconfig "$kc" delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found || rc=1
kubectl --kubeconfig "$kc" delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found || rc=1
exit "$rc"
- name: Remove staged credentials
if: always()
# The tenant kubeconfig is the suite's, fetched by run-proxmox.sh into
# RUNNER_TEMP, which the runner exports into every step (the runner
# context is not allowed in job-level env); its finally removes it too,
# but a run killed mid-step never reaches that.
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY" "$RUNNER_TEMP/proxmox-e2e-tenant.kubeconfig"