mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. `dump_diagnostics` then read whatever sat at that path, so a run that failed before `fetch_tenant_kubeconfig` reported the previous run's tenant as its own. - The file now lives under `RUNNER_TEMP`, which the runner exports into every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it for a local run, and `TMPDIR` is the last fallback. The workflow sets no path of its own: the `runner` context is not available in job-level `env`. - `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell, since the workflow's `umask 077` covers only the staging step. - `apply-tenant`, the suite's first step, removes a leftover before doing anything else, so diagnostics in this run can only see this run's file. - The suite's `finally` removes it, before the `keep_tenant` exit, and the workflow's "Remove staged credentials" step removes it again for a run killed before `finally`. Verification: `sh -n` and shellcheck on the script; actionlint on `e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported, as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases with a stub `kubectl`: the fetched file has no group or other bits under `umask 022`, `apply-tenant` removes a planted leftover, and the suite's `finally` (run out of the Chainsaw file on the `keep_tenant` path) removes the file, so `diagnostics` after it prints no tenant section. Each goes red with its line (`umask 077`, the leftover `rm`, the `finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9, teardown contract 8, step1b 4, talos template 12), also under `hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have cannot dispatch the workflow at the moment, so the workflow steps themselves are unexercised beyond actionlint. Closes #4806 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Proxmox end-to-end test runs store tenant credentials in a temporary work directory and create kubeconfig files with restricted permissions. * Stale tenant kubeconfig files are cleared before a new tenant fixture is applied. Cleanup removes the file even when tenant data is retained or a run is interrupted, preventing credentials from appearing in later diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai -->
168 lines
7.9 KiB
YAML
168 lines
7.9 KiB
YAML
name: E2E (proxmox)
|
|
|
|
# The Proxmox substrate's own lane, separate from every other e2e workflow for
|
|
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
|
|
# the oracle-vm-* runners the other lanes use will never have one.
|
|
#
|
|
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
|
|
# whatever the workflow checks out, and this one holds credentials to a
|
|
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
|
|
# hand both to its author. Manual dispatch and a schedule are the whole trigger
|
|
# surface, and both come from someone who already has write access.
|
|
on:
|
|
workflow_dispatch:
|
|
inputs:
|
|
keep_tenant:
|
|
description: "Leave the tenant in place after the run, for debugging"
|
|
type: boolean
|
|
default: false
|
|
schedule:
|
|
# Nightly, offset from the upstream lanes so a shared hypervisor is not
|
|
# asked to do two things at once.
|
|
- cron: "40 2 * * *"
|
|
|
|
permissions:
|
|
contents: read
|
|
|
|
concurrency:
|
|
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
|
|
# over all three. Never cancel in progress: a killed run leaks VMs.
|
|
group: e2e-proxmox
|
|
cancel-in-progress: false
|
|
|
|
jobs:
|
|
e2e:
|
|
name: Proxmox substrate
|
|
# The job needs a self-hosted runner carrying the `proxmox` label. On a
|
|
# repository without one a scheduled run would sit in the queue until GitHub
|
|
# cancels it, so the schedule fires only where the stand is declared;
|
|
# manual dispatch always runs.
|
|
if: github.event_name == 'workflow_dispatch' || vars.PROXMOX_NIGHTLY == 'true'
|
|
runs-on: [self-hosted, proxmox]
|
|
timeout-minutes: 90
|
|
env:
|
|
COZY_PROXMOX_NS: tenant-e2eproxmox
|
|
COZY_PROXMOX_CLUSTER: k8s-e2e
|
|
# No fallbacks: an unset variable fails the first step below instead of
|
|
# sending a key to a guessed address or probing a storage the stand lacks.
|
|
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE }}
|
|
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
|
|
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
|
|
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
|
|
steps:
|
|
- name: Checkout
|
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
|
with:
|
|
persist-credentials: false
|
|
|
|
- name: Require the stand's variables
|
|
env:
|
|
PVE_URL: ${{ vars.PROXMOX_URL }}
|
|
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
|
run: |
|
|
set -eu
|
|
for v in COZY_PROXMOX_STORAGE COZY_PVE_SSH PVE_URL PVE_REGION; do
|
|
eval "val=\${$v:-}"
|
|
[ -n "$val" ] || { echo "$v is unset: set the PROXMOX_* repository variables for this stand" >&2; exit 1; }
|
|
done
|
|
|
|
- name: Stage credentials
|
|
id: stage
|
|
env:
|
|
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
|
|
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
|
|
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
|
|
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
|
|
PVE_URL: ${{ vars.PROXMOX_URL }}
|
|
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
|
run: |
|
|
set -eu
|
|
umask 077
|
|
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
|
|
# Unless the current context resolves to a server, client-go either
|
|
# fails or falls back to its defaults, in-cluster credentials among
|
|
# them, and the calls below would reach a cluster other than the
|
|
# stand's. Tear down repeats this check.
|
|
server=$(kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" config view --minify \
|
|
-o jsonpath='{.clusters[0].cluster.server}' 2>"$RUNNER_TEMP/kubeconfig-check.err" || true)
|
|
if [ -z "$server" ]; then
|
|
echo "COZY_KUBECONFIG does not resolve to a server" >&2
|
|
# Empty when the context resolves to a cluster with no server.
|
|
cat "$RUNNER_TEMP/kubeconfig-check.err" >&2
|
|
exit 1
|
|
fi
|
|
# OpenSSH refuses a private key whose last line has no newline ("error
|
|
# in libcrypto"), and `gh secret set` strips it from a key read from
|
|
# stdin. A second newline, when the secret kept its own, is harmless.
|
|
printf '%s\n' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
|
|
|
|
# The credentials the management-side CSI controller and CCM read. A
|
|
# Secret rather than values so no token is ever written into a
|
|
# rendered HelmRelease.
|
|
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
|
|
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
|
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
|
|
create secret generic proxmox-credentials \
|
|
--from-literal=url="$PVE_URL" \
|
|
--from-literal=token_id="$PVE_TOKEN_ID" \
|
|
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
|
|
--from-literal=region="$PVE_REGION" \
|
|
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
|
|
|
- name: Enable the suite
|
|
# It ships disabled so the normal lanes cannot select it; this is the
|
|
# one place that is allowed to turn it on.
|
|
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
|
|
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
|
|
|
|
- name: Run the Proxmox suite
|
|
env:
|
|
KUBECONFIG: ${{ github.workspace }}/.kubeconfig
|
|
run: |
|
|
chainsaw test \
|
|
--config hack/e2e-chainsaw/.chainsaw.yaml \
|
|
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
|
|
|
|
- name: Tear down
|
|
# always(), because the whole point of the teardown is the failing run:
|
|
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
|
|
# them.
|
|
if: always() && !inputs.keep_tenant
|
|
env:
|
|
STAGE_OUTCOME: ${{ steps.stage.outcome }}
|
|
run: |
|
|
set -eu
|
|
kc="$GITHUB_WORKSPACE/.kubeconfig"
|
|
# The suite runs only after Stage succeeds, so without that this run
|
|
# created no tenant.
|
|
if [ "$STAGE_OUTCOME" != success ]; then
|
|
echo "::notice::Tear down skipped: Stage credentials ended $STAGE_OUTCOME, so this run created no tenant"
|
|
exit 0
|
|
fi
|
|
# The check Stage makes. Every call names the file, so none falls back
|
|
# to this self-hosted runner's own kubeconfig. Failing it here means
|
|
# the file broke after Stage, and the tenant may still be live.
|
|
server=$(kubectl --kubeconfig "$kc" config view --minify \
|
|
-o jsonpath='{.clusters[0].cluster.server}' 2>"$RUNNER_TEMP/kubeconfig-check.err" || true)
|
|
if [ -z "$server" ]; then
|
|
echo "::error::Tear down did not run: $kc no longer resolves to a server, so tenant e2eproxmox and its VMs may still be live"
|
|
cat "$RUNNER_TEMP/kubeconfig-check.err"
|
|
exit 1
|
|
fi
|
|
# A failed delete must not stop the ones after it: stopping there leaves
|
|
# behind exactly the VMs this step exists to remove. The step still
|
|
# fails at the end.
|
|
rc=0
|
|
kubectl --kubeconfig "$kc" delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found || rc=1
|
|
kubectl --kubeconfig "$kc" delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found || rc=1
|
|
kubectl --kubeconfig "$kc" delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found || rc=1
|
|
exit "$rc"
|
|
|
|
- name: Remove staged credentials
|
|
if: always()
|
|
# The tenant kubeconfig is the suite's, fetched by run-proxmox.sh into
|
|
# RUNNER_TEMP, which the runner exports into every step (the runner
|
|
# context is not allowed in job-level env); its finally removes it too,
|
|
# but a run killed mid-step never reaches that.
|
|
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY" "$RUNNER_TEMP/proxmox-e2e-tenant.kubeconfig"
|