mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-02 18:16:53 +09:00
test(kubernetes): add the Proxmox e2e suite, parked disabled
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.
Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.
Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.
The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.
The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
This commit is contained in:
Marian Koreniuk committed
1 parent
36b04218d7
commit
7500a9fbae
7 files changed
+666
-1
No files matched your search
@@ -0,0 +1,122 @@
|
||||
name: E2E (proxmox)
|
||||
|
||||
# The Proxmox substrate's own lane, separate from every other e2e workflow for
|
||||
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
|
||||
# the oracle-vm-* runners the other lanes use will never have one.
|
||||
#
|
||||
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
|
||||
# whatever the workflow checks out, and this one holds credentials to a
|
||||
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
|
||||
# hand both to its author. Manual dispatch and a schedule are the whole trigger
|
||||
# surface, and both come from someone who already has write access.
|
||||
on:
|
||||
workflow_dispatch:
|
||||
inputs:
|
||||
keep_tenant:
|
||||
description: "Leave the tenant in place after the run, for debugging"
|
||||
type: boolean
|
||||
default: false
|
||||
schedule:
|
||||
# Nightly, offset from the upstream lanes so a shared hypervisor is not
|
||||
# asked to do two things at once.
|
||||
- cron: "40 2 * * *"
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
concurrency:
|
||||
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
|
||||
# over all three. Never cancel in progress: a killed run leaks VMs.
|
||||
group: e2e-proxmox
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
e2e:
|
||||
name: Proxmox substrate
|
||||
# The job needs a self-hosted runner carrying the `proxmox` label. On a
|
||||
# repository without one a scheduled run would sit in the queue until GitHub
|
||||
# cancels it, so the schedule fires only where the stand is declared;
|
||||
# manual dispatch always runs.
|
||||
if: github.event_name == 'workflow_dispatch' || vars.PROXMOX_NIGHTLY == 'true'
|
||||
runs-on: [self-hosted, proxmox]
|
||||
timeout-minutes: 90
|
||||
env:
|
||||
COZY_PROXMOX_NS: tenant-e2eproxmox
|
||||
COZY_PROXMOX_CLUSTER: k8s-e2e
|
||||
# No fallbacks: an unset variable fails the first step below instead of
|
||||
# sending a key to a guessed address or probing a storage the stand lacks.
|
||||
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE }}
|
||||
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
|
||||
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
|
||||
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
|
||||
steps:
|
||||
- name: Checkout
|
||||
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||
with:
|
||||
persist-credentials: false
|
||||
|
||||
- name: Require the stand's variables
|
||||
env:
|
||||
PVE_URL: ${{ vars.PROXMOX_URL }}
|
||||
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
||||
run: |
|
||||
set -eu
|
||||
for v in COZY_PROXMOX_STORAGE COZY_PVE_SSH PVE_URL PVE_REGION; do
|
||||
eval "val=\${$v:-}"
|
||||
[ -n "$val" ] || { echo "$v is unset: set the PROXMOX_* repository variables for this stand" >&2; exit 1; }
|
||||
done
|
||||
|
||||
- name: Stage credentials
|
||||
env:
|
||||
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
|
||||
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
|
||||
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
|
||||
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
|
||||
PVE_URL: ${{ vars.PROXMOX_URL }}
|
||||
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
||||
run: |
|
||||
set -eu
|
||||
umask 077
|
||||
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
|
||||
printf '%s' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
|
||||
echo "KUBECONFIG=$GITHUB_WORKSPACE/.kubeconfig" >> "$GITHUB_ENV"
|
||||
|
||||
# The credentials the management-side CSI controller and CCM read. A
|
||||
# Secret rather than values so no token is ever written into a
|
||||
# rendered HelmRelease.
|
||||
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
|
||||
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
||||
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
|
||||
create secret generic proxmox-credentials \
|
||||
--from-literal=url="$PVE_URL" \
|
||||
--from-literal=token_id="$PVE_TOKEN_ID" \
|
||||
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
|
||||
--from-literal=region="$PVE_REGION" \
|
||||
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
||||
|
||||
- name: Enable the suite
|
||||
# It ships disabled so the normal lanes cannot select it; this is the
|
||||
# one place that is allowed to turn it on.
|
||||
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
|
||||
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
|
||||
|
||||
- name: Run the Proxmox suite
|
||||
run: |
|
||||
chainsaw test \
|
||||
--config hack/e2e-chainsaw/.chainsaw.yaml \
|
||||
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
|
||||
|
||||
- name: Tear down
|
||||
# always(), because the whole point of the teardown is the failing run:
|
||||
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
|
||||
# them.
|
||||
if: always() && !inputs.keep_tenant
|
||||
run: |
|
||||
set -eu
|
||||
kubectl delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
|
||||
kubectl delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
|
||||
kubectl delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found
|
||||
|
||||
- name: Remove staged credentials
|
||||
if: always()
|
||||
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"
|
||||
@@ -10,6 +10,16 @@ The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per a
|
||||
|
||||
Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.
|
||||
|
||||
## The Proxmox lane
|
||||
|
||||
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: suite discovery is `find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml`, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
|
||||
|
||||
Two properties of that workflow are security boundaries rather than preferences. It is never triggered by `pull_request`, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.
|
||||
|
||||
It does not install Cozystack. Every other lane builds a sandbox per run; this one asserts against an existing cluster and an existing hypervisor, which is a weaker guarantee and is stated in the suite header rather than left to be discovered.
|
||||
|
||||
The imperative half lives in `_lib/run-proxmox.sh` and is deliberately not a port of `_lib/run-kubernetes.sh`: that script's diagnostics are virt-launcher Pods, VirtualMachineInstances and guest consoles, none of which exist on an external hypervisor.
|
||||
|
||||
## Conventions
|
||||
|
||||
### 1. No retries on deterministic steps; retry only pure infrastructure
|
||||
|
||||
Executable
+238
@@ -0,0 +1,238 @@
|
||||
#!/bin/sh
|
||||
# Imperative half of the kubernetes-proxmox suite.
|
||||
#
|
||||
# Chainsaw's declarative asserts cover everything that is a Kubernetes object in
|
||||
# THIS cluster, and the suite uses them for exactly that. What is left needs a
|
||||
# shell: the hypervisor is reached over SSH and answers in `qm` output, and the
|
||||
# tenant cluster is reached with a kubeconfig that only exists once its control
|
||||
# plane is up. Those are the two things below.
|
||||
#
|
||||
# Deliberately not a port of _lib/run-kubernetes.sh. That script's diagnostic
|
||||
# half is virt-launcher Pods, VirtualMachineInstances and guest consoles —
|
||||
# objects with no counterpart on an external hypervisor — so sharing it would
|
||||
# mean carrying a thousand lines that can never fire here.
|
||||
#
|
||||
# Every subcommand exits non-zero on failure and says which check failed. There
|
||||
# is no `|| echo` anywhere in this file on purpose: the pattern turns a failed
|
||||
# command into a green run, and this repository has been bitten by it.
|
||||
set -eu
|
||||
|
||||
NS="${COZY_PROXMOX_NS:-tenant-e2eproxmox}"
|
||||
CLUSTER="${COZY_PROXMOX_CLUSTER:-k8s-e2e}"
|
||||
PVE_SSH="${COZY_PVE_SSH:?COZY_PVE_SSH is required: the ssh destination of the Proxmox host, user@host}"
|
||||
PVE_SSH_KEY="${COZY_PVE_SSH_KEY:-$HOME/.ssh/id_ed25519}"
|
||||
STORAGE="${COZY_PROXMOX_STORAGE:?COZY_PROXMOX_STORAGE is required, the PVE storage id for the PVC probe}"
|
||||
TENANT_KUBECONFIG="${TMPDIR:-/tmp}/proxmox-e2e-tenant.kubeconfig"
|
||||
FIXTURE="$(cd "$(dirname "$0")/.." && pwd)/kubernetes-proxmox/tenant.yaml"
|
||||
|
||||
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
|
||||
die() { printf '[%s] FAILED: %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; exit 1; }
|
||||
|
||||
pve() { ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i "$PVE_SSH_KEY" "$PVE_SSH" "$@"; }
|
||||
|
||||
# The Machine carries the VM id in the ProxmoxMachine it references. Reading it
|
||||
# from the cluster rather than guessing keeps this working when a replacement
|
||||
# lands on a different id.
|
||||
vmid_of_machine() {
|
||||
kubectl -n "$NS" get proxmoxmachine -o jsonpath='{.items[0].spec.virtualMachineID}'
|
||||
}
|
||||
|
||||
# --- checks -----------------------------------------------------------------
|
||||
|
||||
# A full clone is a copy of the template's disk; a linked clone references it and
|
||||
# stores only the increment. Nothing fails when this is wrong — the pool comes
|
||||
# up either way — so the only place it can be caught is here, by the shape of the
|
||||
# volume name: a linked clone points at `base-<template>-disk-N/`.
|
||||
check_linked_clone() {
|
||||
vmid=$(vmid_of_machine)
|
||||
[ -n "$vmid" ] || die "no ProxmoxMachine with a virtualMachineID in $NS"
|
||||
log "VM $vmid: checking the boot disk is a linked clone"
|
||||
|
||||
bootdisk=$(pve "qm config $vmid" | sed -n 's/^scsi0: *//p')
|
||||
[ -n "$bootdisk" ] || die "VM $vmid has no scsi0"
|
||||
|
||||
case "$bootdisk" in
|
||||
*:base-*-disk-*/*) log "linked clone confirmed: $bootdisk" ;;
|
||||
*) die "VM $vmid boot disk is a full copy, not a linked clone: $bootdisk" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
# The fixture is stand-agnostic through ${COZY_PVE_*} variables. GNU envsubst
|
||||
# expands ${VAR} and nothing else — a ${VAR:-default} stays in the text as it
|
||||
# is, set or not — so the defaults live here, the variable list is explicit so
|
||||
# nothing else in the file is touched, and a fixture with a variable left in it
|
||||
# is refused rather than applied.
|
||||
render_tenant() {
|
||||
: "${COZY_PVE_ADDRESSES:=10.0.0.180-10.0.0.185}"
|
||||
: "${COZY_PVE_GATEWAY:=10.0.0.1}"
|
||||
: "${COZY_PVE_PREFIX:=24}"
|
||||
: "${COZY_PVE_BRIDGE:=vmbr50}"
|
||||
: "${COZY_PVE_POOL:=}"
|
||||
export COZY_PVE_ADDRESSES COZY_PVE_GATEWAY COZY_PVE_PREFIX COZY_PVE_BRIDGE COZY_PVE_POOL COZY_PROXMOX_STORAGE
|
||||
rendered=$(envsubst '$COZY_PVE_ADDRESSES $COZY_PVE_GATEWAY $COZY_PVE_PREFIX $COZY_PVE_BRIDGE $COZY_PVE_POOL $COZY_PROXMOX_STORAGE' < "$FIXTURE")
|
||||
case "$rendered" in
|
||||
*'${'*) die "tenant.yaml still carries an unexpanded variable after envsubst" ;;
|
||||
esac
|
||||
printf '%s\n' "$rendered"
|
||||
}
|
||||
|
||||
apply_tenant() {
|
||||
render_tenant | kubectl apply -f -
|
||||
}
|
||||
|
||||
# The tenant kubeconfig exists only after Kamaji has issued it. Fetching it is
|
||||
# also the first proof that the control plane came up at all. The .conf key
|
||||
# carries the address a runner outside the management cluster can reach; .svc
|
||||
# is the in-cluster Service the management-side controllers use.
|
||||
fetch_tenant_kubeconfig() {
|
||||
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
|
||||
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
|
||||
[ -s "$TENANT_KUBECONFIG" ] || die "tenant kubeconfig is empty — did the control plane come up?"
|
||||
}
|
||||
|
||||
# providerID and the topology labels are applied by the cloud-controller-manager,
|
||||
# and the CSI node plugin refuses to start without the labels. Asserting them
|
||||
# here is what separates "the node joined" from "the node is usable".
|
||||
check_tenant_node() {
|
||||
fetch_tenant_kubeconfig
|
||||
log "checking the tenant node is Ready and initialised by the CCM"
|
||||
|
||||
node=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
|
||||
[ -n "$node" ] || die "tenant cluster has no nodes"
|
||||
|
||||
provider=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" -o jsonpath='{.spec.providerID}')
|
||||
region=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" \
|
||||
-o jsonpath='{.metadata.labels.topology\.kubernetes\.io/region}')
|
||||
|
||||
[ -n "$provider" ] || die "node $node has no providerID — the CCM did not initialise it"
|
||||
[ -n "$region" ] || die "node $node has no topology region label — the CSI node plugin cannot start"
|
||||
log "node $node: providerID=$provider region=$region"
|
||||
}
|
||||
|
||||
# Bound is not proof: a StorageClass can bind a volume that never becomes a disk.
|
||||
# The disk on the hypervisor is the proof, so the check reads `qm config`, then
|
||||
# removes what it created and reads the storage again: a deleted claim whose disk
|
||||
# is still there is the leak the pre-delete hook's Step 1b exists to prevent.
|
||||
check_pvc_round_trip() {
|
||||
fetch_tenant_kubeconfig
|
||||
vmid=$(vmid_of_machine)
|
||||
log "binding a PVC and looking for its disk on VM $vmid"
|
||||
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl apply -f - <<'EOF'
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: e2e-csi-probe
|
||||
spec:
|
||||
accessModes: [ReadWriteOnce]
|
||||
storageClassName: proxmox
|
||||
resources:
|
||||
requests:
|
||||
storage: 1Gi
|
||||
---
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: e2e-csi-probe
|
||||
spec:
|
||||
containers:
|
||||
- name: c
|
||||
image: busybox:1.36
|
||||
command: ["sh", "-c", "sleep 600"]
|
||||
volumeMounts:
|
||||
- name: v
|
||||
mountPath: /data
|
||||
volumes:
|
||||
- name: v
|
||||
persistentVolumeClaim:
|
||||
claimName: e2e-csi-probe
|
||||
EOF
|
||||
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl wait --for=condition=Ready pod/e2e-csi-probe --timeout=5m \
|
||||
|| die "the probe Pod never became Ready — the volume did not attach"
|
||||
|
||||
disk=$(pve "qm config $vmid" | sed -n "s/^scsi[1-9]: *\($STORAGE:[^,]*\),.*/\1/p" | head -1)
|
||||
if [ -z "$disk" ]; then
|
||||
pve "qm config $vmid" | sed 's/^/ qm-config: /' >&2
|
||||
die "PVC bound but no disk is attached to VM $vmid on storage $STORAGE"
|
||||
fi
|
||||
log "disk attached on the hypervisor: $disk"
|
||||
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pod e2e-csi-probe --wait=true
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pvc e2e-csi-probe --wait=true
|
||||
|
||||
wait_disk_freed "$disk"
|
||||
}
|
||||
|
||||
# The disk counts as freed only when a listing that succeeded no longer shows
|
||||
# it. A listing that fails — ssh down, pvesm erroring — proves nothing, so it
|
||||
# fails the check instead of reading as an empty storage.
|
||||
wait_disk_freed() {
|
||||
disk=$1
|
||||
volid=${disk#*:}
|
||||
deadline=$(( $(date +%s) + 180 ))
|
||||
while :; do
|
||||
listing=$(pve "pvesm list $STORAGE") \
|
||||
|| die "cannot list $STORAGE on the hypervisor, so cannot tell whether $disk was freed"
|
||||
printf '%s\n' "$listing" | grep -qF "$volid" || break
|
||||
[ "$(date +%s)" -lt "$deadline" ] || die "PVC deleted but $disk is still on $STORAGE: the driver did not free it"
|
||||
sleep 5
|
||||
done
|
||||
log "disk $disk is gone from $STORAGE"
|
||||
}
|
||||
|
||||
# A replaced Machine has to converge without an operator: the departed Node must
|
||||
# be removed by the cloud-controller-manager and the replacement must register
|
||||
# in its place. "Without intervention" is the whole assertion, so nothing here
|
||||
# touches the cluster beyond the initial delete.
|
||||
check_replacement_converges() {
|
||||
deadline=$(( $(date +%s) + 900 ))
|
||||
old=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}')
|
||||
[ -n "$old" ] || die "no Machine to replace in $NS"
|
||||
log "deleting Machine $old and waiting for an unattended replacement"
|
||||
|
||||
kubectl -n "$NS" delete machine "$old" --wait=false
|
||||
|
||||
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||
name=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
|
||||
phase=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
|
||||
noderef=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.nodeRef.name}' 2>/dev/null || true)
|
||||
if [ -n "$name" ] && [ "$name" != "$old" ] && [ "$phase" = "Running" ] && [ -n "$noderef" ]; then
|
||||
log "replacement $name reached Running with nodeRef $noderef"
|
||||
return 0
|
||||
fi
|
||||
sleep 15
|
||||
done
|
||||
|
||||
kubectl -n "$NS" get machine,machinedeployment -o wide >&2 || true
|
||||
die "no replacement reached Running with a nodeRef within 15m: the departed Node was not removed, or the replacement never registered"
|
||||
}
|
||||
|
||||
# Runs from the suite's catch. Cheap reads first, so a truncated dump still
|
||||
# carries the ones that discriminate between failure modes.
|
||||
dump_diagnostics() {
|
||||
log "=== Machines and their infrastructure ==="
|
||||
kubectl -n "$NS" get machine,proxmoxmachine,machinedeployment,proxmoxcluster -o wide || true
|
||||
log "=== capmox controller ==="
|
||||
kubectl -n cozy-cluster-api logs -l control-plane=controller-manager --tail=80 \
|
||||
--selector=cluster.x-k8s.io/provider=infrastructure-proxmox || true
|
||||
log "=== VMs on the hypervisor ==="
|
||||
pve "qm list" || true
|
||||
if [ -s "$TENANT_KUBECONFIG" ]; then
|
||||
log "=== tenant nodes and pods ==="
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o wide || true
|
||||
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get pods -A -o wide || true
|
||||
fi
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
render-tenant) render_tenant ;;
|
||||
apply-tenant) apply_tenant ;;
|
||||
linked-clone) check_linked_clone ;;
|
||||
tenant-node) check_tenant_node ;;
|
||||
pvc-round-trip) check_pvc_round_trip ;;
|
||||
disk-freed) wait_disk_freed "${2:?disk-freed needs <storage>:<volume>}" ;;
|
||||
replacement) check_replacement_converges ;;
|
||||
diagnostics) dump_diagnostics ;;
|
||||
*) die "usage: $0 {render-tenant|apply-tenant|linked-clone|tenant-node|pvc-round-trip|disk-freed <storage>:<volume>|replacement|diagnostics}" ;;
|
||||
esac
|
||||
@@ -0,0 +1,135 @@
|
||||
# End-to-end coverage for the Proxmox substrate.
|
||||
#
|
||||
# Parked as .disabled on purpose. Suites are discovered with
|
||||
# find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml
|
||||
# so this one is registered nowhere and cannot be selected by the normal lanes.
|
||||
# That is the point: it needs a Proxmox hypervisor, and the runners those lanes
|
||||
# use will never have one. .github/workflows/e2e-proxmox.yaml drops the suffix
|
||||
# on a self-hosted runner that does.
|
||||
#
|
||||
# What this asserts is what NO other suite can, and every item is a bug this
|
||||
# integration actually hit rather than a shape someone imagined:
|
||||
#
|
||||
# ProxmoxCluster Ready — a missing IPAM provider, or the managed-by
|
||||
# annotation that stops CAPI setting an ownerRef
|
||||
# linked-clone boot disk — 8 KiB per worker against 10.2 GiB, and nothing
|
||||
# fails when it regresses
|
||||
# Machine Running + nodeRef — capmox writes proxmox://<UUID>, the CCM's default
|
||||
# mode writes proxmox://<region>/<vmid>, and CAPI
|
||||
# pairs on an exact match
|
||||
# node Ready + region label — the CSI node plugin exits fatally without it
|
||||
# PVC bound AND on the host — Bound is not proof that a disk exists
|
||||
# unattended replacement — the controller that removes a departed Node
|
||||
# must not itself depend on that Node
|
||||
#
|
||||
# It does NOT install Cozystack. The upstream lanes build a sandbox per run;
|
||||
# this one runs against an existing cluster and an existing hypervisor, which is
|
||||
# a weaker guarantee and is stated rather than implied.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: kubernetes-proxmox
|
||||
spec:
|
||||
# Teardown is the finally below, which must run whether the suite passed or
|
||||
# failed: a leaked tenant leaves VMs on the hypervisor.
|
||||
skipDelete: true
|
||||
catch:
|
||||
- events: {}
|
||||
- script:
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh diagnostics
|
||||
steps:
|
||||
- name: tenant-and-cluster
|
||||
try:
|
||||
# Rendered by run-proxmox.sh rather than applied as is: every stand-specific
|
||||
# value in tenant.yaml is a ${COZY_PVE_*} variable, the script supplies the
|
||||
# defaults and expands them, and it refuses a fixture with a variable left
|
||||
# in it. Another stand sets variables instead of editing the file.
|
||||
- script:
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh apply-tenant
|
||||
# Declarative where the object lives in this cluster, per the conventions in
|
||||
# docs/agents/e2e-testing.md — the assert polls to its own timeout and
|
||||
# reports a structured diff rather than a generic non-zero exit.
|
||||
# Both the name and the namespace matter here. The resource definition
|
||||
# prefixes the release, so the Kubernetes CR named k8s-e2e produces
|
||||
# ProxmoxCluster/kubernetes-k8s-e2e, and it lives in the tenant's own
|
||||
# namespace rather than the suite's default one. Asserted with the wrong
|
||||
# pair this step simply spends its 10m and fails on a healthy stand.
|
||||
- assert:
|
||||
timeout: 10m
|
||||
resource:
|
||||
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
|
||||
kind: ProxmoxCluster
|
||||
metadata:
|
||||
name: kubernetes-k8s-e2e
|
||||
namespace: tenant-e2eproxmox
|
||||
status:
|
||||
ready: true
|
||||
- assert:
|
||||
timeout: 20m
|
||||
resource:
|
||||
apiVersion: cluster.x-k8s.io/v1beta1
|
||||
kind: Machine
|
||||
metadata:
|
||||
namespace: tenant-e2eproxmox
|
||||
status:
|
||||
phase: Running
|
||||
# The controller half of the CSI driver and the cloud-controller-manager
|
||||
# run in THIS cluster, not in the tenant; that is the property the storage
|
||||
# PR is about, so the suite asserts it rather than only the tenant side.
|
||||
- assert:
|
||||
timeout: 5m
|
||||
resource:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: kubernetes-k8s-e2e-pcsi-controller
|
||||
namespace: tenant-e2eproxmox
|
||||
status:
|
||||
readyReplicas: 1
|
||||
- assert:
|
||||
timeout: 5m
|
||||
resource:
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: kubernetes-k8s-e2e-pccm
|
||||
namespace: tenant-e2eproxmox
|
||||
status:
|
||||
readyReplicas: 1
|
||||
|
||||
- name: vm-shape-on-the-hypervisor
|
||||
try:
|
||||
- script:
|
||||
timeout: 3m
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh linked-clone
|
||||
|
||||
- name: tenant-node-usable
|
||||
try:
|
||||
- script:
|
||||
timeout: 10m
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh tenant-node
|
||||
|
||||
- name: storage-round-trip
|
||||
try:
|
||||
- script:
|
||||
timeout: 10m
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh pvc-round-trip
|
||||
|
||||
- name: replacement-converges-unattended
|
||||
try:
|
||||
- script:
|
||||
timeout: 20m
|
||||
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh replacement
|
||||
finally:
|
||||
- script:
|
||||
content: |
|
||||
set -eu
|
||||
# keep_tenant is honoured here as well as in the workflow's own
|
||||
# teardown, or the suite would delete what the operator asked to keep.
|
||||
if [ "${COZY_KEEP_TENANT:-false}" = "true" ]; then
|
||||
echo "COZY_KEEP_TENANT=true: leaving the tenant in place"
|
||||
exit 0
|
||||
fi
|
||||
kubectl delete kubernetesnodes.apps.cozystack.io -n "${COZY_PROXMOX_NS:-tenant-e2eproxmox}" --all --ignore-not-found
|
||||
kubectl delete kubernetes.apps.cozystack.io -n "${COZY_PROXMOX_NS:-tenant-e2eproxmox}" --all --ignore-not-found
|
||||
kubectl delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found
|
||||
@@ -0,0 +1,93 @@
|
||||
# The tenant this suite builds and tears down. Its own namespace rather than the
|
||||
# shared tenant-test: these objects create VMs on a hypervisor, and a leak is
|
||||
# visible in `qm list` rather than only in the API.
|
||||
#
|
||||
# etcd: true is not optional. KamajiControlPlane.spec.dataStoreName reads
|
||||
# _namespace.etcd, and the DataStore is created by the tenant's own etcd module
|
||||
# (packages/extra/etcd/templates/datastore.yaml) — without it the control plane
|
||||
# does not render at all.
|
||||
# Tenant names are validated as lowercase letters and digits only — no dashes —
|
||||
# so this is e2eproxmox rather than the e2e-proxmox the namespace would suggest.
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: Tenant
|
||||
metadata:
|
||||
name: e2eproxmox
|
||||
namespace: tenant-root
|
||||
spec:
|
||||
etcd: true
|
||||
host: ""
|
||||
ingress: false
|
||||
monitoring: false
|
||||
seaweedfs: false
|
||||
resourceQuotas: {}
|
||||
---
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: Kubernetes
|
||||
metadata:
|
||||
name: k8s-e2e
|
||||
namespace: tenant-e2eproxmox
|
||||
spec:
|
||||
version: "v1.35"
|
||||
substrate: proxmox
|
||||
proxmox:
|
||||
# The stand's Proxmox serves its self-signed certificate; the chart verifies
|
||||
# by default, so the suite says so rather than relying on a default.
|
||||
insecure: true
|
||||
# Empty means any node in the Proxmox cluster; the suite must not hardcode a
|
||||
# hostname it would then only pass on one stand.
|
||||
allowedNodes: []
|
||||
# An off-cluster worker cannot resolve through the management cluster's
|
||||
# CoreDNS, so the resolvers are its own. The chart refuses to render without
|
||||
# them rather than writing an address that cannot answer.
|
||||
dnsServers: ["8.8.8.8", "1.1.1.1"]
|
||||
ipv4Config:
|
||||
# Must not overlap the DHCP range, the CAPI IPAM pool or the MetalLB pool
|
||||
# on the same L2 — four owners hand out addresses there and a collision is
|
||||
# silent. Substituted per stand: run-proxmox.sh render-tenant expands the
|
||||
# COZY_PVE_* variables in this file with envsubst and supplies the defaults
|
||||
# itself, because GNU envsubst leaves a shell-style default in the text
|
||||
# exactly as it is. A variable it does not know stays put and is refused.
|
||||
addresses: ["${COZY_PVE_ADDRESSES}"]
|
||||
gateway: "${COZY_PVE_GATEWAY}"
|
||||
prefix: ${COZY_PVE_PREFIX}
|
||||
ccm:
|
||||
credentialsSecretName: proxmox-credentials
|
||||
csi:
|
||||
enabled: true
|
||||
credentialsSecretName: proxmox-credentials
|
||||
storageClasses:
|
||||
- name: proxmox
|
||||
# The same storage the PVC probe looks on, so the two cannot drift apart.
|
||||
storage: "${COZY_PROXMOX_STORAGE}"
|
||||
fstype: ext4
|
||||
reclaimPolicy: Delete
|
||||
---
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: KubernetesNodes
|
||||
metadata:
|
||||
name: md0
|
||||
namespace: tenant-e2eproxmox
|
||||
spec:
|
||||
cluster: k8s-e2e
|
||||
minReplicas: 1
|
||||
maxReplicas: 1
|
||||
instanceType: ""
|
||||
resources:
|
||||
cpu: 2
|
||||
memory: 4Gi
|
||||
substrate: proxmox
|
||||
proxmox:
|
||||
# capmox matches tags as a SET, not by containment: a template carrying four
|
||||
# tags is not selected by naming one of them.
|
||||
templateTags: [capmox-template, cozystack, talos, talos-v1.13.6]
|
||||
# false is the default and the reason the suite asserts it on the hypervisor:
|
||||
# a full clone costs the whole template disk per worker and breaks nothing.
|
||||
# No `storage` with it: a linked clone lives on the template's storage, and
|
||||
# the chart refuses the pair.
|
||||
full: false
|
||||
network:
|
||||
bridge: "${COZY_PVE_BRIDGE}"
|
||||
# The stand's resource pool, the one ACL path the tenant's tokens are
|
||||
# scoped to; empty adds the workers to none.
|
||||
pool: "${COZY_PVE_POOL}"
|
||||
dnsServers: ["8.8.8.8", "1.1.1.1"]
|
||||
Executable
+60
@@ -0,0 +1,60 @@
|
||||
#!/usr/bin/env bats
|
||||
# The Proxmox e2e fixture is stand-agnostic through ${COZY_PVE_*} variables that
|
||||
# hack/e2e-chainsaw/_lib/run-proxmox.sh expands with envsubst. GNU envsubst
|
||||
# expands ${VAR} and nothing else: a ${VAR:-default} stays in the text as it is,
|
||||
# set or not, and reaches the apiserver as a string where the schema wants an
|
||||
# integer. So the defaults live in the script, the fixture carries bare
|
||||
# variables, and the render refuses to hand out a fixture with one left in it.
|
||||
|
||||
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||
FIXTURE="$REPO_ROOT/hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml"
|
||||
RUNNER="$REPO_ROOT/hack/e2e-chainsaw/_lib/run-proxmox.sh"
|
||||
|
||||
@test "the fixture carries no shell-style default that envsubst would leave in place" {
|
||||
! grep -qE '\$\{[A-Za-z_]+:[-=?+]' "$FIXTURE"
|
||||
}
|
||||
|
||||
@test "render-tenant expands every variable with the runner's defaults" {
|
||||
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=teststore "$RUNNER" render-tenant)"
|
||||
! printf '%s' "$out" | grep -q '\${'
|
||||
printf '%s' "$out" | grep -qE '^ +prefix: 24$'
|
||||
printf '%s' "$out" | grep -qE '^ +storage: "teststore"$'
|
||||
printf '%s' "$out" | grep -qE '^ +bridge: "vmbr50"$'
|
||||
}
|
||||
|
||||
@test "render-tenant honours a stand's own values and keeps prefix an integer" {
|
||||
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s COZY_PVE_PREFIX=16 COZY_PVE_BRIDGE=vmbr1 "$RUNNER" render-tenant)"
|
||||
printf '%s' "$out" | grep -qE '^ +prefix: 16$'
|
||||
printf '%s' "$out" | grep -qE '^ +bridge: "vmbr1"$'
|
||||
}
|
||||
|
||||
@test "render-tenant passes the stand's resource pool through, and none by default" {
|
||||
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s COZY_PVE_POOL=tenant-e2e "$RUNNER" render-tenant)"
|
||||
printf '%s' "$out" | grep -qE '^ +pool: "tenant-e2e"$'
|
||||
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s "$RUNNER" render-tenant)"
|
||||
printf '%s' "$out" | grep -qE '^ +pool: ""$'
|
||||
}
|
||||
|
||||
# The PVC probe waits for the disk to leave the storage. A listing that fails
|
||||
# must fail the check: read as empty, it would report a leaked disk as freed.
|
||||
# (hack/cozytest.sh runs these as POSIX sh: no `run`, no [[ ]].)
|
||||
@test "disk-freed fails when the storage cannot be listed, instead of calling the disk gone" {
|
||||
stub="$(mktemp -d)"
|
||||
printf '#!/bin/sh\necho "ssh: connect to host pve: Connection refused" >&2\nexit 255\n' > "$stub/ssh"
|
||||
chmod +x "$stub/ssh"
|
||||
rc=0
|
||||
out="$(PATH="$stub:$PATH" COZY_PVE_SSH=root@pve COZY_PROXMOX_STORAGE=main-pool "$RUNNER" disk-freed main-pool:vm-1-pvc-x 2>&1)" || rc=$?
|
||||
rm -rf "$stub"
|
||||
[ "$rc" -ne 0 ]
|
||||
case "$out" in *"cannot list main-pool"*) ;; *) echo "$out" >&2; exit 1 ;; esac
|
||||
case "$out" in *"is gone"*) echo "$out" >&2; exit 1 ;; esac
|
||||
}
|
||||
|
||||
@test "disk-freed reports the disk gone once a successful listing no longer shows it" {
|
||||
stub="$(mktemp -d)"
|
||||
printf '#!/bin/sh\necho "Volid Format Type Size VMID"\necho "main-pool:vm-2-pvc-y raw images 1073741824 2"\n' > "$stub/ssh"
|
||||
chmod +x "$stub/ssh"
|
||||
out="$(PATH="$stub:$PATH" COZY_PVE_SSH=root@pve COZY_PROXMOX_STORAGE=main-pool "$RUNNER" disk-freed main-pool:vm-1-pvc-x 2>&1)"
|
||||
rm -rf "$stub"
|
||||
case "$out" in *"is gone from main-pool"*) ;; *) echo "$out" >&2; exit 1 ;; esac
|
||||
}
|
||||
@@ -1569,9 +1569,16 @@ EOF
|
||||
# not a filter by itself either: what a creator can be
|
||||
# written in is decided beside the sweep, so that prose acquiring a fenced
|
||||
# manifest reds nothing here.
|
||||
#
|
||||
# The Proxmox fixture is the second entry and is deliberately not governed by
|
||||
# the rules above: those read run-kubernetes.sh's shell, and this one is a
|
||||
# declarative manifest a chainsaw `apply:` block applies. It creates a cluster
|
||||
# all the same, which is why it is named here rather than excluded — a reader
|
||||
# asking "what creates a tenant cluster in this tree" gets both answers. The
|
||||
# suite that applies it ships `.disabled`, so nothing runs it in CI today.
|
||||
found="$(_kind_creators "$REPO_ROOT/hack/e2e-chainsaw" \
|
||||
| sed "s|^$REPO_ROOT/||" | tr '\n' ' ')"
|
||||
want="hack/e2e-chainsaw/_lib/run-kubernetes.sh "
|
||||
want="hack/e2e-chainsaw/_lib/run-kubernetes.sh hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml "
|
||||
if [ "$found" != "$want" ]; then
|
||||
echo "scripts under hack/e2e-chainsaw creating a Kubernetes CR: [$found]" >&2
|
||||
echo "Check first whether the new file APPLIES the kind or only asserts on" >&2
|
||||
|
||||
Reference in new issue
Block a user