mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-02 18:16:53 +09:00
test(kubernetes): add the Proxmox e2e suite, parked disabled
Upstream CI has no hypervisor, so this suite cannot run there; it ships
`.disabled` and is run by a lane that owns a Proxmox stand.
Named for what the chart actually creates. The resource definition prefixes the
release, so a Kubernetes CR called k8s-e2e produces
ProxmoxCluster/kubernetes-k8s-e2e in the tenant's own namespace, and the admin
kubeconfig Secret is kubernetes-k8s-e2e-admin-kubeconfig.
Stand-specific values are variables rather than hardcoded. Every one of them is
a bare ${COZY_PVE_*} that run-proxmox.sh expands with envsubst, supplying the
defaults itself — GNU envsubst leaves a ${VAR:-default} in the text exactly as
it is, set or not, and that reaches the apiserver as a string where the schema
wants an integer — and refusing a fixture with a variable still in it. The
StorageClass and the PVC probe read the same COZY_PROXMOX_STORAGE, so the volume
lands where the probe looks. The runner reads the tenant kubeconfig's .conf key,
the address a runner outside the management cluster can reach.
The stand's resource pool comes in through COZY_PVE_POOL, empty by default.
hack/proxmox-e2e-fixture_test.bats pins the shape of the fixture and the render.
The PVC probe reads the storage again after the claim is deleted: a deleted
claim whose disk is still there is the leak the pre-delete hook's Step 1b
exists to prevent. keep_tenant is honoured by the suite's own teardown, not
only by the workflow's. The workflow carries no fallback for the stand's
variables: an unset one fails the job before a key is sent anywhere. The
schedule runs only where vars.PROXMOX_NIGHTLY is true: the job targets a
self-hosted runner with the `proxmox` label, and a scheduled run on a repository
without one would sit in the queue until GitHub cancels it.
The suite also asserts the half of the deployment that lives in the management
cluster — the CSI controller and the cloud-controller-manager — because that is
the property the pair is designed around and a tenant-only assertion would pass
with the token in the wrong place.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
This commit is contained in:
Marian Koreniuk committed
1 parent
36b04218d7
commit
7500a9fbae
7 files changed
+666
-1
No files matched your search
@@ -0,0 +1,122 @@
|
|||||||
|
name: E2E (proxmox)
|
||||||
|
|
||||||
|
# The Proxmox substrate's own lane, separate from every other e2e workflow for
|
||||||
|
# one reason that cannot be engineered away: it needs a Proxmox hypervisor, and
|
||||||
|
# the oracle-vm-* runners the other lanes use will never have one.
|
||||||
|
#
|
||||||
|
# Deliberately NOT triggered by pull_request. A self-hosted runner executes
|
||||||
|
# whatever the workflow checks out, and this one holds credentials to a
|
||||||
|
# hypervisor and a live cluster; letting an arbitrary fork PR start it would
|
||||||
|
# hand both to its author. Manual dispatch and a schedule are the whole trigger
|
||||||
|
# surface, and both come from someone who already has write access.
|
||||||
|
on:
|
||||||
|
workflow_dispatch:
|
||||||
|
inputs:
|
||||||
|
keep_tenant:
|
||||||
|
description: "Leave the tenant in place after the run, for debugging"
|
||||||
|
type: boolean
|
||||||
|
default: false
|
||||||
|
schedule:
|
||||||
|
# Nightly, offset from the upstream lanes so a shared hypervisor is not
|
||||||
|
# asked to do two things at once.
|
||||||
|
- cron: "40 2 * * *"
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
|
||||||
|
concurrency:
|
||||||
|
# One hypervisor, one tenant namespace, fixed VM ids — two runs would fight
|
||||||
|
# over all three. Never cancel in progress: a killed run leaks VMs.
|
||||||
|
group: e2e-proxmox
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
e2e:
|
||||||
|
name: Proxmox substrate
|
||||||
|
# The job needs a self-hosted runner carrying the `proxmox` label. On a
|
||||||
|
# repository without one a scheduled run would sit in the queue until GitHub
|
||||||
|
# cancels it, so the schedule fires only where the stand is declared;
|
||||||
|
# manual dispatch always runs.
|
||||||
|
if: github.event_name == 'workflow_dispatch' || vars.PROXMOX_NIGHTLY == 'true'
|
||||||
|
runs-on: [self-hosted, proxmox]
|
||||||
|
timeout-minutes: 90
|
||||||
|
env:
|
||||||
|
COZY_PROXMOX_NS: tenant-e2eproxmox
|
||||||
|
COZY_PROXMOX_CLUSTER: k8s-e2e
|
||||||
|
# No fallbacks: an unset variable fails the first step below instead of
|
||||||
|
# sending a key to a guessed address or probing a storage the stand lacks.
|
||||||
|
COZY_PROXMOX_STORAGE: ${{ vars.PROXMOX_STORAGE }}
|
||||||
|
COZY_PVE_SSH: ${{ vars.PROXMOX_SSH }}
|
||||||
|
COZY_PVE_SSH_KEY: ${{ github.workspace }}/.ssh-pve
|
||||||
|
COZY_KEEP_TENANT: ${{ inputs.keep_tenant == true }}
|
||||||
|
steps:
|
||||||
|
- name: Checkout
|
||||||
|
uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4
|
||||||
|
with:
|
||||||
|
persist-credentials: false
|
||||||
|
|
||||||
|
- name: Require the stand's variables
|
||||||
|
env:
|
||||||
|
PVE_URL: ${{ vars.PROXMOX_URL }}
|
||||||
|
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
for v in COZY_PROXMOX_STORAGE COZY_PVE_SSH PVE_URL PVE_REGION; do
|
||||||
|
eval "val=\${$v:-}"
|
||||||
|
[ -n "$val" ] || { echo "$v is unset: set the PROXMOX_* repository variables for this stand" >&2; exit 1; }
|
||||||
|
done
|
||||||
|
|
||||||
|
- name: Stage credentials
|
||||||
|
env:
|
||||||
|
KUBECONFIG_B64: ${{ secrets.COZY_KUBECONFIG }}
|
||||||
|
PVE_SSH_KEY: ${{ secrets.PROXMOX_SSH_KEY }}
|
||||||
|
PVE_TOKEN_ID: ${{ secrets.PROXMOX_TOKEN_ID }}
|
||||||
|
PVE_TOKEN_SECRET: ${{ secrets.PROXMOX_TOKEN_SECRET }}
|
||||||
|
PVE_URL: ${{ vars.PROXMOX_URL }}
|
||||||
|
PVE_REGION: ${{ vars.PROXMOX_REGION }}
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
umask 077
|
||||||
|
printf '%s' "$KUBECONFIG_B64" | base64 -d > "$GITHUB_WORKSPACE/.kubeconfig"
|
||||||
|
printf '%s' "$PVE_SSH_KEY" > "$COZY_PVE_SSH_KEY"
|
||||||
|
echo "KUBECONFIG=$GITHUB_WORKSPACE/.kubeconfig" >> "$GITHUB_ENV"
|
||||||
|
|
||||||
|
# The credentials the management-side CSI controller and CCM read. A
|
||||||
|
# Secret rather than values so no token is ever written into a
|
||||||
|
# rendered HelmRelease.
|
||||||
|
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" create namespace "$COZY_PROXMOX_NS" \
|
||||||
|
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
||||||
|
kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" -n "$COZY_PROXMOX_NS" \
|
||||||
|
create secret generic proxmox-credentials \
|
||||||
|
--from-literal=url="$PVE_URL" \
|
||||||
|
--from-literal=token_id="$PVE_TOKEN_ID" \
|
||||||
|
--from-literal=token_secret="$PVE_TOKEN_SECRET" \
|
||||||
|
--from-literal=region="$PVE_REGION" \
|
||||||
|
--dry-run=client -o yaml | kubectl --kubeconfig "$GITHUB_WORKSPACE/.kubeconfig" apply -f -
|
||||||
|
|
||||||
|
- name: Enable the suite
|
||||||
|
# It ships disabled so the normal lanes cannot select it; this is the
|
||||||
|
# one place that is allowed to turn it on.
|
||||||
|
run: mv hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled
|
||||||
|
hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml
|
||||||
|
|
||||||
|
- name: Run the Proxmox suite
|
||||||
|
run: |
|
||||||
|
chainsaw test \
|
||||||
|
--config hack/e2e-chainsaw/.chainsaw.yaml \
|
||||||
|
--test-dir hack/e2e-chainsaw/kubernetes-proxmox
|
||||||
|
|
||||||
|
- name: Tear down
|
||||||
|
# always(), because the whole point of the teardown is the failing run:
|
||||||
|
# a leaked tenant leaves VMs on the hypervisor and the next run inherits
|
||||||
|
# them.
|
||||||
|
if: always() && !inputs.keep_tenant
|
||||||
|
run: |
|
||||||
|
set -eu
|
||||||
|
kubectl delete kubernetesnodes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
|
||||||
|
kubectl delete kubernetes.apps.cozystack.io -n "$COZY_PROXMOX_NS" --all --ignore-not-found
|
||||||
|
kubectl delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found
|
||||||
|
|
||||||
|
- name: Remove staged credentials
|
||||||
|
if: always()
|
||||||
|
run: rm -f "$GITHUB_WORKSPACE/.kubeconfig" "$COZY_PVE_SSH_KEY"
|
||||||
@@ -10,6 +10,16 @@ The app suite is **Kyverno Chainsaw** (`hack/e2e-chainsaw/`, one directory per a
|
|||||||
|
|
||||||
Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.
|
Every convention below follows from that finding: **fail fast, fail loud, make the failure legible.** A test that flakes is a test (or a product) with a real race to fix, not a test to wrap in a retry. Chainsaw's per-assertion polling is the structural expression of this: each `assert` waits on its own condition to its own timeout, and a failure is reported as a structured diff plus auto-captured events/describe/logs — not a generic non-zero exit.
|
||||||
|
|
||||||
|
## The Proxmox lane
|
||||||
|
|
||||||
|
`hack/e2e-chainsaw/kubernetes-proxmox/` covers the Proxmox substrate and is the one suite in this tree that the normal lanes cannot run. It needs a Proxmox hypervisor, and the `oracle-vm-*` runners have none, so it ships as `chainsaw-test.yaml.disabled`: suite discovery is `find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml`, which registers it nowhere and makes it unselectable rather than merely unselected. `.github/workflows/e2e-proxmox.yaml` drops the suffix on a self-hosted runner that has one.
|
||||||
|
|
||||||
|
Two properties of that workflow are security boundaries rather than preferences. It is never triggered by `pull_request`, because a self-hosted runner executes what the workflow checks out and this one stages a hypervisor token and a live kubeconfig — a fork PR trigger would hand both to its author. And its concurrency group never cancels in progress, because a killed run leaks VMs on the hypervisor rather than merely losing a result.
|
||||||
|
|
||||||
|
It does not install Cozystack. Every other lane builds a sandbox per run; this one asserts against an existing cluster and an existing hypervisor, which is a weaker guarantee and is stated in the suite header rather than left to be discovered.
|
||||||
|
|
||||||
|
The imperative half lives in `_lib/run-proxmox.sh` and is deliberately not a port of `_lib/run-kubernetes.sh`: that script's diagnostics are virt-launcher Pods, VirtualMachineInstances and guest consoles, none of which exist on an external hypervisor.
|
||||||
|
|
||||||
## Conventions
|
## Conventions
|
||||||
|
|
||||||
### 1. No retries on deterministic steps; retry only pure infrastructure
|
### 1. No retries on deterministic steps; retry only pure infrastructure
|
||||||
|
|||||||
Executable
+238
@@ -0,0 +1,238 @@
|
|||||||
|
#!/bin/sh
|
||||||
|
# Imperative half of the kubernetes-proxmox suite.
|
||||||
|
#
|
||||||
|
# Chainsaw's declarative asserts cover everything that is a Kubernetes object in
|
||||||
|
# THIS cluster, and the suite uses them for exactly that. What is left needs a
|
||||||
|
# shell: the hypervisor is reached over SSH and answers in `qm` output, and the
|
||||||
|
# tenant cluster is reached with a kubeconfig that only exists once its control
|
||||||
|
# plane is up. Those are the two things below.
|
||||||
|
#
|
||||||
|
# Deliberately not a port of _lib/run-kubernetes.sh. That script's diagnostic
|
||||||
|
# half is virt-launcher Pods, VirtualMachineInstances and guest consoles —
|
||||||
|
# objects with no counterpart on an external hypervisor — so sharing it would
|
||||||
|
# mean carrying a thousand lines that can never fire here.
|
||||||
|
#
|
||||||
|
# Every subcommand exits non-zero on failure and says which check failed. There
|
||||||
|
# is no `|| echo` anywhere in this file on purpose: the pattern turns a failed
|
||||||
|
# command into a green run, and this repository has been bitten by it.
|
||||||
|
set -eu
|
||||||
|
|
||||||
|
NS="${COZY_PROXMOX_NS:-tenant-e2eproxmox}"
|
||||||
|
CLUSTER="${COZY_PROXMOX_CLUSTER:-k8s-e2e}"
|
||||||
|
PVE_SSH="${COZY_PVE_SSH:?COZY_PVE_SSH is required: the ssh destination of the Proxmox host, user@host}"
|
||||||
|
PVE_SSH_KEY="${COZY_PVE_SSH_KEY:-$HOME/.ssh/id_ed25519}"
|
||||||
|
STORAGE="${COZY_PROXMOX_STORAGE:?COZY_PROXMOX_STORAGE is required, the PVE storage id for the PVC probe}"
|
||||||
|
TENANT_KUBECONFIG="${TMPDIR:-/tmp}/proxmox-e2e-tenant.kubeconfig"
|
||||||
|
FIXTURE="$(cd "$(dirname "$0")/.." && pwd)/kubernetes-proxmox/tenant.yaml"
|
||||||
|
|
||||||
|
log() { printf '[%s] %s\n' "$(date -u +%H:%M:%S)" "$*"; }
|
||||||
|
die() { printf '[%s] FAILED: %s\n' "$(date -u +%H:%M:%S)" "$*" >&2; exit 1; }
|
||||||
|
|
||||||
|
pve() { ssh -o BatchMode=yes -o StrictHostKeyChecking=accept-new -i "$PVE_SSH_KEY" "$PVE_SSH" "$@"; }
|
||||||
|
|
||||||
|
# The Machine carries the VM id in the ProxmoxMachine it references. Reading it
|
||||||
|
# from the cluster rather than guessing keeps this working when a replacement
|
||||||
|
# lands on a different id.
|
||||||
|
vmid_of_machine() {
|
||||||
|
kubectl -n "$NS" get proxmoxmachine -o jsonpath='{.items[0].spec.virtualMachineID}'
|
||||||
|
}
|
||||||
|
|
||||||
|
# --- checks -----------------------------------------------------------------
|
||||||
|
|
||||||
|
# A full clone is a copy of the template's disk; a linked clone references it and
|
||||||
|
# stores only the increment. Nothing fails when this is wrong — the pool comes
|
||||||
|
# up either way — so the only place it can be caught is here, by the shape of the
|
||||||
|
# volume name: a linked clone points at `base-<template>-disk-N/`.
|
||||||
|
check_linked_clone() {
|
||||||
|
vmid=$(vmid_of_machine)
|
||||||
|
[ -n "$vmid" ] || die "no ProxmoxMachine with a virtualMachineID in $NS"
|
||||||
|
log "VM $vmid: checking the boot disk is a linked clone"
|
||||||
|
|
||||||
|
bootdisk=$(pve "qm config $vmid" | sed -n 's/^scsi0: *//p')
|
||||||
|
[ -n "$bootdisk" ] || die "VM $vmid has no scsi0"
|
||||||
|
|
||||||
|
case "$bootdisk" in
|
||||||
|
*:base-*-disk-*/*) log "linked clone confirmed: $bootdisk" ;;
|
||||||
|
*) die "VM $vmid boot disk is a full copy, not a linked clone: $bootdisk" ;;
|
||||||
|
esac
|
||||||
|
}
|
||||||
|
|
||||||
|
# The fixture is stand-agnostic through ${COZY_PVE_*} variables. GNU envsubst
|
||||||
|
# expands ${VAR} and nothing else — a ${VAR:-default} stays in the text as it
|
||||||
|
# is, set or not — so the defaults live here, the variable list is explicit so
|
||||||
|
# nothing else in the file is touched, and a fixture with a variable left in it
|
||||||
|
# is refused rather than applied.
|
||||||
|
render_tenant() {
|
||||||
|
: "${COZY_PVE_ADDRESSES:=10.0.0.180-10.0.0.185}"
|
||||||
|
: "${COZY_PVE_GATEWAY:=10.0.0.1}"
|
||||||
|
: "${COZY_PVE_PREFIX:=24}"
|
||||||
|
: "${COZY_PVE_BRIDGE:=vmbr50}"
|
||||||
|
: "${COZY_PVE_POOL:=}"
|
||||||
|
export COZY_PVE_ADDRESSES COZY_PVE_GATEWAY COZY_PVE_PREFIX COZY_PVE_BRIDGE COZY_PVE_POOL COZY_PROXMOX_STORAGE
|
||||||
|
rendered=$(envsubst '$COZY_PVE_ADDRESSES $COZY_PVE_GATEWAY $COZY_PVE_PREFIX $COZY_PVE_BRIDGE $COZY_PVE_POOL $COZY_PROXMOX_STORAGE' < "$FIXTURE")
|
||||||
|
case "$rendered" in
|
||||||
|
*'${'*) die "tenant.yaml still carries an unexpanded variable after envsubst" ;;
|
||||||
|
esac
|
||||||
|
printf '%s\n' "$rendered"
|
||||||
|
}
|
||||||
|
|
||||||
|
apply_tenant() {
|
||||||
|
render_tenant | kubectl apply -f -
|
||||||
|
}
|
||||||
|
|
||||||
|
# The tenant kubeconfig exists only after Kamaji has issued it. Fetching it is
|
||||||
|
# also the first proof that the control plane came up at all. The .conf key
|
||||||
|
# carries the address a runner outside the management cluster can reach; .svc
|
||||||
|
# is the in-cluster Service the management-side controllers use.
|
||||||
|
fetch_tenant_kubeconfig() {
|
||||||
|
kubectl -n "$NS" get secret "kubernetes-${CLUSTER}-admin-kubeconfig" \
|
||||||
|
-o jsonpath='{.data.super-admin\.conf}' | base64 -d > "$TENANT_KUBECONFIG"
|
||||||
|
[ -s "$TENANT_KUBECONFIG" ] || die "tenant kubeconfig is empty — did the control plane come up?"
|
||||||
|
}
|
||||||
|
|
||||||
|
# providerID and the topology labels are applied by the cloud-controller-manager,
|
||||||
|
# and the CSI node plugin refuses to start without the labels. Asserting them
|
||||||
|
# here is what separates "the node joined" from "the node is usable".
|
||||||
|
check_tenant_node() {
|
||||||
|
fetch_tenant_kubeconfig
|
||||||
|
log "checking the tenant node is Ready and initialised by the CCM"
|
||||||
|
|
||||||
|
node=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o jsonpath='{.items[0].metadata.name}')
|
||||||
|
[ -n "$node" ] || die "tenant cluster has no nodes"
|
||||||
|
|
||||||
|
provider=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" -o jsonpath='{.spec.providerID}')
|
||||||
|
region=$(KUBECONFIG="$TENANT_KUBECONFIG" kubectl get node "$node" \
|
||||||
|
-o jsonpath='{.metadata.labels.topology\.kubernetes\.io/region}')
|
||||||
|
|
||||||
|
[ -n "$provider" ] || die "node $node has no providerID — the CCM did not initialise it"
|
||||||
|
[ -n "$region" ] || die "node $node has no topology region label — the CSI node plugin cannot start"
|
||||||
|
log "node $node: providerID=$provider region=$region"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Bound is not proof: a StorageClass can bind a volume that never becomes a disk.
|
||||||
|
# The disk on the hypervisor is the proof, so the check reads `qm config`, then
|
||||||
|
# removes what it created and reads the storage again: a deleted claim whose disk
|
||||||
|
# is still there is the leak the pre-delete hook's Step 1b exists to prevent.
|
||||||
|
check_pvc_round_trip() {
|
||||||
|
fetch_tenant_kubeconfig
|
||||||
|
vmid=$(vmid_of_machine)
|
||||||
|
log "binding a PVC and looking for its disk on VM $vmid"
|
||||||
|
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl apply -f - <<'EOF'
|
||||||
|
apiVersion: v1
|
||||||
|
kind: PersistentVolumeClaim
|
||||||
|
metadata:
|
||||||
|
name: e2e-csi-probe
|
||||||
|
spec:
|
||||||
|
accessModes: [ReadWriteOnce]
|
||||||
|
storageClassName: proxmox
|
||||||
|
resources:
|
||||||
|
requests:
|
||||||
|
storage: 1Gi
|
||||||
|
---
|
||||||
|
apiVersion: v1
|
||||||
|
kind: Pod
|
||||||
|
metadata:
|
||||||
|
name: e2e-csi-probe
|
||||||
|
spec:
|
||||||
|
containers:
|
||||||
|
- name: c
|
||||||
|
image: busybox:1.36
|
||||||
|
command: ["sh", "-c", "sleep 600"]
|
||||||
|
volumeMounts:
|
||||||
|
- name: v
|
||||||
|
mountPath: /data
|
||||||
|
volumes:
|
||||||
|
- name: v
|
||||||
|
persistentVolumeClaim:
|
||||||
|
claimName: e2e-csi-probe
|
||||||
|
EOF
|
||||||
|
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl wait --for=condition=Ready pod/e2e-csi-probe --timeout=5m \
|
||||||
|
|| die "the probe Pod never became Ready — the volume did not attach"
|
||||||
|
|
||||||
|
disk=$(pve "qm config $vmid" | sed -n "s/^scsi[1-9]: *\($STORAGE:[^,]*\),.*/\1/p" | head -1)
|
||||||
|
if [ -z "$disk" ]; then
|
||||||
|
pve "qm config $vmid" | sed 's/^/ qm-config: /' >&2
|
||||||
|
die "PVC bound but no disk is attached to VM $vmid on storage $STORAGE"
|
||||||
|
fi
|
||||||
|
log "disk attached on the hypervisor: $disk"
|
||||||
|
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pod e2e-csi-probe --wait=true
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl delete pvc e2e-csi-probe --wait=true
|
||||||
|
|
||||||
|
wait_disk_freed "$disk"
|
||||||
|
}
|
||||||
|
|
||||||
|
# The disk counts as freed only when a listing that succeeded no longer shows
|
||||||
|
# it. A listing that fails — ssh down, pvesm erroring — proves nothing, so it
|
||||||
|
# fails the check instead of reading as an empty storage.
|
||||||
|
wait_disk_freed() {
|
||||||
|
disk=$1
|
||||||
|
volid=${disk#*:}
|
||||||
|
deadline=$(( $(date +%s) + 180 ))
|
||||||
|
while :; do
|
||||||
|
listing=$(pve "pvesm list $STORAGE") \
|
||||||
|
|| die "cannot list $STORAGE on the hypervisor, so cannot tell whether $disk was freed"
|
||||||
|
printf '%s\n' "$listing" | grep -qF "$volid" || break
|
||||||
|
[ "$(date +%s)" -lt "$deadline" ] || die "PVC deleted but $disk is still on $STORAGE: the driver did not free it"
|
||||||
|
sleep 5
|
||||||
|
done
|
||||||
|
log "disk $disk is gone from $STORAGE"
|
||||||
|
}
|
||||||
|
|
||||||
|
# A replaced Machine has to converge without an operator: the departed Node must
|
||||||
|
# be removed by the cloud-controller-manager and the replacement must register
|
||||||
|
# in its place. "Without intervention" is the whole assertion, so nothing here
|
||||||
|
# touches the cluster beyond the initial delete.
|
||||||
|
check_replacement_converges() {
|
||||||
|
deadline=$(( $(date +%s) + 900 ))
|
||||||
|
old=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}')
|
||||||
|
[ -n "$old" ] || die "no Machine to replace in $NS"
|
||||||
|
log "deleting Machine $old and waiting for an unattended replacement"
|
||||||
|
|
||||||
|
kubectl -n "$NS" delete machine "$old" --wait=false
|
||||||
|
|
||||||
|
while [ "$(date +%s)" -lt "$deadline" ]; do
|
||||||
|
name=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].metadata.name}' 2>/dev/null || true)
|
||||||
|
phase=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.phase}' 2>/dev/null || true)
|
||||||
|
noderef=$(kubectl -n "$NS" get machine -o jsonpath='{.items[0].status.nodeRef.name}' 2>/dev/null || true)
|
||||||
|
if [ -n "$name" ] && [ "$name" != "$old" ] && [ "$phase" = "Running" ] && [ -n "$noderef" ]; then
|
||||||
|
log "replacement $name reached Running with nodeRef $noderef"
|
||||||
|
return 0
|
||||||
|
fi
|
||||||
|
sleep 15
|
||||||
|
done
|
||||||
|
|
||||||
|
kubectl -n "$NS" get machine,machinedeployment -o wide >&2 || true
|
||||||
|
die "no replacement reached Running with a nodeRef within 15m: the departed Node was not removed, or the replacement never registered"
|
||||||
|
}
|
||||||
|
|
||||||
|
# Runs from the suite's catch. Cheap reads first, so a truncated dump still
|
||||||
|
# carries the ones that discriminate between failure modes.
|
||||||
|
dump_diagnostics() {
|
||||||
|
log "=== Machines and their infrastructure ==="
|
||||||
|
kubectl -n "$NS" get machine,proxmoxmachine,machinedeployment,proxmoxcluster -o wide || true
|
||||||
|
log "=== capmox controller ==="
|
||||||
|
kubectl -n cozy-cluster-api logs -l control-plane=controller-manager --tail=80 \
|
||||||
|
--selector=cluster.x-k8s.io/provider=infrastructure-proxmox || true
|
||||||
|
log "=== VMs on the hypervisor ==="
|
||||||
|
pve "qm list" || true
|
||||||
|
if [ -s "$TENANT_KUBECONFIG" ]; then
|
||||||
|
log "=== tenant nodes and pods ==="
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get nodes -o wide || true
|
||||||
|
KUBECONFIG="$TENANT_KUBECONFIG" kubectl get pods -A -o wide || true
|
||||||
|
fi
|
||||||
|
}
|
||||||
|
|
||||||
|
case "${1:-}" in
|
||||||
|
render-tenant) render_tenant ;;
|
||||||
|
apply-tenant) apply_tenant ;;
|
||||||
|
linked-clone) check_linked_clone ;;
|
||||||
|
tenant-node) check_tenant_node ;;
|
||||||
|
pvc-round-trip) check_pvc_round_trip ;;
|
||||||
|
disk-freed) wait_disk_freed "${2:?disk-freed needs <storage>:<volume>}" ;;
|
||||||
|
replacement) check_replacement_converges ;;
|
||||||
|
diagnostics) dump_diagnostics ;;
|
||||||
|
*) die "usage: $0 {render-tenant|apply-tenant|linked-clone|tenant-node|pvc-round-trip|disk-freed <storage>:<volume>|replacement|diagnostics}" ;;
|
||||||
|
esac
|
||||||
@@ -0,0 +1,135 @@
|
|||||||
|
# End-to-end coverage for the Proxmox substrate.
|
||||||
|
#
|
||||||
|
# Parked as .disabled on purpose. Suites are discovered with
|
||||||
|
# find hack/e2e-chainsaw -mindepth 2 -maxdepth 2 -name chainsaw-test.yaml
|
||||||
|
# so this one is registered nowhere and cannot be selected by the normal lanes.
|
||||||
|
# That is the point: it needs a Proxmox hypervisor, and the runners those lanes
|
||||||
|
# use will never have one. .github/workflows/e2e-proxmox.yaml drops the suffix
|
||||||
|
# on a self-hosted runner that does.
|
||||||
|
#
|
||||||
|
# What this asserts is what NO other suite can, and every item is a bug this
|
||||||
|
# integration actually hit rather than a shape someone imagined:
|
||||||
|
#
|
||||||
|
# ProxmoxCluster Ready — a missing IPAM provider, or the managed-by
|
||||||
|
# annotation that stops CAPI setting an ownerRef
|
||||||
|
# linked-clone boot disk — 8 KiB per worker against 10.2 GiB, and nothing
|
||||||
|
# fails when it regresses
|
||||||
|
# Machine Running + nodeRef — capmox writes proxmox://<UUID>, the CCM's default
|
||||||
|
# mode writes proxmox://<region>/<vmid>, and CAPI
|
||||||
|
# pairs on an exact match
|
||||||
|
# node Ready + region label — the CSI node plugin exits fatally without it
|
||||||
|
# PVC bound AND on the host — Bound is not proof that a disk exists
|
||||||
|
# unattended replacement — the controller that removes a departed Node
|
||||||
|
# must not itself depend on that Node
|
||||||
|
#
|
||||||
|
# It does NOT install Cozystack. The upstream lanes build a sandbox per run;
|
||||||
|
# this one runs against an existing cluster and an existing hypervisor, which is
|
||||||
|
# a weaker guarantee and is stated rather than implied.
|
||||||
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||||
|
kind: Test
|
||||||
|
metadata:
|
||||||
|
name: kubernetes-proxmox
|
||||||
|
spec:
|
||||||
|
# Teardown is the finally below, which must run whether the suite passed or
|
||||||
|
# failed: a leaked tenant leaves VMs on the hypervisor.
|
||||||
|
skipDelete: true
|
||||||
|
catch:
|
||||||
|
- events: {}
|
||||||
|
- script:
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh diagnostics
|
||||||
|
steps:
|
||||||
|
- name: tenant-and-cluster
|
||||||
|
try:
|
||||||
|
# Rendered by run-proxmox.sh rather than applied as is: every stand-specific
|
||||||
|
# value in tenant.yaml is a ${COZY_PVE_*} variable, the script supplies the
|
||||||
|
# defaults and expands them, and it refuses a fixture with a variable left
|
||||||
|
# in it. Another stand sets variables instead of editing the file.
|
||||||
|
- script:
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh apply-tenant
|
||||||
|
# Declarative where the object lives in this cluster, per the conventions in
|
||||||
|
# docs/agents/e2e-testing.md — the assert polls to its own timeout and
|
||||||
|
# reports a structured diff rather than a generic non-zero exit.
|
||||||
|
# Both the name and the namespace matter here. The resource definition
|
||||||
|
# prefixes the release, so the Kubernetes CR named k8s-e2e produces
|
||||||
|
# ProxmoxCluster/kubernetes-k8s-e2e, and it lives in the tenant's own
|
||||||
|
# namespace rather than the suite's default one. Asserted with the wrong
|
||||||
|
# pair this step simply spends its 10m and fails on a healthy stand.
|
||||||
|
- assert:
|
||||||
|
timeout: 10m
|
||||||
|
resource:
|
||||||
|
apiVersion: infrastructure.cluster.x-k8s.io/v1alpha1
|
||||||
|
kind: ProxmoxCluster
|
||||||
|
metadata:
|
||||||
|
name: kubernetes-k8s-e2e
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
status:
|
||||||
|
ready: true
|
||||||
|
- assert:
|
||||||
|
timeout: 20m
|
||||||
|
resource:
|
||||||
|
apiVersion: cluster.x-k8s.io/v1beta1
|
||||||
|
kind: Machine
|
||||||
|
metadata:
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
status:
|
||||||
|
phase: Running
|
||||||
|
# The controller half of the CSI driver and the cloud-controller-manager
|
||||||
|
# run in THIS cluster, not in the tenant; that is the property the storage
|
||||||
|
# PR is about, so the suite asserts it rather than only the tenant side.
|
||||||
|
- assert:
|
||||||
|
timeout: 5m
|
||||||
|
resource:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: kubernetes-k8s-e2e-pcsi-controller
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
status:
|
||||||
|
readyReplicas: 1
|
||||||
|
- assert:
|
||||||
|
timeout: 5m
|
||||||
|
resource:
|
||||||
|
apiVersion: apps/v1
|
||||||
|
kind: Deployment
|
||||||
|
metadata:
|
||||||
|
name: kubernetes-k8s-e2e-pccm
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
status:
|
||||||
|
readyReplicas: 1
|
||||||
|
|
||||||
|
- name: vm-shape-on-the-hypervisor
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
timeout: 3m
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh linked-clone
|
||||||
|
|
||||||
|
- name: tenant-node-usable
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
timeout: 10m
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh tenant-node
|
||||||
|
|
||||||
|
- name: storage-round-trip
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
timeout: 10m
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh pvc-round-trip
|
||||||
|
|
||||||
|
- name: replacement-converges-unattended
|
||||||
|
try:
|
||||||
|
- script:
|
||||||
|
timeout: 20m
|
||||||
|
content: cd ../../.. && hack/e2e-chainsaw/_lib/run-proxmox.sh replacement
|
||||||
|
finally:
|
||||||
|
- script:
|
||||||
|
content: |
|
||||||
|
set -eu
|
||||||
|
# keep_tenant is honoured here as well as in the workflow's own
|
||||||
|
# teardown, or the suite would delete what the operator asked to keep.
|
||||||
|
if [ "${COZY_KEEP_TENANT:-false}" = "true" ]; then
|
||||||
|
echo "COZY_KEEP_TENANT=true: leaving the tenant in place"
|
||||||
|
exit 0
|
||||||
|
fi
|
||||||
|
kubectl delete kubernetesnodes.apps.cozystack.io -n "${COZY_PROXMOX_NS:-tenant-e2eproxmox}" --all --ignore-not-found
|
||||||
|
kubectl delete kubernetes.apps.cozystack.io -n "${COZY_PROXMOX_NS:-tenant-e2eproxmox}" --all --ignore-not-found
|
||||||
|
kubectl delete tenant.apps.cozystack.io e2eproxmox -n tenant-root --ignore-not-found
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
# The tenant this suite builds and tears down. Its own namespace rather than the
|
||||||
|
# shared tenant-test: these objects create VMs on a hypervisor, and a leak is
|
||||||
|
# visible in `qm list` rather than only in the API.
|
||||||
|
#
|
||||||
|
# etcd: true is not optional. KamajiControlPlane.spec.dataStoreName reads
|
||||||
|
# _namespace.etcd, and the DataStore is created by the tenant's own etcd module
|
||||||
|
# (packages/extra/etcd/templates/datastore.yaml) — without it the control plane
|
||||||
|
# does not render at all.
|
||||||
|
# Tenant names are validated as lowercase letters and digits only — no dashes —
|
||||||
|
# so this is e2eproxmox rather than the e2e-proxmox the namespace would suggest.
|
||||||
|
apiVersion: apps.cozystack.io/v1alpha1
|
||||||
|
kind: Tenant
|
||||||
|
metadata:
|
||||||
|
name: e2eproxmox
|
||||||
|
namespace: tenant-root
|
||||||
|
spec:
|
||||||
|
etcd: true
|
||||||
|
host: ""
|
||||||
|
ingress: false
|
||||||
|
monitoring: false
|
||||||
|
seaweedfs: false
|
||||||
|
resourceQuotas: {}
|
||||||
|
---
|
||||||
|
apiVersion: apps.cozystack.io/v1alpha1
|
||||||
|
kind: Kubernetes
|
||||||
|
metadata:
|
||||||
|
name: k8s-e2e
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
spec:
|
||||||
|
version: "v1.35"
|
||||||
|
substrate: proxmox
|
||||||
|
proxmox:
|
||||||
|
# The stand's Proxmox serves its self-signed certificate; the chart verifies
|
||||||
|
# by default, so the suite says so rather than relying on a default.
|
||||||
|
insecure: true
|
||||||
|
# Empty means any node in the Proxmox cluster; the suite must not hardcode a
|
||||||
|
# hostname it would then only pass on one stand.
|
||||||
|
allowedNodes: []
|
||||||
|
# An off-cluster worker cannot resolve through the management cluster's
|
||||||
|
# CoreDNS, so the resolvers are its own. The chart refuses to render without
|
||||||
|
# them rather than writing an address that cannot answer.
|
||||||
|
dnsServers: ["8.8.8.8", "1.1.1.1"]
|
||||||
|
ipv4Config:
|
||||||
|
# Must not overlap the DHCP range, the CAPI IPAM pool or the MetalLB pool
|
||||||
|
# on the same L2 — four owners hand out addresses there and a collision is
|
||||||
|
# silent. Substituted per stand: run-proxmox.sh render-tenant expands the
|
||||||
|
# COZY_PVE_* variables in this file with envsubst and supplies the defaults
|
||||||
|
# itself, because GNU envsubst leaves a shell-style default in the text
|
||||||
|
# exactly as it is. A variable it does not know stays put and is refused.
|
||||||
|
addresses: ["${COZY_PVE_ADDRESSES}"]
|
||||||
|
gateway: "${COZY_PVE_GATEWAY}"
|
||||||
|
prefix: ${COZY_PVE_PREFIX}
|
||||||
|
ccm:
|
||||||
|
credentialsSecretName: proxmox-credentials
|
||||||
|
csi:
|
||||||
|
enabled: true
|
||||||
|
credentialsSecretName: proxmox-credentials
|
||||||
|
storageClasses:
|
||||||
|
- name: proxmox
|
||||||
|
# The same storage the PVC probe looks on, so the two cannot drift apart.
|
||||||
|
storage: "${COZY_PROXMOX_STORAGE}"
|
||||||
|
fstype: ext4
|
||||||
|
reclaimPolicy: Delete
|
||||||
|
---
|
||||||
|
apiVersion: apps.cozystack.io/v1alpha1
|
||||||
|
kind: KubernetesNodes
|
||||||
|
metadata:
|
||||||
|
name: md0
|
||||||
|
namespace: tenant-e2eproxmox
|
||||||
|
spec:
|
||||||
|
cluster: k8s-e2e
|
||||||
|
minReplicas: 1
|
||||||
|
maxReplicas: 1
|
||||||
|
instanceType: ""
|
||||||
|
resources:
|
||||||
|
cpu: 2
|
||||||
|
memory: 4Gi
|
||||||
|
substrate: proxmox
|
||||||
|
proxmox:
|
||||||
|
# capmox matches tags as a SET, not by containment: a template carrying four
|
||||||
|
# tags is not selected by naming one of them.
|
||||||
|
templateTags: [capmox-template, cozystack, talos, talos-v1.13.6]
|
||||||
|
# false is the default and the reason the suite asserts it on the hypervisor:
|
||||||
|
# a full clone costs the whole template disk per worker and breaks nothing.
|
||||||
|
# No `storage` with it: a linked clone lives on the template's storage, and
|
||||||
|
# the chart refuses the pair.
|
||||||
|
full: false
|
||||||
|
network:
|
||||||
|
bridge: "${COZY_PVE_BRIDGE}"
|
||||||
|
# The stand's resource pool, the one ACL path the tenant's tokens are
|
||||||
|
# scoped to; empty adds the workers to none.
|
||||||
|
pool: "${COZY_PVE_POOL}"
|
||||||
|
dnsServers: ["8.8.8.8", "1.1.1.1"]
|
||||||
Executable
+60
@@ -0,0 +1,60 @@
|
|||||||
|
#!/usr/bin/env bats
|
||||||
|
# The Proxmox e2e fixture is stand-agnostic through ${COZY_PVE_*} variables that
|
||||||
|
# hack/e2e-chainsaw/_lib/run-proxmox.sh expands with envsubst. GNU envsubst
|
||||||
|
# expands ${VAR} and nothing else: a ${VAR:-default} stays in the text as it is,
|
||||||
|
# set or not, and reaches the apiserver as a string where the schema wants an
|
||||||
|
# integer. So the defaults live in the script, the fixture carries bare
|
||||||
|
# variables, and the render refuses to hand out a fixture with one left in it.
|
||||||
|
|
||||||
|
REPO_ROOT="$(cd "$(dirname "${BATS_TEST_FILENAME:-$0}")/.." && pwd)"
|
||||||
|
FIXTURE="$REPO_ROOT/hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml"
|
||||||
|
RUNNER="$REPO_ROOT/hack/e2e-chainsaw/_lib/run-proxmox.sh"
|
||||||
|
|
||||||
|
@test "the fixture carries no shell-style default that envsubst would leave in place" {
|
||||||
|
! grep -qE '\$\{[A-Za-z_]+:[-=?+]' "$FIXTURE"
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "render-tenant expands every variable with the runner's defaults" {
|
||||||
|
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=teststore "$RUNNER" render-tenant)"
|
||||||
|
! printf '%s' "$out" | grep -q '\${'
|
||||||
|
printf '%s' "$out" | grep -qE '^ +prefix: 24$'
|
||||||
|
printf '%s' "$out" | grep -qE '^ +storage: "teststore"$'
|
||||||
|
printf '%s' "$out" | grep -qE '^ +bridge: "vmbr50"$'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "render-tenant honours a stand's own values and keeps prefix an integer" {
|
||||||
|
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s COZY_PVE_PREFIX=16 COZY_PVE_BRIDGE=vmbr1 "$RUNNER" render-tenant)"
|
||||||
|
printf '%s' "$out" | grep -qE '^ +prefix: 16$'
|
||||||
|
printf '%s' "$out" | grep -qE '^ +bridge: "vmbr1"$'
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "render-tenant passes the stand's resource pool through, and none by default" {
|
||||||
|
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s COZY_PVE_POOL=tenant-e2e "$RUNNER" render-tenant)"
|
||||||
|
printf '%s' "$out" | grep -qE '^ +pool: "tenant-e2e"$'
|
||||||
|
out="$(COZY_PVE_SSH=x COZY_PROXMOX_STORAGE=s "$RUNNER" render-tenant)"
|
||||||
|
printf '%s' "$out" | grep -qE '^ +pool: ""$'
|
||||||
|
}
|
||||||
|
|
||||||
|
# The PVC probe waits for the disk to leave the storage. A listing that fails
|
||||||
|
# must fail the check: read as empty, it would report a leaked disk as freed.
|
||||||
|
# (hack/cozytest.sh runs these as POSIX sh: no `run`, no [[ ]].)
|
||||||
|
@test "disk-freed fails when the storage cannot be listed, instead of calling the disk gone" {
|
||||||
|
stub="$(mktemp -d)"
|
||||||
|
printf '#!/bin/sh\necho "ssh: connect to host pve: Connection refused" >&2\nexit 255\n' > "$stub/ssh"
|
||||||
|
chmod +x "$stub/ssh"
|
||||||
|
rc=0
|
||||||
|
out="$(PATH="$stub:$PATH" COZY_PVE_SSH=root@pve COZY_PROXMOX_STORAGE=main-pool "$RUNNER" disk-freed main-pool:vm-1-pvc-x 2>&1)" || rc=$?
|
||||||
|
rm -rf "$stub"
|
||||||
|
[ "$rc" -ne 0 ]
|
||||||
|
case "$out" in *"cannot list main-pool"*) ;; *) echo "$out" >&2; exit 1 ;; esac
|
||||||
|
case "$out" in *"is gone"*) echo "$out" >&2; exit 1 ;; esac
|
||||||
|
}
|
||||||
|
|
||||||
|
@test "disk-freed reports the disk gone once a successful listing no longer shows it" {
|
||||||
|
stub="$(mktemp -d)"
|
||||||
|
printf '#!/bin/sh\necho "Volid Format Type Size VMID"\necho "main-pool:vm-2-pvc-y raw images 1073741824 2"\n' > "$stub/ssh"
|
||||||
|
chmod +x "$stub/ssh"
|
||||||
|
out="$(PATH="$stub:$PATH" COZY_PVE_SSH=root@pve COZY_PROXMOX_STORAGE=main-pool "$RUNNER" disk-freed main-pool:vm-1-pvc-x 2>&1)"
|
||||||
|
rm -rf "$stub"
|
||||||
|
case "$out" in *"is gone from main-pool"*) ;; *) echo "$out" >&2; exit 1 ;; esac
|
||||||
|
}
|
||||||
@@ -1569,9 +1569,16 @@ EOF
|
|||||||
# not a filter by itself either: what a creator can be
|
# not a filter by itself either: what a creator can be
|
||||||
# written in is decided beside the sweep, so that prose acquiring a fenced
|
# written in is decided beside the sweep, so that prose acquiring a fenced
|
||||||
# manifest reds nothing here.
|
# manifest reds nothing here.
|
||||||
|
#
|
||||||
|
# The Proxmox fixture is the second entry and is deliberately not governed by
|
||||||
|
# the rules above: those read run-kubernetes.sh's shell, and this one is a
|
||||||
|
# declarative manifest a chainsaw `apply:` block applies. It creates a cluster
|
||||||
|
# all the same, which is why it is named here rather than excluded — a reader
|
||||||
|
# asking "what creates a tenant cluster in this tree" gets both answers. The
|
||||||
|
# suite that applies it ships `.disabled`, so nothing runs it in CI today.
|
||||||
found="$(_kind_creators "$REPO_ROOT/hack/e2e-chainsaw" \
|
found="$(_kind_creators "$REPO_ROOT/hack/e2e-chainsaw" \
|
||||||
| sed "s|^$REPO_ROOT/||" | tr '\n' ' ')"
|
| sed "s|^$REPO_ROOT/||" | tr '\n' ' ')"
|
||||||
want="hack/e2e-chainsaw/_lib/run-kubernetes.sh "
|
want="hack/e2e-chainsaw/_lib/run-kubernetes.sh hack/e2e-chainsaw/kubernetes-proxmox/tenant.yaml "
|
||||||
if [ "$found" != "$want" ]; then
|
if [ "$found" != "$want" ]; then
|
||||||
echo "scripts under hack/e2e-chainsaw creating a Kubernetes CR: [$found]" >&2
|
echo "scripts under hack/e2e-chainsaw creating a Kubernetes CR: [$found]" >&2
|
||||||
echo "Check first whether the new file APPLIES the kind or only asserts on" >&2
|
echo "Check first whether the new file APPLIES the kind or only asserts on" >&2
|
||||||
|
|||||||
Reference in new issue
Block a user