mirror of
https://github.com/cozystack/cozystack.git
synced 2026-10-23 03:27:50 +09:00
fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#3427)
## What this PR does Closes the uncovered VM backup path from #3236. `hack/e2e-chainsaw/backup/` held only a disabled Chainsaw port of the old `backup.bats`, written against the retired API (`kind: VirtualMachine`, `BackupJob.storageRef`/`strategyRef`, `velero-strategy-default`), so it guarded nothing. This removes it, fixes the in-place restore of a standalone VMDisk in the backup controller, and adds two real round-trips in the `vminstance` suite that drive the `examples/backups/{vmi,vmdisk}` scripts, so the documented demo and the automated test stay in step. ### Controller: standalone VMDisk in-place restore `collectUnderlyingResources` in `internal/backupcontroller/velerostrategy_controller.go` returned nothing for any kind but VMInstance, so `prepareForRestore` did nothing for a standalone VMDisk: no HelmRelease suspend, no DataVolume delete, no `keepOriginalPVC` rename. An in-place VMDisk restore then raced Flux and CDI over the PVC name the data mover was repopulating. A standalone VMDisk now reports its own disk as its underlying resource, so the restore preparation already used for VMInstance disks suspends the `vm-disk-<name>` HelmRelease, deletes its DataVolume and renames the live PVC to `<name>-orig-<hash>` before Velero recreates it. Unit tests cover both the VMDisk and the VMInstance branch. A VMInstance is halted before its disks are touched, but a VMDisk target is not, and every VMInstance disk is a VMDisk. Renaming the PVC of a disk that a running VM mounts would leave the original `Terminating` behind `pvc-protection`; Velero's PVC restore action skips a PVC that still exists, so the RestoreJob would report `Succeeded` having moved nothing, and the VM would lose its disk on the next restart. An in-place VMDisk restore therefore fails before changing anything when a Pod that has not terminated mounts the disk's PVC. The message names the Pod and tells the user to stop the VM or restore the VMInstance instead. `TestPrepareForRestore_VMDiskInUse` covers a running mounting Pod (refused, PVC not renamed, HelmRelease not suspended), a finished Pod, a Pod on another disk, and a VMInstance target whose own launcher Pod must not refuse it. A VMDisk Backup taken before the disk was recorded as its underlying resource has none, so the disk is derived from the application name (`vm-disk-<name>`): such a restore is prepared and checked for a mounting Pod too. The per-disk `orLabelSelectors` stay a VMInstance concern: a VMDisk strategy already selects its disk, and Velero rejects a Backup or Restore carrying both `labelSelector` and `orLabelSelectors`, so appending them for a VMDisk would break a strategy written with `labelSelector`. When an in-place restore fails after preparation, nothing is rolled back. Resuming the disk's HelmRelease while its PVC is renamed and its DataVolume deleted would have CDI import a fresh disk under the original name, which looks healthy and hides that the data sits in the `-orig-` PVC. The failure message and a Warning event instead name the HelmReleases preparation suspended and the `-orig-` PVCs holding the original data, read back from the cluster so that a failure before a step ran does not name what it would have left, and say that a HelmRelease Velero already restored is resumed, so the disk has to be checked; `examples/backups/vmdisk/README.md` describes the recovery. Tests cover both selector paths, the pre-upgrade Backup, a failed Pod list (the restore requeues and suspends nothing), and all three failures after preparation: a `PartiallyFailed` Velero Restore, several Restores owning the job, and a Restore that cannot be created. Because preparation takes the target apart before Velero validates the Restore, the RestoreJob is refused before preparation, with nothing changed, when the Velero Backup is gone (expired or deleted) or not `Completed`. This is the same single check #4672 added, now run on every pass behind the preparation failure path, so a Backup that expires after preparation started names the leftovers too. A Velero Restore that ends in `FailedValidation` fails the RestoreJob like `Failed`, with Velero's validation errors and the leftovers in the message, instead of polling forever. A HelmRelease suspend or a VM halt that does not happen now stops preparation (requeue) rather than letting the rename and DataVolume delete run under a live release or VM, and a VMInstance's disks are renamed only once no Pod mounts them. A VMDisk held only by a Pod already being deleted is waited for rather than refused, for up to 30 minutes after the RestoreJob started, so a Pod stuck terminating on a lost node fails the job instead of keeping it `Running`. Each of these is pinned by a test that turns red when the guard is removed. Between renaming a kept disk's PVC and deleting its DataVolume, CDI can recreate an empty PVC under the original name for the still-present DataVolume. Velero then skips that PVC as already present ("PVC already exists. Skip restore this PVC."), the RestoreJob reports `Succeeded`, and the VM boots a blank disk. The round-trip's new DataDownload check caught this on a VMInstance restore, where the uid, label and `-orig-` checks had all passed. Once the DataVolume is gone (the wait #4672 added), preparation now deletes a PVC found under the original name (read uncached) before the Velero Restore is created; `TestPrepareForRestore_ClearsPVCRecreatedAfterRename` reproduces the recreation inside the DataVolume delete. Preparation spans several reconciles, so the RestoreJob records a `PreparationStarted` condition before it first changes the target; if recording it fails, preparation retries without changing anything, so a later failure cannot report leftovers that were never made. A refusal on a later pass (the Velero Backup expired, a VM started on the disk) then names what preparation left behind instead of saying nothing was changed, and a preparation still waiting 30 minutes after it started fails the same way instead of polling forever. The vmdisk script starts its consumer Pod only on a PVC that carries the restore label, so it cannot pin a PVC CDI recreated in the rename window. Now that a VMDisk target is prepared, `keepOriginalPVC: false` reaches it as well. Its DataVolume is deleted with orphan propagation, and the bound PVC is deleted and waited for before the Restore (the path #4672 added for VMInstance disks). Left in place, Velero would skip the PVC and the CDI webhook would refuse the DataVolume the resumed HelmRelease creates, leaving the disk's HelmRelease stuck. `TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC` pins it. ### `vminstance-2-backup-roundtrip` (un-gated in CI) A tiny cirros guest (`u1.nano`, 2Gi, in-place restore only) runs the current API end to end: Velero strategy, BackupClass, a booted VMInstance, an ad-hoc `BackupJob` and an in-place `RestoreJob`, each waited to `Succeeded` and failing fast on `Failed`. The VM must come back `Running`, and its disk PVC must have been created by the restore: a new `metadata.uid`, Velero's `velero.io/restore-name` label, and the `vm-disk-<name>-orig-<hash>` PVC that the rename leaves behind. The label alone does not prove the data moved, because with `existingResourcePolicy: update` Velero also stamps it on a PVC it only patched, and the patch path never reaches the data mover. An in-guest signal is not available on this platform: the KubeVirt CR disables serial-console logging cluster-wide and cirros ships no qemu-guest-agent. ### `vmdisk-2-backup-roundtrip` (un-gated in CI) The disk-only companion: import a 5Gi Ubuntu disk, back it up, restore it in place relying on the controller fix above, and require the same three signals (new uid, restore label, `-orig-` PVC), a `Completed` DataDownload for the restore that wrote bytes, and a Ready HelmRelease. On a `WaitForFirstConsumer` StorageClass, such as the `local` class the CI lane uses, the Velero data mover waits until a scheduled Pod has selected a node for the restored PVC (`WaitPVCConsumed` in Velero's generic restore exposer). A VMInstance gets that from its virt-launcher Pod; a detached disk has none, so its `RestoreJob` stays `Running` until the 30m data-mover prepare timeout fails it. The script therefore starts a restricted-profile `pause` Pod on the new PVC once Velero has created it, standing in for the VM that would attach the disk, and removes it after the restore. `examples/backups/vmdisk/README.md` documents this for operators restoring a detached disk. ### Harness details - The disk `storageClass` is an override (default `replicated`). The Chainsaw steps pass the lane's `COZY_E2E_STORAGE_CLASS` (`local` on the DRBD-off PR gate). - Each harness stands up a throwaway in-cluster BSL (`cozy-e2e-vmi`/`cozy-e2e-vmdisk`) at `seaweedfs-s3.tenant-root:8333` with `insecureSkipTLSVerify` and `checksumAlgorithm: ""` (the #3236 fixes), reusing the platform bucket and `cozy-backups-creds` under a distinct prefix, because the platform `cozy-default` BSL derives the external S3 ingress endpoint, unroutable in the CI cluster. - The script waits are bounded so that their worst case stays under the 40m op timeout (38m for the VMDisk step), and a hung wait trips the script's own named timeout rather than a bare `signal: killed`. - On failure, `catch` also describes the Velero `Restore`, `DataDownload` and `DataUpload` objects in `cozy-velero`. - `skipDelete` plus a `finally` that resumes the suspended HelmReleases, reclaims the `-orig-` PVC and its LINSTOR volume (the PV is flipped back to `Delete` first), and drains the Velero backups on the throwaway BSL, branching on kubectl's exit status, before deleting it. When the drain is not confirmed, the BSL is kept and the step fails, so the Velero Backups are not orphaned behind a green step. - HelmRelease waits use the shared `wait_hr_ready`, which fails fast on `Stalled`. After the VMDisk restore the script first waits for `observedGeneration` to reach the restored spec's `generation`, because the `Ready=True` from before the suspend would otherwise satisfy the wait at once. The `vmi` example is CI-drivable: `run-all.sh`/`cleanup.sh` are non-interactive, the VM is waited to boot before the backup, `volumeSnapshotLocations` is dropped from the demo strategies (`snapshotMoveData` routes data through the BSL), and `BACKUP_STORAGE_LOCATION` defaults to `cozy-default`. `select-e2e.sh` maps `examples/backups/{vmi,vmdisk}` edits to the `vminstance` suite, except `07-restore-to-copy.sh`, which the suite skips (`SKIP_RESTORE_TO_COPY=1`) and so stays docs-only. A `packages/system/velero` change selects the `vminstance` suite as well, since these are the only Tests that run a Velero backup and restore. ### Screenshots N/A — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff (backup controller restore preparation for VMDisk, shell scripts under `examples/backups/`, Chainsaw tests, `hack/select-e2e.sh` routing, a deleted stale suite, one doc line): no `packages/apps|extra` add/rename, no `values.schema.json`/enum/default changes, no CRD/ApplicationDefinition/label/annotation/namespace/metric changes. The controller change alters what a VMDisk `RestoreJob` does in the cluster, not any API surface a downstream repository consumes. `cozystack/examples` holds Terraform for bare nodes, unrelated to the in-repo `examples/` tree. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(backups): an in-place RestoreJob for a standalone VMDisk now suspends the disk's HelmRelease, deletes its DataVolume and (with keepOriginalPVC) renames the original PVC before Velero restores it, instead of racing Flux and CDI over the same PVC; without keepOriginalPVC the original PVC is deleted so Velero recreates it from the backup. Such a restore fails before changing anything when a running Pod, such as the launcher of a VM that uses the disk, mounts the disk's PVC: stop the VM or restore the VMInstance instead. If such a restore fails after the disk was prepared, its message and a Warning event name the HelmRelease preparation suspended and the PVC holding the original data. On a WaitForFirstConsumer StorageClass the data mover still starts only once a Pod consumes the restored PVC, for example when the disk is attached to a VMInstance. An in-place restore with keepOriginalPVC no longer reports Succeeded over an empty PVC that CDI recreated under the disk's name while the original was being kept aside. Adds Chainsaw backup/restore round-trips for VMInstance and standalone VMDisk that run a real Velero backup and restore in CI. ```
This commit is contained in:
28 files changed
+2542
-395
No files matched your search
@@ -70,7 +70,7 @@ Trap-based cleanup was the single biggest source of false failures in the BATS s
|
||||
|
||||
### 4. Do not mask cleanup or teardown failures
|
||||
|
||||
- Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`, `BackupStorageLocation`, and credentials secret in `cozy-velero` in the disabled `hack/e2e-chainsaw/backup/` suite) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run.
|
||||
- Resources a test creates that Chainsaw cannot reclaim — because a controller, not the test, owns them (e.g. the Velero `Backup`/`Restore` in `cozy-velero`, which the `vminstance` backup round-trips prune by deleting the cozystack `Backup` and `RestoreJob` CRs, whose finalizers then clean up the Velero objects asynchronously — the `BackupJob` has no finalizer and its `Backup` carries no owner reference, so deleting a `BackupJob` reclaims nothing — plus an explicit delete of the throwaway `BackupStorageLocation`, sequenced after the Velero `Backup` is gone so it does not race the finalizer) — must be pruned explicitly. Do not `|| true` over a stuck delete and leave stale state for the next run.
|
||||
- For nested tenants, tear down **child → parent with a hard wait for deletion between** each. Deleting the parent while a child is still uninstalling wedges the parent's cleanup Job on the child namespace, and both stuck uninstalls occupy helm-controller workers past the end of the test — starving whichever suite runs next. The kubernetes suites encode this ordering in `hack/e2e-chainsaw/_lib/run-kubernetes.sh`.
|
||||
- A test that creates a tenant `Kubernetes` cluster waits for that release to become **Ready** before the assertions it ends on — not before every read of the cluster, which is a different rule and a false one. The chart's post-install hook (`packages/apps/kubernetes/templates/oidc-rbac-job.yaml`) polls for the Kamaji-issued admin kubeconfig in the modes that have one, and that Secret is what the test needs to reach the tenant API, so bringing up a worker or waiting on a child release happens while the parent is still installing. Reading mid-install is fine. What the rule is about is the COMPLETION path: a run that asserts, passes and ends while the parent is still installing turns a green result into a red one in teardown. A failure path that diagnoses and exits mid-install does the same to the install, but that run is already red, so the second red costs noise rather than a wrong verdict — and the script has a number of such exits between the CR and the wait, which the guard does not and should not treat as violations.
|
||||
- A Helm release cannot be uninstalled while its install action is running, and for this kind the action can outlive the fast assertions — by how much depends on the mode, per the bullet below: `release.cozystack.io/helm-install-disable-wait` on the `ApplicationDefinition` drops the wait on the main resources, but the chart's blocking post-install hook still holds the install. A test that asserts and ends therefore hands the remainder of the install to teardown and is reported red on a step that tests nothing.
|
||||
|
||||
Executable
+44
@@ -0,0 +1,44 @@
|
||||
#!/bin/bash
|
||||
# Helper functions and variables for the VMDisk backup/restore demo.
|
||||
# Source this file in other scripts: source "$(dirname "$0")/00-helpers.sh"
|
||||
|
||||
# ANSI color codes
|
||||
export RED='\033[0;31m'
|
||||
export GREEN='\033[0;32m'
|
||||
export YELLOW='\033[1;33m'
|
||||
export MAGENTA='\033[0;35m'
|
||||
export CYAN='\033[0;36m'
|
||||
export WHITE='\033[1;37m'
|
||||
export NC='\033[0m'
|
||||
export BOLD='\033[1m'
|
||||
|
||||
# Default settings
|
||||
export NAMESPACE="${NAMESPACE:-tenant-root}"
|
||||
# The platform ships the SeaweedFS-backed BackupStorageLocation as cozy-default
|
||||
# (packages/system/backupstrategy-controller). Point the demo/e2e strategy at it
|
||||
# by default so the flow works out of the box on a backups-enabled cluster;
|
||||
# override for an external S3 whose BSL carries a different name.
|
||||
export BACKUP_STORAGE_LOCATION="${BACKUP_STORAGE_LOCATION:-cozy-default}"
|
||||
|
||||
# Logging functions (output to stderr to avoid polluting captured output)
|
||||
log_info() { echo -e "${CYAN}ℹ${NC} $*" >&2; }
|
||||
log_success() { echo -e "${GREEN}✔${NC} $*" >&2; }
|
||||
log_warning() { echo -e "${YELLOW}⚠${NC} $*" >&2; }
|
||||
log_error() { echo -e "${RED}✖${NC} $*" >&2; }
|
||||
log_step() { echo -e "\n${MAGENTA}${BOLD}▶ $*${NC}" >&2; }
|
||||
log_command() { echo -e "${WHITE} \$ $*${NC}" >&2; }
|
||||
log_substep() { echo -e "${CYAN} → $*${NC}" >&2; }
|
||||
|
||||
separator() {
|
||||
echo -e "\n${CYAN}────────────────────────────────────────────────────────────${NC}\n" >&2
|
||||
}
|
||||
|
||||
print_header() {
|
||||
local title="$1"
|
||||
echo -e "\n${MAGENTA}${BOLD}== ${title} ==${NC}\n" >&2
|
||||
}
|
||||
|
||||
# wait_for_field, wait_hr_ready and wait_deleted live in one file shared by
|
||||
# every backup walkthrough, so a fix to one reaches all of them.
|
||||
# shellcheck source-path=SCRIPTDIR source=../_lib/wait-helpers.sh
|
||||
source "$(dirname "${BASH_SOURCE[0]}")/../_lib/wait-helpers.sh"
|
||||
+63
@@ -0,0 +1,63 @@
|
||||
#!/bin/bash
|
||||
# Step 01: Create the Velero backup/restore strategy for a standalone VMDisk.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 1: Create VMDisk Velero strategy"
|
||||
|
||||
log_step "Creating Velero strategy 'vmdisk-backup-strategy'..."
|
||||
log_command "kubectl apply -f - (Velero strategy: vmdisk-backup-strategy)"
|
||||
|
||||
# A standalone VMDisk is just an HelmRelease + a DataVolume-backed PVC (+ its
|
||||
# configmaps), all labelled app.kubernetes.io/instance: vm-disk-<name>. Unlike a
|
||||
# VMInstance there is no VirtualMachine / VMI / launcher pod to capture. With
|
||||
# snapshotMoveData the CSI data mover streams the volume through the
|
||||
# BackupStorageLocation, so no VolumeSnapshotLocation is required (matching the
|
||||
# platform's own cozy-default-velero-vmdisk strategy).
|
||||
kubectl apply -f - <<'EOF'
|
||||
apiVersion: strategy.backups.cozystack.io/v1alpha1
|
||||
kind: Velero
|
||||
metadata:
|
||||
name: vmdisk-backup-strategy
|
||||
spec:
|
||||
template:
|
||||
restoreSpec:
|
||||
existingResourcePolicy: update
|
||||
includedNamespaces:
|
||||
- '{{ .Application.metadata.namespace }}'
|
||||
orLabelSelectors:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
|
||||
- matchLabels:
|
||||
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
|
||||
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
|
||||
includedResources:
|
||||
- helmreleases.helm.toolkit.fluxcd.io
|
||||
- persistentvolumeclaims
|
||||
- configmaps
|
||||
includeClusterResources: false
|
||||
spec:
|
||||
includedNamespaces:
|
||||
- '{{ .Application.metadata.namespace }}'
|
||||
orLabelSelectors:
|
||||
- matchLabels:
|
||||
app.kubernetes.io/instance: 'vm-disk-{{ .Application.metadata.name }}'
|
||||
- matchLabels:
|
||||
apps.cozystack.io/application.kind: '{{ .Application.kind }}'
|
||||
apps.cozystack.io/application.name: '{{ .Application.metadata.name }}'
|
||||
includedResources:
|
||||
- helmreleases.helm.toolkit.fluxcd.io
|
||||
- persistentvolumeclaims
|
||||
- configmaps
|
||||
includeClusterResources: false
|
||||
storageLocation: '{{ .Parameters.backupStorageLocationName }}'
|
||||
snapshotVolumes: true
|
||||
snapshotMoveData: true
|
||||
ttl: 720h0m0s
|
||||
itemOperationTimeout: 24h0m0s
|
||||
EOF
|
||||
|
||||
log_success "Velero strategy created"
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./02-create-backupclass.sh" >&2
|
||||
+33
@@ -0,0 +1,33 @@
|
||||
#!/bin/bash
|
||||
# Step 02: Create a BackupClass binding VMDisk to the Velero strategy.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 2: Create BackupClass"
|
||||
|
||||
log_step "Creating BackupClass 'vmdisk-velero'..."
|
||||
log_command "kubectl apply -f - (BackupClass: vmdisk-velero)"
|
||||
|
||||
# BackupClass is cluster-scoped; a single instance covers every namespace.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupClass
|
||||
metadata:
|
||||
name: vmdisk-velero
|
||||
spec:
|
||||
strategies:
|
||||
- application:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: VMDisk
|
||||
strategyRef:
|
||||
apiGroup: strategy.backups.cozystack.io
|
||||
kind: Velero
|
||||
name: vmdisk-backup-strategy
|
||||
parameters:
|
||||
backupStorageLocationName: ${BACKUP_STORAGE_LOCATION}
|
||||
EOF
|
||||
|
||||
log_success "BackupClass created"
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./03-create-vmdisk.sh" >&2
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
# Step 03: Create a standalone VMDisk and wait for it to be ready to back up.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 3: Create VMDisk 'backup-src'"
|
||||
|
||||
log_step "Creating VMDisk 'backup-src' in namespace $NAMESPACE..."
|
||||
log_info "Imports the Ubuntu Noble cloud image into a 5Gi ${VMI_DISK_STORAGE_CLASS:-replicated} disk"
|
||||
log_command "kubectl apply -f - (VMDisk: backup-src)"
|
||||
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: VMDisk
|
||||
metadata:
|
||||
name: backup-src
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
optical: false
|
||||
source:
|
||||
http:
|
||||
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
||||
storage: 5Gi
|
||||
# Default 'replicated' for a real DRBD cluster; overridable so the CI lane can
|
||||
# pass its own class (the PR gate runs DRBD-off and only renders 'local', so a
|
||||
# hardcoded 'replicated' never binds a PVC and the import hangs).
|
||||
storageClass: ${VMI_DISK_STORAGE_CLASS:-replicated}
|
||||
EOF
|
||||
|
||||
log_success "VMDisk created"
|
||||
|
||||
# Back up only a fully imported disk: the DataVolume import must finish and the
|
||||
# PVC bind before the backup runs, or Velero captures an empty/partial volume.
|
||||
log_step "Waiting for the VMDisk to import (HelmRelease Ready, DataVolume Ready, PVC Bound)..."
|
||||
wait_hr_ready vm-disk-backup-src "${IMPORT_WAIT:-600}"
|
||||
wait_for_field datavolume.cdi.kubevirt.io vm-disk-backup-src \
|
||||
'{.status.phase}' Succeeded "$NAMESPACE" "${IMPORT_WAIT:-600}"
|
||||
wait_for_field pvc vm-disk-backup-src '{.status.phase}' Bound "$NAMESPACE" 300
|
||||
|
||||
log_success "VMDisk 'backup-src' is imported and ready to back up"
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./04-create-backupjob.sh" >&2
|
||||
+41
@@ -0,0 +1,41 @@
|
||||
#!/bin/bash
|
||||
# Step 04: Back up the standalone VMDisk via a BackupJob.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 4: Create BackupJob"
|
||||
|
||||
log_step "Creating BackupJob 'vmdisk-backup' in namespace $NAMESPACE..."
|
||||
log_command "kubectl apply -f - (BackupJob: vmdisk-backup)"
|
||||
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
metadata:
|
||||
name: vmdisk-backup
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
applicationRef:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: VMDisk
|
||||
name: backup-src
|
||||
backupClassName: vmdisk-velero
|
||||
EOF
|
||||
|
||||
log_success "BackupJob created"
|
||||
|
||||
log_step "Waiting for BackupJob to complete..."
|
||||
# Default 1800s: the Velero CSI data mover copies the whole disk (5Gi) to S3.
|
||||
# Overridable (BACKUP_WAIT) so the CI harness can bound it under the Chainsaw op
|
||||
# timeout. Fail fast if the job flips to Failed instead of polling out the budget.
|
||||
wait_for_field backupjob vmdisk-backup '{.status.phase}' Succeeded "$NAMESPACE" "${BACKUP_WAIT:-1800}" Failed
|
||||
|
||||
BACKUP_NAME=$(kubectl -n "$NAMESPACE" get backupjob vmdisk-backup \
|
||||
-o jsonpath='{.status.backupRef.name}')
|
||||
[[ -n "$BACKUP_NAME" ]] || { log_error "BackupJob succeeded but reported no backupRef"; exit 1; }
|
||||
log_success "BackupJob completed; backup artefact: ${BACKUP_NAME}"
|
||||
|
||||
kubectl get backups -n "$NAMESPACE" >&2 || true
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./05-restore-in-place.sh" >&2
|
||||
+159
@@ -0,0 +1,159 @@
|
||||
#!/bin/bash
|
||||
# Step 05: Restore the standalone VMDisk in-place from its backup.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 5: Restore VMDisk In-Place"
|
||||
|
||||
PVC=vm-disk-backup-src
|
||||
CONSUMER=vm-disk-backup-src-restore-consumer
|
||||
|
||||
orig_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
|
||||
[[ -n "$orig_uid" ]] || { log_error "PVC $PVC has no uid before the restore"; exit 1; }
|
||||
log_info "PVC $PVC before restore: uid $orig_uid"
|
||||
|
||||
log_step "Creating RestoreJob 'vmdisk-restore-in-place' in namespace $NAMESPACE..."
|
||||
log_info "In-place restore: the original PVC is renamed and the disk is restored from backup"
|
||||
log_command "kubectl apply -f - (RestoreJob: vmdisk-restore-in-place)"
|
||||
|
||||
# The controller suspends the vm-disk-<name> HelmRelease, deletes its DataVolume
|
||||
# and (keepOriginalPVC) renames the live PVC to <name>-orig-<hash>; Velero then
|
||||
# creates a fresh PVC under the original name for the data mover to populate.
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: RestoreJob
|
||||
metadata:
|
||||
name: vmdisk-restore-in-place
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
backupRef:
|
||||
name: vmdisk-backup
|
||||
targetApplicationRef:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: VMDisk
|
||||
name: backup-src
|
||||
options:
|
||||
keepOriginalPVC: true # rename the original PVC to <name>-orig-<hash> before restoring
|
||||
EOF
|
||||
|
||||
log_success "RestoreJob created"
|
||||
|
||||
# On a WaitForFirstConsumer StorageClass the Velero data mover does not start
|
||||
# until a scheduled Pod has picked a node for the restored PVC (Velero's
|
||||
# WaitPVCConsumed). A VMInstance restore gets that from the virt-launcher Pod; a
|
||||
# detached disk has no consumer, so stand in for the VM that would attach it.
|
||||
# Only once the new PVC exists: a Pod mounting the original would pin it with
|
||||
# pvc-protection and block the rename.
|
||||
start_consumer() {
|
||||
local mode volume mount
|
||||
mode=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.spec.volumeMode}')
|
||||
if [[ "$mode" == "Block" ]]; then
|
||||
mount='volumeDevices: [{name: disk, devicePath: /dev/disk0}]'
|
||||
else
|
||||
mount='volumeMounts: [{name: disk, mountPath: /disk}]'
|
||||
fi
|
||||
volume="{name: disk, persistentVolumeClaim: {claimName: $PVC}}"
|
||||
log_info "Starting consumer Pod $CONSUMER for the restored PVC ($mode)"
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: v1
|
||||
kind: Pod
|
||||
metadata:
|
||||
name: ${CONSUMER}
|
||||
namespace: ${NAMESPACE}
|
||||
spec:
|
||||
securityContext:
|
||||
runAsNonRoot: true
|
||||
runAsUser: 65532
|
||||
seccompProfile:
|
||||
type: RuntimeDefault
|
||||
containers:
|
||||
- name: pause
|
||||
image: registry.k8s.io/pause:3.9
|
||||
securityContext:
|
||||
allowPrivilegeEscalation: false
|
||||
readOnlyRootFilesystem: true
|
||||
capabilities:
|
||||
drop: ["ALL"]
|
||||
${mount}
|
||||
volumes:
|
||||
- ${volume}
|
||||
EOF
|
||||
}
|
||||
|
||||
log_step "Waiting for RestoreJob to complete..."
|
||||
log_command "waiting for restorejob/vmdisk-restore-in-place {.status.phase} == 'Succeeded'"
|
||||
consumer_started=false
|
||||
elapsed=0
|
||||
timeout="${RESTORE_WAIT:-1800}"
|
||||
while true; do
|
||||
phase=$(kubectl -n "$NAMESPACE" get restorejob vmdisk-restore-in-place -o jsonpath='{.status.phase}' || true)
|
||||
[[ "$phase" == "Succeeded" ]] && break
|
||||
if [[ "$phase" == "Failed" ]]; then
|
||||
log_error "restorejob/vmdisk-restore-in-place reached terminal failure state 'Failed'"
|
||||
exit 1
|
||||
fi
|
||||
if [[ "$consumer_started" == false ]]; then
|
||||
# A PVC CDI recreated in the rename window has a new uid too but no
|
||||
# restore label; mounting it would pin it while the controller clears it.
|
||||
read -r uid restored < <(kubectl -n "$NAMESPACE" get pvc "$PVC" --ignore-not-found \
|
||||
-o jsonpath='{.metadata.uid} {.metadata.labels.velero\.io/restore-name}' || true) || true
|
||||
if [[ -n "${uid:-}" && "$uid" != "$orig_uid" && -n "${restored:-}" ]]; then
|
||||
start_consumer
|
||||
consumer_started=true
|
||||
fi
|
||||
fi
|
||||
if [[ $elapsed -ge $timeout ]]; then
|
||||
log_error "Timeout waiting for restorejob/vmdisk-restore-in-place (current: '$phase', expected: 'Succeeded', consumer started: $consumer_started)"
|
||||
exit 1
|
||||
fi
|
||||
sleep 5
|
||||
elapsed=$((elapsed + 5))
|
||||
echo -n "." >&2
|
||||
done
|
||||
log_success "restorejob/vmdisk-restore-in-place reached 'Succeeded'"
|
||||
|
||||
kubectl -n "$NAMESPACE" delete pod "$CONSUMER" --ignore-not-found --timeout=120s
|
||||
|
||||
# Neither the restore-name label nor a new uid proves the data moved: Velero's
|
||||
# existingResourcePolicy: update stamps the label on a PVC it only patched, and
|
||||
# CDI recreating the disk would also give it a new uid. The DataDownload check
|
||||
# below does; the uid and the -orig- PVC show the original was kept aside.
|
||||
log_step "Verifying the PVC was recreated from the backup..."
|
||||
wait_for_field pvc "$PVC" '{.status.phase}' Bound "$NAMESPACE" 120
|
||||
new_uid=$(kubectl -n "$NAMESPACE" get pvc "$PVC" -o jsonpath='{.metadata.uid}')
|
||||
[[ "$new_uid" != "$orig_uid" ]] || {
|
||||
log_error "PVC $PVC still has its pre-restore uid $orig_uid — the restore did not recreate it"
|
||||
exit 1
|
||||
}
|
||||
restore_label=$(kubectl -n "$NAMESPACE" get pvc "$PVC" \
|
||||
-o "jsonpath={.metadata.labels.velero\.io/restore-name}")
|
||||
[[ -n "$restore_label" ]] || {
|
||||
log_error "PVC $PVC carries no velero.io/restore-name label — Velero did not restore it"
|
||||
exit 1
|
||||
}
|
||||
all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name)
|
||||
[[ "$all_pvcs" == *"persistentvolumeclaim/${PVC}-orig-"* ]] || {
|
||||
log_error "no ${PVC}-orig-* PVC — keepOriginalPVC did not rename the original disk"
|
||||
exit 1
|
||||
}
|
||||
# The DataDownload carries the same restore-name label as the PVC; one that
|
||||
# Completed having written bytes is what shows the data mover filled the disk.
|
||||
downloads=$(kubectl -n cozy-velero get datadownloads.velero.io -l "velero.io/restore-name=${restore_label}" \
|
||||
-o jsonpath='{range .items[*]}{.status.phase} {.status.progress.bytesDone}{"\n"}{end}')
|
||||
grep -Eq '^Completed [1-9][0-9]*$' <<<"$downloads" || {
|
||||
log_error "no Completed DataDownload with data for restore ${restore_label} — the data mover did not restore $PVC (got: ${downloads:-none})"
|
||||
exit 1
|
||||
}
|
||||
|
||||
log_step "Waiting for the restored VMDisk to become Ready again..."
|
||||
# The HelmRelease was Ready before the controller suspended it, and that
|
||||
# condition outlives the restore; only once Flux has reconciled the restored
|
||||
# spec does Ready describe the restored release.
|
||||
hr_generation=$(kubectl -n "$NAMESPACE" get hr vm-disk-backup-src -o jsonpath='{.metadata.generation}')
|
||||
wait_for_field hr vm-disk-backup-src '{.status.observedGeneration}' "$hr_generation" "$NAMESPACE" 120
|
||||
wait_hr_ready vm-disk-backup-src 180
|
||||
|
||||
kubectl get restorejob vmdisk-restore-in-place -n "$NAMESPACE" -o wide >&2 || true
|
||||
log_success "Restore completed; VMDisk 'backup-src' recreated from backup (${restore_label}, uid ${new_uid}), Bound and Ready"
|
||||
@@ -0,0 +1,29 @@
|
||||
# VMDisk backup/restore example
|
||||
|
||||
This directory shows how to back up and restore a standalone Cozystack `VMDisk` (a HelmRelease plus a DataVolume-backed PVC, with no VMInstance attached) using the cluster's `Velero` backup strategy driver. Each `BackupJob` drives a Velero CSI-snapshot backup whose data mover (kopia) streams the LINSTOR block volume to S3; each in-place `RestoreJob` recreates the disk from that artefact.
|
||||
|
||||
The numbered scripts run in order (or via `run-all.sh`):
|
||||
|
||||
1. `01-create-strategy.sh` — a `Velero` strategy for the `VMDisk` kind.
|
||||
2. `02-create-backupclass.sh` — a `BackupClass` binding `VMDisk` to it.
|
||||
3. `03-create-vmdisk.sh` — a standalone `VMDisk`, waited until imported.
|
||||
4. `04-create-backupjob.sh` — an ad-hoc `BackupJob`, waited to `Succeeded`.
|
||||
5. `05-restore-in-place.sh` — an in-place `RestoreJob`, then asserts that the PVC was recreated from the backup: it has a new uid, carries Velero's `velero.io/restore-name` label, and the original survives as `<name>-orig-<hash>`.
|
||||
|
||||
`cleanup.sh` tears everything down non-interactively.
|
||||
|
||||
## In-place restore
|
||||
|
||||
The driver prepares a `VMDisk` restore target the same way it does a VMInstance disk: it suspends the `vm-disk-<name>` HelmRelease and deletes its DataVolume (so Flux/CDI do not recreate the PVC and race the data mover), and, with `keepOriginalPVC: true`, renames the live PVC to `<name>-orig-<hash>` so that Velero creates a fresh PVC under the original name. `cleanup.sh` resumes the HelmRelease and reclaims the renamed PVC.
|
||||
|
||||
A disk that a running VM mounts cannot be restored in place: the `RestoreJob` fails before it changes anything and names the Pod that holds the disk. Stop the VM, or restore the VMInstance instead.
|
||||
|
||||
If the restore fails after this preparation, nothing is rolled back. The failure message and a Warning event name the HelmRelease that preparation suspended and the `<name>-orig-<hash>` PVC that holds the original data. If Velero got as far as restoring the HelmRelease it is no longer suspended, and Flux may already have recreated the disk; with no restored PVC under the disk's name, CDI imports a fresh disk there. Either way, check what the disk holds, and restore again or bind the original PV back to a PVC under the disk's name if it is not what you expect.
|
||||
|
||||
## Detached disks and WaitForFirstConsumer
|
||||
|
||||
On a StorageClass with `volumeBindingMode: WaitForFirstConsumer`, such as the `local` class the CI lane uses, the Velero data mover does not start until a scheduled Pod has selected a node for the restored PVC. A VMInstance restore gets that from its virt-launcher Pod. A detached disk has no consumer, so its `RestoreJob` stays `Running` until something mounts the PVC, and fails once Velero's expose timeout passes. `05-restore-in-place.sh` therefore starts a minimal `pause` Pod on the restored PVC, standing in for the VM that would attach the disk, and deletes it once the restore has succeeded. On a real cluster, attaching the disk to a VMInstance does the same.
|
||||
|
||||
## Overrides
|
||||
|
||||
Environment variables (see `00-helpers.sh`) tune the flow: `NAMESPACE`, `BACKUP_STORAGE_LOCATION` (defaults to the platform `cozy-default` BSL), `VMI_DISK_STORAGE_CLASS` (defaults to `replicated`; set it to your cluster's class), and the `IMPORT_WAIT` / `BACKUP_WAIT` / `RESTORE_WAIT` budgets. `hack/e2e-chainsaw/vminstance/` drives this directory as the `vmdisk-2-backup-roundtrip` e2e test.
|
||||
Executable
+70
@@ -0,0 +1,70 @@
|
||||
#!/bin/bash
|
||||
# Clean up all resources created by the demo. Non-interactive: safe to call from
|
||||
# the e2e harness's finally block, and idempotent (a no-op when a gated-out run
|
||||
# created nothing). --ignore-not-found returns success for an absent resource but
|
||||
# does NOT mask a stuck delete: the delete still blocks and, under set -e, a
|
||||
# teardown that never settles fails this script instead of leaking state into the
|
||||
# next run (docs/agents/e2e-testing.md convention 4).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Cleanup Demo Resources"
|
||||
|
||||
# The controller suspends the VMDisk's HelmRelease before a restore and never
|
||||
# resumes it. A restore that gets as far as Velero clears it, but one that fails
|
||||
# earlier leaves it suspended, so resume it best-effort BEFORE deleting the
|
||||
# RestoreJob (same reasoning as examples/backups/vmi/cleanup.sh).
|
||||
if kubectl -n "$NAMESPACE" get hr vm-disk-backup-src >/dev/null 2>&1; then
|
||||
log_substep "Resuming HelmRelease vm-disk-backup-src (no-op if not suspended)..."
|
||||
kubectl -n "$NAMESPACE" patch hr vm-disk-backup-src --type=merge \
|
||||
-p '{"spec":{"suspend":false}}' >/dev/null || true
|
||||
fi
|
||||
|
||||
# Namespaced request objects first. Deleting the cozystack BackupJob/RestoreJob
|
||||
# and Backup CRs cascades to the Velero Backup/Restore the controller created.
|
||||
log_step "Deleting RestoreJob..."
|
||||
kubectl delete restorejob vmdisk-restore-in-place -n "$NAMESPACE" --ignore-not-found
|
||||
|
||||
log_step "Deleting BackupJob and Backup..."
|
||||
kubectl delete backupjob vmdisk-backup -n "$NAMESPACE" --ignore-not-found
|
||||
kubectl delete backup vmdisk-backup -n "$NAMESPACE" --ignore-not-found
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting VMDisk and its PVC..."
|
||||
kubectl delete pod vm-disk-backup-src-restore-consumer -n "$NAMESPACE" --ignore-not-found
|
||||
kubectl delete vmdisk backup-src -n "$NAMESPACE" --ignore-not-found
|
||||
kubectl delete pvc -n "$NAMESPACE" \
|
||||
-l app.kubernetes.io/instance=vm-disk-backup-src --ignore-not-found
|
||||
|
||||
# keepOriginalPVC renamed the pre-restore PVC to <name>-orig-<hash> with a bare
|
||||
# ObjectMeta (no labels/owner) and flipped its PV to Retain, so the label delete
|
||||
# above misses it. Reclaim it by name and flip the PV back to Delete BEFORE
|
||||
# removing the PVC so the CSI provisioner reclaims the LINSTOR volume rather than
|
||||
# stranding it. Let the lookup and patch fail the script (see the vmi cleanup).
|
||||
if ! all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name 2>&1); then
|
||||
log_error "cannot list PVCs to reclaim orphaned disks: $all_pvcs"
|
||||
exit 1
|
||||
fi
|
||||
orig_pvcs=$(printf '%s\n' "$all_pvcs" | grep -E '/vm-disk-backup-src-orig-' || true)
|
||||
for pvc in $orig_pvcs; do
|
||||
pv=$(kubectl -n "$NAMESPACE" get "$pvc" -o jsonpath='{.spec.volumeName}')
|
||||
log_substep "Reclaiming orphaned ${pvc#*/} and its LINSTOR volume..."
|
||||
if [[ -n "$pv" ]]; then
|
||||
kubectl patch pv "$pv" --type=merge \
|
||||
-p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' >/dev/null
|
||||
fi
|
||||
kubectl -n "$NAMESPACE" delete "$pvc" --ignore-not-found
|
||||
done
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting BackupClass and Velero strategy..."
|
||||
kubectl delete backupclass vmdisk-velero --ignore-not-found
|
||||
kubectl delete velero.strategy.backups.cozystack.io vmdisk-backup-strategy --ignore-not-found
|
||||
|
||||
separator
|
||||
|
||||
log_success "Cleanup complete"
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/bin/bash
|
||||
# Convenience runner + e2e harness for the standalone VMDisk backup/restore demo.
|
||||
#
|
||||
# Runs the numbered steps 01..05 a human reads, in order, non-interactively,
|
||||
# stopping on the first failure (set -e). This is the SAME sequence the docs walk
|
||||
# through, so the documented demo and the automated test can never drift.
|
||||
#
|
||||
# Flow: Velero strategy -> BackupClass -> a standalone VMDisk (waits imported) ->
|
||||
# ad-hoc BackupJob (waits Succeeded) -> in-place RestoreJob (waits Succeeded, then
|
||||
# waits for the restored disk to be Bound + Ready).
|
||||
#
|
||||
# Override NAMESPACE / BACKUP_STORAGE_LOCATION via the environment; see
|
||||
# 00-helpers.sh. hack/e2e-chainsaw/vminstance/ drives this file as the
|
||||
# vmdisk-2-backup-roundtrip e2e test, run un-gated in CI.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "VMDisk Backup & Restore Demo"
|
||||
log_info "Namespace: ${NAMESPACE} BackupStorageLocation: ${BACKUP_STORAGE_LOCATION}"
|
||||
|
||||
SCRIPTS=(
|
||||
"01-create-strategy.sh"
|
||||
"02-create-backupclass.sh"
|
||||
"03-create-vmdisk.sh"
|
||||
"04-create-backupjob.sh"
|
||||
"05-restore-in-place.sh"
|
||||
)
|
||||
|
||||
for script in "${SCRIPTS[@]}"; do
|
||||
if [[ -x "$SCRIPT_DIR/$script" ]]; then
|
||||
"$SCRIPT_DIR/$script"
|
||||
separator
|
||||
else
|
||||
log_error "Script not found or not executable: $script"
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
print_header "Demo Complete"
|
||||
log_success "All steps completed successfully!"
|
||||
log_info "To clean up resources: ./cleanup.sh"
|
||||
@@ -15,7 +15,11 @@ export BOLD='\033[1m'
|
||||
|
||||
# Default settings
|
||||
export NAMESPACE="${NAMESPACE:-tenant-root}"
|
||||
export BACKUP_STORAGE_LOCATION="${BACKUP_STORAGE_LOCATION:-default}"
|
||||
# The platform ships the SeaweedFS-backed BackupStorageLocation as cozy-default
|
||||
# (packages/system/backupstrategy-controller). Point the demo/e2e strategies at
|
||||
# it by default so the flow works out of the box on a backups-enabled cluster;
|
||||
# override for an external S3 whose BSL carries a different name.
|
||||
export BACKUP_STORAGE_LOCATION="${BACKUP_STORAGE_LOCATION:-cozy-default}"
|
||||
|
||||
# Logging functions (output to stderr to avoid polluting captured output)
|
||||
log_info() {
|
||||
|
||||
@@ -70,8 +70,11 @@ spec:
|
||||
- controllerrevisions.apps
|
||||
includeClusterResources: false
|
||||
storageLocation: '{{ .Parameters.backupStorageLocationName }}'
|
||||
volumeSnapshotLocations:
|
||||
- '{{ .Parameters.backupStorageLocationName }}'
|
||||
# No volumeSnapshotLocations: with snapshotMoveData the CSI data mover
|
||||
# uploads the volume data through the BackupStorageLocation, so a VSL of
|
||||
# the same name is not required (and referencing a non-existent one fails
|
||||
# the backup). The platform's own cozy-default-velero-vminstance strategy
|
||||
# omits it for the same reason.
|
||||
snapshotVolumes: true
|
||||
snapshotMoveData: true
|
||||
ttl: 720h0m0s
|
||||
@@ -123,8 +126,8 @@ spec:
|
||||
- configmaps
|
||||
includeClusterResources: false
|
||||
storageLocation: '{{ .Parameters.backupStorageLocationName }}'
|
||||
volumeSnapshotLocations:
|
||||
- '{{ .Parameters.backupStorageLocationName }}'
|
||||
# See the VMInstance strategy above: snapshotMoveData routes volume data
|
||||
# through the BackupStorageLocation, so no VolumeSnapshotLocation is needed.
|
||||
snapshotVolumes: true
|
||||
snapshotMoveData: true
|
||||
ttl: 720h0m0s
|
||||
|
||||
@@ -7,8 +7,13 @@ source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 3: Create VMDisk"
|
||||
|
||||
# Image + size are overridable so the same flow can run a heavy ubuntu demo or a
|
||||
# tiny cirros disk for a lightweight e2e (see hack/e2e-chainsaw/vminstance).
|
||||
VMI_IMAGE_URL="${VMI_IMAGE_URL:-https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img}"
|
||||
VMI_DISK_STORAGE="${VMI_DISK_STORAGE:-20Gi}"
|
||||
|
||||
log_step "Creating VMDisk 'ubuntu-source' in namespace $NAMESPACE..."
|
||||
log_info "This will download the Ubuntu Noble cloud image (~700MB)"
|
||||
log_info "Importing ${VMI_IMAGE_URL} into a ${VMI_DISK_STORAGE} disk"
|
||||
log_command "kubectl apply -f - (VMDisk: ubuntu-source)"
|
||||
|
||||
kubectl apply -f - <<EOF
|
||||
@@ -21,9 +26,12 @@ spec:
|
||||
optical: false
|
||||
source:
|
||||
http:
|
||||
url: https://cloud-images.ubuntu.com/noble/current/noble-server-cloudimg-amd64.img
|
||||
storage: 20Gi
|
||||
storageClass: replicated
|
||||
url: ${VMI_IMAGE_URL}
|
||||
storage: ${VMI_DISK_STORAGE}
|
||||
# Default 'replicated' for a real DRBD cluster; overridable so the CI lane can
|
||||
# pass its own class (the PR gate runs DRBD-off and only renders 'local', so a
|
||||
# hardcoded 'replicated' never binds a PVC and the import hangs).
|
||||
storageClass: ${VMI_DISK_STORAGE_CLASS:-replicated}
|
||||
EOF
|
||||
|
||||
log_success "VMDisk created"
|
||||
|
||||
@@ -10,6 +10,11 @@ print_header "Step 4: Create VMInstance"
|
||||
log_step "Creating VMInstance 'test' in namespace $NAMESPACE..."
|
||||
log_command "kubectl apply -f - (VMInstance: test)"
|
||||
|
||||
# Profile + instance type are overridable so the same flow can boot a heavy
|
||||
# ubuntu demo or a tiny cirros VM for a lightweight e2e (u1.nano = 1 vCPU/512Mi).
|
||||
VMI_INSTANCE_PROFILE="${VMI_INSTANCE_PROFILE:-ubuntu}"
|
||||
VMI_INSTANCE_TYPE="${VMI_INSTANCE_TYPE:-u1.medium}"
|
||||
|
||||
kubectl apply -f - <<EOF
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: VMInstance
|
||||
@@ -19,8 +24,8 @@ metadata:
|
||||
spec:
|
||||
disks:
|
||||
- name: ubuntu-source
|
||||
instanceProfile: ubuntu
|
||||
instanceType: "u1.medium"
|
||||
instanceProfile: ${VMI_INSTANCE_PROFILE}
|
||||
instanceType: "${VMI_INSTANCE_TYPE}"
|
||||
running: true
|
||||
sshKeys:
|
||||
#- <paste your ssh public key here>
|
||||
@@ -34,11 +39,24 @@ log_success "VMInstance created"
|
||||
|
||||
separator
|
||||
|
||||
# Wait for the VM to be fully booted before backing it up. Backing up a VM that
|
||||
# is still importing its disk or has not started yields an incomplete, useless
|
||||
# backup — so, like the etcd/postgres source steps, block on readiness here.
|
||||
# The Ubuntu cloud image download (~700MB) plus nested KubeVirt boot is slow,
|
||||
# hence the generous 900s bound.
|
||||
log_step "Waiting for the VMInstance to boot (HelmRelease + VirtualMachine Ready)..."
|
||||
wait_hr_ready vm-instance-test "${IMPORT_WAIT:-900}"
|
||||
log_command "kubectl -n $NAMESPACE wait virtualmachine.kubevirt.io/vm-instance-test --for=condition=Ready"
|
||||
kubectl -n "$NAMESPACE" wait virtualmachine.kubevirt.io/vm-instance-test \
|
||||
--for=condition=Ready --timeout="${VMI_BOOT_WAIT:-600}s"
|
||||
|
||||
separator
|
||||
|
||||
log_step "Verifying VMInstance..."
|
||||
log_command "kubectl get vminstance test -n $NAMESPACE"
|
||||
kubectl get vminstance test -n "$NAMESPACE"
|
||||
|
||||
separator
|
||||
|
||||
log_success "VMInstance is ready"
|
||||
log_success "VMInstance is running and ready to back up"
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./05-create-backupjob.sh"
|
||||
@@ -30,7 +30,11 @@ log_success "BackupJob created"
|
||||
separator
|
||||
|
||||
log_step "Waiting for BackupJob to complete..."
|
||||
wait_for_field backupjob test-backup '{.status.phase}' Succeeded "$NAMESPACE" 600
|
||||
# Default 1800s: the Velero CSI data mover copies the whole VM disk (20Gi) to S3,
|
||||
# far heavier than a database dump. Overridable (BACKUP_WAIT) so the CI harness
|
||||
# can bound it well under the Chainsaw op timeout on its tiny guest. Fail fast if
|
||||
# the job flips to Failed.
|
||||
wait_for_field backupjob test-backup '{.status.phase}' Succeeded "$NAMESPACE" "${BACKUP_WAIT:-1800}" Failed
|
||||
|
||||
separator
|
||||
|
||||
|
||||
@@ -7,6 +7,9 @@ source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "Step 6: Restore VMInstance In-Place"
|
||||
|
||||
orig_uid=$(kubectl -n "$NAMESPACE" get pvc vm-disk-ubuntu-source -o jsonpath='{.metadata.uid}')
|
||||
[[ -n "$orig_uid" ]] || { log_error "PVC vm-disk-ubuntu-source has no uid before the restore"; exit 1; }
|
||||
|
||||
log_step "Creating RestoreJob 'restore-in-place-test' in namespace $NAMESPACE..."
|
||||
log_info "In-place restore: the VM will be halted, PVCs renamed, and data restored from backup"
|
||||
log_command "kubectl apply -f - (RestoreJob: restore-in-place-test)"
|
||||
@@ -25,8 +28,8 @@ spec:
|
||||
kind: VMInstance
|
||||
name: test
|
||||
options:
|
||||
failIfTargetExists: true # if true, restore will fail when the target resource already exists
|
||||
keepOriginalPVC: true # renames original VMI PVC before restore to `<name>-orig-<hash>`, only for in-place restore
|
||||
failIfTargetExists: true # only enforced for restore-to-copy; a no-op for in-place, whose source is expected to exist
|
||||
keepOriginalPVC: true # renames original VMI PVC before restore to <name>-orig-<hash>, only for in-place restore
|
||||
keepOriginalIpAndMac: true # restores original IP and MAC address of VMI via OVN annotations
|
||||
EOF
|
||||
|
||||
@@ -35,7 +38,7 @@ log_success "RestoreJob created"
|
||||
separator
|
||||
|
||||
log_step "Waiting for RestoreJob to complete..."
|
||||
wait_for_field restorejob restore-in-place-test '{.status.phase}' Succeeded "$NAMESPACE" 600
|
||||
wait_for_field restorejob restore-in-place-test '{.status.phase}' Succeeded "$NAMESPACE" "${RESTORE_WAIT:-1800}" Failed
|
||||
|
||||
separator
|
||||
|
||||
@@ -45,5 +48,49 @@ kubectl get restorejob restore-in-place-test -n "$NAMESPACE" -o wide
|
||||
|
||||
separator
|
||||
|
||||
log_success "In-place restore completed successfully"
|
||||
# The RestoreJob reaching Succeeded means Velero re-created the objects; wait
|
||||
# for the restored VM to come back before checking where its disk came from.
|
||||
log_step "Waiting for the restored VMInstance to boot again..."
|
||||
log_command "kubectl -n $NAMESPACE wait virtualmachine.kubevirt.io/vm-instance-test --for=condition=Ready"
|
||||
kubectl -n "$NAMESPACE" wait virtualmachine.kubevirt.io/vm-instance-test \
|
||||
--for=condition=Ready --timeout="${VMI_BOOT_WAIT:-600}s"
|
||||
|
||||
# VirtualMachine Ready is the virt-launcher Pod's readiness, reported before the
|
||||
# guest runs, so it cannot tell a restored disk from a blank one, and this
|
||||
# platform offers no guest-side signal (serial-console logging is disabled
|
||||
# cluster-wide, the cirros guest has no qemu-guest-agent). Prove provenance on the
|
||||
# PVC instead. Neither the restore-name label nor a new uid proves the data
|
||||
# moved (existingResourcePolicy: update stamps the label on a PVC it only
|
||||
# patched, and CDI recreating the disk gives it a new uid); the DataDownload
|
||||
# check below does, and the uid and -orig- PVC show the original was kept aside.
|
||||
log_step "Verifying the VM booted off a disk the restore recreated from the backup..."
|
||||
wait_for_field pvc vm-disk-ubuntu-source '{.status.phase}' Bound "$NAMESPACE" 120
|
||||
new_uid=$(kubectl -n "$NAMESPACE" get pvc vm-disk-ubuntu-source -o jsonpath='{.metadata.uid}')
|
||||
[[ "$new_uid" != "$orig_uid" ]] || {
|
||||
log_error "PVC vm-disk-ubuntu-source still has its pre-restore uid $orig_uid — the restore did not recreate it"
|
||||
exit 1
|
||||
}
|
||||
restore_label=$(kubectl -n "$NAMESPACE" get pvc vm-disk-ubuntu-source \
|
||||
-o "jsonpath={.metadata.labels.velero\.io/restore-name}")
|
||||
[[ -n "$restore_label" ]] || {
|
||||
log_error "restored VM disk PVC carries no velero.io/restore-name label — Velero did not restore it"
|
||||
exit 1
|
||||
}
|
||||
all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name)
|
||||
[[ "$all_pvcs" == *"persistentvolumeclaim/vm-disk-ubuntu-source-orig-"* ]] || {
|
||||
log_error "no vm-disk-ubuntu-source-orig-* PVC — keepOriginalPVC did not rename the original disk"
|
||||
exit 1
|
||||
}
|
||||
# The DataDownload carries the same restore-name label as the PVC; one that
|
||||
# Completed having written bytes is what shows the data mover filled the disk.
|
||||
downloads=$(kubectl -n cozy-velero get datadownloads.velero.io -l "velero.io/restore-name=${restore_label}" \
|
||||
-o jsonpath='{range .items[*]}{.status.phase} {.status.progress.bytesDone}{"\n"}{end}')
|
||||
grep -Eq '^Completed [1-9][0-9]*$' <<<"$downloads" || {
|
||||
log_error "no Completed DataDownload with data for restore ${restore_label} — the data mover did not restore vm-disk-ubuntu-source (got: ${downloads:-none})"
|
||||
exit 1
|
||||
}
|
||||
|
||||
separator
|
||||
|
||||
log_success "In-place restore completed; VM disk recreated from backup (${restore_label}, uid ${new_uid}) and the VM is Running off it"
|
||||
echo -e "\n${GREEN}${BOLD}Next step:${NC} ./07-restore-to-copy.sh"
|
||||
@@ -40,7 +40,7 @@ spec:
|
||||
options: # runtime.RawExtension, typed based on targetApplicationRef and current controller implementation (for additional restore options)
|
||||
targetNamespace: ${TARGET_NAMESPACE} # when set to a different namespace, triggers cross-namespace restore via Velero namespaceMapping
|
||||
failIfTargetExists: true # if true, restore will fail when the target resource already exists
|
||||
keepOriginalPVC: false # renames original VMI PVC before restore to `<name>-orig-<hash>`, only for in-place restore
|
||||
keepOriginalPVC: false # renames original VMI PVC before restore to <name>-orig-<hash>, only for in-place restore
|
||||
keepOriginalIpAndMac: false # restores original IP and MAC address of VMI via OVN annotations
|
||||
EOF
|
||||
|
||||
@@ -49,7 +49,7 @@ log_success "RestoreJob created"
|
||||
separator
|
||||
|
||||
log_step "Waiting for RestoreJob to complete..."
|
||||
wait_for_field restorejob restore-to-copy-test '{.status.phase}' Succeeded "$NAMESPACE" 600
|
||||
wait_for_field restorejob restore-to-copy-test '{.status.phase}' Succeeded "$NAMESPACE" 1800 Failed
|
||||
|
||||
separator
|
||||
|
||||
|
||||
@@ -1,5 +1,10 @@
|
||||
#!/bin/bash
|
||||
# Clean up all resources created by the demo
|
||||
# Clean up all resources created by the demo. Non-interactive: safe to call from
|
||||
# the e2e harness's finally block, and idempotent (a no-op when a gated-out run
|
||||
# created nothing). --ignore-not-found returns success for an absent resource
|
||||
# but does NOT mask a stuck delete: the delete still blocks and, under set -e,
|
||||
# a teardown that never settles fails this script instead of leaking state into
|
||||
# the next run (docs/agents/e2e-testing.md convention 4).
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
@@ -9,62 +14,74 @@ TARGET_NAMESPACE="${TARGET_NAMESPACE:-tenant-root-copy}"
|
||||
|
||||
print_header "Cleanup Demo Resources"
|
||||
|
||||
log_warning "This script will delete all resources created during the demo"
|
||||
echo -e "\n${YELLOW}The following will be deleted:${NC}"
|
||||
echo " - RestoreJobs (restore-in-place-test, restore-to-copy-test)"
|
||||
echo " - BackupJob (test-backup) and associated Backup"
|
||||
echo " - VMInstance (test)"
|
||||
echo " - VMDisk (ubuntu-source)"
|
||||
echo " - BackupClass (velero)"
|
||||
echo " - Velero strategies (vminstance-strategy, vmdisk-strategy)"
|
||||
echo " - Target namespace ($TARGET_NAMESPACE)"
|
||||
echo ""
|
||||
|
||||
read -p "Continue? (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
log_info "Cancelled"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
separator
|
||||
# The driver suspends the VMInstance/VMDisk HelmReleases while a RestoreJob runs
|
||||
# and never resumes them (suspendHelmRelease only ever sets suspend=true), so an
|
||||
# aborted or completed restore leaves them suspended. Resume best-effort BEFORE
|
||||
# deleting the RestoreJobs — deleting a RestoreJob without resuming strands the
|
||||
# app suspended (same reasoning as examples/backups/postgres/cleanup.sh).
|
||||
for hr in vm-instance-test vm-disk-ubuntu-source; do
|
||||
if kubectl -n "$NAMESPACE" get hr "$hr" >/dev/null 2>&1; then
|
||||
log_substep "Resuming HelmRelease $hr (no-op if not suspended)..."
|
||||
kubectl -n "$NAMESPACE" patch hr "$hr" --type=merge \
|
||||
-p '{"spec":{"suspend":false}}' >/dev/null || true
|
||||
fi
|
||||
done
|
||||
|
||||
# Namespaced request objects first. Deleting the cozystack BackupJob/RestoreJob
|
||||
# and Backup CRs cascades to the Velero Backup/Restore the controller created.
|
||||
log_step "Deleting RestoreJobs..."
|
||||
kubectl delete restorejob restore-to-copy-test -n "$NAMESPACE" 2>/dev/null || log_warning "RestoreJob restore-to-copy-test not found"
|
||||
kubectl delete restorejob restore-in-place-test -n "$NAMESPACE" 2>/dev/null || log_warning "RestoreJob restore-in-place-test not found"
|
||||
|
||||
separator
|
||||
kubectl delete restorejob restore-to-copy-test restore-in-place-test \
|
||||
-n "$NAMESPACE" --ignore-not-found
|
||||
|
||||
log_step "Deleting BackupJob and Backup..."
|
||||
kubectl delete backupjob test-backup -n "$NAMESPACE" 2>/dev/null || log_warning "BackupJob test-backup not found"
|
||||
kubectl delete backup test-backup -n "$NAMESPACE" 2>/dev/null || log_warning "Backup test-backup not found"
|
||||
kubectl delete backupjob test-backup -n "$NAMESPACE" --ignore-not-found
|
||||
kubectl delete backup test-backup -n "$NAMESPACE" --ignore-not-found
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting VMInstance..."
|
||||
kubectl delete vminstance test -n "$NAMESPACE" 2>/dev/null || log_warning "VMInstance test not found"
|
||||
log_step "Deleting VMInstance and VMDisk..."
|
||||
kubectl delete vminstance test -n "$NAMESPACE" --ignore-not-found
|
||||
kubectl delete vmdisk ubuntu-source -n "$NAMESPACE" --ignore-not-found
|
||||
|
||||
log_step "Deleting VMDisk..."
|
||||
kubectl delete vmdisk ubuntu-source -n "$NAMESPACE" 2>/dev/null || log_warning "VMDisk ubuntu-source not found"
|
||||
# keepOriginalPVC renames the VMInstance's disk PVC to <name>-orig-<hash> with a
|
||||
# bare ObjectMeta (no labels, no ownerRefs) and flips its PV to Retain. Nothing
|
||||
# above reclaims it: the app delete does not own it and no label selector matches
|
||||
# it, so under skipDelete a LINSTOR volume and a Released PV leak on every run.
|
||||
# List loudly (a failed query must abort, not silently skip the reclaim it exists
|
||||
# for); grep's no-match stays tolerant.
|
||||
if ! all_pvcs=$(kubectl -n "$NAMESPACE" get pvc -o name 2>&1); then
|
||||
log_error "cannot list PVCs to reclaim orphaned disks: $all_pvcs"
|
||||
exit 1
|
||||
fi
|
||||
orig_pvcs=$(printf '%s\n' "$all_pvcs" | grep -E '/vm-disk-ubuntu-source-orig-' || true)
|
||||
for pvc in $orig_pvcs; do
|
||||
# Flip the retained PV back to Delete BEFORE removing the PVC, so releasing it
|
||||
# makes the CSI provisioner call DeleteVolume and reclaim the LINSTOR volume —
|
||||
# deleting the PV object alone would drop the record and strand the volume. Let
|
||||
# both the lookup and the patch fail the script: swallowing them would delete
|
||||
# the PVC anyway and strand the volume with no PVC left to find it by.
|
||||
pv=$(kubectl -n "$NAMESPACE" get "$pvc" -o jsonpath='{.spec.volumeName}')
|
||||
log_substep "Reclaiming orphaned ${pvc#*/} and its LINSTOR volume..."
|
||||
if [[ -n "$pv" ]]; then
|
||||
kubectl patch pv "$pv" --type=merge \
|
||||
-p '{"spec":{"persistentVolumeReclaimPolicy":"Delete"}}' >/dev/null
|
||||
fi
|
||||
kubectl -n "$NAMESPACE" delete "$pvc" --ignore-not-found
|
||||
done
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting BackupClass..."
|
||||
kubectl delete backupclass velero 2>/dev/null || log_warning "BackupClass velero not found"
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting Velero strategies..."
|
||||
kubectl delete velero.strategy.backups.cozystack.io vminstance-strategy 2>/dev/null || log_warning "Strategy vminstance-strategy not found"
|
||||
kubectl delete velero.strategy.backups.cozystack.io vmdisk-strategy 2>/dev/null || log_warning "Strategy vmdisk-strategy not found"
|
||||
# Cluster-scoped BackupClass + strategies.
|
||||
log_step "Deleting BackupClass and Velero strategies..."
|
||||
kubectl delete backupclass velero --ignore-not-found
|
||||
kubectl delete velero.strategy.backups.cozystack.io \
|
||||
vminstance-strategy vmdisk-strategy --ignore-not-found
|
||||
|
||||
separator
|
||||
|
||||
log_step "Deleting target namespace..."
|
||||
kubectl delete namespace "$TARGET_NAMESPACE" 2>/dev/null || log_warning "Namespace $TARGET_NAMESPACE not found"
|
||||
kubectl delete namespace "$TARGET_NAMESPACE" --ignore-not-found
|
||||
|
||||
separator
|
||||
|
||||
log_success "Cleanup complete"
|
||||
log_info "All demo resources have been deleted"
|
||||
echo -e "\n${GREEN}${BOLD}To re-run the demo:${NC} ./01-create-strategies.sh"
|
||||
@@ -1,15 +1,25 @@
|
||||
#!/bin/bash
|
||||
# Run the full VMInstance backup/restore demo sequentially
|
||||
# Convenience runner + e2e harness for the VMInstance backup/restore demo.
|
||||
#
|
||||
# Runs the numbered steps 01..07 a human reads, in order, non-interactively,
|
||||
# stopping on the first failure (set -e). This is the SAME sequence the docs
|
||||
# walk through, so the documented flow and the automated test can never drift.
|
||||
#
|
||||
# Flow: Velero strategies -> BackupClass -> VMDisk + VMInstance (waits until the
|
||||
# VM is booted) -> ad-hoc BackupJob (waits Succeeded) -> in-place RestoreJob
|
||||
# (waits Succeeded, waits for the VM to boot again) -> cross-namespace to-copy
|
||||
# RestoreJob (waits Succeeded).
|
||||
#
|
||||
# Override NAMESPACE / BACKUP_STORAGE_LOCATION / TARGET_NAMESPACE via the
|
||||
# environment; see 00-helpers.sh. hack/e2e-chainsaw/vminstance/ drives this file
|
||||
# as the vminstance-2-backup-roundtrip e2e test, run un-gated in CI.
|
||||
set -euo pipefail
|
||||
|
||||
SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
|
||||
source "$SCRIPT_DIR/00-helpers.sh"
|
||||
|
||||
print_header "VMInstance Backup & Restore Demo"
|
||||
|
||||
log_info "This script will run all demo steps sequentially"
|
||||
log_info "There will be a pause between steps for observation"
|
||||
echo ""
|
||||
log_info "Namespace: ${NAMESPACE} BackupStorageLocation: ${BACKUP_STORAGE_LOCATION}"
|
||||
|
||||
SCRIPTS=(
|
||||
"01-create-strategies.sh"
|
||||
@@ -18,29 +28,18 @@ SCRIPTS=(
|
||||
"04-create-vminstance.sh"
|
||||
"05-create-backupjob.sh"
|
||||
"06-restore-in-place.sh"
|
||||
"07-restore-to-copy.sh"
|
||||
)
|
||||
|
||||
echo -e "${CYAN}Demo steps:${NC}"
|
||||
for i in "${!SCRIPTS[@]}"; do
|
||||
echo " $((i+1)). ${SCRIPTS[$i]}"
|
||||
done
|
||||
echo ""
|
||||
|
||||
read -p "Run demo? (y/N): " -n 1 -r
|
||||
echo
|
||||
if [[ ! $REPLY =~ ^[Yy]$ ]]; then
|
||||
log_info "Cancelled"
|
||||
exit 0
|
||||
# Cross-namespace to-copy restore is the heaviest step (a second full VM in a
|
||||
# fresh namespace). SKIP_RESTORE_TO_COPY=1 drops it — used by the lightweight CI
|
||||
# round-trip, which proves backup + in-place restore is enough.
|
||||
if [[ "${SKIP_RESTORE_TO_COPY:-0}" != "1" ]]; then
|
||||
SCRIPTS+=("07-restore-to-copy.sh")
|
||||
fi
|
||||
|
||||
separator
|
||||
|
||||
for script in "${SCRIPTS[@]}"; do
|
||||
if [[ -x "$SCRIPT_DIR/$script" ]]; then
|
||||
"$SCRIPT_DIR/$script"
|
||||
separator
|
||||
wait_for_enter
|
||||
else
|
||||
log_error "Script not found or not executable: $script"
|
||||
exit 1
|
||||
@@ -50,13 +49,13 @@ done
|
||||
print_header "Demo Complete"
|
||||
|
||||
log_success "All steps completed successfully!"
|
||||
echo ""
|
||||
log_info "What was demonstrated:"
|
||||
echo " 1. Created Velero backup strategies for VMInstance and VMDisk"
|
||||
echo " 2. Created BackupClass binding strategies to application types"
|
||||
echo " 3. Provisioned a VMDisk and VMInstance"
|
||||
echo " 4. Created a backup of the VMInstance via BackupJob"
|
||||
echo " 5. Restored the VMInstance in-place"
|
||||
echo " 6. Restored the VMInstance to a copy in a different namespace"
|
||||
echo ""
|
||||
echo " 1. Created Velero backup strategies for VMInstance and VMDisk" >&2
|
||||
echo " 2. Created BackupClass binding strategies to application types" >&2
|
||||
echo " 3. Provisioned a VMDisk and VMInstance" >&2
|
||||
echo " 4. Created a backup of the VMInstance via BackupJob" >&2
|
||||
echo " 5. Restored the VMInstance in-place" >&2
|
||||
if [[ "${SKIP_RESTORE_TO_COPY:-0}" != "1" ]]; then
|
||||
echo " 6. Restored the VMInstance to a copy in a different namespace" >&2
|
||||
fi
|
||||
log_info "To clean up resources: ./cleanup.sh"
|
||||
@@ -1,21 +0,0 @@
|
||||
# Backup E2E (disabled / opt-in)
|
||||
|
||||
`chainsaw-test.yaml.disabled` is a Chainsaw port of the former `hack/e2e-apps/backup.bats.disabled`. It boots a VirtualMachine, drives a `BackupJob` through the default Velero strategy, and verifies the resulting Velero backup landed in S3.
|
||||
|
||||
It is **not** part of the automated suite — the `.disabled` suffix keeps it out of chainsaw's default `chainsaw-test.yaml` discovery, and `hack/select-e2e.sh` does not enumerate it as a selectable suite. This preserves the previous `backup.bats.disabled` behavior: the test is heavy (a full VM plus a Velero backup and S3 round-trip) and depends on backup infrastructure the standard e2e sandbox install does not guarantee.
|
||||
|
||||
## Preconditions
|
||||
|
||||
Run it only against a cozystack cluster that has backups enabled, i.e. with the `backup-controller`, `backupstrategy-controller`, and Velero installed, and a `velero-strategy-default` Velero strategy present (the install provides it). The test uses the `tenant-test` namespace, as configured in `hack/e2e-chainsaw/.chainsaw.yaml`.
|
||||
|
||||
## Running
|
||||
|
||||
```sh
|
||||
chainsaw test --test-file chainsaw-test.yaml.disabled hack/e2e-chainsaw/backup
|
||||
```
|
||||
|
||||
To enable it permanently, rename it to `chainsaw-test.yaml` — it then joins the full suite and becomes selectable by Test Impact Analysis.
|
||||
|
||||
## Cleanup
|
||||
|
||||
Chainsaw deletes the Bucket / VirtualMachine / BackupJob it applied. The Velero `Backup`, `BackupStorageLocation`, and credentials secret in the `cozy-velero` namespace are created by the backup-controller (not chainsaw), so prune them by hand after an opt-in run.
|
||||
@@ -1,245 +0,0 @@
|
||||
# Chainsaw port of hack/e2e-apps/backup.bats ("Create Backup for Virtual Machine").
|
||||
#
|
||||
# DISABLED ON PURPOSE. The filename ends in `.disabled` so chainsaw's default
|
||||
# test-file discovery (chainsaw-test.yaml) skips it and hack/select-e2e.sh does
|
||||
# not enumerate `backup` as a selectable suite — exactly matching the previous
|
||||
# `backup.bats.disabled` state. It is heavy (boots a VM, drives a Velero backup,
|
||||
# verifies S3) and depends on backup infrastructure that the standard e2e
|
||||
# sandbox install does not guarantee, so it stays opt-in.
|
||||
#
|
||||
# Run it by hand against a cozystack cluster that has the backup-controller,
|
||||
# backupstrategy-controller and Velero installed (see README.md):
|
||||
# chainsaw test --test-file chainsaw-test.yaml.disabled hack/e2e-chainsaw/backup
|
||||
#
|
||||
# Deviation from the bats: the bats re-applied the BackupJob/Velero CRDs and the
|
||||
# default Velero strategy defensively (its Step 0). That belongs to the install,
|
||||
# not the test — this port assumes `velero-strategy-default` already exists, as
|
||||
# it does on any cluster with backups enabled.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: backup
|
||||
spec:
|
||||
catch:
|
||||
- events: {}
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
name: test-backup-job
|
||||
- describe:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
name: virtual-machine-test-backup-vm
|
||||
- describe:
|
||||
apiVersion: objectstorage.k8s.io/v1alpha1
|
||||
kind: BucketClaim
|
||||
name: bucket-test-backup-bucket
|
||||
steps:
|
||||
# Step 1 — backing object store for the backup target.
|
||||
- name: create-bucket
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: Bucket
|
||||
metadata:
|
||||
name: test-backup-bucket
|
||||
spec: {}
|
||||
- assert:
|
||||
timeout: 6m
|
||||
resource:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: bucket-test-backup-bucket
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: objectstorage.k8s.io/v1alpha1
|
||||
kind: BucketClaim
|
||||
metadata:
|
||||
name: bucket-test-backup-bucket
|
||||
status:
|
||||
bucketReady: true
|
||||
- assert:
|
||||
resource:
|
||||
apiVersion: objectstorage.k8s.io/v1alpha1
|
||||
kind: BucketAccess
|
||||
metadata:
|
||||
name: bucket-test-backup-bucket
|
||||
status:
|
||||
accessGranted: true
|
||||
|
||||
# Step 2 — the application to back up: a small Ubuntu VM. Mirrors the
|
||||
# readiness chain of the vminstance suite (HR -> DataVolume -> PVC -> VM).
|
||||
- name: create-vm
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: apps.cozystack.io/v1alpha1
|
||||
kind: VirtualMachine
|
||||
metadata:
|
||||
name: test-backup-vm
|
||||
spec:
|
||||
external: false
|
||||
externalMethod: PortList
|
||||
externalPorts:
|
||||
- 22
|
||||
instanceType: "u1.medium"
|
||||
instanceProfile: ubuntu
|
||||
systemDisk:
|
||||
image: ubuntu
|
||||
storage: 5Gi
|
||||
storageClass: replicated
|
||||
gpus: []
|
||||
resources: {}
|
||||
sshKeys:
|
||||
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPht0dPk5qQ+54g1hSX7A6AUxXJW5T6n/3d7Ga2F8gTF test@test
|
||||
cloudInit: |
|
||||
#cloud-config
|
||||
users:
|
||||
- name: test
|
||||
shell: /bin/bash
|
||||
sudo: ['ALL=(ALL) NOPASSWD: ALL']
|
||||
groups: sudo
|
||||
ssh_authorized_keys:
|
||||
- ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIPht0dPk5qQ+54g1hSX7A6AUxXJW5T6n/3d7Ga2F8gTF test@test
|
||||
cloudInitSeed: ""
|
||||
- assert:
|
||||
timeout: 6m
|
||||
resource:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
metadata:
|
||||
name: virtual-machine-test-backup-vm
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
- assert:
|
||||
timeout: 5m
|
||||
resource:
|
||||
apiVersion: cdi.kubevirt.io/v1beta1
|
||||
kind: DataVolume
|
||||
metadata:
|
||||
name: virtual-machine-test-backup-vm
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
- assert:
|
||||
timeout: 4m
|
||||
resource:
|
||||
apiVersion: v1
|
||||
kind: PersistentVolumeClaim
|
||||
metadata:
|
||||
name: virtual-machine-test-backup-vm
|
||||
status:
|
||||
phase: Bound
|
||||
- assert:
|
||||
timeout: 5m
|
||||
resource:
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: VirtualMachine
|
||||
metadata:
|
||||
name: virtual-machine-test-backup-vm
|
||||
status:
|
||||
(conditions[?type == 'Ready']):
|
||||
- status: "True"
|
||||
|
||||
# Step 3 — trigger the backup and assert it reaches Succeeded with a backupRef.
|
||||
- name: create-backupjob
|
||||
try:
|
||||
- apply:
|
||||
resource:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
metadata:
|
||||
name: test-backup-job
|
||||
labels:
|
||||
backups.cozystack.io/triggered-by: e2e-test
|
||||
spec:
|
||||
applicationRef:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: VirtualMachine
|
||||
name: test-backup-vm
|
||||
storageRef:
|
||||
apiGroup: apps.cozystack.io
|
||||
kind: Bucket
|
||||
name: test-backup-bucket
|
||||
strategyRef:
|
||||
apiGroup: strategy.backups.cozystack.io
|
||||
kind: Velero
|
||||
name: velero-strategy-default
|
||||
- assert:
|
||||
timeout: 6m
|
||||
resource:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
metadata:
|
||||
name: test-backup-job
|
||||
status:
|
||||
phase: Succeeded
|
||||
(backupRef.name != null): true
|
||||
|
||||
# Step 4 — confirm the underlying Velero Backup completed and that the
|
||||
# backup objects actually landed in S3. The port-forward + mc verification is
|
||||
# imperative, so it stays a single script step (same shape as the bucket
|
||||
# suite); $NAMESPACE is the chainsaw-managed test namespace (tenant-test).
|
||||
- name: verify-velero-backup-in-s3
|
||||
try:
|
||||
- script:
|
||||
timeout: 8m
|
||||
content: |
|
||||
set -eu
|
||||
workdir=$(mktemp -d)
|
||||
trap '[ -n "${pf_pid:-}" ] && kill "$pf_pid" 2>/dev/null || true; rm -rf "$workdir"' EXIT
|
||||
|
||||
# Bucket S3 credentials for the verification client.
|
||||
kubectl -n "$NAMESPACE" get secret bucket-test-backup-bucket \
|
||||
-o jsonpath='{.data.BucketInfo}' | base64 -d > "$workdir/bucket.json"
|
||||
ACCESS_KEY=$(jq -r '.spec.secretS3.accessKeyID' "$workdir/bucket.json")
|
||||
SECRET_KEY=$(jq -r '.spec.secretS3.accessSecretKey' "$workdir/bucket.json")
|
||||
BUCKET_NAME=$(jq -r '.spec.bucketName' "$workdir/bucket.json")
|
||||
|
||||
# Find the Velero backup created for this job. Velero names it
|
||||
# "<namespace>-<backupjob>-<timestamp>"; wait for it to appear.
|
||||
echo "Waiting for the Velero backup to appear..."
|
||||
VELERO_BACKUP_NAME=$(timeout 120 sh -ec '
|
||||
while true; do
|
||||
for b in $(kubectl -n cozy-velero get backups.velero.io \
|
||||
-o jsonpath="{.items[*].metadata.name}" 2>/dev/null); do
|
||||
case "$b" in "'"$NAMESPACE"'-test-backup-job-"*) echo "$b"; exit 0;; esac
|
||||
done
|
||||
sleep 5
|
||||
done')
|
||||
[ -n "$VELERO_BACKUP_NAME" ]
|
||||
echo "Velero backup: $VELERO_BACKUP_NAME"
|
||||
|
||||
# Wait for the Velero Backup to reach a terminal phase, then assert Completed.
|
||||
timeout 300 sh -ec "
|
||||
until kubectl -n cozy-velero get backups.velero.io '$VELERO_BACKUP_NAME' \
|
||||
-o jsonpath='{.status.phase}' 2>/dev/null | grep -qE 'Completed|Failed|PartiallyFailed'; do
|
||||
sleep 5
|
||||
done"
|
||||
PHASE=$(kubectl -n cozy-velero get backups.velero.io "$VELERO_BACKUP_NAME" \
|
||||
-o jsonpath='{.status.phase}')
|
||||
[ "$PHASE" = "Completed" ]
|
||||
|
||||
# Port-forward to the S3 endpoint; cleaned up by the trap.
|
||||
kubectl port-forward service/seaweedfs-s3 -n tenant-root 8333:8333 >/dev/null 2>&1 &
|
||||
pf_pid=$!
|
||||
timeout 30 sh -ec 'until nc -z localhost 8333; do sleep 1; done'
|
||||
|
||||
mc alias set backup-s3 https://localhost:8333 "$ACCESS_KEY" "$SECRET_KEY" --insecure
|
||||
|
||||
# The backup directory must exist and hold at least the metadata objects.
|
||||
BACKUP_PATH="$BUCKET_NAME/backups/$VELERO_BACKUP_NAME"
|
||||
mc ls --insecure "backup-s3/$BACKUP_PATH/"
|
||||
[ "$(mc ls --insecure "backup-s3/$BACKUP_PATH/" | wc -l)" -gt 0 ]
|
||||
check:
|
||||
($error == null): true
|
||||
# Chainsaw deletes the Bucket / VirtualMachine / BackupJob CRs it applied during
|
||||
# cleanup. The Velero Backup + BackupStorageLocation + credentials secret in
|
||||
# cozy-velero are created by the backup-controller, not chainsaw, so an opt-in
|
||||
# run should prune them by hand afterwards (the bats teardown did this).
|
||||
@@ -386,3 +386,358 @@ spec:
|
||||
echo "FAIL: a dry-run eviction recorded an evacuation on VMI $vmi" >&2
|
||||
exit 1
|
||||
fi
|
||||
---
|
||||
# vminstance-2-backup-roundtrip — REAL Velero backup/restore round-trip for a
|
||||
# booted VMInstance, run un-gated in CI on a deliberately tiny VM.
|
||||
#
|
||||
# This is the in-CI proof of the cozy-default-velero-vminstance path: a running
|
||||
# VM's disk is CSI-snapshotted, moved to S3 by Velero's data mover (kopia), and
|
||||
# restored in place — after which the VM boots again. Booting does not tell a
|
||||
# restored disk from a blank one; the proof is on the disk PVC: a new uid,
|
||||
# Velero's restore-name label and the -orig- PVC the rename leaves behind.
|
||||
# It drives the validated example (examples/backups/vmi/run-all.sh) so the demo
|
||||
# and the test can never drift.
|
||||
#
|
||||
# Kept light enough for kind: a cirros guest (~20MB image, instanceProfile
|
||||
# cirros) on the smallest instance type (u1.nano = 1 vCPU / 512Mi) with a 2Gi
|
||||
# disk, and SKIP_RESTORE_TO_COPY=1 so only the in-place restore runs (the
|
||||
# cross-namespace to-copy restore stands up a second full VM — too heavy for CI
|
||||
# and redundant for proving the round-trip). cirros has no qemu-guest-agent, so
|
||||
# Velero takes a crash-consistent snapshot without a filesystem freeze, which is
|
||||
# fine for exercising the mechanics.
|
||||
#
|
||||
# S3 target: like vmdisk-2-backup-roundtrip, the harness stands up a throwaway
|
||||
# BSL cozy-e2e-vmi at the IN-CLUSTER seaweedfs endpoint (the platform
|
||||
# cozy-default BSL derives the external ingress endpoint, unroutable in kind)
|
||||
# with insecureSkipTLSVerify + checksumAlgorithm="" (the cozystack#3236 fixes),
|
||||
# reusing the platform bucket + cozy-backups-creds. Runs in tenant-root so the
|
||||
# data-mover DataUpload Pods reach seaweedfs over same-tenant egress.
|
||||
#
|
||||
# Teardown is explicit (skipDelete: true): cleanup.sh prunes the cluster-scoped
|
||||
# BackupClass/strategies and the cozy-velero Velero Backup/Restore, and the
|
||||
# harness deletes the throwaway BSL.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: vminstance-2-backup-roundtrip
|
||||
spec:
|
||||
namespace: tenant-root
|
||||
skipDelete: true
|
||||
catch:
|
||||
- events: {}
|
||||
- describe:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
name: vm-instance-test
|
||||
- describe:
|
||||
apiVersion: kubevirt.io/v1
|
||||
kind: VirtualMachineInstance
|
||||
name: vm-instance-test
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: RestoreJob
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: Backup
|
||||
- describe:
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
namespace: cozy-velero
|
||||
- describe:
|
||||
apiVersion: velero.io/v2alpha1
|
||||
kind: DataDownload
|
||||
namespace: cozy-velero
|
||||
- describe:
|
||||
apiVersion: velero.io/v2alpha1
|
||||
kind: DataUpload
|
||||
namespace: cozy-velero
|
||||
# Velero server + node-agent (the CSI data mover) run in cozy-velero; a failed
|
||||
# upload or a TLS handshake against seaweedfs leaves its error only there.
|
||||
- podLogs:
|
||||
namespace: cozy-velero
|
||||
selector: app.kubernetes.io/name=velero
|
||||
tail: 200
|
||||
steps:
|
||||
- name: run
|
||||
try:
|
||||
# 40m budget: build the in-cluster BSL, import the tiny cirros image + boot
|
||||
# the VM, a CSI-snapshot data-mover backup to S3, and an in-place restore
|
||||
# with a second boot — all fast on a 2Gi cirros guest (realistic ~15m). BSL
|
||||
# setup is in this step (not a separate one) so the finally below always tears
|
||||
# the BSL down, even if setup fails. run-all.sh is `set -e` and fails fast on
|
||||
# a terminal BackupJob/RestoreJob phase. The script waits are bounded (the
|
||||
# IMPORT/BOOT/BACKUP/RESTORE_WAIT values below plus fixed ones in the example
|
||||
# scripts) so their worst-case sum stays under this op timeout, and a hung
|
||||
# wait trips the script's own named "Timeout waiting for ..." rather than a
|
||||
# bare `signal: killed` (docs/agents/e2e-testing.md). Sized small on purpose:
|
||||
# this suite is execution.parallel:1 and shares the job's 215m cap with every
|
||||
# other Test in it and their finally blocks — an oversized op timeout lets the
|
||||
# job cap fire first and lose the report before any op timeout can.
|
||||
- script:
|
||||
timeout: 40m
|
||||
content: |
|
||||
set -eu
|
||||
# Build the in-cluster BSL the example will target (reuse the platform
|
||||
# bucket read off cozy-default + cozy-backups-creds).
|
||||
echo "Waiting for the platform cozy-default BSL to expose its bucket name..."
|
||||
timeout 180 sh -ec 'until kubectl -n cozy-velero get bsl cozy-default -o jsonpath="{.spec.objectStorage.bucket}" 2>/dev/null | grep -q .; do sleep 5; done'
|
||||
BUCKET=$(kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.spec.objectStorage.bucket}')
|
||||
REGION=$(kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.spec.config.region}')
|
||||
[ -n "$REGION" ] || REGION=us-east-1
|
||||
echo "Platform bucket: $BUCKET (region $REGION)"
|
||||
kubectl apply -f - <<YAML
|
||||
apiVersion: velero.io/v1
|
||||
kind: BackupStorageLocation
|
||||
metadata:
|
||||
name: cozy-e2e-vmi
|
||||
namespace: cozy-velero
|
||||
labels:
|
||||
cozystack.io/e2e: vminstance-backup
|
||||
spec:
|
||||
provider: aws
|
||||
default: false
|
||||
objectStorage:
|
||||
bucket: "$BUCKET"
|
||||
prefix: velero-e2e-vmi
|
||||
credential:
|
||||
name: cozy-backups-creds
|
||||
key: cloud
|
||||
config:
|
||||
region: "$REGION"
|
||||
s3ForcePathStyle: "true"
|
||||
s3Url: https://seaweedfs-s3.tenant-root.svc.cozy.local:8333
|
||||
insecureSkipTLSVerify: "true"
|
||||
checksumAlgorithm: ""
|
||||
YAML
|
||||
echo "Waiting for BSL cozy-e2e-vmi to validate Available..."
|
||||
timeout 180 sh -ec 'until [ "$(kubectl -n cozy-velero get bsl cozy-e2e-vmi -o jsonpath="{.status.phase}" 2>/dev/null)" = "Available" ]; do sleep 5; done'
|
||||
kubectl -n cozy-velero get bsl cozy-e2e-vmi
|
||||
|
||||
cd ../../..
|
||||
[ -x examples/backups/vmi/run-all.sh ] || { echo "vmi backup example scripts not found at examples/backups/vmi" >&2; exit 1; }
|
||||
# Tiny VM: cirros guest on u1.nano with a 2Gi disk; in-place restore only.
|
||||
# storageClass tracks the lane (COZY_E2E_STORAGE_CLASS, 'local' on the
|
||||
# DRBD-off PR gate); 'replicated' is unrenderable there and never binds.
|
||||
# Waits bounded well under the 40m op timeout above so a hung wait trips
|
||||
# the script's named fail-fast, not Chainsaw's killer.
|
||||
NAMESPACE="$NAMESPACE" BACKUP_STORAGE_LOCATION=cozy-e2e-vmi \
|
||||
VMI_DISK_STORAGE_CLASS="${COZY_E2E_STORAGE_CLASS:-replicated}" \
|
||||
VMI_IMAGE_URL=https://download.cirros-cloud.net/0.6.2/cirros-0.6.2-x86_64-disk.img \
|
||||
VMI_DISK_STORAGE=2Gi VMI_INSTANCE_PROFILE=cirros VMI_INSTANCE_TYPE=u1.nano \
|
||||
SKIP_RESTORE_TO_COPY=1 \
|
||||
IMPORT_WAIT=240 VMI_BOOT_WAIT=180 BACKUP_WAIT=300 RESTORE_WAIT=240 \
|
||||
examples/backups/vmi/run-all.sh
|
||||
check:
|
||||
($error == null): true
|
||||
finally:
|
||||
# Prune everything the harness created — the example cleanup (cluster-scoped
|
||||
# BackupClass/strategies, namespaced BackupJob/RestoreJob/Backup which cascade
|
||||
# to the Velero Backup/Restore in cozy-velero, the VMInstance/VMDisk), then
|
||||
# the throwaway BSL. Has teeth — a teardown that does not settle fails the step.
|
||||
- description: vmi backup-flow cleanup
|
||||
script:
|
||||
timeout: 15m
|
||||
content: |
|
||||
set -eu
|
||||
cd ../../..
|
||||
# Always tear down the harness-owned BSL, but keep a cleanup.sh failure
|
||||
# fatal (convention 4): capture its rc, then tear down, then re-raise.
|
||||
rc=0
|
||||
NAMESPACE="$NAMESPACE" examples/backups/vmi/cleanup.sh || rc=$?
|
||||
# cleanup.sh deletes the cozystack Backup, whose finalizer asks Velero to
|
||||
# delete the Velero Backup asynchronously. Deleting the BSL before that
|
||||
# finishes makes Velero fail the DeleteBackupRequest ("backup storage
|
||||
# location not found") and orphan the Velero Backup and its S3 objects.
|
||||
# Drain the Velero Backups on this BSL first, then delete it. Branch on
|
||||
# kubectl's exit status, not on empty stdout: a 403/timeout also yields
|
||||
# no lines and must not read as "drained" (that would delete the BSL
|
||||
# under the finalizer and orphan the objects this exists to protect).
|
||||
timeout 180 sh -ec '
|
||||
while out=$(kubectl -n cozy-velero get backups.velero.io \
|
||||
-o custom-columns=BSL:.spec.storageLocation --no-headers); do
|
||||
printf "%s\n" "$out" | grep -qx cozy-e2e-vmi || exit 0
|
||||
sleep 5
|
||||
done
|
||||
exit 3' && drain_rc=0 || drain_rc=$?
|
||||
if [ "$drain_rc" != 0 ]; then
|
||||
echo "error: Velero backups on cozy-e2e-vmi not confirmed drained (rc=$drain_rc); keeping the BSL so they are not orphaned" >&2
|
||||
exit 1
|
||||
fi
|
||||
kubectl -n cozy-velero delete bsl cozy-e2e-vmi --ignore-not-found
|
||||
exit "$rc"
|
||||
---
|
||||
# vmdisk-2-backup-roundtrip — REAL Velero backup/restore round-trip for a
|
||||
# standalone VMDisk, run un-gated in CI. This is the in-CI proof that the
|
||||
# cozy-default-velero-vmdisk path works end to end: a CSI VolumeSnapshot of the
|
||||
# LINSTOR block disk (the lane's storage class, 'local' on the DRBD-off PR gate),
|
||||
# moved to S3 by Velero's data mover (kopia), then restored back to a Bound disk.
|
||||
#
|
||||
# This is the disk-only companion to vminstance-2-backup-roundtrip above: same
|
||||
# Velero + CSI-data-mover + S3 machinery, but it captures only HelmRelease + PVC
|
||||
# (+ configmaps) with no VM at all — no nested boot, no guest-agent freeze — so
|
||||
# it isolates the block-volume backup path from anything VM-specific. Both run
|
||||
# un-gated in CI now.
|
||||
#
|
||||
# S3 target: the platform cozy-default BSL derives the EXTERNAL S3 ingress
|
||||
# endpoint, which is an unroutable placeholder inside kind (the same reason the
|
||||
# postgres round-trip targets the in-cluster endpoint). So the harness stands up
|
||||
# a throwaway BSL `cozy-e2e-vmdisk` pointing at the IN-CLUSTER seaweedfs endpoint
|
||||
# with insecureSkipTLSVerify (kind cannot trust the self-signed SeaweedFS CA) and
|
||||
# checksumAlgorithm="" (the velero-plugin-for-aws v1.14/AWS-SDK-v2 vs SeaweedFS
|
||||
# incompatibility) — i.e. the cozystack/cozystack#3236 fixes applied at the BSL.
|
||||
# It reuses the platform bucket + cozy-backups-creds (a distinct prefix isolates
|
||||
# the objects). The example scripts stay generic — they just consume
|
||||
# BACKUP_STORAGE_LOCATION.
|
||||
#
|
||||
# Runs in tenant-root: Velero's data-mover DataUpload Pods run in the source
|
||||
# namespace and reach tenant-root's seaweedfs over same-tenant egress. Teardown
|
||||
# is explicit (skipDelete: true) — cleanup.sh prunes the cluster-scoped
|
||||
# BackupClass/strategy and the cozy-velero Velero Backup/Restore, and the harness
|
||||
# deletes the throwaway BSL.
|
||||
apiVersion: chainsaw.kyverno.io/v1alpha1
|
||||
kind: Test
|
||||
metadata:
|
||||
name: vmdisk-2-backup-roundtrip
|
||||
spec:
|
||||
namespace: tenant-root
|
||||
skipDelete: true
|
||||
catch:
|
||||
- events: {}
|
||||
- describe:
|
||||
apiVersion: helm.toolkit.fluxcd.io/v2
|
||||
kind: HelmRelease
|
||||
name: vm-disk-backup-src
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: BackupJob
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: RestoreJob
|
||||
- describe:
|
||||
apiVersion: backups.cozystack.io/v1alpha1
|
||||
kind: Backup
|
||||
- describe:
|
||||
apiVersion: velero.io/v1
|
||||
kind: Restore
|
||||
namespace: cozy-velero
|
||||
- describe:
|
||||
apiVersion: velero.io/v2alpha1
|
||||
kind: DataDownload
|
||||
namespace: cozy-velero
|
||||
- describe:
|
||||
apiVersion: velero.io/v2alpha1
|
||||
kind: DataUpload
|
||||
namespace: cozy-velero
|
||||
# The CSI snapshot + data-mover work lands in these Velero CRs and the
|
||||
# node-agent/velero Pod logs; dump them on failure.
|
||||
- podLogs:
|
||||
namespace: cozy-velero
|
||||
selector: app.kubernetes.io/name=velero
|
||||
tail: 200
|
||||
steps:
|
||||
- name: run
|
||||
try:
|
||||
# 40m budget: build the in-cluster BSL, import the Ubuntu image into a 5Gi
|
||||
# disk (~150s proven by the vmdisk test above), a CSI-snapshot data-mover
|
||||
# backup to S3, and a restore. run-all.sh is `set -e` and fails fast on a
|
||||
# terminal BackupJob/RestoreJob phase rather than polling the budget out. As
|
||||
# with the vminstance step above, the worst-case sum of the script waits
|
||||
# (the BSL waits, IMPORT/BACKUP/RESTORE_WAIT below and the fixed ones in the
|
||||
# example scripts, the consumer Pod delete included) is 38m, under this op
|
||||
# timeout, so a hung wait trips the script's own named fail-fast, not
|
||||
# Chainsaw's bare `signal: killed`; and
|
||||
# it is sized small on purpose, since this suite (execution.parallel:1) shares
|
||||
# the 215m job cap with the other Tests and the finally blocks — an oversized
|
||||
# op timeout lets the cap fire first and lose the report. BSL setup is in this
|
||||
# step (not a separate one) so the finally below always tears the BSL down.
|
||||
- script:
|
||||
timeout: 40m
|
||||
content: |
|
||||
set -eu
|
||||
# Build the in-cluster BSL the example will target. Reuse the platform
|
||||
# bucket (read its COSI-assigned name off cozy-default, created once
|
||||
# the platform BucketClaim reconciles) + cozy-backups-creds.
|
||||
echo "Waiting for the platform cozy-default BSL to expose its bucket name..."
|
||||
timeout 180 sh -ec 'until kubectl -n cozy-velero get bsl cozy-default -o jsonpath="{.spec.objectStorage.bucket}" 2>/dev/null | grep -q .; do sleep 5; done'
|
||||
BUCKET=$(kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.spec.objectStorage.bucket}')
|
||||
REGION=$(kubectl -n cozy-velero get bsl cozy-default -o jsonpath='{.spec.config.region}')
|
||||
[ -n "$REGION" ] || REGION=us-east-1
|
||||
echo "Platform bucket: $BUCKET (region $REGION)"
|
||||
kubectl apply -f - <<YAML
|
||||
apiVersion: velero.io/v1
|
||||
kind: BackupStorageLocation
|
||||
metadata:
|
||||
name: cozy-e2e-vmdisk
|
||||
namespace: cozy-velero
|
||||
labels:
|
||||
cozystack.io/e2e: vmdisk-backup
|
||||
spec:
|
||||
provider: aws
|
||||
default: false
|
||||
objectStorage:
|
||||
bucket: "$BUCKET"
|
||||
prefix: velero-e2e-vmdisk
|
||||
credential:
|
||||
name: cozy-backups-creds
|
||||
key: cloud
|
||||
config:
|
||||
region: "$REGION"
|
||||
s3ForcePathStyle: "true"
|
||||
s3Url: https://seaweedfs-s3.tenant-root.svc.cozy.local:8333
|
||||
insecureSkipTLSVerify: "true"
|
||||
checksumAlgorithm: ""
|
||||
YAML
|
||||
echo "Waiting for BSL cozy-e2e-vmdisk to validate Available..."
|
||||
timeout 180 sh -ec '
|
||||
until [ "$(kubectl -n cozy-velero get bsl cozy-e2e-vmdisk -o jsonpath="{.status.phase}" 2>/dev/null)" = "Available" ]; do
|
||||
sleep 5
|
||||
done'
|
||||
kubectl -n cozy-velero get bsl cozy-e2e-vmdisk
|
||||
|
||||
cd ../../..
|
||||
[ -x examples/backups/vmdisk/run-all.sh ] || { echo "vmdisk backup example scripts not found at examples/backups/vmdisk" >&2; exit 1; }
|
||||
# storageClass tracks the lane (COZY_E2E_STORAGE_CLASS, 'local' on the
|
||||
# DRBD-off PR gate); waits bounded under the 40m op timeout above.
|
||||
NAMESPACE="$NAMESPACE" BACKUP_STORAGE_LOCATION=cozy-e2e-vmdisk \
|
||||
VMI_DISK_STORAGE_CLASS="${COZY_E2E_STORAGE_CLASS:-replicated}" \
|
||||
IMPORT_WAIT=240 BACKUP_WAIT=300 RESTORE_WAIT=300 \
|
||||
examples/backups/vmdisk/run-all.sh
|
||||
check:
|
||||
($error == null): true
|
||||
finally:
|
||||
# Prune the harness-created resources: the example cleanup (cluster-scoped
|
||||
# BackupClass/strategy, namespaced BackupJob/RestoreJob/Backup which cascade
|
||||
# to the Velero Backup/Restore in cozy-velero, the VMDisk and any renamed
|
||||
# original PVC), then the throwaway BSL. Has teeth — a teardown that does not
|
||||
# settle fails the step.
|
||||
- description: vmdisk backup-flow cleanup
|
||||
script:
|
||||
timeout: 15m
|
||||
content: |
|
||||
set -eu
|
||||
cd ../../..
|
||||
# Always tear down the harness-owned BSL, but keep a cleanup.sh failure
|
||||
# fatal (convention 4): capture its rc, then tear down, then re-raise.
|
||||
rc=0
|
||||
NAMESPACE="$NAMESPACE" examples/backups/vmdisk/cleanup.sh || rc=$?
|
||||
# Drain the Velero Backups on this BSL before deleting it, or Velero
|
||||
# fails the finalizer's DeleteBackupRequest ("backup storage location not
|
||||
# found") and orphans the Velero Backup and its S3 objects. Branch on
|
||||
# kubectl's exit status, not empty stdout (a 403/timeout must not read as
|
||||
# "drained").
|
||||
timeout 180 sh -ec '
|
||||
while out=$(kubectl -n cozy-velero get backups.velero.io \
|
||||
-o custom-columns=BSL:.spec.storageLocation --no-headers); do
|
||||
printf "%s\n" "$out" | grep -qx cozy-e2e-vmdisk || exit 0
|
||||
sleep 5
|
||||
done
|
||||
exit 3' && drain_rc=0 || drain_rc=$?
|
||||
if [ "$drain_rc" != 0 ]; then
|
||||
echo "error: Velero backups on cozy-e2e-vmdisk not confirmed drained (rc=$drain_rc); keeping the BSL so they are not orphaned" >&2
|
||||
exit 1
|
||||
fi
|
||||
kubectl -n cozy-velero delete bsl cozy-e2e-vmdisk --ignore-not-found
|
||||
exit "$rc"
|
||||
+14
-1
@@ -251,6 +251,10 @@ src_to_suites() {
|
||||
# not exist, is dropped by intersect_suites(), and a CDI change reaches
|
||||
# nothing runnable through it.
|
||||
vm-disk-application) echo vminstance ;;
|
||||
# The VMInstance and VMDisk backup round-trips are the only Tests that run a
|
||||
# Velero backup and restore; without this the source maps to a `velero`
|
||||
# suite that does not exist and a Velero change runs neither.
|
||||
velero) echo vminstance ;;
|
||||
# Both sources own system/ingress-nginx, and the gateway suite carries the
|
||||
# legacy Ingress admission regression coverage. Without these explicit
|
||||
# mappings their derived names point at suites that do not exist, so an
|
||||
@@ -274,7 +278,7 @@ src_to_suites() {
|
||||
# harbor stores its registry in a BucketClaim the seaweedfs COSI driver
|
||||
# serves. etcd's round-trip is left out because CI gates it off. A test
|
||||
# derives this set from the suites.
|
||||
seaweedfs-application) echo "bucket clickhouse harbor kafka kafka-metadata mariadb mongodb postgres rabbitmq redis seaweedfs valkey" ;;
|
||||
seaweedfs-application) echo "bucket clickhouse harbor kafka kafka-metadata mariadb mongodb postgres rabbitmq redis seaweedfs valkey vminstance" ;;
|
||||
# Both suites create a KeycloakClient and wait for the operator's finalizer
|
||||
# to remove it. kubernetes-application has no dependsOn on keycloak-operator
|
||||
# (#4680), so the graph reaches only monitoring and would drop the tenant
|
||||
@@ -462,8 +466,17 @@ while IFS= read -r file || [ -n "$file" ]; do
|
||||
# across suites: the valkey round-trip runs it with APP_KIND=Valkey, so an
|
||||
# edit there selects both.
|
||||
app=$(echo "$file" | sed -nE 's,^examples/backups/([^/]+)/.*,\1,p')
|
||||
# The vmi and vmdisk backup demos are both harnessed by the vminstance
|
||||
# suite, which owns VMDisk coverage too (see vm-disk-application in
|
||||
# src_to_suites), so their dirs need the same source-style translation.
|
||||
# The suite runs the vmi demo with SKIP_RESTORE_TO_COPY=1, so the to-copy
|
||||
# step is docs-only: selecting the suite would pass without executing it.
|
||||
case "$file" in
|
||||
examples/backups/vmi/07-restore-to-copy.sh) continue ;;
|
||||
esac
|
||||
case "$app" in
|
||||
redis) users="redis valkey" ;;
|
||||
vmi|vmdisk) users=vminstance ;;
|
||||
*) users=$app ;;
|
||||
esac
|
||||
for user in $users; do
|
||||
|
||||
@@ -296,6 +296,35 @@ assert_full_suite() {
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
@test "a velero change selects the vminstance suite" {
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
echo "packages/system/velero/values.yaml" > "$tmp/diff"
|
||||
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
|
||||
rm -rf "$tmp"
|
||||
case " $output " in *" vminstance "*) ;; *) echo "a velero change selected '$output'" >&2; exit 1 ;; esac
|
||||
}
|
||||
|
||||
@test "vmi/vmdisk backup harness edit selects the vminstance suite" {
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
for f in examples/backups/vmi/run-all.sh examples/backups/vmdisk/run-all.sh; do
|
||||
echo "$f" > "$tmp/diff"
|
||||
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
|
||||
[ "$output" = "vminstance" ]
|
||||
done
|
||||
rm -rf "$tmp"
|
||||
}
|
||||
|
||||
@test "vmi restore-to-copy step, which the suite skips, selects nothing" {
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
echo "examples/backups/vmi/07-restore-to-copy.sh" > "$tmp/diff"
|
||||
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources") || true
|
||||
rm -rf "$tmp"
|
||||
[ -z "$output" ]
|
||||
}
|
||||
|
||||
@test "backup example without a matching suite selects nothing" {
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
@@ -495,7 +524,7 @@ assert_full_suite() {
|
||||
# escalated to the full run.
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
echo "hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled" > "$tmp/diff"
|
||||
echo "hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled" > "$tmp/diff"
|
||||
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
|
||||
[ -z "$output" ]
|
||||
rm -rf "$tmp"
|
||||
@@ -507,7 +536,7 @@ assert_full_suite() {
|
||||
# that would silently unselect real work committed in the same change.
|
||||
tmp=$(mktemp -d)
|
||||
cp -r packages/core/platform/sources "$tmp/sources"
|
||||
printf '%s\n' hack/e2e-chainsaw/backup/chainsaw-test.yaml.disabled \
|
||||
printf '%s\n' hack/e2e-chainsaw/kubernetes-proxmox/chainsaw-test.yaml.disabled \
|
||||
packages/apps/postgres/values.yaml > "$tmp/diff"
|
||||
output=$(hack/select-e2e.sh "$tmp/diff" "$tmp/sources")
|
||||
[ "$output" = "postgres" ]
|
||||
|
||||
@@ -6,6 +6,8 @@ import (
|
||||
"encoding/hex"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"slices"
|
||||
"sort"
|
||||
"strings"
|
||||
"time"
|
||||
|
||||
@@ -51,6 +53,14 @@ const (
|
||||
defaultActiveJobPollingInterval = defaultRequeueAfter
|
||||
defaultRestoreRequeueAfter = 5 * time.Second
|
||||
defaultActiveRestorePollingInterval = defaultRestoreRequeueAfter
|
||||
|
||||
// Condition recorded on a RestoreJob before prepareForRestore first changes
|
||||
// the target. From then on a failure has to say what preparation left
|
||||
// behind, and its LastTransitionTime starts the preparation deadline.
|
||||
restoreCondPreparationStarted = "PreparationStarted"
|
||||
// How long preparation may wait (VM shutdown, Pods releasing the disks, the
|
||||
// DataVolume and a recreated PVC going away) before the RestoreJob fails.
|
||||
restorePreparationDeadline = 30 * time.Minute
|
||||
// Velero requires API objects and secrets to be in the cozy-velero namespace
|
||||
veleroNamespace = "cozy-velero"
|
||||
veleroBackupNameMetadataKey = "velero.io/backup-name"
|
||||
@@ -105,13 +115,13 @@ type CommonRestoreOptions struct {
|
||||
|
||||
// RestoreOptions is the typed representation of RestoreJob.Spec.Options for the
|
||||
// Velero driver. The struct is deserialized from runtime.RawExtension and used
|
||||
// for all application kinds. VMInstance-specific fields (KeepOriginalPVC,
|
||||
// KeepOriginalIpAndMac) are only effective when the application kind is VMInstance.
|
||||
// for all application kinds. KeepOriginalPVC applies to VMInstance and VMDisk
|
||||
// targets; KeepOriginalIpAndMac only to VMInstance.
|
||||
type RestoreOptions struct {
|
||||
CommonRestoreOptions `json:",inline"`
|
||||
// KeepOriginalPVC renames the original PVC to <name>-orig-<hash> before restore;
|
||||
// when false, the original PVC is deleted so that Velero recreates it.
|
||||
// Only effective for in-place VMInstance restore (no targetNamespace). Defaults to true when omitted.
|
||||
// Only effective for an in-place VMInstance or VMDisk restore (no targetNamespace). Defaults to true when omitted.
|
||||
KeepOriginalPVC *bool `json:"keepOriginalPVC,omitempty"`
|
||||
// KeepOriginalIpAndMac preserves the original IP and MAC address via OVN
|
||||
// annotations. Only effective for VMInstance restores. Defaults to true when omitted.
|
||||
@@ -363,13 +373,29 @@ func (r *BackupJobReconciler) reconcileVelero(ctx context.Context, j *backupsv1a
|
||||
// Returns nil if the application type has no underlying resources to collect.
|
||||
func (r *BackupJobReconciler) collectUnderlyingResources(ctx context.Context, app *unstructured.Unstructured, appKind, ns string) (*runtime.RawExtension, error) {
|
||||
logger := getLogger(ctx)
|
||||
appName := app.GetName()
|
||||
|
||||
if appKind != vmInstanceKind {
|
||||
logger.Debug("application is not a VMInstance, skipping underlying resource collection", "kind", appKind)
|
||||
return nil, nil
|
||||
// A standalone VMDisk is its own underlying disk. Capturing it lets the
|
||||
// restore controller prepare it as it prepares a VMInstance's disks: keep
|
||||
// the live PVC aside and delete the DataVolume, so the restored PVC is
|
||||
// created fresh under the original name. The HelmRelease is suspended only
|
||||
// until Velero restores it; Flux then recreates the DataVolume and CDI
|
||||
// adopts the restored PVC through the allowClaimAdoption annotation the
|
||||
// restore's resource modifiers set. A standalone VMDisk has no VM, so there
|
||||
// is no IP/MAC to collect.
|
||||
if appKind == vmDiskAppKind {
|
||||
return marshalUnderlyingResources(vmInstanceResources{
|
||||
DataVolumes: []backupsv1alpha1.DataVolumeResource{{
|
||||
DataVolumeName: vmDiskNamePrefix + appName,
|
||||
ApplicationName: appName,
|
||||
}},
|
||||
})
|
||||
}
|
||||
|
||||
appName := app.GetName()
|
||||
if appKind != vmInstanceKind {
|
||||
logger.Debug("application has no underlying resources to collect", "kind", appKind)
|
||||
return nil, nil
|
||||
}
|
||||
|
||||
// Extract disk names from VMInstance spec.disks[].name
|
||||
disks, found, err := unstructured.NestedSlice(app.Object, "spec", "disks")
|
||||
@@ -462,8 +488,11 @@ func (r *BackupJobReconciler) createVeleroBackup(ctx context.Context, backupJob
|
||||
return err
|
||||
}
|
||||
|
||||
// Add label selectors for underlying VMDisk HelmReleases
|
||||
if vmRes := getVMInstanceResources(underlyingResources); vmRes != nil {
|
||||
// Add label selectors for a VMInstance's VMDisk HelmReleases. A VMDisk's own
|
||||
// strategy already selects the disk, and Velero rejects a spec that carries
|
||||
// both labelSelector and orLabelSelectors, so a VMDisk strategy written with
|
||||
// labelSelector must be left as it is.
|
||||
if vmRes := getVMInstanceResources(underlyingResources); vmRes != nil && backupJob.Spec.ApplicationRef.Kind == vmInstanceKind {
|
||||
for _, dv := range vmRes.DataVolumes {
|
||||
veleroBackupSpec.OrLabelSelectors = append(veleroBackupSpec.OrLabelSelectors, &metav1.LabelSelector{
|
||||
MatchLabels: map[string]string{
|
||||
@@ -699,32 +728,41 @@ func (r *RestoreJobReconciler) reconcileVeleroRestore(ctx context.Context, resto
|
||||
}
|
||||
|
||||
// The Backup artifact outlives the Velero Backup it points to, which
|
||||
// Velero garbage-collects once its TTL expires. Checked before
|
||||
// prepareForRestore halts the application and deletes its disks, so a
|
||||
// restore that cannot run leaves them in place.
|
||||
veleroBackup := &velerov1.Backup{}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: veleroBackupName}, veleroBackup); err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
|
||||
"Velero Backup %s/%s not found; it may have expired", veleroNamespace, veleroBackupName))
|
||||
}
|
||||
return ctrl.Result{}, fmt.Errorf("failed to get Velero Backup %s: %w", veleroBackupName, err)
|
||||
}
|
||||
if veleroBackup.Status.Phase != velerov1.BackupPhaseCompleted {
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf(
|
||||
"Velero Backup %s/%s is %q, not %q", veleroNamespace, veleroBackupName,
|
||||
veleroBackup.Status.Phase, velerov1.BackupPhaseCompleted))
|
||||
// Velero garbage-collects once its TTL expires. Checked on every pass
|
||||
// before prepareForRestore halts the application and deletes its disks,
|
||||
// so a restore that cannot run leaves them in place, or, when preparation
|
||||
// already started, names what it left behind.
|
||||
veleroBackup, unrestorable, err := r.veleroBackupForRestore(ctx, veleroBackupName)
|
||||
if err != nil {
|
||||
return ctrl.Result{}, err
|
||||
}
|
||||
|
||||
// Resolve underlying resources once; prefer Backup status, fall back to Velero annotation.
|
||||
ur := resolveUnderlyingResourcesForRestore(backup, veleroBackup)
|
||||
|
||||
// Preparation spans several passes. Once it has changed the target, a
|
||||
// failure must name what it left behind instead of claiming nothing changed.
|
||||
failBeforeRestore := func(msg string) (ctrl.Result, error) {
|
||||
if meta.IsStatusConditionTrue(restoreJob.Status.Conditions, restoreCondPreparationStarted) {
|
||||
return r.markPreparedRestoreFailed(ctx, restoreJob, msg+r.preparedRestoreLeftovers(ctx, restoreJob, backup, ur, target, restoreOpts))
|
||||
}
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, msg+"; nothing was changed")
|
||||
}
|
||||
|
||||
if unrestorable != "" {
|
||||
return failBeforeRestore(unrestorable)
|
||||
}
|
||||
|
||||
// Pre-restore: graceful shutdown, suspend HRs, rename PVCs (skipped for copy)
|
||||
ready, result, err := r.prepareForRestore(ctx, restoreJob, backup, ur, target, restoreOpts)
|
||||
if err != nil {
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("pre-restore preparation failed: %v", err))
|
||||
return failBeforeRestore(fmt.Sprintf("pre-restore preparation failed: %v", err))
|
||||
}
|
||||
if !ready {
|
||||
if cond := meta.FindStatusCondition(restoreJob.Status.Conditions, restoreCondPreparationStarted); cond != nil &&
|
||||
cond.Status == metav1.ConditionTrue && time.Since(cond.LastTransitionTime.Time) > restorePreparationDeadline {
|
||||
return failBeforeRestore(fmt.Sprintf("pre-restore preparation did not finish within %s; see the RestoreJob events for what it was waiting on", restorePreparationDeadline))
|
||||
}
|
||||
logger.Debug("pre-restore preparation in progress, requeuing")
|
||||
return result, nil
|
||||
}
|
||||
@@ -733,7 +771,8 @@ func (r *RestoreJobReconciler) reconcileVeleroRestore(ctx context.Context, resto
|
||||
logger.Debug("Velero Restore not found, creating new one")
|
||||
if err := r.createVeleroRestore(ctx, restoreJob, backup, veleroStrategy, veleroBackupName, ur, target, restoreOpts); err != nil {
|
||||
logger.Error(err, "failed to create Velero Restore")
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, fmt.Sprintf("failed to create Velero Restore: %v", err))
|
||||
return r.markPreparedRestoreFailed(ctx, restoreJob, fmt.Sprintf("failed to create Velero Restore: %v", err)+
|
||||
r.preparedRestoreLeftovers(ctx, restoreJob, backup, ur, target, restoreOpts))
|
||||
}
|
||||
logger.Debug("created Velero Restore, requeuing")
|
||||
// Requeue to check status
|
||||
@@ -742,7 +781,8 @@ func (r *RestoreJobReconciler) reconcileVeleroRestore(ctx context.Context, resto
|
||||
|
||||
if len(veleroRestoreList.Items) > 1 {
|
||||
logger.Error(fmt.Errorf("too many Velero restores for RestoreJob"), "found more than one Velero Restore referencing a single RestoreJob as owner")
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, "found multiple Velero Restores for this RestoreJob")
|
||||
return r.markPreparedRestoreFailed(ctx, restoreJob, "found multiple Velero Restores for this RestoreJob"+
|
||||
r.preparedRestoreLeftovers(ctx, restoreJob, backup, r.restoredUnderlyingResources(ctx, backup, veleroBackupName), target, restoreOpts))
|
||||
}
|
||||
|
||||
veleroRestore := veleroRestoreList.Items[0].DeepCopy()
|
||||
@@ -780,14 +820,21 @@ func (r *RestoreJobReconciler) reconcileVeleroRestore(ctx context.Context, resto
|
||||
return ctrl.Result{}, nil
|
||||
}
|
||||
|
||||
// Step 5: On failure
|
||||
if phase == "Failed" || phase == "PartiallyFailed" {
|
||||
// Step 5: On failure. FailedValidation is terminal too; left to the polling
|
||||
// branch below it would keep a target that preparation already took apart
|
||||
// Running forever.
|
||||
if phase == string(velerov1.RestorePhaseFailed) || phase == string(velerov1.RestorePhasePartiallyFailed) ||
|
||||
phase == string(velerov1.RestorePhaseFailedValidation) {
|
||||
r.cleanupResourceModifierConfigMaps(ctx, restoreJob)
|
||||
message := fmt.Sprintf("Velero Restore failed with phase: %s", phase)
|
||||
if veleroRestore.Status.FailureReason != "" {
|
||||
message = fmt.Sprintf("%s: %s", message, veleroRestore.Status.FailureReason)
|
||||
}
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, message)
|
||||
if len(veleroRestore.Status.ValidationErrors) > 0 {
|
||||
message = fmt.Sprintf("%s: %s", message, strings.Join(veleroRestore.Status.ValidationErrors, "; "))
|
||||
}
|
||||
return r.markPreparedRestoreFailed(ctx, restoreJob, message+
|
||||
r.preparedRestoreLeftovers(ctx, restoreJob, backup, r.restoredUnderlyingResources(ctx, backup, veleroBackupName), target, restoreOpts))
|
||||
}
|
||||
|
||||
// Still in progress (InProgress, New, etc.)
|
||||
@@ -1148,8 +1195,39 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
|
||||
appName := backup.Spec.ApplicationRef.Name
|
||||
appKind := backup.Spec.ApplicationRef.Kind
|
||||
origSuffix := "-orig-" + shortHash(restoreJob.Name)
|
||||
vmRes := restoreDisks(backup, ur)
|
||||
|
||||
vmRes := getVMInstanceResources(ur)
|
||||
// A VMInstance is halted below, which releases its disks; a VMDisk target is
|
||||
// not. Renaming a PVC that a running VM still mounts leaves the original
|
||||
// Terminating behind pvc-protection, and Velero skips restoring a PVC that
|
||||
// still exists, so the restore would report success having moved nothing
|
||||
// while the VM loses its disk on the next restart. Refuse before changing
|
||||
// anything.
|
||||
if appKind != vmInstanceKind && vmRes != nil {
|
||||
inUse, leaving, err := r.podsMountingDisks(ctx, ns, vmRes.DataVolumes)
|
||||
if err != nil {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to check whether the disk is in use: %v", err))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
if len(inUse) > 0 {
|
||||
return false, ctrl.Result{}, fmt.Errorf(
|
||||
"the disk is in use by %s; stop the VM that uses it, or restore that VMInstance instead, then create a new RestoreJob",
|
||||
strings.Join(inUse, ", "))
|
||||
}
|
||||
if len(leaving) > 0 {
|
||||
// Not covered by the preparation deadline, which starts once
|
||||
// preparation changes the target; a Pod stuck terminating on a lost
|
||||
// node would otherwise keep the job Running forever.
|
||||
if started := restoreJob.Status.StartedAt; started != nil && time.Since(started.Time) > restorePreparationDeadline {
|
||||
return false, ctrl.Result{}, fmt.Errorf("%s still held the disk %s after the RestoreJob started",
|
||||
strings.Join(leaving, ", "), restorePreparationDeadline)
|
||||
}
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Waiting for %s to release the disk", strings.Join(leaving, ", ")))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// Refuse a taken -orig name before anything is halted or deleted, rather
|
||||
// than fail halfway through a multi-disk rename.
|
||||
@@ -1167,6 +1245,25 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
|
||||
}
|
||||
}
|
||||
|
||||
if !meta.IsStatusConditionTrue(restoreJob.Status.Conditions, restoreCondPreparationStarted) {
|
||||
before := restoreJob.DeepCopy()
|
||||
meta.SetStatusCondition(&restoreJob.Status.Conditions, metav1.Condition{
|
||||
Type: restoreCondPreparationStarted,
|
||||
Status: metav1.ConditionTrue,
|
||||
Reason: "Preparing",
|
||||
Message: "suspending the target's HelmReleases and keeping its disks aside",
|
||||
})
|
||||
// Nothing has been changed yet: on a failed patch drop the in-memory
|
||||
// condition and retry, or the failure path would report leftovers that
|
||||
// preparation never made.
|
||||
if err := r.Status().Patch(ctx, restoreJob, client.MergeFrom(before)); err != nil {
|
||||
restoreJob.Status.Conditions = before.Status.Conditions
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to record that preparation started: %v", err))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
}
|
||||
|
||||
// --- Step 1: Suspend HelmReleases ---
|
||||
hrNames := []string{}
|
||||
if appKind == vmInstanceKind {
|
||||
@@ -1177,24 +1274,28 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
|
||||
hrNames = append(hrNames, dv.DataVolumeName)
|
||||
}
|
||||
}
|
||||
// A HelmRelease left running would have Flux and CDI recreate the
|
||||
// DataVolume and PVC under the data mover, so a failed suspend stops here.
|
||||
for _, hrName := range hrNames {
|
||||
if err := r.suspendHelmRelease(ctx, ns, hrName); err != nil {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to suspend HelmRelease %s: %v", hrName, err))
|
||||
} else {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Suspended HelmRelease %s", hrName))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Suspended HelmRelease %s", hrName))
|
||||
}
|
||||
|
||||
// --- Step 2: Halt VM and wait for shutdown ---
|
||||
if appKind == vmInstanceKind {
|
||||
vmName := vmNamePrefix + appName
|
||||
// haltVirtualMachine reports a missing VM as halted, so an error here is
|
||||
// a VM that may still be running on the disks.
|
||||
halted, err := r.haltVirtualMachine(ctx, ns, vmName)
|
||||
if err != nil {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to halt VM %s: %v", vmName, err))
|
||||
// Non-fatal: proceed even if halting fails (VM might not exist)
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
} else if !halted {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Waiting for VM %s to shut down", vmName))
|
||||
@@ -1203,6 +1304,21 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("VM %s is halted", vmName))
|
||||
}
|
||||
// The VMI is gone before its launcher Pod is; renaming a PVC that Pod still
|
||||
// mounts would leave it Terminating behind pvc-protection.
|
||||
if vmRes != nil {
|
||||
inUse, leaving, err := r.podsMountingDisks(ctx, ns, vmRes.DataVolumes)
|
||||
if err != nil {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to check whether the disks are in use: %v", err))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
if holders := append(inUse, leaving...); len(holders) > 0 {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Waiting for %s to release the disks", strings.Join(holders, ", ")))
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// --- Step 3: Rename PVCs to <name>-orig-<hash> ---
|
||||
@@ -1270,10 +1386,192 @@ func (r *RestoreJobReconciler) prepareForRestore(ctx context.Context, restoreJob
|
||||
}
|
||||
}
|
||||
|
||||
// --- Step 6: Clear a PVC CDI recreated under the original name ---
|
||||
// Between the rename and the DataVolume delete, CDI can recreate an empty PVC
|
||||
// for the still-present DataVolume. Velero skips restoring a PVC that exists,
|
||||
// so the restore would report success and leave the disk blank. The original
|
||||
// is kept under its -orig name by now, so a PVC under the original name is
|
||||
// such a copy and holds no data.
|
||||
if opts.GetKeepOriginalPVC() && vmRes != nil {
|
||||
for _, dv := range vmRes.DataVolumes {
|
||||
clear, err := r.clearRecreatedPVC(ctx, restoreJob, ns, dv.DataVolumeName)
|
||||
if err != nil {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "PrepareForRestore",
|
||||
fmt.Sprintf("Failed to clear PVC %s: %v", dv.DataVolumeName, err))
|
||||
}
|
||||
if err != nil || !clear {
|
||||
return false, ctrl.Result{RequeueAfter: defaultRestoreRequeueAfter}, nil
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore", "Pre-restore preparation complete")
|
||||
return true, ctrl.Result{}, nil
|
||||
}
|
||||
|
||||
// clearRecreatedPVC reports whether the original name of a kept disk is free
|
||||
// for Velero to restore into, and deletes a PVC found under the name; the
|
||||
// caller requeues until it is gone. Step 4 has already waited for the
|
||||
// DataVolume to be gone, so CDI cannot recreate the PVC again.
|
||||
func (r *RestoreJobReconciler) clearRecreatedPVC(ctx context.Context, restoreJob *backupsv1alpha1.RestoreJob, ns, name string) (bool, error) {
|
||||
// Uncached: the cache can still hold the original PVC the rename just deleted.
|
||||
pvc := &corev1.PersistentVolumeClaim{}
|
||||
if err := r.apiReader().Get(ctx, client.ObjectKey{Namespace: ns, Name: name}, pvc); err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
return true, nil
|
||||
}
|
||||
return false, err
|
||||
}
|
||||
if !pvc.DeletionTimestamp.IsZero() {
|
||||
return false, nil
|
||||
}
|
||||
if err := r.Delete(ctx, pvc); err != nil && !errors.IsNotFound(err) {
|
||||
return false, err
|
||||
}
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeNormal, "PrepareForRestore",
|
||||
fmt.Sprintf("Deleted PVC %s that CDI recreated after the original was kept aside", name))
|
||||
return false, nil
|
||||
}
|
||||
|
||||
// markPreparedRestoreFailed fails a RestoreJob whose target was already
|
||||
// prepared, and repeats the message as a Warning event, since recovering the
|
||||
// target is left to the user.
|
||||
func (r *RestoreJobReconciler) markPreparedRestoreFailed(ctx context.Context, restoreJob *backupsv1alpha1.RestoreJob, message string) (ctrl.Result, error) {
|
||||
r.Recorder.Event(restoreJob, corev1.EventTypeWarning, "RestoreFailed", message)
|
||||
return r.markRestoreJobFailed(ctx, restoreJob, message)
|
||||
}
|
||||
|
||||
// restoreDisks returns the disks an in-place restore prepares. A VMDisk Backup
|
||||
// taken before its underlying resources were recorded has none, but its only
|
||||
// disk is the VMDisk itself, named as collectUnderlyingResources names it.
|
||||
func restoreDisks(backup *backupsv1alpha1.Backup, ur *runtime.RawExtension) *vmInstanceResources {
|
||||
if vmRes := getVMInstanceResources(ur); vmRes != nil {
|
||||
return vmRes
|
||||
}
|
||||
if backup.Spec.ApplicationRef.Kind != vmDiskAppKind {
|
||||
return nil
|
||||
}
|
||||
name := backup.Spec.ApplicationRef.Name
|
||||
return &vmInstanceResources{DataVolumes: []backupsv1alpha1.DataVolumeResource{{
|
||||
DataVolumeName: vmDiskNamePrefix + name,
|
||||
ApplicationName: name,
|
||||
}}}
|
||||
}
|
||||
|
||||
// preparedRestoreLeftovers describes, for the failure message of an in-place
|
||||
// restore that fails after prepareForRestore ran, what preparation left behind.
|
||||
// Nothing is rolled back automatically: resuming a disk's HelmRelease while its
|
||||
// PVC is renamed and its DataVolume deleted would have CDI import a fresh disk
|
||||
// under the original name, which looks healthy and hides that the data sits in
|
||||
// the -orig- PVC. Preparation can fail before it reaches every step, so only a
|
||||
// HelmRelease that is suspended and an -orig- PVC that exists are named; one
|
||||
// that cannot be read is named anyway rather than hidden.
|
||||
func (r *RestoreJobReconciler) preparedRestoreLeftovers(ctx context.Context, restoreJob *backupsv1alpha1.RestoreJob, backup *backupsv1alpha1.Backup, ur *runtime.RawExtension, target restoreTarget, opts RestoreOptions) string {
|
||||
if target.IsCopy {
|
||||
return ""
|
||||
}
|
||||
vmRes := restoreDisks(backup, ur)
|
||||
var hrs, kept []string
|
||||
if backup.Spec.ApplicationRef.Kind == vmInstanceKind {
|
||||
hrs = append(hrs, vmNamePrefix+backup.Spec.ApplicationRef.Name)
|
||||
}
|
||||
if vmRes != nil {
|
||||
for _, dv := range vmRes.DataVolumes {
|
||||
hrs = append(hrs, dv.DataVolumeName)
|
||||
if opts.GetKeepOriginalPVC() {
|
||||
kept = append(kept, dv.DataVolumeName+"-orig-"+shortHash(restoreJob.Name))
|
||||
}
|
||||
}
|
||||
}
|
||||
ns := target.Namespace
|
||||
hrs = slices.DeleteFunc(hrs, func(name string) bool {
|
||||
hr, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, name, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
return errors.IsNotFound(err)
|
||||
}
|
||||
suspended, _, _ := unstructured.NestedBool(hr.Object, "spec", "suspend")
|
||||
return !suspended
|
||||
})
|
||||
kept = slices.DeleteFunc(kept, func(name string) bool {
|
||||
err := r.apiReader().Get(ctx, client.ObjectKey{Namespace: ns, Name: name}, &corev1.PersistentVolumeClaim{})
|
||||
return errors.IsNotFound(err)
|
||||
})
|
||||
if len(hrs) == 0 && len(kept) == 0 {
|
||||
return ""
|
||||
}
|
||||
var parts []string
|
||||
if len(hrs) > 0 {
|
||||
parts = append(parts, fmt.Sprintf("suspended HelmRelease %s", strings.Join(hrs, ", ")))
|
||||
}
|
||||
if len(kept) > 0 {
|
||||
parts = append(parts, fmt.Sprintf("kept the original data in PVC %s", strings.Join(kept, ", ")))
|
||||
}
|
||||
return "; preparation " + strings.Join(parts, " and ") + "; if Velero restored the HelmRelease it is resumed and the disk may have been recreated, so check what the disk holds before relying on it"
|
||||
}
|
||||
|
||||
// podsMountingDisks returns "pod/<pod> (PVC <claim>)" for every Pod in ns that
|
||||
// has not terminated and mounts one of the disks' PVCs, split into Pods that
|
||||
// are running and Pods already being deleted. A terminated Pod no longer holds
|
||||
// pvc-protection, so it does not count.
|
||||
func (r *RestoreJobReconciler) podsMountingDisks(ctx context.Context, ns string, disks []backupsv1alpha1.DataVolumeResource) (inUse, leaving []string, err error) {
|
||||
claims := make(map[string]bool, len(disks))
|
||||
for _, dv := range disks {
|
||||
claims[dv.DataVolumeName] = true
|
||||
}
|
||||
pods := &corev1.PodList{}
|
||||
if err := r.List(ctx, pods, client.InNamespace(ns)); err != nil {
|
||||
return nil, nil, err
|
||||
}
|
||||
for i := range pods.Items {
|
||||
pod := &pods.Items[i]
|
||||
if pod.Status.Phase == corev1.PodSucceeded || pod.Status.Phase == corev1.PodFailed {
|
||||
continue
|
||||
}
|
||||
for _, v := range pod.Spec.Volumes {
|
||||
if v.PersistentVolumeClaim == nil || !claims[v.PersistentVolumeClaim.ClaimName] {
|
||||
continue
|
||||
}
|
||||
holder := fmt.Sprintf("pod/%s (PVC %s)", pod.Name, v.PersistentVolumeClaim.ClaimName)
|
||||
if pod.DeletionTimestamp.IsZero() {
|
||||
inUse = append(inUse, holder)
|
||||
} else {
|
||||
leaving = append(leaving, holder)
|
||||
}
|
||||
}
|
||||
}
|
||||
sort.Strings(inUse)
|
||||
sort.Strings(leaving)
|
||||
return inUse, leaving, nil
|
||||
}
|
||||
|
||||
// veleroBackupForRestore returns the named Velero Backup and why it cannot be
|
||||
// restored, or "" when it can. Only a Completed Backup is restored: Velero
|
||||
// itself validates only that the Backup exists and its storage location is
|
||||
// usable, so a Restore from one in another phase would take the target apart
|
||||
// for nothing. A Backup that is gone comes back empty rather than nil.
|
||||
func (r *RestoreJobReconciler) veleroBackupForRestore(ctx context.Context, name string) (*velerov1.Backup, string, error) {
|
||||
vb := &velerov1.Backup{}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: name}, vb); err != nil {
|
||||
if errors.IsNotFound(err) {
|
||||
return &velerov1.Backup{}, fmt.Sprintf("Velero Backup %s/%s not found; it may have expired", veleroNamespace, name), nil
|
||||
}
|
||||
return nil, "", fmt.Errorf("failed to get Velero Backup %s: %w", name, err)
|
||||
}
|
||||
if vb.Status.Phase != velerov1.BackupPhaseCompleted {
|
||||
return vb, fmt.Sprintf("Velero Backup %s/%s is %q, not %q", veleroNamespace, name, vb.Status.Phase, velerov1.BackupPhaseCompleted), nil
|
||||
}
|
||||
return vb, "", nil
|
||||
}
|
||||
|
||||
// restoredUnderlyingResources resolves the underlying resources of a restore
|
||||
// whose Velero Restore already exists, for naming what preparation left
|
||||
// behind. A Velero Backup that cannot be read leaves the Backup status.
|
||||
func (r *RestoreJobReconciler) restoredUnderlyingResources(ctx context.Context, backup *backupsv1alpha1.Backup, name string) *runtime.RawExtension {
|
||||
vb := &velerov1.Backup{}
|
||||
_ = r.Get(ctx, client.ObjectKey{Namespace: veleroNamespace, Name: name}, vb)
|
||||
return resolveUnderlyingResourcesForRestore(backup, vb)
|
||||
}
|
||||
|
||||
// suspendHelmRelease sets spec.suspend=true on a HelmRelease.
|
||||
func (r *RestoreJobReconciler) suspendHelmRelease(ctx context.Context, ns, name string) error {
|
||||
hr, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, name, metav1.GetOptions{})
|
||||
@@ -1578,7 +1876,7 @@ func (r *RestoreJobReconciler) createVeleroRestore(ctx context.Context, restoreJ
|
||||
|
||||
// Match backup: add OR selectors for each underlying VMDisk so restore applies the same
|
||||
// scope as the intended backup (see createVeleroBackup).
|
||||
if vmRes := getVMInstanceResources(ur); vmRes != nil {
|
||||
if vmRes := getVMInstanceResources(ur); vmRes != nil && backup.Spec.ApplicationRef.Kind == vmInstanceKind {
|
||||
for _, dv := range vmRes.DataVolumes {
|
||||
veleroRestoreSpec.OrLabelSelectors = append(veleroRestoreSpec.OrLabelSelectors, &metav1.LabelSelector{
|
||||
MatchLabels: map[string]string{
|
||||
|
||||
@@ -3,6 +3,7 @@ package backupcontroller
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"reflect"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
@@ -355,6 +356,7 @@ func newTestRestoreJobReconcilerWithDynamic(t *testing.T, dynamicObjects []runti
|
||||
fakeClient := clientfake.NewClientBuilder().
|
||||
WithScheme(testScheme).
|
||||
WithObjects(objects...).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).
|
||||
Build()
|
||||
|
||||
dynamicClient := dynamicfake.NewSimpleDynamicClient(testScheme, dynamicObjects...)
|
||||
@@ -385,6 +387,7 @@ func newTestRestoreJobReconciler(t *testing.T, objects ...client.Object) *Restor
|
||||
fakeClient := clientfake.NewClientBuilder().
|
||||
WithScheme(testScheme).
|
||||
WithObjects(objects...).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).
|
||||
Build()
|
||||
|
||||
dynamicClient := dynamicfake.NewSimpleDynamicClient(testScheme)
|
||||
@@ -730,6 +733,72 @@ func TestReconcileVeleroRestore_UnusableVeleroBackupTouchesNothing(t *testing.T)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectUnderlyingResources_VMDisk(t *testing.T) {
|
||||
// A standalone VMDisk must report its own disk as an underlying resource so
|
||||
// the restore controller suspends the vm-disk-<name> HelmRelease and deletes
|
||||
// its DataVolume before the data mover repopulates the PVC. Without this,
|
||||
// prepareForRestore does nothing for a VMDisk (vmRes == nil) and Flux/CDI
|
||||
// race the data mover over the same PVC name, wedging the restore.
|
||||
r := &BackupJobReconciler{}
|
||||
app := &unstructured.Unstructured{Object: map[string]interface{}{
|
||||
"apiVersion": "apps.cozystack.io/v1alpha1",
|
||||
"kind": "VMDisk",
|
||||
"metadata": map[string]interface{}{"name": "backup-src", "namespace": "tenant-root"},
|
||||
}}
|
||||
|
||||
ur, err := r.collectUnderlyingResources(context.Background(), app, "VMDisk", "tenant-root")
|
||||
if err != nil {
|
||||
t.Fatalf("collectUnderlyingResources returned error: %v", err)
|
||||
}
|
||||
res := getVMInstanceResources(ur)
|
||||
if res == nil {
|
||||
t.Fatal("expected non-nil underlying resources for a VMDisk, got nil (prepareForRestore would skip the HR suspend and DataVolume delete)")
|
||||
}
|
||||
if len(res.DataVolumes) != 1 {
|
||||
t.Fatalf("expected exactly one DataVolume, got %d", len(res.DataVolumes))
|
||||
}
|
||||
if got, want := res.DataVolumes[0].DataVolumeName, "vm-disk-backup-src"; got != want {
|
||||
t.Errorf("DataVolumeName = %q, want %q", got, want)
|
||||
}
|
||||
if got, want := res.DataVolumes[0].ApplicationName, "backup-src"; got != want {
|
||||
t.Errorf("ApplicationName = %q, want %q", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCollectUnderlyingResources_VMInstance(t *testing.T) {
|
||||
testScheme := runtime.NewScheme()
|
||||
_ = scheme.AddToScheme(testScheme)
|
||||
r := &BackupJobReconciler{Client: clientfake.NewClientBuilder().WithScheme(testScheme).Build()}
|
||||
app := &unstructured.Unstructured{Object: map[string]interface{}{
|
||||
"apiVersion": "apps.cozystack.io/v1alpha1",
|
||||
"kind": "VMInstance",
|
||||
"metadata": map[string]interface{}{"name": "test", "namespace": "tenant-root"},
|
||||
"spec": map[string]interface{}{
|
||||
"disks": []interface{}{
|
||||
map[string]interface{}{"name": "system"},
|
||||
map[string]interface{}{"name": "data"},
|
||||
},
|
||||
},
|
||||
}}
|
||||
|
||||
ur, err := r.collectUnderlyingResources(context.Background(), app, "VMInstance", "tenant-root")
|
||||
if err != nil {
|
||||
t.Fatalf("collectUnderlyingResources returned error: %v", err)
|
||||
}
|
||||
res := getVMInstanceResources(ur)
|
||||
if res == nil {
|
||||
t.Fatal("expected non-nil underlying resources for a VMInstance with disks, got nil")
|
||||
}
|
||||
var got []string
|
||||
for _, dv := range res.DataVolumes {
|
||||
got = append(got, dv.DataVolumeName+"/"+dv.ApplicationName)
|
||||
}
|
||||
want := []string{"vm-disk-system/system", "vm-disk-data/data"}
|
||||
if !reflect.DeepEqual(got, want) {
|
||||
t.Errorf("DataVolumes = %v, want %v (one per spec.disks[] entry, not the VMInstance itself)", got, want)
|
||||
}
|
||||
}
|
||||
|
||||
func TestResolveRestoreTarget_VMDisk_CommonRestoreOptions(t *testing.T) {
|
||||
tests := []struct {
|
||||
name string
|
||||
|
||||
@@ -0,0 +1,154 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
package backupcontroller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"strings"
|
||||
"testing"
|
||||
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
"k8s.io/apimachinery/pkg/runtime"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
|
||||
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
|
||||
)
|
||||
|
||||
// An in-place restore of a VMDisk that a running VM mounts must be refused
|
||||
// before anything changes: renaming the PVC under the VM leaves it Terminating
|
||||
// behind pvc-protection, and Velero then skips the PVC as already present.
|
||||
func TestPrepareForRestore_VMDiskInUse(t *testing.T) {
|
||||
const (
|
||||
ns = "tenant-root"
|
||||
claim = "vm-disk-data"
|
||||
)
|
||||
mountingPod := func(name, claimName string, phase corev1.PodPhase) *corev1.Pod {
|
||||
return &corev1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: ns},
|
||||
Spec: corev1.PodSpec{
|
||||
Containers: []corev1.Container{{Name: "compute", Image: "virt-launcher"}},
|
||||
Volumes: []corev1.Volume{{
|
||||
Name: "disk",
|
||||
VolumeSource: corev1.VolumeSource{PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ClaimName: claimName}},
|
||||
}},
|
||||
},
|
||||
Status: corev1.PodStatus{Phase: phase},
|
||||
}
|
||||
}
|
||||
urRaw, _ := json.Marshal(vmInstanceResources{
|
||||
DataVolumes: []backupsv1alpha1.DataVolumeResource{{DataVolumeName: claim, ApplicationName: "data"}},
|
||||
})
|
||||
ur := &runtime.RawExtension{Raw: urRaw}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "restore-disk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
}
|
||||
backupOf := func(kind, name string) *backupsv1alpha1.Backup {
|
||||
return &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: kind, Name: name},
|
||||
},
|
||||
}
|
||||
}
|
||||
pvc := &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{Name: claim, Namespace: ns}}
|
||||
keep := RestoreOptions{KeepOriginalPVC: new(true)}
|
||||
|
||||
t.Run("a running pod that mounts the disk fails the restore and changes nothing", func(t *testing.T) {
|
||||
backup := backupOf("VMDisk", "data")
|
||||
hr := makeUnstructuredHelmRelease(claim, ns, nil)
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{hr},
|
||||
restoreJob.DeepCopy(), backup, pvc.DeepCopy(), mountingPod("virt-launcher-vm-instance-web-abcde", claim, corev1.PodRunning))
|
||||
ctx := context.Background()
|
||||
|
||||
ready, _, err := r.prepareForRestore(ctx, restoreJob.DeepCopy(), backup, ur, restoreTarget{Namespace: ns, AppName: "data", AppKind: "VMDisk"}, keep)
|
||||
if err == nil || ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want an error for a mounted disk", ready, err)
|
||||
}
|
||||
if !strings.Contains(err.Error(), "pod/virt-launcher-vm-instance-web-abcde") {
|
||||
t.Errorf("error %q does not name the pod that mounts the disk", err)
|
||||
}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: ns, Name: claim}, &corev1.PersistentVolumeClaim{}); err != nil {
|
||||
t.Errorf("the mounted PVC must keep its name, got %v", err)
|
||||
}
|
||||
got, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, claim, metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(got.Object, "spec", "suspend"); suspended {
|
||||
t.Errorf("the HelmRelease must not be suspended for a refused restore")
|
||||
}
|
||||
})
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
pod *corev1.Pod
|
||||
}{
|
||||
{name: "a pod that has finished does not hold the disk", pod: mountingPod("importer", claim, corev1.PodSucceeded)},
|
||||
{name: "a pod that mounts another disk does not hold this one", pod: mountingPod("other", "vm-disk-other", corev1.PodRunning)},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
backup := backupOf("VMDisk", "data")
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pvc.DeepCopy(), tc.pod)
|
||||
ready, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: "VMDisk"}, RestoreOptions{KeepOriginalPVC: new(false)})
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// A VMInstance target is halted, so its own launcher Pod is not a reason to
|
||||
// refuse; the rename waits until that Pod has let go of the disks.
|
||||
t.Run("a VMInstance target waits for its own launcher pod instead of being refused", func(t *testing.T) {
|
||||
backup := backupOf("VMInstance", "web")
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pvc.DeepCopy(),
|
||||
mountingPod("virt-launcher-vm-instance-web-abcde", claim, corev1.PodRunning))
|
||||
ready, res, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: "web", AppKind: "VMInstance"}, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a requeue", ready, res.RequeueAfter, err)
|
||||
}
|
||||
if err := r.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: claim}, &corev1.PersistentVolumeClaim{}); err != nil {
|
||||
t.Errorf("the PVC was renamed while the launcher still mounts it: %v", err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a VMInstance target proceeds once no pod mounts its disks", func(t *testing.T) {
|
||||
backup := backupOf("VMInstance", "web")
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pvc.DeepCopy())
|
||||
ready, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: "web", AppKind: "VMInstance"}, RestoreOptions{KeepOriginalPVC: new(false)})
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
})
|
||||
|
||||
// A launcher Pod already being deleted (the VM was just stopped) releases the
|
||||
// disk on its own, so the restore waits rather than fails.
|
||||
t.Run("a VMDisk held by a pod being deleted waits instead of failing", func(t *testing.T) {
|
||||
backup := backupOf("VMDisk", "data")
|
||||
pod := mountingPod("virt-launcher-vm-instance-web-abcde", claim, corev1.PodRunning)
|
||||
now := metav1.Now()
|
||||
pod.DeletionTimestamp = &now
|
||||
pod.Finalizers = []string{"kubernetes"}
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pvc.DeepCopy(), pod)
|
||||
ready, res, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: "VMDisk"}, keep)
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a requeue", ready, res.RequeueAfter, err)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a pod that failed does not hold the disk", func(t *testing.T) {
|
||||
backup := backupOf("VMDisk", "data")
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pvc.DeepCopy(), mountingPod("evicted", claim, corev1.PodFailed))
|
||||
ready, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: "VMDisk"}, RestoreOptions{KeepOriginalPVC: new(false)})
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
@@ -0,0 +1,870 @@
|
||||
// SPDX-License-Identifier: Apache-2.0
|
||||
package backupcontroller
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"errors"
|
||||
"strings"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
velerov1 "github.com/vmware-tanzu/velero/pkg/apis/velero/v1"
|
||||
corev1 "k8s.io/api/core/v1"
|
||||
apierrors "k8s.io/apimachinery/pkg/api/errors"
|
||||
"k8s.io/apimachinery/pkg/api/meta"
|
||||
metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
|
||||
"k8s.io/apimachinery/pkg/apis/meta/v1/unstructured"
|
||||
"k8s.io/apimachinery/pkg/runtime"
|
||||
"k8s.io/apimachinery/pkg/runtime/schema"
|
||||
dynamicfake "k8s.io/client-go/dynamic/fake"
|
||||
"k8s.io/client-go/kubernetes/scheme"
|
||||
k8stesting "k8s.io/client-go/testing"
|
||||
"k8s.io/client-go/tools/record"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client"
|
||||
clientfake "sigs.k8s.io/controller-runtime/pkg/client/fake"
|
||||
"sigs.k8s.io/controller-runtime/pkg/client/interceptor"
|
||||
|
||||
strategyv1alpha1 "github.com/cozystack/cozystack/api/backups/strategy/v1alpha1"
|
||||
backupsv1alpha1 "github.com/cozystack/cozystack/api/backups/v1alpha1"
|
||||
)
|
||||
|
||||
func vmDiskTestScheme() *runtime.Scheme {
|
||||
s := runtime.NewScheme()
|
||||
_ = scheme.AddToScheme(s)
|
||||
_ = backupsv1alpha1.AddToScheme(s)
|
||||
_ = strategyv1alpha1.AddToScheme(s)
|
||||
_ = velerov1.AddToScheme(s)
|
||||
return s
|
||||
}
|
||||
|
||||
func diskSelector() *metav1.LabelSelector {
|
||||
return &metav1.LabelSelector{MatchLabels: map[string]string{"app.kubernetes.io/instance": "vm-disk-{{ .Application.metadata.name }}"}}
|
||||
}
|
||||
|
||||
// Velero rejects a Backup or Restore that carries both labelSelector and
|
||||
// orLabelSelectors, so a VMDisk strategy written with labelSelector must not
|
||||
// get the per-disk orLabelSelectors a VMInstance needs.
|
||||
func TestCreateVeleroBackup_VMDiskKeepsStrategySelector(t *testing.T) {
|
||||
s := vmDiskTestScheme()
|
||||
app := &corev1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "data", Namespace: "tenant-root"},
|
||||
Spec: corev1.PodSpec{Containers: []corev1.Container{{Name: "c", Image: "i"}}},
|
||||
}
|
||||
mapping := &meta.RESTMapping{
|
||||
Resource: schema.GroupVersionResource{Version: "v1", Resource: "pods"},
|
||||
GroupVersionKind: schema.GroupVersionKind{Version: "v1", Kind: "Pod"},
|
||||
Scope: meta.RESTScopeNamespace,
|
||||
}
|
||||
bj := &backupsv1alpha1.BackupJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bj", Namespace: "tenant-root"},
|
||||
Spec: backupsv1alpha1.BackupJobSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
strategy := &strategyv1alpha1.Velero{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "s"},
|
||||
Spec: strategyv1alpha1.VeleroSpec{Template: strategyv1alpha1.VeleroTemplate{
|
||||
Spec: velerov1.BackupSpec{LabelSelector: diskSelector()},
|
||||
}},
|
||||
}
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).WithObjects(bj, strategy).Build()
|
||||
r := &BackupJobReconciler{Client: c, Interface: dynamicfake.NewSimpleDynamicClient(s, app),
|
||||
RESTMapper: &mockRESTMapper{mapping: mapping}, Scheme: s, Recorder: record.NewFakeRecorder(10)}
|
||||
|
||||
if err := r.createVeleroBackup(context.Background(), bj, strategy, &ResolvedBackupConfig{}); err != nil {
|
||||
t.Fatalf("createVeleroBackup: %v", err)
|
||||
}
|
||||
backups := &velerov1.BackupList{}
|
||||
if err := c.List(context.Background(), backups, client.InNamespace(veleroNamespace)); err != nil || len(backups.Items) != 1 {
|
||||
t.Fatalf("list Velero Backups: %v (%d items)", err, len(backups.Items))
|
||||
}
|
||||
spec := backups.Items[0].Spec
|
||||
if spec.LabelSelector == nil || len(spec.OrLabelSelectors) != 0 {
|
||||
t.Fatalf("labelSelector %v, orLabelSelectors %d; want the strategy's labelSelector alone", spec.LabelSelector, len(spec.OrLabelSelectors))
|
||||
}
|
||||
}
|
||||
|
||||
func TestCreateVeleroRestore_VMDiskKeepsStrategySelector(t *testing.T) {
|
||||
s := vmDiskTestScheme()
|
||||
ns := "tenant-root"
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns, UID: "uid-rj"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
}
|
||||
strategy := &strategyv1alpha1.Velero{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "s"},
|
||||
Spec: strategyv1alpha1.VeleroSpec{Template: strategyv1alpha1.VeleroTemplate{
|
||||
RestoreSpec: &velerov1.RestoreSpec{LabelSelector: diskSelector()},
|
||||
}},
|
||||
}
|
||||
urRaw, _ := json.Marshal(vmInstanceResources{
|
||||
DataVolumes: []backupsv1alpha1.DataVolumeResource{{DataVolumeName: "vm-disk-data", ApplicationName: "data"}},
|
||||
})
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).WithObjects(backup, restoreJob, strategy).Build()
|
||||
r := &RestoreJobReconciler{Client: c, Scheme: s, Recorder: record.NewFakeRecorder(10)}
|
||||
|
||||
if err := r.createVeleroRestore(context.Background(), restoreJob, backup, strategy, "velero-bk", &runtime.RawExtension{Raw: urRaw},
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: vmDiskAppKind}, RestoreOptions{}); err != nil {
|
||||
t.Fatalf("createVeleroRestore: %v", err)
|
||||
}
|
||||
restores := &velerov1.RestoreList{}
|
||||
if err := c.List(context.Background(), restores, client.InNamespace(veleroNamespace)); err != nil || len(restores.Items) != 1 {
|
||||
t.Fatalf("list Velero Restores: %v (%d items)", err, len(restores.Items))
|
||||
}
|
||||
spec := restores.Items[0].Spec
|
||||
if spec.LabelSelector == nil || len(spec.OrLabelSelectors) != 0 {
|
||||
t.Fatalf("labelSelector %v, orLabelSelectors %d; want the strategy's labelSelector alone", spec.LabelSelector, len(spec.OrLabelSelectors))
|
||||
}
|
||||
}
|
||||
|
||||
// A VMDisk Backup taken before underlying resources were recorded still
|
||||
// restores its disk prepared, and the in-use refusal still applies to it.
|
||||
func TestPrepareForRestore_VMDiskBackupWithoutUnderlyingResources(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
}
|
||||
target := restoreTarget{Namespace: ns, AppName: "data", AppKind: vmDiskAppKind}
|
||||
|
||||
t.Run("its disk is prepared", func(t *testing.T) {
|
||||
hr := makeUnstructuredHelmRelease("vm-disk-data", ns, nil)
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{hr}, restoreJob.DeepCopy(), backup)
|
||||
ready, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, nil, target, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
got, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(context.Background(), "vm-disk-data", metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(got.Object, "spec", "suspend"); !suspended {
|
||||
t.Fatalf("the disk's HelmRelease was not suspended")
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a mounted disk is refused", func(t *testing.T) {
|
||||
pod := &corev1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "virt-launcher-web", Namespace: ns},
|
||||
Spec: corev1.PodSpec{
|
||||
Containers: []corev1.Container{{Name: "compute", Image: "virt-launcher"}},
|
||||
Volumes: []corev1.Volume{{Name: "disk", VolumeSource: corev1.VolumeSource{
|
||||
PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ClaimName: "vm-disk-data"},
|
||||
}}},
|
||||
},
|
||||
Status: corev1.PodStatus{Phase: corev1.PodRunning},
|
||||
}
|
||||
r := newTestRestoreJobReconciler(t, restoreJob.DeepCopy(), backup, pod)
|
||||
if _, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, nil, target, RestoreOptions{KeepOriginalPVC: new(true)}); err == nil {
|
||||
t.Fatalf("prepareForRestore accepted a mounted disk")
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Failing to list Pods must not read as "not in use": the restore waits and
|
||||
// touches nothing.
|
||||
func TestPrepareForRestore_VMDiskInUseCheckFailsClosed(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
s := vmDiskTestScheme()
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns}}
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).WithObjects(backup, restoreJob).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).
|
||||
WithInterceptorFuncs(interceptor.Funcs{List: func(ctx context.Context, cl client.WithWatch, list client.ObjectList, opts ...client.ListOption) error {
|
||||
if _, ok := list.(*corev1.PodList); ok {
|
||||
return errors.New("pods is forbidden")
|
||||
}
|
||||
return cl.List(ctx, list, opts...)
|
||||
}}).Build()
|
||||
hr := makeUnstructuredHelmRelease("vm-disk-data", ns, nil)
|
||||
r := &RestoreJobReconciler{Client: c, Interface: dynamicfake.NewSimpleDynamicClient(s, hr), Scheme: s, Recorder: record.NewFakeRecorder(10)}
|
||||
|
||||
ready, res, err := r.prepareForRestore(context.Background(), restoreJob, backup, nil,
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: vmDiskAppKind}, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a requeue", ready, res.RequeueAfter, err)
|
||||
}
|
||||
got, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(context.Background(), "vm-disk-data", metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(got.Object, "spec", "suspend"); suspended {
|
||||
t.Fatalf("the HelmRelease was suspended although the in-use check could not run")
|
||||
}
|
||||
}
|
||||
|
||||
// A restore that fails after preparation leaves the disk suspended and its data
|
||||
// in the -orig- PVC; the failure has to say so, since nothing is rolled back,
|
||||
// and must not name what preparation did not get to.
|
||||
func TestPreparedRestoreLeftovers(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns}}
|
||||
vmDisk := &backupsv1alpha1.Backup{Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{Kind: vmDiskAppKind, Name: "data"}}}
|
||||
orig := "vm-disk-data-orig-" + shortHash("rj")
|
||||
origPVC := &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{Name: orig, Namespace: ns}}
|
||||
keep := RestoreOptions{KeepOriginalPVC: new(true)}
|
||||
inPlace := restoreTarget{Namespace: ns}
|
||||
ctx := context.Background()
|
||||
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{suspendedHelmRelease(ns, "vm-disk-data", true)}, origPVC)
|
||||
got := r.preparedRestoreLeftovers(ctx, restoreJob, vmDisk, nil, inPlace, keep)
|
||||
for _, want := range []string{"suspended HelmRelease vm-disk-data", "PVC " + orig} {
|
||||
if !strings.Contains(got, want) {
|
||||
t.Errorf("leftovers %q does not mention %q", got, want)
|
||||
}
|
||||
}
|
||||
if got := r.preparedRestoreLeftovers(ctx, restoreJob, vmDisk, nil, inPlace, RestoreOptions{KeepOriginalPVC: new(false)}); strings.Contains(got, "-orig-") {
|
||||
t.Errorf("leftovers %q names an -orig- PVC although keepOriginalPVC is off", got)
|
||||
}
|
||||
if got := r.preparedRestoreLeftovers(ctx, restoreJob, vmDisk, nil, restoreTarget{Namespace: ns, IsCopy: true}, keep); got != "" {
|
||||
t.Errorf("a restore to a copy prepares nothing, got %q", got)
|
||||
}
|
||||
|
||||
suspendedOnly := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{suspendedHelmRelease(ns, "vm-disk-data", true)})
|
||||
if got := suspendedOnly.preparedRestoreLeftovers(ctx, restoreJob, vmDisk, nil, inPlace, keep); !strings.Contains(got, "suspended HelmRelease vm-disk-data") || strings.Contains(got, "-orig-") {
|
||||
t.Errorf("leftovers %q: want the suspended HelmRelease and no -orig- PVC, which was never created", got)
|
||||
}
|
||||
|
||||
untouched := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{makeUnstructuredHelmRelease("vm-disk-data", ns, nil)})
|
||||
if got := untouched.preparedRestoreLeftovers(ctx, restoreJob, vmDisk, nil, inPlace, keep); got != "" {
|
||||
t.Errorf("leftovers %q for a target preparation never changed, want none", got)
|
||||
}
|
||||
}
|
||||
|
||||
// Every way an in-place restore can fail once its target was prepared has to
|
||||
// name what preparation left behind.
|
||||
func TestReconcileVeleroRestore_FailureAfterPreparationNamesLeftovers(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
orig := "vm-disk-data-orig-" + shortHash("rj")
|
||||
veleroRestore := func(name string, phase velerov1.RestorePhase) *velerov1.Restore {
|
||||
return &velerov1.Restore{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: veleroNamespace, Labels: map[string]string{
|
||||
backupsv1alpha1.OwningJobNameLabel: "rj",
|
||||
backupsv1alpha1.OwningJobNamespaceLabel: ns,
|
||||
}},
|
||||
Status: velerov1.RestoreStatus{Phase: phase},
|
||||
}
|
||||
}
|
||||
failRestoreCreate := interceptor.Funcs{Create: func(ctx context.Context, cl client.WithWatch, obj client.Object, opts ...client.CreateOption) error {
|
||||
if _, ok := obj.(*velerov1.Restore); ok {
|
||||
return errors.New("admission webhook denied the request")
|
||||
}
|
||||
return cl.Create(ctx, obj, opts...)
|
||||
}}
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
restores []client.Object
|
||||
funcs interceptor.Funcs
|
||||
cause string
|
||||
}{
|
||||
{name: "the Velero Restore partially failed", restores: []client.Object{veleroRestore("rj-a", velerov1.RestorePhasePartiallyFailed)}, cause: "PartiallyFailed"},
|
||||
{name: "several Velero Restores own the job", restores: []client.Object{veleroRestore("rj-a", ""), veleroRestore("rj-b", "")}, cause: "multiple Velero Restores"},
|
||||
{name: "Velero rejected the Restore", restores: []client.Object{func() client.Object {
|
||||
vr := veleroRestore("rj-a", velerov1.RestorePhaseFailedValidation)
|
||||
vr.Status.ValidationErrors = []string{"backup velero-bk not found"}
|
||||
return vr
|
||||
}()}, cause: "FailedValidation: backup velero-bk not found"},
|
||||
{name: "the Velero Restore could not be created", funcs: failRestoreCreate, cause: "failed to create Velero Restore"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s := vmDiskTestScheme()
|
||||
started := metav1.Now()
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns, UID: "rj-uid"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{Phase: backupsv1alpha1.RestoreJobPhaseRunning, StartedAt: &started},
|
||||
}
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data"},
|
||||
StrategyRef: corev1.TypedLocalObjectReference{APIGroup: new(strategyv1alpha1.GroupVersion.Group), Kind: strategyv1alpha1.VeleroStrategyKind, Name: "s"},
|
||||
DriverMetadata: map[string]string{veleroBackupNameMetadataKey: "velero-bk"},
|
||||
},
|
||||
}
|
||||
strategy := &strategyv1alpha1.Velero{ObjectMeta: metav1.ObjectMeta{Name: "s"}}
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).
|
||||
WithObjects(append([]client.Object{restoreJob, backup, strategy, completedVeleroBackup("velero-bk")}, append(keptOrigPVC(ns), tc.restores...)...)...).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).
|
||||
WithInterceptorFuncs(tc.funcs).Build()
|
||||
recorder := record.NewFakeRecorder(20)
|
||||
r := &RestoreJobReconciler{Client: c, Interface: dynamicfake.NewSimpleDynamicClient(s, suspendedHelmRelease(ns, "vm-disk-data", true)), Scheme: s, Recorder: recorder}
|
||||
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileVeleroRestore(context.Background(), rj, backup); err != nil {
|
||||
t.Fatalf("reconcileVeleroRestore: %v", err)
|
||||
}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed {
|
||||
t.Fatalf("Phase = %q, want Failed", rj.Status.Phase)
|
||||
}
|
||||
for _, want := range []string{tc.cause, "suspended HelmRelease vm-disk-data", "PVC " + orig} {
|
||||
if !strings.Contains(rj.Status.Message, want) {
|
||||
t.Errorf("message %q does not mention %q", rj.Status.Message, want)
|
||||
}
|
||||
}
|
||||
warned := false
|
||||
for len(recorder.Events) > 0 {
|
||||
ev := <-recorder.Events
|
||||
if strings.HasPrefix(ev, "Warning RestoreFailed") && strings.Contains(ev, orig) {
|
||||
warned = true
|
||||
}
|
||||
}
|
||||
if !warned {
|
||||
t.Errorf("no Warning RestoreFailed event naming %s", orig)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func completedVeleroBackup(name string) *velerov1.Backup {
|
||||
return &velerov1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: veleroNamespace},
|
||||
Status: velerov1.BackupStatus{Phase: velerov1.BackupPhaseCompleted},
|
||||
}
|
||||
}
|
||||
|
||||
// A Velero Backup that is gone or failed cannot be restored; the job fails
|
||||
// before preparation takes the target apart.
|
||||
func TestReconcileVeleroRestore_UnrestorableVeleroBackupFailsBeforePreparation(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
vb []client.Object
|
||||
cause string
|
||||
}{
|
||||
{name: "the Velero Backup is gone", cause: "not found; it may have expired"},
|
||||
{name: "the Velero Backup failed", vb: []client.Object{&velerov1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "velero-bk", Namespace: veleroNamespace},
|
||||
Status: velerov1.BackupStatus{Phase: velerov1.BackupPhaseFailed},
|
||||
}}, cause: `is "Failed"`},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
s := vmDiskTestScheme()
|
||||
started := metav1.Now()
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns, UID: "rj-uid"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{Phase: backupsv1alpha1.RestoreJobPhaseRunning, StartedAt: &started},
|
||||
}
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data"},
|
||||
StrategyRef: corev1.TypedLocalObjectReference{APIGroup: new(strategyv1alpha1.GroupVersion.Group), Kind: strategyv1alpha1.VeleroStrategyKind, Name: "s"},
|
||||
DriverMetadata: map[string]string{veleroBackupNameMetadataKey: "velero-bk"},
|
||||
},
|
||||
}
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).
|
||||
WithObjects(append([]client.Object{restoreJob, backup, &strategyv1alpha1.Velero{ObjectMeta: metav1.ObjectMeta{Name: "s"}}}, tc.vb...)...).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).Build()
|
||||
hr := makeUnstructuredHelmRelease("vm-disk-data", ns, nil)
|
||||
r := &RestoreJobReconciler{Client: c, Interface: dynamicfake.NewSimpleDynamicClient(s, hr), Scheme: s, Recorder: record.NewFakeRecorder(10)}
|
||||
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileVeleroRestore(context.Background(), rj, backup); err != nil {
|
||||
t.Fatalf("reconcileVeleroRestore: %v", err)
|
||||
}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(rj.Status.Message, tc.cause) {
|
||||
t.Fatalf("phase %q, message %q; want Failed mentioning %q", rj.Status.Phase, rj.Status.Message, tc.cause)
|
||||
}
|
||||
got, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(context.Background(), "vm-disk-data", metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(got.Object, "spec", "suspend"); suspended {
|
||||
t.Fatalf("the disk was prepared although its Velero Backup cannot be restored")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// A suspend or halt that failed must stop preparation: renaming the PVC and
|
||||
// deleting the DataVolume under a running HelmRelease or VM is the race the
|
||||
// preparation exists to prevent.
|
||||
func TestPrepareForRestore_StopsWhenSuspendOrHaltFails(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
urRaw, _ := json.Marshal(vmInstanceResources{
|
||||
DataVolumes: []backupsv1alpha1.DataVolumeResource{{DataVolumeName: "vm-disk-data", ApplicationName: "data"}},
|
||||
})
|
||||
ur := &runtime.RawExtension{Raw: urRaw}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns}}
|
||||
pvc := &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{Name: "vm-disk-data", Namespace: ns}}
|
||||
dataVolume := &unstructured.Unstructured{}
|
||||
dataVolume.SetAPIVersion("cdi.kubevirt.io/v1beta1")
|
||||
dataVolume.SetKind("DataVolume")
|
||||
dataVolume.SetName("vm-disk-data")
|
||||
dataVolume.SetNamespace(ns)
|
||||
vm := &unstructured.Unstructured{}
|
||||
vm.SetAPIVersion("kubevirt.io/v1")
|
||||
vm.SetKind("VirtualMachine")
|
||||
vm.SetName("vm-instance-web")
|
||||
vm.SetNamespace(ns)
|
||||
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
kind string
|
||||
app string
|
||||
resource string
|
||||
}{
|
||||
{name: "the HelmRelease suspend fails", kind: vmDiskAppKind, app: "data", resource: "helmreleases"},
|
||||
{name: "the VM halt fails", kind: vmInstanceKind, app: "web", resource: "virtualmachines"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: tc.kind, Name: tc.app}},
|
||||
}
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{
|
||||
makeUnstructuredHelmRelease("vm-disk-data", ns, nil),
|
||||
makeUnstructuredHelmRelease("vm-instance-web", ns, nil),
|
||||
dataVolume.DeepCopy(), vm.DeepCopy(),
|
||||
}, restoreJob.DeepCopy(), backup, pvc.DeepCopy())
|
||||
r.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("update", tc.resource, func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
return true, nil, errors.New("conflict")
|
||||
})
|
||||
|
||||
ready, res, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, ur,
|
||||
restoreTarget{Namespace: ns, AppName: tc.app, AppKind: tc.kind}, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a requeue", ready, res.RequeueAfter, err)
|
||||
}
|
||||
if err := r.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "vm-disk-data"}, &corev1.PersistentVolumeClaim{}); err != nil {
|
||||
t.Errorf("the PVC was renamed: %v", err)
|
||||
}
|
||||
if _, err := r.Resource(dataVolumeGVR).Namespace(ns).Get(context.Background(), "vm-disk-data", metav1.GetOptions{}); err != nil {
|
||||
t.Errorf("the DataVolume was deleted: %v", err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreDisks(t *testing.T) {
|
||||
backupOf := func(kind string) *backupsv1alpha1.Backup {
|
||||
return &backupsv1alpha1.Backup{Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{Kind: kind, Name: "data"}}}
|
||||
}
|
||||
if got := restoreDisks(backupOf(vmDiskAppKind), nil); got == nil || len(got.DataVolumes) != 1 || got.DataVolumes[0].DataVolumeName != "vm-disk-data" {
|
||||
t.Errorf("VMDisk without underlying resources: %+v, want vm-disk-data", got)
|
||||
}
|
||||
for _, kind := range []string{vmInstanceKind, "Postgres"} {
|
||||
if got := restoreDisks(backupOf(kind), nil); got != nil {
|
||||
t.Errorf("%s without underlying resources: %+v, want none; a disk must not be derived for another kind", kind, got)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Kinds other than VMDisk and VMInstance have no disks to prepare and must pass
|
||||
// through preparation untouched.
|
||||
func TestPrepareForRestore_OtherKindWithoutDisks(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: "Postgres", Name: "data"}},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns}}
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{makeUnstructuredHelmRelease("vm-disk-data", ns, nil)}, restoreJob.DeepCopy(), backup)
|
||||
ready, _, err := r.prepareForRestore(context.Background(), restoreJob.DeepCopy(), backup, nil,
|
||||
restoreTarget{Namespace: ns, AppName: "data", AppKind: "Postgres"}, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
got, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(context.Background(), "vm-disk-data", metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(got.Object, "spec", "suspend"); suspended {
|
||||
t.Fatalf("an unrelated vm-disk-data HelmRelease was suspended")
|
||||
}
|
||||
}
|
||||
|
||||
func TestPreparedRestoreLeftovers_VMInstanceNamesItsHelmRelease(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{ObjectMeta: metav1.ObjectMeta{Name: "rj"}}
|
||||
backup := &backupsv1alpha1.Backup{Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{Kind: vmInstanceKind, Name: "web"}}}
|
||||
urRaw, _ := json.Marshal(vmInstanceResources{
|
||||
DataVolumes: []backupsv1alpha1.DataVolumeResource{{DataVolumeName: "vm-disk-root", ApplicationName: "root"}},
|
||||
})
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{suspendedHelmRelease(ns, "vm-instance-web", true), suspendedHelmRelease(ns, "vm-disk-root", true)})
|
||||
got := r.preparedRestoreLeftovers(context.Background(), restoreJob, backup, &runtime.RawExtension{Raw: urRaw}, restoreTarget{Namespace: ns}, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if !strings.Contains(got, "vm-instance-web, vm-disk-root") {
|
||||
t.Errorf("leftovers %q do not name the VM's HelmRelease and its disk's", got)
|
||||
}
|
||||
}
|
||||
|
||||
// CDI can recreate an empty PVC under the original name between the rename and
|
||||
// the DataVolume delete. Velero skips restoring a PVC that exists, so the
|
||||
// restore would report success with a blank disk; preparation has to clear it.
|
||||
func TestPrepareForRestore_ClearsPVCRecreatedAfterRename(t *testing.T) {
|
||||
f := newKeepPVCFixture(t)
|
||||
r := f.reconciler(t, f.livePVC, f.pv)
|
||||
ctx := context.Background()
|
||||
recreated := &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{Name: f.livePVC.Name, Namespace: f.ns}}
|
||||
recreatedOnce := false
|
||||
r.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("delete", "datavolumes", func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
if !recreatedOnce {
|
||||
recreatedOnce = true
|
||||
if err := r.Create(ctx, recreated.DeepCopy()); err != nil {
|
||||
t.Errorf("recreate PVC: %v", err)
|
||||
}
|
||||
}
|
||||
return false, nil, nil
|
||||
})
|
||||
|
||||
ready, res, err := r.prepareForRestore(ctx, f.restoreJob, f.backup, f.ur, f.target, f.opts)
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("pass 1: prepareForRestore = ready %v, requeue %v, err %v; want a requeue while the recreated PVC goes", ready, res.RequeueAfter, err)
|
||||
}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(recreated), &corev1.PersistentVolumeClaim{}); !apierrors.IsNotFound(err) {
|
||||
t.Fatalf("the recreated PVC is still there (err %v); Velero would skip restoring it", err)
|
||||
}
|
||||
|
||||
ready, _, err = r.prepareForRestore(ctx, f.restoreJob, f.backup, f.ur, f.target, f.opts)
|
||||
if err != nil || !ready {
|
||||
t.Fatalf("pass 2: prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
kept := &corev1.PersistentVolumeClaim{}
|
||||
if err := r.Get(ctx, client.ObjectKey{Namespace: f.ns, Name: f.origName}, kept); err != nil || kept.Spec.VolumeName != "pv-current" {
|
||||
t.Fatalf("the original disk is not kept as %s on pv-current: %+v, err %v", f.origName, kept.Spec, err)
|
||||
}
|
||||
}
|
||||
|
||||
// reconcilePreparedVMDisk runs reconcileVeleroRestore for an in-place VMDisk
|
||||
// restore that has no Velero Restore yet; started, when set, is the time the
|
||||
// RestoreJob recorded PreparationStarted.
|
||||
// keptOrigPVC is the -orig- PVC, bound to its retained PV, that preparation
|
||||
// leaves for the RestoreJob "rj" with UID "rj-uid".
|
||||
func keptOrigPVC(ns string) []client.Object {
|
||||
name := "vm-disk-data-orig-" + shortHash("rj")
|
||||
return []client.Object{
|
||||
&corev1.PersistentVolumeClaim{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: ns, Annotations: map[string]string{origPVCRestoreJobUIDAnnotation: "rj-uid"}},
|
||||
Spec: corev1.PersistentVolumeClaimSpec{VolumeName: "pv-data"},
|
||||
},
|
||||
&corev1.PersistentVolume{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "pv-data"},
|
||||
Spec: corev1.PersistentVolumeSpec{
|
||||
PersistentVolumeReclaimPolicy: corev1.PersistentVolumeReclaimRetain,
|
||||
ClaimRef: &corev1.ObjectReference{Namespace: ns, Name: name},
|
||||
},
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
func reconcilePreparedVMDisk(t *testing.T, started *metav1.Time, extra ...client.Object) (*backupsv1alpha1.RestoreJob, *record.FakeRecorder) {
|
||||
t.Helper()
|
||||
const ns = "tenant-root"
|
||||
s := vmDiskTestScheme()
|
||||
now := metav1.Now()
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns, UID: "rj-uid"},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{Phase: backupsv1alpha1.RestoreJobPhaseRunning, StartedAt: &now},
|
||||
}
|
||||
hr := makeUnstructuredHelmRelease("vm-disk-data", ns, nil)
|
||||
if started != nil {
|
||||
hr = suspendedHelmRelease(ns, "vm-disk-data", true)
|
||||
extra = append(extra, keptOrigPVC(ns)...)
|
||||
restoreJob.Status.Conditions = []metav1.Condition{{
|
||||
Type: restoreCondPreparationStarted, Status: metav1.ConditionTrue, Reason: "Preparing", LastTransitionTime: *started,
|
||||
}}
|
||||
}
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{
|
||||
ApplicationRef: corev1.TypedLocalObjectReference{APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data"},
|
||||
StrategyRef: corev1.TypedLocalObjectReference{APIGroup: new(strategyv1alpha1.GroupVersion.Group), Kind: strategyv1alpha1.VeleroStrategyKind, Name: "s"},
|
||||
DriverMetadata: map[string]string{veleroBackupNameMetadataKey: "velero-bk"},
|
||||
},
|
||||
}
|
||||
c := clientfake.NewClientBuilder().WithScheme(s).
|
||||
WithObjects(append([]client.Object{restoreJob, backup, &strategyv1alpha1.Velero{ObjectMeta: metav1.ObjectMeta{Name: "s"}}}, extra...)...).
|
||||
WithStatusSubresource(&backupsv1alpha1.RestoreJob{}).Build()
|
||||
recorder := record.NewFakeRecorder(20)
|
||||
r := &RestoreJobReconciler{Client: c, Interface: dynamicfake.NewSimpleDynamicClient(s, hr), Scheme: s, Recorder: recorder}
|
||||
rj := &backupsv1alpha1.RestoreJob{}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
if _, err := r.reconcileVeleroRestore(context.Background(), rj, backup); err != nil {
|
||||
t.Fatalf("reconcileVeleroRestore: %v", err)
|
||||
}
|
||||
if err := c.Get(context.Background(), client.ObjectKey{Namespace: ns, Name: "rj"}, rj); err != nil {
|
||||
t.Fatalf("get RestoreJob: %v", err)
|
||||
}
|
||||
return rj, recorder
|
||||
}
|
||||
|
||||
func mountingVMDiskPod(name string, deleting bool) *corev1.Pod {
|
||||
pod := &corev1.Pod{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: name, Namespace: "tenant-root"},
|
||||
Spec: corev1.PodSpec{
|
||||
Containers: []corev1.Container{{Name: "compute", Image: "virt-launcher"}},
|
||||
Volumes: []corev1.Volume{{Name: "disk", VolumeSource: corev1.VolumeSource{
|
||||
PersistentVolumeClaim: &corev1.PersistentVolumeClaimVolumeSource{ClaimName: "vm-disk-data"},
|
||||
}}},
|
||||
},
|
||||
Status: corev1.PodStatus{Phase: corev1.PodRunning},
|
||||
}
|
||||
if deleting {
|
||||
now := metav1.Now()
|
||||
pod.DeletionTimestamp = &now
|
||||
pod.Finalizers = []string{"kubernetes"}
|
||||
}
|
||||
return pod
|
||||
}
|
||||
|
||||
// Preparation spans several passes. A refusal on a later pass must say what
|
||||
// preparation already left behind, not that nothing was changed.
|
||||
func TestReconcileVeleroRestore_RefusalAfterPreparationStartedNamesLeftovers(t *testing.T) {
|
||||
started := metav1.Now()
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
extra []client.Object
|
||||
cause string
|
||||
}{
|
||||
{name: "the Velero Backup expired meanwhile", cause: "not found; it may have expired"},
|
||||
{name: "a VM started on the disk meanwhile", extra: []client.Object{completedVeleroBackup("velero-bk"), mountingVMDiskPod("virt-launcher-web", false)}, cause: "in use by pod/virt-launcher-web"},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
rj, recorder := reconcilePreparedVMDisk(t, &started, tc.extra...)
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed {
|
||||
t.Fatalf("Phase = %q, want Failed", rj.Status.Phase)
|
||||
}
|
||||
for _, want := range []string{tc.cause, "suspended HelmRelease vm-disk-data", "PVC vm-disk-data-orig-" + shortHash("rj")} {
|
||||
if !strings.Contains(rj.Status.Message, want) {
|
||||
t.Errorf("message %q does not mention %q", rj.Status.Message, want)
|
||||
}
|
||||
}
|
||||
if strings.Contains(rj.Status.Message, "nothing was changed") {
|
||||
t.Errorf("message %q claims nothing was changed after preparation started", rj.Status.Message)
|
||||
}
|
||||
warned := false
|
||||
for len(recorder.Events) > 0 {
|
||||
if strings.HasPrefix(<-recorder.Events, "Warning RestoreFailed") {
|
||||
warned = true
|
||||
}
|
||||
}
|
||||
if !warned {
|
||||
t.Error("no Warning RestoreFailed event")
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
|
||||
// Before preparation starts, a refusal really has changed nothing and must
|
||||
// leave PreparationStarted unset.
|
||||
func TestReconcileVeleroRestore_RefusalBeforePreparationChangesNothing(t *testing.T) {
|
||||
rj, _ := reconcilePreparedVMDisk(t, nil, completedVeleroBackup("velero-bk"), mountingVMDiskPod("virt-launcher-web", false))
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(rj.Status.Message, "nothing was changed") {
|
||||
t.Fatalf("phase %q, message %q; want Failed saying nothing was changed", rj.Status.Phase, rj.Status.Message)
|
||||
}
|
||||
if meta.IsStatusConditionTrue(rj.Status.Conditions, restoreCondPreparationStarted) {
|
||||
t.Error("PreparationStarted was recorded although the restore was refused first")
|
||||
}
|
||||
}
|
||||
|
||||
func TestReconcileVeleroRestore_PreparationDeadline(t *testing.T) {
|
||||
waiting := []client.Object{completedVeleroBackup("velero-bk"), mountingVMDiskPod("virt-launcher-web", true)}
|
||||
|
||||
t.Run("a preparation still waiting past the deadline fails and names the leftovers", func(t *testing.T) {
|
||||
long := metav1.NewTime(time.Now().Add(-restorePreparationDeadline - time.Minute))
|
||||
rj, _ := reconcilePreparedVMDisk(t, &long, waiting...)
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseFailed || !strings.Contains(rj.Status.Message, "did not finish within") ||
|
||||
!strings.Contains(rj.Status.Message, "suspended HelmRelease vm-disk-data") {
|
||||
t.Fatalf("phase %q, message %q; want Failed past the deadline, naming the leftovers", rj.Status.Phase, rj.Status.Message)
|
||||
}
|
||||
})
|
||||
|
||||
t.Run("a preparation waiting within the deadline keeps running", func(t *testing.T) {
|
||||
recent := metav1.Now()
|
||||
rj, _ := reconcilePreparedVMDisk(t, &recent, waiting...)
|
||||
if rj.Status.Phase != backupsv1alpha1.RestoreJobPhaseRunning {
|
||||
t.Fatalf("Phase = %q, message %q; want Running", rj.Status.Phase, rj.Status.Message)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
func TestPrepareForRestore_RecordsPreparationStarted(t *testing.T) {
|
||||
f := newKeepPVCFixture(t)
|
||||
r := f.reconciler(t, f.livePVC, f.pv)
|
||||
if ready, _, err := r.prepareForRestore(context.Background(), f.restoreJob, f.backup, f.ur, f.target, f.opts); err != nil || !ready {
|
||||
t.Fatalf("prepareForRestore = ready %v, err %v; want ready", ready, err)
|
||||
}
|
||||
got := &backupsv1alpha1.RestoreJob{}
|
||||
if err := r.Get(context.Background(), client.ObjectKeyFromObject(f.restoreJob), got); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if !meta.IsStatusConditionTrue(got.Status.Conditions, restoreCondPreparationStarted) {
|
||||
t.Errorf("PreparationStarted not recorded: %+v", got.Status.Conditions)
|
||||
}
|
||||
}
|
||||
|
||||
// Step 5 waits for the DataVolume to be gone: while it exists, CDI can create
|
||||
// the PVC again after Step 5 cleared it.
|
||||
func TestPrepareForRestore_WaitsForDataVolumeGone(t *testing.T) {
|
||||
f := newKeepPVCFixture(t)
|
||||
r := f.reconciler(t, f.livePVC, f.pv)
|
||||
r.Interface.(*dynamicfake.FakeDynamicClient).PrependReactor("delete", "datavolumes", func(k8stesting.Action) (bool, runtime.Object, error) {
|
||||
return true, nil, nil
|
||||
})
|
||||
ready, res, err := r.prepareForRestore(context.Background(), f.restoreJob, f.backup, f.ur, f.target, f.opts)
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a requeue while the DataVolume exists", ready, res.RequeueAfter, err)
|
||||
}
|
||||
}
|
||||
|
||||
// Without keepOriginalPVC, a VMDisk PVC left in place would be skipped by
|
||||
// Velero and, orphaned from its DataVolume, make CDI refuse the DataVolume
|
||||
// the resumed HelmRelease creates: preparation has to delete it.
|
||||
func TestPrepareForRestore_VMDiskWithoutKeepDeletesOriginalPVC(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
}
|
||||
target := restoreTarget{Namespace: ns, AppName: "data", AppKind: vmDiskAppKind}
|
||||
pvc := &corev1.PersistentVolumeClaim{ObjectMeta: metav1.ObjectMeta{Name: "vm-disk-data", Namespace: ns}}
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t,
|
||||
[]runtime.Object{makeUnstructuredHelmRelease("vm-disk-data", ns, nil), makeUnstructuredDataVolume("vm-disk-data", ns)},
|
||||
restoreJob.DeepCopy(), backup, pvc)
|
||||
ctx := context.Background()
|
||||
|
||||
ready := false
|
||||
for pass := 0; pass < 5 && !ready; pass++ {
|
||||
var err error
|
||||
if ready, _, err = r.prepareForRestore(ctx, restoreJob.DeepCopy(), backup, nil, target, RestoreOptions{KeepOriginalPVC: new(false)}); err != nil {
|
||||
t.Fatalf("prepareForRestore: %v", err)
|
||||
}
|
||||
}
|
||||
if !ready {
|
||||
t.Fatal("preparation did not finish")
|
||||
}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(pvc), &corev1.PersistentVolumeClaim{}); !apierrors.IsNotFound(err) {
|
||||
t.Fatalf("original PVC still present (err %v); Velero would skip restoring it", err)
|
||||
}
|
||||
pvcs := &corev1.PersistentVolumeClaimList{}
|
||||
if err := r.List(ctx, pvcs, client.InNamespace(ns)); err != nil {
|
||||
t.Fatalf("list PVCs: %v", err)
|
||||
}
|
||||
for _, p := range pvcs.Items {
|
||||
t.Errorf("unexpected PVC %s left behind without keepOriginalPVC", p.Name)
|
||||
}
|
||||
if _, err := r.Resource(dataVolumeGVR).Namespace(ns).Get(ctx, "vm-disk-data", metav1.GetOptions{}); !apierrors.IsNotFound(err) {
|
||||
t.Fatalf("DataVolume still present (err %v)", err)
|
||||
}
|
||||
hr, err := r.Resource(helmReleaseGVR).Namespace(ns).Get(ctx, "vm-disk-data", metav1.GetOptions{})
|
||||
if err != nil {
|
||||
t.Fatalf("get HelmRelease: %v", err)
|
||||
}
|
||||
if suspended, _, _ := unstructured.NestedBool(hr.Object, "spec", "suspend"); !suspended {
|
||||
t.Fatal("the disk's HelmRelease was not suspended")
|
||||
}
|
||||
}
|
||||
|
||||
// A failed PreparationStarted patch must leave the target and the in-memory
|
||||
// condition untouched and retry, so a later failure does not report leftovers.
|
||||
func TestPrepareForRestore_PreparationStartedPatchFailureRetries(t *testing.T) {
|
||||
f := newKeepPVCFixture(t)
|
||||
r := f.reconciler(t, f.livePVC, f.pv)
|
||||
r.Client = interceptor.NewClient(r.Client.(client.WithWatch), interceptor.Funcs{
|
||||
SubResourcePatch: func(context.Context, client.Client, string, client.Object, client.Patch, ...client.SubResourcePatchOption) error {
|
||||
return errors.New("the object has been modified")
|
||||
},
|
||||
})
|
||||
ctx := context.Background()
|
||||
ready, res, err := r.prepareForRestore(ctx, f.restoreJob, f.backup, f.ur, f.target, f.opts)
|
||||
if err != nil || ready || res.RequeueAfter == 0 {
|
||||
t.Fatalf("prepareForRestore = ready %v, requeue %v, err %v; want a retry", ready, res.RequeueAfter, err)
|
||||
}
|
||||
if meta.FindStatusCondition(f.restoreJob.Status.Conditions, restoreCondPreparationStarted) != nil {
|
||||
t.Errorf("PreparationStarted kept in memory after the patch failed: %+v", f.restoreJob.Status.Conditions)
|
||||
}
|
||||
if err := r.Get(ctx, client.ObjectKeyFromObject(f.livePVC), &corev1.PersistentVolumeClaim{}); err != nil {
|
||||
t.Errorf("original PVC touched although preparation did not start: %v", err)
|
||||
}
|
||||
if !f.dataVolumeExists(t, r) {
|
||||
t.Error("DataVolume deleted although preparation did not start")
|
||||
}
|
||||
}
|
||||
|
||||
// A Pod stuck terminating holds the disk before preparation changes anything,
|
||||
// so the preparation deadline does not cover it; the wait needs its own.
|
||||
func TestPrepareForRestore_VMDiskLeavingPodWaitHasDeadline(t *testing.T) {
|
||||
const ns = "tenant-root"
|
||||
backup := &backupsv1alpha1.Backup{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "bk", Namespace: ns},
|
||||
Spec: backupsv1alpha1.BackupSpec{ApplicationRef: corev1.TypedLocalObjectReference{
|
||||
APIGroup: new("apps.cozystack.io"), Kind: vmDiskAppKind, Name: "data",
|
||||
}},
|
||||
}
|
||||
target := restoreTarget{Namespace: ns, AppName: "data", AppKind: vmDiskAppKind}
|
||||
for _, tc := range []struct {
|
||||
name string
|
||||
age time.Duration
|
||||
wantFail bool
|
||||
}{
|
||||
{name: "within the deadline it waits", age: time.Minute},
|
||||
{name: "past the deadline it fails", age: restorePreparationDeadline + time.Minute, wantFail: true},
|
||||
} {
|
||||
t.Run(tc.name, func(t *testing.T) {
|
||||
started := metav1.NewTime(time.Now().Add(-tc.age))
|
||||
restoreJob := &backupsv1alpha1.RestoreJob{
|
||||
ObjectMeta: metav1.ObjectMeta{Name: "rj", Namespace: ns},
|
||||
Spec: backupsv1alpha1.RestoreJobSpec{BackupRef: corev1.LocalObjectReference{Name: "bk"}},
|
||||
Status: backupsv1alpha1.RestoreJobStatus{StartedAt: &started},
|
||||
}
|
||||
pod := mountingVMDiskPod("virt-launcher-web", true)
|
||||
pod.Namespace = ns
|
||||
r := newTestRestoreJobReconcilerWithDynamic(t, []runtime.Object{makeUnstructuredHelmRelease("vm-disk-data", ns, nil)}, restoreJob.DeepCopy(), backup, pod)
|
||||
ready, res, err := r.prepareForRestore(context.Background(), restoreJob, backup, nil, target, RestoreOptions{KeepOriginalPVC: new(true)})
|
||||
if ready {
|
||||
t.Fatal("prepareForRestore went ahead while a Pod still holds the disk")
|
||||
}
|
||||
if tc.wantFail {
|
||||
if err == nil || !strings.Contains(err.Error(), "still held the disk") {
|
||||
t.Fatalf("err = %v, want a failure naming the Pod that still holds the disk", err)
|
||||
}
|
||||
return
|
||||
}
|
||||
if err != nil || res.RequeueAfter == 0 {
|
||||
t.Fatalf("requeue %v, err %v; want a requeue while within the deadline", res.RequeueAfter, err)
|
||||
}
|
||||
})
|
||||
}
|
||||
}
|
||||
Reference in new issue
Block a user