mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
A nested <name>-system release takes its install timeout from the template that renders it, not from cozystack-api, and a Package release takes it from cozystack-operator. The external-dns, openbao and qdrant templates set 10m and harbor's sets 15m; the operator gives 10m. Their asserts sat at 5m or 10m, so a failed install reported as a bare timeout. Raise each above its own floor: 11m, and 16m for harbor-test-system. harbor-test was on its 600s floor exactly, which expires first because its clock starts before the install's; it goes to 11m with the generated parents here. kafka keeps its 6m: kafka-rd sets a 7h install timeout that no budget inside the E2E job cap can clear, and the assert now says so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la>
267 lines
11 KiB
YAML
267 lines
11 KiB
YAML
# Chainsaw port of hack/e2e-apps/harbor.bats ("Create Harbor").
|
|
#
|
|
# The BATS test hand-rolled a large diagnostic dump on the harbor-test-system
|
|
# HelmRelease wait; here that becomes the test-wide catch block, which fires on
|
|
# any step failure.
|
|
apiVersion: chainsaw.kyverno.io/v1alpha1
|
|
kind: Test
|
|
metadata:
|
|
name: harbor
|
|
spec:
|
|
catch:
|
|
- events: {}
|
|
- describe:
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
name: harbor-test
|
|
- describe:
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
name: harbor-test-system
|
|
- describe:
|
|
apiVersion: objectstorage.k8s.io/v1alpha1
|
|
kind: BucketClaim
|
|
name: harbor-test-registry
|
|
- describe:
|
|
apiVersion: objectstorage.k8s.io/v1alpha1
|
|
kind: BucketAccess
|
|
name: harbor-test-registry
|
|
# The backend view: cluster-scoped Buckets plus the COSI controller pods,
|
|
# where a frozen-claim readiness propagation failure actually surfaces.
|
|
- get:
|
|
apiVersion: objectstorage.k8s.io/v1alpha1
|
|
kind: Bucket
|
|
format: yaml
|
|
- script:
|
|
content: kubectl -n cozy-objectstorage-controller get pods
|
|
# The app chart wraps the workload in a nested HelmRelease named
|
|
# `<release>-system`, and the chart behind it stamps
|
|
# app.kubernetes.io/instance from its own release name. So the Harbor pods
|
|
# carry `harbor-test-system`, and the unsuffixed value this collector used to
|
|
# hold reached none of them, on every failed run.
|
|
#
|
|
# Coverage boundary, so the absence of a component here is not read as its
|
|
# health: this selector reaches the workload of the nested release — core,
|
|
# registry, portal, jobservice, nginx, exporter, and trivy when it is
|
|
# enabled (the manifest for this suite disables it). The Redis pods carry
|
|
# `harbor-test-redis`, so this selector misses them, but the script below
|
|
# collects their logs by `rf[rs]-` name prefix. The CNPG database is covered
|
|
# by the podLogs op below, on a different label.
|
|
#
|
|
# Nothing here collects the parent chart's post-delete cleanup Job, which
|
|
# labels its pod template with the unsuffixed parent release name. That is
|
|
# not an oversight: this suite has no `delete` op, so the parent release is
|
|
# uninstalled only in Chainsaw's cleanup phase, and catch cannot observe it
|
|
# -- catch is a defer inside the step, while the cleaner runs from t.Cleanup,
|
|
# after the test function has returned. A collector on the unsuffixed value
|
|
# would be a permanently-empty capture, the defect these selectors exist to
|
|
# avoid. The qdrant suite deletes inside a `try:`, so there the Job is at
|
|
# least in scope, and that suite carries the collector with its own note on
|
|
# when it captures anything.
|
|
- podLogs:
|
|
selector: app.kubernetes.io/instance=harbor-test-system
|
|
tail: 50
|
|
# harbor-test readiness gates on this database, so its logs are the evidence
|
|
# for a whole class of failed runs. Selected on `cnpg.io/cluster`, CNPG's own
|
|
# primary label, rather than on app.kubernetes.io/instance, which CNPG only
|
|
# derives from the Cluster name: the primary label has nothing to drift from,
|
|
# and it is what the postgres suite and the WorkloadMonitors already use.
|
|
- podLogs:
|
|
selector: cnpg.io/cluster=harbor-test-db
|
|
tail: 50
|
|
# podLogs cannot report that it matched nothing, so an empty capture above and
|
|
# a selector that has gone stale look identical. Print the instance label of
|
|
# every pod in the namespace instead. Deliberately no selector is repeated
|
|
# here: a copy would only prove itself right, and would keep matching after
|
|
# the collector's own selector drifted. The census shows the values that
|
|
# actually exist, so a reader can see whether the selector above can match.
|
|
#
|
|
# What is and is not guarded, so the next reader does not assume more:
|
|
# the `-system` value above is also pinned on a resource in a `steps:` assert
|
|
# below, so the chart renaming that label turns a step red -- that is the
|
|
# drift this pin exists to catch. It does NOT catch a wrong value typed into the
|
|
# selector here: the assert pins the label on the resource, this pins it in
|
|
# the collector, and they are two literals. It also catches nothing when the
|
|
# run fails before reaching that assert. The census below covers neither, and
|
|
# is not meant to -- it answers what the labels actually are.
|
|
# The `cnpg.io/cluster` selector has no pin at all; it rides on CNPG's
|
|
# primary label instead, which is the reason for choosing it.
|
|
- script:
|
|
# The helper bounds each kubectl attempt with --request-timeout, which on
|
|
# a degraded cluster costs about five times the flag; one call is ~75s at
|
|
# worst, and this op timeout sits above that so an inner budget is what
|
|
# fires. A SIGKILL at the op boundary would take the census out mid-write
|
|
# and yield nothing, where a kubectl that gives up leaves an error to
|
|
# print. The helper carries the rest of the rationale and is unit-tested
|
|
# in hack/pod-label-census_test.bats.
|
|
timeout: 2m
|
|
content: |
|
|
set -eu
|
|
cd ../../..
|
|
. hack/e2e-chainsaw/_lib/pod-label-census.sh
|
|
pod_label_census "$NAMESPACE"
|
|
# harbor-test readiness is gated by the inline RedisFailover
|
|
# harbor-test-redis: its three sentinels stay NotReady until the spotahome
|
|
# redis-operator elects a master and rewrites their monitored-master config.
|
|
# None of the collectors above see that path: the instance selector above
|
|
# matches neither the operator (namespace cozy-redis-operator) nor the
|
|
# rfr-/rfs- pods, which carry `harbor-test-redis`, and the RedisFailover CRD
|
|
# ships no status subresource, so there is nothing to assert on declaratively.
|
|
- describe:
|
|
apiVersion: databases.spotahome.com/v1
|
|
kind: RedisFailover
|
|
name: harbor-test-redis
|
|
- podLogs:
|
|
namespace: cozy-redis-operator
|
|
selector: app.kubernetes.io/name=redis-operator
|
|
tail: 200
|
|
- script:
|
|
# A script rather than more podLogs entries: the -o wide pod tables and
|
|
# the rfr-/rfs- name-prefix match (independent of the operator's
|
|
# pod-label scheme) are not expressible declaratively. Every command is
|
|
# best-effort (|| true) so diagnostics never mask the test's own error;
|
|
# no inner timeout wrappers, so the 2m op timeout is the only budget.
|
|
timeout: 2m
|
|
content: |
|
|
set -u
|
|
echo "== redis-operator pods (cozy-redis-operator) =="
|
|
kubectl -n cozy-redis-operator get pods -o wide || true
|
|
echo "== rfr-/rfs- pod states ($NAMESPACE) =="
|
|
kubectl -n "$NAMESPACE" get pods -o wide | grep -E 'rf[rs]-harbor-test-redis' || true
|
|
for p in $(kubectl -n "$NAMESPACE" get pods -o name | grep -E 'rf[rs]-harbor-test-redis'); do
|
|
echo "== logs $p =="
|
|
kubectl -n "$NAMESPACE" logs "$p" --all-containers --tail=100 --prefix || true
|
|
done
|
|
steps:
|
|
- name: create-harbor
|
|
try:
|
|
- apply:
|
|
file: harbor.yaml
|
|
|
|
- name: wait-helmrelease-ready
|
|
try:
|
|
# harbor-test readiness is gated by the inline RedisFailover, whose three
|
|
# sentinels only go Ready after the freshworks-oss redis-operator elects a
|
|
# master and rewrites their config; normal convergence is 2-3m, but on a
|
|
# CPU-committed runner the tail runs long (steal-induced probe misses,
|
|
# documented in pull-requests.yaml). 11m sits above the 600s install
|
|
# timeout cozystack-api gives this release, so a failed install has
|
|
# recorded its reason first; see "Sizing an HR-Ready budget" in
|
|
# docs/agents/e2e-testing.md. On a timeout, the catch block's
|
|
# redis-operator log shows which reconcile gate stalled.
|
|
- assert:
|
|
timeout: 11m
|
|
resource:
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: harbor-test
|
|
status:
|
|
(conditions[?type == 'Ready']):
|
|
- status: "True"
|
|
|
|
- name: wait-cosi-bucket
|
|
try:
|
|
# 2m, not the default 5m: the objectstorage-controller requeues the
|
|
# BucketClaim until the backend Bucket is ready, so readiness converges
|
|
# within tens of seconds; a claim still not ready after 2m is frozen
|
|
# (readiness-propagation regression), and the tight bound surfaces that
|
|
# instead of masking it (mirrors the bats 120s budget from 63d14e748).
|
|
- assert:
|
|
timeout: 2m
|
|
resource:
|
|
apiVersion: objectstorage.k8s.io/v1alpha1
|
|
kind: BucketClaim
|
|
metadata:
|
|
name: harbor-test-registry
|
|
status:
|
|
bucketReady: true
|
|
- assert:
|
|
timeout: 1m
|
|
resource:
|
|
apiVersion: objectstorage.k8s.io/v1alpha1
|
|
kind: BucketAccess
|
|
metadata:
|
|
name: harbor-test-registry
|
|
status:
|
|
accessGranted: true
|
|
|
|
- name: wait-system-and-deployments
|
|
try:
|
|
# 16m: this release is rendered by
|
|
# packages/apps/harbor/templates/harbor.yaml, whose spec.timeout of 15m is
|
|
# its install timeout; see "Sizing an HR-Ready budget" in
|
|
# docs/agents/e2e-testing.md.
|
|
- assert:
|
|
timeout: 16m
|
|
resource:
|
|
apiVersion: helm.toolkit.fluxcd.io/v2
|
|
kind: HelmRelease
|
|
metadata:
|
|
name: harbor-test-system
|
|
status:
|
|
(conditions[?type == 'Ready']):
|
|
- status: "True"
|
|
- assert:
|
|
timeout: 2m
|
|
resource:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: harbor-test-core
|
|
# Pins the label the catch collector selects on. Harbor's matchLabels
|
|
# carry only release+app, so pin the pod template, which is what
|
|
# podLogs actually matches. Asserts are subset matches, so this costs
|
|
# nothing and turns a silent drift of the instance label into a red
|
|
# step instead of an empty capture.
|
|
spec:
|
|
template:
|
|
metadata:
|
|
labels:
|
|
app.kubernetes.io/instance: harbor-test-system
|
|
status:
|
|
(conditions[?type == 'Available']):
|
|
- status: "True"
|
|
- assert:
|
|
timeout: 2m
|
|
resource:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: harbor-test-registry
|
|
status:
|
|
(conditions[?type == 'Available']):
|
|
- status: "True"
|
|
- assert:
|
|
timeout: 2m
|
|
resource:
|
|
apiVersion: apps/v1
|
|
kind: Deployment
|
|
metadata:
|
|
name: harbor-test-portal
|
|
status:
|
|
(conditions[?type == 'Available']):
|
|
- status: "True"
|
|
|
|
- name: verify-credentials-and-service
|
|
try:
|
|
- assert:
|
|
resource:
|
|
apiVersion: v1
|
|
kind: Service
|
|
metadata:
|
|
name: harbor-test
|
|
spec:
|
|
(ports[0].port): 80
|
|
- script:
|
|
content: |
|
|
set -eu
|
|
pw=$(kubectl -n "$NAMESPACE" get secret harbor-test-credentials \
|
|
-o jsonpath='{.data.admin-password}' | base64 -d)
|
|
url=$(kubectl -n "$NAMESPACE" get secret harbor-test-credentials \
|
|
-o jsonpath='{.data.url}' | base64 -d)
|
|
test -n "$pw"
|
|
echo "$url" | grep -q 'https://'
|
|
check:
|
|
($error == null): true
|