admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-09 14:02:56 +09:00
Code Issues Packages Projects Releases Wiki Activity
Files
f2a611788e21bc4bdbfb9a216a59ef178198ffcc
cozystack/Makefile
T
Alexey Artamonov 42a27567ce fix(fluxcd): keep Helm uninstall working during flux-plunger recovery 
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.

Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.

Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.

Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.

Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.

Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.

Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.

Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.

A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.

Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-15 23:34:52 +03:00

321 lines
15 KiB
Makefile
Raw Blame History

.PHONY: manifests assets prepare-env prepare-env-container unit-tests helm-unit-tests bats-unit-tests bats-unit-files-check bats-posix-compat-tests print-bats-unit-files print-bats-posix-compat-files print-bats-jobs rd-presets-check migrations-target-check test test-controllers test-controllers-envtest test-backport-audit preflight
include hack/common-envs.mk
build-deps:
@command -V find docker skopeo jq gh helm > /dev/null
@yq --version | grep -q "mikefarah" || (echo "mikefarah/yq is required" && exit 1)
@tar --version | grep -q GNU || (echo "GNU tar is required" && exit 1)
@sed --version | grep -q GNU || (echo "GNU sed is required" && exit 1)
@awk --version | grep -q GNU || (echo "GNU awk is required" && exit 1)
build: build-deps
make -C packages/apps/mariadb image
make -C packages/apps/clickhouse image
make -C packages/apps/kubernetes image
make -C packages/apps/vpn image
make -C packages/system/cozystack-api image
make -C packages/system/cozystack-controller image
make -C packages/system/backup-controller image
make -C packages/system/backupstrategy-controller image
make -C packages/system/lineage-controller-webhook image
make -C packages/system/migration-controller image
make -C packages/system/flux-shard-operator image
make -C packages/system/flux-plunger image
make -C packages/system/cilium image
make -C packages/system/linstor image
make -C packages/system/linstor-gui image
make -C packages/system/kubeovn image
make -C packages/system/kubeovn-webhook image
make -C packages/system/kubeovn-plunger image
make -C packages/system/kilo image
make -C packages/system/dashboard image
make -C packages/system/metallb image
make -C packages/system/talos-log-collector image
make -C packages/system/kamaji image
make -C packages/system/capi-providers-cpprovider image
make -C packages/system/capi-providers-infraprovider image
make -C packages/system/multus image
make -C packages/system/bucket image
make -C packages/system/objectstorage-controller image
make -C packages/system/securitygroup-controller image
make -C packages/system/grafana-operator image
make -C packages/system/monitoring image
make -C packages/system/redis-operator image
make -C packages/system/harbor image
make -C packages/system/velero image
make -C packages/system/opensearch-operator image
make -C packages/system/keycloak-operator image
make -C packages/core/testing image
make -C packages/core/talos image
make -C packages/core/platform image
make -C packages/core/installer image
make manifests
manifests:
mkdir -p _out/assets
cat internal/crdinstall/manifests/*.yaml > _out/assets/cozystack-crds.yaml
# kubectl-apply install path: render the bare Namespace resource alongside
# the operator. bareNamespace=true gates a Namespace cozy-system with PSA +
# identity labels (see packages/core/installer/templates/cozy-system-namespace.yaml).
# helm install/upgrade users keep the default (false) and use --create-namespace
# + the pre-install labeler hook.
# platformVersion is rendered into the COZYSTACK_VERSION env on the operator so
# the install artifacts carry the version as deploy-time data (not baked into
# the image). Promotion re-renders these with the stable version, no rebuild.
# Talos variant (default)
helm template installer packages/core/installer -n cozy-system \
--set bareNamespace=true \
--set cozystackOperator.platformVersion=$(if $(COZYSTACK_VERSION),v$(COZYSTACK_VERSION),) \
--show-only templates/cozy-system-namespace.yaml \
--show-only templates/cozystack-operator.yaml \
> _out/assets/cozystack-operator-talos.yaml
# Generic Kubernetes variant (k3s, kubeadm, RKE2)
helm template installer packages/core/installer -n cozy-system \
--set bareNamespace=true \
--set cozystackOperator.variant=generic \
--set cozystackOperator.platformVersion=$(if $(COZYSTACK_VERSION),v$(COZYSTACK_VERSION),) \
--set cozystack.apiServerHost=REPLACE_ME \
--show-only templates/cozy-system-namespace.yaml \
--show-only templates/cozystack-operator.yaml \
> _out/assets/cozystack-operator-generic.yaml
# Hosted variant (managed Kubernetes)
helm template installer packages/core/installer -n cozy-system \
--set bareNamespace=true \
--set cozystackOperator.variant=hosted \
--set cozystackOperator.platformVersion=$(if $(COZYSTACK_VERSION),v$(COZYSTACK_VERSION),) \
--show-only templates/cozy-system-namespace.yaml \
--show-only templates/cozystack-operator.yaml \
> _out/assets/cozystack-operator-hosted.yaml
cozypkg:
go build -ldflags "-X github.com/cozystack/cozystack/cmd/cozypkg/cmd.Version=v$(COZYSTACK_VERSION)" -o _out/bin/cozypkg ./cmd/cozypkg
# Operator diagnostic: reports non-ready cozystack / Flux / Kubernetes resources.
check-readiness:
go build -ldflags "-X main.Version=v$(COZYSTACK_VERSION)" -o _out/bin/check-readiness ./cmd/check-readiness
assets: assets-talos assets-cozypkg openapi-json
openapi-json:
mkdir -p _out/assets
VERSION=$(if $(COZYSTACK_VERSION),v$(COZYSTACK_VERSION),$(shell git describe --tags --always 2>/dev/null || echo dev)) go run ./tools/openapi-gen/ 2>/dev/null > _out/assets/openapi.json
assets-talos:
make -C packages/core/talos assets
assets-cozypkg: assets-cozypkg-linux-amd64 assets-cozypkg-linux-arm64 assets-cozypkg-darwin-amd64 assets-cozypkg-darwin-arm64 assets-cozypkg-windows-amd64 assets-cozypkg-windows-arm64
(cd _out/assets/ && sha256sum cozypkg-*.tar.gz) > _out/assets/cozypkg-checksums.txt
assets-cozypkg-%:
$(eval EXT := $(if $(filter windows,$(firstword $(subst -, ,$*))),.exe,))
mkdir -p _out/assets
GOOS=$(firstword $(subst -, ,$*)) GOARCH=$(lastword $(subst -, ,$*)) go build -ldflags "-X github.com/cozystack/cozystack/cmd/cozypkg/cmd.Version=v$(COZYSTACK_VERSION)" -o _out/bin/cozypkg-$*/cozypkg$(EXT) ./cmd/cozypkg
cp LICENSE _out/bin/cozypkg-$*/LICENSE
tar -C _out/bin/cozypkg-$* -czf _out/assets/cozypkg-$*.tar.gz LICENSE cozypkg$(EXT)
test:
make -C packages/core/testing apply
make -C packages/core/testing e2e
unit-tests: helm-unit-tests bats-unit-tests bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check test-check-readiness test-backport-audit migrations-target-check
helm-unit-tests:
hack/helm-unit-tests.sh
# Pin the resourcesPreset enum in every ApplicationDefinition openAPISchema
# to the canonical 47-value set (40 instance-type names + 7 legacy aliases).
# Catches the regression where one chart's Makefile forgets to invoke
# hack/update-crd.sh and that RD's schema drifts from values.schema.json.
rd-presets-check:
hack/check-rd-presets.sh
# Catch the off-by-one where migrations/<N> is added but
# migrations.targetVersion in packages/core/platform/values.yaml is not
# bumped to >= N+1. run-migrations.sh loops `seq $$CURRENT $$((TARGET-1))`,
# so the new migration is silently skipped on every cluster upgrade.
migrations-target-check:
hack/check-migrations-target.sh
# Scoped go test over the cozystack-api surface that this repo owns. Kept
# narrow intentionally - running `go test ./...` pulls in generated code
# round-trip suites whose behavior depends on tool versions outside this
# repo's control (kubebuilder, openapi-gen, etc.) and is better exercised
# from their generator workflows.
go-unit-tests:
go test ./pkg/registry/... ./pkg/config/... ./pkg/cmd/server/...
# The nested Go modules (image sources and the published API types) are
# outside ./... of the root module, so no other target reaches their tests.
# git ls-files, not find: gitignored checkouts such as .claude/worktrees
# hold whole copies of this repository.
go-module-tests:
@for mod in $$(git ls-files '*/go.mod' | xargs -n1 dirname); do \
echo "--- go test $$mod ---"; \
(cd "$$mod" && go test -count=1 ./...) || exit 1; \
done
# Go tests for the controllers and supporting packages under ./internal.
# Excludes ./pkg/... and ./cmd/... — those are run separately by
# go-unit-tests above (pkg subset) and skipped (cmd) until their tests
# stabilise. CI schedules this target in the same four-slot make invocation as
# unit-tests; locally invoke it directly or chain the two targets.
test-controllers:
go test ./internal/... -count=1
# Controller suites that need a real apiserver (build tag `envtest`): server-side
# apply ownership, apply preconditions and finalizer handling, which a fake client
# does not implement. setup-envtest fetches the kube-apiserver and etcd release
# pinned by ENVTEST_K8S_VERSION into _out/envtest; the HelmRelease CRD comes from
# the in-tree Flux manifests (envtest skips their non-CRD objects). CI runs it
# next to test-controllers. Only the packages listed below are compiled, so a
# package that gains an envtest suite has to be added here.
#
# setup-envtest itself is built into _out/envtest too, and binaries already there
# are used without asking the release index, so a warm _out/envtest (CI caches
# it) needs no network at all.
ENVTEST_K8S_VERSION ?= 1.35.0
SETUP_ENVTEST_VERSION ?= v0.0.0-20260305142021-f9589b9f2b9d
ENVTEST_DIR = $(CURDIR)/_out/envtest
SETUP_ENVTEST = $(ENVTEST_DIR)/bin/setup-envtest-$(SETUP_ENVTEST_VERSION)
$(SETUP_ENVTEST):
GOBIN=$(ENVTEST_DIR)/bin go install sigs.k8s.io/controller-runtime/tools/setup-envtest@$(SETUP_ENVTEST_VERSION)
mv $(ENVTEST_DIR)/bin/setup-envtest $@
test-controllers-envtest: $(SETUP_ENVTEST)
KUBEBUILDER_ASSETS="$$($(SETUP_ENVTEST) use $(ENVTEST_K8S_VERSION) --installed-only --bin-dir $(ENVTEST_DIR) --print path 2>/dev/null || $(SETUP_ENVTEST) use $(ENVTEST_K8S_VERSION) --bin-dir $(ENVTEST_DIR) --print path)" \
HELMRELEASE_CRD_DIR=$(CURDIR)/internal/fluxinstall/manifests \
go test -tags envtest -run '^TestEnvtest' ./internal/controller/fluxplunger/... -count=1
# Black-box golden test for cmd/check-readiness. Builds the binary and runs it
# against a mock kubectl (fixtures under test/check-readiness/testdata),
# diffing stdout/stderr/exit against golden files. Regenerate goldens with:
# go test ./test/check-readiness/ -update
test-check-readiness:
go test ./test/check-readiness/ -count=1
# ./cmd/... is excluded from go-unit-tests, so the audit needs its own target.
test-backport-audit:
go test ./cmd/backport-audit/ -count=1
# Unit discovery is one level deep. Live-cluster files need an invocation in
# packages/core/testing; hack/bats-runner-coverage.bats checks that boundary.
# Bats supplies TAP and JUnit output; the live-cluster runner keeps its tracing.
#
# Caveat: $(wildcard ...) returns space-separated names, so a filename
# containing a literal space would split into multiple tokens here. All
# current bats files use hyphen-separated names; if the project ever
# introduces whitespace-bearing filenames this recipe must be rewritten
# (e.g. to use `find ... -print0 | xargs -0`).
BATS_UNIT_FILES := $(filter-out hack/e2e-%.bats,$(wildcard hack/*.bats))
# The same list, one file per line, for hack/bats-strict-setup.bats. Every unit
# file has to load hack/test_helper.bash to get `set -u` back, and that audit is
# only worth anything if the set it walks is the set that actually runs -- so it
# asks here rather than keeping a second copy of the filter above.
print-bats-unit-files:
@printf '%s\n' $(BATS_UNIT_FILES)
# `bats -j` needs GNU parallel and exits non-zero rather than degrading when it
# is missing. moreutils also ships a command named parallel, so identify the GNU
# implementation from its version output before enabling concurrency. Resolve
# the probe once per make process; `?=` would keep the `$(shell ...)` recursive
# and rerun it at every expansion.
ifndef BATS_JOBS
BATS_JOBS := $(shell parallel --version 2>/dev/null | grep -q '^GNU parallel ' && nproc 2>/dev/null || echo 1)
endif
print-bats-jobs:
@printf '%s\n' "$(BATS_JOBS)"
# JUnit XML is preserved as a CI artifact for inspection. _out is gitignored.
BATS_REPORT_DIR ?= _out/test-reports
bats-unit-files-check:
@if [ -z "$(BATS_UNIT_FILES)" ]; then \
echo "ERROR: no hack/*.bats unit test files found"; \
exit 1; \
fi
bats-unit-tests: bats-unit-files-check
@command -v bats >/dev/null 2>&1 || { \
echo "ERROR: bats not found. Install bats-core >= 1.5 — https://bats-core.readthedocs.io"; \
exit 1; \
}
@mkdir -p "$(BATS_REPORT_DIR)"
bats -j $(BATS_JOBS) --report-formatter junit -o "$(BATS_REPORT_DIR)" $(BATS_UNIT_FILES)
# Real Bats is authoritative. Unit files that source production code whose
# contract is POSIX sh retain a compatibility pass through cozytest.sh's
# /bin/sh translator. Discover literal .sh dependencies wherever they live,
# including hack/lib and package migration helpers, so a new non-Chainsaw
# helper test cannot run only under Bash. The explicit entries retain reviewed
# shell-facing tests, including files whose production path is held in a
# variable and therefore cannot be identified from the source line alone.
BATS_SOURCED_SH_FILES := $(shell grep -El '^[[:space:]]*(\.|source)[[:space:]]+.*\.sh' $(BATS_UNIT_FILES))
BATS_POSIX_COMPAT_FILES := $(sort \
$(BATS_SOURCED_SH_FILES) \
hack/capture-dataplane.bats \
hack/capture-previous-logs.bats \
hack/cilium-leak-healer_test.bats \
hack/container-lane-capacity_test.bats \
hack/cozyreport-talos.bats \
hack/cozyreport.bats \
hack/cozystack-version-stamp.bats \
hack/nightly-mirror_test.bats \
hack/pod-label-census_test.bats \
hack/promote-rewrite-tags_test.bats \
hack/runner-identity.bats \
hack/seaweedfs-naming-audit.bats \
)
# CI's /bin/sh is dash. Keep the interpreter overridable so contributors can
# run the same compatibility lane explicitly on hosts whose /bin/sh differs.
BATS_POSIX_SHELL ?= /bin/sh
COZYTEST_TRACE ?= 0
print-bats-posix-compat-files:
@printf '%s\n' $(BATS_POSIX_COMPAT_FILES)
bats-posix-compat-tests:
@status=0; \
for f in $(BATS_POSIX_COMPAT_FILES); do \
echo "--- running POSIX compatibility: $$f ---"; \
COZYTEST_TRACE=$(COZYTEST_TRACE) "$(BATS_POSIX_SHELL)" hack/cozytest.sh "$$f" || status=1; \
done; \
exit $$status
# No cluster for the unit and controller lanes. A test that misses a stub, or
# whose stub a `timeout` or `bash -c` steps past, otherwise reaches whatever
# cluster the caller's kubeconfig points at. An empty KUBERNETES_SERVICE_HOST
# is needed beside KUBECONFIG: given an empty kubeconfig, client-go falls back
# to in-cluster credentials wherever that variable and a service-account token
# are present.
# Every prerequisite of unit-tests is listed so it is guarded when run on its
# own too; hack/unit-lane-no-cluster.bats fails when one is missing.
NO_CLUSTER_TARGETS := unit-tests test-controllers helm-unit-tests bats-unit-tests \
bats-posix-compat-tests go-unit-tests go-module-tests rd-presets-check \
test-check-readiness test-backport-audit migrations-target-check
$(NO_CLUSTER_TARGETS): export KUBECONFIG := /dev/null
$(NO_CLUSTER_TARGETS): export KUBERNETES_SERVICE_HOST :=
# Operator-facing host preflight check. Warns about a standalone
# containerd.service or docker.service running alongside the embedded
# k3s runtime. Safe to run at any time; always exits 0.
preflight:
@hack/check-host-runtime.sh
prepare-env:
make -C packages/core/testing apply
make -C packages/core/testing prepare-cluster
# Container-lane sibling of prepare-env. Same sandbox, Talos nodes as containers
# instead of QEMU guests, so no nocloud asset is copied in.
prepare-env-container:
make -C packages/core/testing apply
make -C packages/core/testing prepare-cluster-container
generate:
hack/update-codegen.sh
upload_assets: manifests
hack/upload-assets.sh
Reference in new issue View Git Blame Copy Permalink