Migration slot 45 holds a different migration here than on main and
release-1.6. On this branch it is the seaweedfs db-split repair, added
after v1.5.3 by `33611dfe7` (the #3370 backport), which in the same
commit bumped `migrations.targetVersion` from 45 to 46. On main and 1.6
slot 45 is the `helm.sh/resource-policy=keep` pin on
`KubeadmConfigTemplate`, and the seaweedfs repair sits at slot 53
instead.
Nothing is broken today. v1.5.0 through v1.5.3 all ship `targetVersion:
45`, so no released cluster is stamped past it and every one of them
still runs 1.6's slot 45 on the way up. Cutting v1.5.4 is what first
ships 46. After that a 1.5.4 to 1.6 upgrade runs `seq 46 53` and never
executes 1.6's slot 45, so helm prunes `KubeadmConfigTemplate` while the
live kubeadm-backed MachineSet still references it,
`bootstrap.configRef` dangles and controller-manager floods through the
talos worker rollover. Tenant kubernetes only, noisy broken upgrade
rather than data loss.
This folds the keep-pin into release-1.5's slot 45 so it does both jobs,
and puts the annotation on the rendered template in
`packages/apps/kubernetes/templates/cluster.yaml` so templates created
after the upgrade are born pinned. `targetVersion` stays 46.
### Why uninstall does not leak
`keep` suppresses only helm's own delete. CAPI stamps an
`ownerReference` to the `Cluster` on this object, which the chart does
not render (checked on a live v1.5 stand, and `MachineDeployment` and
`KubevirtMachineTemplate` carry the same reference). The `Cluster`
itself is helm-managed with no `keep`, so `helm uninstall` deletes it
and GC reclaims the template through that reference.
Template name is deterministic rather than content-hashed, so `keep`
cannot accumulate a copy per upgrade. One residue is the template of a
removed nodeGroup, which helm no longer prunes. It is inert, re-adopted
in place if the group comes back, and GC takes it with the `Cluster`.
### Tests
Mutation-proven rather than revert-tested. The one worth naming:
swapping the selector to the `meta.helm.sh/release-name` annotation
makes the migration exit 0 reporting `pinned=0 failures=0`, which looks
like a clean run, and the test catches it because the annotation never
reached the object.
Also converts four pre-existing negated-grep assertions in
`hack/seaweedfs-guard-parity.bats`. `! grep -q X f` cannot fail, posix
and bash both exempt a `!`-negated pipeline from errexit, so those were
comments that look like assertions. All of them hold once they are real.
v1.5.4 is cut from this branch, and every job on the release path is
pinned to `[self-hosted]`, a pool that no longer exists. `gh api
repos/cozystack/cozystack/actions/runners` returns `total_count: 0` and
the last confirmed execution anywhere was 2026-07-07. A job pinned to a
label nothing answers does not fail, it sits in `Queued` with no logs,
so the release would appear to hang with nothing red to point at.
Three commits.
`tags.yaml`, three jobs, to what `release-1.6` uses for the same job
names: `prepare-release` to `oracle-vm-24cpu-96gb-x86-64`,
`generate-changelog` and `update-website-docs` to `ubuntu-latest`.
`pull-requests-release.yaml` `finalize` to
`[oracle-vm-4cpu-16gb-x86-64]`, which both live branches agree on. This
one matters more than it looks. This branch still carries the `Create
pull request if not exists` step that 1.6 deleted, so the cut opens a
`release`-labelled PR whose merge fires `finalize`, and `finalize` is
what runs `Publish draft release`. Left queued, v1.5.4 builds, drafts,
and then sits unpublished.
`prepare-release` gets the `Set up build toolchain` step, `run:` body
byte-identical to the one on `main` and `release-1.6`, because `make
build` ends at `packages/core/installer image-packages` calling `flux
push artifact` and flux is not on the ephemeral image. Only the `if:`
differs and it has to: this branch's guard sets `skip` where 1.6 renamed
the output `release_exists`.
Gate, stated as a ceiling rather than a pass. Both files parse, plus a
parsed-yaml assertion that the new step lands between `Checkout code`
and `Login to GHCR`. `pre-commit` cannot pass in a linked worktree in
this repo at all, the `run-make-generate` hook does `flock -x
.git/pre-commit.lock` and `.git` is a file there, exit 66, reproduced
against an untouched file. zizmor does not exist on this branch, no hook
and no config, so it was run by hand instead and reports no new
findings.
Deliberately not fixed: `release-e2e.yaml` is on the same dead pool and
will silently queue rather than go red. It gates nothing by its own
header, and both live branches replaced it with `e2e-tag.yaml`.
`make build` ends at packages/core/installer, whose `image-packages`
target shells out to `flux push artifact`. The decommissioned
self-hosted runner had flux baked in; the ephemeral shape does not, so
without this the release build works through all 28 packages and then
fails on the last one -- roughly an hour in, with the tag already pushed.
Port the step release-1.6's tags.yaml gained for this same job when it
made the same migration. The command is byte-identical to 1.6's and to
the equivalents already in build-main.yaml and pull-requests.yaml on this
branch. Only the `if:` gate differs, and it has to: 1.6 renamed the
check_release output to `release_exists`, while this branch's step sets
`skip` -- the expression here matches its sibling steps.
flux is the only tool 1.6 installs for this job, so nothing else is
carried over.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
`finalize` is the job that turns a prepared release into a published one:
it creates the tag on the merge commit, ensures the release-X.Y
maintenance branch, and publishes the draft release. It was pinned to
`[self-hosted]`, the decommissioned cozy-runner-1 (#2937), which nothing
answers -- so a v1.5.4 cut would build, draft, open its release PR, and
then stall silently on merge with the release left as a draft nobody
notices.
Repoint to `[oracle-vm-4cpu-16gb-x86-64]`, which both `main` and
`release-1.6` use for this same job and which is confirmed answered
(jobs executed on it in this repo today). This branch's finalize does
git and GitHub API work only -- no skopeo, no helm, no docker -- so the
small shape is ample.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
All three jobs in tags.yaml were pinned to `[self-hosted]`, which is the
decommissioned cozy-runner-1 (#2937). No self-hosted runner answers that
label any more: the repo has zero registered runners, and the workflow
would sit in `Queued` forever rather than fail -- a tag push meant to cut
v1.5.4 would appear to hang with nothing in CI explaining why.
Repoint to what release-1.6's tags.yaml uses for the same three jobs:
prepare-release -> oracle-vm-24cpu-96gb-x86-64
generate-changelog -> ubuntu-latest
update-website-docs -> ubuntu-latest
prepare-release runs a serial `make build` plus `make assets`, so it
needs the large ephemeral Oracle shape -- the same one build-main.yaml on
this branch already moved to, and which is confirmed live here (the
release-1.5 "Build cache" run executed on it). The other two are git,
node and CLI work against the GitHub and website APIs with no docker and
no repo toolchain, so a GitHub-hosted runner is enough.
Runner labels only. This branch is retired after one final release, so
nothing else from release-1.6's newer tags.yaml is ported.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since botocore ~1.36 (early 2025) the default
RequestChecksumCalculation is when_supported, so barman-cloud's boto3
attaches a flexible checksum to every PutObject. AWS S3 accepts it,
but non-AWS S3-compatible backends - Ceph RADOS Gateway, the
platform's own SeaweedFS system bucket, some MinIO / Cloudflare R2
builds - reject it with "InvalidArgument: x-amz-content-sha256 must
be UNSIGNED-PAYLOAD, ...", which fails every backup and WAL-archive
upload.
Set AWS_REQUEST_CHECKSUM_CALCULATION=when_required through the CNPG
Cluster's spec.env, which reaches the instance pods and therefore the
barman-cloud subprocess the instance manager execs.
This is a release-1.5-native equivalent of #3417 rather than a
cherry-pick of it. That change sets
spec.instanceSidecarConfiguration.env on barmancloud.cnpg.io
ObjectStore objects, and release-1.5 has no barman-cloud plugin:
there is no ObjectStore CRD on the branch and the rendered Cluster
carries no spec.plugins, so those objects would template cleanly and
then fail to apply, referenced by nothing. release-1.5 still uses
CNPG's native in-tree spec.backup.barmanObjectStore, which shells out
to the same barman-cloud client, so the bug is reachable here - the
plugin is the packaging difference, not the cause.
One spec.env covers all three paths on this branch at once: the
chart-rendered legacy barmanObjectStore, the same field SSA-patched
by the CNPG backup driver in the useSystemBucket flow, and
externalClusters recovery. No Go change is needed - the driver's
patch type carries only spec.backup and spec.bootstrap, so it cannot
express or remove the Helm-owned spec.env.
keycloak is out of scope on this branch. Its chart renders a CNPG
Cluster but configures no backup at all - no backup key in values,
no backup template - so it has no barman-cloud path to fix, and the
upstream hunk for it references values that do not exist here.
Verified against CNPG 1.27.3: spec.env is accepted by the webhook
and stored unmodified, appears in the instance manager's own
environment, and botocore inside the instance image then resolves
request_checksum_calculation as when_required instead of its
when_supported default. Adding the field to an already-running
Cluster converges back to healthy after CNPG recreates the instance
pod, which is the in-place upgrade path for existing releases with
backups enabled.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Backport of #3409 to release-1.5.
gateway-hostname-policy.yaml and route-hostname-policy.yaml rendered
their ValidatingAdmissionPolicy and binding documents unconditionally,
so installing the chart against an API server that does not serve
admissionregistration.k8s.io/v1/ValidatingAdmissionPolicy failed.
Guard both on the capability, matching the existing pattern in
packages/core/platform/templates/deletion-protection.yaml, and declare
the capability in the two rendering test suites so they keep exercising
the policies.
hostname-policies-capability-gate_test.yaml pins the absent-API case at
suite level, since helm-unittest merges per-test capabilities into the
suite's rather than replacing them.
The upstream change also gated a third template,
ingress-hostname-policy.yaml. That legacy-Ingress hostname policy
landed on main in d89ad7069 and was never backported, so it does not
exist on release-1.5 and is deliberately excluded here: importing it
would ship an unreviewed admission policy that starts rejecting tenant
Ingress hostnames in a patch release.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The chart ships tests/delete_test.yaml, but the package Makefile has no
`test:` target, and hack/helm-unit-tests.sh runs a package's suite only
when `make -C <dir> -n test` succeeds. The suite was therefore never
executed on this branch, so the namespace assertion added alongside the
teardown fix shipped as dead weight.
Add the target in the same form the other packages use. On main the same
three lines arrived via an unrelated login-theme commit rather than with
the fix itself, which is why the backport landed without them.
Verified by mutating the rendered patch back to the pre-fix hardcoded
`-n cozy-system`: all three tests go red, so the suite asserts something.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The four assertions in "neither chart classifies on mutable claim
timestamps or liveness" were written as `! grep -qF X "$f"`, which cannot
fail: POSIX and bash both exempt a !-negated pipeline from errexit — "the
-e setting shall be ignored ... if the command's return value is being
inverted with !" — so the command runs, returns 1, and the test carries on
reporting success. Those four were the entire body of the test, so it
asserted nothing whatsoever.
Converting them mechanically turns out to fail, and for a reason worth
keeping: all four names DO appear in both files, in the prose that
explains why each was rejected as a discriminator. `creationTimestamp` and
`readyReplicas` are discussed at length in the comment blocks that record
the decision. So a raw grep would fail on the documentation of the very
invariant the test enforces.
Assert against the template logic instead, with {{/* */}} comment blocks
stripped by a small awk helper. Every comment opener in both files starts
its own line, so dropping whole lines cannot take code with it. What the
test now forbids is a LIVE reference, which is what it always meant.
Proven in both directions: it passes with the explanatory prose in place,
and it fails when a real `readyReplicas` lookup is injected into live
template code in either file.
The awk helper is deliberately a one-liner. cozytest.sh rewrites any bare
`}` in column 0 into `return 0` + `}`, which would corrupt a multi-line
awk body.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Platform migration 45 pins the KubeadmConfigTemplate objects that exist
when a cluster runs it. That leaves a hole: a template created AFTERWARDS
— a new tenant Kubernetes cluster, or a nodeGroup added while on 1.5.4 —
is born without the annotation, and 1.6's own slot 45 never runs for that
cluster because a v1.5.4 cluster is stamped 46 and runs `seq 46 53`. The
1.6 kubernetes chart upgrade then prunes the unpinned template while the
kubeadm-backed MachineSet still references it from bootstrap.configRef,
which is the bug the migration exists to prevent.
Render the annotation in the chart so every template is born pinned. The
migration then covers only the objects that predate it, and the two
together leave no window.
KEEP IS SAFE ON THE UNINSTALL PATH, which is the only behaviour it
changes here. CAPI stamps an ownerReference to the Cluster on this object
— verified on a live v1.5 stand, where KubeadmConfigTemplate,
MachineDeployment and KubevirtMachineTemplate all carry
ownerReferences: [Cluster/<release>] that the chart itself does not
render — and the Cluster is Helm-managed with no keep of its own. So
`helm uninstall` deletes the Cluster and Kubernetes garbage collection
reclaims the template through that reference. keep suppresses Helm's own
delete; it has no bearing on owner-driven GC, so uninstall does not leak.
The name is deterministic, <release>-<nodeGroup>, not content-hashed the
way KubevirtMachineTemplate is, so keep cannot accumulate a copy per
upgrade. The one residue is a removed nodeGroup, whose template Helm no
longer prunes: it is inert because nothing references it, it is re-adopted
in place if that nodeGroup returns, and GC reclaims it with the Cluster.
Not added to KubeadmConfig: this chart renders no bare KubeadmConfig, and
the ones CAPI spawns from the template are owned by their Machine and
carry no app.kubernetes.io/managed-by label, so Helm never manages or
prunes them and the annotation would have nothing to act on. Confirmed on
the same live stand.
Nothing else in this chart sets helm.sh/resource-policy, so no existing
conditional is being overridden.
Tests assert the annotation on the rendered object for a single nodeGroup
and for every group when two are configured — the second because the
annotation has to sit inside the per-nodeGroup range, and a group added
later is exactly the one that would otherwise be left prunable. Both were
confirmed red before being made to pass: with the annotation removed from
the template they fail on `unknown path
metadata.annotations["helm.sh/resource-policy"]`, which is the object
being born unpinned rather than a broken assertion.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Name all six proxy env spellings in the sanitisation list, and replace
the unestablished 'external HTTPS call' / 'flux-aio does not restart'
claims with the grounded in-cluster mechanism: after the
KUBERNETES_SERVICE_HOST drop above, the shard talks to the management
apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does
not match, so that startup call is what stalls behind an unreachable
proxy. Comment-only; no behaviour change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 66df3b20e0)
Model the source liveness probe as flux-aio actually ships it (a bare httpGet
with no timing fields) and tighten the startupProbe assertions:
- assert the complete normalised budget (delay/period/success/failure) instead
of only FailureThreshold, so a later PeriodSeconds regression cannot silently
restore a short crashloop window;
- assert TimeoutSeconds is inherited from the source probe rather than a
hardcoded value, so forcing it stricter is caught;
- assert the startupProbe is a DeepCopy of liveness, not an alias, which would
otherwise stamp the 30-failure startup budget onto liveness and produce a
never-failing liveness probe.
The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to
a plain alias.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a10fcb3e4a)
The BuildShardDeployment doc comment and the operator README are the only
record of how far the cloned shard container has diverged from flux-aio.
Add the two steps this fix introduced (dropping the inherited corporate-proxy
env and deriving a startupProbe) so the list stays the authoritative account
and does not invite re-adding what was deliberately removed.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 036a31b550)
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
inherited from the liveness handler, otherwise a future flux-aio with a
larger liveness timeout would get a stricter startup probe and could
recreate the crashloop. Covered by a regression guard (liveness timeout 5s
must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
external HTTPS call at startup) and note the remote spec.kubeConfig
exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
installer-injected (the installer injects only KUBERNETES_SERVICE_*).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit affdd70e9d)
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit d44c87d939)
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.
Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit c475faae0f)
The webhook http.Server set no read, write, or idle timeout, so a client that
opens a connection and sends headers slowly could hold a goroutine and file
descriptor indefinitely. Add ReadHeaderTimeout, ReadTimeout, WriteTimeout, and
IdleTimeout.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 4151885871)
Review findings from #3436, plus what an adversarial re-check of the first
attempt at them turned up.
The chart-name extraction was a greedy sed, i.e. LAST match. Helm marshals
"config" (the release`s values) after "chart", so a values subtree that spells
chart.metadata.name shadowed the real chart name: the release then read as
non-SeaweedFS, the tenant silently vanished from the report, and the script
exited 0 -- the precise false clean this PR exists to prevent, reachable
without any corruption at all. It now takes the FIRST match. The path stays
adjacent on purpose: a looser "any name after metadata" matches
chart.templates[].name, which Helm emits immediately after metadata on every
healthy release, so the review`s suggested relaxation would have returned a
template path for every tenant. Over-strictness fails loudly and recoverably;
over-looseness deletes data quietly.
Newlines are now folded before matching, so a pretty-printed payload parses at
all rather than aborting, and whitespace around the punctuation is tolerated.
first_deployed got the same treatment -- it had the identical greedy, single-
line-only shape, and a spaced payload silently dropped the PV-vintage row from
the report.
The FATAL message names the path it read and the shape it expected, so a future
Helm format change is diagnosable instead of looking like real corruption.
Tests: three new payload shapes (spaced, pretty multi-line, and a values decoy)
each byte-compared against the same golden as the compact payload, so the shape
must make no difference to the report. Mutation-checked -- restoring last-match
extraction fails the decoy test on its own, restoring the single-line matcher
fails the pretty test.
The test fake answered any unmodelled kubectl invocation with exit 0 and empty
stdout, the fail-open shape this script was rewritten to reject, and enough to
let a newly added query pass the goldens unnoticed. Unmodelled calls now exit 97
naming the invocation. Verified inert first by instrumenting the fake: 88
invocations across the suite, none unmodelled. Also added the missing test for
the by-name PVC GET fatal path, whose FAIL mode existed with nothing behind it.
The runbook now tells the operator to read the exit code: a non-zero exit means
the table is incomplete and no step may be taken on it. That contract lived only
in the script header, while the runbook is what the operator follows.
29/29 tests green; sh -n clean.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit e02adad38f)
Collapse the certReloader (mtime tracking, serialized refresh, retry backoff,
rate-limited logging, last-good-certificate fallback) into a GetCertificate
callback that simply re-reads the key pair from disk on every handshake. The
signature is unchanged, so main.go is untouched.
The extra machinery guarded scenarios this deployment cannot produce:
- Torn reads are unreachable: the cert/key files come from a plain Secret
volume with no subPath, and Kubernetes' atomic writer swaps the whole ..data
directory via a single symlink flip, so a reader always sees a complete old
or complete new generation of both files.
- The per-handshake LoadX509KeyPair cost is negligible next to the asymmetric
crypto the handshake already performs, so no cache is needed.
- The webhook configures no mTLS, so GetCertificate not refreshing client CA
pools does not apply.
If the mounted files genuinely become unreadable (an operator error, not a
renewal) the handshake now fails loudly instead of silently serving a stale
certificate. The core regression test is retained and remains non-vacuous: it
fails against a static tls.Config.Certificates and passes with the reloading
callback.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 8baa449f53)
A previous revision recorded the certificate file mtime even when the
reload failed, to avoid re-reading a persistently bad file on every
handshake. That conflated two failure modes: a transient, content-
independent error (fd exhaustion, a torn read) on a VALID renewed file
advanced the recorded mtime, so the staleness check never fired again
and the renewed certificate was never loaded. The cached certificate
then expired within renewBefore, and failurePolicy: Fail blocked all
tenant pod creation - exactly the outage this reloader prevents.
Advance the recorded mtime only on a successful load, and bound retries
of a failed load with a wall-clock interval instead. A transient failure
is now retried until it succeeds, while a persistently unreadable file
is still retried only once per interval, not on every handshake. Keep a
single reload in flight to avoid a thundering herd, and rate-limit the
failure log. Add a regression test that fails when the mtime is advanced
on failure.
Detect change with mtime inequality, so a backward clock step is also
picked up.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 5b6bb47caf)
Address review of the reload path:
- Observe the cert file mtime before reading the key pair, so a Secret
swap racing the read is retried on the next handshake instead of being
cached under a newer mtime and missed.
- Record the load attempt regardless of outcome, so a persistently
unreadable file is retried only once its mtime advances, not on every
handshake.
- Detect change with mtime inequality instead of a strictly-newer
comparison, catching equal-mtime replacements and backward clock steps.
- Log stat failures instead of swallowing them, so a broken mount is not
invisible until expiry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a537bb647b)
renewBefore: 24h on a one-year certificate leaves a one-day window for
cert-manager to rotate the Secret and for the webhook to pick it up.
Widen it to 720h (30 days) so renewal happens well ahead of expiry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 791d99d8e5)
The webhook loaded its TLS key pair once at startup and served it as a
static tls.Config.Certificates entry for the lifetime of the process. A
cert-manager renewal of the backing Secret was never picked up by a
running pod, so once the certificate expired (about a year after install)
the pod kept presenting the expired certificate. The kube-apiserver's TLS
call to the webhook then failed and, because the MutatingWebhookConfiguration
uses failurePolicy: Fail, every pod creation in tenant namespaces was
rejected (including virt-launcher pods, blocking all VMIs).
Serve the certificate through a reloading tls.Config.GetCertificate
callback that re-reads the key pair when the mounted files change and
keeps serving the last good certificate on a failed reload. Add a
regression test that fails against the static implementation and passes
once the certificate is reloaded.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 44d9f534a9)
hack/seaweedfs-naming-audit.sh was fail-open: any kubectl failure
produced an empty table indistinguishable from an honestly clean fleet
(namespace LIST failure = zero namespaces walked, secret/pvc/sts LIST
failures = zero findings, all under 2>/dev/null with no error handling).
The runbook uses this output as the gate before PVC deletion, as
post-deletion verification, and as an upgrade precondition, so a
transient API error could green-light destroying data (#3431).
Route every kubectl call through a run_kubectl helper: non-zero exit
prints a FATAL line naming the failed query and propagates the code up
the whole chain (enumerations restructured to capture-then-check, since
an exit inside $(...) dies with the subshell). By-name GETs distinguish
legitimate absence from real errors via --ignore-not-found on a
separate existence check; a Secret that exists but has no decodable
release payload, or one that decodes without a chart name, is corrupt
state and fails loudly instead of silently dropping the tenant.
Incomplete PV-age evidence now marks the generation incomplete and
falls to the safe no-direction branch, so a failed GET can no longer
flip OVERLAP into a wrong Step-3 deletion candidate. A successful run
with zero findings still prints the same bytes as before.
Tests: 14 new cozytest cases — failure injection for every LIST and
by-name GET site, corrupt-payload variants, and full-output golden
diffs for the success paths (verified byte-identical to the pre-change
script). 25/25 green.
Fixes#3431
Assisted-By: Claude <noreply@anthropic.com>
Assisted-By: GPT-5 <noreply@openai.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 3dc6cbd4f4)
The pre-delete teardown Job clears the HelmRelease finalizers as its
final step, but targeted a hardcoded namespace that does not match the
namespace where the release and its RBAC live. The Job's Role and
RoleBinding are created in the release namespace and grant patch on the
HelmRelease by release name, so the ServiceAccount was Forbidden to patch
the HelmRelease in the hardcoded namespace. The Job errored and retried
forever, the HelmRelease stuck in Terminating, and the Helm release
wedged in "uninstalling", blocking any teardown or reinstall.
Target the release name and namespace so the patch matches the RBAC that
grants it. Add a helm-unittest pinning the rendered namespace.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit fd568dee87)
Drives the pin end-to-end through the migration script, in the harness the
SeaweedFS half already uses: the real slot 45 executed by path inside the
migrations image's own pinned alpine base, so the interpreter under test
is the busybox ash that run-migrations.sh actually gives it, with only the
cluster boundary faked.
The property asserted is the annotation landing on the object, never a
function being callable. Thirteen cases: an unannotated Helm-managed
KubeadmConfigTemplate gets pinned and the slot stamps 46; several
templates are pinned each in its own namespace; an already-pinned one is
skipped WITHOUT a write; zero matching objects is a clean run; a
CAPI-spawned KubeadmConfig is left alone because it is not Helm-managed; a
cluster with no kubeadm bootstrap provider stamps cleanly; an object that
vanishes between the scan and the pin is skipped rather than fatal; a
failing fleet scan and a failed pin both abort without stamping; a partial
failure still pins the rest and then aborts; both halves run in one pass
with the SeaweedFS hand-over first; a SeaweedFS failure aborts before the
pin is attempted; and migration 43 does not pin, since the two libs must
stay independently sourceable.
The fake kubectl MODELS THE LABEL SELECTOR rather than ignoring it, which
is what makes the characteristic form of this bug detectable. Helm injects
app.kubernetes.io/managed-by=Helm as a LABEL; meta.helm.sh/release-name is
an ANNOTATION and can never be a selector, so using it matches nothing,
the pin silently applies to nothing, and the run reports pinned=0
failures=0 and exits 0. A fake that answered every selector identically
could not tell that apart from a working pin. An ABSENT selector lists
everything, exactly as kubectl does, so dropping the selector shows up as
the pin stamping keep on Machine-owned objects instead of as a quiet
no-op.
Also rewrites the nine pre-existing SeaweedFS "this did not happen"
assertions, which could not fail. POSIX and bash both exempt a !-negated
pipeline from errexit — "the -e setting shall be ignored ... if the
command's return value is being inverted with !" — so `! grep -q X f`
runs, returns 1, and the test carries on to report success. Measured in
this harness, not inferred: with the two halves of slot 45 deliberately
swapped, a `! grep -q 'PIN '` assertion stayed green while the command log
plainly contained the PIN line. The replacement,
[ "$(grep -c X f)" -eq 0 ], puts the result inside `[`, whose non-zero
status does trip errexit, and is the form already used elsewhere in the
file. Their claims turn out to have been true all along, so nothing about
the migrations changes; what changes is that they are now checked — with
the hand-over mutated to re-annotate a Cluster that already carries keep,
"idempotent: a Cluster already owned by <name>-db AND carrying keep is
left alone" now goes red where before it passed.
Every case was confirmed against a deliberately broken pin before being
made to pass: annotation-as-selector, no selector at all, best-effort
instead of fail-closed, an unconditional write instead of read-then-skip,
a vanished object treated as fatal, the halves swapped, and the pin never
called.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Migration slot 45 holds two different migrations on the two branches. On
release-1.5 it is the SeaweedFS db-split repair (#3370 backport, which
also bumped targetVersion to 46); on main and release-1.6 it is the
helm.sh/resource-policy=keep pin on KubeadmConfigTemplate / KubeadmConfig
that stops Helm pruning them while CAPI still references them from the
kubeadm-backed MachineSet.
v1.5.0 through v1.5.3 all ship targetVersion 45, so no released 1.5
cluster is stamped past 45 and every one of them still runs 1.6's slot 45
on the way up. Cutting v1.5.4 off release-1.5 as it stands is what first
ships targetVersion 46: those clusters get stamped 46, a later
1.5.4 -> 1.6 upgrade runs `seq 46 53`, and 1.6's slot 45 never executes.
The keep-pin is then silently skipped, the 1.6 kubernetes chart upgrade
prunes KubeadmConfigTemplate while the live kubeadm-backed MachineSet
still references it, and its bootstrap.configRef dangles through the
Talos worker rollover. Tenant Kubernetes only; a noisy broken upgrade
rather than data loss, but not self-healing.
Fold the pin into release-1.5's slot 45 so that slot does both jobs. A
v1.5.4 cluster stamped 46 then skips 1.6's slot 45 harmlessly, because
the annotation is already on the objects.
Renumbering release-1.5's repair to 1.6's slot 53 was the alternative and
does not work: run-migrations.sh hard-fails on a missing slot file, so it
would need no-op stubs for 46-52, and a cluster stamped 54 would run
`seq 54 53` and skip every real 1.6 migration.
The pin logic is ported into release-1.5's idiom rather than copied from
main: this branch's slots are #!/bin/sh (busybox ash in the migrations
image) and have no lib/cozystack-version.sh, so the helper is POSIX-clean
with prefixed globals instead of `local`, and slot 45 keeps its existing
inline labelled version stamp. The selector is a LABEL,
app.kubernetes.io/managed-by=Helm, which Helm injects into everything it
applies; meta.helm.sh/release-name is an annotation and would match
nothing while reporting a clean run. Confirmed against a live v1.5 stand
stamped 45: the chart's KubeadmConfigTemplate carries that label and no
resource-policy, and the KubeadmConfig children CAPI spawns carry no
managed-by at all, so they are correctly left alone.
Fails closed but aggregates: every object that can be pinned is pinned
before a non-zero return aborts ahead of the version stamp, so the Job
retries the whole slot and reports all failures at once. The single
per-object exception is an object that disappears between the fleet scan
and the annotate — an app being deleted concurrently with the hook. Helm
cannot prune what no longer exists, so that is a skip rather than a
failed pre-upgrade hook blocking the upgrade. "not found" is accepted
only there, never for the fleet scan, where a list never answers NotFound
and accepting it would let a real failure read as an empty fleet.
The SeaweedFS repair runs first and the pin second. Both fail closed
ahead of the stamp and the Job retries the whole slot, so a
half-completed attempt is safe either way; but on an attempt where only
one half gets to run it should be the one whose failure is irreversible,
and running the pin first would stop the SeaweedFS repair from being
attempted at all on that pass.
targetVersion stays 46 and no slot is added or renumbered.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Aligns the vendored CloudNativePG chart metadata and CRDs with the
already-pinned 1.27.3 operator image. CNPG 1.27.3 stores the
instance-manager session identifier in
`Backup.status.instanceID.sessionID`; the 1.27.1 CRD prunes that field,
causing false instance-manager restart errors and failed backups.
Fixes#3479.
### Screenshots
Not applicable.
### Release note
```release-note
fix(postgres-operator): align CloudNativePG CRDs with the pinned 1.27.3 operator image so PostgreSQL backups no longer fail with false instance-manager restart errors
```
## What this PR does
Hand-backport of #3138 to `release-1.5`. The `backport` label was
applied at merge time but no backport PR was ever opened, so the change
never reached this branch — found by auditing the branch with
`cmd/backport-audit`, which reported it as `MISSING`. Like the other
items in that set, it merged before #3155 fixed the bot's
`conflict_resolution` input, so the conflicting cherry-pick was dropped
with no draft and no failing check.
Velero binds its metrics/health endpoint (`:8085`) only after loading
plugins and connecting to the API server. Under a heavy parallel
platform install the upstream liveness probe starts checking at
`initialDelaySeconds=10` and kills the container after
`failureThreshold=5`, so velero BackOff-restarts and never escapes the
loop inside the install-gate window. Because `backupstrategy-controller`
hard-depends on velero, this blocks the gate. The fix adds a
`startupProbe`, which holds liveness and readiness off entirely while it
is failing, granting `30 * 10s = 300s` of startup grace and then handing
over to the unchanged upstream liveness for steady-state crash
detection.
### Backport adaptations
This branch vendors velero chart **11.0.0** (appVersion 1.17.0) rather
than `main`'s 12.0.3 (1.18.1), so three things differ from the original.
All are recorded in the commit message as well.
- **The chart version is not bumped.** Only the patch re-application is
added to `update`. The `helm pull` here was previously unpinned, which
would have let `update` float to a chart the patch does not apply to, so
it is pinned to `11.0.0` — the version already vendored. That makes the
new patch step deterministic and leaves the vendored tree
byte-identical.
- **The `test:` target is added.** It exists on `main` but not here, so
without it the shipped regression guard would never run.
- **The image assertions pin this branch's own versions** (velero
`v1.17.0`, `plugin-for-aws` `v1.12.1`, kubevirt plugin `v0.8.0`), and
the `upgrade-crds` rationale is restated for chart 11.0.0, where that
Job still runs a kubectl image — `main`'s comment about the Job running
the velero image natively is true for 12.x only.
The `startupProbe` hunk itself applies to chart 11.0.0's deployment
template unchanged, in the same position inside the `metrics.enabled`
gate.
### Verification
`helm unittest .` passes 5/5 in `packages/system/velero`. The assertions
are mutually corroborating rather than vacuous: the `equal` assertions
prove the `startupProbe` path actually renders, and the same paths are
asserted absent under `velero.metrics.enabled: false`, proving it
follows the same gate as liveness and readiness so disabling metrics can
never leave a `startupProbe` pointed at an unbound port.
### Screenshots
Not applicable — no UI change.
### Release note
```release-note
fix(velero): add a startupProbe so a slow velero server start under heavy parallel install no longer crashloops the container and blocks the install gate
```
## What this PR does
Hand-backport of #3034 to `release-1.5`. The `backport` label was
applied at merge time but no backport PR was ever opened, so the change
never reached this branch — found by auditing the branch with
`cmd/backport-audit`, which reported it as `MISSING`.
Every candidate that went missing this way merged before #3155 landed,
when `conflict_resolution` was passed as a top-level input instead of
nested under `experimental`. The action ignored it and fell back to
`fail`, which on a conflicting cherry-pick opens no PR and reports no
failure, so these were dropped with no draft to find and no red check to
notice.
The six original commits are cherry-picked with `-x`, unchanged and in
order. No adaptation was needed: the cherry-pick was clean, and the net
diff is identical to the change on `main`.
### Verification
The two `.diff` files are image patches applied at build time against
upstream `container-object-storage-interface`, so what matters is that
this branch pins the same upstream source. It does — both `release-1.5`
and `main` set `ARG COMMIT_REF=v0.2.2` in
`images/objectstorage/Dockerfile`, and the resulting patch set on this
branch (`91`, `92`, `93`) is byte-identical to `main`'s. Patch `91` was
already present here unchanged, so `92` applies over the same base it
does on `main`.
The branch differs from `main` only in its builder pins (`golang:1.24`
versus `1.26`, unpinned base digests), none of which affect patch
application.
### Screenshots
Not applicable — no UI change.
### Release note
```release-note
fix(objectstorage-controller): converge BucketClaim readiness to the backend Bucket and speed up single-replica COSI provisioner failover
```
The tenant.cozystack.io/<ancestor> namespace labels drive the
<ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor
tenant reach a descendant namespace only when the descendant carries
tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by
splitting .Release.Namespace, which only reached one level up: a direct
child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev
under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely.
The root ingress controller in tenant-root then could not reach nested
tenants, surfacing as upstream timeouts.
Derive the chain deterministically from the namespace name instead. Every
tenant descends from tenant-root, and tenant.name builds each child as
<parent-namespace>-<word>, so every progressive dash-prefix of the name is
a real ancestor namespace. This needs no cluster lookup, renders offline
(so it is unit-testable), and converges on every reconcile regardless of
parent/child HelmRelease ordering. The ownerReference lookup is made
non-fatal so the template still renders when the parent namespace lookup
is empty; in a real reconcile the parent always exists and the
ownerReference is always emitted.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 1ac9398a4b)
(cherry picked from commit 6ce09079b7b84ed124268addfd21881d64976541)
(cherry picked from commit ac7176257b517cbb09fc1792f90ce1f2a18d89be)
Velero binds its metrics/health endpoint (:8085, http-monitoring) only
after the server loads its plugins and connects to the API server. Under
a heavy parallel platform install that can take minutes, but the upstream
liveness probe begins checking /metrics at initialDelaySeconds=10 and
kills the container after failureThreshold=5 (~130s total) before :8085
is bound. The kubelet SIGTERMs velero (exitCode 0/Completed), it
BackOff-restarts, and never escapes the loop inside the install-gate
window. Because backupstrategy-controller hard-depends on velero, this
blocks the gate.
Add a startupProbe to the velero server container. While it is failing
the kubelet holds off liveness and readiness entirely, granting
failureThreshold * periodSeconds = 30 * 10s = 300s of startup grace, then
hands over to the unchanged, tight upstream liveness (10s / 30s / 5) for
steady-state crash detection. This is strictly better than loosening
liveness, which would also blunt steady-state detection.
The upstream velero chart (12.0.3) does not render a startupProbe, so the
package patches the deployment template via patches/add-startup-probe.patch
(re-applied after re-vendor by the Makefile update target, matching the
seaweedfs/ingress-nginx convention) and supplies the startupProbe through
the package values override. Pin the startupProbe budget and the unchanged
liveness with a helm-unittest regression guard.
Backport adaptations for release-1.5, which vendors velero chart 11.0.0
(appVersion 1.17.0) rather than main's 12.0.3 (1.18.1):
- The chart version pin stays at the version this branch already
vendors; only the patch re-application is added to `update`. The
`helm pull` was previously unpinned, which would have let `update`
float to a chart the patch does not apply to, so it is pinned to
11.0.0 to keep the new patch step deterministic.
- Added the `test:` target, which exists on main but not here, so the
shipped regression guard actually runs.
- The image assertions pin this branch's own vendored versions
(velero v1.17.0, plugin-for-aws v1.12.1, kubevirt plugin v0.8.0),
and the upgrade-crds rationale is restated for chart 11.0.0, where
that Job still runs a kubectl image.
The startupProbe hunk applies to chart 11.0.0's deployment template
unchanged, in the same position inside the metrics.enabled gate.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit f7271b8011)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
While the backend Bucket is not ready the controller returns an error to
force a requeue, but that error is only logged at klog.V(3) and is
suppressed at default verbosity, so an operator cannot see why the claim
is stuck. Record a WaitingForBucket Warning event on the BucketClaim on
each requeue (event aggregation folds the repetition) and still return
the error so the requeue happens as before. Pin the requeue contract in
the test by asserting the error carries the stable "not ready yet"
substring, not merely that it is non-nil.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 5991808d28)
The two bucketaccesses waits used the bare jsonpath form, which also
matches the literal "false"; require =true so a never-granted access
fails the wait instead of passing against an ungranted object. Also
print the objectstorage-controller pod status in the bucketReady
failure block so controller-side issues are visible from the test log.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 0855ff2d1b)
The objectstorage-controller package ships the COSI control plane; its commits
use the objectstorage-controller scope, which had no area mapping and fell back
to area/uncategorized. Map it to area/storage alongside the other COSI/storage
scopes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 4f5dcea432)
myasnikovdaniil