Promotes `v1.6.2-rc.1` to stable `v1.6.2` without rebuilding containers.
Their digests remain bit-for-bit what passed rc e2e; the tag-rewritten
packages tree is re-serialized as a temporary OCI candidate and pinned
here by digest. On merge, pull-requests-release.yaml verifies that
candidate against this tree before creating the write-once `v1.6.2` tag,
retagging the candidate and container digests to `v1.6.2` (+`:latest`
when this is the newest stable), publishing the stable cozy-installer
chart, and publishing the release. Do NOT squash-merge (decision B): the
stable tag must attach to a real merge commit.
⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.2-rc.1`.
ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.
✅ Includes `docs/changelogs/v1.6.2.md`, which finalize uses verbatim as
the published release body.
✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.2`), generated from the `release-1.6.2` staging
branch. **Do NOT merge that PR until `v1.6.2` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).
Fixes the false `promotion changed the container repository/digest set`
that blocks #3904.
`normalized_refs()` in `hack/verify-promoted-packages.sh` gives a
host-less ref a repository equal to its own tag.
`hack/lib/image-refs.sh` emits two entries per shape-3 map, the
correctly joined ref plus shape 1's recursive scrape of the bare `tag:
<tag>@sha256:...` scalar, and on that bare scalar `repo="${image%:*}"`
is a no-op because there is no `/` and no `:` in it. Promotion rewrites
exactly those tags, so `v1.6.2-rc.1` to `v1.6.2` shows up as six changed
repositories whose digests are identical on both sides.
`promote-retag.sh` never hits this because it drops host-less refs
through its ownership filter, the verifier had no equivalent.
Host-less refs now compare on digest alone, which is the whole of the
container identity such a ref carries. Checked against the real
published artifacts: before the fix 61 entries each side differing by
those six pairs, after it 60 each side with the sets equal and 48
container digests identical.
Two things this deliberately does not do, both written as comments at
the fix. Not dropping host-less refs outright, because
`packages/system/kuberture/values.yaml` carries an `image:` map with a
`tag:` and no `repository:`, so shape 1 is the only rule that ever sees
its digest and a drop would silently stop proving it unchanged. And not
reusing `promote-retag.sh`'s `${REGISTRY}/` ownership filter, because
the verify job's `REGISTRY` names the private build registry while both
artifacts under comparison live on the public one, so that filter drops
all 48 refs, and the emptiness guard above tests the raw collection
rather than the filtered set, so the check would pass by comparing two
empty sets.
Second commit adds the fixture that was missing. The existing suite
writes a host-bearing string and exercises a passing rc to stable
rewrite, so it read as coverage while never building a shape-3 split
map, which the library itself calls the dominant shape. Reverting the
fix reddens the new case on the digest-set assertion, and replacing the
fix with an outright skip of host-less refs makes the verifier exit 0 on
a genuinely changed digest.
Once this is on `release-1.6` the check on #3904 re-runs against this
copy, so no `rc.2` re-cut is needed. `main` carries the same script and
wants the same fix.
No fixture in verify-promoted-packages_test.bats built a shape-3 split
map -- `repository:` on one key, `tag: <tag>@sha256:<digest>` on another
-- which hack/lib/image-refs.sh itself calls the dominant shape in the
tree. The existing fixture's single-string ref carries a registry host,
so its rc-to-stable rewrite normalizes through the other branch and
passes whether or not host-less refs are handled, which is why the suite
looked covered while a clean promotion could still be rejected.
Add two cases. The first promotes a split map whose tag moves from rc to
stable with the repository and digest held constant; reverting the
normalization fix reddens it on
"promotion changed the container repository/digest set", diffing
v9.9.9-rc.3@sha256:eee... against v9.9.9@sha256:eee... -- the same
tag-as-repository pair, with identical digests, that CI reported.
The second pins the property a blunter fix would break. A digest-only
`image:` map with no `repository:` (the kuberture shape) is seen by no
rule but shape 1, so it must still contribute its digest: the case moves
that digest and nothing else, and asserts the rejection comes from the
digest set rather than a neighbouring leg. Replacing the fix with an
outright `continue` over host-less refs makes the verifier exit 0 on that
changed digest and reddens the case at its exit-status assertion.
Negative assertions are counted rather than `!`-negated: under
hack/cozytest.sh a `! grep -q` cannot fail, so it would be a comment
shaped like an assertion.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
hack/verify-promoted-packages.sh reported "promotion changed the
container repository/digest set" for a v1.6.2 promotion that moved no
container bytes at all: all 48 real repo@digest refs and all 48 digests
were byte-identical between the rc artifact and the promotion candidate.
The six entries it flagged as changed repositories were not
repositories. They were bare tag strings -- v1.6.2-rc.1 on the rc side,
v1.6.2 on the candidate side -- pairing one-to-one by identical digest.
hack/lib/image-refs.sh emits two entries per shape-3 split map: the
correctly joined repository@digest, plus shape 1's recursive scrape of
the raw .tag scalar, which carries no repository. The library documents
that degeneracy and leaves host-less refs to its callers;
promote-retag.sh drops them, the verifier had only a
*/cozystack-packages exclusion. So `${image%:*}` -- a no-op on a string
holding no colon -- handed the tag over as the repository, and promotion
rewrites exactly those tags.
Set an empty repository for a ref with no `/` ahead of its digest, so it
compares on the digest, which is the whole container identity such a ref
carries and the only part promotion must not move. Dropping these refs
instead would be wrong rather than blunter: packages/system/kuberture
carries an `image:` map with a `tag:` and no `repository:`, so shape 1
is the only rule that sees its digest, and skipping it would silently
stop proving that digest unchanged.
Reproduced over the release-1.6 packages tree with the shape-3 tags
rewritten rc-to-stable: before, the six pairs above diverge; after, both
sides normalize to 60 identical entries, kuberture's digest included and
kube-ovn's real repository still visible through its shape-4 join.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
v1.6.1 was promoted with the rc e2e gate bypassed, and its promote PR
body says so verbatim. The reason is mechanical rather than anyone's
judgement call: workflows run from the ref they fire on, release-1.6's
`tags.yaml` has no `rc-e2e` job and this branch has no `e2e-tag.yaml` at
all, so when `v1.6.1-rc.1` was pushed there was nothing to produce
evidence the gate could find. Bypass was the only outcome available.
This closes that. Adds `e2e-tag.yaml` verbatim from main, which works as
written here because `hack/e2e-chainsaw/` and the `test-chainsaw` target
already exist on this branch. Splices main's `rc-e2e` job into
`tags.yaml`. Adds the `verify-release-candidate` job to
`pull-requests.yaml`, and the candidate-verification step to
`pull-requests-release.yaml` positioned before tag creation, publish and
retag. Plus `hack/verify-promoted-packages.sh`,
`hack/lib/promoted-packages.sh`, `hack/validate-changelog.sh` and two
bats suites.
All five files `promote-rc.yaml`'s preflight demands are present now
with their markers, where this branch previously satisfied one of five.
The e2e job name is kept byte-identical because promote-rc matches on
exactly that string, and `hack/promote-gate-contract.bats` now pins the
pair so the next drift gets a red test instead of a silent bypass.
One deviation worth knowing about. The verify job is keyed on author
plus a `release-` head branch, not on the presence of the `release`
label. On #3550 the PR was created at 09:35:31Z and the label arrived as
a separate `labeled` event at 09:35:32Z, and this branch's
`on.pull_request.types` has no `labeled`, so a label-keyed job would
satisfy the preflight marker and then never fire. Adding `labeled` was
the other option but on main it is interlocked with a concurrency split,
a `plan` guard complement and `e2e-fork.yaml`, so keying on the author
is the narrower change. Checked against every release-labelled bot PR
back to v1.0.7, the bot's other release PRs use `changelog-v*` heads so
they do not collide.
hack/verify-promoted-packages_test.bats is byte-identical to main: 13
tests driving the verifier against a mocked `flux`, so they need no
registry. They cover the fail-open shapes that matter most — an
unreadable file during the rc-reference scan, a collector that returns
nothing, a digest that moved while candidate and merge tree still match.
hack/promote-gate-contract.bats is NOT main's copy, and the difference is
deliberate. main's version pins a `parse` job, a `skip_e2e_gate` input,
promote-time changelog and website-docs jobs, and the `labeled`
pull_request trigger they depend on — none of which exist here, because
`Promote RC` is a workflow_dispatch and runs from the ref it is
dispatched from, which is main. Run verbatim against this branch it fails
on its first test and the runner aborts the suite.
So this pins the half release-1.6 actually owns — the producer side:
* an rc tag push calls e2e-tag.yaml with the pushed ref, only for
-rc. tags, and grants the static permission ceiling the called
workflow declares;
* the called job's name is the exact literal main's gate composes and
searches for, asserted with grep -F because `${{` as a basic regular
expression matches nothing and would make the pin pass by finding its
own subject absent;
* the suite it runs is the full one (empty CHAINSAW_SUITES), with the
Makefile target and the sandbox chainsaw binary it needs;
* all five files promote-rc.yaml's preflight reads off the target base
are present, plus the two libraries the verifier sources by relative
path;
* verify-release-candidate is keyed on author and head branch and
carries NO label condition, so restoring main's from muscle memory
switches the job off here and fails this test instead;
* the candidate verification precedes the stable tag, the submodule
tag, the release publish, the retag and the chart publish, and
follows the toolchain, the login and the tooling checkout it needs;
* the tag-time changelog is validated and ported, not regenerated.
Every pin was mutation-checked: breaking the thing it describes turns it
red.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Since the promote PR is based on release-1.6, the changelog it carries
merges onto the maintenance line and never reaches main. The backstop job
in tags.yaml only asks whether the file is on origin/main, so it sees
"absent", spends an AI run, and produces text nobody reviewed — which
update-releasenotes.yaml then pushes into the published release body,
overwriting the reviewed notes the release actually shipped with.
Ask the second question too. `at_tag` is whether the changelog is present
in this checkout, which is the tag commit and therefore the promote merge
commit. When it is, port that reviewed file to main verbatim and skip
generation entirely; the AI now runs only when the changelog is genuinely
absent on both sides.
Add hack/validate-changelog.sh, byte-identical to main, and gate the
commit on it. A ported file came from the tag and a generated one came
from a step that is deliberately allowed to fail, so neither had been
checked: the old guard only asked whether the file was non-empty, which a
truncated Copilot stream satisfies. The validator asserts the H1, the
release-link comment, the trailing compare link and at least one section,
all carrying the expected version, so a fragment or a changelog generated
against the wrong tag cannot reach a release body.
Verified against this branch's own shipped changelogs: v1.6.1, v1.5.2 and
v1.5.1 all pass, and v1.6.1 checked against 1.6.2 fails as it should.
This takes effect for tags cut after it lands — tags.yaml and the script
it calls both come from the tag's own tree.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Workflows run from the ref they fire on, so a tag pushed on this branch
executes THIS tags.yaml, not main's. main grew an rc-e2e job on
2026-07-24; release-1.6 never did. The v1.6.1-rc.1 push therefore
produced only Prepare Release, a skipped changelog and skipped website
docs — no E2E Release Candidate job at all. Promote RC looks for its
evidence by exact job name, found none, and v1.6.1 was promoted with the
gate bypassed. That is the hole this closes.
Add e2e-tag.yaml byte-identical to main and call it from a new rc-e2e
job. The suite step works as written here because this branch already
carries hack/e2e-chainsaw/ and a test-chainsaw target, and the sandbox
image already installs chainsaw 0.2.15.
Two things make the wiring load-bearing rather than cosmetic:
* the called job's name must stay "E2E ${{ inputs.tag }} (full suite)".
Promote RC composes the string it searches for from the rc tag, and
GitHub renders a called job as "<caller> / <called>", which the
gate's suffix matcher accepts. Rename either side and the gate fails
closed.
* the caller must grant contents: read AND checks: write. A caller's
permissions are the ceiling for every job in the called workflow and
GitHub validates that ceiling statically, at run creation, before any
`if:` is evaluated. e2e-tag.yaml's e2e job declares checks: write for
a dispatch-only breakpoint workflow_call can never reach, so omitting
the grant would fail the entire tags.yaml run for EVERY tag push, rc
and stable alike, building nothing.
prepare-release already publishes the rc as a non-draft prerelease with
the nocloud-amd64.raw.xz asset and pushes the release-X.Y.Z-rc.N staging
branch before rc-e2e can start, which is exactly what e2e-tag.yaml's
resolve job requires.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Promote RC refuses to promote a release whose target base cannot verify
the packages candidate. It checks five markers on the base branch before
its first registry write and fails closed with "Target base
'release-1.6' lacks '<path>'". This branch satisfied one of the five
(hack/lib/image-refs.sh), so the next patch release off release-1.6
could not be dispatched at all.
Add the two halves the base has to carry:
* hack/verify-promoted-packages.sh and hack/lib/promoted-packages.sh,
byte-identical to main. The verifier pulls the candidate by its
committed digest and proves it matches the merge tree apart from the
artifact's impossible self-reference, that promotion rewrote every
version-line image tag, and that the container repo@digest set is
unchanged from the rc artifact.
* the two jobs that run it. verify-release-candidate in
pull-requests.yaml checks GitHub's prospective merge tree so a
packages/ edit pushed to the promote PR shows up as a failed check,
and a Verify stable packages candidate step in the finalize job of
pull-requests-release.yaml repeats it against the real merge commit.
Position is the whole point of the finalize half: it runs before the
write-once stable tag, the draft-release publish and the retag, so a bad
candidate is refused while nothing stable-named exists yet. The former
"Set up toolchain (skopeo, yq, helm)" and "Login to registry (GHCR)"
steps moved up with it and are no longer duplicated later in the job;
the toolchain now also installs flux and pins yq by version and sha256,
because an irreversible step must not depend on whatever
releases/latest/download resolves to today.
verify-release-candidate is keyed on the promote PR's author and its
head-branch shape rather than on the `release` label main matches. This
branch's on.pull_request.types has no `labeled` event, and the label is
applied about a second after the PR opens (a separate call in
gh pr create), so the `opened` payload carries no labels and a
label-keyed guard would never fire here.
.release-tooling is gitignored, matching main, so the trusted-base
checkout can never be staged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Name all six proxy env spellings in the sanitisation list, and replace
the unestablished 'external HTTPS call' / 'flux-aio does not restart'
claims with the grounded in-cluster mechanism: after the
KUBERNETES_SERVICE_HOST drop above, the shard talks to the management
apiserver over the kubelet-injected ClusterIP, which NO_PROXY=.svc does
not match, so that startup call is what stalls behind an unreachable
proxy. Comment-only; no behaviour change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 66df3b20e0)
Model the source liveness probe as flux-aio actually ships it (a bare httpGet
with no timing fields) and tighten the startupProbe assertions:
- assert the complete normalised budget (delay/period/success/failure) instead
of only FailureThreshold, so a later PeriodSeconds regression cannot silently
restore a short crashloop window;
- assert TimeoutSeconds is inherited from the source probe rather than a
hardcoded value, so forcing it stricter is caught;
- assert the startupProbe is a DeepCopy of liveness, not an alias, which would
otherwise stamp the 30-failure startup budget onto liveness and produce a
never-failing liveness probe.
The alias assertion is non-vacuous: it fails when the DeepCopy is regressed to
a plain alias.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a10fcb3e4a)
The BuildShardDeployment doc comment and the operator README are the only
record of how far the cloned shard container has diverged from flux-aio.
Add the two steps this fix introduced (dropping the inherited corporate-proxy
env and deriving a startupProbe) so the list stays the authoritative account
and does not invite re-adding what was deliberately removed.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 036a31b550)
Review follow-ups:
- Do not force the cloned startupProbe TimeoutSeconds to 1: keep the value
inherited from the liveness handler, otherwise a future flux-aio with a
larger liveness timeout would get a stricter startup probe and could
recreate the crashloop. Covered by a regression guard (liveness timeout 5s
must be inherited).
- Soften the proxy-strip rationale to the A/B-confirmed mechanism (blocking
external HTTPS call at startup) and note the remote spec.kubeConfig
exception that does not apply to cozystack in-cluster guest apiservers.
- Correct the test fixture comment: the proxy env is operator/patch-level, not
installer-injected (the installer injects only KUBERNETES_SERVICE_*).
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit affdd70e9d)
The cloned shard Deployment inherits only a liveness probe (~30s window,
no startupProbe), so a controller still syncing caches on startup is killed
before it serves /healthz and never recovers. Derive a startupProbe from the
liveness handler with a generous budget so a slow start is not fatal, while
liveness still catches a wedged running pod.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit d44c87d939)
A sharded helm-controller inherits HTTP_PROXY/HTTPS_PROXY/NO_PROXY from the
flux-aio helm-controller container. A standalone shard needs no external
egress (source-controller fetches artifacts), so behind an unreachable
corporate proxy it makes a blocking startup HTTPS call that never completes,
the manager never serves /healthz, and the liveness probe crashloops the pod
forever, freezing every HelmRelease carrying a sharding key. flux-aio itself
survives only because it starts once and never restarts.
Drop the proxy env (and the now-pointless NO_PROXY) when cloning, alongside
the existing localhost/KubePrism sanitisation, so the shard talks to the
in-cluster apiserver directly.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit c475faae0f)
Nothing asserted that a published definition name contains no slash,
which is why the core and sdn groups shipped as Go import paths in
v1.6.0 and v1.6.1 without anyone noticing. The same omission on a
future group would be just as invisible.
TestDefinitionNamesAreDottedModelNames fails on any definition name
containing "/", names the offender and points at the fix (the
+k8s:openapi-model-package marker plus an OpenAPIModelName method).
TestDefinitionRefsResolve is the failure mode itself: it builds each
$ref the way kube-openapi's builder does, JSON pointer escaped, then
resolves it the way a client does, trimming "#/definitions/" with no
unescaping, and reports any that dangles. It also checks the declared
Dependencies against the published names, which catches a group whose
types disagree with each other rather than uniformly.
Both assert the invariant instead of listing today's 20 names, so a new
group that omits the marker is caught rather than a changed count, and
both guard against going vacuous if the definition map is ever emptied
or stops covering cozystack's own types.
Both fail on the pre-fix tree: the first names all 20 offending
definitions, and the second reproduces the reported error verbatim,
down to
"github.com~1cozystack~1cozystack~1pkg~1apis~1core~1v1alpha1.OptionSpec".
Refs: https://github.com/cozystack/cozystack/issues/3806
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit cea1703a6d)
The aggregated apiserver publishes the core.cozystack.io and
sdn.cozystack.io models under their Go import path, so a $ref to any of
them does not resolve and client-side validation fails on every
resource of those groups, not only the one named in the error:
error validating data: SchemaError(…core/v1alpha1.Option.spec):
unknown model in reference:
"github.com~1cozystack~1…v1alpha1.OptionSpec"
Since Kubernetes 0.35 the apiserver's DefinitionNamer.GetDefinitionName
returns the model name it is handed verbatim; it no longer converts the
Go import-path form into the "friendly" reversed-path form. Whatever
name the generated openapi map uses therefore becomes both the
published definition key and, JSON pointer escaped, the $ref pointing
at it. A name containing "/" ships as a definition keyed on the raw
path while every reference to it spells each slash "~1", and clients
resolve a $ref by trimming "#/definitions/" without unescaping
(kube-openapi pkg/util/proto/document.go). The two spellings never
meet, the reference dangles, and validation of the whole document
fails.
Mirror b916d7475, which fixed the same disagreement for apps: add the
+k8s:openapi-model-package marker to each doc.go and declare
OpenAPIModelName for all 11 core and 9 sdn types, so
GetCanonicalTypeName, Scheme.ToOpenAPIDefinitionName and the generated
map key all agree on a slash-free dotted name. That takes the number of
slash-bearing definition names in the published document from 20 to 0.
The methods are hand-written rather than emitted by openapi-gen's
--output-model-name-file because that flag also rewrites
zz_generated.model_name.go inside the read-only apimachinery and
apiextensions module-cache packages the shared gen_openapi helper
always passes as inputs, which fails on any consumer including CI that
vendors deps from the cache. With the marker in place openapi-gen emits
Type{}.OpenAPIModelName() for every type it references, so a future
type without a method fails the build instead of silently
reintroducing a Go-path name.
The generated openapi is the output of the root `make generate`;
nothing else in the generated tree moved.
kubectl apply --validate has been broken against any cozystack-api
built after 2026-07-21 on every resource, and this shipped in v1.6.0
and v1.6.1. In e2e it surfaces as the platform install hanging on
cozy-backup-controller/backupstrategy-controller, which takes
tenant-root and everything behind cozystack-basics with it.
Refs: https://github.com/cozystack/cozystack/issues/3806
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 5f3ad9cd58)
helm-unittest coverage for the S3 request-checksum fix (was mutation-negative
before): assert spec.instanceSidecarConfiguration.env pins
AWS_REQUEST_CHECKSUM_CALCULATION=when_required on all rendered barman-cloud
ObjectStores — postgres backup + bootstrap-recovery
(packages/apps/postgres/tests/backup_storage_test.yaml) and the Keycloak DB
(new packages/system/keycloak/tests/db_backup_test.yaml). Verified by mutation:
reverting the template change now fails these assertions.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
(cherry picked from commit 80a03b1e09)
Address review on the S3 request-checksum fix:
- Extend the fix to the platform-managed useSystemBucket=true flow, whose
ObjectStore is built in Go by applyClusterPluginBackup (SSA-applied), not
chart-rendered. Add InstanceSidecarConfiguration to cnpgtypes.ObjectStoreSpec
and set AWS_REQUEST_CHECKSUM_CALCULATION=when_required on it via
barmanSidecarConfiguration(). The platform's own default system bucket is
SeaweedFS, a non-AWS S3 gateway of the same class as Ceph RGW. Unit test in
cnpgstrategy_controller_test.go asserts the applied ObjectStore carries it.
- Factor the duplicated instanceSidecarConfiguration block (3x across two
charts) into a shared cozy-lib helper (cozy-lib.barman.checksumSidecarConfiguration).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
(cherry picked from commit b574e2176f)
The barman-cloud plugin's sidecar uploads via boto3. Since botocore ~1.36
(early 2025) the default RequestChecksumCalculation is when_supported, which
attaches a flexible checksum to every PutObject. Non-AWS S3-compatible
backends (Ceph RADOS Gateway, some MinIO / Cloudflare R2 builds) reject the
accompanying header with "InvalidArgument: x-amz-content-sha256 must be
UNSIGNED-PAYLOAD, STREAMING-AWS4-HMAC-SHA256-PAYLOAD or a valid sha256
value", so every backup/WAL-archive upload fails against them.
Pin the sidecar to when_required via the ObjectStore's
spec.instanceSidecarConfiguration.env (botocore honors the
AWS_REQUEST_CHECKSUM_CALCULATION env var) on both the keycloak system DB and
the postgres app (backup + recovery ObjectStores). when_required is also
accepted by AWS S3 on a plain PutObject, so it is a safe default everywhere.
This mirrors the etcd-operator fix (cozystack/etcd-operator#342, released in
v0.5.3), which applied the same WhenRequired policy to its aws-sdk-go-v2 S3
client for the same Ceph RGW backend.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
(cherry picked from commit 0c586b9d72)
Addresses the four findings on #3524.
**The `fail` in the bucket chart (MAJOR).** The <bucket>-<user>-credentials
Secret was made to `fail` the render while its COSI lookup was empty, so the
release would retry and converge instead of skipping the Secret permanently.
Helm cannot render a partial set, so that fail aborts the WHOLE
<bucket>-system release: the other users' Secrets, and the bucket UI
Deployment/Service/Ingress/HTTPRoute with them. One declared user whose
BucketAccess never provisions takes down every other user of that bucket,
and on an already-installed release parks it in Failed, blocking every later
upgrade.
**No per-bucket escape (MAJOR).** Rather than add one, remove the need for
it. The gate already forces the release that renders the lookup-gated
Strategy CRs; the credentials Secret is the same trap one release earlier,
and the gate reads that Secret to resolve the bucket name in the first
place. So while it is absent, or carries no bucket name, force the bucket's
<bucket>-system release instead of doing nothing. That cannot deadlock its
own precondition: the BucketClaim and the BucketAccess whose COSI Secret the
lookup reads are rendered unconditionally by the parent release, not by the
one being forced.
The two releases are throttled independently — they are forced in sequence
on a bootstrap, so a shared timestamp would delay the second by a full
MinForceInterval for no reason. Coordinates come from the chart under
provisionBucket, and are omitted on external S3 where the Secret is
admin-managed and no release renders it. No RBAC delta: patch on
helmreleases is already cluster-scoped.
The chart goes back to a partial render, with the comment now explaining why
the skip is not self-healing and why failing is the wrong lever, and a test
that renders the UI with every user unresolvable so a reintroduced `fail`
fails the suite. Tenant buckets keep their pre-existing behaviour;
generalising the repair to every bucket needs a controller that owns them
and is tracked separately.
**Stale gauge (MINOR).** cozystack_backup_default_objects_missing was only
written on the happy path, so the state the gate exists to catch reported 0
and the documented alert never fired. It is now written on the unresolved
path too — the credentials Secret counts as one of the objects the default
backups depend on, which is what it always was. On an API error the gauge is
deliberately left alone rather than flapping, so a new
cozystack_backup_default_objects_check_errors_total marks it stale, and the
doc pairs the two instead of overstating the gauge.
**Suspended releases (MINOR).** forceHelmRelease now does a point Get and
skips a release with spec.suspend: true. helm-controller ignores forceAt and
requestedAt while suspended (`cozyhr suspend` sets exactly that), so the gate
was re-stamping every MinForceInterval for the whole suspension and climbing
the force counter — which the runbook attributes to a render that is not
producing the objects, the wrong diagnosis. The skip is logged and not
counted, so a climbing counter keeps its documented meaning.
Reported-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
Assisted-By: Claude <noreply@anthropic.com>
(cherry picked from commit f859b710a4)
The gate reads the BackupClass as the manifest of what must exist, but the Go
fixture listed only CNPG, Etcd and the two Velero routes while the shipped
backupclass-default.yaml also routes MariaDB and Altinity. Add both to the
fixture, and to the RESTMapper and list-kinds derived from it, so the
expected-object set the tests exercise equals the real route set.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 8b006391c2)
The BSL existence check used a hardcoded GVR and the dynamic client, whose
Get returns a plain 404 (IsNotFound) for an unserved API group. The
IsNoMatchError branch meant to skip an absent Velero API was therefore dead
code, and if Velero were uninstalled after bootstrap while bslEnabled=true
the BSL would be counted missing and force a Helm upgrade forever. Route the
lookup through the RESTMapper like the strategy loop, so an absent velero.io
API is a NoMatch we skip. A regression test fails without the change.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit bf152ba864)
The cozy-default BackupClass routes VMInstance/VMDisk to the Velero
Strategy CRs unconditionally, but the chart gates those CRs on
velero.bslEnabled. With the BSL disabled they never render, so the gate
counted them as permanently missing and forced a Helm upgrade every
MinForceInterval forever, climbing force_reconciles_total and pinning the
missing gauge. Skip Velero-kind strategyRefs when the BSL is disabled
(empty VeleroNamespace), the same flag that gates their render. The
regression test now omits the manually-created Velero objects so it fails
without the skip. Correct the values.yaml comment that wrongly claimed the
Strategy CRs still ship in this mode.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit aa3accd128)
Two defects found in review of the periodic recovery loop.
checkAndLog passed the manager's root context straight into Check, which
makes several sequential API calls. That context is only cancelled at
shutdown, so an API server stalling on any one of them blocked the call
indefinitely. The ticker loop is sequential, so one stuck call stopped
every later check for the life of the pod: the recovery this gate exists
to provide went silent, leaving a stale gauge and no further log line,
with only a restart to bring it back. Each check now runs under a
timeout of half the Period, capped at 30s, so it can never overlap the
next tick.
Check also propagated a NotFound on the projector's source Secret,
although the very next branch treats an empty bucket name as an expected
no-op. Those are the same bootstrap state seen a moment apart, so the
absent Secret logged a check failure on every tick of the whole
bootstrap window for a condition the doc comment frames as graceful.
Both are pinned by tests that fail against the previous behaviour: one
reverts to an error on the absent Secret, the other to a timeout equal
to Period.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
(cherry picked from commit e85501e9f5)
The default Strategy CRs and the Velero BackupStorageLocation are
Helm-templated behind a `lookup` of the BucketClaim the same chart
creates, and the <bucket>-<user>-credentials Secret behind a `lookup` of
the COSI Secret. When those lookups are empty at install time the objects
were silently skipped, and helm-controller does not re-render a release
whose chart and values did not change (drift detection is off on
operator-generated HelmReleases), so the skip was permanent: clusters ran
for months with only BackupClass cozy-default and no Strategy CRs at all.
That also fail-closes the pre-adoption snapshot in the v1.6.0 etcd
migration, which reads the projected credentials Secret.
Add a DefaultObjectsGate runnable to backupstrategy-controller. Once the
bucket name is resolvable from the projector's source Secret, it checks
that every object cozy-default routes to exists and forces one real Helm
upgrade (reconcile.fluxcd.io/forceAt + requestedAt) when any is missing.
Helm remains the objects' only author; the gate only makes sure the render
that produces them actually happens. It cannot be a render-time `fail` in
this chart: the chart is the producer of the Bucket its own lookup reads,
so a failed render would deadlock the condition.
The bucket chart's user-credentials template CAN fail safely — the
BucketAccess that produces the COSI Secret belongs to the parent release
and that release retries forever — so make it fail loudly instead of
skipping, with requireUserCredentials=false as the offline-render escape.
Also correct the chart and doc comments that claimed Flux re-renders on
its interval, and document the two-release manual recovery for clusters
already affected.
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
(cherry picked from commit 51ad226706)
The webhook http.Server set no read, write, or idle timeout, so a client that
opens a connection and sends headers slowly could hold a goroutine and file
descriptor indefinitely. Add ReadHeaderTimeout, ReadTimeout, WriteTimeout, and
IdleTimeout.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 4151885871)
Collapse the certReloader (mtime tracking, serialized refresh, retry backoff,
rate-limited logging, last-good-certificate fallback) into a GetCertificate
callback that simply re-reads the key pair from disk on every handshake. The
signature is unchanged, so main.go is untouched.
The extra machinery guarded scenarios this deployment cannot produce:
- Torn reads are unreachable: the cert/key files come from a plain Secret
volume with no subPath, and Kubernetes' atomic writer swaps the whole ..data
directory via a single symlink flip, so a reader always sees a complete old
or complete new generation of both files.
- The per-handshake LoadX509KeyPair cost is negligible next to the asymmetric
crypto the handshake already performs, so no cache is needed.
- The webhook configures no mTLS, so GetCertificate not refreshing client CA
pools does not apply.
If the mounted files genuinely become unreadable (an operator error, not a
renewal) the handshake now fails loudly instead of silently serving a stale
certificate. The core regression test is retained and remains non-vacuous: it
fails against a static tls.Config.Certificates and passes with the reloading
callback.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 8baa449f53)
A previous revision recorded the certificate file mtime even when the
reload failed, to avoid re-reading a persistently bad file on every
handshake. That conflated two failure modes: a transient, content-
independent error (fd exhaustion, a torn read) on a VALID renewed file
advanced the recorded mtime, so the staleness check never fired again
and the renewed certificate was never loaded. The cached certificate
then expired within renewBefore, and failurePolicy: Fail blocked all
tenant pod creation - exactly the outage this reloader prevents.
Advance the recorded mtime only on a successful load, and bound retries
of a failed load with a wall-clock interval instead. A transient failure
is now retried until it succeeds, while a persistently unreadable file
is still retried only once per interval, not on every handshake. Keep a
single reload in flight to avoid a thundering herd, and rate-limit the
failure log. Add a regression test that fails when the mtime is advanced
on failure.
Detect change with mtime inequality, so a backward clock step is also
picked up.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 5b6bb47caf)
Address review of the reload path:
- Observe the cert file mtime before reading the key pair, so a Secret
swap racing the read is retried on the next handshake instead of being
cached under a newer mtime and missed.
- Record the load attempt regardless of outcome, so a persistently
unreadable file is retried only once its mtime advances, not on every
handshake.
- Detect change with mtime inequality instead of a strictly-newer
comparison, catching equal-mtime replacements and backward clock steps.
- Log stat failures instead of swallowing them, so a broken mount is not
invisible until expiry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit a537bb647b)
renewBefore: 24h on a one-year certificate leaves a one-day window for
cert-manager to rotate the Secret and for the webhook to pick it up.
Widen it to 720h (30 days) so renewal happens well ahead of expiry.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 791d99d8e5)
The webhook loaded its TLS key pair once at startup and served it as a
static tls.Config.Certificates entry for the lifetime of the process. A
cert-manager renewal of the backing Secret was never picked up by a
running pod, so once the certificate expired (about a year after install)
the pod kept presenting the expired certificate. The kube-apiserver's TLS
call to the webhook then failed and, because the MutatingWebhookConfiguration
uses failurePolicy: Fail, every pod creation in tenant namespaces was
rejected (including virt-launcher pods, blocking all VMIs).
Serve the certificate through a reloading tls.Config.GetCertificate
callback that re-reads the key pair when the mounted files change and
keeps serving the last good certificate on a failed reload. Add a
regression test that fails against the static implementation and passes
once the certificate is reloaded.
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 44d9f534a9)
velero evolves its CRD schema between versions; v1.18 adds the Queued
and ReadyToStart backup phases. Helm never upgrades CRDs shipped in
crds/, the generated HelmRelease left spec.upgrade.crds at the Skip
default, and the package disables the chart's upgrade-crds Job, so live
CRDs stayed frozen at first-install state: the server upgrades, the
apiserver rejects the new phases, and backups sit in New while the
HelmRelease stays green. Opt the velero PackageSource into
CreateReplace, consistent with mongodb-operator, add a platform test
pinning the policy, and correct the stale Job-disable rationale in the
package.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <3811295@gmail.com>
(cherry picked from commit f4d812cfb9)
Promotes `v1.6.1-rc.1` to stable `v1.6.1` — no rebuild. The tree pins
the rc's images by digest. On merge, pull-requests-release.yaml creates
the write-once `v1.6.1` tag at the merge commit, retags those digests to
`v1.6.1` (+`:latest` when this is the newest stable), publishes the
stable cozy-installer chart, and publishes the release — so the stable
artifacts are bit-for-bit the rc that passed e2e. Do NOT squash-merge
(decision B): the stable tag must attach to a real merge commit.
⚠️ **RC e2e gate bypassed** — this promotion did not verify green full
e2e evidence for `v1.6.1-rc.1`.
ℹ️ E2E already ran against the immutable rc, so this promote PR does not
run E2E by default. A maintainer may add the `full-e2e` label to run the
full suite again.
✅ Includes `docs/changelogs/v1.6.1.md`, which finalize uses verbatim as
the published release body.
✅ Website docs PR opened/refreshed on `cozystack/website` (branch
`update-docs-v1.6.1`), generated from the `release-1.6.1` staging
branch. **Do NOT merge that PR until `v1.6.1` is published** — merging
it early flips the site's latest-version pointer to an unpublished
version and 404s its docs/API links (see the PR body).
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster.
Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit bcb36262ef)
myasnikovdaniil