Commit Graph
1493 Commits
Author SHA1 Message Date
myasnikovdaniil 315b335cf7 [Backport release-1.7] fix(marketplace): do not install a connected repository's components on connect (#5117)
# Description
Backport of #5071 to `release-1.7`.
2026-10-27 12:14:59 +05:00
IvanHunters 25f8f11187 fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.

Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
(cherry picked from commit 2f39e304f0)
2026-10-26 16:11:10 +00:00
Myasnikov Daniil a2d9cc766c fix(kubernetes, opensearch, foundationdb)!: restrict managed application images
Image overrides let a tenant run an arbitrary container in the
management cluster, including helpers with access to the Talos CA
secrets and the tenant cluster's API credentials. Drop those overrides
and run the images the platform ships.

The Kubernetes and KubernetesNodes helpers (the wait-for-kubeconfig
and Proxmox cloud-config init containers, the bootstrap-token and OIDC
RBAC Jobs, the talos-csr-signer sidecar, the Talos reconcile and unpin
Jobs) take the bundled image references through cozy-lib.image, so
they still follow the configured registry. OpenSearch no longer sets
an image and leaves it to the operator, which follows the version the
chart's version map selects; an explicit version: null now fails the
render instead of falling back to v2. The Talos installer settings for
tenant worker VMs stay.

FoundationDB replaces the free-form cluster.version with version, v7.3
or v7.4, mapped to 7.3.63 and 7.4.1; the 7.1 line is no longer
offered. The chart refuses a render where cluster.version and version
name different release lines, which also covers variants without
platform migrations. Migration 60 carries an existing cluster.version
over to its release line and leaves the legacy value in place while
the old chart can still render it. It records what needs attention in
the cozystack.io/migration-60-needs-attention annotation on the
cozystack-version ConfigMap: a FoundationDB release it cannot carry
over or whose patch changes, a release that still sets one of the
removed image overrides, an OpenSearch release storing version: null.
A dry run reports them without patching, annotating or advancing the
stamp. Migration 56 stops carrying the removed kubectl override into
adopted node pools.

OpenSearch and FoundationDB get version-map update scripts that keep
the current default while it is still supported.

Stored image overrides are still accepted and the charts ignore them.
On the first reconcile OpenSearch nodes move to the image of their
selected version and Kubernetes components whose image changed
restart. Air-gapped installations need docker.io and ghcr.io mirrored
on the management-cluster nodes.

Assisted-by: LLM
(cherry picked from commit 749d401899)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-26 12:24:00 +00:00
Myasnikov Daniil 8ba8386a2e fix(hack): keep the backup stack in the vminstance selective install
The vminstance Chainsaw suite carries the VMInstance and VMDisk Velero
round-trips. Both build the cozy-default BackupStorageLocation and the
platform bucket from the backupstrategy-controller package, but the
selective install retained that package only for the database and
broker suites. A selection of vminstance alone therefore installed a
platform with no Bucket, no BucketClaim and no BSL, and both
round-trips timed out on the BSL wait before running a step of the
example scripts.

Every pull request whose selection is exactly vminstance hits this,
and that includes each edit to examples/backups/vmi and
examples/backups/vmdisk, which select that suite on purpose.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 9a782f697a)
2026-10-22 10:31:14 +00:00
myasnikovdaniil cb7213deb6 [Backport release-1.7] fix(backups): prepare standalone VMDisk restore targets and add VM backup/restore e2e round-trips (#5044)
Backport of #3427 to `release-1.7`. The three commits are cherry-picked
from 9af1c0f902,
9cc3a9e459 and
58b4343e5a.

All three apply cleanly on top of #5032, the backport of the Valkey
backup strategy, so there is no conflict to resolve. The diff is the
same 28 files, +2544/-397, as #3427.
2026-10-21 22:52:08 +05:00
myasnikovdaniil efbcce2b71 [Backport release-1.7] fix(seaweedfs): build SeaweedFS in-tree with a bounded S3 read-ahead (#5028)
Backport of #4819 to `release-1.7`. The five commits are cherry-picked
in order from 1913104bd, d2c6b85ca, 5592a3f65, ecdc043e6 and e8635f6e4.

One conflict, in `.github/renovate.json`: the rule #4819 adds sits next
to two e2e sandbox rules from #4867, which is not on this branch. I kept
this branch's rules and added only the seaweedfs rule. The diff is the
same nine files, +281/-2, as #4819.

The bug is the same here. This branch runs the upstream
`chrislusf/seaweedfs:4.45` image, and its S3 gateway is OOM-killed while
serving a large object (#3782).

This needs to land before rc.2 is cut, otherwise rc.2 and v1.7.0 ship
the upstream image.
2026-10-21 19:21:39 +05:00
Andrey Kolkov 4ac0cd5814 ci(hack): route examples/backups/{vmi,vmdisk} edits to the vminstance suite
The two backup round-trips run the examples/backups/{vmi,vmdisk} scripts as their
harness, but select-e2e.sh mapped an examples/backups/<dir> edit to a suite only
when <dir> was itself a suite name. The dirs are vmi/vmdisk while the suite is
vminstance, so a harness-only edit selected no suite and the round-trip that runs
it never ran. Translate both dirs to vminstance, matching the vm-disk-application
entry in src_to_suites, and pin it with a bats case. The suite runs the vmi demo
with SKIP_RESTORE_TO_COPY=1, so 07-restore-to-copy.sh stays docs-only: selecting
the suite for it would pass without executing it. Both round-trips write to
the in-cluster seaweedfs-s3 endpoint, so a seaweedfs change selects the
vminstance suite as well. Also correct the controller-
owned-teardown note in docs/agents/e2e-testing.md (only Backup/RestoreJob
finalizers reach Velero; a BackupJob has none).

The two round-trips are also the only Tests that run a Velero backup and
restore, so the velero PackageSource maps to the vminstance suite; it
otherwise maps to a velero suite that does not exist, and a Velero change
ran neither round-trip.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit 58b4343e5a)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 13:46:50 +00:00
Andrey Kolkov 564ca281e4 test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.

Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
  up via BackupClass + BackupJob, restores in place, and asserts the VM returns
  Running off a disk PVC the restore created: a new uid, Velero's restore-name
  label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
  enough, because existingResourcePolicy: update also stamps it on a PVC Velero
  only patched, which moves no data. A guest-side boot proof is unavailable: the
  KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
  place with the same three assertions. The controller prepares the target
  (suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
  class the Velero data mover waits for a Pod to select a node for the restored
  PVC, which a detached disk never gets, so the script starts a pause Pod on the
  new PVC, standing in for the VM that would attach it.

The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
(cherry picked from commit 9cc3a9e459)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 13:46:50 +00:00
myasnikovdaniil 2c3ff51a2a [Backport release-1.7] feat(backups): back up Valkey with the Redis strategy (#5032)
# Description
Backport of #4898 to `release-1.7`.
2026-10-21 18:45:39 +05:00
Andrey Kolkov e2e0684b19 test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app
kind and the chart prefix the helpers derive sentinel and Secret names
from. The demo BackupClass routes Valkey to the same demo strategy.

The valkey suite gains valkey-backup-roundtrip, which drives that demo
end to end: seed a marker, back it up, restore it in place and into a
copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a
seaweedfs change now selects the valkey suite too, and since the
round-trip runs examples/backups/redis as its harness, an edit there
selects both the redis and the valkey suites. The reduced E2E install
keeps the backup stack for the valkey suite, as it does for the other
backup round-trip suites.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
(cherry picked from commit 43b02bda30)
2026-10-21 10:57:11 +00:00
Andrey Kolkov 82af1a457b feat(backups): back up Valkey with the Redis strategy
Valkey runs on the same spotahome RedisFailover operator as Redis and
speaks the same replication protocol and RDB format, so it needs no
strategy kind of its own. The only difference the backup Job sees is the
chart's resource-name prefix: valkey-<app> instead of redis-<app> for the
sentinel Service and the auth Secret.

The Redis driver now accepts applicationRef kind Valkey as well as Redis
and hands the template .Release.Prefix, derived from the app's kind, which
cozy-default-redis uses for every name it builds. A kind with no known
prefix fails the render instead of producing an empty one. cozy-default
routes apps.cozystack.io/Valkey to cozy-default-redis. A restore must
target an app of the same kind as the backup's source: replaying a Redis
dump into Valkey, or the reverse, is not something the strategy can
promise across engine releases.

The shipped template reads the prefix as `.Release.Prefix | default
"redis-"`: during an upgrade the previous controller, which has no Prefix
in its render context and serves only Redis, renders the new CR, and a
bare reference would print "<no value>" into the names.

The redis:8 image already loads RDB v11, which both shipped Valkey
versions (7.2, 8.1) write.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
(cherry picked from commit 0c2f131192)
2026-10-21 10:57:11 +00:00
Myasnikov Daniil cc8d6a4f05 fix(cluster-api): resync capi-operator every minute
A restarted capi-operator v0.19.0 drops the create events for the
provider Deployments it finds already running
(kubernetes-sigs/cluster-api-operator#1031). A provider whose Ready
condition went False while the operator was down therefore keeps it
until that Deployment changes or the informer resyncs, and the first
resync after a restart lands two periods later because the handler
registers after the informer has listed. With the 10m default that is
about 20 minutes, during which capi-providers-core times out on the
CoreProvider and everything depending on it waits; every further
operator restart starts the clock again.

With a one-minute period the same repro recovered 2m02s after the
operator took its lease, against no recovery within 4m26s with the
default on the same cluster. A resync replays the cache and, with
providers in sync, ends in "No changes detected" and an unchanged
condition, so it writes nothing to the API server; the operator logged
about 15 lines a minute and used 2m CPU and 26Mi of memory.

The upstream chart has no value for the flag, so a package patch that
make update reapplies adds one, and a bats test holds the rendered
argument.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit c2dab12a2e)
2026-10-21 10:54:55 +00:00
myasnikovdaniil f39982a30f [Backport release-1.7] fix(backups): keep the backupstrategy-controller Helm release under the Secret size limit (#5023)
# Description
Backport of #4941 to `release-1.7`.
2026-10-21 15:20:40 +05:00
Myasnikov Daniil b6ed4bb0cd test(hack): keep every chart's stored Helm release within budget
A chart can grow past the 1 MiB Helm release Secret without any test
noticing: it renders, lints and installs, and only the upgrade whose
manifest is large enough fails, on a live cluster, with a release that
stays failed. backupstrategy-controller sat under 1 KB from the limit
before it crossed it.

Estimate the stored size the way Helm encodes it (release JSON with the
chart's templates and files, gzip, base64) and fail any chart under
packages/ above 90 percent of the limit, plus the largest revision of
backupstrategy-controller with every default Strategy rendered. The
estimate counts files a .helmignore names, because the production path
stores them.

A size measured without its manifest is a lower bound that passes for
the wrong reason, so every chart must render offline except an explicit
list of charts that need cluster state, and the worst-case render must
succeed and contain the default strategies before it is measured.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7ad7d39349)
2026-10-21 09:48:25 +00:00
Myasnikov Daniil 22d7f4d04c test(backups): fail on a strategy CRD that inlines a pod schema
The release-size budget only reports that the backupstrategy-controller
release grew too large, not which field did it, and three strategies
once inlined the upstream pod schema before anyone noticed. Check the
generated strategy CRDs for an inlined PodSpec and name the property
path, with the markers that fix it.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 513b5088a8)
2026-10-21 09:48:25 +00:00
Alexey Artamonov 279ad817a0 build(seaweedfs): build SeaweedFS in-tree with a bounded read-ahead
The S3 gateway is OOM-killed while serving a large object. SeaweedFS
starts read-ahead downloads after every read of a sequential stream and
counts only the downloads it newly starts toward its prefetch window,
so a GET reading 256 KiB at a time out of 4 MiB chunks pulls up to
about a gigabyte of the object into memory, past the 512Mi limit of
the S3 deployment. No SeaweedFS release carries a fix yet, so there is
no release to move to instead.

Build weed from the 4.45 sources with a patch that keeps the window at
the next prefetchCount chunks, and lay it over the upstream image of
the same tag so the rest of the image stays upstream's own. The image
build runs the tests of the packages the patch touches, including the
regression test the patch adds, because go build never compiles them.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 1913104bd3)
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-10-21 09:19:39 +00:00
Aleksei Sviridkin fc38a540c0 fix(kafka): skip the OpenAPI patch lookup in the migration hook
The pre-install/pre-upgrade hook applies the node pools and the drain
KafkaRebalance with kubectl, and its container has no resources of its
own, so in a tenant with resource quotas it runs under the LimitRange
default of 128Mi. When it applies an object that already exists,
kubectl downloads the cluster's whole OpenAPI v2 document only to
decide that a custom resource takes a JSON merge patch: with the 374
CRDs of an e2e cluster that peaks at about 300Mi, so every upgrade of a
Kafka in such a tenant OOM-kills the hook and fails with "pre-upgrade
hooks failed".

--openapi-patch=false skips that lookup and builds the same three-way
merge patch from the last-applied annotation, peaking at about 35Mi on
the same cluster. A server-side apply would be as small but changes
what gets removed: the "kafka" pool is created with "kubectl create",
whose manager keeps co-owning every field it set, so a later
server-side apply leaves those fields in place when the chart drops
them. A larger memory limit would charge every upgrade against the
tenant quota instead.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
(cherry picked from commit 6ba440f7a3)
2026-10-21 09:17:09 +00:00
myasnikovdaniil 37c82f301c [Backport release-1.7] fix(fluxcd): close the flux-plunger review follow-ups for the hook, drain and Warnings (#4996)
# Description
Backport of #4914 to `release-1.7`.
2026-10-21 12:03:09 +05:00
myasnikovdaniil 3e4295b4a6 [Backport release-1.7] fix(backups): gate every default Strategy CR on its CRD (#4935)
# Description
Backport of #4929 to `release-1.7`.
2026-10-21 12:02:40 +05:00
Alexey Artamonov a8e491646a fix(fluxcd): keep the hook waiting on a release it cannot read
The pre-delete hook read a release's state once, so a single failed
read was taken as the release being handled: its fence came off before
helm-controller's uninstall, and the report counted it as released
cleanly. A state read is now retried within the release's share of the
window, a release whose state still cannot be read keeps its fence for
the wait like one helm-controller has not handled, and the end of the
wait reads each again before naming what the release was left to.

A release still suspended after the sweep is named as suspended but not
by flux-plunger, which also holds when no manager is recorded for the
field at all.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 7011510456)
2026-10-20 12:44:59 +00:00
Alexey Artamonov 683e80ba0c fix(fluxcd): give each release its own share of the hook's retries
The pre-delete hook's retry budget was one window spent in listing
order, so a release that could never be read or unfenced, listed
first, took all of it and every release after it got a single attempt.
The same transient failure then released a fence or not depending on
where its release sorted. Each release now gets its own share of the
window, capped at the window's end, and the steps that cover every
release keep the whole of it.

Two comments said more than holds. The retry window is measured from
the script's start, not the Job's, so scheduling and the image pull come
out of its one-minute reserve rather than being covered by it. And the
five-minute conflict retry inside kubectl scale never runs here, since
without a precondition it sends a merge patch with no resourceVersion
to conflict on.

helm-controller's finalizer, which the hook reads to tell whether a
release being deleted is still held, is now tied to the constant the
controller uses, as the fence already was.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 699555c7cb)
2026-10-20 12:44:59 +00:00
Alexey Artamonov 8416aea08b fix(fluxcd): let helm-controller uninstall before the hook unfences
The pre-delete hook lifted flux-plunger's suspension of a release being
deleted and removed its fence in the same step. helm-controller runs the
uninstall of a release being deleted once it is no longer suspended, but
when its own finalizer is already gone the fence is all that keeps the
object, so removing the fence at once deleted the release before
helm-controller could reconcile it, without an uninstall. The hook now
keeps such a fence while helm-controller handles the generation the
lifted suspension wrote, until shortly before its retry window closes,
and names a release it still had to let go of as before.

A release being deleted under a suspension another manager also holds
was reported as cleanly released, although helm-controller skips the
uninstall of a suspended release; it is now named as deleted without
one and counted with the others.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit af50c4a6b4)
2026-10-20 12:44:59 +00:00
Alexey Artamonov 021040795b fix(fluxcd): keep Helm uninstall working during flux-plunger recovery
flux-plunger recovers a "has no deployed releases" HelmRelease by
briefly suspending it, deleting the stale storage Secret and
unsuspending. helm-controller v1.5.x skips the uninstall and drops its
finalizer when it handles a delete on a suspended HelmRelease, and it
wakes on the same delete event flux-plunger does, so a delete inside
that window orphaned the release's DataVolume, PVCs, RBAC and storage
Secret. Lifting the suspension on the delete event cannot win that race
reliably.

Hold the release with a flux-plunger finalizer while it is suspended.
The finalizer is applied together with the suspension in one server-side
apply. On a delete flux-plunger lifts its suspension and keeps the
finalizer, helm-controller reconciles the unsuspended generation and
runs the uninstall, and flux-plunger removes the finalizer once
helm-controller's own is gone and that generation is observed. This
holds whichever controller handles the delete first. A completed
recovery lifts the suspension and the finalizer together, and a
terminating release never begins recovery.

Decide ownership of a suspension by which field manager owns
spec.suspend in metadata.managedFields rather than by an annotation. The
apiserver transfers the field away the moment an operator, cozyhr resume
or a backup controller writes a different value to it, so a later
suspension is never misread as flux-plunger's, and one set through the
shared flux-client-side-apply manager (platform migrations, the tenant
delete hook, a manual kubectl apply) is never claimed. A release deleted
under another manager's suspension is left to that manager's decision.

Acquire applies without forcing first and, on a conflict, re-fetches and
forces only if spec.suspend is still false: a backup controller owns the
field at false through a dynamic-client Update, which a non-forced apply
can never take, while true is an active suspension to leave alone. Every
apply is preconditioned on the resourceVersion and uid of the read that
decided on it. Without the resourceVersion, the unforced apply of true
over a foreign true is accepted as shared ownership, a window the
optimistic-locked merge patch this replaces had closed, and lifting the
suspension and the finalizer from a release read as live would delete it
without an uninstall if helm-controller had meanwhile handled a delete.
Without the uid, an apply to a release that has gone would create an
empty one. The apply carries only spec.suspend and the finalizer, so it
cannot leak the required spec.interval.

Read the release, every re-check and the storage Secret list from the
apiserver rather than the informer cache. A lagging cache would let the
processed-version guard delete the revision below the one just
reclaimed, and the ClusterRole grants secrets get and list but not the
watch an informer needs.

Emit a Warning when flux-plunger holds a suspension it failed to clear,
and Normal events when it leaves an external suspension alone or defers
a recovery. An external suspension on a release flux-plunger has
recovered before is a Warning instead: it is most likely one an earlier
flux-plunger left under the shared manager, which nothing lifts unless
an operator does. A release deleted under another manager's suspension
is a Normal event when that was a migration's choice, and a Warning when
the release carries flux-plunger's processed-version annotation, since
then the skipped uninstall is an accident.

Keep a single writer with leader election. A second process able to act
under the same field manager could clear a suspension the first is still
recovering, and no rollout strategy prevents one: the next pod starts as
soon as the old one is terminating, and an eviction or a node drain
replaces a pod the same way. The old pod keeps its Lease, unreleased,
through its stop and drain, and the Lease outlasts both, so the next pod
acts only after the old one is done; the price is that a new pod waits
for the Lease to run out. The strategy sets maxSurge 0 with
maxUnavailable 1 and stays RollingUpdate because existing installs carry
the apiserver-defaulted rollingUpdate block and helm-controller applies
server-side, where type Recreate over it is rejected on upgrade.

Give the finalizer a way out when flux-plunger itself goes. Once the
manager has stopped, within a bounded time, and with it every reconcile,
flux-plunger drains its finalizers under a deadline that, with the stop,
fits inside the pod's termination grace period. A stop that overran its
bound is drained too and still reported, and only a pod that led drains.
The drain writes its events straight to the apiserver, since the
manager's recorder stops with the manager. A recovery in progress is
aborted, the suspension and the finalizer lifted together for the next
flux-plunger to start again, and a release being deleted has the
suspension lifted and keeps the finalizer until helm-controller has run
the uninstall. A release helm-controller does not handle in time keeps
it. The waiting lives in the controller, where the envtest suites hold
it against a real apiserver's generations and finalizers, rather than in
a shell script against a model of them.

A pre-delete hook on the chart is the backstop for what the drain could
not finish, so uninstalling the component never leaves a release that
tenant teardown waits on forever. It stops flux-plunger, waits out its
grace period, then lifts any remaining suspension and finalizer without
waiting again, naming each release it lets go of before helm-controller
uninstalled it. It lifts a suspension only where flux-plunger alone
holds it, with a JSON patch preconditioned on the resourceVersion and
uid it read: kubectl apply --server-side on a release last written by a
client-side apply moves that apply's fields to the applying manager and
strips every one the body leaves out. The Job's deadline sits under the
300s helm-controller gives a pre-delete hook by default. A Deployment
that cannot be stopped or a listing that never succeeds ends the hook
before any finalizer is touched, and a finalizer it cannot remove ends
it after the sweep; either way it exits non-zero: a failed pre-delete
hook leaves the release untouched and helm-controller retries the
uninstall, hook included, from a state it can still converge from, where
exiting 0 would remove the component and strand the finalizer. A
flux-plunger pod still present after its grace period, typically one
stuck Terminating on a NotReady node, is only reported, since failing on
it would fail every retry until the node returns; because it may still
be setting finalizers, they are listed again after the sweep and any
found fails the hook. The Job runs once, since helm-controller's retry
is the one that starts afresh, and is kept after success so that its
report stays readable; helm.sh/hook-output-log-policy would make
helm-controller call a nil output writer (fluxcd/helm-controller#1562).
A delete still held after ten minutes raises a Warning, and the held
state is re-checked on a timer instead of only on helm-controller's next
status write. The chart README documents the finalizer and how to clear
it.

Run the envtest suites on every pull request through a new make target
that fetches a kube-apiserver and etcd with a pinned setup-envtest and
takes the HelmRelease CRD from the in-tree Flux manifests: the unit
tests drive a fake client, and everything the finalizer rests on (apply
preconditions, the finalizer set-merge, real managedFields) only holds
against a real apiserver. CI caches those binaries, and the target uses
them without asking the release index, so a pull request does not depend
on their sources being reachable; the control plane gets a longer start
timeout because it starts beside the other test goals.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
(cherry picked from commit 42a27567ce)
2026-10-20 11:14:46 +00:00
Myasnikov Daniil dee1eef559 test(backups): fail CI on any default Strategy CR rendered without its CRD gate
The CRD gate has now been missed twice: MongoDB shipped without it, and
Kafka, RabbitMQ and Redis followed after it existed. helm-unittest only
checks the templates a case lists, so a new strategy-*-default.yaml
that forgets the gate stays green until an upgrade wedges.

Render the chart with the bucket gate open and no CRD served, and fail
on any strategy.backups.cozystack.io object from any template. Also
check that every rendered Strategy CR gates on the CRD this chart ships
for its kind, because a gate naming the wrong CRD passes the first
check and keeps the CR out of every live render.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit f47bab7280)
2026-10-16 08:02:12 +00:00
myasnikovdaniil ebceef4105 ci(retention): sweep packages named by the published nightly tree (#4826)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

The Artifact Retention workflow has failed every day since mid-August,
and it never pruned anything. It lists the org's container packages with
`GET /orgs/cozystack/packages?package_type=container`, and that endpoint
answers `400 Invalid argument` to the app installation token the job
uses. Others hit the same in actions/github-script#449 and cli/cli#9606.
An old `|| true` used to hide it. This is item A19 in #3278.

The per-package endpoints accept the same token: a probe run confirmed
it, and the dry-run below read every package with it. So the job no
longer lists the org. The new `hack/retention-packages.sh` takes the
package names from the published `cozystack-packages:nightly` tree,
reading refs through the shared `hack/lib/image-refs.sh`. It also adds
`cozy-installer` and `cozystack-nocloud`, which the nightly publishes
outside the tree. The committed tree can't be the source: it misses
images whose digest only the build pins, and those still get dated
nightly tags every day.

Failures now turn the run red instead of passing silently:

- if the name list is empty, the nightly sweep is skipped and the run
fails, but abandoned promotion candidates are still handled, because
their protection doesn't depend on that list
- if a package's versions can't be read, that package is skipped and the
run fails at the end
- if a DELETE is refused, `hack/retention-delete.sh` logs an error with
the HTTP status, goes on with the remaining versions and fails the run
(before, this was only a warning)
- if names resolve but none of the packages is linked to this
repository, nothing gets swept, so the run fails too, after the
promotion sweep has run (that sweep itself is skipped only when
candidate protection can't be resolved)

Promotion-candidate protection itself is unchanged. The contract test
that pinned the old org listing now pins the new behaviour, and both new
scripts have bats tests.

Known gap: a component that disappears from the nightly tree drops off
the list, so its old dated nightlies are never pruned. Today none of the
packages outside the list has more dated nightlies than the keep window.

Dry-run on this branch is green:
https://github.com/cozystack/cozystack/actions/runs/38075159900. It
resolves 59 packages, skips `cozystack/cozystack-ui` with the warning,
and would delete 418 old nightly versions plus one abandoned promotion
candidate. Deleting needs a different token. The app token lists
versions, but a one-version apply run
(https://github.com/cozystack/cozystack/actions/runs/37939905343) got
`404 Package not found` on DELETE for a version it had just listed. The
same DELETE with this repository's `GITHUB_TOKEN` and the job's
`packages: write` worked in a second run
(https://github.com/cozystack/cozystack/actions/runs/37941084456), with
no change to package settings. So `hack/retention-delete.sh` now deletes
with `GITHUB_TOKEN`, and listing stays on the app token.

That delete was shown on a package linked to this repository. One of the
swept packages, `cozystack/cozystack-ui`, is linked to another
repository, and I didn't try deleting there. Such a package is skipped
before its versions are read and named in a warning, so it can't turn
every apply run red. The follow-up for it is in #3278.

`RETENTION_APPLY` stays off, so the scheduled run is a green dry-run
until someone turns it on.

### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

N/A

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
ci(retention): the GHCR retention workflow takes package names from the published nightly tree instead of the org package listing, which rejects the app token, and fails the run whenever names, versions or deletions cannot be resolved.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Nightly package retention now targets packages referenced by the
published nightly tree and keeps the 14 newest eligible versions per
package.
* Promotion candidates older than 30 days are eligible for pruning only
when their digests are not protected by qualifying release pull requests
or branch tips.
* **Bug Fixes**
* Packages linked to another repository are skipped, and sweep failures
are reported without preventing other eligible packages from being
processed.
* **Documentation**
* Clarified which nightly versions and promotion candidates qualify for
retention.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 12:02:35 +05:00
myasnikovdaniil 34488db766 ci(e2e): never tear down the Proxmox tenant on the runner's default context (#4807)
## What this PR does

The Proxmox e2e teardown could delete from the wrong cluster. It runs
under `always()` and got its kubeconfig only from `GITHUB_ENV`, which
the "Stage credentials" step writes. If a step before that fails, for
example a missing stand variable, `KUBECONFIG` is never set. The three
`kubectl delete` calls then go to whatever cluster the self-hosted
runner's own kubeconfig selects.

Every teardown call now passes the staged file with `--kubeconfig`.
Stage fails unless that file's current context resolves to a server, and
prints kubectl's own error when it does not. Before, an empty
`COZY_KUBECONFIG` secret let client-go fall back to its defaults,
in-cluster credentials among them. The suite step gets the staged path
in its own `env`. The `GITHUB_ENV` export is gone, so no step in this
workflow picks its cluster up implicitly.

Tear down decides whether to skip from how Stage ended, not from the
file. If Stage did not succeed, the suite never ran and there is no
tenant to remove, so the step prints a notice and exits 0. If Stage
succeeded and the file no longer resolves, the step fails with kubectl's
error, because the tenant and its VMs may still be there. A side effect:
a run whose Stage fails no longer cleans up a tenant left by an earlier
`keep_tenant` run. The next run that passes Stage without `keep_tenant`
removes it.

The Stage and Tear down scripts now have unit tests in
`hack/proxmox-teardown-contract.bats`, run against a stub `kubectl`. I
also ran both against a real kubectl v1.37.1 for every Stage outcome.
The files were missing, empty, garbage YAML, without a current context,
with a dangling context, without a server, and valid. One more case was
a runner without kubectl.

The deletes also no longer stop at the first failure. Under `set -e`, a
failed first delete skipped the tenant delete, and the tenant's VMs
stayed on the hypervisor. Now each delete runs, and the step fails at
the end if any of them failed. With a recording `kubectl` that fails one
chosen delete, all three ran in every case and the step exited 1. With
the change reverted, a failing first delete was the only one that ran.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): the Proxmox e2e teardown always uses the staged kubeconfig, so it can no longer delete from the runner's default cluster. It skips only when Stage did not succeed and fails when the kubeconfig breaks after Stage. A failed delete no longer skips the tenant delete.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Improved reliability of automated Proxmox test setup and cleanup.
Tests now stop setup when Kubernetes configuration is unavailable, skip
cleanup when setup fails, and attempt all resource deletions before
reporting cleanup failures.
* **Tests**
* Added checks for configuration validation, cleanup behavior, and
handling of failed deletion attempts.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 12:02:25 +05:00
myasnikovdaniil 2db18a0c86 test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport (#4657)
## What this PR does

The kubernetes suites' LINSTOR cleanup check now compares dataset names
per node, and prints the leftover names when it fails (#4651). This PR
puts the rest of what a triager needs into `cozyreport.tgz` on every
run, green or red: the size of each dataset, its `origin` and
`creation`, the snapshots, the pool's pending frees, and the LINSTOR
volume list.

What `hack/cozyreport.sh` now collects:

- `linstor/zfs-pools.txt`: the zpool behind each ZFS storage pool, per
node, taken from `StorDriver/StorPoolName` in `linstor
--machine-readable sp l`.
- `linstor/zfs/<node>.txt`: that node's satellite runs `zpool get -p
freeing,leaked,allocated,free` and `zfs list -p -t all -o
name,used,avail,refer,origin,creation -r` over the node's pools.
- `linstor/volumes.txt`: `linstor v l`, next to the existing
`resources.txt`.

Sizes are exact bytes, so the sizes of leftover datasets can be summed
against the pool root's `avail`. That `avail` can differ from
`storage-pools.txt` without anything leaking: on a node with only thick
pools, `linstor sp l` serves the free capacity cached at the satellite's
last report. A dataset LINSTOR could not destroy because snapshots
depend on it is renamed to `_deleted_<name>` in the same pool, so a
search for leftovers has to look for that prefix as well as `pvc-`.

A pool that `zpool get` or `zfs list` cannot open does not cost the
listing of the other pools. `zpool get`'s message and exit status go
into the listing, and `zfs list` lists the rest. A node LINSTOR names no
ZFS pool for gets a note instead of a listing, because `zfs list -r`
with no pool argument lists every pool on the machine. A satellite whose
`spec.nodeName` cannot be read gets a note that says so. All reads go
through the bounded exec reader, like the other LINSTOR reads, and the
time budget in the script header counts them.

The second commit changes how that reader marks a read that wrote output
and then exited non-zero without hitting the timeout. It used to append
`TRUNCATED ... part way through, not because the command finished`. But
`kubectl exec` passes the remote command's status through, and `zfs
list` over one missing pool prints a complete listing of the others and
exits 1. The marker now names the exit status and says the status can't
tell an error after complete output from a cut stream. This changes the
marker for every LINSTOR exec read. The timeout marker stays as it was.

Tested with `hack/cozytest.sh hack/cozyreport.bats`. The new tests run
the controller and satellite shells against stub `linstor`, `zpool` and
`zfs`, with the real `jq`. They cover a failed and an unparsable
storage-pool read, a pool row with a missing, empty or blank backing
name, a non-ZFS pool, a pool name with a space, a pool that `zpool` and
`zfs` cannot open, a failing `zfs`, and an unreadable node name. A test
goes red if any of these break: the filter that keeps non-ZFS pools out,
the filter that drops a missing, empty or blank pool name, the jq exit
status, the `zpool` fallthrough, the node-name note, or the reader's new
wording.

Refs #4653

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(e2e): cozyreport records the ZFS datasets of each node's LINSTOR pools in exact bytes, and the LINSTOR volume list, on every e2e run.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Collection reports now include LINSTOR volumes, ZFS pool capacity and
usage, and per-node ZFS datasets and snapshots with exact-byte details.
Dataset listings are scoped to the ZFS pools LINSTOR reports for each
node.
* Reports include notes or warnings when node, pool, or dataset
information is missing or cannot be collected.

* **Documentation**
* Clarified storage report contents, artifact markers, bounded reads,
and how unsupported data, incomplete results, collection limits, and
per-object warnings are reported.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 12:00:15 +05:00
Aleksei Sviridkin 33ca8b67bb feat(redis): run Sentinel on the same image as the Redis pods (#4831)
## What this PR does

The Redis application pins its Sentinels to `redis:6.2.6-alpine`
(`packages/apps/redis/templates/redisfailover.yaml`), while the data
pods run the version selected by `version` (`redis:8.4.0` for `v8`,
`redis:7.4.7` for `v7`). The pin was added in fde96dd7d so that the
freshworks-oss operator's new compiled-in default (`redis:7.2.4-alpine`)
could not move the Sentinels silently on upgrade, and its comment asks
for the bump to be made as a deliberate, separate change. This is that
change.

Redis 6.2 is end of life, and today every instance runs a Sentinel one
or two major versions behind the Redis it monitors. With this change
`spec.sentinel.image` is rendered from the same `files/versions.yaml`
entry as `spec.redis.image`, so a `v8` instance runs Redis and Sentinel
on `redis:8.4.0`, and a `v7` instance on `redis:7.4.7`. Nodes then pull
one engine image per instance instead of two.

The rollback concern behind the original pin was dump.rdb compatibility.
It does not apply to Sentinel: a Sentinel pod stores no RDB or AOF, only
a `sentinel.conf` on an `emptyDir` that the operator's
`sentinel-config-copy` init container rewrites from the operator
ConfigMap on every pod start. Rolling back the chart therefore starts
the older Sentinel image from a clean config. The image stays explicit
in the rendered RedisFailover, so an operator upgrade still cannot move
it.

Changes:

- `templates/redisfailover.yaml`: `spec.sentinel.image` uses
`redis.versionMap`, with the comment rewritten to say why it is pinned
and why a rollback is safe.
- `values.yaml` and `hack/update-versions.sh`: the `version` description
says that Redis and Sentinel run the same image (`update-versions.sh`
rewrites that line on `make update`, so it carries the same text).
`README.md`, `values.schema.json`, `api/apps/v1alpha1/redis/types.go`
and `packages/system/redis-rd/cozyrds/redis.yaml` are the `make
generate` output for that description.
- `tests/sentinel_image_test.yaml`: new helm-unittest suite; it asserts
that the Sentinel and Redis images share the major version for `v8` and
`v7`. The patterns pin the major version only, so a patch bump in
`files/versions.yaml` needs no test change.
- `hack/e2e-chainsaw/redis/chainsaw-test.yaml`: comment only. It
described the Sentinel container's `timeout` as busybox (exit 143); the
Sentinel now runs the Debian-based image with coreutils (exit 124). The
checks only read "non-zero", so behaviour is unchanged.

Upgrade behaviour: on the first reconcile after the upgrade, every Redis
instance rolls its Sentinel Deployment once (the operator does not set a
strategy, so the default `maxSurge: 1`, `maxUnavailable: 0` applies to 3
replicas). A new Sentinel pod starts from the initial config, the
operator points it at the current master (`NewSentinelMonitor`, which
also sets `auth-pass`), and only then does it pass the readiness probe
that lets the next old pod go. The data pods are not touched.

Not changed: Harbor's Redis
(`packages/system/harbor/templates/redis.yaml`) keeps both images on
`redis:6.2.6-alpine`. Its data pods are PVC-backed, so the rollback
concern still applies there and it needs its own decision.

Testing done:

- `make generate` in `packages/apps/redis`: regenerated files committed;
a second run leaves no diff.
- `make test` in `packages/apps/redis` (helm-unittest v1.1.2): 50
passed. With the template change reverted, the 2 new tests fail and the
other 48 pass.
- `helm template` with `version=v8` and `version=v7`:
`spec.sentinel.image` and `spec.redis.image` are both `redis:8.4.0` and
both `redis:7.4.7` respectively.
- `helm lint` reports the same two errors as on `main` (icon URL,
`cozy-lib` not declared as a dependency), so nothing new.
- I checked the operator's Sentinel command set (`SENTINEL
REMOVE/MONITOR/SET auth-pass/SET
down-after-milliseconds/RESET/CKQUORUM/MASTER`, `INFO sentinel`) against
a Redis 8.6.2 Sentinel locally. It behaves the same as on 6.2. I have
not run the e2e suite. On a Cozystack v1.6.4 cluster we run Redis 8.4.0
data pods with 6.2.6 Sentinels today, but not this exact chart.

Backwards compatibility: no API change. Existing instances roll their
Sentinels once, as described above. Rolling the chart back rolls the
Sentinels back to `redis:6.2.6-alpine` with a fresh config.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Trigger map walked against the diff: the only user-visible text change
is the `version` description in `values.yaml`/`README.md`. `redis` is in
the website `Makefile` `APPS` list, so the release bot regenerates its
page. No field, enum, default or name changes, so
terraform-provider-cozystack is not affected.

### Release note

```release-note
feat(redis): Sentinel pods now run the same Redis image as the data pods (redis:8.4.0 for v8, redis:7.4.7 for v7) instead of the end-of-life redis:6.2.6-alpine. Existing instances roll their Sentinels once on upgrade; data pods are not restarted.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Redis Sentinel now uses the image selected by the Redis version
setting, keeping Sentinel and Redis pods on the same major-version
image.
* **Tests**
* Added checks confirming Sentinel and Redis images match with the
default version and when Redis 7 is selected.
* **Documentation**
* Clarified that Redis and Sentinel pods use the same image when
choosing a Redis version.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-11 22:26:21 +03:00
Aleksei Sviridkin 9a348eeb50 fix(kubernetes): delete orphaned resources in new post-delete job (#3856)
## What this PR does
When tenant deletes the kubernetes cluster, 3 resources are orphaned and
leftover in the tenant namespace.
- jobs/<name>-oidc-bootstrap
- secrets/<name>-talos-ca
- secrets/<name>-talos-tls-cert

This PR adds a new "step 7" in existing
`packages/apps/kubernetes/templates/delete.yaml` in order to delete
these orphaned resources manually.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:

### Release note

```release-note
fix(kubernetes): Fixed the `<name>-talos-ca` and `<name>-talos-tls-cert` Secrets and the `<name>-oidc-bootstrap` Job being left in the tenant namespace after a Kubernetes cluster is deleted.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Bug Fixes**
* Kubernetes release cleanup now removes the Talos CA and TLS
certificates before their associated Secrets, and also removes the OIDC
bootstrap Job.
* The final cleanup report now includes best-effort failures from Step
7, alongside failures from Steps 3, 5, and 6.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-11 20:08:27 +03:00
Aleksei Sviridkin be55f6eb2b test(ci): load the shared bats helper in the labeler contract test
Every unit bats file under hack/ must load test_helper so it runs with
nounset, and hack/bats-strict-setup.bats fails the suite for a file
that does not.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-11 18:42:11 +03:00
shaurya703 79e95a1b30 test(ci): pin the label-writing workflows with contract tests
pr-labeler.yaml and stale.yaml decide which labels land on issues and
PRs, and neither had a test behind it. A mistyped label name is not an
error, it is a rule that silently stops matching.

hack/label-workflows-contract.bats pins:

- every label either workflow can write or match resolves against a
  name: in .github/labels.yml (typeToKind, scopeToArea, the three
  literal adds, stale/close labels, both exempt lists)
- the typeToKind mappings are exactly the rows of the type table in
  docs/agents/contributing.md, in both directions, and a type the table
  lists under "(no kind label)" has no key in the workflow
- the trigger list stays [opened, reopened, synchronize] and 'edited'
  does not come back in either YAML spelling
- the labeler stays additive-only (addLabels only, no removeLabel or
  setLabels) with the area/uncategorized fallback
- stale policy values: label choices, 12 issue and 4 PR exemptions,
  days-before-stale 60, days-before-close 14, operations-per-run 100
- stale runs from its daily cron and workflow_dispatch only

JS // and YAML # comments are stripped before matching, so a mapping or
a gate demoted to a comment cannot satisfy the contract. Negative checks
use the grep -c / -eq 0 form of the sibling contract files: a negated
pipeline is ignored by set -e, so it cannot fail a test under
hack/cozytest.sh and fails under bats only as the last command.

Every assertion was mutation-checked under both bats and
hack/cozytest.sh. POSIX-only constructs; picked up automatically by
BATS_UNIT_FILES.

Closes #3589

Signed-off-by: shaurya703 <kesar.shaurya@gmail.com>
2026-10-11 18:40:02 +03:00
Ray Low 904bc38fc3 fix(kubernetes): delete the Talos PKI and oidc-bootstrap Job on uninstall
cert-manager runs with enableCertificateOwnerRef: false, so the
<release>-talos-ca and <release>-talos-tls-cert Secrets carry no
ownerReference to their Certificates and outlive the release, keeping
the CA private key of a cluster that no longer exists. A new cluster
created under the same name then adopts that stale CA instead of
issuing its own trust root.

The <release>-oidc-bootstrap Job is a hook whose only delete policy is
before-hook-creation. Hooks are not part of the release manifest, so
uninstall never removes it.

The pre-delete hook deletes the two Certificates before their Secrets,
because deleting a Secret while its Certificate exists makes
cert-manager re-issue it. It runs after the TenantControlPlane is gone,
since the control plane's CSR signer mounts the CA. The Role grants
each delete by resourceName only.

Assisted-by: LLM
Signed-off-by: Ray Low <crossrt@gmail.com>
2026-10-11 22:40:47 +08:00
Yasar Rahman 9a88afe6ca feat(redis): run Sentinel on the same image as the Redis pods
The Sentinels were left on redis:6.2.6-alpine when the operator moved to
the freshworks-oss fork, so that its new compiled-in default could not
move them on upgrade. 6.2 is end of life, and every instance now runs a
Sentinel one or two major versions behind the Redis it monitors.

The rollback risk behind that pin, a newer engine rewriting dump.rdb in
a format the older one cannot load, does not apply to Sentinel. It
stores no RDB or AOF, only sentinel.conf on an emptyDir that the
config-copy init container rewrites from the operator's ConfigMap on
every start, so moving back to an older image starts from a clean
config. Taking the image from the same version map as the data pods also
means one image per node instead of two.

The e2e comment that relied on busybox's timeout exit code in the
Sentinel container is updated, since that container now carries
coreutils.

Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
2026-10-11 13:16:11 +05:00
Marian Koreniuk 6852f723b8 fix(kubernetes-nodes): refuse a newline in the Proxmox template tags
COZY_TEMPLATE_TAGS is comma-separated, but the normalisation also split on a
newline, so `talos-v1.\n13` became two tags where the operator meant one. A
newline there is now refused, like in every other value the program receives.

The stub tests clean up their scratch directory after the last assertion, so
a failing test leaves its qm, curl and ssh records behind for inspection.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-11 01:23:10 +02:00
Marian Koreniuk a8635f4f68 fix(kubernetes-nodes): refuse multi-line values in the Proxmox template script
matches() held each value to its shape with grep -x, which matches line by
line, so a value with a newline passed when any one of its lines did and
carried the others into the program create runs as root. A value with a
newline is now refused, for every field the program receives.

The test file loads test_helper and spells its absence check without `!`,
for the two Bats audits on main. The Talos version check accepts the
pre-release suffix the chart's render accepts, and a failure after qm create
names the qm destroy that frees the VMID for the next run.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-11 01:23:10 +02:00
Marian Koreniuk bbd39c94a4 feat(kubernetes-nodes): build the Proxmox worker template from the pool's Talos values
A proxmox pool clones its workers from a VM template the chart never sees,
and nothing said which Talos image that template has to carry. The one an
operator reaches for, the platform release's nocloud-amd64.raw.xz, is the
bare-metal node image: its zfs extension's service waits for a /dev/zfs no
VM has, the boot sequence never completes, and Talos reboots every worker
each time the boot timeout runs out, about every 70 minutes on v1.13. It
also follows the host nodes' Talos version, not the pool's talos.version
that the installer in the worker machineconfig names.

The chart's default schematic has a trap of its own: its qemu-guest-agent
service waits the same way for the agent's virtio port, which Proxmox adds
only to a VM with the agent enabled.

hack/proxmox-talos-template.sh builds the template from the chart's
talos.schematicID and talos.version in the nocloud flavour with the agent
enabled, refuses a tag set another template already carries, because
capmox then finds two and clones neither, and prints the templateTags
that select it.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-11 01:23:10 +02:00
Aleksei Sviridkin 22e0fdbc63 test(tests): stop unit tests from reaching a cluster (#4804)
## What this PR does

One unit test was reading from whatever cluster the caller's kubeconfig
selected. The knob test in `hack/run-kubernetes-node-join_test.bats`
runs `cozy_report_node_join_failure` in a fresh `bash -c` and stubs only
`kubectl`, as a shell function. `cozy_diag_read` runs every read through
`timeout`, and `timeout` execs the real `kubectl` from PATH, so the stub
never ran. With a recording `kubectl` first on PATH, one run of that
file logged 170 calls such as `kubectl -n tenant-test describe
virtualmachineinstances.kubevirt.io`. All of them were reads, but a unit
suite should not touch a cluster at all.

The PR has three parts:

1. Both `bash -c` blocks in that file now stub `timeout`, as the
post-source test already did. They also put recording `kubectl` and
`talosctl` executables first on the subshell's PATH and fail if either
one is executed. The declined-path block made no calls in my runs,
because a zero budget declines every read. It gets the same guard
because it has the same shape.
2. The unit and controller targets now run with no cluster to reach. The
root `Makefile` sets `KUBECONFIG=/dev/null` and an empty
`KUBERNETES_SERVICE_HOST` for `unit-tests`, each of its prerequisites,
and `test-controllers`, so the CI command and each target run on its own
are covered. The second variable is needed because client-go falls back
to in-cluster credentials when the kubeconfig is empty and that variable
plus a service-account token are present. The new
`hack/unit-lane-no-cluster.bats` runs every prerequisite of `unit-tests`
as its own goal with a sentinel kubeconfig and service host in the
environment, and fails if any target receives them.
3. One unit file was missing the strict-mode helper.
`hack/bats-strict-setup.bats` requires a `load test_helper` line in
every unit file, and `hack/fork-new-images.bats` is the only one without
it, so the bats lane is red on main and would stay red here. #4805
carries the same one-line fix on its own; if it lands first, this commit
drops out on rebase.

I ran the whole lane (`make unit-tests test-controllers`) with recording
fakes for 20 cluster clients first on PATH. It made no `kubectl` call.
`helm` was called 444 times, all `template` or `unittest`, none with
`--validate`. `flux` was called once, for `push artifact --help`.
Deleting either export line from the Makefile, or removing one target
from the list, turns the new test red. Removing the `timeout` stubs
turns the knob test red, and the failure lists the reads that escaped.

Closes #4793

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
test(tests): unit and controller test targets run with no kubeconfig and no in-cluster service host, so a unit test can no longer reach a cluster.
```
2026-10-10 23:47:52 +03:00
Aleksei Sviridkin 2e5fd56954 fix(e2e): stop calling a failed exec read after output a cut
kubectl exec passes the remote command's status through, so a
non-zero exit after output does not say whether the stream broke or
the command finished with an error. zfs list is the common case: it
lists every dataset it can open, skips one it cannot and exits 1, so
a complete listing got a TRUNCATED marker and a note claiming it held
only what was streamed before the exit.

The in-file marker now names the exit status and says the status
cannot separate the two, and the note beside it says the same. The
timeout branch keeps its own wording.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:47:42 +03:00
Aleksei Sviridkin d195a56793 test(e2e): record ZFS datasets and LINSTOR volumes in cozyreport
The report carries the LINSTOR resource list and the pool totals, but
nothing that names the datasets holding a pool's space, so a leftover
zvol, a snapshot, a clone and pool metadata cannot be told apart from
the artifact alone.

Each satellite now lists every dataset of the ZFS pools LINSTOR
records for its node, snapshots included, with the pool's pending
frees. Sizes are exact bytes with avail, so the sizes of leftover
datasets can be summed against the pool root's avail; the rounded
form loses a sub-GiB difference on a large pool. That avail can
legitimately differ from `linstor sp l`, which serves a cached free
capacity on nodes with only thick pools. The pool names come from
LINSTOR rather than from a name pattern. One pool that zpool cannot
open does not cost the listing of the others, and a satellite whose
node name cannot be read says so instead of pointing at the pool map.
The controller's volume list is recorded next to the resource list.
Everything runs on every job, so a red listing can be read against a
green one, and goes through the bounded exec reader like the other
LINSTOR reads.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:47:41 +03:00
Aleksei Sviridkin da1f1cc387 ci(retention): delete with the workflow token
The app installation token lists package versions but its DELETE of a
listed version is answered with 404, while the repository's
GITHUB_TOKEN with packages: write deleted the same version without any
change to package settings. Deletes therefore go through GITHUB_TOKEN;
listing keeps the app token.

That deletion was shown on a package linked to this repository. A
package linked elsewhere is not attempted: it is skipped before its
versions are read and named in a warning, so it cannot turn every
apply run red. A failed read of the link still fails the run, and so
does a sweep in which no resolved package is linked, since that sweep
prunes nothing.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:15:48 +03:00
Aleksei Sviridkin 274f3f7023 ci(retention): fail the run when a version cannot be deleted
A refused DELETE only logged a warning, so a token without delete
rights left an apply run green with nothing removed. Move the delete
into hack/retention-delete.sh, which reports each refusal as an error
carrying the API status, still tries the remaining versions, and exits
non-zero; the workflow turns that into a failed run after the sweep.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:10:34 +03:00
Aleksei Sviridkin f1c0f6f3f6 ci(retention): sweep packages named by the published tree
The org container-package listing answers 400 to the app installation
token, so the job has never reached its sweep. Take the names from the
published nightly tree via hack/retention-packages.sh and keep calling
only the per-package versions endpoints, which accept that token.

An unresolved name list now fails the run: the nightly publishes into
these packages daily, so an empty list cannot mean there is nothing to
prune. It skips only the nightly sweep. The promotion sweep reads one
fixed package and is kept safe by the protected set, which the name
list does not feed, so it still runs.

A versions read that fails, in either sweep, is reported with the API
error and fails the run once the rest is swept, instead of aborting
the step through set -e. A failed candidate read leaves no ids behind.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:10:33 +03:00
Aleksei Sviridkin bde3a01699 test(hack): pin the Proxmox lane's kubeconfig guards
Runs the Stage and Tear down scripts out of the workflow against a stub
kubectl, so a teardown that exits 0 with the tenant still live, stops
at the first failed delete, or hides kubectl's error, or a step that
reaches a cluster without naming the staged kubeconfig, fails in unit
tests instead of on the stand.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 21:09:09 +03:00
Aleksei Sviridkin a2e10312de fix(platform): stop treating a failed ConfigMap read as a fresh cluster (#4792)
## What this PR does

A single failed API read at the start of the platform migration Job
could skip every pending migration and still end green.
`run-migrations.sh` checked whether the `cozystack-version` ConfigMap
exists with a silenced `kubectl get` and looked only at the exit code. A
timeout, a dropped connection or an apiserver restart during the upgrade
all counted as "no ConfigMap", so the script stamped the target version
and exited 0. The hook Job is rendered only when that ConfigMap already
exists, so in practice an error was the only way into this branch.

Now the script creates the ConfigMap only when it is really missing. The
check uses `--ignore-not-found --output name`: kubectl drops NotFound
and nothing else, so empty output means a genuine NotFound. Any other
error keeps a non-zero exit, `set -e` aborts the script, and the Job
retries the pod. The rest of the script already works this way: the read
of the stamped version aborts on error, and the stamp runs under
`pipefail`.

The tests cover both branches. A new case in `hack/run-migrations.bats`
makes the existence check fail with a connection error and expects a
non-zero exit, no migration run and the stamp left where it was. It
fails against the old check. The fake kubectl returns empty output for a
missing ConfigMap only when `--ignore-not-found` is passed, as real
kubectl does, so dropping the flag breaks the existing "no ConfigMap"
case.

No migration is added, so `targetVersion` stays. The script ships in the
migrations image, and the release build restamps that image's digest as
usual.

Several older migrations have the same problem in a smaller form. They
check for an object with `kubectl get ... >/dev/null 2>&1` and skip
their own work when the check fails, for example migrations 31, 32, 36,
48, 50 and 51. A transient error there skips one migration's work, not
all of them. Editing migrations that already shipped has its own risk,
so I'd track that separately.

Closes #4579

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(platform): a transient API error while the platform migration Job checks for the cozystack-version ConfigMap now fails the pod so the Job retries, instead of stamping the target version and silently skipping every pending migration.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Migration checks now distinguish a missing version record from an
error while checking for it. When the check fails, the pod stops instead
of treating the record as missing and proceeding with migrations.
* Added coverage for TLS-handshake timeouts during the check, confirming
that migrations do not run and the stamped version remains unchanged.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 20:37:53 +03:00
Aleksei Sviridkin be2387323b feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does

This keeps the TLS-passthrough services on port 443 of the tenant
Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is
stacked on #3343 and has to land with it (stack #4717).

On 1.20, upstream Cilium marks an HTTPS-terminate listener and a
TLS-passthrough listener on one port `Conflicted` when their hostnames
overlap, and drops both from Envoy. That took `*.<apex>` and the `api`,
`vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down
together on every tenant Gateway in `dns01` and `existingSecret` mode
(#3912). #3343 now ships a patched operator that conflicts the pair only
when the hostnames are equal, so the layout stays as it is on main: one
Gateway, one address, everything on 443. What remains is the
equal-hostname case and the cleanup that 1.20 allows.

Under `http01`, a name a passthrough listener answers is now reserved by
the spec alone. Before, the controller dropped the terminate listener
for such a name only once a TLSRoute was attached to the passthrough
listener and its backend Service resolved, which copied how Cilium
builds filter chains. Without a working TLSRoute, an HTTPRoute on
`api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20
that equal-hostname pair takes both listeners down. Now such an
HTTPRoute gets no listener and no certificate, TLSRoute or not. If all
its hostnames are reserved it reads `Accepted=False` with
`NoMatchingListenerHostname`. If it also claims a free hostname it stays
`Accepted` and is served there. The backend lookup, its Service and
ReferenceGrant watches and their RBAC rules are gone.

Other changes:

- `tlsPassthroughListeners` is accepted under `dns01` and
`existingSecret`. The old refusal relied on 1.19.5 serving all ports
from one Envoy listener; 1.20 splits ports whose SNI overlaps.
- A TenantGateway reads `Ready=False` with reason
`GatewayClassUnsupported` when its GatewayClass lists features in
`status.supportedFeatures` but not `TLSRoute`, while it renders
passthrough listeners. A class that lists nothing is not judged. Cilium
1.20.2 lists `TLSRoute`.
- Comments, the README and one error message no longer describe Cilium
1.19.5 as the shipped release.
- The gateway chart writes `tlsPassthroughServices` on the publishing
tenant only, behind the same `gateway.enabled` switch and
`publishing.exposedServices` filter as the platform TLSRoutes. Before,
every tenant Gateway got the three passthrough listeners with nothing
routed to them, and the reservation above would have taken those names
from a child tenant's own HTTPRoutes.

The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The
Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so
removing it is a separate cleanup.

### Upgrade notes

Only the publishing tenant keeps the `api`, `vm-exportproxy` and
`cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless
`publishing.ingressName` says otherwise, and only for names in
`publishing.exposedServices` while `gateway.enabled` is on. Other
tenants lose these listeners on upgrade. Nothing shipped was attached to
them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its
listener and certificate.

Two narrow cases do change. On the publishing tenant under `http01`, an
HTTPRoute on an exposed passthrough name loses its listener and
certificate when the platform TLSRoute for that name forwards nowhere
(the owning package is not installed, or its backends do not resolve). A
TLSRoute attached by hand to a child tenant's `tls-*` listener loses its
parent. To keep that listener, merge-patch `tlsPassthroughServices` into
the tenant's `TenantGateway` after the upgrade. From then on the chart
does not render the key there, and later upgrades leave the patch alone.
`dns01` and `existingSecret` do not change: the wildcard listener still
covers such a name, and the passthrough listener answers it by its exact
name.

Tested locally: Go tests for the controller, `go vet`, helm-unittest for
`extra/gateway` and `cozystack-controller`, `make generate` and `make
manifests` with no diff. The gateway e2e scenario checks that a child
tenant renders no passthrough listener. It then declares `tls-api` and a
passthrough listener on a native port on that tenant, checks that the
tenant stays Ready with no route behind either, and checks the
reservation against `tls-api`. Unit tests catch a regression in either
change. In e2e, dropping the reservation turns the gateway suite red,
and dropping the chart scoping does not, because CI never sets
`gateway.enabled` (#4846). The controller was not run on a live cluster;
the operator patch it depends on was, see #3343.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/725
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* TLS passthrough listeners are supported with HTTP-01, DNS-01, and
existing-secret certificate modes. Edge-terminated Gateways remain
unsupported.
* Declaring a passthrough hostname reserves it from HTTPS termination
and certificate creation, even when no route is attached.
* Passthrough listeners are rendered only for exposed services on the
enabled publishing Gateway.
* **Bug Fixes**
* Gateways that declare passthrough listeners but explicitly lack
TLSRoute support are now reported as not ready.
* Routes targeting a hostname served by a passthrough listener on a
different Gateway are correctly withdrawn.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 16:59:17 +03:00
Aleksei Sviridkin ad3f15d4f3 feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a
name a passthrough listener answers only once a TLSRoute was attached
there and resolved a backend, mirroring how Cilium builds its filter
chains. That kept a served endpoint up on Cilium 1.19.5, but it tied
every route's status to one implementation's translator and needed
Service and ReferenceGrant watches to land the deferred decision.

The shipped operator accepts a wildcard terminate listener beside a
narrower passthrough listener on 443 and keeps conflicting every other
overlap. A terminate listener for the passthrough name itself is not
narrower, so on 443 the rule now protects the dataplane outright; on a
native port Gateway API would serve the pair, and the rule holds as a
policy that keeps a platform name with its passthrough listener. Reading
it off the spec makes the outcome the same on every implementation.

Under http01 an HTTPRoute claiming such a name gets no listener and no
certificate whether or not a TLSRoute is attached. A route whose every
hostname is reserved reads Accepted=False with
NoMatchingListenerHostname; one that also claims a free hostname stays
Accepted and is served there, as Gateway API accepts a route whose
hostnames intersect at least one listener. dns01 and existingSecret are
unchanged: their wildcard listener still covers such a name, and the
passthrough listener answers it by its exact server name.

The chart writes the passthrough services on the publishing tenant
alone, so the names reserved by default are the ones the platform
publishes there through its own TLSRoutes. The shape an upgrade
changes is an HTTPRoute on such a name whose platform TLSRoute forwards
nowhere, because it never attached (the package owning the service is
not installed) or because none of its backends resolve: that route kept
a terminate listener and now gets none.

The backend resolution, its watches and the RBAC rules that served them
go away.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 06:32:42 +03:00
Aleksei Sviridkin 6144805bea fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)
## What this PR does

The MariaDB chart's my.cnf has not been read by the server since the
mariadb-operator bump to v0.28.1. Up to v0.0.27 the operator mounted the
referenced ConfigMap key as `/etc/mysql/conf.d/my.cnf` whatever the key
was called. Since v0.0.28 the file takes the key's own name, so the
chart's key `config` became `/etc/mysql/conf.d/config`. The server's
`!includedir` reads only `*.cnf` files, so it skipped it, and every
instance has been running on stock MariaDB defaults. On the default
`t1.nano` preset that means a 128M buffer pool inside a 128Mi memory
limit.

This renames the key to `my.cnf` in the ConfigMap and in the MariaDB CR.
The file content is unchanged. I started all four shipped versions
(10.6.24, 10.11.15, 11.4.9, 11.8.5) with the rendered file under a 128m
memory limit, and each one came up with the chart's values. The
MySQL-only options in the file log a warning and are otherwise ignored.
They and the deprecated `innodb_flush_method` are tracked in #4842.

Tests: a helm-unittest suite pins the key on both sides, and the mariadb
chainsaw suite gets a step that compares `@@innodb_buffer_pool_size` on
the running server with the value in the rendered my.cnf.

This changes behaviour on every existing instance, see the release note.
Each instance rolls its pods once because the projected volume in the
pod template changes: replicas first, then the primary with a
switchover. After that the server runs with every setting in the chart's
file. Instances created before the operator bump (May 2024) ran with
these settings before. Newer ones never did.

Closes #4836

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(mariadb): the chart's my.cnf is applied again. Since mariadb-operator v0.28.1 it was mounted under a name the server ignores, so MariaDB instances ran on stock defaults. Every existing instance rolls its pods once on upgrade (replicas first, then the primary with a switchover) and then runs with the chart's settings. That changes behaviour: buffer pool, redo log, connection limits and caches are sized from the instance's resources; sql_mode becomes NO_ENGINE_SUBSTITUTION, so strict mode is off and invalid values are truncated with a warning instead of rejected; innodb_flush_log_at_trx_commit=2 can lose up to about a second of committed transactions if the node crashes; innodb_fast_shutdown=0 makes every pod shutdown slower.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* MariaDB configuration is now stored and read using the `my.cnf` key. A
verification check confirms the running server applies the configured
buffer-pool size.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-10 04:38:28 +03:00
Aleksei Sviridkin 8ab6d4948f test(mariadb): assert the server runs with the chart's my.cnf
The chart's unit tests can pin the ConfigMap key and the CR reference,
but not what mariadb-operator does with them or which files the server
then reads from /etc/mysql/conf.d. Compare innodb_buffer_pool_size as
the live server reports it against the value in the rendered my.cnf,
which differs from the server default on this fixture's preset.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-10 02:47:37 +03:00
Aleksei Sviridkin 957397ed42 ci(retention): derive swept package names from the published tree
GitHub refuses to list org container packages to an app installation
token, so the retention workflow cannot enumerate what to sweep. Name
the packages from the published nightly packages tree instead: it is
the tree the nightly mirror copied images for, so it covers every
component image that receives dated nightly tags, including those
digest-pinned only by the build and therefore absent from the
committed tree. A tree that does not reference the packages artifact
is refused rather than read as a shorter list.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 17:12:14 +03:00
Aleksei Sviridkin da28ea7ee8 ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU
guests, the substrate the merge-gating lanes left behind. On it the
tenant Kubernetes suites have missed their node-join deadline on most
nights since 2026-10-02, each failure spends the whole deadline, and
the job reaches its timeout before the alphabetically later suites
run. Over the same days those suites passed on every PR run that
selected them, where the nodes are Talos containers and a tenant
worker sits one virtualization level shallower.

Both lanes now use the PR lanes' container substrate. Nightly's e2e no
longer waits for the nocloud disk, which it does not boot; the disk is
still built and published. A manual e2e-tag dispatch against a tag
that predates the container lane fails early and names the ref to
dispatch from instead.

DRBD, the replicated StorageClass and the Cozystack Talos node image
lose their last e2e coverage with this. The docs and comments that
pointed at nightly and the rc gate as that coverage now say so.

Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-10-09 16:57:43 +03:00