The hourly defrag CronJob runs etcd-defrag with its defaults, which compact
to the latest revision before defragmenting. Every cluster already compacts
itself with a periodic five-minute retention, so the second compaction
either fails with "required revision has been compacted" when nothing was
written since, or drops the history that retention keeps for watchers to
resume from.
The same defaults evaluate dbQuotaUsage in the defrag rule against 2 GiB
rather than the quotaBackendBytes the cluster runs with, so the 80 percent
threshold fires at a different database size than intended.
Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:
admission webhook "vpa.k8s.io" denied the request:
minAllowed: CPU [62500u] must be a whole number of milli CPUs
packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.
The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".
Fixes#2917
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
The affinity value added in this branch introduced a two-branch conditional
in templates/etcd-cluster.yaml (default soft podAntiAffinity vs. a supplied
value) with no coverage. Add a helm unittest suite pinning both branches:
the default preferred per-host rule, and that a supplied affinity replaces
the default outright (no preferred rule leaks through).
Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
Addresses review on #3265:
- pre-upgrade-selector-fix hook: add runAsUser: 65532. clastix/kubectl's
image user is the non-numeric name `nonroot`, which the kubelet cannot
verify against runAsNonRoot: true — so the Pod fails admission and the
hook never runs, silently blocking the very 1.5->1.6 upgrade it exists to
unblock. Matches postgres-operator's webhook-ready hook (same image).
- Digest-pin the kubectl image (the comment already claimed digest-pinning
but shipped a floating v1.32 tag). Reuse the digest postgres-operator
already vendors, add the renovate annotation, and template repo:tag@digest.
- Tests (both charts have CI helm-unittest suites, none previously covered
these):
- etcd-operator/tests/selector-fix-hook_test.yaml: hook wiring, namespaced
least-privilege RBAC, numeric-non-root securityContext (guards the
runAsUser regression above), digest-pinned image.
- etcd-operator/tests/deployment_test.yaml: assert the 256Mi cold-start
memory floor so it can't silently drop back to an OOMKilling value.
- extra/etcd/tests/etcd-cluster_test.yaml: assert the transitional
etcd-headless Service (headless, publishNotReadyAddresses, member
selector, client/peer ports).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Address CodeRabbit review on #3170: the PVC cleanup Role generates its
resourceNames from .Values.replicas. values.schema.json sets no minimum, so an
explicit replicas: 0 rendered an empty resourceNames list — and an empty
resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard),
silently undoing the least-privilege scoping the previous commit added.
Clamp the count to max(1, replicas) via a shared $replicaCount used by both the
Role resourceNames range and the delete command, so at least data-etcd-0 is
always listed (it exists for any running cluster and the delete uses
--ignore-not-found). Add a regression test asserting replicas: 0 still yields a
name-scoped Role and a single-name delete rather than a namespace-wide grant.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
Address CodeRabbit review on #3170:
- Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to
get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica),
mirroring the least-privilege pattern already used for the TLS secrets. The
Job only ever deletes those deterministic names, so list is unnecessary and a
regression can no longer reach a sibling workload's PVC.
- Extend the Role and RoleBinding unit tests to assert helm.sh/hook and
helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress
into normal release resources, and assert the PVC resourceNames.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
When the etcd tenant module is disabled (etcd: false), the etcd-operator
StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and
the five etcd-*-tls secrets are left orphaned in the tenant namespace:
etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with
helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls /
etcd-peer-tls are cert-manager-issued secrets with no owner reference back to
the chart. None are reclaimed by the Helm uninstall.
Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling
fixes) that deletes the data PVCs by name (one per replica) and the five TLS
secrets by name. check-release-name pins the release to "etcd", so every
target name is deterministic and no sibling workload shares the data-etcd-*
prefix. The Job runs non-root with a read-only root fs, dropped capabilities
and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete
and the five secrets by resourceNames. Deletes use --ignore-not-found and
failures are logged but never block teardown.
The stuck-HelmRelease finalizer noted in the issue is the platform-wide
sharded-helm-controller side-bug tracked separately; this change addresses the
PVC and secret orphans the etcd chart owns.
Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
etcd-operator v0.4.5 hardcodes quay.io/coreos/etcd:v3.5.12 and the
v1alpha1 EtcdCluster CRD exposes no version field, so the tenant
control-plane etcd runs 3.5.12. kube-apiserver only enables the
RequestWatchProgress storage feature on etcd >= 3.5.13 / >= 3.4.31, and
Kubernetes >= 1.31 enables ConsistentListFromCache (locked-to-true in
1.35), which depends on it. On etcd 3.5.12 every consistent watch fails
inside the stream with HTTP 500 ("the required storage feature
RequestWatchProgress is disabled"), breaking watch-based clients of
aggregated APIs in modern tenants.
Override the etcd container image through spec.podTemplate.spec, which
the operator strategically merges over its generated pod spec (matched
by container name), pinning a >= 3.5.13 patch release. This restores
consistent watches without waiting on the etcd-operator v1alpha2
migration.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.
Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.
On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).
Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.
`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.
Depends on #2427.
### Scope deferred to follow-ups
- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.
### Release note
```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.
* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.
* **Documentation**
* Backup configuration guide and parameter reference added.
* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The etcd chart shipped a `post-upgrade` Helm hook that `kubectl delete`d the
etcd TLS chain (`etcd-{ca,peer-ca,client,peer,server}-tls`) and then deleted
etcd pods on every chart upgrade, gated by a semver compare of an
`etcd-deployed-version` ConfigMap against `2.6.1`.
The hook was added as a one-shot migration for the chart `2.6.0 -> 2.6.1`
transition. Since commit f871fbdb ("Remove versions_map logic") all chart
versions are stamped as `0.0.0+<git-hash>`, which per semver is always
`< 2.6.1`. The gate therefore always resolves to "update certs", firing the
destructive hook on every etcd upgrade. On clusters running Kamaji-managed
tenant control planes this wipes the etcd CA, cert-manager re-issues it, and
tenant kube-apiservers hit `x509: certificate signed by unknown authority`
against `etcd.<ns>.svc:2379` until each tenant DataStore is manually
re-reconciled.
Commit 47d81f70 ("Disabled private key rotation in CA certs") already fixed
the underlying `rotationPolicy: Always` issue the migration was papering
over, so the hook has no remaining use. Remove the hook Job, its RBAC, the
version ConfigMap it read, and add a helm-unittest suite under
`packages/extra/etcd/tests/` that guards against re-introducing the hook
or the version ConfigMap.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Fourth review round fixes.
Simplify backup validation by dropping placeholder-regex guards and switching default values of s3AccessKey, s3SecretKey, destinationPath, and endpointURL from angle-bracketed hints to empty strings. A single 'required' check on each field is now the whole validation for presence; the regex guard was brittle (silently stopped matching if the hint ever changed) and dropping it makes the values.yaml defaults match the behavior contract the README describes. s3:// prefix and '//' rejection still live in etcd-backup-schedule.yaml as template-time fail branches.
Add helm-unittest suites in packages/extra/etcd/tests/ covering both templates end-to-end: rendered output assertions for the happy path (bucket/key split, region passthrough, conditional rendering of optional fields) and failedTemplate assertions for every fail branch. A 'test' target is wired into the package Makefile so hack/helm-unit-tests.sh picks it up on 'make unit-tests'.
Add type: Opaque to backup-secret.yaml for consistency with the mongodb template.
E2E: wait for HelmRelease, Secret, and EtcdBackupSchedule deletion in setup() so the empty-backup case does not flake against lingering state from the previous test. Replace the fictitious minio-e2e hostname with no-such-endpoint.invalid and explicitly document that the e2e verifies schedule/CronJob materialization, not backup execution.
Restore via EtcdCluster.spec.bootstrap and the one-shot EtcdBackup CRD are still not exposed through this chart; both are explicit follow-ups called out in the README.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>