Commit Graph
67 Commits
Author SHA1 Message Date
Yan Bondarenko 1ca7c0d891 fix(etcd): stop counting canceled watch streams as failed gRPC requests
The tenant etcd copy of etcdHighNumberOfFailedGRPCRequests counted
every code other than OK. A Watch stream is counted once, when it ends,
and the streams of a kube-apiserver that stops end with Canceled. So
whenever an API server on that etcd goes away, the Watch method of
every member reads as 100 percent failed and the critical rule fires
on all of them.

The rules now count the codes the etcd mixin counts since
etcd-io/etcd#13127, the same list the platform copy in
monitoring-agents already uses. The grouping by method stays, so the
alert labels do not change.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-20 12:05:33 +03:00
Yan Bondarenko fe59a6bf91 fix(etcd): leave compaction to etcd and pass the real quota to defrag
The hourly defrag CronJob runs etcd-defrag with its defaults, which compact
to the latest revision before defragmenting. Every cluster already compacts
itself with a periodic five-minute retention, so the second compaction
either fails with "required revision has been compacted" when nothing was
written since, or drops the history that retention keeps for watchers to
resume from.

The same defaults evaluate dbQuotaUsage in the defrag rule against 2 GiB
rather than the quotaBackendBytes the cluster runs with, so the 80 percent
threshold fires at a different database size than intended.

Assisted-by: LLM
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com>
2026-10-11 20:21:26 +03:00
Mattia Eleuteri 7c9ab0e28e fix(cozy-lib): round the sanitized CPU request up to a whole milli-CPU
cozy-lib.resources.sanitize divides the CPU limit by cpuAllocationRatio
and rendered the quotient as a bare float. When the ratio does not
divide the value evenly the result is finer than a milli-CPU, and the
VPA admission webhook rejects it outright:

  admission webhook "vpa.k8s.io" denied the request:
  minAllowed: CPU [62500u] must be a whole number of milli CPUs

packages/extra/etcd routes its VPA minAllowed/maxAllowed through the
helper, so 250m / 4 = 62.5m left the etcd HelmRelease looping
UpgradeFailed -> RollbackFailed forever: the rollback target renders the
same invalid value. Since v1.6.0 HelmReleases wait on each other, so
that one stuck app also blocks tenant-root, cozystack-basics and
kubevirt-cdi-operator, and without CDI the Talos worker rollover cannot
import the worker OS image. Only the divisors of 250 worked as ratios,
which is an artefact of one chart's hardcoded floor rather than a real
constraint on the operator.

The request is rounded up, never down, so it stays at or above the
intended fraction of the limit. It is rounded only when it has to be:
if the milli-CPU value equals its own ceiling, the float keeps its
current rendering, so nothing changes on the default ratio of 10 or on
any ratio that divides evenly; otherwise it renders as the ceiling in
m. The comparison is exact, with no tolerance. sprig's divf and mulf run
on shopspring/decimal rather than float64, so mulf (divf 0.25 10.0)
1000.0 renders exactly 25 and there is no binary-float noise on this
codepath to absorb. A tolerance would reintroduce the bug: a value
whole-milli only to within it, such as 100.0000004m at ratio 1, would
read as exact and render "0.1000000004", rejected for the same reason
62.5m is. The same predicate covers sub-milli input, since ceil carries
any positive request to at least 1m, and a zero request stays "0".

Fixes #2917

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-09-29 17:04:00 +02:00
Alexey Artamonov 6fdb5e0840 docs(etcd): note that affinity is not applied to existing members
The operator applies spec.affinity only when it creates or replaces a member
Pod; it does not reschedule running members. Document that on the `affinity`
value so operators do not expect the default (or a newly set rule) to
redistribute already co-located members until each is replaced.

Addresses the non-blocking review notes on #3962.

Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-09-01 12:47:46 +03:00
Andrey Kolkov 9858fb63ac test(etcd): cover default and overridden affinity rendering
The affinity value added in this branch introduced a two-branch conditional
in templates/etcd-cluster.yaml (default soft podAntiAffinity vs. a supplied
value) with no coverage. Add a helm unittest suite pinning both branches:
the default preferred per-host rule, and that a supplied affinity replaces
the default outright (no preferred rule leaks through).

Signed-off-by: Andrey Kolkov <andrey.kolkov@aenix.io>
2026-08-26 13:26:26 +04:00
Alexey Artamonov c83a22e077 fix(etcd): spread members across nodes with default soft pod anti-affinity
The etcd chart only set a soft topologySpreadConstraints and never rendered
spec.affinity, so the scheduler could place several members of a 3-node
EtcdCluster on the same node; losing that node then took out quorum (observed
in production after a single node reboot).

Render EtcdCluster.spec.affinity from a new `affinity` value. When it is left
empty the chart applies a soft (preferred) podAntiAffinity on
kubernetes.io/hostname, so members are spread on a best-effort basis without
ever leaving a member Pending on small or node-pinned-storage clusters.
Operators can override the value — e.g. a hard requiredDuringScheduling rule —
where node count and storage allow.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-08-25 23:09:40 +03:00
Myasnikov DaniilandClaude Opus 4.8 ca4f99411f fix(etcd): harden v1alpha2 transition — hook runAsUser, digest-pin kubectl, tests
Addresses review on #3265:

- pre-upgrade-selector-fix hook: add runAsUser: 65532. clastix/kubectl's
  image user is the non-numeric name `nonroot`, which the kubelet cannot
  verify against runAsNonRoot: true — so the Pod fails admission and the
  hook never runs, silently blocking the very 1.5->1.6 upgrade it exists to
  unblock. Matches postgres-operator's webhook-ready hook (same image).

- Digest-pin the kubectl image (the comment already claimed digest-pinning
  but shipped a floating v1.32 tag). Reuse the digest postgres-operator
  already vendors, add the renovate annotation, and template repo:tag@digest.

- Tests (both charts have CI helm-unittest suites, none previously covered
  these):
  - etcd-operator/tests/selector-fix-hook_test.yaml: hook wiring, namespaced
    least-privilege RBAC, numeric-non-root securityContext (guards the
    runAsUser regression above), digest-pinned image.
  - etcd-operator/tests/deployment_test.yaml: assert the 256Mi cold-start
    memory floor so it can't silently drop back to an OOMKilling value.
  - extra/etcd/tests/etcd-cluster_test.yaml: assert the transitional
    etcd-headless Service (headless, publishNotReadyAddresses, member
    selector, client/peer ports).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-07-15 19:11:35 +05:00
Andrey Kolkov d51d6914d7 fix(etcd): keep legacy etcd-headless Service alive during v1alpha2 adoption
etcd-migrate adopts legacy clusters IN PLACE, so the Pods keep their original
spec.subdomain: etcd-headless and are dialed at etcd-<i>.etcd-headless.<ns>.svc
until they eventually roll onto the operator's native <member>.etcd.<ns>.svc
domain. The v1alpha2 operator only creates the native `etcd` Service, and the
legacy `etcd-headless` Service is pruned during the transition, so those names
stop resolving (no such host), the operator's MemberList fails, and
status.readyMembers never populates -- the EtcdCluster never goes Ready even
though the etcd processes are healthy and in quorum.

Ship a chart-managed transitional headless `etcd-headless` Service (selector
mirrors the operator's native `etcd` Service via
etcd-operator.cozystack.io/cluster, publishNotReadyAddresses: true). This is the
DNS counterpart of the legacy *.etcd-headless.<ns>.svc wildcard already kept in
the server/peer cert SANs for the same transition window -- the TLS half of the
compat was done, the DNS half was missing. Safe to remove, together with that
SAN, once the members have rolled onto the native `etcd` subdomain.

Verified live: recreating this Service on an adopted 3-node cluster restored
per-pod DNS and the cluster went readyMembers=3 / Available=True.

Refs: #3243

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-15 19:11:35 +05:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Алексей АртамоновandClaude Opus 4.8 9472bb3412 fix(etcd): clamp cleanup PVC resourceNames so replicas:0 cannot widen RBAC
Address CodeRabbit review on #3170: the PVC cleanup Role generates its
resourceNames from .Values.replicas. values.schema.json sets no minimum, so an
explicit replicas: 0 rendered an empty resourceNames list — and an empty
resourceNames in an RBAC rule matches every PVC in the namespace (a wildcard),
silently undoing the least-privilege scoping the previous commit added.

Clamp the count to max(1, replicas) via a shared $replicaCount used by both the
Role resourceNames range and the delete command, so at least data-etcd-0 is
always listed (it exists for any running cluster and the delete uses
--ignore-not-found). Add a regression test asserting replicas: 0 still yields a
name-scoped Role and a single-name delete rather than a namespace-wide grant.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 17:36:03 +03:00
Алексей АртамоновandClaude Opus 4.8 4a06df92e7 fix(etcd): scope cleanup Role to the data-etcd PVC names and assert hook metadata
Address CodeRabbit review on #3170:
- Tighten the cleanup Role's PVC rule from a namespace-wide get/list/delete to
  get/delete by explicit resourceNames (data-etcd-<ordinal>, one per replica),
  mirroring the least-privilege pattern already used for the TLS secrets. The
  Job only ever deletes those deterministic names, so list is unnecessary and a
  regression can no longer reach a sibling workload's PVC.
- Extend the Role and RoleBinding unit tests to assert helm.sh/hook and
  helm.sh/hook-delete-policy (not just the weight), so the RBAC cannot regress
  into normal release resources, and assert the PVC resourceNames.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:58:09 +03:00
Алексей АртамоновandClaude Opus 4.8 a92f060f63 fix(etcd): clean up orphaned data PVCs and TLS secrets on tenant module delete (#3090)
When the etcd tenant module is disabled (etcd: false), the etcd-operator
StatefulSet data PVCs (data-etcd-<ordinal>) are never garbage-collected, and
the five etcd-*-tls secrets are left orphaned in the tenant namespace:
etcd-ca-tls / etcd-client-tls are pre-install hooks this chart marks with
helm.sh/resource-policy: keep, and etcd-peer-ca-tls / etcd-server-tls /
etcd-peer-tls are cert-manager-issued secrets with no owner reference back to
the chart. None are reclaimed by the Helm uninstall.

Add a post-delete cleanup Job (modeled on the seaweedfs/monitoring sibling
fixes) that deletes the data PVCs by name (one per replica) and the five TLS
secrets by name. check-release-name pins the release to "etcd", so every
target name is deterministic and no sibling workload shares the data-etcd-*
prefix. The Job runs non-root with a read-only root fs, dropped capabilities
and seccompProfile RuntimeDefault; its RBAC is scoped to PVC get/list/delete
and the five secrets by resourceNames. Deletes use --ignore-not-found and
failures are logged but never block teardown.

The stuck-HelmRelease finalizer noted in the issue is the platform-wide
sharded-helm-controller side-bug tracked separately; this change addresses the
PVC and secret orphans the etcd chart owns.

Assisted-By: Claude <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-02 15:58:08 +03:00
Aleksei Sviridkin 26b3110c59 fix(etcd): pin etcd image to v3.5.31 for consistent watches
etcd-operator v0.4.5 hardcodes quay.io/coreos/etcd:v3.5.12 and the
v1alpha1 EtcdCluster CRD exposes no version field, so the tenant
control-plane etcd runs 3.5.12. kube-apiserver only enables the
RequestWatchProgress storage feature on etcd >= 3.5.13 / >= 3.4.31, and
Kubernetes >= 1.31 enables ConsistentListFromCache (locked-to-true in
1.35), which depends on it. On etcd 3.5.12 every consistent watch fails
inside the stream with HTTP 500 ("the required storage feature
RequestWatchProgress is disabled"), breaking watch-based clients of
aggregated APIs in modern tenants.

Override the etcd container image through spec.podTemplate.spec, which
the operator strategically merges over its generated pod spec (matched
by container name), pinning a >= 3.5.13 patch release. This restores
consistent watches without waiting on the etcd-operator v1alpha2
migration.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-28 03:55:36 +03:00
Myasnikov DaniilandClaude Opus 4.8 6e9ff908ed fix(etcd): set VPA updateMode to Initial to stop member-evicting churn
Auto mode evicts running etcd members to apply VPA recommendations, churning the StatefulSet (disruptive to quorum and a Cilium IP-reuse-leak trigger under load). Initial sizes at pod creation without evicting; the existing 250m/256Mi minAllowed floor covers the no-runtime-resize tradeoff. Same eviction-churn class as the monitoring VPA fix.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-18 14:08:48 +05:00
Andrey Kolkov d80f67e799 feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-22 14:35:08 +04:00
Andrei Kvapil b5a1ae57de feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428)
## What this PR does

Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.

Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.

On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).

Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.

`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.

Depends on #2427.

### Scope deferred to follow-ups

- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.

### Release note

```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.

* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.

* **Documentation**
  * Backup configuration guide and parameter reference added.

* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-05-01 13:16:08 +02:00
Aleksei Sviridkin 040d5d4681 docs(etcd): use "custom resource" instead of "CRD" for EtcdBackup
Address review feedback from coderabbitai on
packages/extra/etcd/README.md:9: CRDs are schema definitions; what users
apply to trigger a restore or one-shot backup is a custom resource
instance of the EtcdBackup kind, not the CRD itself. Reword accordingly.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-24 13:43:05 +03:00
Aleksei Sviridkin aaf9dd047c fix(etcd): cast backup history limits to int in template
Address review feedback from gemini-code-assist on
packages/extra/etcd/templates/etcd-backup-schedule.yaml:20-21: explicitly
coerce successfulJobsHistoryLimit and failedJobsHistoryLimit to int so
string values passed via helm --set do not produce a type mismatch in
the EtcdBackupSchedule spec.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-24 13:42:17 +03:00
Myasnikov Daniil c1508940bd fix(etcd): remove destructive post-upgrade cert-regeneration hook
The etcd chart shipped a `post-upgrade` Helm hook that `kubectl delete`d the
etcd TLS chain (`etcd-{ca,peer-ca,client,peer,server}-tls`) and then deleted
etcd pods on every chart upgrade, gated by a semver compare of an
`etcd-deployed-version` ConfigMap against `2.6.1`.

The hook was added as a one-shot migration for the chart `2.6.0 -> 2.6.1`
transition. Since commit f871fbdb ("Remove versions_map logic") all chart
versions are stamped as `0.0.0+<git-hash>`, which per semver is always
`< 2.6.1`. The gate therefore always resolves to "update certs", firing the
destructive hook on every etcd upgrade. On clusters running Kamaji-managed
tenant control planes this wipes the etcd CA, cert-manager re-issues it, and
tenant kube-apiservers hit `x509: certificate signed by unknown authority`
against `etcd.<ns>.svc:2379` until each tenant DataStore is manually
re-reconciled.

Commit 47d81f70 ("Disabled private key rotation in CA certs") already fixed
the underlying `rotationPolicy: Always` issue the migration was papering
over, so the hook has no remaining use. Remove the hook Job, its RBAC, the
version ConfigMap it read, and add a helm-unittest suite under
`packages/extra/etcd/tests/` that guards against re-introducing the hook
or the version ConfigMap.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-23 13:34:51 +05:00
Aleksei Sviridkin 9fc99eed50 chore(etcd): regenerate values.schema.json and cozyrd manifest
Run 'make generate' under packages/extra/etcd to refresh the JSON schema
and the corresponding ResourceDefinition YAML after adding backup and
resources fields. Matches cozyvalues-gen v1.3.0 output used by the
pre-commit CI check.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-21 02:30:27 +03:00
Aleksei Sviridkin ed77e9c6e3 fix(etcd): add helm unit tests, simplify backup guards, harden e2e
Fourth review round fixes.

Simplify backup validation by dropping placeholder-regex guards and switching default values of s3AccessKey, s3SecretKey, destinationPath, and endpointURL from angle-bracketed hints to empty strings. A single 'required' check on each field is now the whole validation for presence; the regex guard was brittle (silently stopped matching if the hint ever changed) and dropping it makes the values.yaml defaults match the behavior contract the README describes. s3:// prefix and '//' rejection still live in etcd-backup-schedule.yaml as template-time fail branches.

Add helm-unittest suites in packages/extra/etcd/tests/ covering both templates end-to-end: rendered output assertions for the happy path (bucket/key split, region passthrough, conditional rendering of optional fields) and failedTemplate assertions for every fail branch. A 'test' target is wired into the package Makefile so hack/helm-unit-tests.sh picks it up on 'make unit-tests'.

Add type: Opaque to backup-secret.yaml for consistency with the mongodb template.

E2E: wait for HelmRelease, Secret, and EtcdBackupSchedule deletion in setup() so the empty-backup case does not flake against lingering state from the previous test. Replace the fictitious minio-e2e hostname with no-such-endpoint.invalid and explicitly document that the e2e verifies schedule/CronJob materialization, not backup execution.

Restore via EtcdCluster.spec.bootstrap and the one-shot EtcdBackup CRD are still not exposed through this chart; both are explicit follow-ups called out in the README.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 17:01:58 +03:00
Aleksei Sviridkin 12fc9cd415 fix(etcd): address third review round
- Use name='etcd' in all three e2e cases. The chart pins the Helm
  release name to 'etcd' via check-release-name.yaml, so any other
  name causes template rendering to fail before the HelmRelease
  can reconcile. This is why the tenant chart hardcodes HR name
  'etcd' in packages/apps/tenant/templates/etcd.yaml.

- Consolidate all backup validation guards into
  etcd-backup-schedule.yaml so Helm's alphabetical render order no
  longer determines which failure a user sees first. Previously, a
  bad destinationPath would be masked by the earlier placeholder
  check in backup-secret.yaml.

- Use regex match '^<.*>$' for placeholder detection instead of
  exact string equality. The previous check silently stopped
  firing if anyone updated the default placeholder string in
  values.yaml; the regex form catches any angle-bracketed default.

- Reject destinationPath strings containing '//' after the s3://
  prefix. Previously 's3://bucket//subdir' silently produced a key
  beginning with '/', confusing S3 clients and retention policies.

- Document the backup contract in README.md: what enabling it
  produces, which fields are required, why placeholders are
  rejected, and that restore is a known follow-up gap.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 16:37:26 +03:00
Aleksei Sviridkin 561bc4dc23 fix(etcd): address second review round
- Mark backup.enabled as optional in the values typedef so
  generated JSON schema no longer lists it in 'required'.
  Previously, 'backup: {}' (relying on the enabled:false default)
  failed schema validation at CRD admission.

- Validate backup.destinationPath has s3:// prefix. Without this
  check, passing a path without the scheme silently used the
  first path segment as the bucket name.

- Add tail newline to values.schema.json to match repo convention
  (cozyvalues-gen output lacks it — worth fixing in that project
  separately).

- Add bats setup() that clears tenant-test etcd resources before
  each test so a failure mid-test does not leak into subsequent
  tests. Hardcoded chart-level resource names ('etcd',
  'etcd-s3-creds') make two Etcd apps in the same namespace
  collide; setup() enforces a single owner per test.

- Add dump_diagnostics() helper that dumps operator logs + the
  relevant CRD state on wait-for-ready timeouts, so CI failures
  surface actionable context instead of bare exit code 124.

- Add a third e2e case 'Create Etcd with empty backup block' that
  asserts backup:{} admits and produces no backup resources — a
  regression test for the required:['enabled'] bug fixed here.

Known scope: the upstream EtcdCluster.spec.bootstrap restore field
(also new in v0.4.3) is not exposed through this values.yaml yet.
Restoring a cluster from an existing snapshot is a follow-up PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 16:15:56 +03:00
Aleksei Sviridkin 0e03ed9ad4 fix(etcd): reject placeholder backup values and harden e2e
Review found three issues:

1. The guard in backup-secret.yaml only triggered on empty strings,
   not on the placeholder defaults ('<your-access-key>' etc). A user
   enabling backup.enabled=true without overriding the values would
   produce a Secret containing the literal placeholder, and the
   backup CronJob would fail at runtime with an opaque 403 far from
   the source. Add explicit template guards that fail rendering when
   any of the four required fields still hold their placeholder
   default.

2. The backup e2e test waited on .spec.schedule of the newly-created
   EtcdBackupSchedule, which is set at apply time — the wait was
   tautological and did not prove the etcd-operator reconciled
   anything. Replace it with a wait on the EtcdCluster's Ready
   condition plus a poll for the CronJob the operator generates
   from the schedule (selected via the
   etcd.aenix.io/etcdbackupschedule-name label).

3. The backup e2e test also did not wait for EtcdCluster readiness
   before tearing down. Add the same Ready wait as the baseline
   test, and verify the Secret contains the real access key value
   rather than just existing.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 16:08:43 +03:00
Aleksei Sviridkin ee8ceecc0e test(etcd): add e2e for etcd application and backup schedule
Adds two bats cases: a baseline Etcd managed-app deployment and a
deployment with backup.enabled=true that verifies the
EtcdBackupSchedule resource and the S3 credentials Secret are
materialized. The backup case uses fake S3 credentials — the
operator will not successfully push backups, but the schedule
resource itself is reconciled, which is what this test validates.

Also hardcodes the EtcdBackupSchedule and backup Secret names to
'etcd' / 'etcd-s3-creds' to match the existing hardcoded
EtcdCluster name in the chart (one etcd per tenant namespace).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 15:58:55 +03:00
Aleksei Sviridkin 51c8192e1e feat(etcd): add S3 backup schedule support
Expose the EtcdBackupSchedule CRD (etcd.aenix.io/v1alpha1) through
optional values so operators can declaratively configure periodic
S3 backups of the platform etcd cluster. Kamaji stores tenant
control-plane etcd data here, so losing this cluster means losing
every tenant cluster — a backup schedule is critical.

The backup is disabled by default (backup.enabled: false), keeping
existing installations unchanged. When enabled, a Secret is created
with AWS_ACCESS_KEY_ID / AWS_SECRET_ACCESS_KEY keys referenced by
the new EtcdBackupSchedule resource. Field naming mirrors the
existing postgres backup pattern (destinationPath, endpointURL,
s3AccessKey, s3SecretKey, schedule) for consistency.

Requires etcd-operator v0.4.3+ (shipped in the previous commit).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-04-20 15:57:51 +03:00
Myasnikov Daniil 9e55552910 [docs] Updated app go types
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-03-25 15:57:25 +05:00
Kirill Ilin ed8ba3beec fix(etcd): add protective limits to defrag CronJob
Without concurrencyPolicy and job limits, the defrag CronJob can
accumulate hundreds of running/failed pods during cluster upgrades
when etcd is temporarily unavailable. This was observed after
upgrading to v1.1.2 where defrag jobs piled up across tenants.

Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-03-17 13:21:28 +05:00
Myasnikov Daniil 47d81f70d7 Disabled private key rotation in CA certs
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-02-27 19:09:10 +05:00
Andrei KvapilandClaude c58c959df6 fix(etcd): increase probe thresholds for better recovery
Increase startup probe failureThreshold to 300 (25 minutes) to allow
etcd members more time to sync with the cluster after restart or
recovery. This prevents pods from being killed during initial
synchronization when VPA assigns minimal resources.

Also increase liveness probe failureThreshold to 10 to reduce
unnecessary restarts during temporary network issues.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-18 21:50:06 +01:00
Andrei KvapilandClaude 3618abed62 refactor: move scripts to hack directory
Move common-envs.mk and package.mk from scripts/ to hack/ directory.
Update all Makefile includes to use new paths. Remove unused
issue-flux-certificates.sh script.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 16:06:56 +01:00
Andrei KvapilandClaude 2e61810547 refactor: replace Helm lookup with valuesFrom mechanism
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.

Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
  containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
  (valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format

This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-05 16:10:55 +01:00
Matthieu ROBIN 675eaa6178 Add resource requests and limits to etcd-defrag
Added resource requests and limits for etcd-defrag container.

Signed-off-by: Matthieu ROBIN <info@matthieurobin.com>
2026-01-02 15:53:54 +01:00
Andrei Kvapil 9632772337 [dx] JSDoc compatible syntax for values.yaml
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-10-29 08:57:26 +05:00
Timofei Larkin 42c9d65c7c [etcd] Add VPA for etcd
The etcd tenant module deploys by default with a large resource
limit/request and these values are not exposed at deploy time. This
patch lowers the default resources and adds a VPA to autoconfigure them
according to the real needs.

```release-note
[etcd] Attach VPA to etcd and lower initial default resource requests.
```

Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2025-10-06 21:57:33 +03:00
Andrei Kvapilandkklinch0 0afc3c1e86 [cozystack-api] Implement TenantNamespace, TenantModules, TenantSecret and TenantSecretsTable resources
[cozystack-controller] Introduce new dashboard-controller
[dashboard] Introduce new dashboard based on openapi-ui

Co-authored-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: kklinch0 <kklinch0@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-24 18:27:54 +02:00
Andrei Kvapil f871fbdb1e Remove versions_map logic
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-24 12:32:37 +02:00
Andrei Kvapil 8c6fc68367 [cozystack-api] Update defaulting API schemas
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-18 04:49:50 +02:00
Andrei Kvapil 601f6bd3c9 [etcd] Fix Global TopologySpreadConstarints
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-10 15:10:47 +02:00
Andrei Kvapil ce21299280 Get rid of bitnami images
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-09-01 13:05:56 +02:00
Nick Volynkin e3a4e284de [apps,extra] Update all app versions after updating OpenAPI schemas
Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-08-18 15:55:01 +03:00
Andrei Kvapil c0f742595f [apps] Use cozyvalues-gen with packages/apps/* (#1321)
- clickhouse
- ferretdb
- http-cache
- kafka
- kubernetes
- mysql
- nats
- rabbitmq
- redis
- tcp-balancer
- vm-disk
- vm-instance
- vpn

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>

<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does


### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
[]
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Many apps now present structured, typed configuration blocks (backup,
keeper/quorum, JetStream, HAProxy/Nginx/httpAndHttps, nodeGroups/addons,
topics, users, etc.) for clearer per-field types and presets.

* **Documentation**
* README, values.yaml and JSON schemas updated with Type columns, enums,
validation, nested fields and examples; some backup schema requirements
relaxed.

* **Chores**
* Generation consolidated to cozyvalues-gen (v0.8.5); CI matrix set to
not fail-fast; e2e VM data disk size increased.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-08-11 13:38:02 +03:00
Nick Volynkin f864b40a85 [apps] Use new OpenAPI schema and README generator for packages/apps
- clickhouse
- ferretdb
- http-cache
- kafka
- kubernetes
- mysql
- nats
- rabbitmq
- redis
- tcp-balancer
- vm-disk
- vm-instance
- vpn

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-08-11 12:59:50 +03:00
kklinch0 92f206cb93 fix etcd topologySpreadConstraints
Signed-off-by: kklinch0 <kklinch0@gmail.com>
2025-08-11 11:29:44 +03:00
Andrei Kvapil 5359c6d991 Update cozyvalues-gen
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-08-08 00:26:51 +02:00
Nick VolynkinandTimofei Larkin d48b5cfa2f [apps] Use new OpenAPI schema and README generator for packages/extra
Co-authored-by: Timofei Larkin <lllamnyp@gmail.com>

Signed-off-by: Nick Volynkin <nick.volynkin@gmail.com>
2025-08-07 14:05:50 +03:00
Andrei Kvapil 72e7b5e0b5 Get rid of bitnami's readme-generator
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-18 00:40:31 +02:00
Andrei Kvapil a5ff9bf65b [etcd] Update etcd application (fix resources and headless services)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-07-02 06:15:38 +03:00
Andrei Kvapil 3b8a9f9d2c Configure all apps to use new function to generate subjects
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-06-16 20:32:11 +02:00
kklinch0 85ec09b8de bugfix/add-limits-for-extra-packages
Signed-off-by: kklinch0 <kklinch0@gmail.com>
2025-03-20 14:59:47 +03:00