Commit Graph
682 Commits
Author SHA1 Message Date
Andrey Kolkov d0740ed8c7 test(etcd): gate S3 backup round-trip behind ETCD_E2E_S3_ROUNDTRIP
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:51:48 +04:00
Andrey Kolkov 011d6d1639 fix FUs
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov f2866edcf0 review fixes
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov 087a8c4be9 chore(etcd-operator): bump to v0.5.2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
Andrey Kolkov cc5d688e15 feat(platform): migrate to etcd-operator v1alpha2
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-07-06 14:41:33 +04:00
IvanHunters a1c779854b fix(vertical-pod-autoscaler): migrate admission-controller TLS to cert-manager to stop reconcile-driven ReplicaSet churn (#3178)
## Summary

The VPA admission-controller webhook TLS Secret is currently rendered by
an in-chart `genCA` + `genSignedCert` template. Those functions are not
idempotent across `helm template` invocations — every Flux HelmRelease
reconcile produces a fresh CA + cert. A `checksum/tls-secret`
podAnnotation on the admission-controller Deployment then flips its
sha, materialising a new ReplicaSet on every reconcile.

Under load — the E2E install phase where cert-manager / CDI / LINSTOR
cascade races repeatedly refire the VPA HelmRelease — this produces
10+ admission-controller ReplicaSets in minutes and blocks the whole
cozystack-platform HR: the admission-controller pod livelocks in
startup (waits on its own election lease which its previous
ReplicaSet is still holding, hits livenessProbe timeout,
CrashLoopBackOffs), VPA never becomes Ready, PackageSources stay
InProgress, and cozystack-platform aborts at its 15-minute install
timeout.

Fix: replace the source of the TLS Secret with a namespace-scoped
cert-manager `Issuer` + `Certificate`. cert-manager only rewrites the
Secret when the certificate is close to expiry (`renewBefore`), so
reconcile-driven cert churn stops. Pod rotation is delegated to a
runtime-lookup-based `checksum/tls-cert` podAnnotation that only
changes when the Secret's `tls.crt` content actually changes — not on
every render, not on every reconcile.

## What

-
`charts/vertical-pod-autoscaler/templates/admission-controller/certificate.yaml`
  (new): namespace-scoped selfSigned Issuer + Certificate. The
  Certificate mirrors the legacy `genSignedCert` shape (single DNS SAN
  `vpa-webhook.<ns>.svc`, 365d validity, matching Secret name), so
  cert-manager on the first upgrade adopts the same Secret name the
  legacy path wrote — no rename cascade in the Deployment volume mount.
  Rendered only when `admissionController.tls.certManagerEnabled=true`
  AND no `existingSecret` is supplied.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/tls-secret.yaml`
  (modified): wrapped in an extra `certManagerEnabled=false` guard so
  the legacy genCA path remains available as an explicit opt-out.
-
`charts/vertical-pod-autoscaler/templates/admission-controller/deployment.yaml`
  (modified): the legacy `checksum/tls-secret` annotation is gated
  behind `certManagerEnabled=false`. Under the cert-manager path the
  Deployment carries a `checksum/tls-cert` annotation computed from
  `lookup`-ing the runtime Secret's `tls.crt` — see the template
  comment for the full timing analysis and the day-350 renewal
  trade-off it documents.
- `charts/vertical-pod-autoscaler/values.yaml` (modified): new
  `admissionController.tls.certManagerEnabled` value defaulting to
  `true`. Values-block header documents the switch as effectively
  one-way (cert-manager writes `type: kubernetes.io/tls`;
  `Secret.type` is immutable, so falling back requires `kubectl
  delete` first).
- `patches/cert-manager-admission-tls.diff` (new) +
  `Makefile update` extended: the four subchart edits above live as a
  single reviewable diff that Makefile re-applies over the freshly
  vendored chart on every `make update`, matching the fluxcd-operator
  package convention. Prevents a bump from silently reverting the
  fix.
- `packages/core/platform/sources/vertical-pod-autoscaler.yaml`
  (modified): VPA PackageSource now depends on `cozystack.cert-manager`.
  Non-circular: cert-manager depends on `cozystack.networking` +
  `cozystack.gateway-api-crds`, neither reaches VPA.
- `tests/vpa_test.yaml` (modified): +11 helm-unittest cases (15 total,
  4 pre-existing) covering `certManagerEnabled={true,false}`,
  `existingSecret` bypass, disabled admission-controller, and
  the offline-render behaviour of the `checksum/tls-cert` lookup.

## Why (evidence)

Snapshot from CI run 28587656430 (this branch, pre-workaround-review):

```
admission-controller ReplicaSets after first-install churn window:
  vertical-pod-autoscaler-admission-controller-55d888d6d4
  vertical-pod-autoscaler-admission-controller-56f8d4c747
  vertical-pod-autoscaler-admission-controller-5775dbc9b7
  vertical-pod-autoscaler-admission-controller-585b7c56c9
  vertical-pod-autoscaler-admission-controller-6fc44944fb
  vertical-pod-autoscaler-admission-controller-74bccf4849
  vertical-pod-autoscaler-admission-controller-84df98f98b
  vertical-pod-autoscaler-admission-controller-8666dffb85
  vertical-pod-autoscaler-admission-controller-86b469457c
  vertical-pod-autoscaler-admission-controller-9b979f9d6
  → 10 ReplicaSets, one per Flux reconcile, each with a different
    genCA-rooted TLS Secret sha in checksum/tls-secret.
```

## Migration for existing clusters

On upgrade the legacy Opaque Secret disappears from the release
manifest, Helm garbage-collects it, and cert-manager immediately
writes a fresh `kubernetes.io/tls` Secret of the same name via the
new Certificate. `Secret.type` is immutable, so in-place adoption is
impossible — the delete/create pass is the correct path, and the
seconds-long window it produces is safe: the `vpa-webhook-config`
MutatingWebhookConfiguration ships with `failurePolicy: Ignore`
(verified on cozy-dev3), so Pod admissions in that window skip VPA
mutation but never get rejected — no cluster-wide disruption.

The Deployment rolls in the same upgrade because the new
`checksum/tls-cert` annotation differs from anything the pod had
before, so the admission-controller re-registers `vpa-webhook-config`
with the fresh caBundle on the same pass.

## Test plan

- [x] `helm unittest packages/system/vertical-pod-autoscaler` — 15/15
      pass.
- [x] `helm template` diff between main and this branch is scoped
      exactly to the cert-manager path (new Certificate + Issuer,
      new `checksum/tls-cert` annotation, guarded legacy Secret, new
      values key, patch file + Makefile step, PackageSource
      dependsOn). No unrelated lines change.
- [x] `make update`-style simulation (rm -rf charts && helm pull &&
      patch) produces a byte-identical subchart to what git tracks —
      the patch stays clean across future upstream bumps until it
      genuinely conflicts.
- [ ] Live migration on cozy-dev3: `helm upgrade` VPA release from
      pre-fix version → observe cert-manager write the new tls Secret
      → verify `checksum/tls-cert` on the new pod's podTemplate →
      verify MutatingWebhookConfiguration `vpa-webhook-config`
      caBundle updated in place → confirm no ReplicaSet churn on
      subsequent Flux reconciles. Blocking the draft's removal.
- [ ] Full E2E on this branch — verify only 1 admission-controller
      ReplicaSet after install, VPA HR reaches Ready inside
      `cozystack-platform`'s 15m install timeout, and
      `kubernetes-latest.bats` / `kubernetes-previous.bats` no longer
      time out on VPA-attributable resource churn.

## Review pass — kvaps' CHANGES_REQUESTED addressed

- ✅ Vendored chart mods now carried as `patches/*.diff` re-applied by
  `Makefile update` (matches fluxcd-operator convention).
- ✅ Retired the tls-migrator hook — kvaps correctly showed it was
  a no-op: Helm's delete decision is driven by the stored release
  manifest not by live labels (`ResourcePolicyAnno` is the only opt-
  out), and `Secret.type` is immutable so cert-manager could not have
  adopted the legacy Opaque Secret in place anyway. ~200 lines
  removed.
- ✅ PR description rewritten to match the actual diff (no more
  reference to the removed `tls-rollout.yaml` post-upgrade hook, real
  test count, dev3 verification honestly listed as pending).
- ✅ `checksum/tls-cert` timing comment corrected: `lookup` runs on
  Helm actions only, day-350 renewal trade-off spelled out.
- ✅ `certManagerEnabled=false` documented as effectively one-way
  (Secret.type immutable).
- ✅ Dropped the `secretTemplate` labels — `app.kubernetes.io/managed-
  by: Helm` on a cert-manager-owned Secret was misleading.
- ✅ Banner comment indentation restored in `vpa_test.yaml`.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* VPA admission-controller webhook TLS now uses cert-manager by default,
with options to disable cert-manager or supply an existing TLS secret.
* Webhook pod restarts are now tied to cert-manager Secret contents via
a content-based checksum.
* **Bug Fixes**
* Improved webhook TLS provisioning reliability when TLS readiness is
delayed.
* Corrected conditional rendering/rollout behavior to avoid mismatched
TLS ownership.
* **Tests**
* Added rendering-invariant tests covering cert-manager, legacy TLS,
existing-secret bypass, and admission-controller disabled behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-07-04 22:48:00 +03:00
IvanHunters 59964043e3 fix(platform): migration 49 backfills tenant ancestor labels
The tenant chart fix makes new and re-reconciled tenant namespaces
self-compute their full tenant.cozystack.io/<ancestor> chain, but existing
namespaces whose HelmRelease does not promptly re-reconcile (suspended or
failed releases) keep the incomplete labels — and with them the broken
<ancestor>-egress reachability that surfaces as upstream timeouts to nested
tenants. Backfill the labels directly so clusters heal on upgrade regardless
of per-tenant HelmRelease state.

Bump targetVersion to 50 so run-migrations.sh iterates the new script
(seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort
and idempotent: kubectl label --overwrite adds only the listed labels, and
per-namespace failures are tolerated so a transient apiserver error cannot
abort the platform upgrade.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-07-03 09:54:12 +03:00
Ivan Okhotnikov 9dc64b9278 feat(platform): VPA depends on cert-manager after admission-controller TLS migration
The VPA admission-controller webhook TLS is now provisioned via a
cert-manager Certificate + Issuer instead of the in-chart genCA
Secret. On a fresh install the VPA package must not reconcile until
cert-manager is Ready; otherwise the Certificate resource sits
Pending forever, the admission-controller Secret is never written,
and the pod CrashLoops on missing TLS mount — cascading up through
the cozystack-platform PackageSource in exactly the way the old
genCA path did, only for a different reason.

Add cozystack.cert-manager to the variant's dependsOn list. Verified
non-circular against packages/core/platform/sources/cert-manager.yaml
(cert-manager depends on cozystack.networking + cozystack.gateway-api-crds,
neither of which reaches VPA).

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-07-02 18:57:01 +03:00
Алексей Артамонов a2ae1621af fix(clickhouse): surface cleanup hook failures and backfill legacy keeper PVC labels
Address review on #3072:

- Cleanup hook no longer masks kubectl delete failures: the command now
  fails (exit 1) when PVC deletion fails instead of swallowing the error
  with || echo. The hook-delete-policy drops hook-succeeded (keeps only
  before-hook-creation) so a failed cleanup Job survives for inspection.
- Add migration 45 (45 -> 46) that backfills app.kubernetes.io/instance
  onto keeper PVCs created before this change; the altinity keeper operator
  never re-stamps volumeClaimTemplate labels onto existing PVCs, so without
  this the cleanup selector misses pre-upgrade keeper PVCs and #3057
  persists for existing releases. Bump migrations.targetVersion to 46.
- Unify helm.sh/hook-weight key quoting across the four hook manifests.
- Extend the unittest to assert the new hook-delete-policy and exit-on-failure.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
2026-07-01 18:31:05 +03:00
Aleksei Sviridkin 41ff94eca9 fix(flux): tune source-watcher concurrency and CPU under load
source-watcher reconciles one ArtifactGenerator per PackageSource and
builds the resulting ExternalArtifacts. On the management cluster that is
~100 artifact builds during install, and the platform HelmRelease blocks
until all of them are Ready.

It was the only Flux controller left untuned. In the bootstrap all-in-one
pod it ran without an explicit --concurrent (binary default 10) and with
the shared 100m CPU request; in the production FluxInstance it was missing
from the --concurrent / CPU patch that source-, kustomize- and
helm-controller already receive. Under load its per-artifact builds finish
at staggered times, leaving a random tail of not-yet-Ready artifacts that
times out the platform install.

Raise source-watcher to --concurrent=20 in both deployments. In the
bootstrap all-in-one pod also give it a 500m CPU request; the pod has no
CPU limits, so the request is what drives its CPU share under contention.
In the production FluxInstance add it to the shared reconciler patch, so it
now receives the same --requeue-dependency=5s and 2000m/2048Mi limits the
other tuned controllers already have.

The flux-aio module exposes a resources override for the watcher (set via
CUE) but no args override, so the bootstrap --concurrent bump is a
post-render patch in the Makefile.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-07-01 07:42:33 +03:00
Aleksei Sviridkin cbc314a8c5 fix(kubevirt-cdi): order behind cozystack-basics so cdi-clone-dv RoleBinding finds cozy-public
The kubevirt-cdi default variant only depended on cozystack.networking,
which reconciles long before cozystack-basics provisions the cozy-public
namespace. kubevirt-cdi ships the cdi-clone-dv RoleBinding into cozy-public,
so on a fresh install the RoleBinding can race namespace creation and fail
with `namespaces "cozy-public" not found`. Flux retries normally mask it,
but under slow reconcile it surfaces as a hard UpgradeFailed.

Add a cozystack.cozystack-basics edge so the namespace exists first,
mirroring the backupstrategy-controller ordering. No cycle: cozystack-basics'
own dependencies (tenant-application, cozystack-engine, gateway-api-crds) do
not reach kubevirt-cdi. Pin the edge with a helm-unittest regression guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-30 19:51:46 +03:00
Aleksei Sviridkin bf14a73b39 feat(api): add SecurityGroup network policy resource (sdn.cozystack.io) (#2922)
## What this PR does

Introduces a tenant-facing `SecurityGroup` resource in a new
`sdn.cozystack.io/v1alpha1` API group, served by the Cozystack
aggregated API server as a 1:1 projection of a `CiliumNetworkPolicy`. A
SecurityGroup attaches to one of the tenant's managed applications by
reference (`spec.targetRef: {apiGroup, kind, name}`) and declares
allow-list ingress/egress for that application. Tenants manage network
policy for their own apps entirely through
`securitygroups.sdn.cozystack.io`, without being granted any access to
the `cilium.io` API group.

The translation is synchronous and stateless — the API server converts
each SecurityGroup into a CiliumNetworkPolicy in the same namespace on
write and back on read; there is no controller. The backing policy's
`endpointSelector` is **derived** from `targetRef` (matched on the
application's lineage labels
`apps.cozystack.io/application.{group,kind,name}`), never copied from
tenant input, so the selector is machine-generated and can only ever
match the referenced application's own pods in the tenant's namespace.
Policies owned by this API carry an `sdn.cozystack.io/securitygroup`
marker label that the storage always re-asserts and hides from the view,
so it only ever surfaces or mutates its own policies and never touches
platform-managed network policies.

The value is self-service network policy on a Cozystack-native,
RBAC-scoped surface, without exposing Cilium's version-coupled CRDs to
tenants. Because the selector is derived from an authorized application
reference rather than free-form tenant input, the RBAC boundary is
structural, not selector-validated: a tenant cannot express network
policy against arbitrary or platform-owned pods. The full design —
selector derivation from lineage labels, the structural boundary, the
additive-policy reality (a SecurityGroup widens a managed application's
allowed traffic but cannot sever its platform-managed management plane,
and equally cannot tighten below the platform baseline), the in-tree
CiliumNetworkPolicy mirror that avoids a Cilium Go-module dependency,
RBAC, and deferred extensions (raw-pod targeting, cluster scope,
reusable groups) — is in `pkg/apis/sdn/DESIGN.md`.

Implements the attachment model from #1614.

### Release note

```release-note
feat(api): add the SecurityGroup resource (sdn.cozystack.io/v1alpha1) — a tenant-facing, namespace-scoped firewall that attaches to a managed application by reference and projects 1:1 onto a CiliumNetworkPolicy with a machine-derived endpoint selector, letting tenants manage their application's network policy without cilium.io access.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added the `SecurityGroup` API resource (`sdn.cozystack.io` v1alpha1)
with a namespaced projection backed by Cilium network policies.
* Enabled the SDN `APIService` and wired SDN routing, scheme, and REST
support.
* Expanded RBAC to include SecurityGroup access and required Cilium
policy permissions.
* **Documentation**
* Added a design document describing projection behavior and rule
semantics.
* **Bug Fixes**
  * Improved API rule validation coverage for missing list-type fields.
* **Tests**
* Added SDN round-trip, fuzzing, RBAC, and comprehensive REST
create/get/list/update/delete/watch/resource-version tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 14:30:23 +03:00
Aleksei Sviridkin f9d1138c6e feat(keycloak): configurable login theme via platform branding (#3124)
## What this PR does

Makes a realm's Keycloak login theme configurable through platform
`branding` values, closing a gap: `branding` could set the realm display
name but not its login theme, and the keycloak chart's `themes:` list
was not reachable from platform values.

Two new keys under `branding` (propagated via `_cluster.branding`, the
same path that already feeds `displayName`):

- `branding.loginTheme` — keycloak-configure renders it as
`ClusterKeycloakRealm.spec.themes.loginTheme`, so the realm selects that
login theme.
- `branding.loginThemeImage` — when set together with `loginTheme`, the
keycloak chart appends `{name: loginTheme, image: loginThemeImage}` to
its effective theme list, so the existing init-container mounts the
theme image into `/opt/keycloak/themes/`. No StatefulSet patching or
per-package values plumbing.

`loginTheme` alone (no image) still selects the theme — for themes
already baked into the Keycloak image. Explicit chart `themes:` entries
keep working; the branding-derived entry is appended after them.

Flow: platform `branding` → `_cluster.branding` (cozystack-values
Secret) → keycloak-configure (realm loginTheme) + keycloak (theme init
container). Covered by helm-unittest in both charts.

### Screenshots

No bundled UI assets — this PR adds the mechanism only; the login theme
is operator-supplied via `loginThemeImage`, so the visual result depends
on that image.

### Release note

```release-note
feat(keycloak): select the realm login theme and ship its theme image through platform `branding` values (`branding.loginTheme`, `branding.loginThemeImage`)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring a custom login theme from platform
branding, including automatic selection in the realm setup.
* Theme entries can now be combined with existing theme settings and
rendered in the expected order.

* **Documentation**
* Expanded configuration guidance for branding and theme-related
settings with clearer examples.

* **Tests**
* Added coverage for login theme rendering and fallback behavior when
theme settings are missing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 12:29:52 +03:00
Aleksei Sviridkin 31c15b282f feat(network): mechanism-agnostic LoadBalancer exposure (ExposureClass + ServiceExposure) (#3081)
## What this PR does

Lays the foundation for migrating tenant LoadBalancer exposure off the
deprecated `Service.spec.externalIPs` (KEP-5707) and, eventually, off
MetalLB onto Cilium-native announcement — the mechanism-agnostic shape
prescribed when the earlier chart-side approach (#2468) was reverted in
#2603.

It adds a new `network.cozystack.io/v1alpha1` API group and a controller
that translates intent into whichever LoadBalancer backend the cluster
admin configured:

- `ExposureClass` (cluster-scoped, admin): binds a logical class name to
a backend (`externalIPs` | `metallb` | `cilium` | `robotlb`) and its VIP
address scope — the StorageClass/IngressClass pattern.
- `ServiceExposure` (namespaced, tenant intent): "expose this Service
via class X"; status reports the resolved backend and assigned IPs.
- A `serviceexposure` controller renders the backend's pool + announcer
once per class (MetalLB `IPAddressPool` + `L2Advertisement`; Cilium
`CiliumLoadBalancerIPPool` + `CiliumL2AnnouncementPolicy`), scoped to
the namespaces that opted in. Backends are pluggable behind a small Go
interface, so adding BGP or another cloud LB is one implementation, not
an API change. The controller never mutates the target Service — the
chart still owns its shape.

Platform wiring: `publishing.exposureClass.*` renders the admin
`ExposureClass` and the host ingress publishes through it. The default
(no class set, `externalIPs`) leaves every existing cluster's behaviour
byte-for-byte unchanged; the MetalLB→Cilium announcer migration becomes
a single `ExposureClass.spec.backend` change.

This is the foundation only — Part of #2586. Converting per-app
exposure, flipping Cilium `l2announcements` on, changing the default,
and removing MetalLB are deferred to later phases.

### Release note

```release-note
feat(network): add ExposureClass and ServiceExposure for mechanism-agnostic LoadBalancer exposure. Opt-in foundation; the default externalIPs publishing path is unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added new `ExposureClass` and `ServiceExposure` custom resources with
support for multiple backends (externalIPs, MetalLB, Cilium, RobotLB),
including optional L2 and address configuration.
* Added Helm wiring and templates to enable service/ingress publishing
via `ServiceExposure`, selecting a configured (or default)
`ExposureClass`.
* **Bug Fixes**
* Improved reconciliation, finalizer-based cleanup, and status updates
for readiness and assigned IP reporting, including safer handling when
backends or managed resources change.
* **Tests**
* Added end-to-end and unit test coverage for backend rendering,
readiness, and cleanup flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-30 12:29:48 +03:00
Myasnikov Daniil b9749cec80 fix(platform): md0 migration -- fail closed and preserve the no-delete label
Migration 47 pins the implicit default md0 node group before the kubernetes
chart upgrade so the live md0 MachineDeployment is not pruned. Two correctness
fixes:

- Stamp the cozystack-version ConfigMap via the shared stamp_cozystack_version
  helper instead of `kubectl create configmap | kubectl apply`. The label-less
  apply by the same field manager stripped the platform.cozystack.io/no-delete
  label migration 42 added, dropping cozystack-version out of the
  cozystack-no-delete-guardrail policy; the helper re-emits the label and is
  enforced for every migration >= 42 by hack/cozystack-version-stamp.bats.
- Fail closed. A non-NotFound read error is now counted as a failure rather
  than silently skipped, and any failure exits 1 so the pre-upgrade hook aborts
  before Helm rolls out the new chart against clusters whose md0 is not yet
  pinned. The version stamp stays at 47 and the migration retries next upgrade.

Also narrow the merge patch to only md0 so a concurrent edit to a sibling node
group between the read and the patch cannot be reverted.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-30 12:08:17 +05:00
mattia-eleuteri 602b8d06fd fix(platform): add migration 47 to pin implicit md0 node group
After the kubernetes chart stops force-merging a default md0 node group,
clusters that relied on the implicit md0 (defined other nodeGroups without
md0) would have their live md0 MachineDeployment pruned by Helm on the next
reconcile. Migration 47 makes md0 explicit on those clusters to preserve the
current node topology; it is idempotent, treats a missing CRD as a no-op, and
skips clusters that already define md0.

Signed-off-by: mattia-eleuteri <mattia@hidora.io>
2026-06-30 12:08:17 +05:00
Aleksei Sviridkin 60c9b39578 docs(platform): document branding keys including login theme
The branding map shipped as an undocumented empty map. Describe the keys it
actually supports (brandName, brandHtmlName) and the new login-theme keys
(loginTheme, loginThemeImage) consumed by the Keycloak charts, with an example.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-28 05:08:47 +03:00
Myasnikov Daniil 70320e0abf fix(platform): stamp migrations 45/46 via the cozystack-version helper
Migrations 45 and 46 (added by #2931) stamp the cozystack-version ConfigMap
with an inline `kubectl apply` heredoc instead of sourcing
lib/cozystack-version.sh and calling stamp_cozystack_version, the way 42/43/44
do. The cozystack-version-stamp.bats guard forbids any migration >= 42 from
bypassing the helper, so `make bats-unit-tests` fails on migration 45.

This only surfaces in PR CI (make unit-tests); main-push CI does not run the
bats suite, so #2931 landed red on this test and now blocks every PR that
merges current main.

Route both stamps through the helper. The helper renders a byte-identical
labeled manifest (golden-pinned in the bats suite), so this is behaviour-
preserving — it just restores the no-label-drift guarantee the heredoc form
could silently lose.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-26 20:40:02 +05:00
myasnikovdaniil 1df8c7c557 feat(kubernetes): bootstrap tenant workers with Talos via CABPT (#2931)
## What this PR does

> **Supersedes #2610.** Same work, rebased cleanly onto current `main`
so it now sits on top of #2867 (the `kube-controller-manager` VAP
type-checker fix). That fix is what was killing the base install in
#2610's E2E *before* the `kubernetes` app test ever ran, leaving the
Talos worker-bootstrap rewrite verified only by code reading. #2610's
67-commit branch is squashed here into 8 logical commits with original
authorship preserved (@kvaps and @IvanHunters as authors); the tree is
**byte-identical** to #2610's head — no code changes, just the rebase
onto `main` (to pick up #2867) plus history cleanup. The detailed review
history lives on #2610.

Phase 1 of the Kubernetes-app split design (see cozystack/community#8):
replace the Ubuntu+kubeadm worker bootstrap path with Talos, driven by
`cluster-api-bootstrap-provider-talos` (CABPT) and a
`clastix/talos-csr-signer` sidecar embedded in the Kamaji control-plane
pod. Existing tenants keep working — old machines roll out and get
replaced by Talos workers without manual intervention.

Highlights:

- Add CABPT (v0.6.12) as a second `BootstrapProvider` alongside the
existing kubeadm one in `capi-providers-bootstrap`.
- Bump the Kamaji control-plane provider with an upstream-bound patch
that exposes `KamajiControlPlane.spec.network.additionalServicePorts`,
used to publish trustd (50001/TCP) on the apiserver service.
- Generate Talos PKI (Ed25519 CA + trustd TLS) via cert-manager and
stable random Talos secrets (`token`, `clusterId`, `clusterSecret`,
`bootstrapToken`) using the helm lookup-and-reuse pattern.
- Materialise a kubeadm-format `bootstrap-token-<id>` Secret inside the
tenant `kube-system` via a Helm post-install/upgrade Job.
- Render a `TalosConfigTemplate` (worker machineconfig, `generateType:
none`) and switch the `MachineDeployment.bootstrap` reference from
`KubeadmConfigTemplate` to `TalosConfigTemplate`, gated on Talos secrets
and the Kamaji apiserver service being ready.
- Boot workers from the Talos openstack image via a CDI `DataVolume`
(`source.http.url` → `factory.talos.dev`), expose the system disk as
virtio-blk with `blockSize.custom: logical=512, physical=4096` so
4Ki-native block storage backends (LINSTOR/DRBD) play nicely with QEMU's
O_DIRECT writes and SeaBIOS still boots, drop the separate kubelet disk
(Talos lays out EPHEMERAL itself).
- Pin pod/service CIDRs in the worker machineconfig to the tenant ranges
(`10.243.0.0/16` / `10.95.0.0/16`) to avoid Talos's address-overlap
diagnostic against the host pod CIDR.
- Set `cluster.controlPlane.endpoint` and `cilium.k8sServiceHost` to the
Kamaji apiserver Service ClusterIP (looked up at render time), so
bootstrap survives the chicken-and-egg moment where tenant DNS does not
exist yet and the host CoreDNS does not serve the tenant zone.

End-to-end on a dev cluster: a fresh tenant brings up a Talos worker,
kubelet CSRs get approved (apiserver-client by Kamaji, kubelet-serving
by the talos-csr-signer sidecar), Cilium initialises with hostNetwork
against the apiserver ClusterIP, CoreDNS comes up, and the node
transitions to `Ready`.

### Release note

```release-note
feat(kubernetes): bootstrap tenant workers with Talos Linux instead of Ubuntu+kubeadm. Existing clusters roll over to Talos workers automatically; new tenants come up on Talos from the start. Powered by cluster-api-bootstrap-provider-talos and the talos-csr-signer sidecar in the Kamaji control-plane pod. Note: the tenant `kubernetes` HelmRelease now reports Ready as soon as Helm completes (Install/Upgrade DisableWait) and no longer blocks on worker/addon readiness — worker-rollout health is tracked by WorkloadMonitor, not the HelmRelease.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

**New Features**
* Added Talos worker configuration for `talos.version` and
`talos.schematicID`
* Introduced `nodeHealthCheck` (`maxUnhealthy`, `nodeStartupTimeout`)
for worker remediation tuning
* Added image overrides for `kubectl` and `talosCsrSigner`
* Exposed additional tenant control-plane Service ports support

**Breaking Changes**
* Removed Kubernetes v1.30; supported versions start at v1.31
* Renamed `nodeGroups.ephemeralStorage` → `nodeGroups.diskSize` with
consolidated Talos system-disk semantics

**Removals**
* Removed Ubuntu container-disk images and related containerd patch
mechanisms
* Air-gapped tenant workers temporarily unsupported during the Talos
transition

**Documentation**
* Updated chart docs, parameters, and GitOps upgrade guidance for
version bumps
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-26 20:06:49 +05:00
Kirill Ilin 5cb6ef435f feat(keycloak): opt-in separate hostname and route for the admin console (#2954)
## What this PR does

Adds an opt-in way to serve the Keycloak admin console and
Administration
REST API on a separate hostname and route from the public login
endpoints,
so the admin surface can be kept off the public internet.

- New `ingress.adminHost` in the keycloak chart. When set:
  - `KC_HOSTNAME_ADMIN` is configured so the admin console generates its
    URLs on that hostname;
  - the public (login) route is narrowed to the OIDC paths only
    (`/realms`, `/resources`, `/.well-known`), so the admin console and
    Administration REST API are no longer served on the login hostname;
- a dedicated route (Ingress or HTTPRoute, depending on
`gateway-enabled`)
    serves the admin hostname.
- New platform `publishing.ingressNameAdmin`, propagated as
  `_cluster.expose-ingress-admin`. The admin route attaches to the
Gateway/ingressClass it names, defaulting to `expose-ingress`. Pointing
it
at a separate (for example, private-IP) Gateway moves admin traffic off
the public one. The separate Gateway itself is provisioned out of band;
  this only selects where the admin route attaches.

Everything is empty by default, so existing deployments are unchanged.

Note: `KC_HOSTNAME_ADMIN` on its own is not a security boundary — the
Administration REST API stays reachable on any hostname routed to
Keycloak
(per the Keycloak docs). The path and network separation introduced here
is
what restricts access; full isolation requires attaching the admin route
to
a non-public Gateway.

Covered by helm-unittest suites for the keycloak chart and the platform
value wiring.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(keycloak): add opt-in `ingress.adminHost` to serve the admin console and Administration REST API on a separate hostname/route, plus `publishing.ingressNameAdmin` (`_cluster.expose-ingress-admin`) to attach that route to a separate (e.g. private) Gateway/ingressClass. Disabled by default; existing deployments are unchanged.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added optional configuration to serve the Keycloak admin console on a
dedicated hostname, with separate routing and ingress/gateway attachment
so admin UI/API can be isolated from public login/OIDC paths.
* Introduced a cluster-level setting to optionally expose admin routes
via a separate ingress/gateway class/namespace.

* **Tests**
* Added new YAML test coverage to validate defaulting, hostname split
behavior, and correct routing for both gateway-enabled and
gateway-disabled modes.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-26 14:08:23 +05:00
Aleksei Sviridkin 67692b4544 feat(platform,ingress): publish host ingress via ServiceExposure
Add the operator-facing knob that drives the new exposure model. When
publishing.exposureClass.name is set, the platform renders a cluster-scoped
ExposureClass binding the chosen backend (externalIPs, metallb, cilium,
robotlb) and VIP address scope, and carries the class name through the
cozystack-values Secret. The host ingress then renders a ServiceExposure
for its ingress-nginx Service, which the cozystack-controller translates
into the backend's pool and announcer.

The default (empty name) renders neither the ExposureClass nor the
ServiceExposure, so clusters on the externalIPs path are unchanged on
upgrade. The ingress Service shape is untouched: it is already published
as type LoadBalancer whenever externalIPs is empty.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-25 20:47:19 +03:00
IvanHunters e0597e3d99 fix(platform): make *-rd sources depend on cozystack-engine (#3041)
## Problem

On an unlucky Flux reconcile order, a `*-application` PackageSource that
ships a `system/*-rd` component can install before the
`applicationdefinitions.cozystack.io` CRD exists. Every `*-rd` chart
renders an `ApplicationDefinition` (`cozystack.io/v1alpha1`); that CRD
is installed by the `application-definition-crd` component of
`cozystack.cozystack-engine`. When an `*-rd` HelmRelease renders first,
it hard-fails with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, and that cascades into a broad install
timeout — surfacing even on changes that have nothing to do with
ordering, because it is a latent race.

## Fix

Add `- cozystack.cozystack-engine` to the `dependsOn` of every
`*-application` source that ships a `system/*-rd` component (29
sources). `gateway-application` already carried this edge — it was added
earlier for the TenantGateway CRD, the same bug class, but only for that
one source. A new invariant test fails if any source with a
`system/*-rd` component is missing the edge.

## Why

The ordering was previously implicit, relying on Flux happening to
reconcile `cozystack-engine` before the `*-application` sources. Making
the edge explicit removes the race deterministically. No dependency
cycle is introduced: `cozystack-engine` depends only on `networking` and
`cert-manager` (plus keycloak in its oidc variant), none of which reach
back to any `*-application`.

## Test

- New generic guard `hack/check-applicationdefinition-crd-ordering.bats`
(auto-discovered by `make bats-unit-tests`, runs in CI via `make
unit-tests`): parses every source and fails if any PackageSource variant
with a `system/*-rd` component lacks the `cozystack.cozystack-engine`
edge — so future sources are covered, not just today's set.
- `helm unittest packages/core/platform`: 13 suites / 66 tests pass; the
source edits do not disturb existing assertions.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved dependency ordering across several platform packages to
ensure related components install in the correct sequence.
* Prevented end-to-end test selection from being overly broad when only
ordering-related dependencies change.

* **Tests**
* Added checks to catch missing dependency requirements and to verify
test selection behavior stays accurate.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-25 19:21:50 +03:00
IvanHunters f6d5086190 fix(platform): order backupstrategy-controller after cozystack-basics (#3037)
## Problem

On a fresh install, the install step can fail with:

```text
Error creating resource via patch ... apps.cozystack.io/v1alpha1 Kind=Bucket
  name=cozy-backups namespace=tenant-root: namespaces "tenant-root" not found
```

`backupstrategy-controller` renders the `cozy-backups` Bucket **into**
the `tenant-root` namespace, but its `dependsOn` only guards the CRDs it
needs (backup-controller / bucket-application / objectstorage-controller
/ velero) — not the existence of the target namespace. `tenant-root` is
provisioned by `cozystack-basics` (the static tenant-root release), and
there is no ordering edge to it. On an unlucky Flux reconcile order the
Bucket patch races namespace creation, fails, and that transient failure
races the install script's `kubectl wait` deadline → the run fails.

This is latent and not introduced by any particular change — the
Bucket-into-tenant-root predates recent branches — so it flakes installs
on changes that have nothing to do with backups.

## Fix

Add `cozystack.cozystack-basics` to `backupstrategy-controller`'s
`dependsOn` so the `tenant-root` namespace exists before the Bucket is
applied. `cozystack-basics` is a dependency-graph leaf (nothing depends
on it), so no cycle is introduced.

Pinned with a helm-unittest regression guard
(`tests/sources_backupstrategy_dependson_test.yaml`) asserting the edge
— mirroring the existing `sources_basics_dependson_test.yaml` pattern.


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Improved install ordering so the backup strategy component waits for
required foundational services before deploying, reducing race
conditions during fresh installs.
* Clarified dependency relationships for cluster-wide platform
components to better reflect safe upgrade and installation behavior.

* **Tests**
* Added regression coverage to verify the backup strategy package
declares all required dependencies, including namespace and CRD
providers.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-25 19:21:25 +03:00
Aleksei Sviridkin 69247ba80c feat(platform): deploy the securitygroup-controller
Add the securitygroup-controller Helm package (Deployment, ServiceAccount,
ClusterRole and binding, image build) and register it as a default system
package so it ships with the platform.

The ClusterRole grants cluster-wide pod patch — the controller stamps the
SecurityGroup membership label on managed-app pods across dynamically-created
tenant namespaces, so the grant cannot be namespace-scoped — plus read and
finalizer patch on the backing CiliumNetworkPolicies, and leader-election
leases. Tenants gain no new permissions; the membership label is written
exclusively by this platform controller. helm-unittest pins the pod-patch and
ciliumnetworkpolicies grants so a dropped verb fails the suite rather than
silently failing closed at runtime.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-25 17:50:33 +03:00
Aleksei Sviridkin 4593b72cd1 feat(operator): support private OCI registry credentials for platform source (#2472)
## Summary

Implements support for private OCI registry and Git repository
credentials in platform sources. Adds `--platform-source-secret` flag to
the operator and corresponding Helm chart parameter.

Closes #2455

## Changes

### Operator (`cmd/cozystack-operator/main.go`)
- Added `--platform-source-secret` flag for specifying Secret name
- Updated `generateOCIRepository()` to set `spec.secretRef` when secret
is provided
- Updated `generateGitRepository()` to set `spec.secretRef` when secret
is provided
- Secret is expected to be in `cozy-system` namespace

### Helm Chart
- **`packages/core/installer/values.yaml`**: Added
`cozystackOperator.platformSourceSecret` parameter with documentation
- **`packages/core/installer/templates/cozystack-operator.yaml`**: Pass
`--platform-source-secret` flag to operator container

## Usage Example

### For private OCI registry (e.g., GitLab Container Registry):

```bash
# Create registry credentials secret
kubectl create secret docker-registry gitlab-registry-creds \
  --namespace cozy-system \
  --docker-server=registry.gitlab.example.com \
  --docker-username=<username> \
  --docker-password=<token>
```

```yaml
# Helm values
cozystackOperator:
  platformSourceUrl: 'oci://registry.gitlab.example.com/mygroup/cozystack-packages'
  platformSourceRef: 'tag=v1.0.0'
  platformSourceSecret: 'gitlab-registry-creds'
```

### For private Git repository:

```bash
# Create Git credentials secret (SSH key or basic auth)
kubectl create secret generic git-creds \
  --namespace cozy-system \
  --from-literal=username=<username> \
  --from-literal=password=<token>
```

```yaml
# Helm values
cozystackOperator:
  platformSourceUrl: 'https://github.com/private-org/cozystack-packages'
  platformSourceRef: 'branch=main'
  platformSourceSecret: 'git-creds'
```

## Testing

Manually verified:
- OCIRepository resource is created with `spec.secretRef` when secret
name is provided
- GitRepository resource is created with `spec.secretRef` when secret
name is provided
- Resources are created without `spec.secretRef` when secret name is
empty (backward compatible)
- Helm chart renders correctly with and without `platformSourceSecret`
parameter

## Checklist

- [x] Code follows project style and conventions
- [x] Operator supports both OCI and Git source types
- [x] Helm chart parameter documented with comments
- [x] Backward compatible (empty secret name is handled correctly)
- [x] Closes referenced issue (#2455)


## Release note

```release-note
feat(operator): add `--platform-source-secret` flag to authenticate the platform package source against private OCI registries (kubernetes.io/dockerconfigjson) and private Git repositories (basic-auth, bearer token, or SSH key). Empty value preserves existing public-registry behaviour.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added support for configuring a Kubernetes Secret for the platform
source.
* Installer settings and startup options now let you provide a secret
name for Flux source credentials.

* **Bug Fixes**
* Generated platform source resources can now reference the provided
Secret, enabling authenticated access for OCI and Git sources.

* **Documentation**
* Updated configuration docs to explain the new secret setting and
required secret format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-25 14:41:37 +03:00
Kirill Ilin caeb2c466f feat(platform): add publishing.ingressNameAdmin for a separate admin gateway
Expose a new opt-in publishing.ingressNameAdmin that is propagated to
_cluster.expose-ingress-admin. System components (currently Keycloak)
use it to attach their admin routes to a separate Gateway/ingressClass,
e.g. one bound to a private IP and unreachable from the public internet.
Empty by default, so admin endpoints keep sharing expose-ingress. The
private Gateway itself is provisioned out of band; this only selects
where the admin route attaches.

Assisted-By: Claude AI
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-06-25 15:28:47 +05:00
Ivan Okhotnikov fff9e827d1 fix(operator): address review feedback on private platform source
- Run `go mod tidy` to move github.com/fluxcd/pkg/apis/meta from
  indirect to direct require: it became a direct import after the
  SecretRef type fix.
- Refine the --platform-source-secret flag docstring and the matching
  values.yaml comment: drop identity.pub from the SSH required-keys
  list (Flux source-controller only consumes identity + known_hosts)
  and mention bearerToken as an alternative to username+password for
  Git over HTTPS.
- Note in values.yaml that the Secret must exist in cozy-system before
  the operator reconciles; otherwise source-controller reports a
  failed condition until the Secret is created.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
2026-06-24 23:22:54 +03:00
IvanHunters 0427b184fa feat(operator): support private OCI registry credentials for platform source
Added --platform-source-secret flag to cozystack-operator that allows
specifying a Secret name for registry authentication. This enables using
private OCI registries and Git repositories as platform sources.

Changes:
- cmd/cozystack-operator/main.go:
  - Add --platform-source-secret flag
  - Pass secretName to generateOCIRepository() and generateGitRepository()
  - Set spec.secretRef in generated resources when secret is specified

- packages/core/installer/values.yaml:
  - Add cozystackOperator.platformSourceSecret parameter

- packages/core/installer/templates/cozystack-operator.yaml:
  - Pass --platform-source-secret flag to operator container

Usage:
1. Create registry credentials secret in cozy-system namespace:
   kubectl create secret docker-registry my-registry-creds \
     --namespace cozy-system \
     --docker-server=registry.example.com \
     --docker-username=user \
     --docker-password=token

2. Configure Helm values:
   cozystackOperator:
     platformSourceUrl: 'oci://registry.example.com/repo/packages'
     platformSourceSecret: 'my-registry-creds'

Closes #2455

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-06-24 22:54:06 +03:00
Aleksei Sviridkin fd6e121b28 fix(platform): make *-rd sources depend on cozystack-engine
Every *-application PackageSource that ships a system/*-rd component
renders an ApplicationDefinition (cozystack.io/v1alpha1). That CRD is
installed by the application-definition-crd component of
cozystack-engine. Without an explicit dependsOn edge, a fresh install
where Flux reconciles a *-application source before cozystack-engine
fails to render with `no matches for kind "ApplicationDefinition" in
version "cozystack.io/v1alpha1"`, which cascades into a broad install
timeout unrelated to whatever change triggered the reconcile.

Add `- cozystack.cozystack-engine` to the dependsOn of every source that
ships a system/*-rd component; gateway-application already carried the
edge for the TenantGateway CRD. Add a generic invariant test that fails
if any such source is missing the edge, so future sources cannot
reintroduce the race.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 22:43:56 +03:00
Aleksei Sviridkin ee853b00bd fix(platform): order backupstrategy-controller after cozystack-basics
backupstrategy-controller creates the cozy-backups Bucket into the
tenant-root namespace, which cozystack-basics provisions. On a fresh
install the Bucket patch can race namespace creation and fail with
`namespaces "tenant-root" not found`, which then races the install
wait deadline and fails the run — on changes that have nothing to do
with backup ordering. Add the cozystack-basics dependsOn edge so the
target namespace exists first, pinned with a helm-unittest regression
guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 22:34:09 +03:00
Myasnikov Daniil bf92f3f3b2 chore(kubernetes): make migration 45 executable and drop tracking-PR refs
Set the executable bit on migration 45 to match its siblings (44, 46). Replace
"PR #2610" references in migrations 45/46 with a description of the change.
Correct migration 45's comment: the KubeadmConfig children CAPI spawns from the
template are owned by their Machine (not Helm), so they carry no
app.kubernetes.io/managed-by=Helm label, are not matched by the selector, and
are never at risk of Helm pruning.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-24 22:19:46 +05:00
Aleksei Sviridkin de44ecf15a test(platform): enforce go-forward version stamps route through the helper
The helper header and run-migrations.sh claimed every migration-driven
stamp routes through the helper, but migrations 1-41 still stamp the
cozystack-version ConfigMap label-less and are backfilled by migration 42.
Scope the comments to go-forward stamps (migration 42 onward and the
bootstrap path), and add a unittest asserting no migration numbered >= 42
nor run-migrations.sh stamps cozystack-version without the helper, so the
no-drift invariant is enforced rather than aspirational.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:32:52 +03:00
Aleksei Sviridkin 985b4b0954 fix(platform): guard stamp_cozystack_version against a missing version
Mirror render_cozystack_version_manifest's argument guard in
stamp_cozystack_version so a missing version aborts before the kubectl
pipe. Without it, a render failure in a shell without pipefail would
leave the pipeline exit status to kubectl, which would apply empty input.
Add a unittest covering the guard.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:32:05 +03:00
Aleksei Sviridkin 3aabb0cb7c fix(platform): stamp the no-delete label on bootstrap cozystack-version
The bootstrap path that creates cozystack-version when the ConfigMap is
absent applied it without the platform.cozystack.io/no-delete label,
leaving it unguarded by the cozystack-no-delete-guardrail
ValidatingAdmissionPolicy. Route it through the shared helper so the
bootstrap ConfigMap carries the label like every migration-driven stamp.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:32:05 +03:00
Aleksei Sviridkin b199b5e438 refactor(platform): stamp cozystack-version via shared helper in migrations
Migrations 42, 43 and 44 each carried a copy-pasted heredoc that applied
the labeled cozystack-version ConfigMap. Source the shared helper and call
it instead, so the labeled manifest has a single definition and future
migrations cannot reintroduce a label-less apply.

Behavior-preserving: the rendered manifest is byte-identical to the
previous heredocs. Migration 42 keeps its MIGRATION_DRY_RUN path, now
building its manifest from the helper's pure renderer.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:32:05 +03:00
Aleksei Sviridkin 50344f1ab9 refactor(platform): add shared cozystack-version stamp helper
Extract the labeled cozystack-version ConfigMap stamp into a sourced
shell helper so migrations cannot drift back to a label-less apply that
strips the platform.cozystack.io/no-delete label.

render_cozystack_version_manifest emits the manifest on stdout (pure,
unit-testable); stamp_cozystack_version renders and applies it. The
namespace defaults to cozy-system and honors $NAMESPACE.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-24 15:32:04 +03:00
Aleksei Sviridkin 2494843599 chore(testing): refresh e2e-sandbox tools and base to current upstream (#3010)
## What this PR does

Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib)
carried by the stale tool binaries and base layers.

- base: `ubuntu:22.04` → `24.04`; `netcat` → `netcat-openbsd` (on 24.04
`netcat` is a candidate-less virtual package, so the concrete provider
must be named)
- kubectl: `1.33.2` → `1.33.13` — stays on the 1.33 line that the e2e
control plane pins at `v1.33.12` (`hack/e2e-prepare-cluster.bats`), so
client/server skew is unchanged
- talosctl: `1.10.4` → `1.13.5` — aligns the client with the v1.13.x
Talos nodes the sandbox provisions
- helm: `3.18.3` → `3.21.2` (latest helm 3) — the e2e flow drives the v3
CLI, so it stays on the 3.x line rather than moving to the breaking 4.x
- yq: `4.44.3` → `4.53.3`
- `values.yaml`: re-pinned to the rebuilt multi-arch digest
(`linux/amd64` + `linux/arm64`)

`crust-gather` and `cozyhr` already track their latest releases; `flux`
and `mc` install the current release at build time. The built image's
tool versions were verified in place (kubectl `v1.33.13`, talosctl
`v1.13.5`, helm `v3.21.2`, yq `v4.53.3`, flux `2.8.8`, ubuntu `24.04`).

Closes #2891

### Release note

```release-note
chore(testing): refresh e2e-sandbox tools and base (ubuntu 24.04, kubectl 1.33.13, talosctl 1.13.5, helm 3.21.2)
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated the e2e sandbox image to a new fixed digest for more
consistent test runs.
* Refreshed the sandbox build environment (Ubuntu 24.04) and upgraded
tooling (Kubernetes, Talos, Helm, and yq) to newer stable versions.
* Improved OS package setup by switching from `netcat` to
`netcat-openbsd` and consolidating apt cleanup to reduce image size.
* **Documentation**
* Updated e2e testing guidance: removed the recommendation to pin the
management cluster Kubernetes version, and added guidance to
snapshot/restore cluster state between test groups.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-23 13:14:20 +03:00
Aleksei Sviridkin bebf4db0df chore(build): refresh first-party Alpine runtime bases to 3.24 (#3009)
## What this PR does

Moves the pinned Alpine runtime bases of the first-party images forward
to `alpine:3.24`, clearing the published `musl` / `zlib` advisories
carried by the stale layers.

| image | base |
| --- | --- |
| mariadb-backup | 3.20 → 3.24 |
| cozystack-operator | 3.22 → 3.24 |
| platform-migrations | 3.22 → 3.24 |
| grafana-dashboards | 3.22 → 3.24 |
| kubeovn-webhook | 3.21.3 → 3.24 |
| redis-operator | latest → 3.24 |

Each runtime `apk` set was verified to resolve on `alpine:3.24`. These
images rebuild on the release asset pipeline, which re-pins the digests
in `values.yaml` on the next build — matching how the rest of the ~30
first-party images pick up the shared base (the issue's "rebuild and run
the standard test pipeline").

**Scope notes:**

- The build-only `alpine AS source` stages (keycloak-operator,
objectstorage-controller) are left as-is: their final images are
distroless, so no Alpine layer ships.
- The s3manager base is owned by the separate bucket bump (#2887), and
kilo's base moves in its own image rebuild (#2898).
- **Side effect for platform-migrations**: its `kubectl` / `helm` come
from the Alpine repo, so the base bump moves them `1.33 → 1.36` and
`3.18 → 3.19`. Both are current upstream stable and in skew with the
modern control plane for the migration scripts' operations (apply / get
/ label / helm release ops).

Closes #2902

### Release note

```release-note
chore(images): refresh first-party Alpine runtime bases to 3.24 to clear the musl/zlib advisories
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated Alpine Linux base images across multiple application
containers to version 3.24 (including backup utilities,
platform/operator components, monitoring dashboards, and
networking/webhook components) to improve security and build
consistency.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-23 13:14:15 +03:00
Aleksei Sviridkin 51c5273d0b feat(platform): ACME wildcard certificate on the default ingress-nginx path (#2988)
## What this PR does

On the default ingress-nginx path (`gateway.enabled=false`) every
published hostname still minted its own per-host ACME certificate via
ingress-shim, even with a DNS-01 solver configured. That hits the Let's
Encrypt rate limit (50 certs per registered domain per week) once a
deployment exceeds ~50 endpoints. The Gateway API path already issues a
single per-apex wildcard in DNS-01 mode; this brings the ingress-nginx
path to parity. It implements the remaining ingress-nginx gap that #2400
was narrowed to in its triage comment (DNS-01 is already multi-provider,
and the Gateway path already issues wildcards).

The new `publishing.certificates.wildcard` toggle is opt-in and OFF by
default. When set to `true` with `solver=dns01` and no operator-provided
wildcard Secret, the platform issues one shared wildcard `Certificate`
for `<root-host>` + `*.<root-host>` and serves the resulting Secret as
the publishing controller's `--default-ssl-certificate`. The system
service Ingresses (dashboard, grafana, keycloak, harbor, …) then stop
requesting a per-host cert.

How it works:

- `core/platform` `apps.yaml` computes the effective wildcard Secret
name and a new `_cluster.wildcard-issue` signal. An operator-provided
`wildcardSecretName` (the BYO path) always wins and never triggers
issuance; HTTP-01 cannot issue wildcards; and when Gateway API is
enabled the TenantGateway controller issues the wildcard instead — so
all three are excluded from auto-issuance.
- `extra/ingress` renders the wildcard `Certificate` only on the
publishing controller (`Release.Namespace == expose-ingress`), because
the Secret must be same-namespace for ingress-nginx to read it. The
Certificate references the DNS-01 `ClusterIssuer` that
`cert-manager-issuers` already renders in `dns01` mode, so no per-tenant
Issuer is minted.

This reuses the existing wildcard-secret consumption path, so no
per-service Ingress template changes were needed — only the issuance
side is new.

Why opt-in (default off): the chosen wildcard Secret name rides the
`_cluster` channel, which every child tenant inherits verbatim, so this
is not root-tenant scoped. A single-label wildcard `*.<root-host>`
covers `<service>.<root-host>` but not a custom service host outside it
(a keycloak `ingress.host`, harbor `host`, or grafana `host` pointed at
another domain) nor a child tenant's nested host
(`<service>.<tenant>.<root-host>`). Enabling issuance makes every such
service drop its per-host ACME cert and fall back to the default
certificate, which does not cover it. Leaving it off by default means a
dns01 cluster is never silently switched on upgrade; an operator who
enables it accepts the same coverage responsibility as the
operator-provided wildcard path, which propagates identically. The
tradeoff is pinned by contract tests (keycloak custom host, bucket
nested child-tenant host). Per-tenant wildcards remain on the Gateway
path and are tracked separately.

Docs: cozystack/website#588

Closes #2400. Part of #2811.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(platform): add opt-in shared wildcard certificate issuance on the default ingress-nginx path via publishing.certificates.wildcard (default false). When enabled with a DNS-01 solver, the platform issues one *.<root-host> wildcard for system services instead of a per-host ACME certificate, avoiding Let's Encrypt rate limits at scale.
```



<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added automatic wildcard certificate issuance support for DNS-01 ACME
solver.
* New `certificates.wildcard` configuration option to enable shared
wildcard TLS certificates.
* Ingress rules now properly utilize wildcard certificates when enabled.

* **Tests**
* Added comprehensive test suites validating wildcard certificate
generation and ingress integration across multiple scenarios.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-23 13:13:48 +03:00
Aleksei Sviridkin 553c0be708 chore(mongodb): bump psmdb-operator and CR to 1.22.0 (#2968)
## What this PR does

Bumps the Percona Operator for MongoDB stack from 1.21.x to 1.22.0
(latest upstream release), picking up the published security fixes in
the operator image that the issue enumerates.

Two coupled changes:

- `system/mongodb-operator`: re-vendors the upstream psmdb-operator
chart 1.21.2 → 1.22.0 via `make update` (operator image →
`percona/percona-server-mongodb-operator:1.22.0`). The vendored tree is
byte-for-byte identical to a fresh upstream pull.
- `apps/mongodb`: bumps the managed CR `spec.crVersion` 1.21.1 → 1.22.0
so the platform tracks the deployed operator instead of running at the
N-1 skew, plus the matching helm-unittest assertion.

The 1.22.0 CRD diff is additive (native MinIO backup storage, storage
scaling, `hookScript`, `PBMReady` condition, new `metrics.k8s.io` RBAC
that activates under our `watchAllNamespaces: true`). The only schema
removal — the restore-spec `replset` string, renamed upstream to
`replsetRemapping` — is unused by the cozystack mongodb app (restore
template uses `clusterName`/`pitr`/`backupSource` only). Deprecations
(PSMDB 6.0, PMM2, `enableVolumeExpansion`) stay backward-compatible. The
PBM (`2.11.0`) and PMM (`2.44.1`, disabled by default) sidecars stay
compatible with 1.22.0 and are out of scope here.

Verified: `helm unittest packages/apps/mongodb` 94/94 pass; `helm
template` renders the operator with the new image.

**Upgrade note:** Helm v3 does not update CRDs on `helm upgrade`. On
existing clusters the 1.22.0 CRD must be applied manually (`kubectl
apply --server-side -f .../crds/crd.yaml`); the operator pod itself runs
fine against the older CRD until then.

Closes #2888

### Release note

```release-note
chore(mongodb): bump psmdb-operator to 1.22.0 and managed CR to crVersion 1.22.0
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* MongoDB Operator upgraded to version 1.22.0 with enhanced custom
resource definitions supporting backup storage configuration, hook
scripts, and storage autoscaling capabilities
* Added configurable namespace override for flexible operator deployment
  * Extended RBAC permissions for metrics collection

* **Chores**
  * Kubernetes minimum version requirement updated to 1.32+
  * Helm chart version bumped to 1.22.0

* **Documentation**
* Expanded CRD upgrade and installation guidance with multiple upgrade
strategies

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-23 13:13:31 +03:00
Aleksei Sviridkin 16041736f2 chore(talos): bump core/talos to Talos v1.13.5 and matchbox v0.11.0 (#3008)
## What this PR does

Refreshes the `core/talos` asset/profile layer onto current upstream to
clear the published advisories the scanner flagged in the stale pinned
images (`libcrypto3` plus the Go-module advisories compiled into the
installer).

- **profiles**: regenerated by `hack/gen-profiles.sh v1.13.5`, moving
the base installer and every system extension to the v1.13.5 line —
installer `v1.13.0` → `v1.13.5`; `amdgpu`/`i915`/`drbd`/`zfs` to their
`-v1.13.5` builds; `amd-ucode` and the firmware blobs to the
`20260519`/`20260512` snapshots. The extension digests come verbatim
from the official `ghcr.io/siderolabs/extensions:v1.13.5`
`image-digests` manifest. v1.13.5 rebuilds the installer on the Go
1.26.4 toolchain and current runtime (kernel, containerd, runc,
OpenSSL).
- **matchbox**: base image `quay.io/poseidon/matchbox` `v0.10.0` →
`v0.11.0`, clearing the `libcrypto3` advisories in the matchbox runtime.

`v1.13.5` is the latest stable Talos release (no `v1.14` line yet).
`cozy-talos` ships no runtime templates — it drives the node-image /
installer / matchbox asset builds. The Talos installer and matchbox
images are rebuilt from these sources by the release asset pipeline (the
privileged imager build), which also re-pins the generated
`matchbox.tag` digest; this change is the source bump.

### Release note

```release-note
chore(talos): bump core/talos to Talos v1.13.5 and matchbox v0.11.0
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Chores
* Updated Talos infrastructure components to v1.13.5
* Updated matchbox service to v0.11.0
* Updated system extension firmware and driver components

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-23 13:12:22 +03:00
Myasnikov DaniilandClaude Opus 4.8 b5cc2ae120 fix(kubernetes): address review findings on Talos worker bootstrap
- talos-reconcile-job: emit a `---` separator inside the nodeGroups
  range so a tenant with multiple node groups renders valid
  multi-document YAML instead of two Jobs concatenated into one doc
- talos-reconcile + bootstrap-token jobs: add a restricted
  securityContext (drop ALL caps, no privilege escalation, read-only
  root fs) and CPU/memory requests+limits; drop the unused management
  SA token mount on the bootstrap-token job, which only talks to the
  tenant apiserver via the mounted admin kubeconfig
- migration 44: list Helm-managed resources by the
  app.kubernetes.io/managed-by=Helm label, not meta.helm.sh/release-name
  (an annotation, so it never matched as a --selector — the keep
  annotation was silently never applied)
- cluster.yaml: fail fast on a malformed nodeHealthCheck.maxUnhealthy
  rather than silently coercing it to 0 via `| int`
- config: make helm-install-disable-wait parsing genuinely
  case-insensitive (EqualFold + trim) to match its documented contract
- capi-providers bootstrap/cpprovider Makefiles: guard the
  patches/*.diff loop against an empty glob

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-23 14:45:53 +05:00
IvanHuntersandMyasnikov Daniil cc9485e425 fix(platform): pre-Talos-rollover migrations to protect existing tenants
Migration 44 pins Helm-managed KubeadmConfigTemplates with resource-policy=keep so Helm cannot prune them mid-rollover; migration 45 bumps tenants on spec.version v1.30 to v1.31 ahead of the Talos support-matrix guard. Both stamp cozystack-version with its deletion-protection label. targetVersion is bumped to 46.

Co-authored-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-06-23 14:43:36 +05:00
Aleksei Sviridkin 381edde331 chore(build): refresh first-party Alpine runtime bases to 3.24
Moves the pinned Alpine runtime bases of the first-party images forward to
alpine:3.24 to clear the published musl / zlib advisories carried by the stale
layers:

- mariadb-backup: 3.20 -> 3.24
- cozystack-operator: 3.22 -> 3.24
- platform-migrations: 3.22 -> 3.24
- grafana-dashboards: 3.22 -> 3.24
- kubeovn-webhook: 3.21.3 -> 3.24 (also drops the over-specific patch pin in
  favour of the minor-float convention used elsewhere)
- redis-operator: latest -> 3.24 (pins the previously-floating runtime base)

These rebuild on the release asset pipeline; the digests pinned in values.yaml
are re-pinned there on the next build, matching how the rest of the ~30
first-party images pick up the shared base.

Side effect for platform-migrations: its kubectl / helm come from the Alpine
repo, so the base bump moves them 1.33 -> 1.36 and 3.18 -> 3.19. Both are
current upstream stable and compatible with the management cluster; for the
cozystack migration scripts (apply / get / label / helm release ops) the newer
kubectl is in skew with the modern control plane.

The build-only `alpine AS source` stages (keycloak-operator,
objectstorage-controller) are left as-is: their final images are distroless, so
no Alpine layer ships. The s3manager base is owned by the separate bucket bump,
and kilo's base moves in its own image rebuild.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-23 05:03:45 +03:00
Aleksei Sviridkin e3ed42a608 chore(testing): refresh e2e-sandbox tools and base to current upstream
Rebuilds the e2e test sandbox on current upstream to clear the published
advisories (containerd, helm, go-git, moby/spdystream, Go stdlib) carried by the
stale tool binaries and base layers:

- base: ubuntu 22.04 -> 24.04; netcat -> netcat-openbsd, since `netcat` is a
  candidate-less virtual package on 24.04
- kubectl: 1.33.2 -> 1.33.13, staying on the 1.33 line that the e2e control
  plane pins at v1.33.12, so client/server skew is unchanged
- talosctl: 1.10.4 -> 1.13.5, aligning the client with the v1.13.x Talos nodes
  the sandbox provisions
- helm: 3.18.3 -> 3.21.2 (latest helm 3); the e2e flow drives the v3 CLI, so it
  stays on the 3.x line rather than moving to the breaking 4.x
- yq: 4.44.3 -> 4.53.3
- values.yaml: re-pinned to the rebuilt multi-arch digest

crust-gather and cozyhr already track their latest releases; flux and mc install
the current release at build time.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-23 02:23:39 +03:00
Aleksei Sviridkin cac799518c chore(talos): bump core/talos to Talos v1.13.5 and matchbox v0.11.0
Refreshes the core/talos asset/profile layer onto current upstream to clear the
published advisories the scanner flagged in the stale pinned images.

- profiles: regenerated by hack/gen-profiles.sh v1.13.5, moving the base
  installer and every system extension to the v1.13.5 line (installer v1.13.0 ->
  v1.13.5; amdgpu/i915/drbd/zfs to their -v1.13.5 builds; amd-ucode and the
  firmware blobs to the 20260519/20260512 snapshots). v1.13.5 rebuilds the
  installer on the Go 1.26.4 toolchain and current runtime (kernel, containerd,
  runc, OpenSSL), clearing the Go and libcrypto advisories compiled into the
  stale v1.13.0 installer.
- matchbox: base image quay.io/poseidon/matchbox v0.10.0 -> v0.11.0, clearing
  the libcrypto3 advisories in the matchbox runtime.

cozy-talos ships no runtime templates -- it drives the node image / installer /
matchbox asset builds. The Talos installer and matchbox images are rebuilt from
these sources by the release asset pipeline (the privileged imager build), which
also re-pins the generated matchbox digest; this change is the source bump.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-23 01:40:23 +03:00
Aleksei Sviridkin 34563ce8c7 feat(platform): add opt-in shared wildcard issuance for ingress path
On the default ingress-nginx path (gateway.enabled=false) a DNS-01 solver
still minted a per-host ACME certificate for every system service, hitting
Let's Encrypt rate limits once a deployment exceeds ~50 endpoints. Add an
opt-in publishing.certificates.wildcard toggle (default false) and compute,
in apps.yaml, an effective wildcard secret name plus a wildcard-issue
signal: when enabled with solver=dns01 and no operator-provided
wildcardSecretName, the platform targets one shared wildcard cert
(*.<root-host> + <root-host>) instead of per-host certs.

Issuance is off by default because a single-label wildcard does not cover
a custom service host outside *.<root-host> (e.g. a keycloak ingress.host,
harbor host, or grafana host pointed at another domain); enabling it would
otherwise switch such a host to the controller's default certificate and
serve the wrong certificate. The operator-provided (BYO) name always wins
and never triggers issuance; http01 cannot issue wildcards; and the
Gateway API path issues its own wildcard via the controller, so all three
are excluded. This reuses the existing wildcard-secret-name consumption
path, so no per-service ingress template changes are needed.

Part of #2811.

Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 20:01:40 +03:00
Aleksei Sviridkin 4a9be4db80 chore(maintenance): align remaining alpine/k8s tools-image refs to 1.36.2
Six further references to the alpine/k8s tools image were still pinned at
1.33.4 across the installer, the cozystack-api post-upgrade hook, the vm-disk
and vm-instance resize hooks, the vertical-pod-autoscaler CRD-install override,
and the e2e cilium leak healer. Move them all to 1.36.2 so the whole repo runs
a single current tools-image version (the two digest-pinned refs get the
matching multi-arch digest sha256:44ef4942...). Same stale-stdlib remediation
as the seaweedfs bump in this PR.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 19:12:33 +03:00
Aleksei Sviridkin a730306774 chore(packages/core/talos): bump matchbox base image to v0.11.0
Refreshes the matchbox PXE image base from the 2021-era v0.10.0 to v0.11.0,
clearing the long-fixed golang.org/x/crypto and x/net versions baked into the
old build. v0.11.0 only bumps butane (v0.17->v0.18) and adds darwin arm64
artifacts; no behavior change for Talos PXE boot.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 19:12:32 +03:00
Aleksei Sviridkin e9aaee286e fix(mongodb-operator): apply CRD updates on upgrade via CreateReplace
psmdb-operator evolves its CRD schema between minor versions; 1.22.0 adds
spec.storageScaling/storageAutoscaling, vault and other fields. Helm never
upgrades CRDs shipped in crds/, so without an explicit policy the operator
binary upgrades on an existing cluster while the live CRD stays behind and
the apiserver rejects the new fields. Opt the mongodb-operator PackageSource
into CreateReplace, consistent with etcd-operator, and add a platform test
pinning the policy.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-06-22 19:05:03 +03:00