## What this PR does
Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.
Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).
Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.
## Commits
**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).
**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.
**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).
**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.
## Scope discipline
This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.
## Verification
- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.
### Release note
```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).
When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
* New SeaweedFS DB Helm chart and optional managed DB release.
* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.
* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
* Removed aggressive HelmRelease force-upgrade/install flags.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address non-blocking review feedback from lexfrei: the flux-tenants
sharded-controller Deployment carried a stale app.kubernetes.io/version: v2.7.3
while running helm-controller v1.5.0 (Flux 2.8.0). The flux-aio Makefile update
target only synced the container image, so the version label drifted on every
bump and make update never self-corrected. Bump the label to v2.8.0 and extend
the yq patch to carry app.kubernetes.io/version from the source flux Deployment.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Re-render internal/fluxinstall/manifests/fluxcd.yaml via
`make update` in packages/core/flux-aio (timoni bundle build against
oci://ghcr.io/stefanprodan/modules/flux-aio at the latest tag) so
the cozystack-operator binary embeds Flux 2.8.0 instead of 2.7.3.
Auto-syncs the helm-controller image into fluxcd-tenants.yaml.
Embedded versions now match the chart bump in the prior commit:
flux: v2.7.3 -> v2.8.0
helm-controller: v1.4.3 -> v1.5.0 (enables healthCheckExprs)
source-controller: v1.7.3 -> v1.8.0
kustomize-controller: v1.7.2 -> v1.8.0
notification-controller:v1.7.4 -> v1.8.0
source-watcher: v2.0.2 -> v2.1.0
Verified by rebuilding cmd/cozystack-operator and running `strings`
on the binary -- the expected v1.5.0 / v1.8.0 / v2.1.0 image strings
are now embedded in place of the v1.4.x / v1.7.x ones.
Structure preserved:
- single AIO Deployment named "flux" in namespace cozy-fluxcd
- all 5 controllers + source-watcher as containers in one pod
- custom tolerations from flux-aio.cue intact
- Namespace pod-security label still "privileged" via the
Makefile yq post-processing step
Net diff is -3664 lines (mostly CRD schema simplification in 2.8).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
## What this PR does
Adds a `ValidatingAdmissionPolicy` that blocks `DELETE` on critical
platform
objects labeled `platform.cozystack.io/no-delete=true`. Native,
in-process CEL
evaluation — no webhook DaemonSet, no Service, no TLS, no image to
maintain.
Requires Kubernetes 1.30+.
This is an alternative to the closed#2402 (admission webhook approach).
### Protected objects (labeled in this PR)
| Resource | Kind | Source |
|---|---|---|
| `cozy-system` | Namespace |
`packages/core/installer/templates/cozy-system-{namespace,labels}.yaml`
|
| `tenant-root` | Namespace |
`packages/system/cozystack-basics/templates/tenant-root.yaml` |
| `tenant-root` | HelmRelease (ns: tenant-root) | same |
| `cozystack-version` | ConfigMap (ns: cozy-system) |
`packages/core/platform/templates/cozystack-version.yaml` |
| `cozystack-packages` | OCIRepository (ns: cozy-system) |
`packages/core/platform/templates/repository.yaml` |
| `letsencrypt-prod`, `letsencrypt-stage`, `selfsigned-cluster-issuer` |
ClusterIssuer |
`packages/system/cert-manager-issuers/templates/cluster-issuers.yaml` |
| `linstorcluster` | LinstorCluster |
`packages/system/linstor/templates/cluster.yaml` |
| `packages.cozystack.io`, `packagesources.cozystack.io` | CRD | label
stamped in `internal/crdinstall/install.go` |
### Bypass
```bash
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
```
### Why VAP instead of a webhook
Same outcome as #2402 but with zero runtime infrastructure: no
DaemonSet, Service,
TLS certificate, or image. The kube-apiserver evaluates the CEL
`expression: "false"` in-process and denies because `validationActions:
[Deny]`.
`objectSelector` on the binding scopes evaluation to labeled objects
only — every
other DELETE in the cluster is unaffected.
### Release note
```release-note
feat(platform): add deletion-protection guardrail (ValidatingAdmissionPolicy)
blocking DELETE on cozystack platform objects labeled
`platform.cozystack.io/no-delete=true`. Bypass by removing the label first.
Requires Kubernetes 1.30+.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Deletion-protection guardrail: objects labeled
platform.cozystack.io/no-delete=true are denied DELETE requests; removal
of the label allows deletion.
* **Chores**
* Platform migration to v43 backfills the no-delete label onto system
resources and namespaces.
* **Tools**
* Added an "unprotect" teardown utility to remove the no-delete label
for uninstall/disaster recovery.
* **Documentation**
* Helm NOTES now warns when the guardrail is inactive due to missing
cluster API.
* **Tests**
* New tests and e2e checks validate deletion-protection behavior.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2650?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What
Implements the `strategy.backups.cozystack.io/Job` backup strategy — a
generic, application-agnostic driver that runs a user-supplied
Kubernetes Job to create (and restore) a backup.
## Background
This PR began as a draft written against an earlier shape of the backups
API. The API has since moved to a `BackupClass`-based dispatch model,
which broke the original draft on three counts:
- `BackupJob.Spec.StrategyRef` and `BackupJob.Spec.StorageRef` were
removed; the strategy and its parameters are now resolved from a
`BackupClass` into a `ResolvedBackupConfig`.
- Strategy handling moved from standalone per-strategy reconcilers to a
dispatch model: `BackupJobReconciler.Reconcile` /
`RestoreJobReconciler.Reconcile` resolve the strategy and switch into
per-strategy methods (`reconcileVelero`, `reconcileCNPG`,
`reconcileAltinity`, …).
- The draft's template context (`.Values`, `.Storage`) diverged from the
convention the other strategies adopted, and its `.Storage` half
targeted a storage API that does not exist.
## What changed
- **`reconcileJob` / `reconcileJobRestore`**
(`internal/backupcontroller/jobstrategy_controller.go`): replaces the
standalone `BackupJobStrategyReconciler` with the two dispatch methods
the `BackupJob` / `RestoreJob` reconcilers call. The Job strategy is the
generic counterpart of the Altinity driver — same lifecycle (render
`PodTemplateSpec` → run a one-shot `batch/v1.Job` → poll its terminal
condition → emit a `Backup`), minus the ClickHouse-specific
`applicationRef` gate, so it runs for any application. Shared package
helpers (`renderJobTemplate`, `jobConditionState`,
`getApplicationUnstructured`, status helpers, deterministic Job naming,
controller refs) are reused.
- **Template context** follows the in-tree convention used by the other
drivers: `.Application` (live application object), `.Release.Name` /
`.Release.Namespace`, `.Mode` (`backup` / `restore`), `.Parameters`
(resolved `BackupClass` parameters), and `.Backup` (restore runs only).
Strategy parameters round-trip through the `Backup`'s `driverMetadata`
so a later restore re-renders against the same values.
- **Restore** re-renders the same template with `.Mode == "restore"`.
Cross-namespace restore is unsupported (the RestoreJob target reference
is namespace-local), matching the Altinity driver.
- **`JobSpec.Template` doc + regenerated CRD**: the field doc previously
advertised `.Values`; it now documents the real context, and
`strategy.backups.cozystack.io_jobs.yaml` is regenerated to match (the
change is limited to that field's description).
- **Tests** (`jobstrategy_controller_test.go`): backup-Job creation /
args / labels / status, completion → `Backup` artifact with parameter
round-trip, Job-failure surfacing, missing-strategy terminal failure,
restore into the target namespace, restore completion, and the
parameter-prefix unit. A deliberately generic application kind pins that
the driver never gates on `applicationRef.kind`.
No RBAC changes are required: the controller already holds
`strategy.backups.cozystack.io/*`, `batch/jobs`, `apps.cozystack.io/*`,
and `backups` permissions.
## Notes for reviewers
- Backup and restore share one template, so a strategy author's
`PodTemplateSpec` must branch on `.Mode` to do anything
restore-specific. This matches the existing convention; worth confirming
it's the intended contract for the Job strategy's docs/examples.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Implemented full backup/restore job strategy with deterministic Job
creation, idempotent execution, and status/condition updates; added demo
example scripts and reusable shell helpers for a NATS JetStream
backup/restore workflow.
* **Documentation**
* Clarified Job template rendering context and listed supported template
variables; added README and step-by-step scenario docs for the NATS
example.
* **Tests**
* Added comprehensive unit tests covering job creation,
completion/failure handling, parameter extraction, and unmappable-target
failure cases.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/1721?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Address review feedback on the Job backup strategy:
- Assert .Application is templated: the rendered backup Job now reads
.Application.spec.replicas, the load-bearing context key that lets a
generic backup tool read live app spec (previously only .Release,
.Mode, .Parameters and .Backup had coverage).
- Pin failure-path messages so a reordered error path can't pass on the
wrong branch: the missing-strategy test asserts the strategy name, and
the unmappable-kind tests assert "kind not registered".
- Document why the full Status().Update in the success branches is safe
(the object is freshly Get() each pass; the StartedAt write requeues).
- Note the fixed batch/v1.Job backoffLimit=2 and link the follow-up to
make it configurable (#2742).
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
Address review feedback from @lllamnyp on internal/crdinstall/install.go:
the deletion-protection label loop was untested — TestInstall_appliesAllCRDs
only checked that Install returned no error, so a refactor dropping the loop
would have shipped green and silently disabled VAP coverage on every Cozystack
CRD.
- Extend TestInstall_appliesAllCRDs to read each applied CRD back from the fake
client and assert platform.cozystack.io/no-delete=true is present.
- Add TestInstall_preservesExistingLabels: feeds in a CRD whose manifest already
carries app.kubernetes.io/managed-by=foo, asserts both that label and the
deletion-protection label end up on the applied object. Locks down the merge
contract in install.go:92-101 (the GetLabels()==nil branch is still covered
by the first test).
Verified the new assertions fail when the label-stamping loop is removed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Block accidental DELETE on critical platform objects (tenant-root,
cozy-system, LinstorCluster, CRDs, ClusterIssuers, cozystack-version
ConfigMap, cozystack-packages OCIRepository, tenant-root HelmRelease)
via a label-scoped ValidatingAdmissionPolicy. No webhook DaemonSet,
Service, TLS, or image — the API server evaluates the CEL expression
in-process. Requires Kubernetes 1.30+.
Bypass for legitimate deletes is two steps:
kubectl label <kind> <name> platform.cozystack.io/no-delete-
kubectl delete <kind> <name>
CRDs installed dynamically by the operator get the label stamped in
internal/crdinstall/install.go before the apply call.
Release note:
Added deletion-protection guardrail (ValidatingAdmissionPolicy) blocking
DELETE on cozystack platform objects labeled platform.cozystack.io/no-delete=true.
Bypass by removing the label first.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The HTTP-01 inheritance flow deadlocked at the route-Accepted step:
the apps/tenant chart correctly labelled the child namespace with
`namespace.cozystack.io/gateway=<owner>`, the parent Gateway's
allowedRoutes selector matched by label, but `collectHostnameClaims`
filtered routes by the static set `{tgw.Namespace} ∪ Spec.AttachedNamespaces`
— so a child route in a namespace reached purely via the inheritance
label was silently dropped. No per-listener Certificate, no per-app
HTTPS listener, no hostname match → `Accepted=False,
Reason=NoMatchingListenerHostname` indefinitely.
Caught by the new `child tenant's HTTPRoute attaches to parent's
Gateway via inheritance label` e2e bats — failed 3/3 retries on the
CI run for `afc7765e6`. The Go controller-level
TestReconcile_HTTP01CollectsHostnamesFromInheritingChildNamespaces
test pins the same contract at a faster feedback loop: an HTTPRoute
in a namespace labelled `namespace.cozystack.io/gateway=<owner>` but
NOT listed in Spec.AttachedNamespaces must still cause the
controller to render a per-listener HTTPS listener for its hostname.
The fix extends the `allowed` set in `collectHostnameClaims` to
also include namespaces matched by the gateway label selector, so the
hostname-claim collector and the Gateway's allowedRoutes selector
agree on which namespaces can attach. DNS-01 mode is unaffected — it
returns nil from `collectHostnameClaims` immediately and uses the
separate `collectInheritingChildApexes` path for SAN expansion.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
In DNS-01 mode, the parent's wildcard Certificate now covers every
inheriting child tenant's apex (`<child-apex>` + `*.<child-apex>`
as additional SANs) and the parent Gateway grows one wildcard HTTPS
listener per child apex referencing the same Certificate. Without
the expansion, a child route hostname like `harbor.alice.example.org`
fails to match the parent's listeners — `*.example.org` is a single-
label wildcard and `alice.example.org` is two labels deep — and the
TLS handshake fails the SNI check because the parent cert never
listed the child apex.
The controller discovers inheriting children by listing namespaces
labelled `namespace.cozystack.io/gateway = <tgw.Namespace>` (the
existing inheritance contract from Stage 1/3) and reading their
`namespace.cozystack.io/host` label, which apps/tenant chart writes
to the child apex. Children with no host label are skipped
(defensive); a child host equal to the parent apex is deduplicated
(cert-manager rejects duplicate dnsNames). HTTP-01 mode is
unchanged — its per-listener / per-cert flow already handles child
hostnames without any wildcard.
Listener naming: `https-child-<first-label>-<8-hex>`, where the hex
suffix is sha256(child-apex)[:8] so two children whose first DNS
label collides produce distinct listener names.
Cap: Gateway API hard-caps spec.listeners at 64 entries. The parent
uses 3 slots (http + https + https-apex) leaving ~61 wildcard
children per parent. Operators whose subtree fans out beyond that
must run the high-fanout branch on its own Gateway via
tenant.spec.gateway=true.
Tests (TDD red→green):
* TestReconcile_DNS01WildcardCertCoversInheritingChildApexes pins
the SAN list contains both `<child-apex>` and `*.<child-apex>`.
* TestReconcile_DNS01WildcardCertDeduplicatesChildApexEqualToParent
pins the dedupe path (child host == parent apex).
* TestReconcile_HTTP01WildcardCertNeverRendered pins the inverse:
HTTP-01 mode never renders a wildcard Certificate.
* TestReconcile_DNS01GatewayHasListenerPerChildApex pins the
listener-side expansion (one `*.<child-apex>` listener per
inheriting tenant, all referencing the parent's wildcard cert).
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The Gateway's HTTPS / TLS-passthrough listeners now match attached
namespaces by the namespace.cozystack.io/gateway label keyed on the
TenantGateway's own namespace, instead of a static
kubernetes.io/metadata.name In [list] whitelist. This restores
inheritance parity with the legacy ingress flow: child tenants of a
Gateway-owning ancestor publish through the ancestor's Gateway by
default — without their own LB IP / Certificate, and without
operator intervention to add each child by name to platform values.
Two writers populate the label:
* apps/tenant chart namespace.yaml — each tenant namespace gets a
label pointing at the nearest ancestor that owns a Gateway (self
if owning, inherited otherwise). The tenant Helm chart is the
source of truth for tenant-tree labels; the controller never
touches them (annotation-gated GC).
* cozystack-controller (ensureNamespaceLabels) — patches the label
onto every namespace in tgw.Spec.AttachedNamespaces, so cozy-*
system namespaces (cert-manager, monitoring, harbor, …) reach the
publishing Gateway alongside the tenant tree. Tracks its own
writes via the cozystack.io/gateway-attached-by annotation; on a
reconcile where an entry is removed from AttachedNamespaces, the
label is garbage-collected (annotation present + namespace no
longer expected → strip both label and annotation).
Also adds the gateway-attach label to tenant-root itself via
cozystack-basics/templates/tenant-root.yaml when
_cluster.gateway-enabled is true, so the Gateway in tenant-root's
own namespace passes its own selector for the controller-owned
http→https redirect HTTPRoute.
Tests (TDD, red→green):
* TestReconcile_HTTPSListenerUsesGatewayLabelSelector pins the new
MatchLabels shape on HTTPS listeners.
* TestReconcile_LabelsAttachedNamespaces pins the patch path: every
namespace in spec.AttachedNamespaces (plus tgw.Namespace) carries
namespace.cozystack.io/gateway=<owner> after reconcile.
* TestReconcile_LabelGCRemovesDroppedAttachedNamespaces pins the GC
path: removing an entry between reconciles strips the label.
* TestReconcile_LabelGCDoesNotStripHelmOwnedLabels pins the safety
contract: labels lacking the controller annotation (Helm-owned
tenant labels) are left alone.
* tenant-root_test.yaml adds the positive case for the conditional
label render under _cluster.gateway-enabled=true.
The HTTP listener (port 80) keeps the narrower static-name selector
— it allows only the tenant namespace plus cozy-cert-manager (for
HTTP-01 ACME challenges) and must NOT accept routes from app
namespaces or child tenants, since unattended routes on the HTTP
listener serve plaintext and leak credentials.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.
The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.
publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.
Security model -- seven independent layers, all admission-time gates
fail-closed:
1. Listener allowedRoutes namespace whitelist via the
kube-apiserver-written kubernetes.io/metadata.name label;
plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
passthrough listeners restrict allowedRoutes.kinds to
HTTPRoute / TLSRoute respectively.
2. cozystack-gateway-hostname-policy VAP on Gateway.
3. cozystack-gateway-attached-namespaces-policy VAP on Package --
blocks tenant-* in gateway.attachedNamespaces.
4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
writes from non-trusted callers.
5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
writes to namespace.cozystack.io/host from non-trusted callers.
6. Render-time fail in cozystack-basics for tenant-* in
gateway.attachedNamespaces.
7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
scoped to tenant-* namespaces.
Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).
Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.
Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.
Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.
Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.
Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.
Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Complete the strategy.backups.cozystack.io/Job driver as the generic,
app-agnostic counterpart to the Altinity driver: render the strategy
PodTemplateSpec, run it as a one-shot batch/v1.Job, poll its terminal
condition, and emit a Backup. Wire it into the existing BackupClass
dispatch via reconcileJob/reconcileJobRestore rather than a standalone
reconciler, matching the post-BackupClass backups API (the old draft
read BackupJob.Spec.StrategyRef/StorageRef, which no longer exist).
Template context follows the in-tree convention (.Application, .Release,
.Mode, .Parameters, .Backup); the JobSpec.Template doc and regenerated
CRD are updated to match. Restore re-renders the same template with
.Mode=restore; cross-namespace restore is unsupported, mirroring
Altinity.
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
The dashboard controller used to project ApplicationDefinitions
into a set of dashboard.cozystack.io CRDs (Sidebar, Factory,
Navigation, Breadcrumb, CustomFormsOverride, CustomFormsPrefill,
CustomColumnsOverride, MarketplacePanel, TableUriMapping,
CFOMapping, BreadcrumbInside) consumed by the legacy UI.
cozystack-ui now reads ApplicationDefinitions directly, so the
controller, its API types, the generated CRDs and the related
RBAC rules are no longer used.
Removes:
- internal/controller/dashboard
- api/dashboard/v1alpha1
- packages/system/cozystack-controller/definitions/dashboard.cozystack.io_*.yaml
- dashboard.cozystack.io wiring from cmd/cozystack-controller/main.go
- dashboard.cozystack.io RBAC rules from cozystack-controller and cozy-dashboard charts
- ./api/dashboard/... codegen path from hack/update-codegen.sh
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
helm-controller's XValidation rule rejects Strategy.Name=RetryOnFailure
with RetryInterval set alongside a Remediation entry, so a future
"for safety" reintroduction of Remediation{Retries: -1} would silently
break every HR. Pin the deliberate omission.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Avoid pointer aliasing across all generated HelmReleases. r.HelmReleaseMaxHistory
is set once at startup today, but a future mutation through the field would
silently affect every in-flight HelmReleaseSpec.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Chart template now uses `ne (toString …) ""` for the five helmRelease*
knobs, so integer 0 (notably MaxHistory=0 = unlimited history per Helm
semantics, documented in values.yaml) is no longer silently dropped.
Extract buildHelmReleaseSpec from Reconcile and add unit tests
asserting Interval, MaxHistory, Install/Upgrade timeouts,
Strategy=RetryOnFailure, RetryInterval, and ChartRef propagate from
PackageReconciler config to the generated HelmRelease.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The HelmRelease generation in package_reconciler hardcoded three values
that have caused recurring pain:
- Install.Timeout = 10m (line 214)
- Upgrade.Timeout = 10m (line 220)
- MaxHistory = unset (Helm default 5)
The 10m timeout in particular contradicts the per-Application timeout
work done in `pkg/config/config.go` and
`pkg/registry/apps/application/rest.go` (commit 7b146cbe), which lets
individual Applications override the install/upgrade timeout via
annotation. The PackageSource side had no equivalent — etcd works around
it by hand-rolling its HR YAML in `packages/apps/tenant/templates/etcd.yaml`
to set a 30m timeout, harbor has had multiple commits chasing the same
problem from the other side. This closes that gap.
New operator flags (each maps to a chart value with empty default;
operator uses its own default when value is empty):
- `--helmrelease-install-timeout` (default 10m) → Spec.Install.Timeout
- `--helmrelease-upgrade-timeout` (default 10m) → Spec.Upgrade.Timeout
- `--helmrelease-max-history` (default 5) → Spec.MaxHistory
Production behaviour unchanged. E2E and edge cases (cert rotation,
slow-installing charts) can override per-cluster without modifying
chart manifests.
Per-component overrides on `ComponentInstall` (mirroring the existing
`UpgradeCRDs` pattern) would be a strictly better fix for charts like
etcd that need an outlier timeout — left as a deliberate followup since
it touches the Package CR API.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The operator generated HelmReleases with `Install.Remediation{Retries: -1}`
and `Upgrade.Remediation{Retries: -1}`, which is the v2 way of saying
"retry forever, never remediate". Combined with the spec-level
`Interval: 5m`, this meant a failed install/upgrade waited a full 5
minutes before the next attempt — even when the underlying race that
caused the failure (e.g. a chart artifact not quite Ready, a HR
dependency reconciling out of order) cleared in seconds.
Audit of the last 30 successful PR runs found that every single run had
its `Install Cozystack` step's `kubectl wait hr/seaweedfs-system` time
out at the 2-minute mark and recover only after an inline
`flux reconcile --force` workaround. Same root cause: 5-minute retry
interval was longer than the test's wait window. That workaround can be
removed once this change lands.
Switch to `Strategy.Name=RetryOnFailure` with `RetryInterval` exposed via
a new `--helmrelease-retry-interval` operator flag (default 30s):
- Functionally equivalent to the previous configuration ("retry forever
on failure") since `Retries: -1` meant remediation never fired anyway.
- Decouples retry-on-failure timing from `spec.Interval` so failed
releases recover at 30s without polling healthy releases at the same
cadence (which would multiply controller load by ~10x for no benefit).
- Helm chart exposes `cozystackOperator.helmReleaseRetryInterval`,
defaulting to empty (operator uses its own 30s default).
- Same flag pattern as the existing `--helmrelease-interval` (5m default),
also added in this PR via cherry-pick.
Verified: go build green, go test ./internal/operator/... pass, helm
template renders both flags conditionally on values.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).
E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.
Production defaults are unchanged. The override is opt-in per install.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Add an opt-in UpgradeCRDs field to ComponentInstall that maps to
HelmRelease.Spec.Upgrade.CRDs, allowing a PackageSource component to
declare how Flux should handle CRDs from the chart's crds/ directory
on upgrade.
The helm-controller default on upgrade is Skip, which means new CRDs
added between chart versions never reach existing clusters and must be
applied manually. Setting upgradeCRDs: CreateReplace makes Flux apply
new CRDs declaratively with the chart.
Allowed values are restricted to Skip, Create, CreateReplace via a
kubebuilder enum marker. Empty / unset preserves the existing Flux
default, so all existing PackageSource resources keep working.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Replace rename-only wording on spec.targetApplicationRef form labels so
users discover that the field also supports restoring into a different
application, not just renaming.
- Drop VMInstance-specific keys from the spec.options form label; the
restore options are driver-specific, so hardcoding one driver's keys is
misleading once other drivers land.
- Render a single "Same as backup" fallback for an omitted
spec.targetApplicationRef on the details page instead of the noisy
"-.-/-" produced by concatenating three missing path fallbacks.
- De-duplicate the time-block / time-icon / time-value component IDs
within the restorejob-details factory by prefixing them with
created-, started-at-, and completed-at-.
- Mark non-CRD-backed stock-project-factory-*-details sidebars (kube-*,
plan, backupjob, backup, restorejob) as static in
upsertMultipleSidebars so they pick up consistent managed-by labels
instead of being left unmanaged.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Replace http.DefaultClient with a package-level *http.Client with an
explicit 10-second timeout. Avoids sharing the process-wide default
transport with other libraries.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Pass bucket names from BucketClaim.Status.BucketName into the PromQL
query as a bucket=~"name1|name2" filter. This prevents O(N²) load
where N WorkloadMonitors each fetch all buckets globally.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Wrap resolvePrometheusURL and queryAllBucketMetrics in a
len(bucketClaimList.Items) > 0 guard. Avoids unnecessary namespace
GET and HTTP request to Prometheus on every reconcile of non-bucket
WorkloadMonitors (postgres, redis, kubernetes, etc.).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace 2×N per-bucket HTTP requests with a single query that fetches
all SeaweedFS bucket size metrics at once:
{__name__=~"SeaweedFS_s3_bucket_(size|physical_size)_bytes"}
Results are keyed by bucket name in memory, then looked up per
BucketClaim. This reduces HTTP round-trips from 2N+1 to 2 (one
namespace lookup + one Prometheus query) regardless of bucket count.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Move resolvePrometheusURL call before the BucketClaim loop and pass
the URL as parameter. Avoids redundant namespace lookups and reduces
reconcile time from 2×N+1 to 2×N HTTP calls (where N = BucketClaims).
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Change queryPrometheusMetric to return (int64, bool) so callers can
emit s3-storage-bytes=0 for empty buckets while omitting the field
entirely when monitoring is not configured.
Also:
- Add RBAC marker for core/namespaces GET (used by resolvePrometheusURL)
- Log namespace read errors instead of silently returning empty URL
- Add TestQueryPrometheusMetricZeroValue test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Bucket existence is already tracked via LifetimeHours. The s3-buckets
count resource produced a meaningless "s3-buckets-Hours" fallback type
in billing. Only storage size metrics (s3-storage-bytes,
s3-physical-storage-bytes) are now set on bucket Workloads.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Initialize workload.Labels map inside CreateOrUpdate mutate function
to prevent nil map panic when existing Workload has no labels
- Use strconv.ParseFloat instead of resource.ParseQuantity for
Prometheus metric values, which may use scientific notation
(e.g. "1.048576e+06") that ParseQuantity does not support
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Replace static --prometheus-url flag with dynamic resolution from
namespace.cozystack.io/monitoring label. Each tenant namespace knows
which tenant hosts its monitoring stack, so the controller constructs
the vmselect URL automatically. This correctly handles multi-tenant
setups where different tenants may use different monitoring instances.
- Remove PrometheusURL field from WorkloadMonitorReconciler struct
- Remove --prometheus-url flag and prometheusUrl chart value
- Add resolvePrometheusURL() that reads namespace label
- queryPrometheusMetric() now accepts prometheusBaseURL as parameter
- Add tests for resolvePrometheusURL with and without label
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Set replicas/minReplicas to 0 in bucket WorkloadMonitor (buckets
have no pods, minReplicas=1 would mark monitor as not operational)
- Remove dead IsNotFound check on List (List returns empty, not 404)
- Trim trailing slash from PrometheusURL before path append
- Add strings import for TrimRight
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
- Check resp.StatusCode before parsing Prometheus response
- Limit response body read to 1 MB via io.LimitReader
- Use strings.HasPrefix in test instead of fragile slice indexing
- Add TestQueryPrometheusMetricServerError for 500 responses
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_physical_size_bytes alongside the logical
size metric. Physical size includes all replicas and reflects actual
disk usage, while logical size reflects what the user stored.
Refactor queryBucketSizeBytes into generic queryPrometheusMetric
to reuse for both metrics.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Query SeaweedFS_s3_bucket_size_bytes from a Prometheus-compatible API
to populate s3-storage-bytes resource on bucket Workloads. The
Prometheus URL is configurable via --prometheus-url flag. When set,
bucket WorkloadMonitors are requeued every 60s to keep sizes current.
When Prometheus is not configured, buckets still get tracked with
s3-buckets=1 for existence-based billing.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
Add reconcileBucketClaimForMonitor() that watches COSI BucketClaim
objects and creates Workload CRDs with s3-buckets resource, following
the same pattern as PVC and Service reconcilers. This enables the
billing pipeline to discover and track S3 buckets per tenant.
Changes:
- Add COSI API types dependency (container-object-storage-interface-api)
- Register cosiv1alpha1 scheme in controller main
- Add BucketClaim watch in SetupWithManager
- Add BucketClaim list + reconcile in Reconcile loop
- Add RBAC annotation for objectstorage.k8s.io/bucketclaims
- Add unit tests for BucketClaim reconciliation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
## What this PR does
Adds two features to improve application observability in the dashboard,
plus a bug fix:
### 1. WorkloadsReady condition on Application status
- Queries WorkloadMonitor resources to determine if all application pods
are running
- Exposes `WorkloadsReady` as a separate condition alongside `Ready`
- `Ready` continues to reflect HelmRelease state only — no override — to
preserve backward compatibility with existing tooling (kubectl wait,
GitOps health checks) and avoid false-negative Ready=False during normal
startup windows
- Handles three states: operational, not operational, unknown (pending
reconciliation)
- Fails open on WorkloadMonitor query errors (prefers availability)
- Integrates with Application Watch to emit MODIFIED events on
WorkloadMonitor changes
- Registers cozystack.io/v1alpha1 types in API server scheme with
informer cache
### 2. Events tab in dashboard
- Shows Kubernetes Events scoped to the application's namespace
- Uses status.namespace for Tenant applications (consistent with
Resource Quotas tab)
- Includes both lastTimestamp and eventTime columns for Kubernetes
version compatibility
### 3. Bug fix: WorkloadMonitor Operational status persistence
- Operational field was written to stale `monitor` variable instead of
`fresh` inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
### Release note
```release-note
[dashboard] Added Events tab to application detail pages showing namespace-scoped Kubernetes Events
[application] Added WorkloadsReady condition exposing aggregated WorkloadMonitor status
[workloadmonitor] Fixed bug where Operational status was never persisted to the cluster
```
core/v1 Events populate .lastTimestamp and .firstTimestamp but leave
.eventTime null; events.k8s.io/v1 Events do the opposite. The previous
column bound to .eventTime alone and rendered 'Invalid Date' for every
Helm-generated event.
Extend createTimestampColumn with an optional second jsonPath that is
encoded as a nested reqsJsonPath fallback in the template, and use
.eventTime → .firstTimestamp for Event Time and .lastTimestamp →
.eventTime for Last Seen so both APIs render correctly.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Inside the RetryOnConflict block, derive the operational status from
fresh.Spec.MinReplicas instead of the stale monitor.Spec.MinReplicas
so that concurrent spec updates observed by the retry are respected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add two features to improve application observability in the dashboard:
1. WorkloadsReady condition on Application status
- Query WorkloadMonitor resources to determine if all pods are running
- Expose WorkloadsReady as a separate condition alongside Ready
- Ready continues to reflect HelmRelease state only (no override) to
preserve backward compatibility with tooling and avoid false-negatives
during normal startup windows when pods are still coming up
- Handle three states: operational, not operational, unknown (pending)
- Fail-open on WorkloadMonitor query errors
- Integrate with Application Watch to emit MODIFIED events on
WorkloadMonitor changes (with initial-events-end safety)
- Register cozystack.io/v1alpha1 types in API server scheme
2. Events tab in dashboard
- Show Kubernetes Events scoped to application namespace
- Use status.namespace for Tenant applications
- Include both lastTimestamp and eventTime columns for K8s compat
3. Bug fix: WorkloadMonitor Operational status persistence
- Operational was written to stale 'monitor' instead of 'fresh'
inside RetryOnConflict, so it was never persisted to the cluster
Closes#2359Closes#2360
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
myasnikovdaniil
