Commit Graph
77 Commits
Author SHA1 Message Date
Aleksei Sviridkin fb070ffa21 feat(hack): rewrite check-readiness as a Go command with test suite (#2755)
## What this PR does

Replaces the `hack/check-readiness.sh` diagnostic with a Go command at
`cmd/check-readiness`, integrated into the build the same way as
`cozypkg`:

- `make check-readiness` builds `_out/bin/check-readiness`, with
`Version`
  injected via `-ldflags`.
- The tool shells out to `kubectl` and reports only **non-ready**
cozystack /
  Flux / Kubernetes resources, parsed from status conditions (not column
heuristics). Flags: `--wait`/`--timeout`, `--watch`, `--core`,
`--parallel`,
  `--verbose`, `-n`/`-l` scoping, `--no-color`, `--version`.

Adds a black-box golden test suite under `test/check-readiness/`:

- A mock `kubectl` serves canned fixtures; the suite diffs
stdout/stderr/exit
against golden files across 30 scenarios (arg parsing,
condition/phase/suspend
handling, CRD-not-installed, kubectl arg propagation, wait/timeout, edge
  cases) plus a PTY test for color output.
- `make test-check-readiness` runs it; wired into `make unit-tests`.

Supersedes #2740, which added the same feature set to the bash script.

### Release note
```release-note
feat(hack): replace check-readiness.sh with a `check-readiness` Go command (build with `make check-readiness`); reports non-ready cozystack/Flux/Kubernetes resources with --wait/--watch/--core/--parallel/--verbose and namespace/label scoping.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

## Release Notes

* **New Features**
* Added `check-readiness` command for verifying Kubernetes resource
readiness status across namespaces
* Supports multiple execution modes: one-shot, wait with timeout, and
watch with configurable intervals
* Includes namespace filtering, label selectors, parallel resource
fetching, and color output control

* **Tests**
* Added comprehensive test suite validating command behavior across
various scenarios

<!-- review_stack_entry_start -->

[![Review Change
Stack](https://storage.googleapis.com/coderabbit_public_assets/review-stack-in-coderabbit-ui.svg)](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2755?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)

<!-- review_stack_entry_end -->

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-06-09 23:52:12 +03:00
Andrey Kolkov f17ead3d26 feat(platform): add default backupclass
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-28 13:49:59 +04:00
Myasnikov Daniil f85ec76e19 feat(hack): rewrite check-readiness as a Go command with test suite
Replace hack/check-readiness.sh with a Go command at cmd/check-readiness,
wired into the build like cozypkg (`make check-readiness`, Version via
-ldflags). The tool shells out to kubectl and reports only non-ready
cozystack / Flux / Kubernetes resources parsed from status conditions,
supporting --wait/--timeout, --watch, --core, --parallel, --verbose,
-n/-l scoping, --no-color and --version.

Add a black-box golden test suite under test/check-readiness driven by a
mock kubectl over fixtures; `make test-check-readiness` runs it and it is
wired into `make unit-tests`.

Supersedes #2740, which added the same feature set to the bash script.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-27 01:49:49 +05:00
Aleksei Sviridkin d2e851f63b feat: Gateway API support via Cilium
Replace ingress-nginx as the default ingress layer for cozystack-native
services with Cilium Gateway API, materialised per-tenant via a new
TenantGateway CRD reconciled by cozystack-controller.

The extra/gateway chart renders one gateway.cozystack.io/v1alpha1
TenantGateway per tenant; the controller owns the resulting Gateway,
per-tenant Issuer, Certificate(s), and the http->https redirect
HTTPRoute. This avoids the helm-vs-controller race on
Gateway.spec.listeners when route-driven listeners are added.

publishing.certificates.solver selects HTTP-01 (default; per-app HTTPS
listener + per-app Certificate) or DNS-01 (opt-in; single wildcard
Certificate per apex; providers: cloudflare, route53, digitalocean,
rfc2136). HTTP-01 keeps zero-config per-app onboarding; DNS-01 saves
Let's Encrypt budget on apex-shared clusters.

Security model -- seven independent layers, all admission-time gates
fail-closed:

  1. Listener allowedRoutes namespace whitelist via the
     kube-apiserver-written kubernetes.io/metadata.name label;
     plain-HTTP listener is strictly narrower than HTTPS, and HTTPS /
     passthrough listeners restrict allowedRoutes.kinds to
     HTTPRoute / TLSRoute respectively.
  2. cozystack-gateway-hostname-policy VAP on Gateway.
  3. cozystack-gateway-attached-namespaces-policy VAP on Package --
     blocks tenant-* in gateway.attachedNamespaces.
  4. cozystack-tenant-host-policy VAP on Tenant -- blocks spec.host
     writes from non-trusted callers.
  5. cozystack-namespace-host-label-policy VAP on Namespace -- blocks
     writes to namespace.cozystack.io/host from non-trusted callers.
  6. Render-time fail in cozystack-basics for tenant-* in
     gateway.attachedNamespaces.
  7. cozystack-route-hostname-policy VAP on HTTPRoute / TLSRoute,
     scoped to tenant-* namespaces.

Within-apex cross-namespace hostname conflict is resolved at reconcile
time by a HostnameConflict condition (cozy-* namespace wins).

Five reconcile paths refuse to rewrite pre-existing objects with the
controller-derived name but no OwnerReference back to the
TenantGateway: Gateway, redirect HTTPRoute, per-tenant Issuer, wildcard
Certificate, per-listener Certificate.

Derived-apex tenants (tenants whose host is a subdomain of a parent
tenant's) auto-enable Gateway when the platform-level gateway.enabled
is on.

Bonus fix: pkg/registry/apps/application/rest.go was ignoring the
createValidation and deleteValidation callbacks that genericapiserver
hands to every storage method, silently bypassing admission for Create
and Delete on every apps.cozystack.io resource (Tenant, MariaDB,
Postgres, Kubernetes, ...). Aligned with the existing updateValidation
pattern.

Always-on changes: cilium.envoy.enabled + cilium.gatewayAPI.enabled
flip to true (extra cilium-envoy DaemonSet, ~100 MB RAM/node idle),
and the cozystack-api admission-chain fix. Everything else is gated
behind gateway.enabled=false and tenant.spec.gateway=false; existing
clusters see no behavioural change until an operator opts in.

Tests: 60+ Go cases in internal/controller/tenantgateway/, admission-
chain regression tests in pkg/registry/apps/application/, helm-unittest
across 7 charts (extra/gateway, core/platform, apps/tenant,
system/cozystack-basics, system/cert-manager-issuers,
system/cozystack-api, system/dashboard), and 14 e2e bats scenarios in
hack/e2e-apps/gateway.bats. New make test-controllers target wired
into pull-requests.yaml.

Deps: sigs.k8s.io/gateway-api v1.4.1, cert-manager v1.17.4.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-05-25 17:07:02 +03:00
Andrey Kolkov d80f67e799 feat(platform): add implementation of backup-strategy controller for etcd
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-22 14:35:08 +04:00
Andrei KvapilandClaude 4d5486cd84 chore(controller): remove dead dashboard controller
The dashboard controller used to project ApplicationDefinitions
into a set of dashboard.cozystack.io CRDs (Sidebar, Factory,
Navigation, Breadcrumb, CustomFormsOverride, CustomFormsPrefill,
CustomColumnsOverride, MarketplacePanel, TableUriMapping,
CFOMapping, BreadcrumbInside) consumed by the legacy UI.

cozystack-ui now reads ApplicationDefinitions directly, so the
controller, its API types, the generated CRDs and the related
RBAC rules are no longer used.

Removes:
- internal/controller/dashboard
- api/dashboard/v1alpha1
- packages/system/cozystack-controller/definitions/dashboard.cozystack.io_*.yaml
- dashboard.cozystack.io wiring from cmd/cozystack-controller/main.go
- dashboard.cozystack.io RBAC rules from cozystack-controller and cozy-dashboard charts
- ./api/dashboard/... codegen path from hack/update-codegen.sh

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-05-20 16:33:21 +02:00
Andrey Kolkov 92930ad1aa feat(platform): add implementation of backup-strategy controller for foundationdb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-13 09:49:19 +04:00
Andrey Kolkov 62605a198b feat(platform): implement backup strategy for mariadb
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-12 13:56:29 +04:00
Andrey Kolkov ed5ae16e36 feat(platform): implement backup strategy for postgres application
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-05-07 13:12:54 +04:00
Myasnikov Daniil d0e4b4dbe5 refactor(operator): extract parsePositiveDuration into testable helper
The previous closure inside main() couldn't be tested because every error
path called os.Exit(1). Lifting it to a package-level function returning
(time.Duration, error) lets a table test exercise happy paths and rejections
("0s", "-5m", "5x", empty input) directly. main() retains the os.Exit
behaviour via a thin wrapper.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-04 22:13:29 +05:00
Myasnikov Daniil 1dc33dff79 fix(operator): reject non-positive HelmRelease duration flags
time.ParseDuration accepts "0s" and negative values, but Flux
HelmRelease.Interval / Install.Timeout / Strategy.RetryInterval require
strictly positive durations to function. Validate at startup so a
misconfigured flag fails fast instead of propagating into every
generated HelmRelease.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-05-04 11:41:35 +05:00
Myasnikov Daniil e410b02e1d feat(operator): expose Install/Upgrade timeouts and MaxHistory as flags
The HelmRelease generation in package_reconciler hardcoded three values
that have caused recurring pain:

- Install.Timeout = 10m (line 214)
- Upgrade.Timeout = 10m (line 220)
- MaxHistory = unset (Helm default 5)

The 10m timeout in particular contradicts the per-Application timeout
work done in `pkg/config/config.go` and
`pkg/registry/apps/application/rest.go` (commit 7b146cbe), which lets
individual Applications override the install/upgrade timeout via
annotation. The PackageSource side had no equivalent — etcd works around
it by hand-rolling its HR YAML in `packages/apps/tenant/templates/etcd.yaml`
to set a 30m timeout, harbor has had multiple commits chasing the same
problem from the other side. This closes that gap.

New operator flags (each maps to a chart value with empty default;
operator uses its own default when value is empty):

- `--helmrelease-install-timeout` (default 10m) → Spec.Install.Timeout
- `--helmrelease-upgrade-timeout` (default 10m) → Spec.Upgrade.Timeout
- `--helmrelease-max-history` (default 5)        → Spec.MaxHistory

Production behaviour unchanged. E2E and edge cases (cert rotation,
slow-installing charts) can override per-cluster without modifying
chart manifests.

Per-component overrides on `ComponentInstall` (mirroring the existing
`UpgradeCRDs` pattern) would be a strictly better fix for charts like
etcd that need an outlier timeout — left as a deliberate followup since
it touches the Package CR API.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:44:06 +05:00
Myasnikov Daniil c4c70e0e18 feat(operator): switch HelmRelease retries to Strategy=RetryOnFailure with configurable RetryInterval
The operator generated HelmReleases with `Install.Remediation{Retries: -1}`
and `Upgrade.Remediation{Retries: -1}`, which is the v2 way of saying
"retry forever, never remediate". Combined with the spec-level
`Interval: 5m`, this meant a failed install/upgrade waited a full 5
minutes before the next attempt — even when the underlying race that
caused the failure (e.g. a chart artifact not quite Ready, a HR
dependency reconciling out of order) cleared in seconds.

Audit of the last 30 successful PR runs found that every single run had
its `Install Cozystack` step's `kubectl wait hr/seaweedfs-system` time
out at the 2-minute mark and recover only after an inline
`flux reconcile --force` workaround. Same root cause: 5-minute retry
interval was longer than the test's wait window. That workaround can be
removed once this change lands.

Switch to `Strategy.Name=RetryOnFailure` with `RetryInterval` exposed via
a new `--helmrelease-retry-interval` operator flag (default 30s):

- Functionally equivalent to the previous configuration ("retry forever
  on failure") since `Retries: -1` meant remediation never fired anyway.
- Decouples retry-on-failure timing from `spec.Interval` so failed
  releases recover at 30s without polling healthy releases at the same
  cadence (which would multiply controller load by ~10x for no benefit).
- Helm chart exposes `cozystackOperator.helmReleaseRetryInterval`,
  defaulting to empty (operator uses its own 30s default).
- Same flag pattern as the existing `--helmrelease-interval` (5m default),
  also added in this PR via cherry-pick.

Verified: go build green, go test ./internal/operator/... pass, helm
template renders both flags conditionally on values.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:40:13 +05:00
Myasnikov Daniil e197aba625 feat(operator): make HelmRelease Interval configurable, override to 30s in E2E
Adds --helmrelease-interval (default 5m, matching today's behaviour) to the
operator and a corresponding cozystackOperator.helmReleaseInterval value
on the cozy-installer chart (default empty -> no flag rendered ->
operator default 5m applies).

E2E install sets it to 30s. Rationale: with the 5m default, dependency-
blocked HRs (waiting on cert-manager webhooks, CRDs) are requeued only
every 5 min, producing an 8-10 min dead zone in the E2E install where
the fast pack of HRs is Ready but the slow tail hasn't been retried yet.
30s collapses the gap.

Production defaults are unchanged. The override is opt-in per install.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-04-28 12:36:28 +05:00
Andrei KvapilandClaude 04cc1633be fix(kube-ovn): scope kubeovn-plunger cache and RBAC to its namespace
The kubeovn-plunger controller-runtime cache attempted cluster-wide
list/watch on Deployments and Pods, which the namespace-scoped Role
cannot satisfy. Additionally, the deployments rule relied on
resourceNames, which does not restrict list/watch verbs and left the
permission effectively unusable.

Scope the manager cache to the kube-ovn namespace so list/watch hit
the namespaced API, and drop resourceNames from the deployments rule.
Wire --kube-ovn-namespace and --ovn-central-name through the
Deployment so values are actually consumed.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-04-21 17:13:27 +02:00
ZverGuyandClaude 7f7f4b218d refactor(controller): resolve Prometheus URL from namespace labels
Replace static --prometheus-url flag with dynamic resolution from
namespace.cozystack.io/monitoring label. Each tenant namespace knows
which tenant hosts its monitoring stack, so the controller constructs
the vmselect URL automatically. This correctly handles multi-tenant
setups where different tenants may use different monitoring instances.

- Remove PrometheusURL field from WorkloadMonitorReconciler struct
- Remove --prometheus-url flag and prometheusUrl chart value
- Add resolvePrometheusURL() that reads namespace label
- queryPrometheusMetric() now accepts prometheusBaseURL as parameter
- Add tests for resolvePrometheusURL with and without label

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-04-17 10:58:47 +03:00
ZverGuyandClaude 0aab19f500 feat(controller): add SeaweedFS bucket size metrics via Prometheus
Query SeaweedFS_s3_bucket_size_bytes from a Prometheus-compatible API
to populate s3-storage-bytes resource on bucket Workloads. The
Prometheus URL is configurable via --prometheus-url flag. When set,
bucket WorkloadMonitors are requeued every 60s to keep sizes current.

When Prometheus is not configured, buckets still get tracked with
s3-buckets=1 for existence-based billing.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-04-17 10:58:47 +03:00
ZverGuyandClaude 92286213ab feat(controller): add BucketClaim support to WorkloadMonitorReconciler
Add reconcileBucketClaimForMonitor() that watches COSI BucketClaim
objects and creates Workload CRDs with s3-buckets resource, following
the same pattern as PVC and Service reconcilers. This enables the
billing pipeline to discover and track S3 buckets per tenant.

Changes:
- Add COSI API types dependency (container-object-storage-interface-api)
- Register cosiv1alpha1 scheme in controller main
- Add BucketClaim watch in SetupWithManager
- Add BucketClaim list + reconcile in Reconcile loop
- Add RBAC annotation for objectstorage.k8s.io/bucketclaims
- Add unit tests for BucketClaim reconciliation

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-04-17 10:58:47 +03:00
Andrey Kolkov 15d319bfa1 feat(backup): no manual actions required to perform restorejob for
VMInstance

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-03-31 12:01:04 +04:00
Aleksei SviridkinandClaude 4843a617bc fix(operator): skip PackageSource on empty URL, add comprehensive tests
Guard PackageSource creation on platformSourceURL != "" to avoid
dangling SourceRef when no Flux source resource exists.

Add tests covering all new and modified functions:
- parsePlatformSourceURL (OCI, HTTPS, SSH, empty, unsupported scheme)
- parseRefSpec (single/multi values, whitespace, equals in value,
  missing equals, empty key/value, trailing comma)
- validateOCIRef / validateGitRef (valid keys, invalid keys, format)
- generateOCIRepository / generateGitRepository (with ref, no ref,
  invalid ref)
- installPlatformPackageSource variant valuesFiles validation
- installPlatformPackageSource custom name

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 18:31:49 +03:00
Aleksei SviridkinandClaude 8b9a11360e fix(operator): skip PackageSource creation when platform source URL is empty
When platformSourceURL is empty no Flux source resource is created, so
creating a PackageSource that references it would leave a dangling
SourceRef in a permanent error state. Guard the creation block on
platformSourceURL != "".

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 18:29:55 +03:00
Aleksei SviridkinandClaude d0a6ddd782 refactor(operator): reduce variant duplication in installPlatformPackageSource
Replace repetitive Variant struct literals with a loop over variant
data, making it easier to add new variants and reducing copy-paste
errors.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 18:25:14 +03:00
Aleksei SviridkinandClaude 4821f025fc fix(operator): correct default platformSourceName, gate PackageSource on installCRDs
Change default --platform-source-name from "cozystack-packages" to
"cozystack-platform" to match the value passed by the Helm template.

Gate PackageSource creation on --install-crds flag: when the operator
does not manage CRDs, the PackageSource CRD may not exist yet, so
skip creation and let an external process handle it.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 18:04:50 +03:00
Aleksei SviridkinandClaude dbfdbc8298 fix(installer): check parsePlatformSourceURL error, wait for PackageSource in E2E
Explicitly check error from parsePlatformSourceURL instead of relying on
the implicit guarantee that installPlatformSourceResource already checked
it. This prevents latent bugs if startup order is ever restructured.

Add wait for platform PackageSource existence in E2E test before creating
Package resource, preventing flaky failures when operator startup is slow.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 17:57:45 +03:00
Aleksei SviridkinandClaude 58e2b646be fix(operator): use SSA for PackageSource, separate context, unconditional creation
Switch installPlatformPackageSource to server-side apply (SSA) with
field manager, matching the pattern used in crdinstall. SSA is idempotent
and preserves metadata fields managed by other controllers.

Create PackageSource unconditionally (not only when platformSourceURL is
set), matching the previous Helm template behavior where PackageSource
was always created regardless of source URL configuration.

Use a dedicated context with its own 2-minute timeout for PackageSource
creation, separate from the platform source resource installation.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 17:54:33 +03:00
Aleksei SviridkinandClaude 8450830f06 fix(installer): add CRD wait in E2E, unit tests for PackageSource creation
Add explicit CRD wait (kubectl wait crd --for=condition=Established) in
E2E test before creating Package resources, preventing race condition
between operator CRD installation and resource creation.

Add unit tests for installPlatformPackageSource covering create, update,
and GitRepository sourceRef kind scenarios.

Document that hardcoded variant list is an intentional design choice
matching the previous Helm template behavior.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 17:50:09 +03:00
Aleksei SviridkinandClaude 0e8b6515af fix(installer): handle parsePlatformSourceURL error, restore variant validation
Check error from parsePlatformSourceURL instead of discarding it, preventing
silent fallback to OCIRepository on invalid URLs.

Move variant validation from deleted packagesource.yaml to
cozystack-operator.yaml template so invalid cozystackOperator.variant
values still fail at helm template/install time.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 17:42:31 +03:00
Aleksei SviridkinandClaude 655133b81c fix(installer): move PackageSource creation from Helm template to operator
Replace the Helm hook approach with programmatic PackageSource creation
in the operator startup sequence. Helm hooks are unsuitable for persistent
resources like PackageSource because before-hook-creation policy causes
cascade deletion of owned ArtifactGenerators during upgrades.

The operator now creates the platform PackageSource after installing CRDs
and the Flux source resource, using the same create-or-update pattern as
installPlatformSourceResource(). The sourceRef.kind is derived from the
platform source URL (OCIRepository for oci://, GitRepository for git).

Also fix stale comment in e2e test referencing deleted crds/ directory.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-19 17:38:23 +03:00
Aleksei SviridkinandClaude 92d261fc1e fix: address review findings in operator and tests
- Remove duplicate "Starting controller manager" log before install
  phases, keep only the one before mgr.Start()
- Rename misleading test "document without kind returns error" to
  "decoder rejects document without kind" to match actual behavior
- Document Helm uninstall CRD behavior in deployment template comment
- Use --health-probe-bind-address=0 consistently with metrics-bind
- Exclude all dotfiles in verify-crds diff, not just .gitattributes

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:49:57 +03:00
Aleksei SviridkinandClaude cecc5861af fix(operator): validate CRD apiVersion, respect SIGTERM during install
- Check both apiVersion and kind when validating embedded CRD manifests
  to prevent applying objects with wrong API group
- Move ctrl.SetupSignalHandler() before install phases so CRD and Flux
  installs respect SIGTERM instead of blocking for up to 2 minutes
- Replace custom contains/searchString helpers with strings.Contains

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:49:56 +03:00
Aleksei SviridkinandClaude 1ddbe68bc2 feat(operator): add --install-crds flag with embedded CRD manifests
Embed Package and PackageSource CRDs in the operator binary using Go
embed, following the same pattern as --install-flux. The operator applies
CRDs at startup using server-side apply, ensuring they are updated on
every operator restart/upgrade.

This addresses the CRD lifecycle concern: Helm crds/ directory handles
initial install, while the operator manages updates on subsequent
deployments.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-02-18 00:49:55 +03:00
Andrei Kvapil 88baceae2c [platform] Make cozystack-api reconciliation always use Deployment
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-02-14 01:59:40 +01:00
Timofei Larkin 000b5ff76c [backups]
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-04 15:50:23 +03:00
Timofei Larkin 75a4b8ecbd Merge branch 'feat/backup-work-consolidation' into feat/backup-recovery
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
2026-02-03 21:20:59 +04:00
Andrei KvapilandClaude af320a86a0 fix(telemetry): use APIReader instead of cached client in operator
The cozystack-operator's manager cache is filtered to only include
specific secrets and namespaces with certain labels. This caused
telemetry collection to fail because resources like kube-system
namespace, nodes, services, and PVs were not in the cache.

Switch to using mgr.GetAPIReader() which bypasses the cache and
queries the API server directly. This is appropriate for telemetry
since it only runs every 15 minutes.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-27 18:06:30 +01:00
Andrey Kolkov ffde02c992 fix setup cache for backupClasses
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 12:59:42 +04:00
Andrey Kolkov 2cca1bc8d8 add webhook for validation backupClass is immutable in BackupJob
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-22 12:30:41 +04:00
Andrei Kvapil b0b1a1f1f9 [platform] Add flux-plunger controller to fix HelmRelease errors (#1843)
## What this PR does

This PR introduces a new flux-plunger controller that automatically
fixes HelmRelease resources experiencing "has no deployed releases"
errors by cleaning up corrupted Helm release secrets.

### Problem

When Helm releases fail with "has no deployed releases" error, FluxCD
cannot recover automatically. This typically happens when:
- Helm release secrets are corrupted
- Failed upgrades leave the release in an inconsistent state
- Manual intervention is required to delete the problematic secret

### Solution

The flux-plunger controller:
1. Watches HelmRelease resources for "has no deployed releases" errors
2. Suspends the HelmRelease (with `flux-client-side-apply` field
manager)
3. Identifies and deletes the latest (problematic) Helm release secret
4. Tracks processed versions via annotation to prevent recursive
deletion
5. Unsuspends the HelmRelease to allow FluxCD to retry

**Key features:**
- Only deletes one secret at a time (prevents cascading deletions)
- Uses version tracking: if `latest+1 == processed`, skips (already
handled)
- Handles suspended HelmReleases: only unsuspends if it was suspended by
flux-plunger itself
- Gracefully handles invalid annotations by treating them as unprocessed

### Components

- **Controller:** `internal/controller/fluxplunger/flux_plunger.go`
- **Main:** `cmd/flux-plunger/main.go`  
- **Helm Chart:** `packages/system/flux-plunger/`
- **Platform Integration:** Added to `paas-full.yaml` bundle

### Release note

```release-note
[platform] Add flux-plunger controller to automatically fix HelmRelease resources with "has no deployed releases" error by cleaning up corrupted Helm secrets
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

## Release Notes

* **New Features**
* Introduced a new controller that automatically resolves HelmRelease
resource failures with built-in health probes and graceful recovery
procedures.
  * Enabled by default in system installations.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-20 01:56:00 +01:00
Andrei KvapilandClaude 987a74ae5a refactor(telemetry): split telemetry between operator and controller
Split telemetry collection between cozystack-operator and cozystack-controller:

cozystack-operator now collects cluster-level metrics:
- cozy_cluster_info (cozystack_version, kubernetes_version)
- cozy_nodes_count (os, kernel)
- cozy_cluster_capacity (cpu, memory, nvidia.com/* resources)
- cozy_loadbalancers_count
- cozy_pvs_count (driver, size)
- cozy_package_info (name, variant)

cozystack-controller now collects application-level metrics:
- cozy_application_count (kind) - counts HelmReleases per ApplicationDefinition

Other changes:
- Add pkg/version for build-time version injection via ldflags
- Remove --cozystack-version flag (version now embedded at build time)
- Remove bundle/oidc configuration from telemetry (replaced by package_info)
- Remove cozy_workloads_count metric (replaced by cozy_application_count)

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-19 13:52:15 +01:00
Andrei KvapilandClaude 2d1c8aae02 feat(platform): add flux-plunger controller
Add flux-plunger controller to automatically fix HelmRelease resources
with "has no deployed releases" error.

The controller watches HelmRelease resources and performs the following:
- Detects HelmRelease with "has no deployed releases" error
- Suspends the HelmRelease with flux-client-side-apply field manager
- Deletes the latest Helm release secret
- Updates annotation with processed version to prevent recursive deletion
- Unsuspends the HelmRelease to allow Flux to retry

Special handling for suspended HelmRelease:
- If suspend=true and latest+1==processed: removes suspend
- Otherwise: skips processing as suspended by external process

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-19 13:20:08 +01:00
Andrei KvapilandClaude 57c8cc26d4 refactor(api): rename CozystackResourceDefinition to ApplicationDefinition
Rename the CRD and all related types for better clarity:
- CozystackResourceDefinition -> ApplicationDefinition
- CozystackResourceDefinitionList -> ApplicationDefinitionList
- CozystackResourceDefinitionSpec -> ApplicationDefinitionSpec
- All related nested types updated accordingly

Updated components:
- API types and generated deepcopy code
- Controllers and reconcilers
- Dashboard, lineagecontrollerwebhook, crdmem packages
- CRD YAML definition and Helm chart
- All 25 cozyrds YAML manifests
- Migration scripts and documentation

Added migration 23 to remove old cozystack-resource-definition-crd HelmRelease.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 22:35:56 +01:00
Andrei KvapilandClaude 6dbfcbb93e feat(cozypkg): add cross-platform build targets with version injection
- Add pattern rule for building cozypkg binaries per platform
  (assets-cozypkg-<os>-<arch>) with checksums generation
- Add Version variable to cozypkg CLI injected via ldflags
- Split manifests into separate cozystack-crds.yaml and
  cozystack-operator.yaml files
- Update CI workflow to handle CRDs and operator as separate artifacts
- Update e2e tests and upload script for new manifest structure
- Suppress git describe stderr when no tags exist

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 17:18:01 +01:00
Andrei KvapilandClaude 99ac2fc710 refactor(installer): migrate installer to cozystack-operator
Remove legacy installer components (cozystack-assets-server,
installer.sh script, cozystack container image) in favor of
cozystack-operator based deployment.

Move migration scripts from scripts/migrations/ to
packages/core/platform/images/migrations/ for containerized execution.

Add grafana-dashboards image for centralized dashboard management.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-15 14:14:48 +01:00
Andrey Kolkov 5e15a75d89 [backups] add restore jobs controller
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-08 18:57:42 +03:00
05b2244b36 feat(operator): add secret replicator and reconciler improvements
Add namespace-based secret replication with label selector approach.
The implementation uses configurable secret name, namespace, and
target namespace selector. Cache filtering optimizes memory usage.

Co-Authored-By: Timofei Larkin <lllamnyp@gmail.com>
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-06 16:31:40 +01:00
Andrei Kvapil 5f36396ccc [platform] Replace Helm lookup with valuesFrom mechanism (#1787)
## What this PR does

Replaces Helm lookup functions with FluxCD valuesFrom mechanism for
passing configuration to HelmReleases. This provides cleaner config
propagation and eliminates the need for force reconcile controllers.

### Changes:

**Platform/Tenant charts:**
- Add Secret `cozystack-values` creation in platform chart (for
tenant-root and system namespaces)
- Add Secret `cozystack-values` creation in tenant chart (for child
namespaces)

**cozystack-api:**
- Add `valuesFrom` references to HelmRelease when creating applications
- Filter keys starting with `_` when returning Application specs
- Validate that user values don't contain `_` prefixed keys

**cozystack-controller:**
- Add validation that HelmRelease contains correct valuesFrom
configuration
- Remove `CozystackConfigReconciler` (no longer needed)
- Remove `TenantHelmReconciler` (no longer needed)

**Helm charts (40+ files):**
- Add helper templates in cozy-lib for `_cluster`/`_namespace` access
- Replace ConfigMap lookups with `.Values._cluster.*`
- Replace Namespace annotation lookups with `.Values._namespace.*`

### Architecture:

```
Secret cozystack-values (in each namespace)
├── _cluster: YAML with data from ConfigMaps (cozystack, cozystack-branding, cozystack-scheduling)
└── _namespace: YAML with namespace service references (etcd, host, ingress, monitoring, seaweedfs)

HelmRelease
└── spec.valuesFrom:
    ├── Secret/cozystack-values → _namespace → .Values._namespace
    └── Secret/cozystack-values → _cluster → .Values._cluster
```

### Release note

```release-note
[platform] Replace Helm lookup functions with FluxCD valuesFrom mechanism for configuration propagation
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Helm releases and namespaces now source centralized cluster/namespace
configuration via a new secret (cozystack-values), and many templates
read values from chart-provided _cluster/_namespace entries.

* **Bug Fixes**
* API now rejects application specs containing reserved keys prefixed
with "_" to prevent invalid configurations.

* **Refactor**
* Two background reconciler controllers were removed from startup,
simplifying controller initialization.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-01-05 17:53:33 +01:00
Andrei KvapilandClaude 2e61810547 refactor: replace Helm lookup with valuesFrom mechanism
Replace Helm lookup functions with FluxCD valuesFrom mechanism for
reading cluster and namespace configuration.

Changes:
- Create Secret cozystack-values in each namespace with values.yaml key
  containing _cluster and _namespace configuration as nested YAML
- Configure HelmReleases to read from this Secret via valuesFrom
  (valuesKey defaults to values.yaml, so it can be omitted)
- Update cozy-lib helpers to access config via .Values._cluster
- Add default values for required _cluster keys to ensure all fields exist
- Update Go code (cozystack-api and helm reconciler) to use new format

This eliminates the need for Helm lookup functions while maintaining
the same configuration interface for charts.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-01-05 16:10:55 +01:00
Andrey Kolkov bfafcaa3ab [backups] Implement Velero strategy controller
## What this PR does

This patch implements the Reconcile function for BackupJobs with a
Velero strategy ref.

### Release note

```release-note
[backups] Implement the Velero backup strategy controller.
```

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-01-04 12:55:24 +03:00
Andrei Kvapil c200a45824 [cozypkg] Add tool for managing Package and PackageSources (#1756)
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>

<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

## What this PR does


### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
[cozypkg] Add tool for managing Package and PackageSources
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
  * New cozy CLI with entrypoint and root command.
* Interactive "add" command to install packages from args or files, with
dependency resolution, topological install order, variant selection, and
progress feedback.
* "del" command to remove packages safely, including dependent analysis,
confirmation prompts, and safe delete ordering.
* "list" command to show available or installed packages, with optional
per-component output and concise status lines.
* "dependencies" command to emit Graphviz DOT graphs of
package/component relationships.
* Common flags: kubeconfig, file inputs, --installed and --components
where applicable.

<sub>✏️ Tip: You can customize this high-level summary in your review
settings.</sub>
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2025-12-30 11:56:31 +01:00
Andrei Kvapil f28d639aea [cozystack-operator] Add Package and PackageSource reconcilers
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2025-12-30 11:55:21 +01:00