Replace deprecated single-node VLogs CR with VLCluster (cluster mode)
for reliability and horizontal scalability.
Changes:
- Replace VLogs (v1beta1) with VLCluster (v1) using vlinsert/vlselect/vlstorage
- Update fluent-bit outputs to vlinsert-generic:9481
- Update Grafana datasource to vlselect:9471
- Update ExternalName service from vlogs-generic to vlinsert-generic
- Add VPA for all VLCluster components
- Update WorkloadMonitors for three-component architecture
- Add migration 35 to preserve old VLogs resources with keep annotation
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[platform] Fixed off-by-one error where the first required migration was always skipped.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Corrected migration range handling so upgrade steps run for the
intended version window, preventing skipped or duplicated migrations.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add helm.sh/resource-policy=keep annotation to the cozy-system Namespace
in the installer helm chart. This prevents Helm from deleting the
namespace when the HelmRelease is removed, which would otherwise destroy
all other HelmReleases within it.
Update the migration script to annotate the cozy-system namespace and
cozystack-version ConfigMap with helm.sh/resource-policy=keep before
generating the Package resource.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
[platform] Prevent cozystack-version configmap from deletion
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated deployment resource configuration to improve system
reliability by ensuring critical components are properly retained and
protected during system operations and maintenance activities.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
Adds OpenBAO (open-source Vault fork) as a new managed PaaS application
in Cozystack.
**Structure follows existing app patterns (qdrant, nats):**
- System chart with vendored upstream `openbao/openbao` (chart v0.25.3,
appVersion v2.5.0)
- App chart with standalone/HA mode switching based on replicas count
- TLS via cert-manager self-signed certificates per instance
- ApplicationDefinition, PackageSource, PaaS bundle entry
- E2E test with init/unseal workflow
**Key design decisions:**
- `replicas: 1` → standalone mode with file storage; `replicas > 1` → HA
with Raft integrated storage and retry_join with TLS peer verification
- TLS enabled by default — each instance gets a self-signed Certificate
with DNS SANs covering services and pod addresses
- `disable_mlock = true` in HCL config since default security context
drops IPC_LOCK capability
- Injector and CSI provider disabled (cluster-scoped components, not
safe per-tenant)
- No auto-init/unseal — OpenBAO requires manual initialization by design
- E2E test performs full lifecycle: deploy, wait for certificate + API,
init, unseal, verify readiness, cleanup
### Release note
```release-note
[apps] Add OpenBAO as a managed secrets management service with standalone and HA Raft modes, TLS enabled by default
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Release Notes
* **New Features**
* Added OpenBAO managed secrets management service with
high-availability and standalone deployment options
* Integrated monitoring and dashboards for operational visibility
* Enabled configurable external access and web UI
* Added automated snapshot backup capability
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Add a safety-net migration for users who already passed migrations 28/29
and still have orphaned -rd HelmReleases in cozy-system:
- ferretdb-rd (replaced by mongodb-rd, never had a dedicated migration)
- mysql-rd (migration 28 only handled user HRs)
- virtual-machine-rd (migration 29 only handled user HRs)
These HRs reference ExternalArtifacts that no longer exist after the
application renames (ferretdb→mongodb, mysql→mariadb,
virtual-machine→vm-disk+vm-instance), causing persistent errors.
Bump targetVersion from 33 to 34.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migrations 28 (mysql→mariadb) and 29 (virtual-machine→vm-disk+vm-instance)
only handled user-facing HelmReleases but left the system-level -rd
HelmReleases (mysql-rd, virtual-machine-rd) orphaned in cozy-system.
These HRs reference ExternalArtifacts that no longer exist, causing
persistent reconciliation failures.
Add cleanup steps to delete the orphaned -rd HRs and their Helm secrets.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
The protection-webhook is not part of the cozystack platform and should
not be managed by the migration script. Old services are now deleted
directly instead of being batched through the webhook disable/enable cycle.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 26 was using apps.cozystack.io/application.kind=Monitoring label
which is added by migration 22 and may not be present on v0.41.1 clusters.
Switch to cozystack.io/ui=true (guaranteed on old HRs) with field-selector
for exact name match.
Also remove redundant bundle enabled flags from migrate-to-version-1.0.sh
since the variant already determines them via its values file.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 27 failed with set -e when Piraeus CRDs did not exist on
clusters without linstor. Add existence check before annotating CRDs.
Also add name-pattern fallback for helm secret deletion, consistent
with migration 26.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
grep returns exit code 1 when no lines match. With set -euo pipefail,
this kills the script when all secrets are helm-release secrets or when
no matching resources exist. Wrap grep calls with { ... || true; }.
Also fix reconcile annotation in migration 29 to use RFC3339 timestamp
format instead of Unix epoch, which Flux v2 expects.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Migration 26 silently skipped namespace processing when kubectl
queries failed, leaving helm release secrets intact. This caused
helm to diff old vs new chart manifests during upgrade, deleting
VLogs/CNPG resources and their PVCs.
- Remove silent error suppression (2>/dev/null || true) from
namespace discovery and HR suspend commands
- Add fallback secret deletion by name pattern when label selector
does not match
- Add verification that all helm release secrets are deleted
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Check error from parsePlatformSourceURL instead of discarding it, preventing
silent fallback to OCIRepository on invalid URLs.
Move variant validation from deleted packagesource.yaml to
cozystack-operator.yaml template so invalid cozystackOperator.variant
values still fail at helm template/install time.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the Helm hook approach with programmatic PackageSource creation
in the operator startup sequence. Helm hooks are unsuitable for persistent
resources like PackageSource because before-hook-creation policy causes
cascade deletion of owned ArtifactGenerators during upgrades.
The operator now creates the platform PackageSource after installing CRDs
and the Flux source resource, using the same create-or-update pattern as
installPlatformSourceResource(). The sourceRef.kind is derived from the
platform source URL (OCIRepository for oci://, GitRepository for git).
Also fix stale comment in e2e test referencing deleted crds/ directory.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Remove the crds/ directory from the cozy-installer Helm chart. The operator
already installs embedded CRDs via server-side apply on every startup with
the --install-crds=true flag, making the Helm crds/ directory redundant.
Convert templates/packagesource.yaml to a Helm post-install/post-upgrade
hook so it is applied after the operator has started and installed CRDs.
Update codegen to write CRDs only to internal/crdinstall/manifests/ (single
source of truth) and update the Makefile to source build assets from there.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## What this PR does
Adds Harbor v2.14.2 as a managed tenant-level container registry service
in the PaaS bundle.
**Architecture:**
- Wrapper chart (`apps/harbor`) — HelmRelease, Ingress,
WorkloadMonitors, BucketClaim, dashboard RBAC
- Vendored upstream chart (`system/harbor`) from helm.goharbor.io
v1.18.2
- System chart (`system/harbor`) provisions PostgreSQL via CloudNativePG
and Redis via redis-operator
- ApplicationDefinition (`system/harbor-rd`) for dynamic `Harbor` CRD
registration
- PackageSource and paas.yaml bundle entry for platform integration
**Key design decisions:**
- Database and Redis provisioned via CPNG and redis-operator (not
internal Helm-based instances) for reliable day-2 operations
- Registry image storage uses S3 via COSI BucketClaim/BucketAccess from
namespace SeaweedFS
- Trivy vulnerability scanner cache uses PVC (S3 not supported by
vendored chart)
- Token CA key/cert persisted across upgrades via Secret lookup
- Per-component resource configuration (core, registry, jobservice,
trivy)
- Ingress with TLS via cert-manager, cloudflare issuer type handling,
proxy timeouts for large image pushes
- Auto-generated admin credentials persisted across upgrades
**E2E test:** Creates Harbor instance, verifies HelmRelease readiness,
deployment availability, credentials secret, service port, then cleans
up.
### Release note
```release-note
[harbor] Add managed Harbor container registry as a tenant-level service
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added Harbor container registry deployment with integrated Kubernetes
support, including database and cache layers.
* Enabled metrics monitoring via Prometheus integration.
* Configured dashboard management interface for Harbor administration.
* **Tests**
* Added end-to-end testing for Harbor deployment and verification.
* **Chores**
* Integrated Harbor into the platform's application package bundle.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- Replace pre-rendered static YAML application (`kubectl apply`) with
direct `helm upgrade --install` of the `packages/core/installer` chart
in E2E tests
- Remove CRD/operator artifact upload/download from CI workflow — the
chart with correct values is already present in the sandbox via
workspace copy and `pr.patch`
- Remove `copy-installer-manifest` Makefile target and its dependencies
## Test plan
- [ ] CI build job completes without uploading CRD/operator artifacts
- [ ] E2E `install-cozystack` step succeeds with `helm upgrade
--install`
- [ ] All existing E2E app tests pass
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* PR workflows now only keep the primary disk asset; publishing/fetching
of auxiliary operator and CRD artifacts removed.
* CRD manifests are produced by concatenation and a verify-crds check
was added to unit tests; file-write permissions for embedded manifests
tightened.
* **New Features**
* Operator can install CRDs at startup to ensure resources exist before
reconcile.
* E2E install now uses the chart-based installer flow.
* **Tests**
* Added comprehensive tests for CRD-install handling and manifest
writing.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Replace PVC-based registry storage with S3 via COSI BucketClaim/BucketAccess.
The system chart parses BucketInfo secret and creates a registry-s3 Secret
with REGISTRY_STORAGE_S3_* env vars that override Harbor's ConfigMap values.
- Add bucket-secret.yaml to system chart (BucketInfo parser)
- Remove storageType/size from registry config (S3 is now the only option)
- Use Harbor's existingSecret support for S3 credentials injection
- Add objectstorage-controller to PackageSource dependencies
- Update E2E test with COSI bucket provisioning waits and diagnostics
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
In v0.41.x the tenant-root Namespace and HelmRelease were applied via
kubectl apply with no Helm release tracking. In v1.0 these resources
are managed by the cozystack-basics Helm release. Without proper Helm
ownership annotations the install of cozystack-basics fails because
the resources already exist.
Add migration 31 that annotates and labels both the Namespace and
HelmRelease so Helm can adopt them, matching the pattern established
in migrations 22 and 27.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add postgres-operator and redis-operator to PackageSource dependsOn
to ensure CRDs are available before Harbor system chart deploys.
Make persistentVolumeClaim conditional to avoid empty YAML mapping
when using S3 storage without Trivy.
Increase E2E system HelmRelease timeout from 300s to 600s to account
for CPNG + Redis + Harbor bootstrap time on QEMU.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Move Harbor from packages/extra/ to packages/apps/ as it is a
self-sufficient end-user application, not a singleton tenant module.
Update bundle entry from system to paas accordingly.
Replace registry PVC storage with S3 via COSI BucketClaim/BucketAccess,
provisioned from the namespace's SeaweedFS instance. S3 credentials are
injected into the HelmRelease via valuesFrom with targetPath.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add Harbor v2.14.2 as a tenant-level managed service with per-component
resource configuration, ingress with TLS termination, and internal
PostgreSQL/Redis.
Includes:
- extra/harbor wrapper chart with HelmRelease, WorkloadMonitors, Ingress
- system/harbor with vendored upstream chart (helm.goharbor.io v1.18.2)
- harbor-rd ApplicationDefinition for dynamic CRD registration
- PackageSource and system.yaml bundle entry
- E2E test with Secret and Service verification
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Remove duplicate "Starting controller manager" log before install
phases, keep only the one before mgr.Start()
- Rename misleading test "document without kind returns error" to
"decoder rejects document without kind" to match actual behavior
- Document Helm uninstall CRD behavior in deployment template comment
- Use --health-probe-bind-address=0 consistently with metrics-bind
- Exclude all dotfiles in verify-crds diff, not just .gitattributes
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Use explicit apiextensions.k8s.io/v1 CRD GVK in waitForCRDsEstablished
instead of fragile objects[0].GroupVersionKind()
- Add TestInstall_crdNotEstablished for context timeout path
- Add --recursive to diff in verify-crds Makefile target
- Document why both crds/ and --install-crds exist in deployment template
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add --install-crds=true to cozystack-operator container args so the
operator applies embedded CRD manifests on startup via server-side apply.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The test-cluster target references non-existent hack/e2e-cluster.bats
file. Remove it and its dependency from the test target.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Helm installs crds/ contents before processing templates, resolving the
chicken-and-egg problem where PackageSource CR validation fails because
its CRD hasn't been registered yet.
- Move definitions/ to crds/ in the installer chart
- Remove templates/crds.yaml (Helm auto-installs from crds/)
- Update codegen script to write CRDs to crds/
- Replace helm template with cat for static CRD manifest generation
- Remove pre-apply CRD workaround from e2e test
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace pre-rendered static YAML application with direct helm chart
installation in e2e tests. The chart directory with correct values is
already present in the sandbox after pr.patch application.
- Remove CRD/operator artifact upload/download from CI workflow
- Remove copy-installer-manifest target from testing Makefile
- Use helm upgrade --install from local chart in e2e-install-cozystack.bats
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
## Summary
Add a new `cilium-kilo` networking variant that combines Cilium as the
CNI with Kilo as the WireGuard mesh overlay. This replaces the
standalone kilo PackageSource with a unified variant under the
networking source.
## Changes
- Add `cilium-kilo` variant to `networking.yaml` PackageSource with
proper component ordering and dependencies
- Add `values-kilo.yaml` for Cilium to disable host firewall when used
with Kilo
- Remove standalone `kilo.yaml` PackageSource (now integrated into
networking source)
- Switch Kilo image to official
`ghcr.io/cozystack/cozystack/kilo:v0.8.2`
- Remove unused `podCIDR`/`serviceCIDR` options and `--service-cidr`
flag from Kilo chart
Add a new networking variant that integrates Kilo with Cilium
pre-configured. Cilium is deployed with host firewall disabled and
enable-ipip-termination enabled, which are required for correct IPIP
encapsulation through Cilium's overlay.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Kilo is now integrated into the cilium-kilo networking variant instead
of being a separate package that users install manually.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## Summary
- Extend the installer chart to support generic and hosted Kubernetes
deployments via the existing `cozystackOperator.variant` parameter
(introduced in #2034)
- For `variant=generic`: render ConfigMaps (`cozystack`,
`cozystack-operator-config`) and an optional Platform Package CR —
resources previously required to be created manually before deploying on
non-Talos clusters
- Add variant validation in `packagesource.yaml` to fail fast on typos
- Publish the installer chart as an OCI Helm artifact
## Motivation
Deploying Cozystack on generic Kubernetes (k3s, kubeadm, RKE2) currently
requires manually creating ConfigMaps before applying the rendered
operator manifest. This change makes the installer chart variant-aware
so that:
1. `helm template -s` workflow continues to produce correct rendered
manifests
2. `helm install --set cozystackOperator.variant=generic` becomes a
viable single-command deployment path for generic clusters
3. Required ConfigMaps and optional Platform Package CR are generated
from values, eliminating manual steps
## OCI Helm chart publishing
The installer chart is now packaged and pushed to the OCI registry as
part of the `image` build target via `make chart`. A `.helmignore` file
ensures only chart-relevant files are included in the published
artifact.
## Test plan
- [ ] `helm template` with `variant=talos` (default) renders: operator +
PackageSource
- [ ] `helm template` with `variant=generic` renders: operator + 2
ConfigMaps + PackageSource
- [ ] `helm template` with `variant=generic` + `platform.enabled=true`
renders: + Package CR
- [ ] `helm template` with `variant=hosted` renders: operator +
PackageSource
- [ ] Invalid variant value produces a clear error message
- [ ] `make manifests` generates all asset files
- [ ] `helm package` produces a clean chart without build artifacts
Add a new "cilium" variant to the kilo PackageSource that deploys kilo
with --compatibility=cilium flag. This enables Cilium-aware IPIP
encapsulation, routing outer packets through Cilium's VxLAN overlay
instead of the host network.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Add migration script that converts VPC HelmRelease values from map
format to array format. The script discovers all VirtualPrivateCloud
HelmReleases, reads their values Secrets, and converts subnets using
yq. Idempotent: skips if subnets are already an array or null.
Bumps migration targetVersion from 30 to 31.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Replace `|| echo ""` with `|| true` to avoid newline bugs in variable
assignments. Switch `for x in $(cmd)` loops to `while read` for safer
iteration over kubectl output.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
