The post-delete hook selected PVCs with an unanchored name prefix
(qdrant-storage-<release>-). Because release names are qdrant-<app>, deleting
release "qdrant-test" matched not only its own PVC
(qdrant-storage-qdrant-test-0) but also a sibling "qdrant-test-2"'s PVC
(qdrant-storage-qdrant-test-2-0) in the same namespace — irreversibly deleting
the sibling's data (or wedging it in Terminating via pvc-protection).
Anchor the ordinal with grep -E "...-[0-9]+$" so only the release's own
data PVCs match. A release name that is a prefix of a sibling no longer
collides.
- cleanup-pvc.yaml: grep -E with anchored "-[0-9]+$".
- tests: assert the anchored selector and add a notMatchRegex guard against
regressing to the unanchored prefix.
- e2e: gate the post-delete wait on a NotFound result specifically, so a
transient API/auth error is not misread as "PVC reclaimed".
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The previous approach routed a release-unique label through
qdrant.persistence.additionalLabels, which the upstream chart renders into
the StatefulSet's volumeClaimTemplates[].metadata.labels. That field is
immutable on an existing StatefulSet, so for any already-deployed Qdrant the
API server would reject the upgrade and the <release>-system HelmRelease
(retries: -1) would retry-fail forever — permanently wedging reconciliation.
Label changes on a volumeClaimTemplate also never apply to PVCs that already
exist, so the cleanup hook could not reclaim those instances anyway.
Switch the post-delete hook to select PVCs by the deterministic StatefulSet
naming prefix (qdrant-storage-<release>-) instead. PVC names derive purely
from the release name, so this needs no StatefulSet change (upgrade-safe) and
also reclaims PVCs from instances created before the hook existed.
- qdrant.yaml: drop the additionalLabels passthrough entirely.
- hooks/cleanup-pvc.yaml: delete by name prefix via kubectl get -o name | grep.
- tests: assert the name-prefix command instead of the label passthrough.
- e2e: assert qdrant-storage-<name>-0 is gone after delete (async-waited).
BREAKING CHANGE: deleting a Qdrant app now permanently removes its data PVC
and the vector data it holds.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The .PHONY: test tweak is out of scope for the PVC cleanup fix; remove it
to keep this PR focused (flagged by review).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Алексей Артамонов <alexeyartamonov1987@gmail.com>
The StatefulSet data PVC (qdrant-storage-<release>-0) was left orphaned
after a Qdrant app CR was deleted. The vendored upstream chart does not
support persistentVolumeClaimRetentionPolicy, so add a post-delete PVC
cleanup hook (modeled on the MariaDB hook) and pass persistence.additionalLabels
through the HelmRelease so the data PVC carries a release-unique
app.kubernetes.io/instance label the hook selects on. The cleanup Job runs
non-root with a read-only root fs and dropped capabilities; failures are
logged but never block namespace teardown.
Signed-off-by: Алексей Артамонов <aleksei.artamonov@aenix.io>
Add the storageclass dynamic-dropdown x-cozystack-options annotation to the
storageClass field of the data apps (openbao, clickhouse, mariadb, redis,
harbor, opensearch, rabbitmq, mongodb, postgres, nats, kafka, qdrant,
foundationdb, http-cache) at the values.yaml source, so the extension is
emitted by cozyvalues-gen instead of hand-edited into values.schema.json.
kafka carries two (Kafka and ZooKeeper structs); foundationdb is nested
under the Storage struct.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
Mark fields that reference cluster resources with the x-cozystack-options
vendor keyword so the dashboard renders them as dynamic dropdowns:
GPU, instanceType, instanceProfile, network (vm-instance), GPU/
instanceType (kubernetes nodeGroups), image (vm-disk), storagePool
(bucket), storageClass (all apps) and disks[].name (vm-instance). Drop
the stale static instanceProfile enum.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io>
## What this PR does
Upgrades Flux v2.7.3 → v2.8.0 across both the vendored tenant chart and
the embedded management-cluster manifests, and folds in the chart fixes
that become hard errors under the new strict helm-controller v1.5.
Flux v2.8's helm-controller v1.5.0 ships:
- **Server-Side Apply with `--force-conflicts`** — strict CRD schema
enforcement; misplaced fields (silently dropped on v2.7) now fail the
apply.
- **kstatus-based health checking** by default — parent HR waits for
every applied resource (including child HRs) to be Ready before flipping
its own Ready, surfacing latent ordering deadlocks.
- **Helm v4 embedded** — `upgrade.force: true` is deprecated and now
conflicts with SSA.
- **`HelmRelease.spec.healthCheckExprs`** — prerequisite for proper
readiness gating (used in PR #2601 split).
Folds in PR #2612 (kubevirt-instancetypes null TPM fix) since the same
Flux upgrade triggers it.
## Commits
**Flux upgrade itself:**
- `feat(fluxcd)`: bump `flux-operator` / `flux-instance` vendored charts
to v0.48.0; web UI opt-in.
- `feat(flux)`: regenerate embedded management-cluster manifests via
`make update` in `packages/core/flux-aio` (timoni bundle build).
**Chart fixes for strict SSA — fields the chart sent that v2.7 silently
dropped, v2.8 rejects:**
- `fix(kubevirt-instancetypes)`: drop persistent strip that produced
null `preferredTPM` (folds #2612).
- `fix(foundationdb)`: relocate `faultDomain`, `imageType`, `labels`,
`minimumUptimeSecondsForBounce` from inside `automationOptions` to
direct children of `spec`.
- `fix(kafka)`: place `enableServiceLinks` under `template.pod`, not a
phantom `template.spec`.
- `fix(vm-instance)`: emit `disk: {}` (not `disk:`/null) when no bus is
set.
- `fix(platform)`: drop deprecated `upgrade.force: true` from
HelmReleases; fix `kafka` WorkloadMonitor `replicas` paths.
**Ordering / deadlock fixes under v2.8 kstatus:**
- `fix(vpa)`: break circular wait between parent install and nested
`vpa-for-vpa` HR.
- `fix(kubernetes)`: drop lookup-guarded parent-HR `dependsOn` on tenant
addon child HRs (parent waits on child via kstatus, child waited on
parent — deadlock).
**E2E waits for v2.8 kstatus timing:**
- `test(e2e)`: bump app HR-Ready waits to 5m (was 20s–100s under v2.7's
faster dispatch).
- `test(e2e)`: wait for parent HR Ready before downstream asserts in
`run-kubernetes.sh` and `vminstance.bats`.
## Scope discipline
This PR is part of the split of #2619 (the consolidated CI fixes branch)
into review-friendly pieces. Companion PRs:
- **PR #2601** (seaweedfs split) — folded into this PR (commits
`29c6afc8`, `0e8b46d7`, `7157158c`, `dccdeb52`, `f880b324`): the
seaweedfs-system → seaweedfs-db + seaweedfs-system split, its adoption
migration 43 (targetVersion 44), and the configurable db resources all
land here, because the strict-SSA `upgrade.force` removal and the
kstatus parent-HR timeout bump only make sense together with the split.
#2601 is superseded.
- **PR #2558** (drop 3× retry on `Run E2E` + `Install Cozystack`) —
independent, lands separately.
- Several smaller standalone fixes lifted out of #2619 (startup probes,
cert-manager `dependsOn`, prepull machinery, CSI HR timeout, NFS/OIDC
test improvements) — opened as separate PRs.
## Verification
- `helm template` renders cleanly for both `fluxcd` and
`fluxcd-operator` packages with `web.enabled=false` (default) and
`web.enabled=true`.
- Embedded `cmd/cozystack-operator` binary contains the v1.5.0 / v1.8.0
/ v2.1.0 controller image strings.
- No references to the v0.39-removed `--disable-wait-interruption` flag
anywhere in `packages/` or `internal/`.
### Release note
```release-note
Flux upgraded to v2.8.0 (helm-controller v1.5 — Helm v4 Server-Side Apply with --force-conflicts, kstatus health checking).
When upgrading existing clusters:
- Kubernetes 1.33+ is now required for the platform (management) cluster, and for any tenant cluster that enables the optional (default-off) Flux addon — that addon ships the bumped Flux too.
- HelmReleases no longer set `upgrade.force: true`. Helm v4 SSA resolves field-ownership conflicts automatically (`--force-conflicts`), but that is not the old client-side replace: immutable-field changes (e.g. StatefulSet volumeClaimTemplates/serviceName) no longer self-heal and require manual recreation — delete the object (e.g. `kubectl delete sts <name> --cascade=orphan`) and let Flux recreate it.
- KubeVirt: persistent TPM/EFI is re-enabled for the Windows 11/2k22/2k25 preferences (KubeVirt 1.8 VMPersistentState); each affected VM provisions an extra RWO backend-storage PVC from the default StorageClass.
- KubeVirt: the EOL centos.7*/centos.stream8* preferences are retained as deprecated, hidden aliases (`tags: hidden`, `instancetype.kubevirt.io/deprecated: "true"`) — existing VMInstances on these profiles keep rendering and need no action on upgrade, but the profiles are no longer offered for new VMs; repoint to centos.stream9/10 when convenient. The gn1.* GPU instancetypes are likewise retained.
- FoundationDB: imageType now reaches the operator (silently dropped pre-SSA); it is pinned to `split` to match the value existing clusters effectively ran, so upgrades stay non-disruptive. Set `imageType: unified` to migrate deliberately.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Flux Status web UI: optional web server, config secret, service port,
HTTPRoute/Ingress, network policy, and RBAC roles.
* New SeaweedFS DB Helm chart and optional managed DB release.
* **Improvements**
* CRD/schema enhancements: new provider kinds, validations, variant
option, and external checksum refs.
* Raised Kubernetes prerequisite to 1.30+; extended e2e timeouts for
reliability.
* **Chores**
* Bumped Flux Operator and Flux versions; documentation links updated to
fluxoperator.dev.
* **Bug Fixes**
* Removed aggressive HelmRelease force-upgrade/install flags.
<!-- review_stack_entry_start -->
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2602?utm_source=github_walkthrough&utm_medium=github&utm_campaign=change_stack)
<!-- review_stack_entry_end -->
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Flux v2.8 / helm-controller v1.5.0 embeds Helm v4, which renamed --force
to --force-replace and rejects combining it with SSA's --force-conflicts.
Existing HelmReleases with spec.upgrade.force: true fail every upgrade with
"cannot use force conflicts and force replace together", looping
upgrade -> rollback -> upgrade and blocking dependants. Observed on PR #2619
CI for tenant-root/{ingress-nginx-system,seaweedfs-db}, tenant-test/info,
and others.
Per Flux docs, upgrade.force has been a no-op since v2.8 anyway, so removing
it is safe on green clusters and unblocks broken ones.
Also fix packages/apps/kafka/templates/workloadmonitor.yaml: it referenced
.Values.replicas (no such field; values.yaml has kafka.replicas and
zookeeper.replicas). Flux v2.7 silently dropped the null; v2.8 SSA forwards
it, and the WorkloadMonitor CRD rejects "spec.replicas: null" as not
type:integer, which blocks every kafka-test install.
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
(cherry picked from commit 7e286cf8e1)
RSA keys are unnecessarily large for internal PKI. ECDSA P-256 provides
equivalent security with smaller key material, faster TLS handshakes,
and smaller certificates — beneficial for high-frequency p2p connections.
CA retains rotationPolicy: Never (root CA stability).
Leaf uses rotationPolicy: Always (new key on every renewal).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Replace the hasKey + index approach with the canonical kindIs "invalid"
check. This is the standard cozystack pattern for detecting an unset
*bool field, making the helper consistent with other charts in the
project.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
cert-manager rotates the leaf cert approximately every 11 months
(renewBefore: 720h, duration: 8760h). Qdrant reads tls.cert/key/ca_cert
at process start and does not watch for file changes, so without a
restart trigger TLS would silently break at the renewal boundary.
Add a targeted Reloader annotation (secret.reloader.stakater.com/reload)
pointing at the specific TLS Secret. This uses the targeted form, not the
auto-watch form, to limit scope to the TLS Secret only. Reloader is
installed cluster-side at packages/system/reloader.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Qdrant's upstream chart constructs bootstrap/uri using short-form pod
hostnames (<release>-N.<release>-headless). The existing wildcard SAN
only covered FQDN form, causing TLS handshake failures in multi-replica
clusters when ca_cert peer verification is enabled.
Add a range loop that emits three SAN variants per replica:
- short form: <release>-N.<release>-headless
- svc form: <release>-N.<release>-headless.<ns>.svc
- FQDN form: <release>-N.<release>-headless.<ns>.svc.<cluster-domain>
Scale-up beyond initial replicas requires cert re-issue; Reloader
(see next commit) will restart pods to pick up the new cert.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Address review feedback: the TLS-enablement block was duplicated in both
qdrant.yaml and certmanager.yaml. Introduce a named helper template
"qdrant.tls.enabled" in _helpers.tpl and replace both inline copies with
a single include call.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Add certmanager.yaml to provision cert-manager Certificate and Issuer for p2p and client TLS
- Extend qdrant.yaml to mount TLS secrets and configure Qdrant with TLS settings when enabled
- TLS activation follows a tri-state: explicit true/false or inherit from cluster external setting
Signed-off-by: Arsolitt <arsolitt@gmail.com>
- Extend QdrantSpec with tls.enabled tri-state bool and expose it via values.yaml and JSON schema
- Update cozyrds integration to pass TLS config to the operator
- Add test: Makefile target and TLS parameter documentation to README
Signed-off-by: Arsolitt <arsolitt@gmail.com>
Annotate storageClass as immutable across every stateful app whose
chart exposes one — clickhouse, foundationdb, harbor, http-cache,
kafka (Kafka + ZooKeeper), kubernetes (top-level only), mariadb,
mongodb, nats, openbao, opensearch, postgres, qdrant, rabbitmq,
redis, vm-disk. Without coverage across the catalogue the UI would
render storageClass read-only on some apps and editable on others —
a worse mental model than uniform behaviour.
The annotation lands at the schema layer (CEL XValidation rule
'self == oldSelf' on values.schema.json / types.go / embedded
openAPISchema in cozyrds/<app>.yaml). The cozystack aggregated
apiserver does not yet evaluate CEL on application-backed resources;
that is tracked at cozystack/cozystack#2657. The UI
(cozystack/cozystack-ui#6) consumes the marker today.
The PVC-pinning argument is identical for every chart: Kubernetes
fixes a PVC's storageClassName at creation time, so editing this
field on an existing resource never migrated data. The schema
annotation makes that contract explicit.
kubernetes.nodeGroups[name].storageClass is intentionally NOT
annotated: the field is optional and undefaulted, so a strict
'self == oldSelf' rule would block any future attempt to set it on
an existing node group. A richer ratcheting form is needed there —
see docs/storage-immutability.md.
A refactor in pkg/registry/apps/application/rest.go extracts the
inline schema-build path into buildSpecSchema so it can be
unit-tested. Three regression tests in rest_spec_schema_test.go pin
the contract that x-kubernetes-validations survives the
v1->internal->structural conversion (loaded from the actual MariaDB
ApplicationDefinition fixture on disk, not a hand-edited mock).
Each per-app README gains a one-line blockquote pointing to a new
docs/storage-immutability.md that explains the contract and which
consumers enforce it today.
Requires cozystack/cozyvalues-gen with @immutable support — bump the
pre-commit pin in .github/workflows/pre-commit.yml once the release
exists.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback from coderabbitai across:
- packages/apps/kubernetes/README.md
- packages/apps/mariadb/README.md
- packages/apps/nats/README.md
- packages/apps/qdrant/README.md
- packages/apps/redis/README.md
- packages/apps/tcp-balancer/README.md
- packages/apps/vpn/README.md
Each of these READMEs left an inline 7-row preset table listing only
the legacy flat names after the parameters block above already
defaulted to an instance-type value (e.g. `t1.nano`). cozyvalues-gen
only regenerates the `## Parameters` block, so the hand-written
section drifted.
Replace the inline tables with the same short paragraph already used
by postgres and kafka READMEs: name the five series, note that the
seven legacy names remain accepted as deprecated aliases, and link to
the canonical reference at docs/operations/resource-presets.md so the
full size matrix and legacy mapping live in exactly one place.
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Rewrite resourcesPreset values in every apps/* and extras/* chart from
the legacy flat names to the new instance-type form, using the 1:1
mapping:
nano -> t1.nano micro -> t1.micro
small -> t1.small medium -> c1.small
large -> c1.medium xlarge -> c1.large
2xlarge -> c1.xlarge
Each affected values.yaml carries the canonical 47-value enum (40
instance-type names plus 7 legacy aliases) so the schema accepts both
forms during the deprecation window. cozyvalues-gen regenerates the
matching api/apps/v1alpha1/*/types.go kubebuilder enum and defaults,
the chart values.schema.json, the README parameter table, and the
ApplicationDefinition openAPISchema in packages/system/*-rd/cozyrds.
Three hardcoded legacy literals in chart templates migrated:
- packages/apps/harbor/templates/harbor.yaml:121 (portal)
- packages/apps/opensearch/templates/opensearch.yaml:43 (dashboards)
- packages/system/keycloak/templates/sts.yaml:115 (heap fallback)
packages/apps/foundationdb/Makefile gains the missing hack/update-crd.sh
call. Without it, every regen left foundationdb-rd cozyrds stale and
new FoundationDB app CRs through cozystack-api were rejected at
validation for any non-legacy resourcesPreset value.
Every default-rendered chart produces identical CPU and memory to the
value applied before the rename, courtesy of the legacy alias table in
cozy-lib and the 1:1 default mapping.
Co-authored-by: Matthieu ROBIN <matthieu@hidora.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Алексей Артамонов
