mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-22 12:43:32 +09:00
f83db514b0eb88cefc2923d17aad59daccb4e358
2787
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
85f576756b |
feat(kubernetes-nodes): add a proxmox substrate for worker pools
Worker pools are already Talos machines created by Cluster API; only the
substrate underneath them is fixed to KubeVirt. This adds `substrate:
kubevirt|proxmox` and swaps exactly two things: the machine template object
and every reference to it. Autoscaler capacity annotations, kubelet
reservations, MachineDeployment, MachineHealthCheck, WorkloadMonitor and the
talos-reconcile Job are substrate-agnostic and rendered once, which is why
this is a branch in the existing chart rather than a second copy of it.
The default path is unchanged down to the bytes: the render snapshot and all
15 existing suites pass untouched, and the hash-named template keeps its
variable name so no live worker VM rolls on upgrade.
Five things the Proxmox path must do differently, each learned from a
failure that pointed somewhere else:
- Skip capmox's guest-agent and cloud-init-status probes. Talos ships
neither, and left on they leave the VM booted and the Machine stuck on
"error waiting for agent: the operation has timed out". This is the same
class of problem `virtualMachineBootstrapCheck.checkStrategy: none` solves
for CAPK's SSH probe; capmox's CRD names TalOS in the skipCloudInitStatus
description.
- Clone a template carrying the nocloud image, not openstack. capmox injects
bootstrap data as a nocloud ISO, and an openstack image ignores it exactly
as silently as a nocloud image ignores the configdrive the KubeVirt path
depends on.
- Pass templateTags through verbatim. capmox compares the template's tag set
with slices.Equal after sorting, so a selector listing a subset matches
nothing and reports "found 0 VM templates with tags ..." while the tags are
plainly present.
- Require resources.cpu and resources.memory. Proxmox has no
VirtualMachineClusterInstancetype, so the lookup that normally supplies
effective CPU and memory has nothing to read — and those two numbers feed
the kubelet reservations and the autoscaler capacity annotations. A pool
that cannot state them would be scheduled against invented capacity, so it
is refused at render time. KubeVirt-only knobs (gpus, logSerialConsole,
podCpuLimit/Request) are refused too rather than accepted and ignored.
- Refuse proxmox.storage unless full is true. A target storage is a full-clone
parameter: a linked clone always lives on the template's storage, and both
Proxmox ("parameter 'storage' not allowed for linked clones") and capmox's
own CRD ("Must set full=true when specifying storage") reject the pair.
Left to them, a pool with storage set and a linked clone fails at apply time
with an error about a ProxmoxMachineTemplate the operator never wrote;
failing the render names the two values instead.
The Talos image is not a chart input on this path: capmox clones a template
that already exists on the hypervisor, so building and rotating that template
is an operator responsibility with no equivalent on the KubeVirt side.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
9934014908 |
fix(capi): ship the core provider manifests the repository actually holds
#2946 added a startupProbe to capi-controller-manager so it survives the certificate provisioning delay. It edited capi-providers-core/files/core-components.yaml, which is the copy a reviewer reads. The chart mounts files/components.gz, and that blob was never regenerated, so the probe never reached a cluster. The blob is regenerated from the YAML it is supposed to carry: gzip -9 -n -c files/core-components.yaml > files/components.gz The decompressed blob now matches the YAML byte for byte, and the only change it carries is the six lines of #2946. This reaches new installs only. capi-operator re-applies a provider when the hash of its spec and its config Secret changes; the ConfigMap holding the components is in neither, so a cluster that already runs capi-providers-core keeps its manager without the probe until the provider's version moves. Whether existing clusters should get the probe, and how, is left open here on purpose: moving spec.version and the ConfigMap name together would force the re-apply, but that is a version decision for the package, not a side effect of regenerating a blob. .helmignore is corrected in the same change: `files/.*-components.yaml` is a glob, so it matched only names beginning with a dot, and `helm package` shipped the readable YAML alongside the blob that duplicates it. hack/capi-core-components_test.bats holds the two files to the same bytes from now on, so the next edit of one without the other fails in CI. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
67db200530 |
feat(cluster-api): add Proxmox infrastructure provider package
Packages ionos-cloud/cluster-api-provider-proxmox (capmox) the same way capi-providers-infraprovider packages CAPK: vendored components gzipped into a ConfigMap, an InfrastructureProvider that fetches them by label. Two details differ from the KubeVirt package on purpose. The fetchConfig label is `infraprovider-provider: proxmox`, not the shared `infraprovider-components: cozy`. A shared label would put both providers in one match. capi-operator would still install the right one — it keys every matched ConfigMap by its name as the version and takes the one equal to spec.version — but it builds that repository in a single pass and returns an error for the whole list as soon as one match is malformed: a name that does not parse as semver, or missing metadata. Sharing the label would let a broken ConfigMap belonging to one provider stop the other from being installed. The bootstrap packages already separate on a `bootstrap-provider` key; this follows that. components.gz is produced with `gzip -n`. Without it the header carries a timestamp, so regenerating from identical input yields a different blob and the committed artifact appears to change when nothing has. `make components` records the version pin, the source URLs and the flag, so refreshing the vendored manifests is reproducible rather than remembered. It also checks both downloads against the SHA-256 sums recorded in the Makefile before packaging them: a release tag is mutable, so the pin is on the bytes, and bumping CAPMOX_VERSION means recording the new sums next to it. Pinned to the v0.7 series: from v0.8 capmox declares the v1beta2 Cluster API contract, while capi-providers-core here is v1.10.1, which serves v1beta1. The package source depends on the in-cluster IPAM provider and deliberately not on cozystack.kubevirt: the KubeVirt provider needs it because its VMs run inside this cluster, whereas a Proxmox hypervisor is external and reached over its API with the credentials named by `credentialsSecretName`. That Secret is a precondition rather than an optional extra, and the chart does not create it — a hypervisor token belongs to the platform's secret management, not to a Helm release. capi-operator reads the Secret before it installs the provider, so while it is missing nothing is deployed at all. values.yaml carries what an operator needs at that point: that PROXMOX_URL is the bare https://<host>:8006 — capmox appends /api2/json itself, and a URL that already carries it sends every call to /api2/json/api2/json/... and gets 501 back — the full user@realm!tokenid form of PROXMOX_TOKEN, and the fact that a rotated token reaches the manager only after the next provider reconcile and a restart of its pod. Both edges are asserted in tests/sources_capi_provider_proxmox_test.yaml; losing the IPAM edge reproduces a failure mode whose symptom points nowhere near its cause. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
86834d30a9 |
feat(cluster-api): add in-cluster IPAM provider package
Cluster API ships address management as a separate provider, and Cozystack
has never needed one: KubeVirt workers lease their addresses from the
cluster network's DHCP. An infrastructure provider that allocates addresses
instead of leasing them cannot work without it.
capmox is such a provider. It builds an InClusterIPPool out of
ProxmoxCluster.spec.ipv4Config and hands every VM a static address from it.
With no IPAM provider installed, the pool kind does not resolve:
Reconciler error: no matches for kind "InClusterIPPool"
in version "ipam.cluster.x-k8s.io/v1alpha2"
ProxmoxCluster then never reaches Ready and every ProxmoxMachine sits in
WaitingForClusterInfrastructure indefinitely — indistinguishable, from the
outside, from a provider that is simply broken.
Pinned to the v1.0 series rather than v1.1.0: both serve InClusterIPPool
v1alpha2, but v1.0 declares the v1beta1 contract that matches
capi-providers-core v1.10.1.
`make components` checks both downloads against the SHA-256 sums recorded
in the Makefile before packaging them. A release tag is mutable, so the pin
is on the bytes rather than on the tag; bumping CAIP_VERSION means recording
the new sums next to it.
The package carries no edge to any infrastructure provider. The dependency
belongs on the consumer's side, which keeps this reusable for any future
provider with the same requirement.
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
|
||
|
|
cffe509dcf |
feat(monitoring): per-tenant OTLP collector (tracing phase 2) (#3763)
## What this PR does Phase 2 of distributed tracing (design cozystack/community#38, epic #3761), built on the phase-1 backend (#3762, merged): a per-tenant **OpenTelemetry Collector** — the app-facing OTLP ingest gateway. It lives in the tenant namespace (like a per-tenant vmagent), so app→collector traffic is intra-namespace and never crosses the tenant boundary. No OTel operator exists in the platform, so this is a plain Deployment + ConfigMap + Service (the fluent-bit pattern). ## Changes - **`templates/vtraces/collector.yaml`** — OTLP receivers (gRPC `:4317`, HTTP `:4318`) → `memory_limiter` + head sampling (`probabilistic_sampler`, default 10%) + optional `attributes/redact` (PII/secret control point) + `batch` → `otlphttp` exporter to the `tracingStorages` backend (`vtinsert:10481` cluster / `vtsingle:10428` single, endpoints per upstream docs). PSS-restricted securityContext, digest-pinned image via `cozy-lib.image`, config-checksum roll. - **app-facing Service `otel-traces`** (4317/4318). - **`tracingCollector` values** — `enabled` / `replicas` / `samplingPercentage` (schema-bounded 0-100) / `redactAttributes` / `resources`. A **fail-guard** rejects an enabled collector fronting more than one `tracingStorages` backend (it exports to a single backend by design). - **`otel-traces` WorkloadMonitor** and dashboard resource-map entry, rendered under the collector's own gate. The same resource-map Role also gets the pre-existing `alermanager` typo corrected to `alertmanager`, so the alertmanager WorkloadMonitor is readable from the dashboard. - helm-unittest coverage for the collector and its WorkloadMonitor; regenerated `values.schema.json` / `README.md` / `monitoring-rd` ApplicationDefinition. ## Verification - `helm unittest` — `system/monitoring` 110, `extra/monitoring` 15, all pass. - `helm template` renders cleanly; `make generate` idempotent. - Live check on a test cluster (single-mode backend): the collector starts with `redactAttributes` containing a numeric, a colon-bearing and an empty key; an OTLP/HTTP span sent to `otel-traces:4318` is accepted and is queryable from the backend through the Jaeger API, with the redacted attributes removed and the others kept. - Local `/branch-review` — **LGTM** (one blocker + four recommendations fixed, each with a test: multi-backend guard, memory_limiter cgroup limit, configurable replicas, sampling bounds, trust-boundary wording). ## Scope / deferred (epic #3761) `AccountID`/`ProjectID` header injection for the shared-central topology (phase 4) and engine integrations (phase 3) are follow-up PRs. ### Screenshots N/A, no UI change. ### Downstream repositories The diff touches only `packages/extra/monitoring`, `packages/system/monitoring` and `packages/system/monitoring-rd`. The website's monitoring reference page is regenerated from the package README by the release bot, and the Terraform provider models only the tenant's `monitoring` flag, not these values, so no follow-up is needed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): deploy a per-tenant OpenTelemetry (OTLP) collector for traces once a VictoriaTraces backend is configured. Applications send OTLP to the in-namespace `otel-traces` Service (gRPC `:4317`, HTTP `:4318`); the collector applies head sampling (`samplingPercentage`, default 10%) and optional attribute redaction (`redactAttributes`) before exporting to the backend. Off by default (no backend configured). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable trace storage in cluster or single-instance modes, with retention settings and Grafana datasources. * Added an enabled-by-default OpenTelemetry collector with configurable replicas, sampling, attribute redaction, and resource limits. * Added workload monitoring and automatic resource recommendations for configured trace components. * Added readiness health checks for both tracing backend modes. * **Bug Fixes** * Corrected dashboard permissions for alert monitoring and trace collector resources. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
368d00b5d4 |
test(e2e): remove unused monitoring suites and audit BATS runners (#4019)
## What this PR does I removed the two monitoring OIDC BATS suites that had no runner after the Chainsaw migration, and brought the runner audit and wrapper OIDC test from [#3778](https://github.com/cozystack/cozystack/pull/3778). Also added checks for audience mapper and `OIDC_MODE=System`. The named `KeycloakRealmGroup` test documents coverage the document-count and kind assertions already provided. Runner audit reports unused BATS suites, missing files in runner paths, and unresolved Make variables (including `$(E2E_SUITE)`). Live Monitoring API admission, Grafana operator reconciliation and mode transitions are not tested here. ### Validation - `make -j4 unit-tests SHELL=/usr/bin/dash` passed. - Monitoring Helm tests: 53 system-chart tests and 10 wrapper-chart tests passed. - Runner audit: 12 cases passed under dash and Bats 1.14.0. The full shell syntax and BATS sweep passed under dash too. - Three chart mutations and seven runner mutations failed as expected. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` |
||
|
|
03e2da5805 |
feat(postgres,mariadb)!: drop plaintext user passwords (#4078)
## What this PR does Removes the plaintext `users[].password` field from the `apps/postgres` and `apps/mariadb` charts (and the mariadb `root` account), so managed-database passwords can no longer be pinned in tenant values or committed to git. Passwords are always chart-generated into the `<release>-credentials` Secret and preserved across reconciles via `lookup`, matching the model `redis`/`valkey` already use. This closes the "User passwords" item of the portal upstream epic (aenix-org/cozyportal#1060). The generated password never lands in the init-script Secret or the SQL it runs: the postgres init-job reads each user's password from the mounted `<release>-credentials` Secret at run time and applies it through a psql variable (`ALTER ROLE … WITH PASSWORD :'pw'`), so it does not linger in the init-script manifest a tenant can read. It does live in cleartext `stringData` in the `<release>-credentials` Secret itself, which is a chart manifest and therefore kept in Helm release history for `MaxHistory` revisions like any chart-generated secret — that residue is exactly why rotation cannot be built on the chart (see below). A `password` or a `passwordRotation` key left over in a tenant's values is accepted-but-inert and draws an admission warning, so an operator editing it is told it has no effect rather than getting a silent 200. Backup/restore into a copy keeps the target's chart-managed credentials working: the CNPG driver clears `bootstrap.enabled` once recovery is healthy so the init-job re-applies the generated passwords onto the recovered roles, and the MariaDB driver excludes `mysql.global_priv` from the dump (`ignoreGlobalPriv`) for backups captured from this release onward, so restoring one of those into a copy does not clobber the target's chart-managed users or `root`. Excluding the grant table has a flip side, and it applies to every MariaDB restore, not only a to-copy one: only accounts declared through `User` CRs (the chart's `users` map) are captured, so an account created out of band directly in MySQL is in no backup and a restore brings back its data and `mysql.db` grants but not the account itself. This is also a backup-side fix only: a restore from a backup taken **before** this change still replays the source's `mysql.global_priv` and overwrites the target's users and `root`, and `root` then needs an out-of-band reset (the operator applies `rootPasswordSecretKeyRef` only at datadir bootstrap, so there is no in-band path). See `docs/operations/backup-classes.md` for the convergence window, the out-of-band-account limitation, and this old-backup caveat. Password **rotation** is deliberately **not** in this PR. A chart-rendered credentials Secret keeps its cleartext in Helm release history for `MaxHistory` revisions, so a rotation built on it cannot revoke a leaked password; rotation is being reworked as a controller that owns the Secret — design proposal in cozystack/community#72. Migration is seamless: existing releases keep their current passwords on upgrade (the value already lives in `<release>-credentials`, which `lookup` re-reads and preserves); a `password` still present in a tenant's values is ignored, so no existing HelmRelease fails validation. Verification: `helm unittest` passes for both charts (postgres 87, mariadb 126, including render-time username guards that reject any name k8s would later reject as an invalid Secret key or User resource name); `go build ./...`, `go vet`, the backup-controller tests and the apps-apiserver tests are green; `make generate` was run with the CI-pinned `cozyvalues-gen` v1.6.0 so `values.schema.json`, the READMEs, the API types, and the `cozyrds` ApplicationDefinitions are all in sync (the branch was rebased onto `main`, which reconciled cleanly with the read-replica autoscaling feature that landed there). ### Screenshots N/A — no UI change. ### Downstream repositories - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: affected — `content/en/docs/next/cozystack-api/go-types.md` is hand-written and still shows `Password:` in the Go examples, and the managed-app reference pages regenerate from these chart READMEs, so both document `users[name].password` until refreshed (not yet filed). - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: affected — the provider hand-maintains the postgres/mariadb schemas and per-field defaults; this PR removes `users[].password`. A follow-up issue/PR is needed there (not yet filed). `terraform-provider-cozystack` and `website` are reached by this diff; both boxes are left unticked because the follow-ups are not yet filed. Same-repo future work this change suggests: the controller-owned password rotation (design in cozystack/community#72), and extending the "passwords only in the Secret" model to the other engines that still accept plaintext (`clickhouse`, `mongodb`, `nats`, `opensearch`, `rabbitmq`, `vpn`). ### Release note ```release-note feat(postgres,mariadb)!: passwords for managed PostgreSQL and MariaDB users can no longer be set via `users[].password` — they are always auto-generated and stored only in the `<release>-credentials` Secret, never rendered into the init-script or the SQL it runs. Existing releases keep their current passwords on upgrade; a `password` (or a `passwordRotation`) left in values is ignored with an admission warning. Password rotation is deferred to a follow-up controller design (cozystack/community#72). MariaDB backups now exclude the `mysql.global_priv` grant table (`ignoreGlobalPriv`), so a restore recreates only accounts declared through the chart's `users` map — an account created out of band directly in MySQL is not captured and must be recreated after a restore. ``` |
||
|
|
60b1b05434 |
feat(monitoring): add per-tenant OTLP collector for distributed tracing
Applications need one in-namespace OTLP endpoint that also enforces
sampling and redaction before spans reach the VictoriaTraces backend.
No OpenTelemetry operator ships with the platform, so the collector is
a plain Deployment, ConfigMap and Service ('otel-traces', gRPC 4317,
HTTP 4318) in the tenant namespace; app-to-collector traffic therefore
never leaves the tenant, and isolation comes from the existing tenant
network policies.
The collector renders only when it is enabled and a tracingStorages
backend is configured. Tracing is off by default, and an empty backend
list is a clean off-state rather than a render failure; an app exporter
that finds no Service simply retries. It writes to the first backend
only and rejects a list with several, instead of silently dropping
spans for the others.
redactAttributes is the PII and secret control point, so its keys are
rendered quoted and empty keys are dropped: an unquoted key with YAML
structural characters, or an empty one, makes the collector fail at
startup, and a key YAML re-types (numbers, booleans, null) silently
escapes redaction. replicas has a lower bound of 1 so a bad value fails
schema validation instead of the whole release at apply time; a zeroed
collector is expressed with enabled: false.
The collector gets a WorkloadMonitor and a dashboard-resources Role
entry under the same render gate, so it is part of tenant workload
accounting like every other workload in the chart.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
|
||
|
|
2d2186d1f0 |
feat(monitoring): VictoriaTraces backend + Grafana datasource (tracing phase 1) (#3762)
## What this PR does Phase 1 of distributed tracing (design: cozystack/community#38; epic: #3761): a **VictoriaTraces backend and Grafana traces datasource** in the monitoring stack, mirroring the logs (`VLCluster`) implementation. No new operator: `VTCluster`/`VTSingle` already ship in victoria-metrics-operator. - **`tracingStorages`** values in `extra/monitoring` (annotated, schema-generating) and `system/monitoring` (defaults), per-entry `mode: cluster|single`. **Empty by default, so tracing is off out of the box**: nothing produces spans until the collector ships, so no backend is provisioned until an operator adds an entry. - **`templates/vtraces/vtraces.yaml`**: `VTCluster` (default) or `VTSingle` (edge/dev/small); `retentionPeriod` plus optional `retentionMaxDiskSpaceUsageBytes`; a release-scoped **PVC cleanup label**, reused by the existing post-delete hook, so no hook change is needed. - **Render-time validation** (`_helpers.tpl`): the installed VictoriaTraces CRDs are the trimmed ones, so a bad value would otherwise reach the operator and fail after admission. Each `tracingStorages` entry is checked before anything renders: `name` must be an RFC 1123 label of at most 42 characters (the `vtstorage-<name>-<hash>` pod revision label is capped at 63) and unique across the list, `mode` must be `cluster` or `single`, `storage` must be a positive quantity (a zero size makes the operator mount an EmptyDir in single mode, and a negative one fails when the PVC is created), and `retentionDiskUsageBytes` must be a VictoriaMetrics `BytesString` (the operator rejects the Kubernetes-quantity `8Gi` grammar that `storage` uses). The `storage` field is also typed as a quantity in the schema, and the rendered name is quoted so a numeric name stays a string. - **`templates/vtraces/grafana-datasource.yaml`**: a Jaeger-typed traces datasource per entry (built-in Grafana core datasource, no plugin). - **VPA**: a `VerticalPodAutoscaler` for each trace component (`vtinsert`/`vtselect` Deployments, `vtstorage` StatefulSet, `vtsingle` Deployment), matching the metrics/logs sizing, with `updateMode: Initial`. - **Readiness gate**: `VTCluster`/`VTSingle` `healthCheckExprs` on `status.updateStatus`, alongside the existing `VLCluster` gate. - **VictoriaLogs datasource fix** (`templates/vlogs/grafana-datasource.yaml`): the range over `logsStorages` emitted no `---` between documents, so two or more entries collapsed into one document and only the last datasource survived. Each entry is now its own document. - **Tests and schema**: helm-unittest suites for off-by-default, cluster vs single, datasource type and both query URLs, retention, PVC label, VPA targets, every validation guard with its boundaries, and per-entry vlogs datasource documents, plus regenerated `values.schema.json`, `README.md` and the `monitoring-rd` ApplicationDefinition. #### Upgrade impact With default values the only rendered object that changes is the `monitoring-system` HelmRelease: it gains the `VTCluster`/`VTSingle` `healthCheckExprs` and `tracingStorages: []`. Flux evaluates those expressions only against objects in the release inventory, so they are inert while tracing is off. No VictoriaTraces object is created on upgrade. Installs with two or more `logsStorages` entries additionally gain the `GrafanaDatasource` objects that were previously dropped (see the VictoriaLogs fix above); nothing else under VictoriaLogs or VictoriaMetrics changes. #### Verification - `helm unittest`: `system/monitoring` (89 tests) and `extra/monitoring` (11 tests) pass. Each validation guard and each branch of the zero-storage pattern was mutation-checked: reverting it turns its own test red. - `helm template` renders `VTCluster`/`VTSingle`/datasource/VPA cleanly. `tracingStorages: []` (the default) renders nothing. An invalid, duplicate or over-long `name`, a zero or negative `storage`, or a bad `retentionDiskUsageBytes` (`8Gi`) is refused at render with an actionable message; a numeric `name` renders as a string. `make generate` is idempotent. - **Datasource URLs verified against upstream docs** ([VictoriaTraces and Grafana](https://docs.victoriametrics.com/victoriatraces/querying/grafana/)): cluster `vtselect:10471/select/jaeger`, single `vtsingle:10428/select/jaeger`. `jaeger` is the stable, built-in Grafana datasource; the Tempo Query API (`/select/tempo`, `type: tempo`, VictoriaTraces v0.9.4+) has a richer TraceQL UX but is upstream-experimental, so Jaeger is the default. Both URLs are pinned by unit tests. - The `retentionDiskUsageBytes` pattern, the zero-storage check and the name limit were checked against victoria-metrics-operator v0.68.4 (`BytesString.UnmarshalJSON`, `isStorageEmpty`, the `vtstorage-<name>` StatefulSet name) and the Kubernetes `ControllerRevisionName` naming. #### Scope / deferred (tracked in #3761) Per-tenant OTLP Collector, engine integrations, shared-central topology, and docs are follow-up PRs. This PR intentionally does **not** enable the OTLP/gRPC listener: apps can use `vtinsert`'s native OTLP/HTTP, and the gRPC listener lands with the collector. Builds on the now-merged design proposal cozystack/community#38. ### Screenshots Not a UI change. ### Downstream repositories Walked against the diff, no box ticked yet: - cozystack/website: `monitoring` is in the website `Makefile` `SERVICES` list, so its reference page is regenerated from `packages/extra/monitoring/README.md` automatically on the next stable tag. A user guide for tracing belongs to the docs follow-up in #3761; no website issue or PR exists for it yet. - cozystack/terraform-provider-cozystack: the provider has no `Monitoring` resource (only the tenant's `monitoring` flag), so the new schema field does not reach it. - The remaining repositories in the trigger map are not reached by this diff: no platform values, variants, installer values, `hack/` tooling, node prerequisites, ApplicationDefinition semantics or telemetry names change. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(monitoring): add a VictoriaTraces backend and a Grafana traces datasource to the monitoring stack (distributed tracing, phase 1). Tracing is off by default; enable it per tenant by adding a `tracingStorages` entry (`cluster` or `single` mode). Also fix the VictoriaLogs Grafana datasources: with two or more `logsStorages` entries, every entry except the last was silently dropped, so existing installs with several log stores gain the missing datasources on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable trace storage with cluster or single-node deployment modes. * Supports retention periods, persistent storage sizing, storage classes, and optional disk-usage limits. * Automatically configures Grafana tracing data sources for each enabled trace backend. * Monitoring readiness reflects the health of configured trace storage services. * Invalid trace storage settings are rejected during configuration. * **Bug Fixes** * Every VictoriaLogs storage now gets its own Grafana datasource; with several log storages, all but the last were previously dropped. * **Tests** * Added coverage for storage rendering, retention settings, validation guards, Grafana endpoints, health checks, and cleanup behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
325feec805 |
fix(migration): restore the class reclaim policy on imported volumes
The handoff sets the transferred PersistentVolume to Retain so the data survives the window in which the Forklift claim is deleted and claimRef is re-pointed at the VMDisk's claim. It never gave the policy back, so every imported disk kept Retain for good while its StorageClass says Delete. Deleting such a VMDisk removes its DataVolume and claim through the normal path and strands the volume Released with its backing storage, billed and out of any tenant's reach. Retain is now held only while it protects something. Once the new claim and the volume both report Bound to each other, the transferred claim is gone, and CDI has adopted the claim into the VMDisk DataVolume as its only owner, the volume gets its class's policy back (Delete when the class names none or no longer exists). The VMInstance is created and the VM reported Succeeded only after that, so a task that fails later still leaves disks that clean up like any other. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
0687124b00 |
fix(seaweedfs): pass the Iceberg catalog port explicitly so null means off
The chart wraps -port.iceberg in an if, and icebergPort defaults to null, so the flag is dropped and weed falls back to its own 8181 default. The chart therefore cannot express "no Iceberg catalog": the value that reads as off is the one that leaves an unmanaged plaintext catalog listening on the pod IP, with the same writable routes and the same admin-when-no-identities fallback as the Lance one. Render the flag the way 4.45 already renders -port.lance, so null reaches weed as 0 and an explicit port still wins. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
ea04e671a7 |
fix(seaweedfs): keep the Lance catalog listener off by default
Chart 4.45 ships upstream's Lance Namespace catalog enabled (lancePort: 9101) and publishes it on the s3 Service in every tenant namespace. weed serves that catalog over plain HTTP with no way to supply a certificate, while every other port on this gateway is TLS by way of the s3-tls-main-port patch, and its namespace and table routes are writable. Nothing in Cozystack consumes the catalog, so pin the port off and keep the s3 port set the bump inherited. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
46b4b35270 |
chore(seaweedfs): bump chart to 4.45
Refreshes the vendored chart from 4.31 through `make update`. All carried patches still apply; the volume-resize-hook one is rebased onto the pod template labels upstream now renders, so it appends the cozystack apiserver label instead of creating the block. Rendered names, StatefulSet selectors and volumeClaimTemplates are identical to 4.31, so running workloads keep their volumes across the upgrade. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
0bae19fccc |
feat(lineage-webhook): refuse a second ApplicationDefinition for a kind
A definition declaring an application kind that another definition already declares now fails when it is applied, instead of sitting next to the first one where it has no effect. An update that keeps its kind is allowed, so a duplicate that predates the check keeps upgrading, and so is a second definition from the same Helm release, since a release that renames its definition object creates the new one before it deletes the old one. The webhook is registered with failurePolicy Ignore: ApplicationDefinitions ship with the platform itself, so an outage of the webhook must never block a bootstrap or an upgrade, and a failure to read the existing definitions lets the request through with a warning rather than answering with an error, which Ignore would not cover. It is the feedback layer; which definition manages a kind is decided by the ownership rule whatever it lets through. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
|
dd1fbf9bbb |
test(kamaji): fail when a nested subchart renders anything
A subchart under charts/kamaji/charts deploys unconditionally through this umbrella, so the suite asserts that no template there produces a document. It is the package's first helm-unittest suite, so the Makefile gains the test target the repository sweep discovers, and the suite directory is kept out of the packaged chart. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
47caa2a227 |
fix(kamaji): drop any subchart nested under the vendored kamaji chart
The vendored kamaji chart declares a kamaji-etcd dependency gated on kamaji-etcd.deploy. That condition is never evaluated here: Helm's processDependencyEnabled returns before recursing when the chart it starts from declares no dependencies, and this umbrella declares none. A chart placed under charts/kamaji/charts would therefore render whatever the values say, deploying an etcd nobody asked for. Upstream does not ship that directory today, so the update recipe produces the same tree as before; removing it after extraction keeps it that way if upstream ever starts committing the packaged subchart. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
564f540f46 |
fix(kubernetes): render the OIDC UID header only from v1.33 (#4407)
## What this PR does Stacked on #4398; review the diff against `fix/kamaji-pin-hash-reset`. The tenant Kubernetes chart now renders `--requestheader-uid-headers=X-Remote-Uid` only from an effective Kubernetes version of v1.33, where `RemoteRequestHeaderUID` is beta and on by default. It no longer renders a `--feature-gates` term of its own on any version. On v1.32 it used to add a term enabling that gate, which is alpha there, whenever it could read the operator's entries and they left the gate unset; no release tag contains that path, so dropping it now changes nothing a user has run. The chart still reads `--emulated-version`, because emulation moves the effective version: `--emulated-version=1.32` on a v1.35 tenant puts the gate back to alpha-and-off, and the header flag is not rendered there. It still reads the operator's `--feature-gates` entries for an opt-out (`RemoteRequestHeaderUID=false`, or `AllBeta=false` with no explicit `=true`), so the header flag never lands next to a disabled gate. `AllAlpha` is no longer read, since it says nothing about a beta gate. An unquoted `--emulated-version=value` entry whose kube version has a non-zero patch component now fails the render and names the entry. The rule mirrors upstream rather than the regex in #4396: `toVersionMap` parses each value with `version.Parse` and refuses only `ver.Patch() != 0` (`staging/src/k8s.io/component-base/compatibility/registry.go` lines 320-326 at v1.35.0, the same code in `staging/src/k8s.io/apiserver/pkg/util/version/registry.go` at v1.31.0). `version.Parse` takes `^\s*v?([0-9]+(?:\.[0-9]+)*)(.*)*$` (`versionMatchRE` in `k8s.io/apimachinery/pkg/util/version/version.go`, identical at v0.31.1 and v0.35.0). So `1.32`, `v1.32`, `1.32-rc`, `1.32.0` and `1.32.0-rc` all start an apiserver emulating 1.32, and `1.32.4` and `1.32.4-rc` do not. A `^[0-9]+\.[0-9]+$` check would have refused the first four. The chart extracts the numeric run with the same pattern, reads the minor from it, and fails only on a non-zero third component. A quoted value is still left unread. Two deliberate narrowings: - The chart no longer refuses a mix of bare and `kube:`-prefixed `--feature-gates` terms. Once the chart writes no gate term, the spelling of the operator's terms no longer concerns anything it renders. The apiserver still refuses the mix (`component-base/compatibility/registry.go:405` at v1.35.0), and the chart now passes it through like the other fatal gate values it already leaves to the apiserver: unknown names, unparseable values, quoted terms. - On v1.32 the chart now neither renders nor judges the UID flags, as it already did on v1.31. An operator's own `--requestheader-uid-headers` entry without the gate reaches an apiserver that refuses it, which is how every released chart version behaved. A cluster built from `main` that already runs a v1.32 tenant with OIDC on loses the chart's `--requestheader-uid-headers` and gate term on the next render; no release carried them. That holds only while no release is cut from `main` before this merges; a release that carries the v1.32 gate term first would turn this into a breaking change for v1.32 tenants, whose migration is to write both flags into `extraArgs` themselves. An element such as `kube=1.32=x` is now split at its first `=` only, the way `toVersionMap` splits it with `SplitN(compVer, "=", 2)`. The chart used to skip that element, which the apiserver reads as 1.32, and so rendered the header flag against an emulated 1.32. That gap predates this branch. Rebased onto #4398 at `fb2e57c3b`, which reads a two-element `--feature-gates` and `--emulated-version` separately. That behaviour carries over unchanged under the v1.33 floor: behind a hidden `--emulated-version` only an explicit `RemoteRequestHeaderUID=false` is refused beside a header entry, while a blanket `AllBeta=false` waits for a reading, because a lower emulated version can make the gate alpha again; nothing is refused as an opt-out behind a hidden `--feature-gates` value; the empty-element refusal applies behind both; and behind either one `X-Remote-Uid` is added only beside a uid-headers list of the operator's. The README breaking-changes note from #4398 now says the chart refuses neither the mixed gate spelling nor two kube emulated versions. In `docs/oidc-tenant.md` the paragraphs that list refusals now name the managed-flag move and refusal from #4398 and no longer carry a count, and the two-element paragraph now says what the chart renders behind a hidden value. Tests: `helm unittest` goes from 308 on #4398 to 298. 26 cases are deleted. 25 of them pinned the v1.32 gate term, `AllAlpha` handling or the mixed-spelling refusal. One emulated a version below 1.32, which the below-floor emulation case now covers. v1.32 cases that tested parsing rather than the gate are retargeted to v1.34. Sixteen cases are added: the patch-level refusal, the version parsing below the floor and above it (on v1.35 emulating `1.33-rc`, `v1.33`, `1.33.0` and `kube=1.33=x`, where only a correct minor renders the header), the first-`=` split, a pin that a mixed gate spelling now passes through, emulation landing exactly on 1.33, and a v1.32 opt-out beside a header entry passing through untouched. Coverage notes, not applied: no case pins that the patch-level refusal stays off with `spec.oidc.mode: None`, where the block that holds it does not run. The case "splits a component element at its first equals sign only" pins that the kube element of `kube=1.32=x` is read. It does not pin the split itself, because the version pattern ignores what follows the numbers either way. The `extraArgs` field description still calls the field the escape hatch for apiserver flags unrelated to OIDC. It is unchanged here, and it stays accurate for the v1.32 advice in the guide, because the UID flags belong to the aggregation layer rather than to OIDC. Optional, not applied: on v1.32 the chart could refuse an operator's `--requestheader-uid-headers` entry that comes without the gate, since the gate is off by default there and the apiserver will not start. The same skip also drops two refusals the branch history had at 1.32, a header entry beside `RemoteRequestHeaderUID=false` and a header list ending in a comma; both now reach an apiserver that refuses them. It stays out so that v1.32 matches v1.31 and the released charts, where the chart judges none of these flags. Not applied: for a value with a zero-prefixed major such as `01.32.4`, the patch refusal suggests `01.32`. The apiserver refuses both values over the leading zero, so the configuration fails either way and only the suggested fix is off. Why the patch refusal stays while the mixed-spelling refusal goes: the patch refusal is about `--emulated-version`, which the chart reads itself to decide what to render. The mixed spelling concerns `--feature-gates`, which the chart no longer writes to. Not split out: the `--emulated-version` parsing and the patch-level refusal share the fix commit with the floor move, because both change how the same value is read. The change also removes a comment from `hack/e2e-chainsaw/_lib/run-kubernetes.sh` that became false, so select-e2e runs the full suite for this PR. Default path: renders of the parent and this branch through `tests/values/common.yaml` and `tests/values/oidc-system.yaml` at v1.31 to v1.35 are byte-identical, once the randomly generated Kamaji token fields are masked, except for OIDC System at v1.32, which loses exactly the two UID lines. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The only field that changes is the description of `version`, which returns to its earlier wording. The website's kubernetes reference page is regenerated from the README on the next tag, and the Terraform provider keeps its own copy of that description, which already reads "Kubernetes major.minor version to deploy.". Closes #4373 Closes #4396 ### Release note ```release-note fix(kubernetes): the tenant chart renders the aggregation-layer UID header flag with OIDC on only from an effective Kubernetes v1.33, and never renders a --feature-gates term of its own. v1.32 tenants render no UID flags, which matches every released version, so nothing regresses. This supersedes the v1.32 clause of the #3539 release note. An unquoted --emulated-version entry with a patch-level kube version such as 1.32.4, which the kube-apiserver refuses, now fails the render and names the entry. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OIDC-enabled clusters on Kubernetes 1.33 and later now receive the UID request-header setting without an added feature-gate flag. * Chart rendering now checks Kubernetes and Talos version compatibility and prevents rendering when worker deployments do not meet ownership or retention requirements. * **Bug Fixes** * Emulated Kubernetes versions with non-zero patch components now fail with a clear error; supported version formats are recognized more consistently. * **Documentation** * Updated version and OIDC guidance to reflect current chart behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
339cffe70f |
feat(kamaji)!: update to 26.9.4-edge and read every apiServer extraArgs entry (#4398)
## What this PR does BREAKING CHANGE: a Kubernetes application whose `controlPlane.apiServer.extraArgs` names a flag the tenant control plane now owns fails to render until that entry is removed, except that an `--enable-admission-plugins=` entry is moved into the new `controlPlane.apiServer.admissionControllers` and a `--kubelet-preferred-address-types=` entry into the new `controlPlane.kubelet.preferredAddressTypes`. Closes #3541. Updates the Kamaji build this repo ships and rewrites the part of the tenant Kubernetes chart whose correctness rested on the old one. The new pin is `26.9.4-edge`, which is what `make update` picks. Two upstream changes reach us. The tenant control plane no longer folds the kube-apiserver argument list into a map keyed by flag name, so entries supplied through the control plane spec arrive at the apiserver verbatim, duplicates included. And the reset that rewrites those arguments from scratch keys on a hash of the list rather than on its length, so swapping one flag for another no longer strands the old one on the running Deployment. The OIDC block in the tenant chart read only the last entry of each flag. That was right while earlier entries were dropped before the apiserver saw them, and it is wrong now. An operator whose `extraArgs` carried `--feature-gates=RemoteRequestHeaderUID=false` followed by an unrelated `--feature-gates` entry got `--requestheader-uid-headers=X-Remote-Uid` rendered next to a gate the apiserver has switched off, and the control plane would not start. A blanket `AllBeta=false` has the same shape, and so does an `--emulated-version` entry naming kube followed by one naming another component. The chart now reads every entry with the accumulation each flag performs. Occurrences of `--requestheader-uid-headers` append into one list, so where a list of the operator's own omits `X-Remote-Uid` the chart adds its own element beside it instead of refusing to render: the union is what the apiserver validates, and their header keeps working. Feature-gate terms collect per component and are applied in one pass, so the chart carries only its own term, placed last where it wins. Naming the kube component both bare and `kube:`-prefixed now fails the render, because the apiserver refuses that configuration and nothing the chart renders repairs it. `--emulated-version` accumulates too, and two entries naming kube stop the apiserver outright, so the chart renders nothing rather than pick a version that will not exist. The same upstream change narrows what `extraArgs` can do, and the chart now refuses the part that stopped working. Kamaji 26.9.4-edge sets a list of kube-apiserver flags itself and drops any `extraArgs` entry that names one before the apiserver sees the list, where `26.3.6-edge` merged `extraArgs` last and let the entry win. A tenant with `--enable-admission-plugins=NodeRestriction,AlwaysPullImages` loses both plugins on the first reconcile after the upgrade, with nothing to say so. The drop happens on the running control plane whatever the chart renders, so the chart cannot prevent it; what it can do is refuse to render such an entry, in every `oidc.mode`, and name it. The list mirrors `buildKubeAPIServerCommand` at the pinned tag for the control plane this chart renders (an etcd DataStore, konnectivity on, no `dataStoreOverrides`), and the `controlPlane.apiServer.extraArgs` description in the chart README carries it. A flag the control plane only defaults, such as `--authorization-mode` or `--requestheader-group-headers`, is still yours to set. `--enable-admission-plugins` gets a replacement: the new `controlPlane.apiServer.admissionControllers` passes a list through to the KamajiControlPlane `spec.admissionControllers`, which the control plane renders into that flag, and the application API checks each name against the enum the KamajiControlPlane CRD declares. The list is rendered only when non-empty, so a tenant that sets nothing renders exactly as before, and a non-empty list replaces the control plane's default set, as the `extraArgs` entry did under the previous build. `--kubelet-preferred-address-types` gets the same treatment: the list the chart used to hardcode on the KamajiControlPlane is now `controlPlane.kubelet.preferredAddressTypes`, which renders that list when left empty. Neither entry is refused where it can be moved: an `--enable-admission-plugins=` or `--kubelet-preferred-address-types=` entry is moved into its field while the field is empty, on every render, and left out of the rendered `extraArgs`, so a tenant carrying one keeps its setting across the Kamaji update with no edit and no ordering to get right. The entry is refused beside a non-empty field, when it names nothing, and in the two-entry spelling. A separate defect in the same package is fixed here. The TenantControlPlane schema reaches existing clusters through the `kamaji-crds` subchart's templated CRDs, which every release upgrade applies; `crds/` holds the copies a fresh install lays down first. Directory-based renders lost that schema entirely, `helm template`, `lint` and `package` alike, because the parent `.helmignore` matched the subchart's `hack/` directory at any depth and that is where the templates keep the schema they read with `.Files.Get`. Helm honours `.helmignore` for a directory and skips it for an archive, so the cluster never saw the difference and no local check could catch a schema defect. The patterns are anchored to the chart root and a unit test asserts the rendered CRDs carry their versions. The bump is not cleanly reversible, and that is the one thing to know before rolling it out. The args annotation holds a checksum once the new build has written it, and the old build reads that annotation with `strconv.Atoi`. The parse fails and `resetKubeAPIServerFlags` returns early, ahead of both the args reset and the annotation write that would put a number back. So after a rollback to `26.3.6-edge` the count-keyed reset is disabled on every TenantControlPlane: a later removal of an `extraArgs` entry no longer clears the flag from the running Deployment, and it stays that way until someone writes a number into `kube-apiserver.kamaji.clastix.io/args` by hand. Findings from a live upgrade on a dev cluster, none of them caused by the diff. The DNS-service-IP check moved out of a CEL rule on the CRD into the admission webhook (`internal/webhook/handlers/tcp_dns.go`), so it now depends on that webhook being reachable where the apiserver used to enforce it by itself. The manager's readiness probe targets port 8081 while the webhook serves on 9443, so a Ready pod does not prove 9443 accepts connections; with `failurePolicy: Fail` on the TenantControlPlane webhook, that window rejects writes rather than letting them through unvalidated. The readiness probe this build adds to kube-scheduler and kube-controller-manager survives a rollback, since the old build neither sets nor clears those two; that one is benign. Seven limits worth naming rather than leaving to be found. The mixed-spelling refusal quotes the entry carrying each spelling, so one entry holding both, `--feature-gates=Foo=true,kube:Bar=true`, is quoted twice in its own message; the refusal is right and the message reads oddly. The unit test that pins the rendered CRD schemas asserts on a count, so a CRD missing from the render altogether fails on an empty comparison rather than naming the CRD that went missing. There is one window during the upgrade itself: the chart used to fold an operator's own `--feature-gates` terms into the entry it rendered, because the old build would otherwise drop them, and it now renders only its own term. Nothing orders a tenant release after the Kamaji operator has rolled, so a `v1.32` tenant that sets gate terms of its own with OIDC on can lose them until the new operator reconciles it, which it then does on its own. The same window takes a `--requestheader-uid-headers` list of the operator's own that lacks `X-Remote-Uid`: the chart now renders its own entry beside it instead of refusing, and the old operator keeps one value per flag name, the chart's, so the operator's header is dropped until the new operator reconciles. Which entry an error message quotes is pinned only where one entry is eligible. Three variables choose between equally eligible entries, the gate dictionary taking the last setting for a name and the uid-headers ones the first, and no test distinguishes first from last when two entries qualify, so a future edit could make a message name the wrong entry of the two without a test noticing. What the chart renders and what it refuses is pinned exhaustively; it is the provenance in three message strings that is not. The block is deliberately inconsistent about two shapes that stop the apiserver equally hard: two `--emulated-version` entries naming kube make it withhold silently, while a kube-prefixed and a bare gate spelling in the same configuration make it refuse the render. The refusal is right where the apiserver rejects the configuration whatever the chart does; the withholding is right where the chart cannot read a version it must render against. They are two rules, not one, and a third that covers both is fair to ask for. Moving the refusal out from under those conditions so it fires everywhere is the obvious third rule and it is not free: the apiserver refuses the mix on `v1.31` too, but from `k8s.io/apiserver/pkg/util/version` rather than from the component-base `componentGlobalsRegistry.Set()` the message names, so hoisting it means rewriting that sentence as well. Reading a flag and its value written as two entries would be better than withholding for it, and is a change of behaviour rather than a correction, so it is not here. And two sentences about `--emulated-version` in `docs/oidc-tenant.md` sit thirty lines apart and read as a retraction in sequence, though each is true at its own layer: `toVersionMap` accepts an element naming a foreign component, and the registry rejects it later. What this leaves open, by name. The guard for a valueless flag at the end of `extraArgs` covers `--requestheader-uid-headers`, `--feature-gates` and `--emulated-version` only; any other flag that takes a value swallows the appended `--authentication-config` the same way, which is an input mistake the chart does not look for. The managed-flag refusal matches names as written, as Kamaji does: an underscore spelling such as `--secure_port` passes both, and the apiserver normalises it to `--secure-port`, where the operator's value wins as the later one, the same as before this change. Rendering `--authentication-config` ahead of the operator's entries would close that class for OIDC, since only the UID flags would then follow them; it changes the rendered order, and so rolls every OIDC tenant that has `extraArgs` of its own, and belongs in a separate change after the OIDC floor move. One pre-existing comment needs checking: the note above `extraArgs` in `templates/cluster.yaml` says that omitting the field would leave the live array untouched under JSON merge patch, which reads against Helm's `createPatch`, where a key the previous manifest carried and the new one does not is patched to null; what that comment was guarding against is still to be verified. The KamajiControlPlane CRD accepts only the plugin names it lists, so a plugin outside that list that the apiserver leaves off by default can no longer be enabled; that limit is Kamaji's, since it owns the flag and restricts the field. A name moved out of an `extraArgs` entry is not checked against that list by the chart, so one outside it fails when helm-controller applies the KamajiControlPlane, with an error that names it, rather than at render; the chart keeps no copy of the list to check it with. A repeated address type in a moved entry is dropped, keeping the first place it appears, because the field is a set. With `oidc.mode: None` the chart does not read `extraArgs` for the OIDC flags at all, and two shapes regress with the bump there: a kube feature-gate term written both bare and `kube:`-prefixed across entries, and two `--emulated-version` entries naming kube. The old build kept one entry per flag, the new one passes both, and the apiserver refuses both at startup. Next to a flag and value written as two entries the chart keeps the refusals that hold whatever the hidden value is, but it does not read positions: a hidden `--feature-gates` term keeps the opt-out refusal off even when it sits before the explicit opt-out and so cannot undo it. What the live run could not exercise: the konnectivity datapath, because the stand had no tenant workers, and `--requestheader-uid-headers`, which comes from the tenant Kubernetes chart and no image swap reaches. Both are covered by render and unit tests only. One shape the old collapse was quietly repairing, with OIDC on, needs a guard now that it is gone. A bare `--requestheader-uid-headers` in `extraArgs` used to be overwritten by the entry of the same name the chart renders, because the collapse kept one value per flag name; the same held for a bare `--feature-gates` on `v1.32`, where the chart renders a gate entry too. A bare `--emulated-version`, or a bare `--feature-gates` on any other version, had no entry of the chart's to be overwritten by and was emitted bare in sorted position, where it took the next flag as its value, so it never worked. Without it those entries reach the apiserver as written, and the apiserver gives a valueless long flag the next argument whatever that argument looks like. What follows the operator's last entry is the `--authentication-config` the chart appends, so the flag swallows it and the AuthenticationConfiguration is never applied. On `--requestheader-uid-headers` the path is accepted as a header name and the control plane comes up with OIDC silently off, failing every login for that tenant with nothing in the cluster to say why; the other two reject it and the apiserver exits. The chart refuses to render that shape and says which entry to fix. Two things in the image build fall out of the bump. The datastore deletion fix carried here as a patch landed upstream verbatim, so its patch no longer applies, and one failing patch fails the whole apply of the directory. The Go module overlay is obsolete for the opposite reason: this tag already carries `golang.org/x/net`, `golang.org/x/crypto` and `go.opentelemetry.io/otel/sdk` above the versions the overlay forced, so keeping it would downgrade all three. What I checked: `helm unittest` passes at 307 cases for the package, with the OIDC suite carrying every shape above, and the image builder stage builds end to end at the new tag with the one remaining patch. Kamaji's own support for Kubernetes 1.36 and 1.37 does not move the tenant version enum here; that is a separate update of the `kubernetes` package. No image reference in `packages/system/kamaji/values.yaml` is touched, since CI stamps that digest. ### Screenshots No UI changes. ### Downstream repositories I walked the trigger map file by file against the diff. The chart templates, the vendored Kamaji chart, the image build and the docs reach nothing downstream. The one generated file that reaches downstream is `packages/apps/kubernetes/values.schema.json`, which gains `controlPlane.apiServer.admissionControllers` and `controlPlane.kubelet.preferredAddressTypes` and a changed `extraArgs` description; both new fields are optional with defaults, so a stored Kubernetes object needs no change, no field is removed or renamed and no version enum moves. The new fields are the one trigger the map fires on: the Terraform provider keeps a hand-written schema. `packages/apps/kubernetes/README.md` feeds the website's managed-apps reference pages, which a bot regenerates on a stable tag. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: model `controlPlane.apiServer.admissionControllers` and `controlPlane.kubelet.preferredAddressTypes` on the kubernetes resource (schema, model, expand/flatten). - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(kamaji): update Kamaji to 26.9.4-edge. Tenant kube-apiserver arguments are now derived from a hash of the control plane's extraArgs instead of from its length, so on the first reconcile after the upgrade every tenant control plane rewrites those arguments from scratch and rolls once. The rebuilt list also orders them differently, Kamaji's own flags sorted first and the operator's kept verbatim after them, so the roll happens whether or not the tenant sets any extraArgs. A tenant with OIDC enabled takes a second roll when the chart's own flags land. Rolling the Kamaji image back to an earlier build does not restore the old behaviour: the older code reads that annotation as a number, gives up when it finds a checksum, and from then on never clears a flag you remove from extraArgs, on any TenantControlPlane, until the annotation is set back to a number by hand. The tenant Kubernetes chart reads every extraArgs entry the way the apiserver accumulates them: a --requestheader-uid-headers list that omits X-Remote-Uid now gets the chart's own element beside it instead of failing the render, a feature gate set in an earlier entry is honoured, and naming the kube feature-gates component both bare and kube:-prefixed fails the render, because the apiserver refuses that configuration outright. Writing --feature-gates or --emulated-version as a flag and a value in two separate entries now keeps the chart from rendering UID flags on its own, because an entry the chart cannot read may hide an opt-out; a --requestheader-uid-headers list of your own still gets X-Remote-Uid beside it, and write --flag=value to get the rest back. Action required for tenants that pass their own apiserver flags: Kamaji now sets a list of flags itself and drops a controlPlane.apiServer.extraArgs entry naming one, where the previous build let that entry win, and the Kubernetes application now refuses to render such an entry, except for the two it moves into a field. A tenant carrying one of the others stops reconciling after the upgrade, and no later change to that tenant applies, until the entry is removed; the entry no longer has an effect either way. The list is in the description of controlPlane.apiServer.extraArgs. An --enable-admission-plugins=value entry is moved into the new controlPlane.apiServer.admissionControllers, a list that replaces the control plane's default set as the entry did, and a --kubelet-preferred-address-types=value entry into the new controlPlane.kubelet.preferredAddressTypes, while that field is empty; nothing has to be edited for them. Before upgrading, list the affected entries with: kubectl get kuberneteses.apps.cozystack.io --all-namespaces --output json | jq --raw-output '["--advertise-address","--client-ca-file","--enable-admission-plugins","--service-cluster-ip-range","--kubelet-client-certificate","--kubelet-client-key","--kubelet-preferred-address-types","--proxy-client-cert-file","--proxy-client-key-file","--requestheader-allowed-names","--requestheader-client-ca-file","--secure-port","--service-account-key-file","--service-account-signing-key-file","--tls-cert-file","--tls-private-key-file","--etcd-servers","--etcd-cafile","--etcd-certfile","--etcd-keyfile","--etcd-prefix","--etcd-compaction-interval","--egress-selector-config-file"] as $owned | .items[] | .metadata as $m | (.spec.controlPlane.apiServer.extraArgs // [])[] | select((split("=")[0]) as $f | $owned | index($f)) | "\($m.namespace)/\($m.name): \(.)"' Entries it lists for --enable-admission-plugins or --kubelet-preferred-address-types need a change only when written as two entries, when they name nothing, or next to a non-empty field of their own. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Configure tenant control-plane admission controllers and kubelet preferred address types. - Choose a cert-manager issuer for webhook certificates; self-signed certificates remain the default. - **Bug Fixes** - Improved OIDC argument handling for repeated, split, quoted, and conflicting settings, with clearer render-time errors. - DataStores are recognized as unused when no tenant control planes reference them. - **Validation** - DataStore endpoints are checked for length, count, and IPv6 address formatting. - Chart-managed tenant control-plane arguments are rejected when supplied through `extraArgs`. - **Documentation** - Updated Kubernetes OIDC and certificate issuer guidance. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
047c715775 |
chore: clean up stray console file and test fixtures
Drop a local kubeconfig that was committed by accident alongside the console sources and ignore the filename so it cannot come back. Replace environment-specific values in the vSphere migration fixtures with synthetic ones: the tests only care about the shape of the inventory record, not where it was captured. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9f857f64e1 |
feat(mariadb)!: generate every user password, root included
users[].password is gone from the chart, its schema and the API type. Every password, root included, is generated into <release>-credentials and kept across reconciles by lookup, and a password left in values is ignored. An upgrade that finds no credentials Secret fails instead of minting new passwords, because mariadb-operator applies root only at datadir bootstrap and would advertise a root it never set. The Secret carries k8s.mariadb.com/watch so the operator applies a changed password without waiting for its requeue interval. A username or database name becomes a Secret key and a resource name, so render-time guards reject a name outside the DNS-1123 rule or past its limit: 80 for a username, as the User CRD allows, and 64 for a database, as the server allows. Names that differ only in "_" and "-" collide on one resource; that is rejected on install only, since an existing release already applied the overwrite. A user in both admin and readonly of a database keeps a single ALL grant under the unchanged Grant name, because renaming a Grant makes the operator revoke the live privilege. A role member past 128 characters, the server's account-name limit, and a database name of 65-80 characters, which the Database CRD accepts but the server does not, are rejected on install and on upgrade only when their Grant or Database does not exist yet, so a release that already applied one keeps upgrading. User CR names and keys are quoted so a username that reads as a YAML scalar stays a string. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2b5877eff9 |
feat(postgres)!: generate user passwords and read them at runtime
users[].password is gone from the chart, its schema and the API type. Every application-user password is generated into <release>-credentials and kept across reconciles by lookup, so an upgraded release keeps the password it already has and a password left in values is ignored. The init-job mounts that Secret and applies each password with ALTER ROLE and a psql variable, so no password is rendered into the init-script Secret that Helm keeps in release history. While bootstrap is enabled the Secret is marked credentials-pending, and NOTES say the generated passwords apply once the init-job runs. Names reach superuser SQL and the init-script shell, so render-time guards reject a username, database or extension name outside [-._a-zA-Z0-9] or past the length PostgreSQL silently truncates: 63 bytes for a role or extension, 54 for a database, whose <db>_readonly role would otherwise truncate and fail the next run. A username over 63 bytes that the credentials Secret already carries still upgrades. users.postgres, the CNPG superuser, is rejected in every bootstrap state. Changing a database's key in values drops the database, so the reject message and the README give the rename that keeps the data: a platform administrator renames it and its roles as the CNPG superuser before the key changes. The init-job cannot hang its Helm hook: the readiness wait ends on elapsed time below activeDeadlineSeconds and the Job never retries, so a purged restore target fails with a diagnostic. The delete-users pass reads managed users from pg_shdescription instead of parsing the version-dependent \du+ layout, which on PostgreSQL 16 and later matched nothing and left a removed user able to log in; it compares names as CREATE ROLE stores them and reassigns a removed user's objects in every database before dropping it. A database without roles renders, and extension names are quoted for hyphenated names and folded to lowercase so they resolve as they did unquoted. A value that carries its own surrounding quotes, the only way a hyphenated extension rendered before, has that pair removed and keeps its case, so such a release still upgrades. The README reads a password with jq, since jsonpath splits a dotted key. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
7a0aab2bdc |
refactor: apply the Go 1.27 modernizers
Run go fix over every first-party module: interface{} becomes any,
hand-written loops become slices, maps, min/max and range-over-int,
errors.As becomes errors.AsType, pointer helpers become new(expr) and
are inlined at their call sites, and so on. Every rewrite keeps
behaviour.
Three rewrites stay out. embedlit flattens embedded-field literals
such as ObjectMeta, syntax only Go 1.27 accepts, which would break
backports to release branches still on Go 1.26. omitzero drops
omitempty from struct fields, and openapi-gen reads omitempty as
optional, so metadata, spec and status would turn required in the
published schemas. And errorsastype rewrote an errors.As whose
target, apierrors.APIStatus, does not implement error, which
errors.AsType rejects at compile time.
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
|
||
|
|
db6c20f841 |
chore(deps): raise Go modules to their latest compatible releases
Several pinned modules carry published advisories: golang.org/x/crypto, net, text and mod, grpc, the opentelemetry SDK and exporters, velero, the etcd client and cel-go. Rather than take only the minimum fixed versions, every module outside the Kubernetes lockstep set moves to its newest release whose own requirements keep k8s.io/* and controller-runtime where they are and fit Go 1.27.1. Where the newest release asks for more, the module stops at the last one that does not: cert-manager v1.19.6, source-controller/api v1.8.5, source-watcher/api v2.1.1, fluxcd meta v1.26.0; helm-controller/api, fluxcd kustomize, gateway-api and external-snapshotter stay put. A few modules stop below their newest release on purpose. cel-go and the etcd modules stay on the lines k8s.io/apiserver v0.35 was built and tested with (cel-go v0.29.2, the version Kubernetes itself ships, and etcd v3.6.15), since the apiserver is their only consumer here. grpc stays on v1.83.2: v1.84.0 predates the fix for GO-2026-6443, a server panic the CSI driver's gRPC server can reach, and no v1.84 patch release carries it yet. kubevirt.io/api and the CDI API stay on the versions upstream csi-driver builds against: from CDI v1.60 StorageSpec.Resources changes type and upstream no longer compiles. velero v1.18 adds RestoreSpec.ResourcePolicy, which the strategy CRD embeds, so the CRD is regenerated. The backup controller only consumes the velero v1 types; the new Queued and ReadyToStart backup phases fall through to its requeue path like any other in-progress phase. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
ec7387109e |
build(go): move the toolchain and builder images to Go 1.27.1
Go 1.26.4 carries standard-library advisories in net/http, net/url, crypto/tls, encoding/asn1 and html/template that govulncheck reports as reachable from the first-party binaries. The shipped stdlib comes from the builder image, so the builder tags move together with the go directive; the digests are the multi-arch indexes that golang:1.27, 1.27-alpine and 1.27-bookworm resolve to at 1.27.1. api/apps/v1alpha1 keeps go 1.26.4. It is a published types-only module, no binary here links its stdlib, and raising its directive would force every consumer onto Go 1.27. linstor-csi v1.11.2 pins grpc v1.81.0, which no longer builds on Go 1.27: golang.org/x/net drops http2.TrailerPrefix there. Its image now raises grpc to v1.83.2 before building, as the kamaji image does for its advisories; the upstream unit tests pass on it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a7dfbe6f8c |
chore(deps): update container image & action digests
Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
09a658d43b |
test(dashboard): pin JWT verification in the token proxy
Only the cold-start path of verifyAndParseJWT was tested; every other test served an empty JWKS, so no token was ever verified and a change that weakened verification would have passed. Cover a valid token and the ones that must be refused: expired, not yet valid, signed by another key, unknown or missing kid, alg none, and HS256 keyed with the RSA public key. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
8a5cfbe1cb |
docs(harbor): say what managementWhitelist overrides and trusts
A per-Ingress whitelist-source-range replaces the controller-wide default that the tenant ingress whitelist sets, so for the portal and API the Harbor list is not an intersection: an address the tenant policy excluded gets in when it is on this list. The description also promised that every other path is restricted, which holds only while the controller sees the real client address. Behind a proxy or with a header-derived client address the check runs against that instead. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
a92e89e969 |
docs(kubernetes): describe the OIDC UID header from v1.33
The tenant OIDC guide, the version field description and the e2e helper comment still described a --feature-gates term rendered on v1.32. Left as they were, they would tell an operator that the chart switches that alpha gate on, which it no longer does. The guide now says what a v1.32 tenant has to write itself to get the UID, and which --emulated-version values fail the render, so an operator who meets that refusal finds it explained. The breaking-changes note in the README said the chart refuses the mixed gate spelling while OIDC is on; with that refusal gone it would promise a render error that no longer comes. The version description returns to its earlier wording in values.yaml and in the update-versions.sh template, because that template rewrites the line on every version bump and would otherwise bring the old text back. make generate carries it into the schema, README, API types and ApplicationDefinition. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
16d23c8a53 |
test(monitoring): preserve OIDC coverage without live BATS suites
Rendering the inner chart alone cannot catch a wrapper dropping tenant OIDC settings. The audience mapper and System-mode reconcile flag also need assertions before the unused live suites are removed. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
46674f80e7 |
docs(kubevirt): state the preconditions of the eviction comments
The comments on evictionStrategy and workloadUpdateMethods described the new behaviour as if it applied to every VMI. The VMI mutator stamps the cluster strategy into the spec at creation, so a VMI started before LiveMigrateIfPossible keeps LiveMigrate: if it cannot migrate, its drain still blocks and the workload updater's eviction is refused until its VM restarts. The worker case pointed at the wrong mechanism too. CAPK reports a non-migratable VMI under evacuation as a terminal failure, and every kubernetes-nodes group has a MachineHealthCheck, so the Machine gets replaced. Where maxUnhealthy holds remediation back the VM restarts in place, but the failure is never cleared, so the Machine is still replaced later. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
4e9f83ac0d |
fix(kubernetes)!: move or refuse extraArgs flags the control plane owns
From 26.9.4-edge the tenant control plane sets a list of kube-apiserver flags itself and drops any extraArgs entry naming one before the apiserver sees the list; 26.3.6-edge let such an entry win. An existing tenant that set, for example, its own admission plugins through extraArgs would lose them on the first reconcile after the upgrade with nothing to say so, and a flag written with its value in a separate entry leaves that value behind as a positional argument the apiserver refuses. Two of those flags have a KamajiControlPlane field the control plane renders them from. An --enable-admission-plugins=value or --kubelet-preferred-address-types=value entry is moved into that field while the field is empty, and left out of the rendered extraArgs, so the tenant keeps its setting across the upgrade without an edit. The move is computed from the values on every render; the last entry wins, as it did while the control plane kept one entry per flag name, and a repeated address type is dropped because that field is a set. The post-upgrade Job that patches extraArgs onto the KamajiControlPlane reads the same list. Every other entry naming an owned flag is refused at render time in every OIDC mode, since the chart cannot stop the drop and the refusal is where the operator learns of it. So are the two moved flags beside a non-empty field, with a value that names nothing or is quoted, and in the two-entry spelling. The list mirrors the Kamaji builder for the shape of control plane this chart renders, and the extraArgs description names the same flags. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0f9b8dc16c |
feat(kubernetes): expose admission controllers and kubelet address types
The KamajiControlPlane carries two lists the tenant control plane renders into kube-apiserver flags: spec.admissionControllers into --enable-admission-plugins and spec.kubelet.preferredAddressTypes into --kubelet-preferred-address-types. The chart rendered neither from the tenant values; the address types were a fixed list. Pass both through. Admission controllers are rendered only when the list is non-empty, so a tenant that sets none keeps the control plane's own default set, and a list replaces that default rather than adding to it. An empty or absent list of address types renders the list the chart rendered before, so a tenant that sets nothing renders exactly as it did. The names are declared as enums copied from the KamajiControlPlane CRD at the provider pin, so the application API refuses a misspelled name where the tenant edits it instead of at apply time. Both fields are optional. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
43cbcf9f46 |
fix(kamaji): anchor the ignore patterns to the chart root
Helm skips .helmignore when it reads a chart out of an archive and honours it when it reads one out of a directory. A pattern with no slash is matched against the base name at any depth, so `hack` reached charts/kamaji-crds/hack, where the CRD templates keep the schema they pull in with .Files.Get. The cluster, which gets the archive, has always had the full CRDs; every local render, lint and package produced three CustomResourceDefinitions with an empty spec, and no check that runs before a cluster sees the chart could have caught a schema defect. There is no hack directory at the top level of this package, so the pattern excluded that subchart and nothing else. `images` is anchored alongside it against the same trap; it matches only the intended directory today and the packaged file list is unchanged by it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
88264216b7 |
docs(kubernetes): restate the OIDC UID flags for the merged reading
The migration note told operators to move the extraArgs entry count in the same edit so the running control plane would rewrite its apiserver arguments; the control plane compares a hash of the list now, so any change already forces the rebuild and no second edit is owed in either direction. The paragraphs describing a value that survives only as the last entry of its flag, and an empty value reaching the apiserver as a bare flag, describe a build that is no longer shipped. What replaces them is the accumulation each flag actually performs: a uid-headers list the apiserver validates as the union of every entry, feature gates settled across entries with the last term winning and one spelling of the kube component throughout, and an emulated version that two entries cannot both name. One contract the earlier text sold is narrowed to what the chart keeps. The render checks run only where the chart renders something, so the three readings it refuses to guess at switch them off as well, and a shape that would otherwise be refused passes through to a control plane that does not start. A flag and value written as two entries is a narrower case, and the page says which checks hold next to one. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
b31aa03a53 |
feat(kamaji)!: update to 26.9.4-edge
The tenant kube-apiserver argument list is no longer collapsed into a map keyed by flag name: entries supplied through the control plane spec reach the apiserver verbatim, duplicates included, and the reset that rewrites them from scratch keys on a hash of the list rather than on its length, so a same-count flag swap no longer strands the old flag on the running Deployment. The datastore deletion fix landed upstream verbatim, so its patch no longer applies, and one failing patch fails the whole apply of the directory. The module overlay is obsolete for the opposite reason: this tag already carries golang.org/x/net, golang.org/x/crypto and go.opentelemetry.io/otel/sdk above the versions the overlay forced, so keeping it would downgrade all three. The etcd.deploy override goes with it. The chart's dependency condition is kamaji-etcd.deploy and the subchart is not vendored here, so the key has never selected anything. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
76b3652297 |
fix(kubevirt)!: evict non-migratable VMs instead of blocking the drain (#4403)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A node drain now completes when a VM on that node cannot live-migrate. The VM is shut down and started again instead of holding the drain forever. VMs that can migrate still migrate. The change is the cluster-wide `evictionStrategy` in the KubeVirt CR, from `LiveMigrate` to `LiveMigrateIfPossible`. The drain is held by virt-api's eviction webhook, not by a PodDisruptionBudget. With `LiveMigrate` and a VMI that is not live-migratable, the webhook denies every eviction with HTTP 429 ([pod-eviction-admitter.go#L147-L150](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L147-L150)). `kubectl drain` retries a 429 until `--timeout`, which is infinite by default. KubeVirt v1.9.0 does not create PDBs for VMIs at all; its disruption-budget controller only deletes old ones ([disruptionbudget.go#L353-L406](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/drain/disruptionbudget/disruptionbudget.go#L353-L406)). With `LiveMigrateIfPossible` the same webhook lets the eviction through for such a VMI ([L152-L155](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L152-L155)). The VM then comes back according to its `runStrategy`. RWO disks are not the only case. A VM with a passthrough GPU is not migratable either, unless the `VGPULiveMigration` gate is on and the GPU is a mediated device ([vm.go#L1310-L1319](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-handler/vm.go#L1310-L1319)). The gate is off in Cozystack, so any `vm-instance` with `gpus` set blocked the drain of its node until now. In KubeVirt a drain and a launcher update go through the same eviction. After a KubeVirt upgrade the workload updater live-migrates outdated VMIs that can migrate and evicts the rest, because `workloadUpdateMethods` lists `Evict` as the fallback ([workload-updater.go#L421-L424](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L421-L424)). The eviction call ([L614-L620](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L614-L620)) hits the same webhook as a drain. So a `vm-instance` VM that cannot migrate either blocks both, which is what happens today, or is restarted by both, which is what happens after this PR once the VM runs under the new strategy. This has a cost. A non-migratable VM now restarts on a drain. That covers VMs on RWO storage and VMs with GPU or host device passthrough. The webhook does not look at who asked for the eviction, so this holds for any eviction through the API, not only a drain. A VM whose disk is bound to the drained node, for example on a LINSTOR class without DRBD, stays down until the node is back. The upgrade itself restarts nothing, and existing VMs switch only after their next restart. KubeVirt writes the cluster strategy into the VMI spec when the VMI is created ([defaults.go#L208-L211](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/defaults/defaults.go#L208-L211)), and the webhook reads that field first ([migrations.go#L116-L122](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/util/migrations/migrations.go#L116-L122)). A VMI that was running before the upgrade keeps `LiveMigrate`, so a drain still waits on it and the updater's eviction still gets 429. After one restart the VMI carries `LiveMigrateIfPossible`. From then on every KubeVirt bump shuts it down through the updater, in batches of 10 every minute, the `batchEvictionSize` and `batchEvictionInterval` set in the CR ([kubevirt-cr.yaml#L139-L140](https://github.com/cozystack/cozystack/blob/25d4a8c951b9fa7249b56ae79328d2cfd4f3a9c0/packages/system/kubevirt/templates/kubevirt-cr.yaml#L139-L140)). Reading the source, a CPU socket increase on such a VM under `vmRolloutStrategy: LiveUpdate` goes the same way, since the `VCPUChange` condition sends it to the updater ([L320-L325](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/workload-updater/workload-updater.go#L320-L325)). I have not run that one. I kept `Evict` even so, because `kubernetes-nodes` workers depend on it. They set `evictionStrategy: External`, and for that strategy the webhook marks the VMI for evacuation instead of evicting it ([pod-eviction-admitter.go#L156-L157](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-api/webhooks/validating-webhook/admitters/pod-eviction-admitter.go#L156-L157)). CAPK sees the mark on a VMI that cannot migrate, drains the tenant node and deletes the VMI ([machine.go#L414-L454](https://github.com/kubernetes-sigs/cluster-api-provider-kubevirt/blob/v0.1.10/pkg/kubevirt/machine.go#L414-L454)). The worker VM has `runStrategy: Always`, so KubeVirt starts it again on the new launcher. That is how a GPU worker or any other non-migratable worker gets the new launcher. Without `Evict` those workers would stay on the old launcher for good. Worker behaviour does not change with this PR, and the new test pins the method list so the fallback is not dropped later. `vm-instance` renders no `evictionStrategy` and has no value for it, so a tenant has no way to opt out of either restart. KubeVirt would honour a per-VM value over the cluster one. I filed #4419 to expose it. I picked this over the options in the issue because KubeVirt already decides migratability per VMI at eviction time. The platform CR does not need to know anything about storage, and a chart shipping a VM does not have to remember an override. Deriving the strategy from the disk access mode in a chart would also miss the GPU case. To get the new drain behaviour for a non-migratable VM that is already running, restart it once after the upgrade. Until then a drain of its node still blocks, as it does today. The comment in the CR is on the same line as the key on purpose. `make update` inserts blocks right before the `evictionStrategy:` line, so a comment above the key would get separated from it. A VM with `runStrategy: RerunOnFailure`, `Manual` or `Once` may stay off after an eviction. If the guest shuts down cleanly on the ACPI request within its grace period, the VMI ends `Succeeded`. `RerunOnFailure` restarts only a failed VMI, and `Manual` and `Once` restart nothing on their own ([vm.go#L1039-L1149](https://github.com/kubevirt/kubevirt/blob/v1.9.0/pkg/virt-controller/watch/vm/vm.go#L1039-L1149)). `Once` stays off even after an unclean stop. Only the default `Always` brings the VM back in every case. What I did not verify. I have not drained a node with such a VM on a live cluster, and I have not watched the updater restart them during an upgrade. Everything above comes from the KubeVirt v1.9.0 and CAPK v0.1.10 source. I did not check how `talosctl upgrade` handles a drain that keeps getting 429. I also did not check how the cluster autoscaler treats a node with a non-migratable VM now that its launcher pod can be evicted. The pod has no `safe-to-evict` annotation, so for a VM on node-local storage that likely comes down to the autoscaler's `skip-nodes-with-local-storage` setting. Finally, I did not check whether the upstream `VMCannotBeEvicted` alert is deployed in Cozystack. For non-migratable VMs it stops firing after this change, since they are evictable now. Closes #4360 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubevirt)!: VMs that cannot live-migrate no longer block a node drain. The cluster-wide eviction strategy is now LiveMigrateIfPossible instead of LiveMigrate. VMs that can migrate still migrate. VMs that cannot, for example with a ReadWriteOnce disk or GPU or host device passthrough, are shut down and started again by their runStrategy. This applies to any pod eviction, not only a drain. A VM whose disk is bound to the drained node stays down until the node returns. A VM that is already running keeps the old strategy until its next restart, so restart non-migratable VMs once after upgrading if their nodes need to drain. The KubeVirt workload update uses the same eviction: once a non-migratable vm-instance VM runs under the new strategy, every later KubeVirt upgrade shuts it down, 10 VMs per minute. A VM with runStrategy RerunOnFailure, Manual or Once that shuts down cleanly on eviction stays off until it is started again. kubernetes-nodes workers are not affected. ``` |
||
|
|
5e3ffbe634 |
feat(harbor): restrict the portal and API to listed source ranges
Harbor serves its web portal and management API on the same host as the registry endpoints, and a tenant had no way to limit who reaches the portal and API without also cutting off image pulls. managementWhitelist restricts every path except /v2 and /service/token, which move to a second Ingress on the same host, so docker login, pull and push keep working from any address. The field is a list of ranges rather than a free-form annotation map: the shared ingress-nginx accepts snippet annotations, so arbitrary annotations from a tenant-writable spec would reach its nginx configuration. Gateway API publishing has no equivalent, so a non-empty list fails the render there instead of being silently dropped. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
2682185bff |
docs(monitoring): warn that renaming a tracingStorages entry replaces its backend
A tracingStorages name is the identity of everything the entry renders: the VTCluster/VTSingle, its Services and PVCs, and its Grafana datasource. Renaming an entry therefore deploys an empty backend and removes the old one, exactly like changing mode, but only mode said so. Give name the same treat-as-immutable note, and state the limits the render guard enforces. Reword the neighbouring mode and retentionDiskUsageBytes descriptions without arrows and dashes, which render awkwardly in the dashboard and in the generated README. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
76445ba9b6 |
fix(monitoring): reject a negative tracingStorages storage
The storage guard only matched zero-shaped values, so -1Gi passed it and the schema's quantity pattern alike, rendered into the VTCluster/VTSingle spec, and left the release failing under retries: -1 once the API server refused the PVC. Refuse a leading minus in the same guard, so the error names the field before anything is applied and matches what the message promises. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
c55ab268e2 |
test(monitoring): remove orphaned OIDC BATS suites
The two pre-Chainsaw monitoring OIDC suites have had no invocation site since the app suite migrated to Chainsaw. Remove them instead of carrying dead tests that force the selector to run every live suite. Keep the existing selector and capture guards for the historical path shape, and update their comments now that the directory is empty. Retain a named chart assertion for the no-KeycloakRealmGroup ownership invariant; the existing count and kind assertions already enforce the same behavior. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
08c535c295 |
fix(monitoring): give kube-apiserver its own scrape size limit (#4427)
## What this PR does kube-apiserver's `/metrics` grows with the number of resources and CRDs, and on a full install it outgrows the 32MB `promscrape.maxScrapeSize` that monitoring-agents sets globally. vmagent then refuses every scrape of the API server, `up` drops to 0, and `KubeAPIDown` and `TargetDown` fire against a healthy control plane while every rule over API server metrics goes blind. The apiserver `VMServiceScrape` endpoint now sets its own `max_scrape_size` (128MB by default, `vmagent.apiserverMaxScrapeSize`). vmagent uses a per-job value in place of the global flag for that job only, so the 32MB guard for every other target is unchanged. The value stays finite on purpose: vmagent reads the whole uncompressed response into memory before parsing, `streamParse` included, so this limit is also a memory ceiling. A new `VMAgentScrapeSizeExceeded` warning fires on `vm_promscrape_max_scrape_size_exceeded_errors_total`, so an oversized target is reported as a size problem instead of only as a target that is down. The counter has no target label, so the description points at the vmagent targets page to find it. Upgrade note: the per-job value replaces the global flag in both directions. An install that raised `promscrape.maxScrapeSize` above 128MB through `vmagent.extraArgs` should raise `vmagent.apiserverMaxScrapeSize` as well. Verified on a dev cluster with the same operator (v0.68.4) and vmagent (v1.136.0): a test vmagent with a 1MB global limit scraped all three API servers with `max_scrape_size: 64MB` on the endpoint and refused all three without it. The new counter reached storage with the `namespace` and `pod` labels the alert uses, and the alert expression returned a series. The `VMRule` passes the operator's admission webhook, which rejects a malformed expression. Guest clusters get the same default, since their HelmRelease overrides other `vmagent` keys only. Fixes #4343 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(monitoring): kube-apiserver is scraped with its own 128MB size limit (`vmagent.apiserverMaxScrapeSize` in the monitoring-agents component) instead of the global 32MB, so a large `/metrics` no longer makes KubeAPIDown fire on a healthy API server. A new VMAgentScrapeSizeExceeded alert reports any target whose response exceeds the scrape size limit. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Kubernetes API server metrics can now use a configurable scrape-size limit, set to 128 MB by default. Leaving the setting empty uses the global limit. * **Bug Fixes** * Added a warning alert when vmagent reports scrape-size-limit errors. The alert identifies the affected vmagent pod and service and provides guidance for finding the target and adjusting its limit. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
15035a915b |
fix(vm-instance): disable helm readiness wait on the VMInstance release (#3735)
## What this PR does A VMInstance is served as a Flux HelmRelease that renders a KubeVirt VirtualMachine. Since v1.5.0 (helm-controller v1.4.3 to v1.5.0) that release is health-assessed by the kstatus poller. vm-instance sets no `healthCheckExprs`, so the VirtualMachine goes through kstatus's own rules in `fluxcd/cli-utils` (`pkg/kstatus/status`): `checkGeneration` returns InProgress for any object whose `status.observedGeneration` differs from `metadata.generation`, and past that check a `Ready=False` condition is InProgress too, so a halted VM is never Current even with its generation in sync. KubeVirt keeps a VirtualMachine's `observedGeneration` behind its `generation` whenever a `spec.template` change is pending a restart, the VM is stopped, or a live-updatable change never converges (kubevirt/kubevirt#18700). So a healthy, Running VM is read as permanently InProgress, the upgrade waits its full timeout, and fails with `timeout waiting for: [VirtualMachine/... status: 'InProgress']`. On releases still carrying the pre-1.5.0 `remediation.retries: -1` + rollback spec, the timeout then reverts the previous manifest, dropping just-applied changes such as opened ports. This sets `release.cozystack.io/helm-install-disable-wait: "true"` on the vm-instance ApplicationDefinition, so cozystack-api generates the HelmRelease with Install/Upgrade DisableWait. The release settles once the VirtualMachine is applied, and VM health is read off the VirtualMachine itself (`status.printableStatus` / `Ready`) and off the WorkloadMonitor the chart already renders, which is what `WorkloadsReady` on the Application reports. This restores the pre-1.5.0 behaviour, under which the VM was never a helm readiness gate. The same mechanism is already used by `kubernetes-rd` and `kubernetes-nodes-rd`. On its own that annotation does not reach a running cluster, so two gaps in cozystack-controller are closed with it. The first is the rollout. `applicationdefinition-controller` hashed each definition's name and spec only, so a change confined to a `release.cozystack.io/*` annotation left the hash unmoved and cozystack-api kept serving the configuration it read at its last start-up, which is the only place those annotations are ever read. The hash now covers them too. Only that prefix, because kubectl and Helm rewrite annotations of their own on every apply and hashing those would roll the api Deployment on every reconcile. The second is the releases that already exist. `applicationdefinition-helm-reconciler` brings chartRef, valuesFrom and labels on existing HelmReleases in line with their definition; it now does the same for the readiness wait. Without it the annotation reaches new instances only, which on an affected cluster is the wrong half: #3734 reports 134 of 136 releases already looping upgrade and timeout, and re-applying every VMInstance by hand was the only way out. The setting is driven in both directions, since cozystack-api rebuilds the whole spec on every write and emits `DisableWait=false` once the annotation is gone. An absent install or upgrade block already behaves as `DisableWait=false` and is materialised only when the definition asks to disable the wait, so a release with nothing to change is not rewritten. Two alternatives were considered. A per-kind `healthCheckExprs` keyed on `status.printableStatus` cannot work: setting it moves the VirtualMachine from kstatus to the CEL status evaluator (`fluxcd/pkg/runtime/cel`), which runs the same `observedGeneration` check ahead of the expressions, and any `healthCheckExprs` added to this kind later is a no-op while the wait is disabled. The typed `spec.release.waitStrategy: legacy` is the narrower option, since the legacy checker never waits on a custom resource, but it still gates the release on the chart's Service, and with `external: true` that is a LoadBalancer the legacy checker holds not ready until it has ingress. So an external VM would still wait on IP allocation, which `WorkloadsReady` already watches. Disable-wait avoids that and is the mechanism this repository already uses for the same class of problem. Tests: a helm-unittest suite (`tests/readiness_gate_test.yaml`) guards that the annotation stays on the rendered ApplicationDefinition, and Go unit tests cover both cozystack-controller paths: the hash moves on a release annotation and stands still on an unrelated one, and the wait lands on an existing release, is withdrawn when the annotation goes, does not materialise empty install/upgrade blocks, does not rewrite a release already in sync, leaves a release alone when the annotation does not parse, and does not reach a release of another kind. Note on existing clusters: no manual step. The definition changes on upgrade, the config-hash rolls cozystack-api, and the helm reconciler brings existing VMInstance releases in line. Releases that predate v1.5.0 also carry a `remediation.retries: -1` + rollback spec, which is left alone: with the wait disabled the upgrade no longer times out, so that remediation no longer fires, and any write through the apps API drops it. Closes #3734. ### Downstream repositories - [x] No downstream repository is affected by this change <!-- Walked the trigger map against the diff. The change is a readiness annotation on an ApplicationDefinition plus two cozystack-controller paths. It does not touch openAPISchema, kind, plural, release.prefix, any app package, any values.schema.json or any installer/platform values key. The website's application-definitions.md describes the config-hash as a checksum "over the sorted definitions", which stays accurate now that the hash also covers the release.cozystack.io/* annotations, and no page documents those annotations today. --> ### Release note ```release-note fix(vm-instance): VMInstance HelmReleases no longer hang InProgress or roll back healthy VMs; the release no longer waits on VirtualMachine readiness, which helm-controller's kstatus poller cannot assess correctly for KubeVirt VMs. Existing VMInstance releases are brought in line automatically on upgrade. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * VM installations no longer wait indefinitely when Kubernetes reports the VM’s observed generation behind its current generation. * Changes to release settings are now detected and applied to generated Helm releases, including install and upgrade wait behavior. * **Tests** * Added coverage for VM readiness and release-setting reconciliation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
cf7a5684d0 |
test(monitoring): make the single-mode VPA absence assertions able to fail
containsDocument with not: true but no any: true is scoped to a single document, so the "no cluster VPA in single mode" and "no VPA when empty" assertions passed regardless of what the render actually held. Add any: true inside the matcher so they scan every document: a mutation that leaks the cluster VPAs into single mode now turns the suite red, and it is green on HEAD. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
37789e1768 |
fix(monitoring): reject zero storage and over-long names in tracingStorages
Two tracingStorages values render cleanly and only fail once the operator
acts on them.
- A zero storage is silently destructive in single mode: the operator's
isStorageEmpty gate checks IsZero() and mounts an EmptyDir instead of a
PVC, the VTSingle reports operational and the release goes Ready, and every
stored span is lost on the next reschedule. Type the storage field as a
quantity so the schema refuses non-quantities at admission, and refuse
every zero spelling ("", 0, 0Gi, 0.0, .0, 0e3, 0m, +0, -0Mi) in the render
guard, since the quantity pattern accepts those.
- A name over 42 characters wedges cluster mode: the operator names the
StatefulSet vtstorage-<name>, and each pod gets the label
controller-revision-hash: vtstorage-<name>-<hash>, whose value is capped at
63 with up to 10 characters of hash. Past that no pod is created, the
VTCluster goes failed and the release stays not-Ready under retries: -1.
Cap the length in the same guard.
- Quote the rendered name, so a numeric name such as 2024 stays a string
instead of failing the apply on metadata.name.
- Rewrite the helper's rationale so it matches the trimmed CRDs and the
pinned operator, and state in the mode field description that the API
cannot enforce immutability on an array-item field, so the note is the only
guard.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
|
||
|
|
05eebbdf4f |
refactor(monitoring): keep tracing rationale in template comments
The `#` prose in the traces datasource and in the HelmRelease health-check
block is not stripped at render the way `{{/* */}}` is, so it shipped into
every tenant's rendered GrafanaDatasource and HelmRelease. Move it into
template comments: the reasoning stays in the chart source but no longer lands
in the emitted objects.
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
Assisted-by: LLM
|
||
|
|
30fcfe6877 |
fix(monitoring): emit each vlogs GrafanaDatasource as its own document
The range over logsStorages emitted its documents with no `---` separator, so two or more entries concatenated into a single document with duplicate keys, each later entry overwrote the earlier one, and every log store but the last silently got no datasource. Add the separator inside the range, matching the traces datasource template. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
f20bd7b334 |
fix(monitoring): validate tracingStorages entries and flag mode/retention hazards
The installed VictoriaTraces CRDs are the trimmed ones (crds.plain: false), so their spec is x-kubernetes-preserve-unknown-fields and a bad value reaches the operator instead of being refused by the API server. Catch the ones a render guard can, and document the one it cannot. - name is used verbatim as the VTCluster/VTSingle and GrafanaDatasource metadata.name, so an invalid name is a late admission error and a duplicate silently overwrites the first entry (last write wins in the range). Guard it against an RFC 1123 label and a uniqueness set. - retentionDiskUsageBytes maps to the operator's BytesString field, whose UnmarshalJSON rejects anything outside ^[0-9]+(kb|mb|gb|tb|KB|MB|GB|TB|KiB|MiB|GiB|TiB)?$, notably not the Gi grammar of the sibling storage field. The operator then marks the CR failed and the release fails well after the value was written. Refuse it at render with the upstream pattern instead. - changing mode re-identifies the object instead of reconfiguring it, so switching single to cluster prunes the VTSingle and the operator garbage-collects its PVC, and cluster to single (or dropping the entry) leaks the vtstorage PVCs until the release is deleted. A stateless template cannot see the prior mode, so record the loss and the one-way migration in the field description. The name, uniqueness, mode and retention checks share one pass in a helper; the inline mode guard in vtraces.yaml moves there. The retention description also notes that the cap is per vtstorage replica in cluster mode. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
41aa2aad1d |
feat(monitoring): add VPA for VictoriaTraces backend components
vpa.yaml sized only metrics and logs components; the trace components got no VerticalPodAutoscaler, so they sat at the operator's static defaults with no way to size them (tracingStorages has no resource knobs and the template hardcodes resources/replicaCount). Add VPAs per tracingStorages entry, branching on mode: cluster mode gets vtinsert and vtselect (Deployments) + vtstorage (StatefulSet), single mode gets vtsingle (Deployment). Kinds verified against victoria-metrics-operator v0.68.4. updateMode: Initial like the rest of the file (no eviction, because of the Cilium IP-reuse race), which is also required for VTSingle's plain RWO PVC (an evicting VPA would deadlock on multi-attach). Bounds are hardcoded, without the dead min/maxAllowed conditionals the logs entries carry (tracingStorages declares no such properties). Empty tracingStorages renders no trace VPA. Tests cover the cluster three-VPA split, the single-mode single VPA with its updateMode, and off-by-default. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
00ff19ec5c |
fix(monitoring): default tracing off and defer fail-on-empty guard
tracingStorages had two defaults (extra + system), so 'tracingStorages: null' silently backfilled the inner default and a VTCluster rendered anyway, while 'tracingStorages: []' aborted the whole monitoring release. There was no clean 'off', and the default shipped a 6-pod VTCluster per tenant for a pipeline that nothing writes spans into yet. - Default tracingStorages to [] in both packages/system/monitoring/values.yaml and packages/extra/monitoring/values.yaml, so absence has nothing to fall through to and the generated schema advertises the shipped (off) state. - Drop the fail-on-empty guard in templates/vtraces/vtraces.yaml. Nothing produces spans without a collector, so an absent backend drops no data. The guard belongs with the collector, where it becomes correct: an enabled producer with no backend really would drop spans, the logsStorages #3181 rationale. - healthCheckExprs stay unconditional (Flux evaluates them only against CRs present in the inventory, so they are inert when nothing renders). - Tests: replace the empty-fails case with off-by-default (renders nothing); set tracingStorages explicitly where cases relied on the old default. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |
||
|
|
dbe7a732ad |
test(monitoring): cover VTSingle PVC cleanup label + single-mode disk cap
Single mode stamps the release-scoping cleanup label via spec.storageMetadata, a different field than the cluster path (spec.storage.storage.volumeClaimTemplate), and it was untested, so a dropped storageMetadata block would orphan the VTSingle PVC on tenant deletion with no test to catch it. Add the VTSingle cleanup-label assertion and a single-mode top-level retentionMaxDiskSpaceUsageBytes assertion. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> Assisted-by: LLM |