mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-10 06:24:09 +09:00
72ff22dc9ebd51c899f024ebefec111d8a36b7b1
4702
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
9e8e10c307 |
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
c6c107fd08 |
docs(backups): state when the gate compares endpoints in the values reference
The values reference read as if the credentials endpoint were always compared and an object without the endpoint annotation never forced a render. Operators sizing the force counter or choosing a custom systemSecretName need both conditions spelled out. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
dc909c778c |
fix(backups): compare the credentials endpoint only for the forced bucket's Secret
The gate forces bucket-<bucketName>-system, which renders the credentials Secret of that bucket's system user and nothing else. The COSI Secret name was derived from systemSecretName instead, so a value naming another bucket's Secret made a lagging endpoint there force the platform bucket release every throttle window without ever refreshing the Secret that was compared. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Alexey Artamonov
|
d0f47f7dd5 |
fix(backups): compare the credentials endpoint only for a Secret derived from COSI
backupStorage.systemSecretName may name a Secret the bucket chart does not render from a COSI Secret: the raw COSI Secret itself, which the projector accepts, or any Secret without the -credentials suffix. The COSI Secret name was then the source name unchanged, so the gate compared a Secret with itself and either reported it stale on every tick or failed the check, and stopped repairing missing default objects as well. Skip the comparison when there is no distinct COSI Secret or no endpoint key to compare. The gate compares the annotation with what the chart's endpoint helper renders, and the helper renders a present but blank endpoint as a bare https:// instead of falling back, so the gate has to compute the same value or its expectation and the chart disagree for that Secret. The gate's docs, metric help and log lines named only missing objects, so they did not cover a stale endpoint they now report. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Kirill Ilin
|
cb3fcd3357 |
feat(operator): let the platform source trust a private registry CA (#5237)
## What this PR does The operator creates the `cozystack-platform` OCIRepository from its own command-line arguments and rewrites the whole object on every start; `cozystack-packages` copies that spec. When the platform source is a registry whose certificate is issued by a private CA (an internal pull-through cache in an air-gapped install, for example), there is no durable way to make Flux trust it: a `certSecretRef` patched onto the live object is dropped on the next operator restart, source-controller then fails TLS, and platform updates silently stop arriving. This adds `--platform-source-cert-secret` to the operator and `cozystackOperator.platformSourceCertSecret` to the installer chart. The named Secret in `cozy-system` is set as `spec.certSecretRef` on the generated OCIRepository, so it holds `ca.crt` for a private CA and/or `tls.crt`+`tls.key` for client certificate auth, with the Flux key layout. `cozystack-packages` inherits it through the existing spec copy. The flag is OCI only. `GitRepository` has no `certSecretRef` (Flux reads a Git CA from the `ca.crt` key of the `secretRef` Secret), so a Git platform source with the flag set is refused at startup instead of having its CA silently ignored. Tests: unit tests for the generated spec, a regression test that the CA survives the operator's update of an existing OCIRepository, a test that a Git source with the flag writes nothing, and a helm-unittest suite for the new chart value. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/791 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The new installer value is optional, so `ansible-cozystack` is not reached. The website's air-gapped guide (`content/en/docs/next/install/kubernetes/air-gapped.md`) added the private CA to the nodes only, while Flux fetches the platform source itself; cozystack/website#791 adds a step for `platformSourceUrl`, `platformSourceSecret` and `platformSourceCertSecret`. ### Release note ```release-note feat(operator): the platform source can be served from a registry behind a private CA. Set `cozystackOperator.platformSourceCertSecret` in the installer to a Secret in `cozy-system` with `ca.crt` (and optionally `tls.crt`/`tls.key`); the operator sets it as `certSecretRef` on the `cozystack-platform` OCIRepository, so it survives operator restarts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added the `platformSourceCertSecret` setting for OCI platform sources. It allows a Secret containing certificates for private CA trust or client authentication to be configured separately from registry credentials. * The configured certificate Secret is applied to the OCI source and inherited by its package source. * **Configuration** * The setting is empty by default. Git platform sources do not support a separate certificate Secret; use the existing platform source Secret for Git TLS certificates. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Kirill Ilin
|
0a448963e3 |
docs(installer): say a CA-only cert secret must be Opaque
kubernetes.io/tls requires both tls.crt and tls.key, so a Secret that carries only ca.crt is rejected under that type. Assisted-by: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
Kirill Ilin
|
fbfc270c71 |
test(installer): tie each platform source flag to its own value
Both secrets were only ever set together, so a cert flag keyed on platformSourceSecret stayed green. A pull-through cache behind a private CA usually needs the CA without credentials, which is exactly the case such a slip breaks. The absence checks match the flag by prefix, since a whole-element comparison misses the flag rendered with any value. Assisted-by: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
myasnikovdaniil
|
970191dc8a |
fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does A ClickHouse `RestoreJob` restored the newest archive under the app prefix, whichever `Backup` it named. Strategy pod generated the archive name with `date -u` and nothing was recorded on the `Backup`, so restore could only list the remote archives and take the last one. Restoring an older `Backup` brought back the latest data. Now the controller names the archive before it creates the backup Job (`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so retried pods write the same one), passes it to the pod as `.ArchiveName` and records it in the `Backup` `driverMetadata`. Restore takes exactly that archive and fails with its name when storage has no such archive. The name is recorded only when the Job's pod got it, so a Job left from the previous controller during an upgrade, or a custom strategy with its own names, leaves no name instead of a wrong one. A retried pod reuses the name, so it waits for an attempt the sidecar still runs (no 423) and asks for `resume`. A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after the schedule slot, so a `BackupJob` created again under a used name met the earlier run's `Backup` when its Job completed, took it as its result and left the archive it just wrote recorded nowhere. Now it fails if that `Backup` records another archive, and the message names both. A retry of the same run still gets its own `Backup` back. Old `Backup`s have no recorded name and stay restorable: the pod logs a `WARNING` and takes the newest archive stamped no later than `takenAt`, and fails when nothing qualifies. While the chart rolls out, previous controller pod can still render the new template and pass no `takenAt`, so a restore of a `Backup` without recorded name fails before it touches any table, and works when created again after the rollout. Fixes https://github.com/cozystack/cozystack/issues/5199 Tests are go unit tests for naming, recording, a reused `BackupJob` name and restore render, and `hack/altinity-backup-strategy.bats`, which runs the shipped script against a stubbed clickhouse-backup API (the restore tests go red on the old template). Checked on a live ClickHouse with two backups of 1500 and 2000 rows: restore of the older one by recorded name ends with 1500 rows, fallback for an unnamed older backup picks the same archive with the warning, a missing archive fails without touching the table, a retry over an existing local backup completes under the same name. The controller binary itself was not run on the cluster, only its rendered output in the pod. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backups now use a consistent archive name tied to the backup job, and completed backups retain that name for future restores. * Restores can select the recorded archive and backup timestamp. If no archive was recorded, the system selects the newest eligible archive for the application. * **Bug Fixes** * A restore now fails when its recorded archive is missing instead of silently selecting a different one. * Backup requests avoid starting a duplicate action when the matching action is already in progress. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
18779dddfc |
fix(backups): report sizeBytes for MariaDB and CNPG backup artifacts (#5223)
## What this PR does Fixes https://github.com/cozystack/cozystack/issues/4316. Continues https://github.com/cozystack/cozystack/pull/5015 by androndo on a separate branch. All ten of their commits are kept as they were (authorship, messages and order, only rebased on current `main`). MariaDB and CNPG backups stored in S3 report `Backup.status.artifact.sizeBytes`, the size of the dump mariadb-operator uploaded or the total of the barman base backup, and the field stays unset when the size cannot be attributed to that one backup. Four commits are added on top: - `test(backups)`: operator Backup fixtures carry the condition mariadb-operator writes on success, `Complete=True` with reason `JobComplete` and message `Success`. Operator sets `Complete=True` for a failed Job too (reason `JobFailed`), so a fixture without reason describes a state that never happens. https://github.com/cozystack/cozystack/pull/5207 makes the driver check that reason, the tests pass with and without it. - `refactor(backups)`: the MariaDB matcher returns the matched dump (key, file name, timestamp from the name, size, LastModified) and tells "not found" from "ambiguous", and the Secret lookup and S3 client live on a small value any reconciler can build from its own client. A MariaDB restore needs this to restore the dump of the Backup it was given and not the newest one under the prefix. `sizeBytes` does not change here. - `fix(backups)`: the probe reads bucket, prefix, endpoint, credentials and TLS from the operator Backup's own `spec.storage.s3`, not from the strategy as it renders now, so editing the strategy during a backup does not send the probe somewhere else. - `fix(backups)`: a dump is attributed only when its ownership is proven. Before, a backup pod with a clock more than a minute behind, right after a previous run, got that run's dump size (666 bytes for a 123-byte dump in a test). Now the dump has to be the only one that either clock places in the operator Backup's lifetime, both clocks have to agree, and neither may place it in the lifetime of another finished operator Backup with the same bucket and prefix. Otherwise the size stays unset. Two cases are still not covered. First, this run's own dump is missing from the listing while the operator Backup of a neighbour was deleted or has not finished, then the neighbour's dump can be taken. Second, backup pod clock and object store clock are both off the same way by more than the minute of slack, which moves a finished neighbour's dump into this run's window, and nothing the clocks record tells them apart. A run whose Job failed has no dump of its own and reaches the probe only because the driver takes `Complete=True` for success, https://github.com/cozystack/cozystack/pull/5207 fixes that. Of the open threads on the original PR, the cache reads, the dump file name layout and the default AWS endpoint test were already handled by its later commits, the clock skew one is the last commit here and the failed Job case is https://github.com/cozystack/cozystack/pull/5207. `minio-go` is new to this tree, this PR adds it as a direct dependency and `internal/backupcontroller/artifact_size.go` is its only importer. The open Bucket strategy PR https://github.com/cozystack/cozystack/pull/4235 picks the same client at the same version. Each clock skew case, and a run whose own dump is gone while the neighbour's operator Backup still exists and has finished, have a unit test that fails on the old matcher, and every new guard has a mutant that turns a test red. I also ran the matcher against a real bucket after four MariaDB backups in a row. Runs that finished more than a minute apart mapped to the dump their operator Job logged, the two back to back came back unset instead of taking each other's dump, and with a run's own dump hidden from the listing the matcher refused the neighbour's dump that the old one returned. ### Screenshots N/A, no UI changes. ### Downstream repositories The only API change is the description of the existing `Backup.status.artifact.sizeBytes` field, no new field, value, default or enum, no package, values or tooling change, and neither cozystack/website nor cozystack/terraform-provider-cozystack references `sizeBytes`. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): MariaDB and CNPG (Postgres) backups stored in S3 now report `status.artifact.sizeBytes`: the size of the dump mariadb-operator uploaded, or the total size of the barman base backup (WAL archive excluded). A MariaDB backup takes the size of the one dump that falls inside its run on both the backup pod's and the object store's clock and inside no other finished run, and the size is left unset rather than set to 0 whenever it cannot be measured or attributed. Velero-based backups intentionally keep it unset. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup artifacts now report the stored size attributable to an individual MariaDB or CNPG backup when it can be determined. Shared or ambiguous storage may leave the size unset. * **Bug Fixes** * Improved handling of backup artifact creation and lookup errors, including retrying when an existing backup cannot be read and failing jobs when artifact creation is rejected. * **Documentation** * Clarified that an unset size means unknown; zero does not indicate an unknown size. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Kirill Ilin
|
7f1227e3bd |
feat(installer): expose platformSourceCertSecret
Installing from a registry behind a private CA needs the operator's new --platform-source-cert-secret flag; without a chart value it can only be reached by editing the Deployment, which the next helm upgrade reverts. Assisted-by: LLM Signed-off-by: Kirill Ilin <stitch14@yandex.ru> |
||
|
myasnikovdaniil
|
e5bb3cda07 |
fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does Found while checking for other places with the failure from https://github.com/cozystack/cozystack/issues/4968: data that grows with configuration is passed to a process as one argument, and Linux allows 128 KiB for that. This one is in the `kubernetes` app and there is no issue about it yet. The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users` entry into its `sh -c` script, 526 bytes per user. I rendered current main with `helm template` and users like `user123@example.org`: 238 users give a 130,889 bytes script that starts, 239 give 131,415 bytes and `exec` fails with E2BIG. Then the container can't start, the post-install and post-upgrade hook fails after its retries and the HelmRelease of the tenant cluster stays unready. Schema has no limit for `oidc.users` and each person needs own entry, so a cluster shared by 240 people gets there. Now the bindings and the list of their names are rendered by the new `oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings` that the Job mounts, so the script is 6,380 bytes for any number of users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417 bytes each. Names, labels and roles of the bindings stay the same. I compared the render of old and new chart for eight emails (ones YAML reads as a boolean or null, a number, a quote, an apostrophe, unicode and two plain) and the bindings are identical, so existing ones are updated in place and nothing is orphaned. The prune step reads the same ConfigMap. If the list is not mounted the Job stops before it deletes anything, because `grep` on a missing file exits 2, `! grep` is true and every binding of the release would be pruned. A release with no users applies nothing (`kubectl apply` fails on an empty file) and prunes all stale bindings, same as before. `hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the script is under the limit without any email in it and that every user has one binding with the old name and role. Three more tests run the rendered script against a fake `kubectl` and check what it applies and deletes. The prune condition forced to true or to false passes all helm-unittest cases, so this part is tested in bats. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one chart template, its tests and a bats file): no package added or removed, no values, schema or default changed, no output Secret or Service renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Kubernetes OIDC configuration now supports per-user role bindings, with administrator and viewer access assigned according to each user’s role. * OIDC role bindings are kept in sync with the configured user list, including removal of obsolete bindings. * Deployments with no OIDC users or with OIDC mode disabled continue to avoid creating user bindings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
ab13f9f72c |
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook (#5212)
## What this PR does Fixes https://github.com/cozystack/cozystack/issues/4968. The cilium IPAM migration hook passed the whole `kubectl get ciliumnodes -o json` to `jq` as one argument, and Linux allows 128 KiB for a single argument. Cilium copies Node labels onto every CiliumNode (22 to 30 KiB per node with Talos, NFD or GPU labels), so the list is over the limit within six nodes. `jq` can't start (`Argument list too long`), the hook Job fails before it moves any node, `cilium` HelmRelease loops on `UpgradeFailed` and everything that depends on networking stays in `DependencyNotReady`. The hook is rendered on every upgrade of cilium in `cluster-pool` mode, so a cluster that already finished the migration hits the same on its next values change. The failing step only logs the pods that still hold an old Ingress IP, but it runs before the node loop, so on a cluster that is not migrated yet it blocks the migration itself. Now the list goes to `jq` on stdin together with the IP objects, same as the other `jq` calls in the script get their input. The new test in `hack/cilium-ipam-migration.bats` makes six CiliumNodes over the limit (one 24 KiB annotation each instead of labels, and it checks the list is over 128 KiB even printed compact). It fails on the old script with status 126 and passes now. It also checks that each pod is named with the node whose Ingress IP it holds and that a pod on an unrelated address is not named, because a join that always names the first node, or reports every IP object, passed the old tests. I did not take two suggestions from the issue. Skipping the hook when every CiliumNode is in the pool changes what one existing test pins (the hook adds the pool to `excludeIps` in that case) and doesn't help clusters that are not migrated yet. A value to disable the hook leaves the default broken. `cozystack-operator` rewriting `disableHooks` and helm-controller rolling the DaemonSet on every retry are other components. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one hook script and its bats test): no package, values, schema, default or output name changed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(cilium): the IPAM migration hook no longer fails with "Argument list too long" on clusters with more than a few nodes, which blocked the cilium upgrade and every package that depends on networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Improved IP address migration handling for environments with large node lists, avoiding failures caused by operating system argument-size limits. * Migration continues to identify affected pods by node and distinguish unrelated pods. * **Tests** * Added coverage for migrations involving large node lists, multiple IP objects, and unrelated pods. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
cd91138f33 |
fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither copy is refreshed. The COSI sidecar writes BucketInfo once per BucketAccess, so the credentials Secret rendered from it keeps its endpoint, and the Velero BSL and the default Strategy CRs keep whatever endpoint the last render of backupstrategy-controller saw. The default objects gate only acted on objects that were missing, so after a publishing.host change every backup kept going to the old host with nothing reporting it. The gate now also forces a render when a copy is stale: the bucket's <bucket>-system release when the credentials Secret's endpoint differs from the COSI Secret it is rendered from, and backupstrategy-controller when the endpoint an object was rendered with differs from the credentials Secret's. That endpoint sits at a different path in every kind, in some only as a bare host, so the chart stamps the rendered value on each object as backups.cozystack.io/endpoint and the gate compares that. Objects rendered before the annotation existed count as stale until one forced render has gone through, which stamps it. The COSI Secret itself still holds the endpoint the BucketAccess was granted with; it follows a host change only once the BucketAccess is re-granted. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
Aleksei Sviridkin
|
c1ba05574f |
feat(monitoring-agents): keep application identity in container log streams (#5217)
## What this PR does
Makes Fluent Bit keep the pod lineage labels
`apps.cozystack.io/application.{group,kind,name}` on container log
lines, as the stream fields
`apps_cozystack_io_application_{group,kind,name}`, for both the default
container-log output and the per-tenant routed output.
Why: the per-application logs scope in the design proposal [Tenant
application logs](https://github.com/cozystack/community/pull/95)
(section 6) filters by these stream fields. Today the kubernetes filter
fetches pod labels, but they are dropped together with the rest of the
`kubernetes_*` keys before output.
What changes in the pipeline:
- A second `nest` lift of the pod labels, then a rename of the three
lineage keys to their final names before the existing `Remove_wildcard
kubernetes_` runs. All other labels are still removed. Every lifted key
is prefixed, so no other label can collide into the identity names.
- The three keys are added to `_stream_fields` of both container-log
outputs. Events and audit pipelines are unchanged.
- A parsed JSON log line is kept under `log_processed` (already on
`main`), so a line's content cannot create the identity keys.
- Guest Kubernetes clusters are not affected: their monitoring-agents
release replaces the filters and outputs wholesale. A regression test in
`packages/apps/kubernetes` pins that guest streams carry no identity
fields.
Effect on existing streams: VictoriaLogs treats a missing stream field
like an empty one, and the rename writes nothing when the label is
absent. Pods without lineage labels keep exactly the same stream. Pods
with lineage labels get a new stream once after the rollout; their
earlier lines stay reachable by namespace.
#### How it was tested
- helm-unittest for `monitoring-agents` (lift order, rename before
`Remove_wildcard`, stream fields on both outputs, events and audit
untouched) and for `kubernetes` (guest streams unchanged). The
events/audit guard was mutation-checked.
- Verified on a test cluster: real pod log lines arrive, Redis pods'
lines carry the three `apps_cozystack_io_application_*` stream fields,
and Fluent Bit logs show no errors.
#### Known limitation
Guest clusters write to the parent log store without authentication, so
a guest administrator can forge these stream fields as well as the
namespace field, within the store their tenant shares. This is the
write-path trust risk already documented in the proposal.
### Screenshots
No UI changes.
### Downstream repositories
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
Downstream: no trigger matched. Candidate: none affected, because the
diff changes `packages/system/monitoring-agents/values.yaml` (a system
package with no schema) and adds tests; the file under
`packages/apps/kubernetes` is a test only, so the app's `values.yaml`
and `values.schema.json` are unchanged.
### Release note
```release-note
feat(monitoring-agents): container log lines of pods that belong to a Cozystack application now carry `apps_cozystack_io_application_{group,kind,name}` stream fields in VictoriaLogs. Streams of pods without application labels are unchanged; streams of application pods start a new stream once after the upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **New Features**
* Container-log streams now include application group, kind, and name
metadata for root and tenant stores.
* Guest monitoring streams include pod name, container name, and
namespace fields.
* **Bug Fixes**
* Application metadata is added to container-log streams without
changing the existing stream fields for event and audit outputs.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrey Kolkov
|
523d74a503 |
fix(backups): retry instead of failing when an existing Backup cannot be read
When Create collides with a Backup an earlier reconcile already made and the read that follows fails, the drivers failed the BackupJob, although the backup and its Backup both exist. Return a distinct error instead, which the drivers hand back to the controller so the reconcile is retried with backoff and then finds the Backup. minio.New already refuses an endpoint with a path prefix and accepts a trailing slash, so drop the duplicate check in the shared client; the CNPG probe keeps its own check because it parses the URL itself and would otherwise drop the path before minio sees it. Say in the sizeBytes API doc what the value covers, such as the base backup without the shared WAL archive for CNPG, and that strategies which cannot attribute stored bytes to one backup leave it unset. Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
d4ad5612d3 |
fix(backups): report sizeBytes for MariaDB backup artifacts
Backup.status.artifact.sizeBytes was only ever set by the etcd strategy, so MariaDB backups always arrived without a size. mariadb-operator writes each logical dump as <prefix>/backup.<RFC3339>[.<compression>].sql under a prefix shared by every backup of the application, and the URI the driver records is synthetic, so there is no single object to HEAD. On completion the driver now lists the prefix with the tenant credentials the operator Backup was rendered with and takes the size of the one dump whose timestamp falls inside the operator Backup's lifetime. When the bucket cannot be read or the window does not match exactly one dump, the field stays unset. The API doc now states that an absent sizeBytes means "unknown", and the Velero strategy documents why it leaves the field unset: kopia stores data content-addressed across backups, so there is no per-backup stored size. Refs #4316 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov Daniil
|
47b497194e |
fix(backups): restore the ClickHouse archive a Backup was taken as
The Altinity strategy named each archive inside its Pod with date -u and recorded nothing on the Backup, so a restore could only guess: it listed the remote archives under the application's prefix and took the newest. A RestoreJob of an older Backup therefore restored whichever backup had run last. The controller now names the archive before it creates the backup Job, from the BackupJob's persisted StartedAt, so every reconcile and every retried Pod of the Job writes the same one. The name is recorded in the Backup's driverMetadata next to the strategy parameters, and a restore renders it into the strategy Pod, which restores exactly that archive and fails, naming it, when remote storage no longer holds it. The name is recorded only when the Job's Pod was given it, so a Job created by an earlier controller, or a strategy that names its own archives, leaves the Backup without one instead of pointing it at an archive that does not exist. Because a retried Pod reuses the name, it waits on an attempt the sidecar is still running instead of being refused with 423, and asks for resume, so a local or partial remote backup an earlier attempt left is completed rather than refused as existing. Backups taken before names were recorded stay restorable: the Pod picks the newest archive whose name is stamped no later than the Backup's takenAt, which is set only after the backup Job finished, and logs a warning that the choice is a best effort. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
7b7444feaf |
fix(kubernetes): move the OIDC users out of the bootstrap Job's script
The bootstrap Job wrote a ClusterRoleBinding per oidc.users entry into its script, and the script reaches the container as one `sh -c` argument. Linux caps a single argument at 128 KiB, which about 240 users cross at roughly 530 bytes each. The container then cannot start, the post-install and post-upgrade hook fails, and the tenant cluster's HelmRelease stays unready. The values schema puts no bound on oidc.users, so a shared cluster reaches that size with ordinary per-person entries. The bindings and the list of their names now reach the Job through a ConfigMap it mounts, so the script is the same size for any list. Names, labels and roles of the bindings are unchanged, so the bindings already in a tenant cluster are updated in place and none is orphaned. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
a2d0e4cb50 |
fix(cilium): pass the CiliumNode list to jq on stdin in the IPAM hook
The hook fed the whole `kubectl get ciliumnodes -o json` to jq as one argument. Cilium copies every Node label onto its CiliumNode, 22 to 30 KiB per node on Talos or with NFD and GPU labels, so the list passes the 128 KiB Linux allows a single argument within six nodes. jq then fails to start with "Argument list too long", the hook Job fails before it moves any node, and the cilium HelmRelease loops on UpgradeFailed with the HelmReleases behind it stuck in DependencyNotReady. The hook renders on every upgrade of cilium in cluster-pool mode, so a cluster that already finished the migration hits the same failure on its next values change. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
ZverGuy
|
95dc591198 |
test(kubernetes): keep application identity out of guest log streams
Guest pods are tenant-controlled, so their lineage labels are not a trustworthy application identity, and the platform must not promote them to the stream fields the parent scopes application logs by. The guest release replaces the monitoring-agents filters and outputs wholesale, which is what keeps the parent's label lift out of guests; pin that so a later change to the guest overrides cannot bring it in. This only covers what the platform ships. A guest admin can still edit the guest's fluent-bit or write to the parent's vlinsert directly; that needs a write-path control, which the logs proposal leaves open. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
ZverGuy
|
e5f1714836 |
feat(monitoring): keep application identity on container log lines
A reader that scopes logs to one application needs the owning
application on every line, and the lineage webhook already puts it on
the pod as apps.cozystack.io/application.{group,kind,name}. Fluent Bit
fetched those labels and then dropped them with the rest of the
kubernetes map, so the store could only be scoped by namespace.
Keep the three labels as apps_cozystack_io_application_{group,kind,name},
the names the metrics side uses, and make them stream fields on every
container-log output so the scope is an indexed stream filter. A pod
without the labels produces no such fields, and VictoriaLogs builds the
stream from the fields present, so existing streams keep their
identity; a labelled pod starts a new stream once.
Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
|
||
|
ZverGuy
|
7de70ac2fc |
feat(monitoring-agents): export PVC labels from kube-state-metrics
The tenant metrics API scopes a query to one application by looking up its pods and PVCs through their apps.cozystack.io/application.* lineage labels. kube-state-metrics emits kube_persistentvolumeclaim_labels only for resources on --metric-labels-allowlist, so without PVCs there the PVC half of that lookup always comes back empty and volume series of an application fall out of its scope. All labels are allowed, as for pods and deployments: kube-state-metrics emits one series per PVC whatever keys are listed, and PVCs are far fewer than pods, which already export every label. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
myasnikovdaniil
|
4087218a02 |
docs: move the repo-only guides onto the website (#5034)
## What this PR does Seven guides under `docs/` were operator facing and written alongside the code, and left in the source tree where no user looks. They are on the website now, so this deletes them here and repoints every reference. ``` docs/operations/backup-classes.md -> operations/services/backup-classes docs/operations/resource-presets.md -> guides/resource-management docs/storage-immutability.md -> operations/storage-class-immutability docs/operations/system-memory-limits.md -> operations/configuration/system-memory-limits docs/gateway-classes.md -> networking/gateway-classes docs/observability/distributed-tracing.md -> operations/services/monitoring/distributed-tracing docs/operations/seaweedfs-capacity-for-backups.md -> operations/services/object-storage/capacity-for-backups ``` `docs/release.md`, `docs/deprecating-applications.md` and the two SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`, a migration and a bats suite read them by path, which is the test from #3777. References point at `https://cozystack.io/docs/v1/...`. The site rewrites that short prefix to the latest version in `layouts/404.html`, so the links follow releases and pin none. Backup Classes, resource management and `storageClass` immutability exist in the latest version already. The other four pages are only in `next/` until 1.7.0 is promoted, and their links start to resolve then. The generated mongodb files come from `values.yaml` through `make generate` with cozyvalues-gen v1.7.0, the version pinned in `.github/workflows/pre-commit.yml`. Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114) and kubernetes-nodes (223), go tests for backupcontroller, presets and the operator, rd-presets-check, the migration-50 bats suite. ### Downstream repositories - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#763 Walked the rest of the trigger map against the diff. No provider follow-up: the mongodb `values.schema.json` diff is one `description` string, with no field added, removed or renamed, no enum, no default, and no `kind`, `plural` or `release.prefix` change. `packages/core/platform/values.yaml` changed in comments only and no key, so the hand written platform-package table on the site is unaffected. Please merge after cozystack/website#763, the links in the code only resolve once those pages deploy. ### Release note ```release-note docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there. ``` |
||
|
Aleksei Sviridkin
|
a2cc9a3016 |
fix(dashboard): link a tenant row to its detail page (#4004)
## What this PR does Fixes #3822 Tenant list showed name as plain text and the only link in a row was Edit button, so tenant detail page with its tabs and Delete was reachable only by opening edit form and cancelling out of it. Name cell is a link to that page now, it appears exactly where Edit appears and points to the same CR. Plain click switches selected tenant to the row's parent first, same as Edit does, because detail page takes namespace from selected tenant. Ctrl, cmd and shift click don't run that switch, so the tab you open a tenant from stays where it was. First commit fixes what the link depends on. Console recovers a tenant's parent from namespace name (`tenant-a-b` sits under `tenant-a`), but root is the one level that naming skips: tenant `acme` ordered in `tenant-root` owns `tenant-acme`, not `tenant-root-acme`. So `realParentNamespace` returned undefined for every top level tenant, which is most tenants on a normal cluster, and Edit button has been missing there for as long as it exists. Ancestor labels carry the parent that the name does not. Root level tenant named `root-x` resolves to CR `root-x` now, not `x`. ### Notes I first tried to make links carry their tenant in the url (`?tenant=`), so a detail page opened in new tab resolves on its own. That turned into questions about tenant context itself: what to do with a tenant in a link the user can't see, what localStorage may hold, and TenantNamespace LIST ignoring `labelSelector`. Detail route addresses a Tenant CR by relative name and takes namespace from selected tenant, and making that robust is a refactor of tenant context and routing, which should start with study and planning, not with patches on patches. So this PR only closes the UX gap (no way from tenant list into a tenant's page), going deeper is out of scope. Known limit that comes with it: link has no tenant in it, so opening it in a new tab (ctrl/cmd/middle click, open in new tab, copied url) resolves against the tenant last selected in any tab, because selection lives in shared localStorage. For a row directly under that tenant it opens the right page. For a row deeper in the tree the page says `Application <name> not found.` and there is no Delete. If that tenant also has a direct child tenant with the same relative name, new tab opens that other tenant's page with working Edit and Delete, and delete confirm says only `Delete Tenant "<name>"?`. Plain click in the same tab is not affected. Detail page's own tabs and Edit link already behave this way, it's how detail route resolves its namespace today and fixing it properly is the refactor above. A row links, and offers Edit, exactly when namespace holding its Tenant CR is one the apiserver returns to this user. It returns a tenant namespace only if the user is subject of a RoleBinding in it, so sub-tenant admin that cant act in `tenant-root` sees no link there. No row level Delete button. Issue allows either that or a route to detail page, I took the route so confirm and mutation stay in one place. Covered by tests, every source change here fails a test when reverted: - top level tenant, nested one, root level tenant named `root-x`, forest root without ancestor labels - bridged row whose real parent is invisible gets neither link nor Edit - plain click switches to the row's parent and opens detail page - ctrl click leaves selection and location alone ### Screenshots Tenant list under root on a dev cluster. Every row name is a link to tenant page now (`pr4078` is hovered), top level tenants had neither link nor Edit before. Host column redacted. <!-- SCREENSHOT: drag in 01-tenant-list-links.png here --> <img width="1440" height="900" alt="01-tenant-list-links" src="https://github.com/user-attachments/assets/39cd723a-1ac5-4d3b-8c36-b70ddc626961" /> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: All changed files are under `packages/system/dashboard/images/console`, console has no downstream copy (standalone UI repo is archived). ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(dashboard): tenant name in the tenant list links to its detail page now (where Delete is), and top level tenants get their Edit button back. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Root-level tenant names are now displayed without incorrectly removing prefixes, and tenants ordered directly under the root are correctly recognized in the hierarchy. * Tenant rows link to their detail pages when the parent tenant is accessible. Same-tab navigation updates the active tenant, while opening a link in another tab leaves the current selection unchanged. * Edit links and navigation now use consistent tenant routes. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
7f55ff5d85 |
fix(dashboard): link a tenant row to its detail page
The tenant list rendered the name as plain text and put the row's only link on an Edit button, so nothing in the list reached the detail page that every other kind gets, tabs and a Delete action included. The only way there was to open the edit form and cancel out of it. The name cell becomes a link wherever Edit is offered, to the same CR. The detail page reads the namespace from the tenant context, so a plain click aligns the context to the row's parent first, the way Edit already does. React runs that handler for a ctrl-, cmd- or shift-click too, which the browser turns into a new tab or window, so it runs only for the click that navigates its own tab; otherwise opening a tenant beside the current one would move the tab left behind as well. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
e44e41d307 |
fix(dashboard): resolve the Tenant CR of a tenant ordered in the root
A tenant's CR lives in its parent's namespace, and the console recovered the parent from the namespace name: `tenant-a-b` sits under `tenant-a`. The root is the one level that naming skips. A tenant `acme` ordered in `tenant-root` owns `tenant-acme`, not `tenant-root-acme`, so the prefix test never matched a top-level tenant and `realParentNamespace` returned undefined for it. That is most tenants on a normal cluster, and the Edit button, offered only where the CR's namespace resolves, was missing for all of them. The ancestor labels carry the parent the name does not, so the lookup falls back to `tenant-root` when the chain names it. Deriving the CR name keys on the parent being the root rather than on the child's name happening to start with it, which keeps a root-level tenant called `root-x` from resolving to the CR `x`, and the row label agrees with it rather than showing `x`. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
8d96c46994 |
fix(monitoring-agents): isolate merged log payload under log_processed
Fluent Bit's kubernetes filter runs with Merge_Log On but without Merge_Log_Key, so a parsed JSON log line is merged into the record root. An application log line can thereby forge the service fields that later pipeline stages rely on for routing and tenant isolation (kubernetes_namespace_name, kubernetes.labels, tenant, cluster, log_source). Setting Merge_Log_Key log_processed nests the parsed payload under a dedicated key so application content can no longer override top-level service fields. Applies to both the platform (system/monitoring-agents) and guest (apps/kubernetes) Fluent Bit configurations. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
3abdff5900 |
fix(kubernetes-nodes): refuse a partial resources spec on every pool (#4973)
The `resources.cpu`/`resources.memory` pair check in `nodegroup.yaml` ran only when an `instanceType` was set, while the field description (values.yaml, README, schema, the KubernetesNodes OpenAPI schema) promised a render-time error for a partial spec on any pool. Without an instancetype a cpu-only pool renders a VM with no memory, which KubeVirt refuses when the Machine is created, and a memory-only pool silently sizes the guest at KubeVirt's default of one vCPU. - The check now runs on every kubevirt pool. The proxmox branch already required both, so this is the kubevirt half of one rule, and the field descriptions are true as written (no doc change). On a pool with no `instanceType` the message ends in "or neither and set instanceType", so it does not send the user straight into the next refusal. - A kubevirt pool with neither an `instanceType` nor both resources fails at render with a message naming both, the way `vm-instance` does. No such pool could have passed admission (no memory), so this refusal breaks nothing that runs today. This is the decision the issue asked for; if the gate should stay narrower, this commit can be dropped on its own. - **Upgrade impact.** One shape that renders on main stops rendering: `instanceType: ""` with only `resources.memory`. On main it gets `domain.memory.guest` and no `domain.cpu`, so KubeVirt runs the worker with one vCPU; with this PR it fails with the partial-spec message, and the pool's HelmRelease sits in a failed upgrade until the values are edited. To keep the current size such a pool needs `resources.cpu: "1"`. Adding that line renames the machine template (for the test values, `kubernetes-myk8s-md0-28d812` becomes `kubernetes-myk8s-md0-1ce103`, the same name on main and on this branch), so the pool's workers are replaced once. cpu-only with `instanceType: ""` is not in this class: it renders a VM with no memory, which KubeVirt refuses, so it never ran. - `cores:` printed an unquoted (float64) core count of one million or more in exponent form (`1e+06`). It is printed as an integer now, with the same bytes for every smaller whole number. Tests: `tests/resources_partial_test.yaml` covers both halves with and without an instancetype, the no-sizing case, both fields set, and the float64 formatting. The existing unquoted-zero case in `resources_cpu_test.yaml` now reads the field as unset on an instancetype-sized pool (with a `kubernetesProvider` stub for the lookup), and zero beside memory is pinned as a partial spec. Byte stability, checked by snapshotting the `KubevirtMachineTemplate` on main for nine spellings of `resources.cpu` and replaying the snapshots on this branch: `"2"`, `2`, `2.0`, `"2.0"`, `"+2"`, `"1e1"`, `"010"`, `999999` render identical bytes and template names. Only unquoted `4294967295` changes, from `cores: 4.294967295e+09` to `cores: 4294967295`. Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38 suites, 233 tests, 12 snapshots), and on a trial merge with current main (39 suites, 237 tests). Closes #4546. The leading-zero inconsistency from the issue's comment is addressed separately, together with #4554. ### Release note ```release-note fix(kubernetes-nodes): a partial `resources` spec (only `cpu` or only `memory`) is now refused at render time on every kubevirt worker pool, not only on a pool with an `instanceType`, and a pool with neither an `instanceType` nor both resources is refused as well. Action required: a pool with `instanceType: ""` and only `resources.memory` rendered before (KubeVirt ran it with one vCPU) and now fails to upgrade; add `resources.cpu: "1"` to keep its current size. Adding that line renames the pool's machine template, so its workers are replaced once. Whole-number core counts of one million or more are now printed as integers instead of in exponent form. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * KubeVirt worker pools now require either an instance type or both CPU and memory values. Configurations with only one resource or no sizing information are rejected. * Whole-number CPU values render without exponent notation, including values written with a decimal fraction. * An unquoted zero CPU value is treated as unset when an instance type is provided; without an instance type, pairing zero CPU with memory fails validation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
60d1f53e82 |
fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets (#4974)
On the kubevirt substrate `resources.cpu` is written into `domain.cpu.cores` as spelled and decoded as YAML, so `"010"` gives the VM eight cores, while the cluster-autoscaler capacity annotation parsed the same string as decimal ten (`capacity.cluster-autoscaler.kubernetes.io/cpu: "9000m"` on a node that offers 7000m). This is the inconsistency reported in #4554 and in the comment on #4546. - The YAML-decoded value the range check already computes is kept, and on kubevirt the capacity annotation and the reserved-CPU check start from it. - The kubelet reservations are not touched. The worker machineconfig (`_talosconfigtemplate.tpl`) sizes them from the raw string read as a decimal, and the TalosConfigTemplate name is a hash of that config, so reading the decoded value there would roll every leading-zero pool once. Instead the annotation subtracts the reservations the machineconfig really carries from the cores the VM really has: for `"010"` that is 8000m − 2 × 500m = `"7000m"`, the node's real allocatable, while `"8"` keeps `"7200m"` (2 × 400m). Tests pin both sides: the annotation and the bootstrap `configRef` name in `resources_cpu_test.yaml`, and `systemReserved.cpu` / `kubeReserved.cpu` in the machineconfig in `talos_worker_config_test.yaml`. - No pool rolls. Checked by rendering `nodegroup.yaml` on main and on this branch for nine spellings (`"2"`, `2`, `2.0`, `"2.0"`, `"+2"`, `"1e1"`, `"010"`, `999999`, `4294967295`): the only difference is the capacity annotation of the `"010"` case, so the `KubevirtMachineTemplate` name and the TalosConfigTemplate `configRef` stay as they are. The machineconfig template itself is not changed. - Proxmox parses the quantity as decimal for the machine template and for the reservations alike, so it never had the inconsistency and is unchanged (pinned by a test). - The `resources` description says the value is decoded as YAML, what that means for a leading zero, and that the reservations still read it as a decimal. README, schema, the RD and the Go type are regenerated from it. This closes #4554 without the migration and the render-time refusal it proposes. A refusal at render breaks a stored pool on its next upgrade, a pattern in the OpenAPI schema breaks every update of an existing CR, and a platform-wide migration over every `KubernetesNodes` and parent `nodeGroups` entry is a lot of blast radius for a spelling nobody has reported in a live cluster. With this change the autoscaler plans for the node the VM really is. What is left of a leading zero is the octal reading itself and reservations sized from the decimal value, and both are documented. The unquoted zero of the issue is already harmless: it is falsy, counts as unset, and renders the same bytes as an absent field. If a migration is still wanted, please reopen. Verified with `helm unittest .` in `packages/apps/kubernetes-nodes` (38 suites, 233 tests, 12 snapshots; pointing the annotation back at the decimal string turns the `"010"` case red with `9000m`), `make generate` (no diff), and a trial merge with #4973 (clean, 243 tests green). Closes #4554 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * KubeVirt VM sizing and autoscaler capacity now reflect YAML-decoded CPU values. For example, `"010"` represents eight VM cores, while kubelet reservations interpret it as decimal ten. `"8"` and `"10"` yield `7200m` and `9000m` of autoscaler capacity, respectively. Proxmox continues to interpret `"010"` as ten cores, yielding `9000m` of capacity. * **Documentation** * Clarified explicit worker CPU and memory sizing behavior, including how leading-zero CPU values affect VM sizing, reservations, and capacity calculations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
44848b9c9d |
fix(kubernetes): run Proxmox CCM v0.16.1 and pin the Proxmox images by digest (#4917)
## What this PR does
`v1.7.0-rc.1` and `release-1.7` ship the Proxmox
cloud-controller-manager as
`ghcr.io/sergelogvinov/proxmox-cloud-controller-manager:v0.15.0`, a bare
tag. That tag was re-pushed upstream on 2026-09-27: it used to name the
2026-08-06 build (`sha256:fbc553921145…`) and now names
`sha256:d81702998ae1…`, a binary whose gitVersion is `0.15.0` without
the leading `v`. component-base v0.37 parses that string in an `init()`
and panics:
```
panic: version string "0.15.0" doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"
```
The CCM Deployment crash-loops, no new Node loses
`node.cloudprovider.kubernetes.io/uninitialized`, and workloads on new
workers stay Pending. On a live Proxmox cluster this went unnoticed for
five days: the existing Nodes were already initialized, and the failure
only showed when the next new worker came up. Every fresh install pulls
the broken build, so as it stands v1.7.0 would be the first stable
release with the Proxmox substrate, shipping a CCM that cannot start.
Two commits, both cherry-pick onto `release-1.7` unchanged:
1. **`fix(kubernetes)`**: run v0.16.1, pinned as tag plus digest in
`images/proxmox-cloud-controller-manager.tag`. That is the same shape
the chart already uses for the KubeVirt cloud provider. v0.16 builds
with goreleaser and stamps the version with its `v`. The config the init
container writes still parses: the cluster entry moved to
go-proxmox-pool's `ClusterConfig` with the same `url`, `insecure`,
`token_id`, `token_secret` and `region` keys, and `features.provider:
capmox` still writes `proxmox://<SMBIOS UUID>`, the providerID capmox
puts on the Machine.
2. **`fix(kubernetes, proxmox-csi)`**: pin the CSI driver from the same
registry at its current v0.20.0, with no version change. The
management-side controller goes through
`images/proxmox-csi-controller.tag`. The tenant-side node plugin goes
through `repository` + `tag` in
`packages/system/proxmox-csi/values.yaml`, which also puts it in
`hack/lib/image-refs.sh` and so in front of the release multi-arch gate.
Pinning takes the version out of the vendored chart's hands, so
`hack/proxmox-csi-driver-pin_test.bats` now fails unless the chart's
`appVersion` and both pins name the same release.
All three digests are OCI indexes for `linux/amd64` and `linux/arm64`.
The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI
driver: registry.k8s.io publishes through image promotion, which does
not move a tag.
### Testing
- `helm unittest`: `packages/apps/kubernetes` 31 suites / 346 tests,
`packages/apps/kubernetes-nodes` 37 / 223, all passing.
- bats: the new test, plus `image-pin-consistency`,
`promote-rewrite-tags_test`, `overlay-main-images_test`,
`nightly-mirror_test`, `promote-retag_test`, `stitch-multiarch_test`,
`bats-strict-setup`, `bats-runner-coverage`: 107/107. The new test was
checked to fail on a node tag at another version.
- `git cherry-pick -x` of both commits onto `release-1.7` (
|
||
|
Marian Koreniuk
|
f7913329b2 |
fix(kubernetes-nodes): plan autoscaler capacity on the cores the VM actually gets
On the kubevirt substrate resources.cpu is written into domain.cpu.cores as spelled and decoded as YAML, so "010" gives the VM eight cores, while the cluster-autoscaler capacity annotation parsed the same string as decimal ten. The autoscaler then planned 9000m on a node that offers 7000m. Keep the YAML-decoded value the range check already computes and start the capacity annotation and the reserved-CPU check from it on kubevirt. The kubelet reservations stay as they are: the worker machineconfig sizes them from the decimal string, and the TalosConfigTemplate name is a hash of that config, so changing them would roll every leading-zero pool once. The annotation therefore subtracts the reservations the machineconfig really carries (2 x 500m for "010") from the cores the VM really has, which is the node's real allocatable. Neither template name changes for any spelling, so no pool rolls. Proxmox parses the quantity as decimal on both sides and is unchanged. The field description says the value is decoded as YAML and that the reservations still read it as a decimal. This closes #4554 without the migration and the render-time refusal it proposes: a refusal breaks stored pools on their next upgrade, a schema pattern breaks updates of existing CRs, and a platform-wide migration of every KubernetesNodes and nodeGroups entry is a lot of blast radius for a spelling nobody has reported in a live cluster. Closes #4554 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Marian Koreniuk
|
780b068df0 |
fix(kubernetes, proxmox-csi): pin the Proxmox CSI driver images by digest
The CSI controller in the management cluster was referenced by the bare tag ghcr.io/sergelogvinov/proxmox-csi-controller:v0.20.0, and the node plugin in the tenant by the vendored chart's default, its appVersion as a bare tag. That registry is the one whose proxmox-cloud-controller- manager:v0.15.0 was re-pushed to a build that panics, so a tag there is not a statement of what runs. Both now carry the v0.20.0 digest: the controller through images/proxmox-csi-controller.tag, like the cloud-controller-manager, and the node plugin through repository and tag in the proxmox-csi values, which also makes it visible to hack/lib/image-refs.sh and therefore to the release multi-arch gate. Both digests are indexes for linux/amd64 and linux/arm64. Pinning takes the version out of the vendored chart's hands: a chart bump would bring a new appVersion while both pins kept naming the old release, and the controller and the node plugin would drift apart from the templates and from each other without anything noticing. hack/proxmox-csi-driver-pin_test.bats fails unless the chart's appVersion and both pins name the same release. The sig-storage sidecars stay on tags, as they do for the KubeVirt CSI driver: registry.k8s.io publishes them through image promotion, which does not overwrite a tag. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
Marian Koreniuk
|
05a347e6f6 |
fix(kubernetes): run proxmox-cloud-controller-manager v0.16.1, pinned by digest
The Proxmox CCM was referenced by its bare v0.15.0 tag, and that tag no
longer names the image it named when the template was written. It was
re-pushed on 2026-09-27 (index sha256:d81702998ae1, replacing the
2026-08-06 build sha256:fbc553921145) to a binary whose gitVersion is
"0.15.0" without the leading v. component-base v0.37 parses that string
in the init() of its legacy metrics registry and panics:
panic: version string "0.15.0" doesn't match expected regular
expression: "^v(\d+\.\d+\.\d+)"
so the Deployment crash-loops, no Node ever loses the
node.cloudprovider.kubernetes.io/uninitialized taint, and workloads on a
new worker stay Pending. Any node that pulls the tag fresh gets it, which
is every fresh install.
v0.16 builds with goreleaser and stamps gitVersion as v{{.Version}}. The
configuration this template writes is read unchanged: the cluster entry
moved to go-proxmox-pool's ClusterConfig with the same url, insecure,
token_id, token_secret and region keys, and features.provider: capmox
still writes proxmox://<SMBIOS UUID>, the providerID capmox puts on the
Machine.
The reference now lives in images/proxmox-cloud-controller-manager.tag
as tag plus digest, the shape the chart already uses for the KubeVirt
cloud provider, so a moving tag can no longer change what runs.
Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
|
||
|
Marian Koreniuk
|
0e77e2b10f |
fix(kubernetes-nodes): name instanceType in the partial-spec refusal
On a pool with no instanceType, "set both resources.cpu and resources.memory, or neither" sent the user straight into the next refusal, since neither sizing source is refused as well. The message now ends in "or neither and set instanceType" on such a pool; a pool sized by an instancetype keeps the old wording. The tests anchor both spellings, so the suffix cannot leak onto, or drop off, either kind of pool. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
Aleksei Sviridkin
|
ae9e7c4c64 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on `strategy.name: RetryOnFailure` for both actions, and the e2e HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets the same strategy for the same reason and had no pin. This adds `packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a mirror of `csi_install_retry_test.yaml` rendered on the proxmox substrate, and updates the guard comment that pointed at the gap. No template change. Verified with `helm unittest .` in `packages/apps/kubernetes` (32 suites, 349 tests). Closes #4731 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded automated coverage for Proxmox CSI installation and upgrade retry behavior, including the retry interval and remediation retry limits. The checks verify that retries continue according to the configured settings during installation and upgrades. This update does not change user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0b3f5dd7e1 |
fix(keycloak)!: make the KMS proxy keep its keys across restarts (#4818)
## What this PR does Two fixes for the optional KMS proxy of the platform Keycloak (`encryption.*`). The default static backend never started. The chart stored 32 alphanumeric characters as the KEK, but the proxy wants base64 of 32 bytes and exits with `config: KKP_KEK must decode to 32 bytes (got 24)`. The KEK is now generated with `randBytes`, and a stored key of the old form is replaced. The proxy never ran with such a key, so nothing was encrypted under it. Any other stored key is kept if the proxy accepts it and fails the render otherwise, so the chart never overwrites key material. The KEK Secret now carries `helm.sh/resource-policy: keep`: the DEK set is wrapped with it, so it has to survive turning encryption off or reinstalling the package. Without `deksetSecretName` the proxy mints a new DEK set on every start (`buildCipher` in `cmd/proxy/main.go` of keycloak-kms-proxy) and keeps it only in memory. After the first restart of the proxy pod it cannot decrypt anything it wrote before. The values comment said this is safe on a fresh DB, it is not. The chart now refuses to render the proxy without a DEK set unless `fields: disabled`. Passthrough still works without one, so the static backend can get its KEK Secret created before the DEK set is minted with it. Breaking: a release with encryption on and no `deksetSecretName` fails to render on upgrade until a DEK set is configured. Such a setup loses its data on the next proxy restart anyway, so failing loudly is the point. A stored static KEK the proxy cannot decode fails the render too, instead of being replaced. Checked with `helm unittest` (100 tests, the new ones fail on the old template) and by starting keycloak-kms-proxy v0.2.3 with the KEK the chart renders now. With the KEK from the old template it fails as quoted above. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/pull/742 documents the setup with a DEK set - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The other repositories do not touch the keycloak chart values. ### Release note ```release-note fix(keycloak)!: The KMS proxy of the platform Keycloak now requires `encryption.deksetSecretName` unless `encryption.fields` is `disabled`. Without it the proxy generated a new data key on every start and could not read data encrypted before a restart. The static backend now generates a KEK the proxy accepts; before, the proxy could not start with it. The KEK Secret is no longer deleted when encryption is turned off or the package is removed. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Configuration** * Enabling field encryption requires a persistent DEK set. Chart rendering reports an error if it is missing; setting fields to `disabled` does not require one. * For the static backend, a valid stored KEK is reused, and a missing or legacy-format KEK is replaced. An unusable stored KEK causes rendering to fail. * The KEK Secret is retained when encryption is disabled or the package is removed. * **Documentation** * Clarified DEK set requirements and sharing across starts and replicas, and updated guidance on configuring multiple proxy replicas. * Warns against deleting the retained KEK while encrypted values remain. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
b36628661f |
fix(ingress-nginx): drop the admission-webhook policy a rebase brought back (#5088)
## What this PR does Rebase behind [#3931](https://github.com/cozystack/cozystack/pull/3931) brought back namespaced `CiliumNetworkPolicy` for admission webhook, which [#3997](https://github.com/cozystack/cozystack/pull/3997) deleted in favour of clusterwide policy. It renders only on `admissionWebhookPolicy.enabled` (true by default) and never checks that Cilium API exists, so on cluster without Cilium every tenant with `ingress: true` fails to install with `no matches for kind "CiliumNetworkPolicy"`. On cluster with Cilium it repeats deny that clusterwide policy already has, and also renders for child tenants where webhook is disabled. This PR removes the template with its test and adds suite that renders whole package without Cilium API and fails if any Cilium resource comes out. Existing capability suite selects only clusterwide template, so it could not catch this file coming back. Clusterwide policy and `admissionWebhookPolicy` value are unchanged. Fixes #4706 On cluster without Cilium nothing in this chart restricts webhook port. Removed policy could not do it either because it needs Cilium CRD to install at all, and clusterwide policy already renders only where Cilium API is served, so no protection is lost, such cluster now just installs ingress instead of failing. Only `tenant-root` runs webhook, child tenants set `admissionWebhooks.enabled: false`. Restricting webhook port on non-Cilium clusters is separate design question, not part of this PR. Checked `helm template` with and without `cilium.io/v2/CiliumClusterwideNetworkPolicy` in api versions: without it no Cilium object renders, with it only clusterwide policy, same as before. New suite fails on tree before this change. ### Screenshots Not a UI change. ### Downstream repositories Only template and tests under `packages/system/ingress-nginx` changed, `admissionWebhookPolicy` value kept, nothing from trigger map in `docs/agents/contributing.md` matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(ingress-nginx): remove the namespaced admission-webhook CiliumNetworkPolicy that was re-added by mistake. It was rendered without checking that the Cilium API exists, so on clusters without Cilium every tenant ingress failed to install with `no matches for kind "CiliumNetworkPolicy"`. The clusterwide admission-webhook policy is unchanged and still renders only where the Cilium API is served. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Configuration** * The ingress-nginx chart no longer generates an admission webhook network policy. Deployments that relied on this policy to restrict access to the webhook will need to configure equivalent traffic controls separately. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
1ac83aebd9 |
fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates (#5086)
## What this PR does After platform upgrade the tenant `gateway` HelmRelease fails with `timeout waiting for: [TenantGateway/<namespace>/cozystack status: 'InProgress']`, retries every 30s without end (`RetryOnFailure`) while holding a helm-controller worker, and parent tenant release that waits on it fails too. helm-controller waits on every object of the release with kstatus on install and on every upgrade, and kstatus reads TenantGateway `Ready=False/ListenersNotReady` as InProgress. TenantGateway stays there while any listener certificate is pending, which depends on ACME or DNS and on routes owned by other releases, so waiting longer doesn't help. It passed before only because the wait runs inside a Helm action and the install happened before those listeners showed up. I added `healthCheckExprs` for `TenantGateway` with `current: "true"` and pinned `waitStrategy: poller`, because expressions are evaluated only under poller. Wait stays on. Flux returns InProgress while `status.observedGeneration` lags `metadata.generation` before it reads any expression, so release still waits until controller observed the new TenantGateway spec, but not for listeners or certificates. tenant-root renders same template and gets the change too. Gateway release that is already stuck picks this up on its next attempt, after the current one times out. New helm-unittest suite (`packages/apps/tenant/tests/gateway_wait_test.yaml`) pins the strategy and the whole expression list, checks there is no `disableWait` on install or upgrade, and that no gateway release is rendered when `gateway` is unset. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(tenant): the tenant gateway release no longer times out on upgrade while TenantGateway listener certificates are pending, it waits only until the controller has observed the current TenantGateway spec. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Gateway releases now determine readiness from the gateway resource’s current status, rather than waiting for its listeners and certificates to become ready. Releases still wait for the latest resource update to be observed. * **Tests** * Added checks for gateway release readiness behavior and for tenants configured without a gateway. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
7dc5c0b5e6 |
fix(marketplace): do not install a connected repository's components on connect (#5071)
## What this PR does Connecting a marketplace repository materialized its `PackageSource`s and then auto-created a tap-managed `Package` per source. The Package reconciler installs a `HelmRelease` for every install-marked component of the default variant, and Flux runs those under its `flux` service account, which is bound to `cluster-admin` with no impersonation (`--default-service-account` is not set, and neither `internal/operator` nor `pkg/registry/apps` set `serviceAccountName` on the HelmRelease). A publisher's install chart therefore ran as `cluster-admin` on connect, with no operator step, contradicting the documented behaviour that tapping installs nothing. The `privileged` flag is a prompt to the admin and does not protect against a publisher acting in bad faith. This removes the on-connect auto-registration (`ensureRegistration`). Connecting now only materializes `PackageSource`s. Installing an app is an explicit `cozypkg add`, which becomes the sole creator of a `Package`; every other `Package` reference in the operator is a read or a watch. Removing the call also fixes the second effect of the same path: a `Package` the admin removed with `cozypkg del` is no longer re-created on the next source revision. A second commit reconciles the texts this behaviour change made wrong: they promised that tapping registers a repository's apps in the catalog and runs its charts. Reworded the `cozypkg tap`/`untap` help, the tap success output (it now prints a `cozypkg add <name>` line per materialized PackageSource), the `cozypkg init` scaffold README, the packages-index trust model, and the dashboard Repositories banner and per-repository empty state. Accepted consequence: a connected repository's apps appear in the catalog only after an admin runs `cozypkg add`, since the dashboard catalog is computed from `ApplicationDefinition`s. Install from a connected repository is CLI-only for now; the console connects and disconnects repositories but does not create a `Package`. Populating the catalog from `PackageSource` metadata without an install is the long-term shape in #4359 and is out of scope here. The privileged-confirmation gaps #4358 and #4359 stay open. Upgrade note: a `Package` that Cozystack 1.7.0 auto-created on connect stays after the upgrade, and the privileged-confirmation gate still applies to it, so this is not a regression for existing installs. For a newly connected repository no tap-labelled `Package` is created, so the gate does not come into play on that path. An operator who does not want a 1.7.0 auto-created package can remove it with `cozypkg del`. Addresses #4978. ### Screenshots Dashboard text-only change (Repositories banner and per-repository empty state). No layout change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. The diff is operator logic plus CLI/dashboard/doc text; it matches no concrete trigger (no package, API, schema, tooling, annotation, metric or commit-convention change). The dashboard console is vendored in this repo (`packages/system/dashboard`), so its text change travels in this PR. Leaving every box empty rather than ticking "No downstream repository is affected". ### Release note ```release-note fix(marketplace): connecting a marketplace repository no longer installs its components. A connected repository's apps are materialized as PackageSources but not installed until an explicit `cozypkg add`; install from a connected repository is CLI-only for now. Previously, connecting auto-created a Package whose install-marked components were deployed by Flux under its cluster-admin service account. A Package that Cozystack 1.7.0 auto-created on connect stays after upgrade and can be removed with `cozypkg del`. ``` |
||
|
myasnikovdaniil
|
7215b02d38 |
feat(keycloak): load additional CA files from Secret or ConfigMap keys (#5039)
## What this PR does Closes #3823. With the stock Cozystack chart, Keycloak cannot connect to an LDAPS server, external identity provider or SMTP server signed by a private CA because the chart cannot deliver a CA file to its truststore. Add `truststores`, a list of existing Secret/ConfigMap references in the release namespace, with one selected key per entry (`ca.crt` by default). ```yaml truststores: - configMapName: freeipa-ca key: ca.crt - secretName: partner-ca key: certificates.pem ``` Each key is mounted read-only in a separate `cozystack-<index>` subdirectory of `/opt/keycloak/conf/truststores`. Keycloak discovers it automatically; the chart adds no environment override, Kubernetes API permission or arbitrary volume path. Existing image certificates, Java default roots, themes and `extraEnv` keep their behavior. References are mandatory: a missing object or selected key prevents the pod from starting. Helm validates the reference shape and safe key paths, but cannot inspect external certificate contents. The trust decision belongs to platform administrators and applies to the entire Keycloak server, across all realms. Write access to a referenced object therefore permits changing that server's trust. If HTTPS client authentication is set to `request` or `required` without a separate `https-trust-store-file`, these CAs also verify incoming client certificates; configure that file explicitly when incoming mTLS requires a different trust boundary. Use public certificates only (PEM bundles or unencrypted PKCS12 without private keys). This does not disable TLS hostname verification. Updating an object's contents does not restart Keycloak or reload its truststore: restart both pods after rotation, or use a versioned object name to trigger a rollout. Validation: - All 113 Keycloak Helm tests pass, including 21 new cases for reference wiring, read-only mounts, selected-key isolation, themes, PKCS12 filenames, defaults and invalid input. - Boundary tests reject source names and keys of 254 characters and the single-dot key, and accept a source name and key of exactly 253 characters. Removing any of the three guards makes its regression test fail; changing either length comparison to reject 253 also fails the boundary test. - Empty-default render matches a clean upstream/main worktree after normalizing the existing randomly generated bootstrap password. Four mutations (wrong mount path, writable mount, wrong selected key and removed validation) each turn the new suite red. - Enabled and default manifests render successfully; enabled YAML parses with yq. `git diff --check` passes. - Runtime check with official Keycloak 26.6.3 in isolated Docker: synthetic LDAPS succeeds with a CA loaded automatically through an emulated Kubernetes AtomicWriter symlink layout (HTTP 204 from the LDAP connection-test Admin API); no custom CA fails with PKIX, and a trusted CA with a mismatched SAN fails hostname verification (both HTTP 400). KC_TRUSTSTORE_PATHS was unset. This checks TLS establishment, not a complete federation bind/failover or deployment to a Kubernetes cluster. - `make generate` has no target in this system chart; no generated schema/API files are involved. `make show` requires the unavailable cozyhr CLI and a live release, so rendering uses `helm template`. `helm lint` reports the same pre-existing undeclared cozy-lib dependency on both this branch and clean main. ### Screenshots Not applicable; no UI changes. ### Downstream repositories The website guide currently says CA mounts are unavailable, so its `next` documentation needs updating once this ships. Other downstream trigger-map entries do not apply: this only adds an opt-in value to the system Keycloak chart, with no app schema, platform configuration, CRD, tooling, naming or node-contract changes. - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/765 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(keycloak): load additional CA files from existing Secret or ConfigMap keys through the new truststores value, enabling TLS to private LDAPS, identity-provider and SMTP endpoints. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit - **New Features** - Added support for configuring Keycloak truststores from existing Secrets or ConfigMaps. - Supports PEM bundles and unencrypted PKCS12 certificates, with optional key selection and read-only mounts. - Truststore certificates apply server-wide, including incoming client-certificate validation when no separate HTTPS truststore is configured. - Invalid truststore configurations produce clear errors. Missing objects or keys prevent startup; certificate rotations require pod restarts unless the referenced object name changes. - **Tests** - Added coverage for default behavior, multiple truststore sources, theme mounts, explicit paths, and invalid configurations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
2f39e304f0 |
fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts wrong: they promised that tapping registers a repository's apps in the catalog and runs its charts. Tapping now installs nothing, so the texts told operators to wait for apps that never appear. Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a `cozypkg add <name>` line per materialized PackageSource), the `cozypkg init` scaffold README, the packages-index trust model, the dashboard Repositories banner and per-repository empty state, and the stale re-registration code comments. No behaviour changes beyond the added `cozypkg add` hint in the tap output. Assisted-by: LLM Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Aleksei Sviridkin
|
749d401899 |
fix(kubernetes, opensearch, foundationdb)!: restrict managed application images (#5085)
## What this PR does
Image overrides let tenants run arbitrary containers in management
cluster, including helpers with access to Talos CA secrets and
Kubernetes API credentials. This PR removes those overrides and uses
images shipped by the platform.
Replaces [the original
PR](https://github.com/cozystack/cozystack/pull/4287) for 1.7. Andrei
Kvapil's commits keep his authorship.
- Kubernetes and KubernetesNodes helpers use the bundled image
references through `cozy-lib.image`. OpenSearch uses its version map.
Talos installer settings for tenant worker VMs stay available.
- FoundationDB uses `version: v7.3` or `v7.4`, mapped to `7.3.63` and
`7.4.1`. Migration 60 carries the old `cluster.version` over to its
release line, records releases needing attention and preserves the
legacy value while the old chart can still render it.
- The FoundationDB chart refuses a render when `cluster.version` and
`version` name different release lines. This also covers variants
without platform migrations.
- OpenSearch and FoundationDB get version-map update scripts. They
retain the current default while it remains supported.
Application types and schemas are still generated from the charts.
### Upgrade notes
Stored image overrides are accepted without new admission warnings and
ignored by the charts. On the first reconcile, OpenSearch nodes move to
the image for their selected version, and Kubernetes components whose
images changed restart. Check custom OpenSearch images before upgrading:
if one runs a newer version than the chart's map, switching back can
leave nodes unable to read their data. Air-gapped installations need
`docker.io` and `ghcr.io` mirrored on management-cluster nodes.
FoundationDB preserves the release line, not an arbitrary patch: 7.3.x
moves to 7.3.63 and 7.4.x to 7.4.1. Check these versions before
upgrading. A cluster on 7.1 needs an upgrade to 7.3 first. On variants
without migrations, set `version` to the line the cluster runs. To
change release lines later, remove the leftover `cluster.version` as
directed by the chart error.
Migration 60 records affected releases in the
`cozystack.io/migration-60-needs-attention` annotation on the
`cozystack-version` ConfigMap. Dry-run reports patch changes without
patching, annotating or advancing the migration stamp. Migration 56 no
longer carries the removed kubectl override into adopted node pools.
An explicit OpenSearch `version: null` now fails rendering instead of
falling back to `v2`.
### Validation
- `make unit-tests` passed, including migrations 56 and 60.
- Four-chart regression run: 750 tests passed.
- API Create/Update, admission, defaulting and existing warning
regression tests passed. `cmd/`, `internal/` and `pkg/` match main.
- Shell test lanes passed with `/bin/sh` as dash throughout the process
tree.
- `make generate manifests` passed with no generated drift, using
cozyvalues-gen v1.7.0.
E2E passed on
|
||
|
myasnikovdaniil
|
55aad6f914 |
fix(tcp-balancer): cap HAProxy fds and threads by the pod's own limits (#5005)
## What this PR does
On a v1.6.5 cluster a TCPBalancer created with the chart defaults
(`t1.nano`, limits 250m CPU and 128Mi memory, `haproxy:3.4.4`)
crash-looped. Both pods were OOMKilled with a memory limit of 134217728,
7 restarts each in about 15 minutes, and HAProxy logged `[NOTICE] (8) :
Automatically setting global.maxconn to 524126.` On another cluster the
same chart picked maxconn 524251 and the pods kept running.
`haproxy.cfg` has no `global` section, so HAProxy sizes itself from the
node. maxconn comes from the nofile limit capped at the built-in
`fd-hard-limit` of 1048576 (`DEFAULT_MAXFD`), and the thread count is
the number of CPUs the process may run on, whatever its CPU limit is.
Both are paid for at startup, before any traffic, and the thread part
grows with the node.
This PR adds a `global` section sized from the replica's own limits,
read through the same `cozy-lib.resources.defaultingSanitize` helper the
Deployment uses, so `resources` and `resourcesPreset` resolve the same
way. For `t1.nano` it renders
```
global
fd-hard-limit 4096
thread-hard-limit 1
```
`fd-hard-limit` is 32 per MiB of the memory limit and HAProxy derives
maxconn from it, 2035 for `t1.nano`, which is about 64 KiB per
connection. For comparison `haproxy -m 128` picks maxconn 3600 on 3.4.4,
about 35 KiB per connection, and the difference is left for the process
itself and for the kernel socket buffers the cgroup also charges. An
explicit `maxconn` would read simpler, but `strict-limits` is on by
default and HAProxy refuses to start when it cannot raise nofile to
about twice that value. With 1024 soft and 4096 hard nofile `maxconn
4096` exits with `Cannot raise FD limit to 8251`, while `fd-hard-limit
8192` starts with maxconn 2018. `thread-hard-limit` is the CPU limit
rounded up, at least 1. It caps the detected thread count and never
forces more threads than the node has.
Reproduced with Docker Desktop on arm64 (12 vCPU, kernel 6.12) and the
same image, running the rendered config the way the Deployment mounts
it. A larger node is stood in for by adding `nbthread 96` or `nbthread
128` to the `global` section. Memory is `memory.peak` of the container
cgroup.
```sh
cd packages/apps/tcp-balancer && mkdir -p /tmp/cfg
helm template tb . --show-only templates/configmap.yaml | yq '.data["haproxy.cfg"]' > /tmp/cfg/haproxy.cfg
docker run --rm --memory 128m --memory-swap 128m --ulimit nofile=1048576:1048576 -v /tmp/cfg:/usr/local/etc/haproxy:ro haproxy:3.4.4
```
| run | main | this PR |
| --- | --- | --- |
| idle, 12 threads | maxconn 524258, peak 89.6 MiB | maxconn 2035, 1
thread, peak 18.7 MiB |
| idle, `nbthread 96` | maxconn 524127, worker exited with code 137,
OOMKilled | 1 thread, peak 20.8 MiB |
| idle, `nbthread 128` | worker exited with code 137, OOMKilled | 1
thread, peak 22.9 MiB |
| 2035 clients echoing 16 KiB through a backend for 25 s, 12 threads |
worker exited with code 137 about a second after the clients started |
running, peak 83.7 MiB |
One case this PR does not cover is clients and servers that both stop
reading. They fill kernel socket buffers that the cgroup charges, and
with this PR the 128Mi container was still OOMKilled after 1225 such
connections, which maxconn does not bound.
`helm unittest .` passes 13 of 13 with helm-unittest 1.1.2 as CI pins
it. Against the template on main the four new cases fail and the nine
existing ones pass, and each of six template mutations fails at least
one case. `haproxy -c` accepts the render for `t1.nano`, `t1.4xlarge`,
`m1.4xlarge`, a zero CPU limit, the whitelist, `kubernetesAndTalos` and
`tcp-with-proxy` inputs. A full `helm template` differs from main only
in the `global` section and the `checksum/config` annotation, so the
pods roll once on upgrade. I have not run the change on a live cluster.
The template is the same on release-1.6, and that backport is a separate
PR.
### Screenshots
Not a UI change.
### Downstream repositories
One chart template and its unittest suite in
`packages/apps/tcp-balancer`. No values key, schema, default, README or
package change, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(tcp-balancer): HAProxy now takes its fd and thread caps from the replica's memory and CPU limits instead of the node, so a t1.nano balancer is no longer OOMKilled on nodes with many CPUs. maxconn follows the memory limit at about 16 connections per MiB, 2035 on t1.nano, and the pods roll once on upgrade.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **New Features**
- TCP Balancer now automatically configures HAProxy file-descriptor and
thread limits based on the selected or explicitly defined resource
settings.
- CPU-based thread limits are rounded up and always maintain a minimum
of one thread.
- Memory-based file-descriptor limits scale automatically with the
configured memory capacity.
- **Tests**
- Added coverage for default, preset-based, explicit resource, rounding,
and minimum-limit scenarios.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
432d02e20c |
fix(etcd): stop counting canceled watch streams as failed gRPC requests (#4985)
## What this PR does
The tenant etcd chart has its own copy of
`etcdHighNumberOfFailedGRPCRequests`, and both of its rules count every
`grpc_code` other than `OK` as a failure. `grpc_server_handled_total`
counts a Watch stream once, when it ends, and the streams of a
kube-apiserver that goes away end with `Canceled`. So whenever an API
server on a tenant etcd stops, the Watch method of every member reads as
100 percent failed and the critical rule fires on all of them.
Both rules now count only
`Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded`,
the list the etcd mixin has used since etcd-io/etcd#13127 and the one
the platform copy in
`packages/system/monitoring-agents/alerts/etcd.yaml` already uses. The
grouping `BY (job, instance, grpc_service, grpc_method)`, the
thresholds, `for` and the annotations stay as they are, so the alerts
keep their labels.
Seen on a v1.6.5 cluster while a Kubernetes cluster on a tenant etcd was
created and deleted a few minutes later. vmalert had the critical alert
firing for `grpc_method="Watch"` on all three members with the value
100. At that point
`rate(grpc_server_handled_total{grpc_code!="OK"}[10m])` was 0.63 for
Watch with `Canceled`, and the only other non-OK codes were
`FailedPrecondition` on `RoleGet` and `UserGet` at 0.0017 each, which is
one call each. The same query an hour earlier returned nothing. The
platform copy runs in the same vmalert over the same series, and none of
the firing alerts came from it. Each alert also showed up seven times
there, once per tenant copy of these rules, because the root tenant's
vmalert loads the rules of every tenant namespace. This PR does not
change that.
`hack/etcd-grpc-alerts.bats` renders the chart's PrometheusRule with
`helm template` and replays it through `vmalert-tool` v1.136.0 pinned by
digest, the same image #4963 uses.
`hack/testdata/etcd-grpc-alerts/unittest.yaml` has three cases with
seven checks. Watch streams that end only with `Canceled` must not fire
at 6, 12 and 30 minutes. `Range` calls failing with `Unavailable` at 25
percent must fire the critical rule at 7 minutes and both rules at 12,
with the labels and the summary pinned. Both together must fire only for
`Range`. Against the rules on main the suite fails 4 of the 7 checks,
all of them on Watch alerts with the value 100. Dropping `Unavailable`
from the list, or grouping by `job` and `instance` only, fails 3 checks
each.
```console
$ bats hack/etcd-grpc-alerts.bats
1..1
ok 1 tenant etcd failed gRPC request alerts pass their vmalert-tool cases
```
The same run on the rules from main prints `not ok 1` and the four
failing checks. With Bats 1.14.0 at the commit CI pins, the suite passes
together with `bats-strict-setup`, `bats-no-negated-assert`,
`bats-no-exit-trap`, `bats-runner-coverage`, `bats-runner-contract`,
`alert-severity-contract` and `image-pin-consistency`, 81 of 81. Like
the migration suites it needs docker, and only the image pull touches
the network. Helm 3.17.3 and 4.2.4 render the same rules. I have not run
the new rules in vmalert on a live cluster.
The rules file is the same on release-1.6 and release-1.7. The commit
cherry-picks onto release-1.6 without conflicts, and that backport is a
separate PR with one more commit that drops the `load test_helper` line,
because release-1.6 runs `hack/*.bats` through `hack/cozytest.sh`.
### Screenshots
Not a UI change.
### Downstream repositories
A rule template in `packages/extra/etcd`, a Bats suite under `hack/` and
its test data. No values, schema, README, image or
`ApplicationDefinition` change, and the website does not mention these
rules, so no trigger map entry applies.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
fix(etcd): the tenant etcd alert etcdHighNumberOfFailedGRPCRequests now counts only the gRPC codes that the etcd mixin and the platform copy of the alert count, so the Canceled watch streams of a Kubernetes API server that stops no longer fire it on every etcd member.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Refined etcd gRPC failure alerts to target specific failure status
codes, reducing alerts for non-failure responses while preserving
existing thresholds and timing.
- Added coverage for canceled watch streams, unavailable range requests,
and combined failure scenarios, including warning and critical alert
evaluations.
- **Tests**
- Added integration validation for rendering etcd alert rules and
running them through the alert-testing tool, including failure handling
when prerequisites are unavailable.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
614847976f |
test(harbor): pin the resources of the nginx proxy and the portal (#5025)
## What this PR does #4095 gave the Harbor nginx proxy the `t1.small` preset and moved the portal from `t1.nano` to `t1.micro`, so neither falls back to the LimitRange default of a tenant namespace, 250m CPU and 128Mi. It came without a test. This adds a helm-unittest suite that pins both presets in the nested HelmRelease and checks that they stay the same when the other components are resized, so dropping them or tying them to another component's values fails the suite. Both containers run nginx with `worker_processes auto`, which starts one worker per CPU of the node whatever the container's CPU limit is. With the proxy config the chart renders and `goharbor/nginx-photon:v2.15.1`, 96 workers idle at about 205 MiB, so on a node with many CPUs the proxy is OOMKilled at 128Mi before it serves a request. That is what happened on a v1.6.5 cluster, where both nginx pods of a default Harbor restarted 5 and 6 times within minutes, because `release-1.6` never got #4095. The backport, with this suite and the measurements, is a separate PR. `helm unittest .` in `packages/apps/harbor` passes 47 of 47 with helm-unittest 1.1.2. With the #4095 hunk reverted the three new cases fail and the 44 existing ones pass. Each of seven template mutations fails at least one new case. They drop the nginx block, give nginx `t1.micro` or the portal `t1.nano`, tie nginx to the core values or the portal to the jobservice values, misspell the `resources` key under nginx, and make core ignore its preset, which the resize case checks so that it cannot pass on overrides that never reach the release. ### Screenshots Not a UI change. ### Downstream repositories One new helm-unittest suite in `packages/apps/harbor/tests`. No template, values key, schema, default, README or package change, so no trigger map entry applies. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Tests** * Added Helm coverage verifying default resource settings for Harbor’s nginx and portal components. * Added coverage confirming nginx and portal memory allocations remain consistent when related service resources are customized. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
5265ee662c |
fix(tenant): stop the gateway release waiting on TenantGateway listeners and certificates
helm-controller waits on every object of a release with the kstatus poller on install and on every upgrade. kstatus reads a TenantGateway at Ready=False/ListenersNotReady as InProgress, and a TenantGateway sits there whenever one listener certificate is pending, which depends on ACME or DNS and, under HTTP-01, on routes that other releases own. After a platform upgrade the gateway release therefore times out, retries without end under RetryOnFailure, holds a helm-controller worker meanwhile, and fails the tenant release that waits on it. It used to pass because health is judged only inside a Helm action: the release was installed while the TenantGateway had no status yet or was still Ready, and the listeners whose certificates do not issue came later with other releases' routes. An always-true expression replaces kstatus for TenantGateway and keeps the wait on. Flux checks status.observedGeneration against the generation before it evaluates any expression, so the release still waits until the controller has observed the new spec, but not for listener or certificate health. The poller strategy is set explicitly because expressions are only evaluated under it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
59932e5098 |
fix(platform): report FoundationDB patch drift in migration dry runs
The carry path skipped diagnostics in dry-run mode, so its summary could report no releases needing attention even when the upgrade would lower a FoundationDB patch version. Skip only the write and share the warning and summary bookkeeping with a real run. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
639ede20be |
fix(api): leave FoundationDB version checks in the chart
The chart still reads cluster.version to guard release-line changes, so calling it ignored in an admission warning contradicts the render error. Keep that decision in the chart, which can explain the version mismatch, and limit removed-field warnings to inert values. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Andrei Kvapil
|
b43196e549 |
fix(kubernetes): pin the Proxmox cloud-config builders to the bundled image
The Proxmox substrate added two init containers that build the cloud config from the hypervisor token Secret, and both still took their image from images.waitForKubeconfig. With that key gone the render failed on a nil map; with it set, a tenant chose the image that reads the token. They now run the bundled busybox through cozy-lib.image, like the wait-for-kubeconfig container next to them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
Andrei Kvapil
|
4cb505a512 |
fix(platform): record 7.1 FoundationDB clusters instead of carrying them
The operator no longer carries the 7.1 client, and the chart no longer offers v7.1, so a 7.1 cluster has no line to be carried to. Migration 59 now records it, together with any release whose version already names a line that is no longer offered; both have to be upgraded or repointed before the chart will render them. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |