mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
e4e991e837be8b7b422d759d6735a16c1217a016
7497
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
e4e991e837 |
fix(backups): skip the CNPG restore purge only for this restore's Cluster
The in-place CNPG restore skips deleting the live Cluster when it carries bootstrap.recovery and was created after the RestoreJob started, taking that as "this restore's own purge and re-render produced it". The timestamp does not say which restore rendered the Cluster. A Cluster deleted while the restore waits for the WAL archive is re-rendered by the still-active HelmRelease from the previous restore's values and is created after StartedAt too; the restore then skipped its purge and reported Succeeded on the previous restore's data. The restore patch already sets bootstrap.newServerName to restoredServerName(cluster, RestoreJob UID). The chart now annotates the Cluster with it as postgres.cozystack.io/restored-server-name, whether or not it renders the barman plugin that also carries the value, and the purge is skipped only when the annotation names this RestoreJob. The mark lives on the rendered Cluster rather than in RestoreJob status, so it still covers the failed status write the skip exists for. A Cluster rendered by a chart that predates the annotation keeps the creation-time check. Fixes #3322 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
3daf942dce |
fix(backups): refuse a restore into an application another restore holds
Nothing stopped two RestoreJobs from rewriting one application at once. Each restores the target from its own Backup, so the application ends up on whichever finishes last, and the CNPG driver's purge decision cannot tell which of the two produced the live Cluster: the second job could skip its purge and report Succeeded on the first job's data. A RestoreJob that has not started now fails with reason ConcurrentRestore when another unfinished RestoreJob in the namespace writes into the same application and goes first. A started restore goes before one that has not; between two that have not, the older one goes first, by creation time and then name, so a running restore is never failed by a newer one. The refused job fails instead of waiting, so the application is not restored a second time later, after the first restore, by a job its author may no longer expect to run. The gate lists RestoreJobs through the manager's API reader rather than the informer cache. Creation times have one-second precision, so two jobs created in the same second are ordered by name, and a job reconciled before the cache held its peer's StartedAt would see that peer as not started and proceed alongside it. The controller runs one worker, so a peer's StartedAt written by an earlier reconcile is on the API server by the time the next job reads it; SetupWithManager now states that. The target is resolved as the BackupJob hold resolves it, so a Velero restore into another namespace does not collide with one in place. Refs #3322 Assisted-by: LLM Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
|
22e0fdbc63 |
test(tests): stop unit tests from reaching a cluster (#4804)
## What this PR does One unit test was reading from whatever cluster the caller's kubeconfig selected. The knob test in `hack/run-kubernetes-node-join_test.bats` runs `cozy_report_node_join_failure` in a fresh `bash -c` and stubs only `kubectl`, as a shell function. `cozy_diag_read` runs every read through `timeout`, and `timeout` execs the real `kubectl` from PATH, so the stub never ran. With a recording `kubectl` first on PATH, one run of that file logged 170 calls such as `kubectl -n tenant-test describe virtualmachineinstances.kubevirt.io`. All of them were reads, but a unit suite should not touch a cluster at all. The PR has three parts: 1. Both `bash -c` blocks in that file now stub `timeout`, as the post-source test already did. They also put recording `kubectl` and `talosctl` executables first on the subshell's PATH and fail if either one is executed. The declined-path block made no calls in my runs, because a zero budget declines every read. It gets the same guard because it has the same shape. 2. The unit and controller targets now run with no cluster to reach. The root `Makefile` sets `KUBECONFIG=/dev/null` and an empty `KUBERNETES_SERVICE_HOST` for `unit-tests`, each of its prerequisites, and `test-controllers`, so the CI command and each target run on its own are covered. The second variable is needed because client-go falls back to in-cluster credentials when the kubeconfig is empty and that variable plus a service-account token are present. The new `hack/unit-lane-no-cluster.bats` runs every prerequisite of `unit-tests` as its own goal with a sentinel kubeconfig and service host in the environment, and fails if any target receives them. 3. One unit file was missing the strict-mode helper. `hack/bats-strict-setup.bats` requires a `load test_helper` line in every unit file, and `hack/fork-new-images.bats` is the only one without it, so the bats lane is red on main and would stay red here. #4805 carries the same one-line fix on its own; if it lands first, this commit drops out on rebase. I ran the whole lane (`make unit-tests test-controllers`) with recording fakes for 20 cluster clients first on PATH. It made no `kubectl` call. `helm` was called 444 times, all `template` or `unittest`, none with `--validate`. `flux` was called once, for `push artifact --help`. Deleting either export line from the Makefile, or removing one target from the list, turns the new test red. Removing the `timeout` stubs turns the knob test red, and the failure lists the reads that escaped. Closes #4793 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note test(tests): unit and controller test targets run with no kubeconfig and no in-cluster service host, so a unit test can no longer reach a cluster. ``` |
||
|
|
bcde187858 |
fix(dashboard): small Console fixes from the #3828 triage (#3837)
## What this PR does Six small Console fixes from the #3828 triage. Each one is small and most of the code sits in `apps/console/src`, so they go together in one PR. `Fixes #3105`: sizes between 1Ki and 1Mi printed as a raw byte count. `humanizeBytes` had branches for Ti, Gi and Mi and then fell through to bytes. It now has a Ki branch with the same `toFixed(0)` as the Mi branch. The `1023B` case still holds, and the test covers 1Ki and 512Ki. `Fixes #3106`: the tenant picker showed on admin pages where the tenant changes nothing. It is now hidden on the cluster-scoped admin pages only: Capacity, Backup classes, External IPs and Modules. The rest of `/admin` mounts the same tenant-scoped pages as `/console`, and those take their namespace from the tenant context, so the picker stays there. `/admin/tenants` keeps it too, because its rows come from the picker-filtered tenant list. `Fixes #3107`: both capacity drill-downs showed one generic error, so a permission failure looked like a broken page. Both now use the `error instanceof K8sApiError && error.status === 403` check that `ClusterStorageSection` already uses, with a test for the forbidden case and one for a generic error on each page. `Fixes #3102`: an immutable leaf was not restored when the YAML edit dropped its parent object. `overlayPath` stopped as soon as the submitted body had nothing at a path segment. It now creates `{}` for the missing parent, but only when the target there is undefined or null and the original really has the leaf to restore. So a scalar or an explicit null the user put there survives. A wildcard with more path under it does not rebuild a deleted parent, since that would put back entries the user removed. The test uses foundationdb's `storage.storageClass`, which is a shipped path of this shape. `Fixes #3135`: a blocked submit did not scroll to the field that blocked it. Plain `focusOnFirstError` crashes here, because RJSF's built-in handler reads `form.elements` and this form is `tagName="div"`. So the form passes a small handler that finds the field by its generated id and scrolls it into view. The ids come from one shared definition, and the widgets that had none now carry it: the `DynamicOptionsWidget` select, which every `x-cozystack-options` field uses, and the `SourceField` controls. The nested form that `AdditionalPropertiesField` mounts per map entry continues the outer id namespace instead of restarting at `root`, so fields like `nodeGroups.*.instanceType` resolve. A map key with a dot in it, such as a mariadb user `john.doe`, resolves by trying both readings of the dotted error path. `Fixes #3108`: `cilium`, `coredns` and `verticalPodAutoscaler` have only `valuesOverride` and no `enabled`, so they rendered as plain groups next to the toggleable addons and looked like a switch that failed to appear. Setting `<addon>.enabled: true` in the YAML editor is accepted, because the schema sets no `additionalProperties`, and the API stores the field and echoes it back. Nothing reads it. The three addons are not the same case. `cilium` and `coredns` are always installed. `verticalPodAutoscaler` has no switch of its own and is installed and removed together with `addons.monitoringAgents.enabled`. The three are now marked in `values.yaml` with a new `x-cozystack-no-enable-switch` schema keyword, and the form renders them with a note that there is no enable switch and an `enabled` field in YAML does nothing. The per-addon reason goes into the schema description, which the form renders and which regenerates into the README, the Go types and the ApplicationDefinition. The generated files are regenerated with cozyvalues-gen v1.7.0, the version CI pins, and the pre-commit check passes. The console typecheck and tests pass in CI. ### Screenshots ##### Ki formatting <img width="1182" height="184" alt="image" src="https://github.com/user-attachments/assets/62662265-4f77-455d-b90e-85e417684bb9" /> ##### Admin without tenant picker <img width="1440" height="245" alt="image" src="https://github.com/user-attachments/assets/99aff4f3-fda9-4c4c-8cf8-0c1eb01fb973" /> ##### Capacity 403 error <img width="1232" height="270" alt="image" src="https://github.com/user-attachments/assets/f4e4b328-ee5e-4516-b78f-9b5084b6bbc8" /> <img width="1232" height="944" alt="image" src="https://github.com/user-attachments/assets/142e438e-f538-436e-aae8-dfb79ec747e8" /> ##### Validation focus <img width="1166" height="135" alt="image" src="https://github.com/user-attachments/assets/89abb73b-a624-406f-9736-3bf38bb8a923" /> ##### Addon lifecycle <img width="1166" height="591" alt="image" src="https://github.com/user-attachments/assets/14a95891-84d1-4073-a009-bc0eab94f114" /> ### Downstream repositories I walked the trigger map in `docs/agents/contributing.md` against the diff, file by file. This PR changes field descriptions, adds the `x-cozystack-no-enable-switch` schema keyword on three addon objects, and changes the console. No field is added, removed or renamed, and no default, version enum, `kind`, `plural`, `release.prefix`, output Secret or Service changes, so none of the terraform provider triggers fire. No package is added or removed, and nothing changes in `packages/core/platform/values.yaml`, the variants, `hack/` or the node prerequisites. cozyvalues-gen itself is unchanged, since the keyword goes through its generic `@x-*` passthrough. The kubernetes reference page on the website regenerates from `README.md` on a stable tag, which is automated rather than a follow-up. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard): Ki-range sizes now render as Ki instead of raw bytes, the tenant picker is hidden on cluster-scoped admin pages, capacity drill-downs distinguish a permission error from a broken page, an immutable field whose parent object is absent is now applied, a blocked submit scrolls to the field that blocked it including fields inside maps and the source and dropdown widgets, and the cluster addons with no enable switch now say so instead of showing a toggle that never appears ``` |
||
|
|
1b0a5a2829 |
fix(velero): bump velero-plugin-for-aws to v1.14.4 so backups work on R2 (#4834)
## What this PR does Bumps the `velero-plugin-for-aws` init container in `packages/system/velero/values.yaml` from `v1.14.1` to `v1.14.4`, and updates the pin in `tests/velero_test.yaml`. `v1.14.1` sets `x-amz-tagging` on every `PutObject`, even when the BackupStorageLocation configures no tags, in which case the header is empty. Cloudflare R2 rejects that with `501 NotImplemented: Header 'x-amz-tagging' with value '' not implemented`. Velero then cannot upload backup metadata, so every Velero backup to an R2-backed location fails. That includes Cozystack's own `cozy-default` BackupStorageLocation when the platform backup bucket is on R2. Upstream fixed this in v1.14.3: velero-plugin-for-aws#309 (a backport of #299, "Only set PutObject Tagging when tags are configured"). `v1.14.4` is the current patch release of the 1.14 line. Its only change on top of v1.14.3 is CVE dependency bumps (#336). v1.14.3 itself also carries #310 (SSE-C presigned URLs), #313 (no amz-checksum encapsulation when `checksumAlgorithm=""`) and CVE/Go bumps. The plugin's compatibility table lists `v1.14.x` for Velero `v1.18.x`, which is what the chart ships (`velero:v1.18.1`), so this stays inside the supported pairing. Testing done: - `make test` in `packages/system/velero` (helm-unittest v1.1.2): 7 passed, with the existing init-container pin assertion moved to `v1.14.4`. - `helm template` renders `velero/velero-plugin-for-aws:v1.14.4` as `initContainers[0]` of the `velero` Deployment. - On a Cozystack v1.6.4 staging cluster with the backup bucket on Cloudflare R2: before the bump, Velero backups failed on the empty `x-amz-tagging` header. After pinning the same plugin image through a Package values override (`docker.io/velero/velero-plugin-for-aws:v1.14.4@sha256:c85a7fa803569e3128c33e4f5766f84ab156c8d171c9f1dd2715581d45ce98fa`), all of our volume backup and restore drills pass. That covers CSI snapshots with the Kopia data mover to R2, and incremental backups confirmed. Backwards compatibility: same plugin minor line and the same Velero version. The init container image changes, so the `velero` Deployment rolls once on upgrade. Users who never set `tagging` on a location now get no tagging header at all; users who do set it get the same header as before. The reference stays in the existing shape (tag only, no digest), like the current `v1.14.1` line. If you'd rather pin third-party init containers by digest, the v1.14.4 multi-arch index digest is `sha256:c85a7fa803569e3128c33e4f5766f84ab156c8d171c9f1dd2715581d45ce98fa`. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Trigger map walked against the diff: a third-party image tag in a system package values file and its test; no app schema, platform values, names or tooling. Fixes #4852 ### Release note ```release-note fix(velero): velero-plugin-for-aws is bumped from v1.14.1 to v1.14.4, which no longer sends an empty x-amz-tagging header, so Velero backups work against S3-compatible stores that reject it, such as Cloudflare R2. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Updates** * Updated the AWS backup plugin used by Velero to version 1.14.4. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
cd1345e661 |
fix(redis): re-send the Sentinel auth-pass when the operator resets a Sentinel (#4833)
Fixes #4851 ## What this PR does Adds a downstream patch to the redis-operator image (`packages/system/redis-operator/images/redis-operator/patches/unauthenticated-sentinel.diff`) so that a Sentinel which lost its `auth-pass` is repaired by the operator instead of being reset forever. ### The failure After a node power cycle on a Cozystack v1.6.4 cluster (3 nodes, Talos, HA Redis with `authEnabled: true`), the Sentinel pod on that node came back monitoring the correct master address, but with no `auth-pass`. The master answered it with `NOAUTH`, it reported the master as `s_down,disconnected`, and `SENTINEL CKQUORUM mymaster` on it returned `NOQUORUM 1 usable Sentinels`. The operator logged `mismatch number of sentinels in memory. resetting` on every resync and sent `SENTINEL RESET *`. That does not touch `auth-pass`, so the Sentinel never recovered on its own: the instance keeps running with two working Sentinels out of three until someone deletes that pod. One more failure then leaves Sentinel without a majority. The mechanism, which I reproduced locally with Redis 8.6.2 (script and output in #4851): 1. The pod restarts in place (same pod, same IP), so the `sentinel-config-copy` init container writes the operator's initial `sentinel.conf` again: `sentinel monitor mymaster 127.0.0.1 6379 2` and no `auth-pass`. 2. The other Sentinels still know that address and keep sending their hello messages to it directly. A failover has raised the config epoch, so the restarted Sentinel adopts the current master address from a peer (`+config-update-from sentinel ... +switch-master mymaster 127.0.0.1 ... <current master>`) before the operator looks at it. 3. `CheckAndHeal` then finds the right address in `CheckSentinelMonitor` and skips `NewSentinelMonitor`, the only call that sends `SENTINEL SET <master> auth-pass`. 4. `INFO sentinel` reports `status=sdown`, so `CheckSentinelNumberInMemory` fails and `RestoreSentinel` sends `SENTINEL RESET *`. The reset keeps the empty `auth-pass`, and the cycle repeats. The same code is on `main`: freshworks-oss v3.3.5 sets `auth-pass` only in `MonitorRedisWithPort`, and none of the three patches Cozystack carries touch this path. ### The change `RestoreSentinel` sends `SENTINEL SET <master> auth-pass <password>` right after `SENTINEL RESET *` when the RedisFailover has auth enabled. It uses the same password `NewSentinelMonitor` uses (`k8s.GetRedisPassword`), the same TLS config, and the same master name (`rf.MasterName()`, so `disableMyMaster` is respected). - It only runs on the unhealthy path. `RestoreSentinel` is called only when a Sentinel's view is already wrong: its sentinel or replica count does not match, or its status is not `ok`. A healthy Sentinel is never touched. - It does not disturb a working Sentinel. `SENTINEL SET auth-pass` drops the Sentinel's links to the monitored instances, but the `RESET` just before it has already done that. - Without auth, `GetRedisPassword` returns an empty string and nothing is sent, so the behaviour is exactly as before. - Bootstrap mode goes through the same function and gets the same password `NewSentinelMonitorWithPort` already sends there. Patch contents: a `SetSentinelAuthPass` method on `service/redis.Client`, its mockery-style mock, the two-step `RestoreSentinel`, and `TestRestoreSentinel` (5 cases: auth off, auth on, reset error, unreadable Secret, `SET` error). The file name sorts after `tls.diff` on purpose: the patch is written on top of it (same functions, new `tlsConfig` parameters), and the Dockerfile applies `patches/*.diff` in glob order. If you prefer numbered patch names, I can rename it. Like `tls.diff`, this belongs upstream too: the same fix is open as freshworks-oss/redis-operator#90. This carried patch can be dropped once a release containing it is vendored here. ### Why not a quorum-checking liveness or readiness probe The RedisFailover CRD does allow `spec.sentinel.customLivenessProbe` / `customReadinessProbe`, so a `SENTINEL CKQUORUM` probe could be set from the chart. I looked at that first and decided against it: - A liveness restart does not repair this. It restarts the container, not the pod. `sentinel.conf` lives on an `emptyDir` that survives a container restart, and only the init container rewrites it, on pod start. The restarted Sentinel reads the same file without `auth-pass` and fails again, so the probe would restart it in a loop. - `CKQUORUM` does not reliably detect it. In the local reproduction the broken Sentinel still answered `OK 3 usable Sentinels`, because its peers reach it directly. On the cluster it answered `NOQUORUM`. What does show the problem every time is `status=sdown` in `INFO sentinel`, which the operator already reads. - It hurts in real outages. With two of three nodes down, a quorum-based liveness probe would restart the last Sentinel, and a quorum-based readiness probe would empty the `rfs-` Service, at the moment clients most need it. The operator already detects this state and already runs a repair for it. The repair just did not include the password. ### Testing done - Applied the four patches to a fresh v3.3.5 tarball in Dockerfile order (`git apply patches/*.diff`). Then ran the Dockerfile's test step: `go test .` and `go vet -tags integration .` in every directory the patches touch, all 11 pass. Also ran `GOOS=linux GOARCH=amd64 go build ./cmd/redisoperator`. I could not run `make image` because Docker was not available on my machine. - `TestRestoreSentinel` fails on 3 of its 5 cases without the `heal.go` change and passes with it. The existing `operator/redisfailover/...` and `service/redis` tests pass unchanged. - End-to-end against real Sentinels (Redis 8.6.2, local): master + replica with `requirepass`, three Sentinels configured the way the operator configures them, a `SENTINEL FAILOVER`, then one Sentinel restarted on the same port with the operator's initial config. Result: that Sentinel switched to the new master from a peer's hello and sat in `status=sdown`, `flags s_down,master,disconnected`. `SENTINEL RESET *` alone left it in `sdown`. The patched `service/redis` client (`ResetSentinel` then `SetSentinelAuthPass`) brought it to `status=ok,...,slaves=1,sentinels=3`. - The mock was written in the format mockery v2.45.0 generates; v2.45.0 itself does not build with Go 1.26+ (`golang.org/x/tools` v0.17.0), so I could not regenerate it mechanically. Backwards compatibility: no API or chart change. The operator image has to be rebuilt to pick up the patch, like the existing patches (the image digest in `packages/system/redis-operator/values.yaml` is stamped by the build). Instances without auth are unaffected. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Trigger map walked against the diff: one new file under `packages/system/redis-operator/images/redis-operator/patches/`; no values, schema, CRD, name or tooling change. ### Release note ```release-note fix(redis): the redis-operator now re-sends the Redis password to a Sentinel it resets, so a Sentinel that restarted in place (for example after a node reboot) and lost its auth-pass recovers on the next check instead of staying down until its pod is deleted. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Sentinel recovery now restores the Redis authentication password after a successful reset when a password is configured. * Recovery stops if the reset fails and reports errors encountered while retrieving or applying the password. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
f4bde9e50d |
fix(backups): delete the original PVC of an in-place VM restore without keepOriginalPVC (#4672)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
An in-place VMInstance restore with `keepOriginalPVC: false` reports
`Succeeded` without restoring the disks, and leaves every disk it
touched unable to come back. `prepareForRestore` deletes each DataVolume
with Orphan propagation and leaves the PVC in place, on the assumption,
stated in the `deleteDataVolume` comment, that Velero overwrites it.
Velero does not overwrite a PVC. Its CSI restore action returns early
for a claim that already exists, so no DataDownload is ever created, and
with `existingResourcePolicy: update` the merge patch it attempts
instead is rejected because a bound claim's spec is immutable. All that
reaches the claim is the labels-only fallback patch, which also drops
the `cdi.kubevirt.io/allowClaimAdoption` annotation the resource
modifier added.
The claim left behind then blocks the disk for good. Orphan propagation
has removed its owner reference to the DataVolume, and it lacks the
`cdi.kubevirt.io/storage.populatedFor` annotation that
kubevirt-velero-plugin writes on the backed-up copy, so when the
HelmRelease resumes, the CDI webhook refuses the DataVolume it creates
with `Destination PVC <ns>/<name> already exists`, on every reconcile.
Deleting the disk through the API leaves the PVC behind, and a new disk
of the same name fails the same way.
Observed on a v1.6.3 cluster, for a VMInstance restore with
`keepOriginalPVC: false` whose disk was attached to the VM. The Velero
log for the restore, trimmed:
```text
level=info msg="Retrieved per-restore resource modifiers from configmap cozy-velero/restore-modifiers-..."
level=warning msg="PVC already exists and ExistingVolumeDataPolicy is not in-place restore. Skip restore this PVC." Action=PVCCSIRestoreItemAction PVC=tenant-datalab/vm-disk-e2edisk
level=info msg="Applying resource modifier patch on tenant-datalab/vm-disk-e2edisk"
level=warning msg="patch attempt failed for /v1, Kind=PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk: PersistentVolumeClaim \"vm-disk-e2edisk\" is invalid: spec: Forbidden: spec is immutable after creation except resources.requests and volumeAttributesClassName for bound claims"
level=info msg="backup/restore labels successfully updated for PersistentVolumeClaim tenant-datalab/vm-disk-e2edisk"
```
`velero restore describe --details` lists the PVC as `(failed)` and the
Restore is `Completed` with two warnings. Afterwards the claim had its
original creation timestamp and UID, no owner reference, no
`allowClaimAdoption`, no `populatedFor`, and the disk's HelmRelease was
`Ready=False` with the webhook error above. That cluster runs a Velero
build from `main`, whose wording is quoted above; the released v1.17.1
and v1.18.3 return early at the same point with `PVC already exists.
Skip restore this PVC.` (`pkg/restore/actions/csi/pvc_action.go`).
The fix deletes the PVC when `keepOriginalPVC` is false, only once its
DataVolume is confirmed gone so that CDI cannot recreate it, and holds
the Velero Restore until the claim has actually disappeared. A claim
still mounted stays `Terminating` behind `kubernetes.io/pvc-protection`,
and creating the Restore then would hit the same existing-claim path.
Velero then creates the PVC from the backup, the data mover fills it,
and the recreated DataVolume adopts it through `populatedFor`. A failed
deletion now fails the RestoreJob instead of letting it report
`Succeeded` over the old data. Before
|
||
|
|
c290b75923 |
docs: link the documentation of the current release (#4761)
## What this PR does Points the documentation links in the README and GOVERNANCE.md at the current release. The unversioned `/docs/...` paths on cozystack.io are Hugo aliases owned by the v0 docs. Each one is a meta-refresh to the v0 page, so these links sent readers to the v0 documentation, a major version behind the current release: | Link | Went to | Now | |---|---|---| | README `/docs/guides/use-cases/public-cloud/` | `/docs/v0/guides/use-cases/public-cloud/` | `/docs/v1.6/guides/use-cases/public-cloud/` | | README `/docs/guides/use-cases/private-cloud/` | `/docs/v0/guides/use-cases/private-cloud/` | `/docs/v1.6/guides/use-cases/private-cloud/` | | README `/docs/guides/use-cases/kubernetes-distribution/` | `/docs/v0/guides/use-cases/kubernetes-distribution/` | `/docs/v1.6/guides/use-cases/kubernetes-distribution/` | | README `/docs/getting-started/` | `/docs/v0/getting-started/` | `/docs/v1.6/getting-started/` | | README `/docs/operations/troubleshooting/` | `/docs/v0/operations/troubleshooting/` | `/docs/v1.6/operations/troubleshooting/` | | GOVERNANCE.md `/docs/operations/configuration/telemetry/` | `/docs/v0/operations/configuration/telemetry/` | `/docs/v1.6/operations/configuration/telemetry/` | In the v1 docs the kubernetes-distribution path holds Build Your Own Platform (BYOP), installing only the packages you need onto an existing cluster that already has networking and storage, rather than the v0 "Kubernetes distribution for bare metal" page. The README entry is rewritten to match: "Using Cozystack to build your own platform", with "You can install only the Cozystack components you need on an existing Kubernetes cluster". The full version path follows the other versioned docs links in the repository, which all name `/docs/v1.6/`. The short `/docs/v1/...` path is not an option: only the site's 404 page redirects it, in JavaScript. All six new URLs return 200 without a redirect, and the same pages exist under the website's `content/en/docs/next/`, so the next release keeps them. The same alias problem in the website's own `llms.txt` is fixed in cozystack/website#737. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note docs: link the v1.6 documentation from the README and GOVERNANCE.md instead of the v0 pages the unversioned cozystack.io paths redirect to, and describe the kubernetes-distribution use case as Build Your Own Platform, which is what that page documents in v1. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated README use-case and documentation links to point to the Cozystack v1.6 documentation, including the Telemetry link in the governance guide. * Renamed the Kubernetes-distribution use case to “Build your own platform” and clarified that selected Cozystack components can be installed on an existing Kubernetes cluster. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b1192bfcbc |
fix(backups): create no driver object for a job that already finished (#4673)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
The BackupJob and RestoreJob reconcilers decide a job is still active
from the informer cache, and the cache can trail their own writes. A
driver's first pass stamps `StartedAt`, which requeues the job, and may
then fail the job in the same pass on a transient error, for instance a
503 from the aggregated `apps.cozystack.io` API while
`createVeleroBackup` reads the application. The requeued pass can start
before the informer has seen `Failed`: it retries the step that failed
and, once the error has cleared, creates the driver object anyway. Its
own status write then conflicts on the stale resourceVersion, the next
pass sees `Failed` and skips the job, and a Velero Backup is left
running that no `Backup` artifact will reference and nothing tracks.
Reproduced against a real apiserver (envtest 1.34) with the real manager
cache and the reconciler wired through `SetupWithManager`, nothing else
writing the BackupJobs. The only fault injected is one 503 on the first
GET of each application through `apps.cozystack.io`. Out of 40
BackupJobs, every one ends `Failed` with `failed to create Velero
Backup: the server is currently unable to handle the request`, and on
`main` (
|
||
|
|
a2e10312de |
fix(platform): stop treating a failed ConfigMap read as a fresh cluster (#4792)
## What this PR does A single failed API read at the start of the platform migration Job could skip every pending migration and still end green. `run-migrations.sh` checked whether the `cozystack-version` ConfigMap exists with a silenced `kubectl get` and looked only at the exit code. A timeout, a dropped connection or an apiserver restart during the upgrade all counted as "no ConfigMap", so the script stamped the target version and exited 0. The hook Job is rendered only when that ConfigMap already exists, so in practice an error was the only way into this branch. Now the script creates the ConfigMap only when it is really missing. The check uses `--ignore-not-found --output name`: kubectl drops NotFound and nothing else, so empty output means a genuine NotFound. Any other error keeps a non-zero exit, `set -e` aborts the script, and the Job retries the pod. The rest of the script already works this way: the read of the stamped version aborts on error, and the stamp runs under `pipefail`. The tests cover both branches. A new case in `hack/run-migrations.bats` makes the existence check fail with a connection error and expects a non-zero exit, no migration run and the stamp left where it was. It fails against the old check. The fake kubectl returns empty output for a missing ConfigMap only when `--ignore-not-found` is passed, as real kubectl does, so dropping the flag breaks the existing "no ConfigMap" case. No migration is added, so `targetVersion` stays. The script ships in the migrations image, and the release build restamps that image's digest as usual. Several older migrations have the same problem in a smaller form. They check for an object with `kubectl get ... >/dev/null 2>&1` and skip their own work when the check fails, for example migrations 31, 32, 36, 48, 50 and 51. A transient error there skips one migration's work, not all of them. Editing migrations that already shipped has its own risk, so I'd track that separately. Closes #4579 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(platform): a transient API error while the platform migration Job checks for the cozystack-version ConfigMap now fails the pod so the Job retries, instead of stamping the target version and silently skipping every pending migration. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Migration checks now distinguish a missing version record from an error while checking for it. When the check fails, the pod stops instead of treating the record as missing and proceeding with migrations. * Added coverage for TLS-handshake timeouts during the check, confirming that migrations do not run and the stamped version remains unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c61fcf0ee4 |
docs(governance): link the Telemetry page of the current release
The unversioned /docs/operations/configuration/telemetry/ path is an alias owned by the v0 docs, so GOVERNANCE.md sent readers to the v0 Telemetry page. It now names the v1.6 page, like the Licenses link a few lines below. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
b39f623338 |
docs(readme): describe the use case the kubernetes-distribution page documents now
In the v1 docs the kubernetes-distribution use-case path holds Build Your Own Platform: installing only the packages you need onto an existing cluster that already has networking and storage. The README entry still described the v0 page, Cozystack as a Kubernetes distribution for bare metal, so its label and description no longer matched the page it links to. Both now describe BYOP. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
59af194b73 |
docs(readme): link the documentation of the current release
The unversioned /docs/... paths on cozystack.io are aliases owned by the v0 docs, so the use-case, getting-started and troubleshooting links sent readers to the v0 documentation, a major version behind the current release. They now name the v1.6 pages, in line with the other versioned links in the repository. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
be2387323b |
feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does This keeps the TLS-passthrough services on port 443 of the tenant Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is stacked on #3343 and has to land with it (stack #4717). On 1.20, upstream Cilium marks an HTTPS-terminate listener and a TLS-passthrough listener on one port `Conflicted` when their hostnames overlap, and drops both from Envoy. That took `*.<apex>` and the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down together on every tenant Gateway in `dns01` and `existingSecret` mode (#3912). #3343 now ships a patched operator that conflicts the pair only when the hostnames are equal, so the layout stays as it is on main: one Gateway, one address, everything on 443. What remains is the equal-hostname case and the cleanup that 1.20 allows. Under `http01`, a name a passthrough listener answers is now reserved by the spec alone. Before, the controller dropped the terminate listener for such a name only once a TLSRoute was attached to the passthrough listener and its backend Service resolved, which copied how Cilium builds filter chains. Without a working TLSRoute, an HTTPRoute on `api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20 that equal-hostname pair takes both listeners down. Now such an HTTPRoute gets no listener and no certificate, TLSRoute or not. If all its hostnames are reserved it reads `Accepted=False` with `NoMatchingListenerHostname`. If it also claims a free hostname it stays `Accepted` and is served there. The backend lookup, its Service and ReferenceGrant watches and their RBAC rules are gone. Other changes: - `tlsPassthroughListeners` is accepted under `dns01` and `existingSecret`. The old refusal relied on 1.19.5 serving all ports from one Envoy listener; 1.20 splits ports whose SNI overlaps. - A TenantGateway reads `Ready=False` with reason `GatewayClassUnsupported` when its GatewayClass lists features in `status.supportedFeatures` but not `TLSRoute`, while it renders passthrough listeners. A class that lists nothing is not judged. Cilium 1.20.2 lists `TLSRoute`. - Comments, the README and one error message no longer describe Cilium 1.19.5 as the shipped release. - The gateway chart writes `tlsPassthroughServices` on the publishing tenant only, behind the same `gateway.enabled` switch and `publishing.exposedServices` filter as the platform TLSRoutes. Before, every tenant Gateway got the three passthrough listeners with nothing routed to them, and the reservation above would have taken those names from a child tenant's own HTTPRoutes. The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so removing it is a separate cleanup. ### Upgrade notes Only the publishing tenant keeps the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless `publishing.ingressName` says otherwise, and only for names in `publishing.exposedServices` while `gateway.enabled` is on. Other tenants lose these listeners on upgrade. Nothing shipped was attached to them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its listener and certificate. Two narrow cases do change. On the publishing tenant under `http01`, an HTTPRoute on an exposed passthrough name loses its listener and certificate when the platform TLSRoute for that name forwards nowhere (the owning package is not installed, or its backends do not resolve). A TLSRoute attached by hand to a child tenant's `tls-*` listener loses its parent. To keep that listener, merge-patch `tlsPassthroughServices` into the tenant's `TenantGateway` after the upgrade. From then on the chart does not render the key there, and later upgrades leave the patch alone. `dns01` and `existingSecret` do not change: the wildcard listener still covers such a name, and the passthrough listener answers it by its exact name. Tested locally: Go tests for the controller, `go vet`, helm-unittest for `extra/gateway` and `cozystack-controller`, `make generate` and `make manifests` with no diff. The gateway e2e scenario checks that a child tenant renders no passthrough listener. It then declares `tls-api` and a passthrough listener on a native port on that tenant, checks that the tenant stays Ready with no route behind either, and checks the reservation against `tls-api`. Unit tests catch a regression in either change. In e2e, dropping the reservation turns the gateway suite red, and dropping the chart scoping does not, because CI never sets `gateway.enabled` (#4846). The controller was not run on a live cluster; the operator patch it depends on was, see #3343. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/725 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * TLS passthrough listeners are supported with HTTP-01, DNS-01, and existing-secret certificate modes. Edge-terminated Gateways remain unsupported. * Declaring a passthrough hostname reserves it from HTTPS termination and certificate creation, even when no route is attached. * Passthrough listeners are rendered only for exposed services on the enabled publishing Gateway. * **Bug Fixes** * Gateways that declare passthrough listeners but explicitly lack TLSRoute support are now reported as not ready. * Routes targeting a hostname served by a passthrough listener on a different Gateway are correctly withdrawn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d9689bd6e7 |
fix(gateway): stop describing Cilium 1.19.5 as the shipped release
With Cilium at 1.20.2 the tenant gateway comments, README and one status message still explained behaviour as that of "the pin" at 1.19.5. Several of those claims no longer hold: a Gateway with two routed passthrough ports now gets an Envoy listener per port, Cilium judges a route's kinds against the listener the route names, and the shipped operator accepts an HTTPS and a passthrough listener on one port when the passthrough hostname is strictly narrower. Behaviour that is unchanged is re-anchored on the 1.20.2 source, behaviour that is gone is stated as history, and the refusal of one hostname on two passthrough ports no longer blames a Cilium limitation the shipped release does not have. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
ad3f15d4f3 |
feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a name a passthrough listener answers only once a TLSRoute was attached there and resolved a backend, mirroring how Cilium builds its filter chains. That kept a served endpoint up on Cilium 1.19.5, but it tied every route's status to one implementation's translator and needed Service and ReferenceGrant watches to land the deferred decision. The shipped operator accepts a wildcard terminate listener beside a narrower passthrough listener on 443 and keeps conflicting every other overlap. A terminate listener for the passthrough name itself is not narrower, so on 443 the rule now protects the dataplane outright; on a native port Gateway API would serve the pair, and the rule holds as a policy that keeps a platform name with its passthrough listener. Reading it off the spec makes the outcome the same on every implementation. Under http01 an HTTPRoute claiming such a name gets no listener and no certificate whether or not a TLSRoute is attached. A route whose every hostname is reserved reads Accepted=False with NoMatchingListenerHostname; one that also claims a free hostname stays Accepted and is served there, as Gateway API accepts a route whose hostnames intersect at least one listener. dns01 and existingSecret are unchanged: their wildcard listener still covers such a name, and the passthrough listener answers it by its exact server name. The chart writes the passthrough services on the publishing tenant alone, so the names reserved by default are the ones the platform publishes there through its own TLSRoutes. The shape an upgrade changes is an HTTPRoute on such a name whose platform TLSRoute forwards nowhere, because it never attached (the package owning the service is not installed) or because none of its backends resolve: that route kept a terminate listener and now gets none. The backend resolution, its watches and the RBAC rules that served them go away. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6144805bea |
fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)
## What this PR does The MariaDB chart's my.cnf has not been read by the server since the mariadb-operator bump to v0.28.1. Up to v0.0.27 the operator mounted the referenced ConfigMap key as `/etc/mysql/conf.d/my.cnf` whatever the key was called. Since v0.0.28 the file takes the key's own name, so the chart's key `config` became `/etc/mysql/conf.d/config`. The server's `!includedir` reads only `*.cnf` files, so it skipped it, and every instance has been running on stock MariaDB defaults. On the default `t1.nano` preset that means a 128M buffer pool inside a 128Mi memory limit. This renames the key to `my.cnf` in the ConfigMap and in the MariaDB CR. The file content is unchanged. I started all four shipped versions (10.6.24, 10.11.15, 11.4.9, 11.8.5) with the rendered file under a 128m memory limit, and each one came up with the chart's values. The MySQL-only options in the file log a warning and are otherwise ignored. They and the deprecated `innodb_flush_method` are tracked in #4842. Tests: a helm-unittest suite pins the key on both sides, and the mariadb chainsaw suite gets a step that compares `@@innodb_buffer_pool_size` on the running server with the value in the rendered my.cnf. This changes behaviour on every existing instance, see the release note. Each instance rolls its pods once because the projected volume in the pod template changes: replicas first, then the primary with a switchover. After that the server runs with every setting in the chart's file. Instances created before the operator bump (May 2024) ran with these settings before. Newer ones never did. Closes #4836 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(mariadb): the chart's my.cnf is applied again. Since mariadb-operator v0.28.1 it was mounted under a name the server ignores, so MariaDB instances ran on stock defaults. Every existing instance rolls its pods once on upgrade (replicas first, then the primary with a switchover) and then runs with the chart's settings. That changes behaviour: buffer pool, redo log, connection limits and caches are sized from the instance's resources; sql_mode becomes NO_ENGINE_SUBSTITUTION, so strict mode is off and invalid values are truncated with a warning instead of rejected; innodb_flush_log_at_trx_commit=2 can lose up to about a second of committed transactions if the node crashes; innodb_fast_shutdown=0 makes every pod shutdown slower. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * MariaDB configuration is now stored and read using the `my.cnf` key. A verification check confirms the running server applies the configured buffer-pool size. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4346a30f4b |
ci(e2e): run nightly and release-tag e2e on the container lane (#4828)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Nightly and the release-tag e2e (`e2e-tag.yaml`, the rc gate) now run on the Talos-container substrate the PR lanes moved to in #4020. This is the interim step from #4642, not the whole issue, so it does not close it. On QEMU the nightly full suite has been cancelled at its 215-minute timeout on every night since 2026-10-01 except 2026-10-07. From 2026-10-02 on, kubernetes-latest and kubernetes-previous failed their tenant node-join deadline almost every night, each failure took about 35 minutes, and the suites after them alphabetically never ran. In the 18 PR runs since 2026-10-03 that I checked and that ran them, they passed every time, at 7 to 10 minutes each. A nightly dispatched from this branch (run 37934155231) passed all 62 tests in 193 minutes. The kubernetes suites took 9 to 11 minutes there. Changes: - `nightly.yaml` and `e2e-tag.yaml` get the same host module step, `prepare-env-container`, endpoint and storage-class env and two-part teardown as `pull-requests.yaml`. Nightly's e2e no longer waits for the nocloud disk, which it does not boot. The disk is still built and published. - `e2e-tag.yaml` still requires the release to carry its disk but no longer downloads it. A dispatch against a tag cut before the container lane (v1.6.3 and older, v1.7.0-alpha.1) fails early and says to dispatch with `--ref <tag>`. When `tags.yaml` calls it, it runs the copy from the tag's own commit, so the guard does not fire there. - The comment above `timeout-minutes: 215` in all four e2e workflows claimed the budget covers two failing tenant Kubernetes suites. It does not, and the comment now says so. The value is unchanged. - `platform-packages_test.bats` now checks the container-lane env on all four workflows, and `collect-images_test.bats` covers the new guard. What loses coverage: DRBD and the `replicated` StorageClass (Immediate binding, tenant StorageClass propagation, RWX), the Cozystack Talos node image with its extensions, and the stock 600M CDI importer limit. None of them can run on containers, so no CI lane tests them until #4642 gets a lane with a kernel per node. The docs and comments that pointed at nightly and rc as that coverage now say this. Release lines keep their own `e2e-tag.yaml`, so rc tags on release-1.6 stay on QEMU until this is backported. That is why `kind/backport` is on this PR. Leftover QEMU-only text and diagnostics are collected in #4825. A dispatch of `e2e-tag.yaml` from this branch against `v1.7.0-alpha.3` is running as run 37958852914. I'll post the result here. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note ci(e2e): nightly and release-tag e2e run on Talos containers instead of QEMU guests. DRBD, the replicated StorageClass and the Cozystack Talos node image are no longer covered by any e2e lane. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **CI and Testing** * Release-tag and nightly end-to-end tests now run against container-based environments rather than booting disk images. * Release-tag checks still verify that the published release includes its disk image. * CI no longer covers QEMU-based DRBD scenarios or certain storage and Talos extension behaviors; these coverage gaps are documented. * **Documentation** * Updated E2E and release guidance to reflect the container-based test coverage and its limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8ab6d4948f |
test(mariadb): assert the server runs with the chart's my.cnf
The chart's unit tests can pin the ConfigMap key and the CR reference, but not what mariadb-operator does with them or which files the server then reads from /etc/mysql/conf.d. Compare innodb_buffer_pool_size as the live server reports it against the value in the rendered my.cnf, which differs from the server default on this fixture's preset. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d76bd1f263 |
fix(mariadb): mount the chart's my.cnf under a .cnf name
mariadb-operator up to v0.0.27 mounted the referenced ConfigMap key as
/etc/mysql/conf.d/my.cnf whatever the key was called. Since v0.0.28 the
file takes the key's own name, so the chart's key "config" became
/etc/mysql/conf.d/config, and the server's !includedir reads only *.cnf
files from that directory. Every setting in the chart's my.cnf has been
ignored since that operator bump: instances run on stock defaults,
including a 128M InnoDB buffer pool inside the 128Mi default limit.
Renaming the key restores the file as intended. Existing instances roll
their pods once, since the projected volume in the pod template changes,
and come back with the chart's settings in effect, including sql-mode
and innodb_flush_log_at_trx_commit, not only the resource sizing.
Fixes:
|
||
|
|
83aaa65ed1 |
fix(velero): bump velero-plugin-for-aws to v1.14.4 so backups work on R2
velero-plugin-for-aws v1.14.1 sets x-amz-tagging on every PutObject,
empty when the BackupStorageLocation configures no tags. Cloudflare R2
answers that with 501 NotImplemented ("Header 'x-amz-tagging' with value
'' not implemented"), so Velero cannot write backup metadata and every
backup to an R2 location fails, the cozy-default location included when
the platform bucket is on R2.
The plugin sends the header only when tags are configured since v1.14.3
(velero-plugin-for-aws#309, a backport of #299). v1.14.4 is the current
patch release of the 1.14 line, which upstream lists as the one
compatible with Velero 1.18, and adds only CVE dependency bumps on top
of v1.14.3.
Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
|
||
|
|
8bfd4e5f5e |
fix(redis): re-send the Sentinel auth-pass when the operator resets a Sentinel
A Sentinel that restarts on the same pod IP, as after a node reboot, comes back with the operator's initial sentinel.conf, which monitors a 127.0.0.1 placeholder and carries no auth-pass. Its peers still know that address and keep publishing hello messages to it, and once a failover has raised the config epoch the restarted Sentinel adopts the current master address from them before the operator looks at it. The operator then finds the right master address and skips NewSentinelMonitor, the only call that sends SENTINEL SET auth-pass. The master answers that Sentinel with NOAUTH, it reports the master as s_down, and the operator's resulting SENTINEL RESET on every check keeps the empty auth-pass, so the instance runs with one Sentinel fewer until someone deletes the pod. RestoreSentinel now sends the RedisFailover's password with SENTINEL SET auth-pass after the reset. It runs only when the Sentinel already looks unhealthy, and the reset has just dropped its links, so a Sentinel that had the password loses nothing. Without auth there is nothing to send. The patch file sorts after tls.diff because it is written on top of it; the Dockerfile applies patches/*.diff in glob order. Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com> Assisted-by: LLM |
||
|
|
f61e13fdef |
fix(cilium): size the agent memory limit for its BPF maps (#4803)
## What this PR does This raises the cilium-agent memory limit from 4Gi to 24Gi. The 4Gi limit from #3495 has no room for the agent's BPF maps. On nodes with roughly a terabyte of RAM or more, the agent is OOM-killed on every start after a reboot, and the node loses pod networking. The maps grow with node RAM. Cilium sizes its CT, NAT, neighbor and socket rev-NAT maps as a share of total RAM (`bpf-map-dynamic-size-ratio`, 0.0025 by default), and caps each map at 16Mi entries. These are LRU hash maps, so the kernel preallocates them in full. Since Linux 5.11 that memory is charged to the memcg of the process that creates the maps, which is the agent container. This is also why the upgrade itself works. A map reused from bpffs stays charged to the cgroup that created it. The failure shows up on the first reboot after the upgrade, when the maps are recreated inside the limited cgroup. Policy maps don't belong on that list. The issue and the upstream chart's configmap comment both say the ratio sizes them, but at this version they are LPM tries sized by `bpf.policyMapMax`, and they allocate per entry. One DaemonSet serves every node, so the limit can't follow node RAM. The map sizes are capped, though, so a fixed limit can cover any node. With every map at 16Mi entries for both IPv4 and IPv6, the maps take just under 20Gi of kernel memory. Add the 4Gi the agent already had and you get 24Gi. The request stays at 256Mi. The limit doesn't reserve memory, and the pod cgroup still gets `memory.max`, which is what #3495 was after. What it costs is the leak backstop. The agent is now stopped at 24Gi minus its own map memory, so on most nodes a leak in its heap is no longer caught before the node feels it. If someone prefers a tighter backstop over IPv6 headroom, they can lower `cilium.resources.limits.memory` in the networking component values. About 13Gi still covers the IPv4-only map ceiling. I didn't pin the map sizes (`bpf.ctTcpMax`, `bpf.natMax` and so on). No single value works for every node. A value small enough for a 16Gi node cuts the conntrack capacity of a 1.5TiB node many times over. A pinned size also differs from what almost every node above ~14Gi of RAM runs today, so the agent would recreate those maps on upgrade and drop their conntrack and NAT state. A lower ratio has the same problem and doesn't lower the ceiling. The new test keeps the ratio at its default and the sizes unpinned, so changing either has to be deliberate. The figures are computed, not measured. I read the sizing code at cilium 1.19.5 (`pkg/option/config.go`), and 1.20.2, which #3343 moves to, has the same code. Memory comes from that formula plus the kernel hash map layout: a 48-byte element header, the key and value rounded up to 8 bytes, and a 16-byte bucket per power of two of `max_entries`. By that math the IPv4 maps alone pass 4Gi at roughly a terabyte of RAM. Bucket rounding moves the exact point, so I wouldn't read more precision into it. For the 1.5TiB node in the report, the issue's 3.9GiB is the agent's own budget (0.25% of RAM), and the kernel's actual charge comes out higher. Follow-ups, not in this PR: - Operators who want smaller maps on large nodes can size them per node with a `CiliumNodeConfig` selected by node label. This needs a documentation note. - The reporter's `bpftool` measurement of the real map memory is still pending. That is the measured check for the numbers above. The cilium helm-unittest suite passes on this branch, and also with this change applied over the 1.20.2 chart from #3343. Each new assertion fails when its value changes: the limit, the CT, NAT and neighbor sizes, the socket rev-NAT size set through `extraConfig`, and the ratio. Closes #4800 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: The diff changes only `packages/system/cilium/values.yaml` and its test. None of the trigger-map entries apply: no app, schema, platform value, variant, release asset or tooling changes, and no website page documents the agent limit. ### Release note ```release-note fix(cilium): raise the cilium-agent memory limit from 4Gi to 24Gi so it covers the agent's BPF maps, which cilium sizes from node RAM. On nodes with around a terabyte of RAM or more, the 4Gi limit got the agent OOM-killed on every start after a reboot, leaving the node without pod networking. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Increased the Cilium agent memory limit to 24 GiB. * Updated validation for BPF map sizing: checks the dynamic size ratio and confirms that specific map-size settings are absent. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
52f09d1aed |
fix(gateway)!: render passthrough services only where platform routes attach
The chart wrote the tlsPassthroughServices default onto every non-edge tenant's TenantGateway, so every tenant Gateway carried tls-api, tls-vm-exportproxy and tls-cdi-uploadproxy. The TLSRoutes that serve those names are platform routes: the packages owning the services render one per name in publishing.exposedServices while gateway.enabled is on, parented to the publishing tenant's Gateway alone. Everywhere else no shipped route attached to the listeners, and a passthrough listener holds its name whether or not anything is routed through it. The chart now writes the field only on the publishing tenant, with the same gateway.enabled switch and the same exposedServices filter those routes use, so a listener exists exactly where a platform route can attach. A TLSRoute attached by hand to one of those listeners on another tenant loses its parent. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
bdca165f4c |
feat(gateway): report a GatewayClass that does not serve TLSRoute
A GatewayClass lists the features it serves in status.supportedFeatures, and TLS passthrough is TLSRoute. On a class that leaves it out the passthrough listeners are accepted and serve nothing, so the TenantGateway now reads Ready=False/GatewayClassUnsupported naming the class instead of Ready. A class that lists no features at all is not judged: implementations predating the field leave it empty. The class writes that list after it is accepted and may change it when its controller is upgraded, so a GatewayClass event requeues the TenantGateways on it. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d624049a3a |
feat(gateway): allow tlsPassthroughListeners under dns01 and existingSecret
The refusal existed because Cilium 1.19.5 served every Gateway port from one Envoy listener matched by SNI alone, so the "*.<apex>" terminate listener of the wildcard modes and a passthrough listener under the apex could reach each other's backend. The shipped Cilium 1.20.2 gives each port its own Envoy listener when SNI overlaps across ports, and Gateway API treats listeners on different ports as compatible, so the pair is served as declared. edge still refuses the field: it renders no TLS listener for an entry to bind to. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e1319b5a6c |
docs(cilium): say the SNAT backport is dropped by hand
The patch header cited a commit of cilium/cilium#49232, an open PR whose branch is force-pushed, so the commit is reachable from no ref. Cite the PR alone. It also read as if the build would flag the patch once upstream carries the fix. It does not: a fix that lands in another place leaves this patch applying cleanly as a second early return, so the drop is a manual step. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e816522366 |
chore(renovate): leave the cilium-operator image pins to make update
The operator rebuild compiles one Cilium release with the Go toolchain upstream's builder image pins at that tag, and make update re-pins the builder line from there, along with the operator-generic base, its digest and the source archive checksum. A Renovate bump of that line, by tag or by digest, would build the release with a toolchain upstream never shipped it with, and the next make update would revert it without notice. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
1bfa64f83a |
fix(cilium): keep per-type xDS while the 1.20 upgrade rolls
The 1.20 chart renders envoy-xds-mode "ads" and an adsConfig bootstrap for cilium-envoy unless xdsMode or upgradeCompatibility says otherwise. A 1.19 agent never registers the aggregated discovery service, and a 1.20 agent in ADS mode registers nothing else. The agent and envoy DaemonSets roll independently, so on every node where their versions differ envoy gets no configuration until both pods are replaced, and Gateway traffic stops on all nodes at once during the upgrade. Pin xdsMode to split, the per-type xDS both versions serve and the agent's own default when the key is absent. Moving to ADS is a separate step once every node runs 1.20. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
5b92ca3fc8 |
fix(cilium): serve HTTPS and passthrough on one port from one listener
When a Gateway needs per-port Envoy listeners, the operator emits one listener per HTTPS port and, with two or more passthrough ports, one per passthrough port, all named listener-<port>. A port carrying both gets two listeners with one name, and the agent skips the whole CiliumEnvoyConfig, so every listener of the Gateway stops serving while its status stays green. With one passthrough port the port is instead mapped to two listeners in the CEC services. A tenant Gateway with HTTPS and passthrough on 443 plus a passthrough listener on a native port reaches the first case. The patch puts the passthrough filter chains of such a port on its HTTPS listener and leaves the port out of the passthrough listeners and service mappings. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
cebf9f1e1e |
feat(cilium): accept wildcard HTTPS beside specific TLS passthrough
Cilium 1.19.6 and later mark an HTTPS-terminate listener and a TLS passthrough listener on the same port Conflicted whenever their hostnames intersect, and 1.20 drops both from Envoy. That rejects *.<apex> HTTPS next to api.<apex> passthrough on 443, which a tenant Gateway with one address needs. Envoy selects the most specific server_names match before the transport protocol, so the pair maps every SNI to exactly one filter chain. The operator is rebuilt from the upstream tag with a patch that accepts the pair when the passthrough hostname is strictly narrower than the HTTPS one, and keeps every other overlap conflicted. The asymmetry is needed because passthrough chains take their server_names from the attached TLSRoutes: under a passthrough wildcard a route may claim the HTTPS listener's exact name, and two chains with one match make Envoy refuse the whole listener. The binary replaces the one in the upstream operator-generic image, and the builder runs the patched packages' tests, so a tag bump that breaks the patch fails the build. make update re-pins both upstream images by the digests the vendored chart carries, the builder by upstream's own Go toolchain, and the source archive by its checksum. The image ref sits under operator.image.override: committed as the upstream operator the rebuild starts from, so a tree rendered without a build still pulls an image that exists, and stamped by make image. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c1fe1e9cb2 |
ci(e2e): say what the 215-minute e2e budget actually covers
The comment on every lane claimed the budget fits two tenant Kubernetes suites failing at their node-join deadline. A green full suite on the container lane already spends most of it, and a suite that runs into the deadline costs about 25 minutes more than one that passes, so the claim does not hold. State the real bound so a cancelled run is read as a red suite, not as a budget to raise. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
da28ea7ee8 |
ci(e2e): run nightly and release-tag e2e on the container lane
Nightly and the rc gate still booted the management nodes as QEMU guests, the substrate the merge-gating lanes left behind. On it the tenant Kubernetes suites have missed their node-join deadline on most nights since 2026-10-02, each failure spends the whole deadline, and the job reaches its timeout before the alphabetically later suites run. Over the same days those suites passed on every PR run that selected them, where the nodes are Talos containers and a tenant worker sits one virtualization level shallower. Both lanes now use the PR lanes' container substrate. Nightly's e2e no longer waits for the nocloud disk, which it does not boot; the disk is still built and published. A manual e2e-tag dispatch against a tag that predates the container lane fails early and names the ref to dispatch from instead. DRBD, the replicated StorageClass and the Cozystack Talos node image lose their last e2e coverage with this. The docs and comments that pointed at nightly and the rc gate as that coverage now say so. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
aa804aaa83 |
fix(dashboard): allow editing free-form objects in application forms (#4682)
## What this PR does Free-form object fields, such as NATS `config.merge` and `config.resolver`, get a JSON editor. The form keeps nested objects, arrays and primitive values. Input that is not a JSON object stays visible, and schema validation blocks the create or update request until it is fixed. A JSON object that fails the schema marks the field invalid and points it at the error list. Typed maps keep their key and value editor, and read-only and immutable fields stay protected. Fixes #4676. ### Validation - All 463 console tests in 63 files pass, and so does the TypeScript check. - Tests cover typing into the editor, nested NATS objects next to the typed users editor, input that is not a JSON object, schema errors on a valid object, existing values, immutable fields and free-form objects inside arrays. On the order page they cover a create after a form and YAML round trip, saving an edited object, and blocking input that is not JSON in create and edit mode. - The two schema-error tests fail on the component without the change. ### Screenshots   ### Downstream repositories This change preserves the public resource schema and client contract. The file-level trigger map was reviewed; no companion repository update is required. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard): allow editing free-form objects in application forms ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a JSON editor for configuration fields that accept objects with arbitrary properties. Edit nested values directly, with changes retained when switching between form and YAML views. * **Bug Fixes** * Invalid JSON is flagged and prevents create or update submissions until corrected. The editor supports empty objects, read-only fields, and objects nested in arrays, and makes validation errors accessible. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6f49e5f427 |
feat(openbao): optional static-key auto-unseal and apiserver egress label (#4766)
## What this PR does This continues #4168 by @txmazing, which has been idle since 2026-09-10. The first commit is the work from #4168 squashed into one, with @txmazing as author. The other two commits constrain the new values and complete the documentation. Five changes to the managed OpenBAO app (`packages/apps/openbao`). 1. **Optional static-key auto-unseal** (#2787, #2483). `seal.type` is `shamir`, the unchanged default, or `static`. With `static` the chart mounts the Secret named in `seal.secretName` through the upstream chart's `server.extraVolumes` and adds OpenBAO's `seal "static"` stanza to the standalone and the Raft config, so the pods unseal themselves after a restart. `seal.previousSecretName` and `seal.previousKeyId` carry the n-1 key during a rotation. The chart never generates the key. A cluster administrator creates the Secret in the tenant namespace, because the `cozy:tenant:*` roles grant nothing on Secrets. A way for tenants to supply secrets by reference is proposed separately in cozystack/community#82. 2. **No silent seal change.** The template reads the seal in use from the `<release>-config` ConfigMap and refuses a different `seal.type` unless `seal.allowMigration: true` acknowledges a migration. Going from `static` back to `shamir` is refused, because the chart cannot render the disabled stanza that direction needs. 3. **Apiserver egress label** (#2793). The server pods carry `policy.cozystack.io/allow-to-apiserver: "true"`, so `service_registration "kubernetes"` reaches the API server and HA pods no longer hang at start. 4. **Constrained values.** The upstream chart passes the server config through `tpl`, and its start script replaces upper case placeholders such as `HOSTNAME` in it. A key identifier with template delimiters would run as a template inside helm-controller, which runs as cluster-admin, so `seal.keyId` and `seal.previousKeyId` take lowercase alphanumerics, hyphens, dots and underscores. The Secret names become pod volume names `userconfig-<name>`, so they take DNS labels of at most 52 characters. A previous Secret or key identifier equal to the current one fails at render. The apps API only defaults against the schema, and Helm enforces these patterns at install and upgrade. 5. **README.** The trade-off of `static`, the key Secret, the rotation with the pod replacement that the `OnDelete` strategy requires, the migration from Shamir step by step after the OpenBAO guide, and the one-time pod replacement after an upgrade so that existing pods get the label. ### Validation - `helm unittest .` in `packages/apps/openbao` passes 28 of 28 with helm-unittest 1.1.2, the version CI pins, and with 1.2.1. - Dropping the key identifier patterns, moving the Secret name limit to 51 or 53, or removing either equality guard turns the suite red each time. - Rendering this chart and then the vendored `openbao` chart with a template in `seal.keyId` now stops at the schema. Without the pattern the same two-stage render executed it, `lookup` included. - With default values, standalone and with `replicas: 3`, `helm template` renders the same as `main` apart from the six-line `extraLabels` block. - `make generate` in the package with cozyvalues-gen v1.7.0 and at the repository root with Go 1.27.1 leaves the tree unchanged. `make unit-tests` passes apart from seven BATS tests that fail the same way on a clean `main` under macOS, most of them because macOS ships no `timeout`. In `make test-controllers` only `internal/backupcontroller` fails, because its tests do not build on current `main` either, which #4768 fixes. `hack/check-commit-trailers.sh` passes and `cmd/api-gate` reports no sizeable API change. - Nothing here was run on a cluster. #4168 reports a Cozystack v1.6.3 cluster where a `replicas: 1` instance with `seal.type: static` came back `Sealed false` after `kubectl delete pod`. TLS between pods and an init or bootstrap job are not covered. Both are on the #2787 list and independent of this change. ### Screenshots No dashboard code change. The application form gains the `seal` group from the regenerated `openAPISchema` and `keysOrder`. ### Downstream repositories Walked the trigger map against the diff. The new `seal` field in `packages/apps/openbao/values.schema.json` reaches the hand-written `cozystack_openbao` resource of the Terraform provider. That is tracked in cozystack/terraform-provider-cozystack#44, which also needs `allowMigration` and the value constraints. The website reference page is generated from this `README.md` for the apps listed in its `Makefile`, and `openbao` is in `APPS`, so the page follows the next stable tag without a follow-up. The website pages that mention the apiserver label describe it in general terms and carry no OpenBAO workaround. There is no `hack/` change, no platform values change, no variant and no change to ApplicationDefinition semantics. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [x] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: https://github.com/cozystack/terraform-provider-cozystack/issues/44 - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(openbao): the OpenBAO app can auto-unseal with OpenBAO's static-key seal (`seal.type: static`) from a Secret that a cluster administrator creates in the tenant namespace, with n-1 rotation through `seal.previousSecretName` and `seal.previousKeyId`. Shamir stays the default, and changing the seal of an existing instance needs `seal.allowMigration`. Server pods now carry `policy.cozystack.io/allow-to-apiserver: "true"`, so HA instances no longer hang at start behind the apiserver egress policy. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * OpenBAO supports static sealing with keys stored in an existing Secret; Shamir sealing remains the default. * Configure current and previous keys to support key rotation in standalone and high-availability deployments. Keep the previous key until stored keys have been re-wrapped. * Seal-type changes are validated. Shamir-to-static migration requires explicit acknowledgement; static-to-Shamir migration is not supported. * Static-seal settings validate required key details and Secret names. Server pods receive a label needed by the configured API-server egress policy. * **Documentation** * Added guidance for configuring static sealing, managing Secrets, rotating keys, and migrating seal types. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8c8f2522ee |
fix(cilium): keep the host mark on host traffic to a local pod
Since v1.20.0 the bpf_host SNAT conflict check fires for any packet carrying the host mark on every device bpf_host runs on, and the SNAT path replaces that mark with SNAT_DONE. With kube-ovn chaining a host connection to a pod on the same node leaves through ovn0, so the pod's to-container program sees SNAT_DONE and classifies the source as world instead of host. The implicit localhost allow then stops matching: kubelet probes fail under any default-deny ingress policy that does not admit world, and a world deny drops the API server calling a webhook on its own node. Carry a verbatim backport of cilium/cilium#49232, which skips the conflict check when the destination is a local endpoint: such traffic never leaves the node and cannot collide with pod egress SNAT. Keeping the hunks identical to upstream lets the patch be dropped as soon as a shipped cilium release contains the fix. Without chaining, host traffic to a local pod does not pass to-netdev and nothing changes. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
b64e12b0cf |
docs(cilium): re-anchor comments the v1.20.2 bump made stale
The admission-webhook deny policies in ingress-nginx and kubeovn-webhook cite Cilium source and the vendored values by line number. The bump moved endpointLockdownOnMapOverflow and the policy-map ordering block in pkg/endpoint/bpf.go, so the cited lines pointed elsewhere. Name the key and the function instead, which survive the next bump, and move the getEndpointSelector citation to v1.20.2, where the cited lines are unchanged. The kubeProxyReplacement pin explained the service translation in terms of v1.19.5. With Gateway API on, v1.20.2 keeps socket LB in the host namespace, so pod-namespace connections are translated per packet in bpf_lxc before egress policy runs, which is what the pin relies on. The same holds for the e2e image mirror, whose egress-policy comment in hack/e2e-talos-image-cache.yaml described socket LB. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
e60461dc56 |
docs(cilium): stop tying the endpoint-leak race note to one release
The note said the restore-window race had no released fix as of v1.19.5, which reads as current on every later pin whether or not anything changed upstream. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0f31e7a148 |
fix(cilium): bump vendored chart to v1.20.2
v1.19.5 rejects platform HTTPRoutes on a shared tenant Gateway with NotAllowedByListeners (cilium#42159). The 1.19 line that fixes it also marks a same-port HTTPS-terminate and TLS-passthrough pair ProtocolConflict while still programming both, which leaves the TenantGateway Ready=False on a layout that serves. 1.20 drops such a pair from Envoy outright, so the listener layout is fixed against 1.20 rather than against a 1.19 status quirk, and the bump goes straight there. The update target now follows the 1.20 line. v1.20 moved the ARP case of do_netdev under ENABLE_IPV4, which broke the anchor of the IPv6 passthrough patch; the patch's ETH_P_IPV6 case now sits right before `default:`, outside the ENABLE_IPV4 block. The other two cases apply unchanged. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
3dc0d046b3 |
fix(postgres): cap the default shared_buffers at a quarter of the memory limit (#4724)
## What this PR does The chart renders no `shared_buffers`, and CloudNativePG does not set one either (its default parameters in `pkg/postgres/configuration.go` at v1.30.0, the operator version this repo ships, have no such key). PostgreSQL therefore starts with its built-in 128MB whatever the container gets. A full buffer pool is the whole memory limit on `t1.nano` and half of it on `t1.micro`, which is the chart's default preset. The pool is shared memory that the kernel cannot reclaim without swap. A small or idle database never fills it, but a dump, index scans over a large table or a replica catching up after a failover can, and then the instance is OOMKilled. Observed on a cluster running Cozystack 1.4.3, whose chart renders the same parameters, with a three-instance `t1.nano` database. A `pg_dumpall` run through `kubectl exec` inside the primary's container got that container OOMKilled. The dump client ran in the same container and shared its limit, so that first kill is weak evidence on its own. After the failover the old primaries could not rejoin. Their `pg_rewind` against the new 128Mi primary kept failing with `could not fetch file list: SSL SYSCALL error: EOF detected` until that primary's memory limit was raised to 512Mi, and then both rewinds completed. Below a 512Mi memory limit the chart defaults `shared_buffers` to a quarter of the limit with a 16MB floor, capped at the rendered memory request rounded down to whole MiB. CloudNativePG rejects a Cluster when `shared_buffers` exceeds that request, which can happen with memory overcommit. That is 32MB on `t1.nano` and 64MB on `t1.micro`. From 512Mi up a quarter is at least 128MB, so nothing is rendered and those clusters keep the PostgreSQL default exactly as before. An explicit `shared_buffers` in `postgresql.parameters` still wins at any size. The limit is read through `cozy-lib.resources.defaultingSanitize`, the helper that fills `spec.resources`, which is also how `packages/apps/mariadb/templates/_tuning.tpl` sizes the InnoDB buffer pool. Presets and partial `resources` overrides therefore resolve exactly as they do for the container. A quarter is the starting point the PostgreSQL documentation gives for a dedicated server with 1GB or more, and it advises a smaller share below 1GB, so this is a ceiling for small instances rather than tuning. Larger presets are left alone on purpose. Deriving a bigger pool for them is a tuning change that would restart every PostgreSQL instance on upgrade, and it does not belong in a fix. ### Upgrade impact `shared_buffers` needs a restart. On upgrade every cluster whose memory limit is below 512Mi and that does not set `shared_buffers` gets the new value, and that includes every cluster created with the default preset. The chart leaves `primaryUpdateStrategy` and `primaryUpdateMethod` at their CloudNativePG defaults, `unsupervised` and `restart`, so the operator restarts the replicas first and then the primary in place, and writes pause for the length of one PostgreSQL restart. A single-instance cluster is unavailable for that time. Clusters from 512Mi up render the same manifest as before and are not touched. ### Validation - `helm unittest` passes all 166 tests in 21 suites. The new suite covers the default preset, `t1.nano`, a `resources` override that sets only cpu, a 500Mi limit just under the threshold, the 16MB floor, `t1.small` and `t1.large` rendering no key, an explicit value on a small and on a large preset, and the other default parameters. Seven additional cases cover memory overcommit, rounding down a fractional MiB request, a request below the 16MB floor, explicit values and the unchanged 512Mi threshold. Four cases fail against the previous PR template. - Breaking the threshold comparison, the 16MB floor or the divisor, or dropping the new block, turns at least one case of the suite red each time. - `helm template` of the whole chart renders `t1.small` exactly as the base does, and the default preset differs only by `shared_buffers: "64MB"`. - `make generate` with cozyvalues-gen v1.7.0, the version CI pins, leaves `README.md`, `values.schema.json`, the Go types and the ApplicationDefinition unchanged, because `values.yaml` only gains a comment. - The new default was not exercised on a live cluster. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. It touches one chart template, a comment in `values.yaml` and a new test file. No package is added, renamed or removed, no field in `values.schema.json` is added, removed or renamed, and no default in `values.yaml` changes, so the Terraform provider has nothing to restate and the generated README is identical. Neither the website nor the provider mentions `shared_buffers`. No entry matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(postgres): default shared_buffers to a quarter of the memory limit with a 16MB floor, capped at the memory request, when the limit is below 512Mi and shared_buffers is not set. PostgreSQL's built-in 128MB was the whole memory limit on t1.nano and half of it on t1.micro, the default preset, so a dump, index scans over a large table or a replica resync could fill the buffer pool and get the instance OOMKilled. Affected clusters restart their instances once on upgrade, the primary last, with a short pause in writes. Clusters from 512Mi up and clusters that set shared_buffers explicitly are unchanged. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL now automatically adjusts its default memory allocation for deployments with memory limits below 512 MiB, while retaining PostgreSQL’s default for larger limits. Explicitly configured values continue to take precedence. * **Documentation** * Clarified how the automatic memory allocation works, including its minimum and memory-request cap. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
e7bd3f4c58 |
fix(mongodb): set the s3 bucket and prefix on the bootstrap restore (#4779)
## What this PR does helm fails on every install or upgrade of a mongodb release with `bootstrap.enabled: true`. the chart renders a `PerconaServerMongoDBRestore` without `spec.backupSource.s3.bucket`, the operator crd marks that field required, and the api server rejects the object. ```text PerconaServerMongoDBRestore.psmdb.percona.com "<release>-restore" is invalid: [spec.backupSource.s3.bucket: Required value, <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid] ``` the bucket is not the only gap. for a `backupSource` restore the operator builds the pbm storage from `backupSource.s3` alone. the only location data it takes from `backupSource.destination` is the backup name, its last path segment. that is `getBackup` and `getStorage` in `pkg/controller/perconaservermongodbrestore` at v1.22.0, the operator version this repo ships. so a restore with a bucket and no prefix would pass validation and then look for the backup at the bucket root. the fix adds two helpers in `templates/_backup.tpl` that take the bucket and the prefix from `backup.destinationPath`, with the same expressions the backup storage already used. `mongodb.yaml` and `restore.yaml` both call them, so the restore reads from where the backups are written. both places quote the values now. a numeric bucket or prefix like `s3://2024/10` used to render as ints that the crd rejects and now stays a string. the backup storage still leaves out an empty prefix, so for every other `destinationPath` running releases keep the same object. the restore always sets the prefix, an empty one as `""`. a `destinationPath` that names no bucket, like `s3://`, now fails at render time instead of reaching the operator with an empty bucket. the bug goes back to when the chart was added. both crd versions the repo has shipped, 1.21.1 and 1.22.0, require the bucket, so no chart version could ever create this restore. release-1.6 has the same template and gets a backport. tested with helm-unittest 1.1.2, the version ci pins. - `helm unittest` passes 151 of 151 tests in 11 suites. before this change it was 143 in 10. - six new cases in `tests/restore_test.yaml` cover the bucket and prefix with and without a trailing slash, a bucket root, a numeric bucket and prefix, the render-time failure, and the check staying off when bootstrap is disabled. - the new `tests/restore-storage_test.yaml` renders the cluster and the restore together and checks that bucket, prefix, endpoint, credentials secret, `insecureSkipTLSVerify` and `forcePathStyle` match on both sides, and that a numeric bucket and prefix stay strings on both. - against the old templates 7 of the 8 new cases fail. the eighth only pins that the new check stays inside the bootstrap block, so it passes either way. - `helm template` of the `PerconaServerMongoDB` for 8 `destinationPath` forms. 5 render the same storage as before. for the two bucket root forms the only difference is that a `prefix: null` the api server prunes anyway is no longer rendered. `s3://2024/10` goes from ints the crd rejects to strings. - the rendered restore was checked offline against the crd in `packages/system/mongodb-operator` with the apiextensions validation code the api server runs. before the fix all 8 forms fail with `spec.backupSource.s3.bucket: Required value`. after it 7 pass and `s3://` stops at render time. - `make generate` with cozyvalues-gen v1.7.0, the version ci pins, leaves every generated file unchanged. - no restore was run against a live cluster. on upgrade, releases with `bootstrap.enabled: false` render the same manifests as before. a release that still has `bootstrap.enabled: true` creates its restore on the first reconcile after the upgrade, and the operator starts restoring into the cluster as it is at that moment. the release note says so. three related things are out of scope here. the restore runs in the pbm-agent sidecar, which the operator only adds when `spec.backup.enabled` is true, so a bootstrap with `backup.enabled: false` still cannot complete. the credential mismatch in #3832 is not addressed, a bootstrap restore goes through the same pbm logical restore that issue describes. and a restored release backs up to the same `backup.destinationPath` it restored from, since the chart has one field for both, so with backups on it writes into the prefix of the source. ### Screenshots not a ui change. ### Downstream repositories walked the trigger map in `docs/agents/contributing.md` against the diff. it touches two templates, one new helper file and two test files in `packages/apps/mongodb`. no package is added or renamed, `values.yaml`, `values.schema.json` and `README.md` are unchanged, and nothing under `hack/` moves. the website page for mongodb is generated from the readme and has no bootstrap walkthrough to update. no entry matches. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(mongodb): a mongodb app created with bootstrap.enabled now gets a valid PerconaServerMongoDBRestore. the restore carries the s3 bucket and prefix of backup.destinationPath, the same location the backup storage writes to, so the api server accepts it and the operator looks for the backup in the right place. before this the api server rejected the restore because the bucket was missing, and the release failed. on upgrade, an app that still has bootstrap.enabled set to true, for example after an earlier restore attempt that failed on the missing bucket, creates its restore on the first reconcile, and the operator restores the backup into the cluster as it is at that moment. set bootstrap.enabled to false before the upgrade on every mongodb app that should not be restored. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * MongoDB bootstrap restores now use the bucket and prefix from the configured S3 backup destination, keeping restore storage settings aligned with backup settings. * **Bug Fixes** * Bootstrap now reports an error when the S3 destination does not specify a bucket. * Numeric-looking bucket and prefix values are preserved as strings in backup and restore settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
4b606036bb |
fix(mongodb): set the s3 bucket and prefix on the bootstrap restore
With bootstrap.enabled the chart rendered a PerconaServerMongoDBRestore without backupSource.s3.bucket. The operator CRD requires that field, so the API server rejected the restore and the HelmRelease failed. The prefix was missing too, and it matters just as much. For a backupSource restore the operator builds the PBM storage from backupSource.s3 alone and takes only the backup name from backupSource.destination, so a restore without the prefix would look for the backup at the bucket root. The restore now takes the bucket and prefix from the same helpers that fill the backup storage, so it reads from where the backups are written. Both places quote the two values, so a numeric bucket or prefix stays a string instead of making the resource invalid. The backup storage still leaves out an empty prefix, so running releases keep the same object. A destinationPath that names no bucket fails at render time instead of reaching the operator with an empty bucket. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
3ea7a95108 |
docs(openbao): document the static seal trade-off and the Shamir migration
With the static seal the key sits in a Secret in the same cluster as the data it unwraps, so the barrier is only as strong as access to Secrets in that namespace, to etcd and to backups. The README now says so before the reader picks a seal, and it says what the key identifier is for, since OpenBAO picks the key for stored data by it. Tenant roles grant nothing on Secrets and no pods/log, so a cluster administrator creates the key Secret and reads the log line that confirms a re-wrap. The StatefulSet uses the OnDelete strategy, so neither a seal migration nor the apiserver egress label reaches a running pod until it is replaced. The README describes the migration from Shamir step by step, following the OpenBAO seal migration guide, and says that pods created before the label have to be replaced once. The migration starts with a backup. The standalone file backend is not transactional, so its volume is copied with the server stopped or snapshotted atomically. The upgrade that sets allowMigration is the point of no return for the chart, and the README says how a cluster administrator backs out of it before any pod is replaced. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
bfbb5ac75f |
fix(openbao): constrain the seal values that reach the server config
The upstream chart passes the server config through tpl, and its start script replaces upper case placeholders such as HOSTNAME anywhere in the copied file. A key identifier with template delimiters would run as a template inside helm-controller, which runs as cluster-admin, and an upper case one could come out different on every pod. Both key identifiers now take lowercase alphanumerics, hyphens, dots and underscores. The Secret names become pod volume names of the form userconfig-<name>, which Kubernetes validates as DNS labels. Both name fields now take a DNS label of at most 52 characters, and the volume entries are quoted. The apps API only defaults against the generated schema, so Helm enforces these patterns when it installs or upgrades the release. A previous Secret equal to the current one mounts the same volume twice, which the API server rejects for the StatefulSet, and OpenBAO refuses to start when equal key identifiers name different key material. Both cases now fail at render. The deepcopy functions for the Seal type are regenerated as well. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
0c52dedbe2 |
feat(openbao): optional static-key auto-unseal and apiserver egress label
The OpenBAO app has no auto-unseal (#2787). Every restart of a server, after a node drain, an image bump or an OOM kill, leaves it sealed until an operator enters the Shamir key shares into each replica. In HA mode the pods also hang at start (#2793). Cozystack's allow-to-apiserver policy only admits egress from pods labelled policy.cozystack.io/allow-to-apiserver, the app never set that label, and service_registration "kubernetes" cannot reach the API server without it. A new seal value selects shamir, which stays the default, or OpenBAO's seal "static". With static the chart renders the static seal stanza into the standalone and the Raft config and mounts the key Secret named in seal.secretName through the upstream chart's server.extraVolumes. The chart never generates, stores or rotates the key, because a missed lookup would replace it and leave the data unreadable. seal.previousSecretName and seal.previousKeyId carry the n-1 key that the static seal accepts during a rotation. The server pods always carry the apiserver egress label. Changing the seal of an initialized instance is a seal migration that OpenBAO performs under bao operator unseal -migrate. The chart reads the seal in use from the config ConfigMap the previous release rendered and refuses a different seal.type unless seal.allowMigration acknowledges the migration. Moving away from static needs the old stanza kept with disabled = "true", which the chart does not render, so that direction is refused regardless. The README documents the key Secret, the rotation with the pod replacement that the OnDelete update strategy requires, and the log line that confirms the re-wrap. Signed-off-by: txmazing <txmazing@gmail.com> Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
9c0085c358 |
fix(postgres): cap shared_buffers at the memory request
CloudNativePG rejects clusters whose shared_buffers exceeds the memory request. Memory overcommit can put that request below both a quarter of the limit and the 16MB floor. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
6dbd76ae50 |
fix(postgres): cap the default shared_buffers at a quarter of the memory limit
The chart renders no shared_buffers, so PostgreSQL starts with its 128MB default whatever the container gets. A full pool is the whole memory limit on t1.nano and half of it on t1.micro, the default preset. The pool is shared memory that the kernel cannot reclaim without swap, so when a dump, index scans over a large table or a replica catching up after a failover fill it, the instance is OOMKilled. Instances below a 512Mi memory limit now default to a quarter of the limit, at least 16MB. From 512Mi up a quarter is at least 128MB, so those instances render exactly as before. shared_buffers needs a restart, so on upgrade CloudNativePG restarts every instance below 512Mi that does not set it explicitly. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
96b35893a5 |
fix(dashboard): mark a free-form field invalid when the schema rejects it
A JSON object that parses but fails the schema, for example on minProperties, left the textarea with aria-invalid false, and nothing linked it to the error list RJSF renders under the field, so assistive technology announced the field as valid. The field now reports the schema errors RJSF passes in and points aria-describedby at their list, unless hideError keeps that list from rendering. The red help text stays tied to input that is not a JSON object. Assisted-by: LLM Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> |
||
|
|
d639113caf |
fix(dashboard): allow editing free-form objects in application forms
Signed-off-by: Yan Bondarenko <202671653+yankawai@users.noreply.github.com> Assisted-by: LLM |
||
|
|
87a64071a7 |
fix(cilium)!: take cilium's own addresses from a block kube-ovn does not hand out (#4778)
## What this PR does
fixes #3740.
on the kube-ovn variants cilium allocates no pod ips, but the agent
still takes its router ip (`cilium_host`, `CiliumInternalIP`) and the
gateway api ingress ip from its own allocation cidr. with `ipam.mode:
kubernetes` that's `Node.spec.podCIDR`, a /24 out of the same /16
kube-ovn hands pods from, so sooner or later kube-ovn gives a pod an
address cilium already holds. we hit it on three clusters, details in
https://github.com/cozystack/cozystack/issues/3740#issuecomment-6022394243.
**what changes**
- the kube-ovn variants run cilium with `ipam.mode: cluster-pool`, a /29
per node out of the last /21 of `networking.podCIDR` (`10.244.248.0/21`
for the default /16). the platform derives the block from
`networking.podCIDR` and passes the same range to kube-ovn as
`--default-exclude-ips`, so a fresh install never hands it to pods.
- the block sits inside the pod cidr on purpose. envoy reaches gateway
api backends from the node's ingress ip, and kube-ovn masquerades host
originated traffic only when the source is in `ovn40subnets`. with the
`100.65.0.0/16` pool from the issue a nodeport through the gateway
answered 200 only on the backend's node and 503 `upstream connect error
... connection timeout` on the other two. with the block inside the pod
cidr it's 200 from all three. worth a look on the ci side too. the ci
sandbox sets talos `podSubnets` to `100.65.0.0/16`, so cilium's ingress
ips there already come from outside `ovn40subnets`, and the gateway e2e
checks Accepted, Programmed and the lb service but sends no request
through a listener, so it wouldn't notice. with this pr the sandbox
takes its block from the pod cidr like everyone else.
- the end of the /16 is what kube-ovn reaches last. it hands out the
lowest free address and reuses released ones only when the free range
runs dry (`pkg/ipam/subnet.go` and `pkg/ipam/ip_range_list.go` at
v1.15.26). a /21 cut into /29s covers 256 nodes.
- the slim variants and guest clusters stay on `ipam.mode: kubernetes`.
cilium is the only cni there and `Node.spec.podCIDR` is the pod range.
- `networking.podCIDR` now needs /20 or wider on the kube-ovn variants,
otherwise the render fails and says why. an install that rendered before
can stop rendering at upgrade, hence the `!`. a /21 cut into /29s also
caps these variants at 256 nodes, a fresh install included.
**existing clusters**
a post-upgrade hook in the cilium chart moves them
(`templates/ipam-migration.yaml`, `files/ipam-migration.sh`). before it
touches anything it checks that the block has a /29 for every CiliumNode
and that it lies inside kube-ovn's default subnet, and stops with both
cidrs printed otherwise. a node cleared past the pool's capacity would
lose pod networking on its next agent restart, and a block outside the
subnet puts the agents outside `ovn40subnets` again. it has to be
post-upgrade. platform migrations run as the platform's pre-upgrade
hook, while every agent still runs `ipam.mode: kubernetes` and writes
`Node.spec.podCIDR` back into a cleared CiliumNode
(`pkg/nodediscovery/nodediscovery.go` at v1.19.5), so the move can only
happen after the cilium release switched the mode. the job runs on the
host network against the agent's own api endpoint (kubeprism on talos),
because it restarts the agents.
1. adds the block and every node's current ingress ip to `excludeIps` of
the default subnet, in one json patch with a `test` op, and reads the
subnet again on conflict, since kube-ovn rewrites that list into its own
order right after a patch. the old ingress ip goes in because cilium
1.19.5 keeps a former ingress ip in the agent's ipcache as
`reserved:ingress` after the move. we saw 2 to 3 such entries per node
hours later. old router ips leave the ipcache and go back to kube-ovn.
2. stops right there if an address in the block is still allocated,
names it, and moves no agent. it first checks the subnet status it
patched against, then waits until `status.v4availableIPrange` stops
offering the block, which is kube-ovn having applied the exclusion, and
then reads the `ips.kubeovn.io` objects of the default subnet. the
status alone can't answer it, kube-ovn leaves excluded addresses out of
`v4usingIPrange` (`GetSubnetIPRangeString` in `pkg/ipam/ipam.go` at
v1.15.26), so a pod still in the block would be invisible on a retry
where the exclusion is already in place. recreate those pods and the
next upgrade retry goes through.
3. one node at a time, clears the inherited /24 from
`spec.ipam.podCIDRs` (again with a `test` op, so a concurrent writer
fails the patch instead of losing an update), waits for the operator to
allocate a /29 from the block and restarts that node's agent. a node
that isn't ready is skipped and its agent picks up the block when it
starts. nodes already moved are left alone, so a retry after a timeout
just continues. the hook runs inside the cilium upgrade, so where
restarting every agent takes longer than the upgrade timeout, 10m by
default, the release first reports a failed upgrade and the move
finishes on the retry.
running pods keep their addresses. every agent restarts twice on
upgrade, once when the config change rolls the daemonset and once from
the hook, and while it restarts host to pod traffic on that node blips
for a few seconds because the router ip moves. a pod that sits on a
router ip today is reachable again after the move, the node no longer
holds that address. a pod on a node's current ingress ip keeps it and
that node keeps seeing it as `reserved:ingress`, so recreate it. the
hook logs those pods by name.
### Validation
- `helm unittest` passes cilium 23/23, platform 246/246 and kube-ovn
22/22. the new suites pin the rendered `cilium-config` keys per variant,
the hook shape, env and rbac, and the platform wiring for the talos and
generic branches, a custom pod cidr, host bits in the network address,
the refusals for a too small or non ipv4 cidr, and the slim variant.
- on a trial merge with main at
|