mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
06e72b26dd73c69beb2933122773277bd7efc8e8
7397
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
06e72b26dd |
test(backups): give the restore stale-cache test a Completed Velero Backup
A Velero RestoreJob is about to require its Velero Backup to exist and be Completed before it halts the application. Without one in the fixture, the first pass of this test fails the RestoreJob before it suspends the HelmRelease, so the test would stop exercising the stale cache it is about. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
eafa441b9c |
fix(backups): create no driver object for a job that already finished
The BackupJob and RestoreJob reconcilers decide a job is still active from the informer cache, and the cache can trail their own writes. A driver's first pass stamps StartedAt, which requeues the job, and may then fail the job in the same pass on a transient error, such as a 503 from the aggregated apps.cozystack.io API while it reads the application. The requeued pass can start before the informer has seen Failed: it retries the step that failed and, once the error has cleared, creates the driver object anyway. Its own status write then conflicts, the pass after it sees Failed and skips the job, and a Velero Backup is left running that no Backup artifact will reference and nothing tracks. Against a real apiserver and the manager's cache, with one 503 on the first application read of each job, 5 to 10 BackupJobs out of 40 ended Failed with a Velero Backup. A restore pass that starts on such a stale copy does worse, since the step it retries can be destructive: suspending the VM's HelmRelease and halting the VM before creating a Velero Restore, or patching the Postgres application and purging its cluster. Both reconcilers now re-read the job through the manager's uncached reader when the cached copy says it is still active, and skip it when the apiserver holds it as finished. The check sits ahead of the dispatch, so it covers every driver, and a job the cache already reports as finished costs no extra request. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
6addef4d3c |
fix(kubevirt): route the export proxy to the Service KubeVirt creates (#4637)
## What this PR does The VM export routes point at a Service that does not exist, and the tenant network policy would block the Ingress even if it did. KubeVirt creates the export proxy Service under the name `virt-exportproxy` (`VirtExportProxyName` in `pkg/virt-operator/resource/generate/components`, v1.9.0). The Ingress and the TLSRoute that publish `vm-exportproxy.<host>` name a `vm-exportproxy` Service, and nothing creates one. The backend was `virt-exportproxy` when the Ingress was added in |
||
|
|
8c1a78dc9b |
feat(operator): default system namespace memory via LimitRange (#3494)
## What this PR does
System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.
The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.
An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.
Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.
### Why the default is 32Gi
The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.
### When the default is withheld
A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.
The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.
### Namespace another LimitRange already governs
If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.
This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.
### Disabling
Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.
### Docs
`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.
### Not in this PR
linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.
The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.
### Verification
`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.
The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
|
||
|
|
84e8fcd477 |
fix(dashboard): re-authenticate on 401 instead of failing silently (#3115)
## What this PR does When the dashboard session expires, the console now sends the user back to sign-in instead of sitting on a stale page. This re-applies cozystack/cozystack-ui#34 by @IvanHunters onto the in-tree console, since the UI was vendored into the monorepo (#2963) and the standalone repo is being archived. It is part of the cozystack-ui retirement tracked in #3097. When the `kc-access` cookie expires, k8s API calls return 401. `K8sClient` now treats a 401 as an auth boundary and makes one top-level navigation to `/oauth2/sign_in?rd=<current-url>`. That path is the sign-in entry both gatekeeper modes serve. With OIDC enabled, oauth2-proxy goes straight to the provider (`--skip-provider-button`) and returns the user to `rd`. Without OIDC, token-proxy renders its token form there, but it does not honour `rd` yet, so the user lands on `/` after sign-in. The handler can be replaced through `K8sClientConfig.onUnauthorized` and does nothing outside the browser. A latch keeps concurrent 401s from redirecting more than once, and any 2xx from either fetch site re-arms it, so a handler that re-authenticates in place still sees the next expiry. Related: #2788. ### Verification `client.unauthorized.test.ts` covers the hook on the request and watch paths, the latch across concurrent 401s, the re-arm after a success, and the default handler's target and relative `rd`. `pnpm typecheck` and the console test suite pass. Original PR: cozystack/cozystack-ui#34, credited via `Co-authored-by`. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard): on a 401 (expired session) the console now sends the user to sign in again instead of silently showing a stale page. With OIDC enabled the user returns to the page they were on; without OIDC they land on the start page after entering a token. ``` |
||
|
|
3e078b79a9 |
ci(e2e): refuse a new fork image before the build starts (#4767)
## What this PR does A fork PR that adds a first-party image cannot pass the fork e2e lane, and today it finds that out only after the whole build. `e2e-fork.yaml` pushes the fork's image archives only under names the base tree already exports, so a new image is refused at the push with `image '<name>' is not in the base-tree allowlist`. Nothing the contributor does on the PR changes that. This PR keeps the push allowlist as strict as it is and moves the failure to the start of the run. On fork PRs, `plan` in `pull-requests.yaml` now runs `hack/fork-new-images.sh`. The script derives the image names of the PR tree and of its base parent with the same `make -n` derivation the push step uses. If the PR tree exports a name the base does not, it fails before any build, names the images and says what has to happen: a maintainer lands the image on the base branch first, or the change runs from a branch in this repository. The check runs from the fork's own tree, so a fork can remove it. That is fine, because it only saves an honest contributor a wasted build. The push allowlist in `e2e-fork.yaml` is still the boundary. That workflow cannot call the script, since it runs from the default branch over an untrusted checkout. So `hack/fork-new-images.bats` pins the two derivations to each other, and also covers the script's behaviour and the fork-only gating of the step. I replayed the script against the merge commit of #4291 and its base parent. It fails with `cluster-api-provider-kubevirt`, the same name the push step refused there. The workflow_run lane itself cannot be exercised locally. This does not lift the limitation itself: a fork still cannot get a green run for a new image. Part of #4329. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): a fork PR that adds a new first-party image now fails in the plan job, naming the image, instead of after the whole build at the image push. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Fork pull requests that add image names not present in the base branch are now rejected during planning, with the new names reported before builds begin. * **Documentation** * Updated guidance describes the early check and clarifies how it relates to the image allowlist used during publishing. * **Tests** * Added coverage for new, unchanged, reduced, and empty image sets, along with checks that the pull request workflow applies the check only to fork submissions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3cce629535 |
feat(redis,postgres): certify the per-engine external hostname (#4627)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Redis and postgres server certificates now carry the per-engine external name `<release>.<engine>.<apex>` (for example `redis-foo.redis.example.org`) next to the flat `<release>.<apex>` they already had. Both names appear under the same condition as before: TLS on and `external: true`, with a tenant apex present. A TLS-passthrough listener on the tenant Gateway forwards the handshake untouched, so the certificate the client checks is the engine's own. The design in #2815 routes databases by SNI under `*.redis.<apex>` and `*.postgres.<apex>`, and without the per-engine name in the SAN list a verifying client would fail the hostname check. Issuing both names now means a later move between the two hostname schemes reissues no certificate and keeps clients on the same trust anchor. This is the certificate step of #2815. Nothing renders a route yet. The SAN list changes, so an existing external TLS release gets its server certificate reissued once on upgrade, from the same CA. For redis the operator then rolls the Redis and Sentinel pods one at a time, the same way it handles a renewal. The READMEs say who can reach the new name through a passthrough listener. For postgres that is only clients using direct TLS negotiation (server 17+, `sslnegotiation=direct`), because by default a plaintext SSLRequest comes before the handshake. `redis-cli` needs `--sni`. MongoDB is not in this PR. PSMDB issues the certificate itself and its CRD has no field for extra SANs, so adding the mongos name means the chart taking over the certificate. That needs its own change. Covered by helm-unittest: the name is present with TLS and external on, and absent with external off, with TLS off, and in a tenant with no apex. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(redis,postgres): server certificates of external redis and postgres releases also carry the `<release>.<engine>.<apex>` name that the tenant Gateway's TLS-passthrough listeners are designed to route by. Existing external TLS releases get their server certificate reissued once on upgrade, from the same CA. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL TLS certificates now cover both the tenant hostname and the per-engine `postgres` hostname for external connections. * Redis TLS certificates now also cover the per-engine `redis` hostname when external services are enabled. * **Documentation** * Clarified connection requirements for per-engine hostnames: PostgreSQL routing requires direct TLS negotiation, PostgreSQL 17 or later, and a compatible client; Redis clients must specify SNI. * Clarified that the hostnames may need to be mapped manually to the LoadBalancer address. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b1c84e9c06 |
feat(platform, tests): select E2E packages and declare controller dependencies (#4022)
## What this PR does E2E installs the whole platform on a fresh cluster for every PR, whatever the PR touches, so install time and flake surface go to packages the selected suites never use. This PR makes the E2E install take only what the selected suites need. `hack/select-install.sh --disabled <suites>` generates `bundles.disabledPackages` for fresh E2E installs using an ordinary platform variant. It keeps suite owners, runtime requirements and the install baseline, then follows their forward `PackageSource` dependencies. The baseline follows the unconditional waits in `hack/e2e-install-cozystack.bats` and stays seeded directly even where the graph reaches the same packages. It also keeps platform services that no suite owns but the rest of the platform uses: `multus`, `cozystack-scheduler`, `linstor-scheduler`, `flux-plunger`, `cozy-proxy` and `ouroboros` (the Kubernetes suites assert its hairpin NAT). Backup suites also keep `backupstrategy-controller` and its dependencies, and the gateway suite keeps `kubeovn-webhook`. The E2E job now selects the suites before it installs, and passes the complement to the platform Package as `disabledPackages`. The `full-e2e` label and an empty selection keep the full install, and the nightly and tag lanes pass no list. The platform sources also change: `mongodb-application` now depends on `mongodb-operator`, and `seaweedfs-application` on `objectstorage-controller`. These dependencies apply to regular platform installs and upgrades too. The new `mongodb-operator` edge also reaches reverse test selection: a change under `packages/system/mongodb-operator` now selects the `mongodb` suite instead of falling back to the full suite. Backup APIs and the gateway webhook remain suite-specific install requirements, so they add no edges. The disable list governs fresh installs because existing Packages carry `helm.sh/resource-policy: keep`. Optional packages still need `bundles.enabledPackages`. Packages that no suite owns and the baseline does not keep stay uninstalled on purpose, also when test selection answers with every suite (monitoring exporters like `goldpinger` and `kubeovn-plunger`, apps without a suite, provider-specific operators). Nothing in the run looks at them, so installing them only adds a readiness wait, and a change to such a package gets no install check from this gate. ### Measured One run per variant on a 32 vCPU / 128 GB VM, container lane, the CI artifacts of this PR, `postgres` suite selected, 55 packages disabled: - HelmReleases 111 to 77, pods 219 to 197 - pod requests 14.8 CPU / 37.7 GiB to 13.5 CPU / 35.7 GiB - host memory peak during install 33.4 GB to 30.2 GB - `install-cozystack` 26m33s in both runs - `postgres` suite 25m50s to 25m17s, passes on the reduced install Install time did not change. The removed packages are mostly operators and apps without instances, and what the install waits on (SeaweedFS, LINSTOR, CDI, the Cluster API providers, Keycloak) stays in the baseline. With one run each, differences of a minute or two are noise. The selection, the install closure and the workflow wiring are covered by bats tests (`hack/select-install_test.bats`, `hack/select-e2e_test.bats`, `hack/platform-packages_test.bats`) and by a helm-unittest suite in the platform chart. ### Screenshots Not a UI change. ### Downstream repositories The new dependencies ship in the platform chart. No coordinated changes are needed in downstream repositories. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(platform, tests): E2E installs only the packages the selected suites need. MongoDB now depends on its operator, and SeaweedFS on the object storage controller. ``` |
||
|
|
798094e725 |
feat(release): add backport-audit, a pre-release backport triage tool (#3527)
## What this PR does
This adds `cmd/backport-audit`, a maintainer tool that checks, before a
release or an rc is cut, whether everything labelled for backport
actually landed on the release branch. It prints the URLs of the PRs
that did not and exits non-zero, so the cut can gate on it.
`docs/release.md` and `docs/agents/releasing.md` now send the patch-cut
check through it.
The docs it replaces inventoried backports with `gh pr list --search
"is:merged label:kind/backport"`. That lists every PR ever labelled, not
what reached the branch. Answering the real question by hand means
reading the branch history per PR, and that is how backports go missing
quietly.
```console
$ go build ./cmd/backport-audit
$ ./backport-audit release-1.4
=== release-1.4 === 30 candidate PRs: MISSING=5 dropped=4 backported=21
MISSING -- labelled for this line, no trace of it here (5):
https://github.com/cozystack/cozystack/pull/2704
#2704 fix(info): use root-host for Keycloak OIDC issuer URL in tenant kubeconfig
label=kind/backport author=myasnikovdaniil merged=2026-05-26 -- no backport PR, nothing on branch
```
A `kind/backport` label does not name a branch. `backport.yaml` resolves
the target when the PR merges, so the same label means `release-1.5` on
a PR merged in June and `release-1.6` on one merged in August. The audit
reproduces that rule instead of trusting the label text. The rule
changed once, on 2026-08-03, when
|
||
|
|
5486d4ffcf |
fix(networking): harden webhook policy assumptions (#4058)
## What this PR does This is follow-up for [#3997](https://github.com/cozystack/cozystack/pull/3997). It reads ingress-nginx admission port from effective subchart values and refuses empty value, and fixes `kubeProxyReplacement=false` status because cozystack has no supported Service translator in that state. Gateway test now requires one real ClusterIP and keeps same ready Pod UID, IP and port for whole probe, so stale EndpointSlice, replaced Pod or wrong Service can't report deny. Policy selectors and two-ended deny from [#3997](https://github.com/cozystack/cozystack/pull/3997) stay unchanged, and this does not claim root cause for [#3983](https://github.com/cozystack/cozystack/issues/3983). ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Admission webhook connectivity checks now validate the configured Service and its exact backend endpoint throughout probing, improving failure detection. - Admission webhook policies now fail closed when the webhook port is not explicitly configured. - **Documentation** - Clarified that supported service routing requires `kubeProxyReplacement: true`. - Updated guidance to explain that disabling this setting or webhook policies does not provide a supported fallback. - **Tests** - Added coverage for missing webhook port configuration and strengthened networking stability checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ac4ab1c4a6 |
chore: drop private cluster and host names from the tree (#4461)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does I remove internal cluster names, a real hostname with its IP addresses, internal mirror hostnames and an internal product name from comments, an example file, test fixtures and a dead commented-out config block. Where a stand-in is needed I use neutral test-cluster wording, RFC 2606 domains or RFC 5737 documentation addresses. I also replace two real Keycloak usernames and a reviewer's name in code comments with placeholders, or drop them. Rendered output does not change. The test fixture hostnames change on the input and the expected side together, and the unit tests of every touched package still pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated backup, OIDC, autoscaling, and platform references to describe test environments without naming specific development clusters. * Replaced development-specific endpoints in the platform example with `example.org` and documentation-only IP addresses. * Updated OIDC username examples. * **Tests** * Updated fixture hostnames and expected URLs to use `example.org`. * **Chores** * Removed commented-out registry mirror entries; the active mirror configuration is unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bc9a594e47 |
test(clickhouse): add backup API render assertions (#4011)
## What this PR does This adds unit tests for how the ClickHouse chart wires up the backup API that the default backup strategy calls. The Pod of the `cozy-default-altinity` strategy reaches the clickhouse-backup sidecar on the named port `ch-backup-api`. It authenticates with the credentials from the `<release>-backup-api-auth` Secret. The new suite is `packages/apps/clickhouse/tests/backup_api_contract_test.yaml`. It checks that: - the sidecar exposes `ch-backup-api` on 7171/TCP; - `API_USERNAME` and `API_PASSWORD` come from the `username` and `password` keys of `<release>-backup-api-auth`; - that Secret renders with `username: backup` and a non-empty password; - no auth Secret renders when backups are disabled. All of this is rendered by the chart, so the unit tests catch a regression on every pull request. The e2e assertions on the same wiring run only when test-impact analysis selects the ClickHouse suite. They stay as they are in `hack/e2e-chainsaw/clickhouse`, because they check the operator-created Pod and the admitted Secret, and rendering can't do that. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
|
a1416655f4 |
fix(tenant): allow tenant egress to the KubeVirt export proxy
The vm-exportproxy Ingress is served by the ingress controller in tenant-root, and tenant egress is default-deny. Nothing allowed traffic to virt-exportproxy in cozy-kubevirt, so the Ingress stayed unreachable even with the right backend Service. The new policy renders wherever allow-to-cdi-upload-proxy does, but narrows the destination to the export proxy pods and their container port. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c069a191fc |
fix(monitoring-agents): give kube-state-metrics a sized memory limit (#3846)
`kube-state-metrics` ran BestEffort with no resources at all: the
subchart ships `resources: {}` and our wrapper never set any. So it put
no `memory.max` on its pod cgroup, and the Talos userspace OOM handler
only ever selects cgroups that lack one, which left it permanently in
the victim set. Any tenant workload driving node memory pressure could
evict the source of every kube-state metric we alert on. Background is
in `docs/operations/system-memory-limits.md`.
## The numbers are measured
On a three node cluster with 322 pods and roughly 25.6k series, which is
the smallest we run: 14 day peak working set 307Mi, peak RSS 280Mi, mean
176Mi.
So the request sits above the mean at 192Mi rather than at the peak, and
the limit carries wide headroom at 2Gi. The headroom is deliberate: this
component's memory scales with the number of objects in the cluster, and
the `--metric-labels-allowlist=pods=[*],deployments=[*]` already in this
file widens every pod and deployment series further, so a cluster larger
than the one measured lands well above what was seen here. `2Gi` is 6.7x
that peak, which read as linear in object count is roughly 2100 pods and
170k series, so the limit stays a ceiling rather than a bound well past
the largest cluster this sizing was drawn from.
No CPU limit, following the node DaemonSet sizing in `c9ba7a3cc`, which
set one zero times across nineteen files. CPU is irrelevant to the OOM
ranking, and a ceiling that binds on a scraper turns into scrape
timeouts rather than backpressure.
## This supersedes #2632
#2632 proposes 256Mi, which is below the 307Mi peak measured on the
smallest cluster we have, so it would convert an eviction risk into a
reliable OOM kill. It also adds a `cpu: 200m` limit, which is the half
that buys nothing. The diagnosis there was right and it sat since May;
the numbers are what needed replacing.
## Checks
`packages/system/monitoring-agents/tests/kube_state_metrics_resources_test.yaml`
pins the limit, both requests, and the absence of a CPU limit. Mutation
checked: dropping the limit to 256Mi turns it red. The chart's suites
are 3 files and 6 tests, all green.
### Release note
```release-note
fix(monitoring-agents): kube-state-metrics now carries a memory request and limit sized from a measured peak, so it is no longer a permanent candidate for the node OOM handler
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configured kube-state-metrics with requests of 50m CPU and 192Mi
memory, plus a 2Gi memory limit. No CPU limit is configured, allowing
CPU usage to scale as needed. These settings help ensure the service has
the memory and CPU resources it needs while keeping memory use bounded.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
9927c299d4 |
fix(operator): keep deleting packages out of memory policy creation
A foreground-deleting Package can survive long enough to reconcile after cleanup retires its default. It must not reinstate that policy or consume an acknowledgement while its components are being removed. An active Package sharing the namespace can still manage the default. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9d5d6d8f45 |
test(operator): exercise failed memory safety scans
Foreign ownership short-circuited the read-error fixtures before their interceptors ran. Require the injected failure to occur and preserve both policy and Namespace state, including a pending acknowledgement. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
5b585d6eda |
fix(operator): persist holds for unsafe memory defaults
An admitted Pod loses the distinction between explicit and defaulted limits. Its disappearance cannot prove a lower default is safe for its replacement. Keep that decision on the Namespace until an administrator acknowledges the particular target, without waiving current blockers or foreign policy ownership. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
30ecc41070 |
fix(operator): close memory policy safety gaps
Preserve foreign same-name objects and keep create-time SSA non-force. Only ignore failed pods controlled by terminal Jobs. Make the failure-containment test inject Apply errors for real. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
22a27e75ac |
test(operator): cover memory cleanup failures
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
e83b4eb492 |
fix(operator): align memory policy ownership messages
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: LLM |
||
|
|
0d5b2e35c4 |
docs(operator): align scan comments with withholding
Replace three stale descriptions of the removed ceiling-raise behavior with the current withhold and withdraw semantics. Historical explanations of why the raise design was rejected remain intact. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b3168a3824 |
test(operator): pin system memory defaults
Keep the parser inputs tied to the constants registered as flag defaults, but compare the results with literal 32Gi and 32Mi expectations. A shipped default can no longer move while the defaults case stays green. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a521f0c68 |
fix(operator): sweep memory defaults when disabled
List LimitRanges by the ownership label once per Package reconcile when the feature is disabled, and remove managed objects even in namespaces outside the current Package target set. Skip the per-namespace path while disabled and keep cleanup failures non-fatal to namespace reconciliation. Update the flag help, installer guidance, and operations documentation to describe the global cleanup boundary. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
492c9acdd0 |
fix(operator): ignore failed pods from finished jobs
Record terminal Job UIDs while scanning templates, then ignore only Failed pods owned by those Jobs. Failures from unfinished controllers and ownerless pods remain evidence because they may still be recreated. Also remove the stale constructor comment that still described raising a namespace ceiling. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2e3c40b125 |
fix(operator): ignore non-pod LimitRange scopes
Kubernetes accepts memory keys on PVC-scoped LimitRange items when a storage bound is also present, but those quantities cannot affect pod admission. Restrict the foreign-policy guard to Container and Pod items. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f6dcb69b61 |
refactor(operator): withhold instead of raising, and drop the grace period
The raise-and-grace machinery cannot work: LimitRanger writes the raised limit onto the very container that justified the raise, so at that container's next admission it carries a limit, the scan skips containers that have one, and the raise has erased its own evidence. The ceiling then drops on schedule and the workload is rejected permanently. Any such machinery approximates admission logic ahead of admission, and that is incomplete by construction: a LimitRange default is one number for a whole namespace and cannot mean "the configured limit, or this container's request, whichever is larger". Only per-pod conditional defaulting expresses that, which is a mutating webhook and not this. The justification for the machinery had also expired. It was designed when the default was 4Gi, where withholding was self-perpetuating for VPA-managed namespaces. At 32Gi nothing in the tree approaches the ceiling -- the largest static memory request is 2Gi -- and VPA sets requests and limits together, so those containers carry their own limit and were never blockers. So a namespace holding a container that requests more than the configured limit with no limit of its own is now left with no default at all, and anything written there earlier is withdrawn. It fails in the safe direction, which is the whole argument: a namespace with no memory.max is a namespace behaving as it did in every release before this, while a default below a request is a namespace whose workload cannot be admitted. It needs no stored state and no timer, because withholding is not a decision that has to survive between reconciles -- once the oversized request is gone the default is applied again on the next pass. Gone with it: the grace period, the clear-since annotation and its clamp, the injected clock, containerMemoryDefault, raisedCeiling, and fourteen tests. The scan stays, because deciding whether to apply or withhold is what it was always for. The foreign-LimitRange ownership guard, the disable path and the 32Gi default are untouched. The residual sharp edge is stated rather than machinery: a workload whose only evidence is a live Pod, built from a custom resource with no template this scan can read, can still be missed if the scan runs while that Pod is absent, and applying the default then rejects its recreation. At 32Gi that is unreachable in this tree. It becomes reachable if an operator lowers the knob below such a workload's request, which the flag help, values.yaml and the operations page now say plainly. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
d3df7c3d4d |
style(operator): gofmt stray blank lines in the package reconciler
Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
588ecbaeeb |
fix(operator): scan pods every reconcile, and key ownership on the label
The pod half of the scan was skipped once a namespace defaulted container memory at exactly the configured limit, on the argument that the shape the scan looks for could no longer be admitted there. That argument proves such a pod cannot come to exist. It does not prove one cannot be attempted, and a failed attempt leaves nothing behind: a workload built from a custom resource whose request grows above the limit after its namespace settled was rejected permanently, with no artefact anywhere that could justify raising the ceiling again. The premise was not sound even for existing pods, since LimitRanger reads LimitRanges through an informer and the shape is admissible in the window before the plugin sees the apply. The skip bought one namespaced List out of six and paid for it with a silent permanent-rejection mode, so it is gone along with memoryDefaultSettledAt and the two tests that pinned it. One of those tests carried the counterexample in its own fixture: a settled 4Gi LimitRange beside an 8Gi pod, asserted invisible. The coexistence guard identified its own object by name. The name is not reserved, so an administrator's LimitRange called cozystack-system-defaults was skipped by the guard and then overwritten by the forced apply, and because LimitRangeSpec.Limits is an atomic list that replaced every item in it, taking their max and cpu policy along. Ownership is now the managed-by label, which is what the delete path has always used. That leaves a same-named object with no label ambiguous: ours with the label stripped, or an administrator's. The two mistakes are not symmetric, so the ambiguous object is treated as foreign and left entirely alone rather than adopted. Re-stamping it with the label goes, and its test is replaced, because the same path reaches an administrator's object and adopting it means the disable path later deletes their policy. Declining to delete is no longer silent either. Also: containerMemoryDefault now takes the last matching item rather than the first, which is what LimitRanger applies, so a multi-item spec no longer reports a ceiling the cluster is not using. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c6342bf885 |
fix(operator): bring the flag help in line with the guard and removal scope
The --help text still carried two claims that values.yaml and the operations page had already narrowed, and it is the one surface an operator reads without opening the tree. Both were wrong in the direction that costs the reader something. "A namespace where another LimitRange already defaults or caps container memory is left alone" described the guard before it was widened, so an administrator holding a container memory min, a maxLimitRequestRatio or any Pod-scoped bound would read it, expect the LimitRange to be applied there anyway, and get a namespace with no memory.max for a reason the help says does not apply. "Removes any previously created one" is the universal removal claim, and removal is scoped to namespaces an active Package still targets. Two comments corrected while here. constrainsMemory said it switched on the item's own maps "rather than a hand-picked few", which contradicts the hand-picked list immediately below it; what is actually true is that a field added upstream has to be added there too, which is the reason it has its own table test. And memoryDefaultSettledAt claimed the count of raised namespaces falls to none once the oversized requests are gone, which stopped being true when the scan started keeping Failed pods: a carcass Kubernetes never collects holds both the raise and that namespace's per-reconcile pod List open. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
81c12bc953 |
fix(operator): withhold on any foreign memory policy, not two fields
The coexistence guard treated a namespace as already-governed only when another LimitRange carried a container memory default or max, and said so in its comment. That is not the set of fields that collide with what this reconciler writes. A container min above the default request rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi default against a 32Mi default request is a ratio of 1024:1 and no cap allows it. So the namespace-wide admission failure the guard exists to prevent stayed reachable through fields the guard did not read, and kube-system is where it is likeliest, this feature having deliberately brought it in scope. Enumerating the colliding fields does not fix it, because at the Pod scope the collision is not decidable here. Whether a per-container default of 32Gi breaches a pod-wide max depends on how many containers the pod has, and the pods in question do not exist yet: any arithmetic about them is a guess, and a wrong guess costs a namespace its entire pod admission. So the rule is ownership rather than arithmetic. If anything else in the namespace has an opinion about memory, at either scope and in any field, the namespace is not this reconciler's to default. Narrow still on the resource, because nothing here writes cpu or storage and a LimitRange bounding those must keep coexisting. This is the rule the tenant-namespace exclusion already rests on. The pod scan no longer skips Failed pods, only Succeeded. A pod that ran to completion is never recreated from its spec, but an eviction or a node crash leaves a Failed carcass holding the same oversized request that its controller will recreate, and that carcass is frequently the only surviving evidence the namespace needs a raised ceiling, since the shape the scan looks for cannot be admitted once the namespace has settled. Dropping it lowered the ceiling under a workload that was coming back, which is what the grace period exists to prevent. A grace stamp dated ahead of the clock is clamped to now. Unclamped it held the ceiling for the skew plus the grace period, silently, so an hour of skew bought ninety minutes instead of thirty. Clamping never shortens the wait, it only stops it being extended. The two flag defaults are named constants the flag registration and its test both read. The test previously duplicated the literal and still asserted 4Gi after the default moved to 32Gi, so it misinformed a reader about the shipped value and pinned nothing. The disable claim is narrowed to what the code guarantees. "Removes any previously created one" was universal, but the reconciler only runs for namespaces an active Package still targets, so a namespace whose Package was removed keeps its LimitRange. The flag help, values.yaml and the operations page now say that, and name the managed-by label for finding leftovers by hand. The test asserting a Pod-scoped memory max should be ignored is gone, since under the ownership rule a Pod-scoped memory bound withholds. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
52aa837821 |
fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout (#4584)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does The kubevirt-csi node plugin now unmounts its temporary NFS mount with a timeout, and retries with `umount -f` when the timeout runs out. For an NFS publish the plugin mounts the export root into an `nfs-init-*` temp dir to create `/data`. It unmounted that dir in a defer with a plain `umount`, which has no deadline. Before this change, if the NFS server was gone by then, the publish could hang on that `umount` forever and the temp mount stayed. The force-unmount helper in `k8s.io/mount-utils` doesn't work for this case. `UnmountWithForce` retries with `-f` only after its first `umount` returns. That `umount` forks the `umount.nfs` helper. When the timeout kills `umount`, the helper stays stuck on the dead server and keeps the output pipe open, and `CombinedOutput` waits for that pipe, so the `-f` retry never runs. The plugin now runs `umount` itself with `WaitDelay` set, so the pipe is closed a second after the kill, and then runs `umount -f` under its own timeout. A "not mounted" answer counts as done, same as in mount-utils. `umount2` detaches the mount before it blocks on tearing down the superblock, so the `-f` may find nothing left. Some umounts can't be killed at all, for example one stuck in uninterruptible sleep. So the whole cleanup also waits at most 30s, well inside the 2m deadline kubelet sets for NodePublishVolume (`csiTimeout` in `pkg/volume/csi/csi_plugin.go`). Once the budget runs out the publish goes on, and the unmount keeps running in the background and logs its result when it ends. The temp dir is removed by that background unmount once it succeeds, so an unmount that finishes late still cleans up after itself. A mounter other than the exec-based one keeps its plain `Unmount`, under the same budget. The temp mount now uses `nosharecache`. A forced unmount runs `nfs_umount_begin`, which kills every RPC in flight on the superblock, and NFS shares a superblock between mounts of the same export with the same options. Without this option, `-f` on the temp mount could fail I/O on the pod mount this same publish creates, and on other pods on the node that mount the same volume. The cost is a separate cache: pods that already have `/data` mounted may not see what the migration moves in until their attribute cache expires. The migration is a one-time backfill, so I'm fine with that. The leak only lives until the container restarts, because `/tmp` in the node plugin container has no hostPath and the mounts go away with the container's mount namespace. So this change stops leaked mounts from piling up inside one container. For the same reason a sweep of `nfs-init-*` dirs at startup would never find anything, so I did not add one. Two cases stay as they were. An `umount` that fails fast, for example with `target is busy`, gets no force retry, is logged, and the dir stays because `rmdir` refuses a mount point. And since the temp mount is a hard mount, a server that disappears before the unmount still hangs the mount and migration calls earlier in this path. That second case is #4581. The tests run the real exec-based mounter against a fake `umount` on `PATH`. The fake covers a plain unmount that succeeds, one that fails fast and gets no `-f`, and one that hangs while an orphaned child holds its output pipe. The last one has to end in `umount -f` inside the budget. A "not mounted" from `-f` counts as done, and any other `-f` failure is reported and keeps the dir. Fake mounters cover the background path: giving up on a stalled unmount, removing the dir after a late unmount, and logging the late result. Other tests check `nosharecache` on the temp mount and the temp dir after a failed mount. I haven't reproduced this against a real NFS server that went away, so it is not checked whether `umount -f` in the almalinux image gets as far as `umount2(MNT_FORCE)` there. The tests are in a new file, so this PR does not overlap with #4562. Closes #4561 ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> No UI change. ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(kubernetes): the kubevirt-csi node plugin no longer hangs an NFS volume publish forever when unmounting its temporary mount after the NFS server went away; the unmount now times out and is retried with force ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Temporary NFS mounts use isolated caching to reduce interference with pod mounts. * Cleanup waits up to 30 seconds for unmounting. If the limit is reached, the publish operation returns an error while unmounting continues in the background. * Temporary directories are removed after a successful unmount or a failed mount, and retained if unmounting fails. * When a standard unmount times out, cleanup retries with a forced unmount when available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bcfa50d7e9 |
test(e2e): install only the packages the selected suites need
The E2E job installs the whole platform on a fresh cluster before it knows which suites will run, so every PR pays install time and flake surface for packages its suites never use. Select the suites first and pass the complement computed by select-install.sh to the platform Package as disabledPackages. The full-e2e label and an empty selection keep the full install, and the nightly and tag lanes pass no list, so they are unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
974aaa0bac |
test(hack): regenerate the selective install fixture
Two optional Cluster API providers, Proxmox and in-cluster IPAM, were added to the platform sources after the fixture was generated. The drift test compares the fixture with the current disable list, so it failed until they were added. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f46a9797b8 |
fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest of the platform relies on: multus for secondary networks, the cozystack and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no PackageSource edge reaches them, and adding one would also change reverse test selection. The remaining packages stay uninstalled on purpose, also when test selection answers with every suite: nothing in the run looks at them, so installing them only adds a readiness wait. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f863967ca5 |
fix(hack): retain the webhook used by gateway tests
The gateway namespace-churn regression requires a registered and ready pod admission webhook. Its availability cannot be tested when selective installation removes the package that registers it. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
59555c895e |
fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS release creates COSI resources even when no backup suite is selected. Without those controllers, a fresh reduced install cannot become ready. The Kubernetes suites create worker pools through KubernetesNodes, and MongoDB renders an operator-owned CR. Selecting only the cluster API or application chart drops resources those suites require. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
064231df47 |
fix(hack): support BWK awk in install graph validation
BWK awk rejects literal newlines in command-line assignments. Every non-empty selection and graph validation therefore aborts before it can check dependencies when this awk implementation is selected. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
361893e6f4 |
fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the application graph does not require. A graph-only closure therefore omits the APIs those suites need on a fresh install. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7fda27385e |
fix(hack): validate selective install graphs before output
Dangling references and cycles make the inventory unsuitable for a fresh install even when the selected suite does not reach them. Accepting that inventory would hide a broken platform graph. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3766579e89 |
test(hack): tighten selective install assertions
PackageSource names contain dots, so regular-expression matching can accept a different source. Selecting a suite also does not prove its install closure retains every API that suite uses. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
610578be16 |
test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a reduced install without the APIs its tests exercise. Graph-only checks cannot establish that the chart preserves the requested package set. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
4b13893911 |
feat(hack): generate installable selective package sets
A reduced E2E install needs the install harness prerequisites and every owner of a selected suite, including requirements absent from the application dependency graph. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b6c33926dc |
fix(dashboard): re-arm the 401 latch once the session recovers
unauthorizedHandled was set once per client and never cleared, so it collapsed every later 401 too, not just the concurrent ones it was meant to. Harmless for the shipped default handler, which navigates away and takes the client with it, but the onUnauthorized hook exists precisely so a caller can re-auth in place, and such a handler would fire exactly once for the client's lifetime and then go silent. Clear the latch on any 2xx from either fetch site: a success means the session is alive again, so the next 401 is a new episode. The storm guard still holds, since concurrent 401s have no success between them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7e5f9ab152 |
fix(dashboard): re-auth via /oauth2/sign_in so token-proxy mode works
The dashboard gatekeeper runs one of two mutually exclusive proxies, chosen by the cluster's oidc-enabled flag (templates/gatekeeper.yaml). /oauth2/start exists only in oauth2-proxy. The token-proxy branch registers only sign_in, sign_out and userinfo under the /oauth2 prefix, so there /oauth2/start falls through to the catch-all reverse-proxy handler, which does not re-verify the token. With a present but expired cookie it proxies the request to the console upstream, the SPA reloads, gets a 401 again and loops. /oauth2/sign_in is the one entry point both modes implement, so the default 401 handler goes there. oauth2-proxy's sign_in hands straight off to OAuthStart because the gatekeeper passes --skip-provider-button, and it resolves rd through the same app-director, so the OIDC path behaves as it does through /oauth2/start. token-proxy serves its token form there. It does not honour rd yet, so after sign-in the user lands on / instead of looping. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
badbec63fe |
test(dashboard): cover the 401 -> oauth2-proxy redirect path
The 401 handling is what brings an expired session back to sign-in,
and a regression in it stays invisible until a cookie expires. The
test drives K8sClient through the vi.stubGlobal("fetch", ...) harness
the sibling k8s-client tests use. It checks that the hook fires once
per 401 from both the request and the watch fetch sites, once across
concurrent 401s, never on other responses, and that the default
handler passes the current location as a relative rd, which
oauth2-proxy's open-redirect check accepts.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
8bf78d3bfe |
fix(dashboard): redirect to oauth2-proxy on 401 instead of failing silently
When the kc-access cookie expires, k8s API calls return 401 and the SPA stays on a stale page until the user clears the cache by hand. K8sClient now treats 401 as an auth boundary and makes one top-level navigation into the oauth2-proxy sign-in flow, passing the current location as rd so the user comes back to the same page. The handler can be replaced through K8sClientConfig.onUnauthorized and does nothing outside the browser. The unauthorizedHandled latch keeps concurrent 401s from redirecting more than once. Re-applied from cozystack/cozystack-ui#34 onto the in-tree console after the UI was vendored (#2963), since the source repo is being archived. Related: #2788. Co-authored-by: IvanHunters <49371933+IvanHunters@users.noreply.github.com> Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3d901ca9b4 |
test(clickhouse): add backup API render assertions
The port the backup strategy dials, the two secretKeyRefs the clickhouse-backup sidecar authenticates with, and the auth Secret that backs them are all rendered by this chart. A helm-unittest suite catches a regression in them on every pull request, while the e2e assertions on the same wiring run only when test-impact analysis selects the ClickHouse suite. The e2e assertions stay, since they check the operator-created Pod and the admitted Secret. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c35cc848f0 |
docs(agents): widen overview.md routing and condense activation rules (#3373)
## What this PR does `AGENTS.md` routes an agent to a documentation file based on how a request is phrased. Two things were wrong with that routing. The `overview.md` route listed only structural triggers — "where should I put X", "what's the convention for Y", "how is the project organized". Go, Helm chart and build/test work matched none of them, so an agent asked to add a field to a CRD, fix a controller, add a value to a chart or regenerate a values schema reached no documentation at all — even though `overview.md` already covers every one of those under `Go Code`, `Helm Charts`, `Values schema generation`, `Build and Development Commands` and `Package Structure`. This widens the trigger phrases to the work agents actually get asked to do and names the sections to land in. No new documentation is added here: the content already existed, it was just unreachable from the router. The activation rule list spent seven bullets on three rules. Bullets 1 and 6 stated the same scope constraint, once positively and once negatively; bullets 3, 4 and 5 each said "follow the documentation exactly" in different words. Condensed to three bullets, with no instruction dropped. ### Screenshots Not a UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff rather than the title. The diff touches exactly one file, `AGENTS.md`, and no trigger in the map matches it: nothing under `packages/apps/`, `packages/extra/` or `packages/core/`, nothing under `hack/`, no platform variant, no annotation or label value, no telemetry metric, and no change to node or network requirements. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated guidance to direct questions about Go code, Helm charts, build and test commands, and project structure to the relevant sections of the project overview. * Clarified that process documentation should be loaded before starting tasks within its scope, followed without skipping steps, and used alongside the overview as a standing reference. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
df889a6419 |
docs(platform): state when the ceiling drop actually lands
Two imprecisions in the page written alongside the grace period. It said a lowered limit reaches previously-raised namespaces "within" the grace period, which is backwards: the grace period is a floor, not a ceiling, and since reconciles are event-driven the drop lands on the first one after it elapses. And the withdraw path was described only as leaving the namespace alone, without saying that it hands the namespace to the administrator's policy in full, including any admission rejection that policy implies -- which is theirs, and would have happened without Cozystack, but is worth naming where an operator is being told the operator stopped writing. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
30ee57f892 |
fix(operator): route every downward ceiling move through the grace period
The grace period guarded only the branch where the scan came back empty, which left the same defect standing in the branch where it finds something smaller than the ceiling already in place. A namespace raised to 16Gi for a pod visible only as a live Pod, with a second oversized pod at 8Gi beside it, loses the larger pod to a node drain. The scan still finds the 8Gi one, so the empty-scan guard never engages, and the ceiling was written straight down to 8Gi -- rejecting the larger pod's recreation exactly as writing the configured limit would have. A blocker smaller than the standing ceiling is not evidence that a larger one is gone. So the rule is now about direction rather than about why: the ceiling the evidence justifies is computed first, and any move that lowers it goes through the grace period, whichever of the three reasons prompted it -- an empty scan, a smaller blocker, or the configured limit itself being lowered beneath a standing ceiling. The eventual drop lands on what the scan justifies rather than on the configured limit, so a smaller blocker that is genuinely still there keeps the ceiling it needs. The raise is also no longer logged when it changes nothing. A namespace holding a permanently oversized pod re-justifies the same ceiling on every reconcile, and announcing that put a line in the log for every such namespace on every HelmRelease status change. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |