mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 12:13:55 +09:00
f76dedf9ca0f60e07db0efbc52eba5af889bd08f
4483
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f76dedf9ca |
fix(kubernetes): keep NFS publish from hanging on a dead server
The kubevirt-csi node plugin mounts the NFS export root into a temp dir to create /data and migrate old files. That mount was hard, so with the server gone the mkdir, readdir and rename calls on it never returned. kubelet gives up after 2m and retries, and every retry blocked the same way, piling up goroutines and nfs-init-* dirs. Mount the temp root soft, so its RPCs fail once the connection to the server is declared dead instead of blocking forever; it only does metadata work that a retried publish redoes. retry=0 keeps mount.nfs from retrying a failed temp mount for 2m. timeo and retrans keep their defaults, because the first mount of a server sets them for the connection the pods' mounts of it share. The volume mount the pod gets stays hard, since a soft mount can report a failed write the server applied. Over NFSv4 that soft failure takes minutes, and a stat of a target that is already hard-mounted can block for good, so a publish for a target path that already has one running now returns Aborted instead of starting another. A soft stat can fail with EIO on a file /data already holds, and a rename issued once the server is back would replace it, so the migration now fails the publish on any stat error other than a missing destination. It checks the destination with lstat, so a user's symlink in /data that loops or dangles counts as present instead of failing every publish or being replaced by the rename. The skip it makes for a destination already in /data leaves the root entry where the pod cannot see it, so it is now logged as a warning. /data is created with a plain mkdir, since with the server gone MkdirAll fails its stat on the mount point and then reports the local temp dir as already existing instead of the EIO. The temp mount errors now name the volume, like the migration errors do. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
0a2bb5ef98 |
fix(cert-manager): enable the Gateway API solver in the host cert-manager (#4039)
## What this PR does With the platform Gateway enabled, the ClusterIssuers in `cert-manager-issuers` and the per-tenant Issuers rendered for `TenantGateway` carry an `http01.gatewayHTTPRoute` solver, but the controller refuses it: the vendored chart never turns Gateway API support on, so every HTTP-01 challenge behind the Gateway fails with `gateway api is not enabled` and no certificate is issued. This passes a `ControllerConfiguration` with `enableGatewayAPI: true` to the host cert-manager through the component values of its platform Package. The host ships the Gateway API CRDs on every variant and orders cert-manager after them, so the flag is safe there. It is deliberately not a default of the `system/cert-manager` chart. The same chart is installed into tenant Kubernetes clusters, where the CRDs are an opt-in addon, and cert-manager v1.20 exits at startup when Gateway API support is on and the CRDs are missing. The tenant HelmRelease in `packages/apps/kubernetes` already passes the same `ControllerConfiguration` when its `gatewayAPI` addon is enabled, so both clusters now opt in from the layer that knows whether the CRDs are there, and the chart itself stays neutral. The new tests pin both halves: the host Package carries the flag and the preconditions that make it safe, and the chart wires `config` through to the controller while shipping none by default. Closes #3725 ### Screenshots Not applicable, no UI changes. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(cert-manager): enable the Gateway API solver in the host cert-manager, so ACME HTTP-01 challenges succeed when the platform Gateway is enabled ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Supported system bundles can now use cert-manager’s Gateway API HTTPRoute solver to issue certificates. - Hosted deployments include the required Gateway API custom resource definitions, installed before cert-manager. - Gateway API solver support remains opt-in in environments that do not provide the required definitions, avoiding unintended changes to certificate management there. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9b0e25b1a0 |
fix(etcd-operator): gate VMServiceScrape on VictoriaMetrics operator CRD (#3960)
## What this PR does `packages/system/etcd-operator/templates/metrics.yaml` rendered a `VMServiceScrape` (`operator.victoriametrics.com/v1beta1`) unconditionally, so the `cozy-etcd-operator/etcd-operator` HelmRelease fails to install on any cluster that does not have the VictoriaMetrics operator CRDs, with `no matches for kind "VMServiceScrape" in version "operator.victoriametrics.com/v1beta1"`. That made the VictoriaMetrics operator a hard dependency of etcd-operator, which it should not be. This wraps only the `VMServiceScrape` document in a `.Capabilities.APIVersions.Has "operator.victoriametrics.com/v1beta1"` gate, following the precedent already established in `packages/system/linstor/templates/podscrape.yaml`. The `etcd-operator-metrics` Service keeps rendering unconditionally, so any scraper — a `VMServiceScrape` when the CRD is present, or something else entirely — can still target the `metrics` port. Note: this is a Cozystack-packaging-only issue. The upstream etcd-operator repo's own chart is unaffected — it already gates its scrape object. Added `packages/system/etcd-operator/tests/metrics_test.yaml` (`helm unittest`) pinning the new behavior: the Service always renders, and the `VMServiceScrape` renders only when the capability is injected. Kept this minimal — a pure capabilities gate with no new values key, matching linstor's style, rather than adding a values toggle. Follow-up (out of scope for this PR): `packages/extra/etcd/templates/podscrape.yaml` (a `VMPodScrape`) is unconditional the same way and would hit the same install failure on a cluster without the VictoriaMetrics operator. `packages/system/monitoring-agents` also ships several unconditional `VMServiceScrape`/`VMPodScrape` objects, though that package may already assume the VictoriaMetrics operator is present as part of the monitoring stack it belongs to. ### Screenshots Not applicable — no UI change. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note fix(etcd-operator): stop requiring the VictoriaMetrics operator to install etcd-operator; the VMServiceScrape now only renders when the operator's CRD is present ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Metrics collection now works without the VictoriaMetrics operator API, while scrape configuration is rendered when that API is available. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
6192d5814b |
feat(local-storage): add opt-in OpenEBS LocalPV hostpath package (#3266)
## What this PR does Adds `packages/system/local-storage`, an optional package that vendors OpenEBS Dynamic LocalPV (chart 4.5.1) and registers it as the `cozystack.local-storage` PackageSource. A cluster without the DRBD and ZFS kernel modules can't run LINSTOR, so it has no dynamic PV provisioning, hostpath volumes cover that. They also fit deliberately node-local workloads. - StorageClass is `cozystack-local-hostpath`, `WaitForFirstConsumer`, `Delete` reclaim. It is not the default and doesn't take any existing class name. - Opt-in: a system bundle can list the package in `bundles.enabledPackages`, no built-in variant enables it. Alone it installs with `cozypkg add cozystack.local-storage`. - Vendored RBAC is cut down by `patches/rbac.diff` (reapplied by `make update`) to what volume reconciliation calls. Helper Pods, leader election Leases and analytics state stay in the install namespace, helpers are still privileged because they work on host directories. - Images stay on docker.io, pinned by index digest with `make update-image-pins`, nothing is rebuilt or mirrored. - `local-storage` scope is mapped to `area/storage` in `pr-labeler.yaml`, without it the title lands in `area/uncategorized` on every run. Tested with helm-unittest (StorageClass, PackageSource registration) and `tests/rbac_test.py` (effective permissions across the bindings). Not tried on a live cluster yet. Nothing installs the package by default, the container cluster e2e lane that will use it is a follow-up ([#3238](https://github.com/cozystack/cozystack/issues/3238)). ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/726 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Website follow-up is an issue, the docs there describe a package that isn't merged yet. The diff adds a platform component and touches no platform values, ApplicationDefinition, tooling or `hack/` path, so nothing else in the trigger map matches. ### Release note ```release-note feat(local-storage): add opt-in cozystack.local-storage package that installs OpenEBS LocalPV hostpath and provides the cozystack-local-hostpath StorageClass, it is not enabled by any built-in variant and is not the default StorageClass. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added optional OpenEBS LocalPV Hostpath storage for provisioning persistent volumes on node-local storage. * Added a configurable `cozy-local-hostpath` StorageClass, disabled by default as the cluster’s default class. * Added controls for storage class creation, reclaim policy, hostpath quotas, and node deployment. * **Documentation** * Added guidance on installation, configuration, storage behavior, and uninstall considerations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
76b9e55102 |
fix(cert-manager): enable the Gateway API solver in the controller
With the platform Gateway enabled, the ClusterIssuers in cert-manager-issuers and the per-tenant Issuers rendered for TenantGateway carry an http01.gatewayHTTPRoute solver, but the controller only honours that solver when Gateway API support is switched on, and the vendored chart leaves it off. Every HTTP-01 challenge behind the Gateway then fails with "gateway api is not enabled" and no ACME certificate is issued. Pass a ControllerConfiguration with enableGatewayAPI to the host cert-manager through the component values of its platform Package. The host always ships the Gateway API CRDs and orders cert-manager after them, so the flag is safe on every variant. It is not a default of the system/cert-manager chart: the same chart is installed into tenant clusters, where the CRDs are an opt-in addon and a controller started with Gateway API support and no CRDs exits at startup. The tenant side keeps opting in on its own. Signed-off-by: Aleksei Sviridkin <f@lex.la> Assisted-by: LLM |
||
|
|
d61fed46cc |
feat(local-storage): add opt-in OpenEBS LocalPV hostpath package
LINSTOR needs the DRBD and ZFS kernel modules, so a cluster without them has no dynamic PV provisioning. OpenEBS LocalPV hostpath needs no kernel modules and also serves deliberately node-local workloads. The package is opt-in so it cannot change an existing cluster. Its class is named cozystack-local-hostpath so it cannot collide with an existing class name, and it is not the default so PVCs that omit storageClassName keep their current binding. The vendored RBAC is patched down to what volume reconciliation calls; helper Pods, Leases and analytics state stay in the install namespace. Images stay on docker.io, pinned by index digest. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6addef4d3c |
fix(kubevirt): route the export proxy to the Service KubeVirt creates (#4637)
## What this PR does The VM export routes point at a Service that does not exist, and the tenant network policy would block the Ingress even if it did. KubeVirt creates the export proxy Service under the name `virt-exportproxy` (`VirtExportProxyName` in `pkg/virt-operator/resource/generate/components`, v1.9.0). The Ingress and the TLSRoute that publish `vm-exportproxy.<host>` name a `vm-exportproxy` Service, and nothing creates one. The backend was `virt-exportproxy` when the Ingress was added in |
||
|
|
8c1a78dc9b |
feat(operator): default system namespace memory via LimitRange (#3494)
## What this PR does
System components now get a default container memory limit, so their pod
cgroups carry a `memory.max` and drop out of the Talos userspace OOM
handler's victim set. `cozystack-operator` maintains a LimitRange named
`cozystack-system-defaults` in every system namespace it reconciles.
The Talos handler (v1.12+) scores cgroups with `memory_max.hasValue() ?
0.0 : {Besteffort: 1.0, Burstable: 0.5, ...}[class] * memory_current`
and drops everything that scores zero. Only a memory limit sets
`memory.max`, so only a limit takes a pod out of the candidate set. A
request just moves a pod from BestEffort to Burstable, which under the
default `strictCgroupClassOrdering: true` still means killed second.
`PriorityClass` does nothing here, because the handler reads cgroupfs
and never talks to the apiserver. Victim selection is also unrelated to
the trigger, so a tenant workload thrashing against its own limit can
drive node memory pressure while a system component with no limit is
what gets killed.
An audit of a running cluster found around 70 workloads across ~30
packages with at least one container missing a memory limit. Per-chart
resources would not cover that set, and would not keep covering it as
packages are added. So the LimitRange is written by
`reconcileNamespaces`, which already creates these namespaces and
already tells system from tenant. It also covers init containers, which
#3495 can't reach: one of them without a limit leaves the pod cgroup
without `memory.max`, whatever the main containers declare.
Tenant namespaces are skipped. Where the tenant chart's own
`tenant-range-limits` exists (only when the operator configured
`resourceQuotas`), a second LimitRange next to it would leave the result
to the order LimitRanger iterates them. Where it doesn't exist, a tenant
namespace is not the place for the platform to impose a default.
### Why the default is 32Gi
The handler grants immunity for `memory.max` existing at all, whatever
its value. So the ceiling doesn't need to be fitted, and 32Gi is high
enough that nothing plausibly reaches it. A ceiling low enough to bind
would turn a rare pressure-driven kill into a deterministic one for any
component whose real usage nobody measured. This repo already has one
such case, recorded in the `vertical-pod-autoscaler` values, where a
110Mi limit OOM-killed the updater and failed HelmReleases with it.
Scheduling is unaffected, since pods are scheduled against requests and
the paired default request is 32Mi. An operator who wants the ceiling to
bind can lower it, per component in that component's chart or fleet-wide
with the knob, once real usage is measured.
### When the default is withheld
A default below a container's own request makes that pod impossible to
admit, so the operator checks before it applies. It scans Deployments,
StatefulSets, DaemonSets, CronJobs, Jobs and Pods in the namespace, init
containers included, for a container that requests more than the
configured limit and declares no limit of its own. If it finds one, it
withdraws the managed LimitRange and records a hold in the
`operator.cozystack.io/system-memory-state` annotation on the Namespace.
The hold does not clear by itself when the blocker disappears, because
an empty scan looks the same as a pod between deletion and recreation.
Lowering a previously applied limit also creates a hold, even with no
visible blocker, because LimitRanger writes the defaulted limit onto
admitted pods and the scan can no longer tell which containers declare
one. A cluster administrator releases the hold by copying the
`<hold>:<target>` token from the state into the
`operator.cozystack.io/system-memory-ack` annotation. The token is used
once and only for that hold and target. If a blocker is still visible,
the token is rejected and the hold stays. Package reconciles are
scheduled at least every five minutes, so an acknowledgement takes
effect on a quiet cluster too. The reason and the token also appear in
the operator log under `withholding the default container memory limit`.
### Namespace another LimitRange already governs
If a system namespace already has a LimitRange that says anything about
memory, at the Container or the Pod scope, the operator writes nothing
there and withdraws what it wrote before. A second default leaves the
effective value to iteration order. A foreign `max` below this default,
a `min` above the default request or a `maxLimitRequestRatio` rejects
pods outright, which takes out a whole namespace. At the Pod scope the
collision can't be decided in advance, because it depends on how many
containers a future pod has. A LimitRange that bounds only cpu or
storage coexists normally. A same-named LimitRange without the
operator's managed-by label is treated as foreign and left alone.
This matters most in `kube-system`, which is in scope because
`cozystack-scheduler` installs there, and where a distribution's or an
administrator's own LimitRange is likeliest to sit.
### Disabling
Setting `cozystackOperator.systemNamespaceMemoryLimit` to `0` disables
the feature. On the next reconcile the operator removes every LimitRange
it owns cluster-wide, found by the managed-by label. While the feature
is on, it also removes its LimitRange from namespaces no active Package
targets any more. The safety state on the Namespace stays in both cases.
### Docs
`docs/operations/system-memory-limits.md` landed in #3496 ahead of this
code, so main describes a `4Gi` default and installer values that do
nothing yet. The page is corrected to the 32Gi default and now describes
withholding, the acknowledgement, and the foreign LimitRange rule.
### Not in this PR
linstor policy. #3824 extends the controller's existing exemption from
limits to the satellite, and a linstor limit here would contradict it,
so linstor stays with that PR.
The scan reads six workload kinds through `APIReader`, uncached, on
every reconcile of every system namespace. That cost is tracked in
#3977, since the system namespace set isn't known when the cache is
built and the right fix depends on a reconcile-rate measurement nobody
has taken.
### Verification
`go vet` and `go test` pass for `cmd/cozystack-operator` and
`internal/operator`, and `helm unittest` passes on the installer chart
for the knobs unset, set, and the limit at `0`. The reconciler tests
cover withholding on an oversized request, the hold surviving the
blocker's removal, the hold on a lowered limit, acknowledgement and
token rejection, every memory field of a foreign LimitRange at both
scopes, ownership by label, the disabled sweep, and a deleting Package
not creating policy.
The withhold design has not been run on a live cluster yet. An earlier
design of this PR was, and that run does not carry over.
### Screenshots
Not a UI change.
### Downstream repositories
I walked the trigger map against the diff.
`packages/core/installer/values.yaml` gains two keys, but the
`ansible-cozystack` trigger is for a required key or a rename of one the
role passes, and these are new, optional and default to `""`, so the
role is unaffected. Nothing else in the map is reached by the Go
changes, the operator template or the page.
- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
```release-note
feat(operator): system namespaces now get a default container memory limit through a LimitRange maintained by cozystack-operator, which keeps system components out of the Talos userspace OOM handler's victim set. Tune it with cozystackOperator.systemNamespaceMemoryLimit (default 32Gi) and cozystackOperator.systemNamespaceMemoryRequest (default 32Mi); set the limit to 0 to disable it. The default is far above any real working set on purpose, because the handler grants immunity for a memory.max existing at all, not for its value. A namespace with a container that requests more than the limit and declares no limit of its own gets no default, and stays without one until a cluster administrator acknowledges the hold recorded on the Namespace (see docs/operations/system-memory-limits.md); lowering a previously applied limit needs the same acknowledgement. A namespace where another LimitRange already sets anything about memory is left alone. A container that declares its own memory limit is never changed.
```
|
||
|
|
84e8fcd477 |
fix(dashboard): re-authenticate on 401 instead of failing silently (#3115)
## What this PR does When the dashboard session expires, the console now sends the user back to sign-in instead of sitting on a stale page. This re-applies cozystack/cozystack-ui#34 by @IvanHunters onto the in-tree console, since the UI was vendored into the monorepo (#2963) and the standalone repo is being archived. It is part of the cozystack-ui retirement tracked in #3097. When the `kc-access` cookie expires, k8s API calls return 401. `K8sClient` now treats a 401 as an auth boundary and makes one top-level navigation to `/oauth2/sign_in?rd=<current-url>`. That path is the sign-in entry both gatekeeper modes serve. With OIDC enabled, oauth2-proxy goes straight to the provider (`--skip-provider-button`) and returns the user to `rd`. Without OIDC, token-proxy renders its token form there, but it does not honour `rd` yet, so the user lands on `/` after sign-in. The handler can be replaced through `K8sClientConfig.onUnauthorized` and does nothing outside the browser. A latch keeps concurrent 401s from redirecting more than once, and any 2xx from either fetch site re-arms it, so a handler that re-authenticates in place still sees the next expiry. Related: #2788. ### Verification `client.unauthorized.test.ts` covers the hook on the request and watch paths, the latch across concurrent 401s, the re-arm after a success, and the default handler's target and relative `rd`. `pnpm typecheck` and the console test suite pass. Original PR: cozystack/cozystack-ui#34, credited via `Co-authored-by`. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(dashboard): on a 401 (expired session) the console now sends the user to sign in again instead of silently showing a stale page. With OIDC enabled the user returns to the page they were on; without OIDC they land on the start page after entering a token. ``` |
||
|
|
3cce629535 |
feat(redis,postgres): certify the per-engine external hostname (#4627)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does Redis and postgres server certificates now carry the per-engine external name `<release>.<engine>.<apex>` (for example `redis-foo.redis.example.org`) next to the flat `<release>.<apex>` they already had. Both names appear under the same condition as before: TLS on and `external: true`, with a tenant apex present. A TLS-passthrough listener on the tenant Gateway forwards the handshake untouched, so the certificate the client checks is the engine's own. The design in #2815 routes databases by SNI under `*.redis.<apex>` and `*.postgres.<apex>`, and without the per-engine name in the SAN list a verifying client would fail the hostname check. Issuing both names now means a later move between the two hostname schemes reissues no certificate and keeps clients on the same trust anchor. This is the certificate step of #2815. Nothing renders a route yet. The SAN list changes, so an existing external TLS release gets its server certificate reissued once on upgrade, from the same CA. For redis the operator then rolls the Redis and Sentinel pods one at a time, the same way it handles a renewal. The READMEs say who can reach the new name through a passthrough listener. For postgres that is only clients using direct TLS negotiation (server 17+, `sslnegotiation=direct`), because by default a plaintext SSLRequest comes before the handshake. `redis-cli` needs `--sni`. MongoDB is not in this PR. PSMDB issues the certificate itself and its CRD has no field for extra SANs, so adding the mongos name means the chart taking over the certificate. That needs its own change. Covered by helm-unittest: the name is present with TLS and external on, and absent with external off, with TLS off, and in a tenant with no apex. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note feat(redis,postgres): server certificates of external redis and postgres releases also carry the `<release>.<engine>.<apex>` name that the tenant Gateway's TLS-passthrough listeners are designed to route by. Existing external TLS releases get their server certificate reissued once on upgrade, from the same CA. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL TLS certificates now cover both the tenant hostname and the per-engine `postgres` hostname for external connections. * Redis TLS certificates now also cover the per-engine `redis` hostname when external services are enabled. * **Documentation** * Clarified connection requirements for per-engine hostnames: PostgreSQL routing requires direct TLS negotiation, PostgreSQL 17 or later, and a compatible client; Redis clients must specify SNI. * Clarified that the hostnames may need to be mapped manually to the LoadBalancer address. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
b1c84e9c06 |
feat(platform, tests): select E2E packages and declare controller dependencies (#4022)
## What this PR does E2E installs the whole platform on a fresh cluster for every PR, whatever the PR touches, so install time and flake surface go to packages the selected suites never use. This PR makes the E2E install take only what the selected suites need. `hack/select-install.sh --disabled <suites>` generates `bundles.disabledPackages` for fresh E2E installs using an ordinary platform variant. It keeps suite owners, runtime requirements and the install baseline, then follows their forward `PackageSource` dependencies. The baseline follows the unconditional waits in `hack/e2e-install-cozystack.bats` and stays seeded directly even where the graph reaches the same packages. It also keeps platform services that no suite owns but the rest of the platform uses: `multus`, `cozystack-scheduler`, `linstor-scheduler`, `flux-plunger`, `cozy-proxy` and `ouroboros` (the Kubernetes suites assert its hairpin NAT). Backup suites also keep `backupstrategy-controller` and its dependencies, and the gateway suite keeps `kubeovn-webhook`. The E2E job now selects the suites before it installs, and passes the complement to the platform Package as `disabledPackages`. The `full-e2e` label and an empty selection keep the full install, and the nightly and tag lanes pass no list. The platform sources also change: `mongodb-application` now depends on `mongodb-operator`, and `seaweedfs-application` on `objectstorage-controller`. These dependencies apply to regular platform installs and upgrades too. The new `mongodb-operator` edge also reaches reverse test selection: a change under `packages/system/mongodb-operator` now selects the `mongodb` suite instead of falling back to the full suite. Backup APIs and the gateway webhook remain suite-specific install requirements, so they add no edges. The disable list governs fresh installs because existing Packages carry `helm.sh/resource-policy: keep`. Optional packages still need `bundles.enabledPackages`. Packages that no suite owns and the baseline does not keep stay uninstalled on purpose, also when test selection answers with every suite (monitoring exporters like `goldpinger` and `kubeovn-plunger`, apps without a suite, provider-specific operators). Nothing in the run looks at them, so installing them only adds a readiness wait, and a change to such a package gets no install check from this gate. ### Measured One run per variant on a 32 vCPU / 128 GB VM, container lane, the CI artifacts of this PR, `postgres` suite selected, 55 packages disabled: - HelmReleases 111 to 77, pods 219 to 197 - pod requests 14.8 CPU / 37.7 GiB to 13.5 CPU / 35.7 GiB - host memory peak during install 33.4 GB to 30.2 GB - `install-cozystack` 26m33s in both runs - `postgres` suite 25m50s to 25m17s, passes on the reduced install Install time did not change. The removed packages are mostly operators and apps without instances, and what the install waits on (SeaweedFS, LINSTOR, CDI, the Cluster API providers, Keycloak) stays in the baseline. With one run each, differences of a minute or two are noise. The selection, the install closure and the workflow wiring are covered by bats tests (`hack/select-install_test.bats`, `hack/select-e2e_test.bats`, `hack/platform-packages_test.bats`) and by a helm-unittest suite in the platform chart. ### Screenshots Not a UI change. ### Downstream repositories The new dependencies ship in the platform chart. No coordinated changes are needed in downstream repositories. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(platform, tests): E2E installs only the packages the selected suites need. MongoDB now depends on its operator, and SeaweedFS on the object storage controller. ``` |
||
|
|
5486d4ffcf |
fix(networking): harden webhook policy assumptions (#4058)
## What this PR does This is follow-up for [#3997](https://github.com/cozystack/cozystack/pull/3997). It reads ingress-nginx admission port from effective subchart values and refuses empty value, and fixes `kubeProxyReplacement=false` status because cozystack has no supported Service translator in that state. Gateway test now requires one real ClusterIP and keeps same ready Pod UID, IP and port for whole probe, so stale EndpointSlice, replaced Pod or wrong Service can't report deny. Policy selectors and two-ended deny from [#3997](https://github.com/cozystack/cozystack/pull/3997) stay unchanged, and this does not claim root cause for [#3983](https://github.com/cozystack/cozystack/issues/3983). ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(networking): use effective ingress admission port, report unsupported Cilium service routing correctly, and reject stale backend evidence in gateway tests ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Bug Fixes** - Admission webhook connectivity checks now validate the configured Service and its exact backend endpoint throughout probing, improving failure detection. - Admission webhook policies now fail closed when the webhook port is not explicitly configured. - **Documentation** - Clarified that supported service routing requires `kubeProxyReplacement: true`. - Updated guidance to explain that disabling this setting or webhook policies does not provide a supported fallback. - **Tests** - Added coverage for missing webhook port configuration and strengthened networking stability checks. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ac4ab1c4a6 |
chore: drop private cluster and host names from the tree (#4461)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does I remove internal cluster names, a real hostname with its IP addresses, internal mirror hostnames and an internal product name from comments, an example file, test fixtures and a dead commented-out config block. Where a stand-in is needed I use neutral test-cluster wording, RFC 2606 domains or RFC 5737 documentation addresses. I also replace two real Keycloak usernames and a reviewer's name in code comments with placeholders, or drop them. Rendered output does not change. The test fixture hostnames change on the input and the expected side together, and the unit tests of every touched package still pass. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Updated backup, OIDC, autoscaling, and platform references to describe test environments without naming specific development clusters. * Replaced development-specific endpoints in the platform example with `example.org` and documentation-only IP addresses. * Updated OIDC username examples. * **Tests** * Updated fixture hostnames and expected URLs to use `example.org`. * **Chores** * Removed commented-out registry mirror entries; the active mirror configuration is unchanged. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
bc9a594e47 |
test(clickhouse): add backup API render assertions (#4011)
## What this PR does This adds unit tests for how the ClickHouse chart wires up the backup API that the default backup strategy calls. The Pod of the `cozy-default-altinity` strategy reaches the clickhouse-backup sidecar on the named port `ch-backup-api`. It authenticates with the credentials from the `<release>-backup-api-auth` Secret. The new suite is `packages/apps/clickhouse/tests/backup_api_contract_test.yaml`. It checks that: - the sidecar exposes `ch-backup-api` on 7171/TCP; - `API_USERNAME` and `API_PASSWORD` come from the `username` and `password` keys of `<release>-backup-api-auth`; - that Secret renders with `username: backup` and a non-empty password; - no auth Secret renders when backups are disabled. All of this is rendered by the chart, so the unit tests catch a regression on every pull request. The e2e assertions on the same wiring run only when test-impact analysis selects the ClickHouse suite. They stay as they are in `hack/e2e-chainsaw/clickhouse`, because they check the operator-created Pod and the admitted Secret, and rendering can't do that. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` |
||
|
|
a1416655f4 |
fix(tenant): allow tenant egress to the KubeVirt export proxy
The vm-exportproxy Ingress is served by the ingress controller in tenant-root, and tenant egress is default-deny. Nothing allowed traffic to virt-exportproxy in cozy-kubevirt, so the Ingress stayed unreachable even with the right backend Service. The new policy renders wherever allow-to-cdi-upload-proxy does, but narrows the destination to the export proxy pods and their container port. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
c069a191fc |
fix(monitoring-agents): give kube-state-metrics a sized memory limit (#3846)
`kube-state-metrics` ran BestEffort with no resources at all: the
subchart ships `resources: {}` and our wrapper never set any. So it put
no `memory.max` on its pod cgroup, and the Talos userspace OOM handler
only ever selects cgroups that lack one, which left it permanently in
the victim set. Any tenant workload driving node memory pressure could
evict the source of every kube-state metric we alert on. Background is
in `docs/operations/system-memory-limits.md`.
## The numbers are measured
On a three node cluster with 322 pods and roughly 25.6k series, which is
the smallest we run: 14 day peak working set 307Mi, peak RSS 280Mi, mean
176Mi.
So the request sits above the mean at 192Mi rather than at the peak, and
the limit carries wide headroom at 2Gi. The headroom is deliberate: this
component's memory scales with the number of objects in the cluster, and
the `--metric-labels-allowlist=pods=[*],deployments=[*]` already in this
file widens every pod and deployment series further, so a cluster larger
than the one measured lands well above what was seen here. `2Gi` is 6.7x
that peak, which read as linear in object count is roughly 2100 pods and
170k series, so the limit stays a ceiling rather than a bound well past
the largest cluster this sizing was drawn from.
No CPU limit, following the node DaemonSet sizing in `c9ba7a3cc`, which
set one zero times across nineteen files. CPU is irrelevant to the OOM
ranking, and a ceiling that binds on a scraper turns into scrape
timeouts rather than backpressure.
## This supersedes #2632
#2632 proposes 256Mi, which is below the 307Mi peak measured on the
smallest cluster we have, so it would convert an eviction risk into a
reliable OOM kill. It also adds a `cpu: 200m` limit, which is the half
that buys nothing. The diagnosis there was right and it sat since May;
the numbers are what needed replacing.
## Checks
`packages/system/monitoring-agents/tests/kube_state_metrics_resources_test.yaml`
pins the limit, both requests, and the absence of a CPU limit. Mutation
checked: dropping the limit to 256Mi turns it red. The chart's suites
are 3 files and 6 tests, all green.
### Release note
```release-note
fix(monitoring-agents): kube-state-metrics now carries a memory request and limit sized from a measured peak, so it is no longer a permanent candidate for the node OOM handler
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Configured kube-state-metrics with requests of 50m CPU and 192Mi
memory, plus a 2Gi memory limit. No CPU limit is configured, allowing
CPU usage to scale as needed. These settings help ensure the service has
the memory and CPU resources it needs while keeping memory use bounded.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
5b585d6eda |
fix(operator): persist holds for unsafe memory defaults
An admitted Pod loses the distinction between explicit and defaulted limits. Its disappearance cannot prove a lower default is safe for its replacement. Keep that decision on the Namespace until an administrator acknowledges the particular target, without waiving current blockers or foreign policy ownership. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
30ecc41070 |
fix(operator): close memory policy safety gaps
Preserve foreign same-name objects and keep create-time SSA non-force. Only ignore failed pods controlled by terminal Jobs. Make the failure-containment test inject Apply errors for real. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
6a521f0c68 |
fix(operator): sweep memory defaults when disabled
List LimitRanges by the ownership label once per Package reconcile when the feature is disabled, and remove managed objects even in namespaces outside the current Package target set. Skip the per-namespace path while disabled and keep cleanup failures non-fatal to namespace reconciliation. Update the flag help, installer guidance, and operations documentation to describe the global cleanup boundary. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f6dcb69b61 |
refactor(operator): withhold instead of raising, and drop the grace period
The raise-and-grace machinery cannot work: LimitRanger writes the raised limit onto the very container that justified the raise, so at that container's next admission it carries a limit, the scan skips containers that have one, and the raise has erased its own evidence. The ceiling then drops on schedule and the workload is rejected permanently. Any such machinery approximates admission logic ahead of admission, and that is incomplete by construction: a LimitRange default is one number for a whole namespace and cannot mean "the configured limit, or this container's request, whichever is larger". Only per-pod conditional defaulting expresses that, which is a mutating webhook and not this. The justification for the machinery had also expired. It was designed when the default was 4Gi, where withholding was self-perpetuating for VPA-managed namespaces. At 32Gi nothing in the tree approaches the ceiling -- the largest static memory request is 2Gi -- and VPA sets requests and limits together, so those containers carry their own limit and were never blockers. So a namespace holding a container that requests more than the configured limit with no limit of its own is now left with no default at all, and anything written there earlier is withdrawn. It fails in the safe direction, which is the whole argument: a namespace with no memory.max is a namespace behaving as it did in every release before this, while a default below a request is a namespace whose workload cannot be admitted. It needs no stored state and no timer, because withholding is not a decision that has to survive between reconciles -- once the oversized request is gone the default is applied again on the next pass. Gone with it: the grace period, the clear-since annotation and its clamp, the injected clock, containerMemoryDefault, raisedCeiling, and fourteen tests. The scan stays, because deciding whether to apply or withhold is what it was always for. The foreign-LimitRange ownership guard, the disable path and the 32Gi default are untouched. The residual sharp edge is stated rather than machinery: a workload whose only evidence is a live Pod, built from a custom resource with no template this scan can read, can still be missed if the scan runs while that Pod is absent, and applying the default then rejects its recreation. At 32Gi that is unreachable in this tree. It becomes reachable if an operator lowers the knob below such a workload's request, which the flag help, values.yaml and the operations page now say plainly. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
81c12bc953 |
fix(operator): withhold on any foreign memory policy, not two fields
The coexistence guard treated a namespace as already-governed only when another LimitRange carried a container memory default or max, and said so in its comment. That is not the set of fields that collide with what this reconciler writes. A container min above the default request rejects the pod, and so does a maxLimitRequestRatio, since a 32Gi default against a 32Mi default request is a ratio of 1024:1 and no cap allows it. So the namespace-wide admission failure the guard exists to prevent stayed reachable through fields the guard did not read, and kube-system is where it is likeliest, this feature having deliberately brought it in scope. Enumerating the colliding fields does not fix it, because at the Pod scope the collision is not decidable here. Whether a per-container default of 32Gi breaches a pod-wide max depends on how many containers the pod has, and the pods in question do not exist yet: any arithmetic about them is a guess, and a wrong guess costs a namespace its entire pod admission. So the rule is ownership rather than arithmetic. If anything else in the namespace has an opinion about memory, at either scope and in any field, the namespace is not this reconciler's to default. Narrow still on the resource, because nothing here writes cpu or storage and a LimitRange bounding those must keep coexisting. This is the rule the tenant-namespace exclusion already rests on. The pod scan no longer skips Failed pods, only Succeeded. A pod that ran to completion is never recreated from its spec, but an eviction or a node crash leaves a Failed carcass holding the same oversized request that its controller will recreate, and that carcass is frequently the only surviving evidence the namespace needs a raised ceiling, since the shape the scan looks for cannot be admitted once the namespace has settled. Dropping it lowered the ceiling under a workload that was coming back, which is what the grace period exists to prevent. A grace stamp dated ahead of the clock is clamped to now. Unclamped it held the ceiling for the skew plus the grace period, silently, so an hour of skew bought ninety minutes instead of thirty. Clamping never shortens the wait, it only stops it being extended. The two flag defaults are named constants the flag registration and its test both read. The test previously duplicated the literal and still asserted 4Gi after the default moved to 32Gi, so it misinformed a reader about the shipped value and pinned nothing. The disable claim is narrowed to what the code guarantees. "Removes any previously created one" was universal, but the reconciler only runs for namespaces an active Package still targets, so a namespace whose Package was removed keeps its LimitRange. The flag help, values.yaml and the operations page now say that, and name the managed-by label for finding leftovers by hand. The test asserting a Pod-scoped memory max should be ignored is gone, since under the ownership rule a Pod-scoped memory bound withholds. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
bcfa50d7e9 |
test(e2e): install only the packages the selected suites need
The E2E job installs the whole platform on a fresh cluster before it knows which suites will run, so every PR pays install time and flake surface for packages its suites never use. Select the suites first and pass the complement computed by select-install.sh to the platform Package as disabledPackages. The full-e2e label and an empty selection keep the full install, and the nightly and tag lanes pass no list, so they are unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
974aaa0bac |
test(hack): regenerate the selective install fixture
Two optional Cluster API providers, Proxmox and in-cluster IPAM, were added to the platform sources after the fixture was generated. The drift test compares the fixture with the current disable list, so it failed until they were added. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
f46a9797b8 |
fix(hack): keep platform services in selective installs
Dropping every package that no suite owns also drops services the rest of the platform relies on: multus for secondary networks, the cozystack and LINSTOR schedulers, flux-plunger, cozy-proxy and ouroboros, whose hairpin NAT the Kubernetes suites assert. Nothing waits for them, so no PackageSource edge reaches them, and adding one would also change reverse test selection. The remaining packages stay uninstalled on purpose, also when test selection answers with every suite: nothing in the run looks at them, so installing them only adds a readiness wait. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
59555c895e |
fix(hack): retain controllers and APIs for selective installs
Tenant HelmReleases need a shard reconciler, and the root SeaweedFS release creates COSI resources even when no backup suite is selected. Without those controllers, a fresh reduced install cannot become ready. The Kubernetes suites create worker pools through KubernetesNodes, and MongoDB renders an operator-owned CR. Selecting only the cluster API or application chart drops resources those suites require. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
361893e6f4 |
fix(hack): retain backup APIs in selective installs
Backup roundtrip suites create Bucket and backup resources that the application graph does not require. A graph-only closure therefore omits the APIs those suites need on a fresh install. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
610578be16 |
test(hack): verify selective install contract
A missing suite owner or a discarded discovery failure can leave a reduced install without the APIs its tests exercise. Graph-only checks cannot establish that the chart preserves the requested package set. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b6c33926dc |
fix(dashboard): re-arm the 401 latch once the session recovers
unauthorizedHandled was set once per client and never cleared, so it collapsed every later 401 too, not just the concurrent ones it was meant to. Harmless for the shipped default handler, which navigates away and takes the client with it, but the onUnauthorized hook exists precisely so a caller can re-auth in place, and such a handler would fire exactly once for the client's lifetime and then go silent. Clear the latch on any 2xx from either fetch site: a success means the session is alive again, so the next 401 is a new episode. The storm guard still holds, since concurrent 401s have no success between them. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
7e5f9ab152 |
fix(dashboard): re-auth via /oauth2/sign_in so token-proxy mode works
The dashboard gatekeeper runs one of two mutually exclusive proxies, chosen by the cluster's oidc-enabled flag (templates/gatekeeper.yaml). /oauth2/start exists only in oauth2-proxy. The token-proxy branch registers only sign_in, sign_out and userinfo under the /oauth2 prefix, so there /oauth2/start falls through to the catch-all reverse-proxy handler, which does not re-verify the token. With a present but expired cookie it proxies the request to the console upstream, the SPA reloads, gets a 401 again and loops. /oauth2/sign_in is the one entry point both modes implement, so the default 401 handler goes there. oauth2-proxy's sign_in hands straight off to OAuthStart because the gatekeeper passes --skip-provider-button, and it resolves rd through the same app-director, so the OIDC path behaves as it does through /oauth2/start. token-proxy serves its token form there. It does not honour rd yet, so after sign-in the user lands on / instead of looping. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
badbec63fe |
test(dashboard): cover the 401 -> oauth2-proxy redirect path
The 401 handling is what brings an expired session back to sign-in,
and a regression in it stays invisible until a cookie expires. The
test drives K8sClient through the vi.stubGlobal("fetch", ...) harness
the sibling k8s-client tests use. It checks that the hook fires once
per 401 from both the request and the watch fetch sites, once across
concurrent 401s, never on other responses, and that the default
handler passes the current location as a relative rd, which
oauth2-proxy's open-redirect check accepts.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
8bf78d3bfe |
fix(dashboard): redirect to oauth2-proxy on 401 instead of failing silently
When the kc-access cookie expires, k8s API calls return 401 and the SPA stays on a stale page until the user clears the cache by hand. K8sClient now treats 401 as an auth boundary and makes one top-level navigation into the oauth2-proxy sign-in flow, passing the current location as rd so the user comes back to the same page. The handler can be replaced through K8sClientConfig.onUnauthorized and does nothing outside the browser. The unauthorizedHandled latch keeps concurrent 401s from redirecting more than once. Re-applied from cozystack/cozystack-ui#34 onto the in-tree console after the UI was vendored (#2963), since the source repo is being archived. Related: #2788. Co-authored-by: IvanHunters <49371933+IvanHunters@users.noreply.github.com> Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
3d901ca9b4 |
test(clickhouse): add backup API render assertions
The port the backup strategy dials, the two secretKeyRefs the clickhouse-backup sidecar authenticates with, and the auth Secret that backs them are all rendered by this chart. A helm-unittest suite catches a regression in them on every pull request, while the e2e assertions on the same wiring run only when test-impact analysis selects the ClickHouse suite. The e2e assertions stay, since they check the operator-created Pod and the admitted Secret. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9fe93a3d7e |
docs(operator): fix three claims that overstated what the code does
The static-pod note said containers built from an on-disk manifest are unaffected "in both directions". True of the container, which the kubelet runs from disk whatever the API says, but the mirror pod is an ordinary API object that passes admission like any other, so a static pod declaring a memory request above the ceiling with no limit of its own has its mirror pod rejected -- leaving the container running with no representation in the API. Worth naming, since kube-system is in scope by design. The scan does cover it: a mirror pod is a live Pod. The ReplicaSet exclusion was justified on live ones being copies of their Deployment's template. That does not reach an ownerless ReplicaSet, which at replicas 0 has neither pods to be seen through nor a Deployment template standing in for it. It stays out of scope -- nothing here ships a bare ReplicaSet -- but as a gap left open rather than one the stated reason closes. The delete guard's comment claimed the enabled apply always adopts and stamps a same-named foreign object. It does not when that object's spec already matches, because then the apply is skipped entirely. The outcome is the same one the guard gives on purpose, but it is reached by the apply not running. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
1df6e3ac8f |
revert(linstor): leave the controller memory limit to its own PR
This branch gave linstor-controller a 10Gi memory limit, justified in the values and pinned in cluster_test.yaml on the grounds that the namespace default was 4Gi, half the -Xmx8G baked into the image. Both halves of that have moved. The default is now 32Gi, four times the heap ceiling rather than half of it, so nothing about the JVM requires a per-chart limit to sit above the namespace default any more. And #3824 argues the opposite direction for the same component -- extending the controller's existing limit-free exemption to linstor-satellite, using that exemption as its precedent. Keeping the 10Gi here would remove the precedent that PR reasons from, and whichever of the two merged second would silently invalidate the other's rationale. So linstor policy belongs to one PR, and it is not this one. The three files go back to their state on main; this branch is now confined to the operator and the installer chart. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
e131a49514 |
feat(operator): default the system namespace memory ceiling to 32Gi
The guard in front of this default reads declared memory requests. It cannot see a container's real working set, so a system component that requests little but legitimately uses more than the ceiling, and declares no limit of its own, was handed a deterministic kernel OOM on its next recreation in exchange for leaving the Talos handler's victim set. Since the feature ships enabled, that trade fell on the whole fleet on upgrade. It is not a trade this feature has to make. The handler discards a cgroup for carrying a memory.max at all, whatever its value, so immunity does not require a fitted ceiling -- only a real one. A default far above any plausible working set buys the immunity and leaves nothing for the ceiling to bind on, which also keeps it clear of the failure mode this repository already carries a scar from, where an undersized limit converted a rare pressure-driven kill into a reliable one. Nothing about scheduling changes: only requests are scheduled against, and the paired default request stays 32Mi. An operator who wants the ceiling to bind lowers it, per component in a chart or fleet-wide with this knob, once real usage is measured -- which is the direction that needs evidence, and now the direction that needs it rather than the default. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
b6cb0def78 |
docs(operator): say that kube-system is in scope for the memory default
The scope of the default LimitRange is every namespace a Package targets whose name does not start with `tenant-`, and cozystack-scheduler installs into kube-system, so a namespace owned by the underlying Kubernetes distribution is defaulted like any other. Neither the flag help nor the installer value said so, and both are what somebody setting the knob reads. docs/operations/system-memory-limits.md already covers this, including the part that surprises: containers the kubelet builds from an on-disk manifest never pass through admission, so a Talos control-plane static pod gets no memory.max from this feature in the first place and the mirror pod is the only thing the LimitRange reaches. Both notes now carry the same statement, so neither reading of the knob leaves it out. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dd36eeddbc |
docs(operator): describe the raise, not the retract, in the shipped help
The flag help and the installer values comment both still said the operator skips a namespace holding a request above the configured limit and retracts any LimitRange it applied there. It stopped doing that in 3127c0fad: it raises that namespace's ceiling to clear the request and drops back once the request is gone. An operator reading the shipped help and then finding an 8Gi LimitRange in cozy-monitoring had no way to connect the two. Both texts now describe the raise, the drop-back and the log line to grep for. The values comment also loses its VPA warning, which this branch's own comment in findRequestAboveDefaultLimit refutes — VPA rescales the limit in proportion to the request under the default controlledValues, so it cannot be what puts a namespace over — and gains a pointer to declaring a limit in the component's own chart, which is how packages/system/linstor now stays out of the default's reach. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
ae431808ea |
fix(linstor): give linstor-controller a memory limit above its baked-in -Xmx8G
linstor-server bakes `-Xms256M -Xmx8G -XX:+CrashOnOutOfMemoryError` into the generated bin/Controller (build.gradle, jvmControllerOpts), nothing in this chart or in piraeus-entry.sh overrides it, and a JVM handed an explicit -Xmx never reads memory.max. The container was deliberately left with a 700Mi request and no limit at all, which was the safe choice while nothing else supplied one. It stops being safe alongside the system-namespace LimitRange: a container declaring no memory limit is given the configured default, 4Gi, so the process's own heap ceiling would sit at twice its cgroup ceiling and a large enough DRBD resource count would have the kernel kill the storage control plane where the JVM would otherwise have crashed itself loudly at 8G. Stating the limit here is also what keeps the container out of the LimitRange's reach, since a container with its own limit is never defaulted one. 10Gi is the 8G heap plus room for metaspace, code cache, thread stacks and direct buffers. CPU stays unlimited. The helm-unittest assertion that pinned "no limits" is replaced by one pinning the memory limit and the absence of a CPU limit, so neither lowering it below the heap ceiling nor dropping it again passes silently. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
546d8e6a54 |
feat(operator): default system namespace memory via LimitRange
The Talos userspace OOM handler (v1.12+) picks its victim by ranking
cgroups with `memory_max.hasValue() ? 0.0 : {Besteffort: 1.0, Burstable:
0.5, ...}[class] * memory_current`, discarding every cgroup that scores
zero. Only a memory limit sets memory.max on the pod cgroup, so only a
limit takes a pod out of the candidate set: a request merely demotes
BestEffort to Burstable, which under the v1.13.6 default
strictCgroupClassOrdering still means "killed second". PriorityClass has
no effect at all, because the handler walks cgroupfs and never talks to
the API server.
Victim selection is decoupled from the trigger, so the cgroup that
caused the pressure and the cgroup that gets SIGKILLed are unrelated by
design. Tenant workloads generally carry limits and system components
generally did not, which inverted the intended order: a tenant workload
thrashing against its own limit drives node-wide memory PSI, and
metallb-speaker or linstor-satellite is killed for it while the workload
responsible is never touched.
An audit found roughly 70 workloads across ~30 packages with at least
one container missing a memory limit, so per-chart resources would
neither cover the current set nor keep covering it. Maintain a
LimitRange in every system namespace instead, at the single choke point
that already creates them and already distinguishes system from tenant
(reconcileNamespaces). Tenant namespaces are left alone because
packages/apps/tenant owns LimitRange policy there.
The limit is a ceiling, not a reservation: it defaults well above real
usage (4Gi against a largest observed system request of 512Mi).
defaultRequest is set explicitly and small, because Kubernetes otherwise
defaults each request to the limit and reserves that much at schedule
time. Setting the limit to 0 disables the LimitRange and removes any the
operator previously created.
Two things keep the feature from being able to hurt more than it helps.
A LimitRange default only reaches a container that declares no memory
limit of its own, and the API server then rejects a request above its
limit. So before applying, scan the namespace for a container that
requests more than the configured limit and sets none, and where one
exists skip the namespace, retract any LimitRange applied earlier, and
log the pod, container, request and limit. This is not hypothetical:
packages/system/monitoring and packages/system/monitoring-agents ship
VPAs for cozy-monitoring with maxAllowed.memory of 8Gi and 6G under
updateMode: Initial, so the VPA admission webhook -- which runs after
the in-tree LimitRanger -- can write a request past 4Gi that no chart in
the tree declares. That is why the check runs against live pods rather
than rendered templates, and why it reads through APIReader rather than
starting a cluster-wide Pod informer.
And no failure inside the feature propagates: a LimitRange error, or a
pod list that cannot be read, is logged and stepped over instead of
failing the Package reconcile. A namespace without a LimitRange is back
to the behaviour of every earlier release, while a namespace that never
reconciles leaves its components uninstalled.
Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
|
f318157d69 |
docs(networking): qualify webhook translation fallback
Limit the fail-open discussion to the actual counterexample: another component translating the Service after Cilium source-side egress policy. A disabled translator by itself leaves no working Service path. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
79e236aa7a |
test(networking): harden webhook regression proof
Probe one exact ready admission EndpointSlice backend throughout the measurement window and reject missing or ambiguous Service ports. Ensure the kubeovn-webhook Helm suites run through package discovery. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
c9558f5658 |
fix(networking): correct webhook policy assumptions
Read the ingress admission port from the effective subchart values and fail if the port is absent. Describe kube-proxy replacement as a required service-routing invariant while retaining target-side enforcement. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
361311c5c9 |
docs(monitoring-agents): note what object count the 2Gi limit covers
The sizing comment named the measured peak without saying how much room 2Gi leaves above it, which left open why the chart pins 2Gi when the default LimitRange limit proposed in #3494 is 4Gi. 2Gi is 6.7x the measured 307Mi peak working set. Read as linear in object count, that is roughly 2100 pods and 170k series against the 322 pods and ~25.6k series the peak was measured on, so the limit stays a ceiling rather than a bound on a cluster several times larger than the one behind the numbers. Comment only. The requests, the limit and the deliberate absence of a CPU limit are all unchanged. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
2e533acf4e |
fix(monitoring-agents): give kube-state-metrics a sized memory limit
It ran BestEffort with no resources at all, so it set no memory.max on
its pod cgroup and the Talos userspace OOM handler kept it permanently
in the victim set: any tenant workload driving node memory pressure
could evict it, and it is the source of every kube-state metric we
alert on.
Numbers are measured rather than guessed. A 14d peak on a 3-node
cluster with 322 pods and ~25.6k series is 307Mi working set, 280Mi
RSS, 176Mi mean. The request sits above the mean and the limit carries
wide headroom, because this component scales with object count and
--metric-labels-allowlist widens every pod and deployment series
further, so a larger cluster lands well above what was measured.
No CPU limit, following the node DaemonSet sizing in
|
||
|
|
48e3d1d611 |
feat(redis,postgres): certify the per-engine external hostname
The tenant Gateway's TLS-passthrough listeners are designed to route databases by SNI under a per-engine subdomain (*.redis.<apex>, *.postgres.<apex>), and a passthrough listener forwards the handshake untouched, so the engine's own server certificate must carry <release>.<engine>.<apex> or a verifying client fails the hostname check. The name is issued next to the existing flat <release>.<apex>, under the same TLS-and-external condition, so that a later move between the two hostname schemes reissues no certificate and leaves clients on the same trust anchor. Nothing renders a route yet; the certificate is prepared ahead of it. Adding the name changes the SAN list, so existing external TLS releases get their server certificate reissued once on upgrade, from the same CA. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
92bf90cc7d |
fix(kubernetes): force the kubevirt-csi temp NFS unmount after a timeout
The NFS publish path temp-mounts the export root and unmounts it in a defer with a plain umount, which has no deadline. If the NFS server is gone by then, that umount can block indefinitely: the publish hangs and the temp mount stays behind for the life of the node plugin container. mount-utils' UnmountWithForce cannot fix this. It retries with -f only after its first umount returns, and that umount forks the umount.nfs helper: killed on the timeout, umount leaves the helper stuck on the vanished server with the output pipe still open, and CombinedOutput waits for the pipe, so -f never runs. The plugin therefore runs umount itself with WaitDelay set, which closes the pipe shortly after the kill, and falls back to umount -f under a timeout of its own. As in mount-utils, "not mounted" counts as done: umount2 detaches the mount before it blocks tearing the superblock down, so the -f after a killed umount can find nothing left. A mounter that is not the exec-based one keeps its plain Unmount. An umount in uninterruptible sleep outlives any kill, so the cleanup also waits at most one fixed budget, well inside the 2m deadline kubelet puts on NodePublishVolume, and leaves a stalled unmount running in the background, where it logs its result once it ends. The temp dir is removed by that background unmount once it succeeds, not by the publish: removed at the deadline it is still a mount point, and nothing would retry. A forced unmount runs nfs_umount_begin, which kills every RPC in flight on the superblock, and NFS shares one superblock between mounts of the same export with the same options. The temp mount therefore gets nosharecache, so -f cannot fail the I/O of pods on the node that mount the same volume. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
520af57905 |
chore: drop private cluster, host and personal names from the tree
Private infrastructure names, a real hostname, a commented-out internal mirror block, an internal product name, and a reviewer's name in a comment do not belong in a public repository; drop them and replace what needs a stand-in with neutral placeholders. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
d2e3eb4f82 |
fix(kubernetes, e2e): unblock autoscaling below min size, size the computeplane pool, fix the backupcontroller build (#4768)
## What this PR does Makes the backupcontroller test package compile again on main. #4587 added a flavor argument to `createCNPGBackupArtifact` and `marshalCNPGBackupSnapshot`, and #4669, whose CI ran before that, added two test calls with the old argument lists. Both PRs were green on their own, and together they break `go test ./internal/backupcontroller/`. The two calls now pass the same default flavor as the other tests in the file. The second commit fixes an e2e suite that fails the required E2E check here. It is the `computeplane` suite from #4713, and its worker pool was one node with a single vCPU. After the kubelet reservations that node has 900m of CPU to allocate. The tenant addons (cilium, coredns, csi, ingress-nginx, metrics-server, cert-manager) request exactly 900m between them. If cert-manager's startupapicheck hook pod is created last, it can't be scheduled and the install times out after 10m. The reinstall after remediation runs into the same full node. Whether that pod comes last depends on install order, so the suite passed on #4713 and failed here. The pool now has two workers with one vCPU each, the same shape the kubernetes suites use. One vCPU per worker stays on purpose, because of #3513. Two workers exposed a deadlock in the tenant cluster-autoscaler. The suite's pool is created at zero replicas and then gets a min size of 2, which CAPI does not apply to the live replica count on upgrade (#4786). The autoscaler's scale-up for a pending pod asks for one node, and the clusterapi provider refuses it because one is under the min size, although it is an increase. That refusal backs the group off, and the min-size enforcement skips groups in backoff, so the pool never leaves zero. Upstream had the same report in kubernetes/autoscaler#4942, closed as stale, and it is now tracked in kubernetes/autoscaler#10400. Our autoscaler image already patches the mirror case for max size, and the patch now refuses only a decrease below the min size. `go vet` and `go test ./internal/backupcontroller/...` pass, and so do the bats pins for the computeplane suite. The autoscaler patch adds a test for each direction, and the clusterapi package tests pass with it applied to 1.32.0. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved cluster autoscaling for worker groups outside configured size limits: groups below their minimum can scale up toward it, and nodes can be removed when a group exceeds its maximum. Scale-down requests below the minimum remain restricted. * **Tests** * Updated backup and restore test calls with PostgreSQL and empty flavor values. * Updated ComputePlane checks for a two-worker pool and clarified cert-manager installation timing expectations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
82ae9f6cf7 |
fix(kubernetes): let the autoscaler grow a node group that sits below min
The clusterapi provider refuses any SetSize under the group's min size, including an increase. A pool whose min size is raised while it stands at fewer replicas then deadlocks: the scale-up for pending pods asks for fewer nodes than the min, is refused, and puts the group in backoff, and the min-size enforcement skips groups in backoff. The group stays where it is for good. Refuse only a decrease below min size. The guard it keeps is the one DecreaseTargetSize relies on, and DeleteNodes checks min size itself. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
9620be0103 |
feat(postgres): add per-user createRole and bypassRls attributes (#3505)
## What this PR does - Adds `users.<name>.createRole` to the postgres chart, which sets the `CREATEROLE` role attribute on a managed login user. - Adds `users.<name>.bypassRls`, which sets `BYPASSRLS` on the same user. The two attributes are needed together to provision a privileged service role, see below. - Sets `createrole_self_grant` as a role-level default alongside `createRole`, without which the attribute is granted but not usable. - Refuses `createRole` on PostgreSQL 15 and older, where it is an escalation path rather than a bounded privilege. - Makes the init job record the roles it manages in a superuser-only marker instead of trusting role comments and refuses to adopt a role it did not create once `createRole` has been used, see below. - Adds `tests/user_createrole_test.yaml`: defaults, both explicit values for each attribute, composition with `replication` and with each other, the `createrole_self_grant` set and reset paths, and the version guard on v15 and v13. Adds `tests/managed_roles_test.yaml` for the marker and the adoption checks. Today a tenant that needs to create its own roles has no way to do it through the chart, because role attributes are not inherited through role membership: granting `CREATEROLE` to the generated `<db>_admin` group role has no effect on the users that are members of it, and `<db>_admin` is `NOINHERIT` anyway. The attribute has to be set on the login user itself, which only the init script can do. The remaining option is to hand the tenant the `postgres` superuser, which also gives it `COPY ... FROM PROGRAM`, arbitrary file access and command execution inside the instance pod. These flags are the narrow version of that: the tenant manages its own roles, and the superuser stays internal. Both attributes are reconciled on every init-script run, like `replication`, so setting one back to `false` emits the `NO...` form and revokes it. ## Why `bypassRls` belongs in the same PR `CREATEROLE` on its own does not let a tenant reproduce a role layout that uses row-level security, which is the common case this feature exists for. PostgreSQL only lets a role create other roles carrying `BYPASSRLS` if it holds the attribute itself, so `CREATE ROLE service_role ... BYPASSRLS` fails with `Only roles with the BYPASSRLS attribute may create roles with the BYPASSRLS attribute`. Exposing `createRole` without `bypassRls` therefore ships a feature that stops one statement short of the goal. `bypassRls` is not gated on a PostgreSQL version, because it carries no escalation of its own: `BYPASSRLS` waives row-level security inside the tenant's own databases, and the database owner already reads every row in them. It grants no reach outside the tenant's data. ## Why `createrole_self_grant` is set with the attribute This is the non-obvious half. PostgreSQL 16 grants a `CREATEROLE` holder only `ADMIN OPTION` on the roles it creates, governed by `createrole_self_grant`, whose default is empty. `ADMIN OPTION` allows granting the role onward but not `SET ROLE` to it, and creating an object owned by another role requires `SET`. The attribute therefore looks functional -- `CREATE ROLE` succeeds -- while the very next statement a tenant writes, `CREATE SCHEMA ... AUTHORIZATION <new role>`, fails with `must be able to SET ROLE`. Every tenant granted `createRole` would hit this and have to discover the parameter, so the chart sets it as a role-level default and resets it when the attribute is withdrawn. ## Why the init job stops trusting role comments and role names The init job runs as the superuser, and it trusted two things a CREATEROLE holder controls for the roles it creates. It picked the users and roles to drop by their comment, and on PostgreSQL 16 a CREATEROLE holder may `COMMENT ON ROLE` every role it created, so it could hand the chart for deletion a role the chart never created. It also adopted any role that already existed under a declared name, so a role created under a name an administrator declares later stayed under its creator's control, for example `debezium` before `users.debezium.replication: true`, or `newdb_admin` before `databases.newdb`. Neither case can be caught from the catalog. Creating a role through an intermediate role and then dropping that role leaves the squatted role with no row in `pg_auth_members` at all, while the objects, role settings and default privileges planted in it stay. On 17.11 such a `debezium` had zero memberships, and after the chart adopted it `migrator` could still run code as `debezium` with `REPLICATION` through a function it had planted. The chart therefore keeps its own record: - Chart-managed roles are members of `cozystack:managed`, a NOLOGIN role that only the superuser can grant. On 16+ the membership is `WITH INHERIT FALSE, SET FALSE`. The name is outside the username and database charsets. The delete passes act only on these members. On its first run the chart marks every role it had commented before, when no role could yet hold CREATEROLE through it. - Once any user has held `createRole`, recorded as `cozystack.createrole_granted` on the `postgres` database (the same mechanism as `cozystack.adopted_restore`), the chart refuses to adopt an existing role it did not mark, both for users and for `<db>_admin`/`<db>_readonly`. The init job fails with the role's name, and a platform administrator must drop the role. Before that point adoption works as before, so a fresh install that declares `users.app` still adopts the role CNPG bootstraps. ## The PostgreSQL 16 floor on `createRole` The floor is a security boundary, not a cosmetic check. Before PostgreSQL 16, `CREATEROLE` allowed granting membership in any non-superuser role, so a user holding it could grant itself `pg_execute_server_program` and reach command execution on the instance, which is exactly what not exposing the superuser is meant to prevent. I verified both sides on a real cluster: on 15.15 `GRANT pg_execute_server_program TO <createrole_user>` succeeds and `COPY ... FROM PROGRAM 'id'` then returns `uid=26(postgres)`; on 16.11 and 18.1 the same grant is refused because it now requires ADMIN option on the role. PostgreSQL 16 restricted `CREATEROLE` to roles the grantee administers, which is what makes the attribute safe to expose. The guard uses the same fail-fast `SECURITY:` idiom as the blocked `postgresql.parameters` list in `templates/db.yaml`. Note that a grant obtained on PostgreSQL 15 survives a major upgrade: after moving an instance to 16.11 the user kept its membership in `pg_execute_server_program` even though new attempts were refused. Operators who exposed `CREATEROLE` out of band on an older major should audit and revoke memberships in `pg_execute_server_program`, `pg_read_server_files` and `pg_write_server_files` before relying on the floor. ## Verification The marker, adoption and `createRole` paths were rendered and run against live PostgreSQL 13, 15, 16 and 17 containers. An upgrade from the previous chart marks the roles it already managed and removes a dropped user. What happens to a dropped role's objects comes from #4574 and is covered by its tests. A fresh install adopts the CNPG `app` role in the same run that grants `createRole`. A forged comment is ignored, and squatted `debezium` and `newdb_admin` roles stop the init job with nothing granted to them. The orphaned-role pass still drops the roles of a removed database, and every re-run is clean. Reverting each part of the fix in turn fails `tests/managed_roles_test.yaml`. Beyond the chart unit tests, I replayed a real migration checklist against a live PostgreSQL 17.7 instance, connected as the managed login user and never as the superuser: creation of four roles with their attributes including `CREATEROLE` and `BYPASSRLS`, `CREATE SCHEMA ... AUTHORIZATION` landing on the requested owner, trusted extensions into a named schema, per-role and per-database `search_path`, per-role `statement_timeout` and `idle_in_transaction_session_timeout`, RLS including `FORCE ROW LEVEL SECURITY` verified end to end (a plain role sees zero rows, the `BYPASSRLS` role sees them all), and `SECURITY DEFINER` functions. All of it succeeds without the superuser. The one thing that remains out of reach is `CREATE EVENT TRIGGER`, which PostgreSQL restricts to superusers and no chart flag can change. Other attributes and the predefined roles that a managed-service admin role usually wants (`pg_monitor`, `pg_read_all_settings`, `pg_maintain`) can follow the same per-user pattern in a later PR, minus the three that grant OS access (`pg_execute_server_program`, `pg_read_server_files`, `pg_write_server_files`). ### Screenshots No UI change. ### Downstream repositories `packages/apps/postgres/values.schema.json` gains two fields, which per the trigger map reaches `cozystack/terraform-provider-cozystack`: its postgres resource carries a hand-maintained schema plus an expand/flatten pair for `users`, with no codegen from `values.schema.json`, so `createRole` and `bypassRls` will be invisible to Terraform users until they are added there by hand. I searched that repository and found no open PR or issue covering it (the only open PR is a dependabot Go module bump). I have not filed the follow-up, so I am leaving the box unticked rather than claiming work that does not exist. Happy to open the issue or the provider PR if a maintainer confirms that is the preferred route. I walked the rest of the map against the diff and found nothing else: no package is added, renamed or removed, no version enum changes, no default changes, `release.prefix` and the output Secret and Service names are untouched, and no CRD outside the regenerated `packages/system/postgres-rd/cozyrds/postgres.yaml` is affected. The `README.md` parameter table is regenerated here and postgres is already in the website generator's app list, so the website path stays automated. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: not filed yet, see the note above - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(postgres): add `users.<name>.createRole` and `users.<name>.bypassRls`, which grant the CREATEROLE and BYPASSRLS attributes to a managed user so a tenant can create and manage its own roles, including roles that bypass row-level security, without the postgres superuser. `createRole` also sets `createrole_self_grant` on the user so that roles it creates can own objects. `createRole` requires PostgreSQL 16 or newer: on older majors CREATEROLE can be used to grant oneself pg_execute_server_program and reach command execution on the instance, so the chart refuses to render the attribute there. The init job now records the roles it manages as members of a superuser-only `cozystack:managed` role instead of trusting role comments. Once `createRole` has been used on an instance, it refuses to adopt an existing role it did not create. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * PostgreSQL user configuration now supports `createRole` and `bypassRls` privileges. * These privileges can be combined with replication access. Enabling `createRole` also configures the role’s self-grant behavior. * `createRole` requires PostgreSQL 16 or newer; configuring it on older versions is rejected. * **Documentation** * Updated configuration schemas, examples, and guides to describe the new options and PostgreSQL version requirements. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |