mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
ci/shellcheck-gate
3262
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
b4e203164a |
fix(kubernetes): retry the tenant CSI install instead of uninstalling it (#3581)
## What this PR does The tenant CSI HelmRelease carried `remediation.retries: -1` with no strategy, so Flux applied its default, `RemediateOnFailure`. Install remediation is an uninstall in every case the API offers, so an install that missed its budget was torn down between attempts: the tenant lost its CSI node plugin and the StorageClasses this release propagates until a later attempt applied them again, and that attempt restarted the DaemonSet rollout instead of resuming one already under way. That teardown races the rollout it exists to recover, and the run recorded in #3576 shows how narrowly. Both DaemonSet pods were scheduled at the same second on both workers, so nothing was waiting on capacity; the ten minutes went to image pulls. The slower worker started its last container six seconds after the install timeout fired, and the uninstall removed it one second after that. `RetryOnFailure` leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry from `.status.lastAttemptedReleaseAction` and performs that retry as an upgrade: set on install alone, the retry still happens, but its own failure is then handled by upgrade remediation, whose default is a rollback with no previous release to roll back to. The `remediation` blocks stay, because an absent release consults `install.remediation` and an out-of-sync one consults `upgrade`, both by retry count rather than by strategy. This is the change #3552 makes for the tenant CNI, and the reason for putting the strategy on both actions is the one given there. It is also the shape upstream prescribes: the "Recommended settings" block in the [HelmRelease spec shipped with the helm-controller this installs](https://github.com/fluxcd/helm-controller/blob/v1.5.1/docs/spec/v2/helmreleases.md#recommended-settings) puts `strategy.name: RetryOnFailure` under both `install` and `upgrade` for production deployments. Nothing changes on the test side, and that is deliberate rather than deferred. Because no manifest is removed at the attempt boundary, the rollout continues across it and the kubelet finishes what it started, while the retry re-applies the same manifests every 30s. The release therefore reports Ready once the rollout completes rather than once a whole attempt has fit inside the timeout, so what the existing e2e wait has to outlast is the rollout and no longer an attempt boundary that discards it. That widens what the wait tolerates without making it unbounded: a rollout slower than the wait itself still fails, and this does not claim otherwise. Scope. #3576 records more than this fixes, so this does not close it. The other tenant addon HelmReleases this chart renders keep the remediation-only shape; CSI is the one this changes. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: The trigger map was walked against the diff. The change is a HelmRelease template and a chart test, not a `values.schema.json` field, a `values.yaml` default, a version enum, a `release.prefix`, an `ApplicationDefinition`, a chart source kind or a namespace. ### Release note ```release-note fix(kubernetes): the tenant CSI HelmRelease now retries a failed install instead of being uninstalled between attempts, so a failed install no longer takes the tenant's CSI node plugin and its propagated StorageClasses away while it retries. The same strategy applies to the upgrade action, which it has to: a retried install runs as an upgrade. One consequence is operator-visible — a failed CSI upgrade in a tenant cluster now retries the new manifests every 30s instead of rolling back to the previous release. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved reliability when installing or upgrading the CSI component by automatically retrying failed actions every 30 seconds. * Retains unlimited remediation retries to help recover from transient failures. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
879d0f6db6 |
fix(installer): make operator readiness honest in the chart and the install gate (#3466)
## What this PR does
The operator Deployment ran `replicas: 1` with `maxUnavailable: 1`, so
the Deployment controller took the minimum available replica count as `1
- 1 = 0` and the `Available` condition reported `True` with no running
pod at all. Measured against an operator whose image tag does not exist:
the condition was `True` with `reason: MinimumReplicasAvailable` while
the only pod sat in `ImagePullBackOff`, `unavailableReplicas: 1`, and no
`availableReplicas` key in the status.
That broke two separate things, so this fixes both.
The E2E install gate asked `kubectl wait --for=condition=Available` and
passed in 156 ms against that dead operator. The install then carried on
and died about two minutes later on the `packages.cozystack.io` CRD wait
further down the file, which sends you after CRD installation when the
operator never started. It now uses `kubectl rollout status`, which
reads updated and available replicas directly, and dumps the deployment,
pods, operator logs and namespace events when it fails. This half is
deliberately independent of the chart: `rollout status` is correct under
any rollout strategy, so a later change to that strategy cannot quietly
make the gate vacuous again.
The chart is the other half, and fixing the gate does not cover it.
`Available` is read by more than this test, and until the strategy
changes an alert on that condition, a health dashboard, or a person
reading `kubectl get deploy` all show green while the operator is dead.
The strategy moves to `maxSurge: 1` / `maxUnavailable: 0`, which
restores the ordinary meaning of the condition. Kubernetes refuses
`maxSurge: 0` and `maxUnavailable: 0` together, so the surge has to move
for the unavailability to reach zero.
That pair is worth reading twice, because it is not a configuration I
picked. At `replicas: 1` the Kubernetes default of `25%` / `25%`
resolves to exactly `maxSurge: 1` and `maxUnavailable: 0`, since the
surge rounds up and the unavailability rounds down. What this PR does is
stop overriding the default, not introduce a tuning of its own, and the
risk of the change should be judged on that basis.
Moving the surge is safe here, and I checked rather than assumed,
because the pair came from somewhere. It was carried over from the
earlier `cozystack` Deployment, where a sidecar published host port 8123
and `hostNetwork` left the surge pod unschedulable on a single-node
cluster. It got here in two steps, which is worth knowing if you go
looking for the reason: by the commit before this file existed the
sidecar was already gone and no port was declared, yet the pair stayed.
It had outlived its own reason before it was ever copied, which is why
there is no justification to find in this file's own history. On a
single-node cluster the surge pod is scheduled next to the outgoing one
and the rollout completes; adding a declared container port reproduces
the original `didn't have free ports for the requested pod ports`
failure exactly. Nothing else in the pod holds a port either: metrics
and health probes are bound to `0`, and the configured webhook server
never starts because the operator registers no webhook and never asks
the manager for the server.
The surge makes leader election load-bearing, though it covers less of
the overlap than it first appears. While `maxSurge: 0` capped total pods
at `replicas` a second manager process could not exist, so
`--leader-elect=true` was defence in depth. With a surge pod the lease
is what keeps the incoming manager's reconcile loops off the cluster
while the outgoing one is still leader, and because there is no
readiness probe the incoming pod counts as available as soon as its
container starts, so that overlap covers the whole image pull. It does
not cover everything: the operator installs CRDs, installs Flux and
creates the platform PackageSource before `mgr.Start()`, and the lease
is only taken inside `mgr.Start()`, so those three privileged installs
run outside leader election entirely. They are safe because both pods
write the same objects through server-side apply under the same field
manager, which is a different guarantee from the lease. The
`k8s-await-election` wrapper this operator replaced did hold the whole
process until it won leadership, so if that property sounds familiar it
belongs to the predecessor rather than to this binary.
The sibling `opensearch-operator` chart turns leader election off at one
replica for a real reason: a transient lease-renewal blip makes
controller-runtime exit the process, which at one replica reads as a
crashlooping Deployment. This operator installs first, when the
apiserver is at its shakiest, so that scenario applies here too. I am
accepting the risk knowingly, because after `maxSurge: 1` the
alternative is two managers reconciling the same cluster, which is
worse. The flag is pinned in the unit test so it cannot be turned off
without the test going red.
One consequence is permanent, so I am writing it down now instead of
leaving it to be found in a year. Because any bound port breaks the
surge pod under `hostNetwork`, an `httpGet` readiness probe is off the
table for this Deployment for good and only an `exec` probe could ever
work. So `Available` still answers "a container started", not "the
operator is healthy": an operator that starts and then dies partway
through installing CRDs is caught by the CRD waits below the gate, not
by the gate itself.
The surge also opens one new window. The operator installs CRDs and Flux
before the manager starts, which puts that work outside leader election,
so the incoming pod runs those applies while the outgoing pod is still
leader. Both apply the same objects server-side and the outgoing pod
exits moments later.
`--wait` is dropped from the helm step in the same change. Helm decides
a Deployment is ready with the same `replicas - maxUnavailable`
arithmetic, so on the old strategy it was satisfied at zero ready pods
and gated nothing, which is why the broken install got past it as well.
Once the strategy is honest it would gate for real, but on its own 2m
budget, and it would fail the install before the rollout gate ran and
hide the diagnostics behind a bare Helm timeout. Hooks are waited for
with or without the flag, and the release otherwise carries only the
Deployment and its RBAC objects, so the rollout gate is the better
single place to decide the operator is up.
A helm unittest suite pins the strategy and all four preconditions the
surge makes load-bearing: no container or initContainer declares a port,
both listener arguments stay disabled, leader election stays on, and no
probe uses `httpGet`. That last one follows from the two disabled
listeners leaving no port for a probe to reach, and it is the worst of
the four to get wrong, because the others wedge an upgrade while a
`livenessProbe` against a disabled listener crashloops the pod on every
cluster, fresh install included. Only the `httpGet` field is forbidden
rather than the probes themselves, so an `exec` probe stays available.
All four run against all three chart variants, because the template
branches on variant inside the container spec and a port added under one
branch would render for that variant alone.
I checked the suite fails for the right reasons instead of trusting it.
Reverting the strategy, declaring a `containerPort` on the operator
container, declaring one under a single variant branch, declaring one on
an initContainer, pointing a bind-address at a real address, turning
leader election off, and adding a readiness, liveness or startup probe
with `httpGet` each turns it red on its own assert. Inserting a
container ahead of the operator trips the container-name pin before the
argument asserts can start describing the wrong container. An `exec`
probe correctly stays green.
The rationale is split by audience. What an operator applying the
rendered manifest has to respect stays in plain comments and travels
into `_out/assets/cozystack-operator-*.yaml`: why the strategy is what
it is, that no port may be declared or bound, and that leader election
has to stay on. Everything that is only about this repository moves into
a `{{/* */}}` template comment that stops at the chart boundary. That
covers the pointer to the test and which assert pins what, where the
strategy was copied from, why the webhook server on 9443 is not a third
port binder, and why `opensearch-operator` reaches the opposite
conclusion on leader election. The other templates in this chart already
keep maintainer notes that way.
### Screenshots
Not applicable, no UI changes.
### Downstream repositories
I left the boxes empty on purpose rather than to skip the question,
because the honest answer is not one of them and someone else should
make the call.
Walking the trigger map in `docs/agents/contributing.md`, nothing fires.
The `hack/` triggers name `hack/e2e-prepare-cluster.bats`,
`hack/package.mk`, `hack/update-crd.sh` and moves or renames under
`hack/`, and this PR only edits the body of
`hack/e2e-install-cozystack.bats`. The `ansible-cozystack` trigger is
renaming an object the role waits on, and
`deployment/cozystack-operator` keeps its name.
The behaviour that role sees does change, though. It runs `kubectl wait
deployment/cozystack-operator --for=condition=Available`, the same
vacuous check this PR replaces here, so the chart change turns its wait
from one that returns instantly into one that means something. Its
`cozystack_operator_wait_timeout` defaults to 300 seconds, the same
budget I gave the gate here, so a cold image pull fits and I do not
think anything needs to change over there. No follow-up PR seemed
warranted for a change that only makes an existing check work, but that
is a judgement call rather than a trigger, which is why the boxes are
empty.
- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
### Release note
```release-note
fix(installer): the cozystack-operator Deployment no longer reports Available with zero running pods. It ran replicas: 1 with maxUnavailable: 1, which made the minimum available replica count 0, so the condition stayed True while the operator was dead and anything reading it, including the E2E install gate, saw green. The rollout strategy is now maxSurge: 1 / maxUnavailable: 0 and the install gate waits for the rollout instead of the condition.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved operator installation checks by monitoring deployment
rollouts and providing deployment, pod, log, and event details when a
rollout fails.
* Updated operator upgrades to maintain availability during rollout,
including in single-node environments.
* Documented rollout requirements and behavior across supported
installation configurations.
* **Tests**
* Added coverage for rollout settings and upgrade compatibility across
supported installation variants.
* Clarified Helm unit-test directory coverage documentation.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
a4a8e4143f |
fix(kubernetes): retry the tenant CNI install instead of uninstalling it (#3552)
## What this PR does The tenant Cilium HelmRelease carried no install strategy, so Flux applied its default, `RemediateOnFailure`, which uninstalls the release between attempts. Cilium is the tenant cluster's only CNI, so every cycle takes it away and gives it back when the next install applies the manifests again, and `retries: -1` lets that repeat without end. What sets a cycle off is an install that outruns its budget, which can happen while the tenant still has no registered node, since the operator Deployment cannot become Available until one exists. Remediation neither causes that delay nor is fixed by this change: what it adds is a repeated teardown of the CNI on top of a delay that resolves itself once the node arrives. This removes the amplifier, not the cause. `RetryOnFailure` keeps the applied manifests in place and retries a failed install as an upgrade, so the agent DaemonSet rolls out as soon as a worker registers. Both actions carry the strategy because helm-controller selects the active retry from `.status.lastAttemptedReleaseAction`, and the retry of a failed install runs as an upgrade. With the strategy on `install` alone that retry still happens, but once it is the last attempted action its own failure is handled by upgrade remediation, whose default is a rollback with no previous release to roll back to. The `remediation` blocks stay because two branches consult the retry count rather than the strategy and refuse to act once it is exhausted: an absent release reads the install block, an out-of-sync one reads upgrade. `retries: -1` keeps both open. The cost is that a genuine upgrade failure no longer rolls back either, it retries on the failed manifests instead. That is forced, since the controller cannot tell a retried install from a real upgrade, and what it replaces was an unbounded rollback-and-retry flap rather than a stable safety net: `upgrade.remediation.retries: -1` with the default rollback strategy already retried forever. This is the strategy and retry interval that `pkg/registry/apps/application/rest.go` and `internal/operator/package_reconciler.go` already set on every HelmRelease they generate, except that they leave `remediation` nil where this release keeps it. The other tenant addons this chart renders still carry the remediation-only shape; this PR changes the CNI alone, where the teardown removes the cluster's only CNI rather than churning a single component. ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): the tenant Cilium HelmRelease now retries a failed install instead of being uninstalled between attempts, so a failed install no longer takes the tenant cluster's only CNI away while it retries. The same strategy applies to the upgrade action, which it has to: a retried install runs as an upgrade. One consequence is operator-visible, a failed Cilium upgrade in a tenant cluster now retries the new manifests every 30s instead of rolling back to the previous release. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved Cilium installation and upgrade reliability by automatically retrying failed operations every 30 seconds. * Preserved deployed resources during retries and continued supporting unlimited remediation attempts. * **Tests** * Added coverage validating installation and upgrade retry behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
0c4c11c845 |
fix(kubernetes): retry the tenant CSI install instead of uninstalling it
Flux defaults a HelmRelease to the RemediateOnFailure strategy, whose install remediation is always an uninstall. A tenant CSI install that misses its timeout was therefore torn down between attempts, taking the CSI node plugin and the StorageClasses this release propagates with it, and the next attempt restarted the DaemonSet rollout from nothing instead of resuming one already under way. That teardown races the rollout it is meant to recover. In one observed run both DaemonSet pods were scheduled immediately and spent the whole budget pulling images; the slower node started its last container six seconds after the install timeout fired, and the uninstall removed it one second later. Retrying in place keeps that work. RetryOnFailure leaves the manifests applied and retries the failed release. Both actions carry it because the controller picks the active retry by .status.lastAttemptedReleaseAction and runs that retry as an upgrade: set on install alone, the retry happens but its own failure falls through to upgrade remediation, which rolls back by default with no previous release to roll back to. The cost is that a real upgrade failure now retries rather than rolls back, and that gives up no stable safety net: with retries: -1 and the default rollback strategy, upgrade remediation already rolled back and retried without end. The remediation blocks stay for a separate reason. An absent release consults install.remediation and an out-of-sync one consults upgrade, both by retry count rather than by strategy, so retries: -1 is what keeps those two branches acting. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
115d9bbaaf |
fix(installer): let the operator Deployment report Available honestly
At replicas: 1 with maxUnavailable: 1 the Deployment controller takes the minimum available replica count as 1 - 1 = 0, so the Available condition reports True with no running pod at all. That reading is what every consumer of the status sees: an alert on Available, a health dashboard, or a person reading kubectl get deploy all show green while the operator is dead. The replacement is not a tuned pair, it is the platform default written down. At replicas: 1 the default 25% / 25% resolves to maxSurge 1 and maxUnavailable 0, because the surge rounds up and the unavailability rounds down. So this change stops overriding the default rather than introducing a configuration of its own, and Kubernetes rejects maxUnavailable: 0 together with maxSurge: 0, so the surge could not have gone the other way regardless. The overridden pair was carried over from the earlier cozystack Deployment, where a darkhttpd sidecar published a host port and hostNetwork left a surge pod unschedulable on a single-node cluster. It arrived here in two steps: by the commit before this file was created the sidecar was already gone and no port was declared, yet the pair stayed, so it had outlived its own reason before it was ever copied. That is why looking for a justification in this file's history finds none. With no port declared anywhere in the pod there is no hostNetwork port conflict left, so the surge pod schedules beside the outgoing one; declaring one reintroduces exactly the scheduling failure the pair was added for. Not exercised on a live cluster in this rework. Stay on RollingUpdate rather than move to Recreate, which would fix the condition too, and by a shorter route: MaxUnavailable() reports 0 for any strategy that is not RollingUpdate, so the minimum available replica count becomes replicas - 0 = 1 and the condition is honest without any surge pod at all. The transition was not attempted, and the failure this project fixed in ouroboros 0.8.1 does not transfer to it. There the rollingUpdate block was the one the apiserver defaults onto a Deployment created without one, so it belonged to no applier and outlived the switch to Recreate. This Deployment has carried an explicit block since the commit that created it, and nothing applies it server-side, so whether Recreate would upgrade cleanly is an open question about the field-ownership history of the live object rather than a known rejection. What keeps RollingUpdate here is availability rather than apply semantics: maxUnavailable: 0 leaves the outgoing operator running when the replacement never starts, where Recreate deletes it first and leaves none. Editing the numbers inside the block is upgrade-safe from every state the field can already be in. The surge introduces new windows, and leader election covers less of them than it looks. The operator runs four privileged install steps before mgr.Start(), and the lease is only acquired inside mgr.Start(): the CRD install, the Flux install, the platform source resource and the platform PackageSource. All four therefore run in the incoming pod outside leader election entirely. What the lease does cover is the reconcile loops, the part that would otherwise conflict continuously instead of once at startup. That leaves three seams, and only the first two happen on a healthy upgrade. The routine one is the incoming pod's install steps against the outgoing pod's live reconcile loops, since the outgoing pod ran its own installs at its startup and has been inside mgr.Start() ever since. Those do not contend for the same fields: the installs write as field manager cozystack-operator, while the reconcilers write as cozystack-packagesource-controller and cozystack-package-controller and go mostly to status subresources. Different owners, different objects, so this seam is safe by separation rather than by any shared manager. The second is version skew rather than field contention, and maxSurge: 1 widens it rather than opening it. Under maxSurge: 0 the surge budget counts desired replicas rather than live pods, so the incoming pod is created while the outgoing one may still be Terminating - the overlap existed. What it did not have was a live counterparty: scale-down sends SIGTERM before the new pod is created, and that cancels the manager context, so the old loops were already unwinding. Here the outgoing pod is not signalled until the incoming one is available, so the overlap runs against a fully live reconciler for a termination grace period instead of a vestigial one. The incoming pod reports Ready the instant its container starts, because there is no readiness probe, so its CRD install runs while the outgoing binary's reconcile loops are still live and now writing against a schema the incoming pod has already moved. Old code against a new schema, which no field manager separates. It is bounded rather than benign: the outgoing pod exits on SIGTERM within seconds, a rejected write is transient and the next reconcile repairs it, and no object is left corrupt. Widening the skew, by adding a probe that delays Ready or by a migration the old code cannot round-trip, is what would make this seam matter. The third seam is install against install, and it needs the outgoing pod to be restarting, since a crashlooping pod re-runs its four steps on every restart, which is exactly the broken operator this gate exists to catch. Three of the four steps use server-side apply under the same field manager, so they never conflict and resolve last-writer-wins whenever the two pods carry different manifests. That ordering matters in one direction, and it is accepted here rather than guarded against: the CRD install and the Flux install both apply with Force: true, so an outgoing pod that writes last silently overwrites what the incoming pod has just applied, with no conflict and no error to exit on. The incoming pod is inside mgr.Start() by then and will not apply again until it restarts, so that revert of the upgrade's install phase persists rather than healing itself. Reaching it needs a restart landing in the seconds between the incoming pod's installs and the outgoing pod's SIGTERM. installPlatformSourceResource is the remaining exception, a read-modify-write of a Get followed by Create or by Update with the observed resourceVersion, so two pods running it at once can collide on AlreadyExists or on a conflict; that failure is loud and self-healing, since any error there exits the process and the pod restarts, and the object is not left corrupt. Pin the strategy and all four preconditions the surge makes load-bearing in a unit test: no container or initContainer declares a port, both listener arguments stay disabled, leader election stays on, and no probe uses httpGet. They fail differently and none is visible to the others, since a declared port stops the scheduler, a bound one only fails once the incoming process starts, and a second manager without a lease fails nothing at all until it writes. The httpGet precondition follows from the two disabled listeners leaving no port to target, and it is the worst of the four to get wrong: the other three wedge an upgrade, while a livenessProbe against a disabled listener crashloops the pod on every cluster, including a fresh install where no surge happens. Only the httpGet field is forbidden rather than the probes themselves, because an exec probe stays legitimate. All four are checked against all three variants, because the template branches on variant inside the container spec and a port added under one branch renders for that variant alone. The strategy type is pinned alongside its two values, since maxSurge and maxUnavailable say nothing on their own once the type is no longer RollingUpdate. The listener preconditions are pinned less tightly than the other two, and the test says so rather than implying coverage it does not have. A membership assert has no notion of position while Go's flag package keeps the last occurrence, so a duplicate after the disabled one wins and the assert still passes, while a duplicate before it loses and the assert passes just the same - the test cannot tell those two apart. Moving the disabled flag last fixes the process without making the assert catch anything. Rejecting the two addresses this binary defaults to closes the shape a values-gated block to enable metrics would take, and nothing available here closes an arbitrary address, since helm-unittest cannot express that a flag appears at most once. Note in hack/helm-unit-tests.sh that the loop covers packages/core, which its comment omitted. The suite added here lives there, so a later edit reconciling the loop with the stale comment would have silently stopped running it. Split the rationale by audience rather than keeping it all in shipped comments. What an operator applying the rendered manifest has to respect stays in plain comments and travels into the install artifacts: that the pair is the default, why maxUnavailable must stay 0, that no port may be declared or bound, that no probe may use httpGet, and that leader election has to stay on. Most of what stays in the template comment is what the chart's own suite pins, so it cannot drift out of step with the chart without a test going red: the four preconditions and which assert covers each, and why the variant loop is needed. Three things there are deliberate exceptions, kept because anyone editing the strategy has to know them and no assert can carry them - that the webhook server configured on 9443 is not a port binder, since controller-runtime adds it to the runnables only inside GetWebhookServer() and this binary never calls it; that two overlaps between the pods sit outside the lease; and why RollingUpdate is kept rather than Recreate. Those name what exists. The seam analysis above, which is how each one works, deliberately stays here instead. It describes another file's internals, nothing pins it, and a commit message is fixed in time, so it cannot quietly fall out of date the way a comment can. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
cd57ee6678 |
test(multus): anchor the master-CNI assertion to the top-level key
The check that multus is pointed at the cilium conflist matched its JSON fragment anywhere in the rendered daemon config. The same text nested inside any other key satisfies it, so a config whose effective multusMasterCNI names a different conflist passes. Multus uses that value as written when it is non-empty -- it delegates to whatever is named, so a wrong-but-existing name silently routes through the other CNI instead of cilium, which is the state this assertion exists to prevent. Auto-discovery happens only when the value is empty, and a name that does not exist fails manager startup rather than falling back. Anchor it to the top level, where the template writes its keys. That pins the position rather than parsing the document; deciding which key is effective needs a JSON parser, which helm-unittest does not have. Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
7739d1e294 |
feat(multus)!: stage reference CNI plugins from the multus image
Multus-backed NetworkAttachmentDefinitions delegate to the upstream containernetworking reference plugins (bridge, macvlan, ipvlan, ...), and nothing guarantees a node has them in the directory Multus searches. On k3s they live under /var/lib/rancher/k3s/data/cni while Multus and Cilium install into /opt/cni/bin, so a bridge-type NAD fails there with `failed to find plugin "bridge" in path [/opt/cni/bin]`. Bundle 14 of the release's plugins into the multus image and add an install-cni-plugins init container that installs them into the host /opt/cni/bin, mirroring the existing install-multus-binary container. loopback, dhcp, tap and dummy are left out; the Dockerfile records why. Gate it on a new chart value, stageCniPlugins, resolved per bundle and on for both. Talos bundles only bridge, firewall, flannel, host-local, loopback and portmap, so the other ten are missing there exactly as on generic Linux; the four that overlap are rewritten with the same upstream version, v1.9.1 on both sides, rather than an older one. A marker in the image Dockerfile records the Talos release that pin was compared against, and a test holds it to the release the platform ships. multus therefore moves out of system.common-packages into the two variant branches, the way linstor already does for talos.enabled. networking.stageCniPlugins overrides the variant default in either direction. Where staging is on the install is unconditional: a plugin is replaced on every pod recreation, not only when the destination is missing. A destination that is already a directory is reported and left alone, since replacing it is not something this can do safely. Turning the value off afterwards does not undo it; nothing here removes or restores a plugin. The container is not privileged and takes no mount propagation. It runs in the spc_t SELinux domain, without which an enforcing node refuses the write. Each plugin is installed under a temporary name and swapped in with a rename: /opt/cni/bin is shared with cilium and kube-ovn. The plugin tarball is checksummed before extraction. A plugin that cannot be installed is reported on stderr and to /dev/termination-log, and the container still exits 0: Multus stays the node's primary CNI and nothing removes its config, so an init container that never completes would leave the shim with no daemon behind it. The manifest change lives in customize-deployment.patch so it survives make update. BREAKING CHANGE: enabling stageCniPlugins replaces reference CNI plugins in /opt/cni/bin on existing nodes, Talos included -- four of the names it writes are ones the Talos node image also carries. Opt out before upgrading: declining afterwards does not restore what was replaced. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
80a2090444 |
fix(kubernetes): retry the tenant CNI install instead of uninstalling it
The tenant cilium HelmRelease carried no install strategy, so Flux applied its default, RemediateOnFailure, which uninstalls the release between attempts. Cilium is the tenant cluster's only CNI, so every cycle takes it away and gives it back when the next install applies the manifests again, and retries: -1 lets that repeat without end. What sets a cycle off is an install that outruns its budget, which can happen while the tenant still has no registered node, since the operator Deployment cannot become Available until one exists. Remediation neither causes that delay nor is fixed by this change; what it adds is a repeated teardown of the CNI on top of a delay that resolves itself once the node arrives. This removes the amplifier, not the cause. RetryOnFailure keeps the applied manifests in place and retries a failed install as an upgrade, so the rollout completes on its own once a worker registers. Both actions carry the strategy because the retry of a failed install runs as an upgrade, which would otherwise fall back to upgrade remediation. It is the strategy and retry interval cozystack-api and cozystack-operator already set on every HelmRelease they generate, except that they leave remediation nil where this release keeps it. The other tenant addons the chart renders still carry the remediation-only shape; this one is where the teardown removes the cluster's only CNI rather than churning a single component. The cost is that a genuine upgrade failure no longer rolls back either, it retries on the failed manifests instead. That is forced, since the controller cannot tell a retried install from a real upgrade, and what it replaces was an unbounded rollback-and-retry flap rather than a stable safety net. The remediation blocks stay, since they cover the branches no strategy reaches: an absent release, which consults the install block, and an out-of-sync one, which consults upgrade. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
myasnikovdaniil
|
ee26a50f0e |
fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix (#3510)
## What this PR does Raises the CloudNativePG operator image **and its CRDs together** to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / cloudnative-pg#9981). Symptom: after a simultaneous `resources` + `size` change on a single-instance Postgres, the operator deletes the sole primary Pod, classifies the PVC as `resizing`, and never recreates the Pod — the cluster wedges with zero instances and the disk FS resize never completes. Upstream publishes a CloudNativePG chart only per minor (`.0`/`.1`), so there is no 1.28.2 chart on the 1.28 line. Following the pattern of #3526 / #3528, `patches/cloudnative-pg-1.28.2.patch` (applied by `make update`) raises the vendored chart's `appVersion` and CRDs from 1.28.1 to 1.28.2 in lockstep. The operator image follows `appVersion`, so there is no `image.tag` pin (keeping operator and CRDs aligned and avoiding #3479). Verified #3479-safe: there are no CRD `status` field changes between 1.28.1 and 1.28.2 — only the `extensions` spec grows. Backportable to release-1.6 as-is (same 0.27.1 base chart). ### Release note ```release-note fix(postgres-operator): align CloudNativePG operator and CRDs to 1.28.2, fixing the PVC resize-deadlock that wedged single-instance PostgreSQL clusters after a simultaneous resources+size change ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Added support for PostgreSQL extension configuration in catalog and cluster schemas. - Made extension image settings optional where appropriate. - **Bug Fixes** - Updated CloudNativePG to version 1.28.2, including the PVC resize deadlock fix. - Kept the operator image and CRDs aligned at the same version. - **Documentation** - Clarified versioning and chart update guidance. - **Tests** - Added checks confirming operator image and CRD version alignment. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Alexey Artamonov
|
bcb36262ef |
fix(postgres-operator): align CNPG operator and CRDs to 1.28.2 for PVC resize-deadlock fix
Raises the CloudNativePG operator image and its CRDs together to 1.28.2, which carries the PVC resize-deadlock fix (cloudnative-pg#9980 / #9981): after a simultaneous resources+size change the operator deletes the sole primary Pod, leaves the PVC in the resizing class, and never recreates the Pod, wedging the cluster. Upstream publishes a chart only per minor .0/.1, so there is no 1.28.2 chart on the 1.28 line. Following #3526/#3528, patches/cloudnative-pg-1.28.2.patch (applied by make update) raises the vendored chart's appVersion and CRDs from 1.28.1 to 1.28.2 in lockstep; the operator image follows appVersion, so no image.tag pin. Verified #3479-safe: no CRD status field changes between 1.28.1 and 1.28.2 (only the extensions spec grows). Backportable to release-1.6. Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
myasnikovdaniil
|
a7149d11a3 |
chore(etcd-operator): bump etcd-operator to v0.5.4 (#3529)
## What this PR does Bumps the cozystack etcd-operator packages from **v0.5.3 to v0.5.4**. `v0.5.4` is a controller bug-fix release — no API, RBAC or values changes: - fix(controllers): derive `--initial-cluster-state` from phase, not from the seed ([cozystack/etcd-operator#355](https://github.com/cozystack/etcd-operator/pull/355)) - fix(controllers): stop exempting the bootstrap seed from self-heal ([cozystack/etcd-operator#354](https://github.com/cozystack/etcd-operator/pull/354)) - fix(controllers): extend crash-loop self-heal to memory members ([cozystack/etcd-operator#352](https://github.com/cozystack/etcd-operator/pull/352)) - fix(controllers): switch the PDB from `maxUnavailable` to `minAvailable` ([cozystack/etcd-operator#351](https://github.com/cozystack/etcd-operator/pull/351)) Changes in this repo: - `packages/system/etcd-operator/Chart.yaml` — `appVersion: v0.5.3 → v0.5.4`. The manager image tag defaults to `.Chart.AppVersion` (`values.yaml` keeps `tag: ""`), so this reimages the controller to `ghcr.io/cozystack/etcd-operator:v0.5.4`. - `packages/system/etcd-operator/Makefile`, `packages/system/etcd-operator-crds/Makefile` — `ETCD_OPERATOR_REF: v0.5.3 → v0.5.4`. - `packages/system/etcd-operator-crds/templates/etcdmembers.yaml` — re-vendored at v0.5.4 via `make update`. Description-only change to the `/scale` `replicas`/`selector` field docs, tracking the PDB `minAvailable` fix. `etcdclusters` and `etcdsnapshots` are byte-identical to v0.5.3. - `templates/rbac.yaml` intentionally left as-is: `manager-role-rules.yaml` is byte-identical between v0.5.3 and v0.5.4. **Upgrade path (PDB switch, cozystack/etcd-operator#351).** This is the one change that rewrites live objects: the operator moves each EtcdCluster's PodDisruptionBudget from `maxUnavailable` to `minAvailable`. Setting both fields is invalid, but upstream `reconcilePDB` handles the migration — it treats a surviving pre-migration `maxUnavailable` as divergence and explicitly clears it (`MaxUnavailable = nil`) before writing `MinAvailable`, so existing clusters are reconciled cleanly on upgrade rather than wedging their PDB. Verified locally: `helm lint` and `helm template` pass for both packages; rendered manager image resolves to `ghcr.io/cozystack/etcd-operator:v0.5.4`. These `packages/system/*` packages have no `generate:` target and no `values.schema.json`, so there is nothing for `make generate` to regenerate. ### Screenshots Not applicable — no UI changes. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` file-by-file against the diff: - The diff touches only `packages/system/etcd-operator*` — no `packages/apps/**` or `packages/extra/**` add/rename/remove, no `packages/core/platform` or `installer` values, no Talos bump, no asset-name or dev-tooling change → **website / ansible-cozystack** not reached. - The CRD edit is description-only, inside etcd-operator's own `etcd-operator.cozystack.io` CRDs — not the provider's hand-typed `Package`/`Plan`/`RestoreJob` types, and no new managed app → **terraform-provider-cozystack** not reached. - No `hack/` change, no `packages/system/<name>-rd/cozyrds/**`, no `ApplicationDefinition` CRD / `chartRef.kind` enum change, no `cozyhr`/`package.mk` contract change, no telemetry-metric or proxy-label rename → **ccp / talm / cozyhr / cozy-proxy / telemetry-server / examples / external-apps-example** not reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note chore(etcd-operator): bump etcd-operator to v0.5.4 (controller bug-fixes: PDB switched to minAvailable, crash-loop self-heal extended to memory members, --initial-cluster-state derived from phase, seed no longer exempt from self-heal) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **Updates** - Updated the etcd operator to version v0.5.4. - Clarified scale-related resource descriptions, including replica counts, selectors, and disruption budget behavior. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrey Kolkov
|
22c1df8b94 |
test(etcd-operator): pin deployment unittest to v0.5.4
The manager image tag derives from .Chart.AppVersion, so bumping appVersion to v0.5.4 without updating the two image literals in the deployment unittest broke the suite (helm-unittest gates the PR via `make unit-tests`). Update both asserts (manager container image and the agent OPERATOR_IMAGE env) to ghcr.io/cozystack/etcd-operator:v0.5.4. Suite is green again (18/18). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Andrey Kolkov
|
19f969b2f8 |
chore(etcd-operator): bump etcd-operator to v0.5.4
Bump the cozystack etcd-operator packages from v0.5.3 to v0.5.4. The release is a controller bug-fix set with no API, RBAC or values changes: - fix(controllers): derive --initial-cluster-state from phase, not from the seed (cozystack/etcd-operator#355) - fix(controllers): stop exempting the bootstrap seed from self-heal (cozystack/etcd-operator#354) - fix(controllers): extend crash-loop self-heal to memory members (cozystack/etcd-operator#352) - fix(controllers): switch the PDB from maxUnavailable to minAvailable (cozystack/etcd-operator#351) Adaptations: - etcd-operator/Chart.yaml: appVersion v0.5.3 -> v0.5.4 (the manager image tag defaults to .Chart.AppVersion, so this reimages the controller). - etcd-operator/Makefile, etcd-operator-crds/Makefile: ETCD_OPERATOR_REF v0.5.3 -> v0.5.4. - etcd-operator-crds/templates/etcdmembers.yaml: re-vendored at v0.5.4 via `make update`; description-only change to the /scale replicas/selector fields tracking the PDB minAvailable fix. etcdclusters/etcdsnapshots unchanged. - templates/rbac.yaml left as-is: manager-role-rules.yaml is byte-identical between v0.5.3 and v0.5.4. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrey Kolkov <androndo@gmail.com> |
||
|
Myasnikov Daniil
|
4e9c5ce8ca |
fix(kubernetes): render the talos-reconcile Job for the default md0 group
The Job in templates/talos/talos-reconcile-job.yaml is the only producer of the TalosConfigTemplate that each worker MachineDeployment names in spec.template.spec.bootstrap.configRef; cluster.yaml renders the reference but deliberately not the object. Its loop read .Values.nodeGroups directly, while the MachineDeployment loop reads the effective set through the kubernetes.nodeGroups helper. For a cluster that supplies no nodeGroups the two disagree: the helper's else-branch emits the built-in md0 group, so the md0 MachineDeployment renders, but the raw map is empty and no Job renders at all. Every Machine the cluster-autoscaler adds to md0 then blocks indefinitely on a TalosConfigTemplate that nothing will ever create, and the KamajiControlPlane spec.network.certSANs patch the same Job performs is skipped too. Range over the helper so the Job set tracks the MachineDeployment set exactly. No change for a cluster that declares its own groups, and md0 stays removable: the helper's if-branch keeps a user-supplied map authoritative. The gap survived because every helm-unittest fixture and e2e suite declares md0 explicitly. tests/nodegroups_default_test.yaml does render the empty-nodeGroups case but lists only templates/cluster.yaml, so it never looked at the Job. The new suite pins both halves of the helper contract and fails without this fix. Fixes #3504 Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
3f967ec7e0 |
fix(platform): restore the isp-hosted dependency closure (#3391)
## Problem Fixes #3376. The `isp-hosted` preset shipped `bundles.system.enabled: false` while `paas` and `naas` were enabled, so it enabled an app catalog whose dependency closure it simultaneously disabled. Every emitted Package sat in `DependenciesNotReady`, the `*-rd` HelmReleases failed with `no matches for kind "ApplicationDefinition" in version "cozystack.io/v1alpha1"`, and the `cozystack-platform` HelmRelease never went Ready. Rendering the preset at `origin/main` shows the scale: **24 emitted Packages, 24 of them blocked**, by 6 missing dependencies — two more than the issue lists (`cert-manager` and `prometheus-operator-crds` are also absent). `isp-full` renders 75 Packages with a fully satisfied closure, which is a clean control. ## Root cause `templates/bundles/system.yaml` is gated end to end on `bundles.system.enabled`, but it carries two separable things: a variant-selected **data-plane head** (kubeovn/cilium/linstor on `isp-full`, a noop networking Package on `isp-hosted`) and a variant-agnostic **control-plane body** — `cozystack-engine`, the sole source of the `application-definition-crd` component, plus `cert-manager`, `prometheus-operator-crds`, `postgres-operator` and `objectstorage-controller`. "Hosted" means the host already provides CNI, storage and virtualisation, so only the data-plane head should be dropped. Disabling the whole bundle also deleted the control-plane body that every variant needs. Three things show the disable was the mistake rather than the intent: the preset still sets `variant: "isp-hosted"`; `system.yaml` contains an `isp-hosted` branch (noop networking plus default scheduler) that could never render; and `iaas.yaml` already refuses to render on this variant, encoding the same "system bundle minus iaas" contract. The preset has never worked — it was born this way in `2d022e38e` (January 2026), the refactor that replaced `bundles/paas-hosted.yaml`, whose contents confirm hosted was always meant to ship the engine, `cert-manager` and `prometheus-operator-crds`. ## Changes 1. **`values-isp-hosted.yaml`** — enable the system bundle. Hosted now emits 53 Packages with every `dependsOn` satisfied, installing 59 components against the 60 of the pre-refactor `paas-hosted` bundle, with no LINSTOR, KubeVirt, CAPI or Kamaji leakage. 2. **Move `cozystack.bucket-application`** from the iaas bundle into the system bundle. Its own `dependsOn` closure is `networking` + `objectstorage-controller` + `cozystack-engine`, none of them iaas, while `cozystack.backupstrategy-controller` — a *default* package of the system bundle — hard-depends on it. Any system-on/iaas-off configuration therefore wedged `backupstrategy-controller`; both `isp-full` presets masked this by enabling iaas. This is a second, independent bug not reported in the issue. 3. **Guards** — `paas`, `naas` and `iaas` now also require `bundles.system.enabled`. The pre-existing checks looked only at `bundles.system.variant`, which keeps its value when the bundle is disabled, so they passed on exactly this configuration. `iaas` had the identical hole (12 Packages, 6 unsatisfied deps). This turns a silent multi-hour `DependenciesNotReady` hang into a render-time failure that names the remedy. 4. **Two helm-unittest suites** — one pinning the dependency closure the catalog needs, one pinning the guards. ## Verification - All four operator-registered variants render dependency-closed: `default` 0 Packages, `isp-hosted` 53, `isp-full` 75, `isp-full-generic` 75. The two `isp-full` variants are byte-identical before and after, so the bucket move is a no-op for them. - Hosted is a strict subset of full with zero data-plane leakage, and its exclusions match the pre-refactor hosted bundle. - 23 suites / 97 tests pass in `packages/core/platform`; repo-wide `hack/helm-unit-tests.sh` is green. - Every change was **mutation-tested** — reverting each one in turn makes the suite fail, so the tests are not vacuous. - The workaround documented in #3376 (`paas`/`naas` disabled) still renders cleanly; the new guards do not fire on it. - Hosted runtime spot-checks: `cozystack-api` is a Deployment with only a *preferred* control-plane affinity plus blanket tolerations, and `cozystack-scheduler` uses its `default` variant (plain Deployment, no LINSTOR extender) — both schedule fine on a cluster with no control-plane nodes. ## What this does not fix This restores the dependency closure. It is **not** by itself sufficient for a fully converged hosted install. Review surfaced further hosted-unaware defaults that the dead bundle had been masking, filed separately: - #3387 — `apps/tenant` renders 14 `cilium.io/v2` policies with no `.Capabilities` guard, so `tenant-root` fails on a non-Cilium cluster. This is the blocker for actual hosted convergence, and it needs a deliberate decision rather than a reflexive capability guard, because those policies implement tenant network isolation. - #3388 — `extra/monitoring` defaults `logsStorages` to the LINSTOR-only `replicated` StorageClass. - #3389 — `bundles.disabledPackages` can still break the closure with no render-time error; the guards here cover the bundle flags only. - #3390 — nothing in CI exercises `isp-hosted`, which is why all of the above accumulated undetected. ## Upgrade note Existing `*-application` Package CRs carry `helm.sh/resource-policy: keep`. An operator who applied the #3376 workaround but did not also delete the retained Package CRs (as that issue instructs) will find them install once this change makes their dependencies Ready, since the reconciler re-enqueues dependents on readiness. Deleting the unwanted Packages before upgrading avoids that. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added fail-fast validation that requires the **System bundle** when enabling **IaaS**, **NaaS**, or **PaaS** to prevent incomplete catalogs. * Updated bundle composition so the **bucket application** is provided by the **System bundle**. * Enabled the **System bundle** by default for the `isp-hosted` variant. * **Bug Fixes** * Improved error messaging to clearly indicate the required bundle/variant configuration. * **Tests** * Added Helm unittest coverage for rendering-gate failures, dependency/package expectations, and hosted data-plane exclusions. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Timofei Larkin
|
e5d4940338 |
chore(codeowners): don't gate per-app generated files on architect review
api/apps/v1alpha1/*/types.go and packages/system/*/cozyrds/*.yaml are entirely produced by `make generate` from each app's own values.yaml, but the /api/ codeowner rule pulled every app's regenerated types.go into an architect-only review, regardless of which area actually changed. Correctness there is already enforced by the pre-commit make-generate CI hooks, not by human review, so list no owner for these generated paths and let that check carry the weight instead. Also fixes a stale api/.gitattributes pattern that never matched any real file (zz_generated_deepcopy.go vs the actual zz_generated.deepcopy.go), and adds a matching linguist-generated attribute for the cozyrds ApplicationDefinition manifests, so both collapse in the PR diff view like the Go files already do. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
myasnikovdaniil
|
10554f9dce |
[cozystack-scheduler] Publish SchedulingClass demo (#2252)
## What this PR does Publish the scripts used for running the Cozystack scheduler demo at the community meetup on 19.03.2026. ### Release note ```release-note [cozystack-scheduler] Add demo scripts to the cozystack-scheduler system packages. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a complete set of interactive Bash demo scripts covering scheduling classes: creating scheduling classes and tenants, colocating pods on a single node, spreading replicas evenly, enforcing one-pod-per-node anti-affinity, and pinning workloads to a specific node. * Included guided scaling steps plus cleanup and teardown scripts to remove demo resources. * **Documentation** * Introduced a shared demo helper with colored, step-by-step terminal logging and Kubernetes polling to wait for pods and Redis readiness, improving clarity and consistency across demos. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Timofei Larkin
|
337bb4933e |
[cozystack-scheduler] Publish SchedulingClass demo
## What this PR does Publish the scripts used for running the Cozystack scheduler demo at the community meetup on 19.03.2026. ### Release note ```release-note [cozystack-scheduler] Add demo scripts to the cozystack-scheduler system packages. ``` Signed-off-by: Timofei Larkin <lllamnyp@gmail.com> |
||
|
Andrei Kvapil
|
3dc0f56477 |
feat(kube-ovn): build the chart and image in-tree, retiring the wrapper repo (#3460)
## What this PR does Brings the `kubeovn-chart` wrapper into the tree, so `cozystack/kubeovn-chart` can be archived. Until now `packages/system/kubeovn` pulled a chart tarball from the latest `cozystack/kubeovn-chart` tag at `make update` time, and that repository also built and pushed the `kubeovn` image. The wrapper existed only to cache the upstream chart; with the current caching approach that reason is gone, and the split cost a second repository, a second release step, and a package whose provenance a reader cannot follow without leaving the tree. The package is now shaped like `packages/system/cilium`: - `update` pulls the upstream `kubeovn/kube-ovn` chart directly and applies `patches/mtu.diff`, then syncs the version into the Dockerfile. The MTU patch already carried monorepo-relative paths, so it applies unchanged. - `image` builds `images/kubeovn` (carrying the existing `fix-keepvmip-portgroup` patch) and pins the digest into `values.yaml`, using the shared `image-tags` / `cache-args` helpers. It also carries `cozystack/kubeovn-chart#5`, which never got a tag: `OVN_IPSEC_KEY_DIR` is pinned under `/var/lib`. The kube-ovn default sits in `/etc/origin`, which a Talos node cannot create, so enabling IPsec left the node without ESP security associations. ### The nightly host rewrite had to be fixed first Moving the image build here activates a gap recorded in `docs/agents/image-refs.md`. The nightly host rewrite is a literal `<src-registry>/` substring replace, and kubeovn keeps its host in `global.registry.address` with the bare repository in a sibling key — no trailing slash, no match. While the image came from a separate repository the baked reference carried the public host and the ownership filter skipped it; once CI builds the image here it stamps the build-registry host into that key, and the result is an image mirrored to the public registry while the published tree keeps pointing at the private build registry. The second commit adds an end-of-line-anchored expression that fires only when the source registry is the complete scalar value, so a contiguous reference — which continues past the host — is still handled by the first expression and the two can never both fire on one reference. `keycloak-operator` splits at a different boundary and stays unfixed; that is documented, and dormant while no CI path rebuilds it. Verified: `hack/nightly-mirror_test.bats` passes with the new case and the case fails without the fix; `promote-retag` and `promote-rewrite-tags` suites unaffected; `helm template` and `helm unittest` on the package pass. The pre-existing `helm lint` complaint about a missing `kube-ovn` dependency reproduces on `main` and is left alone. ### Follow-ups this PR does not do - `cozystack/kubeovn-chart` should be archived once this merges, with a pointer here. Its open PR #5 is carried by this change and needs no separate landing. - `cozystack/website` links the Kube-OVN license at `cozystack/kubeovn-chart` in `operations/configuration/licenses.md` and `guides/platform-stack/_index.md` (all versions, including `next`). Archived repositories stay readable so nothing breaks, but the link should move upstream or in-tree. **No follow-up has been opened yet** — flagging rather than ticking the box. ### Screenshots N/A — build and packaging change, no user-facing or UI impact. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: Walked the trigger map against the diff. The website row matches — not through the app-list or platform-component triggers, but because two of its pages link the repository being archived. Left unticked because no follow-up exists there yet; a human should decide whether the license link moves to `kubeovn/kube-ovn` or to this package. No other row matches: the change touches no `packages/apps/` or `packages/extra/` package, no `values.schema.json`, no `ApplicationDefinition`, no platform variant, and no node prerequisite. ### Release note ```release-note feat(kube-ovn): the Kube-OVN chart and image are now built in this repository instead of cozystack/kubeovn-chart. OVN_IPSEC_KEY_DIR now defaults to /var/lib/ovs_ipsec_keys, so enabling IPsec on Talos nodes no longer leaves them without ESP security associations. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit - **New Features** - Updated Kube-OVN image build to use upstream chart metadata and keep registry/version alignment during `update`. - Added conditional MTU support for the OVN CNI DS via chart values. - Introduced `OVN_IPSEC_KEY_DIR` to standardize the OVN IPsec keys directory. - **Bug Fixes** - Preserved VM networking port-group assignments during live migration when an alive virt-launcher sibling pod exists. - Improved nightly mirror host rewriting for split and standalone registry host formats. - **Tests** - Added coverage for the end-of-line “whole scalar value” host rewrite case. - **Documentation** - Refreshed “Known gaps” guidance for nightly mirror host rewriting behavior and limitations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
5d420475f4 |
refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows) (#3454)
## What this PR does
The E2E job's `if: always()` cleanup ran `rm -rf /tmp/$SANDBOX_NAME`
with no check that `SANDBOX_NAME` was ever set. `Set sandbox ID` carries
no condition, so it inherits the implicit `success()` every
unconditional step gets: any failure ahead of it — a conflicting
`pr.patch`, a `download-artifact` miss, `Checkout code`, the
release-path `curl` — or a cancellation that reaches the job after it
has started (the job condition is `!cancelled()`) skips it, and
`SANDBOX_NAME` is never written to `$GITHUB_ENV`. The `if: always()`
steps then run anyway and interpolate the empty value; `Remove
workspace` is the destructive one, expanding to `rm -rf /tmp/` and
failing the cleanup step against root-owned entries in the runner's
`/tmp`. This is not a "re-run skips the setup" bug: a re-run restarts
the job at step one and re-runs the unconditional `Set sandbox ID` — the
trigger is incomplete initialization, by any of the routes above.
The `/tmp/$SANDBOX_NAME` workspace is a vestige of the retired multi-job
self-hosted-runner design, where several `runs-on: [self-hosted]` jobs
shared no state and rendezvoused through a `sha256(REPO:WORKFLOW:REF)`
key, with a per-run `cp -r` of the checkout into `/tmp`. Since CI moved
onto the ephemeral pool (`runs-on: oracle-vm-32cpu-128gb-x86-64`) it is
a single job: there is no cross-job rendezvous and the checkout is never
reused across runs, so copying it into `/tmp` and removing it afterwards
is dead weight. (An earlier review justified the change by "wiping the
entire `/tmp` of the self-hosted runner" — that describes the topology
we retired; the hazard is real, but not for that reason.)
So this drops the workspace rather than guarding the `rm`. Both the PR
(`pull-requests.yaml`) and nightly (`nightly.yaml`) E2E jobs now run in
`$GITHUB_WORKSPACE`: the `Prepare workspace` and `Remove workspace`
steps are gone, the `cd /tmp/$SANDBOX_NAME` prefixes are dropped, and
the upload paths point at `_out/…` instead of
`/tmp/${SANDBOX_NAME}/_out/…`. That takes the empty-value expansion out
of the `cd`s, the stray `mkdir`, the upload paths and the destructive
`rm` in one move, across both workflows rather than one step of many.
`Set sandbox ID` stays: the job still reads `$SANDBOX_NAME` directly
(failure diagnostics, the chainsaw `docker cp`, teardown, and every
`make SANDBOX_NAME=…` argument) — the container needs a name, just not a
directory.
It also removes the dead code the single-runner retirement left in
`packages/core/testing/Makefile`: the world-writable `mkdir -p
/tmp/${SANDBOX_NAME}` + `chmod 777` in the `apply` target (they backed a
`-v /tmp/${SANDBOX_NAME}:/workspace/hosttmp` bind mount that was later
removed — nothing under `images/e2e-sandbox/` references
`/workspace/hosttmp`), and the `-e SANDBOX_NAME=${SANDBOX_NAME}` passed
into the container (nothing under `images/e2e-sandbox/` reads it). Every
other use of `SANDBOX_NAME` — the container name and the `make`
arguments — is kept.
Note this cannot make a job with an empty `SANDBOX_NAME` go green: such
a job already failed initialization or was cancelled, so its conclusion
is fixed before cleanup runs. It removes a second, destructive failure
the old cleanup piled on top, and it does not change whether re-runs
work in general — they do.
The diagnosis of the real trigger, the corrected blast radius, and the
scope across both workflows plus the Makefile are credited to the
maintainer review on this PR.
### Evidence
Run [30067779342 attempt
3](https://github.com/cozystack/cozystack/actions/runs/30067779342/attempts/3)
exhibits the mechanism: `Download Talos image (regular PR)` failed on
`GetSignedArtifactURL` with a 404, which skipped `Set sandbox ID`, and
the `always()` steps then ran on an empty value — `Collect chainsaw
report` created `/tmp/_out`, the uploads resolved to `/tmp//_out/…`, and
`Remove workspace` did:
```
rm: cannot remove '/tmp/systemd-private-…-systemd-timedated.service-1oADZP': Operation not permitted
rm: cannot remove '/tmp/.X11-unix': Operation not permitted
##[error]Process completed with exit code 1.
```
The step ran unprivileged in `/home/ubuntu/_work/…`, so all it could hit
was `Operation not permitted` on root-owned entries in the sticky `/tmp`
— it deleted nothing. That run is a later, independent occurrence (it
started well after this PR was opened), not this change's motivating
case. The 404 that triggered it is a separate, non-deterministic problem
— the artifact listing succeeded and only the signed-URL call failed
~190 ms in, `download-artifact` treats 404 as non-retryable, and attempt
2 of the same run fetched the artifact fine — and it deserves its own
issue rather than being folded in here.
### Screenshots
Not applicable — CI workflow change only.
### Downstream repositories
The change is confined to this repo's E2E CI
(`.github/workflows/pull-requests.yaml`,
`.github/workflows/nightly.yaml`) and its testing Makefile
(`packages/core/testing/Makefile`). No downstream repository consumes
it.
- [x] No downstream repository is affected by this change
### Release note
```release-note
refactor(ci): drop the vestigial /tmp/$SANDBOX_NAME workspace from the E2E PR and nightly jobs — they now run in $GITHUB_WORKSPACE, removing an unguarded `rm -rf /tmp/$SANDBOX_NAME` cleanup that failed the job on cleanup when SANDBOX_NAME was empty
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved end-to-end test environment setup and sandbox cleanup.
* Updated test reporting and artifact collection for more reliable
results.
* Ensured OpenAPI and full-suite tests run from the correct prepared
environment.
* **Chores**
* Simplified container and workflow configuration for sandbox-based
testing.
* Clarified checkout token handling in nightly workflows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
93338cc434 |
refactor(computeplane): drop the in-cluster Flux pin
The fluxcd addon pin was a holdover from the #3150-era model where app charts rendered nested HelmReleases into the target cluster and needed an in-cluster helm-controller. Consumers now keep their releases on the management cluster and remote-apply via spec.kubeConfig, so the management Flux resolves charts and applies manifests — nothing inside the ComputePlane reconciles HelmReleases. This also decouples the module from the FluxCD addon removal (#3379). certManager and ingressNginx stay pinned: Certificate and Ingress objects land inside the cluster and need their controllers there. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
2ff9c30d04 |
docs(kubernetes): note the etcd gate in the Proxied Service comment
The Proxied ingress Service is gated on _namespace.etcd as well as the addon toggle and exposeMethod, but the comment above it still described only the addon+exposeMethod gate — comment drift from the etcd-gate change. State both gates so the comment matches the condition it sits on. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8fc8d23dfd |
fix(computeplane): address review round three
Gate the Proxied ingress Service on _namespace.etcd like the addon HelmRelease it fronts, so an etcd-less tenant with the addon pinned on does not get a selector-less Service (+ test). Correct the secrets framing: the tenant-visible surface is an allowlist (include: [] withholds everything), so drop the never-load-bearing exclude entry, the bats cross-check and the rename-coupling comments, and state the real mechanism. Fix the etcd prerequisite docs (an ancestor's etcd satisfies the wrapped chart; the awaiting-etcd beacon is the diagnostic), reword the tenant toggle to stop advertising the not-yet-shipped placement routing, require maxReplicas >= 1 on the ingress-role node group, and fix comment drift (Service name after the release rename, placement-hook label, package-guard lookup semantics). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
366268d9f5 |
test(kubernetes): pin that a disabled ingress addon renders nothing with hosts set
hosts with the addon disabled used to render the Ingress; the addon gate makes 'the addon is off' mean nothing renders. Pin the combination so the behaviour change reads as a decision. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
5a3d25b8f5 |
docs(computeplane): document the etcd prerequisite and unlisted-field pass-through
The tenant Kubernetes machinery — including the cluster-autoscaler that brings up the default scale-from-zero md0 group — is backed by the tenant's etcd module, so a ComputePlane without it provisions no workers and gives no diagnostic; state the prerequisite in the README. Also state plainly that unlisted NodeGroup fields pass through to the wrapped chart unvalidated (the schema generator cannot close the object), with a test pinning that behaviour. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
6dcf5373e7 |
fix(computeplane): let Flux derive both release names
The aggregated API cannot represent spec.releaseName: the conversion rebuilds HelmRelease specs without it, so the first tenant edit through the API dropped the module's explicit computeplane-module release name, let it default to the object name computeplane, and collided with the Helm storage record the cluster release owned — with the tenant chart writing the old name back every interval. Drop releaseName from both HelmReleases and derive names from object names, like every other tenant module. Kamaji now writes the admin kubeconfig to computeplane-cluster-admin-kubeconfig; the computeplane-rd secrets.exclude entry follows in the same change, and a bats test pins the exclusion to the name the cluster release actually produces so a future rename cannot silently re-expose the credential. Also add the check-release-name guard the other extra modules carry, and pin the fluxcd addon's distribution artifact to the flux-operator version vendored in system/fluxcd-operator instead of :latest. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
ebeeab77aa |
docs(platform): correct the computeplane dependsOn and variant comments
The dependsOn comment claimed the kubernetes-application edge orders the addon artifacts the wrapped chart references; variant dependsOn only gates install: components — the operative effect is ordering the computeplane-rd install so the ComputePlane kind is not registered before the kubernetes machinery exists. Also document why the variant is named kubevirt. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
96842d4a94 |
fix(computeplane): validate nodeGroups and require an ingress-nginx role
nodeGroups was a bare object: typos surfaced as a stuck release deep in the wrapped chart instead of an admission error, and groups without the ingress-nginx role silently left the pinned ingress-nginx controller (and with it the ComputePlane ingress) Pending forever, since supplied groups are not merged with the wrapped chart's default md0. Reuse the kubernetes app's nodeGroups typedefs for the schema and refuse rendering when no group carries the role. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
76374f6472 |
fix(tenant): fail loudly when computeplane is enabled without its package
The computeplane-application package ships in the iaas bundle while the toggle arrives with tenant-rd in the system bundle, and bundles are independent booleans. Refuse the render with a clear diagnostic instead of producing a HelmRelease that retries a nonexistent artifact forever. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
bca5b3f9fb |
fix(kubernetes): render the Proxied ingress Service only when the addon is enabled
exposeMethod defaults to Proxied while the addon defaults to off, so gating the Service on exposeMethod alone emitted a selector-less Service — surfaced in the tenant dashboard — on every cluster that runs no ingress-nginx at all. Gate on the addon instead; ComputePlane sets enabled: true explicitly, and the Service still renders without a hosts list. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
aa5198b758 |
fix(platform): add cozystack-engine dependency to the computeplane package
The computeplane-rd component installs an ApplicationDefinition, whose CRD is served by cozystack-engine — hack/check-applicationdefinition-crd-ordering.bats enforces the edge for every package with a system/*-rd component. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9cb4f2922d |
fix(kubernetes): create the Proxied ingress Service independently of hosts
The proxy Service was only rendered when addons.ingressNginx.hosts was non-empty, but external consumers — e.g. a ComputePlane app's per-app management Ingress — need to route to the tenant cluster's ingress-nginx without the module knowing every app host in advance. Render the Service whenever exposeMethod is Proxied; the Ingress still requires hosts. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
b2232904b6 |
feat(tenant): add the computeplane module toggle
Wire the computeplane module into the tenant chart as a bool toggle, matching the other tenant modules. The module's own Helm release takes the -module suffix so the canonical release name computeplane (and with it the computeplane-admin-kubeconfig Secret) stays reserved for the cluster HelmRelease the module renders. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
4c3262a4ee |
feat(computeplane): deliver ComputePlane as a tenant module wrapping the kubernetes app
Add the operator-owned computeplane tenant module per the revised design (cozystack/community design-proposals/compute-plane): an extra/computeplane chart that provisions a single-tenant, Cozystack-managed cluster for untrusted-code apps by deploying the ordinary apps/kubernetes chart with operator-fixed values (fluxcd, cert-manager, ingress-nginx Proxied). The package re-declares apps/kubernetes as its own source-only component, so the module is self-contained; the cluster release is named computeplane so Kamaji writes the admin kubeconfig to computeplane-admin-kubeconfig — the contract consumed by placement: ComputePlane applications. The tenant holds no admin kubeconfig and can set cluster shape (nodeGroups) only. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
|
7ebce2e942 |
feat(kubeovn): build the chart and image in-tree
kubeovn was the last package whose chart and image were produced by a separate repository: packages/system/kubeovn pulled a tarball from the latest cozystack/kubeovn-chart tag at update time, and that repository built and pushed the image. The wrapper existed only to cache the upstream chart; with the current caching that reason is gone, and the split cost a second repository, a second release step, and a package whose provenance a reader could not follow without leaving the tree. Move the wrapper in, shaped like packages/system/cilium: update pulls the upstream kube-ovn chart directly and applies patches/mtu.diff, and image builds images/kubeovn and pins the digest into values.yaml. The MTU patch already carried monorepo-relative paths, so it applies here unchanged. Also carries cozystack/kubeovn-chart#5, which never got a tag: pin OVN_IPSEC_KEY_DIR under /var/lib. The kube-ovn default sits in /etc/origin, which a Talos node cannot create, so enabling IPsec left the node without ESP security associations. Co-authored-by: Timofei Larkin <lllamnyp@gmail.com> Co-authored-by: Kirill Ilin <stitch14@yandex.ru> Co-authored-by: Aleksei Sviridkin <f@lex.la> Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
8b461f0c0b |
feat(monitoring): drop the Grafana rebuild for the upstream image + catalog plugins (#3378)
## What this PR does Cozystack builds and publishes `ghcr.io/cozystack/cozystack/grafana` — a rebuilt Grafana image — and uses it by default in the monitoring stack. Grafana is AGPL-3.0, so publishing a rebuilt binary makes the project a redistributor of an AGPL Grafana build. This is a governance concern for CNCF Incubation (there is no precedent for a CNCF project redistributing an AGPL binary). The only reason the image was ever rebuilt was to bundle the VictoriaLogs datasource plugin, which was not in the official Grafana plugin catalog at the time. It is now published there as a signed plugin (`victoriametrics-logs-datasource`), so the rebuild is no longer necessary. This PR moves the deployment to the upstream `grafana/grafana` image (digest-pinned, routed through `cozy-lib.image` so mirrored-registry installs keep working) and installs the plugins at container startup via `GF_INSTALL_PLUGINS`, with versions pinned: - `victoriametrics-logs-datasource 0.29.0` — the VictoriaLogs datasource (signed; the reason the image was rebuilt) - `natel-discrete-panel 0.1.1`, `grafana-worldmap-panel 1.0.6`, `marcusolsson-dynamictext-panel 6.2.0` — reproduce the panel set the removed Dockerfile vendored **Air-gapped installs keep working.** The plugin zips are not downloaded from the internet at pod startup: they are mirrored from the official catalog into the existing `grafana-dashboards` image at build time and served by that in-cluster static file server, next to the dashboards it already serves. `GF_INSTALL_PLUGINS` points at `http://grafana-dashboards.cozy-grafana-operator.svc/plugins/…`, so the runtime dependency stays inside the cluster and the artifact is mirrored into private registries together with every other first-party image. A new `allow-to-grafana-dashboards` CiliumNetworkPolicy (mirroring `allow-to-keycloak`) opens tenant egress to that server, since tenant egress is default-deny. As a result, `ghcr.io/cozystack/cozystack/grafana` is no longer built or published: the Dockerfile, the pinned image tag, the `image` targets in both monitoring Makefiles, and the build step in the root Makefile are removed. Notes: - The AGPL-3.0 Grafana binary is no longer redistributed; the mirrored plugin zips are Apache-2.0 licensed and are served unmodified from the official catalog builds (signatures intact). - `allow_loading_unsigned_plugins` and the custom `/var/lib/grafana-plugins` path are dropped: the catalog datasource is signed, and plugins install into the writable default `GF_PATHS_PLUGINS` (`/var/lib/grafana/plugins`). - The VictoriaLogs datasource provisioning (the `GrafanaDatasource` CRs pointing at `vlselect`) is unchanged. - Plugins are re-installed on every pod start from the in-cluster server (no PVC involved, ~77 MB per start per replica over the cluster network); a failed download fails the pod loudly instead of silently dropping the datasource. - `natel-discrete-panel` and `grafana-worldmap-panel` are Angular plugins; Angular is disabled by default in Grafana 11, so they do not load — this is unchanged from the current rebuilt image (same Grafana 11.6.15, same config). They are kept for parity and can be dropped as a follow-up. ### How this was verified - `helm unittest` across the repo passes, including new regression tests: the monitoring chart asserts the exact upstream image reference (with and without `images-registry` set) and the exact `GF_INSTALL_PLUGINS` value; the tenant chart asserts the new egress policy. - Local end-to-end of the full chain in Docker: built the modified `grafana-dashboards` image, served it on a shared network, and started upstream `grafana/grafana:11.6.15` with the rendered `GF_INSTALL_PLUGINS` value. All four plugins download from the in-cluster-style URL (no internet egress from Grafana), and `GET /api/plugins` reports `victoriametrics-logs-datasource` v0.29.0 as `enabled: true` with `signature: valid`. ### Screenshots Not applicable — no UI change. Grafana renders the same datasources and panels; only the image source and plugin delivery mechanism change. ### Downstream repositories Walked the trigger map against the diff. This is an internal change to how the monitoring stack sources the Grafana image and its plugins (`packages/system/monitoring`, `packages/system/grafana-operator`, `packages/apps/tenant` network policy). No package is added, renamed or removed; no `values.schema.json`, `ApplicationDefinition`, platform values, release asset, shared `hack/` tooling or `package.mk` contract changes; the added `grafana.image` key is an internal system-chart value, not a user-facing app field. No downstream repository is affected. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note feat(monitoring): Grafana now runs on the upstream `grafana/grafana` image; the VictoriaLogs datasource and panel plugins are mirrored from the official Grafana catalog into the in-cluster `grafana-dashboards` server and installed at startup, so air-gapped installs keep working. Cozystack no longer builds or publishes `ghcr.io/cozystack/cozystack/grafana`. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Improvements** * Grafana in monitoring is now based on a digest-pinned upstream image for more consistent deployments. * Required Grafana plugins are installed automatically at startup via `GF_INSTALL_PLUGINS`; unsigned-plugin configuration was removed. * Tenant networking now includes a new egress rule specifically for Grafana dashboards. * Grafana dashboards image now includes required plugin assets prepared during image build. * **Tests** * Added/updated checks covering digest pinning, mirrored-registry behavior, plugin-installation behavior, and the new Grafana dashboards network policy. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
78fc3f9178 |
refactor(ci): drop the vestigial /tmp workspace from e2e (both workflows)
The E2E job once ran as several state-sharing self-hosted-runner jobs that rendezvoused through a sha256(REPO:WORKFLOW:REF) key and a per-run /tmp/$SANDBOX_NAME copy of the checkout. Since the move to the ephemeral pool it is a single job: there is no cross-job rendezvous and the checkout is never reused, so copying it into /tmp and removing it afterwards is dead weight. Run every step in $GITHUB_WORKSPACE instead: drop the Prepare/Remove workspace steps and the `cd /tmp/$SANDBOX_NAME` prefixes, and point the upload paths at _out/. This deletes the unguarded `rm -rf /tmp/$SANDBOX_NAME` rather than guarding it: when an upstream step fails or the job is cancelled, the unconditional `Set sandbox ID` is skipped, SANDBOX_NAME stays empty, and the `if: always()` steps interpolate it — the removal expanding to `rm -rf /tmp/` and failing the cleanup step. `Set sandbox ID` stays, since the container still needs a name. Apply the same treatment to nightly.yaml, and drop the Makefile dead code the single-runner retirement left behind: the world-writable mkdir/chmod of /tmp/$SANDBOX_NAME (backed a bind mount removed long ago) and the `-e SANDBOX_NAME` container env (nothing in the e2e-sandbox image reads it). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
d598e672f2 |
fix(reloader): add startupProbe to survive slow first start under load (#3470)
## What this PR does Adds a startup probe to the reloader deployment, rendered from package values through a patch on the vendored chart. Reloader went into CrashLoopBackOff on a loaded install. Events show `Liveness probe failed: ... connect: connection refused`, repeating on the liveness cadence until the kubelet gave up. A refusal rather than a timeout closes two questions at once. It is an RST from the target netns, so the packet arrived and the pod network is fine. It also rules out the mode in [stakater/Reloader#252](https://github.com/stakater/Reloader/issues/252), where the listener is already up and `/metrics` just answers slowly; that one produces probe timeouts. Nothing was listening on `:9090` yet. `http.ListenAndServe` is the last statement of `startReloader`, so the whole prologue runs before the socket opens. The slow part is not the informers, those are lazy and do their List in a goroutine. It is `IsOpenshift = isOpenshift()` in `pkg/kube`: a package-level var, so it runs at init before `main()`, and it does a `GET /apis/project.openshift.io` on a `context.TODO()` carrying no deadline. The latency is the API server's, and the container waits it out under a 100m CPU limit. Without a startup probe the only grace period is the liveness budget itself, `initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5`. A start that overruns it gets killed mid-prologue, and the restart re-runs the prologue from the top against an API server that is no less busy, so the loop feeds itself. This reaches past reloader. `packages/core/platform/sources/linstor.yaml` lists `cozystack.reloader` in the LINSTOR stack's `dependsOn`, and `dependsOn` fans out to every component of that stack, so a reloader stuck in this loop blocks that whole install. The probe is `/live` at `periodSeconds 5` and `failureThreshold 45`, about 225s, well inside the 10m install timeout that cozystack-operator stamps on generated HelmReleases. Liveness is suspended while it runs and gets its tight budget back as soon as the listener binds. `timeoutSeconds` is 5 to match the other two probes instead of the kubelet default of 1s, since a throttled process at 100m can hold the socket open and still need more than a second to answer. Both liveness knobs the chart already exposes would work without any patch. Widening `failureThreshold` buys the grace permanently and loosens hang detection for the life of the container. `initialDelaySeconds` is start-up-only and would genuinely do, but it is a fixed blind window on every start, including one that binds in three seconds, and nothing restarts a container that wedges inside it. Adaptive against fixed is the trade. The win is smaller than it sounds while `reloader.enableHA` is unset, since `/live` is a constant 200 then and liveness only catches the listener dying outright. A chart bump does not fix this. 2.2.14 still renders no `startupProbe` anywhere, and v1.4.19 added a second eager package-level call next to `IsOpenshift`, so the prologue got longer instead of shorter. `packages/system/velero` already carries a startupProbe patch on its vendored chart for the same reason. `make update` now drives the shared `%-update` rule with an exact `CHART_VERSION`. It has to be exact. With a stale local repo cache `helm pull --version 2.0.0` fails with version-not-found, while `--version "^2.0.0"` quietly resolves to 2.2.14, whose `deployment.yaml` has the ports block at line 179 instead of 161, so the hunk would fuzz or land somewhere unintended. The patch runs with `--forward` because `patch(1)` defaults diverge between implementations over a tree that already carries the change. GNU 2.7.6 refuses and exits 1. Apple 2.0-12u11 takes its `Assume -R? [y]` default, strips the hunk back out and exits 0. GNU under `--batch` does the same, which is the trap, since `--batch` is what you reach for to make a script non-interactive. `--forward` refuses in all three and leaves a `.rej`. Two things keep the probe from disappearing quietly, and they fire at different times. `--forward` catches it at the moment someone re-runs the patch by hand. The test only catches it at the next run of the suite, late enough for a commit to travel in between. `make test` also checks that the vendored `Chart.yaml` still records the pinned version, so bumping `CHART_VERSION` without re-running `make update` fails instead of leaving the pin and the tree disagreeing. Checked by doing the violation and watching it go red. ### Screenshots Not applicable, no UI change. ### Downstream repositories Walked the trigger map in `docs/agents/contributing.md` against the diff. All five files sit inside one `packages/system` package. No package is added, renamed or removed, nothing under `hack/` is touched, and no CRD, namespace, `packages/core` values key, version enum or release asset changes. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(reloader): add a startup probe so a slow first start is not killed by the liveness probe and left in CrashLoopBackOff, which also unblocks the LINSTOR stack that depends on reloader ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a configurable startup health probe for the Reloader service. * The probe allows additional time for the service to start before health checks begin. * Documented the probe’s timing and behavior in the chart configuration. * **Bug Fixes** * Preserved existing liveness and readiness probe behavior. * **Tests** * Added coverage for enabled, disabled, and unaffected health probes. * Added validation to ensure the chart version matches the pinned release. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
9fe6ae2ec1 |
fix(keycloak-configure): patch HelmRelease in release namespace on teardown (#3372)
## What this PR does
The `keycloak-configure` pre-delete teardown Job clears the Flux
`HelmRelease` finalizers as its final step so the release can be
uninstalled. That final `kubectl patch` targeted a hardcoded namespace
that does not match where the release actually installs. The release
installs into `cozy-keycloak`, and the teardown Job's own RBAC — a
`Role` + `RoleBinding` granting `patch` on the `HelmRelease` by release
name — is created in the release namespace, but the patch was aimed at
`cozy-system`.
The result is a deadlock: the Job's `ServiceAccount` is `Forbidden` to
patch the `HelmRelease` in the mismatched namespace, so the Job errors
and retries forever, the `HelmRelease` sticks in `Terminating`, and the
Helm release wedges in status `uninstalling`. This blocks any teardown
or reinstall of the component.
The fix templates both the release name and the namespace from `{{
.Release.Name }}` / `{{ .Release.Namespace }}`, so the patch targets the
namespace where the `HelmRelease` lives and where the teardown RBAC
grants access. I also added a helm-unittest that pins the rendered
target, guards against the hardcoded namespace regressing, and asserts
the patch tracks the release name and namespace.
### Screenshots
N/A — no UI changes.
### Release note
```release-note
fix(keycloak-configure): fix the pre-delete teardown Job so uninstalling the release no longer wedges — it now patches the HelmRelease in the release namespace instead of a hardcoded one
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
- **Bug Fixes**
- Improved Helm chart deletion cleanup to patch the correct HelmRelease
using the chart’s release name and namespace instead of fixed values.
- Reduced risk of teardown targeting resources in an unintended
namespace.
- **Tests**
- Added a new delete/teardown YAML test suite to verify correct
parameterization and that only the intended namespace is modified.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
07cefee158 |
fix(kubernetes): raise kubevirt-csi API client rate limits (#3428)
## What this PR does The kubevirt-csi-driver builds its infra and tenant cluster clients from a `rest.Config` with QPS/Burst left unset, so client-go uses its default 5 QPS / 10 burst. The wrapper's NFS `ControllerPublishVolume` polls the infra PVC once per second for up to two minutes. When a tenant scales out and several volumes attach at once, that shared token bucket starves and the poll returns `client rate limiter Wait returned an error: context deadline exceeded`, which the tenant kubelet reports as `FailedAttachVolume`. It looks like an intermittent E2E failure attaching a tenant volume, with nothing wrong in the storage backend. This sets QPS/Burst to 100/200 on both configs, before any client is built. The new flags `--kube-api-qps` and `--kube-api-burst` make the limits tunable. The compiled-in defaults are the real fix: the deployment passes no client flags, so both configs resolve to the in-cluster config and pick up the new defaults. Non-positive values are rejected, since client-go would read `0` as the starving default and a negative value as no rate limiting. Includes a regression test: a burst of 200 waiters times out against the default 5 QPS / 10 burst limiter and passes against the configured one. ### Screenshots None. No UI changes. ### Downstream repositories The diff touches only the driver's Go source in `packages/apps/kubernetes/images/kubevirt-csi-driver/`. No package was added, renamed or removed, and no `values.schema.json`, chart values, `ApplicationDefinition`, node prerequisite, metric or label changed. Nothing in the downstream trigger map is reached. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(kubernetes): raise the kubevirt-csi-driver API client rate limits (QPS/Burst, tunable via --kube-api-qps and --kube-api-burst) above the client-go defaults so a burst of concurrent volume attaches no longer times out as FailedAttachVolume. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added configurable Kubernetes API request rate limits for improved handling of high-volume operations. * Added validation to prevent invalid or non-positive rate-limit settings. * **Bug Fixes** * Improved reliability during bursts of concurrent volume attachment requests by reducing request throttling and timeout risk. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7f67d02fb7 |
fix(reloader): add startupProbe to survive slow first start under load
Reloader binds its :9090 listener as the last statement of startReloader, so /live and /metrics answer connection refused until everything before it has finished. The blocking part of that prologue is the package-level IsOpenshift = isOpenshift() in pkg/kube, which issues a GET /apis/project.openshift.io against the API server on a context carrying no deadline, before main() is even entered. That latency belongs to the API server, and the container waits it out at a 100m CPU limit. The chart renders no startupProbe, so the only grace period is the liveness budget: initialDelaySeconds 10 + periodSeconds 10 * failureThreshold 5, about 60 seconds. A first start that needs longer is killed mid-prologue, and the restart does not resume it, it runs it again from the beginning, so the loop sustains itself. The blast radius reaches past reloader: the LINSTOR stack lists cozystack.reloader in dependsOn, which fans out to every component of that stack, so a reloader stuck in this loop blocks the whole install. Render a values-driven startupProbe on /live so liveness only begins evaluating once the listener is up. Budget = periodSeconds 5 times failureThreshold 45, so about 225 seconds. timeoutSeconds matches the 5s the chart already gives liveness rather than the kubelet default of 1s, which a throttled process can exceed while holding the socket open. Both liveness knobs the chart already exposes could buy the same grace with no patch to carry. Widening failureThreshold buys it permanently, loosening hang detection for the life of the container. initialDelaySeconds is start-up-only and would genuinely do, but it is a fixed blind window on every start, whereas a startup probe returns liveness to its tight budget the moment the listener binds. The chart is vendored, so the template change is carried as patches/deployment-startup-probe.diff and reapplied by `make update`, which now drives the shared %-update rule with an exact chart version pin instead of pulling whatever is latest. The patch is applied with --forward because patch(1) defaults diverge by implementation over a tree that already carries the change: GNU 2.7.6 refuses and exits 1, Apple 2.0-12u11 takes its "Assume -R? [y]" default and strips the hunk back out with exit 0, and GNU under --batch does the same. --forward refuses in all three. A helm unittest pins the probe, its budget, the no-op path the template promises for an unset value, and the liveness numbers the budget is sized against, so the documented arithmetic cannot drift away from the chart without a red test. The same target asserts that the vendored Chart.yaml still records the pinned version, so raising the pin without re-running `make update` fails instead of leaving the tree and the pin quietly disagreeing. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
fd568dee87 |
fix(keycloak-configure): patch HelmRelease in release namespace on teardown
The pre-delete teardown Job clears the HelmRelease finalizers as its final step, but targeted a hardcoded namespace that does not match the namespace where the release and its RBAC live. The Job's Role and RoleBinding are created in the release namespace and grant patch on the HelmRelease by release name, so the ServiceAccount was Forbidden to patch the HelmRelease in the hardcoded namespace. The Job errored and retried forever, the HelmRelease stuck in Terminating, and the Helm release wedged in "uninstalling", blocking any teardown or reinstall. Target the release name and namespace so the patch matches the RBAC that grants it. Add a helm-unittest pinning the rendered namespace. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
2dfe380ac3 |
fix(kubernetes): raise kubevirt-csi API client rate limits
The kubevirt-csi controller built its infra- and tenant-cluster Kubernetes clients on a rest.Config that left QPS/Burst unset, so client-go applied its 5 QPS / 10 burst default. ControllerPublishVolume polls the infra PVC once per second for up to two minutes; when a tenant scales out and several volumes attach at once, the default limiter starves and the poll fails with "client rate limiter Wait returned an error: context deadline exceeded", which the tenant kubelet surfaces as FailedAttachVolume. Set QPS/Burst to 100/200, overridable via --kube-api-qps and --kube-api-burst, on both the infra and tenant configs so a concurrent-attach burst is admitted promptly. Reject non-positive flag values, which client-go would otherwise turn into the starving default (0) or unlimited (<0). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
82e10878be |
fix(harbor): guarantee CPU for redis sentinels
The inline RedisFailover ran its three sentinels at 50m CPU request and a 100m limit. Sentinels only become Ready after the redis-operator elects a master and rewrites their monitored-master config, and on a CPU-contended node CFS shares cycles proportionally to the request, so underprovisioned sentinels converge slowly and delay harbor readiness behind them. Raise the sentinels to Guaranteed QoS (requests equal limits, 250m CPU / 128Mi) so election is never starved or throttled under load. The redis data container is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
Aleksei Sviridkin
|
8bffa34b54 |
feat(platform): add opt-in in-cluster east-west encryption toggle (#2984)
## What this PR does Adds `networking.encryption.enabled` to the platform chart — a single operator-facing opt-in, off by default, that turns on in-cluster (east-west) transport encryption for the active datapath. When enabled, pod-to-pod traffic across nodes is encrypted at the network layer regardless of whether the workload implements TLS: defense-in-depth for the whole mesh without per-application changes. The default `isp-full` bundle selects the `kubeovn-cilium` datapath (and `isp-full-generic` selects `kubeovn-cilium-generic`), where KubeOVN owns the datapath and the overlay is a Geneve tunnel. The toggle therefore maps to KubeOVN's overlay IPsec (`kube-ovn` `ENABLE_OVN_IPSEC`). KubeOVN's own controller provisions the IPsec CA and per-node certificates through the Kubernetes CSR API, so no cert-manager wiring is needed. This complements, and does not replace, application-level / end-to-end TLS: in-cluster encryption never reaches an external client and gives a tenant no verifiable endpoint certificate. The values comments document the per-datapath scope — including that Kilo's WireGuard mesh (`meshGranularity: location`) covers only cross-location traffic and leaves the intra-location gap this toggle targets, and that the Cilium-native WireGuard/IPsec path on the `cilium` / `cilium-generic` variants is not reached because the system bundle does not select those variants — plus the MTU/throughput overhead. Scope: only the default KubeOVN datapath is wired through the platform toggle, which is what the system bundle actually selects. Wiring the Cilium-native path would require the bundle to expose cilium-only variant selection; that is out of scope here and called out in the values comments. Validation: `helm unittest` on the platform chart passes (66 tests, 7 new) covering on / off / absent / null and verifying the IPsec flag deep-merges without dropping the existing `kube-ovn` overrides, for both `isp-full` and `isp-full-generic`. Closes #2977 Part of #2811 ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(platform): add opt-in `networking.encryption.enabled` (off by default) that enables in-cluster east-west transport encryption for the default KubeOVN datapath via overlay IPsec; complements, and does not replace, application-level TLS. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Introduced opt-in in-cluster network encryption using KubeOVN IPsec for east-west traffic protection. * Added `networking.encryption` configuration toggle (disabled by default) with full backward compatibility. * **Tests** * Added comprehensive test coverage for network encryption configurations across system variants and edge cases. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Timofei Larkin
|
66c96b4329 |
fix(dashboard): stop requesting offline_access so logout can revoke sessions (#3447)
## What this PR does The dashboard and linstor-gui auth-proxies (oauth2-proxy) requested the `offline_access` scope, so Keycloak issued an **offline** refresh token. oauth2-proxy has no server-side session store: it keeps the refresh token in the `kc-access` cookie and silently refreshes it (`--cookie-refresh`). An offline token is decoupled from the SSO session by design, so logout (`end_session`) does not revoke it. As a result a copied `kc-access` cookie stayed usable for the whole offline-session lifetime (Keycloak default ~30 days) even after the user logged out — logout did not invalidate it. This drops `offline_access` from the proxy `--scope` in both packages, so Keycloak issues an **online** refresh token bound to the SSO session, which logout revokes. The `offline_access` optional client scope is also removed for cleanliness (with a note on why). Scope of the fix: - `packages/system/dashboard` — proxy `--scope` + optional client scope - `packages/system/linstor-gui` — proxy `--scope` + optional client scope + ingress auth unittest The public `kubernetes` OIDC clients (kubectl `oidc-login`) are intentionally left untouched: `offline_access` is legitimate there and those clients do not sit behind oauth2-proxy. Notes: - The keycloak-operator only *adds* client scopes, so removing the optional scope does not strip the realm-default `offline_access` from an already-created client. The proxy `--scope` change is what actually prevents issuance. - This prevents *new* offline tokens. Offline sessions already issued to existing users remain valid until the realm's offline-session idle timeout (~30d) or explicit revocation via the Keycloak Admin REST API / Account Console. - After this change dashboard session lifetime is governed by the online SSO session (`client.session.idle.timeout` / `client.session.max.lifespan`). ### Screenshots Not applicable — no UI change (auth-proxy OIDC scope only). ### Downstream repositories Walked the trigger map against the diff: the change touches only helm templates and a unittest under `packages/system/dashboard` and `packages/system/linstor-gui`. No downstream repository restates the auth-proxy OIDC scope. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: ### Release note ```release-note fix(dashboard): stop requesting the offline_access scope in the dashboard and linstor-gui auth-proxies so that logout revokes the session; a copied kc-access cookie is no longer refreshable after logout. Offline sessions already issued to existing users must be revoked manually via Keycloak. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Security Improvements** * Removed requests for offline access and persistent refresh tokens from OIDC authentication. * Sessions now rely on standard scopes, helping ensure logout invalidates authentication as expected. * **Tests** * Updated authentication checks to verify the revised OIDC scope configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
1c7eb22918 |
feat(tenant): allow egress to the grafana-dashboards plugin server
Tenant egress is default-deny plus explicit allow-to-* policies, so the Grafana pods deployed per tenant could not reach the in-cluster plugin server in cozy-grafana-operator. Mirror the allow-to-keycloak pattern and cover the policy with a test. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
705e39d3d7 |
feat(monitoring): install Grafana plugins from the in-cluster mirror
Point GF_INSTALL_PLUGINS at the grafana-dashboards static file server instead of the public Grafana catalog, so plugin installation needs no internet egress at pod startup and keeps working on air-gapped installs. Route the upstream Grafana image reference through cozy-lib.image so mirrored-registry installs keep working, and add regression tests for the image reference and the plugin install path. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
606ffd5b1b |
build(grafana-operator): mirror Grafana catalog plugins into the dashboards image
Download the pinned Grafana plugin zips (VictoriaLogs datasource + panels) from the official catalog at image build time and serve them next to the dashboards. Grafana pods can then install plugins from this in-cluster server instead of reaching the public catalog at startup, which keeps air-gapped and egress-restricted installs working. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |
||
|
Andrei Kvapil
|
9c63df7545 |
build(monitoring): stop building and publishing the Grafana rebuild
Remove the custom Grafana image build now that the deployment runs on the upstream grafana/grafana image with catalog plugins. Drops the Dockerfile, the pinned image tag, the image targets in both monitoring Makefiles, and the build step from the root Makefile, so ghcr.io/cozystack/cozystack/grafana is no longer produced or published. Assisted-By: Claude Signed-off-by: Andrei Kvapil <andrei.kvapil@aenix.io> |