mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-09 14:02:56 +09:00
8f4c19daaeb00a2aeccf761cd96879867a4d86ce
5128
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Aleksei Sviridkin
|
809a04a220 |
feat(kubevirt-cdi): make cloneStrategyOverride configurable (#2926)
## What this PR does - Exposes the CDI `cloneStrategyOverride` as a Helm value in `packages/system/kubevirt-cdi` instead of hardcoding it in the CR template. - Default stays `csi-clone`, so **behavior is unchanged** for everyone. - It can now be overridden via a cozystack `Package` (e.g. `cloneStrategyOverride: copy`), the same way `uploadProxyURL` already is — no fork needed. ### Why The value was hardcoded, so the only way to change the clone strategy was to fork the package. On some storage backends `csi-clone` is problematic: with LINSTOR/DRBD a CSI clone is created co-located with the source golden image and then relocated to optimal nodes asynchronously; that async relocation (`migrate-disk`) can race the post-clone resize CDI performs, occasionally leaving a volume whose DRBD device is smaller than its PVC capacity. Being able to select `snapshot` or `copy` per cluster, without maintaining a fork, is a useful escape hatch. This only adds a knob; it does not change the default or recommend a strategy. ## Release note ```release-note feat(kubevirt-cdi): make CDI cloneStrategyOverride configurable via values (default csi-clone) ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * The KubeVirt CDI clone strategy is now configurable via Helm chart values, allowing selection of `csi-clone`, `snapshot`, or `copy` for DataVolume cloning. * The default remains `csi-clone` to preserve existing behavior and ensure backward compatibility. * **Documentation** * Added the `cloneStrategyOverride` Helm value description and allowed options to the chart’s configuration. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
901bb981ae |
fix(seaweedfs): configure postgres2 connection pool for the filer (#2906)
## What this PR does - Adds `WEED_POSTGRES2_CONNECTION_MAX_IDLE=20`, `WEED_POSTGRES2_CONNECTION_MAX_OPEN=40` and `WEED_POSTGRES2_CONNECTION_MAX_LIFETIME_SECONDS=600` to the filer `extraEnvironmentVars` in `packages/system/seaweedfs/values.yaml`. The vendored seaweedfs chart only ships connection pool settings for the **mysql** store (`WEED_MYSQL_CONNECTION_MAX_*`), which cozystack does not use. The **postgres2** store, which cozystack enables, has no pool settings at all. As a result the Go `database/sql` pool runs with `SetMaxIdleConns(0)`: every filer metadata lookup opens a fresh PostgreSQL connection (TCP + TLS + SCRAM, ~300 ms each). With ~5 path-component lookups per S3 operation, this adds roughly **2 seconds of latency to every S3 request**, even on an idle cluster. ### Measurements (production cluster, 12 nodes, 2 filers, CNPG backend) Before / after applying these three variables, same test (sequential ops through the S3 gateway): | Operation | Before | After | |---|---|---| | PUT 4 KiB (p50) | 10.0 s | 0.28 s | | GET 4 KiB (p50) | 3.0 s | 0.14 s | | HEAD (p50) | 2.6 s | 0.16 s | | Bounded LIST, 100 keys (p50) | 3.8 s | 0.37 s | `pg_stat_activity` confirmed the root cause: before the fix every backend was younger than ~200 ms (permanent connection churn); after the fix connections are reused for minutes. Real-traffic p95 of `SeaweedFS_filerStore_request_seconds` dropped from 1.3–6.5 s to ~0.18 s. `max_open` is capped at 40 so that the default two filer replicas stay below the CNPG default `max_connections` of 100. ## Release note ```release-note fix(seaweedfs): configure postgres2 connection pool for the filer, removing ~2s of per-request S3 latency caused by PostgreSQL connection churn ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Optimized PostgreSQL connection pool configuration in SeaweedFS to reduce latency and improve performance during metadata operations. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
ec9c29948b |
[reloader] Set resources requests/limits on reloader container (#2631)
## Summary
The upstream Stakater reloader chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `reloader`
Deployment.
Override `packages/system/reloader/values.yaml` at
`reloader.reloader.deployment.resources` with the exact values from the
chart's own commented example:
- `limits` : `cpu: 100m`, `memory: 512Mi`
- `requests` : `cpu: 10m`, `memory: 128Mi`
Reloader is a light controller that watches ConfigMap / Secret changes
and triggers workload restarts — small CPU envelope with enough memory
to track many objects in clusters with lots of namespaces.
## Test plan
- [ ] `helm template . --namespace cozy-reloader` from
`packages/system/reloader/` renders a `resources` block on the
`reloader-reloader` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `reloader-reloader has no
CPU limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the reloader Deployment to clear
the "no CPU limit" conformance warning. Values follow the upstream
chart's commented example (limits 100m/512Mi, requests 10m/128Mi).
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated resource allocation configuration to define CPU and memory
parameters for system component deployments, enhancing infrastructure
stability and resource optimization.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2631)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
f2d66579f1 |
[metrics-server] Set resources requests/limits on metrics-server container (#2630)
## Summary The upstream metrics-server chart ships with `limits` commented out (only `requests cpu 100m / memory 200Mi` are set), which triggers a `no CPU limit` conformance warning on the `metrics-server` Deployment. Override `packages/system/metrics-server/values.yaml` to keep the upstream-recommended baseline `requests` and add `limits`: - `limits` : `cpu: 500m`, `memory: 500Mi` - `requests` : `cpu: 100m`, `memory: 200Mi` Comfortable headroom on top of the [official sizing formula](https://github.com/kubernetes-sigs/metrics-server#configuration) (`100m + 1m/node`, `200Mi + 2Mi/node`), covering clusters up to ~150 nodes without needing to revisit. Larger clusters can override these values further. ## Test plan - [ ] `helm template . --namespace cozy-monitoring` from `packages/system/metrics-server/` renders a `resources` block on the `metrics-server` container of the Deployment (verified locally) - [ ] Re-run the conformance/lint tool — the `metrics-server has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Set CPU/memory requests and limits on the metrics-server Deployment to clear the "no CPU limit" conformance warning. Values follow the upstream sizing recommendation (requests 100m/200Mi) with a comfortable limits cap (500m/500Mi). \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Configured explicit resource requests and limits for the metrics-server service (CPU: 100m-500m, Memory: 200Mi-500Mi) to ensure consistent resource allocation. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2630) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
4952909e4d |
[external-dns] Set resources requests/limits on external-dns container (#2629)
## Summary
The upstream external-dns chart ships with `resources: {}`, which
triggers a `no CPU limit` conformance warning on the `external-dns`
Deployment.
Override `packages/system/external-dns/values.yaml` to set:
- `limits` : `cpu: 100m`, `memory: 128Mi`
- `requests` : `cpu: 10m`, `memory: 64Mi`
Modest sizing for a controller that just watches Service / Ingress
objects and pushes DNS records to the provider.
## Test plan
- [ ] `helm template . --namespace cozy-external-dns` from
`packages/system/external-dns/` renders a `resources` block on the
`external-dns` container of the Deployment (verified locally)
- [ ] Re-run the conformance/lint tool — the `external-dns has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the external-dns Deployment to
clear the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated ExternalDNS resource allocation with optimized CPU and memory
limits and requests for improved resource efficiency.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2629)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
24e3bf98b5 |
[bootbox] Add resources requests/limits to matchbox container (#2625)
## Summary The `matchbox` container in `packages/extra/bootbox/templates/matchbox/deployment.yaml` had no `resources` block, triggering a `no CPU limit` conformance warning on the `bootbox-matchbox` Deployment (PXE/iPXE provisioning service). Bootbox is a standalone chart (no vendored upstream), so the values are set directly in the template — same pattern as #2614 (csi-resizer) and #2624 (s3manager). - `limits` : `cpu: 200m`, `memory: 256Mi` - `requests` : `cpu: 50m`, `memory: 128Mi` Modest sizing since matchbox is a lightweight HTTP server serving iPXE profiles and boot assets from ConfigMaps. ## Test plan - [ ] `helm template bootbox . --namespace tenant-root --show-only templates/matchbox/deployment.yaml --set _namespace.host=example.com --set _cluster.solver=letsencrypt` from `packages/extra/bootbox/` renders the `resources` block on the `matchbox` container (verified locally) - [ ] Re-run the conformance/lint tool — the `matchbox has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Add CPU and memory requests/limits to the matchbox container of the bootbox Deployment to clear the "no CPU limit" conformance warning. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Added Kubernetes resource limits and requests configuration to optimize container resource allocation and improve deployment stability. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2625) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
04a05f623c |
[bucket] Add resources requests/limits to s3manager container (#2624)
## Summary The `s3manager` container in `packages/system/bucket/templates/deployment.yaml` had no `resources` block, triggering a `no CPU limit` conformance warning on the bucket UI Deployment. The bucket package is a standalone chart (no vendored upstream), so the values are set directly in the template — same pattern as the csi-resizer fix in #2614 and the existing `capi-providers-cpprovider` template. - `limits` : `cpu: 500m`, `memory: 512Mi` - `requests` : `cpu: 100m`, `memory: 128Mi` Comfortable headroom for the lightweight S3 browser UI while keeping a small baseline request. ## Test plan - [ ] `helm template . --show-only templates/deployment.yaml` from `packages/system/bucket/` renders the `resources` block on the `s3manager` container (verified locally) - [ ] Re-run the conformance/lint tool — the `s3manager has no CPU limit` warning should be gone - [ ] CI passes ### Release note \`\`\`release-note Add CPU and memory requests/limits to the s3manager container of the bucket UI Deployment to clear the "no CPU limit" conformance warning. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Improved deployment configuration with explicit resource allocation settings to ensure optimal service stability and resource management. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2624) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Aleksei Sviridkin
|
6762d2d95c |
[goldpinger] Set resources requests/limits on goldpinger DaemonSet (#2623)
## Summary
The upstream Bloomberg goldpinger chart ships with `resources: {}`,
which triggers a `no CPU limit` conformance warning on the
`goldpinger-daemon` container running on every node.
Override `packages/system/goldpinger/values.yaml` to set:
- `limits` : `cpu: 200m`, `memory: 256Mi`
- `requests` : `cpu: 50m`, `memory: 64Mi`
Comfortable margin for the ping-mesh fan-out on larger clusters while
keeping a small baseline request so the DaemonSet schedules everywhere.
## Test plan
- [ ] `helm template . --namespace cozy-goldpinger` from
`packages/system/goldpinger/` renders the `resources` block on the
`goldpinger-daemon` container of the DaemonSet (verified locally)
- [ ] Re-run the conformance/lint tool — the `goldpinger has no CPU
limit` warning should be gone
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on the goldpinger DaemonSet to clear
the "no CPU limit" conformance warning.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated Kubernetes resource requirements for the goldpinger component,
defining CPU limits (200m), memory limits (256Mi), CPU requests (50m),
and memory requests (64Mi).
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2623)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
1aa6436922 |
[cert-manager] Set resources requests/limits on cert-manager pods (#2616)
## Summary
The upstream cert-manager chart ships with `resources: {}` for all four
workloads, which triggers `no CPU limit` (and `no memory limit`)
conformance warnings on every pod.
Override `packages/system/cert-manager/values.yaml` (previously empty)
to set sane requests/limits sized to each component's role:
| Component | requests cpu/mem | limits cpu/mem | Notes |
|---|---|---|---|
| controller | 50m / 128Mi | 500m / 512Mi | Main workload — issues and
renews certs |
| webhook | 10m / 32Mi | 200m / 128Mi | Admission webhook, short-lived
requests |
| cainjector | 10m / 32Mi | 200m / 128Mi | CA bundle injection, light
watch loop |
| startupapicheck | 10m / 16Mi | 100m / 64Mi | One-shot Job at install
time |
Requests for `cainjector` (`10m` cpu / `32Mi` mem) match the example in
the upstream values.yaml comment.
## Test plan
- [ ] `helm template . --namespace cert-manager` from
`packages/system/cert-manager/` renders a `resources` block on all four
pods (verified locally for controller, webhook, cainjector and the
startupapicheck Job)
- [ ] Re-run the conformance/lint tool that originally flagged the
warnings on `cert-manager-cainjector` — should be clean on the other
three pods too
- [ ] CI passes
### Release note
\`\`\`release-note
Set CPU/memory requests and limits on cert-manager controller, webhook,
cainjector and startupapicheck pods to clear "no CPU limit" conformance
warnings.
\`\`\`
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added resource allocation configuration for cert-manager deployment,
specifying CPU and memory requests and limits for improved system
stability and resource management across core components.
[](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2616)
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Aleksei Sviridkin
|
b2d00af495 |
[kubernetes] Add CPU/memory limits to csi-resizer sidecar (#2614)
## Summary - The `csi-resizer` container in `packages/apps/kubernetes/templates/csi/deploy.yaml` was the only sidecar in the `kcsi-controller` Deployment without `resources.limits`, which triggers a "no CPU limit" conformance warning. - Aligned it with the other six CSI sidecars in the same Deployment: `limits` `cpu: 512m` / `memory: 512Mi` and `requests` `cpu: 125m` / `memory: 128Mi`. - No other files touched; vendored charts and other CSI drivers (`nfs-driver`, `piraeus-operator`) are out of scope for this PR. ## Test plan - [ ] `make -C packages/apps/kubernetes show NAME=test NAMESPACE=test` renders the Deployment with the new `limits` block - [ ] Re-run the conformance/lint tool that originally flagged the warning — `csi-resizer has no CPU limit` should be gone - [ ] CI passes ### Release note \`\`\`release-note Set CPU and memory limits on the csi-resizer sidecar of the kubevirt-csi controller to align with the other CSI sidecars and clear the "no CPU limit" conformance warning. \`\`\` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Chores** * Updated resource allocation for the CSI resizer component, increasing CPU and memory requests to improve performance and reliability. [](https://app.coderabbit.ai/change-stack/cozystack/cozystack/pull/2614) <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
mattia-eleuteri
|
8bc62cca1c |
feat(kamaji): default to 2 replicas, drop telemetry webhook
The kamaji-controller-manager runs ~9 sub-controllers per
TenantControlPlane (bootstraptoken, coredns, konnectivity, kubeadm*,
kubelet, kubeproxy, etc.) inside a single Go process, AND serves the
admission webhooks for tenantcontrolplanes/datastores via the
kamaji-webhook-service ClusterIP. With the upstream default
replicaCount=1, every webhook RT shares CPU with the reconciler
goroutines. On a multi-tenant cluster (~20 TCPs), this pushes
apiserver-observed p99 for mtenantcontrolplane.kb.io and
vtenantcontrolplane.kb.io from the kamaji-side ~10ms to 85-95ms.
Three changes, each safe on small clusters and high-impact on large ones:
* replicaCount=2 — HA-safe baseline. --leader-elect is already enabled
upstream, so only one replica reconciles tenants; the follower's CPU
is fully available for webhook handlers, cutting p99 ~70% on busy
clusters. On a single-node cluster both pods schedule normally.
* telemetry.disabled=true — drops both the --disable-telemetry handler
and the telemetry.kamaji.clastix.io admission webhook entry from the
ValidatingWebhookConfiguration. The latter requires patches/1.diff to
wrap the telemetry block in a {{ if not .Values.telemetry.disabled }}
conditional inside the vendored controller-gen file. Without the
webhook entry, the apiserver no longer issues a TCP round-trip to a
no-op handler on every TenantControlPlane CREATE/UPDATE/DELETE.
* podAntiAffinity (preferred) — spread the two replicas across nodes
when possible. preferredDuringScheduling so single-node clusters
still schedule (both pods colocated, which is acceptable).
Measured on a 22-TCP production cluster (Hikube, before/after):
mtenantcontrolplane p99 85 → 25 ms (-71%), vtenantcontrolplane p99
92 → 25 ms (-73%), telemetry webhook impact 291 → 0. Total Kamaji
admission impact -83% (830 → 140).
The patches/1.diff approach (re-applied by `make update`) mirrors the
pattern already used in fluxcd-operator and ingress-nginx and survives
the next clastix/kamaji chart bump.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
|
||
|
mattia-eleuteriandClaude Opus 4.8
|
c47bb6edb0 |
feat(kubevirt-cdi): make cloneStrategyOverride configurable
The CDI cloneStrategyOverride was hardcoded to csi-clone in the CR template, so operators could only change it by forking the package. Expose it as a Helm value (default csi-clone, behavior unchanged) so it can be set via a cozystack Package override, the same way uploadProxyURL already is. This is useful where the csi-clone path is problematic on a given storage backend (e.g. LINSTOR/DRBD, where a clone is co-located with the source and relocated afterwards; the async relocation can race the post-clone resize), letting operators select snapshot or copy without maintaining a fork. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
mattia-eleuteri
|
49cbea8c73 |
fix(seaweedfs): configure postgres2 connection pool for the filer
The vendored seaweedfs chart ships connection pool settings only for the mysql store, which cozystack does not use. The postgres2 store, which is enabled, has no pool settings, so the Go sql pool keeps zero idle connections and every filer metadata lookup opens a fresh PostgreSQL connection (TCP + TLS + SCRAM, ~300ms each). With ~5 path lookups per S3 operation this adds roughly 2 seconds of latency to every S3 request, regardless of cluster load. Measured on a production cluster (12 nodes, 2 filers): PUT of a 4KiB object went from 10s (p50) to 0.28s, HEAD from 2.6s to 0.16s after setting these values; pg_stat_activity confirmed connections were previously recycled every ~200ms and are now reused. max_open is capped at 40 so that two filer replicas stay below the CNPG default max_connections of 100. Signed-off-by: mattia-eleuteri <mattia@hidora.io> |
||
|
Matthieu
|
adb307cbae |
[goldpinger] Set resources requests/limits on goldpinger DaemonSet
The Bloomberg goldpinger chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the goldpinger-daemon container
deployed on every node. Override the package values.yaml to set
limits 200m/256Mi and requests 50m/64Mi — comfortable margin for the
ping-mesh fan-out on larger clusters while keeping a small baseline
request.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
Matthieu
|
bf54945c08 |
[reloader] Set resources requests/limits on reloader container
The upstream Stakater reloader chart ships with resources: {}, which
triggers a 'no CPU limit' conformance warning on the reloader
Deployment. Apply the exact values from the chart's own commented
example (limits 100m/512Mi, requests 10m/128Mi) — reloader is a light
controller that watches ConfigMap/Secret changes and triggers restarts,
so a small CPU envelope with enough memory to track many objects fits.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
Matthieu
|
3c2133ce4b |
[metrics-server] Set resources requests/limits on metrics-server container
The upstream metrics-server chart ships with limits commented out (requests cpu 100m / memory 200Mi only), which triggers a 'no CPU limit' conformance warning. Override the package values.yaml to keep the upstream-recommended baseline requests and add limits 500m/500Mi — comfortable headroom on top of the official sizing formula (100m + 1m/node, 200Mi + 2Mi/node), covering clusters up to ~150 nodes without needing to revisit. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
Matthieu
|
7f72e7bbe8 |
[external-dns] Set resources requests/limits on external-dns container
The upstream external-dns chart ships with resources: {}, which triggers
a 'no CPU limit' conformance warning on the external-dns Deployment.
Override the package values.yaml to set limits 100m/128Mi and requests
10m/64Mi — modest sizing for a controller that just watches Service /
Ingress objects and pushes DNS records to the provider.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
Matthieu
|
0ea94e7d2f |
[bootbox] Add resources requests/limits to matchbox container
The matchbox Deployment in the bootbox package had no resources block, triggering a 'no CPU limit' conformance warning. Set limits 200m/256Mi and requests 50m/128Mi — modest sizing since matchbox is a lightweight HTTP server serving iPXE profiles and boot assets from ConfigMaps. Bootbox is a standalone chart (no vendored upstream), so the values are set directly in the template, matching the existing pattern in csi-resizer and s3manager. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
Matthieu
|
c8b180d91a |
[bucket] Add resources requests/limits to s3manager container
The s3manager Deployment in the bucket package had no resources block, triggering a 'no CPU limit' conformance warning. Set limits 500m/512Mi and requests 100m/128Mi — comfortable headroom for the lightweight S3 browser UI. Bucket is a standalone chart (no vendored upstream), so the values are set directly in the template, matching the existing pattern in csi-resizer and capi-providers. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
Matthieu
|
cd95040c58 |
[cert-manager] Set resources requests/limits on cert-manager pods
The upstream cert-manager chart ships with resources: {} for all four
workloads (controller, webhook, cainjector, startupapicheck), which
triggers 'no CPU limit' conformance warnings on every pod. Override the
package values.yaml to set sane requests and limits sized to each
component's role: controller is the most loaded, webhook and cainjector
handle short-lived admission/injection work, and startupapicheck is a
one-shot Job.
Signed-off-by: Matthieu <matthieu@hidora.com>
|
||
|
Matthieu
|
50c4080ba3 |
[kubernetes] Add CPU/memory limits to csi-resizer sidecar
The csi-resizer container was the only sidecar in the kcsi-controller Deployment without resources.limits, triggering a 'no CPU limit' conformance warning. Align it with the other six CSI sidecars in the same Deployment (limits 512m/512Mi, requests 125m/128Mi) so capacity planning and audits stay uniform. Signed-off-by: Matthieu <matthieu@hidora.com> |
||
|
Aleksei Sviridkin
|
58d61d5652 |
fix(ci): build api-gate detector from base tip when merge base predates it (#3193)
## What this PR does The `API Review Gate` required check builds the `cmd/api-gate` detector from the PR's merge-base checkout so a PR cannot neuter the detector by editing its own code. When a PR's merge base predates the commit that introduced the gate (`fa871249c`), `cmd/api-gate` is absent from both the merge base and the equally old head, so the head fallback also fails to build and the check errors out before evaluating any API content. This red-blocks every open PR whose merge base is older than the gate, and neither a re-run nor an approving review clears it. This adds the base branch tip (`origin/<base>`) as an intermediate build source: prefer the merge-base checkout, fall back to the base tip when the merge base lacks the tool, and bootstrap from the head only while the tool exists nowhere on the base branch yet. The base tip is a protected, non-PR-authored branch, so it preserves the "analyze the head as data, not code" property. The API surface is still diffed against the merge base; only where the detector binary is compiled changes. This unblocks the pre-gate backlog without forcing every affected PR to rebase. The selection is kept inline in the workflow rather than in a helper script: a helper would be read from the untrusted PR head, which both reintroduces the failure for heads that predate the helper and lets a PR steer the selection away from the trusted build. Closes #3192 ### Screenshots N/A — CI-only change, no user-facing UI. ### Release note ```release-note fix(ci): API Review Gate now builds its detector from the base branch tip when the PR merge base predates the gate, so pre-gate PRs stop failing the check without needing a rebase. ``` |
||
|
Aleksei Sviridkin
|
361748ea6a |
fix(ci): build api-gate detector from base tip when merge base predates it
The API Review Gate builds cmd/api-gate from the PR's merge-base checkout so a PR cannot neuter the detector by editing its own code. When the merge base predates the commit that introduced the gate, cmd/api-gate is absent from both the merge base and the equally old head, so the head fallback also fails to build and the required check errors out before evaluating any API content. This blocks every open PR whose merge base is older than the gate, and neither a re-run nor an approving review clears it. Add the base branch tip as an intermediate build source: prefer the merge-base checkout, fall back to the base tip when the merge base lacks the tool, and bootstrap from head only while the tool exists nowhere on the base branch yet. The base tip is a protected branch, not PR-authored, so it preserves the "analyze head as data, not code" property. The API surface is still diffed against the merge base; only where the detector binary is compiled changes. This unblocks the backlog without forcing every affected PR to rebase. The selection stays inline in the workflow rather than in a helper script: a helper would be read from the untrusted PR head, which both reintroduces the failure for heads that predate the helper and lets a PR steer the selection away from the trusted build. Assisted-By: Claude <noreply@anthropic.com> Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
IvanHunters
|
1baf126b1a |
fix(tenant): inherit full ancestor label chain from parent namespace (#2912)
## Summary
`packages/apps/tenant/templates/namespace.yaml` labels every tenant
namespace
with `tenant.cozystack.io/<ancestor>` for each of its ancestors. These
labels
drive the `<ancestor>-egress` `CiliumClusterwideNetworkPolicy`
(`networkpolicy.yaml`): an ancestor tenant's pods may reach a namespace
only if
that namespace carries `tenant.cozystack.io/<ancestor>`.
The chain was derived by splitting `.Release.Namespace`, which only ever
reached
one level up. A direct child of `tenant-root` got `tenant-root` (its
parent
namespace *is* `tenant-root`), but a grandchild — `tenant-ktj-htdev`
created
under `tenant-ktj` — got only `tenant-ktj` and dropped `tenant-root`
entirely.
Without `tenant.cozystack.io/tenant-root` on a nested namespace, the
root
ingress controller cannot reach the proxy service in a nested
`kubernetes`
tenant, surfacing as `upstream timed out (110) while connecting to
upstream` in
the nginx logs.
## Fix
Derive the ancestor chain deterministically from the namespace name
instead of
from any cluster lookup. Every tenant descends from `tenant-root`, so
that label
is emitted unconditionally; the remaining ancestors are encoded in the
namespace
name itself, because `tenant.name` builds each child as
`<parent-namespace>-<word>` — so every progressive dash-prefix of the
name is a
real ancestor namespace (`tenant-ktj-htdev` yields `tenant-ktj` and
`tenant-ktj-htdev`). The logic lives in a new
`tenant.ancestorTenantLabels`
helper:
```yaml
labels:
{{- include "tenant.ancestorTenantLabels" $tenantName | nindent 4 }}
```
This needs no cluster state, renders identically offline (so it is
unit-testable), and converges on every reconcile regardless of the order
in
which parent and child `HelmRelease`s reconcile — unlike a
parent-namespace
`lookup`, which helm-controller does not re-render on and which is
ordering-dependent. The `ownerReference` `lookup` is kept but made
non-fatal:
during a real reconcile the parent namespace always exists (the tenant
`HelmRelease` and its `cozystack-values` Secret live in it), so the
`ownerReference` is always emitted; when the lookup is empty (offline /
unit
test) it is simply omitted so the labels still render.
Migration `49` backfills the labels on existing clusters so they heal on
upgrade
regardless of per-tenant `HelmRelease` state (e.g. suspended or failed
releases), and `migrations.targetVersion` is bumped to `50` so the
runner
iterates it. The migration derives the same chain from each namespace
name,
is idempotent (`kubectl label --overwrite`) and best-effort
(per-namespace
failures are tolerated so a transient apiserver error cannot abort the
upgrade).
## Test plan
- [x] `helm template packages/apps/tenant` at depth 1/2/3 — emits
exactly the
expected ancestor set at each depth: depth-1 `{tenant-root,
tenant-ktj}`,
depth-2 adds `tenant-ktj-htdev`, depth-3 adds `tenant-ktj-htdev-foo`. No
missing, no stray labels.
- [x] `helm unittest packages/apps/tenant` — new
`tests/namespace_ancestor_labels_test.yaml` pins the chain at depths
1/2/3
plus a negative assertion; whole suite green.
- [x] `bats hack/cozystack-version-stamp.bats` — migration `49` stamps
the
version through the shared `stamp_cozystack_version` helper (all 10
tests
pass, including the guard that migrations >= 42 must use the helper).
- [x] Migration `49` ancestor derivation verified under `/bin/sh` to
match the
helm output byte-for-byte, and dry-runs cleanly under `set -euo
pipefail`.
- [ ] Live cluster: create a 3-level tenant (`root -> ktj -> htdev`),
verify
`tenant-ktj-htdev` has all three `tenant.cozystack.io/*` labels.
- [ ] Live cluster: create a `kubernetes` tenant under `htdev` with
`addons.ingressNginx.hosts`, verify an external HTTP request reaches the
tenant ingress controller via the root-ingress proxy (no `upstream timed
out`).
## Closes
- Closes #2810
- Closes #2171
```release-note
fix(tenant): tenant namespaces now carry the full `tenant.cozystack.io/<ancestor>` label chain (including `tenant-root`) at any nesting depth, restoring root-ingress reachability to nested tenants; a migration backfills the labels on existing clusters
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Improved tenant namespace label inheritance across multi-level tenant
hierarchies, ensuring all ancestor tenant labels are consistently
applied.
* **New Features**
* Standardized ancestor label generation to reliably include the full
tenant chain for derived tenant namespaces.
* **Tests**
* Added regression coverage for ancestor label presence and absence at
multiple hierarchy depths.
* **Chores**
* Added a migration to backfill missing ancestor labels for existing
tenant namespaces during platform upgrades.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Ivan Okhotnikov
|
2ac269a04b |
docs(vertical-pod-autoscaler): fix stale ~350d in checksum/tls-cert ascii timeline
Second self-review pass caught a remaining reference to the old rotation math in the checksum/tls-cert doc block's ascii timeline (`renewal (~350d)` — from `renewBefore: 360h` when 365 - 15 = 350). With the widened `renewBefore: 2160h` rotation now lands at day ~275, matching the bullet-list narrative directly below. Update the number in place and regenerate the vendored-chart patch. No functional change; one-line doc fix. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
d6e57d67e7 |
docs(vertical-pod-autoscaler): sync deployment.yaml trade-off narrative with widened renewBefore
Self-review caught a stale day-math block in deployment.yaml that still
described the previous `renewBefore: 360h` behaviour ("cert-manager
renews at day ~350 and NO Helm action happens in the following ~15
days"). After bumping `renewBefore` to 2160h in the previous commit,
rotation now lands at day ~275 with a ~90-day window before expiry.
Update the numbers in place and regenerate the vendored-chart patch so
`make update` re-applies the corrected text instead of silently
reverting to the stale description on the next chart refresh.
No functional change — cert rotation math already lives in
`certificate.yaml`; this only fixes the parallel rationale block a
future reader would read alongside it.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
79ce69a3b4 |
fix(vertical-pod-autoscaler): widen renewBefore + document selfSigned checksum, add existingSecret belt-and-suspenders assertion
Address the three non-blocking follow-ups on myasnikovdaniil's approval: * renewBefore: 360h → 2160h (15d → 90d). With webhook `failurePolicy: Ignore`, an expired in-memory cert silently drops admission traffic until a Helm action rolls the ReplicaSet. helm-controller does not re-render on reconcile ticks after the release converges, only on install/upgrade/rollback — a 15-day window ties admission-webhook liveness to release cadence. 90 days survives realistic release schedules on customer clusters. Rotate-and-restart requirement is unchanged; the window is widened. * Document on deployment.yaml why hashing only `tls.crt` is safe: the Issuer is `selfSigned`, so `ca.crt == tls.crt` and every rotation re-issues `tls.crt`. If a future refactor ever swaps the Issuer to a CA-backed one (external CA, ACME, Vault), the checksum MUST be extended to also cover `ca.crt` — otherwise a CA-only rotation would leave the self-registered MutatingWebhookConfiguration serving a stale caBundle until the next leaf rotation. * Add a belt-and-suspenders helm-unittest assertion pinning that under `existingSecret` the admission-controller pod-template carries NEITHER `checksum/tls-secret` (legacy amplifier) NOR `checksum/tls-cert` (cert-manager runtime lookup). Templates are already structurally gated; this catches a future refactor that accidentally moves either checksum outside its `if` gate. Regenerated patches/cert-manager-admission-tls.diff against upstream via `make update` flow so the vendored chart edits stay in sync with the patch file (avoids silent revert on the next update). Verified: 16/16 helm-unittest cases pass, `helm lint` clean, `helm template ... certManagerEnabled=true` renders `renewBefore: 2160h`, and re-applying the regenerated patch onto a fresh upstream pull produces a byte-identical chart tree. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
myasnikovdaniil
|
c5ff2178e7 |
docs(release): add changelog for v1.4.5 (#3083)
This PR adds the changelog for release `v1.4.5`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.4.5.md`.
|
||
|
myasnikovdaniil
|
6f39beffb6 |
docs(release): add changelog for v1.5.2 (#3162)
This PR adds the changelog for release `v1.5.2`.
✅ Changelog has been automatically generated in
`docs/changelogs/v1.5.2.md`.
|
||
|
cozystack-ci[bot]
|
ee29392584 |
docs: add changelog for v1.5.2
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
cozystack-ci[bot]
|
a8220fa21f |
docs: add changelog for v1.4.5
Signed-off-by: cozystack-ci[bot] <274107086+cozystack-ci[bot]@users.noreply.github.com> |
||
|
myasnikovdaniil
|
1e5f53881f |
fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122) (#3146)
## What Backports the upstream Kamaji fix [clastix/kamaji#1122](https://github.com/clastix/kamaji/pull/1122) (which fixes clastix/kamaji#1115) as a source patch on top of the pinned `26.3.6-edge` build. No version bump, no CRD/RBAC changes. This lands the fix on `main`; the `backport` / `backport-previous` labels fan it out to `release-1.5` and `release-1.4` via the backport automation. Supersedes the per-release PRs #3021 (release-1.5) and #3020 (release-1.4). ## Why Deleting a Tenant whose etcd `DataStore` has **0 referencing `TenantControlPlane`s** — e.g. a tenant deleted before any Kubernetes app / TCP is ever created in it — leaves the `DataStore` stuck `Terminating` with finalizer `kamaji.clastix.io/TenantControlPlane`. That hangs `helm uninstall <etcd>` (which waits by default), freezes the etcd HelmRelease at `status=uninstalling`, and wedges the tenant namespace in `Terminating` indefinitely. Tracked in #3018. Root cause: in `26.3.x-edge`, DataStore deletion drives an `AllowedDeletion` condition that only transitions `False → True`. With 0 referencing TCPs at deletion time the condition is never initialized, so the reconcile falls through to logging `DataStore can be safely deleted` and the deferred finalizer-removal (gated on `IsStatusConditionTrue`) never runs — with no requeue to retry. The pinned `26.3.6-edge` predates the upstream fix (merged 2026-04-14, first released in `26.4.4-edge`). ## Change Adds `packages/system/kamaji/images/kamaji/patches/fix-datastore-unused-deletion.diff`, applied by the existing `RUN git apply /patches/*.diff` step in the kamaji image build. The one-line guard adds the missing 0-TCP path: ```go - if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) { + if meta.IsStatusConditionFalse(..., DataStoreConditionAllowedDeletionType) || len(tcpList.Items) == 0 { ``` ## Test - `git apply --check` confirms the patch applies cleanly to the `26.3.6-edge` source pinned on `main`; after applying, the guard is present at `controllers/datastore_controller.go`. - Reproduced the deadlock on a v1.4.x cluster (tenant deleted before any TCP existed) and confirmed the manual unstick; this patch removes the need for it. Workaround for an already-stuck DataStore (0 TCP refs): `kubectl patch datastore <name> --type=merge -p '{"metadata":{"finalizers":null}}'`. Relates to #2885 (full kamaji bump for CVEs — supersedes this patch once it lands). ```release-note Fix tenant namespace stuck `Terminating` when an etcd DataStore with no referencing TenantControlPlanes is deleted (Kamaji datastore unused-deletion deadlock). ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Data stores are now recognized as unused even when no matching tenant control plane resources are found, improving deletion handling in that case. * This prevents inactive data stores from being left behind when there are no associated matches. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
myasnikovdaniil
|
5b261b9daf |
fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled (#3130)
## What this PR does
The `victoria-metrics-operator` HelmRelease in the `kubernetes` app
chart is rendered when `addons.monitoringAgents.enabled` is true, but it
listed `cert-manager-crds` in its `dependsOn` **unconditionally**. The
`cert-manager-crds` HelmRelease is only created when
`addons.certManager.enabled` is true. These are two independent toggles.
### The valid combination that triggers it
`addons.monitoringAgents.enabled: true` + `addons.certManager.enabled:
false` (both default-able, both schema-valid). In this configuration
vmop renders but the dependency it points at never does.
### Before / after
Before:
```yaml
dependsOn:
- name: {{ .Release.Name }}-cilium
namespace: {{ .Release.Namespace }}
- name: {{ .Release.Name }}-cert-manager-crds # always present
namespace: {{ .Release.Namespace }}
```
After:
```yaml
dependsOn:
- name: {{ .Release.Name }}-cilium
namespace: {{ .Release.Namespace }}
{{- if .Values.addons.certManager.enabled }}
- name: {{ .Release.Name }}-cert-manager-crds
namespace: {{ .Release.Namespace }}
{{- end }}
```
### The cascade
With cert-manager disabled, Flux reports on the vmop HelmRelease:
`unable to get '...cert-manager-crds' dependency: helmreleases... not
found`. Because vmop never goes Ready, the dependency chain fails behind
it: `monitoring-agents` (dependsOn `cozy-victoria-metrics-operator`) →
not Ready → `vertical-pod-autoscaler` (dependsOn `monitoring-agents`) →
not Ready.
### Why gating (not unconditional CRDs) is correct
vmop does not require cert-manager. The vendored vmop chart
(`packages/system/victoria-metrics-operator`) defaults
`admissionWebhooks.certManager.enabled` to `false`, and the cozystack
wrapper does not override it. With that default the chart self-generates
the webhook CA/cert (`genCA`/`genSignedCert` in `_helpers.tpl`); the
cert-manager Issuer/Certificate path is fully gated behind
`certManager.enabled`. So the dependency only belongs there when
cert-manager is actually enabled.
Principle: a `dependsOn` must never reference a HelmRelease that isn't
created under the same value combination.
### How it was verified
`helm template` of the template with a CI values fixture:
`certManager.enabled=false` → cilium only:
```yaml
dependsOn:
- name: test-cilium
namespace: tenant-root
```
`certManager.enabled=true` → cilium + cert-manager-crds:
```yaml
dependsOn:
- name: test-cilium
namespace: tenant-root
- name: test-cert-manager-crds
namespace: tenant-root
```
A new `helm unittest` suite
(`tests/victoria-metrics-operator_test.yaml`) asserts the `dependsOn`
for both toggle states. Full package suite passes (13 suites / 125
tests).
### Release note
```release-note
fix(kubernetes): enabling monitoring agents without cert-manager no longer blocks the victoria-metrics-operator HelmRelease (and the monitoring-agents / vertical-pod-autoscaler chain behind it) on a non-existent cert-manager-crds dependency
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
## Summary by CodeRabbit
* **New Features**
* Added automated template test coverage for the Victoria Metrics
Operator HelmRelease.
* Validates rendered release name and dependency behavior across key
configuration scenarios.
* **Bug Fixes**
* Updated the HelmRelease dependency wiring so the cert-manager CRDs
dependency is included only when cert-manager is enabled.
* Preserved existing monitoring and outer conditional behavior while
refining dependency rendering.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
Myasnikov Daniil
|
47638f8569 |
fix(kamaji): backport datastore unused-deletion deadlock fix (clastix/kamaji#1122)
Deleting a Tenant whose etcd DataStore has 0 referencing TenantControlPlanes left the DataStore stuck Terminating (finalizer kamaji.clastix.io/TenantControlPlane), hanging the etcd helm uninstall and wedging the tenant namespace. Backports clastix/kamaji#1122 (fixes clastix/kamaji#1115) as a source patch on the pinned 26.3.6-edge build. Refs #3018. Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
myasnikovdaniil
|
dad78bd57e |
fix(ci): nest backport conflict_resolution under experimental input (#3155)
## What this PR does The Automatic Backport workflow passed `conflict_resolution: draft_commit_conflicts` as a **top-level** input to `korthout/backport-action`, but the action only reads `conflict_resolution` from inside its `experimental` JSON input. Passed top-level it was silently ignored — the action run logs warn `Unexpected input(s) 'conflict_resolution'` — so it fell back to the `experimental` default of `fail`. On a cherry-pick conflict the action then created **no** pull request and only left a "Backport failed" comment on the source PR (this bit #3121 and #3119). This moves the key into the `experimental` input, so a conflicting backport instead opens a **draft PR with the first conflict committed** for maintainers to resolve in place. No version bump: the pinned `v3.2.1` already supports `draft_commit_conflicts` — the bug was purely the input placement. ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Backport pull requests now handle conflicts more reliably and open as draft PRs when conflicts are detected. * Improved configuration handling so the expected conflict behavior is applied consistently during backport automation. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
IvanHunters
|
59964043e3 |
fix(platform): migration 49 backfills tenant ancestor labels
The tenant chart fix makes new and re-reconciled tenant namespaces self-compute their full tenant.cozystack.io/<ancestor> chain, but existing namespaces whose HelmRelease does not promptly re-reconcile (suspended or failed releases) keep the incomplete labels — and with them the broken <ancestor>-egress reachability that surfaces as upstream timeouts to nested tenants. Backfill the labels directly so clusters heal on upgrade regardless of per-tenant HelmRelease state. Bump targetVersion to 50 so run-migrations.sh iterates the new script (seq CURRENT..49); the missing-file guard otherwise never fires. Best-effort and idempotent: kubectl label --overwrite adds only the listed labels, and per-namespace failures are tolerated so a transient apiserver error cannot abort the platform upgrade. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
IvanHunters
|
1ac9398a4b |
fix(tenant): emit full ancestor label chain on tenant namespaces
The tenant.cozystack.io/<ancestor> namespace labels drive the <ancestor>-egress CiliumClusterwideNetworkPolicy, which lets an ancestor tenant reach a descendant namespace only when the descendant carries tenant.cozystack.io/<ancestor>. namespace.yaml derived that chain by splitting .Release.Namespace, which only reached one level up: a direct child of tenant-root got tenant-root, but a grandchild (tenant-ktj-htdev under tenant-ktj) got only tenant-ktj and dropped tenant-root entirely. The root ingress controller in tenant-root then could not reach nested tenants, surfacing as upstream timeouts. Derive the chain deterministically from the namespace name instead. Every tenant descends from tenant-root, and tenant.name builds each child as <parent-namespace>-<word>, so every progressive dash-prefix of the name is a real ancestor namespace. This needs no cluster lookup, renders offline (so it is unit-testable), and converges on every reconcile regardless of parent/child HelmRelease ordering. The ownerReference lookup is made non-fatal so the template still renders when the parent namespace lookup is empty; in a real reconcile the parent always exists and the ownerReference is always emitted. Signed-off-by: IvanHunters <xorokhotnikov@gmail.com> |
||
|
Myasnikov DaniilandClaude Opus 4.8
|
2b1d3e904d |
fix(kubernetes): gate vmop cert-manager-crds dependency on certManager.enabled
The victoria-metrics-operator HelmRelease is gated on
monitoringAgents.enabled but listed cert-manager-crds in dependsOn
unconditionally, while the cert-manager-crds HelmRelease is only
created when certManager.enabled is true. The valid combination
monitoringAgents.enabled=true + certManager.enabled=false therefore
left vmop blocked on a non-existent dependency
("...cert-manager-crds ... not found"), cascading to monitoring-agents
and vertical-pod-autoscaler. Gate the dependency on the same condition
that creates it.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
|
||
|
scooby87
|
71ca2b1bae |
fix(mariadb): re-guard replication so single-replica passes the operator webhook (#3180)
## What this PR does Fixes a regression where **single-replica MariaDB can no longer be created**. [PR #2279](https://github.com/cozystack/cozystack/pull/2279) made `replication.enabled: true` unconditional in the MariaDB CR so the operator would always create `<release>-primary` / `<release>-secondary` Services (expected by the dashboard RBAC and the ApplicationDefinition). But the vendored mariadb-operator's validating webhook (`vmariadb-v1alpha1.kb.io`) **rejects `replication.enabled: true` when `spec.replicas == 1`** — replication needs a primary + at least one replica. As a result, single-replica installs never became Ready (the HelmRelease failed at admission). The webhook rule lives in the operator's Go code (not in the vendored CRD — there is no CEL rule tying replication to replica count), so the failing path is reproduced by an e2e test rather than a schema assertion. ### Fix — re-guard replication, teach consumers about the bare service The operator provisions a bare `<release>` Service in **both** topologies; `-primary`/`-secondary` exist **only** when replication is on (`replicas>1`). Instead of forcing replication for everyone, this PR lets replication track the replica count and makes the service-name consumers handle the single-replica bare service: - **`mariadb.yaml`** — gate `replication` on `replicas>1`; restore single-replica external access via `service.type: LoadBalancer`, and keep `primaryService: LoadBalancer` for `replicas>1` only. (Effectively reverts #2279's two hunks.) - **`backup-cronjob.yaml`** — single-replica dumps target the bare service (no `-secondary` exists without replication); `replicas>1` still offloads to `-secondary`. - **`dashboard-resourcemap.yaml`** — always grant RBAC on the bare service, plus `-primary`/`-secondary` when `replicas>1`, matching what the ApplicationDefinition lists. - **`mariadb-rd`** — list the bare service in `services.include` so single-replica installs are visible in the dashboard. - **`hack/e2e-apps/mariadb.bats`** — new `Create single-replica MariaDB` test that reproduces the webhook rejection and guards against regression. Verified: `helm template` at `replicas=1`/`2` renders the intended CR/RBAC/backup host; `helm unittest` passes; `update-crd.sh` keeps the RD stable; `select-e2e.sh` maps the change to the mariadb suite. ### Release note ```release-note fix(mariadb): single-replica instances can be created again — replication is now enabled only for replicas>1 (matching the operator webhook), while the dashboard, RBAC, and backups correctly target the bare service for single-replica setups ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an end-to-end test covering MariaDB deployments with a single replica, including validation of the main service, endpoints, StatefulSet replica count, and metrics components. * **Bug Fixes** * Updated MariaDB chart behavior to correctly handle single-replica vs multi-replica configurations for external access, replication settings, and backup host selection. * Refined dashboard RBAC and secret inclusion so access applies to the main MariaDB service in addition to replica-specific services. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Ivan Okhotnikov
|
b40703503c |
docs(vertical-pod-autoscaler): correct first-upgrade rollout attribution in certificate.yaml
Kvaps' approval comment flagged that certificate.yaml attributed the migration-upgrade rollout to the checksum/tls-cert value flipping. That is technically inaccurate: at first-upgrade render time, \`lookup\` runs BEFORE Helm deletes the legacy Opaque Secret, so the annotation's value at that moment is the sha of the old Secret content, not the yet-to-be-written cert-manager one. The rollout still happens on the first upgrade — but because the pod-template itself carries a structural change: the pre-migration chart did not emit \`checksum/tls-cert\` at all, and the new chart does. That new-annotation-present is what Helm sees as a podTemplate diff and rolls the ReplicaSet. The annotation's VALUE only catches up on the NEXT Helm action (when lookup re-fires and sees the cert-manager-written Secret). That is fine on first upgrade because the structural pod-template change already turns the ReplicaSet over, and the fresh pod re- registers the MutatingWebhookConfiguration with the new caBundle after cert-manager has written the new Secret. Subsequent renewal- driven rotations (or manual \`flux reconcile hr\`) are driven by the annotation VALUE flip, as before. Comment rewritten to lay this out explicitly. Patch file regenerated to include the comment change. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Andrei Kvapil
|
41fd08e2ba |
feat(keycloak): enable UPDATE_EMAIL required action in cozy realm (#3175)
## What this PR does The portal user-settings page lets a user change their email via a Keycloak `kc_action=UPDATE_EMAIL` redirect. That flow only works when the **UPDATE_EMAIL required action is enabled at the realm level** — and on a fresh realm it is registered but **disabled by default**. `update-email` is a supported, default-on feature since Keycloak 26.4 (keycloak/keycloak#39722), so the provider is already loaded on the 26.x server this repo ships — **no `KC_FEATURES` change and no Keycloak restart are needed**. The `edp-keycloak-operator` has no CRD field for required actions, so this adds a small one-shot, idempotent `post-install`/`post-upgrade` Job to `keycloak-configure` that enables it via `kcadm.sh`, waiting for the operator to reconcile the realm first. Verified against a live Keycloak 26.5.2: the `UPDATE_EMAIL` action goes from `enabled=false` to `enabled=true`, and re-running the Job is a no-op. ### Release note ```release-note feat(keycloak): enable the UPDATE_EMAIL required action in the cozy realm so users can change their email from the portal ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Keycloak now automatically enables the email update action during install and upgrade, helping users change their email address in the `cozy` realm. * The configuration now includes the required Keycloak container image for this setup task. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
e4c6802863 |
fix(cozy-lib,harbor,mariadb): digest-pin cleanup-hook kubectl image and route it through cozy-lib.images-registry (#3171)
## What this PR does Fixes #3088. The post-delete cleanup Jobs in the **harbor** and **mariadb** charts hardcoded the cleanup image as `docker.io/clastix/kubectl:v1.32` — a moving Docker Hub tag. This: - **broke reproducibility / was a supply-chain risk** — `v1.32` is mutable and can be re-pushed to a different image; - **could not be satisfied on air-gapped / mirrored clusters** — the hardcoded `docker.io` bypassed the cluster images registry, and it failed at the worst moment: during **uninstall**, when the `post-delete` hook runs and a stuck Job stalls release teardown. ### Changes - **New canonical helper `cozy-lib.image`** (`packages/library/cozy-lib/templates/_cozyconfig.tpl`). Given a registry-relative image path and the global context, it prefixes `cozy-lib.images-registry` when set (`<registry>/<image>`) and returns the image unchanged when empty — **with no leading `/`**, so standard installs still resolve from the default registry. This establishes the previously-undefined "how to reference a cluster image" pattern (`images-registry` was defined but never consumed). - **Digest-pinned image** in both cleanup hooks: `clastix/kubectl:v1.32@sha256:b9ef7d8dbe65bcc81a46c09b8dc7543103055021c4f43287bf59e92a8f4fe05c` (the immutable multi-arch OCI index for the `v1.32` tag; the same digest already pinned for `postgres-operator`'s webhook hook). - **Renovate keeps the pin fresh.** The repo disables the helm-values manager (`enabledManagers`), and the ref is assembled at render time by the helper rather than being a Dockerfile / go.mod dependency, so the built-in managers can't see it. A `custom.regex` manager over the hook templates tracks `clastix/kubectl` by tag+digest — mirroring the existing `extra/etcd` manager — so the supply-chain pin doesn't silently age. - **Hardened `cozy-lib.images-registry`** against a nil `.Values._cluster` so the helper never crashes the render — relevant precisely because this hook renders during teardown. - **Helm unit tests** in both charts assert the rendered image for three cases: empty registry (default), registry set, and the no-leading-slash edge case. Assertions match by **pattern** (routed + digest-pinned + no leading slash) rather than an exact digest, so a Renovate bump keeps them green. ### Verification - `helm unittest` passes for both charts (harbor: 16, mariadb: 10). - Real `helm template` renders confirmed for both charts: - empty registry → `clastix/kubectl:v1.32@sha256:b9ef…` - `images-registry=registry.internal:5000` → `registry.internal:5000/clastix/kubectl:v1.32@sha256:b9ef…` - The Renovate `matchStrings` regex verified to capture `currentValue=v1.32` and the digest from both hook templates. ### Note on the wider footprint The same `docker.io/clastix/kubectl:v1.32` hardcode is already merged on `main` in several other cleanup hooks (bucket, qdrant, gateway, seaweedfs, kubernetes, tenant, dashboard, keycloak-configure) and appears in the open PRs #3170 (etcd), #3094 (monitoring), #3072 (clickhouse). This PR fixes harbor + mariadb per the issue and establishes the `cozy-lib.image` + Renovate pattern; migrating the remaining hooks is a follow-up. ### Release note ```release-note fix(cozy-lib,harbor,mariadb): digest-pin the harbor and mariadb post-delete cleanup-hook kubectl image and route it through the cluster images registry (`cozy-lib.images-registry`) via the new `cozy-lib.image` helper, so mirrored / air-gapped installs can resolve it and uninstall no longer depends on a moving Docker Hub tag; a custom Renovate manager keeps the digest fresh ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code) <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * Updated post-delete cleanup hook Jobs to use digest-pinned container images for more secure, repeatable execution. * Improved image reference rendering so cleanup hooks work correctly with or without an image registry configured, avoiding invalid leading/trailing slash issues. * **Tests** * Extended cleanup hook rendering tests to validate digest pinning and correct registry prefix behavior, including edge cases. * **Chores** * Enhanced automated dependency detection to treat digest-pinned hook image references as stable. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
scooby87
|
6bf836db30 |
fix(dashboard): show LoadBalancer external IP on the app Services tab (#3172)
## What this PR does For an application exposed with `external: true` (which creates a `type: LoadBalancer` Service), the app detail → **Services** tab showed the service Type and Cluster IP but not the external IP assigned by MetalLB — so there was no way to learn the connection address from the per-app view. The pre-1.4.0 dashboard showed this, so it read as a regression (reported in #2789). Root cause: `ServicesTab.tsx` queried Services with a spec-only local type (`{ type, clusterIP, ports }`) and never read `status.loadBalancer.ingress[]`. The data was already reachable — `ExternalIpsPage.tsx` renders it — so this was a wiring gap in the per-app tab, not missing data. Changes: - Add a `ServiceStatus` type and extend the `useK8sList` query to `K8sResource<ServiceSpec, ServiceStatus>`. - Add an **External IP** column: `ip ?? hostname ?? "Pending"` for `LoadBalancer` services, `—` otherwise — mirroring `ExternalIpsPage`. - Add a co-located regression test (`ServicesTab.test.tsx`) covering the IP, hostname→Pending fallback, and the ClusterIP no-external case. Verified: `pnpm typecheck` clean, `pnpm test` green (320 tests, incl. 3 new), `eslint` clean on the changed files. Fixes #3111 ### Screenshots <img width="1220" height="572" alt="Снимок экрана 2026-07-02 в 14 12 11" src="https://github.com/user-attachments/assets/82cd1da8-ff81-4b95-a6bf-18729b6c85c7" /> ### Release note ```release-note fix(dashboard): show the external LoadBalancer IP on the per-app Services tab ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * Added an **External IP** column to the service details view. * Load balancer services now show the external IP or hostname when available. * **Bug Fixes** * Fixed the external IP display for load balancer services so it no longer misses valid IPs. * Improved fallback behavior by showing **Pending** when no external endpoint is assigned. * Non-load balancer services now show an em dash instead of misleading external IP text. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
Ivan Okhotnikov
|
73c73df9ee |
docs(vertical-pod-autoscaler): clean up dangling references to the retired migrator/rollout hooks
Self-review after the tls-migrator removal caught a stale comment in certificate.yaml still promising that a pre-upgrade hook would strip Helm/Flux ownership so cert-manager could adopt the legacy Secret in place. That hook no longer exists and — as kvaps' review pointed out — could not have done what it claimed anyway: \`Secret.type\` is immutable, so cert-manager cannot rewrite an Opaque Secret in place into a \`kubernetes.io/tls\` one. Rewrite the certificate.yaml secretName comment to describe the actual migration path: Helm garbage-collects the legacy Opaque Secret from the release manifest, cert-manager immediately writes a fresh \`kubernetes.io/tls\` Secret of the same name via this Certificate, and the same-upgrade rollout — driven by the fresh \`checksum/tls-cert\` annotation — brings up a pod that re-registers \`vpa-webhook-config\` with the new caBundle before the delete/ create window on the MWC matters (given failurePolicy: Ignore). Also refresh patches/cert-manager-admission-tls.diff to reflect the comment edit and re-verify end-to-end that \`rm -rf charts && helm pull && patch\` produces a byte-identical tree to what git tracks. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
5e65a3e792 |
docs(vertical-pod-autoscaler): tighten cert-manager comments per review
Kvaps' review on #3178 called out three imprecisions in the supporting comments. Fix them without touching runtime behaviour: 1. certificate.yaml previously emitted a secretTemplate whose labels include \`app.kubernetes.io/managed-by: Helm\`. Helm does NOT own the Secret written by cert-manager — the Certificate resource does — and stamping that label on it misleads operators reading the object with \`kubectl get secret -oyaml\`. Drop secretTemplate; cert-manager labels its outputs unambiguously (\`cert-manager.io/*\`, \`controller.cert-manager.io/*\`). 2. deployment.yaml comment on checksum/tls-cert claimed the lookup \"resolves at every reconcile\". Not true. \`lookup\` runs only on Helm actions (install / upgrade / rollback); helm-controller does not re-render a deployed release on interval ticks. Under cozystack's release cadence this is fine — any real chart bump, values change, or manual \`flux reconcile hr\` triggers a re-template within days — but at day ~350 renewal the pod continues serving the certificate it loaded at startup until the next Helm action. If no Helm action lands in the ~15 days between renewal and expiry the webhook silently stops mutating. The comment now spells that trade-off out so a future reader can find it, and points at a small post-renewal Job as the follow-up without re-introducing the ReplicaSet-churn amplifier the whole PR retires. 3. values.yaml comment on certManagerEnabled did not warn that the switch is effectively one-way. Once cert-manager has written the Secret with \`type: kubernetes.io/tls\`, flipping the flag back to false and rolling back to the legacy Opaque path fails — \`Secret.type\` is immutable, so the legacy Secret template cannot overwrite the cert-manager-owned one in place. The doc now says operators must \`kubectl delete\` the Secret first if they need to roll back. Also refreshed patches/cert-manager-admission-tls.diff to reflect these comment-only edits and re-verified end-to-end that \`make update\`-style pull + patch produces a byte-identical tree to what git tracks now. 15/15 helm-unittest cases still pass. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
039bb7e180 |
fix(vertical-pod-autoscaler): carry cert-manager mods as a patches/*.diff on the vendored chart
Kvaps' review on #3178 flagged that this package's \`make update\` target does \`rm -rf charts && helm pull cowboysysop/vertical-pod- autoscaler --version 11.1.1\` with no re-apply step. Any local edits to the vendored subchart were therefore one \`make update\` away from vanishing without warning — exactly what the fluxcd-operator package pre-empts via a \`patches/*.diff\` re-apply loop after the helm pull. Adopt the same convention here: * New file \`patches/cert-manager-admission-tls.diff\` capturing all four modifications the cert-manager path makes to the vendored subchart — the new \`certificate.yaml\` template, the checksum/tls-secret + new checksum/tls-cert annotation gating on the admission-controller Deployment, the certManagerEnabled guard on the legacy \`tls-secret.yaml\`, and the new \`certManagerEnabled: true\` default in the subchart values. * Makefile \`update\` target now runs \`patch --no-backup-if-mismatch -p1 < patches/*.diff\` after the helm pull, so a future upstream bump (or a routine \`make update\` today) re-applies the cozystack modifications over the freshly-fetched vendored files and leaves the chart in the same state git tracks. Verified end-to-end: with the on-disk chart intact I ran cp -r charts/vertical-pod-autoscaler /tmp/reference rm -rf charts helm pull cowboysysop/vertical-pod-autoscaler --untar --untardir charts --version 11.1.1 patch --no-backup-if-mismatch -p1 < patches/cert-manager-admission-tls.diff diff -r charts/vertical-pod-autoscaler /tmp/reference The diff is empty — \`make update\` now produces a byte-identical tree to what git has right now, and the 15 helm-unittest cases still pass unchanged. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
0c4525afe2 |
fix(vertical-pod-autoscaler): drop tls-migrator hook — it was a no-op
Kvaps' review on #3178 correctly identified that the pre-upgrade tls-migrator Job cannot do what its comments claim, for two independent reasons: 1. Helm decides what to delete on upgrade from the STORED release manifest, not from live labels on the object. The only live- object opt-out is the \`helm.sh/resource-policy: keep\` annotation — see the \`original.Difference(target)\` loop in helm's pkg/kube/client.go, which only checks \`ResourcePolicyAnno\`. Stripping \`meta.helm.sh/*\` and \`app.kubernetes.io/managed-by\` does not prevent the deletion; those markers only matter for adoption checks on resources entering a manifest, not on resources leaving it. Likewise, helm-controller has no independent label-based prune for Helm-managed resources — the \`helm.toolkit.fluxcd.io/*\` labels are tracking metadata only, not a deletion gate. 2. Even if the Secret survived the Helm/Flux delete pass, cert-manager still could not adopt it in place: the legacy Secret has \`type: Opaque\`, cert-manager writes \`type: kubernetes.io/tls\`, and Secret.type is immutable in Kubernetes. The Certificate reconcile would fail with \"field is immutable\" and never become Ready. The migration converges anyway because the Job is a no-op: Helm deletes the old Opaque Secret, cert-manager mints a fresh one of the right type, and the Deployment rolls in the same upgrade (its podTemplate carries the new \`checksum/tls-cert\` annotation, which is the sha of the runtime Secret and therefore differs the first time cert-manager writes new content). The webhook's \`failurePolicy: Ignore\` makes the seconds-long window benign. Delete the migrator entirely: template file, RBAC (ServiceAccount + Role + RoleBinding), the hardcoded clastix/kubectl image reference, and the four helm-unittest cases pinning its hook wiring / RBAC scope. ~200 lines removed, one less pinned image, and the documented migration flow now matches what actually happens. Existing helm-unittest suite: 15/15 pass (was 19 with the migrator tests; removed 4 that referenced the retired template). Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
78e199a75e |
fix(vertical-pod-autoscaler): rotate admission-controller pods via checksum/tls-cert on Secret content, not via kubectl rollout hook
Retiring the tls-rollout Job. Every Flux HelmRelease reconcile
re-fires the post-install/post-upgrade hooks, and the hook stamped a
fresh `kubectl.kubernetes.io/restartedAt` timestamp into the
Deployment podTemplate at each firing. That mutation spawns a new
ReplicaSet on every reconcile — the exact ReplicaSet-churn pattern
this whole change was meant to end, resurrected from a different
angle. CI run 28584337959 showed 10 admission-controller
ReplicaSets accumulated in ~20 minutes with no Secret content
change; the cert-manager Certificate was Ready and stable, the pod
churn was entirely driven by the hook itself.
Replace it with a declarative `checksum/tls-cert` podAnnotation on
the admission-controller Deployment. The annotation hashes the
runtime Secret's `tls.crt` via `lookup`, so it mutates only when
the Secret content actually mutates:
* fresh install: Secret not yet present at first render;
annotation absent; pod comes up on whichever
Secret cert-manager writes next.
* steady state: Secret content stable → sha stable → no
rotation on any Flux reconcile.
* cert-manager adopt: cert-manager rewrites Secret once on the
first upgrade (issuer change) → sha flips
→ pod restarts ONCE and picks up the new
CA + cert.
* renewal (~350d): cert-manager rotates the Secret → sha flips
→ pod restarts ONCE.
`lookup` returns nil during offline `helm template --dry-run=client`
(CI, unit tests) — the annotation just does not appear in that mode,
which is fine because the pod does not need to rotate in that mode
either. Flux HelmController runs the Helm SDK against the live
apiserver, so `lookup` resolves at every reconcile there.
The tls-migrator pre-upgrade hook is retained. It performs a
one-shot ownership transfer (strip Helm/Flux labels so cert-manager
can adopt the existing Secret without a delete/create race). It
does not stamp any mutable timestamps into the Deployment, so it
does not amplify. It is idempotent (no-op on fresh install and
after the first successful migration).
helm-unittest updated:
* dropped the four assertions that pinned the tls-rollout Job's
hook wiring and RBAC (template no longer exists);
* added an assertion that `checksum/tls-cert` is absent when the
tests run offline (lookup returns nil, exactly as intended).
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
||
|
Ivan Okhotnikov
|
997224a553 |
fix(vertical-pod-autoscaler): tls-rollout waits on the Certificate's real name
certificate.yaml renders the Certificate with name <fullname>-tls (its secretName matches the legacy chart-owned Secret exactly), but the rollout hook was passing plain <fullname> to `kubectl wait` — so every post-install/post-upgrade cycle sat 3 minutes on NotFound, failed, and burned through the Job's backoffLimit. cozystack-platform then aborted its HR reconcile because the VPA release's post-upgrade hook was Failed. Snapshot from the first CI run of #3178 confirms: - Certificate object: vertical-pod-autoscaler-admission-controller-tls (namespace cozy-vertical-pod-autoscaler) - Rollout Job pods: 10+ BackOff / BackoffLimitExceeded - HR condition: post-upgrade hooks failed - Admission-controller Deployment itself: 1 ReplicaSet, 1 Ready pod (the actual chart change works — churn is gone; only the rollout trigger was broken) Fix propagates the -tls suffix through both places that reference the Certificate: * The rollout container's `cert` variable used in `kubectl wait`. * The Role's resourceNames on cert-manager.io/certificates so RBAC still admits the get/watch/list against the actual object. Regression test in vpa_test.yaml asserts rules[0].resourceNames[0] carries the -tls suffix so this exact off-by-suffix bug cannot come back through a future rename. Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io> |
||
|
Ivan Okhotnikov
|
522fb534a7 |
test(vertical-pod-autoscaler): helm-unittest coverage for cert-manager migration paths
Locks in the four configurations the chart must support:
1. certManagerEnabled=true (default): Certificate + Issuer render
with the same DNS shape (`vpa-webhook.<ns>.svc`) and validity
(365d) the legacy genSignedCert produced, so cert-manager
adoption of an existing Secret rotates content once and stays
stable. Legacy Secret is suppressed. The checksum/tls-secret
podAnnotation — the ReplicaSet-churn amplifier of the legacy
path — is absent. A future refactor that reintroduces that
annotation on the cert-manager branch would silently regress
to the same infinite rotation loop from a different angle;
this test blocks that.
2. certManagerEnabled=false: opt-out for operators without cert-
manager. Legacy Secret must be back AND the
checksum/tls-secret annotation must be back — either one alone
leaves admission-controller pods failing to mount.
3. existingSecret set: operator supplies TLS out-of-band. Chart
mints nothing (no Certificate, no Issuer, no legacy Secret, no
migration or rollout hook). Preserves the pre-existing chart
contract.
4. admissionController.enabled=false (nested vpa-for-vpa release):
no admission-controller means no TLS Secret is needed. Every
admission-controller template must render zero documents,
otherwise the nested release would apply stray
Certificate/Issuer/Job manifests into cozy-vpa-for-vpa for no
reason.
Also locks the hook wiring for tls-migrator (pre-upgrade, weight -10)
and tls-rollout (post-install,post-upgrade, weight 10) and asserts
the RBAC scoping-by-resourceName so a future edit cannot accidentally
grant broader Secret/Deployment access than the Job requires.
All 22 tests pass locally under helm-unittest.
Signed-off-by: Ivan Okhotnikov <ivan.okhotnikov@aenix.io>
|
