## Summary
- Replace `GH_PAT` (cozystack-bot PAT) with `cozystack-ci` GitHub App
token in `pull-requests.yaml`
- This was missed in #2351 and broke the `resolve_assets` / `e2e` jobs
for release PRs (draft releases not visible with invalid token)
## Test plan
- [ ] Re-run the release PR pipeline for `release-1.1.6` and verify
`resolve_assets` job passes
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated continuous integration authentication mechanism to use GitHub
App tokens instead of personal access tokens.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The GH_PAT secret tied to cozystack-bot is no longer valid after
migrating release workflows to the cozystack-ci GitHub App in #2351.
This broke the resolve_assets and e2e jobs for release PRs.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Adds a host preflight diagnostic script (`hack/check-host-runtime.sh`)
that warns operators when a standalone `containerd.service` or
`docker.service` is running on the host alongside the embedded k3s
runtime used by the cozystack `generic` variant (k3s / kubeadm on
Ubuntu).
**Why it matters.** K3s ships its own containerd at
`/run/k3s/containerd/containerd.sock` and
`/var/lib/rancher/k3s/agent/containerd`, while a system-package
containerd or docker uses `/run/containerd/containerd.sock` and
`/var/lib/containerd`. The two runtimes do not fight over sockets, so
both keep running silently. Over time the standalone one accumulates
unpruned images and build cache in `/var/lib/containerd` — enough to
fill the root disk, trigger `DiskPressure`, and put `cozystack-api` into
an eviction loop. This is silent on day zero and surfaces as a
mysterious production incident weeks later. The script exists to warn
the next operator before the failure mode surfaces.
The script is warning-only — it always exits 0 and never blocks the
install. It detects:
- `containerd.service` or `docker.service` active via `systemctl
is-active`
- Standalone runtime sockets at well-known paths, with a fallback that
works on hosts without systemd
- Standalone data directory sizes via `du -sh`
When a warning fires, the HINT names only the detected services and
instructs the operator to disable them with `sudo systemctl disable
--now <service>`. Reclaiming the data directory is called out separately
with an explicit note not to delete it blindly — the data may still be
in use.
**Entry points.**
- `make preflight` runs the script directly (for operators preparing a
generic-variant host)
- `make unit-tests` now runs `bats-unit-tests` alongside
`helm-unit-tests`, auto-discovering every `hack/*.bats` file that is not
an e2e test
**Test coverage.** `hack/check-host-runtime.bats` (11 cases, run via
`hack/cozytest.sh`) covers clean hosts with and without systemd,
single-service detection, both services simultaneously, socket-only
fallback for both runtimes, glob-expansion regression guard, explicit
exit-code-0 assertion, `sudo` prefix assertion, `du` failure robustness,
and the "service + socket = exactly one warning" de-duplication
invariant. Every test is self-contained with `trap 'rm -rf $STUB_DIR'
EXIT` for clean recovery on assertion failure, and has no runtime
dependencies beyond bash and core utilities — no python3, no real
systemd.
Irrelevant on Talos where the container runtime lifecycle is fully
managed by the distribution.
### Release note
```release-note
[hack] Add `check-host-runtime.sh` and `make preflight` target that warn when a standalone containerd or docker runtime is running alongside the embedded k3s runtime on the cozystack generic variant.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* Added a non-blocking preflight check that detects standalone container
runtimes, reports disk-usage estimates, and displays an actionable hint
to disable detected services.
* **Tests**
* Added BATS-based unit tests with comprehensive coverage for the
preflight validations and various host scenarios.
* **Chores**
* Build updated to run the BATS unit tests alongside the existing test
suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
The aggregated `cozystack-api` validates each individual Tenant name
length against the Helm release limit (46 chars for the `tenant-`
prefix), but it never validates the **computed workload namespace** that
is built by dash-joining the parent namespace with the tenant name. A
user can therefore `kubectl apply` a nested `Tenant` whose own name
passes the per-name check, only to have Kubernetes later reject the
generated `Namespace` for exceeding the 63-character DNS-1123 label
limit. The failure surfaces as an opaque HelmRelease reconcile error,
and the Tenant CR is left stranded in the cluster with a
`status.namespace` that can never exist.
This PR closes that gap at the aggregation layer by rejecting such
requests synchronously in `Create()` with a clear `field.Invalid` error
that names the computed namespace and its length, so the failure is
reported at `kubectl apply` time before any HelmRelease is created.
The change is scoped to the `Tenant` kind only. `Update()` is not
touched because Kubernetes names and namespaces are immutable and the
existing code already intentionally skips name validation there. No Helm
chart, CRD, or codegen changes are required.
The PR is split into two commits that document the bug and its fix via
TDD:
1. **Pin the gap** — adds a test that demonstrates the pre-fix behavior:
the pre-fix Create() path has no function that rejects a realistic
nested-tenant combination whose computed namespace exceeds 63 chars.
Reviewers can check out this commit and observe the gap.
2. **The fix** — adds `validateTenantNamespaceLength`, the `kindName ==
"Tenant"` gate in `Create()`, and `TestValidateTenantNamespaceLength`
covering boundary cases (pass at 63, fail at 64, fail at 68). The
pinning test from commit 1 is removed in the same commit now that the
gap is closed.
### Release note
```release-note
[platform] cozystack-api now rejects Tenant creation at admission time when the ancestor-chain namespace would exceed the 63-character Kubernetes namespace limit, instead of allowing the resource through and failing later at HelmRelease reconcile.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Bug Fixes**
* Tenant application creation now validates computed tenant namespace
length against the Kubernetes DNS-1123 63-character limit; creations
that would produce too-long namespace names now fail with a clear
validation error.
* **Tests**
* Added tests covering tenant namespace length validation, including
multiple parent/tenant-name scenarios, boundary conditions, and
verification that error messages include the offending namespace and its
length.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
<!-- Thank you for making a contribution! -->
## What this PR does
Cilium 1.19 init containers `mount-cgroup`, `apply-sysctl-overwrites`
and `clean-cilium-state` call `nsenter` to join the host cgroup/mount
namespace. On any distribution that loads the
`cri-containerd.apparmor.d` AppArmor profile by default for containerd
workloads (Ubuntu 22.04+, Debian with AppArmor), the kernel denies the
operation and the audit log reports it as a blocked `ptrace` operation:
```text
apparmor="DENIED" operation="ptrace" class="ptrace"
profile="cri-containerd.apparmor.d" comm="nsenter"
requested_mask="read" denied_mask="read" peer="unconfined"
```
Symptom on non-Talos clusters: after a cilium rollout the new pod stays
in `Init:CrashLoopBackOff`, and the rest of the platform cascades into
`dependency not ready` until cilium is healthy on every node.
This PR opts the affected containers out of the
`cri-containerd.apparmor.d` profile via deprecated per-container
`container.apparmor.security.beta.kubernetes.io/<name>: unconfined`
annotations. Four containers are annotated: `cilium-agent`,
`clean-cilium-state`, `mount-cgroup`, `apply-sysctl-overwrites`.
`mount-bpf-fs` is intentionally excluded because it renders with its own
`securityContext.privileged: true`, and the kernel does not apply
AppArmor profiles to privileged containers.
### Why pod annotations instead of `podSecurityContext.appArmorProfile`
The upstream Cilium chart already sets
`podSecurityContext.appArmorProfile.type: Unconfined` (the modern k8s >=
1.30 API), but containerd's CRI implementation silently ignores this
field today and keeps applying `cri-containerd.apparmor.d` — see
kubernetes/kubernetes#125069 (still open). Only the deprecated
per-container annotations take effect end-to-end on current k3s /
kubeadm + containerd setups.
### Why a dedicated values file limited to non-Talos variants
On Talos (`isp-full` bundle) `values-talos.yaml` sets
`cgroup.autoMount.enabled: false`, so the upstream chart does not render
the `mount-cgroup` init container. The kube-apiserver rejects a
DaemonSet that carries a per-container AppArmor annotation for a
container that is not in the pod spec:
```
DaemonSet.apps "cilium" is invalid: spec.template.annotations[
container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: "mount-cgroup": container not found
```
To keep the Talos path untouched and only add the annotations where they
are both safe and useful, this PR puts the `cilium.podAnnotations` block
in a new `packages/system/cilium/values-apparmor.yaml` and wires it into
only the non-Talos PackageSource variants (`cilium-generic`,
`kubeovn-cilium-generic`). Those variants always run with
`cgroup.autoMount.enabled: true` (explicit bundle override in
`packages/core/platform/templates/bundles/system.yaml:83-86`), so every
annotated container really exists in the rendered DaemonSet.
On Talos the annotations are omitted entirely — that is safe because the
Talos kernel does not load the AppArmor LSM anyway.
### Vendored chart patch to avoid duplicate keys on unsupported k8s <
1.30
On k8s < 1.30 the upstream cilium chart template emits the same
annotation keys from its own `semverCompare "<1.30.0"` branch in
`cilium-agent/daemonset.yaml`. Letting both sources write to the
annotations mapping would produce a YAML map with duplicate keys.
Cozystack's supported k8s matrix starts at 1.30
(`packages/apps/kubernetes/files/versions.yaml`), so the duplicate
rendering is never observed in practice, but to keep the rendered
manifest clean on any k8s version this PR also adds a `perl -i -0pe`
multi-line delete to `packages/system/cilium/Makefile` that strips the
entire upstream `{{- if not .Values.securityContext.privileged }} / {{-
if semverCompare "<1.30.0" }}` block from the vendored template. A
fail-fast `grep` guard after the `perl` invocation turns a
silently-failed patch (e.g. if upstream reformats the block) into a loud
error. The same patch is applied to the currently vendored copy so the
repository state matches what `make update` produces. This follows the
existing sed-patch pattern already used for `values.yaml` (`Used in
iptables`, `SYS_MODULE`).
### Scope
- `packages/system/cilium/values-apparmor.yaml` — new file containing
the four `cilium.podAnnotations`
- `packages/system/cilium/values.yaml` — unchanged apart from reverting
the previously-added annotations block (the annotations now live in
`values-apparmor.yaml`)
- `packages/system/cilium/Makefile` — extends the `update:` target with
a `perl` multi-line delete (plus a `grep` fail-fast guard) that drops
the entire upstream AppArmor annotation block on every vendoring refresh
-
`packages/system/cilium/charts/cilium/templates/cilium-agent/daemonset.yaml`
— the vendored file with that block already removed to match the
Makefile patch
- `packages/core/platform/sources/networking.yaml` — adds
`values-apparmor.yaml` to the `cilium-generic` and
`kubeovn-cilium-generic` variants only
### Compatibility across supported platforms
- **Talos Linux** (`isp-full` bundle, `kubeovn-cilium` variant) —
AppArmor LSM is not loaded, `cgroup.autoMount.enabled: false`,
`values-apparmor.yaml` is not included. Zero annotations in the rendered
DaemonSet. No behavioural change. Verified with `helm template` +
`kubectl apply --dry-run=server` against a live k3s v1.35.2 cluster: no
errors, no AppArmor lines.
- **Ubuntu 22.04 / 24.04, Debian with AppArmor** (`isp-full-generic`
bundle, `kubeovn-cilium-generic` / `cilium-generic` variants) — the four
annotations are present in the rendered DaemonSet, kubelet starts the
listed containers with the unconfined profile, `nsenter` succeeds.
Verified with `helm template` + `kubectl apply --dry-run=server` against
a live k3s v1.35.2 cluster: no errors, exactly four
`container.apparmor.security.beta.kubernetes.io/*: unconfined` lines.
Verified on a fresh k3s + Ubuntu 24.04 deployment end-to-end.
- **RHEL / Rocky / AlmaLinux / Fedora** — use SELinux, not AppArmor.
When deployed via `*-generic` variants these platforms pick up the
annotations; kubelet silently ignores AppArmor annotations when the LSM
is not loaded, SELinux policy is unaffected.
- **k8s < 1.30 and k8s >= 1.30** — both now render the same four
annotations (on non-Talos), sourced exclusively from
`cilium.podAnnotations`. Verified via `helm template --kube-version
1.29.0` and `--kube-version 1.35.0` → exactly 4 annotations with no
duplicates.
### Release note
```release-note
[cilium] Opt four cilium-agent containers (cilium-agent, clean-cilium-state, mount-cgroup, apply-sysctl-overwrites) out of the cri-containerd.apparmor.d AppArmor profile via per-container annotations, applied only on non-Talos cilium variants (cilium-generic, kubeovn-cilium-generic). This unblocks cilium rollouts on Ubuntu 22.04+, Debian and other distributions that load this profile for containerd by default, where the kernel previously denied nsenter's namespace entry (reported as a blocked ptrace operation) and the cilium agent landed in Init:CrashLoopBackOff. Talos variants are untouched.
```
## Summary
Replaces the `cozystack-bot` user account + personal access token
(`GH_PAT`) with the `cozystack-ci` GitHub App across all CI/CD release
workflows.
### Why
- **Security**: GitHub App tokens are short-lived (1h) and scoped, vs. a
long-lived PAT tied to a user account
- **Auditability**: App actions are clearly attributed to
`cozystack-ci[bot]`
- **No 2FA issue**: The bot account lacked 2FA; GitHub Apps don't
require it
- **Best practice**: GitHub recommends Apps over PATs for automation
### Changes
| Workflow | What changed |
|---|---|
| `tags.yaml` | App token in all 3 jobs: prepare-release,
generate-changelog, update-website-docs |
| `auto-release.yaml` | App token for daily auto-patch tagging |
| `pull-requests-release.yaml` | App token for release PR finalization |
All `cozystack-bot` git identity replaced with `cozystack-ci[bot]`.
### New secrets (already configured at org level)
- `COZYSTACK_CI_APP_ID` — GitHub App ID
- `COZYSTACK_CI_PRIVATE_KEY` — GitHub App private key
### After merge
- [ ] Delete `GH_PAT` repo-level secret
- [ ] Remove `cozystack-bot` from the cozystack org
🤖 Generated with [Claude Code](https://claude.com/claude-code)
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Updated CI/CD automation authentication mechanisms across release and
tagging workflows for improved security practices.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
Reject Tenant creation when the computed workload namespace (parent
namespace + "-" + tenant name) would exceed the 63-character DNS-1123
label limit. Previously only the per-name length was checked against
the Helm release limit, so a deeply-nested tenant could pass Create()
and then fail later as an opaque HelmRelease reconcile error.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The disk_usage helper shells out to 'du -sh $path' for reporting in
the warning message. On the exact directories this script is meant
to warn about (a /var/lib/containerd accumulating millions of files
from months of unpruned builds), traversing the tree with du can
take minutes and stall the preflight indefinitely.
Wrap the call in 'timeout 5s' so the helper returns quickly even on
a pathologically slow filesystem. If the timeout binary is absent
(e.g. minimal busybox userland), the pipeline still exits 0 via the
existing '|| true' guard and usage stays empty — the warning still
prints, just without the size detail — so the change is backward
compatible.
Addresses an inline review comment from gemini-code-assist on the
open PR.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
E2E revealed that the kube-apiserver rejects a DaemonSet carrying a
per-container AppArmor annotation for a container that is not present
in the pod spec:
DaemonSet.apps 'cilium' is invalid:
spec.template.annotations[container.apparmor.security.beta.kubernetes.io/mount-cgroup]:
Invalid value: 'mount-cgroup': container not found
On Talos (isp-full bundle) values-talos.yaml sets
cgroup.autoMount.enabled=false, so the upstream chart does not render
the mount-cgroup init container, and our unconditional podAnnotations
entry for it becomes an invalid reference.
Move the podAnnotations block from the shared values.yaml into a new
values-apparmor.yaml and include it only in the cilium-generic and
kubeovn-cilium-generic PackageSource variants, where
cgroup.autoMount.enabled is explicitly true. Talos variants
(cilium, cilium-kilo, kubeovn-cilium) get no AppArmor annotations,
which is safe because the Talos kernel does not load the AppArmor
LSM anyway.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
- Broaden the grep guard in the Makefile update: target to match any
container.apparmor.security.beta.kubernetes.io key, not only the
quoted cilium-agent line. The previous pattern would silently pass
if upstream switched to an unquoted annotation value, masking a
failed perl patch; grepping for the prefix catches any residual
hardcoded AppArmor annotation regardless of format.
- Rewrite the values.yaml comment to distinguish the two Talos-safe
reasons: mount-cgroup is simply not rendered when
cgroup.autoMount.enabled is false (Talos default), while the other
annotations are harmless because kubelet ignores AppArmor metadata
on nodes without the LSM loaded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Replace the misleading 'nsenter --ptrace' shorthand with an accurate
description: nsenter joins the host cgroup/mount namespace, and the
cri-containerd.apparmor.d profile reports the denial as a blocked
'ptrace' operation (operation name, not an nsenter CLI flag).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The perl multi-line delete in the update: target silently becomes a
no-op if the upstream template is ever reformatted. Verify the patch
applied by grepping for one of the stripped annotation keys after the
perl runs; fail the update: target loudly if it still exists, so a
future upstream reformat is caught instead of silently reintroducing
the duplicate-key rendering on k8s < 1.30.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The previous sed patch removed only the four annotation lines and
left an empty nested '{{- if ... }}{{- end }}{{- end }}{{- end }}'
skeleton behind. That skeleton was dead code, but it also meant that
the next make update would re-fetch the upstream chart with the
annotations back inside the same block, and the sed would strip them
again — leaving the skeleton behind forever.
Replace the per-line sed patch with a single perl -i -0pe multi-line
delete that removes the entire <1.30 AppArmor block in one step, and
apply the same patch to the currently vendored copy for consistency.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
The upstream chart emits per-container AppArmor unconfined annotations
inside a k8s<1.30 branch of cilium-agent/daemonset.yaml. Cozystack now
owns these annotations via cilium.podAnnotations in values.yaml so they
also take effect on k8s>=1.30 (where upstream drops them in favour of
the containerd-broken podSecurityContext.appArmorProfile). Having both
sources emit the same keys produced a duplicate mapping on k8s<1.30.
Add a new SED_INPLACE invocation to the package Makefile that removes
the four hardcoded annotation lines from the vendored template, and
apply the patch to the currently vendored copy so make update will
reproduce the same state on future refreshes.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
$(wildcard ...) returns a space-separated list, so a bats file with
a literal space in its name would silently split into multiple
tokens in the subsequent 'for' loop. All current bats files use
hyphen-separated names, so this is not an immediate risk, but the
caveat is worth calling out so a future contributor who adopts a
different naming convention rewrites the recipe instead of tripping
over it.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address three review findings on the bats suite:
- The 'standalone containerd service active' and 'standalone docker
service active' tests previously asserted 'systemctl disable --now
<service>' but did not require the 'sudo ' prefix, so the two-
service test was the only guard against a silent drop of the
prefix. Extend both single-service tests to require the prefix
exactly as the 'both services active' test does.
- The 'both services active' test now captures the exit code
explicitly with 'bash ... || status=$?' and asserts '$status -eq
0'. The script contract ('always exits 0') was previously enforced
implicitly via 'set -e', which produces a generic test failure on
a regression instead of a contract-specific error. The explicit
check locks in the contract and makes regressions readable.
- New test: 'docker socket paths with glob chars do not expand' —
sets COZYSTACK_DOCKER_SOCKET_PATHS to a literal glob against real
directories and asserts no docker warning fires. Locks in the
array-based parsing of the path list.
- Extend the file header with a 'title syntax constraints' section
documenting how cozytest.sh's awk parser treats double quotes and
non-alphanumeric characters in @test titles, so future
contributors do not stumble on the parser's quirks.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
'set -euo pipefail' does not include 'set -f', so the previous
'for sock in $DOCKER_SOCKET_PATHS' loop relied on both word splitting
AND unintended glob expansion. If the variable ever contained a
literal '*' or '?' — for example a test setting
COZYSTACK_DOCKER_SOCKET_PATHS='/run/docker-*.sock' against a tree
that happens to have real files matching the pattern — the loop
would iterate over directory entries instead of the intended socket
paths, producing false-positive docker warnings.
Replace the implicit split+glob loop with 'read -ra' into a bash
array and iterate 'for sock in "${_socks[@]}"'. This keeps the
space-separated input format, disables glob expansion, and removes
the fragile word splitting that shellcheck otherwise has to be
talked out of.
Covered by a new regression test ('docker socket paths with glob
chars do not expand') that sets the env var to a literal '*' pattern
pointing at real directories; without this fix the test fails with
an unexpected docker.service warning.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
DOCKER_SOCKET_PATHS is a space separated list of socket paths (its
default is "/run/docker.sock /var/run/docker.sock") that the loop
iterates by word splitting. Socket paths never contain whitespace on
Linux hosts, so this is the documented and correct idiom — not a
shellcheck oversight. Add an inline comment so a future maintainer (or
a linter that has not seen the comment) does not break the loop by
adding quotes around the expansion.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Without a guard, running make bats-unit-tests in a tree where the
hack/*.bats wildcard matches nothing (for example a rename, a shallow
clone, or a future reorganization that moves the tests elsewhere)
would silently succeed with zero tests run. Add an explicit guard that
fails the target with a clear error message when BATS_UNIT_FILES
expands to an empty list, so the developer cannot accidentally believe
tests passed when no tests executed.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two review findings:
1. The socket fallback tests previously branched on 'command -v
python3' and fell through to 'return 0' when it was missing.
cozytest.sh has no SKIP concept — 'return 0' is indistinguishable
from a real pass, so on a runner without python3 the socket
fallback paths were silently unverified. Since the script uses
'[ -e "$sock" ]' rather than '[ -S ... ]', a regular file is
sufficient to exercise the detection path. Replace the python3
unix-socket creation with 'touch "$SOCK"' so the tests run
unconditionally on every runner.
2. Extend the 'both services active' HINT assertion to require the
'sudo ' prefix on the 'systemctl disable --now' line. Without sudo
the operator would be instructed to run a privileged command as a
non-root user and hit a silent failure; the prefix is as important
as the verb itself and must be locked in by a test.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small Makefile tweaks:
1. bats-unit-tests now iterates over every hack/*.bats file that is
not an e2e-*.bats file (wildcard + filter-out), so dropping a new
unit bats file into hack/ is enough to get it picked up by CI on
the next run. The previous hard coded single file path would have
required a Makefile edit for every new unit suite.
2. Add a new 'preflight' target that runs hack/check-host-runtime.sh.
This gives the script a discoverable entry point — an operator
preparing a generic (k3s/kubeadm) host for cozystack can now run
'make preflight' from the repository root to get the same warning
that is described in the script header, without having to know
the exact path under hack/.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Expand the bats suite to address a second review pass:
- Assert that a containerd only warning produces a HINT that names
containerd.service and NOT docker.service, and mirror the check in
the docker only test. When both fire, assert the HINT lists both
services in a single systemctl disable invocation.
- New test: 'clean host without systemctl exits silently' — exercises
the COZYSTACK_PREFLIGHT_FORCE_NO_SYSTEMCTL=1 path when no standalone
sockets exist. Previously the tests only covered the systemd enabled
clean host, leaving the non systemd clean path unverified.
- New test: 'docker service plus socket still emits exactly one
warning' — mirrors the existing containerd service+socket test and
locks in the gated check in check_docker.
- Replace silent 'return 0' on missing python3 with a visible
'# SKIP: python3 unavailable' message on stderr so CI logs make it
obvious which tests were skipped on a given runner.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
When only containerd.service was active, the HINT block still advised
the operator to disable both containerd.service and docker.service,
which is misleading and potentially dangerous on hosts where docker is
legitimately in use. Track per service warnings (CONTAINERD_WARN and
DOCKER_WARN) and build the HINT systemctl disable argument from the
services that actually fired.
Also guard the ANSI color escapes behind an 'is stderr a TTY' check so
log files (CI and reviewer captures) do not accumulate raw escape
sequences.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Address review feedback on the preflight test suite:
- Add a trap 'rm -rf $STUB_DIR' EXIT in every test immediately after
mktemp -d, so temp dirs are cleaned up even when an assertion fails
and terminates the test early under set -e.
- Assert the HINT block and 'systemctl disable --now' line in the
'both services active' test so a future silent removal or typo in
the HINT output is caught by CI.
- New test: 'docker socket fallback fires when systemctl is
unavailable' — mirrors the existing containerd socket fallback test
and exercises the same code path in check_docker.
- New test: 'containerd service plus socket still emits exactly one
warning' — documents and locks in the intent of the symmetric gated
check in check_containerd. Uses grep -c to assert the warning is not
double printed when both signals fire.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Two small issues caught in review:
1. check_containerd probed the socket unconditionally even when
service_active had already set found=1, while check_docker short
circuited the same path behind 'if \[ $found -eq 0 \]'. The
asymmetry was not user visible but violated least surprise for
future maintainers. check_containerd now uses the same gated pattern.
2. The HINT block recommended 'sudo rm -rf /var/lib/docker
/var/lib/containerd' as a casual follow up, which could destroy data
the operator still needs. Replace that line with a warning telling
the operator to inspect and reclaim standalone runtime storage
manually rather than deleting it blindly.
Also drop a no op 'printf' from disk_usage that served no purpose.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add a bats-unit-tests Make target that runs hack/cozytest.sh against
hack/check-host-runtime.bats, and make unit-tests depend on it so the
existing CI step (make unit-tests in .github/workflows/pull-requests.yaml)
exercises the preflight script on every PR. Without this the bats file
exists in the tree but is never executed, leaving future regressions
undetected.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Ubuntu hosts running the cozystack "generic" variant (k3s or kubeadm)
sometimes end up with a standalone containerd.service or docker.service
running alongside the embedded k3s runtime. The two runtimes do not fight
over sockets — k3s uses /run/k3s/containerd/containerd.sock while the
standalone package uses /run/containerd/containerd.sock — so both keep
running silently, and the standalone one accumulates unpruned images and
build cache in /var/lib/containerd. Over time this fills the root disk,
triggers DiskPressure, and sends cozystack-api into an eviction loop.
hack/check-host-runtime.sh warns an operator about this before install
without blocking it. The script probes systemctl and well-known socket
paths, reports disk usage of the standalone data directory when present,
prints a hint on how to disable the shadow runtime, and always exits 0.
Covered by hack/check-host-runtime.bats (six test cases).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add hack/check-host-runtime.bats with six self-contained test cases that
exercise the check-host-runtime.sh preflight script: clean host exits
silently, standalone containerd.service warns, standalone docker.service
warns, both services warn, du failures do not suppress warnings, and the
socket-only fallback fires when systemctl is unavailable.
Tests inject a stub systemctl and du binary via PATH and redirect the
script's probe paths through COZYSTACK_PREFLIGHT_* environment variables,
so they run without root and on any host (including non-systemd macOS).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
On Kubernetes versions below 1.30 the upstream cilium chart still emits
the deprecated per-container AppArmor annotations from its own
semverCompare '<1.30.0' branch. Our podAnnotations override then emits
the same keys again, yielding a YAML mapping with duplicate entries
(identical values, last-wins by kubernetes API server JSON semantics).
Cozystack's supported Kubernetes matrix starts at 1.30
(packages/apps/kubernetes/files/versions.yaml), so this rendering
quirk never affects any shipped version. Document the behaviour in a
comment so future readers are not surprised.
No functional change.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
mount-bpf-fs runs as privileged in the vendored chart, and the Linux
kernel does not apply AppArmor profiles to privileged containers. The
per-container unconfined annotation for this container is a no-op and
only widens the pod's admission-policy surface for no reason.
Keep the four meaningful annotations (cilium-agent, clean-cilium-state,
mount-cgroup, apply-sysctl-overwrites) which target unprivileged agent
and init containers actually affected by the cri-containerd.apparmor.d
profile.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Cilium 1.19 init containers mount-cgroup, apply-sysctl-overwrites,
mount-bpf-fs and clean-cilium-state call nsenter --ptrace, which is
denied by the cri-containerd.apparmor.d profile on Ubuntu 22.04+ and
any other distro loading this AppArmor profile. As a result cilium
agent pods land in Init:CrashLoopBackOff and downstream HelmReleases
cascade into "dependency not ready".
The modern k8s >= 1.30 pod-level appArmorProfile: Unconfined field is
accepted into the DaemonSet spec but not honoured by containerd CRI
at runtime (kubernetes/kubernetes#125069), so the deprecated
per-container AppArmor annotations remain the only reliable opt-out.
Add the annotations via podAnnotations in the wrapper values.yaml.
The change is harmless on systems without AppArmor (Talos,
RHEL/Rocky/AlmaLinux with SELinux) because kubelet silently ignores
unknown AppArmor annotations when the LSM is not loaded.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
Fix system postgresql images to 17.7-standard-trixie
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Database Helm charts now preserve and reuse existing PostgreSQL image
settings, normalizing tags to the standard-trixie variant when
appropriate.
* Migration configuration bumped to target version 38.
* **Bug Fixes**
* Migration logic updated to correctly locate and adjust CNPG Cluster
resources and to log per-cluster patch/skip decisions.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## What this PR does
- Adds `cozy-monitoring` namespace to the infra dashboards rendering
condition in `dashboards.yaml`
- The default variant deploys the monitoring chart to `cozy-monitoring`,
but the condition from #2197 only checked for `tenant-root`, causing all
infrastructure dashboards to be missing
- This is consistent with the existing pattern in `vmagent.yaml` which
already handles both namespaces
## Release note
```release-note
[monitoring] Fix infrastructure dashboards not rendering in default variant where monitoring is deployed to cozy-monitoring namespace
```
Fixes#2349
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Infrastructure & Monitoring**
* Expanded Grafana dashboard availability across additional deployment
environments for infrastructure monitoring.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
## Summary
- update `packages/system/linstor` to `linstor-server` `v1.33.1`
- keep only the selected `piraeus-server` backports needed for this
update
- document the upstream/source commits for each retained patch
- drop the older local patch set that is no longer carried in this
package revision
## Selected backports
- PR #475: backport maintainer implementation from
`3d97f71c95a493588d3d521c63eac4d846935fb3`
- PR #476: backport the TCP port preservation fix from
`79d6375c55d6181b35a7b7f0fe8dbdfb86e126cd` and
`bcc89902f4f61ac1589dd07ebb7f5aae1935370d`
- PR #472: backport the LUKS header sizing fix from
`ccc85fbd2c65f0b97c52403fa80f1efdb886ec4e` plus the required
`optimal_io_size` dependency from
`71b601554f41bcb50cd5bd06989c5b0d3a814acd`
## Verification
- applied all retained patches on top of upstream `v1.33.1` with `git
apply --check`
- verified the combined backport builds with `./gradlew --no-daemon
compileJava`
- ran `make image` in `packages/system/linstor` successfully
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Upgraded LINSTOR server to v1.33.2 and refreshed related package
metadata.
* **Bug Fixes / Reliability**
* Safer toggle-disk flows with retry/abort handling and preservation of
DRBD TCP ports.
* Automatic retry/cleanup for stale bitmap attach errors to improve
recovery.
* **Storage / Compatibility**
* Improved LUKS2 header sizing, optimal I/O size detection, and related
patches for more reliable disk formatting and alignment.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
The default variant deploys monitoring to the cozy-monitoring namespace,
but the infra dashboards condition only checked for tenant-root. This
adds cozy-monitoring to the condition, consistent with the existing
pattern in vmagent.yaml.
Fixescozystack/cozystack#2349
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
<!-- Thank you for making a contribution! Here are some tips for you:
- Start the PR title with the [label] of Cozystack component:
- For system components: [platform], [system], [linstor], [cilium],
[kube-ovn], [dashboard], [cluster-api], etc.
- For managed apps: [apps], [tenant], [kubernetes], [postgres],
[virtual-machine] etc.
- For development and maintenance: [tests], [ci], [docs], [maintenance].
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `backport` if it's a bugfix that needs to be backported
to a previous version.
-->
## What this PR does
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same [label] as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
add ability for restoring VMI to copy in another namespace
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **New Features**
* RestoreJob CR supports a free-form spec.options payload; restores now
support in-place preserves (IP/MAC, PVC handling), cross-namespace copy
restores, and safe rename semantics.
* **Documentation**
* Added admin and tenant walkthroughs for Velero-backed
VMInstance/VMDisk backup & restore.
* **Examples**
* New stepwise demo scripts, helpers, orchestration (run-all) and
cleanup scripts.
* **Tests**
* Extensive unit tests covering restore option parsing, target
resolution, and restore side effects.
* **Chores**
* RBAC updated to allow HelmRelease create/delete and namespace listing
for restore flows.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
All existing patches (allow-toggle-disk-retry, fix-duplicate-tcp-ports,
fix-luks-header-size, retry-adjust-after-stale-bitmap) apply cleanly on
the new version without modifications.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
## What this PR does
Adds the `service.kubernetes.io/service-proxy-name: "cozy-proxy"` label
to VM LoadBalancer services when `external: true`. This tells Cilium to
completely ignore the service — no BPF entries, no DNAT, no wildcard
drops.
Routing continues to work through kube-ovn
(`networking.cozystack.io/wholeIP` annotation) and MetalLB L2
advertisement is unaffected.
**This fixes two issues:**
1. **Inter-tenant connectivity via public LB IPs**: With
`kube-proxy-replacement: true`, Cilium DNATs traffic destined to LB IPs
(on ports declared in the Service) to the backend pod IP *before*
evaluating network policies. The `CiliumClusterwideNetworkPolicy` then
sees a cross-tenant pod-to-pod flow and blocks it. Ports not in the
Service pass as "world" traffic and work fine — so does ICMP. With this
label, Cilium doesn't DNAT anything, all traffic is classified as
"world", and the existing `allow-external-communication` policy allows
it. Pod IP isolation between tenants is preserved.
2. **WholeIP broken on Cilium 1.19+** (#2327): Cilium 1.19 introduced
wildcard service drop entries (`cilium/cilium#40684`) that drop all
traffic to a LB IP on ports not declared in the Service, before it
reaches netfilter/cozy-proxy. With this label, no BPF entries are
created at all, so there's nothing to drop.
**Tested on our lab cluster:**
- Cilium 1.18.6: inter-tenant via LB IP ✅, pod IP blocked ✅, external
SSH ✅, WholeIP all ports ✅
- Cilium 1.19.1: same results ✅, no wildcard drops ✅
- MetalLB L2 advertisement: unaffected ✅
- Other services without the label: unaffected ✅
**Note:** The `externalPorts` / PortList filtering is currently not
enforced at the network level (the `wholeIP` annotation causes kube-ovn
to route all ports regardless). This was the case before this change and
is a separate concern — per-VM CiliumNetworkPolicies could be
re-introduced for that (ref: `4cbf562a`, #1601).
```release-note
[virtual-machine] Fix inter-tenant connectivity via public LB IPs and WholeIP on Cilium 1.19+
```
Increase Kafka CR readiness timeout from 60s to 300s to account for
slow Strimzi startup on QEMU-based CI sandbox (4 JVM pods).
Add wait-for-delete before re-applying to prevent race condition where
kubectl apply hits a still-deleting resource on retry attempts.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Aleksei Sviridkin <f@lex.la>
Add service.kubernetes.io/service-proxy-name label to LoadBalancer
services when external: true. This prevents Cilium from adding the
service to its BPF service map, fixing two issues:
1. Inter-tenant connectivity via public LB IPs: Cilium's kube-proxy
replacement DNATs traffic to LB IPs before policy evaluation,
causing the CiliumClusterwideNetworkPolicy to block legitimate
cross-tenant traffic through public IPs.
2. WholeIP broken on Cilium 1.19+ (#2327): wildcard service drop
entries block all ports not declared in the Service spec before
traffic reaches cozy-proxy's nftables rules.
With this label, Cilium completely ignores the Service. Routing is
handled by kube-ovn (via the wholeIP annotation) and MetalLB
continues to advertise the IP via L2 unaffected.
Tested on Cilium 1.18.6 and 1.19.1: inter-tenant via LB IP works,
pod IP isolation preserved, external access unaffected.
Signed-off-by: mattia-eleuteri <mattia@hidora.io>
Replace the cozystack-bot personal access token (GH_PAT) with a
GitHub App (cozystack-ci) installation token across all CI/CD
workflows. This improves security by using short-lived, scoped
tokens instead of a long-lived PAT tied to a user account.
Changes:
- tags.yaml: use app token in all 3 jobs (prepare-release,
generate-changelog, update-website-docs)
- auto-release.yaml: use app token for daily patch releases
- pull-requests-release.yaml: use app token for release finalization
The cozystack-ci GitHub App (ID: 3297617) is installed org-wide
with contents:write and pull-requests:write permissions. Secrets
COZYSTACK_CI_APP_ID and COZYSTACK_CI_PRIVATE_KEY are set at org
level.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
## What this PR does
This PR adds configurable values for
`packages/system/keycloak-configure`:
- `login.userRegistration` to allow user self-registration
- `login.verifyEmail` to enforce email verification
- `smtp` which defines the SMTP configuration which will be used by
Keycloak
Fixes: aenix-org/cozyportal#365
### Release note
```release-note
["cchore(keycloak-configure): verify email and smtp config for Keycloak"]
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
## Summary by CodeRabbit
* **Chores**
* Added configurable Keycloak login settings with user registration and
email verification controls disabled by default.
* Introduced SMTP configuration templates supporting server connection
details, TLS options, and secure credential management for email
delivery.
* Enabled conditional rendering of login and SMTP configuration sections
based on provided environment values.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
CNPG operator in v1.2.0 updates default PostgreSQL image to v18
and cannot migrate from the previous major version automatically,
causing PostgreSQL clusters to fail after upgrade.
Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
Andrei Kvapil