Commit Graph
325 Commits
Author SHA1 Message Date
Mattia Eleuteri 2568db67b1 Merge branch 'main' into fix/kubernetes-nodes-surge-before-drain
Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-13 22:40:18 +01:00
myasnikovdaniil aa36d3fad6 fix(kubernetes)!: remove the talos block and other leftovers of the worker pool split (#5191)
## What this PR does

Cleanup after the Kubernetes app was split into `Kubernetes` (control
plane and addons) and `KubernetesNodes` (one release per worker pool):
what the split left behind in values, docs, templates and tests.

### Kubernetes

The `Kubernetes` application has no `talos` field any more. Worker pools
are `KubernetesNodes` resources and take their Talos release, image,
installer and registry mirrors from their own `talos` and `osImage`, so
the cluster's copies did nothing. The last one, `talos.version`, fed a
duplicate of the Talos to Kubernetes support-matrix check. The pool
chart already runs that check against the Talos release each pool
actually boots, including one picked through an `osImage` override; the
cluster's copy compared the Kubernetes version against a release no node
runs. A stored spec that still carries `talos.*` keeps rendering,
because the values schema does not close the object and the API does not
validate `spec` against it, and a unit test pins that.

The pool's check now has render coverage. Its own suite pins the
pairings it must accept, and a bats test renders the refusals through
`talos.version`, `osImage.factory.version` and `osImage.builtin.version`
with schema validation skipped, since the schema offers no Kubernetes
version the matrix refuses.

I fixed the README and field descriptions that described behavior the
chart no longer has, or never had: a dedicated etcd cluster, a
`blockSize.matchVolume` setting that does not exist, a default `md0`
pool, the host default, the substrate mismatch behavior, and rollout
phase names. The README's Breaking Changes section becomes a pointer to
the per-release upgrade notes on cozystack.io. Unused helpers and stale
comments are removed; the chart renders the same bytes.

### KubernetesNodes

Five field descriptions and the `talos` block told users to keep them in
sync with parent fields that do not exist. NOTES warned about colliding
with a parent-managed `md0` pool, although the parent renders no pools
and `md0` is the normal name of a pool. The installer image was credited
to an upgrade Job that never runs one. The values descriptions, README
intro and NOTES now say what the chart renders today, including which
fields have no effect on `substrate: proxmox`: `storageClass`,
`diskSize`, `instanceType`, `talos.imageFactoryURL` (beyond its format
check) and `talos.imageFactoryCA`.

The render error for a pool whose MachineDeployment belongs to another
release told users to remove the pool from the parent's `nodeGroups`,
which cannot clear the conflict. It now names a remedy that works.
Split-era history is gone from Helm and test comments, along with two
`storageClass` fallbacks that can only return the value they fall back
from, a constant variable, and two `.helmignore` rules that match
nothing.

The tenant pod and service CIDRs are literals in both charts, and
nothing checked that they agree. A new bats test renders both charts and
fails when they differ.

No rendered manifest of the pool chart changes. I rendered it for the
snapshot values, GPU, kubelet-override, factory/CA, storageClass
variants and three Proxmox variants on main and on every commit, and the
output is byte-identical. The snapshot and TalosConfigTemplate hash
tests pass unchanged, so no worker rolls. YAML comments inside the
hashed machine config and machine template are left as they are for that
reason.

### The rule

`docs/agents/overview.md` gains a rule so this does not come back with
the next change: app READMEs and field descriptions describe current
behavior, a change users must act on goes to the website's upgrade page,
and a description does not ask the reader to keep it in sync with
another chart.

```release-note
[kubernetes] The Kubernetes application no longer has a `talos` field. Talos settings for workers, and the Talos to Kubernetes compatibility check, live on each KubernetesNodes pool. Existing clusters that still store `talos` values keep working; the values are ignored. KubernetesNodes field descriptions, README and install notes now describe the worker pool as it works today, including which fields have no effect on Proxmox pools.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Configuration**
* Worker OS image, installer, and registry-mirror settings are
configured per worker pool; these settings are no longer available at
the cluster level.
* Kubernetes and Talos version compatibility is checked for each worker
pool. A worker Kubernetes version cannot be ahead of its parent cluster.
* Worker pools support KubeVirt and Proxmox, with pool-specific sizing
and disk settings.
* Rendering is refused when a pool uses a different substrate from its
cluster or conflicts with an existing worker pool owned by another
release.
  * The default cluster hostname now includes the `kubernetes-` prefix.
* Existing cluster configurations that still include Talos settings
receive a warning; those settings are no longer used.

* **Documentation**
* Updated guidance for cluster and worker-pool settings, storage,
upgrades, and version compatibility.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-14 00:34:44 +05:00
myasnikovdaniil f061739ce2 ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does

Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly
runs were cancelled at the limit, and PRs that select full suite end
same way while every suite that got to run passed (#5283 for example).

This raises limit for all four jobs that run the suite (pull-requests,
e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure.
Comments that state the cap follow it. Making e2e itself shorter is
separate work.

Part of #5261, `Collect report` ceiling in fork workflow from there is
not covered here.

### Screenshots

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
ci(e2e): E2E job limit raised from 215 to 265 minutes.
```

<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Increased the time limit for end-to-end test runs from 215 to 265
minutes across applicable workflows. This allows longer test runs to
complete before the overall job is stopped.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 09:16:13 +03:00
Aleksei Sviridkin 3badbdef74 fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->

## What this PR does

A ClickHouse release with `backup.useSystemBucket: true` can start
before the platform projects `cozy-backups-creds` into its namespace.
Its `clickhouse-backup` sidecar took the S3 settings from environment
variables, which are fixed when the container starts, so the first
BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role
found` until the Pod was restarted, and a rotated key stayed stale the
same way.

The projector now also writes a `clickhouse-backup.yml` key into
`cozy-backups-creds` (bucket, `https://` endpoint, region when set,
keys, `force_path_style` from `backupStorage.forcePathStyle`). On the
system-bucket flow the chart mounts that key as an optional Secret
volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting
`S3_*` variables. clickhouse-backup reloads its config file in each API
handler that reaches S3 (among them the `create_remote`,
`restore_remote` and `list` calls the strategy makes; the `status` and
`actions` reads it polls with do not) and kubelet rewrites a mounted
Secret in place, so credentials projected after the Pod started, or
rotated since, reach the running sidecar. `S3_PATH` stays in the
environment because the prefix belongs to the release.

Kubelet takes about a minute to refresh the volume, so the projector
stamps `backups.cozystack.io/credentials-updated-at` when the projected
content changes, and the Altinity driver holds a new backup or restore
Job for up to two minutes after that stamp, with `Ready=False` and
reason `WaitingForCredentials`; the condition is dropped once the Job
runs. A stamp that cannot be read, or one further ahead than clock skew
explains, is logged and ignored rather than holding the namespace's jobs
for good. The hold applies to every Altinity job in a namespace whose
projected Secret just changed, including releases on their own Secret,
where it is only a delay.

The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a
ClickHouse on the system bucket with no projected Secret, then runs a
backup and an in-place restore through `cozy-default` and checks that
the sidecar that started without credentials served both jobs. In CI the
system bucket's advertised host `s3.example.org` neither resolves nor
gets an ACME certificate, so for the duration of its own step the Test
serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name
and adds a marked `rewrite name exact` line for that host to the host
CoreDNS Corefile, removing both on exit; it proves the served
certificate before any job starts.

The bug was reproduced on a test cluster running the code before this
change: a ClickHouse started before the projection failed its first
BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run
green yet outside CI, because it edits the host CoreDNS, which is shared
on that cluster.


### Screenshots

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#799 (draft until this PR merges):
`operations/services/backup-classes.md` for the mounted config file, the
credentials wait and the upgrade window, without the Pod restart this
change removes
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* ClickHouse system-bucket backups now use a projected configuration
file for S3 settings, supporting HTTPS endpoints and optional region and
path-style settings.
* Backup and restore jobs wait briefly for recently updated projected
credentials before starting.
* Path-style settings accept common true and false values and are
normalized consistently across backup storage integrations.
* **Tests**
* Added end-to-end coverage for system-bucket backup and restore,
verifying restored data without restarting the ClickHouse Pod or backup
sidecar.
* Added checks for TLS certificate validation and safe restoration of
shared network and ingress settings.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 16:15:26 +03:00
Myasnikov Daniil 90f0b043a6 fix(kubernetes)!: remove the talos block the chart no longer uses
The kubernetes chart declared a talos block (version, schematicID,
imageFactoryURL, installerRepository, registryMirrors). Workers are
KubernetesNodes pools and take every one of these from their own talos
values, so an operator who set them on the cluster got no error and no
effect. The only reader left was a copy of the Talos to Kubernetes
support-matrix check, and the pool chart already runs that check
itself, against the Talos release each pool actually boots, including
one an osImage override selects. The cluster's copy compared the
Kubernetes version against a Talos release no node runs.

The block leaves the API schema, hence the "!". Stored specs that still
carry it keep rendering: the values schema does not close the object
and the API does not validate spec against it. A helm-unittest case
pins that.

Readers outside the chart moved with it. The e2e install read
talos.schematicID from this chart's values.yaml in a best-effort block
that only warns, so removing the field would have stopped the image
cache being seeded without failing anything; it now reads the pool
chart, as the proxmox template script already does. The e2e no longer
writes a talos block into the Kubernetes CR, the spec test checks the
pool chart for the key it emits, the release checklist diffs the pool
chart's pins as well, and the catalog guard drops the two tests that
compared or read the parent's block. The null-normalization Go tests
used talos keys of the real Kubernetes schema as their fixtures; they
use oidc.mode, the same required-with-default shape under an optional
root section, and addons.cilium.valuesOverride for the free-form case.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 16:51:38 +05:00
Myasnikov Daniil c63e83466a ci(e2e): raise the E2E job limit from 215 to 265 minutes
A green run of the full Chainsaw suite no longer fits in 215 minutes:
the last four nightly runs were cancelled at the limit, and pull
requests that select the full suite end the same way while every
suite that got to run passed. All lanes that run the suite keep one
figure, so a change that passes on a pull request is not cut off on
nightly or on a tag.

Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-12 15:42:25 +05:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Alexey Artamonov 35823c6c9c test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split;
it still described two of them after the third was added, so a reader
starting from it would miss the only Test that covers the system-bucket
path.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:56:26 +03:00
Alexey Artamonov c26d2773a5 test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever
BackupJob ran in tenant-root before this Test, from this suite or
another: Chainsaw does not fix the order of the Tests in a suite, so no
particular one can be named. The restore budget is also not spent on a
credentials hold: by then the stamp is older than the backup's own
two-minute wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-11 14:42:11 +03:00
Aleksei Sviridkin 5a2582e21c test(observability-proxy): preserve JSON seed request bodies
Send JSON seeds with their content type so VictoriaLogs does not consume
log records while parsing form parameters before ingestion.

Fixes: ebb5a9cec4 ("test(observability-proxy): cover the namespace logs scope end to end")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-10 23:21:56 +03:00
ZverGuy 3e907e6f9a test(observability-proxy): declare the shell of the e2e helper library
The library is sourced by sh scripts and has no shebang, so shellcheck
could not tell which shell to check it as and refused the file.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy f775948d2c test(observability-proxy): replay the early line into the e2e tail
The background tail goes through the console route and the proxy's
reviews before it subscribes, so the early ingest can land first, and
a tail without start_offset returns only lines written after it
starts. The tail now replays the last minute, which still leaves the
hold and the late-line check meaningful.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
Mattia Eleuteri f2ee3ec8ad test(computeplane): pin the maxUnavailable bounds and zero through the API
A zero dropped anywhere between the ComputePlane API and the pool chart would let a GPU pool fall back to the chart default of 1, and the render tests never cross the aggregated API. A negative value let through by the ComputePlane schema would only be refused by the pool chart, after the parent release had already been upgraded.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 13:24:36 +01:00
Alexey Artamonov 72a94ee426 test(clickhouse): give the round-trip's preconditions room for their deletes
The step's ceiling SIGKILLs the shell and reports only "signal: killed",
while a kubectl --timeout that fires first names the object that would
not go. The deletes it runs now wait up to 270s in all, past the 3m
ceiling, so a stuck delete late in the list would be reported by the
ceiling instead of by kubectl.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 14:26:02 +03:00
Alexey Artamonov 0dc5a54731 test(clickhouse): remove the round-trip's TLS objects a killed run left
Chainsaw deletes an applied object only when its own apply created
it; one left behind by a run killed past its cleanup is patched and
kept. The Certificate would then outlive every later run and keep
re-issuing its Secret after each step cleanup, and a probe Pod whose
sleep had ended never turns Ready, failing the next run's assert.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-09 13:49:42 +03:00
Mattia Eleuteri f003851338 fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled
like an application database. The init job made <db>_admin their owner,
installed the auto-grant event trigger in them and, once the entry left
values, ran DROP DATABASE on them. That DROP can only fail (template
databases cannot be dropped, and the job is connected to postgres), so
every later upgrade failed in the init job and the release stayed in
UpgradeFailed even after the entry was put back. template1 also passed
the takeover on to every database created from it afterwards, where the
trigger, naming a role about to be dropped, makes CREATE SCHEMA fail.

A release a previous version let take one of them over has to be
repaired without dropping it, and without touching what this chart did
not create: a role or owner that merely carries an expected name may
belong to a tenant or an administrator, and a role comment can be set
by any CREATEROLE holder, so only the cozystack:managed mark is trusted.
What the marked roles own goes to cozystack:orphaned rather than to the
superuser, since a tenant's SECURITY DEFINER function would otherwise
run with superuser rights. A failed lookup of the databases cloned from
template1 has to stop the run before template1 stops being selected, or
those clones would never be repaired.

Assisted-by: LLM
Signed-off-by: Mattia Eleuteri <mattia@hidora.io>
2026-11-09 11:18:27 +01:00
ZverGuy 8784863632 test(observability-proxy): wait for every seeded line before checking scope
The suite checked that B's line and the account-7 line are absent from
A's answers but only waited for A's line to become searchable, so an
absence could come from a line the store had not indexed yet. The step
budgets in the comments now count the calls the steps make.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy ebb5a9cec4 test(observability-proxy): cover the namespace logs scope end to end
Unit tests mock the API server and the store, so nothing shows that the
console's nginx, the proxy's TokenReview and SubjectAccessReview and a
real VictoriaLogs agree. The suite drives a viewer of one tenant through
the console against a seeded store: it reads its own namespace, is
refused a sibling's, cannot widen the scope with client filters, an
AccountID header or dot segments, and keeps a tail open past a minute.

The root store is off in e2e, so the suite runs its own VictoriaLogs
under the root store's Service name rather than waiting on Fluent Bit.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
Alexey Artamonov 28f3613450 test(clickhouse): delete the round-trip's TLS Secret in its own step's cleanup
Chainsaw applied the Certificate, so the scripted delete of it in the
run step's finally was hand-rolled teardown of a resource Chainsaw
already reclaims. Chainsaw deletes a step's applied resources, waiting
for them to go, before it runs that step's cleanup operations, which
puts the Secret's delete after the Certificate's without any ordering
written into a script.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:28:53 +03:00
Alexey Artamonov 6e23bd5629 test(clickhouse): delete the round-trip's Certificate before its Secret
cert-manager issues the Secret of a Certificate that still exists again
as soon as it is deleted, and the Certificate is only removed by
Chainsaw's own cleanup after the step's finally has run. Deleting the
Secret first therefore left a freshly issued copy behind in tenant-root,
carrying no owner reference to take it down with the Certificate.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:22:43 +03:00
Alexey Artamonov 1003cdb1b6 test(clickhouse): find the round-trip's Ingress Secret by its owner label alone
kubectl refuses a resource name together with a label selector, and the
restore asked for both with its stderr discarded, so the lookup always
came back empty and the Secret the Test serves under the platform's S3
Ingress name was never deleted. The next run then stops on its own
precondition that the Secret must not exist. A lookup that does fail
now fails the restore, since an empty answer from a failed read is what
kept this hidden.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-08 20:22:35 +03:00
Alexey Artamonov 6280d06d01 test(clickhouse): describe the round-trip's guards by the archive-recording restore
Restores now take the archive the Backup recorded rather than the
newest one, and a leftover Backup of the BackupJob's name that records
another archive fails the new BackupJob instead of being adopted. The
comments justifying the sentinel row and the Backup cleanup, in the
round-trip and in its unit test, still gave the earlier reasons, which
would send the next reader after behaviour the driver no longer has.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-06 17:56:59 +03:00
Alexey Artamonov 603114cd7c test(clickhouse): check the round-trip restored this run's sentinel row
Every run inserted the same sentinel row and accepted any row after the
restore, while the Altinity restore takes the newest remote archive of
the release. A restore of an older run's archive therefore passed. The
row now carries a per-run value and exactly that row has to come back.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:09:15 +03:00
Alexey Artamonov dee5e6d8bf test(clickhouse): mark the round-trip's reachability poll as a fixed wait
The probe loop sleeps between attempts because no Kubernetes condition
reports that the host CoreDNS serves the rewrite and the Ingress serves
the Test certificate. The e2e conventions allow such a sleep only under
the marker that records it is waiting for something to model it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:08:59 +03:00
Alexey Artamonov cabe9a07e2 test(clickhouse): reclaim the round-trip Backup by its name
The Altinity driver names the Backup after the BackupJob and adopts one
that already exists under that name. The round-trip's cleanup deleted
the Backup only through the BackupJob's status.backupRef, so a Backup
whose BackupJob never recorded it survived, and the next run's
BackupJob would have adopted it in place of a fresh backup. Both the
cleanup and the preconditions now delete it by name; the preconditions
ceiling grows to cover the extra wait.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:08:50 +03:00
Alexey Artamonov a662198991 test(clickhouse): restore the round-trip's shared objects after a step ceiling
The round-trip changes the host CoreDNS Corefile and the S3 Ingress TLS
Secret and puts both back from its EXIT trap. When the run step's
ceiling fires, Chainsaw kills the shell and the trap never runs, so the
rewrite stays in a cluster-wide object and the next run refuses to
start. The step's finally runs regardless, so it repeats the restore;
after a run whose trap already restored them it finds nothing to undo.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:07:51 +03:00
Alexey Artamonov 003bc3ed0b test(clickhouse): fail the round-trip when CoreDNS cannot be restored
The round-trip puts the host CoreDNS Corefile back from its EXIT trap,
which runs with errexit off. A refused kubectl replace, for instance a
resourceVersion conflict with another writer, fell through to the
success message and returned 0, so the run stayed green while the
rewrite remained in a cluster-wide object and the next run refused to
start. The restore now returns the failure.

The restore moves into a file of its own so a unit test can run it
against a stubbed kubectl and so other callers can share it.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:06:53 +03:00
Alexey Artamonov 0ad642e68d test(clickhouse): size the system-bucket round-trip server for SQL
The round-trip ClickHouse ran on the nano preset, whose 128Mi limit
the clickhouse-1 contract suite never exceeds because it sends no
query. This suite writes a sentinel row before the backup, and the
server was OOM-killed on that first statement (lastState OOMKilled,
exit 137), so the Test failed before it reached the behaviour under
test. clickhouse-2 runs the same statements on the small preset.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-05 21:05:31 +03:00
Alexey Artamonov dbcc2af679 test(clickhouse): require the system-bucket round-trip to run on the sidecar that started without credentials
A Pod restart during the round-trip would hand the sidecar fresh
credentials at start, which is how the old environment-based sidecar
got them, and the Test would pass without the running sidecar ever
picking them up. The Pod uid and the sidecar's restart count must stay
the same across both jobs.

The probe deadline is now measured on the clock: counting only its
pauses let a hanging exec stretch it past the step budget, so the step
timeout would fire before the probe could say what stalled.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:43:55 +03:00
Alexey Artamonov 2e37aaec5f docs(backups): drop the claim that a region change needs a ClickHouse restart
The ClickHouse sidecar on the system bucket no longer reads S3 settings
from environment set at start, so a re-projected region reaches it
without a Pod rollout, and the contracts fixture no longer depends on
optional Secret keys to start its Pod.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:31:59 +03:00
Alexey Artamonov e752952047 test(clickhouse): back up and restore a running release through the system bucket
A ClickHouse release on the system bucket starts before the platform
projects cozy-backups-creds into its namespace, and no suite exercised
that order: the round-trips bring their own Bucket and endpoint, so the
sidecar never had to pick up credentials that arrived after it started.

The new Test starts the release without the projected Secret and runs a
backup and an in-place restore through cozy-default against the bucket
as advertised. In CI that host neither resolves nor has a certificate,
so the step makes it reachable from tenant-root for its own duration
and proves the served certificate before any job starts, which keeps a
DNS or TLS problem from reading as a backup failure.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:31:29 +03:00
Aleksei Sviridkin ba0b1ac6ef fix(e2e): validate the leak allowlist against every suite layout (#4972)
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh`
accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already
see step-file suites and a Test nested below the suite directory
(#4630). A suite in one of those layouts could not be put on the leak
allowlist at all: validation failed and the run stopped before the first
suite.

- `lg_suite_has_test` validates an allowlist entry with the rule
`hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`,
`chainsaw-test.yml`, or a numbered step file, anywhere in or below the
suite directory, with the selector's type test (a step-file name counts
unless it is a directory, a `chainsaw-test.y*ml` counts whatever its
type, so Chainsaw reports a directory by that name); a parked
`chainsaw-test.yaml.disabled` matches none. Both sides carry a comment
pointing at the other so they move together.
- `hack/e2e-chainsaw/README.md` describes the three layouts once, under
Layout, and lists `kubernetes-proxmox` as the parked suite.
`docs/agents/e2e-testing.md` points at it.
- Unit tests: every layout is accepted, including a `chainsaw-test.yaml`
that is a directory; a parked, empty, or not-a-step-file directory is
refused, including `1a-install.yaml` (passes the `find` glob, not the
step-file regex) and a directory named `01-install.yaml`.

Left as they are, with the reasons:
- `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one
parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was
removed in 9cc3a9e45.
- The top-level globs in `hack/e2e-chainsaw-run.sh` are wider than
Chainsaw's step-file pattern, but they only refuse a test at the top
level and fail closed.
- A file under `examples/backups/<other>/` that matches no suite is
documented in `select-e2e.sh` as a docs-only demo that stays ignored.

Verified with Bats and `hack/cozytest.sh` on
`hack/chainsaw-leak-guard_test.bats` (34 tests) and
`hack/select-e2e_test.bats` (65 tests; one case, "a path owned by
several sources counts as covered if any one is", fails on my machine on
main as well, from the local yq build, and does not touch this change).

Closes #4659


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Documentation**
* Clarified that Chainsaw suites may use `chainsaw-test.yaml` or `.yml`,
numbered step files, or either layout in a subdirectory.
* Updated guidance on suite selection and noted that disabled test files
are not selected or allowlisted.
* Identified the parked Kubernetes Proxmox suite and its workflow
availability.
* **Bug Fixes**
  * Suite validation now recognizes tests in the documented layouts.
* Validation rejects empty directories, parked test files, invalid
step-file names, and directories named like step files.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 20:00:50 +03:00
Aleksei Sviridkin 013b166967 fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard (#4970)
`cozy_guard_all_helmreleases` in
`hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the parent
`kubernetes-<test>` release and the addons selected by the
`kubernetes-<test>-` prefix. The worker pool is
`kubernetes-nodes-<test>-md0`, an Application HelmRelease the aggregated
apiserver stamps with `RetryOnFailure` like the parent, and it sits on
the node-join path through the talos-reconcile Job it renders. Its name
does not start with the addon prefix, so a failed action on it was never
read, on the passing path or in the failure-path report.

This reads it on both paths, with the `required` strategy check the
parent gets, since the apiserver stamps both. The pool name is derived
from the parent (`cozy_pool_helmrelease_name`), the way the addon prefix
is, so the two cannot drift. The failure-path report checks that the
release exists before reading it, with `get hr --ignore-not-found -o
name`: a run whose pool apply failed has no pool release (exit 0, empty
output), and the report says so instead of reporting a failed read. For
that run to reach the report at all, `run_kubernetes_test` now arms the
exit handler (`_tenant_failure_on_exit`, with
`CURRENT_TENANT_PARENT_HR`) right after the parent `Kubernetes` apply
instead of after the `KubernetesNodes` apply: before, a failing pool
apply exited under errexit with no handler installed, and the parent
went unclassified. The snapshot half still no-ops until the tenant
kubeconfig is set. A non-zero exit (a timeout, a kill, a Forbidden) is
reported as the failed read it is, through `_cozy_guard_read_failure`,
never as absence. The pool reads are gated on the report's own deadline
as well as the phase clock, like the addon listing.

Unit tests in `hack/run-kubernetes-remediation_test.bats` cover a
failing pool release on the passing path, a clean one (the composite
returns zero), an empty strategy on the pool, the report with and
without the pool release, a cut-off existence read, and a pool read past
the report budget; the ordering pin now also requires the handler to be
armed before the pool apply. The kubectl stub gained an existence read:
an `ABSENT` marker prints nothing and exits 0 under `--ignore-not-found`
(NotFound without it, as kubectl does), and the failure markers apply to
it as `<name>.exists`.

Verified with `hack/cozytest.sh` and Bats on
`hack/run-kubernetes-remediation_test.bats` (51 tests),
`hack/run-kubernetes-node-join_test.bats` and
`hack/remediation-guard.bats`.

Closes #4737


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Bug Fixes**
* Kubernetes release checks now include worker-pool releases alongside
the parent release and add-ons, and flag missing or invalid install
strategies and remediation failures.
* Failure reports distinguish worker-pool releases that are absent from
those whose status cannot be read. Existing releases are assessed
against the phase gate and report time budget; when the budget is
exhausted, skipped checks are identified.
* Failure handling now begins immediately after the parent resource is
applied, covering errors that occur while applying the worker pool.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 18:36:58 +03:00
Marian Koreniuk 75bb70e58d fix(e2e): arm the tenant failure handler before the pool apply
The exit handler was armed after the KubernetesNodes apply, so a failing
pool apply exited under errexit with no handler installed: the parent
went unclassified, and the report's "pool release does not exist" line
described a run that could never reach it. Arm it right after the parent
Kubernetes apply instead. The snapshot half still no-ops until the
tenant kubeconfig is set, so the only change is that a failed pool apply
now gets the parent classified and the absent pool reported.

The ordering pin now also requires the handler to be armed before the
pool apply.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-29 12:34:22 +01:00
Aleksei Sviridkin 048c029e29 fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it (#4971)
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin
kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process
umask and never removed it, so it outlived the run on the self-hosted
runner, world-readable under the usual umask. `dump_diagnostics` then
read whatever sat at that path, so a run that failed before
`fetch_tenant_kubeconfig` reported the previous run's tenant as its own.

- The file now lives under `RUNNER_TEMP`, which the runner exports into
every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it
for a local run, and `TMPDIR` is the last fallback. The workflow sets no
path of its own: the `runner` context is not available in job-level
`env`.
- `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell,
since the workflow's `umask 077` covers only the staging step.
- `apply-tenant`, the suite's first step, removes a leftover before
doing anything else, so diagnostics in this run can only see this run's
file.
- The suite's `finally` removes it, before the `keep_tenant` exit, and
the workflow's "Remove staged credentials" step removes it again for a
run killed before `finally`.

Verification: `sh -n` and shellcheck on the script; actionlint on
`e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported,
as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases
with a stub `kubectl`: the fetched file has no group or other bits under
`umask 022`, `apply-tenant` removes a planted leftover, and the suite's
`finally` (run out of the Chainsaw file on the `keep_tenant` path)
removes the file, so `diagnostics` after it prints no tenant section.
Each goes red with its line (`umask 077`, the leftover `rm`, the
`finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9,
teardown contract 8, step1b 4, talos template 12), also under
`hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have
cannot dispatch the workflow at the moment, so the workflow steps
themselves are unexercised beyond actionlint.

Closes #4806


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Chores**
* Proxmox end-to-end test runs store tenant credentials in a temporary
work directory and create kubeconfig files with restricted permissions.
* Stale tenant kubeconfig files are cleared before a new tenant fixture
is applied. Cleanup removes the file even when tenant data is retained
or a run is interrupted, preventing credentials from appearing in later
diagnostics.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:31:31 +03:00
Aleksei Sviridkin 040ae377ae test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool> (#5000)
## What this PR does

The Proxmox suite (`hack/e2e-chainsaw/kubernetes-proxmox/`) has now run
for the first time on a real self-hosted stand: a dedicated Cozystack
v1.7.0-rc.1 management cluster on Proxmox VE, with the runner CageForge
offered to host in cozystack/community#98. The suite failed before it
reached the hypervisor, because the fixture and the chart disagree on
what a node pool may be called.

`kubernetes-nodes` names its release `kubernetes-nodes-<CR name>` and,
since fd14126be, refuses to render one that does not start with
`kubernetes-nodes-<spec.cluster>-`
(`packages/apps/kubernetes-nodes/templates/_helpers.tpl`). The fixture,
added later in 7500a9fba, names its `KubernetesNodes` `md0`. On the
stand that gave:

```
Helm install failed for release tenant-e2eproxmox/kubernetes-nodes-md0 ...:
kubernetes-nodes: release name "kubernetes-nodes-md0" must start with "kubernetes-nodes-k8s-e2e-"
— name the KubernetesNodes CR <cluster>-<pool> (cluster="k8s-e2e")
```

No `Machine` was ever created. The `tenant-and-cluster` step passed its
`ProxmoxCluster` assert in 20 seconds and then waited out the 20-minute
`Machine` assert.

- The fixture names the pool `k8s-e2e-md0`. Nothing else in the suite,
`run-proxmox.sh` or the workflow refers to the pool by name.
- `hack/proxmox-e2e-fixture_test.bats` gets a case that reads
`metadata.name` and `spec.cluster` of the fixture's `KubernetesNodes`
and fails unless the name is `<cluster>-<pool>`. It is POSIX sh like the
rest of the file, so `hack/cozytest.sh` can still run it.

### Testing

- `bats hack/proxmox-e2e-fixture_test.bats` (bats 1.14.0, the CI
commit): 7/7. The new case was checked to fail against the old fixture:
`FAIL: KubernetesNodes md0 must be named k8s-e2e-<pool>`.
- `hack/check-commit-trailers.sh upstream/main..HEAD`: OK.
- Lane, before the fix (fork `main` @ aed7fa896):
https://github.com/themoriarti/cozystack/actions/runs/39402535098. Red
on the `Machine` assert, as above; the teardown removed the tenant and
left no VM on the hypervisor.
- Lane, with this commit:
https://github.com/themoriarti/cozystack/actions/runs/39442983145. The
pool release `kubernetes-nodes-k8s-e2e-md0` installs, the `Machine` is
created, capmox clones the worker from the template and boots it, and
the worker joins the tenant and goes `Ready` (Talos v1.13.6). The run is
still red, one step later and for a known reason: the tenant's CCM
crash-loops on `pccm:v0.15.0` with `panic: version string "0.15.0"
doesn't match expected regular expression: "^v(\d+\.\d+\.\d+)"`, so the
node keeps `node.cloudprovider.kubernetes.io/uninitialized` and no
providerID, and the `Machine` assert reports `Provisioned`, expected
`Running`. That is #4917. The teardown removed the tenant, the worker VM
and its cloud-init ISO.

Two earlier runs on the same commit, 39406900563 and 39409765857,
stopped at stand configuration rather than at this repository: capmox's
token could not see an ISO storage, and the stand's `publishing.host`
did not resolve from the fixture's public resolvers. Both are fixed on
the stand. They are listed below because the lane does not say a stand
needs them.

### Found on the way, not changed here

- **What a stand must provide**, beyond the `PROXMOX_*` variables:
- an `iso`-content storage on which capmox's token holds
`Datastore.AllocateTemplate` (to upload the cloud-init ISO) and
`Datastore.Allocate` (to delete it with the VM). go-proxmox uploads to
the first `iso` storage the token can see; without delete rights the
`Machine` never finishes deleting.
- a `publishing.host` whose `<cluster>.<tenant>.<host>` resolves through
the fixture's own resolvers (`8.8.8.8`, `1.1.1.1`) to a root ingress
with SSL passthrough: on the Proxmox substrate that name is the tenant's
`controlPlaneEndpoint`.
- **`COZY_PVE_ADDRESSES/GATEWAY/PREFIX/BRIDGE/POOL`** are not passed by
the workflow. Unset, they default to `vmbr50` and `10.0.0.180-185`. This
stand sets them in the runner's environment.
- **The fixture's `templateTags`** (`capmox-template, cozystack, talos,
talos-v1.13.6`) are not the set that `hack/proxmox-talos-template.sh`
writes (`cozystack, schematic-…, talos, talos-v…`). capmox matches tags
as a set, so a template built by the script is not selected. This stand
tags its template to match the fixture.
- **Teardown reports success once the CRs are deleted.** It does not
wait for the VMs to be gone; on this stand a worker VM outlived its
`Machine` for hours while capmox retried a forbidden delete.

The CCM fix itself is #4917. Once a release carries it, this lane should
get past the `Machine` assert.

The lane stays manual. It has no `push` or `pull_request` trigger, and
its nightly schedule is a no-op unless `PROXMOX_NIGHTLY` is set, which
it is not on this stand. The runner is registered to a fork whose only
collaborator is its owner. Running it from cozystack/cozystack needs a
runner group restricted to this workflow and a protected environment,
which is a follow-up for community#98, not this PR.

### Screenshots

Not a UI change.

### Downstream repositories

- [x] No downstream repository is affected by this change

The diff touches one fixture under `hack/e2e-chainsaw/` and one bats
file. It moves or renames nothing under `hack/`, changes no make target
and no node prerequisite.

### Release note

```release-note
NONE
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

- **Bug Fixes**
- Updated the Kubernetes node pool fixture to use the required
cluster-prefixed naming convention, improving compatibility with
Kubernetes node validation.

- **Tests**
- Added validation to confirm that Kubernetes node fixtures include the
required cluster reference and use the expected cluster-based name
format.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-29 11:19:21 +03:00
Marian Koreniuk cd983d416e fix(e2e): take the selector's type test in the allowlist scan and pin its regex
lg_suite_has_test dropped every directory with `! -type d`, but the
selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can
fail loudly on a directory by that name. An allowlisted suite in that
shape failed validation and stopped the run before Chainsaw reported the
real error. Use the selector's own `\( ! -type d -o -name
chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a
directory with a step-file name.

The step-file regex behind the find glob was never reached by a test:
install-01.yaml is already filtered by the glob. 1a-install.yaml passes
the glob and not the regex, so it pins the regex; a directory named
chainsaw-test.yaml and one named 01-install.yaml pin the type test.

The README said the runner recognises the same three layouts. It
recognises none: it hands every top-level directory to Chainsaw.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 18:53:07 +01:00
Marian Koreniuk 3d97e912b4 fix(e2e): drop the job-level runner.temp and pin the tenant kubeconfig handling
jobs.e2e.env set COZY_PROXMOX_WORKDIR from runner.temp, but the runner
context is not available in job-level env, so GitHub would refuse the
whole workflow at load time and the Proxmox lane would stop starting.
actionlint reports it as `context "runner" is not allowed here`.

Drop the variable. run-proxmox.sh already falls back to RUNNER_TEMP,
which the runner exports into every step, and the cleanup step removes
the file from there.

Pin the behaviour the previous commit added, none of which a test
covered: the fetched kubeconfig carries no group or other bits under
umask 022, apply-tenant removes a file an earlier run left, and the
suite's finally removes the file on the keep_tenant path, so diagnostics
after it prints no tenant section. Each case goes red with its line
removed.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 18:49:12 +01:00
Marian Koreniuk 4d7d336efc fix(e2e): tell a failed pool existence read from an absent pool release
The failure-path report checked the pool release with `get hr -o name`
and discarded the exit status and stderr, so a timeout, a kill, a
Forbidden or a refused connection all printed that the run ended before
the worker pool was applied. That is the conflation the guard wrapper
exists to refuse, on the path where a slow apiserver is most likely.

Read it with --ignore-not-found instead: exit 0 with empty output is an
absent release, exit 0 with a name is one to classify, and any non-zero
exit is a failed read named through _cozy_guard_read_failure.

The pool reads also skipped the report's own deadline and checked only
the phase clock, which says yes on every path that opens no phase.
Check the deadline before the existence read, like the listing and the
addons do, and name what was not read.

The stub's ABSENT marker now answers the flag the way kubectl does. New
cases pin the failed read and the deadline; the two budget cases are
re-timed for the extra clock reading.

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-28 18:45:50 +01:00
Marian Koreniuk 28d9d352e1 test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool>
kubernetes-nodes names its release kubernetes-nodes-<CR name> and refuses
to render one that does not start with kubernetes-nodes-<spec.cluster>-
(_helpers.tpl, since fd14126be). The Proxmox fixture named its
KubernetesNodes "md0", so helm install of the pool failed, no Machine was
ever created, and the first real run of the lane on a self-hosted stand
timed out on the Machine assert after 20 minutes.

Name the pool k8s-e2e-md0 and add a fixture test that fails on any name
the chart would refuse.

Assisted-by: LLM
Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
2026-10-28 18:44:04 +01:00
Aleksei Sviridkin ae9e7c4c64 test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on
`strategy.name: RetryOnFailure` for both actions, and the e2e
HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names
those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets
the same strategy for the same reason and had no pin.

This adds
`packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a
mirror of `csi_install_retry_test.yaml` rendered on the proxmox
substrate, and updates the guard comment that pointed at the gap. No
template change.

Verified with `helm unittest .` in `packages/apps/kubernetes` (32
suites, 349 tests).

Closes #4731


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Tests**
* Expanded automated coverage for Proxmox CSI installation and upgrade
retry behavior, including the retry interval and remediation retry
limits. The checks verify that retries continue according to the
configured settings during installation and upgrades. This update does
not change user-facing functionality.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-28 18:30:06 +03:00
Andrei Kvapil d62056ef03 fix(foundationdb)!: pick the version from a fixed list
cluster.version took any string and passed it to the operator as the
FoundationDB version, so a tenant chose the image tag rather than a
supported release line. The application now takes version (v7.4, v7.3,
v7.1) and maps it to a tag through files/versions.yaml, the same way
postgres and mariadb do.

The lines are the ones the pinned operator ships client libraries for,
which hack/update-versions.sh reads from the operator chart. The map
keeps v7.3 on 7.3.63, the previous default, so a cluster that never set
the version renders exactly as before.

Assisted-by: LLM
Signed-off-by: Andrei Kvapil <kvapss@gmail.com>
2026-10-23 19:14:15 +05:00
Marian Koreniuk 782e0ab64d fix(e2e): validate the leak allowlist against every suite layout
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the
selectors already see step-file suites and a Test nested below the suite
directory. A suite in one of those layouts could not be allowlisted at
all: the run stopped before the first suite. Validate the allowlist with
the rule hack/select-e2e.sh enumerates suites with, spelled the same way
on both sides and cross-referenced so they move together.

Describe the three layouts once in the e2e-chainsaw README, point the
agent doc at it, and list kubernetes-proxmox as the parked suite.

Closes #4659

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:20:19 +02:00
Marian Koreniuk 61238500dd fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with
the process umask and never removed it, so it outlived the run on the
self-hosted runner, world-readable under the usual umask. dump_diagnostics
then read whatever sat at that path, so a run that failed before the
fetch reported the previous run's tenant as its own.

The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points
at runner.temp. It is written under umask 077, removed at the start of
apply-tenant so this run never sees a leftover, removed again in the
suite's finally, and once more in the workflow's credential cleanup step
for a run killed before finally runs.

Closes #4806

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:17:50 +02:00
Marian Koreniuk cac152048e fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>-
addons. The worker pool is kubernetes-nodes-<test>-md0, an Application
HelmRelease the apiserver stamps with RetryOnFailure like the parent, on
the node-join path through the talos-reconcile Job it renders. Its name
does not start with the addon prefix, so a failed action on it was never
read, on the passing path or in the failure-path report.

Read it on both paths. The pool name is derived from the parent, the way
the addon prefix is, so the two cannot drift. The failure-path report
checks that the release exists first: a run that failed between the
parent and the pool apply has none, and that is a fact about where the
run stopped rather than a failed read.

Closes #4737

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:14:09 +02:00
Marian Koreniuk 98d0402096 test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on
strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease
guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml
sets the same strategy for the same reason and had no pin. Add the mirror
test and update the guard comment that pointed at the gap.

No template change.

Closes #4731

Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua>
Assisted-by: LLM
2026-10-19 23:08:02 +02:00
Andrey Kolkov 9cc3a9e459 test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.

Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
  up via BackupClass + BackupJob, restores in place, and asserts the VM returns
  Running off a disk PVC the restore created: a new uid, Velero's restore-name
  label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
  enough, because existingResourcePolicy: update also stamps it on a PVC Velero
  only patched, which moves no data. A guest-side boot proof is unavailable: the
  KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
  place with the same three assertions. The controller prepares the target
  (suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
  class the Velero data mover waits for a Pod to select a node for the restored
  PVC, which a detached disk never gets, so the script starts a pause Pod on the
  new PVC, standing in for the VM that would attach it.

The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.

Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
2026-10-13 18:05:45 +04:00
Andrey Kolkov 43b02bda30 test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app
kind and the chart prefix the helpers derive sentinel and Secret names
from. The demo BackupClass routes Valkey to the same demo strategy.

The valkey suite gains valkey-backup-roundtrip, which drives that demo
end to end: seed a marker, back it up, restore it in place and into a
copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a
seaweedfs change now selects the valkey suite too, and since the
round-trip runs examples/backups/redis as its harness, an edit there
selects both the redis and the valkey suites. The reduced E2E install
keeps the backup stack for the valkey suite, as it does for the other
backup round-trip suites.

Signed-off-by: Andrey Kolkov <androndo@gmail.com>
Assisted-By: LLM
2026-10-13 12:34:24 +04:00
Aleksei Sviridkin cfa9c5a5c7 fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does

Chainsaw asserts in the e2e suites can fail long before their timeout.
Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other
API error. When etcd changes its leader, kube-apiserver answers the GET
with a 500, and whichever suite happens to be polling goes red. Between
2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached
chainsaw, which is 6 of the 23 assert failures in that window. Each one
has `etcdserver: leader changed` in the apiserver logs within 0.2 s of
the failure. Five of the six came after 10-09, and four of them were the
computeplane ingress-nginx assert.

The fix is kyverno/chainsaw#2761, which is still open and not in any
release. Until it is released, the e2e sandbox image builds chainsaw
v0.2.15 from source with that PR's commit applied unchanged, using the
same Go minor and build flags as the v0.2.15 release binary. The build
runs the tests of the patched packages. It also runs chainsaw against a
fake apiserver that answers 500: a short run of 500s must not fail the
assert, and an endless one must fail it at its timeout. Plain v0.2.15
fails both cases after the first GET.

The Dockerfile header lists what to remove once a chainsaw release
carries the fix. The README sentence added here has to go too, and so
does the Renovate rule that keeps the builder on Go 1.26.

The upstream PR also breaks one chainsaw test outside the packages it
touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks
the new `; operation terminated with error: context deadline exceeded`
suffix. I found it with a full `go test ./...` on v0.2.15 with and
without the patch. The image build tests only the patched packages, so
that test does not run here.

One trade-off: when the API stays broken, an assert or `error` step now
runs to its timeout instead of failing within seconds, so a red run
caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw
installed on its runner, not the sandbox image, so it does not get this
fix.

Known gaps, left for later: the new test covers only an assert on a
named object, not one that lists objects or an `error` operation, and
its skip guard does not notice if no test runs at all. The patch file
has no `-whitespace` entry in `.gitattributes`, so `git diff --check`
reports whitespace inside the upstream patch. A multi-platform build of
the image would run the builder tests once per platform; the image is
built for amd64 only.

Closes #4720

### Screenshots

Not a UI change.

### Downstream repositories

None of them builds or uses the e2e sandbox image; the change stays
inside this repo's CI.

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

## Summary by CodeRabbit

* **Bug Fixes**
* End-to-end sandbox assertions now retry after transient Kubernetes API
errors instead of failing immediately. Persistent errors still cause
assertions to fail when their timeout expires.
* **Testing**
* Added checks that verify assertions recover after temporary API errors
and fail within the expected timeout when errors persist.
* **Maintenance**
* Updated sandbox image build and dependency update settings for the
Chainsaw testing tool.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-10-12 17:25:39 +03:00
Yasar Rahman 9a88afe6ca feat(redis): run Sentinel on the same image as the Redis pods
The Sentinels were left on redis:6.2.6-alpine when the operator moved to
the freshworks-oss fork, so that its new compiled-in default could not
move them on upgrade. 6.2 is end of life, and every instance now runs a
Sentinel one or two major versions behind the Redis it monitors.

The rollback risk behind that pin, a newer engine rewriting dump.rdb in
a format the older one cannot load, does not apply to Sentinel. It
stores no RDB or AOF, only sentinel.conf on an emptyDir that the
config-copy init container rewrites from the operator's ConfigMap on
every start, so moving back to an older image starts from a clean
config. Taking the image from the same version map as the data pods also
means one image per node instead of two.

The e2e comment that relied on busybox's timeout exit code in the
Sentinel container is updated, since that container now carries
coreutils.

Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com>
Assisted-by: LLM
2026-10-11 13:16:11 +05:00