mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
473cfdf478e1dc8d44e5c2bec93f9006653d27fa
321
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
f061739ce2 |
ci(e2e): raise the E2E job limit from 215 to 265 minutes (#5288)
## What this PR does Full chainsaw run doesn't fit in 215 minutes anymore. Last four nightly runs were cancelled at the limit, and PRs that select full suite end same way while every suite that got to run passed (#5283 for example). This raises limit for all four jobs that run the suite (pull-requests, e2e-fork, e2e-tag, nightly) from 215 to 265, so they stay on one figure. Comments that state the cap follow it. Making e2e itself shorter is separate work. Part of #5261, `Collect report` ceiling in fork workflow from there is not covered here. ### Screenshots ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note ci(e2e): E2E job limit raised from 215 to 265 minutes. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Increased the time limit for end-to-end test runs from 215 to 265 minutes across applicable workflows. This allows longer test runs to complete before the overall job is stopped. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
c63e83466a |
ci(e2e): raise the E2E job limit from 215 to 265 minutes
A green run of the full Chainsaw suite no longer fits in 215 minutes: the last four nightly runs were cancelled at the limit, and pull requests that select the full suite end the same way while every suite that got to run passed. All lanes that run the suite keep one figure, so a change that passes on a pull request is not cut off on nightly or on a tag. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
35823c6c9c |
test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split; it still described two of them after the third was added, so a reader starting from it would miss the only Test that covers the system-bucket path. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c26d2773a5 |
test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever BackupJob ran in tenant-root before this Test, from this suite or another: Chainsaw does not fix the order of the Tests in a suite, so no particular one can be named. The restore budget is also not spent on a credentials hold: by then the stamp is older than the backup's own two-minute wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
5a2582e21c |
test(observability-proxy): preserve JSON seed request bodies
Send JSON seeds with their content type so VictoriaLogs does not consume
log records while parsing form parameters before ingestion.
Fixes:
|
||
|
|
3e907e6f9a |
test(observability-proxy): declare the shell of the e2e helper library
The library is sourced by sh scripts and has no shebang, so shellcheck could not tell which shell to check it as and refused the file. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
f775948d2c |
test(observability-proxy): replay the early line into the e2e tail
The background tail goes through the console route and the proxy's reviews before it subscribes, so the early ingest can land first, and a tail without start_offset returns only lines written after it starts. The tail now replays the last minute, which still leaves the hold and the late-line check meaningful. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
72a94ee426 |
test(clickhouse): give the round-trip's preconditions room for their deletes
The step's ceiling SIGKILLs the shell and reports only "signal: killed", while a kubectl --timeout that fires first names the object that would not go. The deletes it runs now wait up to 270s in all, past the 3m ceiling, so a stuck delete late in the list would be reported by the ceiling instead of by kubectl. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0dc5a54731 |
test(clickhouse): remove the round-trip's TLS objects a killed run left
Chainsaw deletes an applied object only when its own apply created it; one left behind by a run killed past its cleanup is patched and kept. The Certificate would then outlive every later run and keep re-issuing its Secret after each step cleanup, and a probe Pod whose sleep had ended never turns Ready, failing the next run's assert. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f003851338 |
fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled like an application database. The init job made <db>_admin their owner, installed the auto-grant event trigger in them and, once the entry left values, ran DROP DATABASE on them. That DROP can only fail (template databases cannot be dropped, and the job is connected to postgres), so every later upgrade failed in the init job and the release stayed in UpgradeFailed even after the entry was put back. template1 also passed the takeover on to every database created from it afterwards, where the trigger, naming a role about to be dropped, makes CREATE SCHEMA fail. A release a previous version let take one of them over has to be repaired without dropping it, and without touching what this chart did not create: a role or owner that merely carries an expected name may belong to a tenant or an administrator, and a role comment can be set by any CREATEROLE holder, so only the cozystack:managed mark is trusted. What the marked roles own goes to cozystack:orphaned rather than to the superuser, since a tenant's SECURITY DEFINER function would otherwise run with superuser rights. A failed lookup of the databases cloned from template1 has to stop the run before template1 stops being selected, or those clones would never be repaired. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
8784863632 |
test(observability-proxy): wait for every seeded line before checking scope
The suite checked that B's line and the account-7 line are absent from A's answers but only waited for A's line to become searchable, so an absence could come from a line the store had not indexed yet. The step budgets in the comments now count the calls the steps make. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
ebb5a9cec4 |
test(observability-proxy): cover the namespace logs scope end to end
Unit tests mock the API server and the store, so nothing shows that the console's nginx, the proxy's TokenReview and SubjectAccessReview and a real VictoriaLogs agree. The suite drives a viewer of one tenant through the console against a seeded store: it reads its own namespace, is refused a sibling's, cannot widen the scope with client filters, an AccountID header or dot segments, and keeps a tail open past a minute. The root store is off in e2e, so the suite runs its own VictoriaLogs under the root store's Service name rather than waiting on Fluent Bit. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
28f3613450 |
test(clickhouse): delete the round-trip's TLS Secret in its own step's cleanup
Chainsaw applied the Certificate, so the scripted delete of it in the run step's finally was hand-rolled teardown of a resource Chainsaw already reclaims. Chainsaw deletes a step's applied resources, waiting for them to go, before it runs that step's cleanup operations, which puts the Secret's delete after the Certificate's without any ordering written into a script. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6e23bd5629 |
test(clickhouse): delete the round-trip's Certificate before its Secret
cert-manager issues the Secret of a Certificate that still exists again as soon as it is deleted, and the Certificate is only removed by Chainsaw's own cleanup after the step's finally has run. Deleting the Secret first therefore left a freshly issued copy behind in tenant-root, carrying no owner reference to take it down with the Certificate. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1003cdb1b6 |
test(clickhouse): find the round-trip's Ingress Secret by its owner label alone
kubectl refuses a resource name together with a label selector, and the restore asked for both with its stderr discarded, so the lookup always came back empty and the Secret the Test serves under the platform's S3 Ingress name was never deleted. The next run then stops on its own precondition that the Secret must not exist. A lookup that does fail now fails the restore, since an empty answer from a failed read is what kept this hidden. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6280d06d01 |
test(clickhouse): describe the round-trip's guards by the archive-recording restore
Restores now take the archive the Backup recorded rather than the newest one, and a leftover Backup of the BackupJob's name that records another archive fails the new BackupJob instead of being adopted. The comments justifying the sentinel row and the Backup cleanup, in the round-trip and in its unit test, still gave the earlier reasons, which would send the next reader after behaviour the driver no longer has. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
603114cd7c |
test(clickhouse): check the round-trip restored this run's sentinel row
Every run inserted the same sentinel row and accepted any row after the restore, while the Altinity restore takes the newest remote archive of the release. A restore of an older run's archive therefore passed. The row now carries a per-run value and exactly that row has to come back. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dee5e6d8bf |
test(clickhouse): mark the round-trip's reachability poll as a fixed wait
The probe loop sleeps between attempts because no Kubernetes condition reports that the host CoreDNS serves the rewrite and the Ingress serves the Test certificate. The e2e conventions allow such a sleep only under the marker that records it is waiting for something to model it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cabe9a07e2 |
test(clickhouse): reclaim the round-trip Backup by its name
The Altinity driver names the Backup after the BackupJob and adopts one that already exists under that name. The round-trip's cleanup deleted the Backup only through the BackupJob's status.backupRef, so a Backup whose BackupJob never recorded it survived, and the next run's BackupJob would have adopted it in place of a fresh backup. Both the cleanup and the preconditions now delete it by name; the preconditions ceiling grows to cover the extra wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a662198991 |
test(clickhouse): restore the round-trip's shared objects after a step ceiling
The round-trip changes the host CoreDNS Corefile and the S3 Ingress TLS Secret and puts both back from its EXIT trap. When the run step's ceiling fires, Chainsaw kills the shell and the trap never runs, so the rewrite stays in a cluster-wide object and the next run refuses to start. The step's finally runs regardless, so it repeats the restore; after a run whose trap already restored them it finds nothing to undo. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
003bc3ed0b |
test(clickhouse): fail the round-trip when CoreDNS cannot be restored
The round-trip puts the host CoreDNS Corefile back from its EXIT trap, which runs with errexit off. A refused kubectl replace, for instance a resourceVersion conflict with another writer, fell through to the success message and returned 0, so the run stayed green while the rewrite remained in a cluster-wide object and the next run refused to start. The restore now returns the failure. The restore moves into a file of its own so a unit test can run it against a stubbed kubectl and so other callers can share it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0ad642e68d |
test(clickhouse): size the system-bucket round-trip server for SQL
The round-trip ClickHouse ran on the nano preset, whose 128Mi limit the clickhouse-1 contract suite never exceeds because it sends no query. This suite writes a sentinel row before the backup, and the server was OOM-killed on that first statement (lastState OOMKilled, exit 137), so the Test failed before it reached the behaviour under test. clickhouse-2 runs the same statements on the small preset. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dbcc2af679 |
test(clickhouse): require the system-bucket round-trip to run on the sidecar that started without credentials
A Pod restart during the round-trip would hand the sidecar fresh credentials at start, which is how the old environment-based sidecar got them, and the Test would pass without the running sidecar ever picking them up. The Pod uid and the sidecar's restart count must stay the same across both jobs. The probe deadline is now measured on the clock: counting only its pauses let a hanging exec stretch it past the step budget, so the step timeout would fire before the probe could say what stalled. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2e37aaec5f |
docs(backups): drop the claim that a region change needs a ClickHouse restart
The ClickHouse sidecar on the system bucket no longer reads S3 settings from environment set at start, so a re-projected region reaches it without a Pod rollout, and the contracts fixture no longer depends on optional Secret keys to start its Pod. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e752952047 |
test(clickhouse): back up and restore a running release through the system bucket
A ClickHouse release on the system bucket starts before the platform projects cozy-backups-creds into its namespace, and no suite exercised that order: the round-trips bring their own Bucket and endpoint, so the sidecar never had to pick up credentials that arrived after it started. The new Test starts the release without the projected Secret and runs a backup and an in-place restore through cozy-default against the bucket as advertised. In CI that host neither resolves nor has a certificate, so the step makes it reachable from tenant-root for its own duration and proves the served certificate before any job starts, which keeps a DNS or TLS problem from reading as a backup failure. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ba0b1ac6ef |
fix(e2e): validate the leak allowlist against every suite layout (#4972)
`lg_allowlist_validate` in `hack/e2e-chainsaw/_lib/leak-guard.sh`
accepted only `<suite>/chainsaw-test.y*ml`, while the selectors already
see step-file suites and a Test nested below the suite directory
(#4630). A suite in one of those layouts could not be put on the leak
allowlist at all: validation failed and the run stopped before the first
suite.
- `lg_suite_has_test` validates an allowlist entry with the rule
`hack/select-e2e.sh` enumerates suites with: `chainsaw-test.yaml`,
`chainsaw-test.yml`, or a numbered step file, anywhere in or below the
suite directory, with the selector's type test (a step-file name counts
unless it is a directory, a `chainsaw-test.y*ml` counts whatever its
type, so Chainsaw reports a directory by that name); a parked
`chainsaw-test.yaml.disabled` matches none. Both sides carry a comment
pointing at the other so they move together.
- `hack/e2e-chainsaw/README.md` describes the three layouts once, under
Layout, and lists `kubernetes-proxmox` as the parked suite.
`docs/agents/e2e-testing.md` points at it.
- Unit tests: every layout is accepted, including a `chainsaw-test.yaml`
that is a directory; a parked, empty, or not-a-step-file directory is
refused, including `1a-install.yaml` (passes the `find` glob, not the
step-file regex) and a directory named `01-install.yaml`.
Left as they are, with the reasons:
- `docs/agents/e2e-testing.md` calling `kubernetes-proxmox` the one
parked suite is correct on main now: `hack/e2e-chainsaw/backup/` was
removed in
|
||
|
|
013b166967 |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard (#4970)
`cozy_guard_all_helmreleases` in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` reads the parent `kubernetes-<test>` release and the addons selected by the `kubernetes-<test>-` prefix. The worker pool is `kubernetes-nodes-<test>-md0`, an Application HelmRelease the aggregated apiserver stamps with `RetryOnFailure` like the parent, and it sits on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. This reads it on both paths, with the `required` strategy check the parent gets, since the apiserver stamps both. The pool name is derived from the parent (`cozy_pool_helmrelease_name`), the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists before reading it, with `get hr --ignore-not-found -o name`: a run whose pool apply failed has no pool release (exit 0, empty output), and the report says so instead of reporting a failed read. For that run to reach the report at all, `run_kubernetes_test` now arms the exit handler (`_tenant_failure_on_exit`, with `CURRENT_TENANT_PARENT_HR`) right after the parent `Kubernetes` apply instead of after the `KubernetesNodes` apply: before, a failing pool apply exited under errexit with no handler installed, and the parent went unclassified. The snapshot half still no-ops until the tenant kubeconfig is set. A non-zero exit (a timeout, a kill, a Forbidden) is reported as the failed read it is, through `_cozy_guard_read_failure`, never as absence. The pool reads are gated on the report's own deadline as well as the phase clock, like the addon listing. Unit tests in `hack/run-kubernetes-remediation_test.bats` cover a failing pool release on the passing path, a clean one (the composite returns zero), an empty strategy on the pool, the report with and without the pool release, a cut-off existence read, and a pool read past the report budget; the ordering pin now also requires the handler to be armed before the pool apply. The kubectl stub gained an existence read: an `ABSENT` marker prints nothing and exits 0 under `--ignore-not-found` (NotFound without it, as kubectl does), and the failure markers apply to it as `<name>.exists`. Verified with `hack/cozytest.sh` and Bats on `hack/run-kubernetes-remediation_test.bats` (51 tests), `hack/run-kubernetes-node-join_test.bats` and `hack/remediation-guard.bats`. Closes #4737 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Bug Fixes** * Kubernetes release checks now include worker-pool releases alongside the parent release and add-ons, and flag missing or invalid install strategies and remediation failures. * Failure reports distinguish worker-pool releases that are absent from those whose status cannot be read. Existing releases are assessed against the phase gate and report time budget; when the budget is exhausted, skipped checks are identified. * Failure handling now begins immediately after the parent resource is applied, covering errors that occur while applying the worker pool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
75bb70e58d |
fix(e2e): arm the tenant failure handler before the pool apply
The exit handler was armed after the KubernetesNodes apply, so a failing pool apply exited under errexit with no handler installed: the parent went unclassified, and the report's "pool release does not exist" line described a run that could never reach it. Arm it right after the parent Kubernetes apply instead. The snapshot half still no-ops until the tenant kubeconfig is set, so the only change is that a failed pool apply now gets the parent classified and the absent pool reported. The ordering pin now also requires the handler to be armed before the pool apply. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
048c029e29 |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it (#4971)
`hack/e2e-chainsaw/_lib/run-proxmox.sh` wrote the tenant's super-admin kubeconfig to `$TMPDIR/proxmox-e2e-tenant.kubeconfig` with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. `dump_diagnostics` then read whatever sat at that path, so a run that failed before `fetch_tenant_kubeconfig` reported the previous run's tenant as its own. - The file now lives under `RUNNER_TEMP`, which the runner exports into every step and empties between jobs; `COZY_PROXMOX_WORKDIR` overrides it for a local run, and `TMPDIR` is the last fallback. The workflow sets no path of its own: the `runner` context is not available in job-level `env`. - `fetch_tenant_kubeconfig` writes it under `umask 077` in a subshell, since the workflow's `umask 077` covers only the staging step. - `apply-tenant`, the suite's first step, removes a leftover before doing anything else, so diagnostics in this run can only see this run's file. - The suite's `finally` removes it, before the `keep_tenant` exit, and the workflow's "Remove staged credentials" step removes it again for a run killed before `finally`. Verification: `sh -n` and shellcheck on the script; actionlint on `e2e-proxmox.yaml` (only the custom `proxmox` runner label is reported, as on main). `hack/proxmox-e2e-fixture_test.bats` has three new cases with a stub `kubectl`: the fetched file has no group or other bits under `umask 022`, `apply-tenant` removes a planted leftover, and the suite's `finally` (run out of the Chainsaw file on the `keep_tenant` path) removes the file, so `diagnostics` after it prints no tenant section. Each goes red with its line (`umask 077`, the leftover `rm`, the `finally` `rm`) removed. All `hack/proxmox-*.bats` pass (fixture 9, teardown contract 8, step1b 4, talos template 12), also under `hack/cozytest.sh`. Not run on a Proxmox stand: the only stand I have cannot dispatch the workflow at the moment, so the workflow steps themselves are unexercised beyond actionlint. Closes #4806 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Chores** * Proxmox end-to-end test runs store tenant credentials in a temporary work directory and create kubeconfig files with restricted permissions. * Stale tenant kubeconfig files are cleared before a new tenant fixture is applied. Cleanup removes the file even when tenant data is retained or a run is interrupted, preventing credentials from appearing in later diagnostics. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
040ae377ae |
test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool> (#5000)
## What this PR does The Proxmox suite (`hack/e2e-chainsaw/kubernetes-proxmox/`) has now run for the first time on a real self-hosted stand: a dedicated Cozystack v1.7.0-rc.1 management cluster on Proxmox VE, with the runner CageForge offered to host in cozystack/community#98. The suite failed before it reached the hypervisor, because the fixture and the chart disagree on what a node pool may be called. `kubernetes-nodes` names its release `kubernetes-nodes-<CR name>` and, since |
||
|
|
cd983d416e |
fix(e2e): take the selector's type test in the allowlist scan and pin its regex
lg_suite_has_test dropped every directory with `! -type d`, but the selector keeps a chainsaw-test.y*ml whatever its type, so Chainsaw can fail loudly on a directory by that name. An allowlisted suite in that shape failed validation and stopped the run before Chainsaw reported the real error. Use the selector's own `\( ! -type d -o -name chainsaw-test.yaml -o -name chainsaw-test.yml \)`, which still drops a directory with a step-file name. The step-file regex behind the find glob was never reached by a test: install-01.yaml is already filtered by the glob. 1a-install.yaml passes the glob and not the regex, so it pins the regex; a directory named chainsaw-test.yaml and one named 01-install.yaml pin the type test. The README said the runner recognises the same three layouts. It recognises none: it hands every top-level directory to Chainsaw. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
3d97e912b4 |
fix(e2e): drop the job-level runner.temp and pin the tenant kubeconfig handling
jobs.e2e.env set COZY_PROXMOX_WORKDIR from runner.temp, but the runner context is not available in job-level env, so GitHub would refuse the whole workflow at load time and the Proxmox lane would stop starting. actionlint reports it as `context "runner" is not allowed here`. Drop the variable. run-proxmox.sh already falls back to RUNNER_TEMP, which the runner exports into every step, and the cleanup step removes the file from there. Pin the behaviour the previous commit added, none of which a test covered: the fetched kubeconfig carries no group or other bits under umask 022, apply-tenant removes a file an earlier run left, and the suite's finally removes the file on the keep_tenant path, so diagnostics after it prints no tenant section. Each case goes red with its line removed. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
4d7d336efc |
fix(e2e): tell a failed pool existence read from an absent pool release
The failure-path report checked the pool release with `get hr -o name` and discarded the exit status and stderr, so a timeout, a kill, a Forbidden or a refused connection all printed that the run ended before the worker pool was applied. That is the conflation the guard wrapper exists to refuse, on the path where a slow apiserver is most likely. Read it with --ignore-not-found instead: exit 0 with empty output is an absent release, exit 0 with a name is one to classify, and any non-zero exit is a failed read named through _cozy_guard_read_failure. The pool reads also skipped the report's own deadline and checked only the phase clock, which says yes on every path that opens no phase. Check the deadline before the existence read, like the listing and the addons do, and name what was not read. The stub's ABSENT marker now answers the flag the way kubectl does. New cases pin the failed read and the deadline; the two budget cases are re-timed for the extra clock reading. Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
28d9d352e1 |
test(kubernetes): name the Proxmox fixture's node pool <cluster>-<pool>
kubernetes-nodes names its release kubernetes-nodes-<CR name> and refuses
to render one that does not start with kubernetes-nodes-<spec.cluster>-
(_helpers.tpl, since
|
||
|
|
ae9e7c4c64 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release (#4969)
The KubeVirt csi and cilium releases carry a helm-unittest pin on `strategy.name: RetryOnFailure` for both actions, and the e2e HelmRelease guard in `hack/e2e-chainsaw/_lib/run-kubernetes.sh` names those two tests as load-bearing. `helmreleases/csi-proxmox.yaml` sets the same strategy for the same reason and had no pin. This adds `packages/apps/kubernetes/tests/csi_proxmox_install_retry_test.yaml`, a mirror of `csi_install_retry_test.yaml` rendered on the proxmox substrate, and updates the guard comment that pointed at the gap. No template change. Verified with `helm unittest .` in `packages/apps/kubernetes` (32 suites, 349 tests). Closes #4731 <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Tests** * Expanded automated coverage for Proxmox CSI installation and upgrade retry behavior, including the retry interval and remediation retry limits. The checks verify that retries continue according to the configured settings during installation and upgrades. This update does not change user-facing functionality. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d62056ef03 |
fix(foundationdb)!: pick the version from a fixed list
cluster.version took any string and passed it to the operator as the FoundationDB version, so a tenant chose the image tag rather than a supported release line. The application now takes version (v7.4, v7.3, v7.1) and maps it to a tag through files/versions.yaml, the same way postgres and mariadb do. The lines are the ones the pinned operator ships client libraries for, which hack/update-versions.sh reads from the operator chart. The map keeps v7.3 on 7.3.63, the previous default, so a cluster that never set the version renders exactly as before. Assisted-by: LLM Signed-off-by: Andrei Kvapil <kvapss@gmail.com> |
||
|
|
782e0ab64d |
fix(e2e): validate the leak allowlist against every suite layout
lg_allowlist_validate accepted only <suite>/chainsaw-test.y*ml, while the selectors already see step-file suites and a Test nested below the suite directory. A suite in one of those layouts could not be allowlisted at all: the run stopped before the first suite. Validate the allowlist with the rule hack/select-e2e.sh enumerates suites with, spelled the same way on both sides and cross-referenced so they move together. Describe the three layouts once in the e2e-chainsaw README, point the agent doc at it, and list kubernetes-proxmox as the parked suite. Closes #4659 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
61238500dd |
fix(e2e): keep the Proxmox tenant kubeconfig per run and remove it
run-proxmox.sh wrote the tenant's super-admin kubeconfig to $TMPDIR with the process umask and never removed it, so it outlived the run on the self-hosted runner, world-readable under the usual umask. dump_diagnostics then read whatever sat at that path, so a run that failed before the fetch reported the previous run's tenant as its own. The file now lives under COZY_PROXMOX_WORKDIR, which the workflow points at runner.temp. It is written under umask 077, removed at the start of apply-tenant so this run never sees a leftover, removed again in the suite's finally, and once more in the workflow's credential cleanup step for a run killed before finally runs. Closes #4806 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
cac152048e |
fix(e2e): read the KubernetesNodes pool release in the HelmRelease guard
cozy_guard_all_helmreleases reads the parent and the kubernetes-<test>- addons. The worker pool is kubernetes-nodes-<test>-md0, an Application HelmRelease the apiserver stamps with RetryOnFailure like the parent, on the node-join path through the talos-reconcile Job it renders. Its name does not start with the addon prefix, so a failed action on it was never read, on the passing path or in the failure-path report. Read it on both paths. The pool name is derived from the parent, the way the addon prefix is, so the two cannot drift. The failure-path report checks that the release exists first: a run that failed between the parent and the pool apply has none, and that is a fact about where the run stopped rather than a failed read. Closes #4737 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
98d0402096 |
test(kubernetes): pin RetryOnFailure on the Proxmox csi release
The KubeVirt csi and cilium releases carry a helm-unittest pin on strategy.name: RetryOnFailure for both actions, and the e2e HelmRelease guard names those two tests as load-bearing. helmreleases/csi-proxmox.yaml sets the same strategy for the same reason and had no pin. Add the mirror test and update the guard comment that pointed at the gap. No template change. Closes #4731 Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> Assisted-by: LLM |
||
|
|
9cc3a9e459 |
test(vmi): add VMInstance and standalone VMDisk backup/restore e2e round-trips
The disabled hack/e2e-chainsaw/backup/ suite was a Chainsaw port of the retired
backup.bats, written against the old backup API, so the VMInstance and standalone
VMDisk Velero paths had no e2e coverage. Replace it with two round-trips in the
vminstance suite that drive the validated examples/backups/{vmi,vmdisk} scripts,
so the documented demo and the automated test stay in step.
Both run un-gated in CI:
- vminstance-2-backup-roundtrip boots a tiny cirros VM (u1.nano, 2Gi), backs it
up via BackupClass + BackupJob, restores in place, and asserts the VM returns
Running off a disk PVC the restore created: a new uid, Velero's restore-name
label and the -orig- PVC keepOriginalPVC leaves behind. The label alone is not
enough, because existingResourcePolicy: update also stamps it on a PVC Velero
only patched, which moves no data. A guest-side boot proof is unavailable: the
KubeVirt CR disables serial-console logging and cirros ships no guest-agent.
- vmdisk-2-backup-roundtrip imports a 5Gi disk, backs it up, and restores it in
place with the same three assertions. The controller prepares the target
(suspend HelmRelease, delete DataVolume, rename PVC). On a WaitForFirstConsumer
class the Velero data mover waits for a Pod to select a node for the restored
PVC, which a detached disk never gets, so the script starts a pause Pod on the
new PVC, standing in for the VM that would attach it.
The disk storageClass is an override (default replicated; the CI steps pass the
lane's COZY_E2E_STORAGE_CLASS, 'local' on the DRBD-off gate) and the script waits
are bounded so their worst case stays under the 40m op timeout, sized to keep the
whole parallel:1 suite inside the 215m job cap. On failure the catch also dumps
the Velero Restore, DataDownload and DataUpload objects. Each harness stands up a
throwaway in-cluster BSL (the platform cozy-default BSL derives the external S3
endpoint, unroutable in kind); its finally resumes the suspended HelmReleases,
reclaims the keepOriginalPVC orig PVC and its LINSTOR volume, and drains the
Velero backups before deleting the BSL. The example scripts are non-interactive
so the harness can drive them.
Assisted-by: LLM
Signed-off-by: Andrey Kolkov <androndo@gmail.com>
|
||
|
|
43b02bda30 |
test(e2e): run the Redis backup round-trip against Valkey
The Redis backup demo now takes APP_KIND=Valkey, which switches the app kind and the chart prefix the helpers derive sentinel and Secret names from. The demo BackupClass routes Valkey to the same demo strategy. The valkey suite gains valkey-backup-roundtrip, which drives that demo end to end: seed a marker, back it up, restore it in place and into a copy. It writes to tenant-root's seaweedfs through seaweedfs-s3, so a seaweedfs change now selects the valkey suite too, and since the round-trip runs examples/backups/redis as its harness, an edit there selects both the redis and the valkey suites. The reduced E2E install keeps the backup stack for the valkey suite, as it does for the other backup round-trip suites. Signed-off-by: Andrey Kolkov <androndo@gmail.com> Assisted-By: LLM |
||
|
|
cfa9c5a5c7 |
fix(e2e): keep chainsaw asserts polling through transient API errors (#4867)
## What this PR does Chainsaw asserts in the e2e suites can fail long before their timeout. Chainsaw v0.2.15 keeps polling only on NotFound and stops on any other API error. When etcd changes its leader, kube-apiserver answers the GET with a 500, and whichever suite happens to be polling goes red. Between 2026-09-27 and 10-11 this hit 6 of the 397 e2e attempts that reached chainsaw, which is 6 of the 23 assert failures in that window. Each one has `etcdserver: leader changed` in the apiserver logs within 0.2 s of the failure. Five of the six came after 10-09, and four of them were the computeplane ingress-nginx assert. The fix is kyverno/chainsaw#2761, which is still open and not in any release. Until it is released, the e2e sandbox image builds chainsaw v0.2.15 from source with that PR's commit applied unchanged, using the same Go minor and build flags as the v0.2.15 release binary. The build runs the tests of the patched packages. It also runs chainsaw against a fake apiserver that answers 500: a short run of 500s must not fail the assert, and an endless one must fail it at its timeout. Plain v0.2.15 fails both cases after the first GET. The Dockerfile header lists what to remove once a chainsaw release carries the fix. The README sentence added here has to go too, and so does the Renovate rule that keeps the builder on Go 1.26. The upstream PR also breaks one chainsaw test outside the packages it touches, `Test_runE` in `pkg/commands/assert`. Its expected string lacks the new `; operation terminated with error: context deadline exceeded` suffix. I found it with a full `go test ./...` on v0.2.15 with and without the patch. The image build tests only the patched packages, so that test does not run here. One trade-off: when the API stays broken, an assert or `error` step now runs to its timeout instead of failing within seconds, so a red run caused by a dead API gets longer. The proxmox e2e lane runs the chainsaw installed on its runner, not the sandbox image, so it does not get this fix. Known gaps, left for later: the new test covers only an assert on a named object, not one that lists objects or an `error` operation, and its skip guard does not notice if no test runs at all. The patch file has no `-whitespace` entry in `.gitattributes`, so `git diff --check` reports whitespace inside the upstream patch. A multi-platform build of the image would run the builder tests once per platform; the image is built for amd64 only. Closes #4720 ### Screenshots Not a UI change. ### Downstream repositories None of them builds or uses the e2e sandbox image; the change stays inside this repo's CI. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(e2e): a chainsaw assert in the e2e suites keeps polling through a transient API error such as an etcd leader change, instead of failing the suite long before its timeout. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * End-to-end sandbox assertions now retry after transient Kubernetes API errors instead of failing immediately. Persistent errors still cause assertions to fail when their timeout expires. * **Testing** * Added checks that verify assertions recover after temporary API errors and fail within the expected timeout when errors persist. * **Maintenance** * Updated sandbox image build and dependency update settings for the Chainsaw testing tool. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9a88afe6ca |
feat(redis): run Sentinel on the same image as the Redis pods
The Sentinels were left on redis:6.2.6-alpine when the operator moved to the freshworks-oss fork, so that its new compiled-in default could not move them on upgrade. 6.2 is end of life, and every instance now runs a Sentinel one or two major versions behind the Redis it monitors. The rollback risk behind that pin, a newer engine rewriting dump.rdb in a format the older one cannot load, does not apply to Sentinel. It stores no RDB or AOF, only sentinel.conf on an emptyDir that the config-copy init container rewrites from the operator's ConfigMap on every start, so moving back to an older image starts from a clean config. Taking the image from the same version map as the data pods also means one image per node instead of two. The e2e comment that relied on busybox's timeout exit code in the Sentinel container is updated, since that container now carries coreutils. Signed-off-by: Yasar Rahman <yasar.rehman@gmail.com> Assisted-by: LLM |
||
|
|
813c618086 |
fix(e2e): keep chainsaw asserts polling through transient API errors
Chainsaw v0.2.15 ends an assert or error operation on the first API error other than NotFound. kube-apiserver answers a GET with 500 when etcd changes its leader, so a leader change that lands inside a poll fails the step long before its own timeout, on whichever suite happens to be polling. The sandbox image now builds v0.2.15 from source with the fix proposed upstream in kyverno/chainsaw#2761, applied unchanged, until a chainsaw release carries it. The build runs the tests of the patched packages and a test that drives chainsaw built from the patched tree against a fake apiserver: a run of 500s must not fail an assert, and a persistent one must fail it at its timeout rather than at the first error. Closes: #4720 Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
be2387323b |
feat(gateway)!: reserve a passthrough listener's name by the spec alone (#4638)
## What this PR does This keeps the TLS-passthrough services on port 443 of the tenant Gateway and makes the passthrough rules hold on Cilium 1.20.2. It is stacked on #3343 and has to land with it (stack #4717). On 1.20, upstream Cilium marks an HTTPS-terminate listener and a TLS-passthrough listener on one port `Conflicted` when their hostnames overlap, and drops both from Envoy. That took `*.<apex>` and the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners down together on every tenant Gateway in `dns01` and `existingSecret` mode (#3912). #3343 now ships a patched operator that conflicts the pair only when the hostnames are equal, so the layout stays as it is on main: one Gateway, one address, everything on 443. What remains is the equal-hostname case and the cleanup that 1.20 allows. Under `http01`, a name a passthrough listener answers is now reserved by the spec alone. Before, the controller dropped the terminate listener for such a name only once a TLSRoute was attached to the passthrough listener and its backend Service resolved, which copied how Cilium builds filter chains. Without a working TLSRoute, an HTTPRoute on `api.<apex>` kept its terminate listener next to `tls-api`, and on 1.20 that equal-hostname pair takes both listeners down. Now such an HTTPRoute gets no listener and no certificate, TLSRoute or not. If all its hostnames are reserved it reads `Accepted=False` with `NoMatchingListenerHostname`. If it also claims a free hostname it stays `Accepted` and is served there. The backend lookup, its Service and ReferenceGrant watches and their RBAC rules are gone. Other changes: - `tlsPassthroughListeners` is accepted under `dns01` and `existingSecret`. The old refusal relied on 1.19.5 serving all ports from one Envoy listener; 1.20 splits ports whose SNI overlaps. - A TenantGateway reads `Ready=False` with reason `GatewayClassUnsupported` when its GatewayClass lists features in `status.supportedFeatures` but not `TLSRoute`, while it renders passthrough listeners. A class that lists nothing is not judged. Cilium 1.20.2 lists `TLSRoute`. - Comments, the README and one error message no longer describe Cilium 1.19.5 as the shipped release. - The gateway chart writes `tlsPassthroughServices` on the publishing tenant only, behind the same `gateway.enabled` switch and `publishing.exposedServices` filter as the platform TLSRoutes. Before, every tenant Gateway got the three passthrough listeners with nothing routed to them, and the reservation above would have taken those names from a child tenant's own HTTPRoutes. The uniform `allowedRoutes.kinds` on port 443 from #3087 stays. The Cilium bug behind it is fixed since 1.19.6, but several tests pin it, so removing it is a separate cleanup. ### Upgrade notes Only the publishing tenant keeps the `api`, `vm-exportproxy` and `cdi-uploadproxy` passthrough listeners. That is `tenant-root` unless `publishing.ingressName` says otherwise, and only for names in `publishing.exposedServices` while `gateway.enabled` is on. Other tenants lose these listeners on upgrade. Nothing shipped was attached to them, so a child tenant's own HTTPRoute on `api.<child-apex>` keeps its listener and certificate. Two narrow cases do change. On the publishing tenant under `http01`, an HTTPRoute on an exposed passthrough name loses its listener and certificate when the platform TLSRoute for that name forwards nowhere (the owning package is not installed, or its backends do not resolve). A TLSRoute attached by hand to a child tenant's `tls-*` listener loses its parent. To keep that listener, merge-patch `tlsPassthroughServices` into the tenant's `TenantGateway` after the upgrade. From then on the chart does not render the key there, and later upgrades leave the patch alone. `dns01` and `existingSecret` do not change: the wildcard listener still covers such a name, and the passthrough listener answers it by its exact name. Tested locally: Go tests for the controller, `go vet`, helm-unittest for `extra/gateway` and `cozystack-controller`, `make generate` and `make manifests` with no diff. The gateway e2e scenario checks that a child tenant renders no passthrough listener. It then declares `tls-api` and a passthrough listener on a native port on that tenant, checks that the tenant stays Ready with no route behind either, and checks the reservation against `tls-api`. Unit tests catch a regression in either change. In e2e, dropping the reservation turns the gateway suite red, and dropping the chart scoping does not, because CI never sets `gateway.enabled` (#4846). The controller was not run on a live cluster; the operator patch it depends on was, see #3343. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/725 - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note feat(gateway)!: the tenant Gateway renders the api, vm-exportproxy and cdi-uploadproxy passthrough listeners only on the publishing tenant, for the names publishing.exposedServices lists while gateway.enabled is on. Under certMode http01 a name a TLS-passthrough listener answers no longer gets an HTTPS listener on 443, with or without a TLSRoute behind it. On the publishing tenant, an HTTPRoute on such a name whose TLSRoute forwards nowhere loses its listener and certificate. A TLSRoute attached by hand to a tls-* listener of another tenant loses its parent. An HTTPRoute with some reserved and some free hostnames used to be rejected whole; it is now Accepted and its free hostnames get a listener and a certificate. tlsPassthroughListeners is accepted under dns01 and existingSecret. A TenantGateway whose GatewayClass does not list TLSRoute in its supported features reads Ready=False while it renders passthrough listeners. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **New Features** * TLS passthrough listeners are supported with HTTP-01, DNS-01, and existing-secret certificate modes. Edge-terminated Gateways remain unsupported. * Declaring a passthrough hostname reserves it from HTTPS termination and certificate creation, even when no route is attached. * Passthrough listeners are rendered only for exposed services on the enabled publishing Gateway. * **Bug Fixes** * Gateways that declare passthrough listeners but explicitly lack TLSRoute support are now reported as not ready. * Routes targeting a hostname served by a passthrough listener on a different Gateway are correctly withdrawn. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
ad3f15d4f3 |
feat(gateway)!: reserve a passthrough listener's name by the spec alone
The TenantGateway controller withheld the HTTPS-terminate listener for a name a passthrough listener answers only once a TLSRoute was attached there and resolved a backend, mirroring how Cilium builds its filter chains. That kept a served endpoint up on Cilium 1.19.5, but it tied every route's status to one implementation's translator and needed Service and ReferenceGrant watches to land the deferred decision. The shipped operator accepts a wildcard terminate listener beside a narrower passthrough listener on 443 and keeps conflicting every other overlap. A terminate listener for the passthrough name itself is not narrower, so on 443 the rule now protects the dataplane outright; on a native port Gateway API would serve the pair, and the rule holds as a policy that keeps a platform name with its passthrough listener. Reading it off the spec makes the outcome the same on every implementation. Under http01 an HTTPRoute claiming such a name gets no listener and no certificate whether or not a TLSRoute is attached. A route whose every hostname is reserved reads Accepted=False with NoMatchingListenerHostname; one that also claims a free hostname stays Accepted and is served there, as Gateway API accepts a route whose hostnames intersect at least one listener. dns01 and existingSecret are unchanged: their wildcard listener still covers such a name, and the passthrough listener answers it by its exact server name. The chart writes the passthrough services on the publishing tenant alone, so the names reserved by default are the ones the platform publishes there through its own TLSRoutes. The shape an upgrade changes is an HTTPRoute on such a name whose platform TLSRoute forwards nowhere, because it never attached (the package owning the service is not installed) or because none of its backends resolve: that route kept a terminate listener and now gets none. The backend resolution, its watches and the RBAC rules that served them go away. Assisted-by: LLM Signed-off-by: Aleksei Sviridkin <f@lex.la> |
||
|
|
6144805bea |
fix(mariadb): mount the chart's my.cnf under a .cnf name (#4843)
## What this PR does The MariaDB chart's my.cnf has not been read by the server since the mariadb-operator bump to v0.28.1. Up to v0.0.27 the operator mounted the referenced ConfigMap key as `/etc/mysql/conf.d/my.cnf` whatever the key was called. Since v0.0.28 the file takes the key's own name, so the chart's key `config` became `/etc/mysql/conf.d/config`. The server's `!includedir` reads only `*.cnf` files, so it skipped it, and every instance has been running on stock MariaDB defaults. On the default `t1.nano` preset that means a 128M buffer pool inside a 128Mi memory limit. This renames the key to `my.cnf` in the ConfigMap and in the MariaDB CR. The file content is unchanged. I started all four shipped versions (10.6.24, 10.11.15, 11.4.9, 11.8.5) with the rendered file under a 128m memory limit, and each one came up with the chart's values. The MySQL-only options in the file log a warning and are otherwise ignored. They and the deprecated `innodb_flush_method` are tracked in #4842. Tests: a helm-unittest suite pins the key on both sides, and the mariadb chainsaw suite gets a step that compares `@@innodb_buffer_pool_size` on the running server with the value in the rendered my.cnf. This changes behaviour on every existing instance, see the release note. Each instance rolls its pods once because the projected volume in the pod template changes: replicas first, then the primary with a switchover. After that the server runs with every setting in the chart's file. Instances created before the operator bump (May 2024) ran with these settings before. Newer ones never did. Closes #4836 ### Screenshots Not a UI change. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(mariadb): the chart's my.cnf is applied again. Since mariadb-operator v0.28.1 it was mounted under a name the server ignores, so MariaDB instances ran on stock defaults. Every existing instance rolls its pods once on upgrade (replicas first, then the primary with a switchover) and then runs with the chart's settings. That changes behaviour: buffer pool, redo log, connection limits and caches are sized from the instance's resources; sql_mode becomes NO_ENGINE_SUBSTITUTION, so strict mode is off and invalid values are truncated with a warning instead of rejected; innodb_flush_log_at_trx_commit=2 can lose up to about a second of committed transactions if the node crashes; innodb_fast_shutdown=0 makes every pod shutdown slower. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> ## Summary by CodeRabbit * **Bug Fixes** * MariaDB configuration is now stored and read using the `my.cnf` key. A verification check confirms the running server applies the configured buffer-pool size. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |