mirror of
https://github.com/cozystack/cozystack.git
synced 2026-11-21 20:23:39 +09:00
326589282693a0508db50817a0c54ffd8fe925f3
1605
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
|
|
3265892826 |
test(kubernetes): group sysctl safety checks with heredoc tests
The heredoc suite owns escaped-input and render-time rejection coverage. Keeping sysctl safety checks there leaves bootstrap-name agreement tests focused on the shared content hash. Assisted-by: LLM Signed-off-by: Michel Jansen <mjansen@whero.xyz> |
||
|
|
9b6128d3ca |
fix(kubernetes): reject invalid UTF-8 sysctl file values
Raw Helm file inputs can carry invalid UTF-8 that Go quoting turns into different Unicode characters. Reject these bytes rather than silently changing the requested worker configuration. Assisted-by: LLM Signed-off-by: Michel Jansen <mjansen@whero.xyz> |
||
|
|
7d9f913350 |
feat(kubernetes): configure per-pool Talos sysctls
Worker sysctl requirements extend beyond user namespaces. A string map keeps those settings in the Talos machine config without expanding the application API for each kernel knob. Tenant input crosses an unquoted shell heredoc in the reconcile Job, so preserve scalar contents without allowing shell or YAML injection. Assisted-by: LLM Signed-off-by: Michel Jansen <mjansen@whero.xyz> |
||
|
|
3badbdef74 |
fix(clickhouse): let a running backup sidecar pick up system-bucket credentials (#5232)
<!-- Thank you for making a contribution! Here are some tips for you: - Use Conventional Commits for the PR title: `type(scope): description` - Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore - Scopes are not an exhaustive list — pick the most specific scope for the change and extend the list when a genuinely new area appears. Examples: - System components: dashboard, platform, operator, cilium, kube-ovn, linstor, fluxcd, cluster-api - Managed apps: postgres, mariadb, redis, kafka, clickhouse, virtual-machine, kubernetes - Development and maintenance: api, hack, tests, ci, docs, maintenance - Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or add a `BREAKING CHANGE:` footer - If it's a work in progress, consider creating this PR as a draft. - Don't hesistate to ask for opinion and review in the community chats, even if it's still a draft. - Add the label `kind/backport` if it's a bugfix that needs to be backported to a previous version. --> ## What this PR does A ClickHouse release with `backup.useSystemBucket: true` can start before the platform projects `cozy-backups-creds` into its namespace. Its `clickhouse-backup` sidecar took the S3 settings from environment variables, which are fixed when the container starts, so the first BackupJob or RestoreJob of such a release failed with `no EC2 IMDS role found` until the Pod was restarted, and a rotated key stayed stale the same way. The projector now also writes a `clickhouse-backup.yml` key into `cozy-backups-creds` (bucket, `https://` endpoint, region when set, keys, `force_path_style` from `backupStorage.forcePathStyle`). On the system-bucket flow the chart mounts that key as an optional Secret volume and points `CLICKHOUSE_BACKUP_CONFIG` at it, instead of setting `S3_*` variables. clickhouse-backup reloads its config file in each API handler that reaches S3 (among them the `create_remote`, `restore_remote` and `list` calls the strategy makes; the `status` and `actions` reads it polls with do not) and kubelet rewrites a mounted Secret in place, so credentials projected after the Pod started, or rotated since, reach the running sidecar. `S3_PATH` stays in the environment because the prefix belongs to the release. Kubelet takes about a minute to refresh the volume, so the projector stamps `backups.cozystack.io/credentials-updated-at` when the projected content changes, and the Altinity driver holds a new backup or restore Job for up to two minutes after that stamp, with `Ready=False` and reason `WaitingForCredentials`; the condition is dropped once the Job runs. A stamp that cannot be read, or one further ahead than clock skew explains, is logged and ignored rather than holding the namespace's jobs for good. The hold applies to every Altinity job in a namespace whose projected Secret just changed, including releases on their own Secret, where it is only a delay. The new e2e Test `clickhouse-3-system-bucket-roundtrip` starts a ClickHouse on the system bucket with no projected Secret, then runs a backup and an in-place restore through `cozy-default` and checks that the sidecar that started without credentials served both jobs. In CI the system bucket's advertised host `s3.example.org` neither resolves nor gets an ACME certificate, so for the duration of its own step the Test serves a SeaweedFS-CA certificate under the S3 Ingress TLS Secret name and adds a marked `rewrite name exact` line for that host to the host CoreDNS Corefile, removing both on exit; it proves the served certificate before any job starts. The bug was reproduced on a test cluster running the code before this change: a ClickHouse started before the projection failed its first BackupJob with `no EC2 IMDS role found`. The new e2e Test has not run green yet outside CI, because it edits the host CoreDNS, which is shared on that cluster. ### Screenshots <!-- REQUIRED for UI changes: attach screenshots or screen recordings demonstrating the visual impact of your changes. PRs with UI changes without screenshots will not be merged. --> ### Downstream repositories <!-- Cozystack is upstream for repositories that are NOT kept in sync with it automatically, and nothing in CI compares the two sides, so a change here can break them silently. Which change forces what, and which file to touch, is the trigger map in docs/agents/contributing.md under "Downstream Repositories". Walk it before you tick anything. IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM TO FILL IN: 1. Walk the trigger map against the actual diff of this PR, file by file. Do not decide from the PR title or from your memory of what you changed. 2. For every repository you tick, open the follow-up PR there and link it on the same line. A ticked box with no link claims work that does not exist, which is worse than an empty box. Search that repository for an open PR or issue covering it first, and link that instead of filing a duplicate. 3. When the follow-up is out of scope, or needs a decision that is not yours to make, open an issue in that repository instead, link that, and say so in the PR body. Do not push a speculative PR into another repository just to fill a line here. 4. Do not tick "No downstream repository is affected" to make the checklist go away. If you are unsure, leave every box empty and say so in the PR body, so a human decides. --> - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: cozystack/website#799 (draft until this PR merges): `operations/services/backup-classes.md` for the mounted config file, the credentials wait and the upgrade window, without the Pod restart this change removes - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note <!-- Write a release note: - Explain what has changed internally and for users. - Start with the same `type(scope):` prefix as in the PR title - Follow the guidelines at https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md. --> ```release-note fix(clickhouse): a ClickHouse release on the platform system bucket (`backup.useSystemBucket: true`) now backs up and restores without a Pod restart when its namespace had no projected backup credentials at start, and picks up rotated credentials while running. The sidecar reads its S3 settings from a `clickhouse-backup.yml` file in `cozy-backups-creds` instead of environment variables, so the upgrade rolls every such ClickHouse Pod once. The first Altinity backup or restore in a namespace after its projected credentials change waits up to two minutes (`WaitingForCredentials`) for them to reach the sidecar. Upgrade the platform as a whole: a system-bucket BackupJob started while the new chart runs against the previous controller fails, because that controller does not write the new key. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * ClickHouse system-bucket backups now use a projected configuration file for S3 settings, supporting HTTPS endpoints and optional region and path-style settings. * Backup and restore jobs wait briefly for recently updated projected credentials before starting. * Path-style settings accept common true and false values and are normalized consistently across backup storage integrations. * **Tests** * Added end-to-end coverage for system-bucket backup and restore, verifying restored data without restarting the ClickHouse Pod or backup sidecar. * Added checks for TLS certificate validation and safe restoration of shared network and ingress settings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9978771858 |
feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does Adds `observability-proxy`, a new system component that lets a tenant user read the logs of their own namespace from VictoriaLogs with the same Kubernetes credentials they already use for the console, and with Kubernetes RBAC deciding what they can read. This is the first part of the design proposals [Tenant application logs](https://github.com/cozystack/community/pull/95) and [Tenant resource-consumption metrics](https://github.com/cozystack/community/pull/94); this PR implements the namespace logs root only. Metrics and the per-application roots follow in separate PRs. Why a proxy: today the log stores are shared per root tenant and have no notion of a caller, so tenants cannot be given direct read access without seeing other tenants' logs. The proxy authenticates the caller, authorizes the request from its path, and rebuilds the upstream query so that its scope cannot be widened by the client. Backward compatibility: this adds a new package and a new HelmRelease; nothing existing is renamed, removed or changed, so no migration is needed. What is included: - `cmd/observability-proxy` and `internal/observabilityproxy`: the proxy. - Serves `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>` for the VictoriaLogs read endpoints the console needs (`query`, `hits`, `stats_query_range`, `tail`, `field_values`, `stream_field_values`, `facets`). Anything else under the root answers `404`. - Authentication by `TokenReview`, authorization by one `SubjectAccessReview` built from the path (`list` on `cozystack.io/logs` in the namespace), both cached briefly. - The upstream request is rebuilt from an allowlist of parameters; the namespace scope is always forced as `extra_stream_filters`. Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL `options(...)` block, and tenant headers such as `AccountID`/`ProjectID` are not forwarded or are refused. - Non-canonical paths (`..`, `//`, encoded separators) are refused with `400` before any authorization is built, because the console nginx matches on the normalized path but forwards the raw one. - Time window, step, bucket count and result limits are clamped; per-user rate, concurrency and open-tail limits answer `429`. Invalid time, step and limit parameters answer `400`. - The store is resolved from the namespace's `namespace.cozystack.io/monitoring` label. - Live `tail` streams are passed through unbuffered, and the upstream tail request disables compression so headers reach the client before the first line. - `packages/system/observability-proxy`: chart and image build, wired into the root `make build`. - Platform wiring: a new `cozystack.observability-proxy` PackageSource, enabled by default on the full variants and opt-in on the slim ones, the same way as `monitoring-agents`. - RBAC: an aggregated `cozy:observability:view` ClusterRole grants `list` on `cozystack.io/logs` to tenant `view` and above. No API server serves this resource; the rule exists only for the proxy's `SubjectAccessReview`. The proxy's own ClusterRole is limited to the calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch` on namespaces). - Console nginx: when the proxy is not installed or cannot be reached, the observability roots answer `404` with a Kubernetes `Status` body, which the console shows as not available on this cluster; the proxy's own `502`/`504` for an unreachable store pass through unchanged. - Console: one regex `location` in the console nginx routes the four observability roots to the proxy ahead of `/apis`. The upstream is resolved at request time, so the console starts even when the proxy is not installed; those paths then answer `404` with a Kubernetes `Status` body. The console Deployment now receives the cluster domain for that resolution. - E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/` with two tenants, covering `401`/`403`/`200`, scope-widening attempts, path normalization and a long-lived `tail`. #### How it was tested - `go test -race` and `go vet` for the proxy packages. - helm-unittest for `observability-proxy`, `dashboard`, `tenant` and `core/platform`; bats tests for the console nginx routing. - The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs as the root store and refuses to run when a real one exists. Its cases were replayed by a script against the real store on a test cluster. - Verified on a test cluster: - `401` without a token and with an invalid token, `403` for another tenant's namespace, `200` for the caller's own. - Client `extra_stream_filters`/`extra_filters` (GET and POST), an `AccountID` header and another tenant's stream named inside the query do not widen the scope; `options(...)` answers `400`; a path with `..` pointing at another tenant answers `400`; an endpoint outside the allowlist answers `404`. - Through the full browser chain (OIDC login, gatekeeper, gateway, console nginx), a single `tail` stayed open for more than two minutes without reconnecting, and requests from the same session to other tenants' roots answered `403`. - Not verified: the `429` limits under real load. `golangci-lint` has not been run locally. #### Known limitations - The proxy is reachable only through the console route; tenant workloads have no egress rule to it. Every request takes its concurrency or tail slot before the body is read; the token and access reviews and the body read run under `--request-timeout`, so a stalled kube-apiserver answers `503` rather than holding the request; and writes to the client carry a deadline (renewed per line for tails). `--request-timeout=0` disables those bounds and should not be used in production. - The first lines a newly started pod writes can be missing from an already open `tail`, although they are in the store. The likely cause is that Fluent Bit picks up a new log file later than the VictoriaLogs live-tail window; later lines of the same pod arrive. A history query shows them. - The per-application roots (`.../<plural>/<name>/log`, `.../metrics`) and the namespace metrics root answer `404` in this PR. ### Screenshots No UI changes. ### Downstream repositories - [ ] No downstream repository is affected by this change - [x] [cozystack/website](https://github.com/cozystack/website) - follow-up: https://github.com/cozystack/website/issues/787 (docs land once the implementation PRs merge) - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: Downstream: `cozystack/website` is reached. A new platform component (`packages/core/platform/sources/observability-proxy.yaml`) triggers `content/en/docs/next/guides/platform-stack/_index.md` and `operations/configuration/licenses.md`; adding it to the system bundle (`packages/core/platform/templates/bundles/system.yaml`) may also need a line in `operations/configuration/variants.md`. A user-facing page for the logs API and the new tenant `view` permission is not required by the trigger map but would be useful. Tracked in cozystack/website#787. Other repositories: no trigger matched. `packages/core/platform/values.yaml` is not changed (only a test fixture), no package under `packages/apps/` gains a values field, and no CRD changes. The root `Makefile` `build` target builds one more image; this does not touch `hack/` or the make targets `ccp` relies on, so `ccp` is not considered affected, but a reviewer may want to confirm. ### Release note ```release-note feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Added an observability proxy for tenant-scoped log access through the console, including live log streaming. Requests are authenticated, access-controlled, and subject to configurable rate and concurrency limits. * Added platform packaging and deployment support, with console routing for observability requests. * **Tests** * Added automated coverage for access control, tenant scoping, query handling, streaming, proxy behavior, and deployment configuration. * **Documentation** * Updated end-to-end test guidance with host requirements for running the observability proxy suite. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
35823c6c9c |
test(clickhouse): list the system-bucket round-trip in the suite header
The header names the backup Tests of the suite and why they are split; it still described two of them after the third was added, so a reader starting from it would miss the only Test that covers the system-bucket path. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
c26d2773a5 |
test(clickhouse): name what really leaves the round-trip's Secret behind
The projected Secret the preconditions delete is left by whichever BackupJob ran in tenant-root before this Test, from this suite or another: Chainsaw does not fix the order of the Tests in a suite, so no particular one can be named. The restore budget is also not spent on a credentials hold: by then the stamp is older than the backup's own two-minute wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
fb5ac1826a |
test(observability-proxy): prepare seed stubs in both unit runners
The POSIX runner does not invoke Bats setup hooks. Initialize the
capture stub from the test body so the compatibility lane stays offline.
Fixes:
|
||
|
|
66acdb80e1 |
ci(e2e): skip final cleanup on disposable PR runners (#5260)
## What this PR does Remove the final sandbox teardown from the two PR E2E workflows. The runners are documented as disposable VMs, so destroying their local containers and zpools after testing delays the result without protecting a later run. Cleanup between preparation attempts and the Makefile targets for reused hosts stay unchanged. The job limit stays at 215 minutes. Suite-length budgeting, including nightly and tag runs, remains separate in #5261. This change relies on the existing disposable-runner contract; it does not change runner provisioning. ### Screenshots Not applicable. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **CI Workflows** * Removed automatic sandbox and cluster teardown after end-to-end report uploads. * Added checks that cluster cleanup runs before environment preparation is retried. These changes affect automated test workflows and do not alter user-facing product features. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
5a2582e21c |
test(observability-proxy): preserve JSON seed request bodies
Send JSON seeds with their content type so VictoriaLogs does not consume
log records while parsing form parameters before ingestion.
Fixes:
|
||
|
|
9cd51cac2e |
ci(e2e): skip final cleanup on disposable PR runners
Destroying runner-local state after a completed suite can push the job
past its deadline even though the runner is discarded. Keep cleanup
between preparation attempts and for reused hosts instead.
Fixes:
|
||
|
|
b53ca1fede |
fix(postgres): refuse system databases and never drop them (#5155)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Add the label `kind/backport` if it's a bugfix that needs to be
backported to a previous version.
-->
## What this PR does
`databases` accepts `postgres`, `template0` and `template1`, and the
init job treats them like application databases: it comments them
`database managed by helm`, creates `<db>_admin` and `<db>_readonly`,
makes `<db>_admin` the owner of the database and its schemas, and
installs the `trigger_auto_grant` event trigger. When the entry later
leaves values, the delete pass runs `DROP DATABASE` on it, which can
only fail:
- `template1`, `template0`: `cannot drop a template database`;
- `postgres`: `cannot drop the currently open database`, since the job
is connected to it.
The init job then fails on every upgrade, so the HelmRelease sits in
`UpgradeFailed` with `post-upgrade hooks failed`, and putting the entry
back does not help. `template1` also hands the takeover (owner of
`public`, default privileges, trigger) to every database created
afterwards. Once its roles are dropped by hand, the trigger left behind
names a role that no longer exists, so `CREATE SCHEMA` fails in every
new database, even as superuser.
This PR:
- **Refuses** the three names at render with a message that says what to
do. `cozyvalues-gen` has no way to constrain the keys of a
`map[string]Database`, so the refusal sits next to the existing
database-name guard, like the `users.postgres` refusal. The field
description says it too.
- **Never selects them for `DROP DATABASE`.** The delete pass excludes
them whatever values say.
- **Hands a taken-over system database back.** In a new `release system
databases` pass before the delete pass, for each of the three still
commented `database managed by helm`: drop the trigger and
`auto_grant_schema_privileges()`, `REASSIGN OWNED` / `DROP OWNED` for
its two roles in that database, give `public` back to
`pg_database_owner` (PostgreSQL 15+), and clear the comment. The
existing orphaned-roles pass then drops the roles; it never visits
`template1`, hence the per-database step. `template0` accepts no
connection and the job that commented it never got further, so only its
comment is cleared.
**Upgrade note.** A release that declares one of the three names fails
to render until the entry is removed. Removing it is now safe: the next
run releases the database instead of dropping it.
**Testing**
- `tests/system_databases_test.yaml`: 5 of its 6 cases fail on `main`
without the change. `helm unittest packages/apps/postgres`: 215 passed.
- I replayed the scenario against a `postgres:17` container, running the
rendered `init.sh` with `PGUSER=postgres`:
1. The `main` script with `databases: {appdb, template1, postgres}`
reproduces the takeover.
2. The `main` script with only `appdb` then fails with `cannot drop the
currently open database`.
3. This branch's script on that same state exits 0. It releases both
databases (owner `postgres`, no comment, no event trigger, `public`
owned by `pg_database_owner`) and drops `postgres_admin`,
`postgres_readonly`, `template1_admin` and `template1_readonly`. A
database created afterwards gets no trigger, and `CREATE SCHEMA`
succeeds in it.
4. A second run is a no-op.
5. A `template0` commented and given roles by hand is released the same
way.
Open PR #5091 touches the same script (stable database order); the hunks
do not overlap.
Backport: the automatic cherry-pick onto `release-1.6` conflicts in
`init-script.yaml` and the generated cozyrd, so the backport to
`release-1.6` is done by hand in #5157 (same SQL, replayed against that
branch's script). `release-1.7` keeps `kind/backport`.
### Screenshots
<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->
### Downstream repositories
<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.
Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.
IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:
1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->
- [x] No downstream repository is affected by this change (walked the
map: a template guard, the init script, a test suite and one field
description regenerated into README, schema, Go types and cozyrd; no
field, default, enum or name change)
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:
### Release note
<!-- Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->
```release-note
fix(postgres): databases now refuses the PostgreSQL system databases postgres, template0 and template1, and the init job never runs DROP DATABASE on them. A system database a previous version took over is handed back to the superuser on the next upgrade (event trigger removed, ownership and public schema restored, its <db>_admin and <db>_readonly roles dropped). A release that declares one of them must remove the entry before upgrading.
```
<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->
## Summary by CodeRabbit
* **Behavior Changes**
* PostgreSQL system databases (`postgres`, `template0`, and `template1`)
are refused as managed application databases.
* Previously managed system databases are released during cleanup and
excluded from database deletion. Cleanup restores eligible ownership and
security settings and clears system database comments.
* **Documentation**
* Updated PostgreSQL configuration guidance to clarify that these system
database names are refused. Names that merely contain these strings
remain accepted.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
|
||
|
|
a2eb175139 |
test(dashboard): start the pass-through nginx on fresh ports
The second nginx of the not-installed test reused the stand-in port of the first one right after killing it, behind a fixed sleep, so a slow exit could leave the port taken. It now waits for the first nginx to exit and takes a new port. The file header also names the fourth property it now checks. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
f2944c572d |
fix(dashboard): answer not-installed when the observability proxy is absent
The proxy can be left out of an installation, and then the console nginx answered the observability roots with its own 502. The console read that as an unreachable store: three errors, a Retry button and a live tail reconnecting forever, instead of a notice that the feature is not available on this cluster. nginx now turns the 502 and 504 it generates itself, when it cannot resolve or reach the proxy, into a 404 Kubernetes Status, which the console already treats as not installed. Upstream errors are not intercepted, so the proxy's own 502 and 504 for a store still reach the console unchanged. The bats test runs the extracted locations in a real nginx when one is installed. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
3e907e6f9a |
test(observability-proxy): declare the shell of the e2e helper library
The library is sourced by sh scripts and has no shebang, so shellcheck could not tell which shell to check it as and refused the file. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
f775948d2c |
test(observability-proxy): replay the early line into the e2e tail
The background tail goes through the console route and the proxy's reviews before it subscribes, so the early ingest can land first, and a tail without start_offset returns only lines written after it starts. The tail now replays the last minute, which still leaves the hold and the late-line check meaningful. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
72a94ee426 |
test(clickhouse): give the round-trip's preconditions room for their deletes
The step's ceiling SIGKILLs the shell and reports only "signal: killed", while a kubectl --timeout that fires first names the object that would not go. The deletes it runs now wait up to 270s in all, past the 3m ceiling, so a stuck delete late in the list would be reported by the ceiling instead of by kubectl. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0dc5a54731 |
test(clickhouse): remove the round-trip's TLS objects a killed run left
Chainsaw deletes an applied object only when its own apply created it; one left behind by a run killed past its cleanup is patched and kept. The Certificate would then outlive every later run and keep re-issuing its Secret after each step cleanup, and a probe Pod whose sleep had ended never turns Ready, failing the next run's assert. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
f003851338 |
fix(postgres): refuse system databases and never drop them
postgres, template0 and template1 were accepted in databases and handled like an application database. The init job made <db>_admin their owner, installed the auto-grant event trigger in them and, once the entry left values, ran DROP DATABASE on them. That DROP can only fail (template databases cannot be dropped, and the job is connected to postgres), so every later upgrade failed in the init job and the release stayed in UpgradeFailed even after the entry was put back. template1 also passed the takeover on to every database created from it afterwards, where the trigger, naming a role about to be dropped, makes CREATE SCHEMA fail. A release a previous version let take one of them over has to be repaired without dropping it, and without touching what this chart did not create: a role or owner that merely carries an expected name may belong to a tenant or an administrator, and a role comment can be set by any CREATEROLE holder, so only the cozystack:managed mark is trusted. What the marked roles own goes to cozystack:orphaned rather than to the superuser, since a tenant's SECURITY DEFINER function would otherwise run with superuser rights. A failed lookup of the databases cloned from template1 has to stop the run before template1 stops being selected, or those clones would never be repaired. Assisted-by: LLM Signed-off-by: Mattia Eleuteri <mattia@hidora.io> |
||
|
|
da7f2c2f50 |
fix(tests): keep two unit tests green after the release image restamp (#5180)
## What this PR does Same change as #5179, on `main`. Both tests pass here today, because `main` still carries the pre-release image pins; they fail on every release line as soon as the first rc is cut, since the `Prepare release` commit rewrites `packages/*/values.yaml` and two unit tests read exactly the fields it rewrites. On `release-1.7` this has kept `Unit & controller tests` red on every PR since v1.7.0-rc.3 ([job](https://github.com/cozystack/cozystack/actions/runs/41162009892/job/124421449910) on #5172), and `release-1.8` would repeat it. | test | reads | before the release build | after (release-1.7 today) | |---|---|---|---| | `hack/cni-plugins-staging-contract.bats` "the kube-ovn release the CNI plugin pin was checked against is the one vendored" | `.global.images.kubeovn.tag` in `packages/system/kubeovn/values.yaml` | `v1.15.26@sha256:…` | `v1.7.0@sha256:…` | | `packages/system/monitoring/tests/vtraces_collector_test.yaml` "pins the collector image through the cozy-lib.image helper" | `tracingCollector.image` | `docker.io/otel/opentelemetry-collector-contrib:0.136.0@sha256:…` | `ghcr.io/cozystack/cozystack/otelcol-tracing:v0.136.0-cozystack.0@sha256:…` | Changes: - The bats test takes the vendored kube-ovn release from `appVersion` in `packages/system/kubeovn/charts/kube-ovn/Chart.yaml`. That is the value `packages/system/kubeovn/Makefile` builds the image from, and a release build leaves it alone. The hint printed on a mismatch names the new file. - The helm unittest case no longer names the image at all. It sets `_cluster.images-registry` and asserts that the registry lands in front of the rendered image and that the image stays digest-pinned (the `grafana_upstream_image_plugins_test.yaml` pattern). Removing the `cozy-lib.image` call from the template makes it fail; the old regex did not. Same single commit as #5179; the only difference is the unrelated `DOC=` line next to the edit, kept as `main` has it. Verified locally on this branch: `bats hack/cni-plugins-staging-contract.bats` 10/10, `helm unittest packages/system/monitoring -f tests/vtraces_collector_test.yaml` 28/28. Refs #5178 (the release-1.7 report), #5179, #5172. ### Downstream repositories - [x] No downstream repository is affected by this change ### Release note ```release-note NONE ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **Tests** * Version validation now compares the checked version with the chart’s application version and reports missing or mismatched versions with the chart path. * Monitoring image checks now verify use of the configured mirror registry and require a 64-character lowercase SHA-256 digest, without requiring a specific image name. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
9e8e10c307 |
fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does Refs #5205. The S3 endpoint the platform backups use is copied twice, and neither copy is refreshed after `publishing.host` changes. The COSI sidecar writes `BucketInfo` once per `BucketAccess`, so the credentials Secret `bucket-cozy-backups-system-credentials` rendered from it keeps the old endpoint. The Velero BSL and the default Strategy CRs keep whatever endpoint the last render of `backupstrategy-controller` saw. The default objects gate only acted on missing objects, so backups kept going to the old host and nothing reported it. The gate now also forces a render when a copy is stale: - the bucket's `<bucket>-system` release, when the credentials Secret's endpoint differs from the COSI Secret it is rendered from; - `backupstrategy-controller`, when the endpoint an object was rendered with differs from the credentials Secret's. The chart stamps the rendered endpoint on the BSL and on every default Strategy as `backups.cozystack.io/endpoint`, because the endpoint itself sits at a different path in every kind. Objects without the annotation count as stale until a render stamps it; `MinForceInterval` bounds the forces. Stale credentials are forced before the BackupClass and the routed objects are read, so an absent BackupClass or an unreadable Strategy still fails the check but does not keep the old endpoint in the credentials Secret. A `BucketInfo` without an endpoint is not compared, since rendering from it would write an empty endpoint or fail the bucket release. Repairing missing objects keeps working while the credentials comparison is stale or fails, and a `systemSecretName` that is not derived from a COSI Secret (no `-credentials` suffix, or the raw COSI Secret itself) is not compared at all. RBAC is unchanged: the controller already holds cluster-wide `get` on Secrets. No new metric: nothing would consume it. What this does not do: the COSI Secret still holds the endpoint the `BucketAccess` was granted with, so after a host change the endpoint follows only once an operator deletes the `BucketAccess` and forces `bucket-cozy-backups` to recreate it (this issues new S3 keys). From there the gate replaces the remaining manual steps of the recovery described in the issue. Automatic re-grant is left for a separate issue, and the operator runbook goes to the website, hence `Refs` rather than `Fixes`. Known limitation: the credentials comparison runs only when `backupStorage.systemSecretName` is `bucket-<bucketName>-system-credentials`, the Secret the forced `bucket-<bucketName>-system` release renders. If an operator sets `bucketName` and `systemSecretName` inconsistently, the comparison is switched off rather than forcing a release that never rewrites the compared Secret, so a stale endpoint in that Secret is not repaired. Repairing a missing credentials Secret still forces the release derived from `bucketName`, as before this PR. Tested with unit tests for the gate (including the comparison normalisation shared with the chart helpers), helm-unittest for the annotation and the wiring, and a bats guard that every rendered Strategy CR and the BSL carry the annotation. On a live cluster, a server-side render of the chart produced exactly the annotation the gate derives from the real credentials Secret, so the steady state does not force anything. ### Screenshots Not a UI change. ### Downstream repositories - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: an operator runbook for changing `publishing.host` with platform backups enabled (re-grant the `BucketAccess`, then let the gate converge; restart ClickHouse pods that read the credentials from environment variables). Not filed yet, pending approval of the text. - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backup strategies and storage locations now record their storage endpoint, enabling detection and repair of outdated endpoint settings after bucket access changes. * Bucket credentials and backup objects are refreshed independently when their endpoints are missing or out of date. * **Documentation** * Clarified when endpoint changes trigger updates and how suspended releases are handled. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
8784863632 |
test(observability-proxy): wait for every seeded line before checking scope
The suite checked that B's line and the account-7 line are absent from A's answers but only waited for A's line to become searchable, so an absence could come from a line the store had not indexed yet. The step budgets in the comments now count the calls the steps make. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
ebb5a9cec4 |
test(observability-proxy): cover the namespace logs scope end to end
Unit tests mock the API server and the store, so nothing shows that the console's nginx, the proxy's TokenReview and SubjectAccessReview and a real VictoriaLogs agree. The suite drives a viewer of one tenant through the console against a seeded store: it reads its own namespace, is refused a sibling's, cannot widen the scope with client filters, an AccountID header or dot segments, and keeps a tail open past a minute. The root store is off in e2e, so the suite runs its own VictoriaLogs under the root store's Service name rather than waiting on Fluent Bit. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
d452b30e42 |
test(dashboard): name the templated nginx config in the buffer check
The console nginx config now ships as an entrypoint template, and a failure message naming the old conf.d path sends the reader to a file the Containerfile no longer writes. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
74a7a44423 |
feat(dashboard): route the observability proxy roots from the console
The logs and metrics roots under /apis/cozystack.io and /apis/apps.cozystack.io are served by the observability proxy, not by the API server, so the console nginx sends them there ahead of its /apis prefix location, unbuffered and with the same long timeouts, so that a live tail reaches the browser as it is written. nginx resolves a literal upstream once at startup and refuses to start when the name does not resolve, which would take the console down on every cluster that does not run the proxy. The upstream is therefore a variable resolved per request, and such a cluster answers 502 on those paths only. That resolver does not apply the pod's search domains, so the config becomes a template the image's entrypoint renders with the pod's nameservers and the platform cluster domain, which the chart now passes to the console. Assisted-by: LLM Signed-off-by: ZverGuy <maximbel2003@gmail.com> |
||
|
|
7097cc5cda |
feat(release): make backports automatic with backport-batch (#5106)
## What this PR does Backports become automatic. Fix merged to `main` with `fix` title, or `security` for fix that comes through security advisory, reaches release lines through batch that runs on Monday and Thursday, newest line first, nobody requests, labels or opens backports by hand. Batch runs for two newest lines with stable release (`released-lines` in policy block) and every line branched after them that has no stable release yet: with 1.6 and 1.7 released, `release-1.8` created at `v1.8.0-rc.1` joins them, and `v1.8.0` drops 1.6. Line counts as released once `vX.Y.Z` tag without pre-release suffix exists on origin. Rules are in `docs/release/backports.md`, `cmd/backport-batch` applies them, `.github/workflows/backport-batch.yaml` runs it. Design is in cozystack/community#114, merge shape of each original in cozystack/community#119. - `backport-batch run --line release-X.Y` takes PRs merged to `main` since line branched (title and body read from merge commit) and skips what rules exclude: other types and `deps`, breaking changes, platform migrations, `targetVersion`, changed CRD documents, API changes that API review gate calls sizeable (batch runs `internal/apigate` itself whenever PR changes a file the gate parses: `pkg/apiserver/apiserver.go`, an ApplicationDefinition, or YAML under `packages/` or `internal/` naming `CustomResourceDefinition` or `APIService`), and on older line what next newer line doesn't carry yet. The rest goes onto `backport-batch/<line>-<date>` in merge order. PR merged to `main` with merge commit goes as its own commits picked with `cherry-pick -x --signoff`, so each keeps its author, message and sign-offs and gets sign-off of whoever runs the batch (with `-m 1` author was whoever merged on `main`), merges of `main` into PR branch are left out, and merge commit joins them to the batch with original's merge message and `Backported-from:` trailer. Squash-merged PR goes as one pick. Conflict on any commit takes branch back to where it was before that original, it's listed with the commit that failed, run goes on. `--push` opens or updates draft PR, `--continue` carries on open batch PR as fast-forward only, PR body it didn't write is never rewritten. Same-day run over batch branch whose PR never got opened refuses and names `run --line <line> --onto <head> --push`, which continues it. Run or `--defer` that resets local batch branch over commits only it has prints `git branch` command that restores them. - `run --defer <N> --reason <why>` takes original out of open batch (usually the one failing CI is pinned on). It rebuilds branch replaying batch unit by unit without it, so merges and pushed conflict resolutions keep their shape, and only this rebuild pushes with `--force-with-lease`. Deferred original comes back in next batch. - Line carries something only if its own history shows it: picks whose `-x` line, the one git appends, names PR's merge commit or every commit of its own, or pick that changes nothing. Picked commits keep their messages and batch merge carries PR body, so text in them (`-x` lines, `Backported-from:`, `Backport of #N`) counts only on commits from before `batch-only` day in policy block, where it records old hand backports and batches. Otherwise text in a merged fix could mark another fix as already backported, and it would never reach the line. PR against line never counts. Open batch PR counts only from this repo and from author with write access or the batch's own bot (`BACKPORT_BATCH_BOT`, exact login, `cozystack-ci[bot]` when unset, so maintainer's run by hand continues scheduled batch instead of opening second one), and is read the same way. Actions bot isn't trusted anymore, any workflow token writes as it. - Any failed GitHub API or git call stops the run with message naming the call, no retries. Failed permission lookup used to count batch PR out as look-alike, so `--continue` opened second batch PR and `notify` posted second comment; now `run`, `check`, `verify` and `notify` exit 2 and write nothing. - Policy applies fully from `release-1.7`. `release-1.6` takes only fixes merged from 2026-10-26, older ones need `backport/include`. Start days are in `backport-policy` block on the policy page, test keeps page and code in sync. - `backport-batch check --cut` replaces `backport-audit` as gate before patch cut and before every rc after the first, it also fails while a fix is only on open batch PR. Plain `check`, which scheduled run uses, lets those through. `backport-batch fold` checks conflict resolution in the same shape: merge on batch head whose second parent is straight line of picks of PR's own commits in order, each with its `-x`, only PR's files (one pick for squash). `backport-batch verify` holds every original on batch to that shape, clean picks included, and `pr-labeler.yaml` runs it on every push to batch PR: `pull_request_target` so it reaches `release-*`, tool built from `main`, batch head fetched as data, read-only token. Batch PR body says per original how many commits it brought and which picks differ from their commits. `backport-batch notify` reports batch that is still open from previous run and deferred conflicts in one comment on batch PR, edited in place, with `BACKPORT_ALERT_TEAM` mention. - `hack/backport-batch-scheduled.sh` is what the workflow runs and maintainer can run by hand (`--dry-run` changes nothing on GitHub): for each line newest first `run --continue --push`, `check`, `notify`. It pushes with `BACKPORT_BATCH_TOKEN` only and fails without it, never `GITHUB_TOKEN`, because its pushes don't start CI. Workflow runs it as cozystack-ci App: it mints token for this repo only with Contents and Pull requests write, passes it as `BACKPORT_BATCH_TOKEN`, commits as `cozystack-ci[bot]`, and sets `BACKPORT_BATCH_BOT` so the tool trusts that bot's batch PR and comment. Job timeout is 30 minutes, under the token's hour. - App has no workflows permission, so workflow runs the script with `--no-workflow-files`: original that changes `.github/workflows/` isn't picked and waits on batch PR for maintainer, listed with its workflow files in body and in notify comment. It's never a conflict. `check` warns about it while it waits and `check --cut` fails on it. Maintainer brings it over with own token by `run --continue --push` without the flag, or takes it out with `--defer`. - Deferred conflicts and failed batch CI wait for maintainer. `notify` keeps one comment on batch PR listing stalled batch, conflicts and workflow changes, mentioning `BACKPORT_ALERT_TEAM` (team or login) when it's set. Failed batch CI isn't in the comment, it shows on batch PR checks, and `check --cut` fails while batch is unmerged. - Two steps that need judgement (resolve conflict, pin failed CI on original and defer it) are written as runbooks in `docs/release/backport-tasks/`: inputs, the only allowed output, check it must pass before push, what never to do. - `backport/include` and `backport/skip` let maintainer override rules for single PR, they are new `backport/` category in `labels.yml`. Skip wins over include, if open batch already picked it `--defer` takes it out. - Text written by other people (titles, release notes, reasons, file names) goes to batch PR body only inside code spans or release-note block, control, bidi and zero-width characters are dropped, file names in deferred block included, git args go after `--end-of-options`, no shell. - Removed: `cmd/backport-audit`, `backport.yaml`, `kind/backport` and `kind/backport-previous` labels with aliases and stale exemption, trailer-check exemption for bot branches. `Commit trailers` job exempts only `backport-batch/*` branches of this repo, changelog instructions read originals from batch commits. Exemption lives in main's `pre-commit.yml`, and batch PR runs its line's copy: `release-1.7` has its own copy that exempts only old `backport-<pr>-to-<branch>` branches, `release-1.6` has no such job. So on `release-1.7` picked commit with bad trailer fails batch CI until line gets main's copy. - `hack/close-manual-backport.sh` runs from `pr-labeler.yaml` on `opened` and closes PR against `release-*` from outside contributor (association contributor, first-time contributor or none, permission read or none). Owners, members, collaborators, anyone with triage or more and bots are left alone. When GitHub's answer on author's permission can't be read, it closes nothing and fails the job naming the failed read. Maintainer keeps release-only fix by reopening it. Release-only PRs are occasional and opened by maintainers, for fix with no counterpart on `main`, or one that on `main` came inside another PR or was written for `main`; if its counterpart on `main` is a fix the batch would take, the batch skips it once picking it changes nothing on the line, otherwise maintainer labels it `backport/skip`. - `docs/release.md` is split into pages under `docs/release/`, PR template doesn't ask for `kind/backport` anymore. `security` type is added to commit format in `docs/agents/contributing.md`, PR template, Gemini review guide and labeler's list (labeler maps no `kind/*` to it, `labels.yml` has no fitting one). - `SECURITY.md`, `docs/security/incident-response.md` and `GENERAL_TECHNICAL_REVIEW.md` now say two latest minor releases are supported, same window the batch runs for, and link the backport policy for which fixes go there. Dry runs without push, on 2026-11-06, with `--no-workflow-files` as the workflow runs it: `lines` gives `release-1.7` and `release-1.6`. On `release-1.7` batch picks 12 fixes, defers 3 conflicts (#4756, #4867, #5082), leaves #4888 and #4971 to a maintainer because they change workflows, and skips 49. On `release-1.6` it picks #5088, defers 3 conflicts (#5071, #5086, #5087), leaves #5142 to a maintainer, and skips 590. `verify` passes both batches. Setup after merge: no new secret. Batch runs as cozystack-ci App from existing `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY`, job fails when either is missing. App's installation already has Contents and Pull requests write on all org repos, and no Workflows is needed: PRs that change `.github/workflows/` wait for maintainer, who brings them over with own token as "Changes to workflows" in `docs/release/backports.md` says. Today that's #4888 and #4971 on `release-1.7` and #5142 on `release-1.6`. Set `BACKPORT_ALERT_TEAM` to a team or a login (for example `cozystack/maintainers`): unset, batch comment mentions nobody and nobody is notified. Once this PR is merged, `Backport batch shape` can be required check in the release lines ruleset (`release-*` excluding `release-*.*.*`), with "Do not require status checks on creation" on, because `cut-prerelease.yaml` creates `release-X.Y` by push. `PR Auto-Label` with both new jobs runs on `pull_request_target`, so it needs #5135 sorted before 2026-12-02. `kind/backport*` labels have to be deleted by hand, label sync doesn't delete. ### Screenshots Not a UI change. ### Downstream repositories cozystack/community `contributors/README.md` lists commit types, so it needs `security` too, it has no backport label text. Website has no backport label instructions, and its links to `docs/release.md` carry no anchor, the page stays as index. Trigger map lists commit convention change for cozystack/ccp: this adds `security` type, but not message format or trailer, and `package-bump` writes only `chore(...)` commits and restates the trailer, so nothing changes there. - [ ] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: add `security` to commit types in `contributors/README.md` ### Release note ```release-note feat(release): backports are automatic. Fixes merged to main with fix or security title reach release lines through a batch built by cmd/backport-batch every Monday and Thursday, by the rules in docs/release/backports.md. The batch runs for the two newest lines with a stable release and for a line in its rc window: the release-X.Y branch the first rc creates joins at once, and its vX.Y.0 drops the oldest line. Features, chores, docs, dependency bumps, breaking changes, migrations, CRDs and API changes that the API review gate calls sizeable wait for the next minor. The policy applies fully from release-1.7, release-1.6 takes only fixes merged from 2026-10-26. Each original reaches the line as its own commits under a merge commit, keeping their authors and sign-offs. Maintainers override the rules for single PR with backport/include and backport/skip labels and take an original out of a batch with backport-batch run --defer. backport-batch check replaces backport-audit as the gate before a patch release and before a later rc. The label-driven backport workflow and kind/backport labels are removed. Backport PR opened by hand against a release branch by author with read access or less is closed automatically. The scheduled batch runs as the cozystack-ci App and needs no secret of its own; a fix that changes .github/workflows waits on the batch pull request for a maintainer, since the App may not push workflow changes. Release documentation now lives in one page per topic under docs/release/. ``` |
||
|
|
e92bc8664b |
fix(tests): keep two unit tests green after the release image restamp
The release build rewrites packages/*/values.yaml: the kube-ovn image tag becomes the Cozystack version and the traces collector becomes the first-party otelcol-tracing build. Two unit tests read exactly those fields, so every PR against release-1.7 has failed Unit & controller tests since the first rc, and release-1.8 would repeat it. Same change as #5179 on release-1.7. Assisted-by: LLM Signed-off-by: Marian Koreniuk <moriarti@cp.if.ua> |
||
|
|
4345a346cc |
test(clickhouse): check why the round-trip restore failed, not only that it did
The refused-delete and failed-lookup cases asserted a non-zero status alone, which any earlier failure in the restore also produces, so a stub drifting from the script would have kept them green for the wrong reason. They now also require the delete to have been attempted and the lookup failure to be named. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
d20b26d2ed |
test(clickhouse): pin that a refused Secret delete fails the round-trip restore
The owned Ingress Secret's delete only became reachable once its lookup stopped asking for a name and a selector together, and nothing failed the stub's delete, so dropping the delete's status from the restore's result passed every test. The CoreDNS half of the restore already has this guarantee pinned. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
28f3613450 |
test(clickhouse): delete the round-trip's TLS Secret in its own step's cleanup
Chainsaw applied the Certificate, so the scripted delete of it in the run step's finally was hand-rolled teardown of a resource Chainsaw already reclaims. Chainsaw deletes a step's applied resources, waiting for them to go, before it runs that step's cleanup operations, which puts the Secret's delete after the Certificate's without any ordering written into a script. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6e23bd5629 |
test(clickhouse): delete the round-trip's Certificate before its Secret
cert-manager issues the Secret of a Certificate that still exists again as soon as it is deleted, and the Certificate is only removed by Chainsaw's own cleanup after the step's finally has run. Deleting the Secret first therefore left a freshly issued copy behind in tenant-root, carrying no owner reference to take it down with the Certificate. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
1003cdb1b6 |
test(clickhouse): find the round-trip's Ingress Secret by its owner label alone
kubectl refuses a resource name together with a label selector, and the restore asked for both with its stderr discarded, so the lookup always came back empty and the Secret the Test serves under the platform's S3 Ingress name was never deleted. The next run then stops on its own precondition that the Secret must not exist. A lookup that does fail now fails the restore, since an empty answer from a failed read is what kept this hidden. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
ee5e51ee6c |
fix(backups): keep forcing objects that still lack the endpoint annotation
An object without backups.cozystack.io/endpoint stopped counting as stale as soon as one forced render had been requested, whether or not that render succeeded. A failed upgrade left the objects unannotated and the gate silent until the controller restarted, which also hid the climbing force counter meant to report exactly that failure. Such an object now stays stale until a render stamps it; MinForceInterval already bounds the cost. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
6280d06d01 |
test(clickhouse): describe the round-trip's guards by the archive-recording restore
Restores now take the archive the Backup recorded rather than the newest one, and a leftover Backup of the BackupJob's name that records another archive fails the new BackupJob instead of being adopted. The comments justifying the sentinel row and the Backup cleanup, in the round-trip and in its unit test, still gave the earlier reasons, which would send the next reader after behaviour the driver no longer has. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
86c541368d |
Merge branch 'main' into fix/clickhouse-backup-late-credentials
Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
eaa0169ff3 |
chore(release): retire the backport bot, its labels and backport-audit
The batch replaces the label-driven backport workflow and the audit that gated a cut against it. The kind/backport labels, the stale exemption for them, their contract tests and the audit's make target go with them. labels.yml gains a backport/ category instead, with backport/include and backport/skip, the overrides the batch reads off a merged pull request. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
4a750a1e23 |
ci(release): run the backport batch twice a week
The batch runs on Monday and Thursday from a script a maintainer can also run by hand. It runs as the cozystack-ci App with a token for this repository alone and Contents and Pull requests write, so its pushes start CI on the batch and no person's token is involved. The App may not push a workflow change, so such an original waits for a maintainer. pr-labeler holds every batch pull request to the batch's shape, with the tool built from main and the head read as data. It also closes a pull request into a release line opened by someone with read access or none, since fixes reach the lines through the batch. The Commit trailers check exempts a batch branch of this repository, whose picked messages cannot be rewritten. Promotion now says how to bring its own pipeline to a line through the batch. Assisted-By: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
970191dc8a |
fix(clickhouse): restore the archive a Backup was taken as (#5208)
## What this PR does A ClickHouse `RestoreJob` restored the newest archive under the app prefix, whichever `Backup` it named. Strategy pod generated the archive name with `date -u` and nothing was recorded on the `Backup`, so restore could only list the remote archives and take the last one. Restoring an older `Backup` brought back the latest data. Now the controller names the archive before it creates the backup Job (`clickhouse-<app>-<startedAt>` from persisted `status.startedAt`, so retried pods write the same one), passes it to the pod as `.ArchiveName` and records it in the `Backup` `driverMetadata`. Restore takes exactly that archive and fails with its name when storage has no such archive. The name is recorded only when the Job's pod got it, so a Job left from the previous controller during an upgrade, or a custom strategy with its own names, leaves no name instead of a wrong one. A retried pod reuses the name, so it waits for an attempt the sidecar still runs (no 423) and asks for `resume`. A `Backup` outlives its `BackupJob` and a Plan names `BackupJob`s after the schedule slot, so a `BackupJob` created again under a used name met the earlier run's `Backup` when its Job completed, took it as its result and left the archive it just wrote recorded nowhere. Now it fails if that `Backup` records another archive, and the message names both. A retry of the same run still gets its own `Backup` back. Old `Backup`s have no recorded name and stay restorable: the pod logs a `WARNING` and takes the newest archive stamped no later than `takenAt`, and fails when nothing qualifies. While the chart rolls out, previous controller pod can still render the new template and pass no `takenAt`, so a restore of a `Backup` without recorded name fails before it touches any table, and works when created again after the rollout. Fixes https://github.com/cozystack/cozystack/issues/5199 Tests are go unit tests for naming, recording, a reused `BackupJob` name and restore render, and `hack/altinity-backup-strategy.bats`, which runs the shipped script against a stubbed clickhouse-backup API (the restore tests go red on the old template). Checked on a live ClickHouse with two backups of 1500 and 2000 rows: restore of the older one by recorded name ends with 1500 rows, fallback for an unnamed older backup picks the same archive with the warning, a missing archive fails without touching the table, a retry over an existing local backup completes under the same name. The controller binary itself was not run on the cluster, only its rendered output in the pod. ### Downstream repositories - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(clickhouse): a RestoreJob now restores the archive its Backup was taken as, not the newest one. New Backups record the archive name, Backups taken before this release are restored best effort from the newest archive stamped no later than their takenAt, with a warning in the restore pod log. A ClickHouse BackupJob created again under a name whose Backup still exists now fails instead of reporting that Backup as its result. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Backups now use a consistent archive name tied to the backup job, and completed backups retain that name for future restores. * Restores can select the recorded archive and backup timestamp. If no archive was recorded, the system selects the newest eligible archive for the application. * **Bug Fixes** * A restore now fails when its recorded archive is missing instead of silently selecting a different one. * Backup requests avoid starting a duplicate action when the matching action is already in progress. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
603114cd7c |
test(clickhouse): check the round-trip restored this run's sentinel row
Every run inserted the same sentinel row and accepted any row after the restore, while the Altinity restore takes the newest remote archive of the release. A restore of an older run's archive therefore passed. The row now carries a per-run value and exactly that row has to come back. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
dee5e6d8bf |
test(clickhouse): mark the round-trip's reachability poll as a fixed wait
The probe loop sleeps between attempts because no Kubernetes condition reports that the host CoreDNS serves the rewrite and the Ingress serves the Test certificate. The e2e conventions allow such a sleep only under the marker that records it is waiting for something to model it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cabe9a07e2 |
test(clickhouse): reclaim the round-trip Backup by its name
The Altinity driver names the Backup after the BackupJob and adopts one that already exists under that name. The round-trip's cleanup deleted the Backup only through the BackupJob's status.backupRef, so a Backup whose BackupJob never recorded it survived, and the next run's BackupJob would have adopted it in place of a fresh backup. Both the cleanup and the preconditions now delete it by name; the preconditions ceiling grows to cover the extra wait. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
a662198991 |
test(clickhouse): restore the round-trip's shared objects after a step ceiling
The round-trip changes the host CoreDNS Corefile and the S3 Ingress TLS Secret and puts both back from its EXIT trap. When the run step's ceiling fires, Chainsaw kills the shell and the trap never runs, so the rewrite stays in a cluster-wide object and the next run refuses to start. The step's finally runs regardless, so it repeats the restore; after a run whose trap already restored them it finds nothing to undo. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
003bc3ed0b |
test(clickhouse): fail the round-trip when CoreDNS cannot be restored
The round-trip puts the host CoreDNS Corefile back from its EXIT trap, which runs with errexit off. A refused kubectl replace, for instance a resourceVersion conflict with another writer, fell through to the success message and returned 0, so the run stayed green while the rewrite remained in a cluster-wide object and the next run refused to start. The restore now returns the failure. The restore moves into a file of its own so a unit test can run it against a stubbed kubectl and so other callers can share it. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
0ad642e68d |
test(clickhouse): size the system-bucket round-trip server for SQL
The round-trip ClickHouse ran on the nano preset, whose 128Mi limit the clickhouse-1 contract suite never exceeds because it sends no query. This suite writes a sentinel row before the backup, and the server was OOM-killed on that first statement (lastState OOMKilled, exit 137), so the Test failed before it reached the behaviour under test. clickhouse-2 runs the same statements on the small preset. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e5bb3cda07 |
fix(kubernetes): move the OIDC users out of the bootstrap Job's script (#5213)
## What this PR does Found while checking for other places with the failure from https://github.com/cozystack/cozystack/issues/4968: data that grows with configuration is passed to a process as one argument, and Linux allows 128 KiB for that. This one is in the `kubernetes` app and there is no issue about it yet. The OIDC bootstrap Job wrote a ClusterRoleBinding for every `oidc.users` entry into its `sh -c` script, 526 bytes per user. I rendered current main with `helm template` and users like `user123@example.org`: 238 users give a 130,889 bytes script that starts, 239 give 131,415 bytes and `exec` fails with E2BIG. Then the container can't start, the post-install and post-upgrade hook fails after its retries and the HelmRelease of the tenant cluster stays unready. Schema has no limit for `oidc.users` and each person needs own entry, so a cluster shared by 240 people gets there. Now the bindings and the list of their names are rendered by the new `oidc-rbac-bindings.yaml` into a ConfigMap `<release>-oidc-bindings` that the Job mounts, so the script is 6,380 bytes for any number of users. The ConfigMap has its own 1 MiB limit, about 2,500 users at 417 bytes each. Names, labels and roles of the bindings stay the same. I compared the render of old and new chart for eight emails (ones YAML reads as a boolean or null, a number, a quote, an apostrophe, unicode and two plain) and the bindings are identical, so existing ones are updated in place and nothing is orphaned. The prune step reads the same ConfigMap. If the list is not mounted the Job stops before it deletes anything, because `grep` on a missing file exits 2, `! grep` is true and every binding of the release would be pruned. A release with no users applies nothing (`kubectl apply` fails on an empty file) and prunes all stale bindings, same as before. `hack/kubernetes-oidc-bindings.bats` renders 300 users, checks the script is under the limit without any email in it and that every user has one binding with the old name and role. Three more tests run the rendered script against a fake `kubectl` and check what it applies and deletes. The prune condition forced to true or to false passes all helm-unittest cases, so this part is tested in bats. The bug is in release-1.6 and release-1.7 as well. ### Downstream repositories I walked the trigger map against the diff (one chart template, its tests and a bats file): no package added or removed, no values, schema or default changed, no output Secret or Service renamed. - [x] No downstream repository is affected by this change - [ ] [cozystack/website](https://github.com/cozystack/website) - follow-up: - [ ] [cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack) - follow-up: - [ ] [cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack) - follow-up: - [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up: - [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up: - [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) - follow-up: - [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) - follow-up: - [ ] [cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server) - follow-up: - [ ] [cozystack/external-apps-example](https://github.com/cozystack/external-apps-example) - follow-up: - [ ] [cozystack/examples](https://github.com/cozystack/examples) - follow-up: - [ ] [cozystack/community](https://github.com/cozystack/community) - follow-up: ### Release note ```release-note fix(kubernetes): the OIDC bootstrap Job no longer fails to start for a tenant cluster with about 240 or more oidc.users, the bindings now come from a ConfigMap the Job mounts. ``` <!-- This is an auto-generated comment: release notes by coderabbit.ai --> <!-- coderabbit-ai-generated:v1 --> ## Summary by CodeRabbit * **New Features** * Kubernetes OIDC configuration now supports per-user role bindings, with administrator and viewer access assigned according to each user’s role. * OIDC role bindings are kept in sync with the configured user list, including removal of obsolete bindings. * Deployments with no OIDC users or with OIDC mode disabled continue to avoid creating user bindings. <!-- end of auto-generated comment: release notes by coderabbit.ai --> |
||
|
|
d5559bb934 |
test(backups): remove the Altinity strategy tests' scratch directories
Every test in hack/altinity-backup-strategy.bats made a directory with mktemp -d and never removed it, so each passing run of the file left nine directories in TMPDIR, and they pile up on any machine that keeps it between runs. Each test now removes its directory as its last statement. A failed assertion aborts the body before that line, so a failing test keeps its stub log and rendered script for inspection. Assisted-by: LLM Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com> |
||
|
|
dbcc2af679 |
test(clickhouse): require the system-bucket round-trip to run on the sidecar that started without credentials
A Pod restart during the round-trip would hand the sidecar fresh credentials at start, which is how the old environment-based sidecar got them, and the Test would pass without the running sidecar ever picking them up. The Pod uid and the sidecar's restart count must stay the same across both jobs. The probe deadline is now measured on the clock: counting only its pauses let a hanging exec stretch it past the step budget, so the step timeout would fire before the probe could say what stalled. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
2e37aaec5f |
docs(backups): drop the claim that a region change needs a ClickHouse restart
The ClickHouse sidecar on the system bucket no longer reads S3 settings from environment set at start, so a re-projected region reaches it without a Pod rollout, and the contracts fixture no longer depends on optional Secret keys to start its Pod. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
e752952047 |
test(clickhouse): back up and restore a running release through the system bucket
A ClickHouse release on the system bucket starts before the platform projects cozy-backups-creds into its namespace, and no suite exercised that order: the round-trips bring their own Bucket and endpoint, so the sidecar never had to pick up credentials that arrived after it started. The new Test starts the release without the projected Secret and runs a backup and an in-place restore through cozy-default against the bucket as advertised. In CI that host neither resolves nor has a certificate, so the step makes it reachable from tenant-root for its own duration and proves the served certificate before any job starts, which keeps a DNS or TLS problem from reading as a backup failure. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |
||
|
|
cd91138f33 |
fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither copy is refreshed. The COSI sidecar writes BucketInfo once per BucketAccess, so the credentials Secret rendered from it keeps its endpoint, and the Velero BSL and the default Strategy CRs keep whatever endpoint the last render of backupstrategy-controller saw. The default objects gate only acted on objects that were missing, so after a publishing.host change every backup kept going to the old host with nothing reporting it. The gate now also forces a render when a copy is stale: the bucket's <bucket>-system release when the credentials Secret's endpoint differs from the COSI Secret it is rendered from, and backupstrategy-controller when the endpoint an object was rendered with differs from the credentials Secret's. That endpoint sits at a different path in every kind, in some only as a bare host, so the chart stamps the rendered value on each object as backups.cozystack.io/endpoint and the gate compares that. Objects rendered before the annotation existed count as stale until one forced render has gone through, which stamps it. The COSI Secret itself still holds the endpoint the BucketAccess was granted with; it follows a host change only once the BucketAccess is re-granted. Assisted-by: LLM Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io> |