Commit Graph
213 Commits
Author SHA1 Message Date
Maxim Belyy 34843c1a91 Merge branch 'main' into feat/observability-proxy-metrics
Assisted-by: LLM
Signed-off-by: Maxim Belyy <maximbel2003@gmail.com>
2026-11-13 16:07:33 +07:00
myasnikovdaniil 0f169540a1 feat(release): withhold partial backports and park those that build on code their line lacks (#5269)
## What this PR does

Stacked on #5264, that one merges first.

Clean cherry-pick doesn't mean the original landed. On release-1.6 the
batch carried a fix whose two deletions hit files the line never had, so
only its test arrived. Other originals conflicted on every run because
they change code that never reaches the line. Batch now tells these
cases apart instead of picking half a fix or retrying a conflict nobody
can resolve.

- Every file an original modifies or deletes is checked at its
counterpart on the line, renames followed on both sides. If part of the
original has no effect there (change to a file the line lacks, deletion
of a file it never had, deleted file the unit keeps), the original is
parked as partial and its release note stays out. A file that already
matches what the original makes counts as there (same blob, binary too,
or clean `git merge-file`). This runs in the picker, `fold`, `verify`
and over units already on the open batch, so such a unit fails `check
--cut` whatever its label until `run --defer` takes it out.
- Pick that differs from its original keeps its release note under
"Check these notes" with a `git range-diff` command to compare, record
and report mark it `changed`.
- Run names what an original builds on that the line lacks: the commit
on `main` that added a file the fix modifies or deletes, and blame of
removed lines whose text is still missing on the line. Generated files
and generated keys of ApplicationDefinitions are skipped. Lines a fix
only adds are not checked, a dependency that lives only there still
needs `--drop`.
- If a prerequisite never reaches the line (`feat`, direct push,
excluded or itself parked) the original is parked and not tried. If it
is still to come, the original waits for it and is retried later.
Prerequisites the line or open batch already carry, or this run picked,
count as there. `run`, `check` and record-only commands decide in the
same order. On scheduled run a workflow original whose only missing
dependency is in tests or docs waits for a maintainer instead of
parking. `backport/include` lifts neither park, wait nor partial.
- `notify` announces each park once on the open batch and marks it in
the record, nothing is announced while no batch is open.

Record moves to version 3 and report to format 2, format 1 reports still
read. I ran the new tool next to the previous one without `--push` on
today's history. On release-1.7 nothing changes category and `check`
prints the same. On release-1.6 #5071 is parked on the two features it
builds on, #5088 already on the open batch now fails the cut as a
partial unit, and #5142 on scheduled run still waits for a maintainer
with a note that only its tests depend on what 1.6 lacks. Rule 6 holds
twelve originals on 1.6 right now, run over them directly flags the
seven that change code 1.6 never got and none of the other five.
Detection adds under 2 s per line.

After this merges, CI on the open 1.6 batch goes red at its next push
since `verify` refuses #5088 there. `run --line release-1.6 --defer 5088
--reason ... --push` takes it out.

### Downstream repositories

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
feat(release): backport batch holds back a pick that applies only part of its original and fails the cut on such unit already on a batch, sets apart the note of a changed pick, parks an original that builds on code its line never gets, waits for one whose prerequisite is still to come, and announces each park once.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backport batches now identify partial picks and missing prerequisites,
and can park affected originals or leave them waiting.
* Batch reports and notifications provide details about parked, partial,
changed, and waiting picks.
* Release notes for picks that differ from their originals are set apart
for review.
* **Bug Fixes**
* Verification, folding, and batch checks reject picks that omit effects
from the original; renamed paths are supported.
* **Documentation**
* Updated backport guidance to explain parking, prerequisites, partial
picks, and release-note checks.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-13 13:14:12 +05:00
Aleksei Sviridkin 0b284a65d5 test(release): isolate report deduplication from workflow attempts (#5282)
<!-- Thank you for making a contribution! Here are some tips for you:
- Use Conventional Commits for the PR title: `type(scope): description`
- Types: feat, fix, security (a fix that comes through a security
advisory), docs, style, refactor, perf, test, build, ci, chore
- Scopes are not an exhaustive list — pick the most specific scope for
the change and extend the list when a genuinely new area appears.
Examples:
- System components: dashboard, platform, operator, cilium, kube-ovn,
linstor, fluxcd, cluster-api
- Managed apps: postgres, mariadb, redis, kafka, clickhouse,
virtual-machine, kubernetes
- Development and maintenance: api, hack, tests, ci, docs, maintenance
- Breaking changes: append `!` after type/scope (`feat(api)!: ...`) or
add a `BREAKING CHANGE:` footer
- If it's a work in progress, consider creating this PR as a draft.
- Don't hesistate to ask for opinion and review in the community chats,
even if it's still a draft.
- Fixes merged to `main` reach the release lines the backport batch runs
for automatically, so do not open backport PRs by hand:
https://github.com/cozystack/cozystack/blob/main/docs/release/backports.md
-->

## What this PR does

Pins the report deduplication test to the first workflow attempt. In a
CI rerun, the inherited attempt otherwise adds `/attempts/2` to the
synthetic run URL and fails the existing exact-URL assertion. Production
run URLs and the deduplication assertions stay unchanged.

Validation: a source-derived check of the test setup and original URL
assertion fails with an inherited second attempt and passes with the
proposed first-attempt setup. Controls with the pin after preview or set
to 2 still fail. The full Git-backed test was not run locally; CI must
verify it.

### Screenshots

Not applicable; no UI changes.

<!-- REQUIRED for UI changes: attach screenshots or screen recordings
demonstrating
the visual impact of your changes. PRs with UI changes without
screenshots will not be merged. -->

### Downstream repositories

<!--
Cozystack is upstream for repositories that are NOT kept in sync with it
automatically, and nothing in CI compares the two sides, so a change
here can break them silently.

Which change forces what, and which file to touch, is the trigger map in
docs/agents/contributing.md under "Downstream Repositories". Walk it
before you tick anything.

IF YOU ARE AN AI AGENT OR AN LLM, THIS IS PART OF THE TASK, NOT A FORM
TO FILL IN:

1. Walk the trigger map against the actual diff of this PR, file by
file. Do not decide from the PR title or from your memory of what you
changed.
2. For every repository you tick, open the follow-up PR there and link
it on the same line. A ticked box with no link claims work that does not
exist, which is worse than an empty box. Search that repository for an
open PR or issue covering it first, and link that instead of filing a
duplicate.
3. When the follow-up is out of scope, or needs a decision that is not
yours to make, open an issue in that repository instead, link that, and
say so in the PR body. Do not push a speculative PR into another
repository just to fill a line here.
4. Do not tick "No downstream repository is affected" to make the
checklist go away. If you are unsure, leave every box empty and say so
in the PR body, so a human decides.
-->

- [x] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

<!--  Write a release note:
- Explain what has changed internally and for users.
- Start with the same `type(scope):` prefix as in the PR title
- Follow the guidelines at
https://github.com/kubernetes/community/blob/master/contributors/guide/release-notes.md.
-->

```release-note
test(release): make the report deduplication test independent of the surrounding workflow attempt.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->

<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **Tests**
* Updated automated test configuration to account for GitHub Actions run
attempts. No user-facing behavior changes.

<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-12 13:01:17 +03:00
Aleksei Sviridkin 9978771858 feat(observability-proxy): serve tenant namespace logs through an authorizing proxy (#5214)
## What this PR does

Adds `observability-proxy`, a new system component that lets a tenant
user read the logs of their own namespace from VictoriaLogs with the
same Kubernetes credentials they already use for the console, and with
Kubernetes RBAC deciding what they can read. This is the first part of
the design proposals [Tenant application
logs](https://github.com/cozystack/community/pull/95) and [Tenant
resource-consumption
metrics](https://github.com/cozystack/community/pull/94); this PR
implements the namespace logs root only. Metrics and the per-application
roots follow in separate PRs.

Why a proxy: today the log stores are shared per root tenant and have no
notion of a caller, so tenants cannot be given direct read access
without seeing other tenants' logs. The proxy authenticates the caller,
authorizes the request from its path, and rebuilds the upstream query so
that its scope cannot be widened by the client.

Backward compatibility: this adds a new package and a new HelmRelease;
nothing existing is renamed, removed or changed, so no migration is
needed.

What is included:

- `cmd/observability-proxy` and `internal/observabilityproxy`: the
proxy.
- Serves
`/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/select/logsql/<endpoint>`
for the VictoriaLogs read endpoints the console needs (`query`, `hits`,
`stats_query_range`, `tail`, `field_values`, `stream_field_values`,
`facets`). Anything else under the root answers `404`.
- Authentication by `TokenReview`, authorization by one
`SubjectAccessReview` built from the path (`list` on `cozystack.io/logs`
in the namespace), both cached briefly.
- The upstream request is rebuilt from an allowlist of parameters; the
namespace scope is always forced as `extra_stream_filters`.
Client-supplied `extra_filters`, `extra_stream_filters`, any LogsQL
`options(...)` block, and tenant headers such as `AccountID`/`ProjectID`
are not forwarded or are refused.
- Non-canonical paths (`..`, `//`, encoded separators) are refused with
`400` before any authorization is built, because the console nginx
matches on the normalized path but forwards the raw one.
- Time window, step, bucket count and result limits are clamped;
per-user rate, concurrency and open-tail limits answer `429`. Invalid
time, step and limit parameters answer `400`.
- The store is resolved from the namespace's
`namespace.cozystack.io/monitoring` label.
- Live `tail` streams are passed through unbuffered, and the upstream
tail request disables compression so headers reach the client before the
first line.
- `packages/system/observability-proxy`: chart and image build, wired
into the root `make build`.
- Platform wiring: a new `cozystack.observability-proxy` PackageSource,
enabled by default on the full variants and opt-in on the slim ones, the
same way as `monitoring-agents`.
- RBAC: an aggregated `cozy:observability:view` ClusterRole grants
`list` on `cozystack.io/logs` to tenant `view` and above. No API server
serves this resource; the rule exists only for the proxy's
`SubjectAccessReview`. The proxy's own ClusterRole is limited to the
calls it makes (`tokenreviews`, `subjectaccessreviews`, `list`/`watch`
on namespaces).
- Console nginx: when the proxy is not installed or cannot be reached,
the observability roots answer `404` with a Kubernetes `Status` body,
which the console shows as not available on this cluster; the proxy's
own `502`/`504` for an unreachable store pass through unchanged.
- Console: one regex `location` in the console nginx routes the four
observability roots to the proxy ahead of `/apis`. The upstream is
resolved at request time, so the console starts even when the proxy is
not installed; those paths then answer `404` with a Kubernetes `Status`
body. The console Deployment now receives the cluster domain for that
resolution.
- E2E: a new Chainsaw suite `hack/e2e-chainsaw/observability-proxy/`
with two tenants, covering `401`/`403`/`200`, scope-widening attempts,
path normalization and a long-lived `tail`.

#### How it was tested

- `go test -race` and `go vet` for the proxy packages.
- helm-unittest for `observability-proxy`, `dashboard`, `tenant` and
`core/platform`; bats tests for the console nginx routing.
- The Chainsaw suite was not run as-is: it deploys its own VictoriaLogs
as the root store and refuses to run when a real one exists. Its cases
were replayed by a script against the real store on a test cluster.
- Verified on a test cluster:
- `401` without a token and with an invalid token, `403` for another
tenant's namespace, `200` for the caller's own.
- Client `extra_stream_filters`/`extra_filters` (GET and POST), an
`AccountID` header and another tenant's stream named inside the query do
not widen the scope; `options(...)` answers `400`; a path with `..`
pointing at another tenant answers `400`; an endpoint outside the
allowlist answers `404`.
- Through the full browser chain (OIDC login, gatekeeper, gateway,
console nginx), a single `tail` stayed open for more than two minutes
without reconnecting, and requests from the same session to other
tenants' roots answered `403`.
- Not verified: the `429` limits under real load. `golangci-lint` has
not been run locally.

#### Known limitations

- The proxy is reachable only through the console route; tenant
workloads have no egress rule to it. Every request takes its concurrency
or tail slot before the body is read; the token and access reviews and
the body read run under `--request-timeout`, so a stalled kube-apiserver
answers `503` rather than holding the request; and writes to the client
carry a deadline (renewed per line for tails). `--request-timeout=0`
disables those bounds and should not be used in production.
- The first lines a newly started pod writes can be missing from an
already open `tail`, although they are in the store. The likely cause is
that Fluent Bit picks up a new log file later than the VictoriaLogs
live-tail window; later lines of the same pod arrive. A history query
shows them.
- The per-application roots (`.../<plural>/<name>/log`, `.../metrics`)
and the namespace metrics root answer `404` in this PR.

### Screenshots

No UI changes.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: https://github.com/cozystack/website/issues/787 (docs land
once the implementation PRs merge)
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

Downstream: `cozystack/website` is reached. A new platform component
(`packages/core/platform/sources/observability-proxy.yaml`) triggers
`content/en/docs/next/guides/platform-stack/_index.md` and
`operations/configuration/licenses.md`; adding it to the system bundle
(`packages/core/platform/templates/bundles/system.yaml`) may also need a
line in `operations/configuration/variants.md`. A user-facing page for
the logs API and the new tenant `view` permission is not required by the
trigger map but would be useful. Tracked in cozystack/website#787. Other
repositories: no trigger matched. `packages/core/platform/values.yaml`
is not changed (only a test fixture), no package under `packages/apps/`
gains a values field, and no CRD changes. The root `Makefile` `build`
target builds one more image; this does not touch `hack/` or the make
targets `ccp` relies on, so `ccp` is not considered affected, but a
reviewer may want to confirm.

### Release note

```release-note
feat(observability-proxy): add an authorizing proxy that lets tenant users read their namespace logs from VictoriaLogs with their Kubernetes credentials, under `/apis/cozystack.io/v1alpha1/namespaces/<ns>/logs/`. Access requires the tenant `view` role or higher. The component is enabled by default on full variants and opt-in on slim variants.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Added an observability proxy for tenant-scoped log access through the
console, including live log streaming. Requests are authenticated,
access-controlled, and subject to configurable rate and concurrency
limits.
* Added platform packaging and deployment support, with console routing
for observability requests.
* **Tests**
* Added automated coverage for access control, tenant scoping, query
handling, streaming, proxy behavior, and deployment configuration.
* **Documentation**
* Updated end-to-end test guidance with host requirements for running
the observability proxy suite.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-11 16:24:25 +03:00
Aleksei Sviridkin 9fe06a874b test(release): isolate report deduplication from workflow attempts
CI reruns expose GITHUB_RUN_ATTEMPT to a test that expects a URL for
the first attempt. Keep its synthetic run identity independent of the
surrounding workflow.

Fixes: 5a997fb7d4 ("feat(release): post an immutable report of every classification and write record version 2")
Assisted-by: LLM
Signed-off-by: Aleksei Sviridkin <f@lex.la>
2026-11-11 15:35:18 +03:00
Myasnikov Daniil 885752eb5a feat(release): announce a parked original once on the open batch
A park is a decision a maintainer has to act on: port what applies as
a pull request of its own, or drop the original. Listing every park in
every notify comment would make that list noise within a week, and a
park nobody hears about is a fix that silently never ships.

notify now names each park the open batch's record holds that is not
yet marked announced, and once its comment is written marks those parks
in the record with gh pr edit --body-file. It writes only while the
pull request is open and its body still carries the record notify read,
so a run that rewrote the body in between loses nothing; a mark that
fails exits 2 and the next notify names the park once more. run carries
the mark into the next record, a merged or closed carrier's too, only
while the park is the same event: merge, kind, prerequisites, commits
and files. A park that changed is news and is named again. With no
batch open nothing is named (M2); the next batch's first notify does
it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 11:11:15 +00:00
Myasnikov Daniil ce4ab1fd2c feat(release): park an original whose dependency never reaches its line, and hold one whose dependency is pending
On 1.6, the fix in #5071 conflicts on every run: it modifies files
that the feat in #3940 added on main, and that feat never reaches the
line. The fix in #5082 builds on #5071 and on code that never comes
either. Trying them on each run says nothing a maintainer can act on,
and a conflict reads as work to do.

Edges come from the dependencies the previous commit reads. One whose
prerequisite the line or the open batch carries, whatever its rule or
label, that this run picked or found empty, or whose evidence is
present at the branch head once earlier picks moved it, is satisfied
and dropped first. A direct push, an original rules 2-4, the start, a
skip label or a drop keep out, or one itself parked never comes and
parks the original. A prerequisite still to come (a conflict, a
deferral, a workflow wait, rule 6) holds it as deferred{needs}, the
strongest kind deciding: missing, workflow, deferred, rule 6; a chain
inherits.

run, check and the metadata operations share one order: partial unit
on the batch, a direct reason, an unpicked partial, the workflow
exception, a dependency park, a wait, then the original's own state.
In the scheduled mode a workflow original whose remaining
never-dependencies are only tests or docs waits for a maintainer
instead of parking, which is the case of the release fix in #5142.
check takes the mode the open record saved, and the scheduled one
with no batch open, where a closed record's workflow waits wait on
nothing any more and the original is judged as the scheduled run
judges it. check and the record-only operations keep a partial park
the open record holds while the batch head still shows it, since only
the run's picker sees it after the picks before it. Records refuse
waits and parks that form cycles, point at later merges or contradict
each other. notify lists a wait after what decides it, from the
record alone. backport/include lifts neither park nor wait.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 11:11:15 +00:00
Myasnikov Daniil 4a909818e4 feat(release): name what an original builds on that its line lacks
On 1.6, seven of the twelve originals rule 6 holds back build on code
the line never got: #5071 changes files #3940 added on main, #4974
reads a variable only #4386 defines. Each conflicts on every run, and
nothing tells a maintainer which missing pull request the conflict
comes from, which every human backport so far had to work out by hand.

For each original the policy takes and nothing carries, run now reads
what it builds on from git alone. A modified or deleted file the line
lacks is named by the first-parent commit on main that added it (V1).
A removed or changed line is blamed at M^1, mapped to the first-parent
commit that brought the blamed commit, and kept only while its text,
edges trimmed, is absent from the line's file (V2c), so a hand backport
under another SHA clears it. Generated files and the keys
update-crd.sh writes into an ApplicationDefinition are not blamed.

Paths go to git literally, blame gets a validated full SHA and no
--end-of-options, which git refuses there, and every call has a
two-minute deadline. Caps, deadlines, binary files, symlinks,
submodules, unmapped commits and skipped rename detection become
diagnostics shown with the original on every outcome and in the
report; malformed git output stops the run. This commit only
annotates; nothing is parked yet.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 10:58:42 +00:00
Myasnikov Daniil 6696953d26 fix(release): set aside the note of a pick that differs from its original
A release note describes what the original does on main. A pick git had
to fit to a line that already had part of the change, a resolution, or
a unit whose first commit came out empty no longer does the same thing,
yet its note went into the batch's release-note block with the rest,
and the changelog took it from there.

A unit now counts as changed when a surviving pick's patch-id differs
from its source or the unit's net patch-id differs from the original
merge's; the latter catches an empty first commit, which leaves no pick
to compare. A changed original's note moves under "Check these notes",
fenced and neutralised as before, with a range-diff per differing pick
and the two diffs to compare. The range-diff uses a creation factor of
999 because the default leaves a pick that lost most of its change,
#5088's test-only pick for one, unpaired. The record and the report
list such originals as changed.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 10:58:32 +00:00
Myasnikov Daniil c617a99453 fix(release): withhold or refuse a unit that does only part of its original on the line
A clean cherry-pick says nothing about whether the original landed
whole: git deletes a file the line never had without a conflict and
leaves a file the line lacks untouched. #5088 reached the 1.6 batch
that way, carrying only its test while the two deletions that are the
fix had no effect.

Every M and D path of an original is now judged at its counterpart on
the line, followed through renames on main, on the line, or both. A
modified path missing before and after, or left unchanged when the line
holds neither the original's blob nor a clean git merge-file result, is
omitted; a deleted path absent before is omitted. Content merge-file
refuses, mode changes and skipped rename detection become diagnostics
and never count as omissions; I/O failures stop the command.

The same guard runs in four places: the picker, which resets a partial
pick and parks the original; a check-time pass over every unit on the
open batch at its own first parent, whatever its label, which fails
check --cut and verify; the same pass over every unpicked original's
clean net merge, which parks a partial one with no record needed; and
fold. contained accepts the renamed path, so a fix of a renamed file
passes verify and fold. The unit mapping also closes the found() gap: a
merged original whose first pick came out empty is on the batch, not
missing.

The record moves to version 3 with parked entries and partial units,
and reports to format 2 with typed parked entries; format 1 reports
still read.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 10:58:32 +00:00
Myasnikov Daniil 2c1cb2f41c feat(release): drop an original from one line with run --drop
A fix the rules take for every line can still be wrong for one of
them, typically because it builds on code that line does not have.
The only ways to keep it out were backport/skip, which keeps it off
every line, and --defer, which holds it out of one batch: the next
batch tried it again, and a conflict that would never resolve came
back every run.

run --drop N --reason R records a drop in the line's record, bound to
the original's merge commit on main, and every later record carries
it, a new batch's included. What the open batch carries of a dropped
original comes off it the way --defer takes one off; any other drop
changes only the record, so a batch on a line that moved since stays
where it is. --undrop withdraws one. Both need an open batch, as every
write to the record does.

A drop comes after the rules and the carries: backport/skip, rules 2
to 4 or the line itself decide first and the drop stays on record,
and backport/include does not lift it. check notes a dropped original,
the body lists it, fold refuses it, and a record whose drop names
another original's merge stops every command. Each report keeps the
record's drops with the merge commit each is bound to and its reason,
held on read to the same rules, so an earlier report still says why
after an undrop.

A reason now travels from a body to terminals, bodies and comments, so
a reason or a file name in a record is believed only in the form a run
writes it, one clean line, and so is a report's diagnostic. Output
that repeats a reason, file names, a report ID or a state GitHub
reports escapes it.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:32 +00:00
Myasnikov Daniil 79b335e461 fix(release): close the batch --defer leaves empty, recording why while it is open
docs/release/backport-tasks/failed-ci.md says to defer the culprit with
run --defer --push. When the culprit was all the batch carried, the
push stopped with "close batch pull request #N instead", and closing
it by hand left nothing that said why: once closed, the deferred
original was simply missing.

The run now closes the batch itself, and does every write while the
pull request is open: the report, then the body, whose record keeps
the deferrals and says the batch is closing, at the head the rebuild
started from. The close and the deletion of the branch go only while
GitHub and the remote still show that head, and the deletion is
leased on it.

A close that stops part way is finished by the next run --push, before
anything else, and by no other command; a head the close did not
freeze is refused, never adopted. The run that finishes it stops
there, and fails when it was asked for --defer, since no open batch
is left to defer on. Recovery never writes the closed
pull request. A same-day batch that reuses the branch name and fails
to open its pull request leaves a branch the next run names for a
person to delete.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:00 +00:00
Myasnikov Daniil 5a997fb7d4 feat(release): post an immutable report of every classification and write record version 2
Each run rewrote the batch body whole, so the only account of what a
run found was gone at the next run, and a merged batch kept only its
last one. Nothing said what a classification was made of either: not
the heads it read, the policy, the tool or the flags.

A run now makes a report: every original merged into main since the
line branched, in exactly one category, with what the run read. Its
ID hashes the classification and names no pull request, so a new
batch's report is whole before GitHub numbers the pull request, and a
run that finds the same as the last one makes the same report, which
is posted once and keeps its first bytes. The body carries the latest
report under the record; a comment the tool never edits keeps each
one, so report reads any of them back, from merged batches too.

The comment goes up before the push and the body edit on an open
batch, and after the pull request a new batch opens with the report in
its body; a later run posts a report the body kept but no comment did,
byte for byte, before its own. Every write is preceded by a read of
the pull request, and nothing is written once it is merged or closed.
Both the comment and the body are rendered and measured before the
first write, so an oversized one stops the run with nothing written.

The record becomes version 2: what it was built from, with the run's
--no-workflow-files, which --defer keeps rather than resets, and the
report it goes with.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-10 07:20:00 +00:00
ZverGuy b3288be903 test(observability-proxy): cite only sources the repository has
The flag defaults test named a design document that is not in the
repository as the origin of the cost limits; the limits stand on their
own as flag defaults.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 04:03:06 +07:00
ZverGuy 3e2472280e fix(observability-proxy): keep the vmselect timeout below the request's
An equal vmselect timeout still lost the race: the proxy's deadline
starts before the request reaches vmselect, so the proxy cut first
and answered 504 for a store within its time. The vmselect timeout
must now be strictly below a set request timeout, the flag help says
so, and tests pin the accepted side too, including no request timeout.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 0c5d1cd93f fix(observability-proxy): keep the vmselect timeout within the request's
The timeout forwarded to vmselect could be set longer than the proxy's
own request timeout. The proxy then cut a query vmselect was still
allowed to run and answered 504 for a store that was within its time.
Such a combination now stops the proxy at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 81cd278f63 test(observability-proxy): cover metrics concurrency and flag overrides
The per-user concurrency cap on the metrics root had no test, so
dropping it from the handler went unnoticed. The new test holds one
request in the upstream, expects the same user's second one to be
refused and another user's to go through. Overrides of the metrics
flags were only checked for their defaults and refusals.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy cd3d889ee8 feat(observability-proxy): add metrics store and cost limit flags
The store names and the short-term retention mirror metricsStorages in
the monitoring package, which an operator can rename or resize; they
are flags so the chart can follow such a change without a rebuild. The
cost limit defaults are the ones the metrics proposal settles on: 30d
range, 11000 points per series, 30s timeout.

Store names become part of a Service host name, so they are checked
as DNS labels at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:56:18 +07:00
ZverGuy 59cbf4b28e docs(observability-proxy): say what the request timeout bounds
The flag now also bounds the body read and the token and access
reviews, and 0 lifts all of those at once; an operator raising it for
slow store queries needs to know that. The store comment named a
design document the repository does not have.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy d2ef5e031f docs(observability-proxy): describe the proxy as it stands
The package and serve comments described a delivery plan rather than
the code: the command named metrics it does not serve, and serve gave a
plan as the reason to refuse unknown roots before authenticating.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 03:53:24 +07:00
ZverGuy 9d9f5fcb6c docs(observability-proxy): name the source of the logs bucket default
The default of 11000 buckets was justified by a metrics proxy that
does not exist on this branch. The number is Prometheus's cap on
points per series in a range query, which the metrics root reuses.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy b1108e836a fix(observability-proxy): refuse to resolve a store before the cache syncs
An absent namespace resolves to the root store, and before the
informer's first list every namespace is absent, so a request served
in that window read a labeled namespace's logs from the wrong store.
Readiness keeps the Service from routing there, but a client that
addresses the pod directly is not stopped by it.

The resolver now fails while the cache is unsynced, which answers
503 like any other resolution error.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy cd04c3e6c4 fix(observability-proxy): count the aligned buckets in the logs step
vlselect widens the window to step boundaries and returns a
zero-filled bucket for each one, so a start off a boundary yields one
bucket more than the span divided by the step. Sizing the step from
the unaligned span let hits and stats_query_range return MaxBuckets+1
buckets.

The minimum step now spans MaxBuckets-1 steps, and the smallest
accepted --logs-max-buckets rises to 4 so that this step still fits
the third-of-range cap that keeps alignment inside the window.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 2d8de08724 fix(observability-proxy): refuse positional arguments
The flag parser stops at the first argument that is not a flag, so
a stray word in the container args made every flag after it fall back
to its default without an error, the root tenant included.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
ZverGuy 2e3edcf30d fix(observability-proxy): refuse a non-finite per-user rate
NaN fails the negative check, as every comparison does, and +Inf
refills a bucket instantly, so either value switched the per-user rate
limit off while the flag looked set. Both now stop the proxy at
startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-10 00:45:06 +07:00
Myasnikov Daniil 5945d2ddd1 feat(release): read a line's record from its newest batch pull request in any state
The deferrals a batch recorded were read only from the open batch pull
request. Once it merged, a conflict it had deferred showed as missing
and failed check and the cut, until some later run picked something
else and opened a new batch; a run that picked nothing opened none.

run, check, fold and notify now find the line's record in the newest
batch pull request whatever its state: one opened from a batch branch
of this repository by an author the batch trusts. A record that is in
doubt stops the command instead of letting it fall back to an older
one: a block by an author no longer trusted, a bot's batch whose block
was removed, a record that does not read whole, a closed record newer
than the open batch, or no record within the pages read.

The record is still written only to the open batch, and a merged or
closed batch is never written again, so a run that picks nothing with
no batch open keeps nothing and says so. The reader takes the version
2 record the next change writes; the writer stays at version 1.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-09 15:03:43 +00:00
Myasnikov Daniil 3fc444874e fix(release): print the batch pull request's URL when --push publishes it
run --push opened or updated the batch pull request and said nothing
about which one, so the scheduled job's log and a maintainer's
terminal had no link to follow. gh pr create prints the URL; the run
now passes it on, on stderr, so stdout stays the body alone, which the
scheduled script saves as it is. An update prints the open pull
request's URL.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-09 14:54:47 +00:00
Myasnikov Daniil 015fc51c0d fix(release): name policy rules instead of internal numbers in output
fold, run --defer and the run's own stderr printed "rule 101" for an
original merged before its line started under the policy, and "rule
100" for backport/skip. Those numbers exist only inside the tool; the
policy has six rules, so a reader looking for rule 101 finds nothing.
Output now names such a rule in words, and the policy's own rules by
their number.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-09 14:53:29 +00:00
Myasnikov Daniil 35c47f20c1 fix(release): fail check --cut while a workflow original waits alone
check --cut returned 0 whenever nothing was missing and nothing was
only on the batch, so a workflow original waiting for a maintainer
passed the cut once the pick beside it was labelled backport/skip or
reached the line by hand. The waiting original was not even printed:
the case that held it swallowed the warning the plain check gives.

check now sorts every original into one disposition and exits once,
so a wait fails the cut on its own. A backport/skip original the open
batch still carries is named with the run --defer that takes it out.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-09 14:52:13 +00:00
Aleksei Sviridkin 9e8e10c307 fix(backups): re-render the default backup objects when the S3 endpoint changes (#5230)
## What this PR does

Refs #5205.

The S3 endpoint the platform backups use is copied twice, and neither
copy is refreshed after `publishing.host` changes. The COSI sidecar
writes `BucketInfo` once per `BucketAccess`, so the credentials Secret
`bucket-cozy-backups-system-credentials` rendered from it keeps the old
endpoint. The Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of `backupstrategy-controller` saw. The default
objects gate only acted on missing objects, so backups kept going to the
old host and nothing reported it.

The gate now also forces a render when a copy is stale:

- the bucket's `<bucket>-system` release, when the credentials Secret's
endpoint differs from the COSI Secret it is rendered from;
- `backupstrategy-controller`, when the endpoint an object was rendered
with differs from the credentials Secret's. The chart stamps the
rendered endpoint on the BSL and on every default Strategy as
`backups.cozystack.io/endpoint`, because the endpoint itself sits at a
different path in every kind. Objects without the annotation count as
stale until a render stamps it; `MinForceInterval` bounds the forces.

Stale credentials are forced before the BackupClass and the routed
objects are read, so an absent BackupClass or an unreadable Strategy
still fails the check but does not keep the old endpoint in the
credentials Secret. A `BucketInfo` without an endpoint is not compared,
since rendering from it would write an empty endpoint or fail the bucket
release. Repairing missing objects keeps working while the credentials
comparison is stale or fails, and a `systemSecretName` that is not
derived from a COSI Secret (no `-credentials` suffix, or the raw COSI
Secret itself) is not compared at all. RBAC is unchanged: the controller
already holds cluster-wide `get` on Secrets. No new metric: nothing
would consume it.

What this does not do: the COSI Secret still holds the endpoint the
`BucketAccess` was granted with, so after a host change the endpoint
follows only once an operator deletes the `BucketAccess` and forces
`bucket-cozy-backups` to recreate it (this issues new S3 keys). From
there the gate replaces the remaining manual steps of the recovery
described in the issue. Automatic re-grant is left for a separate issue,
and the operator runbook goes to the website, hence `Refs` rather than
`Fixes`.

Known limitation: the credentials comparison runs only when
`backupStorage.systemSecretName` is
`bucket-<bucketName>-system-credentials`, the Secret the forced
`bucket-<bucketName>-system` release renders. If an operator sets
`bucketName` and `systemSecretName` inconsistently, the comparison is
switched off rather than forcing a release that never rewrites the
compared Secret, so a stale endpoint in that Secret is not repaired.
Repairing a missing credentials Secret still forces the release derived
from `bucketName`, as before this PR.

Tested with unit tests for the gate (including the comparison
normalisation shared with the chart helpers), helm-unittest for the
annotation and the wiring, and a bats guard that every rendered Strategy
CR and the BSL carry the annotation. On a live cluster, a server-side
render of the chart produced exactly the annotation the gate derives
from the real credentials Secret, so the steady state does not force
anything.

### Screenshots

Not a UI change.

### Downstream repositories

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up: an operator runbook for changing `publishing.host` with
platform backups enabled (re-grant the `BucketAccess`, then let the gate
converge; restart ClickHouse pods that read the credentials from
environment variables). Not filed yet, pending approval of the text.
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up:

### Release note

```release-note
fix(backups): the backup controller now re-renders the Velero BSL, the default backup strategies and the platform bucket's credentials Secret when their S3 endpoint is stale, so after the platform bucket's BucketAccess is re-granted for a new publishing.host, backups move to the new endpoint without forcing releases by hand.
```


<!-- This is an auto-generated comment: release notes by coderabbit.ai
-->
<!-- coderabbit-ai-generated:v1 -->

## Summary by CodeRabbit

* **New Features**
* Backup strategies and storage locations now record their storage
endpoint, enabling detection and repair of outdated endpoint settings
after bucket access changes.
* Bucket credentials and backup objects are refreshed independently when
their endpoints are missing or out of date.
* **Documentation**
* Clarified when endpoint changes trigger updates and how suspended
releases are handled.
<!-- end of auto-generated comment: release notes by coderabbit.ai -->
2026-11-09 12:02:42 +03:00
ZverGuy a56ff5050f test(observability-proxy): accept the smallest satisfiable step cap
Startup refuses --logs-max-buckets below 3 and --logs-max-range below
3s, but only the refusals were tested, so moving either bound up by
one went unnoticed. Pin the smallest accepted pair.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy ca7f283b6a fix(observability-proxy): require a range and bucket count the step cap fits
The step is capped at a third of the maximum range so that vlselect's
alignment to whole steps cannot widen the window past that range. With
fewer than three buckets, or a range under three seconds, the minimum
step exceeds that cap and the bound no longer holds, so such settings
are refused at startup.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 8eef2fa5d2 fix(observability-proxy): reject negative limits at startup
Zero is the documented way to disable a limit; a negative value
disabled it too, silently. Refuse it so a typo in the chart cannot turn
off a per-user or total cap unnoticed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 1c345da367 fix(observability-proxy): reject negative timeouts at startup
A negative duration silently disables the bound it configures, so a
typo in the chart would turn off the request deadline or the tail idle
close without any sign. Refuse it when the flags are parsed.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
ZverGuy 8b0ac342b5 feat(observability-proxy): add the observability-proxy command
The deploy unit builds an image and chart around this binary, so its
flags follow the agreed contract: listen and probe addresses, cluster
domain, root tenant and auth cache TTL, plus the logs limits. Defaults
are chosen so a stock install is safe without tuning; readiness waits
for the namespace cache because store resolution reads it on every
request. On shutdown open tails are cut after a grace period so their
clients reconnect with since to another replica.

Assisted-by: LLM
Signed-off-by: ZverGuy <maximbel2003@gmail.com>
2026-11-09 15:48:57 +07:00
myasnikovdaniil 7097cc5cda feat(release): make backports automatic with backport-batch (#5106)
## What this PR does

Backports become automatic. Fix merged to `main` with `fix` title, or
`security` for fix that comes through security advisory, reaches release
lines through batch that runs on Monday and Thursday, newest line first,
nobody requests, labels or opens backports by hand. Batch runs for two
newest lines with stable release (`released-lines` in policy block) and
every line branched after them that has no stable release yet: with 1.6
and 1.7 released, `release-1.8` created at `v1.8.0-rc.1` joins them, and
`v1.8.0` drops 1.6. Line counts as released once `vX.Y.Z` tag without
pre-release suffix exists on origin. Rules are in
`docs/release/backports.md`, `cmd/backport-batch` applies them,
`.github/workflows/backport-batch.yaml` runs it. Design is in
cozystack/community#114, merge shape of each original in
cozystack/community#119.

- `backport-batch run --line release-X.Y` takes PRs merged to `main`
since line branched (title and body read from merge commit) and skips
what rules exclude: other types and `deps`, breaking changes, platform
migrations, `targetVersion`, changed CRD documents, API changes that API
review gate calls sizeable (batch runs `internal/apigate` itself
whenever PR changes a file the gate parses:
`pkg/apiserver/apiserver.go`, an ApplicationDefinition, or YAML under
`packages/` or `internal/` naming `CustomResourceDefinition` or
`APIService`), and on older line what next newer line doesn't carry yet.
The rest goes onto `backport-batch/<line>-<date>` in merge order. PR
merged to `main` with merge commit goes as its own commits picked with
`cherry-pick -x --signoff`, so each keeps its author, message and
sign-offs and gets sign-off of whoever runs the batch (with `-m 1`
author was whoever merged on `main`), merges of `main` into PR branch
are left out, and merge commit joins them to the batch with original's
merge message and `Backported-from:` trailer. Squash-merged PR goes as
one pick. Conflict on any commit takes branch back to where it was
before that original, it's listed with the commit that failed, run goes
on. `--push` opens or updates draft PR, `--continue` carries on open
batch PR as fast-forward only, PR body it didn't write is never
rewritten. Same-day run over batch branch whose PR never got opened
refuses and names `run --line <line> --onto <head> --push`, which
continues it. Run or `--defer` that resets local batch branch over
commits only it has prints `git branch` command that restores them.
- `run --defer <N> --reason <why>` takes original out of open batch
(usually the one failing CI is pinned on). It rebuilds branch replaying
batch unit by unit without it, so merges and pushed conflict resolutions
keep their shape, and only this rebuild pushes with
`--force-with-lease`. Deferred original comes back in next batch.
- Line carries something only if its own history shows it: picks whose
`-x` line, the one git appends, names PR's merge commit or every commit
of its own, or pick that changes nothing. Picked commits keep their
messages and batch merge carries PR body, so text in them (`-x` lines,
`Backported-from:`, `Backport of #N`) counts only on commits from before
`batch-only` day in policy block, where it records old hand backports
and batches. Otherwise text in a merged fix could mark another fix as
already backported, and it would never reach the line. PR against line
never counts. Open batch PR counts only from this repo and from author
with write access or the batch's own bot (`BACKPORT_BATCH_BOT`, exact
login, `cozystack-ci[bot]` when unset, so maintainer's run by hand
continues scheduled batch instead of opening second one), and is read
the same way. Actions bot isn't trusted anymore, any workflow token
writes as it.
- Any failed GitHub API or git call stops the run with message naming
the call, no retries. Failed permission lookup used to count batch PR
out as look-alike, so `--continue` opened second batch PR and `notify`
posted second comment; now `run`, `check`, `verify` and `notify` exit 2
and write nothing.
- Policy applies fully from `release-1.7`. `release-1.6` takes only
fixes merged from 2026-10-26, older ones need `backport/include`. Start
days are in `backport-policy` block on the policy page, test keeps page
and code in sync.
- `backport-batch check --cut` replaces `backport-audit` as gate before
patch cut and before every rc after the first, it also fails while a fix
is only on open batch PR. Plain `check`, which scheduled run uses, lets
those through. `backport-batch fold` checks conflict resolution in the
same shape: merge on batch head whose second parent is straight line of
picks of PR's own commits in order, each with its `-x`, only PR's files
(one pick for squash). `backport-batch verify` holds every original on
batch to that shape, clean picks included, and `pr-labeler.yaml` runs it
on every push to batch PR: `pull_request_target` so it reaches
`release-*`, tool built from `main`, batch head fetched as data,
read-only token. Batch PR body says per original how many commits it
brought and which picks differ from their commits. `backport-batch
notify` reports batch that is still open from previous run and deferred
conflicts in one comment on batch PR, edited in place, with
`BACKPORT_ALERT_TEAM` mention.
- `hack/backport-batch-scheduled.sh` is what the workflow runs and
maintainer can run by hand (`--dry-run` changes nothing on GitHub): for
each line newest first `run --continue --push`, `check`, `notify`. It
pushes with `BACKPORT_BATCH_TOKEN` only and fails without it, never
`GITHUB_TOKEN`, because its pushes don't start CI. Workflow runs it as
cozystack-ci App: it mints token for this repo only with Contents and
Pull requests write, passes it as `BACKPORT_BATCH_TOKEN`, commits as
`cozystack-ci[bot]`, and sets `BACKPORT_BATCH_BOT` so the tool trusts
that bot's batch PR and comment. Job timeout is 30 minutes, under the
token's hour.
- App has no workflows permission, so workflow runs the script with
`--no-workflow-files`: original that changes `.github/workflows/` isn't
picked and waits on batch PR for maintainer, listed with its workflow
files in body and in notify comment. It's never a conflict. `check`
warns about it while it waits and `check --cut` fails on it. Maintainer
brings it over with own token by `run --continue --push` without the
flag, or takes it out with `--defer`.
- Deferred conflicts and failed batch CI wait for maintainer. `notify`
keeps one comment on batch PR listing stalled batch, conflicts and
workflow changes, mentioning `BACKPORT_ALERT_TEAM` (team or login) when
it's set. Failed batch CI isn't in the comment, it shows on batch PR
checks, and `check --cut` fails while batch is unmerged.
- Two steps that need judgement (resolve conflict, pin failed CI on
original and defer it) are written as runbooks in
`docs/release/backport-tasks/`: inputs, the only allowed output, check
it must pass before push, what never to do.
- `backport/include` and `backport/skip` let maintainer override rules
for single PR, they are new `backport/` category in `labels.yml`. Skip
wins over include, if open batch already picked it `--defer` takes it
out.
- Text written by other people (titles, release notes, reasons, file
names) goes to batch PR body only inside code spans or release-note
block, control, bidi and zero-width characters are dropped, file names
in deferred block included, git args go after `--end-of-options`, no
shell.
- Removed: `cmd/backport-audit`, `backport.yaml`, `kind/backport` and
`kind/backport-previous` labels with aliases and stale exemption,
trailer-check exemption for bot branches. `Commit trailers` job exempts
only `backport-batch/*` branches of this repo, changelog instructions
read originals from batch commits. Exemption lives in main's
`pre-commit.yml`, and batch PR runs its line's copy: `release-1.7` has
its own copy that exempts only old `backport-<pr>-to-<branch>` branches,
`release-1.6` has no such job. So on `release-1.7` picked commit with
bad trailer fails batch CI until line gets main's copy.
- `hack/close-manual-backport.sh` runs from `pr-labeler.yaml` on
`opened` and closes PR against `release-*` from outside contributor
(association contributor, first-time contributor or none, permission
read or none). Owners, members, collaborators, anyone with triage or
more and bots are left alone. When GitHub's answer on author's
permission can't be read, it closes nothing and fails the job naming the
failed read. Maintainer keeps release-only fix by reopening it.
Release-only PRs are occasional and opened by maintainers, for fix with
no counterpart on `main`, or one that on `main` came inside another PR
or was written for `main`; if its counterpart on `main` is a fix the
batch would take, the batch skips it once picking it changes nothing on
the line, otherwise maintainer labels it `backport/skip`.
- `docs/release.md` is split into pages under `docs/release/`, PR
template doesn't ask for `kind/backport` anymore. `security` type is
added to commit format in `docs/agents/contributing.md`, PR template,
Gemini review guide and labeler's list (labeler maps no `kind/*` to it,
`labels.yml` has no fitting one).
- `SECURITY.md`, `docs/security/incident-response.md` and
`GENERAL_TECHNICAL_REVIEW.md` now say two latest minor releases are
supported, same window the batch runs for, and link the backport policy
for which fixes go there.

Dry runs without push, on 2026-11-06, with `--no-workflow-files` as the
workflow runs it: `lines` gives `release-1.7` and `release-1.6`. On
`release-1.7` batch picks 12 fixes, defers 3 conflicts (#4756, #4867,
#5082), leaves #4888 and #4971 to a maintainer because they change
workflows, and skips 49. On `release-1.6` it picks #5088, defers 3
conflicts (#5071, #5086, #5087), leaves #5142 to a maintainer, and skips
590. `verify` passes both batches.

Setup after merge: no new secret. Batch runs as cozystack-ci App from
existing `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY`, job fails
when either is missing. App's installation already has Contents and Pull
requests write on all org repos, and no Workflows is needed: PRs that
change `.github/workflows/` wait for maintainer, who brings them over
with own token as "Changes to workflows" in `docs/release/backports.md`
says. Today that's #4888 and #4971 on `release-1.7` and #5142 on
`release-1.6`. Set `BACKPORT_ALERT_TEAM` to a team or a login (for
example `cozystack/maintainers`): unset, batch comment mentions nobody
and nobody is notified. Once this PR is merged, `Backport batch shape`
can be required check in the release lines ruleset (`release-*`
excluding `release-*.*.*`), with "Do not require status checks on
creation" on, because `cut-prerelease.yaml` creates `release-X.Y` by
push. `PR Auto-Label` with both new jobs runs on `pull_request_target`,
so it needs #5135 sorted before 2026-12-02. `kind/backport*` labels have
to be deleted by hand, label sync doesn't delete.

### Screenshots

Not a UI change.

### Downstream repositories

cozystack/community `contributors/README.md` lists commit types, so it
needs `security` too, it has no backport label text. Website has no
backport label instructions, and its links to `docs/release.md` carry no
anchor, the page stays as index. Trigger map lists commit convention
change for cozystack/ccp: this adds `security` type, but not message
format or trailer, and `package-bump` writes only `chore(...)` commits
and restates the trailer, so nothing changes there.

- [ ] No downstream repository is affected by this change
- [ ] [cozystack/website](https://github.com/cozystack/website) -
follow-up:
- [ ]
[cozystack/terraform-provider-cozystack](https://github.com/cozystack/terraform-provider-cozystack)
- follow-up:
- [ ]
[cozystack/ansible-cozystack](https://github.com/cozystack/ansible-cozystack)
- follow-up:
- [ ] [cozystack/ccp](https://github.com/cozystack/ccp) - follow-up:
- [ ] [cozystack/talm](https://github.com/cozystack/talm) - follow-up:
- [ ] [cozystack/cozyhr](https://github.com/cozystack/cozyhr) -
follow-up:
- [ ] [cozystack/cozy-proxy](https://github.com/cozystack/cozy-proxy) -
follow-up:
- [ ]
[cozystack/cozystack-telemetry-server](https://github.com/cozystack/cozystack-telemetry-server)
- follow-up:
- [ ]
[cozystack/external-apps-example](https://github.com/cozystack/external-apps-example)
- follow-up:
- [ ] [cozystack/examples](https://github.com/cozystack/examples) -
follow-up:
- [ ] [cozystack/community](https://github.com/cozystack/community) -
follow-up: add `security` to commit types in `contributors/README.md`

### Release note

```release-note
feat(release): backports are automatic. Fixes merged to main with fix or security title reach release lines through a batch built by cmd/backport-batch every Monday and Thursday, by the rules in docs/release/backports.md. The batch runs for the two newest lines with a stable release and for a line in its rc window: the release-X.Y branch the first rc creates joins at once, and its vX.Y.0 drops the oldest line. Features, chores, docs, dependency bumps, breaking changes, migrations, CRDs and API changes that the API review gate calls sizeable wait for the next minor. The policy applies fully from release-1.7, release-1.6 takes only fixes merged from 2026-10-26. Each original reaches the line as its own commits under a merge commit, keeping their authors and sign-offs. Maintainers override the rules for single PR with backport/include and backport/skip labels and take an original out of a batch with backport-batch run --defer. backport-batch check replaces backport-audit as the gate before a patch release and before a later rc. The label-driven backport workflow and kind/backport labels are removed. Backport PR opened by hand against a release branch by author with read access or less is closed automatically. The scheduled batch runs as the cozystack-ci App and needs no secret of its own; a fix that changes .github/workflows waits on the batch pull request for a maintainer, since the App may not push workflow changes. Release documentation now lives in one page per topic under docs/release/.
```
2026-11-09 09:42:38 +05:00
Kirill Ilin dfd7cbed36 fix(operator): reject a Git cert secret before installing CRDs and Flux
CRDs and Flux are installed before the platform source, so refusing a
cert secret on a Git source only when the source is written exits the
operator after a partial install. The check now runs right after flag
parsing, and the source install reuses it.

The guard runs on every start, so a Git source without a cert secret is
now covered over https and ssh: a guard that misfired there would
crash-loop the operator for every Git platform source. The update test
also renames the cert Secret, as a CA rotation does.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-11-06 18:28:45 +05:00
Myasnikov Daniil eaa0169ff3 chore(release): retire the backport bot, its labels and backport-audit
The batch replaces the label-driven backport workflow and the audit
that gated a cut against it. The kind/backport labels, the stale
exemption for them, their contract tests and the audit's make target
go with them. labels.yml gains a backport/ category instead, with
backport/include and backport/skip, the overrides the batch reads off
a merged pull request.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-06 12:33:37 +00:00
Myasnikov Daniil 9f8c88b4f1 feat(release): build the backport batch with backport-batch
The rules need a tool that applies them the same way on every run and
shows what it did. backport-batch reads main's first-parent history,
judges each pull request by the policy block it reads from the page on
main, and picks what qualifies onto one batch branch per line. Each
original lands as its own commits under a merge that names it, so
authors and sign-offs survive and the line's own history shows what it
carries.

check is the gate before a cut. fold holds a hand resolution of a
conflict to the batch's shape, verify holds a whole batch to it, and
notify reports what waits on the batch for a maintainer.

Assisted-By: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
2026-11-06 12:33:36 +00:00
Kirill Ilin c15a0d01b8 feat(operator): let the platform source trust a private registry CA
The operator generates the cozystack-platform OCIRepository from its own
arguments and rewrites it on every start, and cozystack-packages copies
that spec. A registry behind a private CA therefore had no durable way
to be trusted: a certSecretRef patched onto the live object is dropped
on the next operator restart, after which source-controller fails TLS
and platform updates stop arriving.

--platform-source-cert-secret sets spec.certSecretRef from the operator
itself, so the CA (or a client certificate) survives the rewrite. Git
sources are refused instead of ignored, because GitRepository has no
certSecretRef and Flux reads a Git CA from the source secret instead.

Assisted-by: LLM
Signed-off-by: Kirill Ilin <stitch14@yandex.ru>
2026-11-05 20:19:29 +05:00
Alexey Artamonov cd91138f33 fix(backups): re-render the default backup objects when the S3 endpoint changes
The S3 endpoint the platform backups use is copied twice and neither
copy is refreshed. The COSI sidecar writes BucketInfo once per
BucketAccess, so the credentials Secret rendered from it keeps its
endpoint, and the Velero BSL and the default Strategy CRs keep whatever
endpoint the last render of backupstrategy-controller saw. The default
objects gate only acted on objects that were missing, so after a
publishing.host change every backup kept going to the old host with
nothing reporting it.

The gate now also forces a render when a copy is stale: the bucket's
<bucket>-system release when the credentials Secret's endpoint differs
from the COSI Secret it is rendered from, and backupstrategy-controller
when the endpoint an object was rendered with differs from the
credentials Secret's. That endpoint sits at a different path in every
kind, in some only as a bare host, so the chart stamps the rendered
value on each object as backups.cozystack.io/endpoint and the gate
compares that. Objects rendered before the annotation existed count as
stale until one forced render has gone through, which stamps it.

The COSI Secret itself still holds the endpoint the BucketAccess was
granted with; it follows a host change only once the BucketAccess is
re-granted.

Assisted-by: LLM
Signed-off-by: Alexey Artamonov <aleksei.artamonov@aenix.io>
2026-11-04 19:13:55 +03:00
myasnikovdaniil 4087218a02 docs: move the repo-only guides onto the website (#5034)
## What this PR does

Seven guides under `docs/` were operator facing and written alongside
the code, and left in the source tree where no user looks. They are on
the website now, so this deletes them here and repoints every reference.

```
docs/operations/backup-classes.md                  -> operations/services/backup-classes
docs/operations/resource-presets.md                -> guides/resource-management
docs/storage-immutability.md                       -> operations/storage-class-immutability
docs/operations/system-memory-limits.md            -> operations/configuration/system-memory-limits
docs/gateway-classes.md                            -> networking/gateway-classes
docs/observability/distributed-tracing.md          -> operations/services/monitoring/distributed-tracing
docs/operations/seaweedfs-capacity-for-backups.md  -> operations/services/object-storage/capacity-for-backups
```

`docs/release.md`, `docs/deprecating-applications.md` and the two
SeaweedFS runbooks in `docs/operations/` stay. A workflow, `AGENTS.md`,
a migration and a bats suite read them by path, which is the test from
#3777.

References point at `https://cozystack.io/docs/v1/...`. The site
rewrites that short prefix to the latest version in `layouts/404.html`,
so the links follow releases and pin none. Backup Classes, resource
management and `storageClass` immutability exist in the latest version
already. The other four pages are only in `next/` until 1.7.0 is
promoted, and their links start to resolve then. The generated mongodb
files come from `values.yaml` through `make generate` with
cozyvalues-gen v1.7.0, the version pinned in
`.github/workflows/pre-commit.yml`.

Tests: helm unit tests for mongodb (151), postgres (209), vm-disk (114)
and kubernetes-nodes (223), go tests for backupcontroller, presets and
the operator, rd-presets-check, the migration-50 bats suite.

### Downstream repositories

- [x] [cozystack/website](https://github.com/cozystack/website) -
follow-up: cozystack/website#763

Walked the rest of the trigger map against the diff. No provider
follow-up: the mongodb `values.schema.json` diff is one `description`
string, with no field added, removed or renamed, no enum, no default,
and no `kind`, `plural` or `release.prefix` change.
`packages/core/platform/values.yaml` changed in comments only and no
key, so the hand written platform-package table on the site is
unaffected.

Please merge after cozystack/website#763, the links in the code only
resolve once those pages deploy.

### Release note

```release-note
docs: seven guides move from the docs/ directory of this repository to cozystack.io (Backup Classes, resource presets, storageClass immutability, system component memory limits, GatewayClass selection, distributed tracing, SeaweedFS capacity for backups), and in-repo references point there.
```
2026-11-03 23:09:46 +05:00
IvanHunters a444ac4498 Merge branch 'main' into fix/tap-pin-digest
Reconcile the tap digest-pin change with #5071 (no-install-on-connect), now
merged into main. Keep this branch's resetTapPinState, the verify-before-pull
Trust text, and the index-README trust model, while adopting #5071's wording
that installing (not tapping) runs charts and dropping the stale
"re-registration" phrasing from the shared tap.go and rest.go comments.

Conflicts resolved in cmd/cozypkg/cmd/tap.go, pkg/registry/core/tap/rest.go,
and hack/packages-index/README.md.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-30 14:02:24 +03:00
IvanHunters 09b0688abd fix(marketplace): fail closed on an unparseable verified digest, and cover the generation gate
A verified cosign signature whose manifest digest could not be parsed fell back
to an unpinned tag pull, which reopens the tag-mutation window the verification
was meant to close. Return an error instead.

Add a test that isolates the status.observedGeneration == metadata.generation
conjunct of artifactReady; the existing tests exercised only the tag-prefix
conjunct, so a mutation of the generation check went uncaught.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 17:11:52 +03:00
IvanHunters 0195e73d38 fix(marketplace): verify the signature before pulling, and anchor the identity match
With --certificate-identity and --certificate-oidc-issuer, `cozypkg tap` pulled by
tag without verifying and applied the artifact's PackageSources with
ForceOwnership; only the Flux source carried the verify policy. A tag moved by
someone without the signing identity could land unsigned content in the cluster.
Verify the signature at tap time and pin the verified digest, so the content
pulled and applied is what a trusted identity signed.

Flux evaluates a cosign matchOIDCIdentity issuer and subject as Go regexps, and
they were written unanchored. An exact identity such as alice@example.com would
also accept malice@example.com, and the dots would match any character. Anchor
and quote both values at the API and the CLI.

Reject --tag together with a digest reference instead of silently ignoring the
tag.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 16:55:47 +03:00
IvanHunters 96758b9e3d fix(marketplace): drop the index-entry digest from the tap pin change
The index-entry `digest` field, and the gate and README claims around it, assume
a published index that does not exist yet: decision 0002 in the marketplace
proposal defers it, and #4981 tracks that work. The gate also never compared a
recorded digest against the artifact it verified, so the field added surface
without the check that would make it trustworthy.

Drop it from this PR. Digest pinning still works from an explicit
oci://host/org/repo@sha256:... reference and from the digest a tag resolves to;
only the index-entry path is removed. The tag-based pinning and the verify-flag
wording stay in the index README.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 16:55:36 +03:00
IvanHunters 2f39e304f0 fix(marketplace): reconcile tap texts with no-install-on-connect
Removing on-connect auto-registration made several user-facing and doc texts
wrong: they promised that tapping registers a repository's apps in the catalog
and runs its charts. Tapping now installs nothing, so the texts told operators to
wait for apps that never appear.

Reword the `cozypkg tap`/`untap` help, the tap success output (now printing a
`cozypkg add <name>` line per materialized PackageSource), the `cozypkg init`
scaffold README, the packages-index trust model, the dashboard Repositories
banner and per-repository empty state, and the stale re-registration code
comments. No behaviour changes beyond the added `cozypkg add` hint in the tap
output.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-26 15:18:14 +03:00
IvanHunters 0fd970304c fix(marketplace): preserve verify and pull secret on CLI re-tap
A re-tap applies the OCIRepository under field owner cozypkg via server-side
apply; a field this owner set earlier and now omits is removed. Re-tapping to
move the pin without repeating --certificate-identity/--certificate-oidc-issuer
therefore silently dropped the signature-verification policy, and re-tapping a
private repo without --secret dropped the pull credential. The CLI now carries
an existing source's spec.verify and secretRef onto the applied object when the
invocation does not set them, mirroring the API connect path, so moving the pin
never silently downgrades trust or breaks a private pull.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-22 22:35:21 +03:00
IvanHunters 5747f94781 docs(marketplace): document digest pinning and verify on tap
Update the tap command help and the community-index README for the new
behaviour: digest references, the automatic tag-to-digest pin, the
--certificate-identity / --certificate-oidc-issuer pull-time verify flags, and
the optional index entry digest. Extend the generated openapi definitions and
model-name helper for the new TapVerify type (code generation is otherwise
run in CI).

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-22 22:20:34 +03:00
IvanHunters c59ced04cc feat(marketplace): record and pin a digest in community index entries
An index entry can now carry a digest (sha256:<64 hex>) alongside its version.
Resolving a short name pins that digest, so 'cozypkg tap <name>' consumes the
exact content the CI gate verified rather than following a mutable tag; the
version is kept informational. A malformed digest in an entry is a hard load
error, consistent with the rest of the strict index decoding.

Assisted-by: LLM
Signed-off-by: IvanHunters <xorokhotnikov@gmail.com>
2026-10-22 22:16:39 +03:00