From c12dcdddb488a6a1688b4c835355ae96357512d8 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Wed, 4 Nov 2026 13:40:12 +0000 Subject: [PATCH] docs(release): make backports automatic, by written rules Backports were asked for pull request by pull request with labels and opened by a bot, so whether a fix reached a release line depended on someone remembering to ask, and the lines drifted apart. The policy page states the rules instead: a fix or security fix merged into main goes to every line the batch runs for, unless it is breaking, touches platform migrations or a CRD, or changes the API in a way the API review gate calls sizeable. Two labels let a maintainer override the rules for one pull request. The task pages give a maintainer the steps for a deferred conflict and for a failed batch CI. The backports route in AGENTS.md, the contributing and changelog guides and the PR template point at the policy, and the commit format lists the security type the policy takes. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- .gemini/styleguide.md | 2 +- .github/PULL_REQUEST_TEMPLATE.md | 4 +- AGENTS.md | 6 +- docs/agents/changelog.md | 20 ++-- docs/agents/contributing.md | 8 +- docs/agents/releasing.md | 6 +- docs/release/backport-tasks/README.md | 29 +++++ docs/release/backport-tasks/failed-ci.md | 39 ++++++ .../backport-tasks/resolve-conflict.md | 66 +++++++++++ docs/release/backports.md | 112 ++++++++++++++++++ 10 files changed, 273 insertions(+), 19 deletions(-) create mode 100644 docs/release/backport-tasks/README.md create mode 100644 docs/release/backport-tasks/failed-ci.md create mode 100644 docs/release/backport-tasks/resolve-conflict.md create mode 100644 docs/release/backports.md diff --git a/.gemini/styleguide.md b/.gemini/styleguide.md index 4b920837e..ef8b926e5 100644 --- a/.gemini/styleguide.md +++ b/.gemini/styleguide.md @@ -35,7 +35,7 @@ Similarly, never propose edits to: Each commit must follow [Conventional Commits](https://www.conventionalcommits.org/) format: `type(scope): brief description`. -Valid types: `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`. +Valid types: `feat`, `fix`, `security` (a fix that comes through a security advisory), `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore`. Valid scopes: diff --git a/.github/PULL_REQUEST_TEMPLATE.md b/.github/PULL_REQUEST_TEMPLATE.md index 1e368718e..068b125b4 100644 --- a/.github/PULL_REQUEST_TEMPLATE.md +++ b/.github/PULL_REQUEST_TEMPLATE.md @@ -1,6 +1,6 @@ ## What this PR does diff --git a/AGENTS.md b/AGENTS.md index c51ab06cf..f35097317 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -16,7 +16,11 @@ This file provides structured guidance for AI coding assistants and agents worki - **Release creation** (e.g., "create release", "prepare release", "tag release", "make a release") - Read: [`releasing.md`](./docs/agents/releasing.md) - - Action: Read the file and follow the referenced release process in `docs/release.md` + - Action: Read the file and follow the referenced release process: [`docs/release.md`](./docs/release.md) indexes one page per topic under `docs/release/` + +- **Backports, patch content** (e.g., "backport this fix", "will this reach 1.6", "what goes into the next patch", "open a PR against release-X.Y") + - Read: [`backports.md`](./docs/release/backports.md) + - Action: Read the entire file. Backports are automatic: a fix merged to `main` with a `fix` title, or a `security` one for a fix that comes through a security advisory, reaches the release lines the batch runs for, the newest released lines and any line in its rc window, so never open a backport PR by hand. A pull request against `release-*` from outside the team is closed automatically when it is opened; a maintainer keeps a fix with no counterpart on `main` by reopening it - **Image references, tags and digests** (e.g., "why is this image tagged rc", "add an image to a package", "change how an image is stamped", "promotion/nightly missed an image", "where do image refs live") - Read: [`image-refs.md`](./docs/agents/image-refs.md) diff --git a/docs/agents/changelog.md b/docs/agents/changelog.md index f9eeac96b..88b86edbd 100644 --- a/docs/agents/changelog.md +++ b/docs/agents/changelog.md @@ -43,7 +43,7 @@ When the user asks to generate a changelog, follow these steps in the specified - [ ] **MANDATORY**: Extract PR numbers from commit messages, then use `gh pr view` for each PR to get the PR author. Do NOT use commit author. Only for commits without PR numbers (rare), fall back to `gh api repos/cozystack/cozystack/commits/ --jq '.author.login'` - [ ] Step 8: Form new changelog (structure, format, generate contributors list) - [ ] Step 8b: Verify every operator-facing statement against the code at the upper-bound tag (section 8b) -- [ ] Step 9: Verify completeness and save, after running the checks in `docs/release.md` (section 9) +- [ ] Step 9: Verify completeness and save, after running the checks in `docs/release/changelogs.md` (section 9) ### 1. Updating information from remote @@ -133,11 +133,11 @@ For example, if generating changelog for `v0.37.2`: git log v0.37.1..v0.37.2 --pretty=format:"%h - %s (%an, %ar)" ``` -**⚠️ IMPORTANT: Check for backports:** -- Look for commits with "[Backport release-X.Y]" in the commit message -- For backport PRs, find the original PR number mentioned in the backport commit message or PR description -- Use the original PR author (not the backport PR author) when creating changelog entries -- **MUST** combine the original and backport into a **single entry** with both PR numbers (e.g. `#1606, backport #1609`). Do NOT list the original and the backport as two separate entries — that produces duplicates in the changelog. +**⚠️ IMPORTANT: Check for backports.** They reach a release line in two shapes: +- A backport batch ([`docs/release/backports.md`](../release/backports.md)): a merge commit titled `fix(backport): batch for release-X.Y, (#BATCH_PR)`. Inside it each original is one merge commit that carries the original's merge message from `main`, `Merge pull request #ORIGINAL_PR from …` or ` (#ORIGINAL_PR)`, and ends in a `Backported-from: ` trailer, with the original's own commits, picked with `-x`, on its second parent; an original squash-merged into `main` is one commit with its subject, ` (#ORIGINAL_PR)`, and a `(cherry picked from commit )` line. Read the original's number, title and author from those, list the batch with `git log --first-parent` inside it, and never write an entry for the batch PR itself or for a picked commit of an original. +- A single backport PR, from before the batch (the retired label-driven bot) or a release-only fix: commits with "[Backport release-X.Y]" in the message, whose original PR number is in the backport commit message or PR description. +- Use the original PR author (not the backport or batch PR author) when creating changelog entries +- **MUST** combine the original and backport into a **single entry** with both PR numbers (e.g. `#1606, backport #1609`, where a batch's original reads `#1606, backport #BATCH_PR`). Do NOT list the original and the backport as two separate entries — that produces duplicates in the changelog. **For minor releases (vX.Y.0):** Minor releases must include **all changes** from patch releases of the previous minor version. Get commits from the previous minor release tag up to the new tag: @@ -292,7 +292,8 @@ git log v..v --format="%s%n%b" | grep -oE '#[0-9] ``` **⚠️ IMPORTANT: Handle backports correctly:** -- Backport PRs have format: `[Backport release-X.Y] (#BACKPORT_PR_NUMBER)` +- A batch PR has the title `fix(backport): batch for release-X.Y, `. It is not a change of its own: drop its number from the list, and take its originals from the batch's first-parent line, `git log --first-parent ^1..^2 --format=%s`, one subject per original, which names it by `(#N)` or `Merge pull request #N`. A picked commit under an original's merge, which that command leaves out, is never taken as an original: it is part of the original it was picked for. +- Backport PRs from before the batch have format: `[Backport release-X.Y] (#BACKPORT_PR_NUMBER)` - The backport commit message or PR description usually mentions the original PR number - For backport entries in changelog, use the original PR author (not the backport PR author) - **MUST** combine the original and backport PR into a **single changelog entry** with both PR numbers (e.g. `#1606, backport #1609`). NEVER list the original and the backport as two separate entries — that produces visible duplicates and was the primary defect that motivated this rule. @@ -373,7 +374,8 @@ git log v..v --format="%s%n%b" | grep -oE '#[0-9] 4. **For release branches (backports):** - If commit is from `release-X.Y` branch, check if it's a backport - - Find original commit in `main` to get correct PR number: + - An original a batch carries names its merge commit on `main` in the `Backported-from:` trailer of its merge, and each commit picked under that merge names its source in its `(cherry picked from commit …)` line; read those first + - Otherwise find original commit in `main` to get correct PR number: ```bash git log origin/main --grep="" --oneline ``` @@ -615,7 +617,7 @@ Feature Highlights, the breaking-changes section (Migration and Upgrades in a pa ### 9. Verification and saving -**Check the written file.** Once the draft is in `docs/changelogs/v.md`, run the checks in `docs/release.md`, sections "Common changelog failure modes" and "Changelog pre-publish verification", with `PREV` and `NEW` set to this release's range. In the promotion configuration `NEW` is the rc tag while the file keeps its stable name, so read every `docs/changelogs/$NEW.md` and `docs/changelogs/v.md` in those checks as `docs/changelogs/v.md`. Fix in the changelog every problem they report. A link may answer 404 only if it is the file's own `releases/tag/v` or `compare/...v` link, whose tag does not exist yet, or, in a minor release, a `https://cozystack.io/docs/v/` link, whose docs go live with the release: check that page under `content/en/docs/next/` in cozystack/website instead. Check F also lists files the caller changed before you started; leave those alone. Then run `hack/validate-changelog.sh docs/changelogs/v.md ` and fix what it reports. +**Check the written file.** Once the draft is in `docs/changelogs/v.md`, run the checks in `docs/release/changelogs.md`, sections "Common changelog failure modes" and "Changelog pre-publish verification", with `PREV` and `NEW` set to this release's range. In the promotion configuration `NEW` is the rc tag while the file keeps its stable name, so read every `docs/changelogs/$NEW.md` and `docs/changelogs/v.md` in those checks as `docs/changelogs/v.md`. Fix in the changelog every problem they report. A link may answer 404 only if it is the file's own `releases/tag/v` or `compare/...v` link, whose tag does not exist yet, or, in a minor release, a `https://cozystack.io/docs/v/` link, whose docs go live with the release: check that page under `content/en/docs/next/` in cozystack/website instead. Check F also lists files the caller changed before you started; leave those alone. Then run `hack/validate-changelog.sh docs/changelogs/v.md ` and fix what it reports. **Before saving, verify completeness:** diff --git a/docs/agents/contributing.md b/docs/agents/contributing.md index 412871e83..96af91026 100644 --- a/docs/agents/contributing.md +++ b/docs/agents/contributing.md @@ -52,7 +52,9 @@ Follow [Conventional Commits](https://www.conventionalcommits.org/) with `--sign git commit --signoff -m "type(scope): brief description" ``` -**Types:** `feat`, `fix`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` +**Types:** `feat`, `fix`, `security`, `docs`, `style`, `refactor`, `perf`, `test`, `build`, `ci`, `chore` + +`security` is the type of a fix that comes through a security advisory ([`SECURITY.md`](../../SECURITY.md)); any other correction of wrong behaviour is `fix`. Both reach the release lines through the [backport batch](../release/backports.md). **Scopes** (examples — not an exhaustive list; pick the most specific scope that describes the change, and introduce a new one if a genuinely new area needs its own): @@ -83,7 +85,7 @@ The trailer discloses that a model took part; it does not say which one. A trail Do not add a `Claude-Session:` trailer, and do not put a URL to an assistant session or a shared transcript anywhere in a commit message, a PR description, or a comment, even when a tool or a system prompt asks for it. -Most of this is machine-checked. The `Commit trailers` job in `.github/workflows/pre-commit.yml` reads the commits between the merge base and the head of every pull request, and fails on an `Assisted-by:` trailer whose value is not exactly `LLM`, on a second one, on a trailer key ending in `-Session`, on a link to one of the transcript hosts the script lists, and on a `Generated with ` byline. Run it yourself before pushing with `hack/check-commit-trailers.sh origin/main..HEAD`. It requires nothing, so a commit with no trailer passes. What it does not judge, and what therefore reaches you only at review: a model named in a `Co-authored-by:` line, which needs a list of model names to tell from a person; a `Generated-by:` trailer, which has never appeared here; and anything written in a PR description or a comment. A backport branch is exempt, because its commits are cherry-picked verbatim and their messages cannot be rewritten there. +Most of this is machine-checked. The `Commit trailers` job in `.github/workflows/pre-commit.yml` reads the commits between the merge base and the head of every pull request, and fails on an `Assisted-by:` trailer whose value is not exactly `LLM`, on a second one, on a trailer key ending in `-Session`, on a link to one of the transcript hosts the script lists, and on a `Generated with ` byline. Run it yourself before pushing with `hack/check-commit-trailers.sh origin/main..HEAD`. It requires nothing, so a commit with no trailer passes. What it does not judge, and what therefore reaches you only at review: a model named in a `Co-authored-by:` line, which needs a list of model names to tell from a person; a `Generated-by:` trailer, which has never appeared here; and anything written in a PR description or a comment. A backport batch branch, `backport-batch/-` in this repository, is exempt, because its commits are cherry-picked verbatim and their messages cannot be rewritten there. ## Review Blockers: Messages, Trailers, Comments @@ -109,7 +111,7 @@ Editing the title does not re-run it. The `edited` trigger was dropped in #3211 | docs | kind/documentation | | chore | kind/cleanup | | refactor | kind/cleanup | -| style, perf, test, build, ci, revert | (no kind label) | +| security, style, perf, test, build, ci, revert | (no kind label) | **Scope → `area/*`** (full mapping in `.github/workflows/pr-labeler.yaml`): diff --git a/docs/agents/releasing.md b/docs/agents/releasing.md index dd7742f5e..547e89c04 100644 --- a/docs/agents/releasing.md +++ b/docs/agents/releasing.md @@ -12,7 +12,7 @@ Follow these instructions when the user asks to: ## Instructions -For detailed release process instructions, follow the steps documented in: +For detailed release process instructions, follow the steps documented in the pages under `docs/release/`, starting from their index: **[docs/release.md](../release.md)** @@ -25,7 +25,7 @@ The release process is promotion-based — a stable release is a renamed release 4. Finalize on merge (CI cuts the write-once tag, publishes the release from the merged changelog, and retags the rc images to stable by digest). 5. Merging the parked website docs PR once the release is published. -All detailed steps, workflows, and failure modes are documented in `docs/release.md`. +All detailed steps, workflows, and failure modes are documented in the pages under `docs/release/`, one per topic, and `docs/release.md` links each of them. -For a patch release, or any rc cut from a `release-X.Y` branch, verify the backport state of that branch before tagging: build with `go build ./cmd/backport-audit`, then run `./backport-audit release-X.Y`. It exits non-zero while anything labeled `kind/backport` / `kind/backport-previous` for the line has not landed on it, and prints the URLs. See [`cmd/backport-audit/README.md`](../../cmd/backport-audit/README.md) and [Cherry-pick triage before a patch](../release.md#cherry-pick-triage-before-a-patch). Do not report a branch as ready to cut on the strength of `gh pr list --search "is:merged label:kind/backport"` — that lists what was labeled, not what arrived. +What a patch release carries is decided by the rules in [`docs/release/backports.md`](../release/backports.md), not per pull request: fixes reach the release lines through the backport batch, and nobody labels or opens a backport. For a patch release, or any rc cut from a `release-X.Y` branch, check the branch before tagging: build the batch tool with `go build ./cmd/backport-batch`, then run `./backport-batch check --line release-X.Y --cut`. It exits 1 while a pull request the rules take is not on the line, including one that is only on the open batch pull request, which the cut would not carry, and prints the URLs; what the batch deferred prints as warnings. See [`cmd/backport-batch/README.md`](../../cmd/backport-batch/README.md) and [Cherry-pick triage before a patch](../release/backport-mechanics.md#cherry-pick-triage-before-a-patch). Do not report a branch as ready to cut on the strength of a list of merged pull requests or of the batch pull request's description — they list what merged or was picked, not what arrived. diff --git a/docs/release/backport-tasks/README.md b/docs/release/backport-tasks/README.md new file mode 100644 index 000000000..eba2c3738 --- /dev/null +++ b/docs/release/backport-tasks/README.md @@ -0,0 +1,29 @@ +## Backport tasks + +The [backport batch](../backports.md) is built by [`backport-batch`](../../../cmd/backport-batch/README.md), and two steps around it need a maintainer's judgement. Each is a task with fixed inputs, one allowed output and a check in code that the result passes before anything is pushed. Both wait on the batch pull request: the batch's comment lists the deferred conflicts, and a failed batch CI shows on its checks. + +| Task | Input | Output | Check | +|---|---|---|---| +| [Resolve a conflict](./resolve-conflict.md) | a deferred conflict on the batch pull request | the original's commits picked under one merge on the batch branch, or one pick for a squash | `backport-batch fold`, and `backport-batch verify` in CI | +| [Attribute a failed batch CI](./failed-ci.md) | the failing jobs of the batch pull request | the culprit deferred to the next batch | `backport-batch run --defer` | + +### Rules for every task + +- Push nothing but the branch of the open batch pull request, `backport-batch/-` in this repository, and only what the task's check accepted. Never push to `main` or a `release-*` branch, and never open a second batch pull request: the scheduled run continues the open one. +- Change no file the original pull request does not change. A conflict that cannot be resolved inside those files is not resolved: the original stays deferred. +- Never rewrite the batch branch by hand. A plain push to it is a fast-forward; only `backport-batch run --defer --push` replaces it, leased on the head it rebuilt. +- Leave the batch pull request's title and body to the tool: it rewrites both on every run, and a hand edit is lost or, inside the `backport-batch` block, makes `check` refuse the body. +- When the task cannot be done within these rules, push nothing and leave a comment on the batch pull request saying why, so the next maintainer to look finds it. Nothing is lost by waiting: a red batch does not merge, and a deferred original goes to the next batch. + +### Getting the tool + +Build it from `main`, in a clean worktree of this repository whose `origin` is this repository and with `gh` authenticated as someone who can push to the batch branch: + +```bash +git fetch origin +git worktree add --detach ../backport-task origin/main +cd ../backport-task +go build -o /tmp/backport-batch ./cmd/backport-batch +``` + +Run it from that worktree. Every command but `notify` fetches `origin` first, and `run`, `check`, `fold` and `lines` read the policy from `origin/main`. diff --git a/docs/release/backport-tasks/failed-ci.md b/docs/release/backport-tasks/failed-ci.md new file mode 100644 index 000000000..e845eff99 --- /dev/null +++ b/docs/release/backport-tasks/failed-ci.md @@ -0,0 +1,39 @@ +## Attribute a failed batch CI + +CI on the batch pull request runs the line's build and tests against every original the batch carries at once. When a check fails, this task finds the original the failure comes from and takes it out of the batch, so the rest can merge; the original goes to the next batch with the reason recorded. Read the [rules for every task](./README.md#rules-for-every-task) first. + +### Inputs + +- The open batch pull request for the line and its failing checks: `gh pr checks `, and the log of each failing job, `gh run view --log-failed`. +- The originals the batch carries, one unit each on its first-parent line: `git log --first-parent --reverse --format='%h %s' origin/..origin/backport-batch/-`. A merged pull request's unit is a merge whose message is the original's merge message, naming its number, and whose `Backported-from:` trailer names its merge commit on `main`; its commits are `git log --reverse --format='%h %an %s' ^1..^2`. A squash-merged one is one commit whose subject ends in its number. The files a unit changes are `git diff --name-only ^1 `. +- The line's own CI on its head, `gh run list --branch `, to tell a failure the batch brought from one the line already had. + +### Output + +The batch rebuilt without the culprits, by one `backport-batch run --defer` with `--push`, which records each culprit and the reason in the batch pull request. Nothing else: no commit by hand, no revert, no edit to the batch branch. + +### Steps + +1. Read the failing jobs and name what fails: the job, the package or test, and the first error. +2. When the same job fails the same way on the line's head, or the failure is the runner's (a timeout pulling an image, a lost runner, a registry error), the batch did not cause it: defer nothing, and comment on the batch pull request with the job and the line's run that shows the same failure. +3. Otherwise pin it on originals by their files: the originals that change the failing package, chart or test, or what it builds from. Take the fewest that explain the failure; when no original's files reach the failure, or every original's do, the culprit cannot be pinned from files alone: read the failure further before deferring anything. +4. Write the reason as one line naming the job and the failure, for example `e2e-apps: postgres never becomes Ready`. + +### Check before pushing + +```bash +/tmp/backport-batch run --line --defer N[,N] --reason ": " +``` + +It rebuilds the branch locally without those originals and prints the new body, exiting 0. It exits 2 and names the original when dropping one leaves a later commit that no longer applies; add that one to `--defer` only when its files reach the failure too, otherwise comment on the batch pull request naming it and stop. Then publish the same command with `--push`: + +```bash +/tmp/backport-batch run --line --defer N[,N] --reason ": " --push +``` + +It replaces the batch branch only if nobody pushed onto it since the rebuild read it, and updates the pull request body. CI runs again on the new head; if it fails again on another original, repeat from step 1. + +### Never + +- Defer an original the failure does not reach, to make the batch green by taking out more than the culprit. +- Push the rebuilt branch any other way, or push to it by hand. diff --git a/docs/release/backport-tasks/resolve-conflict.md b/docs/release/backport-tasks/resolve-conflict.md new file mode 100644 index 000000000..928bc8966 --- /dev/null +++ b/docs/release/backport-tasks/resolve-conflict.md @@ -0,0 +1,66 @@ +## Resolve a conflict + +The batch carries a pull request merged into `main` as its own commits, each picked with `git cherry-pick -x`, under a merge commit that names the original; a squash-merged one as one pick. When any of an original's commits conflicts, nothing of it stays on the batch, and the batch pull request lists it under **Deferred conflicts** with the commit that failed and its files, and in its `backport-batch` block. This task ports one such original onto the batch by hand, in the same shape. Read the [rules for every task](./README.md#rules-for-every-task) first. + +### Inputs + +- The open batch pull request for the line, its branch `backport-batch/-` and its head commit, as GitHub shows them: `gh pr view --json headRefName,headRefOid`. +- The original's number `N` from the **Deferred conflicts** list, and its merge commit on `main`: `gh pr view N --json mergeCommit --jq .mergeCommit.oid`. It has two parents when the pull request was merged, one when it was squashed. +- For a merged pull request, its own commits, oldest first, without the merges that brought `main` into its branch: `git rev-list --reverse --topo-order --no-merges ^1..^2`. +- The files the original changes, `git diff --name-only ^1 `, which are the only files the resolution may touch. + +### Output + +For a merged pull request, one merge commit on the batch head: its first parent is the head of the batch pull request, and its second parent ends a straight line of picks of the pull request's own commits, in their order, each made with `git cherry-pick -x` so it keeps its author, message and sign-offs and ends in the `(cherry picked from commit …)` line git appends. Its message is the original's merge message with a `Backported-from: ` trailer of its own. For a squash-merged pull request, one pick of its commit. Either way only files the original changes, and nothing else: no extra commit, no fix-up of an earlier pick. + +### Steps + +```bash +git fetch origin +git switch --detach +base=$(git rev-parse HEAD) +``` + +Pick the original's commits, oldest first; for a squash-merged pull request the one commit to pick is its merge commit on `main`: + +```bash +git -c core.hooksPath=/dev/null cherry-pick -x --signoff +``` + +When a pick conflicts, resolve each conflicted file so that it does on the line what that commit does on `main`, keeping the line's own code wherever the commit does not change it, and go on with the next commit: + +```bash +git add +git -c core.hooksPath=/dev/null cherry-pick --continue --no-edit +``` + +When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. + +For a merged pull request, join the picks to the batch head with the merge commit: + +```bash +tip=$(git rev-parse HEAD) +git switch --detach "$base" +{ git log -1 --format=%B ; printf '\nBackported-from: %s\n' ; } > /tmp/batch-merge-message +git -c core.hooksPath=/dev/null merge --no-ff -F /tmp/batch-merge-message "$tip" +``` + +### Check before pushing + +```bash +/tmp/backport-batch fold --line --original N --commit "$(git rev-parse HEAD)" +``` + +It prints `ok` and exits 0, or prints why not and exits 1. Push only on `ok`, as a fast-forward of the batch branch: + +```bash +git push origin HEAD:refs/heads/backport-batch/- +``` + +Resolve one original at a time and push each before starting the next: the next fold has to sit on the new head. CI on the batch pull request runs again on the push, `backport-batch verify` among it, and the next run of the batch lists the original as on the branch, with the picks the resolution changed marked as such. + +### Never + +- Push a commit `fold` did not print `ok` for, or push anywhere but the batch branch. +- Squash the original's commits into one pick, reorder them, or pick a commit of another pull request into the merge. +- Edit a file outside the original's files to make the pick build; that is the dependency case above. diff --git a/docs/release/backports.md b/docs/release/backports.md new file mode 100644 index 000000000..c613cc980 --- /dev/null +++ b/docs/release/backports.md @@ -0,0 +1,112 @@ +## Backports + +Backports are automatic. Everything that can go into a patch release goes in, and nobody decides it pull request by pull request: nobody requests a backport, and contributors never open backport pull requests. The one exception is a maintainer override, below. A batch job takes the fixes merged into `main` and cherry-picks them onto each `release-X.Y` line it runs for, following the rules on this page. + +### Getting a fix into a patch release + +Merge it into `main` with a `fix` title, for example `fix(postgres): keep the replica count on resize`. Nothing else is needed: if the change meets the rules below, the next batch picks it up for every [line the batch runs for](#which-lines-the-batch-runs-for). + +The batch takes a pull request's type from its title alone, so a fix that ships inside a `feat` or `refactor` pull request does not reach a patch release. Put the fix in a pull request of its own. + +### What the batch takes + +The batch takes a pull request from `main` into release line X.Y when all of these hold: + +1. It merged into `main` after `release-X.Y` branched, or after the line started under this policy if that is later (see [Where each line starts](#where-each-line-starts)), and the line does not carry it yet. +2. Its title type is `fix` or `security`, the type of a fix that comes through a security advisory. CI and e2e fixes count, because they keep the line's own CI working. The title is read once any leading `[Backport …]` prefixes are dropped, and a title that does not read as a Conventional Commits title is kept out. +3. It is not breaking: no `!` after the type or scope, and no `BREAKING CHANGE:` or `BREAKING-CHANGE:` footer at the start of any line of the description its merge into `main` recorded. A footer inside an HTML comment, where the pull request template quotes it, does not count. +4. It touches no platform migrations, anything under `packages/core/platform/images/migrations/migrations/`, and no `targetVersion` line of `packages/core/platform/values.yaml`, adds, removes or edits no CustomResourceDefinition document (a CRD in a test fixture under `tests/` or `testdata/` does not count, and neither does an edit to another document in the same file), and makes no change to the API that the [API review gate](../../.github/workflows/api-review-gate.yaml) calls sizeable: a new or removed API group or resource, or a schema change that can reject what the old schema accepted. Those wait for the next minor; a reworded description or a new optional field under `api/` does not. [Backporting code that touches monotonic counters](./backport-mechanics.md#backporting-code-that-touches-monotonic-counters) shows what a migration on a release line breaks. +5. Each of its commits picks cleanly with `git cherry-pick -x`, or its conflict is resolved during the batch, and the batch's build and tests pass. A pick that fails moves to the next batch, and the batch pull request lists it with the files that conflicted. +6. On an older line, it is already on the next newer line the batch runs for. Lines are processed newest first, so an older line takes only what the newer one already carries. + +The batch reads the pull requests off `main`'s first-parent history, one per merge commit or squash commit that names its pull request. A commit there that names none, a direct push for instance, and a merge of more than two parents are never taken; the batch pull request counts them. + +### What waits for the next minor + +`feat`, `chore` and `docs` pull requests, and dependency bumps (scope `deps`), ship with the next minor, not in a patch release; only a maintainer's `backport/include`, below, takes one. If a class of fixes should also wait for a minor, it gets a new rule on this page; a single pull request that the rules get wrong is what the overrides below are for. + +### Maintainer overrides + +A maintainer overrides the rules for one pull request with a label on the merged pull request on `main`: + +- `backport/include` takes it although rules 2 to 4 keep it out, a `feat` for instance, or although it merged before its line started under this policy. It still has to pick cleanly or have its conflict resolved, pass CI on the batch, and, on an older line, already be on the newer one, and a line that already carries it does not take it again. +- `backport/skip` keeps it out although the rules take it, and `backport-batch check` stops asking for it. An open batch that picked it before the label came keeps it until `backport-batch run --defer` takes it out; the batch pull request says so meanwhile. + +`backport/skip` wins when a pull request carries both. Each run reads the labels afresh, and the batch pull request lists every override and what became of it. + +### Which lines the batch runs for + +The batch runs for the two newest lines that have shipped a stable release, and for every line branched after the newest of them that has not shipped one yet. A line has shipped once its first stable `vX.Y.Z` tag exists, which only the merge of its promotion creates ([stable tags come from rc promotion](./stable-tags.md)); an rc, alpha or beta tag does not count. How many released lines it keeps is `released-lines` in the block below. + +With 1.6 and 1.7 released, cutting `v1.8.0-rc.1` creates `release-1.8` ([release candidates](./release-candidates.md)), and from the next run the batch runs for 1.6, 1.7 and 1.8. Once `v1.8.0` ships it runs for 1.7 and 1.8, and 1.6 takes no more batches. Rule 6 holds across all of them, newest first: while 1.8 is in its rc window, a fix reaches 1.7 only after the 1.8 batch that carries it has merged, and 1.6 only after 1.7's. + +During its rc window a line is frozen: every later rc is cut from `release-X.Y` itself, and `main` is open for the next minor. The freeze closes the line to `main`, not to the batch, which is how fixes reach it: a batch pull request against a frozen line is built, checked and merged like any other, and the next rc carries what it merged once `check --cut` passes (see [Running the batch](#running-the-batch)). A line that drops out keeps its open batch pull request as it is, and `backport-batch` no longer runs or checks that line; a maintainer merges the pull request or closes it. + +### Where each line starts + +The policy applies in full from `release-1.7` on: a line released before it took effect keeps the patch content its users chose it with, so it takes only fixes merged after the policy started, and an older one only with `backport/include`. The batch reads the block below from this page, on `main`: a line it lists starts on that day, and any other line starts where it branched from `main`. The block also gives the `batch-only` day and `released-lines`. + +From the `batch-only` day in the block, a change from `main` reaches a line only through the batch, and only the batch's picks show that a line carries it: a pick whose `(cherry picked from commit …)` line, the one git appends, names the original's merge commit or every commit of its own. A picked commit keeps its message and a batch merge carries the original's, so text in them, a `Backport of #N`, a `Backported-from:` trailer or an `-x` line the commit's author wrote, counts only on commits from before that day, where it records the hand backports and earlier batches. + +```backport-policy +# release-X.Y: YYYY-MM-DD, the day the line started under this policy +release-1.6: 2026-10-26 +# batch-only: YYYY-MM-DD, the day from which only the batch's picks show what a line carries +batch-only: 2026-10-26 +# released-lines: N, how many of the newest lines with a stable release the batch runs for +released-lines: 2 +``` + +### Fixes that exist only on a release line + +Some fixes go to a release line as a pull request of their own. Such pull requests are occasional, and maintainers open them: for a fix with no counterpart on `main`, for one that on `main` came as part of another pull request, or for one written for `main` rather than for the line. When the counterpart on `main` is a pull request of its own that the batch would take, the batch skips it as already on the line once picking it would change nothing there; otherwise a maintainer labels it `backport/skip`. + +A pull request opened against a `release-*` branch is closed automatically, with a comment that links to this page, when its author is a user account, not a bot, that GitHub names neither the repository's owner nor a member nor a collaborator, and whose access to the repository GitHub reports as read or none. Anyone with triage access or more keeps their pull request open. When GitHub's answer on the author's access cannot be read, the check closes nothing and fails, naming the read that failed. The check runs only when a pull request is opened, so a maintainer keeps a contributor's release-only fix by reopening it. [`hack/close-manual-backport.sh`](../../hack/close-manual-backport.sh) does this from [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml); run it with `--pr --dry-run` to see what it would decide for a given pull request when it is opened. + +### Cadence + +The batch runs for each of its lines twice a week, on Monday and Thursday: often enough that a fix reaches its line within days, rarely enough that nobody spends time on it every day, and early enough in the week that each run leaves working days to cut an rc, run its e2e and promote it before the weekend. Patch releases ship weekly. + +### Running the batch + +[`backport-batch`](../../cmd/backport-batch/README.md) builds the batch in a clean worktree, newest line first. The scheduled run below does this; by hand it is: + +```bash +go build ./cmd/backport-batch +./backport-batch run --line release-1.7 --continue --push +./backport-batch run --line release-1.6 --continue --push +``` + +A run by hand trusts the open batch pull request the scheduled run opened as `cozystack-ci[bot]` and continues it; `BACKPORT_BATCH_BOT` names another bot, and set empty trusts none. + +`./backport-batch lines` prints the lines, newest first. `run`, `check`, `fold` and `lines` read the policy block from this page on `main`, so until the page with its block is on `main` they need `--policy ` naming a ref that has it, and the scheduled run, which passes none, fails. + +`run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. + +A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. + +Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried. + +### Changes to workflows + +The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. + +A maintainer brings it over with their own token, one that may push workflow changes (the Workflows permission, or the classic `workflow` scope), by continuing the open batch without the flag: + +```bash +./backport-batch run --line --continue --push +``` + +That picks it onto the batch in the batch's shape, after what the batch carries, and the body stops listing it; later scheduled runs continue the batch on top. To cut without it instead, take it out with `./backport-batch run --line --defer --reason --push`. + +### The scheduled run + +[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. + +The batch runs as the cozystack-ci GitHub App, which the release workflows already use, and needs no secret of its own. The job mints a token from `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY` for this repository alone, with Contents and Pull requests write and nothing more, and fails when either secret is missing. Its pushes start CI on the batch pull request, which a push with the job's own `GITHUB_TOKEN` would not. Its commits carry the App's bot, `cozystack-ci[bot]`, whose batch pull request and comment `backport-batch` trusts: `BACKPORT_BATCH_BOT`, which the job sets from the token, names it, and a run without the variable trusts `cozystack-ci[bot]` too. The App's installation has to grant this repository Contents and Pull requests read and write, which an organization admin sees among the organization's installed GitHub Apps. It has no Workflows permission, so a change to `.github/workflows/` waits for a maintainer, as [changes to workflows](#changes-to-workflows) describes. + +It also reads one repository setting, which a repository admin sets up: `BACKPORT_ALERT_TEAM`, a variable, the team or person the comment mentions, as `org/team` or a login. Unset, the comment mentions nobody: the batch pull request is a draft the App opened, so no one is notified of what waits unless they watch the repository. + +Optionally, once the `Backport batch shape` check runs from `main`, a ruleset makes it required on the release lines. It targets `release-*` and excludes `release-*.*.*`: `release-*` alone also matches the `release-X.Y.Z-rc.N` and `release-X.Y.Z` staging branches that [`tags.yaml`](../../.github/workflows/tags.yaml) and [`promote-rc.yaml`](../../.github/workflows/promote-rc.yaml) push, which have no batch on them, and a required check would refuse those pushes. "Do not require status checks on creation" stays on, since [`cut-prerelease.yaml`](../../.github/workflows/cut-prerelease.yaml) creates `release-X.Y` with a push at its first rc. + +To run the scheduled script for real by hand, export your own token as `BACKPORT_BATCH_TOKEN`: without `--dry-run` it uses that variable and no other credential. With a token that may push workflow changes, leave `--no-workflow-files` out and the run brings over what waits.