cozystack/cmd/backport-batch

backport-batch

Builds the automatic backport batch for a release line and gates a patch cut on it. What goes into a batch is decided by the rules of the backport policy alone; this page describes how the tool applies them and does not restate them.

Usage

go build ./cmd/backport-batch
./backport-batch run   --line release-1.7 [--onto SHA] [--date YYYY-MM-DD] [--no-workflow-files] [--push] [--policy REF]
./backport-batch run   --line release-1.7 --continue [--no-workflow-files] [--push] [--policy REF]
./backport-batch run   --line release-1.7 --defer 4321[,4322] --reason "fails the e2e of the app" [--push] [--policy REF]
./backport-batch run   --line release-1.7 --drop 4321[,4322] --reason "builds on the 1.8 storage rework" [--push] [--policy REF]
./backport-batch run   --line release-1.7 --undrop 4321[,4322] [--push] [--policy REF]
./backport-batch check --line release-1.7 [--cut] [--policy REF]
./backport-batch fold  --line release-1.7 --original 4321 --commit SHA [--policy REF]
./backport-batch verify --line release-1.7 [--head SHA]
./backport-batch notify --line release-1.7 [--mention org/team|login] [--push]
./backport-batch report --pr 5246 [--id ID] [--json]
./backport-batch lines [--policy REF]

hack/backport-batch-scheduled.sh runs lines, then run --continue --push, check and notify for each line; backport-batch.yaml runs it on schedule as the cozystack-ci App, with --no-workflow-files, which it hands to run.

Requires git and an authenticated gh on PATH, and a remote called origin. Every command but notify and report fetches origin first; notify reads only GitHub, and report reads GitHub and, for a report that keeps drops, the clone's history. The batch runs only for the lines lines prints.

run, check, fold and lines read the backport-policy block of the policy page from origin/main, or from the ref --policy names, which is how a change to the page is tried before it merges; verify and notify do not read it and take no --policy. A line the block lists takes nothing merged into main before the day it gives, unless the original carries backport/include; any other line starts at its fork point. A page without exactly one readable block stops the run with exit 2.

Build the binary wherever an exit code matters: go run turns any non-zero exit into its own 1, so under it a check that could not be completed looks the same as one that found something missing.

run

Switches the current working tree to backport-batch/<line>-<yyyymmdd>, created at origin/<line>, and picks onto it one original at a time, in the order they merged into main, as one unit each on the branch's first-parent line. A pull request merged into main with a merge commit M goes on as its own commits, git rev-list --reverse --topo-order --no-merges M^1..M^2, each picked with git cherry-pick -x --signoff, so each keeps its author, message and sign-offs and gains the (cherry picked from commit …) line; the merges that brought main into its branch are left out, since main already has what they brought. A merge commit then joins the picks to the batch: its first parent is the batch before the original, its second the last pick, and its message is M's with a paragraph of its own, Backported-from: M. A pull request squash-merged into main goes on as one pick. When any commit of an original conflicts, the pick is aborted, the branch goes back to where it was before that original, and the original is recorded with the commit that failed and its files; the run goes on with the next one. A commit the branch already carries is skipped as empty, and an original none of whose commits changes anything is reported as carried. The tree has to be clean. A local branch of that name is reset to where the run starts; when that drops commits only it has, the run first prints its tip and the git branch command that restores them, and so does --defer. Signing follows your git configuration; hooks are off for the picks and the merge, so each commit carries its message as git writes it.

The pull request body goes to stdout, built from the line's own .github/PULL_REQUEST_TEMPLATE.md: the originals on the branch, each with how many commits it brought and which of them differ from the commit they were picked from by git patch-id, a resolution or a pick git had to fit, the deferred conflicts with the commit that failed and its files, the skipped pull requests counted by rule with the lists collapsed, the originals' release notes, and two blocks of JSON: the line's record in backport-batch, which later runs, check and notify act on, and the run's report in backport-batch-report. The title is fix(backport): batch for <line>, <date>. A body that would pass GitHub's 65536-byte limit leaves out its longest skipped lists, and the run prints them on stderr instead; one that what it has to carry still puts over the limit is refused before anything is pushed.

--onto SHA continues a batch already pushed instead of starting a new one. SHA has to be the head the remote has for a batch branch of the line right now, read with git ls-remote, so a head someone pushed onto since is refused. What that branch carries stays, and only what it does not carry yet is picked on top, so the next push is a fast-forward. --continue takes SHA from the open batch pull request, and starts a new batch when none is open, which is what the scheduled run does.

--no-workflow-files is for a token that may not push a change to .github/workflows/, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the backport-batch block records it with "workflows": true, so check, notify and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one.

--defer N[,N] --reason TEXT takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off origin/<line> without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got backport/skip after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the backport-batch block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and --push closes the batch pull request instead, as the line's record describes.

--drop N[,N] --reason TEXT drops originals from the line: where --defer holds one out of this batch, a drop keeps it out of every later batch of the line too, until --undrop N[,N] withdraws it. A drop is recorded in the line's record, so both need an open batch pull request that carries one, and exit 2 without it; once a batch merges, its drops are carried into the next batch's record. A dropped original the open batch carries is taken off it as --defer would, and the rebuild is pushed the same way; any other drop or undrop changes only the record, so the batch stays as it is even when the line moved since. An undropped original is missing until the next run tries it. A drop names the original's merge commit on main as well as its number, and a record whose drop does not match what that merge commit merged stops every command with exit 2. An original the line already carries cannot be dropped, and neither can a number that merged into main before the line branched. The order is the rules', then the drop: what backport/skip, rules 2 to 4 or the line itself decide comes first, with the drop kept on record beside it, and backport/include does not lift a drop. The body lists dropped originals under their own heading among the skipped ones, check prints a note for each, and fold refuses one.

--push pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with gh pr create --draft, or updates the open batch pull request with gh pr edit, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the run --onto <its head> --push that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no backport-batch block either: someone wrote that body by hand, and the batch is theirs to merge or close. After --defer alone it replaces the branch, with --force-with-lease=refs/heads/<branch>:<head>, where head is what git ls-remote reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. A record that says the batch is being closed is acted on before anything else, and only by run --push.

check

Exits 1 and prints the URL of every pull request the rules take that is neither on the line nor on the open batch pull request, 0 when there is none, and 2 when it cannot tell. A pull request that the line's record lists as deferred, a conflict or one taken out with --defer, prints as a warning and does not fail the check: deferring is a maintainer's call. That holds once the batch that recorded it merged or closed too. So does one the open batch lists as waiting for a maintainer because it changes workflows; once that batch is gone, nothing lists it any more, and it is missing. One a maintainer dropped from the line prints as a note, #N is dropped from <line> on #M (<reason>).

--cut makes it the gate before a patch release or an rc is cut from a line. A cut tags the line, so a pull request that is only on the open batch pull request fails it too, with one line each: #N is only on open batch PR #M, merge it before the cut. One that waits there for a maintainer fails it as well, since nobody decided to leave it out, also when it is all the batch has left: bring it over, or take it out with run --defer and a reason. Everything else is as without it, and a cut that passes says no change the policy takes is missing before the cut. The scheduled run checks without --cut, since its batch is open by design.

An original with backport/skip that the open batch picked before the label came prints a warning in either mode, naming the batch and run --defer, which takes it out; it fails neither.

fold

Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request --commit is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one Backported-from: trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the (cherry picked from commit …) line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose (cherry picked from commit …) line names the original's commit on main. Either has to change something, and only files the original changes. The conflict task gives the commands. fold prints ok and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head.

verify

Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from origin/<line> to --head, or to the head of the open batch pull request, and requires each unit to be one original, merged into main after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With --head it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; pr-labeler.yaml runs verify on every push to a batch pull request, from a build of main, with the batch head fetched as data.

notify

Reads the line's record first, as the other commands do, and exits 2 when it cannot read it or when it says the open batch is being closed, with a batch open or not. Then it reads the open batch pull request of the line, trusted as described below, and prints one line for each thing that waits on it: a batch open for three days or more, the shortest gap between two scheduled runs, so one the previous run already saw; the conflicts it deferred; and the originals waiting for a maintainer because they change workflows. Nothing open prints nothing.

With --push it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the conflict task describes, and the originals that wait because they change workflows.

The comment starts with <!-- backport-batch:notify --> and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions --mention, an org/team or a login, with or without @, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing.

report

Prints a report a batch pull request keeps: the one its body carries, or the one --id names. It reads the report comments the batch trusts and falls back to the body, recomputes the report's ID from what it says, and exits 2 when a trusted comment does not match its ID, when a report says anything a run does not write or is for another line than the pull request's, or when the pull request keeps no such report. A report comment counts only when its author is one the batch trusts with a record, so anyone else's comment, whatever ID it carries, is not read. It prints what the classification was made of, whether the pull request's head or body shows the report was published, which it calls unconfirmed when neither does, and one line per original with its category and what the report recorded about it; --json prints the report as stored. A report whose comment the run never got to post is printed from the body, with a note that the next run --push posts it, or, on a merged or closed batch, that it is kept in the body only. It writes nothing. It reads GitHub, and for a report that keeps drops also the clone's history, without fetching: each drop is bound to main as the record's drops are, over the heads of main and the line the report was made from, so a drop of an original the report classified has to name that original's merge, and any other a merge of its number on main's first-parent line. A clone without those heads exits 2, asking for a fetch.

lines

Prints the release lines the batch runs for, newest first: of the release-X.Y branches on origin, the newest that have a stable vX.Y.Z tag there, as many as released-lines in the policy block says, two when it says nothing, and every branch newer than the newest of them. The tags are read with git ls-remote; a tag with a pre-release suffix does not count. run, check and fold refuse any other line, and rule 6 holds each line to the next newer one among them.

Rule 4 and the API review gate

A change to the API is held back only when the API review gate would call it sizeable. The gate is asked whenever an original changes something it parses: pkg/apiserver/apiserver.go, an ApplicationDefinition under packages/system/*-rd/cozyrds/, or a .yaml or .yml file under packages/ or internal/ that holds CustomResourceDefinition or APIService before or after the merge. An edit to the Go types under api/ alone reaches none of them, so the gate would see no change. The batch calls the gate's own code, apigate.LoadSnapshot and apigate.Classify from internal/apigate, on the trees of the original's first parent and of its merge commit. It reads only what the gate reads, from git ls-tree and git cat-file into a temporary directory, so no attribute or export rule changes what the gate sees. A reworded description or a new optional field passes; a new or removed group or resource, or a schema change that can reject what the old schema accepted, does not. Migrations, targetVersion and CRDs keep their own checks: the gate sees only first-party cozystack.io CRDs, and only their sizeable changes, while a vendored operator's CRD is just as unsafe in a patch.

The CRD check works document by document. In every changed manifest, a YAML, JSON or .tpl file or anything under templates/ or crds/, outside tests/ and testdata/, every document that names CustomResourceDefinition is read as YAML, which takes JSON too, and the CRDs among them are compared between the parent and the merge by metadata.name, compact JSON and quoted keys included. The change counts only when one was added, removed or edited, so an edit to the DaemonSet beside a CRD in one manifest does not, and neither does a sentence or a lookup call that merely names the kind. A document that declares kind: CustomResourceDefinition on a line of its own but cannot be read as an object, a CRD behind a Helm template guard for one, is compared by its text, so the check fails closed on it, and so does a List that holds a CRD.

Maintainer overrides

run, check and fold read the merged pull requests that carry backport/include or backport/skip, through GitHub search, which answers at most 1000 per label; a listing that long is an error rather than a complete answer. backport/skip keeps a pull request out whatever the rules say, and wins over backport/include, which lifts rules 2 to 4 and nothing else: what the line already carries and, on the older line, what the newer one does not carry yet still decide. The batch pull request lists every overridden original and what became of it. Adding a label takes triage access to the repository, which is the trust the labels rest on; who added one is not checked.

What counts as already there

An original is on a line when any of these holds, and the batch does not pick it again:

• its merge commit is reachable from the line;
• picks on the line name its merge commit, or every commit of its own, in the (cherry picked from commit …) line git appended to them;
• picking it onto the line would change nothing.

A pick is known by the line git appends: the last (cherry picked from commit …) line of the message's last paragraph, with nothing after it but trailers such as a sign-off. A picked commit keeps its message, and a batch merge carries the original's merge message, so whatever else a message says was written by someone the batch does not answer for: a line in it naming an original, as an -x line, a Backported-from: trailer or Backport of #N, does not count. On commits from before the policy's batch-only day it does, since those record the hand backports and earlier batches:

• a (cherry picked from commit …) line anywhere in the trailer block naming its merge commit or, across commits, every commit of its own;
• a Backported-from: trailer naming its merge commit;
• Backport of #N, which is what a hand backport merged as one commit carries.

An original is on the open batch pull request when its commits picked there show it the same way. The same rules on the newer line decide rule 6.

Only the line's own commits count, the ones after the point where it branched from main, never a pull request against it: an open backport pull request, its title and its body are anyone's to write. The tool trusts the line's history because only maintainers merge into release-*; if that ever changes, a commit that names an original without carrying all of it would hide the rest. The batch pull request counts only when it comes from a backport-batch/<line>-<yyyymmdd> branch of this repository, not a fork, and was opened by the bot the scheduled batch runs as, the exact login BACKPORT_BATCH_BOT names, cozystack-ci[bot] when it is unset and no bot when it is set empty, or by someone whose permission on the repository GitHub reports as write or admin, and only at the head GitHub reports for it. No other bot counts, GitHub Actions' own included, since anything holding a workflow's token writes as it. So a maintainer's run by hand continues the scheduled batch with nothing set, rather than taking its pull request for a look-alike and opening a second one. The notify comment is trusted the same way. A permission GitHub does not answer stops the command with exit 2 rather than count the pull request or the comment out, which would open a second of either. The author association is not enough: an organization member or an invited collaborator can hold read access only. With more than one such pull request open the tool refuses to go on.

Titles and bodies are read from the merge commits on main, which hold them as a maintainer merged them; the live pull request stays editable by its author.

The line's record and reports

The line's record is the backport-batch block of its newest batch pull request: the newest pull request against the line from a backport-batch/<line>-<yyyymmdd> branch of this repository, opened by an author the batch trusts as above, whether it is open, merged or closed. run, check, fold and notify find it in GitHub's list of the line's pull requests, newest first, reading at most ten pages of 100. A batch pull request a person opened without the block is passed over. A block whose author the batch does not trust, a bot's batch without its block, a block that cannot be read whole, a closed record newer than the open batch, or ten pages with no record stop the command with exit 2, naming the pull request, rather than fall back to an older record.

The record is written only while its pull request is open: a merged or closed batch pull request is never edited or commented on again. What it recorded keeps counting, so a conflict or a deferral a merged batch recorded still prints as a warning. A workflow original it listed as waiting is missing, since no open batch lists it. A run that picks nothing while no batch is open has nowhere to write: it prints the body, says that its classification is not retained, and exits 0.

The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. A reason or a file name in the record counts only in the form a run writes it, one line with no control, zero-width or bidi character, so text someone typed into the body never reaches a terminal, a body or a comment as they typed it. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost.

The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a backport-batch-report block: every original merged into main since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on main that merged no pull request, the labels, and what the classification was made of: the heads of main, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: <!-- backport-batch:report <id> --> and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by report, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next run --push on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report.

A --defer that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with --force-with-lease on it, printing closed <url>. The next batch tries what was deferred again. A close that stops part way is finished by the next run --push, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for --defer, --drop or --undrop, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. run without --push, check and fold stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and notify says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed.

One way for the branch to move is a new batch of the same day: it reuses the branch name, and if its push succeeds and opening its pull request fails, the closed batch is still the newest record while its branch now holds the new batch. The next run stops on it, naming the branch, and so does --onto with its head. Check what it carries before deleting it with git push origin --delete backport-batch/<line>-<yyyymmdd>: the next run picks again what the tool put there from the line, but not a commit someone pushed onto it, such as a resolved conflict. Separate writes to GitHub are not atomic: the re-reads and the leases refuse a race they see, but cannot prevent one.

What CI verifies

The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and verify against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with fold and pushes it, and then CI runs again.

Untrusted text

Titles, release notes and file names in the body come from whoever opened the originals, and a reason from whoever ran --defer. A title, a reason or a file name goes into a code span whose fence no backtick inside can close, and the release notes go into the release-note block with every run of three backticks or tildes defused, so GitHub acts on no mention, reference, link or HTML in them. Inside that block, in case a note is copied out of it, links lose any target outside this repository, and closing keywords, cross-repository references and mentions have their @, # and :// swapped for fullwidth signs, which act on nothing however the backticks around them pair up. The record and the report are one line of JSON each, whose encoding leaves no newline, <, > or & in them, so nothing in a file name or a reason ends their block or opens HTML, and a report comment names no one: numbers are bare, and its only text outside the block is the line, the date and the report's ID. Zero-width, bidi and control characters are dropped first, and everything the tool prints passes the same filter, so nothing it relays can drive a terminal. Commit messages are read through git cat-file --batch, which gives each object's length, and end at their first NUL as git log shows them, so a message cannot pass for a second commit or hide a trailer from a reader of the log. Every value handed to git comes after --end-of-options or --, refs pass git check-ref-format, SHAs have to be 40 hex digits, and nothing runs through a shell.