From 79b335e461797d411caee4456caf3d61ff2d8d8f Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Mon, 9 Nov 2026 15:21:02 +0000 Subject: [PATCH] fix(release): close the batch --defer leaves empty, recording why while it is open docs/release/backport-tasks/failed-ci.md says to defer the culprit with run --defer --push. When the culprit was all the batch carried, the push stopped with "close batch pull request #N instead", and closing it by hand left nothing that said why: once closed, the deferred original was simply missing. The run now closes the batch itself, and does every write while the pull request is open: the report, then the body, whose record keeps the deferrals and says the batch is closing, at the head the rebuild started from. The close and the deletion of the branch go only while GitHub and the remote still show that head, and the deletion is leased on it. A close that stops part way is finished by the next run --push, before anything else, and by no other command; a head the close did not freeze is refused, never adopted. The run that finishes it stops there, and fails when it was asked for --defer, since no open batch is left to defer on. Recovery never writes the closed pull request. A same-day batch that reuses the branch name and fails to open its pull request leaves a branch the next run names for a person to delete. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 8 +- cmd/backport-batch/assemble.go | 163 +++++++++++- cmd/backport-batch/body.go | 3 + cmd/backport-batch/check.go | 28 +++ cmd/backport-batch/defer_test.go | 306 +++++++++++++++++++++++ cmd/backport-batch/fixture_test.go | 30 ++- cmd/backport-batch/github.go | 6 + cmd/backport-batch/main.go | 3 +- cmd/backport-batch/overrides_test.go | 7 + cmd/backport-batch/record_test.go | 11 + cmd/backport-batch/report_test.go | 9 - docs/release/backport-mechanics.md | 2 +- docs/release/backport-tasks/failed-ci.md | 2 + docs/release/backports.md | 2 +- 14 files changed, 560 insertions(+), 20 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index 7a1acabf1..768f21b2f 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -35,9 +35,9 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--no-workflow-files` is for a token that may not push a change to `.github/workflows/`, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the `backport-batch` block records it with `"workflows": true`, so `check`, `notify` and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one. -`--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. +`--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and `--push` closes the batch pull request instead, as [the line's record](#the-lines-record-and-reports) describes. -`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. +`--push` pushes the branch, refusing anything but a fast-forward of what the remote has, then opens the draft pull request against the line with `gh pr create --draft`, or updates the open batch pull request with `gh pr edit`, and prints its URL on stderr. On the open batch it first posts the run's report as a comment, so a body never names a report nobody can read; a new batch carries the report in the body it opens with, and the comment follows. Before every write to the pull request it reads the pull request again, and stops with exit 2 once it is merged or closed. When the branch it refuses to push over has no pull request open on it, as a run that stopped between its push and opening the pull request leaves it, the refusal names the `run --onto --push` that continues it. While a batch pull request is open on another branch it refuses, so a line never has two. It never pushes to or edits a batch pull request whose body has no `backport-batch` block either: someone wrote that body by hand, and the batch is theirs to merge or close. After `--defer` alone it replaces the branch, with `--force-with-lease=refs/heads/:`, where head is what `git ls-remote` reads for the branch and has to be the head the rebuild started from: a commit pushed since is never dropped, by the tool's check or, if one lands between that check and the push, by the remote. A record that says the batch is being closed is acted on before anything else, and only by `run --push`. ### check @@ -111,6 +111,10 @@ The record is only as trustworthy as the body it sits in. Anyone with write acce The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. +A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. + +One way for the branch to move is a new batch of the same day: it reuses the branch name, and if its push succeeds and opening its pull request fails, the closed batch is still the newest record while its branch now holds the new batch. The next run stops on it, naming the branch, and so does `--onto` with its head. Check what it carries before deleting it with `git push origin --delete backport-batch/-`: the next run picks again what the tool put there from the line, but not a commit someone pushed onto it, such as a resolved conflict. Separate writes to GitHub are not atomic: the re-reads and the leases refuse a race they see, but cannot prevent one. + ## What CI verifies The tool does not build or test anything. CI on the batch pull request runs the line's own build and tests against what was picked, and `verify` against its shape, and a maintainer reviews and merges it. A deferred conflict waits for the next batch unless someone resolves it, checks the resolution with `fold` and pushes it, and then CI runs again. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index a2adc5c1d..3ac19c4f9 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -119,6 +119,9 @@ func (a *app) run(args []string) int { if err != nil { return a.fail(err) } + if goOn, code := a.resumeClose(b, *push, deferred != nil); !goOn { + return code + } if deferred != nil { return a.rebuild(b, deferred, reason, *push) } @@ -266,6 +269,13 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r if r.shapes, err = summarize(b, head, r.onBranch); err != nil { return a.fail(err) } + if rebuilt && head == b.lineHead && b.pr != nil { + // Deferring everything the batch carries leaves nothing to push: the + // batch is closed, and its record says so first, with the head the + // close is leased on. + r.closing = &closing{PR: b.pr.Number, Branch: branch, Head: b.prHead} + r.opts.Mode = modeClose + } r.built = builtFrom{Main: b.mainHead, Line: b.lineHead, Newer: b.newerHead, Policy: b.policyAt, Batch: head, Tool: a.tool, Run: runURL(os.LookupEnv), At: a.now().UTC().Format(time.RFC3339), Options: r.opts} doc, err := buildReport(b, r) @@ -321,7 +331,7 @@ func (a *app) finish(b *batch, r *report, branch string, date time.Time, push, r // act on, with the full classification left to the report it names. func (r *report) lineRecord(doc reportDoc) lineRecord { built := doc.Built - return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} + return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: []droppedEntry{}, Stale: slices.Clone(doc.Overlays.Stale)} } // branchDate is the date a batch branch was opened under. @@ -496,8 +506,8 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { return fmt.Errorf("batch pull request #%d has no %s block, so backport-batch did not write it; merge or close it, and the next run opens a batch of its own", b.pr.Number, deferredFence) } if p.head == b.lineHead { - if rebuilt { - return fmt.Errorf("without what is deferred, %s carries nothing beyond %s; close batch pull request #%d instead", p.branch, b.line.branch(), b.pr.Number) + if rebuilt && p.record.Closing != nil { + return a.closeBatch(b, p) } return fmt.Errorf("%s carries nothing beyond %s; nothing to push", p.branch, b.line.branch()) } @@ -528,7 +538,13 @@ func (a *app) publish(b *batch, p publication, rebuilt bool) error { refusal := fmt.Sprintf("%s on %s is at %.12s, which %.12s does not descend from; refusing a non-fast-forward push", p.branch, remote, pushed, p.head) // A run that stopped between its push and opening the pull request // leaves the branch, and the next run that day builds it anew. - if b.pr == nil { + switch c := b.src.carrier; { + case b.pr != nil: + case c != nil && !b.src.open && c.Head.Ref == p.branch: + // Continuing a batch that merged or closed would publish + // what nobody can merge any more. + refusal += fmt.Sprintf("; it is the branch of batch pull request #%d, which is %s; the next run picks again what the tool put on it, but not a commit someone pushed there, such as a resolved conflict: check what %.12s carries, then delete it with git push origin --delete %s and run again", c.Number, prState(c), pushed, p.branch) + default: refusal += fmt.Sprintf("; no batch pull request is open on it, and backport-batch run --line %s --onto %s --push continues it and opens one", b.line.branch(), pushed) } return errors.New(refusal) @@ -636,6 +652,140 @@ func pullURL(pr *pullRequest) string { return fmt.Sprintf("%spull/%d", ourRepoURL, pr.Number) } +// closeBatch closes the open batch that --defer left carrying nothing. Every +// write happens while it is open: the report, then the body whose record +// says it is closing and at which head. The close and the deletion of its +// branch then go only while both are still at that head. +func (a *app) closeBatch(b *batch, p publication) error { + c := *p.record.Closing + if err := p.fits(); err != nil { + return err + } + if err := a.keepReports(b, &p); err != nil { + return err + } + name, err := writeTemp("backport-batch-body-*.md", p.body) + if err != nil { + return err + } + defer os.Remove(name) + if err := a.stillOpen(c.PR); err != nil { + return err + } + if err := a.gh.editPull(c.PR, p.title, name); err != nil { + return fmt.Errorf("recording the close of #%d: %w", c.PR, err) + } + return a.finishClose(b.line, pullURL(b.pr), c) +} + +// finishClose closes the batch pull request a record says is closing, and +// deletes its branch, each only while it is at the head the record froze. +func (a *app) finishClose(l line, url string, c closing) error { + pr, err := a.gh.pull(c.PR) + if err != nil { + return fmt.Errorf("closing #%d, reading it again: %w", c.PR, err) + } + if st := prState(&pr); st != "open" || pr.Head.SHA != c.Head { + return fmt.Errorf("closing #%d: it is %s at %.12s, not open at %.12s, which its record froze", c.PR, st, pr.Head.SHA, c.Head) + } + heads, err := remoteBatchHeads(l) + if err != nil { + return err + } + if h := heads[c.Branch]; h != c.Head { + return fmt.Errorf("closing #%d: %s on %s is at %.12s, not at %.12s, which its record froze", c.PR, c.Branch, remote, h, c.Head) + } + if err := a.gh.closePull(c.PR); err != nil { + return fmt.Errorf("closing #%d: %w; backport-batch run --line %s --push finishes it", c.PR, err, l.branch()) + } + if err := a.deleteBranch(l, c); err != nil { + return err + } + fmt.Fprintf(a.stderr, "closed %s: without what is deferred it carries nothing beyond %s; the next batch tries them again\n", url, l.branch()) + return nil +} + +// deleteBranch deletes the branch of a closed batch, leased on the head it +// was closed at: one someone pushed since is a person's to delete or keep. +func (a *app) deleteBranch(l line, c closing) error { + heads, err := remoteBatchHeads(l) + if err != nil { + return err + } + switch h, ok := heads[c.Branch]; { + case !ok: + return nil + case h != c.Head: + return movedBranch(c, h) + } + if _, err := git(deleteArgs(c.Branch, c.Head)...); err != nil { + return fmt.Errorf("deleting %s, left by the close of #%d: %w; backport-batch run --line %s --push deletes it", c.Branch, c.PR, err, l.branch()) + } + return nil +} + +// takenBranch is the refusal for a branch a close left that an open batch +// pull request now uses, whatever it holds, or nil. +func takenBranch(b *batch, c closing) error { + if b.pr == nil || b.pr.Head.Ref != c.Branch { + return nil + } + return fmt.Errorf("batch pull request #%d is open on %s, which the close of #%d left; the branch is that pull request's now, and the close leaves it alone", b.pr.Number, c.Branch, c.PR) +} + +func movedBranch(c closing, h string) error { + return fmt.Errorf("batch pull request #%d was closed at %.12s, but %s is at %.12s now; the next run picks again what the tool put on it, but not a commit someone pushed there, such as a resolved conflict: check what %.12s carries, then delete it with git push origin --delete %s, and run again", c.PR, c.Head, c.Branch, h, h, c.Branch) +} + +// resumeClose finishes a close the record says is under way, before a run +// does anything else, and only under --push. It says whether the run may go +// on: only once the batch is closed and its branch gone. A run that finishes +// one stops there, and fails when it was asked to change the batch. +func (a *app) resumeClose(b *batch, push, changing bool) (bool, int) { + if b.src.rec == nil || b.src.rec.Closing == nil { + return true, 0 + } + c := *b.src.rec.Closing + heads, err := remoteBatchHeads(b.line) + if err != nil { + return false, a.fail(err) + } + h, onRemote := heads[c.Branch] + switch { + case b.src.open && h != c.Head: + return false, a.fail(fmt.Errorf("batch pull request #%d is being closed, but %s is at %.12s, not at %.12s, which its record froze; a person closes it, or pushes the branch back, by hand", c.PR, c.Branch, h, c.Head)) + case b.src.open && !push: + return false, a.fail(fmt.Errorf("batch pull request #%d is being closed: closing it and deleting %s are left; backport-batch run --line %s --push finishes it", c.PR, c.Branch, b.line.branch())) + case b.src.open: + if err := a.finishClose(b.line, pullURL(b.src.carrier), c); err != nil { + return false, a.fail(err) + } + return false, a.finished(c, changing) + case !onRemote: + return true, 0 + case takenBranch(b, c) != nil: + return false, a.fail(takenBranch(b, c)) + case h != c.Head: + return false, a.fail(movedBranch(c, h)) + case !push: + return false, a.fail(fmt.Errorf("batch pull request #%d was closed, but %s is still at %.12s; backport-batch run --line %s --push deletes it", c.PR, c.Branch, h, b.line.branch())) + } + if err := a.deleteBranch(b.line, c); err != nil { + return false, a.fail(err) + } + fmt.Fprintf(a.stderr, "deleted %s, which the close of #%d left\n", c.Branch, c.PR) + return false, a.finished(c, changing) +} + +// finished is the exit of a run that finished a close: a change it was asked +// for needs an open batch pull request, and none is left to make it on. +func (a *app) finished(c closing, changing bool) int { + if !changing { + return 0 + } + return a.fail(fmt.Errorf("finished the close of #%d and made no other change: what this run was asked to change needs an open batch pull request, which the next run opens", c.PR)) +} + // pushArgs pushes head to branch: as a fast-forward, which the remote checks, // or, given a lease, over exactly that commit and no other. func pushArgs(branch, head, lease string) []string { @@ -645,3 +795,8 @@ func pushArgs(branch, head, lease string) []string { } return append(args, "--end-of-options", remote, head+":refs/heads/"+branch) } + +// deleteArgs deletes branch, only while the remote has it at lease. +func deleteArgs(branch, lease string) []string { + return []string{"push", "--quiet", "--force-with-lease=refs/heads/" + branch + ":" + lease, "--end-of-options", remote, ":refs/heads/" + branch} +} diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index ae34be876..1bd2024f7 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -248,6 +248,9 @@ type report struct { built builtFrom // blocks are the record and the report the body ends with. blocks string + // closing is the close of the open batch this run leaves carrying + // nothing. + closing *closing } // ruleName is how output names what kept an original out: the policy's diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index 9398a37e6..d7bc7e213 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -47,6 +47,9 @@ func (a *app) check(args []string) int { if err != nil { return a.fail(err) } + if err := a.warnLeftBranch(b); err != nil { + return a.fail(err) + } disp, err := b.dispositions(*cut) if err != nil { return a.fail(err) @@ -90,6 +93,28 @@ func (a *app) check(args []string) int { return 1 } +// warnLeftBranch names the branch a closed batch's close left behind. +func (a *app) warnLeftBranch(b *batch) error { + if b.src.open || b.src.rec == nil || b.src.rec.Closing == nil { + return nil + } + c := b.src.rec.Closing + heads, err := remoteBatchHeads(b.line) + if err != nil { + return err + } + switch h, ok := heads[c.Branch]; { + case !ok: + case takenBranch(b, *c) != nil: + a.warn("%v", takenBranch(b, *c)) + case h == c.Head: + a.warn("batch pull request #%d is closed, but its branch %s is still at %.12s; backport-batch run --line %s --push deletes it", c.PR, c.Branch, h, b.line.branch()) + default: + a.warn("%v", movedBranch(*c, h)) + } + return nil +} + // outcome is what check makes of one original. type outcome int @@ -196,6 +221,9 @@ func (a *app) fold(args []string) int { if c := b.src.closingOpen(); c != nil { return a.fail(fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, l.branch())) } + if err := a.warnLeftBranch(b); err != nil { + return a.fail(err) + } why, err := refuseFold(b, *number, *commit) if err != nil { return a.fail(err) diff --git a/cmd/backport-batch/defer_test.go b/cmd/backport-batch/defer_test.go index 59c736083..d0f8798c8 100644 --- a/cmd/backport-batch/defer_test.go +++ b/cmd/backport-batch/defer_test.go @@ -378,3 +378,309 @@ func TestSkipAfterThePick(t *testing.T) { t.Errorf("check = %d\n%s", code, stderr) } } + +// closeWorld is a batch carrying #5 alone, opened and pushed. +func closeWorld(t *testing.T) (*fixture, *fakeGH, string) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + gh := &fakeGH{} + h1 := openBatch(t, f, gh) + return f, gh, h1 +} + +// Deferring everything a batch carries closes it: the record says why and +// what it froze while the pull request is still open, then the pull request +// is closed and its branch deleted, leased on that head. The next batch +// tries what was deferred again. +func TestDeferEverythingClosesTheBatch(t *testing.T) { + f, gh, h1 := closeWorld(t) + var log []string + gh.after = map[string]func(){ + "createComment": func() { log = append(log, "comment") }, + "editPull": func() { log = append(log, "edit") }, + "closePull": func() { log = append(log, "close, branch at "+f.remoteHead(batchBranch1)) }, + } + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push") + if code != 0 || !strings.Contains(stderr, "closed https://github.com/cozystack/cozystack/pull/90: without what is deferred it carries nothing beyond release-1.1; the next batch tries them again") { + t.Fatalf("run --defer of the last pick = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if want := []string{"comment", "edit", "close, branch at " + h1}; !slices.Equal(log, want) || f.remoteHead(batchBranch1) != "" || gh.find(90).State != "closed" { + t.Errorf("writes %q, branch at %q, #90 %s; want %q, the branch deleted and #90 closed", log, f.remoteHead(batchBranch1), gh.find(90).State, want) + } + rec, err := parseRecord(gh.find(90).Body, notifyLine) + if err != nil || rec.Closing == nil || *rec.Closing != (closing{PR: 90, Branch: batchBranch1, Head: h1}) || rec.Built.Options.Mode != modeClose || rec.deferred()[5].Reason != "fails the app e2e" { + t.Errorf("the closed batch's record is %+v, %v; want #5 deferred and the close it froze", rec, err) + } + if !strings.Contains(stdout, `"closing":{"pr":90,"branch":"`+batchBranch1+`","head":"`+h1+`"}`) { + t.Errorf("the body printed does not say what the close froze:\n%s", stdout) + } + code, _, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: deferred on #90 (closed) to the next batch: https://github.com/cozystack/cozystack/pull/5 ") { + t.Errorf("check = %d, want 0 with #5 deferred:\n%s", code, stderr) + } + gh.nextPR = 91 + mustRun(t, f, gh, "run", "--line", "release-1.1", "--continue", "--push") + if got := f.subjects("origin/release-1.1", f.remoteHead(batchBranch1)); !slices.Equal(got, []string{"fix(app): five (#5)"}) || gh.find(91) == nil { + t.Errorf("the next batch carries %q, want #5 tried again on a new pull request", got) + } +} + +// A close that stopped part way is finished by run --push, and by nothing +// else, and never past a head it did not freeze; nothing is ever written to +// the closed pull request. +func TestCloseRecovery(t *testing.T) { + deferAll := []string{"run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the app e2e", "--push"} + push := []string{"run", "--line", "release-1.1", "--continue", "--push"} + // moveBranch pushes a commit onto the batch branch, or recreates it, as + // someone else would. + moveBranch := func(f *fixture, base string) string { + f.git("switch", "-q", "--detach", base) + f.git("commit", "-q", "--allow-empty", "-m", "pushed since") + h2 := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + return h2 + } + writes := func(gh *fakeGH) int { return len(gh.issueComments) + len(gh.edited) } + + t.Run("stopped after the record, then the branch moved", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"editPull": func() { gh.fail = map[string]error{"pull": errString("HTTP 502")} }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 || !strings.Contains(stderr, "HTTP 502") { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + h2 := moveBranch(f, h1) + before := writes(gh) + if code, _, stderr := f.exec(gh, push...); code != 2 || !strings.Contains(stderr, h2[:12]) { + t.Errorf("run --push over a moved branch = %d, want 2 naming it:\n%s", code, stderr) + } + if gh.find(90).State != "open" || f.remoteHead(batchBranch1) != h2 || writes(gh) != before { + t.Error("the run acted on a branch that moved since the close froze it") + } + }) + // GitHub and the remote are read apart, and either may be the first to + // show that someone pushed while the record was written. + for _, seen := range []string{"by the remote", "by GitHub"} { + t.Run("the branch moved while the record was written, seen "+seen, func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"editPull": func() { + gh.remoteHead = func(string) string { return "" } + if seen == "by GitHub" { + gh.find(90).Head.SHA = strings.Repeat("b", 40) + } else { + moveBranch(f, h1) + } + }} + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "which its record froze") || gh.find(90).State != "open" || f.remoteHead(batchBranch1) == "" { + t.Errorf("run = %d, #90 %s, branch %q; want 2, #90 open and the branch kept:\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), stderr) + } + }) + } + t.Run("the close failed", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + before := writes(gh) + // Anything but run --push leaves it as it is. + for _, args := range [][]string{ + {"run", "--line", "release-1.1", "--continue"}, + {"check", "--line", "release-1.1"}, + {"fold", "--line", "release-1.1", "--original", "5", "--commit", strings.Repeat("a", 40)}, + {"notify", "--line", "release-1.1", "--push"}, + } { + if code, _, stderr := f.exec(gh, args...); code != 2 || !strings.Contains(stderr, "being closed") { + t.Errorf("%q = %d, want 2:\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + } + if writes(gh) != before || gh.find(90).State != "open" { + t.Error("a command other than run --push wrote to the batch") + } + code, _, stderr := f.exec(gh, push...) + if code != 0 || gh.find(90).State != "closed" || f.remoteHead(batchBranch1) != "" || writes(gh) != before { + t.Errorf("run --push = %d, #90 %s, branch %q, %d writes; want the close finished and nothing else\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), writes(gh)-before, stderr) + } + }) + // A run that finishes a close stops there: the change it was asked for + // needs an open batch pull request, and none is left to make it on. + t.Run("a change asked for while the close was left", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "finished the close of #90 and made no other change") || gh.find(90).State != "closed" || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --defer = %d, #90 %s, branch %q; want 2, the close finished and nothing deferred:\n%s", code, gh.find(90).State, f.remoteHead(batchBranch1), stderr) + } + }) + t.Run("closed by hand while the close writes", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"createComment": func() { gh.find(90).State = "closed" }} + code, _, stderr := f.exec(gh, deferAll...) + f.git("switch", "-q", "main") + if code != 2 || !strings.Contains(stderr, "batch pull request #90 is closed now; nothing is written to it") || len(gh.edited) != 0 || f.remoteHead(batchBranch1) != h1 { + t.Errorf("run = %d, edited %q, branch %q; want 2, no edit and the branch kept:\n%s", code, gh.edited, f.remoteHead(batchBranch1), stderr) + } + }) + t.Run("the delete failed", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"closePull": func() { f.refusePushes(true) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.refusePushes(false) + code, _, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: batch pull request #90 is closed, but its branch "+batchBranch1+" is still at "+h1[:12]) { + t.Errorf("check = %d, want 0 naming the branch:\n%s", code, stderr) + } + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue"); code != 2 { + t.Errorf("run without --push = %d, want 2:\n%s", code, stderr) + } + f.git("switch", "-q", "main") + if code, _, stderr := f.exec(gh, push...); code != 0 || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --push = %d, branch %q; want it deleted\n%s", code, f.remoteHead(batchBranch1), stderr) + } + mustRun(t, f, gh, "check", "--line", "release-1.1") + }) + // A batch pull request a person opens on the branch a close left owns + // it: neither recovery nor check offers to delete it, moved or not. One + // on another branch does not stop the delete. + for _, tc := range []struct { + name string + branch string + moved bool + }{ + {"a batch opened on the branch the close left", batchBranch1, false}, + {"a batch opened on the branch the close left, moved since", batchBranch1, true}, + {"a batch opened on another branch", "backport-batch/release-1.1-20261027", false}, + } { + t.Run(tc.name, func(t *testing.T) { + f, gh, h1 := closeWorld(t) + gh.after = map[string]func(){"closePull": func() { f.refusePushes(true) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + f.refusePushes(false) + gh.after = nil + if tc.branch != batchBranch1 { + f.git("push", "-q", "origin", h1+":refs/heads/"+tc.branch) + } + left := h1 + if tc.moved { + left = moveBranch(f, h1) + } + gh.prs = append(gh.prs, f.batchPR(91, "release-1.1", tc.branch, "a maintainer's batch")) + taken := tc.branch == batchBranch1 + _, _, stderr := f.exec(gh, "check", "--line", "release-1.1") + f.git("switch", "-q", "main") + if named := strings.Contains(stderr, "batch pull request #91 is open on "+batchBranch1); named != taken || taken == strings.Contains(stderr, "--push deletes it") || taken && strings.Contains(stderr, "git push origin --delete") { + t.Errorf("check names the branch as #91's: %v, want %v, and offers no delete of it:\n%s", named, taken, stderr) + } + code, _, stderr := f.exec(gh, push...) + f.git("switch", "-q", "main") + if taken && (code != 2 || !strings.Contains(stderr, "batch pull request #91 is open on "+batchBranch1) || strings.Contains(stderr, "git push origin --delete") || f.remoteHead(batchBranch1) != left || gh.find(91).State != "open") { + t.Errorf("run --push = %d, branch %q; want 2, no delete offered and #91's branch kept:\n%s", code, f.remoteHead(batchBranch1), stderr) + } + if !taken && (code != 0 || f.remoteHead(batchBranch1) != "") { + t.Errorf("run --push = %d, branch %q; want the left branch deleted:\n%s", code, f.remoteHead(batchBranch1), stderr) + } + }) + } + t.Run("the branch moved after the close", func(t *testing.T) { + f, gh, h1 := closeWorld(t) + var h2 string + gh.after = map[string]func(){"closePull": func() { h2 = moveBranch(f, h1) }} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 || !strings.Contains(stderr, "is at "+h2[:12]+" now") { + t.Fatalf("run = %d, want 2 naming where the branch moved\n%s", code, stderr) + } + f.git("switch", "-q", "main") + // What was pushed onto the branch may be work the next run cannot + // pick again, so the refusal promises nothing about deleting it. + if code, _, stderr := f.exec(gh, push...); code != 2 || !strings.Contains(stderr, h1[:12]) || !strings.Contains(stderr, h2[:12]) || !strings.Contains(stderr, "#90") || strings.Contains(stderr, "nothing on it is lost") || !strings.Contains(stderr, "check what "+h2[:12]+" carries") { + t.Errorf("run --push = %d, want 2 naming both heads and #90, and what to check before deleting:\n%s", code, stderr) + } + if f.remoteHead(batchBranch1) != h2 { + t.Error("the run moved a branch it did not freeze") + } + if _, err := git(deleteArgs(batchBranch1, h1)...); err == nil || f.remoteHead(batchBranch1) != h2 { + t.Error("the remote took a delete whose lease names a head it no longer has") + } + }) + t.Run("a batch of the same day that opened no pull request", func(t *testing.T) { + f, gh, _ := closeWorld(t) + if code, _, stderr := f.exec(gh, deferAll...); code != 0 || f.remoteHead(batchBranch1) != "" { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + // The same day, a new batch reuses the branch name, pushes, and + // fails to open its pull request. + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh.fail = map[string]error{"createPull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, push...); code != 2 { + t.Fatalf("the same-day run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + h2 := f.remoteHead(batchBranch1) + for _, args := range [][]string{push, {"run", "--line", "release-1.1", "--onto", h2, "--push"}} { + if code, _, stderr := f.exec(gh, args...); code != 2 || !strings.Contains(stderr, "git push origin --delete "+batchBranch1) { + t.Errorf("%q = %d, want 2 naming the branch to delete:\n%s", args, code, stderr) + } + f.git("switch", "-q", "main") + } + if f.remoteHead(batchBranch1) != h2 || len(gh.created) != 1 { + t.Errorf("the branch moved to %.12s or a pull request was opened: %q", f.remoteHead(batchBranch1), gh.created) + } + f.git("push", "-q", "origin", "--delete", batchBranch1) + gh.nextPR = 91 + mustRun(t, f, gh, push...) + if gh.find(91) == nil { + t.Error("after the branch was deleted the next run opened no batch") + } + }) + t.Run("a preview", func(t *testing.T) { + f, gh, _ := closeWorld(t) + gh.fail = map[string]error{"closePull": errString("HTTP 502")} + if code, _, stderr := f.exec(gh, deferAll...); code != 2 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + before := writes(gh) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "x") + if code != 2 || !strings.Contains(stderr, "run --line release-1.1 --push finishes") || writes(gh) != before || gh.find(90).State != "open" { + t.Errorf("a preview = %d, want 2 naming the step left and nothing written:\n%s", code, stderr) + } + }) +} + +// A batch that merged on GitHub and left its branch behind is named when a +// run of the same day refuses to push over that branch. +func TestRefusalNamesTheMergedBatchsBranch(t *testing.T) { + f, gh, _ := closeWorld(t) + head := f.remoteHead(batchBranch1) + mergeBatch(f, gh) + // A commit pushed onto the batch after it merged is left on its branch. + f.git("switch", "-q", "--detach", head) + f.git("commit", "-q", "--allow-empty", "-m", "pushed after the merge") + pushed := f.git("rev-parse", "HEAD") + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push") + if code != 2 || !strings.Contains(stderr, "it is the branch of batch pull request #90, which is merged") || strings.Contains(stderr, "--onto") || !strings.Contains(stderr, "check what "+pushed[:12]+" carries") { + t.Errorf("run --push = %d, want a refusal naming #90:\n%s", code, stderr) + } +} diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 1a633134f..4ae5731b6 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -449,9 +449,35 @@ func (g *fakeGH) editPull(number int, title, bodyFile string) error { if err := g.readBody(bodyFile); err != nil { return err } + if pr := g.find(number); pr != nil { + pr.Body = g.bodies[len(g.bodies)-1] + } + if f := g.after["editPull"]; f != nil { + f() + } + return nil +} + +func (g *fakeGH) closePull(number int) error { + if err := g.write(number, "a close"); err != nil { + return err + } + if err := g.inject("closePull"); err != nil { + return err + } + if pr := g.find(number); pr != nil { + pr.State = "closed" + } + if f := g.after["closePull"]; f != nil { + f() + } + return nil +} + +func (g *fakeGH) find(n int) *pullRequest { for i := range g.prs { - if g.prs[i].Number == number { - g.prs[i].Body = g.bodies[len(g.bodies)-1] + if g.prs[i].Number == n { + return &g.prs[i] } } return nil diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index b1f3b5ce3..072673a77 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -127,6 +127,7 @@ type github interface { // createPull opens the draft and returns its number and URL. createPull(base, head, title, bodyFile string) (int, string, error) editPull(number int, title, bodyFile string) error + closePull(number int) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error editComment(id int64, bodyFile string) error @@ -259,6 +260,11 @@ func (ghCLI) editPull(number int, title, bodyFile string) error { return err } +func (ghCLI) closePull(number int) error { + _, err := capture("gh", "pr", "close", strconv.Itoa(number)) + return err +} + func (ghCLI) comments(number int) ([]issueComment, error) { out, err := capture("gh", "api", "--paginate", "--slurp", "repos/{owner}/{repo}/issues/"+strconv.Itoa(number)+"/comments") if err != nil { diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 98ed2b2ac..bd0e02d30 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -54,7 +54,8 @@ run Builds backport-batch/- in this working tree, off --defer rebuilds the open batch off origin/ without those originals and records them as deferred, for --reason: the batch keeps them out, the next one tries them again. --push replaces the branch, - leased on the head the rebuild started from. + leased on the head the rebuild started from; when nothing is left it + closes the batch PR instead, after recording why in it. check Exits 1 while a change the policy takes is neither on the line nor on the open batch PR, 0 when nothing is, 2 when it cannot tell. With --cut, the gate before an rc or a patch is cut, it also exits 1 while a change diff --git a/cmd/backport-batch/overrides_test.go b/cmd/backport-batch/overrides_test.go index a759fd59c..0bcdc426e 100644 --- a/cmd/backport-batch/overrides_test.go +++ b/cmd/backport-batch/overrides_test.go @@ -259,6 +259,13 @@ func TestGHCLI(t *testing.T) { t.Error("pull took GitHub's answer about another pull request") } write("") + if err := (ghCLI{}).closePull(90); err != nil { + t.Fatal(err) + } + if want := "pr close 90"; called() != want { + t.Errorf("gh %q, want gh %q", called(), want) + } + write("") if err := (ghCLI{}).editPull(90, "fix(backport): batch", "/tmp/body.md"); err != nil { t.Fatal(err) } diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index 830685474..10b3b3f7f 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -313,6 +313,17 @@ func TestMetadataOperationsKeepTheMode(t *testing.T) { if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/") { t.Errorf("check --cut = %d, want #5 still waiting:\n%s", code, stdout) } + + // Deferring the last picks closes the batch, and its record + // still says how the batch was assembled. + mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "6,8", "--reason", "fails the e2e", "--push") + rec, err = parseRecord(gh.find(90).Body, notifyLine) + if err != nil || gh.find(90).State != "closed" || rec.Closing == nil || rec.Built.Options != (runOptions{NoWorkflowFiles: true, Mode: modeClose}) { + t.Errorf("after the close the record is %+v, %v; want the flag kept", rec.Built, err) + } + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check --cut = %d, want #5 missing once no batch lists it:\n%s", code, stdout) + } }) } } diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index a0b2942ff..46a76669f 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -55,15 +55,6 @@ func bodyReportOf(t *testing.T, body string, on ...line) storedReport { return *s } -func (g *fakeGH) find(n int) *pullRequest { - for i := range g.prs { - if g.prs[i].Number == n { - return &g.prs[i] - } - } - return nil -} - // refusePushes makes the remote refuse every push, as a lost connection or // a protected branch would, until it is called with false. func (f *fixture) refusePushes(on bool) { diff --git a/docs/release/backport-mechanics.md b/docs/release/backport-mechanics.md index f713d20aa..70da94e85 100644 --- a/docs/release/backport-mechanics.md +++ b/docs/release/backport-mechanics.md @@ -14,7 +14,7 @@ go build ./cmd/backport-batch ./backport-batch check --line release-X.Y --cut ``` -[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check. A change that reached the line by hand counts like any other, by what reached the line. +[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut. A change that reached the line by hand counts like any other, by what reached the line. ### Backporting code that touches monotonic counters diff --git a/docs/release/backport-tasks/failed-ci.md b/docs/release/backport-tasks/failed-ci.md index e845eff99..18b26f4dc 100644 --- a/docs/release/backport-tasks/failed-ci.md +++ b/docs/release/backport-tasks/failed-ci.md @@ -33,6 +33,8 @@ It rebuilds the branch locally without those originals and prints the new body, It replaces the batch branch only if nobody pushed onto it since the rebuild read it, and updates the pull request body. CI runs again on the new head; if it fails again on another original, repeat from step 1. +When the culprits are everything the batch carries, there is no branch left to push: the body printed without `--push` says the batch is closing, and `--push` records the deferrals in the batch pull request, closes it and deletes its branch, still only if nobody pushed onto it since. The next batch tries them again. If it stops part way, the next `run --push` on the line finishes the close, and says so. + ### Never - Defer an original the failure does not reach, to make the batch green by taking out more than the culprit. diff --git a/docs/release/backports.md b/docs/release/backports.md index 0cc8db3b6..37e2e89f9 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -83,7 +83,7 @@ A run by hand trusts the open batch pull request the scheduled run opened as `co `run` picks what the rules take onto `backport-batch/-` and opens a draft pull request against the line. A pull request merged into `main` with a merge commit goes onto the batch as its own commits, oldest first, each picked with `git cherry-pick -x --signoff` so it keeps its author, its message and its sign-offs and gains the sign-off of whoever runs the batch; commits that merged `main` into the pull request's branch are left out. A merge commit joins them to the batch: it carries the original's merge message from `main` and ends in a `Backported-from: ` trailer. A pull request squash-merged into `main` goes on as one pick. When any commit of an original conflicts, nothing of that original stays on the batch. The batch pull request is merged into the line with a merge commit, so all of this reaches the line as it is. Its body lists the originals with how many commits each brought and which of them differ from the commit they came from, the conflicts it deferred with the commit that failed and its files, and what it skipped under which rule, and ends with the [line's record and the run's report](#the-lines-record-and-reports); `--push` prints the pull request's URL. While that pull request is open, the next run continues it, `--continue` reading its head, rather than opening another. CI on the batch pull request runs the line's build and tests, and a maintainer merges it. Since rule 6 reads the newer line itself, an older line takes a fix in the batch after the newer line's batch with that fix has merged. -A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. +A deferred conflict waits for the next batch. To land it sooner, pick the original's commits onto the batch head with `git cherry-pick -x`, resolving each conflict, and join them to the batch head with a merge commit that names the original; an original squash-merged into `main` is one pick of its commit there, with no merge. Check that merge or that pick with `./backport-batch fold --line --original --commit `, and push it onto the batch branch, as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes. CI runs `backport-batch verify` on every push to a batch pull request, which holds every original on it to the same shape. When CI on the batch pull request fails, the original it comes from is taken out with `./backport-batch run --line --defer --reason --push` and goes to the next batch, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the rest of the batch merges without it. When it is all the batch carries, `--push` records why in the batch pull request, closes it and deletes its branch, and the next batch tries it again. Before a patch release or an rc is cut from a line, `./backport-batch check --line --cut` has to exit 0. A cut tags the line, so the check fails while a pull request the rules take is not on the line, including one that is only on the open batch pull request: merge the batch pull request first. It prints the URLs, and prints what the batch deferred as warnings, since deferring is a maintainer's call; so is a pull request labelled `backport/skip` that the open batch still carries. An original that waits on the batch for a maintainer fails the cut, also when it is all the batch has left. A cut that passes says so. Without `--cut`, as the scheduled run uses it, a change on the open batch pull request counts as carried.