From c617a994531d623b9c9c1946ffd2de620477ae13 Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 10 Nov 2026 08:09:37 +0000 Subject: [PATCH 1/5] fix(release): withhold or refuse a unit that does only part of its original on the line A clean cherry-pick says nothing about whether the original landed whole: git deletes a file the line never had without a conflict and leaves a file the line lacks untouched. #5088 reached the 1.6 batch that way, carrying only its test while the two deletions that are the fix had no effect. Every M and D path of an original is now judged at its counterpart on the line, followed through renames on main, on the line, or both. A modified path missing before and after, or left unchanged when the line holds neither the original's blob nor a clean git merge-file result, is omitted; a deleted path absent before is omitted. Content merge-file refuses, mode changes and skipped rename detection become diagnostics and never count as omissions; I/O failures stop the command. The same guard runs in four places: the picker, which resets a partial pick and parks the original; a check-time pass over every unit on the open batch at its own first parent, whatever its label, which fails check --cut and verify; the same pass over every unpicked original's clean net merge, which parks a partial one with no record needed; and fold. contained accepts the renamed path, so a fix of a renamed file passes verify and fold. The unit mapping also closes the found() gap: a merged original whose first pick came out empty is on the batch, not missing. The record moves to version 3 with parked entries and partial units, and reports to format 2 with typed parked entries; format 1 reports still read. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 8 +- cmd/backport-batch/assemble.go | 51 +- cmd/backport-batch/body.go | 54 +- cmd/backport-batch/check.go | 29 +- cmd/backport-batch/defer.go | 13 +- cmd/backport-batch/effects.go | 555 ++++++++++++++++++ cmd/backport-batch/effects_test.go | 350 +++++++++++ cmd/backport-batch/fold_test.go | 56 ++ cmd/backport-batch/history.go | 45 +- cmd/backport-batch/main.go | 20 + cmd/backport-batch/park_test.go | 331 +++++++++++ cmd/backport-batch/record.go | 104 +++- cmd/backport-batch/record_test.go | 43 +- cmd/backport-batch/report.go | 129 +++- cmd/backport-batch/report_test.go | 54 +- cmd/backport-batch/shape.go | 46 +- cmd/backport-batch/verify.go | 9 + docs/release/backport-mechanics.md | 2 +- docs/release/backport-tasks/README.md | 2 +- .../backport-tasks/resolve-conflict.md | 2 +- docs/release/backports.md | 10 +- 21 files changed, 1856 insertions(+), 57 deletions(-) create mode 100644 cmd/backport-batch/effects.go create mode 100644 cmd/backport-batch/effects_test.go create mode 100644 cmd/backport-batch/park_test.go diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index b1837f489..6d9157795 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -37,6 +37,8 @@ The pull request body goes to stdout, built from the line's own `.github/PULL_RE `--no-workflow-files` is for a token that may not push a change to `.github/workflows/`, which the scheduled run's App token may not. An original that changes anything there is not picked: the body lists it under "Waiting for a maintainer" with those files, and the `backport-batch` block records it with `"workflows": true`, so `check`, `notify` and a run that continues the batch know it waits. It is never a conflict. A run without the flag, a maintainer's with their own token, picks it like any other original, onto the open batch when it continues one. +A clean pick can still carry only part of its original: git deletes a file the line never had without a word, and leaves alone a file the line lacks. So every path the original modifies or deletes, `M` or `D` in `git diff --name-status --no-renames M^1 M`, is judged at its counterpart on the line: the path itself, or the one a rename `git diff -M` finds between the line's fork point and `M^1` on `main`, between the fork point and the line, or both in turn. A modified path counts as left without effect when its counterpart exists neither before nor after the pick; when the pick leaves an existing file as it was, the file counts as there only when its blob is the original's, binary files included, or when `git merge-file` of the original's preimage, the line's file and the original's postimage merges cleanly back into the line's file. A deleted path counts as left without effect when its counterpart is absent before a pick that changes something, a deletion the line already made being no change of that pick, or still there after it, whatever the pick changed in it; when the original adds that path itself, as when the line made the same rename, it is judged as a modified file against what the original makes of that path. Picks that change nothing carry the original only when the line already has what it does to every file it modifies or deletes, so a change made where its branch merged `main`, which no pick brings, is still missing. What cannot be judged, a mode or type change, content `merge-file` refuses, or a rename git skipped looking for, is shown with the original and never counts against it. An original whose net merge onto the line, `git merge-tree --merge-base=M^1`, leaves something without effect is not tried, and one whose picks do is taken back off the branch: either is parked as partial, listed under "Parked" with those paths, and its release note stays out of the body. A conflict is never called partial. `backport/include` lifts no park. + `--defer N[,N] --reason TEXT` takes originals out of the open batch pull request, usually the ones a failing check was pinned on. It builds the batch branch again off `origin/` without their units: while the batch still sits on the line's head, the units before the first one it drops stay as they are, and every later one, a pushed resolution included, is picked again in its own shape, a merge's picks one by one and then the merge with its message. When dropping one leaves a later commit that no longer applies, the run stops and names that commit's original, to defer as well. It adds no new original. It also takes out an original that got `backport/skip` after the batch picked it, which a run never removes on its own and the body lists as still on the branch. The body records each deferred original with the reason in the `backport-batch` block; a run that continues this batch keeps them out, and the next batch tries them again. A batch that carries a merge commit other than an original's, as GitHub's update-branch button makes, is refused, since picking such a merge again would lose what it resolved. When it takes out everything the batch carries, there is nothing left to push, and `--push` closes the batch pull request instead, as [the line's record](#the-lines-record-and-reports) describes. `--drop N[,N] --reason TEXT` drops originals from the line: where `--defer` holds one out of this batch, a drop keeps it out of every later batch of the line too, until `--undrop N[,N]` withdraws it. A drop is recorded in the [line's record](#the-lines-record-and-reports), so both need an open batch pull request that carries one, and exit 2 without it; once a batch merges, its drops are carried into the next batch's record. A dropped original the open batch carries is taken off it as `--defer` would, and the rebuild is pushed the same way; any other drop or undrop changes only the record, so the batch stays as it is even when the line moved since. An undropped original is missing until the next run tries it. A drop names the original's merge commit on `main` as well as its number, and a record whose drop does not match what that merge commit merged stops every command with exit 2. An original the line already carries cannot be dropped, and neither can a number that merged into `main` before the line branched. The order is the rules', then the drop: what `backport/skip`, rules 2 to 4 or the line itself decide comes first, with the drop kept on record beside it, and `backport/include` does not lift a drop. The body lists dropped originals under their own heading among the skipped ones, `check` prints a note for each, and `fold` refuses one. @@ -51,13 +53,15 @@ Exits 1 and prints the URL of every pull request the rules take that is neither An original with `backport/skip` that the open batch picked before the label came prints a warning in either mode, naming the batch and `run --defer`, which takes it out; it fails neither. +A unit on the open batch that leaves part of its original without effect, judged as [`run`](#run) judges a pick, at the unit's own first parent and whatever its original's rule, label or drop, prints a warning naming what it leaves out, and fails `--cut` ahead of any skip, drop or stale warning: take it out with `run --defer`. An original that is parked as partial prints as a note, `#N is parked: only part of it applies`, and fails neither. + ### fold -Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request `--commit` is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one `Backported-from:` trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the `(cherry picked from commit …)` line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose `(cherry picked from commit …)` line names the original's commit on `main`. Either has to change something, and only files the original changes. The [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) gives the commands. fold prints `ok` and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head. +Checks a hand-made resolution of a deferred conflict before it is pushed onto the batch, in the shape the run gives an original. For a merged pull request `--commit` is a merge whose first parent is the head of the open batch pull request and whose message names the original in exactly one `Backported-from:` trailer; its second parent ends a straight line from that head of picks of the pull request's own commits, in their order, each ending in the `(cherry picked from commit …)` line of the commit it came from, none claiming another commit of the same original, a commit of another pull request never among them. For a squash-merged one it is a single commit on that head whose `(cherry picked from commit …)` line names the original's commit on `main`. Either has to change something, and only files the original changes or the paths a rename on `main` or on the line gave them, and has to leave nothing the original modifies or deletes without effect, judged as [`run`](#run) judges a pick: a resolution that ports what the line has and leaves out what it lacks is refused, since such a port goes to the line as a pull request of its own. The [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) gives the commands. fold prints `ok` and exits 0, or prints why not and exits 1. It never commits or pushes; push one checked resolution at a time, since the next one has to sit on the new head. ### verify -Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from `origin/` to `--head`, or to the head of the open batch pull request, and requires each unit to be one original, merged into `main` after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With `--head` it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml) runs verify on every push to a batch pull request, from a build of `main`, with the batch head fetched as data. +Holds every unit of a batch to the shape fold checks for one, the clean picks as well as resolutions: it walks the batch's first-parent line from `origin/` to `--head`, or to the head of the open batch pull request, and requires each unit to be one original, merged into `main` after the line branched, in that shape, and no original twice. A commit that carries no original fails it, and so does a merge of the line into the batch, and a unit that leaves part of its original without effect, judged as [`run`](#run) judges a pick. It prints one line per unit, and exits 0 when all pass, 1 when one does not, and 2 when it cannot tell. With `--head` it reads git alone, nothing from GitHub. fold runs only where someone resolved a conflict and chose to run it; [`pr-labeler.yaml`](../../.github/workflows/pr-labeler.yaml) runs verify on every push to a batch pull request, from a build of `main`, with the batch head fetched as data. ### notify diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 7f36ff3f4..53093b018 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -198,8 +198,14 @@ func (a *app) run(args []string) int { } } for _, v := range b.qualifying() { + on := false + if carried != nil { + if on, err = b.onBranch(carried, v); err != nil { + return a.fail(err) + } + } switch { - case carried != nil && found(carried, v): + case on: r.onBranch = append(r.onBranch, v) continue case held[v.o.number].Reason != "": @@ -210,6 +216,17 @@ func (a *app) run(args []string) int { r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) continue } + if v.partial != nil { + // Judged at the line's head; what the batch picked since may + // supply what it lacked there. + if err := a.partialAtHead(b, v); err != nil { + return a.fail(err) + } + if v.partial != nil { + r.parked = append(r.parked, v) + continue + } + } if *noWorkflows { files, err := workflowFiles(v.commits) if err != nil { @@ -220,11 +237,19 @@ func (a *app) run(args []string) int { continue } } - p, err := pickOriginal(v) + p, err := pickOriginal(v, func(base, head string) ([]string, error) { + omitted, diag, err := b.omittedEffects(v, base, head) + v.addDiag(diag...) + return omitted, err + }) if err != nil { return a.fail(err) } switch p.status { + case "partial": + v.partial = p.files + v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(p.files)} + r.parked = append(r.parked, v) case "picked": r.onBranch = append(r.onBranch, v) case "conflict": @@ -246,10 +271,23 @@ func (a *app) run(args []string) int { if !descends { return a.fail(fmt.Errorf("%.12s does not descend from --onto %.12s", head, *onto)) } + if err := b.markPartial(r, carried.head); err != nil { + return a.fail(err) + } } return a.finish(b, r, branch, date, *push, false) } +// partialAtHead judges v again where the run has got to, when earlier picks +// moved the branch past the line's head. +func (a *app) partialAtHead(b *batch, v *verdict) error { + head, err := resolve("HEAD") + if err != nil || head == b.lineHead { + return err + } + return b.partialAt(v, head) +} + // keptOut lists the originals with backport/skip, or dropped, that the batch // still carries: a run never takes a commit off the batch, --defer does. func keptOut(b *batch, carried *history) ([]*verdict, error) { @@ -261,7 +299,11 @@ func keptOut(b *batch, carried *history) ([]*verdict, error) { if err := b.load(v); err != nil { return nil, err } - if found(carried, v) { + on, err := b.onBranch(carried, v) + if err != nil { + return nil, err + } + if on { out = append(out, v) } } @@ -365,7 +407,8 @@ func (r *report) lineRecord(doc reportDoc) lineRecord { dropped = []droppedEntry{} } slices.SortFunc(dropped, func(a, b droppedEntry) int { return a.Original - b.Original }) - return lineRecord{Version: 2, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale)} + return lineRecord{Version: recordVersion, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale), + Parked: slices.Clone(doc.Primary.Parked), Partial: slices.Clone(doc.Overlays.Partial)} } // branchDate is the date a batch branch was opened under. diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index da4691450..282f93669 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -238,8 +238,10 @@ type report struct { overrides []*verdict stale []*verdict start time.Time - // shapes says, for an original on the branch, what it brought. - shapes map[*verdict]string + // shapes says, for an original on the branch, what it brought, and + // partialFiles what a partial unit leaves without effect. + shapes map[*verdict]string + partialFiles map[*verdict][]string // missing are originals the policy takes that the run neither tried nor // found anywhere. missing []*verdict @@ -253,6 +255,11 @@ type report struct { closing *closing // dropped are the maintainers' drops the record keeps. dropped []droppedEntry + // parked are what the batch leaves out because it cannot land whole, + // each with its park, and partial the originals on the branch whose unit + // does only part of what they do on main. + parked []*verdict + partial []*verdict } // ruleName is how output names what kept an original out: the policy's @@ -346,6 +353,18 @@ func (r *report) what(details map[int]bool) string { fmt.Fprintf(&b, "- %s %s: %s\n", prLink(p.v.o.number), titleSpan(p.v.o.title), strings.Join(files, ", ")) } } + if len(r.partial) > 0 { + fmt.Fprintf(&b, "\n### Still on this branch: only part of it applies\n\nEach of these is on this branch but leaves files of its original as they were on %s, so it carries only part of what it does on `main`. Take each out with `backport-batch run --defer` before the cut; what applies of it goes as a pull request of its own.\n\n", codeSpan(r.line.branch())) + for _, v := range r.partial { + fmt.Fprintf(&b, "- %s %s: leaves %s without effect\n", prLink(v.o.number), titleSpan(v.o.title), fileSpans(r.partialFiles[v])) + } + } + if len(r.parked) > 0 { + fmt.Fprintf(&b, "\n### Parked\n\nThese cannot land on %s whole, so the batch does not pick them: each would leave part of it without effect there, or builds on a change that never reaches the line. A maintainer ports what applies as a pull request of its own, or drops the original with `backport-batch run --drop`; `backport/include` does not lift a park.\n\n", codeSpan(r.line.branch())) + for _, v := range r.parked { + fmt.Fprintf(&b, "- %s %s: %s\n", prLink(v.o.number), titleSpan(v.o.title), parkReason(*v.park)) + } + } if len(r.overrides) > 0 { b.WriteString("\n### Maintainer overrides\n\n") for _, v := range r.overrides { @@ -384,6 +403,8 @@ func (r *report) what(details map[int]bool) string { // outcome is what became of an original a label overrode. func (r *report) outcome(v *verdict) string { switch { + case slices.Contains(r.partial, v): + return "still on this branch, but only part of it applies; take it out with `backport-batch run --defer`" case slices.Contains(r.stale, v) && v.rule == ruleDropped: return "dropped, but still on this branch; take it out with `backport-batch run --defer`" case slices.Contains(r.stale, v): @@ -398,12 +419,41 @@ func (r *report) outcome(v *verdict) string { return "deferred to the next batch" case slices.ContainsFunc(r.waiting, func(p pick) bool { return p.v == v }): return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) + case slices.Contains(r.parked, v): + return "parked: " + parkReason(*v.park) case v.rule != 0: return "skipped by " + ruleName(v.rule) + ", " + titleSpan(v.reason) } return "not picked" } +// fileSpans lists files as code spans, the first few of them. +func fileSpans(files []string) string { + var out []string + for _, f := range files[:min(len(files), filesListed)] { + out = append(out, codeSpan(f)) + } + if more := len(files) - len(out); more > 0 { + out = append(out, fmt.Sprintf("%d more", more)) + } + return strings.Join(out, ", ") +} + +// parkReason says why a park leaves its original out, for the body. +func parkReason(p parkedEntry) string { + if p.Kind == parkPartial { + return "only part of it applies; it would leave " + fileSpans(p.Files) + " without effect" + } + var needs []string + for _, n := range p.Needs { + needs = append(needs, prLink(n)) + } + for _, c := range p.Commits { + needs = append(needs, codeSpan(c[:12])) + } + return "builds on " + strings.Join(needs, ", ") + ", which never reaches this line" +} + func (r *report) notes() string { var notes []string for _, v := range r.onBranch { diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index ae5395d45..b9983aca3 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -20,6 +20,7 @@ import ( "fmt" "slices" "strconv" + "strings" ) // check says whether every original the policy takes is on the line, on the @@ -163,6 +164,10 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { named := v.o.url() + " " + terminalSafe(v.o.title) e, isDeferred := deferred[v.o.number] switch { + case b.partialUnits[v] != nil && cut: + out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer before the cut: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} + case b.partialUnits[v] != nil: + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} case slices.Contains(stale, v): out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d is still on batch #%d; take it out with backport-batch run --defer: %s", v.o.number, b.pr.Number, named)} case v.rule == ruleDropped: @@ -173,14 +178,16 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { if cut { out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} } + case isDeferred && e.Reason != "": + out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} + case v.park != nil && v.park.Kind == parkPartial: + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: only part of it applies to %s, which would leave %s without effect; port what applies as a pull request of its own: %s", v.o.number, b.line.branch(), quotedFiles(v.park.Files), named)} case isDeferred && e.Workflows && !b.src.open: out[v] = disposition{outcomeMissing, named} case isDeferred && e.Workflows && cut: out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR %s for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, on, named)} case isDeferred && e.Workflows: out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on %s for a maintainer, since it changes %s: %s", on, workflowsDir, named)} - case isDeferred && e.Reason != "": - out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} case isDeferred: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on %s: %s", on, named)} default: @@ -190,6 +197,18 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { return out, nil } +// quotedFiles names files for a terminal, each quoted, the first few of them. +func quotedFiles(files []string) string { + var out []string + for _, f := range files[:min(len(files), filesListed)] { + out = append(out, strconv.Quote(f)) + } + if more := len(files) - len(out); more > 0 { + out = append(out, fmt.Sprintf("%d more", more)) + } + return strings.Join(out, ", ") +} + // terminalSafe keeps a title from carrying escape sequences into a terminal. func terminalSafe(s string) string { return capped(oneLine(clean(s, false)), titleLimit) } @@ -289,5 +308,9 @@ func refuseFold(b *batch, n int, commit string) (string, error) { case carried != v: return fmt.Sprintf("%.12s carries #%d, not #%d", commit, carried.o.number, n), nil } - return "", nil + omitted, _, err := b.omittedEffects(v, u.parents[0], u.commit) + if err != nil || len(omitted) == 0 { + return "", err + } + return leftOut(v, b.line, omitted), nil } diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 52fa8cf62..8915d4ebd 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -78,7 +78,7 @@ func (a *app) rebuild(b *batch, numbers []int, reason string, push bool) int { switch v := b.find(n); { case v == nil: return a.fail(fmt.Errorf("#%d did not merge into main after %s branched", n, b.line.branch())) - case v.rule != 0 && !slices.Contains(stale, v): + case v.rule != 0 && !slices.Contains(stale, v) && b.partialUnits[v] == nil: return a.fail(fmt.Errorf("#%d is not one the batch takes: %s, %s", n, ruleName(v.rule), terminalSafe(v.reason))) } hold[n] = reason @@ -140,11 +140,17 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] } for _, v := range b.qualifying() { e := recorded[v.o.number] + on, err := b.onBranch(picks, v) + if err != nil { + return a.fail(err) + } switch { - case found(picks, v): + case on: r.onBranch = append(r.onBranch, v) case hold[v.o.number] != "": r.held = append(r.held, pick{v: v, status: "held", reason: hold[v.o.number]}) + case v.park != nil: + r.parked = append(r.parked, v) case e.Workflows: r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: e.Files, more: e.More}) case len(e.Files) > 0: @@ -155,6 +161,9 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] r.missing = append(r.missing, v) } } + if err := b.markPartial(r, picks.head); err != nil { + return a.fail(err) + } return a.finish(b, r, b.pr.Head.Ref, date, push, true) } diff --git a/cmd/backport-batch/effects.go b/cmd/backport-batch/effects.go new file mode 100644 index 000000000..590b70b21 --- /dev/null +++ b/cmd/backport-batch/effects.go @@ -0,0 +1,555 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "fmt" + "os" + "path/filepath" + "slices" + "strconv" + "strings" +) + +// A clean pick can still carry only part of an original: git deletes a file +// the line never had without a word, and leaves alone a file the line lacks. +// What follows tells such a unit from a whole one, by what it leaves without +// effect of the files the original modifies or deletes. + +// treeEntry is one record of git ls-tree. +type treeEntry struct{ mode, kind, oid string } + +// regular says whether git can merge the entry as text: a file, not a +// symlink, a submodule or a tree. +func (e treeEntry) regular() bool { + return e.kind == "blob" && (e.mode == "100644" || e.mode == "100755") +} + +// treeEntries reads the entries paths have at treeish, from full ls-tree +// records: a path that is a tree there, or absent, has none. +func treeEntries(treeish string, paths []string) (map[string]treeEntry, error) { + out := map[string]treeEntry{} + want := map[string]bool{} + for _, p := range paths { + want[p] = true + } + const chunk = 500 + for len(paths) > 0 { + n := min(chunk, len(paths)) + args := append([]string{"--literal-pathspecs", "ls-tree", "-r", "-z", "--full-tree", "--end-of-options", treeish, "--"}, paths[:n]...) + raw, err := git(args...) + if err != nil { + return nil, err + } + for _, rec := range splitNUL(raw) { + meta, path, ok := strings.Cut(rec, "\t") + f := strings.Fields(meta) + if !ok || len(f) != 3 || checkSHA(f[2]) != nil { + return nil, fmt.Errorf("git ls-tree %s: unreadable record %q", treeish, terminalSafe(rec)) + } + if want[path] { + out[path] = treeEntry{mode: f[0], kind: f[1], oid: f[2]} + } + } + paths = paths[n:] + } + return out, nil +} + +// renameSet is what git diff -M found renamed between two commits, old path +// to new, and whether it looked at every candidate. +type renameSet struct { + moved map[string]string + complete bool +} + +// renames reads the renames between from and to, once per pair. git says on +// stderr when it skipped inexact detection for too many files; complete is +// then false, and no path may be called absent for want of a rename. +func (b *batch) renames(from, to string) (renameSet, error) { + key := from + ".." + to + if s, ok := b.renameCache[key]; ok { + return s, nil + } + // -l0 lifts git's own limit, which the picker's merge does not share: + // a rename the pick follows has to be one the checks of its shape see. + stdout, stderr, code, err := gitStatus("diff", "--no-ext-diff", "--no-textconv", "-M", "-l"+strconv.Itoa(b.renameLimit), "--name-status", "-z", "--diff-filter=R", "--end-of-options", from, to) + switch { + case err != nil: + return renameSet{}, err + case code != 0: + return renameSet{}, fmt.Errorf("git diff -M %s %s: exit %d: %s", from, to, code, strings.TrimSpace(stderr)) + } + f := splitNUL(stdout) + if len(f)%3 != 0 { + return renameSet{}, fmt.Errorf("unreadable renames between %s and %s", from, to) + } + s := renameSet{moved: map[string]string{}, complete: !strings.Contains(stderr, "detection was skipped")} + for i := 0; i < len(f); i += 3 { + if !strings.HasPrefix(f[i], "R") { + return renameSet{}, fmt.Errorf("unreadable renames between %s and %s: %q", from, to, terminalSafe(f[i])) + } + s.moved[f[i+1]] = f[i+2] + } + if b.renameCache == nil { + b.renameCache = map[string]renameSet{} + } + b.renameCache[key] = s + return s, nil +} + +// effectPaths are the paths v modifies or deletes; what it adds has nothing +// to correspond to. +func effectPaths(v *verdict) []string { + var out []string + for _, c := range v.changes { + if c.status == 'M' || c.status == 'D' { + out = append(out, c.path) + } + } + return out +} + +// correspondence maps a path of the original (at M^1) to the path holding the +// same file on target, through a main-side rename base..M^1, a line-side +// rename base..target, or both composed through the base path. A path with +// no counterpart maps to itself, and one git skipped rename detection for is +// left out and named in unknown: it is not known to be absent. +func (b *batch) correspondence(v *verdict, target string) (map[string]string, []string, error) { + paths := effectPaths(v) + have, err := treeEntries(target, paths) + if err != nil { + return nil, nil, err + } + corr := map[string]string{} + var missing []string + for _, p := range paths { + if _, ok := have[p]; ok { + corr[p] = p + } else { + missing = append(missing, p) + } + } + if len(missing) == 0 { + return corr, nil, nil + } + onMain, err := b.renames(b.base, v.o.merge+"^1") + if err != nil { + return nil, nil, err + } + onLine, err := b.renames(b.base, target) + if err != nil { + return nil, nil, err + } + fromBase := map[string]string{} + for old, now := range onMain.moved { + fromBase[now] = old + } + candidates := map[string]string{} + var names []string + for _, p := range missing { + q := p + if old, ok := fromBase[p]; ok { + q = old + } + r := q + if now, ok := onLine.moved[q]; ok { + r = now + } + if r != p { + candidates[p] = r + names = append(names, r) + } + } + found, err := treeEntries(target, names) + if err != nil { + return nil, nil, err + } + var unknown []string + for _, p := range missing { + if r, ok := candidates[p]; ok { + if _, ok := found[r]; ok { + corr[p] = r + continue + } + } + if !onMain.complete || !onLine.complete { + unknown = append(unknown, p) + continue + } + corr[p] = p + } + return corr, unknown, nil +} + +// omittedEffects lists the paths v modifies or deletes that the change from +// before to after leaves without effect, each judged at its corresponding +// path c(p) on before. A modified path is omitted when c(p) exists on neither +// side, applied when the change touches it, and when the change leaves an +// existing file alone, satisfied only if the line already has what the +// original makes of it. A deleted path is omitted when c(p) is absent before: +// a deletion the line already did is no change this unit makes. diag says +// what could not be judged, which is never counted as omitted. +func (b *batch) omittedEffects(v *verdict, before, after string) ([]string, []string, error) { + if err := b.load(v); err != nil { + return nil, nil, err + } + corr, unknown, err := b.correspondence(v, before) + if err != nil { + return nil, nil, err + } + var diag []string + for _, p := range unknown { + diag = append(diag, "git skipped rename detection, so where "+strconv.Quote(p)+" is on the line is not known") + } + added := map[string]bool{} + for _, c := range v.changes { + if c.status == 'A' { + added[c.path] = true + } + } + // A deleted path whose counterpart the original adds itself is a rename + // the line already made the same way: it is judged as a modified file, + // against what the original makes of that path. + var at, own, moved []string + movedTo := map[string]string{} + for _, c := range v.changes { + cp, ok := corr[c.path] + if !ok { + continue + } + at = append(at, cp) + switch { + case c.status == 'M': + own = append(own, c.path) + case c.status == 'D' && cp != c.path && added[cp]: + own, moved = append(own, c.path), append(moved, cp) + movedTo[c.path] = cp + } + } + bef, err := treeEntries(before, at) + if err != nil { + return nil, nil, err + } + aft, err := treeEntries(after, at) + if err != nil { + return nil, nil, err + } + pre, err := treeEntries(v.o.merge+"^1", own) + if err != nil { + return nil, nil, err + } + post, err := treeEntries(v.o.merge, append(slices.Clone(own), moved...)) + if err != nil { + return nil, nil, err + } + // A change that makes none is judged by its trees, whatever names them: + // a commit and the tree a net merge wrote can be the same change. + same := before == after + if !same { + bt, err := resolveTree(before) + if err != nil { + return nil, nil, err + } + at, err := resolveTree(after) + if err != nil { + return nil, nil, err + } + same = bt == at + } + var omitted []string + // modified judges a path whose counterpart cp should end up as the + // original's postimage p1 of it. + modified := func(path, cp string, p0, p1 treeEntry) error { + be, inBefore := bef[cp] + ae, inAfter := aft[cp] + switch { + case !inBefore && !inAfter: + omitted = append(omitted, path) + return nil + case inBefore != inAfter || be != ae: + return nil + case be.oid == p1.oid && (p0.mode == p1.mode || be.mode == p1.mode): + return nil + case !be.regular() || !p0.regular() || !p1.regular() || p0.mode != p1.mode: + diag = append(diag, "the mode or type of "+strconv.Quote(path)+" changes, which only a person can judge") + return nil + } + applied, unsupported, err := b.alreadyApplied(p0.oid, be.oid, p1.oid) + switch { + case err != nil: + return fmt.Errorf("judging %s of #%d: %w", strconv.Quote(path), v.o.number, err) + case unsupported: + diag = append(diag, "git cannot merge "+strconv.Quote(path)+" as text, so whether the line has its change is not known") + case !applied: + omitted = append(omitted, path) + } + return nil + } + for _, c := range v.changes { + cp, ok := corr[c.path] + if !ok { + continue + } + switch { + case c.status == 'D' && movedTo[c.path] != "": + if err := modified(c.path, cp, pre[c.path], post[cp]); err != nil { + return nil, nil, err + } + case c.status == 'D': + // Absent before: the line already made the deletion, which is + // no change of a unit that makes others, and is all of one + // that makes none. Still there after: the deletion did not + // happen, whatever the unit changed in the file; keeping what + // the line made of it is a port of its own. + _, inBefore := bef[cp] + _, inAfter := aft[cp] + if !inBefore && !same || inAfter { + omitted = append(omitted, c.path) + } + case c.status == 'M': + if err := modified(c.path, cp, pre[c.path], post[c.path]); err != nil { + return nil, nil, err + } + } + } + for i := range diag { + diag[i] = oneLineText(diag[i]) + } + return omitted, diag, nil +} + +// alreadyApplied says whether cur, a file the unit leaves alone, already +// holds the original's change from pre to post: it is post itself, or git +// merge-file of the three merges cleanly back into cur. unsupported is +// content merge-file refuses, such as differing binary files, which says +// nothing either way. +func (b *batch) alreadyApplied(pre, cur, post string) (applied, unsupported bool, err error) { + if cur == post { + return true, false, nil + } + b.mergeFiles++ + blobs, err := readObjects([]string{pre, cur, post}, "blob") + if err != nil { + return false, false, err + } + dir, err := os.MkdirTemp("", "backport-batch-merge-file-*") + if err != nil { + return false, false, err + } + defer os.RemoveAll(dir) + names := map[string]string{} + for _, f := range []struct{ name, oid string }{{"cur", cur}, {"pre", pre}, {"post", post}} { + names[f.name] = filepath.Join(dir, f.name) + if err := os.WriteFile(names[f.name], blobs[f.oid], 0o600); err != nil { + return false, false, err + } + } + stdout, stderr, code, err := gitStatus("merge-file", "-p", names["cur"], names["pre"], names["post"]) + switch { + case err != nil: + return false, false, err + case code == 0: + return stdout == string(blobs[cur]), false, nil + case code > 0 && code < 128: + // The number of conflicts. + return false, false, nil + case strings.Contains(stderr, "Cannot merge binary files"): + return false, true, nil + } + return false, false, fmt.Errorf("git merge-file: exit %d: %s", code, strings.TrimSpace(stderr)) +} + +// unitsOn maps each original the batch head carries a unit of to that unit, +// by its Backported-from trailer or appended -x line: found needs every +// commit of a merged original, and misses one whose first pick came out +// empty because the line already had that change. +func (b *batch) unitsOn(head string) (map[*verdict]string, error) { + if m, ok := b.unitCache[head]; ok { + return m, nil + } + units, err := batchUnits(head, b.lineHead) + if err != nil { + return nil, err + } + m := map[*verdict]string{} + for _, u := range units { + if v := b.unitOriginal(u); v != nil && m[v] == "" { + m[v] = u.commit + } + } + if b.unitCache == nil { + b.unitCache = map[string]map[*verdict]string{} + } + b.unitCache[head] = m + return m, nil +} + +// onBranch says whether the batch head h carries v. +func (b *batch) onBranch(h *history, v *verdict) (bool, error) { + if found(h, v) { + return true, nil + } + m, err := b.unitsOn(h.head) + return m[v] != "", err +} + +// partialStates finds what applies only in part on the line. Every unit on +// the open batch is checked against its own first parent, whatever the rules +// now say of its original: one that leaves an effect out fails the cut until +// it is taken out. Every original the policy takes and nothing carries is +// checked against the tree its clean net merge onto the line writes, so a +// partial one is found with no record and no adder; a conflict is never +// called partial. +func (b *batch) partialStates() error { + b.partialUnits = map[*verdict][]string{} + if b.pr != nil { + var err error + if b.partialUnits, err = b.unitsPartial(b.prHead); err != nil { + return err + } + on, err := b.unitsOn(b.prHead) + if err != nil { + return err + } + for v, commit := range on { + if v.rule == 0 && v.onBatch == "" { + v.onBatch = commit + } + } + } + for _, v := range b.qualifying() { + if v.onBatch != "" { + continue + } + if err := b.partialAt(v, b.lineHead); err != nil { + return err + } + } + return nil +} + +// unitsPartial maps each original whose unit between the line's head and +// head leaves part of it without effect to what it leaves. +func (b *batch) unitsPartial(head string) (map[*verdict][]string, error) { + units, err := batchUnits(head, b.lineHead) + if err != nil { + return nil, err + } + out := map[*verdict][]string{} + for _, u := range units { + v := b.unitOriginal(u) + if v == nil || len(u.parents) == 0 { + continue + } + omitted, diag, err := b.omittedEffects(v, u.parents[0], u.commit) + if err != nil { + return nil, err + } + v.addDiag(diag...) + if len(omitted) > 0 { + out[v] = append(out[v], omitted...) + } + } + return out, nil +} + +// markPartial puts every original on the branch at head whose unit there +// does only part of it among the partial units, and so among the stale +// ones: still on the batch, and to be taken out before the cut. +func (b *batch) markPartial(r *report, head string) error { + units := b.partialUnits + if b.pr == nil || head != b.prHead { + var err error + if units, err = b.unitsPartial(head); err != nil { + return err + } + } + r.partialFiles = map[*verdict][]string{} + for _, v := range b.verdicts { + files := units[v] + if len(files) == 0 { + continue + } + r.partial = append(r.partial, v) + r.partialFiles[v] = files + if !slices.Contains(r.stale, v) { + r.stale = append(r.stale, v) + } + } + return nil +} + +// partialAt judges v's clean net merge onto head, and parks it, or clears +// the park, by what that merge leaves without effect. +func (b *batch) partialAt(v *verdict, head string) error { + tree, err := b.netTree(head, v.o) + if err != nil { + return err + } + v.partial = nil + if v.park != nil && v.park.Kind == parkPartial { + v.park = nil + } + if tree == "" { + return nil + } + omitted, diag, err := b.omittedEffects(v, head, tree) + if err != nil { + return err + } + v.addDiag(diag...) + if len(omitted) > 0 { + v.partial = omitted + v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(omitted)} + } + return nil +} + +// netTree is the tree a clean net merge of o onto head writes, "" for a +// conflict; the line's head reuses what judging the originals merged. +func (b *batch) netTree(head string, o *original) (string, error) { + if head == b.lineHead && b.lineHist != nil { + return b.lineHist.netTree(o) + } + tree, _, err := noopPick(head, o) + return tree, err +} + +// addDiag keeps what could not be judged of v, each line once. +func (v *verdict) addDiag(lines ...string) { + for _, ln := range lines { + if ln != "" && !slices.Contains(v.diag, ln) { + v.diag = append(v.diag, ln) + } + } +} + +func sortedClean(files []string) []string { + out := cleanFiles(files) + slices.Sort(out) + return slices.Compact(out) +} + +// leftOut is the refusal for a unit that leaves paths of v without effect. +func leftOut(v *verdict, l line, omitted []string) string { + quoted := make([]string, len(omitted)) + for i, p := range omitted { + quoted[i] = strconv.Quote(p) + } + return fmt.Sprintf("#%d leaves %s without effect on %s; port only what the line has as a pull request of its own", v.o.number, strings.Join(quoted, ", "), l.branch()) +} diff --git a/cmd/backport-batch/effects_test.go b/cmd/backport-batch/effects_test.go new file mode 100644 index 000000000..082050cd4 --- /dev/null +++ b/cmd/backport-batch/effects_test.go @@ -0,0 +1,350 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "os" + "path/filepath" + "slices" + "strings" + "testing" +) + +const appFile = "one\ntwo\nthree\nfour\nfive\n" + +// renameWorld is the sign-off's rename history: the line keeps app/old.txt, +// main renames it to app/new.txt with #3, a feat, and #5 fixes app/new.txt. +func renameWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: an app file", "", map[string]string{"app/old.txt": appFile}) + f.lines() + f.pr(3, "feat(app): rename the app file", "", map[string]string{"app/old.txt": "", "app/new.txt": appFile}) + mc5 := f.pr(5, "fix(app): fix the app file", "", map[string]string{"app/new.txt": strings.Replace(appFile, "three", "THREE", 1)}) + return f, mc5 +} + +// emptyBatch pushes the line's head as the batch branch and opens a batch +// pull request on it with body, for a fold to land on. +func emptyBatch(f *fixture, gh *fakeGH, body string) *foldWorld { + f.t.Helper() + f.git("fetch", "-q", "origin") + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+batchBranch1) + f.git("fetch", "-q", "origin") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, body)} + return &foldWorld{f: f, gh: gh, head: f.git("rev-parse", "origin/"+batchBranch1)} +} + +// A fix of a file main renamed picks onto the file's old name on the line, +// and every check of the batch's shape takes the old name as the same file: +// verify, a hand fold, and a rebuild by --defer. +func TestRenamePickThroughFoldAndVerify(t *testing.T) { + f, mc5 := renameWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): fix the app file (#5)", "fix(app): six (#6)"}) { + t.Fatalf("the batch carries %q, want #5 and #6", got) + } + if got := f.git("show", head+":app/old.txt"); !strings.Contains(got, "THREE") { + t.Fatalf("#5 did not land on app/old.txt: %q", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + t.Setenv("GIT_COMMITTER_DATE", "2026-10-27T06:00:00Z") + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e", "--push"); code != 0 { + t.Fatalf("run --defer 6 --push = %d\n%s", code, stderr) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", f.remoteHead(batchBranch1)); code != 0 { + t.Errorf("verify after the rebuild = %d\n%s\n%s", code, out, stderr) + } + + f, mc5 = renameWorld(t) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/old.txt": strings.Replace(appFile, "three", "THREE", 1)}) + fold := w.merge(w.head, pick, "fix(app): fix the app file (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 || out != "ok\n" { + t.Errorf("fold of #5 onto app/old.txt = %d, %q\n%s", code, out, stderr) + } +} + +// shaped reads the line from the fixture's remote as verify does. +func shaped(t *testing.T, f *fixture) *batch { + t.Helper() + f.git("fetch", "-q", "origin") + b, err := shapeOf(line11) + if err != nil { + t.Fatal(err) + } + return b +} + +// commitOn writes files onto parent and returns the commit, leaving every +// branch where it was. +func commitOn(f *fixture, parent string, files map[string]string) string { + f.t.Helper() + f.git("switch", "-q", "--detach", parent) + f.apply(files) + f.git("commit", "-q", "--allow-empty", "-m", "a unit") + sha := f.git("rev-parse", "HEAD") + f.git("switch", "-q", "main") + return sha +} + +// A path of the original is followed to where the line keeps the same file: +// through a rename on main, on the line, or both, composed through the path +// at the base. Where git skipped rename detection the path is unknown, never +// absent. +func TestCorrespondence(t *testing.T) { + corr := func(t *testing.T, f *fixture, n int, limit ...int) (map[string]string, []string) { + t.Helper() + b := shaped(t, f) + if len(limit) > 0 { + b.renameLimit = limit[0] + } + v := b.find(n) + if err := b.load(v); err != nil { + t.Fatal(err) + } + c, unknown, err := b.correspondence(v, b.lineHead) + if err != nil { + t.Fatal(err) + } + return c, unknown + } + t.Run("renamed on main", func(t *testing.T) { + f, _ := renameWorld(t) + if c, unknown := corr(t, f, 5); c["app/new.txt"] != "app/old.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/new.txt on app/old.txt", c, unknown) + } + }) + t.Run("renamed on the line", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: an app file", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames it", map[string]string{"app/a.txt": "", "app/line.txt": appFile}) + f.pr(5, "fix(app): fix it", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + if c, unknown := corr(t, f, 5); c["app/a.txt"] != "app/line.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/a.txt on app/line.txt", c, unknown) + } + }) + t.Run("renamed on both", func(t *testing.T) { + f, _ := renameWorld(t) + f.onLine("release-1.1", "fix(app): the line renames it its own way", map[string]string{"app/old.txt": "", "app/release.txt": appFile}) + if c, unknown := corr(t, f, 5); c["app/new.txt"] != "app/release.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/new.txt on app/release.txt", c, unknown) + } + }) + t.Run("not renamed, absent", func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): x", "", map[string]string{"app/x.txt": appFile}) + f.pr(5, "fix(app): fix x", "", map[string]string{"app/x.txt": strings.Replace(appFile, "two", "TWO", 1)}) + if c, unknown := corr(t, f, 5); c["app/x.txt"] != "app/x.txt" || len(unknown) != 0 { + t.Errorf("correspondence = %v, %v; want app/x.txt on itself", c, unknown) + } + }) + t.Run("rename detection skipped", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: two app files", "", map[string]string{"app/a.txt": appFile, "app/b.txt": appFile + "six\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames both, changed", map[string]string{"app/a.txt": "", "app/b.txt": "", "app/x.txt": appFile + "x\n", "app/y.txt": appFile + "six\ny\n"}) + f.pr(5, "fix(app): fix a", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + // Past the limit, git says it skipped looking; a configured limit + // of its own does not decide for the tool. + f.git("config", "diff.renameLimit", "1") + if c, unknown := corr(t, f, 5, 0); c["app/a.txt"] != "app/x.txt" || len(unknown) != 0 { + t.Errorf("correspondence under diff.renameLimit=1 = %v, %v; want app/a.txt on app/x.txt", c, unknown) + } + if c, unknown := corr(t, f, 5, 1); !slices.Equal(unknown, []string{"app/a.txt"}) || c["app/a.txt"] != "" { + t.Errorf("correspondence past the limit = %v, %v; want app/a.txt unknown", c, unknown) + } + }) +} + +// What a change leaves without effect: a modified file the line lacks on both +// sides of it, a deleted file the line never had. A mode change that only a +// person can judge is a diagnostic, not an omission. +func TestOmittedEffects(t *testing.T) { + omitted := func(t *testing.T, f *fixture, n int, files map[string]string) ([]string, []string) { + t.Helper() + b := shaped(t, f) + after := commitOn(f, b.lineHead, files) + om, diag, err := b.omittedEffects(b.find(n), b.lineHead, after) + if err != nil { + t.Fatal(err) + } + return om, diag + } + t.Run("a modified file missing before and after", func(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): x", "", map[string]string{"app/x.txt": appFile}) + f.pr(5, "fix(app): fix x and five", "", map[string]string{"app/x.txt": "fixed\n", "app/five.txt": "5\n"}) + if om, diag := omitted(t, f, 5, map[string]string{"app/five.txt": "5\n"}); !slices.Equal(om, []string{"app/x.txt"}) || len(diag) != 0 { + t.Errorf("omitted = %q, %q; want app/x.txt", om, diag) + } + }) + t.Run("a deleted file absent before", func(t *testing.T) { + f, _ := policyWorld(t) + if om, _ := omitted(t, f, 5, map[string]string{"chart/tests/values_test.yaml": "suite: values\n"}); !slices.Equal(om, []string{"chart/templates/policy.yaml", "chart/tests/policy_test.yaml"}) { + t.Errorf("omitted = %q; want both policy files", om) + } + }) + t.Run("a mode change", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a script", "", map[string]string{"app/run.sh": "#!/bin/sh\n", "app/b.txt": "old\n"}) + f.lines() + f.git("switch", "-q", "-c", "pr-5") + if err := os.Chmod(filepath.Join(f.dir, "app/run.sh"), 0o755); err != nil { + t.Fatal(err) + } + f.write("app/b.txt", "fixed\n") + f.git("add", "-A") + f.git("commit", "-q", "-m", "run it") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): make the script runnable (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + om, diag := omitted(t, f, 5, map[string]string{"app/b.txt": "fixed\n"}) + if len(om) != 0 || len(diag) != 1 || !strings.Contains(diag[0], `the mode or type of "app/run.sh" changes`) { + t.Errorf("omitted = %q, %q; want a diagnostic for app/run.sh and nothing omitted", om, diag) + } + }) +} + +// fixedWorld is the sign-off's two-file case: the base has a and b as old, +// #5 fixes both, and the line already has a as onLine says, if anything. +func fixedWorld(t *testing.T, a, old, onLine, fixed string) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{a: old, "app/b.txt": "old\n"}) + f.lines() + if onLine != "" { + f.onLine("release-1.1", "fix(app): a, on the line", map[string]string{a: onLine}) + } + mc5 := f.pr(5, "fix(app): a and b", "", map[string]string{a: fixed, "app/b.txt": "fixed\n"}) + return f, mc5 +} + +// The line can already have what an original does to a file: then a pick +// that leaves the file alone carries the original whole. The check-time +// pass, the picker, fold and verify all take it, binary files included, +// without asking git merge-file what blob equality settles. A different fix +// on the line, or a deletion of what the line never had, is still left out. +func TestAlreadyAppliedFileIsNotOmitted(t *testing.T) { + const textPost = "one\nfixed\nthree\nfour\nfive\n" + accepted := func(t *testing.T, a, old, onLine, fixed string) { + t.Helper() + f, _ := fixedWorld(t, a, old, onLine, fixed) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check = %d, want #5 missing and not parked:\n%s\n%s", code, stdout, stderr) + } + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a and b (#5)"}) { + t.Errorf("picked %q, want #5", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || strings.Contains(stdout, "only part of it applies") { + t.Errorf("check --cut = %d, want #5 only on the batch, whole:\n%s\n%s", code, stdout, stderr) + } + f, mc5 := fixedWorld(t, a, old, onLine, fixed) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/b.txt": "fixed\n"}) + fold := w.merge(w.head, pick, "fix(app): a and b (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 { + t.Errorf("fold = %d, %q\n%s", code, out, stderr) + } + } + t.Run("the line has the original's file", func(t *testing.T) { + accepted(t, "app/a.txt", "old\n", "fixed\n", "fixed\n") + }) + t.Run("the line has the fix beside lines of its own", func(t *testing.T) { + accepted(t, "app/a.txt", strings.Replace(textPost, "fixed", "old", 1), textPost+"line only\n", textPost) + }) + t.Run("the line has the original's binary file", func(t *testing.T) { + accepted(t, "app/a.bin", "old\x00payload", "fixed\x00payload", "fixed\x00payload") + f, _ := fixedWorld(t, "app/a.bin", "old\x00payload", "fixed\x00payload", "fixed\x00payload") + b := shaped(t, f) + om, diag, err := b.omittedEffects(b.find(5), b.lineHead, commitOn(f, b.lineHead, map[string]string{"app/b.txt": "fixed\n"})) + if err != nil || len(om)+len(diag) != 0 || b.mergeFiles != 0 { + t.Errorf("omitted = %q, %q, %v, merge-file ran %d times; want nothing, and merge-file never asked", om, diag, err, b.mergeFiles) + } + }) + t.Run("an earlier unit of the batch brought the file", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old\n", "app/b.txt": "old\n"}) + f.lines() + f.pr(4, "fix(app): a", "", map[string]string{"app/a.txt": "fixed\n"}) + f.squash(40, "chore: put a back for now", "", map[string]string{"app/a.txt": "old\n"}) + f.pr(5, "fix(app): a and b", "", map[string]string{"app/a.txt": "fixed\n", "app/b.txt": "fixed\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a (#4)", "fix(app): a and b (#5)"}) { + t.Errorf("picked %q, want #4 and #5", got) + } + if code, out, stderr := f.exec(gh, "verify", "--line", "release-1.1", "--head", head); code != 0 { + t.Errorf("verify = %d\n%s\n%s", code, out, stderr) + } + }) + t.Run("the line has a different fix", func(t *testing.T) { + f, mc5 := fixedWorld(t, "app/a.txt", "old\n", "other\n", "fixed\n") + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/b.txt": "fixed\n"}) + fold := w.merge(w.head, pick, "fix(app): a and b (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/a.txt" without effect`) { + t.Errorf("fold = %d, %q, want it refused for app/a.txt\n%s", code, out, stderr) + } + }) + t.Run("the line has a different binary file", func(t *testing.T) { + f, _ := fixedWorld(t, "app/a.bin", "old\x00payload", "other\x00payload", "fixed\x00payload") + b := shaped(t, f) + om, diag, err := b.omittedEffects(b.find(5), b.lineHead, commitOn(f, b.lineHead, map[string]string{"app/b.txt": "fixed\n"})) + if err != nil || len(om) != 0 || len(diag) != 1 || !strings.Contains(diag[0], `cannot merge "app/a.bin" as text`) { + t.Errorf("omitted = %q, %q, %v; want a diagnostic for app/a.bin, nothing omitted", om, diag, err) + } + }) + t.Run("a deletion of what the line never had", func(t *testing.T) { + f, mc5 := policyWorld(t) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"chart/tests/values_test.yaml": "suite: values\n"}) + fold := w.merge(w.head, pick, "fix(chart): remove the policy a rebase brought back (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, "without effect") { + t.Errorf("fold = %d, %q, want it refused\n%s", code, out, stderr) + } + }) +} + +// A change that makes none, whatever names its two sides, leaves a deletion +// the line already made as it is: the original is there, not partial. +func TestDeletionAlreadyMadeByNoChange(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(5, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": ""}) + b := shaped(t, f) + tree := f.git("rev-parse", b.lineHead+"^{tree}") + for _, after := range []string{b.lineHead, tree} { + if om, _, err := b.omittedEffects(b.find(5), b.lineHead, after); err != nil || len(om) != 0 { + t.Errorf("omitted against %.12s = %q, %v; want nothing", after, om, err) + } + } +} diff --git a/cmd/backport-batch/fold_test.go b/cmd/backport-batch/fold_test.go index 89797d0d5..5c064c2e3 100644 --- a/cmd/backport-batch/fold_test.go +++ b/cmd/backport-batch/fold_test.go @@ -348,3 +348,59 @@ func TestFoldRefusesADroppedOriginal(t *testing.T) { t.Errorf("fold = %d, %q, %s; want a refusal naming the drop", code, stdout, stderr) } } + +// A resolution that ports what the line has of #5 and leaves alone a file +// the line lacks is no fold: what #5 does to that file would be lost, and a +// line-only port is a pull request of its own. +func TestFoldRefusesAnOmittedEffect(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): a feature the line never gets", "", map[string]string{"app/feature2.txt": "v1\n"}) + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/feature2.txt": "v2\n", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): five (#5)\n\n"+batchTrailer+": "+mc5) + code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold) + if code != 1 || out != "refused: #5 leaves \"app/feature2.txt\" without effect on release-1.1; port only what the line has as a pull request of its own\n" { + t.Errorf("fold = %d, %q, want it refused for app/feature2.txt\n%s", code, out, stderr) + } +} + +// A resolution that keeps a file the original deletes leaves the deletion +// without effect. +func TestFoldRefusesAKeptDeletion(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: x", "", map[string]string{"app/x.txt": "x\n"}) + f.lines() + mc5 := f.pr(5, "fix(app): five, and x goes", "", map[string]string{"app/x.txt": "", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): five, and x goes (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/x.txt" without effect`) { + t.Errorf("fold that keeps app/x.txt = %d, %q\n%s", code, out, stderr) + } +} + +// A fold that keeps a file the original deletes leaves the deletion without +// effect, whatever it changes in the file: a port that keeps what the line +// made of it, or the file with a line added, is a pull request of its own. +func TestFoldRefusesAChangedKeptDeletion(t *testing.T) { + for name, kept := range map[string]string{ + "the line's own content": "line only\n", + "the file, one line more": "feature\nline only\n# reviewed\n", + } { + t.Run(name, func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a feature", "", map[string]string{"app/x.txt": "feature\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): a line-only setting beside the feature", map[string]string{"app/x.txt": "feature\nline only\n"}) + mc5 := f.pr(5, "fix(app): drop the feature, and five", "", map[string]string{"app/x.txt": "", "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/x.txt": kept, "app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): drop the feature, and five (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/x.txt" without effect`) { + t.Errorf("fold = %d, %q, want it refused\n%s", code, out, stderr) + } + }) + } +} diff --git a/cmd/backport-batch/history.go b/cmd/backport-batch/history.go index 74f4c66af..a530aa244 100644 --- a/cmd/backport-batch/history.go +++ b/cmd/backport-batch/history.go @@ -205,6 +205,9 @@ type history struct { picks []pickRef batches map[string]string backports map[int]string + // trees caches what a net merge of an original onto head writes, by its + // merge commit: "" for a conflict. + trees map[string]string } func readHistory(head, base string, textUntil time.Time) (*history, error) { @@ -225,7 +228,7 @@ func readHistory(head, base string, textUntil time.Time) (*history, error) { // readPicks reads only the messages of head's commits outside base, which is // all a batch branch is asked about; textUntil is the batch-only day. func readPicks(head, base string, textUntil time.Time) (*history, error) { - h := &history{head: head, reachable: map[string]bool{}, appended: map[string]string{}, batches: map[string]string{}, backports: map[int]string{}} + h := &history{head: head, reachable: map[string]bool{}, appended: map[string]string{}, batches: map[string]string{}, backports: map[int]string{}, trees: map[string]string{}} oids, err := revList("--end-of-options", head, "^"+base) if err != nil { return nil, err @@ -332,34 +335,54 @@ func (h *history) carries(o *original, commits []string) (string, error) { if c, ok := h.backports[o.number]; ok { return `named by "Backport of" in ` + c[:12], nil } - noop, err := noopPick(h.head, o) + tree, err := h.netTree(o) if err != nil { return "", err } - if noop { + headTree, err := resolveTree(h.head) + if err != nil { + return "", err + } + if tree == headTree { return "a pick would change nothing", nil } return "", nil } -// noopPick reports whether cherry-picking the original onto head would leave -// head's tree as it is: the content is there, whatever names it. -func noopPick(head string, o *original) (bool, error) { +// netTree is noopPick's tree for o onto the branch's head, read once. +func (h *history) netTree(o *original) (string, error) { + if tree, ok := h.trees[o.merge]; ok { + return tree, nil + } + tree, _, err := noopPick(h.head, o) + if err == nil { + h.trees[o.merge] = tree + } + return tree, err +} + +// noopPick merges the original onto head as one change and returns the tree +// that writes for a clean merge, "" for a conflict, and whether the tree is +// head's own: the content is there, whatever names it. +func noopPick(head string, o *original) (string, bool, error) { stdout, stderr, code, err := gitStatus("merge-tree", "--write-tree", "--no-messages", "--merge-base="+o.merge+"^1", "--end-of-options", head, o.merge) switch { case err != nil: - return false, err + return "", false, err case code == 1: - return false, nil + return "", false, nil case code != 0: - return false, fmt.Errorf("git merge-tree for #%d: exit %d: %s", o.number, code, strings.TrimSpace(stderr)) + return "", false, fmt.Errorf("git merge-tree for #%d: exit %d: %s", o.number, code, strings.TrimSpace(stderr)) } tree, _, _ := strings.Cut(stdout, "\n") + if err := checkSHA(tree); err != nil { + return "", false, fmt.Errorf("git merge-tree for #%d: %w", o.number, err) + } headTree, err := resolveTree(head) if err != nil { - return false, err + return "", false, err } - return tree == headTree, nil + return tree, tree == headTree, nil } func resolveTree(commit string) (string, error) { diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index c02601240..161df97b9 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -211,6 +211,13 @@ type verdict struct { onBatch string // label is the maintainer override the original carries, if any. label string + // partial lists what its clean net merge onto the line would leave + // without effect, and park why the batch leaves it out, if it does. + partial []string + park *parkedEntry + // diag is what could not be judged about it, shown whatever became of + // it. + diag []string } // batch is everything read about a line before anything is written. @@ -242,6 +249,16 @@ type batch struct { // textUntil is the batch-only day: what a commit's text says counts as // evidence only on commits from before it. textUntil time.Time + // partialUnits are the originals whose unit on the open batch leaves + // what it lists without effect. + partialUnits map[*verdict][]string + // What one command read once and asks again: renames by commit pair, + // units by batch head, and how often git merge-file ran. + renameCache map[string]renameSet + unitCache map[string]map[*verdict]string + mergeFiles int + // renameLimit is the -l rename detection runs with; 0, git's widest. + renameLimit int } // fetch moves to the top of the working tree first: every path the batch @@ -355,6 +372,9 @@ func (a *app) prepare(l line) (*batch, error) { } b.verdicts = append(b.verdicts, v) } + if err := b.partialStates(); err != nil { + return nil, err + } return b, nil } diff --git a/cmd/backport-batch/park_test.go b/cmd/backport-batch/park_test.go new file mode 100644 index 000000000..6c8f51d0a --- /dev/null +++ b/cmd/backport-batch/park_test.go @@ -0,0 +1,331 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "slices" + "strings" + "testing" +) + +var line11, _ = parseLine("release-1.1") + +// handBatch pushes a batch carrying merges in the batch's shape, as a run +// made one before the partial guard, and opens its pull request with a +// record that defers nothing. +func handBatch(f *fixture, gh *fakeGH, merges ...string) string { + f.t.Helper() + f.git("fetch", "-q", "origin") + f.git("switch", "-q", "--detach", "origin/release-1.1") + for _, mc := range merges { + if parents := strings.Fields(f.git("rev-list", "--parents", "-n", "1", mc)); len(parents) == 2 { + f.git("cherry-pick", "-x", mc) + continue + } + base := f.git("rev-parse", "HEAD") + for _, c := range strings.Fields(f.git("rev-list", "--reverse", "--no-merges", mc+"^1.."+mc+"^2")) { + f.git("cherry-pick", "-x", c) + } + merge := f.joined(base, "HEAD", f.git("log", "-1", "--format=%B", mc)+"\n\n"+batchTrailer+": "+mc) + f.git("update-ref", "HEAD", merge) + } + f.git("push", "-q", "origin", "HEAD:refs/heads/"+batchBranch1) + f.git("switch", "-q", "main") + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+deferredBlock(line11, nil, nil, nil)+"\n")} + return f.remoteHead(batchBranch1) +} + +// policyWorld is #5088's shape: #3, a feat the line never gets, adds a +// policy and its test; #5 deletes both again and adds a test of its own. +func policyWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(3, "feat(chart): a network policy", "", map[string]string{"chart/templates/policy.yaml": "kind: NetworkPolicy\n", "chart/tests/policy_test.yaml": "suite: policy\n"}) + mc5 := f.pr(5, "fix(chart): remove the policy a rebase brought back", "```release-note\nfix(chart): the stray network policy is gone.\n```\n", + map[string]string{"chart/templates/policy.yaml": "", "chart/tests/policy_test.yaml": "", "chart/tests/values_test.yaml": "suite: values\n"}) + return f, mc5 +} + +// A clean pick that would delete files the line never had, and so carry only +// #5's test, is withheld: #5 is parked as partial, with the paths it would +// leave without effect, and its release note stays out of the body. +func TestPartialPickIsWithheld(t *testing.T) { + f, mc5 := policyWorld(t) + f.pr(6, "fix(app): six", "```release-note\nfix(app): six works.\n```\n", map[string]string{"app/six.txt": "6\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): six (#6)"}) { + t.Errorf("picked %q, want only #6", got) + } + want := `"parked":[{"original":5,"merge":"` + mc5 + `","kind":"partial","files":["chart/templates/policy.yaml","chart/tests/policy_test.yaml"]}]` + if !strings.Contains(stdout, want) { + t.Errorf("the record does not park #5 as partial (%s):\n%s", want, stdout) + } + if !strings.Contains(stdout, "### Parked") || !strings.Contains(stdout, "`chart/templates/policy.yaml`") { + t.Errorf("the body does not say #5 is parked and why:\n%s", stdout) + } + if strings.Contains(stdout, "stray network policy") || !strings.Contains(stdout, "six works") { + t.Errorf("the release notes are not #6's alone:\n%s", stdout) + } +} + +// A file the line removed itself, which no original added after the line +// branched, still leaves #5's deletion without effect: check finds it with no +// record and parks it, and run picks nothing. +func TestPartialWithoutAnAdder(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: add a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(5, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); len(got) != 0 { + t.Errorf("picked %q, want nothing", got) + } + if !strings.Contains(stdout, "### Parked") { + t.Errorf("the body does not park #5:\n%s", stdout) + } + f.git("switch", "-q", "main") + code, stdout, stderr = f.exec(nil, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "note: #5 is parked: only part of it applies to release-1.1") { + t.Errorf("check = %d, want 0 with a parked-partial note:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } +} + +// The picker judges what its picks did, not what the original's merge did: +// #5's first commit deletes a file the line no longer has and comes out +// empty, its second adds a test, and the unit left would be the test alone. +func TestPartialByAnEmptyOwnCommit(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: add a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.prCommits(5, "fix(app): drop the policy and test it", "```release-note\nfix(app): no more policy.\n```\n", + change{files: map[string]string{"app/policy.yaml": ""}}, + change{files: map[string]string{"app/policy_test.txt": "gone\n"}}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); len(got) != 0 { + t.Errorf("picked %q, want #5 withheld", got) + } + if strings.Contains(stdout, "no more policy") || !strings.Contains(stdout, "### Parked") { + t.Errorf("#5's note is in the body, or #5 is not parked:\n%s", stdout) + } +} + +// A batch built before the guard can already carry a partial unit. It fails +// the cut and verify whatever its label says, and --defer takes it off, +// closing a batch it was the last unit of. +func TestExistingPartialUnitFailsTheCut(t *testing.T) { + f, mc5 := policyWorld(t) + gh := &fakeGH{} + head := handBatch(f, gh, mc5) + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is on batch #90, but only part of it applies to release-1.1") { + t.Errorf("check --cut = %d, want 1 naming the partial unit:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, stdout, stderr = f.exec(gh, "verify", "--line", "release-1.1", "--head", head) + if code != 1 || !strings.Contains(stdout, `#5 leaves "chart/templates/policy.yaml", "chart/tests/policy_test.yaml" without effect on release-1.1`) { + t.Errorf("verify = %d, want 1 naming what #5 leaves out:\n%s\n%s", code, stdout, stderr) + } + gh.labels = map[string][]int{labelSkip: {5}} + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is on batch #90, but only part of it applies to release-1.1") { + t.Errorf("check --cut with backport/skip = %d, want the partial unit to still fail it:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, _, stderr = f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "only part of it applies", "--push") + if code != 0 || prState(gh.find(90)) != "closed" || f.remoteHead(batchBranch1) != "" { + t.Errorf("run --defer 5 --push = %d, #90 %s, branch at %q; want the batch closed and its branch gone:\n%s", code, prState(gh.find(90)), f.remoteHead(batchBranch1), stderr) + } +} + +// An original whose first pick came out empty, because the line already had +// that change, is on the batch all the same: check does not call it missing, +// a continuation keeps it picked, and so does a rebuild without another +// original. +func TestOnBatchAfterAnEmptyFirstPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old\n", "app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): a, on the line first", map[string]string{"app/a.txt": "fixed\n"}) + f.prCommits(5, "fix(app): a and b", "", + change{files: map[string]string{"app/a.txt": "fixed\n"}}, + change{files: map[string]string{"app/b.txt": "fixed\n"}}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + head := openBatch(t, f, gh) + if got := f.subjects("origin/release-1.1", head); !slices.Equal(got, []string{"fix(app): a and b (#5)", "fix(app): six (#6)"}) { + t.Fatalf("the batch carries %q, want #5 and #6", got) + } + code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1") + if code != 0 { + t.Errorf("check = %d, want 0 with #5 on the batch:\nstdout:\n%s\nstderr:\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "check", "--line", "release-1.1", "--cut") + if code != 1 || !strings.Contains(stdout, "#5 is only on open batch PR #90") { + t.Errorf("check --cut = %d, want #5 only on the batch:\n%s", code, stdout) + } + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[5,6]`) { + t.Errorf("run --continue = %d, want #5 and #6 picked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e") + if code != 0 || !strings.Contains(stdout, `"picked":[5]`) || !strings.Contains(stdout, `"missing":[]`) { + t.Errorf("run --defer 6 = %d, want #5 still picked:\n%s\n%s", code, stdout, stderr) + } +} + +// The picker judges what its picks did where the run has got to: #4 +// deletes the file #5 also deletes, so after #4 #5 would bring only its +// test, which the net merge at the line's head cannot see. +func TestPartialByAnEarlierPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.pr(4, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/four.txt": "4\n"}) + f.pr(40, "feat(app): the policy again", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"picked":[4]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #4 picked and #5 parked as partial:\n%s\n%s", code, stdout, stderr) + } +} + +// What an earlier pick brought clears a partial park the line's head +// gave: the line removed the policy, #4 brings it back, and #5's deletion +// then has something to delete. +func TestPartialClearedByAnEarlierPick(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): the line drops the policy", map[string]string{"app/policy.yaml": ""}) + f.pr(3, "feat(app): no policy", "", map[string]string{"app/policy.yaml": ""}) + f.pr(4, "fix(app): the policy, back", "", map[string]string{"app/policy.yaml": "allow: some\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"picked":[4,5]`) { + t.Errorf("run = %d, want #4 and #5 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// A file the line renamed the way the original does is the original's own +// rename, not a deletion left without effect. +func TestSameRenameOnBothSidesIsWhole(t *testing.T) { + const long = "l01\nl02\nl03\nl04\nl05\nl06\nl07\nl08\nl09\nl10\nl11\nl12\n" + f := newFixture(t) + f.squash(2, "chore: a file", "", map[string]string{"app/p.txt": long}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames p to q", map[string]string{"app/p.txt": "", "app/q.txt": long}) + f.pr(5, "fix(app): rename p to q, and fix l10", "", map[string]string{"app/p.txt": "", "app/q.txt": strings.Replace(long, "l10", "L10", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked whole:\n%s\n%s", code, stdout, stderr) + } +} + +// The line's own rename stands for the original's only with the original's +// change in it: a fold that leaves the renamed file as it was is refused. +func TestSameRenameStillNeedsItsFix(t *testing.T) { + const long = "l01\nl02\nl03\nl04\nl05\nl06\nl07\nl08\nl09\nl10\nl11\nl12\n" + f := newFixture(t) + f.squash(2, "chore: a file", "", map[string]string{"app/p.txt": long}) + f.lines() + f.onLine("release-1.1", "fix(app): the line renames p to q", map[string]string{"app/p.txt": "", "app/q.txt": long}) + mc5 := f.pr(5, "fix(app): rename p to q, fix l10, and five", "", map[string]string{"app/p.txt": "", "app/q.txt": strings.Replace(long, "l10", "L10", 1), "app/five.txt": "5\n"}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/five.txt": "5\n"}) + fold := w.merge(w.head, pick, "fix(app): rename p to q, fix l10, and five (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 1 || !strings.Contains(out, `leaves "app/p.txt" without effect`) { + t.Errorf("fold without the fix in q = %d, %q\n%s", code, out, stderr) + } +} + +// A pick that comes out empty is judged too: #5's own commit did half its +// fix, which the line has, and a merge of main into its branch did the +// rest, which no pick carries; #5 is not on the line. +func TestEmptyPicksStillJudged(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: b", "", map[string]string{"app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): b, half of it", map[string]string{"app/b.txt": "middle\n"}) + f.git("switch", "-q", "-c", "pr-5", "main") + f.apply(map[string]string{"app/b.txt": "middle\n"}) + f.git("commit", "-q", "-m", "b, half of it") + f.git("switch", "-q", "main") + f.apply(map[string]string{"app/main.txt": "moved\n"}) + f.git("commit", "-q", "-m", "chore: move main along") + f.git("switch", "-q", "pr-5") + f.git("merge", "-q", "--no-ff", "--no-commit", "main") + f.apply(map[string]string{"app/b.txt": "fixed\n"}) + f.git("commit", "-q", "-m", "Merge branch 'main' into pr-5") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): b (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"on-line":[5]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) + } +} + +// --defer takes off a unit that does only part of its original whatever +// rule now keeps that original out, as check tells it to. +func TestDeferTakesOffAPartialExcludedUnit(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(3, "feat(chart): a network policy", "", map[string]string{"chart/templates/policy.yaml": "kind: NetworkPolicy\n"}) + mc5 := f.pr(5, "feat(chart): remove the policy, and test values", "", map[string]string{"chart/templates/policy.yaml": "", "chart/tests/values_test.yaml": "suite: values\n"}) + gh := &fakeGH{} + handBatch(f, gh, mc5) + if code, stdout, _ := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "only part of it applies") { + t.Fatalf("check --cut = %d, want the partial unit to fail it:\n%s", code, stdout) + } + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "only part of it applies", "--push"); code != 0 || prState(gh.find(90)) != "closed" { + t.Errorf("run --defer 5 --push = %d, #90 %s; want the unit taken off and the batch closed:\n%s", code, prState(gh.find(90)), stderr) + } +} + +// An empty pick leaves a deletion undone that only a merge of main into the +// original's branch made: the original is not on the line. +func TestEmptyPickLeavesADeletionUndone(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": "old feature\n", "app/b.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): b, already", map[string]string{"app/b.txt": "fixed\n"}) + f.git("switch", "-q", "-c", "pr-5", "main") + f.apply(map[string]string{"app/b.txt": "fixed\n"}) + f.git("commit", "-q", "-m", "b") + f.git("switch", "-q", "main") + f.apply(map[string]string{"app/main.txt": "moved\n"}) + f.git("commit", "-q", "-m", "chore: move main along") + f.git("switch", "-q", "pr-5") + f.git("merge", "-q", "--no-ff", "--no-commit", "main") + f.git("rm", "-q", "app/a.txt") + f.git("commit", "-q", "-m", "Merge branch 'main' into pr-5") + f.git("switch", "-q", "main") + f.git("merge", "-q", "--no-ff", "-m", "fix(app): b, and a goes (#5)", "pr-5") + f.git("push", "-q", "origin", "main") + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"on-line":[5]`) || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 4ab43933e..313279670 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -34,7 +34,8 @@ import ( // carries a record of its own. // lineRecord is the one line of JSON in the block. Version 1, the block -// before it had a version, holds only line and deferred. +// before it had a version, holds only line and deferred; version 3 adds what +// is parked and which units on the batch do only part of their original. type lineRecord struct { Version int `json:"version"` Line string `json:"line"` @@ -44,6 +45,80 @@ type lineRecord struct { Deferred []deferredEntry `json:"deferred"` Dropped []droppedEntry `json:"dropped"` Stale []int `json:"stale"` + Parked []parkedEntry `json:"parked,omitempty"` + Partial []int `json:"partial,omitempty"` +} + +// recordVersion is the version a run writes. +const recordVersion = 3 + +// lineRecordV2 is version 2, which has none of version 3's keys. +type lineRecordV2 struct { + Version int `json:"version"` + Line string `json:"line"` + Built *builtFrom `json:"built,omitempty"` + Report string `json:"report,omitempty"` + Closing *closing `json:"closing"` + Deferred []deferredEntry `json:"deferred"` + Dropped []droppedEntry `json:"dropped"` + Stale []int `json:"stale"` +} + +// The kinds of park: an original that builds on what never reaches the +// line, and one whose clean pick would leave part of it without effect. +const ( + parkDependency = "dependency" + parkPartial = "partial" +) + +// parkedEntry is an original the batch leaves out because it cannot land on +// the line whole: what it needs that never comes, or the paths it would +// leave without effect. announced says notify has named it on the open +// batch, for exactly this entry. +type parkedEntry struct { + Original int `json:"original"` + Merge string `json:"merge"` + Kind string `json:"kind"` + Needs []int `json:"needs,omitempty"` + Commits []string `json:"commits,omitempty"` + Files []string `json:"files,omitempty"` + Announced bool `json:"announced,omitempty"` +} + +// validate holds a parked entry to what a run writes. Its errors read after +// "the block" or "report ". +func (p parkedEntry) validate() error { + sortedUnique := func(n int, less func(i int) bool) bool { + for i := 1; i < n; i++ { + if !less(i) { + return false + } + } + return true + } + switch { + case checkSHA(p.Merge) != nil: + return fmt.Errorf("parks #%d by merge %q", p.Original, p.Merge) + case !sortedUnique(len(p.Needs), func(i int) bool { return p.Needs[i-1] < p.Needs[i] }) || slices.ContainsFunc(p.Needs, func(n int) bool { return n <= 0 || n == p.Original }): + return fmt.Errorf("parks #%d for originals %v, which are not what a run writes", p.Original, p.Needs) + case !sortedUnique(len(p.Commits), func(i int) bool { return p.Commits[i-1] < p.Commits[i] }) || slices.ContainsFunc(p.Commits, func(c string) bool { return checkSHA(c) != nil }): + return fmt.Errorf("parks #%d for commits %q, which are not what a run writes", p.Original, p.Commits) + case !sortedUnique(len(p.Files), func(i int) bool { return p.Files[i-1] < p.Files[i] }) || !slices.Equal(p.Files, cleanFiles(p.Files)): + return fmt.Errorf("parks #%d for files %q, which are not what a run writes", p.Original, p.Files) + } + switch p.Kind { + case parkPartial: + if len(p.Files) == 0 || len(p.Needs)+len(p.Commits) > 0 { + return fmt.Errorf("parks #%d as partial with no files it leaves without effect, or with what it needs", p.Original) + } + case parkDependency: + if len(p.Needs)+len(p.Commits) == 0 || len(p.Files) > 0 { + return fmt.Errorf("parks #%d for a dependency it does not name, or with files", p.Original) + } + default: + return fmt.Errorf("parks #%d as %q, which is no kind of park", p.Original, p.Kind) + } + return nil } // builtFrom is what a classification was made of; batch is the head it @@ -173,9 +248,13 @@ func parseRecord(body string, l line) (*lineRecord, error) { err = decodeStrict(blocks[0], &v1) rec = lineRecord{Version: 1, Line: v1.Line, Deferred: v1.Deferred} case 2: + var v2 lineRecordV2 + err = decodeStrict(blocks[0], &v2) + rec = lineRecordWithout(v2) + case 3: err = decodeStrict(blocks[0], &rec) default: - return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 and 2", deferredFence, version) + return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 to %d", deferredFence, version, recordVersion) } if err != nil { return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) @@ -244,6 +323,14 @@ func (rec *lineRecord) validate(l line) error { return fmt.Errorf("drops #%d for %q, which is no reason a run writes", d.Original, d.Reason) } } + for _, p := range rec.Parked { + if err := once(p.Original); err != nil { + return err + } + if err := p.validate(); err != nil { + return err + } + } staleSeen := map[int]bool{} for _, n := range rec.Stale { if n <= 0 || staleSeen[n] { @@ -251,6 +338,14 @@ func (rec *lineRecord) validate(l line) error { } staleSeen[n] = true } + partialSeen := map[int]bool{} + for _, n := range rec.Partial { + // A unit that does only part of its original is still on the batch. + if !staleSeen[n] || partialSeen[n] { + return fmt.Errorf("lists #%d as a partial unit more than once, or not as still on the batch", n) + } + partialSeen[n] = true + } if rec.Version < 2 { return nil } @@ -297,6 +392,11 @@ func (b *builtFrom) validate() error { var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) +// lineRecordWithout is a version 2 record as the newest version holds it. +func lineRecordWithout(v2 lineRecordV2) lineRecord { + return lineRecord{Version: v2.Version, Line: v2.Line, Built: v2.Built, Report: v2.Report, Closing: v2.Closing, Deferred: v2.Deferred, Dropped: v2.Dropped, Stale: v2.Stale} +} + // recordBlock is the block that holds rec. One line of JSON: encoding/json // escapes newlines and <, > and &, so nothing in a file name or a reason can // end the fence or open HTML. diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index 13ab6cc79..ed09d1cd2 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -18,6 +18,7 @@ package main import ( "fmt" + "slices" "strings" "testing" ) @@ -272,7 +273,7 @@ func TestRecordNamesItsInputs(t *testing.T) { want := builtFrom{Main: f.git("rev-parse", "origin/main"), Line: f.git("rev-parse", "origin/release-1.0"), Newer: f.git("rev-parse", "origin/release-1.1"), Policy: f.git("rev-parse", "origin/main"), Batch: f.remoteHead(branch), Tool: f.tool, Run: "https://github.com/cozystack/cozystack/actions/runs/123/attempts/2", At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: true, Mode: modeNew}} - if rec.Version != 2 || *rec.Built != want || bodyReportOf(t, stdout, l).doc.Built != want || rec.Report != bodyReportOf(t, stdout, l).doc.ID { + if rec.Version != recordVersion || *rec.Built != want || bodyReportOf(t, stdout, l).doc.Built != want || rec.Report != bodyReportOf(t, stdout, l).doc.ID { t.Errorf("record = %+v, built %+v; want built %+v and the report it names", rec, rec.Built, want) } @@ -366,3 +367,43 @@ func TestPRStateNamesOnlyKnownStates(t *testing.T) { } } } + +// v3 is v2 as version 3, with extra after the stale list. +func v3(extra string) string { + return strings.Replace(v2(extra), `"version":2`, `"version":3`, 1) +} + +// Version 3 adds what is parked and which units do only part of their +// original, each held to what a run writes, and version 2 has neither. +func TestRecordVersion3(t *testing.T) { + l, _ := parseLine("release-1.1") + block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } + sha, other := strings.Repeat("a", 40), strings.Repeat("c", 40) + parked := `,"parked":[{"original":10,"merge":"` + sha + `","kind":"partial","files":["a","b"]},{"original":11,"merge":"` + sha + `","kind":"dependency","needs":[3,4],"commits":["` + other + `"],"announced":true}],"partial":[9]` + rec, err := parseRecord(block(v3(parked)), l) + if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) { + t.Fatalf("parseRecord of a whole v3 record = %+v, %v", rec, err) + } + for name, json := range map[string]string{ + "version 2 with a park": v2(parked), + "version 4": strings.Replace(v3(""), `"version":3`, `"version":4`, 1), + "a park of a deferred original": v3(strings.Replace(parked, `"original":10`, `"original":5`, 1)), + "a park of a dropped original": v3(strings.Replace(parked, `"original":10`, `"original":8`, 1)), + "a partial park with no files": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":[]`, 1)), + "a partial park that needs": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["a"],"needs":[3]`, 1)), + "unsorted files": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["b","a"]`, 1)), + "a file that drives a terminal": v3(strings.Replace(parked, `"files":["a","b"]`, `"files":["a\u001b[2J"]`, 1)), + "a dependency park with no need": v3(strings.Replace(strings.Replace(parked, `"needs":[3,4],`, ``, 1), `"commits":["`+other+`"],`, ``, 1)), + "unsorted needs": v3(strings.Replace(parked, `"needs":[3,4]`, `"needs":[4,3]`, 1)), + "a park that needs itself": v3(strings.Replace(parked, `"needs":[3,4]`, `"needs":[3,11]`, 1)), + "a short commit": v3(strings.Replace(parked, `"commits":["`+other+`"]`, `"commits":["abc"]`, 1)), + "a short merge": v3(strings.Replace(parked, `"merge":"`+sha+`","kind":"partial"`, `"merge":"abc","kind":"partial"`, 1)), + "an unknown kind": v3(strings.Replace(parked, `"kind":"partial"`, `"kind":"later"`, 1)), + "a partial unit not on the batch": v3(strings.Replace(parked, `"partial":[9]`, `"partial":[10]`, 1)), + "a partial unit twice": v3(strings.Replace(parked, `"partial":[9]`, `"partial":[9,9]`, 1)), + } { + if _, err := parseRecord(block(json), l); err == nil { + t.Errorf("%s: parseRecord accepted it", name) + } + } +} diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index 46e37c955..fc6d4dda8 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -23,6 +23,7 @@ import ( "encoding/json" "errors" "fmt" + "maps" "os" "regexp" "runtime/debug" @@ -43,7 +44,9 @@ import ( const ( reportFence = "```backport-batch-report" reportMarker = "` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. @@ -116,7 +122,7 @@ An original the policy takes can build on code its line does not have: another p - A file it modifies or deletes whose counterpart, as [`run`](#run) follows renames, the line does not have is named by the first-parent commit on `main` that added it since the line branched: `git --literal-pathspecs log --first-parent --diff-merges=first-parent --no-renames --diff-filter=A -1 ..M^1 -- `. A file nothing added since the fork point is shown with the original, since no other pull request brought it. - A line it removes or changes in a file the line has is given by `git blame --porcelain -w`, at `M^1` named by its full commit, to a commit, which counts through the first-parent commit on `main` that brought it: each commit of `main`'s first-parent line since the fork point brings itself and every commit of the range reachable from its other parents that no earlier one brought. A commit from before the fork point, which the line has, and the original's own count for nothing. The commit counts only while one of its lines, with its edges trimmed and nothing else, is absent from the line's version of the file, so a hand backport under another commit clears it. A file the line already has as the original makes it, the same blob or a clean `git merge-file` back into the line's file, builds on nothing. -Lines a tool wrote are not blamed: a generated file whole, `zz_generated*`, one with a `Code generated … DO NOT EDIT` marker in its first 100 lines, `go.sum` and the package-manager lock files, and in an ApplicationDefinition under `packages/system/*-rd/cozyrds/` the keys `hack/update-crd.sh` writes, found by their indentation. Controller-gen CRDs and `values.schema.json` are blamed like any other file. Each `git log`, `git diff` and `git blame` the detector and the rename detection run has two minutes, and rename detection that runs out of time counts as skipped, and an original has at most 256 files, missing or blamed, 20 000 blamed lines and 8 MiB per file read for it. Evidence it cannot read, past one of those bounds, a binary file, a symlink, a submodule, a commit blame names that did not come through `main`, a path that is not UTF-8, or a rename git skipped looking for, is shown with the original, whatever becomes of it, under its line in the body and in the report's `diag`. Output of git the tool cannot read whole stops it with exit 2, and is never taken for an absent dependency. A dependency carried only by lines an original adds is not seen. `run` prints how long reading the line took on stderr. +Lines a tool wrote are not blamed: a generated file whole, `zz_generated*`, one with a `Code generated … DO NOT EDIT` marker in its first 100 lines, `go.sum` and the package-manager lock files, and in an ApplicationDefinition under `packages/system/*-rd/cozyrds/` the keys `hack/update-crd.sh` writes, found by their indentation. Controller-gen CRDs and `values.schema.json` are blamed like any other file. Each `git log`, `git diff` and `git blame` the detector and the rename detection run has two minutes, and rename detection that runs out of time counts as skipped, and an original has at most 256 files, missing or blamed, 20 000 blamed lines and 8 MiB per file read for it. Evidence it cannot read, past one of those bounds, a binary file, a symlink, a submodule, a commit blame names that did not come through `main`, a path that is not UTF-8, or a rename git skipped looking for, is shown with the original, whatever becomes of it, under its line in the body and in the report's `diag`. Output of git the tool cannot read whole stops it with exit 2, and is never taken for an absent dependency. A dependency carried only by lines an original adds is not seen. `run` prints how long reading the line took on stderr. What follows from a dependency, a park or a wait, is in [`run`](#run) and [`check`](#check). ## The line's record and reports @@ -126,7 +132,7 @@ The record is written only while its pull request is open: a merged or closed ba The record is only as trustworthy as the body it sits in. Anyone with write access can edit any body, the bot's included, and neither the bot's authorship nor anything in the block proves who edited it last. A hand edit that leaves a readable record is believed; one that breaks it stops every command with exit 2, naming the pull request. A reason or a file name in the record counts only in the form a run writes it, one line with no control, zero-width or bidi character, so text someone typed into the body never reaches a terminal, a body or a comment as they typed it. When the newest batch pull request is deleted and no later one exists, nothing shows it was there: the older record governs, and what was recorded since is lost. -The record holds only what later commands act on. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. +The record holds only what later commands act on. Version 3, which this tool writes, adds what is parked and why, with the merge commit each park is bound to, the units on the batch that do only part of their original, the picks that differ from theirs, and what each wait on another original waits on; it reads versions 1 to 3, and a tool that reads only 1 and 2 refuses version 3 with exit 2. Every run that classifies the line also makes a report, one line of JSON in a `backport-batch-report` block: every original merged into `main` since the line branched in exactly one category, picked, missing, deferred with what it was deferred for, or skipped with the rule that skipped it, each drop the record keeps with the merge commit it is bound to, its reason and the batch pull request that recorded it, the commits on `main` that merged no pull request, the labels, and what the classification was made of: the heads of `main`, the line and the newer line, the policy, the batch head it describes, the tool's commit, the Actions run and the flags. Its ID is a hash of all that but the time and the run, so two runs that find the same make the same report, and it names no pull request, so a new batch's report is whole before GitHub numbers the pull request. The body carries the latest report beside the record, and a comment on the open batch pull request keeps each one: `` and the same block, folded away. A report already posted is not posted again, and the tool never edits or deletes such a comment, so a report the next run replaces in the body is still read back by `report`, also after the batch merged. A comment that looks like one but whose author the batch does not trust is not read. When a run stops after writing the body but before posting its comment, the next `run --push` on the open batch posts that report first, byte for byte; once the batch is closed nothing is written to it, and the report stays in its body. A run that picks nothing while no batch is open keeps no report. A `--defer` that leaves the batch carrying nothing closes it, and writes everything while the pull request is still open: the report comment, then the body, whose record keeps the deferrals and says the batch is closing, with its number, branch and the head the rebuild started from. It then reads the pull request and the branch again, closes the pull request while both are at that head, and deletes the branch with `--force-with-lease` on it, printing `closed `. The next batch tries what was deferred again. A close that stops part way is finished by the next `run --push`, before it does anything else: it closes and deletes when the pull request is open at the frozen head, deletes the branch when the pull request is closed and the branch still at that head, and goes on as usual once both are gone. A run that finishes a close stops there, with exit 0, or with exit 2 when it was asked for `--defer`, `--drop` or `--undrop`, which it does not apply. It never takes a head it did not freeze: with the branch moved, it stops with exit 2 and names both heads. `run` without `--push`, `check` and `fold` stop with exit 2 while the pull request is still open; once it is closed they go on, warning about a branch the close left, and `notify` says nothing. A closed pull request is never written to, so its record says it was closing for good, and a closed pull request whose branch is gone counts as closed. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 753868f72..35ad92214 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -33,6 +33,8 @@ type pick struct { files []string // failed is the commit whose pick conflicted. failed string + // needs are the originals still to come it waits on. + needs []int // more counts the conflicting files a recorded conflict no longer names. more int reason string @@ -204,6 +206,19 @@ func (a *app) run(args []string) int { return a.fail(err) } } + // cls is what each original became in this run, to an original that + // builds on it; the rules decide the rest. + mode := analysisMode{noWorkflowFiles: *noWorkflows} + cls := map[*verdict]edgeClass{} + classOf := func(f *verdict) edgeClass { + if c, ok := b.ruleClass(f); ok { + return c + } + if c, ok := cls[f]; ok { + return c + } + return edgeMissing + } for _, v := range b.qualifying() { on := false if carried != nil { @@ -214,9 +229,11 @@ func (a *app) run(args []string) int { switch { case on: r.onBranch = append(r.onBranch, v) + cls[v] = edgeSatisfied continue case held[v.o.number].Reason != "": r.held = append(r.held, pick{v: v, status: "held", reason: held[v.o.number].Reason}) + cls[v] = edgeDeferred continue case v.onBatch != "": v.rule, v.reason = ruleOnLine, fmt.Sprintf("on the open batch pull request #%d", b.pr.Number) @@ -231,6 +248,17 @@ func (a *app) run(args []string) int { } if v.partial != nil { r.parked = append(r.parked, v) + cls[v] = edgeNever + continue + } + } + if len(v.deps) > 0 { + kind, err := a.dependsAt(b, r, v, mode, classOf) + if err != nil { + return a.fail(err) + } + if kind != edgeSatisfied { + cls[v] = kind continue } } @@ -241,6 +269,7 @@ func (a *app) run(args []string) int { } if len(files) > 0 { r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: files}) + cls[v] = edgeWorkflow continue } } @@ -257,10 +286,13 @@ func (a *app) run(args []string) int { v.partial = p.files v.park = &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkPartial, Files: sortedClean(p.files)} r.parked = append(r.parked, v) + cls[v] = edgeNever case "picked": r.onBranch = append(r.onBranch, v) + cls[v] = edgeSatisfied case "conflict": r.deferred = append(r.deferred, p) + cls[v] = edgeDeferred case "empty": v.rule, v.reason = ruleOnLine, "the earlier picks already carry it" r.skipped[ruleOnLine] = append(r.skipped[ruleOnLine], v) @@ -285,6 +317,50 @@ func (a *app) run(args []string) int { return a.finish(b, r, branch, date, *push, false) } +// dependsAt decides, from what v builds on that the branch lacks, whether +// the run leaves it out: waiting for a maintainer in the scheduled mode +// when only tests or docs build on what never comes, parked when anything +// else never comes, or waiting on prerequisites still to come. What is +// there, the line's, the batch's, or what this run picked or found empty, +// counts for nothing; once earlier picks moved the branch, v is judged at +// its head. It returns what v became to an original that builds on it, +// edgeSatisfied when the run goes on to pick it. +func (a *app) dependsAt(b *batch, r *report, v *verdict, mode analysisMode, classOf func(*verdict) edgeClass) (edgeClass, error) { + deps := v.deps + head, err := resolve("HEAD") + if err != nil { + return 0, err + } + if head != b.lineHead { + if deps, err = b.depsAt(v, head); err != nil { + return 0, err + } + } + never, pending := edges(deps, classOf) + primary, err := workflowPrimary(mode, v, never, pending) + switch { + case err != nil: + return 0, err + case primary: + files, err := workflowFiles(v.commits) + if err != nil { + return 0, err + } + v.addDiag(testsOnly(v, never, b.line)) + r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: files}) + return edgeWorkflow, nil + case len(never) > 0: + v.park = dependencyPark(v, never) + r.parked = append(r.parked, v) + return edgeNever, nil + case len(pending) > 0: + class, needs := strongest(pending) + r.needs = append(r.needs, pick{v: v, status: "waiting", needs: numbers(needs)}) + return class, nil + } + return edgeSatisfied, nil +} + // partialAtHead judges v again where the run has got to, when earlier picks // moved the branch past the line's head. func (a *app) partialAtHead(b *batch, v *verdict) error { diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index 02dc2fd90..d64bfb5d5 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -176,6 +176,9 @@ type deferredEntry struct { More int `json:"more,omitempty"` Reason string `json:"reason,omitempty"` Workflows bool `json:"workflows,omitempty"` + // Needs, from version 3, are the originals still to come that it waits + // on, those of the kind that decides the wait. + Needs []int `json:"needs,omitempty"` } // How many conflicting files of one original the body names, in the list and @@ -194,7 +197,7 @@ type deferredRecord struct { // had a version. encoding/json leaves bidi and zero-width characters as they // are, so file names go through clean first, as the reason does. func deferredBlock(l line, conflicts, held, waiting []pick) string { - data, _ := json.Marshal(deferredRecord{Line: l.branch(), Deferred: deferredEntries(conflicts, held, waiting)}) + data, _ := json.Marshal(deferredRecord{Line: l.branch(), Deferred: deferredEntries(conflicts, held, waiting, nil)}) return deferredFence + "\n" + string(data) + "\n```" } @@ -231,6 +234,7 @@ type report struct { deferred []pick // conflicts held []pick // what --defer took out, with the reason waiting []pick // what changes workflows, left to a maintainer + needs []pick // what waits on other originals still to come skipped map[int][]*verdict other int // overrides are the originals a backport/ label kept out or let in, and @@ -381,6 +385,12 @@ func (r *report) what(details map[int]bool) string { fmt.Fprintf(&b, "- %s %s: %s\n%s", prLink(v.o.number), titleSpan(v.o.title), parkReason(*v.park), diagItems(v)) } } + if len(r.needs) > 0 { + fmt.Fprintf(&b, "\n### Waiting on another original\n\nThese build on another original that has not reached %s yet, so the batch does not try them; each names the one it waits on, and the batch tries it again once that one is there.\n\n", codeSpan(r.line.branch())) + for _, p := range r.needs { + fmt.Fprintf(&b, "- %s %s: after %s\n%s", prLink(p.v.o.number), titleSpan(p.v.o.title), prLinks(p.needs), diagItems(p.v)) + } + } if len(r.overrides) > 0 { b.WriteString("\n### Maintainer overrides\n\n") for _, v := range r.overrides { @@ -437,6 +447,9 @@ func (r *report) outcome(v *verdict) string { return "waiting for a maintainer: it changes " + codeSpan(workflowsDir) case slices.Contains(r.parked, v): return "parked: " + parkReason(*v.park) + case slices.ContainsFunc(r.needs, func(p pick) bool { return p.v == v }): + i := slices.IndexFunc(r.needs, func(p pick) bool { return p.v == v }) + return "waiting on " + prLinks(r.needs[i].needs) case v.rule != 0: return "skipped by " + ruleName(v.rule) + ", " + titleSpan(v.reason) } @@ -465,6 +478,15 @@ func fileSpans(files []string) string { return strings.Join(out, ", ") } +// prLinks links to pull requests by number. +func prLinks(numbers []int) string { + var out []string + for _, n := range numbers { + out = append(out, prLink(n)) + } + return strings.Join(out, ", ") +} + // parkReason says why a park leaves its original out, for the body. func parkReason(p parkedEntry) string { if p.Kind == parkPartial { diff --git a/cmd/backport-batch/check.go b/cmd/backport-batch/check.go index b9983aca3..d8acf4e37 100644 --- a/cmd/backport-batch/check.go +++ b/cmd/backport-batch/check.go @@ -51,7 +51,13 @@ func (a *app) check(args []string) int { if err := a.warnLeftBranch(b); err != nil { return a.fail(err) } - disp, err := b.dispositions(*cut) + // The scheduled run's mode, as the open batch saved it; with none open, + // the scheduled run's, which is what the next batch is built in. + mode := analysisMode{noWorkflowFiles: true} + if b.src.open { + mode.noWorkflowFiles = b.src.savedNoWorkflowFiles() + } + disp, err := b.dispositions(mode, *cut) if err != nil { return a.fail(err) } @@ -138,8 +144,11 @@ type disposition struct { // dispositions says what check reports about each original that needs a line // of its own; the others pass in silence. The text is what follows the line's -// name, or the original alone for one that is missing. -func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { +// name, or the original alone for one that is missing. Originals are taken in +// merge order, each by the first of: a unit on the batch that does only part +// of it, a reason a person deferred it for, a park, a wait for a maintainer +// or on other originals, and what the record says of it. +func (b *batch) dispositions(mode analysisMode, cut bool) (map[*verdict]disposition, error) { if c := b.src.closingOpen(); c != nil { return nil, fmt.Errorf("batch pull request #%d is being closed; backport-batch run --line %s --push finishes it", c.PR, b.line.branch()) } @@ -159,10 +168,40 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { on += " (" + prState(c) + ")" } } + if err := b.analyze(); err != nil { + return nil, err + } + if b.pr != nil { + if err := b.keepRecordedPartials(b.prHead); err != nil { + return nil, err + } + } + held := map[int]string{} + for n, e := range deferred { + if e.Reason != "" { + held[n] = e.Reason + } + } + states, err := b.classify(mode, held, deferred, b.src.open) + if err != nil { + return nil, err + } + // waitsOn names what an original waits on, as check says it. + waitsOn := func(s *state) string { + var names []string + for _, f := range s.needs { + names = append(names, fmt.Sprintf("#%d", f.o.number)) + } + return strings.Join(names, " and ") + } out := map[*verdict]disposition{} for _, v := range b.verdicts { named := v.o.url() + " " + terminalSafe(v.o.title) e, isDeferred := deferred[v.o.number] + s := states[v] + if s == nil { + s = &state{kind: stOwn} + } switch { case b.partialUnits[v] != nil && cut: out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is on batch #%d, but only part of it applies to %s: it leaves %s without effect; take it out with backport-batch run --defer before the cut: %s", v.o.number, b.pr.Number, b.line.branch(), quotedFiles(b.partialUnits[v]), named)} @@ -178,17 +217,41 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { if cut { out[v] = disposition{outcomeUnmerged, fmt.Sprintf("#%d is only on open batch PR #%d, merge it before the cut: %s", v.o.number, b.pr.Number, named)} } - case isDeferred && e.Reason != "": + case s.kind == stHeld: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred on %s to the next batch: %s (%s)", on, named, terminalSafe(e.Reason))} - case v.park != nil && v.park.Kind == parkPartial: + case s.kind == stPartial: out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: only part of it applies to %s, which would leave %s without effect; port what applies as a pull request of its own: %s", v.o.number, b.line.branch(), quotedFiles(v.park.Files), named)} + case s.kind == stWorkflowPrimary && s.class == edgeMissing: + out[v] = disposition{outcomeMissing, named} + case s.kind == stParked: + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d is parked: it builds on %s, which never reaches %s; port what applies as a pull request of its own, or drop it: %s", v.o.number, parkNames(*v.park), b.line.branch(), named)} + case s.kind == stNeeds && s.wait == edgeMissing: + out[v] = disposition{outcomeMissing, fmt.Sprintf("%s (it waits on %s, which is not on the line either)", named, waitsOn(s))} + case s.kind == stNeeds && s.wait == edgeWorkflow && cut: + out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d waits on %s, which changes %s and waits on batch PR %s for a maintainer; bring it over, or defer both, before the cut: %s", v.o.number, waitsOn(s), workflowsDir, on, named)} + case s.kind == stNeeds && s.wait == edgeWorkflow: + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d waits on %s, which waits on %s for a maintainer: %s", v.o.number, waitsOn(s), on, named)} + case s.kind == stNeeds && s.wait == edgeDeferred: + what := "a deferred conflict on " + on + if f := s.needs[0]; deferred[f.o.number].Reason != "" { + what = "deferred on " + on + " to the next batch" + } else if states[f] != nil && states[f].kind == stNeeds { + what = "which waits itself" + } + out[v] = disposition{outcomeWarning, fmt.Sprintf("#%d waits on %s, %s: %s", v.o.number, waitsOn(s), what, named)} + case s.kind == stNeeds: + newer := "the newer line" + if b.newer != nil { + newer = b.newer.branch() + } + out[v] = disposition{outcomeNote, fmt.Sprintf("#%d waits on %s, which is not on %s yet: %s", v.o.number, waitsOn(s), newer, named)} case isDeferred && e.Workflows && !b.src.open: out[v] = disposition{outcomeMissing, named} case isDeferred && e.Workflows && cut: out[v] = disposition{outcomeWaiting, fmt.Sprintf("#%d changes %s and waits on batch PR %s for a maintainer, bring it over or defer it before the cut: %s", v.o.number, workflowsDir, on, named)} case isDeferred && e.Workflows: out[v] = disposition{outcomeWarning, fmt.Sprintf("waiting on %s for a maintainer, since it changes %s: %s", on, workflowsDir, named)} - case isDeferred: + case isDeferred && len(e.Needs) == 0: out[v] = disposition{outcomeWarning, fmt.Sprintf("deferred conflict on %s: %s", on, named)} default: out[v] = disposition{outcomeMissing, named} @@ -197,6 +260,18 @@ func (b *batch) dispositions(cut bool) (map[*verdict]disposition, error) { return out, nil } +// parkNames names what a park waits on for a terminal. +func parkNames(p parkedEntry) string { + var out []string + for _, n := range p.Needs { + out = append(out, fmt.Sprintf("#%d", n)) + } + for _, c := range p.Commits { + out = append(out, c[:12]) + } + return strings.Join(out, ", ") +} + // quotedFiles names files for a terminal, each quoted, the first few of them. func quotedFiles(files []string) string { var out []string diff --git a/cmd/backport-batch/defer.go b/cmd/backport-batch/defer.go index 8915d4ebd..de77d5d5c 100644 --- a/cmd/backport-batch/defer.go +++ b/cmd/backport-batch/defer.go @@ -138,22 +138,50 @@ func (a *app) reclassify(b *batch, picks *history, hold map[int]string, drops [] if r.stale, err = keptOut(b, picks); err != nil { return a.fail(err) } + // What the rebuild took off is no longer there for what builds on it. + units, err := b.unitsOn(picks.head) + if err != nil { + return a.fail(err) + } + for _, v := range b.verdicts { + v.batched = units[v] + } + on := map[*verdict]bool{} for _, v := range b.qualifying() { - e := recorded[v.o.number] - on, err := b.onBranch(picks, v) - if err != nil { + if on[v], err = b.onBranch(picks, v); err != nil { return a.fail(err) } + if !on[v] { + v.onBatch = "" + } + } + if err := b.keepRecordedPartials(picks.head); err != nil { + return a.fail(err) + } + states, err := b.classify(analysisMode{noWorkflowFiles: r.opts.NoWorkflowFiles}, hold, recorded, true) + if err != nil { + return a.fail(err) + } + for _, v := range b.qualifying() { + e := recorded[v.o.number] + s := states[v] + if s == nil { + s = &state{kind: stOwn} + } switch { - case on: + case on[v]: r.onBranch = append(r.onBranch, v) case hold[v.o.number] != "": r.held = append(r.held, pick{v: v, status: "held", reason: hold[v.o.number]}) - case v.park != nil: + case s.kind == stPartial || s.kind == stParked: r.parked = append(r.parked, v) + case s.kind == stNeeds: + r.needs = append(r.needs, pick{v: v, status: "waiting", needs: numbers(s.needs)}) + case s.kind == stWorkflowPrimary && !e.Workflows: + r.missing = append(r.missing, v) case e.Workflows: r.waiting = append(r.waiting, pick{v: v, status: "waiting", files: e.Files, more: e.More}) - case len(e.Files) > 0: + case len(e.Files) > 0 && len(e.Needs) == 0: r.deferred = append(r.deferred, pick{v: v, status: "conflict", files: e.Files, more: e.More, failed: e.Commit}) default: // Merged into main since the batch was assembled, or recorded diff --git a/cmd/backport-batch/deps.go b/cmd/backport-batch/deps.go index 33133a1e5..895897709 100644 --- a/cmd/backport-batch/deps.go +++ b/cmd/backport-batch/deps.go @@ -436,6 +436,23 @@ func (b *batch) dependencies(v *verdict, target string) ([]dependency, []string, for _, d := range byMerge { slices.Sort(d.paths) deps = append(deps, *d) + // The rules decide before the line is looked at, so whether the + // line carries a prerequisite they keep out is asked here. + if f := d.v; f != nil && f.rule != 0 && f.rule != ruleOnLine && f.rule != ruleNewer && b.lineHist != nil { + if _, asked := b.lineCarries[f]; !asked { + if err := b.load(f); err != nil { + return nil, nil, err + } + how, err := b.lineHist.carries(f.o, f.commits) + if err != nil { + return nil, nil, err + } + if b.lineCarries == nil { + b.lineCarries = map[*verdict]bool{} + } + b.lineCarries[f] = how != "" + } + } } slices.SortFunc(deps, func(x, y dependency) int { return strings.Compare(x.merge, y.merge) }) return deps, diag, nil @@ -614,25 +631,9 @@ func (b *batch) beforeBase(commit string) (bool, error) { return ok, err } -// depNote is how a dependency is shown with its original. -func (d dependency) note(l line) string { - who := fmt.Sprintf("%.12s, a commit on main that merged no pull request", d.merge) - if d.v != nil { - who = fmt.Sprintf("#%d", d.v.o.number) - } - quoted := make([]string, len(d.paths)) - for i, p := range d.paths { - quoted[i] = shown(p) - } - how := "in " + strings.Join(quoted, ", ") - if d.added { - how = "which added " + strings.Join(quoted, ", ") - } - return oneLineText(fmt.Sprintf("builds on %s, %s, which %s lacks", who, how, l.branch())) -} - // analyze reads what every original the policy takes and nothing carries -// builds on that the line lacks, and keeps it with the original. +// builds on that the line lacks, and keeps it with the original, for the +// parks and waits that follow from it. func (b *batch) analyze() error { defer b.timed(time.Now()) for _, v := range b.qualifying() { @@ -645,9 +646,267 @@ func (b *batch) analyze() error { } v.deps = deps v.addDiag(diag...) - for _, d := range deps { - v.addDiag(d.note(b.line)) - } } return nil } + +// analysisMode is how a classification treats an original that changes a +// workflow: as the scheduled run does, whose token cannot push one, or as a +// maintainer's run. +type analysisMode struct{ noWorkflowFiles bool } + +// edgeClass is what a prerequisite is to an original that builds on it, +// from what never matters to what decides first. +type edgeClass int + +const ( + edgeSatisfied edgeClass = iota + // edgeNewer is held by rule 6 on this line. + edgeNewer + // edgeDeferred is a deferred conflict, or deferred for a reason. + edgeDeferred + // edgeWorkflow waits on the open batch for a maintainer. + edgeWorkflow + // edgeMissing is on neither the line, the batch nor the record. + edgeMissing + // edgeNever never reaches the line. + edgeNever +) + +// edge is a dependency whose prerequisite has not reached the line, with +// what that prerequisite is. +type edge struct { + dep dependency + class edgeClass +} + +// ruleClass is what the rules alone make of a prerequisite, or ok false when +// it depends on what this classification makes of it: one the line or the +// open batch carries is there, whatever its label, drop or rule says, and a +// direct push and an original the rules keep out never come. +func (b *batch) ruleClass(f *verdict) (edgeClass, bool) { + switch { + case f == nil: + return edgeNever, true + case f.rule == ruleOnLine || f.onBatch != "" || f.batched != "" || b.lineCarries[f]: + return edgeSatisfied, true + case f.rule == ruleNewer: + return edgeNewer, true + case f.rule != 0: + return edgeNever, true + } + return 0, false +} + +// edges sorts dependencies by what classOf says of each prerequisite into +// those that never come and those still to come, dropping the ones there. +func edges(deps []dependency, classOf func(*verdict) edgeClass) (never, pending []edge) { + for _, d := range deps { + switch c := classOf(d.v); c { + case edgeSatisfied: + case edgeNever: + never = append(never, edge{d, c}) + default: + pending = append(pending, edge{d, c}) + } + } + return never, pending +} + +// strongest is the class of prerequisite that decides a wait, and the +// prerequisites of that class, in merge order. +func strongest(pending []edge) (edgeClass, []*verdict) { + top := edgeSatisfied + for _, e := range pending { + top = max(top, e.class) + } + var out []*verdict + for _, e := range pending { + if e.class == top && !slices.Contains(out, e.dep.v) { + out = append(out, e.dep.v) + } + } + return top, out +} + +// auxiliary says whether a path holds tests or documentation, which an +// original can change without building on what it needs to work. +func auxiliary(p string) bool { + parts := strings.Split(p, "/") + base := parts[len(parts)-1] + for _, d := range parts[:len(parts)-1] { + if d == "test" || d == "tests" || d == "testdata" { + return true + } + } + switch { + case strings.HasSuffix(base, "_test.go"), strings.Contains(base, ".test."), strings.Contains(base, ".spec."), strings.HasSuffix(base, ".bats"): + return true + case len(parts) > 1 && parts[0] == "docs": + return true + } + return len(parts) == 1 && strings.HasSuffix(base, ".md") +} + +// workflowPrimary says whether v waits for a maintainer rather than parks: +// in the scheduled mode, an original that changes a workflow, all of whose +// missing dependencies that never come are tests or docs, and whose +// workflow lines build on nothing still missing. The workflow is what it is +// for, and a maintainer brings over what applies. Only what is still +// missing counts: never and pending are what is left once what is there was +// taken out. +func workflowPrimary(mode analysisMode, v *verdict, never, pending []edge) (bool, error) { + if !mode.noWorkflowFiles || len(never) == 0 { + return false, nil + } + for _, e := range never { + if slices.ContainsFunc(e.dep.paths, func(p string) bool { return !auxiliary(p) }) { + return false, nil + } + } + for _, e := range append(slices.Clone(never), pending...) { + if slices.ContainsFunc(e.dep.paths, func(p string) bool { return strings.HasPrefix(p, workflowsDir) }) { + return false, nil + } + } + files, err := workflowFiles(v.commits) + return len(files) > 0, err +} + +// testsOnly is the note a workflow original gets that waits for a +// maintainer although its tests build on what never comes. +func testsOnly(v *verdict, never []edge, l line) string { + var who []string + for _, e := range never { + if e.dep.v != nil { + who = append(who, fmt.Sprintf("#%d", e.dep.v.o.number)) + } else { + who = append(who, e.dep.merge[:12]) + } + } + slices.Sort(who) + return oneLineText(fmt.Sprintf("only its tests or docs build on %s, which never reaches %s", strings.Join(slices.Compact(who), ", "), l.branch())) +} + +// dependencyPark is the park of v for prerequisites that never come. +func dependencyPark(v *verdict, never []edge) *parkedEntry { + p := &parkedEntry{Original: v.o.number, Merge: v.o.merge, Kind: parkDependency} + for _, e := range never { + if e.dep.v != nil { + p.Needs = append(p.Needs, e.dep.v.o.number) + } else { + p.Commits = append(p.Commits, e.dep.merge) + } + } + slices.Sort(p.Needs) + p.Needs = slices.Compact(p.Needs) + slices.Sort(p.Commits) + p.Commits = slices.Compact(p.Commits) + return p +} + +// depsAt is what v builds on that target lacks, read once per target: the +// run asks again at its head once earlier picks moved the branch. +func (b *batch) depsAt(v *verdict, target string) ([]dependency, error) { + key := target + " " + v.o.merge + if d, ok := b.depCache[key]; ok { + return d, nil + } + deps, diag, err := b.dependencies(v, target) + if err != nil { + return nil, err + } + v.addDiag(diag...) + if b.depCache == nil { + b.depCache = map[string][]dependency{} + } + b.depCache[key] = deps + return deps, nil +} + +// stateKind is what a classification makes of an original the policy takes +// that nothing carries, in the order that decides it. +type stateKind int + +const ( + // stHeld is deferred for a reason a person gave. + stHeld stateKind = iota + 1 + // stPartial would leave part of it without effect on the line. + stPartial + // stWorkflowPrimary waits for a maintainer, though its tests build on + // what never comes. + stWorkflowPrimary + // stParked builds on what never comes. + stParked + // stNeeds waits on prerequisites still to come. + stNeeds + // stOwn is what the record says of it, or missing. + stOwn +) + +// state is a classification of one original: what it is, what it is to an +// original that builds on it, and for a wait the class that decides it and +// the prerequisites of that class. +type state struct { + kind stateKind + class edgeClass + wait edgeClass + needs []*verdict +} + +// classify decides, in merge order, what becomes of every original the +// policy takes that neither the line nor the open batch carries, as an +// operation that picks nothing sees it: check, and the ones that only +// change the record. held are the reasons people deferred originals for, +// recorded what the record defers, and listed says the record's workflow +// waits still wait on an open batch. Parks it finds go into each verdict. +func (b *batch) classify(mode analysisMode, held map[int]string, recorded map[int]deferredEntry, listed bool) (map[*verdict]*state, error) { + states := map[*verdict]*state{} + classOf := func(f *verdict) edgeClass { + if c, ok := b.ruleClass(f); ok { + return c + } + if s := states[f]; s != nil { + return s.class + } + return edgeMissing + } + for _, v := range b.qualifying() { + if v.onBatch != "" { + continue + } + s := &state{} + states[v] = s + e := recorded[v.o.number] + never, pending := edges(v.deps, classOf) + primary, err := workflowPrimary(mode, v, never, pending) + if err != nil { + return nil, err + } + switch { + case held[v.o.number] != "": + s.kind, s.class = stHeld, edgeDeferred + case v.park != nil && v.park.Kind == parkPartial: + s.kind, s.class = stPartial, edgeNever + case primary: + s.kind, s.class = stWorkflowPrimary, edgeMissing + if listed && e.Workflows { + s.class = edgeWorkflow + } + case len(never) > 0: + v.park = dependencyPark(v, never) + s.kind, s.class = stParked, edgeNever + case len(pending) > 0: + s.kind = stNeeds + s.wait, s.needs = strongest(pending) + s.class = s.wait + case listed && e.Workflows: + s.kind, s.class = stOwn, edgeWorkflow + case !e.Workflows && len(e.Needs) == 0 && (e.Commit != "" || len(e.Files) > 0): + s.kind, s.class = stOwn, edgeDeferred + default: + s.kind, s.class = stOwn, edgeMissing + } + } + return states, nil +} diff --git a/cmd/backport-batch/drop.go b/cmd/backport-batch/drop.go index b374081cf..1af2e5b3a 100644 --- a/cmd/backport-batch/drop.go +++ b/cmd/backport-batch/drop.go @@ -117,6 +117,9 @@ func (a *app) redrop(b *batch, numbers []int, reason string, undrop bool, push b if err := b.load(v); err != nil { return a.fail(err) } + if err := b.rejudged(v); err != nil { + return a.fail(err) + } if undrop { continue } @@ -159,6 +162,38 @@ func (b *batch) bindDrops(merges []*original) (map[int]droppedEntry, error) { }) } +// waitOrder refuses a record whose waits or parks need an original merged +// after the one that needs it: a dependency is always on main first. +func (b *batch) waitOrder(merges []*original) error { + if b.src.rec == nil { + return nil + } + at := map[int]int{} + for i, o := range merges { + at[o.number] = i + } + check := func(n int, needs []int) error { + i, ok := at[n] + for _, f := range needs { + if j, known := at[f]; ok && known && j >= i { + return fmt.Errorf("batch pull request #%d has #%d wait on #%d, which merged after it", b.src.carrier.Number, n, f) + } + } + return nil + } + for _, d := range b.src.rec.Deferred { + if err := check(d.Original, d.Needs); err != nil { + return err + } + } + for _, p := range b.src.rec.Parked { + if err := check(p.Original, p.Needs); err != nil { + return err + } + } + return nil +} + // bindEntries binds drops to main: one whose original is among merges, the // originals main merged since the line branched, has to name that // original's merge exactly; any other has to be a merge of its number on diff --git a/cmd/backport-batch/effects.go b/cmd/backport-batch/effects.go index 6bff85d7f..350f9a961 100644 --- a/cmd/backport-batch/effects.go +++ b/cmd/backport-batch/effects.go @@ -444,6 +444,7 @@ func (b *batch) partialStates() error { return err } for v, commit := range on { + v.batched = commit if v.rule == 0 && v.onBatch == "" { v.onBatch = commit } @@ -511,6 +512,58 @@ func (b *batch) markPartial(r *report, head string) error { return nil } +// keepRecordedPartials judges again, at head, what the batch carries now, +// each original the open record parks as partial that the line's head +// judges whole: the run's picker found it partial after the picks before +// it, which check and an operation that picks nothing do not replay. It +// stays parked while its net merge onto head still leaves something +// without effect, and is the line's again once that went, as when the pick +// that made it partial was taken off. +func (b *batch) keepRecordedPartials(head string) error { + if !b.src.open || b.src.rec == nil { + return nil + } + for _, p := range b.src.rec.Parked { + v := b.find(p.Original) + if p.Kind != parkPartial || v == nil || v.rule != 0 || v.onBatch != "" || v.park != nil || v.o.merge != p.Merge { + continue + } + if err := b.partialAt(v, head); err != nil { + return err + } + } + return nil +} + +// rejudged reads again what partialStates and analyze read for v once its +// rules changed, as a withdrawn drop changes them: it is back where the +// line puts it. +func (b *batch) rejudged(v *verdict) error { + if b.pr != nil { + on, err := b.unitsOn(b.prHead) + if err != nil { + return err + } + v.batched = on[v] + if v.rule == 0 && v.onBatch == "" { + v.onBatch = on[v] + } + } + if v.rule != 0 || v.onBatch != "" { + return nil + } + if err := b.partialAt(v, b.lineHead); err != nil { + return err + } + deps, diag, err := b.dependencies(v, b.lineHead) + if err != nil { + return err + } + v.deps = deps + v.addDiag(diag...) + return nil +} + // partialAt judges v's clean net merge onto head, and parks it, or clears // the park, by what that merge leaves without effect. func (b *batch) partialAt(v *verdict, head string) error { diff --git a/cmd/backport-batch/fold_test.go b/cmd/backport-batch/fold_test.go index 5c064c2e3..f783802b2 100644 --- a/cmd/backport-batch/fold_test.go +++ b/cmd/backport-batch/fold_test.go @@ -381,6 +381,22 @@ func TestFoldRefusesAKeptDeletion(t *testing.T) { } } +// An original parked for what it builds on can still land as a complete +// hand adaptation: every file it changes, changed, in the batch's shape. +func TestFoldAcceptsACompleteAdaptationOfAParkedOriginal(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc5 := f.pr(5, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + w := emptyBatch(f, &fakeGH{}, "") + pick := w.pick(w.head, f.git("rev-parse", mc5+"^2"), map[string]string{"app/a.txt": strings.Replace(appFile, "two", "two!", 1)}) + fold := w.merge(w.head, pick, "fix(app): an excited two (#5)\n\n"+batchTrailer+": "+mc5) + if code, out, stderr := f.exec(w.gh, "fold", "--line", "release-1.1", "--original", "5", "--commit", fold); code != 0 || out != "ok\n" { + t.Errorf("fold of an adaptation of #5 = %d, %q\n%s", code, out, stderr) + } +} + // A fold that keeps a file the original deletes leaves the deletion without // effect, whatever it changes in the file: a port that keeps what the line // made of it, or the file with a line added, is a pull request of its own. diff --git a/cmd/backport-batch/main.go b/cmd/backport-batch/main.go index 85bd3ee12..23b357dff 100644 --- a/cmd/backport-batch/main.go +++ b/cmd/backport-batch/main.go @@ -207,8 +207,11 @@ type verdict struct { // and by load for anything else. commits []string changes []fileChange - // onBatch is the commit of the open batch pull request that picks it. + // onBatch is the commit of the open batch pull request that picks it, + // and batched the unit of the open batch that carries it whatever the + // rules now say of it. onBatch string + batched string // label is the maintainer override the original carries, if any. label string // partial lists what its clean net merge onto the line would leave @@ -268,6 +271,10 @@ type batch struct { ancestors map[string]bool // detecting is how long judging partial units and dependencies took. detecting time.Duration + depCache map[string][]dependency + // lineCarries says, for a prerequisite the rules keep out, whether the + // line carries it all the same. + lineCarries map[*verdict]bool } // fetch moves to the top of the working tree first: every path the batch @@ -374,6 +381,9 @@ func (a *app) prepare(l line) (*batch, error) { if b.drops, err = b.bindDrops(merges); err != nil { return nil, err } + if err := b.waitOrder(merges); err != nil { + return nil, err + } for _, o := range merges { v := &verdict{o: o, label: labels[o.number]} if err := b.judge(v, b.drops); err != nil { diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 7191f42b4..34f0d8afb 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -87,20 +87,42 @@ func (a *app) notify(args []string) int { return a.fail(err) } deferred := map[int]deferredEntry{} + var partial []int if src.open { - deferred = src.rec.deferred() + deferred, partial = src.rec.deferred(), src.rec.Partial } + // An original that waits on others is listed once, after what decides + // its wait, where that one is listed. var conflicts, waiting []int + after := map[int][]int{} for n, e := range deferred { - switch { - case e.Workflows: + switch place := placement(deferred, n, map[int]bool{}); { + case place == "workflow": waiting = append(waiting, n) - case e.Reason == "": + case place == "conflict": conflicts = append(conflicts, n) } + if len(e.Needs) > 0 { + after[n] = e.Needs + } } slices.Sort(conflicts) slices.Sort(waiting) + item := func(n int) string { + if needs := after[n]; len(needs) > 0 { + return fmt.Sprintf("\n- %s, after %s", prLink(n), prLinks(needs)) + } + return "\n- " + prLink(n) + } + count := func(ns []int) int { + c := 0 + for _, n := range ns { + if len(after[n]) == 0 { + c++ + } + } + return c + } today := a.now().UTC().Truncate(24 * time.Hour) days := int(today.Sub(opened).Hours() / 24) @@ -110,21 +132,34 @@ func (a *app) notify(args []string) int { fmt.Fprintf(a.stdout, "%s: batch pull request #%d has been open since %s, %d days\n", l.branch(), pr.Number, opened.Format(time.DateOnly), days) text = append(text, fmt.Sprintf("This batch has been open since %s, %d days, through at least one scheduled run. Merge it, or close it and the next run starts a new one; until then each run adds what the rules take to it.", opened.Format(time.DateOnly), days)) } - if len(conflicts) > 0 { - fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), len(conflicts), pr.Number) + if len(partial) > 0 { + fmt.Fprintf(a.stdout, "%s: %d units on #%d carry only part of their originals\n", l.branch(), len(partial), pr.Number) var b strings.Builder - fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch:\n", tasksURL) + b.WriteString("These are on the batch but carry only part of their originals on this line; take each out with `backport-batch run --defer` before the cut, and port what applies as a pull request of its own:\n") + for _, n := range partial { + b.WriteString(item(n)) + } + text = append(text, b.String()) + } + if len(conflicts) > 0 { + fmt.Fprintf(a.stdout, "%s: %d deferred conflicts on #%d wait for a maintainer\n", l.branch(), count(conflicts), pr.Number) + var b strings.Builder + fmt.Fprintf(&b, "These conflicts wait for a maintainer, as [resolving a conflict](%sresolve-conflict.md) describes; unresolved, each goes to the next batch, and so does what waits on one:\n", tasksURL) for _, n := range conflicts { - fmt.Fprintf(&b, "\n- %s", prLink(n)) + b.WriteString(item(n)) } text = append(text, b.String()) } if len(waiting) > 0 { - fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + if count(waiting) == len(waiting) { + fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s and wait for a maintainer\n", l.branch(), len(waiting), pr.Number, workflowsDir) + } else { + fmt.Fprintf(a.stdout, "%s: %d originals on #%d change %s or wait on one that does\n", l.branch(), len(waiting), pr.Number, workflowsDir) + } var b strings.Builder - fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes:\n", workflowsDir, policyURL) + fmt.Fprintf(&b, "These change `%s`, which the batch's token may not push, and wait for a maintainer to bring them over, as [changes to workflows](%s#changes-to-workflows) describes, or wait on one that does:\n", workflowsDir, policyURL) for _, n := range waiting { - fmt.Fprintf(&b, "\n- %s", prLink(n)) + b.WriteString(item(n)) } text = append(text, b.String()) } @@ -134,6 +169,36 @@ func (a *app) notify(args []string) int { return a.comment(pr, *mention, today, text) } +// placement says under which list notify names original n of the record: +// workflow for a wait for a maintainer, conflict for a deferred conflict, "" +// for neither, a deferral for a reason among them. One that waits on others +// goes where what decides its wait goes, followed down the record; one that +// is not in the record is held by rule 6 or missing, and waits on nobody. +func placement(deferred map[int]deferredEntry, n int, seen map[int]bool) string { + e, ok := deferred[n] + switch { + case !ok || seen[n]: + return "" + case e.Workflows: + return "workflow" + case e.Reason != "": + return "" + case len(e.Needs) == 0: + return "conflict" + } + seen[n] = true + place := "" + for _, f := range e.Needs { + switch p := placement(deferred, f, seen); { + case p == "workflow": + return p + case p == "conflict": + place = p + } + } + return place +} + // comment writes what waits into the one notify comment on the batch pull // request: it edits the comment if there is one, and opens it only when // something waits. Only a comment by an author the batch trusts is taken for diff --git a/cmd/backport-batch/park_test.go b/cmd/backport-batch/park_test.go index 6c8f51d0a..94a06ada5 100644 --- a/cmd/backport-batch/park_test.go +++ b/cmd/backport-batch/park_test.go @@ -17,6 +17,7 @@ limitations under the License. package main import ( + "fmt" "slices" "strings" "testing" @@ -198,6 +199,263 @@ func TestOnBatchAfterAnEmptyFirstPick(t *testing.T) { } } +// featWorld is #5071's shape: #3, a feat the line never gets, adds a file, +// and #5 fixes that file; #6 is a plain fix beside it. +func featWorld(t *testing.T) *fixture { + t.Helper() + f := newFixture(t) + f.lines() + f.pr(3, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(5, "fix(app): fix the tap", "```release-note\nfix(app): the tap works.\n```\n", map[string]string{"app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f +} + +// A fix of a file a feat added on main, which the line never gets, is +// parked, not tried, and check takes the park as a note. +func TestParkAFixOfAFeatureTheLineLacks(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + if strings.Contains(stdout, "### Deferred conflicts") || !strings.Contains(stdout, `"parked":[{"original":5,`) || !strings.Contains(stdout, `"kind":"dependency","needs":[3]`) { + t.Errorf("#5 is not parked on #3:\n%s", stdout) + } + if !strings.Contains(stdout, "### Parked") || !strings.Contains(stdout, "builds on [#3]") { + t.Errorf("the body does not say #5 builds on #3:\n%s", stdout) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "note: #5 is parked: it builds on #3, which never reaches release-1.1") { + t.Errorf("check = %d, want 0 with a note that #5 is parked:\n%s\n%s", code, stdout, stderr) + } +} + +// backport/include lifts no park. +func TestIncludeDoesNotLiftThePark(t *testing.T) { + f := featWorld(t) + code, stdout, stderr := f.exec(&fakeGH{labels: map[string][]int{labelInclude: {5}}}, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) || !strings.Contains(stdout, "`backport/include`, parked: builds on") { + t.Errorf("run with backport/include on #5 = %d, want #5 still parked:\n%s\n%s", code, stdout, stderr) + } +} + +// A prerequisite that is itself parked, but adapted onto the batch by hand, +// satisfies what builds on it: #3 is parked on #2, a feat, and #5 changes a +// line #3 wrote. +func TestAdaptedPrerequisiteSatisfiesItsDependent(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.pr(20, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc3 := f.pr(3, "fix(app): an excited two, a loud three", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "three", "THREE", 1)}) + f.pr(5, "fix(app): an excited three", "", map[string]string{"app/b.txt": strings.Replace(appFile, "three", "THREE!", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"kind":"dependency","needs":[3]`) { + t.Fatalf("run = %d, want #5 parked as long as #3 is:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + w := emptyBatch(f, &fakeGH{}, "batch\n\n"+deferredBlock(line11, nil, nil, nil)+"\n") + // #3 adapted by hand: every file it changes, changed, in the batch's + // shape. + pick := w.pick(w.head, f.git("rev-parse", mc3+"^2"), map[string]string{"app/a.txt": strings.Replace(appFile, "two", "two!", 1), "app/b.txt": strings.Replace(appFile, "three", "THREE", 1)}) + fold := w.merge(w.head, pick, "fix(app): an excited two, a loud three (#3)\n\n"+batchTrailer+": "+mc3) + f.git("push", "-q", "origin", fold+":refs/heads/"+batchBranch1) + w.gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, w.gh.prs[0].Body)} + code, stdout, stderr := f.exec(w.gh, "check", "--line", "release-1.1") + if code != 1 || strings.Contains(stderr, "#5 is parked") || strings.Contains(stdout+stderr, "waits on") || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check = %d, want #5 missing, not parked:\n%s\n%s", code, stdout, stderr) + } + code, stdout, stderr = f.exec(w.gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[3,5]`) { + t.Errorf("run --continue = %d, want #5 picked after #3's adaptation:\n%s\n%s", code, stdout, stderr) + } +} + +// What an earlier unit of the batch brought counts: #3 writes the line #8 +// changes, a feat takes it back and another feat, #6, writes it again, so +// blame names #6, which never comes; once #3 is picked the branch has the +// line all the same. +func TestContentFromAnotherUnitSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "fix(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(4, "feat(app): a quiet two", "", map[string]string{"app/a.txt": appFile}) + f.pr(6, "feat(app): a loud two again", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(8, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + if got := f.subjects("origin/release-1.1", "HEAD"); !slices.Equal(got, []string{"fix(app): a loud two (#3)", "fix(app): an excited two (#8)"}) { + t.Errorf("picked %q, want #3 and #8:\n%s", got, stdout) + } +} + +// auxWorld is #5142's shape: #3, a feat, adds a bats test; #5 fixes a +// workflow and that test, so all it builds on that never comes is a test; +// #6 is a plain fix. +func auxWorld(t *testing.T) *fixture { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n"}) + f.lines() + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(5, "fix(ci): run the release on tags", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f +} + +// A workflow original whose only missing dependency is a test waits for a +// maintainer in the scheduled run, whose token cannot push workflows, and +// check holds the cut on it. A maintainer's own run parks it. With no batch +// open, or one whose body carries no record, it is missing; a record from +// before the mode was saved says the scheduled mode by listing a wait. +func TestWorkflowModeCarriesToCheck(t *testing.T) { + f := auxWorld(t) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `{"original":5,"files":[".github/workflows/release.yaml"],"workflows":true}`) { + t.Fatalf("scheduled run = %d, want #5 waiting, not parked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut = %d, want #5 waiting:\n%s\n%s", code, stdout, stderr) + } + v1 := "batch\n\n" + deferredBlock(line11, nil, nil, []pick{{v: &verdict{o: &original{number: 5}}, files: []string{".github/workflows/release.yaml"}}}) + "\n" + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, v1)} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut with a version 1 record = %d, want #5 waiting:\n%s\n%s", code, stdout, stderr) + } + gh.prs = []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "a batch someone wrote")} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check under a batch with no record = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } + + f = auxWorld(t) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check with no batch = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } + gh = &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Fatalf("a maintainer's run = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked") { + t.Errorf("check after a maintainer's run = %d, want 0 with #5 parked:\n%s\n%s", code, stdout, stderr) + } +} + +// Evidence found only in a test file still parks an original that changes +// no workflow: #4973's shape. +func TestTestOnlyEvidenceParksOtherOriginals(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: nodes", "", map[string]string{"app/nodes.go": "package app\n", "app/nodes_test.go": "package app\n\nfunc a() {}\n"}) + f.lines() + f.pr(3, "feat(app): proxmox nodes", "", map[string]string{"app/nodes_test.go": "package app\n\nfunc a() { proxmox() }\n"}) + f.pr(5, "fix(app): nodes", "", map[string]string{"app/nodes.go": "package app\n\nvar fixed = true\n", "app/nodes_test.go": "package app\n\nfunc a() { proxmox(); fixed() }\n"}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || strings.Contains(stdout, "### Deferred conflicts") || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run = %d, want #5 parked on #3:\n%s\n%s", code, stdout, stderr) + } +} + +// What an original builds on through lines it adds is not seen: #4973's +// $isProxmox. The original is picked; CI on the batch is what catches it. +func TestAddedLinesDependencyIsNotSeen(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: nodes", "", map[string]string{"app/nodes.tpl": "a\nb\n"}) + f.lines() + f.pr(3, "feat(app): define isProxmox", "", map[string]string{"app/proxmox.tpl": "isProxmox\n"}) + f.pr(5, "fix(app): use isProxmox", "", map[string]string{"app/nodes.tpl": "a\nif isProxmox\nb\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// With no batch open, check reads a closed record whatever its version and +// whatever mode it saved as the scheduled run's: the workflow original it +// left waiting is missing, not parked, while its drops and its other +// deferrals still count. +func TestClosedRecordModeFallback(t *testing.T) { + for _, version := range []int{2, 3} { + for _, saved := range []bool{false, true} { + t.Run(fmt.Sprintf("version %d, noWorkflowFiles %v", version, saved), func(t *testing.T) { + f := auxWorld(t) + mc6 := f.git("log", "--first-parent", "--format=%H", "-1", "--grep", "(#6)$", "main") + mc7 := f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: version, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: saved, Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 5, Files: []string{".github/workflows/release.yaml"}, Workflows: true}, {Original: 7, Commit: f.git("rev-parse", mc7+"^2"), Files: []string{"app/seven.txt"}}}, + Dropped: []droppedEntry{{Original: 6, Merge: mc6, Reason: "not for this line", On: 90, Date: "2026-10-26"}}, Stale: []int{}} + pr := f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n") + at := "2026-10-26T12:00:00Z" + pr.State, pr.MergedAt = "closed", &at + code, stdout, stderr := f.exec(&fakeGH{prs: []pullRequest{pr}}, "check", "--line", "release-1.1") + if code != 1 || !strings.Contains(stdout, "/pull/5 ") || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 missing and not parked:\n%s\n%s", code, stdout, stderr) + } + if !strings.Contains(stderr, "note: #6 is dropped") || !strings.Contains(stderr, "warning: deferred conflict on #90 (merged): https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check lost the closed record's drop or deferral:\n%s", stderr) + } + }) + } + } +} + +// Withdrawing a drop puts the original back where the rules and the line +// put it: a partial original is parked again, in the record as in check. +func TestUndropRestoresThePark(t *testing.T) { + for name, world := range map[string]func(*testing.T) *fixture{ + "partial": func(t *testing.T) *fixture { + f, _ := policyWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + return f + }, + "dependency": featWorld, + } { + t.Run(name, func(t *testing.T) { + f := world(t) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--drop", "5", "--reason", "not for 1.1", "--push") + reopen(f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--undrop", "5", "--push") + reopen(f, gh) + if rec := recordOf(t, gh); len(rec.Parked) != 1 || rec.Parked[0].Original != 5 || rec.Parked[0].Kind != name { + t.Errorf("after the undrop the record parks %+v, want #5 parked %s", rec.Parked, name) + } + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked") { + t.Errorf("check = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + }) + } +} + +// A prerequisite the open batch carries is there for what builds on it, +// whatever its label now says: check and run agree that #7 is not parked. +func TestPrerequisiteStillOnTheBatchSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + gh := &fakeGH{} + openBatch(t, f, gh) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + gh.labels = map[string][]int{labelSkip: {5}} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#7 is parked") || !strings.Contains(stdout, "/pull/7 ") { + t.Errorf("check = %d, want #7 missing, not parked:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue"); code != 0 || !strings.Contains(stdout, `"picked":[7]`) { + t.Errorf("run --continue = %d, want #7 picked:\n%s\n%s", code, stdout, stderr) + } +} + // The picker judges what its picks did where the run has got to: #4 // deletes the file #5 also deletes, so after #4 #5 would bring only its // test, which the net merge at the line's head cannot see. @@ -231,6 +489,64 @@ func TestPartialClearedByAnEarlierPick(t *testing.T) { } } +// A reason a person deferred an original for comes before its partial +// park. +func TestReasonComesBeforeThePartialPark(t *testing.T) { + f, _ := policyWorld(t) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + mustRun(t, f, gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "waits for the policy rework", "--push") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ") || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 deferred for its reason:\n%s\n%s", code, stdout, stderr) + } +} + +// The workflow exception holds only while what never comes is tests or +// docs, and while the workflow lines build on nothing missing: a +// workflow original that also fixes a file a feat added, or whose workflow +// line another original still to come wrote, parks in the scheduled run too. +func TestWorkflowExceptionNeedsTestsOnly(t *testing.T) { + t.Run("a never dependency outside tests", func(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(ci): test the tap, and fix it", "", map[string]string{".github/workflows/tap.yaml": "on: push\n", "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":7,`) { + t.Errorf("scheduled run = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a workflow line still to come", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n", "app/four.txt": "old\n"}) + f.lines() + f.onLine("release-1.1", "fix(app): four, the line's way", map[string]string{"app/four.txt": "line\n"}) + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(4, "fix(ci): four", "", map[string]string{"app/four.txt": "four\n", ".github/workflows/release.yaml": "on: [push]\njobs: {}\n"}) + f.pr(5, "fix(ci): run the release on tags", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("scheduled run = %d, want #5 parked:\n%s\n%s", code, stdout, stderr) + } + }) +} + +// What a rebuild takes off the batch is no longer there for what builds on +// it: deferring #5 makes #7, which merged after the batch was built, wait +// on it. +func TestRebuildTakesThePrerequisiteAway(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "fails the e2e"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[],"needs":[5]}`) { + t.Errorf("run --defer 5 = %d, want #7 waiting on #5:\n%s\n%s", code, stdout, stderr) + } +} + // A file the line renamed the way the original does is the original's own // rename, not a deletion left without effect. func TestSameRenameOnBothSidesIsWhole(t *testing.T) { @@ -245,6 +561,54 @@ func TestSameRenameOnBothSidesIsWhole(t *testing.T) { } } +// A partial park only the picker saw, where the net merge at the line's +// head looks whole, holds in check and in the operations that pick nothing +// as the open record keeps it: check does not call the original missing. +func TestPickerParkHoldsInCheck(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a policy", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.lines() + f.pr(4, "fix(app): drop the policy", "", map[string]string{"app/policy.yaml": "", "app/four.txt": "4\n"}) + f.pr(40, "feat(app): the policy again", "", map[string]string{"app/policy.yaml": "allow: all\n"}) + f.pr(5, "fix(app): drop the policy and test it", "", map[string]string{"app/policy.yaml": "", "app/policy_test.txt": "gone\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "note: #5 is parked: only part of it applies") { + t.Errorf("check = %d, want #5 parked as the record says:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "6", "--reason", "fails the e2e"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":5,`) { + t.Errorf("run --defer 6 = %d, want #5 still parked:\n%s\n%s", code, stdout, stderr) + } + // Deferring #4, which made #5 partial, ends the park: #5 applies whole + // on what the batch carries now. + f.git("switch", "-q", "main") + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "4", "--reason", "fails the e2e", "--push") + if code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"missing":[5]`) { + t.Errorf("run --defer 4 = %d, want #5 missing, no longer parked:\n%s\n%s", code, stdout, stderr) + } + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#5 is parked") || !strings.Contains(stdout, "/pull/5 ") { + t.Errorf("check after deferring #4 = %d, want #5 missing:\n%s\n%s", code, stdout, stderr) + } +} + +// A workflow original that waits for a maintainer only as the open record +// lists it: one merged since the batch was built is missing, and so is +// what waits on it. +func TestUnlistedWorkflowWaitIsMissing(t *testing.T) { + f := auxWorld(t) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push") + f.git("switch", "-q", "main") + reopen(f, gh) + f.pr(8, "fix(ci): run the new release on tags", "", map[string]string{".github/workflows/new.yaml": "on: tag\n", "hack/changelog.bats": "@test x {\n false\n}\n# eight\n", "app/eight.txt": appFile + "TWO\n"}) + f.pr(9, "fix(app): nine", "", map[string]string{"app/eight.txt": appFile + "TWO!\n"}) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/9 fix(app): nine (it waits on #8, which is not on the line either)") { + t.Errorf("check = %d, want #9 missing, waiting on #8:\n%s\n%s", code, stdout, stderr) + } +} + // The line's own rename stands for the original's only with the original's // change in it: a fold that leaves the renamed file as it was is refused. func TestSameRenameStillNeedsItsFix(t *testing.T) { @@ -262,6 +626,47 @@ func TestSameRenameStillNeedsItsFix(t *testing.T) { } } +// A prerequisite the line carries is there, though the rules would keep it +// out: a feat carried by a hand adaptation does not park what builds on it. +func TestCarriedExcludedPrerequisiteSatisfies(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a", "", map[string]string{"app/a.txt": appFile}) + f.lines() + mc3 := f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.onLine("release-1.1", "feat(app): a loud two, adapted\n\n(cherry picked from commit "+mc3+")", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "Two", 1)}) + f.pr(5, "fix(app): an excited two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) { + t.Errorf("run = %d, want #5 not parked on the carried #3:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || strings.Contains(stderr, "#5 is parked") { + t.Errorf("check = %d, want #5 missing, not parked:\n%s\n%s", code, stdout, stderr) + } +} + +// With no batch open, a workflow wait a closed record keeps waits on +// nothing: the original is judged as the scheduled run judges it, so one +// that also builds on a feat the line lacks is parked, in check as in run. +func TestClosedWorkflowWaitIsJudgedAsRunJudgesIt(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(ci): test the tap, and fix it", "", map[string]string{".github/workflows/tap.yaml": "on: push\n", "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: 2, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{NoWorkflowFiles: true, Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 7, Files: []string{".github/workflows/tap.yaml"}, Workflows: true}}, Dropped: []droppedEntry{}, Stale: []int{}} + pr := f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n") + at := "2026-10-26T12:00:00Z" + pr.State, pr.MergedAt = "closed", &at + gh := &fakeGH{prs: []pullRequest{pr}} + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); strings.Contains(stdout, "/pull/7 ") || !strings.Contains(stderr, "note: #7 is parked: it builds on #20") { + t.Errorf("check --cut = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files"); code != 0 || !strings.Contains(stdout, `"parked":[{"original":7,`) { + t.Errorf("scheduled run = %d, want #7 parked:\n%s\n%s", code, stdout, stderr) + } +} + // A pick that comes out empty is judged too: #5's own commit did half its // fix, which the line has, and a merge of main into its branch did the // rest, which no pick carries; #5 is not on the line. @@ -305,6 +710,51 @@ func TestDeferTakesOffAPartialExcludedUnit(t *testing.T) { } } +// A stored wait on an original merged after the one that waits cannot be +// what a run wrote. +func TestForwardWaitIsRefused(t *testing.T) { + f := newFixture(t) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "5\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/seven.txt": "7\n"}) + sha := f.git("rev-parse", "origin/main") + rec := lineRecord{Version: recordVersion, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{Mode: modeContinue}}, + Deferred: []deferredEntry{{Original: 5, Files: []string{}, Needs: []int{7}}}, Dropped: []droppedEntry{}, Stale: []int{}} + f.git("push", "-q", "origin", "origin/release-1.1:refs/heads/"+batchBranch1) + gh := &fakeGH{prs: []pullRequest{f.batchPR(90, "release-1.1", batchBranch1, "batch\n\n"+recordBlock(rec)+"\n")}} + if code, _, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 2 || !strings.Contains(stderr, "#5 wait on #7") { + t.Errorf("check = %d, want 2 for a wait on a later original:\n%s", code, stderr) + } +} + +// The workflow exception looks only at what still keeps the original from +// the line: a workflow line its prerequisite wrote, which the open batch +// carries though that prerequisite is labelled backport/skip, does not +// stop it, in check as in run. +func TestWorkflowExceptionIgnoresWhatIsThere(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a workflow", "", map[string]string{".github/workflows/release.yaml": "on: push\njobs: {}\n"}) + f.lines() + f.pr(3, "feat(ci): test the changelog", "", map[string]string{"hack/changelog.bats": "@test x {\n true\n}\n"}) + f.pr(4, "fix(ci): run the release on pushes", "", map[string]string{".github/workflows/release.yaml": "on: [push]\njobs: {}\n"}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(ci): run the release on tags too", "", map[string]string{".github/workflows/release.yaml": "on: [push, tag]\njobs: {}\n", "hack/changelog.bats": "@test x {\n false\n}\n"}) + gh := &fakeGH{} + mustRun(t, f, gh, "run", "--line", "release-1.1", "--push") + f.git("switch", "-q", "main") + reopen(f, gh) + gh.labels = map[string][]int{labelSkip: {4}} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--no-workflow-files", "--push"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[".github/workflows/release.yaml"],"workflows":true}`) { + t.Fatalf("scheduled run = %d, want #7 waiting for a maintainer:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#7 changes .github/workflows/ and waits on batch PR #90") { + t.Errorf("check --cut = %d, want #7 waiting, as run has it:\n%s\n%s", code, stdout, stderr) + } +} + // An empty pick leaves a deletion undone that only a merge of main into the // original's branch made: the original is not on the line. func TestEmptyPickLeavesADeletionUndone(t *testing.T) { @@ -329,3 +779,18 @@ func TestEmptyPickLeavesADeletionUndone(t *testing.T) { t.Errorf("run = %d, want #5 parked as partial, not on the line:\n%s\n%s", code, stdout, stderr) } } + +// A file the line already has as the original makes it builds on nothing +// the line lacks: #5 fixes a.txt, which the line fixed the same way, and +// b.txt, so it picks whole and is not parked on the feat that wrote a.txt. +func TestAlreadyAppliedFileBuildsOnNothing(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: a and b", "", map[string]string{"app/a.txt": appFile, "app/b.txt": "old\n"}) + f.lines() + f.pr(3, "feat(app): a loud two", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.onLine("release-1.1", "fix(app): two, the line's way", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(5, "fix(app): an excited two, and b", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": "fixed\n"}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `"picked":[5]`) { + t.Errorf("run = %d, want #5 picked, not parked:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 81b7626e3..0077b5302 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -21,6 +21,7 @@ import ( "errors" "fmt" "io" + "maps" "regexp" "slices" "strings" @@ -258,6 +259,9 @@ func parseRecord(body string, l line) (*lineRecord, error) { default: return nil, fmt.Errorf("the %s block is version %d, which this backport-batch does not read; it reads 1 to %d", deferredFence, version, recordVersion) } + if err == nil && version < 3 && slices.ContainsFunc(rec.Deferred, func(d deferredEntry) bool { return d.Needs != nil }) { + err = fmt.Errorf("version %d has no waits on other originals", version) + } if err != nil { return nil, fmt.Errorf("reading the %s block: %w", deferredFence, err) } @@ -294,7 +298,7 @@ func (rec *lineRecord) validate(l line) error { // A conflict names the commit that failed and its files, a deferral // its reason, and a wait its workflow files. kinds := 0 - for _, is := range []bool{d.Commit != "" || len(d.Files) > 0 && !d.Workflows && d.Reason == "", d.Reason != "", d.Workflows} { + for _, is := range []bool{d.Commit != "" || len(d.Files) > 0 && !d.Workflows && d.Reason == "" && len(d.Needs) == 0, d.Reason != "", d.Workflows, len(d.Needs) > 0} { if is { kinds++ } @@ -312,6 +316,13 @@ func (rec *lineRecord) validate(l line) error { return fmt.Errorf("names files of #%d that are not as a run writes them: %q", d.Original, d.Files) case d.More < 0: return fmt.Errorf("counts %d more files for #%d", d.More, d.Original) + case len(d.Needs) > 0 && (len(d.Files) > 0 || d.More != 0): + return fmt.Errorf("has #%d wait on other originals with files of its own", d.Original) + } + for i, n := range d.Needs { + if n <= 0 || n == d.Original || i > 0 && d.Needs[i-1] >= n { + return fmt.Errorf("has #%d wait on %v, which is not what a run writes", d.Original, d.Needs) + } } } for _, d := range rec.Dropped { @@ -347,6 +358,27 @@ func (rec *lineRecord) validate(l line) error { } changedSeen[n] = true } + if err := rec.waitsHold(); err != nil { + return err + } + // What is on the batch is neither deferred nor parked. + left := map[int]bool{} + for _, d := range rec.Deferred { + left[d.Original] = true + } + for _, p := range rec.Parked { + left[p.Original] = true + if slices.Contains(p.Commits, p.Merge) { + return fmt.Errorf("parks #%d for its own merge", p.Original) + } + } + for _, list := range [][]int{rec.Partial, rec.Changed} { + for _, n := range list { + if left[n] { + return fmt.Errorf("has #%d on the batch and left off it at once", n) + } + } + } partialSeen := map[int]bool{} for _, n := range rec.Partial { // A unit that does only part of its original is still on the batch. @@ -401,6 +433,74 @@ func (b *builtFrom) validate() error { var runURLRE = regexp.MustCompile(`^https://[^/\s]+/[^/\s]+/[^/\s]+/actions/runs/[0-9]+(/attempts/[0-9]+)?$`) +// waitsHold holds what waits and parks need to what the record says of the +// originals they need: a wait needs ones still to come, so none parked or +// dropped, a park ones that never come, so none deferred, and no wait ends +// up waiting on itself. Its errors read after "the block". +func (rec *lineRecord) waitsHold() error { + deferred, never := map[int][]int{}, map[int]bool{} + for _, d := range rec.Deferred { + deferred[d.Original] = d.Needs + } + for _, p := range rec.Parked { + never[p.Original] = true + } + for _, d := range rec.Dropped { + never[d.Original] = true + } + for _, d := range rec.Deferred { + for _, n := range d.Needs { + if never[n] { + return fmt.Errorf("has #%d wait on #%d, which never comes", d.Original, n) + } + } + } + for _, p := range rec.Parked { + for _, n := range p.Needs { + if _, ok := deferred[n]; ok { + return fmt.Errorf("parks #%d for #%d, which is still to come", p.Original, n) + } + } + } + needs := map[int][]int{} + for _, d := range rec.Deferred { + needs[d.Original] = d.Needs + } + for _, p := range rec.Parked { + needs[p.Original] = p.Needs + } + return waitCycle(needs) +} + +// waitCycle refuses waits that lead back to where they started, which no +// run writes: a prerequisite merged before what needs it. +func waitCycle(needs map[int][]int) error { + state := map[int]int{} // 1 on the path, 2 done + var visit func(n int) error + visit = func(n int) error { + switch state[n] { + case 1: + return fmt.Errorf("has #%d wait on itself through what it waits on", n) + case 2: + return nil + } + state[n] = 1 + for _, f := range needs[n] { + if err := visit(f); err != nil { + return err + } + } + state[n] = 2 + return nil + } + for _, n := range slices.Sorted(maps.Keys(needs)) { + if err := visit(n); err != nil { + return err + } + } + return nil +} + // lineRecordWithout is a version 2 record as the newest version holds it. func lineRecordWithout(v2 lineRecordV2) lineRecord { return lineRecord{Version: v2.Version, Line: v2.Line, Built: v2.Built, Report: v2.Report, Closing: v2.Closing, Deferred: v2.Deferred, Dropped: v2.Dropped, Stale: v2.Stale} diff --git a/cmd/backport-batch/record_test.go b/cmd/backport-batch/record_test.go index d50685edb..4bbddc58c 100644 --- a/cmd/backport-batch/record_test.go +++ b/cmd/backport-batch/record_test.go @@ -380,10 +380,32 @@ func TestRecordVersion3(t *testing.T) { block := func(json string) string { return "text\n\n" + deferredFence + "\n" + json + "\n```\n" } sha, other := strings.Repeat("a", 40), strings.Repeat("c", 40) parked := `,"parked":[{"original":10,"merge":"` + sha + `","kind":"partial","files":["a","b"]},{"original":11,"merge":"` + sha + `","kind":"dependency","needs":[3,4],"commits":["` + other + `"],"announced":true}],"partial":[9]` - rec, err := parseRecord(block(v3(parked)), l) - if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) { + waits := strings.Replace(v3(parked), `{"original":6,"files":[],"reason":"fails the e2e"}`, `{"original":6,"files":[],"needs":[3,5]}`, 1) + rec, err := parseRecord(block(waits), l) + if err != nil || rec.Version != 3 || len(rec.Parked) != 2 || !rec.Parked[1].Announced || !slices.Equal(rec.Partial, []int{9}) || !slices.Equal(rec.deferred()[6].Needs, []int{3, 5}) { t.Fatalf("parseRecord of a whole v3 record = %+v, %v", rec, err) } + for name, json := range map[string]string{ + "version 1 with a wait": `{"line":"release-1.1","deferred":[{"original":6,"files":[],"needs":[3,5]}]}`, + "a wait with files": strings.Replace(waits, `"files":[],"needs":[3,5]`, `"files":["a"],"needs":[3,5]`, 1), + "a wait that is a conflict": strings.Replace(waits, `"files":[],"needs":[3,5]`, `"commit":"`+sha+`","files":[],"needs":[3,5]`, 1), + "a wait on itself": strings.Replace(waits, `"needs":[3,5]`, `"needs":[3,6]`, 1), + "an unsorted wait": strings.Replace(waits, `"needs":[3,5]`, `"needs":[5,3]`, 1), + "a wait on nothing": strings.Replace(waits, `"needs":[3,5]`, `"needs":[0]`, 1), + "a wait cycle": strings.Replace(strings.Replace(waits, `"needs":[3,5]`, `"needs":[12]`, 1), `"deferred":[`, `"deferred":[{"original":12,"files":[],"needs":[6]},`, 1), + "a wait on a parked one": strings.Replace(waits, `"needs":[3,5]`, `"needs":[10]`, 1), + "a wait on a dropped one": strings.Replace(waits, `"needs":[3,5]`, `"needs":[8]`, 1), + "a park on a deferred one": strings.Replace(waits, `"needs":[3,4]`, `"needs":[3,5]`, 1), + "a partial unit also deferred": strings.Replace(strings.Replace(waits, `"stale":[9]`, `"stale":[9,5]`, 1), `"partial":[9]`, `"partial":[9,5]`, 1), + "a changed pick also parked": v3(parked + `,"changed":[10]`), + "a changed pick also deferred": v3(parked + `,"changed":[5]`), + "a park on its own merge": strings.Replace(waits, `"commits":["`+other+`"]`, `"commits":["`+sha+`"]`, 1), + "a park cycle": strings.Replace(strings.Replace(waits, `"needs":[3,4]`, `"needs":[3,13]`, 1), `"parked":[`, `"parked":[{"original":13,"merge":"`+sha+`","kind":"dependency","needs":[11]},`, 1), + } { + if _, err := parseRecord(block(json), l); err == nil { + t.Errorf("%s: parseRecord accepted it", name) + } + } for name, json := range map[string]string{ "version 2 with a park": v2(parked), "version 4": strings.Replace(v3(""), `"version":3`, `"version":4`, 1), diff --git a/cmd/backport-batch/report.go b/cmd/backport-batch/report.go index de201fd4a..d1ae4a4b5 100644 --- a/cmd/backport-batch/report.go +++ b/cmd/backport-batch/report.go @@ -145,7 +145,7 @@ func numbers(vs []*verdict) []int { // deferredEntries is what the record and the report say about the conflicts, // the deferrals and the waits of a run. -func deferredEntries(conflicts, held, waiting []pick) []deferredEntry { +func deferredEntries(conflicts, held, waiting, needs []pick) []deferredEntry { out := []deferredEntry{} for _, p := range conflicts { files := cleanFiles(p.files[:min(len(p.files), filesKept)]) @@ -158,6 +158,9 @@ func deferredEntries(conflicts, held, waiting []pick) []deferredEntry { files := cleanFiles(p.files[:min(len(p.files), filesKept)]) out = append(out, deferredEntry{Original: p.v.o.number, Files: files, More: len(p.files) - len(files) + p.more, Workflows: true}) } + for _, p := range needs { + out = append(out, deferredEntry{Original: p.v.o.number, Files: []string{}, Needs: slices.Sorted(slices.Values(p.needs))}) + } return out } @@ -188,7 +191,7 @@ func buildReport(b *batch, r *report) (reportDoc, error) { doc.Overrides.Skip = append(doc.Overrides.Skip, v.o.number) } } - doc.Primary = reportPrimary{Picked: numbers(r.onBranch), Missing: numbers(r.missing), Deferred: deferredEntries(r.deferred, r.held, r.waiting), Parked: parkedEntries(r.parked)} + doc.Primary = reportPrimary{Picked: numbers(r.onBranch), Missing: numbers(r.missing), Deferred: deferredEntries(r.deferred, r.held, r.waiting, r.needs), Parked: parkedEntries(r.parked)} for _, l := range doc.Primary.Skipped.lists() { *l.list = []int{} } @@ -415,6 +418,9 @@ func (d *reportDoc) links() error { return fmt.Errorf("parks #%d for #%d, which it has %s", p.Original, n, or(c, "nowhere")) } } + if other[p.Merge] || slices.Contains(p.Commits, p.Merge) { + return fmt.Errorf("parks #%d by a merge it counts as no pull request's, or for its own merge", p.Original) + } for _, c := range p.Commits { if !other[c] { return fmt.Errorf("parks #%d for %.12s, which it does not count among main's commits that merged no pull request", p.Original, c) @@ -480,6 +486,28 @@ func (d *reportDoc) links() error { if e.Workflows && !d.Built.Options.NoWorkflowFiles { return fmt.Errorf("has #%d waiting for its workflow files, but was made bringing them over", e.Original) } + // What a wait waits on is still to come: not on the line, deferred, + // or held by rule 6. + for _, n := range e.Needs { + switch c := where[n]; c { + case "missing", "deferred", "skipped.newer": + default: + return fmt.Errorf("has #%d wait on #%d, which it has %s", e.Original, n, or(c, "nowhere")) + } + } + if d.Format == 1 && len(e.Needs) > 0 { + return fmt.Errorf("has #%d wait on other originals, which format 1 does not", e.Original) + } + } + needs := map[int][]int{} + for _, e := range d.Primary.Deferred { + needs[e.Original] = e.Needs + } + for _, p := range d.Primary.Parked { + needs[p.Original] = p.Needs + } + if err := waitCycle(needs); err != nil { + return err } if d.Built.Options.Mode == modeClose && len(d.Primary.Picked)+len(d.Overlays.Stale) > 0 { return errors.New("closes the batch, but still has originals on it") @@ -734,6 +762,12 @@ func (a *app) printReport(pr *pullRequest, s storedReport, inBody *storedReport, switch { case e.Workflows: text += "waits for a maintainer, changes " + strings.Join(e.Files, ", ") + case len(e.Needs) > 0: + var needs []string + for _, n := range e.Needs { + needs = append(needs, "#"+strconv.Itoa(n)) + } + text += "waits on " + strings.Join(needs, ", ") case e.Reason != "": text += e.Reason default: diff --git a/cmd/backport-batch/report_test.go b/cmd/backport-batch/report_test.go index 1e5048e04..40a9a0673 100644 --- a/cmd/backport-batch/report_test.go +++ b/cmd/backport-batch/report_test.go @@ -633,6 +633,27 @@ func TestReportRefusesWhatItCannotVouchFor(t *testing.T) { d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Commits: []string{strings.Repeat("c", 40)}}} }, "a partial unit not on the batch": func(d *reportDoc) { d.Overlays.Partial = []int{5} }, + "a wait on an original it picks": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{5}}} + }, + "a wait on an original it puts nowhere": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{19}}} + }, + "a park on its own merge as a commit": func(d *reportDoc) { + d.Other = append(d.Other, strings.Repeat("b", 40)) + d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Commits: []string{strings.Repeat("b", 40)}}} + }, + "a park cycle": func(d *reportDoc) { + d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkDependency, Needs: []int{18}}, {Original: 18, Merge: strings.Repeat("b", 40), Kind: parkDependency, Needs: []int{16}}} + }, + "a wait cycle": func(d *reportDoc) { + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{7}}, {Original: 7, Files: []string{}, Needs: []int{6}}} + }, + "format 1 with a wait": func(d *reportDoc) { + d.Format = 1 + d.Primary.Missing = []int{19} + d.Primary.Deferred = []deferredEntry{{Original: 6, Files: []string{}, Needs: []int{19}}} + }, "format 1 with a park": func(d *reportDoc) { d.Format = 1 d.Primary.Parked = []parkedEntry{{Original: 16, Merge: strings.Repeat("b", 40), Kind: parkPartial, Files: []string{"a"}}} diff --git a/cmd/backport-batch/wait_test.go b/cmd/backport-batch/wait_test.go new file mode 100644 index 000000000..35dd5c1d6 --- /dev/null +++ b/cmd/backport-batch/wait_test.go @@ -0,0 +1,236 @@ +/* +Copyright 2026 The Cozystack Authors. + +Licensed under the Apache License, Version 2.0 (the "License"); +you may not use this file except in compliance with the License. +You may obtain a copy of the License at + + http://www.apache.org/licenses/LICENSE-2.0 + +Unless required by applicable law or agreed to in writing, software +distributed under the License is distributed on an "AS IS" BASIS, +WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +See the License for the specific language governing permissions and +limitations under the License. +*/ + +package main + +import ( + "strings" + "testing" +) + +// chainWorld has #5 conflict on the line and change a shared line, and #7 +// change that line again: #7 builds on #5. +func chainWorld(t *testing.T) (*fixture, string) { + t.Helper() + f := newFixture(t) + f.squash(2, "chore: five and shared", "", map[string]string{"app/five.txt": "old\n", "app/shared.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + mc5 := f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/shared.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/shared.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + return f, mc5 +} + +// An original that builds on a deferred conflict waits for it: check warns, +// notify lists it after the conflict, and once the conflict lands, the next +// run picks it. +func TestWaitForADeferredDependency(t *testing.T) { + f, mc5 := chainWorld(t) + gh := &fakeGH{} + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push") + if code != 0 { + t.Fatalf("run --push = %d\n%s", code, stderr) + } + if !strings.Contains(stdout, `{"original":7,"files":[],"needs":[5]}`) || !strings.Contains(stdout, "### Waiting on another original") { + t.Errorf("#7 does not wait on #5:\n%s", stdout) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || !strings.Contains(stderr, "warning: #7 waits on #5, a deferred conflict on #90: https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check = %d, want a warning that #7 waits on #5:\n%s\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "notify", "--line", "release-1.1", "--push") + if code != 0 || !strings.Contains(gh.issueComments[len(gh.issueComments)-1].Body, "- [#7](https://github.com/cozystack/cozystack/pull/7), after [#5](https://github.com/cozystack/cozystack/pull/5)") { + t.Errorf("notify = %d, want #7 listed after #5:\n%s\n%v", code, stdout, gh.issueComments) + } + // The batch merges, and #5 lands on the line by hand, resolved: the + // next batch picks #7. + mergeBatch(f, gh) + f.onLine("release-1.1", "fix(app): five (#5)\n\n(cherry picked from commit "+mc5+")", map[string]string{"app/five.txt": "five\n", "app/shared.txt": strings.Replace(appFile, "two", "TWO", 1)}) + code, stdout, stderr = f.exec(gh, "run", "--line", "release-1.1", "--continue") + if code != 0 || !strings.Contains(stdout, `"picked":[7]`) { + t.Errorf("run --continue after #5 landed = %d, want #7 picked:\n%s\n%s", code, stdout, stderr) + } +} + +// Of several prerequisites still to come, the strongest decides: one not on +// the line at all, then one waiting for a maintainer, then a conflict or a +// deferral, then one rule 6 holds; a chain inherits what its end decides. +func TestWaitReduction(t *testing.T) { + t.Run("a conflict and a workflow wait", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/five.txt": "old\n", "app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(ci): six", "", map[string]string{".github/workflows/six.yaml": "on: push\n", "app/b.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 || !strings.Contains(stdout, `{"original":7,"files":[],"needs":[6]}`) { + t.Fatalf("run = %d, want #7 waiting on #6 alone:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#7 waits on #6, which changes .github/workflows/ and waits on batch PR #90 for a maintainer") { + t.Errorf("check --cut = %d, want #7 to hold the cut:\n%s\n%s", code, stdout, stderr) + } + if code, stdout, _ := f.exec(gh, "notify", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, "2 originals on #90 change .github/workflows/ or wait on one that does") { + t.Errorf("notify = %d, want #7 among the waits:\n%s", code, stdout) + } + }) + t.Run("a conflict and one not on the line", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/five.txt": "old\n", "app/a.txt": appFile, "app/b.txt": appFile}) + f.lines() + f.onLine("release-1.1", "fix(app): five, the line's way", map[string]string{"app/five.txt": "line\n"}) + f.pr(5, "fix(app): five", "", map[string]string{"app/five.txt": "five\n", "app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(6, "fix(app): six", "", map[string]string{"app/six.txt": "6\n"}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + f.pr(9, "fix(app): nine", "", map[string]string{"app/b.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(10, "fix(app): ten", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/b.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/10 fix(app): ten (it waits on #9, which is not on the line either)") { + t.Errorf("check = %d, want #10 to fail it, waiting on #9:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a prerequisite on the open batch", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + gh := &fakeGH{} + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(stdout, `"picked":[5,7]`) { + t.Fatalf("run = %d, want #5 and #7 picked:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1", "--cut"); code != 1 || !strings.Contains(stdout, "#5 is only on open batch PR #90") || strings.Contains(stdout+stderr, "waits on") { + t.Errorf("check --cut = %d, want #5 and #7 only on the batch and no wait:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a direct reason", func(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "5", "--reason", "needs the tap", "--push"); code != 0 { + t.Fatalf("run --defer = %d\n%s", code, stderr) + } + reopen(f, gh) + if code, stdout, stderr := f.exec(gh, "check", "--line", "release-1.1"); code != 0 || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/5 ") || strings.Contains(stderr, "parked") { + t.Errorf("check = %d, want #5 deferred for its reason, not parked:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a prerequisite rule 6 holds", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + mc7 := f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/seven.txt": "7\n"}) + f.onLine("release-1.1", "fix(app): seven (#7)\n\n(cherry picked from commit "+mc7+")", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1), "app/seven.txt": "7\n"}) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.0"); code != 0 || !strings.Contains(stderr, "note: #7 waits on #5, which is not on release-1.1 yet") { + t.Errorf("check of release-1.0 = %d, want a note that #7 waits on #5:\n%s\n%s", code, stdout, stderr) + } + }) + t.Run("a chain", func(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(4, "fix(app): four", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO", 1)}) + f.pr(5, "fix(app): five", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!", 1)}) + f.pr(7, "fix(app): seven", "", map[string]string{"app/a.txt": strings.Replace(appFile, "two", "TWO!!", 1)}) + if code, stdout, stderr := f.exec(nil, "check", "--line", "release-1.1"); code != 1 || !strings.Contains(stdout, "/pull/7 fix(app): seven (it waits on #5, which is not on the line either)") { + t.Errorf("check = %d, want #7 failing it through #5:\n%s\n%s", code, stdout, stderr) + } + }) +} + +// run's report, check and notify give an original one disposition: a +// workflow original whose only missing dependency is a test waits for a +// maintainer, and one deferred for a reason stays deferred, though it +// builds on what never comes; neither is parked. +func TestDispositionOrderIsSharedByRunCheckNotify(t *testing.T) { + f := auxWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(7, "fix(app): fix the tap", "", map[string]string{"app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + gh := &fakeGH{} + if code, _, stderr := f.exec(gh, "run", "--line", "release-1.1", "--no-workflow-files", "--push"); code != 0 { + t.Fatalf("run = %d\n%s", code, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--defer", "7", "--reason", "waits for the tap", "--push") + if code != 0 { + t.Fatalf("run --defer = %d\n%s", code, stderr) + } + if strings.Contains(stdout, `"parked":[{`) || !strings.Contains(stdout, `{"original":5,"files":[".github/workflows/release.yaml"],"workflows":true}`) || !strings.Contains(stdout, `{"original":7,"files":[],"reason":"waits for the tap"}`) { + t.Errorf("the report does not have #5 waiting and #7 deferred, unparked:\n%s", stdout) + } + reopen(f, gh) + code, stdout, stderr = f.exec(gh, "check", "--line", "release-1.1") + if code != 0 || strings.Contains(stderr, "parked") || !strings.Contains(stderr, "warning: waiting on #90 for a maintainer, since it changes .github/workflows/: https://github.com/cozystack/cozystack/pull/5 ") || !strings.Contains(stderr, "warning: deferred on #90 to the next batch: https://github.com/cozystack/cozystack/pull/7 ") { + t.Errorf("check = %d, want #5 waiting and #7 deferred:\n%s\n%s", code, stdout, stderr) + } + code, stdout, _ = f.exec(gh, "notify", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, "1 originals on #90 change .github/workflows/") || strings.Contains(stdout, "parked") || strings.Contains(stdout, "conflicts") { + t.Errorf("notify = %d, want #5 waiting and nothing for #7:\n%s", code, stdout) + } +} + +// A never and a pending prerequisite together park. +func TestParkWinsOverWait(t *testing.T) { + f, _ := chainWorld(t) + f.pr(20, "feat(app): tap", "", map[string]string{"app/tap.go": appFile}) + f.pr(8, "fix(app): eight", "", map[string]string{"app/shared.txt": strings.Replace(appFile, "two", "TWO!!", 1), "app/tap.go": strings.Replace(appFile, "two", "TWO", 1)}) + code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1") + if code != 0 || !strings.Contains(stdout, `"kind":"dependency","needs":[20]`) || strings.Contains(stdout, `"original":8,"files":[],"needs"`) { + t.Errorf("run = %d, want #8 parked on #20, not waiting on #7:\n%s\n%s", code, stdout, stderr) + } +} + +// A prerequisite picked earlier in the same batch satisfies what builds on +// it, and so does one whose pick came out empty because an earlier pick +// already carried it: #4280 and #3938. +func TestDependencySatisfiedEarlierInTheBatch(t *testing.T) { + f := newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(3, "fix(kafka): delete hooks", "", map[string]string{"app/hooks.yaml": appFile}) + f.pr(4, "fix(kafka): fix the delete hook", "", map[string]string{"app/hooks.yaml": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[3,4]`) { + t.Errorf("run = %d, want #3 and #4 picked:\n%s\n%s", code, stdout, stderr) + } + + // #30 adds back the file #2 brought and a feat took away, so its pick + // comes out empty after #2's, and #4 builds on #30. + f = newFixture(t) + f.squash(2, "chore: files", "", map[string]string{"app/a.txt": appFile}) + f.lines() + f.pr(20, "fix(kafka): delete hooks, and two", "", map[string]string{"app/hooks.yaml": appFile, "app/two.txt": "2\n"}) + f.pr(21, "feat(kafka): no delete hooks", "", map[string]string{"app/hooks.yaml": ""}) + f.pr(30, "fix(kafka): delete hooks again", "", map[string]string{"app/hooks.yaml": appFile}) + f.pr(4, "fix(kafka): fix the delete hook", "", map[string]string{"app/hooks.yaml": strings.Replace(appFile, "two", "TWO", 1)}) + if code, stdout, stderr := f.exec(nil, "run", "--line", "release-1.1"); code != 0 || !strings.Contains(stdout, `"picked":[4,20]`) || strings.Contains(stdout, `"needs"`) { + t.Errorf("run with #30's pick empty = %d, want #4 picked:\n%s\n%s", code, stdout, stderr) + } +} diff --git a/docs/release/backport-mechanics.md b/docs/release/backport-mechanics.md index 1ee035618..3474bffea 100644 --- a/docs/release/backport-mechanics.md +++ b/docs/release/backport-mechanics.md @@ -14,7 +14,7 @@ go build ./cmd/backport-batch ./backport-batch check --line release-X.Y --cut ``` -[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut, and so does a unit on the open batch that carries only part of its original. A change that reached the line by hand counts like any other, by what reached the line. +[`backport-batch check`](../../cmd/backport-batch/README.md) applies the rules to every pull request merged into `main` since the line branched, and looks for each one they take on the line and then on the open batch pull request: by picks whose `-x` lines name its merge commit or every commit of its own, by a pick that would change nothing, and, on commits from before the policy's batch-only day, by what their text says, such as `Backport of #N`. With `--cut` a change found only on the open batch pull request fails the check too, since a cut tags the line and would not carry it: merge the batch pull request first. A conflict the batch deferred, or an original deferred to the next batch, prints as a warning without failing the check, also once that batch merged or closed. An original that waits on the open batch for a maintainer fails the cut, and so does a unit on the open batch that carries only part of its original. An original parked because it builds on a change the line never gets, or would land only in part, prints as a note; one that waits on another original fails, warns or notes by what it waits on. A change that reached the line by hand counts like any other, by what reached the line. ### Backporting code that touches monotonic counters diff --git a/docs/release/backport-tasks/resolve-conflict.md b/docs/release/backport-tasks/resolve-conflict.md index ed3798d10..885e568d4 100644 --- a/docs/release/backport-tasks/resolve-conflict.md +++ b/docs/release/backport-tasks/resolve-conflict.md @@ -34,7 +34,7 @@ git add git -c core.hooksPath=/dev/null cherry-pick --continue --no-edit ``` -When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. When that change will never reach the line, drop the original there instead, with `backport-batch run --line --drop N --reason "" --push`, so later batches stop trying it. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. A resolution that ports only what the line has and leaves a file of the original as it was, one the line does not have for instance, is refused by `fold`: such a port goes to the line as a pull request of its own, as [fixes that exist only on a release line](../backports.md#fixes-that-exist-only-on-a-release-line) describes, and the original is dropped there. +When the original builds on something the line does not have, a helper, a value or another pull request, the resolution would have to bring that along, which this task does not allow: abort with `git cherry-pick --abort`, go back with `git switch --detach "$base"`, and comment on the batch pull request which change the original depends on; it stays deferred until that change is on the line. An original the batch parked because it builds on such a change is the same task, and lands only as a complete adaptation: every file it changes changed, unless the line already has that file as the original makes it. When that change will never reach the line, drop the original there instead, with `backport-batch run --line --drop N --reason "" --push`, so later batches stop trying it. A commit whose change the line already has stops the pick as empty; skip it with `git cherry-pick --skip`. A resolution that ports only what the line has and leaves a file of the original as it was, one the line does not have for instance, is refused by `fold`: such a port goes to the line as a pull request of its own, as [fixes that exist only on a release line](../backports.md#fixes-that-exist-only-on-a-release-line) describes, and the original is dropped there. For a merged pull request, join the picks to the batch head with the merge commit: diff --git a/docs/release/backports.md b/docs/release/backports.md index f76cc3438..71fa467eb 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -27,6 +27,8 @@ A pull request the rules take for every line can still be wrong for one of them, The batch also leaves a pull request out of a line on its own when it cannot land there whole. A pick that would leave part of what the pull request does on `main` without effect on the line, a change to a file the line does not have, or the deletion of a file the line never had or already removed, is not made: the pull request is parked, the batch pull request lists it under "Parked" with those files, and `check` prints it as a note without failing. A file the line already has as the pull request makes it counts as there, and so does a file `main` or the line renamed, under its other name. A unit already on the open batch that does only part of its original fails `check --cut` and `backport-batch verify`, whatever its label says, until `backport-batch run --defer` takes it out. What applies of a parked pull request goes to the line as a pull request of its own, as [fixes that exist only on a release line](#fixes-that-exist-only-on-a-release-line) describes. +A pull request is parked too when it builds on a change the line never gets: it changes a file a `feat` added on `main`, say, or a line such a change wrote, and the line has neither. `backport-batch` reads that from git, as [its README](../../cmd/backport-batch/README.md#what-an-original-builds-on) describes, and names the change in the batch pull request. One that builds on a change that is still to come, a deferred conflict, one waiting for a maintainer, one rule 6 holds or one not tried yet, waits for it instead, listed under "Waiting on another original", and the batch tries it again once that change is on the line; one picked earlier in the same batch counts as there. A complete hand adaptation of a parked pull request, every file it changes changed, still lands on the batch through [resolving a conflict](./backport-tasks/resolve-conflict.md). + ### What waits for the next minor `feat`, `chore` and `docs` pull requests, and dependency bumps (scope `deps`), ship with the next minor, not in a patch release; only a maintainer's `backport/include`, below, takes one. If a class of fixes should also wait for a minor, it gets a new rule on this page; a single pull request that the rules get wrong is what the overrides below are for. @@ -103,7 +105,7 @@ Each run also reports where every pull request merged into `main` since the line ### Changes to workflows -The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. +The scheduled run's token may not push a commit that changes `.github/workflows/`, so the scheduled run leaves such an original out (`run --no-workflow-files`). That holds also for one whose only missing dependency that never comes is a test or documentation, which a maintainer's run, with the token that may push the workflow, parks instead: what lands of it then is a pull request of its own; `check` reads which of the two the open batch was assembled as from its record, and with no batch open takes the scheduled run's. The batch pull request lists it under "Waiting for a maintainer", with the workflow files it changes, and so does the batch's comment. It is no conflict. The policy still takes it: `check` warns about it while it waits, and `check --cut` fails on it. When nothing else is picked there is no batch pull request to list it on, and `check` reports it as missing. A maintainer brings it over with their own token, one that may push workflow changes (the Workflows permission, or the classic `workflow` scope), by continuing the open batch without the flag: From 885752eb5aeed5e31b3fd6dd2922bca64ee990cd Mon Sep 17 00:00:00 2001 From: Myasnikov Daniil Date: Tue, 10 Nov 2026 11:11:15 +0000 Subject: [PATCH 5/5] feat(release): announce a parked original once on the open batch A park is a decision a maintainer has to act on: port what applies as a pull request of its own, or drop the original. Listing every park in every notify comment would make that list noise within a week, and a park nobody hears about is a fix that silently never ships. notify now names each park the open batch's record holds that is not yet marked announced, and once its comment is written marks those parks in the record with gh pr edit --body-file. It writes only while the pull request is open and its body still carries the record notify read, so a run that rewrote the body in between loses nothing; a mark that fails exits 2 and the next notify names the park once more. run carries the mark into the next record, a merged or closed carrier's too, only while the park is the same event: merge, kind, prerequisites, commits and files. A park that changed is news and is named again. With no batch open nothing is named (M2); the next batch's first notify does it. Assisted-By: LLM Signed-off-by: Myasnikov Daniil --- cmd/backport-batch/README.md | 2 + cmd/backport-batch/assemble.go | 4 +- cmd/backport-batch/body.go | 3 + cmd/backport-batch/fixture_test.go | 16 +++ cmd/backport-batch/github.go | 7 ++ cmd/backport-batch/notify.go | 72 ++++++++++++- cmd/backport-batch/notify_test.go | 160 +++++++++++++++++++++++++++++ cmd/backport-batch/record.go | 33 ++++++ docs/release/backports.md | 2 +- 9 files changed, 295 insertions(+), 4 deletions(-) diff --git a/cmd/backport-batch/README.md b/cmd/backport-batch/README.md index c0b34bb3c..81afd0ef6 100644 --- a/cmd/backport-batch/README.md +++ b/cmd/backport-batch/README.md @@ -73,6 +73,8 @@ Reads the [line's record](#the-lines-record-and-reports) first, as the other com An original the record has wait on others is listed once, as `#M, after #F`, where what decides its wait is listed: after a deferred conflict with the conflicts, after a workflow wait with the waits; one waiting on what is missing or held by rule 6 is not listed. A unit on the batch that carries only part of its original is listed apart, to be taken out before the cut. +Each park the record of the open batch holds is named once. `run` marks a park `announced` when the line's record it found, open or closed, holds the same park announced: the same merge, kind, prerequisites, commits and files; a park that changed is news again. `notify` lists the parks not yet marked, and with `--push`, once the comment is written, marks them in the record with `gh pr edit --body-file`, only while the pull request is open and its body still carries the record `notify` read. A mark that fails after the comment exits 2, and the next `notify` names those parks once more. With no batch open nothing is named, and the next batch's first `notify` names what it parks. + With `--push` it writes the same into one comment on the batch pull request: the conflicts, each to be resolved as the [conflict task](../../docs/release/backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. The comment starts with `` and is one per pull request: a later run edits it in place rather than adding another, and once nothing waits it says so. Only a comment whose author the batch trusts counts as it, so one someone else wrote with the same first line is neither edited nor believed. It mentions `--mention`, an `org/team` or a login, with or without `@`, only while something waits. A failed batch CI is not among what it reports: the run has no trigger on the batch's CI finishing. diff --git a/cmd/backport-batch/assemble.go b/cmd/backport-batch/assemble.go index 35ad92214..3c025517e 100644 --- a/cmd/backport-batch/assemble.go +++ b/cmd/backport-batch/assemble.go @@ -401,7 +401,7 @@ func found(h *history, v *verdict) bool { // newReport starts the report of a run with what the rules left out and what // the labels overrode. func newReport(b *batch) *report { - r := &report{line: b.line, newer: b.newer, skipped: map[int][]*verdict{}, other: len(b.other), start: b.start} + r := &report{line: b.line, newer: b.newer, skipped: map[int][]*verdict{}, other: len(b.other), start: b.start, carried: b.src.rec} for _, v := range b.verdicts { if v.rule != 0 { r.skipped[v.rule] = append(r.skipped[v.rule], v) @@ -491,7 +491,7 @@ func (r *report) lineRecord(doc reportDoc) lineRecord { } slices.SortFunc(dropped, func(a, b droppedEntry) int { return a.Original - b.Original }) return lineRecord{Version: recordVersion, Line: doc.Line, Built: &built, Report: doc.ID, Closing: r.closing, Deferred: slices.Clone(doc.Primary.Deferred), Dropped: dropped, Stale: slices.Clone(doc.Overlays.Stale), - Parked: slices.Clone(doc.Primary.Parked), Partial: slices.Clone(doc.Overlays.Partial), Changed: slices.Clone(doc.Overlays.Changed)} + Parked: carryAnnounced(doc.Primary.Parked, r.carried), Partial: slices.Clone(doc.Overlays.Partial), Changed: slices.Clone(doc.Overlays.Changed)} } // branchDate is the date a batch branch was opened under. diff --git a/cmd/backport-batch/body.go b/cmd/backport-batch/body.go index d64bfb5d5..dbb9d6050 100644 --- a/cmd/backport-batch/body.go +++ b/cmd/backport-batch/body.go @@ -264,6 +264,9 @@ type report struct { // does only part of what they do on main. parked []*verdict partial []*verdict + // carried is the line's record this run found, whose announcements the + // parks that did not change keep. + carried *lineRecord } // ruleName is how output names what kept an original out: the policy's diff --git a/cmd/backport-batch/fixture_test.go b/cmd/backport-batch/fixture_test.go index 545e4c6c6..ac1307c46 100644 --- a/cmd/backport-batch/fixture_test.go +++ b/cmd/backport-batch/fixture_test.go @@ -458,6 +458,22 @@ func (g *fakeGH) editPull(number int, title, bodyFile string) error { return nil } +func (g *fakeGH) editBody(number int, bodyFile string) error { + if err := g.write(number, "a body edit"); err != nil { + return err + } + if err := g.inject("editBody"); err != nil { + return err + } + if err := g.readBody(bodyFile); err != nil { + return err + } + if pr := g.find(number); pr != nil { + pr.Body = g.bodies[len(g.bodies)-1] + } + return nil +} + func (g *fakeGH) closePull(number int) error { if err := g.write(number, "a close"); err != nil { return err diff --git a/cmd/backport-batch/github.go b/cmd/backport-batch/github.go index 072673a77..7161ef15b 100644 --- a/cmd/backport-batch/github.go +++ b/cmd/backport-batch/github.go @@ -127,6 +127,8 @@ type github interface { // createPull opens the draft and returns its number and URL. createPull(base, head, title, bodyFile string) (int, string, error) editPull(number int, title, bodyFile string) error + // editBody replaces the body of pull request number and nothing else. + editBody(number int, bodyFile string) error closePull(number int) error comments(number int) ([]issueComment, error) createComment(number int, bodyFile string) error @@ -260,6 +262,11 @@ func (ghCLI) editPull(number int, title, bodyFile string) error { return err } +func (ghCLI) editBody(number int, bodyFile string) error { + _, err := capture("gh", "pr", "edit", strconv.Itoa(number), "--body-file", bodyFile) + return err +} + func (ghCLI) closePull(number int) error { _, err := capture("gh", "pr", "close", strconv.Itoa(number)) return err diff --git a/cmd/backport-batch/notify.go b/cmd/backport-batch/notify.go index 34f0d8afb..fe301a2ff 100644 --- a/cmd/backport-batch/notify.go +++ b/cmd/backport-batch/notify.go @@ -88,8 +88,14 @@ func (a *app) notify(args []string) int { } deferred := map[int]deferredEntry{} var partial []int + var parks []parkedEntry if src.open { deferred, partial = src.rec.deferred(), src.rec.Partial + for _, p := range src.rec.Parked { + if !p.Announced { + parks = append(parks, p) + } + } } // An original that waits on others is listed once, after what decides // its wait, where that one is listed. @@ -163,10 +169,74 @@ func (a *app) notify(args []string) int { } text = append(text, b.String()) } + if len(parks) > 0 { + fmt.Fprintf(a.stdout, "%s: %d parked originals on #%d, named once\n", l.branch(), len(parks), pr.Number) + var b strings.Builder + fmt.Fprintf(&b, "These cannot land on this line whole, so the batch parked them; each is named here once. A maintainer ports what applies as a pull request of its own, as [fixes that exist only on a release line](%s#fixes-that-exist-only-on-a-release-line) describes, or drops the original with `backport-batch run --drop`:\n", policyURL) + for _, p := range parks { + fmt.Fprintf(&b, "\n- %s: %s", prLink(p.Original), parkReason(p)) + } + text = append(text, b.String()) + } if !*push { return 0 } - return a.comment(pr, *mention, today, text) + if code := a.comment(pr, *mention, today, text); code != 0 || len(parks) == 0 { + return code + } + if err := a.announced(pr, src.rec, parks); err != nil { + return a.fail(fmt.Errorf("%w; the next notify names them again", err)) + } + return 0 +} + +// announced marks parks as named in the record of the open batch pull +// request, which the comment just named: only while it is open and its body +// still carries the record notify read, so a run that rewrote the body +// meanwhile loses nothing; the next notify names those parks again. +func (a *app) announced(pr *pullRequest, rec *lineRecord, parks []parkedEntry) error { + marked := *rec + marked.Parked = slices.Clone(rec.Parked) + for i, p := range marked.Parked { + if slices.ContainsFunc(parks, func(q parkedEntry) bool { return q.Original == p.Original && parkEvent(q) == parkEvent(p) }) { + marked.Parked[i].Announced = true + } + } + now, err := a.gh.pull(pr.Number) + if err != nil { + return err + } + if st := prState(&now); st != "open" { + return fmt.Errorf("batch pull request #%d is %s now; nothing is written to it", pr.Number, st) + } + lines, err := fencedLines(now.Body, deferredFence) + if err != nil || len(lines) != 1 { + return fmt.Errorf("batch pull request #%d no longer carries the record notify read", pr.Number) + } + read, err := parseRecord(now.Body, notifyLineOf(rec)) + if err != nil || recordBlock(*read) != recordBlock(*rec) { + return fmt.Errorf("batch pull request #%d no longer carries the record notify read", pr.Number) + } + // GitHub keeps the CRLF line ends a body edited on the web has; the + // record is found, and the body written back, with LF ones. + body := strings.ReplaceAll(now.Body, "\r\n", "\n") + block := deferredFence + "\n" + lines[0] + "\n```" + if strings.Count(body, block) != 1 { + return fmt.Errorf("batch pull request #%d carries the record notify read, but not as one block it can mark", pr.Number) + } + body = strings.Replace(body, block, recordBlock(marked), 1) + name, err := writeTemp("backport-batch-body-*.md", body) + if err != nil { + return err + } + defer os.Remove(name) + return a.gh.editBody(pr.Number, name) +} + +// notifyLineOf is the line a record is for, which it was read against. +func notifyLineOf(rec *lineRecord) line { + l, _ := parseLine(rec.Line) + return l } // placement says under which list notify names original n of the record: diff --git a/cmd/backport-batch/notify_test.go b/cmd/backport-batch/notify_test.go index 4bfb650e9..7f870baee 100644 --- a/cmd/backport-batch/notify_test.go +++ b/cmd/backport-batch/notify_test.go @@ -18,6 +18,7 @@ package main import ( "bytes" + "errors" "slices" "strings" "testing" @@ -334,3 +335,162 @@ func TestNotifyWritesNothingOnABatchClosedMeanwhile(t *testing.T) { }) } } + +// parkedBody is an open batch body whose version 3 record parks p. +func parkedBody(p ...parkedEntry) string { + sha := strings.Repeat("a", 40) + rec := lineRecord{Version: recordVersion, Line: "release-1.1", Report: strings.Repeat("b", 64), + Built: &builtFrom{Main: sha, Line: sha, Policy: sha, Batch: sha, At: "2026-10-26T06:00:00Z", Options: runOptions{Mode: modeContinue}}, + Deferred: []deferredEntry{}, Dropped: []droppedEntry{}, Stale: []int{}, Parked: p} + return "batch\n\n" + recordBlock(rec) + "\n" +} + +// notifyAt runs notify at a time of day, with gh faked. +func notifyAt(gh *fakeGH, at time.Time, args ...string) (int, string, string) { + var stdout, stderr bytes.Buffer + a := &app{gh: gh, stdout: &stdout, stderr: &stderr, now: func() time.Time { return at }} + code := a.main(append([]string{"notify", "--line", "release-1.1"}, args...)) + return code, stdout.String(), stderr.String() +} + +// A park is announced once on the open batch: notify names it in its +// comment and marks it announced in the record; a later notify is silent +// about it. A mark that fails to land repeats the announcement once, the +// next day. With no batch open nothing is announced, and the first notify of +// the next batch does it. +func TestNotifyAnnouncesAParkOnce(t *testing.T) { + park := parkedEntry{Original: 5, Merge: strings.Repeat("c", 40), Kind: parkDependency, Needs: []int{3}} + parked := func(gh *fakeGH) []parkedEntry { + t.Helper() + rec, err := parseRecord(gh.prs[0].Body, notifyLine) + if err != nil { + t.Fatal(err) + } + return rec.Parked + } + gh := &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + code, stdout, stderr := notifyRun(gh, "--push") + if code != 0 || !strings.Contains(stdout, "release-1.1: 1 parked originals on #90, named once") || len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "- [#5](https://github.com/cozystack/cozystack/pull/5): builds on [#3](") { + t.Fatalf("notify = %d, want #5 announced:\n%s\n%s\n%v", code, stdout, stderr, gh.issueComments) + } + if p := parked(gh); len(p) != 1 || !p[0].Announced { + t.Errorf("the record parks %+v, want #5 marked announced", p) + } + if code, stdout, _ = notifyRun(gh, "--push"); code != 0 || strings.Contains(stdout, "parked") || strings.Contains(gh.issueComments[0].Body, "[#5]") { + t.Errorf("notify again = %d, want nothing about #5:\n%s\n%s", code, stdout, gh.issueComments[0].Body) + } + + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}, fail: map[string]error{"editBody": errors.New("HTTP 502")}} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || len(gh.issueComments) != 1 || parked(gh)[0].Announced { + t.Fatalf("notify with a failing mark = %d, want 2 after the comment, the record unmarked", code) + } + code, _, stderr = notifyAt(gh, time.Date(2026, 10, 27, 0, 5, 0, 0, time.UTC), "--push") + if code != 0 || !strings.Contains(gh.issueComments[0].Body, "[#5](") || !parked(gh)[0].Announced { + t.Errorf("notify the next day = %d, want #5 named once more and marked:\n%s\n%s", code, stderr, gh.issueComments[0].Body) + } + + // Closed, or rewritten by a run, between the comment and the mark: the + // mark is not written, and the next notify names #5 again. + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + gh.after = map[string]func(){"createComment": func() { gh.prs[0].State = "closed" }} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || len(gh.closedWrites) != 0 || parked(gh)[0].Announced { + t.Errorf("notify on a batch closed meanwhile = %d, writes %q; want 2 and no mark", code, gh.closedWrites) + } + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", parkedBody(park))}} + rewritten := parkedBody(park, parkedEntry{Original: 6, Merge: strings.Repeat("d", 40), Kind: parkPartial, Files: []string{"x"}}) + gh.after = map[string]func(){"createComment": func() { gh.prs[0].Body = rewritten }} + if code, _, _ := notifyRun(gh, "--push"); code != 2 || gh.prs[0].Body != rewritten || len(gh.bodies) != 0 { + t.Errorf("notify on a body rewritten meanwhile = %d; want 2 and the new body left as it is", code) + } + + // A body someone edited on the web comes back with CRLF line ends; the + // mark lands all the same. + gh = &fakeGH{prs: []pullRequest{notifyPR("20261026", strings.ReplaceAll(parkedBody(park), "\n", "\r\n"))}} + if code, _, stderr := notifyRun(gh, "--push"); code != 0 || !parked(gh)[0].Announced { + t.Errorf("notify on a CRLF body = %d, want #5 marked:\n%s", code, stderr) + } + + closed := notifyPR("20261026", parkedBody(park)) + at := "2026-10-26T12:00:00Z" + closed.State, closed.MergedAt = "closed", &at + gh = &fakeGH{prs: []pullRequest{closed}} + if code, stdout, _ := notifyRun(gh, "--push"); code != 0 || stdout != "" || len(gh.issueComments) != 0 || len(gh.bodies) != 0 { + t.Errorf("notify with no batch open = %d, %q, %d comments, %d edits; want nothing", code, stdout, len(gh.issueComments), len(gh.bodies)) + } + next := notifyPR("20261027", parkedBody(park)) + next.Number = 91 + gh.prs = append(gh.prs, next) + if code, _, _ := notifyRun(gh, "--push"); code != 0 || len(gh.issueComments) != 1 || !strings.Contains(gh.issueComments[0].Body, "[#5](") { + t.Errorf("the next batch's notify = %d, want #5 announced on it: %v", code, gh.issueComments) + } +} + +// A park carries its announcement into the next record only while it is +// the same park: the same merge, kind, prerequisites, commits and files. +func TestParkEventIdentity(t *testing.T) { + a, b := strings.Repeat("a", 40), strings.Repeat("b", 40) + was := parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}, Announced: true} + carrier := &lineRecord{Parked: []parkedEntry{was}} + for name, tc := range map[string]struct { + now parkedEntry + announced bool + }{ + "the same park": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}}, true}, + "another unnumbered commit": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{b}}, false}, + "another kind": {parkedEntry{Original: 5, Merge: a, Kind: parkPartial, Files: []string{"x"}}, false}, + "another prerequisite": {parkedEntry{Original: 5, Merge: a, Kind: parkDependency, Commits: []string{a}, Needs: []int{3}}, false}, + "another merge": {parkedEntry{Original: 5, Merge: b, Kind: parkDependency, Commits: []string{a}}, false}, + "another original": {parkedEntry{Original: 6, Merge: a, Kind: parkDependency, Commits: []string{a}}, false}, + } { + got := carryAnnounced([]parkedEntry{tc.now}, carrier) + if got[0].Announced != tc.announced { + t.Errorf("%s: announced = %v, want %v", name, got[0].Announced, tc.announced) + } + } + if got := carryAnnounced([]parkedEntry{was}, nil); got[0].Announced { + t.Error("a park with no carrier came out announced") + } + unannounced := was + unannounced.Announced = false + if got := carryAnnounced([]parkedEntry{was}, &lineRecord{Parked: []parkedEntry{unannounced}}); got[0].Announced { + t.Error("a park the carrier holds unannounced came out announced") + } +} + +// A park keeps its announcement through a continuation and into the next +// batch, so notify does not name it again. +func TestRolloverKeepsTheAnnouncement(t *testing.T) { + f := featWorld(t) + gh := &fakeGH{} + openBatch(t, f, gh) + if code, _, stderr := f.exec(gh, "notify", "--line", "release-1.1", "--push"); code != 0 || !strings.Contains(notifyComment(gh), "[#5](") { + t.Fatalf("notify = %d, want #5 announced:\n%s", code, stderr) + } + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 || !strings.Contains(stdout, `"needs":[3],"announced":true`) { + t.Fatalf("run --continue = %d, want #5's park still announced:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + reopen(f, gh) + mergeBatch(f, gh) + f.pr(8, "fix(app): eight", "", map[string]string{"app/eight.txt": "8\n"}) + gh.nextPR = 91 + f.now = f.now.Add(24 * time.Hour) + if code, stdout, stderr := f.exec(gh, "run", "--line", "release-1.1", "--continue", "--push"); code != 0 || !strings.Contains(stdout, `"needs":[3],"announced":true`) { + t.Fatalf("the next batch's run = %d, want #5's park announced from the merged batch:\n%s\n%s", code, stdout, stderr) + } + f.git("switch", "-q", "main") + gh.issueComments = nil + if code, stdout, _ := f.exec(gh, "notify", "--line", "release-1.1", "--push"); code != 0 || strings.Contains(stdout, "parked") || strings.Contains(notifyComment(gh), "[#5](") { + t.Errorf("notify on the next batch = %d, want nothing about #5:\n%s\n%s", code, stdout, notifyComment(gh)) + } +} + +// notifyComment is the notify comment on the fake, or "". +func notifyComment(gh *fakeGH) string { + for _, c := range slices.Backward(gh.issueComments) { + if strings.HasPrefix(c.Body, notifyMarker) { + return c.Body + } + } + return "" +} diff --git a/cmd/backport-batch/record.go b/cmd/backport-batch/record.go index 0077b5302..37dd7946a 100644 --- a/cmd/backport-batch/record.go +++ b/cmd/backport-batch/record.go @@ -88,6 +88,39 @@ type parkedEntry struct { Announced bool `json:"announced,omitempty"` } +// parkEvent is what makes a park the same park from one record to the +// next: its merge, kind, prerequisites, commits and files. A park whose +// event changed is news, and notify names it again. +func parkEvent(p parkedEntry) string { + data, _ := json.Marshal(struct { + Merge string `json:"merge"` + Kind string `json:"kind"` + Needs []int `json:"needs"` + Commits []string `json:"commits"` + Files []string `json:"files"` + }{p.Merge, p.Kind, p.Needs, p.Commits, p.Files}) + return string(data) +} + +// carryAnnounced marks each park announced that the carrier's record holds +// announced as the same event, a closed carrier's too, so a new batch does +// not name again what an earlier one named. +func carryAnnounced(parked []parkedEntry, carrier *lineRecord) []parkedEntry { + out := slices.Clone(parked) + for i := range out { + out[i].Announced = false + } + if carrier == nil { + return out + } + for i := range out { + out[i].Announced = slices.ContainsFunc(carrier.Parked, func(c parkedEntry) bool { + return c.Announced && c.Original == out[i].Original && parkEvent(c) == parkEvent(out[i]) + }) + } + return out +} + // validate holds a parked entry to what a run writes. Its errors read after // "the block" or "report ". func (p parkedEntry) validate() error { diff --git a/docs/release/backports.md b/docs/release/backports.md index 71fa467eb..ad7424c40 100644 --- a/docs/release/backports.md +++ b/docs/release/backports.md @@ -117,7 +117,7 @@ That picks it onto the batch in the batch's shape, after what the batch carries, ### The scheduled run -[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. +[`backport-batch.yaml`](../../.github/workflows/backport-batch.yaml) runs [`hack/backport-batch-scheduled.sh`](../../hack/backport-batch-scheduled.sh) on Monday and Thursday. For each line the batch runs for, newest first, it continues the open batch pull request or opens one, runs `check`, and then `backport-batch notify`, which reports what waits on the batch. A batch open for three days or more, so one the previous run already saw, gets a comment on its pull request saying so and a line in the run's summary. The comment is one per pull request, updated in place on every run. It also lists the deferred conflicts, which a maintainer resolves as [resolving a conflict](./backport-tasks/resolve-conflict.md) describes, and the originals that wait because they change workflows, and names each parked original once: a park named on one batch is not named again on the next while it stays the same park, and with no batch open nothing is named until the next batch opens. A failed batch CI waits for a maintainer too, as [attributing a failed batch CI](./backport-tasks/failed-ci.md) describes; the comment does not list it, and it shows on the batch pull request's checks. `hack/backport-batch-scheduled.sh --dry-run` does the same in the current worktree and changes nothing on GitHub. Its `check` sees the line without the batch the dry run just built, so it can exit 1 where the real run's check passes. The batch runs as the cozystack-ci GitHub App, which the release workflows already use, and needs no secret of its own. The job mints a token from `COZYSTACK_CI_APP_ID` and `COZYSTACK_CI_PRIVATE_KEY` for this repository alone, with Contents and Pull requests write and nothing more, and fails when either secret is missing. Its pushes start CI on the batch pull request, which a push with the job's own `GITHUB_TOKEN` would not. Its commits carry the App's bot, `cozystack-ci[bot]`, whose batch pull request and comment `backport-batch` trusts: `BACKPORT_BATCH_BOT`, which the job sets from the token, names it, and a run without the variable trusts `cozystack-ci[bot]` too. The App's installation has to grant this repository Contents and Pull requests read and write, which an organization admin sees among the organization's installed GitHub Apps. It has no Workflows permission, so a change to `.github/workflows/` waits for a maintainer, as [changes to workflows](#changes-to-workflows) describes.