The generator writes each entry from the opening of the PR text, one PR
at a time, and nothing in its instructions asks it to check a statement
against the tree it ships. The v1.7.0 draft had every author and PR
number right and still carried 22 wrong or missing statements, several
in the breaking-changes section, where a wrong instruction breaks an
upgrade: a companion-repository fix credited to a tag that does not
contain it, worker pools said to survive a migration that never pins
their template, migrations said to run on a variant that disables them.
The checks that would have caught them live in docs/release.md, which
the model is never pointed at, and the detailed rules it does read are
about bookkeeping. Add the semantic half: read the whole release note,
describe the final state at the upper-bound tag, compare with the
previous release, take identifiers, migration behaviour and variant
scope from the code, collect required actions beyond `!` commits, pin
companion changes to a tag, and run the pre-publish checks and the
validator on the written file before exiting.
The pre-publish checks name the file after $NEW, which in the promotion
configuration is the rc tag, so they are told to read the stable file
name; otherwise the reference and link checks read a file that does
not exist and report nothing. A minor release's versioned docs are not
live until it publishes, so those links are checked against the
website tree rather than expected to answer 200.
Assisted-by: LLM
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Three review nits, all verified against the tree, plus a tighter scope than the
first attempt at them.
The behaviour suite asserted "promote-rc does not trim the rc tag" against the
whole file. promote-rc.yaml has no .trim() today, so it passed, but any unrelated
.trim() added anywhere in the workflow would have failed a test about tag policy.
Scoping it to the parse JOB was not enough either: that job is ~180 lines, most
of them the e2e gate's github-script, whose string handling has nothing to do
with tag policy. Both greps are now scoped to the parse STEP, and run through a
code_lines filter copied from the contract suite so a YAML comment mentioning
.trim() cannot fail them.
Mutation-checked in both directions: a .trim() injected into the parse step fails
the test, one injected into the e2e-gate step of the same job does not.
- The Step 2 checklist entry still said "must be main" while Step 2 itself now
lists three valid starting points (main, detached at the release tag, detached
at the rc tag being promoted). The checklist is what an agent ticks off, so it
was the half that mattered; it now points at Step 2 rather than re-encoding the
count, so a fourth configuration cannot desync it. The heading said "Checking
current branch", which two of the three states do not have.
- hack/testdata/needs-semantics.yaml named release-changelog-contract.bats as its
consumer; the only consumer is release-changelog-behaviour.bats.
50/50 tests green across the three suites (11 + 24 + 15).
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Two more review passes (Claude + Codex) on the previous commit. Codex
found the one that would have broken the very next promotion.
The agent reads its own instructions from the working tree, which is the
rc tag. Every rc cut before this change carries the old §2 of
docs/agents/changelog.md — two valid configurations, "stop" on anything
else — and a detached rc-tag checkout is exactly that "anything else". So
promoting v1.6.0-rc.4 would have presented a compliant agent with
instructions telling it to refuse, silently, producing no changelog on
every promotion of an existing rc. Verified against the tag. The same
applies to hack/validate-changelog.sh, absent from those trees entirely,
which would have failed the Verify step with "command not found" and, in
open-pr, taken the promote PR down with it. Both are now overlaid from the
dispatch ref, best-effort, in both jobs.
The structural validation added last round rejected every PATCH
changelog. It asserted a `# Cozystack vX.Y.Z` header and a 20-line floor;
patch releases use `# vX.Y.Z (date)` and v1.5.1 is a complete 19-line
release. Good AI output would have been discarded and the promote PR
mislabelled "generation failed". Measured across the tree: the header form
holds for 23 of 115 changelogs, so it was never an invariant.
The predicates now live in hack/validate-changelog.sh, tested directly
against real changelogs rather than through YAML greps — the reason the
defect shipped is that the suite pinned finalize's guard but nothing on
the promote-side validator, and stayed green when the validator was
mutated away. It accepts both header conventions, has no line floor, and
checks the version in the header, the release-link comment and the compare
link, so a changelog generated for the wrong version is caught. All 16
current-era changelogs pass; whitespace, empty, truncated and
wrong-version inputs are rejected.
Also fixed:
- A fatal artifact download in open-pr would abort the job before the PR
step — the same wedge as a skipped job, one step lower. Download is now
best-effort and the commit and PR body key off what is on disk, not an
upstream job output that says nothing about whether the transfer worked.
- A pre-existing changelog (hand-written, or preserved across the
force-push) reached the PR without ever being validated, was advertised
with ✅ and published verbatim. It now goes through the same validator,
as does the tags.yaml backstop.
- `gh pr view <branch>` matches closed PRs, so re-dispatching after
closing an abandoned promote PR edited the dead PR and opened nothing.
Branch on state explicitly.
- `gh pr edit --body` unconditionally overwrote the PR body, destroying
the upstream breaking-change notes docs/release.md instructs reviewers
to paste there. Refresh only while the body is still the generated one.
- open-pr consumed raw dispatch input while its siblings used the
validated parse; the changelog job's bash parse did not trim, so an
input with a stray space passed promote and failed there. Both fixed.
- The prompt and §2 said "trailing release-link comment"; it is the first
thing in every changelog.
Tests grow to 16, each mutation-checked: dropping the overlay, making the
download fatal, reverting the existing-changelog check to `-s`, and
removing the backstop validator each fail the suite.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Dual-model review (Claude + Codex) of the previous two commits found seven
substantive defects, two of them destructive. Both models independently
reached three of them; the split is noted per item.
Descope the website half (Claude, verified by reproduction). The docs job
was moved to promote time, but the website Makefile pins the fetch ref
with `override BRANCH := $(RELEASE_TAG)`, which defeats the command-line
BRANCH assignment. `make -p` resolves BRANCH to v1.6.0 — a tag that
cannot exist at promote time, since promote's own validate step hard-fails
if it does. update_apps.sh logs the failed fetch and exits 0 AFTER
truncating each destination file, so every app doc would become a stub,
`changed=true` would open the PR, and the new auto-merge would land it
unreviewed. Not fixable from this repo. update-website-docs returns to
tags.yaml where RELEASE_TAG resolves to a real tag, and the finalize
auto-merge is dropped. Opening it earlier needs a cozystack/website change
first and is left as a follow-up. This also removes the unordered-sibling
race Codex found, where the release PR could merge before the docs job
opened its PR, leaving finalize nothing to merge and no retry.
Fix the non-blocking guarantee (both models). Only the AI step carried
continue-on-error; the changelog JOB did not, so a flaky `npm i -g
@github/copilot` failed it and `needs` skipped open-pr entirely — staging
branch pushed, release drafted, no promote PR. Gate open-pr on promote
alone.
Generate from a checkout the contract permits (both models). The job
checked out the rc staging branch, which docs/agents/changelog.md §2
rejects — a compliant agent stops and writes nothing, so the normal path
produced no changelog at all. Check out the rc tag detached and document
that configuration in §2. The checkout ref now comes from the validated
parse output rather than raw dispatch input.
Port, don't regenerate, in the backstop (both models). For a patch release
promoted onto release-X.Y the changelog never reaches main, so the
backstop fired and ran a second AI pass; once its PR merged,
update-releasenotes.yaml overwrote the published body with unreviewed
text. Detect the file at the tag and port it verbatim; the AI runs only
when it is absent from both main and the tag.
Survive re-dispatch (Codex). promote force-pushes the staging branch,
erasing a hand-written changelog — the documented recovery when generation
fails. Preserve an existing one across the rebuild, and stop open-pr
overwriting it with generated output.
Validate structurally, not by size (Codex). A truncated or whitespace-only
AI run passed `[ -s ]`, was committed, advertised as valid in the PR body,
and published verbatim as release notes. Assert the header, the compare
link and a plausible line count; finalize additionally refuses
whitespace-only content rather than blanking the notes.
Add hack/release-changelog-contract.bats — nine tests pinning each link in
the chain, mutation-checked: removing open-pr's if:, breaking finalize's
body read, and restoring paths-ignore each fail the suite.
Update docs/release.md (~10 sites) and docs/agents/changelog.md §2, which
described the old flow throughout.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- tags.yaml: the check_changelog step now looks at origin/main rather than
the working tree. After Round 2 we check out the release tag, which never
has the changelog file (the PR opened by this job merges to main after
the tag is cut), so the previous `[ -f ]` test always returned false and
the gate was effectively dead on reruns.
- changelog.md: Step 6's `RELEASE_END=$(git log -1 --format=%ai HEAD)`
(lines 179, 222) was missed in the previous "never HEAD" sweep — now
uses v<new_version>, matching Step 5.
- changelog.md: Step 2 ("on main branch") was incompatible with the CI flow
which now runs detached at the release tag. Relaxed it to list both
configurations (interactive on main; CI detached at v<version>), while
the "Scope and boundaries" section continues to forbid the agent from
switching branches.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- tags.yaml: drop "(or HEAD)" from the changelog prompt; the doc says
"never HEAD" and the workflow now matches.
- docs/agents/changelog.md: exclude the current release file from the
"previous contributors" set used for first-time-contributor detection.
The previous globbing on `docs/changelogs/v*.md` swept up the file
being generated, so `comm -23` would never report any newcomers.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
- Drop the "orphan backport, drop entry" rule: a backport whose original is on
main but hasn't shipped in a stable tag is the first delivery and must stay
in the changelog.
- Add `v` prefix to `<previous_version>`/`<new_version>` placeholders in all
four `git log` invocations for consistency with the rest of the doc.
- Apply the bot/CI account filter to the new-contributors detection (and the
standalone previous-contributors extraction) so bot accounts can no longer
leak into the human Contributors list.
Assisted-By: Claude <noreply@anthropic.com>
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The generate-changelog job in .github/workflows/tags.yaml previously
checked out main and ran the AI agent there. The agent followed
docs/agents/changelog.md, which instructed it to compute the release
range with `git log <previous_version>..HEAD`. For minor releases
(vX.Y.0) cut from main this happens to give the right answer; for
patch releases (vX.Y.Z, Z>0) cut from a release-X.Y branch, HEAD-on-
main is a strict superset of the release tag and contains commits
that were merged to main both before and after the tag.
The v1.3.1 changelog generated by this workflow (#2480) demonstrated
the failure mode: it included 8 PRs merged to main but never shipped
in v1.3.1, 6 backport PRs that landed on release-1.3 *after* v1.3.1
was tagged, both originals and their backports as separate entries,
a hallucinated 2024 PR, and the cozystack-ci bot in the contributors
list. The corrected v1.3.1 changelog contains the single fix that
v1.3.1 actually ships.
This PR fixes the root cause and tightens the agent guardrails:
* tags.yaml: check out the release tag commit (`ref: ${{ steps.tag.
outputs.tag }}`) instead of `main`, so HEAD == the release commit
and `git log v<prev>..HEAD` corresponds to the actual release. The
Create-changelog-branch step still creates the PR branch from
origin/main, so PRs continue to merge cleanly into main.
* tags.yaml: update the AI prompt to state explicitly that HEAD is
the release commit and that the upper bound for the release range
is the new tag, not main.
* docs/agents/changelog.md: replace `..HEAD` with `..v<new_version>`
in every example so the instruction is unambiguous regardless of
which branch is checked out.
* docs/agents/changelog.md: add a hard rule that backport PRs MUST
be merged into the original entry (single line, two PR numbers),
never listed as a second entry. Document the edge case where the
original PR is outside the range — drop the entry.
* docs/agents/changelog.md: forbid using the brief description
verbatim as the detailed description, and forbid inventing entries
for PRs that are not in the release range.
* docs/agents/changelog.md: filter bot/CI accounts (`app/*`,
`*[bot]`, cozystack-ci, github-actions, dependabot, renovate) out
of the human Contributors list. Bot attribution on individual
entries is still valid.
The corresponding correction for the v1.3.1 changelog itself is in
PR #2480 on the changelog-v1.3.1 branch.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
Review feedback on PR #2460: the previous "do not write to refs, HEAD,
or remotes" wording contradicted the explicit allowance of `git fetch`
(which updates remote-tracking refs) and the mandatory cross-repo
checks in Step 6, which `cd` into `_repos/<repo>` and run
`git checkout`, `git pull`, etc.
Sharpen the scope paragraph:
- The git-write ban is now explicitly scoped to the cozystack working
tree — it bans writing to local branches, tags, or HEAD in that
repo, not "refs/HEAD/remotes" globally.
- `git fetch` is called out as expected.
- Local git operations inside disposable `_repos/` clones
(`git checkout`, `git pull`, etc.) are explicitly allowed, with
the remaining rules (no push, no PR creation, no API writes)
applying to any repository.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>
The v1.3.0 release pipeline broke because Copilot, invoked by
.github/workflows/tags.yaml with --allow-all-tools, committed the
generated changelog onto HEAD of main on its own. The workflow's
next step — `git checkout -b ... origin/main` — then wiped the file,
and `git add` failed with a pathspec error.
The root cause is in this document. The checklist ends with "Save
the changelog", which an agent with broad tool access can reasonably
interpret as "also commit it, push it, and open a PR". There was no
explicit boundary.
Add a "Scope and boundaries" section at the top and an explicit
"then exit" at the end of Step 9:
- The single deliverable is docs/changelogs/v<version>.md.
- Forbidden by default: git commit / push / checkout (to switch
branches) / branch / tag / reset / merge / rebase; PR creation;
GitHub API writes (POST/PATCH/DELETE); modifying any file other
than the changelog.
- Read-only analysis (git log/show/fetch/diff, gh pr view, gh api
GET) remains expected.
- Auxiliary repo clones under _repos/ remain allowed for cross-repo
analysis per Step 6.
- Scoped "unless the caller explicitly instructs otherwise" so
interactive use with an IDE remains flexible.
With the rules in the doc, CI and interactive callers share the
same boundary; the workflow can invoke the doc with a one-line
prompt instead of re-stating the constraints every time.
Signed-off-by: Myasnikov Daniil <myasnikovdaniil2001@gmail.com>