diff --git a/.github/workflows/cut-prerelease.yaml b/.github/workflows/cut-prerelease.yaml new file mode 100644 index 000000000..89d487511 --- /dev/null +++ b/.github/workflows/cut-prerelease.yaml @@ -0,0 +1,222 @@ +name: Cut Pre-release Tag + +# Cut a pre-release tag (vX.Y.Z-alpha.N / -beta.N / -rc.N) as the CozyStack CI +# app, so that no human ever pushes a v* tag by hand. +# +# This is the single entry point for creating a pre-release. Stable vX.Y.Z tags +# are never cut here (or by hand) — they are created write-once by +# pull-requests-release.yaml finalize at a promote PR's merge commit. See +# docs/release.md and #2677. +# +# Dispatch from the branch the tag belongs to: `main` for a new minor's +# pre-release (vX.Y.0-rc.N), or `release-X.Y` for a patch line (vX.Y.Z-rc.N). +# The workflow tags that branch's HEAD. +# +# Why a workflow instead of a manual `git push`: +# 1. The tag is pushed with the CI app token. A tag pushed with the default +# GITHUB_TOKEN would NOT trigger tags.yaml (GitHub suppresses workflow +# recursion from GITHUB_TOKEN); the app token is a distinct identity, so +# tags.yaml fires exactly as it does for a maintainer's manual push. +# 2. It pushes `HEAD:refs/tags/` from the checked-out branch tip, so the +# push event's base_ref is populated — tags.yaml's "Get base branch" step +# needs it to detect main vs release-X.Y. +# 3. It makes CI the sole creator of pre-release tags. Repo rulesets can +# already restrict *stable* tag creation to CI (a "Restrict creations" +# ruleset targeting v* while excluding v*-*), but pre-releases were +# hand-pushed, so they could not be CI-locked. With this workflow as their +# only entry point, admins can extend "Restrict creations" (CI app the sole +# bypass) to *all* of v* — pre-release and stable alike. + +on: + workflow_dispatch: + inputs: + tag: + description: "Pre-release tag to cut, e.g. v1.6.0-rc.1 (must be -alpha.N / -beta.N / -rc.N)" + required: true + type: string + +concurrency: + # One global lane: two dispatches racing to push overlapping tags would + # confuse the write-once guard. Pre-release cuts are rare, so serialize. + group: cut-prerelease + cancel-in-progress: false + +# The tag write happens via the app token, not GITHUB_TOKEN, so the job needs +# only read scope on the default token. +permissions: + contents: read + +jobs: + cut: + name: Cut ${{ inputs.tag }} + runs-on: ubuntu-latest + timeout-minutes: 10 + + steps: + # Pure-input validation first — no token minted, no checkout, so a bad tag + # or wrong dispatch branch fails in seconds. + + # Validate the dispatch input against a strict pre-release pattern. A + # stable vX.Y.Z is rejected on purpose: stable tags are cut only by the + # promote flow, at a PR merge commit. + - name: Parse and validate tag + id: parse + env: + TAG: ${{ inputs.tag }} + uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 + with: + script: | + const tag = process.env.TAG.trim(); + const m = tag.match(/^v(\d+)\.(\d+)\.(\d+)-(alpha|beta|rc)\.(\d+)$/); + if (!m) { + core.setFailed( + `tag '${tag}' must match 'vX.Y.Z-(alpha|beta|rc).N'. ` + + `Stable vX.Y.Z tags are cut by the promote flow, not here.` + ); + return; + } + core.setOutput('tag', tag); + core.setOutput('line', `${m[1]}.${m[2]}`); // 1.6 + core.setOutput('patch', m[3]); // 0 + + # Refuse to cut from anything but main or the *matching* release-X.Y line. + # The pushed tag's base_ref (= the dispatch branch) must be one tags.yaml + # accepts, and the branch must actually contain the version being cut. + - name: Validate dispatch branch + env: + REF_NAME: ${{ github.ref_name }} + REF_TYPE: ${{ github.ref_type }} + LINE: ${{ steps.parse.outputs.line }} + PATCH: ${{ steps.parse.outputs.patch }} + run: | + if [ "$REF_TYPE" != "branch" ]; then + echo "::error::Dispatch this workflow from a branch (main or release-X.Y), not a tag or SHA." + exit 1 + fi + case "$REF_NAME" in + main) + # From main only a *new minor's* pre-release (vX.Y.0-*) is valid. + # A patch-line pre-release (Z>0) built from main content would land + # on a wrong-tree release-X.Y.Z-rc.N staging branch — it must be cut + # from its own release-X.Y line instead. + if [ "$PATCH" != "0" ]; then + echo "::error::From 'main' only vX.Y.0-* pre-releases may be cut (got patch .$PATCH). Cut v$LINE.$PATCH-* from 'release-$LINE'." + exit 1 + fi + ;; + release-*) + if [ "release-$LINE" != "$REF_NAME" ]; then + echo "::error::Tag line $LINE does not match dispatch branch '$REF_NAME'. Cut v$LINE.* from 'main' or 'release-$LINE'." + exit 1 + fi + ;; + *) + echo "::error::Dispatch from 'main' or 'release-X.Y' only. Got '$REF_NAME'." + exit 1 + ;; + esac + + - name: Generate GitHub App token + id: app-token + uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1 + with: + app-id: ${{ secrets.COZYSTACK_CI_APP_ID }} + private-key: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }} + owner: cozystack + # Scope the token to this repo only — it never touches other repos. + repositories: ${{ github.event.repository.name }} + + # Cutting a release is privileged. workflow_dispatch already requires write + # access to trigger, so this is defense-in-depth: it makes the requirement + # explicit and auditable, and fails closed if that platform default ever + # changes or the job is reached by another path. Reads permission via the + # app token (fails the step if the API errors — fail closed). + - name: Require write access + uses: actions/github-script@f28e40c7f34bde8b3046d885e986cb6290c5673b # v7 + with: + github-token: ${{ steps.app-token.outputs.token }} + script: | + const { data } = await github.rest.repos.getCollaboratorPermissionLevel({ + owner: context.repo.owner, + repo: context.repo.repo, + username: context.actor, + }); + // getCollaboratorPermissionLevel's `permission` field returns only + // the legacy roles admin/write/read/none (maintain collapses to + // write, triage to read), so admin+write is the whole "can push" set. + const allowed = ['admin', 'write']; + if (!allowed.includes(data.permission)) { + core.setFailed( + `@${context.actor} has '${data.permission}' access — cutting a pre-release tag ` + + `requires write access to ${context.repo.owner}/${context.repo.repo}.` + ); + return; + } + core.info(`@${context.actor}: '${data.permission}' access — authorized.`); + + # Minimal checkout — just enough to configure origin with the app-token + # credential (persist-credentials, default on) so the bare push below is + # app-authenticated, and to resolve HEAD. One ref, shallow, no tags. + - name: Check out the dispatched branch + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 + with: + fetch-depth: 1 + token: ${{ steps.app-token.outputs.token }} + + - name: Cut and push the pre-release tag + env: + TAG: ${{ steps.parse.outputs.tag }} + REF_NAME: ${{ github.ref_name }} + run: | + # Write-once: the tag must not already exist on origin. Distinguish + # "absent" (git ls-remote --exit-code exits 2) from a transport/auth + # failure (any other non-zero) so an infra blip cannot masquerade as + # "free to create". + set +e + git ls-remote --exit-code --tags origin "refs/tags/$TAG" >/dev/null 2>&1 + rc=$? + set -e + if [ "$rc" -eq 0 ]; then + echo "::error::Tag $TAG already exists on origin — pre-release tags are write-once." + exit 1 + elif [ "$rc" -ne 2 ]; then + echo "::error::Could not query origin for $TAG (git ls-remote exit $rc) — refusing to proceed." + exit 1 + fi + + # Stale-tip guard: workflow_dispatch checks out the dispatch-time commit, + # not the branch tip at push time (the gap includes any wait in the + # cut-prerelease concurrency queue). If the branch advanced meanwhile, + # pushing HEAD would tag a stale commit and yield an empty base_ref — + # tags.yaml would then fail, having already burned this write-once name. + # Refuse and ask for a re-run instead. + local_head="$(git rev-parse HEAD)" + # Resolve the remote tip without a pipe masking ls-remote's exit code + # (mirrors the write-once query above): a transport/auth failure must + # not read as "branch not found". + set +e + remote_ls="$(git ls-remote origin "refs/heads/$REF_NAME")" + rc=$? + set -e + if [ "$rc" -ne 0 ]; then + echo "::error::Could not query origin for the tip of '$REF_NAME' (git ls-remote exit $rc) — refusing to proceed." + exit 1 + fi + remote_head="$(printf '%s' "$remote_ls" | cut -f1)" + if [ -z "$remote_head" ]; then + echo "::error::Could not resolve the tip of '$REF_NAME' on origin (no such branch?)." + exit 1 + fi + if [ "$local_head" != "$remote_head" ]; then + echo "::error::'$REF_NAME' advanced since dispatch ($local_head -> $remote_head). Re-run the workflow to tag the current tip." + exit 1 + fi + + # Push HEAD straight to a tag ref: no local tag object, non-forced. The + # branch-tip anchoring makes GitHub populate the push event's base_ref + # (tags.yaml's Get-base-branch step requires it); non-forced makes the + # tag write-once at the git layer. tags.yaml takes it from here (build + + # e2e + draft pre-release + staging branch). + git push origin "HEAD:refs/tags/$TAG" + echo "Pushed $TAG at $local_head from '$REF_NAME'." + echo "tags.yaml will now build, e2e, and draft the pre-release." diff --git a/.github/workflows/nightly.yaml b/.github/workflows/nightly.yaml index ff8865724..eefdc7ec9 100644 --- a/.github/workflows/nightly.yaml +++ b/.github/workflows/nightly.yaml @@ -107,7 +107,7 @@ jobs: echo "Resolved main build: sha=$sha version=$version packages_ref=$packages_ref" - name: Checkout source commit - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: ${{ steps.resolve.outputs.sha }} fetch-depth: 1 @@ -168,7 +168,7 @@ jobs: packages: write steps: - name: Checkout source commit - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: ${{ needs.mirror.outputs.sha }} fetch-depth: 1 @@ -213,7 +213,7 @@ jobs: packages: read steps: - name: Checkout source commit - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: ${{ needs.mirror.outputs.sha }} fetch-depth: 1 diff --git a/.github/workflows/promote-rc.yaml b/.github/workflows/promote-rc.yaml index e8ed2d257..8467149ad 100644 --- a/.github/workflows/promote-rc.yaml +++ b/.github/workflows/promote-rc.yaml @@ -137,7 +137,7 @@ jobs: # The rc's digest-vendored tree (the staging branch the rc build pushed). - name: Checkout rc staging branch - uses: actions/checkout@v4 + uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4 with: ref: ${{ steps.parse.outputs.rc_branch }} fetch-depth: 0 diff --git a/.github/workflows/retention.yaml b/.github/workflows/retention.yaml index 839ec749e..cef8a3100 100644 --- a/.github/workflows/retention.yaml +++ b/.github/workflows/retention.yaml @@ -52,7 +52,7 @@ jobs: steps: - name: Generate GitHub App token id: app-token - uses: actions/create-github-app-token@v1 + uses: actions/create-github-app-token@d72941d797fd3113feb6b93fd0dec494b13a2547 # v1 with: app-id: ${{ secrets.COZYSTACK_CI_APP_ID }} private-key: ${{ secrets.COZYSTACK_CI_PRIVATE_KEY }} diff --git a/.github/workflows/tags.yaml b/.github/workflows/tags.yaml index 5088ff015..a72757041 100644 --- a/.github/workflows/tags.yaml +++ b/.github/workflows/tags.yaml @@ -104,7 +104,7 @@ jobs: - name: Reject hand-pushed stable tags if: steps.check_release.outputs.release_exists == 'false' && steps.tag.outputs.is_stable == 'true' run: | - echo "::error::Stable tag ${GITHUB_REF#refs/tags/} has no pre-existing draft release. Stable tags come from promote-rc.yaml — push a release candidate (vX.Y.Z-rc.N) and promote it; do not push vX.Y.Z by hand." + echo "::error::Stable tag ${GITHUB_REF#refs/tags/} has no pre-existing draft release. Stable tags are finalized by the promote flow at the merge commit — cut a pre-release (vX.Y.Z-rc.N) with the 'Cut Pre-release Tag' workflow and promote it; do not push vX.Y.Z by hand." exit 1 # Detect base branch (main or release-X.Y) the tag was pushed from @@ -116,7 +116,7 @@ jobs: script: | const baseRef = context.payload.base_ref; if (!baseRef) { - core.setFailed(`❌ base_ref is empty. Push the tag via 'git push origin HEAD:refs/tags/'.`); + core.setFailed(`❌ base_ref is empty — the tag was not pushed at a branch tip. Cut pre-release tags with the 'Cut Pre-release Tag' workflow (workflow_dispatch), which pushes the tag correctly.`); return; } const branch = baseRef.replace('refs/heads/', ''); diff --git a/docs/release.md b/docs/release.md index af68eade4..e804cada4 100644 --- a/docs/release.md +++ b/docs/release.md @@ -6,7 +6,7 @@ This document is both the process design (what the release model looks like) and You are about to: -- Tag a release candidate (`vX.Y.0-rc.N`). +- Cut a pre-release (`vX.Y.0-rc.N`, or `vX.Y.0-alpha.N` / `-beta.N`). - Cut a regular release (`vX.Y.0`). - Cut a patch release (`vX.Y.Z` with `Z > 0`). - Triage cherry-picks before a patch release. @@ -25,7 +25,7 @@ There are three types of releases: - **Regular Releases** – Final versions (e.g., `v1.2.0`) that are feature-complete and thoroughly tested. - **Patch Releases** – Bugfix-only updates (e.g., `v1.2.1`) made after a stable release, based on a dedicated release branch. -All three are matched by `tags.yaml`'s regex `^v\d+\.\d+\.\d+(-(alpha|beta|rc)\.\d+)?$`. The tag push is the trigger for the whole pipeline. **Always push tags with `git push origin HEAD:refs/tags/`** so GitHub fills `base_ref` — `tags.yaml`'s `Get base branch` step refuses tags pushed without a base. +All three are matched by `tags.yaml`'s regex `^v\d+\.\d+\.\d+(-(alpha|beta|rc)\.\d+)?$`. The tag push is the trigger for the whole pipeline. **Pre-release tags (`-alpha` / `-beta` / `-rc`) are cut with the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow** (manual dispatch from `main` or `release-X.Y`) — it pushes the tag as the CI app at the dispatch branch's tip, so GitHub fills `base_ref` (`tags.yaml`'s `Get base branch` step refuses tags pushed without a base) and the push triggers `tags.yaml` (a tag pushed with the default `GITHUB_TOKEN` would not). Stable `vX.Y.Z` tags are never pushed by hand at all — they are cut write-once by the promote flow at a PR merge commit. ## Release Candidates @@ -55,7 +55,7 @@ gitGraph A regular release sequence runs as follows: -1. Tag the last good commit on `main` as a release candidate (`v1.2.0-rc.N`) and push it with `git push origin HEAD:refs/tags/v1.2.0-rc.N`. CI builds the rc images, drafts the rc release, and pushes the digest-vendored `release-1.2.0-rc.N` staging branch. +1. Cut a release candidate (`v1.2.0-rc.N`) from the last good commit on `main`: run the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `main`**) with the tag. It tags `main`'s HEAD as the CI app; `tags.yaml` then builds the rc images, drafts the rc release, and pushes the digest-vendored `release-1.2.0-rc.N` staging branch. 2. Validate the rc. New features and fixes can still land on `main` and be picked up by a later rc. 3. Once an rc is green, run the [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) workflow (manual dispatch) with that rc tag. Promotion is **transactional** — it performs no registry mutation at dispatch. It: 1. Pushes the `release-1.2.0` branch: the rc's digest-vendored tree with the rc version substring rewritten to `1.2.0` (the digests are the rc's — unchanged). @@ -142,7 +142,7 @@ gitGraph When all relevant patch commits are cherry-picked, the branch is ready for release. -2. The maintainer tags the `HEAD` commit of branch `release-1.2` as a release candidate (`v1.2.1-rc.N`) and pushes it. CI builds the rc images, drafts the rc release, and pushes the `release-1.2.1-rc.N` staging branch. +2. The maintainer cuts a release candidate (`v1.2.1-rc.N`) via the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow (manual dispatch **from `release-1.2`**), which tags that branch's `HEAD`. CI builds the rc images, drafts the rc release, and pushes the `release-1.2.1-rc.N` staging branch. 3. Validate the rc. 4. Once the rc is green, run [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) (manual dispatch) with that rc tag. It pushes the `release-1.2.1` branch (rc digests, tag string rewritten to `1.2.1`), drafts the `v1.2.1` release with the restamped assets, and opens the promote PR into `release-1.2` (labelled `release` + `full-e2e`). As with a regular release the retag is deferred to the merge — no registry mutation at dispatch. @@ -176,12 +176,13 @@ gitGraph ## What CI does during the release process -The numbered process above is implemented by four workflows. Knowing which job does what makes the failure modes much easier to diagnose. +The numbered process above is implemented by five workflows. Knowing which job does what makes the failure modes much easier to diagnose. -1. [`tags.yaml`](../.github/workflows/tags.yaml) — fires on an rc tag push: runs `prepare-release`, then `generate-changelog`, then `update-website-docs`. -2. [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) — manual dispatch: stages the `release-X.Y.Z` tree (rc digests, tag string rewritten to stable), drafts the stable release, and opens the `release-X.Y.Z` promote PR. No registry mutation — transactional. -3. [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags, publishes, then retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart. -4. [`update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — fires on every push to `main`; syncs `docs/changelogs/v*.md` content into the corresponding GitHub Release body. +1. [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) — manual dispatch (from `main` or `release-X.Y`): the sole entry point for creating a pre-release tag. Validates the `-alpha`/`-beta`/`-rc` tag, then pushes it (write-once, at the branch tip) as the CI app so `tags.yaml` fires. Stable tags are never created here. +2. [`tags.yaml`](../.github/workflows/tags.yaml) — fires on an rc tag push: runs `prepare-release`, then `generate-changelog`, then `update-website-docs`. +3. [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) — manual dispatch: stages the `release-X.Y.Z` tree (rc digests, tag string rewritten to stable), drafts the stable release, and opens the `release-X.Y.Z` promote PR. No registry mutation — transactional. +4. [`pull-requests-release.yaml`](../.github/workflows/pull-requests-release.yaml) — fires when the `release-X.Y.Z` PR merges; finalizes the release: cuts the write-once stable + Go-module tags, publishes, then retags the rc images to stable by digest (`:latest` gated on newest-stable) and publishes the stable chart. +5. [`update-releasenotes.yaml`](../.github/workflows/update-releasenotes.yaml) — fires on every push to `main`; syncs `docs/changelogs/v*.md` content into the corresponding GitHub Release body. ### Phase 1 — `prepare-release` (hard gate) @@ -249,7 +250,7 @@ A stable `vX.Y.Z` is created only by **promoting an existing release-candidate** - [`promote-rc.yaml`](../.github/workflows/promote-rc.yaml) is triggered manually once an rc has gone green. It rewrites the rc version substring in the vendored image tags (the digests stay the rc's) and opens the `release-X.Y.Z` staging PR — no registry mutation. Merging that PR (Phase 5) creates the write-once stable tag at the merge commit, then retags the rc's already-built, e2e-passed image digests to the stable tag **by digest** (no rebuild — see [`hack/promote-retag.sh`](../hack/promote-retag.sh)) and publishes the release. Because the copy source is the immutable rc digest, the stable image is bit-for-bit the rc image that passed e2e — and because the retag runs only after merge, an abandoned promotion never leaves stable-named bytes behind. -So a commit on a supported `release-X.Y` line ships when a maintainer promotes the next rc for that line — not automatically within 24h. Pushing a stable `vX.Y.Z` tag by hand is **not** a supported path: `tags.yaml` fails fast on a stable tag that has no pre-existing draft ("stable tags come from promote-rc.yaml"), and even so the finalize step would refuse to move a pre-existing tag. Release candidates (`vX.Y.Z-rc.N`) are still pushed by hand — [`tags.yaml`](../.github/workflows/tags.yaml) fires on the rc tag push, builds it, and publishes the rc release. +So a commit on a supported `release-X.Y` line ships when a maintainer promotes the next rc for that line — not automatically within 24h. Pushing a stable `vX.Y.Z` tag by hand is **not** a supported path: `tags.yaml` fails fast on a stable tag that has no pre-existing draft ("stable tags come from promote-rc.yaml"), and even so the finalize step would refuse to move a pre-existing tag. Pre-releases (`vX.Y.Z-rc.N`, `-alpha.N`, `-beta.N`) are cut with the [`Cut Pre-release Tag`](../.github/workflows/cut-prerelease.yaml) workflow, which pushes the tag as the CI app; [`tags.yaml`](../.github/workflows/tags.yaml) then fires on that push, builds it, and publishes the pre-release. Cutting via the workflow (rather than a manual `git push`) is what lets repo admins lock `v*` tag creation to the CI app — see the tag-protection note below. ## Nightly builds @@ -553,6 +554,13 @@ Published tags are **write-once** — once a `vX.Y.Z` or rc tag is pushed it is This is the immutable-tag + rc-promotion model from [#2677](https://github.com/cozystack/cozystack/issues/2677): stable `vX.Y.Z` is created only by promoting an existing rc, rc tags are write-once, and `api/apps/v1alpha1/vX.Y.Z` is created only on a stable release. The old nightly `auto-release.yaml` (which delete-recreated auto-bumped patch tags) has been removed. rc and stable tags accrete permanently; the only churning artifacts are GHCR nightlies, which [`retention.yaml`](../.github/workflows/retention.yaml) prunes (see [Nightly builds](#nightly-builds)) — GHCR storage is the cost vector to watch. +### Repo-side enforcement (tag-protection rulesets) + +The guarantees above are enforced in the workflows. Repo admins back them with two GitHub **tag rulesets** targeting `v*`: + +- **Immutability** — *Restrict updates* + *Restrict deletions*, **no bypass**. No actor (not even the CI app) can move or delete a shipped `v*` tag. Safe to enable at any time: nothing in the release flow ever moves or deletes a `v*` tag, it only creates new ones. +- **Creation control** — *Restrict creations* with the **CI app as the sole bypass actor**, plus *Limit branches/tags updated in a single push* = 1. This turns "don't hand-push tags" from convention into hard enforcement: every `v*` tag — pre-release **and** stable — can then be created only by CI. Enable this **only after** [`cut-prerelease.yaml`](../.github/workflows/cut-prerelease.yaml) is in use, since that workflow is what makes CI the sole creator of pre-release tags; enabling it earlier would lock maintainers out of cutting rc/alpha/beta. (The Go-module tag `api/apps/v1alpha1/*` gets an equivalent pair of rulesets.) + ## Splitting a release-blocking bundle PR Sometimes the work that has to land before a release is a 40-commit grab bag (CI stabilization, dependency bumps, chart fixes). Splitting it makes it reviewable and survivable. The strategy that has worked: