Bump release version to 0.9.71

Revert "Prepare user-facing release notes for 0.9.71"
This reverts commit 56b57df407.
2026-09-28 21:44:17 +00:00 · 2026-09-28 14:43:34 -07:00 · 2026-09-28 21:41:59 +00:00 · 2026-09-28 14:41:34 -07:00 · 2026-09-28 19:55:28 +00:00 · 2026-09-28 12:55:06 -07:00
126 changed files with 12107 additions and 8445 deletions
@@ -0,0 +1,42 @@
+# Stable releases require live dev acceptance
+
+The order is **dev CI → dev image publication → automatic Cabbage deployment →
+real capture and browser acceptance → stable publication**. Never
+substitute a published PyPI wheel, healthy container, or local build for this
+acceptance. The release workflow blocks *before* a stable upload unless Cabbage
+has successful evidence from `staging-acceptance.yml` matching all tracked
+source and dependency pins. Only ArchiveBox's own version fields may change
+between dev and stable.
+
+The staging workflow observes the existing image watcher; it cannot deploy or
+restart the application. On Cabbage, install `bin/staging-acceptance-host.sh`
+as root-owned mode 0755 `/usr/local/sbin/archivebox-staging-acceptance`. Its hash
+is checked against the candidate. A dedicated SSH key must use
+`restrict,command="/usr/local/sbin/archivebox-staging-acceptance"` in authorized
+keys. It accepts only `inspect <source-sha>` and `capture <source-sha>`; it cannot
+run caller-provided commands or archive arbitrary URLs.
+
+The `staging` GitHub environment is restricted to dev/main branches. It contains
+`STAGING_SSH_KEY` and `STAGING_KNOWN_HOSTS`, with independently verified host keys.
+Only trusted release events use the private ugNAS runner route. Never expose
+these credentials to PR jobs or copy an operator's existing root key.
+
+Acceptance creates one tagged background crawl per source through the normal
+add/runner path, enabling every snapshot extractor, including TLSNotary. It
+checks installed pins, completed outputs and their files, unexpected/missing
+plugin results, host OOM kills, and real rendered snapshot pages. For this
+release, the explicitly accepted TLSNotary failures and exact missing-Claude-auth
+diagnostics are retained in the report's `limitations` list; they do not disable
+those plugins or excuse missing/truncated successful outputs. Remove the
+TLSNotary exception when its follow-up fix ships. Logs, result inventories and
+screenshots are retained as Actions artifacts. Other failures remain
+failures; do not publish a success deployment record to bypass them. No code
+mounts are allowed. The Cabbage job must pass before its record authorizes a
+stable upload. Queued capture progress is polled; failed tests are not retried.
+
+Browser acceptance follows the default WACZ viewer into the captured document,
+matching its replay base and original URL and checking that its body is visible
+and contains substantial text. An attached viewer iframe can still be
+`about:blank`, so its presence alone cannot prove replay works. Extracted display
+titles and the captured document's `<title>` may legitimately differ; neither
+title equality nor output thumbnails substitute for checking the actual page.
@@ -0,0 +1,3 @@
+self-hosted-runner:
+  labels:
+    - ugnas-linux
@@ -10,4 +10,4 @@ Also review the inline navbar/footer in abx-dl/website/index.html and
 -->
 <footer class="abx-footer" aria-label="ArchiveBox ecosystem">
  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
-  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated
@@ -1,37 +1,78 @@
 #!/usr/bin/env python3
-"""Discover every ArchiveBox test file and expose it as one CI job."""
+"""Discover every test once, batching measured short files."""

 import json
+import os
 from pathlib import Path


 def main() -> None:
-    root = Path.cwd().resolve()
+    root = Path(__file__).resolve().parents[2]
    archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))

    if not archivebox_tests:
        raise SystemExit("No ArchiveBox tests discovered")

-    matrix: list[dict[str, object]] = []
+    # These are observed whole-job seconds, including setup. Unknown/long files,
+    # optional dependencies and hosted-only requirements keep independent jobs.
+    durations = json.loads((root / ".github/test-durations.json").read_text())["seconds"]
+    hosted = {path for path in archivebox_tests if "# ci-runner: hosted" in path.read_text().splitlines()[:5]}
+    requested = os.environ.get("TEST_FILE", "")
+    batches: list[list[str]] = []
+    batch: list[str] = []
+    batch_seconds = 0
    for path in archivebox_tests:
        test_path = path.relative_to(root).as_posix()
+        duration = durations.get(test_path, 60)
+        if duration >= 60 or path in hosted or path.stem == "test_auth_ldap" or test_path == requested:
+            batches.append([test_path])
+            continue
+        if batch and (len(batch) == 8 or batch_seconds + duration > 180):
+            batches.append(batch)
+            batch = []
+            batch_seconds = 0
+        batch.append(test_path)
+        batch_seconds += duration
+    if batch:
+        batches.append(batch)
+
+    matrix: list[dict[str, object]] = []
+    for paths in batches:
        matrix.append(
            {
-                "name": f"main/{path.stem}",
-                "paths": [test_path],
-                "paths_arg": test_path,
-                "extra": "ldap" if test_path.endswith("/test_auth_ldap.py") else "",
-                "count": 1,
+                "name": f"main/{Path(paths[0]).stem}" if len(paths) == 1 else f"batch/{Path(paths[0]).stem}",
+                "paths": paths,
+                "paths_arg": " ".join(paths),
+                "extra": "ldap" if paths == ["archivebox/tests/test_auth_ldap.py"] else "",
+                "count": len(paths),
+                "ugnas": False,
            },
        )

+    # Ordinary Linux tests can use either pool. Only genuine hosted-only
+    # requirements need a file header; filenames never enter runner policy.
+    # Keep a bounded NAS share so hosted capacity continues working in parallel.
+    # This is queued work, not a worker count: several waves keep the capped
+    # runners occupied while hosted jobs run in parallel.
+    nas_job_limit = int(os.environ.get("UGNAS_CI_MAX_JOBS", "3"))
+    # Keep long jobs early in the hosted queue. Giving the longest jobs to the
+    # CPU-capped NAS instead created a tail after hosted work had finished; use
+    # its extra slots for a bounded share of shorter work without filename rules.
+    matrix.sort(key=lambda item: sum(durations.get(path, 60) for path in item["paths"]), reverse=True)
+    eligible = [item for item in reversed(matrix) if not any(root / path in hosted for path in item["paths"])]
+    for item in eligible[:nas_job_limit]:
+        item["ugnas"] = True
+
    discovered_paths = [path for entry in matrix for path in entry["paths"]]
    if len(discovered_paths) != len(set(discovered_paths)):
        raise SystemExit("Tests were not discovered exactly once")
    if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
        raise SystemExit("Discovered test shard coverage does not match test files")

-    print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
+    print(
+        f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs; "
+        f"{sum(item['ugnas'] for item in matrix)} selected for ugNAS, subject to the availability check",
+    )
    print(json.dumps(matrix, separators=(",", ":")))


@@ -0,0 +1,169 @@
+{
+  "source_runs": [
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36302511923",
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36353196552",
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36454992009"
+  ],
+  "measurement": "Whole job seconds, including setup and excluding queue time. Hosted unbatched files use the latest complete successful run; other files retain their individual baseline measurement. Unknown files stay independent.",
+  "seconds": {
+    "archivebox/tests/test_api_v1_auth_browser_session.py": 29,
+    "archivebox/tests/test_api_v1_auth_check_api_token.py": 27,
+    "archivebox/tests/test_api_v1_auth_get_api_token.py": 24,
+    "archivebox/tests/test_api_v1_cli_add.py": 96,
+    "archivebox/tests/test_api_v1_cli_remove.py": 50,
+    "archivebox/tests/test_api_v1_cli_schedule.py": 41,
+    "archivebox/tests/test_api_v1_cli_search.py": 26,
+    "archivebox/tests/test_api_v1_cli_update.py": 74,
+    "archivebox/tests/test_api_v1_cli_workflow_add_search_update_remove.py": 64,
+    "archivebox/tests/test_api_v1_core_any_id.py": 21,
+    "archivebox/tests/test_api_v1_core_archiveresult_archiveresult_id.py": 55,
+    "archivebox/tests/test_api_v1_core_archiveresults.py": 32,
+    "archivebox/tests/test_api_v1_core_snapshot_snapshot_id.py": 199,
+    "archivebox/tests/test_api_v1_core_snapshots_rss.py": 27,
+    "archivebox/tests/test_api_v1_core_snapshots.py": 28,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_rename.py": 23,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_snapshots_jsonl.py": 27,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_urls_txt.py": 28,
+    "archivebox/tests/test_api_v1_core_tag_tag_id.py": 27,
+    "archivebox/tests/test_api_v1_core_tags_add_to_snapshot.py": 27,
+    "archivebox/tests/test_api_v1_core_tags_autocomplete.py": 26,
+    "archivebox/tests/test_api_v1_core_tags_create.py": 21,
+    "archivebox/tests/test_api_v1_core_tags_remove_from_snapshot.py": 25,
+    "archivebox/tests/test_api_v1_core_tags_search.py": 26,
+    "archivebox/tests/test_api_v1_core_tags.py": 25,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_filename.py": 22,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_filename.py": 25,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_subfolder_filename.py": 24,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py": 86,
+    "archivebox/tests/test_api_v1_crawls_crawls.py": 21,
+    "archivebox/tests/test_api_v1_machine_binaries.py": 23,
+    "archivebox/tests/test_api_v1_machine_binary_binary_id.py": 23,
+    "archivebox/tests/test_api_v1_machine_binary_by_name_name.py": 26,
+    "archivebox/tests/test_api_v1_machine_machine_current.py": 23,
+    "archivebox/tests/test_api_v1_machine_machine_machine_id.py": 23,
+    "archivebox/tests/test_api_v1_machine_machines.py": 27,
+    "archivebox/tests/test_api_v1_personas_personas.py": 23,
+    "archivebox/tests/test_api_v1_personas_sync.py": 29,
+    "archivebox/tests/test_api_v1_workflow_core_token_auth_side_effects.py": 36,
+    "archivebox/tests/test_api_v1_workflow_frozen_crawl_config_sources.py": 33,
+    "archivebox/tests/test_archive_result_service.py": 137,
+    "archivebox/tests/test_auth_ldap.py": 44,
+    "archivebox/tests/test_binary_service.py": 83,
+    "archivebox/tests/test_cli_add_1.py": 168,
+    "archivebox/tests/test_cli_add_2.py": 203,
+    "archivebox/tests/test_cli_add_3.py": 215,
+    "archivebox/tests/test_cli_add_4.py": 201,
+    "archivebox/tests/test_cli_archiveresult.py": 199,
+    "archivebox/tests/test_cli_binary.py": 20,
+    "archivebox/tests/test_cli_config.py": 148,
+    "archivebox/tests/test_cli_crawl.py": 177,
+    "archivebox/tests/test_cli_extract_input.py": 155,
+    "archivebox/tests/test_cli_extract.py": 106,
+    "archivebox/tests/test_cli_help.py": 30,
+    "archivebox/tests/test_cli_init.py": 135,
+    "archivebox/tests/test_cli_install.py": 58,
+    "archivebox/tests/test_cli_list.py": 260,
+    "archivebox/tests/test_cli_machine.py": 50,
+    "archivebox/tests/test_cli_manage.py": 48,
+    "archivebox/tests/test_cli_mcp.py": 51,
+    "archivebox/tests/test_cli_oneshot.py": 23,
+    "archivebox/tests/test_cli_persona.py": 82,
+    "archivebox/tests/test_cli_piping.py": 97,
+    "archivebox/tests/test_cli_pluginmap.py": 21,
+    "archivebox/tests/test_cli_process.py": 20,
+    "archivebox/tests/test_cli_remove.py": 162,
+    "archivebox/tests/test_cli_run_1.py": 195,
+    "archivebox/tests/test_cli_run_2.py": 255,
+    "archivebox/tests/test_cli_run_3.py": 167,
+    "archivebox/tests/test_cli_run_4.py": 178,
+    "archivebox/tests/test_cli_schedule.py": 87,
+    "archivebox/tests/test_cli_search.py": 35,
+    "archivebox/tests/test_cli_server.py": 259,
+    "archivebox/tests/test_cli_shell.py": 32,
+    "archivebox/tests/test_cli_snapshot.py": 181,
+    "archivebox/tests/test_cli_status.py": 102,
+    "archivebox/tests/test_cli_tag.py": 21,
+    "archivebox/tests/test_cli_update_reindex_snapshots.py": 163,
+    "archivebox/tests/test_cli_update.py": 142,
+    "archivebox/tests/test_cli_version.py": 143,
+    "archivebox/tests/test_config_DELETE_AFTER.py": 169,
+    "archivebox/tests/test_config_MAX_limits.py": 34,
+    "archivebox/tests/test_config_ONLY_NEW.py": 24,
+    "archivebox/tests/test_config_SAVE_TITLE.py": 99,
+    "archivebox/tests/test_config_URL_filters.py": 26,
+    "archivebox/tests/test_core_config.py": 20,
+    "archivebox/tests/test_core_forms.py": 28,
+    "archivebox/tests/test_crawl_pause.py": 20,
+    "archivebox/tests/test_crawl_runner.py": 45,
+    "archivebox/tests/test_crawl_service.py": 34,
+    "archivebox/tests/test_frozen_crawl_config.py": 47,
+    "archivebox/tests/test_hook_result_lifecycle.py": 78,
+    "archivebox/tests/test_hooks.py": 90,
+    "archivebox/tests/test_machine_models.py": 52,
+    "archivebox/tests/test_machine_service.py": 89,
+    "archivebox/tests/test_migrations_04_to_09.py": 36,
+    "archivebox/tests/test_migrations_07_to_09.py": 183,
+    "archivebox/tests/test_migrations_08_to_09.py": 244,
+    "archivebox/tests/test_migrations_fresh.py": 99,
+    "archivebox/tests/test_misc_checks.py": 22,
+    "archivebox/tests/test_misc_monkey_patches.py": 20,
+    "archivebox/tests/test_opencode_agent_1.py": 158,
+    "archivebox/tests/test_opencode_agent_2.py": 145,
+    "archivebox/tests/test_opencode_agent_3.py": 154,
+    "archivebox/tests/test_opencode_browser_1.py": 262,
+    "archivebox/tests/test_opencode_browser_2.py": 146,
+    "archivebox/tests/test_opencode_browser_3.py": 148,
+    "archivebox/tests/test_orm_helpers.py": 22,
+    "archivebox/tests/test_permissions.py": 21,
+    "archivebox/tests/test_persona_runtime.py": 56,
+    "archivebox/tests/test_postgres_backend.py": 85,
+    "archivebox/tests/test_process_runtime_paths.py": 28,
+    "archivebox/tests/test_process_service.py": 49,
+    "archivebox/tests/test_proof_previews.py": 54,
+    "archivebox/tests/test_recursive_crawl_1.py": 339,
+    "archivebox/tests/test_recursive_crawl_2.py": 171,
+    "archivebox/tests/test_recursive_crawl_3.py": 176,
+    "archivebox/tests/test_release_notes.py": 14,
+    "archivebox/tests/test_release_source_fingerprint.py": 22,
+    "archivebox/tests/test_release_workflow.py": 21,
+    "archivebox/tests/test_resource_admission.py": 21,
+    "archivebox/tests/test_runner_watch.py": 177,
+    "archivebox/tests/test_search.py": 153,
+    "archivebox/tests/test_serve_static.py": 31,
+    "archivebox/tests/test_server_automatic_url.py": 36,
+    "archivebox/tests/test_server_security_browser.py": 278,
+    "archivebox/tests/test_settings_signal_webhooks.py": 21,
+    "archivebox/tests/test_setup_script.py": 21,
+    "archivebox/tests/test_shutdown_util.py": 22,
+    "archivebox/tests/test_snapshot_filesystem_migration.py": 29,
+    "archivebox/tests/test_snapshot_output_stacks.py": 62,
+    "archivebox/tests/test_snapshot_ref_lookup.py": 23,
+    "archivebox/tests/test_snapshot_service.py": 41,
+    "archivebox/tests/test_snapshot_url_length.py": 26,
+    "archivebox/tests/test_tag_service.py": 24,
+    "archivebox/tests/test_takeover_util_1.py": 143,
+    "archivebox/tests/test_takeover_util_2.py": 201,
+    "archivebox/tests/test_takeover_util_3.py": 138,
+    "archivebox/tests/test_takeover_util_4.py": 156,
+    "archivebox/tests/test_test_harness.py": 22,
+    "archivebox/tests/test_ui_add_view_runtime.py": 221,
+    "archivebox/tests/test_ui_add_view.py": 46,
+    "archivebox/tests/test_ui_admin_apitoken.py": 23,
+    "archivebox/tests/test_ui_admin_archiveresult.py": 30,
+    "archivebox/tests/test_ui_admin_config_widget.py": 25,
+    "archivebox/tests/test_ui_admin_crawl.py": 44,
+    "archivebox/tests/test_ui_admin_links.py": 35,
+    "archivebox/tests/test_ui_admin_machine.py": 29,
+    "archivebox/tests/test_ui_admin_persona.py": 28,
+    "archivebox/tests/test_ui_admin_snapshot.py": 273,
+    "archivebox/tests/test_ui_admin_tag.py": 26,
+    "archivebox/tests/test_ui_admin_user.py": 28,
+    "archivebox/tests/test_ui_config_views.py": 43,
+    "archivebox/tests/test_ui_live_progress.py": 297,
+    "archivebox/tests/test_ui_public_snapshot.py": 179,
+    "archivebox/tests/test_ui_savepagenow.py": 48,
+    "archivebox/tests/test_urls.py": 220,
+    "archivebox/tests/test_util.py": 20,
+    "archivebox/tests/test_version_metadata.py": 24
+  }
+}
@@ -18,47 +18,68 @@ concurrency:
  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}

 jobs:
+  prepare:
+    permissions:
+      contents: write
+    uses: ./.github/workflows/prepare-release.yml
+    secrets: inherit
+
  lint:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Linters
    uses: ./.github/workflows/lint.yml
    secrets: inherit

  install-cli-platform:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Install and CLI platform compatibility
    uses: ./.github/workflows/test.yml
    secrets: inherit

  tests:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Discovered test matrix
    uses: ./.github/workflows/test-parallel.yml
    secrets: inherit

  documentation:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Documentation and root tests
    uses: ./.github/workflows/docs.yml
    secrets: inherit

  codeql:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: CodeQL
    uses: ./.github/workflows/codeql.yml
    secrets: inherit

  python-artifacts:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Tested Python artifacts
    uses: ./.github/workflows/pip.yml
    secrets: inherit

  docker-digests:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Tested Docker images
    uses: ./.github/workflows/docker.yml
    with:
-      push_digests: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
+      push_digests: ${{ github.event_name == 'push' && (github.ref_name == 'dev' || github.ref_name == 'main') }}
    secrets: inherit

  required:
    name: All required CI lanes
-    if: ${{ always() }}
+    if: ${{ always() && needs.prepare.outputs.run_jobs == 'true' }}
    needs:
+      - prepare
      - lint
      - install-cli-platform
      - tests
@@ -65,7 +65,7 @@ jobs:
        if: github.event_name == 'workflow_dispatch'
        shell: bash
        env:
-          SOURCE_REF: ${{ inputs.source_ref }}
+          SOURCE_REF: ${{ inputs.source_ref || github.sha }}
        run: |
          set -Eeuo pipefail
          [[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
@@ -295,7 +295,7 @@ jobs:
            '

      - name: Validate exact built image version and commit
-        if: ${{ !inputs.push_digests }}
+        if: inputs.full_tests
        shell: bash
        run: |
          set -Eeuo pipefail
@@ -552,6 +552,9 @@ jobs:

      - name: Export digest
        if: inputs.push_digests
+        env:
+          ABX_DL_IMAGE_REF: ${{ steps.abx_dl_image.outputs.image }}
+          ARTIFACT_NAME: ${{ matrix.artifact_name }}
        shell: bash
        run: |
          set -Eeuo pipefail
@@ -559,6 +562,7 @@ jobs:
          digest="${{ steps.docker_build.outputs.digest }}"
          touch "/tmp/digests/${digest#sha256:}"
          printf '%s\n' '${{ inputs.source_ref || github.sha }}' > /tmp/digests/source-ref
+          printf '%s\n' "$ABX_DL_IMAGE_REF" > "/tmp/digests/abx-dl-image-$ARTIFACT_NAME"

      - name: Upload digest
        if: inputs.push_digests
@@ -67,7 +67,7 @@ jobs:
    timeout-minutes: 15
    strategy:
      fail-fast: false
-      max-parallel: 4
+      max-parallel: 8
      matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -0,0 +1,55 @@
+name: Prepare release version
+
+on:
+  workflow_call:
+    outputs:
+      run_jobs:
+        description: Whether this revision owns its release version and should run expensive jobs
+        value: ${{ jobs.prepare.outputs.run_jobs }}
+
+permissions:
+  contents: write
+
+jobs:
+  prepare:
+    name: Prepare the version before starting expensive jobs
+    permissions:
+      contents: write
+    runs-on: ubuntu-24.04
+    outputs:
+      run_jobs: ${{ github.event_name != 'push' || steps.channel.outputs.stable == 'true' || steps.version.outputs.action == 'ready' || steps.version.outputs.action == 'reserved' }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          submodules: true
+          token: ${{ github.event_name == 'push' && secrets.RELEASE_GH_TOKEN || github.token }}
+
+      - name: Keep synchronized stable branches on the released version
+        id: channel
+        if: github.event_name == 'push'
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
+          git fetch --no-tags origin main
+          MAIN_SHA="$(git rev-parse origin/main)"
+          # Stable reconciliation may preserve dev history with a merge commit.
+          # Its tree still matches main; bumping it would undo synchronization
+          # before the stable publisher can advance the Docker :dev tag.
+          if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] &&
+             git merge-base --is-ancestor "$MAIN_SHA" "$GITHUB_SHA" &&
+             git diff --quiet "$MAIN_SHA" "$GITHUB_SHA"; then
+            echo 'stable=true' >> "$GITHUB_OUTPUT"
+          else
+            echo 'stable=false' >> "$GITHUB_OUTPUT"
+          fi
+
+      - id: version
+        if: github.event_name == 'push' && steps.channel.outputs.stable != 'true'
+        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
+        with:
+          package: archivebox
+          branch: ${{ github.ref_name }}
+          tag-prefix: v
+          version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}
@@ -1,79 +0,0 @@
-name: Release candidate
-
-on:
-  push:
-    branches: [dev, main]
-
-permissions:
-  actions: read
-  contents: write
-  packages: write
-
-concurrency:
-  group: release-candidate-${{ github.ref_name }}
-  cancel-in-progress: true
-
-jobs:
-  prepare:
-    runs-on: ubuntu-24.04
-    outputs:
-      action: ${{ steps.version.outputs.action }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          token: ${{ secrets.RELEASE_GH_TOKEN }}
-
-      - name: Keep synchronized stable branches on the released version
-        id: channel
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
-          MAIN_SHA="$(git ls-remote origin refs/heads/main | cut -f1)"
-          if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ && "$MAIN_SHA" == "$GITHUB_SHA" ]]; then
-            echo 'stable=true' >> "$GITHUB_OUTPUT"
-          else
-            echo 'stable=false' >> "$GITHUB_OUTPUT"
-          fi
-
-      - id: version
-        if: steps.channel.outputs.stable != 'true'
-        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
-        with:
-          package: archivebox
-          branch: ${{ github.ref_name }}
-          tag-prefix: v
-          version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}
-
-  python-artifacts:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    uses: ./.github/workflows/pip.yml
-    with:
-      full_tests: false
-    secrets: inherit
-
-  docker-digests:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    uses: ./.github/workflows/docker.yml
-    with:
-      push_digests: true
-      full_tests: false
-    secrets: inherit
-
-  opencode:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    strategy:
-      fail-fast: false
-      matrix:
-        test_file:
-          - archivebox/tests/test_opencode_agent.py
-          - archivebox/tests/test_opencode_browser.py
-    uses: ./.github/workflows/test-parallel.yml
-    with:
-      test_file: ${{ matrix.test_file }}
-    secrets: inherit
@@ -2,7 +2,7 @@ name: Release

 on:
  workflow_run:
-    workflows: [CI, Release candidate]
+    workflows: [CI]
    types: [completed]
    branches: [dev, main]
  workflow_dispatch:
@@ -111,26 +111,14 @@ jobs:
              --jq '[.status, .conclusion, .headSha, .event, .name] | @tsv')"
            [[ "$CI_STATUS" == "$(printf 'completed\tsuccess\t%s\tpush\tCI' "$RELEASE_SHA")" ]]
          else
-            CANDIDATE_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow release-candidate.yml \
-              --commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
-              --json databaseId,status,conclusion \
-              --jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
            CI_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml \
              --commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
              --json databaseId,status,conclusion \
              --jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
-            [[ -n "$CANDIDATE_RUN_ID" && -n "$CI_RUN_ID" ]] || exit 0
-            if [[ "$RELEASE_BRANCH" == dev ]]; then
-              DIGEST_RUN_ID="$CANDIDATE_RUN_ID"
-            else
-              DIGEST_RUN_ID="$CI_RUN_ID"
-            fi
-          fi
-          if [[ "$RELEASE_BRANCH" == dev && "$GITHUB_EVENT_NAME" != workflow_dispatch ]]; then
-            ARTIFACT_RUN_ID="$CANDIDATE_RUN_ID"
-          else
-            ARTIFACT_RUN_ID="$CI_RUN_ID"
+            [[ -n "$CI_RUN_ID" ]] || exit 0
+            DIGEST_RUN_ID="$CI_RUN_ID"
          fi
+          ARTIFACT_RUN_ID="$CI_RUN_ID"
          {
            echo 'ready=true'
            echo "artifact_run_id=$ARTIFACT_RUN_ID"
@@ -144,10 +132,12 @@ jobs:
    runs-on: ubuntu-24.04
    env:
      CI_RUN_ID: ${{ needs.candidate.outputs.artifact_run_id }}
+      DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
    environment: pypi
    permissions:
      actions: read
      contents: write
+      deployments: read
      id-token: write
    outputs:
      version: ${{ steps.version.outputs.version }}
@@ -171,6 +161,26 @@ jobs:
          version: "0.11.3"
          enable-cache: false

+      # This precedes every stable upload, including workflow_dispatch. A green
+      # build is not evidence that our automatically updated test servers ran it.
+      - name: Download tested Docker base-image metadata
+        if: env.RELEASE_BRANCH == 'main'
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          path: ${{ runner.temp }}/tested-docker-artifacts
+          pattern: digest-*
+          merge-multiple: true
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
+
+      - name: Require matching acceptance on Cabbage
+        if: env.RELEASE_BRANCH == 'main'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          STAGING_DOCKER_ARTIFACT_DIR: ${{ runner.temp }}/tested-docker-artifacts
+        run: uv run --no-project python bin/verify-staging-release.py
+
      - name: Resolve release binaries through abxpkg
        env:
          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
@@ -279,7 +289,7 @@ jobs:
          ref: ${{ env.RELEASE_SHA }}
          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}

-      - name: Fast-forward dev to the tested stable release
+      - name: Reconcile dev to the tested stable release
        if: env.RELEASE_BRANCH == 'main' && needs.python-release.outputs.release_ready == 'true'
        shell: bash
        run: |
@@ -287,10 +297,65 @@ jobs:
          git fetch --no-tags origin main dev
          MAIN_SHA="$(git rev-parse origin/main)"
          DEV_SHA="$(git rev-parse origin/dev)"
-          if [[ "$MAIN_SHA" == "$RELEASE_SHA" && "$DEV_SHA" != "$RELEASE_SHA" ]] &&
-             git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
-            git push origin "$RELEASE_SHA:refs/heads/dev"
+          if [[ "$MAIN_SHA" != "$RELEASE_SHA" || "$DEV_SHA" == "$RELEASE_SHA" ]]; then
+            exit 0
          fi
+          push_dev() {
+            if ! git push origin "$1:refs/heads/dev"; then
+              CURRENT_DEV="$(git ls-remote origin refs/heads/dev | cut -f1)"
+              if [[ "$CURRENT_DEV" == "$DEV_SHA" ]]; then
+                echo "Failed to push dev although it still points to $DEV_SHA" >&2
+                exit 1
+              fi
+              echo "Leaving dev at $CURRENT_DEV: it advanced during release reconciliation"
+            fi
+          }
+          if git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
+            push_dev "$RELEASE_SHA"
+            exit 0
+          else
+            STATUS=$?
+            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
+          fi
+          BASE_SHA="$(git merge-base "$MAIN_SHA" "$DEV_SHA")"
+          if git diff --quiet "$BASE_SHA" "$DEV_SHA" -- . ':!pyproject.toml' ':!uv.lock' ':!etc/package.json'; then
+            :
+          else
+            STATUS=$?
+            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
+            echo "Leaving dev at $DEV_SHA: it contains changes beyond release metadata"
+            exit 0
+          fi
+          normalized_metadata() {
+            case "$2" in
+              pyproject.toml)
+                git show "$1:$2" | awk '
+                  /^\[/ {section = $0}
+                  section == "[project]" && /^version = "[^"]+"$/ {print "version = \"__VERSION__\""; next}
+                  section == "[tool.bumpver]" && /^current_version = "[^"]+"$/ {print "current_version = \"__VERSION__\""; next}
+                  {print}
+                ' ;;
+              uv.lock)
+                git show "$1:$2" | awk 'previous == "name = \"archivebox\"" && /^version = / {print "version = \"__VERSION__\""; previous = $0; next} {print; previous = $0}' ;;
+              etc/package.json)
+                git show "$1:$2" | sed -E 's/^(  "version": )"[^"]+"(,?)$/\1"__VERSION__"\2/' ;;
+            esac
+          }
+          for path in pyproject.toml uv.lock etc/package.json; do
+            BASE_METADATA="$(normalized_metadata "$BASE_SHA" "$path")"
+            DEV_METADATA="$(normalized_metadata "$DEV_SHA" "$path")"
+            if [[ "$BASE_METADATA" != "$DEV_METADATA" ]]; then
+              echo "Leaving dev at $DEV_SHA: $path contains changes beyond release version metadata"
+              exit 0
+            fi
+          done
+          # A concurrent release bot may bump only dev's own version metadata.
+          # Keep that commit as a parent while preserving the exact tested stable tree.
+          git switch --detach "$MAIN_SHA"
+          git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
+            merge -s ours --no-ff -m "Merge tested stable release into dev" "$DEV_SHA"
+          MERGE_SHA="$(git rev-parse HEAD)"
+          push_dev "$MERGE_SHA"

      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
        with:
@@ -409,7 +474,8 @@ jobs:
            DEV_TARGET="$($GIT_BINARY rev-parse origin/dev)"
            # Recovery can add CI/docs fixes after the immutable release tag.
            # Both branches and every shipped application file must agree.
-            if [[ "$MAIN_TARGET" == "$DEV_TARGET" ]] &&
+            if $GIT_BINARY merge-base --is-ancestor "$MAIN_TARGET" "$DEV_TARGET" &&
+               $GIT_BINARY diff --quiet "$MAIN_TARGET" "$DEV_TARGET" &&
               $GIT_BINARY diff --quiet "$RELEASE_SHA" "$MAIN_TARGET" -- . ':!.github' ':!README.md'; then
              SYNC_DEV=true
            fi
@@ -471,6 +537,26 @@ jobs:
            test "$INSTALLED_VERSION" = "$VERSION"
          done

+      - name: Record published image source
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          # workflow_run can report the default branch's head even when this
+          # run published a different branch/SHA. Preserve the verified target.
+          [[ "$RELEASE_BRANCH" == dev || "$RELEASE_BRANCH" == main ]]
+          [[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
+          mkdir -p "$RUNNER_TEMP/published-image-source"
+          # shellcheck disable=SC2016
+          "$JQ_BINARY" -n --arg branch "$RELEASE_BRANCH" --arg sha "$RELEASE_SHA" \
+            '{branch: $branch, sha: $sha}' > "$RUNNER_TEMP/published-image-source/source.json"
+
+      - name: Upload published image source
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: published-image-source
+          path: ${{ runner.temp }}/published-image-source/source.json
+          if-no-files-found: error
+
      - name: Update Docker Hub README
        uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
        with:
@@ -13,6 +13,7 @@ on:
      - pyproject.toml
      - uv.lock
      - .github/workflows/screenshots.yml
+      - .github/workflows/prepare-release.yml
  workflow_dispatch:

 permissions:
@@ -23,7 +24,15 @@ concurrency:
  cancel-in-progress: false

 jobs:
+  prepare:
+    permissions:
+      contents: write
+    uses: ./.github/workflows/prepare-release.yml
+    secrets: inherit
+
  deploy:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
@@ -0,0 +1,109 @@
+name: Staging acceptance
+
+on:
+  workflow_run:
+    workflows: [Release]
+    types: [completed]
+
+permissions: {}
+
+concurrency:
+  group: staging-acceptance
+  cancel-in-progress: false
+
+jobs:
+  published:
+    if: github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-24.04
+    permissions:
+      actions: read
+    outputs:
+      ready: ${{ steps.source.outputs.ready }}
+      sha: ${{ steps.source.outputs.sha }}
+    steps:
+      - id: image
+        env:
+          GH_TOKEN: ${{ github.token }}
+          RELEASE_RUN_ID: ${{ github.event.workflow_run.id }}
+        run: |
+          set -Eeuo pipefail
+          published=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RELEASE_RUN_ID/jobs?per_page=100" --jq 'any(.jobs[]; .name == "Publish tested multiarch image" and .conclusion == "success")')
+          echo "published=$published" >> "$GITHUB_OUTPUT"
+      - name: Download published image source
+        if: steps.image.outputs.published == 'true'
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: published-image-source
+          path: ${{ runner.temp }}/published-image-source
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ github.event.workflow_run.id }}
+      - name: Select published dev image
+        id: source
+        if: steps.image.outputs.published == 'true'
+        run: |
+          set -Eeuo pipefail
+          # The triggering run may report the default branch as its head;
+          # this artifact names the image source verified by the publisher.
+          source="$RUNNER_TEMP/published-image-source/source.json"
+          jq -e 'type == "object" and (keys | sort) == ["branch", "sha"]' "$source" >/dev/null
+          branch=$(jq -er '.branch | select(type == "string")' "$source")
+          sha=$(jq -er '.sha | select(type == "string")' "$source")
+          [[ "$branch" == dev || "$branch" == main ]]
+          [[ "$sha" =~ ^[0-9a-f]{40}$ ]]
+          echo "sha=$sha" >> "$GITHUB_OUTPUT"
+          if [[ "$branch" == dev ]]; then
+            echo 'ready=true' >> "$GITHUB_OUTPUT"
+          else
+            echo 'ready=false' >> "$GITHUB_OUTPUT"
+          fi
+  acceptance:
+    # The privileged key is never exposed to PRs or arbitrary dispatch refs.
+    needs: published
+    if: needs.published.outputs.ready == 'true'
+    runs-on: [self-hosted, Linux, X64, ugnas-linux]
+    environment: staging
+    timeout-minutes: 110
+    permissions:
+      contents: read
+      deployments: write
+    strategy:
+      fail-fast: false
+      matrix:
+        target: [cabbage]
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ needs.published.outputs.sha }}
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e
+        with:
+          version: '0.11.3'
+      - name: Prepare browser and restricted staging identity
+        env:
+          STAGING_SSH_KEY: ${{ secrets.STAGING_SSH_KEY }}
+          STAGING_KNOWN_HOSTS: ${{ secrets.STAGING_KNOWN_HOSTS }}
+        run: |
+          set -Eeuo pipefail
+          umask 077
+          printf '%s\n' "$STAGING_SSH_KEY" > "$RUNNER_TEMP/staging-key"
+          printf '%s\n' "$STAGING_KNOWN_HOSTS" > "$RUNNER_TEMP/staging-known-hosts"
+          uv run --no-project --with playwright==1.63.0 playwright install chromium
+      - name: Verify actual deployment, full captures and rendered snapshots
+        env:
+          GH_TOKEN: ${{ github.token }}
+          STAGING_TARGET: ${{ matrix.target }}
+          STAGING_SHA: ${{ needs.published.outputs.sha }}
+          STAGING_KEY_FILE: ${{ runner.temp }}/staging-key
+          STAGING_KNOWN_HOSTS_FILE: ${{ runner.temp }}/staging-known-hosts
+        run: uv run --no-project --with playwright==1.63.0 python bin/check-staging-deployment.py
+      - name: Remove staging identity
+        if: always()
+        run: rm -f "$RUNNER_TEMP/staging-key" "$RUNNER_TEMP/staging-known-hosts"
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
+        if: always()
+        with:
+          name: staging-${{ matrix.target }}-${{ needs.published.outputs.sha }}
+          path: staging-evidence/
+          if-no-files-found: error
@@ -1,6 +1,13 @@
 name: Parallel Tests

 on:
+  workflow_dispatch:
+    inputs:
+      test_file:
+        description: Run one discovered test file
+        required: true
+        default: archivebox/tests/test_cli_list.py
+        type: string
  workflow_call:
    inputs:
      test_file:
@@ -19,11 +26,27 @@ env:
 jobs:
  discover-tests:
    name: Discover every test
+    # Selection must run on GitHub so an offline NAS cannot block the fallback.
    runs-on: ubuntu-24.04
    outputs:
      tests: ${{ steps.discover.outputs.tests }}
+      runner: ${{ steps.runner.outputs.use-runner || '["ubuntu-24.04"]' }}

    steps:
+      - name: Select available CI capacity
+        id: runner
+        if: vars.UGNAS_CI_ENABLED == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main')
+        uses: mikehardy/runner-fallback-action@dc7732f9e532c451b2527b288ba8d58fd4b0a4ca # v1.1.0
+        with:
+          github-token: ${{ secrets.RUNNER_STATUS_TOKEN }}
+          organization: ArchiveBox
+          primary-runner: self-hosted,ugnas-linux
+          fallback-runner: ubuntu-24.04
+          # Jobs queued for this pool must not be mistaken for idle runner slots:
+          # requiring MAX_JOBS idle runners prevents assigning a second wave.
+          primaries-required: ${{ vars.UGNAS_CI_MIN_IDLE || 1 }}
+          fallback-on-error: true
+
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
        with:
          submodules: true
@@ -66,6 +89,7 @@ jobs:
        id: discover
        env:
          TEST_FILE: ${{ inputs.test_file }}
+          UGNAS_CI_MAX_JOBS: ${{ vars.UGNAS_CI_MAX_JOBS || 3 }}
        run: |
          set -Eeuo pipefail
          mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
@@ -80,13 +104,13 @@ jobs:
  tests:
    name: ${{ matrix.test.name }}
    needs: discover-tests
-    runs-on: ubuntu-24.04
+    # Opt in trusted branches only; PRs always use GitHub-hosted runners.
+    # Ordinary Linux tests are eligible; # ci-runner: hosted declares exceptions.
+    # The hosted discovery job falls back when workers are offline/busy or the
+    # availability API fails. UGNAS_CI_ENABLED=false disables the pool entirely.
+    runs-on: ${{ fromJSON(matrix.test.ugnas && needs.discover-tests.outputs.runner || '["ubuntu-24.04"]') }}
    env:
-      ABXPKG_LIB_DIR: /tmp/archivebox-lib
      CHROME_HEADLESS: "true"
-      DATA_DIR: /tmp/archivebox-test-data
-      PERSONAS_DIR: /tmp/abx-personas
-      CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile

    strategy:
      fail-fast: false
@@ -94,6 +118,15 @@ jobs:
        test: ${{ fromJson(needs.discover-tests.outputs.tests) }}

    steps:
+      - name: Set job-local runtime paths
+        run: |
+          {
+            echo "ABXPKG_LIB_DIR=$RUNNER_TEMP/archivebox-lib"
+            echo "DATA_DIR=$RUNNER_TEMP/archivebox-test-data"
+            echo "PERSONAS_DIR=$RUNNER_TEMP/abx-personas"
+            echo "CHROME_USER_DATA_DIR=$RUNNER_TEMP/abx-personas/Default/chrome_profile"
+          } >> "$GITHUB_ENV"
+
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
        with:
          submodules: true
@@ -156,13 +189,35 @@ jobs:
          for test_path in "${test_paths[@]}"; do
            test -f "$test_path"
          done
+          test_status=0
          for test_path in "${test_paths[@]}"; do
            test_name="$(basename "$test_path" .py)"
            test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
            export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
            export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
            export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
+            if (( ${#test_paths[@]} > 1 )); then
+              export ABXPKG_LIB_DIR="$RUNNER_TEMP/archivebox-lib/$test_slug"
+            fi
            mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
-            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
-              --basetemp="$RUNNER_TEMP/pytest/$test_slug"
+            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" --durations=20 \
+              --basetemp="$RUNNER_TEMP/pytest/$test_slug" \
+              --junitxml="$RUNNER_TEMP/test-results/$test_slug.xml" || test_status=1
          done
+          exit "$test_status"
+      - name: Save test results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: test-results-${{ strategy.job-index }}
+          path: ${{ runner.temp }}/test-results/*.xml
+          if-no-files-found: error
+          retention-days: 14
+      - name: Save failed runtime diagnostics
+        if: failure()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: runtime-logs-${{ strategy.job-index }}
+          path: ${{ runner.temp }}/pytest/**/*.log
+          if-no-files-found: ignore
+          retention-days: 14
@@ -77,11 +77,7 @@ archivebox run

 ## Verification

-Use targeted tests for focused work:
-
-```bash
-uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
-uv run prek run --all-files
-```
+Run the test suite with `uv run pytest archivebox/tests -x` and repository checks
+with `uv run prek run --all-files`. For focused work, select the relevant tests.

 Releases are published only by `.github/workflows/release.yml` after complete CI succeeds for the exact commit. Pushes to `main` prepare stable patch releases automatically; `dev` prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
@@ -125,6 +125,7 @@ FROM archivebox-runtime-base AS archivebox-builder
 WORKDIR "$CODE_DIR"
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
    --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
+    --mount=type=bind,source=uv.lock,target=/app/uv.lock \
    <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
 export UV_NO_CACHE=true
@@ -137,48 +138,14 @@ apt-get install -qq -y --no-install-recommends x11-utils
 # Do not rewrite inherited native libraries merely to strip them again.
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries

-mkdir -p /tmp/archivebox-uv-project
-/venv/bin/python - <<'PY'
-from pathlib import Path
-import json
-import re
-import urllib.request
-
-source = Path("/app/pyproject.toml")
-target = Path("/tmp/archivebox-uv-project/pyproject.toml")
-text = source.read_text()
-text = text.replace(
-    'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
-    'environments = ["sys_platform == \'linux\'"]',
-)
-
-# Docker builds need the just-published internal abx wheels immediately, but
-# PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
-# Docker-only dependency view from the version JSON so the published package
-# metadata stays normal while image builds remain resumable after a release.
-for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
-    match = re.search(
-        rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
-        text,
-    )
-    if not match:
-        continue
-    version = match.group("version")
-    with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
-        data = json.load(response)
-    wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
-    text = re.sub(
-        rf'"{re.escape(package)}(?:==|>=)[^"]+"',
-        f'"{package} @ {wheel_url}"',
-        text,
-        count=1,
-    )
-
-target.write_text(text)
-PY
-
+# The old Docker-only dependency rewrite predated this checkout's lockfile:
+# it fetched version JSON to avoid stale PyPI Simple listings. That extra API
+# can itself lag or return 503s. The verified lock already records exact public
+# artifact URLs and hashes, including extras and platform markers. Consume it
+# directly, retaining the freshness check instead of resolving a second graph.
 /usr/bin/uv sync \
-    --project /tmp/archivebox-uv-project \
+    --project /app \
+    --locked \
    --no-cache \
    --no-dev \
    --inexact \
@@ -209,7 +176,7 @@ rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
    /venv/lib/python3.*/site-packages/wheel* \
    /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
+rm -rf /var/lib/apt/lists/*
 EOF

 COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
@@ -291,7 +258,13 @@ RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=
        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install $ABX_DOCKER_PLUGINS \
    # pnpm includes musl variants that Debian's glibc runtime cannot use.
    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
-    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} +
+    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} + \
+    # pnpm's explicit store lives under ABXPKG_LIB_DIR, regardless of XDG_CACHE_HOME.
+    # Retaining its download copies made image size depend on whether the build
+    # preserved hardlinks to installed files. Remove the store in this layer;
+    # installed files retain their contents, and the offline check below proves
+    # the complete catalog still works without the disposable package payloads.
+    && rm -rf "$ABXPKG_LIB_DIR/cache/pnpm"
 RUN --network=none export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
    bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install $ABX_DOCKER_PLUGINS'
@@ -105,12 +105,11 @@ curl -fsSL 'https://get.archivebox.io' | bash
 </details>
 <br/>

-
 <div align="center" style="text-align: center">
 <br/><br/>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
 <br/><br/>
-<small><a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
+<small><a href="https://demo.archivebox.io">Demo</a> | <a href="https://archivebox.io/screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
 <br/>
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
 <br/><br/>
@@ -381,7 +380,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 > *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*

 <ul>
-<li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
+<li><a href="https://apps.truenas.com/catalog/archivebox_community/">TrueNAS</a></li>
 <li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
 <li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
 <li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
@@ -182,6 +182,14 @@ def crawl_file(request: HttpRequest, crawl_id: str, path: str):
    if not file_path.is_file() or crawl_root not in file_path.parents:
        raise HttpError(404, "Crawl file not found")

+    if path == "chrome_screencast/latest.jpg" and request.GET.get("after"):
+        try:
+            started_at = float(request.GET["after"])
+        except ValueError:
+            raise HttpError(400, "Invalid screencast start time")
+        if file_path.stat().st_mtime < started_at:
+            raise HttpError(404, "No frame for this capture yet")
+
    response = FileResponse(file_path.open("rb"))
    response["Cache-Control"] = "no-store, no-cache, max-age=0, must-revalidate"
    response["Pragma"] = "no-cache"
@@ -174,8 +174,12 @@ def cli(ctx, help=False):
                archivebox_init(quick=True)
                os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
            check_data_folder()
-            if subcommand != "update":
-                check_migrations(auto_apply=True)
+            if subcommand != "update" and not check_migrations(auto_apply=True):
+                from archivebox.machine.models import Machine
+
+                # Host registration and config reconciliation belong to CLI
+                # startup; get_config() is also used by read-only HTTP views.
+                Machine.current()
        except Exception as e:
            STDERR.print(f"[red][X] Error setting up Django or checking data folder: {e}[/red]")
            if subcommand not in ("manage", "shell"):  # not all management commands need django to be setup beforehand
@@ -111,6 +111,11 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
        else:
            os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init

+    from archivebox.machine.models import Machine
+
+    # Registration requires the final schema and reconciles legacy file config.
+    Machine.current()
+
    from archivebox.misc.db import database_display_location, is_postgres

    assert database_exists()
@@ -1113,7 +1113,7 @@ def get_config(
            if apps.ready:
                from archivebox.machine.models import Machine

-                machine = Machine.current()
+                machine = Machine.current_readonly()
        except (DatabaseError, ImportError, RuntimeError, TypeError, ValueError):
            machine = None

@@ -63,7 +63,14 @@ def setup_django(check_db=False) -> None:
            os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
            if CONSTANTS.DATA_DIR.exists():
                for child in CONSTANTS.DATA_DIR.iterdir():
-                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
+                    try:
+                        os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
+                    except FileNotFoundError:
+                        # A concurrent SQLite transaction can remove its journal
+                        # between directory enumeration and chown. There is no
+                        # ownership left to fix; other permission errors must
+                        # still fail startup rather than being hidden.
+                        continue

    # Suppress the "database access during app initialization" warning
    # This warning can be triggered during django.setup() but is safe to ignore
@@ -85,23 +85,31 @@ def get_collection_id(DATA_DIR=DATA_DIR) -> str:


@cache
-def get_machine_id() -> str:
-    """Get a short, stable, unique ID for the current machine (e.g. abc45678)"""
+def get_machine_id(length: int = 8) -> str:
+    """Get a machine ID, using the network identity when no OS ID is available."""
+    import machineid

-    MACHINE_ID = "unknown"
    try:
-        import machineid
+        return machineid.hashed_id("archivebox")[:length]
+    except machineid.MachineIdNotFound:
+        import uuid

-        MACHINE_ID = machineid.hashed_id("archivebox")[:8]
-    except Exception:
-        try:
-            import uuid
-            import hashlib
+        import psutil

-            MACHINE_ID = hashlib.sha256(str(uuid.getnode()).encode()).hexdigest()[:8]
-        except Exception:
-            pass
-    return MACHINE_ID
+        node = uuid.getnode()
+        identity = str(node)
+        if node & (1 << 40):
+            # uuid.getnode() silently generates a per-process random value when
+            # tools such as ip/ifconfig are absent. Read interfaces directly so
+            # separate CLI processes agree even in a minimal container.
+            addresses = sorted(
+                address.address
+                for interface in psutil.net_if_addrs().values()
+                for address in interface
+                if address.family == psutil.AF_LINK and address.address not in ("", "00:00:00:00:00:00")
+            )
+            identity = str(int(addresses[0].replace(":", "").replace("-", ""), 16)) if addresses else f"hostname:{socket.gethostname()}"
+        return hashlib.sha256(identity.encode()).hexdigest()[:length]


@cache
@@ -681,7 +681,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
        request = self.request
        return build_snapshot_files_url(
            str(result.snapshot_id),
-            result.plugin,
+            result.output_folder_path(),
            request=request,
            config=request.archivebox_config,
        )
@@ -690,7 +690,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
        request = self.request
        return build_snapshot_zip_url(
            str(result.snapshot_id),
-            result.plugin,
+            result.output_folder_path(),
            request=request,
            config=request.archivebox_config,
        )
@@ -100,11 +100,12 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
    if schema_editor.connection.vendor != "sqlite":
        return

-    from django.db import connection
+    from django.utils import timezone
    import uuid as uuid_lib
-    from datetime import datetime

+    connection = schema_editor.connection
    cursor = connection.cursor()
+    now = connection.ops.adapt_datetimefield_value(timezone.now())

    # Check if there are any snapshots without a crawl
    cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
@@ -125,14 +126,13 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
            INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
        """,
-            [datetime.now().isoformat()],
+            [now],
        )

    # Create a default crawl for migrated snapshots.
    # Depending on migration graph order, later crawls migrations may already
    # have removed output_dir by the time this data migration runs.
    crawl_id = uuid_lib.uuid4().hex
-    now = datetime.now().isoformat()
    cursor.execute("PRAGMA table_info(crawls_crawl)")
    crawl_columns = {row[1] for row in cursor.fetchall()}
    default_columns: list[str] = []
@@ -155,9 +155,9 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
            status, retry_at, created_by_id, schedule_id, config, persona_id
        ) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
                  'Auto-created crawl for migrated snapshots', {default_values_sql}
-                  'sealed', %s, 1, NULL, '{{}}', NULL)
+                  'sealed', NULL, 1, NULL, '{{}}', NULL)
    """,
-        [crawl_id, now, now, crawl_urls, now],
+        [crawl_id, now, now, crawl_urls],
    )

    # Assign all snapshots without a crawl to the default crawl
@@ -0,0 +1,60 @@
+"""Repair SQLite queue timestamps written as raw ISO strings by old migrations."""
+
+from django.db import migrations
+from django.utils.dateparse import parse_datetime
+
+
+def repair_legacy_retry_timestamps(apps, schema_editor):
+    connection = schema_editor.connection
+    if connection.vendor != "sqlite":
+        return
+
+    with connection.cursor() as cursor:
+        # 0024 created this Crawl already sealed, but left a due retry marker.
+        # Its orphaned snapshots have their own scheduler rows for maintenance.
+        cursor.execute(
+            """
+            UPDATE crawls_crawl SET retry_at = NULL
+            WHERE status = 'sealed'
+              AND label = 'Migrated from v0.7.2/v0.8.6'
+              AND notes = 'Auto-created crawl for migrated snapshots'
+              AND substr(retry_at, 11, 1) = 'T'
+            """,
+        )
+
+        # SQLite compares stored date strings lexically. A raw `T` timestamp
+        # from a legacy migration sorts differently from Django's space-separated
+        # parameter, so a due row can be selected but never claimed. Preserve
+        # the datetime Django already reads from each row while rewriting its
+        # storage representation through the active database adapter.
+        for table in ("crawls_crawl", "core_snapshot", "machine_binary", "machine_process"):
+            last_id = ""
+            while True:
+                cursor.execute(
+                    f"SELECT id, CAST(retry_at AS TEXT) FROM {table} WHERE id > %s AND substr(retry_at, 11, 1) = 'T' ORDER BY id LIMIT 500",
+                    [last_id],
+                )
+                rows = cursor.fetchall()
+                if not rows:
+                    break
+                for row_id, raw_retry_at in rows:
+                    try:
+                        parsed = parse_datetime(raw_retry_at)
+                    except ValueError:
+                        continue
+                    if parsed is None:
+                        continue
+                    normalized = connection.ops.adapt_datetimefield_value(parsed)
+                    cursor.execute(
+                        f"UPDATE {table} SET retry_at = %s WHERE id = %s AND retry_at = %s",
+                        [normalized, row_id, raw_retry_at],
+                    )
+                last_id = rows[-1][0]
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0054_snapshot_fs_version_index"),
+    ]
+
+    operations = [migrations.RunPython(repair_legacy_retry_timestamps, migrations.RunPython.noop)]
@@ -0,0 +1,25 @@
+"""Repair process links after both core's legacy copy and machine migrations."""
+
+from importlib import import_module
+
+from django.db import migrations
+
+
+repair_process_binary_iface_links = import_module(
+    "archivebox.machine.migrations.0020_repair_process_binary_iface_links",
+).repair_process_binary_iface_links
+
+
+class Migration(migrations.Migration):
+    atomic = False
+
+    dependencies = [
+        ("core", "0055_repair_legacy_retry_timestamps"),
+        ("machine", "0022_networkinterface_identity_without_mac"),
+    ]
+
+    operations = [
+        # The original repair can run before core.0027 creates legacy Process
+        # rows. Repair again after the copy, including already upgraded DBs.
+        migrations.RunPython(repair_process_binary_iface_links, migrations.RunPython.noop),
+    ]
@@ -895,7 +895,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
            return False
        if self.status == self.StatusChoices.QUEUED:
            return bool(self.url) and self.start_processing()
-        # abx-dl emits SnapshotCompletedEvent after the complete hook sequence;
+        # SnapshotCompletedEvent seals completed or explicitly aborted attempts;
+        # runner takeover preserves the active capture for its replacement.
        # ArchiveResult projection state never drives Snapshot completion.
        return False

@@ -3755,6 +3756,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
        output_sizes = plugin_output_sizes(archive_results)
        for name, output in outputs_by_name.items():
            output["result_ids"] = ",".join(result_ids_by_name.get(name, ()))
+            output["folder_path"] = (
+                output["result"].output_folder_path() if output.get("result") is not None else str(Path(str(output["path"])).parent)
+            )
+            if output["folder_path"] == ".":
+                output["folder_path"] = ""
            if name in output_sizes:
                output["size"] = output_sizes[name]

@@ -4431,6 +4437,10 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
            return 0

    def output_file_map(self) -> dict[str, dict[str, Any]]:
+        # List views opt into this cache on their read-only, request-local rows.
+        # Runner-owned results must still reflect metadata changed during a hook.
+        if "_render_output_file_map" in self.__dict__:
+            return self._render_output_file_map
        return self._normalize_output_files(self.output_files)

    def output_file_paths(self) -> list[str]:
@@ -4585,11 +4595,20 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
        output_file_map: dict[str, dict[str, Any]] | None = None,
    ) -> str | None:
        ignored = {"stdout.log", "stderr.log", "hook.pid", "listener.pid"}
-        candidates = [
-            path
-            for path in output_file_paths
-            if Path(path).name not in ignored and Path(path).suffix.lower() not in (".pid", ".log", ".sh")
-        ]
+        # Responses/wget manifests can contain thousands of paths. Parse each
+        # once instead of rebuilding Path objects for every preferred filename
+        # and extension group, each time a list row selects a preview.
+        candidates = []
+        names = {}
+        extensions = {}
+        for path in output_file_paths:
+            parsed_path = Path(path)
+            name, extension = parsed_path.name, parsed_path.suffix.lower()
+            if name in ignored or extension in (".pid", ".log", ".sh"):
+                continue
+            candidates.append(path)
+            names.setdefault(name.lower(), path)
+            extensions[path] = extension
        if not candidates:
            return None

@@ -4612,9 +4631,8 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
            "article.json",
        ]
        for preferred_name in preferred_names:
-            for candidate in candidates:
-                if Path(candidate).name.lower() == preferred_name:
-                    return candidate
+            if preferred_name in names:
+                return names[preferred_name]

        from archivebox.plugins.discovery import get_plugin_output_extension_preference

@@ -4625,7 +4643,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
            (".mp4", ".webm", ".mp3", ".opus", ".ogg", ".wav"),
        )
        for ext_group in ext_groups:
-            group_candidates = [candidate for candidate in candidates if Path(candidate).suffix.lower() in ext_group]
+            group_candidates = [candidate for candidate in candidates if extensions[candidate] in ext_group]
            if group_candidates:
                return max(
                    group_candidates,
@@ -4720,6 +4738,14 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
        """
        return self.embed_path_db()

+    def output_folder_path(self) -> str:
+        """Return the physical folder for this result's files, using saved metadata."""
+        output_path = self.embed_path_db()
+        if output_path and not output_path.startswith(f"{self.plugin}/"):
+            parent = Path(output_path).parent.as_posix()
+            return "" if parent == "." else parent
+        return self.plugin
+
    @property
    def output_dir_name(self) -> str:
        return self.plugin
@@ -74,8 +74,9 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
        **snapshot_filter,
    ).update(retry_at=now, modified_at=now)

+    # Find rare uploads through hook_name's index before checking their parents;
+    # a correlated upload probe otherwise visits every sealed snapshot.
    extension_upload_results = ArchiveResult.objects.filter(
-        snapshot_id=OuterRef("pk"),
        hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
    )
    server_results = ArchiveResult.objects.filter(snapshot_id=OuterRef("pk")).exclude(
@@ -83,15 +84,15 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
    )
    extension_only_snapshots = (
        Snapshot.objects.filter(
+            id__in=extension_upload_results.values("snapshot_id"),
            status=Snapshot.StatusChoices.SEALED,
            crawl__status__in=[Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED],
            **snapshot_filter,
        )
        .annotate(
-            has_extension_upload=Exists(extension_upload_results),
            has_server_result=Exists(server_results),
        )
-        .filter(has_extension_upload=True, has_server_result=False)
+        .filter(has_server_result=False)
    )
    # Older browser-extension uploads could win a race with runner startup:
    # their successful external rows made the fresh Snapshot look finished
@@ -1,3 +1,28 @@
+"""Shared foreground signal and child-process cleanup contracts.
+
+Shutdown intent and process exit are separate: takeover must preserve the
+replacement owner's work, while a user abort must not later resume the crawl.
+Callers own that policy; this module preserves the received signal and stops
+only the process objects they explicitly supply.
+
+Cooperative boundaries are essential, not just quieter exception handling.
+A real Linux CI regression interrupted Popen.poll() with SIGINT and stranded
+its wait lock. SIGKILL stopped the child (it became a zombie), but Popen.wait()
+still timed out. Raising a different exception or waiting longer cannot repair
+that interrupted critical section. Polling loops must record the signal first
+and honor it between operations; async owners may cancel their own task while
+their loop is running. Once exiting, the last-resort process cleanup must stay
+synchronous: the original loop may be closed, and creating a replacement loop
+cannot recover tasks or resources owned by it.
+
+Keep immediate signal notices and repeated-signal force exit independent of
+the event bus: it may already be draining or closed during shutdown. Resumable
+pause/retry/skip belongs to the existing crawl controller and must not set
+sticky command-shutdown state. See foreground_shutdown_signals below for both
+cooperative and legacy blocking callers; do not change all callers' policy at
+once without supplying their interruption boundaries.
+"""
+
 from __future__ import annotations

 import os
@@ -23,7 +48,7 @@ _active_shutdown_state: ShutdownSignalState | None = None


 def raise_if_shutdown_requested() -> None:
-    """Let long foreground loops honor a signal even if Python ignored it once."""
+    """Honor sticky shutdown intent at a caller-owned interruption boundary."""

    if _active_shutdown_state and _active_shutdown_state.signal_name:
        raise KeyboardInterrupt
@@ -48,14 +73,37 @@ def wait_popen_and_kill_children(
    timeout: float,
    kill_timeout: float = 2.0,
 ) -> None:
-    """Wait for a Popen parent and then hard-kill any surviving descendants."""
+    """Reap our own Popen child and stop its explicitly captured descendants.
+
+    A dead child still needs its parent to reap it. Keep wait failures visible;
+    bypassing Popen's lock would hide the interrupted-owner bug described above.
+    Descendants must still be cleaned up when that wait fails.
+    """

    try:
-        proc.wait(timeout=timeout)
-    except subprocess.TimeoutExpired:
-        proc.kill()
-        proc.wait(timeout=kill_timeout)
-    kill_remaining_processes(children, timeout=kill_timeout)
+        try:
+            proc.wait(timeout=timeout)
+        except subprocess.TimeoutExpired:
+            proc.kill()
+            proc.wait(timeout=kill_timeout)
+    except subprocess.TimeoutExpired as err:
+        try:
+            state = psutil.Process(proc.pid).as_dict(attrs=["pid", "ppid", "status", "num_threads", "cpu_times"])
+        except psutil.NoSuchProcess:
+            state = {"pid": proc.pid, "status": "exited"}
+        err.add_note(f"Process state after SIGKILL: {state}")
+        # A locked Popen wait can time out even after the OS process exits.
+        # Inspect CPython's lock only for diagnostics; never release it here.
+        waitpid_lock = getattr(proc, "_waitpid_lock", None)
+        err.add_note(
+            f"Popen state: returncode={proc.returncode}, "
+            f"waitpid_lock_locked={waitpid_lock.locked() if waitpid_lock is not None else None}, "
+            f"threads={[thread.name for thread in threading.enumerate()]}",
+        )
+        raise
+    finally:
+        # A parent that cannot be reaped must not leave its descendants running.
+        kill_remaining_processes(children, timeout=kill_timeout)


 def wait_psutil_and_kill_children(
@@ -104,11 +152,14 @@ def foreground_shutdown_signals(
    interrupt_handlers: dict[signal.Signals, Callable[[], None]] | None = None,
    raise_on_first_signal: bool = True,
 ) -> Iterator[ShutdownSignalState]:
-    """Install foreground signal handlers that print an immediate exit notice.
+    """Keep shutdown intent visible until the owning foreground command unwinds.

-    Some log-tail loops intentionally swallow KeyboardInterrupt so that callers
-    can centralize cleanup in finally blocks. The handler writes the signal name
-    immediately, then raises KeyboardInterrupt to break out of the blocking read.
+    Signal state was added because inner loops can swallow KeyboardInterrupt
+    while their caller still needs to stop its children. Cooperative callers use
+    raise_on_first_signal=False and check raise_if_shutdown_requested() between
+    operations, or cancel their owned async task through on_signal. Raising from
+    the handler can otherwise interrupt library locks or partially sent I/O.
+    Blocking callers without a cooperative boundary retain immediate exceptions.
    """

    global _active_shutdown_state
@@ -775,7 +775,9 @@ def snapshot_index_row(context, link) -> str:
    title = unescape(getattr(link, "title", "") or "")
    title_text = title or url
    tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
-    tag_html = "".join(f'<span class="snapshot-tag">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag)
+    tag_html = "".join(
+        f'<span class="snapshot-tag" title="{escape(tag)}">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag
+    )
    if tag_html:
        tag_cell = f'<span class="snapshot-tags">{tag_html}</span>'
    else:
@@ -17,7 +17,6 @@ from django.contrib.auth import HASH_SESSION_KEY, SESSION_KEY, get_user_model
 from django.contrib.auth.mixins import UserPassesTestMixin
 from django.contrib.sessions.models import Session
 from django.core import signing
-from django.core.paginator import InvalidPage
 from django.db.models import Case, IntegerField, Q, Value, When
 from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidden, QueryDict
 from django.shortcuts import redirect, render
@@ -48,7 +47,6 @@ from archivebox.config.configset import BaseConfigSet
 from archivebox.core.forms import AddLinkForm
 from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
 from archivebox.core.permissions import (
-    PERMISSIONS_PRIVATE,
    PERMISSIONS_PUBLIC,
    PERMISSIONS_UNLISTED,
    can_view_snapshot,
@@ -92,6 +90,7 @@ from archivebox.plugins.discovery import (
 from archivebox.plugins.forms import get_plugin_config_binary_urls
 from archivebox.plugins.views import get_config_definition_link
 from archivebox.progressmonitor.views import live_progress_view
+from archivebox.progressmonitor.collection import collection_summary
 from archivebox.search.config import (
    get_search_mode,
    get_search_mode_backend,
@@ -1213,6 +1212,18 @@ class OriginalDomainReplayView(View):


 class PublicIndexView(ListView):
+    """Keep browsing responsive while the same small server is archiving.
+
+    Performance takes priority over perfect initial totals: on a 1 vCPU / 1 GB
+    host with hundreds of thousands of snapshots, even an indexed count can
+    consume the entire roughly 0.5-0.7 second page-load budget. Benchmark with
+    the runner active, since it shares the CPU, memory, and SQLite database.
+
+    The unfiltered list therefore renders rows first and accepts missing/stale
+    display totals. A later ordinary reload can show the background-cached
+    count; do not restore a synchronous COUNT or add polling just to fill it in.
+    """
+
    template_name = "public_index.html"
    model = Snapshot
    ordering: ClassVar[list[str]] = ["-bookmarked_at", "-created_at"]
@@ -1238,14 +1249,28 @@ class PublicIndexView(ListView):
            "status",
            "output_size",
            "permissions",
+            # Modern replay URLs need the filesystem version and owner's name.
+            # Deferring these caused four extra SELECTs per displayed snapshot.
+            "fs_version",
+            "crawl__created_by__username",
        )

-    def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot] | None:
-        target_count = page_number * page_size
+    def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot]:
+        # One extra row determines Next without a full collection count. Cached
+        # totals can be stale while the runner adds/removes snapshots, so they
+        # must never decide whether the requested page exists.
+        target_count = page_number * page_size + 1
        public_snapshots: list[Snapshot] = []
        scanned = 0
-        chunk_size = max(self.public_page_scan_chunk_size, page_size)
-        ordered_snapshots = Snapshot.objects.order_by(*self.ordering).only(*self._base_public_snapshot_fields())
+        chunk_size = max(self.public_page_scan_chunk_size, page_size + 1)
+        # Read in snapshot_public_order_idx order and filter visibility in small
+        # batches. Combining the permissions predicate with ORDER BY can make
+        # SQLite select the permissions index and sort the whole public archive
+        # before LIMIT. This favors the common early pages without a new index;
+        # deep pages or mostly private collections can still require more reads.
+        ordered_snapshots = (
+            Snapshot.objects.select_related("crawl__created_by").order_by(*self.ordering).only(*self._base_public_snapshot_fields())
+        )

        while len(public_snapshots) < target_count:
            chunk = list(ordered_snapshots[scanned : scanned + chunk_size])
@@ -1261,20 +1286,39 @@ class PublicIndexView(ListView):
        if self.request.GET.get("q", default="").strip():
            return super().paginate_queryset(queryset, page_size)

-        public_count = self.get_exact_public_snapshot_count()
-        paginator = self.get_paginator(range(public_count), page_size)
        page_kwarg = self.kwargs.get(self.page_kwarg)
        page_query = self.request.GET.get(self.page_kwarg)
        page_number = page_kwarg or page_query or 1

        try:
-            page = paginator.page(page_number)
-        except InvalidPage as err:
+            page_number = int(page_number)
+            if page_number < 1:
+                raise ValueError("Page must be positive")
+        except (ValueError, TypeError) as err:
            raise Http404(f"Invalid page ({page_number}): {err}") from err

-        object_list = self._ordered_public_page_from_order_index(page_number=page.number, page_size=page_size)
-        page.object_list = object_list
-        return paginator, page, object_list, page.has_other_pages()
+        snapshots = self._ordered_public_page_from_order_index(page_number=page_number, page_size=page_size)
+        if not snapshots and page_number > 1:
+            raise Http404(f"Invalid page ({page_number}): No snapshots")
+        # Exhausting the rows gives an exact total for this read at no extra
+        # cost. Otherwise the lookahead is only a lower bound: an absent count
+        # is preferable to an on-request scan or a cached total smaller than
+        # the rows we just observed. Cached overestimates are acceptable here.
+        observed_count = (page_number - 1) * page_size + len(snapshots)
+        self.public_snapshot_count = observed_count
+        if len(snapshots) > page_size:
+            summary = collection_summary(self.request.user, public=True)
+            cached_count = summary["snapshots"] if summary is not None else None
+            self.public_snapshot_count = cached_count if cached_count is not None and cached_count >= observed_count else None
+
+        # Give Django an in-memory lower bound, never the collection QuerySet
+        # whose Paginator.count would issue COUNT(*). Navigation uses this read's
+        # lookahead even when a stale total would hide a newly added page.
+        # The template must use the separate display totals, not num_pages here.
+        paginator = self.get_paginator(range(observed_count), page_size)
+        page = paginator.page(page_number)
+        page.object_list = snapshots[:page_size]
+        return paginator, page, page.object_list, page.has_other_pages()

    def get_context_data(self, **kwargs):
        runtime_config = self.__dict__.get("runtime_config")
@@ -1303,10 +1347,16 @@ class PublicIndexView(ListView):
            and search_mode_backend
            and context["paginator"].count == 0,
        )
+        context["public_snapshot_count"] = context["paginator"].count if query else self.public_snapshot_count
+        count = context["public_snapshot_count"]
+        context["public_page_count"] = (
+            max(1, (count + context["paginator"].per_page - 1) // context["paginator"].per_page) if count is not None else None
+        )
        snapshots = list(context.get("object_list") or ())
        from archivebox.plugins.output_groups import plugin_output_sizes

        all_results_by_snapshot = {str(snapshot.id): [] for snapshot in snapshots}
+        snapshots_by_id = {str(snapshot.id): snapshot for snapshot in snapshots}
        icons_by_snapshot: dict[str, dict[str, ArchiveResult]] = {str(snapshot.id): {} for snapshot in snapshots}
        tag_names_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
        list_icon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
@@ -1343,8 +1393,17 @@ class PublicIndexView(ListView):
                    "output_size",
                )
            )
-            for result in results.iterator(chunk_size=1000):
+            # This is already bounded by the displayed snapshot page. Finish
+            # the SELECT before parsing output manifests: a streaming cursor
+            # otherwise keeps SQLite's read lock throughout rendering.
+            for result in list(results):
                snapshot_key = str(result.snapshot_id)
+                # Thumbnail URLs traverse result.snapshot too; share the page's
+                # already-loaded parent rather than fetching it once per result.
+                result.snapshot = snapshots_by_id[snapshot_key]
+                # These read-only page objects are shared by icons, sizes, and
+                # thumbnails. Normalize each manifest once for this render.
+                result._render_output_file_map = result.output_file_map()
                all_results_by_snapshot[snapshot_key].append(result)
                progress = progress_by_snapshot[snapshot_key]
                progress["total"] += 1
@@ -1376,13 +1435,12 @@ class PublicIndexView(ListView):
        context["object_list"] = snapshots
        return context

-    def get_exact_public_snapshot_count(self) -> int:
-        hidden_count = Snapshot.objects.filter(permissions=PERMISSIONS_PRIVATE).count()
-        hidden_count += Snapshot.objects.filter(permissions=PERMISSIONS_UNLISTED).count()
-        return Snapshot.objects.count() - hidden_count
-
    def get_queryset(self, **kwargs):
-        qs = public_snapshots_queryset(super().get_queryset(**kwargs)).only(*self._base_public_snapshot_fields())
+        qs = (
+            public_snapshots_queryset(super().get_queryset(**kwargs))
+            .select_related("crawl__created_by")
+            .only(*self._base_public_snapshot_fields())
+        )
        query = self.request.GET.get("q", default="").strip()

        if not query:
@@ -717,7 +717,7 @@ class InlineTagEditorWidget(TagEditorWidget):
                )
            pills_html += f'''
                <span class="tag-pill" data-tag="{self._escape(td["name"])}" data-tag-id="{td["id"]}" style="{self._tag_style(td["name"])}">
-                    <a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link">{self._escape(td["name"])}</a>
+                    <a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link" title="{self._escape(td["name"])}">{self._escape(td["name"])}</a>
                    {remove_button}
                </span>
            '''
@@ -57,7 +57,10 @@ def get_vm_info():

    hw_manufacturer = "Docker" if hw_in_docker else "Unknown"
    hw_product = "Container" if hw_in_docker else "Unknown"
-    hw_uuid = machineid.id()
+    try:
+        hw_uuid = machineid.id()
+    except machineid.MachineIdNotFound:
+        hw_uuid = "Unknown"

    if platform.system().lower() == "darwin":
        # Get macOS machine info
@@ -301,7 +304,9 @@ def get_host_stats() -> dict[str, Any]:


 def get_host_guid() -> str:
-    return machineid.hashed_id("archivebox")
+    from archivebox.config.paths import get_machine_id
+
+    return get_machine_id(length=64)


 # Example usage
@@ -221,35 +221,21 @@ class Machine(ModelWithHealthStats):
        app_label = "machine"

    @classmethod
-    def current(cls, refresh: bool = False) -> Machine:
-        global _CURRENT_MACHINE
-        if refresh:
-            _CURRENT_MACHINE = None
-        if _CURRENT_MACHINE:
-            if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
-                # Reconciliation can save() and clear the shared cache; keep
-                # this call's instance so we never return None after that save.
-                machine = _CURRENT_MACHINE
-                # One-time-per-process reconciliation between ArchiveBox.conf
-                # and Machine.config. Fast-path: bool check + early-return when
-                # the sync has already run, so the cached-machine return path
-                # stays sub-microsecond.
-                try:
-                    from archivebox.config.collection import sync_machine_and_file
+    def current_readonly(cls) -> Machine | None:
+        """Read the current machine without registration, refresh, or config writes."""
+        if _CURRENT_MACHINE is not None:
+            return _CURRENT_MACHINE
+        return cls.objects.filter(guid=get_host_guid()).first()

-                    sync_machine_and_file(machine)
-                except Exception:
-                    pass
-                return machine
-            else:
-                _CURRENT_MACHINE = None
+    @classmethod
+    def current(cls) -> Machine:
+        global _CURRENT_MACHINE
+        if _CURRENT_MACHINE is not None:
+            return _CURRENT_MACHINE

        host_guid = get_host_guid()
-        # save() intentionally invalidates _CURRENT_MACHINE so config edits
-        # made through other instances become visible in this process. Host
-        # refresh (after seven days), sanitation, and reconciliation can all
-        # save here, so the working instance must not live only in that cache.
-        # Otherwise a routine refresh leaves startup dereferencing None.
+        # Host detection, sanitation, and config reconciliation run only on
+        # initial registration, never on cached reads.
        try:
            machine = cls.objects.get(guid=host_guid)
        except cls.DoesNotExist:
@@ -296,17 +282,14 @@ class Machine(ModelWithHealthStats):
                    ],
                )
        machine = cls._sanitize_config(machine)
-        # Same one-time sync as the cached-return path. Triggers here on the
-        # very first ``Machine.current()`` call in a process before any
-        # cached return can occur.
+        # Reconcile once, on the first Machine.current() call in this process.
        try:
            from archivebox.config.collection import sync_machine_and_file

            sync_machine_and_file(machine)
        except Exception:
            pass
-        # Publish only after all save-capable steps; publishing earlier would
-        # let those steps invalidate the cache we are in the middle of filling.
+        # Publish only after initialization is complete.
        _CURRENT_MACHINE = machine
        return machine

@@ -377,18 +360,15 @@ class Machine(ModelWithHealthStats):
            if update_fields is not None:
                kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))

-        # Drop the ``Machine.current()`` module-level cache on every save so
-        # config edits (admin form, from_json, etc.) become live in the same
-        # process without waiting out the 7-day ``MACHINE_RECHECK_INTERVAL``.
-        # The cache reads ``_CURRENT_MACHINE.modified_at`` and that value
-        # never moves forward on the cached object even when the row is
-        # updated in the DB, so without this we'd keep serving stale
-        # ``machine.config`` (incl. ``BASE_URL``) until the worker restarts.
+        # Apply explicit edits to the cached instance without making the next
+        # read repeat host detection, sanitation, or config reconciliation.
        update_fields = kwargs.get("update_fields")
        super().save(*args, **kwargs)
        global _CURRENT_MACHINE
        if _CURRENT_MACHINE is not None and _CURRENT_MACHINE.pk == self.pk:
-            _CURRENT_MACHINE = None
+            for field in self._meta.concrete_fields:
+                if field.attname in self.__dict__ and (update_fields is None or field.name in update_fields):
+                    setattr(_CURRENT_MACHINE, field.attname, self.__dict__[field.attname])

        # Mirror Machine.config into ArchiveBox.conf so the two stores stay
        # 1:1. Skipped when ``update_fields`` is set and doesn't touch
@@ -443,7 +423,7 @@ class NetworkInterface(ModelWithHealthStats):
    @classmethod
    def current(cls, refresh: bool = False) -> NetworkInterface:
        global _CURRENT_INTERFACE
-        machine = Machine.current(refresh=refresh)
+        machine = Machine.current()
        if _CURRENT_INTERFACE and _CURRENT_INTERFACE.machine_id == machine.id:
            if not refresh:
                # Callers that pass refresh=False are asking for attribution to
@@ -1778,7 +1758,8 @@ class Process(ModelWithDeleteAfter, models.Model):
            return None

        # Can't validate processes on other machines
-        if self.machine_id != Machine.current().id:
+        machine = Machine.current_readonly()
+        if machine is None or self.machine_id != machine.id:
            return None

        try:
@@ -445,7 +445,10 @@ def migration_lock(stdout: TextIO | None = None):


 # Migration names that previously existed in ArchiveBox's source tree but have
-# since been deleted (squashed away, renamed, moved between apps, etc.). DBs
+# since been deleted (squashed away, renamed, moved between apps, etc.), plus
+# exact older migrations generated in the installed package by v0.6.2's
+# startup makemigrations. Those runtime-generated names were never in Git, so
+# the deleted-file history alone cannot cover them. DBs
 # upgraded incrementally through 0.8.x → 0.9.x dev rcs accumulate rows for
 # these in ``django_migrations``; the newer-DB guard added in 65dc2521 would
 # otherwise refuse to start with "applied migrations missing from this build"
@@ -454,13 +457,18 @@ def migration_lock(stdout: TextIO | None = None):
 # splits the migration graph when only a *subset* of the replaces list is
 # applied (which is exactly what happens for users at different intermediate
 # dev branch states). This set is the authoritative compat list — extend it
-# when squashing more migrations away. Generated from
-# ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
+# when squashing more migrations away. The source-tracked names were generated
+# from ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
 HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
    {
        # core: 0023→0075 sequence plus every transient dev rename
        ("core", "0002_auto_20190417_0739"),
        ("core", "0006_auto_20200915_2006"),
+        # Reported v0.6.2 runtime-generated 0021 migrations; the checked-in
+        # replacement only changes ArchiveResult.extractor choices.
+        ("core", "0021_auto_20220510_0644"),
+        ("core", "0021_auto_20220724_1254"),
+        ("core", "0021_auto_20221128_1116"),
        ("core", "0023_alter_archiveresult_options_archiveresult_abid_and_more"),
        ("core", "0023_new_schema"),
        ("core", "0024_auto_20240513_1143"),
@@ -518,6 +526,8 @@ HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
        ("core", "0053_remove_snapshottag_snapshot_old"),
        ("core", "0054_alter_snapshot_timestamp"),
        ("core", "0055_alter_tag_slug"),
+        # Dev-only progress indexes removed in favor of existing-index queries.
+        ("core", "0056_progress_covering_indexes"),
        ("core", "0056_remove_tag_uuid"),
        ("core", "0057_rename_id_tag_old_id"),
        ("core", "0058_alter_tag_old_id"),
@@ -3,7 +3,7 @@ __package__ = "archivebox.misc"
 # Bootable utility functions (URL parsing, date parsing, JSON encoding, decorators).
 # MUST NOT import archivebox.config, archivebox.core, or Django — this module is
 # loaded by hooks.py and other early code paths. Only depends on stdlib and the
-# bootable .logging module.
+# bootable .logging module and standalone abx_plugins URL helpers.

 import re
 import json as pyjson
@@ -19,6 +19,9 @@ from urllib.parse import urlparse, quote, unquote
 from html import escape, unescape
 from datetime import datetime, timezone

+from abx_plugins.plugins.base.url_cleaning import URL_REGEX, fix_url_from_markdown
+from abx_plugins.plugins.base.url_cleaning import sanitize_extracted_url as sanitize_plugin_url
+
 from .logging import COLOR_DICT


@@ -90,18 +93,6 @@ def ts_to_date_str(ts: Any) -> str | None:
 COLOR_REGEX = re.compile(r"\[(?P<arg_1>\d+)(;(?P<arg_2>\d+)(;(?P<arg_3>\d+))?)?m")


-# https://mathiasbynens.be/demo/url-regex
-URL_REGEX = re.compile(
-    r"(?=("
-    r"http[s]?://"  # start matching from allowed schemes
-    r"(?:[a-zA-Z]|[0-9]"  # followed by allowed alphanum characters
-    r"|[-_$@.&+!*\(\),]"  #   or allowed symbols (keep hyphen first to match literal hyphen)
-    r"|[^\u0000-\u007F])+"  #   or allowed unicode bytes
-    r'[^\]\[<>"\'\s]+'  # stop parsing at these symbols
-    r"))",
-    re.IGNORECASE | re.UNICODE,
-)
-
 # Maximum supported URL length. Very long URLs are rare but must be supported correctly
 # (e.g. data: URLs, deeply nested query strings). The Snapshot.url column is stored as a
 # variable-length TextField (so short URLs don't reserve space and very long URLs still
@@ -109,29 +100,6 @@ URL_REGEX = re.compile(
 # all keep working.
 MAX_URL_LENGTH = 65535

-QUOTE_DELIMITERS = (
-    '"',
-    "'",
-    "`",
-    "“",
-    "”",
-    "‘",
-    "’",
-)
-QUOTE_ENTITY_DELIMITERS = (
-    "&quot;",
-    "&#34;",
-    "&#x22;",
-    "&apos;",
-    "&#39;",
-    "&#x27;",
-)
-URL_ENTITY_REPLACEMENTS = (
-    ("&amp;", "&"),
-    ("&#38;", "&"),
-    ("&#x26;", "&"),
-)
-
 FILESIZE_UNITS: dict[str, int] = {
    "": 1,
    "b": 1,
@@ -153,36 +121,9 @@ FILESIZE_UNITS: dict[str, int] = {


 def sanitize_extracted_url(url: str) -> str:
-    """Trim quote garbage and dangling prose punctuation from an extracted URL candidate."""
-    cleaned = (url or "").strip()
-    if not cleaned:
-        return cleaned
-
-    lower_cleaned = cleaned.lower()
-    cut_index = len(cleaned)
-
-    for delimiter in QUOTE_DELIMITERS:
-        found_index = cleaned.find(delimiter)
-        if found_index != -1:
-            cut_index = min(cut_index, found_index)
-
-    for delimiter in QUOTE_ENTITY_DELIMITERS:
-        found_index = lower_cleaned.find(delimiter)
-        if found_index != -1:
-            cut_index = min(cut_index, found_index)
-
-    cleaned = cleaned[:cut_index].strip()
-    lower_cleaned = cleaned.lower()
-    for entity, replacement in URL_ENTITY_REPLACEMENTS:
-        while entity in lower_cleaned:
-            entity_index = lower_cleaned.find(entity)
-            cleaned = cleaned[:entity_index] + replacement + cleaned[entity_index + len(entity) :]
-            lower_cleaned = cleaned.lower()
-
-    cleaned = cleaned.rstrip(".,;:!?\\'\"")
-    cleaned = cleaned.rstrip('"')
-
-    return cleaned
+    """Use the standalone parser's quote rules, then trim dangling prose punctuation."""
+    cleaned = sanitize_plugin_url(url)
+    return cleaned.rstrip('.,;:!?\\"')


 def validate_url_length(url: str) -> str:
@@ -215,39 +156,6 @@ def parens_are_matched(string: str, open_char="(", close_char=")"):
    return count == 0


-def fix_url_from_markdown(url_str: str) -> str:
-    """
-    cleanup a regex-parsed url that may contain dangling trailing parens from markdown link syntax
-    helpful to fix URLs parsed from markdown e.g.
-      input:  https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def).somemoretext
-      result: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def
-
-    IMPORTANT ASSUMPTION: valid urls wont have unbalanced or incorrectly nested parentheses
-    e.g. this will fail the user actually wants to ingest a url like 'https://example.com/some_wei)(rd_url'
-         in that case it will return https://example.com/some_wei (truncated up to the first unbalanced paren)
-    This assumption is true 99.9999% of the time, and for the rare edge case the user can use url_list parser.
-    """
-    trimmed_url = url_str
-    if len(trimmed_url) > 2048:
-        return trimmed_url
-
-    # cut off one trailing character at a time
-    # until parens are balanced e.g. /a(b)c).x(y)z -> /a(b)c
-    trim_attempts = 0
-    while trimmed_url and not parens_are_matched(trimmed_url) and trim_attempts < 256:
-        trimmed_url = trimmed_url[:-1]
-        trim_attempts += 1
-
-    if not trimmed_url or not parens_are_matched(trimmed_url):
-        return url_str
-
-    # make sure trimmed url is still valid
-    if any(match == trimmed_url for match in re.findall(URL_REGEX, trimmed_url)):
-        return trimmed_url
-
-    return url_str
-
-
 def split_comma_separated_urls(url: str):
    offset = 0
    while True:
@@ -274,7 +182,8 @@ def find_all_urls(urls_str: str):
        if match.start() in skipped_starts:
            continue

-        cleaned_match = sanitize_extracted_url(fix_url_from_markdown(match.group(1)))
+        cleaned_match = fix_url_from_markdown(match.group(1), preceding_text=urls_str[max(0, match.start(1) - 6) : match.start(1)])
+        cleaned_match = sanitize_extracted_url(cleaned_match)
        for offset, url in split_comma_separated_urls(cleaned_match):
            if offset:
                skipped_starts.add(match.start() + offset)
@@ -1,4 +1,15 @@
-"""Non-blocking, database-only collection totals for the progress monitor."""
+"""Cached display totals for progress polling and the public snapshot list.
+
+Missing/stale statistics are intentional: returning useful rows and progress
+quickly matters more than exact initial counts. Each viewer may poll once per
+second; repeating aggregates per request would consume the small server's CPU
+and hold SQLite read locks needed by the archiving runner. Callers must accept
+None on a cold cache and must never wait for the refresh or count synchronously.
+
+Background reads still take normal SQLite read locks. Existing indexes, short
+batches where needed, and throttled refreshes limit their cost; moving work to
+a thread does not make a scan free or require switching the database to WAL.
+"""

 import logging
 from threading import Lock, Thread
@@ -9,14 +20,77 @@ from django.db import close_old_connections, connections
 from django.db.models import Count, Sum


+# This is only a process-local gate for background work, not a DB/file lock.
+# Every request tries it without waiting; a busy refresh means returning the
+# cached value (or None). Serializing refreshes also avoids competing scans for
+# different crawls/users when many viewer tabs open at once.
 _refresh_lock = Lock()


-def collection_summary(user):
-    user_id = None if user.is_superuser else user.pk
-    key = f"progress-collection:{'all' if user_id is None else user_id}"
+def crawl_summary(crawl_id, *, snapshot_id=None):
+    """Sample sealed snapshot metadata without adding table scans to polls."""
+    key = f"progress-crawl:{crawl_id}:{snapshot_id or 'all'}"
    summary = cache.get(key)
-    # Even an indexed SUM scans the index. Never make an HTTP request wait for
+    if (summary is None or time() - summary["sampled_at"] >= 30) and _refresh_lock.acquire(blocking=False):
+        if not cache.add(f"{key}:refresh", True, timeout=30):
+            _refresh_lock.release()
+            return summary
+
+        def refresh():
+            try:
+                close_old_connections()
+                from archivebox.core.models import Snapshot
+
+                snapshots = Snapshot.objects.filter(crawl_id=crawl_id, status=Snapshot.StatusChoices.SEALED)
+                if snapshot_id is not None:
+                    snapshots = snapshots.filter(pk=snapshot_id)
+                snapshots = snapshots.order_by("modified_at").values_list("modified_at", "downloaded_at", "output_size")
+                totals = {"snapshots": 0, "cancelled": 0, "bytes": 0}
+                modified_at, offset = None, 0
+                while True:
+                    batch_scope = snapshots.filter(modified_at__gte=modified_at) if modified_at is not None else snapshots
+                    # Use (crawl, status, modified_at), releasing the SQLite read
+                    # cursor after each small batch. Never hold a cursor over the
+                    # whole crawl, even in this background worker.
+                    batch = list(batch_scope[offset : offset + 100])
+                    for _, downloaded_at, size in batch:
+                        totals["snapshots"] += 1
+                        totals["cancelled"] += downloaded_at is None
+                        totals["bytes"] += size or 0
+                    if len(batch) < 100:
+                        break
+                    last_modified_at = batch[-1][0]
+                    # Only offset within ties; otherwise advance the indexed
+                    # timestamp range instead of rescanning previous pages.
+                    offset = (offset if last_modified_at == modified_at else 0) + sum(row[0] == last_modified_at for row in batch)
+                    modified_at = last_modified_at
+                totals["sampled_at"] = time()
+                cache.set(key, totals, timeout=3600)
+            except Exception:
+                logging.getLogger(__name__).exception("Could not refresh crawl totals")
+            finally:
+                connections.close_all()
+                _refresh_lock.release()
+
+        try:
+            Thread(target=refresh, name="crawl-summary", daemon=True).start()
+        except RuntimeError:
+            _refresh_lock.release()
+            logging.getLogger(__name__).exception("Could not start crawl totals refresh")
+    return summary
+
+
+def collection_summary(user, *, public=False):
+    """Return cached totals immediately and, when due, start one refresh.
+
+    Public pages need no extra AJAX/polling for these totals: an ordinary later
+    page load reads the populated cache. Keep this best-effort display cache
+    separate from pagination/authorization decisions, which need actual rows.
+    """
+    user_id = None if user.is_superuser else user.pk
+    key = "progress-collection:public" if public else f"progress-collection:{'all' if user_id is None else user_id}"
+    summary = cache.get(key)
+    # Even indexed COUNT/SUM aggregates scan the index. Never make HTTP wait for
    # that scan on a large collection. Serve stale totals while one worker refreshes;
    # a single non-blocking lock prevents a queue of scans from concurrent clients.
    if (summary is None or time() - summary["sampled_at"] >= 60) and _refresh_lock.acquire(blocking=False):
@@ -32,13 +106,29 @@ def collection_summary(user):
                from archivebox.core.models import Snapshot

                snapshots = Snapshot.objects.all()
-                if user_id is not None:
-                    snapshots = snapshots.filter(crawl__created_by_id=user_id)
-                # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
-                # Snapshot.output_size is maintained when ArchiveResult sizes change.
-                totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
-                totals["bytes"] = totals["bytes"] or 0
+                if public:
+                    from archivebox.core.permissions import PERMISSIONS_PUBLIC
+
+                    # SQLite can choose a table scan for equality after ANALYZE
+                    # on an all-public archive. Equal range bounds keep the count
+                    # on the existing permissions index. It is still O(n), so
+                    # public pages share this background cache, never await it.
+                    # Count the indexed, non-null permissions column so this
+                    # needs no Snapshot table reads or new covering-index migration.
+                    totals = snapshots.filter(permissions__gte=PERMISSIONS_PUBLIC, permissions__lte=PERMISSIONS_PUBLIC).aggregate(
+                        snapshots=Count("permissions"),
+                    )
+                else:
+                    if user_id is not None:
+                        snapshots = snapshots.filter(crawl__created_by_id=user_id)
+                    # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
+                    # Snapshot.output_size is maintained when ArchiveResult sizes change.
+                    totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+                totals["bytes"] = totals.get("bytes") or 0
                totals["sampled_at"] = time()
+                # Retain old totals longer than the 60s refresh interval so a
+                # slow/failed refresh does not force every tab back to a cold
+                # count. Staleness is acceptable; blocking page loads is not.
                cache.set(key, totals, timeout=3600)
            except Exception:
                logging.getLogger(__name__).exception("Could not refresh collection totals")
@@ -1904,12 +1904,22 @@
    const progressEndpoint = monitor.dataset.progressEndpoint || '/progress.json';

    function fetchProgress() {
+        // Opening a replay replaces the queued snapshot's progress panel.
+        // Its interval and already-running fetch must stop using that old DOM.
+        if (!monitor.isConnected) {
+            stopPolling();
+            return;
+        }
        fetch(progressEndpoint, { credentials: 'same-origin' })
            .then(response => {
                if (!response.ok) throw new Error(`HTTP ${response.status}`);
                return response.json();
            })
            .then(data => {
+                if (!monitor.isConnected) {
+                    stopPolling();
+                    return;
+                }
                if (data.error) {
                    console.error('Progress API error:', data.error, data.traceback);
                    setOrchestratorState('error', 'Backend not responding');
@@ -1921,6 +1931,10 @@
                updateProgress(data);
            })
            .catch(error => {
+                if (!monitor.isConnected) {
+                    stopPolling();
+                    return;
+                }
                console.error('Progress fetch error:', error);
                setOrchestratorState('error', 'Backend not responding');
                idleMessage.textContent = 'Fetch Error: ' + error.message;
@@ -1929,7 +1943,7 @@
    }

    function startPolling() {
-        if (pollInterval) return;
+        if (pollInterval || !monitor.isConnected) return;
        fetchProgress();
        pollInterval = setInterval(fetchProgress, pollDelayMs);
    }
@@ -2033,7 +2047,13 @@

    // Start polling when page loads
    startPolling();
-    setInterval(updateDurationBadges, 1000);
+    const durationInterval = setInterval(() => {
+        if (!monitor.isConnected) {
+            clearInterval(durationInterval);
+            return;
+        }
+        updateDurationBadges();
+    }, 1000);

    // Pause polling when tab is hidden
    document.addEventListener('visibilitychange', function() {
@@ -7,12 +7,11 @@ from typing import Literal
 from abx_dl.events import PROCESS_EXIT_SKIPPED
 from django.conf import settings
 from django.db import DatabaseError
-from django.db.models import CharField, Count, Q, Sum
+from django.db.models import CharField, Count, Q
 from django.db.models.functions import Cast
 from django.http import HttpResponse, JsonResponse
 from django.utils import timezone

-from archivebox.config.common import get_config
 from archivebox.core.permissions import can_view_snapshot, is_admin_user
 from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_url, get_api_base_url
 from archivebox.misc.logging_util import printable_filesize
@@ -182,53 +181,37 @@ def live_progress_view(request, *, authorized_snapshot=None):

        running_screencasts = {}

-        def screencast_frame_url(crawl_id: str, crawl_dir: Path) -> str:
-            frame_path = crawl_dir / "chrome_screencast" / "latest.jpg"
-            try:
-                frame_stat = frame_path.stat()
-            except OSError:
-                return ""
-            if frame_stat.st_size <= 0:
-                return ""
+        def screencast_frame_url(crawl_id: str) -> str:
            started_at = running_screencasts.get(crawl_id)
-            if started_at is None or frame_stat.st_mtime < started_at.timestamp():
+            if started_at is None:
                return ""
-            rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={frame_stat.st_mtime_ns}"
+            # File existence/freshness belongs to the image request, never to
+            # the one-second JSON poll (archive storage may be remote).
+            rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={int(now.timestamp())}&after={started_at.timestamp()}"
            return f"{api_base}{rel}" if api_base else rel

-        machine_id = Machine.current().id
+        machine = Machine.current_readonly()
+        machine_id = machine.id if machine is not None else None
        orchestrator_proc = (
            Process.objects.filter(
                machine_id=machine_id,
                process_type=Process.TypeChoices.ORCHESTRATOR,
                status=Process.StatusChoices.RUNNING,
            )
-            .only("id", "pid", "started_at", "machine_id", "process_type", "status")
+            .only("id", "pid", "started_at", "machine_id", "process_type", "status", "env", "cmd")
            .order_by("-started_at")
            .first()
            if machine_id is not None
            else None
        )
-        runner_worker = None
        orchestrator_proc_running = bool(orchestrator_proc and orchestrator_proc.is_running)
-        if not orchestrator_proc_running:
-            try:
-                from archivebox.workers.supervisord_util import get_existing_supervisord_process, get_worker
-
-                supervisor = get_existing_supervisord_process(quiet=True)
-                runner_worker = get_worker(supervisor, "worker_runner") if supervisor else None
-            except (OSError, RuntimeError, TimeoutError):
-                runner_worker = None
-
-        runner_worker_running = bool(runner_worker and runner_worker.get("statename") in ("STARTING", "RUNNING"))
-        runner_worker_pid = runner_worker.get("pid") if runner_worker else None
-        orchestrator_running = orchestrator_proc_running or runner_worker_running
-        orchestrator_pid = orchestrator_proc.pid if orchestrator_proc_running and orchestrator_proc else runner_worker_pid
+        orchestrator_running = orchestrator_proc_running
+        orchestrator_pid = orchestrator_proc.pid if orchestrator_proc_running and orchestrator_proc else None

        # Get model counts by status
        crawl_status_counts = Crawl.status_counts(
            crawl_scope,
-            (Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.PAUSED),
+            (Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED),
        )
        crawls_queued = crawl_status_counts.get(Crawl.StatusChoices.QUEUED, 0)
        crawls_active = crawl_status_counts.get(Crawl.StatusChoices.STARTED, 0)
@@ -242,7 +225,7 @@ def live_progress_view(request, *, authorized_snapshot=None):

        snapshot_status_counts = Snapshot.status_counts(
            snapshot_scope,
-            Snapshot.OPEN_STATES,
+            Snapshot.RUNNABLE_STATES,
        )
        snapshots_queued = snapshot_status_counts.get(Snapshot.StatusChoices.QUEUED, 0)
        snapshots_active = snapshot_status_counts.get(Snapshot.StatusChoices.STARTED, 0)
@@ -252,15 +235,24 @@ def live_progress_view(request, *, authorized_snapshot=None):
            ArchiveResult.StatusChoices.QUEUED,
            ArchiveResult.StatusChoices.STARTED,
        )
-        archiveresult_status_counts = ArchiveResult.status_counts(archiveresult_scope, result_statuses)
+        # Start from open results using the existing status index. Filtering by
+        # plugin first scans historical runs, including all completed results.
+        result_counts = list(
+            archiveresult_scope.filter(status__in=result_statuses).order_by().values("status", "plugin").annotate(count=Count("*")),
+        )
+        archiveresult_status_counts = {
+            status: sum(row["count"] for row in result_counts if row["status"] == status) for status in result_statuses
+        }
        download_scope = archiveresult_scope.filter(
            plugin__in=download_plugin_names,
            snapshot__status__in=Snapshot.RUNNABLE_STATES,
            snapshot__crawl__status__in=Crawl.RUNNABLE_STATES,
        )
-        indexing_scope = archiveresult_scope.filter(plugin__in=indexing_plugin_names)
        download_status_counts = ArchiveResult.status_counts(download_scope, result_statuses)
-        indexing_status_counts = ArchiveResult.status_counts(indexing_scope, result_statuses)
+        indexing_status_counts = {
+            status: sum(row["count"] for row in result_counts if row["status"] == status and row["plugin"] in indexing_plugin_names)
+            for status in result_statuses
+        }
        archiveresults_queued = archiveresult_status_counts.get(ArchiveResult.StatusChoices.QUEUED, 0)
        archiveresults_active = archiveresult_status_counts.get(ArchiveResult.StatusChoices.STARTED, 0)

@@ -296,21 +288,27 @@ def live_progress_view(request, *, authorized_snapshot=None):
            .values(*active_crawl_fields)
            .order_by("-modified_at")[:max_active_crawls],
        )
-        paused_crawls = list(
-            crawl_scope.filter(
-                Q(status=Crawl.StatusChoices.PAUSED, created_at__gte=paused_crawl_cutoff)
-                | Q(
-                    status=Crawl.StatusChoices.PAUSED,
-                    snapshot_set__status__in=Snapshot.RUNNABLE_STATES,
-                    snapshot_set__retry_at__lte=now,
-                )
-                | Q(
-                    status=Crawl.StatusChoices.PAUSED,
-                    snapshot_set__archiveresult__status=ArchiveResult.StatusChoices.QUEUED,
-                ),
+        # Start with runnable children, not every historical snapshot belonging
+        # to a paused crawl. These uncorrelated subqueries run once per poll.
+        due_paused_snapshots = (
+            Snapshot.objects.filter(
+                status__in=Snapshot.RUNNABLE_STATES,
+                retry_at__lte=now,
            )
+            .order_by()
+            .values("crawl_id")
+        )
+        queued_paused_results = (
+            ArchiveResult.objects.filter(
+                status=ArchiveResult.StatusChoices.QUEUED,
+            )
+            .order_by()
+            .values("snapshot__crawl_id")
+        )
+        paused_crawls = list(
+            crawl_scope.filter(status=Crawl.StatusChoices.PAUSED)
+            .filter(Q(created_at__gte=paused_crawl_cutoff) | Q(id__in=due_paused_snapshots) | Q(id__in=queued_paused_results))
            .values(*active_crawl_fields)
-            .distinct()
            .order_by("-modified_at")[:max_active_crawls],
        )
        queued_crawls = list(
@@ -324,52 +322,29 @@ def live_progress_view(request, *, authorized_snapshot=None):
                crawl["persona_id"] = str(crawl["persona_id"])
        persona_details_by_id: dict[str, dict[str, str]] = {}
        persona_details_by_name: dict[str, dict[str, str]] = {}
-        persona_objects_by_id = {}
-        persona_objects_by_name = {}
        persona_ids = {crawl["persona_id"] for crawl in active_crawls_list if crawl["persona_id"]}
        persona_names = {"Default"} if any(not crawl["persona_id"] for crawl in active_crawls_list) else set()
        if persona_ids or persona_names:
            from archivebox.personas.models import Persona

-            for persona in Persona.objects.filter(Q(id__in=persona_ids) | Q(name__in=persona_names)).only("id", "name", "config"):
+            for persona in Persona.objects.filter(Q(id__in=persona_ids) | Q(name__in=persona_names)).only("id", "name"):
                persona_details = {
                    "name": persona.name,
                    "admin_url": f"/admin/personas/persona/{persona.pk}/change/",
                }
                persona_details_by_id[str(persona.id)] = persona_details
                persona_details_by_name[persona.name] = persona_details
-                persona_objects_by_id[str(persona.id)] = persona
-                persona_objects_by_name[persona.name] = persona
        active_crawl_ids = [crawl["id"] for crawl in active_crawls_list]
-        active_crawl_objects = {}
-        if active_crawl_ids:
-            for crawl_obj in Crawl.objects.filter(id__in=active_crawl_ids).select_related("created_by", "persona"):
-                crawl_obj._runtime_config = request_config
-                active_crawl_objects[str(crawl_obj.id)] = crawl_obj
        snapshot_counts_by_crawl: dict[str, dict[str, int]] = {str(crawl_id): {} for crawl_id in active_crawl_ids}
-        cancelled_snapshot_counts_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
-        crawl_output_sizes_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
        queued_snapshot_overflow_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
        active_snapshot_scope = snapshot_scope.filter(crawl_id__in=active_crawl_ids)
        if active_crawl_ids:
-            for row in active_snapshot_scope.values("crawl_id", "status").annotate(count=Count("id")):
-                snapshot_counts_by_crawl.setdefault(str(row["crawl_id"]), {})[row["status"]] = row["count"]
-
-            for row in (
-                active_snapshot_scope.filter(status=Snapshot.StatusChoices.SEALED, downloaded_at__isnull=True)
-                .values("crawl_id")
-                .annotate(count=Count("id"))
-            ):
-                cancelled_snapshot_counts_by_crawl[str(row["crawl_id"])] = row["count"]
-
-            for row in (
-                active_snapshot_scope.filter(
-                    status=Snapshot.StatusChoices.SEALED,
-                )
-                .values("crawl_id")
-                .annotate(size=Sum("output_size"))
-            ):
-                crawl_output_sizes_by_crawl[str(row["crawl_id"])] = int(row["size"] or 0)
+            # The existing (crawl, status, modified_at) index covers these counts.
+            # Byte totals and cancellation details require table reads, so sample
+            # those separately and share cached values between viewer tabs.
+            for row in active_snapshot_scope.order_by().values("crawl_id", "status").annotate(count=Count("*")):
+                crawl_id = str(row["crawl_id"])
+                snapshot_counts_by_crawl[crawl_id][row["status"]] = row["count"]

        crawl_process_pids: dict[str, int] = {}
        snapshot_process_pids: dict[str, int] = {}
@@ -588,7 +563,14 @@ def live_progress_view(request, *, authorized_snapshot=None):
            completed_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.SEALED, 0)
            started_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.STARTED, 0)
            pending_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.QUEUED, 0)
-            cancelled_snapshots = cancelled_snapshot_counts_by_crawl.get(crawl_id, 0)
+            from archivebox.progressmonitor.collection import crawl_summary
+
+            totals = (
+                crawl_summary(crawl_id, snapshot_id=scoped_snapshot.pk if scoped_snapshot is not None else None)
+                if completed_snapshots
+                else {"snapshots": 0, "cancelled": 0, "bytes": 0}
+            )
+            cancelled_snapshots = totals["cancelled"] if totals is not None else None

            # Count URLs in the crawl (for when snapshots haven't been created yet)
            urls_count = 0
@@ -607,7 +589,7 @@ def live_progress_view(request, *, authorized_snapshot=None):
            crawl_setup_completed = sum(1 for item in crawl_setup_plugins if item.get("status") == "succeeded")
            crawl_setup_failed = sum(1 for item in crawl_setup_plugins if item.get("status") == "failed")
            crawl_setup_pending = sum(1 for item in crawl_setup_plugins if item.get("status") == "queued")
-            crawl_screencast_url = screencast_frame_url(crawl_id, active_crawl_objects[crawl_id].output_dir)
+            crawl_screencast_url = screencast_frame_url(crawl_id)
            crawl_screencast_link = f"/admin/crawls/crawl/{crawl_id.replace('-', '')}/change/" if crawl_screencast_url else ""

            # Get active snapshots for this crawl (already prefetched)
@@ -661,7 +643,7 @@ def live_progress_view(request, *, authorized_snapshot=None):
                    snapshot_preview_url = snapshot_favicon_url

                if snapshot["status"] == Snapshot.StatusChoices.STARTED:
-                    snapshot_screencast_url = screencast_frame_url(crawl_id, active_crawl_objects[crawl_id].output_dir)
+                    snapshot_screencast_url = screencast_frame_url(crawl_id)
                    snapshot_screencast_link = snapshot_view_url(snapshot) if snapshot_screencast_url else ""

                def plugin_sort_key(ar):
@@ -809,18 +791,18 @@ def live_progress_view(request, *, authorized_snapshot=None):
            persona_details = persona_details_by_id.get(str(crawl["persona_id"])) if crawl["persona_id"] else None
            persona_name = persona_details["name"] if persona_details else "Default"
            persona_details = persona_details or persona_details_by_name.get(persona_name)
-            crawl_output_size = crawl_output_sizes_by_crawl.get(crawl_id, 0)
-            avg_snapshot_size = int(crawl_output_size / completed_snapshots) if completed_snapshots else 0
-            crawl_obj = active_crawl_objects[crawl_id]
-            effective_crawl_config = get_config(crawl=crawl_obj, resolve_plugins=False)
-            max_urls = int(effective_crawl_config.CRAWL_MAX_URLS or 0)
-            crawl_max_size = int(effective_crawl_config.CRAWL_MAX_SIZE or 0)
-            crawl_timeout = int(effective_crawl_config.CRAWL_TIMEOUT or 0)
-            snapshot_max_size = int(effective_crawl_config.SNAPSHOT_MAX_SIZE or 0)
+            crawl_output_size = totals["bytes"] if totals is not None else None
+            avg_snapshot_size = int(totals["bytes"] / max(totals["snapshots"], 1)) if totals is not None else None
+            # These crawl-scope limits are frozen in Crawl.config. Rendering
+            # them must not resolve personas, probe binaries, or reread config.
+            max_urls = int(crawl["config"].get("CRAWL_MAX_URLS", request_config.CRAWL_MAX_URLS) or 0)
+            crawl_max_size = int(crawl["config"].get("CRAWL_MAX_SIZE", request_config.CRAWL_MAX_SIZE) or 0)
+            crawl_timeout = int(crawl["config"].get("CRAWL_TIMEOUT", request_config.CRAWL_TIMEOUT) or 0)
+            snapshot_max_size = int(crawl["config"].get("SNAPSHOT_MAX_SIZE", request_config.SNAPSHOT_MAX_SIZE) or 0)

            # Check if retry_at is in the future (would prevent worker from claiming)
            retry_at_future = crawl["retry_at"] > now if crawl["retry_at"] else False
-            is_paused = crawl_obj.is_paused
+            is_paused = crawl["status"] == Crawl.StatusChoices.PAUSED
            seconds_until_retry = (
                0 if is_paused else int((crawl["retry_at"] - now).total_seconds()) if crawl["retry_at"] and retry_at_future else 0
            )
@@ -860,8 +842,8 @@ def live_progress_view(request, *, authorized_snapshot=None):
                    "max_snapshot_size_display": printable_filesize(snapshot_max_size) if snapshot_max_size else "unlimited",
                    "crawl_output_size": crawl_output_size,
                    "avg_snapshot_size": avg_snapshot_size,
-                    "crawl_output_size_display": printable_filesize(crawl_output_size) if crawl_output_size else "0 B",
-                    "avg_snapshot_size_display": printable_filesize(avg_snapshot_size) if avg_snapshot_size else "0 B",
+                    "crawl_output_size_display": printable_filesize(crawl_output_size or 0) if totals is not None else "…",
+                    "avg_snapshot_size_display": printable_filesize(avg_snapshot_size or 0) if totals is not None else "…",
                    "tags": crawl_tags,
                    "urls_count": urls_count,
                    "total_snapshots": total_snapshots,
@@ -12,8 +12,11 @@ _search_backends_cache: dict | None = None

 def search_backend_command_env(config: dict[str, Any] | None = None, **config_kwargs: Any) -> dict[str, str]:
    """Serialize resolved application config for a standalone plugin command."""
+    from abx_dl.models import PluginEnv
+    from abxpkg import BinProvider
+
    config = config or get_config(**config_kwargs)
-    env = os.environ.copy()
+    env: dict[str, str] = {}
    for key, value in config.items():
        key = str(key)
        if value is None:
@@ -24,7 +27,9 @@ def search_backend_command_env(config: dict[str, Any] | None = None, **config_kw
            env[key] = json.dumps(value)
        elif isinstance(value, (str, int, float, os.PathLike)):
            env[key] = str(value)
-    return env
+    # Use the same runtime bootstrap as hooks so the installation's abxpkg
+    # entrypoint remains available even without an activated virtualenv.
+    return BinProvider.build_exec_env(base_env=PluginEnv(**os.environ).to_env(), extra_env=env)


 def normalize_search_backend_name(backend_name: str | None) -> str:
@@ -2,9 +2,14 @@

 import math
 import os
+import re
+import stat
+import sys
 import time
 from pathlib import Path

+import psutil
+

 RESOURCE_RECHECK_SECONDS = 2.0
 # Scheduling policy, not a minimum-RAM claim: stop increasing extraction
@@ -12,6 +17,65 @@ RESOURCE_RECHECK_SECONDS = 2.0
 # stalls. Require both recent and rolling pressure so transient reclaim does
 # not indefinitely starve otherwise viable work on small hosts.
 MEMORY_FULL_STALL_PERCENT = 10.0
+ZRAM_DEVICE_NAME = re.compile(r"zram\d+")
+
+
+def zram_swap_identifiers(sys_block: Path = Path("/sys/block")) -> set[str]:
+    """Return zram block names and major:minor IDs known to sysfs."""
+    identifiers = set()
+    for device in sys_block.glob("zram*"):
+        if ZRAM_DEVICE_NAME.fullmatch(device.name):
+            identifiers.add(device.name)
+            try:
+                identifiers.add((device / "dev").read_text().strip())
+            except OSError:
+                pass
+    return identifiers
+
+
+def swap_source_identifiers(filename: str) -> set[str]:
+    """Identify a swap source by its visible name and, for block devices, ID."""
+    identifiers = {Path(filename).name}
+    try:
+        source = os.stat(filename)
+        if stat.S_ISBLK(source.st_mode):
+            identifiers.add(f"{os.major(source.st_rdev)}:{os.minor(source.st_rdev)}")
+    except OSError:
+        pass
+    return identifiers
+
+
+def disk_backed_swap_free_bytes(swaps: str, zram_identifiers: set[str]) -> int:
+    """Sum free active swap except zram, whose capacity is already backed by RAM."""
+    lines = swaps.splitlines()
+    free_kib = 0
+    for line in lines[1:]:  # /proc/swaps starts with a column header
+        fields = line.split()
+        if len(fields) < 5:
+            continue
+        filename, _, size_kib, used_kib, _ = fields[:5]
+        source_ids = swap_source_identifiers(filename)
+        if source_ids & zram_identifiers or any(ZRAM_DEVICE_NAME.fullmatch(name) for name in source_ids):
+            continue
+        try:
+            free_kib += max(0, int(size_kib) - int(used_kib))
+        except ValueError:
+            continue
+    return free_kib * 1024
+
+
+def host_swap_free_bytes(meminfo_swap_free: int) -> int:
+    """Use only independently backed swap as extra capacity on Linux."""
+    if not sys.platform.startswith("linux"):
+        return psutil.swap_memory().free
+    try:
+        swaps = Path("/proc/swaps").read_text()
+    except OSError:
+        # If Linux swap sources cannot be classified, fail closed rather than
+        # treating potentially RAM-backed zram capacity as extra headroom.
+        return 0
+    disk_free = disk_backed_swap_free_bytes(swaps, zram_swap_identifiers())
+    return min(max(0, meminfo_swap_free), disk_free)


 class ResourceAdmission:
@@ -44,10 +108,73 @@ class ResourceAdmission:
                    cpus = min(cpus, max(1, math.ceil(int(quota) / int(period))))
            except (OSError, ValueError, ZeroDivisionError):
                continue
-        # CPU quota limits parallel admission, not forward progress. A host
-        # with a fractional CPU must still be able to run one snapshot.
+        # CPU quota bounds parallel work; measured memory admission is done
+        # only after the first complete snapshot establishes its actual cost.
        return max(1, min(configured, cpus))

+    def additional_snapshot_slots(self, configured: int, active: int, observed_cost: int | None) -> int:
+        if not observed_cost:
+            # Stage one complete real capture before spending memory on parallel
+            # hook trees. A zero-cost/failed probe cannot justify fanout.
+            return 1 if active == 0 and configured > 0 else 0
+        headroom = self.memory_headroom()
+        return self.slots_for_headroom(configured, active, observed_cost, headroom[1] if headroom else 0)
+
+    @staticmethod
+    def slots_for_headroom(configured: int, active: int, observed_cost: int, available: int) -> int:
+        slots = max(0, configured - active)
+        if observed_cost > 0:
+            # Active captures can still grow to the measured peak. Reserve for
+            # that growth before promising the remaining headroom to new work.
+            unpromised = max(0, available - active * observed_cost)
+            slots = min(slots, unpromised // observed_cost)
+        return max(1, slots) if active == 0 and configured else slots
+
+    def memory_headroom(self) -> tuple[int, int] | None:
+        """Return workload RAM+swap usage and effective available headroom."""
+        try:
+            meminfo = {
+                name: int(value.split()[0]) * 1024
+                for name, value in (line.split(":", 1) for line in Path("/proc/meminfo").read_text().splitlines() if ":" in line)
+            }
+            host_swap_free = host_swap_free_bytes(meminfo.get("SwapFree", 0))
+            host_available = meminfo["MemAvailable"] + host_swap_free
+        except (OSError, ValueError, KeyError):
+            host_swap_free = 0 if sys.platform.startswith("linux") else psutil.swap_memory().free
+            host_available = psutil.virtual_memory().available + host_swap_free
+
+        available = host_available
+        used_bytes = None
+        for path in self.cgroup_paths():
+            try:
+                memory_max = (path / "memory.max").read_text().strip()
+                if memory_max == "max":
+                    continue
+                current = int((path / "memory.current").read_text())
+                swap_max = (path / "memory.swap.max").read_text().strip()
+                swap_current = int((path / "memory.swap.current").read_text())
+                swap_room = host_swap_free if swap_max == "max" else min(host_swap_free, max(0, int(swap_max) - swap_current))
+                available = min(available, max(0, int(memory_max) - current) + swap_room)
+                if used_bytes is None:
+                    used_bytes = current + swap_current
+            except (OSError, ValueError, KeyError):
+                continue
+        if used_bytes is None:
+            # Bare-metal and non-Linux installs have no finite cgroup. Count
+            # the real runner process tree rather than the entire host.
+            try:
+                process = psutil.Process()
+                processes = [process, *process.children(recursive=True)]
+            except (psutil.Error, OSError):
+                return None
+            used_bytes = 0
+            for child in processes:
+                try:
+                    used_bytes += child.memory_info().rss
+                except (psutil.Error, OSError):
+                    continue
+        return used_bytes, available
+
    def observe_memory_stalls(self, source: str, total: int, avg10: float, sampled_at: float) -> bool:
        previous = self._memory_stalls.get(source)
        self._memory_stalls[source] = (total, sampled_at)
@@ -3,6 +3,7 @@ from __future__ import annotations
 import asyncio
 import contextvars
 import json
+import logging
 import os
 import signal
 import shutil
@@ -223,6 +224,7 @@ class CrawlRunner:
        self.snapshot_service = SnapshotService(
            self.bus,
            crawl_id=str(crawl.id),
+            interrupted=self.execution_interrupted,
        )
        HookArchiveResultService(self.bus, emit_jsonl=False)
        ArchiveResultService(self.bus)
@@ -232,6 +234,10 @@ class CrawlRunner:
        self.snapshot_tasks: dict[str, asyncio.Task[None]] = {}
        self.snapshot_semaphore = asyncio.Semaphore(1)
        self.max_concurrent_snapshots = 1
+        self._warmup_snapshot_id: str | None = None
+        self._memory_baseline: int | None = None
+        self._memory_peak_cost = 0
+        self._observed_snapshot_cost: int | None = None
        self._resource_deferred = False
        self._initial_snapshot_admitted = False
        self.persona = None
@@ -253,6 +259,10 @@ class CrawlRunner:
        self._signal_abort_requested = False
        self._last_lease_heartbeat_at = 0.0

+    def execution_interrupted(self) -> bool:
+        """Owner loss preserves work; explicit user abort keeps its sealing policy."""
+        return self._signal_abort_requested and not self._user_aborted
+
    async def on_CrawlAbortEvent(self, event: CrawlAbortEvent) -> None:
        self._signal_abort_requested = True
        # An abort can arrive while the prompt transport is being established,
@@ -498,10 +508,15 @@ class CrawlRunner:
        if task is not None and not task.done():
            return
        current_event = crawl_start_event or get_current_event()
+        if not isinstance(current_event, CrawlStartEvent) and in_handler_context():
+            return
+        if self._observed_snapshot_cost is None and self._warmup_snapshot_id is None:
+            observation = resource_admission.memory_headroom()
+            self._memory_baseline = observation[0] if observation is not None else None
+            self._memory_peak_cost = 0
+            self._warmup_snapshot_id = snapshot_id
        if isinstance(current_event, CrawlStartEvent):
            task = asyncio.create_task(self.run_snapshot(snapshot_id, current_event), context=_runner_task_context())
-        elif in_handler_context():
-            return
        else:
            task = asyncio.create_task(self.run_snapshot(snapshot_id), context=_runner_task_context())
        self._initial_snapshot_admitted = False
@@ -530,6 +545,7 @@ class CrawlRunner:
                if task.done():
                    if self.snapshot_tasks.get(snapshot_id) is task:
                        self.snapshot_tasks.pop(snapshot_id, None)
+                    self._finish_warmup_snapshot(snapshot_id)
                    try:
                        task.result()
                    except asyncio.CancelledError as err:
@@ -554,13 +570,16 @@ class CrawlRunner:
                    return
                continue
            await self.heartbeat_active_leases()
-            done, _pending = await asyncio.wait(pending_tasks, timeout=10.0, return_when=asyncio.FIRST_COMPLETED)
+            self._observe_snapshot_memory()
+            done, _pending = await asyncio.wait(pending_tasks, timeout=RESOURCE_RECHECK_SECONDS, return_when=asyncio.FIRST_COMPLETED)
+            self._observe_snapshot_memory()
            if not done:
                continue
            for task in done:
                for snapshot_id, tracked_task in list(self.snapshot_tasks.items()):
                    if tracked_task is task:
                        self.snapshot_tasks.pop(snapshot_id, None)
+                        self._finish_warmup_snapshot(snapshot_id)
                        break
                try:
                    task.result()
@@ -578,6 +597,41 @@ class CrawlRunner:
            if not stop_scheduling and enqueue_projected:
                await self.enqueue_pending_snapshots_from_projection()

+    def _observe_snapshot_memory(self) -> None:
+        if self._memory_baseline is None:
+            return
+        observation = resource_admission.memory_headroom()
+        if observation is not None:
+            used, _available = observation
+            # Only attribute this workload's growth to a capture. Other jobs on
+            # the host reduce available headroom, but are not a per-capture cost
+            # to multiply again for every snapshot we want to admit.
+            self._memory_peak_cost = max(
+                self._memory_peak_cost,
+                used - self._memory_baseline,
+            )
+            if self._observed_snapshot_cost is not None:
+                self._observed_snapshot_cost = max(self._observed_snapshot_cost, self._memory_peak_cost)
+
+    def _finish_warmup_snapshot(self, snapshot_id: str) -> None:
+        if snapshot_id != self._warmup_snapshot_id or self._observed_snapshot_cost is not None:
+            return
+        self._observe_snapshot_memory()
+        if self._memory_peak_cost > 0:
+            self._observed_snapshot_cost = self._memory_peak_cost
+            headroom = resource_admission.memory_headroom()
+            logging.getLogger(__name__).info(
+                "Snapshot admission: concurrency limit=%s, observed memory growth=%s bytes, available=%s bytes",
+                self.max_concurrent_snapshots,
+                self._observed_snapshot_cost,
+                headroom[1] if headroom else None,
+            )
+        else:
+            # No observed cost cannot justify parallel admission. Probe the
+            # next real snapshot while the queue keeps moving one at a time.
+            self._memory_baseline = None
+        self._warmup_snapshot_id = None
+
    async def heartbeat_active_leases(self) -> None:
        # These are resumable work-item leases, not orchestrator-election
        # heartbeats. Each update is a short autocommit statement; network and
@@ -645,7 +699,11 @@ class CrawlRunner:
        self.max_concurrent_snapshots = resource_admission.snapshot_limit(int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))

        active_snapshot_ids = [snapshot_id for snapshot_id, task in self.snapshot_tasks.items() if not task.done()]
-        available_slots = max(0, self.max_concurrent_snapshots - len(active_snapshot_ids))
+        available_slots = resource_admission.additional_snapshot_slots(
+            self.max_concurrent_snapshots,
+            len(active_snapshot_ids),
+            self._observed_snapshot_cost,
+        )
        if available_slots <= 0:
            return
        pending_snapshot_ids = await sync_to_async(
@@ -669,6 +727,9 @@ class CrawlRunner:
        from archivebox.machine.models import Machine, NetworkInterface, Process

        self.primary_url = self.crawl.get_urls_list()[0] if self.crawl.get_urls_list() else ""
+        # Validate saved binary paths at the execution boundary, including
+        # config edited since this process first registered its Machine.
+        machine = Machine._sanitize_config(Machine.current())
        current_iface = NetworkInterface.current(refresh=not self.allow_maintenance_on_inactive_crawl)
        current_process = Process.current()
        if current_process.iface_id != current_iface.id or current_process.machine_id != current_iface.machine_id:
@@ -677,7 +738,7 @@ class CrawlRunner:
            current_process.save(update_fields=["iface", "machine", "modified_at"])
        self.persona = self.crawl.resolve_persona()
        self.base_config = get_config(crawl=self.crawl, overrides=self.config_overrides)
-        self.derived_config = dict(Machine.current().config or {})
+        self.derived_config = dict(machine.config or {})
        self.crawl_output_dir = str(self.crawl.output_dir)
        if self.persona:
            self.base_config.update(
@@ -997,7 +1058,8 @@ class CrawlRunner:
            if self.initial_snapshot_ids is not None:
                remaining_snapshot_ids = iter(snapshot_ids)
                while True:
-                    batch = list(zip(range(self.max_concurrent_snapshots), remaining_snapshot_ids, strict=False))
+                    slots = resource_admission.additional_snapshot_slots(self.max_concurrent_snapshots, 0, self._observed_snapshot_cost)
+                    batch = list(zip(range(slots), remaining_snapshot_ids, strict=False))
                    if not batch:
                        return
                    for _slot, snapshot_id in batch:
@@ -1008,7 +1070,8 @@ class CrawlRunner:
                        await self.enqueue_snapshot(snapshot_id)
                    await self.wait_for_snapshot_tasks(enqueue_projected=False)
            else:
-                for snapshot_id in snapshot_ids[: self.max_concurrent_snapshots]:
+                slots = resource_admission.additional_snapshot_slots(self.max_concurrent_snapshots, 0, self._observed_snapshot_cost)
+                for snapshot_id in snapshot_ids[:slots]:
                    if await self.crawl_is_cancelled():
                        break
                    if await self.crawl_is_paused():
@@ -1073,18 +1136,17 @@ class CrawlRunner:
            finally:
                cancel_watcher.cancel()
                await asyncio.gather(cancel_watcher, return_exceptions=True)
-            completed_event = event.emit(
-                CrawlCompletedEvent(
-                    url=snapshot["url"],
-                    snapshot_id=snapshot["id"],
-                    output_dir=str(output_dir),
-                ),
-            )
-            # Same signal lifecycle as CrawlCleanupEvent above: completion is a
-            # normal bus event unless the interpreter is already unwinding from
-            # SIGINT/SIGTERM/SIGHUP, where synchronous bus delivery is no
-            # longer a dependable shutdown primitive.
-            if not self._signal_abort_requested:
+            # emit() schedules handlers immediately; guarding only the await
+            # still lets completion projectors run during takeover. An aborted
+            # execution owner must leave durable work for its replacement.
+            if not self.execution_interrupted():
+                completed_event = event.emit(
+                    CrawlCompletedEvent(
+                        url=snapshot["url"],
+                        snapshot_id=snapshot["id"],
+                        output_dir=str(output_dir),
+                    ),
+                )
                await _run_event_now(completed_event, CrawlCompletedEvent.model_fields["event_timeout"].default)

        on_archivebox_CrawlStartEvent.__name__ = "on_archivebox_CrawlStartEvent__run_snapshots"
@@ -1221,6 +1283,8 @@ class CrawlRunner:
                    raise RuntimeError(f"Snapshot {snapshot_id} did not complete")
                await completed_snapshot.wait(timeout=snapshot_phase_timeout)
                await completed_snapshot.event_results_list()
+                if self.execution_interrupted():
+                    return
                if snapshot["status"] == "sealed":
                    await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
                    return
@@ -1471,7 +1535,9 @@ def _run_due_crawl_locked(crawl, *, lock_seconds: int, interactive_interrupts: b
            # avoids hot-looping on the parent while child work is ready without
            # resurrecting a user cancellation that sealed the crawl after
            # selection.
-            crawl.safe_update(
+            if crawl.status != crawl.StatusChoices.STARTED:
+                return False
+            return crawl.safe_update(
                {
                    "status": crawl.StatusChoices.STARTED,
                    "retry_at": now + timedelta(seconds=ACTIVE_STATE_LEASE_SECONDS),
@@ -1480,7 +1546,6 @@ def _run_due_crawl_locked(crawl, *, lock_seconds: int, interactive_interrupts: b
                refresh=False,
                extra_filter={"status": crawl.StatusChoices.STARTED},
            )
-            return True
        if snapshot_count and not due_active_snapshots:
            if crawl.is_finished():
                if not crawl.claim_processing_lock(lock_seconds=lock_seconds):
@@ -1838,12 +1903,11 @@ def _run_due_crawl_status(
    due_crawl = Crawl.objects.filter(id=due_crawl_id).first()
    if due_crawl is None:
        return True
-    run_due_crawl(
+    return run_due_crawl(
        due_crawl,
        lock_seconds=lock_seconds,
        interactive_interrupts=interactive_interrupts,
    )
-    return True


 def _run_due_snapshot_query(
@@ -1897,8 +1961,7 @@ def _run_due_binary() -> bool:
    due_binary = Binary.objects.filter(id=due_binary_id).first()
    if due_binary is None:
        return True
-    run_due_binary(due_binary, lock_seconds=60)
-    return True
+    return run_due_binary(due_binary, lock_seconds=60)


 def run_pending_crawls(
@@ -1,6 +1,7 @@
 from __future__ import annotations

 import asyncio
+from collections.abc import Callable
 from pathlib import Path

 from asgiref.sync import sync_to_async
@@ -73,10 +74,11 @@ def finalize_completed_snapshot(
            modified_at=timezone.now(),
        )

-    # SnapshotCompletedEvent is abx-dl's authoritative signal that the complete
-    # snapshot hook sequence (including cleanup) finished. ArchiveResult rows
-    # are projections of that work, never prerequisites used to decide whether
-    # the Snapshot may seal.
+    # SnapshotCompletedEvent finishes an attempt, including abort cleanup. An
+    # explicit user abort deliberately seals that point-in-time capture; the
+    # runner's interrupted-owner guard below keeps takeover work resumable.
+    # ArchiveResult rows are projections of that work, never prerequisites used
+    # to decide whether the Snapshot may seal.
    if not was_sealed and snapshot.status == Snapshot.StatusChoices.STARTED and snapshot.retry_at == owned_retry_at:
        snapshot.seal()
        snapshot.refresh_from_db()
@@ -120,8 +122,9 @@ class SnapshotService(BaseService):
    LISTENS_TO = [SnapshotEvent, SnapshotCompletedEvent]
    EMITS = []

-    def __init__(self, bus, *, crawl_id: str):
+    def __init__(self, bus, *, crawl_id: str, interrupted: Callable[[], bool] | None = None):
        self.crawl_id = crawl_id
+        self.interrupted = interrupted
        self._run_ownership: dict[str, tuple[str, object, bool, list[str]]] = {}
        self._ownership_lock = asyncio.Lock()
        super().__init__(bus)
@@ -175,6 +178,15 @@ class SnapshotService(BaseService):
            if ownership is None or ownership[0] != str(event.event_parent_id):
                return
            self._run_ownership.pop(snapshot_id, None)
+        # abx-dl also emits completion after abort cleanup, even before the
+        # first hook. Explicit user aborts seal a point-in-time capture so later
+        # work cannot silently mix observations taken days apart; reopening it
+        # is a separate user action. Takeover only replaces its execution owner.
+        # Consult the runner's synchronous signal state: an exiting owner cannot
+        # rely on another bus event arriving before this completion projector runs.
+        # Leave its durable lease for the next elected runner's startup recovery.
+        if self.interrupted is not None and self.interrupted():
+            return
        _, owned_retry_at, was_sealed, retry_plugins = ownership
        await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(
            event.snapshot_id,
@@ -1527,6 +1527,9 @@
                display: inline-flex;
                flex-wrap: wrap;
                align-items: center;
+                min-width: 0;
+                max-width: 100%;
+                box-sizing: border-box;
                gap: 4px;
                padding: 2px 4px;
                background: transparent;
@@ -1540,15 +1543,27 @@
                flex-wrap: wrap;
                gap: 4px;
                align-items: center;
+                min-width: 0;
+                max-width: 100%;
            }

            .tag-editor-inline .tag-pill {
+                min-width: 0;
+                max-width: min(160px, 100%);
+                box-sizing: border-box;
                padding: 2px 6px 2px 8px;
                font-size: 11px;
                border-radius: 12px;
            }

+            .tag-editor-inline .tag-link {
+                min-width: 0;
+                overflow: hidden;
+                text-overflow: ellipsis;
+            }
+
            .tag-editor-inline .tag-remove-btn {
+                flex-shrink: 0;
                width: 14px;
                height: 14px;
                font-size: 12px;
@@ -79,6 +79,13 @@
  }
  .cards .card .card-tags span {
      display: inline-block;
+      max-width: min(160px, 100%);
+      box-sizing: border-box;
+      overflow: hidden;
+      text-overflow: ellipsis;
+      white-space: nowrap;
+      vertical-align: middle;
+      pointer-events: auto;
      padding: 2px 5px;
      border-radius: 5px;
      opacity: 0.95;
@@ -283,16 +290,16 @@
          <div class="card-media">
            <a href="{% snapshot_detail_url obj %}" class="card-thumbnail">
              {% snapshot_thumbnail obj admin=True width="100%" height="245px" %}
-            </a>
              {% if obj.tags_str %}
                <div class="card-tags">
                  {% for tag in obj.tags_str|split:',' %}
-                    <span>
+                    <span title="{{tag}}">
                      {{tag}}
                    </span>
                  {% endfor %}
                </div>
              {% endif %}
+            </a>
          </div>
          <div class="card-outputs">
            {% if obj.status == 'started' %}
@@ -198,21 +198,29 @@
    }

    .snapshot-preview-cell {
+        position: relative;
        width: 110px;
        min-width: 116px;
        vertical-align: middle;
    }

    .snapshot-preview-cell a {
+        position: absolute;
+        top: 10px;
+        bottom: 10px;
        display: inline-flex;
        align-items: center;
        justify-content: center;
        width: 102px;
-        height: 102px;
        line-height: 0;
        vertical-align: middle;
    }

+    .snapshot-preview-cell .snapshot-thumbnail {
+        width: 100% !important;
+        height: 100% !important;
+    }
+
    #content .public-snapshot-list .snapshot-preview-cell .snapshot-preview,
    #content .public-snapshot-list .snapshot-preview-cell img.snapshot-preview.screenshot {
        box-sizing: border-box;
@@ -330,9 +338,10 @@
    }

    .snapshot-tag {
-        display: inline-flex;
-        align-items: center;
-        max-width: 160px;
+        display: block;
+        min-width: 0;
+        max-width: min(160px, 100%);
+        box-sizing: border-box;
        padding: 2px 7px;
        border: 1px solid #bfdbfe;
        border-radius: 999px;
@@ -585,7 +594,8 @@

        .snapshot-preview-cell a {
            width: 82px;
-            height: 82px;
+            top: 7px;
+            bottom: 7px;
        }

        #content .public-snapshot-list .snapshot-preview-cell .snapshot-preview,
@@ -644,12 +654,19 @@
                </span>
                <input type="submit" value="Search">
            </form>
+            {% comment %}
+                Missing/stale totals are intentional on large collections: show rows
+                immediately and let a later normal reload pick up the cached count.
+                Do not fill this in with an extra count request or paginator.count:
+                the unfiltered paginator only knows the rows needed for navigation.
+                Use the optional public_* totals here and in the footer below.
+            {% endcomment %}
            <div class="public-snapshot-count">
                {% if public_search_stream_pending %}
                    <span class="archivebox-search-stream-status" role="status" aria-live="polite">Searching...</span>
                {% else %}
-                    {{ page_obj.start_index }}-{{ page_obj.end_index }} of {{ page_obj.paginator.count }}
-                    <span>(page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }})</span>
+                    {{ page_obj.start_index }}-{{ page_obj.end_index }}{% if public_snapshot_count != None %} of {{ public_snapshot_count }}{% endif %}
+                    <span>(page {{ page_obj.number }}{% if public_page_count != None %} of {{ public_page_count }}{% endif %})</span>
                {% endif %}
            </div>
        </div>
@@ -659,7 +676,7 @@
                    <tr>
                        <th class="snapshot-time">Saved</th>
                        <th class="snapshot-preview-cell">Preview</th>
-                        <th>Snapshot ({{ page_obj.paginator.count }})</th>
+                        <th>Snapshot{% if public_snapshot_count != None %} ({{ public_snapshot_count }}){% endif %}</th>
                        <th class="snapshot-tags-cell">Tags</th>
                        <th class="snapshot-status-cell">Status</th>
                        <th class="snapshot-files-cell">Files</th>
@@ -687,7 +704,7 @@
            </table>
        </div>
        <div class="public-pagination">
-            Showing {{ page_obj.start_index }}-{{ page_obj.end_index }} of {{ page_obj.paginator.count }} total
+            Showing {{ page_obj.start_index }}-{{ page_obj.end_index }}{% if public_snapshot_count != None %} of {{ public_snapshot_count }} total{% endif %}
            <br>
            <span class="step-links">
                {% if page_obj.has_previous %}
@@ -696,14 +713,14 @@
                {% endif %}

                <span class="current">
-                    Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}
+                    Page {{ page_obj.number }}{% if public_page_count != None %} of {{ public_page_count }}{% endif %}
                </span>

                {% if page_obj.has_next %}
                    <a href="?{% url_replace page=page_obj.next_page_number %}">next</a>
                {% endif %}
-                {% if page_obj.has_next %}
-                    <a href="?{% url_replace page=page_obj.paginator.num_pages %}">last &raquo;</a>
+                {% if page_obj.has_next and public_page_count != None %}
+                    <a href="?{% url_replace page=public_page_count %}">last &raquo;</a>
                {% endif %}
            </span>
        </div>
@@ -1671,6 +1671,13 @@
                    return false
                }

+                // A new snapshot has only the synthetic "Browse all files" card.
+                // Selecting it would replace the live monitor with an empty index.
+                // Keep progress/empty state until the user explicitly opens a file.
+                if (document.getElementById('main-frame-wrapper')?.dataset.hasOutputs !== '1') {
+                    return false
+                }
+
                const selectedCard = document.querySelector('.thumb-card.selected-card') || document.querySelector('.thumb-card')
                const selectedLink = selectedCard && (selectedCard.querySelector('a[target=preview]') || selectedCard.querySelector('a'))
                if (selectedCard && selectedLink) {
@@ -6,7 +6,7 @@
                        <div class="thumb-card{% if forloop.first %} selected-card{% endif %}" data-plugin-name="{{result.name|plugin_name}}" data-output-group="{{result.output_group}}" data-output-size="{{result.size|default:0}}"{% if preview_url %} data-preview-url="{{preview_url}}"{% endif %}{% if display_path %} data-output-path="{{display_path}}"{% endif %}>
                                <div class="thumb-body">
                                    <div class="thumb-actions">
-                                        {% if not STATIC_EXPORT %}<a href="{% snapshot_files_url snapshot result.folder_path|default:result.name %}" data-no-preview="1" title="Open output folder" target="_blank" rel="noopener">📁</a>{% endif %}
+                                        {% if not STATIC_EXPORT %}<a href="{% snapshot_files_url snapshot result.folder_path %}" data-no-preview="1" title="Open output folder" target="_blank" rel="noopener">📁</a>{% endif %}
                                        {% if display_path %}
                                            <a href="{{display_url}}" data-no-preview="1" title="Download output file" download>⬇️</a>
                                        {% endif %}
@@ -85,6 +85,7 @@
            link.href = '/admin/core/snapshot/?tags__id__exact=' + td.id;
            link.className = 'tag-link';
            link.textContent = td.name;
+            link.title = td.name;
            pill.appendChild(link);

            var removeBtn = document.createElement('button');
@@ -1155,7 +1155,19 @@ body.change-list:not(.model-snapshot) #changelist #toolbar,
    opacity: 0.75;
 }

+#result_list tbody td.field-preview_icon {
+    position: relative;
+}
+
+#result_list tbody td.field-preview_icon > a {
+    position: absolute;
+    inset: 4px 2px;
+    display: block;
+}
+
 #result_list tbody td.field-preview_icon .snapshot-thumbnail {
+    width: 100% !important;
+    height: 100% !important;
    transition: box-shadow 140ms ease, transform 140ms ease;
 }

@@ -2226,7 +2238,7 @@ tbody .output-link:hover {opacity: 1;}
    .model-snapshot.change-list #result_list tbody th.field-created_at_display > a,
    .model-snapshot.change-list #result_list tbody td.field-created_at_display > a {
        display: inline-flex;
-        max-width: none;
+        max-width: 100%;
        white-space: nowrap !important;
    }

@@ -2234,8 +2246,10 @@ tbody .output-link:hover {opacity: 1;}
    .model-snapshot.change-list #result_list tbody td.field-created_at_display .snapshot-created-at {
        display: inline-flex;
        flex-direction: row !important;
+        flex-wrap: wrap;
        justify-content: center;
        gap: 5px;
+        row-gap: 0;
        white-space: nowrap !important;
    }

@@ -2250,13 +2264,10 @@ tbody .output-link:hover {opacity: 1;}
        grid-area: preview;
        align-self: stretch;
        place-items: center;
-        min-height: 110px;
        overflow: hidden;
    }

    .model-snapshot.change-list #result_list tbody td.field-preview_icon .snapshot-thumbnail {
-        width: 80px !important;
-        height: 110px !important;
        max-width: 100%;
        box-sizing: border-box;
    }
@@ -2278,7 +2289,7 @@ tbody .output-link:hover {opacity: 1;}
        padding-left: 0;
    }

-    .model-snapshot.change-list #result_list tbody td.field-title_str a {
+    .model-snapshot.change-list #result_list tbody td.field-title_str > a {
        display: block;
        max-width: 100%;
        overflow-wrap: anywhere;
@@ -2514,26 +2525,12 @@ tbody .output-link:hover {opacity: 1;}
    }

    .model-snapshot.change-list #result_list tbody tr {
-        grid-template-columns: 24px 84px minmax(0, 1fr);
        gap: 2px 4px;
-        grid-template-areas:
-            "check preview date"
-            "check preview title"
-            "check preview tags"
-            "check preview status"
-            "check preview files"
-            "check preview size";
    }

    .model-snapshot.change-list #result_list tbody td.field-files .files-icons {
        gap: 0;
    }
-
-    .model-snapshot.change-list #result_list tbody td.field-status_with_progress,
-    .model-snapshot.change-list #result_list tbody td.field-size_with_stats {
-        justify-self: start;
-        text-align: left;
-    }
 }

 body.embedded-change-list #content {
@@ -1010,11 +1010,15 @@ def kill_processes_for_data_dir(data_dir: Path) -> None:
            continue
        if pid != os.getpid():
            try:
-                os.kill(pid, signal.SIGTERM)
-            except ProcessLookupError:
+                # Retain the process identity before signalling: a fast, clean
+                # exit can remove the PID before a subsequent Process lookup.
+                # terminate() also checks that the PID has not been reused.
+                process = psutil.Process(pid)
+                process.terminate()
+            except psutil.NoSuchProcess:
                pass
            else:
-                processes.append(psutil.Process(pid))
+                processes.append(process)
    _gone, alive = psutil.wait_procs(processes, timeout=10)
    assert not alive, f"processes did not stop after SIGTERM: {[proc.pid for proc in alive]}"

@@ -1659,6 +1663,8 @@ def resolve_abxpkg_binary_env(
    deps_from: Path | list[Path] | tuple[Path, ...] | None = None,
 ) -> dict[str, str]:
    """Resolve required test dependencies through abxpkg's normal install path."""
+    from abxpkg.config import apply_exec_env
+
    command_env = dict(env) if env is not None else os.environ.copy()
    command_env["ABXPKG_LIB_DIR"] = str(lib_dir)
    command = [
@@ -1679,7 +1685,11 @@ def resolve_abxpkg_binary_env(
    )
    assert result.returncode == 0, result.stderr or result.stdout
    payload = json.loads(result.stdout)
-    return {str(key): str(value) for key, value in payload.items()}
+    # abxpkg env emits prepend/append deltas, not a replacement environment.
+    # Applying a PATH ending in ':' verbatim hides the caller's abxpkg CLI.
+    delta = {str(key): str(value) for key, value in payload.items()}
+    apply_exec_env(delta, command_env)
+    return {key: command_env[key] for key in delta}


 def run_test_hook(
@@ -1777,7 +1787,7 @@ def install_real_binary(
    from archivebox.machine.models import Binary, Machine

    binary = Binary.objects.create(
-        machine=machine or Machine.current(refresh=True),
+        machine=machine or Machine.current(),
        name=name,
        binproviders=binproviders,
        overrides=overrides or {},
@@ -1,4 +1,6 @@
 import pytest
+import os
+import time

 from archivebox.crawls.models import Crawl

@@ -15,3 +17,19 @@ def test_basic_success_case_request(client, tmp_path, api_admin_user, api_header
    response = client.get(f"/api/v1/crawls/crawl/{crawl.id}/files/folder/basic.txt", **api_headers)

    assert response.status_code == 200, response.content
+
+
+def test_screencast_frame_is_checked_by_image_request(client, api_admin_user, api_headers):
+    crawl = Crawl.objects.create(urls="https://example.com/screencast", created_by=api_admin_user)
+    frame = crawl.output_dir / "chrome_screencast" / "latest.jpg"
+    frame.parent.mkdir(parents=True, exist_ok=True)
+    frame.write_bytes(b"stored frame")
+    started_at = time.time()
+    os.utime(frame, (started_at - 10, started_at - 10))
+    url = f"/api/v1/crawls/crawl/{crawl.id}/files/chrome_screencast/latest.jpg?after={started_at}"
+    assert client.get(url, **api_headers).status_code == 404
+    os.utime(frame, (started_at + 1, started_at + 1))
+    response = client.get(url, **api_headers)
+    assert response.status_code == 200
+    assert b"".join(response.streaming_content) == b"stored frame"
+    assert client.get(url.replace(str(started_at), "invalid"), **api_headers).status_code == 400
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get("/api/v1/machine/binaries", **api_headers)
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get(f"/api/v1/machine/binary/{binary.id}", **api_headers)
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get("/api/v1/machine/binary/by-name/python3", **api_headers)
@@ -7,7 +7,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()

    response = client.get(f"/api/v1/machine/machine/{machine.id}", **api_headers)

@@ -7,7 +7,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    Machine.current(refresh=True)
+    Machine.current()

    response = client.get("/api/v1/machine/machines", **api_headers)

@@ -296,9 +296,11 @@ def test_archiveresult_event_create_uses_one_result_lookup():
    assert len(result_lookups) == 1


-def test_process_completed_projects_failed_archiveresult_from_shipped_hook(tmp_path, hermetic_lib_dir):
+def test_failed_shipped_title_hook_projects_failure_without_overwriting_title(tmp_path, hermetic_lib_dir):
    from archivebox.core.models import ArchiveResult

+    # Result linkage and the unchanged title are consequences of the same failed
+    # hook. Check them together so this isolated runtime is installed only once.
    snapshot = _create_snapshot()
    process, result = _run_shipped_snapshot_hook(
        snapshot,
@@ -310,22 +312,6 @@ def test_process_completed_projects_failed_archiveresult_from_shipped_hook(tmp_p
    assert result.status == ArchiveResult.StatusChoices.FAILED
    assert result.process_id == process.id
    assert "Chrome session" in result.output_str
-    _cleanup_machine_process_rows()
-
-
-def test_failed_title_archiveresult_does_not_overwrite_snapshot_title(tmp_path, hermetic_lib_dir):
-    from archivebox.core.models import ArchiveResult
-
-    snapshot = _create_snapshot()
-    _, result = _run_shipped_snapshot_hook(
-        snapshot,
-        plugin="title",
-        hook_name="on_Snapshot__54_title.js",
-        lib_dir=hermetic_lib_dir,
-        expected_exit_codes=(1,),
-    )
-    assert result.status == ArchiveResult.StatusChoices.FAILED
-    assert "Chrome session" in result.output_str
    snapshot.refresh_from_db()
    assert snapshot.title in (None, "")
    assert snapshot.resolved_title == ""
@@ -0,0 +1,377 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+import signal
+import subprocess
+
+import psutil
+from pathlib import Path
+
+import pytest
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+IMPORT_FORMAT_EXPECTATIONS = {
+    "rss": {
+        "url": "https://example.com/",
+        "title": "RSS Example Import",
+        "date": "2024-01-01",
+        "tags": {"rss-tag", "metadata"},
+    },
+    "netscape": {
+        "url": "https://www.iana.org/domains/reserved",
+        "title": "IANA Reserved Domains",
+        "date": "2024-01-02",
+        "tags": {"netscape-tag", "metadata"},
+    },
+    "dom": {
+        "url": "https://www.iana.org/help/example-domains",
+    },
+    "json": {
+        "url": "https://example.com/?archivebox-json-import=1",
+        "title": "JSON Import Example",
+        "date": "2024-01-03",
+        "tags": {"json-tag", "metadata"},
+    },
+    "jsonl": {
+        "url": "https://example.com/?archivebox-jsonl-import=1",
+        "title": "JSONL Import Example",
+        "date": "2024-01-04",
+        "tags": {"jsonl-tag", "metadata"},
+    },
+    "txt": {
+        "url": "https://example.org/",
+    },
+}
+
+
+def write_import_format_files(base_dir: Path) -> dict[str, Path]:
+    files = {
+        "rss": base_dir / "test_rss.xml",
+        "netscape": base_dir / "test_netscape.html",
+        "dom": base_dir / "test_dom.html",
+        "json": base_dir / "test_bookmarks.json",
+        "jsonl": base_dir / "test_bookmarks.jsonl",
+        "txt": base_dir / "test_urls.txt",
+    }
+    files["rss"].write_text(
+        """<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0">
+  <channel>
+    <title>ArchiveBox RSS import fixture</title>
+    <link>https://example.com/</link>
+    <description>ArchiveBox RSS import fixture</description>
+    <item>
+      <title>RSS Example Import</title>
+      <link>https://example.com/</link>
+      <guid>https://example.com/</guid>
+      <pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate>
+      <category>rss-tag</category>
+      <category>metadata</category>
+    </item>
+  </channel>
+</rss>
+""",
+        encoding="utf-8",
+    )
+    files["netscape"].write_text(
+        """<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL><p>
+  <DT><A HREF="https://www.iana.org/domains/reserved" ADD_DATE="1704153600" TAGS="netscape-tag,metadata">IANA Reserved Domains</A>
+</DL><p>
+""",
+        encoding="utf-8",
+    )
+    files["dom"].write_text(
+        """<!doctype html>
+<html>
+  <head><title>DOM import fixture</title></head>
+  <body>
+    <a href="https://www.iana.org/help/example-domains">IANA Example Domains</a>
+  </body>
+</html>
+""",
+        encoding="utf-8",
+    )
+    files["json"].write_text(
+        json.dumps(
+            {
+                "url": "https://example.com/?archivebox-json-import=1",
+                "title": "JSON Import Example",
+                "tags": ["json-tag", "metadata"],
+                "bookmarked_at": "2024-01-03T00:00:00+00:00",
+            },
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    files["jsonl"].write_text(
+        json.dumps(
+            {
+                "url": "https://example.com/?archivebox-jsonl-import=1",
+                "title": "JSONL Import Example",
+                "tags": "jsonl-tag,metadata",
+                "bookmarked_at": "2024-01-04T00:00:00+00:00",
+            },
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    files["txt"].write_text(
+        "Plain text import fixture containing https://example.org/ as a real live URL.\n",
+        encoding="utf-8",
+    )
+    return files
+
+
+IMPORT_FORMAT_ENV = {
+    "USE_COLOR": "False",
+    "SHOW_PROGRESS": "False",
+    "PLUGINS": "parse_html_urls,parse_jsonl_urls,parse_netscape_urls,parse_rss_urls,parse_txt_urls",
+    "SAVE_WGET": "False",
+    "SAVE_HEADERS": "False",
+    "USE_CHROME": "False",
+    "URL_ALLOWLIST": r"example\.com|example\.org|iana\.org|www\.iana\.org",
+}
+
+
+def assert_expected_import_snapshots(cwd: Path, expected_urls: set[str]) -> None:
+    allowed_statuses = {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+    with use_archivebox_db(cwd):
+        rows = list(Snapshot.objects.filter(url__in=expected_urls).values_list("url", "status"))
+    counts = {url: 0 for url in expected_urls}
+    bad_statuses = []
+    for url, status in rows:
+        counts[url] += 1
+        if status not in allowed_statuses:
+            bad_statuses.append((url, status))
+    assert all(count == 1 for count in counts.values()), counts
+    assert not bad_statuses, bad_statuses
+
+
+def malicious_add_inputs(tmp_path: Path, *, safe_url: str) -> tuple[list[str], Path]:
+    other_crawl_source = tmp_path / "sources" / "other_crawl_source.txt"
+    other_crawl_source.parent.mkdir(parents=True, exist_ok=True)
+    other_crawl_source.write_text("https://example.com/not-owned-by-this-crawl\n", encoding="utf-8")
+    canary = tmp_path / "archivebox_shell_injection_canary"
+    return (
+        [
+            safe_url,
+            "file:///etc/hosts",
+            "/etc/hosts",
+            "../../../../etc/passwd",
+            f"file://{other_crawl_source}",
+            str(other_crawl_source),
+            f"'; touch {canary}; #",
+            f'" && touch {canary} && echo "',
+            f"$(touch {canary})",
+            f"`touch {canary}`",
+            """<?xml version="1.0"?>
+<!DOCTYPE rss [
+  <!ENTITY localfile SYSTEM "file:///etc/hosts">
+]>
+<rss version="2.0" xmlns:xi="http://www.w3.org/2001/XInclude">
+  <channel>
+    <item><title>&localfile;</title><link>file:///etc/passwd</link></item>
+    <xi:include href="file:///etc/hosts" parse="text"/>
+  </channel>
+</rss>""",
+        ],
+        canary,
+    )
+
+
+def assert_no_file_or_shell_payload_snapshots(cwd: Path, *, canary: Path) -> None:
+    with use_archivebox_db(cwd):
+        snapshots = list(Snapshot.objects.all())
+    assert not canary.exists()
+    assert not [snapshot.url for snapshot in snapshots if str(snapshot.url).startswith("file:")]
+    for forbidden in ("/etc/hosts", "/etc/passwd", "other_crawl_source", "archivebox_shell_injection_canary"):
+        assert not [snapshot.url for snapshot in snapshots if forbidden in str(snapshot.url)]
+
+
+@pytest.mark.parametrize(
+    "choice,plugins",
+    [(choice, "chrome") for choice in ["skip", "retry", "abort", "ctrl-c", "noninteractive"]] + [("noninteractive", "archivewebpage")],
+)
+def test_add_interrupts_active_hook(initialized_archive, choice, plugins):
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True")
+    # The timed observation below is for a running navigate hook. Resolve the
+    # selected plugin's real binaries before starting it, so a cold install is
+    # not counted as a hook-start failure.
+    installed = run_archivebox_cmd(["install", plugins], cwd=initialized_archive, env=env, timeout=600)
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", f"--plugins={plugins}", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=subprocess.DEVNULL if choice == "noninteractive" else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        hook = active_hook()
+        assert hook is not None
+        result.send_signal(signal.SIGINT)
+        if choice != "noninteractive":
+            read_until(lambda: b"Choice [skip]:" in output)
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            assert not psutil.pid_exists(hook.pid)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (new_hook := active_hook()) is not None and new_hook.pid != hook.pid)
+                prompt_offset = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[prompt_offset:])
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "ctrl-c":
+                result.send_signal(signal.SIGINT)
+            else:
+                os.write(master, b"a" if choice == "abort" else b"\r")
+        read_until(lambda: result.poll() is not None)
+        assert result.returncode == (130 if choice in {"abort", "ctrl-c", "noninteractive"} else 0), output.decode(errors="replace")
+        if choice == "noninteractive":
+            assert b"Choice [skip]:" not in output
+        assert not psutil.pid_exists(hook.pid)
+        assert b"Traceback" not in output
+        with use_archivebox_db(initialized_archive):
+            assert not ArchiveResult.objects.filter(hook_name="on_Snapshot__30_chrome_navigate").exists()
+            if plugins == "archivewebpage":
+                start = ArchiveResult.objects.get(hook_name="on_Snapshot__16_archivewebpage_start")
+                assert start.status == ArchiveResult.StatusChoices.NORESULTS
+                assert not ArchiveResult.objects.filter(hook_name="on_Snapshot__65_archivewebpage_stop").exists()
+                assert not ArchiveResult.objects.filter(plugin="archivewebpage", status="succeeded").exists()
+                assert not any(output["name"] == "archivewebpage" for output in start.snapshot.get_html_details_context()["archiveresults"])
+        for log in (initialized_archive / "logs").glob("worker_runner_add_*.log"):
+            assert "Choice [skip]:" not in log.read_text()
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_single_url_records_url_in_crawl(initialized_archive):
+    """Test that adding a single URL queues a crawl with the submitted URL."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "Crawl.save() outside runner process" not in result.stderr
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.urls == "https://example.com"
+    assert crawl.get_urls_list() == ["https://example.com"]
+    assert snapshots == []
+
+
+def test_add_with_depth_0_flag(initialized_archive):
+    """Test that --depth=0 flag is accepted and works."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --depth" not in result.stderr
+
+
+def test_add_records_max_url_and_size_limits_on_crawl(initialized_archive):
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=1",
+            "--max-urls=3",
+            "--crawl-max-size=45mb",
+            "--crawl-timeout=120",
+            "--snapshot-max-size=5mb",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    columns = {field.name for field in Crawl._meta.local_fields}
+    with use_archivebox_db(initialized_archive):
+        config = Crawl.objects.values_list("config", flat=True).get() or {}
+
+    assert {"max_urls", "crawl_max_size", "crawl_timeout", "snapshot_max_size"}.isdisjoint(columns)
+    assert config["CRAWL_MAX_URLS"] == 3
+    assert config["CRAWL_MAX_SIZE"] == 45 * 1024 * 1024
+    assert config["CRAWL_TIMEOUT"] == 120
+    assert config["SNAPSHOT_MAX_SIZE"] == 5 * 1024 * 1024
+
+
+def test_add_without_args_shows_usage(initialized_archive):
+    """Test that add without URLs fails with a usage hint instead of crashing."""
+
+    result = run_archivebox_cmd(
+        ["add"],
+    )
+
+    combined = result.stdout + result.stderr
+    assert result.returncode != 0
+    assert "usage" in combined.lower() or "url" in combined.lower()
@@ -0,0 +1,433 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+
+from pathlib import Path
+
+import pytest
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+    run_queued_crawls,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+@pytest.fixture
+def progress_page(httpserver):
+    # Exercise the real HTTP fetch and parser with fixed content. example.com's
+    # outgoing IANA link has changed independently of ArchiveBox releases.
+    httpserver.expect_request("/").respond_with_data(
+        '<html><body><a href="https://iana.org/domains/example">Example domains</a></body></html>',
+        content_type="text/html",
+    )
+    return httpserver.url_for("/")
+
+
+@pytest.mark.parametrize("terminal_stream", ["stdout", "stderr"])
+def test_add_renders_live_progress_in_foreground_terminal(initialized_archive, terminal_stream, progress_page):
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--plugins=parse_txt_urls", progress_page],
+            cwd=initialized_archive,
+            env=cli_env(USE_COLOR="True", SHOW_PROGRESS="True"),
+            **{terminal_stream: slave},
+            wait=False,
+            start_new_session=True,
+        )
+        deadline = time.monotonic() + 90
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            elif result.poll() is not None:
+                break
+        assert result.poll() == 0, output.decode(errors="replace")
+        assert b"\x1b[?25l" in output, output.decode(errors="replace")
+        assert b"\x1b[?25h" in output, output.decode(errors="replace")
+        assert b"on_Snapshot__71_parse_txt_urls" in output
+        assert b"\x1b[2K" in output
+        with use_archivebox_db(initialized_archive):
+            assert Crawl.objects.get().status == Crawl.StatusChoices.SEALED
+            snapshot = Snapshot.objects.get()
+            assert snapshot.url == progress_page
+            parsed_urls = Path(snapshot.output_dir) / "parse_txt_urls" / "urls.jsonl"
+            assert json.loads(parsed_urls.read_text())["url"] == "https://iana.org/domains/example"
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=10)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_redirected_progress_remains_plain_text(initialized_archive, progress_page):
+    result = run_archivebox_cmd(
+        ["add", "--plugins=parse_txt_urls", progress_page],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    assert "on_Snapshot__71_parse_txt_urls" in result.stderr
+    assert "\x1b[?25l" not in result.stdout + result.stderr
+
+
+@pytest.mark.timeout(360)
+def test_add_stdin_import_formats_preserve_metadata_and_crawl_inner_urls(initialized_archive):
+    """`archivebox add < import-file` should normalize rich import formats before crawling URLs."""
+    import_files = write_import_format_files(initialized_archive)
+    expected_urls = {case["url"] for case in IMPORT_FORMAT_EXPECTATIONS.values()}
+    env = cli_env(**IMPORT_FORMAT_ENV)
+
+    for import_path in import_files.values():
+        source_text = import_path.read_text(encoding="utf-8")
+        result = run_archivebox_cmd(
+            ["add", "--bg", "--depth=0", "--tag=cli-stdin-import"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=source_text,
+            timeout=360,
+        )
+        assert result.returncode == 0, result.stderr or result.stdout
+        with use_archivebox_db(initialized_archive):
+            crawl = Crawl.objects.order_by("-created_at").first()
+            assert crawl is not None
+            assert crawl.snapshot_set.count() == 0
+        assert crawl.urls == source_text
+
+    run_queued_crawls(initialized_archive, env=env, timeout=240)
+    with use_archivebox_db(initialized_archive):
+        for crawl in Crawl.objects.all():
+            assert not crawl.snapshot_set.filter(url__startswith="archivebox://").exists()
+            root_input = (crawl.output_dir / "input" / "staticfile" / "stdin.txt").read_text(encoding="utf-8")
+            assert root_input == crawl.urls
+    assert_expected_import_snapshots(initialized_archive, expected_urls)
+
+    list_result = run_archivebox_cmd(
+        ["list", "--json"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert list_result.returncode == 0, list_result.stderr or list_result.stdout
+    for expected_url in expected_urls:
+        assert expected_url in list_result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawls = list(Crawl.objects.order_by("created_at"))
+        snapshots_by_url = {snapshot.url: snapshot for snapshot in Snapshot.objects.prefetch_related("tags").filter(url__in=expected_urls)}
+        tags_by_url = {snapshot.url: set(snapshot.tags.values_list("name", flat=True)) for snapshot in snapshots_by_url.values()}
+
+    assert len(crawls) == len(import_files)
+    assert [crawl.urls for crawl in crawls] == [path.read_text(encoding="utf-8") for path in import_files.values()]
+    assert all(crawl.tags_str == "cli-stdin-import" for crawl in crawls)
+    assert all(crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED} for crawl in crawls)
+    assert len(snapshots_by_url) == len(expected_urls)
+
+    for import_name, expected in IMPORT_FORMAT_EXPECTATIONS.items():
+        snapshot = snapshots_by_url.get(expected["url"])
+        assert snapshot is not None, f"{import_name} did not create Snapshot for {expected['url']}"
+        assert snapshot.status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+        if expected.get("title"):
+            assert snapshot.title == expected["title"]
+        if expected.get("date"):
+            assert snapshot.bookmarked_at.date().isoformat() == expected["date"]
+        if expected.get("tags"):
+            assert expected["tags"] | {"cli-stdin-import"} <= tags_by_url[snapshot.url]
+
+
+def test_add_bg_queues_direct_url_snapshot(initialized_archive):
+    """Background add queues explicit URL arguments as real URL snapshots."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is not None
+    assert crawl.urls == "https://example.com"
+    assert snapshots == []
+
+
+@pytest.mark.timeout(180)
+def test_add_title_hook_env_gets_canonical_runtime_config_without_archivebox_selectors_or_aliases(initialized_archive):
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "False",
+            "SHOW_PROGRESS": "False",
+            "PLUGINS": "title",
+            "SAVE_TITLE": "True",
+            "SECRET_KEY": "hook-env-secret-must-not-leak",
+            "PUBLIC_ADD_VIEW": "True",
+            "ADMIN_PASSWORD": "hook-env-admin-password-must-not-leak",
+            "URL_BLACKLIST": "$^",
+            "ARCHIVEBOX_TEST_ARBITRARY_ENV": "preserved-user-env",
+            "LD_PRELOAD": "",
+            "TIMEOUT": "60",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "1",
+        },
+    )
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--plugins=title",
+            "https://example.com/?archivebox-title-env=1",
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "Crawl.save() outside runner process" not in result.stdout + result.stderr
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot = Snapshot.objects.get()
+        title_result = ArchiveResult.objects.select_related("process").get(snapshot=snapshot, plugin="title")
+        process_env = title_result.process.env
+
+    assert crawl.config["PLUGINS"] == "title"
+    assert snapshot.url == "https://example.com/?archivebox-title-env=1"
+    assert snapshot.status == Snapshot.StatusChoices.SEALED
+    assert snapshot.title == "Example Domain"
+    assert title_result.status == ArchiveResult.StatusChoices.SUCCEEDED
+    assert title_result.output_str == "Example Domain"
+    assert title_result.process.exit_code == 0
+    assert process_env["TITLE_ENABLED"] == "True"
+    assert process_env["CHROME_ENABLED"] == "True"
+    assert process_env["WGET_ENABLED"] == "False"
+    assert process_env["ARCHIVEBOX_TEST_ARBITRARY_ENV"] == "preserved-user-env"
+    assert process_env["LD_PRELOAD"] == ""
+    assert "PLUGINS" not in process_env
+    assert "SAVE_TITLE" not in process_env
+    assert "SEARCH_BACKEND_ENGINE" not in process_env
+    assert "SECRET_KEY" not in process_env
+    assert "ADMIN_PASSWORD" not in process_env
+    assert "PUBLIC_ADD_VIEW" not in process_env
+    assert "URL_BLACKLIST" not in process_env
+    assert "hook-env-secret-must-not-leak" not in json.dumps(process_env)
+    assert "hook-env-admin-password-must-not-leak" not in json.dumps(process_env)
+
+
+def test_add_creates_crawl_record(initialized_archive):
+    """Test that add command creates a Crawl record in the database."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        crawl_count = Crawl.objects.count()
+
+    assert crawl_count == 1
+
+
+def test_add_rejects_file_path_argument(initialized_archive):
+    """Local files must be piped through stdin, not passed as archiveable path arguments."""
+    env = cli_env(disable_extractors=True)
+    urls_file = initialized_archive / "urls.txt"
+    urls_file.write_text("https://example.com\nhttps://example.org\n")
+
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", str(urls_file)],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode != 0
+
+    with use_archivebox_db(initialized_archive):
+        assert Crawl.objects.count() == 0
+        assert Snapshot.objects.count() == 0
+
+    assert "No URLs provided" in (result.stderr or result.stdout)
+
+
+def test_add_records_url_filter_overrides_on_crawl(initialized_archive):
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=0",
+            "--domain-allowlist=example.com,*.example.com",
+            "--domain-denylist=static.example.com",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+
+    assert crawl.config["URL_ALLOWLIST"] == "example.com,*.example.com"
+    assert crawl.config["URL_DENYLIST"] == "static.example.com"
+    assert not (initialized_archive / "personas" / "Default" / "chrome_extensions").exists()
+
+
+def test_add_help_shows_depth_and_tag_options(initialized_archive):
+    """Test that add --help documents the main filter and crawl options."""
+
+    result = run_archivebox_cmd(
+        ["add", "--help"],
+    )
+
+    assert result.returncode == 0
+    assert "--depth" in result.stdout
+    assert "--max-urls" in result.stdout
+    assert "--crawl-max-size" in result.stdout
+    assert "--crawl-timeout" in result.stdout
+    assert "--snapshot-max-size" in result.stdout
+    assert "--tag" in result.stdout
+
+
+def test_add_index_only_creates_direct_url_snapshot(initialized_archive):
+    """Index-only add seeds explicit URL args for the runner to snapshot."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot = Snapshot.objects.get()
+
+    assert crawl.urls == "https://example.com"
+    assert snapshot.url == "https://example.com"
+    assert snapshot.depth == 0
+
+
+def test_snapshot_create_sets_snapshot_timestamp(initialized_archive):
+    """Test the user-facing snapshot creation path sets a timestamp."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["snapshot", "create", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        check=True,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        timestamp = Snapshot.objects.values_list("timestamp", flat=True).get()
+
+    assert timestamp is not None
+    assert len(str(timestamp)) > 0
+
+
+@pytest.mark.timeout(180)
+def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive, recursive_test_site):
+
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "TIMEOUT": "60",
+            "SAVE_WGET": "true",
+            "SAVE_HEADERS": "false",
+            "SAVE_TITLE": "false",
+            "SAVE_READABILITY": "false",
+            "SAVE_SINGLEFILE": "false",
+            "SAVE_MERCURY": "false",
+            "SAVE_SCREENSHOT": "false",
+            "SAVE_PDF": "false",
+            "SAVE_DOM": "false",
+            "SAVE_ARCHIVEDOTORG": "false",
+            "SAVE_GIT": "false",
+            "SAVE_YTDLP": "false",
+            "SAVE_FAVICON": "false",
+            "PARSE_HTML_URLS_ENABLED": "true",
+            "PARSE_DOM_OUTLINKS_ENABLED": "false",
+            "URL_ALLOWLIST": r"127\.0\.0\.1[:/].*",
+        },
+    )
+    root_url = recursive_test_site["root_url"]
+    child_url = recursive_test_site["child_urls"][0]
+
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=2",
+            "--max-urls=2",
+            "--crawl-max-size=50mb",
+            "--tag=recursive-flow",
+            "--parser=url_list",
+            "--plugins=wget,parse_html_urls",
+            root_url,
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()
+        snapshots = list(Snapshot.objects.order_by("depth", "url").values_list("url", "depth", "status"))
+        archive_results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__depth", "snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_files"),
+        )
+
+    assert crawl[0] == 2
+    assert crawl[1] == "recursive-flow"
+    crawl_config = crawl[2] or {}
+    assert crawl_config["CRAWL_MAX_URLS"] == 2
+    assert crawl_config["CRAWL_MAX_SIZE"] == 50 * 1024 * 1024
+    assert crawl_config.get("SNAPSHOT_MAX_SIZE", 0) == 0
+    assert (root_url, 0, "sealed") in snapshots
+    assert any(url == child_url and depth == 1 and status == "sealed" for url, depth, status in snapshots)
+
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _files in archive_results}
+    assert by_url_plugin[(root_url, "wget")] == "succeeded"
+    assert by_url_plugin[(root_url, "parse_html_urls")] == "succeeded"
+    assert by_url_plugin[(child_url, "wget")] == "succeeded"
+
+    urls_outputs = list((initialized_archive / "archive/users/system/snapshots").rglob("parse_html_urls/urls.jsonl"))
+    assert urls_outputs
+    assert any(child_url in path.read_text() for path in urls_outputs)
@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import pty
+import select
+import time
+import termios
+import signal
+import re
+import subprocess
+
+import psutil
+from pathlib import Path
+
+import pytest
+from django.utils import timezone
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+    run_queued_crawls,
+    resolve_abxpkg_chrome_env,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+@pytest.mark.parametrize("choice", ["skip", "retry", "ctrl-c", "double", "noninteractive"])
+def test_add_interrupts_background_only_capture(initialized_archive, choice):
+    """Neither pause nor abort/reclaim semantics may depend on a foreground hook."""
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.05)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(archiveresult__plugin="forumdl", status="running").order_by("-started_at").first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=forumdl", "https://news.ycombinator.com"],
+            cwd=initialized_archive,
+            env=cli_env(),
+            stdin=subprocess.DEVNULL if choice == "noninteractive" else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        hook = active_hook()
+        children = psutil.Process(result.pid).children(recursive=True)
+        result.send_signal(signal.SIGINT)
+        if choice == "double":
+            read_until(lambda: b"Forwarding Ctrl+C" in output)
+            result.send_signal(signal.SIGINT)
+        elif choice != "noninteractive":
+            read_until(lambda: b"Choice [skip]:" in output, timeout=20)
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            assert not psutil.pid_exists(hook.pid)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (next_hook := active_hook()) is not None and next_hook.pid != hook.pid)
+                children.extend(psutil.Process(result.pid).children(recursive=True))
+                offset = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[offset:], timeout=20)
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            os.write(master, b"\r" if choice == "skip" else b"\x03")
+        read_until(lambda: result.poll() is not None, timeout=30)
+        assert result.returncode == (0 if choice == "skip" else 130), output.decode(errors="replace")
+        assert not [child.pid for child in children if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert not psutil.pid_exists(hook.pid)
+        assert b"Traceback" not in output
+        if choice == "noninteractive":
+            assert b"Choice [skip]:" not in output
+        # An explicit abort must not be mistaken for takeover and restart work.
+        with use_archivebox_db(initialized_archive):
+            assert Process.objects.filter(worker_type="worker_runner", process_type="orchestrator").count() == 1
+        if choice != "skip":
+            assert b"Aborting crawl" in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=20)
+        os.close(slave)
+        os.close(master)
+
+
+def test_background_completions_render_above_interrupt_prompt(initialized_archive, recursive_test_site):
+    env = cli_env(WGET_ARGS_EXTRA='["--limit-rate=20"]', INFINISCROLL_SCROLL_DELAY="10000", CHROME_HEADLESS="True")
+    # The timed observation is for a running infinite-scroll hook. Resolve its
+    # real browser and wget dependencies before the readiness window begins.
+    installed = run_archivebox_cmd(
+        ["install", "wget", "chrome"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=600,
+    )
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+    env.update(resolve_abxpkg_chrome_env(Path(env["ABXPKG_LIB_DIR"]), env))
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        for log in sorted(initialized_archive.rglob("on_Snapshot__*.*.log")):
+            print(f"\n{log.relative_to(initialized_archive)}:\n{log.read_text(errors='replace')}")
+        raise AssertionError(output.decode(errors="replace"))
+
+    def result_for(plugin):
+        with use_archivebox_db(initialized_archive):
+            return ArchiveResult.objects.filter(plugin=plugin).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=wget,infiniscroll,parse_html_urls", recursive_test_site["root_url"]],
+            cwd=initialized_archive,
+            # The small test-owned page still transfers slowly enough for wget
+            # to finish while the real foreground hook's prompt is open.
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: (scroll := result_for("infiniscroll")) is not None and scroll.status == "started")
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        before_completion = len(output)
+        read_until(lambda: (wget := result_for("wget")) is not None and wget.status == "succeeded")
+        read_until(lambda: b"Choice [skip]:" in output[before_completion:])
+        screen_updates = re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", output[before_completion:].decode(errors="replace"))
+        completed_at = screen_updates.index("on_Snapshot__35_wget")
+        prompt_at = screen_updates.rindex("Interrupted on_Snapshot__45_infiniscroll")
+        assert "succeeded" in screen_updates[completed_at:prompt_at], screen_updates
+        assert completed_at < prompt_at < screen_updates.rindex("Choice [skip]:"), screen_updates
+        assert result_for("parse_html_urls") is None
+        (initialized_archive / "interrupt-terminal.txt").write_bytes(output)
+        os.write(master, b"a")
+        read_until(lambda: result.poll() is not None)
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_summary_points_to_snapshot_outputs(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--plugins=parse_txt_urls", "https://example.com"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get()
+        assert Path(snapshot.output_dir).is_dir()
+        assert str(snapshot.output_dir) in result.stdout
+        assert "snapshot output saved to:" in result.stdout
+        assert "crawl logs and setup:" in result.stdout
+        assert "crawl output saved to:" not in result.stdout
+
+
+def test_initial_crawl_creation_does_not_warn_as_an_outside_runner_update(initialized_archive, caplog):
+    from archivebox.core.takeover_util import current_command
+
+    with use_archivebox_db(initialized_archive):
+        command = current_command(Process.TypeChoices.ADD, data_dir=initialized_archive)
+        caplog.clear()
+        with caplog.at_level("WARNING", logger="archivebox.workers.models"):
+            crawl = Crawl.objects.create(urls="https://example.com", status=Crawl.StatusChoices.QUEUED)
+        assert "Crawl.save() outside runner process" not in caplog.text
+
+        caplog.clear()
+        with caplog.at_level("WARNING", logger="archivebox.workers.models"):
+            crawl.save(update_fields=["modified_at"])
+        command.mark_exited(exit_code=0)
+
+    assert "Crawl.save() outside runner process" in caplog.text
+
+
+@pytest.mark.timeout(240)
+def test_add_rejects_file_path_and_shell_injection_payloads(initialized_archive):
+    """CLI add must not turn user-supplied local paths or shell payloads into snapshots."""
+    safe_url = "https://example.com/?archivebox-cli-security=1"
+    inputs, canary = malicious_add_inputs(initialized_archive, safe_url=safe_url)
+    env = cli_env(**IMPORT_FORMAT_ENV)
+
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--depth=0", "--tag=cli-security"],
+        cwd=initialized_archive,
+        env=env,
+        stdin="\n".join(inputs),
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    run_queued_crawls(initialized_archive, env=env, timeout=120)
+    assert_expected_import_snapshots(initialized_archive, {safe_url})
+
+    assert_no_file_or_shell_payload_snapshots(initialized_archive, canary=canary)
+    with use_archivebox_db(initialized_archive):
+        snapshots = list(Snapshot.objects.filter(url=safe_url).values_list("url", "status", "tags__name"))
+        crawl = Crawl.objects.get()
+    assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
+    assert len({url for url, _status, _tag in snapshots}) == 1
+    assert all(
+        status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+        for _url, status, _tag in snapshots
+    )
+    assert "cli-security" in {tag for _url, _status, tag in snapshots}
+
+
+@pytest.mark.timeout(180)
+def test_run_rejects_depth_two_file_url_snapshot_injected_directly_with_db_update(initialized_archive):
+    """A queued Snapshot row with a file:// URL must not be allowed to run hooks or recurse."""
+    secret_url = "https://example.com/?archivebox-sql-depth2-file-secret=1"
+    local_source = initialized_archive / "not_owned_by_depth_two_snapshot.txt"
+    local_source.write_text(f"{secret_url}\n", encoding="utf-8")
+    file_url = local_source.resolve().as_uri()
+    env = cli_env(**{**IMPORT_FORMAT_ENV, "PLUGINS": "parse_txt_urls", "SAVE_HEADERS": "False", "SAVE_WGET": "False"})
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/?archivebox-sql-depth2-root=1",
+            max_depth=2,
+            tags_str="sql-depth2-file-url",
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        root_snapshot = Snapshot.objects.create(
+            url="https://example.com/?archivebox-sql-depth2-root=1",
+            crawl=crawl,
+            depth=0,
+            status=Snapshot.StatusChoices.SEALED,
+            retry_at=None,
+        )
+        injected_snapshot = Snapshot.objects.create(
+            url="https://example.com/?archivebox-sql-depth2-placeholder=1",
+            crawl=crawl,
+            parent_snapshot=root_snapshot,
+            depth=2,
+            status=Snapshot.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        Snapshot.objects.filter(pk=injected_snapshot.pk).update(url=file_url)
+
+    result = run_archivebox_cmd(
+        ["run", f"--snapshot-id={injected_snapshot.id}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        injected_snapshot.refresh_from_db()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+        file_results = list(
+            ArchiveResult.objects.filter(
+                snapshot=injected_snapshot,
+            ).values_list("plugin", "status"),
+        )
+
+    assert injected_snapshot.url == file_url
+    assert secret_url not in snapshot_urls
+    assert file_results == []
+
+
+def test_add_index_only_rejected_urls_leave_empty_crawl_for_runner_to_seal(initialized_archive):
+    """Index-only add only creates the crawl; rejected URLs are sealed by the runner."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=0",
+            "--url-denylist=example.com",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshot_urls == set()
+
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status == Crawl.StatusChoices.SEALED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshot_urls == set()
+
+
+def test_add_index_only_rejects_archivebox_internal_urls(initialized_archive):
+    """Index-only add must apply the same internal URL guard as snapshot creation."""
+    env = cli_env(disable_extractors=True)
+    internal_urls = [
+        "http://archivebox.localhost:9292/admin/",
+        "http://web.archivebox.localhost:9292/",
+        "http://api.archivebox.localhost:9292/api/v1/docs",
+        "http://snap-2fb8e923c58c.archivebox.localhost:9292/index.html",
+    ]
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", *internal_urls],
+        cwd=initialized_archive,
+        env={**env, "BASE_URL": "http://archivebox.localhost:9292"},
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.urls.splitlines() == internal_urls
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert snapshot_urls == set()
+
+
+def test_add_multiple_urls_single_command(initialized_archive):
+    """Test adding multiple URLs in a single command records one crawl."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.order_by("url").values_list("url", "depth"))
+
+    assert crawl.urls.splitlines() == ["https://example.com", "https://example.org"]
+    assert snapshots == []
+
+
+def test_add_with_tags(initialized_archive):
+    """Test adding URL with tags stores tags_str in crawl.
+
+    With --index-only, Tag objects are not created until archiving happens.
+    Tags are stored as a string in the Crawl.tags_str field.
+    """
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "--tag=test,example", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        tags_str = Crawl.objects.values_list("tags_str", flat=True).get()
+
+    # Tags are stored as a comma-separated string in crawl
+    assert "test" in tags_str or "example" in tags_str
+
+
+def test_add_with_overwrite_flag(initialized_archive):
+    """Test that --overwrite flag forces re-archiving."""
+    env = cli_env(disable_extractors=True)
+
+    # Add URL first time
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    # Add with overwrite
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--overwrite", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --overwrite" not in result.stderr
+
+
+def test_add_index_only_queues_crawl_without_starting_runner(initialized_archive):
+    """Test that --index-only creates only a queued crawl and returns fast."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=30,  # Should be fast
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshots == []
@@ -0,0 +1,511 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+import signal
+import re
+
+import psutil
+from pathlib import Path
+
+import pytest
+from django.utils import timezone
+
+from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    find_snapshot_dir,
+    run_archivebox_cmd,
+    run_queued_crawls,
+    resolve_abxpkg_chrome_env,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+def test_abort_stops_long_running_background_hook(initialized_archive):
+    env = cli_env(INFINISCROLL_SCROLL_DELAY="10000", CHROME_HEADLESS="True")
+    # Keep first-use dependency installation outside the timed hook-start
+    # observation. This test is about interrupting a running hook; a cold
+    # browser or forumdl install must not consume its readiness window.
+    installed = run_archivebox_cmd(
+        ["install", "forumdl", "chrome"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=600,
+    )
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+    env.update(resolve_abxpkg_chrome_env(Path(env["ABXPKG_LIB_DIR"]), env))
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def result_for(plugin):
+        with use_archivebox_db(initialized_archive):
+            return ArchiveResult.objects.filter(plugin=plugin).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=forumdl,infiniscroll", "https://news.ycombinator.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        try:
+            read_until(lambda: (scroll := result_for("infiniscroll")) is not None and scroll.status == "started")
+        except AssertionError:
+            # Capture state before teardown terminates the runner: otherwise a
+            # missed readiness deadline loses the install/hook failure itself.
+            with use_archivebox_db(initialized_archive):
+                print("Hook readiness:", list(ArchiveResult.objects.values("plugin", "status", "output_str")))
+                print("Process readiness:", list(Process.objects.values("process_type", "status", "exit_code", "stderr")))
+            raise
+        read_until(lambda: "running pid=" in re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", output.decode(errors="replace")))
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        forum = result_for("forumdl")
+        assert forum is not None and forum.status == "started"
+        children = psutil.Process(result.pid).children(recursive=True)
+        abort_offset = len(output)
+        os.write(master, b"\x03")
+        read_until(lambda: b"Aborting crawl" in output[abort_offset:], timeout=3)
+        read_until(lambda: result.poll() is not None, timeout=25)
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert not [child.pid for child in children if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert any("INFO:root:GET" in log.read_text() for log in initialized_archive.rglob("on_Snapshot__33_forumdl.*.stderr.log"))
+        assert b"Stopped during crawl abort" in output
+        assert b"INFO:root:GET" not in output[abort_offset:]
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_bg_persists_queue_without_starting_extraction_workers(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--plugins=wget", "--crawl-max-concurrent-snapshots=4", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        assert crawl.status == Crawl.StatusChoices.QUEUED
+        assert crawl.retry_at is not None
+        assert crawl.get_urls_list() == ["https://example.com", "https://example.org"]
+        assert crawl.config["PLUGINS"] == "wget"
+        assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 4
+        assert not Snapshot.objects.exists()
+        assert not ArchiveResult.objects.exists()
+        assert not Process.objects.filter(process_type__in=[Process.TypeChoices.ORCHESTRATOR, Process.TypeChoices.HOOK]).exists()
+
+
+@pytest.mark.timeout(180)
+def test_run_rejects_file_url_injected_directly_into_crawl_urls_with_db_update(initialized_archive):
+    """Runner must validate Crawl.urls again when DB writes bypass normal add/create paths."""
+    secret_url = "https://example.com/?archivebox-sql-crawl-file-secret=1"
+    local_source = initialized_archive / "not_owned_by_crawl_urls.txt"
+    local_source.write_text(f"{secret_url}\n", encoding="utf-8")
+    file_url = local_source.resolve().as_uri()
+    env = cli_env(**{**IMPORT_FORMAT_ENV, "PLUGINS": "parse_txt_urls", "SAVE_HEADERS": "False", "SAVE_WGET": "False"})
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/?archivebox-sql-crawl-control=1",
+            max_depth=2,
+            tags_str="sql-file-url",
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        bad_jsonl = json.dumps({"type": "Snapshot", "url": file_url, "depth": 0, "tags": "sql-file-url"})
+        Crawl.objects.filter(pk=crawl.pk).update(urls=bad_jsonl)
+
+    result = run_archivebox_cmd(
+        ["run", f"--crawl-id={crawl.id}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl.refresh_from_db()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
+    assert file_url not in snapshot_urls
+    assert secret_url not in snapshot_urls
+
+
+@pytest.mark.timeout(180)
+def test_add_tagged_single_url_seals_without_duplicate_snapshot_tags(initialized_archive):
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "False",
+            "SHOW_PROGRESS": "False",
+            "PLUGINS": "parse_txt_urls,title",
+            "TIMEOUT": "60",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "1",
+        },
+    )
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--bg",
+            "--depth=0",
+            "--tag=tagged-single-url",
+            "https://example.com/?archivebox-tagged-single-url=1",
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    run_queued_crawls(initialized_archive, env, timeout=180)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.order_by("depth", "url"))
+        tag_counts = {
+            snapshot.url: SnapshotTag.objects.filter(snapshot=snapshot, tag__name="tagged-single-url").count() for snapshot in snapshots
+        }
+        results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__depth", "snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_str"),
+        )
+
+    assert crawl.status == Crawl.StatusChoices.SEALED
+    assert [(snapshot.url, snapshot.depth, snapshot.status) for snapshot in snapshots] == [
+        ("https://example.com/?archivebox-tagged-single-url=1", 0, Snapshot.StatusChoices.SEALED),
+    ]
+    assert tag_counts == {
+        "https://example.com/?archivebox-tagged-single-url=1": 1,
+    }
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _output in results}
+    assert by_url_plugin[("https://example.com/?archivebox-tagged-single-url=1", "title")] == "succeeded"
+    unexpected_failures = [(url, plugin, status, output) for url, plugin, status, output in results if status == "failed"]
+    assert not unexpected_failures
+
+
+def test_add_direct_url_creates_snapshot_without_internal_input_file(initialized_archive):
+    """Explicit URL args materialize real snapshots when the runner claims the crawl."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get()
+        assert snapshot.url == "https://example.com"
+        assert snapshot.depth == 0
+        assert not (snapshot.output_dir / "staticfile" / "stdin.txt").exists()
+
+
+def test_add_with_depth_1_flag(initialized_archive):
+    """Test that --depth=1 flag is accepted."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=1", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --depth" not in result.stderr
+
+
+def test_add_rejects_invalid_depth_values(initialized_archive):
+    """Test that add rejects depth values outside the supported range."""
+    env = cli_env(disable_extractors=True)
+
+    for depth in ("5", "-1"):
+        result = run_archivebox_cmd(
+            ["add", "--index-only", f"--depth={depth}", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+        )
+        stderr = result.stderr.lower()
+        assert result.returncode != 0
+        assert "invalid" in stderr or "not one of" in stderr
+
+
+def test_add_records_selected_persona_on_crawl(initialized_archive):
+    """Test add persists the selected persona so browser config derives from it later."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "--persona=Default", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+
+    assert crawl.persona_id
+    assert "ACTIVE_PERSONA" not in crawl.config
+    assert (initialized_archive / "personas" / "Default" / "chrome_profile").is_dir()
+
+
+def test_add_duplicate_url_creates_separate_crawls(initialized_archive):
+    """Test that adding the same URL twice creates separate crawls.
+
+    Each 'add' command creates a new Crawl. Multiple crawls can archive the same URL.
+    This allows re-archiving URLs at different times.
+    """
+
+    env = cli_env(disable_extractors=True)
+    # Add URL first time
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    # Add same URL second time with --update to opt out of ONLY_NEW.
+    run_archivebox_cmd(
+        ["add", "--index-only", "--update", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl_count = Crawl.objects.count()
+        snapshots = list(Snapshot.objects.order_by("created_at").values_list("url", "depth"))
+
+    # Each add creates a new crawl with its own queued work.
+    assert crawl_count == 2
+    assert snapshots == [("https://example.com", 0), ("https://example.com", 0)]
+
+
+def test_snapshot_create_creates_current_output_directory(initialized_archive):
+    """Test the user-facing snapshot creation path creates an output directory."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["snapshot", "create", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        check=True,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        snapshot_id = str(Snapshot.objects.values_list("id", flat=True).get())
+
+    snapshot_dir = find_snapshot_dir(initialized_archive, snapshot_id)
+    assert snapshot_dir is not None, f"Snapshot output directory not found for {snapshot_id}"
+    assert snapshot_dir.is_dir()
+
+
+def test_add_preserves_apostrophe_in_snapshot_url(initialized_archive):
+    url = "https://aaib.gov.in/What's%20New%20Assets/Preliminary%20Report%20VT-EXO.pdf"
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert result.returncode == 0, result.stderr
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        assert Crawl.objects.get().urls == url
+        assert Snapshot.objects.get().url == url
+
+
+@pytest.mark.timeout(180)
+def test_cli_add_real_urls_with_options_writes_inspectable_outputs(initialized_archive):
+
+    wget_urls = [
+        "https://example.com",
+        "https://pirate.github.io/stress-tests/challenge.html",
+    ]
+    chrome_url = "https://example.com/?archivebox-chrome-flow=1"
+    env = os.environ.copy()
+    env.pop("CHROME_BINARY", None)
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "TIMEOUT": "60",
+            "SAVE_WGET": "true",
+            "SAVE_HEADERS": "false",
+            "SAVE_TITLE": "false",
+            "SAVE_READABILITY": "false",
+            "SAVE_SINGLEFILE": "false",
+            "SAVE_MERCURY": "false",
+            "SAVE_SCREENSHOT": "false",
+            "SAVE_PDF": "false",
+            "SAVE_DOM": "false",
+            "SAVE_ARCHIVEDOTORG": "false",
+            "SAVE_GIT": "false",
+            "SAVE_YTDLP": "false",
+            "SAVE_FAVICON": "false",
+        },
+    )
+    _cmd_result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--max-urls=2",
+            "--crawl-max-size=10mb",
+            "--tag=real-flow,challenge",
+            "--parser=url_list",
+            "--plugins=wget",
+            *wget_urls,
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert returncode == 0, stderr or stdout
+
+    chrome_env = env | {
+        "SAVE_WGET": "false",
+        "SAVE_HEADERS": "true",
+        "SAVE_TITLE": "true",
+        "CHROME_HEADLESS": "true",
+        "CHROME_SANDBOX": "false",
+        "CHROME_ISOLATION": "snapshot",
+    }
+    _cmd_result = run_archivebox_cmd(
+        ["install", "chrome"],
+        cwd=initialized_archive,
+        env=chrome_env,
+        timeout=600,
+    )
+    install_stdout, install_stderr, install_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert install_returncode == 0, install_stderr or install_stdout
+    chrome_env.update(resolve_abxpkg_chrome_env(Path(chrome_env["ABXPKG_LIB_DIR"]), chrome_env))
+    _cmd_result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--max-urls=1",
+            "--crawl-max-size=10mb",
+            "--tag=chrome-flow",
+            "--parser=url_list",
+            "--plugins=chrome,wget,headers,title",
+            chrome_url,
+        ],
+        cwd=initialized_archive,
+        env=chrome_env,
+        timeout=180,
+    )
+    chrome_stdout, chrome_stderr, chrome_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert chrome_returncode == 0, chrome_stderr or chrome_stdout
+
+    _cmd_result = run_archivebox_cmd(
+        ["list", "--tag=real-flow"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    list_stdout, list_stderr, list_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert list_returncode == 0, list_stderr or list_stdout
+    listed = [json.loads(line) for line in list_stdout.splitlines() if line.strip()]
+    assert {item["url"] for item in listed} >= set(wget_urls)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()
+        real_flow_crawl = Crawl.objects.filter(tags_str="real-flow,challenge").values_list("max_depth", "tags_str", "config").first()
+        snapshots = list(Snapshot.objects.order_by("url").values_list("id", "url", "depth", "status", "title"))
+        archive_results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_files", "output_size", "output_str"),
+        )
+        processes = list(Process.objects.filter(process_type="hook").values_list("process_type", "status", "exit_code", "pwd", "cmd"))
+
+    assert real_flow_crawl is not None
+    assert real_flow_crawl[0] == 0
+    assert real_flow_crawl[1] == "real-flow,challenge"
+    real_flow_config = real_flow_crawl[2] or {}
+    assert real_flow_config["CRAWL_MAX_URLS"] == 2
+    assert real_flow_config["CRAWL_MAX_SIZE"] == 10 * 1024 * 1024
+    assert real_flow_config.get("SNAPSHOT_MAX_SIZE", 0) == 0
+    assert "wget" in real_flow_config["PLUGINS"]
+    assert crawl is not None
+    assert crawl[1] == "chrome-flow"
+    assert "wget,headers,title" in json.dumps(crawl[2] or {})
+
+    snapshot_urls = {url for _id, url, _depth, _status, _title in snapshots}
+    assert snapshot_urls >= {*wget_urls, chrome_url}
+    assert all(depth == 0 for _id, url, depth, _status, _title in snapshots)
+
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _files, _size, _output in archive_results}
+    assert by_url_plugin[("https://example.com", "wget")] == "succeeded"
+    assert by_url_plugin[("https://pirate.github.io/stress-tests/challenge.html", "wget")] == "succeeded"
+    assert by_url_plugin[(chrome_url, "headers")] == "succeeded"
+    assert by_url_plugin[(chrome_url, "title")] == "succeeded"
+    unexpected_results = [
+        (url, plugin, status, output) for url, plugin, status, _files, _size, output in archive_results if status != "succeeded"
+    ]
+    assert not unexpected_results
+
+    snapshot_root = initialized_archive / "archive/users/system/snapshots"
+    html_outputs = [path for path in snapshot_root.rglob("wget/**/*.html") if path.is_file()]
+    header_outputs = [path for path in snapshot_root.rglob("headers/**/headers.json") if path.is_file() and path.stat().st_size > 0]
+    title_outputs = [path for path in snapshot_root.rglob("title/title.txt") if path.is_file() and path.stat().st_size > 0]
+    index_outputs = [path for path in snapshot_root.rglob("index.jsonl") if path.is_file()]
+    assert html_outputs
+    assert header_outputs
+    assert any("example.com" in path.read_text(errors="ignore").lower() for path in header_outputs)
+    assert title_outputs
+    assert any("Example Domain" in path.read_text(errors="ignore") for path in title_outputs)
+    assert len(index_outputs) >= len(wget_urls) + 1
+
+    combined_html = "\n".join(path.read_text(errors="ignore") for path in html_outputs)
+    assert "Example Domain" in combined_html
+    assert "Browser Agent Challenge for AI Browser Drivers" in combined_html
+
+    assert processes
+    assert any("wget" in (pwd or "") or "wget" in (cmd or "") for _type, _status, _exit, pwd, cmd in processes)
+    assert any("headers" in (pwd or "") or "headers" in (cmd or "") for _type, _status, _exit, pwd, cmd in processes)
@@ -85,7 +85,7 @@ def test_wget_literal_percent_output_url_serves_captured_file(initialized_archiv
    """Replay the real HedgeDoc link-interstitial file saved by wget."""
    import shutil
    from urllib.parse import quote
-    from playwright.sync_api import sync_playwright
+    from playwright.sync_api import expect, sync_playwright

    source_url = "https://docs.sweeting.me/_link?url=http%3A%2F%2Farchive.org%2F"
    ui_snapshot = snapshot
@@ -171,15 +171,16 @@ def test_wget_literal_percent_output_url_serves_captured_file(initialized_archiv
        frame = page.locator("#main-frame")
        assert expected_path in frame.get_attribute("src")
        preview = page.frame_locator("#main-frame")
-        assert "External link" in preview.locator("body").inner_text()
-        assert "Continue to external page" in preview.locator("body").inner_text()
+        # Setting the iframe src does not mean its document has finished loading.
+        expect(preview.locator("body")).to_contain_text("External link")
+        expect(preview.locator("body")).to_contain_text("Continue to external page")

        page.reload(wait_until="domcontentloaded")
        frame = page.locator("#main-frame")
        assert expected_path in frame.get_attribute("src")
        preview = page.frame_locator("#main-frame")
-        assert "External link" in preview.locator("body").inner_text()
-        assert "Continue to external page" in preview.locator("body").inner_text()
+        expect(preview.locator("body")).to_contain_text("External link")
+        expect(preview.locator("body")).to_contain_text("Continue to external page")
        browser.close()


@@ -254,20 +254,22 @@ def test_init_refuses_database_migrated_by_newer_code(tmp_path):


 def test_init_recovers_from_pre_squash_dev_history(tmp_path):
-    """Pre-squash dev DBs (rows for migrations now absorbed by ``replaces=``)
-    must NOT trip the newer-DB guard — every historical squash would otherwise
-    brick beta-tester collections that pre-date the squash commit."""
+    """Known historical migration rows must not trip the newer-DB guard."""
    result = run_archivebox_cmd(["init"])
    assert result.returncode == 0

-    # Sampling — one name per affected app, all listed in the ``replaces=``
-    # declarations of the current squash anchors. If any of these get treated
-    # as missing-from-code, dev DBs that ran the historical chain pre-squash
-    # would refuse to start.
+    # Sampling across affected apps, including v0.6.2 runtime-generated names
+    # absent from Git. These older collections must still pass init.
    historical_pre_squash_rows = [
        ("api", "0002_alter_apitoken_options"),
        ("api", "0009_rename_created_apitoken_created_at_and_more"),
+        # v0.6.2 ran makemigrations at init, so these reported 0021 names were
+        # created in the installed package and never appeared in Git.
+        ("core", "0021_auto_20220510_0644"),
+        ("core", "0021_auto_20220724_1254"),
+        ("core", "0021_auto_20221128_1116"),
        ("core", "0023_alter_archiveresult_options_archiveresult_abid_and_more"),
+        ("core", "0056_progress_covering_indexes"),
        ("core", "0074_alter_snapshot_downloaded_at"),
        ("core", "0075_crawl"),
        ("core", "0075_archiveresult_retry_at"),
@@ -6,6 +6,38 @@ Tests for archivebox machine command.
 from archivebox.tests.conftest import run_archivebox_cmd


+def test_machine_identity_is_stable_across_cli_processes(initialized_archive):
+    """Minimal containers without /etc/machine-id must still share one identity."""
+    import sqlite3
+
+    import machineid
+
+    from archivebox.config.paths import get_machine_id
+    from archivebox.machine.detect import get_host_guid, get_vm_info
+
+    guid = get_host_guid()
+    assert len(guid) == 64
+    assert all(char in "0123456789abcdef" for char in guid)
+    assert get_machine_id() == guid[:8]
+    assert isinstance(get_vm_info()["hw_uuid"], str)
+    try:
+        native_guid = machineid.hashed_id("archivebox")
+    except machineid.MachineIdNotFound:
+        pass
+    else:
+        assert guid == native_guid
+
+    identities = []
+    for _ in range(2):
+        result = run_archivebox_cmd(["status"], cwd=initialized_archive)
+        assert result.returncode == 0, result.stdout + result.stderr
+        with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+            rows = db.execute("SELECT id, guid FROM machine_machine WHERE guid=?", (guid,)).fetchall()
+        assert len(rows) == 1
+        identities.append(rows)
+    assert identities[0] == identities[1]
+
+
 def test_status_refreshes_week_old_machine(initialized_archive):
    """A saved host refresh must not invalidate the instance still in use."""
    import sqlite3
@@ -0,0 +1,177 @@
+"""
+Tests for archivebox run CLI command.
+
+Tests cover:
+- run with stdin JSONL (Crawl, Snapshot, ArchiveResult)
+- create-or-update behavior (records with/without id)
+- pass-through output (for chaining)
+"""
+
+import os
+import pty
+import select
+import signal
+import subprocess
+import termios
+import time
+
+import psutil
+import pytest
+
+from archivebox.tests.conftest import (
+    cli_env,
+    create_test_url,
+    parse_jsonl_output,
+    run_archivebox_cmd,
+)
+
+RUN_TEST_ENV = {
+    "PLUGINS": "favicon",
+    "SAVE_FAVICON": "True",
+}
+
+
+def _install_real_chrome_for_test(data_dir, env, *, isolation):
+    env["CHROME_ISOLATION"] = isolation
+    env["CHROME_HEADLESS"] = "true"
+    env["CHROME_SANDBOX"] = "false"
+    install_process = run_archivebox_cmd(
+        ["install", "chrome"],
+        cwd=data_dir,
+        env=env,
+        timeout=600,
+    )
+    assert install_process.returncode == 0, install_process.stderr or install_process.stdout
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize(
+    "choice,target",
+    [(choice, "crawl") for choice in ["skip", "retry", "abort", "ctrl-c", "noninteractive"]]
+    + [("skip", "snapshot"), ("skip", "global"), ("early-skip", "crawl")]
+    + [(choice, "daemon") for choice in ["daemon-tty", "daemon-noninteractive"]]
+    + [("daemon-term", "daemon")],
+)
+def test_run_crawl_interrupts_active_hook(initialized_archive, choice, target):
+    """Foreground run uses the same terminal interrupt choices as add."""
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True", PLUGINS="chrome")
+    # This test starts its 45-second observation at the active navigate hook.
+    # Install Chrome through the real CLI first so a cold binary download is
+    # not mistaken for a hook that never started.
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    snapshot = next(record for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    crawl_id = snapshot["crawl_id"]
+    run_args = {
+        "crawl": [f"--crawl-id={crawl_id}"],
+        "snapshot": [f"--snapshot-id={snapshot['id']}"],
+        "global": [],
+        "daemon": ["--daemon", f"--crawl-id={crawl_id}"],
+    }[target]
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+    owned_processes = []
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["run", *run_args],
+            cwd=initialized_archive,
+            env=env,
+            stdin=subprocess.DEVNULL if choice in {"noninteractive", "daemon-noninteractive"} else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        first_hook = active_hook()
+        assert first_hook is not None
+        with use_archivebox_db(initialized_archive):
+            hook_pids = Process.objects.filter(status="running", process_type="hook").values_list("pid", flat=True)
+            for pid in hook_pids:
+                try:
+                    process = psutil.Process(pid)
+                    owned_processes.extend([process, *process.children(recursive=True)])
+                except psutil.NoSuchProcess:
+                    continue
+        result.send_signal(signal.SIGTERM if choice == "daemon-term" else signal.SIGINT)
+        if choice not in {"noninteractive", "daemon-tty", "daemon-noninteractive", "daemon-term"}:
+            read_until(lambda: b"Choice [skip]:" in output, timeout=30)
+            if choice != "early-skip":
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (next_hook := active_hook()) is not None and next_hook.pid != first_hook.pid)
+                before_prompt = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[before_prompt:], timeout=30)
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "ctrl-c":
+                result.send_signal(signal.SIGINT)
+            else:
+                os.write(master, b"a" if choice == "abort" else b"\r")
+        read_until(lambda: result.poll() is not None, timeout=45)
+        expected_exit = 143 if choice == "daemon-term" else (0 if choice in {"skip", "early-skip", "retry"} else 130)
+        assert result.returncode == expected_exit, output.decode(errors="replace")
+        assert not psutil.pid_exists(first_hook.pid)
+        _, live_processes = psutil.wait_procs(owned_processes, timeout=5)
+        assert not [process.pid for process in live_processes if process.is_running() and process.status() != psutil.STATUS_ZOMBIE]
+        assert b"Traceback" not in output, output.decode(errors="replace")
+        with use_archivebox_db(initialized_archive):
+            crawl_after = Crawl.objects.get(id=crawl_id)
+            snapshot_after = Snapshot.objects.get(id=snapshot["id"])
+            if choice == "daemon-term":
+                # A supervisor stop displaces this runner; the user's crawl
+                # and snapshot remain resumable for the next owner.
+                assert crawl_after.status == Crawl.StatusChoices.STARTED
+                assert snapshot_after.status == Snapshot.StatusChoices.STARTED
+                assert crawl_after.retry_at is not None and snapshot_after.retry_at is not None
+            else:
+                assert crawl_after.status == Crawl.StatusChoices.SEALED
+                assert snapshot_after.status == Snapshot.StatusChoices.SEALED
+            assert not Process.objects.filter(status="running", pid__isnull=False).exists()
+        if choice in {"noninteractive", "daemon-tty", "daemon-noninteractive", "daemon-term"}:
+            assert b"Choice [skip]:" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            try:
+                result.wait(timeout=20)
+            except subprocess.TimeoutExpired:
+                result.send_signal(signal.SIGKILL)
+                result.wait(timeout=5)
+        for process in owned_processes:
+            try:
+                if process.is_running() and process.status() != psutil.STATUS_ZOMBIE:
+                    process.terminate()
+            except psutil.NoSuchProcess:
+                pass
+        psutil.wait_procs(owned_processes, timeout=3)
+        os.close(slave)
+        os.close(master)
@@ -0,0 +1,489 @@
+"""
+Tests for archivebox run CLI command.
+
+Tests cover:
+- run with stdin JSONL (Crawl, Snapshot, ArchiveResult)
+- create-or-update behavior (records with/without id)
+- pass-through output (for chaining)
+"""
+
+import os
+import pty
+import select
+import signal
+import subprocess
+import termios
+import time
+
+import psutil
+import pytest
+
+from archivebox.tests.conftest import (
+    cleanup_process_group,
+    cli_env,
+    create_test_url,
+    parse_jsonl_output,
+    pid_is_alive,
+    run_archivebox_cmd,
+    wait_for_log,
+    wait_for_pid_to_disappear,
+)
+
+from .test_cli_run_1 import (
+    RUN_TEST_ENV as RUN_TEST_ENV,
+    _install_real_chrome_for_test as _install_real_chrome_for_test,
+)
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("choice", ["typed", "signal"])
+def test_run_third_interrupt_forces_chosen_abort(initialized_archive, choice):
+    """Direct run exits at once when Ctrl+C follows an explicit abort choice."""
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True", PLUGINS="chrome")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    crawl_id = next(record["crawl_id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook_pid():
+        with use_archivebox_db(initialized_archive):
+            hook = Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+            return hook.pid if hook is not None and hook.is_running else None
+
+    try:
+        result = run_archivebox_cmd(
+            ["run", f"--crawl-id={crawl_id}"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook_pid() is not None)
+        hook_pid = active_hook_pid()
+        assert hook_pid is not None
+        owned = psutil.Process(result.pid).children(recursive=True)
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        if choice == "typed":
+            os.write(master, b"a")
+        else:
+            result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Aborting crawl" in output, timeout=5)
+        forced_at = time.monotonic()
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: result.poll() is not None, timeout=5)
+        assert time.monotonic() - forced_at < 2.0
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert not pid_is_alive(hook_pid)
+        assert not [child.pid for child in owned if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=20)
+        os.close(slave)
+        os.close(master)
+
+
+@pytest.mark.django_db(transaction=True)
+def test_cli_run_retries_snapshot_after_filesystem_failure(initialized_archive):
+    from archivebox.core.models import ArchiveResult, Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="hashes")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    snapshot_id = next(record["id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        crawl_id = str(snapshot.crawl_id)
+        directory = snapshot.output_dir
+    directory.mkdir(parents=True, exist_ok=True)
+    (directory / "input.txt").write_text("Capture survives a temporary filesystem failure")
+    broken_directory = directory / "hashes"
+    broken_directory.symlink_to("hashes")
+
+    failed = run_archivebox_cmd(["run", f"--snapshot-id={snapshot_id}"], cwd=initialized_archive, env=env, timeout=60)
+    assert failed.returncode == 1, failed.stdout + failed.stderr
+    assert "FileExistsError: [Errno 17]" in failed.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        assert snapshot.status == Snapshot.StatusChoices.STARTED
+        assert snapshot.retry_at is not None
+        assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
+
+    broken_directory.unlink()
+    resumed = run_archivebox_cmd(["run", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=60)
+    assert resumed.returncode == 0, resumed.stdout + resumed.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        assert snapshot.status == Snapshot.StatusChoices.SEALED
+        result = snapshot.archiveresult_set.get(plugin="hashes")
+        assert result.status == ArchiveResult.StatusChoices.SUCCEEDED
+        assert result.output_size > 0
+    assert (directory / "hashes").is_dir()
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("precreate_snapshot", [False, True])
+def test_newest_crawl_runs_before_older_queued_and_started_crawls(initialized_archive, precreate_snapshot):
+    from datetime import timedelta
+
+    from django.utils import timezone
+
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="hashes")
+    old_url, queued_url, newest_url = create_test_url(), create_test_url(), create_test_url()
+    old = run_archivebox_cmd(["snapshot", "create", old_url], cwd=initialized_archive, env=env)
+    assert old.returncode == 0, old.stdout + old.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(url=old_url)
+        Crawl.objects.filter(pk=snapshot.crawl_id).update(status=Crawl.StatusChoices.STARTED)
+        Snapshot.objects.filter(pk=snapshot.pk).update(retry_at=timezone.now() - timedelta(days=1))
+    queued = run_archivebox_cmd(
+        ["snapshot" if precreate_snapshot else "crawl", "create", queued_url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert queued.returncode == 0, queued.stdout + queued.stderr
+    newest = run_archivebox_cmd(
+        ["snapshot" if precreate_snapshot else "crawl", "create", newest_url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert newest.returncode == 0, newest.stdout + newest.stderr
+
+    result = run_archivebox_cmd(["run", "--no-stdin"], cwd=initialized_archive, env=env, timeout=60)
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert all(url in result.stdout for url in (old_url, queued_url, newest_url))
+    assert result.stdout.index(newest_url) < result.stdout.index(queued_url) < result.stdout.index(old_url), result.stdout
+    with use_archivebox_db(initialized_archive):
+        assert (
+            Snapshot.objects.filter(
+                url__in=[old_url, queued_url, newest_url],
+                status=Snapshot.StatusChoices.SEALED,
+            ).count()
+            == 3
+        )
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.timeout(660)
+def test_due_snapshots_share_one_crawl_scoped_chrome(initialized_archive, recursive_test_site):
+    from django.utils import timezone
+
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    urls = [recursive_test_site["root_url"], recursive_test_site["child_urls"][0]]
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="\n".join(urls),
+            config={"PLUGINS": "chrome", "CHROME_ISOLATION": "crawl"},
+            created_by_id=get_or_create_system_user_pk(),
+            status=Crawl.StatusChoices.STARTED,
+            retry_at=timezone.now(),
+        )
+        for url in urls:
+            Snapshot.objects.create(url=url, crawl=crawl, status=Snapshot.StatusChoices.QUEUED, retry_at=timezone.now())
+        crawl_id = str(crawl.id)
+
+    result = run_archivebox_cmd(["run", "--no-stdin", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=600)
+    assert result.returncode == 0, result.stdout + result.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshots = Snapshot.objects.filter(crawl_id=crawl_id, url__in=urls, status=Snapshot.StatusChoices.SEALED)
+        assert snapshots.count() == 2
+        assert all((snapshot.output_dir / "chrome" / "navigation.json").is_file() for snapshot in snapshots)
+        launches = Process.objects.filter(
+            pwd__contains=crawl_id,
+            process_type=Process.TypeChoices.HOOK,
+            cmd__0__endswith="on_CrawlSetup__90_chrome_launch.daemon.bg.js",
+        )
+        assert launches.count() == 1, [(process.pid, process.exit_code) for process in launches]
+        assert launches.get().exit_code == 0
+
+
+@pytest.mark.django_db(transaction=True)
+def test_busy_crawl_does_not_report_snapshot_progress(initialized_archive):
+    from datetime import timedelta
+    from django.utils import timezone
+
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.services.runner import _run_due_snapshot_id
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/",
+            config={"PLUGINS": "chrome", "CHROME_ISOLATION": "crawl"},
+            created_by_id=get_or_create_system_user_pk(),
+            status=Crawl.StatusChoices.STARTED,
+            retry_at=timezone.now() + timedelta(minutes=1),
+        )
+        snapshot = Snapshot.objects.create(url=crawl.urls, crawl=crawl, retry_at=timezone.now())
+        assert _run_due_snapshot_id(str(snapshot.id), lock_seconds=60, interactive_interrupts=False, runtime_config=None) is False
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.timeout(660)
+def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archive, recursive_test_site):
+    from archivebox.core.models import Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+
+    result = run_archivebox_cmd(
+        ["snapshot", "create", recursive_test_site["root_url"]],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    records = parse_jsonl_output(result.stdout)
+    snapshot_id = next(record["id"] for record in records if record.get("type") == "Snapshot")
+    with use_archivebox_db(initialized_archive):
+        browser_state = Snapshot.objects.get(id=snapshot_id).output_dir / "chrome" / "browser.json"
+
+    run_log = initialized_archive / "run-signal-chrome.log"
+    run_log_handle = run_log.open("w", encoding="utf-8")
+    run_process = run_archivebox_cmd(
+        ["run", f"--snapshot-id={snapshot_id}"],
+        cwd=initialized_archive,
+        env=env,
+        stdout=run_log_handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+        wait=False,
+    )
+    run_log_handle.close()
+    try:
+        wait_for_log(browser_state, '"ready": true', timeout=120)
+        child_pids = [child.pid for child in psutil.Process(run_process.pid).children(recursive=True) if pid_is_alive(child.pid)]
+        assert child_pids
+
+        run_process.send_signal(signal.SIGTERM)
+        run_process.wait(timeout=30)
+        output = run_log.read_text(encoding="utf-8", errors="replace")
+        assert "Runner error" not in output
+        for pid in child_pids:
+            wait_for_pid_to_disappear(pid, timeout=15)
+    finally:
+        cleanup_process_group(run_process.pid)
+
+
+class TestRunWithArchiveResult:
+    """Tests for `archivebox run` with ArchiveResult input."""
+
+    @pytest.mark.django_db(transaction=True)
+    def test_run_treats_no_id_archiveresult_as_parent_snapshot_plugin_request(self, initialized_archive):
+        import json
+
+        from archivebox.core.models import ArchiveResult
+        from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+        create_result = run_archivebox_cmd(
+            ["snapshot", "create", create_test_url()],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        snapshot_id = next(record["id"] for record in parse_jsonl_output(create_result.stdout) if record.get("type") == "Snapshot")
+        missing_hook = "on_Snapshot__99_missing_favicon_hook"
+        request = json.dumps(
+            {
+                "type": "ArchiveResult",
+                "snapshot_id": snapshot_id,
+                "plugin": "favicon",
+                "hook_name": missing_hook,
+                "status": "queued",
+            },
+        )
+
+        result = run_archivebox_cmd(
+            ["run"],
+            stdin=f"{request}\n",
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+
+        assert result.returncode == 0, result.stderr or result.stdout
+        with use_archivebox_db(initialized_archive):
+            rows = list(
+                ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin="favicon").values_list(
+                    "hook_name",
+                    "status",
+                    "output_str",
+                ),
+            )
+        assert len(rows) == 1
+        assert rows[0][0] != missing_hook
+        assert rows[0][0].startswith("on_Snapshot__")
+        assert rows[0][1] in ArchiveResult.FINAL_STATES
+
+    def test_run_requeues_failed_archiveresult(self, initialized_archive):
+        """Run uses a failed ArchiveResult as a parent Snapshot/plugin reference."""
+        url = create_test_url()
+
+        # Create snapshot and archive result
+        _cmd_result = run_archivebox_cmd(
+            ["snapshot", "create", url],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout1, _, _ = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+        _cmd_result = run_archivebox_cmd(
+            ["archiveresult", "create", "--plugin=favicon"],
+            stdin=stdout1,
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout2, _, _ = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+        assert any(record.get("type") == "ArchiveResult" for record in parse_jsonl_output(stdout2))
+
+        initial_run = run_archivebox_cmd(
+            ["run"],
+            stdin=stdout2,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert initial_run.returncode == 0, initial_run.stderr
+        persisted_result = run_archivebox_cmd(
+            ["archiveresult", "list", "--plugin=favicon"],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert persisted_result.returncode == 0, persisted_result.stderr
+        assert any(record.get("type") == "ArchiveResult" for record in parse_jsonl_output(persisted_result.stdout))
+
+        # Update to failed
+        update_result = run_archivebox_cmd(
+            ["archiveresult", "update", "--status=failed"],
+            stdin=persisted_result.stdout,
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert update_result.returncode == 0, update_result.stderr
+        failed_result = run_archivebox_cmd(
+            ["archiveresult", "list", "--plugin=favicon"],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert failed_result.returncode == 0, failed_result.stderr
+        failed_records = [record for record in parse_jsonl_output(failed_result.stdout) if record.get("type") == "ArchiveResult"]
+        assert len(failed_records) == 1
+        assert failed_records[0]["status"] == "failed"
+        failed_jsonl = next(line for line in failed_result.stdout.splitlines() if failed_records[0]["id"] in line) + "\n"
+
+        # Now run should re-queue it
+        _cmd_result = run_archivebox_cmd(
+            ["run"],
+            stdin=failed_jsonl,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout3, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+
+        assert code == 0
+        records = parse_jsonl_output(stdout3)
+        ar_records = [r for r in records if r.get("type") == "ArchiveResult"]
+        assert len(ar_records) >= 1
+
+
+class TestRunMixedInput:
+    """Tests for `archivebox run` with mixed record types."""
+
+    def test_run_handles_mixed_records_emitted_by_cli(self, initialized_archive):
+        """Run handles real Crawl, Snapshot, and Tag records from CLI stages."""
+        tag_result = run_archivebox_cmd(
+            ["tag", "create", "mixed-run-tag"],
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert tag_result.returncode == 0, tag_result.stderr
+        crawl_result = run_archivebox_cmd(
+            ["crawl", "create", create_test_url()],
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert crawl_result.returncode == 0, crawl_result.stderr
+        snapshot_result = run_archivebox_cmd(
+            ["snapshot", "create"],
+            stdin=crawl_result.stdout,
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert snapshot_result.returncode == 0, snapshot_result.stderr
+
+        result = run_archivebox_cmd(
+            ["run"],
+            stdin=tag_result.stdout + snapshot_result.stdout,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
+
+        types = {record.get("type") for record in records}
+        assert {"Crawl", "Snapshot", "Tag"}.issubset(types)
@@ -30,6 +30,7 @@ from archivebox.tests.conftest import (
    run_archivebox_cmd,
    start_archivebox_server,
    stop_archivebox_process,
+    wait_for_log,
    wait_for_log_count,
    wait_for_pid_to_disappear,
 )
@@ -588,7 +589,7 @@ def test_live_server_machine_search_engine_update_reaches_subsequent_snapshot_ru
                "from archivebox.crawls.models import Crawl;"
                "from archivebox.core.models import Snapshot;"
                "from archivebox.machine.models import Machine;"
-                "machine = Machine.current(refresh=True);"
+                "machine = Machine.current();"
                "machine.config = {**dict(machine.config or {}), 'SEARCH_BACKEND_ENGINE': 'sqlite'};"
                "machine.save(update_fields=['config', 'modified_at']);"
                "crawl = Crawl.objects.create("
@@ -813,26 +814,32 @@ def test_live_server_signal_exit_and_resume_uses_existing_supervisor_state(initi
    try:
        server = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}.log", env=env)
        server_log = server.log_path
+        # HTTP readiness can precede the foreground tail scope. Exercise its
+        # actual signal policy, where interrupting Popen.poll stranded reaping.
+        wait_for_log(server_log, "Tailing worker logs (Ctrl+C to stop)...")

        os.kill(server.pid, stop_signal)
        server.wait(timeout=20)

        if expected_notice:
            log_text = server_log.read_text(encoding="utf-8", errors="replace")
-            assert expected_notice in log_text
-            assert "ArchiveBox server shut down gracefully" in log_text
+            assert server.returncode == 0, log_text
+            assert expected_notice in log_text, log_text
+            assert "ArchiveBox server shut down gracefully" in log_text, log_text
            assert_no_processes_for_data_dir(initialized_archive, timeout=12)

        resumed = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}-resumed.log", env=env)
        resumed_log = resumed.log_path
+        wait_for_log(resumed_log, "Tailing worker logs (Ctrl+C to stop)...")
        result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
        assert result.returncode == 0, result.stderr or result.stdout

        os.kill(resumed.pid, signal.SIGTERM)
        resumed.wait(timeout=20)
        resumed_text = resumed_log.read_text(encoding="utf-8", errors="replace")
-        assert "Got SIGTERM" in resumed_text
-        assert "ArchiveBox server shut down gracefully" in resumed_text
+        assert resumed.returncode == 0, resumed_text
+        assert "Got SIGTERM" in resumed_text, resumed_text
+        assert "ArchiveBox server shut down gracefully" in resumed_text, resumed_text
        assert_no_processes_for_data_dir(initialized_archive, timeout=12)
    finally:
        for proc in (server, resumed):
@@ -0,0 +1,61 @@
+"""Takeover preserves a capture; an explicit user abort ends its attempt."""
+
+import asyncio
+
+import pytest
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("user_initiated", [False, True], ids=["takeover", "user-abort"])
+def test_abort_before_first_snapshot_hook_preserves_capture_intent(recursive_test_site, user_initiated):
+    from abx_dl.events import CrawlAbortEvent, CrawlCompletedEvent, ProcessStartedEvent, SnapshotCompletedEvent, SnapshotEvent
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.services.runner import CrawlRunner
+
+    crawl = Crawl.objects.create(
+        urls=recursive_test_site["root_url"],
+        config={"PLUGINS": "wget"},
+        created_by_id=get_or_create_system_user_pk(),
+    )
+    runner = CrawlRunner(crawl)
+    completed = []
+    started = []
+    completed_crawls = []
+
+    async def abort_on_snapshot(event: SnapshotEvent) -> None:
+        # Observe the real snapshot command after its DB owner is recorded but
+        # before the hook service consumes it. This is the takeover window that
+        # used to turn shutdown cleanup into a successful, empty capture.
+        await event.emit(CrawlAbortEvent(user_initiated=user_initiated)).now()
+
+    runner.bus.on(SnapshotEvent, abort_on_snapshot)
+    runner.bus.on(SnapshotCompletedEvent, lambda event: completed.append(event))
+    runner.bus.on(ProcessStartedEvent, lambda event: started.append(event))
+    runner.bus.on(CrawlCompletedEvent, lambda event: completed_crawls.append(event))
+    if user_initiated:
+        with pytest.raises(KeyboardInterrupt):
+            asyncio.run(runner.run())
+    else:
+        asyncio.run(runner.run())
+
+    assert len(completed) == 1, "Aborting must still finish the real snapshot cleanup phase"
+    assert not [event for event in started if event.hook_name.startswith("on_Snapshot")]
+    snapshot = Snapshot.objects.get(crawl=crawl)
+    crawl.refresh_from_db()
+    assert len(completed_crawls) == int(user_initiated)
+    assert snapshot.archiveresult_set.count() == 0
+    if user_initiated:
+        # Preserve the historical point-in-time boundary for an explicit abort.
+        assert snapshot.status == Snapshot.StatusChoices.SEALED
+        assert snapshot.retry_at is None
+        assert snapshot.downloaded_at is not None
+        assert crawl.status == Crawl.StatusChoices.SEALED
+        assert crawl.retry_at is None
+    else:
+        assert snapshot.status in Snapshot.RUNNABLE_STATES, "Takeover must leave the capture resumable"
+        assert snapshot.retry_at is not None
+        assert snapshot.downloaded_at is None
+        assert crawl.status in Crawl.RUNNABLE_STATES
+        assert crawl.retry_at is not None
@@ -194,6 +194,16 @@ class TestRequiredBinaryConfigHandling:

        assert Path(resolved["WGET_BINARY"]).is_absolute()
        assert Path(resolved["WGET_BINARY"]).is_file()
+        runtime_env = {**os.environ, **resolved}
+        assert set(os.environ["PATH"].split(os.pathsep)).issubset(runtime_env["PATH"].split(os.pathsep))
+        result = subprocess.run(
+            ["/usr/bin/env", "abxpkg", "--version"],
+            env=runtime_env,
+            capture_output=True,
+            text=True,
+            timeout=30,
+        )
+        assert result.returncode == 0, result.stderr

    @pytest.mark.django_db(transaction=True)
    def test_binary_env_var_name_only_handling(self, hermetic_lib_dir):
@@ -174,21 +174,32 @@ class TestMachineModel:

        assert machine1.id == machine2.id

-    def test_machine_current_refreshes_after_interval(self):
-        """Machine.current() should refresh after recheck interval."""
-        import archivebox.machine.models as models
+    def test_machine_current_does_not_refresh_after_interval(self):
+        """A long-lived process must not turn a cached machine read into writes."""
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext

        machine1 = Machine.current()
-
-        # Manually expire the cache by modifying modified_at
        machine1.modified_at = timezone.now() - timedelta(seconds=MACHINE_RECHECK_INTERVAL + 1)
-        machine1.save()
-        models._CURRENT_MACHINE = machine1
+        with CaptureQueriesContext(connection) as queries:
+            machine2 = Machine.current()
+            assert Machine.current() is machine1
+        assert machine2 is machine1
+        assert list(queries) == []

-        machine2 = Machine.current()
+    def test_machine_config_save_updates_cache_without_reinitializing(self):
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext

-        # Should have fetched/updated the machine (same GUID)
-        assert machine1.guid == machine2.guid
+        cached = Machine.current()
+        edited = Machine.objects.get(pk=cached.pk)
+        edited.config = {**edited.config, "USER_AGENT": "read-only-machine-test"}
+        edited.save(update_fields=["config"])
+        with CaptureQueriesContext(connection) as queries:
+            current = Machine.current()
+        assert current is cached
+        assert current.config["USER_AGENT"] == "read-only-machine-test"
+        assert list(queries) == []

    def test_machine_current_recreates_stale_cached_row(self):
        """Machine.current() should recreate the cached machine if the row was deleted."""
@@ -284,9 +295,9 @@ class TestMachineModel:
            "WGET_BINARY": "/tmp/archivebox-test-missing-wget",
        }
        machine.save(update_fields=["config"])
-        models._CURRENT_MACHINE = machine
+        models._CURRENT_MACHINE = None

-        refreshed = Machine.current(refresh=True)
+        refreshed = Machine.current()

        # Valid binary paths inside ABXPKG_LIB_DIR survive.
        assert refreshed.config.get("CHROME_BINARY") == str(chrome_path)
@@ -12,12 +12,18 @@ Migration tests from 0.8.x to 0.9.x.

 import sqlite3
 import json
+import signal
+import subprocess
+import time
 import uuid
 from datetime import datetime
 from urllib.parse import urlparse

+import psutil
 import pytest

+from archivebox.tests.conftest import cli_env, run_archivebox_cmd, stop_archivebox_process
+
 from .migrations_helpers import (
    SCHEMA_0_7,
    SCHEMA_0_8,
@@ -183,36 +189,28 @@ def test_migration_preserves_supported_extended_08_metadata(migration_08_data, u
    assert migrated_counts == expected_counts


-def test_migration_preserves_snapshot_count(migration_08_data):
-    """Migration should preserve all snapshots from 0.8.x."""
+def test_migration_preserves_08_collection(migration_08_data):
+    """One real upgrade preserves every field and relationship in the same collection."""
    work_dir, db_path, original_data = migration_08_data
-    expected_count = len(original_data["snapshots"])
+    with sqlite3.connect(db_path) as conn:
+        original_tag_count = conn.execute("SELECT COUNT(*) FROM core_snapshot_tags").fetchone()[0]

    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
    assert result.returncode == 0, f"Init failed: {result.stderr}"

+    # migration_preserves_snapshot_count
+    expected_count = len(original_data["snapshots"])
+
    ok, msg = verify_snapshot_count(db_path, expected_count)
    assert ok, msg

-
-def test_migration_preserves_snapshot_urls(migration_08_data):
-    """Migration should preserve all snapshot URLs from 0.8.x."""
-    work_dir, db_path, original_data = migration_08_data
+    # migration_preserves_snapshot_urls
    expected_urls = [s["url"] for s in original_data["snapshots"]]

-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
    ok, msg = verify_snapshot_urls(db_path, expected_urls)
    assert ok, msg

-
-def test_migration_preserves_crawls(migration_08_data):
-    """Migration should preserve all Crawl records and create default crawl if needed."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
+    # migration_preserves_crawls
    # Count snapshots with NULL crawl_id in original data
    snapshots_without_crawl = sum(1 for s in original_data["snapshots"] if s["crawl_id"] is None)

@@ -224,13 +222,7 @@ def test_migration_preserves_crawls(migration_08_data):
    ok, msg = verify_crawl_count(db_path, expected_count)
    assert ok, msg

-
-def test_migration_preserves_snapshot_crawl_links(migration_08_data):
-    """Migration should preserve snapshot-to-crawl relationships and assign default crawl to orphans."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
+    # migration_preserves_snapshot_crawl_links
    conn = sqlite3.connect(str(db_path))
    cursor = conn.cursor()

@@ -249,20 +241,11 @@ def test_migration_preserves_snapshot_crawl_links(migration_08_data):

    conn.close()

-
-def test_migration_preserves_tags(migration_08_data):
-    """Migration should preserve all tags."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
+    # migration_preserves_tags
    ok, msg = verify_tag_count(db_path, len(original_data["tags"]))
    assert ok, msg

-
-def test_migration_preserves_archiveresults(migration_08_data):
-    """Migration should preserve ArchiveResult rows and link each one to a Process."""
-    work_dir, db_path, original_data = migration_08_data
+    # migration_preserves_archiveresults
    expected_count = len(original_data["archiveresults"])
    expected_counts = {}
    for result in original_data["archiveresults"]:
@@ -270,9 +253,6 @@ def test_migration_preserves_archiveresults(migration_08_data):
        key = (result["extractor"], status)
        expected_counts[key] = expected_counts.get(key, 0) + 1

-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
    ok, msg = verify_archiveresult_count(db_path, expected_count)
    assert ok, msg

@@ -290,6 +270,255 @@ def test_migration_preserves_archiveresults(migration_08_data):
    assert missing_process_count == 0
    assert process_count == expected_count

+    # migration_preserves_archiveresult_status
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+
+    # Get status counts
+    cursor.execute("SELECT status, COUNT(*) FROM core_archiveresult GROUP BY status")
+    status_counts = dict(cursor.fetchall())
+    conn.close()
+
+    # Original data has known status distribution: succeeded, failed, skipped
+    assert "succeeded" in status_counts, "Should have succeeded results"
+    assert "failed" in status_counts, "Should have failed results"
+    assert "skipped" in status_counts, "Should have skipped results"
+
+    # migration_preserves_snapshot_titles
+    expected_titles = {s["url"]: s["title"] for s in original_data["snapshots"]}
+
+    ok, msg = verify_snapshot_titles(db_path, expected_titles)
+    assert ok, msg
+
+    # migration_preserves_foreign_keys
+    ok, msg = verify_foreign_keys(db_path)
+    assert ok, msg
+
+    # migration_creates_process_records
+    # Verify Process records created
+    expected_count = len(original_data["archiveresults"])
+    ok, msg = verify_process_migration(db_path, expected_count)
+    assert ok, msg
+
+    # migration_creates_binary_records
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+
+    # Check Binary records exist
+    cursor.execute("SELECT COUNT(*) FROM machine_binary")
+    binary_count = cursor.fetchone()[0]
+
+    # Should have at least one binary per unique extractor
+    extractors = {ar["extractor"] for ar in original_data["archiveresults"]}
+    assert binary_count >= len(extractors), f"Expected at least {len(extractors)} Binaries, got {binary_count}"
+
+    cursor.execute("""
+        SELECT COUNT(*)
+        FROM machine_process
+        WHERE cmd != '[]' AND binary_id IS NULL
+    """)
+    missing_binary_count = cursor.fetchone()[0]
+    assert missing_binary_count == 0
+
+    cursor.execute("""
+        SELECT p.cmd, b.name, b.abspath
+        FROM machine_process p
+        JOIN machine_binary b ON p.binary_id = b.id
+        WHERE p.cmd != '[]'
+    """)
+    rows = cursor.fetchall()
+    assert rows
+    for cmd_raw, binary_name, binary_abspath in rows:
+        cmd = json.loads(cmd_raw)
+        assert binary_name == cmd[0]
+        assert binary_abspath == cmd[0]
+
+    cursor.execute("SELECT COUNT(*) FROM machine_process WHERE iface_id IS NULL")
+    missing_iface_count = cursor.fetchone()[0]
+    assert missing_iface_count == 0
+
+    conn.close()
+
+    # migration_preserves_cmd_data
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+
+    # Check that Process records have cmd arrays
+    cursor.execute("SELECT cmd FROM machine_process WHERE cmd != '[]'")
+    cmd_records = cursor.fetchall()
+
+    # All Processes should have non-empty cmd (test data has json.dumps([extractor, '--version']))
+    expected_count = len(original_data["archiveresults"])
+    assert len(cmd_records) == expected_count, f"Expected {expected_count} Processes with cmd, got {len(cmd_records)}"
+
+    conn.close()
+
+    # no_duplicate_snapshots_after_migration
+    # Check for duplicate URLs
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+    cursor.execute("""
+        SELECT url, COUNT(*) as cnt FROM core_snapshot
+        GROUP BY url HAVING cnt > 1
+    """)
+    duplicates = cursor.fetchall()
+    conn.close()
+
+    assert len(duplicates) == 0, f"Found duplicate URLs: {duplicates}"
+
+    # no_orphaned_archiveresults_after_migration
+    ok, msg = verify_foreign_keys(db_path)
+    assert ok, msg
+
+    # timestamps_preserved_after_migration
+    original_timestamps = {s["url"]: s["timestamp"] for s in original_data["snapshots"]}
+
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+    cursor.execute("SELECT url, timestamp FROM core_snapshot")
+    migrated_timestamps = {row[0]: row[1] for row in cursor.fetchall()}
+    conn.close()
+
+    for url, original_ts in original_timestamps.items():
+        assert migrated_timestamps.get(url) == original_ts, f"Timestamp changed for {url}: {original_ts} -> {migrated_timestamps.get(url)}"
+
+    # crawl_data_preserved_after_migration
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+
+    # Check each crawl's data is preserved
+    for crawl in original_data["crawls"]:
+        cursor.execute("SELECT urls, label, status FROM crawls_crawl WHERE id = ?", (crawl["id"],))
+        row = cursor.fetchone()
+        assert row is not None, f"Crawl {crawl['id']} not found after migration"
+        assert row[0] == crawl["urls"], f"URLs mismatch for crawl {crawl['id']}"
+        assert row[1] == crawl["label"], f"Label mismatch for crawl {crawl['id']}"
+        assert row[2] == crawl["status"], f"Status mismatch for crawl {crawl['id']}"
+
+    conn.close()
+
+    # tag_associations_preserved_after_migration
+    # Count tag associations after migration
+    conn = sqlite3.connect(str(db_path))
+    cursor = conn.cursor()
+    cursor.execute("SELECT COUNT(*) FROM core_snapshot_tags")
+    migrated_count = cursor.fetchone()[0]
+    conn.close()
+
+    assert migrated_count == original_tag_count, f"Tag associations changed: {original_tag_count} -> {migrated_count}"
+
+
+def test_legacy_default_crawl_is_sealed_without_a_due_retry(tmp_path):
+    """Upgrading a legacy collection must not leave its sealed crawl runnable."""
+    work_dir = tmp_path
+    db_path = work_dir / "index.sqlite3"
+    create_data_dir_structure(work_dir)
+    with sqlite3.connect(db_path) as conn:
+        conn.executescript(SCHEMA_0_7)
+    seed_0_7_data(db_path)
+
+    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=90)
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with sqlite3.connect(db_path) as conn:
+        rows = conn.execute("SELECT status, retry_at FROM crawls_crawl").fetchall()
+        snapshot_statuses = conn.execute("SELECT DISTINCT status FROM core_snapshot").fetchall()
+        result_statuses = conn.execute("SELECT DISTINCT status FROM core_archiveresult").fetchall()
+    assert rows == [("sealed", None)]
+    assert snapshot_statuses == [("sealed",)]
+    assert set(result_statuses) == {("succeeded",), ("failed",), ("skipped",)}
+
+
+def test_old_migrated_crawl_stays_idle_and_forward_migration_repairs_it(tmp_path):
+    """A dated raw ISO retry from an already-upgraded collection must not spin."""
+    work_dir = tmp_path
+    db_path = work_dir / "index.sqlite3"
+    create_data_dir_structure(work_dir)
+    with sqlite3.connect(db_path) as conn:
+        conn.executescript(SCHEMA_0_7)
+    seed_0_7_data(db_path)
+
+    before_repair = run_archivebox_migration_cmd(work_dir, ["manage", "migrate", "core", "0054", "--noinput"], timeout=90)
+    assert before_repair.returncode == 0, before_repair.stderr or before_repair.stdout
+    with sqlite3.connect(db_path) as conn:
+        # 0024 used datetime.now().isoformat() for this sealed Crawl. Represent
+        # a collection upgraded before today so SQLite's lexical due query sees it.
+        conn.execute("UPDATE crawls_crawl SET retry_at = '2024-01-01T00:00:00.123456'")
+        assert conn.execute("SELECT status, retry_at FROM crawls_crawl").fetchall() == [
+            ("sealed", "2024-01-01T00:00:00.123456"),
+        ]
+        assert conn.execute("SELECT DISTINCT status FROM core_snapshot").fetchall() == [("sealed",)]
+
+    env = cli_env(disable_extractors=True)
+    log_path = work_dir / "idle-daemon.log"
+    with log_path.open("w") as log:
+        daemon = run_archivebox_cmd(
+            ["run", "--daemon"],
+            cwd=work_dir,
+            env=env,
+            stdout=log,
+            stderr=subprocess.STDOUT,
+            capture_output=False,
+            wait=False,
+            start_new_session=True,
+        )
+    try:
+        ready_deadline = time.monotonic() + 20
+        while True:
+            with sqlite3.connect(db_path) as conn:
+                ready = conn.execute(
+                    "SELECT 1 FROM machine_process WHERE process_type = 'orchestrator' AND status = 'running' AND pid = ?",
+                    (daemon.pid,),
+                ).fetchone()
+            if ready:
+                break
+            assert daemon.poll() is None, log_path.read_text()
+            assert time.monotonic() < ready_deadline, log_path.read_text()
+            time.sleep(0.1)
+        cpu_percent = psutil.Process(daemon.pid).cpu_percent(interval=5)
+        assert cpu_percent < 30, f"idle daemon used {cpu_percent}% CPU: {log_path.read_text()}"
+    finally:
+        stop_archivebox_process(daemon, signal.SIGTERM)
+
+    repaired = run_archivebox_migration_cmd(work_dir, ["init"], timeout=90)
+    assert repaired.returncode == 0, repaired.stderr or repaired.stdout
+    with sqlite3.connect(db_path) as conn:
+        assert conn.execute("SELECT status, retry_at FROM crawls_crawl").fetchall() == [("sealed", None)]
+
+
+def test_forward_migration_normalizes_active_legacy_retry_timestamps(migration_08_data):
+    """Preserve instants for active rows while leaving canonical timestamps intact."""
+    work_dir, db_path, _original_data = migration_08_data
+    before_repair = run_archivebox_migration_cmd(work_dir, ["manage", "migrate", "core", "0054", "--noinput"], timeout=90)
+    assert before_repair.returncode == 0, before_repair.stderr or before_repair.stdout
+
+    old_offset_timestamp = "2024-01-01T03:04:05.123456+03:00"
+    expected_utc_timestamp = "2024-01-01 00:04:05.123456"
+    canonical_timestamp = "2025-04-05 06:07:08.123456"
+    with sqlite3.connect(db_path) as conn:
+        crawl_ids = [row[0] for row in conn.execute("SELECT id FROM crawls_crawl WHERE status = 'queued' ORDER BY id")]
+        snapshot_ids = [row[0] for row in conn.execute("SELECT id FROM core_snapshot ORDER BY id")]
+        binary_id = conn.execute("SELECT id FROM machine_binary ORDER BY id LIMIT 1").fetchone()[0]
+        process_id = conn.execute("SELECT id FROM machine_process ORDER BY id LIMIT 1").fetchone()[0]
+        assert len(crawl_ids) == 2
+        assert len(snapshot_ids) >= 2
+        conn.execute("UPDATE crawls_crawl SET retry_at = ? WHERE id = ?", (old_offset_timestamp, crawl_ids[0]))
+        conn.execute("UPDATE crawls_crawl SET retry_at = ? WHERE id = ?", (canonical_timestamp, crawl_ids[1]))
+        conn.execute("UPDATE core_snapshot SET retry_at = ? WHERE id = ?", (old_offset_timestamp, snapshot_ids[0]))
+        conn.execute("UPDATE core_snapshot SET retry_at = ? WHERE id = ?", (canonical_timestamp, snapshot_ids[1]))
+        conn.execute("UPDATE machine_binary SET retry_at = ? WHERE id = ?", (old_offset_timestamp, binary_id))
+        conn.execute("UPDATE machine_process SET retry_at = ? WHERE id = ?", (old_offset_timestamp, process_id))
+
+    repaired = run_archivebox_migration_cmd(work_dir, ["init"], timeout=90)
+    assert repaired.returncode == 0, repaired.stderr or repaired.stdout
+    with sqlite3.connect(db_path) as conn:
+        assert conn.execute("SELECT retry_at FROM crawls_crawl WHERE id = ?", (crawl_ids[0],)).fetchone() == (expected_utc_timestamp,)
+        assert conn.execute("SELECT retry_at FROM crawls_crawl WHERE id = ?", (crawl_ids[1],)).fetchone() == (canonical_timestamp,)
+        assert conn.execute("SELECT retry_at FROM core_snapshot WHERE id = ?", (snapshot_ids[0],)).fetchone() == (expected_utc_timestamp,)
+        assert conn.execute("SELECT retry_at FROM core_snapshot WHERE id = ?", (snapshot_ids[1],)).fetchone() == (canonical_timestamp,)
+        assert conn.execute("SELECT retry_at FROM machine_binary WHERE id = ?", (binary_id,)).fetchone() == (expected_utc_timestamp,)
+        assert conn.execute("SELECT retry_at FROM machine_process WHERE id = ?", (process_id,)).fetchone() == (expected_utc_timestamp,)
+

 def test_migration_preserves_users_groups_permissions_api_secrets_and_repeated_init(migration_08_data):
    work_dir, db_path, original_data = migration_08_data
@@ -328,26 +557,6 @@ def test_process_metadata_migration_rolls_back_completely_on_malformed_row(migra
        assert conn.execute("SELECT COUNT(*) FROM machine_binary").fetchone()[0] == 0


-def test_migration_preserves_archiveresult_status(migration_08_data):
-    """Migration should preserve archive result status values."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-
-    # Get status counts
-    cursor.execute("SELECT status, COUNT(*) FROM core_archiveresult GROUP BY status")
-    status_counts = dict(cursor.fetchall())
-    conn.close()
-
-    # Original data has known status distribution: succeeded, failed, skipped
-    assert "succeeded" in status_counts, "Should have succeeded results"
-    assert "failed" in status_counts, "Should have failed results"
-    assert "skipped" in status_counts, "Should have skipped results"
-
-
 def test_status_works_after_migration(migration_08_data):
    """Status command should work after migration."""
    work_dir, db_path, original_data = migration_08_data
@@ -388,28 +597,6 @@ def test_search_works_after_migration(migration_08_data):
    assert ok, msg


-def test_migration_preserves_snapshot_titles(migration_08_data):
-    """Migration should preserve all snapshot titles."""
-    work_dir, db_path, original_data = migration_08_data
-    expected_titles = {s["url"]: s["title"] for s in original_data["snapshots"]}
-
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    ok, msg = verify_snapshot_titles(db_path, expected_titles)
-    assert ok, msg
-
-
-def test_migration_preserves_foreign_keys(migration_08_data):
-    """Migration should maintain foreign key relationships."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    ok, msg = verify_foreign_keys(db_path)
-    assert ok, msg
-
-
@pytest.mark.parametrize("has_snapshot_status", (True, False), ids=("with-status", "without-status"))
 def test_migration_preserves_08_timestamp_meanings(migration_08_data, has_snapshot_status):
    """0.8.x already has separated timestamp/bookmarked_at/created_at/downloaded_at fields."""
@@ -534,174 +721,6 @@ def test_version_works_after_migration(migration_08_data):
    assert "ArchiveBox" in output or "version" in output.lower(), f"Version output missing expected content: {output[:500]}"


-def test_migration_creates_process_records(migration_08_data):
-    """Migration should create Process records for all ArchiveResults."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    # Verify Process records created
-    expected_count = len(original_data["archiveresults"])
-    ok, msg = verify_process_migration(db_path, expected_count)
-    assert ok, msg
-
-
-def test_migration_creates_binary_records(migration_08_data):
-    """Migration should create and link Binary/NetworkInterface records from migrated Process data."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-
-    # Check Binary records exist
-    cursor.execute("SELECT COUNT(*) FROM machine_binary")
-    binary_count = cursor.fetchone()[0]
-
-    # Should have at least one binary per unique extractor
-    extractors = {ar["extractor"] for ar in original_data["archiveresults"]}
-    assert binary_count >= len(extractors), f"Expected at least {len(extractors)} Binaries, got {binary_count}"
-
-    cursor.execute("""
-        SELECT COUNT(*)
-        FROM machine_process
-        WHERE cmd != '[]' AND binary_id IS NULL
-    """)
-    missing_binary_count = cursor.fetchone()[0]
-    assert missing_binary_count == 0
-
-    cursor.execute("""
-        SELECT p.cmd, b.name, b.abspath
-        FROM machine_process p
-        JOIN machine_binary b ON p.binary_id = b.id
-        WHERE p.cmd != '[]'
-    """)
-    rows = cursor.fetchall()
-    assert rows
-    for cmd_raw, binary_name, binary_abspath in rows:
-        cmd = json.loads(cmd_raw)
-        assert binary_name == cmd[0]
-        assert binary_abspath == cmd[0]
-
-    cursor.execute("SELECT COUNT(*) FROM machine_process WHERE iface_id IS NULL")
-    missing_iface_count = cursor.fetchone()[0]
-    assert missing_iface_count == 0
-
-    conn.close()
-
-
-def test_migration_preserves_cmd_data(migration_08_data):
-    """Migration should preserve cmd data in Process.cmd field."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-
-    # Check that Process records have cmd arrays
-    cursor.execute("SELECT cmd FROM machine_process WHERE cmd != '[]'")
-    cmd_records = cursor.fetchall()
-
-    # All Processes should have non-empty cmd (test data has json.dumps([extractor, '--version']))
-    expected_count = len(original_data["archiveresults"])
-    assert len(cmd_records) == expected_count, f"Expected {expected_count} Processes with cmd, got {len(cmd_records)}"
-
-    conn.close()
-
-
-def test_no_duplicate_snapshots_after_migration(migration_08_data):
-    """Migration should not create duplicate snapshots."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    # Check for duplicate URLs
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-    cursor.execute("""
-        SELECT url, COUNT(*) as cnt FROM core_snapshot
-        GROUP BY url HAVING cnt > 1
-    """)
-    duplicates = cursor.fetchall()
-    conn.close()
-
-    assert len(duplicates) == 0, f"Found duplicate URLs: {duplicates}"
-
-
-def test_no_orphaned_archiveresults_after_migration(migration_08_data):
-    """Migration should not leave orphaned ArchiveResults."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    ok, msg = verify_foreign_keys(db_path)
-    assert ok, msg
-
-
-def test_timestamps_preserved_after_migration(migration_08_data):
-    """Migration should preserve original timestamps."""
-    work_dir, db_path, original_data = migration_08_data
-    original_timestamps = {s["url"]: s["timestamp"] for s in original_data["snapshots"]}
-
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-    cursor.execute("SELECT url, timestamp FROM core_snapshot")
-    migrated_timestamps = {row[0]: row[1] for row in cursor.fetchall()}
-    conn.close()
-
-    for url, original_ts in original_timestamps.items():
-        assert migrated_timestamps.get(url) == original_ts, f"Timestamp changed for {url}: {original_ts} -> {migrated_timestamps.get(url)}"
-
-
-def test_crawl_data_preserved_after_migration(migration_08_data):
-    """Migration should preserve crawl metadata (urls, label, status)."""
-    work_dir, db_path, original_data = migration_08_data
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-
-    # Check each crawl's data is preserved
-    for crawl in original_data["crawls"]:
-        cursor.execute("SELECT urls, label, status FROM crawls_crawl WHERE id = ?", (crawl["id"],))
-        row = cursor.fetchone()
-        assert row is not None, f"Crawl {crawl['id']} not found after migration"
-        assert row[0] == crawl["urls"], f"URLs mismatch for crawl {crawl['id']}"
-        assert row[1] == crawl["label"], f"Label mismatch for crawl {crawl['id']}"
-        assert row[2] == crawl["status"], f"Status mismatch for crawl {crawl['id']}"
-
-    conn.close()
-
-
-def test_tag_associations_preserved_after_migration(migration_08_data):
-    """Migration should preserve snapshot-tag associations."""
-    work_dir, db_path, original_data = migration_08_data
-    # Count tag associations before migration
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-    cursor.execute("SELECT COUNT(*) FROM core_snapshot_tags")
-    original_count = cursor.fetchone()[0]
-    conn.close()
-
-    result = run_archivebox_migration_cmd(work_dir, ["init"], timeout=45)
-    assert result.returncode == 0, f"Init failed: {result.stderr}"
-
-    # Count tag associations after migration
-    conn = sqlite3.connect(str(db_path))
-    cursor = conn.cursor()
-    cursor.execute("SELECT COUNT(*) FROM core_snapshot_tags")
-    migrated_count = cursor.fetchone()[0]
-    conn.close()
-
-    assert migrated_count == original_count, f"Tag associations changed: {original_count} -> {migrated_count}"
-
-
 def test_update_migrates_db_snapshot_when_legacy_index_missing(tmp_path):
    """A legacy folder with no index file should still migrate if its timestamp exists in DB."""
    work_dir = tmp_path
@@ -1,695 +0,0 @@
-import asyncio
-import json
-import os
-import re
-import shutil
-import socket
-from concurrent.futures import ThreadPoolExecutor
-from concurrent.futures import TimeoutError as FutureTimeoutError
-from pathlib import Path
-from types import SimpleNamespace
-from urllib.parse import parse_qs, quote, urlsplit
-
-import pytest
-import requests
-from asgiref.testing import ApplicationCommunicator
-
-from archivebox.tests.conftest import ADMIN_TEST_HOST, run_archivebox_cmd
-from archivebox.config.common import get_config
-
-
-pytestmark = pytest.mark.django_db(transaction=True)
-
-
-def _free_port() -> int:
-    with socket.socket() as sock:
-        sock.bind(("127.0.0.1", 0))
-        return int(sock.getsockname()[1])
-
-
-def _reset_runtime_config() -> None:
-    from archivebox.config import common
-    from archivebox.machine.models import Machine
-
-    for value in vars(common).values():
-        cache_clear = getattr(value, "cache_clear", None)
-        if cache_clear is not None:
-            cache_clear()
-    Machine.current(refresh=True)
-
-
-def _set_archivebox_config(data_dir: Path, *values: str, env: dict[str, str] | None = None) -> None:
-    os.chdir(data_dir)
-    result = run_archivebox_cmd(
-        ["config", "--set", *values],
-        cwd=data_dir,
-        env=env,
-        timeout=120,
-    )
-    assert result.returncode == 0, result.stderr or result.stdout
-    _reset_runtime_config()
-
-
-@pytest.fixture
-def opencode_archive_config(initialized_archive):
-    port = _free_port()
-    state_dir = initialized_archive / "opencode"
-    env = os.environ.copy()
-    env.update(
-        {
-            "ARCHIVEBOX_ALLOW_NO_UNIX_SOCKETS": "true",
-            "OPENCODE_ENABLED": "True",
-            "OPENCODE_HOST": "127.0.0.1",
-            "OPENCODE_PORT": str(port),
-            "OPENCODE_WORKDIR": str(initialized_archive),
-            "OPENCODE_STATE_DIR": str(state_dir),
-            "OPENCODE_TIMEOUT": "60",
-        },
-    )
-    _set_archivebox_config(
-        initialized_archive,
-        "OPENCODE_ENABLED=True",
-        "OPENCODE_HOST=127.0.0.1",
-        f"OPENCODE_PORT={port}",
-        f"OPENCODE_WORKDIR={initialized_archive}",
-        f"OPENCODE_STATE_DIR={state_dir}",
-        "OPENCODE_TIMEOUT=60",
-        env=env,
-    )
-    return SimpleNamespace(data_dir=initialized_archive, port=port, state_dir=state_dir, env=env)
-
-
-@pytest.fixture
-def installed_opencode(opencode_archive_config):
-    from abx_plugins.plugins.opencode import runtime
-
-    install = run_archivebox_cmd(
-        ["install", "opencode", "--binproviders=env,pnpm"],
-        cwd=opencode_archive_config.data_dir,
-        env=opencode_archive_config.env,
-        timeout=1200,
-    )
-    assert install.returncode == 0, install.stderr or install.stdout
-    _reset_runtime_config()
-
-    config = get_config().model_dump(mode="json")
-    settings = runtime._settings(config, opencode_archive_config.data_dir)
-    settings["archivebox_base_url"] = "http://admin.archivebox.localhost:5797"
-    settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:5797/admin"
-    settings["archivebox_api_url"] = "http://admin.archivebox.localhost:5797/api/"
-    binary, binary_env = runtime._resolve_binary(settings["binary"], settings["config"])
-    version = binary.exec(
-        cmd=("--version",),
-        env={**os.environ, **binary_env},
-        timeout=120,
-    )
-    assert version.returncode == 0, version.stderr or version.stdout
-    return SimpleNamespace(config=opencode_archive_config, settings=settings)
-
-
-@pytest.fixture
-def live_opencode(installed_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    settings = installed_opencode.settings
-    ok, error = runtime._ensure_opencode(settings)
-    assert ok, error
-
-    process = runtime._PROCESS
-    assert process is not None
-    try:
-        yield SimpleNamespace(config=installed_opencode.config, settings=settings, process=process)
-    finally:
-        runtime._stop_owned_process()
-
-
-def test_opencode_disabled_route_does_not_start_server(client, initialized_archive):
-    from archivebox.machine.models import Machine
-    from abx_plugins.plugins.opencode import runtime
-
-    os.chdir(initialized_archive)
-    Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
-    _reset_runtime_config()
-    assert get_config().model_dump(mode="json")["OPENCODE_ENABLED"] is False
-
-    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-
-    assert response.status_code == 404
-    assert runtime._PROCESS is None or runtime._PROCESS.poll() is not None
-
-
-def test_opencode_disabled_via_cli_stays_disabled(admin_client, initialized_archive):
-    _set_archivebox_config(initialized_archive, "OPENCODE_ENABLED=False")
-
-    assert get_config().OPENCODE_ENABLED is False
-    assert admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST).status_code == 404
-    for path in ("/add/", "/admin/core/snapshot/"):
-        response = admin_client.get(path, HTTP_HOST=ADMIN_TEST_HOST)
-        assert response.status_code == 200
-        assert b'href="/admin/agent"' not in response.content
-
-
-def test_opencode_agent_requires_superuser_when_enabled(client, db, django_user_model, live_opencode):
-    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    assert response.status_code == 302
-    assert "/admin/login/" in response.headers["Location"]
-
-    next_path = "/admin/agent?x=1&next=https://example.com"
-    response = client.get(next_path, HTTP_HOST=ADMIN_TEST_HOST)
-    assert response.status_code == 302
-    assert parse_qs(urlsplit(response.headers["Location"]).query) == {"next": [next_path]}
-
-    user = django_user_model.objects.create_user(username="regular", password="testpassword")
-    client.force_login(user)
-    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    assert response.status_code == 403
-
-
-def test_opencode_proxy_blocks_cross_origin_mutation(admin_client, db, live_opencode):
-    response = admin_client.post(
-        "/admin/agent/opencode/session",
-        data=b"{}",
-        content_type="application/json",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_ORIGIN="https://evil.example",
-    )
-
-    assert response.status_code == 403
-
-
-def test_opencode_proxy_blocks_cross_site_fetch_metadata(admin_client, db, live_opencode):
-    response = admin_client.post(
-        "/admin/agent/opencode/session",
-        data=b"{}",
-        content_type="application/json",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="cross-site",
-    )
-
-    assert response.status_code == 403
-
-
-def test_opencode_cold_agent_wrapper_returns_before_server_starts(admin_client, installed_opencode):
-    import time
-
-    from abx_plugins.plugins.opencode import runtime
-
-    assert not runtime._owned_process_running()
-    started = time.monotonic()
-    response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-
-    assert response.status_code == 200
-    assert time.monotonic() - started < 3
-    assert not runtime._owned_process_running()
-    assert b'id="opencode-agent-welcome"' in response.content
-    assert f'<iframe src="{runtime._project_route(installed_opencode.config.data_dir)}"'.encode() in response.content
-
-
-def test_opencode_agent_superuser_gets_admin_wrapper(admin_client, live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    frame_path = runtime._project_route(live_opencode.config.data_dir)
-
-    assert response.status_code == 200
-    assert f'<iframe src="{frame_path}"'.encode() in response.content
-    assert b'id="opencode-agent-welcome"' in response.content
-    assert b'id="header"' in response.content
-    assert b'id="progress-monitor"' in response.content
-    assert b'<a href="/admin/agent" class="navbar-item navbar-ai">' in response.content
-    add_page = admin_client.get("/add/", HTTP_HOST=ADMIN_TEST_HOST)
-    assert add_page.status_code == 200
-    assert '<a href="/admin/agent">💬 Crawl with AI</a>'.encode() in add_page.content
-    assert response.context["proxy_prefix"] == runtime._PROXY_PREFIX
-    assert b"/_archivebox/health" not in response.content
-    assert b"window.setInterval(check, 3000)" not in response.content
-    assert response.headers["X-Frame-Options"] == "DENY"
-    assert response.headers["Content-Security-Policy"] == "frame-ancestors 'none'"
-
-    frame = admin_client.get(
-        frame_path,
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert frame.status_code == 302
-    assert frame.headers["Location"].startswith(frame_path + "/")
-    session = admin_client.get(
-        frame.headers["Location"],
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert session.status_code == 200
-    assert session.headers["X-Frame-Options"] == "SAMEORIGIN"
-    assert session.headers["Content-Security-Policy"] == "frame-ancestors 'self'"
-
-
-def test_opencode_proxy_serves_real_project_and_session(admin_client, live_opencode):
-    workdir = str(live_opencode.config.data_dir.resolve())
-    encoded_workdir = quote(workdir)
-
-    agent = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    assert agent.status_code == 200
-    from abx_plugins.plugins.opencode import runtime
-
-    frame_path = runtime._project_route(live_opencode.config.data_dir)
-    frame = admin_client.get(frame_path, HTTP_HOST=ADMIN_TEST_HOST)
-    assert frame.status_code == 302
-    session_id = frame.headers["Location"].rsplit("/", 1)[-1]
-
-    project = admin_client.get(
-        f"/admin/agent/opencode/project/current?directory={encoded_workdir}",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert project.status_code == 200
-    assert project.json()["id"] == "global"
-    assert not project.json().get("vcs")
-
-    path = admin_client.get(
-        f"/admin/agent/opencode/path?directory={encoded_workdir}",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert path.status_code == 200
-    assert path.json()["directory"] == workdir
-
-    sessions = admin_client.get(
-        f"/admin/agent/opencode/session?directory={encoded_workdir}&roots=true&limit=55",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert sessions.status_code == 200
-    assert any(session["id"] == session_id and session["directory"] == workdir for session in sessions.json())
-    assert not (Path(workdir) / ".git").exists()
-
-
-def test_opencode_proxy_restarts_server_for_an_existing_agent_page(admin_client, live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    old_process = runtime._PROCESS
-    runtime._stop_owned_process()
-
-    response = admin_client.get(
-        "/admin/agent/opencode/global/health",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-
-    assert response.status_code == 200
-    assert runtime._PROCESS is not None
-    assert runtime._PROCESS is not old_process
-    assert runtime._PROCESS.poll() is None
-
-
-def test_concurrent_opencode_startup_waits_until_server_is_ready(live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    runtime._stop_owned_process()
-
-    with ThreadPoolExecutor(max_workers=2) as executor:
-        results = list(executor.map(runtime._ensure_opencode, [live_opencode.settings] * 2))
-
-    assert results == [(True, ""), (True, "")]
-    assert runtime._health(live_opencode.settings)
-
-
-def test_opencode_does_not_probe_or_replace_a_ready_owned_process(live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    process = runtime._PROCESS
-    settings = {**live_opencode.settings, "port": _free_port()}
-    settings["origin"] = f"http://{settings['host']}:{settings['port']}"
-
-    ok, error = runtime._ensure_opencode(settings)
-
-    assert ok, error
-    assert process is not None
-    assert runtime._PROCESS is process
-    assert process.poll() is None
-
-
-def test_opencode_proxy_does_not_wait_for_recovery_lock(admin_client, live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    workdir = quote(str(live_opencode.config.data_dir.resolve()))
-    assert runtime._owned_process_ready()
-    executor = ThreadPoolExecutor(max_workers=1)
-    runtime._PROCESS_LOCK.acquire()
-    try:
-        request = executor.submit(
-            admin_client.get,
-            f"/admin/agent/opencode/path?directory={workdir}",
-            HTTP_HOST=ADMIN_TEST_HOST,
-            HTTP_SEC_FETCH_SITE="same-origin",
-        )
-        response = request.result(timeout=5)
-    finally:
-        runtime._PROCESS_LOCK.release()
-        executor.shutdown(wait=True)
-
-    assert response.status_code == 200
-    assert str(live_opencode.config.data_dir.resolve()).encode() in response.content
-
-
-def test_opencode_proxy_waits_for_owned_process_readiness(admin_client, live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    process = runtime._PROCESS
-    assert process is not None
-    runtime._PROCESS_READY = None
-    workdir = quote(str(live_opencode.config.data_dir.resolve()))
-
-    response = admin_client.get(
-        f"/admin/agent/opencode/path?directory={workdir}",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-
-    assert response.status_code == 200
-    assert runtime._PROCESS is process
-    assert runtime._PROCESS_READY is process
-
-
-def test_opencode_proxy_preserves_protocol_headers(admin_client, live_opencode):
-    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
-    created = admin_client.post(
-        "/admin/agent/opencode/pty",
-        data={"command": "/bin/sh", "args": []},
-        content_type="application/json",
-        **headers,
-    )
-    assert created.status_code == 200
-    path = f"/admin/agent/opencode/pty/{created.json()['id']}"
-    try:
-        response = admin_client.post(path + "/connect-token", HTTP_X_OPENCODE_TICKET="1", **headers)
-        assert response.status_code == 200, response.content
-        assert response.json()["ticket"]
-    finally:
-        deleted = admin_client.delete(path, **headers)
-    assert deleted.status_code == 200, deleted.content
-
-
-def test_opencode_oauth_callback_waits_for_user_and_preserves_cancellation(live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    settings = {**live_opencode.settings, "timeout": 1}
-    headers = {"Content-Type": "application/json"}
-    status, _, body = runtime.proxy(settings, "POST", "provider/openai/oauth/authorize", (), headers, b'{"method":0}')
-    assert status == 200
-    authorization = json.loads(body)
-    assert urlsplit(authorization["url"]).hostname == "auth.openai.com"
-    redirect = parse_qs(urlsplit(authorization["url"]).query)["redirect_uri"][0]
-    callback_origin = urlsplit(redirect)
-    assert callback_origin.hostname == "localhost"
-
-    with ThreadPoolExecutor(max_workers=1) as executor:
-        callback = executor.submit(runtime.proxy, settings, "POST", "provider/openai/oauth/callback", (), headers, b'{"method":0}')
-        try:
-            # A real pending authorization must outlive the ordinary API read
-            # timeout. No tokens or substituted provider responses are used.
-            with pytest.raises(FutureTimeoutError):
-                callback.result(timeout=2)
-        finally:
-            cancelled = requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=5)
-            assert cancelled.status_code == 200
-            assert cancelled.text == "Login cancelled"
-        status, response_headers, body = callback.result(timeout=5)
-    assert status == 500
-    assert response_headers["Content-Type"].startswith("application/json")
-    error = json.loads(body)
-    assert error["name"] == "UnknownError"
-    assert error["data"]["ref"].startswith("err_")
-    runtime._stop_owned_process()
-    # The pnpm launcher can exit before its server child. Stopping the owned
-    # process must release both listeners so another collection can authorize.
-    with pytest.raises(requests.ConnectionError):
-        requests.get(settings["origin"] + "/global/health", timeout=2)
-    with pytest.raises(requests.ConnectionError):
-        requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=2)
-
-
-def test_opencode_static_assets_cache_privately_and_revalidate(admin_client, live_opencode):
-    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
-    page = admin_client.get("/admin/agent/opencode/", **headers)
-    assert page.status_code == 200
-    asset = re.search(rb'src="(/admin/agent/opencode/assets/[^\"]+\.js)"', page.content)
-    assert asset is not None
-    path = asset[1].decode()
-    response = admin_client.get(path, **headers)
-    assert response.status_code == 200
-    assert "private" in response.headers["Cache-Control"]
-    assert "max-age=" in response.headers["Cache-Control"]
-    assert b"/admin/agent/opencode" in response.content
-    repeated = admin_client.get(path, HTTP_IF_NONE_MATCH=response.headers["ETag"], **headers)
-    assert repeated.status_code == 304
-    assert repeated.content == b""
-    assert repeated.headers["ETag"] == response.headers["ETag"]
-    assert admin_client.get("/admin/agent/opencode/path", **headers).headers["Cache-Control"] == "no-store"
-
-
-def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
-    response = admin_client.get(
-        "/admin/agent/opencode/global/event",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-
-    assert response.status_code == 200
-    assert response.streaming
-    assert response.is_async
-    assert response.headers["X-Accel-Buffering"] == "no"
-    assert response.headers["Cache-Control"] == "no-store"
-
-
-@pytest.mark.parametrize("path", ["event", "global/event"])
-def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode, path):
-    from abx_plugins.plugins.opencode import runtime
-
-    # The real /event endpoint emits heartbeats indefinitely. A bounded
-    # upstream read exposes accidental buffering without hanging the test.
-    status, headers, body = runtime.proxy(
-        {**live_opencode.settings, "timeout": 1},
-        "GET",
-        path,
-        (),
-        {},
-        b"",
-    )
-    assert status == 200
-    assert not isinstance(body, bytes)
-    assert headers["Content-Type"] == "text/event-stream"
-    asyncio.run(body.aclose())
-    response = admin_client.get(
-        f"/admin/agent/opencode/{path}",
-        HTTP_HOST=ADMIN_TEST_HOST,
-        HTTP_SEC_FETCH_SITE="same-origin",
-    )
-    assert response.status_code == 200
-
-    async def first_event():
-        stream = response.streaming_content
-        try:
-            async with asyncio.timeout(2):
-                return await anext(stream)
-        finally:
-            await stream.aclose()
-
-    assert b"server.connected" in asyncio.run(first_event())
-
-
-def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client, live_opencode):
-    from archivebox.core.asgi import application
-    from abx_plugins.plugins.opencode import runtime
-    from django.conf import settings as django_settings
-
-    owned_process = runtime._PROCESS
-    assert owned_process is not None
-    session_cookie_name = django_settings.SESSION_COOKIE_NAME
-    session_cookie = admin_client.cookies[session_cookie_name].value
-    path = "/admin/agent/opencode/global/event"
-
-    async def request_event_stream():
-        communicator = ApplicationCommunicator(
-            application,
-            {
-                "type": "http",
-                "asgi": {"version": "3.0"},
-                "http_version": "1.1",
-                "method": "GET",
-                "scheme": "http",
-                "path": path,
-                "raw_path": path.encode(),
-                "query_string": b"",
-                "headers": [
-                    (b"host", ADMIN_TEST_HOST.encode()),
-                    (b"cookie", f"{session_cookie_name}={session_cookie}".encode()),
-                    (b"sec-fetch-site", b"same-origin"),
-                ],
-                "client": ("127.0.0.1", 12345),
-                "server": ("127.0.0.1", 5797),
-            },
-        )
-        runtime._PROCESS_LOCK.acquire()
-        runtime._PROCESS = None
-        try:
-            await communicator.send_input({"type": "http.request", "body": b"", "more_body": False})
-            response_start = await communicator.receive_output(timeout=2)
-            assert response_start["type"] == "http.response.start"
-            assert response_start["status"] == 200
-        finally:
-            try:
-                await communicator.send_input({"type": "http.disconnect"})
-                await communicator.wait(timeout=5)
-            finally:
-                runtime._PROCESS = owned_process
-                runtime._PROCESS_LOCK.release()
-                await asyncio.get_running_loop().shutdown_default_executor()
-
-    asyncio.run(request_event_stream())
-    assert runtime._PROCESS is owned_process
-    assert owned_process.poll() is None
-
-
-def test_opencode_starts_with_isolated_state(admin_client, live_opencode):
-    workdir = str(live_opencode.config.data_dir.resolve())
-    state_dir = live_opencode.config.state_dir
-
-    assert not (Path(workdir) / ".git").exists()
-    agent = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    assert agent.status_code == 200
-    from abx_plugins.plugins.opencode import runtime
-
-    frame = admin_client.get(
-        runtime._project_route(live_opencode.config.data_dir),
-        HTTP_HOST=ADMIN_TEST_HOST,
-    )
-    assert frame.status_code == 302
-    assert not (Path(workdir) / ".git").exists()
-
-    project = requests.get(
-        f"{live_opencode.settings['origin']}/project/current",
-        params={"directory": workdir},
-        timeout=live_opencode.settings["timeout"],
-    )
-    project.raise_for_status()
-
-    config = requests.get(
-        f"{live_opencode.settings['origin']}/global/config",
-        timeout=live_opencode.settings["timeout"],
-    )
-    config.raise_for_status()
-
-    assert Path(live_opencode.settings["workdir"]).resolve() == Path(workdir)
-    assert project.json()["id"] == "global"
-    assert not project.json().get("vcs")
-    assert not config.json().get("model")
-    assert config.json()["snapshot"] is False
-    assert live_opencode.process.poll() is None
-    path = requests.get(
-        f"{live_opencode.settings['origin']}/path",
-        params={"directory": workdir},
-        timeout=live_opencode.settings["timeout"],
-    )
-    path.raise_for_status()
-    assert Path(path.json()["directory"]).resolve() == Path(workdir)
-
-    diff = requests.get(
-        f"{live_opencode.settings['origin']}/vcs/diff",
-        params={"directory": workdir, "mode": "git"},
-        timeout=5,
-    )
-    diff.raise_for_status()
-    assert diff.json() == []
-    assert (state_dir / "data" / "opencode" / "opencode.db").is_file()
-    assert (state_dir / "SKILL.md").is_file()
-    assert (state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md").resolve() == state_dir / "SKILL.md"
-
-
-def test_opencode_invalid_state_does_not_break_archivebox(admin_client, live_opencode):
-    from abx_plugins.plugins.opencode import runtime
-
-    runtime._stop_owned_process()
-    invalid_state = live_opencode.config.state_dir / "config"
-    invalid_state.rename(live_opencode.config.state_dir / "saved-config")
-    invalid_state.write_text("Preserve this file.")
-
-    wrapper = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
-    assert wrapper.status_code == 200
-    for url in (
-        runtime._project_route(live_opencode.config.data_dir),
-        "/admin/agent/opencode/global/health",
-    ):
-        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
-        assert response.status_code == 503
-        assert str(invalid_state).encode() not in response.content
-
-    stream = admin_client.get("/admin/agent/opencode/global/event", HTTP_HOST=ADMIN_TEST_HOST)
-    assert stream.status_code == 200
-
-    async def read_failure():
-        return b"".join([chunk async for chunk in stream.streaming_content])
-
-    assert asyncio.run(read_failure()) == b'event: error\ndata: {"error":"OpenCode unavailable"}\n\n'
-
-    for url in ("/health/", "/add/", "/admin/core/snapshot/"):
-        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
-        assert response.status_code == 200
-    assert invalid_state.read_text() == "Preserve this file."
-
-
-@pytest.mark.parametrize(
-    ("damaged_file", "missing"),
-    [
-        ("runtime.py", True),
-        ("templates/agent.html", True),
-        ("templates/agent.html", False),
-        ("templates/navigation.html", False),
-        ("templates/add.html", False),
-    ],
-)
-def test_opencode_incomplete_install_does_not_break_archivebox(installed_opencode, tmp_path, damaged_file, missing):
-    import abx_plugins
-
-    # Exercise a genuinely incomplete installation in a separate process;
-    # never alter the shared package or intercept Python imports.
-    site = tmp_path / "site"
-    installed = site / "abx_plugins"
-    shutil.copytree(Path(abx_plugins.__file__).parent, installed, ignore=shutil.ignore_patterns("__pycache__", "tests"))
-    damaged_path = installed / "plugins" / "opencode" / damaged_file
-    if missing:
-        damaged_path.unlink()
-    else:
-        damaged_path.write_text("{% invalid_template_tag %}")
-    expected_status = 200 if damaged_file in {"templates/navigation.html", "templates/add.html"} else 503
-    script = f"""
-import sys
-from pathlib import Path
-import abx_plugins
-from django.test import Client
-from django.contrib.auth import get_user_model
-assert Path(abx_plugins.__file__).is_relative_to({str(site)!r})
-user = get_user_model().objects.create_superuser(username='optional-service-test')
-client = Client(HTTP_HOST={ADMIN_TEST_HOST!r})
-client.force_login(user)
-for path in ('/health/', '/add/', '/admin/core/snapshot/'):
-    assert client.get(path).status_code == 200, path
-assert 'abx_plugins.plugins.opencode.runtime' not in sys.modules
-response = client.get('/admin/agent')
-assert response.status_code == {expected_status}, response.status_code
-if response.status_code == 503:
-    assert response.content == b'AI service unavailable. See server logs.'
-if {damaged_file != "runtime.py"!r}:
-    from abx_plugins.plugins.opencode import runtime
-    assert runtime._PROCESS is None
-for path in ('/health/', '/add/', '/admin/core/snapshot/'):
-    assert client.get(path).status_code == 200, path
-print('OPTIONAL_SERVICE_FAILURE_ISOLATED')
-"""
-    result = run_archivebox_cmd(
-        ["shell", "-c", script],
-        cwd=installed_opencode.config.data_dir,
-        env={**installed_opencode.config.env, "PYTHONPATH": str(site)},
-        timeout=90,
-    )
-    assert result.returncode == 0, result.stderr or result.stdout
-    assert "OPTIONAL_SERVICE_FAILURE_ISOLATED" in result.stdout
@@ -0,0 +1,291 @@
+import os
+import shutil
+import socket
+from pathlib import Path
+from types import SimpleNamespace
+from urllib.parse import quote
+
+import pytest
+import requests
+
+from archivebox.tests.conftest import ADMIN_TEST_HOST, run_archivebox_cmd
+from archivebox.config.common import get_config
+
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+def _free_port() -> int:
+    with socket.socket() as sock:
+        sock.bind(("127.0.0.1", 0))
+        return int(sock.getsockname()[1])
+
+
+def _reset_runtime_config() -> None:
+    from archivebox.config import common
+    from archivebox.machine.models import Machine
+
+    for value in vars(common).values():
+        cache_clear = getattr(value, "cache_clear", None)
+        if cache_clear is not None:
+            cache_clear()
+    Machine.current()
+
+
+def _set_archivebox_config(data_dir: Path, *values: str, env: dict[str, str] | None = None) -> None:
+    os.chdir(data_dir)
+    result = run_archivebox_cmd(
+        ["config", "--set", *values],
+        cwd=data_dir,
+        env=env,
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    _reset_runtime_config()
+
+
+@pytest.fixture
+def opencode_archive_config(initialized_archive):
+    port = _free_port()
+    state_dir = initialized_archive / "opencode"
+    env = os.environ.copy()
+    env.update(
+        {
+            "ARCHIVEBOX_ALLOW_NO_UNIX_SOCKETS": "true",
+            "OPENCODE_ENABLED": "True",
+            "OPENCODE_HOST": "127.0.0.1",
+            "OPENCODE_PORT": str(port),
+            "OPENCODE_WORKDIR": str(initialized_archive),
+            "OPENCODE_STATE_DIR": str(state_dir),
+            "OPENCODE_TIMEOUT": "60",
+        },
+    )
+    _set_archivebox_config(
+        initialized_archive,
+        "OPENCODE_ENABLED=True",
+        "OPENCODE_HOST=127.0.0.1",
+        f"OPENCODE_PORT={port}",
+        f"OPENCODE_WORKDIR={initialized_archive}",
+        f"OPENCODE_STATE_DIR={state_dir}",
+        "OPENCODE_TIMEOUT=60",
+        env=env,
+    )
+    return SimpleNamespace(data_dir=initialized_archive, port=port, state_dir=state_dir, env=env)
+
+
+@pytest.fixture
+def installed_opencode(opencode_archive_config):
+    from abx_plugins.plugins.opencode import runtime
+
+    install = run_archivebox_cmd(
+        ["install", "opencode", "--binproviders=env,pnpm"],
+        cwd=opencode_archive_config.data_dir,
+        env=opencode_archive_config.env,
+        timeout=1200,
+    )
+    assert install.returncode == 0, install.stderr or install.stdout
+    _reset_runtime_config()
+
+    config = get_config().model_dump(mode="json")
+    settings = runtime._settings(config, opencode_archive_config.data_dir)
+    settings["archivebox_base_url"] = "http://admin.archivebox.localhost:5797"
+    settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:5797/admin"
+    settings["archivebox_api_url"] = "http://admin.archivebox.localhost:5797/api/"
+    binary, binary_env = runtime._resolve_binary(settings["binary"], settings["config"])
+    version = binary.exec(
+        cmd=("--version",),
+        env={**os.environ, **binary_env},
+        timeout=120,
+    )
+    assert version.returncode == 0, version.stderr or version.stdout
+    return SimpleNamespace(config=opencode_archive_config, settings=settings)
+
+
+@pytest.fixture
+def live_opencode(installed_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    settings = installed_opencode.settings
+    ok, error = runtime._ensure_opencode(settings)
+    assert ok, error
+
+    process = runtime._PROCESS
+    assert process is not None
+    try:
+        yield SimpleNamespace(config=installed_opencode.config, settings=settings, process=process)
+    finally:
+        runtime._stop_owned_process()
+
+
+def test_opencode_disabled_via_cli_stays_disabled(admin_client, initialized_archive):
+    _set_archivebox_config(initialized_archive, "OPENCODE_ENABLED=False")
+
+    assert get_config().OPENCODE_ENABLED is False
+    assert admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST).status_code == 404
+    for path in ("/add/", "/admin/core/snapshot/"):
+        response = admin_client.get(path, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 200
+        assert b'href="/admin/agent"' not in response.content
+
+
+def test_opencode_proxy_blocks_cross_site_fetch_metadata(admin_client, db, live_opencode):
+    response = admin_client.post(
+        "/admin/agent/opencode/session",
+        data=b"{}",
+        content_type="application/json",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="cross-site",
+    )
+
+    assert response.status_code == 403
+
+
+def test_opencode_proxy_waits_for_owned_process_readiness(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    process = runtime._PROCESS
+    assert process is not None
+    runtime._PROCESS_READY = None
+    workdir = quote(str(live_opencode.config.data_dir.resolve()))
+
+    response = admin_client.get(
+        f"/admin/agent/opencode/path?directory={workdir}",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+
+    assert response.status_code == 200
+    assert runtime._PROCESS is process
+    assert runtime._PROCESS_READY is process
+
+
+def test_opencode_proxy_preserves_protocol_headers(admin_client, live_opencode):
+    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
+    created = admin_client.post(
+        "/admin/agent/opencode/pty",
+        data={"command": "/bin/sh", "args": []},
+        content_type="application/json",
+        **headers,
+    )
+    assert created.status_code == 200
+    path = f"/admin/agent/opencode/pty/{created.json()['id']}"
+    try:
+        response = admin_client.post(path + "/connect-token", HTTP_X_OPENCODE_TICKET="1", **headers)
+        assert response.status_code == 200, response.content
+        assert response.json()["ticket"]
+    finally:
+        deleted = admin_client.delete(path, **headers)
+    assert deleted.status_code == 200, deleted.content
+
+
+def test_opencode_starts_with_isolated_state(admin_client, live_opencode):
+    workdir = str(live_opencode.config.data_dir.resolve())
+    state_dir = live_opencode.config.state_dir
+
+    assert not (Path(workdir) / ".git").exists()
+    agent = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert agent.status_code == 200
+    from abx_plugins.plugins.opencode import runtime
+
+    frame = admin_client.get(
+        runtime._project_route(live_opencode.config.data_dir),
+        HTTP_HOST=ADMIN_TEST_HOST,
+    )
+    assert frame.status_code == 302
+    assert not (Path(workdir) / ".git").exists()
+
+    project = requests.get(
+        f"{live_opencode.settings['origin']}/project/current",
+        params={"directory": workdir},
+        timeout=live_opencode.settings["timeout"],
+    )
+    project.raise_for_status()
+
+    config = requests.get(
+        f"{live_opencode.settings['origin']}/global/config",
+        timeout=live_opencode.settings["timeout"],
+    )
+    config.raise_for_status()
+
+    assert Path(live_opencode.settings["workdir"]).resolve() == Path(workdir)
+    assert project.json()["id"] == "global"
+    assert not project.json().get("vcs")
+    assert not config.json().get("model")
+    assert config.json()["snapshot"] is False
+    assert live_opencode.process.poll() is None
+    path = requests.get(
+        f"{live_opencode.settings['origin']}/path",
+        params={"directory": workdir},
+        timeout=live_opencode.settings["timeout"],
+    )
+    path.raise_for_status()
+    assert Path(path.json()["directory"]).resolve() == Path(workdir)
+
+    diff = requests.get(
+        f"{live_opencode.settings['origin']}/vcs/diff",
+        params={"directory": workdir, "mode": "git"},
+        timeout=5,
+    )
+    diff.raise_for_status()
+    assert diff.json() == []
+    assert (state_dir / "data" / "opencode" / "opencode.db").is_file()
+    assert (state_dir / "SKILL.md").is_file()
+    assert (state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md").resolve() == state_dir / "SKILL.md"
+
+
+@pytest.mark.parametrize(
+    ("damaged_file", "missing"),
+    [
+        ("runtime.py", True),
+        ("templates/agent.html", True),
+        ("templates/agent.html", False),
+        ("templates/navigation.html", False),
+        ("templates/add.html", False),
+    ],
+)
+def test_opencode_incomplete_install_does_not_break_archivebox(installed_opencode, tmp_path, damaged_file, missing):
+    import abx_plugins
+
+    # Exercise a genuinely incomplete installation in a separate process;
+    # never alter the shared package or intercept Python imports.
+    site = tmp_path / "site"
+    installed = site / "abx_plugins"
+    shutil.copytree(Path(abx_plugins.__file__).parent, installed, ignore=shutil.ignore_patterns("__pycache__", "tests"))
+    damaged_path = installed / "plugins" / "opencode" / damaged_file
+    if missing:
+        damaged_path.unlink()
+    else:
+        damaged_path.write_text("{% invalid_template_tag %}")
+    expected_status = 200 if damaged_file in {"templates/navigation.html", "templates/add.html"} else 503
+    script = f"""
+import sys
+from pathlib import Path
+import abx_plugins
+from django.test import Client
+from django.contrib.auth import get_user_model
+assert Path(abx_plugins.__file__).is_relative_to({str(site)!r})
+user = get_user_model().objects.create_superuser(username='optional-service-test')
+client = Client(HTTP_HOST={ADMIN_TEST_HOST!r})
+client.force_login(user)
+for path in ('/health/', '/add/', '/admin/core/snapshot/'):
+    assert client.get(path).status_code == 200, path
+assert 'abx_plugins.plugins.opencode.runtime' not in sys.modules
+response = client.get('/admin/agent')
+assert response.status_code == {expected_status}, response.status_code
+if response.status_code == 503:
+    assert response.content == b'AI service unavailable. See server logs.'
+if {damaged_file != "runtime.py"!r}:
+    from abx_plugins.plugins.opencode import runtime
+    assert runtime._PROCESS is None
+for path in ('/health/', '/add/', '/admin/core/snapshot/'):
+    assert client.get(path).status_code == 200, path
+print('OPTIONAL_SERVICE_FAILURE_ISOLATED')
+"""
+    result = run_archivebox_cmd(
+        ["shell", "-c", script],
+        cwd=installed_opencode.config.data_dir,
+        env={**installed_opencode.config.env, "PYTHONPATH": str(site)},
+        timeout=90,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "OPTIONAL_SERVICE_FAILURE_ISOLATED" in result.stdout
@@ -0,0 +1,251 @@
+import asyncio
+import json
+import os
+import re
+from concurrent.futures import ThreadPoolExecutor
+from concurrent.futures import TimeoutError as FutureTimeoutError
+from urllib.parse import parse_qs, urlsplit
+
+import pytest
+import requests
+from asgiref.testing import ApplicationCommunicator
+
+from archivebox.tests.conftest import ADMIN_TEST_HOST
+from archivebox.config.common import get_config
+
+
+from .test_opencode_agent_1 import (
+    pytestmark as pytestmark,
+    _free_port as _free_port,
+    _reset_runtime_config as _reset_runtime_config,
+    _set_archivebox_config as _set_archivebox_config,
+    opencode_archive_config as opencode_archive_config,
+    installed_opencode as installed_opencode,
+    live_opencode as live_opencode,
+)
+
+
+def test_opencode_disabled_route_does_not_start_server(client, initialized_archive):
+    from archivebox.machine.models import Machine
+    from abx_plugins.plugins.opencode import runtime
+
+    os.chdir(initialized_archive)
+    Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
+    _reset_runtime_config()
+    assert get_config().model_dump(mode="json")["OPENCODE_ENABLED"] is False
+
+    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+
+    assert response.status_code == 404
+    assert runtime._PROCESS is None or runtime._PROCESS.poll() is not None
+
+
+def test_opencode_agent_requires_superuser_when_enabled(client, db, django_user_model, live_opencode):
+    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert response.status_code == 302
+    assert "/admin/login/" in response.headers["Location"]
+
+    next_path = "/admin/agent?x=1&next=https://example.com"
+    response = client.get(next_path, HTTP_HOST=ADMIN_TEST_HOST)
+    assert response.status_code == 302
+    assert parse_qs(urlsplit(response.headers["Location"]).query) == {"next": [next_path]}
+
+    user = django_user_model.objects.create_user(username="regular", password="testpassword")
+    client.force_login(user)
+    response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert response.status_code == 403
+
+
+def test_opencode_agent_superuser_gets_admin_wrapper(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    frame_path = runtime._project_route(live_opencode.config.data_dir)
+
+    assert response.status_code == 200
+    assert f'<iframe src="{frame_path}"'.encode() in response.content
+    assert b'id="opencode-agent-welcome"' in response.content
+    assert b'id="header"' in response.content
+    assert b'id="progress-monitor"' in response.content
+    assert b'<a href="/admin/agent" class="navbar-item navbar-ai">' in response.content
+    add_page = admin_client.get("/add/", HTTP_HOST=ADMIN_TEST_HOST)
+    assert add_page.status_code == 200
+    assert '<a href="/admin/agent">💬 Crawl with AI</a>'.encode() in add_page.content
+    assert response.context["proxy_prefix"] == runtime._PROXY_PREFIX
+    assert b"/_archivebox/health" not in response.content
+    assert b"window.setInterval(check, 3000)" not in response.content
+    assert response.headers["X-Frame-Options"] == "DENY"
+    assert response.headers["Content-Security-Policy"] == "frame-ancestors 'none'"
+
+    frame = admin_client.get(
+        frame_path,
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert frame.status_code == 302
+    assert frame.headers["Location"].startswith(frame_path + "/")
+    session = admin_client.get(
+        frame.headers["Location"],
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert session.status_code == 200
+    assert session.headers["X-Frame-Options"] == "SAMEORIGIN"
+    assert session.headers["Content-Security-Policy"] == "frame-ancestors 'self'"
+
+
+def test_opencode_oauth_callback_waits_for_user_and_preserves_cancellation(live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    settings = {**live_opencode.settings, "timeout": 1}
+    headers = {"Content-Type": "application/json"}
+    status, _, body = runtime.proxy(settings, "POST", "provider/openai/oauth/authorize", (), headers, b'{"method":0}')
+    assert status == 200
+    authorization = json.loads(body)
+    assert urlsplit(authorization["url"]).hostname == "auth.openai.com"
+    redirect = parse_qs(urlsplit(authorization["url"]).query)["redirect_uri"][0]
+    callback_origin = urlsplit(redirect)
+    assert callback_origin.hostname == "localhost"
+
+    with ThreadPoolExecutor(max_workers=1) as executor:
+        callback = executor.submit(runtime.proxy, settings, "POST", "provider/openai/oauth/callback", (), headers, b'{"method":0}')
+        try:
+            # A real pending authorization must outlive the ordinary API read
+            # timeout. No tokens or substituted provider responses are used.
+            with pytest.raises(FutureTimeoutError):
+                callback.result(timeout=2)
+        finally:
+            cancelled = requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=5)
+            assert cancelled.status_code == 200
+            assert cancelled.text == "Login cancelled"
+        status, response_headers, body = callback.result(timeout=5)
+    assert status == 500
+    assert response_headers["Content-Type"].startswith("application/json")
+    error = json.loads(body)
+    assert error["name"] == "UnknownError"
+    assert error["data"]["ref"].startswith("err_")
+    runtime._stop_owned_process()
+    # The pnpm launcher can exit before its server child. Stopping the owned
+    # process must release both listeners so another collection can authorize.
+    with pytest.raises(requests.ConnectionError):
+        requests.get(settings["origin"] + "/global/health", timeout=2)
+    with pytest.raises(requests.ConnectionError):
+        requests.get(f"{callback_origin.scheme}://{callback_origin.netloc}/cancel", timeout=2)
+
+
+def test_opencode_static_assets_cache_privately_and_revalidate(admin_client, live_opencode):
+    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
+    page = admin_client.get("/admin/agent/opencode/", **headers)
+    assert page.status_code == 200
+    asset = re.search(rb'src="(/admin/agent/opencode/assets/[^\"]+\.js)"', page.content)
+    assert asset is not None
+    path = asset[1].decode()
+    response = admin_client.get(path, **headers)
+    assert response.status_code == 200
+    assert "private" in response.headers["Cache-Control"]
+    assert "max-age=" in response.headers["Cache-Control"]
+    assert b"/admin/agent/opencode" in response.content
+    repeated = admin_client.get(path, HTTP_IF_NONE_MATCH=response.headers["ETag"], **headers)
+    assert repeated.status_code == 304
+    assert repeated.content == b""
+    assert repeated.headers["ETag"] == response.headers["ETag"]
+    assert admin_client.get("/admin/agent/opencode/path", **headers).headers["Cache-Control"] == "no-store"
+
+
+def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
+    response = admin_client.get(
+        "/admin/agent/opencode/global/event",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+
+    assert response.status_code == 200
+    assert response.streaming
+    assert response.is_async
+    assert response.headers["X-Accel-Buffering"] == "no"
+    assert response.headers["Cache-Control"] == "no-store"
+
+
+def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client, live_opencode):
+    from archivebox.core.asgi import application
+    from abx_plugins.plugins.opencode import runtime
+    from django.conf import settings as django_settings
+
+    owned_process = runtime._PROCESS
+    assert owned_process is not None
+    session_cookie_name = django_settings.SESSION_COOKIE_NAME
+    session_cookie = admin_client.cookies[session_cookie_name].value
+    path = "/admin/agent/opencode/global/event"
+
+    async def request_event_stream():
+        communicator = ApplicationCommunicator(
+            application,
+            {
+                "type": "http",
+                "asgi": {"version": "3.0"},
+                "http_version": "1.1",
+                "method": "GET",
+                "scheme": "http",
+                "path": path,
+                "raw_path": path.encode(),
+                "query_string": b"",
+                "headers": [
+                    (b"host", ADMIN_TEST_HOST.encode()),
+                    (b"cookie", f"{session_cookie_name}={session_cookie}".encode()),
+                    (b"sec-fetch-site", b"same-origin"),
+                ],
+                "client": ("127.0.0.1", 12345),
+                "server": ("127.0.0.1", 5797),
+            },
+        )
+        runtime._PROCESS_LOCK.acquire()
+        runtime._PROCESS = None
+        try:
+            await communicator.send_input({"type": "http.request", "body": b"", "more_body": False})
+            response_start = await communicator.receive_output(timeout=2)
+            assert response_start["type"] == "http.response.start"
+            assert response_start["status"] == 200
+        finally:
+            try:
+                await communicator.send_input({"type": "http.disconnect"})
+                await communicator.wait(timeout=5)
+            finally:
+                runtime._PROCESS = owned_process
+                runtime._PROCESS_LOCK.release()
+                await asyncio.get_running_loop().shutdown_default_executor()
+
+    asyncio.run(request_event_stream())
+    assert runtime._PROCESS is owned_process
+    assert owned_process.poll() is None
+
+
+def test_opencode_invalid_state_does_not_break_archivebox(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    runtime._stop_owned_process()
+    invalid_state = live_opencode.config.state_dir / "config"
+    invalid_state.rename(live_opencode.config.state_dir / "saved-config")
+    invalid_state.write_text("Preserve this file.")
+
+    wrapper = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert wrapper.status_code == 200
+    for url in (
+        runtime._project_route(live_opencode.config.data_dir),
+        "/admin/agent/opencode/global/health",
+    ):
+        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 503
+        assert str(invalid_state).encode() not in response.content
+
+    stream = admin_client.get("/admin/agent/opencode/global/event", HTTP_HOST=ADMIN_TEST_HOST)
+    assert stream.status_code == 200
+
+    async def read_failure():
+        return b"".join([chunk async for chunk in stream.streaming_content])
+
+    assert asyncio.run(read_failure()) == b'event: error\ndata: {"error":"OpenCode unavailable"}\n\n'
+
+    for url in ("/health/", "/add/", "/admin/core/snapshot/"):
+        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 200
+    assert invalid_state.read_text() == "Preserve this file."
@@ -0,0 +1,191 @@
+import asyncio
+from concurrent.futures import ThreadPoolExecutor
+from pathlib import Path
+from urllib.parse import quote
+
+import pytest
+
+from archivebox.tests.conftest import ADMIN_TEST_HOST
+
+
+from .test_opencode_agent_1 import (
+    pytestmark as pytestmark,
+    _free_port as _free_port,
+    _reset_runtime_config as _reset_runtime_config,
+    _set_archivebox_config as _set_archivebox_config,
+    opencode_archive_config as opencode_archive_config,
+    installed_opencode as installed_opencode,
+    live_opencode as live_opencode,
+)
+
+
+def test_opencode_proxy_blocks_cross_origin_mutation(admin_client, db, live_opencode):
+    response = admin_client.post(
+        "/admin/agent/opencode/session",
+        data=b"{}",
+        content_type="application/json",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_ORIGIN="https://evil.example",
+    )
+
+    assert response.status_code == 403
+
+
+def test_opencode_cold_agent_wrapper_returns_before_server_starts(admin_client, installed_opencode):
+    import time
+
+    from abx_plugins.plugins.opencode import runtime
+
+    assert not runtime._owned_process_running()
+    started = time.monotonic()
+    response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+
+    assert response.status_code == 200
+    assert time.monotonic() - started < 3
+    assert not runtime._owned_process_running()
+    assert b'id="opencode-agent-welcome"' in response.content
+    assert f'<iframe src="{runtime._project_route(installed_opencode.config.data_dir)}"'.encode() in response.content
+
+
+def test_opencode_proxy_serves_real_project_and_session(admin_client, live_opencode):
+    workdir = str(live_opencode.config.data_dir.resolve())
+    encoded_workdir = quote(workdir)
+
+    agent = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert agent.status_code == 200
+    from abx_plugins.plugins.opencode import runtime
+
+    frame_path = runtime._project_route(live_opencode.config.data_dir)
+    frame = admin_client.get(frame_path, HTTP_HOST=ADMIN_TEST_HOST)
+    assert frame.status_code == 302
+    session_id = frame.headers["Location"].rsplit("/", 1)[-1]
+
+    project = admin_client.get(
+        f"/admin/agent/opencode/project/current?directory={encoded_workdir}",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert project.status_code == 200
+    assert project.json()["id"] == "global"
+    assert not project.json().get("vcs")
+
+    path = admin_client.get(
+        f"/admin/agent/opencode/path?directory={encoded_workdir}",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert path.status_code == 200
+    assert path.json()["directory"] == workdir
+
+    sessions = admin_client.get(
+        f"/admin/agent/opencode/session?directory={encoded_workdir}&roots=true&limit=55",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert sessions.status_code == 200
+    assert any(session["id"] == session_id and session["directory"] == workdir for session in sessions.json())
+    assert not (Path(workdir) / ".git").exists()
+
+
+def test_opencode_proxy_restarts_server_for_an_existing_agent_page(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    old_process = runtime._PROCESS
+    runtime._stop_owned_process()
+
+    response = admin_client.get(
+        "/admin/agent/opencode/global/health",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+
+    assert response.status_code == 200
+    assert runtime._PROCESS is not None
+    assert runtime._PROCESS is not old_process
+    assert runtime._PROCESS.poll() is None
+
+
+def test_concurrent_opencode_startup_waits_until_server_is_ready(live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    runtime._stop_owned_process()
+
+    with ThreadPoolExecutor(max_workers=2) as executor:
+        results = list(executor.map(runtime._ensure_opencode, [live_opencode.settings] * 2))
+
+    assert results == [(True, ""), (True, "")]
+    assert runtime._health(live_opencode.settings)
+
+
+def test_opencode_does_not_probe_or_replace_a_ready_owned_process(live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    process = runtime._PROCESS
+    settings = {**live_opencode.settings, "port": _free_port()}
+    settings["origin"] = f"http://{settings['host']}:{settings['port']}"
+
+    ok, error = runtime._ensure_opencode(settings)
+
+    assert ok, error
+    assert process is not None
+    assert runtime._PROCESS is process
+    assert process.poll() is None
+
+
+def test_opencode_proxy_does_not_wait_for_recovery_lock(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    workdir = quote(str(live_opencode.config.data_dir.resolve()))
+    assert runtime._owned_process_ready()
+    executor = ThreadPoolExecutor(max_workers=1)
+    runtime._PROCESS_LOCK.acquire()
+    try:
+        request = executor.submit(
+            admin_client.get,
+            f"/admin/agent/opencode/path?directory={workdir}",
+            HTTP_HOST=ADMIN_TEST_HOST,
+            HTTP_SEC_FETCH_SITE="same-origin",
+        )
+        response = request.result(timeout=5)
+    finally:
+        runtime._PROCESS_LOCK.release()
+        executor.shutdown(wait=True)
+
+    assert response.status_code == 200
+    assert str(live_opencode.config.data_dir.resolve()).encode() in response.content
+
+
+@pytest.mark.parametrize("path", ["event", "global/event"])
+def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode, path):
+    from abx_plugins.plugins.opencode import runtime
+
+    # The real /event endpoint emits heartbeats indefinitely. A bounded
+    # upstream read exposes accidental buffering without hanging the test.
+    status, headers, body = runtime.proxy(
+        {**live_opencode.settings, "timeout": 1},
+        "GET",
+        path,
+        (),
+        {},
+        b"",
+    )
+    assert status == 200
+    assert not isinstance(body, bytes)
+    assert headers["Content-Type"] == "text/event-stream"
+    asyncio.run(body.aclose())
+    response = admin_client.get(
+        f"/admin/agent/opencode/{path}",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert response.status_code == 200
+
+    async def first_event():
+        stream = response.streaming_content
+        try:
+            async with asyncio.timeout(2):
+                return await anext(stream)
+        finally:
+            await stream.aclose()
+
+    assert b"server.connected" in asyncio.run(first_event())
@@ -1,6 +1,5 @@
 """Exercise the real agent wrapper with Chromium's native storage failures."""

-import asyncio
 import json
 import os
 import re
@@ -10,16 +9,64 @@ import pytest
 import requests

 from .conftest import get_free_port, run_archivebox_cmd, start_archivebox_server, stop_archivebox_process
-from .test_opencode_agent import _set_archivebox_config
-from .test_opencode_agent import installed_opencode as installed_opencode
-from .test_opencode_agent import live_opencode as live_opencode
-from .test_opencode_agent import opencode_archive_config as opencode_archive_config
+from .test_opencode_agent_1 import _set_archivebox_config
+from .test_opencode_agent_1 import installed_opencode as installed_opencode
+from .test_opencode_agent_1 import live_opencode as live_opencode
+from .test_opencode_agent_1 import opencode_archive_config as opencode_archive_config
 from .test_server_security_browser import browser_runtime as browser_runtime


 pytestmark = pytest.mark.django_db(transaction=True)


+@pytest.fixture
+def agent_server(installed_opencode, browser_runtime, request):
+    port = get_free_port()
+    mode, subdomains = getattr(request, "param", ("safe-onedomain-nojsreplay", False))
+    base_url = f"http://archivebox.localhost:{port}" if subdomains else f"http://localhost:{port}"
+    url = f"http://admin.archivebox.localhost:{port}" if subdomains else base_url
+    config = installed_opencode.config
+    _set_archivebox_config(
+        config.data_dir,
+        f"BASE_URL={base_url}",
+        f"SERVER_SECURITY_MODE={mode}",
+    )
+    user = run_archivebox_cmd(
+        [
+            "shell",
+            "-c",
+            "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser(username='agent-browser-test', password='test-password'); User.objects.create_user(username='agent-regular-test', password='test-password', is_staff=True)",
+        ],
+        cwd=config.data_dir,
+        env=config.env,
+    )
+    assert user.returncode == 0, user.stderr or user.stdout
+    process = start_archivebox_server(config.data_dir, port=port, env=config.env, log_name="agent-browser-server.log")
+    try:
+        yield url, config.data_dir, process
+    finally:
+        if process.poll() is None:
+            stop_archivebox_process(process)
+
+
+def _login_cookie(server_url, username="agent-browser-test"):
+    session = requests.Session()
+    login_url = server_url + "/admin/login/"
+    page = session.get(login_url, timeout=10)
+    assert page.status_code == 200
+    token = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', page.text)
+    assert token is not None
+    response = session.post(
+        login_url,
+        data={"username": username, "password": "test-password", "csrfmiddlewaretoken": token[1]},
+        headers={"Referer": login_url},
+        allow_redirects=False,
+        timeout=10,
+    )
+    assert response.status_code == 302
+    return "; ".join(f"{key}={value}" for key, value in session.cookies.items())
+
+
@pytest.mark.parametrize(
    "agent_server",
    [
@@ -231,262 +278,3 @@ const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
    )
    assert result.returncode == 0, result.stderr or result.stdout
    assert "AGENT_NAVIGATION_OK" in result.stdout
-
-
-@pytest.fixture
-def agent_server(installed_opencode, browser_runtime, request):
-    port = get_free_port()
-    mode, subdomains = getattr(request, "param", ("safe-onedomain-nojsreplay", False))
-    base_url = f"http://archivebox.localhost:{port}" if subdomains else f"http://localhost:{port}"
-    url = f"http://admin.archivebox.localhost:{port}" if subdomains else base_url
-    config = installed_opencode.config
-    _set_archivebox_config(
-        config.data_dir,
-        f"BASE_URL={base_url}",
-        f"SERVER_SECURITY_MODE={mode}",
-    )
-    user = run_archivebox_cmd(
-        [
-            "shell",
-            "-c",
-            "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser(username='agent-browser-test', password='test-password'); User.objects.create_user(username='agent-regular-test', password='test-password', is_staff=True)",
-        ],
-        cwd=config.data_dir,
-        env=config.env,
-    )
-    assert user.returncode == 0, user.stderr or user.stdout
-    process = start_archivebox_server(config.data_dir, port=port, env=config.env, log_name="agent-browser-server.log")
-    try:
-        yield url, config.data_dir, process
-    finally:
-        if process.poll() is None:
-            stop_archivebox_process(process)
-
-
-def _login_cookie(server_url, username="agent-browser-test"):
-    session = requests.Session()
-    login_url = server_url + "/admin/login/"
-    page = session.get(login_url, timeout=10)
-    assert page.status_code == 200
-    token = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', page.text)
-    assert token is not None
-    response = session.post(
-        login_url,
-        data={"username": username, "password": "test-password", "csrfmiddlewaretoken": token[1]},
-        headers={"Referer": login_url},
-        allow_redirects=False,
-        timeout=10,
-    )
-    assert response.status_code == 302
-    return "; ".join(f"{key}={value}" for key, value in session.cookies.items())
-
-
-def test_agent_websocket_rejects_unauthorized_access(agent_server):
-    from websockets.asyncio.client import connect
-    from websockets.exceptions import InvalidStatus
-
-    server_url, _, _ = agent_server
-    admin_cookie = _login_cookie(server_url)
-    regular_cookie = _login_cookie(server_url, "agent-regular-test")
-
-    async def check_denials():
-        for path, cookie, origin in (
-            ("/admin/agent/opencode/pty/unknown/connect", "", server_url),
-            ("/admin/agent/opencode/pty/unknown/connect", regular_cookie, server_url),
-            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, "https://untrusted.example"),
-            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, None),
-            ("/health/", admin_cookie, server_url),
-        ):
-            with pytest.raises(InvalidStatus) as error:
-                async with connect(
-                    server_url.replace("http", "ws", 1) + path,
-                    origin=origin,
-                    additional_headers={"Cookie": cookie},
-                    proxy=None,
-                ):
-                    pytest.fail("Unauthorized WebSocket was accepted")
-            assert error.value.response.status_code == 403
-
-    asyncio.run(check_denials())
-    assert requests.get(server_url + "/health/", timeout=10).status_code == 200
-
-
-@pytest.mark.parametrize(
-    "mode,proxy_whitelist,use_cookie,http_status",
-    [
-        ("unsafe-onedomain-noadmin", "", True, 403),
-        ("safe-onedomain-nojsreplay", "127.0.0.1/32", False, 200),
-        ("safe-onedomain-nojsreplay", "192.0.2.0/24", False, 302),
-    ],
-)
-def test_agent_websocket_matches_http_security_policy(agent_server, live_opencode, mode, proxy_whitelist, use_cookie, http_status):
-    from urllib.parse import urlsplit
-    from websockets.asyncio.client import connect
-    from websockets.exceptions import InvalidStatus
-
-    server_url, data_dir, process = agent_server
-    cookie = _login_cookie(server_url)
-    created = requests.post(
-        server_url + "/admin/agent/opencode/pty",
-        headers={"Cookie": cookie, "Origin": server_url},
-        json={"command": "/bin/sh", "args": []},
-        timeout=10,
-    )
-    assert created.status_code == 200
-    pty_id = created.json()["id"]
-    stop_archivebox_process(process)
-    _set_archivebox_config(
-        data_dir,
-        f"SERVER_SECURITY_MODE={mode}",
-        f"REVERSE_PROXY_WHITELIST={proxy_whitelist}",
-        "REVERSE_PROXY_USER_HEADER=Remote-User",
-    )
-    restarted = start_archivebox_server(
-        data_dir,
-        port=urlsplit(server_url).port,
-        env=live_opencode.config.env,
-        log_name="agent-security-server.log",
-    )
-    headers = {"Cookie": cookie} if use_cookie else {"Remote-User": "agent-browser-test"}
-    try:
-        response = requests.get(server_url + "/admin/agent/opencode/global/health", headers=headers, allow_redirects=False, timeout=10)
-        assert response.status_code == http_status
-
-        async def check_socket():
-            connection = connect(
-                server_url.replace("http", "ws", 1) + f"/admin/agent/opencode/pty/{pty_id}/connect",
-                origin=server_url,
-                additional_headers=headers,
-                proxy=None,
-            )
-            if http_status != 200:
-                with pytest.raises(InvalidStatus) as error:
-                    async with connection:
-                        pytest.fail("Disabled control plane or untrusted proxy accepted a WebSocket")
-                assert error.value.response.status_code == 403
-                return
-            async with connection as socket, asyncio.timeout(10):
-                await socket.send("printf 'ABX_%s\\n' PROXY_OK\n")
-                output = ""
-                while "ABX_PROXY_OK" not in output:
-                    chunk = await socket.recv()
-                    output += chunk.decode() if isinstance(chunk, bytes) else chunk
-                assert "ABX_PROXY_OK" in output
-
-        asyncio.run(check_socket())
-        assert requests.get(server_url + "/health/", timeout=10).status_code == 200
-    finally:
-        deleted = requests.delete(live_opencode.settings["origin"] + f"/pty/{pty_id}", timeout=10)
-        assert deleted.status_code == 200
-        stop_archivebox_process(restarted)
-
-
-def test_agent_preserves_projects_and_survives_storage_failure(agent_server, browser_runtime):
-    server_url, data_dir, _ = agent_server
-    script = r"""
-const assert = require('node:assert/strict');
-const puppeteer = require('puppeteer');
-const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
-(async () => {
-  for (const disabled of [false, true]) {
-    const browser = await puppeteer.launch({
-      executablePath: config.chrome,
-      headless: true,
-      // Headless interaction must not depend on the desktop display clock.
-      args: ['--no-sandbox', '--disable-frame-rate-limit', ...(disabled ? ['--disable-local-storage'] : [])],
-    });
-    try {
-      const page = await browser.newPage();
-      await page.goto(new URL('/admin/login/', config.url).href, {waitUntil: 'domcontentloaded'});
-      await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
-      await page.locator('#login-form input[name="password"]').fill('test-password');
-      assert.equal(await page.$eval('#login-form input[name="username"]', element => element.value), 'agent-browser-test');
-      assert.equal(await page.$eval('#login-form input[name="password"]', element => element.value), 'test-password');
-      await Promise.all([
-        page.waitForNavigation({waitUntil: 'domcontentloaded'}),
-        page.locator('#login-form input[type="submit"]').click(),
-      ]);
-      assert.ok(!page.url().includes('/admin/login/'), page.url());
-      assert.equal((await page.goto(config.url, {waitUntil: 'domcontentloaded'})).status(), 200);
-      const welcome = '#opencode-agent-welcome';
-      await page.waitForSelector(welcome, {visible: true});
-      if (disabled) {
-        assert.equal(await page.evaluate(() => {
-          try { return localStorage === null; }
-          catch (error) { if (error.name !== 'SecurityError') throw error; return true; }
-        }), true);
-      } else {
-        await page.click('#opencode-agent-welcome-dismiss');
-        await page.waitForSelector(welcome, {hidden: true});
-        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
-        assert.equal(await page.$eval(welcome, element => element.hidden), true);
-        const existing = {
-          list: [{type: 'http', http: {url: 'http://other.example'}}],
-          projects: {
-            local: [{worktree: '/other-project', expanded: false}],
-            'http://other.example': [{worktree: '/remote-project', expanded: true}],
-          },
-          lastProject: {'http://other.example': '/remote-project'},
-        };
-        await page.evaluate(value => localStorage.setItem('opencode.global.dat:server', JSON.stringify(value)), existing);
-        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
-        await page.waitForFunction(workdir => {
-          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
-          return state.projects.local?.some(project => project.worktree === workdir);
-        }, {}, config.workdir);
-        const saved = await page.evaluate(() => JSON.parse(localStorage.getItem('opencode.global.dat:server')));
-        assert.deepEqual(saved.list, existing.list);
-        assert.deepEqual(saved.projects['http://other.example'], existing.projects['http://other.example']);
-        assert.equal(saved.lastProject['http://other.example'], '/remote-project');
-        const projects = saved.projects.local;
-        assert.ok(Array.isArray(projects), JSON.stringify({url: page.url(), saved, existing}));
-        assert.deepEqual(projects.find(project => project.worktree === '/other-project'), {worktree: '/other-project', expanded: false});
-        assert.equal(projects.filter(project => project.worktree === config.workdir).length, 1);
-        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
-        assert.equal(await page.evaluate(workdir => {
-          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
-          return state.projects.local.filter(project => project.worktree === workdir).length;
-        }, config.workdir), 1);
-
-        // Fill the actual browser quota; do not replace or intercept Storage methods.
-        const failure = await page.evaluate(() => {
-          localStorage.clear();
-          let low = 0, high = 8 * 1024 * 1024;
-          while (low < high) {
-            const size = Math.ceil((low + high) / 2);
-            try { localStorage.setItem('padding', 'x'.repeat(size)); low = size; }
-            catch (error) { if (error.name !== 'QuotaExceededError') throw error; high = size - 1; }
-          }
-          try { localStorage.setItem('extra', '1'); return null; }
-          catch (error) { return error.name; }
-        });
-        assert.equal(failure, 'QuotaExceededError');
-        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
-        await page.waitForSelector(welcome, {visible: true});
-      }
-      await page.click('#opencode-agent-welcome-dismiss');
-      await page.waitForSelector(welcome, {hidden: true});
-      assert.equal((await page.goto(new URL('/add/', config.url).href, {waitUntil: 'domcontentloaded'})).status(), 200);
-    } finally {
-      await browser.close();
-    }
-  }
-  console.log('AGENT_STORAGE_FAILURE_ISOLATED');
-})().catch(error => { console.error(error); process.exitCode = 1; });
-"""
-    result = subprocess.run(
-        [str(browser_runtime["node_binary"]), "-e", script],
-        input=json.dumps(
-            {
-                "chrome": str(browser_runtime["chrome_binary"]),
-                "url": f"{server_url}/admin/agent",
-                "workdir": str(data_dir.resolve()),
-            },
-        ),
-        env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
-        text=True,
-        capture_output=True,
-        timeout=120,
-    )
-    assert result.returncode == 0, result.stderr or result.stdout
-    assert "AGENT_STORAGE_FAILURE_ISOLATED" in result.stdout
@@ -0,0 +1,90 @@
+"""Exercise the real agent wrapper with Chromium's native storage failures."""
+
+import asyncio
+
+import pytest
+import requests
+
+from .conftest import start_archivebox_server, stop_archivebox_process
+from .test_opencode_agent_1 import _set_archivebox_config
+from .test_opencode_agent_1 import installed_opencode as installed_opencode
+from .test_opencode_agent_1 import live_opencode as live_opencode
+from .test_opencode_agent_1 import opencode_archive_config as opencode_archive_config
+from .test_server_security_browser import browser_runtime as browser_runtime
+
+
+from .test_opencode_browser_1 import (
+    pytestmark as pytestmark,
+    agent_server as agent_server,
+    _login_cookie as _login_cookie,
+)
+
+
+@pytest.mark.parametrize(
+    "mode,proxy_whitelist,use_cookie,http_status",
+    [
+        ("unsafe-onedomain-noadmin", "", True, 403),
+        ("safe-onedomain-nojsreplay", "127.0.0.1/32", False, 200),
+        ("safe-onedomain-nojsreplay", "192.0.2.0/24", False, 302),
+    ],
+)
+def test_agent_websocket_matches_http_security_policy(agent_server, live_opencode, mode, proxy_whitelist, use_cookie, http_status):
+    from urllib.parse import urlsplit
+    from websockets.asyncio.client import connect
+    from websockets.exceptions import InvalidStatus
+
+    server_url, data_dir, process = agent_server
+    cookie = _login_cookie(server_url)
+    created = requests.post(
+        server_url + "/admin/agent/opencode/pty",
+        headers={"Cookie": cookie, "Origin": server_url},
+        json={"command": "/bin/sh", "args": []},
+        timeout=10,
+    )
+    assert created.status_code == 200
+    pty_id = created.json()["id"]
+    stop_archivebox_process(process)
+    _set_archivebox_config(
+        data_dir,
+        f"SERVER_SECURITY_MODE={mode}",
+        f"REVERSE_PROXY_WHITELIST={proxy_whitelist}",
+        "REVERSE_PROXY_USER_HEADER=Remote-User",
+    )
+    restarted = start_archivebox_server(
+        data_dir,
+        port=urlsplit(server_url).port,
+        env=live_opencode.config.env,
+        log_name="agent-security-server.log",
+    )
+    headers = {"Cookie": cookie} if use_cookie else {"Remote-User": "agent-browser-test"}
+    try:
+        response = requests.get(server_url + "/admin/agent/opencode/global/health", headers=headers, allow_redirects=False, timeout=10)
+        assert response.status_code == http_status
+
+        async def check_socket():
+            connection = connect(
+                server_url.replace("http", "ws", 1) + f"/admin/agent/opencode/pty/{pty_id}/connect",
+                origin=server_url,
+                additional_headers=headers,
+                proxy=None,
+            )
+            if http_status != 200:
+                with pytest.raises(InvalidStatus) as error:
+                    async with connection:
+                        pytest.fail("Disabled control plane or untrusted proxy accepted a WebSocket")
+                assert error.value.response.status_code == 403
+                return
+            async with connection as socket, asyncio.timeout(10):
+                await socket.send("printf 'ABX_%s\\n' PROXY_OK\n")
+                output = ""
+                while "ABX_PROXY_OK" not in output:
+                    chunk = await socket.recv()
+                    output += chunk.decode() if isinstance(chunk, bytes) else chunk
+                assert "ABX_PROXY_OK" in output
+
+        asyncio.run(check_socket())
+        assert requests.get(server_url + "/health/", timeout=10).status_code == 200
+    finally:
+        deleted = requests.delete(live_opencode.settings["origin"] + f"/pty/{pty_id}", timeout=10)
+        assert deleted.status_code == 200
+        stop_archivebox_process(restarted)
@@ -0,0 +1,162 @@
+"""Exercise the real agent wrapper with Chromium's native storage failures."""
+
+import asyncio
+import json
+import os
+import subprocess
+
+import pytest
+import requests
+
+from .test_opencode_agent_1 import installed_opencode as installed_opencode
+from .test_opencode_agent_1 import live_opencode as live_opencode
+from .test_opencode_agent_1 import opencode_archive_config as opencode_archive_config
+from .test_server_security_browser import browser_runtime as browser_runtime
+
+
+from .test_opencode_browser_1 import (
+    pytestmark as pytestmark,
+    agent_server as agent_server,
+    _login_cookie as _login_cookie,
+)
+
+
+def test_agent_websocket_rejects_unauthorized_access(agent_server):
+    from websockets.asyncio.client import connect
+    from websockets.exceptions import InvalidStatus
+
+    server_url, _, _ = agent_server
+    admin_cookie = _login_cookie(server_url)
+    regular_cookie = _login_cookie(server_url, "agent-regular-test")
+
+    async def check_denials():
+        for path, cookie, origin in (
+            ("/admin/agent/opencode/pty/unknown/connect", "", server_url),
+            ("/admin/agent/opencode/pty/unknown/connect", regular_cookie, server_url),
+            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, "https://untrusted.example"),
+            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, None),
+            ("/health/", admin_cookie, server_url),
+        ):
+            with pytest.raises(InvalidStatus) as error:
+                async with connect(
+                    server_url.replace("http", "ws", 1) + path,
+                    origin=origin,
+                    additional_headers={"Cookie": cookie},
+                    proxy=None,
+                ):
+                    pytest.fail("Unauthorized WebSocket was accepted")
+            assert error.value.response.status_code == 403
+
+    asyncio.run(check_denials())
+    assert requests.get(server_url + "/health/", timeout=10).status_code == 200
+
+
+def test_agent_preserves_projects_and_survives_storage_failure(agent_server, browser_runtime):
+    server_url, data_dir, _ = agent_server
+    script = r"""
+const assert = require('node:assert/strict');
+const puppeteer = require('puppeteer');
+const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
+(async () => {
+  for (const disabled of [false, true]) {
+    const browser = await puppeteer.launch({
+      executablePath: config.chrome,
+      headless: true,
+      // Headless interaction must not depend on the desktop display clock.
+      args: ['--no-sandbox', '--disable-frame-rate-limit', ...(disabled ? ['--disable-local-storage'] : [])],
+    });
+    try {
+      const page = await browser.newPage();
+      await page.goto(new URL('/admin/login/', config.url).href, {waitUntil: 'domcontentloaded'});
+      await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
+      await page.locator('#login-form input[name="password"]').fill('test-password');
+      assert.equal(await page.$eval('#login-form input[name="username"]', element => element.value), 'agent-browser-test');
+      assert.equal(await page.$eval('#login-form input[name="password"]', element => element.value), 'test-password');
+      await Promise.all([
+        page.waitForNavigation({waitUntil: 'domcontentloaded'}),
+        page.locator('#login-form input[type="submit"]').click(),
+      ]);
+      assert.ok(!page.url().includes('/admin/login/'), page.url());
+      assert.equal((await page.goto(config.url, {waitUntil: 'domcontentloaded'})).status(), 200);
+      const welcome = '#opencode-agent-welcome';
+      await page.waitForSelector(welcome, {visible: true});
+      if (disabled) {
+        assert.equal(await page.evaluate(() => {
+          try { return localStorage === null; }
+          catch (error) { if (error.name !== 'SecurityError') throw error; return true; }
+        }), true);
+      } else {
+        await page.click('#opencode-agent-welcome-dismiss');
+        await page.waitForSelector(welcome, {hidden: true});
+        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
+        assert.equal(await page.$eval(welcome, element => element.hidden), true);
+        const existing = {
+          list: [{type: 'http', http: {url: 'http://other.example'}}],
+          projects: {
+            local: [{worktree: '/other-project', expanded: false}],
+            'http://other.example': [{worktree: '/remote-project', expanded: true}],
+          },
+          lastProject: {'http://other.example': '/remote-project'},
+        };
+        await page.evaluate(value => localStorage.setItem('opencode.global.dat:server', JSON.stringify(value)), existing);
+        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
+        await page.waitForFunction(workdir => {
+          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
+          return state.projects.local?.some(project => project.worktree === workdir);
+        }, {}, config.workdir);
+        const saved = await page.evaluate(() => JSON.parse(localStorage.getItem('opencode.global.dat:server')));
+        assert.deepEqual(saved.list, existing.list);
+        assert.deepEqual(saved.projects['http://other.example'], existing.projects['http://other.example']);
+        assert.equal(saved.lastProject['http://other.example'], '/remote-project');
+        const projects = saved.projects.local;
+        assert.ok(Array.isArray(projects), JSON.stringify({url: page.url(), saved, existing}));
+        assert.deepEqual(projects.find(project => project.worktree === '/other-project'), {worktree: '/other-project', expanded: false});
+        assert.equal(projects.filter(project => project.worktree === config.workdir).length, 1);
+        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
+        assert.equal(await page.evaluate(workdir => {
+          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
+          return state.projects.local.filter(project => project.worktree === workdir).length;
+        }, config.workdir), 1);
+
+        // Fill the actual browser quota; do not replace or intercept Storage methods.
+        const failure = await page.evaluate(() => {
+          localStorage.clear();
+          let low = 0, high = 8 * 1024 * 1024;
+          while (low < high) {
+            const size = Math.ceil((low + high) / 2);
+            try { localStorage.setItem('padding', 'x'.repeat(size)); low = size; }
+            catch (error) { if (error.name !== 'QuotaExceededError') throw error; high = size - 1; }
+          }
+          try { localStorage.setItem('extra', '1'); return null; }
+          catch (error) { return error.name; }
+        });
+        assert.equal(failure, 'QuotaExceededError');
+        await page.goto(config.url, {waitUntil: 'domcontentloaded'});
+        await page.waitForSelector(welcome, {visible: true});
+      }
+      await page.click('#opencode-agent-welcome-dismiss');
+      await page.waitForSelector(welcome, {hidden: true});
+      assert.equal((await page.goto(new URL('/add/', config.url).href, {waitUntil: 'domcontentloaded'})).status(), 200);
+    } finally {
+      await browser.close();
+    }
+  }
+  console.log('AGENT_STORAGE_FAILURE_ISOLATED');
+})().catch(error => { console.error(error); process.exitCode = 1; });
+"""
+    result = subprocess.run(
+        [str(browser_runtime["node_binary"]), "-e", script],
+        input=json.dumps(
+            {
+                "chrome": str(browser_runtime["chrome_binary"]),
+                "url": f"{server_url}/admin/agent",
+                "workdir": str(data_dir.resolve()),
+            },
+        ),
+        env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
+        text=True,
+        capture_output=True,
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "AGENT_STORAGE_FAILURE_ISOLATED" in result.stdout
@@ -91,7 +91,12 @@ def test_tlsnotary_preview_verifies_saved_evidence_without_remote_requests(snaps
        assert "Signed response byte range" in topology.inner_text()
        assert topology.locator('a[href*="response.http"]').count() > 0
        assert topology.locator('a[href*="receipt.json"]').count() > 0
-        assert page.locator("a", has_text="Verify independently").get_attribute("href") == "https://tlsnotary.zervice.io/"
+        # The installed plugin owns its verifier domain. Compare its template
+        # with its declared default so a plugin release does not require a
+        # simultaneous core test edit, while stale template links still fail.
+        schema = json.loads((Path(get_plugins_dir()) / "tlsnotary" / "config.json").read_text())
+        verifier_url = schema["properties"]["TLSNOTARY_VERIFIER_URL"]["default"]
+        assert page.locator("a", has_text="Verify independently").get_attribute("href") == f"{verifier_url.rstrip('/')}/"
        assert all(urlsplit(request).hostname == hostname for request in requests)
        page.screenshot(path=str(tmp_path / "tlsnotary-desktop.png"), full_page=True)
        page.set_viewport_size({"width": 390, "height": 844})
@@ -0,0 +1,266 @@
+#!/usr/bin/env python3
+"""Integration tests for recursive crawling functionality."""
+
+import json
+import os
+from pathlib import Path
+
+import pytest
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import run_archivebox_cmd
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+def run_add_until(args, env, condition, timeout=120):
+    assert args[0] == "archivebox"
+    result = run_archivebox_cmd(
+        args[1:],
+        cwd=Path.cwd(),
+        env=env,
+        timeout=timeout,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    with use_archivebox_db("."):
+        assert condition(), f"Condition was false after command completed: {' '.join(args)}"
+    return result.stdout, result.stderr
+
+
+@pytest.mark.timeout(1200)
+def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
+    initialized_archive,
+    free_tcp_port_factory,
+    recursive_test_site,
+):
+    """Run a bounded real depth=2 crawl with all plugins enabled and verify parallel snapshot execution."""
+
+    from archivebox.plugins.discovery import get_plugin_catalog
+
+    root_url = recursive_test_site["root_url"]
+    plugin_selection = ",".join(sorted(plugin for plugin in get_plugin_catalog() if not plugin.startswith("claude")))
+    env = os.environ.copy()
+    for preinstalled_path_key in (
+        "CHROME_BINARY",
+        "LIB_DIR",
+        "DATA_DIR",
+        "NODE_MODULES_DIR",
+        "NODE_PATH",
+        "PNPM_BIN_DIR",
+        "NPM_BIN_DIR",
+        "CHROMEWEBSTORE_EXTENSIONS_DIR",
+    ):
+        env.pop(preinstalled_path_key, None)
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "URL_ALLOWLIST": r"127\.0\.0\.1[:/].*",
+            "ABXPKG_LIB_DIR": str(initialized_archive / "lib"),
+            "CHROMEWEBSTORE_EXTENSIONS_DIR": str(initialized_archive / "lib/chromewebstore/extensions"),
+            "TIMEOUT": "90",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "3",
+            "SEARCH_BACKEND_SONIC_HOST_NAME": "127.0.0.1",
+            "SEARCH_BACKEND_SONIC_PORT": str(free_tcp_port_factory()),
+            "CHROME_HEADLESS": "true",
+            "CHROME_SANDBOX": "false",
+            "CHROME_ISOLATION": "snapshot",
+            "LITEPARSE_OCR_ENABLED": "false",
+            "LITEPARSE_MAX_SOURCES": "4",
+        },
+    )
+
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=2",
+            "--max-urls=8",
+            "--crawl-max-size=100mb",
+            "--tag=recursive-all-plugins",
+            "--parser=url_list",
+            f"--plugins={plugin_selection}",
+            root_url,
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=1200,
+    )
+    stdout, stderr = result.stdout, result.stderr
+
+    if stderr:
+        print(f"\n=== STDERR ===\n{stderr}\n=== END STDERR ===\n")
+    if stdout:
+        print(f"\n=== STDOUT (last 4000 chars) ===\n{stdout[-4000:]}\n=== END STDOUT ===\n")
+    assert result.returncode == 0, stderr or stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get(tags_str="recursive-all-plugins")
+        snapshots = list(
+            Snapshot.objects.filter(crawl=crawl)
+            .order_by("depth", "url")
+            .values_list("id", "url", "depth", "status", "parent_snapshot_id", "downloaded_at"),
+        )
+        archive_results = list(
+            ArchiveResult.objects.filter(snapshot__crawl=crawl)
+            .select_related("snapshot")
+            .order_by("snapshot__depth", "snapshot__url", "plugin", "hook_name")
+            .values_list(
+                "snapshot_id",
+                "snapshot__url",
+                "snapshot__depth",
+                "plugin",
+                "hook_name",
+                "status",
+                "output_files",
+                "output_size",
+                "output_str",
+            ),
+        )
+        process_snapshot_ids = {
+            process_id: str(snapshot_id)
+            for snapshot_id, process_id in ArchiveResult.objects.filter(
+                snapshot__crawl=crawl,
+                process_id__isnull=False,
+            ).values_list("snapshot_id", "process_id")
+        }
+        processes = list(
+            Process.objects.filter(process_type=Process.TypeChoices.HOOK, id__in=process_snapshot_ids)
+            .order_by("started_at")
+            .values_list("id", "pwd", "cmd", "status", "exit_code", "started_at", "ended_at"),
+        )
+
+    assert crawl.max_depth == 2
+    assert crawl.config["CRAWL_MAX_URLS"] == 8
+    assert crawl.config["CRAWL_MAX_SIZE"] == 100 * 1024 * 1024
+    assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 3
+    assert crawl.status == Crawl.StatusChoices.SEALED
+    assert crawl.retry_at is None
+
+    expected_urls = {
+        root_url,
+        *recursive_test_site["child_urls"],
+        *recursive_test_site["deep_urls"],
+    }
+    snapshots_by_url = {url: (snapshot_id, depth, parent_id) for snapshot_id, url, depth, _status, parent_id, _downloaded_at in snapshots}
+    assert set(snapshots_by_url) == expected_urls
+    root_id, root_depth, root_parent_id = snapshots_by_url[root_url]
+    assert root_depth == 0
+    assert root_parent_id is None
+    child_ids_by_url = {}
+    for child_url in recursive_test_site["child_urls"]:
+        child_id, child_depth, child_parent_id = snapshots_by_url[child_url]
+        assert child_depth == 1
+        assert child_parent_id == root_id
+        child_ids_by_url[child_url] = child_id
+    for child_url, deep_url in zip(recursive_test_site["child_urls"], recursive_test_site["deep_urls"], strict=True):
+        _deep_id, deep_depth, deep_parent_id = snapshots_by_url[deep_url]
+        assert deep_depth == 2
+        assert deep_parent_id == child_ids_by_url[child_url]
+    assert all(status == Snapshot.StatusChoices.SEALED for _id, _url, _depth, status, _parent, _downloaded_at in snapshots)
+    assert all(downloaded_at is not None for _id, _url, _depth, _status, _parent, downloaded_at in snapshots)
+
+    assert archive_results
+    allowed_statuses = {
+        ArchiveResult.StatusChoices.SUCCEEDED,
+        ArchiveResult.StatusChoices.NORESULTS,
+        ArchiveResult.StatusChoices.SKIPPED,
+    }
+    unexpected_results = [
+        {
+            "url": url,
+            "depth": depth,
+            "plugin": plugin,
+            "hook_name": hook_name,
+            "status": status,
+            "output_str": output_str,
+        }
+        for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
+        if not (status in allowed_statuses or (plugin == "archivedotorg" and status == ArchiveResult.StatusChoices.FAILED))
+    ]
+    # This fixture serves HTTP. TLSNotary must reject every plaintext document;
+    # a setup, timeout, or verifier error must not count as that expected refusal.
+    tlsnotary_results = [row for row in archive_results if row[3] == "tlsnotary"]
+    assert {row[1] for row in tlsnotary_results} == expected_urls
+    assert len(tlsnotary_results) == len(expected_urls)
+    with use_archivebox_db(initialized_archive):
+        for snapshot_id, _url, _depth, _plugin, hook_name, status, files, size, _output in tlsnotary_results:
+            assert status == ArchiveResult.StatusChoices.FAILED
+            assert not files
+            assert size == 0
+            process = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="tlsnotary", hook_name=hook_name).process
+            assert process is not None
+            assert process.status == Process.StatusChoices.EXITED
+            assert process.exit_code == 1
+            assert process.stderr.strip() == "[tlsnotary] TLSNotary requires an HTTPS document"
+    assert not [result for result in unexpected_results if result["plugin"] != "tlsnotary"]
+    ytdlp_results = [
+        (url, status, output_str)
+        for _snapshot_id, url, _depth, plugin, _hook_name, status, _files, _size, output_str in archive_results
+        if plugin == "ytdlp"
+    ]
+    assert ytdlp_results
+    assert all(status != ArchiveResult.StatusChoices.FAILED for _url, status, _output_str in ytdlp_results), ytdlp_results
+
+    plugins_seen = {plugin for _snapshot_id, _url, _depth, plugin, _hook_name, _status, _files, _size, _output in archive_results}
+    assert {
+        "wget",
+        "headers",
+        "title",
+        "pdf",
+        "screenshot",
+        "dom",
+        "singlefile",
+        "readability",
+        "mercury",
+        "htmltotext",
+        "favicon",
+        "parse_html_urls",
+        "archivedotorg",
+    }.issubset(plugins_seen)
+
+    snapshot_root = initialized_archive / "archive/users/system/snapshots"
+    assert list(snapshot_root.rglob("wget/**/*.html"))
+    assert list(snapshot_root.rglob("headers/**/headers.json"))
+    assert list(snapshot_root.rglob("title/title.txt"))
+    assert list(snapshot_root.rglob("pdf/**/*.pdf"))
+    assert list(snapshot_root.rglob("screenshot/**/*.png"))
+    assert list(snapshot_root.rglob("dom/**/*.html"))
+    assert list(snapshot_root.rglob("singlefile/**/*.html"))
+    assert list(snapshot_root.rglob("readability/**/*.html"))
+    assert list(snapshot_root.rglob("mercury/**/*.html"))
+    assert list(snapshot_root.rglob("htmltotext/**/*.txt"))
+    assert list(snapshot_root.rglob("favicon/**/*"))
+    urls_jsonl_files = list(snapshot_root.rglob("parse_html_urls/urls.jsonl"))
+    assert urls_jsonl_files
+    parsed_urls = {
+        json.loads(line)["url"] for path in urls_jsonl_files for line in path.read_text(errors="ignore").splitlines() if line.strip()
+    }
+    assert set(recursive_test_site["child_urls"]).issubset(parsed_urls)
+    assert set(recursive_test_site["deep_urls"]).issubset(parsed_urls)
+
+    assert processes
+    assert all(status == Process.StatusChoices.EXITED for _id, _pwd, _cmd, status, _exit_code, _started_at, _ended_at in processes)
+
+    intervals = []
+    for process_id, pwd, cmd, _status, _exit_code, started_at, ended_at in processes:
+        if not started_at or not ended_at:
+            continue
+        process_snapshot_id = process_snapshot_ids.get(process_id)
+        if process_snapshot_id is None:
+            continue
+        intervals.append((process_snapshot_id, started_at, ended_at, pwd, cmd))
+
+    overlapping = [
+        (left, right)
+        for index, left in enumerate(intervals)
+        for right in intervals[index + 1 :]
+        if left[0] != right[0] and left[1] < right[2] and right[1] < left[2]
+    ]
+    assert overlapping, f"Expected hook processes from different snapshots to overlap, got intervals: {intervals}"
+
+
+if __name__ == "__main__":
+    pytest.main([__file__, "-v"])
@@ -0,0 +1,79 @@
+#!/usr/bin/env python3
+"""Integration tests for recursive crawling functionality."""
+
+import os
+
+import pytest
+
+from archivebox.core.models import Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import run_archivebox_cmd
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_recursive_crawl_1 import (
+    pytestmark as pytestmark,
+    run_add_until as run_add_until,
+)
+
+
+@pytest.mark.timeout(600)
+def test_direct_url_crawl_measures_first_snapshot_before_parallel_admission(initialized_archive, recursive_test_site):
+    """A real three-URL capture must not launch all Chrome hook trees at once."""
+    urls = [recursive_test_site["root_url"], *recursive_test_site["child_urls"][:2]]
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "URL_ALLOWLIST": r"127\.0\.0\.1[:/].*",
+            "ABXPKG_LIB_DIR": str(initialized_archive / "lib"),
+            "CHROME_HEADLESS": "true",
+            "CHROME_SANDBOX": "false",
+            "CHROME_ISOLATION": "crawl",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "3",
+            "TIMEOUT": "90",
+        },
+    )
+    result = run_archivebox_cmd(
+        ["add", "--parser=url_list", "--plugins=chrome,wget,screenshot", "--tag=measured-admission", *urls],
+        cwd=initialized_archive,
+        env=env,
+        timeout=600,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get(tags_str="measured-admission")
+        snapshots = list(Snapshot.objects.filter(crawl=crawl).order_by("created_at"))
+        hook_intervals = list(
+            Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                archiveresult__snapshot__crawl=crawl,
+            )
+            .values_list("archiveresult__snapshot_id", "started_at", "ended_at")
+            .distinct(),
+        )
+
+    assert len(snapshots) == 3
+    assert all(snapshot.status == Snapshot.StatusChoices.SEALED and snapshot.downloaded_at for snapshot in snapshots)
+    first_id = snapshots[0].id
+    first_end = snapshots[0].downloaded_at
+    later_starts = [started for snapshot_id, started, _ended in hook_intervals if snapshot_id != first_id and started]
+    assert later_starts and first_end <= min(later_starts), hook_intervals
+    later_intervals = [
+        (snapshot_id, started, ended) for snapshot_id, started, ended in hook_intervals if snapshot_id != first_id and started and ended
+    ]
+    assert any(
+        left_id != right_id and left_start < right_end and right_start < left_end
+        for index, (left_id, left_start, left_end) in enumerate(later_intervals)
+        for right_id, right_start, right_end in later_intervals[index + 1 :]
+    ), later_intervals
+
+
+def test_crawl_snapshot_has_parent_snapshot_field(tmp_path, initialized_archive):
+    """Test that Snapshot model has parent_snapshot field."""
+
+    column_names = {field.column for field in Snapshot._meta.local_fields}
+
+    assert "parent_snapshot_id" in column_names, f"Snapshot table should have parent_snapshot_id column. Columns: {column_names}"
@@ -13,21 +13,10 @@ from archivebox.machine.models import Binary, Process
 from archivebox.tests.conftest import run_archivebox_cmd, cli_env
 from archivebox.tests.test_orm_helpers import use_archivebox_db

-pytestmark = pytest.mark.django_db(transaction=True)
-
-
-def run_add_until(args, env, condition, timeout=120):
-    assert args[0] == "archivebox"
-    result = run_archivebox_cmd(
-        args[1:],
-        cwd=Path.cwd(),
-        env=env,
-        timeout=timeout,
-    )
-    assert result.returncode == 0, result.stderr or result.stdout
-    with use_archivebox_db("."):
-        assert condition(), f"Condition was false after command completed: {' '.join(args)}"
-    return result.stdout, result.stderr
+from .test_recursive_crawl_1 import (
+    pytestmark as pytestmark,
+    run_add_until as run_add_until,
+)


 def test_background_hooks_dont_block_parser_extractors(tmp_path, initialized_archive, recursive_test_site):
@@ -492,246 +481,6 @@ def test_add_archivewebpage_installs_required_chrome_dependency(initialized_arch
    )


-@pytest.mark.timeout(1200)
-def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
-    initialized_archive,
-    free_tcp_port_factory,
-    recursive_test_site,
-):
-    """Run a bounded real depth=2 crawl with all plugins enabled and verify parallel snapshot execution."""
-
-    from archivebox.plugins.discovery import get_plugin_catalog
-
-    root_url = recursive_test_site["root_url"]
-    plugin_selection = ",".join(sorted(plugin for plugin in get_plugin_catalog() if not plugin.startswith("claude")))
-    env = os.environ.copy()
-    for preinstalled_path_key in (
-        "CHROME_BINARY",
-        "LIB_DIR",
-        "DATA_DIR",
-        "NODE_MODULES_DIR",
-        "NODE_PATH",
-        "PNPM_BIN_DIR",
-        "NPM_BIN_DIR",
-        "CHROMEWEBSTORE_EXTENSIONS_DIR",
-    ):
-        env.pop(preinstalled_path_key, None)
-    env.update(
-        {
-            "USE_COLOR": "false",
-            "SHOW_PROGRESS": "false",
-            "URL_ALLOWLIST": r"127\.0\.0\.1[:/].*",
-            "ABXPKG_LIB_DIR": str(initialized_archive / "lib"),
-            "CHROMEWEBSTORE_EXTENSIONS_DIR": str(initialized_archive / "lib/chromewebstore/extensions"),
-            "TIMEOUT": "90",
-            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "3",
-            "SEARCH_BACKEND_SONIC_HOST_NAME": "127.0.0.1",
-            "SEARCH_BACKEND_SONIC_PORT": str(free_tcp_port_factory()),
-            "CHROME_HEADLESS": "true",
-            "CHROME_SANDBOX": "false",
-            "CHROME_ISOLATION": "snapshot",
-            "LITEPARSE_OCR_ENABLED": "false",
-            "LITEPARSE_MAX_SOURCES": "4",
-        },
-    )
-
-    result = run_archivebox_cmd(
-        [
-            "add",
-            "--depth=2",
-            "--max-urls=8",
-            "--crawl-max-size=100mb",
-            "--tag=recursive-all-plugins",
-            "--parser=url_list",
-            f"--plugins={plugin_selection}",
-            root_url,
-        ],
-        cwd=initialized_archive,
-        env=env,
-        timeout=1200,
-    )
-    stdout, stderr = result.stdout, result.stderr
-
-    if stderr:
-        print(f"\n=== STDERR ===\n{stderr}\n=== END STDERR ===\n")
-    if stdout:
-        print(f"\n=== STDOUT (last 4000 chars) ===\n{stdout[-4000:]}\n=== END STDOUT ===\n")
-    assert result.returncode == 0, stderr or stdout
-
-    with use_archivebox_db(initialized_archive):
-        crawl = Crawl.objects.get(tags_str="recursive-all-plugins")
-        snapshots = list(
-            Snapshot.objects.filter(crawl=crawl)
-            .order_by("depth", "url")
-            .values_list("id", "url", "depth", "status", "parent_snapshot_id", "downloaded_at"),
-        )
-        archive_results = list(
-            ArchiveResult.objects.filter(snapshot__crawl=crawl)
-            .select_related("snapshot")
-            .order_by("snapshot__depth", "snapshot__url", "plugin", "hook_name")
-            .values_list(
-                "snapshot_id",
-                "snapshot__url",
-                "snapshot__depth",
-                "plugin",
-                "hook_name",
-                "status",
-                "output_files",
-                "output_size",
-                "output_str",
-            ),
-        )
-        process_snapshot_ids = {
-            process_id: str(snapshot_id)
-            for snapshot_id, process_id in ArchiveResult.objects.filter(
-                snapshot__crawl=crawl,
-                process_id__isnull=False,
-            ).values_list("snapshot_id", "process_id")
-        }
-        processes = list(
-            Process.objects.filter(process_type=Process.TypeChoices.HOOK, id__in=process_snapshot_ids)
-            .order_by("started_at")
-            .values_list("id", "pwd", "cmd", "status", "exit_code", "started_at", "ended_at"),
-        )
-
-    assert crawl.max_depth == 2
-    assert crawl.config["CRAWL_MAX_URLS"] == 8
-    assert crawl.config["CRAWL_MAX_SIZE"] == 100 * 1024 * 1024
-    assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 3
-    assert crawl.status == Crawl.StatusChoices.SEALED
-    assert crawl.retry_at is None
-
-    expected_urls = {
-        root_url,
-        *recursive_test_site["child_urls"],
-        *recursive_test_site["deep_urls"],
-    }
-    snapshots_by_url = {url: (snapshot_id, depth, parent_id) for snapshot_id, url, depth, _status, parent_id, _downloaded_at in snapshots}
-    assert set(snapshots_by_url) == expected_urls
-    root_id, root_depth, root_parent_id = snapshots_by_url[root_url]
-    assert root_depth == 0
-    assert root_parent_id is None
-    child_ids_by_url = {}
-    for child_url in recursive_test_site["child_urls"]:
-        child_id, child_depth, child_parent_id = snapshots_by_url[child_url]
-        assert child_depth == 1
-        assert child_parent_id == root_id
-        child_ids_by_url[child_url] = child_id
-    for child_url, deep_url in zip(recursive_test_site["child_urls"], recursive_test_site["deep_urls"], strict=True):
-        _deep_id, deep_depth, deep_parent_id = snapshots_by_url[deep_url]
-        assert deep_depth == 2
-        assert deep_parent_id == child_ids_by_url[child_url]
-    assert all(status == Snapshot.StatusChoices.SEALED for _id, _url, _depth, status, _parent, _downloaded_at in snapshots)
-    assert all(downloaded_at is not None for _id, _url, _depth, _status, _parent, downloaded_at in snapshots)
-
-    assert archive_results
-    allowed_statuses = {
-        ArchiveResult.StatusChoices.SUCCEEDED,
-        ArchiveResult.StatusChoices.NORESULTS,
-        ArchiveResult.StatusChoices.SKIPPED,
-    }
-    unexpected_results = [
-        {
-            "url": url,
-            "depth": depth,
-            "plugin": plugin,
-            "hook_name": hook_name,
-            "status": status,
-            "output_str": output_str,
-        }
-        for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
-        if not (status in allowed_statuses or (plugin == "archivedotorg" and status == ArchiveResult.StatusChoices.FAILED))
-    ]
-    # This fixture serves HTTP. TLSNotary must reject every plaintext document;
-    # a setup, timeout, or verifier error must not count as that expected refusal.
-    tlsnotary_results = [row for row in archive_results if row[3] == "tlsnotary"]
-    assert {row[1] for row in tlsnotary_results} == expected_urls
-    assert len(tlsnotary_results) == len(expected_urls)
-    with use_archivebox_db(initialized_archive):
-        for snapshot_id, _url, _depth, _plugin, hook_name, status, files, size, _output in tlsnotary_results:
-            assert status == ArchiveResult.StatusChoices.FAILED
-            assert not files
-            assert size == 0
-            process = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="tlsnotary", hook_name=hook_name).process
-            assert process is not None
-            assert process.status == Process.StatusChoices.EXITED
-            assert process.exit_code == 1
-            assert process.stderr.strip() == "[tlsnotary] TLSNotary requires an HTTPS document"
-    assert not [result for result in unexpected_results if result["plugin"] != "tlsnotary"]
-    ytdlp_results = [
-        (url, status, output_str)
-        for _snapshot_id, url, _depth, plugin, _hook_name, status, _files, _size, output_str in archive_results
-        if plugin == "ytdlp"
-    ]
-    assert ytdlp_results
-    assert all(status != ArchiveResult.StatusChoices.FAILED for _url, status, _output_str in ytdlp_results), ytdlp_results
-
-    plugins_seen = {plugin for _snapshot_id, _url, _depth, plugin, _hook_name, _status, _files, _size, _output in archive_results}
-    assert {
-        "wget",
-        "headers",
-        "title",
-        "pdf",
-        "screenshot",
-        "dom",
-        "singlefile",
-        "readability",
-        "mercury",
-        "htmltotext",
-        "favicon",
-        "parse_html_urls",
-        "archivedotorg",
-    }.issubset(plugins_seen)
-
-    snapshot_root = initialized_archive / "archive/users/system/snapshots"
-    assert list(snapshot_root.rglob("wget/**/*.html"))
-    assert list(snapshot_root.rglob("headers/**/headers.json"))
-    assert list(snapshot_root.rglob("title/title.txt"))
-    assert list(snapshot_root.rglob("pdf/**/*.pdf"))
-    assert list(snapshot_root.rglob("screenshot/**/*.png"))
-    assert list(snapshot_root.rglob("dom/**/*.html"))
-    assert list(snapshot_root.rglob("singlefile/**/*.html"))
-    assert list(snapshot_root.rglob("readability/**/*.html"))
-    assert list(snapshot_root.rglob("mercury/**/*.html"))
-    assert list(snapshot_root.rglob("htmltotext/**/*.txt"))
-    assert list(snapshot_root.rglob("favicon/**/*"))
-    urls_jsonl_files = list(snapshot_root.rglob("parse_html_urls/urls.jsonl"))
-    assert urls_jsonl_files
-    parsed_urls = {
-        json.loads(line)["url"] for path in urls_jsonl_files for line in path.read_text(errors="ignore").splitlines() if line.strip()
-    }
-    assert set(recursive_test_site["child_urls"]).issubset(parsed_urls)
-    assert set(recursive_test_site["deep_urls"]).issubset(parsed_urls)
-
-    assert processes
-    assert all(status == Process.StatusChoices.EXITED for _id, _pwd, _cmd, status, _exit_code, _started_at, _ended_at in processes)
-
-    intervals = []
-    for process_id, pwd, cmd, _status, _exit_code, started_at, ended_at in processes:
-        if not started_at or not ended_at:
-            continue
-        process_snapshot_id = process_snapshot_ids.get(process_id)
-        if process_snapshot_id is None:
-            continue
-        intervals.append((process_snapshot_id, started_at, ended_at, pwd, cmd))
-
-    overlapping = [
-        (left, right)
-        for index, left in enumerate(intervals)
-        for right in intervals[index + 1 :]
-        if left[0] != right[0] and left[1] < right[2] and right[1] < left[2]
-    ]
-    assert overlapping, f"Expected hook processes from different snapshots to overlap, got intervals: {intervals}"
-
-
-def test_crawl_snapshot_has_parent_snapshot_field(tmp_path, initialized_archive):
-    """Test that Snapshot model has parent_snapshot field."""
-
-    column_names = {field.column for field in Snapshot._meta.local_fields}
-
-    assert "parent_snapshot_id" in column_names, f"Snapshot table should have parent_snapshot_id column. Columns: {column_names}"
-
-
 def test_snapshot_depth_field_exists(tmp_path, initialized_archive):
    """Test that Snapshot model has depth field."""

@@ -803,7 +552,3 @@ def test_archiveresult_worker_queue_filters_by_foreground_extractors(tmp_path, i
        assert len(non_queued) > 0 or len(parser_status) == 0, (
            f"With {len(bg_results)} background hooks started, parser extractors should still run. Got statuses: {parser_statuses}"
        )
-
-
-if __name__ == "__main__":
-    pytest.main([__file__, "-v"])
@@ -0,0 +1,174 @@
+import json
+import importlib.util
+from pathlib import Path
+import re
+import subprocess
+import sys
+
+import pytest
+
+
+REPO_ROOT = Path(__file__).resolve().parents[2]
+FINGERPRINT_SCRIPT = REPO_ROOT / "bin" / "release-source-fingerprint.py"
+_SPEC = importlib.util.spec_from_file_location("release_source_fingerprint", FINGERPRINT_SCRIPT)
+assert _SPEC is not None and _SPEC.loader is not None
+_FINGERPRINT_MODULE = importlib.util.module_from_spec(_SPEC)
+_SPEC.loader.exec_module(_FINGERPRINT_MODULE)
+
+
+def _git(repo: Path, *args: str) -> str:
+    return subprocess.run(
+        ["git", "-C", str(repo), *args],
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout.strip()
+
+
+def _commit(repo: Path, message: str) -> str:
+    _git(repo, "add", "-A")
+    _git(repo, "commit", "-m", message)
+    return _git(repo, "rev-parse", "HEAD")
+
+
+def _fingerprint(repo: Path, revision: str) -> str:
+    return subprocess.run(
+        [sys.executable, str(FINGERPRINT_SCRIPT), revision],
+        cwd=repo,
+        check=True,
+        capture_output=True,
+        text=True,
+    ).stdout.strip()
+
+
+def _write_template_tree(repo: Path) -> None:
+    (repo / "etc").mkdir()
+    project = (REPO_ROOT / "pyproject.toml").read_text()
+    project = re.sub(r'^version = "[^"]+"$', 'version = "0.9.56rc41"', project, count=1, flags=re.M)
+    project = re.sub(r'^current_version = "[^"]+"$', 'current_version = "v0.9.56rc41"', project, count=1, flags=re.M)
+    assert 'version = "0.9.56rc41"' in project
+    assert 'current_version = "v0.9.56rc41"' in project
+    lock, count = re.subn(
+        r'(\[\[package\]\]\nname = "archivebox"\nversion = ")[^"]+',
+        r"\g<1>0.9.56rc41",
+        (REPO_ROOT / "uv.lock").read_text(),
+        count=1,
+    )
+    assert count == 1
+    package = json.loads((REPO_ROOT / "etc/package.json").read_text())
+    package["version"] = "0.9.56rc41"
+    (repo / "pyproject.toml").write_text(project)
+    (repo / "uv.lock").write_text(lock)
+    (repo / "etc/package.json").write_text(json.dumps(package, indent=2) + "\n")
+    (repo / "source.py").write_text("release behavior = unchanged\n")
+
+
+def _new_repo(tmp_path: Path) -> tuple[Path, str]:
+    repo = tmp_path / "repo"
+    repo.mkdir()
+    subprocess.run(["git", "init", "-q", str(repo)], check=True)
+    _git(repo, "config", "user.name", "ArchiveBox Tests")
+    _git(repo, "config", "user.email", "tests@archivebox.io")
+    _write_template_tree(repo)
+    return repo, _commit(repo, "stable acceptance source")
+
+
+def test_fingerprint_ignores_only_archivebox_release_version_fields(tmp_path):
+    repo, accepted = _new_repo(tmp_path)
+    project = repo / "pyproject.toml"
+    text = project.read_text()
+    text = re.sub(r'^version = "[^"]+"$', 'version = "0.9.64"', text, count=1, flags=re.M)
+    text = re.sub(r'^current_version = "[^"]+"$', 'current_version = "v0.9.64"', text, count=1, flags=re.M)
+    project.write_text(text)
+    lock = repo / "uv.lock"
+    lock.write_text(re.sub(r'(?m)^(version = ")0\.9\.56rc41("\s*)$', r"\g<1>0.9.64\2", lock.read_text(), count=1))
+    package = repo / "etc/package.json"
+    package.write_text(package.read_text().replace('"version": "0.9.56rc41"', '"version": "0.9.64"'))
+    stable = _commit(repo, "stable version bump")
+
+    assert _fingerprint(repo, accepted) == _fingerprint(repo, stable)
+
+
+def test_pyproject_version_normalization_matches_exact_template_bytes():
+    project = (REPO_ROOT / "pyproject.toml").read_bytes()
+    normalized = _FINGERPRINT_MODULE.normalize_file(b"pyproject.toml", project)
+    expected = re.sub(
+        rb'(?m)^(version = ")[^"\r\n]+("\s*(?:#.*)?\r?\n?)$',
+        lambda match: match.group(1) + b"<ARCHIVEBOX_PROJECT_VERSION>" + match.group(2),
+        project,
+        count=1,
+    )
+    expected = re.sub(
+        rb'(?m)^(current_version = ")[^"\r\n]+("\s*(?:#.*)?\r?\n?)$',
+        lambda match: match.group(1) + b"<ARCHIVEBOX_PROJECT_VERSION>" + match.group(2),
+        expected,
+        count=1,
+    )
+    assert normalized == expected
+
+
+@pytest.mark.parametrize(
+    "change",
+    [
+        "source",
+        "project_dependency",
+        "lock_dependency",
+        "package_dependency",
+        "other_tool_version",
+        "file_mode",
+    ],
+)
+def test_fingerprint_detects_non_release_version_changes(tmp_path, change):
+    repo, accepted = _new_repo(tmp_path)
+    if change == "source":
+        (repo / "source.py").write_text("release behavior = changed\n")
+    elif change == "project_dependency":
+        path = repo / "pyproject.toml"
+        project = path.read_text()
+        project, count = re.subn(r'("abx-dl==)[^"]+(")', r"\g<1>999.0\2", project, count=1)
+        assert count == 1
+        path.write_text(project)
+    elif change == "lock_dependency":
+        path = repo / "uv.lock"
+        path.write_text(re.sub(r'(?m)(^name = "abx-dl"\nversion = ")[^"]+', r"\g<1>999.0", path.read_text(), count=1))
+    elif change == "package_dependency":
+        path = repo / "etc/package.json"
+        payload = json.loads(path.read_text())
+        name = next(iter(payload["dependencies"]))
+        payload["dependencies"][name] = "999.0"
+        path.write_text(json.dumps(payload, indent=2) + "\n")
+    elif change == "other_tool_version":
+        path = repo / "pyproject.toml"
+        path.write_text(path.read_text() + '\n[tool.other]\nversion = "1.0"\n')
+        _commit(repo, "add unrelated tool version")
+        path.write_text(path.read_text().replace('version = "1.0"', 'version = "2.0"'))
+    elif change == "file_mode":
+        path = repo / "source.py"
+        path.chmod(0o755)
+
+    changed = _commit(repo, change)
+    assert _fingerprint(repo, accepted) != _fingerprint(repo, changed)
+
+
+def test_fingerprint_detects_submodule_gitlink_changes(tmp_path):
+    repo, accepted = _new_repo(tmp_path)
+    subrepo = tmp_path / "dependency"
+    subrepo.mkdir()
+    subprocess.run(["git", "init", "-q", str(subrepo)], check=True)
+    _git(subrepo, "config", "user.name", "ArchiveBox Tests")
+    _git(subrepo, "config", "user.email", "tests@archivebox.io")
+    (subrepo / "dependency.txt").write_text("one\n")
+    first = _commit(subrepo, "first dependency revision")
+    (repo / "vendor").mkdir()
+    _git(repo, "update-index", "--add", "--cacheinfo", f"160000,{first},vendor/dependency")
+    _git(repo, "commit", "-m", "add dependency gitlink")
+    one = _git(repo, "rev-parse", "HEAD")
+
+    (subrepo / "dependency.txt").write_text("two\n")
+    second = _commit(subrepo, "second dependency revision")
+    _git(repo, "update-index", "--cacheinfo", f"160000,{second},vendor/dependency")
+    _git(repo, "commit", "-m", "update dependency gitlink")
+    two = _git(repo, "rev-parse", "HEAD")
+
+    assert _fingerprint(repo, accepted) != _fingerprint(repo, one)
+    assert _fingerprint(repo, one) != _fingerprint(repo, two)
@@ -1,13 +1,168 @@
 from pathlib import Path
+import ast
+import json
+import importlib.util
 import shlex
+import os
+import re
+import shutil
+import subprocess

+import pytest
 import yaml


 REPO_ROOT = Path(__file__).resolve().parents[2]
 RELEASE_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "release.yml"
-RELEASE_CANDIDATE_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "release-candidate.yml"
+CI_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "ci.yml"
 PIP_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "pip.yml"
+DOCKER_WORKFLOW = REPO_ROOT / ".github" / "workflows" / "docker.yml"
+VERIFY_STAGING_SCRIPT = REPO_ROOT / "bin" / "verify-staging-release.py"
+_VERIFY_SPEC = importlib.util.spec_from_file_location("verify_staging_release", VERIFY_STAGING_SCRIPT)
+assert _VERIFY_SPEC is not None and _VERIFY_SPEC.loader is not None
+_VERIFY_STAGING = importlib.util.module_from_spec(_VERIFY_SPEC)
+_VERIFY_SPEC.loader.exec_module(_VERIFY_STAGING)
+
+
+def _git(repo, *args):
+    return subprocess.run(["git", "-C", str(repo), *args], check=True, capture_output=True, text=True).stdout.strip()
+
+
+@pytest.mark.parametrize(
+    "dev_change",
+    [None, "no_bump", "source", "project_dependency", "locked_dependency", "package_dependency", "other_tool_version"],
+)
+def test_stable_release_reconciles_concurrent_version_only_dev_bump(tmp_path, dev_change):
+    workflow = yaml.safe_load(RELEASE_WORKFLOW.read_text())
+    steps = workflow["jobs"]["docker-release"]["steps"]
+    sync_step = next(step for step in steps if step.get("name") == "Reconcile dev to the tested stable release")
+    origin = tmp_path / "origin.git"
+    work = tmp_path / "work"
+    subprocess.run(["git", "init", "--bare", str(origin)], check=True, capture_output=True)
+    subprocess.run(["git", "clone", str(origin), str(work)], check=True, capture_output=True)
+    _git(work, "config", "user.name", "ArchiveBox Tests")
+    _git(work, "config", "user.email", "tests@archivebox.io")
+    _git(work, "switch", "-c", "dev")
+    (work / "etc").mkdir()
+    project_template = (REPO_ROOT / "pyproject.toml").read_text() + '\n[tool.other]\nversion = "1.0"\n'
+    lock_template = (REPO_ROOT / "uv.lock").read_text()
+    package_template = (REPO_ROOT / "etc/package.json").read_text()
+
+    def write_version(version):
+        project, count = re.subn(r'^version = "[^"]+"$', f'version = "{version}"', project_template, count=1, flags=re.M)
+        assert count == 1
+        project, count = re.subn(r'^current_version = "[^"]+"$', f'current_version = "v{version}"', project, count=1, flags=re.M)
+        assert count == 1
+        lock, count = re.subn(
+            r'(\[\[package\]\]\nname = "archivebox"\nversion = ")[^"]+',
+            lambda match: f"{match.group(1)}{version}",
+            lock_template,
+            count=1,
+        )
+        assert count == 1
+        package, count = re.subn(
+            r'^(  "version": ")[^"]+',
+            lambda match: f"{match.group(1)}{version}",
+            package_template,
+            count=1,
+            flags=re.M,
+        )
+        assert count == 1
+        (work / "pyproject.toml").write_text(project)
+        (work / "uv.lock").write_text(lock)
+        (work / "etc/package.json").write_text(package)
+
+    write_version("0.9.56rc41")
+    (work / "source.py").write_text("tested = True\n")
+    _git(work, "add", ".")
+    _git(work, "commit", "-m", "tested source")
+    base = _git(work, "rev-parse", "HEAD")
+    _git(work, "push", "origin", "dev")
+    _git(work, "switch", "-c", "main")
+    write_version("0.9.64")
+    _git(work, "add", ".")
+    _git(work, "commit", "-m", "Bump release version to 0.9.64")
+    stable = _git(work, "rev-parse", "HEAD")
+    _git(work, "push", "origin", "main")
+    _git(work, "switch", "dev")
+    if dev_change != "no_bump":
+        write_version("0.9.56rc42")
+    if dev_change == "source":
+        (work / "source.py").write_text("tested = False\n")
+    elif dev_change == "project_dependency":
+        project = work / "pyproject.toml"
+        project.write_text(re.sub(r'"abx-dl==[^"]+"', '"abx-dl==999.0"', project.read_text(), count=1))
+    elif dev_change == "locked_dependency":
+        lock = work / "uv.lock"
+        lock.write_text(re.sub(r'(name = "abx-dl"\nversion = ")[^"]+', r"\g<1>999.0", lock.read_text(), count=1))
+    elif dev_change == "package_dependency":
+        package = work / "etc/package.json"
+        dependencies = json.loads(package.read_text())["dependencies"]
+        name, value = next(iter(dependencies.items()))
+        package.write_text(package.read_text().replace(f'"{name}": "{value}"', f'"{name}": "999.0"'))
+    elif dev_change == "other_tool_version":
+        project = work / "pyproject.toml"
+        project.write_text(project.read_text().replace('[tool.other]\nversion = "1.0"', '[tool.other]\nversion = "2.0"'))
+    if dev_change != "no_bump":
+        _git(work, "add", ".")
+        _git(work, "commit", "-m", "Bump release version to 0.9.56rc42")
+    candidate = _git(work, "rev-parse", "HEAD")
+    _git(work, "push", "origin", "dev")
+    _git(work, "checkout", "--detach", stable)
+
+    env = {**os.environ, "RELEASE_SHA": stable}
+    result = subprocess.run(["bash", "-c", sync_step["run"]], cwd=work, env=env, capture_output=True, text=True)
+    assert result.returncode == 0, result.stderr
+    merged = _git(work, "ls-remote", "origin", "refs/heads/dev").split()[0]
+
+    prepare_path = yaml.safe_load(CI_WORKFLOW.read_text())["jobs"]["prepare"]["uses"]
+    prepare = yaml.safe_load((REPO_ROOT / prepare_path).read_text())
+    channel_step = next(step for step in prepare["jobs"]["prepare"]["steps"] if step.get("id") == "channel")
+
+    def stable_channel(sha, branch="dev"):
+        output = tmp_path / "channel-output"
+        output.write_text("")
+        channel_env = {**env, "GITHUB_SHA": sha, "GITHUB_REF_NAME": branch, "GITHUB_OUTPUT": str(output)}
+        result = subprocess.run(["bash", "-c", channel_step["run"]], cwd=work, env=channel_env, capture_output=True, text=True)
+        assert result.returncode == 0, result.stderr
+        return output.read_text().strip()
+
+    if dev_change == "no_bump":
+        assert merged == stable
+        assert stable_channel(merged) == "stable=true"
+        return
+    if dev_change:
+        assert merged == candidate
+        assert "contains changes beyond" in result.stdout
+        return
+    assert merged not in {candidate, stable}
+    assert _git(work, "rev-list", "--parents", "-n", "1", merged).split() == [merged, stable, candidate]
+    assert _git(work, "rev-parse", f"{merged}^{{tree}}") == _git(work, "rev-parse", f"{stable}^{{tree}}")
+    assert _git(work, "merge-base", stable, candidate) == base
+    # Reconciliation creates a new merge commit with the tested main tree. Its
+    # version preparation must not immediately bump dev and undo synchronization.
+    assert stable_channel(merged) == "stable=true"
+    assert stable_channel(merged, branch="main") == "stable=false"
+
+    tag_script = next(step["run"] for step in steps if step.get("id") == "docker_meta")
+    start = tag_script.index("SYNC_DEV=false")
+    end = tag_script.index("\n{", start)
+    sync_script = tag_script[start:end] + '\nprintf "%s\\n" "$SYNC_DEV"\n'
+    sync_env = {**env, "GIT_BINARY": "git", "RELEASE_BRANCH": "main"}
+
+    def sync_dev():
+        result = subprocess.run(["bash", "-e", "-c", sync_script], cwd=work, env=sync_env, capture_output=True, text=True)
+        assert result.returncode == 0, result.stderr
+        return result.stdout.strip().splitlines()[-1]
+
+    assert sync_dev() == "true"
+    _git(work, "switch", "-c", "after-merge", merged)
+    (work / "source.py").write_text("tested = False\n")
+    _git(work, "add", "source.py")
+    _git(work, "commit", "-m", "New dev source change")
+    _git(work, "push", "origin", "HEAD:refs/heads/dev")
+    assert sync_dev() == "false"
+    assert stable_channel(_git(work, "rev-parse", "HEAD")) == "stable=false"


 def test_release_uses_registered_publisher_and_authorized_tag_credentials():
@@ -22,14 +177,35 @@ def test_release_uses_registered_publisher_and_authorized_tag_credentials():
    assert python_release["environment"] == "pypi"
    checkout = python_release["steps"][0]
    assert checkout["with"]["token"] == "${{ secrets.RELEASE_GH_TOKEN || github.token }}"
+
    assert docker_release["needs"] == ["candidate", "python-release"]
    assert docker_release["env"]["DOCKER_DIGEST_RUN_ID"] == "${{ needs.candidate.outputs.digest_run_id }}"
    assert "release_ready" not in docker_release["if"]
    assert jobs["cascade"]["if"] == "needs.python-release.outputs.release_ready == 'true'"

    assert all(step.get("name") != "Verify published PyPI package installs and runs" for step in python_release["steps"])
-    candidate = yaml.safe_load(RELEASE_CANDIDATE_WORKFLOW.read_text())
-    assert candidate["jobs"]["python-artifacts"]["with"]["full_tests"] is False
+    ci = yaml.safe_load(CI_WORKFLOW.read_text())
+    screenshots = yaml.safe_load((REPO_ROOT / ".github" / "workflows" / "screenshots.yml").read_text())
+    prepare_path = ci["jobs"]["prepare"]["uses"]
+    assert screenshots["jobs"]["prepare"]["uses"] == prepare_path
+    prepare = yaml.safe_load((REPO_ROOT / prepare_path).read_text())
+    assert prepare["jobs"]["prepare"]["steps"][-1]["uses"] == "ArchiveBox/monorepo/.github/actions/prepare-release-version@main"
+    assert screenshots["jobs"]["deploy"]["needs"] == "prepare"
+    assert screenshots["jobs"]["deploy"]["if"] == "needs.prepare.outputs.run_jobs == 'true'"
+    for name, job in ci["jobs"].items():
+        if name not in {"prepare", "required"}:
+            assert job["needs"] == "prepare"
+            assert job["if"] == "needs.prepare.outputs.run_jobs == 'true'"
+    gate = next(step["run"] for step in jobs["candidate"]["steps"] if step.get("id") == "verified")
+    assert 'ARTIFACT_RUN_ID="$CI_RUN_ID"' in gate
+    assert 'DIGEST_RUN_ID="$CI_RUN_ID"' in gate
+    assert 'ARTIFACT_RUN_ID="$CANDIDATE_RUN_ID"' not in gate
+    assert 'DIGEST_RUN_ID="$CANDIDATE_RUN_ID"' not in gate
+    docker_workflow = yaml.safe_load(DOCKER_WORKFLOW.read_text())
+    version_check = next(
+        step for step in docker_workflow["jobs"]["build"]["steps"] if step.get("name") == "Validate exact built image version and commit"
+    )
+    assert version_check["if"] == "inputs.full_tests"
    pip_workflow = yaml.safe_load(PIP_WORKFLOW.read_text())
    install_script = next(
        step["run"] for step in pip_workflow["jobs"]["build"]["steps"] if step.get("name") == "Release wheel import and CLI smoke"
@@ -47,6 +223,8 @@ def test_release_uses_registered_publisher_and_authorized_tag_credentials():
    assert '[[ "$TAG_TARGET" == "$RELEASE_SHA" ]]' in tag_script
    assert '$GIT_BINARY merge-base --is-ancestor "$RELEASE_SHA" origin/main' in tag_script
    assert '[[ "$MAIN_VERSION" == "$VERSION" ]]' in tag_script
+    assert '$GIT_BINARY merge-base --is-ancestor "$MAIN_TARGET" "$DEV_TARGET"' in tag_script
+    assert '$GIT_BINARY diff --quiet "$MAIN_TARGET" "$DEV_TARGET"' in tag_script
    assert 'echo "${DOCKERHUB_IMAGE}:dev"' in tag_script
    assert 'echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"' in tag_script
    assert 'echo "${DOCKERHUB_IMAGE}:${VERSION}"' in tag_script
@@ -97,3 +275,235 @@ def test_release_uses_registered_publisher_and_authorized_tag_credentials():
    create_release_line = logical_lines.index(next(line for line in logical_lines if line.startswith("$GH_BINARY release create ")))
    upload_release_line = logical_lines.index(next(line for line in logical_lines if line.startswith("$GH_BINARY release upload ")))
    assert publish_line < rc_skip_line < create_release_line < upload_release_line
+
+
+def test_stable_publication_requires_live_acceptance_before_upload():
+    workflow = yaml.safe_load(RELEASE_WORKFLOW.read_text())
+    publisher = workflow["jobs"]["python-release"]
+    steps = publisher["steps"]
+    gate = next(i for i, step in enumerate(steps) if step.get("name") == "Require matching acceptance on Cabbage")
+    upload = next(i for i, step in enumerate(steps) if step.get("name") == "Publish the exact tested distributions")
+    assert gate < upload
+    # The same guard applies to automatic and manually requested stable releases.
+    assert steps[gate]["if"] == "env.RELEASE_BRANCH == 'main'"
+    assert "verify-staging-release.py" in steps[gate]["run"]
+    assert not steps[gate].get("continue-on-error")
+    assert publisher["permissions"]["deployments"] == "read"
+    staging = yaml.safe_load((REPO_ROOT / ".github" / "workflows" / "staging-acceptance.yml").read_text())
+    assert staging["jobs"]["acceptance"]["strategy"]["matrix"]["target"] == ["cabbage"]
+    verifier = VERIFY_STAGING_SCRIPT.read_text()
+    assert 'for environment in ("cabbage",):' in verifier
+    assert "source_fingerprint" in verifier
+    assert 'inspection.get("health") != "healthy"' in verifier
+    assert "expected_abx_dl_digest" in verifier
+    assert 'run["conclusion"] != "success"' in verifier
+
+
+def test_staging_acceptance_uses_published_image_source_not_workflow_run_head():
+    release = yaml.safe_load(RELEASE_WORKFLOW.read_text())
+    docker_steps = release["jobs"]["docker-release"]["steps"]
+    verified = next(i for i, step in enumerate(docker_steps) if step.get("name") == "Verify published Docker images run")
+    source = next(i for i, step in enumerate(docker_steps) if step.get("name") == "Record published image source")
+    uploaded = next(i for i, step in enumerate(docker_steps) if step.get("name") == "Upload published image source")
+    assert verified < source < uploaded
+    assert docker_steps[uploaded]["with"]["name"] == "published-image-source"
+
+    staging_text = (REPO_ROOT / ".github" / "workflows" / "staging-acceptance.yml").read_text()
+    assert "branches: [dev]" not in staging_text
+    assert "workflow_run.head_branch" not in staging_text
+    assert "workflow_run.head_sha" not in staging_text
+    staging = yaml.safe_load(staging_text)
+    published = staging["jobs"]["published"]
+    download = next(step for step in published["steps"] if step.get("name") == "Download published image source")
+    assert download["with"]["name"] == docker_steps[uploaded]["with"]["name"]
+    assert download["with"]["run-id"] == "${{ github.event.workflow_run.id }}"
+    assert "head_branch" not in published["if"]
+    assert published["outputs"]["sha"] == "${{ steps.source.outputs.sha }}"
+    assert published["outputs"]["ready"] == "${{ steps.source.outputs.ready }}"
+    assert '"$RELEASE_BRANCH"' in docker_steps[source]["run"]
+    assert '"$RELEASE_SHA"' in docker_steps[source]["run"]
+    assert '"$branch" == dev' in next(step for step in published["steps"] if step.get("id") == "source")["run"]
+
+    acceptance = staging["jobs"]["acceptance"]
+    assert acceptance["steps"][0]["with"]["ref"] == "${{ needs.published.outputs.sha }}"
+    verify = next(
+        step for step in acceptance["steps"] if step.get("name") == "Verify actual deployment, full captures and rendered snapshots"
+    )
+    assert verify["env"]["STAGING_SHA"] == "${{ needs.published.outputs.sha }}"
+
+
+@pytest.mark.parametrize(("branch", "ready"), [("dev", "true"), ("main", "false")])
+def test_staging_selects_actual_published_branch(tmp_path, branch, ready):
+    release = yaml.safe_load(RELEASE_WORKFLOW.read_text())
+    record = next(step for step in release["jobs"]["docker-release"]["steps"] if step.get("name") == "Record published image source")
+    staging = yaml.safe_load((REPO_ROOT / ".github" / "workflows" / "staging-acceptance.yml").read_text())
+    select = next(step for step in staging["jobs"]["published"]["steps"] if step.get("id") == "source")
+    jq = shutil.which("jq")
+    assert jq is not None
+    sha = "a" * 40
+    output = tmp_path / "github-output"
+    output.touch()
+    env = {
+        **os.environ,
+        "RUNNER_TEMP": str(tmp_path),
+        "GITHUB_OUTPUT": str(output),
+        "JQ_BINARY": jq,
+        "RELEASE_BRANCH": branch,
+        "RELEASE_SHA": sha,
+    }
+
+    recorded = subprocess.run(["bash", "-c", record["run"]], cwd=tmp_path, env=env, capture_output=True, text=True)
+    assert recorded.returncode == 0, recorded.stderr
+    source = tmp_path / "published-image-source" / "source.json"
+    assert json.loads(source.read_text()) == {"branch": branch, "sha": sha}
+
+    selected = subprocess.run(["bash", "-c", select["run"]], cwd=tmp_path, env=env, capture_output=True, text=True)
+    assert selected.returncode == 0, selected.stderr
+    assert output.read_text().splitlines() == [f"sha={sha}", f"ready={ready}"]
+
+
+def test_staging_rejects_published_source_with_malformed_sha(tmp_path):
+    staging = yaml.safe_load((REPO_ROOT / ".github" / "workflows" / "staging-acceptance.yml").read_text())
+    select = next(step for step in staging["jobs"]["published"]["steps"] if step.get("id") == "source")
+    source = tmp_path / "published-image-source" / "source.json"
+    source.parent.mkdir()
+    source.write_text(json.dumps({"branch": "dev", "sha": "not-a-sha"}))
+    output = tmp_path / "github-output"
+    output.touch()
+    env = {**os.environ, "RUNNER_TEMP": str(tmp_path), "GITHUB_OUTPUT": str(output)}
+
+    selected = subprocess.run(["bash", "-c", select["run"]], cwd=tmp_path, env=env, capture_output=True, text=True)
+    assert selected.returncode != 0
+    assert output.read_text() == ""
+
+
+def test_staging_gate_matches_the_exact_tested_docker_base_digest(tmp_path):
+    workflow = yaml.safe_load(RELEASE_WORKFLOW.read_text())
+    publisher = workflow["jobs"]["python-release"]
+    steps = publisher["steps"]
+    metadata = next(step for step in steps if step.get("name") == "Download tested Docker base-image metadata")
+    gate = next(step for step in steps if step.get("name") == "Require matching acceptance on Cabbage")
+    assert metadata["with"]["run-id"] == "${{ env.DOCKER_DIGEST_RUN_ID }}"
+    assert publisher["env"]["DOCKER_DIGEST_RUN_ID"] == "${{ needs.candidate.outputs.digest_run_id }}"
+    assert metadata["with"]["pattern"] == "digest-*"
+    assert metadata["with"]["merge-multiple"] is True
+    assert metadata["with"]["path"] == "${{ runner.temp }}/tested-docker-artifacts"
+    assert steps.index(metadata) < steps.index(gate)
+    assert gate["env"]["STAGING_DOCKER_ARTIFACT_DIR"] == "${{ runner.temp }}/tested-docker-artifacts"
+
+    docker_workflow = yaml.safe_load(DOCKER_WORKFLOW.read_text())
+    export = next(step for step in docker_workflow["jobs"]["build"]["steps"] if step.get("name") == "Export digest")
+    assert export["env"]["ABX_DL_IMAGE_REF"] == "${{ steps.abx_dl_image.outputs.image }}"
+    assert '"/tmp/digests/abx-dl-image-$ARTIFACT_NAME"' in export["run"]
+    helper = (REPO_ROOT / "bin" / "staging-acceptance-host.sh").read_text()
+    assert "abx_dl_image" in helper
+
+    source_sha = "a" * 40
+    base_digest = "sha256:" + "b" * 64
+    (tmp_path / "source-ref").write_text(source_sha + "\n")
+    for platform in ("linux-amd64", "linux-arm64"):
+        (tmp_path / f"abx-dl-image-digest-{platform}").write_text(f"archivebox/abx-dl:1.13.51@{base_digest}\n")
+    assert _VERIFY_STAGING.tested_abx_dl_digest(tmp_path, source_sha) == base_digest
+
+    (tmp_path / "abx-dl-image-digest-linux-arm64").write_text(f"archivebox/abx-dl:1.13.52@{'sha256:' + 'c' * 64}\n")
+    with pytest.raises(SystemExit, match="different abx-dl images"):
+        _VERIFY_STAGING.tested_abx_dl_digest(tmp_path, source_sha)
+    with pytest.raises(SystemExit, match="different source commit"):
+        _VERIFY_STAGING.tested_abx_dl_digest(tmp_path, "d" * 40)
+
+
+def test_staging_acceptance_limits_only_exact_known_auth_and_tls_failures():
+    helper = (REPO_ROOT / "bin" / "staging-acceptance-host.sh").read_text()
+    embedded = re.search(r"script=\"\$\(cat <<'PY'\n(.*?)\nPY\n\)\"", helper, re.S)
+    assert embedded, "staging acceptance Python body not found"
+    tree = ast.parse(embedded.group(1))
+    classifier = next(node for node in tree.body if isinstance(node, ast.FunctionDef) and node.name == "accepted_limitation")
+    namespace = {}
+    exec(compile(ast.Module(body=[classifier], type_ignores=[]), "staging-acceptance", "exec"), namespace)
+    accepts = namespace["accepted_limitation"]
+
+    assert accepts("tlsnotary", "failed", "Not enough memory for TLSNotary")
+    assert accepts("claudechrome", "failed", "ANTHROPIC_API_KEY not set")
+    assert accepts("claudecodeextract", "failed", "Claude Code auth not set")
+    assert accepts("claudecodecleanup", "failed", "Claude Code auth not set")
+    assert not accepts("claudechrome", "failed", "ANTHROPIC_API_KEY not set; other failure")
+    assert not accepts("other-plugin", "failed", "Claude Code auth not set")
+    assert not accepts("claudechrome", "succeeded", "ANTHROPIC_API_KEY not set")
+    assert "'limitations': []" in embedded.group(1)
+    assert "'PLUGINS': ','.join(sorted(catalog))" in embedded.group(1)
+    assert "for plugin in capture_plugins:" in embedded.group(1)
+    assert "missing or truncated" in embedded.group(1)
+    assert "output.stat().st_size != metadata.get('size')" in embedded.group(1)
+    assert "tlsnotary reported unexpected status" in embedded.group(1)
+
+
+@pytest.mark.parametrize("requested", ["", "archivebox/tests/test_urls.py", "archivebox/tests/test_release_workflow.py"])
+def test_ci_batches_preserve_inventory_runner_ownership_and_single_file_runs(tmp_path, requested):
+    hosted_paths = {
+        path.relative_to(REPO_ROOT).as_posix()
+        for path in (REPO_ROOT / "archivebox/tests").glob("test_*.py")
+        if "# ci-runner: hosted" in path.read_text().splitlines()[:5]
+    }
+    result = subprocess.run(
+        ["uv", "run", "--no-cache", "--no-project", "python", str(REPO_ROOT / ".github/scripts/discover_test_matrix.py")],
+        cwd=tmp_path,
+        env={**os.environ, "TEST_FILE": requested, "UGNAS_CI_MAX_JOBS": "3"},
+        capture_output=True,
+        text=True,
+        check=True,
+    )
+    matrix = json.loads(result.stdout.splitlines()[1])
+    paths = [path for item in matrix for path in item["paths"]]
+    expected = sorted(path.relative_to(REPO_ROOT).as_posix() for path in (REPO_ROOT / "archivebox/tests").glob("test_*.py"))
+    assert sorted(paths) == expected
+    assert len(paths) == len(set(paths))
+    assert len(matrix) < len(paths)
+    assert all(1 <= len(item["paths"]) <= 8 for item in matrix)
+    assert sum(item["ugnas"] for item in matrix) == 3
+    assert not any(path in hosted_paths for item in matrix if item["ugnas"] for path in item["paths"])
+    ldap = next(item for item in matrix if item["extra"] == "ldap")
+    assert ldap["paths"] == ["archivebox/tests/test_auth_ldap.py"]
+    if requested:
+        assert len([item for item in matrix if item["paths"] == [requested]]) == 1
+
+
+def test_ci_discovery_handles_added_renamed_and_deleted_files_without_timing_edits(tmp_path):
+    script = tmp_path / ".github/scripts/discover_test_matrix.py"
+    script.parent.mkdir(parents=True)
+    shutil.copy2(REPO_ROOT / ".github/scripts/discover_test_matrix.py", script)
+    shutil.copy2(REPO_ROOT / ".github/test-durations.json", tmp_path / ".github/test-durations.json")
+    tests = tmp_path / "archivebox/tests"
+    tests.mkdir(parents=True)
+    for name in ("test_core_config.py", "test_util.py"):
+        shutil.copy2(REPO_ROOT / "archivebox/tests" / name, tests / name)
+
+    def discover():
+        result = subprocess.run(
+            ["uv", "run", "--no-cache", "--no-project", "python", str(script)],
+            cwd=tmp_path,
+            env={**os.environ, "TEST_FILE": "", "UGNAS_CI_MAX_JOBS": "3"},
+            capture_output=True,
+            text=True,
+            check=True,
+        )
+        matrix = json.loads(result.stdout.splitlines()[1])
+        paths = [path for item in matrix for path in item["paths"]]
+        assert sorted(paths) == sorted(path.relative_to(tmp_path).as_posix() for path in tests.glob("test_*.py"))
+        assert len(paths) == len(set(paths))
+        return matrix
+
+    assert len(discover()) == 1
+    (tests / "test_util.py").rename(tests / "test_renamed.py")
+    assert len(discover()) == 2
+    shutil.copy2(REPO_ROOT / "archivebox/tests/test_util.py", tests / "test_added.py")
+    assert len(discover()) == 3
+    (tests / "test_core_config.py").unlink()
+    assert len(discover()) == 2
+
+    source = REPO_ROOT / "archivebox/tests/test_util.py"
+    (tests / "test_runner.py").write_text("# ci-runner: hosted\n" + source.read_text())
+    assert next(item for item in discover() if "archivebox/tests/test_runner.py" in item["paths"])["ugnas"] is False
+    (tests / "test_runner.py").rename(tests / "test_runner_renamed.py")
+    assert next(item for item in discover() if "archivebox/tests/test_runner_renamed.py" in item["paths"])["ugnas"] is False
+    (tests / "test_runner_renamed.py").unlink()
+    assert all(item["ugnas"] for item in discover())
@@ -1,6 +1,7 @@
 import os
+import subprocess

-from archivebox.services.resource_admission import ResourceAdmission
+from archivebox.services.resource_admission import ResourceAdmission, disk_backed_swap_free_bytes


 def test_cpu_admission_respects_process_affinity():
@@ -10,6 +11,69 @@ def test_cpu_admission_respects_process_affinity():
    assert admission.snapshot_limit(1) == 1


+def test_snapshot_admission_stages_first_capture_then_uses_measured_headroom():
+    admission = ResourceAdmission()
+    assert admission.additional_snapshot_slots(3, active=0, observed_cost=None) == 1
+    assert admission.additional_snapshot_slots(3, active=1, observed_cost=None) == 0
+    # One observed peak costs 500 MB; only measured space is reserved for
+    # each additional in-flight snapshot, with no fixed minimum RAM.
+    assert admission.slots_for_headroom(3, active=0, observed_cost=500_000_000, available=1_100_000_000) == 2
+    assert admission.slots_for_headroom(3, active=1, observed_cost=500_000_000, available=400_000_000) == 0
+    assert admission.slots_for_headroom(3, active=1, observed_cost=500_000_000, available=900_000_000) == 0
+    assert admission.slots_for_headroom(3, active=1, observed_cost=500_000_000, available=1_100_000_000) == 1
+    assert admission.slots_for_headroom(3, active=0, observed_cost=500_000_000, available=0) == 1
+    # Headroom from memory_headroom() also includes disk-backed free swap.
+    assert admission.slots_for_headroom(3, active=0, observed_cost=500_000_000, available=2_000_000_000) == 3
+    assert admission.additional_snapshot_slots(3, active=0, observed_cost=0) == 1
+
+
+def test_memory_observation_tracks_a_real_child_process():
+    admission = ResourceAdmission()
+    before = admission.memory_headroom()
+    assert before is not None
+    child = subprocess.Popen(
+        [
+            "uv",
+            "run",
+            "python",
+            "-c",
+            "import sys; allocation = bytearray(64 * 1024 * 1024); allocation[0] = 1; print(len(allocation), flush=True); sys.stdin.read()",
+        ],
+        stdin=subprocess.PIPE,
+        stdout=subprocess.PIPE,
+        text=True,
+    )
+    try:
+        assert child.stdout is not None
+        assert child.stdout.readline().strip() == str(64 * 1024 * 1024)
+        during = admission.memory_headroom()
+        assert during is not None
+        assert during[0] > before[0] + 32 * 1024 * 1024
+    finally:
+        assert child.stdin is not None
+        child.stdin.close()
+        assert child.wait(timeout=10) == 0
+
+
+def test_disk_backed_swap_headroom_excludes_zram_but_keeps_backing_swap():
+    swaps = """Filename Type Size Used Priority
+/dev/zram0 partition 1000 100 100
+/swapfile file 2000 500 -2
+/dev/disk-swap partition 4000 1000 -3
+"""
+
+    assert disk_backed_swap_free_bytes(swaps, {"zram0", "254:0"}) == 4_500 * 1024
+
+
+def test_disk_backed_swap_headroom_excludes_zram_device_alias():
+    swaps = """Filename Type Size Used Priority
+/dev/block/254:0 partition 800 200 100
+/swapfile file 500 100 -2
+"""
+
+    assert disk_backed_swap_free_bytes(swaps, {"zram0", "254:0"}) == 400 * 1024
+
+
 def test_memory_stall_counter_recovers_without_resetting_monitor():
    # Real kernel PSI units: cumulative microseconds and ten-second average.
    # Decision samples exercise the policy, not simulated hooks/processes.
@@ -153,6 +153,54 @@ def test_search_backend_command_env_serializes_config_without_mutating_process_e
            os.environ["SEARCH_BACKEND_SONIC_HOST_NAME"] = old_env


+@pytest.mark.parametrize("activated_runtime", [True, False])
+def test_search_command_preserves_runtime_path_with_configured_path(tmp_path, activated_runtime):
+    import sqlite3
+    import subprocess
+    import sys
+
+    database = tmp_path / "search.sqlite3"
+    with sqlite3.connect(database) as connection:
+        connection.execute("CREATE VIRTUAL TABLE search_index USING fts5(snapshot_id UNINDEXED, text)")
+        connection.execute("INSERT INTO search_index VALUES (?, ?)", ("snapshot-runtime-path", "needle"))
+
+    configured_bin = tmp_path / "configured-bin"
+    configured_bin.mkdir()
+    env = os.environ.copy()
+    if not activated_runtime:
+        env["PATH"] = str(configured_bin)
+    result = subprocess.run(
+        [
+            sys.executable,
+            "-c",
+            """
+import os
+import sys
+from pathlib import Path
+from abx_dl.execution import iter_plugin_command
+from archivebox.plugins.discovery import get_plugin_catalog
+from archivebox.search.backends import search_backend_command_env
+
+data_dir, configured_bin = sys.argv[1:]
+env = search_backend_command_env(config={"PATH": configured_bin, "DATA_DIR": data_dir})
+assert env["PATH"].split(os.pathsep)[0] == configured_bin
+command = get_plugin_catalog().command("search_backend_sqlite", "search")
+assert command is not None
+for snapshot_id in iter_plugin_command(command, arguments={"query": "needle"}, env=env, cwd=Path(data_dir)):
+    print(snapshot_id)
+""",
+            str(tmp_path),
+            str(configured_bin),
+        ],
+        env=env,
+        capture_output=True,
+        text=True,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr
+    assert result.stdout.splitlines() == ["snapshot-runtime-path"]
+
+
 def test_search_mode_options_use_canonical_backend_names():
    from archivebox.search.config import get_search_mode_options

@@ -546,7 +594,11 @@ class TestPublicIndexSearch:
        assert b"Public Example Website" in response.content
        assert b"No snapshots found." not in response.content

-    def test_public_index_shows_exact_total_count_and_page_count_for_100_plus_snapshots(self, client, crawl, public_snapshot):
+    @pytest.mark.django_db(transaction=True)
+    def test_public_index_shows_cached_total_count_and_page_count_for_100_plus_snapshots(self, client, crawl, public_snapshot):
+        import time
+
+        from django.core.cache import cache
        from archivebox.core.models import Snapshot

        base_time = public_snapshot.bookmarked_at + timedelta(seconds=1)
@@ -580,11 +632,21 @@ class TestPublicIndexSearch:
            ],
        )

+        key = "progress-collection:public"
+        cache.delete_many([key, f"{key}:refresh"])
+        initial_response = client.get("/public/", HTTP_HOST=WEB_HOST)
+        assert initial_response.status_code == 200
+        assert initial_response.context["public_snapshot_count"] is None
+        assert b"last &raquo;" not in initial_response.content
+        deadline = time.monotonic() + 5
+        while cache.get(key) is None and time.monotonic() < deadline:
+            time.sleep(0.01)
+        assert cache.get(key)["snapshots"] == 125
        response = client.get("/public/", HTTP_HOST=WEB_HOST)

        assert response.status_code == 200
-        assert response.context["paginator"].count == 125
-        assert response.context["paginator"].num_pages == 3
+        assert response.context["public_snapshot_count"] == 125
+        assert response.context["public_page_count"] == 3
        assert response.context["page_obj"].has_next() is True
        assert len(response.context["object_list"]) == 50
        content = response.content.decode()
@@ -1281,7 +1343,7 @@ class TestSearchBackendsE2E:
                        surface_name,
                        search_mode,
                        first_positive_elapsed,
-                        counts[:10],
+                        count_events[:10],
                    )
                    assert total_elapsed < 2.0, (surface_name, search_mode, total_elapsed, counts[-10:])

@@ -954,6 +954,17 @@ def test_server_security_modes_in_chrome(
 def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archive: Path, browser_runtime, tmp_path: Path) -> None:
    from archivebox.core.routes_util import get_snapshot_subdomain

+    node_binary = Path(browser_runtime["node_binary"])
+    node_target = node_binary.resolve(strict=True)
+    node_version = subprocess.check_output([str(node_binary), "--version"], text=True).strip()
+
+    def assert_browser_runtime_unchanged(stage):
+        actual_target = node_binary.resolve(strict=True)
+        actual_version = subprocess.check_output([str(node_binary), "--version"], text=True).strip()
+        assert (actual_target, actual_version) == (node_target, node_version), (
+            f"{stage} changed the browser runtime: {node_target} ({node_version}) -> {actual_target} ({actual_version})"
+        )
+
    url = "https://example.com"
    port = get_free_port()
    env = cli_env(
@@ -992,6 +1003,7 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
        assert install_result.returncode == 0, (
            f"archivebox install archivewebpage failed:\nSTDOUT:\n{install_result.stdout}\nSTDERR:\n{install_result.stderr}"
        )
+        assert_browser_runtime_unchanged("archivebox install archivewebpage")

        process = start_daemon_server(
            initialized_archive,
@@ -1001,6 +1013,7 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
            log_name="archivewebpage_server.log",
        )
        get_http_response(port, host=f"archivebox.localhost:{port}", path="/")
+        assert_browser_runtime_unchanged("archivebox server startup")
        command_result = run_archivebox_cmd(
            ["add", "--depth=0", "--max-urls=1", "--plugins=archivewebpage", url],
            cwd=initialized_archive,
@@ -1008,6 +1021,7 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
            timeout=120,
        )
        assert command_result.returncode == 0, f"archivebox add failed:\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
+        assert_browser_runtime_unchanged("archivebox add --plugins=archivewebpage")

        capture = _get_archivewebpage_capture(initialized_archive, url)
        snapshot_host = f"{get_snapshot_subdomain(capture['snapshot_id'])}.archivebox.localhost:{port}"
@@ -83,6 +83,96 @@ def test_foreground_shutdown_can_request_cooperative_shutdown_without_raising():
    assert process.returncode == 130, (stdout, stderr)


+def test_cooperative_shutdown_during_child_poll_keeps_child_reapable():
+    """Interrupt real child polling at different points without stranding reaping.
+
+    Immediate KeyboardInterrupt stranded Popen's wait lock in real CI, leaving
+    a zombie child that wait() could not reap. Exercise the cooperative policy
+    used by the server with the same external signals and child cleanup.
+    """
+    process = _start_signal_process(
+        r"""
+        import os
+        import signal
+        import subprocess
+        import sys
+        import traceback
+        from collections import Counter
+
+        from archivebox.core.shutdown_util import (
+            foreground_shutdown_signals,
+            raise_if_shutdown_requested,
+            wait_popen_and_kill_children,
+        )
+
+        print("READY", flush=True)
+        interruption_sites = Counter()
+        for attempt in range(512):
+            child = subprocess.Popen(
+                [sys.executable, "-c", "import time; time.sleep(60)"],
+                stdin=subprocess.DEVNULL,
+                stdout=subprocess.DEVNULL,
+                stderr=subprocess.DEVNULL,
+            )
+            interruption_site = None
+            try:
+                with foreground_shutdown_signals(first_signal_message=None, raise_on_first_signal=False) as state:
+                    try:
+                        os.write(sys.stdout.fileno(), b"POLLING\n")
+                        while True:
+                            raise_if_shutdown_requested()
+                            assert child.poll() is None
+                    except KeyboardInterrupt as interruption:
+                        assert state.signal_name == "SIGINT"
+                        frame = traceback.extract_tb(interruption.__traceback__)[-1]
+                        assert frame.name == "raise_if_shutdown_requested", frame
+                        interruption_site = f"{frame.name}:{frame.lineno}"
+                        interruption_sites[interruption_site] += 1
+                # The child deliberately remains alive until this unchanged
+                # kill-and-reap path asks it to stop.
+                wait_popen_and_kill_children(child, [], timeout=0.001, kill_timeout=1.0)
+                assert child.returncode == -signal.SIGKILL, (attempt, child.returncode)
+            except BaseException as error:
+                error.add_note(f"Interrupted child polling iteration: {attempt}, at {interruption_site}")
+                raise
+            finally:
+                # Cleanup must also work if the assertion exposes a broken
+                # Popen wait; preserve that failure while reaping the real PID.
+                if child.returncode is None:
+                    try:
+                        os.kill(child.pid, signal.SIGKILL)
+                    except ProcessLookupError:
+                        pass
+                    try:
+                        os.waitpid(child.pid, 0)
+                    except ChildProcessError:
+                        pass
+        print(f"Interrupt locations: {dict(interruption_sites)}", file=sys.stderr, flush=True)
+        print("REAPED:512", flush=True)
+        """,
+    )
+
+    try:
+        # An external sender does not contend for the polling interpreter's
+        # GIL. A Timer thread can bias delivery toward waitpid's GIL release,
+        # missing other interruption points reached by real CLI Ctrl+C.
+        assert process.stdout is not None
+        sent = 0
+        while sent < 512 and process.stdout.readline() == "POLLING\n":
+            process.send_signal(signal.SIGINT)
+            sent += 1
+        stdout, stderr = process.communicate(timeout=30)
+    finally:
+        if process.poll() is None:
+            process.kill()
+            process.wait(timeout=5)
+
+    assert process.returncode == 0, (stdout, stderr)
+    assert sent == 512, (sent, stdout, stderr)
+    assert stdout == "REAPED:512\n", (stdout, stderr)
+    print(stderr)
+
+
 def test_daemon_runner_signal_exit_is_unexpected_for_supervisor():
    process = _start_signal_process(
        """
@@ -311,6 +311,75 @@ def test_responses_card_uses_relative_preview_urls_in_static_export(snapshot):
    assert "localhost" not in html


+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("status", ["queued", "started", "paused"])
+def test_empty_snapshot_keeps_live_progress_until_files_are_opened(snapshot, live_server, status):
+    from urllib.parse import urlsplit
+
+    from playwright.sync_api import sync_playwright
+
+    from archivebox.machine.models import Machine
+
+    port = urlsplit(live_server.url).port
+    machine = Machine.current()
+    machine.config = {**machine.config, "BASE_URL": f"http://archivebox.localhost:{port}"}
+    machine.save(update_fields=["config"])
+    snapshot.permissions = "public"
+    snapshot.status = status
+    snapshot.save(update_fields=["permissions", "status"])
+    Path(snapshot.output_dir).mkdir(parents=True, exist_ok=True)
+    with sync_playwright() as playwright:
+        browser = playwright.chromium.launch(args=["--host-resolver-rules=MAP *.archivebox.localhost 127.0.0.1"])
+        page = browser.new_page()
+        page.goto(f"http://web.archivebox.localhost:{port}{snapshot.get_absolute_url()}")
+        assert page.locator("#progress-monitor").is_visible()
+        assert page.locator("#progress-monitor .progress-content").is_visible()
+        assert page.locator("#snapshot-empty-state").is_visible()
+        assert page.locator("#main-frame").count() == 0
+
+        page.get_by_role("button", name=re.compile("^Other files,")).click()
+        page.locator("#main-frame").wait_for()
+        assert page.locator("#progress-monitor").count() == 0
+        browser.close()
+
+
+@pytest.mark.django_db(transaction=True)
+def test_opening_files_from_queued_snapshot_stops_detached_progress_monitor(snapshot, live_server):
+    from urllib.parse import urlsplit
+
+    from playwright.sync_api import sync_playwright
+
+    from archivebox.machine.models import Machine
+
+    port = urlsplit(live_server.url).port
+    machine = Machine.current()
+    machine.config = {**machine.config, "BASE_URL": f"http://archivebox.localhost:{port}"}
+    machine.save(update_fields=["config"])
+    snapshot.permissions = "public"
+    snapshot.status = "queued"
+    snapshot.save(update_fields=["permissions", "status"])
+    Path(snapshot.output_dir).mkdir(parents=True, exist_ok=True)
+    errors = []
+    progress_requests = []
+    with sync_playwright() as playwright:
+        browser = playwright.chromium.launch(args=["--host-resolver-rules=MAP *.archivebox.localhost 127.0.0.1"])
+        page = browser.new_page()
+        page.on("pageerror", lambda error: errors.append(str(error)))
+        page.on("console", lambda message: errors.append(message.text) if message.type == "error" else None)
+        page.on("request", lambda request: progress_requests.append(request.url) if "progress.json" in request.url else None)
+        page.goto(f"http://web.archivebox.localhost:{port}{snapshot.get_absolute_url()}/index.html#all")
+        page.locator("#main-frame").wait_for()
+        assert page.locator("#progress-monitor").count() == 0
+        # Let the real one-second polling interval and any in-flight response
+        # complete: a detached panel must neither throw nor keep making requests.
+        page.wait_for_timeout(2200)
+        count = len(progress_requests)
+        page.wait_for_timeout(2200)
+        assert len(progress_requests) == count
+        assert errors == []
+        browser.close()
+
+
@pytest.mark.django_db(transaction=True)
 def test_stack_cover_and_expanded_card_load_same_document(snapshot, live_server):
    from urllib.parse import urlsplit
@@ -0,0 +1,207 @@
+#!/usr/bin/env python3
+"""Takeover utility tests and live command handoff flows."""
+
+import os
+import json
+import signal
+import subprocess
+import sys
+import pty
+import select
+import termios
+import time
+from pathlib import Path
+
+import pytest
+import psutil
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    assert_no_processes_for_data_dir,
+    get_free_port,
+    kill_processes_for_data_dir,
+    cli_env,
+    pid_is_alive,
+    run_archivebox_cmd,
+    start_archivebox_server,
+    stop_archivebox_process,
+    get_http_response,
+)
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+def _resolve_sonic_env(env: dict[str, str]) -> dict[str, str]:
+    from abx_plugins import get_plugins_dir
+
+    config = Path(get_plugins_dir()) / "search_backend_sonic" / "config.json"
+    result = subprocess.run(
+        [
+            str(Path(sys.executable).with_name("abxpkg")),
+            "env",
+            "--install",
+            "--json",
+            f"--lib={env['ABXPKG_LIB_DIR']}",
+            f"--deps-from={config}:required_binaries",
+        ],
+        capture_output=True,
+        text=True,
+        env=env,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    payload = {str(key): str(value) for key, value in json.loads(result.stdout).items()}
+    assert Path(payload["SONIC_BINARY"]).is_file()
+    return payload
+
+
+def _archive_pages_for_sqlite_reindexing(data_dir: Path, env: dict[str, str], root_url: str) -> None:
+    add_env = dict(env)
+    add_env["SEARCH_BACKEND_ENGINE"] = "ripgrep"
+    _cmd_result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=2",
+            "--max-urls=20",
+            "--crawl-max-size=50mb",
+            "--plugins=wget,parse_html_urls",
+            root_url,
+        ],
+        cwd=data_dir,
+        env=add_env,
+        timeout=240,
+    )
+    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert returncode == 0, stderr or stdout
+
+    with use_archivebox_db(data_dir):
+        assert Snapshot.objects.filter(status=Snapshot.StatusChoices.SEALED).count() >= 1
+        assert not ArchiveResult.objects.filter(plugin="search_backend_sqlite").exists()
+
+
+def _start_archivebox_shell(tmp_path: Path):
+    process = run_archivebox_cmd(
+        ["manage", "shell"],
+        cwd=tmp_path,
+        env=cli_env(live=True),
+        stdin=subprocess.PIPE,
+        stdout=subprocess.DEVNULL,
+        stderr=subprocess.DEVNULL,
+        capture_output=False,
+        start_new_session=True,
+        wait=False,
+    )
+    assert process.pid is not None
+    assert process.stdin is not None
+    assert pid_is_alive(process.pid)
+    return process
+
+
+def _stop_archivebox_shells(processes) -> None:
+    for process in processes:
+        if process.stdin is not None and not process.stdin.closed:
+            process.stdin.close()
+    for process in processes:
+        process.wait(timeout=20)
+        assert not pid_is_alive(process.pid)
+
+
+@pytest.mark.parametrize("interrupt_before_takeover", [False, True], ids=["running", "prompt"])
+@pytest.mark.parametrize("after_takeover", ["resume", "cancel-standby"])
+def test_interactive_add_takeover(initialized_archive, interrupt_before_takeover, after_takeover):
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    add = None
+    server = None
+    port = get_free_port()
+    env = cli_env(
+        SEARCH_BACKEND_ENGINE="ripgrep",
+        SEARCH_BACKEND_SONIC_ENABLED="False",
+        CHROME_DELAY_AFTER_LOAD="60",
+        CHROME_TIMEOUT="120",
+        CHROME_HEADLESS="True",
+    )
+    installed = run_archivebox_cmd(["install", "chrome"], cwd=initialized_archive, env=env, timeout=600)
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook_for(parent_pid):
+        with use_archivebox_db(initialized_archive):
+            for hook in Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ):
+                if hook.is_running and parent_pid in {parent.pid for parent in psutil.Process(hook.pid).parents()}:
+                    return hook
+        return None
+
+    try:
+        add = run_archivebox_cmd(
+            ["add", "--plugins=chrome", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook_for(add.pid) is not None)
+        old_hook = active_hook_for(add.pid)
+        assert old_hook is not None
+        if interrupt_before_takeover:
+            add.send_signal(signal.SIGINT)
+            read_until(lambda: b"Choice [skip]:" in output)
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            assert not pid_is_alive(old_hook.pid)
+
+        server = start_archivebox_server(initialized_archive, port=port, log_name="interrupt-takeover-server.log", env=env)
+        read_until(lambda: b"A newer archivebox process took over" in output)
+        assert add.poll() is None, output.decode(errors="replace")
+        assert termios.tcgetattr(slave)[3] & termios.ICANON
+        assert not pid_is_alive(old_hook.pid)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        if after_takeover == "cancel-standby":
+            add.send_signal(signal.SIGINT)
+            read_until(lambda: add.poll() is not None)
+            assert add.returncode == 130, output.decode(errors="replace")
+            assert pid_is_alive(server.pid)
+            assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        else:
+            read_until(lambda: active_hook_for(server.pid) is not None)
+            stop_archivebox_process(server, signal.SIGTERM)
+            server = None
+            read_until(lambda: active_hook_for(add.pid) is not None)
+            resumed_hook = active_hook_for(add.pid)
+            assert resumed_hook is not None and resumed_hook.pid != old_hook.pid
+            prompt_offset = len(output)
+            add.send_signal(signal.SIGINT)
+            read_until(lambda: b"Choice [skip]:" in output[prompt_offset:])
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            os.write(master, b"\r")
+            read_until(lambda: add.poll() is not None)
+            assert add.returncode == 0, output.decode(errors="replace")
+            with use_archivebox_db(initialized_archive):
+                assert Crawl.objects.get().status == Crawl.StatusChoices.SEALED
+        assert b"Traceback" not in output
+    finally:
+        for process in (server, add):
+            if process is not None and process.poll() is None:
+                stop_archivebox_process(process, signal.SIGTERM)
+        os.close(slave)
+        os.close(master)
+        kill_processes_for_data_dir(initialized_archive)
+        assert_no_processes_for_data_dir(initialized_archive, timeout=12)
@@ -0,0 +1,369 @@
+#!/usr/bin/env python3
+"""Takeover utility tests and live command handoff flows."""
+
+import os
+import re
+import shlex
+import shutil
+import signal
+import subprocess
+import sys
+import pty
+import select
+import termios
+import time
+from pathlib import Path
+
+import pytest
+
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    assert_no_processes_for_data_dir,
+    get_free_port,
+    kill_processes_for_data_dir,
+    cli_env,
+    pid_is_alive,
+    run_archivebox_cmd,
+    start_archivebox_server,
+    stop_archivebox_process,
+    get_http_response,
+    wait_for_snapshot_capture,
+)
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_takeover_util_1 import (
+    pytestmark as pytestmark,
+    _resolve_sonic_env as _resolve_sonic_env,
+    _archive_pages_for_sqlite_reindexing as _archive_pages_for_sqlite_reindexing,
+    _start_archivebox_shell as _start_archivebox_shell,
+    _stop_archivebox_shells as _stop_archivebox_shells,
+)
+
+
+@pytest.mark.parametrize("force_abort", [None, "typed", "signal"], ids=["graceful", "force-after-choice", "force-after-ctrl-c"])
+def test_interactive_add_borrowing_server_supervisor_can_abort(initialized_archive, force_abort):
+    """A server-owned supervisor must not make an interactive add worker noninteractive."""
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    server = None
+    add = None
+    port = get_free_port()
+    env = cli_env(
+        SEARCH_BACKEND_ENGINE="ripgrep",
+        SEARCH_BACKEND_SONIC_ENABLED="False",
+        CHROME_DELAY_AFTER_LOAD="60",
+        CHROME_TIMEOUT="120",
+        CHROME_HEADLESS="True",
+    )
+    installed = run_archivebox_cmd(["install", "chrome"], cwd=initialized_archive, env=env, timeout=600)
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            for hook in Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ):
+                if hook.is_running:
+                    return hook
+        return None
+
+    try:
+        server = start_archivebox_server(initialized_archive, port=port, log_name="add-borrows-server.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        add = run_archivebox_cmd(
+            ["add", "--plugins=chrome", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        old_hook = active_hook()
+        assert old_hook is not None
+        add.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        assert add.poll() is None, output.decode(errors="replace")
+        assert not pid_is_alive(old_hook.pid)
+
+        if force_abort:
+            # Both abort choices must become forceable before the worker has
+            # time to write its marker or finish the normal hook grace phase.
+            if force_abort == "typed":
+                os.write(master, b"a")
+            else:
+                add.send_signal(signal.SIGINT)
+            read_until(lambda: termios.tcgetattr(slave)[3] & termios.ICANON)
+            forced_at = time.monotonic()
+            add.send_signal(signal.SIGINT)
+        else:
+            add.send_signal(signal.SIGINT)
+        read_until(lambda: add.poll() is not None)
+        if force_abort:
+            assert time.monotonic() - forced_at < 2.0, output.decode(errors="replace")
+            assert b"Forcing aborted crawl to exit now" in output
+        assert add.returncode == 130, output.decode(errors="replace")
+        assert pid_is_alive(server.pid)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        assert b"Traceback" not in output
+    finally:
+        for process in (add, server):
+            if process is not None and process.poll() is None:
+                stop_archivebox_process(process, signal.SIGTERM)
+        os.close(slave)
+        os.close(master)
+        kill_processes_for_data_dir(initialized_archive)
+        assert_no_processes_for_data_dir(initialized_archive, timeout=12)
+
+
+def test_explicit_sonic_binary_is_used_by_real_search_worker(initialized_archive):
+    sonic_port = get_free_port()
+    lib_dir = initialized_archive / "lib"
+    env = cli_env(
+        live=True,
+        ABXPKG_LIB_DIR=str(lib_dir),
+        PLUGINS="search_backend_sonic",
+        SEARCH_BACKEND_ENGINE="sonic",
+        SEARCH_BACKEND_SONIC_PORT=str(sonic_port),
+    )
+    env.update(_resolve_sonic_env(env))
+    env["PATH"] = os.pathsep.join((str(Path(sys.executable).parent), env["PATH"]))
+    installed_binary = Path(env["SONIC_BINARY"]).resolve(strict=True)
+    managed_sonic_dir = lib_dir / "bash" / "bin"
+    managed_sonic_dir.mkdir(parents=True, exist_ok=True)
+    managed_binary = managed_sonic_dir / "sonic"
+    if installed_binary != managed_binary:
+        shutil.copy2(installed_binary, managed_binary)
+    config_result = run_archivebox_cmd(
+        ["config", "--set", f"SONIC_BINARY={managed_binary}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert config_result.returncode == 0, config_result.stderr or config_result.stdout
+    pinned_sonic_dir = initialized_archive / "pinned-sonic"
+    pinned_sonic_dir.mkdir()
+    sonic_binary = pinned_sonic_dir / "sonic"
+    shutil.copy2(installed_binary, sonic_binary)
+    env["SONIC_BINARY"] = str(sonic_binary)
+
+    result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    config_match = re.search(r"Using supervisord config file: (\S+)", result.stderr)
+    assert config_match, result.stderr
+    worker_config = Path(config_match.group(1)).parent / "workers" / "worker_sonic.conf"
+    worker_text = worker_config.read_text(encoding="utf-8")
+    command = next(line.partition("=")[2] for line in worker_text.splitlines() if line.startswith("command="))
+    assert shlex.split(command)[0] == str(sonic_binary), worker_text
+    from archivebox.workers.supervisord_util import _sonic_worker_bind_target
+
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": command}) == ("127.0.0.1", sonic_port)
+    renamed_binary = initialized_archive / "renamed-sonic-binary"
+    shutil.copy2(sonic_binary, renamed_binary)
+    renamed_command = shlex.join([str(renamed_binary), *shlex.split(command)[1:]])
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": renamed_command}) == ("127.0.0.1", sonic_port)
+
+    relative_binary = pinned_sonic_dir / "sonic-alt"
+    shutil.copy2(sonic_binary, relative_binary)
+    relative_env = {
+        **env,
+        "SONIC_BINARY": relative_binary.name,
+        "PATH": os.pathsep.join((str(pinned_sonic_dir), env["PATH"])),
+        "SEARCH_BACKEND_SONIC_PORT": str(get_free_port()),
+    }
+    relative_result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=relative_env,
+        timeout=60,
+    )
+    assert relative_result.returncode == 0, relative_result.stderr or relative_result.stdout
+    relative_worker_text = worker_config.read_text(encoding="utf-8")
+    relative_command = next(line.partition("=")[2] for line in relative_worker_text.splitlines() if line.startswith("command="))
+    assert Path(shlex.split(relative_command)[0]).name == relative_binary.name, relative_worker_text
+
+
+@pytest.mark.timeout(300)
+def test_behavior_foreground_add_keeps_existing_server_http_visible(tmp_path, initialized_archive, recursive_test_site):
+
+    port = get_free_port()
+    env = cli_env(live=True, server=True, port=port, SEARCH_BACKEND_ENGINE="ripgrep")
+    server = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="behavior-server-add.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        add = run_archivebox_cmd(
+            [
+                "add",
+                "--depth=2",
+                "--max-urls=10",
+                "--plugins=wget,parse_html_urls",
+                recursive_test_site["root_url"],
+            ],
+            cwd=tmp_path,
+            env=env,
+            timeout=180,
+        )
+
+        assert add.returncode == 0, add.stderr or add.stdout
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        captured_text = wait_for_snapshot_capture(tmp_path, recursive_test_site["root_url"], timeout=120)
+        assert "Root" in captured_text
+    finally:
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(300)
+def test_behavior_background_add_returns_and_server_archives_visible_url(tmp_path, initialized_archive, recursive_test_site):
+
+    port = get_free_port()
+    env = cli_env(live=True, server=True, port=port, SEARCH_BACKEND_ENGINE="ripgrep")
+    server = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="behavior-server-bg-add.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        add = run_archivebox_cmd(
+            [
+                "add",
+                "--bg",
+                "--depth=0",
+                "--plugins=wget",
+                recursive_test_site["root_url"],
+            ],
+            cwd=tmp_path,
+            env=env,
+            timeout=60,
+        )
+
+        assert add.returncode == 0, add.stderr or add.stdout
+        assert "background runner will process" in add.stdout
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        captured_text = wait_for_snapshot_capture(tmp_path, recursive_test_site["root_url"], timeout=180)
+        assert "Root" in captured_text
+    finally:
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+def test_foreground_runner_owner_prefers_newer_update_over_server(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import foreground_runner_owner, runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for process_type in (Process.TypeChoices.SERVER, Process.TypeChoices.UPDATE):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=process_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+
+        runtime_owner = runtime_stack_owner(data_dir=tmp_path)
+        runner_owner = foreground_runner_owner(data_dir=tmp_path)
+
+        assert runtime_owner is not None
+        assert runtime_owner.process_type == Process.TypeChoices.SERVER
+        assert runner_owner is not None
+        assert runner_owner.process_type == Process.TypeChoices.UPDATE
+        assert runner_owner.pid == procs[-1].pid
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_runtime_stack_owner_ignores_supervised_orphan_runner(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import RUNNER_ACTIVE_WORKER_TYPE, runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for _ in range(2):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+
+        supervisor_row = Process.objects.create(
+            machine=Machine.current(),
+            process_type=Process.TypeChoices.SUPERVISORD,
+            worker_type="supervisord",
+            pwd=str(tmp_path),
+            cmd=[],
+            pid=procs[0].pid,
+            status=Process.StatusChoices.RUNNING,
+        )
+        Process.objects.create(
+            machine=Machine.current(),
+            parent=supervisor_row,
+            process_type=Process.TypeChoices.ORCHESTRATOR,
+            worker_type=RUNNER_ACTIVE_WORKER_TYPE,
+            pwd=str(tmp_path),
+            cmd=[],
+            pid=procs[1].pid,
+            status=Process.StatusChoices.RUNNING,
+        )
+
+        assert runtime_stack_owner(data_dir=tmp_path) is None
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_foreign_machine_runner_only_warns(tmp_path, initialized_archive, capsys):
+    from archivebox.core.takeover_util import RUNNER_ACTIVE_WORKER_TYPE, live_runner_processes
+    from archivebox.machine.models import Machine
+
+    foreign_machine = Machine.objects.create(
+        guid="foreign-machine",
+        hostname="foreign-host",
+        hw_manufacturer="Test",
+        hw_product="Test",
+        hw_uuid="foreign-hardware",
+        os_arch="x86_64",
+        os_family="linux",
+        os_platform="linux",
+        os_release="test",
+        os_kernel="test",
+    )
+    foreign_runner = Process.objects.create(
+        machine=foreign_machine,
+        process_type=Process.TypeChoices.ORCHESTRATOR,
+        worker_type=RUNNER_ACTIVE_WORKER_TYPE,
+        pwd=str(tmp_path),
+        pid=1,
+        status=Process.StatusChoices.RUNNING,
+    )
+
+    assert live_runner_processes(data_dir=tmp_path) == []
+    foreign_runner.refresh_from_db()
+    assert foreign_runner.status == Process.StatusChoices.RUNNING
+    assert "Multiple orchestrators sharing a single collection is not officially supported" in capsys.readouterr().err
@@ -0,0 +1,400 @@
+#!/usr/bin/env python3
+"""Takeover utility tests and live command handoff flows."""
+
+import signal
+import subprocess
+
+import pytest
+import psutil
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    assert_no_processes_for_data_dir,
+    get_free_port,
+    kill_processes_for_data_dir,
+    cli_env,
+    pid_is_alive,
+    run_archivebox_cmd,
+    start_archivebox_server,
+    stop_archivebox_process,
+    supervisor_pid_from_log,
+    get_http_response,
+    wait_for_log_count,
+    wait_for_pid_to_disappear,
+    wait_for_snapshot_capture,
+    worker_pid_from_log,
+)
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_takeover_util_1 import (
+    pytestmark as pytestmark,
+    _resolve_sonic_env as _resolve_sonic_env,
+    _archive_pages_for_sqlite_reindexing as _archive_pages_for_sqlite_reindexing,
+    _start_archivebox_shell as _start_archivebox_shell,
+    _stop_archivebox_shells as _stop_archivebox_shells,
+)
+
+
+@pytest.mark.timeout(300)
+def test_behavior_daemonized_server_restarts_cleanly_after_forced_stop(tmp_path, initialized_archive):
+
+    port = get_free_port()
+    env = cli_env(live=True, server=True, port=port)
+    try:
+        first = start_archivebox_server(tmp_path, port=port, env=env, daemonize=True)
+        assert first.returncode == 0, first.stderr or first.stdout
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+        second = start_archivebox_server(tmp_path, port=port, env=env, daemonize=True)
+        assert second.returncode == 0, second.stderr or second.stdout
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+    finally:
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(240)
+def test_live_second_server_takes_over_existing_server_process(tmp_path, initialized_archive):
+
+    env = cli_env(live=True)
+    port = get_free_port()
+    first = None
+    second = None
+    try:
+        first = start_archivebox_server(tmp_path, port=port, log_name="server-first.log", env=env)
+        first_log = first.log_path
+        second = start_archivebox_server(tmp_path, port=port, log_name="server-second.log", env=env)
+        second_log = second.log_path
+
+        assert pid_is_alive(first.pid)
+        first_text = first_log.read_text(encoding="utf-8", errors="replace")
+        second_text = second_log.read_text(encoding="utf-8", errors="replace")
+        assert "A newer archivebox process took over the orchestrator, server" in first_text
+        assert "Starting orchestrator, server" in second_text
+
+        result = run_archivebox_cmd(
+            ["status"],
+            cwd=tmp_path,
+            env=env,
+            timeout=60,
+        )
+        assert result.returncode == 0, result.stderr or result.stdout
+
+        first_resumes = first_log.read_text(encoding="utf-8", errors="replace").count("Other newer archivebox process")
+        stop_archivebox_process(second, signal.SIGTERM)
+        second = None
+        wait_for_log_count(first_log, "Other newer archivebox process", first_resumes + 1, timeout=35)
+        assert pid_is_alive(first.pid)
+    finally:
+        if second is not None:
+            stop_archivebox_process(second, signal.SIGTERM)
+        if first is not None:
+            stop_archivebox_process(first, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(360)
+def test_live_server_keeps_http_runtime_while_update_runs_real_sqlite_indexer(tmp_path, initialized_archive, recursive_test_site):
+    env = cli_env(
+        live=True,
+        PLUGINS="wget,parse_html_urls,search_backend_sqlite,search_backend_sonic",
+        SEARCH_BACKEND_ENGINE="sqlite",
+        SEARCH_BACKEND_SONIC_PORT=str(get_free_port()),
+    )
+    env.update(_resolve_sonic_env(env))
+    _archive_pages_for_sqlite_reindexing(tmp_path, env, recursive_test_site["root_url"])
+
+    port = get_free_port()
+    server = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="server-real-sqlite-update.log", env=env)
+        server_log = server.log_path
+        supervisor_pid_before = supervisor_pid_from_log(server_log)
+        daphne_pid_before = worker_pid_from_log(server_log, "worker_daphne")
+        runner_pid_before = worker_pid_from_log(server_log, "worker_runner")
+        sonic_pid_before = worker_pid_from_log(server_log, "worker_sonic")
+        supervisord_log = tmp_path / "logs" / "supervisord.log"
+        runner_spawn_text = "spawned: 'worker_runner' with pid"
+        runner_spawn_count = supervisord_log.read_text(encoding="utf-8", errors="replace").count(runner_spawn_text)
+
+        result = run_archivebox_cmd(
+            ["update", "--index-only", "--batch-size=1"],
+            cwd=tmp_path,
+            env=env,
+            timeout=180,
+        )
+
+        assert result.returncode == 0, result.stderr or result.stdout
+        assert pid_is_alive(server.pid)
+        assert supervisor_pid_from_log(server_log) == supervisor_pid_before
+        assert pid_is_alive(daphne_pid_before)
+        assert pid_is_alive(sonic_pid_before)
+        assert worker_pid_from_log(server_log, "worker_daphne") == daphne_pid_before
+        assert worker_pid_from_log(server_log, "worker_sonic") == sonic_pid_before
+        assert "A newer archivebox process took over the orchestrator, server" not in server_log.read_text(
+            encoding="utf-8",
+            errors="replace",
+        )
+        supervisord_text = supervisord_log.read_text(encoding="utf-8", errors="replace")
+        assert supervisord_text.count(runner_spawn_text) == runner_spawn_count
+        assert worker_pid_from_log(server_log, "worker_runner") == runner_pid_before
+        assert pid_is_alive(runner_pid_before)
+
+        with use_archivebox_db(tmp_path):
+            indexed_results = list(
+                ArchiveResult.objects.filter(plugin="search_backend_sqlite").values_list("status", flat=True),
+            )
+        assert indexed_results
+        assert all(status in ArchiveResult.FINAL_STATES for status in indexed_results)
+
+        stop_archivebox_process(server, signal.SIGTERM)
+        server = None
+        wait_for_pid_to_disappear(daphne_pid_before, timeout=20)
+        wait_for_pid_to_disappear(sonic_pid_before, timeout=20)
+        wait_for_pid_to_disappear(runner_pid_before, timeout=20)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+    finally:
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(420)
+def test_live_repeated_server_startups_take_over_cleanly(tmp_path, initialized_archive):
+
+    env = cli_env(live=True)
+    port = get_free_port()
+    servers: list[subprocess.Popen[str]] = []
+    server_pids: list[int] = []
+    daphne_pids: list[int] = []
+    runner_pids: list[int] = []
+    try:
+        for index in range(5):
+            server = start_archivebox_server(tmp_path, port=port, log_name=f"server-chaos-{index}.log", env=env)
+            log_path = server.log_path
+            servers.append(server)
+            server_pids.append(server.pid)
+            daphne_pids.append(worker_pid_from_log(log_path, "worker_daphne"))
+            runner_pids.append(worker_pid_from_log(log_path, "worker_runner"))
+
+            if index > 0:
+                previous_server = servers[index - 1]
+                previous_log = (tmp_path / f"server-chaos-{index - 1}.log").read_text(encoding="utf-8", errors="replace")
+                current_log = log_path.read_text(encoding="utf-8", errors="replace")
+                assert pid_is_alive(previous_server.pid)
+                assert pid_is_alive(server_pids[index - 1])
+                assert "A newer archivebox process took over the orchestrator, server" in previous_log
+                assert "Starting orchestrator, server" in current_log
+                wait_for_pid_to_disappear(daphne_pids[index - 1], timeout=15)
+                wait_for_pid_to_disappear(runner_pids[index - 1], timeout=15)
+
+            result = run_archivebox_cmd(
+                ["status"],
+                cwd=tmp_path,
+                env=env,
+                timeout=60,
+            )
+            assert result.returncode == 0, result.stderr or result.stdout
+
+        assert pid_is_alive(servers[-1].pid)
+        assert all(pid_is_alive(server.pid) for server in servers)
+        # Inspect the actual server worker rather than requiring a host lsof
+        # installation. Previous workers must be gone, and the newest worker
+        # must own exactly one listening socket on our assigned port.
+        assert all(not pid_is_alive(pid) for pid in daphne_pids[:-1])
+        listeners = [
+            connection
+            for connection in psutil.Process(daphne_pids[-1]).net_connections(kind="tcp")
+            if connection.status == psutil.CONN_LISTEN and connection.laddr.port == port
+        ]
+        assert len(listeners) == 1, listeners
+
+        previous_log_path = tmp_path / "server-chaos-3.log"
+        previous_takeovers = previous_log_path.read_text(encoding="utf-8", errors="replace").count(
+            "Other newer archivebox process",
+        )
+        stop_archivebox_process(servers[-1], signal.SIGTERM)
+        wait_for_log_count(previous_log_path, "Other newer archivebox process", previous_takeovers + 1, timeout=35)
+        assert pid_is_alive(servers[3].pid)
+    finally:
+        for server in reversed(servers):
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(420)
+def test_live_background_add_survives_server_exit_and_foreground_run_reclaims(tmp_path, initialized_archive, recursive_test_site):
+    env = cli_env(live=True, SEARCH_BACKEND_ENGINE="ripgrep")
+    port = get_free_port()
+    server = None
+    server2 = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="server-add-owner-1.log", env=env)
+        server_log = server.log_path
+        supervisor_pid_before = supervisor_pid_from_log(server_log)
+
+        result = run_archivebox_cmd(
+            ["update", "--index-only", "--batch-size=10"],
+            cwd=tmp_path,
+            env=env,
+            timeout=90,
+        )
+        assert result.returncode == 0, result.stderr or result.stdout
+        assert pid_is_alive(server.pid)
+        assert pid_is_alive(supervisor_pid_before)
+        assert supervisor_pid_from_log(server_log) == supervisor_pid_before
+
+        add = run_archivebox_cmd(
+            [
+                "add",
+                "--bg",
+                "--depth=1",
+                "--max-urls=20",
+                "--crawl-max-size=50mb",
+                "--plugins=wget,parse_html_urls",
+                recursive_test_site["root_url"],
+            ],
+            cwd=tmp_path,
+            env=env,
+            timeout=60,
+        )
+        assert add.returncode == 0, add.stderr or add.stdout
+        assert "background runner will process" in add.stdout
+
+        stop_archivebox_process(server, signal.SIGTERM)
+        server = None
+        assert "Got SIGTERM" in server_log.read_text(encoding="utf-8", errors="replace")
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+        with use_archivebox_db(tmp_path):
+            crawl_id = str(Crawl.objects.get().id)
+        run = run_archivebox_cmd(
+            ["run", f"--crawl-id={crawl_id}"],
+            cwd=tmp_path,
+            env=env,
+            timeout=180,
+        )
+        assert run.returncode == 0, run.stderr or run.stdout
+
+        server2 = start_archivebox_server(tmp_path, port=port, log_name="server-add-owner-2.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        captured_text = wait_for_snapshot_capture(tmp_path, recursive_test_site["root_url"], timeout=180)
+        assert "Root" in captured_text
+        stop_archivebox_process(server2, signal.SIGTERM)
+        server2 = None
+        with use_archivebox_db(tmp_path):
+            crawls = list(Crawl.objects.order_by("created_at").values_list("status", "retry_at"))
+            snapshots = list(Snapshot.objects.order_by("created_at").values_list("url", "status", "retry_at"))
+            bad_results = list(
+                ArchiveResult.objects.filter(
+                    status__in=[
+                        ArchiveResult.StatusChoices.FAILED,
+                        ArchiveResult.StatusChoices.SKIPPED,
+                    ],
+                ).values_list("plugin", "status", "output_str"),
+            )
+        assert crawls
+        assert snapshots
+        assert all(status == Crawl.StatusChoices.SEALED for status, _retry_at in crawls)
+        assert all(status == Snapshot.StatusChoices.SEALED for _url, status, _retry_at in snapshots)
+        assert not bad_results
+    finally:
+        for proc in (server, server2):
+            if proc is not None:
+                stop_archivebox_process(proc, signal.SIGTERM, timeout=10)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+# Utility-level takeover selection tests.
+
+
+def test_runtime_stack_owner_prefers_newer_server_over_older_update(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for process_type in (Process.TypeChoices.UPDATE, Process.TypeChoices.SERVER):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=process_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+
+        owner = runtime_stack_owner(data_dir=tmp_path)
+
+        assert owner is not None
+        assert owner.process_type == Process.TypeChoices.SERVER
+        assert owner.pid == procs[-1].pid
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_foreground_runner_owner_prefers_newer_server_over_update(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import foreground_runner_owner, runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for process_type in (Process.TypeChoices.UPDATE, Process.TypeChoices.SERVER):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=process_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+
+        runtime_owner = runtime_stack_owner(data_dir=tmp_path)
+        runner_owner = foreground_runner_owner(data_dir=tmp_path)
+
+        assert runtime_owner is not None
+        assert runtime_owner.process_type == Process.TypeChoices.SERVER
+        assert runner_owner is not None
+        assert runner_owner.process_type == Process.TypeChoices.SERVER
+        assert runner_owner.pid == procs[-1].pid
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_runtime_stack_owner_allows_top_level_runner_when_no_parent_command_exists(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import RUNNER_ACTIVE_WORKER_TYPE, runtime_stack_owner
+
+    proc = _start_archivebox_shell(tmp_path)
+    try:
+        runner_row = Process.objects.create(
+            machine=Machine.current(),
+            process_type=Process.TypeChoices.ORCHESTRATOR,
+            worker_type=RUNNER_ACTIVE_WORKER_TYPE,
+            pwd=str(tmp_path),
+            cmd=[],
+            pid=proc.pid,
+            status=Process.StatusChoices.RUNNING,
+        )
+
+        owner = runtime_stack_owner(data_dir=tmp_path)
+
+        assert owner is not None
+        assert owner.id == runner_row.id
+    finally:
+        _stop_archivebox_shells([proc])
@@ -0,0 +1,377 @@
+#!/usr/bin/env python3
+"""Takeover utility tests and live command handoff flows."""
+
+import os
+import signal
+import subprocess
+
+import pytest
+import psutil
+
+from archivebox.core.models import ArchiveResult
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    assert_no_processes_for_data_dir,
+    get_free_port,
+    kill_processes_for_data_dir,
+    cli_env,
+    pid_is_alive,
+    run_archivebox_cmd,
+    start_archivebox_server,
+    stop_archivebox_process,
+    supervisor_pid_from_log,
+    get_http_response,
+    wait_for_log,
+    wait_for_log_pattern,
+    wait_for_pid_to_disappear,
+    wait_for_worker_pid_from_log,
+    worker_pid_from_log,
+)
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_takeover_util_1 import (
+    pytestmark as pytestmark,
+    _resolve_sonic_env as _resolve_sonic_env,
+    _archive_pages_for_sqlite_reindexing as _archive_pages_for_sqlite_reindexing,
+    _start_archivebox_shell as _start_archivebox_shell,
+    _stop_archivebox_shells as _stop_archivebox_shells,
+)
+
+
+def test_pid_is_alive_treats_unreaped_archivebox_cli_as_exited(tmp_path, initialized_archive):
+    proc = run_archivebox_cmd(
+        ["version"],
+        cwd=tmp_path,
+        default_cli_env=True,
+        disable_extractors=True,
+        wait=False,
+    )
+    try:
+        os.waitid(os.P_PID, proc.pid, os.WEXITED | os.WNOWAIT)
+        assert psutil.Process(proc.pid).status() == psutil.STATUS_ZOMBIE
+        assert not pid_is_alive(proc.pid)
+    finally:
+        proc.wait(timeout=5)
+
+
+@pytest.mark.timeout(360)
+def test_behavior_update_index_only_keeps_server_http_and_search_visible(tmp_path, initialized_archive, recursive_test_site):
+    env = cli_env(
+        live=True,
+        PLUGINS="wget,parse_html_urls,search_backend_sqlite",
+        SEARCH_BACKEND_ENGINE="sqlite",
+        SEARCH_BACKEND_SONIC_PORT=str(get_free_port()),
+    )
+    env.update(_resolve_sonic_env(env))
+    root_url = recursive_test_site["root_url"]
+    _archive_pages_for_sqlite_reindexing(tmp_path, env, root_url)
+
+    port = get_free_port()
+    server = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="behavior-server-update.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        update = run_archivebox_cmd(
+            ["update", "--index-only", "--batch-size=1"],
+            cwd=tmp_path,
+            env=env,
+            timeout=180,
+        )
+
+        assert update.returncode == 0, update.stderr or update.stdout
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+
+        search = run_archivebox_cmd(
+            ["list", "--search=contents", "--csv=url", "Root"],
+            cwd=tmp_path,
+            env=env,
+            timeout=60,
+        )
+
+        assert search.returncode == 0, search.stderr or search.stdout
+        assert root_url in search.stdout
+    finally:
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(420)
+def test_behavior_update_yields_to_server_then_finishes_visible_indexing(tmp_path, initialized_archive, recursive_test_site):
+    env = cli_env(
+        live=True,
+        PLUGINS="wget,parse_html_urls,search_backend_sqlite",
+        SEARCH_BACKEND_ENGINE="sqlite",
+        SEARCH_BACKEND_SONIC_PORT=str(get_free_port()),
+    )
+    env.update(_resolve_sonic_env(env))
+    root_url = recursive_test_site["root_url"]
+    _archive_pages_for_sqlite_reindexing(tmp_path, env, root_url)
+
+    port = get_free_port()
+    update_proc = None
+    server = None
+    try:
+        update_log = tmp_path / "behavior-update-yields.log"
+        update_log_handle = update_log.open("w", encoding="utf-8")
+        update_proc = run_archivebox_cmd(
+            ["update", "--index-only", "--batch-size=1"],
+            cwd=tmp_path,
+            env=env,
+            stdout=update_log_handle,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,
+            wait=False,
+        )
+        update_log_handle.close()
+        wait_for_log(update_log, "[*] Reindexing", timeout=90)
+
+        server = start_archivebox_server(tmp_path, port=port, log_name="behavior-server-takes-update.log", env=env)
+        assert get_http_response(port, host=f"archivebox.localhost:{port}").status_code < 500
+        wait_for_log(update_log, "A newer archivebox process took over the orchestrator, sonic", timeout=90)
+
+        stop_archivebox_process(server, signal.SIGTERM)
+        server = None
+        update_proc.wait(timeout=180)
+        update_text = update_log.read_text(encoding="utf-8", errors="replace")
+        assert update_proc.returncode == 0, update_text
+
+        search = run_archivebox_cmd(
+            ["list", "--search=contents", "--csv=url", "Root"],
+            cwd=tmp_path,
+            env=env,
+            timeout=60,
+        )
+
+        assert search.returncode == 0, search.stderr or search.stdout
+        assert root_url in search.stdout
+    finally:
+        if update_proc is not None:
+            stop_archivebox_process(update_proc, signal.SIGTERM)
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(180)
+def test_live_update_index_only_does_not_take_over_server_runtime(tmp_path, initialized_archive):
+
+    env = cli_env(live=True)
+    port = get_free_port()
+    server = None
+    try:
+        server = start_archivebox_server(tmp_path, port=port, log_name="server-update-owner.log", env=env)
+        server_log = server.log_path
+        supervisor_pid_before = supervisor_pid_from_log(server_log)
+        daphne_pid_before = worker_pid_from_log(server_log, "worker_daphne")
+
+        result = run_archivebox_cmd(
+            ["update", "--index-only", "--before=0"],
+            cwd=tmp_path,
+            env=env,
+            timeout=90,
+        )
+
+        assert result.returncode == 0, result.stderr or result.stdout
+        assert pid_is_alive(server.pid)
+        assert pid_is_alive(supervisor_pid_before)
+        assert pid_is_alive(daphne_pid_before)
+        assert supervisor_pid_from_log(server_log) == supervisor_pid_before
+        assert worker_pid_from_log(server_log, "worker_daphne") == daphne_pid_before
+        assert "A newer archivebox process took over the orchestrator, server" not in server_log.read_text(
+            encoding="utf-8",
+            errors="replace",
+        )
+    finally:
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+@pytest.mark.timeout(420)
+def test_live_update_yields_to_server_then_reclaims_real_sqlite_indexing(tmp_path, initialized_archive, recursive_test_site):
+    env = cli_env(
+        live=True,
+        PLUGINS="wget,parse_html_urls,search_backend_sqlite,search_backend_sonic",
+        SEARCH_BACKEND_ENGINE="sqlite",
+        SEARCH_BACKEND_SONIC_PORT=str(get_free_port()),
+    )
+    env.update(_resolve_sonic_env(env))
+    _archive_pages_for_sqlite_reindexing(tmp_path, env, recursive_test_site["root_url"])
+
+    port = get_free_port()
+    update_proc = None
+    server = None
+    try:
+        update_log = tmp_path / "update-real-sqlite-owner.log"
+        update_log_handle = update_log.open("w", encoding="utf-8")
+        update_proc = run_archivebox_cmd(
+            ["update", "--index-only", "--batch-size=1"],
+            cwd=tmp_path,
+            env=env,
+            stdout=update_log_handle,
+            stderr=subprocess.STDOUT,
+            start_new_session=True,
+            wait=False,
+        )
+        update_log_handle.close()
+        wait_for_log(update_log, "[*] Reindexing", timeout=90)
+        update_supervisor_match = wait_for_log_pattern(update_log, r"Supervisord connected \(pid=(\d+)\)", timeout=90)
+        update_supervisor_pid_before = int(update_supervisor_match.group(1))
+        update_sonic_pid_before = wait_for_worker_pid_from_log(update_log, "worker_sonic", timeout=90)
+        assert pid_is_alive(update_supervisor_pid_before)
+        assert pid_is_alive(update_sonic_pid_before)
+        assert "worker_runner_update_" not in update_log.read_text(encoding="utf-8", errors="replace")
+        assert "worker_daphne" not in update_log.read_text(encoding="utf-8", errors="replace")
+
+        server = start_archivebox_server(tmp_path, port=port, log_name="server-takes-real-sqlite-update.log", env=env)
+        server_log = server.log_path
+        wait_for_log(update_log, "A newer archivebox process took over the orchestrator, sonic", timeout=90)
+        assert pid_is_alive(update_proc.pid)
+        assert pid_is_alive(server.pid)
+        server_text = server_log.read_text(encoding="utf-8", errors="replace")
+        # The older update process can yield orchestrator ownership just before
+        # the server logs its takeover, but sonic must always move to the server.
+        assert (
+            "Taking over orchestrator, sonic from older existing archivebox process" in server_text
+            or "Taking over sonic from older existing archivebox process" in server_text
+        )
+        assert "worker_daphne" in server_text
+        assert "worker_sonic" in server_text
+        server_daphne_pid = worker_pid_from_log(server_log, "worker_daphne")
+        server_runner_pid = worker_pid_from_log(server_log, "worker_runner")
+        server_sonic_pid = worker_pid_from_log(server_log, "worker_sonic")
+        assert pid_is_alive(server_daphne_pid)
+        assert pid_is_alive(server_runner_pid)
+        assert pid_is_alive(server_sonic_pid)
+        wait_for_pid_to_disappear(update_supervisor_pid_before, timeout=30)
+        wait_for_pid_to_disappear(update_sonic_pid_before, timeout=30)
+
+        stop_archivebox_process(server, signal.SIGTERM)
+        server = None
+        update_proc.wait(timeout=180)
+        update_text = update_log.read_text(encoding="utf-8", errors="replace")
+        assert update_proc.returncode == 0, update_text
+        wait_for_pid_to_disappear(server_daphne_pid, timeout=20)
+        wait_for_pid_to_disappear(server_runner_pid, timeout=20)
+        wait_for_pid_to_disappear(server_sonic_pid, timeout=20)
+
+        with use_archivebox_db(tmp_path):
+            indexed_results = list(
+                ArchiveResult.objects.filter(plugin="search_backend_sqlite").values_list("status", flat=True),
+            )
+        assert indexed_results
+        assert all(status in ArchiveResult.FINAL_STATES for status in indexed_results)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+    finally:
+        if update_proc is not None:
+            stop_archivebox_process(update_proc, signal.SIGTERM)
+        if server is not None:
+            stop_archivebox_process(server, signal.SIGTERM)
+        kill_processes_for_data_dir(tmp_path)
+        assert_no_processes_for_data_dir(tmp_path, timeout=12)
+
+
+def test_runtime_stack_owner_keeps_server_over_newer_update(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for process_type in (Process.TypeChoices.SERVER, Process.TypeChoices.UPDATE):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=process_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+
+        owner = runtime_stack_owner(data_dir=tmp_path)
+
+        assert owner is not None
+        assert owner.process_type == Process.TypeChoices.SERVER
+        assert owner.pid == procs[0].pid
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_runtime_stack_owner_keeps_server_over_newer_supervised_runner(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import RUNNER_ACTIVE_WORKER_TYPE, runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    try:
+        for process_type, worker_type in (
+            (Process.TypeChoices.SERVER, ""),
+            (Process.TypeChoices.ORCHESTRATOR, RUNNER_ACTIVE_WORKER_TYPE),
+        ):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=worker_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+
+        owner = runtime_stack_owner(data_dir=tmp_path)
+
+        assert owner is not None
+        assert owner.process_type == Process.TypeChoices.SERVER
+        assert owner.pid == procs[0].pid
+    finally:
+        _stop_archivebox_shells(procs)
+
+
+def test_runtime_stack_owner_reaps_dead_server_without_promoting_update(tmp_path, initialized_archive):
+    from archivebox.machine.models import Machine
+    from archivebox.core.takeover_util import runtime_stack_owner
+
+    procs: list[subprocess.Popen[str]] = []
+    older_row = None
+    newer_row = None
+    try:
+        for process_type in (Process.TypeChoices.UPDATE, Process.TypeChoices.SERVER):
+            proc = _start_archivebox_shell(tmp_path)
+            procs.append(proc)
+            row = Process.objects.create(
+                machine=Machine.current(),
+                process_type=process_type,
+                worker_type=process_type,
+                pwd=str(tmp_path),
+                cmd=[],
+                pid=proc.pid,
+                status=Process.StatusChoices.RUNNING,
+            )
+            if process_type == Process.TypeChoices.UPDATE:
+                older_row = row
+            else:
+                newer_row = row
+
+        assert older_row is not None
+        assert newer_row is not None
+
+        os.kill(procs[-1].pid, signal.SIGTERM)
+        procs[-1].wait(timeout=20)
+
+        owner = runtime_stack_owner(data_dir=tmp_path)
+
+        assert owner is None
+        newer_row.refresh_from_db()
+        assert newer_row.status == Process.StatusChoices.EXITED
+        older_row.refresh_from_db()
+        assert older_row.status == Process.StatusChoices.RUNNING
+    finally:
+        _stop_archivebox_shells(procs)