Bump release version to 0.9.72rc15

Expire completed snapshot event history after database projection
Bump release version to 0.9.72rc14
2026-10-02 19:37:09 +00:00 · 2026-10-02 12:34:55 -07:00 · 2026-10-02 19:34:28 +00:00 · 2026-10-02 19:34:08 +00:00 · 2026-10-02 17:18:32 +00:00 · 2026-10-02 10:18:16 -07:00
172 changed files with 15631 additions and 9665 deletions
@@ -0,0 +1,42 @@
+# Stable releases require live dev acceptance
+
+The order is **dev CI → dev image publication → automatic Cabbage deployment →
+real capture and browser acceptance → stable publication**. Never
+substitute a published PyPI wheel, healthy container, or local build for this
+acceptance. The release workflow blocks *before* a stable upload unless Cabbage
+has successful evidence from `staging-acceptance.yml` matching all tracked
+source and dependency pins. Only ArchiveBox's own version fields may change
+between dev and stable.
+
+The staging workflow observes the existing image watcher; it cannot deploy or
+restart the application. On Cabbage, install `bin/staging-acceptance-host.sh`
+as root-owned mode 0755 `/usr/local/sbin/archivebox-staging-acceptance`. Its hash
+is checked against the candidate. A dedicated SSH key must use
+`restrict,command="/usr/local/sbin/archivebox-staging-acceptance"` in authorized
+keys. It accepts only `inspect <source-sha>` and `capture <source-sha>`; it cannot
+run caller-provided commands or archive arbitrary URLs.
+
+The `staging` GitHub environment is restricted to dev/main branches. It contains
+`STAGING_SSH_KEY` and `STAGING_KNOWN_HOSTS`, with independently verified host keys.
+Only trusted release events use the private ugNAS runner route. Never expose
+these credentials to PR jobs or copy an operator's existing root key.
+
+Acceptance creates one tagged background crawl per source through the normal
+add/runner path, enabling every snapshot extractor, including TLSNotary. It
+checks installed pins, completed outputs and their files, unexpected/missing
+plugin results, host OOM kills, and real rendered snapshot pages. For this
+release, the explicitly accepted TLSNotary failures and exact missing-Claude-auth
+diagnostics are retained in the report's `limitations` list; they do not disable
+those plugins or excuse missing/truncated successful outputs. Remove the
+TLSNotary exception when its follow-up fix ships. Logs, result inventories and
+screenshots are retained as Actions artifacts. Other failures remain
+failures; do not publish a success deployment record to bypass them. No code
+mounts are allowed. The Cabbage job must pass before its record authorizes a
+stable upload. Queued capture progress is polled; failed tests are not retried.
+
+Browser acceptance follows the default WACZ viewer into the captured document,
+matching its replay base and original URL and checking that its body is visible
+and contains substantial text. An attached viewer iframe can still be
+`about:blank`, so its presence alone cannot prove replay works. Extracted display
+titles and the captured document's `<title>` may legitimately differ; neither
+title equality nor output thumbnails substitute for checking the actual page.
@@ -0,0 +1,3 @@
+self-hosted-runner:
+  labels:
+    - ugnas-linux
@@ -15,7 +15,7 @@ updates:
        patterns:
          - "*"
  - package-ecosystem: "npm"
-    directory: "/"
+    directory: "/etc"
    target-branch: "dev"
    schedule:
      interval: "monthly"
@@ -10,4 +10,4 @@ Also review the inline navbar/footer in abx-dl/website/index.html and
 -->
 <footer class="abx-footer" aria-label="ArchiveBox ecosystem">
  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
-  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated
@@ -1,37 +1,78 @@
 #!/usr/bin/env python3
-"""Discover every ArchiveBox test file and expose it as one CI job."""
+"""Discover every test once, batching measured short files."""

 import json
+import os
 from pathlib import Path


 def main() -> None:
-    root = Path.cwd().resolve()
+    root = Path(__file__).resolve().parents[2]
    archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))

    if not archivebox_tests:
        raise SystemExit("No ArchiveBox tests discovered")

-    matrix: list[dict[str, object]] = []
+    # These are observed whole-job seconds, including setup. Unknown/long files,
+    # optional dependencies and hosted-only requirements keep independent jobs.
+    durations = json.loads((root / ".github/test-durations.json").read_text())["seconds"]
+    hosted = {path for path in archivebox_tests if "# ci-runner: hosted" in path.read_text().splitlines()[:5]}
+    requested = os.environ.get("TEST_FILE", "")
+    batches: list[list[str]] = []
+    batch: list[str] = []
+    batch_seconds = 0
    for path in archivebox_tests:
        test_path = path.relative_to(root).as_posix()
+        duration = durations.get(test_path, 60)
+        if duration >= 60 or path in hosted or path.stem == "test_auth_ldap" or test_path == requested:
+            batches.append([test_path])
+            continue
+        if batch and (len(batch) == 8 or batch_seconds + duration > 180):
+            batches.append(batch)
+            batch = []
+            batch_seconds = 0
+        batch.append(test_path)
+        batch_seconds += duration
+    if batch:
+        batches.append(batch)
+
+    matrix: list[dict[str, object]] = []
+    for paths in batches:
        matrix.append(
            {
-                "name": f"main/{path.stem}",
-                "paths": [test_path],
-                "paths_arg": test_path,
-                "extra": "ldap" if test_path.endswith("/test_auth_ldap.py") else "",
-                "count": 1,
+                "name": f"main/{Path(paths[0]).stem}" if len(paths) == 1 else f"batch/{Path(paths[0]).stem}",
+                "paths": paths,
+                "paths_arg": " ".join(paths),
+                "extra": "ldap" if paths == ["archivebox/tests/test_auth_ldap.py"] else "",
+                "count": len(paths),
+                "ugnas": False,
            },
        )

+    # Ordinary Linux tests can use either pool. Only genuine hosted-only
+    # requirements need a file header; filenames never enter runner policy.
+    # Keep a bounded NAS share so hosted capacity continues working in parallel.
+    # This is queued work, not a worker count: several waves keep the capped
+    # runners occupied while hosted jobs run in parallel.
+    nas_job_limit = int(os.environ.get("UGNAS_CI_MAX_JOBS", "3"))
+    # Keep long jobs early in the hosted queue. Giving the longest jobs to the
+    # CPU-capped NAS instead created a tail after hosted work had finished; use
+    # its extra slots for a bounded share of shorter work without filename rules.
+    matrix.sort(key=lambda item: sum(durations.get(path, 60) for path in item["paths"]), reverse=True)
+    eligible = [item for item in reversed(matrix) if not any(root / path in hosted for path in item["paths"])]
+    for item in eligible[:nas_job_limit]:
+        item["ugnas"] = True
+
    discovered_paths = [path for entry in matrix for path in entry["paths"]]
    if len(discovered_paths) != len(set(discovered_paths)):
        raise SystemExit("Tests were not discovered exactly once")
    if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
        raise SystemExit("Discovered test shard coverage does not match test files")

-    print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
+    print(
+        f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs; "
+        f"{sum(item['ugnas'] for item in matrix)} selected for ugNAS, subject to the availability check",
+    )
    print(json.dumps(matrix, separators=(",", ":")))


@@ -0,0 +1,169 @@
+{
+  "source_runs": [
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36302511923",
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36353196552",
+    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/36454992009"
+  ],
+  "measurement": "Whole job seconds, including setup and excluding queue time. Hosted unbatched files use the latest complete successful run; other files retain their individual baseline measurement. Unknown files stay independent.",
+  "seconds": {
+    "archivebox/tests/test_api_v1_auth_browser_session.py": 29,
+    "archivebox/tests/test_api_v1_auth_check_api_token.py": 27,
+    "archivebox/tests/test_api_v1_auth_get_api_token.py": 24,
+    "archivebox/tests/test_api_v1_cli_add.py": 96,
+    "archivebox/tests/test_api_v1_cli_remove.py": 50,
+    "archivebox/tests/test_api_v1_cli_schedule.py": 41,
+    "archivebox/tests/test_api_v1_cli_search.py": 26,
+    "archivebox/tests/test_api_v1_cli_update.py": 74,
+    "archivebox/tests/test_api_v1_cli_workflow_add_search_update_remove.py": 64,
+    "archivebox/tests/test_api_v1_core_any_id.py": 21,
+    "archivebox/tests/test_api_v1_core_archiveresult_archiveresult_id.py": 55,
+    "archivebox/tests/test_api_v1_core_archiveresults.py": 32,
+    "archivebox/tests/test_api_v1_core_snapshot_snapshot_id.py": 199,
+    "archivebox/tests/test_api_v1_core_snapshots_rss.py": 27,
+    "archivebox/tests/test_api_v1_core_snapshots.py": 28,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_rename.py": 23,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_snapshots_jsonl.py": 27,
+    "archivebox/tests/test_api_v1_core_tag_tag_id_urls_txt.py": 28,
+    "archivebox/tests/test_api_v1_core_tag_tag_id.py": 27,
+    "archivebox/tests/test_api_v1_core_tags_add_to_snapshot.py": 27,
+    "archivebox/tests/test_api_v1_core_tags_autocomplete.py": 26,
+    "archivebox/tests/test_api_v1_core_tags_create.py": 21,
+    "archivebox/tests/test_api_v1_core_tags_remove_from_snapshot.py": 25,
+    "archivebox/tests/test_api_v1_core_tags_search.py": 26,
+    "archivebox/tests/test_api_v1_core_tags.py": 25,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_filename.py": 22,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_filename.py": 25,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_subfolder_filename.py": 24,
+    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py": 86,
+    "archivebox/tests/test_api_v1_crawls_crawls.py": 21,
+    "archivebox/tests/test_api_v1_machine_binaries.py": 23,
+    "archivebox/tests/test_api_v1_machine_binary_binary_id.py": 23,
+    "archivebox/tests/test_api_v1_machine_binary_by_name_name.py": 26,
+    "archivebox/tests/test_api_v1_machine_machine_current.py": 23,
+    "archivebox/tests/test_api_v1_machine_machine_machine_id.py": 23,
+    "archivebox/tests/test_api_v1_machine_machines.py": 27,
+    "archivebox/tests/test_api_v1_personas_personas.py": 23,
+    "archivebox/tests/test_api_v1_personas_sync.py": 29,
+    "archivebox/tests/test_api_v1_workflow_core_token_auth_side_effects.py": 36,
+    "archivebox/tests/test_api_v1_workflow_frozen_crawl_config_sources.py": 33,
+    "archivebox/tests/test_archive_result_service.py": 137,
+    "archivebox/tests/test_auth_ldap.py": 44,
+    "archivebox/tests/test_binary_service.py": 83,
+    "archivebox/tests/test_cli_add_1.py": 168,
+    "archivebox/tests/test_cli_add_2.py": 203,
+    "archivebox/tests/test_cli_add_3.py": 215,
+    "archivebox/tests/test_cli_add_4.py": 201,
+    "archivebox/tests/test_cli_archiveresult.py": 199,
+    "archivebox/tests/test_cli_binary.py": 20,
+    "archivebox/tests/test_cli_config.py": 148,
+    "archivebox/tests/test_cli_crawl.py": 177,
+    "archivebox/tests/test_cli_extract_input.py": 155,
+    "archivebox/tests/test_cli_extract.py": 106,
+    "archivebox/tests/test_cli_help.py": 30,
+    "archivebox/tests/test_cli_init.py": 135,
+    "archivebox/tests/test_cli_install.py": 58,
+    "archivebox/tests/test_cli_list.py": 260,
+    "archivebox/tests/test_cli_machine.py": 50,
+    "archivebox/tests/test_cli_manage.py": 48,
+    "archivebox/tests/test_cli_mcp.py": 51,
+    "archivebox/tests/test_cli_oneshot.py": 23,
+    "archivebox/tests/test_cli_persona.py": 82,
+    "archivebox/tests/test_cli_piping.py": 97,
+    "archivebox/tests/test_cli_pluginmap.py": 21,
+    "archivebox/tests/test_cli_process.py": 20,
+    "archivebox/tests/test_cli_remove.py": 162,
+    "archivebox/tests/test_cli_run_1.py": 195,
+    "archivebox/tests/test_cli_run_2.py": 255,
+    "archivebox/tests/test_cli_run_3.py": 167,
+    "archivebox/tests/test_cli_run_4.py": 178,
+    "archivebox/tests/test_cli_schedule.py": 87,
+    "archivebox/tests/test_cli_search.py": 35,
+    "archivebox/tests/test_cli_server.py": 259,
+    "archivebox/tests/test_cli_shell.py": 32,
+    "archivebox/tests/test_cli_snapshot.py": 181,
+    "archivebox/tests/test_cli_status.py": 102,
+    "archivebox/tests/test_cli_tag.py": 21,
+    "archivebox/tests/test_cli_update_reindex_snapshots.py": 163,
+    "archivebox/tests/test_cli_update.py": 142,
+    "archivebox/tests/test_cli_version.py": 143,
+    "archivebox/tests/test_config_DELETE_AFTER.py": 169,
+    "archivebox/tests/test_config_MAX_limits.py": 34,
+    "archivebox/tests/test_config_ONLY_NEW.py": 24,
+    "archivebox/tests/test_config_SAVE_TITLE.py": 99,
+    "archivebox/tests/test_config_URL_filters.py": 26,
+    "archivebox/tests/test_core_config.py": 20,
+    "archivebox/tests/test_core_forms.py": 28,
+    "archivebox/tests/test_crawl_pause.py": 20,
+    "archivebox/tests/test_crawl_runner.py": 45,
+    "archivebox/tests/test_crawl_service.py": 34,
+    "archivebox/tests/test_frozen_crawl_config.py": 47,
+    "archivebox/tests/test_hook_result_lifecycle.py": 78,
+    "archivebox/tests/test_hooks.py": 90,
+    "archivebox/tests/test_machine_models.py": 52,
+    "archivebox/tests/test_machine_service.py": 89,
+    "archivebox/tests/test_migrations_04_to_09.py": 36,
+    "archivebox/tests/test_migrations_07_to_09.py": 183,
+    "archivebox/tests/test_migrations_08_to_09.py": 244,
+    "archivebox/tests/test_migrations_fresh.py": 99,
+    "archivebox/tests/test_misc_checks.py": 22,
+    "archivebox/tests/test_misc_monkey_patches.py": 20,
+    "archivebox/tests/test_opencode_agent_1.py": 158,
+    "archivebox/tests/test_opencode_agent_2.py": 145,
+    "archivebox/tests/test_opencode_agent_3.py": 154,
+    "archivebox/tests/test_opencode_browser_1.py": 262,
+    "archivebox/tests/test_opencode_browser_2.py": 146,
+    "archivebox/tests/test_opencode_browser_3.py": 148,
+    "archivebox/tests/test_orm_helpers.py": 22,
+    "archivebox/tests/test_permissions.py": 21,
+    "archivebox/tests/test_persona_runtime.py": 56,
+    "archivebox/tests/test_postgres_backend.py": 85,
+    "archivebox/tests/test_process_runtime_paths.py": 28,
+    "archivebox/tests/test_process_service.py": 49,
+    "archivebox/tests/test_proof_previews.py": 54,
+    "archivebox/tests/test_recursive_crawl_1.py": 339,
+    "archivebox/tests/test_recursive_crawl_2.py": 171,
+    "archivebox/tests/test_recursive_crawl_3.py": 176,
+    "archivebox/tests/test_release_notes.py": 14,
+    "archivebox/tests/test_release_source_fingerprint.py": 22,
+    "archivebox/tests/test_release_workflow.py": 21,
+    "archivebox/tests/test_resource_admission.py": 21,
+    "archivebox/tests/test_runner_watch.py": 177,
+    "archivebox/tests/test_search.py": 153,
+    "archivebox/tests/test_serve_static.py": 31,
+    "archivebox/tests/test_server_automatic_url.py": 36,
+    "archivebox/tests/test_server_security_browser.py": 278,
+    "archivebox/tests/test_settings_signal_webhooks.py": 21,
+    "archivebox/tests/test_setup_script.py": 21,
+    "archivebox/tests/test_shutdown_util.py": 22,
+    "archivebox/tests/test_snapshot_filesystem_migration.py": 29,
+    "archivebox/tests/test_snapshot_output_stacks.py": 62,
+    "archivebox/tests/test_snapshot_ref_lookup.py": 23,
+    "archivebox/tests/test_snapshot_service.py": 41,
+    "archivebox/tests/test_snapshot_url_length.py": 26,
+    "archivebox/tests/test_tag_service.py": 24,
+    "archivebox/tests/test_takeover_util_1.py": 143,
+    "archivebox/tests/test_takeover_util_2.py": 201,
+    "archivebox/tests/test_takeover_util_3.py": 138,
+    "archivebox/tests/test_takeover_util_4.py": 156,
+    "archivebox/tests/test_test_harness.py": 22,
+    "archivebox/tests/test_ui_add_view_runtime.py": 221,
+    "archivebox/tests/test_ui_add_view.py": 46,
+    "archivebox/tests/test_ui_admin_apitoken.py": 23,
+    "archivebox/tests/test_ui_admin_archiveresult.py": 30,
+    "archivebox/tests/test_ui_admin_config_widget.py": 25,
+    "archivebox/tests/test_ui_admin_crawl.py": 44,
+    "archivebox/tests/test_ui_admin_links.py": 35,
+    "archivebox/tests/test_ui_admin_machine.py": 29,
+    "archivebox/tests/test_ui_admin_persona.py": 28,
+    "archivebox/tests/test_ui_admin_snapshot.py": 273,
+    "archivebox/tests/test_ui_admin_tag.py": 26,
+    "archivebox/tests/test_ui_admin_user.py": 28,
+    "archivebox/tests/test_ui_config_views.py": 43,
+    "archivebox/tests/test_ui_live_progress.py": 297,
+    "archivebox/tests/test_ui_public_snapshot.py": 179,
+    "archivebox/tests/test_ui_savepagenow.py": 48,
+    "archivebox/tests/test_urls.py": 220,
+    "archivebox/tests/test_util.py": 20,
+    "archivebox/tests/test_version_metadata.py": 24
+  }
+}
@@ -18,47 +18,68 @@ concurrency:
  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}

 jobs:
+  prepare:
+    permissions:
+      contents: write
+    uses: ./.github/workflows/prepare-release.yml
+    secrets: inherit
+
  lint:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Linters
    uses: ./.github/workflows/lint.yml
    secrets: inherit

  install-cli-platform:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Install and CLI platform compatibility
    uses: ./.github/workflows/test.yml
    secrets: inherit

  tests:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Discovered test matrix
    uses: ./.github/workflows/test-parallel.yml
    secrets: inherit

  documentation:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Documentation and root tests
    uses: ./.github/workflows/docs.yml
    secrets: inherit

  codeql:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: CodeQL
    uses: ./.github/workflows/codeql.yml
    secrets: inherit

  python-artifacts:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Tested Python artifacts
    uses: ./.github/workflows/pip.yml
    secrets: inherit

  docker-digests:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    name: Tested Docker images
    uses: ./.github/workflows/docker.yml
    with:
-      push_digests: ${{ github.event_name == 'push' && github.ref_name == 'main' }}
+      push_digests: ${{ github.event_name == 'push' && (github.ref_name == 'dev' || github.ref_name == 'main') }}
    secrets: inherit

  required:
    name: All required CI lanes
-    if: ${{ always() }}
+    if: ${{ always() && needs.prepare.outputs.run_jobs == 'true' }}
    needs:
+      - prepare
      - lint
      - install-cli-platform
      - tests
@@ -65,7 +65,7 @@ jobs:
        if: github.event_name == 'workflow_dispatch'
        shell: bash
        env:
-          SOURCE_REF: ${{ inputs.source_ref }}
+          SOURCE_REF: ${{ inputs.source_ref || github.sha }}
        run: |
          set -Eeuo pipefail
          [[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
@@ -295,7 +295,7 @@ jobs:
            '

      - name: Validate exact built image version and commit
-        if: ${{ !inputs.push_digests }}
+        if: inputs.full_tests
        shell: bash
        run: |
          set -Eeuo pipefail
@@ -552,6 +552,9 @@ jobs:

      - name: Export digest
        if: inputs.push_digests
+        env:
+          ABX_DL_IMAGE_REF: ${{ steps.abx_dl_image.outputs.image }}
+          ARTIFACT_NAME: ${{ matrix.artifact_name }}
        shell: bash
        run: |
          set -Eeuo pipefail
@@ -559,6 +562,7 @@ jobs:
          digest="${{ steps.docker_build.outputs.digest }}"
          touch "/tmp/digests/${digest#sha256:}"
          printf '%s\n' '${{ inputs.source_ref || github.sha }}' > /tmp/digests/source-ref
+          printf '%s\n' "$ABX_DL_IMAGE_REF" > "/tmp/digests/abx-dl-image-$ARTIFACT_NAME"

      - name: Upload digest
        if: inputs.push_digests
@@ -67,7 +67,7 @@ jobs:
    timeout-minutes: 15
    strategy:
      fail-fast: false
-      max-parallel: 4
+      max-parallel: 8
      matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
    steps:
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -0,0 +1,55 @@
+name: Prepare release version
+
+on:
+  workflow_call:
+    outputs:
+      run_jobs:
+        description: Whether this revision owns its release version and should run expensive jobs
+        value: ${{ jobs.prepare.outputs.run_jobs }}
+
+permissions:
+  contents: write
+
+jobs:
+  prepare:
+    name: Prepare the version before starting expensive jobs
+    permissions:
+      contents: write
+    runs-on: ubuntu-24.04
+    outputs:
+      run_jobs: ${{ github.event_name != 'push' || steps.channel.outputs.stable == 'true' || steps.version.outputs.action == 'ready' || steps.version.outputs.action == 'reserved' }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          submodules: true
+          token: ${{ github.event_name == 'push' && secrets.RELEASE_GH_TOKEN || github.token }}
+
+      - name: Keep synchronized stable branches on the released version
+        id: channel
+        if: github.event_name == 'push'
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
+          git fetch --no-tags origin main
+          MAIN_SHA="$(git rev-parse origin/main)"
+          # Stable reconciliation may preserve dev history with a merge commit.
+          # Its tree still matches main; bumping it would undo synchronization
+          # before the stable publisher can advance the Docker :dev tag.
+          if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] &&
+             git merge-base --is-ancestor "$MAIN_SHA" "$GITHUB_SHA" &&
+             git diff --quiet "$MAIN_SHA" "$GITHUB_SHA"; then
+            echo 'stable=true' >> "$GITHUB_OUTPUT"
+          else
+            echo 'stable=false' >> "$GITHUB_OUTPUT"
+          fi
+
+      - id: version
+        if: github.event_name == 'push' && steps.channel.outputs.stable != 'true'
+        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
+        with:
+          package: archivebox
+          branch: ${{ github.ref_name }}
+          tag-prefix: v
+          version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}
@@ -1,79 +0,0 @@
-name: Release candidate
-
-on:
-  push:
-    branches: [dev, main]
-
-permissions:
-  actions: read
-  contents: write
-  packages: write
-
-concurrency:
-  group: release-candidate-${{ github.ref_name }}
-  cancel-in-progress: true
-
-jobs:
-  prepare:
-    runs-on: ubuntu-24.04
-    outputs:
-      action: ${{ steps.version.outputs.action }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          token: ${{ secrets.RELEASE_GH_TOKEN }}
-
-      - name: Keep synchronized stable branches on the released version
-        id: channel
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
-          MAIN_SHA="$(git ls-remote origin refs/heads/main | cut -f1)"
-          if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ && "$MAIN_SHA" == "$GITHUB_SHA" ]]; then
-            echo 'stable=true' >> "$GITHUB_OUTPUT"
-          else
-            echo 'stable=false' >> "$GITHUB_OUTPUT"
-          fi
-
-      - id: version
-        if: steps.channel.outputs.stable != 'true'
-        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
-        with:
-          package: archivebox
-          branch: ${{ github.ref_name }}
-          tag-prefix: v
-          version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}
-
-  python-artifacts:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    uses: ./.github/workflows/pip.yml
-    with:
-      full_tests: false
-    secrets: inherit
-
-  docker-digests:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    uses: ./.github/workflows/docker.yml
-    with:
-      push_digests: true
-      full_tests: false
-    secrets: inherit
-
-  opencode:
-    needs: prepare
-    if: github.ref_name == 'dev' && (needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved')
-    strategy:
-      fail-fast: false
-      matrix:
-        test_file:
-          - archivebox/tests/test_opencode_agent.py
-          - archivebox/tests/test_opencode_browser.py
-    uses: ./.github/workflows/test-parallel.yml
-    with:
-      test_file: ${{ matrix.test_file }}
-    secrets: inherit
@@ -2,7 +2,7 @@ name: Release

 on:
  workflow_run:
-    workflows: [CI, Release candidate]
+    workflows: [CI]
    types: [completed]
    branches: [dev, main]
  workflow_dispatch:
@@ -68,6 +68,15 @@ jobs:
          fi
          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
          if [[ "$RELEASE_BRANCH" == main ]]; then
+            [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
+              echo "Refusing to publish non-stable version $VERSION as a main release" >&2
+              exit 1
+            }
+            git fetch --no-tags origin main
+            git merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
+              echo "Refusing to publish $RELEASE_SHA: it is not on main" >&2
+              exit 1
+            }
            TAG_TARGET="$(git ls-remote origin "refs/tags/v$VERSION" | cut -f1)"
            if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch && -n "$TAG_TARGET" && "$TAG_TARGET" != "$RELEASE_SHA" ]]; then
              echo 'ready=false' >> "$GITHUB_OUTPUT"
@@ -102,26 +111,14 @@ jobs:
              --jq '[.status, .conclusion, .headSha, .event, .name] | @tsv')"
            [[ "$CI_STATUS" == "$(printf 'completed\tsuccess\t%s\tpush\tCI' "$RELEASE_SHA")" ]]
          else
-            CANDIDATE_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow release-candidate.yml \
-              --commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
-              --json databaseId,status,conclusion \
-              --jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
            CI_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml \
              --commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
              --json databaseId,status,conclusion \
              --jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
-            [[ -n "$CANDIDATE_RUN_ID" && -n "$CI_RUN_ID" ]] || exit 0
-            if [[ "$RELEASE_BRANCH" == dev ]]; then
-              DIGEST_RUN_ID="$CANDIDATE_RUN_ID"
-            else
-              DIGEST_RUN_ID="$CI_RUN_ID"
-            fi
-          fi
-          if [[ "$RELEASE_BRANCH" == dev && "$GITHUB_EVENT_NAME" != workflow_dispatch ]]; then
-            ARTIFACT_RUN_ID="$CANDIDATE_RUN_ID"
-          else
-            ARTIFACT_RUN_ID="$CI_RUN_ID"
+            [[ -n "$CI_RUN_ID" ]] || exit 0
+            DIGEST_RUN_ID="$CI_RUN_ID"
          fi
+          ARTIFACT_RUN_ID="$CI_RUN_ID"
          {
            echo 'ready=true'
            echo "artifact_run_id=$ARTIFACT_RUN_ID"
@@ -135,10 +132,12 @@ jobs:
    runs-on: ubuntu-24.04
    env:
      CI_RUN_ID: ${{ needs.candidate.outputs.artifact_run_id }}
+      DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
    environment: pypi
    permissions:
      actions: read
      contents: write
+      deployments: read
      id-token: write
    outputs:
      version: ${{ steps.version.outputs.version }}
@@ -162,6 +161,26 @@ jobs:
          version: "0.11.3"
          enable-cache: false

+      # This precedes every stable upload, including workflow_dispatch. A green
+      # build is not evidence that our automatically updated test servers ran it.
+      - name: Download tested Docker base-image metadata
+        if: env.RELEASE_BRANCH == 'main'
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          path: ${{ runner.temp }}/tested-docker-artifacts
+          pattern: digest-*
+          merge-multiple: true
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
+
+      - name: Require matching acceptance on Cabbage
+        if: env.RELEASE_BRANCH == 'main'
+        env:
+          GH_TOKEN: ${{ github.token }}
+          STAGING_DOCKER_ARTIFACT_DIR: ${{ runner.temp }}/tested-docker-artifacts
+        run: uv run --no-project python bin/verify-staging-release.py
+
      - name: Resolve release binaries through abxpkg
        env:
          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
@@ -210,6 +229,10 @@ jobs:
          VERSION='${{ steps.version.outputs.version }}'
          case "$RELEASE_BRANCH" in
            main)
+              [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
+                echo "Refusing to publish non-stable version $VERSION from main" >&2
+                exit 1
+              }
              ;;
            dev)
              [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] || {
@@ -266,7 +289,7 @@ jobs:
          ref: ${{ env.RELEASE_SHA }}
          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}

-      - name: Fast-forward dev to the tested stable release
+      - name: Reconcile dev to the tested stable release
        if: env.RELEASE_BRANCH == 'main' && needs.python-release.outputs.release_ready == 'true'
        shell: bash
        run: |
@@ -274,10 +297,65 @@ jobs:
          git fetch --no-tags origin main dev
          MAIN_SHA="$(git rev-parse origin/main)"
          DEV_SHA="$(git rev-parse origin/dev)"
-          if [[ "$MAIN_SHA" == "$RELEASE_SHA" && "$DEV_SHA" != "$RELEASE_SHA" ]] &&
-             git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
-            git push origin "$RELEASE_SHA:refs/heads/dev"
+          if [[ "$MAIN_SHA" != "$RELEASE_SHA" || "$DEV_SHA" == "$RELEASE_SHA" ]]; then
+            exit 0
          fi
+          push_dev() {
+            if ! git push origin "$1:refs/heads/dev"; then
+              CURRENT_DEV="$(git ls-remote origin refs/heads/dev | cut -f1)"
+              if [[ "$CURRENT_DEV" == "$DEV_SHA" ]]; then
+                echo "Failed to push dev although it still points to $DEV_SHA" >&2
+                exit 1
+              fi
+              echo "Leaving dev at $CURRENT_DEV: it advanced during release reconciliation"
+            fi
+          }
+          if git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
+            push_dev "$RELEASE_SHA"
+            exit 0
+          else
+            STATUS=$?
+            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
+          fi
+          BASE_SHA="$(git merge-base "$MAIN_SHA" "$DEV_SHA")"
+          if git diff --quiet "$BASE_SHA" "$DEV_SHA" -- . ':!pyproject.toml' ':!uv.lock' ':!etc/package.json'; then
+            :
+          else
+            STATUS=$?
+            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
+            echo "Leaving dev at $DEV_SHA: it contains changes beyond release metadata"
+            exit 0
+          fi
+          normalized_metadata() {
+            case "$2" in
+              pyproject.toml)
+                git show "$1:$2" | awk '
+                  /^\[/ {section = $0}
+                  section == "[project]" && /^version = "[^"]+"$/ {print "version = \"__VERSION__\""; next}
+                  section == "[tool.bumpver]" && /^current_version = "[^"]+"$/ {print "current_version = \"__VERSION__\""; next}
+                  {print}
+                ' ;;
+              uv.lock)
+                git show "$1:$2" | awk 'previous == "name = \"archivebox\"" && /^version = / {print "version = \"__VERSION__\""; previous = $0; next} {print; previous = $0}' ;;
+              etc/package.json)
+                git show "$1:$2" | sed -E 's/^(  "version": )"[^"]+"(,?)$/\1"__VERSION__"\2/' ;;
+            esac
+          }
+          for path in pyproject.toml uv.lock etc/package.json; do
+            BASE_METADATA="$(normalized_metadata "$BASE_SHA" "$path")"
+            DEV_METADATA="$(normalized_metadata "$DEV_SHA" "$path")"
+            if [[ "$BASE_METADATA" != "$DEV_METADATA" ]]; then
+              echo "Leaving dev at $DEV_SHA: $path contains changes beyond release version metadata"
+              exit 0
+            fi
+          done
+          # A concurrent release bot may bump only dev's own version metadata.
+          # Keep that commit as a parent while preserving the exact tested stable tree.
+          git switch --detach "$MAIN_SHA"
+          git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
+            merge -s ours --no-ff -m "Merge tested stable release into dev" "$DEV_SHA"
+          MERGE_SHA="$(git rev-parse HEAD)"
+          push_dev "$MERGE_SHA"

      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
        with:
@@ -369,6 +447,26 @@ jobs:
            TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}")"
            TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
          fi
+          if [[ "$RELEASE_BRANCH" == main ]]; then
+            [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
+              echo "Refusing to tag :latest with prerelease $VERSION" >&2
+              exit 1
+            }
+            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] || {
+              echo "Refusing to tag :latest: v$VERSION does not point to $RELEASE_SHA" >&2
+              exit 1
+            }
+            $GIT_BINARY fetch --no-tags origin main
+            $GIT_BINARY merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
+              echo "Refusing to tag :latest: $RELEASE_SHA is not on main" >&2
+              exit 1
+            }
+            MAIN_VERSION="$($GIT_BINARY show origin/main:pyproject.toml | sed -nE 's/^version = "([^"]+)".*/\1/p' | head -n 1)"
+            [[ "$MAIN_VERSION" == "$VERSION" ]] || {
+              echo "Refusing to tag :latest: main is now $MAIN_VERSION, not $VERSION" >&2
+              exit 1
+            }
+          fi
          SYNC_DEV=false
          if [[ "$RELEASE_BRANCH" == main ]]; then
            $GIT_BINARY fetch --no-tags origin main dev
@@ -376,7 +474,8 @@ jobs:
            DEV_TARGET="$($GIT_BINARY rev-parse origin/dev)"
            # Recovery can add CI/docs fixes after the immutable release tag.
            # Both branches and every shipped application file must agree.
-            if [[ "$MAIN_TARGET" == "$DEV_TARGET" ]] &&
+            if $GIT_BINARY merge-base --is-ancestor "$MAIN_TARGET" "$DEV_TARGET" &&
+               $GIT_BINARY diff --quiet "$MAIN_TARGET" "$DEV_TARGET" &&
               $GIT_BINARY diff --quiet "$RELEASE_SHA" "$MAIN_TARGET" -- . ':!.github' ':!README.md'; then
              SYNC_DEV=true
            fi
@@ -438,6 +537,26 @@ jobs:
            test "$INSTALLED_VERSION" = "$VERSION"
          done

+      - name: Record published image source
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          # workflow_run can report the default branch's head even when this
+          # run published a different branch/SHA. Preserve the verified target.
+          [[ "$RELEASE_BRANCH" == dev || "$RELEASE_BRANCH" == main ]]
+          [[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
+          mkdir -p "$RUNNER_TEMP/published-image-source"
+          # shellcheck disable=SC2016
+          "$JQ_BINARY" -n --arg branch "$RELEASE_BRANCH" --arg sha "$RELEASE_SHA" \
+            '{branch: $branch, sha: $sha}' > "$RUNNER_TEMP/published-image-source/source.json"
+
+      - name: Upload published image source
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: published-image-source
+          path: ${{ runner.temp }}/published-image-source/source.json
+          if-no-files-found: error
+
      - name: Update Docker Hub README
        uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
        with:
@@ -13,6 +13,7 @@ on:
      - pyproject.toml
      - uv.lock
      - .github/workflows/screenshots.yml
+      - .github/workflows/prepare-release.yml
  workflow_dispatch:

 permissions:
@@ -23,7 +24,15 @@ concurrency:
  cancel-in-progress: false

 jobs:
+  prepare:
+    permissions:
+      contents: write
+    uses: ./.github/workflows/prepare-release.yml
+    secrets: inherit
+
  deploy:
+    needs: prepare
+    if: needs.prepare.outputs.run_jobs == 'true'
    runs-on: ubuntu-24.04
    steps:
      - name: Checkout
@@ -0,0 +1,109 @@
+name: Staging acceptance
+
+on:
+  workflow_run:
+    workflows: [Release]
+    types: [completed]
+
+permissions: {}
+
+concurrency:
+  group: staging-acceptance
+  cancel-in-progress: false
+
+jobs:
+  published:
+    if: github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-24.04
+    permissions:
+      actions: read
+    outputs:
+      ready: ${{ steps.source.outputs.ready }}
+      sha: ${{ steps.source.outputs.sha }}
+    steps:
+      - id: image
+        env:
+          GH_TOKEN: ${{ github.token }}
+          RELEASE_RUN_ID: ${{ github.event.workflow_run.id }}
+        run: |
+          set -Eeuo pipefail
+          published=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RELEASE_RUN_ID/jobs?per_page=100" --jq 'any(.jobs[]; .name == "Publish tested multiarch image" and .conclusion == "success")')
+          echo "published=$published" >> "$GITHUB_OUTPUT"
+      - name: Download published image source
+        if: steps.image.outputs.published == 'true'
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: published-image-source
+          path: ${{ runner.temp }}/published-image-source
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ github.event.workflow_run.id }}
+      - name: Select published dev image
+        id: source
+        if: steps.image.outputs.published == 'true'
+        run: |
+          set -Eeuo pipefail
+          # The triggering run may report the default branch as its head;
+          # this artifact names the image source verified by the publisher.
+          source="$RUNNER_TEMP/published-image-source/source.json"
+          jq -e 'type == "object" and (keys | sort) == ["branch", "sha"]' "$source" >/dev/null
+          branch=$(jq -er '.branch | select(type == "string")' "$source")
+          sha=$(jq -er '.sha | select(type == "string")' "$source")
+          [[ "$branch" == dev || "$branch" == main ]]
+          [[ "$sha" =~ ^[0-9a-f]{40}$ ]]
+          echo "sha=$sha" >> "$GITHUB_OUTPUT"
+          if [[ "$branch" == dev ]]; then
+            echo 'ready=true' >> "$GITHUB_OUTPUT"
+          else
+            echo 'ready=false' >> "$GITHUB_OUTPUT"
+          fi
+  acceptance:
+    # The privileged key is never exposed to PRs or arbitrary dispatch refs.
+    needs: published
+    if: needs.published.outputs.ready == 'true'
+    runs-on: [self-hosted, Linux, X64, ugnas-linux]
+    environment: staging
+    timeout-minutes: 110
+    permissions:
+      contents: read
+      deployments: write
+    strategy:
+      fail-fast: false
+      matrix:
+        target: [cabbage]
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
+        with:
+          ref: ${{ needs.published.outputs.sha }}
+          fetch-depth: 0
+          persist-credentials: false
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e
+        with:
+          version: '0.11.3'
+      - name: Prepare browser and restricted staging identity
+        env:
+          STAGING_SSH_KEY: ${{ secrets.STAGING_SSH_KEY }}
+          STAGING_KNOWN_HOSTS: ${{ secrets.STAGING_KNOWN_HOSTS }}
+        run: |
+          set -Eeuo pipefail
+          umask 077
+          printf '%s\n' "$STAGING_SSH_KEY" > "$RUNNER_TEMP/staging-key"
+          printf '%s\n' "$STAGING_KNOWN_HOSTS" > "$RUNNER_TEMP/staging-known-hosts"
+          uv run --no-project --with playwright==1.63.0 playwright install chromium
+      - name: Verify actual deployment, full captures and rendered snapshots
+        env:
+          GH_TOKEN: ${{ github.token }}
+          STAGING_TARGET: ${{ matrix.target }}
+          STAGING_SHA: ${{ needs.published.outputs.sha }}
+          STAGING_KEY_FILE: ${{ runner.temp }}/staging-key
+          STAGING_KNOWN_HOSTS_FILE: ${{ runner.temp }}/staging-known-hosts
+        run: uv run --no-project --with playwright==1.63.0 python bin/check-staging-deployment.py
+      - name: Remove staging identity
+        if: always()
+        run: rm -f "$RUNNER_TEMP/staging-key" "$RUNNER_TEMP/staging-known-hosts"
+      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
+        if: always()
+        with:
+          name: staging-${{ matrix.target }}-${{ needs.published.outputs.sha }}
+          path: staging-evidence/
+          if-no-files-found: error
@@ -1,6 +1,13 @@
 name: Parallel Tests

 on:
+  workflow_dispatch:
+    inputs:
+      test_file:
+        description: Run one discovered test file
+        required: true
+        default: archivebox/tests/test_cli_list.py
+        type: string
  workflow_call:
    inputs:
      test_file:
@@ -19,11 +26,27 @@ env:
 jobs:
  discover-tests:
    name: Discover every test
+    # Selection must run on GitHub so an offline NAS cannot block the fallback.
    runs-on: ubuntu-24.04
    outputs:
      tests: ${{ steps.discover.outputs.tests }}
+      runner: ${{ steps.runner.outputs.use-runner || '["ubuntu-24.04"]' }}

    steps:
+      - name: Select available CI capacity
+        id: runner
+        if: vars.UGNAS_CI_ENABLED == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main')
+        uses: mikehardy/runner-fallback-action@dc7732f9e532c451b2527b288ba8d58fd4b0a4ca # v1.1.0
+        with:
+          github-token: ${{ secrets.RUNNER_STATUS_TOKEN }}
+          organization: ArchiveBox
+          primary-runner: self-hosted,ugnas-linux
+          fallback-runner: ubuntu-24.04
+          # Jobs queued for this pool must not be mistaken for idle runner slots:
+          # requiring MAX_JOBS idle runners prevents assigning a second wave.
+          primaries-required: ${{ vars.UGNAS_CI_MIN_IDLE || 1 }}
+          fallback-on-error: true
+
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
        with:
          submodules: true
@@ -66,6 +89,7 @@ jobs:
        id: discover
        env:
          TEST_FILE: ${{ inputs.test_file }}
+          UGNAS_CI_MAX_JOBS: ${{ vars.UGNAS_CI_MAX_JOBS || 3 }}
        run: |
          set -Eeuo pipefail
          mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
@@ -80,13 +104,13 @@ jobs:
  tests:
    name: ${{ matrix.test.name }}
    needs: discover-tests
-    runs-on: ubuntu-24.04
+    # Opt in trusted branches only; PRs always use GitHub-hosted runners.
+    # Ordinary Linux tests are eligible; # ci-runner: hosted declares exceptions.
+    # The hosted discovery job falls back when workers are offline/busy or the
+    # availability API fails. UGNAS_CI_ENABLED=false disables the pool entirely.
+    runs-on: ${{ fromJSON(matrix.test.ugnas && needs.discover-tests.outputs.runner || '["ubuntu-24.04"]') }}
    env:
-      ABXPKG_LIB_DIR: /tmp/archivebox-lib
      CHROME_HEADLESS: "true"
-      DATA_DIR: /tmp/archivebox-test-data
-      PERSONAS_DIR: /tmp/abx-personas
-      CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile

    strategy:
      fail-fast: false
@@ -94,6 +118,15 @@ jobs:
        test: ${{ fromJson(needs.discover-tests.outputs.tests) }}

    steps:
+      - name: Set job-local runtime paths
+        run: |
+          {
+            echo "ABXPKG_LIB_DIR=$RUNNER_TEMP/archivebox-lib"
+            echo "DATA_DIR=$RUNNER_TEMP/archivebox-test-data"
+            echo "PERSONAS_DIR=$RUNNER_TEMP/abx-personas"
+            echo "CHROME_USER_DATA_DIR=$RUNNER_TEMP/abx-personas/Default/chrome_profile"
+          } >> "$GITHUB_ENV"
+
      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
        with:
          submodules: true
@@ -144,7 +177,7 @@ jobs:
          fi
          uv sync --no-cache --locked --dev "${extra_args[@]}"
      - name: Install Playwright Chromium for browser tests
-        if: contains(matrix.test.paths_arg, 'test_proof_previews') || contains(matrix.test.paths_arg, 'test_snapshot_output_stacks')
+        if: contains(matrix.test.paths_arg, 'test_proof_previews') || contains(matrix.test.paths_arg, 'test_snapshot_output_stacks') || contains(matrix.test.paths_arg, 'test_ui_admin_snapshot') || contains(matrix.test.paths_arg, 'test_cli_extract')
        run: uv run --no-cache --no-sync --no-sources playwright install chromium

      - name: Run ${{ matrix.test.name }}
@@ -156,13 +189,35 @@ jobs:
          for test_path in "${test_paths[@]}"; do
            test -f "$test_path"
          done
+          test_status=0
          for test_path in "${test_paths[@]}"; do
            test_name="$(basename "$test_path" .py)"
            test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
            export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
            export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
            export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
+            if (( ${#test_paths[@]} > 1 )); then
+              export ABXPKG_LIB_DIR="$RUNNER_TEMP/archivebox-lib/$test_slug"
+            fi
            mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
-            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
-              --basetemp="$RUNNER_TEMP/pytest/$test_slug"
+            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" --durations=20 \
+              --basetemp="$RUNNER_TEMP/pytest/$test_slug" \
+              --junitxml="$RUNNER_TEMP/test-results/$test_slug.xml" || test_status=1
          done
+          exit "$test_status"
+      - name: Save test results
+        if: always()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: test-results-${{ strategy.job-index }}
+          path: ${{ runner.temp }}/test-results/*.xml
+          if-no-files-found: error
+          retention-days: 14
+      - name: Save failed runtime diagnostics
+        if: failure()
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: runtime-logs-${{ strategy.job-index }}
+          path: ${{ runner.temp }}/pytest/**/*.log
+          if-no-files-found: ignore
+          retention-days: 14
@@ -77,11 +77,7 @@ archivebox run

 ## Verification

-Use targeted tests for focused work:
-
-```bash
-uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
-uv run prek run --all-files
-```
+Run the test suite with `uv run pytest archivebox/tests -x` and repository checks
+with `uv run prek run --all-files`. For focused work, select the relevant tests.

 Releases are published only by `.github/workflows/release.yml` after complete CI succeeds for the exact commit. Pushes to `main` prepare stable patch releases automatically; `dev` prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
@@ -125,6 +125,7 @@ FROM archivebox-runtime-base AS archivebox-builder
 WORKDIR "$CODE_DIR"
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
    --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
+    --mount=type=bind,source=uv.lock,target=/app/uv.lock \
    <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
 export UV_NO_CACHE=true
@@ -137,48 +138,14 @@ apt-get install -qq -y --no-install-recommends x11-utils
 # Do not rewrite inherited native libraries merely to strip them again.
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries

-mkdir -p /tmp/archivebox-uv-project
-/venv/bin/python - <<'PY'
-from pathlib import Path
-import json
-import re
-import urllib.request
-
-source = Path("/app/pyproject.toml")
-target = Path("/tmp/archivebox-uv-project/pyproject.toml")
-text = source.read_text()
-text = text.replace(
-    'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
-    'environments = ["sys_platform == \'linux\'"]',
-)
-
-# Docker builds need the just-published internal abx wheels immediately, but
-# PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
-# Docker-only dependency view from the version JSON so the published package
-# metadata stays normal while image builds remain resumable after a release.
-for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
-    match = re.search(
-        rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
-        text,
-    )
-    if not match:
-        continue
-    version = match.group("version")
-    with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
-        data = json.load(response)
-    wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
-    text = re.sub(
-        rf'"{re.escape(package)}(?:==|>=)[^"]+"',
-        f'"{package} @ {wheel_url}"',
-        text,
-        count=1,
-    )
-
-target.write_text(text)
-PY
-
+# The old Docker-only dependency rewrite predated this checkout's lockfile:
+# it fetched version JSON to avoid stale PyPI Simple listings. That extra API
+# can itself lag or return 503s. The verified lock already records exact public
+# artifact URLs and hashes, including extras and platform markers. Consume it
+# directly, retaining the freshness check instead of resolving a second graph.
 /usr/bin/uv sync \
-    --project /tmp/archivebox-uv-project \
+    --project /app \
+    --locked \
    --no-cache \
    --no-dev \
    --inexact \
@@ -209,7 +176,7 @@ rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
    /venv/lib/python3.*/site-packages/wheel* \
    /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
+rm -rf /var/lib/apt/lists/*
 EOF

 COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
@@ -291,7 +258,13 @@ RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=
        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install $ABX_DOCKER_PLUGINS \
    # pnpm includes musl variants that Debian's glibc runtime cannot use.
    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
-    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} +
+    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} + \
+    # pnpm's explicit store lives under ABXPKG_LIB_DIR, regardless of XDG_CACHE_HOME.
+    # Retaining its download copies made image size depend on whether the build
+    # preserved hardlinks to installed files. Remove the store in this layer;
+    # installed files retain their contents, and the offline check below proves
+    # the complete catalog still works without the disposable package payloads.
+    && rm -rf "$ABXPKG_LIB_DIR/cache/pnpm"
 RUN --network=none export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
    bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install $ABX_DOCKER_PLUGINS'
@@ -105,12 +105,11 @@ curl -fsSL 'https://get.archivebox.io' | bash
 </details>
 <br/>

-
 <div align="center" style="text-align: center">
 <br/><br/>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
 <br/><br/>
-<small><a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
+<small><a href="https://demo.archivebox.io">Demo</a> | <a href="https://archivebox.io/screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
 <br/>
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
 <br/><br/>
@@ -381,7 +380,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 > *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*

 <ul>
-<li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
+<li><a href="https://apps.truenas.com/catalog/archivebox_community/">TrueNAS</a></li>
 <li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
 <li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
 <li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
@@ -7,6 +7,11 @@ class APIConfig(AppConfig):
    name = "archivebox.api"
    label = "api"

+    def ready(self):
+        from archivebox.api.webhooks import register_delete_webhook_field_loader
+
+        register_delete_webhook_field_loader()
+

 def register_admin(admin_site):
    from archivebox.api.admin import register_admin
@@ -32,7 +32,7 @@ from archivebox.core.models import Snapshot, ArchiveResult, Tag
 from archivebox.core.permissions import public_snapshots_queryset
 from archivebox.api.auth import authenticated_user_from_request
 from archivebox.config.common import get_config
-from archivebox.core.routes_util import build_web_url
+from archivebox.core.routes_util import build_snapshot_detail_url, build_web_url
 from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url_length
 from archivebox.core.tag_util import (
    add_snapshot_counts,
@@ -800,7 +800,7 @@ def _snapshots_rss_response(
    )

    for snapshot in snapshots:
-        archived_url = build_web_url(f"/{snapshot.archive_path_from_db}", request=request)
+        archived_url = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
        tags = [tag.name for tag in snapshot.tags.all()]
        crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
        description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
@@ -182,6 +182,14 @@ def crawl_file(request: HttpRequest, crawl_id: str, path: str):
    if not file_path.is_file() or crawl_root not in file_path.parents:
        raise HttpError(404, "Crawl file not found")

+    if path == "chrome_screencast/latest.jpg" and request.GET.get("after"):
+        try:
+            started_at = float(request.GET["after"])
+        except ValueError:
+            raise HttpError(400, "Invalid screencast start time")
+        if file_path.stat().st_mtime < started_at:
+            raise HttpError(404, "No frame for this capture yet")
+
    response = FileResponse(file_path.open("rb"))
    response["Cache-Control"] = "no-store, no-cache, max-age=0, must-revalidate"
    response["Pragma"] = "no-cache"
@@ -5,12 +5,40 @@ from collections.abc import Callable
 import logging

 from django.db import transaction
+from django.db.models.signals import pre_delete
 from signal_webhooks.handlers import sync_task_handler


 logger = logging.getLogger(__name__)


+def load_deferred_fields_for_delete_webhook(sender, instance, using, **kwargs) -> None:
+    """Materialize deferred fields while DELETE webhook rows still exist.
+
+    Admin lists defer large fields, but signal_webhooks serializes all concrete
+    fields in post_delete. Loading one then would query an already-deleted row
+    and drop the event. Hydrate only deferred fields on webhook-enabled models
+    here, preserving the normal list-query optimization.
+    """
+    from signal_webhooks.handlers import find_hook_handler
+    from signal_webhooks.utils import reference_for_model
+
+    ref = reference_for_model(type(instance))
+    deferred_fields = instance.get_deferred_fields()
+    if not deferred_fields or find_hook_handler(ref, "DELETE") is None:
+        return
+
+    instance.refresh_from_db(using=using, fields=deferred_fields)
+
+
+def register_delete_webhook_field_loader() -> None:
+    pre_delete.connect(
+        load_deferred_fields_for_delete_webhook,
+        dispatch_uid="archivebox.load_deferred_fields_for_delete_webhook",
+        weak=False,
+    )
+
+
 def warning_error_handler(hook: Any, error: Exception | None) -> None:
    if error is not None:
        logger.warning("Outbound webhook %r failed: %s", hook.name, error)
@@ -282,7 +282,14 @@ class ModelWithOutputDir(ModelWithUUID):
            if path.is_symlink() or path.is_file():
                path.unlink(missing_ok=True)
            elif path.is_dir():
-                shutil.rmtree(path, ignore_errors=True)
+                shutil.rmtree(path)
+            # lstat also catches dangling symlinks and propagates mount and
+            # permission errors rather than mistaking them for absence.
+            try:
+                path.lstat()
+            except FileNotFoundError:
+                continue
+            raise OSError(f"Output path still exists after deletion: {path}")

    def schedule_delete_cleanup(self, *, using: str | None = None) -> None:
        """Capture output paths before DB deletion and remove them after commit."""
@@ -294,7 +301,9 @@ class ModelWithOutputDir(ModelWithUUID):
        if cls._delete_signal_registered:
            return

-        def schedule_output_dir_cleanup(sender, instance, using, **kwargs):
+        def schedule_output_dir_cleanup(sender, instance, using, origin=None, **kwargs):
+            if getattr(origin, "_output_files_deleted", False):
+                return
            if not isinstance(instance, ModelWithOutputDir):
                return
            instance.schedule_delete_cleanup(using=using)
@@ -174,8 +174,12 @@ def cli(ctx, help=False):
                archivebox_init(quick=True)
                os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
            check_data_folder()
-            if subcommand != "update":
-                check_migrations(auto_apply=True)
+            if subcommand != "update" and not check_migrations(auto_apply=True):
+                from archivebox.machine.models import Machine
+
+                # Host registration and config reconciliation belong to CLI
+                # startup; get_config() is also used by read-only HTTP views.
+                Machine.current()
        except Exception as e:
            STDERR.print(f"[red][X] Error setting up Django or checking data folder: {e}[/red]")
            if subcommand not in ("manage", "shell"):  # not all management commands need django to be setup beforehand
@@ -111,6 +111,11 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
        else:
            os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init

+    from archivebox.machine.models import Machine
+
+    # Registration requires the final schema and reconciles legacy file config.
+    Machine.current()
+
    from archivebox.misc.db import database_display_location, is_postgres

    assert database_exists()
@@ -87,9 +87,8 @@ def remove(
    # Delete one snapshot at a time. Each ``.delete()`` is its own short
    # Django-atomic block, so the writer lock is released between rows and
    # an in-flight daemon transaction can interleave instead of deadlocking.
-    # Filesystem cleanup for each row is scheduled via ``transaction.on_commit``
-    # in ``base_models/models.py`` and runs AFTER its row's tx commits — so
-    # rmtree doesn't hold the lock either.
+    # Snapshot.delete() removes and verifies filesystem outputs first, outside
+    # the short database delete transaction. Failed cleanup retains the row.
    #
    deleted_snapshot_pks = []
    timed_out = False
@@ -102,9 +102,19 @@ def _reconcile_directory(directory: Path) -> str:
    results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
    pending = []
    for result in results:
-        result_id = uuid.UUID(result["id"])
        if uuid.UUID(result["snapshot_id"]) != snapshot_id:
            raise ValueError("ArchiveResult belongs to another snapshot")
+        # Legacy JSON conversion produced records without IDs. Recover these
+        # by hook identity while preserving strict collision checks for IDs.
+        if not result.get("id"):
+            if not ArchiveResult.objects.filter(
+                snapshot_id=snapshot_id,
+                plugin=result["plugin"],
+                hook_name=result.get("hook_name", ""),
+            ).exists():
+                pending.append(result)
+            continue
+        result_id = uuid.UUID(result["id"])
        existing = ArchiveResult.objects.filter(pk=result_id).first()
        identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
        if existing:
@@ -154,7 +164,10 @@ def _reconcile_directory(directory: Path) -> str:
    # Each insert is durable. An interruption after the Snapshot insert is not
    # completion: the next pass still imports missing result IDs and tags.
    for result in pending:
-        snapshot._create_archive_result_if_missing(result, {}, preserve_identity=True)
+        snapshot._create_archive_result_if_missing(result, {}, preserve_identity=bool(result.get("id")))
+    if any(not result.get("id") for result in pending):
+        snapshot.hydrate_archiveresult_output_metadata(snapshot_dir=directory)
+        snapshot.write_index_jsonl(output_dir=directory)
    tags_before = set(snapshot.tags.values_list("name", flat=True))
    snapshot._merge_tags_from_index(record)
    tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
@@ -34,6 +34,25 @@ ConfigPayload = dict[str, object]
 PluginSchemaDocuments = dict[str, dict[str, Any]]
 LIVE_CONFIG_BASE_URL = "/admin/environment/config/"

+
+def _is_cookie_file_config_key(key: str) -> bool:
+    return key in {"COOKIES_FILE", "AUTH_STORAGE_FILE"} or key.endswith("_COOKIES_FILE")
+
+
+def _resolve_cookie_file_paths(config: dict[str, Any]) -> dict[str, Any]:
+    """Keep collection-relative auth paths valid after hooks change cwd."""
+    for key, value in config.items():
+        if _is_cookie_file_config_key(key):
+            if value is None or (isinstance(value, str) and not value.strip()):
+                config[key] = ""
+                continue
+            file_path = Path(value).expanduser()
+            if not file_path.is_absolute():
+                file_path = CONSTANTS.DATA_DIR / file_path
+            config[key] = str(file_path.absolute())
+    return config
+
+
 ###################### Config ##########################

 _STDOUT_CONSOLE = Console()
@@ -468,6 +487,11 @@ class ArchivingConfig(BaseConfigSet):
    COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
    AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})

+    @field_validator("COOKIES_FILE", "AUTH_STORAGE_FILE", mode="before")
+    @classmethod
+    def empty_cookie_file_is_unset(cls, value):
+        return None if isinstance(value, str) and not value.strip() else value
+
    URL_DENYLIST: str = Field(
        default=(
            r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$"
@@ -825,6 +849,8 @@ class ArchiveBoxBaseConfig(
        if runtime_overrides:
            config.update(normalize_runtime_config(runtime_overrides, json_safe=False))

+        _resolve_cookie_file_paths(config)
+
        # Hooks should only see concrete plugin-local flags, never the
        # ArchiveBox selectors used to derive them.
        config.pop("PLUGINS", None)
@@ -849,6 +875,13 @@ class ArchiveBoxBaseConfig(
            lib_dir = CONSTANTS.DATA_DIR / lib_dir
        self.ABXPKG_LIB_DIR = lib_dir.resolve()

+        cookie_paths = {key: getattr(self, key) for key in type(self).model_fields if _is_cookie_file_config_key(key)}
+        for key, value in _resolve_cookie_file_paths(cookie_paths).items():
+            if key in {"COOKIES_FILE", "AUTH_STORAGE_FILE"}:
+                setattr(self, key, Path(value) if value else None)
+            else:
+                setattr(self, key, value)
+
        return self

    @model_validator(mode="after")
@@ -1113,7 +1146,7 @@ def get_config(
            if apps.ready:
                from archivebox.machine.models import Machine

-                machine = Machine.current()
+                machine = Machine.current_readonly()
        except (DatabaseError, ImportError, RuntimeError, TypeError, ValueError):
            machine = None

@@ -63,7 +63,14 @@ def setup_django(check_db=False) -> None:
            os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
            if CONSTANTS.DATA_DIR.exists():
                for child in CONSTANTS.DATA_DIR.iterdir():
-                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
+                    try:
+                        os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
+                    except FileNotFoundError:
+                        # A concurrent SQLite transaction can remove its journal
+                        # between directory enumeration and chown. There is no
+                        # ownership left to fix; other permission errors must
+                        # still fail startup rather than being hidden.
+                        continue

    # Suppress the "database access during app initialization" warning
    # This warning can be triggered during django.setup() but is safe to ignore
@@ -85,23 +85,31 @@ def get_collection_id(DATA_DIR=DATA_DIR) -> str:


@cache
-def get_machine_id() -> str:
-    """Get a short, stable, unique ID for the current machine (e.g. abc45678)"""
+def get_machine_id(length: int = 8) -> str:
+    """Get a machine ID, using the network identity when no OS ID is available."""
+    import machineid

-    MACHINE_ID = "unknown"
    try:
-        import machineid
+        return machineid.hashed_id("archivebox")[:length]
+    except machineid.MachineIdNotFound:
+        import uuid

-        MACHINE_ID = machineid.hashed_id("archivebox")[:8]
-    except Exception:
-        try:
-            import uuid
-            import hashlib
+        import psutil

-            MACHINE_ID = hashlib.sha256(str(uuid.getnode()).encode()).hexdigest()[:8]
-        except Exception:
-            pass
-    return MACHINE_ID
+        node = uuid.getnode()
+        identity = str(node)
+        if node & (1 << 40):
+            # uuid.getnode() silently generates a per-process random value when
+            # tools such as ip/ifconfig are absent. Read interfaces directly so
+            # separate CLI processes agree even in a minimal container.
+            addresses = sorted(
+                address.address
+                for interface in psutil.net_if_addrs().values()
+                for address in interface
+                if address.family == psutil.AF_LINK and address.address not in ("", "00:00:00:00:00:00")
+            )
+            identity = str(int(addresses[0].replace(":", "").replace("-", ""), 16)) if addresses else f"hostname:{socket.gethostname()}"
+        return hashlib.sha256(identity.encode()).hexdigest()[:length]


@cache
@@ -150,9 +150,9 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:

    rows = {
        "Binary Name": [],
-        "Found Version": [],
+        "Recorded Version": [],
        "Provided By": [],
-        "Found Abspath": [],
+        "Recorded Abspath": [],
    }

    db_binaries = get_db_binaries_by_name()
@@ -160,18 +160,10 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:

    for name in all_binary_names:
        binary = db_binaries.get(name)
-        binary_is_valid = bool(binary and binary.is_valid)
-
        rows["Binary Name"].append(ItemLink(name, key=name))
-
-        if binary_is_valid:
-            rows["Found Version"].append(f"✅ {binary.version}" if binary.version else "✅ found")
-            rows["Provided By"].append(binary.binprovider or "-")
-            rows["Found Abspath"].append(binary.abspath or "-")
-        else:
-            rows["Found Version"].append("❌ missing")
-            rows["Provided By"].append("-")
-            rows["Found Abspath"].append("-")
+        rows["Recorded Version"].append(binary.version or "-")
+        rows["Provided By"].append(binary.binprovider or "-")
+        rows["Recorded Abspath"].append(binary.abspath or "-")

    return TableContext(
        title="Binaries",
@@ -2,11 +2,9 @@ __package__ = "archivebox.core"

 import html
 import json
-import os
 import shlex
 from functools import reduce
 from operator import and_
-from pathlib import Path
 from urllib.parse import quote

 from django.contrib import admin
@@ -24,7 +22,13 @@ from django.utils.text import smart_split

 from archivebox.base_models.admin import BaseModelAdmin
 from archivebox.core.models import ArchiveResult, Snapshot
-from archivebox.core.routes_util import build_snapshot_url
+from archivebox.core.routes_util import (
+    build_snapshot_detail_url,
+    build_snapshot_files_url,
+    build_snapshot_url,
+    build_snapshot_zip_url,
+    get_snapshot_output_anchor,
+)
 from archivebox.core.widgets import InlineTagEditorWidget
 from archivebox.machine.env_util import env_to_shell_exports
 from archivebox.misc.logging_util import printable_filesize
@@ -186,7 +190,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None, can_del
        if embed_path and result.status == "succeeded":
            output_link = build_snapshot_url(snapshot_id, embed_path, config=config)
        else:
-            output_link = build_snapshot_url(snapshot_id, "", config=config)
+            output_link = build_snapshot_detail_url(result.snapshot.archive_path_from_db, config=config)
        output_link_attr = html.escape(output_link, quote=True)

        # Get version - try cmd_version field
@@ -555,8 +559,8 @@ class ArchiveResultAdmin(BaseModelAdmin):
            try:
                queryset = self.get_queryset(request).filter(pk__in=selected)
                if not queryset.exists():
-                    snapshot = Snapshot.objects.only("id").filter(pk=request.GET.get("snapshot")).first()
-                    return redirect(build_snapshot_url(str(snapshot.id), "index.html", request=request) if snapshot else request.path)
+                    snapshot = Snapshot.objects.filter(pk=request.GET.get("snapshot")).first()
+                    return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request) if snapshot else request.path)
            except (ValidationError, ValueError):
                return redirect(request.path)
            return delete_selected(self, request, queryset)
@@ -568,7 +572,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
            request.GET.clear()
            request.META["QUERY_STRING"] = ""
            try:
-                handoff_snapshot = Snapshot.objects.only("id").filter(pk=handoff_snapshot).first()
+                handoff_snapshot = Snapshot.objects.filter(pk=handoff_snapshot).first()
            except (ValidationError, ValueError):
                handoff_snapshot = None
        saved_list_per_page = self.list_per_page
@@ -582,7 +586,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
                    pk__in=request.POST.getlist(ACTION_CHECKBOX_NAME),
                ).exists()
            ):
-                return redirect(build_snapshot_url(str(handoff_snapshot.id), "index.html", request=request))
+                return redirect(build_snapshot_detail_url(handoff_snapshot.archive_path_from_db, request=request))
            return response
        finally:
            self.list_per_page = saved_list_per_page
@@ -656,22 +660,40 @@ class ArchiveResultAdmin(BaseModelAdmin):

    def get_snapshot_view_url(self, result: ArchiveResult) -> str:
        request = self.request
-        return build_snapshot_url(str(result.snapshot_id), request=request, config=request.archivebox_config)
+        return build_snapshot_detail_url(
+            result.snapshot.archive_path_from_db,
+            request=request,
+            config=request.archivebox_config,
+        )

    def get_output_view_url(self, result: ArchiveResult) -> str:
        request = self.request
        config = request.archivebox_config
-        output_path = result.embed_path()
-        if not output_path:
-            output_path = result.plugin or ""
-        return build_snapshot_url(str(result.snapshot_id), output_path, request=request, config=config)
+        output_path = get_snapshot_output_anchor(result.plugin, result.embed_path() or "")
+        return build_snapshot_detail_url(
+            result.snapshot.archive_path_from_db,
+            output_path=output_path,
+            request=request,
+            config=config,
+        )

    def get_output_files_url(self, result: ArchiveResult) -> str:
        request = self.request
-        return f"{build_snapshot_url(str(result.snapshot_id), result.plugin, request=request, config=request.archivebox_config)}/?files=1"
+        return build_snapshot_files_url(
+            str(result.snapshot_id),
+            result.output_folder_path(),
+            request=request,
+            config=request.archivebox_config,
+        )

    def get_output_zip_url(self, result: ArchiveResult) -> str:
-        return f"{self.get_output_files_url(result)}&download=zip"
+        request = self.request
+        return build_snapshot_zip_url(
+            str(result.snapshot_id),
+            result.output_folder_path(),
+            request=request,
+            config=request.archivebox_config,
+        )

    @admin.display(description="Details", ordering="id")
    def details_link(self, result):
@@ -729,7 +751,11 @@ class ArchiveResultAdmin(BaseModelAdmin):
        request = self.request
        return format_html(
            '<a href="{}"><b><code>[{}]</code></b> &nbsp; {} &nbsp; {}</a><br/>',
-            build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
+            build_snapshot_detail_url(
+                result.snapshot.archive_path_from_db,
+                request=request,
+                config=request.archivebox_config,
+            ),
            snapshot_id[:8],
            result.snapshot.bookmarked_at.strftime("%Y-%m-%d %H:%M"),
            result.snapshot.url[:128],
@@ -827,11 +853,14 @@ class ArchiveResultAdmin(BaseModelAdmin):
        config = request.archivebox_config
        # Determine output link path - use embed_path() which checks output_files
        embed_path = result.embed_path()
-        output_path = embed_path if (result.status == "succeeded" and embed_path) else "index.html"
-        snapshot_id = str(result.snapshot_id)
+        output_url = (
+            build_snapshot_url(str(result.snapshot_id), embed_path, request=request, config=config)
+            if result.status == "succeeded" and embed_path
+            else build_snapshot_detail_url(result.snapshot.archive_path_from_db, request=request, config=config)
+        )
        return format_html(
            '<a href="{}" class="output-link">↗️</a><pre>{}</pre>',
-            build_snapshot_url(snapshot_id, output_path, request=request, config=config),
+            output_url,
            result.output_str_for_display(),
        )

@@ -858,46 +887,16 @@ class ArchiveResultAdmin(BaseModelAdmin):
        )

    def output_summary(self, result):
-        snapshot_dir = Path(result.snapshot.output_dir)
-        output_html = format_html(
-            '<pre style="display: inline-block">{}</pre><br/>',
-            result.output_str_for_display(),
-        )
-        snapshot_id = str(result.snapshot_id)
-        request = self.request
-        output_html += format_html(
-            '<a href="{}#all">See result files ...</a><br/><pre><code>',
-            build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
-        )
-        embed_path = result.embed_path() or ""
-        path_from_embed = snapshot_dir / (embed_path or "")
-        output_html += format_html(
-            '<i style="padding: 1px">{}</i><b style="padding-right: 20px">/</b><i>{}</i><br/><hr/>',
-            str(snapshot_dir),
-            str(embed_path),
-        )
-        if os.access(path_from_embed, os.R_OK):
-            root_dir = str(path_from_embed)
-        else:
-            root_dir = str(snapshot_dir)
-
-        for root, dirs, files in os.walk(root_dir):
-            depth = root.replace(root_dir, "").count(os.sep) + 1
-            if depth > 2:
-                continue
-            indent = " " * 4 * (depth)
-            output_html += format_html('<b style="padding: 1px">{}{}/</b><br/>', indent, os.path.basename(root))
-            indentation_str = " " * 4 * (depth + 1)
-            for filename in sorted(files):
-                is_hidden = filename.startswith(".")
-                output_html += format_html(
-                    '<span style="opacity: {}.2">{}{}</span><br/>',
-                    int(not is_hidden),
-                    indentation_str,
-                    filename.strip(),
-                )
-
-        return output_html + mark_safe("</code></pre>")
+        output_html = format_html('<pre style="display: inline-block">{}</pre>', result.output_str_for_display())
+        file_map = result.output_file_map()
+        paths = sorted({path for key in file_map if (path := result.output_file_path(key, output_file_map=file_map))})
+        if paths:
+            output_html += format_html(
+                '<br/><a href="{}">See result files ...</a><pre><code>{}</code></pre>',
+                self.get_output_files_url(result),
+                "\n".join(paths),
+            )
+        return output_html


 def register_admin(admin_site):
@@ -3,7 +3,6 @@ __package__ = "archivebox.core"
 import json
 from math import ceil
 from typing import Any
-from urllib.parse import quote

 from django.contrib import admin, messages
 from django.core.exceptions import PermissionDenied
@@ -16,7 +15,7 @@ from django.utils.html import format_html, format_html_join
 from django.utils.safestring import mark_safe
 from django.db.models import Q, Count, Exists, F, OuterRef, Prefetch
 from django import forms
-from django.template import Template, RequestContext
+from django.template import Context, Template, RequestContext
 from django.contrib.admin.helpers import ActionForm

 from archivebox.config.common import get_config
@@ -26,17 +25,24 @@ from archivebox.misc.logging_util import printable_filesize
 from archivebox.misc.serve_static import serve_static_with_byterange_support
 from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
 from archivebox.search.views import admin_snapshot_search_stream_view
-from archivebox.core.routes_util import build_snapshot_url, build_web_url
+from archivebox.core.routes_util import (
+    build_snapshot_detail_url,
+    build_snapshot_files_url,
+    build_snapshot_role_output_url,
+    build_snapshot_url,
+    build_snapshot_zip_url,
+    get_snapshot_output_anchor,
+)
 from archivebox.core.tag_util import get_or_create_tag
-from archivebox.core.templatetags.core_tags import snapshot_url_text
+from archivebox.core.templatetags.core_tags import snapshot_thumbnail, snapshot_url_text
 from archivebox.plugins.hooks import discover_hooks
-from archivebox.plugins.discovery import get_plugin_icon
+from archivebox.plugins.discovery import get_plugin_icon, get_snapshot_role_names
 from archivebox.plugins.output_groups import (
-    HIDDEN_ICON_PLUGINS,
    OUTPUT_GROUPS,
    display_plugin_name,
    output_group_for_plugin,
    order_output_plugins,
+    plugin_icon_is_hidden,
    plugin_output_sizes,
 )

@@ -45,8 +51,8 @@ from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
 from archivebox.core.models import Tag, Snapshot, ArchiveResult
 from archivebox.crawls.models import Crawl
 from archivebox.core.admin_archiveresults import render_archiveresults_list
-from archivebox.core.preview_util import PREVIEW_PLUGINS, snapshot_preview_candidates, render_snapshot_preview
 from archivebox.progressmonitor.views import progress_endpoint
+from archivebox.workers.models import RETRY_AT_MAX
 from archivebox.core.permissions import (
    PERMISSIONS_CHOICES,
    PERMISSIONS_META,
@@ -59,12 +65,6 @@ from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
 GLOBAL_CONTEXT = {}

 SNAPSHOT_PERMISSION_META = PERMISSIONS_META
-GRID_PREVIEW_OUTPUTS = {
-    ("screenshot", "screenshot.png"),
-    ("chrome_extension_screenshot", "screenshot-1.png"),
-    ("chrome_extension_screenshot", "screenshot.png"),
-    ("favicon", "favicon.ico"),
-}


 def _format_size_column(num_bytes: int) -> str:
@@ -132,11 +132,11 @@ class SnapshotStatusListFilter(admin.SimpleListFilter):
    parameter_name = "snapshot_status"

    def lookups(self, request, model_admin):
-        return Snapshot.StatusChoices.choices
+        return Snapshot._meta.get_field("status").choices

    def queryset(self, request, queryset):
        value = self.value()
-        if value in Snapshot.StatusChoices.values:
+        if value in dict(Snapshot._meta.get_field("status").choices):
            return queryset.filter(status=value)
        return queryset

@@ -284,7 +284,7 @@ class SnapshotChangeList(SearchResultsChangeList):
        rows = (
            ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
            .order_by("snapshot_id", "plugin")
-            .only("snapshot_id", "plugin", "status", "output_size", "output_files")
+            .only("snapshot_id", "plugin", "status", "output_str", "output_size", "output_files")
        )
        for result in rows.iterator(chunk_size=1000):
            if result.plugin in seen_plugins[result.snapshot_id]:
@@ -292,17 +292,22 @@ class SnapshotChangeList(SearchResultsChangeList):
            seen_plugins[result.snapshot_id].add(result.plugin)
            output_results_by_snapshot[result.snapshot_id].append(result)

-        response_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
-        for result in ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, plugin__in=PREVIEW_PLUGINS).only(
-            "snapshot_id",
-            "plugin",
-            "status",
-            "output_files",
+        card_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
+        for result in (
+            ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids)
+            .exclude(plugin="")
+            .only(
+                "snapshot_id",
+                "plugin",
+                "status",
+                "output_str",
+                "output_size",
+                "output_files",
+            )
        ):
-            response_results_by_snapshot[result.snapshot_id].append(result)
+            card_results_by_snapshot[result.snapshot_id].append(result)

        for obj in self.result_list:
-            obj._preview_results = response_results_by_snapshot[obj.pk]
            counts = status_counts_by_snapshot.get(obj.pk, {})
            total = int(counts.get("total") or 0)
            succeeded = int(counts.get("succeeded") or 0)
@@ -324,6 +329,7 @@ class SnapshotChangeList(SearchResultsChangeList):
                "is_sealed": obj.status not in (obj.StatusChoices.QUEUED, obj.StatusChoices.STARTED, obj.StatusChoices.PAUSED),
            }
            obj.__dict__["_admin_output_results"] = output_results_by_snapshot[obj.pk]
+            obj.__dict__["_snapshot_card_results"] = card_results_by_snapshot[obj.pk]

    def get_results(self, request):
        super().get_results(request)
@@ -595,14 +601,15 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):

    def get_snapshot_view_url(self, obj: Snapshot) -> str:
        request = self.request
-        return build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)
+        return build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=request.archivebox_config)

    def get_snapshot_files_url(self, obj: Snapshot) -> str:
        request = self.request
-        return f"{build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)}/?files=1"
+        return build_snapshot_files_url(str(obj.id), request=request, config=request.archivebox_config)

    def get_snapshot_zip_url(self, obj: Snapshot) -> str:
-        return f"{self.get_snapshot_files_url(obj)}&download=zip"
+        request = self.request
+        return build_snapshot_zip_url(str(obj.id), request=request, config=request.archivebox_config)

    def get_urls(self):
        urls = super().get_urls()
@@ -624,23 +631,17 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
        return custom_urls + urls

    def preview_view(self, request, object_id, plugin, filename):
-        """Serve grid thumbnails from the authenticated admin origin."""
-        if plugin not in PREVIEW_PLUGINS:
-            raise Http404("Unsupported snapshot preview")
+        """Serve a saved image output from the authenticated admin origin."""
        snapshot = get_object_or_404(Snapshot, pk=object_id)
        if not self.has_view_or_change_permission(request, snapshot):
            raise PermissionDenied
-        candidate = next(
-            (
-                candidate
-                for candidate in snapshot_preview_candidates(snapshot)
-                if candidate["plugin"] == plugin and candidate["filename"] == filename
-            ),
-            None,
-        )
-        if candidate is None:
-            raise Http404("Snapshot preview does not exist")
-        output_path = candidate["path"]
+        result = snapshot.archiveresult_set.filter(plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED).first()
+        try:
+            output_path = result.output_file_path(filename) if result else None
+        except (TypeError, ValueError, OverflowError):
+            output_path = None
+        if not output_path:
+            raise Http404("Snapshot image does not exist")
        request.archivebox_cache_policy = "private"
        response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
        if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
@@ -926,7 +927,13 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
    def status_info(self, obj):
        request = self.request
        config = request.archivebox_config
-        favicon_url = build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config)
+        favicon_url = build_snapshot_role_output_url(
+            obj,
+            "list_icon",
+            request=request,
+            config=config,
+            fallback_to_default=True,
+        )
        return format_html(
            """
            Archived: {} ({} files {}) &nbsp; &nbsp;
@@ -964,7 +971,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
        show_title = bool(title_raw) and title_normalized != "pending..." and title_normalized != url_normalized
        css_class = "fetched" if show_title else "pending"

-        detail_url = build_web_url(f"/{obj.archive_path_from_db}/index.html", request=request, config=config)
+        detail_url = build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config)
        title_html = ""
        if show_title:
            title_html = format_html(
@@ -974,22 +981,26 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
                urldecode(htmldecode(title_raw))[:128],
            )

-        preview = self._get_preview_data(obj) or {}
+        favicon = self._get_favicon_data(obj) or {}
        favicon_html = (
            format_html(
                '<img class="snapshot-url-favicon" src="{}" alt="" loading="lazy" decoding="async" onerror="this.hidden=true">',
-                preview["favicon_url"],
+                favicon["url"],
            )
-            if preview.get("favicon_url")
+            if favicon.get("url")
            else ""
        )
        return format_html(
            '<a class="snapshot-title-detail-hitbox" href="{}" aria-label="Open snapshot details"></a>'
            "{}"
+            "{}"
            '<div class="snapshot-title-url">'
            '<a class="snapshot-original-url" href="{}">{}<code style="user-select: all;">{}</code></a>'
            "</div>",
            detail_url,
+            format_html('<strong class="snapshot-delete-status">{}</strong>', "Deleting — verifying file removal")
+            if obj.status == Snapshot.DELETING_STATE
+            else "",
            title_html,
            url_raw or obj.url,
            favicon_html,
@@ -1020,42 +1031,47 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
        )
        return mark_safe(f'<span class="tags-inline-editor">{tags_html}</span>')

-    def _get_preview_data(self, obj):
-        candidates = snapshot_preview_candidates(obj)
-        if not candidates:
-            return None
-        urls = [
-            build_snapshot_url(str(obj.id), candidate["path"], request=self.request, config=self.request.archivebox_config)
-            for candidate in candidates
-        ]
-        favicons = [url for candidate, url in zip(candidates, urls) if candidate["kind"] == "favicon"]
-        return {
-            "img_url": urls[0],
-            "fallback_list": ",".join(urls[1:]),
-            "img_alt": "Favicon" if candidates[0]["kind"] == "favicon" else "Screenshot",
-            "preview_class": "favicon" if candidates[0]["kind"] == "favicon" else "screenshot",
-            "favicon_url": favicons[0] if favicons else "",
-            "favicon_fallback_list": ",".join(favicons[1:]),
-        }
+    def _get_favicon_data(self, obj):
+        list_icon_plugins = set(get_snapshot_role_names("list_icon"))
+        results = self._get_prefetched_results(obj)
+        if results is None:
+            results = obj.archiveresult_set.filter(
+                plugin__in=list_icon_plugins,
+                status=ArchiveResult.StatusChoices.SUCCEEDED,
+            )
+        urls = []
+        for result in results:
+            if result.plugin not in list_icon_plugins or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
+                continue
+            try:
+                output_path = result.embed_path()
+            except (TypeError, ValueError, OverflowError):
+                continue
+            if output_path:
+                urls.append(
+                    build_snapshot_url(str(obj.id), output_path, request=self.request, config=self.request.archivebox_config),
+                )
+        return {"url": urls[0], "fallbacks": ",".join(urls[1:])} if urls else None

-    def _render_preview(self, obj, width="100px", height="100px"):
-        return render_snapshot_preview(
+    def _render_thumbnail(self, obj, width="100px", height="100px"):
+        return snapshot_thumbnail(
+            Context({"request": self.request, "CONFIG": self.request.archivebox_config}),
            obj,
-            lambda candidate: reverse("admin:core_snapshot_preview", args=(obj.pk, candidate["plugin"], candidate["filename"])),
+            admin=True,
            width=width,
            height=height,
        )

    @admin.display(description="", empty_value="")
    def url_favicon(self, obj):
-        preview = self._get_preview_data(obj)
-        if not preview:
+        favicon = self._get_favicon_data(obj)
+        if not favicon:
            return ""

-        favicon_url = preview.get("favicon_url") or ""
+        favicon_url = favicon.get("url") or ""
        if not favicon_url:
            return ""
-        fallback_list = preview.get("favicon_fallback_list") or ""
+        fallback_list = favicon.get("fallbacks") or ""
        onerror_js = (
            "this.dataset.fallbacks && this.dataset.fallbacks.length ? "
            "(this.src=this.dataset.fallbacks.split(',').shift(), "
@@ -1078,8 +1094,8 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
    def preview_icon(self, obj):
        return format_html(
            '<a href="{}" title="Open snapshot details">{}</a>',
-            build_web_url(f"/{obj.archive_path_from_db}/index.html", request=self.request, config=self.request.archivebox_config),
-            self._render_preview(obj, height="38px"),
+            build_snapshot_detail_url(obj.archive_path_from_db, request=self.request, config=self.request.archivebox_config),
+            self._render_thumbnail(obj, height="38px"),
        )

    @admin.display(description=" ", empty_value="")
@@ -1091,8 +1107,8 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
        size_txt = printable_filesize(archive_size) if archive_size else "pending"
        screenshot_html = format_html(
            '<a href="{}" title="Open snapshot live view" style="display:block;flex:0 0 220px;">{}</a>',
-            build_web_url(f"/{obj.archive_path}", request=request, config=config),
-            self._render_preview(obj, width="220px", height="138px"),
+            build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config),
+            self._render_thumbnail(obj, width="220px", height="138px"),
        )

        return format_html(
@@ -1108,8 +1124,8 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
            "</div>",
            screenshot_html,
            size_txt,
-            build_web_url(f"/{obj.archive_path}", request=request, config=config),
-            obj.archive_path,
+            self.get_snapshot_files_url(obj),
+            obj.archive_path_from_db,
        )

    @admin.display(
@@ -1138,7 +1154,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
        request = self.request
        config = request.archivebox_config
        for result in sorted_results:
-            if result.plugin in HIDDEN_ICON_PLUGINS:
+            if plugin_icon_is_hidden(result.plugin):
                continue
            icon = mark_safe(get_plugin_icon(result.plugin))
            if not icon.strip():
@@ -1147,14 +1163,17 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
            if not embed_path:
                continue
            group_id = output_group_for_plugin(result.plugin)
-            output_relative_path = str(embed_path).lstrip("/")
-            anchor_path = output_relative_path if output_relative_path.startswith(f"{result.plugin}/") else result.plugin
-            output_path = quote(anchor_path, safe="/@-._~!$&'()*+,;=")
+            anchor_path = get_snapshot_output_anchor(result.plugin, embed_path)
            icons_by_group[group_id].append(
                {
                    "plugin": result.plugin,
                    "icon": icon,
-                    "href": build_web_url(f"/{obj.archive_path_from_db}/index.html#{output_path}", request=request, config=config),
+                    "href": build_snapshot_detail_url(
+                        obj.archive_path_from_db,
+                        output_path=anchor_path,
+                        request=request,
+                        config=config,
+                    ),
                },
            )

@@ -1223,8 +1242,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):

    @admin.display(empty_value="")
    def size(self, obj):
-        request = self.request
-        config = request.archivebox_config
        archive_size = self._get_progress_stats(obj)["output_size"] or 0
        if archive_size:
            size_txt = _format_size_column(archive_size)
@@ -1234,7 +1251,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
            return ""
        return format_html(
            '<a href="{}" title="View all files">{}</a>',
-            build_web_url(f"/{obj.archive_path}", request=request, config=config),
+            self.get_snapshot_files_url(obj),
            size_txt,
        )

@@ -1523,25 +1540,14 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):

    @admin.action(
        description="🗑️ Delete",
+        permissions=["delete"],
    )
    def delete_snapshots(self, request, queryset):
-        """Delete snapshots in a single transaction to avoid SQLite concurrency issues."""
-        from django.db import transaction
-
-        total = queryset.count()
-
-        # Get list of IDs to delete first (outside transaction)
-        ids_to_delete = list(queryset.values_list("pk", flat=True))
-
-        # Delete everything in a single atomic transaction
-        with transaction.atomic():
-            deleted_count, _ = Snapshot.objects.filter(pk__in=ids_to_delete).delete()
-
+        """Keep the rows until the runner removes and verifies their files."""
+        total = queryset.update(status=Snapshot.DELETING_STATE, retry_at=RETRY_AT_MAX, modified_at=timezone.now())
        messages.success(
            request,
-            mark_safe(
-                f"Successfully deleted {total} Snapshots ({deleted_count} total objects including related records). Don't forget to scrub URLs from import logs (data/sources) and error logs (data/logs) if needed.",
-            ),
+            f"Queued {total} snapshots for deletion. They remain visible as Deleting until ArchiveBox verifies their files are removed.",
        )

    @admin.action(
@@ -7,12 +7,13 @@ from django.contrib import admin, messages
 from django.contrib.admin.options import IS_POPUP_VAR
 from django.http import HttpRequest, HttpResponseRedirect
 from django.template.response import TemplateResponse
+from django.db.models import Prefetch
 from django.urls import reverse
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe

 from archivebox.base_models.admin import BaseModelAdmin
-from archivebox.core.models import SnapshotTag, Tag
+from archivebox.core.models import ArchiveResult, SnapshotTag, Tag
 from archivebox.core.tag_util import (
    TAG_HAS_SNAPSHOTS_CHOICES,
    TAG_SORT_CHOICES,
@@ -24,7 +25,8 @@ from archivebox.core.tag_util import (
    normalize_has_snapshots_filter,
    normalize_tag_sort,
 )
-from archivebox.core.routes_util import build_snapshot_url
+from archivebox.core.routes_util import build_snapshot_role_output_url
+from archivebox.plugins.discovery import get_snapshot_role_names


 class TagInline(admin.TabularInline):
@@ -191,7 +193,26 @@ class TagAdmin(BaseModelAdmin):

    @admin.display(description="Snapshots")
    def snapshots(self, tag: Tag):
-        snapshots = tag.snapshot_set.select_related("crawl__created_by").order_by("-downloaded_at", "-created_at", "-pk")[:10]
+        snapshots = (
+            tag.snapshot_set.select_related("crawl__created_by")
+            .prefetch_related(
+                Prefetch(
+                    "archiveresult_set",
+                    queryset=ArchiveResult.objects.filter(
+                        plugin__in=get_snapshot_role_names("list_icon"),
+                        status=ArchiveResult.StatusChoices.SUCCEEDED,
+                    ).only(
+                        "snapshot_id",
+                        "plugin",
+                        "status",
+                        "output_str",
+                        "output_files",
+                    ),
+                    to_attr="_list_icon_results",
+                ),
+            )
+            .order_by("-downloaded_at", "-created_at", "-pk")[:10]
+        )
        total_count = tag.snapshot_set.count()
        if not snapshots:
            return mark_safe(
@@ -214,7 +235,12 @@ class TagAdmin(BaseModelAdmin):
                </a>
                """,
                    reverse("admin:core_snapshot_change", args=[snapshot.pk]),
-                    build_snapshot_url(str(snapshot.pk), "favicon.ico"),
+                    build_snapshot_role_output_url(
+                        snapshot,
+                        "list_icon",
+                        archive_results=snapshot._list_icon_results,
+                        fallback_to_default=True,
+                    ),
                    title[:120],
                    snapshot.url[:120],
                ),
@@ -33,6 +33,7 @@ from archivebox.core.routes_util import (
    split_host_port,
 )
 from archivebox.core.views import OriginalDomainHostView, SnapshotHostView
+from archivebox.misc.serve_static import FAVICON_CACHE_CONTROL

 ADMIN_LOGIN_HINT_COOKIE = "archivebox_admin_logged_in"

@@ -154,8 +155,21 @@ def CacheControlMiddleware(get_response):
                    response.headers["Last-Modified"] = http_date(mtime)
                return response

-        # Keep artifact freshness/validators, but never let a shared cache store
-        # private or unlisted snapshots. Auth redirects and errors stay unstored.
+        if response.status_code in (200, 206, 304) and response.get("Cache-Control") == FAVICON_CACHE_CONTROL:
+            # The same raw favicon bytes are shared across authenticated viewers.
+            # Session middleware may have added Cookie to Vary during authorization.
+            vary = [
+                value.strip()
+                for value in response.get("Vary", "").split(",")
+                if value.strip().lower() not in {"", "cookie", "authorization"}
+            ]
+            if vary:
+                response["Vary"] = ", ".join(vary)
+            else:
+                response.headers.pop("Vary", None)
+            return response
+
+        # Other private/unlisted artifacts remain private; auth errors stay unstored.
        snapshot_policy = getattr(request, "archivebox_cache_policy", None)
        if snapshot_policy and response.status_code in (200, 206, 304):
            if not response.get("Cache-Control"):
@@ -100,11 +100,12 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
    if schema_editor.connection.vendor != "sqlite":
        return

-    from django.db import connection
+    from django.utils import timezone
    import uuid as uuid_lib
-    from datetime import datetime

+    connection = schema_editor.connection
    cursor = connection.cursor()
+    now = connection.ops.adapt_datetimefield_value(timezone.now())

    # Check if there are any snapshots without a crawl
    cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
@@ -125,14 +126,13 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
            INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
        """,
-            [datetime.now().isoformat()],
+            [now],
        )

    # Create a default crawl for migrated snapshots.
    # Depending on migration graph order, later crawls migrations may already
    # have removed output_dir by the time this data migration runs.
    crawl_id = uuid_lib.uuid4().hex
-    now = datetime.now().isoformat()
    cursor.execute("PRAGMA table_info(crawls_crawl)")
    crawl_columns = {row[1] for row in cursor.fetchall()}
    default_columns: list[str] = []
@@ -155,9 +155,9 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
            status, retry_at, created_by_id, schedule_id, config, persona_id
        ) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
                  'Auto-created crawl for migrated snapshots', {default_values_sql}
-                  'sealed', %s, 1, NULL, '{{}}', NULL)
+                  'sealed', NULL, 1, NULL, '{{}}', NULL)
    """,
-        [crawl_id, now, now, crawl_urls, now],
+        [crawl_id, now, now, crawl_urls],
    )

    # Assign all snapshots without a crawl to the default crawl
@@ -0,0 +1,60 @@
+"""Repair SQLite queue timestamps written as raw ISO strings by old migrations."""
+
+from django.db import migrations
+from django.utils.dateparse import parse_datetime
+
+
+def repair_legacy_retry_timestamps(apps, schema_editor):
+    connection = schema_editor.connection
+    if connection.vendor != "sqlite":
+        return
+
+    with connection.cursor() as cursor:
+        # 0024 created this Crawl already sealed, but left a due retry marker.
+        # Its orphaned snapshots have their own scheduler rows for maintenance.
+        cursor.execute(
+            """
+            UPDATE crawls_crawl SET retry_at = NULL
+            WHERE status = 'sealed'
+              AND label = 'Migrated from v0.7.2/v0.8.6'
+              AND notes = 'Auto-created crawl for migrated snapshots'
+              AND substr(retry_at, 11, 1) = 'T'
+            """,
+        )
+
+        # SQLite compares stored date strings lexically. A raw `T` timestamp
+        # from a legacy migration sorts differently from Django's space-separated
+        # parameter, so a due row can be selected but never claimed. Preserve
+        # the datetime Django already reads from each row while rewriting its
+        # storage representation through the active database adapter.
+        for table in ("crawls_crawl", "core_snapshot", "machine_binary", "machine_process"):
+            last_id = ""
+            while True:
+                cursor.execute(
+                    f"SELECT id, CAST(retry_at AS TEXT) FROM {table} WHERE id > %s AND substr(retry_at, 11, 1) = 'T' ORDER BY id LIMIT 500",
+                    [last_id],
+                )
+                rows = cursor.fetchall()
+                if not rows:
+                    break
+                for row_id, raw_retry_at in rows:
+                    try:
+                        parsed = parse_datetime(raw_retry_at)
+                    except ValueError:
+                        continue
+                    if parsed is None:
+                        continue
+                    normalized = connection.ops.adapt_datetimefield_value(parsed)
+                    cursor.execute(
+                        f"UPDATE {table} SET retry_at = %s WHERE id = %s AND retry_at = %s",
+                        [normalized, row_id, raw_retry_at],
+                    )
+                last_id = rows[-1][0]
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0054_snapshot_fs_version_index"),
+    ]
+
+    operations = [migrations.RunPython(repair_legacy_retry_timestamps, migrations.RunPython.noop)]
@@ -0,0 +1,25 @@
+"""Repair process links after both core's legacy copy and machine migrations."""
+
+from importlib import import_module
+
+from django.db import migrations
+
+
+repair_process_binary_iface_links = import_module(
+    "archivebox.machine.migrations.0020_repair_process_binary_iface_links",
+).repair_process_binary_iface_links
+
+
+class Migration(migrations.Migration):
+    atomic = False
+
+    dependencies = [
+        ("core", "0055_repair_legacy_retry_timestamps"),
+        ("machine", "0022_networkinterface_identity_without_mac"),
+    ]
+
+    operations = [
+        # The original repair can run before core.0027 creates legacy Process
+        # rows. Repair again after the copy, including already upgraded DBs.
+        migrations.RunPython(repair_process_binary_iface_links, migrations.RunPython.noop),
+    ]
@@ -0,0 +1,28 @@
+# Generated by Django 6.1 on 2026-09-30 03:32
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0057_repair_migrated_process_links"),
+    ]
+
+    operations = [
+        migrations.AlterField(
+            model_name="snapshot",
+            name="status",
+            field=models.CharField(
+                choices=[
+                    ("queued", "Queued"),
+                    ("started", "Started"),
+                    ("paused", "Paused"),
+                    ("sealed", "Sealed"),
+                    ("deleting", "Deleting"),
+                ],
+                db_index=True,
+                default="queued",
+                max_length=15,
+            ),
+        ),
+    ]
@@ -1,6 +1,7 @@
 __package__ = "archivebox.core"

 import json
+import logging
 import os
 import uuid
 from collections.abc import Iterable, Mapping, Sequence
@@ -175,6 +176,19 @@ class SnapshotTag(models.Model):
 class SnapshotQuerySet(models.QuerySet):
    """Custom QuerySet for Snapshot model with export methods that persist through .filter() etc."""

+    def delete(self):
+        # Finish the SELECT before filesystem work, so no read cursor or SQLite
+        # transaction is held while deleting remote archive payloads.
+        snapshots = list(self.select_related("crawl__created_by"))
+        total = 0
+        counts = {}
+        for snapshot in snapshots:
+            deleted, details = snapshot.delete(using=self.db)
+            total += deleted
+            for label, count in details.items():
+                counts[label] = counts.get(label, 0) + count
+        return total, counts
+
    def bulk_create(self, objs, *args, **kwargs):
        objs = list(objs)
        missing_crawl_ids = set()
@@ -466,21 +480,23 @@ class SnapshotQuerySet(models.QuerySet):

        from archivebox.config import VERSION
        from archivebox.config.version import get_COMMIT_HASH
+        from archivebox.plugins.discovery import get_snapshot_role_names

        config = get_config()

        template = "static_index.html" if with_headers else "minimal_index.html"
        snapshot_list = list(self.iterator(chunk_size=500))
+        list_icon_plugins = set(get_snapshot_role_names("list_icon"))
        manifest_records = []
        for snapshot in snapshot_list:
            # Unarchived snapshots still need an export destination before discovery.
            snapshot.output_dir.mkdir(parents=True, exist_ok=True)
-            outputs = snapshot.discover_outputs(include_filesystem_fallback=True)
-            output_paths = [str(output.get("path") or "") for output in outputs]
-            snapshot._public_preview_paths = [
-                path for preferred in ("screenshot/screenshot.png", "screenshot.png") for path in output_paths if path == preferred
+            # The list icon is part of the portable export too; an incomplete
+            # hash manifest must not hide a file that is still on disk.
+            outputs = snapshot.discover_outputs(include_filesystem_fallback=True, filesystem_index={})
+            snapshot._public_list_icon_paths = [
+                str(output.get("path") or "") for output in outputs if output.get("name") in list_icon_plugins
            ]
-            snapshot._public_favicon_paths = [path for path in output_paths if path in ("favicon/favicon.ico", "favicon.ico")]
            snapshot.write_html_details()
            if with_headers:
                # Use the same portable schema as the JSON export. Rendering
@@ -539,6 +555,7 @@ class SnapshotManager(models.Manager.from_queryset(SnapshotQuerySet)):  # ty: ig

 class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWithNotes, ModelWithHealthStats, ModelWithQueue):
    BROWSER_EXTENSION_UPLOAD_HOOK_NAME = "on_Snapshot__archivebox_browser_extension_upload"
+    DELETING_STATE = "deleting"

    id = CompactUUIDField(primary_key=True, default=uuid7, editable=False, unique=True)
    created_at = models.DateTimeField(default=timezone.now, db_index=True)
@@ -572,7 +589,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
    )
    retry_at = ModelWithQueue.RetryAtField(default=timezone.now)
    status = ModelWithQueue.StatusField(
-        choices=ModelWithQueue.StatusChoices,
+        choices=[*ModelWithQueue.StatusChoices.choices, (DELETING_STATE, "Deleting")],
        default=ModelWithQueue.StatusChoices.QUEUED,
    )
    config = models.JSONField(default=dict, null=False, blank=False, editable=True)
@@ -659,6 +676,49 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
    def __str__(self):
        return f"[{self.id}] {self.url[:64]}"

+    def output_paths_for_delete(self) -> tuple[Path, ...]:
+        current = self.get_storage_path_for_version(self.fs_version)
+        paths = [Path(self.output_dir), current, CONSTANTS.ARCHIVE_DIR / self.timestamp]
+        if self.fs_version in ("0.9.0", "0.9.1", "0.9.2", "0.9.3", "0.9.4", "1.0.0"):
+            paths.append(current.with_name(str(uuid.UUID(hex=self.id.hex))))
+        if self.crawl_id:
+            domain_dir = self.crawl.output_dir / CONSTANTS.SNAPSHOTS_DIR_NAME / self.extract_domain_from_url(self.url)
+            paths.extend((domain_dir / str(self.id), domain_dir / str(uuid.UUID(hex=self.id.hex))))
+        return tuple(dict.fromkeys(paths))
+
+    def delete(self, using=None, keep_parents=False):
+        from django.db import connections
+
+        using = using or self._state.db or "default"
+        if connections[using].in_atomic_block:
+            raise RuntimeError("Snapshot filesystem deletion must run outside a database transaction")
+        type(self).objects.using(using).filter(pk=self.pk).update(status=self.DELETING_STATE, retry_at=RETRY_AT_MAX)
+        self.status = self.DELETING_STATE
+        from archivebox.machine.models import Process
+
+        process_ids = self.archiveresult_set.filter(process__status=Process.StatusChoices.RUNNING).values_list("process_id", flat=True)
+        for process in list(Process.objects.using(using).filter(pk__in=process_ids)):
+            process.kill_tree()
+            if process.is_running:
+                raise OSError(f"Snapshot output writer is still running: {process.pk}")
+        paths = self.output_paths_for_delete()
+        self.delete_output_paths(paths)
+        # Cascaded ArchiveResults share this directory. Their normal post-commit
+        # cleanup would repeat filesystem work and update an already-deleted row.
+        self._output_files_deleted = True
+        return super().delete(using=using, keep_parents=keep_parents)
+
+    @classmethod
+    def delete_requested(cls, *, batch_size=100):
+        # Finish the bounded read before touching files: even an open SELECT
+        # cursor can hold a SQLite read lock for the duration of slow cleanup.
+        pending = list(cls.objects.filter(status=cls.DELETING_STATE).select_related("crawl__created_by")[:batch_size])
+        for snapshot in pending:
+            try:
+                snapshot.delete()
+            except OSError:
+                logging.getLogger(__name__).exception("Snapshot %s file deletion failed; keeping its database row", snapshot.pk)
+
    @classmethod
    def crawl_count_subquery(cls, *, status: str | None = None, outer_ref: str = "pk") -> QuerySet:
        """Return a scalar subquery counting Snapshots for one outer Crawl."""
@@ -893,7 +953,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
            return False
        if self.status == self.StatusChoices.QUEUED:
            return bool(self.url) and self.start_processing()
-        # abx-dl emits SnapshotCompletedEvent after the complete hook sequence;
+        # SnapshotCompletedEvent seals completed or explicitly aborted attempts;
+        # runner takeover preserves the active capture for its replacement.
        # ArchiveResult projection state never drives Snapshot completion.
        return False

@@ -1195,6 +1256,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW

        target_dir = self.get_storage_path_for_version(target)
        if target_dir.exists():
+            self.reconcile_with_index(output_dir=target_dir, update_existing_archive_results=False)
            self.hydrate_archiveresult_output_metadata(snapshot_dir=target_dir)
        if cleanup:
            old_dir, new_dir = cleanup
@@ -2374,21 +2436,52 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
            return calc_tags_str()
        return calc_tags_str()

+    def plugin_output_path(self, plugin: str, archive_results=None) -> str | None:
+        """Return the first declared successful output path for one plugin."""
+        from archivebox.plugins.discovery import get_archive_result_names
+
+        result_names = get_archive_result_names(plugin)
+        if archive_results is None:
+            for cache_name in ("_snapshot_card_results", "_admin_output_results", "_admin_archiveresults"):
+                if cache_name in self.__dict__:
+                    archive_results = self.__dict__[cache_name]
+                    break
+        if archive_results is None:
+            archive_results = self.__dict__.get("_prefetched_objects_cache", {}).get("archiveresult_set")
+        if archive_results is None:
+            archive_results = self.archiveresult_set.filter(plugin__in=result_names, status=ArchiveResult.StatusChoices.SUCCEEDED).only(
+                "plugin",
+                "status",
+                "output_str",
+                "output_files",
+            )
+
+        for result in archive_results:
+            if result.plugin not in result_names or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
+                continue
+            try:
+                output_path = result.embed_path()
+            except (TypeError, ValueError, OverflowError):
+                continue
+            if output_path:
+                return output_path
+        return None
+
    def icons(self, path: str | None = None, prefix: str = "/", quote_paths: bool = False) -> str:
        """Generate HTML icons showing which extractor plugins have succeeded for this snapshot"""
-        from urllib.parse import quote
-
        from django.utils.html import format_html

+        from archivebox.core.routes_util import build_snapshot_detail_path, build_snapshot_index_path, get_snapshot_output_anchor
+
        compact_icons = self.__dict__.get("_icons_compact", False)

        def calc_icons():
            from archivebox.plugins.output_groups import (
-                HIDDEN_ICON_PLUGINS,
                OUTPUT_GROUPS,
                display_plugin_name,
                output_group_for_plugin,
                order_output_plugins,
+                plugin_icon_is_hidden,
                plugin_output_sizes,
            )

@@ -2426,7 +2519,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                percent = int((completed / total * 100) if total > 0 else 0)
                icons_by_group: dict[str, list[dict[str, object]]] = {group_id: [] for group_id, _, _ in OUTPUT_GROUPS}
                for plugin in ordered_plugins:
-                    if plugin in HIDDEN_ICON_PLUGINS:
+                    if plugin_icon_is_hidden(plugin):
                        continue
                    icon = get_plugin_icon(plugin)
                    if str(icon).strip():
@@ -2520,11 +2613,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                    successful_icons,
                )

-            archive_path = path or self.archive_path
+            archive_path = path or self.archive_path_from_db
            icons_by_group: dict[str, list[dict[str, object]]] = {group_id: [] for group_id, _, _ in OUTPUT_GROUPS}

            for plugin in ordered_plugins:
-                if plugin in HIDDEN_ICON_PLUGINS:
+                if plugin_icon_is_hidden(plugin):
                    continue
                result = archive_results.get(plugin)
                existing = result is True or bool(
@@ -2550,14 +2643,16 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                ):
                    continue
                group_id = output_group_for_plugin(plugin)
-                output_relative_path = str(embed_path).lstrip("/")
-                anchor_path = output_relative_path if output_relative_path.startswith(f"{plugin}/") else plugin
-                quoted_output = quote(anchor_path, safe="/@-._~!$&'()*+,;=")
+                anchor_path = get_snapshot_output_anchor(plugin, embed_path)
                icons_by_group[group_id].append(
                    {
                        "plugin": plugin,
                        "icon": icon,
-                        "href": f"{prefix}{archive_path}/index.html#{quoted_output}",
+                        "href": (
+                            build_snapshot_index_path(archive_path, output_path=anchor_path, prefix=prefix)
+                            if quote_paths
+                            else build_snapshot_detail_path(archive_path, output_path=anchor_path, prefix=prefix)
+                        ),
                        "existing": bool(existing),
                    },
                )
@@ -2638,7 +2733,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
        return str(reverse_lazy("api-1:get_snapshot", args=[self.id]))

    def get_absolute_url(self):
-        return f"/{self.archive_path}"
+        from archivebox.core.routes_util import build_snapshot_detail_path
+
+        return build_snapshot_detail_path(self.archive_path_from_db)

    @cached_property
    def domain(self) -> str:
@@ -2832,30 +2929,49 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
    def legacy_archive_path(self) -> str:
        return f"{CONSTANTS.ARCHIVE_DIR_NAME}/{self.timestamp}"

+    @classmethod
+    def archive_path_for_values(
+        cls,
+        *,
+        fs_version: str,
+        timestamp: str,
+        snapshot_id,
+        url: str,
+        username: str = "web",
+        bookmarked_at=None,
+        created_at=None,
+    ) -> str:
+        """Build the public detail path without consulting the filesystem."""
+        legacy_path = f"{CONSTANTS.ARCHIVE_DIR_NAME}/{timestamp}"
+        if fs_version not in ("0.9.0", "0.9.1", "0.9.2", "0.9.3", "0.9.4", "1.0.0"):
+            return legacy_path
+
+        date_base = bookmarked_at or created_at
+        if not date_base:
+            return legacy_path
+
+        username = username if username and username != "system" else "web"
+        date_str = date_base.strftime("%Y%m%d")
+        domain = cls.extract_domain_from_url(url)
+        return f"{username}/{date_str}/{domain}/{snapshot_id}"
+
    @cached_property
    def archive_path_from_db(self) -> str:
        """Best-effort public URL path derived from DB fields only."""
-        if self.fs_version in ("0.7.0", "0.8.0"):
-            return self.legacy_archive_path
-
+        username = "web"
        if self.fs_version in ("0.9.0", "0.9.1", "0.9.2", "0.9.3", "0.9.4", "1.0.0"):
-            username = "web"
            crawl = self.crawl if self.crawl_id else None
            if crawl and crawl.created_by_id:
                username = crawl.created_by.username
-            if username == "system":
-                username = "web"
-
-            date_base = self.bookmarked_at or self.created_at
-            if date_base:
-                date_str = date_base.strftime("%Y%m%d")
-            else:
-                return self.legacy_archive_path
-
-            domain = self.extract_domain_from_url(self.url)
-            return f"{username}/{date_str}/{domain}/{self.id}"
-
-        return self.legacy_archive_path
+        return self.archive_path_for_values(
+            fs_version=self.fs_version,
+            timestamp=self.timestamp,
+            snapshot_id=self.id,
+            url=self.url,
+            username=username,
+            bookmarked_at=self.bookmarked_at,
+            created_at=self.created_at,
+        )

    @cached_property
    def url_path(self) -> str:
@@ -3529,8 +3645,6 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
            entry = Path(path)
            if len(entry.parts) != 1 or entry.name in ("index.html", "index.json", "favicon.ico"):
                continue
-            if filesystem_index is not None and entry.name == "index.jsonl":
-                continue
            if entry.suffix.lstrip(".").lower() not in embeddable_exts or entry.stem in seen:
                continue
            outputs.append(
@@ -3562,7 +3676,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW

    def to_dict(self, extended: bool = False, static_export: bool = False) -> dict[str, Any]:
        """Convert Snapshot to a dictionary (replacement for Link._asdict())"""
-        from archivebox.core.routes_util import build_snapshot_url
+        from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_index_path

        archive_size = self.archive_size

@@ -3591,8 +3705,12 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
            "extension": self.extension,
            "is_static": self.is_static,
            "is_archived": self.is_archived,
-            "archive_path": self.static_archive_path if static_export else self.archive_path,
-            "archive_url": f"./{self.static_archive_path}/index.html" if static_export else build_snapshot_url(str(self.id), "index.html"),
+            "archive_path": self.static_archive_path if static_export else self.archive_path_from_db,
+            "archive_url": (
+                build_snapshot_index_path(self.static_archive_path, prefix="./")
+                if static_export
+                else build_snapshot_detail_url(self.archive_path_from_db)
+            ),
            "output_dir": self.output_dir,
            "link_dir": self.output_dir,  # backwards compatibility alias
            "archive_size": archive_size,
@@ -3637,7 +3755,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
        from archivebox.core.widgets import TagEditorWidget
        from archivebox.misc.logging_util import printable_filesize
        from archivebox.progressmonitor.views import progress_endpoint
-        from archivebox.plugins.output_groups import OUTPUT_GROUPS, order_snapshot_outputs, plugin_output_sizes
+        from archivebox.plugins.output_groups import OUTPUT_GROUPS, order_snapshot_outputs, plugin_card_is_hidden, plugin_output_sizes

        runtime_config = get_request_config(request) if request is not None else get_config()
        self._runtime_config = runtime_config
@@ -3649,8 +3767,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
        self.__dict__["num_outputs_cached"] = sum(result.status == ArchiveResult.StatusChoices.SUCCEEDED for result in archive_results)
        self.__dict__["num_failures_cached"] = sum(result.status == ArchiveResult.StatusChoices.FAILED for result in archive_results)

-        filesystem_index = {} if discover_files else None
-        hidden_card_plugins = {"archivedotorg", "favicon"}
+        # Static exports must reflect files that still exist even when the
+        # saved hash manifest is incomplete or stale.
+        filesystem_index = {} if discover_files or static_export_dir is not None else None
        outputs = [
            output
            for output in self.discover_outputs(
@@ -3659,7 +3778,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                filesystem_index=filesystem_index,
            )
            if (output.get("size") or 0) > 0
-            and output.get("name") not in hidden_card_plugins
+            and not plugin_card_is_hidden(str(output.get("name") or ""))
            # Files left by failed/started/noresults hooks remain browsable,
            # but only the DB can authorize a successful live output card.
            and (
@@ -3678,6 +3797,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                return not output_path.is_absolute() and ".." not in output_path.parts and (static_export_dir / output_path).exists()

            outputs = [output for output in outputs if static_output_exists(output)]
+            # A snapshot's index.jsonl is a manifest, not another captured
+            # output card. Preserve it as the preview fallback only when the
+            # manifest is the snapshot's sole available output.
+            if any(output.get("path") != "index.jsonl" for output in outputs):
+                outputs = [output for output in outputs if output.get("path") != "index.jsonl"]
        outputs_by_name: dict[str, dict[str, Any]] = {}
        result_ids_by_name: dict[str, list[str]] = {}
        for output in outputs:
@@ -3691,13 +3815,17 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
        output_sizes = plugin_output_sizes(archive_results)
        for name, output in outputs_by_name.items():
            output["result_ids"] = ",".join(result_ids_by_name.get(name, ()))
+            output["folder_path"] = (
+                output["result"].output_folder_path() if output.get("result") is not None else str(Path(str(output["path"])).parent)
+            )
+            if output["folder_path"] == ".":
+                output["folder_path"] = ""
            if name in output_sizes:
                output["size"] = output_sizes[name]

        hash_index = filesystem_index if filesystem_index is not None else (self.hashes_index if static_export_dir is not None else {})
        loose_items, failed_items = self.get_detail_page_auxiliary_items(
            outputs,
-            hidden_card_plugins=hidden_card_plugins,
            archive_results=archive_results,
            hashes_index=hash_index,
        )
@@ -3836,13 +3964,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
    def get_detail_page_auxiliary_items(
        self,
        outputs: list[dict] | None = None,
-        hidden_card_plugins: set[str] | None = None,
        archive_results: list["ArchiveResult"] | None = None,
        hashes_index: dict[str, dict[str, Any]] | None = None,
    ) -> tuple[list[dict[str, object]], list[dict[str, object]]]:
        if outputs is None:
            outputs = self.discover_outputs(include_filesystem_fallback=True)
-        hidden_card_plugins = hidden_card_plugins or set()
        accounted_entries: set[str] = set()
        for output in outputs:
            output_name = str(output.get("name") or "")
@@ -4348,7 +4474,9 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
        return str(reverse_lazy("api-1:get_archiveresult", args=[self.id]))

    def get_absolute_url(self):
-        return f"/{self.snapshot.archive_path}/{self.plugin}"
+        from archivebox.core.routes_util import build_snapshot_detail_path
+
+        return build_snapshot_detail_path(self.snapshot.archive_path_from_db, output_path=self.plugin)

    @property
    def is_paused(self) -> bool:
@@ -4368,11 +4496,37 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
            return 0

    def output_file_map(self) -> dict[str, dict[str, Any]]:
+        # List views opt into this cache on their read-only, request-local rows.
+        # Runner-owned results must still reflect metadata changed during a hook.
+        if "_render_output_file_map" in self.__dict__:
+            return self._render_output_file_map
        return self._normalize_output_files(self.output_files)

    def output_file_paths(self) -> list[str]:
        return list(self.output_file_map().keys())

+    def output_file_path(
+        self,
+        path: str,
+        output_file_map: dict[str, dict[str, Any]] | None = None,
+    ) -> str | None:
+        """Resolve an output_files key to its snapshot-relative path."""
+        requested_path = str(path or "")
+        if not requested_path or requested_path.startswith("/") or ".." in Path(requested_path).parts:
+            return None
+
+        output_file_map = output_file_map if output_file_map is not None else self.output_file_map()
+        plugin_relative_path = requested_path.removeprefix(f"{self.plugin}/")
+        plugin_path = f"{self.plugin}/{plugin_relative_path}"
+        for candidate in dict.fromkeys((requested_path, plugin_relative_path, plugin_path)):
+            file_info = output_file_map.get(candidate)
+            if not isinstance(file_info, dict):
+                continue
+            if file_info.get("root_relative"):
+                return candidate.removeprefix(f"{self.plugin}/")
+            return candidate if candidate.startswith(f"{self.plugin}/") else f"{self.plugin}/{candidate}"
+        return None
+
    def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True, full_scan: bool = False) -> bool:
        from abx_dl.output_files import OutputManifest, output_file_from_path

@@ -4500,11 +4654,20 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
        output_file_map: dict[str, dict[str, Any]] | None = None,
    ) -> str | None:
        ignored = {"stdout.log", "stderr.log", "hook.pid", "listener.pid"}
-        candidates = [
-            path
-            for path in output_file_paths
-            if Path(path).name not in ignored and Path(path).suffix.lower() not in (".pid", ".log", ".sh")
-        ]
+        # Responses/wget manifests can contain thousands of paths. Parse each
+        # once instead of rebuilding Path objects for every preferred filename
+        # and extension group, each time a list row selects a preview.
+        candidates = []
+        names = {}
+        extensions = {}
+        for path in output_file_paths:
+            parsed_path = Path(path)
+            name, extension = parsed_path.name, parsed_path.suffix.lower()
+            if name in ignored or extension in (".pid", ".log", ".sh"):
+                continue
+            candidates.append(path)
+            names.setdefault(name.lower(), path)
+            extensions[path] = extension
        if not candidates:
            return None

@@ -4527,32 +4690,19 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
            "article.json",
        ]
        for preferred_name in preferred_names:
-            for candidate in candidates:
-                if Path(candidate).name.lower() == preferred_name:
-                    return candidate
+            if preferred_name in names:
+                return names[preferred_name]

-        plugin_lower = (plugin_name or "").lower()
-        if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl"):
-            # yt-dlp commonly emits a thumbnail plus several media formats.
-            # Prefer something browsers can play directly; otherwise cards can
-            # select a large MKV or thumbnail that the plugin player cannot use.
-            ext_groups = (
-                (".mp4", ".webm", ".m4v", ".ogv"),
-                (".mp3", ".m4a", ".aac", ".opus", ".ogg", ".wav", ".flac"),
-                (".mkv", ".mov", ".avi", ".flv", ".wmv", ".mpg", ".mpeg", ".ts", ".m2ts", ".mts", ".3gp", ".3g2"),
-                (".html", ".htm", ".mhtml", ".mht", ".pdf"),
-                (".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".ico"),
-                (".json", ".jsonl", ".txt", ".md", ".csv", ".tsv", ".srt", ".vtt"),
-            )
-        else:
-            ext_groups = (
-                (".html", ".htm", ".mhtml", ".mht", ".pdf"),
-                (".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".ico"),
-                (".json", ".jsonl", ".txt", ".md", ".csv", ".tsv"),
-                (".mp4", ".webm", ".mp3", ".opus", ".ogg", ".wav"),
-            )
+        from archivebox.plugins.discovery import get_plugin_output_extension_preference
+
+        ext_groups = get_plugin_output_extension_preference(plugin_name or "") or (
+            (".html", ".htm", ".mhtml", ".mht", ".pdf"),
+            (".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".ico"),
+            (".json", ".jsonl", ".txt", ".md", ".csv", ".tsv"),
+            (".mp4", ".webm", ".mp3", ".opus", ".ogg", ".wav"),
+        )
        for ext_group in ext_groups:
-            group_candidates = [candidate for candidate in candidates if Path(candidate).suffix.lower() in ext_group]
+            group_candidates = [candidate for candidate in candidates if extensions[candidate] in ext_group]
            if group_candidates:
                return max(
                    group_candidates,
@@ -4595,10 +4745,6 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
    def embed_path_db(self, output_file_map: dict[str, dict[str, Any]] | None = None) -> str | None:
        output_file_map = output_file_map if output_file_map is not None else self.output_file_map()

-        def is_root_relative(path: str) -> bool:
-            metadata = output_file_map.get(path) or {}
-            return bool(isinstance(metadata, dict) and metadata.get("root_relative"))
-
        output_str = "" if "output_str" in self.get_deferred_fields() else self.output_str
        if output_str:
            raw_output = str(output_str).strip()
@@ -4629,23 +4775,16 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
                    # payload storage while rendering a live snapshot page.
                    return candidates[0] if ".." not in output_path.parts else None

-                if raw_output in output_file_map and is_root_relative(raw_output):
-                    return raw_output
-
                for relative_path in candidates:
-                    plugin_relative = relative_path.removeprefix(f"{self.plugin}/")
-                    if relative_path in output_file_map:
-                        return f"{self.plugin}/{relative_path}" if not relative_path.startswith(f"{self.plugin}/") else relative_path
-                    if plugin_relative in output_file_map:
-                        return f"{self.plugin}/{plugin_relative}"
+                    resolved_path = self.output_file_path(relative_path, output_file_map=output_file_map)
+                    if resolved_path:
+                        return resolved_path

        output_file_paths = list(output_file_map.keys())
        if output_file_paths:
            fallback_path = self._fallback_output_file_path(output_file_paths, self.plugin, output_file_map)
            if fallback_path:
-                if is_root_relative(fallback_path):
-                    return fallback_path
-                return f"{self.plugin}/{fallback_path}"
+                return self.output_file_path(fallback_path, output_file_map=output_file_map)

        return None

@@ -4658,6 +4797,14 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
        """
        return self.embed_path_db()

+    def output_folder_path(self) -> str:
+        """Return the physical folder for this result's files, using saved metadata."""
+        output_path = self.embed_path_db()
+        if output_path and not output_path.startswith(f"{self.plugin}/"):
+            parent = Path(output_path).parent.as_posix()
+            return "" if parent == "." else parent
+        return self.plugin
+
    @property
    def output_dir_name(self) -> str:
        return self.plugin
@@ -1,114 +0,0 @@
-__package__ = "archivebox.core"
-
-import hashlib
-import json
-from pathlib import PurePosixPath
-from urllib.parse import urlsplit
-
-from django.template.loader import render_to_string
-from django.utils.safestring import mark_safe
-
-EXTENSION_SCREENSHOT_PLUGIN = "chrome_extension_screenshot"
-PREVIEW_PLUGINS = ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "seo", "responses", "favicon")
-IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".gif", ".avif", ".svg", ".ico"}
-
-
-def positive_int(value):
-    try:
-        return max(0, int(value))
-    except (ValueError, TypeError, OverflowError):
-        return 0
-
-
-def preview_candidates(results):
-    """Select only saved outputs. Never read files or fetch remote metadata."""
-    candidates = []
-    for result in results:
-        files = result.output_files
-        if not isinstance(files, dict):
-            continue
-        for path, info in files.items():
-            if not isinstance(path, str) or not isinstance(info, dict) or not positive_int(info.get("size")):
-                continue
-            if path.startswith("/") or ".." in PurePosixPath(path).parts:
-                continue
-            plugin = result.plugin
-            relative = path.removeprefix(f"{plugin}/")
-            output_path = path if info.get("root_relative") else f"{plugin}/{path}"
-            kind = "image"
-            rank = None
-            if plugin == "screenshot" and relative == "screenshot.png":
-                rank = 0
-            elif plugin == EXTENSION_SCREENSHOT_PLUGIN and relative in ("screenshot-1.png", "screenshot.png"):
-                rank = 1 if relative == "screenshot-1.png" else 2
-            elif (
-                plugin == "seo"
-                and PurePosixPath(relative).stem == "featured-image"
-                and PurePosixPath(relative).suffix.lower() in IMAGE_EXTENSIONS
-            ):
-                rank = 3
-            elif plugin == "responses":
-                mime = str(info.get("mimetype") or info.get("mime_type") or "")
-                if not (relative.startswith("image/") or (relative.startswith("all/") and mime.startswith("image/"))):
-                    continue
-                width, height = positive_int(info.get("width")), positive_int(info.get("height"))
-                if width and height and (min(width, height) <= 2 or max(width, height) <= 64):
-                    continue
-                if PurePosixPath(relative).suffix.lower() == ".ico":
-                    continue
-                rank = 4
-                kind = "response"
-            elif plugin == "favicon" and relative == "favicon.ico":
-                rank = 5
-                kind = "favicon"
-            if rank is None:
-                continue
-            width, height = positive_int(info.get("width")), positive_int(info.get("height"))
-            # Prefer usable known dimensions; byte size remains the legacy fallback.
-            usable = bool(width and height and 0.25 <= width / height <= 4)
-            score = (rank, -int(usable), -(width * height if usable else 0), -positive_int(info.get("size")), output_path)
-            candidates.append((score, {"path": output_path, "kind": kind, "plugin": plugin, "filename": path}))
-    seen = set()
-    selected = []
-    for _score, candidate in sorted(candidates, key=lambda item: item[0]):
-        if candidate["path"] not in seen:
-            selected.append(candidate)
-            seen.add(candidate["path"])
-    return selected
-
-
-def snapshot_preview_candidates(snapshot):
-    if "_preview_candidates" not in snapshot.__dict__:
-        results = snapshot.__dict__.get("_preview_results")
-        if results is None:
-            results = snapshot.__dict__.get("_prefetched_objects_cache", {}).get("archiveresult_set")
-        if results is None:
-            results = snapshot.archiveresult_set.filter(plugin__in=PREVIEW_PLUGINS).only("plugin", "output_files")
-        snapshot._preview_candidates = preview_candidates(results)
-    return snapshot._preview_candidates
-
-
-def render_snapshot_preview(snapshot, url_for_candidate, width="100px", height="100px"):
-    candidates = []
-    for candidate in snapshot_preview_candidates(snapshot):
-        candidates.append({"url": url_for_candidate(candidate), "kind": candidate["kind"]})
-    try:
-        host = urlsplit(snapshot.url).hostname or "Archive"
-    except ValueError:
-        host = "Archive"
-    host = host.removeprefix("www.")
-    hue = int(hashlib.sha256(host.encode()).hexdigest()[:6], 16) % 360
-    return mark_safe(
-        render_to_string(
-            "core/snapshot_thumbnail.html",
-            {
-                "candidates": json.dumps(candidates),
-                "first": candidates[0] if candidates else None,
-                "host": host,
-                "initial": host[:1].upper(),
-                "hue": hue,
-                "width": width,
-                "height": height,
-            },
-        ),
-    )
@@ -74,8 +74,9 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
        **snapshot_filter,
    ).update(retry_at=now, modified_at=now)

+    # Find rare uploads through hook_name's index before checking their parents;
+    # a correlated upload probe otherwise visits every sealed snapshot.
    extension_upload_results = ArchiveResult.objects.filter(
-        snapshot_id=OuterRef("pk"),
        hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
    )
    server_results = ArchiveResult.objects.filter(snapshot_id=OuterRef("pk")).exclude(
@@ -83,15 +84,15 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
    )
    extension_only_snapshots = (
        Snapshot.objects.filter(
+            id__in=extension_upload_results.values("snapshot_id"),
            status=Snapshot.StatusChoices.SEALED,
            crawl__status__in=[Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED],
            **snapshot_filter,
        )
        .annotate(
-            has_extension_upload=Exists(extension_upload_results),
            has_server_result=Exists(server_results),
        )
-        .filter(has_extension_upload=True, has_server_result=False)
+        .filter(has_server_result=False)
    )
    # Older browser-extension uploads could win a race with runner startup:
    # their successful external rows made the fresh Snapshot look finished
@@ -2,7 +2,7 @@ from __future__ import annotations

 import re
 from typing import Any
-from urllib.parse import urlparse
+from urllib.parse import parse_qsl, quote, urlencode, urlparse, urlsplit, urlunsplit

 from django.utils.http import url_has_allowed_host_and_scheme

@@ -11,6 +11,8 @@ from archivebox.config.common import get_config, get_request_config
 _SNAPSHOT_ID_RE = re.compile(r"^[0-9a-fA-F-]{8,36}$")
 _SNAPSHOT_SUBDOMAIN_RE = re.compile(r"^snap-(?P<suffix>[0-9a-fA-F]{12})$")
 _ROLE_SUBDOMAIN_LABELS = ("admin", "web", "api")
+_URL_PATH_SAFE = "/@-._~!$&'()*+,;="
+_URL_FRAGMENT_SAFE = "/@-._~!$&'()*+,;="


 def split_host_port(host: str) -> tuple[str, str | None]:
@@ -60,13 +62,6 @@ def _csrf_trusted_origins(config) -> list[str]:
    return seen


-def _allowed_hosts(config) -> set[str]:
-    raw = (config.ALLOWED_HOSTS or "").strip()
-    if not raw:
-        return set()
-    return {entry.strip().lower() for entry in raw.split(",") if entry.strip() and entry.strip() != "*"}
-
-
 def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
    """Pick a single CSRF_TRUSTED_ORIGINS entry to act as the implicit BASE_URL.

@@ -93,11 +88,6 @@ def get_listen_host(config: dict[str, Any] | None = None, **config_kwargs: Any)
    return (config.BIND_ADDR or "").strip()


-def get_listen_parts(config: dict[str, Any] | None = None, **config_kwargs: Any) -> tuple[str, str | None]:
-    config = config or get_config(**config_kwargs)
-    return split_host_port(get_listen_host(config=config))
-
-
 def _with_port(host: str, port: str | None) -> str:
    return f"{host}:{port}" if port else host

@@ -146,7 +136,7 @@ def canonical_base_host_for_request(request_host: str) -> str:

 def _root_host_from_listen(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
    config = config or get_config(**config_kwargs)
-    listen_host, listen_port = get_listen_parts(config=config)
+    listen_host, listen_port = split_host_port(get_listen_host(config=config))
    root_host = "archivebox.localhost" if _is_local_bind_host(listen_host) else listen_host
    return _with_port(root_host, listen_port) if root_host else ""

@@ -380,11 +370,6 @@ def get_snapshot_base_url(snapshot_id: str, request=None, config: dict[str, Any]
    )


-def get_original_base_url(domain: str, request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
-    return _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
-
-
 def build_admin_url(path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
    return _build_url(get_admin_base_url(request, config=config, **config_kwargs), path)

@@ -394,11 +379,109 @@ def build_web_url(path: str = "", request=None, config: dict[str, Any] | None =


 def build_snapshot_url(snapshot_id: str, path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    return _build_url(get_snapshot_base_url(snapshot_id, request=request, config=config, **config_kwargs), path)
+    # `path` is a filesystem-relative output path, not an already-encoded URL.
+    # In particular, wget can save literal `%3A` in filenames; emitting that
+    # unchanged makes the browser request `:`, which cannot match the file.
+    output_path = quote(str(path).lstrip("/"), safe=_URL_PATH_SAFE)
+    return _build_url(get_snapshot_base_url(snapshot_id, request=request, config=config, **config_kwargs), output_path)
+
+
+def build_snapshot_detail_path(archive_path: str, output_path: str = "", prefix: str = "/") -> str:
+    """Build the live Django snapshot-detail path and optional output anchor."""
+    normalized_archive_path = quote(str(archive_path or "").strip("/"), safe=_URL_PATH_SAFE)
+    detail_path = f"{prefix}{normalized_archive_path}"
+    normalized_output_path = str(output_path or "").lstrip("#/")
+    if not normalized_output_path:
+        return detail_path
+    fragment = quote(normalized_output_path, safe=_URL_FRAGMENT_SAFE)
+    return f"{detail_path}#{fragment}"
+
+
+def build_snapshot_index_path(archive_path: str, output_path: str = "", prefix: str = "/") -> str:
+    """Build a path to the literal exported ``index.html`` snapshot file."""
+    normalized_archive_path = quote(str(archive_path or "").strip("/"), safe=_URL_PATH_SAFE)
+    index_path = f"{prefix}{normalized_archive_path}/index.html"
+    normalized_output_path = str(output_path or "").lstrip("#/")
+    if not normalized_output_path:
+        return index_path
+    fragment = quote(normalized_output_path, safe=_URL_FRAGMENT_SAFE)
+    return f"{index_path}#{fragment}"
+
+
+def get_snapshot_output_anchor(plugin: str, output_path: str) -> str:
+    """Map root-relative legacy outputs to the plugin card that displays them."""
+    normalized_output_path = str(output_path or "").lstrip("/")
+    return normalized_output_path if normalized_output_path.startswith(f"{plugin}/") else plugin
+
+
+def build_snapshot_detail_url(
+    archive_path: str,
+    output_path: str = "",
+    request=None,
+    config: dict[str, Any] | None = None,
+    **config_kwargs: Any,
+) -> str:
+    """Build a request-aware URL for trusted snapshot UI on the web origin."""
+    return build_web_url(
+        build_snapshot_detail_path(archive_path, output_path=output_path),
+        request=request,
+        config=config,
+        **config_kwargs,
+    )
+
+
+def build_snapshot_role_output_url(
+    snapshot,
+    role: str,
+    request=None,
+    config: dict[str, Any] | None = None,
+    archive_results=None,
+    fallback_to_default: bool = False,
+    **config_kwargs: Any,
+) -> str:
+    """Build the first available output URL for a plugin-owned snapshot role."""
+    from archivebox.plugins.discovery import get_plugin_default_output_path, get_snapshot_role_names
+
+    for plugin in get_snapshot_role_names(role):
+        output_path = snapshot.plugin_output_path(plugin, archive_results=archive_results)
+        if not output_path and fallback_to_default:
+            output_path = get_plugin_default_output_path(plugin)
+        if output_path:
+            return build_snapshot_url(str(snapshot.id), output_path, request=request, config=config, **config_kwargs)
+    return ""
+
+
+def build_snapshot_files_url(
+    snapshot_id: str,
+    path: str = "",
+    request=None,
+    config: dict[str, Any] | None = None,
+    **config_kwargs: Any,
+) -> str:
+    """Build a request-aware directory-browser URL on the isolated replay origin."""
+    directory_url = build_snapshot_url(snapshot_id, path, request=request, config=config, **config_kwargs)
+    parts = urlsplit(directory_url)
+    normalized_directory_path = f"{parts.path.rstrip('/')}/"
+    directory_url = urlunsplit((parts.scheme, parts.netloc, normalized_directory_path, parts.query, parts.fragment))
+    return _replace_url_query_params(directory_url, files="1")
+
+
+def build_snapshot_zip_url(
+    snapshot_id: str,
+    path: str = "",
+    request=None,
+    config: dict[str, Any] | None = None,
+    **config_kwargs: Any,
+) -> str:
+    """Build a request-aware ZIP download URL for a snapshot directory."""
+    files_url = build_snapshot_files_url(snapshot_id, path, request=request, config=config, **config_kwargs)
+    return _replace_url_query_params(files_url, download="zip")


 def build_original_url(domain: str, path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    return _build_url(get_original_base_url(domain, request=request, config=config, **config_kwargs), path)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
+    base_url = _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
+    return _build_url(base_url, path)


 def _build_url(base_url: str, path: str) -> str:
@@ -409,3 +492,11 @@ def _build_url(base_url: str, path: str) -> str:
    if not path:
        return base_url
    return f"{base_url}{path if path.startswith('/') else f'/{path}'}"
+
+
+def _replace_url_query_params(url: str, **params: str) -> str:
+    parts = urlsplit(url)
+    replacement_keys = set(params)
+    query = [(key, value) for key, value in parse_qsl(parts.query, keep_blank_values=True) if key not in replacement_keys]
+    query.extend((key, value) for key, value in params.items())
+    return urlunsplit((parts.scheme, parts.netloc, parts.path, urlencode(query), parts.fragment))
@@ -1,3 +1,28 @@
+"""Shared foreground signal and child-process cleanup contracts.
+
+Shutdown intent and process exit are separate: takeover must preserve the
+replacement owner's work, while a user abort must not later resume the crawl.
+Callers own that policy; this module preserves the received signal and stops
+only the process objects they explicitly supply.
+
+Cooperative boundaries are essential, not just quieter exception handling.
+A real Linux CI regression interrupted Popen.poll() with SIGINT and stranded
+its wait lock. SIGKILL stopped the child (it became a zombie), but Popen.wait()
+still timed out. Raising a different exception or waiting longer cannot repair
+that interrupted critical section. Polling loops must record the signal first
+and honor it between operations; async owners may cancel their own task while
+their loop is running. Once exiting, the last-resort process cleanup must stay
+synchronous: the original loop may be closed, and creating a replacement loop
+cannot recover tasks or resources owned by it.
+
+Keep immediate signal notices and repeated-signal force exit independent of
+the event bus: it may already be draining or closed during shutdown. Resumable
+pause/retry/skip belongs to the existing crawl controller and must not set
+sticky command-shutdown state. See foreground_shutdown_signals below for both
+cooperative and legacy blocking callers; do not change all callers' policy at
+once without supplying their interruption boundaries.
+"""
+
 from __future__ import annotations

 import os
@@ -23,7 +48,7 @@ _active_shutdown_state: ShutdownSignalState | None = None


 def raise_if_shutdown_requested() -> None:
-    """Let long foreground loops honor a signal even if Python ignored it once."""
+    """Honor sticky shutdown intent at a caller-owned interruption boundary."""

    if _active_shutdown_state and _active_shutdown_state.signal_name:
        raise KeyboardInterrupt
@@ -48,14 +73,37 @@ def wait_popen_and_kill_children(
    timeout: float,
    kill_timeout: float = 2.0,
 ) -> None:
-    """Wait for a Popen parent and then hard-kill any surviving descendants."""
+    """Reap our own Popen child and stop its explicitly captured descendants.
+
+    A dead child still needs its parent to reap it. Keep wait failures visible;
+    bypassing Popen's lock would hide the interrupted-owner bug described above.
+    Descendants must still be cleaned up when that wait fails.
+    """

    try:
-        proc.wait(timeout=timeout)
-    except subprocess.TimeoutExpired:
-        proc.kill()
-        proc.wait(timeout=kill_timeout)
-    kill_remaining_processes(children, timeout=kill_timeout)
+        try:
+            proc.wait(timeout=timeout)
+        except subprocess.TimeoutExpired:
+            proc.kill()
+            proc.wait(timeout=kill_timeout)
+    except subprocess.TimeoutExpired as err:
+        try:
+            state = psutil.Process(proc.pid).as_dict(attrs=["pid", "ppid", "status", "num_threads", "cpu_times"])
+        except psutil.NoSuchProcess:
+            state = {"pid": proc.pid, "status": "exited"}
+        err.add_note(f"Process state after SIGKILL: {state}")
+        # A locked Popen wait can time out even after the OS process exits.
+        # Inspect CPython's lock only for diagnostics; never release it here.
+        waitpid_lock = getattr(proc, "_waitpid_lock", None)
+        err.add_note(
+            f"Popen state: returncode={proc.returncode}, "
+            f"waitpid_lock_locked={waitpid_lock.locked() if waitpid_lock is not None else None}, "
+            f"threads={[thread.name for thread in threading.enumerate()]}",
+        )
+        raise
+    finally:
+        # A parent that cannot be reaped must not leave its descendants running.
+        kill_remaining_processes(children, timeout=kill_timeout)


 def wait_psutil_and_kill_children(
@@ -104,11 +152,14 @@ def foreground_shutdown_signals(
    interrupt_handlers: dict[signal.Signals, Callable[[], None]] | None = None,
    raise_on_first_signal: bool = True,
 ) -> Iterator[ShutdownSignalState]:
-    """Install foreground signal handlers that print an immediate exit notice.
+    """Keep shutdown intent visible until the owning foreground command unwinds.

-    Some log-tail loops intentionally swallow KeyboardInterrupt so that callers
-    can centralize cleanup in finally blocks. The handler writes the signal name
-    immediately, then raises KeyboardInterrupt to break out of the blocking read.
+    Signal state was added because inner loops can swallow KeyboardInterrupt
+    while their caller still needs to stop its children. Cooperative callers use
+    raise_on_first_signal=False and check raise_if_shutdown_requested() between
+    operations, or cancel their owned async task through on_signal. Raising from
+    the handler can otherwise interrupt library locks or partially sent I/O.
+    Blocking callers without a cooperative boundary retain immediate exceptions.
    """

    global _active_shutdown_state
@@ -6,7 +6,7 @@ from django.db.models import QuerySet
 def snapshot_status_values() -> tuple[str, ...]:
    from archivebox.core.models import Snapshot

-    return tuple(Snapshot.StatusChoices.values)
+    return tuple(value for value, _label in Snapshot._meta.get_field("status").choices)


 def normalize_snapshot_status(status: str | None) -> str | None:
@@ -6,15 +6,16 @@ from typing import Any
 from urllib.parse import unquote

 from django.contrib.auth.models import User
-from django.db.models import Count, Exists, F, OuterRef, QuerySet
+from django.db.models import Count, Exists, F, OuterRef, Prefetch, QuerySet
 from django.db.models.functions import Lower
 from django.http import HttpRequest
 from django.urls import reverse

 from archivebox.config.common import get_config
+from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag, Tag
+from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_role_output_url
 from archivebox.misc.util import sanitize_html_text
-from archivebox.core.routes_util import build_snapshot_url, build_web_url
-from archivebox.core.models import Snapshot, SnapshotTag, Tag
+from archivebox.plugins.discovery import get_snapshot_role_names


 TAG_SNAPSHOT_PREVIEW_LIMIT = 10
@@ -202,9 +203,16 @@ def _build_snapshot_preview(snapshot: Snapshot, request: HttpRequest | None = No
        "id": str(snapshot.pk),
        "title": _display_snapshot_title(snapshot),
        "url": snapshot.url,
-        "favicon_url": build_snapshot_url(str(snapshot.pk), "favicon.ico", request=request, config=config),
+        "favicon_url": build_snapshot_role_output_url(
+            snapshot,
+            "list_icon",
+            request=request,
+            config=config,
+            archive_results=snapshot._list_icon_results,
+            fallback_to_default=True,
+        ),
        "admin_url": reverse("admin:core_snapshot_change", args=[snapshot.pk]),
-        "archive_url": build_web_url(f"/{snapshot.archive_path_from_db}/index.html", request=request, config=config),
+        "archive_url": build_snapshot_detail_url(snapshot.archive_path_from_db, request=request, config=config),
        "downloaded_at": snapshot.downloaded_at.isoformat() if snapshot.downloaded_at else None,
    }

@@ -221,6 +229,22 @@ def _build_snapshot_preview_map(
    snapshot_tags = (
        SnapshotTag.objects.filter(tag_id__in=tag_ids)
        .select_related("snapshot__crawl__created_by")
+        .prefetch_related(
+            Prefetch(
+                "snapshot__archiveresult_set",
+                queryset=ArchiveResult.objects.filter(
+                    plugin__in=get_snapshot_role_names("list_icon"),
+                    status=ArchiveResult.StatusChoices.SUCCEEDED,
+                ).only(
+                    "snapshot_id",
+                    "plugin",
+                    "status",
+                    "output_str",
+                    "output_files",
+                ),
+                to_attr="_list_icon_results",
+            ),
+        )
        .order_by(
            "tag_id",
            F("snapshot__downloaded_at").desc(nulls_last=True),
@@ -1,3 +1,4 @@
+import hashlib
 import logging
 import math
 import os
@@ -7,9 +8,7 @@ from pathlib import Path
 from typing import Any
 from urllib.parse import quote, urlparse

-from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
 from django import template
-from django.urls import reverse
 from django.utils import timezone
 from django.utils.html import escape, format_html
 from django.utils.safestring import mark_safe
@@ -17,8 +16,11 @@ from django.utils.text import Truncator

 from archivebox.config import CONSTANTS
 from archivebox.core.routes_util import (
-    build_web_url,
+    build_snapshot_detail_url,
+    build_snapshot_files_url,
+    build_snapshot_role_output_url,
    build_snapshot_url,
+    build_snapshot_zip_url,
    get_admin_base_url,
    get_snapshot_base_url,
    get_web_base_url,
@@ -29,9 +31,18 @@ from archivebox.plugins.discovery import (
    get_plugin_icon,
    get_plugin_name,
    get_plugin_template,
+    plugin_card_is_interactive,
 )

 register = template.Library()
+register.filter("plugin_card_is_interactive", plugin_card_is_interactive)
+
+# Preview cards are clipped windows onto their content, never scroll areas.
+# Apply inside the document too: parent overflow cannot constrain an iframe.
+CARD_OVERFLOW_STYLE = (
+    "<style>html,body{height:100%!important;min-height:0!important}"
+    "html,body,body *{overflow:clip!important;scrollbar-width:none!important}</style>"
+)


@register.filter
@@ -222,7 +233,13 @@ def _list_media_files(result, *, include_filesystem_fallback=True) -> list[dict]
                candidates.append((rel_path, size))

    for rel_path, size in candidates:
-        href = str(Path(result.plugin) / rel_path)
+        if callable(getattr(result, "output_file_path", None)):
+            href = result.output_file_path(str(rel_path), output_file_map=output_files)
+        else:
+            metadata = output_files.get(str(rel_path)) or {}
+            href = str(rel_path) if metadata.get("root_relative") else str(Path(result.plugin) / rel_path)
+        if not href:
+            continue
        suffix = rel_path.suffix.lower()
        media_type = "video" if suffix in _VIDEO_FILE_EXTS else "audio"
        media_files.append(
@@ -301,12 +318,14 @@ def _snapshot_base_url_for_context(context, snapshot) -> str:

 def _snapshot_url_for_context(context, snapshot, path: str = "") -> str:
    if not context.get("STATIC_EXPORT"):
-        return build_snapshot_url(
+        path_part, separator, query = str(path or "").partition("?")
+        url = build_snapshot_url(
            str(_snapshot_id(snapshot)),
-            path,
+            path_part,
            request=context.get("request"),
            config=context.get("CONFIG"),
        )
+        return f"{url}?{query}" if separator else url

    base_url = _static_snapshot_base_url(context, snapshot)
    raw_path = str(path or "")
@@ -318,14 +337,6 @@ def _snapshot_url_for_context(context, snapshot, path: str = "") -> str:
    return f"{base_url.rstrip('/')}/{quoted_path}{suffix}"


-def _build_snapshot_files_url(snapshot_id: str, request=None, config=None, base_url: str | None = None) -> str:
-    return (
-        f"{base_url.rstrip('/')}/index.jsonl"
-        if base_url
-        else build_snapshot_url(str(snapshot_id), "/?files=1", request=request, config=config)
-    )
-
-
 def _build_snapshot_preview_url(
    snapshot_id: str,
    path: str = "",
@@ -337,20 +348,25 @@ def _build_snapshot_preview_url(
    if path == "about:blank":
        return path
    if _is_root_snapshot_output_path(path):
-        return _build_snapshot_files_url(snapshot_id, request=request, config=config, base_url=base_url)
+        return (
+            f"{base_url.rstrip('/')}/index.jsonl"
+            if base_url
+            else build_snapshot_files_url(str(snapshot_id), request=request, config=config)
+        )
    if base_url:
        path_part, separator, query = str(path).lstrip("/").partition("?")
        url = f"{base_url.rstrip('/')}/{quote(path_part, safe=_STATIC_URL_SAFE)}"
        if separator:
            url = f"{url}?{query}"
    else:
-        url = build_snapshot_url(str(snapshot_id), path, request=request, config=config)
+        path_part, separator, query = str(path).partition("?")
+        url = build_snapshot_url(str(snapshot_id), path_part, request=request, config=config)
+        if separator:
+            url = f"{url}?{query}"
    path_parts = Path(path).parts
    plugin = get_plugin_name(plugin) if plugin else (path_parts[0] if len(path_parts) > 1 else "")
    has_plugin_preview = bool(plugin and get_plugin_template(plugin, "full", fallback=False))
-    if not (
-        _is_text_preview_path(path) or _is_image_preview_path(path) or has_plugin_preview or is_archivewebpage_replay_target(path or "")
-    ):
+    if not (_is_text_preview_path(path) or _is_image_preview_path(path) or has_plugin_preview):
        return url
    separator = "&" if "?" in url else "?"
    return f"{url}{separator}preview=1"
@@ -610,8 +626,8 @@ def snapshot_detail_url(context, snapshot) -> str:
    """Keep the trusted snapshot detail page on web.*, not the replay host."""
    if context.get("STATIC_EXPORT"):
        return _snapshot_url_for_context(context, snapshot, "index.html")
-    return build_web_url(
-        f"{snapshot.get_absolute_url().rstrip('/')}/index.html",
+    return build_snapshot_detail_url(
+        snapshot.archive_path_from_db,
        request=context.get("request"),
        config=context.get("CONFIG"),
    )
@@ -623,81 +639,121 @@ def snapshot_url(context, snapshot, path: str = "") -> str:


@register.simple_tag(takes_context=True)
-def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
-    snapshot_id = str(_snapshot_id(snapshot))
-    url_cache = snapshot.__dict__.setdefault("_snapshot_archiveresult_url_cache", {})
+def snapshot_files_url(context, snapshot, path: str = "") -> str:
+    if context.get("STATIC_EXPORT"):
+        return _snapshot_url_for_context(context, snapshot, "index.jsonl")
+    return build_snapshot_files_url(
+        str(_snapshot_id(snapshot)),
+        path,
+        request=context.get("request"),
+        config=context.get("CONFIG"),
+    )
+
+
+@register.simple_tag(takes_context=True)
+def snapshot_zip_url(context, snapshot, path: str = "") -> str:
+    if context.get("STATIC_EXPORT"):
+        return ""
+    return build_snapshot_zip_url(
+        str(_snapshot_id(snapshot)),
+        path,
+        request=context.get("request"),
+        config=context.get("CONFIG"),
+    )
+
+
+@register.simple_tag(takes_context=True)
+def snapshot_role_output_url(context, snapshot, role: str, fallback_to_default: bool = False) -> str:
+    url_cache = snapshot.__dict__.setdefault("_snapshot_role_output_url_cache", {})
    cache_key = (
-        plugin,
-        filename,
+        role,
        bool(context.get("STATIC_EXPORT")),
        str(context.get("STATIC_EXPORT_DIR") or ""),
+        fallback_to_default,
    )
    if cache_key in url_cache:
        return url_cache[cache_key]

-    results = None
-    if "_admin_archiveresults" in snapshot.__dict__:
-        results = snapshot.__dict__["_admin_archiveresults"]
-    elif "archiveresult_set" in snapshot.__dict__.get("_prefetched_objects_cache", {}):
-        results = snapshot.archiveresult_set.all()
-    elif "_snapshot_archiveresult_url_results" in snapshot.__dict__:
-        results = snapshot.__dict__["_snapshot_archiveresult_url_results"]
+    from archivebox.plugins.discovery import get_snapshot_role_names

-    if results is None:
-        from archivebox.core.models import ArchiveResult
-
-        results = list(
-            ArchiveResult.objects.filter(
-                snapshot_id=snapshot_id,
-                plugin__in=("screenshot", "chrome_extension_screenshot", "favicon"),
-                status=ArchiveResult.StatusChoices.SUCCEEDED,
-            ).only(
-                "plugin",
-                "status",
-                "output_files",
-            ),
+    if context.get("STATIC_EXPORT"):
+        output_path = None
+        for plugin in get_snapshot_role_names(role):
+            output_path = snapshot.plugin_output_path(plugin)
+            if output_path:
+                break
+        url_cache[cache_key] = _snapshot_url_for_context(context, snapshot, output_path) if output_path else ""
+    else:
+        url_cache[cache_key] = build_snapshot_role_output_url(
+            snapshot,
+            role,
+            request=context.get("request"),
+            config=context.get("CONFIG"),
+            fallback_to_default=fallback_to_default,
        )
-        snapshot.__dict__["_snapshot_archiveresult_url_results"] = results
-
-    for result in results:
-        if result.plugin != plugin or result.status != "succeeded":
-            continue
-        output_files = result.output_files or {}
-        file_info = output_files.get(filename)
-        if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
-            continue
-        output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
-        url_cache[cache_key] = _snapshot_url_for_context(context, snapshot, output_path)
-        return url_cache[cache_key]
-
-    url_cache[cache_key] = ""
-    return ""
+    return url_cache[cache_key]


@register.simple_tag(takes_context=True)
-def admin_snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
-    if not snapshot_archiveresult_url(context, snapshot, plugin, filename):
+def snapshot_result_file_url(context, snapshot, result, path: str) -> str:
+    """Build a saved-file URL from one ArchiveResult declaration."""
+    try:
+        output_path = result.output_file_path(path)
+    except (AttributeError, TypeError, ValueError, OverflowError):
        return ""
-    return reverse("admin:core_snapshot_preview", args=(snapshot.pk, plugin, filename))
+    if not output_path:
+        return ""
+    return _snapshot_url_for_context(context, snapshot, output_path)


@register.simple_tag(takes_context=True)
 def snapshot_thumbnail(context, snapshot, admin=False, width="100px", height="100px"):
-    from archivebox.core.preview_util import render_snapshot_preview
+    from django.template.loader import render_to_string

-    if admin:
-        url_for_candidate = lambda candidate: reverse(
-            "admin:core_snapshot_preview",
-            args=(snapshot.pk, candidate["plugin"], candidate["filename"]),
+    from archivebox.plugins.output_groups import snapshot_thumbnail_result
+
+    results = snapshot.__dict__.get("_snapshot_card_results")
+    if results is None:
+        results = snapshot.__dict__.get("_admin_output_results")
+    if results is None:
+        results = snapshot.__dict__.get("_admin_archiveresults")
+    if results is None:
+        results = snapshot.__dict__.get("_prefetched_objects_cache", {}).get("archiveresult_set")
+    if results is None:
+        results = snapshot.archiveresult_set.exclude(plugin="").only(
+            "id",
+            "snapshot_id",
+            "plugin",
+            "status",
+            "output_str",
+            "output_files",
+            "output_size",
        )
-    else:
-        url_for_candidate = lambda candidate: _snapshot_url_for_context(context, snapshot, candidate["path"])
-    return render_snapshot_preview(snapshot, url_for_candidate, width=width, height=height)
+
+    try:
+        host = urlparse(snapshot.url).hostname or "Archive"
+    except ValueError:
+        host = "Archive"
+    host = host.removeprefix("www.")
+    hue = int(hashlib.sha256(host.encode()).hexdigest()[:6], 16) % 360
+    return mark_safe(
+        render_to_string(
+            "core/snapshot_thumbnail.html",
+            {
+                **context.flatten(),
+                "card_result": snapshot_thumbnail_result(results),
+                "host": host,
+                "initial": host[:1].upper(),
+                "hue": hue,
+                "width": width,
+                "height": height,
+            },
+        ),
+    )


@register.simple_tag(takes_context=True)
 def snapshot_index_row(context, link) -> str:
-    snapshot_base = _snapshot_base_url_for_context(context, link)
    detail_url = snapshot_detail_url(context, link)

    status = getattr(link, "status", None) or "unknown"
@@ -719,7 +775,9 @@ def snapshot_index_row(context, link) -> str:
    title = unescape(getattr(link, "title", "") or "")
    title_text = title or url
    tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
-    tag_html = "".join(f'<span class="snapshot-tag">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag)
+    tag_html = "".join(
+        f'<span class="snapshot-tag" title="{escape(tag)}">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag
+    )
    if tag_html:
        tag_cell = f'<span class="snapshot-tags">{tag_html}</span>'
    else:
@@ -734,14 +792,14 @@ def snapshot_index_row(context, link) -> str:
    archive_size = int(getattr(link, "archive_size", 0) or 0)
    size_cell = file_size(archive_size) if archive_size else '<span class="empty-value">...</span>'
    output_plural = "" if num_outputs == 1 else "s"
-    files_url = _snapshot_url_for_context(context, link, "index.jsonl") if context.get("STATIC_EXPORT") else f"{snapshot_base}/?files=1"
+    files_url = snapshot_files_url(context, link)

    preview_html = snapshot_thumbnail(context, link)

-    if "_public_favicon_paths" in link.__dict__:
-        favicon_paths = list(getattr(link, "_public_favicon_paths", []) or [])
-        if favicon_paths:
-            favicon_urls = [_snapshot_url_for_context(context, link, path) for path in favicon_paths]
+    if "_public_list_icon_paths" in link.__dict__:
+        list_icon_paths = list(getattr(link, "_public_list_icon_paths", []) or [])
+        if list_icon_paths:
+            favicon_urls = [_snapshot_url_for_context(context, link, path) for path in list_icon_paths]
            favicon_html = (
                f'<img src="{escape(favicon_urls[0])}" '
                f'data-fallbacks="{escape(",".join(favicon_urls[1:]))}" '
@@ -751,7 +809,7 @@ def snapshot_index_row(context, link) -> str:
        else:
            favicon_html = '<span class="link-favicon link-favicon-empty" aria-hidden="true"></span>'
    else:
-        favicon_url = snapshot_archiveresult_url(context, link, "favicon", "favicon.ico")
+        favicon_url = snapshot_role_output_url(context, link, "list_icon")
        favicon_html = (
            (
                f'<img src="{escape(favicon_url)}" '
@@ -884,12 +942,7 @@ def render_plugin_card_document(context, result) -> str:
    output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path or "")

    icon_html = get_plugin_icon(plugin)
-    plugin_lower = (plugin or "").lower()
-    media_files = (
-        _list_media_files(result, include_filesystem_fallback=bool(context.get("STATIC_EXPORT")))
-        if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl")
-        else []
-    )
+    media_files = _list_media_files(result, include_filesystem_fallback=bool(context.get("STATIC_EXPORT")))
    media_file_count = len(media_files)
    if context.get("STATIC_EXPORT") and media_files:
        media_files = [
@@ -938,6 +991,8 @@ def render_plugin_card_document(context, result) -> str:
            rendered = tpl.render(ctx)
            # Only return non-empty content (strip whitespace to check)
            if rendered.strip():
+                if rendered.lstrip().lower().startswith("<!doctype html"):
+                    rendered = rendered.replace("</head>", CARD_OVERFLOW_STYLE + "</head>", 1)
                if context.get("STATIC_EXPORT") and rendered.lstrip().lower().startswith("<!doctype html"):
                    # Static exports have no _card route. Keep the same card
                    # document in both stack positions, with relative saved-file
@@ -1033,6 +1088,7 @@ def plugin_full(context, result) -> str:
                "output_path": output_url,
                "output_path_raw": raw_output_path,
                "plugin": plugin,
+                "plugin_icon": get_plugin_icon(plugin),
            },
        )
        rendered = tpl.render(ctx)
@@ -1,7 +1,5 @@
 __package__ = "archivebox.core"

-from archivebox.core.preview_util import PREVIEW_PLUGINS, preview_candidates
-
 import json
 import os
 import posixpath
@@ -10,7 +8,6 @@ from typing import ClassVar, cast
 from urllib.parse import quote, urlparse
 from uuid import UUID

-from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
 from admin_data_views.typing import ItemContext, SectionData, TableContext
 from admin_data_views.utils import ItemLink, render_with_item_view, render_with_table_view
 from django import template
@@ -20,7 +17,6 @@ from django.contrib.auth import HASH_SESSION_KEY, SESSION_KEY, get_user_model
 from django.contrib.auth.mixins import UserPassesTestMixin
 from django.contrib.sessions.models import Session
 from django.core import signing
-from django.core.paginator import InvalidPage
 from django.db.models import Case, IntegerField, Q, Value, When
 from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidden, QueryDict
 from django.shortcuts import redirect, render
@@ -51,7 +47,6 @@ from archivebox.config.configset import BaseConfigSet
 from archivebox.core.forms import AddLinkForm
 from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
 from archivebox.core.permissions import (
-    PERMISSIONS_PRIVATE,
    PERMISSIONS_PUBLIC,
    PERMISSIONS_UNLISTED,
    can_view_snapshot,
@@ -62,6 +57,8 @@ from archivebox.core.permissions import (
 )
 from archivebox.core.routes_util import (
    build_admin_url,
+    build_snapshot_detail_url,
+    build_snapshot_files_url,
    build_snapshot_url,
    build_web_url,
    get_admin_host,
@@ -72,7 +69,7 @@ from archivebox.core.routes_util import (
 )
 from archivebox.crawls.models import Crawl
 from archivebox.misc.paginators import AcceleratedPaginator
-from archivebox.misc.serve_static import serve_static_with_byterange_support
+from archivebox.misc.serve_static import FAVICON_CACHE_CONTROL, serve_static_with_byterange_support
 from archivebox.misc.util import (
    base_url,
    filter_queryset_by_uuid_substring,
@@ -81,10 +78,19 @@ from archivebox.misc.util import (
    validate_url,
    without_fragment,
 )
-from archivebox.plugins.discovery import get_plugin_catalog, get_plugin_name, get_plugin_template
+from archivebox.plugins.discovery import (
+    get_plugin_catalog,
+    get_plugin_icon,
+    get_plugin_name,
+    get_plugin_template,
+    get_snapshot_role_names,
+    plugin_has_custom_full_response,
+    serve_plugin_replay_asset,
+)
 from archivebox.plugins.forms import get_plugin_config_binary_urls
 from archivebox.plugins.views import get_config_definition_link
 from archivebox.progressmonitor.views import live_progress_view
+from archivebox.progressmonitor.collection import collection_summary
 from archivebox.search.config import (
    get_search_mode,
    get_search_mode_backend,
@@ -364,10 +370,6 @@ class SnapshotView(View):

        # slug is a timestamp
        if slug.replace(".", "").isdigit():
-            # missing trailing slash -> redirect to index
-            if "/" not in path:
-                return redirect(f"{path}/index.html")
-
            try:
                try:
                    snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
@@ -376,7 +378,11 @@ class SnapshotView(View):
                        return _private_snapshot_auth_redirect(request, snapshot, archivefile or "index.html")
                    canonical_base = snapshot.url_path
                    if canonical_base != snapshot.legacy_archive_path:
-                        target_path = f"/{canonical_base}/{archivefile or 'index.html'}"
+                        target_path = (
+                            build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
+                            if not archivefile or archivefile == "index.html"
+                            else build_snapshot_url(str(snapshot.id), archivefile, request=request)
+                        )
                        query = request.META.get("QUERY_STRING")
                        if query:
                            target_path = f"{target_path}?{query}"
@@ -391,8 +397,9 @@ class SnapshotView(View):
                            show_indexes=True,
                            is_archive_replay=True,
                        )
-                    elif archivefile == "index.html":
-                        # if they requested snapshot index, serve live rendered template instead of static html
+                    elif archivefile in ("", "index.html"):
+                        # Both the canonical extensionless URL and the legacy
+                        # index.html URL render the live Django snapshot page.
                        response = self.render_live_index(request, snapshot)
                    else:
                        target = build_snapshot_url(str(snapshot.id), archivefile, request=request)
@@ -426,15 +433,15 @@ class SnapshotView(View):
            except Snapshot.MultipleObjectsReturned:
                snapshot_hrefs = mark_safe("<br/>").join(
                    format_html(
-                        '{} <a href="/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
+                        '{} <a href="{}"><b><code>{}</code></b></a> {} <b>{}</b>',
                        snap.bookmarked_at.strftime("%Y-%m-%d %H:%M:%S"),
-                        snap.archive_path,
+                        build_snapshot_detail_url(snap.archive_path_from_db, request=request),
                        snap.timestamp,
                        snap.url,
                        snap.title_stripped[:64] or "",
                    )
                    for snap in direct_snapshots_queryset(request, Snapshot.objects.filter(timestamp__startswith=slug))
-                    .only("url", "timestamp", "title", "bookmarked_at")
+                    .select_related("crawl__created_by")
                    .order_by("-bookmarked_at")
                )
                return HttpResponse(
@@ -458,27 +465,27 @@ class SnapshotView(View):
                        <title>Snapshot Not Found</title>
                        </head><body>
                        <center><br/><br/><br/>
-                        Snapshot <a href="/{}/index.html" target="_top"><b><code>[{}]</code></b></a>: <a href="{}" target="_blank" rel="noreferrer">{}</a><br/>
-                        was queued on {}, but no files have been saved yet in:<br/><b><a href="/{}/" target="_top"><code>{}</code></a><code>/{}</code></b><br/><br/>
+                        Snapshot <a href="{}" target="_top"><b><code>[{}]</code></b></a>: <a href="{}" target="_blank" rel="noreferrer">{}</a><br/>
+                        was queued on {}, but no files have been saved yet in:<br/><b><a href="{}" target="_top"><code>{}</code></a><code>/{}</code></b><br/><br/>
                        It's possible {} during the last capture on {},<br/>or that the archiving process has not completed yet.<br/>
                        <pre><code># run this cmd to finish/retry archiving this Snapshot</code><br/>
                        <code style="user-select: all; color: #333">archivebox update -t timestamp {}</code></pre><br/><br/>
                        <div class="text-align: left; width: 100%; max-width: 400px">
                        <i><b>Next steps:</i></b><br/>
-                        - list all the <a href="/{}/" target="_top">Snapshot files <code>.*</code></a><br/>
-                        - view the <a href="/{}/index.html" target="_top">Snapshot <code>./index.html</code></a><br/>
+                        - list all the <a href="{}" target="_top">Snapshot files <code>.*</code></a><br/>
+                        - view the <a href="{}" target="_top">Snapshot <code>./index.html</code></a><br/>
                        - go to the <a href="/admin/core/snapshot/{}/change/" target="_top">Snapshot admin</a> to edit<br/>
                        - go to the <a href="/admin/core/snapshot/?id__exact={}" target="_top">Snapshot actions</a> to re-archive<br/>
                        - or return to <a href="/" target="_top">the main index...</a></div>
                        </center>
                        </body></html>
                        """,
-                        snapshot.archive_path,
+                        build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
                        snapshot.timestamp,
                        snapshot.url,
                        snapshot.url,
                        str(snapshot.bookmarked_at).split(".")[0],
-                        snapshot.archive_path,
+                        build_snapshot_files_url(str(snapshot.id), request=request),
                        snapshot.timestamp,
                        archivefile if str(archivefile) != "None" else "",
                        f"the {archivefile} resource could not be fetched"
@@ -486,8 +493,8 @@ class SnapshotView(View):
                        else "the original site was not available",
                        str(snapshot.bookmarked_at).split(".")[0],
                        snapshot.timestamp,
-                        snapshot.archive_path,
-                        snapshot.archive_path,
+                        build_snapshot_files_url(str(snapshot.id), request=request),
+                        build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
                        snapshot.pk,
                        snapshot.id,
                    ),
@@ -534,15 +541,15 @@ class SnapshotView(View):
        except Snapshot.MultipleObjectsReturned:
            snapshot_hrefs = mark_safe("<br/>").join(
                format_html(
-                    '{} <code style="font-size: 0.8em">{}</code> <a href="/{}/index.html"><b><code>{}</code></b></a> {} <b>{}</b>',
+                    '{} <code style="font-size: 0.8em">{}</code> <a href="{}"><b><code>{}</code></b></a> {} <b>{}</b>',
                    snap.bookmarked_at.strftime("%Y-%m-%d %H:%M:%S"),
                    str(snap.id)[:8],
-                    snap.archive_path,
+                    build_snapshot_detail_url(snap.archive_path_from_db, request=request),
                    snap.timestamp,
                    snap.url,
                    snap.title_stripped[:64] or "",
                )
-                for snap in snapshots.only("url", "timestamp", "title", "bookmarked_at").order_by("-bookmarked_at")
+                for snap in snapshots.select_related("crawl__created_by").order_by("-bookmarked_at")
            )
            return HttpResponse(
                format_html(
@@ -555,7 +562,7 @@ class SnapshotView(View):
                status=404,
            )

-        target_path = build_web_url(f"{snapshot.get_absolute_url().rstrip('/')}/index.html", request=request)
+        target_path = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
        query = request.META.get("QUERY_STRING")
        if query:
            target_path = f"{target_path}?{query}"
@@ -660,7 +667,9 @@ class SnapshotPathView(View):
        if snapshot_id:
            requested_base = f"{requested_base}/{snapshot_id}"
        if canonical_base != requested_base:
-            target = f"/{canonical_base}/{path or 'index.html'}"
+            target = f"/{canonical_base}"
+            if path:
+                target = f"{target}/{path}"
            query = request.META.get("QUERY_STRING")
            if query:
                target = f"{target}?{query}"
@@ -706,11 +715,14 @@ def _safe_archive_relpath(path: str) -> str | None:
    return cleaned


-def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[str, ArchiveResult | None]:
-    """Resolve plugin-relative output paths through ArchiveResult.output_files."""
+def _resolve_archiveresult_relpath(
+    snapshot: Snapshot,
+    rel_path: str,
+) -> tuple[str, ArchiveResult | None, tuple[str, ...]]:
+    """Resolve a declared output path and its exact historical fallbacks."""
    parts = Path(rel_path).parts
    if len(parts) < 2:
-        return rel_path, None
+        return rel_path, None, ()

    plugin = parts[0]
    plugin_relpath = posixpath.join(*parts[1:])
@@ -722,19 +734,38 @@ def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[s
        ).only("plugin", "output_files"),
    )
    if not results:
-        return rel_path, None
+        from archivebox.plugins.discovery import get_plugin_default_output_path

+        fallbacks = (plugin_relpath,) if get_plugin_default_output_path(plugin) == rel_path else ()
+        return rel_path, None, fallbacks
+
+    declared_paths: list[tuple[str, ArchiveResult]] = []
+    historical_fallbacks: list[str] = []
    for result in results:
-        output_files = result.output_files or {}
+        output_files = result.output_file_map()
        for candidate in (plugin_relpath, rel_path):
-            file_info = output_files.get(candidate)
-            if not isinstance(file_info, dict):
-                continue
-            if file_info.get("root_relative"):
-                return candidate, result
-            return rel_path, result
+            output_path = result.output_file_path(candidate, output_file_map=output_files)
+            if output_path and output_path not in {path for path, _result in declared_paths}:
+                declared_paths.append((output_path, result))

-    return rel_path, results[0]
+        # Partially migrated 0.7 rows can declare ``singlefile.html`` while
+        # the old file still lives at SNAP_DIR/singlefile.html, without the
+        # later root_relative metadata bit. Keep that one exact declared key as
+        # a fallback after the normal path returns 404. Do not preflight either
+        # path: the static response performs the only happy-path filesystem IO.
+        if plugin_relpath in output_files and plugin_relpath not in historical_fallbacks:
+            historical_fallbacks.append(plugin_relpath)
+
+    if declared_paths:
+        primary_path, primary_result = declared_paths[0]
+        fallbacks = tuple(
+            dict.fromkeys(
+                path for path in (*[path for path, _result in declared_paths[1:]], *historical_fallbacks) if path != primary_path
+            ),
+        )
+        return primary_path, primary_result, fallbacks
+
+    return rel_path, results[0], ()


 def _plugin_full_preview_response(
@@ -755,6 +786,11 @@ def _plugin_full_preview_response(
    template_str = get_plugin_template(plugin, "full", fallback=False)
    if not template_str:
        return None
+    if plugin_has_custom_full_response(plugin):
+        # The plugin owns path selection and rendering for this full template.
+        # Defer to the file-serving response hook so unrelated outputs from the
+        # same plugin continue through their normal MIME-specific renderer.
+        return None

    raw_query = request.GET.copy()
    raw_query.pop("preview", None)
@@ -774,18 +810,21 @@ def _plugin_full_preview_response(
                    "snapshot": snapshot,
                    "output_path": output_url,
                    "output_path_raw": rel_path,
-                    "snapshot_details_url": build_web_url(snapshot.get_absolute_url(), request=request),
+                    "snapshot_details_url": build_snapshot_detail_url(snapshot.archive_path_from_db, request=request),
                    "plugin": plugin,
+                    "plugin_icon": get_plugin_icon(plugin),
                    "preview_base": f"{request.path.rsplit('/', 1)[0]}/",
                },
            ),
        )
    )
    if request.GET.get("titlebar") == "0":
+        from archivebox.core.templatetags.core_tags import CARD_OVERFLOW_STYLE
+
        # The card already has a result header and actions. Only its embedded
        # trusted full viewer drops the duplicate top bar; opening the same
        # output directly keeps the complete viewer and controls.
-        rendered = rendered.replace("</head>", "<style>body > header { display: none !important; }</style></head>", 1)
+        rendered = rendered.replace("</head>", CARD_OVERFLOW_STYLE + "<style>body > header { display: none !important; }</style></head>", 1)
    response = HttpResponse(rendered, content_type="text/html; charset=utf-8")
    response.headers["Content-Disposition"] = f'inline; filename="{Path(rel_path).stem}.html"'
    response.headers["X-Content-Type-Options"] = "nosniff"
@@ -840,7 +879,7 @@ def _plugin_card_document_response(request: HttpRequest, snapshot: Snapshot, res

    # There is exactly one plugin card template. The stack cover and expanded
    # tray point at this same document; only their containing card width differs.
-    from archivebox.core.templatetags.core_tags import render_plugin_card_document
+    from archivebox.core.templatetags.core_tags import CARD_OVERFLOW_STYLE, render_plugin_card_document

    fragment = render_plugin_card_document(
        template.Context({"request": request, "CONFIG": get_request_config(request, resolve_plugins=False)}),
@@ -853,7 +892,7 @@ def _plugin_card_document_response(request: HttpRequest, snapshot: Snapshot, res
            '<!doctype html><html><head><meta charset="utf-8"><style>'
            "html,body{margin:0;width:100%;height:100%;overflow:hidden}"
            "body>iframe{display:block;width:100%;height:100%;border:0}"
-            "</style></head><body>"
+            f"</style>{CARD_OVERFLOW_STYLE}</head><body>"
            f"{fragment}</body></html>"
        )
    response = HttpResponse(card_html, content_type="text/html; charset=utf-8")
@@ -978,7 +1017,7 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
    snapshot._runtime_config = request_config

    if rel_path.startswith("replay/") or rel_path == "replay":
-        response = archivewebpage_replay.serve_replay_asset_response(rel_path, request_config, HttpResponse)
+        response = serve_plugin_replay_asset(rel_path, request_config, HttpResponse)
        if response is not None:
            return response

@@ -998,12 +1037,12 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
    is_directory_request = bool(path) and path.endswith("/")
    show_indexes = bool(request.GET.get("files")) or (request_config.USES_SUBDOMAIN_ROUTING and is_directory_request)
    if not show_indexes and (not rel_path or rel_path == "index.html"):
-        if snapshot.permissions in (PERMISSIONS_PUBLIC, PERMISSIONS_UNLISTED):
+        if request_config.CONTROL_PLANE_ENABLED and snapshot.permissions in (PERMISSIONS_PUBLIC, PERMISSIONS_UNLISTED):
            # Snapshot detail is trusted application UI and belongs on web.*.
            # snap-* is for isolated saved outputs and plugin frames. Keep the
            # private replay-cookie path below: its grant is host-only and must
            # never be mistaken for an admin session on web.*.
-            target = build_web_url(f"{snapshot.get_absolute_url().rstrip('/')}/index.html", request=request, config=request_config)
+            target = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request, config=request_config)
            if request.META.get("QUERY_STRING"):
                target = f"{target}?{request.META['QUERY_STRING']}"
            return redirect(target)
@@ -1018,22 +1057,23 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa
    if rel_path is None:
        raise Http404

-    rel_path, archive_result = _resolve_archiveresult_relpath(snapshot, rel_path)
+    rel_path, archive_result, fallback_relpaths = _resolve_archiveresult_relpath(snapshot, rel_path)

    plugin_preview = _plugin_full_preview_response(request, snapshot, rel_path, archive_result)
    if plugin_preview is not None:
        return plugin_preview

-    try:
-        return serve_static_with_byterange_support(
-            request,
-            rel_path,
-            document_root=snapshot.output_dir,
-            show_indexes=show_indexes,
-            is_archive_replay=True,
-        )
-    except Http404:
-        pass
+    for candidate_path in (rel_path, *fallback_relpaths):
+        try:
+            return serve_static_with_byterange_support(
+                request,
+                candidate_path,
+                document_root=snapshot.output_dir,
+                show_indexes=show_indexes,
+                is_archive_replay=True,
+            )
+        except Http404:
+            continue

    host = urlparse(snapshot.url).hostname or snapshot.domain
    responses_root = Path(snapshot.output_dir) / "responses" / host
@@ -1047,6 +1087,8 @@ def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, pa

 def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
    response = _build_snapshot_replay_response(request, snapshot, path)
+    if response.status_code in (200, 206, 304) and response.get("Cache-Control") == FAVICON_CACHE_CONTROL:
+        return response
    if path != "progress.json" and response.status_code in (200, 206, 304):
        if not response.get("Cache-Control"):
            policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
@@ -1170,6 +1212,18 @@ class OriginalDomainReplayView(View):


 class PublicIndexView(ListView):
+    """Keep browsing responsive while the same small server is archiving.
+
+    Performance takes priority over perfect initial totals: on a 1 vCPU / 1 GB
+    host with hundreds of thousands of snapshots, even an indexed count can
+    consume the entire roughly 0.5-0.7 second page-load budget. Benchmark with
+    the runner active, since it shares the CPU, memory, and SQLite database.
+
+    The unfiltered list therefore renders rows first and accepts missing/stale
+    display totals. A later ordinary reload can show the background-cached
+    count; do not restore a synchronous COUNT or add polling just to fill it in.
+    """
+
    template_name = "public_index.html"
    model = Snapshot
    ordering: ClassVar[list[str]] = ["-bookmarked_at", "-created_at"]
@@ -1195,14 +1249,28 @@ class PublicIndexView(ListView):
            "status",
            "output_size",
            "permissions",
+            # Modern replay URLs need the filesystem version and owner's name.
+            # Deferring these caused four extra SELECTs per displayed snapshot.
+            "fs_version",
+            "crawl__created_by__username",
        )

-    def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot] | None:
-        target_count = page_number * page_size
+    def _ordered_public_page_from_order_index(self, *, page_number: int, page_size: int) -> list[Snapshot]:
+        # One extra row determines Next without a full collection count. Cached
+        # totals can be stale while the runner adds/removes snapshots, so they
+        # must never decide whether the requested page exists.
+        target_count = page_number * page_size + 1
        public_snapshots: list[Snapshot] = []
        scanned = 0
-        chunk_size = max(self.public_page_scan_chunk_size, page_size)
-        ordered_snapshots = Snapshot.objects.order_by(*self.ordering).only(*self._base_public_snapshot_fields())
+        chunk_size = max(self.public_page_scan_chunk_size, page_size + 1)
+        # Read in snapshot_public_order_idx order and filter visibility in small
+        # batches. Combining the permissions predicate with ORDER BY can make
+        # SQLite select the permissions index and sort the whole public archive
+        # before LIMIT. This favors the common early pages without a new index;
+        # deep pages or mostly private collections can still require more reads.
+        ordered_snapshots = (
+            Snapshot.objects.select_related("crawl__created_by").order_by(*self.ordering).only(*self._base_public_snapshot_fields())
+        )

        while len(public_snapshots) < target_count:
            chunk = list(ordered_snapshots[scanned : scanned + chunk_size])
@@ -1218,20 +1286,39 @@ class PublicIndexView(ListView):
        if self.request.GET.get("q", default="").strip():
            return super().paginate_queryset(queryset, page_size)

-        public_count = self.get_exact_public_snapshot_count()
-        paginator = self.get_paginator(range(public_count), page_size)
        page_kwarg = self.kwargs.get(self.page_kwarg)
        page_query = self.request.GET.get(self.page_kwarg)
        page_number = page_kwarg or page_query or 1

        try:
-            page = paginator.page(page_number)
-        except InvalidPage as err:
+            page_number = int(page_number)
+            if page_number < 1:
+                raise ValueError("Page must be positive")
+        except (ValueError, TypeError) as err:
            raise Http404(f"Invalid page ({page_number}): {err}") from err

-        object_list = self._ordered_public_page_from_order_index(page_number=page.number, page_size=page_size)
-        page.object_list = object_list
-        return paginator, page, object_list, page.has_other_pages()
+        snapshots = self._ordered_public_page_from_order_index(page_number=page_number, page_size=page_size)
+        if not snapshots and page_number > 1:
+            raise Http404(f"Invalid page ({page_number}): No snapshots")
+        # Exhausting the rows gives an exact total for this read at no extra
+        # cost. Otherwise the lookahead is only a lower bound: an absent count
+        # is preferable to an on-request scan or a cached total smaller than
+        # the rows we just observed. Cached overestimates are acceptable here.
+        observed_count = (page_number - 1) * page_size + len(snapshots)
+        self.public_snapshot_count = observed_count
+        if len(snapshots) > page_size:
+            summary = collection_summary(self.request.user, public=True)
+            cached_count = summary["snapshots"] if summary is not None else None
+            self.public_snapshot_count = cached_count if cached_count is not None and cached_count >= observed_count else None
+
+        # Give Django an in-memory lower bound, never the collection QuerySet
+        # whose Paginator.count would issue COUNT(*). Navigation uses this read's
+        # lookahead even when a stale total would hide a newly added page.
+        # The template must use the separate display totals, not num_pages here.
+        paginator = self.get_paginator(range(observed_count), page_size)
+        page = paginator.page(page_number)
+        page.object_list = snapshots[:page_size]
+        return paginator, page, page.object_list, page.has_other_pages()

    def get_context_data(self, **kwargs):
        runtime_config = self.__dict__.get("runtime_config")
@@ -1260,14 +1347,20 @@ class PublicIndexView(ListView):
            and search_mode_backend
            and context["paginator"].count == 0,
        )
+        context["public_snapshot_count"] = context["paginator"].count if query else self.public_snapshot_count
+        count = context["public_snapshot_count"]
+        context["public_page_count"] = (
+            max(1, (count + context["paginator"].per_page - 1) // context["paginator"].per_page) if count is not None else None
+        )
        snapshots = list(context.get("object_list") or ())
        from archivebox.plugins.output_groups import plugin_output_sizes

        all_results_by_snapshot = {str(snapshot.id): [] for snapshot in snapshots}
+        snapshots_by_id = {str(snapshot.id): snapshot for snapshot in snapshots}
        icons_by_snapshot: dict[str, dict[str, ArchiveResult]] = {str(snapshot.id): {} for snapshot in snapshots}
        tag_names_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
-        preview_results_by_snapshot = {str(snapshot.id): [] for snapshot in snapshots}
-        favicon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
+        list_icon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
+        list_icon_plugins = set(get_snapshot_role_names("list_icon"))
        progress_by_snapshot: dict[str, dict[str, int]] = {
            str(snapshot.id): {
                "total": 0,
@@ -1295,12 +1388,22 @@ class PublicIndexView(ListView):
                    "snapshot_id",
                    "plugin",
                    "status",
+                    "output_str",
                    "output_files",
                    "output_size",
                )
            )
-            for result in results.iterator(chunk_size=1000):
+            # This is already bounded by the displayed snapshot page. Finish
+            # the SELECT before parsing output manifests: a streaming cursor
+            # otherwise keeps SQLite's read lock throughout rendering.
+            for result in list(results):
                snapshot_key = str(result.snapshot_id)
+                # Thumbnail URLs traverse result.snapshot too; share the page's
+                # already-loaded parent rather than fetching it once per result.
+                result.snapshot = snapshots_by_id[snapshot_key]
+                # These read-only page objects are shared by icons, sizes, and
+                # thumbnails. Normalize each manifest once for this render.
+                result._render_output_file_map = result.output_file_map()
                all_results_by_snapshot[snapshot_key].append(result)
                progress = progress_by_snapshot[snapshot_key]
                progress["total"] += 1
@@ -1315,11 +1418,9 @@ class PublicIndexView(ListView):
                    progress["skipped"] += 1
                elif result.status == ArchiveResult.StatusChoices.NORESULTS:
                    progress["noresults"] += 1
-                if result.plugin in PREVIEW_PLUGINS:
-                    preview_results_by_snapshot[snapshot_key].append(result)
-                    for candidate in preview_candidates([result]):
-                        if candidate["kind"] == "favicon":
-                            favicon_paths_by_snapshot[snapshot_key].append(candidate["path"])
+                if result.plugin in list_icon_plugins and result.status == ArchiveResult.StatusChoices.SUCCEEDED:
+                    if output_path := result.embed_path():
+                        list_icon_paths_by_snapshot[snapshot_key].append(output_path)

        for snapshot in snapshots:
            snapshot._icons_compact = True
@@ -1328,19 +1429,18 @@ class PublicIndexView(ListView):
            snapshot._icons_progress_stats = progress_by_snapshot.get(str(snapshot.id), {})
            snapshot.num_outputs_cached = snapshot._icons_progress_stats.get("succeeded", 0)
            snapshot._tags_str_cached = ",".join(tag_names_by_snapshot.get(str(snapshot.id), []))
-            snapshot._preview_results = preview_results_by_snapshot.get(str(snapshot.id), [])
-            snapshot._public_favicon_paths = favicon_paths_by_snapshot.get(str(snapshot.id), [])
+            snapshot._snapshot_card_results = all_results_by_snapshot[str(snapshot.id)]
+            snapshot._public_list_icon_paths = list_icon_paths_by_snapshot.get(str(snapshot.id), [])
            snapshot._is_archived_cached = bool(snapshot.downloaded_at or snapshot.status == Snapshot.StatusChoices.SEALED)
        context["object_list"] = snapshots
        return context

-    def get_exact_public_snapshot_count(self) -> int:
-        hidden_count = Snapshot.objects.filter(permissions=PERMISSIONS_PRIVATE).count()
-        hidden_count += Snapshot.objects.filter(permissions=PERMISSIONS_UNLISTED).count()
-        return Snapshot.objects.count() - hidden_count
-
    def get_queryset(self, **kwargs):
-        qs = public_snapshots_queryset(super().get_queryset(**kwargs)).only(*self._base_public_snapshot_fields())
+        qs = (
+            public_snapshots_queryset(super().get_queryset(**kwargs))
+            .select_related("crawl__created_by")
+            .only(*self._base_public_snapshot_fields())
+        )
        query = self.request.GET.get("q", default="").strip()

        if not query:
@@ -1619,7 +1719,7 @@ class WebAddView(AddView):
        if requested_url:
            snapshot = self._latest_snapshot_for_url(requested_url)
            if snapshot:
-                return redirect(f"/{snapshot.url_path}")
+                return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))

        request_host = (request.get_host() or "").lower()
        request_config = get_request_config(request)
@@ -1659,7 +1759,7 @@ class WebAddView(AddView):

        snapshot = self._latest_snapshot_for_url(requested_url)
        if snapshot:
-            return redirect(f"/{snapshot.url_path}")
+            return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))

        add_url = self._normalize_add_url(requested_url)
        assert self.form_class is not None
@@ -1691,7 +1791,7 @@ class WebAddView(AddView):
        crawl = self._create_crawl_from_form(form)
        snapshot = Snapshot.from_json({"url": add_url, "tags": form.cleaned_data.get("tag", "")}, overrides={"crawl": crawl})
        assert snapshot is not None
-        return redirect(f"/{snapshot.url_path}")
+        return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request))


 class HealthCheckView(View):
@@ -717,7 +717,7 @@ class InlineTagEditorWidget(TagEditorWidget):
                )
            pills_html += f'''
                <span class="tag-pill" data-tag="{self._escape(td["name"])}" data-tag-id="{td["id"]}" style="{self._tag_style(td["name"])}">
-                    <a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link">{self._escape(td["name"])}</a>
+                    <a href="/admin/core/snapshot/?tags__id__exact={td["id"]}" class="tag-link" title="{self._escape(td["name"])}">{self._escape(td["name"])}</a>
                    {remove_button}
                </span>
            '''
@@ -8,7 +8,7 @@ from urllib.parse import urlencode, urlparse
 from django import forms
 from django.contrib import admin, messages
 from django.core.paginator import Paginator
-from django.db.models import Count, F, Q
+from django.db.models import Count, F, Prefetch, Q
 from django.http import HttpRequest, HttpResponseBadRequest, HttpResponseNotAllowed, JsonResponse
 from django.shortcuts import get_object_or_404, redirect
 from django.template.loader import render_to_string
@@ -30,9 +30,11 @@ from archivebox.core.permissions import (
    PERMISSIONS_VALUES,
    normalize_permissions,
 )
+from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_role_output_url
 from archivebox.core.widgets import TagEditorWidget, URLFiltersWidget
 from archivebox.crawls.models import Crawl, CrawlSchedule
 from archivebox.misc.paginators import AcceleratedPaginator
+from archivebox.plugins.discovery import get_snapshot_role_names
 from archivebox.progressmonitor.views import progress_endpoint
 from archivebox.workers.models import RETRY_AT_MAX

@@ -72,13 +74,32 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,

    # Keep ArchiveResult counters as scalar subqueries so the paginated
    # Snapshot queryset does not become a join+GROUP BY over every result row.
-    snapshots_qs = filtered_qs.order_by("-created_at").annotate(
-        total_results=ArchiveResult.snapshot_count_expr(),
-        succeeded_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SUCCEEDED),
-        failed_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.FAILED),
-        started_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.STARTED),
-        skipped_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SKIPPED),
-        snapshot_permissions=F("permissions"),
+    snapshots_qs = (
+        filtered_qs.order_by("-created_at")
+        .annotate(
+            total_results=ArchiveResult.snapshot_count_expr(),
+            succeeded_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SUCCEEDED),
+            failed_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.FAILED),
+            started_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.STARTED),
+            skipped_results=ArchiveResult.snapshot_count_expr(status=ArchiveResult.StatusChoices.SKIPPED),
+            snapshot_permissions=F("permissions"),
+        )
+        .prefetch_related(
+            Prefetch(
+                "archiveresult_set",
+                queryset=ArchiveResult.objects.filter(
+                    plugin__in=get_snapshot_role_names("list_icon"),
+                    status=ArchiveResult.StatusChoices.SUCCEEDED,
+                ).only(
+                    "snapshot_id",
+                    "plugin",
+                    "status",
+                    "output_str",
+                    "output_files",
+                ),
+                to_attr="_list_icon_results",
+            ),
+        )
    )

    page_number = request.GET.get(page_param, 1) if request is not None else 1
@@ -219,6 +240,14 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,

        # Format date
        date_str = snapshot.created_at.strftime("%Y-%m-%d %H:%M") if snapshot.created_at else "-"
+        detail_url = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
+        favicon_url = build_snapshot_role_output_url(
+            snapshot,
+            "list_icon",
+            request=request,
+            archive_results=snapshot._list_icon_results,
+            fallback_to_default=True,
+        )

        rows.append(f'''
            <tr style="border-bottom: 1px solid #eee;">
@@ -231,8 +260,8 @@ def render_snapshots_list(snapshots_qs, request=None, crawl=None, page_size=50,
                    <span title="{permissions}" style="display:inline-flex; align-items:center; justify-content:center; width:22px; height:22px; border-radius:999px; font-size:12px; color:{permission_fg}; background:{permission_bg};">{permission_icon}</span>
                </td>
                <td style="padding: 6px 8px; white-space: nowrap;">
-                    <a href="/{snapshot.archive_path}/" style="text-decoration: none;">
-                        <img src="/{snapshot.archive_path}/favicon.ico"
+                    <a href="{escape(detail_url)}" style="text-decoration: none;">
+                        <img src="{escape(favicon_url)}"
                             style="width: 16px; height: 16px; vertical-align: middle; margin-right: 4px;"
                             onerror="this.style.display='none'"/>
                    </a>
@@ -57,7 +57,10 @@ def get_vm_info():

    hw_manufacturer = "Docker" if hw_in_docker else "Unknown"
    hw_product = "Container" if hw_in_docker else "Unknown"
-    hw_uuid = machineid.id()
+    try:
+        hw_uuid = machineid.id()
+    except machineid.MachineIdNotFound:
+        hw_uuid = "Unknown"

    if platform.system().lower() == "darwin":
        # Get macOS machine info
@@ -301,7 +304,9 @@ def get_host_stats() -> dict[str, Any]:


 def get_host_guid() -> str:
-    return machineid.hashed_id("archivebox")
+    from archivebox.config.paths import get_machine_id
+
+    return get_machine_id(length=64)


 # Example usage
@@ -221,35 +221,21 @@ class Machine(ModelWithHealthStats):
        app_label = "machine"

    @classmethod
-    def current(cls, refresh: bool = False) -> Machine:
-        global _CURRENT_MACHINE
-        if refresh:
-            _CURRENT_MACHINE = None
-        if _CURRENT_MACHINE:
-            if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
-                # Reconciliation can save() and clear the shared cache; keep
-                # this call's instance so we never return None after that save.
-                machine = _CURRENT_MACHINE
-                # One-time-per-process reconciliation between ArchiveBox.conf
-                # and Machine.config. Fast-path: bool check + early-return when
-                # the sync has already run, so the cached-machine return path
-                # stays sub-microsecond.
-                try:
-                    from archivebox.config.collection import sync_machine_and_file
+    def current_readonly(cls) -> Machine | None:
+        """Read the current machine without registration, refresh, or config writes."""
+        if _CURRENT_MACHINE is not None:
+            return _CURRENT_MACHINE
+        return cls.objects.filter(guid=get_host_guid()).first()

-                    sync_machine_and_file(machine)
-                except Exception:
-                    pass
-                return machine
-            else:
-                _CURRENT_MACHINE = None
+    @classmethod
+    def current(cls) -> Machine:
+        global _CURRENT_MACHINE
+        if _CURRENT_MACHINE is not None:
+            return _CURRENT_MACHINE

        host_guid = get_host_guid()
-        # save() intentionally invalidates _CURRENT_MACHINE so config edits
-        # made through other instances become visible in this process. Host
-        # refresh (after seven days), sanitation, and reconciliation can all
-        # save here, so the working instance must not live only in that cache.
-        # Otherwise a routine refresh leaves startup dereferencing None.
+        # Host detection, sanitation, and config reconciliation run only on
+        # initial registration, never on cached reads.
        try:
            machine = cls.objects.get(guid=host_guid)
        except cls.DoesNotExist:
@@ -269,44 +255,41 @@ class Machine(ModelWithHealthStats):
                **get_vm_info(),
                stats=get_host_stats(),
            )
-        else:
-            if timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
-                for key, value in {
-                    "hostname": socket.gethostname(),
-                    **get_os_info(),
-                    **get_vm_info(),
-                    "stats": get_host_stats(),
-                }.items():
-                    setattr(machine, key, value)
-                machine.save(
-                    update_fields=[
-                        "hostname",
-                        "hw_in_docker",
-                        "hw_in_vm",
-                        "hw_manufacturer",
-                        "hw_product",
-                        "hw_uuid",
-                        "os_arch",
-                        "os_family",
-                        "os_platform",
-                        "os_release",
-                        "os_kernel",
-                        "stats",
-                        "modified_at",
-                    ],
-                )
-        machine = cls._sanitize_config(machine)
-        # Same one-time sync as the cached-return path. Triggers here on the
-        # very first ``Machine.current()`` call in a process before any
-        # cached return can occur.
+        needs_recheck = timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL)
+        # Reconcile before any save can mirror stale DB config over file edits.
        try:
            from archivebox.config.collection import sync_machine_and_file

            sync_machine_and_file(machine)
        except Exception:
            pass
-        # Publish only after all save-capable steps; publishing earlier would
-        # let those steps invalidate the cache we are in the middle of filling.
+        if needs_recheck:
+            for key, value in {
+                "hostname": socket.gethostname(),
+                **get_os_info(),
+                **get_vm_info(),
+                "stats": get_host_stats(),
+            }.items():
+                setattr(machine, key, value)
+            machine.save(
+                update_fields=[
+                    "hostname",
+                    "hw_in_docker",
+                    "hw_in_vm",
+                    "hw_manufacturer",
+                    "hw_product",
+                    "hw_uuid",
+                    "os_arch",
+                    "os_family",
+                    "os_platform",
+                    "os_release",
+                    "os_kernel",
+                    "stats",
+                    "modified_at",
+                ],
+            )
+        machine = cls._sanitize_config(machine)
+        # Publish only after initialization is complete.
        _CURRENT_MACHINE = machine
        return machine

@@ -377,18 +360,15 @@ class Machine(ModelWithHealthStats):
            if update_fields is not None:
                kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))

-        # Drop the ``Machine.current()`` module-level cache on every save so
-        # config edits (admin form, from_json, etc.) become live in the same
-        # process without waiting out the 7-day ``MACHINE_RECHECK_INTERVAL``.
-        # The cache reads ``_CURRENT_MACHINE.modified_at`` and that value
-        # never moves forward on the cached object even when the row is
-        # updated in the DB, so without this we'd keep serving stale
-        # ``machine.config`` (incl. ``BASE_URL``) until the worker restarts.
+        # Apply explicit edits to the cached instance without making the next
+        # read repeat host detection, sanitation, or config reconciliation.
        update_fields = kwargs.get("update_fields")
        super().save(*args, **kwargs)
        global _CURRENT_MACHINE
        if _CURRENT_MACHINE is not None and _CURRENT_MACHINE.pk == self.pk:
-            _CURRENT_MACHINE = None
+            for field in self._meta.concrete_fields:
+                if field.attname in self.__dict__ and (update_fields is None or field.name in update_fields):
+                    setattr(_CURRENT_MACHINE, field.attname, self.__dict__[field.attname])

        # Mirror Machine.config into ArchiveBox.conf so the two stores stay
        # 1:1. Skipped when ``update_fields`` is set and doesn't touch
@@ -443,7 +423,7 @@ class NetworkInterface(ModelWithHealthStats):
    @classmethod
    def current(cls, refresh: bool = False) -> NetworkInterface:
        global _CURRENT_INTERFACE
-        machine = Machine.current(refresh=refresh)
+        machine = Machine.current()
        if _CURRENT_INTERFACE and _CURRENT_INTERFACE.machine_id == machine.id:
            if not refresh:
                # Callers that pass refresh=False are asking for attribution to
@@ -1778,7 +1758,8 @@ class Process(ModelWithDeleteAfter, models.Model):
            return None

        # Can't validate processes on other machines
-        if self.machine_id != Machine.current().id:
+        machine = Machine.current_readonly()
+        if machine is None or self.machine_id != machine.id:
            return None

        try:
@@ -445,7 +445,10 @@ def migration_lock(stdout: TextIO | None = None):


 # Migration names that previously existed in ArchiveBox's source tree but have
-# since been deleted (squashed away, renamed, moved between apps, etc.). DBs
+# since been deleted (squashed away, renamed, moved between apps, etc.), plus
+# exact older migrations generated in the installed package by v0.6.2's
+# startup makemigrations. Those runtime-generated names were never in Git, so
+# the deleted-file history alone cannot cover them. DBs
 # upgraded incrementally through 0.8.x → 0.9.x dev rcs accumulate rows for
 # these in ``django_migrations``; the newer-DB guard added in 65dc2521 would
 # otherwise refuse to start with "applied migrations missing from this build"
@@ -454,13 +457,18 @@ def migration_lock(stdout: TextIO | None = None):
 # splits the migration graph when only a *subset* of the replaces list is
 # applied (which is exactly what happens for users at different intermediate
 # dev branch states). This set is the authoritative compat list — extend it
-# when squashing more migrations away. Generated from
-# ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
+# when squashing more migrations away. The source-tracked names were generated
+# from ``git log --diff-filter=D --name-only`` over each app's migrations/ tree.
 HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
    {
        # core: 0023→0075 sequence plus every transient dev rename
        ("core", "0002_auto_20190417_0739"),
        ("core", "0006_auto_20200915_2006"),
+        # Reported v0.6.2 runtime-generated 0021 migrations; the checked-in
+        # replacement only changes ArchiveResult.extractor choices.
+        ("core", "0021_auto_20220510_0644"),
+        ("core", "0021_auto_20220724_1254"),
+        ("core", "0021_auto_20221128_1116"),
        ("core", "0023_alter_archiveresult_options_archiveresult_abid_and_more"),
        ("core", "0023_new_schema"),
        ("core", "0024_auto_20240513_1143"),
@@ -518,6 +526,8 @@ HISTORICAL_GHOST_MIGRATIONS: frozenset[tuple[str, str]] = frozenset(
        ("core", "0053_remove_snapshottag_snapshot_old"),
        ("core", "0054_alter_snapshot_timestamp"),
        ("core", "0055_alter_tag_slug"),
+        # Dev-only progress indexes removed in favor of existing-index queries.
+        ("core", "0056_progress_covering_indexes"),
        ("core", "0056_remove_tag_uuid"),
        ("core", "0057_rename_id_tag_old_id"),
        ("core", "0058_alter_tag_old_id"),
@@ -18,7 +18,6 @@ from html.parser import HTMLParser
 from pathlib import Path
 from urllib.parse import quote, urlencode, urljoin

-from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
 from django.contrib.staticfiles import finders
 from django.core.handlers.asgi import ASGIRequest
 from django.http import Http404, HttpResponse, HttpResponseNotModified, StreamingHttpResponse
@@ -30,8 +29,10 @@ from django.views import static

 from archivebox.config.common import get_config
 from archivebox.misc.logging_util import printable_filesize
+from archivebox.plugins.discovery import render_plugin_full_response

 _HASHES_CACHE: dict[Path, tuple[float, dict[str, str]]] = {}
+FAVICON_CACHE_CONTROL = "public, max-age=31536000, s-maxage=31536000, immutable"
 IMG_SRC_ATTR_RE = re.compile(r'(<img\b[^>]*?\s(?:src|data-src)=["\'])([^"\']+)(["\'])', re.IGNORECASE)
 TRANSFORMED_HTML_PREVIEW_STYLE = """<style id="archivebox-static-html-preview-style">
 html {
@@ -939,6 +940,15 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
        raise Http404(_("“%(path)s” does not exist") % {"path": fullpath})

    statobj = fullpath.stat()
+    # Captured site icons are intentionally shareable even for private snapshots.
+    # Only raw icon responses qualify, not generated previews or directory pages.
+    favicon_cache_control = (
+        FAVICON_CACHE_CONTROL
+        if fullpath.stem.lower() == "favicon"
+        and fullpath.suffix.lower() in {".ico", ".svg", ".png", ".jpg", ".jpeg", ".gif", ".webp", ".avif"}
+        and not request.GET.get("preview")
+        else None
+    )
    document_root = Path(document_root) if document_root else None
    rel_path = path
    etag = None
@@ -954,7 +964,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
            if etag in inm_list or etag.strip('"') in [i.strip('"') for i in inm_list]:
                not_modified = HttpResponseNotModified()
                not_modified.headers["ETag"] = etag
-                not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
+                not_modified.headers["Cache-Control"] = favicon_cache_control or f"{cache_policy}, max-age=31536000, immutable"
                not_modified.headers["Last-Modified"] = http_date(statobj.st_mtime)
                return _apply_archive_replay_headers(
                    not_modified,
@@ -995,15 +1005,13 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
    preview_as_image_html = (
        bool(request.GET.get("preview")) and content_type.startswith("image/") and not content_type.startswith("image/svg+xml")
    )
-    preview_as_archivewebpage_html = bool(request.GET.get("preview")) and archivewebpage_replay.is_replay_target(fullpath.name)
-
    # Respect the If-Modified-Since header for non-markdown responses.
    if not content_type.startswith(("text/plain", "text/html")) and not static.was_modified_since(
        request.META.get("HTTP_IF_MODIFIED_SINCE"),
        statobj.st_mtime,
    ):
        not_modified = HttpResponseNotModified()
-        not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
+        not_modified.headers["Cache-Control"] = favicon_cache_control or f"{cache_policy}, max-age=60, stale-while-revalidate=300"
        return _apply_archive_replay_headers(
            not_modified,
            fullpath=fullpath,
@@ -1029,7 +1037,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                    config=config,
                )
        except (OSError, UnicodeDecodeError, ValueError):
-            preview_as_text_html = False
+            pass

    if preview_as_image_html:
        try:
@@ -1049,16 +1057,16 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                config=config,
            )
        except (OSError, ValueError):
-            preview_as_image_html = False
+            pass

-    if preview_as_archivewebpage_html:
+    if request.GET.get("preview"):
        try:
            raw_query = request.GET.copy()
            raw_query.pop("preview", None)
            raw_output_path = request.path
            if raw_query:
                raw_output_path = f"{raw_output_path}?{raw_query.urlencode()}"
-            body, preview_content_type, headers = archivewebpage_replay.render_preview_response(
+            plugin_replay = render_plugin_full_response(
                fullpath.name,
                raw_output_path,
                wacz_path=fullpath,
@@ -1070,18 +1078,20 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                ),
                content_encoding=encoding or "",
            )
-            response = HttpResponse(body, content_type=preview_content_type)
-            for key, value in headers.items():
-                response.headers[key] = value
-            return _apply_archive_replay_headers(
-                response,
-                fullpath=fullpath,
-                content_type=preview_content_type,
-                is_archive_replay=is_archive_replay,
-                config=config,
-            )
+            if plugin_replay is not None:
+                body, replay_content_type, headers = plugin_replay
+                response = HttpResponse(body, content_type=replay_content_type)
+                for key, value in headers.items():
+                    response.headers[key] = value
+                return _apply_archive_replay_headers(
+                    response,
+                    fullpath=fullpath,
+                    content_type=replay_content_type,
+                    is_archive_replay=is_archive_replay,
+                    config=config,
+                )
        except (OSError, RuntimeError, ValueError):
-            preview_as_archivewebpage_html = False
+            pass

    # Heuristic fix: some archived HTML outputs are stored with HTML-escaped markup
    # or markdown sources. If so, render sensibly.
@@ -1153,6 +1163,8 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
        response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
    if content_type.startswith("image/"):
        response.headers["Cache-Control"] = f"{cache_policy}, max-age=604800, immutable"
+    if favicon_cache_control:
+        response.headers["Cache-Control"] = favicon_cache_control

    # handle byte-range requests by serving chunk of file
    if stat.S_ISREG(statobj.st_mode):
@@ -3,7 +3,7 @@ __package__ = "archivebox.misc"
 # Bootable utility functions (URL parsing, date parsing, JSON encoding, decorators).
 # MUST NOT import archivebox.config, archivebox.core, or Django — this module is
 # loaded by hooks.py and other early code paths. Only depends on stdlib and the
-# bootable .logging module.
+# bootable .logging module and standalone abx_plugins URL helpers.

 import re
 import json as pyjson
@@ -19,6 +19,9 @@ from urllib.parse import urlparse, quote, unquote
 from html import escape, unescape
 from datetime import datetime, timezone

+from abx_plugins.plugins.base.url_cleaning import URL_REGEX, fix_url_from_markdown
+from abx_plugins.plugins.base.url_cleaning import sanitize_extracted_url as sanitize_plugin_url
+
 from .logging import COLOR_DICT


@@ -90,18 +93,6 @@ def ts_to_date_str(ts: Any) -> str | None:
 COLOR_REGEX = re.compile(r"\[(?P<arg_1>\d+)(;(?P<arg_2>\d+)(;(?P<arg_3>\d+))?)?m")


-# https://mathiasbynens.be/demo/url-regex
-URL_REGEX = re.compile(
-    r"(?=("
-    r"http[s]?://"  # start matching from allowed schemes
-    r"(?:[a-zA-Z]|[0-9]"  # followed by allowed alphanum characters
-    r"|[-_$@.&+!*\(\),]"  #   or allowed symbols (keep hyphen first to match literal hyphen)
-    r"|[^\u0000-\u007F])+"  #   or allowed unicode bytes
-    r'[^\]\[<>"\'\s]+'  # stop parsing at these symbols
-    r"))",
-    re.IGNORECASE | re.UNICODE,
-)
-
 # Maximum supported URL length. Very long URLs are rare but must be supported correctly
 # (e.g. data: URLs, deeply nested query strings). The Snapshot.url column is stored as a
 # variable-length TextField (so short URLs don't reserve space and very long URLs still
@@ -109,29 +100,6 @@ URL_REGEX = re.compile(
 # all keep working.
 MAX_URL_LENGTH = 65535

-QUOTE_DELIMITERS = (
-    '"',
-    "'",
-    "`",
-    "“",
-    "”",
-    "‘",
-    "’",
-)
-QUOTE_ENTITY_DELIMITERS = (
-    "&quot;",
-    "&#34;",
-    "&#x22;",
-    "&apos;",
-    "&#39;",
-    "&#x27;",
-)
-URL_ENTITY_REPLACEMENTS = (
-    ("&amp;", "&"),
-    ("&#38;", "&"),
-    ("&#x26;", "&"),
-)
-
 FILESIZE_UNITS: dict[str, int] = {
    "": 1,
    "b": 1,
@@ -153,36 +121,9 @@ FILESIZE_UNITS: dict[str, int] = {


 def sanitize_extracted_url(url: str) -> str:
-    """Trim quote garbage and dangling prose punctuation from an extracted URL candidate."""
-    cleaned = (url or "").strip()
-    if not cleaned:
-        return cleaned
-
-    lower_cleaned = cleaned.lower()
-    cut_index = len(cleaned)
-
-    for delimiter in QUOTE_DELIMITERS:
-        found_index = cleaned.find(delimiter)
-        if found_index != -1:
-            cut_index = min(cut_index, found_index)
-
-    for delimiter in QUOTE_ENTITY_DELIMITERS:
-        found_index = lower_cleaned.find(delimiter)
-        if found_index != -1:
-            cut_index = min(cut_index, found_index)
-
-    cleaned = cleaned[:cut_index].strip()
-    lower_cleaned = cleaned.lower()
-    for entity, replacement in URL_ENTITY_REPLACEMENTS:
-        while entity in lower_cleaned:
-            entity_index = lower_cleaned.find(entity)
-            cleaned = cleaned[:entity_index] + replacement + cleaned[entity_index + len(entity) :]
-            lower_cleaned = cleaned.lower()
-
-    cleaned = cleaned.rstrip(".,;:!?\\'\"")
-    cleaned = cleaned.rstrip('"')
-
-    return cleaned
+    """Use the standalone parser's quote rules, then trim dangling prose punctuation."""
+    cleaned = sanitize_plugin_url(url)
+    return cleaned.rstrip('.,;:!?\\"')


 def validate_url_length(url: str) -> str:
@@ -215,39 +156,6 @@ def parens_are_matched(string: str, open_char="(", close_char=")"):
    return count == 0


-def fix_url_from_markdown(url_str: str) -> str:
-    """
-    cleanup a regex-parsed url that may contain dangling trailing parens from markdown link syntax
-    helpful to fix URLs parsed from markdown e.g.
-      input:  https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def).somemoretext
-      result: https://wikipedia.org/en/some_article_(Disambiguation).html?abc=def
-
-    IMPORTANT ASSUMPTION: valid urls wont have unbalanced or incorrectly nested parentheses
-    e.g. this will fail the user actually wants to ingest a url like 'https://example.com/some_wei)(rd_url'
-         in that case it will return https://example.com/some_wei (truncated up to the first unbalanced paren)
-    This assumption is true 99.9999% of the time, and for the rare edge case the user can use url_list parser.
-    """
-    trimmed_url = url_str
-    if len(trimmed_url) > 2048:
-        return trimmed_url
-
-    # cut off one trailing character at a time
-    # until parens are balanced e.g. /a(b)c).x(y)z -> /a(b)c
-    trim_attempts = 0
-    while trimmed_url and not parens_are_matched(trimmed_url) and trim_attempts < 256:
-        trimmed_url = trimmed_url[:-1]
-        trim_attempts += 1
-
-    if not trimmed_url or not parens_are_matched(trimmed_url):
-        return url_str
-
-    # make sure trimmed url is still valid
-    if any(match == trimmed_url for match in re.findall(URL_REGEX, trimmed_url)):
-        return trimmed_url
-
-    return url_str
-
-
 def split_comma_separated_urls(url: str):
    offset = 0
    while True:
@@ -274,7 +182,8 @@ def find_all_urls(urls_str: str):
        if match.start() in skipped_starts:
            continue

-        cleaned_match = sanitize_extracted_url(fix_url_from_markdown(match.group(1)))
+        cleaned_match = fix_url_from_markdown(match.group(1), preceding_text=urls_str[max(0, match.start(1) - 6) : match.start(1)])
+        cleaned_match = sanitize_extracted_url(cleaned_match)
        for offset, url in split_comma_separated_urls(cleaned_match):
            if offset:
                skipped_starts.add(match.start() + offset)
@@ -388,9 +297,12 @@ def parse_date(date: Any) -> datetime | None:
        if date.tzinfo is None:
            return date.replace(tzinfo=timezone.utc)

-        offset = date.utcoffset()
-        assert offset == datetime.now(timezone.utc).utcoffset(), "Refusing to load a non-UTC date!"
-        return date
+        # Legacy SQLite ArchiveResult rows can retain explicit timezone offsets
+        # through migration. Django loads those as aware datetimes, whereas JSON
+        # imports reach the string branch below. Normalize both representations
+        # consistently so opening a migrated snapshot does not raise (#1889).
+        # Convert the instant; replacing tzinfo here would shift the capture time.
+        return date.astimezone(timezone.utc)

    if isinstance(date, (float, int)):
        date = str(date)
@@ -74,8 +74,12 @@ def derive_persona_config(*, name: str, config: Mapping[str, Any] | None, person
        derived["COOKIES_FILE"] = str(cookies_path)

    auth_path = persona_dir / "auth.json"
-    if "AUTH_STORAGE_FILE" not in derived and auth_path.exists():
-        derived["AUTH_STORAGE_FILE"] = str(auth_path)
+    if "AUTH_STORAGE_FILE" not in derived:
+        try:
+            if auth_path.stat().st_size > 0:
+                derived["AUTH_STORAGE_FILE"] = str(auth_path)
+        except FileNotFoundError:
+            pass

    derived["ACTIVE_PERSONA"] = name
    return derived
@@ -156,9 +160,12 @@ class Persona(ModelWithConfig):

    @property
    def AUTH_STORAGE_FILE(self) -> str:
-        """Derived path to auth.json for this persona (if it exists)."""
+        """Derived path to a nonempty auth.json export for this persona."""
        auth_path = self.path / "auth.json"
-        return str(auth_path) if auth_path.exists() else ""
+        try:
+            return str(auth_path) if auth_path.stat().st_size > 0 else ""
+        except FileNotFoundError:
+            return ""

    def get_derived_config(self) -> dict:
        """
@@ -315,20 +322,9 @@ class Persona(ModelWithConfig):
            if chrome_binary:
                (runtime_root / "chrome_binary.txt").write_text(chrome_binary)

-        return {
-            # Hooks derive CHROME_USER_DATA_DIR/CHROME_DOWNLOADS_DIR from
-            # PERSONAS_DIR + ACTIVE_PERSONA. Point PERSONAS_DIR at the
-            # per-crawl runtime root here so CHROME_ISOLATION=crawl never
-            # leaks or reuses the template profile while keeping Chrome path
-            # derivation centralized in the Chrome plugin helpers.
-            "PERSONAS_DIR": str(runtime_root.parent),
-            "ACTIVE_PERSONA": self.name,
-            **{
-                key: str(runtime_root / filename)
-                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
-                if (runtime_root / filename).is_file()
-            },
-        }
+        # Hooks derive Chrome profile paths from the runtime persona root.
+        # Use the same optional auth discovery as the template persona.
+        return derive_persona_config(name=self.name, config={}, persona_dir=runtime_root)

    def prepare_runtime_for_snapshot(self, snapshot, chrome_binary: str = "") -> dict[str, str]:
        crawl_runtime_profile_dir = self.runtime_profile_dir_for_crawl(snapshot.crawl)
@@ -358,18 +354,7 @@ class Persona(ModelWithConfig):
        if chrome_binary:
            (runtime_root / "chrome_binary.txt").write_text(chrome_binary)

-        return {
-            # See prepare_runtime_for_crawl(): snapshot isolation changes the
-            # persona root, not individual CHROME_* config keys, so standalone
-            # Chrome hooks and ArchiveBox-driven hooks resolve paths the same way.
-            "PERSONAS_DIR": str(runtime_root.parent),
-            "ACTIVE_PERSONA": self.name,
-            **{
-                key: str(runtime_root / filename)
-                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
-                if (runtime_root / filename).is_file()
-            },
-        }
+        return derive_persona_config(name=self.name, config={}, persona_dir=runtime_root)

    def cleanup_runtime_for_crawl(self, crawl) -> None:
        shutil.rmtree(Path(crawl.output_dir) / ".persona", ignore_errors=True)
@@ -2,7 +2,9 @@ __package__ = "archivebox.plugins"

 from collections.abc import Iterable
 from functools import lru_cache
-from pathlib import Path
+from importlib import import_module
+from pathlib import Path, PurePosixPath
+from types import ModuleType
 from typing import Any, Protocol, TypedDict

 from abx_plugins import get_plugins_dir
@@ -66,8 +68,141 @@ def get_plugin_name(plugin: str) -> str:
    """
    parts = plugin.split("_", 1)
    if len(parts) == 2 and parts[0].isdigit():
-        return parts[1]
-    return plugin
+        plugin = parts[1]
+    return get_archive_result_aliases().get(plugin, plugin)
+
+
+@lru_cache(maxsize=1)
+def get_archive_result_aliases() -> dict[str, str]:
+    aliases: dict[str, str] = {}
+    for plugin in get_plugin_catalog().values():
+        for alias in plugin.manifest.get("archive_result_aliases") or []:
+            alias = str(alias or "").strip()
+            if alias:
+                aliases[alias] = plugin.name
+    return aliases
+
+
+def get_archive_result_names(plugin_name: str) -> tuple[str, ...]:
+    """Return the canonical ArchiveResult name and its declared historic aliases."""
+    canonical = get_plugin_name(plugin_name)
+    aliases = get_archive_result_aliases()
+    return tuple(dict.fromkeys((canonical, plugin_name, *(alias for alias, target in aliases.items() if target == canonical))))
+
+
+def plugin_card_is_interactive(plugin_name: str) -> bool:
+    plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
+    return bool(plugin and plugin.manifest.get("card_interactive"))
+
+
+def get_plugin_output_extension_preference(plugin_name: str) -> tuple[tuple[str, ...], ...] | None:
+    plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
+    raw_groups = plugin.manifest.get("output_extension_preference") if plugin else None
+    if not isinstance(raw_groups, list):
+        return None
+    groups = []
+    for raw_group in raw_groups:
+        if not isinstance(raw_group, list):
+            continue
+        group = tuple(str(extension).lower() for extension in raw_group if str(extension).startswith("."))
+        if group:
+            groups.append(group)
+    return tuple(groups) or None
+
+
+def get_plugin_default_output_path(plugin_name: str) -> str | None:
+    prefix, separator, base_name = plugin_name.partition("_")
+    archive_result_name = base_name if separator and prefix.isdigit() else plugin_name
+    plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
+    raw_path = str(plugin.manifest.get("default_output_path") or "").strip() if plugin else ""
+    path = PurePosixPath(raw_path)
+    if not raw_path or raw_path.startswith("/") or ".." in path.parts or len(path.parts) != 1:
+        return None
+    return f"{archive_result_name}/{raw_path}"
+
+
+@lru_cache(maxsize=1)
+def get_snapshot_thumbnail_card_order() -> dict[str, int]:
+    """Return plugin-owned ArchiveResult names eligible for snapshot thumbnails."""
+    ordering: dict[str, int] = {}
+    for plugin in get_plugin_catalog().values():
+        declarations = plugin.manifest.get("snapshot_thumbnail_cards") or []
+        if not isinstance(declarations, list):
+            continue
+        for declaration in declarations:
+            if not isinstance(declaration, dict):
+                continue
+            result_name = str(declaration.get("archive_result_name") or "").strip()
+            order = declaration.get("order")
+            if not result_name or isinstance(order, bool) or not isinstance(order, int):
+                continue
+            ordering[result_name] = order
+    return ordering
+
+
+@lru_cache(maxsize=None)
+def get_snapshot_role_names(role: str) -> tuple[str, ...]:
+    """Return ArchiveResult names for a plugin-owned snapshot presentation role."""
+    manifest_key = f"snapshot_{role}"
+    names = []
+    for plugin in get_plugin_catalog().values():
+        if plugin.manifest.get(manifest_key):
+            names.extend(get_archive_result_names(plugin.name))
+    return tuple(dict.fromkeys(names))
+
+
+def get_snapshot_role_result(results, role: str):
+    """Select the first successful result declared for a presentation role."""
+    results_by_name = {result.plugin: result for result in results if result.status == "succeeded"}
+    return next((results_by_name[name] for name in get_snapshot_role_names(role) if name in results_by_name), None)
+
+
+@lru_cache(maxsize=1)
+def get_plugin_presentation_modules() -> tuple[ModuleType, ...]:
+    """Load optional plugin-owned presentation hooks declared in manifests."""
+    return tuple(module for plugin in get_plugin_catalog().values() if (module := get_plugin_presentation_module(plugin.name)) is not None)
+
+
+@lru_cache(maxsize=None)
+def get_plugin_presentation_module(plugin_name: str) -> ModuleType | None:
+    """Load one plugin's optional presentation hooks module."""
+    plugin = get_plugin_catalog().get(get_plugin_name(plugin_name))
+    module_name = str(plugin.manifest.get("presentation_module") or "").strip() if plugin else ""
+    return import_module(module_name) if module_name else None
+
+
+def plugin_has_custom_full_response(plugin_name: str) -> bool:
+    """Return whether a plugin renders its full template through a response hook."""
+    module = get_plugin_presentation_module(plugin_name)
+    return bool(module and getattr(module, "render_full_response", None))
+
+
+def get_extra_snapshot_output_cards(outputs: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    """Collect additional cards from plugin-owned presentation hooks."""
+    cards = []
+    for module in get_plugin_presentation_modules():
+        hook = getattr(module, "extra_snapshot_output_cards", None)
+        if hook is not None:
+            cards.extend(hook(outputs) or ())
+    return cards
+
+
+def serve_plugin_replay_asset(path: str, config, response_class):
+    """Give plugin presentation hooks the first chance to serve replay assets."""
+    for module in get_plugin_presentation_modules():
+        hook = getattr(module, "serve_replay_asset_response", None)
+        if hook is not None and (response := hook(path, config, response_class)) is not None:
+            return response
+    return None
+
+
+def render_plugin_full_response(path: str, output_url: str, **kwargs):
+    """Let plugin presentation hooks render a saved output's full template."""
+    for module in get_plugin_presentation_modules():
+        render = getattr(module, "render_full_response", None)
+        if render is not None and (response := render(path, output_url, **kwargs)) is not None:
+            return response
+    return None


 def get_enabled_plugins(config: ConfigLookup | None = None, **config_kwargs: Any) -> list[str]:
@@ -155,8 +290,6 @@ def get_plugin_template(plugin: str, template_name: str, fallback: bool = True)
        fallback: If True, return default template if plugin template not found
    """
    base_name = get_plugin_name(plugin)
-    if base_name in ("yt-dlp", "youtube-dl"):
-        base_name = "ytdlp"

    catalog = get_plugin_catalog()
    if base_name in catalog:
@@ -1,70 +1,83 @@
 """Snapshot output taxonomy. Display preference is independent of hook order."""

-from pathlib import PurePosixPath
 from typing import Any

-from archivebox.plugins.discovery import get_plugin_name
+from archivebox.plugins.discovery import (
+    get_extra_snapshot_output_cards,
+    get_plugin_catalog,
+    get_plugin_name,
+    get_snapshot_thumbnail_card_order,
+)

-# Preferred outputs come first; remaining outputs sort by total output bytes.
-OUTPUT_GROUPS = (
-    ("html", "HTML", ("archivewebpage", "singlefile", "chrome_mhtml", "wget", "dom")),
-    ("raster", "Raster", ("screenshot", "pdf")),
-    ("article_text", "Article text", ("readability", "defuddle", "mercury", "trafilatura", "htmltotext")),
-    ("embedded_media", "Embedded media", ("papersdl",)),
-    ("metadata", "Metadata", ()),
-    ("other", "Other files", ()),
+_GROUP_LABELS = (
+    ("html", "HTML"),
+    ("raster", "Raster"),
+    ("article_text", "Article text"),
+    ("embedded_media", "Embedded media"),
+    ("metadata", "Metadata"),
+    ("other", "Other files"),
+)
+
+
+def _plugin_presentation() -> dict[str, dict[str, Any]]:
+    presentation = {}
+    for plugin in get_plugin_catalog().values():
+        manifest = plugin.manifest
+        order = manifest.get("snapshot_output_order")
+        presentation[plugin.name] = {
+            "group": manifest.get("snapshot_output_group") or "other",
+            "order": order if isinstance(order, int) and not isinstance(order, bool) else None,
+            "display_name": manifest.get("snapshot_display_name"),
+            "icon_hidden": bool(manifest.get("icon_hidden")),
+            "card_hidden": bool(manifest.get("card_hidden")),
+        }
+    return presentation
+
+
+PLUGIN_PRESENTATION = _plugin_presentation()
+OUTPUT_GROUPS = tuple(
+    (
+        group_id,
+        label,
+        tuple(
+            name
+            for name, metadata in sorted(
+                PLUGIN_PRESENTATION.items(),
+                key=lambda item: (item[1]["order"] is None, item[1]["order"] or 0, item[0]),
+            )
+            if metadata["group"] == group_id and metadata["order"] is not None
+        ),
+    )
+    for group_id, label in _GROUP_LABELS
 )
 UNORDERED_OUTPUTS = {
-    "embedded_media": {"responses", "ytdlp", "yt-dlp", "youtube-dl", "gallerydl", "forumdl", "git", "media", "liteparse"},
-    "metadata": {
-        "dns",
-        "headers",
-        "seo",
-        "accessibility",
-        "redirects",
-        "consolelog",
-        "sslcerts",
-        "ssl",
-        "title",
-        "favicon",
-        "hashes",
-        "chrome",
-        "parse_html_urls",
-        "parse_txt_urls",
-        "parse_dom_outlinks",
-        "parse_jsonl_urls",
-        "parse_netscape_urls",
-        "parse_rss_urls",
-        "tlsnotary",
-        "opentimestamps",
-    },
-}
-
-COLOR_ICON_PLUGINS = frozenset(
-    {"singlefile", "archivewebpage", "screenshot", "ytdlp", "readability", "forumdl", "papersdl", "git", "gallerydl"},
-)
-HIDDEN_ICON_PLUGINS = frozenset({"archivedotorg"})
-PLUGIN_DISPLAY_NAMES = {
-    "singlefile": "SingleFile",
-    "archivewebpage": "ArchiveWebpage",
-    "ytdlp": "YouTube-DL",
-    "readability": "Readability",
-    "forumdl": "Forum-DL",
-    "git": "Git",
-    "gallerydl": "Gallery-DL",
+    group_id: {name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["group"] == group_id and metadata["order"] is None}
+    for group_id, _label in _GROUP_LABELS
 }
+HIDDEN_ICON_PLUGINS = frozenset(name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["icon_hidden"])
+HIDDEN_CARD_PLUGINS = frozenset(name for name, metadata in PLUGIN_PRESENTATION.items() if metadata["card_hidden"])


 def output_group_for_plugin(plugin: str) -> str:
    """Return the snapshot-detail stack that contains a plugin output."""
+    plugin = get_plugin_name(plugin)
    for group_id, _, plugins in OUTPUT_GROUPS:
        if plugin in plugins or plugin in UNORDERED_OUTPUTS.get(group_id, set()):
            return group_id
    return "other"


+def plugin_icon_is_hidden(plugin: str) -> bool:
+    return get_plugin_name(plugin) in HIDDEN_ICON_PLUGINS
+
+
+def plugin_card_is_hidden(plugin: str) -> bool:
+    return get_plugin_name(plugin) in HIDDEN_CARD_PLUGINS
+
+
 def display_plugin_name(plugin: str) -> str:
-    return PLUGIN_DISPLAY_NAMES.get(plugin, plugin.replace("_", " ").replace("-", " ").title())
+    canonical = get_plugin_name(plugin)
+    return PLUGIN_PRESENTATION.get(canonical, {}).get("display_name") or plugin.replace("_", " ").replace("-", " ").title()


 def plugin_output_sizes(results) -> dict[str, int]:
@@ -82,34 +95,21 @@ def order_output_plugins(plugins, sizes: dict[str, int]) -> list[str]:
    return [output["name"] for output in order_snapshot_outputs([{"name": plugin, "size": sizes.get(plugin, 0)} for plugin in plugins])]


-def _responses_html_card(outputs: list[dict[str, Any]]) -> dict[str, Any] | None:
-    # Presentation-only exception to the usual 1:1 plugin:card rule. Responses
-    # owns both this HTML view and its media gallery; responses_html is only a
-    # unique card/anchor ID, never a plugin or a synthetic ArchiveResult.
-    # Refactor this exception if plugins gain support for multiple cards.
-    for output in outputs:
-        if output["name"] != "responses" or not (result := output.get("result")):
+def snapshot_thumbnail_result(results):
+    """Select the first available card using plugin-owned thumbnail priority."""
+    card_order = get_snapshot_thumbnail_card_order()
+    candidates = []
+    for result in results:
+        if result.plugin not in card_order:
            continue
-        path = str(output.get("path") or "")
-        if not path.startswith("responses/") or ".." in PurePosixPath(path).parts:
+        try:
+            path = result.embed_path()
+        except (TypeError, ValueError, OverflowError):
            continue
-        files = result.output_file_map()
-        info = files.get(path) or files.get(path.removeprefix("responses/")) or {}
-        mime = str(info.get("mimetype") or "").split(";")[0].lower()
-        if not info or result._coerce_output_file_size(info.get("size")) <= 0:
+        if not path:
            continue
-        if mime not in {"text/html", "application/xhtml+xml"} and PurePosixPath(path).suffix.lower() not in {".html", ".htm"}:
-            continue
-        return {
-            "name": "responses_html",
-            "path": path,
-            "folder_path": "responses",
-            "ts": output.get("ts"),
-            "size": 0,  # Bytes are already counted by the real Responses card.
-            "result": None,
-            "direct_preview_path": f"{path}?card=responses_html",
-        }
-    return None
+        candidates.append((card_order[result.plugin], str(result.id), result))
+    return min(candidates, default=(None, None, None))[2]


 def order_snapshot_outputs(outputs: list[dict[str, Any]]) -> list[dict[str, Any]]:
@@ -123,13 +123,17 @@ def order_snapshot_outputs(outputs: list[dict[str, Any]]) -> list[dict[str, Any]
        for plugin in plugins:
            preferences[plugin] = (group_id, None)

-    if card := _responses_html_card(outputs):
-        outputs = [*outputs, card]
-    preferences["responses_html"] = ("html", len(OUTPUT_GROUPS[0][2]))
+    if cards := get_extra_snapshot_output_cards(outputs):
+        outputs = [*outputs, *cards]

    def sort_key(output):
        plugin = get_plugin_name(output["name"])
-        group_id, rank = preferences.get(plugin, ("other", None))
+        declared_group = output.get("output_group")
+        declared_order = output.get("output_order")
+        if declared_group in group_order and isinstance(declared_order, int) and not isinstance(declared_order, bool):
+            group_id, rank = declared_group, declared_order
+        else:
+            group_id, rank = preferences.get(plugin, ("other", None))
        output["output_group"] = group_id
        return (group_order[group_id], rank is None, rank if rank is not None else -int(output.get("size") or 0), plugin)

@@ -1,4 +1,15 @@
-"""Non-blocking, database-only collection totals for the progress monitor."""
+"""Cached display totals for progress polling and the public snapshot list.
+
+Missing/stale statistics are intentional: returning useful rows and progress
+quickly matters more than exact initial counts. Each viewer may poll once per
+second; repeating aggregates per request would consume the small server's CPU
+and hold SQLite read locks needed by the archiving runner. Callers must accept
+None on a cold cache and must never wait for the refresh or count synchronously.
+
+Background reads still take normal SQLite read locks. Existing indexes, short
+batches where needed, and throttled refreshes limit their cost; moving work to
+a thread does not make a scan free or require switching the database to WAL.
+"""

 import logging
 from threading import Lock, Thread
@@ -9,14 +20,77 @@ from django.db import close_old_connections, connections
 from django.db.models import Count, Sum


+# This is only a process-local gate for background work, not a DB/file lock.
+# Every request tries it without waiting; a busy refresh means returning the
+# cached value (or None). Serializing refreshes also avoids competing scans for
+# different crawls/users when many viewer tabs open at once.
 _refresh_lock = Lock()


-def collection_summary(user):
-    user_id = None if user.is_superuser else user.pk
-    key = f"progress-collection:{'all' if user_id is None else user_id}"
+def crawl_summary(crawl_id, *, snapshot_id=None):
+    """Sample sealed snapshot metadata without adding table scans to polls."""
+    key = f"progress-crawl:{crawl_id}:{snapshot_id or 'all'}"
    summary = cache.get(key)
-    # Even an indexed SUM scans the index. Never make an HTTP request wait for
+    if (summary is None or time() - summary["sampled_at"] >= 30) and _refresh_lock.acquire(blocking=False):
+        if not cache.add(f"{key}:refresh", True, timeout=30):
+            _refresh_lock.release()
+            return summary
+
+        def refresh():
+            try:
+                close_old_connections()
+                from archivebox.core.models import Snapshot
+
+                snapshots = Snapshot.objects.filter(crawl_id=crawl_id, status=Snapshot.StatusChoices.SEALED)
+                if snapshot_id is not None:
+                    snapshots = snapshots.filter(pk=snapshot_id)
+                snapshots = snapshots.order_by("modified_at").values_list("modified_at", "downloaded_at", "output_size")
+                totals = {"snapshots": 0, "cancelled": 0, "bytes": 0}
+                modified_at, offset = None, 0
+                while True:
+                    batch_scope = snapshots.filter(modified_at__gte=modified_at) if modified_at is not None else snapshots
+                    # Use (crawl, status, modified_at), releasing the SQLite read
+                    # cursor after each small batch. Never hold a cursor over the
+                    # whole crawl, even in this background worker.
+                    batch = list(batch_scope[offset : offset + 100])
+                    for _, downloaded_at, size in batch:
+                        totals["snapshots"] += 1
+                        totals["cancelled"] += downloaded_at is None
+                        totals["bytes"] += size or 0
+                    if len(batch) < 100:
+                        break
+                    last_modified_at = batch[-1][0]
+                    # Only offset within ties; otherwise advance the indexed
+                    # timestamp range instead of rescanning previous pages.
+                    offset = (offset if last_modified_at == modified_at else 0) + sum(row[0] == last_modified_at for row in batch)
+                    modified_at = last_modified_at
+                totals["sampled_at"] = time()
+                cache.set(key, totals, timeout=3600)
+            except Exception:
+                logging.getLogger(__name__).exception("Could not refresh crawl totals")
+            finally:
+                connections.close_all()
+                _refresh_lock.release()
+
+        try:
+            Thread(target=refresh, name="crawl-summary", daemon=True).start()
+        except RuntimeError:
+            _refresh_lock.release()
+            logging.getLogger(__name__).exception("Could not start crawl totals refresh")
+    return summary
+
+
+def collection_summary(user, *, public=False):
+    """Return cached totals immediately and, when due, start one refresh.
+
+    Public pages need no extra AJAX/polling for these totals: an ordinary later
+    page load reads the populated cache. Keep this best-effort display cache
+    separate from pagination/authorization decisions, which need actual rows.
+    """
+    user_id = None if user.is_superuser else user.pk
+    key = "progress-collection:public" if public else f"progress-collection:{'all' if user_id is None else user_id}"
+    summary = cache.get(key)
+    # Even indexed COUNT/SUM aggregates scan the index. Never make HTTP wait for
    # that scan on a large collection. Serve stale totals while one worker refreshes;
    # a single non-blocking lock prevents a queue of scans from concurrent clients.
    if (summary is None or time() - summary["sampled_at"] >= 60) and _refresh_lock.acquire(blocking=False):
@@ -32,13 +106,29 @@ def collection_summary(user):
                from archivebox.core.models import Snapshot

                snapshots = Snapshot.objects.all()
-                if user_id is not None:
-                    snapshots = snapshots.filter(crawl__created_by_id=user_id)
-                # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
-                # Snapshot.output_size is maintained when ArchiveResult sizes change.
-                totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
-                totals["bytes"] = totals["bytes"] or 0
+                if public:
+                    from archivebox.core.permissions import PERMISSIONS_PUBLIC
+
+                    # SQLite can choose a table scan for equality after ANALYZE
+                    # on an all-public archive. Equal range bounds keep the count
+                    # on the existing permissions index. It is still O(n), so
+                    # public pages share this background cache, never await it.
+                    # Count the indexed, non-null permissions column so this
+                    # needs no Snapshot table reads or new covering-index migration.
+                    totals = snapshots.filter(permissions__gte=PERMISSIONS_PUBLIC, permissions__lte=PERMISSIONS_PUBLIC).aggregate(
+                        snapshots=Count("permissions"),
+                    )
+                else:
+                    if user_id is not None:
+                        snapshots = snapshots.filter(crawl__created_by_id=user_id)
+                    # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
+                    # Snapshot.output_size is maintained when ArchiveResult sizes change.
+                    totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+                totals["bytes"] = totals.get("bytes") or 0
                totals["sampled_at"] = time()
+                # Retain old totals longer than the 60s refresh interval so a
+                # slow/failed refresh does not force every tab back to a cold
+                # count. Staleness is acceptable; blocking page loads is not.
                cache.set(key, totals, timeout=3600)
            except Exception:
                logging.getLogger(__name__).exception("Could not refresh collection totals")
@@ -1904,12 +1904,22 @@
    const progressEndpoint = monitor.dataset.progressEndpoint || '/progress.json';

    function fetchProgress() {
+        // Opening a replay replaces the queued snapshot's progress panel.
+        // Its interval and already-running fetch must stop using that old DOM.
+        if (!monitor.isConnected) {
+            stopPolling();
+            return;
+        }
        fetch(progressEndpoint, { credentials: 'same-origin' })
            .then(response => {
                if (!response.ok) throw new Error(`HTTP ${response.status}`);
                return response.json();
            })
            .then(data => {
+                if (!monitor.isConnected) {
+                    stopPolling();
+                    return;
+                }
                if (data.error) {
                    console.error('Progress API error:', data.error, data.traceback);
                    setOrchestratorState('error', 'Backend not responding');
@@ -1921,6 +1931,10 @@
                updateProgress(data);
            })
            .catch(error => {
+                if (!monitor.isConnected) {
+                    stopPolling();
+                    return;
+                }
                console.error('Progress fetch error:', error);
                setOrchestratorState('error', 'Backend not responding');
                idleMessage.textContent = 'Fetch Error: ' + error.message;
@@ -1929,7 +1943,7 @@
    }

    function startPolling() {
-        if (pollInterval) return;
+        if (pollInterval || !monitor.isConnected) return;
        fetchProgress();
        pollInterval = setInterval(fetchProgress, pollDelayMs);
    }
@@ -2033,7 +2047,13 @@

    // Start polling when page loads
    startPolling();
-    setInterval(updateDurationBadges, 1000);
+    const durationInterval = setInterval(() => {
+        if (!monitor.isConnected) {
+            clearInterval(durationInterval);
+            return;
+        }
+        updateDurationBadges();
+    }, 1000);

    // Pause polling when tab is hidden
    document.addEventListener('visibilitychange', function() {
@@ -7,17 +7,15 @@ from typing import Literal
 from abx_dl.events import PROCESS_EXIT_SKIPPED
 from django.conf import settings
 from django.db import DatabaseError
-from django.db.models import CharField, Count, Q, Sum
+from django.db.models import CharField, Count, Q
 from django.db.models.functions import Cast
 from django.http import HttpResponse, JsonResponse
 from django.utils import timezone

-from archivebox.config import CONSTANTS
-from archivebox.config.common import get_config
 from archivebox.core.permissions import can_view_snapshot, is_admin_user
-from archivebox.core.routes_util import build_snapshot_url, build_web_url, get_api_base_url
+from archivebox.core.routes_util import build_snapshot_detail_url, build_snapshot_url, get_api_base_url
 from archivebox.misc.logging_util import printable_filesize
-from archivebox.plugins.discovery import discover_plugin_configs
+from archivebox.plugins.discovery import discover_plugin_configs, get_plugin_default_output_path, get_snapshot_role_result


 def progress_endpoint(scope: Literal["crawl", "snapshot"] | None = None, object_id: object | None = None) -> str:
@@ -152,68 +150,25 @@ def live_progress_view(request, *, authorized_snapshot=None):

            return hook_details(Path(hook_path).name, plugin=Path(hook_path).parent.name or "setup")

-        def archiveresult_output_path(ar) -> str | None:
-            output_file_map = ar.output_files if isinstance(ar.output_files, dict) else {}
-
-            def is_root_relative(path: str) -> bool:
-                metadata = output_file_map.get(path) or {}
-                return bool(isinstance(metadata, dict) and metadata.get("root_relative"))
-
-            if ar.output_str:
-                raw_output = str(ar.output_str).strip()
-                if ar._looks_like_output_path(raw_output, ar.plugin):
-                    output_path = Path(raw_output)
-                    if output_path.is_absolute():
-                        return None
-
-                    if raw_output.startswith(f"{ar.plugin}/"):
-                        candidates = [raw_output]
-                    elif len(output_path.parts) == 1:
-                        candidates = [f"{ar.plugin}/{raw_output}", raw_output]
-                    else:
-                        candidates = [raw_output]
-
-                    if raw_output in output_file_map and is_root_relative(raw_output):
-                        return raw_output
-
-                    for relative_path in candidates:
-                        plugin_relative = relative_path.removeprefix(f"{ar.plugin}/")
-                        if relative_path in output_file_map:
-                            return f"{ar.plugin}/{relative_path}" if not relative_path.startswith(f"{ar.plugin}/") else relative_path
-                        if plugin_relative in output_file_map:
-                            return f"{ar.plugin}/{plugin_relative}"
-
-            output_file_paths = list(output_file_map.keys())
-            if output_file_paths:
-                fallback_path = ArchiveResult._fallback_output_file_path(output_file_paths, ar.plugin, output_file_map)
-                if fallback_path:
-                    if is_root_relative(fallback_path):
-                        return fallback_path
-                    return f"{ar.plugin}/{fallback_path}"
-
-            return None
-
        def snapshot_output_url(snapshot, output_path: str) -> str:
            return build_snapshot_url(str(snapshot["id"]), output_path, request=request, config=request_config)

        def snapshot_archive_path(snapshot) -> str:
-            if snapshot["fs_version"] in ("0.7.0", "0.8.0"):
-                return f"{CONSTANTS.ARCHIVE_DIR_NAME}/{snapshot['timestamp']}"
            crawl = crawls_by_id.get(str(snapshot["crawl_id"]))
-            username = "web"
-            if crawl is not None and crawl["created_by_id"]:
-                username = crawl["created_by__username"]
-            if username == "system":
-                username = "web"
-            date_base = snapshot["bookmarked_at"] or snapshot["created_at"]
-            date_str = date_base.strftime("%Y%m%d") if date_base else "unknown"
-            domain = Snapshot.extract_domain_from_url(snapshot["url"])
-            return f"{username}/{date_str}/{domain}/{snapshot['id']}"
+            return Snapshot.archive_path_for_values(
+                fs_version=snapshot["fs_version"],
+                timestamp=snapshot["timestamp"],
+                snapshot_id=snapshot["id"],
+                url=snapshot["url"],
+                username=crawl["created_by__username"] if crawl is not None and crawl["created_by_id"] else "web",
+                bookmarked_at=snapshot["bookmarked_at"],
+                created_at=snapshot["created_at"],
+            )

        def snapshot_view_url(snapshot, output_path: str = "") -> str:
-            anchor = f"#{output_path}" if output_path else ""
-            return build_web_url(
-                f"/{snapshot_archive_path(snapshot)}/index.html{anchor}",
+            return build_snapshot_detail_url(
+                snapshot_archive_path(snapshot),
+                output_path=output_path,
                request=request,
                config=request_config,
            )
@@ -226,53 +181,37 @@ def live_progress_view(request, *, authorized_snapshot=None):

        running_screencasts = {}

-        def screencast_frame_url(crawl_id: str, crawl_dir: Path) -> str:
-            frame_path = crawl_dir / "chrome_screencast" / "latest.jpg"
-            try:
-                frame_stat = frame_path.stat()
-            except OSError:
-                return ""
-            if frame_stat.st_size <= 0:
-                return ""
+        def screencast_frame_url(crawl_id: str) -> str:
            started_at = running_screencasts.get(crawl_id)
-            if started_at is None or frame_stat.st_mtime < started_at.timestamp():
+            if started_at is None:
                return ""
-            rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={frame_stat.st_mtime_ns}"
+            # File existence/freshness belongs to the image request, never to
+            # the one-second JSON poll (archive storage may be remote).
+            rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={int(now.timestamp())}&after={started_at.timestamp()}"
            return f"{api_base}{rel}" if api_base else rel

-        machine_id = Machine.current().id
+        machine = Machine.current_readonly()
+        machine_id = machine.id if machine is not None else None
        orchestrator_proc = (
            Process.objects.filter(
                machine_id=machine_id,
                process_type=Process.TypeChoices.ORCHESTRATOR,
                status=Process.StatusChoices.RUNNING,
            )
-            .only("id", "pid", "started_at", "machine_id", "process_type", "status")
+            .only("id", "pid", "started_at", "machine_id", "process_type", "status", "env", "cmd")
            .order_by("-started_at")
            .first()
            if machine_id is not None
            else None
        )
-        runner_worker = None
        orchestrator_proc_running = bool(orchestrator_proc and orchestrator_proc.is_running)
-        if not orchestrator_proc_running:
-            try:
-                from archivebox.workers.supervisord_util import get_existing_supervisord_process, get_worker
-
-                supervisor = get_existing_supervisord_process(quiet=True)
-                runner_worker = get_worker(supervisor, "worker_runner") if supervisor else None
-            except (OSError, RuntimeError, TimeoutError):
-                runner_worker = None
-
-        runner_worker_running = bool(runner_worker and runner_worker.get("statename") in ("STARTING", "RUNNING"))
-        runner_worker_pid = runner_worker.get("pid") if runner_worker else None
-        orchestrator_running = orchestrator_proc_running or runner_worker_running
-        orchestrator_pid = orchestrator_proc.pid if orchestrator_proc_running and orchestrator_proc else runner_worker_pid
+        orchestrator_running = orchestrator_proc_running
+        orchestrator_pid = orchestrator_proc.pid if orchestrator_proc_running and orchestrator_proc else None

        # Get model counts by status
        crawl_status_counts = Crawl.status_counts(
            crawl_scope,
-            (Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.PAUSED),
+            (Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED),
        )
        crawls_queued = crawl_status_counts.get(Crawl.StatusChoices.QUEUED, 0)
        crawls_active = crawl_status_counts.get(Crawl.StatusChoices.STARTED, 0)
@@ -286,7 +225,7 @@ def live_progress_view(request, *, authorized_snapshot=None):

        snapshot_status_counts = Snapshot.status_counts(
            snapshot_scope,
-            Snapshot.OPEN_STATES,
+            Snapshot.RUNNABLE_STATES,
        )
        snapshots_queued = snapshot_status_counts.get(Snapshot.StatusChoices.QUEUED, 0)
        snapshots_active = snapshot_status_counts.get(Snapshot.StatusChoices.STARTED, 0)
@@ -296,15 +235,24 @@ def live_progress_view(request, *, authorized_snapshot=None):
            ArchiveResult.StatusChoices.QUEUED,
            ArchiveResult.StatusChoices.STARTED,
        )
-        archiveresult_status_counts = ArchiveResult.status_counts(archiveresult_scope, result_statuses)
+        # Start from open results using the existing status index. Filtering by
+        # plugin first scans historical runs, including all completed results.
+        result_counts = list(
+            archiveresult_scope.filter(status__in=result_statuses).order_by().values("status", "plugin").annotate(count=Count("*")),
+        )
+        archiveresult_status_counts = {
+            status: sum(row["count"] for row in result_counts if row["status"] == status) for status in result_statuses
+        }
        download_scope = archiveresult_scope.filter(
            plugin__in=download_plugin_names,
            snapshot__status__in=Snapshot.RUNNABLE_STATES,
            snapshot__crawl__status__in=Crawl.RUNNABLE_STATES,
        )
-        indexing_scope = archiveresult_scope.filter(plugin__in=indexing_plugin_names)
        download_status_counts = ArchiveResult.status_counts(download_scope, result_statuses)
-        indexing_status_counts = ArchiveResult.status_counts(indexing_scope, result_statuses)
+        indexing_status_counts = {
+            status: sum(row["count"] for row in result_counts if row["status"] == status and row["plugin"] in indexing_plugin_names)
+            for status in result_statuses
+        }
        archiveresults_queued = archiveresult_status_counts.get(ArchiveResult.StatusChoices.QUEUED, 0)
        archiveresults_active = archiveresult_status_counts.get(ArchiveResult.StatusChoices.STARTED, 0)

@@ -340,21 +288,27 @@ def live_progress_view(request, *, authorized_snapshot=None):
            .values(*active_crawl_fields)
            .order_by("-modified_at")[:max_active_crawls],
        )
-        paused_crawls = list(
-            crawl_scope.filter(
-                Q(status=Crawl.StatusChoices.PAUSED, created_at__gte=paused_crawl_cutoff)
-                | Q(
-                    status=Crawl.StatusChoices.PAUSED,
-                    snapshot_set__status__in=Snapshot.RUNNABLE_STATES,
-                    snapshot_set__retry_at__lte=now,
-                )
-                | Q(
-                    status=Crawl.StatusChoices.PAUSED,
-                    snapshot_set__archiveresult__status=ArchiveResult.StatusChoices.QUEUED,
-                ),
+        # Start with runnable children, not every historical snapshot belonging
+        # to a paused crawl. These uncorrelated subqueries run once per poll.
+        due_paused_snapshots = (
+            Snapshot.objects.filter(
+                status__in=Snapshot.RUNNABLE_STATES,
+                retry_at__lte=now,
            )
+            .order_by()
+            .values("crawl_id")
+        )
+        queued_paused_results = (
+            ArchiveResult.objects.filter(
+                status=ArchiveResult.StatusChoices.QUEUED,
+            )
+            .order_by()
+            .values("snapshot__crawl_id")
+        )
+        paused_crawls = list(
+            crawl_scope.filter(status=Crawl.StatusChoices.PAUSED)
+            .filter(Q(created_at__gte=paused_crawl_cutoff) | Q(id__in=due_paused_snapshots) | Q(id__in=queued_paused_results))
            .values(*active_crawl_fields)
-            .distinct()
            .order_by("-modified_at")[:max_active_crawls],
        )
        queued_crawls = list(
@@ -368,52 +322,29 @@ def live_progress_view(request, *, authorized_snapshot=None):
                crawl["persona_id"] = str(crawl["persona_id"])
        persona_details_by_id: dict[str, dict[str, str]] = {}
        persona_details_by_name: dict[str, dict[str, str]] = {}
-        persona_objects_by_id = {}
-        persona_objects_by_name = {}
        persona_ids = {crawl["persona_id"] for crawl in active_crawls_list if crawl["persona_id"]}
        persona_names = {"Default"} if any(not crawl["persona_id"] for crawl in active_crawls_list) else set()
        if persona_ids or persona_names:
            from archivebox.personas.models import Persona

-            for persona in Persona.objects.filter(Q(id__in=persona_ids) | Q(name__in=persona_names)).only("id", "name", "config"):
+            for persona in Persona.objects.filter(Q(id__in=persona_ids) | Q(name__in=persona_names)).only("id", "name"):
                persona_details = {
                    "name": persona.name,
                    "admin_url": f"/admin/personas/persona/{persona.pk}/change/",
                }
                persona_details_by_id[str(persona.id)] = persona_details
                persona_details_by_name[persona.name] = persona_details
-                persona_objects_by_id[str(persona.id)] = persona
-                persona_objects_by_name[persona.name] = persona
        active_crawl_ids = [crawl["id"] for crawl in active_crawls_list]
-        active_crawl_objects = {}
-        if active_crawl_ids:
-            for crawl_obj in Crawl.objects.filter(id__in=active_crawl_ids).select_related("created_by", "persona"):
-                crawl_obj._runtime_config = request_config
-                active_crawl_objects[str(crawl_obj.id)] = crawl_obj
        snapshot_counts_by_crawl: dict[str, dict[str, int]] = {str(crawl_id): {} for crawl_id in active_crawl_ids}
-        cancelled_snapshot_counts_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
-        crawl_output_sizes_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
        queued_snapshot_overflow_by_crawl: dict[str, int] = {str(crawl_id): 0 for crawl_id in active_crawl_ids}
        active_snapshot_scope = snapshot_scope.filter(crawl_id__in=active_crawl_ids)
        if active_crawl_ids:
-            for row in active_snapshot_scope.values("crawl_id", "status").annotate(count=Count("id")):
-                snapshot_counts_by_crawl.setdefault(str(row["crawl_id"]), {})[row["status"]] = row["count"]
-
-            for row in (
-                active_snapshot_scope.filter(status=Snapshot.StatusChoices.SEALED, downloaded_at__isnull=True)
-                .values("crawl_id")
-                .annotate(count=Count("id"))
-            ):
-                cancelled_snapshot_counts_by_crawl[str(row["crawl_id"])] = row["count"]
-
-            for row in (
-                active_snapshot_scope.filter(
-                    status=Snapshot.StatusChoices.SEALED,
-                )
-                .values("crawl_id")
-                .annotate(size=Sum("output_size"))
-            ):
-                crawl_output_sizes_by_crawl[str(row["crawl_id"])] = int(row["size"] or 0)
+            # The existing (crawl, status, modified_at) index covers these counts.
+            # Byte totals and cancellation details require table reads, so sample
+            # those separately and share cached values between viewer tabs.
+            for row in active_snapshot_scope.order_by().values("crawl_id", "status").annotate(count=Count("*")):
+                crawl_id = str(row["crawl_id"])
+                snapshot_counts_by_crawl[crawl_id][row["status"]] = row["count"]

        crawl_process_pids: dict[str, int] = {}
        snapshot_process_pids: dict[str, int] = {}
@@ -632,7 +563,14 @@ def live_progress_view(request, *, authorized_snapshot=None):
            completed_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.SEALED, 0)
            started_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.STARTED, 0)
            pending_snapshots = crawl_snapshot_counts.get(Snapshot.StatusChoices.QUEUED, 0)
-            cancelled_snapshots = cancelled_snapshot_counts_by_crawl.get(crawl_id, 0)
+            from archivebox.progressmonitor.collection import crawl_summary
+
+            totals = (
+                crawl_summary(crawl_id, snapshot_id=scoped_snapshot.pk if scoped_snapshot is not None else None)
+                if completed_snapshots
+                else {"snapshots": 0, "cancelled": 0, "bytes": 0}
+            )
+            cancelled_snapshots = totals["cancelled"] if totals is not None else None

            # Count URLs in the crawl (for when snapshots haven't been created yet)
            urls_count = 0
@@ -651,7 +589,7 @@ def live_progress_view(request, *, authorized_snapshot=None):
            crawl_setup_completed = sum(1 for item in crawl_setup_plugins if item.get("status") == "succeeded")
            crawl_setup_failed = sum(1 for item in crawl_setup_plugins if item.get("status") == "failed")
            crawl_setup_pending = sum(1 for item in crawl_setup_plugins if item.get("status") == "queued")
-            crawl_screencast_url = screencast_frame_url(crawl_id, active_crawl_objects[crawl_id].output_dir)
+            crawl_screencast_url = screencast_frame_url(crawl_id)
            crawl_screencast_link = f"/admin/crawls/crawl/{crawl_id.replace('-', '')}/change/" if crawl_screencast_url else ""

            # Get active snapshots for this crawl (already prefetched)
@@ -684,27 +622,28 @@ def live_progress_view(request, *, authorized_snapshot=None):
                snapshot_screencast_url = ""
                snapshot_screencast_link = ""
                snapshot_fallback_urls: list[str] = []
-                result_by_plugin = {result.plugin: result for result in snapshot_results}
-                title_result = result_by_plugin.get("title")
-                if not snapshot_title and title_result is not None and title_result.status == ArchiveResult.StatusChoices.SUCCEEDED:
+                title_result = get_snapshot_role_result(snapshot_results, "title_source")
+                if not snapshot_title and title_result is not None:
                    snapshot_title = Snapshot._normalize_title_candidate(title_result.output_str, snapshot_url=snapshot["url"])
-                favicon_result = result_by_plugin.get("favicon")
-                if favicon_result is not None and favicon_result.status == ArchiveResult.StatusChoices.SUCCEEDED:
-                    favicon_path = archiveresult_output_path(favicon_result) or "favicon/favicon.ico"
-                    snapshot_favicon_url = snapshot_output_url(snapshot, favicon_path)
-                screenshot_result = result_by_plugin.get("screenshot")
-                if screenshot_result is not None and screenshot_result.status == ArchiveResult.StatusChoices.SUCCEEDED:
+                favicon_result = get_snapshot_role_result(snapshot_results, "list_icon")
+                if favicon_result is not None:
+                    favicon_path = favicon_result.embed_path_db() or get_plugin_default_output_path(favicon_result.plugin)
+                    if favicon_path:
+                        snapshot_favicon_url = snapshot_output_url(snapshot, favicon_path)
+                screenshot_result = get_snapshot_role_result(snapshot_results, "primary_preview")
+                if screenshot_result is not None:
                    snapshot_preview_link = snapshot_view_url(snapshot)
-                    screenshot_path = archiveresult_output_path(screenshot_result) or "screenshot/screenshot.png"
-                    snapshot_preview_url = snapshot_output_url(snapshot, screenshot_path)
-                    snapshot_preview_link = snapshot_view_url(snapshot, screenshot_path)
+                    screenshot_path = screenshot_result.embed_path_db() or get_plugin_default_output_path(screenshot_result.plugin)
+                    if screenshot_path:
+                        snapshot_preview_url = snapshot_output_url(snapshot, screenshot_path)
+                        snapshot_preview_link = snapshot_view_url(snapshot, screenshot_path)
                    if snapshot_favicon_url:
                        snapshot_fallback_urls.append(snapshot_favicon_url)
                elif snapshot_favicon_url:
                    snapshot_preview_url = snapshot_favicon_url

                if snapshot["status"] == Snapshot.StatusChoices.STARTED:
-                    snapshot_screencast_url = screencast_frame_url(crawl_id, active_crawl_objects[crawl_id].output_dir)
+                    snapshot_screencast_url = screencast_frame_url(crawl_id)
                    snapshot_screencast_link = snapshot_view_url(snapshot) if snapshot_screencast_url else ""

                def plugin_sort_key(ar):
@@ -752,7 +691,7 @@ def live_progress_view(request, *, authorized_snapshot=None):
                        "process_id": str(process.id) if process else None,
                        "admin_url": f"/admin/core/archiveresult/{ar.id}/change/",
                    }
-                    output_path = archiveresult_output_path(ar)
+                    output_path = ar.embed_path_db()
                    if output_path:
                        plugin_payload["output_path"] = output_path
                        plugin_payload["output_url"] = snapshot_view_url(snapshot, output_path)
@@ -852,18 +791,18 @@ def live_progress_view(request, *, authorized_snapshot=None):
            persona_details = persona_details_by_id.get(str(crawl["persona_id"])) if crawl["persona_id"] else None
            persona_name = persona_details["name"] if persona_details else "Default"
            persona_details = persona_details or persona_details_by_name.get(persona_name)
-            crawl_output_size = crawl_output_sizes_by_crawl.get(crawl_id, 0)
-            avg_snapshot_size = int(crawl_output_size / completed_snapshots) if completed_snapshots else 0
-            crawl_obj = active_crawl_objects[crawl_id]
-            effective_crawl_config = get_config(crawl=crawl_obj, resolve_plugins=False)
-            max_urls = int(effective_crawl_config.CRAWL_MAX_URLS or 0)
-            crawl_max_size = int(effective_crawl_config.CRAWL_MAX_SIZE or 0)
-            crawl_timeout = int(effective_crawl_config.CRAWL_TIMEOUT or 0)
-            snapshot_max_size = int(effective_crawl_config.SNAPSHOT_MAX_SIZE or 0)
+            crawl_output_size = totals["bytes"] if totals is not None else None
+            avg_snapshot_size = int(totals["bytes"] / max(totals["snapshots"], 1)) if totals is not None else None
+            # These crawl-scope limits are frozen in Crawl.config. Rendering
+            # them must not resolve personas, probe binaries, or reread config.
+            max_urls = int(crawl["config"].get("CRAWL_MAX_URLS", request_config.CRAWL_MAX_URLS) or 0)
+            crawl_max_size = int(crawl["config"].get("CRAWL_MAX_SIZE", request_config.CRAWL_MAX_SIZE) or 0)
+            crawl_timeout = int(crawl["config"].get("CRAWL_TIMEOUT", request_config.CRAWL_TIMEOUT) or 0)
+            snapshot_max_size = int(crawl["config"].get("SNAPSHOT_MAX_SIZE", request_config.SNAPSHOT_MAX_SIZE) or 0)

            # Check if retry_at is in the future (would prevent worker from claiming)
            retry_at_future = crawl["retry_at"] > now if crawl["retry_at"] else False
-            is_paused = crawl_obj.is_paused
+            is_paused = crawl["status"] == Crawl.StatusChoices.PAUSED
            seconds_until_retry = (
                0 if is_paused else int((crawl["retry_at"] - now).total_seconds()) if crawl["retry_at"] and retry_at_future else 0
            )
@@ -903,8 +842,8 @@ def live_progress_view(request, *, authorized_snapshot=None):
                    "max_snapshot_size_display": printable_filesize(snapshot_max_size) if snapshot_max_size else "unlimited",
                    "crawl_output_size": crawl_output_size,
                    "avg_snapshot_size": avg_snapshot_size,
-                    "crawl_output_size_display": printable_filesize(crawl_output_size) if crawl_output_size else "0 B",
-                    "avg_snapshot_size_display": printable_filesize(avg_snapshot_size) if avg_snapshot_size else "0 B",
+                    "crawl_output_size_display": printable_filesize(crawl_output_size or 0) if totals is not None else "…",
+                    "avg_snapshot_size_display": printable_filesize(avg_snapshot_size or 0) if totals is not None else "…",
                    "tags": crawl_tags,
                    "urls_count": urls_count,
                    "total_snapshots": total_snapshots,
@@ -12,8 +12,11 @@ _search_backends_cache: dict | None = None

 def search_backend_command_env(config: dict[str, Any] | None = None, **config_kwargs: Any) -> dict[str, str]:
    """Serialize resolved application config for a standalone plugin command."""
+    from abx_dl.models import PluginEnv
+    from abxpkg import BinProvider
+
    config = config or get_config(**config_kwargs)
-    env = os.environ.copy()
+    env: dict[str, str] = {}
    for key, value in config.items():
        key = str(key)
        if value is None:
@@ -24,7 +27,9 @@ def search_backend_command_env(config: dict[str, Any] | None = None, **config_kw
            env[key] = json.dumps(value)
        elif isinstance(value, (str, int, float, os.PathLike)):
            env[key] = str(value)
-    return env
+    # Use the same runtime bootstrap as hooks so the installation's abxpkg
+    # entrypoint remains available even without an activated virtualenv.
+    return BinProvider.build_exec_env(base_env=PluginEnv(**os.environ).to_env(), extra_env=env)


 def normalize_search_backend_name(backend_name: str | None) -> str:
@@ -2,9 +2,14 @@

 import math
 import os
+import re
+import stat
+import sys
 import time
 from pathlib import Path

+import psutil
+

 RESOURCE_RECHECK_SECONDS = 2.0
 # Scheduling policy, not a minimum-RAM claim: stop increasing extraction
@@ -12,6 +17,65 @@ RESOURCE_RECHECK_SECONDS = 2.0
 # stalls. Require both recent and rolling pressure so transient reclaim does
 # not indefinitely starve otherwise viable work on small hosts.
 MEMORY_FULL_STALL_PERCENT = 10.0
+ZRAM_DEVICE_NAME = re.compile(r"zram\d+")
+
+
+def zram_swap_identifiers(sys_block: Path = Path("/sys/block")) -> set[str]:
+    """Return zram block names and major:minor IDs known to sysfs."""
+    identifiers = set()
+    for device in sys_block.glob("zram*"):
+        if ZRAM_DEVICE_NAME.fullmatch(device.name):
+            identifiers.add(device.name)
+            try:
+                identifiers.add((device / "dev").read_text().strip())
+            except OSError:
+                pass
+    return identifiers
+
+
+def swap_source_identifiers(filename: str) -> set[str]:
+    """Identify a swap source by its visible name and, for block devices, ID."""
+    identifiers = {Path(filename).name}
+    try:
+        source = os.stat(filename)
+        if stat.S_ISBLK(source.st_mode):
+            identifiers.add(f"{os.major(source.st_rdev)}:{os.minor(source.st_rdev)}")
+    except OSError:
+        pass
+    return identifiers
+
+
+def disk_backed_swap_free_bytes(swaps: str, zram_identifiers: set[str]) -> int:
+    """Sum free active swap except zram, whose capacity is already backed by RAM."""
+    lines = swaps.splitlines()
+    free_kib = 0
+    for line in lines[1:]:  # /proc/swaps starts with a column header
+        fields = line.split()
+        if len(fields) < 5:
+            continue
+        filename, _, size_kib, used_kib, _ = fields[:5]
+        source_ids = swap_source_identifiers(filename)
+        if source_ids & zram_identifiers or any(ZRAM_DEVICE_NAME.fullmatch(name) for name in source_ids):
+            continue
+        try:
+            free_kib += max(0, int(size_kib) - int(used_kib))
+        except ValueError:
+            continue
+    return free_kib * 1024
+
+
+def host_swap_free_bytes(meminfo_swap_free: int) -> int:
+    """Use only independently backed swap as extra capacity on Linux."""
+    if not sys.platform.startswith("linux"):
+        return psutil.swap_memory().free
+    try:
+        swaps = Path("/proc/swaps").read_text()
+    except OSError:
+        # If Linux swap sources cannot be classified, fail closed rather than
+        # treating potentially RAM-backed zram capacity as extra headroom.
+        return 0
+    disk_free = disk_backed_swap_free_bytes(swaps, zram_swap_identifiers())
+    return min(max(0, meminfo_swap_free), disk_free)


 class ResourceAdmission:
@@ -44,10 +108,73 @@ class ResourceAdmission:
                    cpus = min(cpus, max(1, math.ceil(int(quota) / int(period))))
            except (OSError, ValueError, ZeroDivisionError):
                continue
-        # CPU quota limits parallel admission, not forward progress. A host
-        # with a fractional CPU must still be able to run one snapshot.
+        # CPU quota bounds parallel work; measured memory admission is done
+        # only after the first complete snapshot establishes its actual cost.
        return max(1, min(configured, cpus))

+    def additional_snapshot_slots(self, configured: int, active: int, observed_cost: int | None) -> int:
+        if not observed_cost:
+            # Stage one complete real capture before spending memory on parallel
+            # hook trees. A zero-cost/failed probe cannot justify fanout.
+            return 1 if active == 0 and configured > 0 else 0
+        headroom = self.memory_headroom()
+        return self.slots_for_headroom(configured, active, observed_cost, headroom[1] if headroom else 0)
+
+    @staticmethod
+    def slots_for_headroom(configured: int, active: int, observed_cost: int, available: int) -> int:
+        slots = max(0, configured - active)
+        if observed_cost > 0:
+            # Active captures can still grow to the measured peak. Reserve for
+            # that growth before promising the remaining headroom to new work.
+            unpromised = max(0, available - active * observed_cost)
+            slots = min(slots, unpromised // observed_cost)
+        return max(1, slots) if active == 0 and configured else slots
+
+    def memory_headroom(self) -> tuple[int, int] | None:
+        """Return workload RAM+swap usage and effective available headroom."""
+        try:
+            meminfo = {
+                name: int(value.split()[0]) * 1024
+                for name, value in (line.split(":", 1) for line in Path("/proc/meminfo").read_text().splitlines() if ":" in line)
+            }
+            host_swap_free = host_swap_free_bytes(meminfo.get("SwapFree", 0))
+            host_available = meminfo["MemAvailable"] + host_swap_free
+        except (OSError, ValueError, KeyError):
+            host_swap_free = 0 if sys.platform.startswith("linux") else psutil.swap_memory().free
+            host_available = psutil.virtual_memory().available + host_swap_free
+
+        available = host_available
+        used_bytes = None
+        for path in self.cgroup_paths():
+            try:
+                memory_max = (path / "memory.max").read_text().strip()
+                if memory_max == "max":
+                    continue
+                current = int((path / "memory.current").read_text())
+                swap_max = (path / "memory.swap.max").read_text().strip()
+                swap_current = int((path / "memory.swap.current").read_text())
+                swap_room = host_swap_free if swap_max == "max" else min(host_swap_free, max(0, int(swap_max) - swap_current))
+                available = min(available, max(0, int(memory_max) - current) + swap_room)
+                if used_bytes is None:
+                    used_bytes = current + swap_current
+            except (OSError, ValueError, KeyError):
+                continue
+        if used_bytes is None:
+            # Bare-metal and non-Linux installs have no finite cgroup. Count
+            # the real runner process tree rather than the entire host.
+            try:
+                process = psutil.Process()
+                processes = [process, *process.children(recursive=True)]
+            except (psutil.Error, OSError):
+                return None
+            used_bytes = 0
+            for child in processes:
+                try:
+                    used_bytes += child.memory_info().rss
+                except (psutil.Error, OSError):
+                    continue
+        return used_bytes, available
+
    def observe_memory_stalls(self, source: str, total: int, avg10: float, sampled_at: float) -> bool:
        previous = self._memory_stalls.get(source)
        self._memory_stalls[source] = (total, sampled_at)
@@ -3,6 +3,7 @@ from __future__ import annotations
 import asyncio
 import contextvars
 import json
+import logging
 import os
 import signal
 import shutil
@@ -64,6 +65,7 @@ from archivebox.services.resource_admission import RESOURCE_RECHECK_SECONDS, res
 from archivebox.config.common import (
    ArchiveBoxBaseConfig,
    normalize_runtime_config,
+    _is_cookie_file_config_key,
    _plugin_enabled_config_keys,
 )
 from archivebox.misc.db import run_db_analyze_batch
@@ -223,6 +225,7 @@ class CrawlRunner:
        self.snapshot_service = SnapshotService(
            self.bus,
            crawl_id=str(crawl.id),
+            interrupted=self.execution_interrupted,
        )
        HookArchiveResultService(self.bus, emit_jsonl=False)
        ArchiveResultService(self.bus)
@@ -232,6 +235,10 @@ class CrawlRunner:
        self.snapshot_tasks: dict[str, asyncio.Task[None]] = {}
        self.snapshot_semaphore = asyncio.Semaphore(1)
        self.max_concurrent_snapshots = 1
+        self._warmup_snapshot_id: str | None = None
+        self._memory_baseline: int | None = None
+        self._memory_peak_cost = 0
+        self._observed_snapshot_cost: int | None = None
        self._resource_deferred = False
        self._initial_snapshot_admitted = False
        self.persona = None
@@ -253,6 +260,10 @@ class CrawlRunner:
        self._signal_abort_requested = False
        self._last_lease_heartbeat_at = 0.0

+    def execution_interrupted(self) -> bool:
+        """Owner loss preserves work; explicit user abort keeps its sealing policy."""
+        return self._signal_abort_requested and not self._user_aborted
+
    async def on_CrawlAbortEvent(self, event: CrawlAbortEvent) -> None:
        self._signal_abort_requested = True
        # An abort can arrive while the prompt transport is being established,
@@ -498,10 +509,15 @@ class CrawlRunner:
        if task is not None and not task.done():
            return
        current_event = crawl_start_event or get_current_event()
+        if not isinstance(current_event, CrawlStartEvent) and in_handler_context():
+            return
+        if self._observed_snapshot_cost is None and self._warmup_snapshot_id is None:
+            observation = resource_admission.memory_headroom()
+            self._memory_baseline = observation[0] if observation is not None else None
+            self._memory_peak_cost = 0
+            self._warmup_snapshot_id = snapshot_id
        if isinstance(current_event, CrawlStartEvent):
            task = asyncio.create_task(self.run_snapshot(snapshot_id, current_event), context=_runner_task_context())
-        elif in_handler_context():
-            return
        else:
            task = asyncio.create_task(self.run_snapshot(snapshot_id), context=_runner_task_context())
        self._initial_snapshot_admitted = False
@@ -530,6 +546,7 @@ class CrawlRunner:
                if task.done():
                    if self.snapshot_tasks.get(snapshot_id) is task:
                        self.snapshot_tasks.pop(snapshot_id, None)
+                    self._finish_warmup_snapshot(snapshot_id)
                    try:
                        task.result()
                    except asyncio.CancelledError as err:
@@ -554,13 +571,16 @@ class CrawlRunner:
                    return
                continue
            await self.heartbeat_active_leases()
-            done, _pending = await asyncio.wait(pending_tasks, timeout=10.0, return_when=asyncio.FIRST_COMPLETED)
+            self._observe_snapshot_memory()
+            done, _pending = await asyncio.wait(pending_tasks, timeout=RESOURCE_RECHECK_SECONDS, return_when=asyncio.FIRST_COMPLETED)
+            self._observe_snapshot_memory()
            if not done:
                continue
            for task in done:
                for snapshot_id, tracked_task in list(self.snapshot_tasks.items()):
                    if tracked_task is task:
                        self.snapshot_tasks.pop(snapshot_id, None)
+                        self._finish_warmup_snapshot(snapshot_id)
                        break
                try:
                    task.result()
@@ -578,6 +598,41 @@ class CrawlRunner:
            if not stop_scheduling and enqueue_projected:
                await self.enqueue_pending_snapshots_from_projection()

+    def _observe_snapshot_memory(self) -> None:
+        if self._memory_baseline is None:
+            return
+        observation = resource_admission.memory_headroom()
+        if observation is not None:
+            used, _available = observation
+            # Only attribute this workload's growth to a capture. Other jobs on
+            # the host reduce available headroom, but are not a per-capture cost
+            # to multiply again for every snapshot we want to admit.
+            self._memory_peak_cost = max(
+                self._memory_peak_cost,
+                used - self._memory_baseline,
+            )
+            if self._observed_snapshot_cost is not None:
+                self._observed_snapshot_cost = max(self._observed_snapshot_cost, self._memory_peak_cost)
+
+    def _finish_warmup_snapshot(self, snapshot_id: str) -> None:
+        if snapshot_id != self._warmup_snapshot_id or self._observed_snapshot_cost is not None:
+            return
+        self._observe_snapshot_memory()
+        if self._memory_peak_cost > 0:
+            self._observed_snapshot_cost = self._memory_peak_cost
+            headroom = resource_admission.memory_headroom()
+            logging.getLogger(__name__).info(
+                "Snapshot admission: concurrency limit=%s, observed memory growth=%s bytes, available=%s bytes",
+                self.max_concurrent_snapshots,
+                self._observed_snapshot_cost,
+                headroom[1] if headroom else None,
+            )
+        else:
+            # No observed cost cannot justify parallel admission. Probe the
+            # next real snapshot while the queue keeps moving one at a time.
+            self._memory_baseline = None
+        self._warmup_snapshot_id = None
+
    async def heartbeat_active_leases(self) -> None:
        # These are resumable work-item leases, not orchestrator-election
        # heartbeats. Each update is a short autocommit statement; network and
@@ -645,7 +700,11 @@ class CrawlRunner:
        self.max_concurrent_snapshots = resource_admission.snapshot_limit(int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))

        active_snapshot_ids = [snapshot_id for snapshot_id, task in self.snapshot_tasks.items() if not task.done()]
-        available_slots = max(0, self.max_concurrent_snapshots - len(active_snapshot_ids))
+        available_slots = resource_admission.additional_snapshot_slots(
+            self.max_concurrent_snapshots,
+            len(active_snapshot_ids),
+            self._observed_snapshot_cost,
+        )
        if available_slots <= 0:
            return
        pending_snapshot_ids = await sync_to_async(
@@ -669,6 +728,9 @@ class CrawlRunner:
        from archivebox.machine.models import Machine, NetworkInterface, Process

        self.primary_url = self.crawl.get_urls_list()[0] if self.crawl.get_urls_list() else ""
+        # Validate saved binary paths at the execution boundary, including
+        # config edited since this process first registered its Machine.
+        machine = Machine._sanitize_config(Machine.current())
        current_iface = NetworkInterface.current(refresh=not self.allow_maintenance_on_inactive_crawl)
        current_process = Process.current()
        if current_process.iface_id != current_iface.id or current_process.machine_id != current_iface.machine_id:
@@ -677,7 +739,14 @@ class CrawlRunner:
            current_process.save(update_fields=["iface", "machine", "modified_at"])
        self.persona = self.crawl.resolve_persona()
        self.base_config = get_config(crawl=self.crawl, overrides=self.config_overrides)
-        self.derived_config = dict(Machine.current().config or {})
+        # The merged user payload owns auth paths and binary selections. The
+        # install phase publishes resolved binaries; stale machine paths must
+        # not override explicit runtime settings before that phase runs.
+        self.derived_config = {
+            key: value
+            for key, value in (machine.config or {}).items()
+            if not _is_cookie_file_config_key(key) and (not key.endswith("_BINARY") or value == self.base_config.get(key))
+        }
        self.crawl_output_dir = str(self.crawl.output_dir)
        if self.persona:
            self.base_config.update(
@@ -997,7 +1066,8 @@ class CrawlRunner:
            if self.initial_snapshot_ids is not None:
                remaining_snapshot_ids = iter(snapshot_ids)
                while True:
-                    batch = list(zip(range(self.max_concurrent_snapshots), remaining_snapshot_ids, strict=False))
+                    slots = resource_admission.additional_snapshot_slots(self.max_concurrent_snapshots, 0, self._observed_snapshot_cost)
+                    batch = list(zip(range(slots), remaining_snapshot_ids, strict=False))
                    if not batch:
                        return
                    for _slot, snapshot_id in batch:
@@ -1008,7 +1078,8 @@ class CrawlRunner:
                        await self.enqueue_snapshot(snapshot_id)
                    await self.wait_for_snapshot_tasks(enqueue_projected=False)
            else:
-                for snapshot_id in snapshot_ids[: self.max_concurrent_snapshots]:
+                slots = resource_admission.additional_snapshot_slots(self.max_concurrent_snapshots, 0, self._observed_snapshot_cost)
+                for snapshot_id in snapshot_ids[:slots]:
                    if await self.crawl_is_cancelled():
                        break
                    if await self.crawl_is_paused():
@@ -1073,18 +1144,17 @@ class CrawlRunner:
            finally:
                cancel_watcher.cancel()
                await asyncio.gather(cancel_watcher, return_exceptions=True)
-            completed_event = event.emit(
-                CrawlCompletedEvent(
-                    url=snapshot["url"],
-                    snapshot_id=snapshot["id"],
-                    output_dir=str(output_dir),
-                ),
-            )
-            # Same signal lifecycle as CrawlCleanupEvent above: completion is a
-            # normal bus event unless the interpreter is already unwinding from
-            # SIGINT/SIGTERM/SIGHUP, where synchronous bus delivery is no
-            # longer a dependable shutdown primitive.
-            if not self._signal_abort_requested:
+            # emit() schedules handlers immediately; guarding only the await
+            # still lets completion projectors run during takeover. An aborted
+            # execution owner must leave durable work for its replacement.
+            if not self.execution_interrupted():
+                completed_event = event.emit(
+                    CrawlCompletedEvent(
+                        url=snapshot["url"],
+                        snapshot_id=snapshot["id"],
+                        output_dir=str(output_dir),
+                    ),
+                )
                await _run_event_now(completed_event, CrawlCompletedEvent.model_fields["event_timeout"].default)

        on_archivebox_CrawlStartEvent.__name__ = "on_archivebox_CrawlStartEvent__run_snapshots"
@@ -1183,6 +1253,7 @@ class CrawlRunner:
            user_config_event = MachineEvent(config=config, config_type="user")
            user_config_event.event_parent_id = crawl_start_event.event_id
            await self.bus.emit(user_config_event).now()
+            derived_config_event = None
            if derived_config:
                derived_config_event = MachineEvent(config=derived_config, config_type="derived")
                derived_config_event.event_parent_id = crawl_start_event.event_id
@@ -1198,6 +1269,7 @@ class CrawlRunner:
                snapshot_cleanup_phase_timeout=snapshot_phase_timeout,
                abort_requested=self.crawl_is_cancelled,
            )
+            projected = False
            try:
                snapshot_event = SnapshotEvent(
                    url=snapshot["url"],
@@ -1221,6 +1293,9 @@ class CrawlRunner:
                    raise RuntimeError(f"Snapshot {snapshot_id} did not complete")
                await completed_snapshot.wait(timeout=snapshot_phase_timeout)
                await completed_snapshot.event_results_list()
+                projected = True
+                if self.execution_interrupted():
+                    return
                if snapshot["status"] == "sealed":
                    await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
                    return
@@ -1242,6 +1317,16 @@ class CrawlRunner:
                await sync_to_async(_seal_when_last_snapshot_finished, thread_sensitive=True)()
            finally:
                snapshot_service.close()
+                # Projection and snapshot cleanup have finished. Historical
+                # capture results now live in the database; only active work
+                # needs its event ancestry available to the downloader.
+                expired = [snapshot_event, user_config_event] if projected else []
+                if projected and derived_config_event is not None:
+                    expired.append(derived_config_event)
+                if projected:
+                    expired.extend(await self.bus.filter("*", child_of=snapshot_event, past=True, future=False))
+                for event in expired:
+                    event.event_ttl = 0

    def seal_snapshot_due_to_limit(self, snapshot_id: str) -> None:
        from archivebox.core.models import Snapshot
@@ -1471,7 +1556,9 @@ def _run_due_crawl_locked(crawl, *, lock_seconds: int, interactive_interrupts: b
            # avoids hot-looping on the parent while child work is ready without
            # resurrecting a user cancellation that sealed the crawl after
            # selection.
-            crawl.safe_update(
+            if crawl.status != crawl.StatusChoices.STARTED:
+                return False
+            return crawl.safe_update(
                {
                    "status": crawl.StatusChoices.STARTED,
                    "retry_at": now + timedelta(seconds=ACTIVE_STATE_LEASE_SECONDS),
@@ -1480,7 +1567,6 @@ def _run_due_crawl_locked(crawl, *, lock_seconds: int, interactive_interrupts: b
                refresh=False,
                extra_filter={"status": crawl.StatusChoices.STARTED},
            )
-            return True
        if snapshot_count and not due_active_snapshots:
            if crawl.is_finished():
                if not crawl.claim_processing_lock(lock_seconds=lock_seconds):
@@ -1608,6 +1694,24 @@ def _run_due_snapshot_locked(snapshot, *, lock_seconds: int, interactive_interru
        if any(process.is_running for process in snapshot.process_set.filter(status="running").iterator()):
            snapshot.update_and_requeue(retry_at=timezone.now() + timedelta(seconds=lock_seconds))
            return True
+        if snapshot.fs_migration_needed:
+            # Crawl isolation bypasses the per-Snapshot path below, so migrate
+            # this admitted Snapshot before the CrawlRunner derives its output
+            # directory. Claim the Snapshot as well as holding the lifecycle
+            # lock: maintenance-only workers use the Snapshot lease directly.
+            if not snapshot.claim_processing_lock(lock_seconds=lock_seconds):
+                return False
+            snapshot.refresh_from_db()
+            claimed_status = snapshot.status
+            claimed_retry_at = snapshot.retry_at
+            if snapshot.fs_migration_needed:
+                snapshot.migrate_filesystem_to_current_version()
+            if not snapshot.safe_update(
+                {"retry_at": timezone.now()},
+                refresh=True,
+                extra_filter={"status": claimed_status, "retry_at": claimed_retry_at},
+            ):
+                return False
        if not snapshot.crawl.claim_processing_lock(lock_seconds=lock_seconds):
            return False
        _runner_console_line(crawl=snapshot.crawl)
@@ -1820,12 +1924,11 @@ def _run_due_crawl_status(
    due_crawl = Crawl.objects.filter(id=due_crawl_id).first()
    if due_crawl is None:
        return True
-    run_due_crawl(
+    return run_due_crawl(
        due_crawl,
        lock_seconds=lock_seconds,
        interactive_interrupts=interactive_interrupts,
    )
-    return True


 def _run_due_snapshot_query(
@@ -1879,8 +1982,7 @@ def _run_due_binary() -> bool:
    due_binary = Binary.objects.filter(id=due_binary_id).first()
    if due_binary is None:
        return True
-    run_due_binary(due_binary, lock_seconds=60)
-    return True
+    return run_due_binary(due_binary, lock_seconds=60)


 def run_pending_crawls(
@@ -1900,14 +2002,35 @@ def run_pending_crawls(
    last_recovery_at = 0.0
    last_retention_at = 0.0
    last_retention_repair_at = 0.0
+    deletion_thread: threading.Thread | None = None
    last_analyze_at = 0.0
    analyze_queue: list[str] | None = None
    analyze_sweep_started_at = 0.0
    orchestrator_started_at = time.monotonic()
+
+    def delete_in_background() -> None:
+        from django.db import connections
+
+        try:
+            Snapshot.delete_requested(batch_size=100)
+        finally:
+            connections.close_all()
+
    while True:
        raise_if_shutdown_requested()
        now_monotonic = time.monotonic()
        if crawl_id is None and now_monotonic - last_retention_at >= (60.0 if daemon else 1.0):
+            if daemon:
+                # At most one batch can be doing filesystem I/O. A stuck mount
+                # must not block capture scheduling or runner shutdown; rows
+                # remain pending until cleanup succeeds, including on restart.
+                if (deletion_thread is None or not deletion_thread.is_alive()) and Snapshot.objects.filter(
+                    status=Snapshot.DELETING_STATE,
+                ).exists():
+                    deletion_thread = threading.Thread(target=delete_in_background, name="snapshot-delete", daemon=True)
+                    deletion_thread.start()
+            else:
+                Snapshot.delete_requested(batch_size=100)
            for model in (ArchiveResult, Snapshot, Crawl, Process):
                # Keep the tight scheduler loop anchored on indexed delete_at
                # columns only. Backfilling missing delete_at values has to read
@@ -1,6 +1,7 @@
 from __future__ import annotations

 import asyncio
+from collections.abc import Callable
 from pathlib import Path

 from asgiref.sync import sync_to_async
@@ -73,10 +74,11 @@ def finalize_completed_snapshot(
            modified_at=timezone.now(),
        )

-    # SnapshotCompletedEvent is abx-dl's authoritative signal that the complete
-    # snapshot hook sequence (including cleanup) finished. ArchiveResult rows
-    # are projections of that work, never prerequisites used to decide whether
-    # the Snapshot may seal.
+    # SnapshotCompletedEvent finishes an attempt, including abort cleanup. An
+    # explicit user abort deliberately seals that point-in-time capture; the
+    # runner's interrupted-owner guard below keeps takeover work resumable.
+    # ArchiveResult rows are projections of that work, never prerequisites used
+    # to decide whether the Snapshot may seal.
    if not was_sealed and snapshot.status == Snapshot.StatusChoices.STARTED and snapshot.retry_at == owned_retry_at:
        snapshot.seal()
        snapshot.refresh_from_db()
@@ -120,8 +122,9 @@ class SnapshotService(BaseService):
    LISTENS_TO = [SnapshotEvent, SnapshotCompletedEvent]
    EMITS = []

-    def __init__(self, bus, *, crawl_id: str):
+    def __init__(self, bus, *, crawl_id: str, interrupted: Callable[[], bool] | None = None):
        self.crawl_id = crawl_id
+        self.interrupted = interrupted
        self._run_ownership: dict[str, tuple[str, object, bool, list[str]]] = {}
        self._ownership_lock = asyncio.Lock()
        super().__init__(bus)
@@ -175,6 +178,15 @@ class SnapshotService(BaseService):
            if ownership is None or ownership[0] != str(event.event_parent_id):
                return
            self._run_ownership.pop(snapshot_id, None)
+        # abx-dl also emits completion after abort cleanup, even before the
+        # first hook. Explicit user aborts seal a point-in-time capture so later
+        # work cannot silently mix observations taken days apart; reopening it
+        # is a separate user action. Takeover only replaces its execution owner.
+        # Consult the runner's synchronous signal state: an exiting owner cannot
+        # rely on another bus event arriving before this completion projector runs.
+        # Leave its durable lease for the next elected runner's startup recovery.
+        if self.interrupted is not None and self.interrupted():
+            return
        _, owned_retry_at, was_sealed, retry_plugins = ownership
        await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(
            event.snapshot_id,
@@ -1527,6 +1527,9 @@
                display: inline-flex;
                flex-wrap: wrap;
                align-items: center;
+                min-width: 0;
+                max-width: 100%;
+                box-sizing: border-box;
                gap: 4px;
                padding: 2px 4px;
                background: transparent;
@@ -1540,15 +1543,27 @@
                flex-wrap: wrap;
                gap: 4px;
                align-items: center;
+                min-width: 0;
+                max-width: 100%;
            }

            .tag-editor-inline .tag-pill {
+                min-width: 0;
+                max-width: min(160px, 100%);
+                box-sizing: border-box;
                padding: 2px 6px 2px 8px;
                font-size: 11px;
                border-radius: 12px;
            }

+            .tag-editor-inline .tag-link {
+                min-width: 0;
+                overflow: hidden;
+                text-overflow: ellipsis;
+            }
+
            .tag-editor-inline .tag-remove-btn {
+                flex-shrink: 0;
                width: 14px;
                height: 14px;
                font-size: 12px;
@@ -79,6 +79,13 @@
  }
  .cards .card .card-tags span {
      display: inline-block;
+      max-width: min(160px, 100%);
+      box-sizing: border-box;
+      overflow: hidden;
+      text-overflow: ellipsis;
+      white-space: nowrap;
+      vertical-align: middle;
+      pointer-events: auto;
      padding: 2px 5px;
      border-radius: 5px;
      opacity: 0.95;
@@ -270,11 +277,12 @@
            <a href="{% snapshot_detail_url obj %}">
              <h4>
                {% if obj.is_archived %}
-                    <img src="{% snapshot_base_url obj %}/favicon.ico" onerror="this.style.display='none'" class="link-favicon" decoding="async"/>
+                    {% snapshot_role_output_url obj 'list_icon' True as favicon_url %}
+                    {% if favicon_url %}<img src="{{ favicon_url }}" onerror="this.style.display='none'" class="link-favicon" decoding="async"/>{% endif %}
                {% else %}
                    <img src="{% static 'spinner.gif' %}" onerror="this.style.display='none'" class="link-favicon" decoding="async"/>
                {% endif %}
-                <span class="title-text">{{obj.title|default:'Pending...' }}</span>
+                <span class="title-text">{{ obj.title|default:obj.domain }}</span>
              </h4>
            </a>
          </div>
@@ -282,16 +290,16 @@
          <div class="card-media">
            <a href="{% snapshot_detail_url obj %}" class="card-thumbnail">
              {% snapshot_thumbnail obj admin=True width="100%" height="245px" %}
-            </a>
              {% if obj.tags_str %}
                <div class="card-tags">
                  {% for tag in obj.tags_str|split:',' %}
-                    <span>
+                    <span title="{{tag}}">
                      {{tag}}
                    </span>
                  {% endfor %}
                </div>
              {% endif %}
+            </a>
          </div>
          <div class="card-outputs">
            {% if obj.status == 'started' %}
@@ -11,6 +11,7 @@
 {% endblock %}

 {% block extra_head %}
+    <link rel="stylesheet" href="{% static 'admin/css/responsive.css' %}" />
    <link rel="stylesheet" href="{% static 'add.css' %}" />
 {% endblock %}

@@ -198,21 +198,29 @@
    }

    .snapshot-preview-cell {
+        position: relative;
        width: 110px;
        min-width: 116px;
        vertical-align: middle;
    }

    .snapshot-preview-cell a {
+        position: absolute;
+        top: 10px;
+        bottom: 10px;
        display: inline-flex;
        align-items: center;
        justify-content: center;
        width: 102px;
-        height: 102px;
        line-height: 0;
        vertical-align: middle;
    }

+    .snapshot-preview-cell .snapshot-thumbnail {
+        width: 100% !important;
+        height: 100% !important;
+    }
+
    #content .public-snapshot-list .snapshot-preview-cell .snapshot-preview,
    #content .public-snapshot-list .snapshot-preview-cell img.snapshot-preview.screenshot {
        box-sizing: border-box;
@@ -330,9 +338,10 @@
    }

    .snapshot-tag {
-        display: inline-flex;
-        align-items: center;
-        max-width: 160px;
+        display: block;
+        min-width: 0;
+        max-width: min(160px, 100%);
+        box-sizing: border-box;
        padding: 2px 7px;
        border: 1px solid #bfdbfe;
        border-radius: 999px;
@@ -585,7 +594,8 @@

        .snapshot-preview-cell a {
            width: 82px;
-            height: 82px;
+            top: 7px;
+            bottom: 7px;
        }

        #content .public-snapshot-list .snapshot-preview-cell .snapshot-preview,
@@ -644,12 +654,19 @@
                </span>
                <input type="submit" value="Search">
            </form>
+            {% comment %}
+                Missing/stale totals are intentional on large collections: show rows
+                immediately and let a later normal reload pick up the cached count.
+                Do not fill this in with an extra count request or paginator.count:
+                the unfiltered paginator only knows the rows needed for navigation.
+                Use the optional public_* totals here and in the footer below.
+            {% endcomment %}
            <div class="public-snapshot-count">
                {% if public_search_stream_pending %}
                    <span class="archivebox-search-stream-status" role="status" aria-live="polite">Searching...</span>
                {% else %}
-                    {{ page_obj.start_index }}-{{ page_obj.end_index }} of {{ page_obj.paginator.count }}
-                    <span>(page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }})</span>
+                    {{ page_obj.start_index }}-{{ page_obj.end_index }}{% if public_snapshot_count != None %} of {{ public_snapshot_count }}{% endif %}
+                    <span>(page {{ page_obj.number }}{% if public_page_count != None %} of {{ public_page_count }}{% endif %})</span>
                {% endif %}
            </div>
        </div>
@@ -659,7 +676,7 @@
                    <tr>
                        <th class="snapshot-time">Saved</th>
                        <th class="snapshot-preview-cell">Preview</th>
-                        <th>Snapshot ({{ page_obj.paginator.count }})</th>
+                        <th>Snapshot{% if public_snapshot_count != None %} ({{ public_snapshot_count }}){% endif %}</th>
                        <th class="snapshot-tags-cell">Tags</th>
                        <th class="snapshot-status-cell">Status</th>
                        <th class="snapshot-files-cell">Files</th>
@@ -687,7 +704,7 @@
            </table>
        </div>
        <div class="public-pagination">
-            Showing {{ page_obj.start_index }}-{{ page_obj.end_index }} of {{ page_obj.paginator.count }} total
+            Showing {{ page_obj.start_index }}-{{ page_obj.end_index }}{% if public_snapshot_count != None %} of {{ public_snapshot_count }} total{% endif %}
            <br>
            <span class="step-links">
                {% if page_obj.has_previous %}
@@ -696,14 +713,14 @@
                {% endif %}

                <span class="current">
-                    Page {{ page_obj.number }} of {{ page_obj.paginator.num_pages }}
+                    Page {{ page_obj.number }}{% if public_page_count != None %} of {{ public_page_count }}{% endif %}
                </span>

                {% if page_obj.has_next %}
                    <a href="?{% url_replace page=page_obj.next_page_number %}">next</a>
                {% endif %}
-                {% if page_obj.has_next %}
-                    <a href="?{% url_replace page=page_obj.paginator.num_pages %}">last &raquo;</a>
+                {% if page_obj.has_next and public_page_count != None %}
+                    <a href="?{% url_replace page=public_page_count %}">last &raquo;</a>
                {% endif %}
            </span>
        </div>
@@ -193,29 +193,9 @@
                background: #c65580;
            }
            .year-capture-count { opacity: .85; font-size: 11px; }
-            .current-year .year-variants { z-index: 2; }
-            .current-year .year-variants[open] { z-index: 60; }
-            .current-year .year-variants > summary::after {
-                content: "";
-                position: absolute;
-                right: -5px;
-                top: calc(50% - 5px);
-                width: 10px;
-                height: 10px;
-                transform: rotate(45deg);
-                background: #c65580;
-            }
-            .selected-capture > summary {
-                display: flex;
-                align-items: center;
-                min-height: 26px;
-                padding: 3px 8px 3px 12px;
-                margin-left: -2px;
-                border-radius: 0 4px 4px 0;
-                background: #861840;
-                color: #fff;
-                font-size: 12px;
-            }
+            .current-year .year-label { position: relative; z-index: 1; background: #c65580; border-radius: 5px; }
+            .current-year .year-label::after { content: ""; position: absolute; right: -5px; top: calc(50% - 5px); width: 10px; height: 10px; transform: rotate(45deg); background: #c65580; }
+            .header-date { background: #861840; color: #fff; border-radius: 0 5px 5px 0; }
            .snapshot-variants-list {
                position: absolute;
                right: 0;
@@ -233,16 +213,6 @@
            }
            .snapshot-variants-list a { display: block; padding: 6px 8px; font-size: 12px; border-radius: 4px; color: #fff; }
            .snapshot-variants-list a:hover { background: #ffffff18; }
-            @container snapshot-header (max-width: 1100px) {
-                .header-nav { grid-template-columns: 32px minmax(0, 1fr); gap: 4px 6px; }
-                .header-archivebox { width: 32px; height: 32px; font-size: 0; gap: 0; }
-                .header-capture-row { grid-column: 2; grid-row: 2; justify-content: flex-end; }
-                .snapshot-variants-list { right: 0; }
-            }
-            @container snapshot-header (max-width: 480px) {
-                .header-top { padding: 5px 30px 5px 8px; }
-                .header-capture-row { grid-column: 1 / -1; gap: 4px; }
-            }

            .header-top .header-url,
            .header-top .header-badges > .badge,
@@ -265,6 +235,8 @@
                white-space: nowrap;
            }
            .header-top .header-url { display: flex; gap: 0; padding: 0; }
+            .header-page-link, .header-page-metadata { display: contents; }
+            .header-url-location { display: flex; align-items: center; flex: 1; min-width: 0; }
            .header-url-content { display: flex; flex: 1; align-items: center; gap: 6px; min-width: 0; padding: 0 7px; overflow: hidden; }
            .header-top .header-url-text { flex: 0 1 auto; min-width: 40px; padding: 0; color: #fff; font-size: 13px; line-height: 26px; }
            .header-title-text { flex: 0 1 auto; min-width: 0; color: #efc9d8; font-size: 12px; }
@@ -308,21 +280,124 @@
            .header-url-action.copied .copy-symbol { display: none; }
            .header-url-action.copied .copy-success { display: block; }
            .header-top .tag-pill { background: var(--tag-bg, #ffffff20); color: var(--tag-fg, #fff); }
-            .header-top .selected-capture > summary { padding-left: 12px; border-left: 0; border-radius: 0 5px 5px 0; }
+            .header-top .current-year summary { gap: 0; padding: 0; border: 0; background: none; }
+            .current-year .year-label, .current-year .header-date { display: inline-flex; align-items: center; gap: 4px; height: 28px; padding: 3px 7px; border: 1px solid #ffffff30; }
+            .current-year .header-date { padding-left: 12px; border-left: 0; }
+            .mobile-size { display: none; }
            .header-top .badge small { font-size: inherit; font-weight: inherit; }
-            @container snapshot-header (max-width: 480px) {
-                .header-top .header-url { flex-wrap: wrap; height: auto; }
-                .header-url-content { flex-basis: calc(100% - 25px); }
-                .header-url-tools { flex-wrap: wrap; justify-content: flex-end; max-width: 100%; }
-                .header-url-actions { width: 100%; flex-wrap: wrap; justify-content: flex-end; min-height: 27px; border-top: 1px solid #ffffff20; }
-                .header-top .header-url-action { width: 24px; }
-                .header-top .header-url-download { width: auto; }
-                .header-top .header-url .permission-pill { padding: 0 4px; font-size: 10px; }
+            @container snapshot-header (max-width: 1100px) {
+                .header-nav { grid-template-columns: 32px minmax(0, 1fr) auto; gap: 4px 6px; }
+                .header-archivebox { width: 32px; height: 32px; font-size: 0; gap: 0; }
+                .header-top .header-url { display: contents; }
+                .header-url-location { grid-column: 2 / -1; grid-row: 1; height: 28px; border: 1px solid #ffffff30; border-radius: 5px; background: #00000012; }
+                .header-url-actions { grid-column: 1 / 3; grid-row: 2; align-items: center; min-width: 0; }
+                .header-url-tools { border: 1px solid #ffffff30; border-radius: 5px; overflow: hidden; }
+                .header-url-action:first-child { border-left: 0; }
+                .header-top .header-url .permission-pill { margin-left: 0; margin-right: 6px; height: 28px; }
+                .header-capture-row { grid-column: 3; grid-row: 2; justify-content: flex-end; }
+                .snapshot-variants-list { right: 0; }
+            }
+            @container snapshot-header (max-width: 600px) {
+                .header-top { padding: 5px 30px 5px 8px; }
+                .header-nav { grid-template-columns: 22px minmax(0, 1fr) auto; column-gap: 5px; }
+                .header-left { grid-column: 1 / 3; width: 32px; }
+                .header-url-location, .header-url-content, .header-page-link { display: contents; }
+                .header-top .header-url-text { grid-column: 1 / -1; grid-row: 1; margin-left: 38px; min-width: 0; padding: 0 7px; height: 28px; border: 1px solid #ffffff30; border-radius: 5px; background: #00000012; }
+                .header-url-favicon { grid-column: 1; grid-row: 2; margin-left: 0; }
+                .header-page-metadata { display: flex; align-items: center; gap: 6px; grid-column: 2 / -1; grid-row: 2; min-width: 0; }
+                .header-title-text { flex: 1 1 auto; max-width: none; font-size: 12px; }
+                .header-text-dot { display: none; }
+                .header-tags { flex: 0 1 auto; max-width: 60%; overflow-x: auto; scrollbar-width: none; }
+                .header-tags::-webkit-scrollbar { display: none; }
+                .header-url-actions { grid-column: 1 / 3; grid-row: 3; justify-content: flex-start; }
+                .header-capture-row { grid-column: 3; grid-row: 3; justify-content: flex-end; max-width: 45vw; }
+                .header-top .header-url-action { width: 26px; }
+                .header-top #copy-original-url, .header-top [aria-label="Open original URL"], .permission-text, .desktop-size { display: none; }
+                .mobile-size { display: inline; }
+                .header-top .header-url-download { width: auto; gap: 2px; padding: 0 4px; }
+                .header-top .header-url .permission-pill { padding: 0 4px; margin-right: 4px; font-size: 10px; }
+                .header-top .header-url-tools [aria-label="Search Archive.org"] { border-left: 0; }
            }
            @container snapshot-header (max-width: 360px) {
-                .header-top .header-url-action { width: 20px; }
-                .header-top .header-url-download { width: auto; padding: 0 4px; font-size: 10px; }
+                .header-top .header-url-action { width: 22px; }
+                .header-top .header-url-download { width: auto; gap: 2px; padding: 0 3px; font-size: 10px; }
+                .header-top .header-status .status-count { margin: 0 1px; padding: 0 2px; }
+                .header-top .year-variants > summary { font-size: 10px; }
+                .current-year .year-label { padding: 3px 4px; }
+                .current-year .header-date { padding: 3px 4px 3px 9px; }
+                .year-capture-count { font-size: 9px; }
            }
+            /* Native ArchiveBox app presentation lives with the snapshot template,
+               so it stays aligned with the server's responsive header markup. */
+            html.archivebox-ios-app #header.archivebox-app-header {
+                box-sizing: border-box; min-height: 60px; padding-left: 64px !important;
+            }
+            @media (max-width: 520px) {
+                html.archivebox-ios-app #header.archivebox-app-header { padding-right: 64px !important; }
+            }
+            html.archivebox-ios-app header:has(> .header-top),
+            html.archivebox-ios-app .header-top { background: #a51c50 !important; }
+            html.archivebox-ios-app .header-top { padding-left: 8px !important; padding-right: 8px !important; }
+            html.archivebox-ios-app .header-top .header-left {
+                box-sizing: border-box; padding-left: 52px !important; min-height: 48px;
+                display: flex; align-items: center;
+            }
+            @container snapshot-header (max-width: 1100px) {
+                html.archivebox-ios-app .header-top .header-nav { grid-template-columns: 84px minmax(0, 1fr) auto !important; }
+                html.archivebox-ios-app .header-top .header-url-location { grid-column: 2 / -1; }
+                html.archivebox-ios-app .header-top .header-url-actions { grid-column: 1 / 3; }
+                html.archivebox-ios-app .header-top .header-capture-row { grid-column: 3; }
+                html.archivebox-ios-app .header-top .header-url-favicon { margin-left: 0; }
+            }
+            @container snapshot-header (max-width: 600px) {
+                /* Keep the native Back clearance on the URL row only. All other
+                   rows use the same narrow tracks as the regular mobile layout. */
+                html.archivebox-ios-app .header-top .header-nav { grid-template-columns: minmax(0, 1fr) auto !important; }
+                html.archivebox-ios-app .header-top .header-left { display: none !important; }
+                html.archivebox-ios-app .header-top .header-url-text {
+                    grid-column: 1 / -1; grid-row: 1; margin-left: 44px !important;
+                    max-width: calc(100% - 44px);
+                }
+                html.archivebox-ios-app .header-top .header-url-favicon { grid-column: 1; grid-row: 2; margin-left: 0; }
+                html.archivebox-ios-app .header-top .header-page-metadata {
+                    grid-column: 1 / -1; grid-row: 2; margin-left: 28px; max-width: calc(100% - 28px);
+                }
+                html.archivebox-ios-app .header-top .header-url-actions {
+                    grid-column: 1; grid-row: 3; justify-self: start; min-width: 0; justify-content: flex-start;
+                }
+                html.archivebox-ios-app .header-top .header-capture-row {
+                    grid-column: 2; grid-row: 3; justify-self: end; max-width: none; flex-wrap: nowrap;
+                }
+                html.archivebox-ios-app .header-top .header-year-badges { flex-wrap: nowrap; }
+            }
+            /* WKWebView can report a desktop-sized layout viewport while the
+               actual iPhone screen is narrow. Use the native phone marker for
+               the same row arrangement when container queries miss it. */
+            html.archivebox-ios-phone .header-top .header-nav { grid-template-columns: minmax(0, 1fr) auto !important; }
+            html.archivebox-ios-phone .header-top .header-url,
+            html.archivebox-ios-phone .header-top .header-url-location,
+            html.archivebox-ios-phone .header-top .header-url-content,
+            html.archivebox-ios-phone .header-top .header-page-link { display: contents !important; }
+            html.archivebox-ios-phone .header-top .header-left { display: none !important; }
+            html.archivebox-ios-phone .header-top .header-url-text {
+                grid-column: 1 / -1; grid-row: 1; margin-left: 44px !important;
+                max-width: calc(100% - 44px);
+            }
+            html.archivebox-ios-phone .header-top .header-url-favicon { grid-column: 1; grid-row: 2; margin-left: 0; }
+            html.archivebox-ios-phone .header-top .header-page-metadata {
+                display: flex; align-items: center; gap: 6px; grid-column: 1 / -1; grid-row: 2;
+                margin-left: 28px; min-width: 0; max-width: calc(100% - 28px);
+            }
+            html.archivebox-ios-phone .header-top .header-url-actions {
+                grid-column: 1; grid-row: 3; justify-self: start; min-width: 0; justify-content: flex-start;
+            }
+            html.archivebox-ios-phone .header-top .header-capture-row {
+                grid-column: 2; grid-row: 3; justify-self: end; max-width: none; flex-wrap: nowrap;
+            }
+            html.archivebox-ios-phone .header-top .header-year-badges { flex-wrap: nowrap; }
+            .header-title-line .header-title-text { min-width: 0; flex: 1 1 0; }
+            .header-title-line .favicon { flex-shrink: 0; }
+            .header-archivebox { white-space: nowrap; }
            .header-top .header-url .header-status { margin: 0; padding: 0; border: 0; }
            .header-status .status-count { margin: 0 3px; padding: 0 4px; border-radius: 3px; line-height: 18px; font-weight: 600; }
            .header-status .status-count-success { background: #acd4b6; color: #18552b; }
@@ -1030,7 +1105,7 @@
                    sandbox="allow-same-origin allow-top-navigation-by-user-activation allow-scripts allow-forms allow-downloads allow-popups allow-popups-to-escape-sandbox"
                    class="full-page-iframe"
                    src="about:blank"
-                    data-default-src="{% if best_result.direct_preview_path %}{% snapshot_url snapshot best_result.direct_preview_path %}{% elif best_result.path %}{% snapshot_preview_url snapshot best_result.path %}{% elif STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_base_url snapshot %}/?files=1{% endif %}"
+                    data-default-src="{% if best_result.direct_preview_path %}{% snapshot_url snapshot best_result.direct_preview_path %}{% elif best_result.path %}{% snapshot_preview_url snapshot best_result.path %}{% else %}{% snapshot_files_url snapshot %}{% endif %}"
                    name="preview"
                    loading="eager"
                    fetchpriority="high"></iframe>
@@ -1082,16 +1157,14 @@
                const frame = document.getElementById('main-frame')
                if (!frame) return
                const snapshotBaseUrlEarly = "{% snapshot_base_url snapshot %}"
-                const snapshotFilesUrlEarly = "{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_base_url snapshot %}/?files=1{% endif %}"
+                const snapshotFilesUrlEarly = "{% snapshot_files_url snapshot %}"
                const defaultSrc = frame.dataset.defaultSrc || snapshotFilesUrlEarly
+                // Hash fragments hold an encoded output path; decoding changes literal percent bytes in filenames.
                const rawHash = window.location.hash ? window.location.hash.slice(1) : ''

                function resolveEarlyPreviewUrl(raw) {
                    if (!raw) return defaultSrc
                    let value = raw
-                    try {
-                        value = decodeURIComponent(value)
-                    } catch (err) {}
                    if (value === 'all') return snapshotFilesUrlEarly
                    if (value.startsWith('http://') || value.startsWith('https://')) return value
                    if (value.startsWith('//')) return window.location.protocol + value
@@ -1122,25 +1195,31 @@
                    </div>
                    <div class="header-col header-main">
                        <div class="header-url">
+                            <div class="header-url-location">
                            <span class="header-url-favicon">
-                            {% snapshot_archiveresult_url snapshot 'favicon' 'favicon.ico' as header_favicon_url %}
+                            {% snapshot_role_output_url snapshot 'list_icon' as header_favicon_url %}
                            {% if header_favicon_url %}
-                                <img src="{{ header_favicon_url }}?raw=1" onerror="this.onerror=null;this.src=&quot;data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;&quot;" alt="Favicon" class="favicon"/>
+                                <img src="{{ header_favicon_url }}{% if not STATIC_EXPORT %}?raw=1{% endif %}" onerror="this.onerror=null;this.src=&quot;data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;&quot;" alt="Favicon" class="favicon"/>
                            {% else %}
                                <img src="data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;" alt="Favicon" class="favicon"/>
                            {% endif %}
                            </span>
                            <div class="header-url-content">
+                                <span class="header-page-link">
                                <a class="header-url-text" href="{{url}}" title="{{url}}" target="_blank" rel="noreferrer">{{url}}</a>
+                                <span class="header-page-metadata">
                                <span class="header-text-dot" aria-hidden="true">·</span>
-                                <span class="header-title-text" title="{{title}}">{{title|truncatechars:120}}</span>
+                                <span class="header-title-text" title="{{title}}">{{title}}</span>
                                {% if title_tags %}
-                                    <span class="header-text-dot" aria-hidden="true">·</span>
+                                    <span class="header-text-dot header-tags-dot" aria-hidden="true">·</span>
                                    <span class="header-tags">{% for tag in title_tags %}<span class="tag-pill" style="{{tag.style}}" title="{{tag.name}}">{{tag.name}}</span>{% endfor %}</span>
                                {% endif %}
+                                </span>
+                                </span>
+                            </div>
                            </div>
                            <div class="header-url-actions">
-                                <span class="badge permission-pill" title="Snapshot visibility">{{ snapshot_permissions_icon }} {{ snapshot_permissions }}</span>
+                                <span class="badge permission-pill" title="{{ snapshot_permissions|capfirst }}" aria-label="{{ snapshot_permissions|capfirst }}">{{ snapshot_permissions_icon }}<span class="permission-text"> {{ snapshot_permissions }}</span></span>
                                <div class="header-url-tools">
                                <button type="button" class="header-url-action" id="copy-original-url" data-url="{{url}}" aria-label="Copy original URL" title="Copy original URL">
                                    <svg class="copy-symbol" viewBox="0 0 20 20" aria-hidden="true"><rect x="7" y="7" width="10" height="11" rx="2"/><path d="M12 7V4a2 2 0 0 0-2-2H4a2 2 0 0 0-2 2v8a2 2 0 0 0 2 2h3"/></svg>
@@ -1158,7 +1237,7 @@
                            </a>
                            {% endif %}
                                {% if not STATIC_EXPORT %}
-                                <a class="header-url-action header-url-download" href="{% snapshot_base_url snapshot %}/?files=1&amp;download=zip" aria-label="Download snapshot ZIP" title="Download the entire snapshot directory as ZIP">
+                                <a class="header-url-action header-url-download" href="{% snapshot_zip_url snapshot %}" aria-label="Download snapshot ZIP" title="Download the entire snapshot directory as ZIP">
                                {% endif %}
                                    <span class="header-status" title="Snapshot output counts">
                                {% if num_outputs or num_failures %}
@@ -1169,7 +1248,7 @@
                                {% endif %}
                                    </span>
                                {% if not STATIC_EXPORT %}
-                                    <span class="header-download-size">{{size}}</span>
+                                    <span class="header-download-size"><span class="desktop-size">{{size}}</span><span class="mobile-size">{% with parts=size.split %}{% if parts.1 == "MB" or parts.1 == "KB" or parts.1 == "Bytes" %}{{parts.0|floatformat:"0"}} {% if parts.1 == "Bytes" %}B{% else %}{{parts.1}}{% endif %}{% else %}{{size}}{% endif %}{% endwith %}</span></span>
                                    <svg viewBox="0 0 20 20" aria-hidden="true"><path d="M10 2v11m-4-4 4 4 4-4M3 13v4a1 1 0 0 0 1 1h12a1 1 0 0 0 1-1v-4"/></svg>
                                </a>
                                {% endif %}
@@ -1182,17 +1261,16 @@
                            {% for entry in related_years %}
                            <div class="capture-year{% if entry.selected_date %} current-year{% endif %}" data-year="{{ entry.year }}">
                                <details class="snapshot-variants year-variants">
-                                    <summary class="badge badge-default">{{ entry.year }} <span class="year-capture-count">({{ entry.snapshots|length }})</span></summary>
-                                    {% include "core/snapshot_capture_list.html" with captures=entry.snapshots %}
-                                </details>
-                                {% if entry.selected_date %}
-                                <details class="snapshot-variants selected-capture">
-                                    <summary class="header-date" title="Click to see captures from {{ entry.year }}" aria-label="Selected capture {{ entry.selected_date|date:'Y-m-d' }}">
-                                        {{ entry.selected_date|date:"Y-m-d" }}
+                                    <summary class="badge badge-default"{% if entry.selected_date %} aria-label="All captures; selected {{ entry.selected_date|date:'Y-m-d' }}"{% endif %}>
+                                        <span class="year-label">{{ entry.year }} <span class="year-capture-count">{{ entry.snapshots|length }}</span></span>
+                                        {% if entry.selected_date %}<span class="header-date">{{ entry.selected_date|date:"Y-m-d" }}</span>{% endif %}
                                    </summary>
-                                    {% include "core/snapshot_capture_list.html" with captures=entry.snapshots %}
+                                    {% if entry.selected_date %}
+                                        {% include "core/snapshot_capture_list.html" with captures=url_snapshots %}
+                                    {% else %}
+                                        {% include "core/snapshot_capture_list.html" with captures=entry.snapshots %}
+                                    {% endif %}
                                </details>
-                                {% endif %}
                            </div>
                            {% endfor %}
                        </div>
@@ -1209,7 +1287,7 @@
                    <div class="thumb-card">
                            <div class="thumb-body">
                                <div class="thumb-actions">
-                                    <a href="{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_base_url snapshot %}/?files=1{% endif %}" data-no-preview="1" title="{% if STATIC_EXPORT %}Open snapshot manifest{% else %}Browse all snapshot files{% endif %}" target="_blank" rel="noopener">📁</a>
+                                    <a href="{% snapshot_files_url snapshot %}" data-no-preview="1" title="{% if STATIC_EXPORT %}Open snapshot manifest{% else %}Browse all snapshot files{% endif %}" target="_blank" rel="noopener">📁</a>
                                </div>
                                <h4><span class="card-title-icon">⚠️</span> Failed</h4>
                            </div>
@@ -1217,7 +1295,7 @@
                                <div class="loose-items failed-items">
                                    {% for item in failed_items %}
                                        {% if item.is_dir %}
-                                            <a href="{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_url snapshot item.path %}/?files=1{% endif %}" data-no-preview="1" target="_blank" rel="noopener">📁 {{item.name}}</a>
+                                            <a href="{% snapshot_files_url snapshot item.path %}" data-no-preview="1" target="_blank" rel="noopener">📁 {{item.name}}</a>
                                        {% else %}
                                            <a href="{% snapshot_url snapshot item.path %}" data-no-preview="1" target="_blank" rel="noopener">📄 {{item.name}}</a>
                                        {% endif %}
@@ -1235,7 +1313,7 @@

        <script>
            const snapshotBaseUrl = "{% snapshot_base_url snapshot %}";
-            const snapshotFilesUrl = "{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_base_url snapshot %}/?files=1{% endif %}";
+            const snapshotFilesUrl = "{% snapshot_files_url snapshot %}";

            function tryCenterImageFrame(frame) {
                try {
@@ -1432,6 +1510,31 @@
                }
            }

+            function isIsolatedLiteParsePreview(urlValue) {
+                if (!urlValue || !snapshotBaseUrl) return false
+                try {
+                    const target = new URL(urlValue, window.location.href)
+                    const base = new URL(snapshotBaseUrl, window.location.href)
+                    const basePath = (base.pathname || '').replace(/\/+$/, '')
+                    // LiteParse's full viewer is an ArchiveBox-authored template, not captured
+                    // page content. It embeds the saved PDF in its own iframe; Chromium/Brave's
+                    // native PDF viewer refuses to run through an ancestor sandbox, even when
+                    // that sandbox allows scripts and same-origin access. Remove the outer
+                    // sandbox only when this exact viewer is served from the isolated snapshot
+                    // origin. On one-domain installs that origin shares the control-plane origin,
+                    // so it must remain sandboxed; all ordinary replays remain sandboxed too.
+                    return (
+                        base.origin !== window.location.origin
+                        && target.origin === base.origin
+                        && target.pathname.startsWith(`${basePath}/liteparse/`)
+                        && target.searchParams.get('preview') === '1'
+                        && target.pathname.toLowerCase().endsWith('.txt')
+                    )
+                } catch (err) {
+                    return false
+                }
+            }
+
            function createMainFrame(previousFrame) {
                const frame = document.createElement('iframe')
                frame.id = 'main-frame'
@@ -1498,7 +1601,10 @@
                // Preview wrappers retain the .pdf path but add ?preview=1.
                // The native PDF renderer must be allowed before navigation,
                // not after a failed sandboxed load has already completed.
-                if (new URL(target, window.location.href).pathname.endsWith('.pdf')) {
+                // Native PDF embedding fails when any ancestor keeps a sandbox. A direct PDF is
+                // already handled below; LiteParse needs the same treatment for its trusted
+                // viewer frame, but only after isIsolatedLiteParsePreview proves origin isolation.
+                if (new URL(target, window.location.href).pathname.endsWith('.pdf') || isIsolatedLiteParsePreview(target)) {
                    iframe_elem.removeAttribute('sandbox')
                } else {
                    iframe_elem.sandbox = "allow-same-origin allow-top-navigation-by-user-activation allow-scripts allow-forms allow-downloads allow-popups allow-popups-to-escape-sandbox"
@@ -1548,8 +1654,8 @@
            }

            function selectInitialPreview(revealStack=true) {
-                let selectedPreviewHash = window.location.hash.slice(1)
-                try { selectedPreviewHash = decodeURIComponent(selectedPreviewHash) } catch (err) {}
+                // Match the encoded href/path as stored so `%3A` in a filename stays `%253A` in its URL.
+                const selectedPreviewHash = window.location.hash.slice(1)
                const matchingLink = findPreviewLinkForHash(selectedPreviewHash)
                if (selectedPreviewHash) {
                    // An explicit file hash must never fall back to the default card.
@@ -1565,6 +1671,13 @@
                    return false
                }

+                // A new snapshot has only the synthetic "Browse all files" card.
+                // Selecting it would replace the live monitor with an empty index.
+                // Keep progress/empty state until the user explicitly opens a file.
+                if (document.getElementById('main-frame-wrapper')?.dataset.hasOutputs !== '1') {
+                    return false
+                }
+
                const selectedCard = document.querySelector('.thumb-card.selected-card') || document.querySelector('.thumb-card')
                const selectedLink = selectedCard && (selectedCard.querySelector('a[target=preview]') || selectedCard.querySelector('a'))
                if (selectedCard && selectedLink) {
@@ -6,7 +6,7 @@
                        <div class="thumb-card{% if forloop.first %} selected-card{% endif %}" data-plugin-name="{{result.name|plugin_name}}" data-output-group="{{result.output_group}}" data-output-size="{{result.size|default:0}}"{% if preview_url %} data-preview-url="{{preview_url}}"{% endif %}{% if display_path %} data-output-path="{{display_path}}"{% endif %}>
                                <div class="thumb-body">
                                    <div class="thumb-actions">
-                                        {% if not STATIC_EXPORT %}<a href="{% snapshot_url snapshot result.folder_path|default:result.name %}/?files=1" data-no-preview="1" title="Open output folder" target="_blank" rel="noopener">📁</a>{% endif %}
+                                        {% if not STATIC_EXPORT %}<a href="{% snapshot_files_url snapshot result.folder_path %}" data-no-preview="1" title="Open output folder" target="_blank" rel="noopener">📁</a>{% endif %}
                                        {% if display_path %}
                                            <a href="{{display_url}}" data-no-preview="1" title="Download output file" download>⬇️</a>
                                        {% endif %}
@@ -28,11 +28,9 @@
                                           live plugin card uses the same scoped _card document on the
                                           snapshot origin in both stack positions. Never run saved
                                           HTML/JS/CSS on web.* or share the admin session with snap-*.{% endcomment %}
-                                        {% with plugin_base=result.name|plugin_name %}
-                                        {% if plugin_base != 'ytdlp' and plugin_base != 'yt-dlp' and plugin_base != 'youtube-dl' %}
+                                        {% if not result.name|plugin_card_is_interactive %}
                                        <a class="thumbnail-click-overlay" href="{{preview_url}}" target="preview" title="./{{display_path}} (downloaded {{result.ts}})" aria-label="Preview {{result.name|plugin_name}}"></a>
                                        {% endif %}
-                                        {% endwith %}
                                        {% plugin_card result.result %}
                                    </div>
                                {% elif result.is_metadata and display_path %}
@@ -54,7 +52,7 @@
                    <div class="thumb-card" data-other-files data-output-size="0"{% if not STATIC_EXPORT %} data-discover-files{% endif %}>
                            <div class="thumb-body">
                                <div class="thumb-actions">
-                                    <a href="{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_base_url snapshot %}/?files=1{% endif %}" data-no-preview="1" title="{% if STATIC_EXPORT %}Open snapshot manifest{% else %}Browse all snapshot files{% endif %}" target="_blank" rel="noopener">📁</a>
+                                    <a href="{% snapshot_files_url snapshot %}" data-no-preview="1" title="{% if STATIC_EXPORT %}Open snapshot manifest{% else %}Browse all snapshot files{% endif %}" target="_blank" rel="noopener">📁</a>
                                </div>
                                <h4><span class="card-title-icon">📦</span> Other files</h4>
                            </div>
@@ -63,12 +61,12 @@
                                <div class="loose-items">
                                    {% for item in loose_items %}
                                        {% if item.is_dir %}
-                                            <a href="{% if STATIC_EXPORT %}{% snapshot_url snapshot 'index.jsonl' %}{% else %}{% snapshot_url snapshot item.path %}/?files=1{% endif %}" data-output-size="{{item.size|default:0}}" data-no-preview="1" target="_blank" rel="noopener">📁 {{item.name}}</a>
+                                            <a href="{% snapshot_files_url snapshot item.path %}" data-output-size="{{item.size|default:0}}" data-no-preview="1" target="_blank" rel="noopener">📁 {{item.name}}</a>
                                        {% else %}
                                            <a href="{% snapshot_url snapshot item.path %}" data-output-size="{{item.size|default:0}}" data-no-preview="1" target="_blank" rel="noopener">📄 {{item.name}}</a>
                                        {% endif %}
                                    {% empty %}
-                                        <a href="{% snapshot_base_url snapshot %}/?files=1" data-no-preview="1" target="_blank" rel="noopener">📁 Browse all files</a>
+                                        <a href="{% snapshot_files_url snapshot %}" data-no-preview="1" target="_blank" rel="noopener">📁 Browse all files</a>
                                    {% endfor %}
                                </div>
                            </div>
@@ -1,13 +1,13 @@
+{% load core_tags %}
 <span class="snapshot-thumbnail" role="img" aria-label="Preview of {{ host }}" style="display:inline-flex;position:relative;vertical-align:middle;align-items:center;justify-content:center;overflow:hidden;border-radius:4px;box-shadow:4px 4px 4px rgba(0,0,0,0.02);width:{{ width }};height:{{ height }};background:hsl({{ hue }} 35% 92%);color:hsl({{ hue }} 35% 30%);">
  <span style="display:flex;flex-direction:column;align-items:center;gap:4px;max-width:90%;">
    <span aria-hidden="true" style="font:bold 28px sans-serif;">{{ initial }}</span>
    <span style="font:10px sans-serif;max-width:100%;overflow:hidden;text-overflow:ellipsis;white-space:nowrap;">{{ host }}</span>
    <span aria-hidden="true" style="font-size:12px;">▤</span>
  </span>
-  {% if first %}
-  <img src="{{ first.url }}" alt="" loading="lazy" decoding="async" data-candidates="{{ candidates }}" data-index="0"
-       style="position:absolute;inset:0;margin:auto;display:block;border:0;max-width:100%;max-height:100%;width:{% if first.kind == 'favicon' %}28px{% else %}100%{% endif %};height:{% if first.kind == 'favicon' %}28px{% else %}100%{% endif %};object-fit:{% if first.kind == 'favicon' %}contain{% else %}cover{% endif %};object-position:top;background:#f8fafc;"
-       onload="try{const c=JSON.parse(this.dataset.candidates)[Number(this.dataset.index)];if(c.kind==='response'&&(Math.min(this.naturalWidth,this.naturalHeight)&lt;=2||Math.max(this.naturalWidth,this.naturalHeight)&lt;=64)){this.onerror();return}if(c.kind==='favicon'){this.previousElementSibling.style.visibility='hidden'}}catch(e){this.remove()}"
-       onerror="try{const cs=JSON.parse(this.dataset.candidates);const i=Number(this.dataset.index)+1;this.dataset.index=i;if(i>=cs.length){this.remove();return}const c=cs[i];this.style.width=this.style.height=c.kind==='favicon'?'28px':'100%';this.style.objectFit=c.kind==='favicon'?'contain':'cover';this.style.objectPosition=c.kind==='favicon'?'center':'top';this.src=c.url}catch(e){this.remove()}">
+  {% if card_result %}
+  <span aria-hidden="true" style="position:absolute;inset:0;display:block;pointer-events:none;background:#f8fafc;">
+    {% plugin_card card_result %}
+  </span>
  {% endif %}
 </span>
@@ -1049,3 +1049,17 @@ input:focus, select:focus, textarea:focus, button:focus {
    text-align: center;
  }
 }
+html.archivebox-native-app #container {
+  min-width: 0;
+  width: 100%;
+}
+
+html.archivebox-native-app #content {
+  min-width: 0;
+  max-width: 100%;
+  box-sizing: border-box;
+}
+
+html.archivebox-native-app .add-page .crawl-explanation:has(.crawl-tip) {
+  display: none !important;
+}
@@ -85,6 +85,7 @@
            link.href = '/admin/core/snapshot/?tags__id__exact=' + td.id;
            link.className = 'tag-link';
            link.textContent = td.name;
+            link.title = td.name;
            pill.appendChild(link);

            var removeBtn = document.createElement('button');
@@ -1155,7 +1155,19 @@ body.change-list:not(.model-snapshot) #changelist #toolbar,
    opacity: 0.75;
 }

+#result_list tbody td.field-preview_icon {
+    position: relative;
+}
+
+#result_list tbody td.field-preview_icon > a {
+    position: absolute;
+    inset: 4px 2px;
+    display: block;
+}
+
 #result_list tbody td.field-preview_icon .snapshot-thumbnail {
+    width: 100% !important;
+    height: 100% !important;
    transition: box-shadow 140ms ease, transform 140ms ease;
 }

@@ -2226,7 +2238,7 @@ tbody .output-link:hover {opacity: 1;}
    .model-snapshot.change-list #result_list tbody th.field-created_at_display > a,
    .model-snapshot.change-list #result_list tbody td.field-created_at_display > a {
        display: inline-flex;
-        max-width: none;
+        max-width: 100%;
        white-space: nowrap !important;
    }

@@ -2234,8 +2246,10 @@ tbody .output-link:hover {opacity: 1;}
    .model-snapshot.change-list #result_list tbody td.field-created_at_display .snapshot-created-at {
        display: inline-flex;
        flex-direction: row !important;
+        flex-wrap: wrap;
        justify-content: center;
        gap: 5px;
+        row-gap: 0;
        white-space: nowrap !important;
    }

@@ -2250,13 +2264,10 @@ tbody .output-link:hover {opacity: 1;}
        grid-area: preview;
        align-self: stretch;
        place-items: center;
-        min-height: 110px;
        overflow: hidden;
    }

    .model-snapshot.change-list #result_list tbody td.field-preview_icon .snapshot-thumbnail {
-        width: 80px !important;
-        height: 110px !important;
        max-width: 100%;
        box-sizing: border-box;
    }
@@ -2278,7 +2289,7 @@ tbody .output-link:hover {opacity: 1;}
        padding-left: 0;
    }

-    .model-snapshot.change-list #result_list tbody td.field-title_str a {
+    .model-snapshot.change-list #result_list tbody td.field-title_str > a {
        display: block;
        max-width: 100%;
        overflow-wrap: anywhere;
@@ -2514,26 +2525,12 @@ tbody .output-link:hover {opacity: 1;}
    }

    .model-snapshot.change-list #result_list tbody tr {
-        grid-template-columns: 24px 84px minmax(0, 1fr);
        gap: 2px 4px;
-        grid-template-areas:
-            "check preview date"
-            "check preview title"
-            "check preview tags"
-            "check preview status"
-            "check preview files"
-            "check preview size";
    }

    .model-snapshot.change-list #result_list tbody td.field-files .files-icons {
        gap: 0;
    }
-
-    .model-snapshot.change-list #result_list tbody td.field-status_with_progress,
-    .model-snapshot.change-list #result_list tbody td.field-size_with_stats {
-        justify-self: start;
-        text-align: left;
-    }
 }

 body.embedded-change-list #content {
@@ -4086,7 +4083,7 @@ body.change-form #content-main > form > div > .submit-row {
        min-width: 0 !important;
    }

-    body.change-form #content-main form fieldset > div,
+body.change-form #content-main form fieldset > div,
    body.change-form #content form fieldset > div,
    body.change-form #content-main form fieldset .form-row,
    body.change-form #content form fieldset .form-row,
@@ -4100,3 +4097,61 @@ body.change-form #content-main > form > div > .submit-row {
        box-sizing: border-box;
    }
 }
+
+/* Presentation adjustments for the native ArchiveBox shell. The app only
+   marks its document; the server owns the layout and colors. */
+html.archivebox-native-app #header.archivebox-app-header {
+    display: flex !important;
+    align-items: center !important;
+    gap: 12px !important;
+    box-sizing: border-box;
+    width: 100%;
+    min-width: 0;
+    height: auto !important;
+    min-height: 44px;
+    padding: 8px 12px !important;
+    flex-shrink: 0;
+    background: #a51c50 !important;
+    color: #fff !important;
+    font: 500 13px/1.5 -apple-system, BlinkMacSystemFont, sans-serif;
+}
+html.archivebox-native-app #header.archivebox-app-header > :not(#branding):not(.breadcrumbs),
+html.archivebox-native-app #header.archivebox-app-header .branding-label,
+html.archivebox-native-app #header.archivebox-app-header #progress-monitor { display: none !important; }
+html.archivebox-native-app #header.archivebox-app-header #branding,
+html.archivebox-native-app #header.archivebox-app-header #site-name { margin: 0 !important; padding: 0 !important; }
+html.archivebox-native-app #header.archivebox-app-header #branding { flex: 0 0 auto; }
+html.archivebox-native-app #header.archivebox-app-header #site-name { line-height: 1 !important; }
+html.archivebox-native-app #header.archivebox-app-header #branding a { display: flex; align-items: center; }
+html.archivebox-native-app #header.archivebox-app-header #branding img {
+    width: 26px !important; height: 26px !important; margin: 0 !important; object-fit: contain;
+}
+html.archivebox-native-app #header.archivebox-app-header .breadcrumbs {
+    display: block !important;
+    visibility: visible !important;
+    opacity: 1 !important;
+    flex: 1 1 0;
+    min-width: 0;
+    margin: 0 !important;
+    padding: 0 !important;
+    background: transparent !important;
+    color: #fff !important;
+    font: inherit !important;
+    white-space: normal !important;
+    overflow-wrap: anywhere;
+}
+html.archivebox-native-app #header.archivebox-app-header .breadcrumbs a {
+    color: #fff !important; font: inherit !important; text-decoration: none; text-underline-offset: 3px;
+}
+html.archivebox-native-app #header.archivebox-app-header .breadcrumbs a:hover { text-decoration: underline; }
+html.archivebox-native-app #header.archivebox-app-header a:focus-visible { outline: 2px solid #fff; outline-offset: 3px; }
+
+html.archivebox-native-app:has(body > #progress-monitor),
+html.archivebox-native-app:has(body > #progress-monitor) body {
+    margin: 0 !important; padding: 0 !important; background: #0d1117 !important;
+}
+html.archivebox-native-app:has(body > #progress-monitor) #progress-monitor { display: block !important; min-height: 100vh; }
+html.archivebox-native-app:has(body > #progress-monitor) #progress-monitor .progress-content { display: flex !important; }
+html.archivebox-native-app:has(body > #progress-monitor) #progress-monitor .tree-container { max-height: calc(100vh - 80px) !important; }
+html.archivebox-native-app:has(body > #progress-monitor) #progress-monitor .header-bar { pointer-events: none; }
+html.archivebox-native-app:has(body > #progress-monitor) #progress-monitor #progress-collapse { display: none !important; }
@@ -324,6 +324,28 @@ def pytest_configure():
        django.setup()


+@pytest.fixture(scope="session")
+def django_db_modify_db_settings(
+    django_db_modify_db_settings_parallel_suffix,
+    request,
+):
+    """Use file-backed SQLite for threaded live_server requests in tests.
+
+    pytest-django shares an in-memory SQLite DatabaseWrapper with live_server.
+    Its threaded request handlers close that wrapper after each response, so
+    concurrent browser requests can race over the same connection.
+    """
+    from django.conf import settings
+
+    database = settings.DATABASES["default"]
+    if "sqlite" in database["ENGINE"]:
+        worker_id = getattr(request.config, "workerinput", {}).get("workerid", "")
+        tox_env = os.environ.get("TOX_PARALLEL_ENV", "")
+        suffix = "-".join(part for part in (tox_env, worker_id) if part)
+        filename = f"pytest{('-' + suffix) if suffix else ''}.sqlite3"
+        database.setdefault("TEST", {})["NAME"] = str(SESSION_DATA_DIR / filename)
+
+
@pytest.fixture(autouse=True)
 def isolate_test_runtime(tmp_path):
    """
@@ -988,11 +1010,15 @@ def kill_processes_for_data_dir(data_dir: Path) -> None:
            continue
        if pid != os.getpid():
            try:
-                os.kill(pid, signal.SIGTERM)
-            except ProcessLookupError:
+                # Retain the process identity before signalling: a fast, clean
+                # exit can remove the PID before a subsequent Process lookup.
+                # terminate() also checks that the PID has not been reused.
+                process = psutil.Process(pid)
+                process.terminate()
+            except psutil.NoSuchProcess:
                pass
            else:
-                processes.append(psutil.Process(pid))
+                processes.append(process)
    _gone, alive = psutil.wait_procs(processes, timeout=10)
    assert not alive, f"processes did not stop after SIGTERM: {[proc.pid for proc in alive]}"

@@ -1637,6 +1663,8 @@ def resolve_abxpkg_binary_env(
    deps_from: Path | list[Path] | tuple[Path, ...] | None = None,
 ) -> dict[str, str]:
    """Resolve required test dependencies through abxpkg's normal install path."""
+    from abxpkg.config import apply_exec_env
+
    command_env = dict(env) if env is not None else os.environ.copy()
    command_env["ABXPKG_LIB_DIR"] = str(lib_dir)
    command = [
@@ -1657,7 +1685,11 @@ def resolve_abxpkg_binary_env(
    )
    assert result.returncode == 0, result.stderr or result.stdout
    payload = json.loads(result.stdout)
-    return {str(key): str(value) for key, value in payload.items()}
+    # abxpkg env emits prepend/append deltas, not a replacement environment.
+    # Applying a PATH ending in ':' verbatim hides the caller's abxpkg CLI.
+    delta = {str(key): str(value) for key, value in payload.items()}
+    apply_exec_env(delta, command_env)
+    return {key: command_env[key] for key in delta}


 def run_test_hook(
@@ -1755,7 +1787,7 @@ def install_real_binary(
    from archivebox.machine.models import Binary, Machine

    binary = Binary.objects.create(
-        machine=machine or Machine.current(refresh=True),
+        machine=machine or Machine.current(),
        name=name,
        binproviders=binproviders,
        overrides=overrides or {},
@@ -1,4 +1,6 @@
 import pytest
+import os
+import time

 from archivebox.crawls.models import Crawl

@@ -15,3 +17,19 @@ def test_basic_success_case_request(client, tmp_path, api_admin_user, api_header
    response = client.get(f"/api/v1/crawls/crawl/{crawl.id}/files/folder/basic.txt", **api_headers)

    assert response.status_code == 200, response.content
+
+
+def test_screencast_frame_is_checked_by_image_request(client, api_admin_user, api_headers):
+    crawl = Crawl.objects.create(urls="https://example.com/screencast", created_by=api_admin_user)
+    frame = crawl.output_dir / "chrome_screencast" / "latest.jpg"
+    frame.parent.mkdir(parents=True, exist_ok=True)
+    frame.write_bytes(b"stored frame")
+    started_at = time.time()
+    os.utime(frame, (started_at - 10, started_at - 10))
+    url = f"/api/v1/crawls/crawl/{crawl.id}/files/chrome_screencast/latest.jpg?after={started_at}"
+    assert client.get(url, **api_headers).status_code == 404
+    os.utime(frame, (started_at + 1, started_at + 1))
+    response = client.get(url, **api_headers)
+    assert response.status_code == 200
+    assert b"".join(response.streaming_content) == b"stored frame"
+    assert client.get(url.replace(str(started_at), "invalid"), **api_headers).status_code == 400
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get("/api/v1/machine/binaries", **api_headers)
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get(f"/api/v1/machine/binary/{binary.id}", **api_headers)
@@ -8,7 +8,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()
    binary = install_real_binary("python3", machine=machine)

    response = client.get("/api/v1/machine/binary/by-name/python3", **api_headers)
@@ -7,7 +7,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    machine = Machine.current(refresh=True)
+    machine = Machine.current()

    response = client.get(f"/api/v1/machine/machine/{machine.id}", **api_headers)

@@ -7,7 +7,7 @@ pytestmark = pytest.mark.django_db(transaction=True)


 def test_basic_success_case_request(client, tmp_path, api_headers):
-    Machine.current(refresh=True)
+    Machine.current()

    response = client.get("/api/v1/machine/machines", **api_headers)

@@ -296,9 +296,11 @@ def test_archiveresult_event_create_uses_one_result_lookup():
    assert len(result_lookups) == 1


-def test_process_completed_projects_failed_archiveresult_from_shipped_hook(tmp_path, hermetic_lib_dir):
+def test_failed_shipped_title_hook_projects_failure_without_overwriting_title(tmp_path, hermetic_lib_dir):
    from archivebox.core.models import ArchiveResult

+    # Result linkage and the unchanged title are consequences of the same failed
+    # hook. Check them together so this isolated runtime is installed only once.
    snapshot = _create_snapshot()
    process, result = _run_shipped_snapshot_hook(
        snapshot,
@@ -310,22 +312,6 @@ def test_process_completed_projects_failed_archiveresult_from_shipped_hook(tmp_p
    assert result.status == ArchiveResult.StatusChoices.FAILED
    assert result.process_id == process.id
    assert "Chrome session" in result.output_str
-    _cleanup_machine_process_rows()
-
-
-def test_failed_title_archiveresult_does_not_overwrite_snapshot_title(tmp_path, hermetic_lib_dir):
-    from archivebox.core.models import ArchiveResult
-
-    snapshot = _create_snapshot()
-    _, result = _run_shipped_snapshot_hook(
-        snapshot,
-        plugin="title",
-        hook_name="on_Snapshot__54_title.js",
-        lib_dir=hermetic_lib_dir,
-        expected_exit_codes=(1,),
-    )
-    assert result.status == ArchiveResult.StatusChoices.FAILED
-    assert "Chrome session" in result.output_str
    snapshot.refresh_from_db()
    assert snapshot.title in (None, "")
    assert snapshot.resolved_title == ""
@@ -0,0 +1,377 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+import signal
+import subprocess
+
+import psutil
+from pathlib import Path
+
+import pytest
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+IMPORT_FORMAT_EXPECTATIONS = {
+    "rss": {
+        "url": "https://example.com/",
+        "title": "RSS Example Import",
+        "date": "2024-01-01",
+        "tags": {"rss-tag", "metadata"},
+    },
+    "netscape": {
+        "url": "https://www.iana.org/domains/reserved",
+        "title": "IANA Reserved Domains",
+        "date": "2024-01-02",
+        "tags": {"netscape-tag", "metadata"},
+    },
+    "dom": {
+        "url": "https://www.iana.org/help/example-domains",
+    },
+    "json": {
+        "url": "https://example.com/?archivebox-json-import=1",
+        "title": "JSON Import Example",
+        "date": "2024-01-03",
+        "tags": {"json-tag", "metadata"},
+    },
+    "jsonl": {
+        "url": "https://example.com/?archivebox-jsonl-import=1",
+        "title": "JSONL Import Example",
+        "date": "2024-01-04",
+        "tags": {"jsonl-tag", "metadata"},
+    },
+    "txt": {
+        "url": "https://example.org/",
+    },
+}
+
+
+def write_import_format_files(base_dir: Path) -> dict[str, Path]:
+    files = {
+        "rss": base_dir / "test_rss.xml",
+        "netscape": base_dir / "test_netscape.html",
+        "dom": base_dir / "test_dom.html",
+        "json": base_dir / "test_bookmarks.json",
+        "jsonl": base_dir / "test_bookmarks.jsonl",
+        "txt": base_dir / "test_urls.txt",
+    }
+    files["rss"].write_text(
+        """<?xml version="1.0" encoding="UTF-8"?>
+<rss version="2.0">
+  <channel>
+    <title>ArchiveBox RSS import fixture</title>
+    <link>https://example.com/</link>
+    <description>ArchiveBox RSS import fixture</description>
+    <item>
+      <title>RSS Example Import</title>
+      <link>https://example.com/</link>
+      <guid>https://example.com/</guid>
+      <pubDate>Mon, 01 Jan 2024 00:00:00 GMT</pubDate>
+      <category>rss-tag</category>
+      <category>metadata</category>
+    </item>
+  </channel>
+</rss>
+""",
+        encoding="utf-8",
+    )
+    files["netscape"].write_text(
+        """<!DOCTYPE NETSCAPE-Bookmark-file-1>
+<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=UTF-8">
+<TITLE>Bookmarks</TITLE>
+<H1>Bookmarks</H1>
+<DL><p>
+  <DT><A HREF="https://www.iana.org/domains/reserved" ADD_DATE="1704153600" TAGS="netscape-tag,metadata">IANA Reserved Domains</A>
+</DL><p>
+""",
+        encoding="utf-8",
+    )
+    files["dom"].write_text(
+        """<!doctype html>
+<html>
+  <head><title>DOM import fixture</title></head>
+  <body>
+    <a href="https://www.iana.org/help/example-domains">IANA Example Domains</a>
+  </body>
+</html>
+""",
+        encoding="utf-8",
+    )
+    files["json"].write_text(
+        json.dumps(
+            {
+                "url": "https://example.com/?archivebox-json-import=1",
+                "title": "JSON Import Example",
+                "tags": ["json-tag", "metadata"],
+                "bookmarked_at": "2024-01-03T00:00:00+00:00",
+            },
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    files["jsonl"].write_text(
+        json.dumps(
+            {
+                "url": "https://example.com/?archivebox-jsonl-import=1",
+                "title": "JSONL Import Example",
+                "tags": "jsonl-tag,metadata",
+                "bookmarked_at": "2024-01-04T00:00:00+00:00",
+            },
+        )
+        + "\n",
+        encoding="utf-8",
+    )
+    files["txt"].write_text(
+        "Plain text import fixture containing https://example.org/ as a real live URL.\n",
+        encoding="utf-8",
+    )
+    return files
+
+
+IMPORT_FORMAT_ENV = {
+    "USE_COLOR": "False",
+    "SHOW_PROGRESS": "False",
+    "PLUGINS": "parse_html_urls,parse_jsonl_urls,parse_netscape_urls,parse_rss_urls,parse_txt_urls",
+    "SAVE_WGET": "False",
+    "SAVE_HEADERS": "False",
+    "USE_CHROME": "False",
+    "URL_ALLOWLIST": r"example\.com|example\.org|iana\.org|www\.iana\.org",
+}
+
+
+def assert_expected_import_snapshots(cwd: Path, expected_urls: set[str]) -> None:
+    allowed_statuses = {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+    with use_archivebox_db(cwd):
+        rows = list(Snapshot.objects.filter(url__in=expected_urls).values_list("url", "status"))
+    counts = {url: 0 for url in expected_urls}
+    bad_statuses = []
+    for url, status in rows:
+        counts[url] += 1
+        if status not in allowed_statuses:
+            bad_statuses.append((url, status))
+    assert all(count == 1 for count in counts.values()), counts
+    assert not bad_statuses, bad_statuses
+
+
+def malicious_add_inputs(tmp_path: Path, *, safe_url: str) -> tuple[list[str], Path]:
+    other_crawl_source = tmp_path / "sources" / "other_crawl_source.txt"
+    other_crawl_source.parent.mkdir(parents=True, exist_ok=True)
+    other_crawl_source.write_text("https://example.com/not-owned-by-this-crawl\n", encoding="utf-8")
+    canary = tmp_path / "archivebox_shell_injection_canary"
+    return (
+        [
+            safe_url,
+            "file:///etc/hosts",
+            "/etc/hosts",
+            "../../../../etc/passwd",
+            f"file://{other_crawl_source}",
+            str(other_crawl_source),
+            f"'; touch {canary}; #",
+            f'" && touch {canary} && echo "',
+            f"$(touch {canary})",
+            f"`touch {canary}`",
+            """<?xml version="1.0"?>
+<!DOCTYPE rss [
+  <!ENTITY localfile SYSTEM "file:///etc/hosts">
+]>
+<rss version="2.0" xmlns:xi="http://www.w3.org/2001/XInclude">
+  <channel>
+    <item><title>&localfile;</title><link>file:///etc/passwd</link></item>
+    <xi:include href="file:///etc/hosts" parse="text"/>
+  </channel>
+</rss>""",
+        ],
+        canary,
+    )
+
+
+def assert_no_file_or_shell_payload_snapshots(cwd: Path, *, canary: Path) -> None:
+    with use_archivebox_db(cwd):
+        snapshots = list(Snapshot.objects.all())
+    assert not canary.exists()
+    assert not [snapshot.url for snapshot in snapshots if str(snapshot.url).startswith("file:")]
+    for forbidden in ("/etc/hosts", "/etc/passwd", "other_crawl_source", "archivebox_shell_injection_canary"):
+        assert not [snapshot.url for snapshot in snapshots if forbidden in str(snapshot.url)]
+
+
+@pytest.mark.parametrize(
+    "choice,plugins",
+    [(choice, "chrome") for choice in ["skip", "retry", "abort", "ctrl-c", "noninteractive"]] + [("noninteractive", "archivewebpage")],
+)
+def test_add_interrupts_active_hook(initialized_archive, choice, plugins):
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True")
+    # The timed observation below is for a running navigate hook. Resolve the
+    # selected plugin's real binaries before starting it, so a cold install is
+    # not counted as a hook-start failure.
+    installed = run_archivebox_cmd(["install", plugins], cwd=initialized_archive, env=env, timeout=600)
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", f"--plugins={plugins}", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=subprocess.DEVNULL if choice == "noninteractive" else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        hook = active_hook()
+        assert hook is not None
+        result.send_signal(signal.SIGINT)
+        if choice != "noninteractive":
+            read_until(lambda: b"Choice [skip]:" in output)
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            assert not psutil.pid_exists(hook.pid)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (new_hook := active_hook()) is not None and new_hook.pid != hook.pid)
+                prompt_offset = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[prompt_offset:])
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "ctrl-c":
+                result.send_signal(signal.SIGINT)
+            else:
+                os.write(master, b"a" if choice == "abort" else b"\r")
+        read_until(lambda: result.poll() is not None)
+        assert result.returncode == (130 if choice in {"abort", "ctrl-c", "noninteractive"} else 0), output.decode(errors="replace")
+        if choice == "noninteractive":
+            assert b"Choice [skip]:" not in output
+        assert not psutil.pid_exists(hook.pid)
+        assert b"Traceback" not in output
+        with use_archivebox_db(initialized_archive):
+            assert not ArchiveResult.objects.filter(hook_name="on_Snapshot__30_chrome_navigate").exists()
+            if plugins == "archivewebpage":
+                start = ArchiveResult.objects.get(hook_name="on_Snapshot__16_archivewebpage_start")
+                assert start.status == ArchiveResult.StatusChoices.NORESULTS
+                assert not ArchiveResult.objects.filter(hook_name="on_Snapshot__65_archivewebpage_stop").exists()
+                assert not ArchiveResult.objects.filter(plugin="archivewebpage", status="succeeded").exists()
+                assert not any(output["name"] == "archivewebpage" for output in start.snapshot.get_html_details_context()["archiveresults"])
+        for log in (initialized_archive / "logs").glob("worker_runner_add_*.log"):
+            assert "Choice [skip]:" not in log.read_text()
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_single_url_records_url_in_crawl(initialized_archive):
+    """Test that adding a single URL queues a crawl with the submitted URL."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "Crawl.save() outside runner process" not in result.stderr
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.urls == "https://example.com"
+    assert crawl.get_urls_list() == ["https://example.com"]
+    assert snapshots == []
+
+
+def test_add_with_depth_0_flag(initialized_archive):
+    """Test that --depth=0 flag is accepted and works."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --depth" not in result.stderr
+
+
+def test_add_records_max_url_and_size_limits_on_crawl(initialized_archive):
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=1",
+            "--max-urls=3",
+            "--crawl-max-size=45mb",
+            "--crawl-timeout=120",
+            "--snapshot-max-size=5mb",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    columns = {field.name for field in Crawl._meta.local_fields}
+    with use_archivebox_db(initialized_archive):
+        config = Crawl.objects.values_list("config", flat=True).get() or {}
+
+    assert {"max_urls", "crawl_max_size", "crawl_timeout", "snapshot_max_size"}.isdisjoint(columns)
+    assert config["CRAWL_MAX_URLS"] == 3
+    assert config["CRAWL_MAX_SIZE"] == 45 * 1024 * 1024
+    assert config["CRAWL_TIMEOUT"] == 120
+    assert config["SNAPSHOT_MAX_SIZE"] == 5 * 1024 * 1024
+
+
+def test_add_without_args_shows_usage(initialized_archive):
+    """Test that add without URLs fails with a usage hint instead of crashing."""
+
+    result = run_archivebox_cmd(
+        ["add"],
+    )
+
+    combined = result.stdout + result.stderr
+    assert result.returncode != 0
+    assert "usage" in combined.lower() or "url" in combined.lower()
@@ -0,0 +1,433 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+
+from pathlib import Path
+
+import pytest
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+    run_queued_crawls,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+@pytest.fixture
+def progress_page(httpserver):
+    # Exercise the real HTTP fetch and parser with fixed content. example.com's
+    # outgoing IANA link has changed independently of ArchiveBox releases.
+    httpserver.expect_request("/").respond_with_data(
+        '<html><body><a href="https://iana.org/domains/example">Example domains</a></body></html>',
+        content_type="text/html",
+    )
+    return httpserver.url_for("/")
+
+
+@pytest.mark.parametrize("terminal_stream", ["stdout", "stderr"])
+def test_add_renders_live_progress_in_foreground_terminal(initialized_archive, terminal_stream, progress_page):
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--plugins=parse_txt_urls", progress_page],
+            cwd=initialized_archive,
+            env=cli_env(USE_COLOR="True", SHOW_PROGRESS="True"),
+            **{terminal_stream: slave},
+            wait=False,
+            start_new_session=True,
+        )
+        deadline = time.monotonic() + 90
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            elif result.poll() is not None:
+                break
+        assert result.poll() == 0, output.decode(errors="replace")
+        assert b"\x1b[?25l" in output, output.decode(errors="replace")
+        assert b"\x1b[?25h" in output, output.decode(errors="replace")
+        assert b"on_Snapshot__71_parse_txt_urls" in output
+        assert b"\x1b[2K" in output
+        with use_archivebox_db(initialized_archive):
+            assert Crawl.objects.get().status == Crawl.StatusChoices.SEALED
+            snapshot = Snapshot.objects.get()
+            assert snapshot.url == progress_page
+            parsed_urls = Path(snapshot.output_dir) / "parse_txt_urls" / "urls.jsonl"
+            assert json.loads(parsed_urls.read_text())["url"] == "https://iana.org/domains/example"
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=10)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_redirected_progress_remains_plain_text(initialized_archive, progress_page):
+    result = run_archivebox_cmd(
+        ["add", "--plugins=parse_txt_urls", progress_page],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    assert "on_Snapshot__71_parse_txt_urls" in result.stderr
+    assert "\x1b[?25l" not in result.stdout + result.stderr
+
+
+@pytest.mark.timeout(360)
+def test_add_stdin_import_formats_preserve_metadata_and_crawl_inner_urls(initialized_archive):
+    """`archivebox add < import-file` should normalize rich import formats before crawling URLs."""
+    import_files = write_import_format_files(initialized_archive)
+    expected_urls = {case["url"] for case in IMPORT_FORMAT_EXPECTATIONS.values()}
+    env = cli_env(**IMPORT_FORMAT_ENV)
+
+    for import_path in import_files.values():
+        source_text = import_path.read_text(encoding="utf-8")
+        result = run_archivebox_cmd(
+            ["add", "--bg", "--depth=0", "--tag=cli-stdin-import"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=source_text,
+            timeout=360,
+        )
+        assert result.returncode == 0, result.stderr or result.stdout
+        with use_archivebox_db(initialized_archive):
+            crawl = Crawl.objects.order_by("-created_at").first()
+            assert crawl is not None
+            assert crawl.snapshot_set.count() == 0
+        assert crawl.urls == source_text
+
+    run_queued_crawls(initialized_archive, env=env, timeout=240)
+    with use_archivebox_db(initialized_archive):
+        for crawl in Crawl.objects.all():
+            assert not crawl.snapshot_set.filter(url__startswith="archivebox://").exists()
+            root_input = (crawl.output_dir / "input" / "staticfile" / "stdin.txt").read_text(encoding="utf-8")
+            assert root_input == crawl.urls
+    assert_expected_import_snapshots(initialized_archive, expected_urls)
+
+    list_result = run_archivebox_cmd(
+        ["list", "--json"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert list_result.returncode == 0, list_result.stderr or list_result.stdout
+    for expected_url in expected_urls:
+        assert expected_url in list_result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawls = list(Crawl.objects.order_by("created_at"))
+        snapshots_by_url = {snapshot.url: snapshot for snapshot in Snapshot.objects.prefetch_related("tags").filter(url__in=expected_urls)}
+        tags_by_url = {snapshot.url: set(snapshot.tags.values_list("name", flat=True)) for snapshot in snapshots_by_url.values()}
+
+    assert len(crawls) == len(import_files)
+    assert [crawl.urls for crawl in crawls] == [path.read_text(encoding="utf-8") for path in import_files.values()]
+    assert all(crawl.tags_str == "cli-stdin-import" for crawl in crawls)
+    assert all(crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED} for crawl in crawls)
+    assert len(snapshots_by_url) == len(expected_urls)
+
+    for import_name, expected in IMPORT_FORMAT_EXPECTATIONS.items():
+        snapshot = snapshots_by_url.get(expected["url"])
+        assert snapshot is not None, f"{import_name} did not create Snapshot for {expected['url']}"
+        assert snapshot.status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+        if expected.get("title"):
+            assert snapshot.title == expected["title"]
+        if expected.get("date"):
+            assert snapshot.bookmarked_at.date().isoformat() == expected["date"]
+        if expected.get("tags"):
+            assert expected["tags"] | {"cli-stdin-import"} <= tags_by_url[snapshot.url]
+
+
+def test_add_bg_queues_direct_url_snapshot(initialized_archive):
+    """Background add queues explicit URL arguments as real URL snapshots."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is not None
+    assert crawl.urls == "https://example.com"
+    assert snapshots == []
+
+
+@pytest.mark.timeout(180)
+def test_add_title_hook_env_gets_canonical_runtime_config_without_archivebox_selectors_or_aliases(initialized_archive):
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "False",
+            "SHOW_PROGRESS": "False",
+            "PLUGINS": "title",
+            "SAVE_TITLE": "True",
+            "SECRET_KEY": "hook-env-secret-must-not-leak",
+            "PUBLIC_ADD_VIEW": "True",
+            "ADMIN_PASSWORD": "hook-env-admin-password-must-not-leak",
+            "URL_BLACKLIST": "$^",
+            "ARCHIVEBOX_TEST_ARBITRARY_ENV": "preserved-user-env",
+            "LD_PRELOAD": "",
+            "TIMEOUT": "60",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "1",
+        },
+    )
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--plugins=title",
+            "https://example.com/?archivebox-title-env=1",
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "Crawl.save() outside runner process" not in result.stdout + result.stderr
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot = Snapshot.objects.get()
+        title_result = ArchiveResult.objects.select_related("process").get(snapshot=snapshot, plugin="title")
+        process_env = title_result.process.env
+
+    assert crawl.config["PLUGINS"] == "title"
+    assert snapshot.url == "https://example.com/?archivebox-title-env=1"
+    assert snapshot.status == Snapshot.StatusChoices.SEALED
+    assert snapshot.title == "Example Domain"
+    assert title_result.status == ArchiveResult.StatusChoices.SUCCEEDED
+    assert title_result.output_str == "Example Domain"
+    assert title_result.process.exit_code == 0
+    assert process_env["TITLE_ENABLED"] == "True"
+    assert process_env["CHROME_ENABLED"] == "True"
+    assert process_env["WGET_ENABLED"] == "False"
+    assert process_env["ARCHIVEBOX_TEST_ARBITRARY_ENV"] == "preserved-user-env"
+    assert process_env["LD_PRELOAD"] == ""
+    assert "PLUGINS" not in process_env
+    assert "SAVE_TITLE" not in process_env
+    assert "SEARCH_BACKEND_ENGINE" not in process_env
+    assert "SECRET_KEY" not in process_env
+    assert "ADMIN_PASSWORD" not in process_env
+    assert "PUBLIC_ADD_VIEW" not in process_env
+    assert "URL_BLACKLIST" not in process_env
+    assert "hook-env-secret-must-not-leak" not in json.dumps(process_env)
+    assert "hook-env-admin-password-must-not-leak" not in json.dumps(process_env)
+
+
+def test_add_creates_crawl_record(initialized_archive):
+    """Test that add command creates a Crawl record in the database."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        crawl_count = Crawl.objects.count()
+
+    assert crawl_count == 1
+
+
+def test_add_rejects_file_path_argument(initialized_archive):
+    """Local files must be piped through stdin, not passed as archiveable path arguments."""
+    env = cli_env(disable_extractors=True)
+    urls_file = initialized_archive / "urls.txt"
+    urls_file.write_text("https://example.com\nhttps://example.org\n")
+
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", str(urls_file)],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode != 0
+
+    with use_archivebox_db(initialized_archive):
+        assert Crawl.objects.count() == 0
+        assert Snapshot.objects.count() == 0
+
+    assert "No URLs provided" in (result.stderr or result.stdout)
+
+
+def test_add_records_url_filter_overrides_on_crawl(initialized_archive):
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=0",
+            "--domain-allowlist=example.com,*.example.com",
+            "--domain-denylist=static.example.com",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+
+    assert crawl.config["URL_ALLOWLIST"] == "example.com,*.example.com"
+    assert crawl.config["URL_DENYLIST"] == "static.example.com"
+    assert not (initialized_archive / "personas" / "Default" / "chrome_extensions").exists()
+
+
+def test_add_help_shows_depth_and_tag_options(initialized_archive):
+    """Test that add --help documents the main filter and crawl options."""
+
+    result = run_archivebox_cmd(
+        ["add", "--help"],
+    )
+
+    assert result.returncode == 0
+    assert "--depth" in result.stdout
+    assert "--max-urls" in result.stdout
+    assert "--crawl-max-size" in result.stdout
+    assert "--crawl-timeout" in result.stdout
+    assert "--snapshot-max-size" in result.stdout
+    assert "--tag" in result.stdout
+
+
+def test_add_index_only_creates_direct_url_snapshot(initialized_archive):
+    """Index-only add seeds explicit URL args for the runner to snapshot."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot = Snapshot.objects.get()
+
+    assert crawl.urls == "https://example.com"
+    assert snapshot.url == "https://example.com"
+    assert snapshot.depth == 0
+
+
+def test_snapshot_create_sets_snapshot_timestamp(initialized_archive):
+    """Test the user-facing snapshot creation path sets a timestamp."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["snapshot", "create", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        check=True,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        timestamp = Snapshot.objects.values_list("timestamp", flat=True).get()
+
+    assert timestamp is not None
+    assert len(str(timestamp)) > 0
+
+
+@pytest.mark.timeout(180)
+def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive, recursive_test_site):
+
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "TIMEOUT": "60",
+            "SAVE_WGET": "true",
+            "SAVE_HEADERS": "false",
+            "SAVE_TITLE": "false",
+            "SAVE_READABILITY": "false",
+            "SAVE_SINGLEFILE": "false",
+            "SAVE_MERCURY": "false",
+            "SAVE_SCREENSHOT": "false",
+            "SAVE_PDF": "false",
+            "SAVE_DOM": "false",
+            "SAVE_ARCHIVEDOTORG": "false",
+            "SAVE_GIT": "false",
+            "SAVE_YTDLP": "false",
+            "SAVE_FAVICON": "false",
+            "PARSE_HTML_URLS_ENABLED": "true",
+            "PARSE_DOM_OUTLINKS_ENABLED": "false",
+            "URL_ALLOWLIST": r"127\.0\.0\.1[:/].*",
+        },
+    )
+    root_url = recursive_test_site["root_url"]
+    child_url = recursive_test_site["child_urls"][0]
+
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=2",
+            "--max-urls=2",
+            "--crawl-max-size=50mb",
+            "--tag=recursive-flow",
+            "--parser=url_list",
+            "--plugins=wget,parse_html_urls",
+            root_url,
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()
+        snapshots = list(Snapshot.objects.order_by("depth", "url").values_list("url", "depth", "status"))
+        archive_results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__depth", "snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_files"),
+        )
+
+    assert crawl[0] == 2
+    assert crawl[1] == "recursive-flow"
+    crawl_config = crawl[2] or {}
+    assert crawl_config["CRAWL_MAX_URLS"] == 2
+    assert crawl_config["CRAWL_MAX_SIZE"] == 50 * 1024 * 1024
+    assert crawl_config.get("SNAPSHOT_MAX_SIZE", 0) == 0
+    assert (root_url, 0, "sealed") in snapshots
+    assert any(url == child_url and depth == 1 and status == "sealed" for url, depth, status in snapshots)
+
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _files in archive_results}
+    assert by_url_plugin[(root_url, "wget")] == "succeeded"
+    assert by_url_plugin[(root_url, "parse_html_urls")] == "succeeded"
+    assert by_url_plugin[(child_url, "wget")] == "succeeded"
+
+    urls_outputs = list((initialized_archive / "archive/users/system/snapshots").rglob("parse_html_urls/urls.jsonl"))
+    assert urls_outputs
+    assert any(child_url in path.read_text() for path in urls_outputs)
@@ -0,0 +1,456 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import pty
+import select
+import time
+import termios
+import signal
+import re
+import subprocess
+
+import psutil
+from pathlib import Path
+
+import pytest
+from django.utils import timezone
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    run_archivebox_cmd,
+    run_queued_crawls,
+    resolve_abxpkg_chrome_env,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+@pytest.mark.parametrize("choice", ["skip", "retry", "ctrl-c", "double", "noninteractive"])
+def test_add_interrupts_background_only_capture(initialized_archive, choice):
+    """Neither pause nor abort/reclaim semantics may depend on a foreground hook."""
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.05)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(archiveresult__plugin="forumdl", status="running").order_by("-started_at").first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=forumdl", "https://news.ycombinator.com"],
+            cwd=initialized_archive,
+            env=cli_env(),
+            stdin=subprocess.DEVNULL if choice == "noninteractive" else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        hook = active_hook()
+        children = psutil.Process(result.pid).children(recursive=True)
+        result.send_signal(signal.SIGINT)
+        if choice == "double":
+            read_until(lambda: b"Forwarding Ctrl+C" in output)
+            result.send_signal(signal.SIGINT)
+        elif choice != "noninteractive":
+            read_until(lambda: b"Choice [skip]:" in output, timeout=20)
+            read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            assert not psutil.pid_exists(hook.pid)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (next_hook := active_hook()) is not None and next_hook.pid != hook.pid)
+                children.extend(psutil.Process(result.pid).children(recursive=True))
+                offset = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[offset:], timeout=20)
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            os.write(master, b"\r" if choice == "skip" else b"\x03")
+        read_until(lambda: result.poll() is not None, timeout=30)
+        assert result.returncode == (0 if choice == "skip" else 130), output.decode(errors="replace")
+        assert not [child.pid for child in children if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert not psutil.pid_exists(hook.pid)
+        assert b"Traceback" not in output
+        if choice == "noninteractive":
+            assert b"Choice [skip]:" not in output
+        # An explicit abort must not be mistaken for takeover and restart work.
+        with use_archivebox_db(initialized_archive):
+            assert Process.objects.filter(worker_type="worker_runner", process_type="orchestrator").count() == 1
+        if choice != "skip":
+            assert b"Aborting crawl" in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=20)
+        os.close(slave)
+        os.close(master)
+
+
+def test_background_completions_render_above_interrupt_prompt(initialized_archive, recursive_test_site):
+    env = cli_env(WGET_ARGS_EXTRA='["--limit-rate=20"]', INFINISCROLL_SCROLL_DELAY="10000", CHROME_HEADLESS="True")
+    # The timed observation is for a running infinite-scroll hook. Resolve its
+    # real browser and wget dependencies before the readiness window begins.
+    installed = run_archivebox_cmd(
+        ["install", "wget", "chrome"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=600,
+    )
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+    env.update(resolve_abxpkg_chrome_env(Path(env["ABXPKG_LIB_DIR"]), env))
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        for log in sorted(initialized_archive.rglob("on_Snapshot__*.*.log")):
+            print(f"\n{log.relative_to(initialized_archive)}:\n{log.read_text(errors='replace')}")
+        raise AssertionError(output.decode(errors="replace"))
+
+    def result_for(plugin):
+        with use_archivebox_db(initialized_archive):
+            return ArchiveResult.objects.filter(plugin=plugin).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=wget,infiniscroll,parse_html_urls", recursive_test_site["root_url"]],
+            cwd=initialized_archive,
+            # The small test-owned page still transfers slowly enough for wget
+            # to finish while the real foreground hook's prompt is open.
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: (scroll := result_for("infiniscroll")) is not None and scroll.status == "started")
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        before_completion = len(output)
+        read_until(lambda: (wget := result_for("wget")) is not None and wget.status == "succeeded")
+        read_until(lambda: b"Choice [skip]:" in output[before_completion:])
+        screen_updates = re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", output[before_completion:].decode(errors="replace"))
+        completed_at = screen_updates.index("on_Snapshot__35_wget")
+        prompt_at = screen_updates.rindex("Interrupted on_Snapshot__45_infiniscroll")
+        assert "succeeded" in screen_updates[completed_at:prompt_at], screen_updates
+        assert completed_at < prompt_at < screen_updates.rindex("Choice [skip]:"), screen_updates
+        assert result_for("parse_html_urls") is None
+        (initialized_archive / "interrupt-terminal.txt").write_bytes(output)
+        os.write(master, b"a")
+        read_until(lambda: result.poll() is not None)
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_summary_points_to_snapshot_outputs(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--plugins=parse_txt_urls", "https://example.com"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get()
+        assert Path(snapshot.output_dir).is_dir()
+        assert str(snapshot.output_dir) in result.stdout
+        assert "snapshot output saved to:" in result.stdout
+        assert "crawl logs and setup:" in result.stdout
+        assert "crawl output saved to:" not in result.stdout
+
+
+def test_initial_crawl_creation_does_not_warn_as_an_outside_runner_update(initialized_archive, caplog):
+    from archivebox.core.takeover_util import current_command
+
+    with use_archivebox_db(initialized_archive):
+        command = current_command(Process.TypeChoices.ADD, data_dir=initialized_archive)
+        caplog.clear()
+        with caplog.at_level("WARNING", logger="archivebox.workers.models"):
+            crawl = Crawl.objects.create(urls="https://example.com", status=Crawl.StatusChoices.QUEUED)
+        assert "Crawl.save() outside runner process" not in caplog.text
+
+        caplog.clear()
+        with caplog.at_level("WARNING", logger="archivebox.workers.models"):
+            crawl.save(update_fields=["modified_at"])
+        command.mark_exited(exit_code=0)
+
+    assert "Crawl.save() outside runner process" in caplog.text
+
+
+@pytest.mark.timeout(240)
+def test_add_rejects_file_path_and_shell_injection_payloads(initialized_archive):
+    """CLI add must not turn user-supplied local paths or shell payloads into snapshots."""
+    safe_url = "https://example.com/?archivebox-cli-security=1"
+    inputs, canary = malicious_add_inputs(initialized_archive, safe_url=safe_url)
+    env = cli_env(**IMPORT_FORMAT_ENV)
+
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--depth=0", "--tag=cli-security"],
+        cwd=initialized_archive,
+        env=env,
+        stdin="\n".join(inputs),
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    run_queued_crawls(initialized_archive, env=env, timeout=120)
+    assert_expected_import_snapshots(initialized_archive, {safe_url})
+
+    assert_no_file_or_shell_payload_snapshots(initialized_archive, canary=canary)
+    with use_archivebox_db(initialized_archive):
+        snapshots = list(Snapshot.objects.filter(url=safe_url).values_list("url", "status", "tags__name"))
+        crawl = Crawl.objects.get()
+    assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
+    assert len({url for url, _status, _tag in snapshots}) == 1
+    assert all(
+        status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
+        for _url, status, _tag in snapshots
+    )
+    assert "cli-security" in {tag for _url, _status, tag in snapshots}
+
+
+@pytest.mark.timeout(180)
+def test_run_rejects_depth_two_file_url_snapshot_injected_directly_with_db_update(initialized_archive):
+    """A queued Snapshot row with a file:// URL must not be allowed to run hooks or recurse."""
+    secret_url = "https://example.com/?archivebox-sql-depth2-file-secret=1"
+    local_source = initialized_archive / "not_owned_by_depth_two_snapshot.txt"
+    local_source.write_text(f"{secret_url}\n", encoding="utf-8")
+    file_url = local_source.resolve().as_uri()
+    env = cli_env(**{**IMPORT_FORMAT_ENV, "PLUGINS": "parse_txt_urls", "SAVE_HEADERS": "False", "SAVE_WGET": "False"})
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/?archivebox-sql-depth2-root=1",
+            max_depth=2,
+            tags_str="sql-depth2-file-url",
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        root_snapshot = Snapshot.objects.create(
+            url="https://example.com/?archivebox-sql-depth2-root=1",
+            crawl=crawl,
+            depth=0,
+            status=Snapshot.StatusChoices.SEALED,
+            retry_at=None,
+        )
+        injected_snapshot = Snapshot.objects.create(
+            url="https://example.com/?archivebox-sql-depth2-placeholder=1",
+            crawl=crawl,
+            parent_snapshot=root_snapshot,
+            depth=2,
+            status=Snapshot.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        Snapshot.objects.filter(pk=injected_snapshot.pk).update(url=file_url)
+
+    result = run_archivebox_cmd(
+        ["run", f"--snapshot-id={injected_snapshot.id}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        injected_snapshot.refresh_from_db()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+        file_results = list(
+            ArchiveResult.objects.filter(
+                snapshot=injected_snapshot,
+            ).values_list("plugin", "status"),
+        )
+
+    assert injected_snapshot.url == file_url
+    assert secret_url not in snapshot_urls
+    assert file_results == []
+
+
+def test_add_index_only_rejected_urls_leave_empty_crawl_for_runner_to_seal(initialized_archive):
+    """Index-only add only creates the crawl; rejected URLs are sealed by the runner."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--index-only",
+            "--depth=0",
+            "--url-denylist=example.com",
+            "https://example.com",
+        ],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshot_urls == set()
+
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status == Crawl.StatusChoices.SEALED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshot_urls == set()
+
+
+def test_add_index_only_rejects_archivebox_internal_urls(initialized_archive):
+    """Index-only add must apply the same internal URL guard as snapshot creation."""
+    env = cli_env(disable_extractors=True)
+    internal_urls = [
+        "http://archivebox.localhost:9292/admin/",
+        "http://web.archivebox.localhost:9292/",
+        "http://api.archivebox.localhost:9292/api/v1/docs",
+        "http://snap-2fb8e923c58c.archivebox.localhost:9292/index.html",
+    ]
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", *internal_urls],
+        cwd=initialized_archive,
+        env={**env, "BASE_URL": "http://archivebox.localhost:9292"},
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.urls.splitlines() == internal_urls
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert snapshot_urls == set()
+
+
+def test_add_multiple_urls_single_command(initialized_archive):
+    """Test adding multiple URLs in a single command records one crawl."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.order_by("url").values_list("url", "depth"))
+
+    assert crawl.urls.splitlines() == ["https://example.com", "https://example.org"]
+    assert snapshots == []
+
+
+def test_add_with_tags(initialized_archive):
+    """Test adding URL with tags stores tags_str in crawl.
+
+    With --index-only, Tag objects are not created until archiving happens.
+    Tags are stored as a string in the Crawl.tags_str field.
+    """
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "--tag=test,example", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        tags_str = Crawl.objects.values_list("tags_str", flat=True).get()
+
+    # Tags are stored as a comma-separated string in crawl
+    assert "test" in tags_str or "example" in tags_str
+
+
+def test_add_with_overwrite_flag(initialized_archive):
+    """Test that --overwrite flag forces re-archiving."""
+    env = cli_env(disable_extractors=True)
+
+    # Add URL first time
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    # Add with overwrite
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--overwrite", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --overwrite" not in result.stderr
+
+
+def test_add_index_only_queues_crawl_without_starting_runner(initialized_archive):
+    """Test that --index-only creates only a queued crawl and returns fast."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=30,  # Should be fast
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.all())
+
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.retry_at is None
+    assert crawl.urls == "https://example.com"
+    assert snapshots == []
@@ -0,0 +1,511 @@
+#!/usr/bin/env python3
+"""
+Comprehensive tests for archivebox add command.
+Verify add creates snapshots in DB, crawls, source files, and archive directories.
+"""
+
+import os
+import json
+import pty
+import select
+import time
+import termios
+import signal
+import re
+
+import psutil
+from pathlib import Path
+
+import pytest
+from django.utils import timezone
+
+from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Process
+from archivebox.tests.conftest import (
+    cli_env,
+    find_snapshot_dir,
+    run_archivebox_cmd,
+    run_queued_crawls,
+    resolve_abxpkg_chrome_env,
+)
+
+from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+from .test_cli_add_1 import (
+    pytestmark as pytestmark,
+    IMPORT_FORMAT_EXPECTATIONS as IMPORT_FORMAT_EXPECTATIONS,
+    write_import_format_files as write_import_format_files,
+    IMPORT_FORMAT_ENV as IMPORT_FORMAT_ENV,
+    assert_expected_import_snapshots as assert_expected_import_snapshots,
+    malicious_add_inputs as malicious_add_inputs,
+    assert_no_file_or_shell_payload_snapshots as assert_no_file_or_shell_payload_snapshots,
+)
+
+
+def test_abort_stops_long_running_background_hook(initialized_archive):
+    env = cli_env(INFINISCROLL_SCROLL_DELAY="10000", CHROME_HEADLESS="True")
+    # Keep first-use dependency installation outside the timed hook-start
+    # observation. This test is about interrupting a running hook; a cold
+    # browser or forumdl install must not consume its readiness window.
+    installed = run_archivebox_cmd(
+        ["install", "forumdl", "chrome"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=600,
+    )
+    assert installed.returncode == 0, installed.stderr or installed.stdout
+    env.update(resolve_abxpkg_chrome_env(Path(env["ABXPKG_LIB_DIR"]), env))
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 200))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def result_for(plugin):
+        with use_archivebox_db(initialized_archive):
+            return ArchiveResult.objects.filter(plugin=plugin).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["add", "--depth=1", "--plugins=forumdl,infiniscroll", "https://news.ycombinator.com"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        try:
+            read_until(lambda: (scroll := result_for("infiniscroll")) is not None and scroll.status == "started")
+        except AssertionError:
+            # Capture state before teardown terminates the runner: otherwise a
+            # missed readiness deadline loses the install/hook failure itself.
+            with use_archivebox_db(initialized_archive):
+                print("Hook readiness:", list(ArchiveResult.objects.values("plugin", "status", "output_str")))
+                print("Process readiness:", list(Process.objects.values("process_type", "status", "exit_code", "stderr")))
+            raise
+        read_until(lambda: "running pid=" in re.sub(r"\x1b\[[0-?]*[ -/]*[@-~]", "", output.decode(errors="replace")))
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        forum = result_for("forumdl")
+        assert forum is not None and forum.status == "started"
+        children = psutil.Process(result.pid).children(recursive=True)
+        abort_offset = len(output)
+        os.write(master, b"\x03")
+        read_until(lambda: b"Aborting crawl" in output[abort_offset:], timeout=3)
+        read_until(lambda: result.poll() is not None, timeout=25)
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert not [child.pid for child in children if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert any("INFO:root:GET" in log.read_text() for log in initialized_archive.rglob("on_Snapshot__33_forumdl.*.stderr.log"))
+        assert b"Stopped during crawl abort" in output
+        assert b"INFO:root:GET" not in output[abort_offset:]
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=15)
+        os.close(slave)
+        os.close(master)
+
+
+def test_add_bg_persists_queue_without_starting_extraction_workers(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--plugins=wget", "--crawl-max-concurrent-snapshots=4", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        assert crawl.status == Crawl.StatusChoices.QUEUED
+        assert crawl.retry_at is not None
+        assert crawl.get_urls_list() == ["https://example.com", "https://example.org"]
+        assert crawl.config["PLUGINS"] == "wget"
+        assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 4
+        assert not Snapshot.objects.exists()
+        assert not ArchiveResult.objects.exists()
+        assert not Process.objects.filter(process_type__in=[Process.TypeChoices.ORCHESTRATOR, Process.TypeChoices.HOOK]).exists()
+
+
+@pytest.mark.timeout(180)
+def test_run_rejects_file_url_injected_directly_into_crawl_urls_with_db_update(initialized_archive):
+    """Runner must validate Crawl.urls again when DB writes bypass normal add/create paths."""
+    secret_url = "https://example.com/?archivebox-sql-crawl-file-secret=1"
+    local_source = initialized_archive / "not_owned_by_crawl_urls.txt"
+    local_source.write_text(f"{secret_url}\n", encoding="utf-8")
+    file_url = local_source.resolve().as_uri()
+    env = cli_env(**{**IMPORT_FORMAT_ENV, "PLUGINS": "parse_txt_urls", "SAVE_HEADERS": "False", "SAVE_WGET": "False"})
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/?archivebox-sql-crawl-control=1",
+            max_depth=2,
+            tags_str="sql-file-url",
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+        )
+        bad_jsonl = json.dumps({"type": "Snapshot", "url": file_url, "depth": 0, "tags": "sql-file-url"})
+        Crawl.objects.filter(pk=crawl.pk).update(urls=bad_jsonl)
+
+    result = run_archivebox_cmd(
+        ["run", f"--crawl-id={crawl.id}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl.refresh_from_db()
+        snapshot_urls = set(Snapshot.objects.values_list("url", flat=True))
+
+    assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
+    assert file_url not in snapshot_urls
+    assert secret_url not in snapshot_urls
+
+
+@pytest.mark.timeout(180)
+def test_add_tagged_single_url_seals_without_duplicate_snapshot_tags(initialized_archive):
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "False",
+            "SHOW_PROGRESS": "False",
+            "PLUGINS": "parse_txt_urls,title",
+            "TIMEOUT": "60",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "1",
+        },
+    )
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--bg",
+            "--depth=0",
+            "--tag=tagged-single-url",
+            "https://example.com/?archivebox-tagged-single-url=1",
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    run_queued_crawls(initialized_archive, env, timeout=180)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshots = list(Snapshot.objects.order_by("depth", "url"))
+        tag_counts = {
+            snapshot.url: SnapshotTag.objects.filter(snapshot=snapshot, tag__name="tagged-single-url").count() for snapshot in snapshots
+        }
+        results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__depth", "snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_str"),
+        )
+
+    assert crawl.status == Crawl.StatusChoices.SEALED
+    assert [(snapshot.url, snapshot.depth, snapshot.status) for snapshot in snapshots] == [
+        ("https://example.com/?archivebox-tagged-single-url=1", 0, Snapshot.StatusChoices.SEALED),
+    ]
+    assert tag_counts == {
+        "https://example.com/?archivebox-tagged-single-url=1": 1,
+    }
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _output in results}
+    assert by_url_plugin[("https://example.com/?archivebox-tagged-single-url=1", "title")] == "succeeded"
+    unexpected_failures = [(url, plugin, status, output) for url, plugin, status, output in results if status == "failed"]
+    assert not unexpected_failures
+
+
+def test_add_direct_url_creates_snapshot_without_internal_input_file(initialized_archive):
+    """Explicit URL args materialize real snapshots when the runner claims the crawl."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get()
+        assert snapshot.url == "https://example.com"
+        assert snapshot.depth == 0
+        assert not (snapshot.output_dir / "staticfile" / "stdin.txt").exists()
+
+
+def test_add_with_depth_1_flag(initialized_archive):
+    """Test that --depth=1 flag is accepted."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=1", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+    assert "unrecognized arguments: --depth" not in result.stderr
+
+
+def test_add_rejects_invalid_depth_values(initialized_archive):
+    """Test that add rejects depth values outside the supported range."""
+    env = cli_env(disable_extractors=True)
+
+    for depth in ("5", "-1"):
+        result = run_archivebox_cmd(
+            ["add", "--index-only", f"--depth={depth}", "https://example.com"],
+            cwd=initialized_archive,
+            env=env,
+        )
+        stderr = result.stderr.lower()
+        assert result.returncode != 0
+        assert "invalid" in stderr or "not one of" in stderr
+
+
+def test_add_records_selected_persona_on_crawl(initialized_archive):
+    """Test add persists the selected persona so browser config derives from it later."""
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "--persona=Default", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    assert result.returncode == 0
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+
+    assert crawl.persona_id
+    assert "ACTIVE_PERSONA" not in crawl.config
+    assert (initialized_archive / "personas" / "Default" / "chrome_profile").is_dir()
+
+
+def test_add_duplicate_url_creates_separate_crawls(initialized_archive):
+    """Test that adding the same URL twice creates separate crawls.
+
+    Each 'add' command creates a new Crawl. Multiple crawls can archive the same URL.
+    This allows re-archiving URLs at different times.
+    """
+
+    env = cli_env(disable_extractors=True)
+    # Add URL first time
+    run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+
+    # Add same URL second time with --update to opt out of ONLY_NEW.
+    run_archivebox_cmd(
+        ["add", "--index-only", "--update", "--depth=0", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        crawl_count = Crawl.objects.count()
+        snapshots = list(Snapshot.objects.order_by("created_at").values_list("url", "depth"))
+
+    # Each add creates a new crawl with its own queued work.
+    assert crawl_count == 2
+    assert snapshots == [("https://example.com", 0), ("https://example.com", 0)]
+
+
+def test_snapshot_create_creates_current_output_directory(initialized_archive):
+    """Test the user-facing snapshot creation path creates an output directory."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(
+        ["snapshot", "create", "https://example.com"],
+        cwd=initialized_archive,
+        env=env,
+        check=True,
+    )
+
+    with use_archivebox_db(initialized_archive):
+        snapshot_id = str(Snapshot.objects.values_list("id", flat=True).get())
+
+    snapshot_dir = find_snapshot_dir(initialized_archive, snapshot_id)
+    assert snapshot_dir is not None, f"Snapshot output directory not found for {snapshot_id}"
+    assert snapshot_dir.is_dir()
+
+
+def test_add_preserves_apostrophe_in_snapshot_url(initialized_archive):
+    url = "https://aaib.gov.in/What's%20New%20Assets/Preliminary%20Report%20VT-EXO.pdf"
+    env = cli_env(disable_extractors=True)
+    result = run_archivebox_cmd(
+        ["add", "--index-only", "--depth=0", url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert result.returncode == 0, result.stderr
+    run_queued_crawls(initialized_archive, env)
+
+    with use_archivebox_db(initialized_archive):
+        assert Crawl.objects.get().urls == url
+        assert Snapshot.objects.get().url == url
+
+
+@pytest.mark.timeout(180)
+def test_cli_add_real_urls_with_options_writes_inspectable_outputs(initialized_archive):
+
+    wget_urls = [
+        "https://example.com",
+        "https://pirate.github.io/stress-tests/challenge.html",
+    ]
+    chrome_url = "https://example.com/?archivebox-chrome-flow=1"
+    env = os.environ.copy()
+    env.pop("CHROME_BINARY", None)
+    env.update(
+        {
+            "USE_COLOR": "false",
+            "SHOW_PROGRESS": "false",
+            "TIMEOUT": "60",
+            "SAVE_WGET": "true",
+            "SAVE_HEADERS": "false",
+            "SAVE_TITLE": "false",
+            "SAVE_READABILITY": "false",
+            "SAVE_SINGLEFILE": "false",
+            "SAVE_MERCURY": "false",
+            "SAVE_SCREENSHOT": "false",
+            "SAVE_PDF": "false",
+            "SAVE_DOM": "false",
+            "SAVE_ARCHIVEDOTORG": "false",
+            "SAVE_GIT": "false",
+            "SAVE_YTDLP": "false",
+            "SAVE_FAVICON": "false",
+        },
+    )
+    _cmd_result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--max-urls=2",
+            "--crawl-max-size=10mb",
+            "--tag=real-flow,challenge",
+            "--parser=url_list",
+            "--plugins=wget",
+            *wget_urls,
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert returncode == 0, stderr or stdout
+
+    chrome_env = env | {
+        "SAVE_WGET": "false",
+        "SAVE_HEADERS": "true",
+        "SAVE_TITLE": "true",
+        "CHROME_HEADLESS": "true",
+        "CHROME_SANDBOX": "false",
+        "CHROME_ISOLATION": "snapshot",
+    }
+    _cmd_result = run_archivebox_cmd(
+        ["install", "chrome"],
+        cwd=initialized_archive,
+        env=chrome_env,
+        timeout=600,
+    )
+    install_stdout, install_stderr, install_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert install_returncode == 0, install_stderr or install_stdout
+    chrome_env.update(resolve_abxpkg_chrome_env(Path(chrome_env["ABXPKG_LIB_DIR"]), chrome_env))
+    _cmd_result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--max-urls=1",
+            "--crawl-max-size=10mb",
+            "--tag=chrome-flow",
+            "--parser=url_list",
+            "--plugins=chrome,wget,headers,title",
+            chrome_url,
+        ],
+        cwd=initialized_archive,
+        env=chrome_env,
+        timeout=180,
+    )
+    chrome_stdout, chrome_stderr, chrome_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert chrome_returncode == 0, chrome_stderr or chrome_stdout
+
+    _cmd_result = run_archivebox_cmd(
+        ["list", "--tag=real-flow"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    list_stdout, list_stderr, list_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    assert list_returncode == 0, list_stderr or list_stdout
+    listed = [json.loads(line) for line in list_stdout.splitlines() if line.strip()]
+    assert {item["url"] for item in listed} >= set(wget_urls)
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()
+        real_flow_crawl = Crawl.objects.filter(tags_str="real-flow,challenge").values_list("max_depth", "tags_str", "config").first()
+        snapshots = list(Snapshot.objects.order_by("url").values_list("id", "url", "depth", "status", "title"))
+        archive_results = list(
+            ArchiveResult.objects.select_related("snapshot")
+            .order_by("snapshot__url", "plugin")
+            .values_list("snapshot__url", "plugin", "status", "output_files", "output_size", "output_str"),
+        )
+        processes = list(Process.objects.filter(process_type="hook").values_list("process_type", "status", "exit_code", "pwd", "cmd"))
+
+    assert real_flow_crawl is not None
+    assert real_flow_crawl[0] == 0
+    assert real_flow_crawl[1] == "real-flow,challenge"
+    real_flow_config = real_flow_crawl[2] or {}
+    assert real_flow_config["CRAWL_MAX_URLS"] == 2
+    assert real_flow_config["CRAWL_MAX_SIZE"] == 10 * 1024 * 1024
+    assert real_flow_config.get("SNAPSHOT_MAX_SIZE", 0) == 0
+    assert "wget" in real_flow_config["PLUGINS"]
+    assert crawl is not None
+    assert crawl[1] == "chrome-flow"
+    assert "wget,headers,title" in json.dumps(crawl[2] or {})
+
+    snapshot_urls = {url for _id, url, _depth, _status, _title in snapshots}
+    assert snapshot_urls >= {*wget_urls, chrome_url}
+    assert all(depth == 0 for _id, url, depth, _status, _title in snapshots)
+
+    by_url_plugin = {(url, plugin): status for url, plugin, status, _files, _size, _output in archive_results}
+    assert by_url_plugin[("https://example.com", "wget")] == "succeeded"
+    assert by_url_plugin[("https://pirate.github.io/stress-tests/challenge.html", "wget")] == "succeeded"
+    assert by_url_plugin[(chrome_url, "headers")] == "succeeded"
+    assert by_url_plugin[(chrome_url, "title")] == "succeeded"
+    unexpected_results = [
+        (url, plugin, status, output) for url, plugin, status, _files, _size, output in archive_results if status != "succeeded"
+    ]
+    assert not unexpected_results
+
+    snapshot_root = initialized_archive / "archive/users/system/snapshots"
+    html_outputs = [path for path in snapshot_root.rglob("wget/**/*.html") if path.is_file()]
+    header_outputs = [path for path in snapshot_root.rglob("headers/**/headers.json") if path.is_file() and path.stat().st_size > 0]
+    title_outputs = [path for path in snapshot_root.rglob("title/title.txt") if path.is_file() and path.stat().st_size > 0]
+    index_outputs = [path for path in snapshot_root.rglob("index.jsonl") if path.is_file()]
+    assert html_outputs
+    assert header_outputs
+    assert any("example.com" in path.read_text(errors="ignore").lower() for path in header_outputs)
+    assert title_outputs
+    assert any("Example Domain" in path.read_text(errors="ignore") for path in title_outputs)
+    assert len(index_outputs) >= len(wget_urls) + 1
+
+    combined_html = "\n".join(path.read_text(errors="ignore") for path in html_outputs)
+    assert "Example Domain" in combined_html
+    assert "Browser Agent Challenge for AI Browser Drivers" in combined_html
+
+    assert processes
+    assert any("wget" in (pwd or "") or "wget" in (cmd or "") for _type, _status, _exit, pwd, cmd in processes)
+    assert any("headers" in (pwd or "") or "headers" in (cmd or "") for _type, _status, _exit, pwd, cmd in processes)
@@ -3,11 +3,14 @@

 import shutil
 from pathlib import Path
+from urllib.parse import urlsplit

 import pytest
 from abx_plugins import get_plugins_dir
+from django.test import Client

 from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.core.routes_util import build_snapshot_url
 from archivebox.tests.conftest import cli_env, find_snapshot_dir, parse_jsonl_output, run_archivebox_cmd

 from archivebox.tests.test_orm_helpers import use_archivebox_db
@@ -78,6 +81,109 @@ def test_extract_runs_on_existing_snapshots(archive_with_extractors):
    assert archiveresults["wget"].output_files["example.com/index.html"]["size"] == wget_path.stat().st_size


+def test_wget_literal_percent_output_url_serves_captured_file(initialized_archive, snapshot, live_server):
+    """Replay the real HedgeDoc link-interstitial file saved by wget."""
+    import shutil
+    from urllib.parse import quote
+    from playwright.sync_api import expect, sync_playwright
+
+    source_url = "https://docs.sweeting.me/_link?url=http%3A%2F%2Farchive.org%2F"
+    ui_snapshot = snapshot
+    env = cli_env(PLUGINS="wget", SAVE_WGET="True")
+
+    created = run_archivebox_cmd(["snapshot", "create", source_url], cwd=initialized_archive, env=env, check=True)
+    snapshot_id = next(record["id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    extracted = run_archivebox_cmd(
+        ["extract", "--plugins=wget", snapshot_id],
+        cwd=initialized_archive,
+        env=env,
+        timeout=90,
+    )
+    assert extracted.returncode == 0, extracted.stderr or extracted.stdout
+
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(id=snapshot_id)
+        wget = ArchiveResult.objects.get(snapshot=snapshot, plugin="wget")
+        assert wget.status == ArchiveResult.StatusChoices.SUCCEEDED
+        output_path = wget.embed_path()
+        assert output_path and "%3A" in output_path, output_path
+        captured = Path(snapshot.output_dir) / output_path
+        assert captured.is_file()
+        captured_bytes = captured.read_bytes()
+        assert captured_bytes and b"html" in captured_bytes.lower()
+
+        target = urlsplit(build_snapshot_url(snapshot_id, output_path))
+        assert "%253A" in target.path and "%252F" in target.path
+        response = Client().get(target.path, HTTP_HOST=target.netloc)
+        assert response.status_code == 200, (target, response.status_code)
+        assert b"".join(response.streaming_content) == captured_bytes
+        preview = Client().get(f"{target.path}?preview=1", HTTP_HOST=target.netloc)
+        assert preview.status_code == 200
+        card = Client().get(f"/_card/{wget.id}", HTTP_HOST=target.netloc)
+        assert card.status_code == 200
+        assert target.path in card.content.decode()
+
+        captured_wget_dir = Path(ui_snapshot.output_dir) / "wget"
+        shutil.copytree(Path(snapshot.output_dir) / "wget", captured_wget_dir, dirs_exist_ok=True)
+        captured_result = {
+            "hook_name": wget.hook_name,
+            "output_str": wget.output_str,
+            "output_files": wget.output_files,
+            "output_size": wget.output_size,
+        }
+
+    # Exercise the actual captured file through the snapshot detail UI and its
+    # iframe preview. The second hash-driven selection must retain literal `%`
+    # sequences in wget's filesystem filename.
+    from archivebox.machine.models import Machine
+
+    machine = Machine.current()
+    port = urlsplit(live_server.url).port
+    machine.config = {
+        **machine.config,
+        "BASE_URL": f"http://archivebox.localhost:{port}",
+        "SERVER_SECURITY_MODE": "safe-subdomains-fullreplay",
+    }
+    machine.save(update_fields=["config"])
+    ui_snapshot.permissions = "public"
+    ui_snapshot.save(update_fields=["permissions"])
+    ArchiveResult.objects.create(
+        snapshot=ui_snapshot,
+        plugin="wget",
+        status=ArchiveResult.StatusChoices.SUCCEEDED,
+        **captured_result,
+    )
+
+    with sync_playwright() as playwright:
+        browser = playwright.chromium.launch(args=["--host-resolver-rules=MAP *.archivebox.localhost 127.0.0.1"])
+        page = browser.new_page()
+        page.goto(f"http://web.archivebox.localhost:{port}{ui_snapshot.get_absolute_url()}/index.html", wait_until="domcontentloaded")
+        expected_path = quote(str(output_path), safe="/@=")
+        page.evaluate("""() => {
+            window.__previewHashChanges = 0;
+            window.addEventListener('hashchange', () => window.__previewHashChanges++);
+        }""")
+        page.locator(".output-stack-html").click()
+        page.wait_for_function("window.__previewHashChanges > 0")
+
+        link = page.locator('.stack-tray .thumb-card[data-plugin-name="wget"] a[target="preview"]').first
+        assert expected_path in link.get_attribute("href")
+        frame = page.locator("#main-frame")
+        assert expected_path in frame.get_attribute("src")
+        preview = page.frame_locator("#main-frame")
+        # Setting the iframe src does not mean its document has finished loading.
+        expect(preview.locator("body")).to_contain_text("External link")
+        expect(preview.locator("body")).to_contain_text("Continue to external page")
+
+        page.reload(wait_until="domcontentloaded")
+        frame = page.locator("#main-frame")
+        assert expected_path in frame.get_attribute("src")
+        preview = page.frame_locator("#main-frame")
+        expect(preview.locator("body")).to_contain_text("External link")
+        expect(preview.locator("body")).to_contain_text("Continue to external page")
+        browser.close()
+
+
 def test_extract_runs_custom_plugin_discovered_from_data_dir(initialized_archive):
    """Custom plugins discovered by ArchiveBox must execute through abx-dl."""
    custom_plugin_name = "custom_parse_txt_urls"
@@ -254,20 +254,22 @@ def test_init_refuses_database_migrated_by_newer_code(tmp_path):


 def test_init_recovers_from_pre_squash_dev_history(tmp_path):
-    """Pre-squash dev DBs (rows for migrations now absorbed by ``replaces=``)
-    must NOT trip the newer-DB guard — every historical squash would otherwise
-    brick beta-tester collections that pre-date the squash commit."""
+    """Known historical migration rows must not trip the newer-DB guard."""
    result = run_archivebox_cmd(["init"])
    assert result.returncode == 0

-    # Sampling — one name per affected app, all listed in the ``replaces=``
-    # declarations of the current squash anchors. If any of these get treated
-    # as missing-from-code, dev DBs that ran the historical chain pre-squash
-    # would refuse to start.
+    # Sampling across affected apps, including v0.6.2 runtime-generated names
+    # absent from Git. These older collections must still pass init.
    historical_pre_squash_rows = [
        ("api", "0002_alter_apitoken_options"),
        ("api", "0009_rename_created_apitoken_created_at_and_more"),
+        # v0.6.2 ran makemigrations at init, so these reported 0021 names were
+        # created in the installed package and never appeared in Git.
+        ("core", "0021_auto_20220510_0644"),
+        ("core", "0021_auto_20220724_1254"),
+        ("core", "0021_auto_20221128_1116"),
        ("core", "0023_alter_archiveresult_options_archiveresult_abid_and_more"),
+        ("core", "0056_progress_covering_indexes"),
        ("core", "0074_alter_snapshot_downloaded_at"),
        ("core", "0075_crawl"),
        ("core", "0075_archiveresult_retry_at"),
@@ -144,7 +144,7 @@ def test_static_exports_use_filesystem_paths_not_live_django_routes(snapshot):

    assert f"./{static_path}/index.html" in html
    assert f"./{static_path}/screenshot/screenshot.png" in html
-    assert f"./{static_path}/wget/index%253A.html" in html
+    assert f"./{static_path}/index.html#wget/index%253A.html" in html
    assert f"./{static_path}/index.jsonl" in html
    assert f"/snapshot/{snapshot.id.hex}" not in html
    assert "/web/" not in html
@@ -157,6 +157,7 @@ def test_static_exports_use_filesystem_paths_not_live_django_routes(snapshot):
    rendered_detail = detail_html.read_text()
    assert "core/snapshot.html" not in rendered_detail
    assert "screenshot/screenshot.png" in rendered_detail
+    assert 'href="./wget/index%253A.html"' in rendered_detail
    assert "staticfile/prenav.json" not in rendered_detail
    assert "ytdlp/saved.m4a" in rendered_detail
    assert "ytdlp/deleted.temp.m4a" not in rendered_detail
@@ -6,6 +6,38 @@ Tests for archivebox machine command.
 from archivebox.tests.conftest import run_archivebox_cmd


+def test_machine_identity_is_stable_across_cli_processes(initialized_archive):
+    """Minimal containers without /etc/machine-id must still share one identity."""
+    import sqlite3
+
+    import machineid
+
+    from archivebox.config.paths import get_machine_id
+    from archivebox.machine.detect import get_host_guid, get_vm_info
+
+    guid = get_host_guid()
+    assert len(guid) == 64
+    assert all(char in "0123456789abcdef" for char in guid)
+    assert get_machine_id() == guid[:8]
+    assert isinstance(get_vm_info()["hw_uuid"], str)
+    try:
+        native_guid = machineid.hashed_id("archivebox")
+    except machineid.MachineIdNotFound:
+        pass
+    else:
+        assert guid == native_guid
+
+    identities = []
+    for _ in range(2):
+        result = run_archivebox_cmd(["status"], cwd=initialized_archive)
+        assert result.returncode == 0, result.stdout + result.stderr
+        with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+            rows = db.execute("SELECT id, guid FROM machine_machine WHERE guid=?", (guid,)).fetchall()
+        assert len(rows) == 1
+        identities.append(rows)
+    assert identities[0] == identities[1]
+
+
 def test_status_refreshes_week_old_machine(initialized_archive):
    """A saved host refresh must not invalidate the instance still in use."""
    import sqlite3
@@ -50,12 +50,14 @@ def test_persona_import_real_browser_session(initialized_archive, tmp_path, http
    """Import an actual Chromium-created session and archive its private page."""
    import hashlib
    import json
+    import shlex
    import subprocess

    from werkzeug.wrappers import Response
    from archivebox.tests.conftest import resolve_abxpkg_chrome_env

    browser_env = resolve_abxpkg_chrome_env(tmp_path / "lib")
+    browser_env["ABXPKG_LIB_DIR"] = str(tmp_path / "lib")
    source = tmp_path / "browser"
    httpserver.expect_request("/login").respond_with_data(
        '<html><title>Signed in</title><script>localStorage.setItem("persona-preference", "forest-green")</script>Signed in</html>',
@@ -90,6 +92,7 @@ const puppeteer = require(process.argv[1]);
        const page = await browser.newPage();
        await page.goto(process.argv[4]);
        console.log(await page.title());
+        console.log(await page.evaluate(() => localStorage.getItem('persona-preference')));
    } finally { await browser.close(); }
 })().catch(error => { console.error(error); process.exit(1); });
 """,
@@ -104,6 +107,7 @@ const puppeteer = require(process.argv[1]);
    )
    assert seeded.returncode == 0, seeded.stderr
    assert "Signed in" in seeded.stdout
+    assert "forest-green" in seeded.stdout
    preferences = source / "Profile 2" / "Preferences"
    original_preferences = preferences.read_bytes()

@@ -161,6 +165,9 @@ const puppeteer = require(process.argv[1]);
        env={**browser_env, "CHROME_SANDBOX": "false"},
    )
    assert archived.returncode == 0, archived.stderr
+    launch_commands = list((initialized_archive / "archive" / "users").glob("*/crawls/**/chrome/cmd.sh"))
+    assert len(launch_commands) == 1
+    assert shlex.split(launch_commands[0].read_text().splitlines()[1])[0] == browser_env["CHROME_BINARY"]
    outputs = list((initialized_archive / "archive" / "users").glob("*/snapshots/**/dom/output.html"))
    assert len(outputs) == 1
    html = outputs[0].read_text()
@@ -0,0 +1,177 @@
+"""
+Tests for archivebox run CLI command.
+
+Tests cover:
+- run with stdin JSONL (Crawl, Snapshot, ArchiveResult)
+- create-or-update behavior (records with/without id)
+- pass-through output (for chaining)
+"""
+
+import os
+import pty
+import select
+import signal
+import subprocess
+import termios
+import time
+
+import psutil
+import pytest
+
+from archivebox.tests.conftest import (
+    cli_env,
+    create_test_url,
+    parse_jsonl_output,
+    run_archivebox_cmd,
+)
+
+RUN_TEST_ENV = {
+    "PLUGINS": "favicon",
+    "SAVE_FAVICON": "True",
+}
+
+
+def _install_real_chrome_for_test(data_dir, env, *, isolation):
+    env["CHROME_ISOLATION"] = isolation
+    env["CHROME_HEADLESS"] = "true"
+    env["CHROME_SANDBOX"] = "false"
+    install_process = run_archivebox_cmd(
+        ["install", "chrome"],
+        cwd=data_dir,
+        env=env,
+        timeout=600,
+    )
+    assert install_process.returncode == 0, install_process.stderr or install_process.stdout
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize(
+    "choice,target",
+    [(choice, "crawl") for choice in ["skip", "retry", "abort", "ctrl-c", "noninteractive"]]
+    + [("skip", "snapshot"), ("skip", "global"), ("early-skip", "crawl")]
+    + [(choice, "daemon") for choice in ["daemon-tty", "daemon-noninteractive"]]
+    + [("daemon-term", "daemon")],
+)
+def test_run_crawl_interrupts_active_hook(initialized_archive, choice, target):
+    """Foreground run uses the same terminal interrupt choices as add."""
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True", PLUGINS="chrome")
+    # This test starts its 45-second observation at the active navigate hook.
+    # Install Chrome through the real CLI first so a cold binary download is
+    # not mistaken for a hook that never started.
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    snapshot = next(record for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    crawl_id = snapshot["crawl_id"]
+    run_args = {
+        "crawl": [f"--crawl-id={crawl_id}"],
+        "snapshot": [f"--snapshot-id={snapshot['id']}"],
+        "global": [],
+        "daemon": ["--daemon", f"--crawl-id={crawl_id}"],
+    }[target]
+
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+    owned_processes = []
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook():
+        with use_archivebox_db(initialized_archive):
+            return Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+
+    try:
+        result = run_archivebox_cmd(
+            ["run", *run_args],
+            cwd=initialized_archive,
+            env=env,
+            stdin=subprocess.DEVNULL if choice in {"noninteractive", "daemon-noninteractive"} else slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook() is not None)
+        first_hook = active_hook()
+        assert first_hook is not None
+        with use_archivebox_db(initialized_archive):
+            hook_pids = Process.objects.filter(status="running", process_type="hook").values_list("pid", flat=True)
+            for pid in hook_pids:
+                try:
+                    process = psutil.Process(pid)
+                    owned_processes.extend([process, *process.children(recursive=True)])
+                except psutil.NoSuchProcess:
+                    continue
+        result.send_signal(signal.SIGTERM if choice == "daemon-term" else signal.SIGINT)
+        if choice not in {"noninteractive", "daemon-tty", "daemon-noninteractive", "daemon-term"}:
+            read_until(lambda: b"Choice [skip]:" in output, timeout=30)
+            if choice != "early-skip":
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "retry":
+                os.write(master, b"r")
+                read_until(lambda: (next_hook := active_hook()) is not None and next_hook.pid != first_hook.pid)
+                before_prompt = len(output)
+                result.send_signal(signal.SIGINT)
+                read_until(lambda: b"Choice [skip]:" in output[before_prompt:], timeout=30)
+                read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+            if choice == "ctrl-c":
+                result.send_signal(signal.SIGINT)
+            else:
+                os.write(master, b"a" if choice == "abort" else b"\r")
+        read_until(lambda: result.poll() is not None, timeout=45)
+        expected_exit = 143 if choice == "daemon-term" else (0 if choice in {"skip", "early-skip", "retry"} else 130)
+        assert result.returncode == expected_exit, output.decode(errors="replace")
+        assert not psutil.pid_exists(first_hook.pid)
+        _, live_processes = psutil.wait_procs(owned_processes, timeout=5)
+        assert not [process.pid for process in live_processes if process.is_running() and process.status() != psutil.STATUS_ZOMBIE]
+        assert b"Traceback" not in output, output.decode(errors="replace")
+        with use_archivebox_db(initialized_archive):
+            crawl_after = Crawl.objects.get(id=crawl_id)
+            snapshot_after = Snapshot.objects.get(id=snapshot["id"])
+            if choice == "daemon-term":
+                # A supervisor stop displaces this runner; the user's crawl
+                # and snapshot remain resumable for the next owner.
+                assert crawl_after.status == Crawl.StatusChoices.STARTED
+                assert snapshot_after.status == Snapshot.StatusChoices.STARTED
+                assert crawl_after.retry_at is not None and snapshot_after.retry_at is not None
+            else:
+                assert crawl_after.status == Crawl.StatusChoices.SEALED
+                assert snapshot_after.status == Snapshot.StatusChoices.SEALED
+            assert not Process.objects.filter(status="running", pid__isnull=False).exists()
+        if choice in {"noninteractive", "daemon-tty", "daemon-noninteractive", "daemon-term"}:
+            assert b"Choice [skip]:" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            try:
+                result.wait(timeout=20)
+            except subprocess.TimeoutExpired:
+                result.send_signal(signal.SIGKILL)
+                result.wait(timeout=5)
+        for process in owned_processes:
+            try:
+                if process.is_running() and process.status() != psutil.STATUS_ZOMBIE:
+                    process.terminate()
+            except psutil.NoSuchProcess:
+                pass
+        psutil.wait_procs(owned_processes, timeout=3)
+        os.close(slave)
+        os.close(master)
@@ -0,0 +1,489 @@
+"""
+Tests for archivebox run CLI command.
+
+Tests cover:
+- run with stdin JSONL (Crawl, Snapshot, ArchiveResult)
+- create-or-update behavior (records with/without id)
+- pass-through output (for chaining)
+"""
+
+import os
+import pty
+import select
+import signal
+import subprocess
+import termios
+import time
+
+import psutil
+import pytest
+
+from archivebox.tests.conftest import (
+    cleanup_process_group,
+    cli_env,
+    create_test_url,
+    parse_jsonl_output,
+    pid_is_alive,
+    run_archivebox_cmd,
+    wait_for_log,
+    wait_for_pid_to_disappear,
+)
+
+from .test_cli_run_1 import (
+    RUN_TEST_ENV as RUN_TEST_ENV,
+    _install_real_chrome_for_test as _install_real_chrome_for_test,
+)
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("choice", ["typed", "signal"])
+def test_run_third_interrupt_forces_chosen_abort(initialized_archive, choice):
+    """Direct run exits at once when Ctrl+C follows an explicit abort choice."""
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(CHROME_DELAY_AFTER_LOAD="60", CHROME_TIMEOUT="120", CHROME_HEADLESS="True", PLUGINS="chrome")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    crawl_id = next(record["crawl_id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    master, slave = pty.openpty()
+    termios.tcsetwinsize(slave, (40, 160))
+    output = bytearray()
+    result = None
+
+    def read_until(predicate, timeout=45):
+        deadline = time.monotonic() + timeout
+        while time.monotonic() < deadline:
+            if select.select([master], [], [], 0.1)[0]:
+                output.extend(os.read(master, 65536))
+            if predicate():
+                return
+        raise AssertionError(output.decode(errors="replace"))
+
+    def active_hook_pid():
+        with use_archivebox_db(initialized_archive):
+            hook = Process.objects.filter(
+                process_type=Process.TypeChoices.HOOK,
+                cmd__0__endswith="on_Snapshot__30_chrome_navigate.js",
+                status="running",
+            ).first()
+            return hook.pid if hook is not None and hook.is_running else None
+
+    try:
+        result = run_archivebox_cmd(
+            ["run", f"--crawl-id={crawl_id}"],
+            cwd=initialized_archive,
+            env=env,
+            stdin=slave,
+            stdout=slave,
+            stderr=slave,
+            wait=False,
+            start_new_session=True,
+        )
+        read_until(lambda: active_hook_pid() is not None)
+        hook_pid = active_hook_pid()
+        assert hook_pid is not None
+        owned = psutil.Process(result.pid).children(recursive=True)
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Choice [skip]:" in output)
+        read_until(lambda: not termios.tcgetattr(slave)[3] & termios.ICANON)
+        if choice == "typed":
+            os.write(master, b"a")
+        else:
+            result.send_signal(signal.SIGINT)
+        read_until(lambda: b"Aborting crawl" in output, timeout=5)
+        forced_at = time.monotonic()
+        result.send_signal(signal.SIGINT)
+        read_until(lambda: result.poll() is not None, timeout=5)
+        assert time.monotonic() - forced_at < 2.0
+        assert result.returncode == 130, output.decode(errors="replace")
+        assert not pid_is_alive(hook_pid)
+        assert not [child.pid for child in owned if child.is_running() and child.status() != psutil.STATUS_ZOMBIE]
+        assert b"Traceback" not in output
+    finally:
+        if result is not None and result.poll() is None:
+            result.terminate()
+            result.wait(timeout=20)
+        os.close(slave)
+        os.close(master)
+
+
+@pytest.mark.django_db(transaction=True)
+def test_cli_run_retries_snapshot_after_filesystem_failure(initialized_archive):
+    from archivebox.core.models import ArchiveResult, Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="hashes")
+    created = run_archivebox_cmd(["snapshot", "create", create_test_url()], cwd=initialized_archive, env=env)
+    assert created.returncode == 0, created.stdout + created.stderr
+    snapshot_id = next(record["id"] for record in parse_jsonl_output(created.stdout) if record.get("type") == "Snapshot")
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        crawl_id = str(snapshot.crawl_id)
+        directory = snapshot.output_dir
+    directory.mkdir(parents=True, exist_ok=True)
+    (directory / "input.txt").write_text("Capture survives a temporary filesystem failure")
+    broken_directory = directory / "hashes"
+    broken_directory.symlink_to("hashes")
+
+    failed = run_archivebox_cmd(["run", f"--snapshot-id={snapshot_id}"], cwd=initialized_archive, env=env, timeout=60)
+    assert failed.returncode == 1, failed.stdout + failed.stderr
+    assert "FileExistsError: [Errno 17]" in failed.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        assert snapshot.status == Snapshot.StatusChoices.STARTED
+        assert snapshot.retry_at is not None
+        assert not ArchiveResult.objects.filter(snapshot=snapshot).exists()
+
+    broken_directory.unlink()
+    resumed = run_archivebox_cmd(["run", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=60)
+    assert resumed.returncode == 0, resumed.stdout + resumed.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(pk=snapshot_id)
+        assert snapshot.status == Snapshot.StatusChoices.SEALED
+        result = snapshot.archiveresult_set.get(plugin="hashes")
+        assert result.status == ArchiveResult.StatusChoices.SUCCEEDED
+        assert result.output_size > 0
+    assert (directory / "hashes").is_dir()
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.parametrize("precreate_snapshot", [False, True])
+def test_newest_crawl_runs_before_older_queued_and_started_crawls(initialized_archive, precreate_snapshot):
+    from datetime import timedelta
+
+    from django.utils import timezone
+
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="hashes")
+    old_url, queued_url, newest_url = create_test_url(), create_test_url(), create_test_url()
+    old = run_archivebox_cmd(["snapshot", "create", old_url], cwd=initialized_archive, env=env)
+    assert old.returncode == 0, old.stdout + old.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(url=old_url)
+        Crawl.objects.filter(pk=snapshot.crawl_id).update(status=Crawl.StatusChoices.STARTED)
+        Snapshot.objects.filter(pk=snapshot.pk).update(retry_at=timezone.now() - timedelta(days=1))
+    queued = run_archivebox_cmd(
+        ["snapshot" if precreate_snapshot else "crawl", "create", queued_url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert queued.returncode == 0, queued.stdout + queued.stderr
+    newest = run_archivebox_cmd(
+        ["snapshot" if precreate_snapshot else "crawl", "create", newest_url],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert newest.returncode == 0, newest.stdout + newest.stderr
+
+    result = run_archivebox_cmd(["run", "--no-stdin"], cwd=initialized_archive, env=env, timeout=60)
+    assert result.returncode == 0, result.stdout + result.stderr
+    assert all(url in result.stdout for url in (old_url, queued_url, newest_url))
+    assert result.stdout.index(newest_url) < result.stdout.index(queued_url) < result.stdout.index(old_url), result.stdout
+    with use_archivebox_db(initialized_archive):
+        assert (
+            Snapshot.objects.filter(
+                url__in=[old_url, queued_url, newest_url],
+                status=Snapshot.StatusChoices.SEALED,
+            ).count()
+            == 3
+        )
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.timeout(660)
+def test_due_snapshots_share_one_crawl_scoped_chrome(initialized_archive, recursive_test_site):
+    from django.utils import timezone
+
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.machine.models import Process
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+    urls = [recursive_test_site["root_url"], recursive_test_site["child_urls"][0]]
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="\n".join(urls),
+            config={"PLUGINS": "chrome", "CHROME_ISOLATION": "crawl"},
+            created_by_id=get_or_create_system_user_pk(),
+            status=Crawl.StatusChoices.STARTED,
+            retry_at=timezone.now(),
+        )
+        for url in urls:
+            Snapshot.objects.create(url=url, crawl=crawl, status=Snapshot.StatusChoices.QUEUED, retry_at=timezone.now())
+        crawl_id = str(crawl.id)
+
+    result = run_archivebox_cmd(["run", "--no-stdin", f"--crawl-id={crawl_id}"], cwd=initialized_archive, env=env, timeout=600)
+    assert result.returncode == 0, result.stdout + result.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshots = Snapshot.objects.filter(crawl_id=crawl_id, url__in=urls, status=Snapshot.StatusChoices.SEALED)
+        assert snapshots.count() == 2
+        assert all((snapshot.output_dir / "chrome" / "navigation.json").is_file() for snapshot in snapshots)
+        launches = Process.objects.filter(
+            pwd__contains=crawl_id,
+            process_type=Process.TypeChoices.HOOK,
+            cmd__0__endswith="on_CrawlSetup__90_chrome_launch.daemon.bg.js",
+        )
+        assert launches.count() == 1, [(process.pid, process.exit_code) for process in launches]
+        assert launches.get().exit_code == 0
+
+
+@pytest.mark.django_db(transaction=True)
+def test_busy_crawl_does_not_report_snapshot_progress(initialized_archive):
+    from datetime import timedelta
+    from django.utils import timezone
+
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
+    from archivebox.services.runner import _run_due_snapshot_id
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.create(
+            urls="https://example.com/",
+            config={"PLUGINS": "chrome", "CHROME_ISOLATION": "crawl"},
+            created_by_id=get_or_create_system_user_pk(),
+            status=Crawl.StatusChoices.STARTED,
+            retry_at=timezone.now() + timedelta(minutes=1),
+        )
+        snapshot = Snapshot.objects.create(url=crawl.urls, crawl=crawl, retry_at=timezone.now())
+        assert _run_due_snapshot_id(str(snapshot.id), lock_seconds=60, interactive_interrupts=False, runtime_config=None) is False
+
+
+@pytest.mark.django_db(transaction=True)
+@pytest.mark.timeout(660)
+def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archive, recursive_test_site):
+    from archivebox.core.models import Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
+    _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
+
+    result = run_archivebox_cmd(
+        ["snapshot", "create", recursive_test_site["root_url"]],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    records = parse_jsonl_output(result.stdout)
+    snapshot_id = next(record["id"] for record in records if record.get("type") == "Snapshot")
+    with use_archivebox_db(initialized_archive):
+        browser_state = Snapshot.objects.get(id=snapshot_id).output_dir / "chrome" / "browser.json"
+
+    run_log = initialized_archive / "run-signal-chrome.log"
+    run_log_handle = run_log.open("w", encoding="utf-8")
+    run_process = run_archivebox_cmd(
+        ["run", f"--snapshot-id={snapshot_id}"],
+        cwd=initialized_archive,
+        env=env,
+        stdout=run_log_handle,
+        stderr=subprocess.STDOUT,
+        start_new_session=True,
+        wait=False,
+    )
+    run_log_handle.close()
+    try:
+        wait_for_log(browser_state, '"ready": true', timeout=120)
+        child_pids = [child.pid for child in psutil.Process(run_process.pid).children(recursive=True) if pid_is_alive(child.pid)]
+        assert child_pids
+
+        run_process.send_signal(signal.SIGTERM)
+        run_process.wait(timeout=30)
+        output = run_log.read_text(encoding="utf-8", errors="replace")
+        assert "Runner error" not in output
+        for pid in child_pids:
+            wait_for_pid_to_disappear(pid, timeout=15)
+    finally:
+        cleanup_process_group(run_process.pid)
+
+
+class TestRunWithArchiveResult:
+    """Tests for `archivebox run` with ArchiveResult input."""
+
+    @pytest.mark.django_db(transaction=True)
+    def test_run_treats_no_id_archiveresult_as_parent_snapshot_plugin_request(self, initialized_archive):
+        import json
+
+        from archivebox.core.models import ArchiveResult
+        from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+        create_result = run_archivebox_cmd(
+            ["snapshot", "create", create_test_url()],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        snapshot_id = next(record["id"] for record in parse_jsonl_output(create_result.stdout) if record.get("type") == "Snapshot")
+        missing_hook = "on_Snapshot__99_missing_favicon_hook"
+        request = json.dumps(
+            {
+                "type": "ArchiveResult",
+                "snapshot_id": snapshot_id,
+                "plugin": "favicon",
+                "hook_name": missing_hook,
+                "status": "queued",
+            },
+        )
+
+        result = run_archivebox_cmd(
+            ["run"],
+            stdin=f"{request}\n",
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+
+        assert result.returncode == 0, result.stderr or result.stdout
+        with use_archivebox_db(initialized_archive):
+            rows = list(
+                ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin="favicon").values_list(
+                    "hook_name",
+                    "status",
+                    "output_str",
+                ),
+            )
+        assert len(rows) == 1
+        assert rows[0][0] != missing_hook
+        assert rows[0][0].startswith("on_Snapshot__")
+        assert rows[0][1] in ArchiveResult.FINAL_STATES
+
+    def test_run_requeues_failed_archiveresult(self, initialized_archive):
+        """Run uses a failed ArchiveResult as a parent Snapshot/plugin reference."""
+        url = create_test_url()
+
+        # Create snapshot and archive result
+        _cmd_result = run_archivebox_cmd(
+            ["snapshot", "create", url],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout1, _, _ = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+        _cmd_result = run_archivebox_cmd(
+            ["archiveresult", "create", "--plugin=favicon"],
+            stdin=stdout1,
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout2, _, _ = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+        assert any(record.get("type") == "ArchiveResult" for record in parse_jsonl_output(stdout2))
+
+        initial_run = run_archivebox_cmd(
+            ["run"],
+            stdin=stdout2,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert initial_run.returncode == 0, initial_run.stderr
+        persisted_result = run_archivebox_cmd(
+            ["archiveresult", "list", "--plugin=favicon"],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert persisted_result.returncode == 0, persisted_result.stderr
+        assert any(record.get("type") == "ArchiveResult" for record in parse_jsonl_output(persisted_result.stdout))
+
+        # Update to failed
+        update_result = run_archivebox_cmd(
+            ["archiveresult", "update", "--status=failed"],
+            stdin=persisted_result.stdout,
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert update_result.returncode == 0, update_result.stderr
+        failed_result = run_archivebox_cmd(
+            ["archiveresult", "list", "--plugin=favicon"],
+            cwd=initialized_archive,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert failed_result.returncode == 0, failed_result.stderr
+        failed_records = [record for record in parse_jsonl_output(failed_result.stdout) if record.get("type") == "ArchiveResult"]
+        assert len(failed_records) == 1
+        assert failed_records[0]["status"] == "failed"
+        failed_jsonl = next(line for line in failed_result.stdout.splitlines() if failed_records[0]["id"] in line) + "\n"
+
+        # Now run should re-queue it
+        _cmd_result = run_archivebox_cmd(
+            ["run"],
+            stdin=failed_jsonl,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        stdout3, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+
+        assert code == 0
+        records = parse_jsonl_output(stdout3)
+        ar_records = [r for r in records if r.get("type") == "ArchiveResult"]
+        assert len(ar_records) >= 1
+
+
+class TestRunMixedInput:
+    """Tests for `archivebox run` with mixed record types."""
+
+    def test_run_handles_mixed_records_emitted_by_cli(self, initialized_archive):
+        """Run handles real Crawl, Snapshot, and Tag records from CLI stages."""
+        tag_result = run_archivebox_cmd(
+            ["tag", "create", "mixed-run-tag"],
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert tag_result.returncode == 0, tag_result.stderr
+        crawl_result = run_archivebox_cmd(
+            ["crawl", "create", create_test_url()],
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert crawl_result.returncode == 0, crawl_result.stderr
+        snapshot_result = run_archivebox_cmd(
+            ["snapshot", "create"],
+            stdin=crawl_result.stdout,
+            cwd=initialized_archive,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+        assert snapshot_result.returncode == 0, snapshot_result.stderr
+
+        result = run_archivebox_cmd(
+            ["run"],
+            stdin=tag_result.stdout + snapshot_result.stdout,
+            cwd=initialized_archive,
+            timeout=120,
+            env=RUN_TEST_ENV,
+            default_cli_env=True,
+            disable_extractors=True,
+        )
+
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
+
+        types = {record.get("type") for record in records}
+        assert {"Crawl", "Snapshot", "Tag"}.issubset(types)
@@ -30,6 +30,7 @@ from archivebox.tests.conftest import (
    run_archivebox_cmd,
    start_archivebox_server,
    stop_archivebox_process,
+    wait_for_log,
    wait_for_log_count,
    wait_for_pid_to_disappear,
 )
@@ -329,6 +330,50 @@ def test_runner_worker_uses_active_archivebox_module():
    assert 'ARCHIVEBOX_RUNNER_DAEMON="1"' in worker["environment"]


+def test_runner_worker_hook_uses_its_own_python_environment(tmp_path):
+    """A supervisor started in another venv must launch hooks from the runner's venv."""
+    import abx_plugins.plugins.base.utils as current_utils
+
+    from archivebox.workers.supervisord_util import RUNNER_WORKER, _worker_environment_value
+
+    stale_venv = tmp_path / "stale-venv"
+    subprocess.run(["uv", "venv", "--python", sys.executable, str(stale_venv)], check=True, capture_output=True, text=True)
+
+    worker = RUNNER_WORKER()
+    env = os.environ.copy()
+    env["PATH"] = os.pathsep.join((str(stale_venv / "bin"), "/usr/bin", "/bin"))
+    env["VIRTUAL_ENV"] = str(stale_venv)
+    env.pop("PYTHONPATH", None)
+    for key in ("PATH", "VIRTUAL_ENV"):
+        value = _worker_environment_value(worker, key)
+        if value is not None:
+            env[key] = value
+
+    result = subprocess.run(
+        [
+            "/usr/bin/env",
+            "python3",
+            "-c",
+            "import abx_plugins.plugins.base.utils as utils; print(utils.__file__)",
+        ],
+        cwd=tmp_path,
+        env=env,
+        capture_output=True,
+        text=True,
+    )
+    assert result.returncode == 0, result.stderr
+    assert Path(result.stdout.strip()).resolve() == Path(current_utils.__file__).resolve()
+
+
+def test_archivebox_worker_environment_clears_stale_venv_for_global_executable():
+    from archivebox.workers.supervisord_util import _archivebox_worker_environment, _worker_environment_value
+
+    executable = Path("/usr/bin/env")
+    worker = {"environment": _archivebox_worker_environment([str(executable)])}
+    assert _worker_environment_value(worker, "VIRTUAL_ENV") == ""
+    assert _worker_environment_value(worker, "PATH").split(os.pathsep)[0] == str(executable.parent)
+
+
 def test_runtime_binary_wins_over_ambient_system_path(tmp_path):
    env = os.environ.copy()
    env["ABXPKG_LIB_DIR"] = str(tmp_path / "lib")
@@ -357,7 +402,29 @@ def test_daphne_worker_uses_default_application_close_timeout():
    assert "--application-close-timeout=0" not in command


-@pytest.mark.parametrize("in_docker,port", [(False, "5797"), (False, "8000"), (True, "9579")])
+def test_daphne_worker_uses_callers_python_runtime(tmp_path):
+    from archivebox.workers.supervisord_util import SERVER_WORKER, _worker_environment_value
+
+    worker = SERVER_WORKER("127.0.0.1", "5797")
+    command = shlex.split(worker["command"])
+    assert command[:3] == [sys.executable, "-m", "daphne"]
+
+    stale_venv = tmp_path / "stale-venv"
+    subprocess.run(["uv", "venv", "--python", sys.executable, str(stale_venv)], check=True, capture_output=True, text=True)
+    env = os.environ.copy()
+    env["PATH"] = os.pathsep.join((str(stale_venv / "bin"), "/usr/bin", "/bin"))
+    env["VIRTUAL_ENV"] = str(stale_venv)
+    for key in ("PATH", "VIRTUAL_ENV"):
+        value = _worker_environment_value(worker, key)
+        if value is not None:
+            env[key] = value
+
+    result = subprocess.run([*command, "--help"], cwd=tmp_path, env=env, capture_output=True, text=True)
+    assert result.returncode == 0, result.stderr
+    assert "usage:" in result.stdout
+
+
+@pytest.mark.parametrize("in_docker,port", [(False, "5797"), (False, "8000"), (True, "5797"), (True, "9579")])
@pytest.mark.parametrize("debug", [False, True])
 def test_port_compatibility_does_not_add_listeners_outside_docker_defaults(tmp_path, in_docker, port, debug):
    result = subprocess.run(
@@ -381,7 +448,10 @@ def test_port_compatibility_does_not_add_listeners_outside_docker_defaults(tmp_p
        check=True,
    )
    workers = json.loads(result.stdout)
-    assert [worker["name"] for worker in workers] == ["worker_runserver" if debug else "worker_daphne", "worker_runner"]
+    assert [worker["name"] for worker in workers] == [
+        "worker_runserver" if debug else "worker_daphne",
+        "worker_runner",
+    ]
    command = shlex.split(workers[0]["command"])
    assert "--endpoint" not in command
    assert (f"127.0.0.1:{port}" if debug else f"--port={port}") in command
@@ -522,7 +592,7 @@ def test_live_server_machine_search_engine_update_reaches_subsequent_snapshot_ru
                "from archivebox.crawls.models import Crawl;"
                "from archivebox.core.models import Snapshot;"
                "from archivebox.machine.models import Machine;"
-                "machine = Machine.current(refresh=True);"
+                "machine = Machine.current();"
                "machine.config = {**dict(machine.config or {}), 'SEARCH_BACKEND_ENGINE': 'sqlite'};"
                "machine.save(update_fields=['config', 'modified_at']);"
                "crawl = Crawl.objects.create("
@@ -747,26 +817,32 @@ def test_live_server_signal_exit_and_resume_uses_existing_supervisor_state(initi
    try:
        server = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}.log", env=env)
        server_log = server.log_path
+        # HTTP readiness can precede the foreground tail scope. Exercise its
+        # actual signal policy, where interrupting Popen.poll stranded reaping.
+        wait_for_log(server_log, "Tailing worker logs (Ctrl+C to stop)...")

        os.kill(server.pid, stop_signal)
        server.wait(timeout=20)

        if expected_notice:
            log_text = server_log.read_text(encoding="utf-8", errors="replace")
-            assert expected_notice in log_text
-            assert "ArchiveBox server shut down gracefully" in log_text
+            assert server.returncode == 0, log_text
+            assert expected_notice in log_text, log_text
+            assert "ArchiveBox server shut down gracefully" in log_text, log_text
            assert_no_processes_for_data_dir(initialized_archive, timeout=12)

        resumed = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}-resumed.log", env=env)
        resumed_log = resumed.log_path
+        wait_for_log(resumed_log, "Tailing worker logs (Ctrl+C to stop)...")
        result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
        assert result.returncode == 0, result.stderr or result.stdout

        os.kill(resumed.pid, signal.SIGTERM)
        resumed.wait(timeout=20)
        resumed_text = resumed_log.read_text(encoding="utf-8", errors="replace")
-        assert "Got SIGTERM" in resumed_text
-        assert "ArchiveBox server shut down gracefully" in resumed_text
+        assert resumed.returncode == 0, resumed_text
+        assert "Got SIGTERM" in resumed_text, resumed_text
+        assert "ArchiveBox server shut down gracefully" in resumed_text, resumed_text
        assert_no_processes_for_data_dir(initialized_archive, timeout=12)
    finally:
        for proc in (server, resumed):
@@ -19,6 +19,50 @@ from archivebox.tests.test_orm_helpers import use_archivebox_db
 pytestmark = pytest.mark.django_db(transaction=True)


+def test_rescan_recovers_already_migrated_idless_result(initialized_archive, tmp_path):
+    """Repair collections already migrated by versions affected by #1899."""
+    env = cli_env(disable_extractors=True)
+    run_archivebox_cmd(["add", "--index-only", "https://example.com/legacy-result"], env=env, check=True)
+    run_queued_crawls(initialized_archive, env)
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(url="https://example.com/legacy-result")
+        snapshot.write_index_jsonl()
+        output_dir = snapshot.output_dir
+        snapshot_id = snapshot.pk
+    payload = b"<html>Recovered after a previous migration</html>"
+    (output_dir / "singlefile.html").write_bytes(payload)
+    with (output_dir / "index.jsonl").open("a") as index:
+        index.write(
+            json.dumps(
+                {
+                    "type": "ArchiveResult",
+                    "snapshot_id": str(snapshot_id),
+                    "plugin": "singlefile",
+                    "hook_name": "",
+                    "status": "succeeded",
+                    "output_str": "singlefile.html",
+                    "start_ts": "2021-03-07T23:56:16-08:00",
+                },
+            )
+            + "\n",
+        )
+    result_id = None
+    for expected in ("Repaired: 1", "Unchanged: 1"):
+        result = run_archivebox_cmd(["update", "--rescan", "--migrate-only"], env=env, timeout=60)
+        assert result.returncode == 0, result.stdout + result.stderr
+        assert expected in result.stdout
+        with use_archivebox_db(initialized_archive):
+            archived = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="singlefile")
+            assert archived.status == "succeeded"
+            assert archived.output_size == len(payload)
+            assert archived.output_files
+            assert archived.snapshot.output_size == len(payload)
+            if result_id is not None:
+                assert archived.pk == result_id
+            result_id = archived.pk
+        assert (output_dir / "singlefile.html").read_bytes() == payload
+
+
 def test_rescan_imports_current_layout_and_is_idempotent(initialized_archive, tmp_path):
    env = cli_env(disable_extractors=True)
    run_archivebox_cmd(["add", "--index-only", "https://example.com/rescan"], cwd=initialized_archive, env=env, check=True)