ArchiveBox 0.9.36 release merge

Merge latest dev release state into main with stable 0.9.36 metadata.

.github/ISSUE_TEMPLATE/1-bug_report.yml

@@ -30,8 +30,8 @@ body:
       description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
       render: markdown
       placeholder: |
-        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
-        2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
+        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 5797:5797 archivebox/archivebox` in iTerm2
+        2. Went to the https://127.0.0.1:5797/add/ page in Google Chrome
         3. Typed 'https://example.com/xyz' into the 'Add URL' input field
         4. Clicked the 'Add+' button
         5. Got a 500 error and saw the errors below in terminal
@@ -138,7 +138,7 @@ body:
             archivebox:
                 image: archivebox/archivebox:latest
                 ports:
-                    - 8000:8000
+                    - 5797:5797
                 volumes:
                     - ./data:/data
                 environment:

.github/configs/ci-tooling.json

@@ -116,6 +116,10 @@
     }
   ],
   "docker_binaries": [
+    {
+      "name": "{BASH_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
     {
       "name": "{DOCKER_BINARY}",
       "binproviders": "env"

.github/pages/README.md

@@ -0,0 +1,119 @@
+# archivebox.io website
+
+This repository owns its complete website. Build and deploy it without checking
+out another repository's website, fetching shared snippets, or calling a shared
+workflow. Common files are ordinary committed copies; update them deliberately.
+
+## Repository-specific files
+
+- Content: `publicsite/index.html`.
+- Site/gallery generation: `bin/generate_ui_screenshot_gallery.py`.
+- Application capture: `bin/collect_ui_screenshots.sh`.
+- Deployment branch: `dev`.
+
+## Common publication contract
+
+1. Build pull requests without deploying them.
+2. Build website pushes immediately, independently of application capture CI.
+3. Restore validated screenshots from trusted default-branch capture jobs.
+   Keep the original capture revision, timestamp, coverage and image checksums.
+4. Retain `site-screenshots` artifacts for 90 days. After expiry, restore the
+   published image set. Never replace a complete gallery with an empty one when
+   restoration fails. Apple device groups are independently validated artifacts;
+   Android's checked-in bootstrap captures retain their local provenance.
+5. Render current content and layout around those captures. Site revision and
+   capture revision are separate facts; reusing captures must not relabel them.
+6. Serialize production restore/build/deploy under one concurrency group and
+   check out the latest default-branch revision when each build starts. Keep the
+   built revision in provenance; later release-bot commits must not block deploys.
+7. Keep platform-specific build, installation, capture, and completeness checks
+   in the application's existing capture workflow. Site-only edits do not need
+   to rerun those jobs.
+
+## Common presentation contract
+
+The base header, Apps menu and icons, footer, brand styles and navigation behavior
+should remain identical copies across the seven product sites. Put this site's
+name, navigation, CTA and extra footer resources outside the common base files.
+Preserve real links, anchor IDs, no-JavaScript navigation, keyboard operation,
+mobile layout, reduced-motion preferences and existing product-specific content.
+Screenshots stay in English. Translation content, when added, lives in separate
+HTML files; no runtime translation service or cross-repository dependency.
+
+## Editing and building
+
+`base/` contains identical independent copies across the seven product sites.
+Common Python scripts share the same behavior; repository formatters may change
+line wrapping without changing their syntax trees. The base owns the shared logo, Apps menu/icons, footer directory, navigation
+behavior and styles. `site.json` and `nav.html` contain this site's name, routes,
+links and CTA. Product content and additional footer resources stay in the native
+HTML/Jinja/Liquid templates listed above. The `ARCHIVEBOX:*` comments are build
+slots; the renderer fills them with static HTML, never browser-side fetches.
+
+The existing builder calls `site.py render` after generating its content. This
+copies `base/chrome.css` and `base/chrome.js` to `site-base/`, assembles the common
+HTML, validates local resources and writes the site revision to root `build.json`.
+Screenshot manifests are left untouched. No build reads another repository.
+
+Common changes are manual copy/paste edits. Compare `base/` and `site.py` with
+another product repository when updating them; do not add a synchronization job,
+package dependency or reusable cross-repository workflow. `nav.html`, `site.json`
+and the marked workflow build section are intentionally repository-specific.
+
+Build: `uv run --no-project python .github/pages/site.py render _site --source publicsite` after restoring screenshots and refreshing the gallery.
+
+Check the built site with real Chromium (also run by Pages CI):
+
+```sh
+uv run --no-config --no-project --with playwright==1.63.0 playwright install chromium
+uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py SITE_OUTPUT --evidence /tmp/site-evidence
+```
+
+Use this workflow's `SITE_OUTPUT` directory. The check covers desktop/mobile
+layout, keyboard dismissal, no-JavaScript links, local resources and visible
+images, and uploads its screenshots as `site-verification` for 14 days.
+
+## Branding and future translations
+
+`base/chrome.css` owns the shared system font stacks, plum accent and CTA shape,
+including the existing `.button` and `.ab-btn` classes. Product styles own their
+layouts, feature sections and platform-specific button states. Shared system
+fonts need no external font downloads. `site.py` emits the same complete SEO/OG/
+Twitter tags everywhere while retaining each native template's title, description,
+canonical URL, locale and social artwork; edit those values with that page's copy.
+
+For translations, keep complete separate HTML files under `es/`, `fr/` and `zh/`,
+and list their output paths in `site.json` alongside the English pages. Share CSS,
+JS, social artwork and the English screenshot gallery. Translate each page's title
+and description, set its own canonical and `lang` (`zh` for Chinese), and add
+reciprocal `hreflang` links only for translations that actually exist. The renderer
+preserves these tags and computes shared asset paths for nested pages. Localized
+header/footer copy and the small language selector can be added with those pages;
+there is no language redirect, placeholder translation or translation runtime now.
+
+## Screenshot strips
+
+`marquee.py` and `base/marquee.css` / `base/marquee.js` are also identical copies.
+Product homepages choose their placement with an `ARCHIVEBOX:MARQUEE` comment.
+The renderer takes real images, captions, dimensions and links from the generated
+`screenshots/index.html`; there is no second screenshot list or browser fetch.
+Desktop/tablet/mobile galleries contribute their desktop views. Apple uses
+`MARQUEE-CLIENT` and `MARQUEE-SERVER` slots to keep its two products separate.
+Catalog sites without a slot do not load the strip assets.
+
+The strip scrolls once, pauses on hover, focus or manual interaction, and has an
+explicit Play/Pause button. Reduced-motion users start paused. Without JavaScript,
+normal image links and horizontal scrolling remain available. Screenshot capture
+manifests and provenance are never rewritten by this presentation step.
+
+Standalone one- or two-image illustrations remain in their authored source.
+Gallery strips with three or more images use the latest restored captures.
+
+## Keeping the common layout in sync
+
+The comments in `base/header.html`, `base/footer-start.html`,
+`base/footer-end.html`, `base/chrome.css`, and `base/chrome.js` list the peer
+repositories. Update those committed copies together when shared branding,
+Apps links, keyboard behavior, or footer resources change. Preserve each site's
+local navigation and content. The former monorepo `site-chrome` generator is
+retired; builds must use only this repository's files.

.github/pages/artifacts.py

@@ -0,0 +1,102 @@
+"""Common screenshot artifact transport; coverage checks belong to each app."""
+
+import json
+import subprocess
+from pathlib import PurePosixPath
+from urllib.request import urlopen
+
+
+def api(repo, path):
+    return json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/{path}"]))
+
+
+def trusted(run, repo, workflow, branch, successful=True):
+    return (
+        (run.get("head_repository") or {}).get("full_name", "").lower() == repo.lower()
+        and run["head_branch"] == branch
+        and run["event"] in ("push", "workflow_dispatch")
+        and run["path"] == f".github/workflows/{workflow}"
+        and (not successful or run["conclusion"] == "success")
+    )
+
+
+def runs(repo, workflow, branch, successful=True, artifact_names=("site-screenshots",)):
+    # Ask for capture artifacts directly instead of checking every historical
+    # workflow run (most app runs never produce a complete screenshot set).
+    # Artifact creation time also includes fresh captures from old rerun jobs.
+    candidates = []
+    for name in artifact_names:
+        rows = subprocess.check_output(
+            [
+                "gh",
+                "api",
+                f"repos/{repo}/actions/artifacts?name={name}&per_page=100",
+                "--paginate",
+                "--jq",
+                ".artifacts[] | select(.expired == false) | @json",
+            ],
+            text=True,
+        )
+        candidates.extend(json.loads(row) for row in rows.splitlines())
+    seen = set()
+    for artifact in sorted(candidates, key=lambda item: item["created_at"], reverse=True):
+        source = artifact.get("workflow_run") or {}
+        run_id = source.get("id")
+        if (
+            artifact["name"] not in artifact_names
+            or not run_id
+            or run_id in seen
+            or source.get("head_branch") != branch
+            or source.get("head_repository_id") != source.get("repository_id")
+        ):
+            continue
+        seen.add(run_id)
+        run = api(repo, f"actions/runs/{run_id}")
+        if trusted(run, repo, workflow, branch, successful):
+            yield run
+
+
+def names(repo, run):
+    rows = subprocess.check_output(
+        [
+            "gh",
+            "api",
+            f"repos/{repo}/actions/runs/{run['id']}/artifacts?per_page=100",
+            "--paginate",
+            "--jq",
+            ".artifacts[] | @json",
+        ],
+        text=True,
+    )
+    return {artifact["name"] for row in rows.splitlines() if not (artifact := json.loads(row))["expired"]}
+
+
+def download(repo, run, name, destination):
+    subprocess.run(
+        [
+            "gh",
+            "run",
+            "download",
+            str(run["id"]),
+            "--repo",
+            repo,
+            "--name",
+            name,
+            "--dir",
+            str(destination),
+        ],
+        check=True,
+    )
+
+
+def relative_path(name):
+    path = PurePosixPath(name)
+    if not name or path.is_absolute() or ".." in path.parts or any(c in name for c in ":\\?#%"):
+        raise ValueError(f"Unsafe screenshot path: {name}")
+    return path
+
+
+def fetch(base, name):
+    relative_path(name)
+    with urlopen(base + name, timeout=60) as response:
+        return response.read()

.github/pages/base/chrome.css

@@ -0,0 +1,156 @@
+/*
+Shared ArchiveBox website navigation styles and behavior.
+Keep this file in sync with .github/pages/base/chrome.css in:
+ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
+electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
+Each site builds its committed copy independently; there is no central generator.
+Keep site-specific navigation, CTAs, and extra footer links outside this base.
+Also review the inline navbar/footer in abx-dl/website/index.html and
+ debian-archivebox/website/index.html when shared links or branding change.
+*/
+/* Common brand tokens. Native templates own product layout and content. */
+:root {
+  --abx-font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
+  --abx-font-mono: ui-monospace, "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace;
+  --abx-accent: #9b2854;
+  --abx-accent-hover: #782040;
+  --abx-paper: #fcfaf7;
+  --abx-ink: #28252a;
+  --abx-muted: #716c74;
+  --abx-line: #e9e2e3;
+  --font-sans: var(--abx-font-sans);
+  --font-mono: var(--abx-font-mono);
+  --ab-font-sans: var(--abx-font-sans);
+  --ab-font-serif: var(--abx-font-sans);
+  --ab-font-label: var(--abx-font-sans);
+  --ab-font-mono: var(--abx-font-mono);
+  --ab-accent: var(--abx-accent);
+  --ab-accent-hover: var(--abx-accent-hover);
+  --ab-on-accent: #fff;
+}
+body { font-family: var(--abx-font-sans); }
+code, pre, kbd, samp { font-family: var(--abx-font-mono); }
+/* Reuse existing CTA classes; platform selection continues to control colors. */
+.button, .ab-btn {
+  display: inline-flex; align-items: center; justify-content: center; gap: 10px;
+  min-height: 44px; height: auto; max-width: 100%; padding: 10px 20px;
+  border-radius: 24px; font-family: var(--abx-font-sans);
+  font-size: 14px; font-weight: 650; line-height: 1.4;
+  text-align: center; white-space: normal; text-decoration: none;
+}
+.button.primary, .ab-btn--primary { background: var(--abx-accent); border-color: var(--abx-accent); color: #fff; }
+.button.primary:hover, .ab-btn--primary:hover { background: var(--abx-accent-hover); border-color: var(--abx-accent-hover); color: #fff; }
+.button:focus-visible, .ab-btn:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
+.button svg, .button img, .ab-btn svg, .ab-btn img { flex-shrink: 0; }
+@media (prefers-reduced-motion: reduce) { .button, .ab-btn { transition: none; } .button:hover { transform: none; } }
+
+/* Shared ArchiveBox site chrome. Independent copy; edit alongside the other product sites, without a build dependency. */
+/* Native anchor navigation leaves room for the heading and its surrounding content. */
+html { scroll-behavior: smooth; scroll-padding-top: clamp(96px, 25vh, 240px); }
+@media (prefers-reduced-motion: reduce) { html { scroll-behavior: auto; } }
+.abx-header, .abx-footer { box-sizing: border-box; font-family: var(--abx-font-sans); font-size: 14px; line-height: 1.5; letter-spacing: normal; text-align: left; }
+.abx-header *, .abx-footer * { box-sizing: border-box; }
+.abx-header { position: sticky; top: 0; z-index: 100; display: flex; align-items: center; justify-content: space-between; gap: 24px; width: 100%; max-width: none; margin: 0; padding: 15px max(24px, calc((100% - 1280px) / 2)); border: 0; border-bottom: 1px solid #e9e2e3; background: #fcfaf7f5; color: #28252a; backdrop-filter: blur(16px); }
+.abx-header a, .abx-footer a { text-decoration: none; text-underline-offset: .24em; }
+.abx-header .abx-brand, .abx-footer .abx-brand { display: inline-flex; flex: 0 0 auto; align-items: center; gap: 11px; margin: 0; padding: 0; color: inherit; font-size: 19px; font-weight: 750; line-height: 1.2; letter-spacing: -.5px; white-space: nowrap; }
+.abx-brand .abx-logo { display: block; flex: 0 0 auto; width: 32px; height: 32px; margin: 0; }
+.abx-header .abx-subsite { transform: translateY(3px); margin-left: 2px; color: #716c74; font-size: 12px; font-weight: 450; letter-spacing: 0; }
+.abx-header .abx-nav { display: flex; align-items: center; justify-content: flex-end; flex-wrap: wrap; gap: 0; width: auto; max-width: none; margin: 0; padding: 0; font-size: 13px; font-weight: 600; }
+.abx-nav > a, .abx-nav > details { margin: 0 0 0 15px; padding: 0 0 0 15px; border: 0; border-left: 1px solid #ded4d8; border-radius: 0; background: none; }
+.abx-nav > :first-child { margin-left: 0; padding-left: 0; border-left: 0; }
+.abx-header .abx-nav a { display: inline-flex; align-items: center; min-height: 36px; color: #4d424b; white-space: nowrap; }
+.abx-header .abx-nav a:hover { color: #9b2854; }
+.abx-header .abx-nav .abx-cta { margin-left: 18px; padding: 9px 17px; border: 1px solid #9b2854; border-radius: 24px; background: #9b2854; color: #fff; }
+.abx-header .abx-nav .abx-cta:hover { background: #782040; color: #fff; }
+.abx-header .abx-apps { position: relative; }
+.abx-apps > summary { display: flex; align-items: center; gap: 7px; min-height: 36px; list-style: none; cursor: pointer; color: #4d424b; }
+.abx-apps > summary::-webkit-details-marker { display: none; }
+.abx-apps > summary::before { content: '›'; display: inline-block; color: #9b2854; font-size: 22px; line-height: 1; transition: transform .15s; }
+.abx-apps[open] > summary::before { transform: rotate(90deg); }
+.abx-header .abx-app-links { position: absolute; right: 0; top: calc(100% + 12px); width: 440px; max-width: calc(100vw - 24px); max-height: min(70vh, 580px); overflow-y: auto; padding: 8px; border: 1px solid #e9e2e3; border-radius: 16px; background: #fcfaf7; box-shadow: 0 18px 48px #2d152522, 0 3px 8px #2d152508; }
+.abx-header .abx-app-row { display: flex; align-items: center; gap: 8px; min-height: 44px; padding: 4px 6px; border-radius: 10px; transition: background .15s; }
+.abx-header .abx-app-row:hover, .abx-header .abx-app-row:focus-within { background: #f2e8ed; }
+.abx-header .abx-app-links .abx-app-name { flex: 1; min-width: 0; min-height: 34px; padding: 5px 4px; border-radius: 6px; font-size: 13px; line-height: 1.35; white-space: normal; }
+.abx-header .abx-platforms { display: flex; flex: 0 0 auto; align-items: center; gap: 5px; }
+.abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 auto; height: 30px; min-height: 30px; padding: 5px; border: 1px solid #e8dfe3; border-radius: 9px; background: #fffdfb; color: #4d424b; box-shadow: 0 2px 3px #2d152510, inset 0 1px 0 #fff; transition: transform .15s, box-shadow .15s, border-color .15s; }
+.abx-header .abx-app-links .abx-platform { width: 30px; }
+.abx-header .abx-platform svg { display: block; width: 18px; height: 18px; margin: 0; fill: currentColor; }
+.abx-header .abx-app-links .abx-platform-docker { color: #2386ce; }
+.abx-header .abx-app-links .abx-platform-windows, .abx-header .abx-app-links .abx-platform-app-store, .abx-header .abx-app-links .abx-platform-safari { color: #1879c4; }
+.abx-header .abx-app-links .abx-platform-android { color: #29965d; }
+.abx-header .abx-app-links .abx-platform-firefox { color: #e46b24; }
+.abx-header .abx-app-links .abx-platform-brave { color: #e8552b; }
+.abx-header .abx-app-links .abx-platform-edge { color: #168b95; }
+.abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-platform:focus-visible, .abx-header .abx-app-links .abx-tool:hover, .abx-header .abx-app-links .abx-tool:focus-visible { transform: translateY(-2px); border-color: #c99bb0; background: #fff; box-shadow: 0 5px 10px #2d15251c; }
+.abx-header .abx-app-links .abx-tool { padding: 5px 8px; font-size: 11px; white-space: nowrap; }
+.abx-header .abx-tools-row { margin-top: 5px; padding-top: 9px; border-top: 1px solid #e9e2e3; border-radius: 0 0 10px 10px; }
+.abx-header .abx-tools-row .abx-app-name { flex: 0 0 auto; }
+.abx-header .abx-tools-row .abx-platforms { flex: 0 1 auto; flex-wrap: wrap; justify-content: flex-end; margin-left: auto; }
+.abx-header .abx-app-links code { display: inline; vertical-align: baseline; font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 1em; font-weight: inherit; line-height: inherit; margin: 0; padding: 0; border: 0; background: transparent; color: inherit; }
+.abx-header .abx-icon-stack { isolation: isolate; gap: 0; padding: 4px 5px; }
+.abx-header .abx-icon-stack .abx-integration { position: relative; z-index: 1; transform: translateY(var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); transition: transform .18s ease, box-shadow .18s ease, border-color .18s ease; }
+.abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -10px; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(2) { --stack-angle: 9deg; --stack-y: -2px; z-index: 2; color: #569a31; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(3) { --stack-angle: -6deg; --stack-y: 2px; z-index: 3; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(4) { --stack-angle: 7deg; --stack-y: -1px; z-index: 4; color: #f00; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(5) { --stack-angle: -10deg; --stack-y: 1px; z-index: 5; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(6) { --stack-angle: 6deg; --stack-y: -2px; z-index: 6; }
+.abx-header .abx-icon-stack .abx-integration:nth-child(7) { --stack-angle: -5deg; z-index: 7; }
+.abx-header .abx-icon-stack .abx-integration:not(:hover, :focus-visible):has(~ .abx-integration:is(:hover, :focus-visible)) { transform: translate(-6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
+.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) ~ .abx-integration:not(:hover, :focus-visible) { transform: translate(6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
+.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { z-index: 20; transform: translateY(-7px) rotate(0deg) scale(1.12); border-color: #c99bb0; box-shadow: 0 8px 14px #2d15252b; }
+@media (hover: none) { .abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -6px; } }
+@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration { transition: none; } }
+.abx-footer { position: relative; z-index: 1; display: block; width: 100%; max-width: none; margin: 72px 0 0; padding: 60px max(24px, calc((100% - 1280px) / 2)) 24px; border: 0; border-top: 3px solid #7e284a; background: radial-gradient(ellipse at 12% 0, #54243b55, transparent 48%), linear-gradient(120deg, #201923, #14151b 75%); color: #ddd2dd; color-scheme: dark; }
+.abx-footer .abx-footer-intro { display: flex; align-items: flex-start; justify-content: space-between; gap: 32px; margin: 0 0 42px; }
+.abx-footer .abx-brand { color: #fff4ed; font-size: 24px; }
+.abx-footer .abx-brand .abx-logo { width: 40px; height: 40px; }
+.abx-footer .abx-tagline { max-width: 430px; margin: 12px 0 0; color: #b8acbb; font-size: 14px; line-height: 1.7; }
+.abx-footer .abx-footer-motto { margin: 2px 0 0; color: #d8b39f; font: italic 25px/1.45 Georgia, serif; text-align: right; }
+.abx-footer .abx-footer-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 42px; margin: 0; }
+.abx-footer .abx-footer-column { margin: 0; padding: 20px 0 0; border: 0; border-top: 1px solid #67505e66; background: none; }
+.abx-footer .abx-footer-column h2 { margin: 0 0 19px; padding: 0; border: 0; color: #e4acbb; font: 650 12px/1.4 var(--abx-font-sans); letter-spacing: .12em; text-transform: uppercase; }
+.abx-footer .abx-footer-column ul { display: block; list-style: none; margin: 0; padding: 0; }
+.abx-footer .abx-footer-column li { display: flex; align-items: center; gap: 10px; margin: 0; padding: 0; min-height: 36px; }
+.abx-footer .abx-footer-column li::before { content: none; }
+.abx-footer .abx-footer-column a { color: #ddd2dd; font-size: 13px; line-height: 1.55; }
+.abx-footer .abx-footer-column a:first-child { padding: 6px 0; }
+.abx-footer a:hover { color: #fff; text-decoration: underline; }
+.abx-footer .abx-source { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 28px; width: 28px; height: 28px; border: 1px solid transparent; border-radius: 50%; color: #a58d9f; }
+.abx-footer .abx-source:hover { color: #f3b5cb; border-color: #80516a; background: #ffffff08; }
+.abx-source svg { width: 15px; height: 15px; fill: currentColor; }
+.abx-footer .abx-footer-bottom { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 12px 24px; margin-top: 32px; padding-top: 21px; border-top: 1px solid #67505e66; color: #a99bab; font-size: 12px; }
+.abx-footer .abx-footer-bottom p { margin: 0; font-size: inherit; color: inherit; }
+.abx-footer .abx-footer-bottom a { color: #cebbc9; }
+.abx-footer .abx-footer-local { margin: 24px 0 0; padding: 16px 0 0; border: 0; border-top: 1px solid #67505e66; border-radius: 0; background: none; color: #baacbc; font-size: 12px; }
+.abx-footer-local > summary { cursor: pointer; color: #d6bfce; }
+.abx-footer-local > div { padding-top: 18px; }
+.abx-footer-local a { color: #e5aabd; }
+.abx-footer-local .brand, .abx-footer-local .footer-intro, .abx-footer-local .footer-grid, .abx-footer-local .footer-meta { color: inherit; }
+.abx-footer-local .footer-card { background: transparent; border-color: #67505e66; }
+.abx-footer-local h2, .abx-footer-local h3, .abx-footer-local strong { color: #e4d8e3; }
+.abx-footer-local .footer-link-copy span, .abx-footer-local .footer-card-subtitle { color: #baacbc; }
+.abx-header a:focus-visible, .abx-header summary:focus-visible, .abx-footer a:focus-visible, .abx-footer summary:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
+@media (max-width: 1100px) {
+  .abx-header { flex-wrap: wrap; gap: 8px 24px; padding: 12px 20px; }
+  .abx-header .abx-nav { justify-content: flex-start; }
+  .abx-header .abx-app-links { position: fixed; top: auto; left: 20px; right: auto; width: 440px; margin-top: 10px; }
+}
+@media (max-width: 700px) {
+  .abx-header { position: relative; padding: 14px 18px; }
+  .abx-header .abx-brand { flex-wrap: wrap; row-gap: 5px; }
+  .abx-header .abx-nav { width: 100%; gap: 4px 0; font-size: 12px; }
+  .abx-nav > a, .abx-nav > details { margin-left: 10px; padding-left: 10px; }
+  .abx-header .abx-nav .abx-cta { margin-left: 10px; padding: 7px 13px; }
+  .abx-header .abx-app-links { left: 12px; right: 12px; width: auto; max-height: 60vh; }
+  .abx-footer { padding: 40px 22px 22px; margin-top: 48px; }
+  .abx-footer .abx-footer-intro { display: block; margin-bottom: 30px; }
+  .abx-footer .abx-footer-motto { margin-top: 20px; text-align: left; font-size: 22px; }
+  .abx-footer .abx-footer-grid { grid-template-columns: 1fr; gap: 26px; }
+  .abx-footer .abx-footer-column li { min-height: 40px; }
+}
+@media (prefers-reduced-motion: reduce) { .abx-apps > summary::before, .abx-header .abx-app-row, .abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { transition: none; } .abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-tool:hover { transform: none; } }
+
+.abx-footer-local > div > a + a { margin-left: 12px; }
+
+@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { transform: none; } }

.github/pages/base/chrome.js

@@ -0,0 +1,28 @@
+/*
+Shared ArchiveBox website navigation styles and behavior.
+Keep this file in sync with .github/pages/base/chrome.js in:
+ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
+electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
+Each site builds its committed copy independently; there is no central generator.
+Keep site-specific navigation, CTAs, and extra footer links outside this base.
+Also review the inline navbar/footer in abx-dl/website/index.html and
+ debian-archivebox/website/index.html when shared links or branding change.
+*/
+(() => {
+  const menu = document.querySelector('.abx-apps');
+  document.addEventListener('keydown', event => {
+    if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
+  });
+  document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
+  function revealAnchor() {
+    let target;
+    try { target = document.getElementById(decodeURIComponent(window.location.hash.slice(1))); } catch { return; }
+    if (!target) return;
+    let details = target.closest('details');
+    let opened = false;
+    while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
+    if (opened) window.requestAnimationFrame(() => target.scrollIntoView());
+  }
+  window.addEventListener('hashchange', revealAnchor);
+  revealAnchor();
+})();

.github/pages/base/footer-end.html

@@ -0,0 +1,12 @@
+<!--
+Shared ArchiveBox website footer.
+Keep this file in sync with .github/pages/base/footer-end.html in:
+ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
+electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
+Each site builds its committed copy independently; there is no central generator.
+Keep site-specific navigation, CTAs, and extra footer links outside this base.
+Also review the inline navbar/footer in abx-dl/website/index.html and
+ debian-archivebox/website/index.html when shared links or branding change.
+-->
+  <div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
+</footer>

.github/pages/base/footer-start.html

@@ -0,0 +1,13 @@
+<!--
+Shared ArchiveBox website footer.
+Keep this file in sync with .github/pages/base/footer-start.html in:
+ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
+electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
+Each site builds its committed copy independently; there is no central generator.
+Keep site-specific navigation, CTAs, and extra footer links outside this base.
+Also review the inline navbar/footer in abx-dl/website/index.html and
+ debian-archivebox/website/index.html when shared links or branding change.
+-->
+<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
+  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated

.github/pages/base/header.html

@@ -0,0 +1,24 @@
+<!--
+Shared ArchiveBox website navbar and Apps menu.
+Keep this file in sync with .github/pages/base/header.html in:
+ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
+electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
+Each site builds its committed copy independently; there is no central generator.
+Keep site-specific navigation, CTAs, and extra footer links outside this base.
+Also review the inline navbar/footer in abx-dl/website/index.html and
+ debian-archivebox/website/index.html when shared links or branding change.
+-->
+<header class="abx-header">
+  <a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">__SITE_NAME__</span></a>
+  <nav class="abx-nav" aria-label="Main navigation">
+    <details class="abx-apps"><summary>Apps</summary><div class="abx-app-links">
+      <div class="abx-app-row"><a class="abx-app-name" href="https://archivebox.io/#install-compose">ArchiveBox Server</a><div class="abx-platforms"><a class="abx-platform abx-platform-docker" href="https://archivebox.io/#install-compose" aria-label="ArchiveBox Server for Docker" title="ArchiveBox Server for Docker"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M349.9 236.3l-66.1 0 0-59.4 66.1 0 0 59.4zm0-204.3l-66.1 0 0 60.7 66.1 0 0-60.7zm78.2 144.8l-66.1 0 0 59.4 66.1 0 0-59.4zM271.8 104.7l-66.1 0 0 60.1 66.1 0 0-60.1zm78.1 0l-66.1 0 0 60.1 66.1 0 0-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7l-434.7 0c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zM115.6 176.8l-66 0 0 59.4 66.1 0 0-59.4-.1 0zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm-78.1-72.1l-66.1 0 0 60.1 66.1 0 0-60.1z"/></svg></a><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/#archivebox-serverapp" aria-label="ArchiveBox Server for macOS" title="ArchiveBox Server for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://electron.archivebox.io/" aria-label="ArchiveBox Server for Windows" title="ArchiveBox Server for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://archivebox.io/#install-apt" aria-label="ArchiveBox Server for Linux" title="ArchiveBox Server for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-1Line truncated
+      <div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Desktop</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Desktop for macOS" title="ArchiveBox Desktop for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://windows.archivebox.io/" aria-label="ArchiveBox Desktop for Windows" title="ArchiveBox Desktop for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://linux.archivebox.io/" aria-label="ArchiveBox Desktop for Linux" title="ArchiveBox Desktop for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-10 14.1-10.5 4.7-4.4 13.5-14.2 27.9-14.2 7.1 0 15.6 2.3 25.9 8.9 6.3 4.1 11.3 4.4 22.6 9.3 8.4 3.5 13.7 9.7 10.5 18.2-2.6 7.1-11 14.4-22.7 18.1-11.1 3.6-19.8 16-38.2 14.9-3.9-.2-7-1-9.6-2.1-8-3.5-12.2-10.4-20-15-8.6-4.8-13.2-10.4-14.7-15.3-1.4-4.9 0-9 4.2-12.3zm3.3 334c-2.7 35.1-43.9 34.4-75.3 18-29.9-15.8-68.6-6.5-76.5-21.9-2.4-4.7-2.4-12.7 2.6-26.4l0-.2c2.4-7.6 .6-16-.6-23.9-1.2-7.8-1.8-15 .9-20 3.5-6.7 8.5-9.1 14.8-11.3 10.3-3.7 11.8-3.4 19.6-9.9 5.5-5.7 9.5-12.9 14.3-18 5.1-5.5 10-8.1 17.7-6.9 8.1 1.2 15.1 6.8 21.9 16l19.6 35.6c9.5 19.9 43.1 48.4 41 68.9zm-1.4-25.9c-4.1-6.6-9.6-13.6-14.4-19.6 7.1 0 14.2-2.2 16.7-8.9 2.3-6.2 0-14.9-7.4-24.9-13.5-18.2-38.3-32.5-38.3-32.5-13.5-8.4-21.1-18.7-24.6-29.9s-3-23.3-.3-35.2c5.2-22.9 18.6-45.2 27.2-59.2 2.3-1.7 .8 3.2-8.7 20.8-8.5 16.1-24.4 53.3-2.6 82.4 .6-20.7 5.5-41.8 13.8-61.5 12-27.4 37.3-74.9 39.3-112.7 1.1 .8 4.6 3.2 6.2 4.1 4.6 2.7 8.1 6.7 12.6 10.3 12.4 10 28.5 9.2 42.4 1.2 6.2-3.5 11.2-7.5 15.9-9 9.9-3.1 17.8-8.6 22.3-15 7.7 30.4 25.7 74.3 37.2 95.7 6.1 11.4 18.3 35.5 23.6 64.6 3.3-.1 7 .4 10.9 1.4 13.8-35.7-11.7-74.2-23.3-84.9-4.7-4.6-4.9-6.6-2.6-6.5 12.6 11.2 29.2 33.7 35.2 59 2.8 11.6 3.3 23.7 .4 35.7 16.4 6.8 35.9 17.9 30.7 34.8-2.2-.1-3.2 0-4Line truncated
+      <div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Mobile</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Mobile for iOS" title="ArchiveBox Mobile for iOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-android" href="https://android.archivebox.io/" aria-label="ArchiveBox Mobile for Android" title="ArchiveBox Mobile for Android"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!-- Font Awesome Free 5.0.13 by Fonticons, Inc. https://fontawesome.com License: https://creativecommons.org/licenses/by/4.0/ --><path d="M89.6 204.5v115.8c0 15.4-12.1 27.7-27.5 27.7-15.3 0-30.1-12.4-30.1-27.7V204.5c0-15.1 14.8-27.5 30.1-27.5 15.1 0 27.5 12.4 27.5 27.5zm10.8 157c0 16.4 13.2 29.6 29.6 29.6h19.9l.3 61.1c0 36.9 55.2 36.6 55.2 0v-61.1h37.2v61.1c0 36.7 55.5 36.8 55.5 0v-61.1h20.2c16.2 0 29.4-13.2 29.4-29.6V182.1H100.4v179.4zm248-189.1H99.3c0-42.8 25.6-80 63.6-99.4l-19.1-35.3c-2.8-4.9 4.3-8 6.7-3.8l19.4 35.6c34.9-15.5 75-14.7 108.3 0L297.5 34c2.5-4.3 9.5-1.1 6.7 3.8L285.1 73c37.7 19.4 63.3 56.6 63.3 99.4zm-170.7-55.5c0-5.7-4.6-10.5-10.5-10.5-5.7 0-10.2 4.8-10.2 10.5s4.6 10.5 10.2 10.5c5.9 0 10.5-4.8 10.5-10.5zm113.4 0c0-5.7-4.6-10.5-10.2-10.5-5.9 0-10.5 4.8-10.5 10.5s4.6 10.5 10.5 10.5c5.6 0 10.2-4.8 10.2-10.5zm94.8 60.1c-15.1 0-27.5 12.1-27.5 27.5v115.8c0 15.4 12.4 27.7 27.5 27.7 15.4 0 30.1-12.4 30.1-27.7V204.5c0-15.4-14.8-27.5-30.1-27.5z"/></svg></a></div></div>
+      <div class="abx-app-row"><a class="abx-app-name" href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><div class="abx-platforms"><a class="abx-platform abx-platform-chrome" href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj" aria-label="ArchiveBox Browser Extension for Chrome" title="ArchiveBox Browser Extension for Chrome"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2024 Fonticons, Inc. --><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg></a><a class="abx-platform abx-platform-safari" href="https://extension.archivebox.io/" aria-label="ArchiveBox Browser Extension for Safari" title="ArchiveBox Browser Extension for Safari"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M274.7 274.7L237.3 237.3 166 346 274.7 274.7zM256 8a248 248 0 1 0 0 496 248 248 0 1 0 0-496zM411.8 182.8l14.8-6.1c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.8 1 2.6 1.7 1.3 1.6 1.7 2.6 .6 2 .6 3.1-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.3-2.6 1.7L418 197.6c-2 .8-4.2 .8-6.1 0s-3.5-2.4-4.3-4.3-.8-4.2 0-6.1 2.4-3.5 4.3-4.3zM314.4 94l6.1-14.8c.8-2 2.4-3.5 4.3-4.3s4.2-.8 6.1 0 3.5 2.4 4.3 4.3 .8 4.2 0 6.1l-6.1 14.8c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0c-1-.4-1.9-1-2.6-1.7s-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1zM256 60c2.1 0 4.2 .8 5.7 2.3S264 65.9 264 68l0 16c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7S253.9 60 256 60zM181 74.9c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.9 1 2.6 1.7 1.3 1.6 1.7 2.6L197.6 94c.4 1 .7 2 .7 3.1s-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.4-2.6 1.8-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.8-1.3-1.7-1.7-2.7l-6.1-14.8c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3zm-63.6 42.5c1.5-1.5 3.5-2.3 5.7-2.3s4.2 .8 5.7 2.3L140 128.7c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3l-11.3-11.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6l0 0zM60 256c0-2.1 .8-4.2 2.3-5.7S65.9 248 68 248l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3l-16 0c-2.1 0-4.2-.8-5.7-2.3S60 258.1 60 256zm40.2 73.2l-14.8 6.1c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1 1-1.9 1.7-2.6 1.6-1.3 2.6-1.7L94 314.4c2-.8 4.2-.8 6.1 0s3.5 2.4 4.3 4.3 .8 4.2 0 6.1-2.4 3.5-4.3 4.3zm4.3-136c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1 2.4-3.5 4.3-4.3 4.2-.8 6.1 0l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1l0 0zM197.6 418l-6.1 14.8c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1l6.1-14.8c.4-1 1-1.9 1.7-2.7s1.6-1.4 2.6-1.8 2-.6 3.1-.6 2.1 .2 3.1 .6 1.9 1 2.6 1.8 1.3 1.7 1.7 2.6 .6 2 .6 3.1-.2 2.1-.7 3.1zM264 444c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7s3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3 2.3 3.5 2.3 5.7l0 16zm67-6.9c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6L314.4 418c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3 4.2-.8 6.1 0 3.5 2.4 4.3 4.3l6.1 14.8c.8 2 .8 4.2 0 6.1s-2.4 3.5-4.3 4.3zm63.6-42.5c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3L372 383.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6 3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3l11.3 11.3c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6l0 0zM286.2 286.2L110.3 401.7 225.8 225.8 401.7 110.3 286.2 286.2zM437.1 331c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1c.4-1 1-1.9 1.7-2.6s1.6-1.3 2.6-1.7 2-.6 3.1-.6 2.1 .2 3.1 .6l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1zm6.9-67l-16 0c-2.1 0-4.2-.8-5.7-2.3s-2.3-3.5-2.3-5.7 .8-4.2 2.3-5.7 3.5-2.3 5.7-2.3l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3z"/></svg></a><a class="abx-platform abx-platform-firefox" href="https://addons.mozilla.org/firefox/addon/archivebox-exporter/" aria-label="ArchiveBox Browser Extension for Firefox" title="ArchiveBox Browser Extension for Firefox"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512Line truncated
+      <div class="abx-app-row abx-plugin-row"><a class="abx-app-name" href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><div class="abx-platforms abx-icon-stack"><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Google Drive storage" title="Google Drive storage"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m38.7 419.3 22.6 39c4.7 8.2 11.4 14.7 19.4 19.4l80.6-139.6H0c0 9.1 2.3 18.2 7 26.4z" style="fill:#0066da"/><path d="M256 173.9 175.4 34.3c-7.9 4.7-14.7 11.1-19.4 19.4L7 311.7c-4.6 8-7 17.1-7 26.4h161.3z" style="fill:#00ac47"/><path d="M431.4 477.7c7.9-4.7 14.7-11.1 19.4-19.4l9.4-16.1 44.9-77.7c4.7-8.2 7-17.3 7-26.4H350.7l34.3 67.4z" style="fill:#ea4335"/><path d="m256 173.9 80.6-139.6c-7.9-4.7-17-7-26.4-7H201.8c-9.4 0-18.5 2.6-26.4 7z" style="fill:#00832d"/><path d="M350.7 338.1H161.3L80.6 477.7c7.9 4.7 17 7 26.4 7h298c9.4 0 18.5-2.6 26.4-7z" style="fill:#2684fc"/><path d="M430.5 182.7 356 53.7c-4.7-8.2-11.4-14.7-19.4-19.4L256 173.9l94.7 164.2h161c0-9.1-2.3-18.2-7-26.4z" style="fill:#ffba00"/></svg></a><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Amazon S3 storage" title="Amazon S3 storage"><svg aria-hidden="true" focusable="false" fill="#569A31" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M20.913 13.147l.12-.895c.947.576 1.258.922 1.354 1.071-.16.031-.562.046-1.474-.176zm-2.174 7.988a.547.547 0 0 0-.005.073c0 .084-.207.405-1.124.768a10.28 10.28 0 0 1-1.438.432c-1.405.325-3.128.504-4.853.504-4.612 0-7.412-1.184-7.412-1.704a.547.547 0 0 0-.005-.073L1.81 5.602c.135.078.28.154.432.227.042.02.086.038.128.057.134.062.272.122.417.18l.179.069c.154.058.314.114.478.168.043.013.084.029.13.043.207.065.423.127.646.187l.176.044c.175.044.353.087.534.127a23.414 23.414 0 0 0 .843.17l.121.023c.252.045.508.085.768.122.071.011.144.02.216.03.2.027.4.053.604.077l.24.027c.245.026.49.05.74.07l.081.009c.275.022.552.04.83.056l.233.012c.21.01.422.018.633.025a33.088 33.088 0 0 0 2.795-.026l.232-.011c.278-.016.555-.034.83-.056l.08-.008c.25-.02.497-.045.742-.072l.238-.026c.205-.024.408-.05.609-.077.07-.01.141-.019.211-.03.261-.037.519-.078.772-.122l.111-.02c.215-.04.427-.082.634-.125l.212-.047c.186-.041.368-.085.546-.13l.166-.042c.225-.06.444-.122.654-.189.04-.012.077-.026.115-.038a10.6 10.6 0 0 0 .493-.173c.058-.021.114-.044.17-.066.15-.06.293-.12.43-.185.038-.017.079-.034.116-.052.153-.073.3-.15.436-.228l-.976 7.245c-2.488-.78-5.805-2.292-7.311-3a1.09 1.09 0 0 0-1.088-1.085c-.6 0-1.088.489-1.088 1.088 0 .6.488 1.089 1.088 1.089.196 0 .378-.056.537-.148 1.72.812 5.144 2.367 7.715 3.15zm-7.42-20.047c5.677 0 9.676 1.759 9.75 2.736l-.014.113c-.01.033-.031.067-.048.101-.015.028-.026.057-.047.087-.024.033-.058.068-.09.102-.028.03-.051.06-.084.09-.038.035-.087.07-.133.105-.04.03-.074.06-.119.091-.053.036-.116.071-.177.107-.05.03-.095.06-.15.09-.068.036-.147.073-.222.11-.059.028-.114.057-.177.085-.084.038-.177.074-.268.111-.068.027-.13.054-.203.082-.097.036-.205.072-.31.107-.075.026-.148.053-.228.079-.111.035-.233.069-.35.103-.085.024-.165.05-.253.073-.124.034-.258.065-.389.098-.093.022-.181.046-.278.068-.139.032-.287.061-.433.091-.098.02-.191.041-.293.06-.155.03-.32.057-.482.084-.1.018-.198.036-.302.052-.166.026-.342.048-.515.072-.11.014-.213.03-.325.044-.181.023-.372.041-.56.06-.11.012-.218.025-.332.036-.188.016-.386.029-.58.043-.122.009-.24.02-.364.028-.207.012-.422.02-.635.028-.12.005-.234.012-.354.016a35.605 35.605 0 0 1-2.069 0c-.12-.004-.234-.011-.352-.016-.214-.008-.43-.016-.637-.028-.122-.008-.238-.02-.36-.027-.195-.015-.394-.028-.584-.044-.11-.01-.215-.024-.324-.035-.19-.02-.384-.038-.568-.06l-.315-.044c-.176-.024-.355-.046-.525-.073-.1-.015-.192-.033-.29-.05-.167-.028-.335-.055-.494-.086-.096-.018-.183-.038-.276-.056-.151-.032-.305-.062-.45-.095-.09-.02-.173-.043-.26-.064-.138-.034-.277-.067-.407-.102-.082-.022-.157-.046-.235-.069a11.75 11.75 0 0 1-.368-.108c-.075-.024-.141-.049-.213-.073-.11-.037-.223-.075-.325-.113-.067-.025-.125-.051-.188-.077-.096-.038-.195-.076-.282-.115-.06-.027-.11-.054-.166-.08-.08-.039-.162-.077-.233-.116-.052-.028-.094-.055-.142-.084-.063-.038-.13-.075-.185-.113-.043-.029-.075-.058-.113-.086-.048-.037-.098-.073-.139-.11-.032-.029-.054-.057-.08-.087-.033-.035-.069-.07-.093-.104-.02-.03-.031-.058-.046-.086-.018-.035-.039-.068-.049-.102l-.015-.113c.076-.977 4.074-2.736 9.748-2.736zm12.182 12.124c-.118-.628-.84-1.291-2.31-2.128l.963-7.16a.531.531 0 0 0 .005-.073C22.16 1.581 16.447 0 11.32 0 6.194 0 .482 1.581.482 3.851a.58.58 0 0 0 .005.072L2.819 21.25c.071 2.002 5.236 2.75 8.5 2.75 1.805 0 3.615-.188 5.098-.531.598-.138 1.133-.3 1.592-.48 1.18-.467 1.789-1.053 1.813-1.739l.945-7.018c.557.131 1.016.197 1.389.197.54 0 .902-.137 1.134-.413aLine truncated
+      <div class="abx-app-row abx-tools-row"><a class="abx-app-name" href="https://abx-dl.archivebox.io/">Tools</a><div class="abx-platforms"><a class="abx-tool" href="https://abx-dl.archivebox.io/"><span>One-shot <code>abx-dl</code> CLI</span></a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/mcp">MCP</a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage">API</a><a class="abx-tool" href="https://web.archivebox.io">Demo</a></div></div>
+    </div></details>
+    __SITE_NAV__
+  </nav>
+</header>

.github/pages/base/marquee.css

@@ -0,0 +1,11 @@
+/* Identical standalone gallery strip across ArchiveBox product sites. */
+/* The root query container measures the usable width, excluding scrollbars. */
+html { container-type: inline-size; }
+.abx-marquee { min-width: 0; width: 100cqw; max-width: none; margin: 24px calc(50% - 50cqw) 40px; padding: 0; overflow: hidden; }
+.abx-marquee .abx-marquee-viewport { width: 100%; overflow-x: auto; scroll-behavior: auto; overscroll-behavior-x: contain; scrollbar-width: thin; padding: 16px 24px 28px; box-sizing: border-box; }
+.abx-marquee .abx-marquee-track { display: flex; align-items: flex-start; gap: 24px; width: max-content; margin-inline: auto; }
+.abx-marquee .abx-marquee-card { display: block; flex: none; width: min-content; overflow: hidden; border-radius: 10px; background: #fff; box-shadow: 0 0 0 1px rgb(0 0 0 / 12%), 0 2px 4px rgb(0 0 0 / 10%), 0 8px 16px rgb(0 0 0 / 16%); text-decoration: none; color: #533344; }
+.abx-marquee .abx-marquee-card img { display: block; height: clamp(240px, 30vw, 400px); width: auto; max-width: none; max-height: none; object-fit: contain; box-shadow: none; border: 0; border-radius: 0; background: transparent; padding: 0; margin: 0; }
+.abx-marquee :focus-visible { outline: 2px solid #9b2854; outline-offset: 3px; }
+.abx-marquee .abx-marquee-viewport:focus-visible { outline-offset: -3px; }
+@media (max-width: 600px) { .abx-marquee .abx-marquee-track { gap: 16px; } .abx-marquee .abx-marquee-card img { height: 240px; } }

.github/pages/base/marquee.js

@@ -0,0 +1,44 @@
+/* Scroll real gallery links once; keep them usable without JavaScript. */
+(() => {
+  document.querySelectorAll('.abx-marquee').forEach(gallery => {
+    const viewport = gallery.querySelector('.abx-marquee-viewport');
+    const reducedMotion = window.matchMedia('(prefers-reduced-motion: reduce)');
+    let paused = reducedMotion.matches;
+    let hovering = false;
+    let focused = false;
+    let visible = false;
+    let frame;
+    let previous;
+    let offset = viewport.scrollLeft;
+    function tick(time) {
+      const distance = viewport.scrollWidth - viewport.clientWidth;
+      if (distance <= 0) return;
+      if (previous !== undefined) {
+        offset = Math.min(offset + Math.min(time - previous, 50) * 0.035, distance);
+        viewport.scrollTo({ left: offset, behavior: 'instant' });
+      }
+      if (offset >= distance - 1) { paused = true; return; }
+      previous = time;
+      frame = window.requestAnimationFrame(tick);
+    }
+    function sync() {
+      window.cancelAnimationFrame(frame);
+      previous = undefined;
+      offset = viewport.scrollLeft;
+      if (!paused && !hovering && !focused && visible && !document.hidden) frame = window.requestAnimationFrame(tick);
+    }
+    viewport.addEventListener('pointerenter', event => { if (event.pointerType === 'mouse') { hovering = true; sync(); } });
+    viewport.addEventListener('pointerleave', () => { hovering = false; sync(); });
+    viewport.addEventListener('focusin', () => { focused = true; sync(); });
+    viewport.addEventListener('focusout', event => { focused = viewport.contains(event.relatedTarget); sync(); });
+    function pauseForInteraction() { paused = true; sync(); }
+    viewport.addEventListener('pointerdown', pauseForInteraction);
+    viewport.addEventListener('wheel', pauseForInteraction, { passive: true });
+    viewport.addEventListener('keydown', pauseForInteraction);
+    reducedMotion.addEventListener('change', () => { paused = reducedMotion.matches; sync(); });
+    document.addEventListener('visibilitychange', sync);
+    new window.IntersectionObserver(([entry]) => { visible = entry.isIntersecting; sync(); }).observe(gallery);
+    new window.ResizeObserver(sync).observe(viewport);
+    viewport.querySelectorAll('img').forEach(image => image.addEventListener('load', sync));
+  });
+})();

.github/pages/marquee.py

@@ -0,0 +1,96 @@
+"""Render homepage screenshot strips from the site's existing gallery HTML."""
+
+import re
+from html import escape
+from html.parser import HTMLParser
+from urllib.parse import urljoin
+
+
+class Gallery(HTMLParser):
+    """Only gallery figures are captures; navigation and decorative images are not."""
+
+    def __init__(self, base):
+        super().__init__()
+        self.base = base
+        self.stack = []
+        self.captures = []
+
+    def handle_starttag(self, tag, attrs):
+        attrs = dict(attrs)
+        if tag == "img" and any(name == "figure" for name, _ in self.stack):
+            context = [item for _, item in self.stack]
+            profile = next(
+                (
+                    item.get("data-profile") or item.get("data-viewport")
+                    for item in reversed(context)
+                    if item.get("data-profile") or item.get("data-viewport")
+                ),
+                None,
+            )
+            if profile not in (None, "desktop"):
+                return
+            source = attrs.get("src", "")
+            if not source or not attrs.get("alt"):
+                return
+            anchor = next((item["id"] for name, item in reversed(self.stack) if name in ("article", "figure") and item.get("id")), "")
+            server = any(item.get("data-platform") == "server" for item in context)
+            self.captures.append(
+                {
+                    "src": urljoin(self.base, source),
+                    "href": self.base + "#" + anchor if anchor else urljoin(self.base, source),
+                    "title": attrs["alt"],
+                    "width": attrs.get("width", ""),
+                    "height": attrs.get("height", ""),
+                    "product": "server" if server else "client",
+                },
+            )
+        if tag not in {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"}:
+            self.stack.append((tag, attrs))
+
+    def handle_endtag(self, tag):
+        for index in range(len(self.stack) - 1, -1, -1):
+            if self.stack[index][0] == tag:
+                del self.stack[index:]
+                break
+
+
+def render(output, baseurl="", featured=()):
+    homepage = output / "index.html"
+    text = homepage.read_text()
+    slots = re.findall(r"<!-- ARCHIVEBOX:(MARQUEE(?:-CLIENT|-SERVER)?) -->", text)
+    if not slots:
+        return False
+    base = "/" + baseurl.strip("/") + "/" if baseurl.strip("/") else "/"
+    gallery = Gallery(base + "screenshots/")
+    gallery.feed((output / "screenshots/index.html").read_text())
+    for slot in slots:
+        product = slot.removeprefix("MARQUEE-").lower() if "-" in slot else None
+        captures = [capture for capture in gallery.captures if product is None or capture["product"] == product]
+        priority = {title: index for index, title in enumerate(featured)}
+        captures.sort(key=lambda capture: priority.get(capture["title"], len(priority)))
+        cards = []
+        for capture in captures:
+            dimensions = "".join(f' {key}="{escape(capture[key])}"' for key in ("width", "height") if capture[key])
+            cards.append(
+                f'<a class="abx-marquee-card" href="{escape(capture["href"])}"><img src="{escape(capture["src"])}" alt="{escape(capture["title"])}"{dimensions} loading="lazy" decoding="async"></a>',
+            )
+        viewport_id = slot.lower()
+        content = (
+            f'<section class="abx-marquee" aria-label="App screenshots"><div class="abx-marquee-viewport" id="{viewport_id}" tabindex="0" aria-label="Screenshots. Scroll sideways to explore."><div class="abx-marquee-track">{"".join(cards)}</div></div></section>'
+            if cards
+            else ""
+        )
+        start, end = f"<!-- ARCHIVEBOX:{slot} -->", f"<!-- /ARCHIVEBOX:{slot} -->"
+        replacement = start + "\n" + content + "\n" + end
+        if end in text:
+            text = re.sub(
+                re.escape(start) + r".*?" + re.escape(end),
+                lambda _, replacement=replacement: replacement,
+                text,
+                count=1,
+                flags=re.DOTALL,
+            )
+        else:
+            text = text.replace(start, replacement, 1)
+    homepage.write_text(text)
+    return True

.github/pages/nav.html

@@ -0,0 +1,4 @@
+<!-- Repository-specific navigation and primary CTA. -->
+<a href="__SITE_BASE__#use-cases">Use cases</a><a href="__SITE_BASE__#configuration">Configuration</a><a href="__SITE_BASE__#layout-security">Layout &amp; security</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
+    <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
+    <a class="abx-cta" href="__SITE_BASE__#install">Get started</a>

.github/pages/site.json

@@ -0,0 +1,16 @@
+{
+  "name": "Web Archive",
+  "url": "https://archivebox.io/",
+  "pages": [
+    "index.html",
+    "screenshots/index.html"
+  ],
+  "marquee_featured": [
+    "Snapshots grid — desktop",
+    "Snapshot View (header collapsed) — desktop",
+    "AI agent — desktop",
+    "Archive results — desktop",
+    "Add URLs — desktop",
+    "Snapshots table — desktop"
+  ]
+}

.github/pages/site.py

@@ -0,0 +1,215 @@
+"""Assemble and verify this repository's static Pages site (standard library only)."""
+
+import argparse
+import json
+import re
+import shutil
+import subprocess
+from datetime import UTC, datetime
+from html import escape, unescape
+from html.parser import HTMLParser
+from pathlib import Path
+from urllib.parse import unquote, urlsplit
+
+from marquee import render as render_marquee
+
+HERE = Path(__file__).resolve().parent
+ROOT = HERE.parents[1]
+
+
+def replace_block(text, name, content):
+    start, end = f"<!-- ARCHIVEBOX:{name} -->", f"<!-- /ARCHIVEBOX:{name} -->"
+    pattern = re.escape(start) + r".*?" + re.escape(end)
+    if re.search(pattern, text, re.DOTALL):
+        return re.sub(
+            pattern,
+            lambda _: f"{start}\n{content}\n{end}",
+            text,
+            count=1,
+            flags=re.DOTALL,
+        )
+    if text.count(start) != 1:
+        raise ValueError(f"Expected one {name} placeholder")
+    return text.replace(start, f"{start}\n{content}\n{end}", 1)
+
+
+def render_metadata(text):
+    """Keep product copy in native templates; render one consistent SEO block."""
+    page = Page()
+    page.feed(text)
+    title = unescape(re.search(r"<title>(.*?)</title>", text, re.DOTALL)[1])
+    meta = page.metadata
+    description = meta.get("description", meta.get("og:description", ""))
+    image = meta.get("og:image", meta.get("twitter:image", ""))
+    if not description or not image or not page.canonical:
+        raise ValueError("Each page needs a description, social image and canonical")
+    values = {
+        "description": description,
+        "robots": "index,follow,max-image-preview:large",
+        "theme-color": "#9b2854",
+        "og:type": "website",
+        "og:site_name": meta.get("og:site_name", "ArchiveBox"),
+        "og:locale": meta.get("og:locale", "en_US"),
+        "og:url": page.canonical,
+        "og:title": title,
+        "og:description": description,
+        "og:image": image,
+        "og:image:type": meta.get("og:image:type", "image/png"),
+        "og:image:width": meta.get("og:image:width", "1200"),
+        "og:image:height": meta.get("og:image:height", "630"),
+        "og:image:alt": meta.get("og:image:alt", title),
+        "twitter:card": "summary_large_image",
+        "twitter:site": "@ArchiveBoxApp",
+        "twitter:title": title,
+        "twitter:description": description,
+        "twitter:image": image,
+        "twitter:image:alt": meta.get("og:image:alt", title),
+    }
+
+    def strip_managed_tag(match):
+        tag = Page()
+        tag.feed(match[0])
+        if tag.canonicals or any(key in values for key in tag.metadata):
+            return ""
+        # System fonts are shared across sites, without a network font dependency.
+        if "fonts.googleapis.com" in match[0] or "fonts.gstatic.com" in match[0]:
+            return ""
+        return match[0]
+
+    text = re.sub(r"<(?:meta|link)\b[^>]*>", strip_managed_tag, text)
+    text = re.sub(r"<title>.*?</title>", "", text, flags=re.DOTALL)
+    tags = [
+        f"<title>{escape(title)}</title>",
+        f'<link rel="canonical" href="{escape(page.canonical, quote=True)}">',
+    ]
+    for key, value in values.items():
+        attr = "property" if key.startswith("og:") else "name"
+        tags.append(f'<meta {attr}="{key}" content="{escape(value, quote=True)}">')
+    if "<!-- ARCHIVEBOX:METADATA -->" not in text:
+        text = text.replace("</head>", "<!-- ARCHIVEBOX:METADATA -->\n</head>", 1)
+    return replace_block(text, "METADATA", "\n".join(tags))
+
+
+def render(output, source=None, baseurl=""):
+    config = json.loads((HERE / "site.json").read_text())
+    output = output.resolve()
+    if source:
+        source = source.resolve()
+        if output == source or output in source.parents or source in output.parents:
+            raise ValueError("Build output must be separate from the source directory")
+        if output == ROOT or output in ROOT.parents:
+            raise ValueError("Build output must not replace a repository")
+        if output.exists():
+            shutil.rmtree(output)
+        shutil.copytree(source, output)
+    revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
+    assets = output / "site-base"
+    assets.mkdir(parents=True, exist_ok=True)
+    has_marquee = render_marquee(output, baseurl, config.get("marquee_featured", []))
+    common_assets = ["chrome.css", "chrome.js"] + (["marquee.css", "marquee.js"] if has_marquee else [])
+    for name in common_assets:
+        shutil.copyfile(HERE / "base" / name, assets / name)
+    for name in config["pages"]:
+        page = output / name
+        text = render_metadata(page.read_text())
+        base = "../" * (len(Path(name).parts) - 1) or "./"
+        header = (HERE / "base/header.html").read_text().replace("__SITE_NAME__", escape(config["name"]))
+        header = header.replace("__SITE_NAV__", (HERE / "nav.html").read_text().strip())
+        text = replace_block(text, "HEADER", header)
+        for kind in ["FOOTER-START", "FOOTER-END"]:
+            text = replace_block(text, kind, (HERE / "base" / f"{kind.lower()}.html").read_text().strip())
+        text = text.replace("__SITE_BASE__", base).replace("__SITE_REVISION__", revision)
+        resources = f'<link rel="stylesheet" href="{base}site-base/chrome.css?v={revision}">\n<script src="{base}site-base/chrome.js?v={revision}" defer></script>'
+        if has_marquee and name == "index.html":
+            resources += f'\n<link rel="stylesheet" href="{base}site-base/marquee.css?v={revision}">\n<script src="{base}site-base/marquee.js?v={revision}" defer></script>'
+        if "<!-- ARCHIVEBOX:ASSETS -->" not in text:
+            text = text.replace("</head>", "<!-- ARCHIVEBOX:ASSETS -->\n</head>", 1)
+        text = replace_block(text, "ASSETS", resources)
+        page.write_text(text)
+    # Site build provenance is independent of all original screenshot manifests.
+    provenance_path = output / "build.json"
+    provenance = json.loads(provenance_path.read_text()) if provenance_path.exists() else {}
+    provenance.update(
+        revision=revision,
+        generatedAt=datetime.now(UTC).isoformat(),
+        site=config["url"],
+    )
+    provenance_path.write_text(json.dumps(provenance, indent=2) + "\n")
+    (output / ".nojekyll").touch()
+    verify(output, baseurl)
+    print(f"Assembled {len(config['pages'])} pages at {output} from {revision}")
+
+
+class Page(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.ids = set()
+        self.metadata = {}
+        self.canonical = ""
+        self.resources = []
+        self.headers = self.footers = self.canonicals = 0
+
+    def handle_starttag(self, tag, attrs):
+        attrs = dict(attrs)
+        if tag == "meta":
+            for key in ["name", "property"]:
+                if attrs.get(key):
+                    self.metadata[attrs[key]] = attrs.get("content", "")
+        if tag == "link" and attrs.get("rel") == "canonical":
+            self.canonical = attrs["href"]
+        if attrs.get("id"):
+            self.ids.add(attrs["id"])
+        classes = attrs.get("class", "").split()
+        self.headers += "abx-header" in classes
+        self.footers += "abx-footer" in classes
+        self.canonicals += tag == "link" and attrs.get("rel") == "canonical"
+        if tag in ["img", "script"] and attrs.get("src"):
+            self.resources.append(attrs["src"])
+        if tag == "link" and attrs.get("rel") in [
+            "stylesheet",
+            "icon",
+            "apple-touch-icon",
+        ]:
+            self.resources.append(attrs["href"])
+
+
+def verify(output, baseurl=""):
+    config = json.loads((HERE / "site.json").read_text())
+    output = output.resolve()
+    prefix = "/" + baseurl.strip("/") if baseurl.strip("/") else ""
+    for name in config["pages"]:
+        path = output / name
+        text = path.read_text()
+        page = Page()
+        page.feed(text)
+        if (page.headers, page.footers, page.canonicals) != (1, 1, 1):
+            raise ValueError(f"{name}: expected exactly one shared header, footer and canonical")
+        if "__SITE_" in text:
+            raise ValueError(f"{name}: unresolved site placeholder")
+        for resource in page.resources:
+            url = urlsplit(resource)
+            if url.scheme or url.netloc or not url.path:
+                continue
+            resource_path = unquote(url.path)
+            if prefix and resource_path.startswith(prefix + "/"):
+                resource_path = resource_path[len(prefix) :]
+            local = (output / resource_path.lstrip("/") if resource_path.startswith("/") else path.parent / resource_path).resolve()
+            if not local.is_relative_to(output) or not local.is_file():
+                raise ValueError(f"{name}: missing local resource {resource}")
+        for asset in ["chrome.css", "chrome.js"]:
+            if (output / "site-base" / asset).read_bytes() != (HERE / "base" / asset).read_bytes():
+                raise ValueError(f"{name}: stale common asset {asset}")
+    print(f"Verified shared layout and local resources for {len(config['pages'])} pages")
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("command", choices=["render", "verify"])
+    parser.add_argument("output", type=Path)
+    parser.add_argument("--source", type=Path)
+    parser.add_argument("--baseurl", default="")
+    args = parser.parse_args()
+    if args.command == "render":
+        render(args.output, args.source, args.baseurl)
+    else:
+        verify(args.output, args.baseurl)

.github/pages/verify.py

@@ -0,0 +1,255 @@
+"""Check the built site in real Chromium at desktop and phone widths."""
+
+import argparse
+import json
+import re
+from functools import partial
+from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
+from html.parser import HTMLParser
+from pathlib import Path
+from threading import Thread
+from urllib.parse import unquote, urljoin, urlsplit
+
+from playwright.sync_api import expect, sync_playwright
+
+HERE = Path(__file__).resolve().parent
+
+
+class Handler(SimpleHTTPRequestHandler):
+    def handle(self):
+        # Browser navigation can cancel an in-flight lazy image response.
+        try:
+            super().handle()
+        except (BrokenPipeError, ConnectionResetError):
+            pass
+
+    def log_message(self, *args):
+        pass
+
+
+class ScreenshotReferences(HTMLParser):
+    def __init__(self):
+        super().__init__()
+        self.references = []
+        self.ids = set()
+
+    def handle_starttag(self, tag, attributes):
+        attrs = dict(attributes)
+        if attrs.get("id"):
+            self.ids.add(attrs["id"])
+        for attribute in ("src", "href"):
+            if attrs.get(attribute):
+                self.references.append(attrs[attribute])
+
+
+def verify_screenshot_references(output, canonical):
+    """Check gallery embeds on every static page, including optional translations."""
+    pages = {}
+    for path in output.rglob("*.html"):
+        page = ScreenshotReferences()
+        page.feed(path.read_text())
+        pages[path.resolve()] = page
+    for path, page in pages.items():
+        page_url = urljoin(canonical, path.relative_to(output.resolve()).as_posix())
+        for reference in page.references:
+            url = urlsplit(urljoin(page_url, reference))
+            if url.netloc != urlsplit(canonical).netloc or "/screenshots/" not in url.path:
+                continue
+            target = output / unquote(url.path).lstrip("/")
+            if target.is_dir():
+                target /= "index.html"
+            target = target.resolve()
+            assert target.is_relative_to(output.resolve()) and target.is_file(), f"{path}: missing screenshot target {reference}"
+            assert not re.match(r"^\d+-", target.name), f"{path}: screenshot filename depends on capture order: {reference}"
+            if url.fragment and target in pages:
+                assert unquote(url.fragment) in pages[target].ids, f"{path}: missing screenshot anchor {reference}"
+
+
+def verify(output, evidence):
+    config = json.loads((HERE / "site.json").read_text())
+    verify_screenshot_references(output, config["url"])
+    evidence.mkdir(parents=True, exist_ok=True)
+    server = ThreadingHTTPServer(("127.0.0.1", 0), partial(Handler, directory=str(output.resolve())))
+    thread = Thread(target=server.serve_forever, daemon=True)
+    thread.start()
+    origin = f"http://127.0.0.1:{server.server_port}"
+    try:
+        with sync_playwright() as playwright:
+            browser = playwright.chromium.launch(ignore_default_args=["--hide-scrollbars"])
+            for name in config["pages"]:
+                route = name.removesuffix("index.html")
+                page = browser.new_page(reduced_motion="reduce")
+                errors = []
+                missing = []
+                page.on("pageerror", lambda error, errors=errors: errors.append(str(error)))
+                page.on(
+                    "response",
+                    lambda response, missing=missing: (
+                        missing.append(response.url) if response.url.startswith(origin) and response.status >= 400 else None
+                    ),
+                )
+                for width in [1716, 2560, 390]:
+                    page.set_viewport_size({"width": width, "height": 900})
+                    page.goto(f"{origin}/{route}", wait_until="domcontentloaded")
+                    header = page.locator(".abx-header")
+                    expect(header).to_have_count(1)
+                    expect(header).to_be_visible()
+                    expect(header).to_be_in_viewport()
+                    brand = header.locator(".abx-brand")
+                    expect(brand).to_be_visible()
+                    expect(brand).to_contain_text("ArchiveBox")
+                    expect(brand.locator(".abx-logo")).to_be_visible()
+                    expect(brand.locator(".abx-logo")).to_be_in_viewport()
+                    expect(header).to_have_css("display", "flex")
+                    expect(page.locator(".abx-footer")).to_have_count(1)
+                    expect(page.locator(".abx-footer-column")).to_have_count(3)
+                    expect(header.locator(".abx-cta")).to_be_visible()
+                    menu = header.locator(".abx-apps")
+                    menu.locator("summary").focus()
+                    page.keyboard.press("Enter")
+                    expect(menu).to_have_attribute("open", "")
+                    expect(menu.locator("a").first).to_be_visible()
+                    page.keyboard.press("Escape")
+                    expect(menu).not_to_have_attribute("open", "")
+                    menu.locator("summary").click()
+                    box = header.bounding_box()
+                    assert box
+                    page.mouse.click(box["x"] + 2, box["y"] + 5)
+                    expect(menu).not_to_have_attribute("open", "")
+                    assert page.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth"), (
+                        f"{name}: overflow at {width}"
+                    )
+                    if not route and "screenshots/index.html" in config["pages"]:
+                        strips = page.locator(".abx-marquee")
+                        assert strips.count() > 0, "Homepage must use its screenshot gallery"
+                        for strip in strips.all():
+                            strip.scroll_into_view_if_needed()
+                            bounds = strip.bounding_box()
+                            usable_width = page.evaluate("document.documentElement.clientWidth")
+                            assert bounds and abs(bounds["x"]) < 1
+                            assert abs(bounds["width"] - usable_width) < 1
+                            assert not strip.inner_text().strip(), "Screenshot strips should have no visible captions or links"
+                            cards = strip.locator(".abx-marquee-card")
+                            assert cards.count() > 0, "Screenshot strip is empty"
+                            expect(strip.locator("button")).to_have_count(0)
+                            expect(strip.locator("a:not(.abx-marquee-card)")).to_have_count(0)
+                            viewport = strip.locator(".abx-marquee-viewport")
+                            overflowing = viewport.evaluate(
+                                "node => node.scrollWidth > node.clientWidth",
+                            )
+                            if overflowing:
+                                page.mouse.move(0, 0)
+                                page.emulate_media(reduced_motion="no-preference")
+                                initial = viewport.evaluate("node => node.scrollLeft")
+                                page.wait_for_function(
+                                    "([node, initial]) => node.scrollLeft > initial",
+                                    arg=[viewport.element_handle(), initial],
+                                )
+                                viewport.hover()
+                                stopped = viewport.evaluate("node => node.scrollLeft")
+                                viewport.evaluate(
+                                    "node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
+                                )
+                                assert viewport.evaluate("node => node.scrollLeft") == stopped
+                                page.emulate_media(reduced_motion="reduce")
+                                page.mouse.move(0, 0)
+                                stopped = viewport.evaluate("node => node.scrollLeft")
+                                viewport.evaluate(
+                                    "node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
+                                )
+                                assert viewport.evaluate("node => node.scrollLeft") == stopped
+                                viewport.focus()
+                                page.keyboard.press("ArrowRight")
+                                assert viewport.evaluate("node => node.scrollWidth > node.clientWidth")
+                            else:
+                                centered = viewport.evaluate(
+                                    """node => {
+                                        const view = node.getBoundingClientRect();
+                                        const track = node.querySelector('.abx-marquee-track').getBoundingClientRect();
+                                        return Math.abs((track.left - view.left) - (view.right - track.right)) < 2;
+                                    }""",
+                                )
+                                assert centered, "Short screenshot strips must be centered"
+                            for card in cards.all():
+                                href = card.get_attribute("href")
+                                assert href and "/screenshots/" in href
+                                response = page.request.get(urljoin(page.url, href))
+                                assert response.ok, f"Broken screenshot link: {href}"
+                                anchor = urlsplit(href).fragment
+                                if anchor:
+                                    assert f'id="{anchor}"' in response.text(), f"Missing screenshot anchor: {href}"
+                    # Decode every local content image using the real browser loader,
+                    # including images in offscreen carousel tracks and lazy galleries.
+                    images = page.locator("main img").evaluate_all("nodes => [...new Set(nodes.map(image => image.src))]")
+                    local_images = [url for url in images if url.startswith(origin + "/")]
+                    page.evaluate(
+                        """async urls => {
+                        for (const url of urls) {
+                            const image = new Image();
+                            image.src = url;
+                            await image.decode();
+                            if (!image.naturalWidth) throw new Error(`Broken image: ${url}`);
+                        }
+                    }""",
+                        local_images,
+                    )
+                    page.evaluate("scrollTo({top: 0, behavior: 'instant'})")
+                    page.evaluate(
+                        "() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
+                    )
+                    page.screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}{width}.png"))
+                    page.locator(".abx-footer").screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}footer-{width}.png"))
+                    for link in header.locator(".abx-nav > a").all():
+                        url = urlsplit(link.get_attribute("href") or "")
+                        if url.scheme or url.netloc:
+                            continue
+                        response = page.request.get(urljoin(page.url, url.geturl()))
+                        assert response.ok, f"Broken navigation: {url.geturl()}"
+                        if url.fragment:
+                            body = response.text()
+                            assert f'id="{url.fragment}"' in body or f"id='{url.fragment}'" in body, (
+                                f"Missing navigation anchor: {url.geturl()}"
+                            )
+                    if route == "" and page.locator("#resources").count():
+                        page.goto(f"{origin}/#resources")
+                        expect(page.locator("#resources")).to_be_visible()
+                assert not errors, errors
+                assert not missing, missing
+                page.close()
+                context = browser.new_context(java_script_enabled=False, viewport={"width": 390, "height": 900})
+                plain = context.new_page()
+                plain.goto(f"{origin}/{route}", wait_until="domcontentloaded")
+                expect(plain.locator(".abx-header")).to_be_visible()
+                expect(plain.locator(".abx-header .abx-brand")).to_be_in_viewport()
+                expect(plain.locator(".abx-header .abx-logo")).to_be_visible()
+                plain.locator(".abx-apps summary").click()
+                expect(plain.locator(".abx-app-links a").first).to_be_visible()
+                expect(plain.locator(".abx-footer-column a").first).to_be_visible()
+                if not route and "screenshots/index.html" in config["pages"]:
+                    assert plain.locator(".abx-marquee-card").count() > 0
+                    for strip in plain.locator(".abx-marquee").all():
+                        bounds = strip.bounding_box()
+                        usable_width = plain.evaluate("document.documentElement.clientWidth")
+                        assert bounds and abs(bounds["x"]) < 1
+                        assert abs(bounds["width"] - usable_width) < 1
+                    assert plain.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth")
+                    expect(plain.locator(".abx-marquee button")).to_have_count(0)
+                    expect(plain.locator(".abx-marquee-card").first).to_be_visible()
+                context.close()
+                print(
+                    f"PASS {name}: desktop/mobile, keyboard, no-JS, images and resources",
+                    flush=True,
+                )
+            browser.close()
+    finally:
+        server.shutdown()
+        server.server_close()
+        thread.join()
+
+
+if __name__ == "__main__":
+    parser = argparse.ArgumentParser(description=__doc__)
+    parser.add_argument("output", type=Path)
+    parser.add_argument("--evidence", type=Path, default=Path("test-results/site"))
+    args = parser.parse_args()
+    verify(args.output, args.evidence)

.github/workflows/deploy-publicsite.yml

@@ -1,79 +1,87 @@
-name: Deploy Publicsite to GitHub Pages
+name: Publish website
 
 on:
   push:
-    branches:
-      - dev
-    paths:
-      - publicsite/**
-      - archivebox/**
-      - bin/collect_ui_screenshots.*
-      - bin/generate_ui_screenshot_gallery.py
-      - bin/take_screenshot.js
-      - docs/Screenshots.md
-      - pyproject.toml
-      - uv.lock
-      - .github/workflows/deploy-publicsite.yml
+    branches: [dev]
+  pull_request:
   workflow_dispatch:
+  workflow_run:
+    workflows: ['Capture publicsite screenshots']
+    types: [completed]
+    branches: [dev]
 
 permissions:
   contents: read
-  pages: write
-  id-token: write
+  actions: read
 
+env:
+  # Repository-specific output; all publication logic below is a local copy.
+  SITE_OUTPUT: _site
+  DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
+
+# One production queue covers restore, build and deploy. PRs have separate queues.
 concurrency:
-  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
-  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
+  group: pages-${{ github.event_name == 'pull_request' && github.ref || 'production' }}
+  cancel-in-progress: false
 
 jobs:
+  build:
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
+    outputs:
+      revision: ${{ steps.source.outputs.revision }}
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event_name == 'pull_request' && github.ref || env.DEFAULT_BRANCH }}
+          persist-credentials: false
+      - name: Record website source revision
+        id: source
+        run: echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+
+      # BEGIN REPOSITORY-SPECIFIC BUILD
+      - uses: astral-sh/setup-uv@v7
+      - name: Restore latest complete screenshots
+        run: uv run --no-project python bin/restore_site_screenshots.py publicsite/screenshots
+        env:
+          GH_TOKEN: ${{ github.token }}
+      - name: Render current gallery layout without recapturing images
+        run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
+      - name: Assemble website with the local base components
+        run: uv run --no-project python .github/pages/site.py render _site --source publicsite
+      # END REPOSITORY-SPECIFIC BUILD
+
+      - name: Verify website in Chromium
+        run: |
+          uv run --no-config --no-project --with playwright==1.63.0 playwright install --with-deps chromium
+          uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py "$SITE_OUTPUT" --evidence "$RUNNER_TEMP/site-evidence"
+      - name: Save website verification evidence
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: site-verification
+          path: ${{ runner.temp }}/site-evidence
+          if-no-files-found: warn
+          retention-days: 14
+
+      - name: Upload website
+        uses: actions/upload-pages-artifact@v3
+        with:
+          path: ${{ env.SITE_OUTPUT }}
+
   deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      pages: write
+      id-token: write
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-24.04
     steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Setup Python and uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          enable-cache: false
-
-      - name: Install ArchiveBox and screenshot dependencies
-        run: uv sync --no-cache --frozen
-
-      - name: Build UI screenshot gallery
-        run: bin/collect_ui_screenshots.sh
-        env:
-          BASE_URL: http://archivebox.localhost:8000
-
-      - name: Verify gallery was generated from this exact revision
-        run: |
-          uv run --no-cache --no-sync python - <<'PY'
-          import json
-          import os
-          import tomllib
-          from pathlib import Path
-
-          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
-          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
-          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
-          assert provenance["version"] == version, provenance
-          assert provenance["capture_count"] >= 3 * 35, provenance
-          assert len(provenance["files"]) == provenance["capture_count"], provenance
-          PY
-
-      - name: Setup Pages
-        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
-        with:
-          path: ./publicsite
-
-      - name: Deploy to GitHub Pages
+      - uses: actions/configure-pages@v5
+      - uses: actions/deploy-pages@v4
         id: deployment
-        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

.github/workflows/docker.yml

@@ -246,6 +246,26 @@ jobs:
         if: inputs.push_digests
         run: echo ${{ steps.docker_build.outputs.digest }}
 
+      - name: Verify OpenCode install with a read-only browser profile
+        shell: bash
+        env:
+          IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
+        run: |
+          set -Eeuo pipefail
+          PROFILE_DIR="$(mktemp -d)"
+          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
+            --volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
+            "$IMAGE" bash -c '
+              set -Eeuo pipefail
+              test -w "$XDG_CONFIG_HOME"
+              test ! -w "$XDG_CONFIG_HOME/chromium"
+              # Check the shipped executable before the installer can repair
+              # an image that accidentally omitted this opt-in dependency.
+              "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
+              archivebox install opencode
+              "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
+            '
+
       - name: Validate exact built image version and commit
         if: ${{ !inputs.push_digests }}
         shell: bash
@@ -296,8 +316,8 @@ jobs:
             MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
           fi
           COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
-          LIMIT=$((750 * 1024 * 1024))
-          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
+          LIMIT=$((1025 * 1024 * 1024))
+          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=1025 MiB"
           (( COMPRESSED_SIZE < LIMIT ))
 
       - name: Resolve real Docker CLI test dependencies through abxpkg
@@ -357,7 +377,7 @@ jobs:
           "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
           "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+            "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
             "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
 
@@ -378,6 +398,12 @@ jobs:
           IMAGE="${{ matrix.local_tag }}" \
             "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
 
+      - name: Validate Docker port compatibility
+        shell: bash
+        env:
+          IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
+        run: '"$BASH_BINARY" bin/validate_docker_ports.sh'
+
       - name: Validate Docker CLI add/list and compose behavior
         if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
@@ -401,13 +427,13 @@ jobs:
           "$DOCKER_BINARY" compose run --rm archivebox init
           "$DOCKER_BINARY" compose up -d --wait
           "$DOCKER_BINARY" network connect archivebox_default www-nginx
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
-          admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
+          admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/static/admin/js/jquery.init.js')"
           [[ "$index_html" == *ArchiveBox* ]]
           [[ "$admin_js" == *window.django* ]]
 
           "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
           [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
           "$DOCKER_BINARY" compose down
 
@@ -423,7 +449,7 @@ jobs:
             '${{ runner.temp }}/archivebox-image.tar' \
             '${{ steps.docker_build_pr.outputs.digest }}' \
             "$LOCAL_IMAGE_ID" \
-            "$((750 * 1024 * 1024))" <<'PY'
+            "$((1025 * 1024 * 1024))" <<'PY'
           from hashlib import sha256
           import json
           import os

.github/workflows/docs.yml

@@ -99,9 +99,10 @@ jobs:
           if [[ '${{ matrix.environment }}' != macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries")
           fi
-          if [[ '${{ matrix.environment }}' == ubuntu ]]; then
+          if [[ '${{ matrix.name }}' == ubuntu-AGENTS || '${{ matrix.name }}' == ubuntu-skills-archivebox-SKILL || '${{ matrix.name }}' == ubuntu-docs-Configuration || '${{ matrix.name }}' == ubuntu-docs-Setting-up-Authentication ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
-          elif [[ '${{ matrix.environment }}' == docker ]]; then
+          fi
+          if [[ '${{ matrix.environment }}' == docker ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries")
           elif [[ '${{ matrix.environment }}' == macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries")
@@ -145,7 +146,9 @@ jobs:
           fi
       - name: Build the exact checkout's documentation image
         if: ${{ matrix.environment == 'docker' }}
-        run: '"$DOCKER_BINARY" build --tag archivebox/archivebox:dev .'
+        run: |
+          ABX_DL_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(dep.removeprefix("abx-dl==") for dep in tomllib.load(open("pyproject.toml", "rb"))["project"]["dependencies"] if dep.startswith("abx-dl==")))')"
+          "$DOCKER_BINARY" build --build-arg "ABX_DL_IMAGE=archivebox/abx-dl:${ABX_DL_VERSION}" --tag archivebox/archivebox:dev .
       - name: Run the Ubuntu, Docker, or macOS documentation lane
         if: ${{ !contains(fromJSON('["root", "freebsd", "openbsd"]'), matrix.environment) }}
         run: uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'

.github/workflows/duplicate-issue-detection.yml

@@ -1,96 +0,0 @@
-name: Duplicate Issue Detection
-
-on:
-  issues:
-    types: [opened]
-
-jobs:
-  check-duplicates:
-    runs-on: ubuntu-24.04
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-
-      - name: Resolve OpenCode and workflow tools through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          uv sync --no-cache --locked --no-sources --no-install-project
-          OPENCODE_CONFIG="$(uv run --no-cache --no-sync --no-sources python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "opencode" / "config.json")')"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
-          opencode_env="$(
-            uv run --no-cache --no-sync --no-sources abxpkg env \
-              --install \
-              --json \
-              --lib="$ABXPKG_LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:github_binaries" \
-              --deps-from="$OPENCODE_CONFIG:required_binaries"
-          )"
-          jq_binary="$ABXPKG_LIB_DIR/env/bin/jq"
-          "$jq_binary" -r 'to_entries[] | "\(.key)=\(.value)"' \
-            <<< "$opencode_env" >> "$GITHUB_ENV"
-
-          opencode_binary="$ABXPKG_LIB_DIR/env/bin/opencode"
-          echo "OPENCODE_BINARY=$opencode_binary" >> "$GITHUB_ENV"
-          test -L "$opencode_binary"
-          test -x "$opencode_binary"
-
-      - name: Check for duplicate issues
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          OPENCODE_PERMISSION: |
-            {
-              "bash": {
-                "gh issue*": "allow",
-                "gh pr*": "allow",
-                "*": "deny"
-              },
-              "webfetch": "allow"
-            }
-        run: |
-          "$OPENCODE_BINARY" run -m anthropic/claude-haiku-4-5 "A new issue has been created:
-
-          Issue number:
-          ${{ github.event.issue.number }}
-
-          Lookup this issue and search through existing issues and PRs (excluding #${{ github.event.issue.number }}) in this repository to find any potential duplicates of this new issue.
-          Consider:
-          1. Similar titles or descriptions
-          2. Same error messages or symptoms
-          3. Related functionality or components
-          4. Similar feature requests
-
-          If you find any potential duplicates, please comment on the new issue with:
-          - A brief explanation of why it might be a duplicate
-          - Links to the potentially duplicate issues or PRs
-          - A suggestion to check those issues first
-
-          Use this format for the comment:
-          'This issue might be a duplicate of existing issues. Please check:
-          - #[issue_number]: [brief description of similarity]
-
-          Feel free to ignore if none of these address your specific case.'
-
-          If no clear duplicates are found, do not comment."

.github/workflows/release-candidate.yml

@@ -49,3 +49,17 @@ jobs:
       push_digests: true
       full_tests: false
     secrets: inherit
+
+  opencode:
+    needs: prepare
+    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
+    strategy:
+      fail-fast: false
+      matrix:
+        test_file:
+          - archivebox/tests/test_opencode_agent.py
+          - archivebox/tests/test_opencode_browser.py
+    uses: ./.github/workflows/test-parallel.yml
+    with:
+      test_file: ${{ matrix.test_file }}
+    secrets: inherit

.github/workflows/release.yml

@@ -27,8 +27,11 @@ jobs:
       ((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
        (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
     runs-on: ubuntu-24.04
+    permissions:
+      actions: read
+      contents: read
     outputs:
-      ready: ${{ steps.owner.outputs.ready }}
+      ready: ${{ steps.verified.outputs.ready }}
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
@@ -45,6 +48,32 @@ jobs:
           TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
           [[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
 
+      - name: Require successful full CI for this commit
+        id: verified
+        if: steps.owner.outputs.ready == 'true'
+        timeout-minutes: 40
+        env:
+          GH_TOKEN: ${{ github.token }}
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          CI_RUN_ID=''
+          for _ in {1..12}; do
+            CI_RUN_ID="$(gh run list \
+              --repo "$GITHUB_REPOSITORY" \
+              --workflow ci.yml \
+              --commit "$RELEASE_SHA" \
+              --event push \
+              --limit 1 \
+              --json databaseId \
+              --jq '.[0].databaseId // empty')"
+            [[ -n "$CI_RUN_ID" ]] && break
+            sleep 5
+          done
+          [[ -n "$CI_RUN_ID" ]]
+          gh run watch "$CI_RUN_ID" --repo "$GITHUB_REPOSITORY" --exit-status
+          echo 'ready=true' >> "$GITHUB_OUTPUT"
+
   python-release:
     name: Publish tested Python artifacts
     needs: candidate

.github/workflows/screenshots.yml

@@ -0,0 +1,71 @@
+name: Capture publicsite screenshots
+
+on:
+  push:
+    branches:
+      - dev
+    paths:
+      - archivebox/**
+      - bin/collect_ui_screenshots.*
+      - bin/generate_ui_screenshot_gallery.py
+      - bin/take_screenshot.js
+      - docs/Screenshots.md
+      - pyproject.toml
+      - uv.lock
+      - .github/workflows/screenshots.yml
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.event_name == 'workflow_dispatch' && format('publicsite-captures-manual-{0}', github.run_id) || 'publicsite-captures' }}
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Setup Python and uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: false
+
+      - name: Install ArchiveBox and screenshot dependencies
+        run: uv sync --no-cache --frozen
+
+      - name: Build UI screenshot gallery
+        run: bin/collect_ui_screenshots.sh
+        env:
+          BASE_URL: http://archivebox.localhost:5797
+
+      - name: Verify gallery was generated from this exact revision
+        run: |
+          uv run --no-cache --no-sync python - <<'PY'
+          import json
+          import os
+          import re
+          import tomllib
+          from pathlib import Path
+
+          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
+          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
+          assert provenance["version"] == version, provenance
+          assert provenance["capture_count"] >= 3 * 35, provenance
+          assert len(provenance["files"]) == provenance["capture_count"], provenance
+          assert all(not re.match(r"^\d+-", name) for name in provenance["files"]), provenance
+          PY
+
+      - name: Save validated screenshots
+        uses: actions/upload-artifact@v4
+        with:
+          name: site-screenshots
+          path: publicsite/screenshots
+          if-no-files-found: error
+          retention-days: 90

.github/workflows/test-parallel.yml

@@ -2,6 +2,12 @@ name: Parallel Tests
 
 on:
   workflow_call:
+    inputs:
+      test_file:
+        description: Run one discovered test file; empty runs the complete matrix
+        required: false
+        default: ""
+        type: string
 
 env:
   PYTHONIOENCODING: utf-8
@@ -58,10 +64,15 @@ jobs:
 
       - name: Discover every test exactly once
         id: discover
+        env:
+          TEST_FILE: ${{ inputs.test_file }}
         run: |
           set -Eeuo pipefail
           mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
           test "${#output[@]}" -eq 2
+          if [[ -n "$TEST_FILE" ]]; then
+            output[1]="$("$JQ_BINARY" -ce '[.[] | select(.paths == [env.TEST_FILE])] | select(length == 1)' <<< "${output[1]}")"
+          fi
           echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
           echo "${output[0]}"
           "$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
@@ -132,6 +143,9 @@ jobs:
             extra_args=(--extra "$TEST_EXTRA")
           fi
           uv sync --no-cache --locked --dev "${extra_args[@]}"
+      - name: Install Playwright Chromium for browser tests
+        if: contains(matrix.test.paths_arg, 'test_proof_previews') || contains(matrix.test.paths_arg, 'test_snapshot_output_stacks')
+        run: uv run --no-cache --no-sync --no-sources playwright install chromium
 
       - name: Run ${{ matrix.test.name }}
         env:

.gitignore

@@ -68,3 +68,5 @@ ArchiveBox.conf
 
 # Local Claude Code task locks (not project state)
 .claude/scheduled_tasks.lock
+
+/_site/

Dockerfile

@@ -1,8 +1,38 @@
 # syntax=docker/dockerfile:1.7
 
 # Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
-# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
-# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
+# WHY: abx-dl owns Python, Node, Chromium, and extraction dependencies so the
+# standalone downloader stays independent of ArchiveBox's server features.
+# ArchiveBox-only dependencies (Sonic, OpenCode, supervisor, Django) belong in
+# the layers added by this Dockerfile, never in the shared abx-dl base image.
+# REQUIRED INSTALLATION CONTRACT — do not work around failures:
+# 1. IMAGE CONSTRUCTION: preinstall ALL plugins supported by runtime="archivebox"
+#    and ALL their required_binaries, regardless of default enabled state.
+#    Inherit the complete downloader dependencies from the tested abx-dl image;
+#    install every remaining ArchiveBox-only dependency in this image's layers,
+#    including OpenCode and Sonic. Never put server dependencies in abx-dl.
+#    Use the normal plugin dependency pipeline: install command -> config.json
+#    required_binaries -> abxpkg. Never add manual apt/pip/npm installs to hide
+#    failures in that pipeline. OS/interpreter bootstrap is a separate concern,
+#    not a second hand-maintained list of plugin dependencies.
+# 2. FINISHED IMAGE: archivebox install and the full plugin installation check
+#    must discover everything already present and be NO-OPS offline as UID 911.
+#    No downloads, package installs, sudo, or root repair may be needed to enable
+#    ANY plugin or start a new crawl. Never run verification as root or manually
+#    preinstall packages before it. If the inherited image lacks a downloader
+#    dependency, fix/release abx-dl and consume that tested image; do not patch
+#    over the missing base dependency here or repair deployed containers by SSH.
+# 3. NON-CONTAINER CI: exercise real runtime auto-installation through the same
+#    CLI/required_binaries/abxpkg path users invoke. GitHub's runner has
+#    passwordless sudo; abxpkg must escalate automatically when needed. Never
+#    manually preinstall dependencies or wrap the tested command in sudo to make
+#    a failure disappear. This is distinct from the finished-image no-op test.
+# 4. SIZE: the complete compressed image budget is 1025 MiB. Preserve ALL
+#    user-facing functionality. Only mechanical/internal
+#    optimizations are allowed. Never remove dependencies, disable features or
+#    plugins, weaken assertions, or skip checks to meet a size limit. If the
+#    complete image exceeds the configured limit, LET CI FAIL and explicitly
+#    ask the maintainer what to do. Do not raise the limit unilaterally.
 # Build abx-dl first, then point this file at it:
 #   docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
 #       --build-context abxbus=../abxbus \
@@ -10,10 +40,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.233 \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:dev \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.233
+ARG ABX_DL_IMAGE=archivebox/abx-dl:main
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -47,6 +77,7 @@ ENV ARCHIVEBOX_USER=archivebox \
     IN_DOCKER=True
 
 ENV CODE_DIR=/app \
+    ABX_RUNTIME=archivebox \
     DATA_DIR=/data \
     CONFIG_DIR=/opt/archivebox \
     ABXPKG_LIB_DIR=/opt/archivebox/lib \
@@ -96,17 +127,15 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$T
     --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
     <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
-echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
-echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
+export UV_NO_CACHE=true
 rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
-apt-get install -qq -y --no-install-recommends \
-    build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+apt-get install -qq -y --no-install-recommends x11-utils
 # Extend abx-dl's existing venv instead of clearing it. Clearing and later
 # copying a complete replacement over the base duplicates every venv byte in
 # the final overlay history even when most packages are unchanged.
-/usr/bin/uv pip install --no-cache setuptools pip wheel
+# Do not rewrite inherited native libraries merely to strip them again.
+/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries
 
 mkdir -p /tmp/archivebox-uv-project
 /venv/bin/python - <<'PY'
@@ -155,25 +184,31 @@ PY
     --inexact \
     --no-install-project \
     --no-install-workspace \
+    --no-install-package abxbus \
+    --no-install-package abxpkg \
+    --no-install-package abx-plugins \
+    --no-install-package abx-dl \
     --no-sources
 builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
 ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
 while IFS= read -r -d '' native_library; do
+    if grep -Fxq "$native_library" /tmp/archivebox-inherited-libraries; then
+        continue
+    fi
     magic=''
     if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
         "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
     fi
 done < /tmp/archivebox-native-libraries
-rm -f /tmp/archivebox-native-libraries
+rm -f /tmp/archivebox-native-libraries /tmp/archivebox-inherited-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils
 abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
 rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
     /venv/lib/python3.*/site-packages/wheel* \
     /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
 rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
 EOF
 
@@ -202,7 +237,7 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
-# The builder installs and purges compilers in one layer, so its final
+# The builder installs and purges stripping tools in one layer, so its final
 # filesystem is already runtime-clean. Preserve that ancestry: starting again
 # from archivebox-runtime-base and COPYing /venv would bake a second full venv
 # over the inherited one instead of recording only ArchiveBox's package delta.
@@ -225,7 +260,8 @@ COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 # The builder invokes abxpkg as root for temporary ELF stripping, which can
 # rewrite derived state ownership. Restore UID 911 before resolving binaries
 # as the runtime user; the old copy-based final stage hid this dependency.
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
+RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777,id=archivebox-runtime-config-$TARGETARCH \
+    echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
     && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
     && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
     && ln -sf /venv/bin/daphne /usr/local/bin/daphne \
@@ -239,12 +275,27 @@ RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..."
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
     && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
-    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
+    && find "$ABXPKG_LIB_DIR" \( ! -user "$DEFAULT_ARCHIVEBOX_UID" -o ! -group "$DEFAULT_ARCHIVEBOX_GID" \) -exec chown -h "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" {} + \
+    && UV_NO_CACHE=true setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
     && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
 
+# Install the complete ArchiveBox catalog, including disabled server plugins.
+# Explicit plugin names make installation independent of enabled defaults.
+RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=archivebox-opencode-$TARGETARCH \
+    chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" /var/tmp/abxpkg-cache \
+    && chmod 1777 /var/tmp/abxpkg-cache \
+    && export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
+    && env ABXPKG_TMP_CACHE_DIR=/var/tmp/abxpkg-cache XDG_CACHE_HOME=/var/tmp/abxpkg-cache \
+        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install $ABX_DOCKER_PLUGINS \
+    # pnpm includes musl variants that Debian's glibc runtime cannot use.
+    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules" -type l -name 'opencode-linux-*-musl' -delete \
+    && find "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.pnpm" -maxdepth 1 -type d -name 'opencode-linux-*-musl@*' -exec rm -rf {} +
+RUN --network=none export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
+    bash -c '"$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version && abx-dl install $ABX_DOCKER_PLUGINS'
+
 WORKDIR "$DATA_DIR"
 RUN echo "[+] Initializing image collection..." \
     && find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
@@ -277,10 +328,10 @@ RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully
 
 WORKDIR "$DATA_DIR"
 VOLUME "$DATA_DIR"
-EXPOSE 8000
+EXPOSE 5797
 
 HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
-    CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
+    CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:5797/health/' | grep -q 'OK'
 
 ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
-CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]
+CMD ["archivebox", "server", "--init", "0.0.0.0:5797"]

README.md

@@ -73,15 +73,15 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
 curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
 docker compose pull
 docker compose up -d --wait                                                # initializes new collections automatically
-# open http://admin.archivebox.localhost:8000 to finish setup
+# open http://admin.archivebox.localhost:5797 to finish setup
 # docker compose run --rm archivebox add 'https://example.com'
 # docker compose run --rm archivebox help
 <br/>
 <br/>
 # Option B: Or use it as a plain Docker container:
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
-# open http://admin.archivebox.localhost:8000 to finish setup
+docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
+# open http://admin.archivebox.localhost:5797 to finish setup
 # docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
 # docker run -it -v $PWD:/data archivebox/archivebox:dev help
 <br/>
@@ -93,14 +93,14 @@ archivebox init
 archivebox install
 # archivebox add 'https://example.com'
 # archivebox help
-# archivebox server 0.0.0.0:8000
+# archivebox server 0.0.0.0:5797
 <br/>
 <br/>
 # Option D: Or use the uv install shortcut for Option C
 curl -fsSL 'https://get.archivebox.io' | bash
 </code></pre>
 <br/>
-<sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
+<sub>Open <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI and <a href="http://admin.archivebox.localhost:5797"><code>http://admin.archivebox.localhost:5797</code></a> for the admin UI ➡️</sub><br/>
 <sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
 </details>
 <br/>
@@ -129,10 +129,10 @@ curl -fsSL 'https://get.archivebox.io' | bash
 - [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
 - [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
 - [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
-- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
+- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
 - [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
-- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
-- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
+- Support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
+- Support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [infiniscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
 
 <br/>
 
@@ -151,6 +151,17 @@ curl -fsSL 'https://get.archivebox.io' | bash
 
 #### ✳️&nbsp; Easy Setup
 
+<details>
+<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
+<br/>
+<ol>
+<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
+<li>Set up an admin account and generate an API Key.</li>
+<li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
+</ol>
+<br/>
+</details>
+
 <details>
 <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b>  (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
 <br/>
@@ -166,7 +177,7 @@ docker compose pull
 <li>Start the server, which initializes a new collection automatically.
 <pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
 </code></pre></li>
-<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
 <pre lang="bash"><code style="white-space: pre-line">
 # run CLI commands inside the server container started above
 docker compose exec archivebox archivebox add 'https://example.com'
@@ -187,10 +198,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
 <li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
+docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
 </code></pre>
 </li>
-<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
 <pre lang="bash"><code style="white-space: pre-line">
 # completely optional, CLI can always be used without running a server
 # docker exec archivebox archivebox [subcommand] [--help]
@@ -245,8 +256,8 @@ archivebox init     # initialize a new collection
 archivebox install  # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
 </code></pre>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -281,8 +292,8 @@ archivebox add 'https://example.com'
 </code></pre>
 <br/>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -314,8 +325,8 @@ archivebox init
 archivebox install
 </code></pre>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -354,17 +365,12 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 <br/>
 <ol>
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
-<li>Download a binary release for your OS or build the native app from source<br/>
-<ul>
-<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
-<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
-<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
-</ul>
-</li>
+<li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
+<li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
 </ol>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
 <br/>
-<i>✨ Alpha (contributors wanted!)</i>: for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
+<a href="https://electron.archivebox.io/">Desktop app guide</a> · <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
 <br/>
 </details>
 
@@ -426,7 +432,7 @@ For more discussion on third-party hosting options see here: <a href="https://gi
 #### ➡️&nbsp; Next Steps
 
 - Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
-- (Optional) Create a persona and import browser cookies to archive logged-in sites: `archivebox persona create --import=chrome personal`
+- (Optional) [Import browser cookies and settings into a persona](docs/Chromium-Install.md#import-an-existing-browser-profile).
 - Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
 - Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
 - Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
@@ -454,11 +460,6 @@ docker compose run --rm archivebox help
 # equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
 docker run -it -v $PWD:/data archivebox/archivebox:dev help
 
-# optional: import your browser cookies into a persona for logged-in archiving
-archivebox persona create --import=chrome personal
-# supported: chrome/chromium/brave/edge (Chromium-based only)
-# use --profile to target a specific profile (e.g. Default, Profile 1)
-# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
 ```
 
 #### ArchiveBox Subcommands
@@ -538,16 +539,16 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html'  # inspect snapshot da
 <summary><b>🖥&nbsp; Web UI & API Usage</b></summary>
 <pre lang="bash"><code style="white-space: pre-line">
 # Start the server on bare metal (uv/apt/brew):
-archivebox server 0.0.0.0:8000                 # start the server
+archivebox server 0.0.0.0:5797                 # start the server
 <br/>
 # Or with Docker Compose:
 docker compose up                              # start the server
 <br/>
 # Or with a Docker container:
-docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
+docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
 </code></pre>
 
-<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
+<sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI. ➡️</sup><br/>
 <sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
 <sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
 <br/><br/>
@@ -564,6 +565,22 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
 
 # restart the server to apply any config changes
 </code></pre>
+
+Snapshot caching follows each snapshot's saved `PERMISSIONS`, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses `Cache-Control: private` so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.
+</details>
+
+<br/>
+
+<details>
+<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
+<br/>
+
+- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
+- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
+- 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
+- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
+
+<br/>
 </details>
 
 <br/>
@@ -619,7 +636,10 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>  
-  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
+  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
+
+- <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://app.archivebox.io/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
+  <i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files  
   <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
@@ -662,6 +682,8 @@ It also includes a built-in scheduled import feature with `archivebox schedule`,
 
 ## Output Formats: What ArchiveBox saves for each URL
 
+Snapshot detail pages group output cards into expandable stacks: HTML, Raster, Article text, Embedded media, OCR, and Metadata. HTML, Raster, Article text, and OCR use the preference order defined in [`archivebox/plugins/output_groups.py`](archivebox/plugins/output_groups.py); media and metadata put the largest output directory first. Unclassified plugins and auxiliary files remain available under Other files. The same layout is included in portable static exports.
+
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ace0954a-ddac-4520-9d18-1c77b1ec50b2" width="330px" align="right" style="float: right"/>
 
 
@@ -932,7 +954,7 @@ archivebox manage createsuperuser
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
 </ul>
 
@@ -985,14 +1007,11 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
 
 
 <ul>
-<li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
-<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
+<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">importing a chrome profile into a persona</a></li>
 <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
 <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
 </ul>
 
-In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/51">Issue #51</a>.
-
 </details>
 <br/>
 
@@ -1041,8 +1060,8 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 </li>
 <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
 </li>
-<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
-<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
+<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
+<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
 </li>
 </ul>
 
@@ -1323,10 +1342,10 @@ archivebox init
 archivebox install         # detect and install all extractor dependencies
 
 # Run the development server w/ autoreloading (but no bg workers)
-archivebox server --debug --reload 0.0.0.0:8000
+archivebox server --debug --reload 0.0.0.0:5797
 
 # Run the production server (with bg workers but no autoreloading)
-archivebox server 0.0.0.0:8000
+archivebox server 0.0.0.0:5797
 ```
 
 #### 2. Option B: Build the docker container and use that for development instead
@@ -1338,10 +1357,10 @@ archivebox server 0.0.0.0:8000
 ./bin/build_docker.sh dev
 
 # Run the development server w/ autoreloading (but no bg workers)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
 
 # Run the production server (with bg workers but no autoreloading)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
 
 # (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
 # When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
@@ -1363,11 +1382,11 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
 archivebox config --set DEBUG=True
 
 # OR you can run a dev server with DEBUG=True in a few ways:
-archivebox server --debug --reload 0.0.0.0:8000
+archivebox server --debug --reload 0.0.0.0:5797
 # or
-archivebox server --debug 0.0.0.0:8000
+archivebox server --debug 0.0.0.0:5797
 # or
-env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application
+env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
 ```
 
 https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
@@ -1402,7 +1421,7 @@ services:
 
 # or with plain Docker:
 docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
-docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
+docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
 
 # or with uv:
 uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'

archivebox/api/v1_auth.py

@@ -4,7 +4,7 @@ from django.http import HttpRequest
 
 from ninja import Router, Schema
 
-from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
+from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token, HeaderTokenAuth, BearerTokenAuth
 
 
 router = Router(tags=["Authentication"], auth=None)
@@ -63,3 +63,44 @@ def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
         return {"success": True, "user_id": str(user.pk)}
 
     return {"success": False, "user_id": None}
+
+
+@router.post(
+    "/browser_session",
+    auth=[HeaderTokenAuth(), BearerTokenAuth()],
+    summary="Exchange an administrator API key for a normal browser session",
+)
+def browser_session(request: HttpRequest):
+    from django.conf import settings
+    from django.contrib.auth import login
+    from django.http import JsonResponse
+    from django.utils import timezone
+    from ninja.errors import HttpError
+    from archivebox.core.routes_util import get_admin_base_url
+
+    user = request.auth
+    if not user.is_active or not user.is_superuser:
+        raise HttpError(403, "An active administrator is required")
+    login(request, user, backend="django.contrib.auth.backends.ModelBackend")
+    # A browser session must not outlive the key used to establish it.
+    token = request._api_token
+    lifetime = settings.SESSION_COOKIE_AGE
+    if token.expires:
+        lifetime = min(lifetime, max(0, int((token.expires - timezone.now()).total_seconds())))
+    request.session.set_expiry(lifetime)
+    request.session.save()
+    response = JsonResponse(
+        {
+            "admin_url": get_admin_base_url(request=request).rstrip("/") + "/admin/",
+            "cookie": {
+                "name": settings.SESSION_COOKIE_NAME,
+                "value": request.session.session_key,
+                "expires": request.session.get_expiry_date().timestamp(),
+                "secure": settings.SESSION_COOKIE_SECURE,
+            },
+        },
+    )
+    # The native client installs a host-only HttpOnly cookie on the admin origin;
+    # API and admin hosts can differ, so a Set-Cookie on the API host is insufficient.
+    response["Cache-Control"] = "no-store"
+    return response

archivebox/api/v1_cli.py

@@ -133,6 +133,15 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
             validate_url(args.urls[0])
         except ValueError:
             submitted_urls = args.urls[0]
+    else:
+        submitted_urls = []
+        for url in args.urls:
+            try:
+                submitted_urls.append(validate_url(url))
+            except ValueError:
+                continue
+        if not submitted_urls:
+            raise HttpError(400, "No valid URLs were submitted")
     crawl, snapshots = add(
         urls=submitted_urls,
         snapshot_ids=args.snapshot_ids,
@@ -156,7 +165,7 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
         "crawl_id": str(crawl.id),
         "num_snapshots": len(snapshot_ids),
         "snapshot_ids": snapshot_ids,
-        "queued_urls": args.urls,
+        "queued_urls": args.urls if isinstance(submitted_urls, str) else submitted_urls,
     }
     stdout = request.__dict__.get("stdout")
     stderr = request.__dict__.get("stderr")

archivebox/api/v1_core.py

@@ -11,7 +11,7 @@ from uuid import UUID
 from typing import Union, Any, Annotated
 from datetime import datetime, time
 
-from django.db import transaction
+from django.db import IntegrityError
 from django.db.models import Model, Q
 from django.http import HttpRequest, HttpResponse
 from django.http.multipartparser import MultiPartParser, MultiPartParserError
@@ -247,7 +247,7 @@ def get_archiveresult(request: HttpRequest, archiveresult_id: str):
 
 def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
     normalized = str(plugin or "").strip().strip("/")
-    if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
+    if normalized in (".", "..") or not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
         raise HttpError(400, "Invalid ArchiveResult plugin name")
     return normalized
 
@@ -386,6 +386,7 @@ def _write_archiveresult_files(
     existing_output_files: dict[str, dict[str, Any]] | None = None,
     allow_empty: bool = False,
 ) -> dict[str, dict[str, Any]]:
+    plugin_name = _normalize_uploaded_archiveresult_plugin(plugin_name)
     files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
     output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
     mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
@@ -517,52 +518,53 @@ def create_archiveresult(
     normalized_status = ArchiveResult.normalize_status(status)
     parsed_output_json = _parse_archiveresult_output_json(output_json)
     hook = hook_name or ARCHIVERESULT_UPLOAD_HOOK_NAME
-    existing_result = ArchiveResult.objects.filter(snapshot=snapshot, plugin=plugin_name).first()
-    existing_output_files = dict(existing_result.output_files or {}) if existing_result else {}
-    output_files = _write_archiveresult_files(
+    result_lookup = {
+        "snapshot": snapshot,
+        "plugin": plugin_name,
+        "hook_name": hook,
+    }
+    uploaded_output_files = _write_archiveresult_files(
         request,
         snapshot,
         plugin_name,
-        existing_output_files=existing_output_files,
         allow_empty=True,
     )
-    now = timezone.now()
-
-    with transaction.atomic():
-        Snapshot.objects.select_for_update().get(pk=snapshot.pk)
-        existing_result = ArchiveResult.objects.filter(snapshot=snapshot, plugin=plugin_name).first()
-        if existing_result:
-            output_files = {
-                **dict(existing_result.output_files or {}),
-                **output_files,
-            }
-            result = existing_result
-        else:
-            existing_result = None
-            result = ArchiveResult(
-                snapshot=snapshot,
-                plugin=plugin_name,
-                hook_name=hook,
-            )
-
-        if (
-            existing_result
-            and normalized_status == ArchiveResult.StatusChoices.STARTED
-            and existing_result.status != ArchiveResult.StatusChoices.STARTED
-        ):
-            normalized_status = existing_result.status
+    result = ArchiveResult.objects.filter(**result_lookup).first()
+    for _attempt in range(3):
+        output_files = {
+            **(result.output_file_map() if result else {}),
+            **uploaded_output_files,
+        }
         output_size, output_mimetypes = _summarize_archiveresult_output_files(output_files)
         output_file_paths = list(output_files.keys())
-        result.status = normalized_status
-        result.hook_name = hook
-        result.output_str = output_str or (output_file_paths[0] if output_file_paths else "")
-        result.output_json = parsed_output_json
-        result.output_files = output_files
-        result.output_size = output_size
-        result.output_mimetypes = output_mimetypes
-        result.start_ts = result.start_ts or now
-        result.end_ts = now
-        result.save()
+        result_status = normalized_status
+        if result and result_status == ArchiveResult.StatusChoices.STARTED and result.status != ArchiveResult.StatusChoices.STARTED:
+            result_status = result.status
+        now = timezone.now()
+        values = {
+            "status": result_status,
+            "output_str": output_str or (output_file_paths[0] if output_file_paths else ""),
+            "output_json": parsed_output_json,
+            "output_files": output_files,
+            "output_size": output_size,
+            "output_mimetypes": output_mimetypes,
+            "start_ts": result.start_ts or now if result else now,
+            "end_ts": now,
+        }
+        if result:
+            if result.safe_update(values):
+                break
+            continue
+        result, created = ArchiveResult.get_or_create_by_hook(
+            snapshot,
+            plugin_name,
+            hook,
+            defaults=values,
+        )
+        if created:
+            break
+    else:
+        raise HttpError(409, "ArchiveResult changed while upload metadata was being updated")
 
     if result.status != ArchiveResult.StatusChoices.STARTED:
         _queue_archiveresult_snapshot_maintenance(snapshot)
@@ -576,44 +578,46 @@ def patch_archiveresult(
 ):
     """Append or replace files on an existing ArchiveResult."""
     result = ArchiveResult.objects.select_related("snapshot__crawl__created_by").get(_uuid_ref_query("id", archiveresult_id))
-    output_files = _write_archiveresult_files(
+    uploaded_output_files = _write_archiveresult_files(
         request,
         result.snapshot,
         result.plugin,
-        existing_output_files=result.output_file_map(),
     )
-    latest_result = ArchiveResult.objects.only("output_files", "status").get(pk=result.pk)
-    output_files = {
-        **latest_result.output_file_map(),
-        **output_files,
-    }
-    output_size, output_mimetypes = _summarize_archiveresult_output_files(output_files)
-
-    update_fields = ["output_files", "output_size", "output_mimetypes", "end_ts", "modified_at"]
-    result.output_files = output_files
-    result.output_size = output_size
-    result.output_mimetypes = output_mimetypes
-    result.end_ts = timezone.now()
     output_str = _get_archiveresult_upload_form_value(request, "output_str")
     status = _get_archiveresult_upload_form_value(request, "status")
     output_json = _get_archiveresult_upload_form_value(request, "output_json")
-    if output_str:
-        result.output_str = output_str
-        update_fields.append("output_str")
-    if status:
-        normalized_status = ArchiveResult.normalize_status(status)
-        if normalized_status == ArchiveResult.StatusChoices.STARTED and latest_result.status != ArchiveResult.StatusChoices.STARTED:
-            normalized_status = latest_result.status
-        result.status = normalized_status
-        update_fields.append("status")
-    elif latest_result.status == ArchiveResult.StatusChoices.QUEUED and ArchiveResult.output_files_upload_complete(output_files):
-        result.status = ArchiveResult.StatusChoices.SUCCEEDED
-        update_fields.append("status")
-    if output_json:
-        result.output_json = _parse_archiveresult_output_json(output_json)
-        update_fields.append("output_json")
+    parsed_output_json = _parse_archiveresult_output_json(output_json) if output_json else None
+
+    if not ArchiveResult.output_files_upload_complete(uploaded_output_files):
+        result.output_files = {**result.output_file_map(), **uploaded_output_files}
+        result.output_size, result.output_mimetypes = _summarize_archiveresult_output_files(result.output_files)
+        return result
+
+    for _attempt in range(3):
+        output_files = {**result.output_file_map(), **uploaded_output_files}
+        output_size, output_mimetypes = _summarize_archiveresult_output_files(output_files)
+        values: dict[str, Any] = {
+            "output_files": output_files,
+            "output_size": output_size,
+            "output_mimetypes": output_mimetypes,
+            "end_ts": timezone.now(),
+        }
+        if output_str:
+            values["output_str"] = output_str
+        if status:
+            normalized_status = ArchiveResult.normalize_status(status)
+            if normalized_status == ArchiveResult.StatusChoices.STARTED and result.status != ArchiveResult.StatusChoices.STARTED:
+                normalized_status = result.status
+            values["status"] = normalized_status
+        elif result.status == ArchiveResult.StatusChoices.QUEUED:
+            values["status"] = ArchiveResult.StatusChoices.SUCCEEDED
+        if output_json:
+            values["output_json"] = parsed_output_json
+        if result.safe_update(values):
+            break
+    else:
+        raise HttpError(409, "ArchiveResult changed while upload metadata was being updated")
 
-    result.save(update_fields=update_fields)
     if result.status != ArchiveResult.StatusChoices.STARTED:
         _queue_archiveresult_snapshot_maintenance(result.snapshot)
 
@@ -930,52 +934,47 @@ def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
             created_by=request.user if isinstance(request.user, User) else None,
         )
 
-    # Browser clients sync metadata and upload capture artifacts immediately
-    # after queueing a URL. The runner holds the crawl lifecycle lock for the
-    # whole crawl, so idempotently posting an already-created snapshot must not
-    # wait behind that potentially long-running archive job.
-    existing_snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
-    if existing_snapshot is not None and (status is None or existing_snapshot.status == status):
-        if data.title is not None and existing_snapshot.title != data.title:
-            Snapshot.objects.filter(pk=existing_snapshot.pk).update(title=data.title, modified_at=timezone.now())
-            existing_snapshot.title = data.title
-        if tags:
-            existing_snapshot.save_tags(tags)
-        setattr(request, "with_archiveresults", False)
-        return existing_snapshot
+    # Browser uploads must not wait behind the runner's crawl-wide lifecycle
+    # lock. The unique insert recovery and CAS update below are the request-side
+    # coordination boundary for this idempotent metadata sync.
+    snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
+    if snapshot is None:
+        try:
+            snapshot = Snapshot.objects.create(
+                url=data.url,
+                crawl=crawl,
+                depth=data.depth,
+                title=data.title,
+                timestamp=str(timezone.now().timestamp()),
+                status=status or Snapshot.StatusChoices.QUEUED,
+                retry_at=timezone.now(),
+            )
+        except IntegrityError:
+            snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
+            if snapshot is None:
+                raise
 
-    with crawl_lifecycle_lock(str(crawl.id)):
-        snapshot_defaults = {
-            "depth": data.depth,
-            "title": data.title,
-            "timestamp": str(timezone.now().timestamp()),
-            "status": status or Snapshot.StatusChoices.QUEUED,
-            "retry_at": timezone.now(),
-        }
-        snapshot, _ = Snapshot.objects.get_or_create(
-            url=data.url,
-            crawl=crawl,
-            defaults=snapshot_defaults,
+    for _attempt in range(3):
+        updates: dict[str, Any] = {}
+        if data.title is not None and snapshot.title != data.title:
+            updates["title"] = data.title
+        if status is not None and snapshot.status != status:
+            updates["status"] = status
+        if not updates or snapshot.safe_update(updates, extra_filter={"modified_at": snapshot.modified_at}):
+            break
+    else:
+        raise HttpError(409, "Snapshot changed while metadata was being updated")
+
+    if tags:
+        snapshot.save_tags(
+            tags,
+            created_by=request.user if isinstance(request.user, User) else None,
         )
 
-        update_fields: list[str] = []
-        if data.title is not None and snapshot.title != data.title:
-            snapshot.title = data.title
-            update_fields.append("title")
-        if status is not None and snapshot.status != status:
-            snapshot.status = status
-            update_fields.append("status")
-        if update_fields:
-            update_fields.append("modified_at")
-            snapshot.save(update_fields=update_fields)
-
-        if tags:
-            snapshot.save_tags(tags)
-
-        try:
-            snapshot.ensure_crawl_symlink()
-        except Exception:
-            pass
+    try:
+        snapshot.ensure_crawl_symlink()
+    except Exception:
+        pass
 
     setattr(request, "with_archiveresults", False)
     return snapshot
@@ -1023,7 +1022,10 @@ def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateS
             update_fields.append("retry_at")
 
         if tags is not None:
-            snapshot.save_tags(normalize_tag_list(tags))
+            snapshot.save_tags(
+                normalize_tag_list(tags),
+                created_by=request.user if isinstance(request.user, User) else None,
+            )
 
         if payload.get("status") == Snapshot.StatusChoices.SEALED:
             snapshot.cancel()
@@ -1410,8 +1412,7 @@ def tags_add_to_snapshot(request: HttpRequest, data: TagSnapshotRequestSchema):
     else:
         raise HttpError(400, "Either tag_name or tag_id is required")
 
-    # Add the tag to the snapshot
-    snapshot.tags.add(tag.pk)
+    snapshot.add_tag_ids([tag.pk])
 
     return {
         "success": True,
@@ -1439,8 +1440,7 @@ def tags_remove_from_snapshot(request: HttpRequest, data: TagSnapshotRequestSche
     else:
         raise HttpError(400, "Either tag_name or tag_id is required")
 
-    # Remove the tag from the snapshot
-    snapshot.tags.remove(tag.pk)
+    snapshot.remove_tag_ids([tag.pk])
 
     return {
         "success": True,

archivebox/api/v1_crawls.py

@@ -1,6 +1,5 @@
 __package__ = "archivebox.api"
 
-import json
 from pathlib import Path
 from uuid import UUID
 from datetime import datetime
@@ -23,7 +22,7 @@ from archivebox.core.permissions import (
 )
 from archivebox.config.common import get_config
 from archivebox.crawls.models import Crawl
-from archivebox.misc.util import filter_queryset_by_uuid_substring
+from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url
 
 from .auth import API_AUTH_METHODS, authenticated_user_from_request
 
@@ -113,7 +112,10 @@ def get_crawls(request: HttpRequest):
 
 @router.post("/crawls", response=CrawlSchema, url_name="create_crawl")
 def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
-    urls = [url.strip() for url in data.urls if url and url.strip()]
+    try:
+        urls = [validate_url(url) for url in data.urls if url and url.strip()]
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
     if not urls:
         raise HttpError(400, "At least one URL is required")
     if data.max_depth not in (0, 1, 2, 3, 4):
@@ -121,9 +123,10 @@ def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
 
     tags = normalize_tag_list(data.tags, data.tags_str)
     config = dict(data.config or {})
+    config.setdefault("PARSER", "url_list")
     config.setdefault("PERMISSIONS", str(get_config().PERMISSIONS))
     crawl = Crawl.objects.create(
-        urls="\n".join(json.dumps({"type": "CrawlSeed", "url": url, "depth": 0}, separators=(",", ":")) for url in urls),
+        urls="\n".join(urls),
         max_depth=data.max_depth,
         tags_str=",".join(tags),
         label=data.label,

archivebox/api/v1_personas.py

@@ -2,7 +2,7 @@ __package__ = "archivebox.api"
 
 import json
 from datetime import datetime
-from typing import Any
+from typing import Any, Literal
 from uuid import UUID
 
 from django.db.models import Q
@@ -23,6 +23,7 @@ class PersonaBrowserSettingsSchema(Schema):
     user_agent: str = ""
     viewport_size: str = ""
     viewport_device_scale_factor: float | None = None
+    color_scheme: Literal["", "light", "dark"] = ""
     language: str = ""
     timezone: str = ""
     geolocation: dict[str, Any] | None = None
@@ -94,6 +95,8 @@ def browser_settings_to_config(extension_persona_id: str, settings: PersonaBrows
         )
     if settings.viewport_device_scale_factor is not None:
         config["BROWSER_DEVICE_SCALE_FACTOR"] = settings.viewport_device_scale_factor
+    if "color_scheme" in settings.model_fields_set:
+        config["BROWSER_COLOR_SCHEME"] = settings.color_scheme
     if settings.language:
         config["BROWSER_LANGUAGE"] = settings.language
     if settings.timezone:
@@ -150,12 +153,12 @@ def sync_persona(request: HttpRequest, payload: PersonaSyncSchema):
     persona.ensure_dirs()
 
     cookies_written = False
-    if payload.cookies_txt.strip():
+    if "cookies_txt" in payload.model_fields_set:
         (persona.path / "cookies.txt").write_text(payload.cookies_txt)
         cookies_written = True
 
     auth_written = False
-    if payload.auth_json:
+    if "auth_json" in payload.model_fields_set:
         (persona.path / "auth.json").write_text(json.dumps(payload.auth_json, indent=2, sort_keys=True) + "\n")
         auth_written = True
 

archivebox/cli/archivebox_add.py

@@ -6,7 +6,6 @@ __command__ = "archivebox add"
 
 import sys
 import os
-import json
 from pathlib import Path
 
 from typing import Any, TYPE_CHECKING
@@ -71,17 +70,7 @@ def add(
     created_by_id: int | None = None,
     config: dict[str, Any] | None = None,
 ) -> tuple[Crawl, QuerySet[Snapshot]]:
-    """Add a new URL or list of URLs to your archive.
-
-    The flow is:
-    1. Save URLs to sources file
-    2. Create Crawl with URLs and max_depth
-    3. Crawl runner creates Snapshots from Crawl URLs (depth=0)
-    4. Crawl runner runs parser extractors on root snapshots
-    5. Parser extractors output to urls.jsonl
-    6. URLs are added to Crawl.urls and child Snapshots are created
-    7. Repeat until max_depth is reached
-    """
+    """Add a URL list or imported URL document to a new Crawl."""
 
     from rich import print
 
@@ -125,12 +114,12 @@ def add(
     created_by_id = created_by_id or get_or_create_system_user_pk()
     started_at = timezone.now()
 
-    use_internal_input_root = isinstance(urls, str)
-    source_text = (
-        urls
-        if use_internal_input_root
-        else "\n".join(json.dumps({"type": "CrawlSeed", "url": str(url), "depth": 0}, separators=(",", ":")) for url in urls)
-    )
+    if isinstance(urls, str):
+        source_text = urls
+    else:
+        from archivebox.misc.util import validate_url
+
+        source_text = "\n".join(validate_url(str(url)) for url in urls)
 
     # 2. Create a new Crawl with inline URLs
     # Foreground add must claim runner ownership before publishing runnable
@@ -187,12 +176,7 @@ def add(
     try:
         crawl = Crawl.objects.create(
             urls=source_text,
-            # Stdin/import text gets an extra hop because the synthetic
-            # archivebox://internal root lives at depth 0 and parser-discovered
-            # URLs land at depth 1; direct URL args become the depth=0 input
-            # snapshots themselves so --depth=N matches the deepest hop the user
-            # asked for.
-            max_depth=depth + 1 if use_internal_input_root else depth,
+            max_depth=depth,
             tags_str=tag,
             persona_id=persona_obj.id,
             label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
@@ -213,9 +197,8 @@ def add(
     print(f"[green]\\[+] Created Crawl {crawl.id} with max_depth={depth}[/green]")
     print(f"    [dim]First URL: {first_url}[/dim]")
 
-    # 3. The runner will create Snapshots from all URLs after claiming the Crawl
-    #    Parser extractors run on snapshots and discover more URLs
-    #    Discovered URLs become child Snapshots (depth+1)
+    # The runner parses Crawl.urls after claiming the Crawl, then persists the
+    # resulting depth-0 Snapshot facts. Recursive discoveries start at depth 1.
 
     if index_only:
         print("[yellow]\\[*] Index-only mode - URLs queued, runner not started[/yellow]")
@@ -405,10 +388,6 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
-        if not kwargs.get("index_only"):
-            from archivebox.workers.supervisord_util import require_crawl_memory
-
-            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.
@@ -421,13 +400,13 @@ def main(**kwargs):
         if only_new is not None:
             kwargs["config"] = {"ONLY_NEW": bool(only_new)}
         if extract:
-            from abx_dl.models import discover_plugins, plugins_matching_output
+            from archivebox.plugins.discovery import get_plugin_catalog
 
-            all_plugins = discover_plugins()
+            catalog = get_plugin_catalog()
             tokens = [token.strip() for token in extract.split(",") if token.strip()]
-            plugin_names = {name.lower(): name for name in all_plugins}
+            plugin_names = {name.lower(): name for name in catalog}
             selected = [plugin_names[token.lower()] for token in tokens if token.lower() in plugin_names]
-            selected += plugins_matching_output(all_plugins, tokens)
+            selected += catalog.matching_output(tokens)
             if not selected:
                 raise click.UsageError(f"No plugins found matching extract types: {extract}")
             existing = [token.strip() for token in (kwargs.get("plugins") or "").split(",") if token.strip()]

archivebox/cli/archivebox_archiveresult.py

@@ -6,13 +6,13 @@ archivebox archiveresult <action> [args...] [--filters]
 Manage ArchiveResult records (plugin extraction results).
 
 Actions:
-    create  - Create ArchiveResults for Snapshots (queue extractions)
+    create  - Emit plugin extraction request records for Snapshots
     list    - List ArchiveResults as JSONL (with optional filters)
     update  - Update ArchiveResults from stdin JSONL
     delete  - Delete ArchiveResults from stdin JSONL
 
 Examples:
-    # Create ArchiveResults for snapshots (queue for extraction)
+    # Emit extraction requests; `archivebox run` schedules their parent snapshots
     archivebox snapshot list --status=queued | archivebox archiveresult create
     archivebox archiveresult create --plugin=screenshot --snapshot-id=<uuid>
 
@@ -20,9 +20,6 @@ Examples:
     archivebox archiveresult list --status=failed
     archivebox archiveresult list --plugin=screenshot --status=succeeded
 
-    # Update (reset failed extractions to queued)
-    archivebox archiveresult list --status=failed | archivebox archiveresult update --status=queued
-
     # Delete
     archivebox archiveresult list --plugin=singlefile | archivebox archiveresult delete --yes
 
@@ -67,7 +64,7 @@ def create_archiveresults(
     Reads Snapshot records from stdin and emits ArchiveResult request JSONL.
     Pass-through: Non-Snapshot/ArchiveResult records are output unchanged.
     If --plugin is specified, only emits requests for that plugin.
-    Otherwise, emits one request for each enabled Snapshot plugin.
+    Otherwise, emits requests for all enabled snapshot hooks.
 
     Exit codes:
         0: Success
@@ -144,19 +141,20 @@ def create_archiveresults(
 
     created_count = 0
     for snapshot in snapshots:
-        if plugin:
+        config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
+        hooks = [
+            hook
+            for hook in discover_hooks("Snapshot", filter_disabled=not plugin, config=config)
+            if not plugin or hook.parent.name == plugin
+        ]
+        for hook_path in hooks:
+            hook_name = hook_path.stem
+            plugin_name = hook_path.parent.name
             if not is_tty:
-                write_record(build_archiveresult_request(snapshot.id, plugin, status=status))
+                write_record(build_archiveresult_request(snapshot.id, plugin_name, hook_name=hook_name, status=status))
             created_count += 1
-        else:
-            config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
-            hooks = discover_hooks("Snapshot", filter_disabled=True, config=config)
-            for plugin_name in dict.fromkeys(hook_path.parent.name for hook_path in hooks):
-                if not is_tty:
-                    write_record(build_archiveresult_request(snapshot.id, plugin_name, status=status))
-                created_count += 1
 
-    rprint(f"[green]Created {created_count} archive result request records[/green]", file=sys.stderr)
+    rprint(f"[green]Created {created_count} extraction request records[/green]", file=sys.stderr)
     return 0
 
 
@@ -254,11 +252,9 @@ def update_archiveresults(
 
             # Apply updates from CLI flags
             if status:
-                if status == ArchiveResult.StatusChoices.QUEUED:
-                    result.reset_for_retry()
-                else:
-                    result.status = status
-                    result.save(update_fields=["status", "modified_at"])
+                result.status = status
+
+            result.save()
             updated_count += 1
 
             if not is_tty:
@@ -342,7 +338,7 @@ def main():
 @click.option("--plugin", "-p", help="Plugin name (e.g., screenshot, singlefile)")
 @click.option("--status", "-s", default="queued", help="Initial status (default: queued)")
 def create_cmd(snapshot_id: str | None, plugin: str | None, status: str):
-    """Create ArchiveResults for Snapshots from stdin JSONL."""
+    """Emit Snapshot plugin extraction requests as JSONL."""
     sys.exit(create_archiveresults(snapshot_id=snapshot_id, plugin=plugin, status=status))
 
 

archivebox/cli/archivebox_extract.py

@@ -1,95 +1,135 @@
 #!/usr/bin/env python3
 
-"""
-archivebox extract [snapshot_ids...] [--plugins=NAMES]
-
-Run plugins on Snapshots. Accepts snapshot IDs as arguments, from stdin, or via JSONL.
-
-Input formats:
-    - Snapshot UUIDs (one per line)
-    - JSONL: {"type": "Snapshot", "id": "...", "url": "..."}
-    - JSONL: {"type": "ArchiveResult", "snapshot_id": "...", "plugin": "..."}
-
-Output (JSONL):
-    {"type": "ArchiveResult", "id": "...", "snapshot_id": "...", "plugin": "...", "status": "..."}
-
-Examples:
-    # Extract specific snapshot
-    archivebox extract 01234567-89ab-cdef-0123-456789abcdef
-
-    # Pipe from snapshot command
-    archivebox snapshot https://example.com | archivebox extract
-
-    # Run specific plugins only
-    archivebox extract --plugins=screenshot,singlefile 01234567-89ab-cdef-0123-456789abcdef
-
-    # Chain commands
-    archivebox crawl https://example.com | archivebox snapshot | archivebox extract
-"""
+"""Run abx-dl snapshot hooks for existing ArchiveBox snapshots."""
 
 __package__ = "archivebox.cli"
 __command__ = "archivebox extract"
 
 import sys
 from collections import defaultdict
-from itertools import product
+from contextlib import redirect_stdout
 
 import rich_click as click
 
 
-def process_archiveresult_by_id(archiveresult_id: str) -> int:
-    """
-    Re-run extraction for a single ArchiveResult by ID.
+def _resolve_requests(records: list[dict], plugins: str) -> tuple[dict[str, set[str]], set[str]]:
+    """Resolve CLI records to snapshot-level execution requests.
 
-    ArchiveResults are projected status rows, not queued work items. Re-running
-    a single result means resetting that row and queueing its parent snapshot
-    through the shared crawl runner with the corresponding plugin selected.
+    ArchiveResult input is accepted as a convenient reference to its parent
+    Snapshot and plugin. It is never reset or converted into queued work.
     """
-    from rich import print as rprint
-    from archivebox.core.models import ArchiveResult
     from archivebox.api.v1_core import _uuid_ref_query
+    from archivebox.core.models import ArchiveResult, Snapshot
+    from archivebox.misc.jsonl import TYPE_ARCHIVERESULT
+
+    explicit_plugins = {name.strip() for name in plugins.split(",") if name.strip()}
+    requested: dict[str, set[str]] = defaultdict(set)
+    missing: set[str] = set()
+
+    for record in records:
+        record_type = record.get("type")
+        record_id = str(record.get("id") or "")
+
+        if record_type == TYPE_ARCHIVERESULT and record.get("snapshot_id"):
+            requested[str(record["snapshot_id"])].update(explicit_plugins or {str(record.get("plugin") or "")})
+            requested[str(record["snapshot_id"])].discard("")
+            continue
+
+        if record_type == TYPE_ARCHIVERESULT:
+            result = ArchiveResult.objects.filter(_uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
+            if result is not None:
+                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
+                continue
+
+        snapshot_id = str(record.get("snapshot_id") or record_id or "")
+        snapshot = Snapshot.objects.filter(id=snapshot_id).only("id").first() if snapshot_id else None
+        if snapshot is None and record.get("url"):
+            snapshot = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").only("id").first()
+        # Bare UUID CLI arguments are parsed as Snapshot records because the
+        # input layer cannot know which model owns them. Preserve the public
+        # convenience of passing an ArchiveResult ID by trying that reference
+        # only after the Snapshot lookup misses.
+        if snapshot is None and record_id and not record.get("url"):
+            result = ArchiveResult.objects.filter(_uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
+            if result is not None:
+                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
+                continue
+        if snapshot is None:
+            missing.add(snapshot_id or str(record.get("url") or ""))
+            continue
+        requested[str(snapshot.id)].update(explicit_plugins)
+
+    return requested, missing
+
+
+def _run_snapshot_requests(requested: dict[str, set[str]], *, wait: bool, show_progress: bool) -> int:
+    from django.utils import timezone
+    from rich import print as rprint
+
+    from archivebox.core.models import Snapshot
     from archivebox.services.runner import run_crawl
 
-    try:
-        archiveresult = ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
-    except ArchiveResult.DoesNotExist:
-        rprint(f"[red]ArchiveResult {archiveresult_id} not found[/red]", file=sys.stderr)
+    snapshots = {str(snapshot.id): snapshot for snapshot in Snapshot.objects.filter(id__in=requested).select_related("crawl")}
+    if not snapshots:
+        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
         return 1
 
-    rprint(f"[blue]Extracting {archiveresult.plugin} for {archiveresult.snapshot.url}[/blue]", file=sys.stderr)
+    from archivebox.config.common import get_config
+    from archivebox.plugins.discovery import get_enabled_plugins
 
-    try:
-        archiveresult.reset_for_retry()
-        snapshot = archiveresult.snapshot
-        snapshot.queue_for_extraction()
-        crawl = snapshot.crawl
-        if not crawl.claim_processing_lock(lock_seconds=10):
-            rprint(
-                f"[yellow]Crawl {crawl.id} is already owned by another runner[/yellow]",
-                file=sys.stderr,
-            )
-            return 1
+    for snapshot_id, plugin_names in requested.items():
+        snapshot = snapshots.get(snapshot_id)
+        if snapshot is not None and not plugin_names:
+            plugin_names.update(get_enabled_plugins(config=get_config(crawl=snapshot.crawl, snapshot=snapshot)))
 
-        run_crawl(str(snapshot.crawl_id), snapshot_ids=[str(snapshot.id)], selected_plugins=[archiveresult.plugin])
-        archiveresult.refresh_from_db()
+    if wait and any("search_backend_sonic" in plugin_names for plugin_names in requested.values()):
+        from archivebox.core.takeover_util import ensure_daemon_stack
 
-        if archiveresult.status == ArchiveResult.StatusChoices.SUCCEEDED:
-            print(f"[green]Extraction succeeded: {archiveresult.output_str}[/green]")
-            return 0
-        elif archiveresult.status == ArchiveResult.StatusChoices.NORESULTS:
-            print(f"[dim]Extraction completed with no results: {archiveresult.output_str}[/dim]")
-            return 0
-        elif archiveresult.status == ArchiveResult.StatusChoices.FAILED:
-            print(f"[red]Extraction failed: {archiveresult.output_str}[/red]", file=sys.stderr)
-            return 1
-        else:
-            # Still in progress or backoff - not a failure
-            print(f"[yellow]Extraction status: {archiveresult.status}[/yellow]")
-            return 0
+        with redirect_stdout(sys.stderr):
+            ensure_daemon_stack(reason="Sonic snapshot indexing")
 
-    except Exception as e:
-        print(f"[red]Extraction error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-        return 1
+    # Explicit extraction resumes open/paused snapshots at the snapshot level.
+    # Sealed snapshots stay sealed during targeted maintenance backfills.
+    if wait:
+        for snapshot in snapshots.values():
+            if snapshot.status != Snapshot.StatusChoices.SEALED:
+                snapshot.update_and_requeue(
+                    status=Snapshot.StatusChoices.QUEUED,
+                    retry_at=timezone.now(),
+                )
+
+    if not wait:
+        for snapshot_id, plugin_names in requested.items():
+            snapshot = snapshots.get(snapshot_id)
+            if snapshot is None:
+                continue
+            if plugin_names:
+                snapshot.schedule_plugin_run(plugin_names, when=timezone.now())
+            elif snapshot.status == Snapshot.StatusChoices.SEALED:
+                snapshot.update_and_requeue(retry_at=timezone.now())
+            else:
+                snapshot.update_and_requeue(
+                    status=Snapshot.StatusChoices.QUEUED,
+                    retry_at=timezone.now(),
+                )
+        if show_progress:
+            rprint(f"[blue]Queued {len(snapshots)} snapshots for extraction[/blue]", file=sys.stderr)
+        return 0
+
+    grouped: dict[tuple[str, tuple[str, ...]], list[str]] = defaultdict(list)
+    for snapshot_id, plugin_names in requested.items():
+        snapshot = snapshots.get(snapshot_id)
+        if snapshot is not None:
+            grouped[(str(snapshot.crawl_id), tuple(sorted(plugin_names)))].append(snapshot_id)
+
+    for (crawl_id, plugin_names), snapshot_ids in grouped.items():
+        run_crawl(
+            crawl_id,
+            snapshot_ids=sorted(snapshot_ids),
+            selected_plugins=list(plugin_names) or None,
+            show_progress=show_progress,
+        )
+    return 0
 
 
 def run_plugins(
@@ -99,356 +139,64 @@ def run_plugins(
     wait: bool = True,
     emit_results: bool = True,
     show_progress: bool = True,
-    preserve_queued: bool = False,
 ) -> int:
-    """
-    Run plugins on Snapshots from input.
-
-    Reads Snapshot IDs or JSONL from args/stdin, runs plugins, outputs JSONL.
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
+    """Execute selected plugins through the snapshot-level runner."""
     from rich import print as rprint
-    from django.utils import timezone
 
-    from archivebox.misc.jsonl import (
-        read_args_or_stdin,
-        write_record,
-        TYPE_SNAPSHOT,
-        TYPE_ARCHIVERESULT,
-    )
     from archivebox.core.models import Snapshot
-    from archivebox.core.models import ArchiveResult
-    from archivebox.services.runner import run_crawl
+    from archivebox.misc.jsonl import read_args_or_stdin, write_record
 
-    is_tty = sys.stdout.isatty()
-
-    # Parse comma-separated plugins list once (reused in creation and filtering)
-    plugins_list = [p.strip() for p in plugins.split(",") if p.strip()] if plugins else []
-
-    # Parse stdin/args exactly once per CLI invocation.
-    # `main()` may already have consumed stdin to distinguish Snapshot input from
-    # ArchiveResult IDs; if so, it must pass the parsed records through here
-    # instead of asking this helper to reread an already-drained pipe.
     if records is None:
         records = list(read_args_or_stdin(args))
-
     if not records:
         rprint("[yellow]No snapshots provided. Pass snapshot IDs as arguments or via stdin.[/yellow]", file=sys.stderr)
         return 1
 
-    # Gather snapshot IDs and optional plugin constraints to process
-    snapshot_ids = set()
-    requested_plugins_by_snapshot: dict[str, set[str]] = defaultdict(set)
-    for record in records:
-        record_type = record.get("type")
-
-        if record_type == TYPE_SNAPSHOT:
-            snapshot_id = record.get("id")
-            if snapshot_id:
-                snapshot_ids.add(str(snapshot_id))
-            elif record.get("url"):
-                # Look up by URL (get most recent if multiple exist)
-                snap = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").first()
-                if snap:
-                    snapshot_ids.add(str(snap.id))
-                else:
-                    rprint(f"[yellow]Snapshot not found for URL: {record['url']}[/yellow]", file=sys.stderr)
-
-        elif record_type == TYPE_ARCHIVERESULT:
-            snapshot_id = record.get("snapshot_id")
-            if snapshot_id:
-                snapshot_ids.add(str(snapshot_id))
-                plugin_name = record.get("plugin")
-                if plugin_name and not plugins_list:
-                    requested_plugins_by_snapshot[str(snapshot_id)].add(str(plugin_name))
-
-        elif "id" in record:
-            # Assume it's a snapshot ID
-            snapshot_ids.add(str(record["id"]))
-
-    if not snapshot_ids:
-        rprint("[red]No valid snapshot IDs found in input[/red]", file=sys.stderr)
+    requested, missing = _resolve_requests(records, plugins)
+    for value in sorted(missing):
+        rprint(f"[yellow]Snapshot or ArchiveResult not found: {value}[/yellow]", file=sys.stderr)
+    if not requested:
         return 1
 
-    existing_snapshots = list(Snapshot.objects.filter(id__in=snapshot_ids).values_list("id", "crawl_id"))
-    existing_snapshot_ids = {str(snapshot_id) for snapshot_id, _crawl_id in existing_snapshots}
-    existing_crawl_ids = {str(crawl_id) for _snapshot_id, crawl_id in existing_snapshots}
-    missing_snapshot_ids = sorted(str(snapshot_id) for snapshot_id in snapshot_ids - existing_snapshot_ids)
-    for snapshot_id in missing_snapshot_ids:
-        rprint(f"[yellow]Snapshot {snapshot_id} not found[/yellow]", file=sys.stderr)
-
-    # Queue only the target plugin rows. Bulk updates keep large reindex runs
-    # from doing one SELECT+UPDATE per snapshot/plugin before hooks even start.
-    requested_pairs: set[tuple[str, str]] = set()
-    if plugins_list:
-        requested_pairs.update((snapshot_id, plugin_name) for snapshot_id, plugin_name in product(existing_snapshot_ids, plugins_list))
-    else:
-        requested_pairs.update(
-            (snapshot_id, plugin_name)
-            for snapshot_id, plugin_names in requested_plugins_by_snapshot.items()
-            if snapshot_id in existing_snapshot_ids
-            for plugin_name in plugin_names
-        )
-    queued_rows: set[tuple[str, str]] = set()
-    if preserve_queued and requested_pairs:
-        queued_rows = {
-            (str(snapshot_id), plugin_name)
-            for snapshot_id, plugin_name in ArchiveResult.objects.filter(
-                snapshot_id__in=existing_snapshot_ids,
-                plugin__in={plugin_name for _snapshot_id, plugin_name in requested_pairs},
-                status=ArchiveResult.StatusChoices.QUEUED,
-            ).values_list("snapshot_id", "plugin")
-        }
-    rows_to_queue = requested_pairs - queued_rows
-
-    reset_fields = {
-        "status": ArchiveResult.StatusChoices.QUEUED,
-        "hook_name": "",
-        "output_str": "",
-        "output_json": None,
-        "output_files": {},
-        "output_size": 0,
-        "output_mimetypes": "",
-        "start_ts": None,
-        "end_ts": None,
-        "modified_at": timezone.now(),
-    }
-    if rows_to_queue and plugins_list:
-        rows_to_reset_by_plugin: dict[str, set[str]] = defaultdict(set)
-        for snapshot_id, plugin_name in rows_to_queue:
-            rows_to_reset_by_plugin[plugin_name].add(snapshot_id)
-        for plugin_name, plugin_snapshot_ids in rows_to_reset_by_plugin.items():
-            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name).update(
-                **reset_fields,
-            )
-    elif rows_to_queue and requested_plugins_by_snapshot:
-        snapshot_ids_by_plugin: dict[str, set[str]] = defaultdict(set)
-        for snapshot_id, plugin_name in rows_to_queue:
-            snapshot_ids_by_plugin[plugin_name].add(snapshot_id)
-        for plugin_name, plugin_snapshot_ids in snapshot_ids_by_plugin.items():
-            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name).update(
-                **reset_fields,
-            )
-    existing_rows = (
-        {
-            (str(snapshot_id), plugin_name)
-            for snapshot_id, plugin_name in ArchiveResult.objects.filter(
-                snapshot_id__in=existing_snapshot_ids,
-                plugin__in={plugin_name for _snapshot_id, plugin_name in rows_to_queue},
-            ).values_list("snapshot_id", "plugin")
-        }
-        if rows_to_queue
-        else set()
-    )
-    missing_rows = rows_to_queue - existing_rows
-    if missing_rows:
-        ArchiveResult.objects.bulk_create(
-            [
-                ArchiveResult(
-                    snapshot_id=snapshot_id,
-                    plugin=plugin_name,
-                    hook_name="",
-                    status=ArchiveResult.StatusChoices.QUEUED,
-                )
-                for snapshot_id, plugin_name in sorted(missing_rows)
-            ],
-            batch_size=500,
-        )
-
-    processed_count = len(existing_snapshot_ids)
-    queue_at = timezone.now()
-    if existing_snapshot_ids:
-        if requested_pairs:
-            # Search indexing on a sealed Snapshot is the only targeted plugin
-            # allowed to bypass the normal lifecycle. Every other requested
-            # plugin requeues its Snapshot through the unified lifecycle.
-            affected_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name in rows_to_queue}
-            if preserve_queued and queued_rows:
-                queued_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name in queued_rows}
-                affected_snapshot_ids.update(
-                    str(snapshot_id)
-                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids)
-                    .filter(retry_at__gt=queue_at)
-                    .values_list("id", flat=True)
-                )
-                affected_snapshot_ids.update(
-                    str(snapshot_id)
-                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids, retry_at__isnull=True).values_list(
-                        "id",
-                        flat=True,
-                    )
-                )
-            requested_plugins_by_id: dict[str, set[str]] = defaultdict(set)
-            for snapshot_id, plugin_name in requested_pairs:
-                requested_plugins_by_id[snapshot_id].add(plugin_name)
-            for snapshot in Snapshot.objects.filter(id__in=affected_snapshot_ids).only("id", "status", "modified_at"):
-                # Guard the read-time status so we never bump retry_at on a
-                # row that's been re-queued / started by a concurrent runner.
-                plugin_names = requested_plugins_by_id.get(str(snapshot.id), set())
-                sealed_search_backfill = (
-                    snapshot.status == Snapshot.StatusChoices.SEALED
-                    and plugin_names
-                    and all(plugin_name.startswith("search_backend_") for plugin_name in plugin_names)
-                )
-                if sealed_search_backfill:
-                    snapshot.safe_update(
-                        {"retry_at": queue_at, "modified_at": queue_at},
-                        refresh=False,
-                        extra_filter={"status": snapshot.status},
-                    )
-                else:
-                    snapshot.update_and_requeue(
-                        status=Snapshot.StatusChoices.QUEUED,
-                        retry_at=queue_at,
-                        current_step=0,
-                    )
-        else:
-            # No plugin rows were requested, so this is a full snapshot retry.
-            for snapshot in Snapshot.objects.filter(id__in=existing_snapshot_ids).only("id", "status", "retry_at", "modified_at"):
-                snapshot.safe_update(
-                    {
-                        "status": Snapshot.StatusChoices.QUEUED,
-                        "retry_at": queue_at,
-                        "current_step": 0,
-                        "modified_at": queue_at,
-                    },
-                    refresh=False,
-                    extra_filter={"status": snapshot.status},
-                )
-    if existing_crawl_ids and not requested_pairs:
-        from archivebox.crawls.models import Crawl
-
-        for crawl in Crawl.objects.filter(id__in=existing_crawl_ids).only("id", "status", "retry_at", "modified_at"):
-            update_fields = {
-                "retry_at": queue_at,
-                "modified_at": queue_at,
-            }
-            if crawl.status != Crawl.StatusChoices.STARTED:
-                update_fields["status"] = Crawl.StatusChoices.QUEUED
-            crawl.safe_update(
-                update_fields,
-                refresh=False,
-                extra_filter={"status": crawl.status},
-            )
-
-    if processed_count == 0:
-        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
-        return 1
-
-    if show_progress:
-        rprint(f"[blue]Queued {processed_count} snapshots for extraction[/blue]", file=sys.stderr)
-
-    # Run orchestrator if --wait (default)
-    if wait:
-        if show_progress:
-            rprint("[blue]Running plugins...[/blue]", file=sys.stderr)
-        snapshot_ids_by_crawl: dict[str, set[str]] = defaultdict(set)
-        for snapshot_id, crawl_id in existing_snapshots:
-            snapshot_ids_by_crawl[str(crawl_id)].add(str(snapshot_id))
-
-        for crawl_id, crawl_snapshot_ids in snapshot_ids_by_crawl.items():
-            from archivebox.crawls.models import Crawl
-
-            crawl = Crawl.objects.get(id=crawl_id)
-            if not crawl.claim_processing_lock(lock_seconds=10):
-                rprint(
-                    f"[yellow]Crawl {crawl_id} is already owned by another runner[/yellow]",
-                    file=sys.stderr,
-                )
-                return 1
-            selected_plugins = (
-                plugins_list
-                or sorted(
-                    {plugin for snapshot_id in crawl_snapshot_ids for plugin in requested_plugins_by_snapshot.get(str(snapshot_id), set())},
-                )
-                or None
-            )
-            run_crawl(
-                crawl_id,
-                snapshot_ids=sorted(crawl_snapshot_ids),
-                selected_plugins=selected_plugins,
-                show_progress=show_progress,
-            )
-
-    if not emit_results:
-        return 0
-
-    # Output results as JSONL (when piped) or human-readable (when TTY)
-    for snapshot_id in snapshot_ids:
-        try:
-            snapshot = Snapshot.objects.get(id=snapshot_id)
-            results = snapshot.archiveresult_set.all()
-            if plugins_list:
-                results = results.filter(plugin__in=plugins_list)
-
-            for result in results:
-                if is_tty:
-                    status_color = {
-                        "succeeded": "green",
-                        "failed": "red",
-                        "skipped": "yellow",
-                    }.get(result.status, "dim")
-                    rprint(
-                        f"  [{status_color}]{result.status}[/{status_color}] {result.plugin} → {result.output_str or ''}",
-                        file=sys.stderr,
-                    )
-                else:
-                    write_record(result.to_json())
-        except Snapshot.DoesNotExist:
-            continue
+    exit_code = _run_snapshot_requests(requested, wait=wait, show_progress=show_progress)
+    if exit_code or not emit_results:
+        return exit_code
 
+    is_tty = sys.stdout.isatty()
+    for snapshot in Snapshot.objects.filter(id__in=requested):
+        results = snapshot.archiveresult_set.all()
+        requested_plugins = requested[str(snapshot.id)]
+        if requested_plugins:
+            results = results.filter(plugin__in=requested_plugins)
+        for result in results:
+            if is_tty:
+                color = {"succeeded": "green", "failed": "red", "skipped": "yellow"}.get(result.status, "dim")
+                rprint(f"  [{color}]{result.status}[/{color}] {result.plugin} → {result.output_str or ''}", file=sys.stderr)
+            else:
+                write_record(result.to_json())
     return 0
 
 
-def is_archiveresult_id(value: str) -> bool:
-    """Check if value resolves to an ArchiveResult ID."""
-    from archivebox.core.models import ArchiveResult
-    from archivebox.api.v1_core import _uuid_ref_query
-
-    return ArchiveResult.objects.filter(_uuid_ref_query("id", value)).exists()
+def process_archiveresult_by_id(archiveresult_id: str) -> int:
+    """Re-run the parent Snapshot plugin referenced by an ArchiveResult."""
+    return run_plugins((), records=[{"id": archiveresult_id}], wait=True)
 
 
 @click.command()
-@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run (e.g., screenshot,singlefile)")
+@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run")
 @click.option("--wait/--no-wait", default=True, help="Wait for plugins to complete (default: wait)")
 @click.argument("args", nargs=-1)
 def main(plugins: str, wait: bool, args: tuple):
-    """Run plugins on Snapshots, or process existing ArchiveResults by ID"""
+    """Run plugins on Snapshots; ArchiveResult IDs select their parent plugin."""
     from archivebox.misc.jsonl import read_args_or_stdin
 
-    # Read all input
     records = list(read_args_or_stdin(args))
-
     if not records:
         from rich import print as rprint
 
         rprint("[yellow]No Snapshot IDs or ArchiveResult IDs provided. Pass as arguments or via stdin.[/yellow]", file=sys.stderr)
         sys.exit(1)
-
-    # Check if input looks like existing ArchiveResult IDs to process
-    all_are_archiveresult_ids = all(is_archiveresult_id(r.get("id") or r.get("url", "")) for r in records)
-
-    if all_are_archiveresult_ids:
-        # Process existing ArchiveResults by ID
-        from rich import print as rprint
-
-        exit_code = 0
-        for record in records:
-            archiveresult_id = record.get("id") or record.get("url")
-            if not isinstance(archiveresult_id, str):
-                rprint(f"[red]Invalid ArchiveResult input: {record}[/red]", file=sys.stderr)
-                exit_code = 1
-                continue
-            result = process_archiveresult_by_id(archiveresult_id)
-            if result != 0:
-                exit_code = result
-        sys.exit(exit_code)
-    else:
-        # Default behavior: run plugins on Snapshots from input
-        sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
+    sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_help.py

@@ -68,7 +68,7 @@ def help() -> None:
     [blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
 
     [grey53]# using Docker:[/grey53]
-    [blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
+    [blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 5797:5797[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
 """
         if IN_DOCKER
         else ""
@@ -109,7 +109,7 @@ def help() -> None:
     [dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
     [dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
     [dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
-    [dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue]                [grey53]# Start the Web UI / API server[/grey53]
+    [dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:5797[/blue]                [grey53]# Start the Web UI / API server[/grey53]
 """
         print(
             Panel(

archivebox/cli/archivebox_init.py

@@ -134,7 +134,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
     print("    > Skipping orphan snapshot import during init.")
     print()
     print("    [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
-    print("        archivebox update")
+    print("        archivebox update --rescan --migrate-only")
 
     print("\n[green]----------------------------------------------------------------------[/green]")
 

archivebox/cli/archivebox_install.py

@@ -17,12 +17,12 @@ def _resolve_install_targets(
     requested_names: tuple[str, ...],
 ) -> tuple[list[str], list[str]]:
     """Resolve plugin names and declared binary aliases, leaving unknown tokens raw."""
-    from abx_dl.models import discover_plugins, filter_plugins
+    from archivebox.plugins.discovery import get_plugin_catalog
 
-    plugins = discover_plugins(runtime="archivebox")
-    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in plugins}
+    catalog = get_plugin_catalog()
+    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in catalog}
     plugin_names_by_binary_alias: dict[str, set[str]] = {}
-    for plugin_name, plugin in plugins.items():
+    for plugin_name, plugin in catalog.items():
         for required_binary in plugin.config.required_binaries:
             aliases = {required_binary.name.lower()}
             template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", required_binary.name)
@@ -47,7 +47,7 @@ def _resolve_install_targets(
         else:
             raw_binary_names.append(name)
 
-    selected_plugins = filter_plugins(plugins, requested_plugins, include_providers=True) if requested_plugins else {}
+    selected_plugins = catalog.select(requested_plugins) if requested_plugins else catalog.select([])
     selected_plugin_names = {name.lower() for name in selected_plugins}
     raw_binary_names = [name for name in raw_binary_names if name.lower() not in selected_plugin_names]
     return sorted(selected_plugins), sorted(set(raw_binary_names))

archivebox/cli/archivebox_persona.py

@@ -27,11 +27,10 @@ Examples:
 __package__ = "archivebox.cli"
 __command__ = "archivebox persona"
 
-import os
 import sys
 import shutil
-import platform
 from pathlib import Path
+from dataclasses import replace
 from collections.abc import Iterable
 
 import rich_click as click
@@ -41,116 +40,6 @@ from archivebox.cli.cli_util import apply_filters
 from archivebox.personas import importers as persona_importers
 
 
-# =============================================================================
-# Browser Profile Locations
-# =============================================================================
-
-
-def get_chrome_user_data_dir() -> Path | None:
-    """Get the default Chrome user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":  # macOS
-        candidates = [
-            home / "Library" / "Application Support" / "Google" / "Chrome",
-            home / "Library" / "Application Support" / "Chromium",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "google-chrome",
-            home / ".config" / "chromium",
-            home / ".config" / "chrome",
-            home / "snap" / "chromium" / "common" / "chromium",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Google" / "Chrome" / "User Data",
-            local_app_data / "Chromium" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_brave_user_data_dir() -> Path | None:
-    """Get the default Brave user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "BraveSoftware" / "Brave-Browser" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_edge_user_data_dir() -> Path | None:
-    """Get the default Edge user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "Microsoft Edge",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "microsoft-edge",
-            home / ".config" / "microsoft-edge-beta",
-            home / ".config" / "microsoft-edge-dev",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Microsoft" / "Edge" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-BROWSER_PROFILE_FINDERS = {
-    "chrome": get_chrome_user_data_dir,
-    "chromium": get_chrome_user_data_dir,  # Same locations
-    "brave": get_brave_user_data_dir,
-    "edge": get_edge_user_data_dir,
-}
-
-CHROMIUM_BROWSERS = {"chrome", "chromium", "brave", "edge"}
-
-
-# =============================================================================
-# Cookie Extraction via CDP
-# =============================================================================
-
 # =============================================================================
 # Validation Helpers
 # =============================================================================
@@ -218,6 +107,9 @@ def create_personas(
     names: Iterable[str],
     import_from: str | None = None,
     profile: str | None = None,
+    source: str | None = None,
+    browser_binary: str | None = None,
+    permissions: str | None = None,
 ) -> int:
     """
     Create Personas from names.
@@ -239,25 +131,26 @@ def create_personas(
         rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
         return 1
 
-    # Validate import source if specified
-    source_profile_dir = None
+    import_source = None
+    if source and not import_from:
+        rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
+        return 1
     if import_from:
-        import_from = import_from.lower()
-        if import_from not in BROWSER_PROFILE_FINDERS:
-            rprint(f"[red]Unknown browser: {import_from}[/red]", file=sys.stderr)
-            rprint(f"[dim]Supported browsers: {', '.join(BROWSER_PROFILE_FINDERS.keys())}[/dim]", file=sys.stderr)
+        try:
+            if source:
+                import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
+                if import_source.browser != "persona":
+                    import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
+            elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
+                import_source = persona_importers.resolve_custom_import_source(import_from)
+            else:
+                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+            if browser_binary:
+                import_source = replace(import_source, browser_binary=browser_binary)
+        except ValueError as err:
+            rprint(f"[red]{err}[/red]", file=sys.stderr)
             return 1
 
-        source_profile_dir = BROWSER_PROFILE_FINDERS[import_from]()
-        if not source_profile_dir:
-            rprint(f"[red]Could not find {import_from} profile directory[/red]", file=sys.stderr)
-            return 1
-
-        rprint(f"[dim]Found {import_from} profile: {source_profile_dir}[/dim]", file=sys.stderr)
-
-        if profile is None and (source_profile_dir / "Default").exists():
-            profile = "Default"
-
     created_count = 0
     for name in name_list:
         name = name.strip()
@@ -270,21 +163,24 @@ def create_personas(
             rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
             continue
 
-        persona, created = Persona.objects.get_or_create(name=name)
+        persona, created = Persona.objects.get_or_create(
+            name=name,
+            defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
+        )
 
         if created:
             persona.ensure_dirs()
             created_count += 1
-            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
+
         else:
             rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
 
         cookies_file = Path(persona.path) / "cookies.txt"
 
         # Import browser profile if requested
-        if import_from in CHROMIUM_BROWSERS and source_profile_dir is not None:
+        if import_source is not None:
             try:
-                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+                rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
                 import_result = persona_importers.import_persona_from_source(
                     persona,
                     import_source,
@@ -292,20 +188,25 @@ def create_personas(
                     import_cookies=True,
                     capture_storage=False,
                 )
-            except Exception as e:
+            except (Exception, KeyboardInterrupt) as e:
+                if created:
+                    shutil.rmtree(persona.path, ignore_errors=True)
+                    persona.delete()
                 rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
                 return 1
 
             if import_result.profile_copied:
                 rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
             if import_result.cookies_imported:
-                rprint(f"[green]Extracted cookies to {cookies_file}[/green]", file=sys.stderr)
+                rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
             elif not import_result.profile_copied:
                 rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
 
             for warning in import_result.warnings:
                 rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
 
+        if created:
+            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
         if not is_tty:
             write_record(
                 {
@@ -544,11 +445,42 @@ def main():
 
 @main.command("create")
 @click.argument("names", nargs=-1)
-@click.option("--import", "import_from", help="Import profile from browser (chrome, chromium, brave, edge)")
+@click.option(
+    "--import",
+    "import_from",
+    help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
+)
+@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
+@click.option(
+    "--browser-binary",
+    type=click.Path(exists=True, dir_okay=False),
+    help="Source Chromium browser executable; required for other Chromium-based browsers",
+)
 @click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
-def create_cmd(names: tuple, import_from: str | None, profile: str | None):
+@click.option(
+    "--permissions",
+    type=click.Choice(["public", "unlisted", "private"]),
+    help="Visibility for a newly created persona (existing personas are unchanged)",
+)
+def create_cmd(
+    names: tuple,
+    import_from: str | None,
+    profile: str | None,
+    source: str | None,
+    browser_binary: str | None,
+    permissions: str | None,
+):
     """Create Personas, optionally importing from a browser profile."""
-    sys.exit(create_personas(names, import_from=import_from, profile=profile))
+    sys.exit(
+        create_personas(
+            names,
+            import_from=import_from,
+            profile=profile,
+            source=source,
+            browser_binary=browser_binary,
+            permissions=permissions,
+        ),
+    )
 
 
 @main.command("list")
@@ -560,6 +492,84 @@ def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
     sys.exit(list_personas(name=name, name__icontains=name__icontains, limit=limit))
 
 
+@main.command("open")
+@click.argument("name", default="Default")
+def open_cmd(name: str):
+    """Open a persona's browser, including its imported logins."""
+    import json
+    import os
+    import subprocess
+
+    import psutil
+
+    from abx_plugins import get_plugins_dir
+    from archivebox.config.common import get_config
+    from archivebox.personas.models import Persona
+
+    valid, reason = validate_persona_name(name)
+    if not valid:
+        raise click.ClickException(reason)
+    persona = Persona.get_or_create_named(name)
+    persona.ensure_dirs()
+    config = get_config(persona=persona)
+    if os.environ.get("DISPLAY"):
+        from archivebox.config import CONSTANTS
+
+        for process in psutil.process_iter():
+            try:
+                args = process.cmdline()
+                profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
+                if (
+                    not profile
+                    or any(arg.startswith(("--type=", "--headless")) for arg in args)
+                    or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
+                    or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
+                ):
+                    continue
+                rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
+                process.wait()
+            except (psutil.NoSuchProcess, psutil.AccessDenied):
+                continue
+    plugins_dir = Path(get_plugins_dir()).resolve()
+    env = os.environ.copy()
+    dependencies = subprocess.run(
+        [
+            str(Path(sys.executable).with_name("abxpkg")),
+            "env",
+            "--install",
+            "--json",
+            f"--lib={config.ABXPKG_LIB_DIR}",
+            f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
+        ],
+        capture_output=True,
+        text=True,
+        env=env,
+    )
+    if dependencies.returncode:
+        raise click.ClickException(dependencies.stderr.strip())
+    env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
+    env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
+    env["ACTIVE_PERSONA"] = persona.name
+    env["PERSONAS_DIR"] = str(persona.path.parent)
+    env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
+    payload = config.model_dump(mode="json")
+    payload.update(
+        ACTIVE_PERSONA=persona.name,
+        CHROME_HEADLESS=False,
+        CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
+        AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
+        COOKIES_FILE=str(persona.COOKIES_FILE or ""),
+    )
+    script = Path(__file__).parent.parent / "personas" / "open_browser.js"
+    result = subprocess.run(
+        [env.get("NODE_BINARY") or "node", str(script)],
+        input=json.dumps(payload),
+        text=True,
+        env=env,
+    )
+    sys.exit(result.returncode)
+
+
 @main.command("update")
 @click.option("--name", "-n", help="Set new name")
 def update_cmd(name: str | None):

archivebox/cli/archivebox_run.py

@@ -72,8 +72,8 @@ def process_stdin_records() -> int:
 
     Outputs JSONL of all processed records (for chaining).
 
-    Handles any record type: Crawl, Snapshot, ArchiveResult.
-    Auto-cascades: Crawl → Snapshots → ArchiveResults.
+    Handles Crawl and Snapshot work records. ArchiveResult records are accepted
+    as references to their parent Snapshot and plugin.
 
     Returns exit code (0 = success, 1 = error).
     """
@@ -90,6 +90,7 @@ def process_stdin_records() -> int:
     )
     from archivebox.base_models.models import get_or_create_system_user_pk
     from archivebox.core.models import Snapshot, ArchiveResult
+    from archivebox.api.v1_core import _uuid_ref_query
     from archivebox.crawls.models import Crawl
     from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
     from archivebox.machine.models import Binary
@@ -156,33 +157,15 @@ def process_stdin_records() -> int:
                     queued_count += 1
 
             elif record_type == TYPE_ARCHIVERESULT:
-                if record_id:
-                    # Existing archiveresult - re-queue
-                    try:
-                        archiveresult = ArchiveResult.objects.get(id=record_id)
-                    except ArchiveResult.DoesNotExist:
-                        archiveresult = None
-                else:
-                    archiveresult = None
-
-                snapshot_id = record.get("snapshot_id")
-                plugin_name = record.get("plugin")
-                snapshot = None
-                if archiveresult:
-                    if archiveresult.status in [
-                        ArchiveResult.StatusChoices.FAILED,
-                        ArchiveResult.StatusChoices.SKIPPED,
-                        ArchiveResult.StatusChoices.NORESULTS,
-                        ArchiveResult.StatusChoices.BACKOFF,
-                    ]:
-                        archiveresult.reset_for_retry()
-                    snapshot = archiveresult.snapshot
-                    plugin_name = plugin_name or archiveresult.plugin
-                elif snapshot_id:
-                    try:
-                        snapshot = Snapshot.objects.get(id=snapshot_id)
-                    except Snapshot.DoesNotExist:
-                        snapshot = None
+                snapshot_id = str(record.get("snapshot_id") or "")
+                plugin_name = str(record.get("plugin") or "")
+                archiveresult = None
+                if not snapshot_id and record_id:
+                    archiveresult = ArchiveResult.objects.filter(_uuid_ref_query("id", str(record_id))).select_related("snapshot").first()
+                    if archiveresult:
+                        snapshot_id = str(archiveresult.snapshot_id)
+                        plugin_name = plugin_name or archiveresult.plugin
+                snapshot = Snapshot.objects.filter(id=snapshot_id).first() if snapshot_id else None
 
                 if snapshot:
                     snapshot.queue_for_extraction()
@@ -190,7 +173,7 @@ def process_stdin_records() -> int:
                     snapshot_ids_by_crawl[crawl_id].add(str(snapshot.id))
                     if plugin_name:
                         plugin_names_by_crawl[crawl_id].add(str(plugin_name))
-                    output_records.append(record if not archiveresult else archiveresult.to_json())
+                    output_records.append(archiveresult.to_json() if archiveresult else record)
                     queued_count += 1
 
             elif record_type in {TYPE_BINARYREQUEST, TYPE_BINARY}:
@@ -248,7 +231,6 @@ def process_stdin_records() -> int:
                     crawl_id,
                     snapshot_ids=None if crawl_id in full_crawl_ids else sorted(snapshot_ids_by_crawl[crawl_id]),
                     selected_plugins=None if crawl_id in run_all_plugins_for_crawl else sorted(plugin_names_by_crawl[crawl_id]),
-                    selected_plugins_are_explicit=False,
                 )
     return 0
 
@@ -257,7 +239,6 @@ def run_runner(
     daemon: bool = False,
     crawl_id: str | None = None,
     maintenance_only: bool = False,
-    maintenance_batch_size: int | None = None,
 ) -> int:
     """
     Run the background runner loop.
@@ -324,7 +305,6 @@ def run_runner(
                 crawl_id=crawl_id,
                 maintenance_only=maintenance_only,
                 interactive_interrupts=interactive_interrupts,
-                **({"maintenance_batch_size": maintenance_batch_size} if maintenance_batch_size else {}),
             )
         return 0
     except KeyboardInterrupt:
@@ -349,13 +329,6 @@ def run_runner(
 @click.option("--snapshot-id", help="Run one snapshot through its crawl")
 @click.option("--binary-id", help="Run one queued binary install directly on the bus")
 @click.option("--maintenance-only", is_flag=True, help="Only process sealed Snapshot maintenance and search-index backfills")
-@click.option(
-    "--maintenance-batch-size",
-    type=int,
-    default=None,
-    hidden=True,
-    help="Limit queued maintenance snapshots claimed per scheduler tick",
-)
 @click.option("--no-stdin", is_flag=True, hidden=True, help="Run the scheduler even when stdin is not a TTY")
 def main(
     daemon: bool,
@@ -363,7 +336,6 @@ def main(
     snapshot_id: str,
     binary_id: str,
     maintenance_only: bool,
-    maintenance_batch_size: int | None,
     no_stdin: bool,
 ):
     """
@@ -388,7 +360,7 @@ def main(
                 ),
                 foreground_parent_watchdog(enabled=False),
             ):
-                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
+                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only))
         except KeyboardInterrupt:
             sys.exit(0)
 
@@ -417,17 +389,16 @@ def main(
                     daemon=False,
                     crawl_id=crawl_id,
                     maintenance_only=maintenance_only,
-                    maintenance_batch_size=maintenance_batch_size,
                 ),
             )
 
         if maintenance_only:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=True, maintenance_batch_size=maintenance_batch_size))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=True))
 
         if not no_stdin and not sys.stdin.isatty():
             sys.exit(process_stdin_records())
         else:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only))
 
 
 def run_snapshot_worker(snapshot_id: str) -> int:

archivebox/cli/archivebox_schedule.py

@@ -37,6 +37,7 @@ def schedule(
         "created_schedule_ids": [],
         "disabled_count": 0,
         "run_all_enqueued": 0,
+        "run_all_maintained": 0,
         "active_schedule_ids": [],
     }
 
@@ -57,12 +58,12 @@ def schedule(
 
         created_by_id = get_or_create_system_user_pk()
         is_update_schedule = not import_path
-        template_urls = import_path or "archivebox://update"
+        template_urls = import_path or ""
         template_label = (f"Scheduled import: {template_urls}" if import_path else "Scheduled ArchiveBox update")[:64]
         template_notes = (
             f"Created by archivebox schedule for {template_urls}"
             if import_path
-            else "Created by archivebox schedule to queue recurring archivebox://update maintenance crawls."
+            else "Created by archivebox schedule to run recurring ArchiveBox maintenance."
         )
 
         template = Crawl.objects.create(
@@ -76,7 +77,6 @@ def schedule(
             retry_at=None,
             config={
                 "DEPTH": 0 if is_update_schedule else depth,
-                "SCHEDULE_KIND": "update" if is_update_schedule else "crawl",
                 # Caller-supplied overrides (e.g. {"ONLY_NEW": False}) win over the
                 # template defaults. Anything left unset falls through to the
                 # standard config stack at crawl-resolution time.
@@ -86,6 +86,7 @@ def schedule(
         crawl_schedule = CrawlSchedule.objects.create(
             template=template,
             schedule=schedule_str,
+            config={**template.config, "SCHEDULE_KIND": "update" if is_update_schedule else "crawl"},
             is_enabled=True,
             label=template_label,
             notes=template_notes,
@@ -119,12 +120,19 @@ def schedule(
 
     if run_all:
         enqueued = 0
+        maintained = 0
         now = timezone.now()
         for scheduled_crawl in schedules:
-            scheduled_crawl.enqueue(queued_at=now)
-            enqueued += 1
+            if scheduled_crawl.dispatch(queued_at=now) is None:
+                maintained += 1
+            else:
+                enqueued += 1
         result["run_all_enqueued"] = enqueued
+        result["run_all_maintained"] = maintained
         print(f"[green]\\[*] Enqueued {enqueued} scheduled crawl(s) immediately.[/green]")
+        if maintained:
+            run_pending_crawls(maintenance_only=True)
+            print(f"[green]\\[*] Ran {maintained} scheduled maintenance update(s).[/green]")
         if enqueued:
             print(
                 "[yellow]\\[*] Start `archivebox server`, `archivebox run --daemon`, or `archivebox schedule --foreground` to process the queued crawls.[/yellow]",
@@ -171,7 +179,7 @@ def schedule(
 @click.option("--clear", is_flag=True, help="Disable all currently enabled schedules")
 @click.option("--show", is_flag=True, help="Print all currently enabled schedules")
 @click.option("--foreground", "-f", is_flag=True, help="Run the global crawl runner in the foreground (no crontab required)")
-@click.option("--run-all", is_flag=True, help="Enqueue all enabled schedules immediately and process them once")
+@click.option("--run-all", is_flag=True, help="Dispatch all enabled schedules immediately")
 @click.argument("import_path", required=False)
 @docstring(schedule.__doc__)
 def main(**kwargs):

archivebox/cli/archivebox_server.py

@@ -46,13 +46,13 @@ def _split_bind_spec(spec: str) -> tuple[str, str]:
     """Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
 
     The empty strings stand in for "not provided"; the caller fills in
-    defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
+    defaults. Bracketed IPv6 literals like ``[::1]:5797`` are handled.
     """
     spec = (spec or "").strip()
     if not spec:
         return "", ""
     if spec.startswith("["):
-        # Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
+        # Bracketed IPv6: ``[::1]`` or ``[::1]:5797``
         end = spec.find("]")
         if end == -1:
             return spec, ""  # malformed; let validator reject it
@@ -77,11 +77,11 @@ def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
     (normalized to ``127.0.0.1``). Bare hostnames are rejected because the
     bind address feeds Daphne, which has to listen on a numeric address;
     public hostnames belong in ``BASE_URL`` instead. Empty values fall back
-    to ``127.0.0.1`` / ``8000``.
+    to ``127.0.0.1`` / ``5797``.
     """
     raw_host, raw_port = _split_bind_spec(spec)
     host = raw_host.strip()
-    port = (raw_port or "").strip() or "8000"
+    port = (raw_port or "").strip() or "5797"
 
     if host == "" or host.lower() == "localhost":
         host = "127.0.0.1"
@@ -198,7 +198,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
             "[yellow]    routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
         )
         print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
+            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
         )
         if config.USES_SUBDOMAIN_ROUTING:
             print(
@@ -221,7 +221,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
             "[yellow]    a reverse proxy / ingress / public hostname, e.g.[/yellow]",
         )
         print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
+            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
         )
         print()
 
@@ -254,7 +254,7 @@ def server(
         print()
 
     # First non-empty positional arg is the bind spec; otherwise inherit from
-    # config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
+    # config (which defaults to "127.0.0.1:5797"). _parse_and_validate_bind_spec
     # hard-errors on hostnames so the rest of the server can assume a numeric
     # bind host.
     bind_spec = next((arg for arg in runserver_args if arg), "")
@@ -363,12 +363,10 @@ def server(
     command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
 
     def still_owns_runtime_stack() -> bool:
-        from django.db import connections
-
-        try:
-            return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
-        finally:
-            connections.close_all()
+        # This is polled at the log-tail cadence. Keep the autocommit
+        # connection open so SQLite can reuse its page cache between polls;
+        # reconnecting here repeatedly rereads the process table while idle.
+        return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
 
     shutdown_state = None
     try:
@@ -416,8 +414,13 @@ def server(
     except KeyboardInterrupt:
         pass
     finally:
-        if not shutdown_state or not shutdown_state.signal_name:
-            command.mark_exited()
+        from django.db import connections
+
+        try:
+            if not shutdown_state or not shutdown_state.signal_name:
+                command.mark_exited()
+        finally:
+            connections.close_all()
     print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
 
 

archivebox/cli/archivebox_snapshot.py

@@ -44,6 +44,7 @@ SNAPSHOT_LIST_CHUNK_SIZE = 5000
 
 def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
     from archivebox.config import VERSION
+    from archivebox.config.common import redact_sensitive_config
     from archivebox.core.models import SnapshotTag
 
     fields = (
@@ -53,6 +54,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
         "title",
         "bookmarked_at",
         "created_at",
+        "modified_at",
+        "config",
+        "notes",
         "timestamp",
         "depth",
         "status",
@@ -80,6 +84,9 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
                 "tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
                 "bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
                 "created_at": row["created_at"].isoformat() if row["created_at"] else None,
+                "modified_at": row["modified_at"].isoformat() if row["modified_at"] else None,
+                "config": redact_sensitive_config(row["config"]),
+                "notes": row["notes"],
                 "timestamp": row["timestamp"],
                 "depth": row["depth"],
                 "status": row["status"],

archivebox/cli/archivebox_update.py

@@ -7,6 +7,7 @@ import os
 import asyncio
 import shlex
 import time
+from itertools import islice
 
 from typing import TYPE_CHECKING, Any
 from collections.abc import Iterable
@@ -52,17 +53,20 @@ def _get_search_indexing_plugins() -> list[str]:
 def _build_filtered_snapshots_queryset(
     **kwargs,
 ):
-    from archivebox.core.models import Snapshot
     from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
 
     limit = kwargs.pop("limit", None)
+    reverse = kwargs.pop("reverse", False)
     snapshots = build_snapshot_queryset(**kwargs)
+    snapshots = snapshots.order_by(kwargs.get("sort") or "-created_at", "-id")
+    if reverse:
+        snapshots = snapshots.reverse()
     if kwargs.get("resume"):
-        snapshots = snapshots.filter(timestamp__lte=kwargs["resume"])
+        snapshots = snapshots.filter(**{"timestamp__gte" if reverse else "timestamp__lte": kwargs["resume"]})
     snapshots = snapshots.select_related("crawl")
     if limit is not None and limit > 0:
         snapshot_ids = list(snapshots.values_list("id", flat=True)[:limit])
-        snapshots = Snapshot.objects.filter(id__in=snapshot_ids).select_related("crawl")
+        snapshots = snapshots.filter(id__in=snapshot_ids)
 
     return snapshots
 
@@ -73,111 +77,75 @@ def reindex_snapshots(
     search_plugins: list[str],
     batch_size: int,
     collect_ids: bool = False,
+    reverse: bool = False,
     wait_for_turn=None,
 ) -> dict[str, Any]:
     from archivebox.cli.archivebox_extract import run_plugins
     from archivebox.core.models import ArchiveResult, Snapshot
-    from abx_dl.models import discover_plugins
+    from django.db.models import Exists, OuterRef
 
     # Search backfill is the one maintenance hook allowed to execute without
     # reopening a Snapshot. Restrict that exception to already-sealed rows;
     # every open lifecycle state remains owned by the normal runner lifecycle.
     snapshots = snapshots.filter(status=Snapshot.StatusChoices.SEALED)
 
-    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "skipped_queued": 0, "reindexed": 0, "snapshot_ids": []}
-    records: list[dict[str, str]] = []
-    plugins_by_name = discover_plugins(runtime="archivebox")
-    required_hooks_by_plugin = {
-        plugin_name: frozenset(hook.name for hook in plugins_by_name[plugin_name].filter_hooks("Snapshot"))
-        for plugin_name in search_plugins
-        if plugin_name in plugins_by_name
-    }
+    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
+    print(f"[*] Reindexing missing search indexes with: {', '.join(search_plugins)}")
 
-    total = snapshots.count()
-    print(f"[*] Reindexing {total} snapshots with search plugins: {', '.join(search_plugins)}")
-
-    def run_batch() -> None:
-        if not records:
-            return
-        if wait_for_turn:
-            wait_for_turn()
-        batch_records = list(records)
-        snapshot_ids = {record["snapshot_id"] for record in batch_records}
-        plugin_names = {record["plugin"] for record in batch_records}
-        queued_rows = {
-            (str(snapshot_id), plugin_name, hook_name)
-            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
-                snapshot_id__in=snapshot_ids,
-                plugin__in=plugin_names,
-                status=ArchiveResult.StatusChoices.QUEUED,
-            ).values_list("snapshot_id", "plugin", "hook_name")
-        }
-        records_to_queue = []
-        for record in batch_records:
-            snapshot_id = record["snapshot_id"]
-            plugin_name = record["plugin"]
-            required_hooks = required_hooks_by_plugin.get(plugin_name, frozenset())
-            if required_hooks and all((snapshot_id, plugin_name, hook_name) in queued_rows for hook_name in required_hooks):
-                stats["skipped_queued"] += 1
-                continue
-            records_to_queue.append(record)
-        if not records_to_queue:
-            print(
-                f"    [{stats['processed']}/{total}] Already queued {len(batch_records)} index jobs",
-            )
-            records.clear()
-            return
-        # `archivebox update --index-only` intentionally breaks the usual
-        # "runner discovers work" rule by inserting synthetic queued
-        # ArchiveResult rows for search backends. run_plugins() keeps this as
-        # statement-sized UPDATE/bulk_create work, then bumps Snapshot.retry_at
-        # so the orchestrator owns actual hook execution. Paused snapshots stay
-        # PAUSED; run_due_snapshot restores retry_at=MAX after targeted rows
-        # finish.
-        exit_code = run_plugins(
-            args=(),
-            records=records_to_queue,
-            wait=False,
-            emit_results=False,
-            show_progress=False,
-            preserve_queued=True,
+    completed_statuses = [ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS]
+    for plugin_name in search_plugins:
+        completed_result = ArchiveResult.objects.filter(
+            snapshot_id=OuterRef("pk"),
+            plugin=plugin_name,
+            status__in=completed_statuses,
         )
-        if exit_code != 0:
-            raise SystemExit(exit_code)
-        stats["queued"] += len(records_to_queue)
-        print(
-            f"    [{stats['processed']}/{total}] Queued {len(records_to_queue)} index jobs for orchestrator",
-        )
-        records.clear()
-
-    for snapshot in snapshots.select_related("crawl").paged_iterator(chunk_size=batch_size):
-        try:
-            stats["processed"] += 1
-
-            if _get_snapshot_crawl(snapshot) is None:
-                continue
-
+        candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False)
+        candidates = candidates.order_by("created_at", "id") if reverse else candidates.order_by("-created_at", "-id")
+        remaining = candidates.select_related(None).only("id", "timestamp", "created_at").paged_iterator(chunk_size=batch_size)
+        while True:
+            if wait_for_turn:
+                wait_for_turn()
+            batch = list(islice(remaining, batch_size))
+            if not batch:
+                break
+            records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
+            stats["processed"] += len(batch)
+            stats["requested"] += len(batch)
             if collect_ids:
-                stats["snapshot_ids"].append(str(snapshot.id))
-            for plugin_name in search_plugins:
-                records.append(
-                    {
-                        "type": "ArchiveResult",
-                        "snapshot_id": str(snapshot.id),
-                        "plugin": plugin_name,
-                    },
-                )
-                stats["requested"] += 1
-            if len(records) >= batch_size:
-                run_batch()
-        except KeyboardInterrupt as err:
-            err.archivebox_resume = snapshot.timestamp
-            raise
-
-    run_batch()
+                stats["snapshot_ids"].extend(str(snapshot.id) for snapshot in batch)
+            exit_code = run_plugins(
+                args=(),
+                records=records,
+                plugins=plugin_name,
+                wait=True,
+                emit_results=False,
+                show_progress=False,
+            )
+            if exit_code != 0:
+                raise SystemExit(exit_code)
+            stats["reindexed"] += len(batch)
+            print(f"    [{plugin_name}] indexed {stats['reindexed']} missing snapshots")
     return stats
 
 
+def run_scheduled_maintenance(*, batch_size: int = 500) -> dict[str, Any]:
+    """Queue stale filesystem rows and backfill missing search facts."""
+    from archivebox.core.models import Snapshot
+
+    filesystem_stats = process_all_db_snapshots(batch_size=batch_size)
+    search_plugins = _get_search_indexing_plugins()
+    search_stats = (
+        reindex_snapshots(
+            Snapshot.objects.filter(fs_version=Snapshot._fs_current_version()),
+            search_plugins=search_plugins,
+            batch_size=batch_size,
+        )
+        if search_plugins
+        else {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
+    )
+    return {"filesystem": filesystem_stats, "search": search_stats}
+
+
 @enforce_types
 def update(
     filter_patterns: Iterable[str] = (),
@@ -197,18 +165,18 @@ def update(
     continuous: bool = False,
     index_only: bool = False,
     migrate_only: bool = False,
+    rescan: bool = False,
+    reverse: bool = False,
     stop_daemon_stack: bool = True,
 ) -> None:
     """
-    Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
+    Update the database without forcing a filesystem scan by default.
 
-    Three-phase operation (without filters):
-    - Phase 1: Drain old archive/ dirs by moving to new fs location (0.8.x → 0.9.x)
-    - Phase 2: O(n) scan over entire DB from most recent to least recent
-    - No orphan scans needed (trust 1:1 mapping between DB and filesystem after phase 1)
-
-    With filters: Only phase 2 (DB query), no filesystem operations.
-    Without filters: All phases (full update).
+    --migrate-only: eagerly finish pending filesystem migrations for known rows.
+    --index-only: reconcile known snapshot assets/metadata and backfill search.
+    --rescan: discover filesystem orphans first, then repair known snapshots.
+    Explicit scans default to newest first; --reverse selects oldest first.
+    Filters select existing database snapshots for targeted updates.
     """
 
     from rich import print
@@ -232,7 +200,6 @@ def update(
     from archivebox.core.takeover_util import (
         command_owns_foreground_runner,
         current_command,
-        foreground_runner_owner,
         standby_until_foreground_runner_needed,
     )
     from archivebox.workers.supervisord_util import run_runner_worker, stop_own_supervisord_process
@@ -285,17 +252,28 @@ def update(
     exit_code = 0
 
     try:
+        if rescan and (is_filtered_update or resume or continuous):
+            raise click.UsageError("--rescan cannot be combined with filters, --resume, or --continuous; rerun it to resume automatically.")
+        if rescan:
+            migrate_only = True  # Discovery must not schedule new captures.
         wait_for_turn()
 
         with foreground_shutdown_signals(), foreground_parent_watchdog():
             while True:
-                do_migrate = migrate_only or not index_only
-                do_index = index_only or not migrate_only
-                do_run_until_idle = do_migrate or do_index
-                ran_post_migrate_runner = False
-                full_update_empty = False
-                maintenance_work_queued = False
-                runner_work_queued = False
+                if rescan:
+                    from archivebox.cli.rescan import rescan_snapshots
+
+                    rescan_snapshots(
+                        wait_for_turn=wait_for_turn,
+                        reverse=reverse,
+                        scan_legacy=lambda: drain_old_archive_dirs(batch_size=batch_size, reverse=reverse),
+                    )
+                do_migrate = migrate_only or is_filtered_update and not index_only
+                do_index = index_only
+                if not do_migrate and not do_index:
+                    print(
+                        "[*] Collection ready. Filesystem migrations remain lazy; use --migrate-only, --index-only, or --rescan for explicit scans.",
+                    )
 
                 if do_migrate:
                     if (
@@ -327,178 +305,86 @@ def update(
                             after=after,
                             resume=resume,
                             batch_size=batch_size,
-                            queue_for_archiving=do_run_until_idle,
+                            queue_for_archiving=not migrate_only,
+                            reverse=reverse,
                             wait_for_turn=wait_for_turn,
                         )
                         print_stats(stats)
                         touched_snapshot_ids.update(stats.get("snapshot_ids", []))
-                        maintenance_work_queued = stats.get("queued", 0) > 0
-                        runner_work_queued = runner_work_queued or maintenance_work_queued
                     else:
                         stats_combined = {"phase1": {}, "phase2": {}}
 
-                        print("[*] Phase 1: Draining old archive/ directories (0.8.x → 0.9.x migration)...")
-                        stats_combined["phase1"] = drain_old_archive_dirs(
-                            resume_from=resume,
-                            batch_size=batch_size,
-                        )
-
-                        print("[*] Phase 2: Processing all database snapshots (most recent first)...")
+                        print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
                         stats_combined["phase2"] = process_all_db_snapshots(
                             batch_size=batch_size,
                             resume=resume,
                             wait_for_turn=wait_for_turn,
+                            reverse=reverse,
+                            migrate=True,
                         )
                         print_combined_stats(stats_combined)
-                        full_update_empty = (
-                            stats_combined["phase1"].get("processed", 0) == 0 and stats_combined["phase2"].get("snapshots", 0) == 0
-                        )
-                        maintenance_work_queued = any(
-                            (
-                                stats_combined["phase1"].get("queued", 0),
-                                stats_combined["phase2"].get("queued", 0),
-                                stats_combined["phase2"].get("crawls_sealed", 0),
-                            ),
-                        )
-                        runner_work_queued = runner_work_queued or maintenance_work_queued
-
-                    if do_run_until_idle:
-                        # Filesystem migration is maintenance on existing
-                        # Snapshot rows: Snapshot.save() moves archive/<ts> to
-                        # the current output_dir and preserves the lifecycle
-                        # status. Drain those retry_at ticks before queuing
-                        # search backfill below. Otherwise the sealed search
-                        # runner branch correctly sees queued ArchiveResult
-                        # rows first, runs the targeted plugins, and may leave
-                        # the fs_version maintenance tick hidden behind that
-                        # plugin work until another update pass.
-                        if full_update_empty:
-                            print("[*] No snapshots or legacy archive directories found; skipping filesystem maintenance runner.")
-                        elif not maintenance_work_queued:
-                            print("[*] No filesystem maintenance work queued; skipping filesystem maintenance runner.")
-                        else:
-                            print("[*] Phase 3: Running filesystem maintenance until idle...")
-                        if full_update_empty:
-                            pass
-                        elif not maintenance_work_queued:
-                            pass
-                        elif is_filtered_update:
-                            if not touched_snapshot_ids:
-                                print("[*] No matching snapshots queued work for the runner.")
-                            for snapshot_id in sorted(touched_snapshot_ids):
-                                run_scoped_runner("--snapshot-id", snapshot_id)
-                        else:
-                            run_scoped_runner("--maintenance-only", "--maintenance-batch-size", str(batch_size))
-                        ran_post_migrate_runner = True
+                    # The due selectors are indexed and cheap when empty, so
+                    # always drain them instead of preceding the runner with
+                    # whole-table counts merely to decide whether to call it.
+                    print("[*] Phase 3: Running filesystem maintenance until idle...")
+                    if is_filtered_update:
+                        for snapshot_id in sorted(touched_snapshot_ids, reverse=not reverse):
+                            run_scoped_runner("--snapshot-id", snapshot_id)
+                    elif migrate_only:
+                        run_scoped_runner("--maintenance-only")
+                    else:
+                        run_scoped_runner()
 
                 if do_index:
-                    if full_update_empty:
-                        print("[*] No snapshots found; skipping search indexing backfill.")
-                    else:
-                        search_plugins = _get_search_indexing_plugins()
-                        if not search_plugins:
-                            print("[*] No search indexing plugins are available, nothing to backfill.")
-                        else:
-                            snapshots = _build_filtered_snapshots_queryset(
-                                filter_patterns=filter_patterns,
-                                filter_type=filter_type,
-                                status=status,
-                                url__icontains=url__icontains,
-                                url__istartswith=url__istartswith,
-                                tag=tag,
-                                crawl_id=crawl_id,
-                                limit=limit,
-                                sort=sort,
-                                search=search,
-                                before=before,
-                                after=after,
-                                resume=resume,
-                            )
-                            from django.db.models import Exists, OuterRef, Q
-                            from django.utils import timezone
-                            from archivebox.core.models import ArchiveResult, Snapshot
+                    from archivebox.cli.rescan import reconcile_known_snapshots
 
-                            scoped_snapshot_ids = snapshots.order_by().values("id") if is_filtered_update else None
-                            queued_index_results = ArchiveResult.objects.filter(
-                                status=ArchiveResult.StatusChoices.QUEUED,
-                                plugin__in=search_plugins,
-                            )
-                            if scoped_snapshot_ids is not None:
-                                queued_index_results = queued_index_results.filter(snapshot_id__in=scoped_snapshot_ids)
-
-                            if queued_index_results.exists():
-                                runner_work_queued = True
-                                now = timezone.now()
-                                queued_result_for_snapshot = queued_index_results.filter(snapshot_id=OuterRef("pk"))
-                                snapshots_to_wake = (
-                                    Snapshot.objects.filter(status=Snapshot.StatusChoices.SEALED)
-                                    .annotate(
-                                        has_queued_index_result=Exists(queued_result_for_snapshot),
-                                    )
-                                    .filter(
-                                        has_queued_index_result=True,
-                                    )
-                                    .filter(
-                                        Q(retry_at__isnull=True) | Q(retry_at__gt=now),
-                                    )
-                                )
-                                if scoped_snapshot_ids is not None:
-                                    snapshots_to_wake = snapshots_to_wake.filter(id__in=scoped_snapshot_ids)
-                                woken_count = snapshots_to_wake.update(
-                                    retry_at=now,
-                                    modified_at=now,
-                                )
-                                print(
-                                    "[*] Existing queued search index jobs found; "
-                                    f"skipping backfill scan and waking {woken_count} snapshot(s) for the runner.",
-                                )
-                            else:
-                                collect_index_ids = (
-                                    is_filtered_update
-                                    or foreground_runner_owner(
-                                        data_dir=CONSTANTS.DATA_DIR,
-                                        exclude_id=command.id,
-                                    )
-                                    is not None
-                                )
-                                stats = reindex_snapshots(
-                                    snapshots,
-                                    search_plugins=search_plugins,
-                                    batch_size=batch_size,
-                                    collect_ids=collect_index_ids,
-                                    wait_for_turn=wait_for_turn,
-                                )
-                                print_index_stats(stats)
-                                touched_snapshot_ids.update(stats.get("snapshot_ids", []))
-                                runner_work_queued = runner_work_queued or stats["queued"] > 0
-
-                if do_run_until_idle and (do_index or not ran_post_migrate_runner):
-                    # Search/index backfill intentionally queues targeted
-                    # ArchiveResult rows without reopening sealed snapshots.
-                    # This second runner pass drains those plugin
-                    # rows after filesystem maintenance has had its own turn.
-                    # For a normal unfiltered `archivebox update`, keep the
-                    # historical final pass broad enough to resume genuinely
-                    # queued/interrupted crawl work after maintenance is done.
-                    if full_update_empty:
-                        print("[*] No snapshots found; skipping queued/interrupted crawl runner.")
-                    elif not runner_work_queued:
-                        print("[*] No queued/interrupted crawl work found; skipping queued/interrupted crawl runner.")
+                    snapshots = _build_filtered_snapshots_queryset(
+                        filter_patterns=filter_patterns,
+                        filter_type=filter_type,
+                        status=status,
+                        url__icontains=url__icontains,
+                        url__istartswith=url__istartswith,
+                        tag=tag,
+                        crawl_id=crawl_id,
+                        limit=limit,
+                        sort=sort,
+                        search=search,
+                        before=before,
+                        after=after,
+                        resume=resume,
+                        reverse=reverse,
+                    )
+                    reconcile_known_snapshots(snapshots, wait_for_turn=wait_for_turn)
+                    search_plugins = _get_search_indexing_plugins()
+                    if not search_plugins:
+                        print("[*] No search indexing plugins are available, nothing to backfill.")
                     else:
-                        print("[*] Phase 3: Running queued/interrupted crawl work until idle...")
-                    if full_update_empty:
-                        pass
-                    elif not runner_work_queued:
-                        pass
-                    elif touched_snapshot_ids and is_filtered_update:
-                        if not touched_snapshot_ids:
-                            print("[*] No matching snapshots queued work for the runner.")
-                        for snapshot_id in sorted(touched_snapshot_ids):
-                            run_scoped_runner("--snapshot-id", snapshot_id)
-                    else:
-                        run_scoped_runner(
-                            *(["--maintenance-only", "--maintenance-batch-size", str(batch_size)] if index_only or migrate_only else []),
+                        snapshots = _build_filtered_snapshots_queryset(
+                            filter_patterns=filter_patterns,
+                            filter_type=filter_type,
+                            status=status,
+                            url__icontains=url__icontains,
+                            url__istartswith=url__istartswith,
+                            tag=tag,
+                            crawl_id=crawl_id,
+                            limit=limit,
+                            sort=sort,
+                            search=search,
+                            before=before,
+                            after=after,
+                            resume=resume,
+                            reverse=reverse,
                         )
+                        stats = reindex_snapshots(
+                            snapshots,
+                            search_plugins=search_plugins,
+                            batch_size=batch_size,
+                            collect_ids=is_filtered_update,
+                            reverse=reverse,
+                            wait_for_turn=wait_for_turn,
+                        )
+                        print_index_stats(stats)
+                        touched_snapshot_ids.update(stats.get("snapshot_ids", []))
 
                 if not continuous:
                     break
@@ -510,6 +396,10 @@ def update(
         exit_code = 130
         exact_resume = err.__dict__.get("archivebox_resume")
         resume_cmd = ["archivebox", "update"]
+        if rescan:
+            resume_cmd.append("--rescan")
+        if reverse:
+            resume_cmd.append("--reverse")
         if migrate_only:
             resume_cmd.append("--migrate-only")
         if index_only:
@@ -557,7 +447,7 @@ def update(
             stop_own_supervisord_process()
 
 
-def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500) -> dict[str, int]:
+def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500, reverse: bool = False) -> dict[str, int]:
     """
     Drain old archive/ directories (0.8.x → 0.9.x migration).
 
@@ -623,18 +513,18 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
 
     # Scan real legacy directories only; these still contain data to migrate.
     entries = [
-        (e.stat().st_mtime, e.path)
+        (float(e.name), e.path)
         for e in all_entries
         if e.is_dir(follow_symlinks=False) and Snapshot.is_legacy_archive_dir(Path(e.path))  # Skip symlinks and 0.9.x roots
     ]
-    entries.sort(reverse=True)  # Newest first
+    entries.sort(reverse=not reverse)  # Newest first
     print(f"[*] Found {len(entries)} old directories to drain")
 
     for mtime, entry_path in entries:
         entry_path = Path(entry_path)
 
         # Resume from timestamp if specified
-        if resume_from and entry_path.name > resume_from:
+        if resume_from and (entry_path.name < resume_from if reverse else entry_path.name > resume_from):
             continue
 
         stats["processed"] += 1
@@ -736,19 +626,15 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
     return stats
 
 
-def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
-    """
-    O(n) scan over entire DB from most recent to least recent.
-
-    For each snapshot:
-      1. Reconcile index.json with DB (merge titles, tags, archive results)
-      2. Mark migrated snapshots sealed unless explicitly re-queued elsewhere
-
-    No orphan detection needed - we trust 1:1 mapping between DB and filesystem
-    after Phase 1 has drained all old archive/ directories.
-    """
+def process_all_db_snapshots(
+    batch_size: int = 500,
+    resume: str | None = None,
+    wait_for_turn=None,
+    reverse: bool = False,
+    migrate: bool = False,
+) -> dict[str, int]:
+    """Queue only snapshots whose indexed filesystem version is stale."""
     from archivebox.core.models import Snapshot
-    from archivebox.crawls.models import Crawl
     from django.db.models import Q
     from django.utils import timezone
 
@@ -761,128 +647,41 @@ def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, w
         "sealed": 0,
         "crawls_sealed": 0,
     }
-    current_fs_version = Snapshot._fs_current_version()
-
-    queryset = Snapshot.objects.all()
+    queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
     if resume:
-        queryset = queryset.filter(timestamp__lte=resume)
-    total = queryset.count()
-    stats["snapshots"] = total
-    print(f"[*] Processing {total} snapshots from database (most recent first)...")
-
-    def update_in_batches(rows, *, label: str, **updates) -> int:
-        updated = 0
-        checked = 0
-        while True:
-            if wait_for_turn:
-                wait_for_turn()
-            batch = list(rows.only("id", "modified_at").order_by("-timestamp")[:batch_size])
-            if not batch:
-                if updated:
-                    print(f"    [{label}] complete: {updated} rows updated")
-                return updated
-            checked += len(batch)
-            print(f"    [{label}] updating next {len(batch)} rows (seen {checked})...")
-            for snapshot in batch:
-                # This maintenance scan intentionally bypasses save(); it is
-                # only normalizing scheduler fields, and Snapshot.save() may
-                # do filesystem migration work that belongs in the runner.
-                # Guard each single-row UPDATE with modified_at so stale scan
-                # pages cannot overwrite newer runner/admin writes.
-                updated += int(
-                    snapshot.safe_update(
-                        updates,
-                        refresh=False,
-                        extra_filter={"modified_at": snapshot.modified_at},
-                    ),
-                )
-            print(f"    [{label}] updated {updated} rows so far")
-
-    now = timezone.now()
-    updated_rows = update_in_batches(
-        queryset.exclude(
-            status__in=[
-                Snapshot.StatusChoices.QUEUED,
-                Snapshot.StatusChoices.STARTED,
-                Snapshot.StatusChoices.PAUSED,
-                Snapshot.StatusChoices.SEALED,
-            ],
-        ),
-        label="snapshot status normalization",
-        status=Snapshot.StatusChoices.SEALED,
-        retry_at=None,
-        modified_at=now,
-    )
-    stats["sealed"] += updated_rows
-    stats["updated_db"] += updated_rows
-
-    fs_version_rows = queryset.exclude(fs_version=current_fs_version).filter(Q(retry_at__isnull=True) | Q(retry_at__gt=now))
-    stale_batch = []
-
-    def queue_stale_fs_batch() -> None:
-        if not stale_batch:
-            return
+        queryset = queryset.filter(**{"timestamp__gte" if reverse else "timestamp__lte": resume})
+    initial_now = timezone.now()
+    rows_to_wake = queryset if migrate else queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
+    ordered = rows_to_wake.order_by("created_at", "id") if reverse else rows_to_wake.order_by("-created_at", "-id")
+    remaining = ordered.select_related("crawl__created_by").paged_iterator(chunk_size=batch_size)
+    while True:
         if wait_for_turn:
             wait_for_turn()
+        batch = list(islice(remaining, batch_size))
+        if not batch:
+            break
         now = timezone.now()
-        # Do not bump fs_version here. The orchestrator calls Snapshot.save(),
-        # which performs the idempotent filesystem migration and commits the new
-        # fs_version in the same serialized worker path as normal crawls.
         updated = 0
-        for snapshot in stale_batch:
-            # Each row gets its own short autocommit UPDATE because this scan
-            # can touch millions of snapshots while a server is also alive.
-            # The modified_at predicate is the CAS guard: if the runner or
-            # admin changed the snapshot after paged_iterator read it, skip it
-            # and let the newer state decide whether migration is still due.
+        for snapshot in batch:
+            if migrate:
+                from archivebox.services.runner import run_snapshot_maintenance
+
+                if wait_for_turn:
+                    wait_for_turn()
+                run_snapshot_maintenance(str(snapshot.id))
+                continue
             updated += int(
                 snapshot.safe_update(
-                    {
-                        "retry_at": now,
-                        "modified_at": now,
-                    },
+                    {"retry_at": now, "modified_at": now},
                     refresh=False,
                     extra_filter={"fs_version": snapshot.fs_version},
                 ),
             )
-        stats["processed"] += len(stale_batch)
+        stats["processed"] += len(batch)
         stats["updated_db"] += updated
         stats["queued"] += updated
-        print(f"    [{stats['processed']}/{total}] Queued {updated} filesystem migrations for orchestrator...")
-        stale_batch.clear()
-
-    for snapshot in (
-        fs_version_rows.only("id", "crawl_id", "timestamp", "fs_version", "modified_at")
-        .order_by("-timestamp")
-        .paged_iterator(chunk_size=batch_size)
-    ):
-        try:
-            stale_batch.append(snapshot)
-            if len(stale_batch) >= batch_size:
-                queue_stale_fs_batch()
-        except KeyboardInterrupt as err:
-            err.archivebox_resume = snapshot.timestamp
-            raise
-    queue_stale_fs_batch()
-
-    now = timezone.now()
-    # Crawls with no open child snapshots are already finished. Seal them here
-    # instead of waking the foreground runner; otherwise migration/update can
-    # accidentally re-enter full crawl execution for historical rows.
-    stats["crawls_sealed"] = (
-        Crawl.objects.filter(
-            status__in=Crawl.RUNNABLE_STATES,
-        )
-        .exclude(
-            snapshot_set__status__in=Snapshot.OPEN_STATES,
-        )
-        .update(
-            status=Crawl.StatusChoices.SEALED,
-            retry_at=None,
-            modified_at=now,
-        )
-    )
-    stats["updated_db"] += stats["crawls_sealed"]
+        print(f"    Queued {stats['queued']} stale filesystem snapshots so far...")
+    stats["snapshots"] = stats["processed"]
     return stats
 
 
@@ -902,6 +701,7 @@ def process_filtered_snapshots(
     resume: str | None,
     batch_size: int,
     queue_for_archiving: bool = True,
+    reverse: bool = False,
     wait_for_turn=None,
 ) -> dict[str, Any]:
     """Process snapshots matching filters (DB query only)."""
@@ -924,6 +724,7 @@ def process_filtered_snapshots(
         before=before,
         after=after,
         resume=resume,
+        reverse=reverse,
     )
 
     total = snapshots.count()
@@ -942,8 +743,10 @@ def process_filtered_snapshots(
             stats["snapshot_ids"].append(str(snapshot.id))
             update_values = {}
             updated = 0
-            if not isinstance(snapshot.current_step, int):
-                update_values["current_step"] = 0
+            if not queue_for_archiving and snapshot.fs_migration_needed:
+                from archivebox.services.runner import run_snapshot_maintenance
+
+                run_snapshot_maintenance(str(snapshot.id))
             if queue_for_archiving:
                 update_values.update(
                     {
@@ -1029,10 +832,9 @@ def print_index_stats(stats: dict[str, Any]) -> None:
 
     print(f"""
 [green]Search Reindex Complete[/green]
-  Scanned rows:      {stats["processed"]}
-  Requested jobs:    {stats.get("requested", stats["queued"])}
-  Queued index jobs: {stats["queued"]}
-  Already queued:    {stats.get("skipped_queued", 0)}
+  Missing index runs: {stats["processed"]}
+  Requested runs:    {stats.get("requested", 0)}
+  Indexed snapshots: {stats.get("reindexed", 0)}
 """)
 
 
@@ -1042,6 +844,12 @@ def print_index_stats(stats: dict[str, Any]) -> None:
 @click.option("--continuous", is_flag=True, help="Run continuously as background worker")
 @click.option("--index-only", is_flag=True, help="Backfill available search indexes from existing archived content")
 @click.option("--migrate-only", is_flag=True, help="Only migrate filesystem and update database/index state")
+@click.option("--reverse", is_flag=True, help="Process oldest first instead of newest first for explicit scans")
+@click.option(
+    "--rescan",
+    is_flag=True,
+    help="Scan all snapshot directories for orphans and metadata repairs (slow, opt-in, safe to restart)",
+)
 @snapshot_filter_options(default_filter_type="exact")
 @docstring(update.__doc__)
 def main(**kwargs):

archivebox/cli/archivebox_version.py

@@ -265,15 +265,14 @@ def version(
     seen_failures: set[str] = set()
     seen_rows: set[tuple[str, str, str, str]] = set()
 
-    from archivebox.plugins.discovery import get_enabled_plugins
+    from archivebox.plugins.discovery import get_enabled_plugins, get_plugin_catalog
     from abx_dl.config import get_required_binary_requests
     from abx_dl.dependencies import resolve_binary_requests
-    from abx_dl.models import discover_plugins, filter_plugins
     from abx_dl.orchestrator import create_bus
     from abxpkg.binary_service import BinaryEvent, BinaryService
 
-    plugins = discover_plugins(runtime="archivebox")
-    enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
+    plugins = get_plugin_catalog()
+    enabled_plugins = plugins.select(get_enabled_plugins(config=config))
     enabled_plugin_names = set(enabled_plugins)
     runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
     derived_config: dict[str, object] = {}

archivebox/cli/rescan.py

@@ -0,0 +1,222 @@
+"""Explicit discovery of portable snapshots; no checkpoint or default tree walk."""
+
+import json
+import os
+import uuid
+from itertools import groupby
+from pathlib import Path
+
+from django.contrib.auth import get_user_model
+from django.core.exceptions import ValidationError
+from django.db import IntegrityError
+from rich import print
+
+from archivebox.config import CONSTANTS
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.crawls.models import Crawl
+from archivebox.misc.util import parse_date
+
+
+def _directories(parent: Path):
+    # Names encode dates / UUIDv7 creation time. Newest first within each
+    # subtree, with bounded memory and no traversal into plugin payloads.
+    with os.scandir(parent) as entries:
+        paths = [Path(entry.path) for entry in entries if entry.is_dir(follow_symlinks=False)]
+    yield from sorted(paths, key=lambda path: path.name, reverse=True)
+
+
+def _snapshot_directories(reverse=False):
+    dates = []
+    for user in _directories(CONSTANTS.USERS_DIR):
+        root = user / CONSTANTS.SNAPSHOTS_DIR_NAME
+        if root.is_symlink() or not root.is_dir():
+            continue
+        dates.extend(_directories(root))
+    dates.sort(key=lambda path: path.name, reverse=not reverse)
+    for _, date_group in groupby(dates, key=lambda path: path.name):
+        directories = [directory for date in date_group for domain in _directories(date) for directory in _directories(domain)]
+        yield from sorted(directories, key=lambda path: path.name, reverse=not reverse)
+
+
+def _fields(record, names):
+    return {name: record[name] for name in names.split() if name in record}
+
+
+def _dates(record):
+    return {name: parse_date(record[name]) for name in ("created_at", "modified_at") if record.get(name)}
+
+
+def _restore_dates(obj, record):
+    dates = _dates(record)
+    if dates:
+        type(obj).objects.filter(pk=obj.pk).update(**dates)
+        for name, value in dates.items():
+            setattr(obj, name, value)
+
+
+def _reconcile_directory(directory: Path) -> str:
+    snapshot_id = uuid.UUID(directory.name)
+    snapshot = Snapshot.objects.select_related("crawl__created_by").filter(pk=snapshot_id).first()
+    index = directory / CONSTANTS.JSONL_INDEX_FILENAME
+    if index.is_symlink():
+        raise ValueError("metadata must not be a symlink")
+    if not index.exists():
+        if snapshot is None or Path(snapshot.output_dir) != directory or snapshot.status != "sealed":
+            raise ValueError("missing index.jsonl; cannot reconstruct an unknown or active snapshot")
+        snapshot.write_index_jsonl(output_dir=directory)
+        snapshot.reconcile_filesystem_links()
+        return "Repaired"
+
+    records = [json.loads(line) for line in index.read_text().splitlines() if line.strip()]
+    if any(not isinstance(record, dict) for record in records):
+        raise ValueError("metadata records must be JSON objects")
+    snapshots = [record for record in records if record.get("type") == "Snapshot"]
+    if len(snapshots) != 1:
+        raise ValueError("expected exactly one Snapshot record")
+    record = snapshots[0]
+    if uuid.UUID(record["id"]) != snapshot_id:
+        raise ValueError("snapshot ID conflicts with directory name")
+    if record.get("status") != "sealed":
+        raise ValueError("snapshot is not sealed; finish the source capture first")
+    if record.get("fs_version") not in {Snapshot._fs_current_version(), *Snapshot._FS_VERSION_MIGRATION_PATHS}:
+        raise ValueError("unsupported filesystem version")
+    crawl_id = uuid.UUID(record["crawl_id"])
+    owner = directory.parents[3].name
+    crawl_records = [item for item in records if item.get("type") == "Crawl" and uuid.UUID(item["id"]) == crawl_id]
+    crawl_record = crawl_records[0] if crawl_records else {}
+    if crawl_record.get("created_by", owner) != owner:
+        raise ValueError("crawl owner conflicts with directory path")
+    crawl = Crawl.objects.select_related("created_by").filter(pk=crawl_id).first()
+    imported_crawl = crawl is None
+    if crawl and crawl.created_by.username != owner:
+        raise ValueError("existing crawl ID belongs to a different owner")
+    if snapshot and (snapshot.url != record["url"] or snapshot.crawl_id != crawl_id or snapshot.timestamp != record["timestamp"]):
+        raise ValueError("existing snapshot ID has conflicting identity; refusing to overwrite")
+    if snapshot and Path(snapshot.output_dir) != directory:
+        raise ValueError("existing snapshot has a different storage path")
+
+    # Validate identities before inserting anything. Never turn an ID collision
+    # into a new ID, and never overwrite a newer destination row.
+    # write_index_jsonl retains historical hook records before appending the
+    # canonical DB rows. The final record for each hook is authoritative.
+    results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
+    pending = []
+    for result in results:
+        result_id = uuid.UUID(result["id"])
+        if uuid.UUID(result["snapshot_id"]) != snapshot_id:
+            raise ValueError("ArchiveResult belongs to another snapshot")
+        existing = ArchiveResult.objects.filter(pk=result_id).first()
+        identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
+        if existing:
+            if (existing.snapshot_id, existing.plugin, existing.hook_name) != identity:
+                raise ValueError("existing ArchiveResult ID has conflicting identity")
+        elif ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin=identity[1], hook_name=identity[2]).exists():
+            raise ValueError("existing hook has a different ArchiveResult ID")
+        else:
+            pending.append(result)
+
+    imported = snapshot is None
+    if imported:
+        if Snapshot.objects.filter(timestamp=record["timestamp"]).exists():
+            raise ValueError("timestamp belongs to another snapshot")
+        if Snapshot.objects.filter(crawl_id=crawl_id, url=record["url"]).exists():
+            raise ValueError("crawl URL belongs to another snapshot")
+        user, _ = get_user_model().objects.get_or_create(
+            username=owner,
+            defaults={"is_active": False, "password": "!"},
+        )
+        if crawl is None:
+            # Old exports contain only crawl_id. Keep that identity but report
+            # that absent crawl metadata cannot be reconstructed exactly.
+            if not crawl_record:
+                print(f"[yellow]    {directory}: no Crawl record; preserving ID with recovered metadata[/yellow]")
+            crawl = Crawl(
+                id=crawl_id,
+                created_by=user,
+                status="sealed",
+                retry_at=None,
+                **_fields(crawl_record, "urls max_depth config tags_str label"),
+            )
+            crawl.save(force_insert=True)
+            _restore_dates(crawl, crawl_record)
+        snapshot = Snapshot.create_from_directory(directory, crawl=crawl, preserve_identity=True)
+        if snapshot is None:
+            raise ValueError("invalid snapshot metadata")
+        snapshot.status = "sealed"
+        snapshot.retry_at = None
+        if "config" not in record:
+            snapshot.config = {"PERMISSIONS": "private"}
+        if Path(snapshot.get_storage_path_for_version(Snapshot._fs_current_version())) != directory:
+            raise ValueError("metadata does not match its owner/date/domain directory")
+        snapshot.save(force_insert=True)
+        _restore_dates(snapshot, record)
+
+    # Each insert is durable. An interruption after the Snapshot insert is not
+    # completion: the next pass still imports missing result IDs and tags.
+    for result in pending:
+        snapshot._create_archive_result_if_missing(result, {}, preserve_identity=True)
+    tags_before = set(snapshot.tags.values_list("name", flat=True))
+    snapshot._merge_tags_from_index(record)
+    tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
+    link = Path(snapshot.crawl.output_dir) / "snapshots" / snapshot.extract_domain_from_url(snapshot.url) / str(snapshot.id)
+    links_ok = link.is_symlink() and link.resolve() == directory.resolve()
+    if not links_ok:
+        snapshot.reconcile_filesystem_links()
+    if snapshot.fs_migration_needed:
+        snapshot.migrate_filesystem_to_current_version(source_dir=directory)
+    if imported:
+        _restore_dates(snapshot, record)
+    if imported_crawl:
+        _restore_dates(crawl, crawl_record)
+    # Leave source metadata untouched, including process/binary records that
+    # are historical evidence rather than runnable destination process state.
+    return "Imported" if imported else "Repaired" if pending or tags_changed or not links_ok else "Unchanged"
+
+
+def reconcile_known_snapshots(snapshots, wait_for_turn=None):
+    """Explicit metadata / asset repair; normal updates never call this scan."""
+    for snapshot in snapshots.filter(status="sealed").paged_iterator():
+        if wait_for_turn:
+            wait_for_turn()
+        if snapshot.fs_migration_needed:
+            snapshot.migrate_filesystem_to_current_version()
+        directory = Path(snapshot.output_dir)
+        if not directory.is_dir():
+            raise ValueError(f"Missing snapshot directory: {directory}")
+        _reconcile_directory(directory)
+        for result in snapshot.archiveresult_set.all():
+            result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
+
+
+def rescan_snapshots(wait_for_turn=None, reverse=False, scan_legacy=None):
+    stats = dict.fromkeys(("Imported", "Repaired", "Unchanged", "Errors"), 0)
+    print("[*] Rescanning snapshot directories (O(N), orphans first; completed records are reused)...")
+    known = {str(pk) for pk in Snapshot.objects.values_list("pk", flat=True)}
+    # Two streaming passes avoid enumerating the entire archive before the
+    # first import. Rows imported before interruption move to the second pass
+    # next time; only one day's directory names are held in memory at a time.
+    for existing in (False, True):
+        if existing and scan_legacy:
+            scan_legacy()
+        for directory in _snapshot_directories(reverse=reverse):
+            if (directory.name.replace("-", "") in known) != existing:
+                continue
+            if wait_for_turn:
+                wait_for_turn()
+            try:
+                outcome = _reconcile_directory(directory)
+                if directory.name.replace("-", "") in known:
+                    for result in ArchiveResult.objects.filter(snapshot_id=uuid.UUID(directory.name)):
+                        result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
+                stats[outcome] += 1
+                if outcome != "Unchanged":
+                    print(f"    {outcome}: {directory}")
+            except (OSError, ValueError, KeyError, TypeError, ValidationError, IntegrityError) as err:
+                stats["Errors"] += 1
+                print(f"[yellow]    Unresolved: {directory}: {err}[/yellow]")
+            if sum(stats.values()) % 100 == 0:
+                print("    " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
+    print("    " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
+    if stats["Errors"]:
+        raise SystemExit(1)
+    return stats

archivebox/config/collection.py

@@ -17,7 +17,7 @@ CONFIG_FILE_HEADER = (
     "# You can add options here manually in INI format, or automatically by running:\n"
     "#    archivebox config --set KEY=VALUE\n"
     "#\n"
-    "# This file is kept in sync 1:1 with Machine.config in the index DB —\n"
+    "# This file is kept in sync 1:1 with the current Machine.config in the DB —\n"
     "# editing either side propagates to the other. ``archivebox init`` reads\n"
     "# this file on startup; the admin Machine.config editor writes both.\n"
     "#\n"
@@ -160,7 +160,7 @@ def _coerce_from_str_dict(file_config: dict[str, str]) -> dict[str, Any]:
     so they round-trip through INI's string-only storage. When reading the
     file back into ``Machine.config`` (a JSONField that holds native types)
     those strings have to be decoded — otherwise downstream consumers like
-    ``_emit_machine_config`` → ``MachineEvent`` → abx-dl see a JSON string
+    ``MachineEvent`` → abx-dl see a JSON string
     where they expect a dict and raise ``TypeError``.
     Declared fields go through pydantic-settings' own ``field_is_complex`` /
     ``prepare_field_value`` so they're decoded per annotation. Undeclared

archivebox/config/common.py

@@ -18,7 +18,7 @@ from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.base.utils import build_config_model
 from django.db import DatabaseError
-from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
+from pydantic import BaseModel, Field, PrivateAttr, TypeAdapter, create_model, field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 from rich.console import Console
 
@@ -83,12 +83,16 @@ def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
             return candidate
 
     listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
-    if listen_host and listen_host.lower() != "archivebox.localhost:8000":
+    if listen_host and listen_host.lower() not in {"archivebox.localhost:8000", "archivebox.localhost:5797"}:
         return normalize(listen_host)
 
     csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
     csrf_origins = [origin for origin in csrf_origins if origin]
-    nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
+    nondefault_csrf_origins = [
+        origin
+        for origin in csrf_origins
+        if origin not in {"http://admin.archivebox.localhost:8000", "http://admin.archivebox.localhost:5797"}
+    ]
     if len(nondefault_csrf_origins) == 1:
         return nondefault_csrf_origins[0]
 
@@ -314,7 +318,7 @@ class ServerConfig(BaseConfigSet):
     )
 
     SECRET_KEY: str = Field(default_factory=lambda: "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789_") for _ in range(50)))
-    BIND_ADDR: str = Field(default="127.0.0.1:8000")
+    BIND_ADDR: str = Field(default="127.0.0.1:5797")
     BASE_URL: str = Field(default="")
     ALLOWED_HOSTS: str = Field(default="*")
     CSRF_TRUSTED_ORIGINS: str = Field(default="")
@@ -441,6 +445,7 @@ class ArchivingConfig(BaseConfigSet):
         default="",
         description="Comma-separated plugin selection for this run. Empty means use enabled plugin defaults.",
     )
+    PARSER: str = Field(default="auto", description="Parser used to read the submitted Crawl input document.")
 
     ONLY_NEW: bool = Field(default=True)
     INDEX_ONLY: bool = Field(default=False)
@@ -460,7 +465,8 @@ class ArchivingConfig(BaseConfigSet):
     USER_AGENT: str = Field(
         default=f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)",
     )
-    COOKIES_FILE: Path | None = Field(default=None)
+    COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
     URL_DENYLIST: str = Field(default=r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$", alias="URL_BLACKLIST")
     URL_ALLOWLIST: str | None = Field(default=None, alias="URL_WHITELIST")
@@ -852,13 +858,11 @@ class ArchiveBoxBaseConfig(
             disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
         selected_plugin_roots = plugin_names
         if selected_plugin_roots:
-            from abx_dl.models import discover_plugins, filter_plugins
+            from archivebox.plugins.discovery import get_plugin_catalog
 
             selected_plugins = set(
-                filter_plugins(
-                    discover_plugins(runtime="archivebox"),
+                get_plugin_catalog().select(
                     sorted(selected_plugin_roots),
-                    include_providers=True,
                     disabled_names=disabled_plugins,
                 ),
             )
@@ -1254,7 +1258,7 @@ def get_config(
             config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
     for key in explicit_plugin_enabled_keys:
         if key in config_data:
-            setattr(config, key, config_data[key])
+            setattr(config, key, TypeAdapter(bool).validate_python(config_data[key]))
     if config.PLUGINS:
         config._derive_plugin_enabled_config(respect_current_enabled=True)
     if redact_sensitive:

archivebox/core/admin_snapshots.py

@@ -5,9 +5,10 @@ from functools import lru_cache
 from types import SimpleNamespace
 
 from django.contrib import admin, messages
+from django.core.exceptions import PermissionDenied
 from django.urls import path, reverse
 from django.shortcuts import get_object_or_404, redirect
-from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
+from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed, Http404
 from django.utils import timezone
 from django.utils.html import format_html, format_html_join
 from django.utils.safestring import mark_safe
@@ -20,6 +21,7 @@ from archivebox.config.common import get_config
 from archivebox.misc.util import htmldecode, urldecode
 from archivebox.misc.paginators import AcceleratedPaginator
 from archivebox.misc.logging_util import printable_filesize
+from archivebox.misc.serve_static import serve_static_with_byterange_support
 from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
 from archivebox.search.views import admin_snapshot_search_stream_view
 from archivebox.core.routes_util import build_snapshot_url, build_web_url
@@ -32,7 +34,7 @@ from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
 from archivebox.core.models import Tag, Snapshot, ArchiveResult
 from archivebox.crawls.models import Crawl
 from archivebox.core.admin_archiveresults import render_archiveresults_list
-from archivebox.core.preview_util import EXTENSION_SCREENSHOT_PLUGIN
+from archivebox.core.preview_util import PREVIEW_PLUGINS, snapshot_preview_candidates, render_snapshot_preview
 from archivebox.progressmonitor.views import progress_endpoint
 from archivebox.core.permissions import (
     PERMISSIONS_CHOICES,
@@ -46,6 +48,12 @@ from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
 GLOBAL_CONTEXT = {}
 
 SNAPSHOT_PERMISSION_META = PERMISSIONS_META
+GRID_PREVIEW_OUTPUTS = {
+    ("screenshot", "screenshot.png"),
+    ("chrome_extension_screenshot", "screenshot-1.png"),
+    ("chrome_extension_screenshot", "screenshot.png"),
+    ("favicon", "favicon.ico"),
+}
 
 
 @lru_cache(maxsize=1)
@@ -272,7 +280,17 @@ class SnapshotChangeList(SearchResultsChangeList):
                 SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
             )
 
+        response_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
+        for result in ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, plugin__in=PREVIEW_PLUGINS).only(
+            "snapshot_id",
+            "plugin",
+            "status",
+            "output_files",
+        ):
+            response_results_by_snapshot[result.snapshot_id].append(result)
+
         for obj in self.result_list:
+            obj._preview_results = response_results_by_snapshot[obj.pk]
             counts = status_counts_by_snapshot.get(obj.pk, {})
             total = int(counts.get("total") or 0)
             succeeded = int(counts.get("succeeded") or 0)
@@ -346,19 +364,8 @@ class SnapshotAdminForm(forms.ModelForm):
 
             # Parse and save tags from tags_editor
             tags_str = self.cleaned_data.get("tags_editor", "")
-            if tags_str:
-                tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
-                tags = []
-                for name in tag_names:
-                    tag, _ = Tag.objects.get_or_create(
-                        name__iexact=name,
-                        defaults={"name": name},
-                    )
-                    tag = Tag.objects.filter(name__iexact=name).first() or tag
-                    tags.append(tag)
-                instance.tags.set(tags)
-            else:
-                instance.tags.clear()
+            tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
+            instance.save_tags(tag_names)
 
         return instance
 
@@ -367,6 +374,16 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     form = SnapshotAdminForm
     raw_id_fields = ("crawl", "parent_snapshot")
     list_select_related = ()
+
+    def save_related(self, request, form, formsets, change):
+        super().save_related(request, form, formsets, change)
+        tags_str = form.cleaned_data.get("tags_editor", "")
+        tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
+        form.instance.save_tags(
+            tag_names,
+            created_by=request.user if request.user.is_authenticated else None,
+        )
+
     list_display = (
         "permissions_badge",
         "created_at",
@@ -570,6 +587,11 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         custom_urls = [
             path("grid/", self.admin_site.admin_view(self.grid_view), name="grid"),
             path("search-stream/", self.admin_site.admin_view(self.search_stream_view), name="core_snapshot_search_stream"),
+            path(
+                "<path:object_id>/preview/<str:plugin>/<path:filename>",
+                self.admin_site.admin_view(self.preview_view, cacheable=True),
+                name="core_snapshot_preview",
+            ),
             path("<path:object_id>/redo-failed/", self.admin_site.admin_view(self.redo_failed_view), name="core_snapshot_redo_failed"),
             path(
                 "<path:object_id>/set-permissions/",
@@ -579,6 +601,32 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         ]
         return custom_urls + urls
 
+    def preview_view(self, request, object_id, plugin, filename):
+        """Serve grid thumbnails from the authenticated admin origin."""
+        if plugin not in PREVIEW_PLUGINS:
+            raise Http404("Unsupported snapshot preview")
+        snapshot = get_object_or_404(Snapshot, pk=object_id)
+        if not self.has_view_or_change_permission(request, snapshot):
+            raise PermissionDenied
+        candidate = next(
+            (
+                candidate
+                for candidate in snapshot_preview_candidates(snapshot)
+                if candidate["plugin"] == plugin and candidate["filename"] == filename
+            ),
+            None,
+        )
+        if candidate is None:
+            raise Http404("Snapshot preview does not exist")
+        output_path = candidate["path"]
+        request.archivebox_cache_policy = "private"
+        response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
+        if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
+            raise Http404("Snapshot preview is not an image")
+        response.headers["Vary"] = "Cookie"
+        response.headers["X-Content-Type-Options"] = "nosniff"
+        return response
+
     def search_stream_view(self, request):
         return admin_snapshot_search_stream_view(self, request)
 
@@ -939,79 +987,31 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         return mark_safe(f'<span class="tags-inline-editor">{tags_html}</span>')
 
     def _get_preview_data(self, obj):
-        request = self.request
-        config = request.archivebox_config
-        results = self._get_prefetched_results(obj)
-        if results is not None:
-            results = [r for r in results if r.plugin in ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")]
-        else:
-            results = list(
-                obj.archiveresult_set.filter(plugin__in=("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")).only(
-                    "snapshot_id",
-                    "plugin",
-                    "status",
-                    "output_files",
-                    "output_str",
-                ),
-            )
-
-        def result_output_path(result, filename: str) -> str | None:
-            output_files = result.output_files or {}
-            file_info = output_files.get(filename)
-            if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
-                return None
-            if file_info.get("root_relative"):
-                return filename
-            return f"{result.plugin}/{filename}"
-
-        def result_urls(plugin: str, filenames: tuple[str, ...]) -> list[str]:
-            urls: list[str] = []
-            for result in results:
-                if result.plugin != plugin or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
-                    continue
-                for filename in filenames:
-                    output_path = result_output_path(result, filename)
-                    if output_path:
-                        urls.append(build_snapshot_url(str(obj.id), output_path, request=request, config=config))
-            return urls
-
-        screenshot_urls = result_urls("screenshot", ("screenshot.png",))
-        extension_screenshot_urls = result_urls(EXTENSION_SCREENSHOT_PLUGIN, ("screenshot-1.png", "screenshot.png"))
-        favicon_urls = result_urls("favicon", ("favicon.ico",))
-
-        if not screenshot_urls and not extension_screenshot_urls and not favicon_urls:
+        candidates = snapshot_preview_candidates(obj)
+        if not candidates:
             return None
-
-        all_screenshot_urls = [*screenshot_urls, *extension_screenshot_urls]
-        if all_screenshot_urls:
-            img_url = all_screenshot_urls[0]
-            fallbacks = [*all_screenshot_urls[1:], *favicon_urls]
-            img_alt = "Screenshot"
-            preview_class = "screenshot"
-        else:
-            img_url = favicon_urls[0]
-            fallbacks = favicon_urls[1:]
-            img_alt = "Favicon"
-            preview_class = "favicon"
-
-        fallback_list = ",".join(fallbacks)
-        onerror_js = (
-            "this.dataset.fallbacks && this.dataset.fallbacks.length ? "
-            "(this.src=this.dataset.fallbacks.split(',').shift(), "
-            "this.dataset.fallbacks=this.dataset.fallbacks.split(',').slice(1).join(',')) : "
-            "this.remove()"
-        )
-
+        urls = [
+            build_snapshot_url(str(obj.id), candidate["path"], request=self.request, config=self.request.archivebox_config)
+            for candidate in candidates
+        ]
+        favicons = [url for candidate, url in zip(candidates, urls) if candidate["kind"] == "favicon"]
         return {
-            "img_url": img_url,
-            "img_alt": img_alt,
-            "preview_class": preview_class,
-            "onerror_js": onerror_js,
-            "fallback_list": fallback_list,
-            "favicon_url": favicon_urls[0] if favicon_urls else "",
-            "favicon_fallback_list": ",".join(favicon_urls[1:]),
+            "img_url": urls[0],
+            "fallback_list": ",".join(urls[1:]),
+            "img_alt": "Favicon" if candidates[0]["kind"] == "favicon" else "Screenshot",
+            "preview_class": "favicon" if candidates[0]["kind"] == "favicon" else "screenshot",
+            "favicon_url": favicons[0] if favicons else "",
+            "favicon_fallback_list": ",".join(favicons[1:]),
         }
 
+    def _render_preview(self, obj, width="100px", height="100px"):
+        return render_snapshot_preview(
+            obj,
+            lambda candidate: reverse("admin:core_snapshot_preview", args=(obj.pk, candidate["plugin"], candidate["filename"])),
+            width=width,
+            height=height,
+        )
+
     @admin.display(description="", empty_value="")
     def url_favicon(self, obj):
         preview = self._get_preview_data(obj)
@@ -1042,42 +1042,24 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
     @admin.display(description="Preview", empty_value="")
     def preview_icon(self, obj):
-        preview = self._get_preview_data(obj)
-        if not preview:
-            return None
-
         return format_html(
-            '<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">',
-            preview["img_url"],
-            preview["img_alt"],
-            preview["preview_class"],
-            preview["onerror_js"],
-            preview["fallback_list"],
+            '<a href="{}" title="Open snapshot details">{}</a>',
+            build_web_url(f"/{obj.archive_path_from_db}/index.html", request=self.request, config=self.request.archivebox_config),
+            self._render_preview(obj),
         )
 
     @admin.display(description=" ", empty_value="")
     def snapshot_summary(self, obj):
         request = self.request
         config = request.archivebox_config
-        preview = self._get_preview_data(obj)
         stats = self._get_progress_stats(obj)
         archive_size = stats["output_size"] or 0
         size_txt = printable_filesize(archive_size) if archive_size else "pending"
-        screenshot_html = ""
-
-        if preview:
-            screenshot_html = format_html(
-                '<a href="{href}" title="Open snapshot live view" style="display:block; flex:0 0 220px; width:220px;">'
-                '<img src="{src}" alt="{alt}" decoding="async" loading="lazy" onerror="{onerror}" data-fallbacks="{fallbacks}" '
-                'style="display:block; width:100%; max-width:220px; aspect-ratio: 16 / 10; object-fit: cover; object-position: top; '
-                'border-radius: 10px; border: 1px solid #e2e8f0; background: #f8fafc;">'
-                "</a>",
-                href=build_web_url(f"/{obj.archive_path}", request=request, config=config),
-                src=preview["img_url"],
-                alt=preview["img_alt"],
-                onerror=preview["onerror_js"],
-                fallbacks=preview["fallback_list"],
-            )
+        screenshot_html = format_html(
+            '<a href="{}" title="Open snapshot live view" style="display:block;flex:0 0 220px;">{}</a>',
+            build_web_url(f"/{obj.archive_path}", request=request, config=config),
+            self._render_preview(obj, width="220px", height="138px"),
+        )
 
         return format_html(
             '<div style="display:flex; gap:16px; align-items:flex-start;">'

archivebox/core/asgi.py

@@ -7,70 +7,31 @@ For more information on this file, see
 https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
 """
 
-from django.core.asgi import get_asgi_application
+from django.core.handlers.asgi import ASGIHandler
 
 from archivebox.config.django import setup_django
 
 setup_django(check_db=True)
 
 
-def _patch_thread_sensitive_context_shutdown() -> None:
-    """Stop ``ThreadSensitiveContext.__aexit__`` from blocking the daphne loop.
-
-    Django 6.0's ASGIHandler wraps every request in ``async with
-    ThreadSensitiveContext():`` (django/core/handlers/asgi.py:169). On exit
-    asgiref calls ``executor.shutdown()`` with the default ``wait=True``
-    (asgiref/sync.py:148), which is a *synchronous* ``Thread.join()`` inside
-    an async function — so it blocks the daphne event loop until the
-    executor's worker thread exits.
-
-    That's normally fine because the request handler has already awaited
-    every ``sync_to_async`` it submitted, so the worker is idle and dies as
-    soon as the shutdown sentinel reaches it. The blocking turns into a
-    problem when a client disconnects mid-request:
-
-    * ``SyncToAsync.__call__`` shields the executor work with
-      ``await asyncio.shield(exec_coro)`` (asgiref/sync.py:506) so that the
-      sync DB call doesn't get torn down halfway through.
-    * On cancellation it calls ``exec_coro.cancel()`` (line 522) which only
-      flips the asyncio ``Future`` to cancelled — the underlying thread
-      keeps running the SQL query.
-    * Control unwinds to ``__aexit__`` while the orphaned thread is still
-      mid-query. ``shutdown(wait=True)`` then blocks the event loop until
-      that orphan finishes.
-
-    Under heavy SQLite contention (the load-test scenario that surfaced
-    this on cabbage) those orphan threads can take 30 seconds each waiting
-    for write locks, and the daphne loop is single-threaded — so every
-    such orphan stalls every other in-flight request, healthchecks time
-    out, and the container goes ``unhealthy``.
-
-    Switching to ``shutdown(wait=False)`` queues the sentinel and returns
-    immediately; the worker thread still exits cleanly once its current
-    task finishes, and asgiref's ``WeakKeyDictionary`` releases the
-    executor as soon as the request's context is GC'd. No per-request
-    teardown guarantee is lost — there was no caller relying on it.
-    """
-    from asgiref import sync as _asgiref_sync
-
-    original_aexit = _asgiref_sync.ThreadSensitiveContext.__aexit__
-
-    async def __aexit__(self, exc, value, tb):  # type: ignore[no-redef]
-        if not self.token:
-            return
-        executor = _asgiref_sync.SyncToAsync.context_to_thread_executor.pop(self, None)
-        if executor is not None:
-            executor.shutdown(wait=False)
-        _asgiref_sync.SyncToAsync.thread_sensitive_context.reset(self.token)
-
-    # Idempotent: only patch once even if asgi.py is reloaded.
-    if getattr(original_aexit, "_archivebox_patched", False):
-        return
-    __aexit__._archivebox_patched = True  # type: ignore[attr-defined]
-    _asgiref_sync.ThreadSensitiveContext.__aexit__ = __aexit__
+class ArchiveBoxASGIHandler(ASGIHandler):
+    def create_request(self, scope, body_file):
+        # ASGI already spools the complete body. Django's multipart parser
+        # otherwise treats chunked uploads without Content-Length as empty.
+        if not any(name == b"content-length" for name, _value in scope.get("headers", [])):
+            body_file.seek(0, 2)
+            length = body_file.tell()
+            body_file.seek(0)
+            scope = {**scope, "headers": [*scope.get("headers", []), (b"content-length", str(length).encode("ascii"))]}
+        return super().create_request(scope, body_file)
 
 
-_patch_thread_sensitive_context_shutdown()
+django_application = ArchiveBoxASGIHandler()
 
-# Standard Django ASGI application (no websockets/channels needed)
-application = get_asgi_application()
+
+async def application(scope, receive, send):
+    if scope["type"] == "websocket":
+        from archivebox.opencode.views import websocket_view
+
+        return await websocket_view(scope, receive, send)
+    return await django_application(scope, receive, send)

archivebox/core/middleware.py

@@ -13,6 +13,7 @@ from django.http import HttpResponseForbidden, HttpResponseNotModified
 from django.shortcuts import redirect
 from django.utils import timezone
 from django.utils.http import http_date
+from django.utils.cache import patch_cache_control, patch_vary_headers
 
 from archivebox.config import VERSION
 from archivebox.config.common import get_config, get_request_config
@@ -89,8 +90,9 @@ def AdminCookieIsolationMiddleware(get_response):
         response = get_response(request)
 
         if request.path == "/admin" or request.path.startswith("/admin/"):
-            response.headers["X-Frame-Options"] = "DENY"
-            response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
+            if not getattr(response, "xframe_options_exempt", False):
+                response.headers["X-Frame-Options"] = "DENY"
+                response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
 
         config = request.__dict__.get("archivebox_config")
         if config is None or config.SERVER_SECURITY_MODE == "auto":
@@ -122,13 +124,12 @@ def AdminCookieIsolationMiddleware(get_response):
 
 
 def CacheControlMiddleware(get_response):
-    snapshot_path_re = re.compile(r"^/[^/]+/\\d{8}/[^/]+/[0-9a-fA-F-]{8,36}/")
     static_cache_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
 
     def middleware(request):
         response = get_response(request)
 
-        if request.path.startswith("/static/"):
+        if request.path.startswith("/static/") and response.status_code in (200, 304):
             rel_path = request.path[len("/static/") :]
             static_path = finders.find(rel_path)
             if static_path:
@@ -153,15 +154,17 @@ def CacheControlMiddleware(get_response):
                     response.headers["Last-Modified"] = http_date(mtime)
                 return response
 
-        if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
-            "Cache-Control",
-        ):
-            config = request.__dict__.get("archivebox_config")
-            if config is None:
-                config = get_config(resolve_plugins=False)
-                request.archivebox_config = config
-            policy = "private" if config.PERMISSIONS == "private" else "public"
-            response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
+        # Keep artifact freshness/validators, but never let a shared cache store
+        # private or unlisted snapshots. Auth redirects and errors stay unstored.
+        snapshot_policy = getattr(request, "archivebox_cache_policy", None)
+        if snapshot_policy and response.status_code in (200, 206, 304):
+            if not response.get("Cache-Control"):
+                response["Cache-Control"] = f"{snapshot_policy}, max-age=60, stale-while-revalidate=300"
+            if snapshot_policy == "private":
+                patch_cache_control(response, private=True)
+        elif snapshot_policy or not response.get("Cache-Control"):
+            response["Cache-Control"] = "private, no-store"
+        patch_vary_headers(response, ("Cookie", "Authorization"))
         return response
 
     return middleware

archivebox/core/migrations/0052_unique_archiveresult_per_snapshot_plugin.py

@@ -1,71 +0,0 @@
-from django.db import migrations, models
-from django.db.models import Sum
-
-
-def consolidate_archiveresults_per_plugin(apps, schema_editor):
-    ArchiveResult = apps.get_model("core", "ArchiveResult")
-    Snapshot = apps.get_model("core", "Snapshot")
-    duplicate_groups = ArchiveResult.objects.values("snapshot_id", "plugin").annotate(count=models.Count("id")).filter(count__gt=1)
-
-    affected_snapshot_ids = set()
-    for group in duplicate_groups.iterator(chunk_size=200):
-        rows = list(
-            ArchiveResult.objects.filter(
-                snapshot_id=group["snapshot_id"],
-                plugin=group["plugin"],
-            ).order_by("created_at", "id"),
-        )
-        winner = max(
-            rows,
-            key=lambda row: (
-                bool(row.output_files),
-                int(row.output_size or 0),
-                row.modified_at,
-                str(row.id),
-            ),
-        )
-        output_files = {}
-        mimetypes = set()
-        for row in rows:
-            output_files.update(row.output_files or {})
-            mimetypes.update(part.strip() for part in (row.output_mimetypes or "").split(",") if part.strip())
-
-        winner.output_files = output_files
-        winner.output_size = max(
-            sum(int(metadata.get("size") or 0) for metadata in output_files.values() if isinstance(metadata, dict)),
-            *(int(row.output_size or 0) for row in rows),
-        )
-        winner.output_mimetypes = ",".join(sorted(mimetypes))
-        winner.start_ts = min((row.start_ts for row in rows if row.start_ts), default=None)
-        winner.end_ts = max((row.end_ts for row in rows if row.end_ts), default=None)
-        winner.save(update_fields=["output_files", "output_size", "output_mimetypes", "start_ts", "end_ts"])
-        ArchiveResult.objects.filter(id__in=[row.id for row in rows if row.id != winner.id]).delete()
-        affected_snapshot_ids.add(group["snapshot_id"])
-
-    for snapshot_id in affected_snapshot_ids:
-        total = ArchiveResult.objects.filter(snapshot_id=snapshot_id).aggregate(total=Sum("output_size"))["total"] or 0
-        Snapshot.objects.filter(id=snapshot_id).update(output_size=total)
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("core", "0051_postgres_url_pattern_ops_index"),
-    ]
-
-    operations = [
-        migrations.RemoveConstraint(
-            model_name="archiveresult",
-            name="unique_archiveresult_per_snapshot_hook",
-        ),
-        migrations.RunPython(
-            consolidate_archiveresults_per_plugin,
-            reverse_code=migrations.RunPython.noop,
-        ),
-        migrations.AddConstraint(
-            model_name="archiveresult",
-            constraint=models.UniqueConstraint(
-                fields=("snapshot", "plugin"),
-                name="unique_archiveresult_per_snapshot_plugin",
-            ),
-        ),
-    ]

archivebox/core/migrations/0053_alter_archiveresult_options.py

@@ -5,7 +5,7 @@ from django.db import migrations
 
 class Migration(migrations.Migration):
     dependencies = [
-        ("core", "0052_unique_archiveresult_per_snapshot_plugin"),
+        ("core", "0051_postgres_url_pattern_ops_index"),
     ]
 
     operations = [

archivebox/core/migrations/0054_snapshot_fs_version_index.py

@@ -0,0 +1,26 @@
+# Generated by Django 6.1 on 2026-09-02
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0053_alter_archiveresult_options"),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name="snapshot",
+            name="current_step",
+        ),
+        migrations.AlterField(
+            model_name="snapshot",
+            name="fs_version",
+            field=models.CharField(
+                db_index=True,
+                default="0.9.0",
+                help_text='Filesystem version of this snapshot (e.g., "0.7.0", "0.8.0", "0.9.4").',
+                max_length=10,
+            ),
+        ),
+    ]

archivebox/core/preview_util.py

@@ -1,4 +1,114 @@
 __package__ = "archivebox.core"
 
+import hashlib
+import json
+from pathlib import PurePosixPath
+from urllib.parse import urlsplit
+
+from django.template.loader import render_to_string
+from django.utils.safestring import mark_safe
 
 EXTENSION_SCREENSHOT_PLUGIN = "chrome_extension_screenshot"
+PREVIEW_PLUGINS = ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "seo", "responses", "favicon")
+IMAGE_EXTENSIONS = {".png", ".jpg", ".jpeg", ".webp", ".gif", ".avif", ".svg", ".ico"}
+
+
+def positive_int(value):
+    try:
+        return max(0, int(value))
+    except (ValueError, TypeError, OverflowError):
+        return 0
+
+
+def preview_candidates(results):
+    """Select only saved outputs. Never read files or fetch remote metadata."""
+    candidates = []
+    for result in results:
+        files = result.output_files
+        if not isinstance(files, dict):
+            continue
+        for path, info in files.items():
+            if not isinstance(path, str) or not isinstance(info, dict) or not positive_int(info.get("size")):
+                continue
+            if path.startswith("/") or ".." in PurePosixPath(path).parts:
+                continue
+            plugin = result.plugin
+            relative = path.removeprefix(f"{plugin}/")
+            output_path = path if info.get("root_relative") else f"{plugin}/{path}"
+            kind = "image"
+            rank = None
+            if plugin == "screenshot" and relative == "screenshot.png":
+                rank = 0
+            elif plugin == EXTENSION_SCREENSHOT_PLUGIN and relative in ("screenshot-1.png", "screenshot.png"):
+                rank = 1 if relative == "screenshot-1.png" else 2
+            elif (
+                plugin == "seo"
+                and PurePosixPath(relative).stem == "featured-image"
+                and PurePosixPath(relative).suffix.lower() in IMAGE_EXTENSIONS
+            ):
+                rank = 3
+            elif plugin == "responses":
+                mime = str(info.get("mimetype") or info.get("mime_type") or "")
+                if not (relative.startswith("image/") or (relative.startswith("all/") and mime.startswith("image/"))):
+                    continue
+                width, height = positive_int(info.get("width")), positive_int(info.get("height"))
+                if width and height and (min(width, height) <= 2 or max(width, height) <= 64):
+                    continue
+                if PurePosixPath(relative).suffix.lower() == ".ico":
+                    continue
+                rank = 4
+                kind = "response"
+            elif plugin == "favicon" and relative == "favicon.ico":
+                rank = 5
+                kind = "favicon"
+            if rank is None:
+                continue
+            width, height = positive_int(info.get("width")), positive_int(info.get("height"))
+            # Prefer usable known dimensions; byte size remains the legacy fallback.
+            usable = bool(width and height and 0.25 <= width / height <= 4)
+            score = (rank, -int(usable), -(width * height if usable else 0), -positive_int(info.get("size")), output_path)
+            candidates.append((score, {"path": output_path, "kind": kind, "plugin": plugin, "filename": path}))
+    seen = set()
+    selected = []
+    for _score, candidate in sorted(candidates, key=lambda item: item[0]):
+        if candidate["path"] not in seen:
+            selected.append(candidate)
+            seen.add(candidate["path"])
+    return selected
+
+
+def snapshot_preview_candidates(snapshot):
+    if "_preview_candidates" not in snapshot.__dict__:
+        results = snapshot.__dict__.get("_preview_results")
+        if results is None:
+            results = snapshot.__dict__.get("_prefetched_objects_cache", {}).get("archiveresult_set")
+        if results is None:
+            results = snapshot.archiveresult_set.filter(plugin__in=PREVIEW_PLUGINS).only("plugin", "output_files")
+        snapshot._preview_candidates = preview_candidates(results)
+    return snapshot._preview_candidates
+
+
+def render_snapshot_preview(snapshot, url_for_candidate, width="100px", height="100px"):
+    candidates = []
+    for candidate in snapshot_preview_candidates(snapshot):
+        candidates.append({"url": url_for_candidate(candidate), "kind": candidate["kind"]})
+    try:
+        host = urlsplit(snapshot.url).hostname or "Archive"
+    except ValueError:
+        host = "Archive"
+    host = host.removeprefix("www.")
+    hue = int(hashlib.sha256(host.encode()).hexdigest()[:6], 16) % 360
+    return mark_safe(
+        render_to_string(
+            "core/snapshot_thumbnail.html",
+            {
+                "candidates": json.dumps(candidates),
+                "first": candidates[0] if candidates else None,
+                "host": host,
+                "initial": host[:1].upper(),
+                "hue": hue,
+                "width": width,
+                "height": height,
+            },
+        ),
+    )

archivebox/core/recovery_util.py

@@ -10,6 +10,11 @@ def _is_signal_interrupted_exit(exit_code: int | None) -> bool:
     return exit_code is not None and (exit_code < 0 or exit_code >= 128)
 
 
+def _canonical_hook_name(hook_name: str) -> str:
+    hook_name = Path(hook_name).name
+    return Path(hook_name).stem if Path(hook_name).suffix in {".py", ".js", ".sh"} else hook_name
+
+
 def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str | None = None) -> dict[str, int]:
     from archivebox.crawls.models import Crawl
     from archivebox.core.models import ArchiveResult, Snapshot
@@ -31,9 +36,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "crawls_queued_without_retry_at": 0,
         "snapshots_queued_without_retry_at": 0,
         "snapshots_sealed_with_extension_uploads_only": 0,
-        "archiveresults_backoff": 0,
-        "snapshots_queued_plugin_rows_waiting_on_stale_lease": 0,
-        "archiveresults_started_without_running_process": 0,
+        "archiveresults_interrupted_without_running_process": 0,
         "archiveresults_missing_for_orphaned_hook_processes": 0,
         "snapshots_started_without_running_results": 0,
         "crawls_started_with_due_snapshots": 0,
@@ -41,10 +44,10 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "crawls_started_without_active_snapshots": 0,
     }
 
-    running_archiveresults = ArchiveResult.objects.filter(
-        snapshot_id=OuterRef("pk"),
-        status=ArchiveResult.StatusChoices.STARTED,
-        process__status=Process.StatusChoices.RUNNING,
+    running_hook_processes = Process.objects.filter(
+        archiveresult__snapshot_id=OuterRef("pk"),
+        process_type=Process.TypeChoices.HOOK,
+        status=Process.StatusChoices.RUNNING,
     )
     active_child_snapshots = Snapshot.objects.filter(
         crawl_id=OuterRef("pk"),
@@ -104,49 +107,18 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         retry_at=now,
         modified_at=now,
     )
-    backoff_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.BACKOFF, **result_filter)
     orphaned_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.STARTED, **result_filter).exclude(
         process__status=Process.StatusChoices.RUNNING,
     )
-    # ArchiveResult has no retry_at scheduler. Wake only the parent Snapshots
-    # for result rows we are about to repair, then requeue those exact rows.
-    # Using subqueries keeps million-row recovery in SQLite instead of building
-    # Python ID lists or scanning all sealed snapshots.
-    Snapshot.objects.filter(
-        id__in=backoff_results.values("snapshot_id"),
-        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        retry_at__isnull=True,
-    ).update(retry_at=now, modified_at=now)
+    # ArchiveResult rows are projections, never work items. Close interrupted
+    # projections as failed and wake their parent Snapshot so abx-dl can replay
+    # the snapshot-level sequence. Indexed subqueries keep this bounded.
     Snapshot.objects.filter(
         id__in=orphaned_results.values("snapshot_id"),
-        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        retry_at__isnull=True,
+        status=Snapshot.StatusChoices.STARTED,
     ).update(retry_at=now, modified_at=now)
-    cleaned["archiveresults_backoff"] = backoff_results.update(status=ArchiveResult.StatusChoices.QUEUED, modified_at=now)
-    # Targeted plugin rows on final/paused Snapshots are scheduled through the
-    # parent Snapshot.retry_at. retry_at=NULL is the normal idle marker for a
-    # sealed Snapshot and must not be interpreted as queued work just because
-    # old/synthetic ArchiveResult rows exist. If takeover kills the runner
-    # after it leases the Snapshot but before queued ArchiveResult rows finish,
-    # the rows remain QUEUED while retry_at sits in the future. Recovery runs
-    # only after this runner has won the single-runner gate, so it can safely
-    # unlock those stale plugin leases for immediate processing instead of
-    # waiting out the previous owner's full lock timeout.
-    queued_plugin_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.QUEUED, **result_filter)
-    cleaned["snapshots_queued_plugin_rows_waiting_on_stale_lease"] = (
-        Snapshot.objects.filter(
-            id__in=queued_plugin_results.values("snapshot_id"),
-            status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        )
-        .filter(retry_at__gt=now)
-        .update(retry_at=now, modified_at=now)
-    )
-    # Impossible state repair: STARTED ArchiveResults without a live Process
-    # have no owner left to emit completion. Requeue only the result row; the
-    # snapshot/crawl schedulers will pick up normal retry processing.
-    cleaned["archiveresults_started_without_running_process"] = orphaned_results.update(
-        status=ArchiveResult.StatusChoices.QUEUED,
-        process=None,
+    cleaned["archiveresults_interrupted_without_running_process"] = orphaned_results.update(
+        status=ArchiveResult.StatusChoices.FAILED,
         modified_at=now,
     )
     orphaned_hook_processes = Process.objects.filter(
@@ -168,14 +140,30 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
         else:
             orphaned_hook_processes = orphaned_hook_processes.none()
-    for process in orphaned_hook_processes.only("id", "pwd", "cmd", "process_type", "status", "started_at", "ended_at").order_by(
-        "-started_at",
-        "-id",
-    ):
+    else:
+        # Global startup recovery must stay database-only. Process.pwd points
+        # into archive payload storage, which may be a cold FUSE mount; scanning
+        # every historical unlinked hook here blocks the sole scheduler before
+        # it can claim newly queued work. Explicit crawl recovery below remains
+        # available for the work the caller asked to resume.
+        orphaned_hook_processes = orphaned_hook_processes.none()
+    for process in orphaned_hook_processes.only(
+        "id",
+        "pwd",
+        "cmd",
+        "process_type",
+        "status",
+        "exit_code",
+        "stdout",
+        "stderr",
+        "started_at",
+        "ended_at",
+    ).order_by("-started_at", "-id"):
         hook_script_name = process.hook_script_name
         if not hook_script_name or not process.pwd:
             continue
         plugin_dir = Path(process.pwd)
+        hook_name = _canonical_hook_name(hook_script_name)
         if crawl_snapshot_ids is not None and plugin_dir.parent.name not in crawl_snapshot_ids:
             continue
         try:
@@ -187,19 +175,20 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             continue
         if snapshot is None:
             continue
-        result, created = ArchiveResult.objects.get_or_create(
-            snapshot=snapshot,
-            plugin=plugin_dir.name,
+        result, created = ArchiveResult.get_or_create_by_hook(
+            snapshot,
+            plugin_dir.name,
+            hook_name,
             defaults={
-                "hook_name": Path(hook_script_name).stem,
-                "status": ArchiveResult.StatusChoices.QUEUED,
+                "status": ArchiveResult.StatusChoices.FAILED,
             },
         )
-        if created or result.status == ArchiveResult.StatusChoices.QUEUED:
+        process_is_newer = bool(process.started_at and (result.start_ts is None or process.started_at >= result.start_ts))
+        if created or process_is_newer:
             requeue_snapshot = False
             # A runner can die after the hook Process exits but before the
             # ProcessCompletedEvent projector links/finalizes ArchiveResult.
-            # Reconstruct the plugin row from its newest durable Process row.
+            # Reconstruct only that exact hook row from the durable Process row.
             manifest = OutputManifest.scan(plugin_dir, containment_root=snapshot.output_dir)
             output_files = manifest.as_mapping()
             output_size = manifest.total_size
@@ -209,23 +198,21 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
                 for record in Process.parse_records_from_text(process.stdout or "")
                 if record.get("type") == "ArchiveResult"
                 and (record.get("plugin") or plugin_dir.name) == plugin_dir.name
-                and (record.get("hook_name") or Path(hook_script_name).stem) == Path(hook_script_name).stem
+                and _canonical_hook_name(str(record.get("hook_name") or hook_name)) == hook_name
             ]
             emitted_result = emitted_records[-1] if emitted_records else {}
-            result.hook_name = Path(hook_script_name).stem
             result.process = process
             result.start_ts = process.started_at
             result.end_ts = process.ended_at
             if _is_signal_interrupted_exit(process.exit_code):
-                # The owning runner died or was asked to stop while the hook was
-                # still active. Keep the work item queued so takeover retries the
-                # same hook; treating an unknown signal exit as success would
-                # silently skip unfinished side effects.
+                # The Process is a durable fact; record the interruption as a
+                # failure and wake the parent Snapshot for replay.
                 result.output_files = {}
                 result.output_size = 0
                 result.output_mimetypes = ""
                 result.output_str = ""
-                result.status = ArchiveResult.StatusChoices.QUEUED
+                result.output_json = None
+                result.status = ArchiveResult.StatusChoices.FAILED
                 requeue_snapshot = True
             else:
                 result.output_files = output_files
@@ -249,7 +236,6 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
                 )
             result.save(
                 update_fields=[
-                    "hook_name",
                     "process",
                     "start_ts",
                     "end_ts",
@@ -266,7 +252,6 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
                 Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
         if created:
             cleaned["archiveresults_missing_for_orphaned_hook_processes"] += 1
-            Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
     started_snapshots = Snapshot.objects.filter(status=Snapshot.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
         **snapshot_filter,
@@ -280,8 +265,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
     # We only unlock scheduling; normal Snapshot runner code owns the next
     # transition and side effects.
     cleaned["snapshots_started_without_running_results"] = (
-        started_snapshots.annotate(has_running_results=Exists(running_archiveresults))
-        .filter(has_running_results=False)
+        started_snapshots.annotate(has_running_process=Exists(running_hook_processes))
+        .filter(has_running_process=False)
         .update(
             retry_at=now,
             modified_at=now,
@@ -338,13 +323,9 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             "Finishing {count} browser-extension Snapshot(s) that received uploaded files before server extractors started "
             "(uploaded and server-created results will remain together on the same Snapshot)."
         ),
-        "archiveresults_backoff": (
-            "Retrying {count} extractor result(s) that were waiting to retry "
-            "(ArchiveBox may have been interrupted before it was able to try them again; affected outputs will be retried)."
-        ),
-        "archiveresults_started_without_running_process": (
-            "Retrying {count} extractor result(s) that were interrupted before finishing "
-            "(ArchiveBox may have been interrupted before it was able to save their final status; partial files will be overwritten with fresh results upon retry)."
+        "archiveresults_interrupted_without_running_process": (
+            "Closing {count} interrupted extractor result projection(s) "
+            "(their parent Snapshots are resumed through the normal snapshot-level runner)."
         ),
         "snapshots_started_without_running_results": (
             "Resuming {count} Snapshot(s) that were interrupted before finishing "

archivebox/core/routes_util.py

@@ -74,7 +74,7 @@ def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwa
     had ``CSRF_TRUSTED_ORIGINS=https://archive.example.com`` set (required for
     admin login to work). On upgrade, ``BASE_URL`` is the new knob — but it
     defaults to empty, and falling through to ``BIND_ADDR`` produces an
-    unreachable URL like ``http://0.0.0.0:8000``. If the user has exactly one
+    unreachable URL like ``http://0.0.0.0:5797``. If the user has exactly one
     CSRF origin we treat it as the implicit BASE_URL so links/redirects keep
     pointing at the public hostname they already configured.
 
@@ -133,7 +133,7 @@ def canonical_base_host_for_request(request_host: str) -> str:
     """Strip role subdomains and remap loopback hostnames to ``archivebox.localhost``.
 
     Used by the banner suggestion and the in-browser pin endpoint: when the
-    user is hitting the server on raw ``localhost:9292`` or ``127.0.0.1:9292``
+    user is hitting the server on raw ``localhost:5797`` or ``127.0.0.1:5797``
     we want to suggest the wildcard-friendly ``archivebox.localhost`` family
     instead, so the eventual pinned ``BASE_URL`` plays nicely with subdomain
     routing without forcing the user to add a /etc/hosts entry.

archivebox/core/settings.py

@@ -76,7 +76,6 @@ INSTALLED_APPS = [
     "archivebox.crawls",  # handles Crawl and CrawlSchedule models and management (depends on core)
     "archivebox.progressmonitor",  # live progress endpoint and admin monitor template
     "archivebox.api",  # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.
-    "archivebox.opencode",
     # 3rd-party apps from PyPI that need to be loaded last
     "admin_data_views",  # handles rendering some convenient automatic read-only views of data in Django admin
     "django_extensions",  # provides Django Debug Toolbar (and other non-debug helpers)
@@ -88,6 +87,7 @@ DJANGO_OBJECT_ACTIONS_DEFAULT_HTTP_METHOD = "POST"
 MIDDLEWARE = [
     "archivebox.core.middleware.TimezoneMiddleware",
     "django.middleware.security.SecurityMiddleware",
+    "archivebox.core.middleware.CacheControlMiddleware",
     "archivebox.core.middleware.AdminCookieIsolationMiddleware",
     "django.contrib.sessions.middleware.SessionMiddleware",
     "django.middleware.common.CommonMiddleware",
@@ -98,7 +98,6 @@ MIDDLEWARE = [
     "archivebox.core.middleware.ServerSecurityModeMiddleware",
     "archivebox.core.middleware.HostRoutingMiddleware",
     "django.contrib.messages.middleware.MessageMiddleware",
-    "archivebox.core.middleware.CacheControlMiddleware",
     # Additional middlewares from plugins (if any)
 ]
 
@@ -391,7 +390,7 @@ SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
 # get_base_url() also covers deployments that only set CSRF_TRUSTED_ORIGINS (the
 # implicit-BASE_URL fallback used on 0.7.x->0.9.x upgrades), so HTTPS hardening
 # isn't lost until BASE_URL is migrated. An explicit plain-http base (e.g. local
-# http://archivebox.localhost:8000) disables proxy HTTPS handling.
+# http://archivebox.localhost:5797) disables proxy HTTPS handling.
 BASE_URL_IS_HTTPS = get_base_url(config=CONFIG).strip().lower().startswith("https://")
 if BASE_URL_IS_HTTPS or not CONFIG.BASE_URL:
     SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")

archivebox/core/tag_util.py

@@ -152,15 +152,8 @@ def get_or_create_tag(name: str, created_by: User | None = None) -> tuple[Tag, b
     if not normalized_name:
         raise ValueError("Tag name is required")
 
-    existing = Tag.objects.filter(name__iexact=normalized_name).first()
-    if existing:
-        return existing, False
-
-    tag = Tag.objects.create(
-        name=normalized_name,
-        created_by=created_by,
-    )
-    return tag, True
+    defaults = {"created_by": created_by} if created_by is not None else None
+    return Tag.get_or_create_by_name(normalized_name, defaults=defaults)
 
 
 def rename_tag(tag: Tag, name: str) -> Tag:

archivebox/core/takeover_util.py

@@ -126,12 +126,25 @@ def ensure_daemon_stack(*, reason: str = ""):
 
     supervisor = get_existing_supervisord_process() or get_or_create_supervisord_process(daemonize=False)
     worker = get_worker(supervisor, sonic_worker["name"])
-    if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
-        return worker
+    if not (isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING")):
+        if reason:
+            rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
+        worker = start_worker(supervisor, sonic_worker)
 
-    if reason:
-        rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
-    return start_worker(supervisor, sonic_worker)
+    # Supervisord marks this worker RUNNING with startsecs=0, before Sonic has
+    # opened its search socket. A query launched at that point fails even though
+    # the daemon is about to become ready.
+    deadline = time.monotonic() + 30.0
+    while time.monotonic() < deadline:
+        if is_port_listening(sonic_event.host, sonic_event.port):
+            if time.monotonic() < deadline:
+                return worker
+            break
+        worker = get_worker(supervisor, sonic_worker["name"])
+        if not isinstance(worker, dict) or worker.get("statename") not in ("STARTING", "RUNNING"):
+            raise RuntimeError(f"Sonic search backend worker failed to start: {worker}")
+        time.sleep(min(0.1, max(0.0, deadline - time.monotonic())))
+    raise RuntimeError(f"Sonic search backend is not listening at {sonic_event.url}")
 
 
 def live_runner_processes(*, data_dir: str | Path):

archivebox/core/templatetags/core_tags.py

@@ -1,12 +1,13 @@
+import logging
 import os
 from html import unescape
 from pathlib import Path
 from typing import Any
-from urllib.parse import quote
+from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
 from django import template
-from django.templatetags.static import static
+from django.urls import reverse
 from django.utils import timezone
 from django.utils.html import escape
 from django.utils.safestring import mark_safe
@@ -21,6 +22,7 @@ from archivebox.core.routes_util import (
 )
 from archivebox.core.setup_wizard import get_base_url_mismatch_context, get_setup_wizard_context
 from archivebox.plugins.discovery import (
+    get_plugin_catalog,
     get_plugin_icon,
     get_plugin_name,
     get_plugin_template,
@@ -28,11 +30,25 @@ from archivebox.plugins.discovery import (
 
 register = template.Library()
 
+
+@register.simple_tag(takes_context=True)
+def plugin_ui(context, plugin: str, slot: str):
+    """Optional plugin UI must never prevent the host page from rendering."""
+    try:
+        source = get_plugin_template(plugin, slot, fallback=False)
+        if not source:
+            return ""
+        return template.engines["django"].from_string(source).render(context.flatten())
+    except Exception:
+        logging.getLogger(__name__).exception("Unable to render optional plugin UI: %s/%s", plugin, slot)
+        return ""
+
+
 _TEXT_PREVIEW_EXTS = (".json", ".jsonl", ".txt", ".csv", ".tsv", ".xml", ".yml", ".yaml", ".md", ".log")
 _IMAGE_PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".ico", ".avif")
 _STATIC_URL_SAFE = "/@-._~!$&'()*+,;="
 
-_MEDIA_FILE_EXTS = {
+_VIDEO_FILE_EXTS = {
     ".mp4",
     ".webm",
     ".mkv",
@@ -49,6 +65,9 @@ _MEDIA_FILE_EXTS = {
     ".3gp",
     ".3g2",
     ".ogv",
+}
+
+_AUDIO_FILE_EXTS = {
     ".mp3",
     ".m4a",
     ".aac",
@@ -66,6 +85,25 @@ _MEDIA_FILE_EXTS = {
     ".dts",
 }
 
+_MEDIA_FILE_EXTS = _VIDEO_FILE_EXTS | _AUDIO_FILE_EXTS
+_BROWSER_VIDEO_FILE_EXTS = {".mp4", ".webm", ".m4v", ".ogv"}
+_BROWSER_AUDIO_FILE_EXTS = {".mp3", ".m4a", ".aac", ".ogg", ".oga", ".opus", ".wav", ".flac"}
+
+
+def _is_web_subdomain_alias(browser_url: str, configured_base_url: str) -> bool:
+    """Hide the informational mismatch banner on the conventional public web alias.
+
+    This is deliberately presentation-only: request routing and the restriction
+    on state-changing requests continue to use the configured BASE_URL.
+    """
+    browser = urlparse(browser_url)
+    configured = urlparse(configured_base_url)
+    return (
+        browser.scheme.lower() == configured.scheme.lower()
+        and browser.hostname == f"web.{configured.hostname}"
+        and browser.port == configured.port
+    )
+
 
 def _normalize_output_files(output_files: Any) -> dict[str, dict[str, Any]]:
     from abx_dl.output_files import OutputManifest
@@ -117,10 +155,10 @@ def _count_media_files(result) -> int:
     return count
 
 
-def _list_media_files(result) -> list[dict]:
+def _list_media_files(result, *, include_filesystem_fallback=True) -> list[dict]:
     media_files: list[dict] = []
     try:
-        plugin_dir = Path(result.snapshot_dir) / result.plugin
+        plugin_dir = Path(result.snapshot_dir) / result.plugin if include_filesystem_fallback else None
     except (AttributeError, TypeError, ValueError):
         return media_files
 
@@ -132,7 +170,7 @@ def _list_media_files(result) -> list[dict]:
             if rel_path.suffix.lower() in _MEDIA_FILE_EXTS:
                 candidates.append((rel_path, _coerce_output_file_size(metadata.get("size"))))
 
-    if not candidates and plugin_dir.exists():
+    if not candidates and plugin_dir is not None and plugin_dir.exists():
         scanned = 0
         max_scan = 2000
         for file_path in plugin_dir.rglob("*"):
@@ -154,15 +192,27 @@ def _list_media_files(result) -> list[dict]:
 
     for rel_path, size in candidates:
         href = str(Path(result.plugin) / rel_path)
+        suffix = rel_path.suffix.lower()
+        media_type = "video" if suffix in _VIDEO_FILE_EXTS else "audio"
         media_files.append(
             {
                 "name": rel_path.name,
                 "path": href,
                 "size": size,
+                "media_type": media_type,
+                "is_video": media_type == "video",
+                "is_audio": media_type == "audio",
+                "is_browser_playable": suffix in _BROWSER_VIDEO_FILE_EXTS or suffix in _BROWSER_AUDIO_FILE_EXTS,
             },
         )
 
-    media_files.sort(key=lambda item: item["name"].lower())
+    media_files.sort(
+        key=lambda item: (
+            0 if item["is_video"] and item["is_browser_playable"] else 1 if item["is_audio"] and item["is_browser_playable"] else 2,
+            -int(item.get("size") or 0),
+            item["name"].lower(),
+        ),
+    )
     return media_files
 
 
@@ -280,7 +330,7 @@ def _render_text_preview(plugin: str, icon_html: str, snippet: str) -> str:
     plugin_label = escape(plugin or "")
     escaped = escape(snippet)
     return (
-        f'<div class="thumbnail-text" data-plugin="{plugin_attr}" data-compact="1">'
+        f'<div class="thumbnail-text" data-plugin="{plugin_attr}">'
         f'<div class="thumbnail-text-header">'
         f'<span class="thumbnail-compact-icon">{icon_html}</span>'
         f'<span class="thumbnail-text-title">{plugin_label}</span>'
@@ -295,7 +345,7 @@ def _render_fallback_card(plugin: str, icon_html: str, fallback_label: str) -> s
     plugin_label = escape(plugin or "")
     fallback_attr = escape(fallback_label)
     return (
-        f'<div class="thumbnail-compact" data-plugin="{plugin_attr}" data-compact="1">'
+        f'<div class="thumbnail-compact" data-plugin="{plugin_attr}">'
         f'<span class="thumbnail-compact-icon">{icon_html}</span>'
         f'<span class="thumbnail-compact-label">{plugin_label}</span>'
         f'<span class="thumbnail-compact-meta">{fallback_attr}</span>'
@@ -335,12 +385,12 @@ def index(value, position):
 
 
 @register.filter
-def file_size(num_bytes: float) -> str:
+def file_size(num_bytes: float, decimal_places: int = 1) -> str:
     for count in ["Bytes", "KB", "MB", "GB"]:
         if num_bytes > -1024.0 and num_bytes < 1024.0:
-            return f"{num_bytes:3.1f} {count}"
+            return f"{num_bytes:.{decimal_places}f} {count}"
         num_bytes /= 1024.0
-    return "{:3.1f} {}".format(num_bytes, "TB")
+    return f"{num_bytes:.{decimal_places}f} TB"
 
 
 @register.filter
@@ -411,9 +461,9 @@ def system_warnings_banner(context):
     """Render the top-of-page warning banner for one of the conditions below,
     in priority order (highest first):
 
-    1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
-       issue: until it's pinned, generated URLs can echo any Host the client
-       sends, and admin/web/api routing has no canonical anchor.
+    1. ``mode="unconfigured"``— ``BASE_URL`` is empty and the operator has
+       not selected safe single-domain routing. That mode intentionally uses
+       each device's connection address, including LAN and Tailscale IPs.
     2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
        an origin that differs from the configured canonical ``BASE_URL``.
     3. ``mode="unsafe"``      — ``SERVER_SECURITY_MODE`` explicitly enables
@@ -440,10 +490,10 @@ def system_warnings_banner(context):
 
         config = get_config(resolve_plugins=False)
 
-    if not config.BASE_URL:
+    if not config.BASE_URL and config.SERVER_SECURITY_MODE != "safe-onedomain-nojsreplay":
         return get_setup_wizard_context(context.get("request"), config)
     mismatch = get_base_url_mismatch_context(context.get("request"), config)
-    if mismatch:
+    if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
         return mismatch
     if config.IS_LOWER_SECURITY_MODE:
         return {"mode": "unsafe"}
@@ -564,6 +614,27 @@ def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) ->
     return ""
 
 
+@register.simple_tag(takes_context=True)
+def admin_snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
+    if not snapshot_archiveresult_url(context, snapshot, plugin, filename):
+        return ""
+    return reverse("admin:core_snapshot_preview", args=(snapshot.pk, plugin, filename))
+
+
+@register.simple_tag(takes_context=True)
+def snapshot_thumbnail(context, snapshot, admin=False, width="100px", height="100px"):
+    from archivebox.core.preview_util import render_snapshot_preview
+
+    if admin:
+        url_for_candidate = lambda candidate: reverse(
+            "admin:core_snapshot_preview",
+            args=(snapshot.pk, candidate["plugin"], candidate["filename"]),
+        )
+    else:
+        url_for_candidate = lambda candidate: _snapshot_url_for_context(context, snapshot, candidate["path"])
+    return render_snapshot_preview(snapshot, url_for_candidate, width=width, height=height)
+
+
 @register.simple_tag(takes_context=True)
 def snapshot_index_row(context, link) -> str:
     snapshot_base = _snapshot_base_url_for_context(context, link)
@@ -585,7 +656,6 @@ def snapshot_index_row(context, link) -> str:
 
     url = getattr(link, "url", "") or ""
     title = unescape(getattr(link, "title", "") or "")
-    is_pending = status in {"queued", "started", "backoff"}
     title_text = title or url
     tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
     tag_html = "".join(f'<span class="snapshot-tag">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag)
@@ -605,43 +675,7 @@ def snapshot_index_row(context, link) -> str:
     output_plural = "" if num_outputs == 1 else "s"
     files_url = _snapshot_url_for_context(context, link, "index.jsonl") if context.get("STATIC_EXPORT") else f"{snapshot_base}/?files=1"
 
-    if is_pending:
-        preview_html = '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress"></span>'
-        if not context.get("STATIC_EXPORT"):
-            preview_html = (
-                '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress">'
-                f'<img src="{escape(static("spinner.gif"))}" alt="" decoding="async" loading="lazy">'
-                "</span>"
-            )
-    elif "_public_preview_paths" in link.__dict__:
-        preview_paths = list(getattr(link, "_public_preview_paths", []) or [])
-        if preview_paths:
-            preview_urls = [_snapshot_url_for_context(context, link, path) for path in preview_paths]
-            preview_html = (
-                f'<img src="{escape(preview_urls[0])}" '
-                f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
-                'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
-            )
-        else:
-            preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
-    else:
-        preview_urls = [
-            url
-            for url in (
-                snapshot_archiveresult_url(context, link, "screenshot", "screenshot.png"),
-                snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot-1.png"),
-                snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot.png"),
-            )
-            if url
-        ]
-        if preview_urls:
-            preview_html = (
-                f'<img src="{escape(preview_urls[0])}" '
-                f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
-                'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
-            )
-        else:
-            preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
+    preview_html = snapshot_thumbnail(context, link)
 
     if "_public_favicon_paths" in link.__dict__:
         favicon_paths = list(getattr(link, "_public_favicon_paths", []) or [])
@@ -763,6 +797,7 @@ def plugin_card(context, result) -> str:
         return ""
 
     plugin = get_plugin_name(result.plugin)
+    has_card_template = bool(get_plugin_template(plugin, "card", fallback=False))
     template_str = get_plugin_template(plugin, "card")
 
     # Use embed_path() for the display path
@@ -771,8 +806,12 @@ def plugin_card(context, result) -> str:
 
     icon_html = get_plugin_icon(plugin)
     plugin_lower = (plugin or "").lower()
-    media_file_count = _count_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else 0
-    media_files = _list_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else []
+    media_files = (
+        _list_media_files(result, include_filesystem_fallback=bool(context.get("STATIC_EXPORT")))
+        if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl")
+        else []
+    )
+    media_file_count = len(media_files)
     if context.get("STATIC_EXPORT") and media_files:
         media_files = [
             item
@@ -791,12 +830,22 @@ def plugin_card(context, result) -> str:
     output_lower = (raw_output_path or "").lower()
     force_text_preview = output_lower.endswith(_TEXT_PREVIEW_EXTS)
 
+    # Static exports cannot execute live preview routes; keep their text snippets.
+    if context.get("STATIC_EXPORT") and force_text_preview:
+        template_str = None
+
     # Create a mini template and render it with context
     try:
-        if template_str and raw_output_path and str(raw_output_path).strip() not in (".", "/", "./") and not force_text_preview:
+        if (
+            template_str
+            and raw_output_path
+            and str(raw_output_path).strip() not in (".", "/", "./")
+            and (has_card_template or not force_text_preview)
+        ):
             tpl = template.Template(template_str)
             ctx = template.Context(
                 {
+                    **context.flatten(),
                     "result": result,
                     "snapshot": result.snapshot,
                     "output_path": output_url,
@@ -814,7 +863,23 @@ def plugin_card(context, result) -> str:
     except (template.TemplateSyntaxError, AttributeError, TypeError, ValueError):
         rendered = ""
 
-    if force_text_preview:
+    if force_text_preview and not context.get("STATIC_EXPORT"):
+        # Read archived text only when the browser requests its preview, never
+        # while rendering the outer snapshot HTML (payloads may live on B2).
+        preview_url = _build_snapshot_preview_url(
+            str(_snapshot_id(result.snapshot)),
+            raw_output_path,
+            request=context.get("request"),
+            config=context.get("CONFIG"),
+            plugin=plugin,
+        )
+        return mark_safe(
+            f'<iframe src="{escape(preview_url)}" title="{escape(plugin)} preview" '
+            'loading="lazy" fetchpriority="low" sandbox="allow-scripts allow-same-origin" '
+            'style="border:0;pointer-events:none"></iframe>',
+        )
+
+    if force_text_preview and context.get("STATIC_EXPORT"):
         preview = _render_text_file_preview(result.snapshot_dir, raw_output_path, plugin, icon_html)
         if preview:
             return mark_safe(preview)
@@ -867,7 +932,11 @@ def plugin_full(context, result) -> str:
     output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path)
 
     try:
-        tpl = template.Template(template_str)
+        catalog = get_plugin_catalog()
+        tpl = template.Engine(
+            dirs=[catalog[plugin].path] if plugin in catalog else [],
+            libraries=template.Engine.get_default().libraries,
+        ).from_string(template_str)
         ctx = template.Context(
             {
                 "result": result,

archivebox/core/views.py

@@ -1,5 +1,7 @@
 __package__ = "archivebox.core"
 
+from archivebox.core.preview_util import PREVIEW_PLUGINS, preview_candidates
+
 import json
 import os
 import posixpath
@@ -23,6 +25,7 @@ from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidde
 from django.shortcuts import redirect, render
 from django.utils.decorators import method_decorator
 from django.utils.html import format_html, format_html_join
+from django.utils.cache import patch_cache_control, patch_vary_headers
 from django.utils.safestring import mark_safe
 from django.views import View
 from django.views.decorators.csrf import csrf_exempt, csrf_protect
@@ -77,7 +80,7 @@ from archivebox.misc.util import (
     validate_url,
     without_fragment,
 )
-from archivebox.plugins.discovery import get_plugin_name, get_plugin_template
+from archivebox.plugins.discovery import get_plugin_catalog, get_plugin_name, get_plugin_template
 from archivebox.plugins.forms import get_plugin_config_binary_urls
 from archivebox.plugins.views import get_config_definition_link
 from archivebox.progressmonitor.views import live_progress_view
@@ -110,7 +113,10 @@ def _find_snapshot_by_ref(snapshot_ref: str) -> Snapshot | None:
     snapshots = Snapshot.objects.select_related("crawl", "crawl__created_by")
 
     if len(lookup) == 12 and "-" not in lookup:
-        return snapshots.filter(id__endswith=lookup).order_by("-created_at", "-downloaded_at").first()
+        # Resolve suffixes using the covering primary-key index before fetching
+        # wide snapshot rows and joining their crawl/user data.
+        matches = list(Snapshot.objects.filter(id__endswith=lookup).order_by().values_list("id", flat=True))
+        return snapshots.filter(pk__in=matches).order_by("-created_at", "-downloaded_at").first()
 
     try:
         return snapshots.get(pk=lookup)
@@ -327,6 +333,20 @@ class SnapshotView(View):
 
     @staticmethod
     def render_live_index(request, snapshot):
+        if request.GET.get("discover_outputs") == "1":
+            # Reuse the existing output discovery and card/badge classification.
+            request.archivebox_cache_policy = "private"
+            initial_context = snapshot.get_html_details_context(request=request)
+            known_paths = {output["path"] for output in initial_context["archiveresults"]}
+            context = snapshot.get_html_details_context(request=request, discover_files=True)
+            context["archiveresults"] = [output for output in context["archiveresults"] if output["path"] not in known_paths]
+            for output in context["archiveresults"]:
+                output["output_group"] = "other"
+                output["folder_path"] = str(Path(output["path"]).parent)
+            response = render(request, "core/snapshot_output_cards.html", context)
+            patch_cache_control(response, private=True, no_store=True)
+            return response
+        request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
         return render(
             template_name="core/snapshot.html",
             request=request,
@@ -350,6 +370,7 @@ class SnapshotView(View):
             try:
                 try:
                     snapshot = Snapshot.objects.get(Q(timestamp=slug) | Q(id__startswith=slug))
+                    request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
                     if not can_view_snapshot(request, snapshot):
                         return _private_snapshot_auth_redirect(request, snapshot, archivefile or "index.html")
                     canonical_base = snapshot.url_path
@@ -629,6 +650,7 @@ class SnapshotPathView(View):
                 status=404,
             )
 
+        request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
         canonical_base = snapshot.url_path
         if date:
             requested_base = f"{username}/{date}/{domain or url or ''}"
@@ -691,26 +713,27 @@ def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[s
 
     plugin = parts[0]
     plugin_relpath = posixpath.join(*parts[1:])
-    result = (
-        ArchiveResult.objects.filter(snapshot=snapshot, plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED)
-        .only("plugin", "output_files")
-        .first()
+    results = list(
+        ArchiveResult.objects.filter(
+            snapshot=snapshot,
+            plugin=plugin,
+            status=ArchiveResult.StatusChoices.SUCCEEDED,
+        ).only("plugin", "output_files"),
     )
-    if not result:
+    if not results:
         return rel_path, None
-    if not result.output_files:
-        return rel_path, result
 
-    output_files = result.output_files or {}
-    for candidate in (plugin_relpath, rel_path):
-        file_info = output_files.get(candidate)
-        if not isinstance(file_info, dict):
-            continue
-        if file_info.get("root_relative"):
-            return candidate, result
-        return rel_path, result
+    for result in results:
+        output_files = result.output_files or {}
+        for candidate in (plugin_relpath, rel_path):
+            file_info = output_files.get(candidate)
+            if not isinstance(file_info, dict):
+                continue
+            if file_info.get("root_relative"):
+                return candidate, result
+            return rel_path, result
 
-    return rel_path, result
+    return rel_path, results[0]
 
 
 def _plugin_full_preview_response(
@@ -720,7 +743,7 @@ def _plugin_full_preview_response(
     result: ArchiveResult | None,
 ) -> HttpResponse | None:
     """Render an explicit plugin full template as a trusted preview wrapper."""
-    if not request.GET.get("preview"):
+    if not request.GET.get("preview") or request.GET.get("raw") or request.GET.get("files"):
         return None
 
     path_parts = Path(rel_path).parts
@@ -743,8 +766,9 @@ def _plugin_full_preview_response(
     if raw_query:
         output_url = f"{output_url}?{raw_query.urlencode()}"
 
+    catalog = get_plugin_catalog()
     rendered = (
-        template.Engine(debug=False)
+        template.Engine(debug=False, dirs=[catalog[plugin].path] if plugin in catalog else [])
         .from_string(template_str)
         .render(
             template.Context(
@@ -753,6 +777,7 @@ def _plugin_full_preview_response(
                     "snapshot": snapshot,
                     "output_path": output_url,
                     "output_path_raw": rel_path,
+                    "snapshot_details_url": build_web_url(snapshot.get_absolute_url(), request=request),
                     "plugin": plugin,
                     "preview_base": f"{request.path.rsplit('/', 1)[0]}/",
                 },
@@ -764,6 +789,11 @@ def _plugin_full_preview_response(
     response.headers["X-Content-Type-Options"] = "nosniff"
     response.headers["X-ArchiveBox-Security-Mode"] = request.archivebox_config.SERVER_SECURITY_MODE
     response.headers["Referrer-Policy"] = "no-referrer"
+    # Trusted viewers live on the snapshot origin, but the collection UI lives
+    # on web.* (and can be embedded by admin.*). Permit only those configured
+    # origins, not arbitrary sites or other snapshots. This does not grant the
+    # archived document access to the parent frame or its admin session cookies.
+    viewer_origins = " ".join(dict.fromkeys((build_web_url(request=request), build_admin_url(request=request))))
     response.headers["Content-Security-Policy"] = (
         "default-src 'self' data: blob:; "
         "script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; "
@@ -777,7 +807,7 @@ def _plugin_full_preview_response(
         "object-src 'none'; "
         "base-uri 'none'; "
         "form-action 'none'; "
-        "frame-ancestors 'self';"
+        f"frame-ancestors 'self' {viewer_origins};"
     )
     return response
 
@@ -885,7 +915,7 @@ def _serve_responses_path(request, responses_root: Path, rel_path: str, show_ind
     return None
 
 
-def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
+def _build_snapshot_replay_response(request: HttpRequest, snapshot: Snapshot, path: str = ""):
     rel_path = path or ""
     request_config = get_request_config(
         request,
@@ -893,6 +923,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
     )
     request.archivebox_config = request_config
     request.archivebox_snapshot_url = snapshot.url
+    request.archivebox_cache_policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
     snapshot._runtime_config = request_config
 
     if rel_path.startswith("replay/") or rel_path == "replay":
@@ -905,7 +936,7 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
         # /progress.json on through to the same view used everywhere else. The caller
         # passes snapshot_id explicitly in the query string — we don't read it from the
         # subdomain (this keeps the endpoint identical across all security modes).
-        return live_progress_view(request)
+        return live_progress_view(request, authorized_snapshot=snapshot)
 
     is_directory_request = bool(path) and path.endswith("/")
     show_indexes = bool(request.GET.get("files")) or (request_config.USES_SUBDOMAIN_ROUTING and is_directory_request)
@@ -948,6 +979,20 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
     raise Http404
 
 
+def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str = ""):
+    response = _build_snapshot_replay_response(request, snapshot, path)
+    if path != "progress.json" and response.status_code in (200, 206, 304):
+        if not response.get("Cache-Control"):
+            policy = "public" if snapshot.permissions == PERMISSIONS_PUBLIC else "private"
+            response.headers["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
+        if snapshot.permissions != PERMISSIONS_PUBLIC:
+            patch_cache_control(response, private=True)
+    else:
+        response.headers["Cache-Control"] = "private, no-store"
+    patch_vary_headers(response, ("Cookie", "Authorization"))
+    return response
+
+
 def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str = ""):
     request_config = get_request_config(request, resolve_plugins=False)
     request.archivebox_config = request_config
@@ -971,6 +1016,8 @@ def _serve_original_domain_replay(request: HttpRequest, domain: str, path: str =
 
     root_match = (match[0], match[1]) if match else _latest_responses_root(snapshots, domain)
     responses_root = root_match[1] if root_match else None
+    if root_match:
+        request.archivebox_cache_policy = "public" if root_match[0].permissions == PERMISSIONS_PUBLIC else "private"
     if request_config.USES_SUBDOMAIN_ROUTING:
         snapshot = root_match[0] if root_match else (snapshots[0] if requested_root_index and snapshots else None)
         if snapshot:
@@ -1150,7 +1197,7 @@ class PublicIndexView(ListView):
         snapshots = list(context.get("object_list") or ())
         icons_by_snapshot: dict[str, set[str]] = {str(snapshot.id): set() for snapshot in snapshots}
         tag_names_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
-        preview_paths_by_snapshot: dict[str, list[tuple[int, int, str]]] = {str(snapshot.id): [] for snapshot in snapshots}
+        preview_results_by_snapshot = {str(snapshot.id): [] for snapshot in snapshots}
         favicon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
         progress_by_snapshot: dict[str, dict[str, int]] = {
             str(snapshot.id): {
@@ -1172,24 +1219,6 @@ class PublicIndexView(ListView):
             ):
                 tag_names_by_snapshot[str(snapshot_id)].append(tag_name)
 
-            preview_plugin_order = {
-                "screenshot": 0,
-                "chrome_extension_screenshot": 1,
-            }
-            preview_candidates = {
-                "screenshot": ("screenshot.png",),
-                "chrome_extension_screenshot": ("screenshot-1.png", "screenshot.png"),
-            }
-
-            def result_output_path(result: ArchiveResult, filename: str) -> str | None:
-                output_files = result.output_files or {}
-                file_info = output_files.get(filename)
-                if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
-                    return None
-                if file_info.get("root_relative"):
-                    return filename
-                return f"{result.plugin}/{filename}"
-
             for snapshot_id, plugin, status in (
                 ArchiveResult.objects.filter(
                     snapshot_id__in=icons_by_snapshot.keys(),
@@ -1216,23 +1245,16 @@ class PublicIndexView(ListView):
             for result in (
                 ArchiveResult.objects.filter(
                     snapshot_id__in=icons_by_snapshot.keys(),
-                    status=ArchiveResult.StatusChoices.SUCCEEDED,
-                    plugin__in=(*preview_candidates, "favicon"),
+                    plugin__in=PREVIEW_PLUGINS,
                 )
-                .only("snapshot_id", "plugin", "output_files")
+                .only("snapshot_id", "plugin", "status", "output_files")
                 .iterator(chunk_size=1000)
             ):
                 snapshot_key = str(result.snapshot_id)
-                if result.plugin in preview_candidates:
-                    plugin_rank = preview_plugin_order[result.plugin]
-                    for filename_rank, filename in enumerate(preview_candidates[result.plugin]):
-                        output_path = result_output_path(result, filename)
-                        if output_path:
-                            preview_paths_by_snapshot[snapshot_key].append((plugin_rank, filename_rank, output_path))
-                elif result.plugin == "favicon":
-                    output_path = result_output_path(result, "favicon.ico")
-                    if output_path:
-                        favicon_paths_by_snapshot[snapshot_key].append(output_path)
+                preview_results_by_snapshot[snapshot_key].append(result)
+                for candidate in preview_candidates([result]):
+                    if candidate["kind"] == "favicon":
+                        favicon_paths_by_snapshot[snapshot_key].append(candidate["path"])
 
         for snapshot in snapshots:
             snapshot._icons_compact = True
@@ -1240,9 +1262,7 @@ class PublicIndexView(ListView):
             snapshot._icons_progress_stats = progress_by_snapshot.get(str(snapshot.id), {})
             snapshot.num_outputs_cached = snapshot._icons_progress_stats.get("succeeded", 0)
             snapshot._tags_str_cached = ",".join(tag_names_by_snapshot.get(str(snapshot.id), []))
-            snapshot._public_preview_paths = [
-                output_path for _plugin_rank, _filename_rank, output_path in sorted(preview_paths_by_snapshot.get(str(snapshot.id), []))
-            ]
+            snapshot._preview_results = preview_results_by_snapshot.get(str(snapshot.id), [])
             snapshot._public_favicon_paths = favicon_paths_by_snapshot.get(str(snapshot.id), [])
             snapshot._is_archived_cached = bool(snapshot.downloaded_at or snapshot.status == Snapshot.StatusChoices.SEALED)
         context["object_list"] = snapshots

archivebox/crawls/admin.py

@@ -1230,7 +1230,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
         (
             "Configuration",
             {
-                "fields": ("schedule", "template"),
+                "fields": ("is_enabled", "schedule", "template"),
                 "classes": ("card",),
             },
         ),
@@ -1276,7 +1276,21 @@ class CrawlScheduleAdmin(BaseModelAdmin):
 
     def change_view(self, request, object_id, form_url="", extra_context=None):
         self.request = request
-        return super().change_view(request, object_id, form_url, extra_context)
+        from django.db import connections, router
+
+        connection = connections[router.db_for_write(self.model)]
+        if request.method != "POST" or connection.vendor != "sqlite":
+            return super().change_view(request, object_id, form_url, extra_context)
+        # Reserve the writer before Django's atomic admin form reads. A
+        # deferred read transaction cannot safely upgrade while the runner
+        # writes, even when SQLite's existing busy timeout has time left.
+        connection.ensure_connection()
+        previous_mode = connection.transaction_mode
+        connection.transaction_mode = "IMMEDIATE"
+        try:
+            return super().change_view(request, object_id, form_url, extra_context)
+        finally:
+            connection.transaction_mode = previous_mode
 
     def add_view(self, request, form_url="", extra_context=None):
         return redirect("/add/#schedule")

archivebox/crawls/models.py

@@ -90,6 +90,8 @@ class CrawlSchedule(ModelWithUUID, ModelWithNotes):
 
     @property
     def last_run_at(self):
+        if self.kind == "update":
+            return self.modified_at
         latest_crawl = self.crawl_set.order_by("-created_at").first()
         if latest_crawl:
             return latest_crawl.created_at
@@ -105,6 +107,22 @@ class CrawlSchedule(ModelWithUUID, ModelWithNotes):
         now = now or timezone.now()
         return self.is_enabled and self.next_run_at <= now
 
+    @property
+    def kind(self) -> str:
+        return str((self.config or {}).get("SCHEDULE_KIND") or "crawl")
+
+    def dispatch(self, queued_at=None) -> "Crawl | None":
+        """Run maintenance directly or enqueue one ordinary Crawl."""
+        queued_at = queued_at or timezone.now()
+        if self.kind == "update":
+            from archivebox.cli.archivebox_update import run_scheduled_maintenance
+
+            run_scheduled_maintenance()
+            type(self).objects.filter(pk=self.pk).update(modified_at=queued_at)
+            self.modified_at = queued_at
+            return None
+        return self.enqueue(queued_at=queued_at)
+
     def enqueue(self, queued_at=None) -> "Crawl":
         from archivebox.config.common import build_crawl_config_snapshot
 
@@ -112,10 +130,11 @@ class CrawlSchedule(ModelWithUUID, ModelWithNotes):
         template = self.template
         label = template.label or self.label
         persona = template.persona if template.persona_id else None
+        crawl_config = {key: value for key, value in (self.config or {}).items() if key != "SCHEDULE_KIND"}
 
         return Crawl.objects.create(
             urls=template.urls,
-            config=build_crawl_config_snapshot(persona=persona, overrides=self.config or {}),
+            config=build_crawl_config_snapshot(persona=persona, overrides=crawl_config),
             max_depth=template.max_depth,
             tags_str=template.tags_str,
             persona_id=template.persona_id,
@@ -217,7 +236,7 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
     def resume(self, *, when=None, save: bool = True) -> bool:
         resumed = super().resume(when=when, save=save)
         if resumed and self.pk:
-            from archivebox.core.models import ArchiveResult, Snapshot
+            from archivebox.core.models import Snapshot
 
             resume_at = when or timezone.now()
             active_snapshots = self.snapshot_set.filter(
@@ -228,7 +247,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
                 retry_at=resume_at,
                 modified_at=timezone.now(),
             )
-            ArchiveResult.resume_queryset(ArchiveResult.objects.filter(snapshot__crawl=self), when=resume_at)
         return resumed
 
     def cancel(self) -> None:
@@ -269,8 +287,8 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
 
         now = timezone.now()
         # Parent pause is a scheduler command. Wake child rows only; each
-        # Snapshot runner claim performs the real pause transition and cascades
-        # its own ArchiveResults, keeping request/admin transactions tiny.
+        # Snapshot runner claim performs the real pause transition, keeping
+        # request/admin transactions tiny.
         active_children = self.snapshot_set.filter(
             status__in=Snapshot.RUNNABLE_STATES,
         )
@@ -449,6 +467,8 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
             "tags_str": self.tags_str,
             "label": self.label,
             "created_at": self.created_at.isoformat() if self.created_at else None,
+            "modified_at": self.modified_at.isoformat() if self.modified_at else None,
+            "created_by": self.created_by.username,
         }
 
     @staticmethod
@@ -515,18 +535,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
             return []
         return [url for _raw_line, url in self._iter_url_lines() if url]
 
-    def has_internal_input_root(self) -> bool:
-        """Return True when Crawl.urls is preserved source text, not the work queue.
-
-        The runner creates a synthetic root snapshot for raw import text so
-        parser hooks use the same Snapshot lifecycle as every other extractor.
-        In that mode the submitted text must remain in Crawl.urls forever;
-        parsed URLs live as child Snapshot rows and should not be appended back.
-        """
-        from archivebox.core.models import Snapshot
-
-        return self.snapshot_set.filter(url=Snapshot.INTERNAL_INPUT_URL, depth=0).exists()
-
     @staticmethod
     def normalize_domain(value: str) -> str:
         candidate = (value or "").strip().lower()
@@ -796,15 +804,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
             "deleted_snapshots": filter_result["deleted_snapshots"],
         }
 
-    def get_system_task(self) -> str | None:
-        urls = self.get_urls_list()
-        if len(urls) != 1:
-            return None
-        system_url = urls[0].strip().lower()
-        if system_url.startswith("archivebox://"):
-            return system_url
-        return None
-
     def resolve_persona(self):
         from archivebox.personas.models import Persona
 
@@ -968,14 +967,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
 
         if self.status == self.StatusChoices.SEALED:
             return []
-        # Internal-input crawls preserve the submitted text verbatim in
-        # Crawl.urls. The root snapshot's parser hooks are the only supported
-        # path for turning that text into child snapshots, otherwise a later
-        # runner pass could reinterpret plain URL-looking lines as direct
-        # depth-0 work and bypass format-specific metadata parsing.
-        if self.has_internal_input_root():
-            return []
-
         created_snapshots = []
         crawl_tag_names = self.current_tag_names()
         tags_by_name: dict[str, Tag] = {}
@@ -1191,7 +1182,7 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
                 ),
             }
             if tag_names:
-                tag_ids = [Tag.objects.get_or_create(name=tag_name)[0].pk for tag_name in tag_names]
+                tag_ids = [Tag.get_or_create_by_name(tag_name)[0].pk for tag_name in tag_names]
                 snapshot.add_tag_ids(tag_ids)
 
         existing_scope = Snapshot.objects if bool(self._config_value(config, "ONLY_NEW", True)) else self.snapshot_set
@@ -1248,16 +1239,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
         if not created_snapshots:
             return []
 
-        crawl_urls = {url for _raw_line, url in self._iter_url_lines() if url}
-        new_url_lines = [snapshot.url for snapshot in created_snapshots if snapshot.url not in crawl_urls]
-        # For internal-input crawls, Crawl.urls is the immutable source text.
-        # Child snapshots are the parsed/indexed representation, so appending
-        # discovered URLs here would both duplicate state and destroy the exact
-        # import artifact users submitted through CLI/API/UI.
-        if new_url_lines and not self.has_internal_input_root():
-            self.urls = (self.urls.rstrip() + "\n" + "\n".join(new_url_lines)).lstrip("\n")
-            self.save(update_fields=["urls", "modified_at"])
-
         tag_names_by_url: dict[str, set[str]] = {}
         for snapshot in created_snapshots:
             tag_names = {

archivebox/machine/detect.py

@@ -88,7 +88,7 @@ def get_vm_info():
         try:
             # Getting SMBIOS data from sysfs.
             # SMBIOS 2.8 present.
-            # argo-1        | 2024-10-01T10:40:51Z ERR  error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:8000                                                                               │
+            # argo-1        | 2024-10-01T10:40:51Z ERR  error="Incoming request ended abruptly: context canceled" connIndex=2 event=1 ingressRule=0 originService=http://archivebox:5797                                                                               │
             # Handle 0x0100, DMI type 1, 27 bytes
             # System Information
             #         Manufacturer: DigitalOcean

archivebox/machine/models.py

@@ -227,6 +227,9 @@ class Machine(ModelWithHealthStats):
             _CURRENT_MACHINE = None
         if _CURRENT_MACHINE:
             if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+                # Reconciliation can save() and clear the shared cache; keep
+                # this call's instance so we never return None after that save.
+                machine = _CURRENT_MACHINE
                 # One-time-per-process reconciliation between ArchiveBox.conf
                 # and Machine.config. Fast-path: bool check + early-return when
                 # the sync has already run, so the cached-machine return path
@@ -234,16 +237,21 @@ class Machine(ModelWithHealthStats):
                 try:
                     from archivebox.config.collection import sync_machine_and_file
 
-                    sync_machine_and_file(_CURRENT_MACHINE)
+                    sync_machine_and_file(machine)
                 except Exception:
                     pass
-                return _CURRENT_MACHINE
+                return machine
             else:
                 _CURRENT_MACHINE = None
 
         host_guid = get_host_guid()
+        # save() intentionally invalidates _CURRENT_MACHINE so config edits
+        # made through other instances become visible in this process. Host
+        # refresh (after seven days), sanitation, and reconciliation can all
+        # save here, so the working instance must not live only in that cache.
+        # Otherwise a routine refresh leaves startup dereferencing None.
         try:
-            _CURRENT_MACHINE = cls.objects.get(guid=host_guid)
+            machine = cls.objects.get(guid=host_guid)
         except cls.DoesNotExist:
             config = {}
             try:
@@ -253,7 +261,7 @@ class Machine(ModelWithHealthStats):
                 config = _coerce_from_str_dict(file_config)
             except Exception:
                 config = {}
-            _CURRENT_MACHINE = cls.objects.create(
+            machine = cls.objects.create(
                 guid=host_guid,
                 hostname=socket.gethostname(),
                 config=config,
@@ -262,15 +270,15 @@ class Machine(ModelWithHealthStats):
                 stats=get_host_stats(),
             )
         else:
-            if timezone.now() >= _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+            if timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
                 for key, value in {
                     "hostname": socket.gethostname(),
                     **get_os_info(),
                     **get_vm_info(),
                     "stats": get_host_stats(),
                 }.items():
-                    setattr(_CURRENT_MACHINE, key, value)
-                _CURRENT_MACHINE.save(
+                    setattr(machine, key, value)
+                machine.save(
                     update_fields=[
                         "hostname",
                         "hw_in_docker",
@@ -287,16 +295,19 @@ class Machine(ModelWithHealthStats):
                         "modified_at",
                     ],
                 )
-        machine = cls._sanitize_config(_CURRENT_MACHINE)
+        machine = cls._sanitize_config(machine)
         # Same one-time sync as the cached-return path. Triggers here on the
         # very first ``Machine.current()`` call in a process before any
         # cached return can occur.
         try:
             from archivebox.config.collection import sync_machine_and_file
 
-            sync_machine_and_file(_CURRENT_MACHINE)
+            sync_machine_and_file(machine)
         except Exception:
             pass
+        # Publish only after all save-capable steps; publishing earlier would
+        # let those steps invalidate the cache we are in the middle of filling.
+        _CURRENT_MACHINE = machine
         return machine
 
     @classmethod

archivebox/misc/checks.py

@@ -255,7 +255,7 @@ def check_data_dir_permissions(config=None, **config_kwargs):
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions]https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions[/link]",
         )
         STDERR.print(
-            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid[/link]",
+            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid[/link]",
         )
         STDERR.print(
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts]https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts[/link]",

archivebox/misc/db.py

@@ -380,29 +380,40 @@ def log_sqlite_lock_holders(console: Any, *, db_path: Path = CONSTANTS.DATABASE_
 def sqlite_lock_error(error: BaseException) -> bool:
     from django.db import OperationalError as DjangoOperationalError
 
-    return isinstance(error, (SQLiteOperationalError, DjangoOperationalError)) and "database is locked" in str(error).lower()
+    message = str(error).lower()
+    return isinstance(error, (SQLiteOperationalError, DjangoOperationalError)) and (
+        "database is locked" in message or "database table is locked" in message
+    )
 
 
 def retry_sqlite_locks(action: Callable[[], Any], *, label: str, stderr: TextIO | None = None) -> Any:
+    from django.conf import settings
     from django.db import OperationalError, connections
     from rich.console import Console
 
     console = Console(file=stderr or None, stderr=stderr is None)
+    started_at = time.monotonic()
+    retry_timeout = settings.CONFIG.SQLITE_LOCK_RETRY_TIMEOUT
+    retry_interval = settings.CONFIG.SQLITE_LOCK_RETRY_INTERVAL
     while True:
         try:
             return action()
-        except OperationalError as err:
-            if "database is locked" not in str(err).lower():
-                raise
-        except SQLiteOperationalError as err:
+        except (OperationalError, SQLiteOperationalError) as err:
             if not sqlite_lock_error(err):
                 raise
+            if retry_timeout:
+                remaining = retry_timeout - (time.monotonic() - started_at)
+                if remaining <= 0:
+                    raise
+                sleep_for = min(retry_interval, remaining)
+            else:
+                sleep_for = retry_interval
 
         connections.close_all()
-        console.print(f"[yellow][*] SQLite database is locked while {label}; retrying in 5s...[/yellow]")
+        console.print(f"[yellow][*] SQLite database is locked while {label}; retrying in {sleep_for:g}s...[/yellow]")
         log_sqlite_lock_holders(console)
         with console.status("[yellow]Waiting for SQLite database lock to clear...[/yellow]", spinner="dots"):
-            time.sleep(5.0)
+            time.sleep(sleep_for)
 
 
 @contextmanager

archivebox/misc/hashing.py

@@ -1,260 +0,0 @@
-# Bootable hashing helpers (file content + mtime/size cache, mime detection).
-# MUST NOT import archivebox.config, archivebox.core, or Django — stdlib only.
-
-import hashlib
-import mimetypes
-from functools import lru_cache
-from pathlib import Path
-from collections.abc import Callable
-from datetime import datetime
-
-
-@lru_cache(maxsize=1024)
-def _cached_file_hash(filepath: str, size: int, mtime: float) -> str:
-    """Internal function to calculate file hash with cache key based on path, size and mtime."""
-    sha256_hash = hashlib.sha256()
-
-    with open(filepath, "rb") as f:
-        for chunk in iter(lambda: f.read(4096), b""):
-            sha256_hash.update(chunk)
-
-    return sha256_hash.hexdigest()
-
-
-@lru_cache(maxsize=10)
-def hash_file(file_path: Path, pwd: Path | None = None) -> str:
-    """Calculate SHA256 hash of a file with caching based on path, size and mtime."""
-    pwd = Path(pwd) if pwd else None
-    file_path = Path(file_path)
-    if not file_path.is_absolute():
-        file_path = pwd / file_path if pwd else file_path.absolute()
-
-    abs_path = file_path.resolve()
-    stat_info = abs_path.stat()
-
-    return _cached_file_hash(
-        str(abs_path),
-        stat_info.st_size,
-        stat_info.st_mtime,
-    )
-
-
-@lru_cache(maxsize=10)
-def get_dir_hashes(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict[str, str]:
-    """Calculate SHA256 hashes for all files and directories recursively."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    if not dir_path.is_dir():
-        raise ValueError(f"Not a directory: {dir_path}")
-    if max_depth < -1:
-        raise ValueError(f"max_depth must be >= -1, got {max_depth}")
-
-    # Get all files recursively
-    all_files = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=True,
-        include_dirs=False,
-        filter_func=filter_func,
-    )
-
-    hashes: dict[str, str] = {}
-    hashable_summary = []
-
-    # Calculate hashes for all files
-    for subfile in all_files:
-        subfile_path = dir_path / subfile
-        sha256_hash = hash_file(subfile_path)
-        hashes[subfile] = sha256_hash
-        hashable_summary.append(f"{sha256_hash}  ./{subfile}")
-
-    # Calculate hashes for all directories
-    subdirs = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=False,
-        include_dirs=True,
-        include_hidden=False,
-        filter_func=filter_func,
-        max_depth=max_depth,
-    )
-
-    for subdir in subdirs:
-        subdir_path = dir_path / subdir
-        subdir_hashes = get_dir_hashes(
-            subdir_path,
-            filter_func=filter_func,
-            max_depth=0,
-        )
-        hashes[subdir] = subdir_hashes["."]
-
-    # Filter results by max_depth
-    if max_depth >= 0:
-        hashes = {path: value for path, value in hashes.items() if len(Path(path).parts) <= max_depth + 1}
-
-    # Calculate root directory hash
-    hashable_summary.sort()
-    root_sha256 = hashlib.sha256("\n".join(hashable_summary).encode()).hexdigest()
-    hashes["."] = root_sha256
-
-    return hashes
-
-
-@lru_cache(maxsize=128)
-def get_dir_entries(
-    dir_path: Path,
-    pwd: Path | None = None,
-    recursive: bool = True,
-    include_files: bool = True,
-    include_dirs: bool = True,
-    include_hidden: bool = False,
-    filter_func: Callable | None = None,
-    max_depth: int = -1,
-) -> tuple[str, ...]:
-    """Get filtered list of directory entries."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    results = []
-
-    def process_path(path: Path, depth: int):
-        if not include_hidden and path.name.startswith("."):
-            return False
-        if max_depth >= 0 and depth > max_depth:
-            return False
-        if filter_func:
-            info = {
-                "abspath": str(path.absolute()),
-                "relpath": str(path.relative_to(dir_path)),
-            }
-            if not filter_func(info):
-                return False
-        return True
-
-    for path in dir_path.rglob("*") if recursive else dir_path.glob("*"):
-        current_depth = len(path.relative_to(dir_path).parts)
-
-        if path.is_file() and include_files and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-        elif path.is_dir() and include_dirs and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-
-        if not recursive:
-            break
-
-    return tuple(sorted(results))  # Make immutable for caching
-
-
-@lru_cache(maxsize=1024)
-def get_dir_sizes(dir_path: Path, pwd: Path | None = None, **kwargs) -> dict[str, int]:
-    """Calculate sizes for all files and directories recursively."""
-    sizes: dict[str, int] = {}
-    hashes = get_dir_hashes(dir_path, pwd=pwd, **kwargs)
-    dir_path = Path(dir_path)
-
-    for path_key in hashes:
-        full_path = dir_path / path_key
-        if full_path.is_file():
-            sizes[path_key] = full_path.stat().st_size
-        else:
-            total = 0
-            for file_path in full_path.rglob("*"):
-                if file_path.is_file() and not file_path.name.startswith("."):
-                    total += file_path.stat().st_size
-            sizes[path_key + "/"] = total
-
-    return sizes
-
-
-@lru_cache(maxsize=10)
-def get_dir_info(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict:
-    """Get detailed information about directory contents including hashes and sizes."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    hashes = get_dir_hashes(dir_path, pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-    sizes = get_dir_sizes(str(dir_path), pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-
-    num_total_subpaths = sum(1 for name in hashes if name != ".")
-    details = {}
-
-    for filename, sha256_hash in sorted(hashes.items()):
-        abs_path = (dir_path / filename).resolve()
-        stat_info = abs_path.stat()
-        num_subpaths = sum(1 for p in hashes if p.startswith(filename + "/"))
-        is_dir = abs_path.is_dir()
-        if is_dir:
-            mime_type = "inode/directory"
-            basename = abs_path.name
-            extension = ""
-            num_bytes = sizes[filename + "/"]
-            if filename == ".":
-                num_subpaths = num_total_subpaths
-            else:
-                filename += "/"
-                num_subpaths = num_subpaths
-        else:  # is_file
-            num_subpaths = None
-            mime_type = mimetypes.guess_type(str(abs_path))[0]
-            extension = abs_path.suffix
-            basename = abs_path.name.rsplit(extension, 1)[0]
-            num_bytes = sizes[filename]
-
-        details[filename] = {
-            "basename": basename,
-            "mime_type": mime_type,
-            "extension": extension,
-            "num_subpaths": num_subpaths,
-            "num_bytes": num_bytes,
-            "hash_sha256": sha256_hash,
-            "created_at": datetime.fromtimestamp(stat_info.st_ctime).isoformat(),
-            "modified_at": datetime.fromtimestamp(stat_info.st_mtime).isoformat(),
-        }
-
-        if filter_func and not filter_func(details[filename]):
-            del details[filename]
-
-    return details
-
-
-if __name__ == "__main__":
-    import json
-
-    dir_info = get_dir_info(Path("."), max_depth=6)
-    with open(".hashes.json", "w") as f:
-        json.dump(dir_info, f, indent=4)
-    print("Wrote .hashes.json")
-
-# Example output:
-# {
-#     ".": {
-#         "basename": "misc",
-#         "mime_type": "inode/directory",
-#         "extension": "",
-#         "num_subpaths": 25,
-#         "num_bytes": 214677,
-#         "hash_sha256": "addfacf88b2ff6b564846415fb7b21dcb7e63ee4e911bc0aec255ee354958530",
-#         "created_at": "2024-12-04T00:08:38.537449",
-#         "modified_at": "2024-12-04T00:08:38.537449"
-#     },
-#     "__init__.py": {
-#         "basename": "__init__",
-#         "mime_type": "text/x-python",
-#         "extension": ".py",
-#         "num_subpaths": null,
-#         "num_bytes": 32,
-#         "hash_sha256": "b0e5e7ff17db3b60535cf664282787767c336e3e203a43e21b6326c6fe457551",
-#         "created_at": "2024-10-08T00:51:41.001359",
-#         "modified_at": "2024-10-08T00:51:41.001359"
-#     },
-#     ...
-# }

archivebox/misc/jsonl.py

@@ -32,7 +32,6 @@ import sys
 import json
 from typing import Any, TextIO
 from collections.abc import Iterable, Iterator
-from pathlib import Path
 
 
 # Type constants for JSONL records
@@ -120,19 +119,6 @@ def read_stdin(stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
             yield record
 
 
-def read_file(path: Path) -> Iterator[dict[str, Any]]:
-    """
-    Read JSONL or plain URLs from a file.
-
-    Yields parsed records as dicts.
-    """
-    with open(path) as f:
-        for line in f:
-            record = parse_line(line)
-            if record:
-                yield record
-
-
 def read_args_or_stdin(args: Iterable[str], stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
     """
     Read from CLI arguments if provided, otherwise from stdin.

archivebox/misc/serve_static.py

@@ -14,6 +14,7 @@ import time
 import zipfile
 from collections.abc import Callable
 from datetime import UTC, datetime
+from html.parser import HTMLParser
 from pathlib import Path
 from urllib.parse import quote, urlencode, urljoin
 
@@ -129,7 +130,7 @@ def _resolve_archive_path(document_root: str | Path, rel_path: str) -> tuple[Pat
 
 def _cache_policy(config=None, **config_kwargs) -> str:
     config = config or get_config(resolve_plugins=False, **config_kwargs)
-    return "private" if config.PERMISSIONS == "private" else "public"
+    return "public" if config.PERMISSIONS == "public" else "private"
 
 
 def _format_direntry_timestamp(stat_result: os.stat_result) -> str:
@@ -527,6 +528,26 @@ def _index_responses_paths_for_html_images(
 
     best_matches: dict[str, str] = {}
     image_matches: set[str] = set()
+    # Optional responses plugin file contract: index.jsonl records contain method,
+    # url, status, and a path relative to responses/. No plugin code is imported.
+    # Legacy captures without an index still use the filename lookup below.
+    # The index retains complete URLs even when response filenames are shortened.
+    index = responses_root / "index.jsonl"
+    if index.is_file():
+        with index.open(errors="replace") as records:
+            for line in records:
+                try:
+                    record = json.loads(line)
+                    encoded = _encoded_responses_image_url(record.get("url", ""), None)
+                    if encoded not in encoded_urls or record.get("method") != "GET" or record.get("status") != 200:
+                        continue
+                    candidate = (responses_root / record.get("path", "")).resolve()
+                    if not candidate.is_relative_to(responses_root.resolve()) or not candidate.is_file():
+                        continue
+                    best_matches[encoded] = posixpath.relpath(candidate, start=snapshot_root / posixpath.dirname(html_rel_path))
+                    image_matches.add(encoded)
+                except (ValueError, TypeError, AttributeError):
+                    continue
     image_suffixes = {".png", ".jpg", ".jpeg", ".gif", ".webp", ".svg", ".ico"}
     for candidate in responses_root.rglob("*"):
         if not candidate.is_file():
@@ -561,10 +582,30 @@ def _rewrite_html_image_sources_to_responses(
         if image_url not in encoded_urls_by_src:
             encoded_urls_by_src[image_url] = _encoded_responses_image_url(image_url, page_url)
 
+    # Optional dom plugin artifact: dom/output.html. Only image src and
+    # data-canonical-src attributes are read; no scripts or plugin code execute.
+    # Absence of this artifact simply disables proxy-to-original aliases.
+    aliases: dict[str, str] = {}
+
+    class ImageAliases(HTMLParser):
+        def handle_starttag(self, tag, attrs):
+            attrs = dict(attrs)
+            original = attrs.get("data-canonical-src")
+            proxy = attrs.get("src")
+            if tag == "img" and original and proxy:
+                original_key = _encoded_responses_image_url(original, page_url)
+                proxy_key = _encoded_responses_image_url(proxy, page_url)
+                if original_key and proxy_key:
+                    aliases[original_key] = proxy_key
+
+    dom = snapshot_root / "dom" / "output.html"
+    if dom.is_file() and dom.stat().st_size <= 10 * 1024 * 1024:
+        ImageAliases().feed(dom.read_text(errors="replace"))
+    wanted = {encoded for encoded in encoded_urls_by_src.values() if encoded}
     response_paths = _index_responses_paths_for_html_images(
         snapshot_root,
         html_rel_path,
-        {encoded_url for encoded_url in encoded_urls_by_src.values() if encoded_url},
+        wanted | {aliases[key] for key in wanted if key in aliases},
     )
     rewrites = 0
 
@@ -572,7 +613,7 @@ def _rewrite_html_image_sources_to_responses(
         nonlocal rewrites
         image_url = html.unescape(match.group(2))
         encoded_url = encoded_urls_by_src.get(image_url)
-        local_path = response_paths.get(encoded_url or "")
+        local_path = response_paths.get(encoded_url or "") or response_paths.get(aliases.get(encoded_url or "", ""))
         if not local_path:
             return match.group(0)
         rewrites += 1
@@ -589,12 +630,17 @@ def _rewrite_html_image_sources_for_request(
 ) -> tuple[str, int]:
     if not document_root:
         return html_text, 0
-    return _rewrite_html_image_sources_to_responses(
-        html_text,
-        document_root,
-        html_rel_path,
-        request.__dict__.get("archivebox_snapshot_url"),
-    )
+    try:
+        return _rewrite_html_image_sources_to_responses(
+            html_text,
+            document_root,
+            html_rel_path,
+            request.__dict__.get("archivebox_snapshot_url"),
+        )
+    except (OSError, ValueError, TypeError, RecursionError):
+        # Saved plugin artifacts are optional and may be incomplete, malformed,
+        # or removed concurrently. Image enrichment must not break file replay.
+        return html_text, 0
 
 
 def _apply_transformed_html_preview_style(html_text: str) -> str:
@@ -605,16 +651,22 @@ def _apply_transformed_html_preview_style(html_text: str) -> str:
     return f"{TRANSFORMED_HTML_PREVIEW_STYLE}\n{html_text}"
 
 
-def _set_transformed_response_headers(response, fullpath: Path, statobj: os.stat_result, encoding: str | None, config) -> None:
+def _set_transformed_response_headers(
+    response,
+    fullpath: Path,
+    statobj: os.stat_result,
+    encoding: str | None,
+    cache_policy: str,
+) -> None:
     response.headers["Last-Modified"] = http_date(statobj.st_mtime)
-    response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
+    response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
     response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
     if encoding:
         response.headers["Content-Encoding"] = encoding
 
 
 def _render_markdown_fallback(text: str) -> str:
-    if _markdown is not None and not HTML_TAG_RE.search(text):
+    if _markdown is not None:
         try:
             return _markdown(
                 text,
@@ -858,19 +910,22 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
     config = request.__dict__.get("archivebox_config")
     if config is None:
         config = get_config(resolve_plugins=False)
+    cache_policy = getattr(request, "archivebox_cache_policy", None) or _cache_policy(config=config)
     fullpath, path = _resolve_archive_path(document_root, path)
     if os.access(fullpath, os.R_OK) and fullpath.is_dir():
         if request.GET.get("download") == "zip" and show_indexes:
-            return _build_directory_zip_response(
+            response = _build_directory_zip_response(
                 fullpath,
                 path,
                 is_archive_replay=is_archive_replay,
                 use_async_stream=_is_asgi_request(request),
                 config=config,
             )
+            response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
+            return response
         if show_indexes:
             response = _render_directory_index(request, path, fullpath)
-            response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
+            response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
             response.headers["Last-Modified"] = http_date(fullpath.stat().st_mtime)
             return _apply_archive_replay_headers(
                 response,
@@ -899,7 +954,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
             if etag in inm_list or etag.strip('"') in [i.strip('"') for i in inm_list]:
                 not_modified = HttpResponseNotModified()
                 not_modified.headers["ETag"] = etag
-                not_modified.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
+                not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
                 not_modified.headers["Last-Modified"] = http_date(statobj.st_mtime)
                 return _apply_archive_replay_headers(
                     not_modified,
@@ -910,6 +965,9 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 )
 
     content_type, encoding = mimetypes.guess_type(str(fullpath))
+    preserve_plain_text = fullpath.suffix.lower() in {".log", ".sh", ".jsonl"}
+    if preserve_plain_text:
+        content_type = "text/plain"
     content_type = content_type or "application/octet-stream"
     # Add charset for text-like types (best guess), but don't override the type.
     is_text_like = content_type.startswith("text/") or content_type in {
@@ -924,7 +982,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
     preview_as_text_html = (
         bool(request.GET.get("preview"))
         and is_text_like
-        and not content_type.startswith("text/html")
+        and (bool(request.GET.get("raw")) or not content_type.startswith("text/html"))
         and not content_type.startswith("image/svg+xml")
     )
     preview_as_image_html = (
@@ -937,8 +995,10 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
         request.META.get("HTTP_IF_MODIFIED_SINCE"),
         statobj.st_mtime,
     ):
+        not_modified = HttpResponseNotModified()
+        not_modified.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
         return _apply_archive_replay_headers(
-            HttpResponseNotModified(),
+            not_modified,
             fullpath=fullpath,
             content_type=content_type,
             is_archive_replay=is_archive_replay,
@@ -953,7 +1013,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 decoded = fullpath.read_text(encoding="utf-8", errors="replace")
                 wrapped = _render_text_preview_document(decoded, fullpath.name)
                 response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
-                _set_transformed_response_headers(response, fullpath, statobj, encoding, config)
+                _set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
                 return _apply_archive_replay_headers(
                     response,
                     fullpath=fullpath,
@@ -973,7 +1033,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 raw_image_url = f"{raw_image_url}?{urlencode(list(preview_query.lists()), doseq=True)}"
             wrapped = _render_image_preview_document(raw_image_url, fullpath.name)
             response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
-            _set_transformed_response_headers(response, fullpath, statobj, encoding, config)
+            _set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
             return _apply_archive_replay_headers(
                 response,
                 fullpath=fullpath,
@@ -999,9 +1059,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 last_modified=http_date(statobj.st_mtime),
                 etag=etag or "",
                 cache_control=(
-                    f"{_cache_policy(config=config)}, max-age=31536000, immutable"
-                    if etag
-                    else f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
+                    f"{cache_policy}, max-age=31536000, immutable" if etag else f"{cache_policy}, max-age=60, stale-while-revalidate=300"
                 ),
                 content_encoding=encoding or "",
             )
@@ -1020,7 +1078,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
 
     # Heuristic fix: some archived HTML outputs are stored with HTML-escaped markup
     # or markdown sources. If so, render sensibly.
-    if content_type.startswith(("text/plain", "text/html")):
+    if not request.GET.get("raw") and not preserve_plain_text and content_type.startswith(("text/plain", "text/html", "text/markdown")):
         try:
             max_unescape_size = 10 * 1024 * 1024  # 10MB cap to avoid heavy memory use
             if statobj.st_size <= max_unescape_size:
@@ -1034,12 +1092,12 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 if content_type.startswith("text/html") and document_root:
                     rewritten_html, rewritten_count = _rewrite_html_image_sources_for_request(request, decoded, document_root, rel_path)
                 markdown_candidate = _extract_markdown_candidate(decoded)
-                if _looks_like_markdown(markdown_candidate):
+                if content_type.startswith("text/markdown") or _looks_like_markdown(markdown_candidate):
                     wrapped = _render_markdown_document(markdown_candidate)
                     wrapped, _rewrite_count = _rewrite_html_image_sources_for_request(request, wrapped, document_root, rel_path)
                     wrapped = _apply_transformed_html_preview_style(wrapped)
                     response = HttpResponse(wrapped, content_type="text/html; charset=utf-8")
-                    _set_transformed_response_headers(response, fullpath, statobj, encoding, config)
+                    _set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
                     return _apply_archive_replay_headers(
                         response,
                         fullpath=fullpath,
@@ -1050,7 +1108,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 if rewritten_count:
                     rewritten_html = _apply_transformed_html_preview_style(rewritten_html)
                     response = HttpResponse(rewritten_html, content_type=content_type)
-                    _set_transformed_response_headers(response, fullpath, statobj, encoding, config)
+                    _set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
                     return _apply_archive_replay_headers(
                         response,
                         fullpath=fullpath,
@@ -1061,7 +1119,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 if escaped_count and escaped_count > tag_count * 2:
                     decoded = _apply_transformed_html_preview_style(decoded)
                     response = HttpResponse(decoded, content_type=content_type)
-                    _set_transformed_response_headers(response, fullpath, statobj, encoding, config)
+                    _set_transformed_response_headers(response, fullpath, statobj, encoding, cache_policy)
                     return _apply_archive_replay_headers(
                         response,
                         fullpath=fullpath,
@@ -1081,13 +1139,13 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
     response.headers["Last-Modified"] = http_date(statobj.st_mtime)
     if etag:
         response.headers["ETag"] = etag
-        response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=31536000, immutable"
+        response.headers["Cache-Control"] = f"{cache_policy}, max-age=31536000, immutable"
     else:
-        response.headers["Cache-Control"] = f"{_cache_policy(config=config)}, max-age=60, stale-while-revalidate=300"
+        response.headers["Cache-Control"] = f"{cache_policy}, max-age=60, stale-while-revalidate=300"
     if is_text_like:
         response.headers["Content-Disposition"] = f'inline; filename="{fullpath.name}"'
     if content_type.startswith("image/"):
-        response.headers["Cache-Control"] = "public, max-age=604800, immutable"
+        response.headers["Cache-Control"] = f"{cache_policy}, max-age=604800, immutable"
 
     # handle byte-range requests by serving chunk of file
     if stat.S_ISREG(statobj.st_mode):

archivebox/misc/toml_util.py

@@ -2,94 +2,11 @@ from typing import Any, cast
 from collections.abc import Callable
 
 import json
-import ast
-import inspect
 import toml
 import re
-import configparser
 
 from pathlib import Path, PosixPath
 
-from pydantic.json_schema import GenerateJsonSchema
-from pydantic_core import to_jsonable_python
-
-JSONValue = str | bool | int | None | list["JSONValue"]
-
-TOML_HEADER = "# Converted from INI to TOML format: https://toml.io/en/\n\n"
-
-
-def load_ini_value(val: str) -> JSONValue:
-    """Convert lax INI values into strict TOML-compliant (JSON) values"""
-    if val.lower() in ("true", "yes", "1"):
-        return True
-    if val.lower() in ("false", "no", "0"):
-        return False
-    if val.isdigit():
-        return int(val)
-
-    try:
-        return ast.literal_eval(val)
-    except Exception:
-        pass
-
-    try:
-        return json.loads(val)
-    except Exception:
-        pass
-
-    return val
-
-
-def convert(ini_str: str) -> str:
-    """Convert a string of INI config into its TOML equivalent (warning: strips comments)"""
-
-    config = configparser.ConfigParser()
-    setattr(config, "optionxform", str)  # capitalize key names
-    config.read_string(ini_str)
-
-    # Initialize an empty dictionary to store the TOML representation
-    toml_dict = {}
-
-    # Iterate over each section in the INI configuration
-    for section in config.sections():
-        toml_dict[section] = {}
-
-        # Iterate over each key-value pair in the section
-        for key, value in config.items(section):
-            parsed_value = load_ini_value(value)
-
-            # Convert the parsed value to its TOML-compatible JSON representation
-            toml_dict[section.upper()][key.upper()] = json.dumps(parsed_value)
-
-    # Build the TOML string
-    toml_str = TOML_HEADER
-    for section, items in toml_dict.items():
-        toml_str += f"[{section}]\n"
-        for key, value in items.items():
-            toml_str += f"{key} = {value}\n"
-        toml_str += "\n"
-
-    return toml_str.strip()
-
-
-class JSONSchemaWithLambdas(GenerateJsonSchema):
-    """
-    Encode lambda functions in default values properly.
-    Usage:
-    >>> json.dumps(value, encoder=JSONSchemaWithLambdas())
-    """
-
-    def encode_default(self, dft: Any) -> Any:
-        config = self._config
-        if isinstance(dft, Callable):
-            return "{{lambda " + inspect.getsource(dft).split("=lambda ")[-1].strip()[:-1] + "}}"
-        return to_jsonable_python(
-            dft,
-            timedelta_mode=config.ser_json_timedelta,
-            bytes_mode=config.ser_json_bytes,
-            serialize_unknown=True,
-        )
-
 
 def better_toml_dump_str(val: Any) -> str:
     try:

archivebox/misc/util.py

@@ -192,7 +192,10 @@ def validate_url_length(url: str) -> str:
 
 
 def validate_url(url: str) -> str:
-    url = validate_url_length((url or "").strip())
+    url = (url or "").strip()
+    if "\n" in url or "\r" in url:
+        raise ValueError("URL must be a single line.")
+    url = validate_url_length(url)
     parsed = urlparse(url)
     if parsed.scheme.lower() not in ("http", "https") or not parsed.hostname:
         raise ValueError("URL must start with http:// or https:// and include a hostname.")

archivebox/opencode/apps.py

@@ -1,6 +0,0 @@
-from django.apps import AppConfig
-
-
-class OpencodeConfig(AppConfig):
-    name = "archivebox.opencode"
-    label = "opencode_plugin"

archivebox/opencode/templates/opencode/agent.html

@@ -1,211 +0,0 @@
-{% extends "admin/base.html" %}
-{% load i18n %}
-
-{% block title %}Agent{% endblock %}
-
-{% block breadcrumbs %}
-    <div class="breadcrumbs">
-        <a href="{% url 'admin:index' %}">{% trans 'Home' %}</a>
-        &rsaquo; Agent
-    </div>
-{% endblock %}
-
-{% block extrastyle %}
-    {{ block.super }}
-    <style>
-        body.opencode-agent #content {
-            padding: 0;
-        }
-        body.opencode-agent #content-main {
-            height: calc(100vh - 150px);
-            min-height: 560px;
-        }
-        .opencode-agent-error {
-            margin: 24px;
-            color: #0f172a;
-            font: 14px -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
-        }
-        .opencode-agent-error code {
-            color: #0369a1;
-        }
-        .opencode-agent-frame {
-            width: 100%;
-            height: 100%;
-            border: 0;
-            display: block;
-        }
-        .opencode-agent-shell {
-            position: relative;
-            height: 100%;
-        }
-        .opencode-agent-welcome {
-            position: absolute;
-            inset: 0;
-            z-index: 20;
-            display: flex;
-            align-items: center;
-            justify-content: center;
-            padding: 24px;
-            background: rgba(0, 0, 0, 0.48);
-            color: #111827;
-        }
-        .opencode-agent-welcome[hidden] {
-            display: none;
-        }
-        .opencode-agent-welcome-panel {
-            width: min(720px, 100%);
-            max-height: min(720px, calc(100vh - 190px));
-            overflow: auto;
-            border-radius: 8px;
-            border: 1px solid #d1d5db;
-            background: Canvas;
-            color: CanvasText;
-            box-shadow: 0 20px 60px rgba(0, 0, 0, 0.32);
-            padding: 24px;
-            font: 14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
-        }
-        .opencode-agent-welcome-panel h1 {
-            margin: 0 0 10px;
-            font-size: 22px;
-            line-height: 1.25;
-        }
-        .opencode-agent-welcome-panel p {
-            margin: 0 0 14px;
-        }
-        .opencode-agent-welcome-panel ul {
-            margin: 0 0 16px 20px;
-            padding: 0;
-        }
-        .opencode-agent-welcome-panel li {
-            margin: 0 0 8px;
-        }
-        .opencode-agent-welcome-warning {
-            margin: 16px 0;
-            padding: 12px 14px;
-            border: 1px solid #f59e0b;
-            border-radius: 6px;
-            background: color-mix(in srgb, #f59e0b 12%, Canvas);
-        }
-        .opencode-agent-welcome-actions {
-            display: flex;
-            justify-content: flex-end;
-            gap: 8px;
-        }
-        .opencode-agent-welcome button {
-            cursor: pointer;
-            border: 1px solid #2563eb;
-            border-radius: 6px;
-            background: #2563eb;
-            color: #fff;
-            padding: 8px 14px;
-            font-weight: 600;
-        }
-        @media (max-width: 700px) {
-            .opencode-agent-welcome {
-                padding: 12px;
-            }
-            .opencode-agent-welcome-panel {
-                max-height: calc(100vh - 170px);
-                padding: 18px;
-            }
-        }
-    </style>
-{% endblock %}
-
-{% block bodyclass %}{{ block.super }} opencode-agent{% endblock %}
-
-{% block content %}
-    <div id="content-main">
-        {% if error %}
-            <div class="opencode-agent-error">
-                {{ error }}
-                {% if command %}<br><br><code>{{ command }}</code>{% endif %}
-            </div>
-        {% else %}
-            <script>
-                (() => {
-                    const workdir = "{{ workdir|escapejs }}";
-                    const recentSessionId = "{{ recent_session_id|escapejs }}";
-                    const merge = (left, right) => {
-                        if (!left || typeof left !== "object" || Array.isArray(left)) return right;
-                        const out = {...left};
-                        for (const [key, value] of Object.entries(right)) {
-                            out[key] = value && typeof value === "object" && !Array.isArray(value) ? merge(out[key], value) : value;
-                        }
-                        return out;
-                    };
-                    const seed = (key, value) => {
-                        try {
-                            localStorage.setItem(key, JSON.stringify(merge(JSON.parse(localStorage.getItem(key) || "{}"), value)));
-                        } catch {
-                            localStorage.setItem(key, JSON.stringify(value));
-                        }
-                    };
-                    const setJSON = (key, value) => localStorage.setItem(key, JSON.stringify(value));
-                    const page = {
-                        activeProject: workdir,
-                        activeWorkspace: workdir,
-                        gettingStartedDismissed: true,
-                        workspaceExpanded: {[workdir]: true},
-                    };
-                    if (recentSessionId) {
-                        page.lastProjectSession = {[workdir]: {directory: workdir, id: recentSessionId, at: Date.now()}};
-                    }
-                    seed("opencode.global.dat:layout.page", page);
-                    seed("opencode.global.dat:layout", {
-                        sidebar: {opened: true, width: 280},
-                        fileTree: {opened: true, width: 280, tab: "all"},
-                    });
-                    const serverKey = `${location.origin}/admin/agent/opencode`;
-                    setJSON("opencode.global.dat:server", {
-                        list: [],
-                        projects: {
-                            [serverKey]: [{worktree: workdir, expanded: true}],
-                        },
-                        lastProject: {
-                            [serverKey]: workdir,
-                        },
-                    });
-                    localStorage.setItem("opencode.settings.dat:defaultServerUrl", serverKey);
-                })();
-            </script>
-            <div class="opencode-agent-shell">
-                <iframe src="{{ proxy_url }}" title="OpenCode Agent" class="opencode-agent-frame"></iframe>
-                <div class="opencode-agent-welcome" id="opencode-agent-welcome" hidden>
-                    <div class="opencode-agent-welcome-panel" role="dialog" aria-modal="true" aria-labelledby="opencode-agent-welcome-title">
-                        <h1 id="opencode-agent-welcome-title">ArchiveBox AI Agent</h1>
-                        <p>This agent can work directly with your ArchiveBox collection. It can inspect the crawl database, create and monitor crawls, run maintenance operations, answer research questions, query archived data, and help with complex collection workflows.</p>
-                        <p>Example prompts:</p>
-                        <ul>
-                            <li>Archive this list of URLs with depth 0, then report which ones failed and why.</li>
-                            <li>Find snapshots from the last month that failed PDF or screenshot extraction and retry them.</li>
-                            <li>Create a constrained crawl for this site, avoid login/privacy/sitemap URLs, and watch the logs as it runs.</li>
-                            <li>Summarize what my collection contains about this topic and link to the best saved pages.</li>
-                        </ul>
-                        <div class="opencode-agent-welcome-warning">
-                            The agent has unrestricted access to this collection and can make destructive edits. Review requests carefully. Permission settings can be changed from the OpenCode gear icon in the lower left.
-                        </div>
-                        <div class="opencode-agent-welcome-actions">
-                            <button type="button" id="opencode-agent-welcome-dismiss">Start using Agent</button>
-                        </div>
-                    </div>
-                </div>
-            </div>
-            <script>
-                (() => {
-                    const key = "archivebox.opencode.agentWelcomeDismissed.v1";
-                    const welcome = document.getElementById("opencode-agent-welcome");
-                    const dismiss = document.getElementById("opencode-agent-welcome-dismiss");
-                    if (!welcome || !dismiss) return;
-                    if (localStorage.getItem(key) !== "1") {
-                        welcome.hidden = false;
-                    }
-                    dismiss.addEventListener("click", () => {
-                        localStorage.setItem(key, "1");
-                        welcome.hidden = true;
-                    });
-                })();
-            </script>
-        {% endif %}
-    </div>
-{% endblock %}

archivebox/opencode/views.py

@@ -1,716 +1,140 @@
-from __future__ import annotations
+"""Django adapter for the optional plugin; never import its runtime at startup."""
 
-import atexit
-import base64
-import os
-import re
-import shutil
-import subprocess
-import threading
-import time
-from pathlib import Path
-from typing import Any
-from urllib.parse import urljoin, urlsplit
+import logging
+from io import BytesIO
 
-import httpx
-import requests
-from abx_plugins.plugins import opencode as opencode_plugin
-from archivebox.config import CONSTANTS
-from archivebox.config.common import get_config
-from archivebox.core.routes_util import build_admin_url, get_api_base_url, get_base_url
-from django.http import (
-    Http404,
-    HttpRequest,
-    HttpResponse,
-    HttpResponseForbidden,
-    StreamingHttpResponse,
-)
-from django.shortcuts import redirect, render
+from asgiref.sync import sync_to_async
+
+from django.contrib.auth.middleware import AuthenticationMiddleware
+from django.contrib.auth.views import redirect_to_login
+from django.contrib.sessions.middleware import SessionMiddleware
+from django.core.exceptions import PermissionDenied
+from django.core.handlers.asgi import ASGIRequest
+from django.db import close_old_connections
+from django.http import Http404, HttpResponse, HttpResponseForbidden, StreamingHttpResponse
+from django.template import engines
+from django.urls import Resolver404, resolve
 from django.views.decorators.csrf import csrf_exempt
 
+from archivebox.config import CONSTANTS
+from archivebox.config.common import get_request_config
+from archivebox.core.middleware import ReverseProxyAuthMiddleware
+from archivebox.core.routes_util import build_admin_url, get_admin_host, get_api_base_url, get_base_url, host_matches
+from archivebox.plugins.discovery import get_plugin_template
 
-_PROCESS: subprocess.Popen | None = None
-_PROCESS_LOCK = threading.Lock()
-_PROXY_PREFIX = "/admin/agent/opencode"
-_PROXY_PREFIX_REGEX = _PROXY_PREFIX.replace("/", r"\/")
-_PROXY_PREFIX_NO_SLASH_REGEX = _PROXY_PREFIX.lstrip("/").replace("/", r"\/")
-_CONFIG_PATH = Path(opencode_plugin.__file__).with_name("config.json")
-
-_TEXT_CONTENT_TYPES = (
-    "text/",
-    "application/javascript",
-    "application/json",
-    "application/x-javascript",
-)
-_HOP_BY_HOP_HEADERS = {
-    "connection",
-    "keep-alive",
-    "proxy-authenticate",
-    "proxy-authorization",
-    "te",
-    "trailers",
-    "transfer-encoding",
-    "upgrade",
-}
-_ARCHIVEBOX_SKILL = """---
-name: archivebox
-description: Use ArchiveBox's CLI and local REST API from an ArchiveBox collection.
----
-
-You are running inside an ArchiveBox collection directory.
-
-- ArchiveBox collection directory: {archivebox_data_dir}
-- ArchiveBox BASE_URL: {archivebox_base_url}
-- ArchiveBox Admin URL: {archivebox_admin_url}
-- ArchiveBox REST API URL: {archivebox_api_url}
-- Prefer the `archivebox` CLI for authenticated changes, e.g. `archivebox add`, `archivebox schedule`, `archivebox update`, and `archivebox shell`.
-- Run ArchiveBox CLI commands from the ArchiveBox collection directory above.
-- Get command help with `archivebox list --help`, `archivebox add --help`, `archivebox schedule --help`, etc. Do not use `archivebox help <command>`.
-- Use `--depth=0` by default. Only use recursive crawling when the user explicitly asks for it; use `--depth=1` when you need pages one hop out.
-- Before any recursive crawl, constrain scope with ArchiveBox config such as `CRAWL_MAX_URLS`, `CRAWL_MAX_SIZE`, `SNAPSHOT_MAX_*`, `URL_ALLOWLIST`, `URL_DENYLIST`, and related limits.
-- Respect the configured `archivebox config --get ONLY_NEW` behavior unless the user explicitly says otherwise. Remind users that expected crawl URLs can be skipped when the collection already contains snapshots with the same URL.
-- Always audit newly discovered crawl URLs before letting a crawl run broadly. Treat junk URLs such as privacy policies, legal pages, tag archives, sitemap files, feeds, login/logout URLs, and other low-value boilerplate as unwanted unless the user explicitly asked to archive them.
-- Always watch crawl output and logs as the crawl progresses, and correct errors early instead of waiting until the crawl finishes.
-- If a crawl contains bad URLs, pause it, edit the crawl's `urls` field to remove them, delete any unneeded snapshots already created under that crawl, then resume the crawl.
-- Use `archivebox shell -c '...'` or `archivebox shell <<'PY' ... PY` for Django ORM work. Shell Plus prints an import banner first; keep stderr visible while debugging.
-- Use full ArchiveBox module paths in shell code: `from archivebox.crawls.models import Crawl, CrawlSchedule` and `from archivebox.core.models import Snapshot, ArchiveResult`.
-- If a model/field/relation is unclear, inspect `_meta.fields` before guessing, e.g. `archivebox shell -c "from archivebox.crawls.models import Crawl; print([f.name for f in Crawl._meta.fields])"`.
-- Use `archivebox config --get BASE_URL` only to verify the configured base URL; prefer the seeded URLs above for API/admin requests.
-- Use `$ARCHIVEBOX_API_URL` for REST API inspection when helpful. Do not assume admin session cookies authenticate API subdomain requests; prefer CLI/shell for authenticated mutations unless the admin provides or asks you to create an API token.
-- Discover REST endpoints from `${{ARCHIVEBOX_API_URL}}v1/openapi.json`; crawl endpoints live under `/api/v1/crawls/`, snapshots under `/api/v1/core/`.
-- Do not bypass ArchiveBox auth, expose API keys, or modify config unless the admin explicitly asks.
-- After creating crawls or snapshots, report the crawl/snapshot IDs and the exact command or API request used.
-"""
+_LOGGER = logging.getLogger(__name__)
 
 
-def _stop_owned_process(process: subprocess.Popen | None = None) -> None:
-    global _PROCESS
-    owned_process = process or _PROCESS
-    if owned_process is None:
-        return
-    if owned_process.poll() is None:
-        owned_process.terminate()
-        try:
-            owned_process.wait(timeout=5)
-        except subprocess.TimeoutExpired:
-            owned_process.kill()
-            owned_process.wait()
-    if _PROCESS is owned_process:
-        _PROCESS = None
+def _runtime_settings(request, config):
+    from abx_plugins.plugins.opencode import runtime
 
-
-atexit.register(_stop_owned_process)
-
-
-def _machine_config() -> dict[str, Any]:
-    resolved = get_config()
-    return dict(resolved.model_dump(mode="json"))
-
-
-def _archivebox_data_dir_default() -> Path:
-    return Path(CONSTANTS.DATA_DIR)
-
-
-def _archivebox_route_urls(request: HttpRequest, route_config) -> tuple[str, str, str]:
-    base_url = get_base_url(
-        request=request,
-        config=route_config,
-    ).rstrip("/")
-    admin_url = build_admin_url(
-        "/admin/",
-        request=request,
-        config=route_config,
-    ).rstrip("/")
-    api_url = f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/"
-    return base_url, admin_url, api_url
-
-
-def _config_value(config: dict, key: str, default):
-    value = config.get(key, default)
-    if value in (None, ""):
-        return default
-    return value
-
-
-def _opencode_enabled(config: dict) -> bool:
-    value = config.get("OPENCODE_ENABLED", False)
-    if isinstance(value, bool):
-        return value
-    return str(value).strip().lower() in {"1", "true", "yes", "on"}
-
-
-def _require_enabled(config: dict) -> None:
-    if not _opencode_enabled(config):
-        raise Http404
-
-
-def _require_superuser(request: HttpRequest):
-    user = getattr(request, "user", None)
-    if user is None:
-        return redirect(f"/admin/login/?next={request.get_full_path()}")
-    if (
-        bool(getattr(user, "is_authenticated", False))
-        and bool(getattr(user, "is_active", False))
-        and bool(getattr(user, "is_superuser", False))
-    ):
-        return None
-    if bool(getattr(user, "is_authenticated", False)):
-        return HttpResponseForbidden(
-            b"ArchiveBox agent access requires a superuser account.",
-        )
-    return redirect(f"/admin/login/?next={request.get_full_path()}")
-
-
-def _origin_allowed(request: HttpRequest, path: str | None = None) -> bool:
-    if request.method in {"GET", "HEAD", "OPTIONS", "TRACE"}:
-        return True
-
-    expected_host = request.get_host()
-    pty_connect = bool(
-        path and path.startswith("pty/") and path.endswith("/connect-token"),
-    )
-    if pty_connect:
-        return True
-
-    origin = _request_header(request, "Origin")
-    if origin:
-        return _same_host(origin, expected_host)
-
-    referer = _request_header(request, "Referer")
-    if referer:
-        return _same_host(referer, expected_host)
-
-    fetch_site = _request_header(request, "Sec-Fetch-Site")
-    if fetch_site:
-        return fetch_site in {"same-origin", "same-site", "none"}
-
-    return False
-
-
-def _same_host(value: str, expected_host: str) -> bool:
-    parsed = urlsplit(value)
-    return parsed.scheme in {"http", "https"} and parsed.netloc == expected_host
-
-
-def _settings(config: dict) -> dict:
-    host = str(_config_value(config, "OPENCODE_HOST", "127.0.0.1"))
-    port = int(_config_value(config, "OPENCODE_PORT", 4096))
-    default_data_dir = _archivebox_data_dir_default()
-    workdir = Path(
-        str(_config_value(config, "OPENCODE_WORKDIR", default_data_dir)),
-    ).expanduser()
-    opencode_dir = Path(
-        str(_config_value(config, "OPENCODE_STATE_DIR", workdir / "opencode")),
-    ).expanduser()
-    binary = str(_config_value(config, "OPENCODE_BINARY", "opencode"))
-    timeout = int(_config_value(config, "OPENCODE_TIMEOUT", 30))
-    return {
-        "host": host,
-        "port": port,
-        "origin": f"http://{host}:{port}",
-        "workdir": workdir,
-        "opencode_dir": opencode_dir,
-        "config_home": opencode_dir / "config",
-        "data_home": opencode_dir / "data",
-        "state_home": opencode_dir / "state",
-        "cache_home": opencode_dir / "cache",
-        "home": opencode_dir / "home",
-        "binary": binary,
-        "config": config,
-        "timeout": timeout,
-    }
-
-
-def _resolve_binary(binary: str, config: dict) -> tuple[Any, Any, dict[str, str]]:
-    try:
-        from abxpkg import BinProvider
-        from abx_plugins.plugins.base.utils import load_required_binary_from_config
-
-        binary_environ = os.environ.copy()
-        lib_dir = config.get("ABXPKG_LIB_DIR")
-        if lib_dir:
-            binary_environ["ABXPKG_LIB_DIR"] = str(lib_dir)
-        loaded_dependencies = [
-            load_required_binary_from_config(
-                required_binary,
-                _CONFIG_PATH,
-                global_config=config,
-                environ=binary_environ,
-                install=False,
-            )
-            for required_binary in (
-                str(config.get("NODE_BINARY") or "node"),
-                str(config.get("GIT_BINARY") or "git"),
-                binary,
-            )
-        ]
-    except Exception as err:
-        raise RuntimeError(
-            f"OpenCode dependency is not installed from required_binaries: {err}",
-        ) from err
-
-    if any(not loaded.loaded_abspath for loaded in loaded_dependencies):
-        raise RuntimeError(
-            "OpenCode dependency is not installed from required_binaries.",
-        )
-
-    providers = [loaded.loaded_binprovider for loaded in loaded_dependencies if loaded.loaded_binprovider is not None]
-    binary_env = BinProvider.build_exec_env(
-        providers=providers,
-        base_env=binary_environ,
-    )
-    return (
-        loaded_dependencies[-1],
-        loaded_dependencies[1],
-        binary_env,
-    )
-
-
-def _project_route(workdir: Path, session_id: str = "") -> str:
-    encoded = base64.b64encode(str(workdir.resolve()).encode()).decode()
-    encoded = encoded.replace("+", "-").replace("/", "_").rstrip("=")
-    route = f"{_PROXY_PREFIX}/{encoded}/session"
-    return f"{route}/{session_id}" if session_id else route
-
-
-def _ensure_project_files(settings: dict) -> None:
-    workdir = settings["workdir"].resolve()
-    workdir.mkdir(parents=True, exist_ok=True)
-    git_marker = workdir / ".git" / "not-a-git"
-    if git_marker.exists():
-        # Current OpenCode hangs on the legacy fake marker, so remove only
-        # that invalid shape before initializing the real worktree.
-        shutil.rmtree(git_marker.parent)
-
-    editable_skill_path = settings["opencode_dir"] / "SKILL.md"
-    editable_skill_path.parent.mkdir(parents=True, exist_ok=True)
-    if not editable_skill_path.exists():
-        editable_skill_path.write_text(
-            _ARCHIVEBOX_SKILL.format(
-                archivebox_data_dir=workdir,
-                archivebox_base_url=settings.get("archivebox_base_url", ""),
-                archivebox_admin_url=settings.get("archivebox_admin_url", ""),
-                archivebox_api_url=settings.get("archivebox_api_url", ""),
-            ),
-        )
-
-    opencode_skill_path = settings["config_home"] / "opencode" / "skills" / "archivebox" / "SKILL.md"
-    opencode_skill_path.parent.mkdir(parents=True, exist_ok=True)
-    if opencode_skill_path.resolve() != editable_skill_path.resolve():
-        if opencode_skill_path.exists() or opencode_skill_path.is_symlink():
-            opencode_skill_path.unlink()
-        opencode_skill_path.symlink_to(editable_skill_path)
-
-
-def _ensure_default_session(settings: dict) -> str:
-    workdir = settings["workdir"].resolve()
-    params = {"directory": str(workdir)}
-    timeout = settings["timeout"]
-    project = requests.post(
-        f"{settings['origin']}/project/git/init",
-        params=params,
-        timeout=timeout,
-    )
-    project.raise_for_status()
-    project_data = project.json()
-    if Path(str(project_data.get("worktree") or "/")).resolve() != workdir:
-        raise RuntimeError(
-            f"OpenCode initialized the wrong project worktree: {project_data.get('worktree')!r}",
-        )
-
-    sessions = requests.get(
-        f"{settings['origin']}/session",
-        params={**params, "roots": "true", "limit": 55},
-        timeout=timeout,
-    )
-    sessions.raise_for_status()
-    session_data = sessions.json()
-    if not isinstance(session_data, list):
-        raise RuntimeError("OpenCode returned an invalid project session list.")
-    for session_data_item in session_data:
-        if not isinstance(session_data_item, dict):
-            continue
-        session_id = str(session_data_item.get("id") or "")
-        session_directory = session_data_item.get("directory")
-        if session_id and session_directory and Path(str(session_directory)).resolve() == workdir:
-            return session_id
-
-    session = requests.post(
-        f"{settings['origin']}/session",
-        params=params,
-        json={},
-        timeout=timeout,
-    )
-    session.raise_for_status()
-    session_data = session.json()
-    session_id = str(session_data.get("id") or "")
-    session_directory = session_data.get("directory")
-    if not session_id or not session_directory or Path(str(session_directory)).resolve() != workdir:
-        raise RuntimeError(
-            "OpenCode did not create a session for the requested worktree.",
-        )
-    return session_id
-
-
-def _recent_session_id(settings: dict) -> str:
-    workdir = str(settings["workdir"].resolve())
-    try:
-        response = requests.get(
-            f"{settings['origin']}/session",
-            params={"directory": workdir, "roots": "true", "limit": 1},
-            timeout=settings["timeout"],
-        )
-        response.raise_for_status()
-        sessions = response.json()
-        if sessions:
-            return str(sessions[0].get("id") or "")
-    except requests.RequestException:
-        pass
-    return ""
-
-
-def _health(settings: dict) -> bool:
-    try:
-        response = requests.get(
-            f"{settings['origin']}/global/health",
-            timeout=2,
-        )
-        return response.status_code == 200
-    except requests.RequestException:
-        return False
-
-
-def _ensure_opencode(settings: dict) -> tuple[bool, str]:
-    global _PROCESS
-    started_process: subprocess.Popen | None = None
-    workdir = settings["workdir"].resolve()
-    try:
-        binary, git_binary, binary_env = _resolve_binary(
-            settings["binary"],
-            settings["config"],
-        )
-    except RuntimeError as err:
-        return False, str(err)
-
-    env = {
-        **os.environ,
-        **binary_env,
-        "ARCHIVEBOX_BASE_URL": str(settings.get("archivebox_base_url", "")),
-        "ARCHIVEBOX_ADMIN_URL": str(settings.get("archivebox_admin_url", "")),
-        "ARCHIVEBOX_API_URL": str(settings.get("archivebox_api_url", "")),
-        "BROWSER": "false",
-        "GIT_CEILING_DIRECTORIES": str(workdir),
-        "HOME": str(settings["home"]),
-        "OPENCODE_DISABLE_PROJECT_CONFIG": "true",
-        "XDG_CONFIG_HOME": str(settings["config_home"]),
-        "XDG_DATA_HOME": str(settings["data_home"]),
-        "XDG_STATE_HOME": str(settings["state_home"]),
-        "XDG_CACHE_HOME": str(settings["cache_home"]),
-    }
-
-    with _PROCESS_LOCK:
-        settings["workdir"].mkdir(parents=True, exist_ok=True)
-        settings["config_home"].mkdir(parents=True, exist_ok=True)
-        settings["data_home"].mkdir(parents=True, exist_ok=True)
-        settings["state_home"].mkdir(parents=True, exist_ok=True)
-        settings["cache_home"].mkdir(parents=True, exist_ok=True)
-        settings["home"].mkdir(parents=True, exist_ok=True)
-        _ensure_project_files(settings)
-
-        if not (workdir / ".git").exists():
-            try:
-                git_init = git_binary.exec(
-                    cmd=("init", "--quiet"),
-                    cwd=workdir,
-                    env=env,
-                    timeout=settings["timeout"],
-                )
-            except (AssertionError, OSError, subprocess.SubprocessError) as err:
-                return False, f"OpenCode project initialization failed: {err}"
-            if git_init.returncode != 0:
-                output = (git_init.stderr or git_init.stdout or "").strip()
-                return (
-                    False,
-                    f"OpenCode project initialization failed: {output or f'git exited with {git_init.returncode}'}",
-                )
-
-        if _health(settings):
-            try:
-                _ensure_default_session(settings)
-            except (requests.RequestException, RuntimeError, ValueError) as err:
-                return False, f"OpenCode project initialization failed: {err}"
-            return True, ""
-
-        binary_abspath = binary.loaded_abspath
-        if binary.loaded_binprovider is not None:
-            binary_abspath = binary.loaded_binprovider._exec_bin_abspath(
-                Path(binary.loaded_abspath),
-            )
-        cmd = [
-            str(binary_abspath),
-            "serve",
-            "--hostname",
-            settings["host"],
-            "--port",
-            str(settings["port"]),
-        ]
-        try:
-            _PROCESS = subprocess.Popen(
-                cmd,
-                cwd=workdir,
-                env=env,
-                stdin=subprocess.DEVNULL,
-                stdout=subprocess.DEVNULL,
-                stderr=subprocess.DEVNULL,
-                start_new_session=True,
-            )
-            started_process = _PROCESS
-        except FileNotFoundError:
-            return False, f"OpenCode binary not found: {settings['binary']}"
-
-        deadline = time.monotonic() + settings["timeout"]
-        while time.monotonic() < deadline:
-            if _health(settings):
-                try:
-                    _ensure_default_session(settings)
-                except (requests.RequestException, RuntimeError, ValueError) as err:
-                    _stop_owned_process(started_process)
-                    return False, f"OpenCode project initialization failed: {err}"
-                return True, ""
-            if started_process and started_process.poll() is not None:
-                if _PROCESS is started_process:
-                    _PROCESS = None
-                return False, "OpenCode exited before the web server became ready."
-            time.sleep(0.25)
-
-        _stop_owned_process(started_process)
-        return False, "Timed out waiting for OpenCode to start."
-
-
-def agent_view(request: HttpRequest):
-    config = _machine_config()
-    _require_enabled(config)
-    auth_response = _require_superuser(request)
-    if auth_response:
-        return auth_response
-
-    settings = _settings(config)
+    settings = runtime._settings(config, CONSTANTS.DATA_DIR)
     route_config = request.__dict__.get("archivebox_config")
-    base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
-    settings["archivebox_base_url"] = base_url
-    settings["archivebox_admin_url"] = admin_url
-    settings["archivebox_api_url"] = api_url
-    ok, error = _ensure_opencode(settings)
-    from archivebox.core.admin_site import archivebox_admin
-
-    recent_session_id = _recent_session_id(settings) if ok else ""
-    context = {
-        **archivebox_admin.each_context(request),
-        "title": "Agent",
-        "error": "" if ok else error,
-        "command": f"{settings['binary']} serve --hostname {settings['host']} --port {settings['port']}" if error else "",
-        # OpenCode 1.17+ keeps durable sessions on the explicit
-        # /<dirBase64>/session/<sessionId> route. We still seed localStorage
-        # below because the sidebar state uses it, but the iframe itself must
-        # open the durable session URL so a fresh browser does not land on the
-        # transient new-session route and appear to have lost prior sessions.
-        "proxy_url": _project_route(settings["workdir"], recent_session_id),
-        "workdir": str(settings["workdir"].resolve()),
-        "recent_session_id": recent_session_id,
-    }
-    return render(
-        request,
-        "opencode/agent.html",
-        context,
-        status=200 if ok else 502,
+    settings.update(
+        archivebox_base_url=get_base_url(request=request, config=route_config).rstrip("/"),
+        archivebox_admin_url=build_admin_url("/admin/", request=request, config=route_config).rstrip("/"),
+        archivebox_api_url=f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/",
     )
+    return runtime, settings
 
 
-def _proxy_url(settings: dict, path: str | None) -> str:
-    rel = "/" if not path else f"/{path}"
-    return urljoin(settings["origin"], rel)
+def _dispatch(request, path=None):
+    try:
+        # Middleware already merged server defaults and live Machine overrides.
+        # Re-resolving every extractor's config here stalls the UI request burst.
+        config = get_request_config(request).model_dump(mode="json")
+        if not config.get("OPENCODE_ENABLED", False):
+            raise Http404
+        if not request.user.is_authenticated:
+            return redirect_to_login(request.get_full_path(), login_url="/admin/login/")
+        if not request.user.is_active or not request.user.is_superuser:
+            return HttpResponseForbidden("Agent access requires a superuser account.")
+
+        runtime, settings = _runtime_settings(request, config)
+        if path is None:
+            from archivebox.core.admin_site import archivebox_admin
+
+            context = {**archivebox_admin.each_context(request), **runtime.agent_context(settings)}
+            source = get_plugin_template("opencode", "agent", fallback=False)
+            if source is None:
+                raise RuntimeError("Agent template unavailable")
+            return HttpResponse(engines["django"].from_string(source).render(context, request))
+
+        if not runtime._origin_allowed(request.method, request.get_host(), request.headers):
+            return HttpResponseForbidden("Cross-origin agent requests are blocked.")
+        status, headers, body = runtime.proxy(
+            settings,
+            request.method,
+            path,
+            tuple((key, value) for key, values in request.GET.lists() for value in values),
+            request.headers,
+            request.body,
+        )
+        response_type = HttpResponse if isinstance(body, bytes) else StreamingHttpResponse
+        response = response_type(body, status=status, headers=headers)
+        response.xframe_options_exempt = True
+        response.headers["X-Frame-Options"] = "SAMEORIGIN"
+        response.headers["Content-Security-Policy"] = "frame-ancestors 'self'"
+        return response
+    except Http404:
+        raise
+    except Exception:
+        # Optional-service boundary, including imports and template rendering.
+        _LOGGER.exception("Optional AI service failed")
+        return HttpResponse("AI service unavailable. See server logs.", status=503, content_type="text/plain")
 
 
-def _request_header(request: HttpRequest, name: str) -> str | None:
-    meta = getattr(request, "META", {})
-    if name == "Content-Type":
-        value = meta.get("CONTENT_TYPE")
-    elif name == "Content-Length":
-        value = meta.get("CONTENT_LENGTH")
-    else:
-        value = meta.get(f"HTTP_{name.upper().replace('-', '_')}")
-    return str(value) if value else None
-
-
-def _request_headers(request: HttpRequest, settings: dict) -> dict[str, str]:
-    forwarded = {}
-    for key in ("Accept", "Accept-Language", "Content-Type", "Range", "User-Agent"):
-        value = _request_header(request, key)
-        if value:
-            forwarded[key] = value
-    return forwarded
-
-
-def _request_params(request: HttpRequest) -> tuple[tuple[str, str], ...]:
-    if hasattr(request.GET, "lists"):
-        return tuple((key, str(value)) for key, values in request.GET.lists() for value in values)
-    return tuple((key, str(value)) for key, value in dict(request.GET).items())
-
-
-async def _event_chunks(request: HttpRequest, settings: dict, path: str | None):
-    timeout = httpx.Timeout(settings["timeout"], read=None)
-    url = _proxy_url(settings, path)
-    method = request.method or "GET"
-    async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
-        async with client.stream(
-            method,
-            url,
-            params=_request_params(request),
-            headers=_request_headers(request, settings),
-        ) as upstream:
-            async for chunk in upstream.aiter_raw(chunk_size=512):
-                yield chunk
-
-
-def _rewrite_text(body: bytes, settings: dict) -> bytes:
-    text = body.decode("utf-8", errors="replace")
-    text = text.replace(settings["origin"], _PROXY_PREFIX)
-    text = text.replace("location.origin", f'location.origin+"{_PROXY_PREFIX}"')
-    text = text.replace(
-        "k(k5,{get component(){return t.router??Az},",
-        f'k(k5,{{base:"{_PROXY_PREFIX}",get component(){{return t.router??Az}},',
-    )
-    text = text.replace('"/assets/', f'"{_PROXY_PREFIX}/assets/')
-    text = text.replace("'/assets/", f"'{_PROXY_PREFIX}/assets/")
-    proxy_path = rf'(\1.replace(/^{_PROXY_PREFIX_REGEX}(?=\/|$)/,"")||"/")'
-    text = re.sub(r"\b(window\.location\.pathname)\b", proxy_path, text)
-    text = re.sub(r"(?<![.\w])(location\.pathname)\b", proxy_path, text)
-    text = text.replace(
-        'window.history.replaceState(nz(o),"",r):window.history.pushState(o,"",r)',
-        (
-            f'window.history.replaceState(nz(o),"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r):'
-            f'window.history.pushState(o,"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r)'
-        ),
-    )
-    text = text.replace(
-        'const BL="modulepreload",UL=function(t){return"/"+t}',
-        f'const BL="modulepreload",UL=function(t){{return"{_PROXY_PREFIX}/"+t}}',
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\b(?:href|src|action)=["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\b(?:fetch|EventSource)\(["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\burl\(["']?)/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    return text.encode("utf-8")
-
-
-def _response_headers(upstream: requests.Response, settings: dict) -> dict[str, str]:
-    headers = {}
-    for key, value in upstream.headers.items():
-        lower = key.lower()
-        if lower in _HOP_BY_HOP_HEADERS or lower in {
-            "content-length",
-            "content-encoding",
-            "x-frame-options",
-        }:
-            continue
-        if lower == "location":
-            if value.startswith(settings["origin"]):
-                value = value.replace(settings["origin"], _PROXY_PREFIX, 1)
-            elif value.startswith("/"):
-                value = f"{_PROXY_PREFIX}{value}"
-        headers[key] = value
-    return headers
+def agent_view(request):
+    return _dispatch(request)
 
 
 @csrf_exempt
-def opencode_proxy_view(request: HttpRequest, path: str | None = None):
-    config = _machine_config()
-    _require_enabled(config)
-    auth_response = _require_superuser(request)
-    if auth_response:
-        return auth_response
-    if not _origin_allowed(request, path):
-        return HttpResponseForbidden(
-            b"Cross-origin OpenCode agent requests are blocked.",
-        )
+def opencode_proxy_view(request, path=None):
+    return _dispatch(request, path=path or "")
 
-    settings = _settings(config)
-    route_config = request.__dict__.get("archivebox_config")
-    base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
-    settings["archivebox_base_url"] = base_url
-    settings["archivebox_admin_url"] = admin_url
-    settings["archivebox_api_url"] = api_url
-    ok, error = _ensure_opencode(settings)
-    if not ok:
-        return HttpResponse(
-            error.encode(),
-            status=502,
-            content_type="text/plain; charset=utf-8",
-        )
-
-    if request.method == "GET" and (path or "").endswith("/event"):
-        response = StreamingHttpResponse(
-            _event_chunks(request, settings, path),
-            content_type="text/event-stream",
-        )
-        response.headers["Cache-Control"] = "no-store"
-        response.headers["X-Accel-Buffering"] = "no"
-        return response
 
+def _websocket_context(scope):
+    close_old_connections()
     try:
-        method = request.method or "GET"
-        upstream = requests.request(
-            method,
-            _proxy_url(settings, path),
-            params=_request_params(request),
-            data=request.body if method not in {"GET", "HEAD"} else None,
-            headers=_request_headers(request, settings),
-            stream=True,
-            timeout=(settings["timeout"], None),
-            allow_redirects=False,
-        )
-    except requests.RequestException as err:
-        return HttpResponse(
-            str(err).encode(),
-            status=502,
-            content_type="text/plain; charset=utf-8",
+        match = resolve(scope["path"])
+        if match.url_name != "opencode-proxy":
+            raise PermissionDenied
+        request = ASGIRequest(
+            {**scope, "type": "http", "method": "GET", "scheme": "https" if scope.get("scheme") == "wss" else "http"},
+            BytesIO(),
         )
+        route_config = get_request_config(request, resolve_plugins=False)
+        request.archivebox_config = route_config
+        if not route_config.CONTROL_PLANE_ENABLED:
+            raise PermissionDenied
+        if (
+            route_config.USES_SUBDOMAIN_ROUTING
+            and route_config.BASE_URL
+            and not host_matches(request.get_host(), get_admin_host(config=route_config, request=request))
+        ):
+            raise PermissionDenied
+        SessionMiddleware(_dispatch).process_request(request)
+        AuthenticationMiddleware(_dispatch).process_request(request)
+        ReverseProxyAuthMiddleware(_dispatch).process_request(request)
+        config = route_config.model_dump(mode="json")
+        if not config.get("OPENCODE_ENABLED") or not request.user.is_active or not request.user.is_superuser:
+            raise PermissionDenied
+        runtime, settings = _runtime_settings(request, config)
+        if not request.headers.get("Origin") or not runtime._origin_allowed("POST", request.get_host(), request.headers):
+            raise PermissionDenied
+        return runtime, settings, match.kwargs.get("path", "")
+    finally:
+        close_old_connections()
 
-    content_type = upstream.headers.get("Content-Type", "")
-    is_event_stream = content_type.startswith("text/event-stream")
-    is_text = not is_event_stream and any(content_type.startswith(prefix) for prefix in _TEXT_CONTENT_TYPES)
-    headers = _response_headers(upstream, settings)
-    if is_text:
-        body = _rewrite_text(upstream.content, settings)
-        response = HttpResponse(
-            body,
-            status=upstream.status_code,
-            content_type=content_type or "text/plain; charset=utf-8",
-        )
-    elif is_event_stream:
-        response = StreamingHttpResponse(
-            upstream.iter_lines(chunk_size=1),
-            status=upstream.status_code,
-            content_type=content_type or "text/event-stream",
-        )
-    else:
-        response = StreamingHttpResponse(
-            upstream.iter_content(chunk_size=64 * 1024),
-            status=upstream.status_code,
-            content_type=content_type or "application/octet-stream",
-        )
-    for key, value in headers.items():
-        response.headers[key] = value
-    response.headers["Cache-Control"] = "no-store"
-    return response
+
+async def websocket_view(scope, receive, send):
+    if (await receive())["type"] != "websocket.connect":
+        return
+    try:
+        runtime, settings, path = await sync_to_async(_websocket_context)(scope)
+        await runtime.websocket_proxy(settings, path, scope.get("query_string", b""), scope.get("subprotocols", []), receive, send)
+    except (PermissionDenied, Resolver404, Http404):
+        await send({"type": "websocket.close", "code": 1008})
+    except Exception:
+        _LOGGER.exception("Optional AI WebSocket failed")
+        await send({"type": "websocket.close", "code": 1011})

archivebox/personas/admin.py

@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
             "Persona",
             {
                 "fields": ("name", "created_by", "permissions"),
-                "classes": ("card", "persona-card-primary"),
+                "classes": ("card", "wide", "persona-card-primary"),
             },
         ),
         (
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
                     "import_extract_cookies",
                     "import_capture_storage",
                 ),
-                "classes": ("card", "wide"),
+                "classes": ("card", "wide", "persona-card-import"),
             },
         ),
         (
@@ -63,7 +63,6 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
                 "classes": ("card", "persona-card-timestamps"),
             },
         ),
-        add_fieldsets[1],
         add_fieldsets[2],
         (
             "Artifacts",
@@ -136,7 +135,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
         return super().get_form(request, obj=obj, change=change, **kwargs)
 
     def render_change_form(self, request, context, add=False, change=False, form_url="", obj=None):
-        context["detected_profile_count"] = len(discover_local_browser_profiles())
+        context["detected_profile_count"] = len(discover_local_browser_profiles()) if add else 0
         return super().render_change_form(request, context, add=add, change=change, form_url=form_url, obj=obj)
 
     def save_model(self, request, obj, form, change):

archivebox/personas/export_browser_state.js

@@ -9,11 +9,11 @@
  *   COOKIES_OUTPUT_FILE         Optional output path for Netscape cookies.txt
  *   AUTH_STORAGE_OUTPUT_FILE    Optional output path for auth.json
  *   CHROME_BINARY               Absolute browser path resolved by abxpkg
+ *   CHROME_SANDBOX              Set to false when the Linux sandbox is unavailable
  *   NODE_MODULES_DIR            node_modules path resolved by abxpkg
  */
 
 const fs = require('fs');
-const os = require('os');
 const path = require('path');
 
 const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR || process.env.ABX_PLUGINS_DIR;
@@ -26,7 +26,7 @@ const baseUtils = require(path.join(pluginsDir, 'base', 'utils.js'));
 baseUtils.ensureNodeModuleResolution(module);
 
 const chromeUtils = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
-const puppeteer = require('puppeteer-core');
+const puppeteer = require('puppeteer');
 
 function cookieToNetscape(cookie) {
     let domain = cookie.domain;
@@ -115,38 +115,28 @@ async function openBrowser() {
         throw new Error(`User data directory does not exist: ${userDataDir}`);
     }
 
-    const outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'abx-browser-state-'));
     const binary = process.env.CHROME_BINARY;
     if (!binary || !path.isAbsolute(binary) || !fs.existsSync(binary)) {
         throw new Error('CHROME_BINARY must be an absolute executable path resolved by abxpkg');
     }
 
-    const launched = await chromeUtils.launchChromium({
-        binary,
-        outputDir,
+    // The ordinary archiving launcher uses a mock keychain. An import must
+    // instead let the original signed browser decrypt its own profile cookies.
+    const sandboxDisabled = ['false', '0', 'no', 'off'].includes(
+        String(process.env.CHROME_SANDBOX || '').trim().toLowerCase(),
+    );
+    const browser = await puppeteer.launch({
+        executablePath: binary,
         userDataDir,
         headless: true,
-        killZombies: false,
+        ignoreDefaultArgs: ['--use-mock-keychain', '--password-store=basic'],
+        args: ['--no-first-run', '--disable-sync', '--disable-background-networking',
+               '--profile-directory=Default', ...(process.platform === 'linux' && (process.getuid?.() === 0 || sandboxDisabled) ? ['--no-sandbox'] : [])],
     });
-
-    if (!launched.success) {
-        throw new Error(launched.error || 'Chrome launch failed');
-    }
-
-    const browser = await chromeUtils.connectToBrowserEndpoint(puppeteer, launched.cdpUrl, { defaultViewport: null });
-
     return {
         browser,
         async cleanup() {
-            try {
-                await browser.disconnect();
-            } catch (error) {}
-            try {
-                await chromeUtils.killChrome(launched.pid, outputDir);
-            } catch (error) {}
-            try {
-                fs.rmSync(outputDir, { recursive: true, force: true });
-            } catch (error) {}
+            await browser.close();
         },
         sourceDescription: userDataDir,
     };

archivebox/personas/forms.py

@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
         label="Autodiscovered profiles",
         widget=forms.RadioSelect,
         choices=(),
-        help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
+        help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
     )
     import_source = forms.CharField(
         required=False,

archivebox/personas/importers.py

@@ -11,10 +11,14 @@ import json
 import os
 import platform
 import shutil
+import sqlite3
+import time
+from http.cookiejar import MozillaCookieJar
 import subprocess
 import sys
 import tempfile
 from dataclasses import dataclass, field
+from functools import partial
 from pathlib import Path
 from typing import TYPE_CHECKING
 from urllib.parse import urlparse
@@ -28,6 +32,7 @@ if TYPE_CHECKING:
 
 BROWSER_LABELS = {
     "chrome": "Google Chrome",
+    "chrome-beta": "Google Chrome Beta",
     "chromium": "Chromium",
     "brave": "Brave",
     "edge": "Microsoft Edge",
@@ -42,25 +47,27 @@ BROWSER_PROFILE_DIR_NAMES = (
 )
 
 VOLATILE_PROFILE_COPY_PATTERNS = (
+    # Chromium/macOS atomic-write staging files (e.g.
+    # .com.brave.Browser.TransportSecurity.Ze7UBU) disappear after rename.
+    ".*.??????",
     "Cache",
     "Code Cache",
     "GPUCache",
     "ShaderCache",
-    "Service Worker",
-    "GCM Store",
-    "chrome-extension_*",
-    "DNR Extension Rules",
-    "Extension Rules",
-    "Extension Scripts",
-    "Extension State",
-    "Local Extension Settings",
-    "*.log",
     "Crashpad",
     "BrowserMetrics",
     "BrowserMetrics-spare.pma",
+    "RunningChromeVersion",
+    "DevToolsActivePort",
     "SingletonLock",
     "SingletonSocket",
     "SingletonCookie",
+    "Sessions",
+    "Sessions_Encrypted",
+    "Current Session",
+    "Current Tabs",
+    "Last Session",
+    "Last Tabs",
 )
 
 PERSONA_PROFILE_DIR_CANDIDATES = (
@@ -201,6 +208,7 @@ class PersonaImportResult:
     source: PersonaImportSource
     profile_copied: bool = False
     cookies_imported: bool = False
+    cookie_count: int = 0
     storage_captured: bool = False
     user_agent_imported: bool = False
     warnings: list[str] = field(default_factory=list)
@@ -210,28 +218,29 @@ class PersonaImportResult:
         return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
 
 
-def get_chrome_user_data_dir() -> Path | None:
+def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
     """Get the default Chrome user data directory for the current platform."""
     system = platform.system()
     home = Path.home()
+    app_name = "Chrome Beta" if beta else "Chrome"
 
     if system == "Darwin":
         candidates = [
-            home / "Library" / "Application Support" / "Google" / "Chrome",
-            home / "Library" / "Application Support" / "Chromium",
+            home / "Library" / "Application Support" / "Google" / app_name,
         ]
     elif system == "Linux":
-        candidates = [
-            home / ".config" / "google-chrome",
-            home / ".config" / "chromium",
-            home / ".config" / "chrome",
-            home / "snap" / "chromium" / "common" / "chromium",
-        ]
+        candidates = (
+            [home / ".config" / "google-chrome-beta"]
+            if beta
+            else [
+                home / ".config" / "google-chrome",
+                home / ".config" / "chrome",
+            ]
+        )
     elif system == "Windows":
         local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
         candidates = [
-            local_app_data / "Google" / "Chrome" / "User Data",
-            local_app_data / "Chromium" / "User Data",
+            local_app_data / "Google" / app_name / "User Data",
         ]
     else:
         candidates = []
@@ -243,6 +252,21 @@ def get_chrome_user_data_dir() -> Path | None:
     return None
 
 
+def get_chromium_user_data_dir() -> Path | None:
+    """Select Chromium's own profile even when Google Chrome is installed."""
+    home = Path.home()
+    if platform.system() == "Darwin":
+        candidates = [home / "Library/Application Support/Chromium"]
+    elif platform.system() == "Windows":
+        candidates = [Path(os.environ.get("LOCALAPPDATA", home / "AppData/Local")) / "Chromium/User Data"]
+    else:
+        candidates = [
+            Path(os.environ.get("XDG_CONFIG_HOME", home / ".config")) / "chromium",
+            home / "snap/chromium/common/chromium",
+        ]
+    return next((path for path in candidates if path.is_dir() and _list_profile_names(path)), None)
+
+
 def get_brave_user_data_dir() -> Path | None:
     """Get the default Brave user data directory for the current platform."""
     system = platform.system()
@@ -303,7 +327,8 @@ def get_edge_user_data_dir() -> Path | None:
 
 BROWSER_PROFILE_FINDERS = {
     "chrome": get_chrome_user_data_dir,
-    "chromium": get_chrome_user_data_dir,
+    "chrome-beta": partial(get_chrome_user_data_dir, beta=True),
+    "chromium": get_chromium_user_data_dir,
     "brave": get_brave_user_data_dir,
     "edge": get_edge_user_data_dir,
 }
@@ -337,6 +362,12 @@ def validate_persona_name(name: str) -> tuple[bool, str]:
 
 
 def discover_local_browser_profiles() -> list[PersonaImportSource]:
+    # The macOS companion exports with the host Keychain before creating personas.
+    # Raw mounts alone cannot decrypt macOS cookies in the Linux container.
+    if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
+        host_templates = discover_persona_template_profiles(Path(host_profiles))
+        host_names = {source.source_name for source in host_templates}
+        return host_templates + [source for source in discover_persona_template_profiles() if source.source_name not in host_names]
     discovered: list[PersonaImportSource] = []
 
     for browser, finder in BROWSER_PROFILE_FINDERS.items():
@@ -413,13 +444,21 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
 
 def resolve_browser_import_source(browser: str, profile_dir: str | None = None) -> PersonaImportSource:
     browser = browser.lower().strip()
+    if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
+        label = BROWSER_LABELS.get(browser, browser)
+        exported = Path(host_profiles) / f"{label} - {profile_dir or 'Default'}"
+        if exported.is_dir():
+            return resolve_custom_import_source(str(exported))
     if browser not in BROWSER_PROFILE_FINDERS:
         supported = ", ".join(BROWSER_PROFILE_FINDERS)
         raise ValueError(f"Unknown browser: {browser}. Supported browsers: {supported}")
 
     user_data_dir = BROWSER_PROFILE_FINDERS[browser]()
     if not user_data_dir:
-        raise ValueError(f"Could not find {browser} profile directory")
+        raise ValueError(
+            f"Could not find {browser} profile directory. Use --source /path/to/browser-data. "
+            "For Docker on macOS/Windows, import on the host into the shared data directory first.",
+        )
 
     chosen_profile = profile_dir or pick_default_profile_dir(user_data_dir)
     if not chosen_profile:
@@ -444,6 +483,8 @@ def resolve_browser_profile_source(
         resolved_root = resolved_root.resolve()
     if not resolved_root.exists():
         raise ValueError(f"Profile root does not exist: {resolved_root}")
+    if Path(profile_dir).name != profile_dir or profile_dir in {".", ".."}:
+        raise ValueError("Profile must be a directory name within the source browser root.")
     if not profile_dir.strip():
         raise ValueError("Profile directory name cannot be empty.")
 
@@ -476,6 +517,17 @@ def resolve_custom_import_source(raw_value: str, profile_dir: str | None = None)
         raise ValueError(f"Custom browser path does not exist: {source_path}")
 
     explicit_profile = profile_dir.strip() if profile_dir else ""
+    for profile_root in PERSONA_PROFILE_DIR_CANDIDATES:
+        root = source_path / profile_root
+        if root.is_dir():
+            selected = explicit_profile or pick_default_profile_dir(root)
+            if selected:
+                return resolve_browser_profile_source(
+                    browser="persona",
+                    source_name=source_path.name,
+                    user_data_dir=root.resolve(),
+                    profile_dir=selected,
+                )
     if _looks_like_profile_dir(source_path):
         if explicit_profile and explicit_profile != source_path.name:
             raise ValueError("Profile name does not match the provided profile directory path.")
@@ -508,6 +560,48 @@ def pick_default_profile_dir(user_data_dir: Path) -> str | None:
     return profiles[0]
 
 
+def resolve_source_browser_binary(source: PersonaImportSource) -> str:
+    """Use the originating browser so OS-protected cookies use the correct keychain."""
+    if source.browser_binary:
+        return str(Path(source.browser_binary).expanduser().resolve())
+    names = {
+        "chrome": ("Google Chrome", "google-chrome"),
+        "chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
+        "chromium": ("Chromium", "chromium"),
+        "brave": ("Brave Browser", "brave-browser"),
+        "edge": ("Microsoft Edge", "microsoft-edge"),
+    }
+    app, executable = names.get(source.browser, ("", ""))
+    candidates = []
+    if platform.system() == "Darwin" and app:
+        candidates = [
+            Path("/Applications") / f"{app}.app/Contents/MacOS/{app}",
+            Path.home() / f"Applications/{app}.app/Contents/MacOS/{app}",
+        ]
+    elif platform.system() == "Windows":
+        relative = {
+            "chrome": "Google/Chrome/Application/chrome.exe",
+            "chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
+            "edge": "Microsoft/Edge/Application/msedge.exe",
+            "brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
+            "chromium": "Chromium/Application/chrome.exe",
+        }.get(source.browser)
+        if relative:
+            candidates = [
+                Path(os.environ[root]) / relative for root in ("LOCALAPPDATA", "PROGRAMFILES", "PROGRAMFILES(X86)") if os.environ.get(root)
+            ]
+    for candidate in candidates:
+        if candidate.is_file() and os.access(candidate, os.X_OK):
+            return str(candidate)
+    if executable and (found := shutil.which(executable)):
+        return found
+    raise ValueError(
+        f"Source {source.browser} browser executable not found. Use --browser-binary with the originating Chromium browser. "
+        "For Docker, run persona create --import on the browser's host into the shared data directory first: "
+        "the container cannot decrypt cookies using the host's macOS Keychain or Windows credentials.",
+    )
+
+
 def import_persona_from_source(
     persona: Persona,
     source: PersonaImportSource,
@@ -516,67 +610,182 @@ def import_persona_from_source(
     import_cookies: bool = True,
     capture_storage: bool = False,
 ) -> PersonaImportResult:
+    # Stage all work before replacing an existing, working identity.
     persona.ensure_dirs()
     result = PersonaImportResult(source=source)
+    expected_cookies = 0
+    browser_binary = None
+    native_cookie_browsers = {"chrome", "chromium", "brave", "edge", "vivaldi", "opera", "opera_gx"}
+    # browser-cookie3 needs a desktop D-Bus session on Linux. In a headless
+    # environment, let the originating browser decode its own profile instead.
+    has_native_cookie_decoder = platform.system() != "Linux" or bool(os.environ.get("DBUS_SESSION_BUS_ADDRESS"))
+    native_cookie_import = source.kind == "browser-profile" and source.browser in native_cookie_browsers and has_native_cookie_decoder
+    portable_import = source.kind == "browser-profile" and source.browser == "persona"
+    if source.kind == "browser-profile" and not portable_import and (import_cookies or capture_storage):
+        if not native_cookie_import:
+            browser_binary = resolve_source_browser_binary(source)
+        for cookie_db in (source.profile_path / "Network" / "Cookies", source.profile_path / "Cookies"):
+            if cookie_db.is_file():
+                with sqlite3.connect(f"{cookie_db.as_uri()}?mode=ro", uri=True) as conn:
+                    expected_cookies = conn.execute(
+                        "SELECT COUNT(*) FROM cookies WHERE expires_utc = 0 OR expires_utc > ?",
+                        (int((time.time() + 11644473600) * 1_000_000),),
+                    ).fetchone()[0]
+                break
+    with tempfile.TemporaryDirectory(prefix=".import-", dir=persona.path.parent) as tmp:
+        stage = Path(tmp)
+        staged_profile = stage / "chrome_profile"
+        if source.kind == "browser-profile":
+            assert source.user_data_dir and source.profile_path
+            if copy_profile or import_cookies or capture_storage:
+                staged_profile.mkdir()
+                # Only the selected profile belongs to this persona. Normalize its
+                # name so downstream Chromium launches always select it.
+                copy_browser_user_data_dir(source.profile_path, staged_profile / "Default")
+                local_state = source.user_data_dir / "Local State"
+                if local_state.exists():
+                    state = json.loads(local_state.read_text())
+                    profile_state = state.setdefault("profile", {})
+                    profile_state["last_used"] = "Default"
+                    profile_state["last_active_profiles"] = ["Default"]
+                    info = profile_state.get("info_cache", {}).get(source.profile_dir)
+                    profile_state["info_cache"] = {"Default": info} if info else {}
+                    (staged_profile / "Local State").write_text(json.dumps(state))
+                persona.cleanup_chrome_profile(staged_profile)
+                result.profile_copied = copy_profile
+        elif copy_profile:
+            result.warnings.append(
+                "CDP imports capture cookies and open-tab storage, not browser settings. Use --source for a full profile import.",
+            )
 
-    persona_chrome_dir = Path(persona.CHROME_USER_DATA_DIR)
-    cookies_file = persona.path / "cookies.txt"
-    auth_file = persona.path / "auth.json"
-
-    launch_user_data_dir: Path | None = None
-
-    if source.kind == "browser-profile":
-        if copy_profile and source.user_data_dir:
-            resolved_source_root = source.user_data_dir.resolve()
-            resolved_persona_root = persona_chrome_dir.resolve()
-            if resolved_source_root == resolved_persona_root:
-                result.warnings.append(
-                    "Skipped profile copy because the selected source is already this persona's chrome_profile directory.",
-                )
+        if import_cookies or capture_storage:
+            if portable_import:
+                export_dir = source.user_data_dir.parent
+                if not all((export_dir / name).is_file() for name in ("auth.json", "cookies.txt")):
+                    raise ValueError("This persona has no portable cookie export. Re-import it on the original browser host first.")
+                auth_payload = json.loads((export_dir / "auth.json").read_text())
+                if not isinstance(auth_payload, dict) or not isinstance(auth_payload.get("cookies"), list):
+                    raise ValueError("Invalid persona auth.json: expected a cookies list.")
+                for name in ("auth.json", "cookies.txt"):
+                    shutil.copy2(export_dir / name, stage / name)
+                success, message = True, ""
+            elif native_cookie_import:
+                auth_payload = export_profile_cookies(source, staged_profile, stage)
+                success, message = True, ""
             else:
-                copy_browser_user_data_dir(resolved_source_root, resolved_persona_root)
-                persona.cleanup_chrome_profile(resolved_persona_root)
-                result.profile_copied = True
-            launch_user_data_dir = resolved_persona_root
-        else:
-            launch_user_data_dir = source.user_data_dir
-    elif copy_profile:
-        result.warnings.append(
-            "Profile copying is only available for local Chromium profile paths. CDP imports can only pull cookies and open-tab storage.",
-        )
-
-    if not import_cookies and not capture_storage:
-        return result
-
-    if source.kind == "cdp":
-        export_success, auth_payload, export_message = export_browser_state(
-            cdp_url=source.cdp_url,
-            cookies_output_file=cookies_file if import_cookies else None,
-            auth_output_file=auth_file if capture_storage else None,
-        )
-    else:
-        export_success, auth_payload, export_message = export_browser_state(
-            user_data_dir=launch_user_data_dir,
-            profile_dir=source.profile_dir,
-            chrome_binary=source.browser_binary,
-            cookies_output_file=cookies_file if import_cookies else None,
-            auth_output_file=auth_file if capture_storage else None,
-        )
-
-    if not export_success:
-        result.warnings.append(export_message or "Browser import failed.")
-        return result
-
-    if import_cookies and cookies_file.exists():
-        result.cookies_imported = True
-    if capture_storage and auth_file.exists():
-        result.storage_captured = True
-    if _apply_imported_user_agent(persona, auth_payload):
-        result.user_agent_imported = True
+                # Custom browsers and headless Linux use the originating browser.
+                # Desktop imports use their OS decoder without launching or probing.
+                launch_profile = stage / "export_profile"
+                if source.kind == "browser-profile":
+                    copy_browser_user_data_dir(staged_profile, launch_profile)
+                success, auth_payload, message = export_browser_state(
+                    user_data_dir=launch_profile if source.kind == "browser-profile" else None,
+                    cdp_url=source.cdp_url,
+                    profile_dir="Default" if source.kind == "browser-profile" else None,
+                    chrome_binary=browser_binary,
+                    cookies_output_file=stage / "cookies.txt" if import_cookies else None,
+                    auth_output_file=stage / "auth.json",
+                )
+            if not success:
+                raise ValueError(message or "Browser state export failed; persona was not replaced.")
+            if expected_cookies and not (auth_payload or {}).get("cookies"):
+                raise ValueError(
+                    "The source has cookies but the browser exported none. Run the import on the source host "
+                    "with the original browser and unlock its OS keychain; the existing persona was not replaced.",
+                )
+            if import_cookies:
+                result.cookies_imported = True
+                result.cookie_count = len((auth_payload or {}).get("cookies", []))
+            if capture_storage:
+                result.storage_captured = True
+            # Keep full CDP cookie attributes (sameSite, httpOnly, partition keys)
+            # as well as cookies.txt for non-browser extractors.
+            result.user_agent_imported = _apply_imported_user_agent(persona, auth_payload)
 
+        if result.profile_copied:
+            target = Path(persona.CHROME_USER_DATA_DIR)
+            if target.exists():
+                shutil.rmtree(target)
+            staged_profile.rename(target)
+        for filename in ("cookies.txt", "auth.json"):
+            exported = stage / filename
+            if exported.exists():
+                exported.chmod(0o600)
+                exported.replace(persona.path / filename)
     return result
 
 
+def export_profile_cookies(source: PersonaImportSource, staged_profile: Path, output: Path) -> dict:
+    """Decode with the host keychain, never by starting the user's browser."""
+    import browser_cookie3
+
+    payload = {"TYPE": "auth", "cookies": [], "localStorage": {}, "sessionStorage": {}}
+    jar = MozillaCookieJar(str(output / "cookies.txt"))
+    assert source.profile_path
+    for relative in (Path("Network/Cookies"), Path("Cookies")):
+        original = source.profile_path / relative
+        if not original.is_file():
+            continue
+        database = staged_profile / "Default" / relative
+        for suffix in ("", "-wal", "-shm"):
+            database.with_name(database.name + suffix).unlink(missing_ok=True)
+        # SQLite backup includes committed WAL data even while the browser is open.
+        with sqlite3.connect(f"{original.as_uri()}?mode=ro", uri=True) as reader:
+            with sqlite3.connect(database) as writer:
+                reader.backup(writer)
+        try:
+            cookies = getattr(browser_cookie3, source.browser)(
+                cookie_file=str(database),
+                key_file=str(staged_profile / "Local State"),
+            )
+        except Exception as err:
+            raise ValueError(
+                "Could not decrypt browser cookies. Run the import as your desktop user on the browser host "
+                "with its OS keychain unlocked. Mounting macOS/Windows browser files into Linux does not provide "
+                "the decryption keys. The existing persona was not replaced.",
+            ) from err
+        with sqlite3.connect(database) as connection:
+            connection.row_factory = sqlite3.Row
+            columns = {row[1] for row in connection.execute("PRAGMA table_info(cookies)")}
+            metadata_columns = [
+                name
+                for name in ("host_key", "path", "name", "samesite", "top_frame_site_key", "has_cross_site_ancestor")
+                if name in columns
+            ]
+            rows = {
+                (row["host_key"], row["path"], row["name"]): dict(row)
+                for row in connection.execute("SELECT " + ", ".join(metadata_columns) + " FROM cookies")
+            }
+        for cookie in cookies:
+            if cookie.is_expired():
+                continue
+            jar.set_cookie(cookie)
+            item = {
+                "name": cookie.name,
+                "value": cookie.value,
+                "domain": cookie.domain,
+                "path": cookie.path,
+                "secure": bool(cookie.secure),
+                "httpOnly": cookie.has_nonstandard_attr("HTTPOnly"),
+            }
+            if cookie.expires:
+                item["expires"] = cookie.expires
+            row = rows.get((cookie.domain, cookie.path, cookie.name), {})
+            same_site = {0: "None", 1: "Lax", 2: "Strict"}.get(row.get("samesite"))
+            if same_site:
+                item["sameSite"] = same_site
+            if row.get("top_frame_site_key"):
+                item["partitionKey"] = {
+                    "topLevelSite": row["top_frame_site_key"],
+                    "hasCrossSiteAncestor": bool(row.get("has_cross_site_ancestor", False)),
+                }
+            payload["cookies"].append(item)
+        break
+    jar.save(ignore_discard=True, ignore_expires=False)
+    (output / "auth.json").write_text(json.dumps(payload) + "\n")
+    return payload
+
+
 def copy_browser_user_data_dir(source_dir: Path, destination_dir: Path) -> None:
     destination_dir.parent.mkdir(parents=True, exist_ok=True)
     shutil.rmtree(destination_dir, ignore_errors=True)
@@ -611,21 +820,27 @@ def export_browser_state(
     chrome_config = chrome_plugin_dir / "chrome" / "config.json"
 
     env = os.environ.copy()
-    if chrome_binary:
-        env["CHROME_BINARY"] = str(chrome_binary)
-    dependency_env = subprocess.run(
-        [
-            str(Path(sys.executable).with_name("abxpkg")),
-            "env",
-            "--install",
-            "--json",
-            f"--lib={get_config().ABXPKG_LIB_DIR}",
-            f"--deps-from={chrome_config}:required_binaries",
-        ],
-        capture_output=True,
-        text=True,
-        env=env,
-    )
+    dependency_config = json.loads(chrome_config.read_text())
+    dependency_config["required_binaries"] = [dep for dep in dependency_config["required_binaries"] if dep["name"] != "{CHROME_BINARY}"]
+    dependency_file = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False)
+    with dependency_file:
+        json.dump(dependency_config, dependency_file)
+    try:
+        dependency_env = subprocess.run(
+            [
+                str(Path(sys.executable).with_name("abxpkg")),
+                "env",
+                "--install",
+                "--json",
+                f"--lib={get_config().ABXPKG_LIB_DIR}",
+                f"--deps-from={dependency_file.name}:required_binaries",
+            ],
+            capture_output=True,
+            text=True,
+            env=env,
+        )
+    finally:
+        Path(dependency_file.name).unlink()
     if dependency_env.returncode != 0:
         return False, None, dependency_env.stderr.strip() or "abxpkg could not resolve browser export dependencies."
     try:
@@ -639,6 +854,9 @@ def export_browser_state(
     if not node_projection.is_symlink() or not os.access(node_projection, os.X_OK):
         return False, None, f"abxpkg did not resolve Node.js into {node_projection}."
     env.update({str(key): str(value) for key, value in resolved_env.items()})
+    if chrome_binary:
+        env["CHROME_BINARY"] = chrome_binary
+    env["NODE_MODULES_DIR"] = str(abxpkg_lib_dir / "pnpm" / "packages" / "chrome" / "node_modules")
     env["NODE_BINARY"] = str(node_projection)
     env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(chrome_plugin_dir)
 
@@ -865,7 +1083,7 @@ def _apply_imported_user_agent(persona: Persona, auth_payload: dict | None) -> b
     if not auth_payload:
         return False
 
-    user_agent = str(auth_payload.get("user_agent") or "").strip()
+    user_agent = str(auth_payload.get("user_agent") or "").strip().replace("HeadlessChrome/", "Chrome/")
     if not user_agent:
         return False
 

archivebox/personas/models.py

@@ -45,17 +45,23 @@ VOLATILE_PROFILE_DIR_NAMES = {
     "Code Cache",
     "GPUCache",
     "ShaderCache",
-    "Service Worker",
-    "GCM Store",
     "Crashpad",
     "BrowserMetrics",
+    "Sessions",
+    "Sessions_Encrypted",
 }
 
 VOLATILE_PROFILE_FILE_NAMES = {
     "BrowserMetrics-spare.pma",
+    "RunningChromeVersion",
+    "DevToolsActivePort",
     "SingletonCookie",
     "SingletonLock",
     "SingletonSocket",
+    "Current Session",
+    "Current Tabs",
+    "Last Session",
+    "Last Tabs",
 }
 
 
@@ -211,7 +217,7 @@ class Persona(ModelWithConfig):
                 cleaned = True
 
         for path in profile_paths():
-            if not path.match("*.log"):
+            if path.name not in {"chrome.log", "chrome_debug.log"}:
                 continue
             try:
                 path.unlink()
@@ -297,6 +303,10 @@ class Persona(ModelWithConfig):
             else:
                 runtime_profile_dir.mkdir(parents=True, exist_ok=True)
 
+            for filename in ("cookies.txt", "auth.json"):
+                source = self.path / filename
+                if source.is_file():
+                    shutil.copy2(source, runtime_root / filename)
             runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
             self.cleanup_chrome_profile(runtime_profile_dir)
 
@@ -313,6 +323,11 @@ class Persona(ModelWithConfig):
             # derivation centralized in the Chrome plugin helpers.
             "PERSONAS_DIR": str(runtime_root.parent),
             "ACTIVE_PERSONA": self.name,
+            **{
+                key: str(runtime_root / filename)
+                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
+                if (runtime_root / filename).is_file()
+            },
         }
 
     def prepare_runtime_for_snapshot(self, snapshot, chrome_binary: str = "") -> dict[str, str]:
@@ -329,6 +344,12 @@ class Persona(ModelWithConfig):
         else:
             runtime_profile_dir.mkdir(parents=True, exist_ok=True)
 
+        for filename in ("cookies.txt", "auth.json"):
+            source = self.runtime_root_for_crawl(snapshot.crawl) / filename
+            if not source.is_file():
+                source = self.path / filename
+            if source.is_file():
+                shutil.copy2(source, runtime_root / filename)
         runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
         self.cleanup_chrome_profile(runtime_profile_dir)
 
@@ -343,6 +364,11 @@ class Persona(ModelWithConfig):
             # Chrome hooks and ArchiveBox-driven hooks resolve paths the same way.
             "PERSONAS_DIR": str(runtime_root.parent),
             "ACTIVE_PERSONA": self.name,
+            **{
+                key: str(runtime_root / filename)
+                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
+                if (runtime_root / filename).is_file()
+            },
         }
 
     def cleanup_runtime_for_crawl(self, crawl) -> None:

archivebox/personas/open_browser.js

@@ -0,0 +1,61 @@
+#!/usr/bin/env node
+
+const fs = require('fs');
+const path = require('path');
+const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR;
+const {ensureNodeModuleResolution} = require(path.join(pluginsDir, 'base', 'utils.js'));
+ensureNodeModuleResolution(module);
+const installShutdownHandler = require(path.join(pluginsDir, 'base', 'daemon_lifecycle.js')).captureShutdownSignals();
+const {
+    acquireSessionLock,
+    ensureChromeSession,
+    getChromeSessionOptionsFromConfig,
+    getChromeLaunchPrerequisites,
+    connectToBrowserEndpoint,
+    closeBrowserInChromeSession,
+} = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
+
+async function main() {
+    const config = JSON.parse(fs.readFileSync(0, 'utf8'));
+    const outputDir = path.join(path.dirname(config.CHROME_USER_DATA_DIR), '.browser');
+    fs.mkdirSync(outputDir, {recursive: true});
+    const release = await acquireSessionLock(path.join(outputDir, 'launch.lock'));
+    let session;
+    const {puppeteer, binary} = getChromeLaunchPrerequisites();
+    try {
+        session = await ensureChromeSession({
+            ...getChromeSessionOptionsFromConfig(config),
+            outputDir, puppeteer, binary,
+        });
+    } finally {
+        release();
+    }
+    const browser = await connectToBrowserEndpoint(puppeteer, session.cdpUrl, {defaultViewport: null});
+    installShutdownHandler(async () => {
+        if (!session.reusedExisting) {
+            await closeBrowserInChromeSession({
+                outputDir, puppeteer, cdpUrl: session.cdpUrl, pid: session.pid, processIsLocal: true,
+            });
+        }
+        await browser.disconnect();
+        process.exit(0);
+    });
+    const pages = await browser.pages();
+    const page = pages[0] || await browser.newPage();
+    const windowSession = await page.createCDPSession();
+    const {windowId} = await windowSession.send('Browser.getWindowForTarget');
+    await windowSession.send('Browser.setWindowBounds', {windowId, bounds: {windowState: 'maximized'}});
+    await windowSession.detach();
+    await page.bringToFront();
+    console.log(`Browser ready for persona ${config.ACTIVE_PERSONA || 'Default'}`);
+    if (session.reusedExisting) {
+        await browser.disconnect();
+        return;
+    }
+    await new Promise(resolve => browser.once('disconnected', resolve));
+}
+
+main().catch(error => {
+    console.error(error.message);
+    process.exit(1);
+});

archivebox/plugins/discovery.py

@@ -55,10 +55,6 @@ def get_plugins() -> list[str]:
     return sorted(get_plugin_catalog())
 
 
-def get_plugin_models():
-    return get_plugin_catalog().plugins
-
-
 def get_plugin_name(plugin: str) -> str:
     """
     Get the base plugin name without numeric prefix.
@@ -88,70 +84,14 @@ def get_enabled_plugins(config: ConfigLookup | None = None, **config_kwargs: Any
     return get_plugin_config_resolver().enabled_plugin_names_from_flat(dict(config.items()))
 
 
-def discover_plugins_that_provide_interface(
-    module_name: str,
-    required_attrs: list[str],
-    plugin_prefix: str | None = None,
-) -> dict[str, Any]:
-    """
-    Discover plugins that provide a specific Python module with required interface.
-
-    This enables dynamic plugin discovery for features like search backends,
-    storage backends, etc. without hardcoding imports.
-    """
-    import importlib.util
-
-    backends = {}
-
-    for plugin_dir in iter_plugin_dirs():
-        plugin_name = plugin_dir.name
-        if plugin_prefix and not plugin_name.startswith(plugin_prefix):
-            continue
-
-        module_path = plugin_dir / f"{module_name}.py"
-        if not module_path.exists():
-            continue
-
-        try:
-            spec = importlib.util.spec_from_file_location(
-                f"archivebox.dynamic_plugins.{plugin_name}.{module_name}",
-                module_path,
-            )
-            if spec is None or spec.loader is None:
-                continue
-
-            module = importlib.util.module_from_spec(spec)
-            spec.loader.exec_module(module)
-
-            if not all(attr in vars(module) for attr in required_attrs):
-                continue
-
-            if plugin_prefix:
-                backend_name = plugin_name[len(plugin_prefix) :]
-            else:
-                backend_name = plugin_name
-
-            backends[backend_name] = module
-
-        except Exception:
-            continue
-
-    return backends
-
-
-def get_search_backends() -> dict[str, Any]:
-    """
-    Discover all available search backend plugins.
-
-    Search backends must provide a search.py module with:
-        - search(query: str) -> List[str]  (returns snapshot IDs)
-        - flush(snapshot_ids: Iterable[str]) -> None
-    """
-    return discover_plugins_that_provide_interface(
-        module_name="search",
-        required_attrs=["search", "flush"],
-        plugin_prefix="search_backend_",
-    )
+def get_search_backends():
+    """Return plugins that declare both standalone search commands."""
+    catalog = get_plugin_catalog()
+    return {
+        plugin.name.removeprefix("search_backend_"): plugin
+        for plugin in catalog.values()
+        if catalog.command(plugin.name, "search") is not None and catalog.command(plugin.name, "flush") is not None
+    }
 
 
 @lru_cache(maxsize=1)

archivebox/plugins/forms.py

@@ -213,7 +213,7 @@ class PluginConfigFormMixin:
                     "enabled_config_key": enabled_config_key,
                     "description": str(schema.get("description") or "").strip(),
                     "source_url": f"https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/{plugin_name}",
-                    "docs_url": f"https://archivebox.github.io/abx-plugins/#{plugin_name}",
+                    "docs_url": f"https://plugins.archivebox.io/#{plugin_name}",
                     "required_plugins": [str(item) for item in schema.get("required_plugins") or []],
                     "required_binary_links": _build_required_binary_links(
                         schema.get("required_binaries") or [],
@@ -276,7 +276,7 @@ class PluginConfigFormMixin:
         elif is_sensitive:
             input_type = "password"
         else:
-            input_value = "" if is_sensitive else str(current_value)
+            input_value = "" if current_value is None else str(current_value)
 
         return {
             "key": config_key,
@@ -290,7 +290,7 @@ class PluginConfigFormMixin:
             "default": _jsonish(default_display),
             "current": "configured"
             if is_sensitive and current_value
-            else (str(current_value) if "string" in schema_types else _jsonish(current_value)),
+            else (("" if current_value is None else str(current_value)) if "string" in schema_types else _jsonish(current_value)),
             "current_url": self.plugin_config_binary_urls.get(config_key, "") if str(config_key).endswith("_BINARY") else "",
             "is_sensitive": is_sensitive,
             "minimum": prop_schema.get("minimum"),
@@ -342,6 +342,8 @@ class PluginConfigFormMixin:
                     continue
 
                 base_value = effective_config.get(config_key, prop_schema.get("default", ""))
+                if base_value is None and coerced_value == "":
+                    continue
                 if _same_config_value(coerced_value, base_value):
                     continue
 

archivebox/plugins/hooks.py

@@ -8,40 +8,13 @@ from __future__ import annotations
 
 __package__ = "archivebox.plugins"
 
-import os
-from collections.abc import Mapping
 from pathlib import Path
-from typing import TYPE_CHECKING, Any, Protocol, TypeGuard, runtime_checkable
+from typing import Any
 
-from asgiref.sync import async_to_sync
+from abx_dl.models import parse_hook_filename
 
-from abx_dl.execution import execute_hook
-from abx_dl.models import Hook, parse_hook_filename
-
-from archivebox.config.constants import CONSTANTS
-from archivebox.config.version import VERSION
 from archivebox.misc.util import fix_url_from_markdown, sanitize_extracted_url
-from archivebox.plugins.discovery import ConfigLookup, get_enabled_plugins, get_plugin_catalog, get_plugin_special_config
-
-if TYPE_CHECKING:
-    from archivebox.machine.models import Process
-
-
-@runtime_checkable
-class ConfigDump(Protocol):
-    def as_dict(self) -> dict[str, Any]: ...
-
-
-def _has_config_dump(config: object) -> TypeGuard[ConfigDump]:
-    return isinstance(config, ConfigDump)
-
-
-def _config_to_overrides(config: ConfigLookup | Mapping[str, Any] | None) -> dict[str, Any]:
-    if config is None:
-        return {}
-    if _has_config_dump(config):
-        return dict(config.as_dict())
-    return dict(config.items())
+from archivebox.plugins.discovery import ConfigLookup, get_enabled_plugins, get_plugin_catalog
 
 
 def is_background_hook(hook_name: str) -> bool:
@@ -76,149 +49,6 @@ def discover_hooks(
     return [hook.path for _plugin, hook in get_plugin_catalog().hooks(normalized, names=names)]
 
 
-def _catalog_hook(script: Path) -> Hook:
-    script = script.resolve()
-    for plugin in get_plugin_catalog().values():
-        for hook in plugin.hooks:
-            if hook.path.resolve() == script:
-                return hook
-    parsed = parse_hook_filename(script.name)
-    if parsed is None:
-        raise ValueError(f"Not a valid plugin hook filename: {script.name}")
-    event, order, is_background = parsed
-    return Hook(
-        name=script.name,
-        event=event,
-        plugin_name=script.parent.name,
-        path=script,
-        order=order,
-        is_background=is_background,
-    )
-
-
-def _hook_environment(config: ConfigLookup | Mapping[str, Any] | None, **config_scope: Any) -> tuple[dict[str, str], Any]:
-    from archivebox.config.common import (
-        ArchiveBoxConfig,
-        _archivebox_config_input_names,
-        get_config,
-        normalize_runtime_config,
-    )
-
-    overrides = _config_to_overrides(config)
-    resolved = get_config(overrides=overrides, **config_scope)
-    runtime = normalize_runtime_config(
-        resolved.for_crawl_runtime(runtime_overrides=overrides),
-        json_safe=False,
-    )
-    runtime.update(normalize_runtime_config(overrides, json_safe=False))
-
-    env = os.environ.copy()
-    config_input_names = _archivebox_config_input_names()
-    for key in config_input_names:
-        env.pop(key, None)
-    env.pop("PLUGINS", None)
-    env["PATH"] = os.environ.get("PATH", "")
-    env["DATA_DIR"] = str(CONSTANTS.DATA_DIR)
-    env["LIBRARY_VERSION"] = VERSION
-    env.setdefault("MACHINE_ID", os.environ.get("MACHINE_ID", CONSTANTS.MACHINE_ID))
-
-    canonical_config_keys = set(ArchiveBoxConfig.model_fields)
-    for key, value in runtime.items():
-        if key == "PATH" or value is None:
-            continue
-        if key in config_input_names and key not in canonical_config_keys:
-            continue
-        if isinstance(value, bool):
-            env[key] = "true" if value else "false"
-        elif isinstance(value, (dict, list)):
-            import json
-
-            env[key] = json.dumps(value)
-        else:
-            env[key] = str(value)
-
-    node_modules_dir = runtime.get("NODE_MODULES_DIR")
-    lib_dir = runtime.get("ABXPKG_LIB_DIR")
-    if not node_modules_dir and lib_dir:
-        node_modules_dir = Path(lib_dir) / "pnpm" / "packages" / "chrome" / "node_modules"
-    if node_modules_dir:
-        env["NODE_MODULES_DIR"] = str(node_modules_dir)
-        env["NODE_MODULE_DIR"] = str(node_modules_dir)
-        node_path = [part for part in str(runtime.get("NODE_PATH") or "").split(os.pathsep) if part]
-        if str(node_modules_dir) not in node_path:
-            node_path.append(str(node_modules_dir))
-        env["NODE_PATH"] = os.pathsep.join(node_path)
-    return env, resolved
-
-
-def run_hook(
-    script: Path,
-    output_dir: Path,
-    config: ConfigLookup | Mapping[str, Any] | None = None,
-    timeout: int | None = None,
-    parent: Process | None = None,
-    **kwargs: Any,
-) -> Process:
-    """Compatibility adapter for finite direct calls; abx-dl owns execution."""
-    from archivebox.machine.models import Process
-    from archivebox.services.process_service import ProcessService as PersistedProcessService
-    from archivebox.services.process_service import parse_event_datetime
-    from abx_dl.orchestrator import create_bus
-
-    # Preserve the old direct-call contract: hooks are children of an explicit
-    # parent, or of the current ArchiveBox process when one can be identified.
-    # This belongs on the DB projection adapter, not in hook CLI arguments.
-    if parent is None:
-        try:
-            parent = Process.current()
-        except Exception:
-            parent = None
-    config_scope = {key.removeprefix("config_"): kwargs.pop(key) for key in list(kwargs) if key.startswith("config_")}
-    env, resolved = _hook_environment(config, **config_scope)
-    hook = _catalog_hook(script)
-    if timeout is None:
-        timeout = get_plugin_special_config(hook.plugin_name, resolved)["timeout"]
-    timeout = min(int(timeout or 300), int(CONSTANTS.MAX_HOOK_RUNTIME_SECONDS))
-
-    bus = create_bus(name=f"ArchiveBoxHook_{hook.plugin_name}", total_timeout=float(timeout) + 30.0)
-    PersistedProcessService(bus, parent_process_id=str(parent.id) if parent is not None else None)
-
-    async def execute_and_close():
-        try:
-            return await execute_hook(
-                hook,
-                output_dir=output_dir,
-                env=env,
-                arguments=kwargs,
-                timeout=timeout,
-                bus=bus,
-                process_type=Process.TypeChoices.HOOK,
-            )
-        finally:
-            await bus.wait_until_idle()
-            await bus.destroy(clear=False)
-
-    completed = async_to_sync(execute_and_close)()
-    started_at = parse_event_datetime(completed.start_ts)
-    process = Process.objects.filter(pid=completed.pid or None, started_at=started_at).order_by("-modified_at").first()
-    if process is None:
-        raise RuntimeError(f"Hook {hook.full_name} completed without an ArchiveBox Process projection")
-    return process
-
-
-def extract_records_from_process(process: Process) -> list[dict[str, Any]]:
-    """Return hook JSONL records with generic catalog identity attached."""
-    records = process.get_records()
-    plugin_name = Path(process.pwd).name if process.pwd else "unknown"
-    plugin_hook = next((str(arg) for arg in process.cmd if Path(str(arg)).name.startswith("on_")), "")
-    hook_name = Path(plugin_hook).name if plugin_hook else "unknown"
-    for record in records:
-        record.setdefault("plugin", plugin_name)
-        record.setdefault("hook_name", hook_name)
-        record.setdefault("plugin_hook", plugin_hook)
-    return records
-
-
 def collect_urls_from_plugins(snapshot_dir: Path) -> list[dict[str, Any]]:
     """Read the durable urls.jsonl interface emitted by parser plugins."""
     urls: list[dict[str, Any]] = []

archivebox/plugins/output_groups.py

@@ -0,0 +1,95 @@
+"""Snapshot output taxonomy. Display preference is independent of hook order."""
+
+from pathlib import PurePosixPath
+from typing import Any
+
+from archivebox.plugins.discovery import get_plugin_name
+
+# Ordered tuples are preference lists; unordered groups sort by total output bytes.
+OUTPUT_GROUPS = (
+    ("html", "HTML", ("archivewebpage", "singlefile", "chrome_mhtml", "wget", "dom")),
+    ("raster", "Raster", ("screenshot", "pdf")),
+    ("article_text", "Article text", ("readability", "defuddle", "mercury", "trafilatura", "htmltotext")),
+    ("embedded_media", "Embedded media", ()),
+    ("metadata", "Metadata", ()),
+    ("other", "Other files", ()),
+)
+UNORDERED_OUTPUTS = {
+    "embedded_media": {"responses", "ytdlp", "yt-dlp", "youtube-dl", "gallerydl", "forumdl", "git", "media", "liteparse"},
+    "metadata": {
+        "dns",
+        "headers",
+        "seo",
+        "accessibility",
+        "redirects",
+        "consolelog",
+        "sslcerts",
+        "ssl",
+        "title",
+        "favicon",
+        "hashes",
+        "chrome",
+        "parse_html_urls",
+        "parse_txt_urls",
+        "parse_dom_outlinks",
+        "parse_jsonl_urls",
+        "parse_netscape_urls",
+        "parse_rss_urls",
+        "tlsnotary",
+        "opentimestamps",
+    },
+}
+
+
+def _responses_html_card(outputs: list[dict[str, Any]]) -> dict[str, Any] | None:
+    # Presentation-only exception to the usual 1:1 plugin:card rule. Responses
+    # owns both this HTML view and its media gallery; responses_html is only a
+    # unique card/anchor ID, never a plugin or a synthetic ArchiveResult.
+    # Refactor this exception if plugins gain support for multiple cards.
+    for output in outputs:
+        if output["name"] != "responses" or not (result := output.get("result")):
+            continue
+        path = str(output.get("path") or "")
+        if not path.startswith("responses/") or ".." in PurePosixPath(path).parts:
+            continue
+        files = result.output_file_map()
+        info = files.get(path) or files.get(path.removeprefix("responses/")) or {}
+        mime = str(info.get("mimetype") or "").split(";")[0].lower()
+        if not info or result._coerce_output_file_size(info.get("size")) <= 0:
+            continue
+        if mime not in {"text/html", "application/xhtml+xml"} and PurePosixPath(path).suffix.lower() not in {".html", ".htm"}:
+            continue
+        return {
+            "name": "responses_html",
+            "path": path,
+            "folder_path": "responses",
+            "ts": output.get("ts"),
+            "size": 0,  # Bytes are already counted by the real Responses card.
+            "result": None,
+            "direct_preview_path": f"{path}?card=responses_html",
+        }
+    return None
+
+
+def order_snapshot_outputs(outputs: list[dict[str, Any]]) -> list[dict[str, Any]]:
+    """Annotate and order available cards; unknown plugins stay visible in Other files."""
+    group_order = {group_id: index for index, (group_id, _, _) in enumerate(OUTPUT_GROUPS)}
+    preferences = {}
+    for group_id, _, plugins in OUTPUT_GROUPS:
+        for rank, plugin in enumerate(plugins):
+            preferences[plugin] = (group_id, rank)
+    for group_id, plugins in UNORDERED_OUTPUTS.items():
+        for plugin in plugins:
+            preferences[plugin] = (group_id, None)
+
+    if card := _responses_html_card(outputs):
+        outputs = [*outputs, card]
+    preferences["responses_html"] = ("html", len(OUTPUT_GROUPS[0][2]))
+
+    def sort_key(output):
+        plugin = get_plugin_name(output["name"])
+        group_id, rank = preferences.get(plugin, ("other", None))
+        output["output_group"] = group_id
+        return (group_order[group_id], rank if rank is not None else -int(output.get("size") or 0), plugin)
+
+    return sorted(outputs, key=sort_key)

archivebox/plugins/views.py

@@ -17,7 +17,7 @@ from archivebox.config.common import get_live_config_url
 from archivebox.config.views import get_environment_binary_url, is_superuser
 from archivebox.plugins.discovery import BUILTIN_PLUGINS_DIR, USER_PLUGINS_DIR, discover_plugin_configs, iter_plugin_dirs
 
-ABX_PLUGINS_DOCS_BASE_URL = "https://archivebox.github.io/abx-plugins/"
+ABX_PLUGINS_DOCS_BASE_URL = "https://plugins.archivebox.io/"
 ABX_PLUGINS_GITHUB_BASE_URL = "https://github.com/ArchiveBox/abx-plugins/tree/main/abx_plugins/plugins/"
 LIVE_PLUGIN_BASE_URL = "/admin/environment/plugins/"
 

archivebox/progressmonitor/collection.py

@@ -0,0 +1,54 @@
+"""Non-blocking, database-only collection totals for the progress monitor."""
+
+import logging
+from threading import Lock, Thread
+from time import time
+
+from django.core.cache import cache
+from django.db import close_old_connections, connections
+from django.db.models import Count, Sum
+
+
+_refresh_lock = Lock()
+
+
+def collection_summary(user):
+    user_id = None if user.is_superuser else user.pk
+    key = f"progress-collection:{'all' if user_id is None else user_id}"
+    summary = cache.get(key)
+    # Even an indexed SUM scans the index. Never make an HTTP request wait for
+    # that scan on a large collection. Serve stale totals while one worker refreshes;
+    # a single non-blocking lock prevents a queue of scans from concurrent clients.
+    if (summary is None or time() - summary["sampled_at"] >= 60) and _refresh_lock.acquire(blocking=False):
+        # Throttle failures too: a broken/locked database must not trigger work
+        # and log output on every sidebar poll. This cache is local to each worker.
+        if not cache.add(f"{key}:refresh", True, timeout=60):
+            _refresh_lock.release()
+            return summary
+
+        def refresh():
+            try:
+                close_old_connections()
+                from archivebox.core.models import Snapshot
+
+                snapshots = Snapshot.objects.all()
+                if user_id is not None:
+                    snapshots = snapshots.filter(crawl__created_by_id=user_id)
+                # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
+                # Snapshot.output_size is maintained when ArchiveResult sizes change.
+                totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+                totals["bytes"] = totals["bytes"] or 0
+                totals["sampled_at"] = time()
+                cache.set(key, totals, timeout=3600)
+            except Exception:
+                logging.getLogger(__name__).exception("Could not refresh collection totals")
+            finally:
+                connections.close_all()
+                _refresh_lock.release()
+
+        try:
+            Thread(target=refresh, name="collection-summary", daemon=True).start()
+        except RuntimeError:
+            _refresh_lock.release()
+            logging.getLogger(__name__).exception("Could not start collection totals refresh")
+    return summary

archivebox/progressmonitor/views.py

@@ -30,16 +30,20 @@ def progress_endpoint(scope: Literal["crawl", "snapshot"] | None = None, object_
 @lru_cache(maxsize=1)
 def _live_progress_plugin_names() -> tuple[frozenset[str], frozenset[str]]:
     plugin_configs = discover_plugin_configs()
+    indexing_plugin_names = frozenset(
+        plugin_name
+        for plugin_name, plugin_config in plugin_configs.items()
+        if {"search", "flush"}.issubset(plugin_config.get("commands", {}))
+    )
     download_plugin_names = frozenset(
         plugin_name
         for plugin_name, plugin_config in plugin_configs.items()
-        if plugin_config.get("output_mimetypes") and not plugin_name.startswith("search_backend_")
+        if plugin_config.get("output_mimetypes") and plugin_name not in indexing_plugin_names
     )
-    indexing_plugin_names = frozenset(plugin_name for plugin_name in plugin_configs if plugin_name.startswith("search_backend_"))
     return download_plugin_names, indexing_plugin_names
 
 
-def live_progress_view(request):
+def live_progress_view(request, *, authorized_snapshot=None):
     """Simple JSON endpoint for live progress status - used by admin progress monitor."""
     try:
         from archivebox.core.models import ArchiveResult, Snapshot
@@ -59,7 +63,10 @@ def live_progress_view(request):
             except (TypeError, ValueError):
                 return JsonResponse({"error": "Invalid snapshot_id"}, status=400)
             scoped_snapshot = Snapshot.objects.filter(id=snapshot_id_filter).select_related("crawl").first()
-            if scoped_snapshot is None or not can_view_snapshot(request, scoped_snapshot):
+            authorized_snapshot_id = str(getattr(authorized_snapshot, "id", "")).replace("-", "")
+            if scoped_snapshot is None or (
+                authorized_snapshot_id != str(scoped_snapshot.id).replace("-", "") and not can_view_snapshot(request, scoped_snapshot)
+            ):
                 return JsonResponse({"error": "Permission denied"}, status=403)
         elif crawl_id_filter:
             # Crawl-only scope still requires staff: there's no per-crawl ACL helper,
@@ -217,6 +224,8 @@ def live_progress_view(request):
 
         api_base = get_api_base_url(request=request, config=request_config) if scoped_snapshot is not None else ""
 
+        running_screencasts = {}
+
         def screencast_frame_url(crawl_id: str, crawl_dir: Path) -> str:
             frame_path = crawl_dir / "chrome_screencast" / "latest.jpg"
             try:
@@ -225,7 +234,8 @@ def live_progress_view(request):
                 return ""
             if frame_stat.st_size <= 0:
                 return ""
-            if now.timestamp() - frame_stat.st_mtime > 15:
+            started_at = running_screencasts.get(crawl_id)
+            if started_at is None or frame_stat.st_mtime < started_at.timestamp():
                 return ""
             rel = f"/api/v1/crawls/crawl/{crawl_id}/files/chrome_screencast/latest.jpg?v={frame_stat.st_mtime_ns}"
             return f"{api_base}{rel}" if api_base else rel
@@ -471,7 +481,10 @@ def live_progress_view(request):
         detailed_snapshot_ids = [snapshot["id"] for snapshot in snapshots if snapshot["status"] != Snapshot.StatusChoices.QUEUED]
         process_value_fields = ("id", "process_type", "status", "pwd", "cmd", "pid", "exit_code", "started_at", "ended_at", "modified_at")
         if active_crawl_ids or displayed_snapshot_ids:
+            relevant_process_ids = tuple(dict.fromkeys([*active_crawl_ids, *displayed_snapshot_ids]))
+            relevant_process_path_pattern = "|".join(relevant_process_ids)
             process_scope = Process.objects.filter(
+                pwd__regex=relevant_process_path_pattern,
                 machine_id=machine_id,
                 process_type__in=[
                     Process.TypeChoices.HOOK,
@@ -548,6 +561,10 @@ def live_progress_view(request):
                 snapshot_id = str(matched_snapshot["id"])
             running_worker_ids.add(str(proc["id"]))
             _plugin, _label, phase, _hook_name = process_label(proc["cmd"])
+            if _plugin == "chrome_screencast" and proc["started_at"] and Process(**proc, machine_id=machine_id).is_running:
+                # CDP sends frames only when the page changes. Keep a still
+                # frame visible for its live capture, never for a previous run.
+                running_screencasts[crawl_id] = max(proc["started_at"], running_screencasts.get(crawl_id, proc["started_at"]))
             if crawl_id and proc["pid"]:
                 crawl_process_pids.setdefault(crawl_id, proc["pid"])
             if phase == "snapshot" and snapshot_id and proc["pid"]:
@@ -938,6 +955,10 @@ def live_progress_view(request):
             "queued_crawls_hidden": queued_crawls_hidden,
             "server_time": timezone.now().isoformat(),
         }
+        if is_admin and not snapshot_id_filter and not crawl_id_filter and request.GET.get("collection") == "1":
+            from archivebox.progressmonitor.collection import collection_summary
+
+            payload["collection"] = collection_summary(request.user)
         try:
             import ujson
 

archivebox/search/backends.py

@@ -1,7 +1,7 @@
 __package__ = "archivebox.search"
 
+import json
 import os
-from contextlib import contextmanager
 from typing import Any
 
 from archivebox.config.common import get_config
@@ -10,29 +10,21 @@ from archivebox.config.common import get_config
 _search_backends_cache: dict | None = None
 
 
-@contextmanager
-def search_backend_env(config: dict[str, Any] | None = None, **config_kwargs: Any):
-    """Temporarily expose resolved search config through os.environ for backend code."""
+def search_backend_command_env(config: dict[str, Any] | None = None, **config_kwargs: Any) -> dict[str, str]:
+    """Serialize resolved application config for a standalone plugin command."""
     config = config or get_config(**config_kwargs)
-    updates = {}
+    env = os.environ.copy()
     for key, value in config.items():
         key = str(key)
-        if not (key.startswith("SEARCH_BACKEND_") or key.endswith("_BINARY")):
-            continue
         if value is None:
             continue
-        if isinstance(value, (str, int, float, bool, os.PathLike)):
-            updates[key] = str(value)
-    previous = {key: os.environ.get(key) for key in updates}
-    os.environ.update(updates)
-    try:
-        yield
-    finally:
-        for key, value in previous.items():
-            if value is None:
-                os.environ.pop(key, None)
-            else:
-                os.environ[key] = value
+        if isinstance(value, bool):
+            env[key] = "true" if value else "false"
+        elif isinstance(value, (dict, list, tuple)):
+            env[key] = json.dumps(value)
+        elif isinstance(value, (str, int, float, os.PathLike)):
+            env[key] = str(value)
+    return env
 
 
 def normalize_search_backend_name(backend_name: str | None) -> str:
@@ -41,7 +33,7 @@ def normalize_search_backend_name(backend_name: str | None) -> str:
 
 
 def get_available_backends() -> dict:
-    """Discover search backend plugin modules and cache them in memory."""
+    """Discover search-capable plugins and cache their catalog entries."""
     global _search_backends_cache
 
     if _search_backends_cache is None:
@@ -53,7 +45,7 @@ def get_available_backends() -> dict:
 
 
 def get_backend(config: dict[str, Any] | None = None, **config_kwargs: Any) -> Any:
-    """Resolve the configured search backend module."""
+    """Resolve the configured search-capable plugin."""
     config = config or get_config(**config_kwargs)
     backend_name = normalize_search_backend_name(config.SEARCH_BACKEND_ENGINE)
     backends = get_available_backends()

archivebox/search/query.py

@@ -1,5 +1,6 @@
 __package__ = "archivebox.search"
 
+from subprocess import CalledProcessError
 from typing import Any
 
 from django.db import connection
@@ -8,7 +9,7 @@ from django.db.models import Case, IntegerField, Q, QuerySet, Value, When
 from archivebox.config.common import get_config
 from archivebox.misc.logging import stderr
 from archivebox.misc.util import enforce_types
-from archivebox.search.backends import get_available_backends, get_backend, normalize_search_backend_name, search_backend_env
+from archivebox.search.backends import get_available_backends, get_backend, normalize_search_backend_name, search_backend_command_env
 from archivebox.search.config import get_search_mode, get_search_mode_backend, get_search_mode_base
 
 
@@ -220,6 +221,7 @@ def iter_query_search_ids(
     search_mode: str | None = None,
     config: dict[str, Any] | None = None,
     max_results: int | None = None,
+    stop_event=None,
     **config_kwargs: Any,
 ):
     """Yield snapshot IDs from configured search backend modules."""
@@ -266,6 +268,8 @@ def iter_query_search_ids(
     seen: set[str] = set()
     try:
         for backend_name in backend_names:
+            if stop_event is not None and stop_event.is_set():
+                return
             backend = backends[backend_name]
             try:
                 if backend_name == "sonic":
@@ -278,11 +282,22 @@ def iter_query_search_ids(
                     # --csv/--json. Keep daemon diagnostics on stderr.
                     with redirect_stdout(sys.stderr):
                         ensure_daemon_stack(reason="search query")
-                with search_backend_env(config=config):
-                    if backend_name == "ripgrep":
-                        ids = backend.iter_search(query, search_mode=search_mode_base)
-                    else:
-                        ids = backend.search(query)
+                from abx_dl.execution import iter_plugin_command
+                from archivebox.config.constants import CONSTANTS
+                from archivebox.plugins.discovery import get_plugin_catalog
+
+                command = get_plugin_catalog().command(backend.name, "search")
+                if command is None:
+                    raise RuntimeError(f'Plugin "{backend.name}" does not expose a search command')
+                ids = iter_plugin_command(
+                    command,
+                    arguments={"query": query, "search_mode": search_mode_base},
+                    env=search_backend_command_env(config=config),
+                    cwd=CONSTANTS.DATA_DIR,
+                    timeout=max(1, int(config.get("TIMEOUT", 60))) * 4,
+                    stop_event=stop_event,
+                )
+                try:
                     for snapshot_id in ids:
                         if snapshot_id in seen:
                             continue
@@ -290,21 +305,24 @@ def iter_query_search_ids(
                         yield snapshot_id
                         if max_results and len(seen) >= max_results:
                             return
+                finally:
+                    ids.close()
                 successful_backends += 1
             except Exception as err:
                 errors.append(err)
                 if not fallback_enabled:
                     raise
+        if not successful_backends and errors and search_mode_base == "deep":
+            raise errors[0]
     except Exception as err:
         stderr()
         stderr(
             f"[X] The search backend threw an exception={err}:",
             color="red",
         )
+        if isinstance(err, CalledProcessError) and err.stderr:
+            stderr(err.stderr.rstrip(), color="red")
         raise
-    else:
-        if not successful_backends and errors and search_mode_base == "deep":
-            raise errors[0]
 
 
 @enforce_types
@@ -318,8 +336,22 @@ def flush_search_index(snapshots: QuerySet, config: dict[str, Any] | None = None
     snapshot_pks = [str(pk) for pk in snapshots.values_list("pk", flat=True)]
 
     try:
-        with search_backend_env(config=config):
-            backend.flush(snapshot_pks)
+        from abx_dl.execution import iter_plugin_command
+        from archivebox.config.constants import CONSTANTS
+        from archivebox.plugins.discovery import get_plugin_catalog
+
+        command = get_plugin_catalog().command(backend.name, "flush")
+        if command is None:
+            raise RuntimeError(f'Plugin "{backend.name}" does not expose a flush command')
+        list(
+            iter_plugin_command(
+                command,
+                stdin=snapshot_pks,
+                env=search_backend_command_env(config=config),
+                cwd=CONSTANTS.DATA_DIR,
+                timeout=max(1, int(config.get("TIMEOUT", 60))) * 4,
+            ),
+        )
     except Exception as err:
         stderr()
         stderr(

archivebox/search/views.py

@@ -6,7 +6,7 @@ import json
 import threading
 import time
 from copy import copy
-from queue import Full, Queue
+from queue import Empty, Full, Queue
 from urllib.parse import urlsplit
 from uuid import UUID
 
@@ -254,15 +254,18 @@ def iter_filtered_search_result_ids(iterator, queryset, *, flush_max_delay=0.05)
         yield from flush_batch()
 
 
-def iter_search_result_ids(query, base_queryset, *, search_mode, config):
+def iter_search_result_ids(query, base_queryset, *, search_mode, config, stop_event=None):
     """Yield filtered Snapshot IDs from the selected search provider."""
     search_mode_base = get_search_mode_base(search_mode, config=config)
     provider = (
         iter_meta_search_ids(query, base_queryset)
         if search_mode_base == "meta"
-        else iter_query_search_ids(query, search_mode=search_mode, config=config)
+        else iter_query_search_ids(query, search_mode=search_mode, config=config, stop_event=stop_event)
     )
-    yield from iter_filtered_search_result_ids(provider, base_queryset)
+    try:
+        yield from iter_filtered_search_result_ids(provider, base_queryset)
+    finally:
+        provider.close()
 
 
 def snapshot_search_stream_response(query, base_queryset, *, search_mode, config, cache_key, thread_name):
@@ -300,7 +303,7 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
             iterator = None
             try:
                 close_old_connections()
-                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config)
+                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config, stop_event=stop_event)
                 for snapshot_id in iterator:
                     if stop_event.is_set():
                         break
@@ -326,10 +329,13 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
                 emit(None)
 
         threading.Thread(target=run_search, name=thread_name, daemon=True).start()
-        yield f"0{stream_padding}\n"
         try:
+            yield f"0{stream_padding}\n"
             while True:
-                item = await asyncio.to_thread(queue.get)
+                try:
+                    item = await asyncio.to_thread(queue.get, True, 0.1)
+                except Empty:
+                    continue
                 if item is None:
                     break
                 if isinstance(item, BaseException):

archivebox/services/archive_result_service.py

@@ -2,10 +2,7 @@ from __future__ import annotations
 
 import asyncio
 import inspect
-import json
 import os
-import re
-import signal
 import sys
 import time
 from contextlib import contextmanager
@@ -14,10 +11,9 @@ from pathlib import Path
 from typing import Any
 
 from asgiref.sync import sync_to_async
-from django.db import IntegrityError
 from django.utils import timezone
 
-from abx_dl.events import PROCESS_EXIT_SKIPPED, ArchiveResultEvent, ProcessCompletedEvent, ProcessStartedEvent, SnapshotEvent
+from abx_dl.events import ArchiveResultEvent, ProcessStartedEvent
 from abx_dl.output_files import OutputManifest
 from abx_dl.services.base import BaseService
 
@@ -87,11 +83,6 @@ def _normalize_status(status: str) -> str:
     return status or "failed"
 
 
-def _snapshot_hook_order(hook_name: str) -> int:
-    match = re.match(r"^on_Snapshot__(\d+)", hook_name or "")
-    return int(match.group(1)) if match else -1
-
-
 def _normalize_snapshot_title(candidate: str, *, snapshot_url: str) -> str:
     title = " ".join(line.strip() for line in str(candidate or "").splitlines() if line.strip()).strip()
     if not title:
@@ -128,37 +119,6 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh
     return len(next_title) > len(current)
 
 
-def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str:
-    if event.exit_code == PROCESS_EXIT_SKIPPED:
-        return "skipped"
-    if event.exit_code in {128 + signal.SIGHUP, 128 + signal.SIGINT, 128 + signal.SIGTERM}:
-        # This fallback only runs when a snapshot hook exited before emitting a
-        # structured ArchiveResult. A polite shutdown signal means the runner
-        # was interrupted during ownership transfer, not that the extractor
-        # produced a durable negative result. Keep the hook queued so the next
-        # runner can retry the exact work item instead of sealing in a transient
-        # process-lifecycle failure.
-        return "queued"
-    if event.exit_code != 0:
-        return "failed"
-    return "noresults"
-
-
-def _iter_archiveresult_records(stdout: str) -> list[dict]:
-    records: list[dict] = []
-    for raw_line in stdout.splitlines():
-        line = raw_line.strip()
-        if not line.startswith("{"):
-            continue
-        try:
-            record = json.loads(line)
-        except json.JSONDecodeError:
-            continue
-        if record.get("type") == "ArchiveResult":
-            records.append(record)
-    return records
-
-
 @_perf_trace("archivebox.ArchiveResultService._save_archiveresult_event_sync")
 def _save_archiveresult_event_to_db(
     event: ArchiveResultEvent,
@@ -209,7 +169,6 @@ def _save_archiveresult_event_to_db(
         start_ts = parse_event_datetime(event.start_ts)
         end_ts = parse_event_datetime(event.end_ts) or timezone.now()
         defaults = {
-            "hook_name": event.hook_name,
             "status": _normalize_status(event.status),
             "output_str": event.output_str,
             "output_json": event.output_json,
@@ -224,48 +183,13 @@ def _save_archiveresult_event_to_db(
         if event.error:
             defaults["notes"] = event.error
 
-    with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.result_lookup"):
-        result = ArchiveResult.objects.filter(
-            snapshot=snapshot,
-            plugin=event.plugin,
-        ).first()
-    if result is None:
-        try:
-            with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.result_create"):
-                result = ArchiveResult.objects.create(
-                    snapshot=snapshot,
-                    plugin=event.plugin,
-                    **defaults,
-                )
-        except IntegrityError:
-            with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.result_get_after_integrity"):
-                result = ArchiveResult.objects.get(
-                    snapshot=snapshot,
-                    plugin=event.plugin,
-                )
-
-    if result.output_files:
-        merged_output_files = {**result.output_files, **defaults["output_files"]}
-        defaults["output_files"], defaults["output_size"], defaults["output_mimetypes"] = _manifest_metadata(
-            OutputManifest.from_value(merged_output_files),
+    with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.result_get_or_create"):
+        result, _created = ArchiveResult.get_or_create_by_hook(
+            snapshot,
+            event.plugin,
+            event.hook_name,
+            defaults=defaults,
         )
-        defaults["output_size"] = max(defaults["output_size"], int(result.output_size or 0))
-        defaults["output_mimetypes"] = ",".join(
-            dict.fromkeys(
-                mimetype.strip()
-                for value in (defaults["output_mimetypes"], result.output_mimetypes)
-                for mimetype in value.split(",")
-                if mimetype.strip()
-            ),
-        )
-    if result.start_ts and defaults["start_ts"]:
-        defaults["start_ts"] = min(result.start_ts, defaults["start_ts"])
-    if result.end_ts and defaults["end_ts"]:
-        defaults["end_ts"] = max(result.end_ts, defaults["end_ts"])
-    if _snapshot_hook_order(result.hook_name) > _snapshot_hook_order(event.hook_name):
-        for field in ("hook_name", "status", "output_str", "output_json", "process_id", "notes"):
-            if field in result.__dict__:
-                defaults[field] = result.__dict__[field]
 
     with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.diff_fields"):
         update_fields = []
@@ -277,13 +201,6 @@ def _save_archiveresult_event_to_db(
         with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.result_update"):
             result.save(update_fields=[*update_fields, "modified_at"])
 
-    if result.status == ArchiveResult.StatusChoices.QUEUED:
-        # ArchiveResult has no retry_at column. If a shutdown/takeover projects
-        # a killed hook back to QUEUED, wake the parent Snapshot/Crawl so the
-        # next runner retries that exact hook instead of waiting on a stale
-        # active-state lease.
-        snapshot.update_and_requeue(retry_at=timezone.now())
-
     if result.status in (ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS):
         with _perf_span("archivebox.ArchiveResultService.on_ArchiveResultEvent.title_update"):
             title_output_str = result.output_str if result.status == ArchiveResult.StatusChoices.SUCCEEDED else ""
@@ -307,36 +224,45 @@ def _save_archiveresult_event_to_db(
 
 
 def mark_archiveresult_started(event: ProcessStartedEvent, *, snapshot_id: str, process_id: str) -> None:
-    """Advance an existing queued plugin row after its OS process is persisted."""
-    from archivebox.core.models import ArchiveResult
+    """Project a running abx-dl hook after its OS process is persisted."""
+    from archivebox.core.models import ArchiveResult, Snapshot
 
     started_at = parse_event_datetime(event.start_ts)
     if started_at is None:
         raise ValueError("ProcessStartedEvent.start_ts is required")
-    ArchiveResult.objects.filter(
-        snapshot_id=snapshot_id,
-        plugin=event.plugin_name,
-        status=ArchiveResult.StatusChoices.QUEUED,
-    ).update(
-        hook_name=event.hook_name,
-        status=ArchiveResult.StatusChoices.STARTED,
-        start_ts=started_at,
-        end_ts=None,
-        process_id=process_id,
-        modified_at=timezone.now(),
+    snapshot = Snapshot.objects.filter(id=snapshot_id).first()
+    if snapshot is None:
+        return
+    result, _created = ArchiveResult.get_or_create_by_hook(
+        snapshot,
+        event.plugin_name,
+        event.hook_name,
+        defaults={
+            "status": ArchiveResult.StatusChoices.STARTED,
+            "start_ts": started_at,
+            "end_ts": None,
+            "process_id": process_id,
+        },
     )
+    if result.start_ts is not None and started_at <= result.start_ts:
+        return
+    result.status = ArchiveResult.StatusChoices.STARTED
+    result.start_ts = started_at
+    result.end_ts = None
+    result.process_id = process_id
+    result.save(update_fields=["status", "start_ts", "end_ts", "process_id", "modified_at"])
 
 
 class ArchiveResultService(BaseService):
-    LISTENS_TO = [ArchiveResultEvent, ProcessCompletedEvent]
+    """Project abx-dl ArchiveResult facts into Django models."""
+
+    LISTENS_TO = [ArchiveResultEvent]
     EMITS = []
 
     def __init__(self, bus):
-        self._completed_process_event_ids: set[str] = set()
-        self._save_locks: dict[tuple[str, str], asyncio.Lock] = {}
+        self._save_locks: dict[tuple[str, str, str], asyncio.Lock] = {}
         super().__init__(bus)
         self.bus.on(ArchiveResultEvent, self.on_ArchiveResultEvent__save_to_db)
-        self.bus.on(ProcessCompletedEvent, self.on_ProcessCompletedEvent__save_to_db)
 
     @_perf_trace("archivebox.ArchiveResultService.on_ArchiveResultEvent__save_to_db")
     async def on_ArchiveResultEvent__save_to_db(self, event: ArchiveResultEvent) -> None:
@@ -348,68 +274,7 @@ class ArchiveResultService(BaseService):
                 where=lambda candidate: self.bus.event_is_child_of(event, candidate),
             )
 
-        key = (str(event.snapshot_id), event.plugin)
+        key = (str(event.snapshot_id), event.plugin, event.hook_name)
         lock = self._save_locks.setdefault(key, asyncio.Lock())
         async with lock:
             await sync_to_async(_save_archiveresult_event_to_db, thread_sensitive=True)(event, process_started)
-
-    @_perf_trace("archivebox.ArchiveResultService.on_ProcessCompletedEvent__save_to_db")
-    async def on_ProcessCompletedEvent__save_to_db(self, event: ProcessCompletedEvent) -> None:
-        if event.event_id in self._completed_process_event_ids:
-            return
-        self._completed_process_event_ids.add(event.event_id)
-
-        if not event.hook_name.startswith("on_Snapshot"):
-            return
-        with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.find_snapshot_event"):
-            snapshot_event = await self.bus.find(
-                SnapshotEvent,
-                past=True,
-                future=False,
-                where=lambda candidate: self.bus.event_is_child_of(event, candidate),
-            )
-        if snapshot_event is None:
-            return
-
-        with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.parse_stdout_records"):
-            records = _iter_archiveresult_records(event.stdout)
-        if records:
-            if len(records) > 1:
-                raise RuntimeError(
-                    f"Hook {event.plugin_name}:{event.hook_name} emitted {len(records)} ArchiveResult records; expected exactly one",
-                )
-            for record in records:
-                record_status = _normalize_status(record.get("status") or "")
-                record_failed = record_status == "failed" or (not record_status and event.exit_code not in (0, PROCESS_EXIT_SKIPPED))
-                with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.emit_archive_result_record"):
-                    await event.emit(
-                        ArchiveResultEvent(
-                            snapshot_id=record.get("snapshot_id") or snapshot_event.snapshot_id,
-                            plugin=record.get("plugin") or event.plugin_name,
-                            hook_name=record.get("hook_name") or event.hook_name,
-                            status=record_status,
-                            output_str=record.get("output_str") or "",
-                            output_json=record.get("output_json") if isinstance(record.get("output_json"), dict) else None,
-                            output_files=event.output_files,
-                            start_ts=event.start_ts,
-                            end_ts=event.end_ts,
-                            error=record.get("error") or (event.stderr if record_failed else ""),
-                        ),
-                    ).now()
-            return
-
-        process_failed = _status_for_process_without_archive_result(event) == "failed"
-        with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.emit_archive_result_fallback"):
-            await event.emit(
-                ArchiveResultEvent(
-                    snapshot_id=snapshot_event.snapshot_id,
-                    plugin=event.plugin_name,
-                    hook_name=event.hook_name,
-                    status=_status_for_process_without_archive_result(event),
-                    output_str=event.stderr if process_failed else "",
-                    output_files=event.output_files,
-                    start_ts=event.start_ts,
-                    end_ts=event.end_ts,
-                    error=event.stderr if process_failed else "",
-                ),
-            ).now()

archivebox/services/process_service.py

@@ -43,10 +43,10 @@ def normalize_process_env(env: dict) -> dict:
         if is_sensitive_config_key(key) or (key in config_input_names and key not in allowed_config_keys):
             normalized.pop(key, None)
     if selected_plugins:
-        from abx_dl.models import discover_plugins, filter_plugins
         from archivebox.config.common import _plugin_enabled_config_keys
+        from archivebox.plugins.discovery import get_plugin_catalog
 
-        selected_plugins = set(filter_plugins(discover_plugins(runtime="archivebox"), sorted(selected_plugins), include_providers=True))
+        selected_plugins = set(get_plugin_catalog().select(sorted(selected_plugins)))
         for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
             normalized.setdefault(enabled_key, "True" if plugin_name in selected_plugins else "False")
     return normalized
@@ -61,12 +61,8 @@ class ProcessService(BaseService):
     ]
     EMITS: ClassVar[list[type[BaseEvent]]] = []
 
-    def __init__(self, bus, *, parent_process_id: str | None = None):
+    def __init__(self, bus):
         self._iface = None
-        # A direct run_hook() call owns a private bus, so its caller-supplied
-        # parent applies to every process projected from that bus. Crawl buses
-        # leave this unset and derive their hierarchy from lifecycle events.
-        self.parent_process_id = parent_process_id
         self._completed_queue: asyncio.Queue[ProcessCompletedEvent | None] = asyncio.Queue()
         self._completed_worker: asyncio.Task | None = None
         super().__init__(bus)
@@ -104,7 +100,7 @@ class ProcessService(BaseService):
             process = await Process.objects.acreate(
                 machine=iface.machine,
                 iface=iface,
-                parent_id=self.parent_process_id,
+                parent_id=None,
                 process_type=process_type,
                 worker_type=worker_type,
                 pwd=event.output_dir,
@@ -138,7 +134,7 @@ class ProcessService(BaseService):
             hook_path=event.hook_path,
         )
         await Process.objects.filter(id=process.id).aupdate(
-            parent_id=self.parent_process_id or process.parent_id,
+            parent_id=process.parent_id,
             pwd=process.pwd,
             cmd=process.cmd,
             env=process.env,
@@ -225,7 +221,7 @@ class ProcessService(BaseService):
             await Process.objects.acreate(
                 machine=iface.machine,
                 iface=iface,
-                parent_id=self.parent_process_id,
+                parent_id=None,
                 process_type=process_type,
                 worker_type=worker_type,
                 pwd=event.output_dir,
@@ -246,7 +242,7 @@ class ProcessService(BaseService):
         updates = {
             "machine_id": iface.machine_id,
             "iface_id": iface.id,
-            "parent_id": self.parent_process_id or process.parent_id,
+            "parent_id": process.parent_id,
             "pwd": event.output_dir,
             "env": process_env,
             "pid": event.pid or process.pid,

archivebox/services/resource_admission.py

@@ -0,0 +1,91 @@
+"""Admission signals, not estimates of how much RAM an extractor will need."""
+
+import math
+import os
+import time
+from pathlib import Path
+
+
+RESOURCE_RECHECK_SECONDS = 2.0
+# Scheduling policy, not a minimum-RAM claim: stop increasing extraction
+# load when all non-idle work loses at least one second out of ten to memory
+# stalls. Require both recent and rolling pressure so transient reclaim does
+# not indefinitely starve otherwise viable work on small hosts.
+MEMORY_FULL_STALL_PERCENT = 10.0
+
+
+class ResourceAdmission:
+    def __init__(self):
+        self._memory_stalls: dict[str, tuple[int, float]] = {}
+        self._checked_at = float("-inf")
+        self._reason: str | None = None
+
+    @staticmethod
+    def cgroup_paths() -> list[Path]:
+        root = Path("/sys/fs/cgroup")
+        paths = [root]
+        try:
+            for line in Path("/proc/self/cgroup").read_text().splitlines():
+                if line.startswith("0::"):
+                    current = root / line[3:].lstrip("/")
+                    if current.is_dir() and current.is_relative_to(root):
+                        paths = [current, *[parent for parent in current.parents if parent.is_relative_to(root)]]
+                    break
+        except OSError:
+            pass
+        return paths
+
+    def snapshot_limit(self, configured: int) -> int:
+        cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 1)
+        for path in self.cgroup_paths():
+            try:
+                quota, period = (path / "cpu.max").read_text().split()
+                if quota != "max":
+                    cpus = min(cpus, max(1, math.ceil(int(quota) / int(period))))
+            except (OSError, ValueError, ZeroDivisionError):
+                continue
+        # CPU quota limits parallel admission, not forward progress. A host
+        # with a fractional CPU must still be able to run one snapshot.
+        return max(1, min(configured, cpus))
+
+    def observe_memory_stalls(self, source: str, total: int, avg10: float, sampled_at: float) -> bool:
+        previous = self._memory_stalls.get(source)
+        self._memory_stalls[source] = (total, sampled_at)
+        if previous is None:
+            return avg10 >= MEMORY_FULL_STALL_PERCENT
+        previous_total, previous_time = previous
+        elapsed = sampled_at - previous_time
+        if elapsed <= 0 or total < previous_total:
+            return False
+        stalled_percent = (total - previous_total) / (elapsed * 1_000_000) * 100
+        return avg10 >= MEMORY_FULL_STALL_PERCENT and stalled_percent >= MEMORY_FULL_STALL_PERCENT
+
+    def memory_pressure(self) -> str | None:
+        now = time.monotonic()
+        if now - self._checked_at < RESOURCE_RECHECK_SECONDS:
+            return self._reason
+        self._checked_at = now
+        reasons = []
+        cgroups = self.cgroup_paths()
+        for path in cgroups:
+            try:
+                current = int((path / "memory.current").read_text())
+                for name in ("memory.high", "memory.max"):
+                    limit = (path / name).read_text().strip()
+                    if limit != "max" and current >= int(limit):
+                        reasons.append(f"{name} reached")
+            except (OSError, ValueError):
+                continue
+        for path in [Path("/proc/pressure/memory"), *[p / "memory.pressure" for p in cgroups]]:
+            try:
+                full = next(line for line in path.read_text().splitlines() if line.startswith("full "))
+                fields = dict(field.split("=", 1) for field in full.split()[1:])
+                if self.observe_memory_stalls(str(path), int(fields["total"]), float(fields["avg10"]), now):
+                    reasons.append("memory stalls")
+            except (OSError, ValueError, KeyError, StopIteration):
+                continue
+        self._reason = ", ".join(dict.fromkeys(reasons)) or None
+        return self._reason
+
+
+resource_admission = ResourceAdmission()

archivebox/services/snapshot_service.py

@@ -1,12 +1,10 @@
 from __future__ import annotations
 
-import sys
+import asyncio
 from pathlib import Path
 
 from asgiref.sync import sync_to_async
 from django.utils import timezone
-from django.core.exceptions import ValidationError
-from rich import print as rprint
 from abx_dl.events import SnapshotCompletedEvent, SnapshotEvent
 from abx_dl.limits import CrawlLimitState
 from abx_dl.services.base import BaseService
@@ -44,6 +42,9 @@ def project_discovered_snapshots(snapshot_id: str) -> list:
 def finalize_completed_snapshot(
     snapshot_id: str,
     *,
+    owned_retry_at,
+    was_sealed: bool,
+    consumed_retry_plugins: list[str] | None = None,
     output_dir=None,
     crawl_limit_stop_reason: str | None = None,
 ) -> None:
@@ -73,13 +74,40 @@ def finalize_completed_snapshot(
             modified_at=timezone.now(),
         )
 
-    if snapshot.status == Snapshot.StatusChoices.QUEUED:
-        snapshot.advance_lifecycle()
-        snapshot.refresh_from_db()
-    if snapshot.status == Snapshot.StatusChoices.STARTED and snapshot.is_finished_processing():
+    # SnapshotCompletedEvent is abx-dl's authoritative signal that the complete
+    # snapshot hook sequence (including cleanup) finished. ArchiveResult rows
+    # are projections of that work, never prerequisites used to decide whether
+    # the Snapshot may seal.
+    if not was_sealed and snapshot.status == Snapshot.StatusChoices.STARTED and snapshot.retry_at == owned_retry_at:
         snapshot.seal()
         snapshot.refresh_from_db()
 
+    consumed_plugins = sorted({str(name).strip() for name in (consumed_retry_plugins or []) if str(name).strip()})
+    if consumed_plugins and snapshot.status == Snapshot.StatusChoices.SEALED:
+        for _attempt in range(3):
+            current = Snapshot.objects.only("status", "retry_at", "config").get(pk=snapshot.pk)
+            pending_plugins = sorted(
+                {str(name).strip() for name in (current.config or {}).get("RETRY_PLUGINS", []) if str(name).strip()},
+            )
+            if pending_plugins != consumed_plugins:
+                break
+            owned_filter = {"retry_at": owned_retry_at} if was_sealed else {"retry_at__isnull": True}
+            updates: dict[str, object] = {
+                "config": {key: value for key, value in current.config.items() if key != "RETRY_PLUGINS"},
+                "modified_at": timezone.now(),
+            }
+            if was_sealed:
+                updates["retry_at"] = None
+            updated = Snapshot.objects.filter(
+                pk=current.pk,
+                status=Snapshot.StatusChoices.SEALED,
+                config=current.config,
+                **owned_filter,
+            ).update(**updates)
+            if updated:
+                snapshot.refresh_from_db()
+                break
+
     snapshot.write_index_jsonl(output_dir=output_dir)
 
 
@@ -100,6 +128,8 @@ class SnapshotService(BaseService):
 
     def __init__(self, bus, *, crawl_id: str):
         self.crawl_id = crawl_id
+        self._run_ownership: dict[str, tuple[str, object, bool, list[str]]] = {}
+        self._ownership_lock = asyncio.Lock()
         super().__init__(bus)
         self.bus.on(SnapshotEvent, self.on_SnapshotEvent)
         self.bus.on(SnapshotCompletedEvent, self.on_SnapshotCompletedEvent)
@@ -112,33 +142,49 @@ class SnapshotService(BaseService):
         if snapshot is not None:
             if snapshot.is_paused:
                 return
+            was_sealed = snapshot.status == Snapshot.StatusChoices.SEALED
             if snapshot.status == Snapshot.StatusChoices.QUEUED:
-                if not await snapshot.archiveresult_set.aexists():
-                    from archivebox.services.runner import snapshot_hooks_for_pending_archiveresults
-
-                    hooks = await sync_to_async(snapshot_hooks_for_pending_archiveresults, thread_sensitive=True)(snapshot)
-                    await sync_to_async(snapshot.create_pending_archiveresults, thread_sensitive=True)(hooks=hooks)
-                try:
-                    await sync_to_async(snapshot.advance_lifecycle, thread_sensitive=True)()
-                except ValidationError as err:
-                    if "ArchiveBox cannot archive its own admin, web, api, or snapshot URLs." not in str(err):
-                        raise
-                    await Snapshot.objects.filter(id=snapshot.id).aupdate(
-                        status=Snapshot.StatusChoices.SEALED,
-                        retry_at=None,
-                        modified_at=timezone.now(),
-                    )
-                    rprint(
-                        f"[red][X] Refusing to archive ArchiveBox internal URL for security: {snapshot.url}[/red]",
-                        file=sys.stderr,
-                    )
-                    return
+                await sync_to_async(snapshot.advance_lifecycle, thread_sensitive=True)()
                 await sync_to_async(snapshot.refresh_from_db, thread_sensitive=True)()
-            elif snapshot.status != Snapshot.StatusChoices.STARTED:
+            elif snapshot.status not in (Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED):
                 return
-            if snapshot.status != Snapshot.StatusChoices.STARTED:
-                return
-            await sync_to_async(snapshot.ensure_crawl_symlink, thread_sensitive=True)()
+            if snapshot.status == Snapshot.StatusChoices.STARTED:
+                await sync_to_async(snapshot.ensure_crawl_symlink, thread_sensitive=True)()
+            retry_plugins = [str(name).strip() for name in (snapshot.config or {}).get("RETRY_PLUGINS", []) if str(name).strip()]
+            async with self._ownership_lock:
+                self._run_ownership[str(event.snapshot_id)] = (str(event.event_id), snapshot.retry_at, was_sealed, retry_plugins)
+
+    async def renew_lease(self, snapshot_id: str, lease_until) -> bool | None:
+        from archivebox.core.models import Snapshot
+
+        async with self._ownership_lock:
+            ownership = self._run_ownership.get(str(snapshot_id))
+            if ownership is None or ownership[2]:
+                return None
+            event_id, owned_retry_at, was_sealed, retry_plugins = ownership
+            updated = await Snapshot.objects.filter(
+                id=snapshot_id,
+                status=Snapshot.StatusChoices.STARTED,
+                retry_at=owned_retry_at,
+            ).aupdate(
+                retry_at=lease_until,
+                modified_at=timezone.now(),
+            )
+            if updated:
+                self._run_ownership[str(snapshot_id)] = (event_id, lease_until, was_sealed, retry_plugins)
+            return bool(updated)
 
     async def on_SnapshotCompletedEvent(self, event: SnapshotCompletedEvent) -> None:
-        await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(event.snapshot_id)
+        snapshot_id = str(event.snapshot_id)
+        async with self._ownership_lock:
+            ownership = self._run_ownership.get(snapshot_id)
+            if ownership is None or ownership[0] != str(event.event_parent_id):
+                return
+            self._run_ownership.pop(snapshot_id, None)
+        _, owned_retry_at, was_sealed, retry_plugins = ownership
+        await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(
+            event.snapshot_id,
+            owned_retry_at=owned_retry_at,
+            was_sealed=was_sealed,
+            consumed_retry_plugins=retry_plugins,
+        )

archivebox/services/tag_service.py

@@ -1,5 +1,7 @@
 from __future__ import annotations
 
+from asgiref.sync import sync_to_async
+
 from abx_dl.events import TagEvent
 from abx_dl.services.base import BaseService
 
@@ -13,10 +15,10 @@ class TagService(BaseService):
         self.bus.on(TagEvent, self.on_TagEvent__save_to_db)
 
     async def on_TagEvent__save_to_db(self, event: TagEvent) -> None:
-        from archivebox.core.models import Snapshot, SnapshotTag, Tag
+        from archivebox.core.models import Snapshot, Tag
 
         snapshot = await Snapshot.objects.filter(id=event.snapshot_id).afirst()
         if snapshot is None:
             return
-        tag, _ = await Tag.objects.aget_or_create(name=event.name)
-        await SnapshotTag.objects.aget_or_create(snapshot=snapshot, tag=tag)
+        tag, _ = await sync_to_async(Tag.get_or_create_by_name, thread_sensitive=True)(event.name)
+        await sync_to_async(snapshot.add_tag_ids, thread_sensitive=True)([tag.pk])

archivebox/templates/admin/base.html

@@ -1339,20 +1339,36 @@
                 display: inline-block;
             }
 
-            .action-rearchive-select,
-            .action-permissions-select {
+            /* The native select receives the actual click, including in WKWebView. */
+            body.change-list #changelist .actions .action-rearchive-select,
+            body.change-list #changelist .actions .action-permissions-select {
                 position: absolute;
-                left: 0;
-                bottom: 0;
-                width: 1px;
-                height: 1px;
+                inset: 0;
+                width: 100%;
+                height: 100%;
                 opacity: 0;
-                pointer-events: none;
-                border: 0;
-                padding: 0;
+                cursor: pointer;
+                z-index: 1;
                 margin: 0;
             }
 
+            body.change-list #changelist .action-rearchive-wrapper > .button,
+            body.change-list #changelist .action-permissions-wrapper > .button {
+                padding: 8px 16px !important;
+                width: max-content;
+                min-width: max-content;
+                flex: 0 0 auto;
+                gap: 6px;
+                white-space: nowrap;
+            }
+
+            .action-rearchive-wrapper:focus-within,
+            .action-permissions-wrapper:focus-within {
+                outline: 2px solid #2563eb;
+                outline-offset: 2px;
+                border-radius: 6px;
+            }
+
             /* Object count */
             .actions .action-counter {
                 color: #64748b;
@@ -1960,22 +1976,10 @@
                             .addClass('action-rearchive-select')
                             .attr('aria-label', 'Re-Archive')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .addClass('button')
                             .html('Re-Archive <span aria-hidden="true">▾</span>')
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         rearchiveActions.forEach(function (action_value) {
@@ -2021,23 +2025,11 @@
                             .addClass('action-permissions-select')
                             .attr('aria-label', 'Permissions')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .attr('name', action_type)
                             .addClass('button')
                             .text(this.text)
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         ;[