Honor explicit allauth opt-in

# Conflicts:
#	archivebox/config/common.py
#	archivebox/templates/admin/login.html
#	uv.lock

.github/CONTRIBUTING.md

@@ -12,7 +12,7 @@
 - https://github.com/ArchiveBox/ArchiveBox/issues
 - https://github.com/ArchiveBox/ArchiveBox/pulls
 - https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-c-bare-metal-setup
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
 
 ### Development Setup
 

.github/ISSUE_TEMPLATE/1-bug_report.yml

@@ -30,8 +30,8 @@ body:
       description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
       render: markdown
       placeholder: |
-        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 5797:5797 archivebox/archivebox` in iTerm2
-        2. Went to the https://127.0.0.1:5797/add/ page in Google Chrome
+        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
+        2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
         3. Typed 'https://example.com/xyz' into the 'Add URL' input field
         4. Clicked the 'Add+' button
         5. Got a 500 error and saw the errors below in terminal
@@ -138,7 +138,7 @@ body:
             archivebox:
                 image: archivebox/archivebox:latest
                 ports:
-                    - 5797:5797
+                    - 8000:8000
                 volumes:
                     - ./data:/data
                 environment:

.github/STAGING.md

@@ -1,42 +0,0 @@
-# Stable releases require live dev acceptance
-
-The order is **dev CI → dev image publication → automatic Cabbage deployment →
-real capture and browser acceptance → stable publication**. Never
-substitute a published PyPI wheel, healthy container, or local build for this
-acceptance. The release workflow blocks *before* a stable upload unless Cabbage
-has successful evidence from `staging-acceptance.yml` matching all tracked
-source and dependency pins. Only ArchiveBox's own version fields may change
-between dev and stable.
-
-The staging workflow observes the existing image watcher; it cannot deploy or
-restart the application. On Cabbage, install `bin/staging-acceptance-host.sh`
-as root-owned mode 0755 `/usr/local/sbin/archivebox-staging-acceptance`. Its hash
-is checked against the candidate. A dedicated SSH key must use
-`restrict,command="/usr/local/sbin/archivebox-staging-acceptance"` in authorized
-keys. It accepts only `inspect <source-sha>` and `capture <source-sha>`; it cannot
-run caller-provided commands or archive arbitrary URLs.
-
-The `staging` GitHub environment is restricted to dev/main branches. It contains
-`STAGING_SSH_KEY` and `STAGING_KNOWN_HOSTS`, with independently verified host keys.
-Only trusted release events use the private ugNAS runner route. Never expose
-these credentials to PR jobs or copy an operator's existing root key.
-
-Acceptance creates one tagged background crawl per source through the normal
-add/runner path, enabling every snapshot extractor, including TLSNotary. It
-checks installed pins, completed outputs and their files, unexpected/missing
-plugin results, host OOM kills, and real rendered snapshot pages. For this
-release, the explicitly accepted TLSNotary failures and exact missing-Claude-auth
-diagnostics are retained in the report's `limitations` list; they do not disable
-those plugins or excuse missing/truncated successful outputs. Remove the
-TLSNotary exception when its follow-up fix ships. Logs, result inventories and
-screenshots are retained as Actions artifacts. Other failures remain
-failures; do not publish a success deployment record to bypass them. No code
-mounts are allowed. The Cabbage job must pass before its record authorizes a
-stable upload. Queued capture progress is polled; failed tests are not retried.
-
-Browser acceptance follows the default WACZ viewer into the captured document,
-matching its replay base and original URL and checking that its body is visible
-and contains substantial text. An attached viewer iframe can still be
-`about:blank`, so its presence alone cannot prove replay works. Extracted display
-titles and the captured document's `<title>` may legitimately differ; neither
-title equality nor output thumbnails substitute for checking the actual page.

.github/actionlint.yaml

@@ -1,3 +0,0 @@
-self-hosted-runner:
-  labels:
-    - ugnas-linux

.github/configs/ci-tooling.json

@@ -116,10 +116,6 @@
     }
   ],
   "docker_binaries": [
-    {
-      "name": "{BASH_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
     {
       "name": "{DOCKER_BINARY}",
       "binproviders": "env"

.github/dependabot.yml

@@ -15,7 +15,7 @@ updates:
         patterns:
           - "*"
   - package-ecosystem: "npm"
-    directory: "/etc"
+    directory: "/"
     target-branch: "dev"
     schedule:
       interval: "monthly"

.github/lychee.toml

@@ -1,38 +0,0 @@
-# Documentation destinations only: do not crawl archived URLs, app routes, or image assets.
-# Manual run from the repository root with Lychee 0.24.2:
-# git ls-files -- '*.md' '*.html' '*.py' '*.rst' '*.yml' '*.yaml' '*.sh' '*.toml' > /tmp/archivebox-link-inputs.txt
-# lychee --config .github/lychee.toml --root-dir "$PWD" --files-from /tmp/archivebox-link-inputs.txt
-# Explicit tracked inputs include README symlinks and let errors in one file leave others checkable.
-include_fragments = "anchor-only"
-include_verbatim = true
-max_retries = 0
-max_concurrency = 4
-timeout = 20
-no_progress = true
-hidden = true
-extensions = ["md", "html", "py", "rst", "yml", "yaml", "sh", "toml"]
-exclude_path = ['(^|/)(tests|data|apidocs|node_modules|\.git)(/|$)']
-# Includes override this exclude; this is a documentation-link allowlist.
-exclude = ['.*']
-include = [
- '^https://github\.com/ArchiveBox/ArchiveBox/blob/[^/]+/.*\.md(?:#|$)',
- '^https://raw\.githubusercontent\.com/wiki/ArchiveBox/ArchiveBox/',
- '^https?://github\.com/(?:ArchiveBox|pirate)/ArchiveBox(?:/wiki(?:/|#|$)|/?#)',
- '^https?://docs\.archivebox\.io(?:/|$)',
- '^file:.*\.md(?:#|$)',
- '^file:.*/docs/[^/.#]+(?:#|$)',
-]
-fallback_extensions = ["md"]
-remap = [
- # Lychee's Markdown slugger drops variation selectors. Check these against rendered HTML.
-
- '^file:.*/README\.md(#%EF%B8%8F-cli-usage)$ https://github.com/ArchiveBox/ArchiveBox${1}',
- '^file:.*/docs/Install\.md(#option-a-docker--docker-compose-setup-%EF%B8%8F)$ https://github.com/ArchiveBox/ArchiveBox/wiki/Install${1}',
- # The wiki accepts this case alias, but raw file paths are case-sensitive.
- '^https?://github\.com/(?:ArchiveBox|pirate)/ArchiveBox/wiki/Setting-up-Storage$ https://raw.githubusercontent.com/wiki/ArchiveBox/ArchiveBox/Setting-Up-Storage.md',
-
- # Bare missing wiki pages silently redirect to Home; raw pages return an actual 404.
- # Keep fragment URLs on the rendered wiki to catch published heading drift.
- '^https?://github\.com/(?:ArchiveBox|pirate)/ArchiveBox/wiki/([^#/?]+)$ https://raw.githubusercontent.com/wiki/ArchiveBox/ArchiveBox/${1}.md',
-]
-preprocess = { command = "bin/lychee_preprocess.py" }

.github/pages/README.md

@@ -1,124 +0,0 @@
-# archivebox.io website
-
-This repository owns its complete website. Build and deploy it without checking
-out another repository's website, fetching shared snippets, or calling a shared
-workflow. Common files are ordinary committed copies; update them deliberately.
-
-## Repository-specific files
-
-- Content: `publicsite/index.html`.
-- Site/gallery generation: `bin/generate_ui_screenshot_gallery.py`.
-- Application capture: `bin/collect_ui_screenshots.sh`.
-- Deployment branch: `dev`.
-
-## Common publication contract
-
-1. Build pull requests without deploying them.
-2. Build website pushes immediately, independently of application capture CI.
-3. Restore validated screenshots from trusted default-branch capture jobs.
-   Keep the original capture revision, timestamp, coverage and image checksums.
-4. Retain `site-screenshots` artifacts for 90 days. After expiry, restore the
-   published image set. Never replace a complete gallery with an empty one when
-   restoration fails. Apple device groups are independently validated artifacts;
-   Android's checked-in bootstrap captures retain their local provenance.
-5. Render current content and layout around those captures. Site revision and
-   capture revision are separate facts; reusing captures must not relabel them.
-6. Serialize production restore/build/deploy under one concurrency group and
-   check out the latest default-branch revision when each build starts. Keep the
-   built revision in provenance; later release-bot commits must not block deploys.
-7. Keep platform-specific build, installation, capture, and completeness checks
-   in the application's existing capture workflow. Site-only edits do not need
-   to rerun those jobs.
-
-## Common presentation contract
-
-The base header, Apps menu and icons, footer, brand styles and navigation behavior
-should remain identical copies across the seven product sites. Put this site's
-name, navigation, CTA and extra footer resources outside the common base files.
-Preserve real links, anchor IDs, no-JavaScript navigation, keyboard operation,
-mobile layout, reduced-motion preferences and existing product-specific content.
-Screenshots stay in English. Translation content, when added, lives in separate
-HTML files; no runtime translation service or cross-repository dependency.
-
-## Editing and building
-
-`base/` contains identical independent copies across the seven product sites.
-Common Python scripts share the same behavior; repository formatters may change
-line wrapping without changing their syntax trees. The base owns the shared logo, Apps menu/icons, footer directory, navigation
-behavior and styles. `site.json` and `nav.html` contain this site's name, routes,
-links and CTA. Product content and additional footer resources stay in the native
-HTML/Jinja/Liquid templates listed above. The `ARCHIVEBOX:*` comments are build
-slots; the renderer fills them with static HTML, never browser-side fetches.
-
-The existing builder calls `site.py render` after generating its content. This
-copies `base/chrome.css` and `base/chrome.js` to `site-base/`, assembles the common
-HTML, validates local resources and writes the site revision to root `build.json`.
-Screenshot manifests are left untouched. No build reads another repository.
-
-Common changes are manual copy/paste edits. Compare `base/` and `site.py` with
-another product repository when updating them; do not add a synchronization job,
-package dependency or reusable cross-repository workflow. `nav.html`, `site.json`
-and the marked workflow build section are intentionally repository-specific.
-
-Build: `uv run --no-project python .github/pages/site.py render _site --source publicsite` after restoring screenshots and refreshing the gallery.
-
-Check the built site with real Chromium (also run by Pages CI):
-
-```sh
-export ABXPKG_LIB_DIR=/tmp/archivebox-site-browser
-CHROME_SCHEMA="$(uv run python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "chrome" / "config.json")')"
-uv run abxpkg env --install --deps-from="$CHROME_SCHEMA:required_binaries"
-export CHROME_BINARY="$ABXPKG_LIB_DIR/env/bin/chromium"
-uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py SITE_OUTPUT --evidence /tmp/site-evidence
-```
-
-The browser comes from the normal abxpkg provider/cache path; Playwright drives
-that resolved executable instead of installing a second Chromium build.
-Use this workflow's `SITE_OUTPUT` directory. The check covers desktop/mobile
-layout, keyboard dismissal, no-JavaScript links, local resources and visible
-images, and uploads its screenshots as `site-verification` for 14 days.
-
-## Branding and future translations
-
-`base/chrome.css` owns the shared system font stacks, plum accent and CTA shape,
-including the existing `.button` and `.ab-btn` classes. Product styles own their
-layouts, feature sections and platform-specific button states. Shared system
-fonts need no external font downloads. `site.py` emits the same complete SEO/OG/
-Twitter tags everywhere while retaining each native template's title, description,
-canonical URL, locale and social artwork; edit those values with that page's copy.
-
-For translations, keep complete separate HTML files under `es/`, `fr/` and `zh/`,
-and list their output paths in `site.json` alongside the English pages. Share CSS,
-JS, social artwork and the English screenshot gallery. Translate each page's title
-and description, set its own canonical and `lang` (`zh` for Chinese), and add
-reciprocal `hreflang` links only for translations that actually exist. The renderer
-preserves these tags and computes shared asset paths for nested pages. Localized
-header/footer copy and the small language selector can be added with those pages;
-there is no language redirect, placeholder translation or translation runtime now.
-
-## Screenshot strips
-
-`marquee.py` and `base/marquee.css` / `base/marquee.js` are also identical copies.
-Product homepages choose their placement with an `ARCHIVEBOX:MARQUEE` comment.
-The renderer takes real images, captions, dimensions and links from the generated
-`screenshots/index.html`; there is no second screenshot list or browser fetch.
-Desktop/tablet/mobile galleries contribute their desktop views. Apple uses
-`MARQUEE-CLIENT` and `MARQUEE-SERVER` slots to keep its two products separate.
-Catalog sites without a slot do not load the strip assets.
-
-The strip scrolls once, pauses on hover, focus or manual interaction, and has an
-explicit Play/Pause button. Reduced-motion users start paused. Without JavaScript,
-normal image links and horizontal scrolling remain available. Screenshot capture
-manifests and provenance are never rewritten by this presentation step.
-
-Standalone one- or two-image illustrations remain in their authored source.
-Gallery strips with three or more images use the latest restored captures.
-
-## Keeping the common layout in sync
-
-The comments in `base/header.html`, `base/footer-start.html`,
-`base/footer-end.html`, `base/chrome.css`, and `base/chrome.js` list the peer
-repositories. Update those committed copies together when shared branding,
-Apps links, keyboard behavior, or footer resources change. Preserve each site's
-local navigation and content. The former monorepo `site-chrome` generator is
-retired; builds must use only this repository's files.

.github/pages/artifacts.py

@@ -1,102 +0,0 @@
-"""Common screenshot artifact transport; coverage checks belong to each app."""
-
-import json
-import subprocess
-from pathlib import PurePosixPath
-from urllib.request import urlopen
-
-
-def api(repo, path):
-    return json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/{path}"]))
-
-
-def trusted(run, repo, workflow, branch, successful=True):
-    return (
-        (run.get("head_repository") or {}).get("full_name", "").lower() == repo.lower()
-        and run["head_branch"] == branch
-        and run["event"] in ("push", "workflow_dispatch", "schedule")
-        and run["path"] == f".github/workflows/{workflow}"
-        and (not successful or run["conclusion"] == "success")
-    )
-
-
-def runs(repo, workflow, branch, successful=True, artifact_names=("site-screenshots",)):
-    # Ask for capture artifacts directly instead of checking every historical
-    # workflow run (most app runs never produce a complete screenshot set).
-    # Artifact creation time also includes fresh captures from old rerun jobs.
-    candidates = []
-    for name in artifact_names:
-        rows = subprocess.check_output(
-            [
-                "gh",
-                "api",
-                f"repos/{repo}/actions/artifacts?name={name}&per_page=100",
-                "--paginate",
-                "--jq",
-                ".artifacts[] | select(.expired == false) | @json",
-            ],
-            text=True,
-        )
-        candidates.extend(json.loads(row) for row in rows.splitlines())
-    seen = set()
-    for artifact in sorted(candidates, key=lambda item: item["created_at"], reverse=True):
-        source = artifact.get("workflow_run") or {}
-        run_id = source.get("id")
-        if (
-            artifact["name"] not in artifact_names
-            or not run_id
-            or run_id in seen
-            or source.get("head_branch") != branch
-            or source.get("head_repository_id") != source.get("repository_id")
-        ):
-            continue
-        seen.add(run_id)
-        run = api(repo, f"actions/runs/{run_id}")
-        if trusted(run, repo, workflow, branch, successful):
-            yield run
-
-
-def names(repo, run):
-    rows = subprocess.check_output(
-        [
-            "gh",
-            "api",
-            f"repos/{repo}/actions/runs/{run['id']}/artifacts?per_page=100",
-            "--paginate",
-            "--jq",
-            ".artifacts[] | @json",
-        ],
-        text=True,
-    )
-    return {artifact["name"] for row in rows.splitlines() if not (artifact := json.loads(row))["expired"]}
-
-
-def download(repo, run, name, destination):
-    subprocess.run(
-        [
-            "gh",
-            "run",
-            "download",
-            str(run["id"]),
-            "--repo",
-            repo,
-            "--name",
-            name,
-            "--dir",
-            str(destination),
-        ],
-        check=True,
-    )
-
-
-def relative_path(name):
-    path = PurePosixPath(name)
-    if not name or path.is_absolute() or ".." in path.parts or any(c in name for c in ":\\?#%"):
-        raise ValueError(f"Unsafe screenshot path: {name}")
-    return path
-
-
-def fetch(base, name):
-    relative_path(name)
-    with urlopen(base + name, timeout=60) as response:
-        return response.read()

.github/pages/base/chrome.css

@@ -1,156 +0,0 @@
-/*
-Shared ArchiveBox website navigation styles and behavior.
-Keep this file in sync with .github/pages/base/chrome.css in:
-ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
-electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
-Each site builds its committed copy independently; there is no central generator.
-Keep site-specific navigation, CTAs, and extra footer links outside this base.
-Also review the inline navbar/footer in abx-dl/website/index.html and
- debian-archivebox/website/index.html when shared links or branding change.
-*/
-/* Common brand tokens. Native templates own product layout and content. */
-:root {
-  --abx-font-sans: -apple-system, BlinkMacSystemFont, "Segoe UI", Roboto, Helvetica, Arial, sans-serif;
-  --abx-font-mono: ui-monospace, "SFMono-Regular", Consolas, "Liberation Mono", Menlo, monospace;
-  --abx-accent: #9b2854;
-  --abx-accent-hover: #782040;
-  --abx-paper: #fcfaf7;
-  --abx-ink: #28252a;
-  --abx-muted: #716c74;
-  --abx-line: #e9e2e3;
-  --font-sans: var(--abx-font-sans);
-  --font-mono: var(--abx-font-mono);
-  --ab-font-sans: var(--abx-font-sans);
-  --ab-font-serif: var(--abx-font-sans);
-  --ab-font-label: var(--abx-font-sans);
-  --ab-font-mono: var(--abx-font-mono);
-  --ab-accent: var(--abx-accent);
-  --ab-accent-hover: var(--abx-accent-hover);
-  --ab-on-accent: #fff;
-}
-body { font-family: var(--abx-font-sans); }
-code, pre, kbd, samp { font-family: var(--abx-font-mono); }
-/* Reuse existing CTA classes; platform selection continues to control colors. */
-.button, .ab-btn {
-  display: inline-flex; align-items: center; justify-content: center; gap: 10px;
-  min-height: 44px; height: auto; max-width: 100%; padding: 10px 20px;
-  border-radius: 24px; font-family: var(--abx-font-sans);
-  font-size: 14px; font-weight: 650; line-height: 1.4;
-  text-align: center; white-space: normal; text-decoration: none;
-}
-.button.primary, .ab-btn--primary { background: var(--abx-accent); border-color: var(--abx-accent); color: #fff; }
-.button.primary:hover, .ab-btn--primary:hover { background: var(--abx-accent-hover); border-color: var(--abx-accent-hover); color: #fff; }
-.button:focus-visible, .ab-btn:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
-.button svg, .button img, .ab-btn svg, .ab-btn img { flex-shrink: 0; }
-@media (prefers-reduced-motion: reduce) { .button, .ab-btn { transition: none; } .button:hover { transform: none; } }
-
-/* Shared ArchiveBox site chrome. Independent copy; edit alongside the other product sites, without a build dependency. */
-/* Native anchor navigation leaves room for the heading and its surrounding content. */
-html { scroll-behavior: smooth; scroll-padding-top: clamp(96px, 25vh, 240px); }
-@media (prefers-reduced-motion: reduce) { html { scroll-behavior: auto; } }
-.abx-header, .abx-footer { box-sizing: border-box; font-family: var(--abx-font-sans); font-size: 14px; line-height: 1.5; letter-spacing: normal; text-align: left; }
-.abx-header *, .abx-footer * { box-sizing: border-box; }
-.abx-header { position: sticky; top: 0; z-index: 100; display: flex; align-items: center; justify-content: space-between; gap: 24px; width: 100%; max-width: none; margin: 0; padding: 15px max(24px, calc((100% - 1280px) / 2)); border: 0; border-bottom: 1px solid #e9e2e3; background: #fcfaf7f5; color: #28252a; backdrop-filter: blur(16px); }
-.abx-header a, .abx-footer a { text-decoration: none; text-underline-offset: .24em; }
-.abx-header .abx-brand, .abx-footer .abx-brand { display: inline-flex; flex: 0 0 auto; align-items: center; gap: 11px; margin: 0; padding: 0; color: inherit; font-size: 19px; font-weight: 750; line-height: 1.2; letter-spacing: -.5px; white-space: nowrap; }
-.abx-brand .abx-logo { display: block; flex: 0 0 auto; width: 32px; height: 32px; margin: 0; }
-.abx-header .abx-subsite { transform: translateY(3px); margin-left: 2px; color: #716c74; font-size: 12px; font-weight: 450; letter-spacing: 0; }
-.abx-header .abx-nav { display: flex; align-items: center; justify-content: flex-end; flex-wrap: wrap; gap: 0; width: auto; max-width: none; margin: 0; padding: 0; font-size: 13px; font-weight: 600; }
-.abx-nav > a, .abx-nav > details { margin: 0 0 0 15px; padding: 0 0 0 15px; border: 0; border-left: 1px solid #ded4d8; border-radius: 0; background: none; }
-.abx-nav > :first-child { margin-left: 0; padding-left: 0; border-left: 0; }
-.abx-header .abx-nav a { display: inline-flex; align-items: center; min-height: 36px; color: #4d424b; white-space: nowrap; }
-.abx-header .abx-nav a:hover { color: #9b2854; }
-.abx-header .abx-nav .abx-cta { margin-left: 18px; padding: 9px 17px; border: 1px solid #9b2854; border-radius: 24px; background: #9b2854; color: #fff; }
-.abx-header .abx-nav .abx-cta:hover { background: #782040; color: #fff; }
-.abx-header .abx-apps { position: relative; }
-.abx-apps > summary { display: flex; align-items: center; gap: 7px; min-height: 36px; list-style: none; cursor: pointer; color: #4d424b; }
-.abx-apps > summary::-webkit-details-marker { display: none; }
-.abx-apps > summary::before { content: '›'; display: inline-block; color: #9b2854; font-size: 22px; line-height: 1; transition: transform .15s; }
-.abx-apps[open] > summary::before { transform: rotate(90deg); }
-.abx-header .abx-app-links { position: absolute; right: 0; top: calc(100% + 12px); width: 440px; max-width: calc(100vw - 24px); max-height: min(70vh, 580px); overflow-y: auto; padding: 8px; border: 1px solid #e9e2e3; border-radius: 16px; background: #fcfaf7; box-shadow: 0 18px 48px #2d152522, 0 3px 8px #2d152508; }
-.abx-header .abx-app-row { display: flex; align-items: center; gap: 8px; min-height: 44px; padding: 4px 6px; border-radius: 10px; transition: background .15s; }
-.abx-header .abx-app-row:hover, .abx-header .abx-app-row:focus-within { background: #f2e8ed; }
-.abx-header .abx-app-links .abx-app-name { flex: 1; min-width: 0; min-height: 34px; padding: 5px 4px; border-radius: 6px; font-size: 13px; line-height: 1.35; white-space: normal; }
-.abx-header .abx-platforms { display: flex; flex: 0 0 auto; align-items: center; gap: 5px; }
-.abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 auto; height: 30px; min-height: 30px; padding: 5px; border: 1px solid #e8dfe3; border-radius: 9px; background: #fffdfb; color: #4d424b; box-shadow: 0 2px 3px #2d152510, inset 0 1px 0 #fff; transition: transform .15s, box-shadow .15s, border-color .15s; }
-.abx-header .abx-app-links .abx-platform { width: 30px; }
-.abx-header .abx-platform svg { display: block; width: 18px; height: 18px; margin: 0; fill: currentColor; }
-.abx-header .abx-app-links .abx-platform-docker { color: #2386ce; }
-.abx-header .abx-app-links .abx-platform-windows, .abx-header .abx-app-links .abx-platform-app-store, .abx-header .abx-app-links .abx-platform-safari { color: #1879c4; }
-.abx-header .abx-app-links .abx-platform-android { color: #29965d; }
-.abx-header .abx-app-links .abx-platform-firefox { color: #e46b24; }
-.abx-header .abx-app-links .abx-platform-brave { color: #e8552b; }
-.abx-header .abx-app-links .abx-platform-edge { color: #168b95; }
-.abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-platform:focus-visible, .abx-header .abx-app-links .abx-tool:hover, .abx-header .abx-app-links .abx-tool:focus-visible { transform: translateY(-2px); border-color: #c99bb0; background: #fff; box-shadow: 0 5px 10px #2d15251c; }
-.abx-header .abx-app-links .abx-tool { padding: 5px 8px; font-size: 11px; white-space: nowrap; }
-.abx-header .abx-tools-row { margin-top: 5px; padding-top: 9px; border-top: 1px solid #e9e2e3; border-radius: 0 0 10px 10px; }
-.abx-header .abx-tools-row .abx-app-name { flex: 0 0 auto; }
-.abx-header .abx-tools-row .abx-platforms { flex: 0 1 auto; flex-wrap: wrap; justify-content: flex-end; margin-left: auto; }
-.abx-header .abx-app-links code { display: inline; vertical-align: baseline; font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 1em; font-weight: inherit; line-height: inherit; margin: 0; padding: 0; border: 0; background: transparent; color: inherit; }
-.abx-header .abx-icon-stack { isolation: isolate; gap: 0; padding: 4px 5px; }
-.abx-header .abx-icon-stack .abx-integration { position: relative; z-index: 1; transform: translateY(var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); transition: transform .18s ease, box-shadow .18s ease, border-color .18s ease; }
-.abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -10px; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(2) { --stack-angle: 9deg; --stack-y: -2px; z-index: 2; color: #569a31; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(3) { --stack-angle: -6deg; --stack-y: 2px; z-index: 3; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(4) { --stack-angle: 7deg; --stack-y: -1px; z-index: 4; color: #f00; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(5) { --stack-angle: -10deg; --stack-y: 1px; z-index: 5; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(6) { --stack-angle: 6deg; --stack-y: -2px; z-index: 6; }
-.abx-header .abx-icon-stack .abx-integration:nth-child(7) { --stack-angle: -5deg; z-index: 7; }
-.abx-header .abx-icon-stack .abx-integration:not(:hover, :focus-visible):has(~ .abx-integration:is(:hover, :focus-visible)) { transform: translate(-6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
-.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) ~ .abx-integration:not(:hover, :focus-visible) { transform: translate(6px, var(--stack-y, 0px)) rotate(var(--stack-angle, -8deg)); }
-.abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { z-index: 20; transform: translateY(-7px) rotate(0deg) scale(1.12); border-color: #c99bb0; box-shadow: 0 8px 14px #2d15252b; }
-@media (hover: none) { .abx-header .abx-icon-stack .abx-integration + .abx-integration { margin-left: -6px; } }
-@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration { transition: none; } }
-.abx-footer { position: relative; z-index: 1; display: block; width: 100%; max-width: none; margin: 72px 0 0; padding: 60px max(24px, calc((100% - 1280px) / 2)) 24px; border: 0; border-top: 3px solid #7e284a; background: radial-gradient(ellipse at 12% 0, #54243b55, transparent 48%), linear-gradient(120deg, #201923, #14151b 75%); color: #ddd2dd; color-scheme: dark; }
-.abx-footer .abx-footer-intro { display: flex; align-items: flex-start; justify-content: space-between; gap: 32px; margin: 0 0 42px; }
-.abx-footer .abx-brand { color: #fff4ed; font-size: 24px; }
-.abx-footer .abx-brand .abx-logo { width: 40px; height: 40px; }
-.abx-footer .abx-tagline { max-width: 430px; margin: 12px 0 0; color: #b8acbb; font-size: 14px; line-height: 1.7; }
-.abx-footer .abx-footer-motto { margin: 2px 0 0; color: #d8b39f; font: italic 25px/1.45 Georgia, serif; text-align: right; }
-.abx-footer .abx-footer-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 42px; margin: 0; }
-.abx-footer .abx-footer-column { margin: 0; padding: 20px 0 0; border: 0; border-top: 1px solid #67505e66; background: none; }
-.abx-footer .abx-footer-column h2 { margin: 0 0 19px; padding: 0; border: 0; color: #e4acbb; font: 650 12px/1.4 var(--abx-font-sans); letter-spacing: .12em; text-transform: uppercase; }
-.abx-footer .abx-footer-column ul { display: block; list-style: none; margin: 0; padding: 0; }
-.abx-footer .abx-footer-column li { display: flex; align-items: center; gap: 10px; margin: 0; padding: 0; min-height: 36px; }
-.abx-footer .abx-footer-column li::before { content: none; }
-.abx-footer .abx-footer-column a { color: #ddd2dd; font-size: 13px; line-height: 1.55; }
-.abx-footer .abx-footer-column a:first-child { padding: 6px 0; }
-.abx-footer a:hover { color: #fff; text-decoration: underline; }
-.abx-footer .abx-source { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 28px; width: 28px; height: 28px; border: 1px solid transparent; border-radius: 50%; color: #a58d9f; }
-.abx-footer .abx-source:hover { color: #f3b5cb; border-color: #80516a; background: #ffffff08; }
-.abx-source svg { width: 15px; height: 15px; fill: currentColor; }
-.abx-footer .abx-footer-bottom { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 12px 24px; margin-top: 32px; padding-top: 21px; border-top: 1px solid #67505e66; color: #a99bab; font-size: 12px; }
-.abx-footer .abx-footer-bottom p { margin: 0; font-size: inherit; color: inherit; }
-.abx-footer .abx-footer-bottom a { color: #cebbc9; }
-.abx-footer .abx-footer-local { margin: 24px 0 0; padding: 16px 0 0; border: 0; border-top: 1px solid #67505e66; border-radius: 0; background: none; color: #baacbc; font-size: 12px; }
-.abx-footer-local > summary { cursor: pointer; color: #d6bfce; }
-.abx-footer-local > div { padding-top: 18px; }
-.abx-footer-local a { color: #e5aabd; }
-.abx-footer-local .brand, .abx-footer-local .footer-intro, .abx-footer-local .footer-grid, .abx-footer-local .footer-meta { color: inherit; }
-.abx-footer-local .footer-card { background: transparent; border-color: #67505e66; }
-.abx-footer-local h2, .abx-footer-local h3, .abx-footer-local strong { color: #e4d8e3; }
-.abx-footer-local .footer-link-copy span, .abx-footer-local .footer-card-subtitle { color: #baacbc; }
-.abx-header a:focus-visible, .abx-header summary:focus-visible, .abx-footer a:focus-visible, .abx-footer summary:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
-@media (max-width: 1100px) {
-  .abx-header { flex-wrap: wrap; gap: 8px 24px; padding: 12px 20px; }
-  .abx-header .abx-nav { justify-content: flex-start; }
-  .abx-header .abx-app-links { position: fixed; top: auto; left: 20px; right: auto; width: 440px; margin-top: 10px; }
-}
-@media (max-width: 700px) {
-  .abx-header { position: relative; padding: 14px 18px; }
-  .abx-header .abx-brand { flex-wrap: wrap; row-gap: 5px; }
-  .abx-header .abx-nav { width: 100%; gap: 4px 0; font-size: 12px; }
-  .abx-nav > a, .abx-nav > details { margin-left: 10px; padding-left: 10px; }
-  .abx-header .abx-nav .abx-cta { margin-left: 10px; padding: 7px 13px; }
-  .abx-header .abx-app-links { left: 12px; right: 12px; width: auto; max-height: 60vh; }
-  .abx-footer { padding: 40px 22px 22px; margin-top: 48px; }
-  .abx-footer .abx-footer-intro { display: block; margin-bottom: 30px; }
-  .abx-footer .abx-footer-motto { margin-top: 20px; text-align: left; font-size: 22px; }
-  .abx-footer .abx-footer-grid { grid-template-columns: 1fr; gap: 26px; }
-  .abx-footer .abx-footer-column li { min-height: 40px; }
-}
-@media (prefers-reduced-motion: reduce) { .abx-apps > summary::before, .abx-header .abx-app-row, .abx-header .abx-app-links .abx-platform, .abx-header .abx-app-links .abx-tool { transition: none; } .abx-header .abx-app-links .abx-platform:hover, .abx-header .abx-app-links .abx-tool:hover { transform: none; } }
-
-.abx-footer-local > div > a + a { margin-left: 12px; }
-
-@media (prefers-reduced-motion: reduce) { .abx-header .abx-icon-stack .abx-integration:is(:hover, :focus-visible) { transform: none; } }

.github/pages/base/chrome.js

@@ -1,28 +0,0 @@
-/*
-Shared ArchiveBox website navigation styles and behavior.
-Keep this file in sync with .github/pages/base/chrome.js in:
-ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
-electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
-Each site builds its committed copy independently; there is no central generator.
-Keep site-specific navigation, CTAs, and extra footer links outside this base.
-Also review the inline navbar/footer in abx-dl/website/index.html and
- debian-archivebox/website/index.html when shared links or branding change.
-*/
-(() => {
-  const menu = document.querySelector('.abx-apps');
-  document.addEventListener('keydown', event => {
-    if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
-  });
-  document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
-  function revealAnchor() {
-    let target;
-    try { target = document.getElementById(decodeURIComponent(window.location.hash.slice(1))); } catch { return; }
-    if (!target) return;
-    let details = target.closest('details');
-    let opened = false;
-    while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
-    if (opened) window.requestAnimationFrame(() => target.scrollIntoView());
-  }
-  window.addEventListener('hashchange', revealAnchor);
-  revealAnchor();
-})();

.github/pages/base/footer-end.html

@@ -1,12 +0,0 @@
-<!--
-Shared ArchiveBox website footer.
-Keep this file in sync with .github/pages/base/footer-end.html in:
-ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
-electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
-Each site builds its committed copy independently; there is no central generator.
-Keep site-specific navigation, CTAs, and extra footer links outside this base.
-Also review the inline navbar/footer in abx-dl/website/index.html and
- debian-archivebox/website/index.html when shared links or branding change.
--->
-  <div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
-</footer>

.github/pages/base/footer-start.html

@@ -1,13 +0,0 @@
-<!--
-Shared ArchiveBox website footer.
-Keep this file in sync with .github/pages/base/footer-start.html in:
-ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
-electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
-Each site builds its committed copy independently; there is no central generator.
-Keep site-specific navigation, CTAs, and extra footer links outside this base.
-Also review the inline navbar/footer in abx-dl/website/index.html and
- debian-archivebox/website/index.html when shared links or branding change.
--->
-<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
-  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
-  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://app.archivebox.io/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://android.archivebox.io/">ArchiveBox for Android</a><a class="abx-source" href="https://github.com/ArchiveBox/android-archivebox" aria-label="ArchiveBox for Android source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://electron.archivebox.io/">ArchiveBox Desktop</a><a class="abx-source" href="https://github.com/ArchiveBox/electron-archivebox" aria-label="ArchiveBox Desktop source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://abx-dl.archivebox.io/">One-shot abx-Line truncated

.github/pages/base/header.html

@@ -1,24 +0,0 @@
-<!--
-Shared ArchiveBox website navbar and Apps menu.
-Keep this file in sync with .github/pages/base/header.html in:
-ArchiveBox/ArchiveBox, abx-plugins, abxpkg, android-archivebox,
-electron-archivebox, ios-archivebox, and archivebox-browser-extension (all repositories are in the ArchiveBox GitHub organization).
-Each site builds its committed copy independently; there is no central generator.
-Keep site-specific navigation, CTAs, and extra footer links outside this base.
-Also review the inline navbar/footer in abx-dl/website/index.html and
- debian-archivebox/website/index.html when shared links or branding change.
--->
-<header class="abx-header">
-  <a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">__SITE_NAME__</span></a>
-  <nav class="abx-nav" aria-label="Main navigation">
-    <details class="abx-apps"><summary>Apps</summary><div class="abx-app-links">
-      <div class="abx-app-row"><a class="abx-app-name" href="https://archivebox.io/#install-compose">ArchiveBox Server</a><div class="abx-platforms"><a class="abx-platform abx-platform-docker" href="https://archivebox.io/#install-compose" aria-label="ArchiveBox Server for Docker" title="ArchiveBox Server for Docker"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 640 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M349.9 236.3l-66.1 0 0-59.4 66.1 0 0 59.4zm0-204.3l-66.1 0 0 60.7 66.1 0 0-60.7zm78.2 144.8l-66.1 0 0 59.4 66.1 0 0-59.4zM271.8 104.7l-66.1 0 0 60.1 66.1 0 0-60.1zm78.1 0l-66.1 0 0 60.1 66.1 0 0-60.1zm276.8 100c-14.4-9.7-47.6-13.2-73.1-8.4-3.3-24-16.7-44.9-41.1-63.7l-14-9.3-9.3 14c-18.4 27.8-23.4 73.6-3.7 103.8-8.7 4.7-25.8 11.1-48.4 10.7l-434.7 0c-8.7 50.8 5.8 116.8 44 162.1 37.1 43.9 92.7 66.2 165.4 66.2 157.4 0 273.9-72.5 328.4-204.2 21.4 .4 67.6 .1 91.3-45.2 1.5-2.5 6.6-13.2 8.5-17.1l-13.3-8.9zM115.6 176.8l-66 0 0 59.4 66.1 0 0-59.4-.1 0zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm78.1 0l-66.1 0 0 59.4 66.1 0 0-59.4zm-78.1-72.1l-66.1 0 0 60.1 66.1 0 0-60.1z"/></svg></a><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/#archivebox-serverapp" aria-label="ArchiveBox Server for macOS" title="ArchiveBox Server for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://electron.archivebox.io/" aria-label="ArchiveBox Server for Windows" title="ArchiveBox Server for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://archivebox.io/#install-apt" aria-label="ArchiveBox Server for Linux" title="ArchiveBox Server for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-1Line truncated
-      <div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Desktop</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Desktop for macOS" title="ArchiveBox Desktop for macOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-windows" href="https://windows.archivebox.io/" aria-label="ArchiveBox Desktop for Windows" title="ArchiveBox Desktop for Windows"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M0 93.7l183.6-25.3 0 177.4-183.6 0 0-152.1zM0 418.3l183.6 25.3 0-175.2-183.6 0 0 149.9zm203.8 28l244.2 33.7 0-211.6-244.2 0 0 177.9zm0-380.6l0 180.1 244.2 0 0-213.8-244.2 33.7z"/></svg></a><a class="abx-platform abx-platform-linux" href="https://linux.archivebox.io/" aria-label="ArchiveBox Desktop for Linux" title="ArchiveBox Desktop for Linux"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M220.9 123.3c1 .5 1.8 1.7 3 1.7 1.1 0 2.8-.4 2.9-1.5 .2-1.4-1.9-2.3-3.2-2.9-1.7-.7-3.9-1-5.5-.1-.4 .2-.8 .7-.6 1.1 .3 1.3 2.3 1.1 3.4 1.7zM199 125c1.2 0 2-1.2 3-1.7 1.1-.6 3.1-.4 3.5-1.6 .2-.4-.2-.9-.6-1.1-1.6-.9-3.8-.6-5.5 .1-1.3 .6-3.4 1.5-3.2 2.9 .1 1 1.8 1.5 2.8 1.4zM420 403.8c-3.6-4-5.3-11.6-7.2-19.7-1.8-8.1-3.9-16.8-10.5-22.4-1.3-1.1-2.6-2.1-4-2.9-1.3-.8-2.7-1.5-4.1-2 9.2-27.3 5.6-54.5-3.7-79.1-11.4-30.1-31.3-56.4-46.5-74.4-17.1-21.5-33.7-41.9-33.4-72 .5-45.9 5.1-131.2-75.8-131.3-102.4-.2-76.8 103.4-77.9 135.2-1.7 23.4-6.4 41.8-22.5 64.7-18.9 22.5-45.5 58.8-58.1 96.7-6 17.9-8.8 36.1-6.2 53.3-6.5 5.8-11.4 14.7-16.6 20.2-4.2 4.3-10.3 5.9-17 8.3s-14 6-18.5 14.5c-2.1 3.9-2.8 8.1-2.8 12.4 0 3.9 .6 7.9 1.2 11.8 1.2 8.1 2.5 15.7 .8 20.8-5.2 14.4-5.9 24.4-2.2 31.7 3.8 7.3 11.4 10.5 20.1 12.3 17.3 3.6 40.8 2.7 59.3 12.5 19.8 10.4 39.9 14.1 55.9 10.4 11.6-2.6 21.1-9.6 25.9-20.2 12.5-.1 26.3-5.4 48.3-6.6 14.9-1.2 33.6 5.3 55.1 4.1 .6 2.3 1.4 4.6 2.5 6.7l0 .1c8.3 16.7 23.8 24.3 40.3 23 16.6-1.3 34.1-11 48.3-27.9 13.6-16.4 36-23.2 50.9-32.2 7.4-4.5 13.4-10.1 13.9-18.3 .4-8.2-4.4-17.3-15.5-29.7zM223.8 87.3c9.8-22.2 34.2-21.8 44-.4 6.5 14.2 3.6 30.9-4.3 40.4-1.6-.8-5.9-2.6-12.6-4.9 1.1-1.2 3.1-2.7 3.9-4.6 4.8-11.8-.2-27-9.1-27.3-7.3-.5-13.9 10.8-11.8 23-4.1-2-9.4-3.5-13-4.4-1-6.9-.3-14.6 2.9-21.8zM183.1 75.8c10.1 0 20.8 14.2 19.1 33.5-3.5 1-7.1 2.5-10.2 4.6 1.2-8.9-3.3-20.1-9.6-19.6-8.4 .7-9.8 21.2-1.8 28.1 1 .8 1.9-.2-5.9 5.5-15.6-14.6-10.5-52.1 8.4-52.1zm-13.6 60.7c6.2-4.6 13.6-10 14.1-10.5 4.7-4.4 13.5-14.2 27.9-14.2 7.1 0 15.6 2.3 25.9 8.9 6.3 4.1 11.3 4.4 22.6 9.3 8.4 3.5 13.7 9.7 10.5 18.2-2.6 7.1-11 14.4-22.7 18.1-11.1 3.6-19.8 16-38.2 14.9-3.9-.2-7-1-9.6-2.1-8-3.5-12.2-10.4-20-15-8.6-4.8-13.2-10.4-14.7-15.3-1.4-4.9 0-9 4.2-12.3zm3.3 334c-2.7 35.1-43.9 34.4-75.3 18-29.9-15.8-68.6-6.5-76.5-21.9-2.4-4.7-2.4-12.7 2.6-26.4l0-.2c2.4-7.6 .6-16-.6-23.9-1.2-7.8-1.8-15 .9-20 3.5-6.7 8.5-9.1 14.8-11.3 10.3-3.7 11.8-3.4 19.6-9.9 5.5-5.7 9.5-12.9 14.3-18 5.1-5.5 10-8.1 17.7-6.9 8.1 1.2 15.1 6.8 21.9 16l19.6 35.6c9.5 19.9 43.1 48.4 41 68.9zm-1.4-25.9c-4.1-6.6-9.6-13.6-14.4-19.6 7.1 0 14.2-2.2 16.7-8.9 2.3-6.2 0-14.9-7.4-24.9-13.5-18.2-38.3-32.5-38.3-32.5-13.5-8.4-21.1-18.7-24.6-29.9s-3-23.3-.3-35.2c5.2-22.9 18.6-45.2 27.2-59.2 2.3-1.7 .8 3.2-8.7 20.8-8.5 16.1-24.4 53.3-2.6 82.4 .6-20.7 5.5-41.8 13.8-61.5 12-27.4 37.3-74.9 39.3-112.7 1.1 .8 4.6 3.2 6.2 4.1 4.6 2.7 8.1 6.7 12.6 10.3 12.4 10 28.5 9.2 42.4 1.2 6.2-3.5 11.2-7.5 15.9-9 9.9-3.1 17.8-8.6 22.3-15 7.7 30.4 25.7 74.3 37.2 95.7 6.1 11.4 18.3 35.5 23.6 64.6 3.3-.1 7 .4 10.9 1.4 13.8-35.7-11.7-74.2-23.3-84.9-4.7-4.6-4.9-6.6-2.6-6.5 12.6 11.2 29.2 33.7 35.2 59 2.8 11.6 3.3 23.7 .4 35.7 16.4 6.8 35.9 17.9 30.7 34.8-2.2-.1-3.2 0-4Line truncated
-      <div class="abx-app-row"><a class="abx-app-name" href="https://app.archivebox.io/">ArchiveBox Mobile</a><div class="abx-platforms"><a class="abx-platform abx-platform-apple" href="https://app.archivebox.io/" aria-label="ArchiveBox Mobile for iOS" title="ArchiveBox Mobile for iOS"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 384 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M319.1 268.7c-.2-36.7 16.4-64.4 50-84.8-18.8-26.9-47.2-41.7-84.7-44.6-35.5-2.8-74.3 20.7-88.5 20.7-15 0-49.4-19.7-76.4-19.7-55.8 .9-115.1 44.5-115.1 133.2 0 26.2 4.8 53.3 14.4 81.2 12.8 36.7 59 126.7 107.2 125.2 25.2-.6 43-17.9 75.8-17.9 31.8 0 48.3 17.9 76.4 17.9 48.6-.7 90.4-82.5 102.6-119.3-65.2-30.7-61.7-90-61.7-91.9zM262.5 104.5c27.3-32.4 24.8-61.9 24-72.5-24.1 1.4-52 16.4-67.9 34.9-17.5 19.8-27.8 44.3-25.6 71.9 26.1 2 49.9-11.4 69.5-34.3z"/></svg></a><a class="abx-platform abx-platform-android" href="https://android.archivebox.io/" aria-label="ArchiveBox Mobile for Android" title="ArchiveBox Mobile for Android"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 448 512"><!-- Font Awesome Free 5.0.13 by Fonticons, Inc. https://fontawesome.com License: https://creativecommons.org/licenses/by/4.0/ --><path d="M89.6 204.5v115.8c0 15.4-12.1 27.7-27.5 27.7-15.3 0-30.1-12.4-30.1-27.7V204.5c0-15.1 14.8-27.5 30.1-27.5 15.1 0 27.5 12.4 27.5 27.5zm10.8 157c0 16.4 13.2 29.6 29.6 29.6h19.9l.3 61.1c0 36.9 55.2 36.6 55.2 0v-61.1h37.2v61.1c0 36.7 55.5 36.8 55.5 0v-61.1h20.2c16.2 0 29.4-13.2 29.4-29.6V182.1H100.4v179.4zm248-189.1H99.3c0-42.8 25.6-80 63.6-99.4l-19.1-35.3c-2.8-4.9 4.3-8 6.7-3.8l19.4 35.6c34.9-15.5 75-14.7 108.3 0L297.5 34c2.5-4.3 9.5-1.1 6.7 3.8L285.1 73c37.7 19.4 63.3 56.6 63.3 99.4zm-170.7-55.5c0-5.7-4.6-10.5-10.5-10.5-5.7 0-10.2 4.8-10.2 10.5s4.6 10.5 10.2 10.5c5.9 0 10.5-4.8 10.5-10.5zm113.4 0c0-5.7-4.6-10.5-10.2-10.5-5.9 0-10.5 4.8-10.5 10.5s4.6 10.5 10.5 10.5c5.6 0 10.2-4.8 10.2-10.5zm94.8 60.1c-15.1 0-27.5 12.1-27.5 27.5v115.8c0 15.4 12.4 27.7 27.5 27.7 15.4 0 30.1-12.4 30.1-27.7V204.5c0-15.4-14.8-27.5-30.1-27.5z"/></svg></a></div></div>
-      <div class="abx-app-row"><a class="abx-app-name" href="https://extension.archivebox.io/">ArchiveBox Browser Extension</a><div class="abx-platforms"><a class="abx-platform abx-platform-chrome" href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj" aria-label="ArchiveBox Browser Extension for Chrome" title="ArchiveBox Browser Extension for Chrome"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 6.7.2 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2024 Fonticons, Inc. --><path d="M0 256C0 209.4 12.47 165.6 34.27 127.1L144.1 318.3C166 357.5 207.9 384 256 384C270.3 384 283.1 381.7 296.8 377.4L220.5 509.6C95.9 492.3 0 385.3 0 256zM365.1 321.6C377.4 302.4 384 279.1 384 256C384 217.8 367.2 183.5 340.7 160H493.4C505.4 189.6 512 222.1 512 256C512 397.4 397.4 511.1 256 512L365.1 321.6zM477.8 128H256C193.1 128 142.3 172.1 130.5 230.7L54.19 98.47C101 38.53 174 0 256 0C350.8 0 433.5 51.48 477.8 128V128zM168 256C168 207.4 207.4 168 256 168C304.6 168 344 207.4 344 256C344 304.6 304.6 344 256 344C207.4 344 168 304.6 168 256z"/></svg></a><a class="abx-platform abx-platform-safari" href="https://extension.archivebox.io/" aria-label="ArchiveBox Browser Extension for Safari" title="ArchiveBox Browser Extension for Safari"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512 512"><!--! Font Awesome Free 7.3.1 by @fontawesome - https://fontawesome.com License - https://fontawesome.com/license/free (Icons: CC BY 4.0, Fonts: SIL OFL 1.1, Code: MIT License) Copyright 2026 Fonticons, Inc. --><path fill="currentColor" d="M274.7 274.7L237.3 237.3 166 346 274.7 274.7zM256 8a248 248 0 1 0 0 496 248 248 0 1 0 0-496zM411.8 182.8l14.8-6.1c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.8 1 2.6 1.7 1.3 1.6 1.7 2.6 .6 2 .6 3.1-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.3-2.6 1.7L418 197.6c-2 .8-4.2 .8-6.1 0s-3.5-2.4-4.3-4.3-.8-4.2 0-6.1 2.4-3.5 4.3-4.3zM314.4 94l6.1-14.8c.8-2 2.4-3.5 4.3-4.3s4.2-.8 6.1 0 3.5 2.4 4.3 4.3 .8 4.2 0 6.1l-6.1 14.8c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0c-1-.4-1.9-1-2.6-1.7s-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1zM256 60c2.1 0 4.2 .8 5.7 2.3S264 65.9 264 68l0 16c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7S253.9 60 256 60zM181 74.9c1-.4 2-.6 3.1-.6s2.1 .2 3.1 .6 1.9 1 2.6 1.7 1.3 1.6 1.7 2.6L197.6 94c.4 1 .7 2 .7 3.1s-.2 2.1-.6 3.1-1 1.9-1.7 2.6-1.6 1.4-2.6 1.8-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.8-1.3-1.7-1.7-2.7l-6.1-14.8c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3zm-63.6 42.5c1.5-1.5 3.5-2.3 5.7-2.3s4.2 .8 5.7 2.3L140 128.7c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3l-11.3-11.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6l0 0zM60 256c0-2.1 .8-4.2 2.3-5.7S65.9 248 68 248l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3l-16 0c-2.1 0-4.2-.8-5.7-2.3S60 258.1 60 256zm40.2 73.2l-14.8 6.1c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1 1-1.9 1.7-2.6 1.6-1.3 2.6-1.7L94 314.4c2-.8 4.2-.8 6.1 0s3.5 2.4 4.3 4.3 .8 4.2 0 6.1-2.4 3.5-4.3 4.3zm4.3-136c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1 2.4-3.5 4.3-4.3 4.2-.8 6.1 0l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1l0 0zM197.6 418l-6.1 14.8c-.4 1-1 1.9-1.7 2.6s-1.6 1.3-2.6 1.7-2 .6-3.1 .6-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6-.6-2-.6-3.1 .2-2.1 .6-3.1l6.1-14.8c.4-1 1-1.9 1.7-2.7s1.6-1.4 2.6-1.8 2-.6 3.1-.6 2.1 .2 3.1 .6 1.9 1 2.6 1.8 1.3 1.7 1.7 2.6 .6 2 .6 3.1-.2 2.1-.7 3.1zM264 444c0 2.1-.8 4.2-2.3 5.7s-3.5 2.3-5.7 2.3-4.2-.8-5.7-2.3-2.3-3.5-2.3-5.7l0-16c0-2.1 .8-4.2 2.3-5.7s3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3 2.3 3.5 2.3 5.7l0 16zm67-6.9c-1 .4-2 .6-3.1 .6s-2.1-.2-3.1-.6-1.9-1-2.6-1.7-1.3-1.6-1.7-2.6L314.4 418c-.8-2-.8-4.2 0-6.1s2.4-3.5 4.3-4.3 4.2-.8 6.1 0 3.5 2.4 4.3 4.3l6.1 14.8c.8 2 .8 4.2 0 6.1s-2.4 3.5-4.3 4.3zm63.6-42.5c-1.5 1.5-3.5 2.3-5.7 2.3s-4.2-.8-5.7-2.3L372 383.3c-1.5-1.5-2.3-3.5-2.3-5.6s.8-4.1 2.3-5.6 3.5-2.3 5.7-2.3 4.2 .8 5.7 2.3l11.3 11.3c1.5 1.5 2.3 3.5 2.3 5.6s-.8 4.1-2.3 5.6l0 0zM286.2 286.2L110.3 401.7 225.8 225.8 401.7 110.3 286.2 286.2zM437.1 331c-.8 2-2.4 3.5-4.3 4.3s-4.2 .8-6.1 0l-14.8-6.1c-2-.8-3.5-2.4-4.3-4.3s-.8-4.2 0-6.1c.4-1 1-1.9 1.7-2.6s1.6-1.3 2.6-1.7 2-.6 3.1-.6 2.1 .2 3.1 .6l14.8 6.1c2 .8 3.5 2.4 4.3 4.3s.8 4.2 0 6.1zm6.9-67l-16 0c-2.1 0-4.2-.8-5.7-2.3s-2.3-3.5-2.3-5.7 .8-4.2 2.3-5.7 3.5-2.3 5.7-2.3l16 0c2.1 0 4.2 .8 5.7 2.3s2.3 3.5 2.3 5.7-.8 4.2-2.3 5.7-3.5 2.3-5.7 2.3z"/></svg></a><a class="abx-platform abx-platform-firefox" href="https://addons.mozilla.org/firefox/addon/archivebox-exporter/" aria-label="ArchiveBox Browser Extension for Firefox" title="ArchiveBox Browser Extension for Firefox"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 512Line truncated
-      <div class="abx-app-row abx-plugin-row"><a class="abx-app-name" href="https://plugins.archivebox.io/">ArchiveBox Plugin Library</a><div class="abx-platforms abx-icon-stack"><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Google Drive storage" title="Google Drive storage"><svg aria-hidden="true" focusable="false" xmlns="http://www.w3.org/2000/svg" xml:space="preserve" viewBox="0 0 512 512"><path d="m38.7 419.3 22.6 39c4.7 8.2 11.4 14.7 19.4 19.4l80.6-139.6H0c0 9.1 2.3 18.2 7 26.4z" style="fill:#0066da"/><path d="M256 173.9 175.4 34.3c-7.9 4.7-14.7 11.1-19.4 19.4L7 311.7c-4.6 8-7 17.1-7 26.4h161.3z" style="fill:#00ac47"/><path d="M431.4 477.7c7.9-4.7 14.7-11.1 19.4-19.4l9.4-16.1 44.9-77.7c4.7-8.2 7-17.3 7-26.4H350.7l34.3 67.4z" style="fill:#ea4335"/><path d="m256 173.9 80.6-139.6c-7.9-4.7-17-7-26.4-7H201.8c-9.4 0-18.5 2.6-26.4 7z" style="fill:#00832d"/><path d="M350.7 338.1H161.3L80.6 477.7c7.9 4.7 17 7 26.4 7h298c9.4 0 18.5-2.6 26.4-7z" style="fill:#2684fc"/><path d="M430.5 182.7 356 53.7c-4.7-8.2-11.4-14.7-19.4-19.4L256 173.9l94.7 164.2h161c0-9.1-2.3-18.2-7-26.4z" style="fill:#ffba00"/></svg></a><a class="abx-platform abx-integration" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#amazon-s3--backblaze-b2--google-drive--etc-rclone" aria-label="Amazon S3 storage" title="Amazon S3 storage"><svg aria-hidden="true" focusable="false" fill="#569A31" viewBox="0 0 24 24" xmlns="http://www.w3.org/2000/svg"><path d="M20.913 13.147l.12-.895c.947.576 1.258.922 1.354 1.071-.16.031-.562.046-1.474-.176zm-2.174 7.988a.547.547 0 0 0-.005.073c0 .084-.207.405-1.124.768a10.28 10.28 0 0 1-1.438.432c-1.405.325-3.128.504-4.853.504-4.612 0-7.412-1.184-7.412-1.704a.547.547 0 0 0-.005-.073L1.81 5.602c.135.078.28.154.432.227.042.02.086.038.128.057.134.062.272.122.417.18l.179.069c.154.058.314.114.478.168.043.013.084.029.13.043.207.065.423.127.646.187l.176.044c.175.044.353.087.534.127a23.414 23.414 0 0 0 .843.17l.121.023c.252.045.508.085.768.122.071.011.144.02.216.03.2.027.4.053.604.077l.24.027c.245.026.49.05.74.07l.081.009c.275.022.552.04.83.056l.233.012c.21.01.422.018.633.025a33.088 33.088 0 0 0 2.795-.026l.232-.011c.278-.016.555-.034.83-.056l.08-.008c.25-.02.497-.045.742-.072l.238-.026c.205-.024.408-.05.609-.077.07-.01.141-.019.211-.03.261-.037.519-.078.772-.122l.111-.02c.215-.04.427-.082.634-.125l.212-.047c.186-.041.368-.085.546-.13l.166-.042c.225-.06.444-.122.654-.189.04-.012.077-.026.115-.038a10.6 10.6 0 0 0 .493-.173c.058-.021.114-.044.17-.066.15-.06.293-.12.43-.185.038-.017.079-.034.116-.052.153-.073.3-.15.436-.228l-.976 7.245c-2.488-.78-5.805-2.292-7.311-3a1.09 1.09 0 0 0-1.088-1.085c-.6 0-1.088.489-1.088 1.088 0 .6.488 1.089 1.088 1.089.196 0 .378-.056.537-.148 1.72.812 5.144 2.367 7.715 3.15zm-7.42-20.047c5.677 0 9.676 1.759 9.75 2.736l-.014.113c-.01.033-.031.067-.048.101-.015.028-.026.057-.047.087-.024.033-.058.068-.09.102-.028.03-.051.06-.084.09-.038.035-.087.07-.133.105-.04.03-.074.06-.119.091-.053.036-.116.071-.177.107-.05.03-.095.06-.15.09-.068.036-.147.073-.222.11-.059.028-.114.057-.177.085-.084.038-.177.074-.268.111-.068.027-.13.054-.203.082-.097.036-.205.072-.31.107-.075.026-.148.053-.228.079-.111.035-.233.069-.35.103-.085.024-.165.05-.253.073-.124.034-.258.065-.389.098-.093.022-.181.046-.278.068-.139.032-.287.061-.433.091-.098.02-.191.041-.293.06-.155.03-.32.057-.482.084-.1.018-.198.036-.302.052-.166.026-.342.048-.515.072-.11.014-.213.03-.325.044-.181.023-.372.041-.56.06-.11.012-.218.025-.332.036-.188.016-.386.029-.58.043-.122.009-.24.02-.364.028-.207.012-.422.02-.635.028-.12.005-.234.012-.354.016a35.605 35.605 0 0 1-2.069 0c-.12-.004-.234-.011-.352-.016-.214-.008-.43-.016-.637-.028-.122-.008-.238-.02-.36-.027-.195-.015-.394-.028-.584-.044-.11-.01-.215-.024-.324-.035-.19-.02-.384-.038-.568-.06l-.315-.044c-.176-.024-.355-.046-.525-.073-.1-.015-.192-.033-.29-.05-.167-.028-.335-.055-.494-.086-.096-.018-.183-.038-.276-.056-.151-.032-.305-.062-.45-.095-.09-.02-.173-.043-.26-.064-.138-.034-.277-.067-.407-.102-.082-.022-.157-.046-.235-.069a11.75 11.75 0 0 1-.368-.108c-.075-.024-.141-.049-.213-.073-.11-.037-.223-.075-.325-.113-.067-.025-.125-.051-.188-.077-.096-.038-.195-.076-.282-.115-.06-.027-.11-.054-.166-.08-.08-.039-.162-.077-.233-.116-.052-.028-.094-.055-.142-.084-.063-.038-.13-.075-.185-.113-.043-.029-.075-.058-.113-.086-.048-.037-.098-.073-.139-.11-.032-.029-.054-.057-.08-.087-.033-.035-.069-.07-.093-.104-.02-.03-.031-.058-.046-.086-.018-.035-.039-.068-.049-.102l-.015-.113c.076-.977 4.074-2.736 9.748-2.736zm12.182 12.124c-.118-.628-.84-1.291-2.31-2.128l.963-7.16a.531.531 0 0 0 .005-.073C22.16 1.581 16.447 0 11.32 0 6.194 0 .482 1.581.482 3.851a.58.58 0 0 0 .005.072L2.819 21.25c.071 2.002 5.236 2.75 8.5 2.75 1.805 0 3.615-.188 5.098-.531.598-.138 1.133-.3 1.592-.48 1.18-.467 1.789-1.053 1.813-1.739l.945-7.018c.557.131 1.016.197 1.389.197.54 0 .902-.137 1.134-.413aLine truncated
-      <div class="abx-app-row abx-tools-row"><a class="abx-app-name" href="https://abx-dl.archivebox.io/">Tools</a><div class="abx-platforms"><a class="abx-tool" href="https://abx-dl.archivebox.io/"><span>One-shot <code>abx-dl</code> CLI</span></a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/tree/dev/archivebox/mcp">MCP</a><a class="abx-tool" href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-api-usage">API</a><a class="abx-tool" href="https://web.archivebox.io">Demo</a></div></div>
-    </div></details>
-    __SITE_NAV__
-  </nav>
-</header>

.github/pages/base/marquee.css

@@ -1,11 +0,0 @@
-/* Identical standalone gallery strip across ArchiveBox product sites. */
-/* The root query container measures the usable width, excluding scrollbars. */
-html { container-type: inline-size; }
-.abx-marquee { min-width: 0; width: 100cqw; max-width: none; margin: 24px calc(50% - 50cqw) 40px; padding: 0; overflow: hidden; }
-.abx-marquee .abx-marquee-viewport { width: 100%; overflow-x: auto; scroll-behavior: auto; overscroll-behavior-x: contain; scrollbar-width: thin; padding: 16px 24px 28px; box-sizing: border-box; }
-.abx-marquee .abx-marquee-track { display: flex; align-items: flex-start; gap: 24px; width: max-content; margin-inline: auto; }
-.abx-marquee .abx-marquee-card { display: block; flex: none; width: min-content; overflow: hidden; border-radius: 10px; background: #fff; box-shadow: 0 0 0 1px rgb(0 0 0 / 12%), 0 2px 4px rgb(0 0 0 / 10%), 0 8px 16px rgb(0 0 0 / 16%); text-decoration: none; color: #533344; }
-.abx-marquee .abx-marquee-card img { display: block; height: clamp(240px, 30vw, 400px); width: auto; max-width: none; max-height: none; object-fit: contain; box-shadow: none; border: 0; border-radius: 0; background: transparent; padding: 0; margin: 0; }
-.abx-marquee :focus-visible { outline: 2px solid #9b2854; outline-offset: 3px; }
-.abx-marquee .abx-marquee-viewport:focus-visible { outline-offset: -3px; }
-@media (max-width: 600px) { .abx-marquee .abx-marquee-track { gap: 16px; } .abx-marquee .abx-marquee-card img { height: 240px; } }

.github/pages/base/marquee.js

@@ -1,44 +0,0 @@
-/* Scroll real gallery links once; keep them usable without JavaScript. */
-(() => {
-  document.querySelectorAll('.abx-marquee').forEach(gallery => {
-    const viewport = gallery.querySelector('.abx-marquee-viewport');
-    const reducedMotion = window.matchMedia('(prefers-reduced-motion: reduce)');
-    let paused = reducedMotion.matches;
-    let hovering = false;
-    let focused = false;
-    let visible = false;
-    let frame;
-    let previous;
-    let offset = viewport.scrollLeft;
-    function tick(time) {
-      const distance = viewport.scrollWidth - viewport.clientWidth;
-      if (distance <= 0) return;
-      if (previous !== undefined) {
-        offset = Math.min(offset + Math.min(time - previous, 50) * 0.035, distance);
-        viewport.scrollTo({ left: offset, behavior: 'instant' });
-      }
-      if (offset >= distance - 1) { paused = true; return; }
-      previous = time;
-      frame = window.requestAnimationFrame(tick);
-    }
-    function sync() {
-      window.cancelAnimationFrame(frame);
-      previous = undefined;
-      offset = viewport.scrollLeft;
-      if (!paused && !hovering && !focused && visible && !document.hidden) frame = window.requestAnimationFrame(tick);
-    }
-    viewport.addEventListener('pointerenter', event => { if (event.pointerType === 'mouse') { hovering = true; sync(); } });
-    viewport.addEventListener('pointerleave', () => { hovering = false; sync(); });
-    viewport.addEventListener('focusin', () => { focused = true; sync(); });
-    viewport.addEventListener('focusout', event => { focused = viewport.contains(event.relatedTarget); sync(); });
-    function pauseForInteraction() { paused = true; sync(); }
-    viewport.addEventListener('pointerdown', pauseForInteraction);
-    viewport.addEventListener('wheel', pauseForInteraction, { passive: true });
-    viewport.addEventListener('keydown', pauseForInteraction);
-    reducedMotion.addEventListener('change', () => { paused = reducedMotion.matches; sync(); });
-    document.addEventListener('visibilitychange', sync);
-    new window.IntersectionObserver(([entry]) => { visible = entry.isIntersecting; sync(); }).observe(gallery);
-    new window.ResizeObserver(sync).observe(viewport);
-    viewport.querySelectorAll('img').forEach(image => image.addEventListener('load', sync));
-  });
-})();

.github/pages/marquee.py

@@ -1,96 +0,0 @@
-"""Render homepage screenshot strips from the site's existing gallery HTML."""
-
-import re
-from html import escape
-from html.parser import HTMLParser
-from urllib.parse import urljoin
-
-
-class Gallery(HTMLParser):
-    """Only gallery figures are captures; navigation and decorative images are not."""
-
-    def __init__(self, base):
-        super().__init__()
-        self.base = base
-        self.stack = []
-        self.captures = []
-
-    def handle_starttag(self, tag, attrs):
-        attrs = dict(attrs)
-        if tag == "img" and any(name == "figure" for name, _ in self.stack):
-            context = [item for _, item in self.stack]
-            profile = next(
-                (
-                    item.get("data-profile") or item.get("data-viewport")
-                    for item in reversed(context)
-                    if item.get("data-profile") or item.get("data-viewport")
-                ),
-                None,
-            )
-            if profile not in (None, "desktop"):
-                return
-            source = attrs.get("src", "")
-            if not source or not attrs.get("alt"):
-                return
-            anchor = next((item["id"] for name, item in reversed(self.stack) if name in ("article", "figure") and item.get("id")), "")
-            server = any(item.get("data-platform") == "server" for item in context)
-            self.captures.append(
-                {
-                    "src": urljoin(self.base, source),
-                    "href": self.base + "#" + anchor if anchor else urljoin(self.base, source),
-                    "title": attrs["alt"],
-                    "width": attrs.get("width", ""),
-                    "height": attrs.get("height", ""),
-                    "product": "server" if server else "client",
-                },
-            )
-        if tag not in {"area", "base", "br", "col", "embed", "hr", "img", "input", "link", "meta", "param", "source", "track", "wbr"}:
-            self.stack.append((tag, attrs))
-
-    def handle_endtag(self, tag):
-        for index in range(len(self.stack) - 1, -1, -1):
-            if self.stack[index][0] == tag:
-                del self.stack[index:]
-                break
-
-
-def render(output, baseurl="", featured=()):
-    homepage = output / "index.html"
-    text = homepage.read_text()
-    slots = re.findall(r"<!-- ARCHIVEBOX:(MARQUEE(?:-CLIENT|-SERVER)?) -->", text)
-    if not slots:
-        return False
-    base = "/" + baseurl.strip("/") + "/" if baseurl.strip("/") else "/"
-    gallery = Gallery(base + "screenshots/")
-    gallery.feed((output / "screenshots/index.html").read_text())
-    for slot in slots:
-        product = slot.removeprefix("MARQUEE-").lower() if "-" in slot else None
-        captures = [capture for capture in gallery.captures if product is None or capture["product"] == product]
-        priority = {title: index for index, title in enumerate(featured)}
-        captures.sort(key=lambda capture: priority.get(capture["title"], len(priority)))
-        cards = []
-        for capture in captures:
-            dimensions = "".join(f' {key}="{escape(capture[key])}"' for key in ("width", "height") if capture[key])
-            cards.append(
-                f'<a class="abx-marquee-card" href="{escape(capture["href"])}"><img src="{escape(capture["src"])}" alt="{escape(capture["title"])}"{dimensions} loading="lazy" decoding="async"></a>',
-            )
-        viewport_id = slot.lower()
-        content = (
-            f'<section class="abx-marquee" aria-label="App screenshots"><div class="abx-marquee-viewport" id="{viewport_id}" tabindex="0" aria-label="Screenshots. Scroll sideways to explore."><div class="abx-marquee-track">{"".join(cards)}</div></div></section>'
-            if cards
-            else ""
-        )
-        start, end = f"<!-- ARCHIVEBOX:{slot} -->", f"<!-- /ARCHIVEBOX:{slot} -->"
-        replacement = start + "\n" + content + "\n" + end
-        if end in text:
-            text = re.sub(
-                re.escape(start) + r".*?" + re.escape(end),
-                lambda _, replacement=replacement: replacement,
-                text,
-                count=1,
-                flags=re.DOTALL,
-            )
-        else:
-            text = text.replace(start, replacement, 1)
-    homepage.write_text(text)
-    return True

.github/pages/nav.html

@@ -1,4 +0,0 @@
-<!-- Repository-specific navigation and primary CTA. -->
-<a href="__SITE_BASE__#use-cases">Use cases</a><a href="__SITE_BASE__#configuration">Configuration</a><a href="__SITE_BASE__#layout-security">Layout &amp; security</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
-    <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
-    <a class="abx-cta" href="__SITE_BASE__#install">Get started</a>

.github/pages/site.json

@@ -1,16 +0,0 @@
-{
-  "name": "Web Archive",
-  "url": "https://archivebox.io/",
-  "pages": [
-    "index.html",
-    "screenshots/index.html"
-  ],
-  "marquee_featured": [
-    "Snapshots grid — desktop",
-    "Snapshot View (header collapsed) — desktop",
-    "AI agent — desktop",
-    "Archive results — desktop",
-    "Add URLs — desktop",
-    "Snapshots table — desktop"
-  ]
-}

.github/pages/site.py

@@ -1,215 +0,0 @@
-"""Assemble and verify this repository's static Pages site (standard library only)."""
-
-import argparse
-import json
-import re
-import shutil
-import subprocess
-from datetime import UTC, datetime
-from html import escape, unescape
-from html.parser import HTMLParser
-from pathlib import Path
-from urllib.parse import unquote, urlsplit
-
-from marquee import render as render_marquee
-
-HERE = Path(__file__).resolve().parent
-ROOT = HERE.parents[1]
-
-
-def replace_block(text, name, content):
-    start, end = f"<!-- ARCHIVEBOX:{name} -->", f"<!-- /ARCHIVEBOX:{name} -->"
-    pattern = re.escape(start) + r".*?" + re.escape(end)
-    if re.search(pattern, text, re.DOTALL):
-        return re.sub(
-            pattern,
-            lambda _: f"{start}\n{content}\n{end}",
-            text,
-            count=1,
-            flags=re.DOTALL,
-        )
-    if text.count(start) != 1:
-        raise ValueError(f"Expected one {name} placeholder")
-    return text.replace(start, f"{start}\n{content}\n{end}", 1)
-
-
-def render_metadata(text):
-    """Keep product copy in native templates; render one consistent SEO block."""
-    page = Page()
-    page.feed(text)
-    title = unescape(re.search(r"<title>(.*?)</title>", text, re.DOTALL)[1])
-    meta = page.metadata
-    description = meta.get("description", meta.get("og:description", ""))
-    image = meta.get("og:image", meta.get("twitter:image", ""))
-    if not description or not image or not page.canonical:
-        raise ValueError("Each page needs a description, social image and canonical")
-    values = {
-        "description": description,
-        "robots": "index,follow,max-image-preview:large",
-        "theme-color": "#9b2854",
-        "og:type": "website",
-        "og:site_name": meta.get("og:site_name", "ArchiveBox"),
-        "og:locale": meta.get("og:locale", "en_US"),
-        "og:url": page.canonical,
-        "og:title": title,
-        "og:description": description,
-        "og:image": image,
-        "og:image:type": meta.get("og:image:type", "image/png"),
-        "og:image:width": meta.get("og:image:width", "1200"),
-        "og:image:height": meta.get("og:image:height", "630"),
-        "og:image:alt": meta.get("og:image:alt", title),
-        "twitter:card": "summary_large_image",
-        "twitter:site": "@ArchiveBoxApp",
-        "twitter:title": title,
-        "twitter:description": description,
-        "twitter:image": image,
-        "twitter:image:alt": meta.get("og:image:alt", title),
-    }
-
-    def strip_managed_tag(match):
-        tag = Page()
-        tag.feed(match[0])
-        if tag.canonicals or any(key in values for key in tag.metadata):
-            return ""
-        # System fonts are shared across sites, without a network font dependency.
-        if "fonts.googleapis.com" in match[0] or "fonts.gstatic.com" in match[0]:
-            return ""
-        return match[0]
-
-    text = re.sub(r"<(?:meta|link)\b[^>]*>", strip_managed_tag, text)
-    text = re.sub(r"<title>.*?</title>", "", text, flags=re.DOTALL)
-    tags = [
-        f"<title>{escape(title)}</title>",
-        f'<link rel="canonical" href="{escape(page.canonical, quote=True)}">',
-    ]
-    for key, value in values.items():
-        attr = "property" if key.startswith("og:") else "name"
-        tags.append(f'<meta {attr}="{key}" content="{escape(value, quote=True)}">')
-    if "<!-- ARCHIVEBOX:METADATA -->" not in text:
-        text = text.replace("</head>", "<!-- ARCHIVEBOX:METADATA -->\n</head>", 1)
-    return replace_block(text, "METADATA", "\n".join(tags))
-
-
-def render(output, source=None, baseurl=""):
-    config = json.loads((HERE / "site.json").read_text())
-    output = output.resolve()
-    if source:
-        source = source.resolve()
-        if output == source or output in source.parents or source in output.parents:
-            raise ValueError("Build output must be separate from the source directory")
-        if output == ROOT or output in ROOT.parents:
-            raise ValueError("Build output must not replace a repository")
-        if output.exists():
-            shutil.rmtree(output)
-        shutil.copytree(source, output)
-    revision = subprocess.check_output(["git", "rev-parse", "HEAD"], cwd=ROOT, text=True).strip()
-    assets = output / "site-base"
-    assets.mkdir(parents=True, exist_ok=True)
-    has_marquee = render_marquee(output, baseurl, config.get("marquee_featured", []))
-    common_assets = ["chrome.css", "chrome.js"] + (["marquee.css", "marquee.js"] if has_marquee else [])
-    for name in common_assets:
-        shutil.copyfile(HERE / "base" / name, assets / name)
-    for name in config["pages"]:
-        page = output / name
-        text = render_metadata(page.read_text())
-        base = "../" * (len(Path(name).parts) - 1) or "./"
-        header = (HERE / "base/header.html").read_text().replace("__SITE_NAME__", escape(config["name"]))
-        header = header.replace("__SITE_NAV__", (HERE / "nav.html").read_text().strip())
-        text = replace_block(text, "HEADER", header)
-        for kind in ["FOOTER-START", "FOOTER-END"]:
-            text = replace_block(text, kind, (HERE / "base" / f"{kind.lower()}.html").read_text().strip())
-        text = text.replace("__SITE_BASE__", base).replace("__SITE_REVISION__", revision)
-        resources = f'<link rel="stylesheet" href="{base}site-base/chrome.css?v={revision}">\n<script src="{base}site-base/chrome.js?v={revision}" defer></script>'
-        if has_marquee and name == "index.html":
-            resources += f'\n<link rel="stylesheet" href="{base}site-base/marquee.css?v={revision}">\n<script src="{base}site-base/marquee.js?v={revision}" defer></script>'
-        if "<!-- ARCHIVEBOX:ASSETS -->" not in text:
-            text = text.replace("</head>", "<!-- ARCHIVEBOX:ASSETS -->\n</head>", 1)
-        text = replace_block(text, "ASSETS", resources)
-        page.write_text(text)
-    # Site build provenance is independent of all original screenshot manifests.
-    provenance_path = output / "build.json"
-    provenance = json.loads(provenance_path.read_text()) if provenance_path.exists() else {}
-    provenance.update(
-        revision=revision,
-        generatedAt=datetime.now(UTC).isoformat(),
-        site=config["url"],
-    )
-    provenance_path.write_text(json.dumps(provenance, indent=2) + "\n")
-    (output / ".nojekyll").touch()
-    verify(output, baseurl)
-    print(f"Assembled {len(config['pages'])} pages at {output} from {revision}")
-
-
-class Page(HTMLParser):
-    def __init__(self):
-        super().__init__()
-        self.ids = set()
-        self.metadata = {}
-        self.canonical = ""
-        self.resources = []
-        self.headers = self.footers = self.canonicals = 0
-
-    def handle_starttag(self, tag, attrs):
-        attrs = dict(attrs)
-        if tag == "meta":
-            for key in ["name", "property"]:
-                if attrs.get(key):
-                    self.metadata[attrs[key]] = attrs.get("content", "")
-        if tag == "link" and attrs.get("rel") == "canonical":
-            self.canonical = attrs["href"]
-        if attrs.get("id"):
-            self.ids.add(attrs["id"])
-        classes = attrs.get("class", "").split()
-        self.headers += "abx-header" in classes
-        self.footers += "abx-footer" in classes
-        self.canonicals += tag == "link" and attrs.get("rel") == "canonical"
-        if tag in ["img", "script"] and attrs.get("src"):
-            self.resources.append(attrs["src"])
-        if tag == "link" and attrs.get("rel") in [
-            "stylesheet",
-            "icon",
-            "apple-touch-icon",
-        ]:
-            self.resources.append(attrs["href"])
-
-
-def verify(output, baseurl=""):
-    config = json.loads((HERE / "site.json").read_text())
-    output = output.resolve()
-    prefix = "/" + baseurl.strip("/") if baseurl.strip("/") else ""
-    for name in config["pages"]:
-        path = output / name
-        text = path.read_text()
-        page = Page()
-        page.feed(text)
-        if (page.headers, page.footers, page.canonicals) != (1, 1, 1):
-            raise ValueError(f"{name}: expected exactly one shared header, footer and canonical")
-        if "__SITE_" in text:
-            raise ValueError(f"{name}: unresolved site placeholder")
-        for resource in page.resources:
-            url = urlsplit(resource)
-            if url.scheme or url.netloc or not url.path:
-                continue
-            resource_path = unquote(url.path)
-            if prefix and resource_path.startswith(prefix + "/"):
-                resource_path = resource_path[len(prefix) :]
-            local = (output / resource_path.lstrip("/") if resource_path.startswith("/") else path.parent / resource_path).resolve()
-            if not local.is_relative_to(output) or not local.is_file():
-                raise ValueError(f"{name}: missing local resource {resource}")
-        for asset in ["chrome.css", "chrome.js"]:
-            if (output / "site-base" / asset).read_bytes() != (HERE / "base" / asset).read_bytes():
-                raise ValueError(f"{name}: stale common asset {asset}")
-    print(f"Verified shared layout and local resources for {len(config['pages'])} pages")
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("command", choices=["render", "verify"])
-    parser.add_argument("output", type=Path)
-    parser.add_argument("--source", type=Path)
-    parser.add_argument("--baseurl", default="")
-    args = parser.parse_args()
-    if args.command == "render":
-        render(args.output, args.source, args.baseurl)
-    else:
-        verify(args.output, args.baseurl)

.github/pages/verify.py

@@ -1,256 +0,0 @@
-"""Check the built site in real Chromium at desktop and phone widths."""
-
-import argparse
-import json
-import os
-import re
-from functools import partial
-from http.server import SimpleHTTPRequestHandler, ThreadingHTTPServer
-from html.parser import HTMLParser
-from pathlib import Path
-from threading import Thread
-from urllib.parse import unquote, urljoin, urlsplit
-
-from playwright.sync_api import expect, sync_playwright
-
-HERE = Path(__file__).resolve().parent
-
-
-class Handler(SimpleHTTPRequestHandler):
-    def handle(self):
-        # Browser navigation can cancel an in-flight lazy image response.
-        try:
-            super().handle()
-        except (BrokenPipeError, ConnectionResetError):
-            pass
-
-    def log_message(self, *args):
-        pass
-
-
-class ScreenshotReferences(HTMLParser):
-    def __init__(self):
-        super().__init__()
-        self.references = []
-        self.ids = set()
-
-    def handle_starttag(self, tag, attributes):
-        attrs = dict(attributes)
-        if attrs.get("id"):
-            self.ids.add(attrs["id"])
-        for attribute in ("src", "href"):
-            if attrs.get(attribute):
-                self.references.append(attrs[attribute])
-
-
-def verify_screenshot_references(output, canonical):
-    """Check gallery embeds on every static page, including optional translations."""
-    pages = {}
-    for path in output.rglob("*.html"):
-        page = ScreenshotReferences()
-        page.feed(path.read_text())
-        pages[path.resolve()] = page
-    for path, page in pages.items():
-        page_url = urljoin(canonical, path.relative_to(output.resolve()).as_posix())
-        for reference in page.references:
-            url = urlsplit(urljoin(page_url, reference))
-            if url.netloc != urlsplit(canonical).netloc or "/screenshots/" not in url.path:
-                continue
-            target = output / unquote(url.path).lstrip("/")
-            if target.is_dir():
-                target /= "index.html"
-            target = target.resolve()
-            assert target.is_relative_to(output.resolve()) and target.is_file(), f"{path}: missing screenshot target {reference}"
-            assert not re.match(r"^\d+-", target.name), f"{path}: screenshot filename depends on capture order: {reference}"
-            if url.fragment and target in pages:
-                assert unquote(url.fragment) in pages[target].ids, f"{path}: missing screenshot anchor {reference}"
-
-
-def verify(output, evidence):
-    config = json.loads((HERE / "site.json").read_text())
-    verify_screenshot_references(output, config["url"])
-    evidence.mkdir(parents=True, exist_ok=True)
-    server = ThreadingHTTPServer(("127.0.0.1", 0), partial(Handler, directory=str(output.resolve())))
-    thread = Thread(target=server.serve_forever, daemon=True)
-    thread.start()
-    origin = f"http://127.0.0.1:{server.server_port}"
-    try:
-        with sync_playwright() as playwright:
-            browser = playwright.chromium.launch(executable_path=os.environ.get("CHROME_BINARY"), ignore_default_args=["--hide-scrollbars"])
-            for name in config["pages"]:
-                route = name.removesuffix("index.html")
-                page = browser.new_page(reduced_motion="reduce")
-                errors = []
-                missing = []
-                page.on("pageerror", lambda error, errors=errors: errors.append(str(error)))
-                page.on(
-                    "response",
-                    lambda response, missing=missing: (
-                        missing.append(response.url) if response.url.startswith(origin) and response.status >= 400 else None
-                    ),
-                )
-                for width in [1716, 2560, 390]:
-                    page.set_viewport_size({"width": width, "height": 900})
-                    page.goto(f"{origin}/{route}", wait_until="domcontentloaded")
-                    header = page.locator(".abx-header")
-                    expect(header).to_have_count(1)
-                    expect(header).to_be_visible()
-                    expect(header).to_be_in_viewport()
-                    brand = header.locator(".abx-brand")
-                    expect(brand).to_be_visible()
-                    expect(brand).to_contain_text("ArchiveBox")
-                    expect(brand.locator(".abx-logo")).to_be_visible()
-                    expect(brand.locator(".abx-logo")).to_be_in_viewport()
-                    expect(header).to_have_css("display", "flex")
-                    expect(page.locator(".abx-footer")).to_have_count(1)
-                    expect(page.locator(".abx-footer-column")).to_have_count(3)
-                    expect(header.locator(".abx-cta")).to_be_visible()
-                    menu = header.locator(".abx-apps")
-                    menu.locator("summary").focus()
-                    page.keyboard.press("Enter")
-                    expect(menu).to_have_attribute("open", "")
-                    expect(menu.locator("a").first).to_be_visible()
-                    page.keyboard.press("Escape")
-                    expect(menu).not_to_have_attribute("open", "")
-                    menu.locator("summary").click()
-                    box = header.bounding_box()
-                    assert box
-                    page.mouse.click(box["x"] + 2, box["y"] + 5)
-                    expect(menu).not_to_have_attribute("open", "")
-                    assert page.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth"), (
-                        f"{name}: overflow at {width}"
-                    )
-                    if not route and "screenshots/index.html" in config["pages"]:
-                        strips = page.locator(".abx-marquee")
-                        assert strips.count() > 0, "Homepage must use its screenshot gallery"
-                        for strip in strips.all():
-                            strip.scroll_into_view_if_needed()
-                            bounds = strip.bounding_box()
-                            usable_width = page.evaluate("document.documentElement.clientWidth")
-                            assert bounds and abs(bounds["x"]) < 1
-                            assert abs(bounds["width"] - usable_width) < 1
-                            assert not strip.inner_text().strip(), "Screenshot strips should have no visible captions or links"
-                            cards = strip.locator(".abx-marquee-card")
-                            assert cards.count() > 0, "Screenshot strip is empty"
-                            expect(strip.locator("button")).to_have_count(0)
-                            expect(strip.locator("a:not(.abx-marquee-card)")).to_have_count(0)
-                            viewport = strip.locator(".abx-marquee-viewport")
-                            overflowing = viewport.evaluate(
-                                "node => node.scrollWidth > node.clientWidth",
-                            )
-                            if overflowing:
-                                page.mouse.move(0, 0)
-                                page.emulate_media(reduced_motion="no-preference")
-                                initial = viewport.evaluate("node => node.scrollLeft")
-                                page.wait_for_function(
-                                    "([node, initial]) => node.scrollLeft > initial",
-                                    arg=[viewport.element_handle(), initial],
-                                )
-                                viewport.hover()
-                                stopped = viewport.evaluate("node => node.scrollLeft")
-                                viewport.evaluate(
-                                    "node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
-                                )
-                                assert viewport.evaluate("node => node.scrollLeft") == stopped
-                                page.emulate_media(reduced_motion="reduce")
-                                page.mouse.move(0, 0)
-                                stopped = viewport.evaluate("node => node.scrollLeft")
-                                viewport.evaluate(
-                                    "node => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
-                                )
-                                assert viewport.evaluate("node => node.scrollLeft") == stopped
-                                viewport.focus()
-                                page.keyboard.press("ArrowRight")
-                                assert viewport.evaluate("node => node.scrollWidth > node.clientWidth")
-                            else:
-                                centered = viewport.evaluate(
-                                    """node => {
-                                        const view = node.getBoundingClientRect();
-                                        const track = node.querySelector('.abx-marquee-track').getBoundingClientRect();
-                                        return Math.abs((track.left - view.left) - (view.right - track.right)) < 2;
-                                    }""",
-                                )
-                                assert centered, "Short screenshot strips must be centered"
-                            for card in cards.all():
-                                href = card.get_attribute("href")
-                                assert href and "/screenshots/" in href
-                                response = page.request.get(urljoin(page.url, href))
-                                assert response.ok, f"Broken screenshot link: {href}"
-                                anchor = urlsplit(href).fragment
-                                if anchor:
-                                    assert f'id="{anchor}"' in response.text(), f"Missing screenshot anchor: {href}"
-                    # Decode every local content image using the real browser loader,
-                    # including images in offscreen carousel tracks and lazy galleries.
-                    images = page.locator("main img").evaluate_all("nodes => [...new Set(nodes.map(image => image.src))]")
-                    local_images = [url for url in images if url.startswith(origin + "/")]
-                    page.evaluate(
-                        """async urls => {
-                        for (const url of urls) {
-                            const image = new Image();
-                            image.src = url;
-                            await image.decode();
-                            if (!image.naturalWidth) throw new Error(`Broken image: ${url}`);
-                        }
-                    }""",
-                        local_images,
-                    )
-                    page.evaluate("scrollTo({top: 0, behavior: 'instant'})")
-                    page.evaluate(
-                        "() => new Promise(resolve => requestAnimationFrame(() => requestAnimationFrame(resolve)))",
-                    )
-                    page.screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}{width}.png"))
-                    page.locator(".abx-footer").screenshot(path=str(evidence / f"{route.replace('/', '-') or 'home-'}footer-{width}.png"))
-                    for link in header.locator(".abx-nav > a").all():
-                        url = urlsplit(link.get_attribute("href") or "")
-                        if url.scheme or url.netloc:
-                            continue
-                        response = page.request.get(urljoin(page.url, url.geturl()))
-                        assert response.ok, f"Broken navigation: {url.geturl()}"
-                        if url.fragment:
-                            body = response.text()
-                            assert f'id="{url.fragment}"' in body or f"id='{url.fragment}'" in body, (
-                                f"Missing navigation anchor: {url.geturl()}"
-                            )
-                    if route == "" and page.locator("#resources").count():
-                        page.goto(f"{origin}/#resources")
-                        expect(page.locator("#resources")).to_be_visible()
-                assert not errors, errors
-                assert not missing, missing
-                page.close()
-                context = browser.new_context(java_script_enabled=False, viewport={"width": 390, "height": 900})
-                plain = context.new_page()
-                plain.goto(f"{origin}/{route}", wait_until="domcontentloaded")
-                expect(plain.locator(".abx-header")).to_be_visible()
-                expect(plain.locator(".abx-header .abx-brand")).to_be_in_viewport()
-                expect(plain.locator(".abx-header .abx-logo")).to_be_visible()
-                plain.locator(".abx-apps summary").click()
-                expect(plain.locator(".abx-app-links a").first).to_be_visible()
-                expect(plain.locator(".abx-footer-column a").first).to_be_visible()
-                if not route and "screenshots/index.html" in config["pages"]:
-                    assert plain.locator(".abx-marquee-card").count() > 0
-                    for strip in plain.locator(".abx-marquee").all():
-                        bounds = strip.bounding_box()
-                        usable_width = plain.evaluate("document.documentElement.clientWidth")
-                        assert bounds and abs(bounds["x"]) < 1
-                        assert abs(bounds["width"] - usable_width) < 1
-                    assert plain.evaluate("document.documentElement.scrollWidth <= document.documentElement.clientWidth")
-                    expect(plain.locator(".abx-marquee button")).to_have_count(0)
-                    expect(plain.locator(".abx-marquee-card").first).to_be_visible()
-                context.close()
-                print(
-                    f"PASS {name}: desktop/mobile, keyboard, no-JS, images and resources",
-                    flush=True,
-                )
-            browser.close()
-    finally:
-        server.shutdown()
-        server.server_close()
-        thread.join()
-
-
-if __name__ == "__main__":
-    parser = argparse.ArgumentParser(description=__doc__)
-    parser.add_argument("output", type=Path)
-    parser.add_argument("--evidence", type=Path, default=Path("test-results/site"))
-    args = parser.parse_args()
-    verify(args.output, args.evidence)

.github/scripts/discover_test_matrix.py

@@ -1,79 +1,43 @@
 #!/usr/bin/env python3
-"""Discover every test once, batching measured short files."""
+"""Discover every ArchiveBox test file and expose it as one CI job."""
 
 import json
-import os
 from pathlib import Path
 
 
 def main() -> None:
-    root = Path(__file__).resolve().parents[2]
+    root = Path.cwd().resolve()
     archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))
 
     if not archivebox_tests:
         raise SystemExit("No ArchiveBox tests discovered")
 
-    # These are observed per-file pytest seconds, including fixture/startup
-    # time. Shared job setup is paid once per batch. Unknown/long files, optional
-    # dependencies and hosted-only requirements keep independent jobs.
-    durations = json.loads((root / ".github/test-durations.json").read_text())["seconds"]
-    hosted = {path for path in archivebox_tests if "# ci-runner: hosted" in path.read_text().splitlines()[:5]}
-    requested = os.environ.get("TEST_FILE", "")
-    batches: list[list[str]] = []
-    batch: list[str] = []
-    batch_seconds = 0
+    matrix: list[dict[str, object]] = []
     for path in archivebox_tests:
         test_path = path.relative_to(root).as_posix()
-        duration = durations.get(test_path, 60)
-        if duration >= 60 or path in hosted or path.stem == "test_auth_ldap" or test_path == requested:
-            batches.append([test_path])
-            continue
-        if batch and (len(batch) == 8 or batch_seconds + duration > 180):
-            batches.append(batch)
-            batch = []
-            batch_seconds = 0
-        batch.append(test_path)
-        batch_seconds += duration
-    if batch:
-        batches.append(batch)
-
-    matrix: list[dict[str, object]] = []
-    for paths in batches:
         matrix.append(
             {
-                "name": f"main/{Path(paths[0]).stem}" if len(paths) == 1 else f"batch/{Path(paths[0]).stem}",
-                "paths": paths,
-                "paths_arg": " ".join(paths),
-                "extra": "ldap" if paths == ["archivebox/tests/test_auth_ldap.py"] else "",
-                "count": len(paths),
-                "ugnas": False,
+                "name": f"main/{path.stem}",
+                "paths": [test_path],
+                "paths_arg": test_path,
+                "extra": (
+                    "ldap"
+                    if test_path.endswith("/test_auth_ldap.py")
+                    else "allauth"
+                    if test_path.endswith("/test_allauth_integration.py")
+                    else ""
+                ),
+                "count": 1,
             },
         )
 
-    # Ordinary Linux tests can use either pool. Only genuine hosted-only
-    # requirements need a file header; filenames never enter runner policy.
-    # Keep a bounded NAS share so hosted capacity continues working in parallel.
-    # This is queued work, not a worker count: several waves keep the capped
-    # runners occupied while hosted jobs run in parallel.
-    nas_job_limit = int(os.environ.get("UGNAS_CI_MAX_JOBS", "3"))
-    # Keep long jobs early in the hosted queue. Giving the longest jobs to the
-    # CPU-capped NAS instead created a tail after hosted work had finished; use
-    # its extra slots for a bounded share of shorter work without filename rules.
-    matrix.sort(key=lambda item: sum(durations.get(path, 60) for path in item["paths"]), reverse=True)
-    eligible = [item for item in reversed(matrix) if not any(root / path in hosted for path in item["paths"])]
-    for item in eligible[:nas_job_limit]:
-        item["ugnas"] = True
-
     discovered_paths = [path for entry in matrix for path in entry["paths"]]
     if len(discovered_paths) != len(set(discovered_paths)):
         raise SystemExit("Tests were not discovered exactly once")
     if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
         raise SystemExit("Discovered test shard coverage does not match test files")
 
-    print(
-        f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs; "
-        f"{sum(item['ugnas'] for item in matrix)} selected for ugNAS, subject to the availability check",
-    )
+    print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
     print(json.dumps(matrix, separators=(",", ":")))
 
 

.github/test-durations.json

@@ -1,176 +0,0 @@
-{
-  "source_runs": [
-    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/37190995942",
-    "https://github.com/ArchiveBox/ArchiveBox/actions/runs/37237793977"
-  ],
-  "measurement": "Unbatched files use successful per-file pytest step wall seconds from run 37237793977, including process and fixture startup, on their actual hosted or ugNAS runner. Already-batched files retain prior JUnit measurements plus 9 seconds startup. Queue and shared job setup are excluded; unknown files remain independent.",
-  "seconds": {
-    "archivebox/tests/test_api_discovery_performance.py": 28,
-    "archivebox/tests/test_api_v1_auth_browser_session.py": 17,
-    "archivebox/tests/test_api_v1_auth_check_api_token.py": 15,
-    "archivebox/tests/test_api_v1_auth_get_api_token.py": 16,
-    "archivebox/tests/test_api_v1_cli_add.py": 52,
-    "archivebox/tests/test_api_v1_cli_remove.py": 33,
-    "archivebox/tests/test_api_v1_cli_schedule.py": 29,
-    "archivebox/tests/test_api_v1_cli_search.py": 15,
-    "archivebox/tests/test_api_v1_cli_update.py": 60,
-    "archivebox/tests/test_api_v1_cli_workflow_add_search_update_remove.py": 59,
-    "archivebox/tests/test_api_v1_core_any_id.py": 13,
-    "archivebox/tests/test_api_v1_core_archiveresult_archiveresult_id.py": 24,
-    "archivebox/tests/test_api_v1_core_archiveresults.py": 19,
-    "archivebox/tests/test_api_v1_core_snapshot_snapshot_id.py": 106,
-    "archivebox/tests/test_api_v1_core_snapshots.py": 17,
-    "archivebox/tests/test_api_v1_core_snapshots_rss.py": 16,
-    "archivebox/tests/test_api_v1_core_tag_tag_id.py": 15,
-    "archivebox/tests/test_api_v1_core_tag_tag_id_rename.py": 16,
-    "archivebox/tests/test_api_v1_core_tag_tag_id_snapshots_jsonl.py": 16,
-    "archivebox/tests/test_api_v1_core_tag_tag_id_urls_txt.py": 16,
-    "archivebox/tests/test_api_v1_core_tags.py": 15,
-    "archivebox/tests/test_api_v1_core_tags_add_to_snapshot.py": 16,
-    "archivebox/tests/test_api_v1_core_tags_autocomplete.py": 15,
-    "archivebox/tests/test_api_v1_core_tags_create.py": 15,
-    "archivebox/tests/test_api_v1_core_tags_remove_from_snapshot.py": 15,
-    "archivebox/tests/test_api_v1_core_tags_search.py": 17,
-    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id.py": 69,
-    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_filename.py": 15,
-    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_filename.py": 15,
-    "archivebox/tests/test_api_v1_crawls_crawl_crawl_id_files_folder_subfolder_filename.py": 16,
-    "archivebox/tests/test_api_v1_crawls_crawls.py": 16,
-    "archivebox/tests/test_api_v1_machine_binaries.py": 16,
-    "archivebox/tests/test_api_v1_machine_binary_binary_id.py": 16,
-    "archivebox/tests/test_api_v1_machine_binary_by_name_name.py": 16,
-    "archivebox/tests/test_api_v1_machine_machine_current.py": 15,
-    "archivebox/tests/test_api_v1_machine_machine_machine_id.py": 16,
-    "archivebox/tests/test_api_v1_machine_machines.py": 15,
-    "archivebox/tests/test_api_v1_personas_personas.py": 15,
-    "archivebox/tests/test_api_v1_personas_sync.py": 16,
-    "archivebox/tests/test_api_v1_workflow_core_token_auth_side_effects.py": 27,
-    "archivebox/tests/test_api_v1_workflow_frozen_crawl_config_sources.py": 19,
-    "archivebox/tests/test_archive_result_service.py": 49,
-    "archivebox/tests/test_auth_ldap.py": 37,
-    "archivebox/tests/test_binary_service.py": 46,
-    "archivebox/tests/test_cli_add_1.py": 75,
-    "archivebox/tests/test_cli_add_2.py": 98,
-    "archivebox/tests/test_cli_add_3.py": 93,
-    "archivebox/tests/test_cli_add_4.py": 94,
-    "archivebox/tests/test_cli_archiveresult.py": 48,
-    "archivebox/tests/test_cli_background_screenshot.py": 49,
-    "archivebox/tests/test_cli_binary.py": 12,
-    "archivebox/tests/test_cli_config.py": 88,
-    "archivebox/tests/test_cli_crawl.py": 81,
-    "archivebox/tests/test_cli_extract.py": 97,
-    "archivebox/tests/test_cli_extract_input.py": 84,
-    "archivebox/tests/test_cli_help.py": 16,
-    "archivebox/tests/test_cli_init.py": 111,
-    "archivebox/tests/test_cli_install.py": 38,
-    "archivebox/tests/test_cli_list.py": 116,
-    "archivebox/tests/test_cli_machine.py": 19,
-    "archivebox/tests/test_cli_manage.py": 32,
-    "archivebox/tests/test_cli_mcp.py": 30,
-    "archivebox/tests/test_cli_oneshot.py": 15,
-    "archivebox/tests/test_cli_persona.py": 56,
-    "archivebox/tests/test_cli_piping.py": 118,
-    "archivebox/tests/test_cli_pluginmap.py": 12,
-    "archivebox/tests/test_cli_process.py": 12,
-    "archivebox/tests/test_cli_remove.py": 46,
-    "archivebox/tests/test_cli_run_1.py": 93,
-    "archivebox/tests/test_cli_run_2.py": 111,
-    "archivebox/tests/test_cli_run_3.py": 51,
-    "archivebox/tests/test_cli_run_4.py": 78,
-    "archivebox/tests/test_cli_schedule.py": 59,
-    "archivebox/tests/test_cli_search.py": 25,
-    "archivebox/tests/test_cli_server.py": 126,
-    "archivebox/tests/test_cli_shell.py": 17,
-    "archivebox/tests/test_cli_snapshot.py": 55,
-    "archivebox/tests/test_cli_status.py": 46,
-    "archivebox/tests/test_cli_tag.py": 12,
-    "archivebox/tests/test_cli_update.py": 62,
-    "archivebox/tests/test_cli_update_reindex_snapshots.py": 84,
-    "archivebox/tests/test_cli_version.py": 81,
-    "archivebox/tests/test_config_DELETE_AFTER.py": 145,
-    "archivebox/tests/test_config_MAX_limits.py": 25,
-    "archivebox/tests/test_config_ONLY_NEW.py": 15,
-    "archivebox/tests/test_config_SAVE_TITLE.py": 59,
-    "archivebox/tests/test_config_URL_filters.py": 15,
-    "archivebox/tests/test_core_config.py": 11,
-    "archivebox/tests/test_core_forms.py": 14,
-    "archivebox/tests/test_crawl_history.py": 60,
-    "archivebox/tests/test_crawl_pause.py": 11,
-    "archivebox/tests/test_crawl_runner.py": 22,
-    "archivebox/tests/test_crawl_runner_2.py": 27,
-    "archivebox/tests/test_crawl_service.py": 19,
-    "archivebox/tests/test_frozen_crawl_config.py": 26,
-    "archivebox/tests/test_hook_result_lifecycle.py": 48,
-    "archivebox/tests/test_hooks.py": 54,
-    "archivebox/tests/test_issue_1896.py": 115,
-    "archivebox/tests/test_machine_models.py": 26,
-    "archivebox/tests/test_machine_service.py": 77,
-    "archivebox/tests/test_migrations_04_to_09.py": 25,
-    "archivebox/tests/test_migrations_07_to_09.py": 86,
-    "archivebox/tests/test_migrations_08_to_09.py": 85,
-    "archivebox/tests/test_migrations_fresh.py": 55,
-    "archivebox/tests/test_misc_checks.py": 12,
-    "archivebox/tests/test_misc_monkey_patches.py": 12,
-    "archivebox/tests/test_opencode_agent_1.py": 73,
-    "archivebox/tests/test_opencode_agent_2.py": 141,
-    "archivebox/tests/test_opencode_agent_3.py": 61,
-    "archivebox/tests/test_opencode_browser_1.py": 139,
-    "archivebox/tests/test_opencode_browser_2.py": 164,
-    "archivebox/tests/test_opencode_browser_3.py": 125,
-    "archivebox/tests/test_orm_helpers.py": 14,
-    "archivebox/tests/test_permissions.py": 12,
-    "archivebox/tests/test_persona_casing.py": 45,
-    "archivebox/tests/test_persona_runtime.py": 30,
-    "archivebox/tests/test_postgres_backend.py": 79,
-    "archivebox/tests/test_process_runtime_paths.py": 20,
-    "archivebox/tests/test_process_service.py": 40,
-    "archivebox/tests/test_proof_previews.py": 24,
-    "archivebox/tests/test_recursive_crawl_1.py": 361,
-    "archivebox/tests/test_recursive_crawl_2.py": 55,
-    "archivebox/tests/test_recursive_crawl_3.py": 67,
-    "archivebox/tests/test_release_notes.py": 12,
-    "archivebox/tests/test_release_source_fingerprint.py": 13,
-    "archivebox/tests/test_release_workflow.py": 15,
-    "archivebox/tests/test_resource_admission.py": 13,
-    "archivebox/tests/test_runner_deletion.py": 91,
-    "archivebox/tests/test_runner_watch.py": 75,
-    "archivebox/tests/test_search.py": 142,
-    "archivebox/tests/test_serve_static.py": 19,
-    "archivebox/tests/test_server_automatic_url.py": 22,
-    "archivebox/tests/test_server_security_browser.py": 121,
-    "archivebox/tests/test_settings_signal_webhooks.py": 13,
-    "archivebox/tests/test_setup_script.py": 12,
-    "archivebox/tests/test_shutdown_util.py": 13,
-    "archivebox/tests/test_snapshot_filesystem_migration.py": 16,
-    "archivebox/tests/test_snapshot_output_stacks.py": 63,
-    "archivebox/tests/test_snapshot_ref_lookup.py": 15,
-    "archivebox/tests/test_snapshot_service.py": 48,
-    "archivebox/tests/test_snapshot_url_length.py": 16,
-    "archivebox/tests/test_tag_service.py": 15,
-    "archivebox/tests/test_takeover_util_1.py": 60,
-    "archivebox/tests/test_takeover_util_2.py": 62,
-    "archivebox/tests/test_takeover_util_3.py": 76,
-    "archivebox/tests/test_takeover_util_4.py": 84,
-    "archivebox/tests/test_test_harness.py": 12,
-    "archivebox/tests/test_ui_add_view.py": 26,
-    "archivebox/tests/test_ui_add_view_runtime.py": 120,
-    "archivebox/tests/test_ui_admin_apitoken.py": 16,
-    "archivebox/tests/test_ui_admin_archiveresult.py": 18,
-    "archivebox/tests/test_ui_admin_config_widget.py": 13,
-    "archivebox/tests/test_ui_admin_crawl.py": 25,
-    "archivebox/tests/test_ui_admin_links.py": 23,
-    "archivebox/tests/test_ui_admin_machine.py": 18,
-    "archivebox/tests/test_ui_admin_persona.py": 19,
-    "archivebox/tests/test_ui_admin_snapshot.py": 138,
-    "archivebox/tests/test_ui_admin_tag.py": 16,
-    "archivebox/tests/test_ui_admin_user.py": 16,
-    "archivebox/tests/test_ui_config_views.py": 20,
-    "archivebox/tests/test_ui_live_progress.py": 121,
-    "archivebox/tests/test_ui_public_snapshot.py": 72,
-    "archivebox/tests/test_ui_savepagenow.py": 31,
-    "archivebox/tests/test_urls.py": 105,
-    "archivebox/tests/test_util.py": 13,
-    "archivebox/tests/test_version_metadata.py": 18,
-    "archivebox/tests/test_zip_preview.py": 29
-  }
-}

.github/workflows/ci.yml

@@ -18,64 +18,51 @@ concurrency:
   cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
 
 jobs:
-  prepare:
-    permissions:
-      contents: write
-    uses: ./.github/workflows/prepare-release.yml
-    secrets: inherit
-
   lint:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: Linters
     uses: ./.github/workflows/lint.yml
     secrets: inherit
 
   install-cli-platform:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: Install and CLI platform compatibility
     uses: ./.github/workflows/test.yml
     secrets: inherit
 
   tests:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: Discovered test matrix
     uses: ./.github/workflows/test-parallel.yml
     secrets: inherit
 
+  documentation:
+    name: Documentation and root tests
+    uses: ./.github/workflows/docs.yml
+    secrets: inherit
+
   codeql:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: CodeQL
     uses: ./.github/workflows/codeql.yml
     secrets: inherit
 
   python-artifacts:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: Tested Python artifacts
     uses: ./.github/workflows/pip.yml
     secrets: inherit
 
   docker-digests:
-    needs: prepare
-    if: needs.prepare.outputs.run_jobs == 'true'
     name: Tested Docker images
     uses: ./.github/workflows/docker.yml
     with:
-      push_digests: ${{ github.event_name == 'push' && (github.ref_name == 'dev' || github.ref_name == 'main') }}
+      push_digests: false
     secrets: inherit
 
   required:
     name: All required CI lanes
-    if: ${{ always() && needs.prepare.outputs.run_jobs == 'true' }}
+    if: ${{ always() }}
     needs:
-      - prepare
       - lint
       - install-cli-platform
       - tests
+      - documentation
       - codeql
       - python-artifacts
       - docker-digests
@@ -86,6 +73,7 @@ jobs:
           LINT_RESULT: ${{ needs.lint.result }}
           INSTALL_CLI_PLATFORM_RESULT: ${{ needs.install-cli-platform.result }}
           TESTS_RESULT: ${{ needs.tests.result }}
+          DOCUMENTATION_RESULT: ${{ needs.documentation.result }}
           CODEQL_RESULT: ${{ needs.codeql.result }}
           PYTHON_ARTIFACTS_RESULT: ${{ needs.python-artifacts.result }}
           DOCKER_DIGESTS_RESULT: ${{ needs.docker-digests.result }}
@@ -94,6 +82,7 @@ jobs:
             "$LINT_RESULT" \
             "$INSTALL_CLI_PLATFORM_RESULT" \
             "$TESTS_RESULT" \
+            "$DOCUMENTATION_RESULT" \
             "$CODEQL_RESULT" \
             "$PYTHON_ARTIFACTS_RESULT" \
             "$DOCKER_DIGESTS_RESULT"

.github/workflows/deploy-publicsite.yml

@@ -1,96 +1,79 @@
-name: Publish website
+name: Deploy Publicsite to GitHub Pages
 
 on:
   push:
-    branches: [dev]
-  pull_request:
+    branches:
+      - dev
+    paths:
+      - publicsite/**
+      - archivebox/**
+      - bin/collect_ui_screenshots.*
+      - bin/generate_ui_screenshot_gallery.py
+      - bin/take_screenshot.js
+      - docs/Screenshots.md
+      - pyproject.toml
+      - uv.lock
+      - .github/workflows/deploy-publicsite.yml
   workflow_dispatch:
-  workflow_run:
-    workflows: ['Capture publicsite screenshots']
-    types: [completed]
-    branches: [dev]
 
 permissions:
   contents: read
-  actions: read
+  pages: write
+  id-token: write
 
-env:
-  # Repository-specific output; all publication logic below is a local copy.
-  SITE_OUTPUT: _site
-  DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
-
-# One production queue covers restore, build and deploy. PRs have separate queues.
 concurrency:
-  group: pages-${{ github.event_name == 'pull_request' && github.ref || 'production' }}
-  cancel-in-progress: false
+  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
 
 jobs:
-  build:
-    runs-on: ubuntu-latest
-    timeout-minutes: 20
-    outputs:
-      revision: ${{ steps.source.outputs.revision }}
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          ref: ${{ github.event_name == 'pull_request' && github.ref || env.DEFAULT_BRANCH }}
-          persist-credentials: false
-      - name: Record website source revision
-        id: source
-        run: echo "revision=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
-
-      # BEGIN REPOSITORY-SPECIFIC BUILD
-      - uses: astral-sh/setup-uv@v7
-      - name: Restore latest complete screenshots
-        run: uv run --no-project python bin/restore_site_screenshots.py publicsite/screenshots
-        env:
-          GH_TOKEN: ${{ github.token }}
-      - name: Render current gallery layout without recapturing images
-        run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
-      - name: Assemble website with the local base components
-        run: uv run --no-project python .github/pages/site.py render _site --source publicsite
-      # END REPOSITORY-SPECIFIC BUILD
-
-      - name: Verify website in Chromium
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.tool_cache }}/abxpkg/site-browser
-        run: |
-          # Resolve the same supported browser as captures; Playwright is the
-          # driver only and must not provision a separate browser/runtime.
-          ABX_DL_VERSION="$(uv run --no-project python -c 'import tomllib; print(next(p["version"] for p in tomllib.load(open("uv.lock", "rb"))["package"] if p["name"] == "abx-dl"))')"
-          CHROME_SCHEMA="$(uv run --no-project --with "abx-dl==$ABX_DL_VERSION" python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "chrome" / "config.json")')"
-          uv run --no-project --with "abx-dl==$ABX_DL_VERSION" abxpkg env --install --json --deps-from="$CHROME_SCHEMA:required_binaries" > "$RUNNER_TEMP/browser-env.json"
-          CHROME_BINARY="$(uv run --no-project python -c 'import json, sys; print(json.load(open(sys.argv[1]))["CHROME_BINARY"])' "$RUNNER_TEMP/browser-env.json")"
-          export CHROME_BINARY
-          test -x "$CHROME_BINARY"
-          uv run --no-config --no-project --with playwright==1.63.0 python .github/pages/verify.py "$SITE_OUTPUT" --evidence "$RUNNER_TEMP/site-evidence"
-      - name: Save website verification evidence
-        if: always()
-        uses: actions/upload-artifact@v4
-        with:
-          name: site-verification
-          path: ${{ runner.temp }}/site-evidence
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Upload website
-        uses: actions/upload-pages-artifact@v3
-        with:
-          path: ${{ env.SITE_OUTPUT }}
-
   deploy:
-    if: github.event_name != 'pull_request'
-    needs: build
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      pages: write
-      id-token: write
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
+    runs-on: ubuntu-24.04
     steps:
-      - uses: actions/configure-pages@v5
-      - uses: actions/deploy-pages@v4
+      - name: Checkout
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Setup Python and uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: false
+
+      - name: Install ArchiveBox and screenshot dependencies
+        run: uv sync --no-cache --frozen
+
+      - name: Build UI screenshot gallery
+        run: bin/collect_ui_screenshots.sh
+        env:
+          BASE_URL: http://archivebox.localhost:8000
+
+      - name: Verify gallery was generated from this exact revision
+        run: |
+          uv run --no-cache --no-sync python - <<'PY'
+          import json
+          import os
+          import tomllib
+          from pathlib import Path
+
+          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
+          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
+          assert provenance["version"] == version, provenance
+          assert provenance["capture_count"] >= 3 * 35, provenance
+          assert len(provenance["files"]) == provenance["capture_count"], provenance
+          PY
+
+      - name: Setup Pages
+        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
+
+      - name: Upload artifact
+        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
+        with:
+          path: ./publicsite
+
+      - name: Deploy to GitHub Pages
         id: deployment
+        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

.github/workflows/docker.yml

@@ -13,22 +13,6 @@ on:
         required: false
         default: true
         type: boolean
-  workflow_dispatch:
-    inputs:
-      source_ref:
-        description: Exact 40-character commit SHA to build
-        required: true
-        type: string
-      push_digests:
-        description: Push validated per-platform images by digest for the release workflow
-        required: false
-        default: true
-        type: boolean
-      full_tests:
-        description: Run storage, CLI, and size validation in addition to release smoke checks
-        required: false
-        default: true
-        type: boolean
 
 env:
   DOCKERHUB_IMAGE: archivebox/archivebox
@@ -58,18 +42,6 @@ jobs:
     steps:
       - name: Checkout
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ inputs.source_ref || github.sha }}
-
-      - name: Verify exact source commit
-        if: github.event_name == 'workflow_dispatch'
-        shell: bash
-        env:
-          SOURCE_REF: ${{ inputs.source_ref || github.sha }}
-        run: |
-          set -Eeuo pipefail
-          [[ "$SOURCE_REF" =~ ^[0-9a-f]{40}$ ]]
-          [[ "$(git rev-parse HEAD)" == "$SOURCE_REF" ]]
 
       - name: Set up Python
         uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
@@ -211,7 +183,7 @@ jobs:
           {
             echo 'labels<<EOF'
             echo "org.opencontainers.image.version=${VERSION}"
-            echo "org.opencontainers.image.revision=${{ inputs.source_ref || github.sha }}"
+            echo "org.opencontainers.image.revision=${GITHUB_SHA}"
             echo 'org.opencontainers.image.source=https://github.com/ArchiveBox/ArchiveBox'
             echo "io.archivebox.abx-dl.image=${{ steps.abx_dl_image.outputs.image }}"
             echo 'EOF'
@@ -231,7 +203,7 @@ jobs:
           labels: ${{ steps.docker_meta.outputs.labels }}
           build-args: |
             ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
-            ARCHIVEBOX_COMMIT_HASH=${{ inputs.source_ref || github.sha }}
+            ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
           pull: true
           platforms: ${{ matrix.platform }}
           # Keep the released image as the one cache authority. A second GHA
@@ -260,7 +232,7 @@ jobs:
           labels: ${{ steps.docker_meta.outputs.labels }}
           build-args: |
             ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
-            ARCHIVEBOX_COMMIT_HASH=${{ inputs.source_ref || github.sha }}
+            ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
           pull: true
           platforms: ${{ matrix.platform }}
           # Test the same layer lineage users receive. Reusing :dev avoids a
@@ -274,31 +246,13 @@ jobs:
         if: inputs.push_digests
         run: echo ${{ steps.docker_build.outputs.digest }}
 
-      - name: Verify optional agent install with a read-only browser profile
-        shell: bash
-        env:
-          IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
-        run: |
-          set -Eeuo pipefail
-          PROFILE_DIR="$(mktemp -d)"
-          # shellcheck disable=SC2016
-          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
-            --volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
-            "$IMAGE" bash -c '
-              set -Eeuo pipefail
-              test -w "$XDG_CONFIG_HOME"
-              test ! -w "$XDG_CONFIG_HOME/chromium"
-              uv run --no-project python -c "from abx_plugins.plugins.opencode.image import verify_archivebox_install; verify_archivebox_install()"
-            '
-
       - name: Validate exact built image version and commit
-        if: inputs.full_tests
+        if: ${{ !inputs.push_digests }}
         shell: bash
         run: |
           set -Eeuo pipefail
-          IMAGE="${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}"
-          SHORT_SHA="${{ (inputs.source_ref || github.sha) }}"
-          SHORT_SHA="${SHORT_SHA::7}"
+          IMAGE="${{ matrix.local_tag }}"
+          SHORT_SHA="${GITHUB_SHA::7}"
 
           DATA_DIR="$(mktemp -d)"
           "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
@@ -342,12 +296,11 @@ jobs:
             MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
           fi
           COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
-          LIMIT=$((1025 * 1024 * 1024))
-          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=1025 MiB"
+          LIMIT=$((750 * 1024 * 1024))
+          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
           (( COMPRESSED_SIZE < LIMIT ))
 
       - name: Resolve real Docker CLI test dependencies through abxpkg
-        id: mount_dependencies
         if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         run: |
@@ -369,142 +322,94 @@ jobs:
             test -x "$binary"
           done
 
-      - name: Diagnose failed mount dependency installation
-        if: failure() && steps.mount_dependencies.conclusion == 'failure'
+      - name: Validate Docker UID and GID behavior on real NFS and SMB mounts
+        if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         run: |
-          sudo tail -n 120 /var/log/apt/term.log /var/log/dpkg.log
-          sudo journalctl --since '-10 minutes' --no-pager \
-            -u smbd -u nmbd -u samba-ad-dc -u nfs-server
-          ps -eo pid,ppid,stat,etimes,wchan:32,comm
+          set -Eeuo pipefail
+          NFS_SOURCE="/srv/archivebox-ci-nfs"
+          SMB_SOURCE="/srv/archivebox-ci-smb"
+          NFS_MOUNT="$RUNNER_TEMP/archivebox-nfs-mount"
+          SMB_MOUNT="$RUNNER_TEMP/archivebox-smb-mount"
+          mounted_nfs=0
+          mounted_smb=0
 
-      - name: Validate Docker mounts, ports, CLI and compose concurrently
+          cleanup() {
+            local original_status=$? cleanup_status=0
+            trap - EXIT
+            if [[ "$mounted_smb" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$SMB_MOUNT"; then
+              cleanup_status=1
+            fi
+            if [[ "$mounted_nfs" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$NFS_MOUNT"; then
+              cleanup_status=1
+            fi
+            if ! "$SUDO_BINARY" "$SYSTEMCTL_BINARY" stop smbd.service nfs-kernel-server.service; then
+              cleanup_status=1
+            fi
+            if [[ "$original_status" != "0" ]]; then
+              exit "$original_status"
+            fi
+            exit "$cleanup_status"
+          }
+          trap cleanup EXIT
+
+          "$SUDO_BINARY" "$MKDIR_BINARY" -p "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
+          "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
+          "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
+          "$SUDO_BINARY" "$BASH_BINARY" -c \
+            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+          "$SUDO_BINARY" "$BASH_BINARY" -c \
+            "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
+
+          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart nfs-kernel-server.service
+          "$SUDO_BINARY" "$EXPORTFS_BINARY" -ra
+          "$SYSTEMCTL_BINARY" is-active --quiet nfs-kernel-server.service
+          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart smbd.service
+          "$SYSTEMCTL_BINARY" is-active --quiet smbd.service
+
+          "$SUDO_BINARY" "$MOUNT_NFS_BINARY" -o vers=3,nolock "127.0.0.1:$NFS_SOURCE" "$NFS_MOUNT"
+          mounted_nfs=1
+          "$SUDO_BINARY" "$MOUNT_CIFS_BINARY" //127.0.0.1/archivebox-ci "$SMB_MOUNT" \
+            -o "guest,vers=3.0,uid=0,gid=0,forceuid,forcegid,file_mode=0777,dir_mode=0777,nounix"
+          mounted_smb=1
+
+          NFS_TEST_DIR="$NFS_MOUNT" \
+          SMB_TEST_DIR="$SMB_MOUNT" \
+          IMAGE="${{ matrix.local_tag }}" \
+            "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
+
+      - name: Validate Docker CLI add/list and compose behavior
         if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         env:
-          ARCHIVEBOX_IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
+          ARCHIVEBOX_IMAGE: ${{ matrix.local_tag }}
         run: |
           set -Eeuo pipefail
-          # These checks own separate collections, Compose projects and ports.
-          # Subshells keep mount cleanup traps and environment changes isolated.
-          validate_mounts() (
-            set -Eeuo pipefail
-            NFS_SOURCE="/srv/archivebox-ci-nfs"
-            SMB_SOURCE="/srv/archivebox-ci-smb"
-            NFS_MOUNT="$RUNNER_TEMP/archivebox-nfs-mount"
-            SMB_MOUNT="$RUNNER_TEMP/archivebox-smb-mount"
-            mounted_nfs=0
-            mounted_smb=0
+          "$MKDIR_BINARY" "${{ github.workspace }}/data"
+          "$DOCKER_BINARY" run -v "${{ github.workspace }}/data":/data "$ARCHIVEBOX_IMAGE" init
 
-            # Invoked by the subshell EXIT trap.
-            # shellcheck disable=SC2329
-            cleanup() {
-              local original_status=$? cleanup_status=0
-              trap - EXIT
-              if [[ "$mounted_smb" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$SMB_MOUNT"; then
-                cleanup_status=1
-              fi
-              if [[ "$mounted_nfs" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$NFS_MOUNT"; then
-                cleanup_status=1
-              fi
-              if ! "$SUDO_BINARY" "$SYSTEMCTL_BINARY" stop smbd.service nfs-kernel-server.service; then
-                cleanup_status=1
-              fi
-              if [[ "$original_status" != "0" ]]; then
-                exit "$original_status"
-              fi
-              exit "$cleanup_status"
-            }
-            trap cleanup EXIT
+          "$SUDO_BINARY" "$BASH_BINARY" -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
+          "$DOCKER_BINARY" run --name www-nginx -p 80:80 -d nginx:1.29.5@sha256:0236ee02dcbce00b9bd83e0f5fbc51069e7e1161bd59d99885b3ae1734f3392e
 
-            "$SUDO_BINARY" "$MKDIR_BINARY" -p "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
-            "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
-            "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
-            "$SUDO_BINARY" "$BASH_BINARY" -c \
-              "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
-            "$SUDO_BINARY" "$BASH_BINARY" -c \
-              "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
+          "$DOCKER_BINARY" run -v "$PWD"/data:/data --network host "$ARCHIVEBOX_IMAGE" add http://www.test-nginx-1.local
+          echo "http://www.test-nginx-2.local" | "$DOCKER_BINARY" run -i --network host -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" add
 
-            "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart nfs-kernel-server.service
-            "$SUDO_BINARY" "$EXPORTFS_BINARY" -ra
-            "$SYSTEMCTL_BINARY" is-active --quiet nfs-kernel-server.service
-            "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart smbd.service
-            "$SYSTEMCTL_BINARY" is-active --quiet smbd.service
+          archive_list="$("$DOCKER_BINARY" run -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" list)"
+          [[ "$archive_list" == *"www.test-nginx-1.local"* ]] || { echo "The site 1 isn't in the list"; exit 1; }
+          [[ "$archive_list" == *"www.test-nginx-2.local"* ]] || { echo "The site 2 isn't in the list"; exit 1; }
 
-            "$SUDO_BINARY" "$MOUNT_NFS_BINARY" -o vers=3,nolock "127.0.0.1:$NFS_SOURCE" "$NFS_MOUNT"
-            mounted_nfs=1
-            "$SUDO_BINARY" "$MOUNT_CIFS_BINARY" //127.0.0.1/archivebox-ci "$SMB_MOUNT" \
-              -o "guest,vers=3.0,uid=0,gid=0,forceuid,forcegid,file_mode=0777,dir_mode=0777,nounix"
-            mounted_smb=1
+          "$DOCKER_BINARY" compose run --rm archivebox init
+          "$DOCKER_BINARY" compose up -d --wait
+          "$DOCKER_BINARY" network connect archivebox_default www-nginx
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
+          admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
+          [[ "$index_html" == *ArchiveBox* ]]
+          [[ "$admin_js" == *window.django* ]]
 
-            NFS_TEST_DIR="$NFS_MOUNT" \
-            SMB_TEST_DIR="$SMB_MOUNT" \
-            IMAGE="${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}" \
-              "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
-          )
-          validate_cli() (
-            set -Eeuo pipefail
-            "$MKDIR_BINARY" "${{ github.workspace }}/data"
-            "$DOCKER_BINARY" run -v "${{ github.workspace }}/data":/data "$ARCHIVEBOX_IMAGE" init
-
-            "$SUDO_BINARY" "$BASH_BINARY" -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
-            "$DOCKER_BINARY" run --name www-nginx -p 80:80 -d nginx:1.29.5@sha256:0236ee02dcbce00b9bd83e0f5fbc51069e7e1161bd59d99885b3ae1734f3392e
-
-            "$DOCKER_BINARY" run -v "$PWD"/data:/data --network host "$ARCHIVEBOX_IMAGE" add http://www.test-nginx-1.local
-            # The argument form above exercises the complete default pipeline.
-            # Keep a real capture for stdin parsing without repeating every plugin.
-            echo "http://www.test-nginx-2.local" | "$DOCKER_BINARY" run -i --network host -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" add --plugins=wget,hashes
-
-            archive_list="$("$DOCKER_BINARY" run -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" list)"
-            [[ "$archive_list" == *"www.test-nginx-1.local"* ]] || { echo "The site 1 isn't in the list"; exit 1; }
-            [[ "$archive_list" == *"www.test-nginx-2.local"* ]] || { echo "The site 2 isn't in the list"; exit 1; }
-
-            "$DOCKER_BINARY" compose run --rm archivebox init
-            "$DOCKER_BINARY" compose up -d --wait
-            "$DOCKER_BINARY" network connect archivebox_default www-nginx
-            index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
-            admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/static/admin/js/jquery.init.js')"
-            [[ "$index_html" == *ArchiveBox* ]]
-            [[ "$admin_js" == *window.django* ]]
-
-            "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
-            index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:5797/public/')"
-            [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
-            "$DOCKER_BINARY" compose down
-          )
-          validate_ports() (
-            IMAGE="$ARCHIVEBOX_IMAGE" "$BASH_BINARY" bin/validate_docker_ports.sh
-          )
-
-          checks=(mounts ports cli)
-          pids=()
-          validate_mounts > "$RUNNER_TEMP/docker-mounts.log" 2>&1 &
-          pids+=("$!")
-          validate_ports > "$RUNNER_TEMP/docker-ports.log" 2>&1 &
-          pids+=("$!")
-          validate_cli > "$RUNNER_TEMP/docker-cli.log" 2>&1 &
-          pids+=("$!")
-          status=0
-          for index in "${!checks[@]}"; do
-            check="${checks[$index]}"
-            echo "::group::Docker $check validation"
-            if wait "${pids[$index]}"; then
-              echo "PASS: Docker $check validation"
-            else
-              echo "::error::Docker $check validation failed"
-              status=1
-            fi
-            cat "$RUNNER_TEMP/docker-$check.log"
-            echo "::endgroup::"
-          done
-          exit "$status"
-
-      - name: Validate Docker port compatibility
-        if: ${{ !(inputs.full_tests && matrix.platform == 'linux/amd64') }}
-        shell: bash
-        env:
-          IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
-        run: '"$BASH_BINARY" bin/validate_docker_ports.sh'
+          "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
+          [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
+          "$DOCKER_BINARY" compose down
 
       - name: Validate compressed image size
         if: inputs.full_tests && !inputs.push_digests
@@ -518,7 +423,7 @@ jobs:
             '${{ runner.temp }}/archivebox-image.tar' \
             '${{ steps.docker_build_pr.outputs.digest }}' \
             "$LOCAL_IMAGE_ID" \
-            "$((1025 * 1024 * 1024))" <<'PY'
+            "$((750 * 1024 * 1024))" <<'PY'
           from hashlib import sha256
           import json
           import os
@@ -592,17 +497,12 @@ jobs:
 
       - name: Export digest
         if: inputs.push_digests
-        env:
-          ABX_DL_IMAGE_REF: ${{ steps.abx_dl_image.outputs.image }}
-          ARTIFACT_NAME: ${{ matrix.artifact_name }}
         shell: bash
         run: |
           set -Eeuo pipefail
           mkdir -p /tmp/digests
           digest="${{ steps.docker_build.outputs.digest }}"
           touch "/tmp/digests/${digest#sha256:}"
-          printf '%s\n' '${{ inputs.source_ref || github.sha }}' > /tmp/digests/source-ref
-          printf '%s\n' "$ABX_DL_IMAGE_REF" > "/tmp/digests/abx-dl-image-$ARTIFACT_NAME"
 
       - name: Upload digest
         if: inputs.push_digests

.github/workflows/docs-links.yml

@@ -1,38 +0,0 @@
-name: Documentation link checks
-
-on:
-  schedule:
-    - cron: '17 10 * * *' # Daily at 10:17 UTC, away from the busy top of the hour.
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-env:
-  PYTHONIOENCODING: utf-8
-
-jobs:
-  links:
-    name: Documentation links and anchors
-    runs-on: ubuntu-24.04
-    timeout-minutes: 5
-    permissions:
-      contents: read
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-      - name: List tracked documentation and source inputs
-        run: |
-          git ls-files -- '*.md' '*.html' '*.py' '*.rst' '*.yml' '*.yaml' '*.sh' '*.toml' > "$RUNNER_TEMP/link-inputs.txt"
-      - uses: lycheeverse/lychee-action@e7477775783ea5526144ba13e8db5eec57747ce8 # v2
-        with:
-          lycheeVersion: v0.24.2
-          args: >-
-            --config .github/lychee.toml
-            --root-dir "${{ github.workspace }}"
-            --files-from "${{ runner.temp }}/link-inputs.txt"
-          fail: true
-          failIfEmpty: true

.github/workflows/docs.yml

@@ -1,17 +1,7 @@
 name: Documentation tests
 
 on:
-  schedule:
-    - cron: '7 5,17 * * *'
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  actions: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
+  workflow_call:
 
 env:
   PYTHONIOENCODING: utf-8
@@ -19,18 +9,7 @@ env:
   SHOW_PROGRESS: "False"
 
 jobs:
-  changes:
-    runs-on: ubuntu-24.04
-    outputs:
-      changed: ${{ steps.check.outputs.changed }}
-      key: ${{ steps.check.outputs.key }}
-    steps:
-      - id: check
-        uses: ArchiveBox/monorepo/.github/actions/changed-scheduled-inputs@main
-
   discover:
-    needs: changes
-    if: needs.changes.outputs.changed == 'true'
     name: Discover documentation examples
     runs-on: ubuntu-24.04
     timeout-minutes: 5
@@ -88,7 +67,7 @@ jobs:
     timeout-minutes: 15
     strategy:
       fail-fast: false
-      max-parallel: 8
+      max-parallel: 4
       matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -120,10 +99,9 @@ jobs:
           if [[ '${{ matrix.environment }}' != macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries")
           fi
-          if [[ '${{ matrix.name }}' == ubuntu-AGENTS || '${{ matrix.name }}' == ubuntu-skills-archivebox-SKILL || '${{ matrix.name }}' == ubuntu-docs-Configuration || '${{ matrix.name }}' == ubuntu-docs-Setting-up-Authentication ]]; then
+          if [[ '${{ matrix.environment }}' == ubuntu ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
-          fi
-          if [[ '${{ matrix.environment }}' == docker ]]; then
+          elif [[ '${{ matrix.environment }}' == docker ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries")
           elif [[ '${{ matrix.environment }}' == macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries")
@@ -167,9 +145,7 @@ jobs:
           fi
       - name: Build the exact checkout's documentation image
         if: ${{ matrix.environment == 'docker' }}
-        run: |
-          ABX_DL_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(dep.removeprefix("abx-dl==") for dep in tomllib.load(open("pyproject.toml", "rb"))["project"]["dependencies"] if dep.startswith("abx-dl==")))')"
-          "$DOCKER_BINARY" build --build-arg "ABX_DL_IMAGE=archivebox/abx-dl:${ABX_DL_VERSION}" --tag archivebox/archivebox:dev .
+        run: '"$DOCKER_BINARY" build --tag archivebox/archivebox:dev .'
       - name: Run the Ubuntu, Docker, or macOS documentation lane
         if: ${{ !contains(fromJSON('["root", "freebsd", "openbsd"]'), matrix.environment) }}
         run: uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
@@ -179,17 +155,3 @@ jobs:
           "$SUDO_BINARY" \
             --preserve-env=PATH,ABXPKG_LIB_DIR,BASH_BINARY,PYTHON_BINARY,SUDO_BINARY,UV_BINARY \
             "$UV_BINARY" run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
-      - name: Diagnose failed Linux package installation
-        if: failure() && runner.os == 'Linux'
-        run: |
-          sudo tail -n 120 /var/log/apt/term.log /var/log/dpkg.log
-          apt-config dump Acquire::Retries Acquire::http::Timeout Acquire::https::Timeout
-          ps -eo pid,ppid,stat,etimes,wchan:32,comm
-
-  complete:
-    name: 'Scheduled inputs: ${{ needs.changes.outputs.key }}'
-    needs: [changes, discover, examples]
-    if: needs.changes.outputs.changed == 'true'
-    runs-on: ubuntu-24.04
-    steps:
-      - run: echo 'Scheduled work completed successfully.'

.github/workflows/duplicate-issue-detection.yml

@@ -0,0 +1,96 @@
+name: Duplicate Issue Detection
+
+on:
+  issues:
+    types: [opened]
+
+jobs:
+  check-duplicates:
+    runs-on: ubuntu-24.04
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+
+      - name: Resolve OpenCode and workflow tools through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
+        run: |
+          set -Eeuo pipefail
+          uv sync --no-cache --locked --no-sources --no-install-project
+          OPENCODE_CONFIG="$(uv run --no-cache --no-sync --no-sources python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "opencode" / "config.json")')"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
+          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
+          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
+          opencode_env="$(
+            uv run --no-cache --no-sync --no-sources abxpkg env \
+              --install \
+              --json \
+              --lib="$ABXPKG_LIB_DIR" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:github_binaries" \
+              --deps-from="$OPENCODE_CONFIG:required_binaries"
+          )"
+          jq_binary="$ABXPKG_LIB_DIR/env/bin/jq"
+          "$jq_binary" -r 'to_entries[] | "\(.key)=\(.value)"' \
+            <<< "$opencode_env" >> "$GITHUB_ENV"
+
+          opencode_binary="$ABXPKG_LIB_DIR/env/bin/opencode"
+          echo "OPENCODE_BINARY=$opencode_binary" >> "$GITHUB_ENV"
+          test -L "$opencode_binary"
+          test -x "$opencode_binary"
+
+      - name: Check for duplicate issues
+        env:
+          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
+          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+          OPENCODE_PERMISSION: |
+            {
+              "bash": {
+                "gh issue*": "allow",
+                "gh pr*": "allow",
+                "*": "deny"
+              },
+              "webfetch": "allow"
+            }
+        run: |
+          "$OPENCODE_BINARY" run -m anthropic/claude-haiku-4-5 "A new issue has been created:
+
+          Issue number:
+          ${{ github.event.issue.number }}
+
+          Lookup this issue and search through existing issues and PRs (excluding #${{ github.event.issue.number }}) in this repository to find any potential duplicates of this new issue.
+          Consider:
+          1. Similar titles or descriptions
+          2. Same error messages or symptoms
+          3. Related functionality or components
+          4. Similar feature requests
+
+          If you find any potential duplicates, please comment on the new issue with:
+          - A brief explanation of why it might be a duplicate
+          - Links to the potentially duplicate issues or PRs
+          - A suggestion to check those issues first
+
+          Use this format for the comment:
+          'This issue might be a duplicate of existing issues. Please check:
+          - #[issue_number]: [brief description of similarity]
+
+          Feel free to ignore if none of these address your specific case.'
+
+          If no clear duplicates are found, do not comment."

.github/workflows/macos.yml

@@ -1,48 +0,0 @@
-name: macOS compatibility
-
-on:
-  schedule:
-    - cron: '51 2,10,18 * * *'
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  actions: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
-jobs:
-  changes:
-    runs-on: ubuntu-24.04
-    outputs:
-      changed: ${{ steps.check.outputs.changed }}
-      key: ${{ steps.check.outputs.key }}
-    steps:
-      - id: check
-        uses: ArchiveBox/monorepo/.github/actions/changed-scheduled-inputs@main
-
-  install-cli:
-    needs: changes
-    if: needs.changes.outputs.changed == 'true'
-    uses: ./.github/workflows/test.yml
-    with:
-      platform: macos
-    secrets: inherit
-
-  python-artifacts:
-    needs: changes
-    if: needs.changes.outputs.changed == 'true'
-    uses: ./.github/workflows/pip.yml
-    with:
-      platforms: '["macos-15"]'
-    secrets: inherit
-
-  complete:
-    name: 'Scheduled inputs: ${{ needs.changes.outputs.key }}'
-    needs: [changes, install-cli, python-artifacts]
-    if: needs.changes.outputs.changed == 'true'
-    runs-on: ubuntu-24.04
-    steps:
-      - run: echo 'Scheduled macOS coverage completed successfully.'

.github/workflows/pip.yml

@@ -3,11 +3,6 @@ name: Build Pip package
 on:
   workflow_call:
     inputs:
-      platforms:
-        description: Runner matrix for package compatibility
-        required: false
-        default: '["ubuntu-24.04"]'
-        type: string
       full_tests:
         description: Run the full package compatibility matrix instead of one release smoke check
         required: false
@@ -100,7 +95,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        os: ${{ fromJSON(inputs.platforms) }}
+        os: [ubuntu-24.04, macos-15]
         python: ["3.13", "3.14"]
         artifact: [wheel, sdist]
 

.github/workflows/prepare-release.yml

@@ -1,55 +0,0 @@
-name: Prepare release version
-
-on:
-  workflow_call:
-    outputs:
-      run_jobs:
-        description: Whether this revision owns its release version and should run expensive jobs
-        value: ${{ jobs.prepare.outputs.run_jobs }}
-
-permissions:
-  contents: write
-
-jobs:
-  prepare:
-    name: Prepare the version before starting expensive jobs
-    permissions:
-      contents: write
-    runs-on: ubuntu-24.04
-    outputs:
-      run_jobs: ${{ github.event_name != 'push' || steps.channel.outputs.stable == 'true' || steps.version.outputs.action == 'ready' || steps.version.outputs.action == 'reserved' }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          token: ${{ github.event_name == 'push' && secrets.RELEASE_GH_TOKEN || github.token }}
-
-      - name: Keep synchronized stable branches on the released version
-        id: channel
-        if: github.event_name == 'push'
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
-          git fetch --no-tags origin main
-          MAIN_SHA="$(git rev-parse origin/main)"
-          # Stable reconciliation may preserve dev history with a merge commit.
-          # Its tree still matches main; bumping it would undo synchronization
-          # before the stable publisher can advance the Docker :dev tag.
-          if [[ "$GITHUB_REF_NAME" == dev && "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] &&
-             git merge-base --is-ancestor "$MAIN_SHA" "$GITHUB_SHA" &&
-             git diff --quiet "$MAIN_SHA" "$GITHUB_SHA"; then
-            echo 'stable=true' >> "$GITHUB_OUTPUT"
-          else
-            echo 'stable=false' >> "$GITHUB_OUTPUT"
-          fi
-
-      - id: version
-        if: github.event_name == 'push' && steps.channel.outputs.stable != 'true'
-        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
-        with:
-          package: archivebox
-          branch: ${{ github.ref_name }}
-          tag-prefix: v
-          version-scheme: ${{ github.ref_name == 'main' && 'patch' || 'rc' }}

.github/workflows/release-candidate.yml

@@ -0,0 +1,51 @@
+name: Release candidate
+
+on:
+  push:
+    branches: [dev]
+
+permissions:
+  actions: read
+  contents: write
+  packages: write
+
+concurrency:
+  group: release-candidate-dev
+  cancel-in-progress: true
+
+jobs:
+  prepare:
+    runs-on: ubuntu-24.04
+    outputs:
+      action: ${{ steps.version.outputs.action }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          submodules: true
+          token: ${{ secrets.RELEASE_GH_TOKEN }}
+
+      - id: version
+        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
+        with:
+          package: archivebox
+          branch: dev
+          tag-prefix: v
+          version-scheme: rc
+
+  python-artifacts:
+    needs: prepare
+    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
+    uses: ./.github/workflows/pip.yml
+    with:
+      full_tests: false
+    secrets: inherit
+
+  docker-digests:
+    needs: prepare
+    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
+    uses: ./.github/workflows/docker.yml
+    with:
+      push_digests: true
+      full_tests: false
+    secrets: inherit

.github/workflows/release.yml

@@ -2,56 +2,33 @@ name: Release
 
 on:
   workflow_run:
-    workflows: [CI]
+    workflows: [CI, Release candidate]
     types: [completed]
     branches: [dev, main]
-  workflow_dispatch:
-    inputs:
-      release_branch:
-        description: Branch being promoted
-        required: true
-        type: choice
-        options: [main, dev]
-      release_sha:
-        description: Exact tested commit to release
-        required: true
-        type: string
-      ci_run_id:
-        description: Successful CI run containing the tested Python artifacts
-        required: true
-        type: string
-      docker_digest_run_id:
-        description: Manual Docker digest build run containing the tested image digests
-        required: true
-        type: string
 
 permissions: {}
 
 concurrency:
-  group: release-${{ github.repository }}-${{ github.event_name == 'workflow_dispatch' && inputs.release_branch || github.event.workflow_run.head_branch }}
+  group: release-${{ github.repository }}-${{ github.event.workflow_run.head_branch }}
   cancel-in-progress: false
 
 env:
   DOCKERHUB_IMAGE: archivebox/archivebox
   GHCR_IMAGE: ghcr.io/archivebox/archivebox
-  RELEASE_BRANCH: ${{ github.event_name == 'workflow_dispatch' && inputs.release_branch || github.event.workflow_run.head_branch }}
-  RELEASE_SHA: ${{ github.event_name == 'workflow_dispatch' && inputs.release_sha || github.event.workflow_run.head_sha }}
+  RELEASE_BRANCH: ${{ github.event.workflow_run.head_branch }}
+  RELEASE_SHA: ${{ github.event.workflow_run.head_sha }}
+  CI_RUN_ID: ${{ github.event.workflow_run.id }}
 
 jobs:
   candidate:
     if: >-
       github.event.workflow_run.conclusion == 'success' &&
       github.event.workflow_run.event == 'push' &&
-      (github.event.workflow_run.head_branch == 'dev' || github.event.workflow_run.head_branch == 'main') ||
-      (github.event_name == 'workflow_dispatch' && inputs.release_branch == 'main')
+      ((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
+       (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
     runs-on: ubuntu-24.04
-    permissions:
-      actions: read
-      contents: read
     outputs:
-      ready: ${{ steps.verified.outputs.ready }}
-      artifact_run_id: ${{ steps.verified.outputs.artifact_run_id }}
-      digest_run_id: ${{ steps.verified.outputs.digest_run_id }}
+      ready: ${{ steps.owner.outputs.ready }}
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
@@ -60,84 +37,23 @@ jobs:
       - id: owner
         shell: bash
         run: |
-          set -Eeuo pipefail
-          BRANCH_SHA="$(git ls-remote origin "refs/heads/$RELEASE_BRANCH" | cut -f1)"
-          if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch && "$BRANCH_SHA" != "$RELEASE_SHA" ]]; then
-            echo 'ready=false' >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
           if [[ "$RELEASE_BRANCH" == main ]]; then
-            [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
-              echo "Refusing to publish non-stable version $VERSION as a main release" >&2
-              exit 1
-            }
-            git fetch --no-tags origin main
-            git merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
-              echo "Refusing to publish $RELEASE_SHA: it is not on main" >&2
-              exit 1
-            }
-            TAG_TARGET="$(git ls-remote origin "refs/tags/v$VERSION" | cut -f1)"
-            if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch && -n "$TAG_TARGET" && "$TAG_TARGET" != "$RELEASE_SHA" ]]; then
-              echo 'ready=false' >> "$GITHUB_OUTPUT"
-              exit 0
-            fi
             echo 'ready=true' >> "$GITHUB_OUTPUT"
             exit 0
           fi
-          # A stable release fast-forwards dev to main. Only main publishes it;
-          # the resulting dev workflow is not a new release candidate.
-          if [[ ! "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]]; then
-            echo 'ready=false' >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
+          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
           TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
           [[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
 
-      - name: Require successful full CI for this commit
-        id: verified
-        if: steps.owner.outputs.ready == 'true'
-        env:
-          GH_TOKEN: ${{ github.token }}
-          MANUAL_CI_RUN_ID: ${{ inputs.ci_run_id }}
-          MANUAL_DIGEST_RUN_ID: ${{ inputs.docker_digest_run_id }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          if [[ "$GITHUB_EVENT_NAME" == workflow_dispatch ]]; then
-            CI_RUN_ID="$MANUAL_CI_RUN_ID"
-            DIGEST_RUN_ID="$MANUAL_DIGEST_RUN_ID"
-            CI_STATUS="$(gh run view "$CI_RUN_ID" --repo "$GITHUB_REPOSITORY" --json status,conclusion,headSha,event,name \
-              --jq '[.status, .conclusion, .headSha, .event, .name] | @tsv')"
-            [[ "$CI_STATUS" == "$(printf 'completed\tsuccess\t%s\tpush\tCI' "$RELEASE_SHA")" ]]
-          else
-            CI_RUN_ID="$(gh run list --repo "$GITHUB_REPOSITORY" --workflow ci.yml \
-              --commit "$RELEASE_SHA" --branch "$RELEASE_BRANCH" --event push --limit 1 \
-              --json databaseId,status,conclusion \
-              --jq '.[0] | select(.status == "completed" and .conclusion == "success") | .databaseId // empty')"
-            [[ -n "$CI_RUN_ID" ]] || exit 0
-            DIGEST_RUN_ID="$CI_RUN_ID"
-          fi
-          ARTIFACT_RUN_ID="$CI_RUN_ID"
-          {
-            echo 'ready=true'
-            echo "artifact_run_id=$ARTIFACT_RUN_ID"
-            echo "digest_run_id=$DIGEST_RUN_ID"
-          } >> "$GITHUB_OUTPUT"
-
   python-release:
     name: Publish tested Python artifacts
     needs: candidate
     if: needs.candidate.outputs.ready == 'true'
     runs-on: ubuntu-24.04
-    env:
-      CI_RUN_ID: ${{ needs.candidate.outputs.artifact_run_id }}
-      DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
     environment: pypi
     permissions:
       actions: read
       contents: write
-      deployments: read
       id-token: write
     outputs:
       version: ${{ steps.version.outputs.version }}
@@ -161,26 +77,6 @@ jobs:
           version: "0.11.3"
           enable-cache: false
 
-      # This precedes every stable upload, including workflow_dispatch. A green
-      # build is not evidence that our automatically updated test servers ran it.
-      - name: Download tested Docker base-image metadata
-        if: env.RELEASE_BRANCH == 'main'
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
-        with:
-          path: ${{ runner.temp }}/tested-docker-artifacts
-          pattern: digest-*
-          merge-multiple: true
-          github-token: ${{ github.token }}
-          repository: ${{ github.repository }}
-          run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
-
-      - name: Require matching acceptance on Cabbage
-        if: env.RELEASE_BRANCH == 'main'
-        env:
-          GH_TOKEN: ${{ github.token }}
-          STAGING_DOCKER_ARTIFACT_DIR: ${{ runner.temp }}/tested-docker-artifacts
-        run: uv run --no-project python bin/verify-staging-release.py
-
       - name: Resolve release binaries through abxpkg
         env:
           ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
@@ -229,10 +125,6 @@ jobs:
           VERSION='${{ steps.version.outputs.version }}'
           case "$RELEASE_BRANCH" in
             main)
-              [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
-                echo "Refusing to publish non-stable version $VERSION from main" >&2
-                exit 1
-              }
               ;;
             dev)
               [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] || {
@@ -270,92 +162,20 @@ jobs:
   docker-release:
     name: Publish tested multiarch image
     if: >-
-      (github.event_name == 'workflow_dispatch') ||
-      (github.event.workflow_run.conclusion == 'success' &&
-       github.event.workflow_run.event == 'push')
-    needs: [candidate, python-release]
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.event == 'push'
+    needs: python-release
     runs-on: ubuntu-24.04
-    env:
-      DOCKER_DIGEST_RUN_ID: ${{ needs.candidate.outputs.digest_run_id }}
     permissions:
       actions: read
-      contents: write
+      contents: read
       packages: write
     steps:
       - name: Checkout the tested commit
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
-          fetch-depth: 0
+          fetch-depth: 1
           ref: ${{ env.RELEASE_SHA }}
-          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-
-      - name: Reconcile dev to the tested stable release
-        if: env.RELEASE_BRANCH == 'main' && needs.python-release.outputs.release_ready == 'true'
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          git fetch --no-tags origin main dev
-          MAIN_SHA="$(git rev-parse origin/main)"
-          DEV_SHA="$(git rev-parse origin/dev)"
-          if [[ "$MAIN_SHA" != "$RELEASE_SHA" || "$DEV_SHA" == "$RELEASE_SHA" ]]; then
-            exit 0
-          fi
-          push_dev() {
-            if ! git push origin "$1:refs/heads/dev"; then
-              CURRENT_DEV="$(git ls-remote origin refs/heads/dev | cut -f1)"
-              if [[ "$CURRENT_DEV" == "$DEV_SHA" ]]; then
-                echo "Failed to push dev although it still points to $DEV_SHA" >&2
-                exit 1
-              fi
-              echo "Leaving dev at $CURRENT_DEV: it advanced during release reconciliation"
-            fi
-          }
-          if git merge-base --is-ancestor "$DEV_SHA" "$RELEASE_SHA"; then
-            push_dev "$RELEASE_SHA"
-            exit 0
-          else
-            STATUS=$?
-            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
-          fi
-          BASE_SHA="$(git merge-base "$MAIN_SHA" "$DEV_SHA")"
-          if git diff --quiet "$BASE_SHA" "$DEV_SHA" -- . ':!pyproject.toml' ':!uv.lock' ':!etc/package.json'; then
-            :
-          else
-            STATUS=$?
-            [[ "$STATUS" -eq 1 ]] || exit "$STATUS"
-            echo "Leaving dev at $DEV_SHA: it contains changes beyond release metadata"
-            exit 0
-          fi
-          normalized_metadata() {
-            case "$2" in
-              pyproject.toml)
-                git show "$1:$2" | awk '
-                  /^\[/ {section = $0}
-                  section == "[project]" && /^version = "[^"]+"$/ {print "version = \"__VERSION__\""; next}
-                  section == "[tool.bumpver]" && /^current_version = "[^"]+"$/ {print "current_version = \"__VERSION__\""; next}
-                  {print}
-                ' ;;
-              uv.lock)
-                git show "$1:$2" | awk 'previous == "name = \"archivebox\"" && /^version = / {print "version = \"__VERSION__\""; previous = $0; next} {print; previous = $0}' ;;
-              etc/package.json)
-                git show "$1:$2" | sed -E 's/^(  "version": )"[^"]+"(,?)$/\1"__VERSION__"\2/' ;;
-            esac
-          }
-          for path in pyproject.toml uv.lock etc/package.json; do
-            BASE_METADATA="$(normalized_metadata "$BASE_SHA" "$path")"
-            DEV_METADATA="$(normalized_metadata "$DEV_SHA" "$path")"
-            if [[ "$BASE_METADATA" != "$DEV_METADATA" ]]; then
-              echo "Leaving dev at $DEV_SHA: $path contains changes beyond release version metadata"
-              exit 0
-            fi
-          done
-          # A concurrent release bot may bump only dev's own version metadata.
-          # Keep that commit as a parent while preserving the exact tested stable tree.
-          git switch --detach "$MAIN_SHA"
-          git -c user.name='github-actions[bot]' -c user.email='41898282+github-actions[bot]@users.noreply.github.com' \
-            merge -s ours --no-ff -m "Merge tested stable release into dev" "$DEV_SHA"
-          MERGE_SHA="$(git rev-parse HEAD)"
-          push_dev "$MERGE_SHA"
 
       - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
@@ -378,22 +198,20 @@ jobs:
             --lib="$ABXPKG_LIB_DIR" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries" >/dev/null
-          for name in uv gh git jq docker; do
+          for name in uv git jq docker; do
             test -L "$ABXPKG_LIB_DIR/env/bin/$name"
             test -x "$ABXPKG_LIB_DIR/env/bin/$name"
           done
           {
             echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
             echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
-            echo "GH_BINARY=$ABXPKG_LIB_DIR/env/bin/gh"
             echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
             echo "DOCKER_BINARY=$ABXPKG_LIB_DIR/env/bin/docker"
           } >> "$GITHUB_ENV"
 
       - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
         with:
-          # These images were already built and tested; only publish manifests.
-          driver: docker
+          install: true
 
       - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
         with:
@@ -414,31 +232,16 @@ jobs:
           merge-multiple: true
           github-token: ${{ github.token }}
           repository: ${{ github.repository }}
-          run-id: ${{ env.DOCKER_DIGEST_RUN_ID }}
-
-      - name: Verify exact successful digest build run
-        shell: bash
-        env:
-          GH_TOKEN: ${{ github.token }}
-        run: |
-          set -Eeuo pipefail
-          RUN_STATUS="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json status --jq '.status')"
-          RUN_CONCLUSION="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json conclusion --jq '.conclusion')"
-          [[ "$RUN_STATUS" == completed && "$RUN_CONCLUSION" == success ]]
-          if [[ "$GITHUB_EVENT_NAME" != workflow_dispatch ]]; then
-            RUN_SHA="$($GH_BINARY run view "$DOCKER_DIGEST_RUN_ID" --repo "$GITHUB_REPOSITORY" --json headSha --jq '.headSha')"
-            [[ "$RUN_SHA" == "$RELEASE_SHA" ]]
-          fi
+          run-id: ${{ env.CI_RUN_ID }}
 
       - name: Verify digest set and collect tags
         id: docker_meta
         shell: bash
         run: |
           set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
           [[ "${#DIGESTS[@]}" -eq 2 ]]
           for digest in "${DIGESTS[@]}"; do [[ "$digest" =~ ^[0-9a-f]{64}$ ]]; done
-          [[ "$(< /tmp/digests/source-ref)" == "$RELEASE_SHA" ]]
 
           VERSION='${{ needs.python-release.outputs.version }}'
           SHORT_SHA="${RELEASE_SHA::12}"
@@ -448,46 +251,12 @@ jobs:
             TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}")"
             TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
           fi
-          if [[ "$RELEASE_BRANCH" == main ]]; then
-            [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+$ ]] || {
-              echo "Refusing to tag :latest with prerelease $VERSION" >&2
-              exit 1
-            }
-            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] || {
-              echo "Refusing to tag :latest: v$VERSION does not point to $RELEASE_SHA" >&2
-              exit 1
-            }
-            $GIT_BINARY fetch --no-tags origin main
-            $GIT_BINARY merge-base --is-ancestor "$RELEASE_SHA" origin/main || {
-              echo "Refusing to tag :latest: $RELEASE_SHA is not on main" >&2
-              exit 1
-            }
-            MAIN_VERSION="$($GIT_BINARY show origin/main:pyproject.toml | sed -nE 's/^version = "([^"]+)".*/\1/p' | head -n 1)"
-            [[ "$MAIN_VERSION" == "$VERSION" ]] || {
-              echo "Refusing to tag :latest: main is now $MAIN_VERSION, not $VERSION" >&2
-              exit 1
-            }
-          fi
-          SYNC_DEV=false
-          if [[ "$RELEASE_BRANCH" == main ]]; then
-            $GIT_BINARY fetch --no-tags origin main dev
-            MAIN_TARGET="$($GIT_BINARY rev-parse origin/main)"
-            DEV_TARGET="$($GIT_BINARY rev-parse origin/dev)"
-            # Recovery can add CI/docs fixes after the immutable release tag.
-            # Both branches and every shipped application file must agree.
-            if $GIT_BINARY merge-base --is-ancestor "$MAIN_TARGET" "$DEV_TARGET" &&
-               $GIT_BINARY diff --quiet "$MAIN_TARGET" "$DEV_TARGET" &&
-               $GIT_BINARY diff --quiet "$RELEASE_SHA" "$MAIN_TARGET" -- . ':!.github' ':!README.md'; then
-              SYNC_DEV=true
-            fi
-          fi
           {
             echo 'dockerhub_tags<<EOF'
             if [[ "$RELEASE_BRANCH" == dev ]]; then
               echo "${DOCKERHUB_IMAGE}:dev"
             elif [[ "$RELEASE_BRANCH" == main ]]; then
               echo "${DOCKERHUB_IMAGE}:latest"
-              [[ "$SYNC_DEV" == true ]] && echo "${DOCKERHUB_IMAGE}:dev"
             fi
             echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
             [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${DOCKERHUB_IMAGE}:${VERSION}"
@@ -497,7 +266,6 @@ jobs:
               echo "${GHCR_IMAGE}:dev"
             elif [[ "$RELEASE_BRANCH" == main ]]; then
               echo "${GHCR_IMAGE}:latest"
-              [[ "$SYNC_DEV" == true ]] && echo "${GHCR_IMAGE}:dev"
             fi
             echo "${GHCR_IMAGE}:sha-${SHORT_SHA}"
             [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${GHCR_IMAGE}:${VERSION}"
@@ -508,7 +276,7 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
           TAG_ARGS=()
           while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.dockerhub_tags }}'
           REFS=()
@@ -519,54 +287,24 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
           TAG_ARGS=()
           while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.ghcr_tags }}'
           REFS=()
           for digest in "${DIGESTS[@]}"; do REFS+=("${GHCR_IMAGE}@sha256:${digest}"); done
           $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
 
-      - name: Verify published tags contain the exact tested images
-        env:
-          DOCKERHUB_TAGS: ${{ steps.docker_meta.outputs.dockerhub_tags }}
-          GHCR_TAGS: ${{ steps.docker_meta.outputs.ghcr_tags }}
+      - name: Verify published Docker images run
         shell: bash
         run: |
           set -Eeuo pipefail
-          # Both architectures already passed CLI/version/capture acceptance by
-          # digest. Verify promotion preserves those exact images and attestations
-          # instead of downloading and running the same image again on one arch.
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'import re; from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file() and re.fullmatch(r"[0-9a-f]{64}", path.name))))')
-          REFS=()
-          for digest in "${DIGESTS[@]}"; do REFS+=("${DOCKERHUB_IMAGE}@sha256:${digest}"); done
-          EXPECTED="$("$DOCKER_BINARY" buildx imagetools create --dry-run "${REFS[@]}" | "$JQ_BINARY" -ceS '.manifests | sort_by(.digest)')"
-          "$JQ_BINARY" -e 'map(select(.platform.os == "linux") | .platform.architecture) | sort == ["amd64", "arm64"]' <<< "$EXPECTED"
-          while IFS= read -r image; do
-            [[ -n "$image" ]] || continue
-            ACTUAL="$("$DOCKER_BINARY" buildx imagetools inspect --raw "$image" | "$JQ_BINARY" -ceS 'select(.schemaVersion == 2) | .manifests | sort_by(.digest)')"
-            [[ "$ACTUAL" == "$EXPECTED" ]] || { echo "Published tag differs from tested images: $image" >&2; exit 1; }
-            echo "Verified exact tested images: $image"
-          done < <(printf '%s\n' "$DOCKERHUB_TAGS" "$GHCR_TAGS")
-
-      - name: Record published image source
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          # workflow_run can report the default branch's head even when this
-          # run published a different branch/SHA. Preserve the verified target.
-          [[ "$RELEASE_BRANCH" == dev || "$RELEASE_BRANCH" == main ]]
-          [[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
-          mkdir -p "$RUNNER_TEMP/published-image-source"
-          # shellcheck disable=SC2016
-          "$JQ_BINARY" -n --arg branch "$RELEASE_BRANCH" --arg sha "$RELEASE_SHA" \
-            '{branch: $branch, sha: $sha}' > "$RUNNER_TEMP/published-image-source/source.json"
-
-      - name: Upload published image source
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
-        with:
-          name: published-image-source
-          path: ${{ runner.temp }}/published-image-source/source.json
-          if-no-files-found: error
+          VERSION='${{ needs.python-release.outputs.version }}'
+          SHORT_SHA="${RELEASE_SHA::12}"
+          for image in "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}" "${GHCR_IMAGE}:sha-${SHORT_SHA}"; do
+            "$DOCKER_BINARY" buildx imagetools inspect "$image" >/dev/null
+            INSTALLED_VERSION="$("$DOCKER_BINARY" run --rm "$image" --version)"
+            test "$INSTALLED_VERSION" = "$VERSION"
+          done
 
       - name: Update Docker Hub README
         uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
@@ -579,7 +317,6 @@ jobs:
     name: Cascade verified release
     if: needs.python-release.outputs.release_ready == 'true'
     needs:
-      - candidate
       - python-release
       - docker-release
     permissions:
@@ -590,6 +327,5 @@ jobs:
       upstream_package: archivebox
       upstream_version: ${{ needs.python-release.outputs.version }}
       upstream_artifact_name: python-distributions
-      upstream_run_id: ${{ needs.candidate.outputs.artifact_run_id }}
     secrets:
       release_token: ${{ secrets.RELEASE_GH_TOKEN }}

.github/workflows/screenshots.yml

@@ -1,81 +0,0 @@
-name: Capture publicsite screenshots
-
-on:
-  schedule:
-    - cron: '37 5,17 * * *'
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  actions: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
-jobs:
-  changes:
-    runs-on: ubuntu-24.04
-    outputs:
-      changed: ${{ steps.check.outputs.changed }}
-      key: ${{ steps.check.outputs.key }}
-    steps:
-      - id: check
-        uses: ArchiveBox/monorepo/.github/actions/changed-scheduled-inputs@main
-
-  deploy:
-    needs: changes
-    if: needs.changes.outputs.changed == 'true'
-    runs-on: ubuntu-24.04
-    steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Setup Python and uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          enable-cache: false
-
-      - name: Install ArchiveBox and screenshot dependencies
-        run: uv sync --no-cache --frozen
-
-      - name: Build UI screenshot gallery
-        run: bin/collect_ui_screenshots.sh
-        env:
-          BASE_URL: http://archivebox.localhost:5797
-
-      - name: Verify gallery was generated from this exact revision
-        run: |
-          uv run --no-cache --no-sync python - <<'PY'
-          import json
-          import os
-          import re
-          import tomllib
-          from pathlib import Path
-
-          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
-          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
-          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
-          assert provenance["version"] == version, provenance
-          assert provenance["capture_count"] >= 3 * 35, provenance
-          assert len(provenance["files"]) == provenance["capture_count"], provenance
-          assert all(not re.match(r"^\d+-", name) for name in provenance["files"]), provenance
-          PY
-
-      - name: Save validated screenshots
-        uses: actions/upload-artifact@v4
-        with:
-          name: site-screenshots
-          path: publicsite/screenshots
-          if-no-files-found: error
-          retention-days: 90
-
-  complete:
-    name: 'Scheduled inputs: ${{ needs.changes.outputs.key }}'
-    needs: [changes, deploy]
-    if: needs.changes.outputs.changed == 'true'
-    runs-on: ubuntu-24.04
-    steps:
-      - run: echo 'Scheduled work completed successfully.'

.github/workflows/staging-acceptance.yml

@@ -1,117 +0,0 @@
-name: Staging acceptance
-
-on:
-  workflow_run:
-    workflows: [Release]
-    types: [completed]
-
-permissions: {}
-
-concurrency:
-  group: staging-acceptance
-  cancel-in-progress: false
-
-jobs:
-  published:
-    if: github.event.workflow_run.conclusion == 'success'
-    runs-on: ubuntu-24.04
-    permissions:
-      actions: read
-    outputs:
-      ready: ${{ steps.source.outputs.ready }}
-      sha: ${{ steps.source.outputs.sha }}
-    steps:
-      - id: image
-        env:
-          GH_TOKEN: ${{ github.token }}
-          RELEASE_RUN_ID: ${{ github.event.workflow_run.id }}
-        run: |
-          set -Eeuo pipefail
-          published=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs/$RELEASE_RUN_ID/jobs?per_page=100" --jq 'any(.jobs[]; .name == "Publish tested multiarch image" and .conclusion == "success")')
-          echo "published=$published" >> "$GITHUB_OUTPUT"
-      - name: Download published image source
-        if: steps.image.outputs.published == 'true'
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
-        with:
-          name: published-image-source
-          path: ${{ runner.temp }}/published-image-source
-          github-token: ${{ github.token }}
-          repository: ${{ github.repository }}
-          run-id: ${{ github.event.workflow_run.id }}
-      - name: Select published dev image
-        id: source
-        if: steps.image.outputs.published == 'true'
-        run: |
-          set -Eeuo pipefail
-          # The triggering run may report the default branch as its head;
-          # this artifact names the image source verified by the publisher.
-          source="$RUNNER_TEMP/published-image-source/source.json"
-          jq -e 'type == "object" and (keys | sort) == ["branch", "sha"]' "$source" >/dev/null
-          branch=$(jq -er '.branch | select(type == "string")' "$source")
-          sha=$(jq -er '.sha | select(type == "string")' "$source")
-          [[ "$branch" == dev || "$branch" == main ]]
-          [[ "$sha" =~ ^[0-9a-f]{40}$ ]]
-          echo "sha=$sha" >> "$GITHUB_OUTPUT"
-          if [[ "$branch" == dev ]]; then
-            echo 'ready=true' >> "$GITHUB_OUTPUT"
-          else
-            echo 'ready=false' >> "$GITHUB_OUTPUT"
-          fi
-  acceptance:
-    # The privileged key is never exposed to PRs or arbitrary dispatch refs.
-    needs: published
-    if: needs.published.outputs.ready == 'true'
-    runs-on: [self-hosted, Linux, X64, ugnas-linux]
-    environment: staging
-    timeout-minutes: 110
-    permissions:
-      contents: read
-      deployments: write
-    strategy:
-      fail-fast: false
-      matrix:
-        target: [cabbage]
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262
-        with:
-          ref: ${{ needs.published.outputs.sha }}
-          fetch-depth: 0
-          persist-credentials: false
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e
-        with:
-          version: '0.11.3'
-      - name: Prepare browser and restricted staging identity
-        env:
-          STAGING_SSH_KEY: ${{ secrets.STAGING_SSH_KEY }}
-          STAGING_KNOWN_HOSTS: ${{ secrets.STAGING_KNOWN_HOSTS }}
-          ABXPKG_LIB_DIR: ${{ runner.tool_cache }}/abxpkg/staging-browser
-        run: |
-          set -Eeuo pipefail
-          umask 077
-          printf '%s\n' "$STAGING_SSH_KEY" > "$RUNNER_TEMP/staging-key"
-          printf '%s\n' "$STAGING_KNOWN_HOSTS" > "$RUNNER_TEMP/staging-known-hosts"
-          # Use the candidate's normal provider/cache path, not a second
-          # Playwright-managed browser that bypasses abxpkg validation.
-          ABX_DL_VERSION="$(uv run --no-project python -c 'import tomllib; print(next(p["version"] for p in tomllib.load(open("uv.lock", "rb"))["package"] if p["name"] == "abx-dl"))')"
-          CHROME_SCHEMA="$(uv run --no-project --with "abx-dl==$ABX_DL_VERSION" python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "chrome" / "config.json")')"
-          uv run --no-project --with "abx-dl==$ABX_DL_VERSION" abxpkg env --install --json --deps-from="$CHROME_SCHEMA:required_binaries" > "$RUNNER_TEMP/browser-env.json"
-          CHROME_BINARY="$(uv run --no-project python -c 'import json, sys; print(json.load(open(sys.argv[1]))["CHROME_BINARY"])' "$RUNNER_TEMP/browser-env.json")"
-          test -x "$CHROME_BINARY"
-          echo "CHROME_BINARY=$CHROME_BINARY" >> "$GITHUB_ENV"
-      - name: Verify actual deployment, full captures and rendered snapshots
-        env:
-          GH_TOKEN: ${{ github.token }}
-          STAGING_TARGET: ${{ matrix.target }}
-          STAGING_SHA: ${{ needs.published.outputs.sha }}
-          STAGING_KEY_FILE: ${{ runner.temp }}/staging-key
-          STAGING_KNOWN_HOSTS_FILE: ${{ runner.temp }}/staging-known-hosts
-        run: uv run --no-project --with playwright==1.63.0 python bin/check-staging-deployment.py
-      - name: Remove staging identity
-        if: always()
-        run: rm -f "$RUNNER_TEMP/staging-key" "$RUNNER_TEMP/staging-known-hosts"
-      - uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02
-        if: always()
-        with:
-          name: staging-${{ matrix.target }}-${{ needs.published.outputs.sha }}
-          path: staging-evidence/
-          if-no-files-found: error

.github/workflows/test-parallel.yml

@@ -1,20 +1,7 @@
 name: Parallel Tests
 
 on:
-  workflow_dispatch:
-    inputs:
-      test_file:
-        description: Run one discovered test file
-        required: true
-        default: archivebox/tests/test_cli_list.py
-        type: string
   workflow_call:
-    inputs:
-      test_file:
-        description: Run one discovered test file; empty runs the complete matrix
-        required: false
-        default: ""
-        type: string
 
 env:
   PYTHONIOENCODING: utf-8
@@ -26,20 +13,11 @@ env:
 jobs:
   discover-tests:
     name: Discover every test
-    # Selection must run on GitHub so an offline NAS cannot block the fallback.
     runs-on: ubuntu-24.04
     outputs:
       tests: ${{ steps.discover.outputs.tests }}
-      runner: ${{ steps.runner.outputs.use-runner || '["ubuntu-24.04"]' }}
 
     steps:
-      - name: Select available CI capacity
-        id: runner
-        if: vars.UGNAS_CI_ENABLED == 'true' && (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && (github.ref == 'refs/heads/dev' || github.ref == 'refs/heads/main')
-        uses: ArchiveBox/monorepo/.github/actions/select-ci-runner@main
-        with:
-          github-token: ${{ secrets.RUNNER_STATUS_TOKEN }}
-
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
@@ -80,16 +58,10 @@ jobs:
 
       - name: Discover every test exactly once
         id: discover
-        env:
-          TEST_FILE: ${{ inputs.test_file }}
-          UGNAS_CI_MAX_JOBS: ${{ vars.UGNAS_CI_MAX_JOBS || 3 }}
         run: |
           set -Eeuo pipefail
           mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
           test "${#output[@]}" -eq 2
-          if [[ -n "$TEST_FILE" ]]; then
-            output[1]="$("$JQ_BINARY" -ce '[.[] | select(.paths == [env.TEST_FILE])] | select(length == 1)' <<< "${output[1]}")"
-          fi
           echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
           echo "${output[0]}"
           "$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
@@ -97,13 +69,13 @@ jobs:
   tests:
     name: ${{ matrix.test.name }}
     needs: discover-tests
-    # Opt in trusted branches only; PRs always use GitHub-hosted runners.
-    # Ordinary Linux tests are eligible; # ci-runner: hosted declares exceptions.
-    # The hosted discovery job falls back when workers are offline or the
-    # availability API fails. UGNAS_CI_ENABLED=false disables the pool entirely.
-    runs-on: ${{ fromJSON(matrix.test.ugnas && needs.discover-tests.outputs.runner || '["ubuntu-24.04"]') }}
+    runs-on: ubuntu-24.04
     env:
+      ABXPKG_LIB_DIR: /tmp/archivebox-lib
       CHROME_HEADLESS: "true"
+      DATA_DIR: /tmp/archivebox-test-data
+      PERSONAS_DIR: /tmp/abx-personas
+      CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile
 
     strategy:
       fail-fast: false
@@ -111,28 +83,6 @@ jobs:
         test: ${{ fromJson(needs.discover-tests.outputs.tests) }}
 
     steps:
-      - name: Set job-local runtime paths
-        env:
-          CI_RUNNER_ENVIRONMENT: ${{ runner.environment }}
-        run: |
-          workers=2
-          cache_suffix=''
-          if [[ "$CI_RUNNER_ENVIRONMENT" == github-hosted ]]; then
-            workers="$(nproc)"
-            cache_suffix="-workers-$workers"
-            # Avoid the hosted image's cold Chromium executable page faults.
-            # abxpkg owns installation and reuse in the existing runtime cache.
-            echo "CHROME_BINPROVIDERS=playwright,puppeteer" >> "$GITHUB_ENV"
-          fi
-          {
-            echo "CI_TEST_WORKERS=$workers"
-            echo "ABXPKG_CACHE_WORKERS_SUFFIX=$cache_suffix"
-            echo "ABXPKG_LIB_DIR=$RUNNER_TEMP/archivebox-lib"
-            echo "DATA_DIR=$RUNNER_TEMP/archivebox-test-data"
-            echo "PERSONAS_DIR=$RUNNER_TEMP/abx-personas"
-            echo "CHROME_USER_DATA_DIR=$RUNNER_TEMP/abx-personas/Default/chrome_profile"
-          } >> "$GITHUB_ENV"
-
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
@@ -142,7 +92,6 @@ jobs:
         with:
           python-version: ${{ env.CI_PYTHON_VERSION }}
           architecture: x64
-        id: python
 
       - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
         with:
@@ -150,41 +99,6 @@ jobs:
           enable-cache: false
           cache-dependency-glob: uv.lock
 
-      # Self-hosted tool storage survives jobs. Normal abxpkg validation and
-      # install locks allow reuse across files; cold-install tests use tmp_path.
-      - name: Reuse persistent runner tools
-        if: runner.environment == 'self-hosted'
-        env:
-          PERSISTENT_ABXPKG_LIB: ${{ runner.tool_cache }}/abxpkg/${{ github.repository }}/${{ steps.python.outputs.python-version }}
-        run: |
-          mkdir -p "$PERSISTENT_ABXPKG_LIB"
-          echo "ABXPKG_LIB_DIR=$PERSISTENT_ABXPKG_LIB" >> "$GITHUB_ENV"
-
-      # Restored binaries still pass normal abxpkg resolution/version checks.
-      # Installation and isolation tests explicitly choose fresh temporary libs.
-      - name: Identify runtime dependency cache
-        shell: bash
-        run: |
-          uv run --no-project python - <<'PYTHON' >> "$GITHUB_ENV"
-          import hashlib
-          import json
-          import tomllib
-          from pathlib import Path
-
-          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
-          versions = {p["name"]: p["version"] for p in packages if p["name"] in {"abxpkg", "abx-plugins"} and "editable" not in p["source"]}
-          print("ABXPKG_CACHE_DEPS=" + hashlib.sha256(json.dumps(versions, sort_keys=True).encode()).hexdigest())
-          PYTHON
-
-      - name: Reuse validated runtime dependencies
-        if: runner.environment != 'self-hosted'
-        uses: actions/cache@0057852bfaa89a56745cba8c7296529d2fc39830 # v4
-        with:
-          path: ${{ runner.temp }}/archivebox-lib
-          key: abxpkg-v1-${{ runner.os }}-${{ runner.arch }}-${{ runner.environment }}-${{ steps.python.outputs.python-version }}-${{ matrix.test.name }}-${{ env.ABXPKG_CACHE_DEPS }}${{ env.ABXPKG_CACHE_WORKERS_SUFFIX }}
-          restore-keys: |
-            abxpkg-v1-${{ runner.os }}-${{ runner.arch }}-${{ runner.environment }}-${{ steps.python.outputs.python-version }}-${{ matrix.test.name }}-
-
       - name: Resolve optional build dependencies through abxpkg
         if: matrix.test.extra != '' || contains(matrix.test.paths_arg, 'test_postgres_backend')
         env:
@@ -218,38 +132,23 @@ jobs:
             extra_args=(--extra "$TEST_EXTRA")
           fi
           uv sync --no-cache --locked --dev "${extra_args[@]}"
+
       - name: Run ${{ matrix.test.name }}
         env:
           TEST_PATHS: ${{ matrix.test.paths_arg }}
-          CI_RUNNER_ENVIRONMENT: ${{ runner.environment }}
         run: |
           set -Eeuo pipefail
-          echo "Running pytest with $CI_TEST_WORKERS workers on $CI_RUNNER_ENVIRONMENT"
           read -r -a test_paths <<< "$TEST_PATHS"
-          test_args=()
           for test_path in "${test_paths[@]}"; do
             test -f "$test_path"
-            test_args+=("$GITHUB_WORKSPACE/$test_path")
           done
-          # Each test still gets isolated DB/filesystem state from conftest.
-          # Keep one worker pool per batch instead of starting it for every file.
-          mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest"
-          uv run --no-cache --no-sync --no-sources pytest -vs -n "$CI_TEST_WORKERS" "${test_args[@]}" --durations=20 \
-            --basetemp="$RUNNER_TEMP/pytest" \
-            --junitxml="$RUNNER_TEMP/test-results/results.xml"
-      - name: Save test results
-        if: always()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
-        with:
-          name: test-results-${{ strategy.job-index }}
-          path: ${{ runner.temp }}/test-results/*.xml
-          if-no-files-found: error
-          retention-days: 14
-      - name: Save failed runtime diagnostics
-        if: failure()
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
-        with:
-          name: runtime-logs-${{ strategy.job-index }}
-          path: ${{ runner.temp }}/pytest/**/*.log
-          if-no-files-found: ignore
-          retention-days: 14
+          for test_path in "${test_paths[@]}"; do
+            test_name="$(basename "$test_path" .py)"
+            test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
+            export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
+            export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
+            export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
+            mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
+            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
+              --basetemp="$RUNNER_TEMP/pytest/$test_slug"
+          done

.github/workflows/test.yml

@@ -2,12 +2,6 @@ name: Install and CLI Compatibility
 
 on:
   workflow_call:
-    inputs:
-      platform:
-        description: Platform for install and CLI compatibility
-        required: false
-        default: linux
-        type: string
 
 env:
   DOCKER_IMAGE: archivebox-ci
@@ -25,7 +19,16 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        include: ${{ fromJSON(inputs.platform == 'macos' && '[{"os":"macos-15","os_name":"macOS","python":"3.13"},{"os":"macos-15","os_name":"macOS","python":"3.14"}]' || '[{"os":"ubuntu-24.04","os_name":"linux","python":"3.14"}]') }}
+        include:
+          - os: ubuntu-24.04
+            os_name: linux
+            python: "3.14"
+          - os: macos-15
+            os_name: macOS
+            python: "3.13"
+          - os: macos-15
+            os_name: macOS
+            python: "3.14"
 
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4

.gitignore

@@ -68,5 +68,3 @@ ArchiveBox.conf
 
 # Local Claude Code task locks (not project state)
 .claude/scheduled_tasks.lock
-
-/_site/

.pre-commit-config.yaml

@@ -56,8 +56,6 @@ repos:
       - id: mixed-line-ending
       - id: fix-byte-order-marker
       - id: end-of-file-fixer
-        # Preserve the exact bytes of captured cryptographic proof fixtures.
-        exclude: ^archivebox/tests/fixtures/legacy_proofs/.*\.json$
       - id: detect-private-key
       - id: debug-statements
       - id: forbid-submodules

AGENTS.md

@@ -77,7 +77,11 @@ archivebox run
 
 ## Verification
 
-Run the test suite with `uv run pytest archivebox/tests -x` and repository checks
-with `uv run prek run --all-files`. For focused work, select the relevant tests.
+Use targeted tests for focused work:
 
-Releases are published only by `.github/workflows/release.yml` after complete CI succeeds for the exact commit. Pushes to `main` prepare stable patch releases automatically; `dev` prepares release candidates. Version selection and dependency repinning are owned by CI and the monorepo coordinator. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
+```bash
+uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
+uv run prek run --all-files
+```
+
+Releases are published only by `.github/workflows/release.yml` after the complete `dev` CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.

Dockerfile

@@ -1,38 +1,8 @@
 # syntax=docker/dockerfile:1.7
 
 # Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
-# WHY: abx-dl owns Python, Node, Chromium, and extraction dependencies so the
-# standalone downloader stays independent of ArchiveBox's server features.
-# ArchiveBox-only dependencies (server plugins, supervisor, Django) belong in
-# the layers added by this Dockerfile, never in the shared abx-dl base image.
-# REQUIRED INSTALLATION CONTRACT — do not work around failures:
-# 1. IMAGE CONSTRUCTION: preinstall ALL plugins supported by runtime="archivebox"
-#    and ALL their required_binaries, regardless of default enabled state.
-#    Inherit the complete downloader dependencies from the tested abx-dl image;
-#    install every remaining ArchiveBox-only dependency in this image's layers,
-#    including optional server plugins. Never put server dependencies in abx-dl.
-#    Use the normal plugin dependency pipeline: install command -> config.json
-#    required_binaries -> abxpkg. Never add manual apt/pip/npm installs to hide
-#    failures in that pipeline. OS/interpreter bootstrap is a separate concern,
-#    not a second hand-maintained list of plugin dependencies.
-# 2. FINISHED IMAGE: archivebox install and the full plugin installation check
-#    must discover everything already present and be NO-OPS offline as UID 911.
-#    No downloads, package installs, sudo, or root repair may be needed to enable
-#    ANY plugin or start a new crawl. Never run verification as root or manually
-#    preinstall packages before it. If the inherited image lacks a downloader
-#    dependency, fix/release abx-dl and consume that tested image; do not patch
-#    over the missing base dependency here or repair deployed containers by SSH.
-# 3. NON-CONTAINER CI: exercise real runtime auto-installation through the same
-#    CLI/required_binaries/abxpkg path users invoke. GitHub's runner has
-#    passwordless sudo; abxpkg must escalate automatically when needed. Never
-#    manually preinstall dependencies or wrap the tested command in sudo to make
-#    a failure disappear. This is distinct from the finished-image no-op test.
-# 4. SIZE: the complete compressed image budget is 1025 MiB. Preserve ALL
-#    user-facing functionality. Only mechanical/internal
-#    optimizations are allowed. Never remove dependencies, disable features or
-#    plugins, weaken assertions, or skip checks to meet a size limit. If the
-#    complete image exceeds the configured limit, LET CI FAIL and explicitly
-#    ask the maintainer what to do. Do not raise the limit unilaterally.
+# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
+# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
 # Build abx-dl first, then point this file at it:
 #   docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
 #       --build-context abxbus=../abxbus \
@@ -40,10 +10,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:dev \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.233 \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:main
+ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.233
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -77,7 +47,6 @@ ENV ARCHIVEBOX_USER=archivebox \
     IN_DOCKER=True
 
 ENV CODE_DIR=/app \
-    ABX_RUNTIME=archivebox \
     DATA_DIR=/data \
     CONFIG_DIR=/opt/archivebox \
     ABXPKG_LIB_DIR=/opt/archivebox/lib \
@@ -125,74 +94,87 @@ FROM archivebox-runtime-base AS archivebox-builder
 WORKDIR "$CODE_DIR"
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
     --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
-    --mount=type=bind,source=uv.lock,target=/app/uv.lock \
     <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
-export UV_NO_CACHE=true
+echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
+echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
+echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
 rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
-apt-get install -qq -y --no-install-recommends x11-utils
+apt-get install -qq -y --no-install-recommends \
+    build-essential gcc libldap2-dev libsasl2-dev libssl-dev
 # Extend abx-dl's existing venv instead of clearing it. Clearing and later
 # copying a complete replacement over the base duplicates every venv byte in
 # the final overlay history even when most packages are unchanged.
-# Do not rewrite inherited native libraries merely to strip them again.
-/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries
+/usr/bin/uv pip install --no-cache setuptools pip wheel
+
+mkdir -p /tmp/archivebox-uv-project
+/venv/bin/python - <<'PY'
+from pathlib import Path
+import json
+import re
+import urllib.request
+
+source = Path("/app/pyproject.toml")
+target = Path("/tmp/archivebox-uv-project/pyproject.toml")
+text = source.read_text()
+text = text.replace(
+    'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
+    'environments = ["sys_platform == \'linux\'"]',
+)
+
+# Docker builds need the just-published internal abx wheels immediately, but
+# PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
+# Docker-only dependency view from the version JSON so the published package
+# metadata stays normal while image builds remain resumable after a release.
+for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
+    match = re.search(
+        rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
+        text,
+    )
+    if not match:
+        continue
+    version = match.group("version")
+    with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
+        data = json.load(response)
+    wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
+    text = re.sub(
+        rf'"{re.escape(package)}(?:==|>=)[^"]+"',
+        f'"{package} @ {wheel_url}"',
+        text,
+        count=1,
+    )
+
+target.write_text(text)
+PY
 
-# The old Docker-only dependency rewrite predated this checkout's lockfile:
-# it fetched version JSON to avoid stale PyPI Simple listings. That extra API
-# can itself lag or return 503s. The verified lock already records exact public
-# artifact URLs and hashes, including extras and platform markers. Consume it
-# directly, retaining the freshness check instead of resolving a second graph.
 /usr/bin/uv sync \
-    --project /app \
-    --locked \
+    --project /tmp/archivebox-uv-project \
     --no-cache \
     --no-dev \
     --inexact \
     --no-install-project \
     --no-install-workspace \
-    --no-install-package abxbus \
-    --no-install-package abxpkg \
-    --no-install-package abx-plugins \
-    --no-install-package abx-dl \
     --no-sources
-# Sync may replace inherited dependencies (e.g. pydantic/rich-click), removing
-# their bytecode. Runtime writes are disabled, so missing caches would make
-# every hook compile them again. Compile only missing entries: compileall's
-# timestamp fast path rewrites even valid checked-hash caches, copying the
-# entire inherited venv into this OCI layer for no runtime benefit.
-/usr/bin/uv run --no-project --no-sync python - <<'PY'
-import importlib.util
-from pathlib import Path
-import py_compile
-import sysconfig
-
-for source in Path(sysconfig.get_path("purelib")).rglob("*.py"):
-    cache = Path(importlib.util.cache_from_source(str(source)))
-    if not cache.is_file():
-        py_compile.compile(str(source), doraise=True, invalidation_mode=py_compile.PycInvalidationMode.CHECKED_HASH)
-PY
 builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
 ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
 while IFS= read -r -d '' native_library; do
-    if grep -Fxq "$native_library" /tmp/archivebox-inherited-libraries; then
-        continue
-    fi
     magic=''
     if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
         "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
     fi
 done < /tmp/archivebox-native-libraries
-rm -f /tmp/archivebox-native-libraries /tmp/archivebox-inherited-libraries
+rm -f /tmp/archivebox-native-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
 abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
 rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
     /venv/lib/python3.*/site-packages/wheel* \
     /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-rm -rf /var/lib/apt/lists/*
+echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
+rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
 EOF
 
 COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
@@ -220,7 +202,7 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
-# The builder installs and purges stripping tools in one layer, so its final
+# The builder installs and purges compilers in one layer, so its final
 # filesystem is already runtime-clean. Preserve that ancestry: starting again
 # from archivebox-runtime-base and COPYing /venv would bake a second full venv
 # over the inherited one instead of recording only ArchiveBox's package delta.
@@ -243,8 +225,7 @@ COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 # The builder invokes abxpkg as root for temporary ELF stripping, which can
 # rewrite derived state ownership. Restore UID 911 before resolving binaries
 # as the runtime user; the old copy-based final stage hid this dependency.
-RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777,id=archivebox-runtime-config-$TARGETARCH \
-    echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
+RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
     && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
     && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
     && ln -sf /venv/bin/daphne /usr/local/bin/daphne \
@@ -258,31 +239,12 @@ RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
     && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
-    && find "$ABXPKG_LIB_DIR" \( ! -user "$DEFAULT_ARCHIVEBOX_UID" -o ! -group "$DEFAULT_ARCHIVEBOX_GID" \) -exec chown -h "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" {} + \
-    && UV_NO_CACHE=true setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
+    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
     && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
 
-# Install the complete ArchiveBox catalog, including disabled server plugins.
-# Explicit plugin names make installation independent of enabled defaults.
-RUN --mount=type=cache,target=/var/tmp/abxpkg-cache,sharing=locked,mode=1777,id=archivebox-server-plugins-$TARGETARCH \
-    chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" /var/tmp/abxpkg-cache \
-    && chmod 1777 /var/tmp/abxpkg-cache \
-    && export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
-    && env ABXPKG_TMP_CACHE_DIR=/var/tmp/abxpkg-cache XDG_CACHE_HOME=/var/tmp/abxpkg-cache \
-        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abx-dl install $ABX_DOCKER_PLUGINS \
-    && UV_NO_CACHE=true /usr/bin/uv run --no-project python -c 'from abx_plugins.plugins.opencode.image import prune_incompatible_image_files; prune_incompatible_image_files()' \
-    # pnpm's explicit store lives under ABXPKG_LIB_DIR, regardless of XDG_CACHE_HOME.
-    # Retaining its download copies made image size depend on whether the build
-    # preserved hardlinks to installed files. Remove the store in this layer;
-    # installed files retain their contents, and the offline check below proves
-    # the complete catalog still works without the disposable package payloads.
-    && rm -rf "$ABXPKG_LIB_DIR/cache/pnpm"
-RUN --network=none export ABX_DOCKER_PLUGINS="$(/venv/bin/python3 -c 'from abx_dl.catalog import PluginCatalog; print(" ".join(PluginCatalog.discover(runtime="archivebox")))')" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups \
-    bash -c 'uv run --no-project python -c "from abx_plugins.plugins.opencode.image import verify_installed; verify_installed()" && abx-dl install $ABX_DOCKER_PLUGINS'
-
 WORKDIR "$DATA_DIR"
 RUN echo "[+] Initializing image collection..." \
     && find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
@@ -301,9 +263,6 @@ RUN chmod +x "$CODE_DIR"/bin/*.sh \
     && chmod g+w "$TMP_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
 
 RUN for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
-    && test -f "$(/venv/bin/python -c 'import pydantic; print(pydantic.__cached__)')" \
-    && test -f "$(/venv/bin/python -c 'import rich_click; print(rich_click.__cached__)')" \
-    && test -f "$(/venv/bin/python -c 'import platformdirs; print(platformdirs.__cached__)')" \
     && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$CONFIG_DIR" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$ABXPKG_LIB_DIR" \
@@ -318,10 +277,10 @@ RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully
 
 WORKDIR "$DATA_DIR"
 VOLUME "$DATA_DIR"
-EXPOSE 5797
+EXPOSE 8000
 
 HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
-    CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:5797/health/' | grep -q 'OK'
+    CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
 
 ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
-CMD ["archivebox", "server", "--init", "0.0.0.0:5797"]
+CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]

README.md

@@ -73,15 +73,15 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
 curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
 docker compose pull
 docker compose up -d --wait                                                # initializes new collections automatically
-# open http://admin.archivebox.localhost:5797 to finish setup
+# open http://admin.archivebox.localhost:8000 to finish setup
 # docker compose run --rm archivebox add 'https://example.com'
 # docker compose run --rm archivebox help
 <br/>
 <br/>
 # Option B: Or use it as a plain Docker container:
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -d --name archivebox -v "$PWD:/data" -p 5797:5797 archivebox/archivebox:dev
-# open http://admin.archivebox.localhost:5797 to finish setup
+docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
+# open http://admin.archivebox.localhost:8000 to finish setup
 # docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
 # docker run -it -v $PWD:/data archivebox/archivebox:dev help
 <br/>
@@ -93,23 +93,24 @@ archivebox init
 archivebox install
 # archivebox add 'https://example.com'
 # archivebox help
-# archivebox server 0.0.0.0:5797
+# archivebox server 0.0.0.0:8000
 <br/>
 <br/>
 # Option D: Or use the uv install shortcut for Option C
 curl -fsSL 'https://get.archivebox.io' | bash
 </code></pre>
 <br/>
-<sub>Open <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI and <a href="http://admin.archivebox.localhost:5797"><code>http://admin.archivebox.localhost:5797</code></a> for the admin UI ➡️</sub><br/>
+<sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
 <sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
 </details>
 <br/>
 
+
 <div align="center" style="text-align: center">
 <br/><br/>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
 <br/><br/>
-<small><a href="https://demo.archivebox.io">Demo</a> | <a href="https://archivebox.io/screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
+<small><a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
 <br/>
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
 <br/><br/>
@@ -123,15 +124,15 @@ curl -fsSL 'https://get.archivebox.io' | bash
 ## Key Features
 
 - [**Free & open source**](https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE), own your own data & maintain your privacy by self-hosting
-- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) with [modular dependencies](#dependencies) and [support for Google Drive/NFS/SMB/S3/B2/etc.](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage)
+- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) with [modular dependencies](#dependencies) and [support for Google Drive/NFS/SMB/S3/B2/etc.](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage)
 - [**Comprehensive documentation**](https://github.com/ArchiveBox/ArchiveBox/wiki), [active development](https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap), and [rich community](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community)
 - [**Extracts a wide variety of content out-of-the-box**](https://github.com/ArchiveBox/ArchiveBox/issues/51): [media (yt-dlp), articles (readability), code (git), etc.](#output-formats)
 - [**Supports scheduled/realtime importing**](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving) from [many types of sources](#input-formats)
 - [**Uses standard, durable, long-term formats**](#output-formats) like HTML, JSON, PDF, PNG, MP4, TXT, and WARC
-- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage), [Python API](https://docs.archivebox.io/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://electron.archivebox.io/)
-- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archivedotorg_enabled) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archivedotorg_enabled) for local-only mode)
-- Support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
-- Support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [infiniscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
+- [**Powerful CLI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage), [**self-hosted web UI**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#UI-Usage), [Python API](https://docs.archivebox.io/en/dev/apidocs/archivebox/archivebox.html) (BETA), [REST API](https://github.com/ArchiveBox/ArchiveBox/issues/496) (ALPHA), or [desktop app](https://github.com/ArchiveBox/electron-archivebox)
+- [**Saves all pages to archive.org as well**](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#save_archive_dot_org) by default for redundancy (can be [disabled](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) for local-only mode)
+- Advanced users: support for archiving [content requiring login/paywall/cookies](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir) (see wiki security caveats!)
+- Planned: support for running [JS during archiving](https://github.com/ArchiveBox/ArchiveBox/issues/51) to adblock, [autoscroll](https://github.com/ArchiveBox/ArchiveBox/issues/80), [modal-hide](https://github.com/ArchiveBox/ArchiveBox/issues/175), [thread-expand](https://github.com/ArchiveBox/ArchiveBox/issues/345)
 
 <br/>
 
@@ -150,17 +151,6 @@ curl -fsSL 'https://get.archivebox.io' | bash
 
 #### ✳️&nbsp; Easy Setup
 
-<details>
-<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
-<br/>
-<ol>
-<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
-<li>Set up an admin account and generate an API Key.</li>
-<li>Use the Web UI or <a href="https://app.archivebox.io/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://app.archivebox.io/">mobile app</a>, the <a href="https://extension.archivebox.io/">browser extension</a>, and other integrations to your server using your API Key.</li>
-</ol>
-<br/>
-</details>
-
 <details>
 <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b>  (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
 <br/>
@@ -176,7 +166,7 @@ docker compose pull
 <li>Start the server, which initializes a new collection automatically.
 <pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
 </code></pre></li>
-<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
 <pre lang="bash"><code style="white-space: pre-line">
 # run CLI commands inside the server container started above
 docker compose exec archivebox archivebox add 'https://example.com'
@@ -197,10 +187,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
 <li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -d --name archivebox -v $PWD:/data -p 5797:5797 archivebox/archivebox:dev
+docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 </li>
-<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
 <pre lang="bash"><code style="white-space: pre-line">
 # completely optional, CLI can always be used without running a server
 # docker exec archivebox archivebox [subcommand] [--help]
@@ -220,7 +210,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <ol>
 <li>Run the shortcut for the <code>uv</code> install method. It installs <code>uv</code> first when needed.
 <pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | bash</code></pre>
-<i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-uv-install-shortcut">Install: Bare Metal</a> in the Wiki. ➡️</i>
+<i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-automatic-setup-script">Install: Bare Metal</a> in the Wiki. ➡️</i>
 </li>
 </ol>
 
@@ -255,8 +245,8 @@ archivebox init     # initialize a new collection
 archivebox install  # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
 </code></pre>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -291,8 +281,8 @@ archivebox add 'https://example.com'
 </code></pre>
 <br/>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -324,8 +314,8 @@ archivebox init
 archivebox install
 </code></pre>
 </li>
-<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:5797/admin/">http://admin.archivebox.localhost:5797/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:5797
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
+<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -364,12 +354,17 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 <br/>
 <ol>
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
-<li><a href="https://electron.archivebox.io/#get-started">Download the latest installer for Windows, Linux, or Mac</a>.</li>
-<li>Install and open ArchiveBox Desktop, then create your local administrator account to start archiving.</li>
+<li>Download a binary release for your OS or build the native app from source<br/>
+<ul>
+<li>macOS: <a href="https://github.com/ArchiveBox/ArchiveBox/releases/download/v0.6.2/Electron-ArchiveBox-macOS-x64-0.6.2.app.zip" download><code>ArchiveBox.app.zip</code></a></li>
+<li>Linux: <code>ArchiveBox.deb</code> (alpha: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
+<li>Windows: <code>ArchiveBox.exe</code> (beta: <a href="https://github.com/ArchiveBox/electron-archivebox#quickstart">build manually</a>)</li>
+</ul>
+</li>
 </ol>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/575ef92f-bb3e-4a7c-a4ba-986c1fd76ecf" width="320px">
 <br/>
-<a href="https://electron.archivebox.io/">Desktop app guide</a> · <a href="https://electron.archivebox.io/screenshots/">Screenshots</a>
+<i>✨ Alpha (contributors wanted!)</i>: for more info, see the: <a href="https://github.com/ArchiveBox/electron-archivebox">Electron ArchiveBox</a> repo.
 <br/>
 </details>
 
@@ -380,7 +375,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 > *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*
 
 <ul>
-<li><a href="https://apps.truenas.com/catalog/archivebox_community/">TrueNAS</a></li>
+<li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
 <li><a href="https://unraid.net/community/apps?q=archivebox#r">UnRaid</a></li>
 <li><a href="https://community-scripts.github.io/ProxmoxVE/scripts?id=archivebox">Proxmox</a></li>
 <li><a href="https://github.com/YunoHost-Apps/archivebox_ynh">Yunohost</a></li>
@@ -431,7 +426,7 @@ For more discussion on third-party hosting options see here: <a href="https://gi
 #### ➡️&nbsp; Next Steps
 
 - Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
-- (Optional) [Import browser cookies and settings into a persona](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile).
+- (Optional) Create a persona and import browser cookies to archive logged-in sites: `archivebox persona create --import=chrome personal`
 - Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
 - Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
 - Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
@@ -459,6 +454,11 @@ docker compose run --rm archivebox help
 # equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
 docker run -it -v $PWD:/data archivebox/archivebox:dev help
 
+# optional: import your browser cookies into a persona for logged-in archiving
+archivebox persona create --import=chrome personal
+# supported: chrome/chromium/brave/edge (Chromium-based only)
+# use --profile to target a specific profile (e.g. Default, Profile 1)
+# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
 ```
 
 #### ArchiveBox Subcommands
@@ -538,16 +538,16 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html'  # inspect snapshot da
 <summary><b>🖥&nbsp; Web UI & API Usage</b></summary>
 <pre lang="bash"><code style="white-space: pre-line">
 # Start the server on bare metal (uv/apt/brew):
-archivebox server 0.0.0.0:5797                 # start the server
+archivebox server 0.0.0.0:8000                 # start the server
 <br/>
 # Or with Docker Compose:
 docker compose up                              # start the server
 <br/>
 # Or with a Docker container:
-docker run -v $PWD:/data -it -p 5797:5797 archivebox/archivebox:dev
+docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 
-<sup>Open <a href="http://admin.archivebox.localhost:5797/admin/"><code>http://admin.archivebox.localhost:5797/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:5797"><code>http://web.archivebox.localhost:5797</code></a> for the public UI. ➡️</sup><br/>
+<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
 <sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
 <sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
 <br/><br/>
@@ -564,22 +564,6 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
 
 # restart the server to apply any config changes
 </code></pre>
-
-Snapshot caching follows each snapshot's saved `PERMISSIONS`, including inherited defaults and snapshot overrides. Public content can be cached by browsers and shared proxies/CDNs; private and unlisted content uses `Cache-Control: private` so only the user's browser can cache it. Both retain normal artifact cache lifetimes and validators for fast repeat viewing. Authentication handoffs are not stored. Changing permissions does not revoke copies already cached under the previous policy; purge any affected shared cache when needed.
-</details>
-
-<br/>
-
-<details>
-<summary><b><img src="https://app.archivebox.io/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
-<br/>
-
-- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://app.archivebox.io/)
-- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://app.archivebox.io/)
-- 🧩 [ArchiveBox Browser Extension](https://extension.archivebox.io/)
-- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://electron.archivebox.io/)
-
-<br/>
 </details>
 
 <br/>
@@ -613,7 +597,7 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 <sub>. . . . . . . . . . . . . . . . . . . . . . . . . . . .</sub>
 <br/><br/>
 <a href="https://demo.archivebox.io">DEMO: <code>https://demo.archivebox.io</code></a><br/>
-<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a> | <a href="#caveats">Caveats</a>
+<a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration">Configuration</a> | <a href="#Caveats">Caveats</a>
 <br/>
 </div>
 
@@ -635,13 +619,10 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>  
-  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
-
-- <img src="https://app.archivebox.io/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://app.archivebox.io/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
-  <i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
+  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files  
-  <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#import-a-list-of-urls-from-a-text-file).</i>
+  <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> From manually exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (in Netscape format)  
   <i>Instructions: <a href="https://support.google.com/chrome/answer/96816?hl=en">Chrome</a>, <a href="https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer">Firefox</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a">Safari</a>, <a href="https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows">IE</a>, <a href="https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom">Opera</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></i>
@@ -670,7 +651,7 @@ echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox a
 # echo 'https://example.com' | docker compose run --rm -T archivebox add
 ```
 
-See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage) page for documentation and examples.
+See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
 
 It also includes a built-in scheduled import feature with `archivebox schedule`, handled by the same orchestrator that powers `archivebox server`, so you can pull in URLs from RSS feeds and websites regularly without a separate cron container.
 
@@ -811,7 +792,7 @@ Installing directly on <strong>Windows without Docker or WSL/WSL2/Cygwin is not
 
 <h4>Learn More</h4>
 <ul>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#3-install-runtime-dependencies-using-archivebox-install">Wiki: Install (Dependencies)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#dependencies">Wiki: Install (Dependencies)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install">Wiki: Chromium Install</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives">Wiki: Upgrading or Merging Archives</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#installing">Wiki: Troubleshooting (Installing)</a></li>
@@ -865,7 +846,7 @@ Each snapshot subfolder includes static metadata and plain extractor output file
 <h4>Learn More</h4>
 <ul>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage">Wiki: Setting Up Storage (SMB, NFS, S3, B2, Google Drive, etc.)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout">Wiki: Usage (Disk Layout)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Disk-Layout">Wiki: Usage (Disk Layout)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output Folder)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
@@ -951,7 +932,7 @@ archivebox manage createsuperuser
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
 </ul>
 
@@ -1004,11 +985,14 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
 
 
 <ul>
-<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/docs/Chromium-Install.md#import-an-existing-browser-profile">importing a chrome profile into a persona</a></li>
+<li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
+<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
 <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
 <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
 </ul>
 
+In the future we plan on adding support for running JS scripts during archiving to block ads, cookie popups, modals, and fix other issues. Follow here for progress: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/51">Issue #51</a>.
+
 </details>
 <br/>
 
@@ -1057,15 +1041,15 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 </li>
 <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
 </li>
-<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
-<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
+<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
+<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
 </li>
 </ul>
 
 <h4>Learn More</h4>
 
 <ul>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout">Wiki: Usage (Disk Layout)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Disk-Layout">Wiki: Usage (Disk Layout)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output-Folder)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#run-archivebox-as-an-unprivileged-user">Wiki: Security Overview (Privilege Dropping)</a></li>
@@ -1190,7 +1174,7 @@ By having users store their content locally or within their organizations, we ca
 
 <h3>Comparison With Other Self-Hosted Archiving Options</h3>
 
-ArchiveBox differentiates itself from [similar self-hosted projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects) by providing both a comprehensive CLI interface for managing your archive, a Web UI that can be used either independently or together with the CLI, and a simple on-disk data format that can be used without either.
+ArchiveBox differentiates itself from [similar self-hosted projects](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects) by providing both a comprehensive CLI interface for managing your archive, a Web UI that can be used either independently or together with the CLI, and a simple on-disk data format that can be used without either.
 
 
 *If you want better fidelity for very complex interactive pages with heavy JS/streams/API requests, check out [ArchiveWeb.page](https://archiveweb.page) and [ReplayWeb.page](https://replayweb.page).*
@@ -1199,7 +1183,7 @@ ArchiveBox differentiates itself from [similar self-hosted projects](https://git
 
 *If you need more advanced recursive spider/crawling ability beyond `--depth=1`, check out [Browsertrix](https://github.com/webrecorder/browsertrix-crawler), [Photon](https://github.com/s0md3v/Photon), or [Scrapy](https://scrapy.org/) and pipe the outputted URLs into ArchiveBox.*
 
-For more alternatives, see our [list here](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#web-archiving-projects)...
+For more alternatives, see our [list here](https://github.com/ArchiveBox/ArchiveBox/wiki/Web-Archiving-Community#Web-Archiving-Projects)...
 
 ArchiveBox is neither the highest fidelity nor the simplest tool available for self-hosted archiving, rather it's a jack-of-all-trades that tries to do most things well by default. We encourage you to try these other tools made by our friends if ArchiveBox isn't suited to your needs.
 
@@ -1254,7 +1238,7 @@ We use the [ArchiveBox GitHub Wiki](https://github.com/ArchiveBox/ArchiveBox/wik
 - [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage)
 - [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration)
 - [Supported Sources](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive)
-- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox#output-formats)
+- [Supported Outputs](https://github.com/ArchiveBox/ArchiveBox/wiki#can-save-these-things-for-each-site)
 - [Scheduled Archiving](https://github.com/ArchiveBox/ArchiveBox/wiki/Scheduled-Archiving)
 
 ## Advanced
@@ -1339,10 +1323,10 @@ archivebox init
 archivebox install         # detect and install all extractor dependencies
 
 # Run the development server w/ autoreloading (but no bg workers)
-archivebox server --debug --reload 0.0.0.0:5797
+archivebox server --debug --reload 0.0.0.0:8000
 
 # Run the production server (with bg workers but no autoreloading)
-archivebox server 0.0.0.0:5797
+archivebox server 0.0.0.0:8000
 ```
 
 #### 2. Option B: Build the docker container and use that for development instead
@@ -1354,10 +1338,10 @@ archivebox server 0.0.0.0:5797
 ./bin/build_docker.sh dev
 
 # Run the development server w/ autoreloading (but no bg workers)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:5797
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
 
 # Run the production server (with bg workers but no autoreloading)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 5797:5797 archivebox/archivebox:dev server --init
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
 
 # (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
 # When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
@@ -1379,11 +1363,11 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
 archivebox config --set DEBUG=True
 
 # OR you can run a dev server with DEBUG=True in a few ways:
-archivebox server --debug --reload 0.0.0.0:5797
+archivebox server --debug --reload 0.0.0.0:8000
 # or
-archivebox server --debug 0.0.0.0:5797
+archivebox server --debug 0.0.0.0:8000
 # or
-env DEBUG=True daphne -b 0.0.0.0 -p 5797 archivebox.core.asgi:application
+env DEBUG=True daphne -b 0.0.0.0 -p 8000 archivebox.core.asgi:application
 ```
 
 https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-django-is-running
@@ -1418,7 +1402,7 @@ services:
 
 # or with plain Docker:
 docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
-docker run -it -v $PWD:/data -p 5797:5797 archivebox:dev
+docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
 
 # or with uv:
 uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'

archivebox/api/apps.py

@@ -7,11 +7,6 @@ class APIConfig(AppConfig):
     name = "archivebox.api"
     label = "api"
 
-    def ready(self):
-        from archivebox.api.webhooks import register_delete_webhook_field_loader
-
-        register_delete_webhook_field_loader()
-
 
 def register_admin(admin_site):
     from archivebox.api.admin import register_admin

archivebox/api/v1_api.py

@@ -2,8 +2,6 @@ __package__ = "archivebox.api"
 
 
 from io import StringIO
-from functools import lru_cache
-from typing import Any
 from traceback import format_exception
 from contextlib import redirect_stdout, redirect_stderr
 
@@ -12,7 +10,6 @@ from django.core.exceptions import ObjectDoesNotExist, EmptyResultSet, Permissio
 from django.contrib.auth.models import User
 
 from ninja import NinjaAPI, Swagger
-from ninja.openapi.schema import OpenAPISchema
 
 # TODO: explore adding https://eadwincode.github.io/django-ninja-extra/
 
@@ -52,16 +49,6 @@ def register_urls(api: NinjaAPI) -> NinjaAPI:
 
 
 class NinjaAPIWithIOCapture(NinjaAPI):
-    def get_openapi_schema(self, *, path_prefix: str | None = None, path_params: dict[str, Any] | None = None) -> OpenAPISchema:
-        prefix = path_prefix if path_prefix is not None else self.get_root_path(path_params or {})
-        return self._schema_for_prefix(prefix)
-
-    @lru_cache(maxsize=8)
-    def _schema_for_prefix(self, path_prefix: str) -> OpenAPISchema:
-        # Routes are registered at startup. Server discovery can reuse their
-        # schema while preserving distinct URL prefixes for mounted instances.
-        return super().get_openapi_schema(path_prefix=path_prefix)
-
     def create_temporal_response(self, request: HttpRequest) -> HttpResponse:
         stdout, stderr = StringIO(), StringIO()
 

archivebox/api/v1_auth.py

@@ -4,7 +4,7 @@ from django.http import HttpRequest
 
 from ninja import Router, Schema
 
-from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token, HeaderTokenAuth, BearerTokenAuth
+from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
 
 
 router = Router(tags=["Authentication"], auth=None)
@@ -63,44 +63,3 @@ def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
         return {"success": True, "user_id": str(user.pk)}
 
     return {"success": False, "user_id": None}
-
-
-@router.post(
-    "/browser_session",
-    auth=[HeaderTokenAuth(), BearerTokenAuth()],
-    summary="Exchange an administrator API key for a normal browser session",
-)
-def browser_session(request: HttpRequest):
-    from django.conf import settings
-    from django.contrib.auth import login
-    from django.http import JsonResponse
-    from django.utils import timezone
-    from ninja.errors import HttpError
-    from archivebox.core.routes_util import get_admin_base_url
-
-    user = request.auth
-    if not user.is_active or not user.is_superuser:
-        raise HttpError(403, "An active administrator is required")
-    login(request, user, backend="django.contrib.auth.backends.ModelBackend")
-    # A browser session must not outlive the key used to establish it.
-    token = request._api_token
-    lifetime = settings.SESSION_COOKIE_AGE
-    if token.expires:
-        lifetime = min(lifetime, max(0, int((token.expires - timezone.now()).total_seconds())))
-    request.session.set_expiry(lifetime)
-    request.session.save()
-    response = JsonResponse(
-        {
-            "admin_url": get_admin_base_url(request=request).rstrip("/") + "/admin/",
-            "cookie": {
-                "name": settings.SESSION_COOKIE_NAME,
-                "value": request.session.session_key,
-                "expires": request.session.get_expiry_date().timestamp(),
-                "secure": settings.SESSION_COOKIE_SECURE,
-            },
-        },
-    )
-    # The native client installs a host-only HttpOnly cookie on the admin origin;
-    # API and admin hosts can differ, so a Set-Cookie on the API host is insufficient.
-    response["Cache-Control"] = "no-store"
-    return response

archivebox/api/v1_cli.py

@@ -133,15 +133,6 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
             validate_url(args.urls[0])
         except ValueError:
             submitted_urls = args.urls[0]
-    else:
-        submitted_urls = []
-        for url in args.urls:
-            try:
-                submitted_urls.append(validate_url(url))
-            except ValueError:
-                continue
-        if not submitted_urls:
-            raise HttpError(400, "No valid URLs were submitted")
     crawl, snapshots = add(
         urls=submitted_urls,
         snapshot_ids=args.snapshot_ids,
@@ -165,7 +156,7 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
         "crawl_id": str(crawl.id),
         "num_snapshots": len(snapshot_ids),
         "snapshot_ids": snapshot_ids,
-        "queued_urls": args.urls if isinstance(submitted_urls, str) else submitted_urls,
+        "queued_urls": args.urls,
     }
     stdout = request.__dict__.get("stdout")
     stderr = request.__dict__.get("stderr")

archivebox/api/v1_core.py

@@ -32,7 +32,7 @@ from archivebox.core.models import Snapshot, ArchiveResult, Tag
 from archivebox.core.permissions import public_snapshots_queryset
 from archivebox.api.auth import authenticated_user_from_request
 from archivebox.config.common import get_config
-from archivebox.core.routes_util import build_snapshot_detail_url, build_web_url
+from archivebox.core.routes_util import build_web_url
 from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url_length
 from archivebox.core.tag_util import (
     add_snapshot_counts,
@@ -211,22 +211,11 @@ class ArchiveResultFilterSchema(FilterSchema):
     created_at__lt: Annotated[datetime | None, FilterLookup("created_at__lt")] = None
 
 
-def _with_archiveresult_processes(queryset):
-    # cmd/pwd/version delegate to Process and Binary. Without this join a full
-    # capture's metadata response performs two extra indexed lookups per hook
-    # (roughly 100 queries), despite the initial Snapshot UUID lookup being fast.
-    # Logs and the hydrated hook environment are not response fields; loading
-    # those potentially large values would replace round trips with excess I/O.
-    return queryset.select_related("process__binary").defer("process__stdout", "process__stderr", "process__env")
-
-
 @router.get("/archiveresults", response=list[ArchiveResultSchema], url_name="get_archiveresult")
 @paginate(CustomPagination)
 def get_archiveresults(request: HttpRequest, filters: Query[ArchiveResultFilterSchema]):
     """List all ArchiveResult entries matching these filters."""
-    queryset = filters.filter(
-        _with_archiveresult_processes(ArchiveResult.objects.all()).prefetch_related("snapshot__crawl__created_by", "snapshot__tags"),
-    )
+    queryset = filters.filter(ArchiveResult.objects.all())
     if filters.search or filters.snapshot_tag:
         return queryset.distinct()
     return queryset
@@ -234,36 +223,31 @@ def get_archiveresults(request: HttpRequest, filters: Query[ArchiveResultFilterS
 
 def _uuid_ref_query(field_name: str, ref: str) -> Q:
     raw_ref = str(ref or "").strip()
+    query = Q(**{f"{field_name}__startswith": raw_ref})
+    if raw_ref:
+        query |= Q(**{f"{field_name}__icontains": raw_ref})
     try:
         parsed_uuid = UUID(raw_ref)
     except (TypeError, ValueError):
-        query = Q(**{f"{field_name}__startswith": raw_ref})
-        if raw_ref:
-            query |= Q(**{f"{field_name}__icontains": raw_ref})
         normalized_ref = raw_ref.replace("-", "")
         if normalized_ref and normalized_ref != raw_ref:
             query |= Q(**{f"{field_name}__startswith": normalized_ref})
             query |= Q(**{f"{field_name}__icontains": normalized_ref})
-        return query
-    # An exact UUID cannot need a substring fallback. OR-ing it with LIKE
-    # prevents an indexed lookup and made Cabbage metadata reads take seconds.
-    return Q(**{field_name: parsed_uuid})
+    else:
+        query |= Q(**{field_name: parsed_uuid})
+        query |= Q(**{f"{field_name}__startswith": parsed_uuid.hex})
+    return query
 
 
 @router.get("/archiveresult/{archiveresult_id}", response=ArchiveResultSchema, url_name="get_archiveresult")
 def get_archiveresult(request: HttpRequest, archiveresult_id: str):
     """Get a specific ArchiveResult by id."""
-    return (
-        _with_archiveresult_processes(ArchiveResult.objects.all())
-        .select_related("snapshot__crawl__created_by")
-        .prefetch_related("snapshot__tags")
-        .get(_uuid_ref_query("id", archiveresult_id))
-    )
+    return ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
 
 
 def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
     normalized = str(plugin or "").strip().strip("/")
-    if normalized in (".", "..") or not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
+    if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
         raise HttpError(400, "Invalid ArchiveResult plugin name")
     return normalized
 
@@ -366,12 +350,6 @@ def _summarize_archiveresult_output_files(output_files: dict[str, dict[str, Any]
 
 def _get_snapshot_by_ref(snapshot_id: str):
     queryset = Snapshot.objects.select_related("crawl__created_by")
-    try:
-        full_id = UUID(snapshot_id)
-    except (TypeError, ValueError):
-        pass
-    else:
-        return queryset.get(pk=full_id)
     try:
         return queryset.get(_uuid_ref_query("id", snapshot_id) | Q(timestamp__startswith=snapshot_id))
     except Snapshot.DoesNotExist:
@@ -408,19 +386,18 @@ def _write_archiveresult_files(
     existing_output_files: dict[str, dict[str, Any]] | None = None,
     allow_empty: bool = False,
 ) -> dict[str, dict[str, Any]]:
-    plugin_name = _normalize_uploaded_archiveresult_plugin(plugin_name)
     files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
     output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
     mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
     chunk_output_path = _get_archiveresult_upload_form_value(request, "chunk_output_path")
 
-    output_files = dict(existing_output_files or {})
-    if not files:
-        return output_files
-
     snapshot_dir = snapshot.output_dir
     plugin_dir = snapshot_dir / plugin_name
     storage = FileSystemStorage(location=str(plugin_dir))
+    output_files = dict(existing_output_files or {})
+
+    if not files:
+        return output_files
 
     if chunk_output_path:
         if len(files) != 1:
@@ -652,8 +629,6 @@ def patch_archiveresult(
 class SnapshotSchema(Schema):
     TYPE: str = "core.models.Snapshot"
     id: UUID
-    crawl_id: UUID
-    persona: str | None
     created_by_id: str
     created_by_username: str
     created_at: datetime
@@ -672,17 +647,6 @@ class SnapshotSchema(Schema):
     num_archiveresults: int
     archiveresults: list[MinimalArchiveResultSchema]
 
-    @staticmethod
-    def resolve_archive_path(obj):
-        return obj.archive_path_from_db
-
-    @staticmethod
-    def resolve_persona(obj):
-        # A deduplicated submission can reuse a capture from a different
-        # persona. Report its actual DB-backed owner, not the new request's choice.
-        persona = obj.crawl.resolve_persona()
-        return persona.name if persona else None
-
     @staticmethod
     def resolve_created_by_id(obj):
         return str(obj.created_by.pk)
@@ -705,14 +669,12 @@ class SnapshotSchema(Schema):
 
     @staticmethod
     def resolve_num_archiveresults(obj, context):
-        # This reverse FK contains each result once. DISTINCT would materialize
-        # wide JSON output columns just to count rows through snapshot_id's index.
-        return obj.archiveresult_set.count()
+        return obj.archiveresult_set.all().distinct().count()
 
     @staticmethod
     def resolve_archiveresults(obj, context):
         if bool(context["request"].__dict__.get("with_archiveresults", False)):
-            return _with_archiveresult_processes(obj.archiveresult_set.all())
+            return obj.archiveresult_set.all().distinct()
         return ArchiveResult.objects.none()
 
 
@@ -737,7 +699,6 @@ class SnapshotDeleteResponseSchema(Schema):
     snapshot_id: str
     crawl_id: str
     deleted_count: int
-    queued_count: int
 
 
 def normalize_tag_list(tags: list[str] | None = None) -> list[str]:
@@ -838,7 +799,7 @@ def _snapshots_rss_response(
     )
 
     for snapshot in snapshots:
-        archived_url = build_snapshot_detail_url(snapshot.archive_path_from_db, request=request)
+        archived_url = build_web_url(f"/{snapshot.archive_path_from_db}", request=request)
         tags = [tag.name for tag in snapshot.tags.all()]
         crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
         description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
@@ -976,15 +937,6 @@ def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
     # lock. The unique insert recovery and CAS update below are the request-side
     # coordination boundary for this idempotent metadata sync.
     snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
-    if snapshot is None and bool(crawl.get_current_config().get("ONLY_NEW", True)):
-        # Uploading browser metadata is part of the same submission as /add.
-        # It must honor the runner's deduplication policy, even if it arrives
-        # before the runner has processed that crawl. Report the actual owner
-        # so clients never mistake a reused capture for one they just created.
-        snapshot = Snapshot.objects.filter(url=data.url).exclude(status=Snapshot.DELETING_STATE).order_by("-created_at").first()
-        if snapshot is not None:
-            setattr(request, "with_archiveresults", False)
-            return snapshot
     if snapshot is None:
         try:
             snapshot = Snapshot.objects.create(
@@ -1018,6 +970,11 @@ def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
             created_by=request.user if isinstance(request.user, User) else None,
         )
 
+    try:
+        snapshot.ensure_crawl_symlink()
+    except Exception:
+        pass
+
     setattr(request, "with_archiveresults", False)
     return snapshot
 
@@ -1079,23 +1036,22 @@ def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateS
 
 @router.delete("/snapshot/{snapshot_id}", response=SnapshotDeleteResponseSchema, url_name="delete_snapshot")
 def delete_snapshot(request: HttpRequest, snapshot_id: str):
-    """Queue deletion; the normal server runner stops writers and removes files.
-
-    The snapshot remains readable with status=deleting until cleanup succeeds.
-    queued_count reports acceptance; deleted_count is zero in this response.
-    """
     snapshot = get_snapshot(request, snapshot_id, with_archiveresults=False)
     snapshot_id_str = str(snapshot.id)
-    crawl_id_str = str(snapshot.crawl_id)
-    # Use the same durable queue as the admin delete action. Stopping writers
-    # and removing archive files belongs to the runner, never the HTTP request.
-    queued_count = Snapshot.objects.filter(pk=snapshot.pk).request_delete()
+    crawl_id_str = str(snapshot.crawl.pk)
+    snapshot.cancel()
+
+    from archivebox.services.runner import run_pending_crawls
+
+    with crawl_lifecycle_lock(crawl_id_str):
+        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
+        snapshot = get_snapshot(request, snapshot_id_str, with_archiveresults=False)
+        deleted_count, _ = snapshot.delete()
     return {
         "success": True,
         "snapshot_id": snapshot_id_str,
         "crawl_id": crawl_id_str,
-        "deleted_count": 0,
-        "queued_count": queued_count,
+        "deleted_count": deleted_count,
     }
 
 

archivebox/api/v1_crawls.py

@@ -1,5 +1,6 @@
 __package__ = "archivebox.api"
 
+import json
 from pathlib import Path
 from uuid import UUID
 from datetime import datetime
@@ -22,7 +23,7 @@ from archivebox.core.permissions import (
 )
 from archivebox.config.common import get_config
 from archivebox.crawls.models import Crawl
-from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url
+from archivebox.misc.util import filter_queryset_by_uuid_substring
 
 from .auth import API_AUTH_METHODS, authenticated_user_from_request
 
@@ -112,10 +113,7 @@ def get_crawls(request: HttpRequest):
 
 @router.post("/crawls", response=CrawlSchema, url_name="create_crawl")
 def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
-    try:
-        urls = [validate_url(url) for url in data.urls if url and url.strip()]
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
+    urls = [url.strip() for url in data.urls if url and url.strip()]
     if not urls:
         raise HttpError(400, "At least one URL is required")
     if data.max_depth not in (0, 1, 2, 3, 4):
@@ -123,10 +121,9 @@ def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
 
     tags = normalize_tag_list(data.tags, data.tags_str)
     config = dict(data.config or {})
-    config.setdefault("PARSER", "url_list")
     config.setdefault("PERMISSIONS", str(get_config().PERMISSIONS))
     crawl = Crawl.objects.create(
-        urls="\n".join(urls),
+        urls="\n".join(json.dumps({"type": "CrawlSeed", "url": url, "depth": 0}, separators=(",", ":")) for url in urls),
         max_depth=data.max_depth,
         tags_str=",".join(tags),
         label=data.label,
@@ -182,14 +179,6 @@ def crawl_file(request: HttpRequest, crawl_id: str, path: str):
     if not file_path.is_file() or crawl_root not in file_path.parents:
         raise HttpError(404, "Crawl file not found")
 
-    if path == "chrome_screencast/latest.jpg" and request.GET.get("after"):
-        try:
-            started_at = float(request.GET["after"])
-        except ValueError:
-            raise HttpError(400, "Invalid screencast start time")
-        if file_path.stat().st_mtime < started_at:
-            raise HttpError(404, "No frame for this capture yet")
-
     response = FileResponse(file_path.open("rb"))
     response["Cache-Control"] = "no-store, no-cache, max-age=0, must-revalidate"
     response["Pragma"] = "no-cache"

archivebox/api/v1_personas.py

@@ -2,14 +2,12 @@ __package__ = "archivebox.api"
 
 import json
 from datetime import datetime
-from typing import Any, Literal
+from typing import Any
 from uuid import UUID
 
-from django.core.exceptions import ValidationError
 from django.db.models import Q
 from django.http import HttpRequest
 from ninja import Router, Schema
-from ninja.errors import HttpError
 from ninja.pagination import paginate
 from pydantic import Field
 
@@ -25,7 +23,6 @@ class PersonaBrowserSettingsSchema(Schema):
     user_agent: str = ""
     viewport_size: str = ""
     viewport_device_scale_factor: float | None = None
-    color_scheme: Literal["", "light", "dark"] = ""
     language: str = ""
     timezone: str = ""
     geolocation: dict[str, Any] | None = None
@@ -97,8 +94,6 @@ def browser_settings_to_config(extension_persona_id: str, settings: PersonaBrows
         )
     if settings.viewport_device_scale_factor is not None:
         config["BROWSER_DEVICE_SCALE_FACTOR"] = settings.viewport_device_scale_factor
-    if "color_scheme" in settings.model_fields_set:
-        config["BROWSER_COLOR_SCHEME"] = settings.color_scheme
     if settings.language:
         config["BROWSER_LANGUAGE"] = settings.language
     if settings.timezone:
@@ -110,9 +105,10 @@ def browser_settings_to_config(extension_persona_id: str, settings: PersonaBrows
 
 
 def find_persona(extension_persona_id: str, name: str) -> Persona | None:
-    named = Persona.find_named(name)
     return (
-        Persona.objects.filter(Q(config__BROWSER_EXTENSION_PERSONA_ID=extension_persona_id) | Q(pk=named.pk if named else None))
+        Persona.objects.filter(
+            Q(config__BROWSER_EXTENSION_PERSONA_ID=extension_persona_id) | Q(name=name),
+        )
         .order_by("created_at")
         .first()
     )
@@ -139,11 +135,7 @@ def sync_persona(request: HttpRequest, payload: PersonaSyncSchema):
     if not is_valid:
         raise ValueError(error_message)
 
-    try:
-        persona = find_persona(payload.extension_persona_id, name)
-        (persona or Persona(name=name)).validate_name(persona.name if persona else name)
-    except ValidationError as err:
-        raise HttpError(409, "; ".join(err.messages)) from err
+    persona = find_persona(payload.extension_persona_id, name)
     created = persona is None
     if persona is None:
         persona = Persona(name=name)
@@ -158,12 +150,12 @@ def sync_persona(request: HttpRequest, payload: PersonaSyncSchema):
     persona.ensure_dirs()
 
     cookies_written = False
-    if "cookies_txt" in payload.model_fields_set:
+    if payload.cookies_txt.strip():
         (persona.path / "cookies.txt").write_text(payload.cookies_txt)
         cookies_written = True
 
     auth_written = False
-    if "auth_json" in payload.model_fields_set:
+    if payload.auth_json:
         (persona.path / "auth.json").write_text(json.dumps(payload.auth_json, indent=2, sort_keys=True) + "\n")
         auth_written = True
 

archivebox/api/webhooks.py

@@ -5,40 +5,12 @@ from collections.abc import Callable
 import logging
 
 from django.db import transaction
-from django.db.models.signals import pre_delete
 from signal_webhooks.handlers import sync_task_handler
 
 
 logger = logging.getLogger(__name__)
 
 
-def load_deferred_fields_for_delete_webhook(sender, instance, using, **kwargs) -> None:
-    """Materialize deferred fields while DELETE webhook rows still exist.
-
-    Admin lists defer large fields, but signal_webhooks serializes all concrete
-    fields in post_delete. Loading one then would query an already-deleted row
-    and drop the event. Hydrate only deferred fields on webhook-enabled models
-    here, preserving the normal list-query optimization.
-    """
-    from signal_webhooks.handlers import find_hook_handler
-    from signal_webhooks.utils import reference_for_model
-
-    ref = reference_for_model(type(instance))
-    deferred_fields = instance.get_deferred_fields()
-    if not deferred_fields or find_hook_handler(ref, "DELETE") is None:
-        return
-
-    instance.refresh_from_db(using=using, fields=deferred_fields)
-
-
-def register_delete_webhook_field_loader() -> None:
-    pre_delete.connect(
-        load_deferred_fields_for_delete_webhook,
-        dispatch_uid="archivebox.load_deferred_fields_for_delete_webhook",
-        weak=False,
-    )
-
-
 def warning_error_handler(hook: Any, error: Exception | None) -> None:
     if error is not None:
         logger.warning("Outbound webhook %r failed: %s", hook.name, error)

archivebox/auth/__init__.py

@@ -0,0 +1,24 @@
+"""
+django-allauth integration for ArchiveBox.
+
+Provides social login, email/password auth, and configurable user registration
+as a replacement for the default admin-only login flow.
+
+Only activates if ALLAUTH_ENABLED=True in config and django-allauth is installed.
+
+To install allauth support:
+    pip install archivebox[allauth]
+    # or
+    uv add "django-allauth[socialaccount]>=65.0"
+
+To configure social providers, set SOCIALACCOUNT_PROVIDERS in ArchiveBox.conf:
+    [ALLAUTH_CONFIG]
+    ALLAUTH_ENABLED = true
+    SOCIALACCOUNT_ENABLED = true
+    REGISTRATION_MODE = open     # open | invite | approval
+    EMAIL_VERIFICATION = none    # none | optional | mandatory
+
+    SOCIALACCOUNT_PROVIDERS = {"github": {"APP": {"client_id": "...", "secret": "..."}}}
+"""
+
+__package__ = "archivebox.auth"

archivebox/auth/adapters.py

@@ -0,0 +1,73 @@
+__package__ = "archivebox.auth"
+
+import importlib
+
+try:
+    DefaultAccountAdapter = importlib.import_module("allauth.account.adapter").DefaultAccountAdapter
+    DefaultSocialAccountAdapter = importlib.import_module("allauth.socialaccount.adapter").DefaultSocialAccountAdapter
+except (ImportError, RuntimeError):
+
+    class DefaultAccountAdapter:  # type: ignore[no-redef]
+        """Stub when django-allauth is not installed."""
+
+        pass
+
+    class DefaultSocialAccountAdapter:  # type: ignore[no-redef]
+        """Stub when django-allauth is not installed."""
+
+        pass
+
+
+def _get_registration_enabled() -> bool:
+    from archivebox.config.common import get_config
+
+    return get_config().REGISTRATION_ENABLED
+
+
+def _get_registration_mode() -> str:
+    from archivebox.config.common import get_config
+
+    return get_config().REGISTRATION_MODE
+
+
+class ArchiveBoxAccountAdapter(DefaultAccountAdapter):
+    """
+    Custom account adapter that enforces ArchiveBox registration policy:
+    - REGISTRATION_ENABLED=False → signup form is closed entirely
+    - REGISTRATION_MODE="invite" → signup form is closed (invite link required)
+    - REGISTRATION_MODE="approval" → user is saved inactive, pending admin activation
+    """
+
+    def is_open_for_signup(self, request):
+        if not _get_registration_enabled():
+            return False
+        if _get_registration_mode() == "invite":
+            return False
+        return True
+
+    def save_user(self, request, user, form, commit=True):
+        user = super().save_user(request, user, form, commit=False)
+        if _get_registration_mode() == "approval":
+            user.is_active = False
+        if commit:
+            user.save()
+        return user
+
+
+class ArchiveBoxSocialAccountAdapter(DefaultSocialAccountAdapter):
+    """
+    Custom social account adapter that enforces the same registration policy
+    as ArchiveBoxAccountAdapter for social logins.
+
+    Auto-connecting social accounts by verified email is controlled by the
+    django-allauth setting SOCIALACCOUNT_EMAIL_AUTHENTICATION_AUTO_CONNECT,
+    which ArchiveBox exposes as SOCIALACCOUNT_EMAIL_AUTO_CONNECT (default: False).
+    See archivebox/config/allauth.py for the security rationale.
+    """
+
+    def is_open_for_signup(self, request, sociallogin):
+        if not _get_registration_enabled():
+            return False
+        if _get_registration_mode() == "invite":
+            return False
+        return True

archivebox/auth/apps.py

@@ -0,0 +1,13 @@
+__package__ = "archivebox.auth"
+
+from django.apps import AppConfig
+
+
+class AuthConfig(AppConfig):
+    name = "archivebox.auth"
+    label = "archivebox_auth"
+    verbose_name = "ArchiveBox Auth"
+
+    def ready(self):
+        # Connect signal handlers
+        from archivebox.auth import signals  # noqa: F401

archivebox/auth/signals.py

@@ -0,0 +1,34 @@
+__package__ = "archivebox.auth"
+
+from django.dispatch import receiver
+
+
+def _get_default_permissions() -> str:
+    from archivebox.config.common import get_config
+
+    return get_config().DEFAULT_USER_PERMISSIONS
+
+
+def _apply_default_permissions(user) -> None:
+    """Apply DEFAULT_USER_PERMISSIONS to a newly signed-up user."""
+    level = _get_default_permissions()
+
+    if level == "none":
+        return
+
+    if level == "admin":
+        user.is_staff = True
+        user.is_superuser = True
+        user.save(update_fields=["is_staff", "is_superuser"])
+        return
+
+
+try:
+    from allauth.account.signals import user_signed_up
+
+    @receiver(user_signed_up)
+    def on_user_signed_up(sender, request, user, **kwargs):
+        _apply_default_permissions(user)
+
+except ImportError:
+    pass

archivebox/auth/templates/account/email_confirm.html

@@ -0,0 +1,19 @@
+{% extends "admin/base_site.html" %}
+{% load i18n %}
+{% block content_title %}<center>{% trans "Confirm your email address." %}</center><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+{% block content %}
+<div id="content-main">
+{% if confirmation %}
+<p>{% blocktrans with email=confirmation.email_address.email %}Please confirm that <strong>{{ email }}</strong> is your email address by clicking below.{% endblocktrans %}</p>
+<form method="POST" action="{% url 'account_confirm_email' confirmation.key %}">
+  {% csrf_token %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Confirm email' %}">
+  </div>
+</form>
+{% else %}
+<p class="errornote">{% trans "This email confirmation link is invalid or has expired." %}</p>
+{% endif %}
+</div>
+{% endblock %}

archivebox/auth/templates/account/login.html

@@ -0,0 +1,64 @@
+{% extends "admin/base_site.html" %}
+{% load i18n static allauth account socialaccount %}
+
+{% block extrastyle %}{{ block.super }}
+<link rel="stylesheet" type="text/css" href="{% static "admin/css/login.css" %}">
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} login{% endblock %}
+
+<a href="{% url 'Home' %}">
+{% block branding %}<h1>ArchiveBox</h1>{% endblock %}
+</a>
+{% block usertools %}{% endblock %}
+{% block nav-global %}{% endblock %}
+
+{% block content_title %}
+  <center>Log in to your archive.</center><br/><br/>
+  <img src="{% static 'archive.png' %}" style="width: 80px; display: block; margin: auto"/><br/>
+{% endblock %}
+
+{% block breadcrumbs %}{% endblock %}
+
+{% block content %}
+<div id="content-main">
+
+{% if form.errors %}
+<p class="errornote">{% trans "Please correct the error below." %}</p>
+{% endif %}
+{{ form.non_field_errors }}
+
+<form method="POST" action="{% url 'account_login' %}">
+  {% csrf_token %}
+  {% for field in form %}
+  <div class="form-row">
+    {{ field.errors }}
+    {{ field.label_tag }}{{ field }}
+  </div>
+  {% endfor %}
+  {% if redirect_field_value %}
+  <input type="hidden" name="{{ redirect_field_name }}" value="{{ redirect_field_value }}">
+  {% endif %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Log in' %}">
+  </div>
+</form>
+
+{% get_providers as socialaccount_providers %}
+{% if socialaccount_providers %}
+<hr/>
+<center><strong>{% trans "Or sign in with:" %}</strong></center><br/>
+{% include "socialaccount/snippets/provider_list.html" with process="login" %}
+{% endif %}
+
+{% if ACCOUNT_ALLOW_REGISTRATION %}
+<hr/>
+<center>
+  <a href="{% url 'account_signup' %}">{% trans "Create an account" %}</a>
+  &nbsp;|&nbsp;
+  <a href="{% url 'account_reset_password' %}">{% trans "Forgot your password?" %}</a>
+</center>
+{% endif %}
+
+</div>
+{% endblock %}

archivebox/auth/templates/account/password_reset.html

@@ -0,0 +1,28 @@
+{% extends "admin/base_site.html" %}
+{% load i18n static %}
+
+{% block extrastyle %}{{ block.super }}
+<link rel="stylesheet" type="text/css" href="{% static "admin/css/login.css" %}">
+{% endblock %}
+{% block bodyclass %}{{ block.super }} login{% endblock %}
+<a href="{% url 'Home' %}">{% block branding %}<h1>ArchiveBox</h1>{% endblock %}</a>
+{% block usertools %}{% endblock %}
+{% block nav-global %}{% endblock %}
+{% block content_title %}<center>Reset your password.</center><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+
+{% block content %}
+<div id="content-main">
+<form method="POST" action="{% url 'account_reset_password' %}">
+  {% csrf_token %}
+  {% for field in form %}
+  <div class="form-row">{{ field.errors }}{{ field.label_tag }}{{ field }}</div>
+  {% endfor %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Send reset link' %}">
+  </div>
+</form>
+<hr/>
+<center><a href="{% url 'account_login' %}">{% trans "Back to login" %}</a></center>
+</div>
+{% endblock %}

archivebox/auth/templates/account/password_reset_from_key.html

@@ -0,0 +1,31 @@
+{% extends "admin/base_site.html" %}
+{% load i18n static %}
+
+{% block extrastyle %}{{ block.super }}
+<link rel="stylesheet" type="text/css" href="{% static "admin/css/login.css" %}">
+{% endblock %}
+{% block bodyclass %}{{ block.super }} login{% endblock %}
+<a href="{% url 'Home' %}">{% block branding %}<h1>ArchiveBox</h1>{% endblock %}</a>
+{% block usertools %}{% endblock %}
+{% block nav-global %}{% endblock %}
+{% block content_title %}<center>Set a new password.</center><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+
+{% block content %}
+<div id="content-main">
+{% if token_fail %}
+<p class="errornote">{% trans "This password reset link is invalid or has expired. Please request a new one." %}</p>
+<center><a href="{% url 'account_reset_password' %}">{% trans "Request new link" %}</a></center>
+{% else %}
+<form method="POST">
+  {% csrf_token %}
+  {% for field in form %}
+  <div class="form-row">{{ field.errors }}{{ field.label_tag }}{{ field }}</div>
+  {% endfor %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Set new password' %}">
+  </div>
+</form>
+{% endif %}
+</div>
+{% endblock %}

archivebox/auth/templates/account/signup.html

@@ -0,0 +1,40 @@
+{% extends "admin/base_site.html" %}
+{% load i18n static %}
+
+{% block extrastyle %}{{ block.super }}
+<link rel="stylesheet" type="text/css" href="{% static "admin/css/login.css" %}">
+{% endblock %}
+
+{% block bodyclass %}{{ block.super }} login{% endblock %}
+
+<a href="{% url 'Home' %}">{% block branding %}<h1>ArchiveBox</h1>{% endblock %}</a>
+{% block usertools %}{% endblock %}
+{% block nav-global %}{% endblock %}
+{% block content_title %}<center>Create an account.</center><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+
+{% block content %}
+<div id="content-main">
+{% if form.errors %}
+<p class="errornote">{% trans "Please correct the error below." %}</p>
+{% endif %}
+{{ form.non_field_errors }}
+<form method="POST" action="{% url 'account_signup' %}">
+  {% csrf_token %}
+  {% for field in form %}
+  <div class="form-row">
+    {{ field.errors }}
+    {{ field.label_tag }}{{ field }}
+  </div>
+  {% endfor %}
+  {% if redirect_field_value %}
+  <input type="hidden" name="{{ redirect_field_name }}" value="{{ redirect_field_value }}">
+  {% endif %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Sign up' %}">
+  </div>
+</form>
+<hr/>
+<center><a href="{% url 'account_login' %}">{% trans "Already have an account? Log in" %}</a></center>
+</div>
+{% endblock %}

archivebox/auth/templates/socialaccount/connections.html

@@ -0,0 +1,26 @@
+{% extends "admin/base_site.html" %}
+{% load i18n %}
+{% block content_title %}<div style="text-align: center;">{% trans "Connected accounts." %}</div><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+{% block content %}
+<div id="content-main">
+{% if socialaccounts %}
+<p>{% trans "The following social accounts are connected to your account:" %}</p>
+<ul>
+{% for account in socialaccounts %}
+<li>{{ account.get_provider_display }}: {{ account.uid }}
+  <form method="POST" action="{% url 'socialaccount_connections' %}">
+    {% csrf_token %}
+    <input type="hidden" name="account" value="{{ account.id }}">
+    <button type="submit">{% trans "Disconnect" %}</button>
+  </form>
+</li>
+{% endfor %}
+</ul>
+{% else %}
+<p>{% trans "No social accounts connected." %}</p>
+{% endif %}
+<hr/>
+<a href="{% url 'account_login' %}">{% trans "Back" %}</a>
+</div>
+{% endblock %}

archivebox/auth/templates/socialaccount/signup.html

@@ -0,0 +1,28 @@
+{% extends "admin/base_site.html" %}
+{% load i18n static %}
+
+{% block extrastyle %}{{ block.super }}
+<link rel="stylesheet" type="text/css" href="{% static "admin/css/login.css" %}">
+{% endblock %}
+{% block bodyclass %}{{ block.super }} login{% endblock %}
+<a href="{% url 'Home' %}">{% block branding %}<h1>ArchiveBox</h1>{% endblock %}</a>
+{% block usertools %}{% endblock %}
+{% block nav-global %}{% endblock %}
+{% block content_title %}<center>Complete your sign-up.</center><br/>{% endblock %}
+{% block breadcrumbs %}{% endblock %}
+
+{% block content %}
+<div id="content-main">
+<p>{% blocktrans with provider=sociallogin.account.get_provider.name %}You are signing in with <strong>{{ provider }}</strong>. Please confirm your details below.{% endblocktrans %}</p>
+<form method="POST" action="{% url 'socialaccount_signup' %}">
+  {% csrf_token %}
+  {{ redirect_field }}
+  {% for field in form %}
+  <div class="form-row">{{ field.errors }}{{ field.label_tag }}{{ field }}</div>
+  {% endfor %}
+  <div class="submit-row" style="border: none;">
+    <label>&nbsp;</label><input type="submit" value="{% trans 'Continue' %}">
+  </div>
+</form>
+</div>
+{% endblock %}

archivebox/base_models/models.py

@@ -9,7 +9,6 @@ from typing import Any
 from archivebox.uuid_compat import CompactUUIDField, uuid7
 from pathlib import Path
 
-from django.core.exceptions import FieldDoesNotExist
 from django.db import models
 from django.db.models import F
 from django.db import transaction
@@ -24,110 +23,6 @@ from django_stubs_ext.db.models import TypedModelMeta
 from archivebox.config import CONSTANTS
 
 
-class PagedQuerySet(models.QuerySet):
-    def paged_iterator(self, chunk_size: int = 500):
-        """
-        Iterate rows using bounded keyset pages instead of one streaming cursor.
-
-        Django's iterator(chunk_size=...) still keeps a single SQLite SELECT
-        cursor open until the full queryset is exhausted. That is fine for
-        read-only exports, but update/migration code does filesystem work and
-        writes while iterating; a long-lived read cursor there can stretch lock
-        waits across thousands of rows. This respects the queryset's existing
-        filters, order_by(), select_related(), and prefetch_related() state; if
-        no ordering is defined, it falls back to primary-key order.
-        """
-        pk_field = self.model._meta.pk.name
-        raw_ordering = tuple(self.query.order_by or self.model._meta.ordering or (pk_field,))
-
-        if any(not isinstance(term, str) or term == "?" for term in raw_ordering):
-            offset = 0
-            while True:
-                batch = list(self[offset : offset + chunk_size])
-                if not batch:
-                    break
-                yield from batch
-                offset += chunk_size
-            return
-
-        ordering = []
-        for term in raw_ordering:
-            descending = term.startswith("-")
-            field_name = term[1:] if descending else term
-            if field_name == "pk":
-                field_name = pk_field
-            ordering.append(f"-{field_name}" if descending else field_name)
-
-        ordered_field_names = [term.removeprefix("-") for term in ordering]
-        try:
-            if any(self.model._meta.get_field(field_name).null for field_name in ordered_field_names):
-                offset = 0
-                while True:
-                    batch = list(self[offset : offset + chunk_size])
-                    if not batch:
-                        break
-                    yield from batch
-                    offset += chunk_size
-                return
-        except (AttributeError, FieldDoesNotExist):
-            offset = 0
-            while True:
-                batch = list(self[offset : offset + chunk_size])
-                if not batch:
-                    break
-                yield from batch
-                offset += chunk_size
-            return
-
-        unique_field_names = {pk_field, *(field.name for field in self.model._meta.fields if field.unique)}
-        if not any(field_name in unique_field_names for field_name in ordered_field_names):
-            offset = 0
-            while True:
-                batch = list(self[offset : offset + chunk_size])
-                if not batch:
-                    break
-                yield from batch
-                offset += chunk_size
-            return
-
-        last_values = None
-        value_field_names = tuple(dict.fromkeys([*ordered_field_names, pk_field]))
-        while True:
-            batch_qs = self.order_by(*ordering)
-            if last_values is not None:
-                page_filter = models.Q()
-                for idx, term in enumerate(ordering):
-                    descending = term.startswith("-")
-                    field_name = term[1:] if descending else term
-                    prefix = {ordered_field_names[i]: last_values[i] for i in range(idx)}
-                    comparison = "lt" if descending else "gt"
-                    page_filter |= models.Q(**prefix, **{f"{field_name}__{comparison}": last_values[idx]})
-                # Anchor the OR tie-breaks with an explicit leading range so
-                # SQLite/PG can seek the existing order index instead of
-                # rescanning its prefix for every keyset page.
-                first_comparison = "lte" if ordering[0].startswith("-") else "gte"
-                batch_qs = batch_qs.filter(
-                    page_filter,
-                    **{f"{ordered_field_names[0]}__{first_comparison}": last_values[0]},
-                )
-
-            batch_rows = list(batch_qs.values_list(*value_field_names)[:chunk_size])
-            if not batch_rows:
-                break
-
-            pk_idx = value_field_names.index(pk_field)
-            row_ids = [row[pk_idx] for row in batch_rows]
-            rows_by_id = {obj.pk: obj for obj in self.filter(pk__in=row_ids).order_by()}
-
-            for row in batch_rows:
-                row_id = row[pk_idx]
-                obj = rows_by_id.get(row_id)
-                if obj is not None:
-                    yield obj
-
-            last_values = batch_rows[-1][: len(ordered_field_names)]
-
-
 def normalize_config_json_values(config: Any) -> Any:
     if not isinstance(config, dict):
         return config
@@ -296,21 +191,9 @@ class ModelWithDeleteAfter(models.Model):
         return cls.objects.none()
 
     @classmethod
-    def delete_expired(cls, *, batch_size: int = 100, backfill_missing: bool = True, backfill_cursors: dict | None = None) -> int:
+    def delete_expired(cls, *, batch_size: int = 100, backfill_missing: bool = True) -> int:
         if backfill_missing:
-            # Bound the scanned rows BEFORE evaluating JSON policy/parent
-            # joins. LIMIT on the policy query only bounded its output: finding
-            # no matches could scan the entire history with a SQLite read lock.
-            # The runner keeps a keyset cursor between idle passes, so disabled
-            # policies cannot starve older rows. Restarting safely repeats work.
-            page = cls.objects.order_by("-pk")
-            before = backfill_cursors.get(cls) if backfill_cursors is not None else None
-            if before is not None:
-                page = page.filter(pk__lt=before)
-            page_ids = list(page.values_list("pk", flat=True)[:batch_size])
-            if backfill_cursors is not None:
-                backfill_cursors[cls] = page_ids[-1] if len(page_ids) == batch_size else None
-            missing_delete_at = list(cls.missing_delete_at_candidates().filter(pk__in=page_ids).order_by())
+            missing_delete_at = list(cls.missing_delete_at_candidates().order_by("created_at", "pk")[:batch_size])
             for obj in missing_delete_at:
                 if obj.set_delete_at_from_config():
                     cls.objects.filter(pk=obj.pk, delete_at__isnull=True).update(
@@ -348,9 +231,12 @@ class ModelWithOutputDir(ModelWithUUID):
 
     _delete_signal_registered = False
 
-    # Saving metadata must not resolve or create output paths. The runner and
-    # explicit artifact writers create their own directories when writing files;
-    # on_commit would still make HTTP requests wait on remote archive storage.
+    def save(self, *args, **kwargs):
+        super().save(*args, **kwargs)
+        output_dir = Path(self.output_dir)
+        # Avoid holding SQLite write transactions open across slow filesystem work.
+        transaction.on_commit(lambda: output_dir.mkdir(parents=True, exist_ok=True))
+        # Note: index.json is deprecated, models should use write_index_jsonl() for full data
 
     @property
     def output_dir_parent(self) -> str:
@@ -396,14 +282,7 @@ class ModelWithOutputDir(ModelWithUUID):
             if path.is_symlink() or path.is_file():
                 path.unlink(missing_ok=True)
             elif path.is_dir():
-                shutil.rmtree(path)
-            # lstat also catches dangling symlinks and propagates mount and
-            # permission errors rather than mistaking them for absence.
-            try:
-                path.lstat()
-            except FileNotFoundError:
-                continue
-            raise OSError(f"Output path still exists after deletion: {path}")
+                shutil.rmtree(path, ignore_errors=True)
 
     def schedule_delete_cleanup(self, *, using: str | None = None) -> None:
         """Capture output paths before DB deletion and remove them after commit."""
@@ -415,9 +294,7 @@ class ModelWithOutputDir(ModelWithUUID):
         if cls._delete_signal_registered:
             return
 
-        def schedule_output_dir_cleanup(sender, instance, using, origin=None, **kwargs):
-            if getattr(origin, "_output_files_deleted", False):
-                return
+        def schedule_output_dir_cleanup(sender, instance, using, **kwargs):
             if not isinstance(instance, ModelWithOutputDir):
                 return
             instance.schedule_delete_cleanup(using=using)

archivebox/cli/__init__.py

@@ -2,7 +2,6 @@ __package__ = "archivebox.cli"
 __command__ = "archivebox"
 import os
 import sys
-from contextlib import redirect_stdout
 from importlib import import_module
 
 import rich_click as click
@@ -156,10 +155,13 @@ def cli(ctx, help=False):
     wants_help = any(arg in ("-h", "--help", "--version") for arg in sys.argv[1:])
     if not wants_help and (subcommand in ArchiveBoxGroup.archive_commands or subcommand in ArchiveBoxGroup.model_commands):
         try:
-            # RUNSERVER_WORKER sets the web-worker markers on its own child.
-            # Setting them on the server parent leaks them through supervisord
-            # into runners waiting for takeover, which then register themselves
-            # as web workers and trigger spurious runner_watch reloads.
+            if subcommand == "server":
+                run_in_debug = "--reload" in sys.argv or os.environ.get("DEBUG") in ("1", "true", "True", "TRUE", "yes")
+                if run_in_debug:
+                    os.environ["ARCHIVEBOX_RUNSERVER"] = "1"
+                    if "--reload" in sys.argv:
+                        os.environ["ARCHIVEBOX_AUTORELOAD"] = "1"
+
             from archivebox.config.django import setup_django
             from archivebox.misc.checks import check_data_folder, check_migrations
 
@@ -175,12 +177,8 @@ def cli(ctx, help=False):
                 archivebox_init(quick=True)
                 os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
             check_data_folder()
-            if subcommand != "update" and not check_migrations(auto_apply=True):
-                from archivebox.machine.models import Machine
-
-                # Host registration and config reconciliation belong to CLI
-                # startup; get_config() is also used by read-only HTTP views.
-                Machine.current()
+            if subcommand != "update":
+                check_migrations(auto_apply=True)
         except Exception as e:
             STDERR.print(f"[red][X] Error setting up Django or checking data folder: {e}[/red]")
             if subcommand not in ("manage", "shell"):  # not all management commands need django to be setup beforehand
@@ -220,12 +218,6 @@ def main(args=None, prog_name=None):
         raise SystemExit(130) from None
     finally:
         sys.unraisablehook = previous_unraisablehook
-        supervisor_util = sys.modules.get("archivebox.workers.supervisord_util")
-        if supervisor_util is not None:
-            # Short commands can start a foreground-owned search daemon too.
-            # Finish its cleanup before another CLI can borrow a dying stack.
-            with redirect_stdout(sys.stderr):
-                supervisor_util.stop_own_supervisord_process()
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_add.py

@@ -6,7 +6,7 @@ __command__ = "archivebox add"
 
 import sys
 import os
-import signal
+import json
 from pathlib import Path
 
 from typing import Any, TYPE_CHECKING
@@ -71,7 +71,17 @@ def add(
     created_by_id: int | None = None,
     config: dict[str, Any] | None = None,
 ) -> tuple[Crawl, QuerySet[Snapshot]]:
-    """Add a URL list or imported URL document to a new Crawl."""
+    """Add a new URL or list of URLs to your archive.
+
+    The flow is:
+    1. Save URLs to sources file
+    2. Create Crawl with URLs and max_depth
+    3. Crawl runner creates Snapshots from Crawl URLs (depth=0)
+    4. Crawl runner runs parser extractors on root snapshots
+    5. Parser extractors output to urls.jsonl
+    6. URLs are added to Crawl.urls and child Snapshots are created
+    7. Repeat until max_depth is reached
+    """
 
     from rich import print
 
@@ -115,12 +125,12 @@ def add(
     created_by_id = created_by_id or get_or_create_system_user_pk()
     started_at = timezone.now()
 
-    if isinstance(urls, str):
-        source_text = urls
-    else:
-        from archivebox.misc.util import validate_url
-
-        source_text = "\n".join(validate_url(str(url)) for url in urls)
+    use_internal_input_root = isinstance(urls, str)
+    source_text = (
+        urls
+        if use_internal_input_root
+        else "\n".join(json.dumps({"type": "CrawlSeed", "url": str(url), "depth": 0}, separators=(",", ":")) for url in urls)
+    )
 
     # 2. Create a new Crawl with inline URLs
     # Foreground add must claim runner ownership before publishing runnable
@@ -144,6 +154,7 @@ def add(
     persona_name = (persona or "Default").strip() or "Default"
     plugins = plugins or ""
     persona_obj = Persona.get_or_create_named(persona_name)
+    persona_obj.ensure_dirs()
     effective_persona_config = get_config(persona=persona_obj)
 
     crawl_config = {
@@ -176,7 +187,12 @@ def add(
     try:
         crawl = Crawl.objects.create(
             urls=source_text,
-            max_depth=depth,
+            # Stdin/import text gets an extra hop because the synthetic
+            # archivebox://internal root lives at depth 0 and parser-discovered
+            # URLs land at depth 1; direct URL args become the depth=0 input
+            # snapshots themselves so --depth=N matches the deepest hop the user
+            # asked for.
+            max_depth=depth + 1 if use_internal_input_root else depth,
             tags_str=tag,
             persona_id=persona_obj.id,
             label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
@@ -197,8 +213,9 @@ def add(
     print(f"[green]\\[+] Created Crawl {crawl.id} with max_depth={depth}[/green]")
     print(f"    [dim]First URL: {first_url}[/dim]")
 
-    # The runner parses Crawl.urls after claiming the Crawl, then persists the
-    # resulting depth-0 Snapshot facts. Recursive discoveries start at depth 1.
+    # 3. The runner will create Snapshots from all URLs after claiming the Crawl
+    #    Parser extractors run on snapshots and discover more URLs
+    #    Discovered URLs become child Snapshots (depth+1)
 
     if index_only:
         print("[yellow]\\[*] Index-only mode - URLs queued, runner not started[/yellow]")
@@ -216,6 +233,9 @@ def add(
         print(
             "[yellow]\\[*] URLs queued. The background runner will process them (run `archivebox server` or `archivebox run --daemon` if not already running).[/yellow]",
         )
+        from archivebox.services.runner import ensure_background_runner
+
+        ensure_background_runner()
     else:
         # Foreground mode: run full crawl runner until all work is done
         print("[green]\\[*] Starting crawl runner to process crawl...[/green]")
@@ -224,10 +244,6 @@ def add(
 
         assert command is not None
         exit_code = 0
-        interactive_interrupts = sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())
-
-        def interrupt_hook() -> None:
-            raise KeyboardInterrupt
 
         def crawl_is_complete() -> bool:
             crawl.refresh_from_db(fields=["status"])
@@ -235,13 +251,7 @@ def add(
 
         try:
             try:
-                with (
-                    foreground_shutdown_signals(
-                        first_signal_message=None,
-                        interrupt_handlers={signal.SIGINT: interrupt_hook},
-                    ),
-                    foreground_parent_watchdog(),
-                ):
+                with foreground_shutdown_signals(first_signal_message=None), foreground_parent_watchdog():
                     while True:
                         standby = standby_until_foreground_runner_needed(
                             command,
@@ -253,7 +263,7 @@ def add(
                         exit_code = run_runner_worker(
                             ["--crawl-id", str(crawl.id)],
                             name=f"worker_runner_add_{os.getpid()}",
-                            interactive_interrupts=interactive_interrupts,
+                            interactive_interrupts=True,
                             config=get_config(crawl=crawl),
                         )
                         crawl.refresh_from_db(fields=["status", "retry_at"])
@@ -318,13 +328,9 @@ def add(
 
             admin_url = build_admin_url(f"/admin/crawls/crawl/{crawl.id}/change/", config=config)
 
-            if snapshots_count:
-                print("\n[bold]snapshot output saved to:[/bold]")
-                for snapshot in crawl.snapshot_set.order_by("created_at").iterator():
-                    click.echo(f"  {snapshot.output_dir}")
-            print("\n[bold]crawl logs and setup:[/bold]")
-            click.echo(f"  {rel_output_str}")
-            click.echo(f"  {admin_url}")
+            print("\n[bold]crawl output saved to:[/bold]")
+            print(f"  {rel_output_str}")
+            print(f"  {admin_url}")
             print(f"\n[bold]total urls snapshotted:[/bold] {snapshots_count}")
             print(f"[bold]total size:[/bold] {total_size}")
             print(f"[bold]total time:[/bold] {duration_str}")
@@ -399,6 +405,10 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
+        if not kwargs.get("index_only"):
+            from archivebox.workers.supervisord_util import require_crawl_memory
+
+            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.
@@ -411,35 +421,19 @@ def main(**kwargs):
         if only_new is not None:
             kwargs["config"] = {"ONLY_NEW": bool(only_new)}
         if extract:
-            from archivebox.plugins.discovery import get_plugin_catalog
+            from abx_dl.models import discover_plugins, plugins_matching_output
 
-            catalog = get_plugin_catalog()
+            all_plugins = discover_plugins()
             tokens = [token.strip() for token in extract.split(",") if token.strip()]
-            plugin_names = {name.lower(): name for name in catalog}
+            plugin_names = {name.lower(): name for name in all_plugins}
             selected = [plugin_names[token.lower()] for token in tokens if token.lower() in plugin_names]
-            selected += catalog.matching_output(tokens)
+            selected += plugins_matching_output(all_plugins, tokens)
             if not selected:
                 raise click.UsageError(f"No plugins found matching extract types: {extract}")
             existing = [token.strip() for token in (kwargs.get("plugins") or "").split(",") if token.strip()]
             kwargs["plugins"] = ",".join(dict.fromkeys([*existing, *selected]))
 
-        from archivebox.machine.models import Process
-
-        # CLI commands own their process record and logs. Queue model saves
-        # must not register a process implicitly: HTTP add() shares that path.
-        process = Process.current()
-        exit_code = 0
-        try:
-            add(urls=urls, **kwargs)
-        except BaseException:
-            exit_code = 1
-            raise
-        finally:
-            # --bg/--index-only return without the foreground runner's cleanup.
-            # Leaving them RUNNING makes the active-process index accumulate
-            # historical CLI commands forever, defeating bounded recovery.
-            if process.status == Process.StatusChoices.RUNNING:
-                process.mark_exited(exit_code=exit_code)
+        add(urls=urls, **kwargs)
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_archiveresult.py

@@ -6,13 +6,13 @@ archivebox archiveresult <action> [args...] [--filters]
 Manage ArchiveResult records (plugin extraction results).
 
 Actions:
-    create  - Emit plugin extraction request records for Snapshots
+    create  - Create ArchiveResults for Snapshots (queue extractions)
     list    - List ArchiveResults as JSONL (with optional filters)
     update  - Update ArchiveResults from stdin JSONL
     delete  - Delete ArchiveResults from stdin JSONL
 
 Examples:
-    # Emit extraction requests; `archivebox run` schedules their parent snapshots
+    # Create ArchiveResults for snapshots (queue for extraction)
     archivebox snapshot list --status=queued | archivebox archiveresult create
     archivebox archiveresult create --plugin=screenshot --snapshot-id=<uuid>
 
@@ -20,6 +20,9 @@ Examples:
     archivebox archiveresult list --status=failed
     archivebox archiveresult list --plugin=screenshot --status=succeeded
 
+    # Update (reset failed extractions to queued)
+    archivebox archiveresult list --status=failed | archivebox archiveresult update --status=queued
+
     # Delete
     archivebox archiveresult list --plugin=singlefile | archivebox archiveresult delete --yes
 
@@ -154,7 +157,7 @@ def create_archiveresults(
                 write_record(build_archiveresult_request(snapshot.id, plugin_name, hook_name=hook_name, status=status))
             created_count += 1
 
-    rprint(f"[green]Created {created_count} extraction request records[/green]", file=sys.stderr)
+    rprint(f"[green]Created {created_count} archive result request records[/green]", file=sys.stderr)
     return 0
 
 
@@ -338,7 +341,7 @@ def main():
 @click.option("--plugin", "-p", help="Plugin name (e.g., screenshot, singlefile)")
 @click.option("--status", "-s", default="queued", help="Initial status (default: queued)")
 def create_cmd(snapshot_id: str | None, plugin: str | None, status: str):
-    """Emit Snapshot plugin extraction requests as JSONL."""
+    """Create ArchiveResults for Snapshots from stdin JSONL."""
     sys.exit(create_archiveresults(snapshot_id=snapshot_id, plugin=plugin, status=status))
 
 

archivebox/cli/archivebox_extract.py

@@ -1,135 +1,100 @@
 #!/usr/bin/env python3
 
-"""Run abx-dl snapshot hooks for existing ArchiveBox snapshots."""
+"""
+archivebox extract [snapshot_ids...] [--plugins=NAMES]
+
+Run plugins on Snapshots. Accepts snapshot IDs as arguments, from stdin, or via JSONL.
+
+Input formats:
+    - Snapshot UUIDs (one per line)
+    - JSONL: {"type": "Snapshot", "id": "...", "url": "..."}
+    - JSONL: {"type": "ArchiveResult", "snapshot_id": "...", "plugin": "..."}
+
+Output (JSONL):
+    {"type": "ArchiveResult", "id": "...", "snapshot_id": "...", "plugin": "...", "status": "..."}
+
+Examples:
+    # Extract specific snapshot
+    archivebox extract 01234567-89ab-cdef-0123-456789abcdef
+
+    # Pipe from snapshot command
+    archivebox snapshot https://example.com | archivebox extract
+
+    # Run specific plugins only
+    archivebox extract --plugins=screenshot,singlefile 01234567-89ab-cdef-0123-456789abcdef
+
+    # Chain commands
+    archivebox crawl https://example.com | archivebox snapshot | archivebox extract
+"""
 
 __package__ = "archivebox.cli"
 __command__ = "archivebox extract"
 
 import sys
 from collections import defaultdict
-from contextlib import redirect_stdout
+from itertools import product
 
 import rich_click as click
 
 
-def _resolve_requests(records: list[dict], plugins: str) -> tuple[dict[str, set[str]], set[str]]:
-    """Resolve CLI records to snapshot-level execution requests.
-
-    ArchiveResult input is accepted as a convenient reference to its parent
-    Snapshot and plugin. It is never reset or converted into queued work.
+def process_archiveresult_by_id(archiveresult_id: str) -> int:
     """
-    from archivebox.api.v1_core import _uuid_ref_query
-    from archivebox.core.models import ArchiveResult, Snapshot
-    from archivebox.misc.jsonl import TYPE_ARCHIVERESULT
+    Re-run extraction for a single ArchiveResult by ID.
 
-    explicit_plugins = {name.strip() for name in plugins.split(",") if name.strip()}
-    requested: dict[str, set[str]] = defaultdict(set)
-    missing: set[str] = set()
-
-    for record in records:
-        record_type = record.get("type")
-        record_id = str(record.get("id") or "")
-
-        if record_type == TYPE_ARCHIVERESULT and record.get("snapshot_id"):
-            requested[str(record["snapshot_id"])].update(explicit_plugins or {str(record.get("plugin") or "")})
-            requested[str(record["snapshot_id"])].discard("")
-            continue
-
-        if record_type == TYPE_ARCHIVERESULT:
-            result = ArchiveResult.objects.filter(_uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
-            if result is not None:
-                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
-                continue
-
-        snapshot_id = str(record.get("snapshot_id") or record_id or "")
-        snapshot = Snapshot.objects.filter(id=snapshot_id).only("id").first() if snapshot_id else None
-        if snapshot is None and record.get("url"):
-            snapshot = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").only("id").first()
-        # Bare UUID CLI arguments are parsed as Snapshot records because the
-        # input layer cannot know which model owns them. Preserve the public
-        # convenience of passing an ArchiveResult ID by trying that reference
-        # only after the Snapshot lookup misses.
-        if snapshot is None and record_id and not record.get("url"):
-            result = ArchiveResult.objects.filter(_uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
-            if result is not None:
-                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
-                continue
-        if snapshot is None:
-            missing.add(snapshot_id or str(record.get("url") or ""))
-            continue
-        requested[str(snapshot.id)].update(explicit_plugins)
-
-    return requested, missing
-
-
-def _run_snapshot_requests(requested: dict[str, set[str]], *, wait: bool, show_progress: bool) -> int:
-    from django.utils import timezone
+    ArchiveResults are projected status rows, not queued work items. Re-running
+    a single result means resetting that row and queueing its parent snapshot
+    through the shared crawl runner with the corresponding plugin selected.
+    """
     from rich import print as rprint
-
-    from archivebox.core.models import Snapshot
+    from archivebox.core.models import ArchiveResult
+    from archivebox.api.v1_core import _uuid_ref_query
     from archivebox.services.runner import run_crawl
 
-    snapshots = {str(snapshot.id): snapshot for snapshot in Snapshot.objects.filter(id__in=requested).select_related("crawl")}
-    if not snapshots:
-        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
+    try:
+        archiveresult = ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
+    except ArchiveResult.DoesNotExist:
+        rprint(f"[red]ArchiveResult {archiveresult_id} not found[/red]", file=sys.stderr)
         return 1
 
-    from archivebox.config.common import get_config
-    from archivebox.plugins.discovery import get_enabled_plugins
+    rprint(f"[blue]Extracting {archiveresult.plugin} for {archiveresult.snapshot.url}[/blue]", file=sys.stderr)
 
-    for snapshot_id, plugin_names in requested.items():
-        snapshot = snapshots.get(snapshot_id)
-        if snapshot is not None and not plugin_names:
-            plugin_names.update(get_enabled_plugins(config=get_config(crawl=snapshot.crawl, snapshot=snapshot)))
+    try:
+        archiveresult.reset_for_retry()
+        snapshot = archiveresult.snapshot
+        snapshot.queue_for_extraction()
+        crawl = snapshot.crawl
+        if not crawl.claim_processing_lock(lock_seconds=10):
+            rprint(
+                f"[yellow]Crawl {crawl.id} is already owned by another runner[/yellow]",
+                file=sys.stderr,
+            )
+            return 1
 
-    if wait and any("search_backend_sonic" in plugin_names for plugin_names in requested.values()):
-        from archivebox.core.takeover_util import ensure_daemon_stack
-
-        with redirect_stdout(sys.stderr):
-            ensure_daemon_stack(reason="Sonic snapshot indexing")
-
-    # Explicit extraction resumes open/paused snapshots at the snapshot level.
-    # Sealed snapshots stay sealed during targeted maintenance backfills.
-    if wait:
-        for snapshot in snapshots.values():
-            if snapshot.status != Snapshot.StatusChoices.SEALED:
-                snapshot.update_and_requeue(
-                    status=Snapshot.StatusChoices.QUEUED,
-                    retry_at=timezone.now(),
-                )
-
-    if not wait:
-        for snapshot_id, plugin_names in requested.items():
-            snapshot = snapshots.get(snapshot_id)
-            if snapshot is None:
-                continue
-            if plugin_names:
-                snapshot.schedule_plugin_run(plugin_names, when=timezone.now())
-            elif snapshot.status == Snapshot.StatusChoices.SEALED:
-                snapshot.update_and_requeue(retry_at=timezone.now())
-            else:
-                snapshot.update_and_requeue(
-                    status=Snapshot.StatusChoices.QUEUED,
-                    retry_at=timezone.now(),
-                )
-        if show_progress:
-            rprint(f"[blue]Queued {len(snapshots)} snapshots for extraction[/blue]", file=sys.stderr)
-        return 0
-
-    grouped: dict[tuple[str, tuple[str, ...]], list[str]] = defaultdict(list)
-    for snapshot_id, plugin_names in requested.items():
-        snapshot = snapshots.get(snapshot_id)
-        if snapshot is not None:
-            grouped[(str(snapshot.crawl_id), tuple(sorted(plugin_names)))].append(snapshot_id)
-
-    for (crawl_id, plugin_names), snapshot_ids in grouped.items():
         run_crawl(
-            crawl_id,
-            snapshot_ids=sorted(snapshot_ids),
-            selected_plugins=list(plugin_names) or None,
-            show_progress=show_progress,
+            str(snapshot.crawl_id),
+            snapshot_ids=[str(snapshot.id)],
+            selected_plugins=[archiveresult.plugin],
+            selected_plugins_are_explicit=False,
         )
-    return 0
+        archiveresult.refresh_from_db()
+
+        if archiveresult.status == ArchiveResult.StatusChoices.SUCCEEDED:
+            print(f"[green]Extraction succeeded: {archiveresult.output_str}[/green]")
+            return 0
+        elif archiveresult.status == ArchiveResult.StatusChoices.NORESULTS:
+            print(f"[dim]Extraction completed with no results: {archiveresult.output_str}[/dim]")
+            return 0
+        elif archiveresult.status == ArchiveResult.StatusChoices.FAILED:
+            print(f"[red]Extraction failed: {archiveresult.output_str}[/red]", file=sys.stderr)
+            return 1
+        else:
+            # Still in progress or backoff - not a failure
+            print(f"[yellow]Extraction status: {archiveresult.status}[/yellow]")
+            return 0
+
+    except Exception as e:
+        print(f"[red]Extraction error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
+        return 1
 
 
 def run_plugins(
@@ -139,64 +104,384 @@ def run_plugins(
     wait: bool = True,
     emit_results: bool = True,
     show_progress: bool = True,
+    preserve_queued: bool = False,
 ) -> int:
-    """Execute selected plugins through the snapshot-level runner."""
+    """
+    Run plugins on Snapshots from input.
+
+    Reads Snapshot IDs or JSONL from args/stdin, runs plugins, outputs JSONL.
+
+    Exit codes:
+        0: Success
+        1: Failure
+    """
     from rich import print as rprint
+    from django.utils import timezone
 
+    from archivebox.misc.jsonl import (
+        read_args_or_stdin,
+        write_record,
+        TYPE_SNAPSHOT,
+        TYPE_ARCHIVERESULT,
+    )
     from archivebox.core.models import Snapshot
-    from archivebox.misc.jsonl import read_args_or_stdin, write_record
+    from archivebox.core.models import ArchiveResult
+    from archivebox.services.runner import run_crawl
+    from abx_dl.models import discover_plugins
 
+    is_tty = sys.stdout.isatty()
+
+    # Parse comma-separated plugins list once (reused in creation and filtering)
+    plugins_list = [p.strip() for p in plugins.split(",") if p.strip()] if plugins else []
+
+    # Parse stdin/args exactly once per CLI invocation.
+    # `main()` may already have consumed stdin to distinguish Snapshot input from
+    # ArchiveResult IDs; if so, it must pass the parsed records through here
+    # instead of asking this helper to reread an already-drained pipe.
     if records is None:
         records = list(read_args_or_stdin(args))
+
     if not records:
         rprint("[yellow]No snapshots provided. Pass snapshot IDs as arguments or via stdin.[/yellow]", file=sys.stderr)
         return 1
 
-    requested, missing = _resolve_requests(records, plugins)
-    for value in sorted(missing):
-        rprint(f"[yellow]Snapshot or ArchiveResult not found: {value}[/yellow]", file=sys.stderr)
-    if not requested:
+    # Gather snapshot IDs and optional plugin constraints to process
+    snapshot_ids = set()
+    requested_plugins_by_snapshot: dict[str, set[str]] = defaultdict(set)
+    requested_hooks_by_snapshot: dict[str, set[tuple[str, str]]] = defaultdict(set)
+    plugin_level_requests_by_snapshot: dict[str, set[str]] = defaultdict(set)
+    for record in records:
+        record_type = record.get("type")
+
+        if record_type == TYPE_SNAPSHOT:
+            snapshot_id = record.get("id")
+            if snapshot_id:
+                snapshot_ids.add(str(snapshot_id))
+            elif record.get("url"):
+                # Look up by URL (get most recent if multiple exist)
+                snap = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").first()
+                if snap:
+                    snapshot_ids.add(str(snap.id))
+                else:
+                    rprint(f"[yellow]Snapshot not found for URL: {record['url']}[/yellow]", file=sys.stderr)
+
+        elif record_type == TYPE_ARCHIVERESULT:
+            snapshot_id = record.get("snapshot_id")
+            if snapshot_id:
+                snapshot_ids.add(str(snapshot_id))
+                plugin_name = record.get("plugin")
+                if plugin_name and not plugins_list:
+                    snapshot_key = str(snapshot_id)
+                    plugin_key = str(plugin_name)
+                    requested_plugins_by_snapshot[snapshot_key].add(plugin_key)
+                    hook_name = str(record.get("hook_name") or "")
+                    if hook_name:
+                        requested_hooks_by_snapshot[snapshot_key].add((plugin_key, hook_name))
+                    else:
+                        plugin_level_requests_by_snapshot[snapshot_key].add(plugin_key)
+
+        elif "id" in record:
+            # Assume it's a snapshot ID
+            snapshot_ids.add(str(record["id"]))
+
+    if not snapshot_ids:
+        rprint("[red]No valid snapshot IDs found in input[/red]", file=sys.stderr)
         return 1
 
-    exit_code = _run_snapshot_requests(requested, wait=wait, show_progress=show_progress)
-    if exit_code or not emit_results:
-        return exit_code
+    existing_snapshots = list(Snapshot.objects.filter(id__in=snapshot_ids).values_list("id", "crawl_id"))
+    existing_snapshot_ids = {str(snapshot_id) for snapshot_id, _crawl_id in existing_snapshots}
+    existing_crawl_ids = {str(crawl_id) for _snapshot_id, crawl_id in existing_snapshots}
+    missing_snapshot_ids = sorted(str(snapshot_id) for snapshot_id in snapshot_ids - existing_snapshot_ids)
+    for snapshot_id in missing_snapshot_ids:
+        rprint(f"[yellow]Snapshot {snapshot_id} not found[/yellow]", file=sys.stderr)
+
+    # Queue only the target plugin rows. Bulk updates keep large reindex runs
+    # from doing one SELECT+UPDATE per snapshot/plugin before hooks even start.
+    requested_pairs: set[tuple[str, str]] = set()
+    if plugins_list:
+        requested_pairs.update((snapshot_id, plugin_name) for snapshot_id, plugin_name in product(existing_snapshot_ids, plugins_list))
+    else:
+        requested_pairs.update(
+            (snapshot_id, plugin_name)
+            for snapshot_id, plugin_names in requested_plugins_by_snapshot.items()
+            if snapshot_id in existing_snapshot_ids
+            for plugin_name in plugin_names
+        )
+    plugins_by_name = discover_plugins(runtime="archivebox")
+    requested_rows: set[tuple[str, str, str]] = set()
+    for snapshot_id, plugin_name in requested_pairs:
+        exact_hook_names = {
+            hook_name
+            for requested_plugin, hook_name in requested_hooks_by_snapshot.get(snapshot_id, set())
+            if requested_plugin == plugin_name
+        }
+        if exact_hook_names and plugin_name not in plugin_level_requests_by_snapshot.get(snapshot_id, set()):
+            requested_rows.update((snapshot_id, plugin_name, hook_name) for hook_name in exact_hook_names)
+            continue
+        plugin = plugins_by_name.get(plugin_name)
+        hooks = plugin.filter_hooks("Snapshot") if plugin is not None else []
+        if hooks:
+            requested_rows.update((snapshot_id, plugin_name, hook.name) for hook in hooks)
+        else:
+            requested_rows.add((snapshot_id, plugin_name, ""))
+
+    queued_rows: set[tuple[str, str, str]] = set()
+    if preserve_queued and requested_rows:
+        queued_rows = {
+            (str(snapshot_id), plugin_name, hook_name)
+            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
+                snapshot_id__in=existing_snapshot_ids,
+                plugin__in={plugin_name for _snapshot_id, plugin_name, _hook_name in requested_rows},
+                status=ArchiveResult.StatusChoices.QUEUED,
+            ).values_list("snapshot_id", "plugin", "hook_name")
+        }
+    rows_to_queue = requested_rows - queued_rows
+
+    reset_fields = {
+        "status": ArchiveResult.StatusChoices.QUEUED,
+        "output_str": "",
+        "output_json": None,
+        "output_files": {},
+        "output_size": 0,
+        "output_mimetypes": "",
+        "start_ts": None,
+        "end_ts": None,
+        "modified_at": timezone.now(),
+    }
+    if rows_to_queue and plugins_list:
+        rows_to_reset_by_hook: dict[tuple[str, str], set[str]] = defaultdict(set)
+        for snapshot_id, plugin_name, hook_name in rows_to_queue:
+            rows_to_reset_by_hook[(plugin_name, hook_name)].add(snapshot_id)
+        for (plugin_name, hook_name), plugin_snapshot_ids in rows_to_reset_by_hook.items():
+            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name, hook_name=hook_name).update(
+                **reset_fields,
+            )
+    elif rows_to_queue and requested_plugins_by_snapshot:
+        snapshot_ids_by_hook: dict[tuple[str, str], set[str]] = defaultdict(set)
+        for snapshot_id, plugin_name, hook_name in rows_to_queue:
+            snapshot_ids_by_hook[(plugin_name, hook_name)].add(snapshot_id)
+        for (plugin_name, hook_name), plugin_snapshot_ids in snapshot_ids_by_hook.items():
+            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name, hook_name=hook_name).update(
+                **reset_fields,
+            )
+    existing_rows = (
+        {
+            (str(snapshot_id), plugin_name, hook_name)
+            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
+                snapshot_id__in=existing_snapshot_ids,
+                plugin__in={plugin_name for _snapshot_id, plugin_name, _hook_name in rows_to_queue},
+            ).values_list("snapshot_id", "plugin", "hook_name")
+        }
+        if rows_to_queue
+        else set()
+    )
+    missing_rows = rows_to_queue - existing_rows
+    if missing_rows:
+        ArchiveResult.objects.bulk_create(
+            [
+                ArchiveResult(
+                    snapshot_id=snapshot_id,
+                    plugin=plugin_name,
+                    hook_name=hook_name,
+                    status=ArchiveResult.StatusChoices.QUEUED,
+                )
+                for snapshot_id, plugin_name, hook_name in sorted(missing_rows)
+            ],
+            batch_size=500,
+        )
+
+    processed_count = len(existing_snapshot_ids)
+    queue_at = timezone.now()
+    if existing_snapshot_ids:
+        if requested_rows:
+            # Search indexing on a sealed Snapshot is the only targeted hook
+            # allowed to bypass the normal lifecycle. Every other requested
+            # plugin requeues its Snapshot through the unified lifecycle.
+            affected_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in rows_to_queue}
+            if preserve_queued and queued_rows:
+                queued_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in queued_rows}
+                affected_snapshot_ids.update(
+                    str(snapshot_id)
+                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids)
+                    .filter(retry_at__gt=queue_at)
+                    .values_list("id", flat=True)
+                )
+                affected_snapshot_ids.update(
+                    str(snapshot_id)
+                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids, retry_at__isnull=True).values_list(
+                        "id",
+                        flat=True,
+                    )
+                )
+            requested_plugins_by_id: dict[str, set[str]] = defaultdict(set)
+            for snapshot_id, plugin_name, _hook_name in requested_rows:
+                requested_plugins_by_id[snapshot_id].add(plugin_name)
+            for snapshot in Snapshot.objects.filter(id__in=affected_snapshot_ids).only("id", "status", "modified_at"):
+                # Guard the read-time status so we never bump retry_at on a
+                # row that's been re-queued / started by a concurrent runner.
+                plugin_names = requested_plugins_by_id.get(str(snapshot.id), set())
+                sealed_search_backfill = (
+                    snapshot.status == Snapshot.StatusChoices.SEALED
+                    and plugin_names
+                    and all(plugin_name.startswith("search_backend_") for plugin_name in plugin_names)
+                )
+                if sealed_search_backfill:
+                    snapshot.safe_update(
+                        {"retry_at": queue_at, "modified_at": queue_at},
+                        refresh=False,
+                        extra_filter={"status": snapshot.status},
+                    )
+                else:
+                    snapshot.update_and_requeue(
+                        status=Snapshot.StatusChoices.QUEUED,
+                        retry_at=queue_at,
+                        current_step=0,
+                    )
+        else:
+            # No plugin rows were requested, so this is a full snapshot retry.
+            for snapshot in Snapshot.objects.filter(id__in=existing_snapshot_ids).only("id", "status", "retry_at", "modified_at"):
+                snapshot.safe_update(
+                    {
+                        "status": Snapshot.StatusChoices.QUEUED,
+                        "retry_at": queue_at,
+                        "current_step": 0,
+                        "modified_at": queue_at,
+                    },
+                    refresh=False,
+                    extra_filter={"status": snapshot.status},
+                )
+    if existing_crawl_ids and not requested_rows:
+        from archivebox.crawls.models import Crawl
+
+        for crawl in Crawl.objects.filter(id__in=existing_crawl_ids).only("id", "status", "retry_at", "modified_at"):
+            update_fields = {
+                "retry_at": queue_at,
+                "modified_at": queue_at,
+            }
+            if crawl.status != Crawl.StatusChoices.STARTED:
+                update_fields["status"] = Crawl.StatusChoices.QUEUED
+            crawl.safe_update(
+                update_fields,
+                refresh=False,
+                extra_filter={"status": crawl.status},
+            )
+
+    if processed_count == 0:
+        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
+        return 1
+
+    if show_progress:
+        rprint(f"[blue]Queued {processed_count} snapshots for extraction[/blue]", file=sys.stderr)
+
+    # Run orchestrator if --wait (default)
+    if wait:
+        if show_progress:
+            rprint("[blue]Running plugins...[/blue]", file=sys.stderr)
+        snapshot_ids_by_crawl: dict[str, set[str]] = defaultdict(set)
+        for snapshot_id, crawl_id in existing_snapshots:
+            snapshot_ids_by_crawl[str(crawl_id)].add(str(snapshot_id))
+
+        for crawl_id, crawl_snapshot_ids in snapshot_ids_by_crawl.items():
+            from archivebox.crawls.models import Crawl
+
+            crawl = Crawl.objects.get(id=crawl_id)
+            if not crawl.claim_processing_lock(lock_seconds=10):
+                rprint(
+                    f"[yellow]Crawl {crawl_id} is already owned by another runner[/yellow]",
+                    file=sys.stderr,
+                )
+                return 1
+            selected_plugins = (
+                plugins_list
+                or sorted(
+                    {plugin for snapshot_id, plugin, _hook_name in requested_rows if snapshot_id in crawl_snapshot_ids},
+                )
+                or None
+            )
+            run_crawl(
+                crawl_id,
+                snapshot_ids=sorted(crawl_snapshot_ids),
+                selected_plugins=selected_plugins,
+                show_progress=show_progress,
+                selected_plugins_are_explicit=bool(plugins_list),
+            )
+
+    if not emit_results:
+        return 0
+
+    # Output results as JSONL (when piped) or human-readable (when TTY)
+    for snapshot_id in snapshot_ids:
+        try:
+            snapshot = Snapshot.objects.get(id=snapshot_id)
+            results = snapshot.archiveresult_set.all()
+            if plugins_list:
+                results = results.filter(plugin__in=plugins_list)
+
+            for result in results:
+                if is_tty:
+                    status_color = {
+                        "succeeded": "green",
+                        "failed": "red",
+                        "skipped": "yellow",
+                    }.get(result.status, "dim")
+                    rprint(
+                        f"  [{status_color}]{result.status}[/{status_color}] {result.plugin} → {result.output_str or ''}",
+                        file=sys.stderr,
+                    )
+                else:
+                    write_record(result.to_json())
+        except Snapshot.DoesNotExist:
+            continue
 
-    is_tty = sys.stdout.isatty()
-    for snapshot in Snapshot.objects.filter(id__in=requested):
-        results = snapshot.archiveresult_set.all()
-        requested_plugins = requested[str(snapshot.id)]
-        if requested_plugins:
-            results = results.filter(plugin__in=requested_plugins)
-        for result in results:
-            if is_tty:
-                color = {"succeeded": "green", "failed": "red", "skipped": "yellow"}.get(result.status, "dim")
-                rprint(f"  [{color}]{result.status}[/{color}] {result.plugin} → {result.output_str or ''}", file=sys.stderr)
-            else:
-                write_record(result.to_json())
     return 0
 
 
-def process_archiveresult_by_id(archiveresult_id: str) -> int:
-    """Re-run the parent Snapshot plugin referenced by an ArchiveResult."""
-    return run_plugins((), records=[{"id": archiveresult_id}], wait=True)
+def is_archiveresult_id(value: str) -> bool:
+    """Check if value resolves to an ArchiveResult ID."""
+    from archivebox.core.models import ArchiveResult
+    from archivebox.api.v1_core import _uuid_ref_query
+
+    return ArchiveResult.objects.filter(_uuid_ref_query("id", value)).exists()
 
 
 @click.command()
-@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run")
+@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run (e.g., screenshot,singlefile)")
 @click.option("--wait/--no-wait", default=True, help="Wait for plugins to complete (default: wait)")
 @click.argument("args", nargs=-1)
 def main(plugins: str, wait: bool, args: tuple):
-    """Run plugins on Snapshots; ArchiveResult IDs select their parent plugin."""
+    """Run plugins on Snapshots, or process existing ArchiveResults by ID"""
     from archivebox.misc.jsonl import read_args_or_stdin
 
+    # Read all input
     records = list(read_args_or_stdin(args))
+
     if not records:
         from rich import print as rprint
 
         rprint("[yellow]No Snapshot IDs or ArchiveResult IDs provided. Pass as arguments or via stdin.[/yellow]", file=sys.stderr)
         sys.exit(1)
-    sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
+
+    # Check if input looks like existing ArchiveResult IDs to process
+    all_are_archiveresult_ids = all(is_archiveresult_id(r.get("id") or r.get("url", "")) for r in records)
+
+    if all_are_archiveresult_ids:
+        # Process existing ArchiveResults by ID
+        from rich import print as rprint
+
+        exit_code = 0
+        for record in records:
+            archiveresult_id = record.get("id") or record.get("url")
+            if not isinstance(archiveresult_id, str):
+                rprint(f"[red]Invalid ArchiveResult input: {record}[/red]", file=sys.stderr)
+                exit_code = 1
+                continue
+            result = process_archiveresult_by_id(archiveresult_id)
+            if result != 0:
+                exit_code = result
+        sys.exit(exit_code)
+    else:
+        # Default behavior: run plugins on Snapshots from input
+        sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_help.py

@@ -68,7 +68,7 @@ def help() -> None:
     [blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
 
     [grey53]# using Docker:[/grey53]
-    [blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 5797:5797[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
+    [blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
 """
         if IN_DOCKER
         else ""
@@ -109,7 +109,7 @@ def help() -> None:
     [dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
     [dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
     [dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
-    [dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:5797[/blue]                [grey53]# Start the Web UI / API server[/grey53]
+    [dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue]                [grey53]# Start the Web UI / API server[/grey53]
 """
         print(
             Panel(

archivebox/cli/archivebox_init.py

@@ -111,16 +111,6 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
         else:
             os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init
 
-    from archivebox.machine.models import Machine
-
-    # Registration requires the final schema and reconciles legacy file config.
-    Machine.current()
-
-    from archivebox.personas.models import Persona
-
-    # Initialize here, not when /add/ is rendered: a GET must not create profiles.
-    Persona.get_or_create_default().ensure_dirs()
-
     from archivebox.misc.db import database_display_location, is_postgres
 
     assert database_exists()
@@ -144,7 +134,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
     print("    > Skipping orphan snapshot import during init.")
     print()
     print("    [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
-    print("        archivebox update --rescan --migrate-only")
+    print("        archivebox update")
 
     print("\n[green]----------------------------------------------------------------------[/green]")
 

archivebox/cli/archivebox_install.py

@@ -17,12 +17,12 @@ def _resolve_install_targets(
     requested_names: tuple[str, ...],
 ) -> tuple[list[str], list[str]]:
     """Resolve plugin names and declared binary aliases, leaving unknown tokens raw."""
-    from archivebox.plugins.discovery import get_plugin_catalog
+    from abx_dl.models import discover_plugins, filter_plugins
 
-    catalog = get_plugin_catalog()
-    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in catalog}
+    plugins = discover_plugins(runtime="archivebox")
+    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in plugins}
     plugin_names_by_binary_alias: dict[str, set[str]] = {}
-    for plugin_name, plugin in catalog.items():
+    for plugin_name, plugin in plugins.items():
         for required_binary in plugin.config.required_binaries:
             aliases = {required_binary.name.lower()}
             template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", required_binary.name)
@@ -47,7 +47,7 @@ def _resolve_install_targets(
         else:
             raw_binary_names.append(name)
 
-    selected_plugins = catalog.select(requested_plugins) if requested_plugins else catalog.select([])
+    selected_plugins = filter_plugins(plugins, requested_plugins, include_providers=True) if requested_plugins else {}
     selected_plugin_names = {name.lower() for name in selected_plugins}
     raw_binary_names = [name for name in raw_binary_names if name.lower() not in selected_plugin_names]
     return sorted(selected_plugins), sorted(set(raw_binary_names))
@@ -58,6 +58,7 @@ def _install_raw_binary_names(binary_names: list[str], binproviders: str) -> Non
     from django.utils import timezone
 
     from archivebox.machine.models import Binary, Machine, _canonical_binary_name
+    from archivebox.services.runner import run_due_binary
 
     machine = Machine.current()
     for requested_name in binary_names:
@@ -79,13 +80,7 @@ def _install_raw_binary_names(binary_names: list[str], binproviders: str) -> Non
             binary.status = Binary.StatusChoices.QUEUED
             binary.retry_at = timezone.now()
             binary.save(update_fields=["binproviders", "overrides", "status", "retry_at", "modified_at"])
-        if binary.status == Binary.StatusChoices.INSTALLED:
-            continue
-        # The synchronous CLI needs the lifecycle's verified installation result,
-        # not the scheduler's "claimed work" result. Reuse its failure/requeue and
-        # health-stat handling so a failed install cannot exit successfully.
-        if not binary.install_claimed(lock_seconds=60):
-            raise click.ClickException(f"Binary {binary_name} is already being installed")
+        run_due_binary(binary, lock_seconds=60)
 
 
 def ensure_data_dir_lib_symlink(data_dir: Path, abxpkg_lib_dir: Path) -> Path | None:

archivebox/cli/archivebox_persona.py

@@ -27,10 +27,11 @@ Examples:
 __package__ = "archivebox.cli"
 __command__ = "archivebox persona"
 
+import os
 import sys
 import shutil
+import platform
 from pathlib import Path
-from dataclasses import replace
 from collections.abc import Iterable
 
 import rich_click as click
@@ -40,6 +41,116 @@ from archivebox.cli.cli_util import apply_filters
 from archivebox.personas import importers as persona_importers
 
 
+# =============================================================================
+# Browser Profile Locations
+# =============================================================================
+
+
+def get_chrome_user_data_dir() -> Path | None:
+    """Get the default Chrome user data directory for the current platform."""
+    system = platform.system()
+    home = Path.home()
+
+    if system == "Darwin":  # macOS
+        candidates = [
+            home / "Library" / "Application Support" / "Google" / "Chrome",
+            home / "Library" / "Application Support" / "Chromium",
+        ]
+    elif system == "Linux":
+        candidates = [
+            home / ".config" / "google-chrome",
+            home / ".config" / "chromium",
+            home / ".config" / "chrome",
+            home / "snap" / "chromium" / "common" / "chromium",
+        ]
+    elif system == "Windows":
+        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
+        candidates = [
+            local_app_data / "Google" / "Chrome" / "User Data",
+            local_app_data / "Chromium" / "User Data",
+        ]
+    else:
+        candidates = []
+
+    for candidate in candidates:
+        if candidate.exists() and (candidate / "Default").exists():
+            return candidate
+
+    return None
+
+
+def get_brave_user_data_dir() -> Path | None:
+    """Get the default Brave user data directory for the current platform."""
+    system = platform.system()
+    home = Path.home()
+
+    if system == "Darwin":
+        candidates = [
+            home / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser",
+        ]
+    elif system == "Linux":
+        candidates = [
+            home / ".config" / "BraveSoftware" / "Brave-Browser",
+        ]
+    elif system == "Windows":
+        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
+        candidates = [
+            local_app_data / "BraveSoftware" / "Brave-Browser" / "User Data",
+        ]
+    else:
+        candidates = []
+
+    for candidate in candidates:
+        if candidate.exists() and (candidate / "Default").exists():
+            return candidate
+
+    return None
+
+
+def get_edge_user_data_dir() -> Path | None:
+    """Get the default Edge user data directory for the current platform."""
+    system = platform.system()
+    home = Path.home()
+
+    if system == "Darwin":
+        candidates = [
+            home / "Library" / "Application Support" / "Microsoft Edge",
+        ]
+    elif system == "Linux":
+        candidates = [
+            home / ".config" / "microsoft-edge",
+            home / ".config" / "microsoft-edge-beta",
+            home / ".config" / "microsoft-edge-dev",
+        ]
+    elif system == "Windows":
+        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
+        candidates = [
+            local_app_data / "Microsoft" / "Edge" / "User Data",
+        ]
+    else:
+        candidates = []
+
+    for candidate in candidates:
+        if candidate.exists() and (candidate / "Default").exists():
+            return candidate
+
+    return None
+
+
+BROWSER_PROFILE_FINDERS = {
+    "chrome": get_chrome_user_data_dir,
+    "chromium": get_chrome_user_data_dir,  # Same locations
+    "brave": get_brave_user_data_dir,
+    "edge": get_edge_user_data_dir,
+}
+
+CHROMIUM_BROWSERS = {"chrome", "chromium", "brave", "edge"}
+
+
+# =============================================================================
+# Cookie Extraction via CDP
+# =============================================================================
+
 # =============================================================================
 # Validation Helpers
 # =============================================================================
@@ -107,9 +218,6 @@ def create_personas(
     names: Iterable[str],
     import_from: str | None = None,
     profile: str | None = None,
-    source: str | None = None,
-    browser_binary: str | None = None,
-    permissions: str | None = None,
 ) -> int:
     """
     Create Personas from names.
@@ -131,26 +239,25 @@ def create_personas(
         rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
         return 1
 
-    import_source = None
-    if source and not import_from:
-        rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
-        return 1
+    # Validate import source if specified
+    source_profile_dir = None
     if import_from:
-        try:
-            if source:
-                import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
-                if import_source.browser != "persona":
-                    import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
-            elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
-                import_source = persona_importers.resolve_custom_import_source(import_from)
-            else:
-                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
-            if browser_binary:
-                import_source = replace(import_source, browser_binary=browser_binary)
-        except ValueError as err:
-            rprint(f"[red]{err}[/red]", file=sys.stderr)
+        import_from = import_from.lower()
+        if import_from not in BROWSER_PROFILE_FINDERS:
+            rprint(f"[red]Unknown browser: {import_from}[/red]", file=sys.stderr)
+            rprint(f"[dim]Supported browsers: {', '.join(BROWSER_PROFILE_FINDERS.keys())}[/dim]", file=sys.stderr)
             return 1
 
+        source_profile_dir = BROWSER_PROFILE_FINDERS[import_from]()
+        if not source_profile_dir:
+            rprint(f"[red]Could not find {import_from} profile directory[/red]", file=sys.stderr)
+            return 1
+
+        rprint(f"[dim]Found {import_from} profile: {source_profile_dir}[/dim]", file=sys.stderr)
+
+        if profile is None and (source_profile_dir / "Default").exists():
+            profile = "Default"
+
     created_count = 0
     for name in name_list:
         name = name.strip()
@@ -163,25 +270,21 @@ def create_personas(
             rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
             continue
 
-        created = Persona.find_named(name) is None
-        persona = Persona.get_or_create_named(
-            name,
-            defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
-        )
+        persona, created = Persona.objects.get_or_create(name=name)
 
         if created:
             persona.ensure_dirs()
             created_count += 1
-
+            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
         else:
             rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
 
         cookies_file = Path(persona.path) / "cookies.txt"
 
         # Import browser profile if requested
-        if import_source is not None:
+        if import_from in CHROMIUM_BROWSERS and source_profile_dir is not None:
             try:
-                rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
+                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
                 import_result = persona_importers.import_persona_from_source(
                     persona,
                     import_source,
@@ -189,25 +292,20 @@ def create_personas(
                     import_cookies=True,
                     capture_storage=False,
                 )
-            except (Exception, KeyboardInterrupt) as e:
-                if created:
-                    shutil.rmtree(persona.path, ignore_errors=True)
-                    persona.delete()
+            except Exception as e:
                 rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
                 return 1
 
             if import_result.profile_copied:
                 rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
             if import_result.cookies_imported:
-                rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
+                rprint(f"[green]Extracted cookies to {cookies_file}[/green]", file=sys.stderr)
             elif not import_result.profile_copied:
                 rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
 
             for warning in import_result.warnings:
                 rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
 
-        if created:
-            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
         if not is_tty:
             write_record(
                 {
@@ -314,9 +412,7 @@ def update_personas(name: str | None = None) -> int:
             if persona_id:
                 persona = Persona.objects.get(id=persona_id)
             else:
-                persona = Persona.find_named(old_name)
-                if persona is None:
-                    raise Persona.DoesNotExist
+                persona = Persona.objects.get(name=old_name)
 
             # Apply updates from CLI flags
             if name:
@@ -328,7 +424,7 @@ def update_personas(name: str | None = None) -> int:
 
                 # Rename the persona directory too
                 old_path = persona.path
-                persona.name = persona.validate_name(name)
+                persona.name = name
                 new_path = persona.path
 
                 if old_path.exists() and old_path != new_path:
@@ -448,42 +544,11 @@ def main():
 
 @main.command("create")
 @click.argument("names", nargs=-1)
-@click.option(
-    "--import",
-    "import_from",
-    help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
-)
-@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
-@click.option(
-    "--browser-binary",
-    type=click.Path(exists=True, dir_okay=False),
-    help="Source Chromium browser executable; required for other Chromium-based browsers",
-)
+@click.option("--import", "import_from", help="Import profile from browser (chrome, chromium, brave, edge)")
 @click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
-@click.option(
-    "--permissions",
-    type=click.Choice(["public", "unlisted", "private"]),
-    help="Visibility for a newly created persona (existing personas are unchanged)",
-)
-def create_cmd(
-    names: tuple,
-    import_from: str | None,
-    profile: str | None,
-    source: str | None,
-    browser_binary: str | None,
-    permissions: str | None,
-):
+def create_cmd(names: tuple, import_from: str | None, profile: str | None):
     """Create Personas, optionally importing from a browser profile."""
-    sys.exit(
-        create_personas(
-            names,
-            import_from=import_from,
-            profile=profile,
-            source=source,
-            browser_binary=browser_binary,
-            permissions=permissions,
-        ),
-    )
+    sys.exit(create_personas(names, import_from=import_from, profile=profile))
 
 
 @main.command("list")
@@ -495,84 +560,6 @@ def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
     sys.exit(list_personas(name=name, name__icontains=name__icontains, limit=limit))
 
 
-@main.command("open")
-@click.argument("name", default="Default")
-def open_cmd(name: str):
-    """Open a persona's browser, including its imported logins."""
-    import json
-    import os
-    import subprocess
-
-    import psutil
-
-    from abx_plugins import get_plugins_dir
-    from archivebox.config.common import get_config
-    from archivebox.personas.models import Persona
-
-    valid, reason = validate_persona_name(name)
-    if not valid:
-        raise click.ClickException(reason)
-    persona = Persona.get_or_create_named(name)
-    persona.ensure_dirs()
-    config = get_config(persona=persona)
-    if os.environ.get("DISPLAY"):
-        from archivebox.config import CONSTANTS
-
-        for process in psutil.process_iter():
-            try:
-                args = process.cmdline()
-                profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
-                if (
-                    not profile
-                    or any(arg.startswith(("--type=", "--headless")) for arg in args)
-                    or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
-                    or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
-                ):
-                    continue
-                rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
-                process.wait()
-            except (psutil.NoSuchProcess, psutil.AccessDenied):
-                continue
-    plugins_dir = Path(get_plugins_dir()).resolve()
-    env = os.environ.copy()
-    dependencies = subprocess.run(
-        [
-            str(Path(sys.executable).with_name("abxpkg")),
-            "env",
-            "--install",
-            "--json",
-            f"--lib={config.ABXPKG_LIB_DIR}",
-            f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
-        ],
-        capture_output=True,
-        text=True,
-        env=env,
-    )
-    if dependencies.returncode:
-        raise click.ClickException(dependencies.stderr.strip())
-    env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
-    env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
-    env["ACTIVE_PERSONA"] = persona.name
-    env["PERSONAS_DIR"] = str(persona.path.parent)
-    env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
-    payload = config.model_dump(mode="json")
-    payload.update(
-        ACTIVE_PERSONA=persona.name,
-        CHROME_HEADLESS=False,
-        CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
-        AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
-        COOKIES_FILE=str(persona.COOKIES_FILE or ""),
-    )
-    script = Path(__file__).parent.parent / "personas" / "open_browser.js"
-    result = subprocess.run(
-        [env.get("NODE_BINARY") or "node", str(script)],
-        input=json.dumps(payload),
-        text=True,
-        env=env,
-    )
-    sys.exit(result.returncode)
-
-
 @main.command("update")
 @click.option("--name", "-n", help="Set new name")
 def update_cmd(name: str | None):

archivebox/cli/archivebox_remove.py

@@ -87,8 +87,9 @@ def remove(
     # Delete one snapshot at a time. Each ``.delete()`` is its own short
     # Django-atomic block, so the writer lock is released between rows and
     # an in-flight daemon transaction can interleave instead of deadlocking.
-    # Snapshot.delete() removes and verifies filesystem outputs first, outside
-    # the short database delete transaction. Failed cleanup retains the row.
+    # Filesystem cleanup for each row is scheduled via ``transaction.on_commit``
+    # in ``base_models/models.py`` and runs AFTER its row's tx commits — so
+    # rmtree doesn't hold the lock either.
     #
     deleted_snapshot_pks = []
     timed_out = False

archivebox/cli/archivebox_run.py

@@ -54,9 +54,12 @@ RUNNER_DAEMON_ENV = "ARCHIVEBOX_RUNNER_DAEMON"
 
 
 def _exit_daemon_runner_on_signal(sig: signal.Signals) -> None:
-    # A supervised daemon must exit non-zero so supervisord restarts it, while
-    # still unwinding the active crawl's hooks and persisted process state.
-    raise SystemExit(128 + sig.value)
+    # A supervised `archivebox run --daemon` is intentionally a disposable
+    # child. If it receives SIGINT/SIGTERM directly, exit with the conventional
+    # signal status so supervisord treats it as an unexpected worker death and
+    # restarts only the runner. The parent `archivebox server` owns supervisord
+    # shutdown and must not be pulled down by a killed daemon worker.
+    os._exit(128 + int(sig))
 
 
 def process_stdin_records() -> int:
@@ -69,8 +72,8 @@ def process_stdin_records() -> int:
 
     Outputs JSONL of all processed records (for chaining).
 
-    Handles Crawl and Snapshot work records. ArchiveResult records are accepted
-    as references to their parent Snapshot and plugin.
+    Handles any record type: Crawl, Snapshot, ArchiveResult.
+    Auto-cascades: Crawl → Snapshots → ArchiveResults.
 
     Returns exit code (0 = success, 1 = error).
     """
@@ -87,7 +90,6 @@ def process_stdin_records() -> int:
     )
     from archivebox.base_models.models import get_or_create_system_user_pk
     from archivebox.core.models import Snapshot, ArchiveResult
-    from archivebox.api.v1_core import _uuid_ref_query
     from archivebox.crawls.models import Crawl
     from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
     from archivebox.machine.models import Binary
@@ -154,15 +156,20 @@ def process_stdin_records() -> int:
                     queued_count += 1
 
             elif record_type == TYPE_ARCHIVERESULT:
-                snapshot_id = str(record.get("snapshot_id") or "")
-                plugin_name = str(record.get("plugin") or "")
-                archiveresult = None
-                if not snapshot_id and record_id:
-                    archiveresult = ArchiveResult.objects.filter(_uuid_ref_query("id", str(record_id))).select_related("snapshot").first()
-                    if archiveresult:
-                        snapshot_id = str(archiveresult.snapshot_id)
-                        plugin_name = plugin_name or archiveresult.plugin
-                snapshot = Snapshot.objects.filter(id=snapshot_id).first() if snapshot_id else None
+                archiveresult = ArchiveResult.from_json(record)
+                if archiveresult:
+                    if archiveresult.status in [
+                        ArchiveResult.StatusChoices.FAILED,
+                        ArchiveResult.StatusChoices.SKIPPED,
+                        ArchiveResult.StatusChoices.NORESULTS,
+                        ArchiveResult.StatusChoices.BACKOFF,
+                    ]:
+                        archiveresult.reset_for_retry()
+                    snapshot = archiveresult.snapshot
+                    plugin_name = archiveresult.plugin
+                else:
+                    snapshot = None
+                    plugin_name = None
 
                 if snapshot:
                     snapshot.queue_for_extraction()
@@ -170,7 +177,7 @@ def process_stdin_records() -> int:
                     snapshot_ids_by_crawl[crawl_id].add(str(snapshot.id))
                     if plugin_name:
                         plugin_names_by_crawl[crawl_id].add(str(plugin_name))
-                    output_records.append(archiveresult.to_json() if archiveresult else record)
+                    output_records.append(record if not archiveresult else archiveresult.to_json())
                     queued_count += 1
 
             elif record_type in {TYPE_BINARYREQUEST, TYPE_BINARY}:
@@ -228,6 +235,7 @@ def process_stdin_records() -> int:
                     crawl_id,
                     snapshot_ids=None if crawl_id in full_crawl_ids else sorted(snapshot_ids_by_crawl[crawl_id]),
                     selected_plugins=None if crawl_id in run_all_plugins_for_crawl else sorted(plugin_names_by_crawl[crawl_id]),
+                    selected_plugins_are_explicit=False,
                 )
     return 0
 
@@ -236,6 +244,7 @@ def run_runner(
     daemon: bool = False,
     crawl_id: str | None = None,
     maintenance_only: bool = False,
+    maintenance_batch_size: int | None = None,
 ) -> int:
     """
     Run the background runner loop.
@@ -282,26 +291,19 @@ def run_runner(
         # future ownership lease cannot hide it from the unified scheduler.
         if crawl is not None:
             crawl.update_and_requeue(retry_at=now, refresh=False)
-    # Interactivity follows terminal ownership, not the spelling of the command.
-    # A supervised add worker has pipes and receives terminal intent/answers from
-    # its parent; direct run owns the terminal itself. Daemons and redirected
-    # commands must never wait for a person who cannot answer their prompt.
-    # Trust the explicit terminal-owner flag, not Process.root: when add borrows
-    # a server-owned supervisord, its worker's OS ancestry still leads to server.
-    # Gating on root=ADD silently made the first Ctrl+C abort after takeover.
-    interactive_interrupts = not daemon and (
-        (sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())) or os.environ.get("ARCHIVEBOX_INTERACTIVE_INTERRUPTS") == "1"
-    )
+    # Only a foreground `archivebox add` gets the interactive "abort current
+    # hook, continue/retry, second Ctrl+C exits" flow. Server/update/run owned
+    # orchestrators should shut down immediately and cleanly on the first signal.
+    interactive_interrupts = current.root.process_type == Process.TypeChoices.ADD
     if daemon:
         os.environ[RUNNER_DAEMON_ENV] = "1"
 
-    shutdown_state = None
     try:
         with (
             foreground_shutdown_signals(
                 on_signal=_exit_daemon_runner_on_signal if daemon else None,
                 raise_on_first_signal=not daemon,
-            ) as shutdown_state,
+            ),
             foreground_parent_watchdog(enabled=not daemon),
         ):
             run_pending_crawls(
@@ -309,17 +311,12 @@ def run_runner(
                 crawl_id=crawl_id,
                 maintenance_only=maintenance_only,
                 interactive_interrupts=interactive_interrupts,
+                **({"maintenance_batch_size": maintenance_batch_size} if maintenance_batch_size else {}),
             )
-        if daemon and shutdown_state.signal_name:
-            return 128 + signal.Signals[shutdown_state.signal_name].value
         return 0
     except KeyboardInterrupt:
-        if daemon and shutdown_state and shutdown_state.signal_name:
-            return 128 + signal.Signals[shutdown_state.signal_name].value
-        return 130
+        return 0
     except asyncio.CancelledError as e:
-        if daemon and shutdown_state and shutdown_state.signal_name:
-            return 128 + signal.Signals[shutdown_state.signal_name].value
         if daemon:
             rprint(f"[red]Runner cancelled unexpectedly: {type(e).__name__}: {e}[/red]", file=sys.stderr)
             return 1
@@ -339,6 +336,13 @@ def run_runner(
 @click.option("--snapshot-id", help="Run one snapshot through its crawl")
 @click.option("--binary-id", help="Run one queued binary install directly on the bus")
 @click.option("--maintenance-only", is_flag=True, help="Only process sealed Snapshot maintenance and search-index backfills")
+@click.option(
+    "--maintenance-batch-size",
+    type=int,
+    default=None,
+    hidden=True,
+    help="Limit queued maintenance snapshots claimed per scheduler tick",
+)
 @click.option("--no-stdin", is_flag=True, hidden=True, help="Run the scheduler even when stdin is not a TTY")
 def main(
     daemon: bool,
@@ -346,6 +350,7 @@ def main(
     snapshot_id: str,
     binary_id: str,
     maintenance_only: bool,
+    maintenance_batch_size: int | None,
     no_stdin: bool,
 ):
     """
@@ -370,13 +375,13 @@ def main(
                 ),
                 foreground_parent_watchdog(enabled=False),
             ):
-                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only))
+                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
         except KeyboardInterrupt:
-            sys.exit(130)
+            sys.exit(0)
 
     with foreground_shutdown_signals(), foreground_parent_watchdog(enabled=not daemon):
         if snapshot_id:
-            sys.exit(run_snapshot_worker(snapshot_id, daemon=daemon))
+            sys.exit(run_snapshot_worker(snapshot_id))
 
         if binary_id:
             try:
@@ -396,22 +401,23 @@ def main(
         if crawl_id:
             sys.exit(
                 run_runner(
-                    daemon=daemon,
+                    daemon=False,
                     crawl_id=crawl_id,
                     maintenance_only=maintenance_only,
+                    maintenance_batch_size=maintenance_batch_size,
                 ),
             )
 
         if maintenance_only:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=True))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=True, maintenance_batch_size=maintenance_batch_size))
 
         if not no_stdin and not sys.stdin.isatty():
             sys.exit(process_stdin_records())
         else:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
 
 
-def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
+def run_snapshot_worker(snapshot_id: str) -> int:
     from archivebox.config import CONSTANTS
     from archivebox.core.takeover_util import enter_single_runner_gate
     from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
@@ -426,7 +432,6 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
         return 0
 
     snapshot = None
-    interactive_interrupts = not daemon and sys.stdin.isatty() and (sys.stdout.isatty() or sys.stderr.isatty())
     try:
         with foreground_shutdown_signals(), foreground_parent_watchdog():
             for _ in range(10):
@@ -435,7 +440,7 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
                     snapshot.update_and_requeue(retry_at=timezone.now())
                 elif snapshot.retry_at > timezone.now():
                     break
-                if not run_due_snapshot(snapshot, lock_seconds=60, interactive_interrupts=interactive_interrupts):
+                if not run_due_snapshot(snapshot, lock_seconds=60):
                     break
         return 0
     except KeyboardInterrupt:
@@ -448,7 +453,7 @@ def run_snapshot_worker(snapshot_id: str, *, daemon: bool = False) -> int:
                 snapshot.update_and_requeue(retry_at=timezone.now())
         except Exception:
             pass
-        return 130
+        return 0
     except Exception as e:
         rprint(f"[red]Runner error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
         import traceback

archivebox/cli/archivebox_schedule.py

@@ -37,7 +37,6 @@ def schedule(
         "created_schedule_ids": [],
         "disabled_count": 0,
         "run_all_enqueued": 0,
-        "run_all_maintained": 0,
         "active_schedule_ids": [],
     }
 
@@ -58,12 +57,12 @@ def schedule(
 
         created_by_id = get_or_create_system_user_pk()
         is_update_schedule = not import_path
-        template_urls = import_path or ""
+        template_urls = import_path or "archivebox://update"
         template_label = (f"Scheduled import: {template_urls}" if import_path else "Scheduled ArchiveBox update")[:64]
         template_notes = (
             f"Created by archivebox schedule for {template_urls}"
             if import_path
-            else "Created by archivebox schedule to run recurring ArchiveBox maintenance."
+            else "Created by archivebox schedule to queue recurring archivebox://update maintenance crawls."
         )
 
         template = Crawl.objects.create(
@@ -77,6 +76,7 @@ def schedule(
             retry_at=None,
             config={
                 "DEPTH": 0 if is_update_schedule else depth,
+                "SCHEDULE_KIND": "update" if is_update_schedule else "crawl",
                 # Caller-supplied overrides (e.g. {"ONLY_NEW": False}) win over the
                 # template defaults. Anything left unset falls through to the
                 # standard config stack at crawl-resolution time.
@@ -86,7 +86,6 @@ def schedule(
         crawl_schedule = CrawlSchedule.objects.create(
             template=template,
             schedule=schedule_str,
-            config={**template.config, "SCHEDULE_KIND": "update" if is_update_schedule else "crawl"},
             is_enabled=True,
             label=template_label,
             notes=template_notes,
@@ -120,19 +119,12 @@ def schedule(
 
     if run_all:
         enqueued = 0
-        maintained = 0
         now = timezone.now()
         for scheduled_crawl in schedules:
-            if scheduled_crawl.dispatch(queued_at=now) is None:
-                maintained += 1
-            else:
-                enqueued += 1
+            scheduled_crawl.enqueue(queued_at=now)
+            enqueued += 1
         result["run_all_enqueued"] = enqueued
-        result["run_all_maintained"] = maintained
         print(f"[green]\\[*] Enqueued {enqueued} scheduled crawl(s) immediately.[/green]")
-        if maintained:
-            run_pending_crawls(maintenance_only=True)
-            print(f"[green]\\[*] Ran {maintained} scheduled maintenance update(s).[/green]")
         if enqueued:
             print(
                 "[yellow]\\[*] Start `archivebox server`, `archivebox run --daemon`, or `archivebox schedule --foreground` to process the queued crawls.[/yellow]",
@@ -179,7 +171,7 @@ def schedule(
 @click.option("--clear", is_flag=True, help="Disable all currently enabled schedules")
 @click.option("--show", is_flag=True, help="Print all currently enabled schedules")
 @click.option("--foreground", "-f", is_flag=True, help="Run the global crawl runner in the foreground (no crontab required)")
-@click.option("--run-all", is_flag=True, help="Dispatch all enabled schedules immediately")
+@click.option("--run-all", is_flag=True, help="Enqueue all enabled schedules immediately and process them once")
 @click.argument("import_path", required=False)
 @docstring(schedule.__doc__)
 def main(**kwargs):

archivebox/cli/archivebox_server.py

@@ -46,13 +46,13 @@ def _split_bind_spec(spec: str) -> tuple[str, str]:
     """Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
 
     The empty strings stand in for "not provided"; the caller fills in
-    defaults. Bracketed IPv6 literals like ``[::1]:5797`` are handled.
+    defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
     """
     spec = (spec or "").strip()
     if not spec:
         return "", ""
     if spec.startswith("["):
-        # Bracketed IPv6: ``[::1]`` or ``[::1]:5797``
+        # Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
         end = spec.find("]")
         if end == -1:
             return spec, ""  # malformed; let validator reject it
@@ -77,11 +77,11 @@ def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
     (normalized to ``127.0.0.1``). Bare hostnames are rejected because the
     bind address feeds Daphne, which has to listen on a numeric address;
     public hostnames belong in ``BASE_URL`` instead. Empty values fall back
-    to ``127.0.0.1`` / ``5797``.
+    to ``127.0.0.1`` / ``8000``.
     """
     raw_host, raw_port = _split_bind_spec(spec)
     host = raw_host.strip()
-    port = (raw_port or "").strip() or "5797"
+    port = (raw_port or "").strip() or "8000"
 
     if host == "" or host.lower() == "localhost":
         host = "127.0.0.1"
@@ -198,7 +198,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
             "[yellow]    routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
         )
         print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
+            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
         )
         if config.USES_SUBDOMAIN_ROUTING:
             print(
@@ -221,7 +221,7 @@ def _print_server_startup_warnings(config, host: str, port: str) -> None:
             "[yellow]    a reverse proxy / ingress / public hostname, e.g.[/yellow]",
         )
         print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:5797[/yellow]",
+            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
         )
         print()
 
@@ -254,18 +254,12 @@ def server(
         print()
 
     # First non-empty positional arg is the bind spec; otherwise inherit from
-    # config (which defaults to "127.0.0.1:5797"). _parse_and_validate_bind_spec
+    # config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
     # hard-errors on hostnames so the rest of the server can assume a numeric
     # bind host.
     bind_spec = next((arg for arg in runserver_args if arg), "")
     host, port = _parse_and_validate_bind_spec(bind_spec)
 
-    from archivebox.personas.models import Persona
-
-    # Older collections deferred this until /add/. Reconcile before serving any
-    # requests, and report ambiguous profile names before Chrome can use them.
-    Persona.get_or_create_default().ensure_dirs()
-
     if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
         from archivebox.core.routes_util import build_admin_url
 
@@ -369,10 +363,12 @@ def server(
     command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
 
     def still_owns_runtime_stack() -> bool:
-        # This is polled at the log-tail cadence. Keep the autocommit
-        # connection open so SQLite can reuse its page cache between polls;
-        # reconnecting here repeatedly rereads the process table while idle.
-        return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
+        from django.db import connections
+
+        try:
+            return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
+        finally:
+            connections.close_all()
 
     shutdown_state = None
     try:
@@ -420,13 +416,8 @@ def server(
     except KeyboardInterrupt:
         pass
     finally:
-        from django.db import connections
-
-        try:
-            if not shutdown_state or not shutdown_state.signal_name:
-                command.mark_exited()
-        finally:
-            connections.close_all()
+        if not shutdown_state or not shutdown_state.signal_name:
+            command.mark_exited()
     print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
 
 

archivebox/cli/archivebox_snapshot.py

@@ -44,7 +44,6 @@ SNAPSHOT_LIST_CHUNK_SIZE = 5000
 
 def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
     from archivebox.config import VERSION
-    from archivebox.config.common import redact_sensitive_config
     from archivebox.core.models import SnapshotTag
 
     fields = (
@@ -54,9 +53,6 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
         "title",
         "bookmarked_at",
         "created_at",
-        "modified_at",
-        "config",
-        "notes",
         "timestamp",
         "depth",
         "status",
@@ -84,9 +80,6 @@ def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
                 "tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
                 "bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
                 "created_at": row["created_at"].isoformat() if row["created_at"] else None,
-                "modified_at": row["modified_at"].isoformat() if row["modified_at"] else None,
-                "config": redact_sensitive_config(row["config"]),
-                "notes": row["notes"],
                 "timestamp": row["timestamp"],
                 "depth": row["depth"],
                 "status": row["status"],
@@ -298,7 +291,7 @@ def list_snapshots(
     Exit codes:
         0: Success (even if no results)
     """
-    from archivebox.misc.jsonl import write_records
+    from archivebox.misc.jsonl import write_record
 
     output_formats = sum(bool(output_format) for output_format in (csv, as_json, as_html))
     if output_formats > 1:
@@ -351,7 +344,9 @@ def list_snapshots(
         return 0
 
     if not is_tty:
-        count = write_records(iter_snapshot_json(queryset))
+        for snapshot_json in iter_snapshot_json(queryset):
+            write_record(snapshot_json)
+            count += 1
         rprint(f"[dim]Listed {count} snapshots[/dim]", file=sys.stderr)
         return 0
 

archivebox/cli/archivebox_update.py

@@ -7,7 +7,6 @@ import os
 import asyncio
 import shlex
 import time
-from itertools import islice
 
 from typing import TYPE_CHECKING, Any
 from collections.abc import Iterable
@@ -53,20 +52,17 @@ def _get_search_indexing_plugins() -> list[str]:
 def _build_filtered_snapshots_queryset(
     **kwargs,
 ):
+    from archivebox.core.models import Snapshot
     from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
 
     limit = kwargs.pop("limit", None)
-    reverse = kwargs.pop("reverse", False)
     snapshots = build_snapshot_queryset(**kwargs)
-    snapshots = snapshots.order_by(kwargs.get("sort") or "-created_at", "-id")
-    if reverse:
-        snapshots = snapshots.reverse()
     if kwargs.get("resume"):
-        snapshots = snapshots.filter(**{"timestamp__gte" if reverse else "timestamp__lte": kwargs["resume"]})
+        snapshots = snapshots.filter(timestamp__lte=kwargs["resume"])
     snapshots = snapshots.select_related("crawl")
     if limit is not None and limit > 0:
         snapshot_ids = list(snapshots.values_list("id", flat=True)[:limit])
-        snapshots = snapshots.filter(id__in=snapshot_ids)
+        snapshots = Snapshot.objects.filter(id__in=snapshot_ids).select_related("crawl")
 
     return snapshots
 
@@ -77,73 +73,109 @@ def reindex_snapshots(
     search_plugins: list[str],
     batch_size: int,
     collect_ids: bool = False,
-    reverse: bool = False,
     wait_for_turn=None,
 ) -> dict[str, Any]:
     from archivebox.cli.archivebox_extract import run_plugins
     from archivebox.core.models import ArchiveResult, Snapshot
-    from django.db.models import Exists, OuterRef
+    from abx_dl.models import discover_plugins
 
     # Search backfill is the one maintenance hook allowed to execute without
     # reopening a Snapshot. Restrict that exception to already-sealed rows;
     # every open lifecycle state remains owned by the normal runner lifecycle.
     snapshots = snapshots.filter(status=Snapshot.StatusChoices.SEALED)
 
-    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
-    print(f"[*] Reindexing missing search indexes with: {', '.join(search_plugins)}")
+    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "skipped_queued": 0, "reindexed": 0, "snapshot_ids": []}
+    records: list[dict[str, str]] = []
+    plugins_by_name = discover_plugins(runtime="archivebox")
+    required_hooks_by_plugin = {
+        plugin_name: frozenset(hook.name for hook in plugins_by_name[plugin_name].filter_hooks("Snapshot"))
+        for plugin_name in search_plugins
+        if plugin_name in plugins_by_name
+    }
 
-    completed_statuses = [ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS]
-    for plugin_name in search_plugins:
-        completed_result = ArchiveResult.objects.filter(
-            snapshot_id=OuterRef("pk"),
-            plugin=plugin_name,
-            status__in=completed_statuses,
-        )
-        candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False)
-        candidates = candidates.order_by("created_at", "id") if reverse else candidates.order_by("-created_at", "-id")
-        remaining = candidates.select_related(None).only("id", "timestamp", "created_at").paged_iterator(chunk_size=batch_size)
-        while True:
-            if wait_for_turn:
-                wait_for_turn()
-            batch = list(islice(remaining, batch_size))
-            if not batch:
-                break
-            records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
-            stats["processed"] += len(batch)
-            stats["requested"] += len(batch)
-            if collect_ids:
-                stats["snapshot_ids"].extend(str(snapshot.id) for snapshot in batch)
-            exit_code = run_plugins(
-                args=(),
-                records=records,
-                plugins=plugin_name,
-                wait=True,
-                emit_results=False,
-                show_progress=False,
+    total = snapshots.count()
+    print(f"[*] Reindexing {total} snapshots with search plugins: {', '.join(search_plugins)}")
+
+    def run_batch() -> None:
+        if not records:
+            return
+        if wait_for_turn:
+            wait_for_turn()
+        batch_records = list(records)
+        snapshot_ids = {record["snapshot_id"] for record in batch_records}
+        plugin_names = {record["plugin"] for record in batch_records}
+        queued_rows = {
+            (str(snapshot_id), plugin_name, hook_name)
+            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
+                snapshot_id__in=snapshot_ids,
+                plugin__in=plugin_names,
+                status=ArchiveResult.StatusChoices.QUEUED,
+            ).values_list("snapshot_id", "plugin", "hook_name")
+        }
+        records_to_queue = []
+        for record in batch_records:
+            snapshot_id = record["snapshot_id"]
+            plugin_name = record["plugin"]
+            required_hooks = required_hooks_by_plugin.get(plugin_name, frozenset())
+            if required_hooks and all((snapshot_id, plugin_name, hook_name) in queued_rows for hook_name in required_hooks):
+                stats["skipped_queued"] += 1
+                continue
+            records_to_queue.append(record)
+        if not records_to_queue:
+            print(
+                f"    [{stats['processed']}/{total}] Already queued {len(batch_records)} index jobs",
             )
-            if exit_code != 0:
-                raise SystemExit(exit_code)
-            stats["reindexed"] += len(batch)
-            print(f"    [{plugin_name}] indexed {stats['reindexed']} missing snapshots")
-    return stats
-
-
-def run_scheduled_maintenance(*, batch_size: int = 500) -> dict[str, Any]:
-    """Queue stale filesystem rows and backfill missing search facts."""
-    from archivebox.core.models import Snapshot
-
-    filesystem_stats = process_all_db_snapshots(batch_size=batch_size)
-    search_plugins = _get_search_indexing_plugins()
-    search_stats = (
-        reindex_snapshots(
-            Snapshot.objects.filter(fs_version=Snapshot._fs_current_version()),
-            search_plugins=search_plugins,
-            batch_size=batch_size,
+            records.clear()
+            return
+        # `archivebox update --index-only` intentionally breaks the usual
+        # "runner discovers work" rule by inserting synthetic queued
+        # ArchiveResult rows for search backends. run_plugins() keeps this as
+        # statement-sized UPDATE/bulk_create work, then bumps Snapshot.retry_at
+        # so the orchestrator owns actual hook execution. Paused snapshots stay
+        # PAUSED; run_due_snapshot restores retry_at=MAX after targeted rows
+        # finish.
+        exit_code = run_plugins(
+            args=(),
+            records=records_to_queue,
+            wait=False,
+            emit_results=False,
+            show_progress=False,
+            preserve_queued=True,
         )
-        if search_plugins
-        else {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
-    )
-    return {"filesystem": filesystem_stats, "search": search_stats}
+        if exit_code != 0:
+            raise SystemExit(exit_code)
+        stats["queued"] += len(records_to_queue)
+        print(
+            f"    [{stats['processed']}/{total}] Queued {len(records_to_queue)} index jobs for orchestrator",
+        )
+        records.clear()
+
+    for snapshot in snapshots.select_related("crawl").paged_iterator(chunk_size=batch_size):
+        try:
+            stats["processed"] += 1
+
+            if _get_snapshot_crawl(snapshot) is None:
+                continue
+
+            if collect_ids:
+                stats["snapshot_ids"].append(str(snapshot.id))
+            for plugin_name in search_plugins:
+                records.append(
+                    {
+                        "type": "ArchiveResult",
+                        "snapshot_id": str(snapshot.id),
+                        "plugin": plugin_name,
+                    },
+                )
+                stats["requested"] += 1
+            if len(records) >= batch_size:
+                run_batch()
+        except KeyboardInterrupt as err:
+            err.archivebox_resume = snapshot.timestamp
+            raise
+
+    run_batch()
+    return stats
 
 
 @enforce_types
@@ -165,18 +197,18 @@ def update(
     continuous: bool = False,
     index_only: bool = False,
     migrate_only: bool = False,
-    rescan: bool = False,
-    reverse: bool = False,
     stop_daemon_stack: bool = True,
 ) -> None:
     """
-    Update the database without forcing a filesystem scan by default.
+    Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
 
-    --migrate-only: eagerly finish pending filesystem migrations for known rows.
-    --index-only: reconcile known snapshot assets/metadata and backfill search.
-    --rescan: discover filesystem orphans first, then repair known snapshots.
-    Explicit scans default to newest first; --reverse selects oldest first.
-    Filters select existing database snapshots for targeted updates.
+    Three-phase operation (without filters):
+    - Phase 1: Drain old archive/ dirs by moving to new fs location (0.8.x → 0.9.x)
+    - Phase 2: O(n) scan over entire DB from most recent to least recent
+    - No orphan scans needed (trust 1:1 mapping between DB and filesystem after phase 1)
+
+    With filters: Only phase 2 (DB query), no filesystem operations.
+    Without filters: All phases (full update).
     """
 
     from rich import print
@@ -200,6 +232,7 @@ def update(
     from archivebox.core.takeover_util import (
         command_owns_foreground_runner,
         current_command,
+        foreground_runner_owner,
         standby_until_foreground_runner_needed,
     )
     from archivebox.workers.supervisord_util import run_runner_worker, stop_own_supervisord_process
@@ -252,28 +285,17 @@ def update(
     exit_code = 0
 
     try:
-        if rescan and (is_filtered_update or resume or continuous):
-            raise click.UsageError("--rescan cannot be combined with filters, --resume, or --continuous; rerun it to resume automatically.")
-        if rescan:
-            migrate_only = True  # Discovery must not schedule new captures.
         wait_for_turn()
 
         with foreground_shutdown_signals(), foreground_parent_watchdog():
             while True:
-                if rescan:
-                    from archivebox.cli.rescan import rescan_snapshots
-
-                    rescan_snapshots(
-                        wait_for_turn=wait_for_turn,
-                        reverse=reverse,
-                        scan_legacy=lambda: drain_old_archive_dirs(batch_size=batch_size, reverse=reverse),
-                    )
-                do_migrate = migrate_only or is_filtered_update and not index_only
-                do_index = index_only
-                if not do_migrate and not do_index:
-                    print(
-                        "[*] Collection ready. Filesystem migrations remain lazy; use --migrate-only, --index-only, or --rescan for explicit scans.",
-                    )
+                do_migrate = migrate_only or not index_only
+                do_index = index_only or not migrate_only
+                do_run_until_idle = do_migrate or do_index
+                ran_post_migrate_runner = False
+                full_update_empty = False
+                maintenance_work_queued = False
+                runner_work_queued = False
 
                 if do_migrate:
                     if (
@@ -305,86 +327,178 @@ def update(
                             after=after,
                             resume=resume,
                             batch_size=batch_size,
-                            queue_for_archiving=not migrate_only,
-                            reverse=reverse,
+                            queue_for_archiving=do_run_until_idle,
                             wait_for_turn=wait_for_turn,
                         )
                         print_stats(stats)
                         touched_snapshot_ids.update(stats.get("snapshot_ids", []))
+                        maintenance_work_queued = stats.get("queued", 0) > 0
+                        runner_work_queued = runner_work_queued or maintenance_work_queued
                     else:
                         stats_combined = {"phase1": {}, "phase2": {}}
 
-                        print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
+                        print("[*] Phase 1: Draining old archive/ directories (0.8.x → 0.9.x migration)...")
+                        stats_combined["phase1"] = drain_old_archive_dirs(
+                            resume_from=resume,
+                            batch_size=batch_size,
+                        )
+
+                        print("[*] Phase 2: Processing all database snapshots (most recent first)...")
                         stats_combined["phase2"] = process_all_db_snapshots(
                             batch_size=batch_size,
                             resume=resume,
                             wait_for_turn=wait_for_turn,
-                            reverse=reverse,
-                            migrate=True,
                         )
                         print_combined_stats(stats_combined)
-                    # The due selectors are indexed and cheap when empty, so
-                    # always drain them instead of preceding the runner with
-                    # whole-table counts merely to decide whether to call it.
-                    print("[*] Phase 3: Running filesystem maintenance until idle...")
-                    if is_filtered_update:
-                        for snapshot_id in sorted(touched_snapshot_ids, reverse=not reverse):
-                            run_scoped_runner("--snapshot-id", snapshot_id)
-                    elif migrate_only:
-                        run_scoped_runner("--maintenance-only")
-                    else:
-                        run_scoped_runner()
+                        full_update_empty = (
+                            stats_combined["phase1"].get("processed", 0) == 0 and stats_combined["phase2"].get("snapshots", 0) == 0
+                        )
+                        maintenance_work_queued = any(
+                            (
+                                stats_combined["phase1"].get("queued", 0),
+                                stats_combined["phase2"].get("queued", 0),
+                                stats_combined["phase2"].get("crawls_sealed", 0),
+                            ),
+                        )
+                        runner_work_queued = runner_work_queued or maintenance_work_queued
+
+                    if do_run_until_idle:
+                        # Filesystem migration is maintenance on existing
+                        # Snapshot rows: Snapshot.save() moves archive/<ts> to
+                        # the current output_dir and preserves the lifecycle
+                        # status. Drain those retry_at ticks before queuing
+                        # search backfill below. Otherwise the sealed search
+                        # runner branch correctly sees queued ArchiveResult
+                        # rows first, runs the targeted plugins, and may leave
+                        # the fs_version maintenance tick hidden behind that
+                        # plugin work until another update pass.
+                        if full_update_empty:
+                            print("[*] No snapshots or legacy archive directories found; skipping filesystem maintenance runner.")
+                        elif not maintenance_work_queued:
+                            print("[*] No filesystem maintenance work queued; skipping filesystem maintenance runner.")
+                        else:
+                            print("[*] Phase 3: Running filesystem maintenance until idle...")
+                        if full_update_empty:
+                            pass
+                        elif not maintenance_work_queued:
+                            pass
+                        elif is_filtered_update:
+                            if not touched_snapshot_ids:
+                                print("[*] No matching snapshots queued work for the runner.")
+                            for snapshot_id in sorted(touched_snapshot_ids):
+                                run_scoped_runner("--snapshot-id", snapshot_id)
+                        else:
+                            run_scoped_runner("--maintenance-only", "--maintenance-batch-size", str(batch_size))
+                        ran_post_migrate_runner = True
 
                 if do_index:
-                    from archivebox.cli.rescan import reconcile_known_snapshots
-
-                    snapshots = _build_filtered_snapshots_queryset(
-                        filter_patterns=filter_patterns,
-                        filter_type=filter_type,
-                        status=status,
-                        url__icontains=url__icontains,
-                        url__istartswith=url__istartswith,
-                        tag=tag,
-                        crawl_id=crawl_id,
-                        limit=limit,
-                        sort=sort,
-                        search=search,
-                        before=before,
-                        after=after,
-                        resume=resume,
-                        reverse=reverse,
-                    )
-                    reconcile_known_snapshots(snapshots, wait_for_turn=wait_for_turn)
-                    search_plugins = _get_search_indexing_plugins()
-                    if not search_plugins:
-                        print("[*] No search indexing plugins are available, nothing to backfill.")
+                    if full_update_empty:
+                        print("[*] No snapshots found; skipping search indexing backfill.")
                     else:
-                        snapshots = _build_filtered_snapshots_queryset(
-                            filter_patterns=filter_patterns,
-                            filter_type=filter_type,
-                            status=status,
-                            url__icontains=url__icontains,
-                            url__istartswith=url__istartswith,
-                            tag=tag,
-                            crawl_id=crawl_id,
-                            limit=limit,
-                            sort=sort,
-                            search=search,
-                            before=before,
-                            after=after,
-                            resume=resume,
-                            reverse=reverse,
+                        search_plugins = _get_search_indexing_plugins()
+                        if not search_plugins:
+                            print("[*] No search indexing plugins are available, nothing to backfill.")
+                        else:
+                            snapshots = _build_filtered_snapshots_queryset(
+                                filter_patterns=filter_patterns,
+                                filter_type=filter_type,
+                                status=status,
+                                url__icontains=url__icontains,
+                                url__istartswith=url__istartswith,
+                                tag=tag,
+                                crawl_id=crawl_id,
+                                limit=limit,
+                                sort=sort,
+                                search=search,
+                                before=before,
+                                after=after,
+                                resume=resume,
+                            )
+                            from django.db.models import Exists, OuterRef, Q
+                            from django.utils import timezone
+                            from archivebox.core.models import ArchiveResult, Snapshot
+
+                            scoped_snapshot_ids = snapshots.order_by().values("id") if is_filtered_update else None
+                            queued_index_results = ArchiveResult.objects.filter(
+                                status=ArchiveResult.StatusChoices.QUEUED,
+                                plugin__in=search_plugins,
+                            )
+                            if scoped_snapshot_ids is not None:
+                                queued_index_results = queued_index_results.filter(snapshot_id__in=scoped_snapshot_ids)
+
+                            if queued_index_results.exists():
+                                runner_work_queued = True
+                                now = timezone.now()
+                                queued_result_for_snapshot = queued_index_results.filter(snapshot_id=OuterRef("pk"))
+                                snapshots_to_wake = (
+                                    Snapshot.objects.filter(status=Snapshot.StatusChoices.SEALED)
+                                    .annotate(
+                                        has_queued_index_result=Exists(queued_result_for_snapshot),
+                                    )
+                                    .filter(
+                                        has_queued_index_result=True,
+                                    )
+                                    .filter(
+                                        Q(retry_at__isnull=True) | Q(retry_at__gt=now),
+                                    )
+                                )
+                                if scoped_snapshot_ids is not None:
+                                    snapshots_to_wake = snapshots_to_wake.filter(id__in=scoped_snapshot_ids)
+                                woken_count = snapshots_to_wake.update(
+                                    retry_at=now,
+                                    modified_at=now,
+                                )
+                                print(
+                                    "[*] Existing queued search index jobs found; "
+                                    f"skipping backfill scan and waking {woken_count} snapshot(s) for the runner.",
+                                )
+                            else:
+                                collect_index_ids = (
+                                    is_filtered_update
+                                    or foreground_runner_owner(
+                                        data_dir=CONSTANTS.DATA_DIR,
+                                        exclude_id=command.id,
+                                    )
+                                    is not None
+                                )
+                                stats = reindex_snapshots(
+                                    snapshots,
+                                    search_plugins=search_plugins,
+                                    batch_size=batch_size,
+                                    collect_ids=collect_index_ids,
+                                    wait_for_turn=wait_for_turn,
+                                )
+                                print_index_stats(stats)
+                                touched_snapshot_ids.update(stats.get("snapshot_ids", []))
+                                runner_work_queued = runner_work_queued or stats["queued"] > 0
+
+                if do_run_until_idle and (do_index or not ran_post_migrate_runner):
+                    # Search/index backfill intentionally queues targeted
+                    # ArchiveResult rows without reopening sealed snapshots.
+                    # This second runner pass drains those plugin
+                    # rows after filesystem maintenance has had its own turn.
+                    # For a normal unfiltered `archivebox update`, keep the
+                    # historical final pass broad enough to resume genuinely
+                    # queued/interrupted crawl work after maintenance is done.
+                    if full_update_empty:
+                        print("[*] No snapshots found; skipping queued/interrupted crawl runner.")
+                    elif not runner_work_queued:
+                        print("[*] No queued/interrupted crawl work found; skipping queued/interrupted crawl runner.")
+                    else:
+                        print("[*] Phase 3: Running queued/interrupted crawl work until idle...")
+                    if full_update_empty:
+                        pass
+                    elif not runner_work_queued:
+                        pass
+                    elif touched_snapshot_ids and is_filtered_update:
+                        if not touched_snapshot_ids:
+                            print("[*] No matching snapshots queued work for the runner.")
+                        for snapshot_id in sorted(touched_snapshot_ids):
+                            run_scoped_runner("--snapshot-id", snapshot_id)
+                    else:
+                        run_scoped_runner(
+                            *(["--maintenance-only", "--maintenance-batch-size", str(batch_size)] if index_only or migrate_only else []),
                         )
-                        stats = reindex_snapshots(
-                            snapshots,
-                            search_plugins=search_plugins,
-                            batch_size=batch_size,
-                            collect_ids=is_filtered_update,
-                            reverse=reverse,
-                            wait_for_turn=wait_for_turn,
-                        )
-                        print_index_stats(stats)
-                        touched_snapshot_ids.update(stats.get("snapshot_ids", []))
 
                 if not continuous:
                     break
@@ -396,10 +510,6 @@ def update(
         exit_code = 130
         exact_resume = err.__dict__.get("archivebox_resume")
         resume_cmd = ["archivebox", "update"]
-        if rescan:
-            resume_cmd.append("--rescan")
-        if reverse:
-            resume_cmd.append("--reverse")
         if migrate_only:
             resume_cmd.append("--migrate-only")
         if index_only:
@@ -447,7 +557,7 @@ def update(
             stop_own_supervisord_process()
 
 
-def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500, reverse: bool = False) -> dict[str, int]:
+def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500) -> dict[str, int]:
     """
     Drain old archive/ directories (0.8.x → 0.9.x migration).
 
@@ -513,18 +623,18 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
 
     # Scan real legacy directories only; these still contain data to migrate.
     entries = [
-        (float(e.name), e.path)
+        (e.stat().st_mtime, e.path)
         for e in all_entries
         if e.is_dir(follow_symlinks=False) and Snapshot.is_legacy_archive_dir(Path(e.path))  # Skip symlinks and 0.9.x roots
     ]
-    entries.sort(reverse=not reverse)  # Newest first
+    entries.sort(reverse=True)  # Newest first
     print(f"[*] Found {len(entries)} old directories to drain")
 
     for mtime, entry_path in entries:
         entry_path = Path(entry_path)
 
         # Resume from timestamp if specified
-        if resume_from and (entry_path.name < resume_from if reverse else entry_path.name > resume_from):
+        if resume_from and entry_path.name > resume_from:
             continue
 
         stats["processed"] += 1
@@ -626,15 +736,19 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
     return stats
 
 
-def process_all_db_snapshots(
-    batch_size: int = 500,
-    resume: str | None = None,
-    wait_for_turn=None,
-    reverse: bool = False,
-    migrate: bool = False,
-) -> dict[str, int]:
-    """Queue only snapshots whose indexed filesystem version is stale."""
+def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
+    """
+    O(n) scan over entire DB from most recent to least recent.
+
+    For each snapshot:
+      1. Reconcile index.json with DB (merge titles, tags, archive results)
+      2. Mark migrated snapshots sealed unless explicitly re-queued elsewhere
+
+    No orphan detection needed - we trust 1:1 mapping between DB and filesystem
+    after Phase 1 has drained all old archive/ directories.
+    """
     from archivebox.core.models import Snapshot
+    from archivebox.crawls.models import Crawl
     from django.db.models import Q
     from django.utils import timezone
 
@@ -647,41 +761,128 @@ def process_all_db_snapshots(
         "sealed": 0,
         "crawls_sealed": 0,
     }
-    queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
+    current_fs_version = Snapshot._fs_current_version()
+
+    queryset = Snapshot.objects.all()
     if resume:
-        queryset = queryset.filter(**{"timestamp__gte" if reverse else "timestamp__lte": resume})
-    initial_now = timezone.now()
-    rows_to_wake = queryset if migrate else queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
-    ordered = rows_to_wake.order_by("created_at", "id") if reverse else rows_to_wake.order_by("-created_at", "-id")
-    remaining = ordered.select_related("crawl__created_by").paged_iterator(chunk_size=batch_size)
-    while True:
+        queryset = queryset.filter(timestamp__lte=resume)
+    total = queryset.count()
+    stats["snapshots"] = total
+    print(f"[*] Processing {total} snapshots from database (most recent first)...")
+
+    def update_in_batches(rows, *, label: str, **updates) -> int:
+        updated = 0
+        checked = 0
+        while True:
+            if wait_for_turn:
+                wait_for_turn()
+            batch = list(rows.only("id", "modified_at").order_by("-timestamp")[:batch_size])
+            if not batch:
+                if updated:
+                    print(f"    [{label}] complete: {updated} rows updated")
+                return updated
+            checked += len(batch)
+            print(f"    [{label}] updating next {len(batch)} rows (seen {checked})...")
+            for snapshot in batch:
+                # This maintenance scan intentionally bypasses save(); it is
+                # only normalizing scheduler fields, and Snapshot.save() may
+                # do filesystem migration work that belongs in the runner.
+                # Guard each single-row UPDATE with modified_at so stale scan
+                # pages cannot overwrite newer runner/admin writes.
+                updated += int(
+                    snapshot.safe_update(
+                        updates,
+                        refresh=False,
+                        extra_filter={"modified_at": snapshot.modified_at},
+                    ),
+                )
+            print(f"    [{label}] updated {updated} rows so far")
+
+    now = timezone.now()
+    updated_rows = update_in_batches(
+        queryset.exclude(
+            status__in=[
+                Snapshot.StatusChoices.QUEUED,
+                Snapshot.StatusChoices.STARTED,
+                Snapshot.StatusChoices.PAUSED,
+                Snapshot.StatusChoices.SEALED,
+            ],
+        ),
+        label="snapshot status normalization",
+        status=Snapshot.StatusChoices.SEALED,
+        retry_at=None,
+        modified_at=now,
+    )
+    stats["sealed"] += updated_rows
+    stats["updated_db"] += updated_rows
+
+    fs_version_rows = queryset.exclude(fs_version=current_fs_version).filter(Q(retry_at__isnull=True) | Q(retry_at__gt=now))
+    stale_batch = []
+
+    def queue_stale_fs_batch() -> None:
+        if not stale_batch:
+            return
         if wait_for_turn:
             wait_for_turn()
-        batch = list(islice(remaining, batch_size))
-        if not batch:
-            break
         now = timezone.now()
+        # Do not bump fs_version here. The orchestrator calls Snapshot.save(),
+        # which performs the idempotent filesystem migration and commits the new
+        # fs_version in the same serialized worker path as normal crawls.
         updated = 0
-        for snapshot in batch:
-            if migrate:
-                from archivebox.services.runner import run_snapshot_maintenance
-
-                if wait_for_turn:
-                    wait_for_turn()
-                run_snapshot_maintenance(str(snapshot.id))
-                continue
+        for snapshot in stale_batch:
+            # Each row gets its own short autocommit UPDATE because this scan
+            # can touch millions of snapshots while a server is also alive.
+            # The modified_at predicate is the CAS guard: if the runner or
+            # admin changed the snapshot after paged_iterator read it, skip it
+            # and let the newer state decide whether migration is still due.
             updated += int(
                 snapshot.safe_update(
-                    {"retry_at": now, "modified_at": now},
+                    {
+                        "retry_at": now,
+                        "modified_at": now,
+                    },
                     refresh=False,
                     extra_filter={"fs_version": snapshot.fs_version},
                 ),
             )
-        stats["processed"] += len(batch)
+        stats["processed"] += len(stale_batch)
         stats["updated_db"] += updated
         stats["queued"] += updated
-        print(f"    Queued {stats['queued']} stale filesystem snapshots so far...")
-    stats["snapshots"] = stats["processed"]
+        print(f"    [{stats['processed']}/{total}] Queued {updated} filesystem migrations for orchestrator...")
+        stale_batch.clear()
+
+    for snapshot in (
+        fs_version_rows.only("id", "crawl_id", "timestamp", "fs_version", "modified_at")
+        .order_by("-timestamp")
+        .paged_iterator(chunk_size=batch_size)
+    ):
+        try:
+            stale_batch.append(snapshot)
+            if len(stale_batch) >= batch_size:
+                queue_stale_fs_batch()
+        except KeyboardInterrupt as err:
+            err.archivebox_resume = snapshot.timestamp
+            raise
+    queue_stale_fs_batch()
+
+    now = timezone.now()
+    # Crawls with no open child snapshots are already finished. Seal them here
+    # instead of waking the foreground runner; otherwise migration/update can
+    # accidentally re-enter full crawl execution for historical rows.
+    stats["crawls_sealed"] = (
+        Crawl.objects.filter(
+            status__in=Crawl.RUNNABLE_STATES,
+        )
+        .exclude(
+            snapshot_set__status__in=Snapshot.OPEN_STATES,
+        )
+        .update(
+            status=Crawl.StatusChoices.SEALED,
+            retry_at=None,
+            modified_at=now,
+        )
+    )
+    stats["updated_db"] += stats["crawls_sealed"]
     return stats
 
 
@@ -701,7 +902,6 @@ def process_filtered_snapshots(
     resume: str | None,
     batch_size: int,
     queue_for_archiving: bool = True,
-    reverse: bool = False,
     wait_for_turn=None,
 ) -> dict[str, Any]:
     """Process snapshots matching filters (DB query only)."""
@@ -724,7 +924,6 @@ def process_filtered_snapshots(
         before=before,
         after=after,
         resume=resume,
-        reverse=reverse,
     )
 
     total = snapshots.count()
@@ -743,10 +942,8 @@ def process_filtered_snapshots(
             stats["snapshot_ids"].append(str(snapshot.id))
             update_values = {}
             updated = 0
-            if not queue_for_archiving and snapshot.fs_migration_needed:
-                from archivebox.services.runner import run_snapshot_maintenance
-
-                run_snapshot_maintenance(str(snapshot.id))
+            if not isinstance(snapshot.current_step, int):
+                update_values["current_step"] = 0
             if queue_for_archiving:
                 update_values.update(
                     {
@@ -832,9 +1029,10 @@ def print_index_stats(stats: dict[str, Any]) -> None:
 
     print(f"""
 [green]Search Reindex Complete[/green]
-  Missing index runs: {stats["processed"]}
-  Requested runs:    {stats.get("requested", 0)}
-  Indexed snapshots: {stats.get("reindexed", 0)}
+  Scanned rows:      {stats["processed"]}
+  Requested jobs:    {stats.get("requested", stats["queued"])}
+  Queued index jobs: {stats["queued"]}
+  Already queued:    {stats.get("skipped_queued", 0)}
 """)
 
 
@@ -844,12 +1042,6 @@ def print_index_stats(stats: dict[str, Any]) -> None:
 @click.option("--continuous", is_flag=True, help="Run continuously as background worker")
 @click.option("--index-only", is_flag=True, help="Backfill available search indexes from existing archived content")
 @click.option("--migrate-only", is_flag=True, help="Only migrate filesystem and update database/index state")
-@click.option("--reverse", is_flag=True, help="Process oldest first instead of newest first for explicit scans")
-@click.option(
-    "--rescan",
-    is_flag=True,
-    help="Scan all snapshot directories for orphans and metadata repairs (slow, opt-in, safe to restart)",
-)
 @snapshot_filter_options(default_filter_type="exact")
 @docstring(update.__doc__)
 def main(**kwargs):

archivebox/cli/archivebox_version.py

@@ -80,6 +80,20 @@ def _render_binary_abspath(abspath: str):
     return Text(abspath, style="green")
 
 
+def _binary_record_matches_runtime(installed, lib_dir: Path) -> bool:
+    if not installed or not installed.is_valid or not installed.version:
+        return False
+    try:
+        abspath = Path(installed.abspath).expanduser().resolve(strict=False)
+        if not abspath.exists():
+            return False
+        if installed.binprovider not in {"", "env", "apt", "brew"}:
+            abspath.relative_to(lib_dir)
+    except (OSError, ValueError):
+        return False
+    return True
+
+
 def _binary_row_dedupe_key(
     *,
     display_name: str,
@@ -101,7 +115,6 @@ def _binary_row_dedupe_key(
 def version(
     quiet: bool = False,
     binaries: Iterable[str] = (),
-    no_cache: bool = False,
 ) -> list[str]:
     """Print the ArchiveBox version, debug metadata, and installed dependency versions"""
 
@@ -114,6 +127,7 @@ def version(
 
     from rich.panel import Panel
     from rich.console import Console
+    from rich.live import Live
     from abx_dl.tables import binary_dependency_status, binary_dependency_table
 
     from archivebox.config import CONSTANTS
@@ -249,38 +263,47 @@ def version(
     prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
     failures = []
     seen_failures: set[str] = set()
+    seen_rows: set[tuple[str, str, str, str]] = set()
 
-    from archivebox.plugins.discovery import get_enabled_plugins, get_plugin_catalog
+    from archivebox.plugins.discovery import get_enabled_plugins
     from abx_dl.config import get_required_binary_requests
     from abx_dl.dependencies import resolve_binary_requests
+    from abx_dl.models import discover_plugins, filter_plugins
     from abx_dl.orchestrator import create_bus
     from abxpkg.binary_service import BinaryEvent, BinaryService
 
-    plugins = get_plugin_catalog()
-    enabled_plugin_names = set(get_enabled_plugins(config=config))
-    runtime_config = normalize_runtime_config(dict(config.items()), json_safe=False)
+    plugins = discover_plugins(runtime="archivebox")
+    enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
+    enabled_plugin_names = set(enabled_plugins)
+    runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
     derived_config: dict[str, object] = {}
+    db_binaries = {}
+    db_available = False
     if in_data_dir:
         try:
             from archivebox.config.django import setup_django
 
             setup_django()
 
-            from archivebox.machine.models import Machine
+            from archivebox.machine.models import Machine, Binary
 
             machine = Machine.current()
             derived_config = normalize_runtime_config(machine.config, json_safe=False)
+            for binary in Binary.objects.filter(machine=machine).order_by("name", "-modified_at"):
+                if _binary_record_matches_runtime(binary, config.ABXPKG_LIB_DIR):
+                    db_binaries.setdefault(binary.name, binary)
+            db_available = True
 
         except Exception as e:
             prnt()
-            prnt("", f"[yellow]Warning: Could not query collection machine config; resolving through abxpkg: {e}[/yellow]")
+            prnt("", f"[yellow]Warning: Could not query collection binary records; resolving through abxpkg: {e}[/yellow]")
 
     declared_binary_specs: dict[str, dict[str, object]] = {}
-    binary_records_by_plugin: dict[str, list[dict[str, object]]] = {}
     for plugin_name, plugin in plugins.items():
         if not plugin_may_have_requested_binary(plugin):
             continue
-        plugin_requested = plugin_name.lower() in {name.lower() for name in requested_names}
+        plugin_requested = bool(requested_names)
+        plugin_enabled = plugin_name in enabled_plugin_names
         binary_records = get_required_binary_requests(
             plugin,
             plugin.config.required_binaries,
@@ -288,10 +311,6 @@ def version(
             derived_overrides=derived_config,
             run_output_dir=CONSTANTS.DATA_DIR,
         )
-        # Render the exact requests resolved below. Rehydrating all schemas for
-        # display doubled config work, and could describe different requests if
-        # a config file changed while dependency resolution was running.
-        binary_records_by_plugin[plugin_name] = binary_records
         for binary_record in binary_records:
             actual_name = str(binary_record["name"])
             logical_name = (
@@ -302,13 +321,10 @@ def version(
             display_name = logical_name
             if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
                 continue
-            # Resolve through abxpkg, never the DB's last-known installation row.
-            # Its cache validates current paths, permissions, package fingerprints
-            # and runtime options. Forcing cold resolution here rehashed every
-            # large binary and reran every version subprocess on each diagnostic,
-            # even in an image that had already validated all of them at build.
-            # Keep an explicit full recheck for diagnosing a suspect installation.
-            binary_record["no_cache"] = no_cache
+            if not plugin_enabled and not requested_names:
+                continue
+            if _binary_record_matches_runtime(db_binaries.get(logical_name), config.ABXPKG_LIB_DIR):
+                continue
             signature = json.dumps(binary_record, sort_keys=True, default=str)
             declared_binary_specs.setdefault(signature, binary_record)
 
@@ -326,41 +342,126 @@ def version(
 
         loaded_binaries = asyncio.run(resolve_declared_binaries())
 
-    rows_by_binary: dict[str, dict] = {}
+    rows: list[dict[str, object]] = []
     any_rows = False
     any_available = False
     compact_paths = console.is_terminal
-    for plugin_name, plugin in plugins.items():
-        if not plugin_may_have_requested_binary(plugin):
-            continue
-        plugin_requested = plugin_name.lower() in {name.lower() for name in requested_names}
-        plugin_enabled = plugin_name in enabled_plugin_names
-        disabled_by = ", ".join(
-            f"{dependency.enabled_key}=False"
-            for dependency in plugins.select([plugin_name]).values()
-            if not config.get(dependency.enabled_key, True)
-        )
-        binary_records = binary_records_by_plugin[plugin_name]
-        for binary_record in binary_records:
-            binary_record["no_cache"] = no_cache
-            actual_name = str(binary_record["name"])
-            logical_name = (
-                Path(actual_name).expanduser().name
-                if ("/" in actual_name or "\\" in actual_name or actual_name.startswith("~"))
-                else actual_name
-            )
-            display_name = logical_name
-            if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
+    live_enabled = console.is_terminal
+    live_cm = Live(binary_dependency_table(rows), console=console, refresh_per_second=8) if live_enabled else None
+
+    def emit_row(row: dict[str, object]) -> None:
+        rows.append(row)
+        if live_cm is not None:
+            live_cm.update(binary_dependency_table(rows), refresh=True)
+
+    if live_cm is not None:
+        live_cm.start()
+    try:
+        for plugin_name, plugin in plugins.items():
+            if not plugin_may_have_requested_binary(plugin):
                 continue
+            plugin_requested = bool(requested_names)
+            plugin_enabled = plugin_name in enabled_plugin_names
+            binary_records = get_required_binary_requests(
+                plugin,
+                plugin.config.required_binaries,
+                overrides=runtime_config,
+                derived_overrides=derived_config,
+                run_output_dir=CONSTANTS.DATA_DIR,
+            )
+            for binary_record in binary_records:
+                actual_name = str(binary_record["name"])
+                logical_name = (
+                    Path(actual_name).expanduser().name
+                    if ("/" in actual_name or "\\" in actual_name or actual_name.startswith("~"))
+                    else actual_name
+                )
+                display_name = logical_name
+                if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
+                    continue
 
-            loaded = loaded_binaries[json.dumps(binary_record, sort_keys=True, default=str)]
-            abspath = loaded.abspath if loaded is not None else ""
-            version_str = str(loaded.version or "unknown") if loaded is not None else "unknown"
-            provider = str(loaded.binprovider or "env") if loaded is not None else "env"
-            valid = loaded is not None and Path(abspath).is_file()
+                installed = db_binaries.get(logical_name) if db_available else None
+                if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
+                    abspath = installed.abspath
+                    version_str = (installed.version or "unknown")[:15]
+                    provider = (installed.binprovider or "env")[:8]
+                    valid = True
+                elif not plugin_enabled and not requested_names:
+                    # `archivebox version` is expected to verify the active runtime, not
+                    # cold-load every optional plugin provider. Migration and status
+                    # checks often run with PLUGINS narrowed to a tiny set; resolving
+                    # disabled plugin binaries there can spend most of the command on
+                    # providers the current collection will never execute.
+                    continue
+                else:
+                    loaded = loaded_binaries[json.dumps(binary_record, sort_keys=True, default=str)]
+                    abspath = loaded.abspath if loaded is not None else ""
+                    version_str = str(loaded.version or "unknown")[:15] if loaded is not None else "unknown"
+                    provider = str(loaded.binprovider or "env")[:8] if loaded is not None else "env"
+                    valid = loaded is not None
+
+                any_rows = True
+                if valid:
+                    display_path = (
+                        _format_binary_abspath(
+                            abspath,
+                            pwd=Path.cwd(),
+                            lib_dir=config.ABXPKG_LIB_DIR,
+                            personas_dir=CONSTANTS.PERSONAS_DIR,
+                            home=Path.home(),
+                        )
+                        if compact_paths
+                        else abspath
+                    )
+                    rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
+                    any_available = True
+                else:
+                    rendered_path = "[grey53]not installed[/grey53]"
+                    if plugin_enabled and display_name not in seen_failures:
+                        failures.append(display_name)
+                        seen_failures.add(display_name)
+
+                row_key = _binary_row_dedupe_key(
+                    display_name=display_name,
+                    valid=valid,
+                    version=version_str if valid else "-",
+                    provider=provider if valid else "-",
+                    abspath=abspath,
+                )
+                if row_key in seen_rows:
+                    continue
+                seen_rows.add(row_key)
+
+                emit_row(
+                    {
+                        "plugin": plugin_name,
+                        "status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
+                        "binary": display_name,
+                        "version": version_str if valid else "-",
+                        "provider": provider if valid else "-",
+                        "path": rendered_path,
+                        "style": "" if plugin_enabled else "dim",
+                    },
+                )
+
+        if db_available:
+            for binary_name, installed in db_binaries.items():
+                if requested_names and binary_name not in requested_names:
+                    continue
+                abspath = installed.abspath
+                version_str = (installed.version or "unknown")[:15]
+                provider = (installed.binprovider or "env")[:8]
+                row_key = _binary_row_dedupe_key(
+                    display_name=binary_name,
+                    valid=True,
+                    version=version_str,
+                    provider=provider,
+                    abspath=abspath,
+                )
+                if row_key in seen_rows:
+                    continue
+                seen_rows.add(row_key)
 
-            any_rows = True
-            if valid:
                 display_path = (
                     _format_binary_abspath(
                         abspath,
@@ -372,67 +473,24 @@ def version(
                     if compact_paths
                     else abspath
                 )
-                rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
+                emit_row(
+                    {
+                        "plugin": "(database)",
+                        "status": binary_dependency_status(enabled=True, valid=True),
+                        "binary": binary_name,
+                        "version": version_str,
+                        "provider": provider,
+                        "path": _render_binary_abspath(display_path) if compact_paths else display_path,
+                        "style": "dim",
+                    },
+                )
+                any_rows = True
                 any_available = True
-            else:
-                rendered_path = "[grey53]not installed[/grey53]"
-                if plugin_enabled and display_name not in seen_failures:
-                    failures.append(display_name)
-                    seen_failures.add(display_name)
-
-            if not plugin_enabled:
-                rendered_path = f"[grey53]disabled by {disabled_by or 'PLUGINS configuration'}[/grey53]"
-
-            row = {
-                "plugin": plugin_name,
-                "status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
-                "binary": display_name,
-                "version": version_str if valid else "-",
-                "provider": provider if valid else "-",
-                "path": rendered_path,
-                "style": "" if plugin_enabled else "dim",
-                "enabled": plugin_enabled,
-                "valid": valid,
-                "plugins": [plugin_name],
-                "disabled_by": [disabled_by or "PLUGINS configuration"] if not plugin_enabled else [],
-            }
-            existing = rows_by_binary.get(display_name)
-            if existing is None:
-                rows_by_binary[display_name] = row
-            else:
-                consumers = list(dict.fromkeys([*existing["plugins"], plugin_name]))
-                disabled_reasons = list(dict.fromkeys([*existing["disabled_by"], *row["disabled_by"]]))
-                # Disabled consumers cannot mask an active requirement. If any
-                # active request fails (e.g. a version floor), keep that failure.
-                if plugin_enabled and (not existing["enabled"] or not valid):
-                    existing.update(row)
-                existing["plugins"] = consumers
-                existing["disabled_by"] = disabled_reasons
-                existing["plugin"] = ", ".join(consumers)
-                if not existing["enabled"]:
-                    existing["path"] = f"[grey53]disabled by {', '.join(disabled_reasons)}[/grey53]"
-
-    rows = list(rows_by_binary.values())
-
-    if console.is_terminal and console.width < 120:
-        from rich.text import Text
-
-        for row in rows:
-            heading = Text.from_markup(str(row["status"]))
-            heading.append(f" {row['binary']} {row['version']} ({row['provider']}) · {row['plugin']}")
-            prnt(heading, style=str(row["style"]))
-            path = row["path"]
-            prnt(Text.assemble("   ", path if isinstance(path, Text) else Text.from_markup(str(path))), style=str(row["style"]))
-    else:
-        table = binary_dependency_table(rows)
-        table.expand = False
-        table.title = None
-        table.box = None
-        for column in table.columns:
-            column.width = None
-            column.max_width = None
-            column.ratio = None
-        prnt(table)
+    finally:
+        if live_cm is not None:
+            live_cm.stop()
+        else:
+            prnt(binary_dependency_table(rows))
 
     if not any_rows:
         prnt("", "[grey53]No required binaries declared for discovered plugins.[/grey53]")
@@ -489,11 +547,6 @@ def version(
     is_flag=True,
     help="Only print ArchiveBox version number and nothing else. (equivalent to archivebox --version)",
 )
-@click.option(
-    "--no-cache",
-    is_flag=True,
-    help="Reprobe binary versions and hashes instead of reusing validated provider caches.",
-)
 @click.option(
     "--binaries",
     "-b",

archivebox/cli/rescan.py

@@ -1,239 +0,0 @@
-"""Explicit discovery of portable snapshots; no checkpoint or default tree walk."""
-
-import json
-import os
-import uuid
-from itertools import groupby
-from pathlib import Path
-
-from django.contrib.auth import get_user_model
-from django.core.exceptions import ValidationError
-from django.db import IntegrityError
-from rich import print
-
-from archivebox.config import CONSTANTS
-from archivebox.core.models import ArchiveResult, Snapshot
-from archivebox.crawls.models import Crawl
-from archivebox.misc.util import parse_date
-
-
-def _directories(parent: Path):
-    # Names encode dates / UUIDv7 creation time. Newest first within each
-    # subtree, with bounded memory and no traversal into plugin payloads.
-    with os.scandir(parent) as entries:
-        paths = [Path(entry.path) for entry in entries if entry.is_dir(follow_symlinks=False)]
-    yield from sorted(paths, key=lambda path: path.name, reverse=True)
-
-
-def _snapshot_directories(reverse=False):
-    dates = []
-    for user in _directories(CONSTANTS.USERS_DIR):
-        root = user / CONSTANTS.SNAPSHOTS_DIR_NAME
-        if root.is_symlink() or not root.is_dir():
-            continue
-        dates.extend(_directories(root))
-    dates.sort(key=lambda path: path.name, reverse=not reverse)
-    for _, date_group in groupby(dates, key=lambda path: path.name):
-        directories = [directory for date in date_group for domain in _directories(date) for directory in _directories(domain)]
-        yield from sorted(directories, key=lambda path: path.name, reverse=not reverse)
-
-
-def _fields(record, names):
-    return {name: record[name] for name in names.split() if name in record}
-
-
-def _dates(record):
-    return {name: parse_date(record[name]) for name in ("created_at", "modified_at") if record.get(name)}
-
-
-def _restore_dates(obj, record):
-    dates = _dates(record)
-    if dates:
-        type(obj).objects.filter(pk=obj.pk).update(**dates)
-        for name, value in dates.items():
-            setattr(obj, name, value)
-
-
-def _reconcile_directory(directory: Path) -> str:
-    snapshot_id = uuid.UUID(directory.name)
-    snapshot = Snapshot.objects.select_related("crawl__created_by").filter(pk=snapshot_id).first()
-    index = directory / CONSTANTS.JSONL_INDEX_FILENAME
-    if index.is_symlink():
-        raise ValueError("metadata must not be a symlink")
-    if not index.exists():
-        if snapshot is None or Path(snapshot.output_dir) != directory or snapshot.status != "sealed":
-            raise ValueError("missing index.jsonl; cannot reconstruct an unknown or active snapshot")
-        snapshot.write_index_jsonl(output_dir=directory)
-        snapshot.reconcile_filesystem_links()
-        return "Repaired"
-
-    records = [json.loads(line) for line in index.read_text().splitlines() if line.strip()]
-    if any(not isinstance(record, dict) for record in records):
-        raise ValueError("metadata records must be JSON objects")
-    snapshots = [record for record in records if record.get("type") == "Snapshot"]
-    if len(snapshots) != 1:
-        raise ValueError("expected exactly one Snapshot record")
-    record = snapshots[0]
-    if uuid.UUID(record["id"]) != snapshot_id:
-        raise ValueError("snapshot ID conflicts with directory name")
-    if record.get("status") != "sealed":
-        raise ValueError("snapshot is not sealed; finish the source capture first")
-    if record.get("fs_version") not in {Snapshot._fs_current_version(), *Snapshot._FS_VERSION_MIGRATION_PATHS}:
-        raise ValueError("unsupported filesystem version")
-    crawl_id = uuid.UUID(record["crawl_id"])
-    owner = directory.parents[3].name
-    crawl_records = [item for item in records if item.get("type") == "Crawl" and uuid.UUID(item["id"]) == crawl_id]
-    crawl_record = crawl_records[0] if crawl_records else {}
-    if crawl_record.get("created_by", owner) != owner:
-        raise ValueError("crawl owner conflicts with directory path")
-    crawl = Crawl.objects.select_related("created_by").filter(pk=crawl_id).first()
-    imported_crawl = crawl is None
-    if crawl and crawl.created_by.username != owner:
-        raise ValueError("existing crawl ID belongs to a different owner")
-    if snapshot and (snapshot.url != record["url"] or snapshot.crawl_id != crawl_id or snapshot.timestamp != record["timestamp"]):
-        raise ValueError("existing snapshot ID has conflicting identity; refusing to overwrite")
-    if snapshot and Path(snapshot.output_dir) != directory:
-        raise ValueError("existing snapshot has a different storage path")
-
-    # Validate identities before inserting anything. Never turn an ID collision
-    # into a new ID, and never overwrite a newer destination row.
-    # write_index_jsonl retains historical hook records before appending the
-    # canonical DB rows. The final record for each hook is authoritative.
-    results = {(item.get("plugin"), item.get("hook_name", "")): item for item in records if item.get("type") == "ArchiveResult"}.values()
-    pending = []
-    for result in results:
-        if uuid.UUID(result["snapshot_id"]) != snapshot_id:
-            raise ValueError("ArchiveResult belongs to another snapshot")
-        # Legacy JSON conversion produced records without IDs. Recover these
-        # by hook identity while preserving strict collision checks for IDs.
-        if not result.get("id"):
-            if not ArchiveResult.objects.filter(
-                snapshot_id=snapshot_id,
-                plugin=result["plugin"],
-                hook_name=result.get("hook_name", ""),
-            ).exists():
-                pending.append(result)
-            continue
-        result_id = uuid.UUID(result["id"])
-        existing = ArchiveResult.objects.filter(pk=result_id).first()
-        identity = (snapshot_id, result["plugin"], result.get("hook_name", ""))
-        if existing:
-            if (existing.snapshot_id, existing.plugin, existing.hook_name) != identity:
-                raise ValueError("existing ArchiveResult ID has conflicting identity")
-        elif ArchiveResult.objects.filter(snapshot_id=snapshot_id, plugin=identity[1], hook_name=identity[2]).exists():
-            raise ValueError("existing hook has a different ArchiveResult ID")
-        else:
-            pending.append(result)
-
-    imported = snapshot is None
-    if imported:
-        if Snapshot.objects.filter(timestamp=record["timestamp"]).exists():
-            raise ValueError("timestamp belongs to another snapshot")
-        if Snapshot.objects.filter(crawl_id=crawl_id, url=record["url"]).exists():
-            raise ValueError("crawl URL belongs to another snapshot")
-        user, _ = get_user_model().objects.get_or_create(
-            username=owner,
-            defaults={"is_active": False, "password": "!"},
-        )
-        if crawl is None:
-            # Old exports contain only crawl_id. Keep that identity but report
-            # that absent crawl metadata cannot be reconstructed exactly.
-            if not crawl_record:
-                print(f"[yellow]    {directory}: no Crawl record; preserving ID with recovered metadata[/yellow]")
-            crawl = Crawl(
-                id=crawl_id,
-                created_by=user,
-                status="sealed",
-                retry_at=None,
-                **_fields(crawl_record, "urls max_depth config tags_str label"),
-            )
-            crawl.save(force_insert=True)
-            _restore_dates(crawl, crawl_record)
-        snapshot = Snapshot.create_from_directory(directory, crawl=crawl, preserve_identity=True)
-        if snapshot is None:
-            raise ValueError("invalid snapshot metadata")
-        snapshot.status = "sealed"
-        snapshot.retry_at = None
-        if "config" not in record:
-            snapshot.config = {"PERMISSIONS": "private"}
-        if Path(snapshot.get_storage_path_for_version(Snapshot._fs_current_version())) != directory:
-            raise ValueError("metadata does not match its owner/date/domain directory")
-        snapshot.save(force_insert=True)
-        _restore_dates(snapshot, record)
-
-    # Each insert is durable. An interruption after the Snapshot insert is not
-    # completion: the next pass still imports missing result IDs and tags.
-    for result in pending:
-        snapshot._create_archive_result_if_missing(result, {}, preserve_identity=bool(result.get("id")))
-    if any(not result.get("id") for result in pending):
-        snapshot.hydrate_archiveresult_output_metadata(snapshot_dir=directory)
-        snapshot.write_index_jsonl(output_dir=directory)
-    tags_before = set(snapshot.tags.values_list("name", flat=True))
-    snapshot._merge_tags_from_index(record)
-    tags_changed = tags_before != set(snapshot.tags.values_list("name", flat=True))
-    link = Path(snapshot.crawl.output_dir) / "snapshots" / snapshot.extract_domain_from_url(snapshot.url) / str(snapshot.id)
-    links_ok = link.is_symlink() and link.resolve() == directory.resolve()
-    if not links_ok:
-        snapshot.reconcile_filesystem_links()
-    if snapshot.fs_migration_needed:
-        snapshot.migrate_filesystem_to_current_version(source_dir=directory)
-    if imported:
-        _restore_dates(snapshot, record)
-    if imported_crawl:
-        _restore_dates(crawl, crawl_record)
-    # Leave source metadata untouched, including process/binary records that
-    # are historical evidence rather than runnable destination process state.
-    return "Imported" if imported else "Repaired" if pending or tags_changed or not links_ok else "Unchanged"
-
-
-def reconcile_known_snapshots(snapshots, wait_for_turn=None):
-    """Explicit metadata / asset repair; normal updates never call this scan."""
-    for snapshot in snapshots.filter(status="sealed").paged_iterator():
-        if wait_for_turn:
-            wait_for_turn()
-        if snapshot.fs_migration_needed:
-            snapshot.migrate_filesystem_to_current_version()
-        directory = Path(snapshot.output_dir)
-        if not directory.is_dir():
-            # Canceling queued metadata can seal a row before the runner ever
-            # creates storage. There is nothing on disk to reconcile for it.
-            if snapshot.downloaded_at is None and not snapshot.archiveresult_set.exists():
-                continue
-            raise ValueError(f"Missing snapshot directory: {directory}")
-        _reconcile_directory(directory)
-        for result in snapshot.archiveresult_set.all():
-            result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
-
-
-def rescan_snapshots(wait_for_turn=None, reverse=False, scan_legacy=None):
-    stats = dict.fromkeys(("Imported", "Repaired", "Unchanged", "Errors"), 0)
-    print("[*] Rescanning snapshot directories (O(N), orphans first; completed records are reused)...")
-    known = {str(pk) for pk in Snapshot.objects.values_list("pk", flat=True)}
-    # Two streaming passes avoid enumerating the entire archive before the
-    # first import. Rows imported before interruption move to the second pass
-    # next time; only one day's directory names are held in memory at a time.
-    for existing in (False, True):
-        if existing and scan_legacy:
-            scan_legacy()
-        for directory in _snapshot_directories(reverse=reverse):
-            if (directory.name.replace("-", "") in known) != existing:
-                continue
-            if wait_for_turn:
-                wait_for_turn()
-            try:
-                outcome = _reconcile_directory(directory)
-                if directory.name.replace("-", "") in known:
-                    for result in ArchiveResult.objects.filter(snapshot_id=uuid.UUID(directory.name)):
-                        result.update_output_metadata_from_filesystem(snapshot_dir=directory, full_scan=True)
-                stats[outcome] += 1
-                if outcome != "Unchanged":
-                    print(f"    {outcome}: {directory}")
-            except (OSError, ValueError, KeyError, TypeError, ValidationError, IntegrityError) as err:
-                stats["Errors"] += 1
-                print(f"[yellow]    Unresolved: {directory}: {err}[/yellow]")
-            if sum(stats.values()) % 100 == 0:
-                print("    " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
-    print("    " + ", ".join(f"{key}: {value}" for key, value in stats.items()))
-    if stats["Errors"]:
-        raise SystemExit(1)
-    return stats

archivebox/config/allauth.py

@@ -0,0 +1,58 @@
+__package__ = "archivebox.config"
+
+from pydantic import Field, PrivateAttr, field_validator
+
+from archivebox.config.configset import BaseConfigSet
+
+
+class AllauthConfig(BaseConfigSet):
+    """
+    django-allauth authentication configuration.
+
+    Only activates if ALLAUTH_ENABLED=True and django-allauth is installed.
+    """
+
+    toml_section_header: str = "ALLAUTH_CONFIG"
+    _scope: str = PrivateAttr(default="server")
+
+    ALLAUTH_ENABLED: bool = Field(default=False)
+    SOCIALACCOUNT_ENABLED: bool = Field(default=False)
+    REGISTRATION_ENABLED: bool = Field(default=True)
+    REGISTRATION_MODE: str = Field(default="open")
+    EMAIL_VERIFICATION: str = Field(default="none")
+    DEFAULT_USER_PERMISSIONS: str = Field(default="none")
+    SOCIALACCOUNT_PROVIDERS: dict = Field(default_factory=dict)
+
+    # SECURITY: Auto-connecting social accounts by verified email is disabled by default.
+    # When True, a social login whose provider-reported email matches an existing local
+    # account will be silently linked to that account without user confirmation.
+    # This creates an account-takeover risk if the OIDC/OAuth provider's email
+    # verification is weak or if the provider itself is compromised.
+    # Only enable this if you fully trust all configured social providers AND require
+    # mandatory email verification (EMAIL_VERIFICATION=mandatory) in your setup.
+    # See: https://docs.allauth.org/en/latest/socialaccount/configuration.html
+    SOCIALACCOUNT_EMAIL_AUTO_CONNECT: bool = Field(default=False)
+
+    @field_validator("REGISTRATION_MODE", mode="after")
+    @classmethod
+    def validate_registration_mode(cls, v: str) -> str:
+        allowed = {"open", "invite", "approval"}
+        if v not in allowed:
+            raise ValueError(f"REGISTRATION_MODE must be one of: {', '.join(sorted(allowed))}")
+        return v
+
+    @field_validator("EMAIL_VERIFICATION", mode="after")
+    @classmethod
+    def validate_email_verification(cls, v: str) -> str:
+        allowed = {"mandatory", "optional", "none"}
+        if v not in allowed:
+            raise ValueError(f"EMAIL_VERIFICATION must be one of: {', '.join(sorted(allowed))}")
+        return v
+
+    @field_validator("DEFAULT_USER_PERMISSIONS", mode="after")
+    @classmethod
+    def validate_default_user_permissions(cls, v: str) -> str:
+        allowed = {"none", "admin"}
+        if v not in allowed:
+            raise ValueError(f"DEFAULT_USER_PERMISSIONS must be one of: {', '.join(sorted(allowed))}")
+        return v

archivebox/config/collection.py

@@ -17,7 +17,7 @@ CONFIG_FILE_HEADER = (
     "# You can add options here manually in INI format, or automatically by running:\n"
     "#    archivebox config --set KEY=VALUE\n"
     "#\n"
-    "# This file is kept in sync 1:1 with the current Machine.config in the DB —\n"
+    "# This file is kept in sync 1:1 with Machine.config in the index DB —\n"
     "# editing either side propagates to the other. ``archivebox init`` reads\n"
     "# this file on startup; the admin Machine.config editor writes both.\n"
     "#\n"
@@ -160,7 +160,7 @@ def _coerce_from_str_dict(file_config: dict[str, str]) -> dict[str, Any]:
     so they round-trip through INI's string-only storage. When reading the
     file back into ``Machine.config`` (a JSONField that holds native types)
     those strings have to be decoded — otherwise downstream consumers like
-    ``MachineEvent`` → abx-dl see a JSON string
+    ``_emit_machine_config`` → ``MachineEvent`` → abx-dl see a JSON string
     where they expect a dict and raise ``TypeError``.
     Declared fields go through pydantic-settings' own ``field_is_complex`` /
     ``prepare_field_value`` so they're decoded per annotation. Undeclared

archivebox/config/common.py

@@ -18,12 +18,13 @@ from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.base.utils import build_config_model
 from django.db import DatabaseError
-from pydantic import BaseModel, Field, PrivateAttr, TypeAdapter, create_model, field_validator, model_validator
+from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 from rich.console import Console
 
 from archivebox.config.configset import COMPUTED_CONFIG_KEYS, BaseConfigSet, decode_config_inputs
 
+from .allauth import AllauthConfig
 from .constants import CONSTANTS
 from .ldap import LDAPConfig
 from .permissions import IN_DOCKER
@@ -34,25 +35,6 @@ ConfigPayload = dict[str, object]
 PluginSchemaDocuments = dict[str, dict[str, Any]]
 LIVE_CONFIG_BASE_URL = "/admin/environment/config/"
 
-
-def _is_cookie_file_config_key(key: str) -> bool:
-    return key in {"COOKIES_FILE", "AUTH_STORAGE_FILE"} or key.endswith("_COOKIES_FILE")
-
-
-def _resolve_cookie_file_paths(config: dict[str, Any]) -> dict[str, Any]:
-    """Keep collection-relative auth paths valid after hooks change cwd."""
-    for key, value in config.items():
-        if _is_cookie_file_config_key(key):
-            if value is None or (isinstance(value, str) and not value.strip()):
-                config[key] = ""
-                continue
-            file_path = Path(value).expanduser()
-            if not file_path.is_absolute():
-                file_path = CONSTANTS.DATA_DIR / file_path
-            config[key] = str(file_path.absolute())
-    return config
-
-
 ###################### Config ##########################
 
 _STDOUT_CONSOLE = Console()
@@ -102,16 +84,12 @@ def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
             return candidate
 
     listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
-    if listen_host and listen_host.lower() not in {"archivebox.localhost:8000", "archivebox.localhost:5797"}:
+    if listen_host and listen_host.lower() != "archivebox.localhost:8000":
         return normalize(listen_host)
 
     csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
     csrf_origins = [origin for origin in csrf_origins if origin]
-    nondefault_csrf_origins = [
-        origin
-        for origin in csrf_origins
-        if origin not in {"http://admin.archivebox.localhost:8000", "http://admin.archivebox.localhost:5797"}
-    ]
+    nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
     if len(nondefault_csrf_origins) == 1:
         return nondefault_csrf_origins[0]
 
@@ -135,6 +113,7 @@ def resolve_delete_after_config_value(*configs: Mapping[str, Any] | None) -> str
 
 
 _SENSITIVE_CONFIG_KEY_NEEDLES = ("TOKEN", "SECRET", "API_KEY", "APIKEY", "PASSWORD")
+_SENSITIVE_CONFIG_KEY_EXACT = frozenset({"SOCIALACCOUNT_PROVIDERS"})
 SENSITIVE_CONFIG_VALUE_REDACTED = "********"
 _SCOPE_CRAWL_FROZEN = "crawl_frozen"
 _SCOPE_CRAWL_EXECUTION = "crawl_execution"
@@ -163,7 +142,11 @@ def is_sensitive_config_key(key: str) -> bool:
     """
     key = str(key or "")
     upper = key.upper()
-    return key in _plugin_sensitive_config_keys() or any(needle in upper for needle in _SENSITIVE_CONFIG_KEY_NEEDLES)
+    return (
+        key in _plugin_sensitive_config_keys()
+        or key in _SENSITIVE_CONFIG_KEY_EXACT
+        or any(needle in upper for needle in _SENSITIVE_CONFIG_KEY_NEEDLES)
+    )
 
 
 def redact_sensitive_config(config: Mapping[str, Any] | None) -> dict[str, Any]:
@@ -337,7 +320,7 @@ class ServerConfig(BaseConfigSet):
     )
 
     SECRET_KEY: str = Field(default_factory=lambda: "".join(secrets.choice("abcdefghijklmnopqrstuvwxyz0123456789_") for _ in range(50)))
-    BIND_ADDR: str = Field(default="127.0.0.1:5797")
+    BIND_ADDR: str = Field(default="127.0.0.1:8000")
     BASE_URL: str = Field(default="")
     ALLOWED_HOSTS: str = Field(default="*")
     CSRF_TRUSTED_ORIGINS: str = Field(default="")
@@ -414,36 +397,6 @@ class ServerConfig(BaseConfigSet):
         )
 
 
-@lru_cache(maxsize=16)
-def _default_sqlite_journal_mode(database_name: str) -> str:
-    if not IN_DOCKER:
-        return "WAL"
-
-    # Native Linux bind mounts share the kernel's locks and mmap; Docker alone
-    # does not make WAL unsafe. DELETE there lets any slow reader block tiny
-    # writes. Desktop VM shares (virtiofs/9p/FUSE), network filesystems and
-    # unknown mounts keep rollback journaling: their host-side readers may not
-    # share SQLite's -shm locking domain. Inspect the DB mount, not archive/.
-    database_path = Path(database_name).expanduser().resolve()
-    try:
-        mounts = Path("/proc/self/mountinfo").read_text().splitlines()
-    except OSError:
-        return "DELETE"
-    filesystem = ""
-    mount_depth = -1
-    for line in mounts:
-        fields, separator, details = line.partition(" - ")
-        if not separator:
-            continue
-        mount_path = fields.split()[4]
-        mount_path = re.sub(r"\\([0-7]{3})", lambda match: chr(int(match[1], 8)), mount_path)
-        mount = Path(mount_path)
-        if database_path.is_relative_to(mount) and len(mount.parts) > mount_depth:
-            filesystem = details.split()[0]
-            mount_depth = len(mount.parts)
-    return "WAL" if filesystem in {"ext2", "ext3", "ext4", "xfs", "btrfs", "zfs", "overlay", "tmpfs", "f2fs"} else "DELETE"
-
-
 class DatabaseConfig(BaseConfigSet):
     toml_section_header: str = "DATABASE_CONFIG"
     _scope: str = PrivateAttr(default=_SCOPE_SERVER)
@@ -459,7 +412,11 @@ class DatabaseConfig(BaseConfigSet):
     DATABASE_USER: str = Field(default="archivebox", alias="ARCHIVEBOX_DATABASE_USER")
     DATABASE_PASSWORD: str = Field(default="", alias="ARCHIVEBOX_DATABASE_PASSWORD")
     SQLITE_JOURNAL_MODE: str = Field(
-        default_factory=lambda data: _default_sqlite_journal_mode(str(data["DATABASE_NAME"])),
+        # Docker collections commonly live on a host bind mount. WAL's -shm
+        # locking is only safe when every SQLite process is on the same host;
+        # Docker Desktop/OrbStack place the container and host in different
+        # locking domains and a host-side reader can corrupt the live DB.
+        default="DELETE" if IN_DOCKER else "WAL",
         alias="ARCHIVEBOX_SQLITE_JOURNAL_MODE",
         pattern=r"(?i)^(DELETE|TRUNCATE|PERSIST|MEMORY|WAL|OFF)$",
     )
@@ -473,16 +430,11 @@ class DatabaseConfig(BaseConfigSet):
     SQLITE_LOCK_RETRY_INTERVAL: float = Field(default=5.0, alias="ARCHIVEBOX_SQLITE_LOCK_RETRY_INTERVAL", gt=0)
 
     @model_validator(mode="after")
-    def reject_shared_filesystem_sqlite_wal(self):
-        if (
-            IN_DOCKER
-            and self.DATABASE_ENGINE.lower() == "sqlite"
-            and self.SQLITE_JOURNAL_MODE.upper() == "WAL"
-            and _default_sqlite_journal_mode(self.DATABASE_NAME) != "WAL"
-        ):
+    def reject_docker_sqlite_wal(self):
+        if IN_DOCKER and self.DATABASE_ENGINE.lower() == "sqlite" and self.SQLITE_JOURNAL_MODE.upper() == "WAL":
             raise ValueError(
-                "SQLITE_JOURNAL_MODE=WAL requires a local filesystem with shared SQLite locking; "
-                "keep the database on a local Docker volume, use DELETE for host/VM shared mounts, or use PostgreSQL",
+                "SQLITE_JOURNAL_MODE=WAL is unsafe for Docker collections because host bind mounts cross SQLite "
+                "locking domains; use DELETE (the Docker default) or PostgreSQL",
             )
         return self
 
@@ -495,7 +447,6 @@ class ArchivingConfig(BaseConfigSet):
         default="",
         description="Comma-separated plugin selection for this run. Empty means use enabled plugin defaults.",
     )
-    PARSER: str = Field(default="auto", description="Parser used to read the submitted Crawl input document.")
 
     ONLY_NEW: bool = Field(default=True)
     INDEX_ONLY: bool = Field(default=False)
@@ -515,21 +466,9 @@ class ArchivingConfig(BaseConfigSet):
     USER_AGENT: str = Field(
         default=f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)",
     )
-    COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
-    AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    COOKIES_FILE: Path | None = Field(default=None)
 
-    @field_validator("COOKIES_FILE", "AUTH_STORAGE_FILE", mode="before")
-    @classmethod
-    def empty_cookie_file_is_unset(cls, value):
-        return None if isinstance(value, str) and not value.strip() else value
-
-    URL_DENYLIST: str = Field(
-        default=(
-            r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$"
-            r"|^https?://news\.ycombinator\.com/(?:vote|hide)/?(?:[?#]|$)"
-        ),
-        alias="URL_BLACKLIST",
-    )
+    URL_DENYLIST: str = Field(default=r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$", alias="URL_BLACKLIST")
     URL_ALLOWLIST: str | None = Field(default=None, alias="URL_WHITELIST")
 
     DEFAULT_PERSONA: str = Field(default="Default", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
@@ -553,7 +492,7 @@ class ArchivingConfig(BaseConfigSet):
             rprint("    (Setting it to somewhere between 30 and 3000 seconds is recommended)", file=sys.stderr)
             rprint(file=sys.stderr)
             rprint("    If you want to make ArchiveBox run faster, disable specific archive methods instead:", file=sys.stderr)
-            rprint("        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#plugins", file=sys.stderr)
+            rprint("        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles", file=sys.stderr)
             rprint(file=sys.stderr)
 
     @field_validator("CHECK_SSL_VALIDITY", mode="after")
@@ -708,6 +647,7 @@ class ArchiveBoxBaseConfig(
     ArchivingConfig,
     SearchBackendConfig,
     LDAPConfig,
+    AllauthConfig,
 ):
     """Merged, typed ArchiveBox config.
 
@@ -738,6 +678,7 @@ class ArchiveBoxBaseConfig(
             ArchivingConfig,
             SearchBackendConfig,
             LDAPConfig,
+            AllauthConfig,
         )
 
     @classmethod
@@ -880,8 +821,6 @@ class ArchiveBoxBaseConfig(
         if runtime_overrides:
             config.update(normalize_runtime_config(runtime_overrides, json_safe=False))
 
-        _resolve_cookie_file_paths(config)
-
         # Hooks should only see concrete plugin-local flags, never the
         # ArchiveBox selectors used to derive them.
         config.pop("PLUGINS", None)
@@ -906,13 +845,6 @@ class ArchiveBoxBaseConfig(
             lib_dir = CONSTANTS.DATA_DIR / lib_dir
         self.ABXPKG_LIB_DIR = lib_dir.resolve()
 
-        cookie_paths = {key: getattr(self, key) for key in type(self).model_fields if _is_cookie_file_config_key(key)}
-        for key, value in _resolve_cookie_file_paths(cookie_paths).items():
-            if key in {"COOKIES_FILE", "AUTH_STORAGE_FILE"}:
-                setattr(self, key, Path(value) if value else None)
-            else:
-                setattr(self, key, value)
-
         return self
 
     @model_validator(mode="after")
@@ -928,11 +860,13 @@ class ArchiveBoxBaseConfig(
             disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
         selected_plugin_roots = plugin_names
         if selected_plugin_roots:
-            from archivebox.plugins.discovery import get_plugin_catalog
+            from abx_dl.models import discover_plugins, filter_plugins
 
             selected_plugins = set(
-                get_plugin_catalog().select(
+                filter_plugins(
+                    discover_plugins(runtime="archivebox"),
                     sorted(selected_plugin_roots),
+                    include_providers=True,
                     disabled_names=disabled_plugins,
                 ),
             )
@@ -1177,7 +1111,7 @@ def get_config(
             if apps.ready:
                 from archivebox.machine.models import Machine
 
-                machine = Machine.current_readonly()
+                machine = Machine.current()
         except (DatabaseError, ImportError, RuntimeError, TypeError, ValueError):
             machine = None
 
@@ -1220,15 +1154,11 @@ def get_config(
     scope_overrides: ConfigPayload = {}
 
     if include_machine and machine is not None and machine.config:
-        machine_config = machine.config
-        if resolve_plugins:
-            from archivebox.machine.models import _sanitize_machine_config
-
-            machine_config = _sanitize_machine_config(machine_config, lib_dir=config_data.get("ABXPKG_LIB_DIR"))
+        from archivebox.machine.models import _sanitize_machine_config
 
         scope_overrides.update(
             normalize_runtime_config(
-                machine_config,
+                _sanitize_machine_config(machine.config, lib_dir=config_data.get("ABXPKG_LIB_DIR")),
                 only_crawl_execution=crawl_config_base,
                 exclude_runtime_derived=True,
                 json_safe=False,
@@ -1332,7 +1262,7 @@ def get_config(
             config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
     for key in explicit_plugin_enabled_keys:
         if key in config_data:
-            setattr(config, key, TypeAdapter(bool).validate_python(config_data[key]))
+            setattr(config, key, config_data[key])
     if config.PLUGINS:
         config._derive_plugin_enabled_config(respect_current_enabled=True)
     if redact_sensitive:
@@ -1359,4 +1289,5 @@ def get_all_configs() -> dict[str, BaseConfigSet]:
         "ARCHIVING_CONFIG": ArchivingConfig.model_validate(resolved),
         "SEARCH_BACKEND_CONFIG": SearchBackendConfig.model_validate(resolved),
         "LDAP_CONFIG": LDAPConfig.model_validate(resolved),
+        "ALLAUTH_CONFIG": AllauthConfig.model_validate(resolved),
     }

archivebox/config/django.py

@@ -63,14 +63,7 @@ def setup_django(check_db=False) -> None:
             os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
             if CONSTANTS.DATA_DIR.exists():
                 for child in CONSTANTS.DATA_DIR.iterdir():
-                    try:
-                        os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
-                    except FileNotFoundError:
-                        # A concurrent SQLite transaction can remove its journal
-                        # between directory enumeration and chown. There is no
-                        # ownership left to fix; other permission errors must
-                        # still fail startup rather than being hidden.
-                        continue
+                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
 
     # Suppress the "database access during app initialization" warning
     # This warning can be triggered during django.setup() but is safe to ignore

archivebox/config/paths.py

@@ -85,31 +85,23 @@ def get_collection_id(DATA_DIR=DATA_DIR) -> str:
 
 
 @cache
-def get_machine_id(length: int = 8) -> str:
-    """Get a machine ID, using the network identity when no OS ID is available."""
-    import machineid
+def get_machine_id() -> str:
+    """Get a short, stable, unique ID for the current machine (e.g. abc45678)"""
 
+    MACHINE_ID = "unknown"
     try:
-        return machineid.hashed_id("archivebox")[:length]
-    except machineid.MachineIdNotFound:
-        import uuid
+        import machineid
 
-        import psutil
+        MACHINE_ID = machineid.hashed_id("archivebox")[:8]
+    except Exception:
+        try:
+            import uuid
+            import hashlib
 
-        node = uuid.getnode()
-        identity = str(node)
-        if node & (1 << 40):
-            # uuid.getnode() silently generates a per-process random value when
-            # tools such as ip/ifconfig are absent. Read interfaces directly so
-            # separate CLI processes agree even in a minimal container.
-            addresses = sorted(
-                address.address
-                for interface in psutil.net_if_addrs().values()
-                for address in interface
-                if address.family == psutil.AF_LINK and address.address not in ("", "00:00:00:00:00:00")
-            )
-            identity = str(int(addresses[0].replace(":", "").replace("-", ""), 16)) if addresses else f"hostname:{socket.gethostname()}"
-        return hashlib.sha256(identity.encode()).hexdigest()[:length]
+            MACHINE_ID = hashlib.sha256(str(uuid.getnode()).encode()).hexdigest()[:8]
+        except Exception:
+            pass
+    return MACHINE_ID
 
 
 @cache

archivebox/config/views.py

@@ -150,9 +150,9 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:
 
     rows = {
         "Binary Name": [],
-        "Recorded Version": [],
+        "Found Version": [],
         "Provided By": [],
-        "Recorded Abspath": [],
+        "Found Abspath": [],
     }
 
     db_binaries = get_db_binaries_by_name()
@@ -160,10 +160,18 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:
 
     for name in all_binary_names:
         binary = db_binaries.get(name)
+        binary_is_valid = bool(binary and binary.is_valid)
+
         rows["Binary Name"].append(ItemLink(name, key=name))
-        rows["Recorded Version"].append(binary.version or "-")
-        rows["Provided By"].append(binary.binprovider or "-")
-        rows["Recorded Abspath"].append(binary.abspath or "-")
+
+        if binary_is_valid:
+            rows["Found Version"].append(f"✅ {binary.version}" if binary.version else "✅ found")
+            rows["Provided By"].append(binary.binprovider or "-")
+            rows["Found Abspath"].append(binary.abspath or "-")
+        else:
+            rows["Found Version"].append("❌ missing")
+            rows["Provided By"].append("-")
+            rows["Found Abspath"].append("-")
 
     return TableContext(
         title="Binaries",

archivebox/core/admin_archiveresults.py

@@ -2,9 +2,11 @@ __package__ = "archivebox.core"
 
 import html
 import json
+import os
 import shlex
 from functools import reduce
 from operator import and_
+from pathlib import Path
 from urllib.parse import quote
 
 from django.contrib import admin
@@ -22,13 +24,7 @@ from django.utils.text import smart_split
 
 from archivebox.base_models.admin import BaseModelAdmin
 from archivebox.core.models import ArchiveResult, Snapshot
-from archivebox.core.routes_util import (
-    build_snapshot_detail_url,
-    build_snapshot_files_url,
-    build_snapshot_url,
-    build_snapshot_zip_url,
-    get_snapshot_output_anchor,
-)
+from archivebox.core.routes_util import build_snapshot_url
 from archivebox.core.widgets import InlineTagEditorWidget
 from archivebox.machine.env_util import env_to_shell_exports
 from archivebox.misc.logging_util import printable_filesize
@@ -190,7 +186,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None, can_del
         if embed_path and result.status == "succeeded":
             output_link = build_snapshot_url(snapshot_id, embed_path, config=config)
         else:
-            output_link = build_snapshot_detail_url(result.snapshot.archive_path_from_db, config=config)
+            output_link = build_snapshot_url(snapshot_id, "", config=config)
         output_link_attr = html.escape(output_link, quote=True)
 
         # Get version - try cmd_version field
@@ -559,8 +555,8 @@ class ArchiveResultAdmin(BaseModelAdmin):
             try:
                 queryset = self.get_queryset(request).filter(pk__in=selected)
                 if not queryset.exists():
-                    snapshot = Snapshot.objects.filter(pk=request.GET.get("snapshot")).first()
-                    return redirect(build_snapshot_detail_url(snapshot.archive_path_from_db, request=request) if snapshot else request.path)
+                    snapshot = Snapshot.objects.only("id").filter(pk=request.GET.get("snapshot")).first()
+                    return redirect(build_snapshot_url(str(snapshot.id), "index.html", request=request) if snapshot else request.path)
             except (ValidationError, ValueError):
                 return redirect(request.path)
             return delete_selected(self, request, queryset)
@@ -572,7 +568,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
             request.GET.clear()
             request.META["QUERY_STRING"] = ""
             try:
-                handoff_snapshot = Snapshot.objects.filter(pk=handoff_snapshot).first()
+                handoff_snapshot = Snapshot.objects.only("id").filter(pk=handoff_snapshot).first()
             except (ValidationError, ValueError):
                 handoff_snapshot = None
         saved_list_per_page = self.list_per_page
@@ -586,7 +582,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
                     pk__in=request.POST.getlist(ACTION_CHECKBOX_NAME),
                 ).exists()
             ):
-                return redirect(build_snapshot_detail_url(handoff_snapshot.archive_path_from_db, request=request))
+                return redirect(build_snapshot_url(str(handoff_snapshot.id), "index.html", request=request))
             return response
         finally:
             self.list_per_page = saved_list_per_page
@@ -660,40 +656,22 @@ class ArchiveResultAdmin(BaseModelAdmin):
 
     def get_snapshot_view_url(self, result: ArchiveResult) -> str:
         request = self.request
-        return build_snapshot_detail_url(
-            result.snapshot.archive_path_from_db,
-            request=request,
-            config=request.archivebox_config,
-        )
+        return build_snapshot_url(str(result.snapshot_id), request=request, config=request.archivebox_config)
 
     def get_output_view_url(self, result: ArchiveResult) -> str:
         request = self.request
         config = request.archivebox_config
-        output_path = get_snapshot_output_anchor(result.plugin, result.embed_path() or "")
-        return build_snapshot_detail_url(
-            result.snapshot.archive_path_from_db,
-            output_path=output_path,
-            request=request,
-            config=config,
-        )
+        output_path = result.embed_path()
+        if not output_path:
+            output_path = result.plugin or ""
+        return build_snapshot_url(str(result.snapshot_id), output_path, request=request, config=config)
 
     def get_output_files_url(self, result: ArchiveResult) -> str:
         request = self.request
-        return build_snapshot_files_url(
-            str(result.snapshot_id),
-            result.output_folder_path(),
-            request=request,
-            config=request.archivebox_config,
-        )
+        return f"{build_snapshot_url(str(result.snapshot_id), result.plugin, request=request, config=request.archivebox_config)}/?files=1"
 
     def get_output_zip_url(self, result: ArchiveResult) -> str:
-        request = self.request
-        return build_snapshot_zip_url(
-            str(result.snapshot_id),
-            result.output_folder_path(),
-            request=request,
-            config=request.archivebox_config,
-        )
+        return f"{self.get_output_files_url(result)}&download=zip"
 
     @admin.display(description="Details", ordering="id")
     def details_link(self, result):
@@ -751,11 +729,7 @@ class ArchiveResultAdmin(BaseModelAdmin):
         request = self.request
         return format_html(
             '<a href="{}"><b><code>[{}]</code></b> &nbsp; {} &nbsp; {}</a><br/>',
-            build_snapshot_detail_url(
-                result.snapshot.archive_path_from_db,
-                request=request,
-                config=request.archivebox_config,
-            ),
+            build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
             snapshot_id[:8],
             result.snapshot.bookmarked_at.strftime("%Y-%m-%d %H:%M"),
             result.snapshot.url[:128],
@@ -853,14 +827,11 @@ class ArchiveResultAdmin(BaseModelAdmin):
         config = request.archivebox_config
         # Determine output link path - use embed_path() which checks output_files
         embed_path = result.embed_path()
-        output_url = (
-            build_snapshot_url(str(result.snapshot_id), embed_path, request=request, config=config)
-            if result.status == "succeeded" and embed_path
-            else build_snapshot_detail_url(result.snapshot.archive_path_from_db, request=request, config=config)
-        )
+        output_path = embed_path if (result.status == "succeeded" and embed_path) else "index.html"
+        snapshot_id = str(result.snapshot_id)
         return format_html(
             '<a href="{}" class="output-link">↗️</a><pre>{}</pre>',
-            output_url,
+            build_snapshot_url(snapshot_id, output_path, request=request, config=config),
             result.output_str_for_display(),
         )
 
@@ -887,16 +858,46 @@ class ArchiveResultAdmin(BaseModelAdmin):
         )
 
     def output_summary(self, result):
-        output_html = format_html('<pre style="display: inline-block">{}</pre>', result.output_str_for_display())
-        file_map = result.output_file_map()
-        paths = sorted({path for key in file_map if (path := result.output_file_path(key, output_file_map=file_map))})
-        if paths:
-            output_html += format_html(
-                '<br/><a href="{}">See result files ...</a><pre><code>{}</code></pre>',
-                self.get_output_files_url(result),
-                "\n".join(paths),
-            )
-        return output_html
+        snapshot_dir = Path(result.snapshot.output_dir)
+        output_html = format_html(
+            '<pre style="display: inline-block">{}</pre><br/>',
+            result.output_str_for_display(),
+        )
+        snapshot_id = str(result.snapshot_id)
+        request = self.request
+        output_html += format_html(
+            '<a href="{}#all">See result files ...</a><br/><pre><code>',
+            build_snapshot_url(snapshot_id, "index.html", request=request, config=request.archivebox_config),
+        )
+        embed_path = result.embed_path() or ""
+        path_from_embed = snapshot_dir / (embed_path or "")
+        output_html += format_html(
+            '<i style="padding: 1px">{}</i><b style="padding-right: 20px">/</b><i>{}</i><br/><hr/>',
+            str(snapshot_dir),
+            str(embed_path),
+        )
+        if os.access(path_from_embed, os.R_OK):
+            root_dir = str(path_from_embed)
+        else:
+            root_dir = str(snapshot_dir)
+
+        for root, dirs, files in os.walk(root_dir):
+            depth = root.replace(root_dir, "").count(os.sep) + 1
+            if depth > 2:
+                continue
+            indent = " " * 4 * (depth)
+            output_html += format_html('<b style="padding: 1px">{}{}/</b><br/>', indent, os.path.basename(root))
+            indentation_str = " " * 4 * (depth + 1)
+            for filename in sorted(files):
+                is_hidden = filename.startswith(".")
+                output_html += format_html(
+                    '<span style="opacity: {}.2">{}{}</span><br/>',
+                    int(not is_hidden),
+                    indentation_str,
+                    filename.strip(),
+                )
+
+        return output_html + mark_safe("</code></pre>")
 
 
 def register_admin(admin_site):

archivebox/core/admin_site.py

@@ -11,8 +11,10 @@ from admin_data_views.admin import (
 from admin_data_views.admin import (
     get_app_list as adv_get_app_list,
 )
+from django.conf import settings
 from django.contrib import admin
-from django.contrib.auth import REDIRECT_FIELD_NAME, get_user_model, login as auth_login
+from django.contrib.auth import REDIRECT_FIELD_NAME, get_user_model
+from django.contrib.auth import login as auth_login
 from django.contrib.auth.decorators import login_not_required
 from django.contrib.auth.forms import UserCreationForm
 from django.contrib.auth.views import LoginView
@@ -31,7 +33,6 @@ from archivebox.core.routes_util import is_allowed_archivebox_redirect_url
 if TYPE_CHECKING:
     from admin_data_views.typing import AppDict
     from django.http import HttpRequest
-    from django.template.response import TemplateResponse
     from django.urls import URLPattern, URLResolver
 
 
@@ -114,6 +115,7 @@ class ArchiveBoxAdmin(admin.AdminSite):
             "app_path": request.get_full_path(),
             "username": request.user.get_username(),
             "first_admin_setup": first_admin_setup,
+            "allauth_enabled": settings.ALLAUTH_ENABLED,
         }
         if REDIRECT_FIELD_NAME not in request.GET and REDIRECT_FIELD_NAME not in request.POST:
             context[REDIRECT_FIELD_NAME] = reverse("admin:index", current_app=self.name)

archivebox/core/admin_snapshots.py

@@ -1,56 +1,38 @@
 __package__ = "archivebox.core"
 
 import json
-from math import ceil
-from typing import Any
+from functools import lru_cache
+from types import SimpleNamespace
 
 from django.contrib import admin, messages
-from django.core.exceptions import PermissionDenied
 from django.urls import path, reverse
 from django.shortcuts import get_object_or_404, redirect
-from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed, Http404
+from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
 from django.utils import timezone
-from django.utils.formats import date_format
 from django.utils.html import format_html, format_html_join
 from django.utils.safestring import mark_safe
 from django.db.models import Q, Count, Exists, F, OuterRef, Prefetch
 from django import forms
-from django.template import Context, Template, RequestContext
+from django.template import Template, RequestContext
 from django.contrib.admin.helpers import ActionForm
 
 from archivebox.config.common import get_config
 from archivebox.misc.util import htmldecode, urldecode
 from archivebox.misc.paginators import AcceleratedPaginator
 from archivebox.misc.logging_util import printable_filesize
-from archivebox.misc.serve_static import serve_static_with_byterange_support
 from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
 from archivebox.search.views import admin_snapshot_search_stream_view
-from archivebox.core.routes_util import (
-    build_snapshot_detail_url,
-    build_snapshot_files_url,
-    build_snapshot_role_output_url,
-    build_snapshot_url,
-    build_snapshot_zip_url,
-    get_snapshot_output_anchor,
-)
+from archivebox.core.routes_util import build_snapshot_url, build_web_url
 from archivebox.core.tag_util import get_or_create_tag
-from archivebox.core.templatetags.core_tags import snapshot_thumbnail, snapshot_url_text
 from archivebox.plugins.hooks import discover_hooks
-from archivebox.plugins.discovery import get_plugin_icon, get_snapshot_role_names
-from archivebox.plugins.output_groups import (
-    OUTPUT_GROUPS,
-    display_plugin_name,
-    output_group_for_plugin,
-    order_output_plugins,
-    plugin_icon_is_hidden,
-    plugin_output_sizes,
-)
+from archivebox.plugins.discovery import get_plugin_icon, get_plugin_name, get_plugins
 
 from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
 
 from archivebox.core.models import Tag, Snapshot, ArchiveResult
 from archivebox.crawls.models import Crawl
 from archivebox.core.admin_archiveresults import render_archiveresults_list
+from archivebox.core.preview_util import EXTENSION_SCREENSHOT_PLUGIN
 from archivebox.progressmonitor.views import progress_endpoint
 from archivebox.core.permissions import (
     PERMISSIONS_CHOICES,
@@ -66,12 +48,9 @@ GLOBAL_CONTEXT = {}
 SNAPSHOT_PERMISSION_META = PERMISSIONS_META
 
 
-def _format_size_column(num_bytes: int) -> str:
-    size = num_bytes
-    for unit in ("B", "KB", "MB", "GB", "TB"):
-        if size < 1024 or unit == "TB":
-            return f"{size:.1f} {unit}" if unit == "GB" else f"{ceil(size)} {unit}"
-        size /= 1024
+@lru_cache(maxsize=1)
+def _plugin_sort_order() -> dict[str, int]:
+    return {get_plugin_name(plugin): idx for idx, plugin in enumerate(get_plugins())}
 
 
 class SnapshotActionForm(ActionForm):
@@ -131,11 +110,11 @@ class SnapshotStatusListFilter(admin.SimpleListFilter):
     parameter_name = "snapshot_status"
 
     def lookups(self, request, model_admin):
-        return Snapshot._meta.get_field("status").choices
+        return Snapshot.StatusChoices.choices
 
     def queryset(self, request, queryset):
         value = self.value()
-        if value in dict(Snapshot._meta.get_field("status").choices):
+        if value in Snapshot.StatusChoices.values:
             return queryset.filter(status=value)
         return queryset
 
@@ -283,28 +262,15 @@ class SnapshotChangeList(SearchResultsChangeList):
         rows = (
             ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
             .order_by("snapshot_id", "plugin")
-            .only("snapshot_id", "plugin", "status", "output_str", "output_size", "output_files")
+            .values_list("snapshot_id", "plugin", "status", "output_size", "output_files")
         )
-        for result in rows.iterator(chunk_size=1000):
-            if result.plugin in seen_plugins[result.snapshot_id]:
+        for snapshot_id, plugin, status, output_size, output_files in rows.iterator(chunk_size=1000):
+            if plugin in seen_plugins[snapshot_id]:
                 continue
-            seen_plugins[result.snapshot_id].add(result.plugin)
-            output_results_by_snapshot[result.snapshot_id].append(result)
-
-        card_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
-        for result in (
-            ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids)
-            .exclude(plugin="")
-            .only(
-                "snapshot_id",
-                "plugin",
-                "status",
-                "output_str",
-                "output_size",
-                "output_files",
+            seen_plugins[snapshot_id].add(plugin)
+            output_results_by_snapshot[snapshot_id].append(
+                SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
             )
-        ):
-            card_results_by_snapshot[result.snapshot_id].append(result)
 
         for obj in self.result_list:
             counts = status_counts_by_snapshot.get(obj.pk, {})
@@ -328,7 +294,6 @@ class SnapshotChangeList(SearchResultsChangeList):
                 "is_sealed": obj.status not in (obj.StatusChoices.QUEUED, obj.StatusChoices.STARTED, obj.StatusChoices.PAUSED),
             }
             obj.__dict__["_admin_output_results"] = output_results_by_snapshot[obj.pk]
-            obj.__dict__["_snapshot_card_results"] = card_results_by_snapshot[obj.pk]
 
     def get_results(self, request):
         super().get_results(request)
@@ -341,7 +306,6 @@ class SnapshotChangeList(SearchResultsChangeList):
 class SnapshotAdminForm(forms.ModelForm):
     """Custom form for Snapshot admin with tag editor widget."""
 
-    url = forms.CharField(widget=forms.TextInput)
     tags_editor = forms.CharField(
         label="Tags",
         required=False,
@@ -404,7 +368,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
     list_display = (
         "permissions_badge",
-        "created_at_display",
+        "created_at",
         "preview_icon",
         "title_str",
         "tags_inline",
@@ -412,7 +376,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         "files",
         "size_with_stats",
     )
-    list_display_links = ("created_at_display",)
+    list_display_links = ("created_at",)
     sort_fields = ("title_str", "created_at", "status", "crawl")
     readonly_fields = (
         "admin_actions",
@@ -526,15 +490,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     inlines = []  # Removed TagInline, using TagEditorWidget instead
     list_per_page = 50
 
-    @admin.display(description="Date", ordering="created_at")
-    def created_at_display(self, obj):
-        created_at = timezone.localtime(obj.created_at)
-        return format_html(
-            '<span class="snapshot-created-at"><span class="snapshot-created-at__date">{}</span><span class="snapshot-created-at__time">{}</span></span>',
-            date_format(created_at, "Y-m-d"),
-            date_format(created_at, "g:i A"),
-        )
-
     action_form = SnapshotActionForm
     paginator = AcceleratedPaginator
 
@@ -600,26 +555,20 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
     def get_snapshot_view_url(self, obj: Snapshot) -> str:
         request = self.request
-        return build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=request.archivebox_config)
+        return build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)
 
     def get_snapshot_files_url(self, obj: Snapshot) -> str:
         request = self.request
-        return build_snapshot_files_url(str(obj.id), request=request, config=request.archivebox_config)
+        return f"{build_snapshot_url(str(obj.id), request=request, config=request.archivebox_config)}/?files=1"
 
     def get_snapshot_zip_url(self, obj: Snapshot) -> str:
-        request = self.request
-        return build_snapshot_zip_url(str(obj.id), request=request, config=request.archivebox_config)
+        return f"{self.get_snapshot_files_url(obj)}&download=zip"
 
     def get_urls(self):
         urls = super().get_urls()
         custom_urls = [
             path("grid/", self.admin_site.admin_view(self.grid_view), name="grid"),
             path("search-stream/", self.admin_site.admin_view(self.search_stream_view), name="core_snapshot_search_stream"),
-            path(
-                "<path:object_id>/preview/<str:plugin>/<path:filename>",
-                self.admin_site.admin_view(self.preview_view, cacheable=True),
-                name="core_snapshot_preview",
-            ),
             path("<path:object_id>/redo-failed/", self.admin_site.admin_view(self.redo_failed_view), name="core_snapshot_redo_failed"),
             path(
                 "<path:object_id>/set-permissions/",
@@ -629,26 +578,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         ]
         return custom_urls + urls
 
-    def preview_view(self, request, object_id, plugin, filename):
-        """Serve a saved image output from the authenticated admin origin."""
-        snapshot = get_object_or_404(Snapshot, pk=object_id)
-        if not self.has_view_or_change_permission(request, snapshot):
-            raise PermissionDenied
-        result = snapshot.archiveresult_set.filter(plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED).first()
-        try:
-            output_path = result.output_file_path(filename) if result else None
-        except (TypeError, ValueError, OverflowError):
-            output_path = None
-        if not output_path:
-            raise Http404("Snapshot image does not exist")
-        request.archivebox_cache_policy = "private"
-        response = serve_static_with_byterange_support(request, output_path, document_root=snapshot.output_dir)
-        if response.status_code != 304 and not response.headers.get("Content-Type", "").lower().startswith("image/"):
-            raise Http404("Snapshot preview is not an image")
-        response.headers["Vary"] = "Cookie"
-        response.headers["X-Content-Type-Options"] = "nosniff"
-        return response
-
     def search_stream_view(self, request):
         return admin_snapshot_search_stream_view(self, request)
 
@@ -780,7 +709,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
         return qs
 
-    @admin.display(description="🔒", ordering="permissions")
+    @admin.display(description="👁", ordering="permissions")
     def permissions_badge(self, obj):
         permissions = obj.__dict__.get("snapshot_permissions")
         if permissions is None:
@@ -926,13 +855,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     def status_info(self, obj):
         request = self.request
         config = request.archivebox_config
-        favicon_url = build_snapshot_role_output_url(
-            obj,
-            "list_icon",
-            request=request,
-            config=config,
-            fallback_to_default=True,
-        )
+        favicon_url = build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config)
         return format_html(
             """
             Archived: {} ({} files {}) &nbsp; &nbsp;
@@ -970,7 +893,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         show_title = bool(title_raw) and title_normalized != "pending..." and title_normalized != url_normalized
         css_class = "fetched" if show_title else "pending"
 
-        detail_url = build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config)
+        detail_url = build_web_url(f"/{obj.archive_path_from_db}/index.html", request=request, config=config)
         title_html = ""
         if show_title:
             title_html = format_html(
@@ -980,30 +903,14 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
                 urldecode(htmldecode(title_raw))[:128],
             )
 
-        favicon = self._get_favicon_data(obj) or {}
-        favicon_html = (
-            format_html(
-                '<img class="snapshot-url-favicon" src="{}" alt="" loading="lazy" decoding="async" onerror="this.hidden=true">',
-                favicon["url"],
-            )
-            if favicon.get("url")
-            else ""
-        )
         return format_html(
-            '<a class="snapshot-title-detail-hitbox" href="{}" aria-label="Open snapshot details"></a>'
             "{}"
-            "{}"
-            '<div class="snapshot-title-url">'
-            '<a class="snapshot-original-url" href="{}">{}<code style="user-select: all;">{}</code></a>'
+            '<div style="font-size: 11px; color: #64748b; margin-top: 2px;">'
+            '<a href="{}"><code style="user-select: all;">{}</code></a>'
             "</div>",
-            detail_url,
-            format_html('<strong class="snapshot-delete-status">{}</strong>', "Deleting — verifying file removal")
-            if obj.status == Snapshot.DELETING_STATE
-            else "",
             title_html,
             url_raw or obj.url,
-            favicon_html,
-            snapshot_url_text(url_raw or obj.url),
+            (url_raw or obj.url)[:128],
         )
 
     @admin.display(description="Tags", ordering="tag_count")
@@ -1030,47 +937,90 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         )
         return mark_safe(f'<span class="tags-inline-editor">{tags_html}</span>')
 
-    def _get_favicon_data(self, obj):
-        list_icon_plugins = set(get_snapshot_role_names("list_icon"))
+    def _get_preview_data(self, obj):
+        request = self.request
+        config = request.archivebox_config
         results = self._get_prefetched_results(obj)
-        if results is None:
-            results = obj.archiveresult_set.filter(
-                plugin__in=list_icon_plugins,
-                status=ArchiveResult.StatusChoices.SUCCEEDED,
+        if results is not None:
+            results = [r for r in results if r.plugin in ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")]
+        else:
+            results = list(
+                obj.archiveresult_set.filter(plugin__in=("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")).only(
+                    "snapshot_id",
+                    "plugin",
+                    "status",
+                    "output_files",
+                    "output_str",
+                ),
             )
-        urls = []
-        for result in results:
-            if result.plugin not in list_icon_plugins or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
-                continue
-            try:
-                output_path = result.embed_path()
-            except (TypeError, ValueError, OverflowError):
-                continue
-            if output_path:
-                urls.append(
-                    build_snapshot_url(str(obj.id), output_path, request=self.request, config=self.request.archivebox_config),
-                )
-        return {"url": urls[0], "fallbacks": ",".join(urls[1:])} if urls else None
 
-    def _render_thumbnail(self, obj, width="100px", height="100px"):
-        return snapshot_thumbnail(
-            Context({"request": self.request, "CONFIG": self.request.archivebox_config}),
-            obj,
-            admin=True,
-            width=width,
-            height=height,
+        def result_output_path(result, filename: str) -> str | None:
+            output_files = result.output_files or {}
+            file_info = output_files.get(filename)
+            if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
+                return None
+            if file_info.get("root_relative"):
+                return filename
+            return f"{result.plugin}/{filename}"
+
+        def result_urls(plugin: str, filenames: tuple[str, ...]) -> list[str]:
+            urls: list[str] = []
+            for result in results:
+                if result.plugin != plugin or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
+                    continue
+                for filename in filenames:
+                    output_path = result_output_path(result, filename)
+                    if output_path:
+                        urls.append(build_snapshot_url(str(obj.id), output_path, request=request, config=config))
+            return urls
+
+        screenshot_urls = result_urls("screenshot", ("screenshot.png",))
+        extension_screenshot_urls = result_urls(EXTENSION_SCREENSHOT_PLUGIN, ("screenshot-1.png", "screenshot.png"))
+        favicon_urls = result_urls("favicon", ("favicon.ico",))
+
+        if not screenshot_urls and not extension_screenshot_urls and not favicon_urls:
+            return None
+
+        all_screenshot_urls = [*screenshot_urls, *extension_screenshot_urls]
+        if all_screenshot_urls:
+            img_url = all_screenshot_urls[0]
+            fallbacks = [*all_screenshot_urls[1:], *favicon_urls]
+            img_alt = "Screenshot"
+            preview_class = "screenshot"
+        else:
+            img_url = favicon_urls[0]
+            fallbacks = favicon_urls[1:]
+            img_alt = "Favicon"
+            preview_class = "favicon"
+
+        fallback_list = ",".join(fallbacks)
+        onerror_js = (
+            "this.dataset.fallbacks && this.dataset.fallbacks.length ? "
+            "(this.src=this.dataset.fallbacks.split(',').shift(), "
+            "this.dataset.fallbacks=this.dataset.fallbacks.split(',').slice(1).join(',')) : "
+            "this.remove()"
         )
 
+        return {
+            "img_url": img_url,
+            "img_alt": img_alt,
+            "preview_class": preview_class,
+            "onerror_js": onerror_js,
+            "fallback_list": fallback_list,
+            "favicon_url": favicon_urls[0] if favicon_urls else "",
+            "favicon_fallback_list": ",".join(favicon_urls[1:]),
+        }
+
     @admin.display(description="", empty_value="")
     def url_favicon(self, obj):
-        favicon = self._get_favicon_data(obj)
-        if not favicon:
+        preview = self._get_preview_data(obj)
+        if not preview:
             return ""
 
-        favicon_url = favicon.get("url") or ""
+        favicon_url = preview.get("favicon_url") or ""
         if not favicon_url:
             return ""
-        fallback_list = favicon.get("fallbacks") or ""
+        fallback_list = preview.get("favicon_fallback_list") or ""
         onerror_js = (
             "this.dataset.fallbacks && this.dataset.fallbacks.length ? "
             "(this.src=this.dataset.fallbacks.split(',').shift(), "
@@ -1091,24 +1041,42 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
     @admin.display(description="Preview", empty_value="")
     def preview_icon(self, obj):
+        preview = self._get_preview_data(obj)
+        if not preview:
+            return None
+
         return format_html(
-            '<a href="{}" title="Open snapshot details">{}</a>',
-            build_snapshot_detail_url(obj.archive_path_from_db, request=self.request, config=self.request.archivebox_config),
-            self._render_thumbnail(obj, height="38px"),
+            '<img src="{}" alt="{}" class="snapshot-preview {}" decoding="async" loading="lazy" onerror="{}" data-fallbacks="{}">',
+            preview["img_url"],
+            preview["img_alt"],
+            preview["preview_class"],
+            preview["onerror_js"],
+            preview["fallback_list"],
         )
 
     @admin.display(description=" ", empty_value="")
     def snapshot_summary(self, obj):
         request = self.request
         config = request.archivebox_config
+        preview = self._get_preview_data(obj)
         stats = self._get_progress_stats(obj)
         archive_size = stats["output_size"] or 0
         size_txt = printable_filesize(archive_size) if archive_size else "pending"
-        screenshot_html = format_html(
-            '<a href="{}" title="Open snapshot live view" style="display:block;flex:0 0 220px;">{}</a>',
-            build_snapshot_detail_url(obj.archive_path_from_db, request=request, config=config),
-            self._render_thumbnail(obj, width="220px", height="138px"),
-        )
+        screenshot_html = ""
+
+        if preview:
+            screenshot_html = format_html(
+                '<a href="{href}" title="Open snapshot live view" style="display:block; flex:0 0 220px; width:220px;">'
+                '<img src="{src}" alt="{alt}" decoding="async" loading="lazy" onerror="{onerror}" data-fallbacks="{fallbacks}" '
+                'style="display:block; width:100%; max-width:220px; aspect-ratio: 16 / 10; object-fit: cover; object-position: top; '
+                'border-radius: 10px; border: 1px solid #e2e8f0; background: #f8fafc;">'
+                "</a>",
+                href=build_web_url(f"/{obj.archive_path}", request=request, config=config),
+                src=preview["img_url"],
+                alt=preview["img_alt"],
+                onerror=preview["onerror_js"],
+                fallbacks=preview["fallback_list"],
+            )
 
         return format_html(
             '<div style="display:flex; gap:16px; align-items:flex-start;">'
@@ -1123,19 +1091,18 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             "</div>",
             screenshot_html,
             size_txt,
-            self.get_snapshot_files_url(obj),
-            obj.archive_path_from_db,
+            build_web_url(f"/{obj.archive_path}", request=request, config=config),
+            obj.archive_path,
         )
 
     @admin.display(
         description="Files Saved",
         ordering="ar_succeeded_count",
-        empty_value="",
     )
     def files(self, obj):
         results = self._get_prefetched_results(obj)
         if results is None:
-            results = obj.archiveresult_set.only("plugin", "status", "output_size", "output_files")
+            results = obj.archiveresult_set.only("plugin", "status", "output_size")
 
         plugins_with_output: dict[str, ArchiveResult] = {}
         for result in results:
@@ -1146,111 +1113,56 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             plugins_with_output.setdefault(result.plugin, result)
 
         if not plugins_with_output:
-            return ""
+            return mark_safe('<span style="opacity: 0.35;">...</span>')
 
-        sorted_results = [plugins_with_output[plugin] for plugin in order_output_plugins(plugins_with_output, plugin_output_sizes(results))]
-        icons_by_group: dict[str, list[dict[str, Any]]] = {group_id: [] for group_id, _, _ in OUTPUT_GROUPS}
+        sorted_results = sorted(
+            plugins_with_output.values(),
+            key=lambda result: (_plugin_sort_order().get(result.plugin, 9999), result.plugin),
+        )
+        visible_results = sorted_results[:14]
+        output = []
         request = self.request
         config = request.archivebox_config
-        for result in sorted_results:
-            if plugin_icon_is_hidden(result.plugin):
-                continue
+        for result in visible_results:
             icon = mark_safe(get_plugin_icon(result.plugin))
             if not icon.strip():
                 continue
-            embed_path = result.embed_path()
-            if not embed_path:
-                continue
-            group_id = output_group_for_plugin(result.plugin)
-            anchor_path = get_snapshot_output_anchor(result.plugin, embed_path)
-            icons_by_group[group_id].append(
-                {
-                    "plugin": result.plugin,
-                    "icon": icon,
-                    "href": build_snapshot_detail_url(
-                        obj.archive_path_from_db,
-                        output_path=anchor_path,
-                        request=request,
-                        config=config,
-                    ),
-                },
+            output.append(
+                format_html(
+                    '<a href="{}" class="exists-True" title="{}">{}</a>',
+                    build_web_url(f"/{obj.archive_path_from_db}/{result.plugin}/", request=request, config=config),
+                    result.plugin,
+                    icon,
+                ),
+            )
+        if len(sorted_results) > len(visible_results):
+            output.append(
+                format_html(
+                    '<span title="{} more outputs">+{}</span>',
+                    len(sorted_results) - len(visible_results),
+                    len(sorted_results) - len(visible_results),
+                ),
             )
 
-        output = []
-        for group_id, label, _ in OUTPUT_GROUPS:
-            icons = icons_by_group[group_id]
-            if icons:
-                top = icons[0]
-                front_index = 0
-                pile_column = len(output)
-                top_html = format_html(
-                    '<a href="{}" class="exists-True files-icon-pile-top files-icon-pile-cover files-icon-plugin--{}" aria-label="{}" data-tooltip="{}" style="--files-icon-front-index:{};--files-icon-front-offset:{}px">{}</a>',
-                    top["href"],
-                    top["plugin"],
-                    label,
-                    label,
-                    front_index,
-                    front_index * 2,
-                    top["icon"],
-                )
-                members = []
-                for icon_index, item in enumerate(icons):
-                    row = icon_index // 5
-                    row_count = min(5, len(icons) - row * 5)
-                    member_top = 8 + row * 18
-                    row_offset = (5 - row_count) * 11 + 3
-                    member_left = 3 + row_offset + (icon_index % 5) * 22
-                    members.append(
-                        format_html(
-                            '<a href="{}" class="exists-True files-icon-plugin--{}" aria-label="{}" data-tooltip="{}" style="--files-icon-index:{};--files-icon-row:{};--files-icon-column:{};--files-icon-row-offset:{}px;--files-icon-member-left:{}px;--files-icon-member-top:{}px">{}</a>',
-                            item["href"],
-                            item["plugin"],
-                            display_plugin_name(item["plugin"]),
-                            item["plugin"],
-                            icon_index,
-                            row,
-                            icon_index % 5,
-                            row_offset,
-                            member_left,
-                            member_top,
-                            item["icon"],
-                        ),
-                    )
-                popup = format_html(
-                    '<span class="files-icon-pile-popup" popover="manual">{}<span class="files-icon-pile-label">{}</span></span>',
-                    mark_safe("".join(str(icon) for icon in members)),
-                    label,
-                )
-                pile_contents = format_html("{}{}", top_html, popup)
-                output.append(
-                    format_html(
-                        '<span class="files-icon-pile files-icon-pile--{}" role="group" aria-label="{} saved outputs" style="--files-icon-popup-left:{}px;--files-icon-rows:{};--files-icon-card-width:{}px">{}</span>',
-                        group_id,
-                        label,
-                        pile_column * -22 - front_index * 2,
-                        (len(icons) + 4) // 5,
-                        116,
-                        pile_contents,
-                    ),
-                )
-
         return format_html(
-            '<span class="files-icons files-icons--compact" style="font-size: 1em;">{}</span>',
+            '<span class="files-icons files-icons--compact" style="font-size: 1em; opacity: 0.8;">{}</span>',
             mark_safe("".join(output)),
         )
 
-    @admin.display(empty_value="")
+    @admin.display()
     def size(self, obj):
+        request = self.request
+        config = request.archivebox_config
         archive_size = self._get_progress_stats(obj)["output_size"] or 0
         if archive_size:
-            size_txt = _format_size_column(archive_size)
+            size_txt = printable_filesize(archive_size)
             if archive_size > 52428800:
                 size_txt = mark_safe(f"<b>{size_txt}</b>")
         else:
-            return ""
+            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
         return format_html(
             '<a href="{}" title="View all files">{}</a>',
-            self.get_snapshot_files_url(obj),
+            build_web_url(f"/{obj.archive_path}", request=request, config=config),
             size_txt,
         )
 
@@ -1320,7 +1232,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     @admin.display(
         description="Size",
         ordering="output_size",
-        empty_value="",
     )
     def size_with_stats(self, obj):
         """Show archive size with output size from archive results."""
@@ -1329,11 +1240,24 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         size_bytes = output_size or 0
 
         if size_bytes:
-            size_txt = _format_size_column(size_bytes)
+            size_txt = printable_filesize(size_bytes)
             if size_bytes > 52428800:  # 50MB
                 size_txt = mark_safe(f"<b>{size_txt}</b>")
         else:
-            return ""
+            size_txt = mark_safe('<span style="opacity: 0.3">...</span>')
+
+        # Show hook statistics
+        if stats["total"] > 0:
+            return format_html(
+                '<a href="{}" title="View all files" style="white-space: nowrap;">'
+                "{}</a>"
+                '<div style="font-size: 10px; color: #94a3b8; margin-top: 2px;">'
+                "{}/{} hooks</div>",
+                self.get_snapshot_files_url(obj),
+                size_txt,
+                stats["succeeded"],
+                stats["total"],
+            )
 
         return format_html(
             '<a href="{}" title="View all files">{}</a>',
@@ -1461,9 +1385,9 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     )
     def url_str(self, obj):
         return format_html(
-            '<a class="snapshot-original-url" href="{}"><code style="user-select: all;">{}</code></a>',
+            '<a href="{}"><code style="user-select: all;">{}</code></a>',
             obj.url,
-            snapshot_url_text(obj.url),
+            obj.url[:128],
         )
 
     @admin.display(description="Health", ordering="health")
@@ -1539,14 +1463,25 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
     @admin.action(
         description="🗑️ Delete",
-        permissions=["delete"],
     )
     def delete_snapshots(self, request, queryset):
-        """Keep the rows until the runner removes and verifies their files."""
-        total = queryset.request_delete()
+        """Delete snapshots in a single transaction to avoid SQLite concurrency issues."""
+        from django.db import transaction
+
+        total = queryset.count()
+
+        # Get list of IDs to delete first (outside transaction)
+        ids_to_delete = list(queryset.values_list("pk", flat=True))
+
+        # Delete everything in a single atomic transaction
+        with transaction.atomic():
+            deleted_count, _ = Snapshot.objects.filter(pk__in=ids_to_delete).delete()
+
         messages.success(
             request,
-            f"Queued {total} snapshots for deletion. They remain visible as Deleting until ArchiveBox verifies their files are removed.",
+            mark_safe(
+                f"Successfully deleted {total} Snapshots ({deleted_count} total objects including related records). Don't forget to scrub URLs from import logs (data/sources) and error logs (data/logs) if needed.",
+            ),
         )
 
     @admin.action(

archivebox/core/admin_tags.py

@@ -7,13 +7,12 @@ from django.contrib import admin, messages
 from django.contrib.admin.options import IS_POPUP_VAR
 from django.http import HttpRequest, HttpResponseRedirect
 from django.template.response import TemplateResponse
-from django.db.models import Prefetch
 from django.urls import reverse
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 
 from archivebox.base_models.admin import BaseModelAdmin
-from archivebox.core.models import ArchiveResult, SnapshotTag, Tag
+from archivebox.core.models import SnapshotTag, Tag
 from archivebox.core.tag_util import (
     TAG_HAS_SNAPSHOTS_CHOICES,
     TAG_SORT_CHOICES,
@@ -25,8 +24,7 @@ from archivebox.core.tag_util import (
     normalize_has_snapshots_filter,
     normalize_tag_sort,
 )
-from archivebox.core.routes_util import build_snapshot_role_output_url
-from archivebox.plugins.discovery import get_snapshot_role_names
+from archivebox.core.routes_util import build_snapshot_url
 
 
 class TagInline(admin.TabularInline):
@@ -193,26 +191,7 @@ class TagAdmin(BaseModelAdmin):
 
     @admin.display(description="Snapshots")
     def snapshots(self, tag: Tag):
-        snapshots = (
-            tag.snapshot_set.select_related("crawl__created_by")
-            .prefetch_related(
-                Prefetch(
-                    "archiveresult_set",
-                    queryset=ArchiveResult.objects.filter(
-                        plugin__in=get_snapshot_role_names("list_icon"),
-                        status=ArchiveResult.StatusChoices.SUCCEEDED,
-                    ).only(
-                        "snapshot_id",
-                        "plugin",
-                        "status",
-                        "output_str",
-                        "output_files",
-                    ),
-                    to_attr="_list_icon_results",
-                ),
-            )
-            .order_by("-downloaded_at", "-created_at", "-pk")[:10]
-        )
+        snapshots = tag.snapshot_set.select_related("crawl__created_by").order_by("-downloaded_at", "-created_at", "-pk")[:10]
         total_count = tag.snapshot_set.count()
         if not snapshots:
             return mark_safe(
@@ -235,12 +214,7 @@ class TagAdmin(BaseModelAdmin):
                 </a>
                 """,
                     reverse("admin:core_snapshot_change", args=[snapshot.pk]),
-                    build_snapshot_role_output_url(
-                        snapshot,
-                        "list_icon",
-                        archive_results=snapshot._list_icon_results,
-                        fallback_to_default=True,
-                    ),
+                    build_snapshot_url(str(snapshot.pk), "favicon.ico"),
                     title[:120],
                     snapshot.url[:120],
                 ),

archivebox/core/asgi.py

@@ -7,31 +7,70 @@ For more information on this file, see
 https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
 """
 
-from django.core.handlers.asgi import ASGIHandler
+from django.core.asgi import get_asgi_application
 
 from archivebox.config.django import setup_django
 
 setup_django(check_db=True)
 
 
-class ArchiveBoxASGIHandler(ASGIHandler):
-    def create_request(self, scope, body_file):
-        # ASGI already spools the complete body. Django's multipart parser
-        # otherwise treats chunked uploads without Content-Length as empty.
-        if not any(name == b"content-length" for name, _value in scope.get("headers", [])):
-            body_file.seek(0, 2)
-            length = body_file.tell()
-            body_file.seek(0)
-            scope = {**scope, "headers": [*scope.get("headers", []), (b"content-length", str(length).encode("ascii"))]}
-        return super().create_request(scope, body_file)
+def _patch_thread_sensitive_context_shutdown() -> None:
+    """Stop ``ThreadSensitiveContext.__aexit__`` from blocking the daphne loop.
+
+    Django 6.0's ASGIHandler wraps every request in ``async with
+    ThreadSensitiveContext():`` (django/core/handlers/asgi.py:169). On exit
+    asgiref calls ``executor.shutdown()`` with the default ``wait=True``
+    (asgiref/sync.py:148), which is a *synchronous* ``Thread.join()`` inside
+    an async function — so it blocks the daphne event loop until the
+    executor's worker thread exits.
+
+    That's normally fine because the request handler has already awaited
+    every ``sync_to_async`` it submitted, so the worker is idle and dies as
+    soon as the shutdown sentinel reaches it. The blocking turns into a
+    problem when a client disconnects mid-request:
+
+    * ``SyncToAsync.__call__`` shields the executor work with
+      ``await asyncio.shield(exec_coro)`` (asgiref/sync.py:506) so that the
+      sync DB call doesn't get torn down halfway through.
+    * On cancellation it calls ``exec_coro.cancel()`` (line 522) which only
+      flips the asyncio ``Future`` to cancelled — the underlying thread
+      keeps running the SQL query.
+    * Control unwinds to ``__aexit__`` while the orphaned thread is still
+      mid-query. ``shutdown(wait=True)`` then blocks the event loop until
+      that orphan finishes.
+
+    Under heavy SQLite contention (the load-test scenario that surfaced
+    this on cabbage) those orphan threads can take 30 seconds each waiting
+    for write locks, and the daphne loop is single-threaded — so every
+    such orphan stalls every other in-flight request, healthchecks time
+    out, and the container goes ``unhealthy``.
+
+    Switching to ``shutdown(wait=False)`` queues the sentinel and returns
+    immediately; the worker thread still exits cleanly once its current
+    task finishes, and asgiref's ``WeakKeyDictionary`` releases the
+    executor as soon as the request's context is GC'd. No per-request
+    teardown guarantee is lost — there was no caller relying on it.
+    """
+    from asgiref import sync as _asgiref_sync
+
+    original_aexit = _asgiref_sync.ThreadSensitiveContext.__aexit__
+
+    async def __aexit__(self, exc, value, tb):  # type: ignore[no-redef]
+        if not self.token:
+            return
+        executor = _asgiref_sync.SyncToAsync.context_to_thread_executor.pop(self, None)
+        if executor is not None:
+            executor.shutdown(wait=False)
+        _asgiref_sync.SyncToAsync.thread_sensitive_context.reset(self.token)
+
+    # Idempotent: only patch once even if asgi.py is reloaded.
+    if getattr(original_aexit, "_archivebox_patched", False):
+        return
+    __aexit__._archivebox_patched = True  # type: ignore[attr-defined]
+    _asgiref_sync.ThreadSensitiveContext.__aexit__ = __aexit__
 
 
-django_application = ArchiveBoxASGIHandler()
+_patch_thread_sensitive_context_shutdown()
 
-
-async def application(scope, receive, send):
-    if scope["type"] == "websocket":
-        from abx_plugins.plugins.opencode.archivebox.views import websocket_view
-
-        return await websocket_view(scope, receive, send)
-    return await django_application(scope, receive, send)
+# Standard Django ASGI application (no websockets/channels needed)
+application = get_asgi_application()

archivebox/core/forms.py

@@ -304,13 +304,13 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             self.can_override_crawl_config = bool(user.is_authenticated and user.is_active and user.is_superuser)
         super().__init__(*args, **kwargs)
 
-        # Persona creation belongs to init/server startup, never a GET of /add/.
+        default_persona = Persona.get_or_create_default()
         persona_queryset = Persona.objects.order_by("name")
         if not self.can_override_crawl_config:
             persona_queryset = filter_personas_by_permissions(persona_queryset, {PERMISSIONS_PUBLIC})
         self.fields["persona"].queryset = persona_queryset
 
-        selected_persona = persona_queryset.filter(name__iexact="Default").first() or persona_queryset.first()
+        selected_persona = persona_queryset.filter(id=default_persona.id).first() or persona_queryset.first()
         default_config = get_config(persona=selected_persona) if selected_persona else get_config()
         if selected_persona:
             self.fields["persona"].initial = selected_persona.name
@@ -338,12 +338,6 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
                 get_choice_field(self, field_name).choices = [(plugin.name, plugin.name) for plugin in grouped_plugins.get(category, [])]
             self.plugin_groups = []
 
-    def clean_persona(self):
-        persona = self.cleaned_data.get("persona")
-        if persona:
-            persona.validate_name(persona.name)
-        return persona
-
     def clean(self):
         cleaned_data = super().clean() or {}
 

archivebox/core/middleware.py

@@ -13,7 +13,6 @@ from django.http import HttpResponseForbidden, HttpResponseNotModified
 from django.shortcuts import redirect
 from django.utils import timezone
 from django.utils.http import http_date
-from django.utils.cache import patch_cache_control, patch_vary_headers
 
 from archivebox.config import VERSION
 from archivebox.config.common import get_config, get_request_config
@@ -33,7 +32,6 @@ from archivebox.core.routes_util import (
     split_host_port,
 )
 from archivebox.core.views import OriginalDomainHostView, SnapshotHostView
-from archivebox.misc.serve_static import FAVICON_CACHE_CONTROL
 
 ADMIN_LOGIN_HINT_COOKIE = "archivebox_admin_logged_in"
 
@@ -91,9 +89,8 @@ def AdminCookieIsolationMiddleware(get_response):
         response = get_response(request)
 
         if request.path == "/admin" or request.path.startswith("/admin/"):
-            if not getattr(response, "xframe_options_exempt", False):
-                response.headers["X-Frame-Options"] = "DENY"
-                response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
+            response.headers["X-Frame-Options"] = "DENY"
+            response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
 
         config = request.__dict__.get("archivebox_config")
         if config is None or config.SERVER_SECURITY_MODE == "auto":
@@ -125,12 +122,13 @@ def AdminCookieIsolationMiddleware(get_response):
 
 
 def CacheControlMiddleware(get_response):
+    snapshot_path_re = re.compile(r"^/[^/]+/\\d{8}/[^/]+/[0-9a-fA-F-]{8,36}/")
     static_cache_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
 
     def middleware(request):
         response = get_response(request)
 
-        if request.path.startswith("/static/") and response.status_code in (200, 304):
+        if request.path.startswith("/static/"):
             rel_path = request.path[len("/static/") :]
             static_path = finders.find(rel_path)
             if static_path:
@@ -155,30 +153,15 @@ def CacheControlMiddleware(get_response):
                     response.headers["Last-Modified"] = http_date(mtime)
                 return response
 
-        if response.status_code in (200, 206, 304) and response.get("Cache-Control") == FAVICON_CACHE_CONTROL:
-            # The same raw favicon bytes are shared across authenticated viewers.
-            # Session middleware may have added Cookie to Vary during authorization.
-            vary = [
-                value.strip()
-                for value in response.get("Vary", "").split(",")
-                if value.strip().lower() not in {"", "cookie", "authorization"}
-            ]
-            if vary:
-                response["Vary"] = ", ".join(vary)
-            else:
-                response.headers.pop("Vary", None)
-            return response
-
-        # Other private/unlisted artifacts remain private; auth errors stay unstored.
-        snapshot_policy = getattr(request, "archivebox_cache_policy", None)
-        if snapshot_policy and response.status_code in (200, 206, 304):
-            if not response.get("Cache-Control"):
-                response["Cache-Control"] = f"{snapshot_policy}, max-age=60, stale-while-revalidate=300"
-            if snapshot_policy == "private":
-                patch_cache_control(response, private=True)
-        elif snapshot_policy or not response.get("Cache-Control"):
-            response["Cache-Control"] = "private, no-store"
-        patch_vary_headers(response, ("Cookie", "Authorization"))
+        if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
+            "Cache-Control",
+        ):
+            config = request.__dict__.get("archivebox_config")
+            if config is None:
+                config = get_config(resolve_plugins=False)
+                request.archivebox_config = config
+            policy = "private" if config.PERMISSIONS == "private" else "public"
+            response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
         return response
 
     return middleware

archivebox/core/migrations/0024_assign_default_crawl.py

@@ -100,12 +100,11 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
     if schema_editor.connection.vendor != "sqlite":
         return
 
-    from django.utils import timezone
+    from django.db import connection
     import uuid as uuid_lib
+    from datetime import datetime
 
-    connection = schema_editor.connection
     cursor = connection.cursor()
-    now = connection.ops.adapt_datetimefield_value(timezone.now())
 
     # Check if there are any snapshots without a crawl
     cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
@@ -126,13 +125,14 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
             INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
             VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
         """,
-            [now],
+            [datetime.now().isoformat()],
         )
 
     # Create a default crawl for migrated snapshots.
     # Depending on migration graph order, later crawls migrations may already
     # have removed output_dir by the time this data migration runs.
     crawl_id = uuid_lib.uuid4().hex
+    now = datetime.now().isoformat()
     cursor.execute("PRAGMA table_info(crawls_crawl)")
     crawl_columns = {row[1] for row in cursor.fetchall()}
     default_columns: list[str] = []
@@ -155,9 +155,9 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
             status, retry_at, created_by_id, schedule_id, config, persona_id
         ) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
                   'Auto-created crawl for migrated snapshots', {default_values_sql}
-                  'sealed', NULL, 1, NULL, '{{}}', NULL)
+                  'sealed', %s, 1, NULL, '{{}}', NULL)
     """,
-        [crawl_id, now, now, crawl_urls],
+        [crawl_id, now, now, crawl_urls, now],
     )
 
     # Assign all snapshots without a crawl to the default crawl

archivebox/core/migrations/0054_snapshot_fs_version_index.py

@@ -1,26 +0,0 @@
-# Generated by Django 6.1 on 2026-09-02
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("core", "0053_alter_archiveresult_options"),
-    ]
-
-    operations = [
-        migrations.RemoveField(
-            model_name="snapshot",
-            name="current_step",
-        ),
-        migrations.AlterField(
-            model_name="snapshot",
-            name="fs_version",
-            field=models.CharField(
-                db_index=True,
-                default="0.9.0",
-                help_text='Filesystem version of this snapshot (e.g., "0.7.0", "0.8.0", "0.9.4").',
-                max_length=10,
-            ),
-        ),
-    ]

archivebox/core/migrations/0055_repair_legacy_retry_timestamps.py

@@ -1,60 +0,0 @@
-"""Repair SQLite queue timestamps written as raw ISO strings by old migrations."""
-
-from django.db import migrations
-from django.utils.dateparse import parse_datetime
-
-
-def repair_legacy_retry_timestamps(apps, schema_editor):
-    connection = schema_editor.connection
-    if connection.vendor != "sqlite":
-        return
-
-    with connection.cursor() as cursor:
-        # 0024 created this Crawl already sealed, but left a due retry marker.
-        # Its orphaned snapshots have their own scheduler rows for maintenance.
-        cursor.execute(
-            """
-            UPDATE crawls_crawl SET retry_at = NULL
-            WHERE status = 'sealed'
-              AND label = 'Migrated from v0.7.2/v0.8.6'
-              AND notes = 'Auto-created crawl for migrated snapshots'
-              AND substr(retry_at, 11, 1) = 'T'
-            """,
-        )
-
-        # SQLite compares stored date strings lexically. A raw `T` timestamp
-        # from a legacy migration sorts differently from Django's space-separated
-        # parameter, so a due row can be selected but never claimed. Preserve
-        # the datetime Django already reads from each row while rewriting its
-        # storage representation through the active database adapter.
-        for table in ("crawls_crawl", "core_snapshot", "machine_binary", "machine_process"):
-            last_id = ""
-            while True:
-                cursor.execute(
-                    f"SELECT id, CAST(retry_at AS TEXT) FROM {table} WHERE id > %s AND substr(retry_at, 11, 1) = 'T' ORDER BY id LIMIT 500",
-                    [last_id],
-                )
-                rows = cursor.fetchall()
-                if not rows:
-                    break
-                for row_id, raw_retry_at in rows:
-                    try:
-                        parsed = parse_datetime(raw_retry_at)
-                    except ValueError:
-                        continue
-                    if parsed is None:
-                        continue
-                    normalized = connection.ops.adapt_datetimefield_value(parsed)
-                    cursor.execute(
-                        f"UPDATE {table} SET retry_at = %s WHERE id = %s AND retry_at = %s",
-                        [normalized, row_id, raw_retry_at],
-                    )
-                last_id = rows[-1][0]
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("core", "0054_snapshot_fs_version_index"),
-    ]
-
-    operations = [migrations.RunPython(repair_legacy_retry_timestamps, migrations.RunPython.noop)]

archivebox/core/migrations/0057_repair_migrated_process_links.py

@@ -1,25 +0,0 @@
-"""Repair process links after both core's legacy copy and machine migrations."""
-
-from importlib import import_module
-
-from django.db import migrations
-
-
-repair_process_binary_iface_links = import_module(
-    "archivebox.machine.migrations.0020_repair_process_binary_iface_links",
-).repair_process_binary_iface_links
-
-
-class Migration(migrations.Migration):
-    atomic = False
-
-    dependencies = [
-        ("core", "0055_repair_legacy_retry_timestamps"),
-        ("machine", "0022_networkinterface_identity_without_mac"),
-    ]
-
-    operations = [
-        # The original repair can run before core.0027 creates legacy Process
-        # rows. Repair again after the copy, including already upgraded DBs.
-        migrations.RunPython(repair_process_binary_iface_links, migrations.RunPython.noop),
-    ]

archivebox/core/migrations/0058_snapshot_deleting_status.py

@@ -1,28 +0,0 @@
-# Generated by Django 6.1 on 2026-09-30 03:32
-
-from django.db import migrations, models
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("core", "0057_repair_migrated_process_links"),
-    ]
-
-    operations = [
-        migrations.AlterField(
-            model_name="snapshot",
-            name="status",
-            field=models.CharField(
-                choices=[
-                    ("queued", "Queued"),
-                    ("started", "Started"),
-                    ("paused", "Paused"),
-                    ("sealed", "Sealed"),
-                    ("deleting", "Deleting"),
-                ],
-                db_index=True,
-                default="queued",
-                max_length=15,
-            ),
-        ),
-    ]

archivebox/core/preview_util.py

@@ -0,0 +1,4 @@
+__package__ = "archivebox.core"
+
+
+EXTENSION_SCREENSHOT_PLUGIN = "chrome_extension_screenshot"

archivebox/core/recovery_util.py

@@ -36,7 +36,9 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "crawls_queued_without_retry_at": 0,
         "snapshots_queued_without_retry_at": 0,
         "snapshots_sealed_with_extension_uploads_only": 0,
-        "archiveresults_interrupted_without_running_process": 0,
+        "archiveresults_backoff": 0,
+        "snapshots_queued_plugin_rows_waiting_on_stale_lease": 0,
+        "archiveresults_started_without_running_process": 0,
         "archiveresults_missing_for_orphaned_hook_processes": 0,
         "snapshots_started_without_running_results": 0,
         "crawls_started_with_due_snapshots": 0,
@@ -44,10 +46,10 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "crawls_started_without_active_snapshots": 0,
     }
 
-    running_hook_processes = Process.objects.filter(
-        archiveresult__snapshot_id=OuterRef("pk"),
-        process_type=Process.TypeChoices.HOOK,
-        status=Process.StatusChoices.RUNNING,
+    running_archiveresults = ArchiveResult.objects.filter(
+        snapshot_id=OuterRef("pk"),
+        status=ArchiveResult.StatusChoices.STARTED,
+        process__status=Process.StatusChoices.RUNNING,
     )
     active_child_snapshots = Snapshot.objects.filter(
         crawl_id=OuterRef("pk"),
@@ -74,9 +76,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         **snapshot_filter,
     ).update(retry_at=now, modified_at=now)
 
-    # Find rare uploads through hook_name's index before checking their parents;
-    # a correlated upload probe otherwise visits every sealed snapshot.
     extension_upload_results = ArchiveResult.objects.filter(
+        snapshot_id=OuterRef("pk"),
         hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
     )
     server_results = ArchiveResult.objects.filter(snapshot_id=OuterRef("pk")).exclude(
@@ -84,15 +85,15 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
     )
     extension_only_snapshots = (
         Snapshot.objects.filter(
-            id__in=extension_upload_results.values("snapshot_id"),
             status=Snapshot.StatusChoices.SEALED,
             crawl__status__in=[Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED],
             **snapshot_filter,
         )
         .annotate(
+            has_extension_upload=Exists(extension_upload_results),
             has_server_result=Exists(server_results),
         )
-        .filter(has_server_result=False)
+        .filter(has_extension_upload=True, has_server_result=False)
     )
     # Older browser-extension uploads could win a race with runner startup:
     # their successful external rows made the fresh Snapshot look finished
@@ -108,18 +109,49 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         retry_at=now,
         modified_at=now,
     )
+    backoff_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.BACKOFF, **result_filter)
     orphaned_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.STARTED, **result_filter).exclude(
         process__status=Process.StatusChoices.RUNNING,
     )
-    # ArchiveResult rows are projections, never work items. Close interrupted
-    # projections as failed and wake their parent Snapshot so abx-dl can replay
-    # the snapshot-level sequence. Indexed subqueries keep this bounded.
+    # ArchiveResult has no retry_at scheduler. Wake only the parent Snapshots
+    # for result rows we are about to repair, then requeue those exact rows.
+    # Using subqueries keeps million-row recovery in SQLite instead of building
+    # Python ID lists or scanning all sealed snapshots.
+    Snapshot.objects.filter(
+        id__in=backoff_results.values("snapshot_id"),
+        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
+        retry_at__isnull=True,
+    ).update(retry_at=now, modified_at=now)
     Snapshot.objects.filter(
         id__in=orphaned_results.values("snapshot_id"),
-        status=Snapshot.StatusChoices.STARTED,
+        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
+        retry_at__isnull=True,
     ).update(retry_at=now, modified_at=now)
-    cleaned["archiveresults_interrupted_without_running_process"] = orphaned_results.update(
-        status=ArchiveResult.StatusChoices.FAILED,
+    cleaned["archiveresults_backoff"] = backoff_results.update(status=ArchiveResult.StatusChoices.QUEUED, modified_at=now)
+    # Targeted plugin rows on final/paused Snapshots are scheduled through the
+    # parent Snapshot.retry_at. retry_at=NULL is the normal idle marker for a
+    # sealed Snapshot and must not be interpreted as queued work just because
+    # old/synthetic ArchiveResult rows exist. If takeover kills the runner
+    # after it leases the Snapshot but before queued ArchiveResult rows finish,
+    # the rows remain QUEUED while retry_at sits in the future. Recovery runs
+    # only after this runner has won the single-runner gate, so it can safely
+    # unlock those stale plugin leases for immediate processing instead of
+    # waiting out the previous owner's full lock timeout.
+    queued_plugin_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.QUEUED, **result_filter)
+    cleaned["snapshots_queued_plugin_rows_waiting_on_stale_lease"] = (
+        Snapshot.objects.filter(
+            id__in=queued_plugin_results.values("snapshot_id"),
+            status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
+        )
+        .filter(retry_at__gt=now)
+        .update(retry_at=now, modified_at=now)
+    )
+    # Impossible state repair: STARTED ArchiveResults without a live Process
+    # have no owner left to emit completion. Requeue only the result row; the
+    # snapshot/crawl schedulers will pick up normal retry processing.
+    cleaned["archiveresults_started_without_running_process"] = orphaned_results.update(
+        status=ArchiveResult.StatusChoices.QUEUED,
+        process=None,
         modified_at=now,
     )
     orphaned_hook_processes = Process.objects.filter(
@@ -141,13 +173,6 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
         else:
             orphaned_hook_processes = orphaned_hook_processes.none()
-    else:
-        # Global startup recovery must stay database-only. Process.pwd points
-        # into archive payload storage, which may be a cold FUSE mount; scanning
-        # every historical unlinked hook here blocks the sole scheduler before
-        # it can claim newly queued work. Explicit crawl recovery below remains
-        # available for the work the caller asked to resume.
-        orphaned_hook_processes = orphaned_hook_processes.none()
     for process in orphaned_hook_processes.only(
         "id",
         "pwd",
@@ -181,11 +206,11 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             plugin_dir.name,
             hook_name,
             defaults={
-                "status": ArchiveResult.StatusChoices.FAILED,
+                "status": ArchiveResult.StatusChoices.QUEUED,
             },
         )
         process_is_newer = bool(process.started_at and (result.start_ts is None or process.started_at >= result.start_ts))
-        if created or process_is_newer:
+        if result.status == ArchiveResult.StatusChoices.QUEUED or process_is_newer:
             requeue_snapshot = False
             # A runner can die after the hook Process exits but before the
             # ProcessCompletedEvent projector links/finalizes ArchiveResult.
@@ -206,14 +231,16 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             result.start_ts = process.started_at
             result.end_ts = process.ended_at
             if _is_signal_interrupted_exit(process.exit_code):
-                # The Process is a durable fact; record the interruption as a
-                # failure and wake the parent Snapshot for replay.
+                # The owning runner died or was asked to stop while the hook was
+                # still active. Keep the work item queued so takeover retries the
+                # same hook; treating an unknown signal exit as success would
+                # silently skip unfinished side effects.
                 result.output_files = {}
                 result.output_size = 0
                 result.output_mimetypes = ""
                 result.output_str = ""
                 result.output_json = None
-                result.status = ArchiveResult.StatusChoices.FAILED
+                result.status = ArchiveResult.StatusChoices.QUEUED
                 requeue_snapshot = True
             else:
                 result.output_files = output_files
@@ -253,6 +280,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
                 Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
         if created:
             cleaned["archiveresults_missing_for_orphaned_hook_processes"] += 1
+            Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
     started_snapshots = Snapshot.objects.filter(status=Snapshot.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
         **snapshot_filter,
@@ -266,8 +294,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
     # We only unlock scheduling; normal Snapshot runner code owns the next
     # transition and side effects.
     cleaned["snapshots_started_without_running_results"] = (
-        started_snapshots.annotate(has_running_process=Exists(running_hook_processes))
-        .filter(has_running_process=False)
+        started_snapshots.annotate(has_running_results=Exists(running_archiveresults))
+        .filter(has_running_results=False)
         .update(
             retry_at=now,
             modified_at=now,
@@ -324,9 +352,13 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             "Finishing {count} browser-extension Snapshot(s) that received uploaded files before server extractors started "
             "(uploaded and server-created results will remain together on the same Snapshot)."
         ),
-        "archiveresults_interrupted_without_running_process": (
-            "Closing {count} interrupted extractor result projection(s) "
-            "(their parent Snapshots are resumed through the normal snapshot-level runner)."
+        "archiveresults_backoff": (
+            "Retrying {count} extractor result(s) that were waiting to retry "
+            "(ArchiveBox may have been interrupted before it was able to try them again; affected outputs will be retried)."
+        ),
+        "archiveresults_started_without_running_process": (
+            "Retrying {count} extractor result(s) that were interrupted before finishing "
+            "(ArchiveBox may have been interrupted before it was able to save their final status; partial files will be overwritten with fresh results upon retry)."
         ),
         "snapshots_started_without_running_results": (
             "Resuming {count} Snapshot(s) that were interrupted before finishing "