Update SECURITY.md with version security details

Clarify security risks and protections for versions prior to v0.9.0.

.dockerignore

@@ -13,10 +13,14 @@ __pycache__
 .pytest_cache/
 .ruff_cache/
 .uv-cache/
+.cache/
+.cache*/
 .github/
 .pdm-build/
+.pdm-build/**
 .pdm-python
 .eggs/
+*.egg-info/
 .git/
 !.git/
 .git/*
@@ -42,6 +46,7 @@ chromeprofile/
 chrome_profile/
 lib/
 out/
+tests/out/
 users/
 archive/
 crawls/
@@ -63,6 +68,8 @@ dist/
 brew_dist/
 deb_dist/
 pip_dist/
+cache/
+caches/
 assets/
 docker/
 website/

.github/SECURITY.md

@@ -2,18 +2,15 @@
 
 ---
 
-## Security Information
+## Security Documentation
 
-Please see this wiki page for important notices about ArchiveBox security, publishing your archives securely, and the dangers of executing archived JS:
-
-https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
-
-Also see this section of the README about important caveats when running ArchiveBox:
-
-https://github.com/ArchiveBox/ArchiveBox?tab=readme-ov-file#caveats
-
-You can also read these pages for more information about ArchiveBox's internals, development environment, DB schema, and more:
+Please start by reading these links to understand our existing security model, known issues, API surfaces, and setup steps:
 
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
+- https://github.com/ArchiveBox/ArchiveBox/security/advisories/
+- https://github.com/ArchiveBox/ArchiveBox/tree/v0.7.4#security-risks-of-viewing-archived-js
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#security-concerns
+- https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication
 - https://github.com/ArchiveBox/ArchiveBox#archive-layout
 - https://github.com/ArchiveBox/ArchiveBox#archivebox-development
 - https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives
@@ -32,3 +29,43 @@ We use Github's built-in [Private Reporting](https://docs.github.com/en/code-sec
 3. Fill out the form to submit the details of the report and it will be securely sent to the maintainers
 
 You can also contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp).
+
+---
+
+## CVEs and Report Credits
+
+> [!IMPORTANT]
+> **Reports that only affect `dev` or beta versions will be converted to normal Github issues and will not be issued CVEs.**  
+> Reporters **will still receive credit in release notes** however.
+
+> [!TIP]
+> Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.
+
+Please read our existing published [security advisories](https://github.com/ArchiveBox/ArchiveBox/security/advisories), [Security Overview Docs](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview), and [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and avoid creating duplicates.
+
+---
+
+## Threat Model
+
+### All admin users are assumed to be trusted
+
+ArchiveBox does not support granular user permissions, there are only two types of users available in the Admin UI & REST API:
+
+- logged in admin superusers (which are assumed to have `root` on the ArchiveBox host machine already)
+- non-logged in anonymous guest users
+
+Because admin / REST API / CLI / Python API users are already assumed to have `root` on the host machine, there are deliberately no protections against admins inputting malicious config values, attempts to attack other admin users, or destructive/malicious actions. Permissions for non-logged in users are managed via the [`PERMISSIONS`](https://github.com/archiveBox/archiveBox/wiki/configuration#permissions) + [`PUBLIC_INDEX` + `PUBLIC_ADD_VIEW`](https://github.com/archiveBox/archiveBox/wiki/configuration#public_index--public_add_view) + other config options.
+
+### Sanitization of Config Values, Filesystem Paths, DB Fields, etc.
+
+Non-logged in users can only choose between public Persona config presets on the `/add/` page, but they should never be allowed to directly set config options like `*_ARGS`, `*_BINARY` or DB fields like `url`, `title`, `tag.name`, `notes` without sanitization/filtering on save and/or use.
+
+Non-logged in users should be tightly limited to prevent attacks like SQL/shell injection, path traversal attacks, XSS, cookie leaks between admin/non-admin domains, etc. Any reports related to non-logged-in users will be prioritized as this is a surface we care a lot about hardening.
+
+### XSS, CSRF, CORS, CSP
+
+`< v0.9.x` versions do not include any XSS/CSRF protections at all, as documented in the [readme](https://github.com/ArchiveBox/ArchiveBox/tree/v0.7.4#security-risks-of-viewing-archived-js), [wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#security-concerns), [issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239), and [existing advisory](https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx). Do not open advisories related to XSS or CSRF for those versions, as they are missing fundamental architectural changes added in `v0.9.0` needed to support per-snapshot replay isolation.
+
+[`SERVER_SECURITY_MODE`](https://github.com/archiveBox/archiveBox/wiki/configuration#server_security_mode) + CSRF/XSS defenses were only added in the new 0.9.0 `dev` work and are planned to land in `main` in mid-2026.
+
+Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs. The only URLs that are blocked are URLs hosted by the archivebox server itself.

.github/workflows/docker.yml

@@ -13,7 +13,6 @@ on:
 env:
   DOCKERHUB_IMAGE: archivebox/archivebox
   GHCR_IMAGE: ghcr.io/archivebox/archivebox
-  ABX_DL_IMAGE: archivebox/abx-dl:latest
 
 permissions:
   contents: read
@@ -105,6 +104,23 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
+          ABX_DL_VERSION="$(python3 - <<'PY'
+          import re
+          import tomllib
+
+          with open("pyproject.toml", "rb") as f:
+              deps = tomllib.load(f)["project"]["dependencies"]
+
+          for dep in deps:
+              match = re.match(r"abx-dl\s*(?:==|>=)\s*([^,;\s]+)", dep)
+              if match:
+                  print(match.group(1))
+                  break
+          else:
+              raise SystemExit("Missing abx-dl dependency in pyproject.toml")
+          PY
+          )"
+          ABX_DL_IMAGE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
           deadline=$((SECONDS + 1800))
           until docker buildx imagetools inspect "${ABX_DL_IMAGE}" >/tmp/abx-dl-image.json; do
             if (( SECONDS >= deadline )); then
@@ -173,12 +189,43 @@ jobs:
             ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
           cache-from: type=gha,scope=${{ matrix.cache_scope }}
           cache-to: type=gha,mode=max,scope=${{ matrix.cache_scope }}
+          pull: true
           platforms: ${{ matrix.platform }}
           outputs: type=image,push-by-digest=true,name-canonical=true,push=true
 
       - name: Image digest
         run: echo ${{ steps.docker_build.outputs.digest }}
 
+      - name: Validate pushed image version, commit, and size
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          IMAGE="${DOCKERHUB_IMAGE}@${{ steps.docker_build.outputs.digest }}"
+          SHORT_SHA="${GITHUB_SHA::7}"
+
+          DATA_DIR="$(mktemp -d)"
+          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
+          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox version | tee /tmp/archivebox-version.out
+          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox status
+          docker run --rm --platform '${{ matrix.platform }}' --entrypoint cat "$IMAGE" /VERSION.txt > /tmp/archivebox-image-version.txt
+          grep -F "ArchiveBox v${{ steps.docker_meta.outputs.version }}" /tmp/archivebox-version.out
+          grep -F "COMMIT_HASH=${SHORT_SHA}" /tmp/archivebox-version.out
+          grep -Fx "COMMIT_HASH=${GITHUB_SHA}" /tmp/archivebox-image-version.txt
+
+          python3 - "$IMAGE" <<'PY'
+          import json
+          import subprocess
+          import sys
+
+          image = sys.argv[1]
+          limit = 780 * 1024 * 1024
+          manifest = json.loads(subprocess.check_output(["docker", "manifest", "inspect", image]))
+          total = manifest.get("config", {}).get("size", 0) + sum(layer.get("size", 0) for layer in manifest.get("layers", []))
+          print(f"{image} compressed_size={total / 1024 / 1024:.2f} MiB limit={limit / 1024 / 1024:.2f} MiB")
+          if total > limit:
+              raise SystemExit(f"{image} is over the compressed size limit")
+          PY
+
       - name: Export digest
         shell: bash
         run: |

.github/workflows/pip.yml

@@ -136,8 +136,6 @@ jobs:
         uses: pypa/gh-action-pypi-publish@release/v1
         # ignore when publish to PyPI fails due to duplicate tag
         continue-on-error: true
-        with:
-          password: ${{ secrets.PYPI_PAT_SECRET }}
 
       - name: UV install archivebox and archivebox/pkgs/* locally for tests
         env:
@@ -146,13 +144,14 @@ jobs:
           uv sync --project .ci-uv-project --all-extras --no-install-project --no-install-workspace --no-sources --no-cache
           uv pip install --python .venv/bin/python --no-deps dist/archivebox-*.whl
 
-      - name: UV run archivebox init + archivebox version
+      - name: Verify built package full install
         env:
           UV_PROJECT_ENVIRONMENT: ${{ github.workspace }}/.venv
         run: |
-          mkdir -p data && cd data
-          uv run --project ../.ci-uv-project --no-sync --no-sources archivebox init \
-          && uv run --project ../.ci-uv-project --no-sync --no-sources archivebox version
-          # && uv run archivebox add 'https://example.com' \
-          # && uv run archivebox status \
-          # || (echo "UV Failed to run archivebox!" && exit 1)
+          set -Eeuo pipefail
+          DATA_DIR="$(mktemp -d)"
+          cd "$DATA_DIR"
+          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox init
+          timeout 30m uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox install
+          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox version
+          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox status

.github/workflows/release-runner.yml

@@ -3,26 +3,30 @@ name: Release State
 on:
   push:
     branches:
-      - '**'
+      - dev
   workflow_dispatch:
 
 permissions:
+  actions: write
   contents: write
   id-token: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+  cancel-in-progress: false
 
 jobs:
   release-state:
+    if: github.event_name != 'push' || !startsWith(github.event.head_commit.message, 'release:')
     runs-on: ubuntu-latest
+    environment: pypi
     steps:
       - uses: actions/checkout@v4
         with:
           fetch-depth: 0
           submodules: true
           ref: ${{ github.ref_name }}
+          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
 
       - uses: actions/setup-python@v5
         with:
@@ -44,7 +48,13 @@ jobs:
 
       - name: Run release script
         env:
-          DEFAULT_BRANCH: ${{ github.event.repository.default_branch }}
-          GH_TOKEN: ${{ github.token }}
-          PYPI_PAT_SECRET: ${{ secrets.PYPI_PAT_SECRET }}
+          DEFAULT_BRANCH: dev
+          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
         run: ./bin/release.sh
+
+      - name: Dispatch image and tap updates
+        env:
+          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
+        run: |
+          gh workflow run docker.yml --ref dev
+          gh workflow run update-homebrew-tap.yml --ref dev

.github/workflows/test.yml

@@ -81,10 +81,15 @@ jobs:
           pwd
           ls
 
-      - name: Archivebox version
+      - name: ArchiveBox full install check
         run: |
-          mkdir -p tests/out/data
-          DATA_DIR="$PWD/tests/out/data" uv run --no-sync --no-sources archivebox version
+          set -Eeuo pipefail
+          DATA_DIR="$PWD/tests/out/data"
+          mkdir -p "$DATA_DIR"
+          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox init
+          timeout 30m uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox install
+          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox version
+          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox status
 
       - name: Test built package with pytest
         # TODO: remove this exception for windows once we get tests passing on that platform

.github/workflows/update-homebrew-tap.yml

@@ -1,4 +1,4 @@
-name: Update Homebrew tap
+name: Update downstream packages
 
 on:
   push:
@@ -9,7 +9,7 @@ permissions:
   contents: read
 
 concurrency:
-  group: update-homebrew-tap-${{ github.ref }}
+  group: update-downstream-packages-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
@@ -17,19 +17,21 @@ jobs:
     runs-on: ubuntu-latest
 
     steps:
-      - name: Dispatch tap update
+      - name: Dispatch downstream package updates
         env:
-          HOMEBREW_TAP_TOKEN: ${{ secrets.HOMEBREW_TAP_TOKEN }}
+          DOWNSTREAM_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || secrets.HOMEBREW_TAP_TOKEN }}
         run: |
-          if [ -z "$HOMEBREW_TAP_TOKEN" ]; then
-            echo "HOMEBREW_TAP_TOKEN is not configured; tap hourly fallback will update the formula."
+          if [ -z "$DOWNSTREAM_TOKEN" ]; then
+            echo "RELEASE_GH_TOKEN/HOMEBREW_TAP_TOKEN is not configured; downstream schedules will update packages."
             exit 0
           fi
 
-          curl --fail-with-body \
-            -X POST \
-            -H "Accept: application/vnd.github+json" \
-            -H "Authorization: Bearer ${HOMEBREW_TAP_TOKEN}" \
-            -H "X-GitHub-Api-Version: 2022-11-28" \
-            https://api.github.com/repos/ArchiveBox/homebrew-archivebox/dispatches \
-            -d '{"event_type":"archivebox-dev-updated"}'
+          for repo in homebrew-archivebox debian-archivebox; do
+            curl --fail-with-body \
+              -X POST \
+              -H "Accept: application/vnd.github+json" \
+              -H "Authorization: Bearer ${DOWNSTREAM_TOKEN}" \
+              -H "X-GitHub-Api-Version: 2022-11-28" \
+              "https://api.github.com/repos/ArchiveBox/${repo}/dispatches" \
+              -d '{"event_type":"archivebox-dev-updated","client_payload":{"ref":"dev"}}'
+          done

AGENTS.md

@@ -1,14 +1,87 @@
-I found 31 unique `no mocking` prompts in recent Codex history, across 21 session transcripts. Only one transcript literally started with `no mocking`; most had it later as testing guidance. Consolidated advice:
+# ArchiveBox Agent Guide
 
-- Tests must hit real user-facing code paths: CLI commands, REST/API calls, browser UI, real hooks, real ArchiveBox data dirs, real pytest fixtures, and real subprocess/binary behavior.
-- No mocking, faking, simulating, monkey patching, handwritten fake objects, fake buses, fake hooks, fake binaries, fake handlers, or direct-post shortcuts when the user path is through UI/extension/CLI.
-- No skipped, xfailed, flaky, or “works around platform” tests. Flakiness is treated as a bug, especially on macOS/browser flows.
-- Prefer live integration tests over narrow unit tests when behavior depends on browsers, binaries, ArchiveBox crawls, plugins, LLMs, or server state.
-- Assertions must validate real correctness: returned values, exit codes, DB rows, filesystem contents, field values, uploaded files, rendered output, and side effects. “No error occurred” or “attribute exists” is not enough.
-- Start fixes with failing red tests that reproduce the missing behavior or regression, then implement the minimal fix and confirm the test passes.
-- Use realistic setup patterns “like a user would”: events + bus + handlers, real browser pages/CDP sessions, real URLs or `pytest-httpserver`, real rows, real snapshots, real installs, real local browser/server state.
-- For ArchiveBox/API tests, use existing `conftest.py` fixtures and test harnesses, real test DB rows/data dirs, and user-facing commands/APIs rather than bespoke helpers.
-- For browser/extension tests, trigger behavior through the real extension UI or actual browser session, not direct posting or mocked browser/session objects.
-- For binary/provider tests, use real binaries and real installs; verify constraints and final installed package metadata, not just install success.
-- For coverage quality, keep tests strict, deterministic, grouped consistently, and use a few larger realistic tests when that gives better surface coverage than many tiny fake unit tests.
-- Avoid weakening test coverage, adding compatibility/shim/fallback layers, or guessing from code shape. Trace root causes, verify assumptions with tests/scripts, and let real type/parse errors surface normally.
+ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch.
+
+## Shared Standards
+
+- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
+- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
+- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
+- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
+- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
+- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
+- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
+- Start behavior fixes with a red failing test when a test is requested or practical.
+- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
+- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
+
+## Development Setup
+
+```bash
+uv sync --dev --all-extras
+mkdir -p data
+cd data
+uv run --project .. archivebox init --install
+```
+
+Run collection commands from inside an initialized data directory:
+
+```bash
+cd data
+uv run --project .. archivebox status
+uv run --project .. archivebox add 'https://example.com'
+uv run --project .. archivebox run
+```
+
+## User-Facing Setup
+
+Recommended CLI install:
+
+```bash
+uv tool install archivebox
+mkdir -p ~/archivebox/data
+cd ~/archivebox/data
+archivebox init --install
+archivebox add 'https://example.com'
+archivebox server 0.0.0.0:8000
+```
+
+Alternative install methods:
+
+- Docker Compose / Docker
+- Homebrew
+- Debian package
+- pip
+
+## Basic Usage
+
+```bash
+archivebox version
+archivebox help
+archivebox status
+archivebox install
+archivebox add 'https://example.com'
+archivebox add --extract=title,screenshot,pdf 'https://example.com'
+archivebox list --json --with-headers
+archivebox search 'example'
+archivebox update --filter-type=domain example.com
+archivebox remove --filter-type=exact 'https://example.com'
+archivebox run
+```
+
+## Verification
+
+Use targeted tests for focused work:
+
+<!--pytest.mark.skip(reason="pytest invocation")-->
+```bash
+uv run pytest archivebox/tests/test_cli_add.py -q
+uv run prek run --all-files
+```
+
+Use the full release/deploy loop only when requested:
+
+<!--pytest.mark.skip(reason="release/deploy script")-->
+```bash
+./bin/release_dev_stack.sh
+```

Dockerfile

@@ -2,7 +2,7 @@
 
 # Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
 # abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
-# ArchiveBox owns ripgrep, sonic, supervisor, Django, and the app runtime.
+# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
 # Build abx-dl first, then point this file at it:
 #   docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
 #       --build-context abxbus=../abxbus \
@@ -48,11 +48,9 @@ ENV ARCHIVEBOX_USER=archivebox \
 ENV CODE_DIR=/app \
     DATA_DIR=/data \
     CONFIG_DIR=/opt/archivebox \
-    LIB_DIR=/opt/archivebox/lib \
     ABXPKG_LIB_DIR=/opt/archivebox/lib \
     PLAYWRIGHT_BROWSERS_PATH=/opt/archivebox/lib/playwright/cache \
     PERSONAS_DIR=/data/personas \
-    CHROME_USER_DATA_DIR=/data/personas/Default/chrome_profile \
     CHROME_HEADLESS=true \
     CHROME_SANDBOX=false \
     CHROME_ISOLATION=crawl \
@@ -89,16 +87,10 @@ RUN cp /VERSION.txt /ABX-DL-VERSION.txt \
     && echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
     && uname -a \
     && sed -n '1,7p' /etc/os-release \
-    && which node && node --version \
-    && which uv && uv self version \
+    && abxpkg load --binproviders=env node \
+    && abxpkg load --binproviders=env uv \
     ) | tee -a /VERSION.txt
 
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
-    echo "[+] APT Installing ArchiveBox search dependency ripgrep for $TARGETPLATFORM..." \
-    && apt-get update -qq \
-    && apt-get install -qq -y --no-install-recommends ripgrep \
-    && rm -rf /var/lib/apt/lists/*
-
 FROM archivebox-runtime-base AS archivebox-builder
 
 WORKDIR "$CODE_DIR"
@@ -185,8 +177,7 @@ RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked,id=uv-$TARGETARCH$T
     && if [[ "$COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "COMMIT_HASH=$COMMIT_HASH" | tee -a /VERSION.txt; fi \
     && /usr/bin/uv pip install --no-deps "$CODE_DIR" \
     && rm -f /venv/bin/uv /venv/bin/uvx \
-    && (/usr/bin/uv pip show archivebox && which archivebox) | tee -a /VERSION.txt \
-    && rm -rf "$CODE_DIR/.git"
+    && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
 FROM archivebox-runtime-base
 
@@ -220,42 +211,42 @@ RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..."
     && usermod --append --groups audio,video "$ARCHIVEBOX_USER" \
     && [[ "$(id -u "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_UID" ]] || usermod -u "$DEFAULT_ARCHIVEBOX_UID" "$ARCHIVEBOX_USER" \
     && [[ "$(id -g "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_GID" ]] || groupmod -g "$DEFAULT_ARCHIVEBOX_GID" "$ARCHIVEBOX_USER" \
-    && (which sonic && sonic --version) | tee -a /VERSION.txt \
-    && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
+    && abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
+    && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
-    && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
+    && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
+    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
-    && echo -e "TMP_DIR=$TMP_DIR\nLIB_DIR=$LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
+    && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
 
 WORKDIR "$DATA_DIR"
 RUN echo "[+] Initializing image collection..." \
     && find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
-    && archivebox init \
+    && HOME="$TMP_DIR" archivebox init \
     && (chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" \
         "$DATA_DIR" "$DATA_DIR"/.archivebox_id "$DATA_DIR"/ArchiveBox.conf "$DATA_DIR"/index.sqlite3 \
         "$DATA_DIR"/logs "$DATA_DIR"/logs/* "$DATA_DIR"/sources \
         "$DATA_DIR"/archive "$DATA_DIR"/archive/users "$DATA_DIR"/personas \
         "$DATA_DIR"/tmp "$DATA_DIR"/tmp/* \
         "$CONFIG_DIR" "$CONFIG_DIR"/config.env "$CONFIG_DIR"/derived.env \
-        "$TMP_DIR" "$LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
+        "$TMP_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
         2>/dev/null || true) \
     && find "$TMP_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
 
 RUN chmod +x "$CODE_DIR"/bin/*.sh \
-    && chmod g+w "$TMP_DIR" "$LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
+    && chmod g+w "$TMP_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
 
-RUN "$LIB_DIR/playwright/bin/chromium" --version | tee -a /VERSION.txt \
-    && "$LIB_DIR/uv/packages/papers-dl/venv/bin/papers-dl" --version | tee -a /VERSION.txt \
-    && /usr/bin/rg --version | head -1 | tee -a /VERSION.txt \
-    && /usr/local/bin/sonic --version | tee -a /VERSION.txt \
-    && /venv/bin/supervisord --version | tee -a /VERSION.txt \
-    && for forbidden_bin in gcc g++ make; do ! command -v "$forbidden_bin" || (echo "Unexpected build tool in runtime: $forbidden_bin=$(command -v "$forbidden_bin")" >&2 && exit 1); done \
-    && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
+RUN --mount=type=cache,target=/tmp/abxpkg-cache,sharing=locked,mode=1777 \
+    for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
+    && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$CONFIG_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox version 2>&1 | tee -a /VERSION.txt \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox install \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$ABXPKG_LIB_DIR" \
+    && python3 -c 'from abx_dl.models import discover_plugins; [print(f"export {plugin.enabled_key}=True") for plugin in discover_plugins(runtime="archivebox").values() if plugin.enabled_key in plugin.config.properties]' > /tmp/archivebox-enable-plugins.env \
+    && sort /tmp/archivebox-enable-plugins.env | tee -a /VERSION.txt \
+    && source /tmp/archivebox-enable-plugins.env \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache ABXPKG_NO_CACHE=True ABXPKG_TMP_CACHE_DIR=/tmp/abxpkg-cache archivebox install \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache archivebox version 2>&1 | tee -a /VERSION.txt \
     && rm -rf /root/.cache /var/cache/apt/* /var/lib/apt/lists/*
 
 RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully." \

archivebox/cli/__init__.py

@@ -18,10 +18,11 @@ if "--debug" in sys.argv:
 
 # Universal `--init` flag: when passed to ANY subcommand (e.g. `archivebox server --init`,
 # `archivebox add --init`, `archivebox shell --init`), run a `quick` archivebox init before
-# the subcommand executes. Strip it from argv here so each subcommand's own click parser
-# never sees it. Ignored for `help` and `init` themselves.
-if "--init" in sys.argv:
-    sys.argv = [arg for arg in sys.argv if arg != "--init"]
+# the subcommand executes. `--quick-init` is the old server spelling. Strip either from
+# argv here so each subcommand's own click parser never sees it. Ignored for `help` and
+# `init` themselves.
+if "--init" in sys.argv or "--quick-init" in sys.argv:
+    sys.argv = [arg for arg in sys.argv if arg not in ("--init", "--quick-init")]
     os.environ["ARCHIVEBOX_WANTS_INIT"] = "1"
 
 
@@ -32,6 +33,7 @@ class ArchiveBoxGroup(click.Group):
         "help": "archivebox.cli.archivebox_help.main",
         "version": "archivebox.cli.archivebox_version.main",
         "mcp": "archivebox.cli.archivebox_mcp.main",
+        "oneshot": "archivebox.cli.archivebox_oneshot.main",
     }
     setup_commands = {
         "init": "archivebox.cli.archivebox_init.main",

archivebox/cli/archivebox_add.py

@@ -331,6 +331,7 @@ def add(
 @click.option("--url-denylist", "--domain-denylist", default="", help="Comma-separated URL/domain denylist for this crawl")
 @click.option("--parser", default="auto", help="Parser for reading input URLs (auto, txt, html, rss, json, jsonl, netscape, ...)")
 @click.option("--plugins", "-p", default="", help="Comma-separated list of plugins to run e.g. title,favicon,screenshot,singlefile,...")
+@click.option("--extract", default="", help="Comma-separated plugins or output types to extract e.g. title,pdf,text/html,image,...")
 @click.option("--persona", default="Default", help="Authentication profile to use when archiving")
 @click.option(
     "--only-new/--no-only-new",
@@ -380,10 +381,23 @@ def main(**kwargs):
         overwrite = kwargs.pop("overwrite", False)
         update = kwargs.pop("update", False)
         only_new = kwargs.pop("only_new", None)
+        extract = kwargs.pop("extract", "")
         if overwrite or update:
             only_new = False
         if only_new is not None:
             kwargs["config"] = {"ONLY_NEW": bool(only_new)}
+        if extract:
+            from abx_dl.models import discover_plugins, plugins_matching_output
+
+            all_plugins = discover_plugins()
+            tokens = [token.strip() for token in extract.split(",") if token.strip()]
+            plugin_names = {name.lower(): name for name in all_plugins}
+            selected = [plugin_names[token.lower()] for token in tokens if token.lower() in plugin_names]
+            selected += plugins_matching_output(all_plugins, tokens)
+            if not selected:
+                raise click.UsageError(f"No plugins found matching extract types: {extract}")
+            existing = [token.strip() for token in (kwargs.get("plugins") or "").split(",") if token.strip()]
+            kwargs["plugins"] = ",".join(dict.fromkeys([*existing, *selected]))
 
         add(urls=urls, **kwargs)
 

archivebox/cli/archivebox_init.py

@@ -153,8 +153,8 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
 
     config = get_config()
     config.TMP_DIR.mkdir(parents=True, exist_ok=True)
-    config.LIB_DIR.mkdir(parents=True, exist_ok=True)
-    (config.LIB_DIR / "bin").mkdir(parents=True, exist_ok=True)
+    config.ABXPKG_LIB_DIR.mkdir(parents=True, exist_ok=True)
+    (config.ABXPKG_LIB_DIR / "bin").mkdir(parents=True, exist_ok=True)
 
     working_tmp_dir = get_or_create_working_tmp_dir(autofix=True, quiet=True)
     if working_tmp_dir:
@@ -186,7 +186,7 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
 @click.command()
 @click.option("--force", "-f", is_flag=True, help="Ignore unrecognized files in current directory and initialize anyway")
 @click.option("--quick", "-q", is_flag=True, help="Run any updates or migrations without rechecking all snapshot dirs")
-@click.option("--install", "-s", is_flag=True, help="Automatically install dependencies and extras used for archiving")
+@click.option("--install", "--setup", "-s", is_flag=True, help="Automatically install dependencies and extras used for archiving")
 @docstring(init.__doc__)
 def main(**kwargs) -> None:
     init(**kwargs)

archivebox/cli/archivebox_install.py

@@ -3,6 +3,8 @@
 __package__ = "archivebox.cli"
 
 import os
+import shutil
+from pathlib import Path
 
 import rich_click as click
 from rich import print
@@ -10,6 +12,38 @@ from rich import print
 from archivebox.misc.util import docstring, enforce_types
 
 
+def ensure_data_dir_lib_symlink(data_dir: Path, abxpkg_lib_dir: Path) -> Path | None:
+    """Expose the shared abxpkg lib dir at DATA_DIR/lib after install."""
+    link_path = data_dir / "lib"
+    target_config_path = abxpkg_lib_dir.expanduser().absolute()
+    target_path = abxpkg_lib_dir.expanduser().resolve()
+    from archivebox.config.paths import get_machine_type
+
+    scoped_data_lib_path = (link_path / get_machine_type()).absolute()
+
+    if target_config_path == scoped_data_lib_path:
+        if link_path.is_symlink() or link_path.is_file():
+            link_path.unlink()
+        target_config_path.mkdir(parents=True, exist_ok=True)
+        return None
+
+    if link_path.resolve(strict=False) == target_path:
+        return None
+
+    target_path.mkdir(parents=True, exist_ok=True)
+
+    if link_path.is_symlink():
+        if link_path.resolve(strict=False) == target_path:
+            return link_path
+        link_path.unlink()
+    elif link_path.exists():
+        shutil.rmtree(link_path)
+
+    relative_target = os.path.relpath(target_path, start=link_path.parent)
+    link_path.symlink_to(relative_target, target_is_directory=True)
+    return link_path
+
+
 @enforce_types
 def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bool = False) -> None:
     """Detect and install ArchiveBox dependencies by running the abx-dl install flow
@@ -67,6 +101,11 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
 
     run_install(plugin_names=plugin_names or None)
 
+    from archivebox.config.common import get_config
+
+    config = get_config(include_machine=False)
+    ensure_data_dir_lib_symlink(CONSTANTS.DATA_DIR, config.ABXPKG_LIB_DIR)
+
     print()
 
     # Check for superuser

archivebox/cli/archivebox_oneshot.py

@@ -0,0 +1,27 @@
+#!/usr/bin/env python3
+
+__package__ = "archivebox.cli"
+__command__ = "archivebox oneshot"
+
+import subprocess
+from pathlib import Path
+
+import rich_click as click
+
+from archivebox.config import CONSTANTS
+
+
+@click.command(add_help_option=False, context_settings=dict(ignore_unknown_options=True))
+@click.argument("args", nargs=-1)
+def main(args: tuple[str, ...] = ()) -> None:
+    """Download URLs using abx-dl"""
+    cwd = Path.cwd()
+    if any((path / CONSTANTS.SQL_INDEX_FILENAME).exists() for path in (cwd, *cwd.parents)):
+        raise click.ClickException(
+            "Refusing to run `archivebox oneshot` inside an ArchiveBox DATA_DIR. Use `archivebox add` here, or run oneshot from another directory.",
+        )
+    raise SystemExit(subprocess.run(["abx-dl", *args]).returncode)
+
+
+if __name__ == "__main__":
+    main()

archivebox/cli/archivebox_remove.py

@@ -34,6 +34,7 @@ def remove(
     after: float | None = None,
     before: float | None = None,
     yes: bool = False,
+    delete: bool = False,
     out_dir: Path = CONSTANTS.DATA_DIR,
     timeout: float | None = None,
     **kwargs,
@@ -142,6 +143,7 @@ def remove(
 
 @click.command()
 @click.option("--yes", is_flag=True, help="Remove links instantly without prompting to confirm")
+@click.option("--delete", is_flag=True, help="Compatibility no-op; archived content is deleted automatically")
 @click.option("--timeout", type=float, default=None, help="Maximum seconds to spend deleting snapshots")
 @snapshot_filter_options(default_filter_type="exact")
 @docstring(remove.__doc__)

archivebox/cli/archivebox_run.py

@@ -287,7 +287,7 @@ def run_runner(
         current.mark_exited()
         return 0
 
-    recover_orchestrator_state(include_chrome=True)
+    recover_orchestrator_state(include_chrome=crawl_id is None, crawl_id=crawl_id)
     if crawl_id:
         from django.utils import timezone
         from archivebox.crawls.models import Crawl

archivebox/cli/archivebox_server.py

@@ -231,6 +231,7 @@ def server(
     debug: bool = False,
     daemonize: bool = False,
     nothreading: bool = False,
+    createsuperuser: bool = False,
 ) -> None:
     """Run the ArchiveBox HTTP server"""
     from archivebox.config.common import get_config
@@ -244,6 +245,12 @@ def server(
 
     from django.contrib.auth.models import User
 
+    if createsuperuser:
+        from archivebox.cli.archivebox_manage import manage
+
+        manage(args=["createsuperuser"])
+        print()
+
     if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
         print()
         print(
@@ -404,6 +411,7 @@ def server(
 @click.option("--debug", is_flag=True, help="Enable DEBUG=True mode with more verbose errors")
 @click.option("--nothreading", is_flag=True, help="Force runserver to run in single-threaded mode")
 @click.option("--daemonize", is_flag=True, help="Run the server in the background as a daemon")
+@click.option("--createsuperuser", is_flag=True, help="Run archivebox manage createsuperuser before starting the server")
 @docstring(server.__doc__)
 def main(**kwargs):
     server(**kwargs)

archivebox/cli/archivebox_version.py

@@ -29,8 +29,7 @@ def _format_binary_abspath(
 
     candidate_bases: tuple[tuple[Path, str], ...] = (
         (pwd, "./"),
-        (lib_dir, "LIB_DIR/"),
-        (Path(os.environ.get("LIB_DIR", "")), "LIB_DIR/") if os.environ.get("LIB_DIR") else (Path(), ""),
+        (lib_dir, "ABXPKG_LIB_DIR/"),
         (personas_dir, "PERSONAS_DIR/"),
         (home, "~/"),
     )
@@ -47,8 +46,8 @@ def _format_binary_abspath(
             relative_str = relative.as_posix()
             if prefix == "./":
                 return "." if not relative_str else f"./{relative_str}"
-            if prefix == "LIB_DIR/":
-                return "LIB_DIR" if not relative_str else f"LIB_DIR/{relative_str}"
+            if prefix == "ABXPKG_LIB_DIR/":
+                return "ABXPKG_LIB_DIR" if not relative_str else f"ABXPKG_LIB_DIR/{relative_str}"
             if prefix == "PERSONAS_DIR/":
                 return "PERSONAS_DIR" if not relative_str else f"PERSONAS_DIR/{relative_str}"
             return "~" if not relative_str else f"~/{relative_str}"
@@ -59,10 +58,10 @@ def _format_binary_abspath(
 def _render_binary_abspath(abspath: str):
     from rich.text import Text
 
-    if abspath.startswith("LIB_DIR/"):
-        return Text.assemble(("LIB_DIR", "bright_blue"), (abspath.removeprefix("LIB_DIR"), "green"))
-    if abspath == "LIB_DIR":
-        return Text("LIB_DIR", style="bright_blue")
+    if abspath.startswith("ABXPKG_LIB_DIR/"):
+        return Text.assemble(("ABXPKG_LIB_DIR", "bright_blue"), (abspath.removeprefix("ABXPKG_LIB_DIR"), "green"))
+    if abspath == "ABXPKG_LIB_DIR":
+        return Text("ABXPKG_LIB_DIR", style="bright_blue")
     if abspath.startswith("PERSONAS_DIR/"):
         return Text.assemble(("PERSONAS_DIR", "medium_purple"), (abspath.removeprefix("PERSONAS_DIR"), "green"))
     if abspath == "PERSONAS_DIR":
@@ -78,8 +77,50 @@ def _render_binary_abspath(abspath: str):
     return Text(abspath, style="green")
 
 
+def _build_binary_table(rows: list[dict[str, object]]):
+    from rich import box
+    from rich.table import Table
+
+    table = Table(title="Binary Dependencies", box=box.SIMPLE_HEAVY, expand=True)
+    table.add_column("Plugin", no_wrap=True, max_width=24)
+    table.add_column("State", no_wrap=True, width=8)
+    table.add_column("Status", justify="center", no_wrap=True, width=6)
+    table.add_column("Binary", no_wrap=True, max_width=28)
+    table.add_column("Version", no_wrap=True, width=16)
+    table.add_column("Provider", no_wrap=True, width=8)
+    table.add_column("Path", overflow="fold", ratio=1)
+    for row in rows:
+        table.add_row(
+            str(row["plugin"]),
+            str(row["state"]),
+            str(row["status"]),
+            str(row["binary"]),
+            str(row["version"]),
+            str(row["provider"]),
+            row["path"],
+            style=str(row.get("style") or ""),
+        )
+    return table
+
+
+def _print_binary_row(prnt, row: dict[str, object]) -> None:
+    prnt(
+        "",
+        row["status"],
+        str(row["plugin"]).ljust(24),
+        str(row["state"]).ljust(8),
+        str(row["binary"]).ljust(28),
+        str(row["version"]).ljust(16),
+        str(row["provider"]).ljust(8),
+        row["path"],
+        overflow="ignore",
+        crop=False,
+        style=str(row.get("style") or ""),
+    )
+
+
 def _binary_record_matches_runtime(installed, lib_dir: Path) -> bool:
-    if not installed or not installed.is_valid:
+    if not installed or not installed.is_valid or not installed.version:
         return False
     try:
         abspath = Path(installed.abspath).expanduser().resolve(strict=False)
@@ -108,13 +149,14 @@ def version(
 
     from rich.panel import Panel
     from rich.console import Console
+    from rich.live import Live
 
     from archivebox.config import CONSTANTS
     from archivebox.config.version import get_COMMIT_HASH, get_BUILD_TIME
     from archivebox.config.permissions import ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, RUNNING_AS_UID, RUNNING_AS_GID, IN_DOCKER
     from archivebox.config.paths import get_data_locations, get_code_locations
     from archivebox.misc.logging_util import printable_folder_status
-    from archivebox.config.common import get_config
+    from archivebox.config.common import get_config, normalize_runtime_config
 
     console = Console()
     prnt = console.print
@@ -168,7 +210,8 @@ def version(
     )
     prnt()
 
-    if not (os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)):
+    in_data_dir = os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
+    if not in_data_dir:
         PANEL_TEXT = "\n".join(
             (
                 "",
@@ -189,99 +232,149 @@ def version(
             ),
         )
         prnt()
-        return []
 
     prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
     failures = []
 
-    # Setup Django before importing models
+    from archivebox.plugins.discovery import get_enabled_plugins
+    from abx_dl.config import get_required_binary_requests
+    from abx_dl.dependencies import load_binary
+    from abx_dl.models import discover_plugins, filter_plugins
+
+    if isinstance(binaries, str):
+        requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
+    else:
+        requested_names = {name for name in (binaries or ()) if name}
+
+    plugins = discover_plugins(runtime="archivebox")
+    enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
+    enabled_plugin_names = set(enabled_plugins)
+    runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
+    derived_config: dict[str, object] = {}
+    db_binaries = {}
+    db_available = False
+    if in_data_dir:
+        try:
+            from archivebox.config.django import setup_django
+
+            setup_django()
+
+            from archivebox.machine.models import Machine, Binary
+
+            machine = Machine.current()
+            derived_config = normalize_runtime_config(machine.config, json_safe=False)
+            for binary in Binary.objects.filter(machine=machine).order_by("name", "-modified_at"):
+                if _binary_record_matches_runtime(binary, config.ABXPKG_LIB_DIR):
+                    db_binaries.setdefault(binary.name, binary)
+            db_available = True
+
+        except Exception as e:
+            prnt()
+            prnt("", f"[yellow]Warning: Could not query collection binary records, falling back to abxpkg state: {e}[/yellow]")
+
+    rows: list[dict[str, object]] = []
+    any_rows = False
+    any_available = False
+    compact_paths = console.is_terminal
+    live_enabled = console.is_terminal
+    live_cm = Live(_build_binary_table(rows), console=console, refresh_per_second=8) if live_enabled else None
+    if not live_enabled:
+        prnt("", "Status", "Plugin".ljust(24), "State".ljust(8), "Binary".ljust(28), "Version".ljust(16), "Provider".ljust(8), "Path")
+
+    def emit_row(row: dict[str, object]) -> None:
+        rows.append(row)
+        if live_cm is not None:
+            live_cm.update(_build_binary_table(rows), refresh=True)
+        else:
+            _print_binary_row(prnt, row)
+
+    if live_cm is not None:
+        live_cm.start()
     try:
-        from archivebox.config.django import setup_django
-
-        setup_django()
-
-        from archivebox.machine.models import Machine, Binary
-
-        machine = Machine.current()
-
-        if isinstance(binaries, str):
-            requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
-        else:
-            requested_names = {name for name in (binaries or ()) if name}
-
-        db_binaries: dict[str, Binary] = {}
-        for binary in Binary.objects.filter(machine=machine).order_by("name", "-modified_at"):
-            if _binary_record_matches_runtime(binary, config.LIB_DIR):
-                db_binaries.setdefault(binary.name, binary)
-                continue
-            if binary.status == Binary.StatusChoices.INSTALLED and binary.abspath:
-                binary.status = Binary.StatusChoices.QUEUED
-                binary.retry_at = None
-                binary.abspath = ""
-                binary.save(update_fields=["status", "retry_at", "abspath", "modified_at"])
-
-        all_binary_names = sorted(requested_names or set(db_binaries.keys()))
-
-        if not all_binary_names:
-            prnt("", "[grey53]No binaries detected. Run [green]archivebox install[/green] to detect dependencies.[/grey53]")
-        else:
-            any_available = False
-            compact_paths = console.is_terminal
-            for name in all_binary_names:
-                if requested_names and name not in requested_names:
+        for plugin_name, plugin in plugins.items():
+            plugin_enabled = plugin_name in enabled_plugin_names
+            logical_records = get_required_binary_requests(
+                plugin,
+                plugin.config.required_binaries,
+                overrides=runtime_config,
+                derived_overrides=derived_config,
+                run_output_dir=CONSTANTS.DATA_DIR,
+            )
+            actual_records = get_required_binary_requests(
+                plugin,
+                plugin.config.required_binaries,
+                overrides=runtime_config,
+                derived_overrides=derived_config,
+                run_output_dir=CONSTANTS.DATA_DIR,
+                logical_names=False,
+            )
+            for logical_record, actual_record in zip(logical_records, actual_records, strict=False):
+                logical_name = str(logical_record["name"])
+                actual_name = str(actual_record["name"])
+                display_name = Path(actual_name).expanduser().name if ("/" in actual_name or actual_name.startswith("~")) else logical_name
+                if (
+                    requested_names
+                    and logical_name not in requested_names
+                    and actual_name not in requested_names
+                    and display_name not in requested_names
+                ):
                     continue
 
-                installed = db_binaries.get(name)
-                if _binary_record_matches_runtime(installed, config.LIB_DIR):
-                    display_name = Path(name).expanduser().name if ("/" in name or name.startswith("~")) else name
+                any_rows = True
+                installed = db_binaries.get(logical_name) if db_available else None
+                if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
+                    abspath = installed.abspath
+                    version_str = (installed.version or "unknown")[:15]
+                    provider = (installed.binprovider or "env")[:8]
+                    valid = True
+                else:
+                    loaded = load_binary(actual_record)
+                    abspath = str(loaded.loaded_abspath or "")
+                    version_str = str(loaded.loaded_version or "unknown")[:15]
+                    provider = (loaded.loaded_binprovider.name if loaded.loaded_binprovider else "env")[:8]
+                    valid = loaded.is_valid
+
+                if valid:
                     display_path = (
                         _format_binary_abspath(
-                            installed.abspath,
+                            abspath,
                             pwd=Path.cwd(),
-                            lib_dir=config.LIB_DIR,
+                            lib_dir=config.ABXPKG_LIB_DIR,
                             personas_dir=CONSTANTS.PERSONAS_DIR,
                             home=Path.home(),
                         )
                         if compact_paths
-                        else installed.abspath
+                        else abspath
                     )
                     rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
-                    version_str = (installed.version or "unknown")[:15]
-                    provider = (installed.binprovider or "env")[:8]
-                    prnt(
-                        "",
-                        "[green]√[/green]",
-                        "",
-                        display_name.ljust(18),
-                        version_str.ljust(16),
-                        provider.ljust(8),
-                        rendered_path,
-                        overflow="ignore",
-                        crop=False,
-                    )
+                    status = "[green]√[/green]" if plugin_enabled else "[grey53]-[/grey53]"
                     any_available = True
-                    continue
+                else:
+                    rendered_path = "[grey53]not installed[/grey53]"
+                    status = "[red]X[/red]" if plugin_enabled else "[grey53]-[/grey53]"
+                    if plugin_enabled:
+                        failures.append(display_name)
 
-                status = (
-                    "[grey53]not recorded[/grey53]" if name in requested_names and installed is None else "[grey53]not installed[/grey53]"
+                emit_row(
+                    {
+                        "plugin": plugin_name,
+                        "state": "enabled" if plugin_enabled else "disabled",
+                        "status": status,
+                        "binary": display_name,
+                        "version": version_str if valid else "-",
+                        "provider": provider if valid else "-",
+                        "path": rendered_path,
+                        "style": "" if plugin_enabled else "dim",
+                    },
                 )
-                prnt("", "[red]X[/red]", "", name.ljust(18), status, overflow="ignore", crop=False)
-                failures.append(name)
+    finally:
+        if live_cm is not None:
+            live_cm.stop()
 
-            if not any_available:
-                prnt("", "[grey53]No binaries detected. Run [green]archivebox install[/green] to detect dependencies.[/grey53]")
-
-        # Show hint if no binaries are installed yet
-        has_any_installed = Binary.objects.filter(machine=machine).exclude(abspath="").exists()
-        if not has_any_installed:
-            prnt()
-            prnt("", "[grey53]Run [green]archivebox install[/green] to detect and install dependencies.[/grey53]")
-
-    except Exception as e:
-        # Handle database errors gracefully (locked, missing, etc.)
-        prnt()
-        prnt("", f"[yellow]Warning: Could not query binaries from database: {e}[/yellow]")
-        prnt("", "[grey53]Run [green]archivebox init[/green] and [green]archivebox install[/green] to set up dependencies.[/grey53]")
+    if not any_rows:
+        prnt("", "[grey53]No required binaries declared for discovered plugins.[/grey53]")
+    elif not any_available:
+        prnt("", "[grey53]No binaries detected. Run [green]archivebox install[/green] to detect dependencies.[/grey53]")
 
     if not binaries:
         # Show code and data locations

archivebox/config/common.py

@@ -250,10 +250,10 @@ class StorageConfig(BaseConfigSet):
     # must be a local SSD/tmpfs for speed and because bind mounts/network mounts/FUSE dont support unix sockets
     TMP_DIR: Path = Field(default=CONSTANTS.DEFAULT_TMP_DIR, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
-    # LIB_DIR must be a local, fast, readable/writable dir by archivebox user,
+    # ABXPKG_LIB_DIR must be a local, fast, readable/writable dir by archivebox user,
     # must be able to contain executable binaries (up to 5GB size)
     # should not be a remote/network/FUSE mount for speed reasons, otherwise extractors will be slow
-    LIB_DIR: Path = Field(default=CONSTANTS.DEFAULT_LIB_DIR, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    ABXPKG_LIB_DIR: Path = Field(default=CONSTANTS.DEFAULT_ABXPKG_LIB_DIR, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
     OUTPUT_PERMISSIONS: str = Field(default="644")
     ENFORCE_ATOMIC_WRITES: bool = Field(default=True)
@@ -536,6 +536,10 @@ def _archivebox_config_input_names() -> set[str]:
     for field in ArchiveBoxConfig.model_fields.values():
         if isinstance(field.alias, str):
             names.add(field.alias)
+    for prop in _plugin_config_properties(PLUGIN_CONFIG_SCHEMAS).values():
+        aliases = prop.get("x-aliases") if isinstance(prop, Mapping) else None
+        if isinstance(aliases, list):
+            names.update(str(alias) for alias in aliases if str(alias).strip())
     return names
 
 
@@ -647,11 +651,10 @@ class ArchiveBoxBaseConfig(
         runtime_derived_keys = {
             "ABX_INSTALL_CACHE",
             "ACTIVE_PERSONA",
-            "CHROME_DOWNLOADS_DIR",
-            "CHROME_USER_DATA_DIR",
             "CRAWL_DIR",
             "DEFAULT_PERSONA",
             "EXTRA_CONTEXT",
+            "PERSONAS_DIR",
             "SNAP_DIR",
         }
         return frozenset(
@@ -697,9 +700,9 @@ class ArchiveBoxBaseConfig(
         config["DATA_DIR"] = str(CONSTANTS.DATA_DIR)
         scope_by_key = type(self)._scope_by_key()
         model_fields = type(self).model_fields
-        # ArchiveBox owns SEARCH_BACKEND_ENGINE and uses it during model
-        # validation to derive the selected backend's *_ENABLED flag. Hooks
-        # only receive the backend-local flags, never the selector itself.
+        # ArchiveBox-only selectors are consumed during model validation to
+        # derive hook-owned *_ENABLED flags. They are stripped below after
+        # crawl/snapshot overlays have had a chance to apply.
         config.pop("SEARCH_BACKEND_ENGINE", None)
         if persona is not None:
             for key, value in persona.get_derived_config().items():
@@ -721,6 +724,11 @@ class ArchiveBoxBaseConfig(
         if runtime_overrides:
             config.update(normalize_runtime_config(runtime_overrides, json_safe=False))
 
+        # Hooks should only see concrete plugin-local flags, never the
+        # ArchiveBox selectors used to derive them.
+        config.pop("PLUGINS", None)
+        config.pop("SEARCH_BACKEND_ENGINE", None)
+
         if extra_context:
             context: dict[str, Any] = {}
             if config.get("EXTRA_CONTEXT"):
@@ -735,10 +743,10 @@ class ArchiveBoxBaseConfig(
 
     @model_validator(mode="after")
     def resolve_runtime_paths(self):
-        lib_dir = self.LIB_DIR.expanduser()
+        lib_dir = self.ABXPKG_LIB_DIR.expanduser()
         if not lib_dir.is_absolute():
             lib_dir = CONSTANTS.DATA_DIR / lib_dir
-        self.LIB_DIR = lib_dir.resolve()
+        self.ABXPKG_LIB_DIR = lib_dir.resolve()
 
         return self
 
@@ -997,6 +1005,7 @@ def get_config(
         persona = crawl.resolve_persona()
 
     config_data: ConfigPayload = dict(defaults or {})
+    config_data["PERSONAS_DIR"] = str(CONSTANTS.PERSONAS_DIR)
     base_config_payload: ConfigPayload = {}
     if crawl_config_base:
         config_data.update(
@@ -1024,7 +1033,7 @@ def get_config(
 
         scope_overrides.update(
             normalize_runtime_config(
-                _sanitize_machine_config(machine.config, lib_dir=config_data.get("LIB_DIR")),
+                _sanitize_machine_config(machine.config, lib_dir=config_data.get("ABXPKG_LIB_DIR")),
                 only_crawl_execution=crawl_config_base,
                 exclude_runtime_derived=True,
                 json_safe=False,
@@ -1133,8 +1142,7 @@ def get_config(
             value = config[key]
             if is_sensitive_config_key(key) and value not in (None, ""):
                 setattr(config, key, SENSITIVE_CONFIG_VALUE_REDACTED)
-    os.environ["LIB_DIR"] = str(config.LIB_DIR)
-    os.environ["ABXPKG_LIB_DIR"] = str(config.LIB_DIR)
+    os.environ["ABXPKG_LIB_DIR"] = str(config.ABXPKG_LIB_DIR)
     archiving_warning_key = (config.TIMEOUT, config.USE_COLOR)
     if archiving_warning_key not in _WARNED_ARCHIVING_CONFIGS:
         config.warn_if_invalid()

archivebox/config/constants.py

@@ -113,10 +113,9 @@ class ConstantsDict:
     TMP_DIR_NAME: str = "tmp"
     DEFAULT_TMP_DIR: Path = DATA_DIR / TMP_DIR_NAME / MACHINE_ID  # ./data/tmp/abc3244323
 
-    LIB_DIR_NAME: str = "lib"
-    DEFAULT_LIB_DIR: Path = _env_path(
-        "LIB_DIR",
-        user_config_path("abx") / LIB_DIR_NAME,
+    DEFAULT_ABXPKG_LIB_DIR: Path = _env_path(
+        "ABXPKG_LIB_DIR",
+        user_config_path("abx") / "lib",
     )
 
     RESERVED_ARCHIVE_DIR_NAMES: frozenset[str] = frozenset(
@@ -240,7 +239,7 @@ class ConstantsDict:
             SOURCES_DIR_NAME,
             LOGS_DIR_NAME,
             CACHE_DIR_NAME,
-            LIB_DIR_NAME,
+            "lib",
             TMP_DIR_NAME,
             PERSONAS_DIR_NAME,
             CUSTOM_TEMPLATES_DIR_NAME,

archivebox/config/paths.py

@@ -254,8 +254,8 @@ def get_or_create_working_lib_dir(autofix=True, quiet=False, config: "ArchiveBox
 
     config = config or get_config(**config_kwargs)
 
-    # LIB_DIR is either the shared platformdirs default or an explicit env/config override.
-    CANDIDATES = [config.LIB_DIR]
+    # ABXPKG_LIB_DIR is either the shared platformdirs default or an explicit env/config override.
+    CANDIDATES = [config.ABXPKG_LIB_DIR]
 
     for candidate in CANDIDATES:
         try:
@@ -263,12 +263,12 @@ def get_or_create_working_lib_dir(autofix=True, quiet=False, config: "ArchiveBox
         except Exception:
             pass
         if check_lib_dir(candidate, throw=False, quiet=True, must_exist=True):
-            if autofix and config.LIB_DIR != candidate:
-                os.environ["LIB_DIR"] = str(candidate)
+            if autofix and config.ABXPKG_LIB_DIR != candidate:
+                os.environ["ABXPKG_LIB_DIR"] = str(candidate)
             return candidate
 
     if not quiet:
-        raise OSError(f"ArchiveBox is unable to find a writable LIB_DIR, tried {CANDIDATES}!")
+        raise OSError(f"ArchiveBox is unable to find a writable ABXPKG_LIB_DIR, tried {CANDIDATES}!")
 
 
 def get_data_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs):
@@ -364,9 +364,9 @@ def get_code_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs
 
     config = config or get_config(**config_kwargs)
     try:
-        lib_dir = get_or_create_working_lib_dir(autofix=True, quiet=True, config=config) or config.LIB_DIR
+        lib_dir = get_or_create_working_lib_dir(autofix=True, quiet=True, config=config) or config.ABXPKG_LIB_DIR
     except Exception:
-        lib_dir = config.LIB_DIR
+        lib_dir = config.ABXPKG_LIB_DIR
 
     return AttrDict(
         {
@@ -390,7 +390,7 @@ def get_code_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs
                 "enabled": os.path.isdir(CONSTANTS.USER_PLUGINS_DIR),
                 "is_valid": os.path.isdir(CONSTANTS.USER_PLUGINS_DIR) and os.access(CONSTANTS.USER_PLUGINS_DIR, os.R_OK),  # read
             },
-            "LIB_DIR": {
+            "ABXPKG_LIB_DIR": {
                 "path": lib_dir.resolve(),
                 "enabled": True,
                 "is_valid": os.path.isdir(lib_dir) and os.access(lib_dir, os.R_OK) and os.access(lib_dir, os.W_OK),  # read + write

archivebox/config/version.py

@@ -49,10 +49,10 @@ def get_COMMIT_HASH() -> str | None:
 
     if IN_DOCKER:
         try:
-            version_txt = Path("/VERSION.txt").read_text()
-            docker_commit_hashes = re.findall(r"COMMIT_HASH=([0-9a-fA-F]{40})", version_txt)
-            if docker_commit_hashes:
-                return docker_commit_hashes[-1]
+            version_text = Path("/VERSION.txt").read_text()
+            match = re.search(r"^COMMIT_HASH=([0-9a-fA-F]{40})$", version_text, re.MULTILINE)
+            if match:
+                return match.group(1)
         except Exception:
             pass
 
@@ -106,13 +106,10 @@ def get_COMMIT_HASH() -> str | None:
 
 @cache
 def get_BUILD_TIME() -> str:
-    if IN_DOCKER:
-        try:
-            # if we're in the archivebox official docker image, /VERSION.txt will contain the build time
-            docker_build_end_time = Path("/VERSION.txt").read_text().rsplit("BUILD_END_TIME=")[-1].split("\n", 1)[0]
-            return docker_build_end_time
-        except Exception:
-            pass
+    for env_var in ("ARCHIVEBOX_BUILD_TIME", "BUILD_TIME"):
+        build_time = os.environ.get(env_var, "").strip()
+        if build_time:
+            return build_time
 
     src_last_modified_unix_timestamp = (PACKAGE_DIR / "README.md").stat().st_mtime
     return datetime.fromtimestamp(src_last_modified_unix_timestamp).strftime("%Y-%m-%d %H:%M:%S %s")

archivebox/core/admin_archiveresults.py

@@ -142,18 +142,22 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
         process = result.process_record
         process_display = "-"
         if process:
+            process_url = html.escape(reverse("admin:machine_process_change", args=[process.id]), quote=True)
+            process_label = html.escape(get_process_link_label(process), quote=True)
             process_display = f'''
-                <a href="{reverse("admin:machine_process_change", args=[process.id])}"
+                <a href="{process_url}"
                    style="color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 12px;"
-                   title="View process">{get_process_link_label(process)}</a>
+                   title="View process">{process_label}</a>
             '''
 
         machine_display = "-"
         if process and process.machine_id:
+            machine_url = html.escape(reverse("admin:machine_machine_change", args=[process.machine_id]), quote=True)
+            machine_label = html.escape(str(process.machine.hostname or ""), quote=True)
             machine_display = f'''
-                <a href="{reverse("admin:machine_machine_change", args=[process.machine_id])}"
+                <a href="{machine_url}"
                    style="color: #2563eb; text-decoration: none; font-size: 12px;"
-                   title="View machine">{process.machine.hostname}</a>
+                   title="View machine">{machine_label}</a>
             '''
 
         # Truncate output for display
@@ -176,9 +180,13 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
             output_link = build_snapshot_url(snapshot_id, embed_path, config=config)
         else:
             output_link = build_snapshot_url(snapshot_id, "", config=config)
+        output_link_attr = html.escape(output_link, quote=True)
 
         # Get version - try cmd_version field
-        version = result.cmd_version if result.cmd_version else "-"
+        version = html.escape(str(result.cmd_version if result.cmd_version else "-"), quote=True)
+        plugin_text = html.escape(str(result.plugin or ""), quote=True)
+        status_text = html.escape(str(status), quote=True)
+        pwd_text = html.escape(str(result.pwd or "-"), quote=True)
 
         # Unique ID for this row's expandable output
         row_id = f"output_{idx}_{str(result.id)[:8]}"
@@ -195,18 +203,18 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                 <td style="padding: 10px 12px; white-space: nowrap;">
                     <span style="display: inline-block; padding: 3px 10px; border-radius: 12px;
                                  font-size: 11px; font-weight: 600; text-transform: uppercase;
-                                 color: {color}; background: {bg};">{status}</span>
+                                 color: {color}; background: {bg};">{status_text}</span>
                 </td>
-                <td style="padding: 10px 12px; white-space: nowrap; font-size: 20px;" title="{result.plugin}">
+                <td style="padding: 10px 12px; white-space: nowrap; font-size: 20px;" title="{plugin_text}">
                     {icon}
                 </td>
                 <td style="padding: 10px 12px; font-weight: 500; color: #334155;">
-                        <a href="{output_link}" target="_blank"
+                        <a href="{output_link_attr}" target="_blank"
                            style="color: #334155; text-decoration: none;"
                        title="View output fullscreen"
                        onmouseover="this.style.color='#2563eb'; this.style.textDecoration='underline';"
                        onmouseout="this.style.color='#334155'; this.style.textDecoration='none';">
-                        {result.plugin}
+                        {plugin_text}
                     </a>
                 </td>
                 <td style="padding: 10px 12px; max-width: 280px;">
@@ -233,7 +241,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                 </td>
                 <td style="padding: 10px 8px; white-space: nowrap;">
                     <div style="display: flex; gap: 4px;">
-                        <a href="{output_link}" target="_blank"
+                        <a href="{output_link_attr}" target="_blank"
                            style="padding: 4px 8px; background: #f1f5f9; border-radius: 4px; color: #475569; text-decoration: none; font-size: 11px;"
                            title="View output">📄</a>
                         <a href="{reverse("admin:core_archiveresult_change", args=[result.id])}"
@@ -252,7 +260,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                             <div style="font-size: 11px; color: #64748b; margin-bottom: 8px;">
                                 <span style="margin-right: 16px;"><b>ID:</b> <code>{str(result.id)}</code></span>
                                 <span style="margin-right: 16px;"><b>Version:</b> <code>{version}</code></span>
-                                <span style="margin-right: 16px;"><b>PWD:</b> <code>{result.pwd or "-"}</code></span>
+                                <span style="margin-right: 16px;"><b>PWD:</b> <code>{pwd_text}</code></span>
                             </div>
                             <div style="font-size: 11px; color: #64748b; margin-bottom: 8px;">
                                 <b>Output:</b>

archivebox/core/admin_site.py

@@ -3,7 +3,15 @@ __package__ = "archivebox.core"
 from typing import TYPE_CHECKING, Any
 
 from django.contrib import admin
+from django.contrib.auth import REDIRECT_FIELD_NAME
+from django.contrib.auth.decorators import login_not_required
+from django.contrib.auth.views import LoginView
 from django.db import DatabaseError, connection
+from django.http import HttpResponseRedirect
+from django.urls import reverse
+from django.utils.decorators import method_decorator
+from django.utils.translation import gettext as _
+from django.views.decorators.cache import never_cache
 from admin_data_views.admin import (
     admin_data_index_view as adv_admin_data_index_view,
     get_admin_data_urls as adv_get_admin_data_urls,
@@ -12,6 +20,7 @@ from admin_data_views.admin import (
 
 from archivebox.config import VERSION
 from archivebox.config.version import get_COMMIT_HASH
+from archivebox.core.routes_util import is_allowed_archivebox_redirect_url
 
 if TYPE_CHECKING:
     from django.http import HttpRequest
@@ -21,6 +30,17 @@ if TYPE_CHECKING:
     from admin_data_views.typing import AppDict
 
 
+class ArchiveBoxLoginView(LoginView):
+    def get_redirect_url(self) -> str:
+        redirect_to = self.request.POST.get(
+            self.redirect_field_name,
+            self.request.GET.get(self.redirect_field_name),
+        )
+        if is_allowed_archivebox_redirect_url(redirect_to, request=self.request):
+            return redirect_to
+        return ""
+
+
 class ArchiveBoxAdmin(admin.AdminSite):
     site_header = "ArchiveBox"
     index_title = "Admin Views"
@@ -72,6 +92,34 @@ class ArchiveBoxAdmin(admin.AdminSite):
     def admin_data_index_view(self, request: "HttpRequest", **kwargs: Any) -> "TemplateResponse":
         return adv_admin_data_index_view(self, request, **kwargs)
 
+    @method_decorator(never_cache)
+    @login_not_required
+    def login(self, request: "HttpRequest", extra_context: dict[str, Any] | None = None) -> "TemplateResponse":
+        if request.method == "GET" and self.has_permission(request):
+            return HttpResponseRedirect(reverse("admin:index", current_app=self.name))
+
+        from django.contrib.admin.forms import AdminAuthenticationForm
+
+        context = {
+            **self.each_context(request),
+            "title": _("Log in"),
+            "subtitle": None,
+            "app_path": request.get_full_path(),
+            "username": request.user.get_username(),
+        }
+        if REDIRECT_FIELD_NAME not in request.GET and REDIRECT_FIELD_NAME not in request.POST:
+            context[REDIRECT_FIELD_NAME] = reverse("admin:index", current_app=self.name)
+        context.update(extra_context or {})
+
+        index_path = reverse("admin:index", current_app=self.name)
+        request.current_app = self.name
+        return ArchiveBoxLoginView.as_view(
+            extra_context=context,
+            authentication_form=self.login_form or AdminAuthenticationForm,
+            template_name=self.login_template or "admin/login.html",
+            next_page=index_path,
+        )(request)
+
     def index(self, request: "HttpRequest", extra_context: dict[str, Any] | None = None) -> "TemplateResponse":
         response = super().index(request, extra_context)
         if connection.vendor != "sqlite":

archivebox/core/admin_snapshots.py

@@ -940,38 +940,54 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         config = request.archivebox_config
         results = self._get_prefetched_results(obj)
         if results is not None:
-            has_screenshot = any(r.plugin == "screenshot" for r in results)
-            has_favicon = any(r.plugin == "favicon" for r in results)
-            has_extension_screenshot = any(r.plugin == EXTENSION_SCREENSHOT_PLUGIN for r in results)
+            results = [r for r in results if r.plugin in ("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")]
         else:
-            available_plugins = set(
-                obj.archiveresult_set.filter(plugin__in=("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")).values_list(
+            results = list(
+                obj.archiveresult_set.filter(plugin__in=("screenshot", EXTENSION_SCREENSHOT_PLUGIN, "favicon")).only(
+                    "snapshot_id",
                     "plugin",
-                    flat=True,
+                    "status",
+                    "output_files",
+                    "output_str",
                 ),
             )
-            has_screenshot = "screenshot" in available_plugins
-            has_favicon = "favicon" in available_plugins
-            has_extension_screenshot = EXTENSION_SCREENSHOT_PLUGIN in available_plugins
 
-        extension_screenshot_urls = [
-            build_snapshot_url(str(obj.id), f"{EXTENSION_SCREENSHOT_PLUGIN}/screenshot-1.png", request=request, config=config),
-            build_snapshot_url(str(obj.id), f"{EXTENSION_SCREENSHOT_PLUGIN}/screenshot.png", request=request, config=config),
-        ]
+        def result_output_path(result, filename: str) -> str | None:
+            output_files = result.output_files or {}
+            file_info = output_files.get(filename)
+            if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
+                return None
+            if file_info.get("root_relative"):
+                return filename
+            return f"{result.plugin}/{filename}"
 
-        if not has_screenshot and not has_extension_screenshot and not has_favicon:
+        def result_urls(plugin: str, filenames: tuple[str, ...]) -> list[str]:
+            urls: list[str] = []
+            for result in results:
+                if result.plugin != plugin or result.status != ArchiveResult.StatusChoices.SUCCEEDED:
+                    continue
+                for filename in filenames:
+                    output_path = result_output_path(result, filename)
+                    if output_path:
+                        urls.append(build_snapshot_url(str(obj.id), output_path, request=request, config=config))
+            return urls
+
+        screenshot_urls = result_urls("screenshot", ("screenshot.png",))
+        extension_screenshot_urls = result_urls(EXTENSION_SCREENSHOT_PLUGIN, ("screenshot-1.png", "screenshot.png"))
+        favicon_urls = result_urls("favicon", ("favicon.ico",))
+
+        if not screenshot_urls and not extension_screenshot_urls and not favicon_urls:
             return None
 
-        if has_screenshot or has_extension_screenshot:
-            img_url = build_snapshot_url(str(obj.id), "screenshot/screenshot.png", request=request, config=config)
-            fallbacks = extension_screenshot_urls
+        all_screenshot_urls = [*screenshot_urls, *extension_screenshot_urls]
+        if all_screenshot_urls:
+            img_url = all_screenshot_urls[0]
+            fallbacks = [*all_screenshot_urls[1:], *favicon_urls]
             img_alt = "Screenshot"
             preview_class = "screenshot"
         else:
-            img_url = build_snapshot_url(str(obj.id), "favicon/favicon.ico", request=request, config=config)
-            fallbacks = [
-                build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config),
-            ]
+            img_url = favicon_urls[0]
+            fallbacks = favicon_urls[1:]
             img_alt = "Favicon"
             preview_class = "favicon"
 
@@ -989,6 +1005,8 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             "preview_class": preview_class,
             "onerror_js": onerror_js,
             "fallback_list": fallback_list,
+            "favicon_url": favicon_urls[0] if favicon_urls else "",
+            "favicon_fallback_list": ",".join(favicon_urls[1:]),
         }
 
     @admin.display(description="", empty_value="")
@@ -997,10 +1015,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         if not preview:
             return ""
 
-        request = self.request
-        config = request.archivebox_config
-        favicon_url = build_snapshot_url(str(obj.id), "favicon/favicon.ico", request=request, config=config)
-        fallback_list = ",".join([build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config)])
+        favicon_url = preview.get("favicon_url") or ""
+        if not favicon_url:
+            return ""
+        fallback_list = preview.get("favicon_fallback_list") or ""
         onerror_js = (
             "this.dataset.fallbacks && this.dataset.fallbacks.length ? "
             "(this.src=this.dataset.fallbacks.split(',').shift(), "

archivebox/core/forms.py

@@ -9,7 +9,7 @@ from archivebox.misc.util import URL_REGEX, find_all_urls, parse_filesize_to_byt
 from archivebox.base_models.admin import KeyValueWidget
 from archivebox.crawls.schedule_util import validate_schedule
 from archivebox.config.common import get_config, parse_delete_after
-from archivebox.core.permissions import PERMISSIONS_CHOICES, PERMISSIONS_PUBLIC, filter_personas_by_permissions, is_admin_user
+from archivebox.core.permissions import PERMISSIONS_CHOICES, PERMISSIONS_PUBLIC, filter_personas_by_permissions
 from archivebox.core.widgets import TagEditorWidget, URLFiltersWidget
 from archivebox.plugins.discovery import get_plugins
 from archivebox.plugins.forms import (
@@ -299,7 +299,10 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
 
     def __init__(self, *args, **kwargs):
         self.request = kwargs.pop("request", None)
-        self.can_override_crawl_config = bool(self.request and is_admin_user(self.request))
+        self.can_override_crawl_config = False
+        if self.request is not None:
+            user = self.request.user
+            self.can_override_crawl_config = bool(user.is_authenticated and user.is_active and user.is_superuser)
         super().__init__(*args, **kwargs)
 
         default_persona = Persona.get_or_create_default()

archivebox/core/models.py

@@ -36,7 +36,7 @@ from archivebox.misc.util import (
     to_json,
     ts_to_date_str,
     urlencode,
-    htmlencode,
+    htmldecode,
     sanitize_html_text,
     urldecode,
     validate_url,
@@ -3434,8 +3434,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         context = {
             **self.to_dict(extended=True),
             "snapshot": self,
-            "title": htmlencode(self.resolved_title or (self.base_url if is_archived else TITLE_LOADING_MSG)),
-            "url_str": htmlencode(urldecode(self.base_url)),
+            "title": htmldecode(self.resolved_title or (self.base_url if is_archived else TITLE_LOADING_MSG)),
+            "url_str": htmldecode(urldecode(self.base_url)),
             "archive_url": urlencode(f"warc/{self.timestamp}" or (self.domain if is_archived else "")) or "about:blank",
             "extension": self.extension or "html",
             "tags": self.tags_str() or "untagged",

archivebox/core/recovery_util.py

@@ -10,7 +10,7 @@ def _is_signal_interrupted_exit(exit_code: int | None) -> bool:
     return exit_code is not None and (exit_code < 0 or exit_code >= 128)
 
 
-def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int]:
+def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str | None = None) -> dict[str, int]:
     from archivebox.crawls.models import Crawl
     from archivebox.core.models import ArchiveResult, Snapshot
     from archivebox.services.archive_result_service import _collect_output_metadata
@@ -21,10 +21,13 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
 
     now = timezone.now()
     recovery_console = Console(stderr=True, highlight=False, soft_wrap=True)
+    crawl_filter = {"id": crawl_id} if crawl_id else {}
+    snapshot_filter = {"crawl_id": crawl_id} if crawl_id else {}
+    result_filter = {"snapshot__crawl_id": crawl_id} if crawl_id else {}
     cleaned = {
-        "processes_stale_running": Process.cleanup_stale_running(),
-        "processes_orphaned_workers": Process.cleanup_orphaned_workers(),
-        "chrome_processes_orphaned": Process.cleanup_orphaned_chrome() if include_chrome else 0,
+        "processes_stale_running": 0 if crawl_id else Process.cleanup_stale_running(),
+        "processes_orphaned_workers": 0 if crawl_id else Process.cleanup_orphaned_workers(),
+        "chrome_processes_orphaned": Process.cleanup_orphaned_chrome() if include_chrome and not crawl_id else 0,
         "crawls_queued_without_retry_at": 0,
         "snapshots_queued_without_retry_at": 0,
         "archiveresults_backoff": 0,
@@ -58,14 +61,16 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
     cleaned["crawls_queued_without_retry_at"] = Crawl.objects.filter(
         status=Crawl.StatusChoices.QUEUED,
         retry_at__isnull=True,
+        **crawl_filter,
     ).update(retry_at=now, modified_at=now)
     cleaned["snapshots_queued_without_retry_at"] = Snapshot.objects.filter(
         status=Snapshot.StatusChoices.QUEUED,
         retry_at__isnull=True,
         crawl__status__in=Crawl.RUNNABLE_STATES,
+        **snapshot_filter,
     ).update(retry_at=now, modified_at=now)
-    backoff_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.BACKOFF)
-    orphaned_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.STARTED).exclude(
+    backoff_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.BACKOFF, **result_filter)
+    orphaned_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.STARTED, **result_filter).exclude(
         process__status=Process.StatusChoices.RUNNING,
     )
     # ArchiveResult has no retry_at scheduler. Wake only the parent Snapshots
@@ -92,7 +97,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
     # only after this runner has won the single-runner gate, so it can safely
     # unlock those stale plugin leases for immediate processing instead of
     # waiting out the previous owner's full lock timeout.
-    queued_plugin_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.QUEUED)
+    queued_plugin_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.QUEUED, **result_filter)
     cleaned["snapshots_queued_plugin_rows_waiting_on_stale_lease"] = (
         Snapshot.objects.filter(
             id__in=queued_plugin_results.values("snapshot_id"),
@@ -113,16 +118,33 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
         process_type=Process.TypeChoices.HOOK,
         archiveresult__isnull=True,
     ).exclude(status=Process.StatusChoices.RUNNING)
+    crawl_snapshot_ids = (
+        {str(snapshot_id) for snapshot_id in Snapshot.objects.filter(crawl_id=crawl_id).values_list("id", flat=True)} if crawl_id else None
+    )
+    if crawl_snapshot_ids is not None:
+        # Targeted `archivebox run --crawl-id ...` is used by foreground add/update
+        # commands and by resume of one existing crawl. It must still repair bad
+        # hook state for that crawl, but it must not scan historical orphaned hook
+        # rows from unrelated crawls before the first snapshot can run.
+        if crawl_snapshot_ids:
+            snapshot_pwd_filter = Q()
+            for snapshot_id in crawl_snapshot_ids:
+                snapshot_pwd_filter |= Q(pwd__contains=snapshot_id)
+            orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
+        else:
+            orphaned_hook_processes = orphaned_hook_processes.none()
     for process in orphaned_hook_processes.only("id", "pwd", "cmd", "process_type", "status"):
         hook_script_name = process.hook_script_name
         if not hook_script_name or not process.pwd:
             continue
         plugin_dir = Path(process.pwd)
+        if crawl_snapshot_ids is not None and plugin_dir.parent.name not in crawl_snapshot_ids:
+            continue
         try:
             # Old or synthetic hook Process rows can point at arbitrary paths.
             # Only paths whose parent directory is a valid Snapshot id can be
             # reconstructed into ArchiveResult rows.
-            snapshot = Snapshot.objects.filter(id=plugin_dir.parent.name).first()
+            snapshot = Snapshot.objects.filter(id=plugin_dir.parent.name, **snapshot_filter).first()
         except ValidationError:
             continue
         if snapshot is None:
@@ -181,6 +203,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
             Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
     started_snapshots = Snapshot.objects.filter(status=Snapshot.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
+        **snapshot_filter,
     )
 
     # Broken lock repair: STARTED + retry_at=NULL or retry_at in the future
@@ -205,6 +228,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False) -> dict[str, int
     # are already final.
     recoverable_started_crawls = Crawl.objects.filter(status=Crawl.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
+        **crawl_filter,
     )
 
     due_started_crawls = recoverable_started_crawls.annotate(has_due_child=Exists(due_child_snapshots)).filter(has_due_child=True)

archivebox/core/routes_util.py

@@ -4,6 +4,8 @@ import re
 from typing import Any
 from urllib.parse import urlparse
 
+from django.utils.http import url_has_allowed_host_and_scheme
+
 from archivebox.config.common import get_config
 
 
@@ -284,6 +286,54 @@ def host_matches(request_host: str, target_host: str) -> bool:
     return True
 
 
+def is_allowed_archivebox_redirect_url(url: str | None, request=None, config: dict[str, Any] | None = None) -> bool:
+    """Allow login redirects to this host or to the explicit ArchiveBox BASE_URL family.
+
+    Django already rejects arbitrary absolute ``next=`` URLs for the current host.
+    ArchiveBox also has legitimate cross-subdomain handoffs (admin/web/api/snap-*),
+    but only when BASE_URL is pinned. Without an explicit BASE_URL we do not widen
+    trust based on a request Host header that may be user supplied.
+    """
+    if not url:
+        return False
+
+    require_https = bool(request and request.is_secure())
+    request_host = request.get_host() if request else ""
+    if url_has_allowed_host_and_scheme(url=url, allowed_hosts={request_host}, require_https=require_https):
+        return True
+
+    parsed = urlparse(url)
+    if not parsed.netloc:
+        return False
+    if parsed.scheme not in {"http", "https"}:
+        return False
+    if require_https and parsed.scheme != "https":
+        return False
+    if parsed.username or parsed.password:
+        return False
+
+    config = config or get_config()
+    base_url = _normalize_base_url(config.BASE_URL)
+    if not base_url:
+        return False
+
+    base_host, base_port = split_host_port(urlparse(base_url).netloc)
+    if not base_host:
+        return False
+
+    redirect_host = (parsed.hostname or "").lower()
+    if redirect_host != base_host and not redirect_host.endswith(f".{base_host}"):
+        return False
+
+    try:
+        redirect_port = str(parsed.port) if parsed.port else None
+    except ValueError:
+        return False
+    if base_port:
+        return redirect_port == base_port
+    return redirect_port is None
+
+
 def _scheme_from_request(request=None, config: dict[str, Any] | None = None) -> str:
     config = config or get_config()
     override = _normalize_base_url(config.BASE_URL)

archivebox/core/templatetags/core_tags.py

@@ -509,17 +509,59 @@ def snapshot_url(context, snapshot, path: str = "") -> str:
     return build_snapshot_url(str(snapshot_id), path, request=context.get("request"), config=context.get("CONFIG"))
 
 
+@register.simple_tag(takes_context=True)
+def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
+    snapshot_id = str(_snapshot_id(snapshot))
+    url_cache = snapshot.__dict__.setdefault("_snapshot_archiveresult_url_cache", {})
+    cache_key = (plugin, filename)
+    if cache_key in url_cache:
+        return url_cache[cache_key]
+
+    results = None
+    if "_admin_archiveresults" in snapshot.__dict__:
+        results = snapshot.__dict__["_admin_archiveresults"]
+    elif "archiveresult_set" in snapshot.__dict__.get("_prefetched_objects_cache", {}):
+        results = snapshot.archiveresult_set.all()
+    elif "_snapshot_archiveresult_url_results" in snapshot.__dict__:
+        results = snapshot.__dict__["_snapshot_archiveresult_url_results"]
+
+    if results is None:
+        from archivebox.core.models import ArchiveResult
+
+        results = list(
+            ArchiveResult.objects.filter(
+                snapshot_id=snapshot_id,
+                plugin__in=("screenshot", "chrome_extension_screenshot", "favicon"),
+                status=ArchiveResult.StatusChoices.SUCCEEDED,
+            ).only(
+                "plugin",
+                "status",
+                "output_files",
+            ),
+        )
+        snapshot.__dict__["_snapshot_archiveresult_url_results"] = results
+
+    for result in results:
+        if result.plugin != plugin or result.status != "succeeded":
+            continue
+        output_files = result.output_files or {}
+        file_info = output_files.get(filename)
+        if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
+            continue
+        output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
+        url_cache[cache_key] = build_snapshot_url(snapshot_id, output_path, request=context.get("request"), config=context.get("CONFIG"))
+        return url_cache[cache_key]
+
+    url_cache[cache_key] = ""
+    return ""
+
+
 @register.simple_tag(takes_context=True)
 def snapshot_index_row(context, link) -> str:
     snapshot_id = str(_snapshot_id(link))
     request = context.get("request")
     config = context.get("CONFIG")
     snapshot_base = get_snapshot_base_url(snapshot_id, request=request, config=config)
-    screenshot_plugin_url = f"{snapshot_base}/screenshot/screenshot.png"
-    extension_screenshot_1_url = f"{snapshot_base}/chrome_extension_screenshot/screenshot-1.png"
-    extension_screenshot_url = f"{snapshot_base}/chrome_extension_screenshot/screenshot.png"
-    favicon_plugin_url = f"{snapshot_base}/favicon/favicon.ico"
-    favicon_root_url = f"{snapshot_base}/favicon.ico"
 
     status = getattr(link, "status", None) or "unknown"
     bookmarked_at = getattr(link, "bookmarked_at", None)
@@ -560,11 +602,59 @@ def snapshot_index_row(context, link) -> str:
             f'<img src="{escape(static("spinner.gif"))}" alt="" decoding="async" loading="lazy">'
             "</span>"
         )
+    elif "_public_preview_paths" in link.__dict__:
+        preview_paths = list(getattr(link, "_public_preview_paths", []) or [])
+        if preview_paths:
+            preview_urls = [build_snapshot_url(snapshot_id, path, request=request, config=config) for path in preview_paths]
+            preview_html = (
+                f'<img src="{escape(preview_urls[0])}" '
+                f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
+                'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
+            )
+        else:
+            preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
     else:
-        preview_html = (
-            f'<img src="{escape(screenshot_plugin_url)}" '
-            f'data-fallbacks="{escape(extension_screenshot_1_url)},{escape(extension_screenshot_url)}" '
-            'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
+        preview_urls = [
+            url
+            for url in (
+                snapshot_archiveresult_url(context, link, "screenshot", "screenshot.png"),
+                snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot-1.png"),
+                snapshot_archiveresult_url(context, link, "chrome_extension_screenshot", "screenshot.png"),
+            )
+            if url
+        ]
+        if preview_urls:
+            preview_html = (
+                f'<img src="{escape(preview_urls[0])}" '
+                f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
+                'onerror="nextPublicSnapshotPreview(this)" class="snapshot-preview screenshot" alt="" decoding="async" loading="lazy">'
+            )
+        else:
+            preview_html = '<span class="snapshot-preview snapshot-preview-empty" aria-label="No preview available"></span>'
+
+    if "_public_favicon_paths" in link.__dict__:
+        favicon_paths = list(getattr(link, "_public_favicon_paths", []) or [])
+        if favicon_paths:
+            favicon_urls = [build_snapshot_url(snapshot_id, path, request=request, config=config) for path in favicon_paths]
+            favicon_html = (
+                f'<img src="{escape(favicon_urls[0])}" '
+                f'data-fallbacks="{escape(",".join(favicon_urls[1:]))}" '
+                'onerror="nextPublicSnapshotPreview(this)" '
+                'class="link-favicon" alt="" decoding="async" loading="lazy">'
+            )
+        else:
+            favicon_html = '<span class="link-favicon link-favicon-empty" aria-hidden="true"></span>'
+    else:
+        favicon_url = snapshot_archiveresult_url(context, link, "favicon", "favicon.ico")
+        favicon_html = (
+            (
+                f'<img src="{escape(favicon_url)}" '
+                'data-fallbacks="" '
+                'onerror="nextPublicSnapshotPreview(this)" '
+                'class="link-favicon" alt="" decoding="async" loading="lazy">'
+            )
+            if favicon_url
+            else '<span class="link-favicon link-favicon-empty" aria-hidden="true"></span>'
         )
 
     html = f"""
@@ -583,13 +673,7 @@ def snapshot_index_row(context, link) -> str:
     <td class="snapshot-title-cell" title="{escape(title or url)}">
         <div class="snapshot-title-line">
             <a href="{escape(snapshot_base)}/index.html" class="snapshot-favicon-link" title="Open archived snapshot">
-                <img src="{escape(favicon_plugin_url)}"
-                     data-fallbacks="{escape(favicon_root_url)}"
-                     onerror="nextPublicSnapshotPreview(this)"
-                     class="link-favicon"
-                     alt=""
-                     decoding="async"
-                     loading="lazy">
+                {favicon_html}
             </a>
             <a href="{escape(snapshot_base)}/index.html" class="snapshot-title">
                 {escape(Truncator(title_text).chars(110))}

archivebox/core/views.py

@@ -51,7 +51,7 @@ from archivebox.misc.paginators import AcceleratedPaginator
 from archivebox.misc.util import (
     base_url,
     filter_queryset_by_uuid_substring,
-    htmlencode,
+    htmldecode,
     sanitize_html_text,
     ts_to_date_str,
     urldecode,
@@ -441,7 +441,7 @@ class SnapshotView(View):
             "progress_endpoint": progress_endpoint("snapshot", snapshot.id),
             "url": snapshot.url,
             "archive_path": snapshot.archive_path_from_db,
-            "title": htmlencode(snapshot.resolved_title or (snapshot.base_url if is_archived else TITLE_LOADING_MSG)),
+            "title": htmldecode(snapshot.resolved_title or (snapshot.base_url if is_archived else TITLE_LOADING_MSG)),
             "extension": snapshot.extension or "html",
             "tags": snapshot.tags_str() or "untagged",
             "size": printable_filesize(output_size) if output_size else "—",
@@ -820,6 +820,34 @@ def _safe_archive_relpath(path: str) -> str | None:
     return cleaned
 
 
+def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[str, bool]:
+    """Resolve plugin-relative output paths through ArchiveResult.output_files."""
+    parts = Path(rel_path).parts
+    if len(parts) < 2:
+        return rel_path, False
+
+    plugin = parts[0]
+    plugin_relpath = posixpath.join(*parts[1:])
+    result = (
+        ArchiveResult.objects.filter(snapshot=snapshot, plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED)
+        .only("output_files")
+        .first()
+    )
+    if not result or not result.output_files:
+        return rel_path, False
+
+    output_files = result.output_files or {}
+    for candidate in (plugin_relpath, rel_path):
+        file_info = output_files.get(candidate)
+        if not isinstance(file_info, dict):
+            continue
+        if file_info.get("root_relative"):
+            return candidate, True
+        return rel_path, True
+
+    return rel_path, True
+
+
 def _coerce_sort_timestamp(value: str | float | None) -> float:
     if value is None:
         return 0.0
@@ -1013,6 +1041,8 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
     if rel_path is None:
         raise Http404
 
+    rel_path, _is_known_plugin_output = _resolve_archiveresult_relpath(snapshot, rel_path)
+
     try:
         return serve_static_with_byterange_support(
             request,
@@ -1226,6 +1256,8 @@ class PublicIndexView(ListView):
         snapshots = list(context.get("object_list") or ())
         icons_by_snapshot: dict[str, set[str]] = {str(snapshot.id): set() for snapshot in snapshots}
         tag_names_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
+        preview_paths_by_snapshot: dict[str, list[tuple[int, int, str]]] = {str(snapshot.id): [] for snapshot in snapshots}
+        favicon_paths_by_snapshot: dict[str, list[str]] = {str(snapshot.id): [] for snapshot in snapshots}
         progress_by_snapshot: dict[str, dict[str, int]] = {
             str(snapshot.id): {
                 "total": 0,
@@ -1246,20 +1278,51 @@ class PublicIndexView(ListView):
             ):
                 tag_names_by_snapshot[str(snapshot_id)].append(tag_name)
 
-            for snapshot_id, plugin, status in (
+            preview_plugin_order = {
+                "screenshot": 0,
+                "chrome_extension_screenshot": 1,
+            }
+            preview_candidates = {
+                "screenshot": ("screenshot.png",),
+                "chrome_extension_screenshot": ("screenshot-1.png", "screenshot.png"),
+            }
+
+            def result_output_path(result: ArchiveResult, filename: str) -> str | None:
+                output_files = result.output_files or {}
+                file_info = output_files.get(filename)
+                if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
+                    return None
+                if file_info.get("root_relative"):
+                    return filename
+                return f"{result.plugin}/{filename}"
+
+            for result in (
                 ArchiveResult.objects.filter(
                     snapshot_id__in=icons_by_snapshot.keys(),
                 )
                 .exclude(plugin="")
-                .values_list("snapshot_id", "plugin", "status")
+                .only("snapshot_id", "plugin", "status", "output_files", "output_str")
                 .iterator(chunk_size=1000)
             ):
+                snapshot_id = result.snapshot_id
+                plugin = result.plugin
+                status = result.status
                 snapshot_key = str(snapshot_id)
                 progress = progress_by_snapshot[snapshot_key]
                 progress["total"] += 1
                 if status == ArchiveResult.StatusChoices.SUCCEEDED:
                     icons_by_snapshot[snapshot_key].add(plugin)
                     progress["succeeded"] += 1
+                    if plugin in preview_candidates:
+                        plugin_rank = preview_plugin_order[plugin]
+                        for filename_rank, filename in enumerate(preview_candidates[plugin]):
+                            output_path = result_output_path(result, filename)
+                            if output_path:
+                                preview_paths_by_snapshot[snapshot_key].append((plugin_rank, filename_rank, output_path))
+                    elif plugin == "favicon":
+                        output_path = result_output_path(result, "favicon.ico")
+                        if output_path:
+                            favicon_paths_by_snapshot[snapshot_key].append(output_path)
                 elif status == ArchiveResult.StatusChoices.FAILED:
                     progress["failed"] += 1
                 elif status == ArchiveResult.StatusChoices.STARTED:
@@ -1275,6 +1338,10 @@ class PublicIndexView(ListView):
             snapshot._icons_progress_stats = progress_by_snapshot.get(str(snapshot.id), {})
             snapshot.num_outputs_cached = snapshot._icons_progress_stats.get("succeeded", 0)
             snapshot._tags_str_cached = ",".join(tag_names_by_snapshot.get(str(snapshot.id), []))
+            snapshot._public_preview_paths = [
+                output_path for _plugin_rank, _filename_rank, output_path in sorted(preview_paths_by_snapshot.get(str(snapshot.id), []))
+            ]
+            snapshot._public_favicon_paths = favicon_paths_by_snapshot.get(str(snapshot.id), [])
             snapshot._is_archived_cached = bool(snapshot.downloaded_at or snapshot.status == Snapshot.StatusChoices.SEALED)
         context["object_list"] = snapshots
         return context
@@ -1338,7 +1405,8 @@ class AddView(UserPassesTestMixin, FormView):
         return get_request_config(self.request).PUBLIC_ADD_VIEW or self.request.user.is_authenticated
 
     def _can_override_crawl_config(self) -> bool:
-        return is_admin_user(self.request)
+        user = self.request.user
+        return bool(user.is_authenticated and user.is_active and user.is_superuser)
 
     def _get_custom_config_overrides(self, form: AddLinkForm) -> dict:
         custom_config = form.cleaned_data.get("config") or {}

archivebox/machine/models.py

@@ -140,7 +140,7 @@ def _sanitize_machine_config(config: dict[str, Any] | None, *, lib_dir: str | Pa
     """Validate ``Machine.config`` in place.
 
     Drops stale ``*_BINARY`` overrides whose path no longer exists or whose
-    path falls outside of ``LIB_DIR`` (so a binary uninstall or a lib_dir
+    path falls outside of ``ABXPKG_LIB_DIR`` (so a binary uninstall or a lib_dir
     move clears the override automatically). Non-``_BINARY`` keys
     (``BASE_URL``, ``SERVER_SECURITY_MODE``, plugin tunables, etc.) are
     pass-through — they're arbitrary config overrides and not ours to filter.
@@ -287,7 +287,7 @@ class Machine(ModelWithHealthStats):
     def _sanitize_config(cls, machine: Machine) -> Machine:
         from archivebox.config.common import get_config
 
-        sanitized = _sanitize_machine_config(machine.config, lib_dir=get_config(include_machine=False).LIB_DIR)
+        sanitized = _sanitize_machine_config(machine.config, lib_dir=get_config(include_machine=False).ABXPKG_LIB_DIR)
         current = machine.config or {}
         if sanitized != current:
             machine.config = sanitized
@@ -681,7 +681,7 @@ class Binary(ModelWithHealthStats, ModelWithStateMachine):
             )
             from archivebox.config.common import get_config
 
-            binary.symlink_to_lib_bin_after_commit(get_config().LIB_DIR / "bin")
+            binary.symlink_to_lib_bin_after_commit(get_config().ABXPKG_LIB_DIR / "bin")
             return binary
 
         # Case 2: From binaries.json - create queued binary (needs installation)
@@ -714,7 +714,7 @@ class Binary(ModelWithHealthStats, ModelWithStateMachine):
             )
             from archivebox.config.common import get_config
 
-            binary.symlink_to_lib_bin_after_commit(get_config().LIB_DIR / "bin")
+            binary.symlink_to_lib_bin_after_commit(get_config().ABXPKG_LIB_DIR / "bin")
             return binary
 
         return None
@@ -754,7 +754,7 @@ class Binary(ModelWithHealthStats, ModelWithStateMachine):
 
         After a binary is installed by any binprovider (pip, npm, brew, apt, etc),
         we can optionally expose a flat convenience directory for shell users.
-        ArchiveBox/abx-dl runtime lookup must use the provider-specific LIB_DIR
+        ArchiveBox/abx-dl runtime lookup must use the provider-specific ABXPKG_LIB_DIR
         paths, not this indirection.
 
         Args:

archivebox/misc/checks.py

@@ -273,9 +273,9 @@ def check_data_dir_permissions(config=None, **config_kwargs):
         tmp_dir = config.TMP_DIR
 
     try:
-        lib_dir = get_or_create_working_lib_dir(autofix=True, quiet=True, config=config) or config.LIB_DIR
+        lib_dir = get_or_create_working_lib_dir(autofix=True, quiet=True, config=config) or config.ABXPKG_LIB_DIR
     except Exception:
-        lib_dir = config.LIB_DIR
+        lib_dir = config.ABXPKG_LIB_DIR
 
     # Check /tmp dir permissions
     check_tmp_dir(tmp_dir, throw=False, must_exist=True, config=config)
@@ -365,7 +365,7 @@ def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_ex
     from archivebox.config.common import get_config
 
     config = config or get_config(**config_kwargs)
-    lib_dir = lib_dir or config.LIB_DIR
+    lib_dir = lib_dir or config.ABXPKG_LIB_DIR
 
     if not must_exist and not os.path.isdir(lib_dir):
         return True
@@ -373,7 +373,7 @@ def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_ex
     lib_is_valid = False
     try:
         lib_is_valid = dir_is_writable(lib_dir)
-        assert lib_is_valid, f"ArchiveBox user {ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP} is unable to write to LIB_DIR={lib_dir}"
+        assert lib_is_valid, f"ArchiveBox user {ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP} is unable to write to ABXPKG_LIB_DIR={lib_dir}"
         return True
     except Exception as e:
         if not quiet:
@@ -381,16 +381,16 @@ def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_ex
             ERROR_TEXT = "\n".join(
                 (
                     "",
-                    f"[red]:cross_mark: ArchiveBox is unable to use LIB_DIR={pretty_path(lib_dir)}[/red]",
+                    f"[red]:cross_mark: ArchiveBox is unable to use ABXPKG_LIB_DIR={pretty_path(lib_dir)}[/red]",
                     f"   [yellow]{e}[/yellow]",
                     "",
-                    "[blue]Info:[/blue] [grey53]The LIB_DIR is used to store ArchiveBox auto-installed plugin library and binary dependencies.",
+                    "[blue]Info:[/blue] [grey53]The ABXPKG_LIB_DIR is used to store ArchiveBox auto-installed plugin library and binary dependencies.",
                     f"  - It [red]must[/red] be readable and writable by the ArchiveBox user ({ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP}).",
                     "  - It [yellow]should[/yellow] be on a local (ideally fast) drive like an SSD or HDD (not on a network drive or external HDD).",
                     "  - It [yellow]should[/yellow] be able to hold at least 1GB of data (some dependencies like Chrome can be large).[/grey53]",
                     "",
-                    "[violet]Hint:[/violet] Fix it by setting LIB_DIR to a path that meets these requirements, e.g.:",
-                    f"      [green]archivebox config --set LIB_DIR={get_or_create_working_lib_dir(autofix=False, quiet=True) or config.LIB_DIR}[/green]",
+                    "[violet]Hint:[/violet] Fix it by setting ABXPKG_LIB_DIR to a path that meets these requirements, e.g.:",
+                    f"      [green]archivebox config --set ABXPKG_LIB_DIR={get_or_create_working_lib_dir(autofix=False, quiet=True) or config.ABXPKG_LIB_DIR}[/green]",
                     "",
                 ),
             )
@@ -399,11 +399,13 @@ def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_ex
                     ERROR_TEXT,
                     expand=False,
                     border_style="red",
-                    title="[red]:cross_mark: Error with configured LIB_DIR[/red]",
+                    title="[red]:cross_mark: Error with configured ABXPKG_LIB_DIR[/red]",
                     subtitle="[yellow]Dependencies may not auto-install properly until fixed.[/yellow]",
                 ),
             )
             STDERR.print()
         if throw:
-            raise OSError(f"LIB_DIR={lib_dir} is invalid, ArchiveBox is unable to use it and dependencies will fail to install.") from e
+            raise OSError(
+                f"ABXPKG_LIB_DIR={lib_dir} is invalid, ArchiveBox is unable to use it and dependencies will fail to install.",
+            ) from e
     return False

archivebox/personas/importers.py

@@ -47,6 +47,12 @@ VOLATILE_PROFILE_COPY_PATTERNS = (
     "ShaderCache",
     "Service Worker",
     "GCM Store",
+    "chrome-extension_*",
+    "DNR Extension Rules",
+    "Extension Rules",
+    "Extension Scripts",
+    "Extension State",
+    "Local Extension Settings",
     "*.log",
     "Crashpad",
     "BrowserMetrics",
@@ -611,7 +617,7 @@ def export_browser_state(
     if not state_script.exists():
         return False, None, f"Browser state export script not found at {state_script}"
 
-    node_modules_dir = get_config().LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules"
+    node_modules_dir = get_config().ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules"
     chrome_plugin_dir = Path(get_plugins_dir()).resolve()
 
     env = os.environ.copy()

archivebox/personas/models.py

@@ -11,8 +11,6 @@ Each persona has its own:
 __package__ = "archivebox.personas"
 
 import shutil
-import subprocess
-import sys
 from contextlib import contextmanager
 from pathlib import Path
 from typing import TYPE_CHECKING, Any
@@ -62,11 +60,7 @@ VOLATILE_PROFILE_FILE_NAMES = {
 
 def derive_persona_config(*, name: str, config: Mapping[str, Any] | None, persona_dir: Path) -> dict[str, Any]:
     derived = dict(config or {})
-
-    if "CHROME_USER_DATA_DIR" not in derived:
-        derived["CHROME_USER_DATA_DIR"] = str(persona_dir / "chrome_profile")
-    if "CHROME_DOWNLOADS_DIR" not in derived:
-        derived["CHROME_DOWNLOADS_DIR"] = str(persona_dir / "chrome_downloads")
+    derived["PERSONAS_DIR"] = str(persona_dir.parent)
 
     cookies_path = persona_dir / "cookies.txt"
     if "COOKIES_FILE" not in derived and cookies_path.exists():
@@ -165,8 +159,7 @@ class Persona(ModelWithConfig):
 
         Returns dict with:
         - All values from self.config JSONField
-        - CHROME_USER_DATA_DIR (derived from persona path)
-        - CHROME_DOWNLOADS_DIR (derived from persona path)
+        - PERSONAS_DIR (derived from DATA_DIR/personas)
         - COOKIES_FILE (derived from persona path, if file exists)
         - AUTH_STORAGE_FILE (derived from persona path, if file exists)
         - ACTIVE_PERSONA (set to this persona's name)
@@ -259,27 +252,16 @@ class Persona(ModelWithConfig):
         return self.runtime_root_for_snapshot(snapshot) / "chrome_downloads"
 
     def copy_chrome_profile(self, source_dir: Path, destination_dir: Path) -> None:
+        from archivebox.personas.importers import VOLATILE_PROFILE_COPY_PATTERNS
+
         destination_dir.parent.mkdir(parents=True, exist_ok=True)
         shutil.rmtree(destination_dir, ignore_errors=True)
-        destination_dir.mkdir(parents=True, exist_ok=True)
-
-        copy_cmd: list[str] | None = None
-        source_contents = f"{source_dir}/."
-
-        if sys.platform == "darwin":
-            copy_cmd = ["cp", "-cR", source_contents, str(destination_dir)]
-        elif sys.platform.startswith("linux"):
-            copy_cmd = ["cp", "-a", source_contents, str(destination_dir)]
-
-        if copy_cmd:
-            result = subprocess.run(copy_cmd, capture_output=True, text=True)
-            if result.returncode == 0:
-                return
-
-            shutil.rmtree(destination_dir, ignore_errors=True)
-            destination_dir.mkdir(parents=True, exist_ok=True)
-
-        shutil.copytree(source_dir, destination_dir, symlinks=True, dirs_exist_ok=True)
+        shutil.copytree(
+            source_dir,
+            destination_dir,
+            symlinks=True,
+            ignore=shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS),
+        )
 
     def prepare_runtime_for_crawl(self, crawl, chrome_binary: str = "") -> dict[str, str]:
         self.ensure_dirs()
@@ -306,8 +288,13 @@ class Persona(ModelWithConfig):
                 (runtime_root / "chrome_binary.txt").write_text(chrome_binary)
 
         return {
-            "CHROME_USER_DATA_DIR": str(runtime_profile_dir),
-            "CHROME_DOWNLOADS_DIR": str(runtime_downloads_dir),
+            # Hooks derive CHROME_USER_DATA_DIR/CHROME_DOWNLOADS_DIR from
+            # PERSONAS_DIR + ACTIVE_PERSONA. Point PERSONAS_DIR at the
+            # per-crawl runtime root here so CHROME_ISOLATION=crawl never
+            # leaks or reuses the template profile while keeping Chrome path
+            # derivation centralized in the Chrome plugin helpers.
+            "PERSONAS_DIR": str(runtime_root.parent),
+            "ACTIVE_PERSONA": self.name,
         }
 
     def prepare_runtime_for_snapshot(self, snapshot, chrome_binary: str = "") -> dict[str, str]:
@@ -333,8 +320,11 @@ class Persona(ModelWithConfig):
             (runtime_root / "chrome_binary.txt").write_text(chrome_binary)
 
         return {
-            "CHROME_USER_DATA_DIR": str(runtime_profile_dir),
-            "CHROME_DOWNLOADS_DIR": str(runtime_downloads_dir),
+            # See prepare_runtime_for_crawl(): snapshot isolation changes the
+            # persona root, not individual CHROME_* config keys, so standalone
+            # Chrome hooks and ArchiveBox-driven hooks resolve paths the same way.
+            "PERSONAS_DIR": str(runtime_root.parent),
+            "ACTIVE_PERSONA": self.name,
         }
 
     def cleanup_runtime_for_crawl(self, crawl) -> None:

archivebox/plugins/hooks.py

@@ -279,13 +279,13 @@ def run_hook(
             records = process.get_records()  # Get parsed JSONL output
     """
     from archivebox.machine.models import Process, Machine, NetworkInterface
-    from archivebox.config.common import get_config, normalize_runtime_config
+    from archivebox.config.common import ArchiveBoxConfig, _archivebox_config_input_names, get_config, normalize_runtime_config
 
     config_scope = {key.removeprefix("config_"): kwargs.pop(key) for key in list(kwargs) if key.startswith("config_")}
     config_overrides = _config_to_overrides(config)
     resolved_config = get_config(overrides=config_overrides, **config_scope)
     hook_config = normalize_runtime_config(
-        config_overrides if config is not None else resolved_config.for_crawl(),
+        resolved_config.for_crawl_runtime(),
         json_safe=False,
     )
 
@@ -365,6 +365,9 @@ def run_hook(
 
     # Set up environment with base paths
     env = os.environ.copy()
+    archivebox_config_input_names = _archivebox_config_input_names()
+    for key in archivebox_config_input_names:
+        env.pop(key, None)
     env["DATA_DIR"] = str(CONSTANTS.DATA_DIR)
     env["LIBRARY_VERSION"] = VERSION
     env.setdefault("MACHINE_ID", os.environ.get("MACHINE_ID", CONSTANTS.MACHINE_ID))
@@ -375,10 +378,9 @@ def run_hook(
     if crawl_dir:
         env["CRAWL_DIR"] = str(crawl_dir)
 
-    # Export runtime library roots; abx-dl/abxpkg own executable lookup env.
-    lib_dir = hook_config.get("LIB_DIR")
+    # Export the runtime library root; abx-dl/abxpkg own executable lookup env.
+    lib_dir = hook_config.get("ABXPKG_LIB_DIR")
     if lib_dir:
-        env["LIB_DIR"] = str(lib_dir)
         env["ABXPKG_LIB_DIR"] = str(lib_dir)
 
     # Set Node.js module resolution paths.
@@ -400,10 +402,9 @@ def run_hook(
         env["NODE_PATH"] = os.pathsep.join(node_path_parts)
 
     # Export all config values to environment (already merged by get_config())
-    # Skip keys we've already handled specially above (PATH, LIB_DIR, NODE_PATH, etc.)
+    # Skip keys we've already handled specially above (PATH, ABXPKG_LIB_DIR, NODE_PATH, etc.)
     SKIP_KEYS = {
         "PATH",
-        "LIB_DIR",
         "ABXPKG_LIB_DIR",
         "NODE_PATH",
         "NODE_MODULES_DIR",
@@ -413,9 +414,12 @@ def run_hook(
         "SNAP_DIR",
         "CRAWL_DIR",
     }
+    canonical_config_keys = set(ArchiveBoxConfig.model_fields)
     for key, value in hook_config.items():
         if key in SKIP_KEYS:
             continue  # Already handled specially above, don't overwrite
+        if key in archivebox_config_input_names and key not in canonical_config_keys:
+            continue
         if value is None:
             continue
         elif isinstance(value, bool):

archivebox/services/archive_result_service.py

@@ -237,12 +237,12 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh
     return len(next_title) > len(current)
 
 
-def _has_content_files(output_files: Any) -> bool:
-    return any(Path(path).suffix not in {".log", ".pid", ".sh"} for path in _normalize_output_files(output_files))
-
-
-def _is_signal_interrupted_exit(exit_code: int) -> bool:
-    return exit_code < 0 or (exit_code >= 128 and exit_code != PROCESS_EXIT_SKIPPED)
+def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str:
+    if event.exit_code == PROCESS_EXIT_SKIPPED:
+        return "skipped"
+    if event.exit_code != 0:
+        return "failed"
+    return "noresults"
 
 
 def _iter_archiveresult_records(stdout: str) -> list[dict]:
@@ -444,24 +444,14 @@ class ArchiveResultService(BaseService):
                     ).now()
             return
 
-        # TODO: consider moving this fallback derivation into abx-dl itself.
-        # First try both patterns: if the whole abx-dl process crashes, restarting
-        # the snapshot may be enough, but don't guess before validating it.
-        process_interrupted = _is_signal_interrupted_exit(event.exit_code)
-        process_failed = event.exit_code not in (0, PROCESS_EXIT_SKIPPED) and not process_interrupted
+        process_failed = _status_for_process_without_archive_result(event) == "failed"
         with _perf_span("archivebox.ArchiveResultService.on_ProcessCompletedEvent.emit_archive_result_fallback"):
             await event.emit(
                 ArchiveResultEvent(
                     snapshot_id=snapshot_event.snapshot_id,
                     plugin=event.plugin_name,
                     hook_name=event.hook_name,
-                    status=(
-                        "queued"
-                        if process_interrupted
-                        else "failed"
-                        if process_failed
-                        else ("succeeded" if _has_content_files(event.output_files) else "noresult")
-                    ),
+                    status=_status_for_process_without_archive_result(event),
                     output_str=event.stderr if process_failed else "",
                     output_files=event.output_files,
                     start_ts=event.start_ts,

archivebox/services/runner.py

@@ -566,7 +566,7 @@ class CrawlRunner:
             current_process.machine = current_iface.machine
             current_process.save(update_fields=["iface", "machine", "modified_at"])
         self.persona = self.crawl.resolve_persona()
-        self.base_config = get_config(crawl=self.crawl)
+        self.base_config = get_config(crawl=self.crawl, overrides=self.config_overrides)
         self.derived_config = dict(Machine.current().config or {})
         self.crawl_output_dir = str(self.crawl.output_dir)
         if self.persona:
@@ -576,7 +576,6 @@ class CrawlRunner:
                     chrome_binary=self.base_config["CHROME_BINARY"],
                 ),
             )
-        self.base_config.update(self.config_overrides)
         if self.selected_plugins is None:
             raw_plugins = str(self.base_config.get("PLUGINS") or "").strip()
             if raw_plugins:
@@ -765,8 +764,7 @@ class CrawlRunner:
         snapshot = Snapshot.objects.select_related("crawl", "crawl__created_by").get(id=snapshot_id)
         self.crawl = snapshot.crawl
         self.persona = snapshot.crawl.resolve_persona()
-        self.base_config = get_config(crawl=snapshot.crawl, persona=self.persona)
-        self.base_config.update(self.config_overrides)
+        self.base_config = get_config(crawl=snapshot.crawl, persona=self.persona, overrides=self.config_overrides)
         self.crawl_output_dir = str(snapshot.crawl.output_dir)
         runtime_chrome_overrides = {}
         if self.persona:
@@ -782,8 +780,8 @@ class CrawlRunner:
                 crawl_downloads_dir.mkdir(parents=True, exist_ok=True)
                 runtime_chrome_overrides.update(
                     {
-                        "CHROME_USER_DATA_DIR": str(self.persona.runtime_profile_dir_for_crawl(snapshot.crawl)),
-                        "CHROME_DOWNLOADS_DIR": str(crawl_downloads_dir),
+                        "PERSONAS_DIR": str(self.persona.runtime_root_for_crawl(snapshot.crawl).parent),
+                        "ACTIVE_PERSONA": self.persona.name,
                     },
                 )
         snapshot_output_dir = str(snapshot.output_dir)
@@ -802,7 +800,6 @@ class CrawlRunner:
             },
         )
         normalized_config = normalize_runtime_config(config)
-        normalized_config.update(self.config_overrides)
         return {
             "id": str(snapshot.id),
             "url": snapshot.url,
@@ -1857,6 +1854,7 @@ def run_due_binary(binary, *, lock_seconds: int) -> bool:
 async def _run_install(plugin_names: list[str] | None = None) -> None:
     from archivebox.config.common import get_config
     from archivebox.machine.models import Machine
+    from archivebox.plugins.discovery import get_enabled_plugins
 
     plugins = _discover_archivebox_plugins()
     config = get_config(include_machine=False)
@@ -1877,10 +1875,13 @@ async def _run_install(plugin_names: list[str] | None = None) -> None:
     bus_destroyed = False
 
     try:
-        selected_plugins = filter_plugins(plugins, list(plugin_names), include_providers=True) if plugin_names else plugins
+        if plugin_names:
+            selected_plugins = filter_plugins(plugins, list(plugin_names), include_providers=True)
+        else:
+            selected_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
         if not selected_plugins:
             return
-        plugins_label = ", ".join(plugin_names) if plugin_names else f"all ({len(plugins)} available)"
+        plugins_label = ", ".join(plugin_names) if plugin_names else f"enabled ({len(selected_plugins)} of {len(plugins)} available)"
         timeout_seconds = config["TIMEOUT"]
         stdout_is_tty = sys.stdout.isatty()
         stderr_is_tty = sys.stderr.isatty()
@@ -1951,7 +1952,7 @@ async def _run_install(plugin_names: list[str] | None = None) -> None:
                         await bus.destroy(clear=False)
                         bus_destroyed = True
             if live_ui is not None:
-                live_ui.print_summary(output_dir=output_dir)
+                live_ui.print_summary(output_dir=output_dir, archive_results=[])
     finally:
         if not bus_destroyed:
             await bus.destroy(clear=False)
@@ -2240,7 +2241,7 @@ def run_pending_crawls(
     while True:
         raise_if_shutdown_requested()
         now_monotonic = time.monotonic()
-        if now_monotonic - last_retention_at >= (60.0 if daemon else 1.0):
+        if crawl_id is None and now_monotonic - last_retention_at >= (60.0 if daemon else 1.0):
             for model in (ArchiveResult, Snapshot, Crawl, Process):
                 # Keep the tight scheduler loop anchored on indexed delete_at
                 # columns only. Backfilling missing delete_at values has to read
@@ -2422,7 +2423,7 @@ def run_pending_crawls(
                 continue
 
         now_monotonic = time.monotonic()
-        if now_monotonic - last_retention_repair_at >= (60.0 if daemon else 0.0):
+        if crawl_id is None and now_monotonic - last_retention_repair_at >= (60.0 if daemon else 0.0):
             for model in (ArchiveResult, Snapshot, Crawl, Process):
                 # No runnable work was found on this scheduler pass. This is
                 # the bounded repair point for missing retention deadlines,

archivebox/templates/admin/snapshots_grid.html

@@ -229,13 +229,14 @@
           <code class="card-url" title="{{obj.url}}">{{obj.url}}</code>
           <div class="card-media">
             <a href="{% snapshot_base_url obj %}/index.html" class="card-thumbnail">
-              {% snapshot_url obj 'screenshot/screenshot.png' as screenshot_plugin_url %}
-              {% snapshot_url obj 'screenshot.png' as screenshot_root_url %}
-              {% snapshot_url obj 'favicon.ico' as favicon_root_url %}
+              {% snapshot_archiveresult_url obj 'screenshot' 'screenshot.png' as screenshot_url %}
+              {% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot-1.png' as extension_screenshot_1_url %}
+              {% snapshot_archiveresult_url obj 'chrome_extension_screenshot' 'screenshot.png' as extension_screenshot_url %}
+              {% snapshot_archiveresult_url obj 'favicon' 'favicon.ico' as favicon_url %}
               <img
-                src="{{ screenshot_plugin_url }}"
+                src="{% firstof screenshot_url extension_screenshot_1_url extension_screenshot_url favicon_url '/static/spinner.gif' %}"
                 alt="{{obj.title|default:'Not yet archived...'}}"
-                data-fallbacks="{{ screenshot_root_url }},{{ favicon_root_url }}"
+                data-fallbacks="{{ extension_screenshot_1_url }},{{ extension_screenshot_url }},{{ favicon_url }}"
                 onerror="const fallbacks=(this.dataset.fallbacks || '').split(',').filter(Boolean); if (fallbacks.length) { this.dataset.fallbacks=fallbacks.slice(1).join(','); this.src=fallbacks[0]; } else { this.onerror=null; this.closest('.card-thumbnail').classList.add('missing'); this.src='/static/spinner.gif'; }"
               />
             </a>

archivebox/templates/core/public_index.html

@@ -656,18 +656,22 @@
                     return url
                 }
 
+                function replaceRegionFromDocument(doc, selector) {
+                    const next = doc.querySelector(selector)
+                    const current = document.querySelector(selector)
+                    if (next && current) current.innerHTML = next.innerHTML
+                }
+
                 async function renderNow() {
                     const url = buildListUrl()
                     const response = await fetch(url, {headers: {'X-Requested-With': 'XMLHttpRequest'}, signal: controller.signal})
                     const html = await response.text()
                     const doc = new DOMParser().parseFromString(html, 'text/html')
-                    const nextList = doc.querySelector('.public-snapshot-list')
-                    const currentList = document.querySelector('.public-snapshot-list')
-                    if (nextList && currentList) {
-                        currentList.replaceWith(nextList)
-                        initPublicSearchMode()
-                        setSearchLoading(controller, resultVersion ? 'Searching... ' + resultVersion + ' found' : 'Searching...')
-                    }
+                    replaceRegionFromDocument(doc, '.public-snapshot-count')
+                    replaceRegionFromDocument(doc, '#table-bookmarks thead th:nth-child(3)')
+                    replaceRegionFromDocument(doc, '#table-bookmarks tbody')
+                    replaceRegionFromDocument(doc, '.public-pagination')
+                    setSearchLoading(controller, resultVersion ? 'Searching... ' + resultVersion + ' found' : 'Searching...')
                 }
 
                 const listUrl = buildListUrl()

archivebox/templates/core/snapshot.html

@@ -1150,8 +1150,13 @@
                             </a>
                         </div>
                         <div class="header-title-line header-toggle-trigger">
-                            <img src="{% snapshot_url snapshot 'favicon/favicon.ico' %}" onerror="this.onerror=null;this.src=&quot;data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;&quot;" alt="Favicon" class="favicon"/>
-                            <span class="header-title-text">{{title|truncatechars:120|safe}}</span>
+                            {% snapshot_archiveresult_url snapshot 'favicon' 'favicon.ico' as header_favicon_url %}
+                            {% if header_favicon_url %}
+                                <img src="{{ header_favicon_url }}" onerror="this.onerror=null;this.src=&quot;data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;&quot;" alt="Favicon" class="favicon"/>
+                            {% else %}
+                                <img src="data:image/svg+xml;utf8,&lt;svg xmlns='http://www.w3.org/2000/svg' viewBox='0 0 20 20' fill='none' stroke='rgba(255,255,255,0.65)' stroke-width='1.4'&gt;&lt;circle cx='10' cy='10' r='7.25'/&gt;&lt;ellipse cx='10' cy='10' rx='3.25' ry='7.25'/&gt;&lt;line x1='2.75' y1='10' x2='17.25' y2='10'/&gt;&lt;/svg&gt;" alt="Favicon" class="favicon"/>
+                            {% endif %}
+                            <span class="header-title-text">{{title|truncatechars:120}}</span>
                             {% if title_tags %}
                                 <span class="header-tags">
                                     <span class="tag-pill permission-pill">{{ snapshot_permissions_icon }} {{ snapshot_permissions }}</span>

archivebox/tests/conftest.py

@@ -378,19 +378,18 @@ def isolated_data_dir(tmp_path):
 @pytest.fixture
 def hermetic_lib_dir(tmp_path, monkeypatch):
     """
-    Point LIB_DIR at a tmp directory so the test can write fake binaries
+    Point ABXPKG_LIB_DIR at a tmp directory so the test can write fake binaries
     without touching the real ``~/Library/Application Support/abx/lib`` (which
     can contain symlinks to SIP-protected system binaries on macOS).
 
-    Opt-in only: most tests should reuse the cached real LIB_DIR for speed —
+    Opt-in only: most tests should reuse the cached real ABXPKG_LIB_DIR for speed —
     rebuilding from scratch per-test adds ~10× overhead. Use this only when
-    the test synthesizes binary paths or validates LIB_DIR-relative behavior.
+    the test synthesizes binary paths or validates ABXPKG_LIB_DIR-relative behavior.
     """
     import archivebox.machine.models as machine_models
 
     lib_dir = tmp_path / "lib"
     (lib_dir / "bin").mkdir(parents=True, exist_ok=True)
-    monkeypatch.setenv("LIB_DIR", str(lib_dir))
     monkeypatch.setenv("ABXPKG_LIB_DIR", str(lib_dir))
     machine_models._CURRENT_MACHINE = None
     machine_models._CURRENT_PROCESS = None

archivebox/tests/test_archive_result_service.py

@@ -3,7 +3,7 @@ import pytest
 
 
 from abxpkg.binary_service import BinaryRequestEvent
-from abx_dl.events import ArchiveResultEvent, ProcessEvent, ProcessStartedEvent
+from abx_dl.events import ArchiveResultEvent, ProcessCompletedEvent, ProcessEvent, ProcessStartedEvent, SnapshotEvent
 from abx_dl.orchestrator import create_bus
 from abx_dl.output_files import OutputFile
 
@@ -331,6 +331,55 @@ def test_process_completed_projects_noresults_archiveresult():
     assert result.output_str == "No title found"
 
 
+def test_process_completed_without_archive_result_does_not_infer_success_from_output_files(snapshot):
+    from archivebox.core.models import ArchiveResult
+    from archivebox.services.archive_result_service import ArchiveResultService
+    import asyncio
+
+    plugin_dir = Path(snapshot.output_dir) / "wget"
+    plugin_dir.mkdir(parents=True, exist_ok=True)
+    (plugin_dir / "index.html").write_text("<html>downloaded but not reported</html>")
+
+    bus = create_bus(name="test_process_completed_without_archive_result_output_files")
+    ArchiveResultService(bus)
+
+    snapshot_event = SnapshotEvent(
+        url=snapshot.url,
+        snapshot_id=str(snapshot.id),
+        output_dir=str(snapshot.output_dir),
+    )
+    completed_event = ProcessCompletedEvent(
+        plugin_name="wget",
+        hook_name="on_Snapshot__06_wget.finite.bg",
+        hook_path="/usr/bin/env",
+        hook_args=[],
+        env={},
+        timeout=60,
+        stdout="",
+        stderr="",
+        exit_code=0,
+        status="succeeded",
+        output_dir=str(plugin_dir),
+        output_files=[OutputFile(path="index.html", extension="html", mimetype="text/html", size=36)],
+        start_ts="2026-03-22T12:00:00+00:00",
+        end_ts="2026-03-22T12:00:01+00:00",
+        event_parent_id=snapshot_event.event_id,
+    )
+
+    async def emit_events() -> None:
+        await bus.emit(snapshot_event).now()
+        await bus.emit(completed_event).now()
+        await bus.wait_until_idle()
+
+    asyncio.run(emit_events())
+
+    result = ArchiveResult.objects.get(snapshot=snapshot, plugin="wget", hook_name="on_Snapshot__06_wget.finite.bg")
+    assert result.status == ArchiveResult.StatusChoices.NORESULTS
+    assert result.output_str == ""
+    assert result.output_files == {"index.html": {"extension": "html", "mimetype": "text/html", "size": 36}}
+    _cleanup_machine_process_rows()
+
+
 def test_retry_failed_archiveresults_requeues_snapshot_in_queued_state():
     from archivebox.core.models import ArchiveResult, Snapshot
 

archivebox/tests/test_binary_service.py

@@ -29,7 +29,6 @@ def _runtime_env(data_dir: Path, bin_dir: Path, *, lib_dir: Path | None = None)
     assert archivebox_bin, "archivebox console script must be available for CLI tests"
     lib_dir = lib_dir or data_dir / "lib"
     return {
-        "LIB_DIR": str(lib_dir),
         "ABXPKG_LIB_DIR": str(lib_dir),
         "PATH": os.pathsep.join([str(bin_dir), str(Path(archivebox_bin).parent), "/usr/bin", "/bin", "/usr/sbin", "/sbin"]),
     }

archivebox/tests/test_cli_add.py

@@ -539,6 +539,67 @@ def test_add_tagged_single_url_seals_without_duplicate_snapshot_tags(initialized
     assert not unexpected_failures
 
 
+@pytest.mark.timeout(180)
+def test_add_title_hook_env_gets_canonical_runtime_config_without_archivebox_selectors_or_aliases(initialized_archive):
+    env = os.environ.copy()
+    env.update(
+        {
+            "USE_COLOR": "False",
+            "SHOW_PROGRESS": "False",
+            "PLUGINS": "title",
+            "SAVE_TITLE": "True",
+            "SECRET_KEY": "hook-env-secret-must-not-leak",
+            "PUBLIC_ADD_VIEW": "True",
+            "ADMIN_PASSWORD": "hook-env-admin-password-must-not-leak",
+            "URL_BLACKLIST": "$^",
+            "ARCHIVEBOX_TEST_ARBITRARY_ENV": "preserved-user-env",
+            "LD_PRELOAD": "",
+            "TIMEOUT": "60",
+            "CRAWL_MAX_CONCURRENT_SNAPSHOTS": "1",
+        },
+    )
+    result = run_archivebox_cmd(
+        [
+            "add",
+            "--depth=0",
+            "--plugins=title",
+            "https://example.com/?archivebox-title-env=1",
+        ],
+        cwd=initialized_archive,
+        env=env,
+        timeout=180,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+
+    with use_archivebox_db(initialized_archive):
+        crawl = Crawl.objects.get()
+        snapshot = Snapshot.objects.get()
+        title_result = ArchiveResult.objects.select_related("process").get(snapshot=snapshot, plugin="title")
+        process_env = title_result.process.env
+
+    assert crawl.config["PLUGINS"] == "title"
+    assert snapshot.url == "https://example.com/?archivebox-title-env=1"
+    assert snapshot.status == Snapshot.StatusChoices.SEALED
+    assert snapshot.title == "Example Domain"
+    assert title_result.status == ArchiveResult.StatusChoices.SUCCEEDED
+    assert title_result.output_str == "Example Domain"
+    assert title_result.process.exit_code == 0
+    assert process_env["TITLE_ENABLED"] == "True"
+    assert process_env["CHROME_ENABLED"] == "True"
+    assert process_env["WGET_ENABLED"] == "False"
+    assert process_env["ARCHIVEBOX_TEST_ARBITRARY_ENV"] == "preserved-user-env"
+    assert process_env["LD_PRELOAD"] == ""
+    assert "PLUGINS" not in process_env
+    assert "SAVE_TITLE" not in process_env
+    assert "SEARCH_BACKEND_ENGINE" not in process_env
+    assert "SECRET_KEY" not in process_env
+    assert "ADMIN_PASSWORD" not in process_env
+    assert "PUBLIC_ADD_VIEW" not in process_env
+    assert "URL_BLACKLIST" not in process_env
+    assert "hook-env-secret-must-not-leak" not in json.dumps(process_env)
+    assert "hook-env-admin-password-must-not-leak" not in json.dumps(process_env)
+
+
 def test_add_index_only_rejected_urls_leave_empty_crawl_for_runner_to_seal(initialized_archive):
     """Index-only add only creates the crawl; rejected URLs are sealed by the runner."""
     env = cli_env(disable_extractors=True)

archivebox/tests/test_cli_install.py

@@ -11,6 +11,8 @@ from archivebox.tests.conftest import run_archivebox_cmd
 
 import pytest
 
+from archivebox.cli.archivebox_install import ensure_data_dir_lib_symlink
+from archivebox.config.paths import get_machine_type
 from archivebox.core.models import Snapshot
 from archivebox.crawls.models import Crawl
 from archivebox.machine.models import Binary
@@ -114,6 +116,9 @@ def test_install_from_empty_dir_initializes_collection(tmp_path):
     env = os.environ.copy()
     tmp_short = Path("/tmp") / f"abx-install-empty-{tmp_path.name}"
     tmp_short.mkdir(parents=True, exist_ok=True)
+    stale_lib_dir = tmp_path / "lib"
+    stale_lib_dir.mkdir()
+    (stale_lib_dir / "stale.txt").write_text("stale")
     env.update(
         {
             "TMP_DIR": str(tmp_short),
@@ -134,6 +139,8 @@ def test_install_from_empty_dir_initializes_collection(tmp_path):
     assert "Installing specific binaries: git" in output
     assert (tmp_path / "ArchiveBox.conf").is_file()
     assert (tmp_path / "index.sqlite3").is_file()
+    assert (tmp_path / "lib").is_symlink()
+    assert (tmp_path / "lib").resolve().is_dir()
 
     with use_archivebox_db(tmp_path):
         assert Snapshot.objects.count() == 0
@@ -141,6 +148,16 @@ def test_install_from_empty_dir_initializes_collection(tmp_path):
         assert Binary.objects.filter(status="installed", name="git").count() == 1
 
 
+def test_install_lib_symlink_preserves_scoped_data_lib_dir(tmp_path):
+    """DATA_DIR/lib/<machine_type> is a valid ABXPKG_LIB_DIR and must not be replaced by a parent symlink."""
+    scoped_lib_dir = tmp_path / "lib" / get_machine_type()
+
+    assert ensure_data_dir_lib_symlink(tmp_path, scoped_lib_dir) is None
+    assert (tmp_path / "lib").is_dir()
+    assert not (tmp_path / "lib").is_symlink()
+    assert scoped_lib_dir.is_dir()
+
+
 def test_install_updates_binary_table(initialized_archive):
     """Test that install completes and only mutates dependency state."""
     env = os.environ.copy()
@@ -174,3 +191,5 @@ def test_install_updates_binary_table(initialized_archive):
     assert snapshot_count == 0
     assert binary_counts.get("installed", 0) > 0
     assert installed_git == 1
+    assert (initialized_archive / "lib").is_symlink()
+    assert (initialized_archive / "lib").resolve().is_dir()

archivebox/tests/test_cli_run.py

@@ -566,7 +566,9 @@ class TestRunDaemonMode:
                 if proc.poll() is not None:
                     stdout = proc.stdout.read()
                     stderr = proc.stderr.read()
-                    pytest.fail(f"daemon exited before starting runner: code={proc.returncode}\nstdout={stdout}\nstderr={stderr}")
+                    raise AssertionError(
+                        f"daemon exited before starting runner: code={proc.returncode}\nstdout={stdout}\nstderr={stderr}",
+                    )
                 with use_archivebox_db(initialized_archive):
                     started = Process.objects.filter(
                         process_type=Process.TypeChoices.ORCHESTRATOR,

archivebox/tests/test_cli_server.py

@@ -207,8 +207,7 @@ def test_reload_workers_use_current_interpreter_and_supervisord_managed_runner()
 
 
 def test_server_daemon_starts_real_plugin_owned_sonic_worker(archivebox_daemon_server):
-    if shutil.which("sonic") is None:
-        pytest.skip("sonic server binary is required for Sonic worker integration tests")
+    assert shutil.which("sonic") is not None, "sonic server binary is required for Sonic worker integration tests"
 
     server = archivebox_daemon_server(
         SEARCH_BACKEND_ENGINE="sonic",
@@ -332,8 +331,7 @@ def test_sonic_worker_is_disabled_when_sonic_disabled(tmp_path):
 
 
 def test_sonic_daemon_event_handler_accepts_real_running_worker(archivebox_daemon_server):
-    if shutil.which("sonic") is None:
-        pytest.skip("sonic server binary is required for Sonic worker integration tests")
+    assert shutil.which("sonic") is not None, "sonic server binary is required for Sonic worker integration tests"
 
     from abx_dl.events import ProcessStdoutEvent
     from abx_dl.orchestrator import create_bus

archivebox/tests/test_config_MAX_limits.py

@@ -115,7 +115,7 @@ def test_run_snapshot_seals_descendant_when_crawl_max_size_is_reached(tmp_path):
     crawl = Crawl.objects.create(
         urls="https://example.com",
         config={
-            "LIB_DIR": str(tmp_path / "lib"),
+            "ABXPKG_LIB_DIR": str(tmp_path / "lib"),
             "PLUGINS": "__archivebox_test_no_plugins__",
             "CHROME_BINARY": "",
             "CRAWL_MAX_SIZE": 16,

archivebox/tests/test_config_SAVE_TITLE.py

@@ -14,7 +14,7 @@ pytestmark = pytest.mark.django_db(transaction=True)
 
 def _install_chrome(tmp_path, env):
     env["CHROME_ISOLATION"] = "snapshot"
-    env["LIB_DIR"] = str(tmp_path / "lib")
+    env["ABXPKG_LIB_DIR"] = str(tmp_path / "lib")
     install_process = run_archivebox_cmd(
         ["install", "chrome"],
         cwd=tmp_path,
@@ -22,7 +22,7 @@ def _install_chrome(tmp_path, env):
         timeout=600,
     )
     assert install_process.returncode == 0, install_process.stderr or install_process.stdout
-    cached_browser = _find_cached_chrome(Path(env["LIB_DIR"]))
+    cached_browser = _find_cached_chrome(Path(env["ABXPKG_LIB_DIR"]))
     if cached_browser is not None:
         env["CHROME_BINARY"] = str(cached_browser)
         return

archivebox/tests/test_crawl_runner.py

@@ -89,12 +89,13 @@ def test_snapshot_payload_uses_crawl_chrome_dirs_by_default():
     config = payload["config"]
     other_config = other_payload["config"]
 
-    assert Path(config["CHROME_USER_DATA_DIR"]).is_relative_to(crawl.output_dir)
-    assert Path(config["CHROME_DOWNLOADS_DIR"]).is_relative_to(crawl.output_dir)
-    assert Path(config["CHROME_USER_DATA_DIR"]).name == "chrome_profile"
-    assert Path(config["CHROME_DOWNLOADS_DIR"]).name == "chrome_downloads"
-    assert Path(config["CHROME_USER_DATA_DIR"]) == Path(other_config["CHROME_USER_DATA_DIR"])
-    assert Path(config["CHROME_DOWNLOADS_DIR"]) == Path(other_config["CHROME_DOWNLOADS_DIR"])
+    personas_dir = Path(config["PERSONAS_DIR"])
+    other_personas_dir = Path(other_config["PERSONAS_DIR"])
+    assert personas_dir.is_relative_to(crawl.output_dir)
+    assert personas_dir == persona.runtime_root_for_crawl(crawl).parent
+    assert personas_dir == other_personas_dir
+    assert personas_dir / config["ACTIVE_PERSONA"] / "chrome_profile" == persona.runtime_profile_dir_for_crawl(crawl)
+    assert personas_dir / config["ACTIVE_PERSONA"] / "chrome_downloads" == persona.runtime_downloads_dir_for_crawl(crawl)
     assert crawl_downloads_sentinel.read_text() == "keep"
     assert config["ACTIVE_PERSONA"] == "RuntimePersona"
     assert Path(config["CRAWL_DIR"]) == crawl.output_dir
@@ -130,12 +131,15 @@ def test_snapshot_payload_uses_snapshot_chrome_dirs_when_snapshot_isolated():
     config = payload["config"]
     other_config = other_payload["config"]
 
-    assert Path(config["CHROME_USER_DATA_DIR"]).is_relative_to(snapshot.output_dir)
-    assert Path(config["CHROME_DOWNLOADS_DIR"]).is_relative_to(snapshot.output_dir)
-    assert Path(other_config["CHROME_USER_DATA_DIR"]).is_relative_to(other_snapshot.output_dir)
-    assert Path(other_config["CHROME_DOWNLOADS_DIR"]).is_relative_to(other_snapshot.output_dir)
-    assert Path(config["CHROME_USER_DATA_DIR"]) != Path(other_config["CHROME_USER_DATA_DIR"])
-    assert Path(config["CHROME_DOWNLOADS_DIR"]) != Path(other_config["CHROME_DOWNLOADS_DIR"])
+    personas_dir = Path(config["PERSONAS_DIR"])
+    other_personas_dir = Path(other_config["PERSONAS_DIR"])
+    assert personas_dir.is_relative_to(snapshot.output_dir)
+    assert other_personas_dir.is_relative_to(other_snapshot.output_dir)
+    assert personas_dir == persona.runtime_root_for_snapshot(snapshot).parent
+    assert other_personas_dir == persona.runtime_root_for_snapshot(other_snapshot).parent
+    assert personas_dir != other_personas_dir
+    assert personas_dir / config["ACTIVE_PERSONA"] / "chrome_profile" == persona.runtime_profile_dir_for_snapshot(snapshot)
+    assert personas_dir / config["ACTIVE_PERSONA"] / "chrome_downloads" == persona.runtime_downloads_dir_for_snapshot(snapshot)
     assert Path(config["CRAWL_DIR"]) == crawl.output_dir
     assert Path(config["SNAP_DIR"]) == snapshot.output_dir
 
@@ -465,7 +469,7 @@ def test_machine_service_persists_only_derived_config_events(tmp_path, hermetic_
     # path / install cache — that's the security boundary that stops plugins
     # from rewriting arbitrary user config via events. So CHROME_USER_DATA_DIR
     # from the derived payload is dropped; WGET_BINARY made it in (inside
-    # LIB_DIR) then the unset removed it; ABX_INSTALL_CACHE survives.
+    # ABXPKG_LIB_DIR) then the unset removed it; ABX_INSTALL_CACHE survives.
     assert machine.config == {
         "ABX_INSTALL_CACHE": {"wget": "2026-03-24T00:00:00+00:00"},
         "ABX_PNPM_CACHE": "/tmp/pnpm-cache",
@@ -483,8 +487,8 @@ def test_load_run_state_uses_real_lib_dir_for_machine_binary_config(tmp_path, he
     from archivebox.machine.models import Machine
     from archivebox.services.runner import CrawlRunner
 
-    resolved_lib_dir = get_config(include_machine=False).LIB_DIR
-    assert resolved_lib_dir == hermetic_lib_dir, f"LIB_DIR override not applied: {resolved_lib_dir!r} != {hermetic_lib_dir!r}"
+    resolved_lib_dir = get_config(include_machine=False).ABXPKG_LIB_DIR
+    assert resolved_lib_dir == hermetic_lib_dir, f"ABXPKG_LIB_DIR override not applied: {resolved_lib_dir!r} != {hermetic_lib_dir!r}"
 
     wget_binary = resolved_lib_dir / "bin" / "wget"
     wget_binary.write_text("#!/bin/sh\n", encoding="utf-8")
@@ -517,8 +521,8 @@ def test_load_run_state_uses_real_lib_dir_for_machine_binary_config(tmp_path, he
     runner = CrawlRunner(crawl)
     snapshot_ids = runner.load_run_state()
 
-    # ``derived_config`` is Machine.config sanitized against LIB_DIR. Binary
-    # paths outside LIB_DIR drop out (YTDLP_BINARY → ``/tmp/...``); the
+    # ``derived_config`` is Machine.config sanitized against ABXPKG_LIB_DIR. Binary
+    # paths outside ABXPKG_LIB_DIR drop out (YTDLP_BINARY → ``/tmp/...``); the
     # ArchiveBox.conf mirror values (CHROME_ISOLATION, CHROME_USER_DATA_DIR,
     # ABX_INSTALL_CACHE) survive so plugin hooks see the same runtime cache
     # the user/runner persisted.
@@ -528,7 +532,7 @@ def test_load_run_state_uses_real_lib_dir_for_machine_binary_config(tmp_path, he
         "CHROME_ISOLATION": "snapshot",
         "CHROME_USER_DATA_DIR": "/tmp/stale-profile",
     }
-    assert runner.base_config["LIB_DIR"] == resolved_lib_dir
+    assert runner.base_config["ABXPKG_LIB_DIR"] == resolved_lib_dir
     assert runner.base_config["CHROME_KEEPALIVE"] is False
     assert runner.selected_plugins == ["__archivebox_test_no_plugins__"]
     assert Snapshot.objects.filter(id__in=snapshot_ids, crawl=crawl, url="https://example.com").count() == 1
@@ -546,7 +550,7 @@ def test_crawl_runner_empty_plugin_selection_emits_lifecycle_and_seals_crawl(tmp
     crawl = Crawl.objects.create(
         urls="https://example.com",
         config={
-            "LIB_DIR": str(tmp_path / "lib"),
+            "ABXPKG_LIB_DIR": str(tmp_path / "lib"),
             "PLUGINS": "__archivebox_test_no_plugins__",
             "CHROME_BINARY": "",
         },
@@ -694,7 +698,7 @@ def test_run_pending_crawls_processes_queued_crawl_before_missing_binary_backlog
     crawl = Crawl.objects.create(
         urls="https://example.com",
         config={
-            "LIB_DIR": str(tmp_path / "lib"),
+            "ABXPKG_LIB_DIR": str(tmp_path / "lib"),
             "PLUGINS": "__archivebox_test_no_plugins__",
             "CHROME_BINARY": "",
         },

archivebox/tests/test_hooks.py

@@ -766,7 +766,7 @@ print(json.dumps({
         hook_path,
         output_dir,
         config={
-            "LIB_DIR": str(lib_dir),
+            "ABXPKG_LIB_DIR": str(lib_dir),
             "NODE_PATH": configured_node_path,
         },
         timeout=10,
@@ -812,7 +812,7 @@ print(json.dumps({
         hook_path,
         output_dir,
         config={
-            "LIB_DIR": str(tmp_path / "lib"),
+            "ABXPKG_LIB_DIR": str(tmp_path / "lib"),
         },
         timeout=10,
     )

archivebox/tests/test_machine_models.py

@@ -154,7 +154,7 @@ class TestMachineModel:
         from archivebox.config.constants import CONSTANTS
 
         Machine.current()  # Ensure machine exists
-        wget_path = CONSTANTS.DEFAULT_LIB_DIR / "wget"
+        wget_path = CONSTANTS.DEFAULT_ABXPKG_LIB_DIR / "wget"
         wget_path.parent.mkdir(parents=True, exist_ok=True)
         wget_path.write_text("#!/bin/sh\n")
         cleanup_paths.append(wget_path)
@@ -179,7 +179,7 @@ class TestMachineModel:
         from archivebox.config.constants import CONSTANTS
 
         Machine.current()  # Ensure machine exists
-        wget_path = CONSTANTS.DEFAULT_LIB_DIR / "wget"
+        wget_path = CONSTANTS.DEFAULT_ABXPKG_LIB_DIR / "wget"
         wget_path.parent.mkdir(parents=True, exist_ok=True)
         wget_path.write_text("#!/bin/sh\n")
         cleanup_paths.append(wget_path)
@@ -209,12 +209,12 @@ class TestMachineModel:
         ``Machine.config`` is the file ↔ DB mirror of ``ArchiveBox.conf``, so
         non-binary keys (``CHROME_ISOLATION``, ``CHROMIUM_VERSION``, etc.) are
         preserved on read. Only ``_BINARY`` paths get validated against
-        ``LIB_DIR`` and dropped when stale/missing.
+        ``ABXPKG_LIB_DIR`` and dropped when stale/missing.
         """
         import archivebox.machine.models as models
         from archivebox.config.constants import CONSTANTS
 
-        active_lib_dir = CONSTANTS.DEFAULT_LIB_DIR
+        active_lib_dir = CONSTANTS.DEFAULT_ABXPKG_LIB_DIR
         active_lib_dir.mkdir(parents=True, exist_ok=True)
         chrome_path = active_lib_dir / "chromium"
         node_path = active_lib_dir / "node"
@@ -239,7 +239,7 @@ class TestMachineModel:
 
         refreshed = Machine.current(refresh=True)
 
-        # Valid binary paths inside LIB_DIR survive.
+        # Valid binary paths inside ABXPKG_LIB_DIR survive.
         assert refreshed.config.get("CHROME_BINARY") == str(chrome_path)
         assert refreshed.config.get("NODE_BINARY") == str(node_path)
         # Non-binary mirror keys survive — they belong to ArchiveBox.conf.
@@ -247,7 +247,7 @@ class TestMachineModel:
         assert refreshed.config.get("CHROME_ISOLATION") == "snapshot"
         assert refreshed.config.get("CHROME_USER_DATA_DIR") == "/tmp/profile"
         assert refreshed.config.get("CHROMIUM_VERSION") == "123.4.5"
-        # Stale binary paths get dropped: YTDLP_BINARY outside LIB_DIR,
+        # Stale binary paths get dropped: YTDLP_BINARY outside ABXPKG_LIB_DIR,
         # WGET_BINARY path doesn't exist.
         assert "YTDLP_BINARY" not in refreshed.config
         assert "WGET_BINARY" not in refreshed.config
@@ -262,7 +262,7 @@ class TestMachineModel:
         import archivebox.machine.models as models
         from archivebox.config.common import get_config
 
-        lib_dir = get_config(include_machine=False).LIB_DIR
+        lib_dir = get_config(include_machine=False).ABXPKG_LIB_DIR
         chrome_path = lib_dir / "chromium"
         chrome_path.parent.mkdir(parents=True, exist_ok=True)
         chrome_path.write_text("#!/bin/sh\n")

archivebox/tests/test_machine_service.py

@@ -37,7 +37,6 @@ def _runtime_env(data_dir: Path, bin_dir: Path) -> dict[str, str]:
         os.environ.get("PATH", ""),
     ]
     return {
-        "LIB_DIR": str(data_dir / "lib"),
         "ABXPKG_LIB_DIR": str(data_dir / "lib"),
         "LITEPARSE_ENABLED": "True",
         "TIMEOUT": "180",

archivebox/tests/test_persona_runtime.py

@@ -38,13 +38,15 @@ def test_persona_prepare_runtime_for_crawl_clones_and_cleans_profile(initialized
         )
 
         runtime_root = persona.runtime_root_for_crawl(crawl)
-        runtime_profile = Path(overrides['CHROME_USER_DATA_DIR'])
-        runtime_downloads = Path(overrides['CHROME_DOWNLOADS_DIR'])
+        runtime_profile = persona.runtime_profile_dir_for_crawl(crawl)
+        runtime_downloads = persona.runtime_downloads_dir_for_crawl(crawl)
 
         print(json.dumps({
             'runtime_root_exists': runtime_root.exists(),
             'runtime_profile_exists': runtime_profile.exists(),
             'runtime_downloads_exists': runtime_downloads.exists(),
+            'runtime_personas_dir': overrides['PERSONAS_DIR'],
+            'active_persona': overrides['ACTIVE_PERSONA'],
             'preferences_copied': (runtime_profile / 'Default' / 'Preferences').exists(),
             'singleton_removed': not (runtime_profile / 'SingletonLock').exists(),
             'cache_removed': not (runtime_profile / 'Default' / 'GPUCache').exists(),
@@ -63,6 +65,8 @@ def test_persona_prepare_runtime_for_crawl_clones_and_cleans_profile(initialized
     assert payload["runtime_root_exists"] is True
     assert payload["runtime_profile_exists"] is True
     assert payload["runtime_downloads_exists"] is True
+    assert payload["runtime_personas_dir"].endswith("/.persona")
+    assert payload["active_persona"] == "Default"
     assert payload["preferences_copied"] is True
     assert payload["singleton_removed"] is True
     assert payload["cache_removed"] is True

archivebox/tests/test_recursive_crawl.py

@@ -427,12 +427,11 @@ def test_add_archivewebpage_installs_required_chrome_dependency(initialized_arch
             "SHOW_PROGRESS": "false",
             "TIMEOUT": "120",
             "ABXPKG_INSTALL_TIMEOUT": "900",
-            "LIB_DIR": str(initialized_archive / "lib"),
             "ABXPKG_LIB_DIR": str(initialized_archive / "lib"),
             "CHROME_HEADLESS": "true",
             "CHROME_SANDBOX": "false",
             "CHROME_ISOLATION": "snapshot",
-            "CHROME_EXTENSIONS_DIR": str(initialized_archive / "lib/chromewebstore/extensions"),
+            "CHROMEWEBSTORE_EXTENSIONS_DIR": str(initialized_archive / "lib/chromewebstore/extensions"),
         },
     )
 
@@ -506,7 +505,7 @@ def test_add_archivewebpage_installs_required_chrome_dependency(initialized_arch
         if process_type == Process.TypeChoices.HOOK and "chrome_launch" in str(cmd)
     ]
     assert chrome_hook_envs
-    assert all("{LIB_DIR}" not in str(env) for env in chrome_hook_envs)
+    assert all("{ABXPKG_LIB_DIR}" not in str(env) for env in chrome_hook_envs)
     assert any(process_type == Process.TypeChoices.BINARY for process_type, _status, _exit_code, _cmd, _env in process_rows)
     assert all(
         status == Process.StatusChoices.EXITED and exit_code == 0
@@ -531,7 +530,6 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(initia
         {
             "USE_COLOR": "false",
             "SHOW_PROGRESS": "false",
-            "LIB_DIR": str(initialized_archive / "lib"),
             "ABXPKG_LIB_DIR": str(initialized_archive / "lib"),
             "TIMEOUT": "90",
             "ABXPKG_INSTALL_TIMEOUT": "900",

archivebox/tests/test_search.py

@@ -479,6 +479,15 @@ class TestPublicIndexSearch:
         assert response.status_code == 200
         assert b"archivebox-search-stream-status" in response.content
 
+    @override_settings(PUBLIC_INDEX=True)
+    def test_public_search_stream_preserves_search_form_dom(self, client, public_snapshot):
+        response = client.get("/public/", {"q": "Public Example"}, HTTP_HOST=WEB_HOST)
+
+        assert response.status_code == 200
+        assert b"replaceRegionFromDocument(doc, '.public-snapshot-count')" in response.content
+        assert b"replaceRegionFromDocument(doc, '#table-bookmarks tbody')" in response.content
+        assert b"currentList.replaceWith(nextList)" not in response.content
+
     @override_settings(PUBLIC_INDEX=True)
     def test_public_search_stream_populates_public_results_cache(self, client, public_snapshot):
         search_params = {"q": "Public Example", "search_mode": "meta"}
@@ -562,6 +571,47 @@ class TestPublicIndexSearch:
         assert "last »" not in last_content
         assert "private-page-test" not in last_content
 
+    @override_settings(PUBLIC_INDEX=True)
+    def test_public_index_preview_respects_root_relative_screenshot_output(self, client, public_snapshot):
+        from archivebox.core.models import ArchiveResult
+
+        ArchiveResult.objects.create(
+            snapshot=public_snapshot,
+            plugin="screenshot",
+            status=ArchiveResult.StatusChoices.SUCCEEDED,
+            output_files={
+                "screenshot.png": {"size": 3, "root_relative": True},
+            },
+            output_str="screenshot.png",
+        )
+
+        response = client.get("/public/", HTTP_HOST=WEB_HOST)
+
+        assert response.status_code == 200
+        content = response.content.decode()
+        assert "/screenshot.png" in content
+        assert "screenshot/screenshot.png" not in content
+
+    @override_settings(PUBLIC_INDEX=True)
+    def test_public_index_preview_respects_plugin_relative_screenshot_output(self, client, public_snapshot):
+        from archivebox.core.models import ArchiveResult
+
+        ArchiveResult.objects.create(
+            snapshot=public_snapshot,
+            plugin="screenshot",
+            status=ArchiveResult.StatusChoices.SUCCEEDED,
+            output_files={
+                "screenshot.png": {"size": 3},
+            },
+            output_str="screenshot.png",
+        )
+
+        response = client.get("/public/", HTTP_HOST=WEB_HOST)
+
+        assert response.status_code == 200
+        content = response.content.decode()
+        assert "screenshot/screenshot.png" in content
+
     @override_settings(PUBLIC_INDEX=True)
     def test_public_index_preview_falls_back_to_extension_screenshots(self, client, public_snapshot):
         from archivebox.core.models import ArchiveResult
@@ -581,12 +631,21 @@ class TestPublicIndexSearch:
 
         assert response.status_code == 200
         content = response.content.decode()
-        assert "screenshot/screenshot.png" in content
         first = content.index("chrome_extension_screenshot/screenshot-1.png")
         second = content.index("chrome_extension_screenshot/screenshot.png")
         assert first < second
+        assert "screenshot/screenshot.png" not in content
         assert "chrome_extension_screenshot/screenshot-2.png" not in content
 
+    @override_settings(PUBLIC_INDEX=True)
+    def test_public_index_snapshot_without_preview_renders_placeholder(self, client, public_snapshot):
+        response = client.get("/public/", HTTP_HOST=WEB_HOST)
+
+        assert response.status_code == 200
+        content = response.content.decode()
+        assert "snapshot-preview-empty" in content
+        assert "screenshot/screenshot.png" not in content
+
     @override_settings(PUBLIC_INDEX=True)
     def test_public_index_pending_snapshot_uses_small_preview_spinner(self, client, crawl):
         from archivebox.core.models import Snapshot

archivebox/tests/test_server_security_browser.py

@@ -297,7 +297,6 @@ def browser_runtime(initialized_archive: Path):
     env = cli_env(
         ABXPKG_INSTALL_TIMEOUT="900",
         ABXPKG_MIN_RELEASE_AGE="0",
-        LIB_DIR=str(shared_lib),
         ABXPKG_LIB_DIR=str(shared_lib),
         CHROME_HEADLESS="True",
         CHROME_SANDBOX="False",
@@ -776,11 +775,10 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
         ARCHIVEWEBPAGE_TIMEOUT="90",
         TIMEOUT="90",
     )
-    env["LIB_DIR"] = str(browser_runtime["lib_dir"])
     env["ABXPKG_LIB_DIR"] = str(browser_runtime["lib_dir"])
     env["NODE_PATH"] = str(browser_runtime["node_modules_dir"])
     env["NODE_MODULES_DIR"] = str(browser_runtime["node_modules_dir"])
-    env["CHROME_EXTENSIONS_DIR"] = str(
+    env["CHROMEWEBSTORE_EXTENSIONS_DIR"] = str(
         browser_runtime["lib_dir"] / "chromewebstore" / "extensions",
     )
 

archivebox/tests/test_takeover_util.py

@@ -39,8 +39,7 @@ pytestmark = pytest.mark.django_db(transaction=True)
 
 
 def _require_sonic_binary() -> None:
-    if shutil.which("sonic") is None:
-        pytest.skip("sonic server binary is required for Sonic worker takeover tests")
+    assert shutil.which("sonic") is not None, "sonic server binary is required for Sonic worker takeover tests"
 
 
 def _archive_pages_for_sqlite_reindexing(data_dir: Path, env: dict[str, str], root_url: str) -> None:

archivebox/tests/test_ui_add_view.py

@@ -99,6 +99,61 @@ def test_add_view_admin_renders_plugin_config_grid(client, admin_user, monkeypat
     assert b"plugin_config__wget__WGET_ENABLED" not in response.content
 
 
+def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
+    staff_user = User.objects.create_user(
+        username="addviewstaff",
+        email="addviewstaff@test.com",
+        password="testpassword",
+        is_staff=True,
+        is_superuser=False,
+    )
+    client.force_login(staff_user)
+
+    response = client.get(reverse("add"), HTTP_HOST=ADMIN_HOST)
+    assert response.status_code == 200
+    assert response.context["can_override_crawl_config"] is False
+    assert response.context["form"].plugin_groups == []
+    assert b"plugin_config__wget__WGET_TIMEOUT" not in response.content
+    assert b"Custom config overrides" not in response.content
+
+    response = client.post(
+        reverse("add"),
+        data={
+            "url": "https://example.com/staff-config",
+            "tag": "",
+            "depth": "0",
+            "max_urls": "0",
+            "crawl_max_size": "0",
+            "crawl_timeout": "0",
+            "timeout": "60",
+            "snapshot_max_size": "0",
+            "delete_after": "0",
+            "crawl_max_concurrent_snapshots": "1",
+            "url_filters_allowlist": "",
+            "url_filters_denylist": "",
+            "notes": "",
+            "schedule": "daily",
+            "persona": "Default",
+            "permissions": "public",
+            "start_paused": "on",
+            "main_plugins": ["wget"],
+            "plugin_config__wget__WGET_TIMEOUT": "77",
+            "config": '{"WGET_TIMEOUT": 77, "YTDLP_ARGS_EXTRA": ["--exec", "touch /tmp/owned"]}',
+        },
+        HTTP_HOST=ADMIN_HOST,
+    )
+
+    assert response.status_code == 302
+    crawl = Crawl.objects.order_by("-created_at").first()
+    assert crawl is not None
+    assert crawl.created_by == staff_user
+    assert crawl.status == Crawl.StatusChoices.QUEUED
+    assert crawl.schedule is None
+    assert crawl.config.get("PLUGINS", "") == ""
+    assert crawl.config.get("WGET_TIMEOUT") != 77
+    assert crawl.config.get("YTDLP_ARGS_EXTRA") != ["--exec", "touch /tmp/owned"]
+
+
 def test_add_view_embeds_selected_persona_config_for_ui_hydration(client, admin_user, monkeypatch):
     monkeypatch.setenv("PUBLIC_ADD_VIEW", "true")
     client.force_login(admin_user)

archivebox/tests/test_ui_admin_snapshot.py

@@ -71,6 +71,72 @@ def test_snapshot_admin_archive_results_escape_extractor_output(admin_client, sn
     assert b"<img src=x onerror="window.__archivebox_archiveresult_xss__=1">" in body
 
 
+def test_snapshot_admin_archive_result_table_escapes_legacy_string_fields(admin_client, snapshot):
+    from uuid import uuid4
+
+    from archivebox.core.models import ArchiveResult
+    from archivebox.machine.models import Binary, Machine, Process
+
+    machine = Machine.objects.create(
+        guid=f"xss-machine-{uuid4()}",
+        hostname='<script id="machine-xss">x</script>',
+        hw_in_docker=False,
+        hw_in_vm=False,
+        hw_manufacturer="Test",
+        hw_product="Test Product",
+        hw_uuid=f"xss-hw-{uuid4()}",
+        os_arch="arm64",
+        os_family="darwin",
+        os_platform="macOS",
+        os_release="14.0",
+        os_kernel="Darwin",
+        stats={},
+        config={},
+    )
+    binary = Binary.objects.create(
+        machine=machine,
+        name="staticfile",
+        abspath="/usr/bin/staticfile",
+        version='<b id="version-xss">v</b>',
+        binprovider="env",
+        binproviders="env",
+        status=Binary.StatusChoices.INSTALLED,
+    )
+    process = Process.objects.create(
+        machine=machine,
+        binary=binary,
+        process_type=Process.TypeChoices.HOOK,
+        pwd='/tmp/archivebox"><script id="pwd-xss">x</script>',
+        cmd=["staticfile"],
+        status=Process.StatusChoices.EXITED,
+    )
+    result = ArchiveResult.objects.create(
+        snapshot=snapshot,
+        plugin="staticfile",
+        hook_name="on_Snapshot__00_staticfile.py",
+        process=process,
+        status=ArchiveResult.StatusChoices.SUCCEEDED,
+        output_files={'evil"><script id="file-xss">x</script>.txt': {"size": 12, "mimetype": "text/plain"}},
+        output_str='staticfile/evil"><script id="file-xss">x</script>.txt',
+    )
+    ArchiveResult.objects.filter(pk=result.pk).update(plugin='<script id="plugin-xss">x</script>')
+
+    response = admin_client.get(reverse("admin:core_snapshot_change", args=[snapshot.pk]), HTTP_HOST=ADMIN_TEST_HOST)
+    body = response.content
+
+    assert response.status_code == 200
+    assert b'<script id="machine-xss">' not in body
+    assert b'<script id="version-xss">' not in body
+    assert b'<script id="pwd-xss">' not in body
+    assert b'<script id="file-xss">' not in body
+    assert b'<script id="plugin-xss">' not in body
+    assert b"&lt;script id=&quot;machine-xss&quot;&gt;" in body
+    assert b"&lt;b id=&quot;version-xss&quot;&gt;v&lt;/b&gt;" in body
+    assert b"&lt;script id=&quot;pwd-xss&quot;&gt;" in body
+    assert b"&lt;script id=&quot;file-xss&quot;&gt;" in body
+    assert b"&lt;script id=&quot;plugin-xss&quot;&gt;" in body
+
+
 def test_snapshot_changelist_bulk_permissions_action_updates_selected_snapshots(client, admin_user, crawl, snapshot):
     client.force_login(admin_user)
     url = reverse("admin:core_snapshot_changelist")

archivebox/tests/test_ui_public_snapshot.py

@@ -282,7 +282,7 @@ class TestPublicIndex:
 
     @override_settings(PUBLIC_INDEX=True)
     def test_public_snapshot_surfaces_escape_legacy_raw_title_and_tag_values(self, client, admin_user):
-        from archivebox.core.models import Snapshot, Tag
+        from archivebox.core.models import ArchiveResult, Snapshot, Tag
         from archivebox.crawls.models import Crawl
 
         crawl = Crawl.objects.create(urls="https://public-xss.example", created_by=admin_user, config={"PERMISSIONS": "public"})
@@ -296,11 +296,19 @@ class TestPublicIndex:
         snapshot.tags.add(tag)
         snapshot_archive_path = snapshot.archive_path
 
-        title_payload = "</script><script id=public-title-xss>window.__archivebox_public_title_xss__=1</script>"
+        title_payload = "Legacy title 1 < 2 & 3 </script><script id=public-title-xss>window.__archivebox_public_title_xss__=1</script>"
         tag_payload = "</script><script id=public-tag-xss>window.__archivebox_public_tag_xss__=1</script>"
         url_payload = "https://public-xss.example/</script><script id=public-url-xss>window.__archivebox_public_url_xss__=1</script>"
+        filename_payload = 'evil"><script id=public-file-xss>window.__archivebox_public_file_xss__=1</script>.txt'
         Snapshot.objects.filter(pk=snapshot.pk).update(title=title_payload, url=url_payload)
         Tag.objects.filter(pk=tag.pk).update(name=tag_payload)
+        ArchiveResult.objects.create(
+            snapshot=snapshot,
+            plugin="staticfile",
+            hook_name="on_Snapshot__00_staticfile.py",
+            status=ArchiveResult.StatusChoices.SUCCEEDED,
+            output_files={filename_payload: {"size": 12, "mimetype": "text/plain"}},
+        )
 
         public_index = client.get("/public/", HTTP_HOST=WEB_TEST_HOST)
         snapshot_detail = client.get(f"/{snapshot_archive_path}/index.html", HTTP_HOST=WEB_TEST_HOST)
@@ -311,9 +319,13 @@ class TestPublicIndex:
             assert b"<script id=public-title-xss>" not in response.content
             assert b"<script id=public-tag-xss>" not in response.content
             assert b"<script id=public-url-xss>" not in response.content
+            assert b"<script id=public-file-xss>" not in response.content
             assert b"&lt;/script&gt;&lt;script id=public-tag-xss&gt;" in response.content
         assert b"&lt;/script&gt;&lt;script id=public-title-xss&gt;" in public_index.content
         assert b"window.__archivebox_public_title_xss__=1" in snapshot_detail.content
+        assert b"Legacy title 1 &lt; 2 &amp; 3" in snapshot_detail.content
+        assert b"Legacy title 1 &amp;lt; 2 &amp;amp; 3" not in snapshot_detail.content
+        assert b"&lt;script id=public-file-xss&gt;" in snapshot_detail.content
 
     def test_direct_snapshot_urls_allow_unlisted_but_not_private_for_guests(self, client, admin_user):
         from archivebox.core.models import Snapshot

archivebox/tests/test_urls.py

@@ -332,6 +332,57 @@ class TestUrlRouting:
             """,
         )
 
+    def test_admin_login_next_allows_archivebox_hosts_only(self) -> None:
+        self._run(
+            """
+            ensure_admin_user()
+            snapshot = get_snapshot()
+            admin_host = get_admin_host()
+            web_host = get_web_host()
+            api_host = get_api_host()
+            snapshot_id = str(snapshot.id)
+
+            allowed_next_urls = [
+                "/admin/core/snapshot/",
+                f"http://archivebox.localhost:8000/public/",
+                f"http://{web_host}/public/",
+                f"http://{admin_host}/admin/core/snapshot/",
+                f"http://{api_host}/api/v1/docs",
+                build_snapshot_url(snapshot_id, "index.html"),
+                build_original_url(snapshot.domain, "index.html"),
+            ]
+            for next_url in allowed_next_urls:
+                client = Client()
+                resp = client.post(
+                    "/admin/login/",
+                    data={"username": "testadmin", "password": "testpassword", "next": next_url},
+                    HTTP_HOST=admin_host,
+                )
+                assert resp.status_code in (301, 302), (next_url, resp.status_code, response_body(resp)[:500])
+                assert resp["Location"] == next_url, (next_url, resp["Location"])
+
+            blocked_next_urls = [
+                "https://anything.attacker.com/admin/",
+                "//anything.attacker.com/admin/",
+                "https://archivebox.localhost.attacker.com/public/",
+                "http://web.archivebox.localhost:9999/public/",
+                "javascript:alert(1)",
+            ]
+            for next_url in blocked_next_urls:
+                client = Client()
+                resp = client.post(
+                    "/admin/login/",
+                    data={"username": "testadmin", "password": "testpassword", "next": next_url},
+                    HTTP_HOST=admin_host,
+                )
+                assert resp.status_code in (301, 302), (next_url, resp.status_code, response_body(resp)[:500])
+                assert resp["Location"] == "/admin/", (next_url, resp["Location"])
+                assert "attacker.com" not in resp["Location"]
+
+            print("OK")
+            """,
+        )
+
     def test_snapshot_routing_and_hosts(self) -> None:
         self._run(
             """
@@ -743,14 +794,14 @@ class TestUrlRouting:
             self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
 
     def test_subdomain_replay_assets_route_without_base_url(self) -> None:
-        chrome_extensions_dir = self.data_dir / "test-chrome-extensions"
+        lib_dir = self.data_dir / "test-lib"
         try:
             self._set_config("BIND_ADDR=127.0.0.1:8766", "BASE_URL=")
             self._run(
                 """
                 snapshot = get_snapshot()
                 snapshot_host = get_snapshot_host(str(snapshot.id))
-                extension_dir = Path(SERVER_CONFIG.CHROME_EXTENSIONS_DIR) / "test__archivewebpage"
+                extension_dir = Path(SERVER_CONFIG.ABXPKG_LIB_DIR) / "chromewebstore" / "extensions" / "test__archivewebpage"
                 extension_dir.mkdir(parents=True, exist_ok=True)
                 (extension_dir / "ui.js").write_text("window.__archivebox_replay_ui__ = true;\\n", encoding="utf-8")
                 (extension_dir / "sw.js").write_text("self.__archivebox_replay_sw__ = true;\\n", encoding="utf-8")
@@ -766,7 +817,7 @@ class TestUrlRouting:
                 print("OK")
                 """,
                 mode="safe-subdomains-fullreplay",
-                env_overrides={"CHROME_EXTENSIONS_DIR": str(chrome_extensions_dir)},
+                env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
             )
         finally:
             self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
@@ -779,10 +830,9 @@ class TestUrlRouting:
                 """
                 snapshot = get_snapshot()
                 snapshot_host = get_snapshot_host(str(snapshot.id))
-                expected_extensions_dir = Path(SERVER_CONFIG.LIB_DIR) / "chromewebstore" / "extensions"
-                assert Path(SERVER_CONFIG.CHROME_EXTENSIONS_DIR).resolve() == expected_extensions_dir.resolve()
+                expected_extensions_dir = Path(SERVER_CONFIG.ABXPKG_LIB_DIR) / "chromewebstore" / "extensions"
 
-                extension_dir = Path(SERVER_CONFIG.CHROME_EXTENSIONS_DIR) / "test__archivewebpage"
+                extension_dir = expected_extensions_dir / "test__archivewebpage"
                 extension_dir.mkdir(parents=True, exist_ok=True)
                 (extension_dir / "ui.js").write_text("window.__archivebox_replay_ui_from_lib__ = true;\\n", encoding="utf-8")
                 (extension_dir / "sw.js").write_text("self.__archivebox_replay_sw_from_lib__ = true;\\n", encoding="utf-8")
@@ -798,7 +848,7 @@ class TestUrlRouting:
                 print("OK")
                 """,
                 mode="safe-subdomains-fullreplay",
-                env_overrides={"LIB_DIR": str(lib_dir)},
+                env_overrides={"ABXPKG_LIB_DIR": str(lib_dir)},
             )
         finally:
             self._set_config("BIND_ADDR=127.0.0.1:8000", "BASE_URL=http://archivebox.localhost:8000")
@@ -998,6 +1048,27 @@ class TestUrlRouting:
             assert resp.status_code == 200
             assert resp["Content-Type"].startswith("text/html")
 
+            root_screenshot = bytes.fromhex(
+                "89504e470d0a1a0a"
+                "0000000d49484452000000010000000108060000001f15c489"
+                "0000000d49444154789c63606060600000050001a5f64540"
+                "0000000049454e44ae426082"
+            )
+            (Path(snapshot.output_dir) / "screenshot.png").write_bytes(root_screenshot)
+            ArchiveResult.objects.update_or_create(
+                snapshot=snapshot,
+                plugin="screenshot",
+                defaults={
+                    "status": ArchiveResult.StatusChoices.SUCCEEDED,
+                    "output_files": {"screenshot.png": {"size": len(root_screenshot), "root_relative": True}},
+                    "output_str": "screenshot.png",
+                },
+            )
+            resp = client.get("/screenshot/screenshot.png", HTTP_HOST=snapshot_host)
+            assert resp.status_code == 200
+            assert resp["Content-Type"].startswith("image/png")
+            assert response_body(resp) == root_screenshot
+
             mhtml_dir = Path(snapshot.output_dir) / "chrome_mhtml"
             mhtml_dir.mkdir(parents=True, exist_ok=True)
             (mhtml_dir / "snapshot.mhtml").write_text(

bin/build_docker.sh

@@ -22,6 +22,23 @@ declare -a TAG_NAMES=("$@")
 BRANCH_NAME="${1:-$(git rev-parse --abbrev-ref HEAD)}"
 VERSION="$(grep '^version = ' "${REPO_DIR}/pyproject.toml" | awk -F'"' '{print $2}')"
 GIT_SHA=sha-"$(git rev-parse --short HEAD)"
+ABX_DL_VERSION="$(python3 - <<'PY'
+import re
+import tomllib
+
+with open("pyproject.toml", "rb") as f:
+    deps = tomllib.load(f)["project"]["dependencies"]
+
+for dep in deps:
+    match = re.match(r"abx-dl\s*(?:==|>=)\s*([^,;\s]+)", dep)
+    if match:
+        print(match.group(1))
+        break
+else:
+    raise SystemExit("Missing abx-dl dependency in pyproject.toml")
+PY
+)"
+ABX_DL_IMAGE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
 NATIVE_ARCH="$(docker info --format '{{.Architecture}}' 2>/dev/null || uname -m)"
 case "$NATIVE_ARCH" in
     x86_64|amd64) NATIVE_PLATFORM="linux/amd64" ;;
@@ -41,7 +58,7 @@ if ! echo "${TAG_NAMES[@]}" | grep -q "$VERSION"; then
     TAG_NAMES+=("$VERSION")
 fi
 
-echo "[+] Building Docker image for $SELECTED_PLATFORMS: branch=$BRANCH_NAME version=$VERSION tags=${TAG_NAMES[*]}"
+echo "[+] Building Docker image for $SELECTED_PLATFORMS: branch=$BRANCH_NAME version=$VERSION abx_dl_image=$ABX_DL_IMAGE tags=${TAG_NAMES[*]}"
 
 declare -a FULL_TAG_NAMES
 # for each tag in TAG_NAMES, add archivebox/archivebox:tag and its mirrors to FULL_TAG_NAMES
@@ -97,6 +114,7 @@ check_platforms || (recreate_builder && check_platforms) || exit 1
 
 echo "[+] Building archivebox:$VERSION docker image..."
 mkdir -p "$HOME/.cache/docker/archivebox"
+docker buildx imagetools inspect "$ABX_DL_IMAGE"
 # docker builder prune
 # docker build . --no-cache -t archivebox-dev \
 # replace --load with --push to deploy
@@ -107,6 +125,8 @@ if [[ "$SELECTED_PLATFORMS" == *,* ]]; then
 fi
 docker buildx build \
     --platform "$SELECTED_PLATFORMS" \
+    --pull \
+    --build-arg "ABX_DL_IMAGE=$ABX_DL_IMAGE" \
     --cache-from type=local,src="$HOME/.cache/docker/archivebox" \
     --cache-to type=local,compression=zstd,mode=min,oci-mediatypes=true,dest="$HOME/.cache/docker/archivebox" \
     --load . ${FULL_TAG_NAMES[@]}

bin/docker_entrypoint.sh

@@ -30,12 +30,11 @@ ulimit -c 0 >/dev/null 2>&1 || true
 export DATA_DIR="${DATA_DIR:-/data}"
 export CONFIG_DIR="${CONFIG_DIR:-/opt/archivebox}"
 export TMP_DIR="${TMP_DIR:-/tmp/archivebox}"
-export LIB_DIR="${LIB_DIR:-/opt/archivebox/lib}"
-export ABXPKG_LIB_DIR="${ABXPKG_LIB_DIR:-$LIB_DIR}"
+export ABXPKG_LIB_DIR="${ABXPKG_LIB_DIR:-/opt/archivebox/lib}"
 export ARCHIVEBOX_USER="${ARCHIVEBOX_USER:-archivebox}"
 export PERSONAS_DIR="${PERSONAS_DIR:-$DATA_DIR/personas}"
-export PLAYWRIGHT_BROWSERS_PATH="${PLAYWRIGHT_BROWSERS_PATH:-$LIB_DIR/playwright/cache}"
-export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$LIB_DIR/cache}"
+export PLAYWRIGHT_BROWSERS_PATH="${PLAYWRIGHT_BROWSERS_PATH:-$ABXPKG_LIB_DIR/playwright/cache}"
+export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$ABXPKG_LIB_DIR/cache}"
 export ABXBUS_CACHE_DIR="${ABXBUS_CACHE_DIR:-$XDG_CACHE_HOME/abxbus}"
 export UV_CACHE_DIR="${UV_CACHE_DIR:-$XDG_CACHE_HOME/uv}"
 
@@ -213,7 +212,7 @@ ensure_small_runtime_tree "$ABXBUS_CACHE_DIR/semaphores"
 ensure_small_runtime_tree "$UV_CACHE_DIR"
 ensure_runtime_tree "$PLAYWRIGHT_BROWSERS_PATH"
 ensure_runtime_tmp_tree
-ensure_runtime_tree "$LIB_DIR"
+ensure_runtime_tree "$ABXPKG_LIB_DIR"
 run_as_archivebox touch "$ABXBUS_CACHE_DIR/semaphores/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$ABXBUS_CACHE_DIR/semaphores"
 rm -f "$ABXBUS_CACHE_DIR/semaphores/.permissions_test_safe_to_delete"
 run_as_archivebox touch "$UV_CACHE_DIR/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$UV_CACHE_DIR"

bin/release.sh

@@ -221,6 +221,26 @@ else:
 PY
 }
 
+latest_pypi_version() {
+    local package_name="$1"
+    local releases_json
+    releases_json="$(curl -fsSL "https://pypi.org/pypi/${package_name}/json" | jq -r '.releases | keys[]' || true)"
+    RELEASE_TAGS="${releases_json}" python3 - <<'PY'
+import os
+import re
+
+def parse(version: str) -> tuple[int, int, int, int, int]:
+    match = re.fullmatch(r'(\d+)\.(\d+)\.(\d+)(?:-?rc(\d*))?$', version)
+    if not match:
+        return (-1, -1, -1, -1, -1)
+    major, minor, patch, rc = match.groups()
+    return (int(major), int(minor), int(patch), 0 if 'rc' in version else 1, int(rc or '0'))
+
+versions = [line.strip() for line in os.environ.get('RELEASE_TAGS', '').splitlines() if line.strip()]
+print(max(versions, key=parse) if versions else '')
+PY
+}
+
 github_release_enabled() {
     local version="$1"
     if [[ "${version}" == *rc* && "${CREATE_GITHUB_RC_RELEASES:-0}" != "1" ]]; then
@@ -384,7 +404,6 @@ create_release() {
 
 publish_artifacts() {
     local version="$1"
-    local pypi_token="${UV_PUBLISH_TOKEN:-${PYPI_TOKEN:-${PYPI_PAT_SECRET:-}}}"
     local artifact_prefix="${PYPI_PACKAGE//-/_}"
     local artifacts=()
     local dist_dir
@@ -406,28 +425,19 @@ publish_artifacts() {
             return 1
         fi
 
-        if [[ -n "${pypi_token}" ]]; then
-            UV_PUBLISH_TOKEN="${pypi_token}" uv publish --username=__token__ "${artifacts[@]}"
-        else
-            uv publish --username=__token__ "${artifacts[@]}"
-        fi
+        uv publish --trusted-publishing always "${artifacts[@]}"
     fi
 
     wait_for_pypi "${PYPI_PACKAGE}" "${version}"
 }
 
 main() {
-    local slug branch version latest relation
+    local slug branch version latest pypi_latest relation
 
     source_optional_env
     slug="$(repo_slug)"
     branch="$(default_branch)"
 
-    if [[ "${GITHUB_EVENT_NAME:-}" == "push" ]]; then
-        validate_release_state "${slug}" "${branch}"
-        return 0
-    fi
-
     if [[ "$(git branch --show-current)" != "${branch}" ]]; then
         echo "Release must run from ${branch}, found $(git branch --show-current)" >&2
         return 1
@@ -435,6 +445,10 @@ main() {
 
     version="$(current_version)"
     latest="$(latest_release_version "${slug}")"
+    pypi_latest="$(latest_pypi_version "${PYPI_PACKAGE}")"
+    if [[ -n "${pypi_latest}" && ( -z "${latest}" || "$(compare_versions "${pypi_latest}" "${latest}")" == "gt" ) ]]; then
+        latest="${pypi_latest}"
+    fi
     if [[ -z "${latest}" ]]; then
         relation="gt"
     else
@@ -449,16 +463,12 @@ main() {
         git add -A
         git commit -m "release: ${TAG_PREFIX}${version}"
         git push origin "${branch}"
-
-        wait_for_runs "${slug}" push "$(git rev-parse HEAD)" "push"
     elif [[ "${relation}" == "gt" ]]; then
         run_checks
         if [[ -n "$(git status --short)" ]]; then
             git add -A
             git commit -m "release: ${TAG_PREFIX}${version}"
             git push origin "${branch}"
-
-            wait_for_runs "${slug}" push "$(git rev-parse HEAD)" "push"
         fi
     else
         echo "Current version ${version} is behind latest GitHub release ${latest}" >&2

bin/release_dev_stack.sh

@@ -5,8 +5,8 @@ IFS=$'\n\t'
 
 ARCHIVEBOX_REPO="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
 WORKSPACE_DIR="$(cd "${ARCHIVEBOX_REPO}/.." && pwd)"
-PYPI_USERNAME="${PYPI_USERNAME:-__token__}"
 PYPI_WAIT_ATTEMPTS="${PYPI_WAIT_ATTEMPTS:-90}"
+DOCKER_IMAGE_REPOS="${DOCKER_IMAGE_REPOS:-archivebox/archivebox ghcr.io/archivebox/archivebox}"
 
 cd "${WORKSPACE_DIR}"
 
@@ -121,7 +121,7 @@ build_and_prek() {
     local repo="$1"
     (
         cd "$repo"
-        rm -rf dist
+        rm -rf dist .pdm-build
         uv --no-cache build --out-dir dist
         # prek auto-fixes (ruff-format, add-trailing-comma, end-of-file-fixer,
         # …) exit with status 1 whenever they modify files. Some hooks expose
@@ -138,7 +138,7 @@ build_and_prek() {
             fi
             echo "[*] prek auto-fixed files in $(basename "$repo"); re-running to verify clean…"
         done
-        rm -rf dist
+        rm -rf dist .pdm-build
         uv --no-cache build --out-dir dist
     )
 }
@@ -174,7 +174,7 @@ commit_push_publish() {
         if pypi_has_release "$package" "$version"; then
             echo "[*] ${package}==${version} is already on PyPI; skipping upload."
         else
-            uv --no-cache publish --username="${PYPI_USERNAME}" dist/*
+            uv --no-cache publish --trusted-publishing always dist/*
         fi
     )
 }
@@ -213,6 +213,23 @@ wait_for_pypi() {
     done
 }
 
+wait_for_docker_image() {
+    local image="$1"
+    local attempts=0
+    local max_attempts="${DOCKER_IMAGE_WAIT_ATTEMPTS:-60}"
+
+    until docker buildx imagetools inspect "$image" >/dev/null
+    do
+        attempts=$((attempts + 1))
+        if [[ "$attempts" -ge "$max_attempts" ]]; then
+            echo "[X] Timed out waiting for ${image}" >&2
+            exit 1
+        fi
+        echo "[*] ${image} is not published yet; waiting..."
+        sleep 30
+    done
+}
+
 release_python_repo() {
     local repo_name="$1"
     local branch="$2"
@@ -255,6 +272,7 @@ commit_push_publish "$ARCHIVEBOX_REPO" dev archivebox "$ARCHIVEBOX_VERSION"
 
 (
     cd "$ARCHIVEBOX_REPO"
+    wait_for_docker_image "archivebox/abx-dl:${ABX_SHARED_VERSION}"
     ./bin/release_docker.sh dev "$ARCHIVEBOX_VERSION" "sha-$(git rev-parse --short HEAD)"
     DEPLOY_IMAGE="${DOCKER_IMAGE_REPOS%% *}:dev" DEPLOY_EXPECT_VERSION="$ARCHIVEBOX_VERSION" SKIP_DOCKER=1 ./bin/deploy_dev_demo.sh
 )

bin/release_docker.sh

@@ -19,6 +19,23 @@ VERSION="$(grep '^version = ' "${REPO_DIR}/pyproject.toml" | awk -F'"' '{print $
 GIT_SHA=sha-"$(git rev-parse --short HEAD)"
 SELECTED_PLATFORMS="${DOCKER_PLATFORMS:-${SELECTED_PLATFORMS:-linux/amd64,linux/arm64}}"
 DOCKER_IMAGE_REPOS="${DOCKER_IMAGE_REPOS:-archivebox/archivebox ghcr.io/archivebox/archivebox}"
+ABX_DL_VERSION="$(python3 - <<'PY'
+import re
+import tomllib
+
+with open("pyproject.toml", "rb") as f:
+    deps = tomllib.load(f)["project"]["dependencies"]
+
+for dep in deps:
+    match = re.match(r"abx-dl\s*(?:==|>=)\s*([^,;\s]+)", dep)
+    if match:
+        print(match.group(1))
+        break
+else:
+    raise SystemExit("Missing abx-dl dependency in pyproject.toml")
+PY
+)"
+ABX_DL_IMAGE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
 
 # if not already in TAG_NAMES, add GIT_SHA and BRANCH_NAME  
 if ! echo "${TAG_NAMES[@]}" | grep -q "$GIT_SHA"; then
@@ -31,7 +48,7 @@ if ! echo "${TAG_NAMES[@]}" | grep -q "$VERSION"; then
    TAG_NAMES+=("$VERSION")
 fi
 
-echo "[+] Building + releasing Docker image for $SELECTED_PLATFORMS: branch=$BRANCH_NAME version=$VERSION tags=${TAG_NAMES[*]}"
+echo "[+] Building + releasing Docker image for $SELECTED_PLATFORMS: branch=$BRANCH_NAME version=$VERSION abx_dl_image=$ABX_DL_IMAGE tags=${TAG_NAMES[*]}"
 
 declare -a FULL_TAG_NAMES
 for TAG_NAME in "${TAG_NAMES[@]}"; do
@@ -85,11 +102,14 @@ check_platforms || (recreate_builder && check_platforms) || exit 1
 
 echo "[^] Uploading docker image"
 mkdir -p "$HOME/.cache/docker/archivebox"
+docker buildx imagetools inspect "$ABX_DL_IMAGE"
 
 # https://docs.docker.com/build/cache/backends/
 # shellcheck disable=SC2068
 docker buildx build \
    --platform "$SELECTED_PLATFORMS" \
+   --pull \
+   --build-arg "ABX_DL_IMAGE=$ABX_DL_IMAGE" \
    --cache-from type=local,src="$HOME/.cache/docker/archivebox" \
    --cache-to type=local,compression=zstd,mode=min,oci-mediatypes=true,dest="$HOME/.cache/docker/archivebox" \
    --push . ${FULL_TAG_NAMES[@]}   

bin/release_pip.sh

@@ -17,4 +17,4 @@ source "$REPO_DIR/.venv/bin/activate"
 echo "[^] Publishing to PyPI..."
 rm -Rf dist
 uv build
-uv publish
+uv publish --trusted-publishing always

bin/validate_docker_uid_gid.sh

@@ -87,7 +87,7 @@ docker_setup() {
             $setup_script"
 }
 
-default_cmd='printf "ABX_UID=%s\nABX_GID=%s\nABX_USER=%s\nABX_GROUPS=%s\n" "$(id -u)" "$(id -g)" "$(whoami 2>/dev/null || true)" "$(id -Gn)"; touch /data/logs/probe /data/archive/probe "$LIB_DIR/probe" "$PERSONAS_DIR/Default/chrome_profile/probe"; stat -c "ABX_STAT %u:%g:%a %n" /data /data/logs /data/archive "$LIB_DIR" "$PERSONAS_DIR" "$PERSONAS_DIR/Default" "$PERSONAS_DIR/Default/chrome_profile"; echo ABX_PERSONA_PROFILE_OK; echo ABX_OK'
+default_cmd='printf "ABX_UID=%s\nABX_GID=%s\nABX_USER=%s\nABX_GROUPS=%s\n" "$(id -u)" "$(id -g)" "$(whoami 2>/dev/null || true)" "$(id -Gn)"; touch /data/logs/probe /data/archive/probe "$ABXPKG_LIB_DIR/probe" "$PERSONAS_DIR/Default/chrome_profile/probe"; stat -c "ABX_STAT %u:%g:%a %n" /data /data/logs /data/archive "$ABXPKG_LIB_DIR" "$PERSONAS_DIR" "$PERSONAS_DIR/Default" "$PERSONAS_DIR/Default/chrome_profile"; echo ABX_PERSONA_PROFILE_OK; echo ABX_OK'
 version_cmd='printf "ABX_UID=%s\nABX_GID=%s\nABX_USER=%s\nABX_GROUPS=%s\n" "$(id -u)" "$(id -g)" "$(whoami 2>/dev/null || true)" "$(id -Gn)"; archivebox version >/tmp/archivebox-version.out; tail -n 12 /tmp/archivebox-version.out; echo ABX_OK'
 full_flow_cmd='set -Eeuo pipefail
 printf "ABX_UID=%s\nABX_GID=%s\nABX_USER=%s\nABX_GROUPS=%s\n" "$(id -u)" "$(id -g)" "$(whoami 2>/dev/null || true)" "$(id -Gn)"
@@ -163,7 +163,6 @@ run_case() {
     fi
     run_args+=(
         -e DATA_DIR=/data
-        -e LIB_DIR=/libdir
         -e ABXPKG_LIB_DIR=/libdir
         -e PLAYWRIGHT_BROWSERS_PATH=/browsers
     )
@@ -260,7 +259,6 @@ run_readonly_case() {
     fi
     run_args+=(
         -e DATA_DIR=/data
-        -e LIB_DIR=/libdir
         -e ABXPKG_LIB_DIR=/libdir
         -e PLAYWRIGHT_BROWSERS_PATH=/browsers
     )

bin/watch_dev_image.sh

@@ -0,0 +1,73 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+IFS=$'\n\t'
+
+DEPLOY_PATH="${DEPLOY_PATH:-/opt/archivebox.demo}"
+DEPLOY_SERVICE="${DEPLOY_SERVICE:-archivebox}"
+DEPLOY_IMAGE="${DEPLOY_IMAGE:-archivebox/archivebox:dev}"
+DEPLOY_INTERVAL="${DEPLOY_INTERVAL:-60}"
+STATE_FILE="${STATE_FILE:-${DEPLOY_PATH}/.archivebox-dev-image.digest}"
+OVERRIDE_FILE="${OVERRIDE_FILE:-${DEPLOY_PATH}/.archivebox-deploy.override.yml}"
+
+cd "$DEPLOY_PATH"
+
+if [[ -f compose.yml ]]; then
+    COMPOSE_FILE=compose.yml
+elif [[ -f compose.yaml ]]; then
+    COMPOSE_FILE=compose.yaml
+elif [[ -f docker-compose.yml ]]; then
+    COMPOSE_FILE=docker-compose.yml
+else
+    echo "[X] No compose file found in $DEPLOY_PATH" >&2
+    exit 1
+fi
+
+cat > "$OVERRIDE_FILE" <<EOF
+services:
+  $DEPLOY_SERVICE:
+    image: $DEPLOY_IMAGE
+EOF
+
+COMPOSE=(docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE")
+
+remote_digest() {
+    docker buildx imagetools inspect "$DEPLOY_IMAGE" | awk '/^Digest:/ {print $2; exit}'
+}
+
+deploy_digest() {
+    local digest="$1"
+
+    echo "[+] Deploying ${DEPLOY_IMAGE}@${digest}"
+    "${COMPOSE[@]}" pull "$DEPLOY_SERVICE"
+    "${COMPOSE[@]}" up -d "$DEPLOY_SERVICE"
+
+    if "${COMPOSE[@]}" config --services | grep -qx argo; then
+        "${COMPOSE[@]}" up -d argo
+    fi
+
+    "${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" archivebox version </dev/null | sed -n '1,40p'
+    "${COMPOSE[@]}" exec -T "$DEPLOY_SERVICE" curl -fsS --max-time 10 --connect-timeout 2 -H 'Host: admin.archivebox.io' http://127.0.0.1:8000/health/ </dev/null
+    printf '%s\n' "$digest" > "$STATE_FILE"
+}
+
+mkdir -p "$(dirname "$STATE_FILE")"
+touch "$STATE_FILE"
+
+while :; do
+    digest="$(remote_digest || true)"
+    last_digest="$(cat "$STATE_FILE" 2>/dev/null || true)"
+
+    if [[ -z "$digest" ]]; then
+        echo "[!] Could not resolve ${DEPLOY_IMAGE}; retrying in ${DEPLOY_INTERVAL}s" >&2
+    elif [[ "$digest" != "$last_digest" ]]; then
+        deploy_digest "$digest"
+    else
+        echo "[=] ${DEPLOY_IMAGE} already deployed at ${digest}"
+    fi
+
+    if [[ "${WATCH_ONCE:-0}" == "1" ]]; then
+        break
+    fi
+    sleep "$DEPLOY_INTERVAL"
+done

docs/Configuration.md

@@ -616,15 +616,15 @@ Path for temporary files, the supervisord unix socket, and generated supervisor
 If ArchiveBox detects the configured `TMP_DIR` is unwritable or too long, it will auto-fall-back to `/tmp/archivebox/<collection_id>` at startup.
 
 *Related options:*
-[`LIB_DIR`](#lib_dir), [`ALLOW_NO_UNIX_SOCKETS`](#allow_no_unix_sockets)
+[`ABXPKG_LIB_DIR`](#abxpkg_lib_dir), [`ALLOW_NO_UNIX_SOCKETS`](#allow_no_unix_sockets)
 
 ---
-#### `LIB_DIR`
+#### `ABXPKG_LIB_DIR`
 **Possible Values:** [`<user-config>/abx/lib`]/`/opt/archivebox/lib`/`~/.config/abx/lib`/...
 Path for installed binary dependencies (`chromium`, `single-file`, `yt-dlp`, `ripgrep`, etc.) managed by `abxpkg`. The default is the platform-standard user-config location (`~/.config/abx/lib` on Linux, `~/Library/Application Support/abx/lib` on macOS, `%APPDATA%\abx\lib` on Windows) so a single binary installation can be shared across multiple collections on the same machine without re-downloading.
 
 > [!NOTE]
-> `LIB_DIR` can grow to several GB. Put it on a fast local disk — running extractors off a network-mounted `LIB_DIR` will be painfully slow.
+> `ABXPKG_LIB_DIR` can grow to several GB. Put it on a fast local disk — running extractors off a network-mounted `ABXPKG_LIB_DIR` will be painfully slow.
 
 *Related options:*
 [`TMP_DIR`](#tmp_dir)

docs/apidocs/archivebox/archivebox.config.common.md

@@ -451,13 +451,13 @@ Bases: {py:obj}`archivebox.config.configset.BaseConfigSet`
 
 ````
 
-````{py:attribute} LIB_DIR
-:canonical: archivebox.config.common.StorageConfig.LIB_DIR
+````{py:attribute} ABXPKG_LIB_DIR
+:canonical: archivebox.config.common.StorageConfig.ABXPKG_LIB_DIR
 :type: pathlib.Path
 :value: >
    'Field(...)'
 
-```{autodoc2-docstring} archivebox.config.common.StorageConfig.LIB_DIR
+```{autodoc2-docstring} archivebox.config.common.StorageConfig.ABXPKG_LIB_DIR
 ```
 
 ````

docs/apidocs/archivebox/archivebox.config.constants.md

@@ -562,24 +562,13 @@
 
 ````
 
-````{py:attribute} LIB_DIR_NAME
-:canonical: archivebox.config.constants.ConstantsDict.LIB_DIR_NAME
-:type: str
-:value: >
-   'lib'
-
-```{autodoc2-docstring} archivebox.config.constants.ConstantsDict.LIB_DIR_NAME
-```
-
-````
-
-````{py:attribute} DEFAULT_LIB_DIR
-:canonical: archivebox.config.constants.ConstantsDict.DEFAULT_LIB_DIR
+````{py:attribute} DEFAULT_ABXPKG_LIB_DIR
+:canonical: archivebox.config.constants.ConstantsDict.DEFAULT_ABXPKG_LIB_DIR
 :type: pathlib.Path
 :value: >
    '_env_path(...)'
 
-```{autodoc2-docstring} archivebox.config.constants.ConstantsDict.DEFAULT_LIB_DIR
+```{autodoc2-docstring} archivebox.config.constants.ConstantsDict.DEFAULT_ABXPKG_LIB_DIR
 ```
 
 ````

docs/apidocs/archivebox/archivebox.personas.models.md

@@ -183,15 +183,6 @@ Bases: {py:obj}`archivebox.base_models.models.ModelWithConfig.Meta`
 
 ````
 
-````{py:property} CHROME_EXTENSIONS_DIR
-:canonical: archivebox.personas.models.Persona.CHROME_EXTENSIONS_DIR
-:type: str
-
-```{autodoc2-docstring} archivebox.personas.models.Persona.CHROME_EXTENSIONS_DIR
-```
-
-````
-
 ````{py:property} CHROME_DOWNLOADS_DIR
 :canonical: archivebox.personas.models.Persona.CHROME_DOWNLOADS_DIR
 :type: str

etc/package.json

@@ -1,6 +1,6 @@
 {
   "name": "archivebox",
-  "version": "0.9.35rc28",
+  "version": "0.9.35rc44",
   "repository": "github:ArchiveBox/ArchiveBox",
   "license": "MIT",
   "dependencies": {

old/archivebox.ts

@@ -179,8 +179,8 @@ if (!fs.existsSync(ARCHIVE_DIR))
 const PERSONA_DIR = path.join(DATA_DIR, 'personas', ACTIVE_PERSONA)
 const CHROME_PROFILE_PATH = path.join(PERSONA_DIR, 'chrome_profile')
 const CHROME_DOWNLOADS_DIR = path.join(PERSONA_DIR, 'chrome_downloads')
-const CHROME_EXTENSIONS_DIR =  path.join(PERSONA_DIR, 'chrome_extensions')
-const CHROME_EXTENSIONS_JSON_PATH = path.join(CHROME_EXTENSIONS_DIR, 'extensions.json')
+const CHROMEWEBSTORE_EXTENSIONS_DIR =  path.join(DATA_DIR, 'lib', 'chromewebstore', 'extensions')
+const CHROMEWEBSTORE_EXTENSIONS_JSON_PATH = path.join(CHROMEWEBSTORE_EXTENSIONS_DIR, 'extensions.json')
 const AUTH_JSON_PATH = path.join(PERSONA_DIR, 'auth.json')
 const COOKIES_TXT_PATH = path.join(PERSONA_DIR, 'cookies.txt')
 const SPEEDTESTS_DIR = path.join(PERSONA_DIR, 'speedtests')
@@ -191,7 +191,7 @@ const SPEEDTESTS_DIR = path.join(PERSONA_DIR, 'speedtests')
 fs.mkdirSync(PERSONA_DIR, {recursive: true})
 fs.mkdirSync(SPEEDTESTS_DIR, {recursive: true})
 fs.mkdirSync(CHROME_PROFILE_PATH, {recursive: true})
-fs.mkdirSync(CHROME_EXTENSIONS_DIR, {recursive: true})
+fs.mkdirSync(CHROMEWEBSTORE_EXTENSIONS_DIR, {recursive: true})
 fs.mkdirSync(CHROME_DOWNLOADS_DIR, {recursive: true})
 
 // cruft directories
@@ -870,8 +870,8 @@ async function loadOrInstallExtension(ext) {
     ext.name =              ext.name            || ext.webstore_id
     ext.webstore_url =      ext.webstore_url    || `https://chromewebstore.google.com/detail/${ext.webstore_id}`
     ext.crx_url =           ext.crx_url         || `https://clients2.google.com/service/update2/crx?response=redirect&prodversion=1230&acceptformat=crx3&x=id%3D${ext.webstore_id}%26uc`
-    ext.crx_path =          ext.crx_path        || path.join(CHROME_EXTENSIONS_DIR, `${ext.webstore_id}__${ext.name}.crx`)
-    ext.unpacked_path =     ext.unpacked_path   || path.join(CHROME_EXTENSIONS_DIR, `${ext.webstore_id}__${ext.name}`)
+    ext.crx_path =          ext.crx_path        || path.join(CHROMEWEBSTORE_EXTENSIONS_DIR, `${ext.webstore_id}__${ext.name}.crx`)
+    ext.unpacked_path =     ext.unpacked_path   || path.join(CHROMEWEBSTORE_EXTENSIONS_DIR, `${ext.webstore_id}__${ext.name}`)
     
     const manifest_path =   path.join(ext.unpacked_path, 'manifest.json')
     ext.read_manifest =     () => JSON.parse(fs.readFileSync(manifest_path, 'utf-8'))
@@ -1036,7 +1036,7 @@ async function loadExtensionFromTarget(extensions, target) {
 
 
 
-async function getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROME_EXTENSIONS_DIR}) {
+async function getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROMEWEBSTORE_EXTENSIONS_DIR}) {
     console.log('*************************************************************************')
     console.log(`[⚙️] Installing ${CHROME_EXTENSIONS.length} chrome extensions from CHROME_EXTENSIONS...`)
     try {
@@ -1047,7 +1047,7 @@ async function getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROME_EXTENSI
 
         // for easier debugging, write parsed extension info to filesystem
         await overwriteFile(
-            CHROME_EXTENSIONS_JSON_PATH.replace('.json', '.present.json'),
+            CHROMEWEBSTORE_EXTENSIONS_JSON_PATH.replace('.json', '.present.json'),
             CHROME_EXTENSIONS,
         )
     } catch(err) {
@@ -1058,7 +1058,7 @@ async function getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROME_EXTENSI
 }
 
 let _EXTENSIONS_CACHE = null
-async function getChromeExtensionsFromCache({browser, extensions=CHROME_EXTENSIONS, extensions_dir=CHROME_EXTENSIONS_DIR}) {
+async function getChromeExtensionsFromCache({browser, extensions=CHROME_EXTENSIONS, extensions_dir=CHROMEWEBSTORE_EXTENSIONS_DIR}) {
     if (_EXTENSIONS_CACHE === null) {
         console.log(`[⚙️] Loading ${CHROME_EXTENSIONS.length} chrome extensions from CHROME_EXTENSIONS...`)
 
@@ -1072,12 +1072,12 @@ async function getChromeExtensionsFromCache({browser, extensions=CHROME_EXTENSIO
 
         // write installed extension metadata to filesystem extensions.json for easier debugging
         await overwriteFile(
-            CHROME_EXTENSIONS_JSON_PATH.replace('.json', '.loaded.json'),
+            CHROMEWEBSTORE_EXTENSIONS_JSON_PATH.replace('.json', '.loaded.json'),
             extensions,
         )
         await overwriteSymlink(
-            CHROME_EXTENSIONS_JSON_PATH.replace('.json', '.loaded.json'),
-            CHROME_EXTENSIONS_JSON_PATH,
+            CHROMEWEBSTORE_EXTENSIONS_JSON_PATH.replace('.json', '.loaded.json'),
+            CHROMEWEBSTORE_EXTENSIONS_JSON_PATH,
         )
     }
     
@@ -5851,7 +5851,7 @@ async function startAPIServer(port=API_SERVER_PORT, host=API_SERVER_HOST, taskCa
 async function main(urls, cluster=CHROME_CLUSTER) {
     process.chdir(DATA_DIR)
 
-    const extensions =      await getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROME_EXTENSIONS_DIR})
+    const extensions =      await getChromeExtensionsFromPersona({CHROME_EXTENSIONS, CHROMEWEBSTORE_EXTENSIONS_DIR})
     const args =            getChromeArgs({...CHROME_LAUNCH_OPTIONS, CHROME_EXTENSIONS: extensions})
     const preferences =     getChromePreferences({CHROME_PREFERENCES_DEFAULT, CHROME_PREFERENCES_EXTRA, CHROME_DOWNLOADS_DIR, CHROME_EXTENSIONS: extensions})
     const Puppeteer =       applyChromePreferences(PupeteerExtra, CHROME_PREFERENCES_PATH, preferences)

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "archivebox"
-version = "0.9.35rc28"
+version = "0.9.35rc44"
 requires-python = ">=3.13"
 description = "Self-hosted internet archiving solution."
 authors = [{name = "Nick Sweeting", email = "pyproject.toml@archivebox.io"}]
@@ -79,9 +79,9 @@ dependencies = [
     ### Extractor dependencies (optional binaries detected at runtime via shutil.which)
     ### Binary/Package Management
     "abxbus==2.5.10",         # EventBus API
-    "abxpkg>=1.11.210",        # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
-    "abx-plugins>=1.11.213",  # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
-    "abx-dl>=1.11.213",       # shared ArchiveBox downloader package with blocking install preflight
+    "abxpkg>=1.11.219",        # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
+    "abx-plugins>=1.11.223",  # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
+    "abx-dl>=1.11.223",       # shared ArchiveBox downloader package with blocking install preflight
     ### UUID7 backport for Python <3.14
     "uuid7>=0.1.0; python_version < '3.14'",  # provides the uuid_extensions module on Python 3.13
 ]
@@ -138,10 +138,10 @@ dev = [
     # "snakeviz",                                      # usage: python -m cProfile -o flamegraph.prof ../.venv/bin/archivebox manage check
     ### TESTING
     "pytest-django>=4.11.1",
-    "pytest>=8.3.3",
+    "pytest>=8.3.3,<9",                               # pytest-codeblocks 0.17.0 still uses the pre-pytest-9 collect hook signature.
     "pytest-codeblocks>=0.17.0",
     "pytest-cov>=6.0.0",
-    "pytest-github-actions-annotate-failures>=0.3.0",
+    "pytest-github-actions-annotate-failures>=0.4.0",
     "pytest-httpserver>=1.1.0",
     "pytest-sugar>=1.0.0",
     "pytest-timeout>=2.3.1",
@@ -173,20 +173,53 @@ source-includes = [
     ".git/refs/heads/dev",
 ]
 excludes = [
-    "archivebox/**/.DS_Store",
-    "archivebox/**/.flake8",
-    "archivebox/**/__pycache__",
-    "archivebox/**/*.pyc",
-    "archivebox/**/*.pyo",
-    "archivebox/**/*.sqlite3",
-    "archivebox/**/*.sqlite3-*",
+    ".cache",
+    ".mypy_cache",
+    ".pdm-build",
+    ".pytest_cache",
+    ".ruff_cache",
+    ".tmp",
+    ".uv-cache",
+    ".venv",
+    "ArchiveBox.conf",
+    "archive",
+    "archivebox/.DS_Store",
+    "archivebox/*/.DS_Store",
+    "archivebox/.flake8",
+    "archivebox/__pycache__",
+    "archivebox/index.sqlite3",
+    "archivebox/index.sqlite3-*",
+    "archivebox/machine/tests",
     "archivebox/mcp/README.md",
     "archivebox/tests/",
-    "archivebox/tests/**",
-    "archivebox/**/tests/",
-    "archivebox/**/tests/**",
-    "tests/",
-    "tests/**",
+    "archivebox/workers/tests",
+    "build",
+    "cache",
+    "caches",
+    "chrome_profile",
+    "chromeprofile",
+    "crawls",
+    "data",
+    "data*",
+    "deb_dist",
+    "dist",
+    "index.sqlite3",
+    "index.sqlite3-*",
+    "lib",
+    "logs",
+    "node_modules",
+    "out",
+    "output",
+    "personas",
+    "pip_dist",
+    "queue.sqlite3",
+    "queue.sqlite3-*",
+    "snapshots",
+    "sources",
+    "tests",
+    "tmp",
+    "users",
+    "venv",
 ]
 
 [tool.setuptools]
@@ -209,7 +242,8 @@ ignore-words-list = "abx,archivebox,adminsnapshots,bu,wit,dont,cant,wont,havent,
 skip = "*.json,*.min.js,*.min.css,uv.lock,old/*,publicsite/*,docs/apidocs/*"
 
 [tool.pytest.ini_options]
-testpaths = [ "archivebox/tests" ]
+testpaths = [ "archivebox/tests", "README.md", "AGENTS.md", "skills", "docs" ]
+addopts = ["--codeblocks"]
 norecursedirs = ["archivebox/tests/data"]
 timeout = 600
 timeout_method = "thread"

skills/archivebox/SKILL.md

@@ -1,250 +1,81 @@
 ---
 name: archivebox
-description: Use this when an agent needs to install, run, automate, inspect, or troubleshoot ArchiveBox collections. Covers setup, data-folder workflow, CLI usage, Docker usage, Admin UI, REST API, SQLite index.sqlite3 access, filesystem layout, and `archivebox shell -c '...'` Django ORM snippets.
+description: Use this when working on the ArchiveBox app, CLI, server, Docker image, Admin UI, REST API, data dirs, crawls, snapshots, and release/deploy scripts.
 ---
 
 # ArchiveBox
 
-Use this skill when an agent needs to operate a full ArchiveBox collection: add URLs, run the server, inspect archived snapshots, automate via API, query `index.sqlite3`, or use the Django shell/ORM.
+## Purpose
 
-ArchiveBox is the full self-hosted archiving app. For one-off extraction without a collection/server, prefer the `abx-dl` skill.
+ArchiveBox is the full self-hosted web archiving app. Use this skill for collection operations, app code, Docker, Admin UI, API, and release work.
 
-## Core Model
+## Shared Rules
 
-- Always `cd` into the ArchiveBox data directory before running `archivebox ...`.
-- A data directory contains `ArchiveBox.conf`, `index.sqlite3`, and `archive/`.
-- Run `archivebox init` before other collection commands; it is safe to rerun and applies migrations.
-- Do not rely on `DATA_DIR=/path/to/data archivebox ...` for agent workflows; commands must be run from inside the initialized `data/` directory.
-- Avoid running local/dev ArchiveBox commands as root. The official Docker image handles its own runtime environment.
+- Keep this repo on branch `dev`.
+- Use `uv` and `uv run` for Python commands.
+- Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
+- Use existing ArchiveBox CLI/API/UI/runner paths for setup and side effects.
+- Do not mock, monkeypatch, fake, simulate, skip, xfail, or weaken tests.
+- Verify behavior with real commands, real data dirs, real DB rows, real hooks, real browsers, and real filesystem outputs.
+- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
 
-Current default layout:
-
-```text
-data/
-  ArchiveBox.conf
-  index.sqlite3
-  archive/
-    users/
-      <username>/
-        crawls/
-          YYYYMMDD/
-            <domain>/
-              <crawl-uuid>/
-                ...crawl-level state
-        snapshots/
-          YYYYMMDD/
-            <domain>/
-              <snapshot-uuid>/
-                index.jsonl
-                <plugin>/
-                  ...plugin output files
-```
-
-
-## Setup
-
-Preferred Docker Compose workflow:
+## Development Setup
 
 ```bash
-mkdir -p ~/archivebox/data
-cd ~/archivebox
-curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml' -o docker-compose.yml
-docker compose run archivebox init
-docker compose up
-```
-
-Local checkout workflow:
-
-```bash
-cd /path/to/ArchiveBox/archivebox
 uv sync --dev --all-extras
-mkdir -p ./data
-cd ./data
+mkdir -p data
+cd data
 uv run --project .. archivebox init --install
 ```
 
-Published CLI workflow:
+## User-Facing Setup
+
+Recommended CLI install:
 
 ```bash
+uv tool install archivebox
 mkdir -p ~/archivebox/data
 cd ~/archivebox/data
 archivebox init --install
-```
-
-## Basic CLI Usage
-
-Run commands from inside the data directory:
-
-```bash
-cd ~/archivebox/data
-archivebox version
-archivebox help
-archivebox status
 archivebox add 'https://example.com'
-archivebox add --depth=1 'https://news.ycombinator.com'
-echo 'https://example.com' | archivebox add
-archivebox list --json --with-headers > index.json
-archivebox list --html --with-headers > index.html
-archivebox search 'example'
-archivebox update --filter-type=domain example.com
-archivebox remove --filter-type=exact 'https://example.com'
-```
-
-Docker equivalents:
-
-```bash
-docker compose run archivebox init --install
-docker compose run archivebox add --depth=1 'https://example.com'
-echo 'https://example.com' | docker compose run -T archivebox add
-docker compose run -T archivebox list --json --with-headers > index.json
-```
-
-Useful subcommands:
-
-- `archivebox init`, `install`, `config`, `status`, `version`, `help`
-- `archivebox add`, `update`, `list`, `search`, `remove`, `schedule`
-- `archivebox server`, `manage`, `shell`
-- Model-oriented commands: `crawl`, `snapshot`, `archiveresult`, `tag`, `binary`, `process`, `machine`, `persona`
-
-## Configuration
-
-ArchiveBox config can come from environment variables, `ArchiveBox.conf`, or `archivebox config --set`.
-
-```bash
-archivebox config
-archivebox config --get CHROME_BINARY
-archivebox config --set TIMEOUT=240
-archivebox config --set CHECK_SSL_VALIDITY=False
-archivebox config --set PUBLIC_INDEX=False PUBLIC_SNAPSHOTS=False PUBLIC_ADD_VIEW=False
-CHROME_BINARY=chromium archivebox add 'https://example.com'
-```
-
-Common knobs:
-
-- `TIMEOUT`, `USER_AGENT`, `CHECK_SSL_VALIDITY`
-- `PUBLIC_INDEX`, `PUBLIC_SNAPSHOTS`, `PUBLIC_ADD_VIEW`
-- `CHROME_BINARY`, `SAVE_WGET`, `SAVE_DOM`, `SAVE_PDF`, `SAVE_SCREENSHOT`
-- `LIB_DIR`, `TMP_DIR`
-
-## Admin UI
-
-Create an admin user and start the server:
-
-```bash
-archivebox manage createsuperuser
 archivebox server 0.0.0.0:8000
 ```
 
-Docker Compose:
+Alternative install methods:
+
+- Docker Compose / Docker
+- Homebrew
+- Debian package
+- pip
+
+## Basic Usage
+
+Run from inside an initialized data dir:
 
 ```bash
-docker compose run archivebox manage createsuperuser
-docker compose up
+archivebox version
+archivebox status
+archivebox install
+archivebox add 'https://example.com'
+archivebox add --extract=title,screenshot,pdf 'https://example.com'
+archivebox list --json --with-headers
+archivebox search 'example'
+archivebox update --filter-type=domain example.com
+archivebox remove --filter-type=exact 'https://example.com'
+archivebox run
 ```
 
-Open:
-
-- Public UI: `http://web.archivebox.localhost:8000`
-- Admin UI: `http://admin.archivebox.localhost:8000`
-- API docs: `http://web.archivebox.localhost:8000/api/v1/docs`
-
-If custom hostnames are not resolving, try `/admin` and `/api/v1/docs` on the server base URL.
-
-## REST API
-
-The REST API is alpha and served by Django Ninja. Always check the live Swagger docs for the exact schema:
-
-```text
-GET /api/v1/docs
-GET /api/v1/openapi.json
-```
-
-Get an API token:
+## Verification
 
+<!--pytest.mark.skip(reason="pytest invocation")-->
 ```bash
-curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/auth/get_api_token' \
-  -H 'Content-Type: application/json' \
-  -d '{"username":"admin","password":"password"}'
+uv run pytest archivebox/tests/test_cli_add.py -q
+uv run prek run --all-files
 ```
 
-Authenticate with any of:
+Use the full release/deploy loop only when requested:
 
+<!--pytest.mark.skip(reason="release/deploy script")-->
 ```bash
-Authorization: Bearer <token>
-X-ArchiveBox-API-Key: <token>
-?api_key=<token>
+./bin/release_dev_stack.sh
 ```
-
-Useful endpoints:
-
-- `POST /api/v1/cli/add` mirrors `archivebox add` and queues background work.
-- `GET /api/v1/core/snapshots`
-- `GET /api/v1/core/snapshot/{snapshot_id}`
-- `POST /api/v1/core/snapshots`
-- `GET /api/v1/crawls/crawls`
-- `POST /api/v1/crawls/crawls`
-- `GET /api/v1/machine/binaries`
-
-Example add via API:
-
-```bash
-TOKEN='...'
-curl -sS -X POST 'http://web.archivebox.localhost:8000/api/v1/cli/add' \
-  -H "Authorization: Bearer $TOKEN" \
-  -H 'Content-Type: application/json' \
-  -d '{"urls":["https://example.com"],"depth":0,"tag":"agent-run"}'
-```
-
-Keep `archivebox server` running to process queued UI/API jobs.
-
-## SQLite Index
-
-The main metadata database is `index.sqlite3` in the data directory. Prefer the ORM for writes. Direct SQLite is useful for read-only inspection, exports, and debugging.
-
-```bash
-sqlite3 ./index.sqlite3 '.tables'
-sqlite3 ./index.sqlite3 'SELECT COUNT(*) FROM core_snapshot;'
-sqlite3 ./index.sqlite3 "SELECT timestamp, url, title, status FROM core_snapshot ORDER BY bookmarked_at DESC LIMIT 20;"
-sqlite3 ./index.sqlite3 "SELECT plugin, status, COUNT(*) FROM core_archiveresult GROUP BY plugin, status ORDER BY plugin, status;"
-sqlite3 ./index.sqlite3 "SELECT id, status, substr(urls, 1, 120) FROM crawls_crawl ORDER BY created_at DESC LIMIT 20;"
-```
-
-Useful tables usually include:
-
-- `core_snapshot`: URL, title, timestamp, status, crawl linkage, output size.
-- `core_archiveresult`: extractor/plugin run status and output metadata.
-- `core_tag`, `core_snapshottag`: tags and snapshot-tag links.
-- `crawls_crawl`: crawl batches, queued URLs, config, status.
-- Django tables such as `auth_user` and `django_migrations`.
-
-For large or active archives, avoid long write transactions and keep `index.sqlite3` on local disk/SSD when possible.
-
-## Django Shell
-
-Use `archivebox shell` for interactive ORM access and `archivebox shell -c '...'` for agent-friendly one-liners. Run from the data directory.
-
-Read-only examples:
-
-```bash
-archivebox shell -c 'from archivebox.core.models import Snapshot; print(Snapshot.objects.count())'
-archivebox shell -c 'from archivebox.core.models import Snapshot; print(list(Snapshot.objects.values("timestamp", "url", "title", "status").order_by("-bookmarked_at")[:10]))'
-archivebox shell -c 'from archivebox.core.models import ArchiveResult; print(list(ArchiveResult.objects.values("plugin", "status").order_by("plugin", "status").distinct()))'
-archivebox shell -c 'from archivebox.crawls.models import Crawl; print(list(Crawl.objects.values("id", "status", "urls").order_by("-created_at")[:5]))'
-```
-
-Safe write examples:
-
-```bash
-archivebox shell -c 'from archivebox.core.models import Snapshot; s=Snapshot.objects.get(timestamp="1700000000"); s.notes="reviewed by agent"; s.save(update_fields=["notes", "modified_at"])'
-archivebox shell -c 'from archivebox.core.models import Snapshot, Tag; s=Snapshot.objects.get(timestamp="1700000000"); t,_=Tag.objects.get_or_create(name="important"); s.tags.add(t)'
-```
-
-Prefer `archivebox add`, the REST API, or model methods for creating crawl/snapshot work. Use direct ORM writes only when you understand the model lifecycle.
-
-## Recommended Agent Workflow
-
-1. Identify the data directory and run `archivebox status`.
-2. Use `archivebox config` to confirm privacy, timeout, and browser settings before ingesting sensitive URLs.
-3. Add URLs with `archivebox add` or `POST /api/v1/cli/add`.
-4. Inspect progress with the Admin UI, `archivebox list`, `archivebox shell -c`, or read-only SQLite queries.
-5. Use filesystem outputs under `archive/` for artifacts, but use `index.sqlite3`/ORM/API for authoritative metadata.