Bump release version to 0.9.35rc482

An explicit `DISPLAY=novnc:0.0` was silently removed by the Docker
entrypoint whenever `xdpyinfo` failed. In the reported `0.9.35rc381`
image, `xdpyinfo` is missing, so `docker compose run --rm` clears the
display even with a healthy noVNC server and Chromium exits with
“Missing X server or $DISPLAY”.

Preserve explicitly supplied `DISPLAY`, including an empty value to opt
out. Probe only the implicit `novnc:0.0` default, retaining automatic
headless fallback when no display was configured. Current dev already
includes `x11-utils`; this change prevents probe failure from overriding
user configuration.

Document the one-off `persona create` / `persona open` Compose flow.
`persona open` resolves Chromium automatically, avoiding the obsolete
`/data/lib/env/bin/chromium` and architecture/version-specific
Playwright paths.

Validation:
- Reproduced persona creation followed by the exact Chromium X11 failure
on the published `0.9.35rc381` image with real, healthy `theasp/novnc`.
- Mounted only the patched entrypoint into that affected image: explicit
DISPLAY survived despite the missing `xdpyinfo`, and the same Chromium
command opened a window visible through noVNC.
- Built the complete branch Docker image using the Dockerfile's default
abx-dl base. Real `docker compose run --rm archivebox persona create
personal` and `persona open personal` succeeded; viewed and controlled
Chromium through noVNC, closed it normally, and reopened in another
one-off container. A bookmark created in the UI persisted across both
runs.
- Added `IMAGE=<built-image> ./bin/validate_docker_display.sh`: six real
Compose checks cover working implicit/explicit displays, unavailable
explicit display, explicit empty opt-out, and implicit/explicit behavior
with noVNC stopped. Passed on the rebuilt image; failed on the unpatched
image at the explicit unavailable-display assertion.
- `bash -n`, `git diff --check`, and pre-commit hooks passed. No mocks
or fake binaries.

Fixes #1872.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the Docker entrypoint silently dropping an explicit `DISPLAY` when
`xdpyinfo` fails, so Chromium can connect to a healthy noVNC server in
images like `0.9.35rc381` where `xdpyinfo` is missing. The entrypoint
previously unset the display on probe failure, causing Chromium to exit
with "Missing X server or $DISPLAY"; now only the implicit `novnc:0.0`
default is probed, an explicit value (including empty to opt out) is
preserved, and headless fallback still applies when no display was
configured. Fixes #1872.

**Validation**
- Adds `bin/validate_docker_display.sh` with six real Compose checks
covering implicit/explicit displays, unavailable display, empty opt-out,
and noVNC stopped behavior.
- Documents the one-off `persona create` / `persona open` Compose flow
and that `persona open` resolves the Chromium binary automatically.

<sup>Written for commit 953f29b65e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1882?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

.github/workflows/deploy-publicsite.yml

@@ -1,79 +1,50 @@
 name: Deploy Publicsite to GitHub Pages
-
 on:
   push:
-    branches:
-      - dev
-    paths:
-      - publicsite/**
-      - archivebox/**
-      - bin/collect_ui_screenshots.*
-      - bin/generate_ui_screenshot_gallery.py
-      - bin/take_screenshot.js
-      - docs/Screenshots.md
-      - pyproject.toml
-      - uv.lock
-      - .github/workflows/deploy-publicsite.yml
+    branches: [dev]
+  pull_request:
   workflow_dispatch:
-
+  workflow_run:
+    workflows: ['Capture publicsite screenshots']
+    types: [completed]
+    branches: [dev]
 permissions:
   contents: read
-  pages: write
-  id-token: write
-
+  actions: read
 concurrency:
-  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
-  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
-
+  # Failed captures must not displace a pending push deployment.
+  group: fast-pages-${{ github.event_name == 'workflow_run' && github.event.workflow_run.conclusion != 'success' && github.run_id || github.ref }}
+  cancel-in-progress: false
 jobs:
+  build:
+    if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event_name == 'pull_request' && github.ref || 'dev' }}
+      - uses: astral-sh/setup-uv@v7
+      - name: Restore latest complete screenshots
+        run: uv run --no-project python bin/restore_site_screenshots.py archivebox publicsite/screenshots
+        env:
+          GH_TOKEN: ${{ github.token }}
+      - name: Render current gallery layout without recapturing images
+        run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: publicsite
   deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-24.04
     steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Setup Python and uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          enable-cache: false
-
-      - name: Install ArchiveBox and screenshot dependencies
-        run: uv sync --no-cache --frozen
-
-      - name: Build UI screenshot gallery
-        run: bin/collect_ui_screenshots.sh
-        env:
-          BASE_URL: http://archivebox.localhost:8000
-
-      - name: Verify gallery was generated from this exact revision
-        run: |
-          uv run --no-cache --no-sync python - <<'PY'
-          import json
-          import os
-          import tomllib
-          from pathlib import Path
-
-          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
-          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
-          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
-          assert provenance["version"] == version, provenance
-          assert provenance["capture_count"] >= 3 * 35, provenance
-          assert len(provenance["files"]) == provenance["capture_count"], provenance
-          PY
-
-      - name: Setup Pages
-        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
-        with:
-          path: ./publicsite
-
-      - name: Deploy to GitHub Pages
+      - uses: actions/configure-pages@v5
+      - uses: actions/deploy-pages@v4
         id: deployment
-        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

.github/workflows/docker.yml

@@ -357,7 +357,7 @@ jobs:
           "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
           "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+            "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
             "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
 

.github/workflows/screenshots.yml

@@ -0,0 +1,70 @@
+name: Capture publicsite screenshots
+
+on:
+  push:
+    branches:
+      - dev
+    paths:
+      - publicsite/**
+      - archivebox/**
+      - bin/collect_ui_screenshots.*
+      - bin/generate_ui_screenshot_gallery.py
+      - bin/take_screenshot.js
+      - docs/Screenshots.md
+      - pyproject.toml
+      - uv.lock
+      - .github/workflows/screenshots.yml
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.event_name == 'workflow_dispatch' && format('publicsite-captures-manual-{0}', github.run_id) || 'publicsite-captures' }}
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Setup Python and uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: false
+
+      - name: Install ArchiveBox and screenshot dependencies
+        run: uv sync --no-cache --frozen
+
+      - name: Build UI screenshot gallery
+        run: bin/collect_ui_screenshots.sh
+        env:
+          BASE_URL: http://archivebox.localhost:8000
+
+      - name: Verify gallery was generated from this exact revision
+        run: |
+          uv run --no-cache --no-sync python - <<'PY'
+          import json
+          import os
+          import tomllib
+          from pathlib import Path
+
+          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
+          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
+          assert provenance["version"] == version, provenance
+          assert provenance["capture_count"] >= 3 * 35, provenance
+          assert len(provenance["files"]) == provenance["capture_count"], provenance
+          PY
+
+      - name: Save validated screenshots
+        uses: actions/upload-artifact@v4
+        with:
+          name: site-screenshots
+          path: publicsite/screenshots
+          if-no-files-found: error
+          retention-days: 90

Dockerfile

@@ -10,10 +10,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.273 \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.300 \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.273
+ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.300
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -96,17 +96,15 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$T
     --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
     <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
-echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
-echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
+export UV_NO_CACHE=true
 rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
-apt-get install -qq -y --no-install-recommends \
-    build-essential gcc libldap2-dev libsasl2-dev libssl-dev x11-utils
+apt-get install -qq -y --no-install-recommends x11-utils
 # Extend abx-dl's existing venv instead of clearing it. Clearing and later
 # copying a complete replacement over the base duplicates every venv byte in
 # the final overlay history even when most packages are unchanged.
-/usr/bin/uv pip install --no-cache setuptools pip wheel
+# Do not rewrite inherited native libraries merely to strip them again.
+/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries
 
 mkdir -p /tmp/archivebox-uv-project
 /venv/bin/python - <<'PY'
@@ -155,25 +153,31 @@ PY
     --inexact \
     --no-install-project \
     --no-install-workspace \
+    --no-install-package abxbus \
+    --no-install-package abxpkg \
+    --no-install-package abx-plugins \
+    --no-install-package abx-dl \
     --no-sources
 builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
 ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
 while IFS= read -r -d '' native_library; do
+    if grep -Fxq "$native_library" /tmp/archivebox-inherited-libraries; then
+        continue
+    fi
     magic=''
     if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
         "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
     fi
 done < /tmp/archivebox-native-libraries
-rm -f /tmp/archivebox-native-libraries
+rm -f /tmp/archivebox-native-libraries /tmp/archivebox-inherited-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils
 abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
 rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
     /venv/lib/python3.*/site-packages/wheel* \
     /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
 rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
 EOF
 
@@ -202,7 +206,7 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
-# The builder installs and purges compilers in one layer, so its final
+# The builder installs and purges stripping tools in one layer, so its final
 # filesystem is already runtime-clean. Preserve that ancestry: starting again
 # from archivebox-runtime-base and COPYing /venv would bake a second full venv
 # over the inherited one instead of recording only ArchiveBox's package delta.
@@ -225,7 +229,8 @@ COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 # The builder invokes abxpkg as root for temporary ELF stripping, which can
 # rewrite derived state ownership. Restore UID 911 before resolving binaries
 # as the runtime user; the old copy-based final stage hid this dependency.
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
+RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777,id=archivebox-runtime-config-$TARGETARCH \
+    echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
     && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
     && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
     && ln -sf /venv/bin/daphne /usr/local/bin/daphne \
@@ -239,8 +244,8 @@ RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..."
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
     && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
-    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
+    && find "$ABXPKG_LIB_DIR" \( ! -user "$DEFAULT_ARCHIVEBOX_UID" -o ! -group "$DEFAULT_ARCHIVEBOX_GID" \) -exec chown -h "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" {} + \
+    && UV_NO_CACHE=true setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
     && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt

README.md

@@ -151,6 +151,17 @@ curl -fsSL 'https://get.archivebox.io' | bash
 
 #### ✳️  Easy Setup
 
+<details>
+<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
+<br/>
+<ol>
+<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
+<li>Set up an admin account and generate an API Key.</li>
+<li>Use the Web UI or <a href="https://archivebox.github.io/ios-archivebox/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://archivebox.github.io/ios-archivebox/">mobile app</a>, the <a href="https://archivebox.github.io/archivebox-browser-extension/">browser extension</a>, and other integrations to your server using your API Key.</li>
+</ol>
+<br/>
+</details>
+
 <details>
 <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b>  (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
 <br/>
@@ -561,6 +572,20 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
 </code></pre>
 </details>
 
+<br/>
+
+<details>
+<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
+<br/>
+
+- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://archivebox.github.io/ios-archivebox/)
+- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://archivebox.github.io/ios-archivebox/)
+- 🧩 [ArchiveBox Browser Extension](https://archivebox.github.io/archivebox-browser-extension/)
+- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://github.com/ArchiveBox/electron-archivebox)
+
+<br/>
+</details>
+
 <br/>
 <br/>
 
@@ -614,7 +639,10 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>  
-  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
+  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
+
+- <img src="https://archivebox.github.io/ios-archivebox/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
+  <i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files  
   <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
@@ -1036,8 +1064,8 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 </li>
 <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
 </li>
-<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
-<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
+<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
+<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
 </li>
 </ul>
 

archivebox/api/v1_auth.py

@@ -4,7 +4,7 @@ from django.http import HttpRequest
 
 from ninja import Router, Schema
 
-from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
+from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token, HeaderTokenAuth, BearerTokenAuth
 
 
 router = Router(tags=["Authentication"], auth=None)
@@ -63,3 +63,44 @@ def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
         return {"success": True, "user_id": str(user.pk)}
 
     return {"success": False, "user_id": None}
+
+
+@router.post(
+    "/browser_session",
+    auth=[HeaderTokenAuth(), BearerTokenAuth()],
+    summary="Exchange an administrator API key for a normal browser session",
+)
+def browser_session(request: HttpRequest):
+    from django.conf import settings
+    from django.contrib.auth import login
+    from django.http import JsonResponse
+    from django.utils import timezone
+    from ninja.errors import HttpError
+    from archivebox.core.routes_util import get_admin_base_url
+
+    user = request.auth
+    if not user.is_active or not user.is_superuser:
+        raise HttpError(403, "An active administrator is required")
+    login(request, user, backend="django.contrib.auth.backends.ModelBackend")
+    # A browser session must not outlive the key used to establish it.
+    token = request._api_token
+    lifetime = settings.SESSION_COOKIE_AGE
+    if token.expires:
+        lifetime = min(lifetime, max(0, int((token.expires - timezone.now()).total_seconds())))
+    request.session.set_expiry(lifetime)
+    request.session.save()
+    response = JsonResponse(
+        {
+            "admin_url": get_admin_base_url(request=request).rstrip("/") + "/admin/",
+            "cookie": {
+                "name": settings.SESSION_COOKIE_NAME,
+                "value": request.session.session_key,
+                "expires": request.session.get_expiry_date().timestamp(),
+                "secure": settings.SESSION_COOKIE_SECURE,
+            },
+        },
+    )
+    # The native client installs a host-only HttpOnly cookie on the admin origin;
+    # API and admin hosts can differ, so a Set-Cookie on the API host is insufficient.
+    response["Cache-Control"] = "no-store"
+    return response

archivebox/cli/archivebox_add.py

@@ -388,10 +388,6 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
-        if not kwargs.get("index_only"):
-            from archivebox.workers.supervisord_util import require_crawl_memory
-
-            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.

archivebox/cli/archivebox_server.py

@@ -363,12 +363,10 @@ def server(
     command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
 
     def still_owns_runtime_stack() -> bool:
-        from django.db import connections
-
-        try:
-            return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
-        finally:
-            connections.close_all()
+        # This is polled at the log-tail cadence. Keep the autocommit
+        # connection open so SQLite can reuse its page cache between polls;
+        # reconnecting here repeatedly rereads the process table while idle.
+        return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
 
     shutdown_state = None
     try:
@@ -416,8 +414,13 @@ def server(
     except KeyboardInterrupt:
         pass
     finally:
-        if not shutdown_state or not shutdown_state.signal_name:
-            command.mark_exited()
+        from django.db import connections
+
+        try:
+            if not shutdown_state or not shutdown_state.signal_name:
+                command.mark_exited()
+        finally:
+            connections.close_all()
     print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
 
 

archivebox/core/takeover_util.py

@@ -126,12 +126,25 @@ def ensure_daemon_stack(*, reason: str = ""):
 
     supervisor = get_existing_supervisord_process() or get_or_create_supervisord_process(daemonize=False)
     worker = get_worker(supervisor, sonic_worker["name"])
-    if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
-        return worker
+    if not (isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING")):
+        if reason:
+            rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
+        worker = start_worker(supervisor, sonic_worker)
 
-    if reason:
-        rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
-    return start_worker(supervisor, sonic_worker)
+    # Supervisord marks this worker RUNNING with startsecs=0, before Sonic has
+    # opened its search socket. A query launched at that point fails even though
+    # the daemon is about to become ready.
+    deadline = time.monotonic() + 30.0
+    while time.monotonic() < deadline:
+        if is_port_listening(sonic_event.host, sonic_event.port):
+            if time.monotonic() < deadline:
+                return worker
+            break
+        worker = get_worker(supervisor, sonic_worker["name"])
+        if not isinstance(worker, dict) or worker.get("statename") not in ("STARTING", "RUNNING"):
+            raise RuntimeError(f"Sonic search backend worker failed to start: {worker}")
+        time.sleep(min(0.1, max(0.0, deadline - time.monotonic())))
+    raise RuntimeError(f"Sonic search backend is not listening at {sonic_event.url}")
 
 
 def live_runner_processes(*, data_dir: str | Path):

archivebox/core/views.py

@@ -765,6 +765,11 @@ def _plugin_full_preview_response(
     response.headers["X-Content-Type-Options"] = "nosniff"
     response.headers["X-ArchiveBox-Security-Mode"] = request.archivebox_config.SERVER_SECURITY_MODE
     response.headers["Referrer-Policy"] = "no-referrer"
+    # Trusted viewers live on the snapshot origin, but the collection UI lives
+    # on web.* (and can be embedded by admin.*). Permit only those configured
+    # origins, not arbitrary sites or other snapshots. This does not grant the
+    # archived document access to the parent frame or its admin session cookies.
+    viewer_origins = " ".join(dict.fromkeys((build_web_url(request=request), build_admin_url(request=request))))
     response.headers["Content-Security-Policy"] = (
         "default-src 'self' data: blob:; "
         "script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; "
@@ -778,7 +783,7 @@ def _plugin_full_preview_response(
         "object-src 'none'; "
         "base-uri 'none'; "
         "form-action 'none'; "
-        "frame-ancestors 'self';"
+        f"frame-ancestors 'self' {viewer_origins};"
     )
     return response
 

archivebox/crawls/admin.py

@@ -1230,7 +1230,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
         (
             "Configuration",
             {
-                "fields": ("schedule", "template"),
+                "fields": ("is_enabled", "schedule", "template"),
                 "classes": ("card",),
             },
         ),
@@ -1276,7 +1276,21 @@ class CrawlScheduleAdmin(BaseModelAdmin):
 
     def change_view(self, request, object_id, form_url="", extra_context=None):
         self.request = request
-        return super().change_view(request, object_id, form_url, extra_context)
+        from django.db import connections, router
+
+        connection = connections[router.db_for_write(self.model)]
+        if request.method != "POST" or connection.vendor != "sqlite":
+            return super().change_view(request, object_id, form_url, extra_context)
+        # Reserve the writer before Django's atomic admin form reads. A
+        # deferred read transaction cannot safely upgrade while the runner
+        # writes, even when SQLite's existing busy timeout has time left.
+        connection.ensure_connection()
+        previous_mode = connection.transaction_mode
+        connection.transaction_mode = "IMMEDIATE"
+        try:
+            return super().change_view(request, object_id, form_url, extra_context)
+        finally:
+            connection.transaction_mode = previous_mode
 
     def add_view(self, request, form_url="", extra_context=None):
         return redirect("/add/#schedule")

archivebox/machine/models.py

@@ -227,6 +227,9 @@ class Machine(ModelWithHealthStats):
             _CURRENT_MACHINE = None
         if _CURRENT_MACHINE:
             if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+                # Reconciliation can save() and clear the shared cache; keep
+                # this call's instance so we never return None after that save.
+                machine = _CURRENT_MACHINE
                 # One-time-per-process reconciliation between ArchiveBox.conf
                 # and Machine.config. Fast-path: bool check + early-return when
                 # the sync has already run, so the cached-machine return path
@@ -234,16 +237,21 @@ class Machine(ModelWithHealthStats):
                 try:
                     from archivebox.config.collection import sync_machine_and_file
 
-                    sync_machine_and_file(_CURRENT_MACHINE)
+                    sync_machine_and_file(machine)
                 except Exception:
                     pass
-                return _CURRENT_MACHINE
+                return machine
             else:
                 _CURRENT_MACHINE = None
 
         host_guid = get_host_guid()
+        # save() intentionally invalidates _CURRENT_MACHINE so config edits
+        # made through other instances become visible in this process. Host
+        # refresh (after seven days), sanitation, and reconciliation can all
+        # save here, so the working instance must not live only in that cache.
+        # Otherwise a routine refresh leaves startup dereferencing None.
         try:
-            _CURRENT_MACHINE = cls.objects.get(guid=host_guid)
+            machine = cls.objects.get(guid=host_guid)
         except cls.DoesNotExist:
             config = {}
             try:
@@ -253,7 +261,7 @@ class Machine(ModelWithHealthStats):
                 config = _coerce_from_str_dict(file_config)
             except Exception:
                 config = {}
-            _CURRENT_MACHINE = cls.objects.create(
+            machine = cls.objects.create(
                 guid=host_guid,
                 hostname=socket.gethostname(),
                 config=config,
@@ -262,15 +270,15 @@ class Machine(ModelWithHealthStats):
                 stats=get_host_stats(),
             )
         else:
-            if timezone.now() >= _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+            if timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
                 for key, value in {
                     "hostname": socket.gethostname(),
                     **get_os_info(),
                     **get_vm_info(),
                     "stats": get_host_stats(),
                 }.items():
-                    setattr(_CURRENT_MACHINE, key, value)
-                _CURRENT_MACHINE.save(
+                    setattr(machine, key, value)
+                machine.save(
                     update_fields=[
                         "hostname",
                         "hw_in_docker",
@@ -287,16 +295,19 @@ class Machine(ModelWithHealthStats):
                         "modified_at",
                     ],
                 )
-        machine = cls._sanitize_config(_CURRENT_MACHINE)
+        machine = cls._sanitize_config(machine)
         # Same one-time sync as the cached-return path. Triggers here on the
         # very first ``Machine.current()`` call in a process before any
         # cached return can occur.
         try:
             from archivebox.config.collection import sync_machine_and_file
 
-            sync_machine_and_file(_CURRENT_MACHINE)
+            sync_machine_and_file(machine)
         except Exception:
             pass
+        # Publish only after all save-capable steps; publishing earlier would
+        # let those steps invalidate the cache we are in the middle of filling.
+        _CURRENT_MACHINE = machine
         return machine
 
     @classmethod

archivebox/misc/checks.py

@@ -255,7 +255,7 @@ def check_data_dir_permissions(config=None, **config_kwargs):
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions]https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions[/link]",
         )
         STDERR.print(
-            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid[/link]",
+            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid[/link]",
         )
         STDERR.print(
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts]https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts[/link]",

archivebox/misc/hashing.py

@@ -1,260 +0,0 @@
-# Bootable hashing helpers (file content + mtime/size cache, mime detection).
-# MUST NOT import archivebox.config, archivebox.core, or Django — stdlib only.
-
-import hashlib
-import mimetypes
-from functools import lru_cache
-from pathlib import Path
-from collections.abc import Callable
-from datetime import datetime
-
-
-@lru_cache(maxsize=1024)
-def _cached_file_hash(filepath: str, size: int, mtime: float) -> str:
-    """Internal function to calculate file hash with cache key based on path, size and mtime."""
-    sha256_hash = hashlib.sha256()
-
-    with open(filepath, "rb") as f:
-        for chunk in iter(lambda: f.read(4096), b""):
-            sha256_hash.update(chunk)
-
-    return sha256_hash.hexdigest()
-
-
-@lru_cache(maxsize=10)
-def hash_file(file_path: Path, pwd: Path | None = None) -> str:
-    """Calculate SHA256 hash of a file with caching based on path, size and mtime."""
-    pwd = Path(pwd) if pwd else None
-    file_path = Path(file_path)
-    if not file_path.is_absolute():
-        file_path = pwd / file_path if pwd else file_path.absolute()
-
-    abs_path = file_path.resolve()
-    stat_info = abs_path.stat()
-
-    return _cached_file_hash(
-        str(abs_path),
-        stat_info.st_size,
-        stat_info.st_mtime,
-    )
-
-
-@lru_cache(maxsize=10)
-def get_dir_hashes(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict[str, str]:
-    """Calculate SHA256 hashes for all files and directories recursively."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    if not dir_path.is_dir():
-        raise ValueError(f"Not a directory: {dir_path}")
-    if max_depth < -1:
-        raise ValueError(f"max_depth must be >= -1, got {max_depth}")
-
-    # Get all files recursively
-    all_files = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=True,
-        include_dirs=False,
-        filter_func=filter_func,
-    )
-
-    hashes: dict[str, str] = {}
-    hashable_summary = []
-
-    # Calculate hashes for all files
-    for subfile in all_files:
-        subfile_path = dir_path / subfile
-        sha256_hash = hash_file(subfile_path)
-        hashes[subfile] = sha256_hash
-        hashable_summary.append(f"{sha256_hash}  ./{subfile}")
-
-    # Calculate hashes for all directories
-    subdirs = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=False,
-        include_dirs=True,
-        include_hidden=False,
-        filter_func=filter_func,
-        max_depth=max_depth,
-    )
-
-    for subdir in subdirs:
-        subdir_path = dir_path / subdir
-        subdir_hashes = get_dir_hashes(
-            subdir_path,
-            filter_func=filter_func,
-            max_depth=0,
-        )
-        hashes[subdir] = subdir_hashes["."]
-
-    # Filter results by max_depth
-    if max_depth >= 0:
-        hashes = {path: value for path, value in hashes.items() if len(Path(path).parts) <= max_depth + 1}
-
-    # Calculate root directory hash
-    hashable_summary.sort()
-    root_sha256 = hashlib.sha256("\n".join(hashable_summary).encode()).hexdigest()
-    hashes["."] = root_sha256
-
-    return hashes
-
-
-@lru_cache(maxsize=128)
-def get_dir_entries(
-    dir_path: Path,
-    pwd: Path | None = None,
-    recursive: bool = True,
-    include_files: bool = True,
-    include_dirs: bool = True,
-    include_hidden: bool = False,
-    filter_func: Callable | None = None,
-    max_depth: int = -1,
-) -> tuple[str, ...]:
-    """Get filtered list of directory entries."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    results = []
-
-    def process_path(path: Path, depth: int):
-        if not include_hidden and path.name.startswith("."):
-            return False
-        if max_depth >= 0 and depth > max_depth:
-            return False
-        if filter_func:
-            info = {
-                "abspath": str(path.absolute()),
-                "relpath": str(path.relative_to(dir_path)),
-            }
-            if not filter_func(info):
-                return False
-        return True
-
-    for path in dir_path.rglob("*") if recursive else dir_path.glob("*"):
-        current_depth = len(path.relative_to(dir_path).parts)
-
-        if path.is_file() and include_files and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-        elif path.is_dir() and include_dirs and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-
-        if not recursive:
-            break
-
-    return tuple(sorted(results))  # Make immutable for caching
-
-
-@lru_cache(maxsize=1024)
-def get_dir_sizes(dir_path: Path, pwd: Path | None = None, **kwargs) -> dict[str, int]:
-    """Calculate sizes for all files and directories recursively."""
-    sizes: dict[str, int] = {}
-    hashes = get_dir_hashes(dir_path, pwd=pwd, **kwargs)
-    dir_path = Path(dir_path)
-
-    for path_key in hashes:
-        full_path = dir_path / path_key
-        if full_path.is_file():
-            sizes[path_key] = full_path.stat().st_size
-        else:
-            total = 0
-            for file_path in full_path.rglob("*"):
-                if file_path.is_file() and not file_path.name.startswith("."):
-                    total += file_path.stat().st_size
-            sizes[path_key + "/"] = total
-
-    return sizes
-
-
-@lru_cache(maxsize=10)
-def get_dir_info(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict:
-    """Get detailed information about directory contents including hashes and sizes."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    hashes = get_dir_hashes(dir_path, pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-    sizes = get_dir_sizes(str(dir_path), pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-
-    num_total_subpaths = sum(1 for name in hashes if name != ".")
-    details = {}
-
-    for filename, sha256_hash in sorted(hashes.items()):
-        abs_path = (dir_path / filename).resolve()
-        stat_info = abs_path.stat()
-        num_subpaths = sum(1 for p in hashes if p.startswith(filename + "/"))
-        is_dir = abs_path.is_dir()
-        if is_dir:
-            mime_type = "inode/directory"
-            basename = abs_path.name
-            extension = ""
-            num_bytes = sizes[filename + "/"]
-            if filename == ".":
-                num_subpaths = num_total_subpaths
-            else:
-                filename += "/"
-                num_subpaths = num_subpaths
-        else:  # is_file
-            num_subpaths = None
-            mime_type = mimetypes.guess_type(str(abs_path))[0]
-            extension = abs_path.suffix
-            basename = abs_path.name.rsplit(extension, 1)[0]
-            num_bytes = sizes[filename]
-
-        details[filename] = {
-            "basename": basename,
-            "mime_type": mime_type,
-            "extension": extension,
-            "num_subpaths": num_subpaths,
-            "num_bytes": num_bytes,
-            "hash_sha256": sha256_hash,
-            "created_at": datetime.fromtimestamp(stat_info.st_ctime).isoformat(),
-            "modified_at": datetime.fromtimestamp(stat_info.st_mtime).isoformat(),
-        }
-
-        if filter_func and not filter_func(details[filename]):
-            del details[filename]
-
-    return details
-
-
-if __name__ == "__main__":
-    import json
-
-    dir_info = get_dir_info(Path("."), max_depth=6)
-    with open(".hashes.json", "w") as f:
-        json.dump(dir_info, f, indent=4)
-    print("Wrote .hashes.json")
-
-# Example output:
-# {
-#     ".": {
-#         "basename": "misc",
-#         "mime_type": "inode/directory",
-#         "extension": "",
-#         "num_subpaths": 25,
-#         "num_bytes": 214677,
-#         "hash_sha256": "addfacf88b2ff6b564846415fb7b21dcb7e63ee4e911bc0aec255ee354958530",
-#         "created_at": "2024-12-04T00:08:38.537449",
-#         "modified_at": "2024-12-04T00:08:38.537449"
-#     },
-#     "__init__.py": {
-#         "basename": "__init__",
-#         "mime_type": "text/x-python",
-#         "extension": ".py",
-#         "num_subpaths": null,
-#         "num_bytes": 32,
-#         "hash_sha256": "b0e5e7ff17db3b60535cf664282787767c336e3e203a43e21b6326c6fe457551",
-#         "created_at": "2024-10-08T00:51:41.001359",
-#         "modified_at": "2024-10-08T00:51:41.001359"
-#     },
-#     ...
-# }

archivebox/misc/jsonl.py

@@ -32,7 +32,6 @@ import sys
 import json
 from typing import Any, TextIO
 from collections.abc import Iterable, Iterator
-from pathlib import Path
 
 
 # Type constants for JSONL records
@@ -120,19 +119,6 @@ def read_stdin(stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
             yield record
 
 
-def read_file(path: Path) -> Iterator[dict[str, Any]]:
-    """
-    Read JSONL or plain URLs from a file.
-
-    Yields parsed records as dicts.
-    """
-    with open(path) as f:
-        for line in f:
-            record = parse_line(line)
-            if record:
-                yield record
-
-
 def read_args_or_stdin(args: Iterable[str], stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
     """
     Read from CLI arguments if provided, otherwise from stdin.

archivebox/misc/serve_static.py

@@ -910,6 +910,9 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 )
 
     content_type, encoding = mimetypes.guess_type(str(fullpath))
+    preserve_plain_text = fullpath.suffix.lower() in {".log", ".sh"}
+    if preserve_plain_text:
+        content_type = "text/plain"
     content_type = content_type or "application/octet-stream"
     # Add charset for text-like types (best guess), but don't override the type.
     is_text_like = content_type.startswith("text/") or content_type in {
@@ -1020,7 +1023,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
 
     # Heuristic fix: some archived HTML outputs are stored with HTML-escaped markup
     # or markdown sources. If so, render sensibly.
-    if content_type.startswith(("text/plain", "text/html")):
+    if not preserve_plain_text and content_type.startswith(("text/plain", "text/html")):
         try:
             max_unescape_size = 10 * 1024 * 1024  # 10MB cap to avoid heavy memory use
             if statobj.st_size <= max_unescape_size:

archivebox/misc/toml_util.py

@@ -2,94 +2,11 @@ from typing import Any, cast
 from collections.abc import Callable
 
 import json
-import ast
-import inspect
 import toml
 import re
-import configparser
 
 from pathlib import Path, PosixPath
 
-from pydantic.json_schema import GenerateJsonSchema
-from pydantic_core import to_jsonable_python
-
-JSONValue = str | bool | int | None | list["JSONValue"]
-
-TOML_HEADER = "# Converted from INI to TOML format: https://toml.io/en/\n\n"
-
-
-def load_ini_value(val: str) -> JSONValue:
-    """Convert lax INI values into strict TOML-compliant (JSON) values"""
-    if val.lower() in ("true", "yes", "1"):
-        return True
-    if val.lower() in ("false", "no", "0"):
-        return False
-    if val.isdigit():
-        return int(val)
-
-    try:
-        return ast.literal_eval(val)
-    except Exception:
-        pass
-
-    try:
-        return json.loads(val)
-    except Exception:
-        pass
-
-    return val
-
-
-def convert(ini_str: str) -> str:
-    """Convert a string of INI config into its TOML equivalent (warning: strips comments)"""
-
-    config = configparser.ConfigParser()
-    setattr(config, "optionxform", str)  # capitalize key names
-    config.read_string(ini_str)
-
-    # Initialize an empty dictionary to store the TOML representation
-    toml_dict = {}
-
-    # Iterate over each section in the INI configuration
-    for section in config.sections():
-        toml_dict[section] = {}
-
-        # Iterate over each key-value pair in the section
-        for key, value in config.items(section):
-            parsed_value = load_ini_value(value)
-
-            # Convert the parsed value to its TOML-compatible JSON representation
-            toml_dict[section.upper()][key.upper()] = json.dumps(parsed_value)
-
-    # Build the TOML string
-    toml_str = TOML_HEADER
-    for section, items in toml_dict.items():
-        toml_str += f"[{section}]\n"
-        for key, value in items.items():
-            toml_str += f"{key} = {value}\n"
-        toml_str += "\n"
-
-    return toml_str.strip()
-
-
-class JSONSchemaWithLambdas(GenerateJsonSchema):
-    """
-    Encode lambda functions in default values properly.
-    Usage:
-    >>> json.dumps(value, encoder=JSONSchemaWithLambdas())
-    """
-
-    def encode_default(self, dft: Any) -> Any:
-        config = self._config
-        if isinstance(dft, Callable):
-            return "{{lambda " + inspect.getsource(dft).split("=lambda ")[-1].strip()[:-1] + "}}"
-        return to_jsonable_python(
-            dft,
-            timedelta_mode=config.ser_json_timedelta,
-            bytes_mode=config.ser_json_bytes,
-            serialize_unknown=True,
-        )
-
 
 def better_toml_dump_str(val: Any) -> str:
     try:

archivebox/personas/export_browser_state.js

@@ -9,6 +9,7 @@
  *   COOKIES_OUTPUT_FILE         Optional output path for Netscape cookies.txt
  *   AUTH_STORAGE_OUTPUT_FILE    Optional output path for auth.json
  *   CHROME_BINARY               Absolute browser path resolved by abxpkg
+ *   CHROME_SANDBOX              Set to false when the Linux sandbox is unavailable
  *   NODE_MODULES_DIR            node_modules path resolved by abxpkg
  */
 
@@ -121,13 +122,16 @@ async function openBrowser() {
 
     // The ordinary archiving launcher uses a mock keychain. An import must
     // instead let the original signed browser decrypt its own profile cookies.
+    const sandboxDisabled = ['false', '0', 'no', 'off'].includes(
+        String(process.env.CHROME_SANDBOX || '').trim().toLowerCase(),
+    );
     const browser = await puppeteer.launch({
         executablePath: binary,
         userDataDir,
         headless: true,
         ignoreDefaultArgs: ['--use-mock-keychain', '--password-store=basic'],
         args: ['--no-first-run', '--disable-sync', '--disable-background-networking',
-               '--profile-directory=Default', ...(process.platform === 'linux' && process.getuid?.() === 0 ? ['--no-sandbox'] : [])],
+               '--profile-directory=Default', ...(process.platform === 'linux' && (process.getuid?.() === 0 || sandboxDisabled) ? ['--no-sandbox'] : [])],
     });
     return {
         browser,

archivebox/personas/importers.py

@@ -504,8 +504,10 @@ def resolve_custom_import_source(raw_value: str, profile_dir: str | None = None)
             selected = explicit_profile or pick_default_profile_dir(root)
             if selected:
                 return resolve_browser_profile_source(
-                    browser="persona", source_name=source_path.name,
-                    user_data_dir=root.resolve(), profile_dir=selected,
+                    browser="persona",
+                    source_name=source_path.name,
+                    user_data_dir=root.resolve(),
+                    profile_dir=selected,
                 )
     if _looks_like_profile_dir(source_path):
         if explicit_profile and explicit_profile != source_path.name:

archivebox/plugins/discovery.py

@@ -55,10 +55,6 @@ def get_plugins() -> list[str]:
     return sorted(get_plugin_catalog())
 
 
-def get_plugin_models():
-    return get_plugin_catalog().plugins
-
-
 def get_plugin_name(plugin: str) -> str:
     """
     Get the base plugin name without numeric prefix.

archivebox/progressmonitor/collection.py

@@ -0,0 +1,54 @@
+"""Non-blocking, database-only collection totals for the progress monitor."""
+
+import logging
+from threading import Lock, Thread
+from time import time
+
+from django.core.cache import cache
+from django.db import close_old_connections, connections
+from django.db.models import Count, Sum
+
+
+_refresh_lock = Lock()
+
+
+def collection_summary(user):
+    user_id = None if user.is_superuser else user.pk
+    key = f"progress-collection:{'all' if user_id is None else user_id}"
+    summary = cache.get(key)
+    # Even an indexed SUM scans the index. Never make an HTTP request wait for
+    # that scan on a large collection. Serve stale totals while one worker refreshes;
+    # a single non-blocking lock prevents a queue of scans from concurrent clients.
+    if (summary is None or time() - summary["sampled_at"] >= 60) and _refresh_lock.acquire(blocking=False):
+        # Throttle failures too: a broken/locked database must not trigger work
+        # and log output on every sidebar poll. This cache is local to each worker.
+        if not cache.add(f"{key}:refresh", True, timeout=60):
+            _refresh_lock.release()
+            return summary
+
+        def refresh():
+            try:
+                close_old_connections()
+                from archivebox.core.models import Snapshot
+
+                snapshots = Snapshot.objects.all()
+                if user_id is not None:
+                    snapshots = snapshots.filter(crawl__created_by_id=user_id)
+                # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
+                # Snapshot.output_size is maintained when ArchiveResult sizes change.
+                totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+                totals["bytes"] = totals["bytes"] or 0
+                totals["sampled_at"] = time()
+                cache.set(key, totals, timeout=3600)
+            except Exception:
+                logging.getLogger(__name__).exception("Could not refresh collection totals")
+            finally:
+                connections.close_all()
+                _refresh_lock.release()
+
+        try:
+            Thread(target=refresh, name="collection-summary", daemon=True).start()
+        except RuntimeError:
+            _refresh_lock.release()
+            logging.getLogger(__name__).exception("Could not start collection totals refresh")
+    return summary

archivebox/progressmonitor/views.py

@@ -942,6 +942,10 @@ def live_progress_view(request):
             "queued_crawls_hidden": queued_crawls_hidden,
             "server_time": timezone.now().isoformat(),
         }
+        if is_admin and not snapshot_id_filter and not crawl_id_filter and request.GET.get("collection") == "1":
+            from archivebox.progressmonitor.collection import collection_summary
+
+            payload["collection"] = collection_summary(request.user)
         try:
             import ujson
 

archivebox/search/query.py

@@ -1,5 +1,6 @@
 __package__ = "archivebox.search"
 
+from subprocess import CalledProcessError
 from typing import Any
 
 from django.db import connection
@@ -220,6 +221,7 @@ def iter_query_search_ids(
     search_mode: str | None = None,
     config: dict[str, Any] | None = None,
     max_results: int | None = None,
+    stop_event=None,
     **config_kwargs: Any,
 ):
     """Yield snapshot IDs from configured search backend modules."""
@@ -266,6 +268,8 @@ def iter_query_search_ids(
     seen: set[str] = set()
     try:
         for backend_name in backend_names:
+            if stop_event is not None and stop_event.is_set():
+                return
             backend = backends[backend_name]
             try:
                 if backend_name == "sonic":
@@ -291,29 +295,34 @@ def iter_query_search_ids(
                     env=search_backend_command_env(config=config),
                     cwd=CONSTANTS.DATA_DIR,
                     timeout=max(1, int(config.get("TIMEOUT", 60))) * 4,
+                    stop_event=stop_event,
                 )
-                for snapshot_id in ids:
-                    if snapshot_id in seen:
-                        continue
-                    seen.add(snapshot_id)
-                    yield snapshot_id
-                    if max_results and len(seen) >= max_results:
-                        return
+                try:
+                    for snapshot_id in ids:
+                        if snapshot_id in seen:
+                            continue
+                        seen.add(snapshot_id)
+                        yield snapshot_id
+                        if max_results and len(seen) >= max_results:
+                            return
+                finally:
+                    ids.close()
                 successful_backends += 1
             except Exception as err:
                 errors.append(err)
                 if not fallback_enabled:
                     raise
+        if not successful_backends and errors and search_mode_base == "deep":
+            raise errors[0]
     except Exception as err:
         stderr()
         stderr(
             f"[X] The search backend threw an exception={err}:",
             color="red",
         )
+        if isinstance(err, CalledProcessError) and err.stderr:
+            stderr(err.stderr.rstrip(), color="red")
         raise
-    else:
-        if not successful_backends and errors and search_mode_base == "deep":
-            raise errors[0]
 
 
 @enforce_types

archivebox/search/views.py

@@ -6,7 +6,7 @@ import json
 import threading
 import time
 from copy import copy
-from queue import Full, Queue
+from queue import Empty, Full, Queue
 from urllib.parse import urlsplit
 from uuid import UUID
 
@@ -254,15 +254,18 @@ def iter_filtered_search_result_ids(iterator, queryset, *, flush_max_delay=0.05)
         yield from flush_batch()
 
 
-def iter_search_result_ids(query, base_queryset, *, search_mode, config):
+def iter_search_result_ids(query, base_queryset, *, search_mode, config, stop_event=None):
     """Yield filtered Snapshot IDs from the selected search provider."""
     search_mode_base = get_search_mode_base(search_mode, config=config)
     provider = (
         iter_meta_search_ids(query, base_queryset)
         if search_mode_base == "meta"
-        else iter_query_search_ids(query, search_mode=search_mode, config=config)
+        else iter_query_search_ids(query, search_mode=search_mode, config=config, stop_event=stop_event)
     )
-    yield from iter_filtered_search_result_ids(provider, base_queryset)
+    try:
+        yield from iter_filtered_search_result_ids(provider, base_queryset)
+    finally:
+        provider.close()
 
 
 def snapshot_search_stream_response(query, base_queryset, *, search_mode, config, cache_key, thread_name):
@@ -300,7 +303,7 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
             iterator = None
             try:
                 close_old_connections()
-                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config)
+                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config, stop_event=stop_event)
                 for snapshot_id in iterator:
                     if stop_event.is_set():
                         break
@@ -326,10 +329,13 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
                 emit(None)
 
         threading.Thread(target=run_search, name=thread_name, daemon=True).start()
-        yield f"0{stream_padding}\n"
         try:
+            yield f"0{stream_padding}\n"
             while True:
-                item = await asyncio.to_thread(queue.get)
+                try:
+                    item = await asyncio.to_thread(queue.get, True, 0.1)
+                except Empty:
+                    continue
                 if item is None:
                     break
                 if isinstance(item, BaseException):

archivebox/services/resource_admission.py

@@ -0,0 +1,91 @@
+"""Admission signals, not estimates of how much RAM an extractor will need."""
+
+import math
+import os
+import time
+from pathlib import Path
+
+
+RESOURCE_RECHECK_SECONDS = 2.0
+# Scheduling policy, not a minimum-RAM claim: stop increasing extraction
+# load when all non-idle work loses at least one second out of ten to memory
+# stalls. Require both recent and rolling pressure so transient reclaim does
+# not indefinitely starve otherwise viable work on small hosts.
+MEMORY_FULL_STALL_PERCENT = 10.0
+
+
+class ResourceAdmission:
+    def __init__(self):
+        self._memory_stalls: dict[str, tuple[int, float]] = {}
+        self._checked_at = float("-inf")
+        self._reason: str | None = None
+
+    @staticmethod
+    def cgroup_paths() -> list[Path]:
+        root = Path("/sys/fs/cgroup")
+        paths = [root]
+        try:
+            for line in Path("/proc/self/cgroup").read_text().splitlines():
+                if line.startswith("0::"):
+                    current = root / line[3:].lstrip("/")
+                    if current.is_dir() and current.is_relative_to(root):
+                        paths = [current, *[parent for parent in current.parents if parent.is_relative_to(root)]]
+                    break
+        except OSError:
+            pass
+        return paths
+
+    def snapshot_limit(self, configured: int) -> int:
+        cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 1)
+        for path in self.cgroup_paths():
+            try:
+                quota, period = (path / "cpu.max").read_text().split()
+                if quota != "max":
+                    cpus = min(cpus, max(1, math.ceil(int(quota) / int(period))))
+            except (OSError, ValueError, ZeroDivisionError):
+                continue
+        # CPU quota limits parallel admission, not forward progress. A host
+        # with a fractional CPU must still be able to run one snapshot.
+        return max(1, min(configured, cpus))
+
+    def observe_memory_stalls(self, source: str, total: int, avg10: float, sampled_at: float) -> bool:
+        previous = self._memory_stalls.get(source)
+        self._memory_stalls[source] = (total, sampled_at)
+        if previous is None:
+            return avg10 >= MEMORY_FULL_STALL_PERCENT
+        previous_total, previous_time = previous
+        elapsed = sampled_at - previous_time
+        if elapsed <= 0 or total < previous_total:
+            return False
+        stalled_percent = (total - previous_total) / (elapsed * 1_000_000) * 100
+        return avg10 >= MEMORY_FULL_STALL_PERCENT and stalled_percent >= MEMORY_FULL_STALL_PERCENT
+
+    def memory_pressure(self) -> str | None:
+        now = time.monotonic()
+        if now - self._checked_at < RESOURCE_RECHECK_SECONDS:
+            return self._reason
+        self._checked_at = now
+        reasons = []
+        cgroups = self.cgroup_paths()
+        for path in cgroups:
+            try:
+                current = int((path / "memory.current").read_text())
+                for name in ("memory.high", "memory.max"):
+                    limit = (path / name).read_text().strip()
+                    if limit != "max" and current >= int(limit):
+                        reasons.append(f"{name} reached")
+            except (OSError, ValueError):
+                continue
+        for path in [Path("/proc/pressure/memory"), *[p / "memory.pressure" for p in cgroups]]:
+            try:
+                full = next(line for line in path.read_text().splitlines() if line.startswith("full "))
+                fields = dict(field.split("=", 1) for field in full.split()[1:])
+                if self.observe_memory_stalls(str(path), int(fields["total"]), float(fields["avg10"]), now):
+                    reasons.append("memory stalls")
+            except (OSError, ValueError, KeyError, StopIteration):
+                continue
+        self._reason = ", ".join(dict.fromkeys(reasons)) or None
+        return self._reason
+
+
+resource_admission = ResourceAdmission()

archivebox/services/runner.py

@@ -58,6 +58,8 @@ from abxbus import BaseEvent
 from abxbus.event_bus import EventBus, get_current_event, in_handler_context
 from abxbus.event_handler import EventHandlerAbortedError, EventHandlerCancelledError
 
+from archivebox.services.resource_admission import RESOURCE_RECHECK_SECONDS, resource_admission
+
 from archivebox.config.common import (
     ArchiveBoxBaseConfig,
     normalize_runtime_config,
@@ -221,6 +223,8 @@ class CrawlRunner:
         self.snapshot_tasks: dict[str, asyncio.Task[None]] = {}
         self.snapshot_semaphore = asyncio.Semaphore(1)
         self.max_concurrent_snapshots = 1
+        self._resource_deferred = False
+        self._initial_snapshot_admitted = False
         self.persona = None
         self.base_config: ArchiveBoxBaseConfig | dict[str, Any] = {}
         self.derived_config: dict[str, Any] = {}
@@ -294,6 +298,7 @@ class CrawlRunner:
     async def run(self) -> None:
         root_snapshot_id: str | None = None
         bus_destroyed = False
+        run_state_loaded = False
         try:
             first_signal_message = (
                 "\n[🛑] Got {signal_name}, aborting the active hook...\n"
@@ -313,8 +318,15 @@ class CrawlRunner:
                 on_signal=self._request_abort_from_signal,
                 raise_on_first_signal=False,
             ):
+                if await sync_to_async(defer_crawl_for_resources, thread_sensitive=True)(self.crawl):
+                    return
+                # Startup and its first snapshot are one admission. Browser
+                # startup can itself cause reclaim; checking again immediately
+                # would tear it down and repeat startup without doing any work.
+                self._initial_snapshot_admitted = True
+                run_state_loaded = True
                 snapshot_ids = await sync_to_async(self.load_run_state, thread_sensitive=True)()
-                max_concurrent_snapshots = max(1, int(self.base_config.get("CRAWL_MAX_CONCURRENT_SNAPSHOTS", 1)))
+                max_concurrent_snapshots = resource_admission.snapshot_limit(int(self.base_config.get("CRAWL_MAX_CONCURRENT_SNAPSHOTS", 1)))
                 self.max_concurrent_snapshots = max_concurrent_snapshots
                 self.snapshot_semaphore = asyncio.Semaphore(max_concurrent_snapshots)
                 live_ui = self._create_live_ui()
@@ -344,10 +356,19 @@ class CrawlRunner:
                 except Exception:
                     pass
                 self._live_stream = None
-            await sync_to_async(project_abxpkg_derived_cache_to_db, thread_sensitive=True)(self.base_config.get("ABXPKG_LIB_DIR"))
-            await sync_to_async(self.finalize_run_state, thread_sensitive=True)()
+            if run_state_loaded:
+                await sync_to_async(project_abxpkg_derived_cache_to_db, thread_sensitive=True)(self.base_config.get("ABXPKG_LIB_DIR"))
+                await sync_to_async(self.finalize_run_state, thread_sensitive=True)()
 
     async def enqueue_snapshot(self, snapshot_id: str, crawl_start_event: CrawlStartEvent | None = None) -> None:
+        if self._resource_deferred:
+            return
+        if not self._initial_snapshot_admitted and await sync_to_async(defer_crawl_for_resources, thread_sensitive=True)(
+            self.crawl,
+            excluded_snapshot_ids=list(self.snapshot_tasks),
+        ):
+            self._resource_deferred = True
+            return
         if await self.crawl_is_cancelled():
             return
         if await self.crawl_is_paused() and not self.allow_maintenance_on_inactive_crawl:
@@ -362,6 +383,7 @@ class CrawlRunner:
             return
         else:
             task = asyncio.create_task(self.run_snapshot(snapshot_id), context=_runner_task_context())
+        self._initial_snapshot_admitted = False
         self.snapshot_tasks[snapshot_id] = task
 
     async def stop_snapshot_tasks(self) -> None:
@@ -488,6 +510,8 @@ class CrawlRunner:
         from archivebox.core.models import Snapshot
         from archivebox.config.common import get_config
 
+        if self._resource_deferred:
+            return
         if not isinstance(get_current_event(), CrawlStartEvent):
             return
         if await self.crawl_is_cancelled():
@@ -497,7 +521,7 @@ class CrawlRunner:
 
         await sync_to_async(self.crawl.refresh_from_db, thread_sensitive=True)()
         config = await sync_to_async(lambda: get_config(crawl=self.crawl), thread_sensitive=True)()
-        self.max_concurrent_snapshots = max(1, int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))
+        self.max_concurrent_snapshots = resource_admission.snapshot_limit(int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))
 
         active_snapshot_ids = [snapshot_id for snapshot_id, task in self.snapshot_tasks.items() if not task.done()]
         available_slots = max(0, self.max_concurrent_snapshots - len(active_snapshot_ids))
@@ -1266,6 +1290,29 @@ def run_snapshot_maintenance(snapshot_id: str, *, output_dir: Path | None = None
     return True
 
 
+def defer_crawl_for_resources(crawl, *, excluded_snapshot_ids=()) -> bool:
+    """Leave extraction durable; never delay cancellation or final cleanup."""
+    from archivebox.core.models import Snapshot
+
+    if crawl.status not in (crawl.StatusChoices.QUEUED, crawl.StatusChoices.STARTED):
+        return False
+    reason = resource_admission.memory_pressure()
+    if reason is None:
+        return False
+    pending = crawl.snapshot_set.filter(status__in=Snapshot.RUNNABLE_STATES).exclude(id__in=excluded_snapshot_ids)
+    if crawl.snapshot_set.exists() and not pending.exists():
+        return False
+    now = timezone.now()
+    retry_at = now + timedelta(seconds=RESOURCE_RECHECK_SECONDS)
+    # Conditional short updates cannot resurrect a concurrent pause/cancel.
+    type(crawl).objects.filter(id=crawl.id, status__in=[crawl.StatusChoices.QUEUED, crawl.StatusChoices.STARTED]).update(
+        retry_at=retry_at,
+        modified_at=now,
+    )
+    pending.filter(retry_at__lte=now).update(retry_at=retry_at, modified_at=now)
+    return True
+
+
 def run_due_crawl(crawl, *, lock_seconds: int, interactive_interrupts: bool = False) -> bool:
     with crawl_lifecycle_lock(str(crawl.id)):
         return _run_due_crawl_locked(

archivebox/templates/admin/base.html

@@ -1339,20 +1339,36 @@
                 display: inline-block;
             }
 
-            .action-rearchive-select,
-            .action-permissions-select {
+            /* The native select receives the actual click, including in WKWebView. */
+            body.change-list #changelist .actions .action-rearchive-select,
+            body.change-list #changelist .actions .action-permissions-select {
                 position: absolute;
-                left: 0;
-                bottom: 0;
-                width: 1px;
-                height: 1px;
+                inset: 0;
+                width: 100%;
+                height: 100%;
                 opacity: 0;
-                pointer-events: none;
-                border: 0;
-                padding: 0;
+                cursor: pointer;
+                z-index: 1;
                 margin: 0;
             }
 
+            body.change-list #changelist .action-rearchive-wrapper > .button,
+            body.change-list #changelist .action-permissions-wrapper > .button {
+                padding: 8px 16px !important;
+                width: max-content;
+                min-width: max-content;
+                flex: 0 0 auto;
+                gap: 6px;
+                white-space: nowrap;
+            }
+
+            .action-rearchive-wrapper:focus-within,
+            .action-permissions-wrapper:focus-within {
+                outline: 2px solid #2563eb;
+                outline-offset: 2px;
+                border-radius: 6px;
+            }
+
             /* Object count */
             .actions .action-counter {
                 color: #64748b;
@@ -1960,22 +1976,10 @@
                             .addClass('action-rearchive-select')
                             .attr('aria-label', 'Re-Archive')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .addClass('button')
                             .html('Re-Archive <span aria-hidden="true">▾</span>')
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         rearchiveActions.forEach(function (action_value) {
@@ -2021,23 +2025,11 @@
                             .addClass('action-permissions-select')
                             .attr('aria-label', 'Permissions')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .attr('name', action_type)
                             .addClass('button')
                             .text(this.text)
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         ;[

archivebox/templates/core/snapshot.html

@@ -1890,14 +1890,13 @@
 
             function attachPreviewFrameHandlers(frame) {
                 resetPreviewFrameSize(frame)
-                if (frame.src.endsWith('.pdf')) {
+                if (new URL(frame.src || 'about:blank', window.location.href).pathname.endsWith('.pdf')) {
                     frame.removeAttribute('sandbox')
                     frame.src = frame.src + '#toolbar=0'
                 }
                 frame.onload = function() {
-                    if (this.src.includes('.pdf')) {
+                    if (new URL(this.src || 'about:blank', window.location.href).pathname.endsWith('.pdf')) {
                         this.removeAttribute('sandbox')
-                        this.src = this.src.split('?autoplay=')[0] + '#toolbar=0'
                     }
                     tryCenterImageFrame(this)
                     try {
@@ -2083,7 +2082,10 @@
                     nextSrcAbs = new URL(nextSrc, window.location.href).href
                 } catch (err) {}
                 const iframe_elem = ensureMainFrame(currentSrc !== nextSrcAbs)
-                if (target.endsWith('.pdf')) {
+                // Preview wrappers retain the .pdf path but add ?preview=1.
+                // The native PDF renderer must be allowed before navigation,
+                // not after a failed sandboxed load has already completed.
+                if (new URL(target, window.location.href).pathname.endsWith('.pdf')) {
                     iframe_elem.removeAttribute('sandbox')
                 } else {
                     iframe_elem.sandbox = "allow-same-origin allow-top-navigation-by-user-activation allow-scripts allow-forms"

archivebox/templates/core/system_warnings_banner.html

@@ -34,7 +34,7 @@
     {% elif mode == "high_load" %}
       15-min loadavg <code style="background:#000;color:#fff;padding:1px 6px;border-radius:3px;font:700 11px/1.3 ui-monospace,Menlo,Consolas,monospace;text-transform:none;">{{ load_15 }}</code>
       exceeds <code style="background:#000;color:#fff;padding:1px 6px;border-radius:3px;font:700 11px/1.3 ui-monospace,Menlo,Consolas,monospace;text-transform:none;">{{ load_threshold }}</code>
-      ({{ cpu_count }} cores × 3) — the host is saturated; crawls are queuing faster than the runner can finish them.
+      ({{ cpu_count }} cores × 3) — recent CPU or I/O contention may slow crawls. This average can stay elevated after load subsides; it does not measure queue growth.
     {% elif mode == "unsafe" %}
       ArchiveBox single-domain mode — archived pages share an origin with this site
       <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Modes" target="_blank" rel="noopener noreferrer" style="color:#fff;text-decoration:underline;margin-left:6px;">why?</a>

archivebox/tests/test_api_v1_auth_browser_session.py

@@ -0,0 +1,56 @@
+from datetime import timedelta
+from urllib.parse import urlsplit
+
+import pytest
+from django.contrib.sessions.models import Session
+from django.utils import timezone
+
+from archivebox.api.models import APIToken
+from archivebox.tests.conftest import API_TEST_HOST, api_client_request
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+@pytest.mark.parametrize("header", ["HTTP_AUTHORIZATION", "HTTP_X_ARCHIVEBOX_API_KEY"])
+def test_browser_session_authenticates_admin_and_expires_with_key(client, api_admin_user, header):
+    token = APIToken.objects.create(created_by=api_admin_user, expires=timezone.now() + timedelta(minutes=5))
+    response = api_client_request(
+        client,
+        "post",
+        "/api/v1/auth/browser_session",
+        payload={},
+        headers={
+            "HTTP_HOST": API_TEST_HOST,
+            header: f"Bearer {token.token}" if header == "HTTP_AUTHORIZATION" else token.token,
+        },
+    )
+    assert response.status_code == 200, response.content
+    assert response["Cache-Control"] == "no-store"
+    data = response.json()
+    session = Session.objects.get(session_key=data["cookie"]["value"])
+    assert session.get_decoded()["_auth_user_id"] == str(api_admin_user.pk)
+    assert timezone.now() < session.expire_date <= token.expires
+    client.cookies.clear()
+    client.cookies[data["cookie"]["name"]] = data["cookie"]["value"]
+    response = client.get("/admin/", HTTP_HOST=urlsplit(data["admin_url"]).netloc)
+    assert response.status_code == 200, response.content
+    assert response.wsgi_request.user.pk == api_admin_user.pk
+
+
+@pytest.mark.parametrize("kind", ["missing", "invalid", "expired", "inactive", "nonadmin", "query"])
+def test_browser_session_rejects_invalid_credentials(client, api_admin_user, kind):
+    token = APIToken.objects.create(created_by=api_admin_user, expires=timezone.now() + timedelta(minutes=5))
+    if kind == "expired":
+        token.expires = timezone.now() - timedelta(seconds=1)
+        token.save()
+    if kind in ("inactive", "nonadmin"):
+        setattr(api_admin_user, "is_active" if kind == "inactive" else "is_superuser", False)
+        api_admin_user.save()
+    headers = {"HTTP_HOST": API_TEST_HOST}
+    if kind not in ("missing", "query"):
+        headers["HTTP_AUTHORIZATION"] = "Bearer " + ("invalid" if kind == "invalid" else token.token)
+    path = "/api/v1/auth/browser_session" + (f"?api_key={token.token}" if kind == "query" else "")
+    before = Session.objects.count()
+    response = api_client_request(client, "post", path, payload={}, headers=headers)
+    assert response.status_code in (401, 403), response.content
+    assert Session.objects.count() == before

archivebox/tests/test_auth_ldap.py

@@ -153,7 +153,7 @@ class TestArchiveBoxWithLDAP:
 
     def test_archivebox_init_without_ldap(self, tmp_path):
         """Test that archivebox init works without LDAP enabled."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["init"],
             cwd=tmp_path,
             timeout=45,
@@ -161,14 +161,13 @@ class TestArchiveBoxWithLDAP:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should succeed
-        assert code == 0, f"archivebox init failed: {stderr}"
+        assert result.returncode == 0, f"archivebox init failed: {result.stderr}"
 
     def test_archivebox_version_with_ldap_config(self, tmp_path):
         """Test that archivebox version works with LDAP config set."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["version"],
             cwd=tmp_path,
             timeout=10,
@@ -179,10 +178,9 @@ class TestArchiveBoxWithLDAP:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should succeed
-        assert code == 0, f"archivebox version failed: {stderr}"
+        assert result.returncode == 0, f"archivebox version failed: {result.stderr}"
 
 
 class TestLDAPConfigValidationInArchiveBox:
@@ -190,7 +188,7 @@ class TestLDAPConfigValidationInArchiveBox:
 
     def test_archivebox_init_with_incomplete_ldap_config(self, tmp_path):
         """Test that archivebox init fails with helpful error when LDAP config is incomplete."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["init"],
             cwd=tmp_path,
             timeout=45,
@@ -201,10 +199,9 @@ class TestLDAPConfigValidationInArchiveBox:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should fail with validation error
-        assert code != 0, "Should fail with incomplete LDAP config"
+        assert result.returncode != 0, "Should fail with incomplete LDAP config"
 
         # Check error message
-        assert "LDAP_* config options must all be set" in stderr, f"Expected validation error message in: {stderr}"
+        assert "LDAP_* config options must all be set" in result.stderr, f"Expected validation error message in: {result.stderr}"

archivebox/tests/test_cli_add.py

@@ -230,27 +230,23 @@ def test_add_single_url_records_url_in_crawl(initialized_archive):
     assert snapshots == []
 
 
-def test_add_memory_preflight_fails_before_creating_crawl(initialized_archive, capsys):
-    from archivebox.workers.supervisord_util import MIN_CRAWL_AVAILABLE_MEMORY_BYTES, require_crawl_memory
-
-    assert MIN_CRAWL_AVAILABLE_MEMORY_BYTES == 512 * 1024 * 1024
-    require_crawl_memory(MIN_CRAWL_AVAILABLE_MEMORY_BYTES)
-
+def test_add_bg_persists_queue_without_starting_extraction_workers(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--plugins=wget", "--crawl-max-concurrent-snapshots=4", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
     with use_archivebox_db(initialized_archive):
-        before_crawls = Crawl.objects.count()
-        before_processes = Process.objects.count()
-
-        with pytest.raises(SystemExit) as err:
-            require_crawl_memory(MIN_CRAWL_AVAILABLE_MEMORY_BYTES - 1)
-
-        assert Crawl.objects.count() == before_crawls
-        assert Process.objects.count() == before_processes
-
-    assert err.value.code == 1
-    output = capsys.readouterr().err
-    assert "Not enough available memory to archive a crawl" in output
-    assert "No crawl, runner, or Sonic workers were started" in output
-    assert "configure swap" in output
+        crawl = Crawl.objects.get()
+        assert crawl.status == Crawl.StatusChoices.QUEUED
+        assert crawl.retry_at is not None
+        assert crawl.get_urls_list() == ["https://example.com", "https://example.org"]
+        assert crawl.config["PLUGINS"] == "wget"
+        assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 4
+        assert not Snapshot.objects.exists()
+        assert not ArchiveResult.objects.exists()
+        assert not Process.objects.filter(process_type__in=[Process.TypeChoices.ORCHESTRATOR, Process.TypeChoices.HOOK]).exists()
 
 
 def test_initial_crawl_creation_does_not_warn_as_an_outside_runner_update(initialized_archive, caplog):
@@ -1208,7 +1204,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
     root_url = recursive_test_site["root_url"]
     child_url = recursive_test_site["child_urls"][0]
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         [
             "add",
             "--depth=2",
@@ -1223,8 +1219,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
         env=env,
         timeout=180,
     )
-    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert returncode == 0, stderr or stdout
+    assert result.returncode == 0, result.stderr or result.stdout
 
     with use_archivebox_db(initialized_archive):
         crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()

archivebox/tests/test_cli_archiveresult.py

@@ -142,18 +142,17 @@ class TestArchiveResultCreate:
         assert crawl_result.returncode == 0, crawl_result.stderr
         crawl_record = parse_jsonl_output(crawl_result.stdout)[0]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["archiveresult", "create"],
             stdin=crawl_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Passed through" in stderr
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        assert "Passed through" in result.stderr
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["id"] == crawl_record["id"]
 
@@ -163,16 +162,15 @@ class TestArchiveResultList:
 
     def test_list_empty(self, initialized_archive):
         """List with no archive results returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["archiveresult", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 archive results" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 archive results" in result.stderr
 
     def test_list_filter_by_status(self, initialized_archive):
         """Filter archive results by status."""

archivebox/tests/test_cli_crawl.py

@@ -33,19 +33,18 @@ class TestCrawlCreate:
         """Create crawl from URL arguments."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        assert "Created crawl" in stderr
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        assert "Created crawl" in result.stderr
 
         # Check JSONL output
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["type"] == "Crawl"
         assert url in records[0]["urls"]
@@ -55,18 +54,17 @@ class TestCrawlCreate:
         urls = [create_test_url() for _ in range(3)]
         stdin = "\n".join(urls)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         crawl = records[0]
         assert crawl["type"] == "Crawl"
@@ -78,32 +76,30 @@ class TestCrawlCreate:
         """Create crawl with --depth flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", "--depth=2", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert records[0]["max_depth"] == 2
 
     def test_create_with_tag(self, initialized_archive):
         """Create crawl with --tag flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", "--tag=test-tag", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert "test-tag" in records[0].get("tags_str", "")
 
     def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -119,17 +115,16 @@ class TestCrawlCreate:
         url = create_test_url()
         stdin = tag_result.stdout + json.dumps({"url": url}) + "\n"
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
         assert any(record.get("type") == "Crawl" and url in record["urls"] for record in records)
@@ -163,32 +158,30 @@ class TestCrawlList:
 
     def test_list_empty(self, initialized_archive):
         """List with no crawls returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 crawls" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 crawls" in result.stderr
 
     def test_list_returns_created(self, initialized_archive):
         """List returns previously created crawls."""
         url = create_test_url()
         run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
         assert any(url in r.get("urls", "") for r in records)
 
@@ -197,16 +190,15 @@ class TestCrawlList:
         url = create_test_url()
         run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list", "--status=queued"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         for r in records:
             assert r["status"] == "queued"
 
@@ -221,16 +213,15 @@ class TestCrawlList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list", "--limit=2"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
 

archivebox/tests/test_cli_list.py

@@ -3,6 +3,8 @@ Tests for archivebox list command.
 Verify list emits snapshot JSONL and applies the documented filters.
 """
 
+import csv
+import io
 import json
 from pathlib import Path
 
@@ -427,16 +429,15 @@ def test_snapshot_list_search_meta(initialized_archive):
     url = create_test_url(domain="meta-search-example.com")
     run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["snapshot", "list", "--search=meta", "meta-search-example.com"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, f"Command failed: {stderr}"
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, f"Command failed: {result.stderr}"
+    records = parse_jsonl_output(result.stdout)
     assert len(records) == 1
     assert "meta-search-example.com" in records[0]["url"]
 
@@ -446,16 +447,15 @@ def test_list_search_meta_matches_metadata(initialized_archive):
     url = create_test_url(domain="top-level-meta-search-example.com")
     run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["list", "--search=meta", "top-level-meta-search-example.com"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, f"Command failed: {stderr}"
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, f"Command failed: {result.stderr}"
+    records = parse_jsonl_output(result.stdout)
     assert len(records) == 1
     assert "top-level-meta-search-example.com" in records[0]["url"]
 
@@ -468,11 +468,10 @@ def test_search_command_finds_snapshots(initialized_archive):
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0, stderr
-    assert "example" in stdout
+    assert result.returncode == 0, result.stderr
+    assert "example" in result.stdout
 
 
 def test_search_command_returns_no_results_for_missing_term(initialized_archive):
@@ -483,22 +482,20 @@ def test_search_command_returns_no_results_for_missing_term(initialized_archive)
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "nonexistentterm12345"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code in [0, 1]
+    assert result.returncode in [0, 1]
 
 
 def test_search_command_on_empty_archive(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code in [0, 1]
+    assert result.returncode in [0, 1]
 
 
 def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive):
@@ -509,11 +506,10 @@ def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive)
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0, stderr
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, result.stderr
+    records = parse_jsonl_output(result.stdout)
     assert any("example.com" in row.get("url", "") for row in records)
 
 
@@ -576,52 +572,47 @@ def test_search_command_csv_outputs_requested_column(initialized_archive):
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "--csv", "url", "--with-headers"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, stderr
-    assert "url" in stdout
-    assert "example.com" in stdout
+    assert result.returncode == 0, result.stderr
+    assert list(csv.reader(io.StringIO(result.stdout))) == [["url"], ["https://example.com"]]
 
 
 def test_search_command_with_headers_requires_structured_output_format(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code != 0
-    assert "requires" in stderr.lower()
-    assert "json" in stderr.lower()
+    assert result.returncode != 0
+    assert "requires" in result.stderr.lower()
+    assert "json" in result.stderr.lower()
 
 
 def test_search_command_sort_option_runs_successfully(initialized_archive):
     for url in ["https://iana.org", "https://example.com"]:
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "--csv", "url", "--sort=url"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, stderr
-    assert "example.com" in stdout or "iana.org" in stdout
+    assert result.returncode == 0, result.stderr
+    assert list(csv.reader(io.StringIO(result.stdout))) == [["https://example.com"], ["https://iana.org"]]
 
 
 def test_search_command_help_lists_supported_filters(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0
-    assert "--url__icontains" in stdout
-    assert "--crawl-id" in stdout
-    assert "--status" in stdout
-    assert "--sort" in stdout
-    assert "--json" in stdout
-    assert "--html" in stdout
+    assert result.returncode == 0
+    assert "--url__icontains" in result.stdout
+    assert "--crawl-id" in result.stdout
+    assert "--status" in result.stdout
+    assert "--sort" in result.stdout
+    assert "--json" in result.stdout
+    assert "--html" in result.stdout

archivebox/tests/test_cli_machine.py

@@ -6,6 +6,32 @@ Tests for archivebox machine command.
 from archivebox.tests.conftest import run_archivebox_cmd
 
 
+def test_status_refreshes_week_old_machine(initialized_archive):
+    """A saved host refresh must not invalidate the instance still in use."""
+    import sqlite3
+    from datetime import datetime, timedelta, timezone
+
+    from archivebox.machine.detect import get_host_guid
+
+    # Age real rows instead of mocking time or save(): the failure happens
+    # when normal CLI startup refreshes a persisted host after seven days.
+    expired = (datetime.now(timezone.utc) - timedelta(days=8)).isoformat()
+    with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+        before = db.execute("SELECT id, guid FROM machine_machine ORDER BY id").fetchall()
+        assert before
+        db.execute("UPDATE machine_machine SET modified_at=?", (expired,))
+
+    result = run_archivebox_cmd(["status"], cwd=initialized_archive)
+    assert result.returncode == 0, result.stdout + result.stderr
+    with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+        after = db.execute("SELECT id, guid FROM machine_machine ORDER BY id").fetchall()
+        assert after == before
+        refreshed = db.execute("SELECT modified_at FROM machine_machine WHERE guid=?", (get_host_guid(),)).fetchone()
+        assert refreshed is not None
+        assert datetime.fromisoformat(refreshed[0]).replace(tzinfo=timezone.utc) > datetime.fromisoformat(expired)
+        assert all(row[0] == expired for row in db.execute("SELECT modified_at FROM machine_machine WHERE guid!=?", (get_host_guid(),)))
+
+
 def test_machine_help_runs_successfully(tmp_path):
     """The machine command should be registered and expose help."""
 

archivebox/tests/test_cli_persona.py

@@ -104,6 +104,7 @@ const puppeteer = require(process.argv[1]);
         ],
         cwd=initialized_archive,
         timeout=120,
+        env={**browser_env, "CHROME_SANDBOX": "false"},
     )
     assert imported.returncode == 0, imported.stderr
     persona = initialized_archive / "personas" / "session"

archivebox/tests/test_cli_run.py

@@ -53,15 +53,14 @@ def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archiv
     env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
     _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["snapshot", "create", recursive_test_site["root_url"]],
         cwd=initialized_archive,
         env=env,
         timeout=60,
     )
-    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert returncode == 0, stderr or stdout
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, result.stderr or result.stdout
+    records = parse_jsonl_output(result.stdout)
     snapshot_id = next(record["id"] for record in records if record.get("type") == "Snapshot")
     with use_archivebox_db(initialized_archive):
         browser_state = Snapshot.objects.get(id=snapshot_id).output_dir / "chrome" / "browser.json"
@@ -107,7 +106,7 @@ class TestRunWithCrawl:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -116,12 +115,11 @@ class TestRunWithCrawl:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
         # Should output the created Crawl
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         crawl_records = [r for r in records if r.get("type") == "Crawl"]
         assert len(crawl_records) >= 1
         assert crawl_records[0].get("id")  # Should have an id now
@@ -170,7 +168,7 @@ class TestRunWithSnapshot:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -179,11 +177,10 @@ class TestRunWithSnapshot:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         snapshot_records = [r for r in records if r.get("type") == "Snapshot"]
         assert len(snapshot_records) >= 1
         assert snapshot_records[0].get("id")
@@ -220,7 +217,7 @@ class TestRunWithSnapshot:
     def test_run_with_plain_url(self, initialized_archive):
         """Run accepts plain URL records (no type field)."""
         url = create_test_url()
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=url + "\n",
             cwd=initialized_archive,
@@ -229,10 +226,9 @@ class TestRunWithSnapshot:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
 
 
@@ -405,7 +401,7 @@ class TestRunRecovery:
             crawl_id = crawl.id
             snapshot_id = snapshot.id
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run", "--maintenance-only"],
             cwd=initialized_archive,
             timeout=90,
@@ -413,13 +409,12 @@ class TestRunRecovery:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, stdout + stderr
-        assert "Repairing" in stderr
-        assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in stderr
-        assert "interrupted before" in stderr
-        assert "remaining URLs" in stderr
+        assert result.returncode == 0, result.stdout + result.stderr
+        assert "Repairing" in result.stderr
+        assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in result.stderr
+        assert "interrupted before" in result.stderr
+        assert "remaining URLs" in result.stderr
 
         with use_archivebox_db(initialized_archive):
             crawl = Crawl.objects.get(id=crawl_id)
@@ -444,17 +439,16 @@ class TestRunPassThrough:
         assert tag_result.returncode == 0, tag_result.stderr
         tag_record = parse_jsonl_output(tag_result.stdout)[0]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         tag_records = [record for record in records if record.get("type") == "Tag"]
         assert len(tag_records) == 1
         assert tag_records[0]["id"] == tag_record["id"]
@@ -471,7 +465,7 @@ class TestRunPassThrough:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -480,10 +474,9 @@ class TestRunPassThrough:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         # Should have at least the Crawl in output
         assert len(records) >= 1
 
@@ -516,7 +509,7 @@ class TestRunMixedInput:
         )
         assert snapshot_result.returncode == 0, snapshot_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout + snapshot_result.stdout,
             cwd=initialized_archive,
@@ -525,10 +518,9 @@ class TestRunMixedInput:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         types = {record.get("type") for record in records}
         assert {"Crawl", "Snapshot", "Tag"}.issubset(types)
@@ -539,16 +531,15 @@ class TestRunEmpty:
 
     def test_run_empty_stdin(self, initialized_archive):
         """Run with empty stdin returns success."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin="",
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
+        assert result.returncode == 0
 
     def test_run_no_runnable_records_to_process(self, initialized_archive):
         """Run with only a real non-runnable Tag reports no work."""
@@ -560,17 +551,16 @@ class TestRunEmpty:
         )
         assert tag_result.returncode == 0, tag_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "No records to process" in stderr
+        assert result.returncode == 0
+        assert "No records to process" in result.stderr
 
 
 class TestRunDaemonMode:

archivebox/tests/test_cli_server.py

@@ -124,10 +124,17 @@ def test_server_auth_secret_and_cookie_settings_are_restart_stable(tmp_path):
     assert first_lines[3:] == ["None", "False", "False"]
 
 
-def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
+@pytest.mark.parametrize(
+    "base_url, secure",
+    [
+        ("https://archive.example.com", True),
+        ("", False),
+    ],
+)
+def test_base_url_controls_secure_cookies(tmp_path, base_url, secure):
     (tmp_path / ".archivebox_id").write_text("testcoll")
     env = os.environ.copy()
-    env["BASE_URL"] = "https://archive.example.com"
+    env["BASE_URL"] = base_url
     env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
     repo_root = Path(__file__).resolve().parents[2]
     env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
@@ -155,45 +162,8 @@ def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
     )
 
     assert json.loads(result.stdout) == {
-        "csrf_secure": True,
-        "session_secure": True,
-        "proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
-    }
-
-
-def test_unconfigured_base_url_enables_proxy_ssl_header_without_secure_cookies(tmp_path):
-    (tmp_path / ".archivebox_id").write_text("testcoll")
-    env = os.environ.copy()
-    env["BASE_URL"] = ""
-    env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
-    repo_root = Path(__file__).resolve().parents[2]
-    env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
-
-    result = subprocess.run(
-        [
-            sys.executable,
-            "-c",
-            (
-                "import django, json;"
-                "django.setup();"
-                "from django.conf import settings;"
-                "print(json.dumps({"
-                "'csrf_secure': settings.CSRF_COOKIE_SECURE,"
-                "'session_secure': settings.SESSION_COOKIE_SECURE,"
-                "'proxy_ssl_header': settings.SECURE_PROXY_SSL_HEADER,"
-                "}))"
-            ),
-        ],
-        capture_output=True,
-        text=True,
-        check=True,
-        env=env,
-        cwd=tmp_path,
-    )
-
-    assert json.loads(result.stdout) == {
-        "csrf_secure": False,
-        "session_secure": False,
+        "csrf_secure": secure,
+        "session_secure": secure,
         "proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
     }
 
@@ -759,9 +729,8 @@ def test_live_server_signal_exit_and_resume_uses_existing_supervisor_state(initi
 
         resumed = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}-resumed.log", env=env)
         resumed_log = resumed.log_path
-        _cmd_result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
+        assert result.returncode == 0, result.stderr or result.stdout
 
         os.kill(resumed.pid, signal.SIGTERM)
         resumed.wait(timeout=20)
@@ -783,14 +752,13 @@ def test_live_daemonized_server_keeps_supervisord_owned_by_archivebox_parent(ini
     port = get_free_port()
     bind_url = f"http://127.0.0.1:{port}"
     try:
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["server", "--daemonize", f"127.0.0.1:{port}"],
             cwd=initialized_archive,
             env=env,
             timeout=90,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         _wait_for_archivebox_workers(initialized_archive, env, ("worker_daphne", "worker_runner"), timeout=30)
         assert_port_open("127.0.0.1", port, timeout=30)
 

archivebox/tests/test_cli_snapshot.py

@@ -33,18 +33,17 @@ class TestSnapshotCreate:
         """Create snapshot from URL arguments."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        assert "Created" in stderr
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        assert "Created" in result.stderr
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["type"] == "Snapshot"
         assert records[0]["url"] == url
@@ -83,16 +82,15 @@ class TestSnapshotCreate:
         """Create snapshot with --tag flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create", "--tag=test-tag", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert "test-tag" in records[0].get("tags", "")
 
     def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -108,17 +106,16 @@ class TestSnapshotCreate:
         url = create_test_url()
         stdin = tag_result.stdout + url + "\n"
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
         assert any(record.get("type") == "Snapshot" and record["url"] == url for record in records)
@@ -127,16 +124,15 @@ class TestSnapshotCreate:
         """Create snapshots from multiple URLs."""
         urls = [create_test_url() for _ in range(3)]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create"] + urls,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 3
 
         created_urls = {r["url"] for r in records}
@@ -149,32 +145,30 @@ class TestSnapshotList:
 
     def test_list_empty(self, initialized_archive):
         """List with no snapshots returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 snapshots" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 snapshots" in result.stderr
 
     def test_list_returns_created(self, initialized_archive):
         """List returns previously created snapshots."""
         url = create_test_url()
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
         assert any(r.get("url") == url for r in records)
 
@@ -183,16 +177,15 @@ class TestSnapshotList:
         url = create_test_url()
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--status=queued"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         for r in records:
             assert r["status"] == "queued"
 
@@ -201,16 +194,15 @@ class TestSnapshotList:
         url = create_test_url(domain="unique-domain-12345.com")
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--url__icontains=unique-domain-12345"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert "unique-domain-12345" in records[0]["url"]
 
@@ -224,16 +216,15 @@ class TestSnapshotList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--limit=2"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
     def test_list_with_sort_and_limit(self, initialized_archive):
@@ -246,16 +237,15 @@ class TestSnapshotList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--limit=2", "--sort=-created_at"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
 

archivebox/tests/test_permissions.py

@@ -177,6 +177,40 @@ def test_permission_repairs_avoid_recursive_collection_and_abxpkg_chown():
     assert "chown -R" not in abxpkg_repairs
 
 
+def test_docker_entrypoint_uses_non_root_functional_checks_before_metadata_repairs():
+    entrypoint = (Path(__file__).parents[2] / "bin" / "docker_entrypoint.sh").read_text(encoding="utf-8")
+    ensure_dir = entrypoint.partition("ensure_dir() {")[2].partition("\n}")[0]
+    writable_probe = entrypoint.partition("assert_writable_dir() {")[2].partition("\n}")[0]
+
+    assert 'run_as_archivebox mkdir -p "$path"' in ensure_dir
+    assert 'path_is_writable "$path" && return 0' in ensure_dir
+    assert ensure_dir.index('path_is_writable "$path" && return 0') < ensure_dir.index('chown_if_needed "$path"')
+    assert ensure_dir.index('path_is_writable "$path" && return 0') < ensure_dir.index('chmod_if_possible "$path"')
+    assert 'run_as_archivebox test -w "$path"' in entrypoint
+    assert 'run_as_archivebox test -x "$path"' in entrypoint
+    assert 'run_as_archivebox mktemp "$path/.permissions_test.XXXXXX"' in writable_probe
+    assert 'run_as_archivebox rm -f "$probe"' in writable_probe
+    assert "du -s" not in entrypoint
+    assert "du -sb" not in entrypoint
+    assert 'find "$DATA_DIR"' not in entrypoint
+    assert 'find "$PERSONAS_DIR"' not in entrypoint
+    for line in entrypoint.splitlines():
+        assert not ("chown -R" in line and ("$DATA_DIR" in line or "$DATA_DIR/archive" in line))
+        assert not ("chmod -R" in line and ("$DATA_DIR" in line or "$DATA_DIR/archive" in line))
+
+
+def test_docker_entrypoint_keeps_root_pgid_and_validates_it_in_real_docker():
+    entrypoint = (Path(__file__).parents[2] / "bin" / "docker_entrypoint.sh").read_text(encoding="utf-8")
+    docker_validator = (Path(__file__).parents[2] / "bin" / "validate_docker_uid_gid.sh").read_text(encoding="utf-8")
+
+    assert 'TARGET_UID="${PUID:-$DETECTED_UID}"' in entrypoint
+    assert 'TARGET_GID="${PGID:-$DETECTED_GID}"' in entrypoint
+    assert '[[ "$TARGET_UID" == "0" ]]' in entrypoint
+    assert '[[ "$TARGET_GID" == "0" ]]' not in entrypoint.partition('[[ "$TARGET_UID" == "0" ]]')[2].partition("fi")[0]
+    assert 'run_case "explicit non-root PUID with PGID zero is supported"' in docker_validator
+    assert '"PUID=1201 PGID=0" "-" pass 1201 0' in docker_validator
+
+
 def test_root_handoff_never_selects_filesystem_root():
     from archivebox.config.permissions import root_data_dir_handoff_paths
 

archivebox/tests/test_recursive_crawl.py

@@ -635,7 +635,22 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
         for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
         if not (status in allowed_statuses or (plugin == "archivedotorg" and status == ArchiveResult.StatusChoices.FAILED))
     ]
-    assert not unexpected_results
+    # This fixture serves HTTP. TLSNotary must reject every plaintext document;
+    # a setup, timeout, or verifier error must not count as that expected refusal.
+    tlsnotary_results = [row for row in archive_results if row[3] == "tlsnotary"]
+    assert {row[1] for row in tlsnotary_results} == expected_urls
+    assert len(tlsnotary_results) == len(expected_urls)
+    with use_archivebox_db(initialized_archive):
+        for snapshot_id, _url, _depth, _plugin, hook_name, status, files, size, _output in tlsnotary_results:
+            assert status == ArchiveResult.StatusChoices.FAILED
+            assert not files
+            assert size == 0
+            process = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="tlsnotary", hook_name=hook_name).process
+            assert process is not None
+            assert process.status == Process.StatusChoices.EXITED
+            assert process.exit_code == 1
+            assert process.stderr.strip() == "[tlsnotary] TLSNotary requires an HTTPS document"
+    assert not [result for result in unexpected_results if result["plugin"] != "tlsnotary"]
     ytdlp_results = [
         (url, status, output_str)
         for _snapshot_id, url, _depth, plugin, _hook_name, status, _files, _size, output_str in archive_results
@@ -682,19 +697,6 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
     assert set(recursive_test_site["deep_urls"]).issubset(parsed_urls)
 
     assert processes
-    failed_hook_results = [
-        {
-            "url": url,
-            "depth": depth,
-            "plugin": plugin,
-            "hook_name": hook_name,
-            "status": status,
-            "output_str": output_str,
-        }
-        for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
-        if status == ArchiveResult.StatusChoices.FAILED and plugin != "archivedotorg"
-    ]
-    assert not failed_hook_results
     assert all(status == Process.StatusChoices.EXITED for _id, _pwd, _cmd, status, _exit_code, _started_at, _ended_at in processes)
 
     intervals = []

archivebox/tests/test_resource_admission.py

@@ -0,0 +1,30 @@
+import os
+
+from archivebox.services.resource_admission import ResourceAdmission
+
+
+def test_cpu_admission_respects_process_affinity():
+    admission = ResourceAdmission()
+    available = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()
+    assert 1 <= admission.snapshot_limit(100000) <= available
+    assert admission.snapshot_limit(1) == 1
+
+
+def test_memory_stall_counter_recovers_without_resetting_monitor():
+    # Real kernel PSI units: cumulative microseconds and ten-second average.
+    # Decision samples exercise the policy, not simulated hooks/processes.
+    admission = ResourceAdmission()
+    assert admission.observe_memory_stalls("host", 100, 0.0, 0.0) is False
+    assert admission.observe_memory_stalls("host", 400100, 20.0, 2.0) is True
+    assert admission.observe_memory_stalls("host", 400100, 20.0, 4.0) is False
+    assert admission.observe_memory_stalls("new-cgroup", 900, 20.0, 0.0) is True
+    assert admission.observe_memory_stalls("new-cgroup", 900, 20.0, 2.0) is False
+
+
+def test_transient_and_low_memory_stalls_do_not_starve_admission():
+    admission = ResourceAdmission()
+    for sample in range(20):
+        assert admission.observe_memory_stalls("host", sample * 2600, 0.13, sample * 2.0) is False
+    # One large interval alone is not sustained pressure; the rolling average
+    # must agree before deferring work.
+    assert admission.observe_memory_stalls("host", 1_000_000, 0.13, 40.0) is False

archivebox/tests/test_search.py

@@ -1,5 +1,6 @@
 import os
 import re
+import shutil
 import time
 from datetime import timedelta
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -745,6 +746,14 @@ class TestSearchBackendsE2E:
         )
         assert Path(binary_env["RIPGREP_BINARY"]).is_file()
         assert Path(binary_env["SONIC_BINARY"]).is_file()
+        # abxpkg may replace both env/bin and provider install paths on later
+        # hydration. Keep a real executable outside its managed lib tree so
+        # every index/query process uses the same on-disk KV format.
+        pinned_sonic_dir = initialized_archive / "pinned-sonic"
+        pinned_sonic_dir.mkdir()
+        sonic_binary = pinned_sonic_dir / "sonic"
+        shutil.copy2(Path(binary_env["SONIC_BINARY"]).resolve(strict=True), sonic_binary)
+        binary_env["SONIC_BINARY"] = str(sonic_binary)
 
         page_count = 23
         first_batch_count = 12
@@ -939,7 +948,7 @@ class TestSearchBackendsE2E:
                 )
                 assert create_result.returncode == 0, create_result.stderr or create_result.stdout
                 metadata_create_outputs.append(create_result.stdout)
-            from archivebox.core.models import Snapshot
+            from archivebox.core.models import ArchiveResult, Snapshot
             from archivebox.tests.test_orm_helpers import use_archivebox_db
 
             with use_archivebox_db(initialized_archive):
@@ -972,6 +981,17 @@ class TestSearchBackendsE2E:
                 timeout=180,
             )
             assert index_update.returncode == 0, index_update.stderr or index_update.stdout
+            with use_archivebox_db(initialized_archive):
+                sonic_results = list(
+                    ArchiveResult.objects.filter(
+                        plugin="search_backend_sonic",
+                        snapshot__url__in=[*first_wget_urls[:3], *second_wget_urls[:2]],
+                    ).values_list("snapshot__url", "status", "output_str", "process__stderr"),
+                )
+            assert {url for url, status, _, _ in sonic_results if status == "succeeded"} == {
+                *first_wget_urls[:3],
+                *second_wget_urls[:2],
+            }, (sonic_results, index_update.stdout, index_update.stderr)
 
             cli_backend_expectations = (
                 ("ripgrep", shared_content_needle, matrix_urls),
@@ -986,7 +1006,12 @@ class TestSearchBackendsE2E:
                     env={**env, "SEARCH_BACKEND_ENGINE": backend_name},
                     timeout=60,
                 )
-                assert backend_result.returncode == 0, backend_result.stderr or backend_result.stdout
+                sonic_log = initialized_archive / "logs" / "worker_sonic.log"
+                assert backend_result.returncode == 0, (
+                    backend_result.stderr or backend_result.stdout,
+                    sonic_log.read_text(encoding="utf-8", errors="replace") if backend_name == "sonic" and sonic_log.is_file() else "",
+                    str(sonic_binary),
+                )
                 backend_urls = [line.strip().strip('"') for line in backend_result.stdout.splitlines() if line.strip()]
                 assert set(backend_urls) == set(expected_urls), (backend_name, query, backend_result.stdout)
 
@@ -1264,3 +1289,75 @@ class TestSearchBackendsE2E:
             fixture_server.shutdown()
             fixture_server.server_close()
             fixture_thread.join(timeout=5)
+
+
+@pytest.mark.timeout(90)
+def test_live_public_search_disconnect_reaps_backend(initialized_archive):
+    """A real HTTP disconnect cancels a silent ripgrep process and its wrapper."""
+    import signal
+    from uuid import uuid4
+
+    import psutil
+
+    from archivebox.core.models import Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    rg = shutil.which("rg")
+    assert rg
+    query = "disconnect" + uuid4().hex
+    port = get_free_port()
+    env = cli_env(
+        live=True,
+        PUBLIC_INDEX="True",
+        PERMISSIONS="public",
+        SEARCH_BACKEND_ENGINE="ripgrep",
+        SEARCH_BACKEND_RIPGREP_ENABLED="True",
+        RIPGREP_BINARY=rg,
+        RIPGREP_ARGS_EXTRA='["--no-mmap", "--text", "--sort", "path"]',
+        RIPGREP_TIMEOUT="30",
+        ALLOWED_HOSTS="*",
+    )
+    created = run_archivebox_cmd(
+        ["snapshot", "create", "--status=sealed", "https://example.com/disconnect-test"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert created.returncode == 0, created.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(url="https://example.com/disconnect-test")
+        directory = snapshot.output_dir / "dom"
+        directory.mkdir(parents=True, exist_ok=True)
+        (directory / "a.txt").write_text(query + "\n")
+        with (directory / "z.txt").open("wb") as large:
+            large.truncate(8 * 1024**3)
+            for offset in range(0, 8 * 1024**3, 1024**2):
+                large.seek(offset)
+                large.write(b"\n")
+    server = start_archivebox_server(initialized_archive, port=port, env=env, daemonize=False, log_name="disconnect-server.log")
+    children = []
+    try:
+        with requests.get(
+            f"http://127.0.0.1:{port}/public/search-stream/",
+            params={"q": query, "search_mode": "deep:ripgrep"},
+            headers={"Host": f"web.archivebox.localhost:{port}"},
+            stream=True,
+            timeout=(5, 10),
+        ) as response:
+            assert response.status_code == 200, response.text
+            lines = response.iter_lines()
+            assert int(next(lines).strip()) == 0
+            assert int(next(lines).strip()) == 1
+            children = [child for child in psutil.Process(server.pid).children(recursive=True) if query in " ".join(child.cmdline())]
+            assert len(children) == 2
+            for child in children:
+                if Path(child.exe()).name == Path(rg).name:
+                    child.send_signal(signal.SIGSTOP)
+        _, alive = psutil.wait_procs(children, timeout=5)
+        assert not alive, [(child.pid, child.status()) for child in alive]
+    finally:
+        for child in children:
+            try:
+                child.kill()
+            except psutil.NoSuchProcess:
+                pass
+        stop_archivebox_process(server)

archivebox/tests/test_serve_static.py

@@ -1,10 +1,26 @@
 from pathlib import Path
 
+import pytest
 from django.test import RequestFactory
 
 from archivebox.misc.serve_static import serve_static_with_byterange_support
 
 
+@pytest.mark.parametrize("filename", ["output.log", "hook.sh"])
+@pytest.mark.parametrize("byte_range", [None, "bytes=2-5"])
+def test_logs_and_shell_scripts_are_served_as_plain_text(tmp_path: Path, filename: str, byte_range: str | None):
+    content = b"# Heading\n\n- item\n- another item\n\n<literal>\n"
+    (tmp_path / filename).write_bytes(content)
+    request = RequestFactory().get(f"/{filename}", **({"HTTP_RANGE": byte_range} if byte_range else {}))
+
+    response = serve_static_with_byterange_support(request, filename, document_root=tmp_path)
+
+    assert response.status_code == (206 if byte_range else 200)
+    assert response["Content-Type"] == "text/plain; charset=utf-8"
+    assert response["Content-Disposition"] == f'inline; filename="{filename}"'
+    assert b"".join(response.streaming_content) == (content[2:6] if byte_range else content)
+
+
 def test_archive_file_response_uses_async_iterator_under_asgi(tmp_path: Path):
     output = tmp_path / "screenshot" / "output.png"
     output.parent.mkdir()

archivebox/tests/test_server_security_browser.py

@@ -963,14 +963,13 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
             log_name="archivewebpage_server.log",
         )
         get_http_response(port, host=f"archivebox.localhost:{port}", path="/")
-        _cmd_result = run_archivebox_cmd(
+        command_result = run_archivebox_cmd(
             ["add", "--depth=0", "--max-urls=1", "--plugins=archivewebpage", url],
             cwd=initialized_archive,
             env=env,
             timeout=120,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, f"archivebox add failed:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
+        assert command_result.returncode == 0, f"archivebox add failed:\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
 
         capture = _get_archivewebpage_capture(initialized_archive, url)
         snapshot_host = f"{get_snapshot_subdomain(capture['snapshot_id'])}.archivebox.localhost:{port}"

archivebox/tests/test_snapshot_service.py

@@ -163,14 +163,15 @@ def test_snapshot_service_cli_add_seals_snapshot_and_writes_indexes(tmp_path, re
 
     port = get_free_port()
     env = cli_env(port=port, server=True, PLUGINS="wget", SAVE_WGET="True")
-    _cmd_result = run_archivebox_cmd(
+    command_result = run_archivebox_cmd(
         ["add", "--depth=0", "--plugins=wget", recursive_test_site["root_url"]],
         cwd=tmp_path,
         env=env,
         timeout=180,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert code == 0, f"archivebox add failed with code {code}\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
+    assert command_result.returncode == 0, (
+        f"archivebox add failed with code {command_result.returncode}\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
+    )
 
     state = _snapshot_state(tmp_path, recursive_test_site["root_url"])
     snapshot_dir = state["snapshot_dir"]

archivebox/tests/test_takeover_util.py

@@ -3,6 +3,9 @@
 
 import os
 import json
+import re
+import shlex
+import shutil
 import signal
 import subprocess
 import sys
@@ -77,6 +80,78 @@ def _resolve_sonic_env(env: dict[str, str]) -> dict[str, str]:
     return payload
 
 
+def test_explicit_sonic_binary_is_used_by_real_search_worker(initialized_archive):
+    sonic_port = get_free_port()
+    lib_dir = initialized_archive / "lib"
+    env = cli_env(
+        live=True,
+        ABXPKG_LIB_DIR=str(lib_dir),
+        PLUGINS="search_backend_sonic",
+        SEARCH_BACKEND_ENGINE="sonic",
+        SEARCH_BACKEND_SONIC_PORT=str(sonic_port),
+    )
+    env.update(_resolve_sonic_env(env))
+    env["PATH"] = os.pathsep.join((str(Path(sys.executable).parent), env["PATH"]))
+    installed_binary = Path(env["SONIC_BINARY"]).resolve(strict=True)
+    managed_sonic_dir = lib_dir / "bash" / "bin"
+    managed_sonic_dir.mkdir(parents=True, exist_ok=True)
+    managed_binary = managed_sonic_dir / "sonic"
+    if installed_binary != managed_binary:
+        shutil.copy2(installed_binary, managed_binary)
+    config_result = run_archivebox_cmd(
+        ["config", "--set", f"SONIC_BINARY={managed_binary}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert config_result.returncode == 0, config_result.stderr or config_result.stdout
+    pinned_sonic_dir = initialized_archive / "pinned-sonic"
+    pinned_sonic_dir.mkdir()
+    sonic_binary = pinned_sonic_dir / "sonic"
+    shutil.copy2(installed_binary, sonic_binary)
+    env["SONIC_BINARY"] = str(sonic_binary)
+
+    result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    config_match = re.search(r"Using supervisord config file: (\S+)", result.stderr)
+    assert config_match, result.stderr
+    worker_config = Path(config_match.group(1)).parent / "workers" / "worker_sonic.conf"
+    worker_text = worker_config.read_text(encoding="utf-8")
+    command = next(line.partition("=")[2] for line in worker_text.splitlines() if line.startswith("command="))
+    assert shlex.split(command)[0] == str(sonic_binary), worker_text
+    from archivebox.workers.supervisord_util import _sonic_worker_bind_target
+
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": command}) == ("127.0.0.1", sonic_port)
+    renamed_binary = initialized_archive / "renamed-sonic-binary"
+    shutil.copy2(sonic_binary, renamed_binary)
+    renamed_command = shlex.join([str(renamed_binary), *shlex.split(command)[1:]])
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": renamed_command}) == ("127.0.0.1", sonic_port)
+
+    relative_binary = pinned_sonic_dir / "sonic-alt"
+    shutil.copy2(sonic_binary, relative_binary)
+    relative_env = {
+        **env,
+        "SONIC_BINARY": relative_binary.name,
+        "PATH": os.pathsep.join((str(pinned_sonic_dir), env["PATH"])),
+        "SEARCH_BACKEND_SONIC_PORT": str(get_free_port()),
+    }
+    relative_result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=relative_env,
+        timeout=60,
+    )
+    assert relative_result.returncode == 0, relative_result.stderr or relative_result.stdout
+    relative_worker_text = worker_config.read_text(encoding="utf-8")
+    relative_command = next(line.partition("=")[2] for line in relative_worker_text.splitlines() if line.startswith("command="))
+    assert Path(shlex.split(relative_command)[0]).name == relative_binary.name, relative_worker_text
+
+
 def _archive_pages_for_sqlite_reindexing(data_dir: Path, env: dict[str, str], root_url: str) -> None:
     add_env = dict(env)
     add_env["SEARCH_BACKEND_ENGINE"] = "ripgrep"
@@ -312,14 +387,13 @@ def test_live_second_server_takes_over_existing_server_process(tmp_path, initial
         assert "A newer archivebox process took over the orchestrator, server" in first_text
         assert "Starting orchestrator, server" in second_text
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["status"],
             cwd=tmp_path,
             env=env,
             timeout=60,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        assert result.returncode == 0, result.stderr or result.stdout
 
         first_resumes = first_log.read_text(encoding="utf-8", errors="replace").count("Other newer archivebox process")
         stop_archivebox_process(second, signal.SIGTERM)
@@ -347,15 +421,14 @@ def test_live_update_index_only_does_not_take_over_server_runtime(tmp_path, init
         supervisor_pid_before = supervisor_pid_from_log(server_log)
         daphne_pid_before = worker_pid_from_log(server_log, "worker_daphne")
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--before=0"],
             cwd=tmp_path,
             env=env,
             timeout=90,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert pid_is_alive(supervisor_pid_before)
         assert pid_is_alive(daphne_pid_before)
@@ -396,15 +469,14 @@ def test_live_server_keeps_http_runtime_while_update_runs_real_sqlite_indexer(tm
         runner_spawn_text = "spawned: 'worker_runner' with pid"
         runner_spawn_count = supervisord_log.read_text(encoding="utf-8", errors="replace").count(runner_spawn_text)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--batch-size=1"],
             cwd=tmp_path,
             env=env,
             timeout=180,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert supervisor_pid_from_log(server_log) == supervisor_pid_before
         assert pid_is_alive(daphne_pid_before)
@@ -553,14 +625,13 @@ def test_live_repeated_server_startups_take_over_cleanly(tmp_path, initialized_a
                 wait_for_pid_to_disappear(daphne_pids[index - 1], timeout=15)
                 wait_for_pid_to_disappear(runner_pids[index - 1], timeout=15)
 
-            _cmd_result = run_archivebox_cmd(
+            result = run_archivebox_cmd(
                 ["status"],
                 cwd=tmp_path,
                 env=env,
                 timeout=60,
             )
-            stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-            assert returncode == 0, stderr or stdout
+            assert result.returncode == 0, result.stderr or result.stdout
 
         assert pid_is_alive(servers[-1].pid)
         assert all(pid_is_alive(server.pid) for server in servers)
@@ -598,14 +669,13 @@ def test_live_background_add_survives_server_exit_and_foreground_run_reclaims(tm
         server_log = server.log_path
         supervisor_pid_before = supervisor_pid_from_log(server_log)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--batch-size=10"],
             cwd=tmp_path,
             env=env,
             timeout=90,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert pid_is_alive(supervisor_pid_before)
         assert supervisor_pid_from_log(server_log) == supervisor_pid_before

archivebox/tests/test_ui_add_view.py

@@ -20,6 +20,23 @@ WEB_HOST = "web.archivebox.localhost:8000"
 ADMIN_HOST = "admin.archivebox.localhost:8000"
 
 
+ADD_FORM_DEFAULTS = {
+    "tag": "",
+    "depth": "0",
+    "max_urls": "0",
+    "crawl_max_size": "0",
+    "snapshot_max_size": "0",
+    "url_filters_allowlist": "",
+    "url_filters_denylist": "",
+    "notes": "",
+    "schedule": "",
+    "persona": "Default",
+    "permissions": "public",
+    "start_paused": "",
+    "config": "{}",
+}
+
+
 @pytest.fixture
 def admin_user(db):
     return User.objects.create_superuser(
@@ -140,22 +157,13 @@ def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/staff-config",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
             "crawl_timeout": "0",
             "timeout": "60",
-            "snapshot_max_size": "0",
             "delete_after": "0",
             "crawl_max_concurrent_snapshots": "1",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
-            "notes": "",
             "schedule": "daily",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
             "main_plugins": ["wget"],
             "plugin_config__wget__WGET_TIMEOUT": "77",
@@ -246,6 +254,7 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com\nhttps://cdn.example.com/asset.js",
             "tag": "alpha,beta",
             "depth": "1",
@@ -260,11 +269,6 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
             "url_filters_denylist": "cdn.example.com",
             "url_filters_only_new": "1",
             "notes": "Created from /add/",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -290,24 +294,12 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
 
 def _csrf_test_add_data(url: str) -> dict[str, str]:
     return {
+        **ADD_FORM_DEFAULTS,
         "url": url,
-        "tag": "",
-        "depth": "0",
-        "max_urls": "0",
-        "crawl_max_size": "0",
         "crawl_timeout": "0",
         "timeout": "",
-        "snapshot_max_size": "0",
         "delete_after": "0",
         "crawl_max_concurrent_snapshots": "1",
-        "url_filters_allowlist": "",
-        "url_filters_denylist": "",
-        "notes": "",
-        "schedule": "",
-        "persona": "Default",
-        "permissions": "public",
-        "start_paused": "",
-        "config": "{}",
     }
 
 
@@ -337,24 +329,14 @@ def test_add_view_sanitizes_crawl_notes_before_safe_update(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/notes-xss",
-            "tag": "",
-            "depth": "0",
             "max_urls": "1",
-            "crawl_max_size": "0",
             "crawl_timeout": "0",
             "timeout": "",
-            "snapshot_max_size": "0",
             "delete_after": "0",
             "crawl_max_concurrent_snapshots": "1",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "notes": malicious_notes,
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -373,20 +355,8 @@ def test_add_view_unchecked_only_new_sets_crawl_override(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/rearchive",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -407,20 +377,10 @@ def test_add_view_selected_persona_wins_over_stale_config_override(client, admin
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/private",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
             "persona": "Private",
-            "permissions": "public",
-            "start_paused": "",
             "config": '{"DEFAULT_PERSONA": "Default"}',
         },
         HTTP_HOST=ADMIN_HOST,
@@ -447,20 +407,9 @@ def test_add_view_applies_plugin_config_overrides(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/plugin-config",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
             "main_plugins": ["wget"],
             "plugin_config__wget__WGET_ENABLED": "false",
             "plugin_config__wget__WGET_TIMEOUT": "77",
@@ -488,9 +437,8 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/public-safe",
-            "tag": "",
-            "depth": "0",
             "max_urls": "10",
             "crawl_max_size": "45mb",
             "crawl_timeout": "120",
@@ -503,8 +451,6 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
             "url_filters_only_new": "1",
             "notes": "public add",
             "schedule": "daily",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
             "main_plugins": ["wget"],
             "plugin_config__twocaptcha__TWOCAPTCHA_API_KEY": "posted-token",
@@ -541,21 +487,9 @@ def test_add_view_queues_crawl_for_background_runner(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -575,21 +509,11 @@ def test_add_view_start_paused_creates_paused_crawl_without_snapshots(client, ad
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/paused",
-            "tag": "",
             "depth": "1",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -610,6 +534,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": (
                 "https://sweeting.me,https://google.com\n"
                 "Notes: [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox), https://news.ycombinator.com\n"
@@ -617,20 +542,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
                 '{"items":["https://example.com/three"]}\n'
                 "csv,https://example.com/four"
             ),
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -656,21 +568,9 @@ def test_add_view_trims_trailing_punctuation_from_markdown_urls(client, admin_us
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": ("Docs: https://github.com/ArchiveBox/ArchiveBox.\nIssue: https://github.com/abc?abc#234234?."),
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )

archivebox/tests/test_ui_add_view_runtime.py

@@ -431,6 +431,10 @@ def test_public_add_view_import_text_formats_preserve_metadata_and_resume_withou
         URL_ALLOWLIST=r"127\.0\.0\.1[:/].*",
     )
     create_admin_and_token(tmp_path)
+    # These tests deliberately stop the server while crawls are queued. Finish
+    # dependency installation first so shutdown cannot leave a half-built venv.
+    install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
+    assert install_result.returncode == 0, install_result.stderr or install_result.stdout
 
     try:
         start_archivebox_server(tmp_path, env=env, port=port)
@@ -536,6 +540,10 @@ def test_public_add_view_rejects_file_path_and_shell_injection_payloads(tmp_path
         URL_ALLOWLIST=r"example\.com|example\.org|iana\.org|www\.iana\.org",
     )
     create_admin_and_token(tmp_path)
+    # These tests deliberately stop the server while crawls are queued. Finish
+    # dependency installation first so shutdown cannot leave a half-built venv.
+    install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
+    assert install_result.returncode == 0, install_result.stderr or install_result.stdout
 
     try:
         start_archivebox_server(tmp_path, env=env, port=port)

archivebox/tests/test_ui_admin_crawl.py

@@ -14,6 +14,7 @@ pytestmark = pytest.mark.django_db
 
 
 class TestCrawlScheduleAdmin:
+    @pytest.mark.django_db(transaction=True)
     def test_crawlschedule_change_view_renders_and_saves(self, client, admin_user, crawl):
         from archivebox.crawls.models import CrawlSchedule
 
@@ -35,24 +36,33 @@ class TestCrawlScheduleAdmin:
         assert b"No Crawls yet..." not in get_response.content
         assert b"No Snapshots yet..." not in get_response.content
 
-        post_response = client.post(
-            change_url,
-            {
-                "label": "Morning crawl",
-                "notes": "updated",
-                "schedule": "0 8 * * *",
-                "template": str(crawl.pk),
-                "created_by": str(admin_user.pk),
-                "_save": "Save",
-            },
-            HTTP_HOST=ADMIN_TEST_HOST,
-        )
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext
+
+        previous_mode = getattr(connection, "transaction_mode", None)
+        with CaptureQueriesContext(connection) as queries:
+            post_response = client.post(
+                change_url,
+                {
+                    "label": "Morning crawl",
+                    "notes": "updated",
+                    "schedule": "0 8 * * *",
+                    "template": str(crawl.pk),
+                    "created_by": str(admin_user.pk),
+                    "_save": "Save",
+                },
+                HTTP_HOST=ADMIN_TEST_HOST,
+            )
+        if connection.vendor == "sqlite":
+            assert any(query["sql"] == "BEGIN IMMEDIATE" for query in queries)
+            assert connection.transaction_mode == previous_mode
 
         assert post_response.status_code == 302
         schedule.refresh_from_db()
         assert schedule.label == "Morning crawl"
         assert schedule.notes == "updated"
         assert schedule.schedule == "0 8 * * *"
+        assert schedule.is_enabled is False
         assert schedule.template_id == crawl.pk
         assert schedule.created_by_id == admin_user.pk
 

archivebox/tests/test_ui_admin_snapshot.py

@@ -607,7 +607,14 @@ class TestSnapshotProgressStats:
 
         assert result.embed_path_db() is None
 
-    def test_embed_path_db_prefers_valid_output_str_over_first_output_file(self, snapshot, real_hash_projection):
+    @pytest.mark.parametrize(
+        "output_str",
+        [
+            ("wget/example.com/index.html"),
+            (""),
+        ],
+    )
+    def test_embed_path_db_selects_html_output(self, snapshot, real_hash_projection, output_str):
         from archivebox.core.models import ArchiveResult
 
         output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
@@ -619,32 +626,7 @@ class TestSnapshotProgressStats:
         _process, result = real_hash_projection
         ArchiveResult.objects.filter(pk=result.pk).update(
             plugin="wget",
-            output_str="wget/example.com/index.html",
-            output_files={
-                "example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
-                "example.com/index.html": {
-                    "size": (Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").stat().st_size,
-                    "mimetype": "text/html",
-                },
-            },
-        )
-        result.refresh_from_db()
-
-        assert result.embed_path_db() == "wget/example.com/index.html"
-
-    def test_embed_path_db_scores_output_files_instead_of_using_first_entry(self, snapshot, real_hash_projection):
-        from archivebox.core.models import ArchiveResult
-
-        output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
-        output_dir.mkdir(parents=True, exist_ok=True)
-        (Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").parent.mkdir(parents=True, exist_ok=True)
-        shutil.copyfile(REPO_ROOT / "README.md", Path(snapshot.output_dir) / "wget" / "example.com" / "index.html")
-        shutil.copyfile(REPO_ROOT / "archivebox" / "templates" / "static" / "bootstrap.min.css", output_dir / "mobile.css")
-
-        _process, result = real_hash_projection
-        ArchiveResult.objects.filter(pk=result.pk).update(
-            plugin="wget",
-            output_str="",
+            output_str=output_str,
             output_files={
                 "example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
                 "example.com/index.html": {

archivebox/tests/test_ui_live_progress.py

@@ -104,6 +104,74 @@ def real_crawl_setup_process(snapshot, hermetic_lib_dir):
 
 
 class TestLiveProgressView:
+    def test_collection_summary_reports_stored_sizes(self, client, admin_user, snapshot):
+        import time
+        from django.core.cache import cache
+        from django.db.models import Count, Sum
+        from archivebox.core.models import Snapshot
+
+        Snapshot.objects.filter(pk=snapshot.pk).update(output_size=12345)
+        expected = Snapshot.objects.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+        cache.delete_many(["progress-collection:all", "progress-collection:all:refresh"])
+        client.force_login(admin_user)
+        deadline = time.monotonic() + 5
+        while True:
+            response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+            assert response.status_code == 200
+            summary = response.json().get("collection")
+            if summary is not None or time.monotonic() >= deadline:
+                break
+            time.sleep(0.05)
+        assert summary is not None
+        assert summary["snapshots"] == expected["snapshots"]
+        assert summary["bytes"] == expected["bytes"]
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext
+
+        if connection.vendor == "sqlite":
+            with connection.cursor() as cursor:
+                cursor.execute("EXPLAIN QUERY PLAN SELECT COUNT(*), SUM(output_size) FROM core_snapshot")
+                plan = " ".join(str(row) for row in cursor.fetchall())
+            assert "COVERING INDEX" in plan, plan
+
+        timings = []
+        with CaptureQueriesContext(connection) as queries:
+            for _ in range(20):
+                start = time.perf_counter()
+                response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+                timings.append((time.perf_counter() - start) * 1000)
+                assert response.status_code == 200
+                assert response.json()["collection"] == summary
+        assert not any('AS "bytes"' in query["sql"] for query in queries)
+        print(f"cached progress response: median={sorted(timings)[10]:.1f}ms max={max(timings):.1f}ms")
+
+    def test_collection_summary_is_scoped_to_staff_owner(self, client, admin_user, snapshot):
+        import time
+        from django.core.cache import cache
+        from archivebox.core.models import Snapshot
+
+        admin_user.is_superuser = False
+        admin_user.save(update_fields=["is_superuser"])
+        from django.contrib.auth import get_user_model
+
+        # Give the snapshot another owner; staff must not learn its size/count.
+        snapshot.crawl.created_by = get_user_model().objects.create_user(username="other-summary-owner")
+        snapshot.crawl.save(update_fields=["created_by"])
+        Snapshot.objects.filter(pk=snapshot.pk).update(output_size=12345)
+        cache.delete_many([f"progress-collection:{admin_user.pk}", f"progress-collection:{admin_user.pk}:refresh"])
+        client.force_login(admin_user)
+        deadline = time.monotonic() + 5
+        while True:
+            response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+            assert response.status_code == 200
+            summary = response.json().get("collection")
+            if summary is not None or time.monotonic() >= deadline:
+                break
+            time.sleep(0.05)
+        assert summary is not None
+        assert summary["snapshots"] == 0
+        assert summary["bytes"] == 0
+
     def test_live_progress_rejects_unauthenticated_unscoped_request(self, client):
         response = client.get(reverse("live_progress"), HTTP_HOST=ADMIN_TEST_HOST)
 

archivebox/tests/test_urls.py

@@ -1431,7 +1431,7 @@ class TestUrlRouting:
             [
                 "add",
                 "--depth=0",
-                "--plugins=consolelog,screenshot,chrome_mhtml",
+                "--plugins=consolelog,screenshot,pdf,chrome_mhtml",
                 console_source_url,
             ],
             cwd=self.data_dir,
@@ -1475,6 +1475,14 @@ class TestUrlRouting:
             resp = client.get(f"/{screenshot_rel}?preview=1", HTTP_HOST=snapshot_host)
             assert resp.status_code == 200
             assert resp["Content-Type"].startswith("text/html")
+            assert f"frame-ancestors 'self' http://{web_host}" in resp["Content-Security-Policy"]
+
+            resp = client.get("/pdf/output.pdf?preview=1", HTTP_HOST=snapshot_host)
+            assert resp.status_code == 200
+            pdf_preview = response_body(resp).decode("utf-8")
+            assert '<iframe ' in pdf_preview
+            assert '<embed ' not in pdf_preview
+            assert "object-src 'none'" in resp["Content-Security-Policy"]
 
             root_screenshot = screenshot_file.read_bytes()
             import shutil
@@ -1507,6 +1515,58 @@ class TestUrlRouting:
             """.replace("__CONSOLE_MARKER__", console_marker),
         )
 
+        # Exercise the same real captured files with each request-time posture,
+        # including both auto branches and an HTTPS canonical reverse-proxy URL.
+        for mode, base in (
+            ("auto", "http://archivebox.localhost:8000"),
+            ("auto", "http://archivebox.example:8000"),
+            ("safe-subdomains-fullreplay", "https://archivebox.example"),
+            ("safe-onedomain-nojsreplay", "http://archivebox.example:8000"),
+            ("unsafe-onedomain-noadmin", "http://archivebox.example:8000"),
+            ("danger-onedomain-fullreplay", "http://archivebox.example:8000"),
+        ):
+            self._run(
+                """
+                from urllib.parse import urlsplit
+                from archivebox.config.common import get_request_config
+                snapshot = get_snapshot()
+                client = Client()
+                for path in ('screenshot/screenshot.png', 'pdf/output.pdf', 'chrome_mhtml/snapshot.mhtml'):
+                    target = urlsplit(build_snapshot_url(str(snapshot.id), path))
+                    response = client.get(target.path + '?preview=1', HTTP_HOST=target.netloc)
+                    assert response.status_code == 200, (target, response.status_code)
+                    csp = response['Content-Security-Policy']
+                    config = get_request_config(response.wsgi_request)
+                    origins = tuple(dict.fromkeys((get_web_base_url(config=config), get_admin_base_url(config=config))))
+                    ancestors = next(d.strip() for d in csp.split(';') if d.strip().startswith('frame-ancestors'))
+                    assert ancestors == "frame-ancestors 'self' " + ' '.join(origins), ancestors
+                    assert '*' not in ancestors  # no unrelated snapshots/sites
+                    assert "object-src 'none'" in csp
+                    assert "form-action 'none'" in csp
+                    assert "base-uri 'none'" in csp
+                    assert "script-src 'self'" in csp  # trusted viewer can run
+                    if path.endswith('.mhtml'):
+                        assert 'sandbox="allow-popups allow-popups-to-escape-sandbox allow-same-origin"' in response_body(response).decode()
+
+                dangerous, _, _ = write_replay_fixtures(snapshot)
+                target = urlsplit(build_snapshot_url(str(snapshot.id), dangerous))
+                response = client.get(target.path, HTTP_HOST=target.netloc)
+                assert response.status_code == 200
+                config = get_request_config(response.wsgi_request)
+                csp = response.headers.get('Content-Security-Policy', '')
+                assert ("script-src 'none'" in csp) == config.SHOULD_NEUTER_RISKY_REPLAY
+                assert ('sandbox;' in csp) == config.SHOULD_NEUTER_RISKY_REPLAY
+                admin = client.get('/admin/login/', HTTP_HOST=get_admin_host())
+                assert admin.status_code == (200 if config.CONTROL_PLANE_ENABLED else 403)
+                if not config.CONTROL_PLANE_ENABLED:
+                    assert client.get('/api/v1/docs', HTTP_HOST=get_api_host()).status_code == 403
+                    assert client.post('/public/', HTTP_HOST=get_web_host()).status_code == 403
+                print('OK')
+                """,
+                mode=mode,
+                env_overrides={"BASE_URL": base},
+            )
+
     def test_api_available_on_admin_and_api_hosts(self) -> None:
         self._run(
             """

archivebox/workers/supervisord_util.py

@@ -47,7 +47,6 @@ _SUPERVISORD_ERRORS = (XmlRpcError, OSError, RuntimeError, TimeoutError)
 _PROCESS_STATE_ERRORS = (DatabaseError, OSError, RuntimeError, ValueError, psutil.Error)
 MIN_SERVER_WORKER_AVAILABLE_MEMORY_BYTES = 256 * 1024 * 1024
 MIN_DEPENDENCY_INSTALL_AVAILABLE_MEMORY_BYTES = MIN_SERVER_WORKER_AVAILABLE_MEMORY_BYTES
-MIN_CRAWL_AVAILABLE_MEMORY_BYTES = 512 * 1024 * 1024
 
 
 def _shell_join(args: list[str]) -> str:
@@ -118,16 +117,6 @@ def require_dependency_install_memory(available_bytes: int | None = None) -> Non
     )
 
 
-def require_crawl_memory(available_bytes: int | None = None) -> None:
-    available_bytes = effective_available_memory_bytes() if available_bytes is None else available_bytes
-    _require_available_memory(
-        "archive a crawl",
-        "    No crawl, runner, or Sonic workers were started.",
-        available_bytes,
-        MIN_CRAWL_AVAILABLE_MEMORY_BYTES,
-    )
-
-
 def archivebox_cmd(*args: str) -> list[str]:
     return [str(resolve_env_binary("archivebox")), *args]
 
@@ -405,7 +394,7 @@ def is_port_in_use(host: str, port: int) -> bool:
 def _sonic_worker_bind_target(worker: dict[str, str]) -> tuple[str, int] | None:
     """Read the plugin-owned Sonic config before starting its supervisord worker."""
     command = shlex.split(worker.get("command") or "")
-    if not command or Path(command[0]).name != "sonic" or "-c" not in command:
+    if worker.get("name") != "worker_sonic" or not command or "-c" not in command:
         return None
     config_index = command.index("-c") + 1
     if config_index >= len(command):
@@ -1205,7 +1194,7 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
                     "redirect_stderr": "true",
                 },
                 False,
-            )
+            ),
         )
         log_files.append("logs/worker_vnc_browser.log")
     components = runtime_components_for_worker_names([worker["name"] for worker, _lazy in workers])
@@ -1280,11 +1269,18 @@ def tail_multiple_worker_logs(log_files: list[str], follow=True, proc=None, keep
 
     print()
 
+    # Log display needs a short refresh interval, but each ownership check
+    # queries the database and validates OS processes. Check leadership once
+    # per second even when workers continuously produce log output.
+    next_ownership_check = 0.0
     try:
         while follow:
-            if keep_running is not None and not keep_running():
-                print("\n[newer ArchiveBox process is now running the orchestrator and server]")
-                return "transferred"
+            now = time.monotonic()
+            if keep_running is not None and now >= next_ownership_check:
+                if not keep_running():
+                    print("\n[newer ArchiveBox process is now running the orchestrator and server]")
+                    return "transferred"
+                next_ownership_check = time.monotonic() + 1.0
 
             # Check if the monitored process has exited
             if proc is not None and proc.poll() is not None:
@@ -1330,7 +1326,22 @@ def get_sonic_supervisord_worker_from_plugin(config) -> dict[str, str] | None:
             raise
         return None
 
+    explicit_binary = os.environ.get("SONIC_BINARY")
+    requested_binary = Path(explicit_binary or str(config.SONIC_BINARY)).expanduser()
+    if explicit_binary:
+        # Machine.config can outrank the process environment; an explicit
+        # executable must win before the plugin hydrates its binary provider.
+        config = config.model_copy(update={"SONIC_BINARY": str(requested_binary)})
+
     worker = get_sonic_supervisord_worker(config)
+    if worker is not None:
+        if requested_binary.is_absolute():
+            # The plugin may project an explicit binary through env/bin/sonic.
+            # That symlink can be retargeted to a different Sonic release,
+            # whose on-disk index format may be incompatible with this one.
+            command = shlex.split(worker["command"])
+            command[0] = str(requested_binary.resolve(strict=True))
+            worker["command"] = shlex.join(command)
     return cast(dict[str, str] | None, worker)
 
 

bin/collect_ui_screenshots.sh

@@ -193,6 +193,22 @@ echo "[*] Creating a temporary screenshot admin"
 )
 CREATED_TEMP_USER=1
 
+# Configure the optional UI before the server snapshots its startup config.
+OPENCODE_PORT="$(uv run --no-cache --project "$REPO_DIR" python - <<'PYPORT'
+import socket
+
+with socket.socket() as sock:
+    sock.bind(("127.0.0.1", 0))
+    print(sock.getsockname()[1])
+PYPORT
+)"
+(
+    cd "$DATA_DIR"
+    uv run --no-cache --project "$REPO_DIR" archivebox config --set \
+        OPENCODE_ENABLED=True "OPENCODE_PORT=$OPENCODE_PORT" TLSNOTARY_ENABLED=True
+    uv run --no-cache --project "$REPO_DIR" archivebox install opencode --binproviders=env,pnpm
+)
+
 echo "[*] Starting ArchiveBox on port $PORT"
 (
     cd "$DATA_DIR"
@@ -248,6 +264,10 @@ while [[ "$capture_index" -lt "${#VIEWS[@]}" ]]; do
     capture_index=$((capture_index + 1))
     slug="$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]' | tr -cs '[:alnum:]' '-' | sed 's/^-//; s/-$//')"
     expected_plugin=""
+    expected_frame_text=""
+    if [[ "$capture_mode" == wait-frame-text:* ]]; then
+        expected_frame_text="${capture_mode#wait-frame-text:}"
+    fi
     if [[ "$name" == "Snapshot View ("*")" && "$capture_mode" != "live-progress" && "$capture_mode" != "snapshot-collapsed" ]]; then
         expected_plugin="${name#Snapshot View (}"
         expected_plugin="${expected_plugin%)}"
@@ -407,6 +427,7 @@ PY
                     SCREENSHOT_VARIANTS_JSON="$output_variants" \
                     SCREENSHOT_COLLAPSE_FILTERS=1 \
                     SCREENSHOT_EXPECT_PLUGIN="$expected_plugin" \
+                    SCREENSHOT_EXPECT_FRAME_TEXT="$expected_frame_text" \
                     node "$REPO_DIR/bin/take_screenshot.js" "$url" "$screenshot_path" >"$capture_dir/report.json"
                 view_timing_report="$capture_dir/report.json"
                 timing_report_path="$view_timing_report"
@@ -494,21 +515,6 @@ PY
             CREATE_WEBHOOK="$CREATE_WEBHOOK" \
             node "$REPO_DIR/bin/setup_ui_screenshot_data.js" "$ADMIN_BASE_URL" "$USERNAME"
 
-        OPENCODE_PORT="$(uv run --no-cache --project "$REPO_DIR" python - <<'PY'
-import socket
-
-with socket.socket() as sock:
-    sock.bind(("127.0.0.1", 0))
-    print(sock.getsockname()[1])
-PY
-)"
-        (
-            cd "$DATA_DIR"
-            OPENCODE_PORT="$OPENCODE_PORT" uv run --no-cache --project "$REPO_DIR" archivebox manage shell --no-imports -c \
-                'import os; from archivebox.machine.models import Machine; Machine.from_json({"config": {"OPENCODE_ENABLED": True, "OPENCODE_PORT": int(os.environ["OPENCODE_PORT"])}})'
-            uv run --no-cache --project "$REPO_DIR" archivebox install opencode --binproviders=env,pnpm
-        )
-
         SWEETING_CAPTURE_STARTED_AT="$( (
             cd "$DATA_DIR"
             uv run --no-cache --project "$REPO_DIR" archivebox manage shell --no-imports -c \
@@ -561,7 +567,7 @@ PY
         VIEWS+=(
             "Add URLs|$ADMIN_BASE_URL/add/|/add/|archivebox/core/views.py"
             "Admin dashboard|$ADMIN_BASE_URL/admin/|/admin/|archivebox/core/admin_site.py"
-            "AI agent|$ADMIN_BASE_URL/admin/agent/|/admin/agent/|abx_plugins/plugins/opencode/views.py|wait-text:ArchiveBox AI Agent"
+            "AI agent|$ADMIN_BASE_URL/admin/agent/|/admin/agent/|archivebox/opencode/views.py|wait-text:ArchiveBox AI Agent"
             "Snapshots table|$ADMIN_BASE_URL/admin/core/snapshot/|/admin/core/snapshot/|archivebox/core/admin_snapshots.py"
             "Snapshots grid|$ADMIN_BASE_URL/admin/core/snapshot/grid/|/admin/core/snapshot/grid/|archivebox/templates/admin/snapshots_grid.html"
             "Snapshot admin detail|$ADMIN_BASE_URL/admin/core/snapshot/$SNAPSHOT_ID/change/|/admin/core/snapshot/$SNAPSHOT_ID/change/|archivebox/core/admin_snapshots.py"
@@ -625,11 +631,17 @@ PY
             SCREENSHOT_HEIGHT=1000 \
             node "$REPO_DIR/bin/take_screenshot.js" "$LIVE_SNAPSHOT_VIEW_URL" "$CAPTURE_ROOT/snapshot-output-discovery.png" >"$SNAPSHOT_DISCOVERY_REPORT"
         SNAPSHOT_OUTPUT_PLUGINS="$(UI_SCREENSHOT_DISCOVERY_REPORT="$SNAPSHOT_DISCOVERY_REPORT" uv run --no-cache --project "$REPO_DIR" python -c \
-            'import json, os; from urllib.parse import urlsplit; report=json.load(open(os.environ["UI_SCREENSHOT_DISCOVERY_REPORT"])); print("\n".join("{}\t{}".format(output["plugin"], "wait-replay:Nick Sweeting" if urlsplit(output["previewUrl"]).path.endswith(".wacz") else "") for output in report["checks"]["snapshotOutputs"]))')"
+            'import json, os; from urllib.parse import urlsplit; report=json.load(open(os.environ["UI_SCREENSHOT_DISCOVERY_REPORT"])); print("\n".join("{}\t{}".format(output["plugin"], "wait-frame-text:Verified · signature and archived response match" if output["plugin"] == "tlsnotary" else "wait-replay:Nick Sweeting" if urlsplit(output["previewUrl"]).path.endswith(".wacz") else "") for output in report["checks"]["snapshotOutputs"]))')"
         if [[ -z "$SNAPSHOT_OUTPUT_PLUGINS" ]]; then
             echo "[!] The Sweeting.me snapshot detail page exposed no selectable outputs" >&2
             exit 1
         fi
+        if ! printf '%s\n' "$SNAPSHOT_OUTPUT_PLUGINS" | cut -f1 | grep -qx tlsnotary; then
+            echo "[!] Live capture did not expose the required TLSNotary verifier output" >&2
+            tail -200 "$CAPTURE_ROOT/sweeting-live-capture.log" >&2
+            cat "$SNAPSHOT_DISCOVERY_REPORT" >&2
+            exit 1
+        fi
         while IFS=$'\t' read -r plugin_name output_capture_mode; do
             [[ -z "$plugin_name" ]] && continue
             VIEWS+=("Snapshot View ($plugin_name)|$LIVE_SNAPSHOT_VIEW_URL#$plugin_name|/|archivebox/templates/core/snapshot.html|$output_capture_mode")

bin/docker_entrypoint.sh

@@ -4,8 +4,9 @@
 # It takes a CMD as $* shell arguments and runs it following these setup steps:
 
 # - Set the archivebox user to match the existing /data owner when possible
-#     1. use the first non-root owner detected from existing collection files
-#     2. fall back to the image's default archivebox uid/gid when /data is root-owned
+#     1. use explicit PUID/PGID overrides when provided
+#     2. otherwise use the first non-root owner detected from collection files
+#     3. fall back to the image's default archivebox uid/gid when /data is root-owned
 # - Create a new /data dir if necessary and set the correct ownership on it
 # - Create a new /browsers dir if necessary and set the correct ownership on it
 # - Check whether we're running inside QEMU emulation and show a warning if so.
@@ -34,7 +35,8 @@ export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$ABXPKG_LIB_DIR/cache}"
 export ABXBUS_CACHE_DIR="${ABXBUS_CACHE_DIR:-$XDG_CACHE_HOME/abxbus}"
 export UV_CACHE_DIR="${UV_CACHE_DIR:-$XDG_CACHE_HOME/uv}"
 
-# Global default uid/gid used when /data is empty or root-owned.
+# Global default uid/gid used when /data is empty or root-owned. PUID/PGID may
+# override the detected values for mounts with fixed server-side identities.
 export DEFAULT_ARCHIVEBOX_UID="${DEFAULT_ARCHIVEBOX_UID:-911}"
 export DEFAULT_ARCHIVEBOX_GID="${DEFAULT_ARCHIVEBOX_GID:-911}"
 
@@ -44,7 +46,7 @@ detect_data_owner() {
         if [[ -e "$path" ]]; then
             uid="$(stat -c '%u' "$path" 2>/dev/null || echo "$DEFAULT_ARCHIVEBOX_UID")"
             gid="$(stat -c '%g' "$path" 2>/dev/null || echo "$DEFAULT_ARCHIVEBOX_GID")"
-            if [[ "$uid" != "0" && "$gid" != "0" ]]; then
+            if [[ "$uid" != "0" ]]; then
                 echo "$uid:$gid"
                 return
             fi
@@ -54,8 +56,23 @@ detect_data_owner() {
 }
 
 export DETECTED_OWNER="$(detect_data_owner)"
-export TARGET_UID="${DETECTED_OWNER%%:*}"
-export TARGET_GID="${DETECTED_OWNER##*:}"
+export DETECTED_UID="${DETECTED_OWNER%%:*}"
+export DETECTED_GID="${DETECTED_OWNER##*:}"
+export TARGET_UID="${PUID:-$DETECTED_UID}"
+export TARGET_GID="${PGID:-$DETECTED_GID}"
+
+if [[ ! "$TARGET_UID" =~ ^[0-9]+$ || ! "$TARGET_GID" =~ ^[0-9]+$ ]]; then
+    echo -e "\n[X] Error: PUID and PGID must be numeric when set (got PUID=${PUID:-unset} PGID=${PGID:-unset})." > /dev/stderr
+    exit 3
+fi
+
+# The entrypoint needs root for mount preparation, but ArchiveBox and Chrome
+# must always run as a non-root user. Ignore PUID=0 injected by rootless
+# runtimes/providers and retain the detected/default non-root UID instead.
+# GID 0 remains valid for group-writable mounts.
+if [[ "$TARGET_UID" == "0" ]]; then
+    export TARGET_UID="$DETECTED_UID"
+fi
 
 if [[ "$(id -u)" == "0" ]]; then
     # Root is only used for startup permission repair. ArchiveBox/Chrome run as
@@ -77,6 +94,14 @@ else
     export TARGET_GID="$(id -g)"
 fi
 
+run_as_archivebox() {
+    if [[ "$(id -u)" == "0" ]]; then
+        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$@"
+    else
+        "$@"
+    fi
+}
+
 # Check if user attempted to run it in the root of their home folder or hard drive (common mistake)
 if [[ -d "$DATA_DIR/Documents" || -d "$DATA_DIR/.config" || -d "$DATA_DIR/usr" || -f "$DATA_DIR/.bashrc" || -f "$DATA_DIR/.zshrc" ]]; then
     echo -e "\n[X] ERROR: ArchiveBox was run from inside a home folder"
@@ -98,59 +123,87 @@ chmod_if_possible() {
     chmod u+rwX,g+rwX "$path" 2>/dev/null || true
 }
 
+path_is_writable() {
+    local path="$1"
+    [[ -d "$path" ]] \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        && run_as_archivebox test -x "$path" 2>/dev/null
+}
+
 ensure_dir() {
     local path="$1"
+
+    # Functional access is authoritative for NFS/CIFS/FUSE and Docker Desktop:
+    # they may report synthetic ownership or reject chown while writes work.
+    path_is_writable "$path" && return 0
+
+    # Create missing paths as the eventual application user first. This is the
+    # only reliable path on root-squashed mounts where container root is mapped
+    # to an unprivileged identity but the configured PUID can write.
+    if [[ ! -e "$path" ]] && run_as_archivebox mkdir -p "$path" 2>/dev/null; then
+        path_is_writable "$path" && return 0
+    fi
+
+    # Fall back to bounded, shallow root repair. Never walk the collection or
+    # archive tree during startup.
     mkdir -p "$path" 2>/dev/null || true
     chown_if_needed "$path"
     chmod_if_possible "$path"
+    path_is_writable "$path" || permission_error "$path"
 }
 
 ensure_file_owner() {
     local path="$1"
     [[ -e "$path" ]] || return 0
+    run_as_archivebox test -r "$path" 2>/dev/null \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        && return 0
     chown_if_needed "$path"
     chmod_if_possible "$path"
+    run_as_archivebox test -r "$path" 2>/dev/null \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        || permission_error "$path"
 }
 
 ensure_runtime_tmp_tree() {
-    mkdir -p "$TMP_DIR" 2>/dev/null || true
-    [[ -e "$TMP_DIR" ]] || return 0
+    if [[ ! -e "$TMP_DIR" ]]; then
+        ensure_dir "$TMP_DIR"
+        return
+    fi
+    path_is_writable "$TMP_DIR" && return 0
     if [[ "$(id -u)" == "0" ]]; then
         chown -R "$TARGET_UID:$TARGET_GID" "$TMP_DIR" 2>/dev/null || true
     fi
     chmod_if_possible "$TMP_DIR"
+    path_is_writable "$TMP_DIR" || permission_error "$TMP_DIR"
 }
 
 ensure_small_runtime_tree() {
     local path="$1"
-    mkdir -p "$path" 2>/dev/null || true
-    [[ -e "$path" ]] || return 0
+    if [[ ! -e "$path" ]]; then
+        ensure_dir "$path"
+        return
+    fi
+    path_is_writable "$path" && return 0
     if [[ "$(id -u)" == "0" ]]; then
         chown -R "$TARGET_UID:$TARGET_GID" "$path" 2>/dev/null || true
     fi
     chmod -R u+rwX,g+rwX "$path" 2>/dev/null || true
-}
-
-run_as_archivebox() {
-    if [[ "$(id -u)" == "0" ]]; then
-        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$@"
-    else
-        "$@"
-    fi
+    path_is_writable "$path" || permission_error "$path"
 }
 
 permission_error() {
     local path="$1"
     echo -e "\n[X] Error: archivebox user (uid=$TARGET_UID gid=$TARGET_GID) cannot write to $path." > /dev/stderr
     echo -e "    Current owner is $(stat -c '%u:%g' "$path" 2>/dev/null || echo 'unknown')." > /dev/stderr
-    echo -e "    Fix ownership of the matching host path for the intended archivebox user, e.g.:" > /dev/stderr
-    echo -e "       sudo chown $TARGET_UID:$TARGET_GID ./data" > /dev/stderr
-    echo -e "    Repair any nested path named above explicitly; never recursively chown a large archive." > /dev/stderr
+    echo -e "    The entrypoint tried non-root creation first, then bounded root ownership/mode repair." > /dev/stderr
+    echo -e "    The mount is read-only or its server-side identity/ACL does not allow uid=$TARGET_UID gid=$TARGET_GID." > /dev/stderr
     exit 3
 }
 
-# Create and repair only the small set of top-level writable paths. Do not recurse
-# through /data/archive; large collections can take days to recursively chown/chmod.
+# Create and repair only the small set of required paths. Functional checks run
+# as the final non-root identity first, so writable NFS/CIFS/FUSE mounts never
+# receive repeated chown attempts. Never recursively walk /data/archive here.
 ensure_dir "$DATA_DIR"
 ensure_dir "$CONFIG_DIR"
 ensure_dir "$DATA_DIR/logs"
@@ -166,30 +219,36 @@ ensure_file_owner "$DATA_DIR/ArchiveBox.conf"
 ensure_file_owner "$CONFIG_DIR/config.env"
 ensure_file_owner "$CONFIG_DIR/derived.env"
 
-run_as_archivebox touch "$DATA_DIR/logs/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$DATA_DIR/logs"
-rm -f "$DATA_DIR/logs/.permissions_test_safe_to_delete"
-run_as_archivebox touch "$DATA_DIR/archive/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$DATA_DIR/archive"
-rm -f "$DATA_DIR/archive/.permissions_test_safe_to_delete"
-run_as_archivebox touch "$PERSONAS_DIR/Default/chrome_profile/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$PERSONAS_DIR/Default/chrome_profile"
-rm -f "$PERSONAS_DIR/Default/chrome_profile/.permissions_test_safe_to_delete"
+assert_writable_dir() {
+    local path="$1"
+    local probe
+    probe="$(run_as_archivebox mktemp "$path/.permissions_test.XXXXXX" 2>/dev/null)" || permission_error "$path"
+    run_as_archivebox rm -f "$probe" 2>/dev/null || permission_error "$path"
+}
 
-# check if novnc x11 $DISPLAY is available
-export DISPLAY="${DISPLAY:-"novnc:0.0"}"
-if ! xdpyinfo > /dev/null 2>&1; then
-    # cant connect to x11 display, unset it so that chrome doesn't try to connect to it and hang indefinitely
-    unset DISPLAY
+assert_writable_dir "$DATA_DIR/logs"
+assert_writable_dir "$DATA_DIR/archive"
+assert_writable_dir "$PERSONAS_DIR/Default/chrome_profile"
+
+# Preserve an explicit DISPLAY (including an empty value to disable X11).
+# Only probe the implicit noVNC default: a missing probe or unavailable server
+# must not silently discard the user's display configuration.
+if [[ -z "${DISPLAY+x}" ]]; then
+    export DISPLAY="novnc:0.0"
+    if ! xdpyinfo > /dev/null 2>&1; then
+        unset DISPLAY
+    fi
 fi
 
 # Active browser processes do not survive container restarts, but their lock
-# files can. Clear stale browser state before dropping privileges.
-find "$PERSONAS_DIR" -type f \( \
-    -name "SingletonLock" \
-    -o -name "SingletonSocket" \
-    -o -name "SingletonCookie" \
-    -o -name "DevToolsActivePort" \
-    -o -name ".launch.lock" \
-    -o -name ".target.lock" \
-\) -delete >/dev/null 2>&1 || true
+# files can. Chromium keeps these at the user-data root; only inspect the
+# known root of each persona, never walk its potentially large profile tree.
+for persona_profile in "$PERSONAS_DIR"/*/chrome_profile; do
+    [[ -d "$persona_profile" ]] || continue
+    for lock_name in SingletonLock SingletonSocket SingletonCookie DevToolsActivePort .launch.lock .target.lock; do
+        rm -f "$persona_profile/$lock_name" 2>/dev/null || true
+    done
+done
 find /tmp "$TMP_DIR" -maxdepth 1 -type d -name "archivebox-chrome-profile.*" -mmin +30 -exec rm -rf {} + >/dev/null 2>&1 || true
     
 

bin/generate_ui_screenshot_gallery.py

@@ -4,6 +4,7 @@ import hashlib
 import html
 import json
 import os
+import re
 import subprocess
 import struct
 import sys
@@ -123,6 +124,53 @@ def append_manifest(manifest_path: Path, screenshot_path: Path) -> None:
         manifest.write(json.dumps(item) + "\n")
 
 
+def render_gallery(gallery_header: str, gallery_sections: str) -> str:
+    return (
+        '<!doctype html><html lang="en"><head><meta charset="utf-8">'
+        '<meta name="viewport" content="width=device-width, initial-scale=1">'
+        "<title>ArchiveBox UI Screenshots</title><style>"
+        ":root{--ink:#28252a;--muted:#716c74;--accent:#9b2854;--paper:#fcfaf7;--line:#e9e2e3}"
+        "*{box-sizing:border-box}body{margin:0;font:16px/1.75 system-ui,sans-serif;"
+        "background:var(--paper);color:var(--ink)}.screenshot-content{max-width:1100px;margin:0 auto;padding:28px}"
+        "a{color:var(--accent)}h1,h2{line-height:1.15;letter-spacing:-.045em}"
+        "article{margin:40px 0 65px}article h2{font-size:29px;margin:0 0 10px}article p{font-size:13px;color:var(--muted);margin:0 0 20px}"
+        ".profile-switch{display:flex;flex-wrap:wrap;gap:4px;background:#f0e8eb;border:1px solid var(--line);padding:5px;"
+        "border-radius:28px;width:max-content;max-width:100%;margin:30px 0}.profile-switch button{font:600 13px system-ui;"
+        "cursor:pointer;border:0;border-radius:22px;background:transparent;color:var(--muted);padding:10px 20px}"
+        ".profile-switch button[aria-pressed=true]{background:var(--accent);color:white;box-shadow:0 3px 10px #9b285425}"
+        "a:focus-visible,button:focus-visible{outline:3px solid var(--accent);outline-offset:4px}"
+        "figure{min-width:0;margin:0 0 20px;padding:18px;border:1px solid var(--line);border-radius:20px;"
+        "background:linear-gradient(140deg,#fff,#f8edf2)}figure[hidden]{display:none}"
+        ".shot-mobile{max-width:440px;margin-inline:auto}.shot-tablet{max-width:800px;margin-inline:auto}"
+        "figcaption{font-size:12px;color:var(--muted);text-align:center;margin:0 0 12px}"
+        "img{display:block;width:100%;height:auto;border-radius:8px}code{overflow-wrap:anywhere}"
+        "@media(max-width:550px){.screenshot-content{padding:20px}figure{padding:8px;border-radius:12px}.profile-switch button{padding:9px 16px}}"
+        '</style><link rel="stylesheet" href="../site-chrome.css"></head><body>'
+        + (REPO_DIR / "bin/templates/screenshots-header.html").read_text(encoding="utf-8")
+        + '<div class="screenshot-content">'
+        + gallery_header
+        + '<div class="profile-switch" role="group" aria-label="Screenshot viewport">'
+        + "".join(
+            f'<button type="button" data-viewport="{profile}" aria-pressed="{str(profile == "desktop").lower()}">{profile.title()}</button>'
+            for profile in CAPTURE_PROFILES
+        )
+        + "</div><main>"
+        + gallery_sections
+        + """</main><script>
+const figures = document.querySelectorAll('.shot');
+const buttons = document.querySelectorAll('[data-viewport]');
+function selectProfile(profile) {
+  for (const figure of figures) figure.hidden = figure.dataset.profile !== profile;
+  for (const button of buttons) button.setAttribute('aria-pressed', String(button.dataset.viewport === profile));
+}
+for (const button of buttons) button.addEventListener('click', () => selectProfile(button.dataset.viewport));
+selectProfile('desktop');
+</script></div>"""
+        + (REPO_DIR / "bin/templates/screenshots-footer.html").read_text(encoding="utf-8")
+        + "</body></html>\n"
+    )
+
+
 def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -> None:
     provenance = build_provenance()
     source_base_url = f"https://github.com/ArchiveBox/ArchiveBox/blob/{provenance['revision']}/"
@@ -205,7 +253,7 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
                 f'alt="{html.escape(str(capture["name"]))} — {profile}" width="{width}"></td>',
             )
             html_figures.append(
-                f'<figure class="shot shot-{profile}"><figcaption>{label}</figcaption>'
+                f'<figure class="shot shot-{profile}" data-profile="{profile}"><figcaption>{label}</figcaption>'
                 f'<a href="./{versioned_filename}">'
                 f'<img src="./{versioned_filename}" width="{width}" height="{height}" loading="lazy" '
                 f'alt="{html.escape(str(capture["name"]))} — {profile}"></a></figure>',
@@ -252,25 +300,15 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
         encoding="utf-8",
     )
 
-    html_path.write_text(
-        '<!doctype html><html lang="en"><head><meta charset="utf-8">'
-        '<meta name="viewport" content="width=device-width, initial-scale=1">'
-        "<title>ArchiveBox UI Screenshots</title><style>"
-        "body{max-width:1900px;margin:0 auto;padding:20px;font:16px system-ui,sans-serif;background:#f6f7f9;color:#18202a}"
-        "a{color:#2563eb}article{margin:32px 0 64px}.shots{display:grid;grid-template-columns:2fr 1.35fr .8fr;gap:14px;align-items:start}"
-        "figure{min-width:0;margin:0}figcaption{font-weight:650;margin:0 0 8px}img{display:block;width:100%;height:auto;"
-        "border:1px solid #cbd5e1;border-radius:8px;background:white;box-shadow:0 8px 24px #0f172a18}code{overflow-wrap:anywhere}"
-        "@media(max-width:900px){.shots{grid-template-columns:1fr}body{padding:12px}}"
-        '</style></head><body><header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
-        f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
+    gallery_header = (
+        '<header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
+        + f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
         f'<a href="https://github.com/ArchiveBox/ArchiveBox/commit/{html.escape(provenance["revision"])}">'
         f"<code>{html.escape(provenance['revision'][:12])}</code></a> on "
         f'<time datetime="{html.escape(provenance["generated_at"])}">{html.escape(provenance["generated_at"])}</time>. '
-        "Desktop, tablet, and mobile viewports are captured from the same build.</p></header><main>"
-        + "".join(html_sections)
-        + "</main></body></html>\n",
-        encoding="utf-8",
+        "Desktop, tablet, and mobile viewports are captured from the same build.</p></header>"
     )
+    html_path.write_text(render_gallery(gallery_header, "".join(html_sections)), encoding="utf-8")
 
     file_hashes = {
         capture["filename"]: hashlib.sha256((html_path.parent / capture["filename"]).read_bytes()).hexdigest() for capture in captures
@@ -297,7 +335,27 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
                     screenshot_path.unlink()
 
 
+def refresh_gallery(html_path: Path) -> None:
+    """Apply current presentation to published capture content without changing provenance."""
+    document = html_path.read_text(encoding="utf-8")
+    header = re.search(r"<header>.*?</header>", document, re.DOTALL)
+    sections = re.search(r"<main>(.*?)</main>", document, re.DOTALL)
+    if not header or not sections:
+        raise SystemExit("Published screenshot gallery is missing its capture header or content")
+    content = sections[1]
+    for profile in CAPTURE_PROFILES:
+        # Galleries published before the switcher used classes alone.
+        content = content.replace(
+            f'<figure class="shot shot-{profile}">',
+            f'<figure class="shot shot-{profile}" data-profile="{profile}">',
+        )
+    html_path.write_text(render_gallery(header[0], content), encoding="utf-8")
+
+
 def main() -> None:
+    if len(sys.argv) == 3 and sys.argv[1] == "refresh":
+        refresh_gallery(Path(sys.argv[2]))
+        return
     if len(sys.argv) == 4 and sys.argv[1] == "validate":
         validate_navigation(Path(sys.argv[2]), sys.argv[3])
         return

bin/restore_site_screenshots.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+"""Restore validated capture output without running application CI."""
+
+import argparse
+from concurrent.futures import ThreadPoolExecutor
+import hashlib
+import json
+import os
+from pathlib import Path, PurePosixPath
+import subprocess
+from urllib.error import HTTPError
+from urllib.request import urlopen
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument("kind", choices=["archivebox", "ios", "extension"])
+parser.add_argument("destination", type=Path)
+args = parser.parse_args()
+args.destination.mkdir(parents=True, exist_ok=True)
+# Only successful default-branch capture runs are eligible, never PR artifacts.
+if os.environ.get("GH_TOKEN"):
+    repo = os.environ["GITHUB_REPOSITORY"]
+    branch = "dev" if args.kind == "archivebox" else "main"
+    runs = json.loads(
+        subprocess.check_output(
+            ["gh", "api", f"repos/{repo}/actions/workflows/screenshots.yml/runs?branch={branch}&status=success&per_page=100"],
+        ),
+    )
+    for run in runs["workflow_runs"]:
+        if run["event"] not in ("push", "workflow_dispatch"):
+            continue
+        artifacts = json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/actions/runs/{run['id']}/artifacts"]))
+        if any(a["name"] == "site-screenshots" and not a["expired"] for a in artifacts["artifacts"]):
+            subprocess.run(
+                ["gh", "run", "download", str(run["id"]), "--repo", repo, "--name", "site-screenshots", "--dir", str(args.destination)],
+                check=True,
+            )
+            print(f"Restored validated captures from run {run['id']} ({run['head_sha']})")
+            raise SystemExit(0)
+# Published captures remain available even after Actions artifacts expire.
+base, manifest_name = {
+    "archivebox": ("https://archivebox.io/screenshots/", "build.json"),
+    "ios": ("https://archivebox.github.io/ios-archivebox/screenshots/", "manifest.json"),
+    "extension": ("https://archivebox.github.io/archivebox-browser-extension/screenshots/", "manifest.json"),
+}[args.kind]
+
+
+def fetch(name):
+    path = PurePosixPath(name)
+    if path.is_absolute() or ".." in path.parts or ":" in name or "\\" in name:
+        raise ValueError(f"Unsafe capture path: {name}")
+    with urlopen(base + name, timeout=60) as response:
+        data = response.read()
+    expected = hashes.get(name)
+    if expected and hashlib.sha256(data).hexdigest() != expected:
+        raise ValueError(f"Capture checksum mismatch: {name}")
+    target = args.destination / name
+    target.parent.mkdir(parents=True, exist_ok=True)
+    target.write_bytes(data)
+
+
+try:
+    with urlopen(base + manifest_name, timeout=60) as response:
+        raw = response.read()
+except HTTPError as error:
+    if error.code == 404 and args.kind == "ios":
+        print("No complete native gallery published yet; use the curated README screenshots.")
+        raise SystemExit(0)
+    raise
+manifest = json.loads(raw)
+hashes = {}
+if args.kind == "archivebox":
+    hashes = manifest["files"]
+    names = [*hashes, "index.html"]
+elif args.kind == "extension":
+    names = [image["file"] for capture in manifest["screenshots"] for image in capture["images"]]
+else:
+    if manifest.get("complete") is not True:
+        raise ValueError("Published native gallery must be complete")
+    names = [capture["path"] for capture in manifest["captures"]]
+with ThreadPoolExecutor(max_workers=8) as pool:
+    list(pool.map(fetch, names))
+(args.destination / manifest_name).write_bytes(raw)
+print(f"Restored {len(names)} published capture files, retaining revision {manifest['revision']}")

bin/take_screenshot.js

@@ -32,6 +32,7 @@ Environment:
   SCREENSHOT_SNAPSHOT_VIEW   Set to list or grid before loading the page
   SCREENSHOT_SNAPSHOT_HEADER Set to expanded or collapsed before loading a snapshot detail page
   SCREENSHOT_EXPECT_PLUGIN   Require this snapshot output plugin to be selected
+  SCREENSHOT_EXPECT_FRAME_TEXT  Require this text in the selected output frame
   SCREENSHOT_EXPECT_LIVE_PROGRESS  Require real progress bars and a loaded screencast frame
   SCREENSHOT_COLLAPSE_FILTERS Set to 1 to keep admin filters out of screenshots
   SCREENSHOT_RESET_FILTERS   Set to 1 to clear the admin filter collapsed preference
@@ -231,6 +232,12 @@ async function main() {
           }
         });
       }, { timeout: 45000, polling: 250 }).catch(() => {});
+      if (process.env.SCREENSHOT_EXPECT_FRAME_TEXT) {
+        const frameElement = await page.$('#main-frame');
+        const frame = await frameElement.contentFrame();
+        await frame.waitForFunction((text) => document.body.innerText.includes(text),
+          { timeout: 45000 }, process.env.SCREENSHOT_EXPECT_FRAME_TEXT);
+      }
     }
 
     if (process.env.SCREENSHOT_EXPECT_LIVE_PROGRESS === '1') {

bin/templates/screenshots-footer.html

@@ -0,0 +1,26 @@
+<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
+  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
+  
+  <div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
+</footer>
+<script>
+(() => {
+  const menu = document.querySelector('.abx-apps');
+  document.addEventListener('keydown', event => {
+    if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
+  });
+  document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
+  function revealAnchor() {
+    let target;
+    try { target = document.getElementById(decodeURIComponent(location.hash.slice(1))); } catch { return; }
+    if (!target) return;
+    let details = target.closest('details');
+    let opened = false;
+    while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
+    if (opened) requestAnimationFrame(() => target.scrollIntoView());
+  }
+  addEventListener('hashchange', revealAnchor);
+  revealAnchor();
+})();
+</script>

bin/templates/screenshots-header.html

@@ -0,0 +1,9 @@
+<header class="abx-header">
+  <a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">UI Screenshots</span></a>
+  <nav class="abx-nav" aria-label="Main navigation">
+    <details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
+    <a href="https://archivebox.io/screenshots/">Screenshots</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
+    <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
+    <a class="abx-cta" href="https://archivebox.io/#install">Get started</a>
+  </nav>
+</header>

bin/validate_docker_display.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# Real Compose regression checks. Build the image under test first, then run:
+# IMAGE=archivebox/archivebox:issue-1872 ./bin/validate_docker_display.sh
+set -Eeuo pipefail
+
+export IMAGE="${IMAGE:-archivebox/archivebox:dev}"
+validation_dir="$(mktemp -d "${TMPDIR:-/tmp}/archivebox-display.XXXXXX")"
+compose=(docker compose --project-name "archivebox-display-$$" --file "$validation_dir/compose.yml")
+cleanup() {
+    "${compose[@]}" down --volumes --remove-orphans
+    echo "Validation files: $validation_dir"
+}
+trap cleanup EXIT
+
+cat > "$validation_dir/compose.yml" <<'YAML'
+services:
+  archivebox:
+    image: ${IMAGE}
+    volumes:
+      - ./data:/data
+      - ./data/personas:/data/personas
+      - tmp:/tmp/archivebox
+    shm_size: 1gb
+  novnc:
+    image: theasp/novnc:latest
+    environment:
+      DISPLAY_WIDTH: 1280
+      DISPLAY_HEIGHT: 800
+      RUN_XTERM: "no"
+    healthcheck:
+      test: ["CMD", "xdpyinfo", "-display", ":0"]
+      interval: 2s
+      timeout: 2s
+      retries: 10
+volumes:
+  tmp:
+YAML
+
+"${compose[@]}" up -d --wait novnc
+
+# Check the real runtime dependency as well as successful X11 connectivity.
+"${compose[@]}" run --rm archivebox sh -ec '
+    command -v xdpyinfo
+    test "$DISPLAY" = novnc:0.0
+    xdpyinfo >/dev/null
+'
+"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
+    test "$DISPLAY" = novnc:0.0
+    xdpyinfo >/dev/null
+'
+
+# Explicit values belong to the caller, even when X11 is unavailable.
+"${compose[@]}" run --rm -e DISPLAY=unavailable:0 archivebox sh -ec '
+    test "$DISPLAY" = unavailable:0
+'
+# An explicit empty value disables automatic display discovery.
+"${compose[@]}" run --rm -e DISPLAY= archivebox sh -ec '
+    test "${DISPLAY+x}" = x
+    test -z "$DISPLAY"
+'
+
+"${compose[@]}" stop novnc
+"${compose[@]}" run --rm archivebox sh -ec '
+    test "${DISPLAY+x}" != x
+'
+"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
+    test "$DISPLAY" = novnc:0.0
+'
+echo 'PASS: Compose DISPLAY discovery, explicit override, empty opt-out, and headless fallback'

bin/validate_docker_uid_gid.sh

@@ -246,6 +246,10 @@ run_case() {
         local nested_stat
         nested_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data/archive/existing/file" 2>/dev/null || true)"
         [[ "$nested_stat" == "0:0" ]] || ok=0
+    elif [[ "$post_assert" == "data-owner-stays" ]]; then
+        local data_stat
+        data_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data" 2>/dev/null || true)"
+        [[ "$data_stat" == "1000:1000" ]] || ok=0
     elif [[ "$post_assert" == runtime-nested-root-stays ]]; then
         local lib_stat browsers_stat chrome_rules_stat chrome_metadata_stat
         lib_stat="$("${docker_base[@]}" -v "$case_dir/lib:/libdir" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /libdir/existing/file" 2>/dev/null || true)"
@@ -256,6 +260,11 @@ run_case() {
             && "$browsers_stat" == "0:0" \
             && "$chrome_rules_stat" == "$expected_uid:$expected_gid" \
             && "$chrome_metadata_stat" == "$expected_uid:$expected_gid" ]] || ok=0
+    elif [[ "$post_assert" == cache-and-profile-shallow ]]; then
+        local cache_stat profile_state
+        cache_stat="$("${docker_base[@]}" -v "$case_dir/lib:/libdir" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /libdir/cache/uv/nested/marker" 2>/dev/null || true)"
+        profile_state="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "test ! -e /data/personas/Default/chrome_profile/SingletonLock && test -e /data/personas/Default/chrome_profile/nested/SingletonLock && echo shallow" 2>/dev/null || true)"
+        [[ "$cache_stat" == "0:0" && "$profile_state" == shallow ]] || ok=0
     elif [[ "$post_assert" == users-dir-repaired ]]; then
         local users_stat
         users_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data/users" 2>/dev/null || true)"
@@ -419,6 +428,62 @@ run_mount_case() {
     VALIDATION_ROOT="$previous_root"
 }
 
+run_nfs_root_squash_case() {
+    local mount_dir="$1"
+
+    [[ -n "$mount_dir" ]] || { log "FAIL NFS root_squash case: mount dir not provided"; exit 1; }
+    [[ -d "$mount_dir" ]] || { log "FAIL NFS root_squash case: $mount_dir does not exist"; exit 1; }
+    [[ -w "$mount_dir" ]] || { log "FAIL NFS root_squash case: $mount_dir is not writable by host user"; exit 1; }
+
+    total=$((total + 1))
+    local case_dir="$mount_dir/archivebox-root-squash-validation-$RUN_ID"
+    local log_file="$case_dir/output.log"
+    local owner uid gid status ok root_is_squashed=0
+    mkdir -p "$case_dir/data"
+    owner="$(stat -c '%u:%g' "$case_dir/data")"
+    uid="${owner%%:*}"
+    gid="${owner##*:}"
+
+    if ! "${docker_base[@]}" \
+        --user 0:0 \
+        -v "$case_dir/data:/data" \
+        --entrypoint /bin/touch \
+        "$IMAGE" \
+        /data/root-must-not-write 2>/dev/null; then
+        root_is_squashed=1
+    fi
+    rm -f "$case_dir/data/root-must-not-write"
+
+    set +e
+    "${docker_base[@]}" \
+        -e "PUID=$uid" \
+        -e "PGID=$gid" \
+        -v "$ENTRYPOINT_PATH:/app/bin/docker_entrypoint.sh:ro" \
+        -v "$case_dir/data:/data" \
+        --entrypoint /app/bin/docker_entrypoint.sh \
+        "$IMAGE" \
+        sh -c "$default_cmd" >"$log_file" 2>&1
+    status=$?
+    set -e
+
+    ok=1
+    [[ "$root_is_squashed" == "1" ]] || ok=0
+    [[ "$uid" != "0" ]] || ok=0
+    [[ "$status" == "0" ]] || ok=0
+    grep -q "^ABX_UID=$uid$" "$log_file" || ok=0
+    grep -q "^ABX_GID=$gid$" "$log_file" || ok=0
+    grep -q '^ABX_OK$' "$log_file" || ok=0
+
+    if [[ "$ok" == "1" ]]; then
+        passed=$((passed + 1))
+        log "PASS NFS root_squash mount uses the writable non-root identity without root repair"
+    else
+        failed=$((failed + 1))
+        log "FAIL NFS root_squash mount (status=$status owner=$owner log=$log_file)"
+        sed -n '1,180p' "$log_file"
+    fi
+}
+
 run_compose_personas_case() {
     total=$((total + 1))
     local case_dir="$VALIDATION_ROOT/compose-personas-persist"
@@ -451,6 +516,54 @@ run_compose_personas_case() {
     fi
 }
 
+run_compose_fresh_up_case() {
+    total=$((total + 1))
+    local case_dir="$VALIDATION_ROOT/compose-fresh-up"
+    local log_file="$case_dir/output.log"
+    local status=0
+    mkdir -p "$case_dir"
+    local ARCHIVEBOX_IMAGE="$IMAGE"
+    local ARCHIVEBOX_PORT=0
+    export ARCHIVEBOX_IMAGE ARCHIVEBOX_PORT
+
+    set +e
+    "$DOCKER_BINARY" compose \
+        --project-directory "$case_dir" \
+        -f "$COMPOSE_PATH" \
+        up -d --wait >"$log_file" 2>&1
+    status=$?
+    if [[ "$status" == "0" ]]; then
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" \
+            exec -T archivebox sh -c \
+            'server_identity="$(ps -eo uid=,gid=,args= | awk '\''/\/archivebox server --init/ && !/awk/ {print $1 ":" $2; exit}'\'')"; printf "ABX_SERVER_ID=%s\n" "$server_identity"; test "$server_identity" = 911:911; setpriv --reuid=911 --regid=911 --clear-groups sh -c '\''touch /data/logs/fresh-up-probe /data/archive/fresh-up-probe "$PERSONAS_DIR/Default/chrome_profile/fresh-up-probe" && rm -f /data/logs/fresh-up-probe /data/archive/fresh-up-probe "$PERSONAS_DIR/Default/chrome_profile/fresh-up-probe"'\''' \
+            >>"$log_file" 2>&1 || status=$?
+    fi
+    if [[ "$status" == "0" ]]; then
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" restart archivebox \
+            >>"$log_file" 2>&1 || status=$?
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" up -d --wait \
+            >>"$log_file" 2>&1 || status=$?
+    fi
+    "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" logs --no-color archivebox \
+        >>"$log_file" 2>&1 || true
+    "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" down --volumes --remove-orphans \
+        >>"$log_file" 2>&1 || status=$?
+    set -e
+
+    if [[ "$status" == "0" ]] \
+        && [[ -d "$case_dir/data/logs" ]] \
+        && [[ -d "$case_dir/data/personas/Default/chrome_profile" ]] \
+        && grep -q '^ABX_SERVER_ID=911:911$' "$log_file" \
+        && ! grep -q 'cannot write to /data' "$log_file"; then
+        passed=$((passed + 1))
+        log "PASS fresh compose up creates writable data and personas paths and survives restart"
+    else
+        failed=$((failed + 1))
+        log "FAIL fresh compose up permission initialization (status=$status log=$log_file)"
+        sed -n '1,220p' "$log_file"
+    fi
+}
+
 log "Running UID/GID validation on ${HOSTNAME:-unknown-host} using image=$IMAGE entrypoint=$ENTRYPOINT_PATH root=$VALIDATION_ROOT platform=${DOCKER_PLATFORM:-native}"
 
 run_case "root-owned empty data auto-detect default" \
@@ -477,6 +590,42 @@ run_case "nested root-owned archive content is not recursively chowned" \
     "chown 0:0 /case/data && chmod 755 /case/data && mkdir -p /case/data/archive/existing && touch /case/data/archive/existing/file && chown -R 0:0 /case/data/archive/existing" \
     "-" "-" pass 911 911 "$default_cmd" nested-root-stays
 
+run_case "explicit PUID and PGID override root-owned data" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=1234 PGID=2345" "-" pass 1234 2345
+
+run_case "PUID-only override retains detected group" \
+    "chown 501:20 /case/data && chmod 755 /case/data" \
+    "PUID=1234" "-" pass 1234 20
+
+run_case "PGID-only override retains detected user" \
+    "chown 501:20 /case/data && chmod 755 /case/data" \
+    "PGID=2345" "-" pass 501 2345
+
+run_case "PUID zero falls back to the non-root default" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=0 PGID=911" "-" pass 911 911
+
+run_case "nonnumeric PUID is rejected" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=archivebox PGID=911" "-" fail "" ""
+
+run_case "non-root UID with root group is supported" \
+    "chown 1201:0 /case/data && chmod 775 /case/data" \
+    "-" "-" pass 1201 0
+
+run_case "explicit non-root PUID with PGID zero is supported" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=1201 PGID=0" "-" pass 1201 0
+
+run_case "writable cache and persona profile skip recursive startup scans" \
+    "chown 911:911 /case/data && mkdir -p /case/lib/cache/uv/nested /case/data/personas/Default/chrome_profile/nested && touch /case/lib/cache/uv/nested/marker /case/data/personas/Default/chrome_profile/SingletonLock /case/data/personas/Default/chrome_profile/nested/SingletonLock && chown 911:911 /case/lib/cache/uv /case/data/personas/Default/chrome_profile && chown 0:0 /case/lib/cache/uv/nested/marker" \
+    "-" "-" pass 911 911 "$default_cmd" cache-and-profile-shallow
+
+run_case "writable forced-owner mount skips metadata changes" \
+    "chown 1000:1000 /case/data && chmod 777 /case/data" \
+    "PUID=911 PGID=911" "-" pass 911 911 "$default_cmd" data-owner-stays
+
 run_case "only the writable Chrome extension cache is recursively repaired" \
     "chown 501:20 /case/data && mkdir -p /case/lib/existing /case/lib/chromewebstore/extensions/cookie /case/browsers/existing && touch /case/lib/existing/file /case/lib/chromewebstore/extensions/cookie/rules.json /case/browsers/existing/file && chown -R 0:0 /case/lib/existing /case/lib/chromewebstore /case/browsers/existing" \
     "-" "-" pass 501 20 "$runtime_dependency_cmd" runtime-nested-root-stays
@@ -506,8 +655,9 @@ run_case "root-owned ArchiveBox.conf only is repaired" \
     "-" "-" pass 911 911 "$default_cmd" config-files-repaired
 
 run_compose_personas_case
+run_compose_fresh_up_case
 
-run_mount_case "NFS" "${NFS_TEST_DIR:-}"
+run_nfs_root_squash_case "${NFS_TEST_DIR:-}"
 run_mount_case "SMB" "${SMB_TEST_DIR:-}"
 
 log "SUMMARY passed=$passed failed=$failed total=$total"

docker-compose.yml

@@ -29,6 +29,8 @@ services:
             - tmp:/tmp/archivebox
         pids_limit: 2048                       # avoids low systemd task caps with Chrome and parallel extractors
         environment:
+            - PUID                                                                   # optional numeric UID for NFS/CIFS/FUSE mounts (defaults to collection owner or 911)
+            - PGID                                                                   # optional numeric GID for NFS/CIFS/FUSE mounts (defaults to collection group or 911)
             # - ADMIN_USERNAME=admin                             # creates an admin user on first run with the given user/pass combo
             # - ADMIN_PASSWORD=SomeSecretPassword
             # - ARCHIVEBOX_VNC_PERSONA=personal                    # persona to open in VNC (defaults to DEFAULT_PERSONA)

docs/Chromium-Install.md

@@ -149,6 +149,22 @@ ssh -N -L 8080:127.0.0.1:8080 user@your-server
 
 4. Open a page and confirm you can view it while logged in. For a new persona, log in to the sites you want to archive. Then follow [Save and check the page](#3-save-and-check-the-page), selecting the same persona.
 
+#### One-off Compose setup
+
+To set up a persona without starting the ArchiveBox server, enable the same `novnc` service and healthy `depends_on` block, then run:
+
+```bash
+docker compose up -d --wait novnc
+docker compose run --rm archivebox persona create personal
+docker compose run --rm -e DISPLAY=novnc:0.0 archivebox persona open personal
+```
+
+Connect at <http://localhost:8080/vnc.html>, log in, and close the browser when finished. The persona remains in `data/personas/personal` across one-off containers.
+
+`persona open` resolves the installed Chromium binary automatically. Do not hard-code `/data/lib/env/bin/chromium` or a versioned Playwright cache path: the browser location varies by image version and architecture.
+
+An explicitly set `DISPLAY` is preserved. When it is unset, ArchiveBox probes `novnc:0.0` and falls back to headless operation if unavailable. Set `DISPLAY=` to disable this automatic display selection.
+
 ## Troubleshooting Chromium Install
 
 <a name="more-info--troubleshooting"></a>

docs/Configuration.md

@@ -411,6 +411,8 @@ ArchiveBox splits its surfaces across three logical hosts: `admin.*` (Django adm
 
 SingleFile output is served as ordinary HTML in every mode; it remains usable in no-JS modes because SingleFile removes the page's scripts during capture. ArchiveWeb.page/ReplayWeb.page and MHTML use their existing trusted preview templates. ArchiveBox always attempts to load these viewers, including when the incoming request is plain HTTP, because HTTPS may be terminated by an upstream proxy. Browser service-worker rules still require ReplayWeb.page to be reached through HTTPS or localhost for replay to initialize.
 
+Trusted preview wrappers allow embedding by their own origin and the configured web/admin origins, so the collection UI can display snapshot-host previews without opening them to arbitrary embedding sites. PDF previews use a document frame rather than an object/embed, retaining `object-src 'none'` on the wrapper. Allowing a trusted viewer's scripts does not exempt the archived documents inside it from their replay policy; the MHTML viewer keeps its archived child document sandboxed without scripts.
+
 > [!WARNING]
 > Switching to any mode whose name starts with `unsafe-` or `danger-` is logged at startup and surfaces a banner in the UI. **Don't use these modes on a public hostname** — archived JavaScript will run on the same origin as your admin session.
 
@@ -585,6 +587,9 @@ When `True`, every LDAP user who successfully authenticates is auto-promoted to
 
 *Options for the on-disk layout, file permissions, and temp/lib directories that ArchiveBox reads and writes during archiving.*
 
+> [!NOTE]
+> Docker-only [`PUID` and `PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid) are entrypoint environment variables, not ArchiveBox configuration keys.
+
 ---
 <a id="dir_output_permissions"></a>
 #### `OUTPUT_PERMISSIONS`

docs/Docker.md

@@ -128,7 +128,18 @@ docker compose exec archivebox archivebox add --depth=1 'https://example.com/som
 
 ### Accessing the data
 
-The outputted archive data is stored in `data/` (relative to the project root), or whatever folder path you specified in the `docker-compose.yml` `volumes:` section. The mounted directory must be writable by its current owner; the entrypoint detects that non-root owner and runs ArchiveBox with matching permissions.
+The outputted archive data is stored in `data/` (relative to the project root), or whatever folder path you specified in the `docker-compose.yml` `volumes:` section. The mounted directory must be writable by its current owner; by default the entrypoint detects that non-root owner and runs ArchiveBox with matching permissions.
+
+<a id="puid--pgid"></a>
+#### Docker `PUID` / `PGID`
+
+`PUID` and `PGID` are optional Docker-entrypoint environment variables for mounts with a fixed numeric identity, such as NFS/CIFS/FUSE. They override owner autodetection and are not ArchiveBox config keys: do not put them in `ArchiveBox.conf` or use `archivebox config --set`. Both must be numeric. `PUID=0` is ignored in favor of the detected/default non-root UID so ArchiveBox and Chrome never run as root; `PGID=0` is permitted for a group-writable mount.
+
+```bash
+PUID=1000 PGID=1000 docker compose up -d --wait
+```
+
+Leave them unset for normal local bind mounts. When you do set them, match the effective numeric owner/group or ACL configured by the storage server; container root cannot override server-side permission enforcement such as NFS `root_squash`.
 
 To access a result directly via the filesystem, follow its backwards-compatible `./data/archive/<timestamp>` symlink, or browse the canonical `./data/archive/users/<user>/snapshots/<date>/<domain>/<uuid>/` tree.
 
@@ -207,7 +218,7 @@ Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local exampl
 
 *(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
 
-If you encounter permissions issues, make sure the mounted data directory is writable by its intended owner. Docker startup automatically uses the first non-root owner detected from the existing collection, or the default `archivebox` user when the data directory is root-owned.
+If you encounter permissions issues, make sure the mounted data directory is writable by its intended owner. Docker startup uses explicit [`PUID`/`PGID`](#puid--pgid) values when supplied, otherwise the first non-root owner detected from the existing collection or the default `archivebox` user when the data directory is root-owned.
 
 <br/>
 

docs/Security-Overview.md

@@ -150,7 +150,7 @@ How much are you planning to archive? Only a few bookmarked articles, or thousan
 
 Consider what permissioning to apply to your archive folder carefully. Limit access to the fewest possible users by checking folder ownership and setting [`OUTPUT_PERMISSIONS`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#output_permissions) accordingly. Generally the `index.sqlite3` file, `archive/` folder, and `ArchiveBox.conf` file must all be owned and writable by the `archivebox` user or a dedicated non-root user.
 
-When running with Docker, the entrypoint uses the existing non-root owner of the mounted data directory when possible, otherwise it falls back to the image's `archivebox` user.
+When running with Docker, the entrypoint uses explicit [`PUID`/`PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid) values when supplied, otherwise the existing non-root owner of the mounted data directory or the image's `archivebox` user.
 
 More info:
 - https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout

docs/Setting-Up-Storage.md

@@ -127,6 +127,8 @@ The `data/archive/` subfolder contains the bulk archived content, and it support
 
 ### `NFS` (Docker Driver)
 
+Configure the export's ownership or ACL so a non-root numeric identity can create and remove files, then set Docker `PUID`/`PGID` to that identity if it differs from the local collection owner. NFS `root_squash` is compatible; ArchiveBox does not rely on container root to write the export.
+
 `docker-compose.yml`:
 ```yaml
 services:
@@ -148,6 +150,8 @@ volumes:
 
 ### `SMB` / `Ceph` (Docker CIFS Driver)
 
+The example below presents the mount as the image default `911:911`. If you choose another forced `uid`/`gid`, set the ArchiveBox service's `PUID`/`PGID` to the same values.
+
 `docker-compose.yml`:
 ```yaml
 services:
@@ -331,7 +335,8 @@ volumes:
             allow_other: 'true'
             vfs_cache_mode: full
             vfs_links: 'true'
-            # Match these to the numeric owner of ./data; 911:911 is the image default.
+            # Match these and the ArchiveBox PUID/PGID to the effective mount owner;
+            # 911:911 is the image default.
             uid: 911
             gid: 911
             transfers: 16

docs/Troubleshooting.md

@@ -134,7 +134,23 @@ if you have problem with a particular nginx config.
 
 #### Docker Permissions issues
 
-Make sure the mounted data directory is writable by its intended non-root owner. The current Docker entrypoint detects the first non-root collection owner and runs ArchiveBox with matching numeric UID/GID; a new root-owned collection falls back to the image's `archivebox` user. Check the host directory's numeric ownership and the entrypoint's startup output before changing permissions.
+The Docker entrypoint starts as root for mount and runtime-cache setup, then runs ArchiveBox and Chrome as a non-root user. It checks required paths as that user first, so normal starts do not traverse cache trees; an inaccessible runtime/cache tree may need a one-time recursive repair. Browser-lock cleanup checks only known files at each persona profile root, not the contents of large profiles. By default the entrypoint uses the first non-root numeric owner found in the collection, falling back to `911:911` for a new or root-owned collection. Set `PUID` and `PGID` when an NFS/CIFS/FUSE server or host requires a specific numeric identity.
+
+Required collection directories get a non-root access check or creation attempt first, followed by real create/delete probes on the critical output paths. Only a failed check triggers a shallow root `chown`/`chmod` of the exact collection path. This avoids repeated metadata changes on writable `root_squash` or synthetic-permission mounts. Startup never recursively scans or changes `data/archive`: byte size is not a safe proxy for traversal cost, and a large or metadata-heavy archive can take hours or days to walk.
+
+With SQLite, `index.sqlite3`, configuration, logs, temp/runtime files, and sidecar databases must remain on reliable local storage; only `data/archive/` may be remote. PostgreSQL is the supported exception for the main index.
+
+The default `restart: unless-stopped` policy repeats a startup failure but does not create it and should not be removed. If the entrypoint still cannot create and delete its probe file, stop the noisy loop, inspect the resolved volume sources and logs, then retry once after fixing the server-side UID/GID mapping or ACL:
+
+```bash
+docker compose stop archivebox
+docker compose config
+docker compose logs --tail=200 archivebox
+docker compose run --rm archivebox init
+docker compose up -d --wait
+```
+
+A read-only mount or NFS export that denies both the selected user and root cannot be repaired from inside the container.
 
 <br/>
 

docs/codeblocks.toml

@@ -21,7 +21,7 @@ version = 2
 "8fe4a2c15abb0ab2-1" = "run"
 "2b08779e4d24d305-1" = "run"
 "588be84a1f0fda9a-3" = "illustration"
-"1b9438e093866ff0-1" = "illustration"
+"9ab59dbbf9d5cedb-1" = "illustration"
 "6392c7b5f3593d07-1" = "run"
 "fe137daf1317bcd6-1" = "run"
 "724cf9fe3251b66f-1" = "illustration"
@@ -83,13 +83,15 @@ version = 2
 
 # docs/Chromium-Install.md
 "201988374bff6d14-1" = "run"
-"d85a11895d69713a-1" = "run"
-"d8061ea425363f8c-1" = "illustration"
-"bd5e6d3fc3193a25-1" = "illustration"
-"e41aab6197f32cb4-1" = "illustration"
-"c5c46f8a1c693a4c-1" = "illustration"
-"ce2edd9be87280fc-1" = "run"
-"7890998169008ad0-1" = "illustration"
+"e305c234db136566-1" = "run"
+"56056d409ad85f85-1" = "illustration"
+"baaf307bd7ee0f36-1" = "illustration"
+"b274b9afd4304207-1" = "illustration"
+"3bc4ad1dc41c32c6-1" = "illustration"
+"3dcfedd701e87748-1" = "illustration"
+"6c64433356e03458-1" = "illustration"
+"e9088da849bc5755-1" = "illustration"
+"b32c747a730e6f8b-1" = "illustration"
 
 # docs/Configuration.md
 "129eb9ca56f17500-1" = "illustration"
@@ -113,6 +115,7 @@ version = 2
 "746a3e228d29f9d2-1" = "run"
 "66a7d54ea81d64dc-1" = "illustration"
 "aa4884ebe3101cf6-1" = "illustration"
+"0b7277780e3efb1a-1" = "illustration"
 "150ae2c7f6d012ab-1" = "illustration"
 "8d86727108f1dbad-1" = "run"
 "6bcb6c2e8770907a-1" = "illustration"
@@ -166,6 +169,12 @@ version = 2
 "16a101113c74c070-1" = "illustration"
 "c84835e713d1c625-1" = "run"
 
+# docs/Release-Notes-v0.9.md
+"7f9cf63fad110997-1" = "illustration"
+"22c61747fad47c75-1" = "illustration"
+"693b5aec2007eea7-1" = "illustration"
+"c28c2cb9ba1f19ae-1" = "illustration"
+
 # docs/Scheduled-Archiving.md
 "e6102e50b4491a0d-1" = "illustration"
 "e256179d1504c37d-1" = "illustration"
@@ -195,7 +204,7 @@ version = 2
 "0fdcf228bb957933-1" = "run"
 "d76a2cce93150387-1" = "illustration"
 "acb33c3336d33ac5-1" = "illustration"
-"8bc58240c72e3eaa-1" = "illustration"
+"64a482f8906997f7-1" = "illustration"
 "74e75b4a84bbb636-1" = "run"
 
 # docs/Setting-up-Authentication.md
@@ -224,6 +233,7 @@ version = 2
 "201988374bff6d14-2" = "run"
 "725a7a8c6f829ece-1" = "run"
 "7658fabf90ed3c2e-1" = "run"
+"1192620577a0dc5a-1" = "illustration"
 "9612e7b40b348ab1-1" = "illustration"
 "2f491912a15ecd93-1" = "output"
 "e0cbb64ddf1d7135-1" = "run"
@@ -246,7 +256,6 @@ version = 2
 "6611c2dee47b1092-1" = "illustration"
 "2560680733c8e908-1" = "illustration"
 "debc2548308dec0c-1" = "illustration"
-"d29b422c54745a7a-1" = "illustration"
 "f6a482c5e0582b05-1" = "illustration"
 "5cd641727f4b04bb-1" = "output"
 "f0e895e5b375c198-1" = "illustration"
@@ -284,8 +293,7 @@ version = 2
 "46fe9d81940ea385-1" = "collection"
 "396490664ce459e9-2" = "project"
 "201988374bff6d14-1" = "collection"
-"d85a11895d69713a-1" = "collection"
-"ce2edd9be87280fc-1" = "collection"
+"e305c234db136566-1" = "collection"
 "798793d9d6b2e12e-1" = "collection"
 "ce0de8ed0d7cd552-1" = "collection"
 "83cb9be168d203ca-1" = "collection"
@@ -364,8 +372,7 @@ version = 2
 "46fe9d81940ea385-1" = "ubuntu"
 "396490664ce459e9-2" = "ubuntu"
 "201988374bff6d14-1" = "ubuntu"
-"d85a11895d69713a-1" = "ubuntu"
-"ce2edd9be87280fc-1" = "ubuntu"
+"e305c234db136566-1" = "ubuntu"
 "798793d9d6b2e12e-1" = "ubuntu"
 "ce0de8ed0d7cd552-1" = "ubuntu"
 "83cb9be168d203ca-1" = "ubuntu"

etc/package.json

@@ -1,6 +1,6 @@
 {
   "name": "archivebox",
-  "version": "0.9.35rc442",
+  "version": "0.9.35rc482",
   "repository": "github:ArchiveBox/ArchiveBox",
   "license": "MIT",
   "dependencies": {

publicsite/index.html

@@ -42,25 +42,20 @@
     }
   }
   </script>
+<link rel="stylesheet" href="site-chrome.css">
 </head>
 <body>
-  <header class="site-header">
-    <a class="brand" href="#top" aria-label="ArchiveBox home">
-      <img src="./icon.png" alt="" width="42" height="42">
-      <span>ArchiveBox</span>
-    </a>
-    <nav aria-label="Primary navigation">
-      <a href="#install"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 3v11"/><path d="m7 10 5 5 5-5"/><path d="M5 21h14"/></svg><span>Install</span></a>
-      <a href="#use-cases"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M16 21v-2a4 4 0 0 0-4-4H6a4 4 0 0 0-4 4v2"/><circle cx="9" cy="7" r="4"/><path d="M22 21v-2a4 4 0 0 0-3-3.87"/><path d="M16 3.13a4 4 0 0 1 0 7.75"/></svg><span>Use cases</span></a>
-      <a href="#configuration"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 21v-7"/><path d="M4 10V3"/><path d="M12 21v-9"/><path d="M12 8V3"/><path d="M20 21v-5"/><path d="M20 12V3"/><path d="M2 14h4"/><path d="M10 8h4"/><path d="M18 16h4"/></svg><span>Configuration</span></a>
-      <a href="#layout-security"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 22s8-4 8-10V5l-8-3-8 3v7c0 6 8 10 8 10Z"/><path d="m9 12 2 2 4-4"/></svg><span>Layout &amp; security</span></a>
-      <a href="#background"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 8v4l3 3"/><circle cx="12" cy="12" r="9"/></svg><span>Backstory</span></a>
-      <a href="#ecosystem"><svg viewBox="0 0 24 24" aria-hidden="true"><rect x="3" y="3" width="7" height="7" rx="1.5"/><rect x="14" y="3" width="7" height="7" rx="1.5"/><rect x="14" y="14" width="7" height="7" rx="1.5"/><path d="M10 6.5h4"/><path d="M17.5 10v4"/><path d="M10 17.5h4"/></svg><span>Ecosystem</span></a>
-      <a href="https://github.com/ArchiveBox/ArchiveBox/wiki"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M4 19.5A2.5 2.5 0 0 1 6.5 17H20"/><path d="M6.5 2H20v20H6.5A2.5 2.5 0 0 1 4 19.5v-15A2.5 2.5 0 0 1 6.5 2Z"/></svg><span>Docs</span></a>
-      <a href="https://github.com/ArchiveBox/ArchiveBox"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M15 22v-4a4.8 4.8 0 0 0-.09-.93"/><path d="M9 22v-4a4.8 4.8 0 0 1 .09-.93"/><path d="M17.8 20.2c1.4-.5 2.2-1.7 2.2-3.7V12.7c0-.7-.2-1.4-.7-2 .1-.3.3-1.7-.1-3.2 0 0-1.1-.4-3.4.9a11.5 11.5 0 0 0-6 0C7.5 7.1 6.4 7.5 6.4 7.5c-.4 1.5-.2 2.9-.1 3.2-.5.6-.7 1.3-.7 2v3.8c0 2 1 3.2 2.4 3.7"/><path d="M9 18c-3 1.3-5 0-6-2"/></svg><span>GitHub</span></a>
-      <a href="https://pypi.org/project/archivebox/"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M16.5 9.4 7.5 4.2"/><path d="M21 16V8a2 2 0 0 0-1-1.73l-7-4a2 2 0 0 0-2 0l-7 4A2 2 0 0 0 3 8v8a2 2 0 0 0 1 1.73l7 4a2 2 0 0 0 2 0l7-4A2 2 0 0 0 21 16Z"/><path d="M3.3 7 12 12l8.7-5"/><path d="M12 22V12"/></svg><span>PyPI</span></a>
-    </nav>
-  </header>
+  <!-- BEGIN ARCHIVEBOX HEADER — generated by monorepo/site-chrome/sync.py -->
+<header class="abx-header">
+  <a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">Web Archive</span></a>
+  <nav class="abx-nav" aria-label="Main navigation">
+    <details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
+    <a href="#use-cases">Use cases</a><a href="#configuration">Configuration</a><a href="#layout-security">Layout &amp; security</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
+    <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
+    <a class="abx-cta" href="#install">Get started</a>
+  </nav>
+</header>
+<!-- END ARCHIVEBOX HEADER -->
 
   <main id="top">
     <section class="hero section">
@@ -463,18 +458,39 @@
     </section>
   </main>
 
-  <footer class="site-footer">
-    <div class="brand">
-      <img src="./icon.png" alt="" width="34" height="34">
-      <span>ArchiveBox</span>
-    </div>
-    <p>Open-source self-hosted web archiving. MIT licensed, nonprofit-supported, and built for durable collections.</p>
+  <!-- BEGIN ARCHIVEBOX FOOTER — generated by monorepo/site-chrome/sync.py -->
+<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
+  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
+  <div class="abx-footer-local"><p>Open-source self-hosted web archiving. MIT licensed, nonprofit-supported, and built for durable collections.</p>
     <div>
       <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE">License</a>
       <a href="https://github.com/ArchiveBox/ArchiveBox/releases">Changelog</a>
       <a href="https://zulip.archivebox.io">Community chat</a>
       <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Donations">Donations</a>
-    </div>
-  </footer>
+    </div></div>
+  <div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
+</footer>
+<script>
+(() => {
+  const menu = document.querySelector('.abx-apps');
+  document.addEventListener('keydown', event => {
+    if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
+  });
+  document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
+  function revealAnchor() {
+    let target;
+    try { target = document.getElementById(decodeURIComponent(location.hash.slice(1))); } catch { return; }
+    if (!target) return;
+    let details = target.closest('details');
+    let opened = false;
+    while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
+    if (opened) requestAnimationFrame(() => target.scrollIntoView());
+  }
+  addEventListener('hashchange', revealAnchor);
+  revealAnchor();
+})();
+</script>
+<!-- END ARCHIVEBOX FOOTER -->
 </body>
 </html>

publicsite/site-chrome.css

@@ -0,0 +1,75 @@
+/* Shared ArchiveBox site chrome. Source: ArchiveBox/monorepo/site-chrome. */
+.abx-header, .abx-footer { box-sizing: border-box; font-family: -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; font-size: 14px; line-height: 1.5; letter-spacing: normal; text-align: left; }
+.abx-header *, .abx-footer * { box-sizing: border-box; }
+.abx-header { position: sticky; top: 0; z-index: 100; display: flex; align-items: center; justify-content: space-between; gap: 24px; width: 100%; max-width: none; margin: 0; padding: 15px max(24px, calc((100% - 1280px) / 2)); border: 0; border-bottom: 1px solid #e9e2e3; background: #fcfaf7f5; color: #28252a; backdrop-filter: blur(16px); }
+.abx-header a, .abx-footer a { text-decoration: none; text-underline-offset: .24em; }
+.abx-header .abx-brand, .abx-footer .abx-brand { display: inline-flex; flex: 0 0 auto; align-items: center; gap: 11px; margin: 0; padding: 0; color: inherit; font-size: 19px; font-weight: 750; line-height: 1.2; letter-spacing: -.5px; white-space: nowrap; }
+.abx-brand .abx-logo { display: block; flex: 0 0 auto; width: 32px; height: 32px; margin: 0; }
+.abx-header .abx-subsite { margin-left: 2px; color: #716c74; font-size: 12px; font-weight: 450; letter-spacing: 0; }
+.abx-header .abx-nav { display: flex; align-items: center; justify-content: flex-end; flex-wrap: wrap; gap: 0; width: auto; max-width: none; margin: 0; padding: 0; font-size: 13px; font-weight: 600; }
+.abx-nav > a, .abx-nav > details { margin: 0 0 0 15px; padding: 0 0 0 15px; border: 0; border-left: 1px solid #ded4d8; border-radius: 0; background: none; }
+.abx-nav > :first-child { margin-left: 0; padding-left: 0; border-left: 0; }
+.abx-header .abx-nav a { display: inline-flex; align-items: center; min-height: 36px; color: #4d424b; white-space: nowrap; }
+.abx-header .abx-nav a:hover { color: #9b2854; }
+.abx-header .abx-nav .abx-cta { margin-left: 18px; padding: 9px 17px; border: 1px solid #9b2854; border-radius: 24px; background: #9b2854; color: #fff; }
+.abx-header .abx-nav .abx-cta:hover { background: #782040; color: #fff; }
+.abx-header .abx-apps { position: relative; }
+.abx-apps > summary { display: flex; align-items: center; gap: 7px; min-height: 36px; list-style: none; cursor: pointer; color: #4d424b; }
+.abx-apps > summary::-webkit-details-marker { display: none; }
+.abx-apps > summary::before { content: '›'; display: inline-block; color: #9b2854; font-size: 22px; line-height: 1; transition: transform .15s; }
+.abx-apps[open] > summary::before { transform: rotate(90deg); }
+.abx-header .abx-app-links { position: absolute; right: 0; top: calc(100% + 12px); width: 305px; max-height: min(70vh, 580px); overflow-y: auto; padding: 10px; border: 1px solid #e9e2e3; border-radius: 14px; background: #fcfaf7; box-shadow: 0 16px 48px #2d152522; }
+.abx-header .abx-app-links a { display: flex; min-height: 42px; padding: 9px 12px; border-radius: 7px; white-space: normal; }
+.abx-header .abx-app-links code { display: inline; vertical-align: baseline; font-family: ui-monospace, SFMono-Regular, Consolas, monospace; font-size: 1em; font-weight: inherit; line-height: inherit; margin: 0 .3em; padding: 0; border: 0; background: transparent; color: inherit; }
+.abx-header .abx-app-links a:hover { background: #f2e8ed; }
+.abx-footer { position: relative; z-index: 1; display: block; width: 100%; max-width: none; margin: 72px 0 0; padding: 60px max(24px, calc((100% - 1280px) / 2)) 24px; border: 0; border-top: 3px solid #7e284a; background: radial-gradient(ellipse at 12% 0, #54243b55, transparent 48%), linear-gradient(120deg, #201923, #14151b 75%); color: #ddd2dd; color-scheme: dark; }
+.abx-footer .abx-footer-intro { display: flex; align-items: flex-start; justify-content: space-between; gap: 32px; margin: 0 0 42px; }
+.abx-footer .abx-brand { color: #fff4ed; font-size: 24px; }
+.abx-footer .abx-brand .abx-logo { width: 40px; height: 40px; }
+.abx-footer .abx-tagline { max-width: 430px; margin: 12px 0 0; color: #b8acbb; font-size: 14px; line-height: 1.7; }
+.abx-footer .abx-footer-motto { margin: 2px 0 0; color: #d8b39f; font: italic 25px/1.45 Georgia, serif; text-align: right; }
+.abx-footer .abx-footer-grid { display: grid; grid-template-columns: repeat(3, minmax(0, 1fr)); gap: 42px; margin: 0; }
+.abx-footer .abx-footer-column { margin: 0; padding: 20px 0 0; border: 0; border-top: 1px solid #67505e66; background: none; }
+.abx-footer .abx-footer-column h2 { margin: 0 0 19px; padding: 0; border: 0; color: #e4acbb; font: 650 12px/1.4 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif; letter-spacing: .12em; text-transform: uppercase; }
+.abx-footer .abx-footer-column ul { display: block; list-style: none; margin: 0; padding: 0; }
+.abx-footer .abx-footer-column li { display: flex; align-items: center; gap: 10px; margin: 0; padding: 0; min-height: 36px; }
+.abx-footer .abx-footer-column li::before { content: none; }
+.abx-footer .abx-footer-column a { color: #ddd2dd; font-size: 13px; line-height: 1.55; }
+.abx-footer .abx-footer-column a:first-child { padding: 6px 0; }
+.abx-footer a:hover { color: #fff; text-decoration: underline; }
+.abx-footer .abx-source { display: inline-flex; align-items: center; justify-content: center; flex: 0 0 28px; width: 28px; height: 28px; border: 1px solid transparent; border-radius: 50%; color: #a58d9f; }
+.abx-footer .abx-source:hover { color: #f3b5cb; border-color: #80516a; background: #ffffff08; }
+.abx-source svg { width: 15px; height: 15px; fill: currentColor; }
+.abx-footer .abx-footer-bottom { display: flex; flex-wrap: wrap; align-items: center; justify-content: space-between; gap: 12px 24px; margin-top: 32px; padding-top: 21px; border-top: 1px solid #67505e66; color: #a99bab; font-size: 12px; }
+.abx-footer .abx-footer-bottom p { margin: 0; font-size: inherit; color: inherit; }
+.abx-footer .abx-footer-bottom a { color: #cebbc9; }
+.abx-footer .abx-footer-local { margin: 24px 0 0; padding: 16px 0 0; border: 0; border-top: 1px solid #67505e66; border-radius: 0; background: none; color: #baacbc; font-size: 12px; }
+.abx-footer-local > summary { cursor: pointer; color: #d6bfce; }
+.abx-footer-local > div { padding-top: 18px; }
+.abx-footer-local a { color: #e5aabd; }
+.abx-footer-local .brand, .abx-footer-local .footer-intro, .abx-footer-local .footer-grid, .abx-footer-local .footer-meta { color: inherit; }
+.abx-footer-local .footer-card { background: transparent; border-color: #67505e66; }
+.abx-footer-local h2, .abx-footer-local h3, .abx-footer-local strong { color: #e4d8e3; }
+.abx-footer-local .footer-link-copy span, .abx-footer-local .footer-card-subtitle { color: #baacbc; }
+.abx-header a:focus-visible, .abx-header summary:focus-visible, .abx-footer a:focus-visible, .abx-footer summary:focus-visible { outline: 3px solid #d67c9f; outline-offset: 4px; }
+@media (max-width: 1100px) {
+  .abx-header { flex-wrap: wrap; gap: 8px 24px; padding: 12px 20px; }
+  .abx-header .abx-nav { justify-content: flex-start; }
+  .abx-header .abx-app-links { position: fixed; top: auto; left: 20px; right: 20px; width: auto; margin-top: 10px; }
+}
+@media (max-width: 700px) {
+  .abx-header { position: relative; padding: 14px 18px; }
+  .abx-header .abx-brand { flex-wrap: wrap; row-gap: 5px; }
+  .abx-header .abx-nav { width: 100%; gap: 4px 0; font-size: 12px; }
+  .abx-nav > a, .abx-nav > details { margin-left: 10px; padding-left: 10px; }
+  .abx-header .abx-nav .abx-cta { margin-left: 10px; padding: 7px 13px; }
+  .abx-header .abx-app-links { left: 12px; right: 12px; max-height: 60vh; }
+  .abx-footer { padding: 40px 22px 22px; margin-top: 48px; }
+  .abx-footer .abx-footer-intro { display: block; margin-bottom: 30px; }
+  .abx-footer .abx-footer-motto { margin-top: 20px; text-align: left; font-size: 22px; }
+  .abx-footer .abx-footer-grid { grid-template-columns: 1fr; gap: 26px; }
+  .abx-footer .abx-footer-column li { min-height: 40px; }
+}
+@media (prefers-reduced-motion: reduce) { .abx-apps > summary::before { transition: none; } }
+
+.abx-footer-local > div > a + a { margin-left: 12px; }

publicsite/styles.css

@@ -125,7 +125,8 @@ code {
   gap: 8px 18px;
 }
 
-.site-header nav a {
+.site-header nav a,
+.site-header nav .project-menu > summary {
   display: inline-flex;
   align-items: center;
   gap: 8px;
@@ -140,7 +141,8 @@ code {
   transition: border-color 160ms ease, color 160ms ease;
 }
 
-.site-header nav a:hover {
+.site-header nav a:hover,
+.site-header nav .project-menu > summary:hover {
   color: var(--text);
   text-decoration: none;
   border-color: hsl(var(--primary) / 0.42);
@@ -806,7 +808,7 @@ h3 {
   }
 
   .site-header {
-    position: static;
+    position: relative;
     width: 100%;
     margin: 0;
     padding: 14px 16px;
@@ -816,7 +818,8 @@ h3 {
     width: 100%;
   }
 
-  .site-header nav a {
+  .site-header nav a,
+  .site-header nav .project-menu > summary {
     font-size: 0.74rem;
     letter-spacing: 0.1em;
   }
@@ -881,3 +884,16 @@ h3 {
     font-size: 0.82rem;
   }
 }
+
+/* Shared project navigation stays usable without JavaScript. */
+.project-menu { position: relative; flex: 0 0 auto; }
+.project-menu > summary { cursor: pointer; list-style: none; }
+.project-menu > summary::-webkit-details-marker { display: none; }
+.project-menu > summary::after { content: "▾"; }
+.project-menu > summary:focus-visible, .project-menu__links a:focus-visible { outline: 3px solid #9b2854; outline-offset: 3px; }
+.project-menu__links { position: absolute; right: 0; top: 100%; z-index: 30; width: 220px; max-width: calc(100vw - 32px); padding: 8px; border: 1px solid #e9e2e3; border-radius: 12px; background: #fff; box-shadow: 0 12px 32px #28252a20; }
+header nav .project-menu__links a { display: flex; align-items: center; min-height: 44px; padding: 8px 12px; color: #28252a; font-size: 13px; letter-spacing: normal; text-transform: none; text-decoration: none; white-space: nowrap; }
+header nav .project-menu__links a:hover, header nav .project-menu__links a[aria-current="page"] { background: #f8edf2; color: #9b2854; border-radius: 6px; }
+@media (max-width: 700px) { header.site-header { flex-wrap: wrap; } header.site-header nav { flex-wrap: wrap; gap: 8px 16px; } }
+
+@media (max-width: 1100px) { .project-menu__links { position: fixed; left: 16px; right: 16px; top: auto; width: auto; } }

pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "archivebox"
-version = "0.9.35rc442"
+version = "0.9.35rc482"
 requires-python = ">=3.13"
 description = "Self-hosted internet archiving solution."
 authors = [{name = "Nick Sweeting", email = "pyproject.toml@archivebox.io"}]
@@ -78,9 +78,9 @@ dependencies = [
     ### Extractor dependencies (runtime binaries resolved through abxpkg)
     ### Binary/Package Management
     "abxbus==2.5.56",                 # direct imports only; version constrained by abx-dl -> abx-plugins
-    "abxpkg==1.12.117",                # direct imports only; version constrained by abx-dl -> abx-plugins
-    "abx-plugins[opencode]==1.12.235", # optional AI runtime clients are owned by the plugin
-    "abx-dl==1.12.273",       # shared ArchiveBox downloader package
+    "abxpkg==1.12.125",                # direct imports only; version constrained by abx-dl -> abx-plugins
+    "abx-plugins[opencode]==1.12.266", # optional AI runtime clients are owned by the plugin
+    "abx-dl==1.12.300",       # shared ArchiveBox downloader package
     ### UUID7 backport for Python <3.14
     "uuid7>=0.1.0; python_version < '3.14'",  # provides the uuid_extensions module on Python 3.13
     "browser-cookie3>=0.20.1",
@@ -342,7 +342,7 @@ Donate = "https://github.com/ArchiveBox/ArchiveBox/wiki/Donations"
 
 
 [tool.bumpver]
-current_version = "v0.9.35rc442"
+current_version = "v0.9.35rc482"
 version_pattern = "vMAJOR.MINOR.PATCH[PYTAGNUM]"
 commit_message = "bump version {old_version} -> {new_version}"
 tag_message = "{new_version}"

uv.lock

@@ -13,18 +13,18 @@ supported-markers = [
 ]
 
 [options]
-exclude-newer = "2026-09-10T23:06:34.37843694Z"
+exclude-newer = "2026-09-13T08:21:42.97648167Z"
 exclude-newer-span = "P5D"
 
 [options.exclude-newer-package]
-abxbus = { timestamp = "2026-09-15T23:06:33.378450375Z", span = "PT1S" }
+abxbus = { timestamp = "2026-09-18T08:21:41.97649376Z", span = "PT1S" }
 abx-plugins = "2100-01-01T00:00:00Z"
 abx-dl = "2100-01-01T00:00:00Z"
 abxpkg = "2100-01-01T00:00:00Z"
 
 [[package]]
 name = "abx-dl"
-version = "1.12.273"
+version = "1.12.300"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "abx-plugins", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -38,14 +38,14 @@ dependencies = [
     { name = "rich", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/15/4e/653927d17263b986557f1a2d3d31bcd2bcbac945fe662555fba1a8bd9759/abx_dl-1.12.273.tar.gz", hash = "sha256:a5392f701cf695c3cf4aca20d9f7493475b6ee79071c0bbf6cec883ca17c6ee2", size = 87274 }
+sdist = { url = "https://files.pythonhosted.org/packages/92/8b/2a5c6fcdc38979e015744f7012b3819f0986a25fd9190c15bdb1c7ca7b64/abx_dl-1.12.300.tar.gz", hash = "sha256:915a2eef1a359b5ee2edb57117faafd51a899679bd06980ede886e7c22794ec7", size = 87713 }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/95/b3/660a4eae38d974c7688e0f4b4605c5219f73fffff35c88e47d58657aaf9d/abx_dl-1.12.273-py3-none-any.whl", hash = "sha256:edf652c999fcbbc67664a39c78327fddafaea31faa611a0c77a4ec1dd6a34ed3", size = 92247 },
+    { url = "https://files.pythonhosted.org/packages/af/fb/fcbb6385082de109b7187b6a5476a4082972edb527cad44126c28d2e5c45/abx_dl-1.12.300-py3-none-any.whl", hash = "sha256:5ffac3d16edd967d6622cee4fa4155329539708e6e9ab92ba1e4c262819364b5", size = 93004 },
 ]
 
 [[package]]
 name = "abx-plugins"
-version = "1.12.235"
+version = "1.12.266"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "abxbus", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -54,9 +54,9 @@ dependencies = [
     { name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "uv", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/85/d7/513351281663445d60e2b1bd6d6469066cdc3ca94aff2875069a6014914d/abx_plugins-1.12.235.tar.gz", hash = "sha256:49622aca8dcb1cf649d0b6798949b3458c1d1a046256980afe571ac8cf2da7ee", size = 269431, upload-time = "2026-09-15T22:58:49.789Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/90/09/26b3aaf382cd318be9210cf1694c47d0a5637b51b4853d60b5716ad7edcb/abx_plugins-1.12.266.tar.gz", hash = "sha256:018fe8cba456f45d282324794a0c2311e3e8fe6a6a68e2205c0b7b05def4d087", size = 335452, upload-time = "2026-09-18T08:12:26.252Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/f4/63/93d9f155477d3a086255f44c17f3d34beae9f4380df702bd9bf9ec9070ae/abx_plugins-1.12.235-py3-none-any.whl", hash = "sha256:669cde167360d78fa1e571596614e70ba9f6f03c0d6d75ee744af43cd1271341", size = 424393, upload-time = "2026-09-15T22:58:51.14Z" },
+    { url = "https://files.pythonhosted.org/packages/36/4c/d321fce97e2efa3f84a4dede773e64f0aa081944bd4382c1c7e7684af3bb/abx_plugins-1.12.266-py3-none-any.whl", hash = "sha256:bbf3d85d417d9a92c8f7662ca44da8aaf00024282707b67488685205515d65b9", size = 500937, upload-time = "2026-09-18T08:12:24.558Z" },
 ]
 
 [package.optional-dependencies]
@@ -82,7 +82,7 @@ wheels = [
 
 [[package]]
 name = "abxpkg"
-version = "1.12.117"
+version = "1.12.125"
 source = { registry = "https://pypi.org/simple" }
 dependencies = [
     { name = "platformdirs", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -90,9 +90,9 @@ dependencies = [
     { name = "rich-click", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
     { name = "typing-extensions", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
 ]
-sdist = { url = "https://files.pythonhosted.org/packages/00/35/bbfaccae109328c716cea4e0cdbf2d2dc5975f448203bb3f4daa605591a6/abxpkg-1.12.117.tar.gz", hash = "sha256:4d88cc293f343eee2be7ecaab1203dfbc20571a4a39b2afa2f866c9ef8032dec", size = 239387, upload-time = "2026-09-02T19:57:26.142Z" }
+sdist = { url = "https://files.pythonhosted.org/packages/87/fc/cf3f9d370ee93202c2e9239d267ca0b0dece7dc3595d36eebbedf3ab2946/abxpkg-1.12.125.tar.gz", hash = "sha256:14d98f4ff305f5ac84030f4d1f16a2f6c511ff5480ebfb08ba133f367a0770e1", size = 239374, upload-time = "2026-09-18T07:29:21.549Z" }
 wheels = [
-    { url = "https://files.pythonhosted.org/packages/73/46/372be1f8fa915eb254ad865b97fb41073acdb5573dbce3e23a6083417d87/abxpkg-1.12.117-py3-none-any.whl", hash = "sha256:76ec8afaac7b2d7aece7bdda2bbf606e6373db51eb4dcb906008052d09e59577", size = 253749, upload-time = "2026-09-02T19:57:27.437Z" },
+    { url = "https://files.pythonhosted.org/packages/bf/5c/dd0d1690e4c1d6241f44a5194a525af6769acdf33dd366a6b4b56f33c2b0/abxpkg-1.12.125-py3-none-any.whl", hash = "sha256:cfef2e06ed01b8774e8bdf0b8de36ab592a0dfa273993c4824628ab17d7d0fd1", size = 253748, upload-time = "2026-09-18T07:29:23.043Z" },
 ]
 
 [[package]]
@@ -127,7 +127,7 @@ wheels = [
 
 [[package]]
 name = "archivebox"
-version = "0.9.35rc442"
+version = "0.9.35rc482"
 source = { editable = "." }
 dependencies = [
     { name = "abx-dl", marker = "sys_platform == 'darwin' or sys_platform == 'linux'" },
@@ -226,10 +226,10 @@ dev = [
 
 [package.metadata]
 requires-dist = [
-    { name = "abx-dl", specifier = "==1.12.273" },
-    { name = "abx-plugins", extras = ["opencode"], specifier = "==1.12.235" },
+    { name = "abx-dl", specifier = "==1.12.300" },
+    { name = "abx-plugins", extras = ["opencode"], specifier = "==1.12.266" },
     { name = "abxbus", specifier = "==2.5.56" },
-    { name = "abxpkg", specifier = "==1.12.117" },
+    { name = "abxpkg", specifier = "==1.12.125" },
     { name = "archivebox", extras = ["sonic", "ldap", "debug"], marker = "extra == 'all'" },
     { name = "atomicwrites", specifier = "==1.4.1" },
     { name = "base32-crockford", specifier = ">=0.3.0" },