Bump release version to 0.9.35rc487

An explicit `DISPLAY=novnc:0.0` was silently removed by the Docker
entrypoint whenever `xdpyinfo` failed. In the reported `0.9.35rc381`
image, `xdpyinfo` is missing, so `docker compose run --rm` clears the
display even with a healthy noVNC server and Chromium exits with
“Missing X server or $DISPLAY”.

Preserve explicitly supplied `DISPLAY`, including an empty value to opt
out. Probe only the implicit `novnc:0.0` default, retaining automatic
headless fallback when no display was configured. Current dev already
includes `x11-utils`; this change prevents probe failure from overriding
user configuration.

Document the one-off `persona create` / `persona open` Compose flow.
`persona open` resolves Chromium automatically, avoiding the obsolete
`/data/lib/env/bin/chromium` and architecture/version-specific
Playwright paths.

Validation:
- Reproduced persona creation followed by the exact Chromium X11 failure
on the published `0.9.35rc381` image with real, healthy `theasp/novnc`.
- Mounted only the patched entrypoint into that affected image: explicit
DISPLAY survived despite the missing `xdpyinfo`, and the same Chromium
command opened a window visible through noVNC.
- Built the complete branch Docker image using the Dockerfile's default
abx-dl base. Real `docker compose run --rm archivebox persona create
personal` and `persona open personal` succeeded; viewed and controlled
Chromium through noVNC, closed it normally, and reopened in another
one-off container. A bookmark created in the UI persisted across both
runs.
- Added `IMAGE=<built-image> ./bin/validate_docker_display.sh`: six real
Compose checks cover working implicit/explicit displays, unavailable
explicit display, explicit empty opt-out, and implicit/explicit behavior
with noVNC stopped. Passed on the rebuilt image; failed on the unpatched
image at the explicit unavailable-display assertion.
- `bash -n`, `git diff --check`, and pre-commit hooks passed. No mocks
or fake binaries.

Fixes #1872.


<!-- This is an auto-generated description by cubic. -->
---
## Summary by cubic
Fixes the Docker entrypoint silently dropping an explicit `DISPLAY` when
`xdpyinfo` fails, so Chromium can connect to a healthy noVNC server in
images like `0.9.35rc381` where `xdpyinfo` is missing. The entrypoint
previously unset the display on probe failure, causing Chromium to exit
with "Missing X server or $DISPLAY"; now only the implicit `novnc:0.0`
default is probed, an explicit value (including empty to opt out) is
preserved, and headless fallback still applies when no display was
configured. Fixes #1872.

**Validation**
- Adds `bin/validate_docker_display.sh` with six real Compose checks
covering implicit/explicit displays, unavailable display, empty opt-out,
and noVNC stopped behavior.
- Documents the one-off `persona create` / `persona open` Compose flow
and that `persona open` resolves the Chromium binary automatically.

<sup>Written for commit 953f29b65e.
Summary will update on new commits.</sup>

<a
href="https://cubic.dev/pr/ArchiveBox/ArchiveBox/pull/1882?utm_source=github"
target="_blank" rel="noopener noreferrer"
data-no-image-dialog="true"><picture><source
media="(prefers-color-scheme: dark)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"><source
media="(prefers-color-scheme: light)"
srcset="https://www.cubic.dev/buttons/review-in-cubic-light.svg"><img
alt="Review in cubic"
src="https://www.cubic.dev/buttons/review-in-cubic-dark.svg"></picture></a>

<!-- End of auto-generated description by cubic. -->

.github/workflows/deploy-publicsite.yml

@@ -1,79 +1,50 @@
 name: Deploy Publicsite to GitHub Pages
-
 on:
   push:
-    branches:
-      - dev
-    paths:
-      - publicsite/**
-      - archivebox/**
-      - bin/collect_ui_screenshots.*
-      - bin/generate_ui_screenshot_gallery.py
-      - bin/take_screenshot.js
-      - docs/Screenshots.md
-      - pyproject.toml
-      - uv.lock
-      - .github/workflows/deploy-publicsite.yml
+    branches: [dev]
+  pull_request:
   workflow_dispatch:
-
+  workflow_run:
+    workflows: ['Capture publicsite screenshots']
+    types: [completed]
+    branches: [dev]
 permissions:
   contents: read
-  pages: write
-  id-token: write
-
+  actions: read
 concurrency:
-  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
-  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
-
+  # Failed captures must not displace a pending push deployment.
+  group: fast-pages-${{ github.event_name == 'workflow_run' && github.event.workflow_run.conclusion != 'success' && github.run_id || github.ref }}
+  cancel-in-progress: false
 jobs:
+  build:
+    if: github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          ref: ${{ github.event_name == 'pull_request' && github.ref || 'dev' }}
+      - uses: astral-sh/setup-uv@v7
+      - name: Restore latest complete screenshots
+        run: uv run --no-project python bin/restore_site_screenshots.py archivebox publicsite/screenshots
+        env:
+          GH_TOKEN: ${{ github.token }}
+      - name: Render current gallery layout without recapturing images
+        run: uv run --no-project python bin/generate_ui_screenshot_gallery.py refresh publicsite/screenshots/index.html
+      - uses: actions/upload-pages-artifact@v3
+        with:
+          path: publicsite
   deploy:
+    if: github.event_name != 'pull_request'
+    needs: build
+    runs-on: ubuntu-latest
+    permissions:
+      pages: write
+      id-token: write
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-24.04
     steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Setup Python and uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          enable-cache: false
-
-      - name: Install ArchiveBox and screenshot dependencies
-        run: uv sync --no-cache --frozen
-
-      - name: Build UI screenshot gallery
-        run: bin/collect_ui_screenshots.sh
-        env:
-          BASE_URL: http://archivebox.localhost:8000
-
-      - name: Verify gallery was generated from this exact revision
-        run: |
-          uv run --no-cache --no-sync python - <<'PY'
-          import json
-          import os
-          import tomllib
-          from pathlib import Path
-
-          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
-          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
-          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
-          assert provenance["version"] == version, provenance
-          assert provenance["capture_count"] >= 3 * 35, provenance
-          assert len(provenance["files"]) == provenance["capture_count"], provenance
-          PY
-
-      - name: Setup Pages
-        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
-        with:
-          path: ./publicsite
-
-      - name: Deploy to GitHub Pages
+      - uses: actions/configure-pages@v5
+      - uses: actions/deploy-pages@v4
         id: deployment
-        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

.github/workflows/docker.yml

@@ -246,6 +246,23 @@ jobs:
         if: inputs.push_digests
         run: echo ${{ steps.docker_build.outputs.digest }}
 
+      - name: Verify OpenCode install with a read-only browser profile
+        shell: bash
+        env:
+          IMAGE: ${{ inputs.push_digests && format('{0}@{1}', env.DOCKERHUB_IMAGE, steps.docker_build.outputs.digest) || matrix.local_tag }}
+        run: |
+          set -Eeuo pipefail
+          PROFILE_DIR="$(mktemp -d)"
+          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' \
+            --volume "$PROFILE_DIR:/home/archivebox/.config/chromium:ro" \
+            "$IMAGE" bash -c '
+              set -Eeuo pipefail
+              test -w "$XDG_CONFIG_HOME"
+              test ! -w "$XDG_CONFIG_HOME/chromium"
+              archivebox install opencode
+              "$ABXPKG_LIB_DIR/pnpm/packages/opencode/node_modules/.bin/opencode" --version
+            '
+
       - name: Validate exact built image version and commit
         if: ${{ !inputs.push_digests }}
         shell: bash
@@ -357,7 +374,7 @@ jobs:
           "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
           "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+            "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
             "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
 

.github/workflows/screenshots.yml

@@ -0,0 +1,70 @@
+name: Capture publicsite screenshots
+
+on:
+  push:
+    branches:
+      - dev
+    paths:
+      - publicsite/**
+      - archivebox/**
+      - bin/collect_ui_screenshots.*
+      - bin/generate_ui_screenshot_gallery.py
+      - bin/take_screenshot.js
+      - docs/Screenshots.md
+      - pyproject.toml
+      - uv.lock
+      - .github/workflows/screenshots.yml
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.event_name == 'workflow_dispatch' && format('publicsite-captures-manual-{0}', github.run_id) || 'publicsite-captures' }}
+  cancel-in-progress: false
+
+jobs:
+  deploy:
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Checkout
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Setup Python and uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: false
+
+      - name: Install ArchiveBox and screenshot dependencies
+        run: uv sync --no-cache --frozen
+
+      - name: Build UI screenshot gallery
+        run: bin/collect_ui_screenshots.sh
+        env:
+          BASE_URL: http://archivebox.localhost:8000
+
+      - name: Verify gallery was generated from this exact revision
+        run: |
+          uv run --no-cache --no-sync python - <<'PY'
+          import json
+          import os
+          import tomllib
+          from pathlib import Path
+
+          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
+          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
+          assert provenance["version"] == version, provenance
+          assert provenance["capture_count"] >= 3 * 35, provenance
+          assert len(provenance["files"]) == provenance["capture_count"], provenance
+          PY
+
+      - name: Save validated screenshots
+        uses: actions/upload-artifact@v4
+        with:
+          name: site-screenshots
+          path: publicsite/screenshots
+          if-no-files-found: error
+          retention-days: 90

Dockerfile

@@ -10,10 +10,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.264 \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.302 \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.264
+ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.302
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -96,17 +96,15 @@ RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$T
     --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
     <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
-echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
-echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
+export UV_NO_CACHE=true
 rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
-apt-get install -qq -y --no-install-recommends \
-    build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+apt-get install -qq -y --no-install-recommends x11-utils
 # Extend abx-dl's existing venv instead of clearing it. Clearing and later
 # copying a complete replacement over the base duplicates every venv byte in
 # the final overlay history even when most packages are unchanged.
-/usr/bin/uv pip install --no-cache setuptools pip wheel
+# Do not rewrite inherited native libraries merely to strip them again.
+/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print > /tmp/archivebox-inherited-libraries
 
 mkdir -p /tmp/archivebox-uv-project
 /venv/bin/python - <<'PY'
@@ -155,25 +153,31 @@ PY
     --inexact \
     --no-install-project \
     --no-install-workspace \
+    --no-install-package abxbus \
+    --no-install-package abxpkg \
+    --no-install-package abx-plugins \
+    --no-install-package abx-dl \
     --no-sources
 builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
 ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
 /usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
 while IFS= read -r -d '' native_library; do
+    if grep -Fxq "$native_library" /tmp/archivebox-inherited-libraries; then
+        continue
+    fi
     magic=''
     if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
         "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
     fi
 done < /tmp/archivebox-native-libraries
-rm -f /tmp/archivebox-native-libraries
+rm -f /tmp/archivebox-native-libraries /tmp/archivebox-inherited-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils
 abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
 rm -rf "$builder_abxpkg_lib_dir"
 rm -rf /venv/lib/python3.*/site-packages/pip* \
     /venv/lib/python3.*/site-packages/wheel* \
     /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
 rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
 EOF
 
@@ -202,7 +206,7 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
-# The builder installs and purges compilers in one layer, so its final
+# The builder installs and purges stripping tools in one layer, so its final
 # filesystem is already runtime-clean. Preserve that ancestry: starting again
 # from archivebox-runtime-base and COPYing /venv would bake a second full venv
 # over the inherited one instead of recording only ArchiveBox's package delta.
@@ -225,7 +229,8 @@ COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 # The builder invokes abxpkg as root for temporary ELF stripping, which can
 # rewrite derived state ownership. Restore UID 911 before resolving binaries
 # as the runtime user; the old copy-based final stage hid this dependency.
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
+RUN --mount=type=cache,target=/opt/archivebox/lib/cache,sharing=locked,mode=1777,id=archivebox-runtime-config-$TARGETARCH \
+    echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
     && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
     && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
     && ln -sf /venv/bin/daphne /usr/local/bin/daphne \
@@ -239,8 +244,8 @@ RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..."
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
     && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
-    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
+    && find "$ABXPKG_LIB_DIR" \( ! -user "$DEFAULT_ARCHIVEBOX_UID" -o ! -group "$DEFAULT_ARCHIVEBOX_GID" \) -exec chown -h "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" {} + \
+    && UV_NO_CACHE=true setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
     && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt

README.md

@@ -151,6 +151,17 @@ curl -fsSL 'https://get.archivebox.io' | bash
 
 #### ✳️  Easy Setup
 
+<details>
+<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="28px" align="top"/> <code>ArchiveBox Server.app</code></b> (macOS)</summary>
+<br/>
+<ol>
+<li><a href="https://github.com/ArchiveBox/ios-archivebox/releases">Download <code>ArchiveBox Server.app</code></a>, move it to Applications, and open it.</li>
+<li>Set up an admin account and generate an API Key.</li>
+<li>Use the Web UI or <a href="https://archivebox.github.io/ios-archivebox/"><code>ArchiveBox.app</code></a>, and optionally connect REST API clients, the <a href="https://archivebox.github.io/ios-archivebox/">mobile app</a>, the <a href="https://archivebox.github.io/archivebox-browser-extension/">browser extension</a>, and other integrations to your server using your API Key.</li>
+</ol>
+<br/>
+</details>
+
 <details>
 <summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b>  (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
 <br/>
@@ -426,7 +437,7 @@ For more discussion on third-party hosting options see here: <a href="https://gi
 #### ➡️&nbsp; Next Steps
 
 - Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
-- (Optional) Create a persona and import browser cookies to archive logged-in sites: `archivebox persona create --import=chrome personal`
+- (Optional) [Import browser cookies and settings into a persona](docs/Chromium-Install.md#import-an-existing-browser-profile).
 - Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
 - Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
 - Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
@@ -454,11 +465,6 @@ docker compose run --rm archivebox help
 # equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
 docker run -it -v $PWD:/data archivebox/archivebox:dev help
 
-# optional: import your browser cookies into a persona for logged-in archiving
-archivebox persona create --import=chrome personal
-# supported: chrome/chromium/brave/edge (Chromium-based only)
-# use --profile to target a specific profile (e.g. Default, Profile 1)
-# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
 ```
 
 #### ArchiveBox Subcommands
@@ -566,6 +572,20 @@ docker compose run --rm archivebox config --set PERMISSIONS=public
 </code></pre>
 </details>
 
+<br/>
+
+<details>
+<summary><b><img src="https://archivebox.github.io/ios-archivebox/docs/icons/apple.svg" alt="Apple" height="22px" align="top"/> <img src="https://cdn.jsdelivr.net/gh/devicons/devicon@v2.17.0/icons/windows11/windows11-original.svg" alt="Windows" height="22px" align="top"/> 🐧&nbsp; Desktop / Mobile / Extension Usage</b></summary>
+<br/>
+
+- 📱 [<code>ArchiveBox</code> iOS Mobile App](https://archivebox.github.io/ios-archivebox/)
+- 🖥️ [<code>ArchiveBox.app</code> macOS Desktop App](https://archivebox.github.io/ios-archivebox/)
+- 🧩 [ArchiveBox Browser Extension](https://archivebox.github.io/archivebox-browser-extension/)
+- 💻 [Linux/Windows Desktop App (Requires Docker Desktop)](https://github.com/ArchiveBox/electron-archivebox)
+
+<br/>
+</details>
+
 <br/>
 <br/>
 
@@ -619,7 +639,10 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>  
-  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
+  <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox/Safari browsers.</i>
+
+- <img src="https://archivebox.github.io/ios-archivebox/App/AppIcon.icon/Assets/ArchiveBox.png" alt="ArchiveBox app icon" height="28px"/> <b>From the official <a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox macOS Desktop App or iOS Mobile App</a></b><br/>
+  <i>Save directly from the Share menu or Siri/Shortcuts to your server.</i>
 
 - <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files  
   <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
@@ -932,7 +955,7 @@ archivebox manage createsuperuser
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
 </ul>
 
@@ -986,7 +1009,7 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
 
 <ul>
 <li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
-<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
+<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">importing a dedicated browser profile into a persona</a></li>
 <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
 <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
 </ul>
@@ -1041,8 +1064,8 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 </li>
 <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
 </li>
-<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
-<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
+<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
+<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
 </li>
 </ul>
 

archivebox/api/v1_auth.py

@@ -4,7 +4,7 @@ from django.http import HttpRequest
 
 from ninja import Router, Schema
 
-from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
+from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token, HeaderTokenAuth, BearerTokenAuth
 
 
 router = Router(tags=["Authentication"], auth=None)
@@ -63,3 +63,44 @@ def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
         return {"success": True, "user_id": str(user.pk)}
 
     return {"success": False, "user_id": None}
+
+
+@router.post(
+    "/browser_session",
+    auth=[HeaderTokenAuth(), BearerTokenAuth()],
+    summary="Exchange an administrator API key for a normal browser session",
+)
+def browser_session(request: HttpRequest):
+    from django.conf import settings
+    from django.contrib.auth import login
+    from django.http import JsonResponse
+    from django.utils import timezone
+    from ninja.errors import HttpError
+    from archivebox.core.routes_util import get_admin_base_url
+
+    user = request.auth
+    if not user.is_active or not user.is_superuser:
+        raise HttpError(403, "An active administrator is required")
+    login(request, user, backend="django.contrib.auth.backends.ModelBackend")
+    # A browser session must not outlive the key used to establish it.
+    token = request._api_token
+    lifetime = settings.SESSION_COOKIE_AGE
+    if token.expires:
+        lifetime = min(lifetime, max(0, int((token.expires - timezone.now()).total_seconds())))
+    request.session.set_expiry(lifetime)
+    request.session.save()
+    response = JsonResponse(
+        {
+            "admin_url": get_admin_base_url(request=request).rstrip("/") + "/admin/",
+            "cookie": {
+                "name": settings.SESSION_COOKIE_NAME,
+                "value": request.session.session_key,
+                "expires": request.session.get_expiry_date().timestamp(),
+                "secure": settings.SESSION_COOKIE_SECURE,
+            },
+        },
+    )
+    # The native client installs a host-only HttpOnly cookie on the admin origin;
+    # API and admin hosts can differ, so a Set-Cookie on the API host is insufficient.
+    response["Cache-Control"] = "no-store"
+    return response

archivebox/cli/archivebox_add.py

@@ -388,10 +388,6 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
-        if not kwargs.get("index_only"):
-            from archivebox.workers.supervisord_util import require_crawl_memory
-
-            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.

archivebox/cli/archivebox_persona.py

@@ -27,11 +27,10 @@ Examples:
 __package__ = "archivebox.cli"
 __command__ = "archivebox persona"
 
-import os
 import sys
 import shutil
-import platform
 from pathlib import Path
+from dataclasses import replace
 from collections.abc import Iterable
 
 import rich_click as click
@@ -41,116 +40,6 @@ from archivebox.cli.cli_util import apply_filters
 from archivebox.personas import importers as persona_importers
 
 
-# =============================================================================
-# Browser Profile Locations
-# =============================================================================
-
-
-def get_chrome_user_data_dir() -> Path | None:
-    """Get the default Chrome user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":  # macOS
-        candidates = [
-            home / "Library" / "Application Support" / "Google" / "Chrome",
-            home / "Library" / "Application Support" / "Chromium",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "google-chrome",
-            home / ".config" / "chromium",
-            home / ".config" / "chrome",
-            home / "snap" / "chromium" / "common" / "chromium",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Google" / "Chrome" / "User Data",
-            local_app_data / "Chromium" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_brave_user_data_dir() -> Path | None:
-    """Get the default Brave user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "BraveSoftware" / "Brave-Browser" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_edge_user_data_dir() -> Path | None:
-    """Get the default Edge user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "Microsoft Edge",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "microsoft-edge",
-            home / ".config" / "microsoft-edge-beta",
-            home / ".config" / "microsoft-edge-dev",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Microsoft" / "Edge" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-BROWSER_PROFILE_FINDERS = {
-    "chrome": get_chrome_user_data_dir,
-    "chromium": get_chrome_user_data_dir,  # Same locations
-    "brave": get_brave_user_data_dir,
-    "edge": get_edge_user_data_dir,
-}
-
-CHROMIUM_BROWSERS = {"chrome", "chromium", "brave", "edge"}
-
-
-# =============================================================================
-# Cookie Extraction via CDP
-# =============================================================================
-
 # =============================================================================
 # Validation Helpers
 # =============================================================================
@@ -218,6 +107,9 @@ def create_personas(
     names: Iterable[str],
     import_from: str | None = None,
     profile: str | None = None,
+    source: str | None = None,
+    browser_binary: str | None = None,
+    permissions: str | None = None,
 ) -> int:
     """
     Create Personas from names.
@@ -239,25 +131,26 @@ def create_personas(
         rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
         return 1
 
-    # Validate import source if specified
-    source_profile_dir = None
+    import_source = None
+    if source and not import_from:
+        rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
+        return 1
     if import_from:
-        import_from = import_from.lower()
-        if import_from not in BROWSER_PROFILE_FINDERS:
-            rprint(f"[red]Unknown browser: {import_from}[/red]", file=sys.stderr)
-            rprint(f"[dim]Supported browsers: {', '.join(BROWSER_PROFILE_FINDERS.keys())}[/dim]", file=sys.stderr)
+        try:
+            if source:
+                import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
+                if import_source.browser != "persona":
+                    import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
+            elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
+                import_source = persona_importers.resolve_custom_import_source(import_from)
+            else:
+                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+            if browser_binary:
+                import_source = replace(import_source, browser_binary=browser_binary)
+        except ValueError as err:
+            rprint(f"[red]{err}[/red]", file=sys.stderr)
             return 1
 
-        source_profile_dir = BROWSER_PROFILE_FINDERS[import_from]()
-        if not source_profile_dir:
-            rprint(f"[red]Could not find {import_from} profile directory[/red]", file=sys.stderr)
-            return 1
-
-        rprint(f"[dim]Found {import_from} profile: {source_profile_dir}[/dim]", file=sys.stderr)
-
-        if profile is None and (source_profile_dir / "Default").exists():
-            profile = "Default"
-
     created_count = 0
     for name in name_list:
         name = name.strip()
@@ -270,21 +163,24 @@ def create_personas(
             rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
             continue
 
-        persona, created = Persona.objects.get_or_create(name=name)
+        persona, created = Persona.objects.get_or_create(
+            name=name,
+            defaults={"config": {"PERMISSIONS": permissions}} if permissions else {},
+        )
 
         if created:
             persona.ensure_dirs()
             created_count += 1
-            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
+
         else:
             rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
 
         cookies_file = Path(persona.path) / "cookies.txt"
 
         # Import browser profile if requested
-        if import_from in CHROMIUM_BROWSERS and source_profile_dir is not None:
+        if import_source is not None:
             try:
-                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+                rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
                 import_result = persona_importers.import_persona_from_source(
                     persona,
                     import_source,
@@ -292,20 +188,25 @@ def create_personas(
                     import_cookies=True,
                     capture_storage=False,
                 )
-            except Exception as e:
+            except (Exception, KeyboardInterrupt) as e:
+                if created:
+                    shutil.rmtree(persona.path, ignore_errors=True)
+                    persona.delete()
                 rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
                 return 1
 
             if import_result.profile_copied:
                 rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
             if import_result.cookies_imported:
-                rprint(f"[green]Extracted cookies to {cookies_file}[/green]", file=sys.stderr)
+                rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
             elif not import_result.profile_copied:
                 rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
 
             for warning in import_result.warnings:
                 rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
 
+        if created:
+            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
         if not is_tty:
             write_record(
                 {
@@ -544,11 +445,42 @@ def main():
 
 @main.command("create")
 @click.argument("names", nargs=-1)
-@click.option("--import", "import_from", help="Import profile from browser (chrome, chromium, brave, edge)")
+@click.option(
+    "--import",
+    "import_from",
+    help="Import from chrome, chrome-beta, chromium, brave, edge, a persona directory, or a live CDP URL",
+)
+@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
+@click.option(
+    "--browser-binary",
+    type=click.Path(exists=True, dir_okay=False),
+    help="Source Chromium browser executable; required for other Chromium-based browsers",
+)
 @click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
-def create_cmd(names: tuple, import_from: str | None, profile: str | None):
+@click.option(
+    "--permissions",
+    type=click.Choice(["public", "unlisted", "private"]),
+    help="Visibility for a newly created persona (existing personas are unchanged)",
+)
+def create_cmd(
+    names: tuple,
+    import_from: str | None,
+    profile: str | None,
+    source: str | None,
+    browser_binary: str | None,
+    permissions: str | None,
+):
     """Create Personas, optionally importing from a browser profile."""
-    sys.exit(create_personas(names, import_from=import_from, profile=profile))
+    sys.exit(
+        create_personas(
+            names,
+            import_from=import_from,
+            profile=profile,
+            source=source,
+            browser_binary=browser_binary,
+            permissions=permissions,
+        ),
+    )
 
 
 @main.command("list")
@@ -560,6 +492,84 @@ def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
     sys.exit(list_personas(name=name, name__icontains=name__icontains, limit=limit))
 
 
+@main.command("open")
+@click.argument("name", default="Default")
+def open_cmd(name: str):
+    """Open a persona's browser, including its imported logins."""
+    import json
+    import os
+    import subprocess
+
+    import psutil
+
+    from abx_plugins import get_plugins_dir
+    from archivebox.config.common import get_config
+    from archivebox.personas.models import Persona
+
+    valid, reason = validate_persona_name(name)
+    if not valid:
+        raise click.ClickException(reason)
+    persona = Persona.get_or_create_named(name)
+    persona.ensure_dirs()
+    config = get_config(persona=persona)
+    if os.environ.get("DISPLAY"):
+        from archivebox.config import CONSTANTS
+
+        for process in psutil.process_iter():
+            try:
+                args = process.cmdline()
+                profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
+                if (
+                    not profile
+                    or any(arg.startswith(("--type=", "--headless")) for arg in args)
+                    or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
+                    or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
+                ):
+                    continue
+                rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
+                process.wait()
+            except (psutil.NoSuchProcess, psutil.AccessDenied):
+                continue
+    plugins_dir = Path(get_plugins_dir()).resolve()
+    env = os.environ.copy()
+    dependencies = subprocess.run(
+        [
+            str(Path(sys.executable).with_name("abxpkg")),
+            "env",
+            "--install",
+            "--json",
+            f"--lib={config.ABXPKG_LIB_DIR}",
+            f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
+        ],
+        capture_output=True,
+        text=True,
+        env=env,
+    )
+    if dependencies.returncode:
+        raise click.ClickException(dependencies.stderr.strip())
+    env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
+    env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
+    env["ACTIVE_PERSONA"] = persona.name
+    env["PERSONAS_DIR"] = str(persona.path.parent)
+    env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
+    payload = config.model_dump(mode="json")
+    payload.update(
+        ACTIVE_PERSONA=persona.name,
+        CHROME_HEADLESS=False,
+        CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
+        AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
+        COOKIES_FILE=str(persona.COOKIES_FILE or ""),
+    )
+    script = Path(__file__).parent.parent / "personas" / "open_browser.js"
+    result = subprocess.run(
+        [env.get("NODE_BINARY") or "node", str(script)],
+        input=json.dumps(payload),
+        text=True,
+        env=env,
+    )
+    sys.exit(result.returncode)
+
+
 @main.command("update")
 @click.option("--name", "-n", help="Set new name")
 def update_cmd(name: str | None):

archivebox/cli/archivebox_server.py

@@ -363,12 +363,10 @@ def server(
     command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
 
     def still_owns_runtime_stack() -> bool:
-        from django.db import connections
-
-        try:
-            return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
-        finally:
-            connections.close_all()
+        # This is polled at the log-tail cadence. Keep the autocommit
+        # connection open so SQLite can reuse its page cache between polls;
+        # reconnecting here repeatedly rereads the process table while idle.
+        return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
 
     shutdown_state = None
     try:
@@ -416,8 +414,13 @@ def server(
     except KeyboardInterrupt:
         pass
     finally:
-        if not shutdown_state or not shutdown_state.signal_name:
-            command.mark_exited()
+        from django.db import connections
+
+        try:
+            if not shutdown_state or not shutdown_state.signal_name:
+                command.mark_exited()
+        finally:
+            connections.close_all()
     print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
 
 

archivebox/config/collection.py

@@ -17,7 +17,7 @@ CONFIG_FILE_HEADER = (
     "# You can add options here manually in INI format, or automatically by running:\n"
     "#    archivebox config --set KEY=VALUE\n"
     "#\n"
-    "# This file is kept in sync 1:1 with Machine.config in the index DB —\n"
+    "# This file is kept in sync 1:1 with the current Machine.config in the DB —\n"
     "# editing either side propagates to the other. ``archivebox init`` reads\n"
     "# this file on startup; the admin Machine.config editor writes both.\n"
     "#\n"

archivebox/config/common.py

@@ -18,7 +18,7 @@ from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.base.utils import build_config_model
 from django.db import DatabaseError
-from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
+from pydantic import BaseModel, Field, PrivateAttr, TypeAdapter, create_model, field_validator, model_validator
 from pydantic_settings import BaseSettings, SettingsConfigDict
 from rich.console import Console
 
@@ -461,7 +461,8 @@ class ArchivingConfig(BaseConfigSet):
     USER_AGENT: str = Field(
         default=f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)",
     )
-    COOKIES_FILE: Path | None = Field(default=None)
+    COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
     URL_DENYLIST: str = Field(default=r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$", alias="URL_BLACKLIST")
     URL_ALLOWLIST: str | None = Field(default=None, alias="URL_WHITELIST")
@@ -1253,7 +1254,7 @@ def get_config(
             config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
     for key in explicit_plugin_enabled_keys:
         if key in config_data:
-            setattr(config, key, config_data[key])
+            setattr(config, key, TypeAdapter(bool).validate_python(config_data[key]))
     if config.PLUGINS:
         config._derive_plugin_enabled_config(respect_current_enabled=True)
     if redact_sensitive:

archivebox/core/asgi.py

@@ -13,5 +13,12 @@ from archivebox.config.django import setup_django
 
 setup_django(check_db=True)
 
-# Standard Django ASGI application (no websockets/channels needed)
-application = get_asgi_application()
+django_application = get_asgi_application()
+
+
+async def application(scope, receive, send):
+    if scope["type"] == "websocket":
+        from archivebox.opencode.views import websocket_view
+
+        return await websocket_view(scope, receive, send)
+    return await django_application(scope, receive, send)

archivebox/core/middleware.py

@@ -89,13 +89,7 @@ def AdminCookieIsolationMiddleware(get_response):
         response = get_response(request)
 
         if request.path == "/admin" or request.path.startswith("/admin/"):
-            from archivebox.opencode.views import _PROXY_PREFIX
-
-            is_opencode_proxy = request.path == _PROXY_PREFIX or request.path.startswith(f"{_PROXY_PREFIX}/")
-            if is_opencode_proxy:
-                response.headers["X-Frame-Options"] = "SAMEORIGIN"
-                response.headers["Content-Security-Policy"] = "frame-ancestors 'self'"
-            else:
+            if not getattr(response, "xframe_options_exempt", False):
                 response.headers["X-Frame-Options"] = "DENY"
                 response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
 

archivebox/core/settings.py

@@ -76,7 +76,6 @@ INSTALLED_APPS = [
     "archivebox.crawls",  # handles Crawl and CrawlSchedule models and management (depends on core)
     "archivebox.progressmonitor",  # live progress endpoint and admin monitor template
     "archivebox.api",  # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.
-    "archivebox.opencode",
     # 3rd-party apps from PyPI that need to be loaded last
     "admin_data_views",  # handles rendering some convenient automatic read-only views of data in Django admin
     "django_extensions",  # provides Django Debug Toolbar (and other non-debug helpers)

archivebox/core/takeover_util.py

@@ -126,12 +126,25 @@ def ensure_daemon_stack(*, reason: str = ""):
 
     supervisor = get_existing_supervisord_process() or get_or_create_supervisord_process(daemonize=False)
     worker = get_worker(supervisor, sonic_worker["name"])
-    if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
-        return worker
+    if not (isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING")):
+        if reason:
+            rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
+        worker = start_worker(supervisor, sonic_worker)
 
-    if reason:
-        rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
-    return start_worker(supervisor, sonic_worker)
+    # Supervisord marks this worker RUNNING with startsecs=0, before Sonic has
+    # opened its search socket. A query launched at that point fails even though
+    # the daemon is about to become ready.
+    deadline = time.monotonic() + 30.0
+    while time.monotonic() < deadline:
+        if is_port_listening(sonic_event.host, sonic_event.port):
+            if time.monotonic() < deadline:
+                return worker
+            break
+        worker = get_worker(supervisor, sonic_worker["name"])
+        if not isinstance(worker, dict) or worker.get("statename") not in ("STARTING", "RUNNING"):
+            raise RuntimeError(f"Sonic search backend worker failed to start: {worker}")
+        time.sleep(min(0.1, max(0.0, deadline - time.monotonic())))
+    raise RuntimeError(f"Sonic search backend is not listening at {sonic_event.url}")
 
 
 def live_runner_processes(*, data_dir: str | Path):

archivebox/core/templatetags/core_tags.py

@@ -1,8 +1,9 @@
+import logging
 import os
 from html import unescape
 from pathlib import Path
 from typing import Any
-from urllib.parse import quote
+from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
 from django import template
@@ -28,6 +29,20 @@ from archivebox.plugins.discovery import (
 
 register = template.Library()
 
+
+@register.simple_tag(takes_context=True)
+def plugin_ui(context, plugin: str, slot: str):
+    """Optional plugin UI must never prevent the host page from rendering."""
+    try:
+        source = get_plugin_template(plugin, slot, fallback=False)
+        if not source:
+            return ""
+        return template.engines["django"].from_string(source).render(context.flatten())
+    except Exception:
+        logging.getLogger(__name__).exception("Unable to render optional plugin UI: %s/%s", plugin, slot)
+        return ""
+
+
 _TEXT_PREVIEW_EXTS = (".json", ".jsonl", ".txt", ".csv", ".tsv", ".xml", ".yml", ".yaml", ".md", ".log")
 _IMAGE_PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".ico", ".avif")
 _STATIC_URL_SAFE = "/@-._~!$&'()*+,;="
@@ -74,6 +89,21 @@ _BROWSER_VIDEO_FILE_EXTS = {".mp4", ".webm", ".m4v", ".ogv"}
 _BROWSER_AUDIO_FILE_EXTS = {".mp3", ".m4a", ".aac", ".ogg", ".oga", ".opus", ".wav", ".flac"}
 
 
+def _is_web_subdomain_alias(browser_url: str, configured_base_url: str) -> bool:
+    """Hide the informational mismatch banner on the conventional public web alias.
+
+    This is deliberately presentation-only: request routing and the restriction
+    on state-changing requests continue to use the configured BASE_URL.
+    """
+    browser = urlparse(browser_url)
+    configured = urlparse(configured_base_url)
+    return (
+        browser.scheme.lower() == configured.scheme.lower()
+        and browser.hostname == f"web.{configured.hostname}"
+        and browser.port == configured.port
+    )
+
+
 def _normalize_output_files(output_files: Any) -> dict[str, dict[str, Any]]:
     from abx_dl.output_files import OutputManifest
 
@@ -430,9 +460,9 @@ def system_warnings_banner(context):
     """Render the top-of-page warning banner for one of the conditions below,
     in priority order (highest first):
 
-    1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
-       issue: until it's pinned, generated URLs can echo any Host the client
-       sends, and admin/web/api routing has no canonical anchor.
+    1. ``mode="unconfigured"``— ``BASE_URL`` is empty and the operator has
+       not selected safe single-domain routing. That mode intentionally uses
+       each device's connection address, including LAN and Tailscale IPs.
     2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
        an origin that differs from the configured canonical ``BASE_URL``.
     3. ``mode="unsafe"``      — ``SERVER_SECURITY_MODE`` explicitly enables
@@ -459,10 +489,10 @@ def system_warnings_banner(context):
 
         config = get_config(resolve_plugins=False)
 
-    if not config.BASE_URL:
+    if not config.BASE_URL and config.SERVER_SECURITY_MODE != "safe-onedomain-nojsreplay":
         return get_setup_wizard_context(context.get("request"), config)
     mismatch = get_base_url_mismatch_context(context.get("request"), config)
-    if mismatch:
+    if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
         return mismatch
     if config.IS_LOWER_SECURITY_MODE:
         return {"mode": "unsafe"}

archivebox/core/views.py

@@ -765,6 +765,11 @@ def _plugin_full_preview_response(
     response.headers["X-Content-Type-Options"] = "nosniff"
     response.headers["X-ArchiveBox-Security-Mode"] = request.archivebox_config.SERVER_SECURITY_MODE
     response.headers["Referrer-Policy"] = "no-referrer"
+    # Trusted viewers live on the snapshot origin, but the collection UI lives
+    # on web.* (and can be embedded by admin.*). Permit only those configured
+    # origins, not arbitrary sites or other snapshots. This does not grant the
+    # archived document access to the parent frame or its admin session cookies.
+    viewer_origins = " ".join(dict.fromkeys((build_web_url(request=request), build_admin_url(request=request))))
     response.headers["Content-Security-Policy"] = (
         "default-src 'self' data: blob:; "
         "script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; "
@@ -778,7 +783,7 @@ def _plugin_full_preview_response(
         "object-src 'none'; "
         "base-uri 'none'; "
         "form-action 'none'; "
-        "frame-ancestors 'self';"
+        f"frame-ancestors 'self' {viewer_origins};"
     )
     return response
 

archivebox/crawls/admin.py

@@ -1230,7 +1230,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
         (
             "Configuration",
             {
-                "fields": ("schedule", "template"),
+                "fields": ("is_enabled", "schedule", "template"),
                 "classes": ("card",),
             },
         ),
@@ -1276,7 +1276,21 @@ class CrawlScheduleAdmin(BaseModelAdmin):
 
     def change_view(self, request, object_id, form_url="", extra_context=None):
         self.request = request
-        return super().change_view(request, object_id, form_url, extra_context)
+        from django.db import connections, router
+
+        connection = connections[router.db_for_write(self.model)]
+        if request.method != "POST" or connection.vendor != "sqlite":
+            return super().change_view(request, object_id, form_url, extra_context)
+        # Reserve the writer before Django's atomic admin form reads. A
+        # deferred read transaction cannot safely upgrade while the runner
+        # writes, even when SQLite's existing busy timeout has time left.
+        connection.ensure_connection()
+        previous_mode = connection.transaction_mode
+        connection.transaction_mode = "IMMEDIATE"
+        try:
+            return super().change_view(request, object_id, form_url, extra_context)
+        finally:
+            connection.transaction_mode = previous_mode
 
     def add_view(self, request, form_url="", extra_context=None):
         return redirect("/add/#schedule")

archivebox/machine/models.py

@@ -227,6 +227,9 @@ class Machine(ModelWithHealthStats):
             _CURRENT_MACHINE = None
         if _CURRENT_MACHINE:
             if timezone.now() < _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+                # Reconciliation can save() and clear the shared cache; keep
+                # this call's instance so we never return None after that save.
+                machine = _CURRENT_MACHINE
                 # One-time-per-process reconciliation between ArchiveBox.conf
                 # and Machine.config. Fast-path: bool check + early-return when
                 # the sync has already run, so the cached-machine return path
@@ -234,16 +237,21 @@ class Machine(ModelWithHealthStats):
                 try:
                     from archivebox.config.collection import sync_machine_and_file
 
-                    sync_machine_and_file(_CURRENT_MACHINE)
+                    sync_machine_and_file(machine)
                 except Exception:
                     pass
-                return _CURRENT_MACHINE
+                return machine
             else:
                 _CURRENT_MACHINE = None
 
         host_guid = get_host_guid()
+        # save() intentionally invalidates _CURRENT_MACHINE so config edits
+        # made through other instances become visible in this process. Host
+        # refresh (after seven days), sanitation, and reconciliation can all
+        # save here, so the working instance must not live only in that cache.
+        # Otherwise a routine refresh leaves startup dereferencing None.
         try:
-            _CURRENT_MACHINE = cls.objects.get(guid=host_guid)
+            machine = cls.objects.get(guid=host_guid)
         except cls.DoesNotExist:
             config = {}
             try:
@@ -253,7 +261,7 @@ class Machine(ModelWithHealthStats):
                 config = _coerce_from_str_dict(file_config)
             except Exception:
                 config = {}
-            _CURRENT_MACHINE = cls.objects.create(
+            machine = cls.objects.create(
                 guid=host_guid,
                 hostname=socket.gethostname(),
                 config=config,
@@ -262,15 +270,15 @@ class Machine(ModelWithHealthStats):
                 stats=get_host_stats(),
             )
         else:
-            if timezone.now() >= _CURRENT_MACHINE.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
+            if timezone.now() >= machine.modified_at + timedelta(seconds=MACHINE_RECHECK_INTERVAL):
                 for key, value in {
                     "hostname": socket.gethostname(),
                     **get_os_info(),
                     **get_vm_info(),
                     "stats": get_host_stats(),
                 }.items():
-                    setattr(_CURRENT_MACHINE, key, value)
-                _CURRENT_MACHINE.save(
+                    setattr(machine, key, value)
+                machine.save(
                     update_fields=[
                         "hostname",
                         "hw_in_docker",
@@ -287,16 +295,19 @@ class Machine(ModelWithHealthStats):
                         "modified_at",
                     ],
                 )
-        machine = cls._sanitize_config(_CURRENT_MACHINE)
+        machine = cls._sanitize_config(machine)
         # Same one-time sync as the cached-return path. Triggers here on the
         # very first ``Machine.current()`` call in a process before any
         # cached return can occur.
         try:
             from archivebox.config.collection import sync_machine_and_file
 
-            sync_machine_and_file(_CURRENT_MACHINE)
+            sync_machine_and_file(machine)
         except Exception:
             pass
+        # Publish only after all save-capable steps; publishing earlier would
+        # let those steps invalidate the cache we are in the middle of filling.
+        _CURRENT_MACHINE = machine
         return machine
 
     @classmethod

archivebox/misc/checks.py

@@ -255,7 +255,7 @@ def check_data_dir_permissions(config=None, **config_kwargs):
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions]https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#permissions[/link]",
         )
         STDERR.print(
-            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#puid--pgid[/link]",
+            "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid]https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid[/link]",
         )
         STDERR.print(
             "    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts]https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#filesystem-doesnt-support-fsync-eg-network-mounts[/link]",

archivebox/misc/hashing.py

@@ -1,260 +0,0 @@
-# Bootable hashing helpers (file content + mtime/size cache, mime detection).
-# MUST NOT import archivebox.config, archivebox.core, or Django — stdlib only.
-
-import hashlib
-import mimetypes
-from functools import lru_cache
-from pathlib import Path
-from collections.abc import Callable
-from datetime import datetime
-
-
-@lru_cache(maxsize=1024)
-def _cached_file_hash(filepath: str, size: int, mtime: float) -> str:
-    """Internal function to calculate file hash with cache key based on path, size and mtime."""
-    sha256_hash = hashlib.sha256()
-
-    with open(filepath, "rb") as f:
-        for chunk in iter(lambda: f.read(4096), b""):
-            sha256_hash.update(chunk)
-
-    return sha256_hash.hexdigest()
-
-
-@lru_cache(maxsize=10)
-def hash_file(file_path: Path, pwd: Path | None = None) -> str:
-    """Calculate SHA256 hash of a file with caching based on path, size and mtime."""
-    pwd = Path(pwd) if pwd else None
-    file_path = Path(file_path)
-    if not file_path.is_absolute():
-        file_path = pwd / file_path if pwd else file_path.absolute()
-
-    abs_path = file_path.resolve()
-    stat_info = abs_path.stat()
-
-    return _cached_file_hash(
-        str(abs_path),
-        stat_info.st_size,
-        stat_info.st_mtime,
-    )
-
-
-@lru_cache(maxsize=10)
-def get_dir_hashes(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict[str, str]:
-    """Calculate SHA256 hashes for all files and directories recursively."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    if not dir_path.is_dir():
-        raise ValueError(f"Not a directory: {dir_path}")
-    if max_depth < -1:
-        raise ValueError(f"max_depth must be >= -1, got {max_depth}")
-
-    # Get all files recursively
-    all_files = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=True,
-        include_dirs=False,
-        filter_func=filter_func,
-    )
-
-    hashes: dict[str, str] = {}
-    hashable_summary = []
-
-    # Calculate hashes for all files
-    for subfile in all_files:
-        subfile_path = dir_path / subfile
-        sha256_hash = hash_file(subfile_path)
-        hashes[subfile] = sha256_hash
-        hashable_summary.append(f"{sha256_hash}  ./{subfile}")
-
-    # Calculate hashes for all directories
-    subdirs = get_dir_entries(
-        dir_path,
-        pwd=pwd,
-        recursive=True,
-        include_files=False,
-        include_dirs=True,
-        include_hidden=False,
-        filter_func=filter_func,
-        max_depth=max_depth,
-    )
-
-    for subdir in subdirs:
-        subdir_path = dir_path / subdir
-        subdir_hashes = get_dir_hashes(
-            subdir_path,
-            filter_func=filter_func,
-            max_depth=0,
-        )
-        hashes[subdir] = subdir_hashes["."]
-
-    # Filter results by max_depth
-    if max_depth >= 0:
-        hashes = {path: value for path, value in hashes.items() if len(Path(path).parts) <= max_depth + 1}
-
-    # Calculate root directory hash
-    hashable_summary.sort()
-    root_sha256 = hashlib.sha256("\n".join(hashable_summary).encode()).hexdigest()
-    hashes["."] = root_sha256
-
-    return hashes
-
-
-@lru_cache(maxsize=128)
-def get_dir_entries(
-    dir_path: Path,
-    pwd: Path | None = None,
-    recursive: bool = True,
-    include_files: bool = True,
-    include_dirs: bool = True,
-    include_hidden: bool = False,
-    filter_func: Callable | None = None,
-    max_depth: int = -1,
-) -> tuple[str, ...]:
-    """Get filtered list of directory entries."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    results = []
-
-    def process_path(path: Path, depth: int):
-        if not include_hidden and path.name.startswith("."):
-            return False
-        if max_depth >= 0 and depth > max_depth:
-            return False
-        if filter_func:
-            info = {
-                "abspath": str(path.absolute()),
-                "relpath": str(path.relative_to(dir_path)),
-            }
-            if not filter_func(info):
-                return False
-        return True
-
-    for path in dir_path.rglob("*") if recursive else dir_path.glob("*"):
-        current_depth = len(path.relative_to(dir_path).parts)
-
-        if path.is_file() and include_files and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-        elif path.is_dir() and include_dirs and process_path(path, current_depth):
-            results.append(str(path.relative_to(dir_path)))
-
-        if not recursive:
-            break
-
-    return tuple(sorted(results))  # Make immutable for caching
-
-
-@lru_cache(maxsize=1024)
-def get_dir_sizes(dir_path: Path, pwd: Path | None = None, **kwargs) -> dict[str, int]:
-    """Calculate sizes for all files and directories recursively."""
-    sizes: dict[str, int] = {}
-    hashes = get_dir_hashes(dir_path, pwd=pwd, **kwargs)
-    dir_path = Path(dir_path)
-
-    for path_key in hashes:
-        full_path = dir_path / path_key
-        if full_path.is_file():
-            sizes[path_key] = full_path.stat().st_size
-        else:
-            total = 0
-            for file_path in full_path.rglob("*"):
-                if file_path.is_file() and not file_path.name.startswith("."):
-                    total += file_path.stat().st_size
-            sizes[path_key + "/"] = total
-
-    return sizes
-
-
-@lru_cache(maxsize=10)
-def get_dir_info(dir_path: Path, pwd: Path | None = None, filter_func: Callable | None = None, max_depth: int = -1) -> dict:
-    """Get detailed information about directory contents including hashes and sizes."""
-    pwd = Path(pwd) if pwd else None
-    dir_path = Path(dir_path)
-    if not dir_path.is_absolute():
-        dir_path = pwd / dir_path if pwd else dir_path.absolute()
-
-    hashes = get_dir_hashes(dir_path, pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-    sizes = get_dir_sizes(str(dir_path), pwd=pwd, filter_func=filter_func, max_depth=max_depth)
-
-    num_total_subpaths = sum(1 for name in hashes if name != ".")
-    details = {}
-
-    for filename, sha256_hash in sorted(hashes.items()):
-        abs_path = (dir_path / filename).resolve()
-        stat_info = abs_path.stat()
-        num_subpaths = sum(1 for p in hashes if p.startswith(filename + "/"))
-        is_dir = abs_path.is_dir()
-        if is_dir:
-            mime_type = "inode/directory"
-            basename = abs_path.name
-            extension = ""
-            num_bytes = sizes[filename + "/"]
-            if filename == ".":
-                num_subpaths = num_total_subpaths
-            else:
-                filename += "/"
-                num_subpaths = num_subpaths
-        else:  # is_file
-            num_subpaths = None
-            mime_type = mimetypes.guess_type(str(abs_path))[0]
-            extension = abs_path.suffix
-            basename = abs_path.name.rsplit(extension, 1)[0]
-            num_bytes = sizes[filename]
-
-        details[filename] = {
-            "basename": basename,
-            "mime_type": mime_type,
-            "extension": extension,
-            "num_subpaths": num_subpaths,
-            "num_bytes": num_bytes,
-            "hash_sha256": sha256_hash,
-            "created_at": datetime.fromtimestamp(stat_info.st_ctime).isoformat(),
-            "modified_at": datetime.fromtimestamp(stat_info.st_mtime).isoformat(),
-        }
-
-        if filter_func and not filter_func(details[filename]):
-            del details[filename]
-
-    return details
-
-
-if __name__ == "__main__":
-    import json
-
-    dir_info = get_dir_info(Path("."), max_depth=6)
-    with open(".hashes.json", "w") as f:
-        json.dump(dir_info, f, indent=4)
-    print("Wrote .hashes.json")
-
-# Example output:
-# {
-#     ".": {
-#         "basename": "misc",
-#         "mime_type": "inode/directory",
-#         "extension": "",
-#         "num_subpaths": 25,
-#         "num_bytes": 214677,
-#         "hash_sha256": "addfacf88b2ff6b564846415fb7b21dcb7e63ee4e911bc0aec255ee354958530",
-#         "created_at": "2024-12-04T00:08:38.537449",
-#         "modified_at": "2024-12-04T00:08:38.537449"
-#     },
-#     "__init__.py": {
-#         "basename": "__init__",
-#         "mime_type": "text/x-python",
-#         "extension": ".py",
-#         "num_subpaths": null,
-#         "num_bytes": 32,
-#         "hash_sha256": "b0e5e7ff17db3b60535cf664282787767c336e3e203a43e21b6326c6fe457551",
-#         "created_at": "2024-10-08T00:51:41.001359",
-#         "modified_at": "2024-10-08T00:51:41.001359"
-#     },
-#     ...
-# }

archivebox/misc/jsonl.py

@@ -32,7 +32,6 @@ import sys
 import json
 from typing import Any, TextIO
 from collections.abc import Iterable, Iterator
-from pathlib import Path
 
 
 # Type constants for JSONL records
@@ -120,19 +119,6 @@ def read_stdin(stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
             yield record
 
 
-def read_file(path: Path) -> Iterator[dict[str, Any]]:
-    """
-    Read JSONL or plain URLs from a file.
-
-    Yields parsed records as dicts.
-    """
-    with open(path) as f:
-        for line in f:
-            record = parse_line(line)
-            if record:
-                yield record
-
-
 def read_args_or_stdin(args: Iterable[str], stream: TextIO | None = None) -> Iterator[dict[str, Any]]:
     """
     Read from CLI arguments if provided, otherwise from stdin.

archivebox/misc/serve_static.py

@@ -910,6 +910,9 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
                 )
 
     content_type, encoding = mimetypes.guess_type(str(fullpath))
+    preserve_plain_text = fullpath.suffix.lower() in {".log", ".sh"}
+    if preserve_plain_text:
+        content_type = "text/plain"
     content_type = content_type or "application/octet-stream"
     # Add charset for text-like types (best guess), but don't override the type.
     is_text_like = content_type.startswith("text/") or content_type in {
@@ -1020,7 +1023,7 @@ def serve_static_with_byterange_support(request, path, document_root=None, show_
 
     # Heuristic fix: some archived HTML outputs are stored with HTML-escaped markup
     # or markdown sources. If so, render sensibly.
-    if content_type.startswith(("text/plain", "text/html")):
+    if not preserve_plain_text and content_type.startswith(("text/plain", "text/html")):
         try:
             max_unescape_size = 10 * 1024 * 1024  # 10MB cap to avoid heavy memory use
             if statobj.st_size <= max_unescape_size:

archivebox/misc/toml_util.py

@@ -2,94 +2,11 @@ from typing import Any, cast
 from collections.abc import Callable
 
 import json
-import ast
-import inspect
 import toml
 import re
-import configparser
 
 from pathlib import Path, PosixPath
 
-from pydantic.json_schema import GenerateJsonSchema
-from pydantic_core import to_jsonable_python
-
-JSONValue = str | bool | int | None | list["JSONValue"]
-
-TOML_HEADER = "# Converted from INI to TOML format: https://toml.io/en/\n\n"
-
-
-def load_ini_value(val: str) -> JSONValue:
-    """Convert lax INI values into strict TOML-compliant (JSON) values"""
-    if val.lower() in ("true", "yes", "1"):
-        return True
-    if val.lower() in ("false", "no", "0"):
-        return False
-    if val.isdigit():
-        return int(val)
-
-    try:
-        return ast.literal_eval(val)
-    except Exception:
-        pass
-
-    try:
-        return json.loads(val)
-    except Exception:
-        pass
-
-    return val
-
-
-def convert(ini_str: str) -> str:
-    """Convert a string of INI config into its TOML equivalent (warning: strips comments)"""
-
-    config = configparser.ConfigParser()
-    setattr(config, "optionxform", str)  # capitalize key names
-    config.read_string(ini_str)
-
-    # Initialize an empty dictionary to store the TOML representation
-    toml_dict = {}
-
-    # Iterate over each section in the INI configuration
-    for section in config.sections():
-        toml_dict[section] = {}
-
-        # Iterate over each key-value pair in the section
-        for key, value in config.items(section):
-            parsed_value = load_ini_value(value)
-
-            # Convert the parsed value to its TOML-compatible JSON representation
-            toml_dict[section.upper()][key.upper()] = json.dumps(parsed_value)
-
-    # Build the TOML string
-    toml_str = TOML_HEADER
-    for section, items in toml_dict.items():
-        toml_str += f"[{section}]\n"
-        for key, value in items.items():
-            toml_str += f"{key} = {value}\n"
-        toml_str += "\n"
-
-    return toml_str.strip()
-
-
-class JSONSchemaWithLambdas(GenerateJsonSchema):
-    """
-    Encode lambda functions in default values properly.
-    Usage:
-    >>> json.dumps(value, encoder=JSONSchemaWithLambdas())
-    """
-
-    def encode_default(self, dft: Any) -> Any:
-        config = self._config
-        if isinstance(dft, Callable):
-            return "{{lambda " + inspect.getsource(dft).split("=lambda ")[-1].strip()[:-1] + "}}"
-        return to_jsonable_python(
-            dft,
-            timedelta_mode=config.ser_json_timedelta,
-            bytes_mode=config.ser_json_bytes,
-            serialize_unknown=True,
-        )
-
 
 def better_toml_dump_str(val: Any) -> str:
     try:

archivebox/opencode/apps.py

@@ -1,6 +0,0 @@
-from django.apps import AppConfig
-
-
-class OpencodeConfig(AppConfig):
-    name = "archivebox.opencode"
-    label = "opencode_plugin"

archivebox/opencode/templates/opencode/agent.html

@@ -1,211 +0,0 @@
-{% extends "admin/base.html" %}
-{% load i18n %}
-
-{% block title %}Agent{% endblock %}
-
-{% block breadcrumbs %}
-    <div class="breadcrumbs">
-        <a href="{% url 'admin:index' %}">{% trans 'Home' %}</a>
-        &rsaquo; Agent
-    </div>
-{% endblock %}
-
-{% block extrastyle %}
-    {{ block.super }}
-    <style>
-        body.opencode-agent #content {
-            padding: 0;
-        }
-        body.opencode-agent #content-main {
-            height: calc(100vh - 150px);
-            min-height: 560px;
-        }
-        .opencode-agent-error {
-            margin: 24px;
-            color: #0f172a;
-            font: 14px -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
-        }
-        .opencode-agent-error code {
-            color: #0369a1;
-        }
-        .opencode-agent-frame {
-            width: 100%;
-            height: 100%;
-            border: 0;
-            display: block;
-        }
-        .opencode-agent-shell {
-            position: relative;
-            height: 100%;
-        }
-        .opencode-agent-welcome {
-            position: absolute;
-            inset: 0;
-            z-index: 20;
-            display: flex;
-            align-items: center;
-            justify-content: center;
-            padding: 24px;
-            background: rgba(0, 0, 0, 0.48);
-            color: #111827;
-        }
-        .opencode-agent-welcome[hidden] {
-            display: none;
-        }
-        .opencode-agent-welcome-panel {
-            width: min(720px, 100%);
-            max-height: min(720px, calc(100vh - 190px));
-            overflow: auto;
-            border-radius: 8px;
-            border: 1px solid #d1d5db;
-            background: Canvas;
-            color: CanvasText;
-            box-shadow: 0 20px 60px rgba(0, 0, 0, 0.32);
-            padding: 24px;
-            font: 14px/1.5 -apple-system, BlinkMacSystemFont, "Segoe UI", sans-serif;
-        }
-        .opencode-agent-welcome-panel h1 {
-            margin: 0 0 10px;
-            font-size: 22px;
-            line-height: 1.25;
-        }
-        .opencode-agent-welcome-panel p {
-            margin: 0 0 14px;
-        }
-        .opencode-agent-welcome-panel ul {
-            margin: 0 0 16px 20px;
-            padding: 0;
-        }
-        .opencode-agent-welcome-panel li {
-            margin: 0 0 8px;
-        }
-        .opencode-agent-welcome-warning {
-            margin: 16px 0;
-            padding: 12px 14px;
-            border: 1px solid #f59e0b;
-            border-radius: 6px;
-            background: color-mix(in srgb, #f59e0b 12%, Canvas);
-        }
-        .opencode-agent-welcome-actions {
-            display: flex;
-            justify-content: flex-end;
-            gap: 8px;
-        }
-        .opencode-agent-welcome button {
-            cursor: pointer;
-            border: 1px solid #2563eb;
-            border-radius: 6px;
-            background: #2563eb;
-            color: #fff;
-            padding: 8px 14px;
-            font-weight: 600;
-        }
-        @media (max-width: 700px) {
-            .opencode-agent-welcome {
-                padding: 12px;
-            }
-            .opencode-agent-welcome-panel {
-                max-height: calc(100vh - 170px);
-                padding: 18px;
-            }
-        }
-    </style>
-{% endblock %}
-
-{% block bodyclass %}{{ block.super }} opencode-agent{% endblock %}
-
-{% block content %}
-    <div id="content-main">
-        {% if error %}
-            <div class="opencode-agent-error">
-                {{ error }}
-                {% if command %}<br><br><code>{{ command }}</code>{% endif %}
-            </div>
-        {% else %}
-            <script>
-                (() => {
-                    const workdir = "{{ workdir|escapejs }}";
-                    const recentSessionId = "{{ recent_session_id|escapejs }}";
-                    const merge = (left, right) => {
-                        if (!left || typeof left !== "object" || Array.isArray(left)) return right;
-                        const out = {...left};
-                        for (const [key, value] of Object.entries(right)) {
-                            out[key] = value && typeof value === "object" && !Array.isArray(value) ? merge(out[key], value) : value;
-                        }
-                        return out;
-                    };
-                    const seed = (key, value) => {
-                        try {
-                            localStorage.setItem(key, JSON.stringify(merge(JSON.parse(localStorage.getItem(key) || "{}"), value)));
-                        } catch {
-                            localStorage.setItem(key, JSON.stringify(value));
-                        }
-                    };
-                    const setJSON = (key, value) => localStorage.setItem(key, JSON.stringify(value));
-                    const page = {
-                        activeProject: workdir,
-                        activeWorkspace: workdir,
-                        gettingStartedDismissed: true,
-                        workspaceExpanded: {[workdir]: true},
-                    };
-                    if (recentSessionId) {
-                        page.lastProjectSession = {[workdir]: {directory: workdir, id: recentSessionId, at: Date.now()}};
-                    }
-                    seed("opencode.global.dat:layout.page", page);
-                    seed("opencode.global.dat:layout", {
-                        sidebar: {opened: true, width: 280},
-                        fileTree: {opened: true, width: 280, tab: "all"},
-                    });
-                    const serverKey = `${location.origin}/admin/agent/opencode`;
-                    setJSON("opencode.global.dat:server", {
-                        list: [],
-                        projects: {
-                            [serverKey]: [{worktree: workdir, expanded: true}],
-                        },
-                        lastProject: {
-                            [serverKey]: workdir,
-                        },
-                    });
-                    localStorage.setItem("opencode.settings.dat:defaultServerUrl", serverKey);
-                })();
-            </script>
-            <div class="opencode-agent-shell">
-                <iframe src="{{ proxy_url }}" title="OpenCode Agent" class="opencode-agent-frame"></iframe>
-                <div class="opencode-agent-welcome" id="opencode-agent-welcome" hidden>
-                    <div class="opencode-agent-welcome-panel" role="dialog" aria-modal="true" aria-labelledby="opencode-agent-welcome-title">
-                        <h1 id="opencode-agent-welcome-title">ArchiveBox AI Agent</h1>
-                        <p>This agent can work directly with your ArchiveBox collection. It can inspect the crawl database, create and monitor crawls, run maintenance operations, answer research questions, query archived data, and help with complex collection workflows.</p>
-                        <p>Example prompts:</p>
-                        <ul>
-                            <li>Archive this list of URLs with depth 0, then report which ones failed and why.</li>
-                            <li>Find snapshots from the last month that failed PDF or screenshot extraction and retry them.</li>
-                            <li>Create a constrained crawl for this site, avoid login/privacy/sitemap URLs, and watch the logs as it runs.</li>
-                            <li>Summarize what my collection contains about this topic and link to the best saved pages.</li>
-                        </ul>
-                        <div class="opencode-agent-welcome-warning">
-                            The agent has unrestricted access to this collection and can make destructive edits. Review requests carefully. Permission settings can be changed from the OpenCode gear icon in the lower left.
-                        </div>
-                        <div class="opencode-agent-welcome-actions">
-                            <button type="button" id="opencode-agent-welcome-dismiss">Start using Agent</button>
-                        </div>
-                    </div>
-                </div>
-            </div>
-            <script>
-                (() => {
-                    const key = "archivebox.opencode.agentWelcomeDismissed.v1";
-                    const welcome = document.getElementById("opencode-agent-welcome");
-                    const dismiss = document.getElementById("opencode-agent-welcome-dismiss");
-                    if (!welcome || !dismiss) return;
-                    if (localStorage.getItem(key) !== "1") {
-                        welcome.hidden = false;
-                    }
-                    dismiss.addEventListener("click", () => {
-                        localStorage.setItem(key, "1");
-                        welcome.hidden = true;
-                    });
-                })();
-            </script>
-        {% endif %}
-    </div>
-{% endblock %}

archivebox/opencode/views.py

@@ -1,776 +1,140 @@
-from __future__ import annotations
+"""Django adapter for the optional plugin; never import its runtime at startup."""
 
-import atexit
-import base64
 import logging
-import os
-import re
-import signal
-import shutil
-import subprocess
-import threading
-import time
-from pathlib import Path
-from typing import Any
-from urllib.parse import urljoin, urlsplit
+from io import BytesIO
 
-import httpx
-import requests
-from abx_plugins.plugins import opencode as opencode_plugin
-from archivebox.config import CONSTANTS
-from archivebox.config.common import get_config
-from archivebox.core.routes_util import build_admin_url, get_api_base_url, get_base_url
 from asgiref.sync import sync_to_async
-from django.http import (
-    Http404,
-    HttpRequest,
-    HttpResponse,
-    HttpResponseForbidden,
-    StreamingHttpResponse,
-)
-from django.shortcuts import redirect, render
+
+from django.contrib.auth.middleware import AuthenticationMiddleware
+from django.contrib.auth.views import redirect_to_login
+from django.contrib.sessions.middleware import SessionMiddleware
+from django.core.exceptions import PermissionDenied
+from django.core.handlers.asgi import ASGIRequest
+from django.db import close_old_connections
+from django.http import Http404, HttpResponse, HttpResponseForbidden, StreamingHttpResponse
+from django.template import engines
+from django.urls import Resolver404, resolve
 from django.views.decorators.csrf import csrf_exempt
 
+from archivebox.config import CONSTANTS
+from archivebox.config.common import get_request_config
+from archivebox.core.middleware import ReverseProxyAuthMiddleware
+from archivebox.core.routes_util import build_admin_url, get_admin_host, get_api_base_url, get_base_url, host_matches
+from archivebox.plugins.discovery import get_plugin_template
 
-_PROCESS: subprocess.Popen | None = None
-_PROCESS_READY: subprocess.Popen | None = None
-_PROCESS_LOCK = threading.Lock()
-_SESSION_LOCK = threading.Lock()
 _LOGGER = logging.getLogger(__name__)
-_PROXY_PREFIX = "/admin/agent/opencode"
-_PROXY_PREFIX_REGEX = _PROXY_PREFIX.replace("/", r"\/")
-_PROXY_PREFIX_NO_SLASH_REGEX = _PROXY_PREFIX.lstrip("/").replace("/", r"\/")
-_CONFIG_PATH = Path(opencode_plugin.__file__).with_name("config.json")
-_DEFAULT_MODEL = "opencode/big-pickle"
-_DEFAULT_CONFIG = f'''{{
-  "$schema": "https://opencode.ai/config.json",
-  "model": "{_DEFAULT_MODEL}",
-  "snapshot": false
-}}
-'''
 
-_TEXT_CONTENT_TYPES = (
-    "text/",
-    "application/javascript",
-    "application/x-javascript",
-)
-_HOP_BY_HOP_HEADERS = {
-    "connection",
-    "keep-alive",
-    "proxy-authenticate",
-    "proxy-authorization",
-    "te",
-    "trailers",
-    "transfer-encoding",
-    "upgrade",
-}
-_ARCHIVEBOX_SKILL = """---
-name: archivebox
-description: Use ArchiveBox's CLI and local REST API from an ArchiveBox collection.
----
 
-You are running inside an ArchiveBox collection directory.
+def _runtime_settings(request, config):
+    from abx_plugins.plugins.opencode import runtime
 
-- ArchiveBox collection directory: {archivebox_data_dir}
-- ArchiveBox BASE_URL: {archivebox_base_url}
-- ArchiveBox Admin URL: {archivebox_admin_url}
-- ArchiveBox REST API URL: {archivebox_api_url}
-- Prefer the `archivebox` CLI for authenticated changes, e.g. `archivebox add`, `archivebox schedule`, `archivebox update`, and `archivebox shell`.
-- Run ArchiveBox CLI commands from the ArchiveBox collection directory above.
-- Get command help with `archivebox list --help`, `archivebox add --help`, `archivebox schedule --help`, etc. Do not use `archivebox help <command>`.
-- Use `--depth=0` by default. Only use recursive crawling when the user explicitly asks for it; use `--depth=1` when you need pages one hop out.
-- Before any recursive crawl, constrain scope with ArchiveBox config such as `CRAWL_MAX_URLS`, `CRAWL_MAX_SIZE`, `SNAPSHOT_MAX_*`, `URL_ALLOWLIST`, `URL_DENYLIST`, and related limits.
-- Respect the configured `archivebox config --get ONLY_NEW` behavior unless the user explicitly says otherwise. Remind users that expected crawl URLs can be skipped when the collection already contains snapshots with the same URL.
-- Always audit newly discovered crawl URLs before letting a crawl run broadly. Treat junk URLs such as privacy policies, legal pages, tag archives, sitemap files, feeds, login/logout URLs, and other low-value boilerplate as unwanted unless the user explicitly asked to archive them.
-- Always watch crawl output and logs as the crawl progresses, and correct errors early instead of waiting until the crawl finishes.
-- If a crawl contains bad URLs, pause it, edit the crawl's `urls` field to remove them, delete any unneeded snapshots already created under that crawl, then resume the crawl.
-- Use `archivebox shell -c '...'` or `archivebox shell <<'PY' ... PY` for Django ORM work. Shell Plus prints an import banner first; keep stderr visible while debugging.
-- Use full ArchiveBox module paths in shell code: `from archivebox.crawls.models import Crawl, CrawlSchedule` and `from archivebox.core.models import Snapshot, ArchiveResult`.
-- If a model/field/relation is unclear, inspect `_meta.fields` before guessing, e.g. `archivebox shell -c "from archivebox.crawls.models import Crawl; print([f.name for f in Crawl._meta.fields])"`.
-- Use `archivebox config --get BASE_URL` only to verify the configured base URL; prefer the seeded URLs above for API/admin requests.
-- Use `$ARCHIVEBOX_API_URL` for REST API inspection when helpful. Do not assume admin session cookies authenticate API subdomain requests; prefer CLI/shell for authenticated mutations unless the admin provides or asks you to create an API token.
-- Discover REST endpoints from `${{ARCHIVEBOX_API_URL}}v1/openapi.json`; crawl endpoints live under `/api/v1/crawls/`, snapshots under `/api/v1/core/`.
-- Do not bypass ArchiveBox auth, expose API keys, or modify config unless the admin explicitly asks.
-- After creating crawls or snapshots, report the crawl/snapshot IDs and the exact command or API request used.
-"""
-
-
-def _signal_owned_process(process: subprocess.Popen, sig: signal.Signals) -> None:
-    try:
-        os.killpg(process.pid, sig)
-    except OSError:
-        try:
-            process.send_signal(sig)
-        except ProcessLookupError:
-            pass
-
-
-def _stop_owned_process(process: subprocess.Popen | None = None) -> None:
-    global _PROCESS, _PROCESS_READY
-    owned_process = process or _PROCESS
-    if owned_process is None:
-        return
-    if owned_process.poll() is None:
-        _signal_owned_process(owned_process, signal.SIGCONT)
-        _signal_owned_process(owned_process, signal.SIGTERM)
-        try:
-            owned_process.wait(timeout=5)
-        except subprocess.TimeoutExpired:
-            _signal_owned_process(owned_process, signal.SIGKILL)
-            owned_process.wait()
-    if _PROCESS is owned_process:
-        _PROCESS = None
-    if _PROCESS_READY is owned_process:
-        _PROCESS_READY = None
-
-
-atexit.register(_stop_owned_process)
-
-
-def _machine_config() -> dict[str, Any]:
-    resolved = get_config()
-    return dict(resolved.model_dump(mode="json"))
-
-
-def _archivebox_data_dir_default() -> Path:
-    return Path(CONSTANTS.DATA_DIR)
-
-
-def _archivebox_route_urls(request: HttpRequest, route_config) -> tuple[str, str, str]:
-    base_url = get_base_url(
-        request=request,
-        config=route_config,
-    ).rstrip("/")
-    admin_url = build_admin_url(
-        "/admin/",
-        request=request,
-        config=route_config,
-    ).rstrip("/")
-    api_url = f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/"
-    return base_url, admin_url, api_url
-
-
-def _config_value(config: dict, key: str, default):
-    value = config.get(key, default)
-    if value in (None, ""):
-        return default
-    return value
-
-
-def _opencode_enabled(config: dict) -> bool:
-    value = config.get("OPENCODE_ENABLED", False)
-    if isinstance(value, bool):
-        return value
-    return str(value).strip().lower() in {"1", "true", "yes", "on"}
-
-
-def _require_enabled(config: dict) -> None:
-    if not _opencode_enabled(config):
-        raise Http404
-
-
-def _require_superuser(request: HttpRequest):
-    user = getattr(request, "user", None)
-    if user is None:
-        return redirect(f"/admin/login/?next={request.get_full_path()}")
-    if (
-        bool(getattr(user, "is_authenticated", False))
-        and bool(getattr(user, "is_active", False))
-        and bool(getattr(user, "is_superuser", False))
-    ):
-        return None
-    if bool(getattr(user, "is_authenticated", False)):
-        return HttpResponseForbidden(
-            b"ArchiveBox agent access requires a superuser account.",
-        )
-    return redirect(f"/admin/login/?next={request.get_full_path()}")
-
-
-def _origin_allowed(request: HttpRequest, path: str | None = None) -> bool:
-    if request.method in {"GET", "HEAD", "OPTIONS", "TRACE"}:
-        return True
-
-    expected_host = request.get_host()
-    pty_connect = bool(
-        path and path.startswith("pty/") and path.endswith("/connect-token"),
-    )
-    if pty_connect:
-        return True
-
-    origin = _request_header(request, "Origin")
-    if origin:
-        return _same_host(origin, expected_host)
-
-    referer = _request_header(request, "Referer")
-    if referer:
-        return _same_host(referer, expected_host)
-
-    fetch_site = _request_header(request, "Sec-Fetch-Site")
-    if fetch_site:
-        return fetch_site in {"same-origin", "same-site", "none"}
-
-    return False
-
-
-def _same_host(value: str, expected_host: str) -> bool:
-    parsed = urlsplit(value)
-    return parsed.scheme in {"http", "https"} and parsed.netloc == expected_host
-
-
-def _settings(config: dict) -> dict:
-    host = str(_config_value(config, "OPENCODE_HOST", "127.0.0.1"))
-    port = int(_config_value(config, "OPENCODE_PORT", 4096))
-    default_data_dir = _archivebox_data_dir_default()
-    opencode_dir = Path(
-        str(_config_value(config, "OPENCODE_STATE_DIR", default_data_dir / "opencode")),
-    ).expanduser()
-    workdir = Path(
-        str(_config_value(config, "OPENCODE_WORKDIR", default_data_dir)),
-    ).expanduser()
-    binary = str(_config_value(config, "OPENCODE_BINARY", "opencode"))
-    timeout = int(_config_value(config, "OPENCODE_TIMEOUT", 120))
-    return {
-        "host": host,
-        "port": port,
-        "origin": f"http://{host}:{port}",
-        "archivebox_data_dir": default_data_dir,
-        "workdir": workdir,
-        "opencode_dir": opencode_dir,
-        "config_home": opencode_dir / "config",
-        "data_home": opencode_dir / "data",
-        "state_home": opencode_dir / "state",
-        "cache_home": opencode_dir / "cache",
-        "home": opencode_dir / "home",
-        "binary": binary,
-        "config": config,
-        "timeout": timeout,
-    }
-
-
-def _resolve_binary(binary: str, config: dict) -> tuple[Any, Any, dict[str, str]]:
-    try:
-        from abxpkg import BinProvider
-        from abx_plugins.plugins.base.utils import load_required_binary_from_config
-
-        binary_environ = os.environ.copy()
-        lib_dir = config.get("ABXPKG_LIB_DIR")
-        if lib_dir:
-            binary_environ["ABXPKG_LIB_DIR"] = str(lib_dir)
-        loaded_dependencies = [
-            load_required_binary_from_config(
-                required_binary,
-                _CONFIG_PATH,
-                global_config=config,
-                environ=binary_environ,
-                install=False,
-            )
-            for required_binary in (
-                str(config.get("NODE_BINARY") or "node"),
-                str(config.get("GIT_BINARY") or "git"),
-                binary,
-            )
-        ]
-    except Exception as err:
-        raise RuntimeError(
-            f"OpenCode dependency is not installed from required_binaries: {err}",
-        ) from err
-
-    if any(not loaded.loaded_abspath for loaded in loaded_dependencies):
-        raise RuntimeError(
-            "OpenCode dependency is not installed from required_binaries.",
-        )
-
-    providers = [loaded.loaded_binprovider for loaded in loaded_dependencies if loaded.loaded_binprovider is not None]
-    binary_env = BinProvider.build_exec_env(
-        providers=providers,
-        base_env=binary_environ,
-    )
-    return (
-        loaded_dependencies[-1],
-        loaded_dependencies[1],
-        binary_env,
-    )
-
-
-def _project_route(workdir: Path, session_id: str = "") -> str:
-    encoded = base64.b64encode(str(workdir.resolve()).encode()).decode()
-    encoded = encoded.replace("+", "-").replace("/", "_").rstrip("=")
-    route = f"{_PROXY_PREFIX}/{encoded}/session"
-    return f"{route}/{session_id}" if session_id else route
-
-
-def _ensure_project_files(settings: dict) -> None:
-    workdir = settings["workdir"].resolve()
-    workdir.mkdir(parents=True, exist_ok=True)
-    git_marker = workdir / ".git" / "not-a-git"
-    if git_marker.exists():
-        # Current OpenCode hangs on the legacy fake marker, so remove only
-        # that invalid shape before initializing the real worktree.
-        shutil.rmtree(git_marker.parent)
-
-    editable_skill_path = settings["opencode_dir"] / "SKILL.md"
-    editable_skill_path.parent.mkdir(parents=True, exist_ok=True)
-    if not editable_skill_path.exists():
-        editable_skill_path.write_text(
-            _ARCHIVEBOX_SKILL.format(
-                archivebox_data_dir=settings["archivebox_data_dir"],
-                archivebox_base_url=settings.get("archivebox_base_url", ""),
-                archivebox_admin_url=settings.get("archivebox_admin_url", ""),
-                archivebox_api_url=settings.get("archivebox_api_url", ""),
-            ),
-        )
-
-    opencode_skill_path = settings["config_home"] / "opencode" / "skills" / "archivebox" / "SKILL.md"
-    opencode_skill_path.parent.mkdir(parents=True, exist_ok=True)
-    if opencode_skill_path.resolve() != editable_skill_path.resolve():
-        if opencode_skill_path.exists() or opencode_skill_path.is_symlink():
-            opencode_skill_path.unlink()
-        opencode_skill_path.symlink_to(editable_skill_path)
-
-    opencode_config_path = settings["config_home"] / "opencode" / "opencode.jsonc"
-    if not opencode_config_path.exists():
-        opencode_config_path.write_text(_DEFAULT_CONFIG)
-
-
-def _ensure_default_session(settings: dict) -> str:
-    workdir = settings["workdir"].resolve()
-    params = {"directory": str(workdir)}
-    timeout = settings["timeout"]
-    project = requests.post(
-        f"{settings['origin']}/project/git/init",
-        params=params,
-        timeout=timeout,
-    )
-    project.raise_for_status()
-    project_data = project.json()
-    if Path(str(project_data.get("worktree") or "/")).resolve() != workdir:
-        raise RuntimeError(
-            f"OpenCode initialized the wrong project worktree: {project_data.get('worktree')!r}",
-        )
-
-    sessions = requests.get(
-        f"{settings['origin']}/session",
-        params={**params, "roots": "true", "limit": 55},
-        timeout=timeout,
-    )
-    sessions.raise_for_status()
-    session_data = sessions.json()
-    if not isinstance(session_data, list):
-        raise RuntimeError("OpenCode returned an invalid project session list.")
-    for session_data_item in session_data:
-        if not isinstance(session_data_item, dict):
-            continue
-        session_id = str(session_data_item.get("id") or "")
-        session_directory = session_data_item.get("directory")
-        if session_id and session_directory and Path(str(session_directory)).resolve() == workdir:
-            return session_id
-
-    session = requests.post(
-        f"{settings['origin']}/session",
-        params=params,
-        json={},
-        timeout=timeout,
-    )
-    session.raise_for_status()
-    session_data = session.json()
-    session_id = str(session_data.get("id") or "")
-    session_directory = session_data.get("directory")
-    if not session_id or not session_directory or Path(str(session_directory)).resolve() != workdir:
-        raise RuntimeError(
-            "OpenCode did not create a session for the requested worktree.",
-        )
-    return session_id
-
-
-def _owned_process_running() -> bool:
-    process = _PROCESS
-    return process is not None and process.poll() is None
-
-
-def _owned_process_ready() -> bool:
-    process = _PROCESS_READY
-    return process is not None and process is _PROCESS and process.poll() is None
-
-
-def _health(settings: dict, timeout: float = 2) -> bool:
-    try:
-        response = requests.get(
-            f"{settings['origin']}/global/health",
-            timeout=timeout,
-        )
-        return response.status_code == 200
-    except requests.RequestException:
-        return False
-
-
-def _ensure_opencode(settings: dict) -> tuple[bool, str]:
-    global _PROCESS, _PROCESS_READY
-    started_process: subprocess.Popen | None = None
-    workdir = settings["workdir"].resolve()
-
-    with _PROCESS_LOCK:
-        if _owned_process_ready():
-            return True, ""
-        if _health(settings):
-            if _owned_process_running():
-                _PROCESS_READY = _PROCESS
-            return True, ""
-        if _owned_process_running():
-            _stop_owned_process(_PROCESS)
-
-        try:
-            binary, git_binary, binary_env = _resolve_binary(
-                settings["binary"],
-                settings["config"],
-            )
-        except RuntimeError as err:
-            return False, str(err)
-
-        env = {
-            **os.environ,
-            **binary_env,
-            "ARCHIVEBOX_BASE_URL": str(settings.get("archivebox_base_url", "")),
-            "ARCHIVEBOX_ADMIN_URL": str(settings.get("archivebox_admin_url", "")),
-            "ARCHIVEBOX_API_URL": str(settings.get("archivebox_api_url", "")),
-            "BROWSER": "false",
-            "GIT_CEILING_DIRECTORIES": str(workdir),
-            "HOME": str(settings["home"]),
-            "OPENCODE_DISABLE_PROJECT_CONFIG": "true",
-            "XDG_CONFIG_HOME": str(settings["config_home"]),
-            "XDG_DATA_HOME": str(settings["data_home"]),
-            "XDG_STATE_HOME": str(settings["state_home"]),
-            "XDG_CACHE_HOME": str(settings["cache_home"]),
-        }
-
-        settings["workdir"].mkdir(parents=True, exist_ok=True)
-        settings["config_home"].mkdir(parents=True, exist_ok=True)
-        settings["data_home"].mkdir(parents=True, exist_ok=True)
-        settings["state_home"].mkdir(parents=True, exist_ok=True)
-        settings["cache_home"].mkdir(parents=True, exist_ok=True)
-        settings["home"].mkdir(parents=True, exist_ok=True)
-        _ensure_project_files(settings)
-
-        if not (workdir / ".git").exists():
-            try:
-                git_init = git_binary.exec(
-                    cmd=("init", "--quiet"),
-                    cwd=workdir,
-                    env=env,
-                    timeout=settings["timeout"],
-                )
-            except (AssertionError, OSError, subprocess.SubprocessError) as err:
-                return False, f"OpenCode project initialization failed: {err}"
-            if git_init.returncode != 0:
-                output = (git_init.stderr or git_init.stdout or "").strip()
-                return (
-                    False,
-                    f"OpenCode project initialization failed: {output or f'git exited with {git_init.returncode}'}",
-                )
-
-        if _health(settings):
-            return True, ""
-
-        binary_abspath = binary.loaded_abspath
-        if binary.loaded_binprovider is not None:
-            binary_abspath = binary.loaded_binprovider._exec_bin_abspath(
-                Path(binary.loaded_abspath),
-            )
-        cmd = [
-            str(binary_abspath),
-            "serve",
-            "--hostname",
-            settings["host"],
-            "--port",
-            str(settings["port"]),
-        ]
-        try:
-            _PROCESS = subprocess.Popen(
-                cmd,
-                cwd=workdir,
-                env=env,
-                stdin=subprocess.DEVNULL,
-                stdout=subprocess.DEVNULL,
-                start_new_session=True,
-            )
-            _PROCESS_READY = None
-            started_process = _PROCESS
-        except FileNotFoundError:
-            return False, f"OpenCode binary not found: {settings['binary']}"
-
-        deadline = time.monotonic() + settings["timeout"]
-        while True:
-            remaining = deadline - time.monotonic()
-            if remaining <= 0:
-                break
-            if _health(settings, timeout=min(2, remaining)):
-                if time.monotonic() <= deadline:
-                    _PROCESS_READY = started_process
-                    return True, ""
-                break
-            if started_process and started_process.poll() is not None:
-                if _PROCESS is started_process:
-                    _PROCESS = None
-                if _PROCESS_READY is started_process:
-                    _PROCESS_READY = None
-                return False, "OpenCode exited before the web server became ready."
-            remaining = deadline - time.monotonic()
-            if remaining > 0:
-                time.sleep(min(0.25, remaining))
-
-        _stop_owned_process(started_process)
-        return False, "Timed out waiting for OpenCode to start."
-
-
-def agent_view(request: HttpRequest):
-    config = _machine_config()
-    _require_enabled(config)
-    auth_response = _require_superuser(request)
-    if auth_response:
-        return auth_response
-
-    settings = _settings(config)
+    settings = runtime._settings(config, CONSTANTS.DATA_DIR)
     route_config = request.__dict__.get("archivebox_config")
-    base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
-    settings["archivebox_base_url"] = base_url
-    settings["archivebox_admin_url"] = admin_url
-    settings["archivebox_api_url"] = api_url
-    ok, error = _ensure_opencode(settings)
-    recent_session_id = ""
-    if ok:
-        try:
-            with _SESSION_LOCK:
-                recent_session_id = _ensure_default_session(settings)
-        except (requests.RequestException, RuntimeError, ValueError) as err:
-            if isinstance(err, requests.RequestException) and _owned_process_ready():
-                _stop_owned_process()
-            ok = False
-            error = f"OpenCode project initialization failed: {err}"
-    from archivebox.core.admin_site import archivebox_admin
-
-    context = {
-        **archivebox_admin.each_context(request),
-        "title": "Agent",
-        "error": "" if ok else error,
-        "command": f"{settings['binary']} serve --hostname {settings['host']} --port {settings['port']}" if error else "",
-        # OpenCode 1.17+ keeps durable sessions on the explicit
-        # /<dirBase64>/session/<sessionId> route. We still seed localStorage
-        # below because the sidebar state uses it, but the iframe itself must
-        # open the durable session URL so a fresh browser does not land on the
-        # transient new-session route and appear to have lost prior sessions.
-        "proxy_url": _project_route(settings["workdir"], recent_session_id),
-        "proxy_prefix": _PROXY_PREFIX,
-        "workdir": str(settings["workdir"].resolve()),
-        "recent_session_id": recent_session_id,
-    }
-    return render(
-        request,
-        "opencode/agent.html",
-        context,
-        status=200 if ok else 502,
+    settings.update(
+        archivebox_base_url=get_base_url(request=request, config=route_config).rstrip("/"),
+        archivebox_admin_url=build_admin_url("/admin/", request=request, config=route_config).rstrip("/"),
+        archivebox_api_url=f"{get_api_base_url(request=request, config=route_config).rstrip('/')}/api/",
     )
+    return runtime, settings
 
 
-def _proxy_url(settings: dict, path: str | None) -> str:
-    rel = "/" if not path else f"/{path}"
-    return urljoin(settings["origin"], rel)
-
-
-def _request_header(request: HttpRequest, name: str) -> str | None:
-    meta = getattr(request, "META", {})
-    if name == "Content-Type":
-        value = meta.get("CONTENT_TYPE")
-    elif name == "Content-Length":
-        value = meta.get("CONTENT_LENGTH")
-    else:
-        value = meta.get(f"HTTP_{name.upper().replace('-', '_')}")
-    return str(value) if value else None
-
-
-def _request_headers(request: HttpRequest, settings: dict) -> dict[str, str]:
-    forwarded = {}
-    for key in ("Accept", "Accept-Language", "Content-Type", "Range", "User-Agent"):
-        value = _request_header(request, key)
-        if value:
-            forwarded[key] = value
-    return forwarded
-
-
-def _request_params(request: HttpRequest) -> tuple[tuple[str, str], ...]:
-    if hasattr(request.GET, "lists"):
-        return tuple((key, str(value)) for key, values in request.GET.lists() for value in values)
-    return tuple((key, str(value)) for key, value in dict(request.GET).items())
-
-
-async def _event_chunks(
-    settings: dict,
-    path: str | None,
-    method: str,
-    params: tuple[tuple[str, str], ...],
-    headers: dict[str, str],
-):
-    if not _owned_process_ready():
-        ok, error = await sync_to_async(_ensure_opencode, thread_sensitive=False)(settings)
-        if not ok:
-            _LOGGER.warning("OpenCode event stream unavailable: %s", error)
-            yield b'event: error\ndata: {"error":"OpenCode upstream unavailable"}\n\n'
-            return
-
-    timeout = httpx.Timeout(settings["timeout"], read=None)
-    url = _proxy_url(settings, path)
+def _dispatch(request, path=None):
     try:
-        async with httpx.AsyncClient(timeout=timeout, follow_redirects=False) as client:
-            async with client.stream(
-                method,
-                url,
-                params=params,
-                headers=headers,
-            ) as upstream:
-                async for chunk in upstream.aiter_raw(chunk_size=512):
-                    yield chunk
-    except httpx.RequestError as err:
-        _LOGGER.warning("OpenCode event stream ended: %s", err)
+        # Middleware already merged server defaults and live Machine overrides.
+        # Re-resolving every extractor's config here stalls the UI request burst.
+        config = get_request_config(request).model_dump(mode="json")
+        if not config.get("OPENCODE_ENABLED", False):
+            raise Http404
+        if not request.user.is_authenticated:
+            return redirect_to_login(request.get_full_path(), login_url="/admin/login/")
+        if not request.user.is_active or not request.user.is_superuser:
+            return HttpResponseForbidden("Agent access requires a superuser account.")
+
+        runtime, settings = _runtime_settings(request, config)
+        if path is None:
+            from archivebox.core.admin_site import archivebox_admin
+
+            context = {**archivebox_admin.each_context(request), **runtime.agent_context(settings)}
+            source = get_plugin_template("opencode", "agent", fallback=False)
+            if source is None:
+                raise RuntimeError("Agent template unavailable")
+            return HttpResponse(engines["django"].from_string(source).render(context, request))
+
+        if not runtime._origin_allowed(request.method, request.get_host(), request.headers):
+            return HttpResponseForbidden("Cross-origin agent requests are blocked.")
+        status, headers, body = runtime.proxy(
+            settings,
+            request.method,
+            path,
+            tuple((key, value) for key, values in request.GET.lists() for value in values),
+            request.headers,
+            request.body,
+        )
+        response_type = HttpResponse if isinstance(body, bytes) else StreamingHttpResponse
+        response = response_type(body, status=status, headers=headers)
+        response.xframe_options_exempt = True
+        response.headers["X-Frame-Options"] = "SAMEORIGIN"
+        response.headers["Content-Security-Policy"] = "frame-ancestors 'self'"
+        return response
+    except Http404:
+        raise
+    except Exception:
+        # Optional-service boundary, including imports and template rendering.
+        _LOGGER.exception("Optional AI service failed")
+        return HttpResponse("AI service unavailable. See server logs.", status=503, content_type="text/plain")
 
 
-def _rewrite_text(body: bytes, settings: dict) -> bytes:
-    text = body.decode("utf-8", errors="replace")
-    text = text.replace(settings["origin"], _PROXY_PREFIX)
-    text = text.replace("location.origin", f'location.origin+"{_PROXY_PREFIX}"')
-    text = text.replace(
-        "k(k5,{get component(){return t.router??Az},",
-        f'k(k5,{{base:"{_PROXY_PREFIX}",get component(){{return t.router??Az}},',
-    )
-    text = text.replace('"/assets/', f'"{_PROXY_PREFIX}/assets/')
-    text = text.replace("'/assets/", f"'{_PROXY_PREFIX}/assets/")
-    proxy_path = rf'(\1.replace(/^{_PROXY_PREFIX_REGEX}(?=\/|$)/,"")||"/")'
-    text = re.sub(r"\b(window\.location\.pathname)\b", proxy_path, text)
-    text = re.sub(r"(?<![.\w])(location\.pathname)\b", proxy_path, text)
-    text = text.replace(
-        'window.history.replaceState(nz(o),"",r):window.history.pushState(o,"",r)',
-        (
-            f'window.history.replaceState(nz(o),"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r):'
-            f'window.history.pushState(o,"",r.startsWith("{_PROXY_PREFIX}")?r:r.startsWith("/")?"{_PROXY_PREFIX}"+r:r)'
-        ),
-    )
-    text = text.replace(
-        'const BL="modulepreload",UL=function(t){return"/"+t}',
-        f'const BL="modulepreload",UL=function(t){{return"{_PROXY_PREFIX}/"+t}}',
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\b(?:href|src|action)=["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\b(?:fetch|EventSource)\(["'])/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    text = re.sub(
-        rf"""(?P<prefix>\burl\(["']?)/(?!{_PROXY_PREFIX_NO_SLASH_REGEX}(?:/|$))""",
-        rf"\g<prefix>{_PROXY_PREFIX}/",
-        text,
-    )
-    return text.encode("utf-8")
-
-
-def _response_headers(upstream: requests.Response, settings: dict) -> dict[str, str]:
-    headers = {}
-    for key, value in upstream.headers.items():
-        lower = key.lower()
-        if lower in _HOP_BY_HOP_HEADERS or lower in {
-            "content-length",
-            "content-encoding",
-            "x-frame-options",
-        }:
-            continue
-        if lower == "location":
-            if value.startswith(settings["origin"]):
-                value = value.replace(settings["origin"], _PROXY_PREFIX, 1)
-            elif value.startswith("/"):
-                value = f"{_PROXY_PREFIX}{value}"
-        headers[key] = value
-    return headers
-
-
-def _proxy_error_response(error: Exception | str) -> HttpResponse:
-    _LOGGER.warning("OpenCode upstream request failed: %s", error)
-    return HttpResponse(
-        b"OpenCode upstream request failed.",
-        status=502,
-        content_type="text/plain; charset=utf-8",
-    )
+def agent_view(request):
+    return _dispatch(request)
 
 
 @csrf_exempt
-def opencode_proxy_view(request: HttpRequest, path: str | None = None):
-    config = _machine_config()
-    _require_enabled(config)
-    auth_response = _require_superuser(request)
-    if auth_response:
-        return auth_response
-    if not _origin_allowed(request, path):
-        return HttpResponseForbidden(
-            b"Cross-origin OpenCode agent requests are blocked.",
-        )
+def opencode_proxy_view(request, path=None):
+    return _dispatch(request, path=path or "")
 
-    settings = _settings(config)
-    route_config = request.__dict__.get("archivebox_config")
-    base_url, admin_url, api_url = _archivebox_route_urls(request, route_config)
-    settings["archivebox_base_url"] = base_url
-    settings["archivebox_admin_url"] = admin_url
-    settings["archivebox_api_url"] = api_url
-
-    if request.method == "GET" and (path or "").endswith("/event"):
-        response = StreamingHttpResponse(
-            _event_chunks(
-                settings,
-                path,
-                request.method or "GET",
-                _request_params(request),
-                _request_headers(request, settings),
-            ),
-            content_type="text/event-stream",
-        )
-        response.headers["Cache-Control"] = "no-store"
-        response.headers["X-Accel-Buffering"] = "no"
-        return response
-
-    if path == "global/health" or not _owned_process_ready():
-        ok, error = _ensure_opencode(settings)
-        if not ok:
-            return _proxy_error_response(error)
 
+def _websocket_context(scope):
+    close_old_connections()
     try:
-        method = request.method or "GET"
-        upstream = requests.request(
-            method,
-            _proxy_url(settings, path),
-            params=_request_params(request),
-            data=request.body if method not in {"GET", "HEAD"} else None,
-            headers=_request_headers(request, settings),
-            stream=True,
-            timeout=settings["timeout"],
-            allow_redirects=False,
+        match = resolve(scope["path"])
+        if match.url_name != "opencode-proxy":
+            raise PermissionDenied
+        request = ASGIRequest(
+            {**scope, "type": "http", "method": "GET", "scheme": "https" if scope.get("scheme") == "wss" else "http"},
+            BytesIO(),
         )
-    except requests.RequestException as err:
-        return _proxy_error_response(err)
+        route_config = get_request_config(request, resolve_plugins=False)
+        request.archivebox_config = route_config
+        if not route_config.CONTROL_PLANE_ENABLED:
+            raise PermissionDenied
+        if (
+            route_config.USES_SUBDOMAIN_ROUTING
+            and route_config.BASE_URL
+            and not host_matches(request.get_host(), get_admin_host(config=route_config, request=request))
+        ):
+            raise PermissionDenied
+        SessionMiddleware(_dispatch).process_request(request)
+        AuthenticationMiddleware(_dispatch).process_request(request)
+        ReverseProxyAuthMiddleware(_dispatch).process_request(request)
+        config = route_config.model_dump(mode="json")
+        if not config.get("OPENCODE_ENABLED") or not request.user.is_active or not request.user.is_superuser:
+            raise PermissionDenied
+        runtime, settings = _runtime_settings(request, config)
+        if not request.headers.get("Origin") or not runtime._origin_allowed("POST", request.get_host(), request.headers):
+            raise PermissionDenied
+        return runtime, settings, match.kwargs.get("path", "")
+    finally:
+        close_old_connections()
 
+
+async def websocket_view(scope, receive, send):
+    if (await receive())["type"] != "websocket.connect":
+        return
     try:
-        body = upstream.content
-    except requests.RequestException as err:
-        upstream.close()
-        return _proxy_error_response(err)
-
-    content_type = upstream.headers.get("Content-Type", "")
-    headers = _response_headers(upstream, settings)
-    if any(content_type.startswith(prefix) for prefix in _TEXT_CONTENT_TYPES):
-        body = _rewrite_text(body, settings)
-    response = HttpResponse(
-        body,
-        status=upstream.status_code,
-        content_type=content_type or "application/octet-stream",
-    )
-    for key, value in headers.items():
-        response.headers[key] = value
-    response.headers["Cache-Control"] = "no-store"
-    return response
+        runtime, settings, path = await sync_to_async(_websocket_context)(scope)
+        await runtime.websocket_proxy(settings, path, scope.get("query_string", b""), scope.get("subprotocols", []), receive, send)
+    except (PermissionDenied, Resolver404, Http404):
+        await send({"type": "websocket.close", "code": 1008})
+    except Exception:
+        _LOGGER.exception("Optional AI WebSocket failed")
+        await send({"type": "websocket.close", "code": 1011})

archivebox/personas/admin.py

@@ -27,7 +27,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
             "Persona",
             {
                 "fields": ("name", "created_by", "permissions"),
-                "classes": ("card", "persona-card-primary"),
+                "classes": ("card", "wide", "persona-card-primary"),
             },
         ),
         (
@@ -42,7 +42,7 @@ class PersonaAdmin(ConfigEditorMixin, BaseModelAdmin):
                     "import_extract_cookies",
                     "import_capture_storage",
                 ),
-                "classes": ("card", "wide"),
+                "classes": ("card", "wide", "persona-card-import"),
             },
         ),
         (

archivebox/personas/export_browser_state.js

@@ -9,11 +9,11 @@
  *   COOKIES_OUTPUT_FILE         Optional output path for Netscape cookies.txt
  *   AUTH_STORAGE_OUTPUT_FILE    Optional output path for auth.json
  *   CHROME_BINARY               Absolute browser path resolved by abxpkg
+ *   CHROME_SANDBOX              Set to false when the Linux sandbox is unavailable
  *   NODE_MODULES_DIR            node_modules path resolved by abxpkg
  */
 
 const fs = require('fs');
-const os = require('os');
 const path = require('path');
 
 const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR || process.env.ABX_PLUGINS_DIR;
@@ -26,7 +26,7 @@ const baseUtils = require(path.join(pluginsDir, 'base', 'utils.js'));
 baseUtils.ensureNodeModuleResolution(module);
 
 const chromeUtils = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
-const puppeteer = require('puppeteer-core');
+const puppeteer = require('puppeteer');
 
 function cookieToNetscape(cookie) {
     let domain = cookie.domain;
@@ -115,38 +115,28 @@ async function openBrowser() {
         throw new Error(`User data directory does not exist: ${userDataDir}`);
     }
 
-    const outputDir = fs.mkdtempSync(path.join(os.tmpdir(), 'abx-browser-state-'));
     const binary = process.env.CHROME_BINARY;
     if (!binary || !path.isAbsolute(binary) || !fs.existsSync(binary)) {
         throw new Error('CHROME_BINARY must be an absolute executable path resolved by abxpkg');
     }
 
-    const launched = await chromeUtils.launchChromium({
-        binary,
-        outputDir,
+    // The ordinary archiving launcher uses a mock keychain. An import must
+    // instead let the original signed browser decrypt its own profile cookies.
+    const sandboxDisabled = ['false', '0', 'no', 'off'].includes(
+        String(process.env.CHROME_SANDBOX || '').trim().toLowerCase(),
+    );
+    const browser = await puppeteer.launch({
+        executablePath: binary,
         userDataDir,
         headless: true,
-        killZombies: false,
+        ignoreDefaultArgs: ['--use-mock-keychain', '--password-store=basic'],
+        args: ['--no-first-run', '--disable-sync', '--disable-background-networking',
+               '--profile-directory=Default', ...(process.platform === 'linux' && (process.getuid?.() === 0 || sandboxDisabled) ? ['--no-sandbox'] : [])],
     });
-
-    if (!launched.success) {
-        throw new Error(launched.error || 'Chrome launch failed');
-    }
-
-    const browser = await chromeUtils.connectToBrowserEndpoint(puppeteer, launched.cdpUrl, { defaultViewport: null });
-
     return {
         browser,
         async cleanup() {
-            try {
-                await browser.disconnect();
-            } catch (error) {}
-            try {
-                await chromeUtils.killChrome(launched.pid, outputDir);
-            } catch (error) {}
-            try {
-                fs.rmSync(outputDir, { recursive: true, force: true });
-            } catch (error) {}
+            await browser.close();
         },
         sourceDescription: userDataDir,
     };

archivebox/personas/forms.py

@@ -55,7 +55,7 @@ class PersonaAdminForm(PluginConfigFormMixin, forms.ModelForm):
         label="Autodiscovered profiles",
         widget=forms.RadioSelect,
         choices=(),
-        help_text="Detected from local Chrome, Chromium, Brave, and Edge profile roots.",
+        help_text="Detected from local Chrome, Chrome Beta, Chromium, Brave, and Edge profile roots.",
     )
     import_source = forms.CharField(
         required=False,

archivebox/personas/importers.py

@@ -11,10 +11,14 @@ import json
 import os
 import platform
 import shutil
+import sqlite3
+import time
+from http.cookiejar import MozillaCookieJar
 import subprocess
 import sys
 import tempfile
 from dataclasses import dataclass, field
+from functools import partial
 from pathlib import Path
 from typing import TYPE_CHECKING
 from urllib.parse import urlparse
@@ -28,6 +32,7 @@ if TYPE_CHECKING:
 
 BROWSER_LABELS = {
     "chrome": "Google Chrome",
+    "chrome-beta": "Google Chrome Beta",
     "chromium": "Chromium",
     "brave": "Brave",
     "edge": "Microsoft Edge",
@@ -42,25 +47,27 @@ BROWSER_PROFILE_DIR_NAMES = (
 )
 
 VOLATILE_PROFILE_COPY_PATTERNS = (
+    # Chromium/macOS atomic-write staging files (e.g.
+    # .com.brave.Browser.TransportSecurity.Ze7UBU) disappear after rename.
+    ".*.??????",
     "Cache",
     "Code Cache",
     "GPUCache",
     "ShaderCache",
-    "Service Worker",
-    "GCM Store",
-    "chrome-extension_*",
-    "DNR Extension Rules",
-    "Extension Rules",
-    "Extension Scripts",
-    "Extension State",
-    "Local Extension Settings",
-    "*.log",
     "Crashpad",
     "BrowserMetrics",
     "BrowserMetrics-spare.pma",
+    "RunningChromeVersion",
+    "DevToolsActivePort",
     "SingletonLock",
     "SingletonSocket",
     "SingletonCookie",
+    "Sessions",
+    "Sessions_Encrypted",
+    "Current Session",
+    "Current Tabs",
+    "Last Session",
+    "Last Tabs",
 )
 
 PERSONA_PROFILE_DIR_CANDIDATES = (
@@ -201,6 +208,7 @@ class PersonaImportResult:
     source: PersonaImportSource
     profile_copied: bool = False
     cookies_imported: bool = False
+    cookie_count: int = 0
     storage_captured: bool = False
     user_agent_imported: bool = False
     warnings: list[str] = field(default_factory=list)
@@ -210,28 +218,29 @@ class PersonaImportResult:
         return self.profile_copied or self.cookies_imported or self.storage_captured or self.user_agent_imported
 
 
-def get_chrome_user_data_dir() -> Path | None:
+def get_chrome_user_data_dir(*, beta: bool = False) -> Path | None:
     """Get the default Chrome user data directory for the current platform."""
     system = platform.system()
     home = Path.home()
+    app_name = "Chrome Beta" if beta else "Chrome"
 
     if system == "Darwin":
         candidates = [
-            home / "Library" / "Application Support" / "Google" / "Chrome",
-            home / "Library" / "Application Support" / "Chromium",
+            home / "Library" / "Application Support" / "Google" / app_name,
         ]
     elif system == "Linux":
-        candidates = [
-            home / ".config" / "google-chrome",
-            home / ".config" / "chromium",
-            home / ".config" / "chrome",
-            home / "snap" / "chromium" / "common" / "chromium",
-        ]
+        candidates = (
+            [home / ".config" / "google-chrome-beta"]
+            if beta
+            else [
+                home / ".config" / "google-chrome",
+                home / ".config" / "chrome",
+            ]
+        )
     elif system == "Windows":
         local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
         candidates = [
-            local_app_data / "Google" / "Chrome" / "User Data",
-            local_app_data / "Chromium" / "User Data",
+            local_app_data / "Google" / app_name / "User Data",
         ]
     else:
         candidates = []
@@ -243,6 +252,21 @@ def get_chrome_user_data_dir() -> Path | None:
     return None
 
 
+def get_chromium_user_data_dir() -> Path | None:
+    """Select Chromium's own profile even when Google Chrome is installed."""
+    home = Path.home()
+    if platform.system() == "Darwin":
+        candidates = [home / "Library/Application Support/Chromium"]
+    elif platform.system() == "Windows":
+        candidates = [Path(os.environ.get("LOCALAPPDATA", home / "AppData/Local")) / "Chromium/User Data"]
+    else:
+        candidates = [
+            Path(os.environ.get("XDG_CONFIG_HOME", home / ".config")) / "chromium",
+            home / "snap/chromium/common/chromium",
+        ]
+    return next((path for path in candidates if path.is_dir() and _list_profile_names(path)), None)
+
+
 def get_brave_user_data_dir() -> Path | None:
     """Get the default Brave user data directory for the current platform."""
     system = platform.system()
@@ -303,7 +327,8 @@ def get_edge_user_data_dir() -> Path | None:
 
 BROWSER_PROFILE_FINDERS = {
     "chrome": get_chrome_user_data_dir,
-    "chromium": get_chrome_user_data_dir,
+    "chrome-beta": partial(get_chrome_user_data_dir, beta=True),
+    "chromium": get_chromium_user_data_dir,
     "brave": get_brave_user_data_dir,
     "edge": get_edge_user_data_dir,
 }
@@ -337,6 +362,12 @@ def validate_persona_name(name: str) -> tuple[bool, str]:
 
 
 def discover_local_browser_profiles() -> list[PersonaImportSource]:
+    # The macOS companion exports with the host Keychain before creating personas.
+    # Raw mounts alone cannot decrypt macOS cookies in the Linux container.
+    if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
+        host_templates = discover_persona_template_profiles(Path(host_profiles))
+        host_names = {source.source_name for source in host_templates}
+        return host_templates + [source for source in discover_persona_template_profiles() if source.source_name not in host_names]
     discovered: list[PersonaImportSource] = []
 
     for browser, finder in BROWSER_PROFILE_FINDERS.items():
@@ -413,13 +444,21 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
 
 def resolve_browser_import_source(browser: str, profile_dir: str | None = None) -> PersonaImportSource:
     browser = browser.lower().strip()
+    if host_profiles := os.environ.get("ARCHIVEBOX_HOST_BROWSER_PROFILES"):
+        label = BROWSER_LABELS.get(browser, browser)
+        exported = Path(host_profiles) / f"{label} - {profile_dir or 'Default'}"
+        if exported.is_dir():
+            return resolve_custom_import_source(str(exported))
     if browser not in BROWSER_PROFILE_FINDERS:
         supported = ", ".join(BROWSER_PROFILE_FINDERS)
         raise ValueError(f"Unknown browser: {browser}. Supported browsers: {supported}")
 
     user_data_dir = BROWSER_PROFILE_FINDERS[browser]()
     if not user_data_dir:
-        raise ValueError(f"Could not find {browser} profile directory")
+        raise ValueError(
+            f"Could not find {browser} profile directory. Use --source /path/to/browser-data. "
+            "For Docker on macOS/Windows, import on the host into the shared data directory first.",
+        )
 
     chosen_profile = profile_dir or pick_default_profile_dir(user_data_dir)
     if not chosen_profile:
@@ -444,6 +483,8 @@ def resolve_browser_profile_source(
         resolved_root = resolved_root.resolve()
     if not resolved_root.exists():
         raise ValueError(f"Profile root does not exist: {resolved_root}")
+    if Path(profile_dir).name != profile_dir or profile_dir in {".", ".."}:
+        raise ValueError("Profile must be a directory name within the source browser root.")
     if not profile_dir.strip():
         raise ValueError("Profile directory name cannot be empty.")
 
@@ -476,6 +517,17 @@ def resolve_custom_import_source(raw_value: str, profile_dir: str | None = None)
         raise ValueError(f"Custom browser path does not exist: {source_path}")
 
     explicit_profile = profile_dir.strip() if profile_dir else ""
+    for profile_root in PERSONA_PROFILE_DIR_CANDIDATES:
+        root = source_path / profile_root
+        if root.is_dir():
+            selected = explicit_profile or pick_default_profile_dir(root)
+            if selected:
+                return resolve_browser_profile_source(
+                    browser="persona",
+                    source_name=source_path.name,
+                    user_data_dir=root.resolve(),
+                    profile_dir=selected,
+                )
     if _looks_like_profile_dir(source_path):
         if explicit_profile and explicit_profile != source_path.name:
             raise ValueError("Profile name does not match the provided profile directory path.")
@@ -508,6 +560,48 @@ def pick_default_profile_dir(user_data_dir: Path) -> str | None:
     return profiles[0]
 
 
+def resolve_source_browser_binary(source: PersonaImportSource) -> str:
+    """Use the originating browser so OS-protected cookies use the correct keychain."""
+    if source.browser_binary:
+        return str(Path(source.browser_binary).expanduser().resolve())
+    names = {
+        "chrome": ("Google Chrome", "google-chrome"),
+        "chrome-beta": ("Google Chrome Beta", "google-chrome-beta"),
+        "chromium": ("Chromium", "chromium"),
+        "brave": ("Brave Browser", "brave-browser"),
+        "edge": ("Microsoft Edge", "microsoft-edge"),
+    }
+    app, executable = names.get(source.browser, ("", ""))
+    candidates = []
+    if platform.system() == "Darwin" and app:
+        candidates = [
+            Path("/Applications") / f"{app}.app/Contents/MacOS/{app}",
+            Path.home() / f"Applications/{app}.app/Contents/MacOS/{app}",
+        ]
+    elif platform.system() == "Windows":
+        relative = {
+            "chrome": "Google/Chrome/Application/chrome.exe",
+            "chrome-beta": "Google/Chrome Beta/Application/chrome.exe",
+            "edge": "Microsoft/Edge/Application/msedge.exe",
+            "brave": "BraveSoftware/Brave-Browser/Application/brave.exe",
+            "chromium": "Chromium/Application/chrome.exe",
+        }.get(source.browser)
+        if relative:
+            candidates = [
+                Path(os.environ[root]) / relative for root in ("LOCALAPPDATA", "PROGRAMFILES", "PROGRAMFILES(X86)") if os.environ.get(root)
+            ]
+    for candidate in candidates:
+        if candidate.is_file() and os.access(candidate, os.X_OK):
+            return str(candidate)
+    if executable and (found := shutil.which(executable)):
+        return found
+    raise ValueError(
+        f"Source {source.browser} browser executable not found. Use --browser-binary with the originating Chromium browser. "
+        "For Docker, run persona create --import on the browser's host into the shared data directory first: "
+        "the container cannot decrypt cookies using the host's macOS Keychain or Windows credentials.",
+    )
+
+
 def import_persona_from_source(
     persona: Persona,
     source: PersonaImportSource,
@@ -516,67 +610,182 @@ def import_persona_from_source(
     import_cookies: bool = True,
     capture_storage: bool = False,
 ) -> PersonaImportResult:
+    # Stage all work before replacing an existing, working identity.
     persona.ensure_dirs()
     result = PersonaImportResult(source=source)
+    expected_cookies = 0
+    browser_binary = None
+    native_cookie_browsers = {"chrome", "chromium", "brave", "edge", "vivaldi", "opera", "opera_gx"}
+    # browser-cookie3 needs a desktop D-Bus session on Linux. In a headless
+    # environment, let the originating browser decode its own profile instead.
+    has_native_cookie_decoder = platform.system() != "Linux" or bool(os.environ.get("DBUS_SESSION_BUS_ADDRESS"))
+    native_cookie_import = source.kind == "browser-profile" and source.browser in native_cookie_browsers and has_native_cookie_decoder
+    portable_import = source.kind == "browser-profile" and source.browser == "persona"
+    if source.kind == "browser-profile" and not portable_import and (import_cookies or capture_storage):
+        if not native_cookie_import:
+            browser_binary = resolve_source_browser_binary(source)
+        for cookie_db in (source.profile_path / "Network" / "Cookies", source.profile_path / "Cookies"):
+            if cookie_db.is_file():
+                with sqlite3.connect(f"{cookie_db.as_uri()}?mode=ro", uri=True) as conn:
+                    expected_cookies = conn.execute(
+                        "SELECT COUNT(*) FROM cookies WHERE expires_utc = 0 OR expires_utc > ?",
+                        (int((time.time() + 11644473600) * 1_000_000),),
+                    ).fetchone()[0]
+                break
+    with tempfile.TemporaryDirectory(prefix=".import-", dir=persona.path.parent) as tmp:
+        stage = Path(tmp)
+        staged_profile = stage / "chrome_profile"
+        if source.kind == "browser-profile":
+            assert source.user_data_dir and source.profile_path
+            if copy_profile or import_cookies or capture_storage:
+                staged_profile.mkdir()
+                # Only the selected profile belongs to this persona. Normalize its
+                # name so downstream Chromium launches always select it.
+                copy_browser_user_data_dir(source.profile_path, staged_profile / "Default")
+                local_state = source.user_data_dir / "Local State"
+                if local_state.exists():
+                    state = json.loads(local_state.read_text())
+                    profile_state = state.setdefault("profile", {})
+                    profile_state["last_used"] = "Default"
+                    profile_state["last_active_profiles"] = ["Default"]
+                    info = profile_state.get("info_cache", {}).get(source.profile_dir)
+                    profile_state["info_cache"] = {"Default": info} if info else {}
+                    (staged_profile / "Local State").write_text(json.dumps(state))
+                persona.cleanup_chrome_profile(staged_profile)
+                result.profile_copied = copy_profile
+        elif copy_profile:
+            result.warnings.append(
+                "CDP imports capture cookies and open-tab storage, not browser settings. Use --source for a full profile import.",
+            )
 
-    persona_chrome_dir = Path(persona.CHROME_USER_DATA_DIR)
-    cookies_file = persona.path / "cookies.txt"
-    auth_file = persona.path / "auth.json"
-
-    launch_user_data_dir: Path | None = None
-
-    if source.kind == "browser-profile":
-        if copy_profile and source.user_data_dir:
-            resolved_source_root = source.user_data_dir.resolve()
-            resolved_persona_root = persona_chrome_dir.resolve()
-            if resolved_source_root == resolved_persona_root:
-                result.warnings.append(
-                    "Skipped profile copy because the selected source is already this persona's chrome_profile directory.",
-                )
+        if import_cookies or capture_storage:
+            if portable_import:
+                export_dir = source.user_data_dir.parent
+                if not all((export_dir / name).is_file() for name in ("auth.json", "cookies.txt")):
+                    raise ValueError("This persona has no portable cookie export. Re-import it on the original browser host first.")
+                auth_payload = json.loads((export_dir / "auth.json").read_text())
+                if not isinstance(auth_payload, dict) or not isinstance(auth_payload.get("cookies"), list):
+                    raise ValueError("Invalid persona auth.json: expected a cookies list.")
+                for name in ("auth.json", "cookies.txt"):
+                    shutil.copy2(export_dir / name, stage / name)
+                success, message = True, ""
+            elif native_cookie_import:
+                auth_payload = export_profile_cookies(source, staged_profile, stage)
+                success, message = True, ""
             else:
-                copy_browser_user_data_dir(resolved_source_root, resolved_persona_root)
-                persona.cleanup_chrome_profile(resolved_persona_root)
-                result.profile_copied = True
-            launch_user_data_dir = resolved_persona_root
-        else:
-            launch_user_data_dir = source.user_data_dir
-    elif copy_profile:
-        result.warnings.append(
-            "Profile copying is only available for local Chromium profile paths. CDP imports can only pull cookies and open-tab storage.",
-        )
-
-    if not import_cookies and not capture_storage:
-        return result
-
-    if source.kind == "cdp":
-        export_success, auth_payload, export_message = export_browser_state(
-            cdp_url=source.cdp_url,
-            cookies_output_file=cookies_file if import_cookies else None,
-            auth_output_file=auth_file if capture_storage else None,
-        )
-    else:
-        export_success, auth_payload, export_message = export_browser_state(
-            user_data_dir=launch_user_data_dir,
-            profile_dir=source.profile_dir,
-            chrome_binary=source.browser_binary,
-            cookies_output_file=cookies_file if import_cookies else None,
-            auth_output_file=auth_file if capture_storage else None,
-        )
-
-    if not export_success:
-        result.warnings.append(export_message or "Browser import failed.")
-        return result
-
-    if import_cookies and cookies_file.exists():
-        result.cookies_imported = True
-    if capture_storage and auth_file.exists():
-        result.storage_captured = True
-    if _apply_imported_user_agent(persona, auth_payload):
-        result.user_agent_imported = True
+                # Custom browsers and headless Linux use the originating browser.
+                # Desktop imports use their OS decoder without launching or probing.
+                launch_profile = stage / "export_profile"
+                if source.kind == "browser-profile":
+                    copy_browser_user_data_dir(staged_profile, launch_profile)
+                success, auth_payload, message = export_browser_state(
+                    user_data_dir=launch_profile if source.kind == "browser-profile" else None,
+                    cdp_url=source.cdp_url,
+                    profile_dir="Default" if source.kind == "browser-profile" else None,
+                    chrome_binary=browser_binary,
+                    cookies_output_file=stage / "cookies.txt" if import_cookies else None,
+                    auth_output_file=stage / "auth.json",
+                )
+            if not success:
+                raise ValueError(message or "Browser state export failed; persona was not replaced.")
+            if expected_cookies and not (auth_payload or {}).get("cookies"):
+                raise ValueError(
+                    "The source has cookies but the browser exported none. Run the import on the source host "
+                    "with the original browser and unlock its OS keychain; the existing persona was not replaced.",
+                )
+            if import_cookies:
+                result.cookies_imported = True
+                result.cookie_count = len((auth_payload or {}).get("cookies", []))
+            if capture_storage:
+                result.storage_captured = True
+            # Keep full CDP cookie attributes (sameSite, httpOnly, partition keys)
+            # as well as cookies.txt for non-browser extractors.
+            result.user_agent_imported = _apply_imported_user_agent(persona, auth_payload)
 
+        if result.profile_copied:
+            target = Path(persona.CHROME_USER_DATA_DIR)
+            if target.exists():
+                shutil.rmtree(target)
+            staged_profile.rename(target)
+        for filename in ("cookies.txt", "auth.json"):
+            exported = stage / filename
+            if exported.exists():
+                exported.chmod(0o600)
+                exported.replace(persona.path / filename)
     return result
 
 
+def export_profile_cookies(source: PersonaImportSource, staged_profile: Path, output: Path) -> dict:
+    """Decode with the host keychain, never by starting the user's browser."""
+    import browser_cookie3
+
+    payload = {"TYPE": "auth", "cookies": [], "localStorage": {}, "sessionStorage": {}}
+    jar = MozillaCookieJar(str(output / "cookies.txt"))
+    assert source.profile_path
+    for relative in (Path("Network/Cookies"), Path("Cookies")):
+        original = source.profile_path / relative
+        if not original.is_file():
+            continue
+        database = staged_profile / "Default" / relative
+        for suffix in ("", "-wal", "-shm"):
+            database.with_name(database.name + suffix).unlink(missing_ok=True)
+        # SQLite backup includes committed WAL data even while the browser is open.
+        with sqlite3.connect(f"{original.as_uri()}?mode=ro", uri=True) as reader:
+            with sqlite3.connect(database) as writer:
+                reader.backup(writer)
+        try:
+            cookies = getattr(browser_cookie3, source.browser)(
+                cookie_file=str(database),
+                key_file=str(staged_profile / "Local State"),
+            )
+        except Exception as err:
+            raise ValueError(
+                "Could not decrypt browser cookies. Run the import as your desktop user on the browser host "
+                "with its OS keychain unlocked. Mounting macOS/Windows browser files into Linux does not provide "
+                "the decryption keys. The existing persona was not replaced.",
+            ) from err
+        with sqlite3.connect(database) as connection:
+            connection.row_factory = sqlite3.Row
+            columns = {row[1] for row in connection.execute("PRAGMA table_info(cookies)")}
+            metadata_columns = [
+                name
+                for name in ("host_key", "path", "name", "samesite", "top_frame_site_key", "has_cross_site_ancestor")
+                if name in columns
+            ]
+            rows = {
+                (row["host_key"], row["path"], row["name"]): dict(row)
+                for row in connection.execute("SELECT " + ", ".join(metadata_columns) + " FROM cookies")
+            }
+        for cookie in cookies:
+            if cookie.is_expired():
+                continue
+            jar.set_cookie(cookie)
+            item = {
+                "name": cookie.name,
+                "value": cookie.value,
+                "domain": cookie.domain,
+                "path": cookie.path,
+                "secure": bool(cookie.secure),
+                "httpOnly": cookie.has_nonstandard_attr("HTTPOnly"),
+            }
+            if cookie.expires:
+                item["expires"] = cookie.expires
+            row = rows.get((cookie.domain, cookie.path, cookie.name), {})
+            same_site = {0: "None", 1: "Lax", 2: "Strict"}.get(row.get("samesite"))
+            if same_site:
+                item["sameSite"] = same_site
+            if row.get("top_frame_site_key"):
+                item["partitionKey"] = {
+                    "topLevelSite": row["top_frame_site_key"],
+                    "hasCrossSiteAncestor": bool(row.get("has_cross_site_ancestor", False)),
+                }
+            payload["cookies"].append(item)
+        break
+    jar.save(ignore_discard=True, ignore_expires=False)
+    (output / "auth.json").write_text(json.dumps(payload) + "\n")
+    return payload
+
+
 def copy_browser_user_data_dir(source_dir: Path, destination_dir: Path) -> None:
     destination_dir.parent.mkdir(parents=True, exist_ok=True)
     shutil.rmtree(destination_dir, ignore_errors=True)
@@ -611,21 +820,27 @@ def export_browser_state(
     chrome_config = chrome_plugin_dir / "chrome" / "config.json"
 
     env = os.environ.copy()
-    if chrome_binary:
-        env["CHROME_BINARY"] = str(chrome_binary)
-    dependency_env = subprocess.run(
-        [
-            str(Path(sys.executable).with_name("abxpkg")),
-            "env",
-            "--install",
-            "--json",
-            f"--lib={get_config().ABXPKG_LIB_DIR}",
-            f"--deps-from={chrome_config}:required_binaries",
-        ],
-        capture_output=True,
-        text=True,
-        env=env,
-    )
+    dependency_config = json.loads(chrome_config.read_text())
+    dependency_config["required_binaries"] = [dep for dep in dependency_config["required_binaries"] if dep["name"] != "{CHROME_BINARY}"]
+    dependency_file = tempfile.NamedTemporaryFile(mode="w", suffix=".json", delete=False)
+    with dependency_file:
+        json.dump(dependency_config, dependency_file)
+    try:
+        dependency_env = subprocess.run(
+            [
+                str(Path(sys.executable).with_name("abxpkg")),
+                "env",
+                "--install",
+                "--json",
+                f"--lib={get_config().ABXPKG_LIB_DIR}",
+                f"--deps-from={dependency_file.name}:required_binaries",
+            ],
+            capture_output=True,
+            text=True,
+            env=env,
+        )
+    finally:
+        Path(dependency_file.name).unlink()
     if dependency_env.returncode != 0:
         return False, None, dependency_env.stderr.strip() or "abxpkg could not resolve browser export dependencies."
     try:
@@ -639,6 +854,9 @@ def export_browser_state(
     if not node_projection.is_symlink() or not os.access(node_projection, os.X_OK):
         return False, None, f"abxpkg did not resolve Node.js into {node_projection}."
     env.update({str(key): str(value) for key, value in resolved_env.items()})
+    if chrome_binary:
+        env["CHROME_BINARY"] = chrome_binary
+    env["NODE_MODULES_DIR"] = str(abxpkg_lib_dir / "pnpm" / "packages" / "chrome" / "node_modules")
     env["NODE_BINARY"] = str(node_projection)
     env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(chrome_plugin_dir)
 
@@ -865,7 +1083,7 @@ def _apply_imported_user_agent(persona: Persona, auth_payload: dict | None) -> b
     if not auth_payload:
         return False
 
-    user_agent = str(auth_payload.get("user_agent") or "").strip()
+    user_agent = str(auth_payload.get("user_agent") or "").strip().replace("HeadlessChrome/", "Chrome/")
     if not user_agent:
         return False
 

archivebox/personas/models.py

@@ -45,17 +45,23 @@ VOLATILE_PROFILE_DIR_NAMES = {
     "Code Cache",
     "GPUCache",
     "ShaderCache",
-    "Service Worker",
-    "GCM Store",
     "Crashpad",
     "BrowserMetrics",
+    "Sessions",
+    "Sessions_Encrypted",
 }
 
 VOLATILE_PROFILE_FILE_NAMES = {
     "BrowserMetrics-spare.pma",
+    "RunningChromeVersion",
+    "DevToolsActivePort",
     "SingletonCookie",
     "SingletonLock",
     "SingletonSocket",
+    "Current Session",
+    "Current Tabs",
+    "Last Session",
+    "Last Tabs",
 }
 
 
@@ -211,7 +217,7 @@ class Persona(ModelWithConfig):
                 cleaned = True
 
         for path in profile_paths():
-            if not path.match("*.log"):
+            if path.name not in {"chrome.log", "chrome_debug.log"}:
                 continue
             try:
                 path.unlink()
@@ -297,6 +303,10 @@ class Persona(ModelWithConfig):
             else:
                 runtime_profile_dir.mkdir(parents=True, exist_ok=True)
 
+            for filename in ("cookies.txt", "auth.json"):
+                source = self.path / filename
+                if source.is_file():
+                    shutil.copy2(source, runtime_root / filename)
             runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
             self.cleanup_chrome_profile(runtime_profile_dir)
 
@@ -313,6 +323,11 @@ class Persona(ModelWithConfig):
             # derivation centralized in the Chrome plugin helpers.
             "PERSONAS_DIR": str(runtime_root.parent),
             "ACTIVE_PERSONA": self.name,
+            **{
+                key: str(runtime_root / filename)
+                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
+                if (runtime_root / filename).is_file()
+            },
         }
 
     def prepare_runtime_for_snapshot(self, snapshot, chrome_binary: str = "") -> dict[str, str]:
@@ -329,6 +344,12 @@ class Persona(ModelWithConfig):
         else:
             runtime_profile_dir.mkdir(parents=True, exist_ok=True)
 
+        for filename in ("cookies.txt", "auth.json"):
+            source = self.runtime_root_for_crawl(snapshot.crawl) / filename
+            if not source.is_file():
+                source = self.path / filename
+            if source.is_file():
+                shutil.copy2(source, runtime_root / filename)
         runtime_downloads_dir.mkdir(parents=True, exist_ok=True)
         self.cleanup_chrome_profile(runtime_profile_dir)
 
@@ -343,6 +364,11 @@ class Persona(ModelWithConfig):
             # Chrome hooks and ArchiveBox-driven hooks resolve paths the same way.
             "PERSONAS_DIR": str(runtime_root.parent),
             "ACTIVE_PERSONA": self.name,
+            **{
+                key: str(runtime_root / filename)
+                for key, filename in (("COOKIES_FILE", "cookies.txt"), ("AUTH_STORAGE_FILE", "auth.json"))
+                if (runtime_root / filename).is_file()
+            },
         }
 
     def cleanup_runtime_for_crawl(self, crawl) -> None:

archivebox/personas/open_browser.js

@@ -0,0 +1,61 @@
+#!/usr/bin/env node
+
+const fs = require('fs');
+const path = require('path');
+const pluginsDir = process.env.ARCHIVEBOX_ABX_PLUGINS_DIR;
+const {ensureNodeModuleResolution} = require(path.join(pluginsDir, 'base', 'utils.js'));
+ensureNodeModuleResolution(module);
+const installShutdownHandler = require(path.join(pluginsDir, 'base', 'daemon_lifecycle.js')).captureShutdownSignals();
+const {
+    acquireSessionLock,
+    ensureChromeSession,
+    getChromeSessionOptionsFromConfig,
+    getChromeLaunchPrerequisites,
+    connectToBrowserEndpoint,
+    closeBrowserInChromeSession,
+} = require(path.join(pluginsDir, 'chrome', 'chrome_utils.js'));
+
+async function main() {
+    const config = JSON.parse(fs.readFileSync(0, 'utf8'));
+    const outputDir = path.join(path.dirname(config.CHROME_USER_DATA_DIR), '.browser');
+    fs.mkdirSync(outputDir, {recursive: true});
+    const release = await acquireSessionLock(path.join(outputDir, 'launch.lock'));
+    let session;
+    const {puppeteer, binary} = getChromeLaunchPrerequisites();
+    try {
+        session = await ensureChromeSession({
+            ...getChromeSessionOptionsFromConfig(config),
+            outputDir, puppeteer, binary,
+        });
+    } finally {
+        release();
+    }
+    const browser = await connectToBrowserEndpoint(puppeteer, session.cdpUrl, {defaultViewport: null});
+    installShutdownHandler(async () => {
+        if (!session.reusedExisting) {
+            await closeBrowserInChromeSession({
+                outputDir, puppeteer, cdpUrl: session.cdpUrl, pid: session.pid, processIsLocal: true,
+            });
+        }
+        await browser.disconnect();
+        process.exit(0);
+    });
+    const pages = await browser.pages();
+    const page = pages[0] || await browser.newPage();
+    const windowSession = await page.createCDPSession();
+    const {windowId} = await windowSession.send('Browser.getWindowForTarget');
+    await windowSession.send('Browser.setWindowBounds', {windowId, bounds: {windowState: 'maximized'}});
+    await windowSession.detach();
+    await page.bringToFront();
+    console.log(`Browser ready for persona ${config.ACTIVE_PERSONA || 'Default'}`);
+    if (session.reusedExisting) {
+        await browser.disconnect();
+        return;
+    }
+    await new Promise(resolve => browser.once('disconnected', resolve));
+}
+
+main().catch(error => {
+    console.error(error.message);
+    process.exit(1);
+});

archivebox/plugins/discovery.py

@@ -55,10 +55,6 @@ def get_plugins() -> list[str]:
     return sorted(get_plugin_catalog())
 
 
-def get_plugin_models():
-    return get_plugin_catalog().plugins
-
-
 def get_plugin_name(plugin: str) -> str:
     """
     Get the base plugin name without numeric prefix.

archivebox/plugins/forms.py

@@ -276,7 +276,7 @@ class PluginConfigFormMixin:
         elif is_sensitive:
             input_type = "password"
         else:
-            input_value = "" if is_sensitive else str(current_value)
+            input_value = "" if current_value is None else str(current_value)
 
         return {
             "key": config_key,
@@ -290,7 +290,7 @@ class PluginConfigFormMixin:
             "default": _jsonish(default_display),
             "current": "configured"
             if is_sensitive and current_value
-            else (str(current_value) if "string" in schema_types else _jsonish(current_value)),
+            else (("" if current_value is None else str(current_value)) if "string" in schema_types else _jsonish(current_value)),
             "current_url": self.plugin_config_binary_urls.get(config_key, "") if str(config_key).endswith("_BINARY") else "",
             "is_sensitive": is_sensitive,
             "minimum": prop_schema.get("minimum"),
@@ -342,6 +342,8 @@ class PluginConfigFormMixin:
                     continue
 
                 base_value = effective_config.get(config_key, prop_schema.get("default", ""))
+                if base_value is None and coerced_value == "":
+                    continue
                 if _same_config_value(coerced_value, base_value):
                     continue
 

archivebox/progressmonitor/collection.py

@@ -0,0 +1,54 @@
+"""Non-blocking, database-only collection totals for the progress monitor."""
+
+import logging
+from threading import Lock, Thread
+from time import time
+
+from django.core.cache import cache
+from django.db import close_old_connections, connections
+from django.db.models import Count, Sum
+
+
+_refresh_lock = Lock()
+
+
+def collection_summary(user):
+    user_id = None if user.is_superuser else user.pk
+    key = f"progress-collection:{'all' if user_id is None else user_id}"
+    summary = cache.get(key)
+    # Even an indexed SUM scans the index. Never make an HTTP request wait for
+    # that scan on a large collection. Serve stale totals while one worker refreshes;
+    # a single non-blocking lock prevents a queue of scans from concurrent clients.
+    if (summary is None or time() - summary["sampled_at"] >= 60) and _refresh_lock.acquire(blocking=False):
+        # Throttle failures too: a broken/locked database must not trigger work
+        # and log output on every sidebar poll. This cache is local to each worker.
+        if not cache.add(f"{key}:refresh", True, timeout=60):
+            _refresh_lock.release()
+            return summary
+
+        def refresh():
+            try:
+                close_old_connections()
+                from archivebox.core.models import Snapshot
+
+                snapshots = Snapshot.objects.all()
+                if user_id is not None:
+                    snapshots = snapshots.filter(crawl__created_by_id=user_id)
+                # COUNT(*) + SUM(output_size) can use Snapshot's covering size index.
+                # Snapshot.output_size is maintained when ArchiveResult sizes change.
+                totals = snapshots.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+                totals["bytes"] = totals["bytes"] or 0
+                totals["sampled_at"] = time()
+                cache.set(key, totals, timeout=3600)
+            except Exception:
+                logging.getLogger(__name__).exception("Could not refresh collection totals")
+            finally:
+                connections.close_all()
+                _refresh_lock.release()
+
+        try:
+            Thread(target=refresh, name="collection-summary", daemon=True).start()
+        except RuntimeError:
+            _refresh_lock.release()
+            logging.getLogger(__name__).exception("Could not start collection totals refresh")
+    return summary

archivebox/progressmonitor/views.py

@@ -942,6 +942,10 @@ def live_progress_view(request):
             "queued_crawls_hidden": queued_crawls_hidden,
             "server_time": timezone.now().isoformat(),
         }
+        if is_admin and not snapshot_id_filter and not crawl_id_filter and request.GET.get("collection") == "1":
+            from archivebox.progressmonitor.collection import collection_summary
+
+            payload["collection"] = collection_summary(request.user)
         try:
             import ujson
 

archivebox/search/query.py

@@ -1,5 +1,6 @@
 __package__ = "archivebox.search"
 
+from subprocess import CalledProcessError
 from typing import Any
 
 from django.db import connection
@@ -220,6 +221,7 @@ def iter_query_search_ids(
     search_mode: str | None = None,
     config: dict[str, Any] | None = None,
     max_results: int | None = None,
+    stop_event=None,
     **config_kwargs: Any,
 ):
     """Yield snapshot IDs from configured search backend modules."""
@@ -266,6 +268,8 @@ def iter_query_search_ids(
     seen: set[str] = set()
     try:
         for backend_name in backend_names:
+            if stop_event is not None and stop_event.is_set():
+                return
             backend = backends[backend_name]
             try:
                 if backend_name == "sonic":
@@ -291,29 +295,34 @@ def iter_query_search_ids(
                     env=search_backend_command_env(config=config),
                     cwd=CONSTANTS.DATA_DIR,
                     timeout=max(1, int(config.get("TIMEOUT", 60))) * 4,
+                    stop_event=stop_event,
                 )
-                for snapshot_id in ids:
-                    if snapshot_id in seen:
-                        continue
-                    seen.add(snapshot_id)
-                    yield snapshot_id
-                    if max_results and len(seen) >= max_results:
-                        return
+                try:
+                    for snapshot_id in ids:
+                        if snapshot_id in seen:
+                            continue
+                        seen.add(snapshot_id)
+                        yield snapshot_id
+                        if max_results and len(seen) >= max_results:
+                            return
+                finally:
+                    ids.close()
                 successful_backends += 1
             except Exception as err:
                 errors.append(err)
                 if not fallback_enabled:
                     raise
+        if not successful_backends and errors and search_mode_base == "deep":
+            raise errors[0]
     except Exception as err:
         stderr()
         stderr(
             f"[X] The search backend threw an exception={err}:",
             color="red",
         )
+        if isinstance(err, CalledProcessError) and err.stderr:
+            stderr(err.stderr.rstrip(), color="red")
         raise
-    else:
-        if not successful_backends and errors and search_mode_base == "deep":
-            raise errors[0]
 
 
 @enforce_types

archivebox/search/views.py

@@ -6,7 +6,7 @@ import json
 import threading
 import time
 from copy import copy
-from queue import Full, Queue
+from queue import Empty, Full, Queue
 from urllib.parse import urlsplit
 from uuid import UUID
 
@@ -254,15 +254,18 @@ def iter_filtered_search_result_ids(iterator, queryset, *, flush_max_delay=0.05)
         yield from flush_batch()
 
 
-def iter_search_result_ids(query, base_queryset, *, search_mode, config):
+def iter_search_result_ids(query, base_queryset, *, search_mode, config, stop_event=None):
     """Yield filtered Snapshot IDs from the selected search provider."""
     search_mode_base = get_search_mode_base(search_mode, config=config)
     provider = (
         iter_meta_search_ids(query, base_queryset)
         if search_mode_base == "meta"
-        else iter_query_search_ids(query, search_mode=search_mode, config=config)
+        else iter_query_search_ids(query, search_mode=search_mode, config=config, stop_event=stop_event)
     )
-    yield from iter_filtered_search_result_ids(provider, base_queryset)
+    try:
+        yield from iter_filtered_search_result_ids(provider, base_queryset)
+    finally:
+        provider.close()
 
 
 def snapshot_search_stream_response(query, base_queryset, *, search_mode, config, cache_key, thread_name):
@@ -300,7 +303,7 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
             iterator = None
             try:
                 close_old_connections()
-                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config)
+                iterator = iter_search_result_ids(query, base_queryset, search_mode=search_mode, config=config, stop_event=stop_event)
                 for snapshot_id in iterator:
                     if stop_event.is_set():
                         break
@@ -326,10 +329,13 @@ def snapshot_search_stream_response(query, base_queryset, *, search_mode, config
                 emit(None)
 
         threading.Thread(target=run_search, name=thread_name, daemon=True).start()
-        yield f"0{stream_padding}\n"
         try:
+            yield f"0{stream_padding}\n"
             while True:
-                item = await asyncio.to_thread(queue.get)
+                try:
+                    item = await asyncio.to_thread(queue.get, True, 0.1)
+                except Empty:
+                    continue
                 if item is None:
                     break
                 if isinstance(item, BaseException):

archivebox/services/resource_admission.py

@@ -0,0 +1,91 @@
+"""Admission signals, not estimates of how much RAM an extractor will need."""
+
+import math
+import os
+import time
+from pathlib import Path
+
+
+RESOURCE_RECHECK_SECONDS = 2.0
+# Scheduling policy, not a minimum-RAM claim: stop increasing extraction
+# load when all non-idle work loses at least one second out of ten to memory
+# stalls. Require both recent and rolling pressure so transient reclaim does
+# not indefinitely starve otherwise viable work on small hosts.
+MEMORY_FULL_STALL_PERCENT = 10.0
+
+
+class ResourceAdmission:
+    def __init__(self):
+        self._memory_stalls: dict[str, tuple[int, float]] = {}
+        self._checked_at = float("-inf")
+        self._reason: str | None = None
+
+    @staticmethod
+    def cgroup_paths() -> list[Path]:
+        root = Path("/sys/fs/cgroup")
+        paths = [root]
+        try:
+            for line in Path("/proc/self/cgroup").read_text().splitlines():
+                if line.startswith("0::"):
+                    current = root / line[3:].lstrip("/")
+                    if current.is_dir() and current.is_relative_to(root):
+                        paths = [current, *[parent for parent in current.parents if parent.is_relative_to(root)]]
+                    break
+        except OSError:
+            pass
+        return paths
+
+    def snapshot_limit(self, configured: int) -> int:
+        cpus = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else (os.cpu_count() or 1)
+        for path in self.cgroup_paths():
+            try:
+                quota, period = (path / "cpu.max").read_text().split()
+                if quota != "max":
+                    cpus = min(cpus, max(1, math.ceil(int(quota) / int(period))))
+            except (OSError, ValueError, ZeroDivisionError):
+                continue
+        # CPU quota limits parallel admission, not forward progress. A host
+        # with a fractional CPU must still be able to run one snapshot.
+        return max(1, min(configured, cpus))
+
+    def observe_memory_stalls(self, source: str, total: int, avg10: float, sampled_at: float) -> bool:
+        previous = self._memory_stalls.get(source)
+        self._memory_stalls[source] = (total, sampled_at)
+        if previous is None:
+            return avg10 >= MEMORY_FULL_STALL_PERCENT
+        previous_total, previous_time = previous
+        elapsed = sampled_at - previous_time
+        if elapsed <= 0 or total < previous_total:
+            return False
+        stalled_percent = (total - previous_total) / (elapsed * 1_000_000) * 100
+        return avg10 >= MEMORY_FULL_STALL_PERCENT and stalled_percent >= MEMORY_FULL_STALL_PERCENT
+
+    def memory_pressure(self) -> str | None:
+        now = time.monotonic()
+        if now - self._checked_at < RESOURCE_RECHECK_SECONDS:
+            return self._reason
+        self._checked_at = now
+        reasons = []
+        cgroups = self.cgroup_paths()
+        for path in cgroups:
+            try:
+                current = int((path / "memory.current").read_text())
+                for name in ("memory.high", "memory.max"):
+                    limit = (path / name).read_text().strip()
+                    if limit != "max" and current >= int(limit):
+                        reasons.append(f"{name} reached")
+            except (OSError, ValueError):
+                continue
+        for path in [Path("/proc/pressure/memory"), *[p / "memory.pressure" for p in cgroups]]:
+            try:
+                full = next(line for line in path.read_text().splitlines() if line.startswith("full "))
+                fields = dict(field.split("=", 1) for field in full.split()[1:])
+                if self.observe_memory_stalls(str(path), int(fields["total"]), float(fields["avg10"]), now):
+                    reasons.append("memory stalls")
+            except (OSError, ValueError, KeyError, StopIteration):
+                continue
+        self._reason = ", ".join(dict.fromkeys(reasons)) or None
+        return self._reason
+
+
+resource_admission = ResourceAdmission()

archivebox/services/runner.py

@@ -58,6 +58,8 @@ from abxbus import BaseEvent
 from abxbus.event_bus import EventBus, get_current_event, in_handler_context
 from abxbus.event_handler import EventHandlerAbortedError, EventHandlerCancelledError
 
+from archivebox.services.resource_admission import RESOURCE_RECHECK_SECONDS, resource_admission
+
 from archivebox.config.common import (
     ArchiveBoxBaseConfig,
     normalize_runtime_config,
@@ -221,6 +223,8 @@ class CrawlRunner:
         self.snapshot_tasks: dict[str, asyncio.Task[None]] = {}
         self.snapshot_semaphore = asyncio.Semaphore(1)
         self.max_concurrent_snapshots = 1
+        self._resource_deferred = False
+        self._initial_snapshot_admitted = False
         self.persona = None
         self.base_config: ArchiveBoxBaseConfig | dict[str, Any] = {}
         self.derived_config: dict[str, Any] = {}
@@ -294,6 +298,7 @@ class CrawlRunner:
     async def run(self) -> None:
         root_snapshot_id: str | None = None
         bus_destroyed = False
+        run_state_loaded = False
         try:
             first_signal_message = (
                 "\n[🛑] Got {signal_name}, aborting the active hook...\n"
@@ -313,8 +318,15 @@ class CrawlRunner:
                 on_signal=self._request_abort_from_signal,
                 raise_on_first_signal=False,
             ):
+                if await sync_to_async(defer_crawl_for_resources, thread_sensitive=True)(self.crawl):
+                    return
+                # Startup and its first snapshot are one admission. Browser
+                # startup can itself cause reclaim; checking again immediately
+                # would tear it down and repeat startup without doing any work.
+                self._initial_snapshot_admitted = True
+                run_state_loaded = True
                 snapshot_ids = await sync_to_async(self.load_run_state, thread_sensitive=True)()
-                max_concurrent_snapshots = max(1, int(self.base_config.get("CRAWL_MAX_CONCURRENT_SNAPSHOTS", 1)))
+                max_concurrent_snapshots = resource_admission.snapshot_limit(int(self.base_config.get("CRAWL_MAX_CONCURRENT_SNAPSHOTS", 1)))
                 self.max_concurrent_snapshots = max_concurrent_snapshots
                 self.snapshot_semaphore = asyncio.Semaphore(max_concurrent_snapshots)
                 live_ui = self._create_live_ui()
@@ -344,10 +356,19 @@ class CrawlRunner:
                 except Exception:
                     pass
                 self._live_stream = None
-            await sync_to_async(project_abxpkg_derived_cache_to_db, thread_sensitive=True)(self.base_config.get("ABXPKG_LIB_DIR"))
-            await sync_to_async(self.finalize_run_state, thread_sensitive=True)()
+            if run_state_loaded:
+                await sync_to_async(project_abxpkg_derived_cache_to_db, thread_sensitive=True)(self.base_config.get("ABXPKG_LIB_DIR"))
+                await sync_to_async(self.finalize_run_state, thread_sensitive=True)()
 
     async def enqueue_snapshot(self, snapshot_id: str, crawl_start_event: CrawlStartEvent | None = None) -> None:
+        if self._resource_deferred:
+            return
+        if not self._initial_snapshot_admitted and await sync_to_async(defer_crawl_for_resources, thread_sensitive=True)(
+            self.crawl,
+            excluded_snapshot_ids=list(self.snapshot_tasks),
+        ):
+            self._resource_deferred = True
+            return
         if await self.crawl_is_cancelled():
             return
         if await self.crawl_is_paused() and not self.allow_maintenance_on_inactive_crawl:
@@ -362,6 +383,7 @@ class CrawlRunner:
             return
         else:
             task = asyncio.create_task(self.run_snapshot(snapshot_id), context=_runner_task_context())
+        self._initial_snapshot_admitted = False
         self.snapshot_tasks[snapshot_id] = task
 
     async def stop_snapshot_tasks(self) -> None:
@@ -488,6 +510,8 @@ class CrawlRunner:
         from archivebox.core.models import Snapshot
         from archivebox.config.common import get_config
 
+        if self._resource_deferred:
+            return
         if not isinstance(get_current_event(), CrawlStartEvent):
             return
         if await self.crawl_is_cancelled():
@@ -497,7 +521,7 @@ class CrawlRunner:
 
         await sync_to_async(self.crawl.refresh_from_db, thread_sensitive=True)()
         config = await sync_to_async(lambda: get_config(crawl=self.crawl), thread_sensitive=True)()
-        self.max_concurrent_snapshots = max(1, int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))
+        self.max_concurrent_snapshots = resource_admission.snapshot_limit(int(config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"]))
 
         active_snapshot_ids = [snapshot_id for snapshot_id, task in self.snapshot_tasks.items() if not task.done()]
         available_slots = max(0, self.max_concurrent_snapshots - len(active_snapshot_ids))
@@ -748,10 +772,6 @@ class CrawlRunner:
             runtime_overrides=runtime_chrome_overrides,
             extra_context={
                 "snapshot_id": str(snapshot.id),
-                "snapshot_depth": snapshot.depth,
-                "snapshot_url": snapshot.url,
-                "snapshot_title": snapshot.title or "",
-                "snapshot_tags": tags,
             },
         )
         normalized_config = normalize_runtime_config(config)
@@ -1021,6 +1041,9 @@ class CrawlRunner:
                 return
             derived_config = normalize_runtime_config(self.derived_config)
             output_dir = Path(snapshot["output_dir"])
+            # Hooks read snapshot metadata from the manifest; EXTRA_CONTEXT is
+            # only an opaque record-correlation envelope, never a data input.
+            await sync_to_async(snapshot["_snapshot"].write_index_jsonl, thread_sensitive=True)(output_dir=output_dir)
             plugins = self.catalog.select(snapshot_selected_plugins) if snapshot_selected_plugins else self.catalog
             if self.requested_plugins is not None:
                 _enable_requested_plugins(config, plugins)
@@ -1267,6 +1290,29 @@ def run_snapshot_maintenance(snapshot_id: str, *, output_dir: Path | None = None
     return True
 
 
+def defer_crawl_for_resources(crawl, *, excluded_snapshot_ids=()) -> bool:
+    """Leave extraction durable; never delay cancellation or final cleanup."""
+    from archivebox.core.models import Snapshot
+
+    if crawl.status not in (crawl.StatusChoices.QUEUED, crawl.StatusChoices.STARTED):
+        return False
+    reason = resource_admission.memory_pressure()
+    if reason is None:
+        return False
+    pending = crawl.snapshot_set.filter(status__in=Snapshot.RUNNABLE_STATES).exclude(id__in=excluded_snapshot_ids)
+    if crawl.snapshot_set.exists() and not pending.exists():
+        return False
+    now = timezone.now()
+    retry_at = now + timedelta(seconds=RESOURCE_RECHECK_SECONDS)
+    # Conditional short updates cannot resurrect a concurrent pause/cancel.
+    type(crawl).objects.filter(id=crawl.id, status__in=[crawl.StatusChoices.QUEUED, crawl.StatusChoices.STARTED]).update(
+        retry_at=retry_at,
+        modified_at=now,
+    )
+    pending.filter(retry_at__lte=now).update(retry_at=retry_at, modified_at=now)
+    return True
+
+
 def run_due_crawl(crawl, *, lock_seconds: int, interactive_interrupts: bool = False) -> bool:
     with crawl_lifecycle_lock(str(crawl.id)):
         return _run_due_crawl_locked(

archivebox/templates/admin/base.html

@@ -1339,20 +1339,36 @@
                 display: inline-block;
             }
 
-            .action-rearchive-select,
-            .action-permissions-select {
+            /* The native select receives the actual click, including in WKWebView. */
+            body.change-list #changelist .actions .action-rearchive-select,
+            body.change-list #changelist .actions .action-permissions-select {
                 position: absolute;
-                left: 0;
-                bottom: 0;
-                width: 1px;
-                height: 1px;
+                inset: 0;
+                width: 100%;
+                height: 100%;
                 opacity: 0;
-                pointer-events: none;
-                border: 0;
-                padding: 0;
+                cursor: pointer;
+                z-index: 1;
                 margin: 0;
             }
 
+            body.change-list #changelist .action-rearchive-wrapper > .button,
+            body.change-list #changelist .action-permissions-wrapper > .button {
+                padding: 8px 16px !important;
+                width: max-content;
+                min-width: max-content;
+                flex: 0 0 auto;
+                gap: 6px;
+                white-space: nowrap;
+            }
+
+            .action-rearchive-wrapper:focus-within,
+            .action-permissions-wrapper:focus-within {
+                outline: 2px solid #2563eb;
+                outline-offset: 2px;
+                border-radius: 6px;
+            }
+
             /* Object count */
             .actions .action-counter {
                 color: #64748b;
@@ -1960,22 +1976,10 @@
                             .addClass('action-rearchive-select')
                             .attr('aria-label', 'Re-Archive')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .addClass('button')
                             .html('Re-Archive <span aria-hidden="true">▾</span>')
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         rearchiveActions.forEach(function (action_value) {
@@ -2021,23 +2025,11 @@
                             .addClass('action-permissions-select')
                             .attr('aria-label', 'Permissions')
                             .appendTo(wrapper)
-                        $('<button>')
-                            .attr('type', 'button')
+                        $('<span>')
+                            .attr('aria-hidden', 'true')
                             .attr('name', action_type)
                             .addClass('button')
                             .text(this.text)
-                            .click(function (e) {
-                                e.preventDefault()
-                                e.stopPropagation()
-                                const el = select[0]
-                                if (typeof el.showPicker === 'function') {
-                                    try { el.showPicker() } catch (err) { el.focus(); el.click() }
-                                } else {
-                                    el.focus()
-                                    el.click()
-                                }
-                                return false
-                            })
                             .appendTo(wrapper)
                         $('<option></option>').attr('value', '').attr('hidden', true).appendTo(select)
                         ;[

archivebox/templates/admin/personas/persona/change_form.html

@@ -38,8 +38,8 @@
         border-bottom: 1px dashed #64748b;
     }
 
-    .field-import_mode ul,
-    .field-import_discovered_profile ul {
+    .abx-import-mode,
+    .abx-profile-picker {
         margin: 0;
         padding: 0;
         list-style: none;
@@ -47,20 +47,26 @@
         gap: 12px;
     }
 
-    .field-import_mode ul {
-        grid-template-columns: repeat(auto-fit, minmax(220px, 1fr));
+    .abx-import-mode {
+        grid-template-columns: repeat(auto-fit, minmax(min(220px, 100%), 1fr));
     }
 
-    .field-import_discovered_profile ul {
-        grid-template-columns: repeat(auto-fit, minmax(260px, 1fr));
-        max-height: 460px;
-        overflow: auto;
-        padding-right: 4px;
+    .abx-profile-picker {
+        grid-template-columns: repeat(auto-fit, minmax(min(260px, 100%), 1fr));
     }
 
-    .field-import_mode li,
-    .field-import_discovered_profile li {
+    .abx-import-mode > div,
+    .abx-profile-picker > div {
         margin: 0;
+        min-width: 0;
+    }
+
+    /* Import rows grow with their contents instead of sharing a fixed flex
+       height. Avoid overflow-x:hidden turning overflow-y:visible into auto. */
+    #content-main #persona_form .persona-card-import > .form-row,
+    #content-main #persona_form .persona-card-import fieldset > div {
+        flex: none;
+        overflow: visible;
     }
 
     .field-import_mode label,
@@ -68,7 +74,6 @@
         display: flex;
         gap: 12px;
         align-items: flex-start;
-        min-height: 100%;
         padding: 14px 16px;
         border-radius: 14px;
         border: 1px solid #dbe4ee;
@@ -167,9 +172,14 @@
         color: #991b1b;
     }
 
-    body.model-persona.change-form #content-main form fieldset.persona-card-primary {
-        flex: 1 1 520px !important;
-        max-width: calc(66.66% - 10px) !important;
+    #content-main #persona_form .field-name .flex-container {
+        max-width: 100% !important;
+    }
+
+    #persona_form .field-name input {
+        width: 100%;
+        max-width: none;
+        box-sizing: border-box;
     }
 
     body.model-persona.change-form #content-main form fieldset.persona-card-timestamps {
@@ -257,6 +267,11 @@ document.addEventListener('DOMContentLoaded', function () {
             or by pasting in a CDP url from
             <code class="persona-import-copy" data-copy-text="chrome://inspect#remote-debugging" title="Copy chrome://inspect#remote-debugging">chrome://inspect#remote-debugging</code> (e.g. <code>http://127.0.0.1:9222</code>).
         </p>
+        <p>
+            You can also import cookies into this persona from another browser using the
+            <a href="https://archivebox.github.io/archivebox-browser-extension/" target="_blank" rel="noopener noreferrer">ArchiveBox browser extension</a>.
+            Create the persona here, then select it in the extension's settings to sync cookies from that browser.
+        </p>
     </div>
 </section>
 {{ block.super }}
@@ -264,9 +279,9 @@ document.addEventListener('DOMContentLoaded', function () {
 
 {% block after_field_sets %}
 {{ block.super }}
-<section id="plugin-config" class="module aligned persona-plugin-config">
+<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">
     <h2>Plugin Config</h2>
     <p>These typed controls update the same Persona config JSON shown above. Shared config keys stay synced across plugin sections.</p>
     {% include "plugins/plugin_config_grid.html" with plugin_groups=adminform.form.plugin_groups %}
-</section>
+</fieldset>
 {% endblock %}

archivebox/templates/core/add.html

@@ -1,6 +1,6 @@
 {% extends "core/base.html" %}
 
-{% load static %}
+{% load static core_tags %}
 {% load i18n %}
 
 {% block breadcrumbs %}
@@ -48,9 +48,7 @@
                 <div class="crawl-explanation">
                     <p class="crawl-tip" style="user-select: none;">
                         <span style="float: right">
-                            {% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
-                                <a href="/admin/agent">💬 Crawl with AI</a> &nbsp; | &nbsp;
-                            {% endif %}
+                            {% plugin_ui "opencode" "add" %}
                             <a href="https://github.com/ArchiveBox/archivebox-browser-extension"><img src="{% static 'chrome_extension_icon.png' %}" alt="Chrome Extension" style="height: 33px; margin-top: -5px;">&nbsp; Get the extension</a>
                         </span>
                         💡 <strong>Tip:</strong> Instantly save a single URL by visiting:
@@ -289,7 +287,8 @@
                                 <code>daily</code> - Run once per day<br/>
                                 <code>weekly</code> - Run once per week<br/>
                                 <code>0 */6 * * *</code> - Every 6 hours (cron format)<br/>
-                                <code>0 0 * * 0</code> - Every Sunday at midnight (cron format)
+                                <code>0 0 * * 0</code> - Every Sunday at midnight (cron format)<br/>
+                                <a href="https://crontab.guru/" target="_blank" rel="noopener">Cron Format Helper</a>
                             </div>
                         </div>
 

archivebox/templates/core/navigation.html

@@ -1,4 +1,4 @@
-{% load i18n static %}
+{% load i18n static core_tags %}
 
 <div id="user-tools">
     <a href="{% url 'add' %}" class="navbar-item navbar-add-link">Add ➕</a>
@@ -11,10 +11,7 @@
     <span class="navbar-separator" aria-hidden="true">|</span>
     <a href="/admin/core/tag/" class="navbar-item navbar-tags">Tags</a>
     <span class="navbar-separator navbar-separator-wide" aria-hidden="true"></span>
-    {% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
-        <a href="/admin/agent" class="navbar-item navbar-ai">💬 AI</a>
-        <span class="navbar-separator" aria-hidden="true">|</span>
-    {% endif %}
+    {% plugin_ui "opencode" "navigation" %}
     <a href="{% url 'Docs' %}" class="navbar-item navbar-docs" target="_blank" rel="noopener noreferrer">Docs</a>
     <span class="navbar-separator" aria-hidden="true">|</span>
     <a href="/api/v1/docs" class="navbar-item navbar-api">API</a>

archivebox/templates/core/snapshot.html

@@ -1890,14 +1890,13 @@
 
             function attachPreviewFrameHandlers(frame) {
                 resetPreviewFrameSize(frame)
-                if (frame.src.endsWith('.pdf')) {
+                if (new URL(frame.src || 'about:blank', window.location.href).pathname.endsWith('.pdf')) {
                     frame.removeAttribute('sandbox')
                     frame.src = frame.src + '#toolbar=0'
                 }
                 frame.onload = function() {
-                    if (this.src.includes('.pdf')) {
+                    if (new URL(this.src || 'about:blank', window.location.href).pathname.endsWith('.pdf')) {
                         this.removeAttribute('sandbox')
-                        this.src = this.src.split('?autoplay=')[0] + '#toolbar=0'
                     }
                     tryCenterImageFrame(this)
                     try {
@@ -2083,7 +2082,10 @@
                     nextSrcAbs = new URL(nextSrc, window.location.href).href
                 } catch (err) {}
                 const iframe_elem = ensureMainFrame(currentSrc !== nextSrcAbs)
-                if (target.endsWith('.pdf')) {
+                // Preview wrappers retain the .pdf path but add ?preview=1.
+                // The native PDF renderer must be allowed before navigation,
+                // not after a failed sandboxed load has already completed.
+                if (new URL(target, window.location.href).pathname.endsWith('.pdf')) {
                     iframe_elem.removeAttribute('sandbox')
                 } else {
                     iframe_elem.sandbox = "allow-same-origin allow-top-navigation-by-user-activation allow-scripts allow-forms"

archivebox/templates/core/system_warnings_banner.html

@@ -34,7 +34,7 @@
     {% elif mode == "high_load" %}
       15-min loadavg <code style="background:#000;color:#fff;padding:1px 6px;border-radius:3px;font:700 11px/1.3 ui-monospace,Menlo,Consolas,monospace;text-transform:none;">{{ load_15 }}</code>
       exceeds <code style="background:#000;color:#fff;padding:1px 6px;border-radius:3px;font:700 11px/1.3 ui-monospace,Menlo,Consolas,monospace;text-transform:none;">{{ load_threshold }}</code>
-      ({{ cpu_count }} cores × 3) — the host is saturated; crawls are queuing faster than the runner can finish them.
+      ({{ cpu_count }} cores × 3) — recent CPU or I/O contention may slow crawls. This average can stay elevated after load subsides; it does not measure queue growth.
     {% elif mode == "unsafe" %}
       ArchiveBox single-domain mode — archived pages share an origin with this site
       <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Modes" target="_blank" rel="noopener noreferrer" style="color:#fff;text-decoration:underline;margin-left:6px;">why?</a>

archivebox/tests/conftest.py

@@ -85,7 +85,7 @@ def _set_test_source_pythonpath(env: dict[str, str]) -> None:
         for entry in (env.get("PYTHONPATH") or "").split(os.pathsep)
         if entry and Path(entry).expanduser().is_absolute() and Path(entry).expanduser().exists()
     ]
-    entries = [entry for entry in [*source_pythonpath.split(os.pathsep), *existing_entries] if entry]
+    entries = [entry for entry in [*existing_entries, *source_pythonpath.split(os.pathsep)] if entry]
     if entries:
         env["PYTHONPATH"] = os.pathsep.join(dict.fromkeys(entries))
     else:

archivebox/tests/test_api_v1_auth_browser_session.py

@@ -0,0 +1,56 @@
+from datetime import timedelta
+from urllib.parse import urlsplit
+
+import pytest
+from django.contrib.sessions.models import Session
+from django.utils import timezone
+
+from archivebox.api.models import APIToken
+from archivebox.tests.conftest import API_TEST_HOST, api_client_request
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+@pytest.mark.parametrize("header", ["HTTP_AUTHORIZATION", "HTTP_X_ARCHIVEBOX_API_KEY"])
+def test_browser_session_authenticates_admin_and_expires_with_key(client, api_admin_user, header):
+    token = APIToken.objects.create(created_by=api_admin_user, expires=timezone.now() + timedelta(minutes=5))
+    response = api_client_request(
+        client,
+        "post",
+        "/api/v1/auth/browser_session",
+        payload={},
+        headers={
+            "HTTP_HOST": API_TEST_HOST,
+            header: f"Bearer {token.token}" if header == "HTTP_AUTHORIZATION" else token.token,
+        },
+    )
+    assert response.status_code == 200, response.content
+    assert response["Cache-Control"] == "no-store"
+    data = response.json()
+    session = Session.objects.get(session_key=data["cookie"]["value"])
+    assert session.get_decoded()["_auth_user_id"] == str(api_admin_user.pk)
+    assert timezone.now() < session.expire_date <= token.expires
+    client.cookies.clear()
+    client.cookies[data["cookie"]["name"]] = data["cookie"]["value"]
+    response = client.get("/admin/", HTTP_HOST=urlsplit(data["admin_url"]).netloc)
+    assert response.status_code == 200, response.content
+    assert response.wsgi_request.user.pk == api_admin_user.pk
+
+
+@pytest.mark.parametrize("kind", ["missing", "invalid", "expired", "inactive", "nonadmin", "query"])
+def test_browser_session_rejects_invalid_credentials(client, api_admin_user, kind):
+    token = APIToken.objects.create(created_by=api_admin_user, expires=timezone.now() + timedelta(minutes=5))
+    if kind == "expired":
+        token.expires = timezone.now() - timedelta(seconds=1)
+        token.save()
+    if kind in ("inactive", "nonadmin"):
+        setattr(api_admin_user, "is_active" if kind == "inactive" else "is_superuser", False)
+        api_admin_user.save()
+    headers = {"HTTP_HOST": API_TEST_HOST}
+    if kind not in ("missing", "query"):
+        headers["HTTP_AUTHORIZATION"] = "Bearer " + ("invalid" if kind == "invalid" else token.token)
+    path = "/api/v1/auth/browser_session" + (f"?api_key={token.token}" if kind == "query" else "")
+    before = Session.objects.count()
+    response = api_client_request(client, "post", path, payload={}, headers=headers)
+    assert response.status_code in (401, 403), response.content
+    assert Session.objects.count() == before

archivebox/tests/test_auth_ldap.py

@@ -153,7 +153,7 @@ class TestArchiveBoxWithLDAP:
 
     def test_archivebox_init_without_ldap(self, tmp_path):
         """Test that archivebox init works without LDAP enabled."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["init"],
             cwd=tmp_path,
             timeout=45,
@@ -161,14 +161,13 @@ class TestArchiveBoxWithLDAP:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should succeed
-        assert code == 0, f"archivebox init failed: {stderr}"
+        assert result.returncode == 0, f"archivebox init failed: {result.stderr}"
 
     def test_archivebox_version_with_ldap_config(self, tmp_path):
         """Test that archivebox version works with LDAP config set."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["version"],
             cwd=tmp_path,
             timeout=10,
@@ -179,10 +178,9 @@ class TestArchiveBoxWithLDAP:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should succeed
-        assert code == 0, f"archivebox version failed: {stderr}"
+        assert result.returncode == 0, f"archivebox version failed: {result.stderr}"
 
 
 class TestLDAPConfigValidationInArchiveBox:
@@ -190,7 +188,7 @@ class TestLDAPConfigValidationInArchiveBox:
 
     def test_archivebox_init_with_incomplete_ldap_config(self, tmp_path):
         """Test that archivebox init fails with helpful error when LDAP config is incomplete."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["init"],
             cwd=tmp_path,
             timeout=45,
@@ -201,10 +199,9 @@ class TestLDAPConfigValidationInArchiveBox:
             default_cli_env=True,
             disable_extractors=True,
         )
-        _, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
         # Should fail with validation error
-        assert code != 0, "Should fail with incomplete LDAP config"
+        assert result.returncode != 0, "Should fail with incomplete LDAP config"
 
         # Check error message
-        assert "LDAP_* config options must all be set" in stderr, f"Expected validation error message in: {stderr}"
+        assert "LDAP_* config options must all be set" in result.stderr, f"Expected validation error message in: {result.stderr}"

archivebox/tests/test_cli_add.py

@@ -230,27 +230,23 @@ def test_add_single_url_records_url_in_crawl(initialized_archive):
     assert snapshots == []
 
 
-def test_add_memory_preflight_fails_before_creating_crawl(initialized_archive, capsys):
-    from archivebox.workers.supervisord_util import MIN_CRAWL_AVAILABLE_MEMORY_BYTES, require_crawl_memory
-
-    assert MIN_CRAWL_AVAILABLE_MEMORY_BYTES == 512 * 1024 * 1024
-    require_crawl_memory(MIN_CRAWL_AVAILABLE_MEMORY_BYTES)
-
+def test_add_bg_persists_queue_without_starting_extraction_workers(initialized_archive):
+    result = run_archivebox_cmd(
+        ["add", "--bg", "--plugins=wget", "--crawl-max-concurrent-snapshots=4", "https://example.com", "https://example.org"],
+        cwd=initialized_archive,
+        env=cli_env(),
+    )
+    assert result.returncode == 0, result.stderr
     with use_archivebox_db(initialized_archive):
-        before_crawls = Crawl.objects.count()
-        before_processes = Process.objects.count()
-
-        with pytest.raises(SystemExit) as err:
-            require_crawl_memory(MIN_CRAWL_AVAILABLE_MEMORY_BYTES - 1)
-
-        assert Crawl.objects.count() == before_crawls
-        assert Process.objects.count() == before_processes
-
-    assert err.value.code == 1
-    output = capsys.readouterr().err
-    assert "Not enough available memory to archive a crawl" in output
-    assert "No crawl, runner, or Sonic workers were started" in output
-    assert "configure swap" in output
+        crawl = Crawl.objects.get()
+        assert crawl.status == Crawl.StatusChoices.QUEUED
+        assert crawl.retry_at is not None
+        assert crawl.get_urls_list() == ["https://example.com", "https://example.org"]
+        assert crawl.config["PLUGINS"] == "wget"
+        assert crawl.config["CRAWL_MAX_CONCURRENT_SNAPSHOTS"] == 4
+        assert not Snapshot.objects.exists()
+        assert not ArchiveResult.objects.exists()
+        assert not Process.objects.filter(process_type__in=[Process.TypeChoices.ORCHESTRATOR, Process.TypeChoices.HOOK]).exists()
 
 
 def test_initial_crawl_creation_does_not_warn_as_an_outside_runner_update(initialized_archive, caplog):
@@ -1208,7 +1204,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
     root_url = recursive_test_site["root_url"]
     child_url = recursive_test_site["child_urls"][0]
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         [
             "add",
             "--depth=2",
@@ -1223,8 +1219,7 @@ def test_cli_recursive_crawl_processes_discovered_html_urls(initialized_archive,
         env=env,
         timeout=180,
     )
-    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert returncode == 0, stderr or stdout
+    assert result.returncode == 0, result.stderr or result.stdout
 
     with use_archivebox_db(initialized_archive):
         crawl = Crawl.objects.order_by("-created_at").values_list("max_depth", "tags_str", "config").first()

archivebox/tests/test_cli_archiveresult.py

@@ -142,18 +142,17 @@ class TestArchiveResultCreate:
         assert crawl_result.returncode == 0, crawl_result.stderr
         crawl_record = parse_jsonl_output(crawl_result.stdout)[0]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["archiveresult", "create"],
             stdin=crawl_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Passed through" in stderr
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        assert "Passed through" in result.stderr
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["id"] == crawl_record["id"]
 
@@ -163,16 +162,15 @@ class TestArchiveResultList:
 
     def test_list_empty(self, initialized_archive):
         """List with no archive results returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["archiveresult", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 archive results" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 archive results" in result.stderr
 
     def test_list_filter_by_status(self, initialized_archive):
         """Filter archive results by status."""

archivebox/tests/test_cli_crawl.py

@@ -33,19 +33,18 @@ class TestCrawlCreate:
         """Create crawl from URL arguments."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        assert "Created crawl" in stderr
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        assert "Created crawl" in result.stderr
 
         # Check JSONL output
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["type"] == "Crawl"
         assert url in records[0]["urls"]
@@ -55,18 +54,17 @@ class TestCrawlCreate:
         urls = [create_test_url() for _ in range(3)]
         stdin = "\n".join(urls)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         crawl = records[0]
         assert crawl["type"] == "Crawl"
@@ -78,32 +76,30 @@ class TestCrawlCreate:
         """Create crawl with --depth flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", "--depth=2", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert records[0]["max_depth"] == 2
 
     def test_create_with_tag(self, initialized_archive):
         """Create crawl with --tag flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create", "--tag=test-tag", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert "test-tag" in records[0].get("tags_str", "")
 
     def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -119,17 +115,16 @@ class TestCrawlCreate:
         url = create_test_url()
         stdin = tag_result.stdout + json.dumps({"url": url}) + "\n"
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
         assert any(record.get("type") == "Crawl" and url in record["urls"] for record in records)
@@ -163,32 +158,30 @@ class TestCrawlList:
 
     def test_list_empty(self, initialized_archive):
         """List with no crawls returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 crawls" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 crawls" in result.stderr
 
     def test_list_returns_created(self, initialized_archive):
         """List returns previously created crawls."""
         url = create_test_url()
         run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
         assert any(url in r.get("urls", "") for r in records)
 
@@ -197,16 +190,15 @@ class TestCrawlList:
         url = create_test_url()
         run_archivebox_cmd(["crawl", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list", "--status=queued"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         for r in records:
             assert r["status"] == "queued"
 
@@ -221,16 +213,15 @@ class TestCrawlList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["crawl", "list", "--limit=2"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
 

archivebox/tests/test_cli_extract_input.py

@@ -2,6 +2,7 @@
 
 import json
 import subprocess
+import sqlite3
 
 import pytest
 
@@ -22,6 +23,54 @@ def create_extract_snapshot(initialized_archive, env, url="https://example.com")
     )
 
 
+def test_extract_indexes_large_title_without_passing_content_in_environment(initialized_archive):
+    from archivebox.machine.models import Process
+
+    env = cli_env(PLUGINS="search_backend_sqlite", SEARCH_BACKEND_SQLITE_ENABLED="true")
+    create_extract_snapshot(initialized_archive, env)
+    title = "界" * 70000 + " literal shell syntax $(touch injected) `touch injected`"
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get()
+        snapshot_id = str(snapshot.id)
+        snapshot.save_tags(["metadata preservation"])
+        Snapshot.objects.filter(id=snapshot.id).update(title=title)
+        snapshot_dir = snapshot.output_dir
+    title_path = snapshot_dir / "title" / "title.txt"
+    title_path.parent.mkdir(parents=True, exist_ok=True)
+    title_path.write_text(title)
+
+    result = run_archivebox_cmd(
+        ["extract", "--plugins=search_backend_sqlite", snapshot_id],
+        cwd=initialized_archive,
+        env=env,
+        timeout=90,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(id=snapshot_id)
+        assert snapshot.title == title
+        assert list(snapshot.tags.values_list("name", flat=True)) == ["metadata preservation"]
+        archive_result = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="search_backend_sqlite")
+        assert archive_result.status == ArchiveResult.StatusChoices.SUCCEEDED
+        processes = list(Process.objects.filter(process_type=Process.TypeChoices.HOOK))
+        assert len(processes) == 1
+        process = processes[0]
+        assert process.exit_code == 0
+        assert f"--snapshot-id={snapshot_id}" in process.cmd
+        assert "--depth=0" in process.cmd
+        context = json.loads(process.env["EXTRA_CONTEXT"])
+        assert set(context) == {"snapshot_id", "plugin", "hook_name"}
+        assert title not in str(process.env)
+        snapshot_dir = snapshot.output_dir
+    with sqlite3.connect(initialized_archive / "search.sqlite3") as conn:
+        assert conn.execute("SELECT snapshot_id, title FROM search_index").fetchall() == [(snapshot_id, title)]
+    metadata = json.loads((snapshot_dir / "index.jsonl").read_text().splitlines()[0])
+    assert metadata["title"] == title
+    assert metadata["tags"] == "metadata preservation"
+    assert (snapshot_dir / "title" / "title.txt").read_text() == title
+    assert not list(initialized_archive.rglob("injected"))
+
+
 def test_extract_archiveresult_record_queues_parent_snapshot_plugin(initialized_archive):
     env = cli_env(PLUGINS="archivewebpage")
     create_extract_snapshot(initialized_archive, env)

archivebox/tests/test_cli_list.py

@@ -3,6 +3,8 @@ Tests for archivebox list command.
 Verify list emits snapshot JSONL and applies the documented filters.
 """
 
+import csv
+import io
 import json
 from pathlib import Path
 
@@ -427,16 +429,15 @@ def test_snapshot_list_search_meta(initialized_archive):
     url = create_test_url(domain="meta-search-example.com")
     run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["snapshot", "list", "--search=meta", "meta-search-example.com"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, f"Command failed: {stderr}"
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, f"Command failed: {result.stderr}"
+    records = parse_jsonl_output(result.stdout)
     assert len(records) == 1
     assert "meta-search-example.com" in records[0]["url"]
 
@@ -446,16 +447,15 @@ def test_list_search_meta_matches_metadata(initialized_archive):
     url = create_test_url(domain="top-level-meta-search-example.com")
     run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["list", "--search=meta", "top-level-meta-search-example.com"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, f"Command failed: {stderr}"
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, f"Command failed: {result.stderr}"
+    records = parse_jsonl_output(result.stdout)
     assert len(records) == 1
     assert "top-level-meta-search-example.com" in records[0]["url"]
 
@@ -468,11 +468,10 @@ def test_search_command_finds_snapshots(initialized_archive):
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "example"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0, stderr
-    assert "example" in stdout
+    assert result.returncode == 0, result.stderr
+    assert "example" in result.stdout
 
 
 def test_search_command_returns_no_results_for_missing_term(initialized_archive):
@@ -483,22 +482,20 @@ def test_search_command_returns_no_results_for_missing_term(initialized_archive)
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "nonexistentterm12345"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code in [0, 1]
+    assert result.returncode in [0, 1]
 
 
 def test_search_command_on_empty_archive(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "anything"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code in [0, 1]
+    assert result.returncode in [0, 1]
 
 
 def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive):
@@ -509,11 +506,10 @@ def test_search_command_outputs_matching_snapshots_as_jsonl(initialized_archive)
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0, stderr
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, result.stderr
+    records = parse_jsonl_output(result.stdout)
     assert any("example.com" in row.get("url", "") for row in records)
 
 
@@ -576,52 +572,47 @@ def test_search_command_csv_outputs_requested_column(initialized_archive):
         disable_extractors=True,
     )
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "--csv", "url", "--with-headers"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, stderr
-    assert "url" in stdout
-    assert "example.com" in stdout
+    assert result.returncode == 0, result.stderr
+    assert list(csv.reader(io.StringIO(result.stdout))) == [["url"], ["https://example.com"]]
 
 
 def test_search_command_with_headers_requires_structured_output_format(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "--with-headers"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code != 0
-    assert "requires" in stderr.lower()
-    assert "json" in stderr.lower()
+    assert result.returncode != 0
+    assert "requires" in result.stderr.lower()
+    assert "json" in result.stderr.lower()
 
 
 def test_search_command_sort_option_runs_successfully(initialized_archive):
     for url in ["https://iana.org", "https://example.com"]:
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["search", "--csv", "url", "--sort=url"],
         cwd=initialized_archive,
         default_cli_env=True,
         disable_extractors=True,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-    assert code == 0, stderr
-    assert "example.com" in stdout or "iana.org" in stdout
+    assert result.returncode == 0, result.stderr
+    assert list(csv.reader(io.StringIO(result.stdout))) == [["https://example.com"], ["https://iana.org"]]
 
 
 def test_search_command_help_lists_supported_filters(initialized_archive):
-    _cmd_result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
-    stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
+    result = run_archivebox_cmd(["search", "--help"], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-    assert code == 0
-    assert "--url__icontains" in stdout
-    assert "--crawl-id" in stdout
-    assert "--status" in stdout
-    assert "--sort" in stdout
-    assert "--json" in stdout
-    assert "--html" in stdout
+    assert result.returncode == 0
+    assert "--url__icontains" in result.stdout
+    assert "--crawl-id" in result.stdout
+    assert "--status" in result.stdout
+    assert "--sort" in result.stdout
+    assert "--json" in result.stdout
+    assert "--html" in result.stdout

archivebox/tests/test_cli_machine.py

@@ -6,6 +6,32 @@ Tests for archivebox machine command.
 from archivebox.tests.conftest import run_archivebox_cmd
 
 
+def test_status_refreshes_week_old_machine(initialized_archive):
+    """A saved host refresh must not invalidate the instance still in use."""
+    import sqlite3
+    from datetime import datetime, timedelta, timezone
+
+    from archivebox.machine.detect import get_host_guid
+
+    # Age real rows instead of mocking time or save(): the failure happens
+    # when normal CLI startup refreshes a persisted host after seven days.
+    expired = (datetime.now(timezone.utc) - timedelta(days=8)).isoformat()
+    with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+        before = db.execute("SELECT id, guid FROM machine_machine ORDER BY id").fetchall()
+        assert before
+        db.execute("UPDATE machine_machine SET modified_at=?", (expired,))
+
+    result = run_archivebox_cmd(["status"], cwd=initialized_archive)
+    assert result.returncode == 0, result.stdout + result.stderr
+    with sqlite3.connect(initialized_archive / "index.sqlite3") as db:
+        after = db.execute("SELECT id, guid FROM machine_machine ORDER BY id").fetchall()
+        assert after == before
+        refreshed = db.execute("SELECT modified_at FROM machine_machine WHERE guid=?", (get_host_guid(),)).fetchone()
+        assert refreshed is not None
+        assert datetime.fromisoformat(refreshed[0]).replace(tzinfo=timezone.utc) > datetime.fromisoformat(expired)
+        assert all(row[0] == expired for row in db.execute("SELECT modified_at FROM machine_machine WHERE guid!=?", (get_host_guid(),)))
+
+
 def test_machine_help_runs_successfully(tmp_path):
     """The machine command should be registered and expose help."""
 

archivebox/tests/test_cli_persona.py

@@ -14,3 +14,156 @@ def test_persona_help_runs_successfully(tmp_path):
     assert result.returncode == 0
     assert "persona" in result.stdout.lower()
     assert "list" in result.stdout
+
+
+def test_persona_create_permissions_preserves_existing_persona(initialized_archive):
+    from archivebox.tests.conftest import run_python_cwd
+
+    result = run_archivebox_cmd(["persona", "create", "--permissions=private", "host-profile"], cwd=initialized_archive)
+    assert result.returncode == 0, result.stderr
+    existing = run_archivebox_cmd(["persona", "create", "--permissions=public", "host-profile"], cwd=initialized_archive)
+    assert existing.returncode == 0, existing.stderr
+    stdout, stderr, returncode = run_python_cwd(
+        "import os; os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'archivebox.core.settings'); "
+        "import django; django.setup(); from archivebox.personas.models import Persona; "
+        "p = Persona.objects.get(name='host-profile'); assert p.permissions == 'private'; print('private')",
+        cwd=initialized_archive,
+    )
+    assert returncode == 0, stderr
+    assert "private" in stdout
+
+
+def test_persona_import_missing_source_is_actionable_and_does_not_create(initialized_archive, tmp_path):
+    result = run_archivebox_cmd(
+        ["persona", "create", "--import=brave", "--source", str(tmp_path / "missing"), "broken"],
+        cwd=initialized_archive,
+    )
+    assert result.returncode == 1
+    assert "does not exist" in result.stderr
+    listed = run_archivebox_cmd(["persona", "list", "--name=broken"], cwd=initialized_archive)
+    assert listed.returncode == 0
+    assert not listed.stdout.strip()
+    assert not (initialized_archive / "personas" / "broken").exists()
+
+
+def test_persona_import_real_browser_session(initialized_archive, tmp_path, httpserver):
+    """Import an actual Chromium-created session and archive its private page."""
+    import hashlib
+    import json
+    import subprocess
+
+    from werkzeug.wrappers import Response
+    from archivebox.tests.conftest import resolve_abxpkg_chrome_env
+
+    browser_env = resolve_abxpkg_chrome_env(tmp_path / "lib")
+    source = tmp_path / "browser"
+    httpserver.expect_request("/login").respond_with_data(
+        '<html><title>Signed in</title><script>localStorage.setItem("persona-preference", "forest-green")</script>Signed in</html>',
+        content_type="text/html",
+        headers={"Set-Cookie": "persona_session=real-session; Path=/; Max-Age=3600; HttpOnly; SameSite=Lax"},
+    )
+
+    def private_page(request):
+        authenticated = request.cookies.get("persona_session") == "real-session"
+        return Response(
+            "<html><title>Private notebook</title><p>Authenticated persona notebook</p>"
+            '<script>document.body.append(localStorage.getItem("persona-preference"))</script></html>'
+            if authenticated
+            else "<html>Please log in</html>",
+            content_type="text/html",
+        )
+
+    httpserver.expect_request("/private").respond_with_handler(private_page)
+    seeded = subprocess.run(
+        [
+            browser_env["NODE_BINARY"],
+            "-e",
+            """
+const puppeteer = require(process.argv[1]);
+(async () => {
+    const browser = await puppeteer.launch({
+        executablePath: process.argv[2], userDataDir: process.argv[3], headless: true,
+        ignoreDefaultArgs: ['--use-mock-keychain', '--password-store=basic'],
+        args: ['--no-sandbox', '--profile-directory=Profile 2'],
+    });
+    try {
+        const page = await browser.newPage();
+        await page.goto(process.argv[4]);
+        console.log(await page.title());
+    } finally { await browser.close(); }
+})().catch(error => { console.error(error); process.exit(1); });
+""",
+            str(tmp_path / "lib/pnpm/packages/chrome/node_modules/puppeteer"),
+            browser_env["CHROME_BINARY"],
+            str(source),
+            httpserver.url_for("/login"),
+        ],
+        capture_output=True,
+        text=True,
+        timeout=60,
+    )
+    assert seeded.returncode == 0, seeded.stderr
+    assert "Signed in" in seeded.stdout
+    preferences = source / "Profile 2" / "Preferences"
+    original_preferences = preferences.read_bytes()
+
+    imported = run_archivebox_cmd(
+        [
+            "persona",
+            "create",
+            "--import=chromium",
+            "--source",
+            str(source),
+            "--browser-binary",
+            browser_env["CHROME_BINARY"],
+            "--profile=Profile 2",
+            "session",
+        ],
+        cwd=initialized_archive,
+        timeout=120,
+        env={**browser_env, "CHROME_SANDBOX": "false"},
+    )
+    assert imported.returncode == 0, imported.stderr
+    persona = initialized_archive / "personas" / "session"
+    assert (persona / "chrome_profile" / "Default" / "Preferences").read_bytes() == original_preferences
+    assert preferences.read_bytes() == original_preferences
+    auth = json.loads((persona / "auth.json").read_text())
+    cookie = next(cookie for cookie in auth["cookies"] if cookie["name"] == "persona_session")
+    assert cookie["value"] == "real-session"
+    assert cookie["httpOnly"] is True
+    assert cookie["sameSite"] == "Lax"
+    transferred = run_archivebox_cmd(
+        ["persona", "create", f"--import={persona}", "portable"],
+        cwd=initialized_archive,
+        timeout=120,
+    )
+    assert transferred.returncode == 0, transferred.stderr
+    portable = initialized_archive / "personas" / "portable"
+    assert (portable / "auth.json").read_bytes() == (persona / "auth.json").read_bytes()
+    assert (portable / "chrome_profile/Default/Preferences").read_bytes() == original_preferences
+    before = {
+        path.relative_to(persona): hashlib.sha256(path.read_bytes()).hexdigest()
+        for path in (persona / "cookies.txt", persona / "auth.json", persona / "chrome_profile/Default/Preferences")
+    }
+    failed = run_archivebox_cmd(
+        ["persona", "create", "--import=unsupported-browser", "--source", str(source), "--profile=Profile 2", "session"],
+        cwd=initialized_archive,
+        timeout=60,
+    )
+    assert failed.returncode == 1
+    assert "browser executable not found" in " ".join(failed.stderr.split())
+    assert all(hashlib.sha256((persona / relative).read_bytes()).hexdigest() == digest for relative, digest in before.items())
+
+    archived = run_archivebox_cmd(
+        ["add", "--persona=portable", "--plugins=dom", httpserver.url_for("/private")],
+        cwd=initialized_archive,
+        timeout=180,
+        env={**browser_env, "CHROME_SANDBOX": "false"},
+    )
+    assert archived.returncode == 0, archived.stderr
+    outputs = list((initialized_archive / "archive" / "users").glob("*/snapshots/**/dom/output.html"))
+    assert len(outputs) == 1
+    html = outputs[0].read_text()
+    assert "Authenticated persona notebook" in html
+    assert "forest-green" in html
+    assert "Please log in" not in html

archivebox/tests/test_cli_run.py

@@ -53,15 +53,14 @@ def test_cli_run_signal_cleans_real_chrome_hook_process_group(initialized_archiv
     env = cli_env(live=True, PLUGINS="chrome", CHROME_ISOLATION="crawl", CHROME_HEADLESS="true", CHROME_SANDBOX="false")
     _install_real_chrome_for_test(initialized_archive, env, isolation="crawl")
 
-    _cmd_result = run_archivebox_cmd(
+    result = run_archivebox_cmd(
         ["snapshot", "create", recursive_test_site["root_url"]],
         cwd=initialized_archive,
         env=env,
         timeout=60,
     )
-    stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert returncode == 0, stderr or stdout
-    records = parse_jsonl_output(stdout)
+    assert result.returncode == 0, result.stderr or result.stdout
+    records = parse_jsonl_output(result.stdout)
     snapshot_id = next(record["id"] for record in records if record.get("type") == "Snapshot")
     with use_archivebox_db(initialized_archive):
         browser_state = Snapshot.objects.get(id=snapshot_id).output_dir / "chrome" / "browser.json"
@@ -107,7 +106,7 @@ class TestRunWithCrawl:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -116,12 +115,11 @@ class TestRunWithCrawl:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
         # Should output the created Crawl
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         crawl_records = [r for r in records if r.get("type") == "Crawl"]
         assert len(crawl_records) >= 1
         assert crawl_records[0].get("id")  # Should have an id now
@@ -170,7 +168,7 @@ class TestRunWithSnapshot:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -179,11 +177,10 @@ class TestRunWithSnapshot:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         snapshot_records = [r for r in records if r.get("type") == "Snapshot"]
         assert len(snapshot_records) >= 1
         assert snapshot_records[0].get("id")
@@ -220,7 +217,7 @@ class TestRunWithSnapshot:
     def test_run_with_plain_url(self, initialized_archive):
         """Run accepts plain URL records (no type field)."""
         url = create_test_url()
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=url + "\n",
             cwd=initialized_archive,
@@ -229,10 +226,9 @@ class TestRunWithSnapshot:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
 
 
@@ -405,7 +401,7 @@ class TestRunRecovery:
             crawl_id = crawl.id
             snapshot_id = snapshot.id
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run", "--maintenance-only"],
             cwd=initialized_archive,
             timeout=90,
@@ -413,13 +409,12 @@ class TestRunRecovery:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, stdout + stderr
-        assert "Repairing" in stderr
-        assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in stderr
-        assert "interrupted before" in stderr
-        assert "remaining URLs" in stderr
+        assert result.returncode == 0, result.stdout + result.stderr
+        assert "Repairing" in result.stderr
+        assert "Resuming 1 Crawl(s) with pending URLs ready to archive" in result.stderr
+        assert "interrupted before" in result.stderr
+        assert "remaining URLs" in result.stderr
 
         with use_archivebox_db(initialized_archive):
             crawl = Crawl.objects.get(id=crawl_id)
@@ -444,17 +439,16 @@ class TestRunPassThrough:
         assert tag_result.returncode == 0, tag_result.stderr
         tag_record = parse_jsonl_output(tag_result.stdout)[0]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         tag_records = [record for record in records if record.get("type") == "Tag"]
         assert len(tag_records) == 1
         assert tag_records[0]["id"] == tag_record["id"]
@@ -471,7 +465,7 @@ class TestRunPassThrough:
         )
         assert create_result.returncode == 0, create_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=create_result.stdout,
             cwd=initialized_archive,
@@ -480,10 +474,9 @@ class TestRunPassThrough:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         # Should have at least the Crawl in output
         assert len(records) >= 1
 
@@ -516,7 +509,7 @@ class TestRunMixedInput:
         )
         assert snapshot_result.returncode == 0, snapshot_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout + snapshot_result.stdout,
             cwd=initialized_archive,
@@ -525,10 +518,9 @@ class TestRunMixedInput:
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         types = {record.get("type") for record in records}
         assert {"Crawl", "Snapshot", "Tag"}.issubset(types)
@@ -539,16 +531,15 @@ class TestRunEmpty:
 
     def test_run_empty_stdin(self, initialized_archive):
         """Run with empty stdin returns success."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin="",
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
+        assert result.returncode == 0
 
     def test_run_no_runnable_records_to_process(self, initialized_archive):
         """Run with only a real non-runnable Tag reports no work."""
@@ -560,17 +551,16 @@ class TestRunEmpty:
         )
         assert tag_result.returncode == 0, tag_result.stderr
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["run"],
             stdin=tag_result.stdout,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "No records to process" in stderr
+        assert result.returncode == 0
+        assert "No records to process" in result.stderr
 
 
 class TestRunDaemonMode:

archivebox/tests/test_cli_server.py

@@ -124,10 +124,17 @@ def test_server_auth_secret_and_cookie_settings_are_restart_stable(tmp_path):
     assert first_lines[3:] == ["None", "False", "False"]
 
 
-def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
+@pytest.mark.parametrize(
+    "base_url, secure",
+    [
+        ("https://archive.example.com", True),
+        ("", False),
+    ],
+)
+def test_base_url_controls_secure_cookies(tmp_path, base_url, secure):
     (tmp_path / ".archivebox_id").write_text("testcoll")
     env = os.environ.copy()
-    env["BASE_URL"] = "https://archive.example.com"
+    env["BASE_URL"] = base_url
     env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
     repo_root = Path(__file__).resolve().parents[2]
     env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
@@ -155,45 +162,8 @@ def test_https_base_url_enables_proxy_ssl_header_and_secure_cookies(tmp_path):
     )
 
     assert json.loads(result.stdout) == {
-        "csrf_secure": True,
-        "session_secure": True,
-        "proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
-    }
-
-
-def test_unconfigured_base_url_enables_proxy_ssl_header_without_secure_cookies(tmp_path):
-    (tmp_path / ".archivebox_id").write_text("testcoll")
-    env = os.environ.copy()
-    env["BASE_URL"] = ""
-    env["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
-    repo_root = Path(__file__).resolve().parents[2]
-    env["PYTHONPATH"] = f"{repo_root}{os.pathsep}{env.get('PYTHONPATH', '')}"
-
-    result = subprocess.run(
-        [
-            sys.executable,
-            "-c",
-            (
-                "import django, json;"
-                "django.setup();"
-                "from django.conf import settings;"
-                "print(json.dumps({"
-                "'csrf_secure': settings.CSRF_COOKIE_SECURE,"
-                "'session_secure': settings.SESSION_COOKIE_SECURE,"
-                "'proxy_ssl_header': settings.SECURE_PROXY_SSL_HEADER,"
-                "}))"
-            ),
-        ],
-        capture_output=True,
-        text=True,
-        check=True,
-        env=env,
-        cwd=tmp_path,
-    )
-
-    assert json.loads(result.stdout) == {
-        "csrf_secure": False,
-        "session_secure": False,
+        "csrf_secure": secure,
+        "session_secure": secure,
         "proxy_ssl_header": ["HTTP_X_FORWARDED_PROTO", "https"],
     }
 
@@ -759,9 +729,8 @@ def test_live_server_signal_exit_and_resume_uses_existing_supervisor_state(initi
 
         resumed = start_archivebox_server(initialized_archive, port=port, log_name=f"server-{stop_signal.name}-resumed.log", env=env)
         resumed_log = resumed.log_path
-        _cmd_result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        result = run_archivebox_cmd(["status"], cwd=initialized_archive, env=env, timeout=60)
+        assert result.returncode == 0, result.stderr or result.stdout
 
         os.kill(resumed.pid, signal.SIGTERM)
         resumed.wait(timeout=20)
@@ -783,14 +752,13 @@ def test_live_daemonized_server_keeps_supervisord_owned_by_archivebox_parent(ini
     port = get_free_port()
     bind_url = f"http://127.0.0.1:{port}"
     try:
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["server", "--daemonize", f"127.0.0.1:{port}"],
             cwd=initialized_archive,
             env=env,
             timeout=90,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         _wait_for_archivebox_workers(initialized_archive, env, ("worker_daphne", "worker_runner"), timeout=30)
         assert_port_open("127.0.0.1", port, timeout=30)
 

archivebox/tests/test_cli_snapshot.py

@@ -33,18 +33,17 @@ class TestSnapshotCreate:
         """Create snapshot from URL arguments."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        assert "Created" in stderr
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        assert "Created" in result.stderr
 
-        records = parse_jsonl_output(stdout)
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert records[0]["type"] == "Snapshot"
         assert records[0]["url"] == url
@@ -83,16 +82,15 @@ class TestSnapshotCreate:
         """Create snapshot with --tag flag."""
         url = create_test_url()
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create", "--tag=test-tag", url],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert "test-tag" in records[0].get("tags", "")
 
     def test_create_passes_through_tag_emitted_by_cli(self, initialized_archive):
@@ -108,17 +106,16 @@ class TestSnapshotCreate:
         url = create_test_url()
         stdin = tag_result.stdout + url + "\n"
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create"],
             stdin=stdin,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
 
         assert any(record.get("type") == "Tag" and record["id"] == tag_record["id"] for record in records)
         assert any(record.get("type") == "Snapshot" and record["url"] == url for record in records)
@@ -127,16 +124,15 @@ class TestSnapshotCreate:
         """Create snapshots from multiple URLs."""
         urls = [create_test_url() for _ in range(3)]
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "create"] + urls,
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 3
 
         created_urls = {r["url"] for r in records}
@@ -149,32 +145,30 @@ class TestSnapshotList:
 
     def test_list_empty(self, initialized_archive):
         """List with no snapshots returns empty."""
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        _stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        assert "Listed 0 snapshots" in stderr
+        assert result.returncode == 0
+        assert "Listed 0 snapshots" in result.stderr
 
     def test_list_returns_created(self, initialized_archive):
         """List returns previously created snapshots."""
         url = create_test_url()
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) >= 1
         assert any(r.get("url") == url for r in records)
 
@@ -183,16 +177,15 @@ class TestSnapshotList:
         url = create_test_url()
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--status=queued"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         for r in records:
             assert r["status"] == "queued"
 
@@ -201,16 +194,15 @@ class TestSnapshotList:
         url = create_test_url(domain="unique-domain-12345.com")
         run_archivebox_cmd(["snapshot", "create", url], cwd=initialized_archive, default_cli_env=True, disable_extractors=True)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--url__icontains=unique-domain-12345"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 1
         assert "unique-domain-12345" in records[0]["url"]
 
@@ -224,16 +216,15 @@ class TestSnapshotList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--limit=2"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, _stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
     def test_list_with_sort_and_limit(self, initialized_archive):
@@ -246,16 +237,15 @@ class TestSnapshotList:
                 disable_extractors=True,
             )
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["snapshot", "list", "--limit=2", "--sort=-created_at"],
             cwd=initialized_archive,
             default_cli_env=True,
             disable_extractors=True,
         )
-        stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert code == 0, f"Command failed: {stderr}"
-        records = parse_jsonl_output(stdout)
+        assert result.returncode == 0, f"Command failed: {result.stderr}"
+        records = parse_jsonl_output(result.stdout)
         assert len(records) == 2
 
 

archivebox/tests/test_crawl_runner.py

@@ -1,4 +1,5 @@
 import asyncio
+import json
 from importlib.resources import files
 from pathlib import Path
 import sys
@@ -11,6 +12,34 @@ from archivebox.tests.conftest import install_real_binary, resolve_abxpkg_binary
 pytestmark = pytest.mark.django_db
 
 
+@pytest.mark.django_db(transaction=True)
+def test_snapshot_payload_keeps_unbounded_content_out_of_hook_environment():
+    from archivebox.base_models.models import get_or_create_system_user_pk
+    from archivebox.crawls.models import Crawl
+    from archivebox.core.models import Snapshot
+    from archivebox.services.runner import CrawlRunner
+
+    title = "界" * 70000 + " $(touch injected) `touch injected`"
+    crawl = Crawl.objects.create(urls="https://example.com", created_by_id=get_or_create_system_user_pk())
+    snapshot = Snapshot.objects.create(url="https://example.com", title=title, depth=4, crawl=crawl)
+    snapshot.save_tags(["tag with spaces", "tag;not-a-command"])
+    # Legacy collections may contain titles larger than current model normalization allows.
+    Snapshot.objects.filter(id=snapshot.id).update(title=title)
+    runner = CrawlRunner(crawl)
+    try:
+        runner.load_run_state()
+        payload = runner.load_snapshot_payload(str(snapshot.id))
+        context = json.loads(payload["config"]["EXTRA_CONTEXT"])
+        assert context == {"snapshot_id": str(snapshot.id)}
+        assert payload["title"] == title
+        assert payload["depth"] == 4
+        snapshot.refresh_from_db()
+        assert snapshot.title == title
+        assert set(snapshot.tags.values_list("name", flat=True)) == {"tag with spaces", "tag;not-a-command"}
+    finally:
+        asyncio.run(runner.bus.destroy(clear=False))
+
+
 @pytest.mark.django_db(transaction=True)
 def test_crawl_runner_creates_progress_reporter_without_a_tty(crawl):
     from archivebox.services.runner import CrawlRunner

archivebox/tests/test_opencode_agent.py

@@ -1,19 +1,19 @@
 import asyncio
-import json
 import os
-import signal
+import re
+import shutil
 import socket
-import subprocess
 from concurrent.futures import ThreadPoolExecutor
 from pathlib import Path
 from types import SimpleNamespace
-from urllib.parse import quote
+from urllib.parse import parse_qs, quote, urlsplit
 
 import pytest
 import requests
 from asgiref.testing import ApplicationCommunicator
 
 from archivebox.tests.conftest import ADMIN_TEST_HOST, run_archivebox_cmd
+from archivebox.config.common import get_config
 
 
 pytestmark = pytest.mark.django_db(transaction=True)
@@ -78,8 +78,8 @@ def opencode_archive_config(initialized_archive):
 
 
 @pytest.fixture
-def live_opencode(opencode_archive_config):
-    from archivebox.opencode import views
+def installed_opencode(opencode_archive_config):
+    from abx_plugins.plugins.opencode import runtime
 
     install = run_archivebox_cmd(
         ["install", "opencode", "--binproviders=env,pnpm"],
@@ -90,56 +90,61 @@ def live_opencode(opencode_archive_config):
     assert install.returncode == 0, install.stderr or install.stdout
     _reset_runtime_config()
 
-    config = views._machine_config()
-    settings = views._settings(config)
+    config = get_config().model_dump(mode="json")
+    settings = runtime._settings(config, opencode_archive_config.data_dir)
     settings["archivebox_base_url"] = "http://admin.archivebox.localhost:8000"
     settings["archivebox_admin_url"] = "http://admin.archivebox.localhost:8000/admin"
     settings["archivebox_api_url"] = "http://admin.archivebox.localhost:8000/api/"
-    binary, _, binary_env = views._resolve_binary(settings["binary"], settings["config"])
+    binary, binary_env = runtime._resolve_binary(settings["binary"], settings["config"])
     version = binary.exec(
         cmd=("--version",),
         env={**os.environ, **binary_env},
         timeout=120,
     )
     assert version.returncode == 0, version.stderr or version.stdout
-    ok, error = views._ensure_opencode(settings)
+    return SimpleNamespace(config=opencode_archive_config, settings=settings)
+
+
+@pytest.fixture
+def live_opencode(installed_opencode):
+    from abx_plugins.plugins.opencode import runtime
+
+    settings = installed_opencode.settings
+    ok, error = runtime._ensure_opencode(settings)
     assert ok, error
 
-    process = views._PROCESS
+    process = runtime._PROCESS
     assert process is not None
     try:
-        yield SimpleNamespace(config=opencode_archive_config, settings=settings, process=process)
+        yield SimpleNamespace(config=installed_opencode.config, settings=settings, process=process)
     finally:
-        views._stop_owned_process()
+        runtime._stop_owned_process()
 
 
 def test_opencode_disabled_route_does_not_start_server(client, initialized_archive):
     from archivebox.machine.models import Machine
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
     os.chdir(initialized_archive)
     Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
     _reset_runtime_config()
-    assert views._machine_config()["OPENCODE_ENABLED"] is False
+    assert get_config().model_dump(mode="json")["OPENCODE_ENABLED"] is False
 
     response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
 
     assert response.status_code == 404
-    assert views._PROCESS is None or views._PROCESS.poll() is not None
+    assert runtime._PROCESS is None or runtime._PROCESS.poll() is not None
 
 
-def test_stop_owned_process_falls_back_for_stopped_process_without_dedicated_group():
-    from archivebox.opencode import views
+def test_opencode_disabled_via_cli_stays_disabled(admin_client, initialized_archive):
+    _set_archivebox_config(initialized_archive, "OPENCODE_ENABLED=False")
 
-    process = subprocess.Popen(["sleep", "60"])
-    try:
-        process.send_signal(signal.SIGSTOP)
-        views._stop_owned_process(process)
-        assert process.returncode == -signal.SIGTERM
-    finally:
-        if process.poll() is None:
-            process.kill()
-            process.wait()
+    assert get_config().OPENCODE_ENABLED is False
+    assert admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST).status_code == 404
+    for path in ("/add/", "/admin/core/snapshot/"):
+        response = admin_client.get(path, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 200
+        assert b'href="/admin/agent"' not in response.content
 
 
 def test_opencode_agent_requires_superuser_when_enabled(client, db, django_user_model, live_opencode):
@@ -147,6 +152,11 @@ def test_opencode_agent_requires_superuser_when_enabled(client, db, django_user_
     assert response.status_code == 302
     assert "/admin/login/" in response.headers["Location"]
 
+    next_path = "/admin/agent?x=1&next=https://example.com"
+    response = client.get(next_path, HTTP_HOST=ADMIN_TEST_HOST)
+    assert response.status_code == 302
+    assert parse_qs(urlsplit(response.headers["Location"]).query) == {"next": [next_path]}
+
     user = django_user_model.objects.create_user(username="regular", password="testpassword")
     client.force_login(user)
     response = client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
@@ -178,18 +188,22 @@ def test_opencode_proxy_blocks_cross_site_fetch_metadata(admin_client, db, live_
 
 
 def test_opencode_agent_superuser_gets_admin_wrapper(admin_client, live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
     response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
     recent_session_id = response.context["recent_session_id"]
-    session_path = views._project_route(live_opencode.config.data_dir, recent_session_id)
+    session_path = runtime._project_route(live_opencode.config.data_dir, recent_session_id)
 
     assert response.status_code == 200
     assert recent_session_id
     assert f'<iframe src="{session_path}"'.encode() in response.content
     assert b'id="header"' in response.content
     assert b'id="progress-monitor"' in response.content
-    assert response.context["proxy_prefix"] == views._PROXY_PREFIX
+    assert b'<a href="/admin/agent" class="navbar-item navbar-ai">' in response.content
+    add_page = admin_client.get("/add/", HTTP_HOST=ADMIN_TEST_HOST)
+    assert add_page.status_code == 200
+    assert '<a href="/admin/agent">💬 Crawl with AI</a>'.encode() in add_page.content
+    assert response.context["proxy_prefix"] == runtime._PROXY_PREFIX
     assert b"/_archivebox/health" not in response.content
     assert b"window.setInterval(check, 3000)" not in response.content
     assert response.headers["X-Frame-Options"] == "DENY"
@@ -218,7 +232,8 @@ def test_opencode_proxy_serves_real_project_and_session(admin_client, live_openc
         HTTP_SEC_FETCH_SITE="same-origin",
     )
     assert project.status_code == 200
-    assert workdir.encode() in project.content
+    assert project.json()["id"] == "global"
+    assert not project.json().get("vcs")
 
     path = admin_client.get(
         f"/admin/agent/opencode/path?directory={encoded_workdir}",
@@ -226,7 +241,7 @@ def test_opencode_proxy_serves_real_project_and_session(admin_client, live_openc
         HTTP_SEC_FETCH_SITE="same-origin",
     )
     assert path.status_code == 200
-    assert workdir.encode() in path.content
+    assert path.json()["directory"] == workdir
 
     sessions = admin_client.get(
         f"/admin/agent/opencode/session?directory={encoded_workdir}&roots=true&limit=55",
@@ -234,14 +249,15 @@ def test_opencode_proxy_serves_real_project_and_session(admin_client, live_openc
         HTTP_SEC_FETCH_SITE="same-origin",
     )
     assert sessions.status_code == 200
-    assert b"id" in sessions.content
+    assert any(session["id"] == agent.context["recent_session_id"] and session["directory"] == workdir for session in sessions.json())
+    assert not (Path(workdir) / ".git").exists()
 
 
 def test_opencode_proxy_restarts_server_for_an_existing_agent_page(admin_client, live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
-    old_process = views._PROCESS
-    views._stop_owned_process()
+    old_process = runtime._PROCESS
+    runtime._stop_owned_process()
 
     response = admin_client.get(
         "/admin/agent/opencode/global/health",
@@ -250,45 +266,45 @@ def test_opencode_proxy_restarts_server_for_an_existing_agent_page(admin_client,
     )
 
     assert response.status_code == 200
-    assert views._PROCESS is not None
-    assert views._PROCESS is not old_process
-    assert views._PROCESS.poll() is None
+    assert runtime._PROCESS is not None
+    assert runtime._PROCESS is not old_process
+    assert runtime._PROCESS.poll() is None
 
 
 def test_concurrent_opencode_startup_waits_until_server_is_ready(live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
-    views._stop_owned_process()
+    runtime._stop_owned_process()
 
     with ThreadPoolExecutor(max_workers=2) as executor:
-        results = list(executor.map(views._ensure_opencode, [live_opencode.settings] * 2))
+        results = list(executor.map(runtime._ensure_opencode, [live_opencode.settings] * 2))
 
     assert results == [(True, ""), (True, "")]
-    assert views._health(live_opencode.settings)
+    assert runtime._health(live_opencode.settings)
 
 
 def test_opencode_does_not_probe_or_replace_a_ready_owned_process(live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
-    process = views._PROCESS
+    process = runtime._PROCESS
     settings = {**live_opencode.settings, "port": _free_port()}
     settings["origin"] = f"http://{settings['host']}:{settings['port']}"
 
-    ok, error = views._ensure_opencode(settings)
+    ok, error = runtime._ensure_opencode(settings)
 
     assert ok, error
     assert process is not None
-    assert views._PROCESS is process
+    assert runtime._PROCESS is process
     assert process.poll() is None
 
 
 def test_opencode_proxy_does_not_wait_for_recovery_lock(admin_client, live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
     workdir = quote(str(live_opencode.config.data_dir.resolve()))
-    assert views._owned_process_ready()
+    assert runtime._owned_process_ready()
     executor = ThreadPoolExecutor(max_workers=1)
-    views._PROCESS_LOCK.acquire()
+    runtime._PROCESS_LOCK.acquire()
     try:
         request = executor.submit(
             admin_client.get,
@@ -298,7 +314,7 @@ def test_opencode_proxy_does_not_wait_for_recovery_lock(admin_client, live_openc
         )
         response = request.result(timeout=5)
     finally:
-        views._PROCESS_LOCK.release()
+        runtime._PROCESS_LOCK.release()
         executor.shutdown(wait=True)
 
     assert response.status_code == 200
@@ -306,11 +322,11 @@ def test_opencode_proxy_does_not_wait_for_recovery_lock(admin_client, live_openc
 
 
 def test_opencode_proxy_waits_for_owned_process_readiness(admin_client, live_opencode):
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
 
-    process = views._PROCESS
+    process = runtime._PROCESS
     assert process is not None
-    views._PROCESS_READY = None
+    runtime._PROCESS_READY = None
     workdir = quote(str(live_opencode.config.data_dir.resolve()))
 
     response = admin_client.get(
@@ -320,8 +336,46 @@ def test_opencode_proxy_waits_for_owned_process_readiness(admin_client, live_ope
     )
 
     assert response.status_code == 200
-    assert views._PROCESS is process
-    assert views._PROCESS_READY is process
+    assert runtime._PROCESS is process
+    assert runtime._PROCESS_READY is process
+
+
+def test_opencode_proxy_preserves_protocol_headers(admin_client, live_opencode):
+    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
+    created = admin_client.post(
+        "/admin/agent/opencode/pty",
+        data={"command": "/bin/sh", "args": []},
+        content_type="application/json",
+        **headers,
+    )
+    assert created.status_code == 200
+    path = f"/admin/agent/opencode/pty/{created.json()['id']}"
+    try:
+        response = admin_client.post(path + "/connect-token", HTTP_X_OPENCODE_TICKET="1", **headers)
+        assert response.status_code == 200, response.content
+        assert response.json()["ticket"]
+    finally:
+        deleted = admin_client.delete(path, **headers)
+    assert deleted.status_code == 200, deleted.content
+
+
+def test_opencode_static_assets_cache_privately_and_revalidate(admin_client, live_opencode):
+    headers = {"HTTP_HOST": ADMIN_TEST_HOST, "HTTP_SEC_FETCH_SITE": "same-origin"}
+    page = admin_client.get("/admin/agent/opencode/", **headers)
+    assert page.status_code == 200
+    asset = re.search(rb'src="(/admin/agent/opencode/assets/[^\"]+\.js)"', page.content)
+    assert asset is not None
+    path = asset[1].decode()
+    response = admin_client.get(path, **headers)
+    assert response.status_code == 200
+    assert "private" in response.headers["Cache-Control"]
+    assert "max-age=" in response.headers["Cache-Control"]
+    assert b"/admin/agent/opencode" in response.content
+    repeated = admin_client.get(path, HTTP_IF_NONE_MATCH=response.headers["ETag"], **headers)
+    assert repeated.status_code == 304
+    assert repeated.content == b""
+    assert repeated.headers["ETag"] == response.headers["ETag"]
+    assert admin_client.get("/admin/agent/opencode/path", **headers).headers["Cache-Control"] == "no-store"
 
 
 def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
@@ -338,12 +392,31 @@ def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
     assert response.headers["Cache-Control"] == "no-store"
 
 
+def test_opencode_proxy_sse_delivers_first_event_immediately(admin_client, live_opencode):
+    response = admin_client.get(
+        "/admin/agent/opencode/global/event",
+        HTTP_HOST=ADMIN_TEST_HOST,
+        HTTP_SEC_FETCH_SITE="same-origin",
+    )
+    assert response.status_code == 200
+
+    async def first_event():
+        stream = response.streaming_content
+        try:
+            async with asyncio.timeout(2):
+                return await anext(stream)
+        finally:
+            await stream.aclose()
+
+    assert b"server.connected" in asyncio.run(first_event())
+
+
 def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client, live_opencode):
     from archivebox.core.asgi import application
-    from archivebox.opencode import views
+    from abx_plugins.plugins.opencode import runtime
     from django.conf import settings as django_settings
 
-    owned_process = views._PROCESS
+    owned_process = runtime._PROCESS
     assert owned_process is not None
     session_cookie_name = django_settings.SESSION_COOKIE_NAME
     session_cookie = admin_client.cookies[session_cookie_name].value
@@ -370,8 +443,8 @@ def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client
                 "server": ("127.0.0.1", 8000),
             },
         )
-        views._PROCESS_LOCK.acquire()
-        views._PROCESS = None
+        runtime._PROCESS_LOCK.acquire()
+        runtime._PROCESS = None
         try:
             await communicator.send_input({"type": "http.request", "body": b"", "more_body": False})
             response_start = await communicator.receive_output(timeout=2)
@@ -382,19 +455,25 @@ def test_opencode_proxy_sse_returns_headers_before_restart_finishes(admin_client
                 await communicator.send_input({"type": "http.disconnect"})
                 await communicator.wait(timeout=5)
             finally:
-                views._PROCESS = owned_process
-                views._PROCESS_LOCK.release()
+                runtime._PROCESS = owned_process
+                runtime._PROCESS_LOCK.release()
                 await asyncio.get_running_loop().shutdown_default_executor()
 
     asyncio.run(request_event_stream())
-    assert views._PROCESS is owned_process
+    assert runtime._PROCESS is owned_process
     assert owned_process.poll() is None
 
 
-def test_opencode_starts_with_isolated_state(live_opencode):
+def test_opencode_starts_with_isolated_state(admin_client, live_opencode):
     workdir = str(live_opencode.config.data_dir.resolve())
     state_dir = live_opencode.config.state_dir
 
+    assert not (Path(workdir) / ".git").exists()
+    agent = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
+    assert agent.status_code == 200
+    assert agent.context["recent_session_id"]
+    assert not (Path(workdir) / ".git").exists()
+
     project = requests.get(
         f"{live_opencode.settings['origin']}/project/current",
         params={"directory": workdir},
@@ -409,89 +488,112 @@ def test_opencode_starts_with_isolated_state(live_opencode):
     config.raise_for_status()
 
     assert Path(live_opencode.settings["workdir"]).resolve() == Path(workdir)
-    assert Path(str(project.json()["worktree"])).resolve() == Path(workdir)
-    assert config.json()["model"] == "opencode/big-pickle"
+    assert project.json()["id"] == "global"
+    assert not project.json().get("vcs")
+    assert not config.json().get("model")
     assert config.json()["snapshot"] is False
     assert live_opencode.process.poll() is None
-    assert (live_opencode.config.data_dir / ".git").is_dir()
+    path = requests.get(
+        f"{live_opencode.settings['origin']}/path",
+        params={"directory": workdir},
+        timeout=live_opencode.settings["timeout"],
+    )
+    path.raise_for_status()
+    assert Path(path.json()["directory"]).resolve() == Path(workdir)
+
+    diff = requests.get(
+        f"{live_opencode.settings['origin']}/vcs/diff",
+        params={"directory": workdir, "mode": "git"},
+        timeout=5,
+    )
+    diff.raise_for_status()
+    assert diff.json() == []
     assert (state_dir / "data" / "opencode" / "opencode.db").is_file()
     assert (state_dir / "SKILL.md").is_file()
     assert (state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md").resolve() == state_dir / "SKILL.md"
 
 
-def test_opencode_state_dir_is_separate_from_workdir(tmp_path):
-    from archivebox.opencode import views
+def test_opencode_invalid_state_does_not_break_archivebox(admin_client, live_opencode):
+    from abx_plugins.plugins.opencode import runtime
 
-    workdir = tmp_path / "workdir"
-    state_dir = tmp_path / "state"
-    settings = views._settings(
-        {
-            "OPENCODE_WORKDIR": str(workdir),
-            "OPENCODE_STATE_DIR": str(state_dir),
-        },
-    )
-    views._ensure_project_files(settings)
+    runtime._stop_owned_process()
+    invalid_state = live_opencode.config.state_dir / "config"
+    invalid_state.rename(live_opencode.config.state_dir / "saved-config")
+    invalid_state.write_text("Preserve this file.")
 
-    assert settings["workdir"] == workdir
-    assert settings["opencode_dir"] == state_dir
-    assert settings["config_home"] == state_dir / "config"
-    assert settings["data_home"] == state_dir / "data"
-    assert settings["state_home"] == state_dir / "state"
-    editable_skill = state_dir / "SKILL.md"
-    loaded_skill = state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md"
-    assert editable_skill.exists()
-    assert loaded_skill.is_symlink()
-    assert loaded_skill.resolve() == editable_skill.resolve()
-    assert f"ArchiveBox collection directory: {settings['archivebox_data_dir']}" in editable_skill.read_text()
-    opencode_config = json.loads((state_dir / "config" / "opencode" / "opencode.jsonc").read_text())
-    assert opencode_config["model"] == "opencode/big-pickle"
-    assert opencode_config["snapshot"] is False
+    for url in ("/admin/agent", "/admin/agent/opencode/global/health"):
+        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 503
+        assert str(invalid_state).encode() not in response.content
+
+    stream = admin_client.get("/admin/agent/opencode/global/event", HTTP_HOST=ADMIN_TEST_HOST)
+    assert stream.status_code == 200
+
+    async def read_failure():
+        return b"".join([chunk async for chunk in stream.streaming_content])
+
+    assert asyncio.run(read_failure()) == b'event: error\ndata: {"error":"OpenCode unavailable"}\n\n'
+
+    for url in ("/health/", "/add/", "/admin/core/snapshot/"):
+        response = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
+        assert response.status_code == 200
+    assert invalid_state.read_text() == "Preserve this file."
 
 
 @pytest.mark.parametrize(
-    "existing_config",
+    ("damaged_file", "missing"),
     [
-        '{"model": "anthropic/claude-sonnet-4-5"}\n',
-        '{\n  // Keep the administrator-selected model.\n  "model": "anthropic/claude-sonnet-4-5",\n}\n',
-        '{\n  // Schema-only files are still user-owned.\n  "$schema": "https://opencode.ai/config.json",\n}\n',
+        ("runtime.py", True),
+        ("templates/agent.html", True),
+        ("templates/agent.html", False),
+        ("templates/navigation.html", False),
+        ("templates/add.html", False),
     ],
 )
-def test_opencode_preserves_existing_config(tmp_path, existing_config):
-    from archivebox.opencode import views
+def test_opencode_incomplete_install_does_not_break_archivebox(installed_opencode, tmp_path, damaged_file, missing):
+    import abx_plugins
 
-    state_dir = tmp_path / "state"
-    config_path = state_dir / "config" / "opencode" / "opencode.jsonc"
-    config_path.parent.mkdir(parents=True)
-    config_path.write_text(existing_config)
-
-    views._ensure_project_files(
-        views._settings(
-            {
-                "OPENCODE_WORKDIR": str(tmp_path / "workdir"),
-                "OPENCODE_STATE_DIR": str(state_dir),
-            },
-        ),
+    # Exercise a genuinely incomplete installation in a separate process;
+    # never alter the shared package or intercept Python imports.
+    site = tmp_path / "site"
+    installed = site / "abx_plugins"
+    shutil.copytree(Path(abx_plugins.__file__).parent, installed, ignore=shutil.ignore_patterns("__pycache__", "tests"))
+    damaged_path = installed / "plugins" / "opencode" / damaged_file
+    if missing:
+        damaged_path.unlink()
+    else:
+        damaged_path.write_text("{% invalid_template_tag %}")
+    expected_status = 200 if damaged_file in {"templates/navigation.html", "templates/add.html"} else 503
+    script = f"""
+import sys
+from pathlib import Path
+import abx_plugins
+from django.test import Client
+from django.contrib.auth import get_user_model
+assert Path(abx_plugins.__file__).is_relative_to({str(site)!r})
+user = get_user_model().objects.create_superuser(username='optional-service-test')
+client = Client(HTTP_HOST={ADMIN_TEST_HOST!r})
+client.force_login(user)
+for path in ('/health/', '/add/', '/admin/core/snapshot/'):
+    assert client.get(path).status_code == 200, path
+assert 'abx_plugins.plugins.opencode.runtime' not in sys.modules
+response = client.get('/admin/agent')
+assert response.status_code == {expected_status}, response.status_code
+if response.status_code == 503:
+    assert response.content == b'AI service unavailable. See server logs.'
+if {damaged_file != "runtime.py"!r}:
+    from abx_plugins.plugins.opencode import runtime
+    assert runtime._PROCESS is not None
+    assert runtime._PROCESS.poll() is None
+for path in ('/health/', '/add/', '/admin/core/snapshot/'):
+    assert client.get(path).status_code == 200, path
+print('OPTIONAL_SERVICE_FAILURE_ISOLATED')
+"""
+    result = run_archivebox_cmd(
+        ["shell", "-c", script],
+        cwd=installed_opencode.config.data_dir,
+        env={**installed_opencode.config.env, "PYTHONPATH": str(site)},
+        timeout=90,
     )
-
-    assert config_path.read_text() == existing_config
-
-
-def test_opencode_defaults_to_the_archivebox_collection():
-    from archivebox.opencode import views
-
-    settings = views._settings({})
-
-    assert settings["opencode_dir"] == settings["archivebox_data_dir"] / "opencode"
-    assert settings["workdir"] == settings["archivebox_data_dir"]
-    assert settings["timeout"] == 120
-
-
-def test_opencode_rewrites_vite_preload_assets():
-    from archivebox.opencode import views
-
-    body = b'const BL="modulepreload",UL=function(t){return"/"+t};const icon="/assets/sprite.svg#anthropic"'
-    rewritten = views._rewrite_text(body, {"origin": "http://127.0.0.1:4096"}).decode()
-
-    assert 'return"/"+t' not in rewritten
-    assert 'return"/admin/agent/opencode/"+t' in rewritten
-    assert '"/admin/agent/opencode/assets/sprite.svg#anthropic"' in rewritten
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "OPTIONAL_SERVICE_FAILURE_ISOLATED" in result.stdout

archivebox/tests/test_opencode_browser.py

@@ -0,0 +1,371 @@
+"""Exercise the real agent wrapper with Chromium's native storage failures."""
+
+import asyncio
+import json
+import os
+import re
+import subprocess
+
+import pytest
+import requests
+
+from .conftest import get_free_port, run_archivebox_cmd, start_archivebox_server, stop_archivebox_process
+from .test_opencode_agent import _set_archivebox_config
+from .test_opencode_agent import installed_opencode as installed_opencode
+from .test_opencode_agent import live_opencode as live_opencode
+from .test_opencode_agent import opencode_archive_config as opencode_archive_config
+from .test_server_security_browser import browser_runtime as browser_runtime
+
+
+pytestmark = pytest.mark.django_db(transaction=True)
+
+
+def test_agent_navigation_stays_inside_mount(agent_server, browser_runtime):
+    server_url, _, _ = agent_server
+    script = r"""
+const assert = require('node:assert/strict');
+const puppeteer = require('puppeteer');
+const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
+(async () => {
+  const browser = await puppeteer.launch({executablePath: config.chrome, headless: true,
+    args: ['--no-sandbox', '--disable-frame-rate-limit']});
+  try {
+    const page = await browser.newPage();
+    await page.setViewport({width: 1440, height: 900});
+    await page.goto(config.url + '/admin/login/', {waitUntil: 'domcontentloaded'});
+    await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
+    await page.locator('#login-form input[name="password"]').fill('test-password');
+    await Promise.all([page.waitForNavigation({waitUntil: 'domcontentloaded'}),
+      page.locator('#login-form input[type="submit"]').click()]);
+    assert.equal((await page.goto(config.url + '/admin/agent', {waitUntil: 'domcontentloaded'})).status(), 200);
+    await page.locator('#opencode-agent-welcome-dismiss').click();
+    const frame = await (await page.waitForSelector('iframe')).contentFrame();
+    await frame.waitForSelector('a[href*="/session"]');
+    const links = await frame.$$eval('a[href*="/session"]', nodes => nodes.map(node => node.getAttribute('href')));
+    assert.ok(links.length, 'OpenCode must expose session navigation');
+    for (const href of links) assert.ok(href.startsWith('/admin/agent/opencode/'), href);
+    await frame.locator('::-p-aria(New session[role="button"])').click();
+    await frame.waitForSelector('[contenteditable="true"]');
+    let navigation;
+    for (const link of await frame.$$('a[href*="/session"]')) {
+      if (await link.isVisible() && await link.evaluate(node => node.href) !== frame.url()) {
+        navigation = link;
+        break;
+      }
+    }
+    assert.ok(navigation, 'A visible link must navigate to another session route');
+    await navigation.click();
+    await frame.waitForSelector('[contenteditable="true"]');
+    assert.ok(new URL(frame.url()).pathname.startsWith('/admin/agent/opencode/'), frame.url());
+    const sessionUrl = frame.url();
+    assert.equal((await frame.goto(sessionUrl, {waitUntil: 'domcontentloaded'})).status(), 200);
+    await frame.waitForSelector('[contenteditable="true"]');
+    assert.ok(!(await frame.$eval('body', node => node.innerText)).includes('Something went wrong'));
+    // Exercise the actual public PTY API and native browser WebSocket. No
+    // intercepted traffic or replacement server: this runs a real shell.
+    const terminal = await frame.evaluate(async () => {
+      const base = location.origin + '/admin/agent/opencode';
+      const created = await fetch(base + '/pty', {
+        method: 'POST', headers: {'Content-Type': 'application/json'},
+        body: JSON.stringify({command: '/bin/sh', args: []}),
+      });
+      if (!created.ok) throw new Error('PTY create: ' + created.status);
+      const pty = await created.json();
+      try {
+        const authorized = await fetch(base + '/pty/' + pty.id + '/connect-token', {
+          method: 'POST', headers: {'x-opencode-ticket': '1'},
+        });
+        if (authorized.status !== 200) throw new Error('PTY ticket: ' + authorized.status);
+        const {ticket} = await authorized.json();
+        if (!ticket) throw new Error('PTY ticket missing');
+        return await new Promise((resolve, reject) => {
+          const url = new URL(base + '/pty/' + pty.id + '/connect');
+          url.searchParams.set('ticket', ticket);
+          url.protocol = url.protocol === 'https:' ? 'wss:' : 'ws:';
+          const socket = new WebSocket(url);
+          const timer = setTimeout(() => { socket.close(); reject(new Error('PTY output timed out')); }, 10000);
+          let output = '';
+          socket.onopen = () => socket.send("printf 'ABX_%s\\n' TERMINAL_OK\n");
+          socket.onmessage = async event => {
+            output += typeof event.data === 'string' ? event.data : await event.data.text();
+            if (output.includes('ABX_TERMINAL_OK')) {
+              clearTimeout(timer); socket.close(); resolve(output);
+            }
+          };
+          socket.onerror = () => { clearTimeout(timer); reject(new Error('PTY WebSocket failed')); };
+        });
+      } finally { await fetch(base + '/pty/' + pty.id, {method: 'DELETE'}); }
+    });
+    assert.ok(terminal.includes('ABX_TERMINAL_OK'), terminal);
+    assert.equal((await page.goto(config.url + '/add/', {waitUntil: 'domcontentloaded'})).status(), 200);
+    console.log('AGENT_NAVIGATION_OK');
+  } finally { await browser.close(); }
+})().catch(error => { console.error(error); process.exitCode = 1; });
+"""
+    result = subprocess.run(
+        [str(browser_runtime["node_binary"]), "-e", script],
+        input=json.dumps({"chrome": str(browser_runtime["chrome_binary"]), "url": server_url}),
+        env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
+        text=True,
+        capture_output=True,
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "AGENT_NAVIGATION_OK" in result.stdout
+
+
+@pytest.fixture
+def agent_server(installed_opencode, browser_runtime):
+    port = get_free_port()
+    url = f"http://localhost:{port}"
+    config = installed_opencode.config
+    _set_archivebox_config(
+        config.data_dir,
+        f"BASE_URL={url}",
+        "SERVER_SECURITY_MODE=safe-onedomain-nojsreplay",
+    )
+    user = run_archivebox_cmd(
+        [
+            "shell",
+            "-c",
+            "from django.contrib.auth import get_user_model; User = get_user_model(); User.objects.create_superuser(username='agent-browser-test', password='test-password'); User.objects.create_user(username='agent-regular-test', password='test-password', is_staff=True)",
+        ],
+        cwd=config.data_dir,
+        env=config.env,
+    )
+    assert user.returncode == 0, user.stderr or user.stdout
+    process = start_archivebox_server(config.data_dir, port=port, env=config.env, log_name="agent-browser-server.log")
+    try:
+        yield url, config.data_dir, process
+    finally:
+        if process.poll() is None:
+            stop_archivebox_process(process)
+
+
+def _login_cookie(server_url, username="agent-browser-test"):
+    session = requests.Session()
+    login_url = server_url + "/admin/login/"
+    page = session.get(login_url, timeout=10)
+    assert page.status_code == 200
+    token = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', page.text)
+    assert token is not None
+    response = session.post(
+        login_url,
+        data={"username": username, "password": "test-password", "csrfmiddlewaretoken": token[1]},
+        headers={"Referer": login_url},
+        allow_redirects=False,
+        timeout=10,
+    )
+    assert response.status_code == 302
+    return "; ".join(f"{key}={value}" for key, value in session.cookies.items())
+
+
+def test_agent_websocket_rejects_unauthorized_access(agent_server):
+    from websockets.asyncio.client import connect
+    from websockets.exceptions import InvalidStatus
+
+    server_url, _, _ = agent_server
+    admin_cookie = _login_cookie(server_url)
+    regular_cookie = _login_cookie(server_url, "agent-regular-test")
+
+    async def check_denials():
+        for path, cookie, origin in (
+            ("/admin/agent/opencode/pty/unknown/connect", "", server_url),
+            ("/admin/agent/opencode/pty/unknown/connect", regular_cookie, server_url),
+            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, "https://untrusted.example"),
+            ("/admin/agent/opencode/pty/unknown/connect", admin_cookie, None),
+            ("/health/", admin_cookie, server_url),
+        ):
+            with pytest.raises(InvalidStatus) as error:
+                async with connect(
+                    server_url.replace("http", "ws", 1) + path,
+                    origin=origin,
+                    additional_headers={"Cookie": cookie},
+                    proxy=None,
+                ):
+                    pytest.fail("Unauthorized WebSocket was accepted")
+            assert error.value.response.status_code == 403
+
+    asyncio.run(check_denials())
+    assert requests.get(server_url + "/health/", timeout=10).status_code == 200
+
+
+@pytest.mark.parametrize(
+    "mode,proxy_whitelist,use_cookie,http_status",
+    [
+        ("unsafe-onedomain-noadmin", "", True, 403),
+        ("safe-onedomain-nojsreplay", "127.0.0.1/32", False, 200),
+        ("safe-onedomain-nojsreplay", "192.0.2.0/24", False, 302),
+    ],
+)
+def test_agent_websocket_matches_http_security_policy(agent_server, live_opencode, mode, proxy_whitelist, use_cookie, http_status):
+    from urllib.parse import urlsplit
+    from websockets.asyncio.client import connect
+    from websockets.exceptions import InvalidStatus
+
+    server_url, data_dir, process = agent_server
+    cookie = _login_cookie(server_url)
+    created = requests.post(
+        server_url + "/admin/agent/opencode/pty",
+        headers={"Cookie": cookie, "Origin": server_url},
+        json={"command": "/bin/sh", "args": []},
+        timeout=10,
+    )
+    assert created.status_code == 200
+    pty_id = created.json()["id"]
+    stop_archivebox_process(process)
+    _set_archivebox_config(
+        data_dir,
+        f"SERVER_SECURITY_MODE={mode}",
+        f"REVERSE_PROXY_WHITELIST={proxy_whitelist}",
+        "REVERSE_PROXY_USER_HEADER=Remote-User",
+    )
+    restarted = start_archivebox_server(
+        data_dir,
+        port=urlsplit(server_url).port,
+        env=live_opencode.config.env,
+        log_name="agent-security-server.log",
+    )
+    headers = {"Cookie": cookie} if use_cookie else {"Remote-User": "agent-browser-test"}
+    try:
+        response = requests.get(server_url + "/admin/agent/opencode/global/health", headers=headers, allow_redirects=False, timeout=10)
+        assert response.status_code == http_status
+
+        async def check_socket():
+            connection = connect(
+                server_url.replace("http", "ws", 1) + f"/admin/agent/opencode/pty/{pty_id}/connect",
+                origin=server_url,
+                additional_headers=headers,
+                proxy=None,
+            )
+            if http_status != 200:
+                with pytest.raises(InvalidStatus) as error:
+                    async with connection:
+                        pytest.fail("Disabled control plane or untrusted proxy accepted a WebSocket")
+                assert error.value.response.status_code == 403
+                return
+            async with connection as socket, asyncio.timeout(10):
+                await socket.send("printf 'ABX_%s\\n' PROXY_OK\n")
+                output = ""
+                while "ABX_PROXY_OK" not in output:
+                    chunk = await socket.recv()
+                    output += chunk.decode() if isinstance(chunk, bytes) else chunk
+                assert "ABX_PROXY_OK" in output
+
+        asyncio.run(check_socket())
+        assert requests.get(server_url + "/health/", timeout=10).status_code == 200
+    finally:
+        deleted = requests.delete(live_opencode.settings["origin"] + f"/pty/{pty_id}", timeout=10)
+        assert deleted.status_code == 200
+        stop_archivebox_process(restarted)
+
+
+def test_agent_preserves_projects_and_survives_storage_failure(agent_server, browser_runtime):
+    server_url, data_dir, _ = agent_server
+    script = r"""
+const assert = require('node:assert/strict');
+const puppeteer = require('puppeteer');
+const config = JSON.parse(require('node:fs').readFileSync(0, 'utf8'));
+(async () => {
+  for (const disabled of [false, true]) {
+    const browser = await puppeteer.launch({
+      executablePath: config.chrome,
+      headless: true,
+      // Headless interaction must not depend on the desktop display clock.
+      args: ['--no-sandbox', '--disable-frame-rate-limit', ...(disabled ? ['--disable-local-storage'] : [])],
+    });
+    try {
+      const page = await browser.newPage();
+      await page.goto(new URL('/admin/login/', config.url).href, {waitUntil: 'domcontentloaded'});
+      await page.locator('#login-form input[name="username"]').fill('agent-browser-test');
+      await page.locator('#login-form input[name="password"]').fill('test-password');
+      assert.equal(await page.$eval('#login-form input[name="username"]', element => element.value), 'agent-browser-test');
+      assert.equal(await page.$eval('#login-form input[name="password"]', element => element.value), 'test-password');
+      await Promise.all([
+        page.waitForNavigation({waitUntil: 'domcontentloaded'}),
+        page.locator('#login-form input[type="submit"]').click(),
+      ]);
+      assert.ok(!page.url().includes('/admin/login/'), page.url());
+      assert.equal((await page.goto(config.url, {waitUntil: 'domcontentloaded'})).status(), 200);
+      const welcome = '#opencode-agent-welcome';
+      await page.waitForSelector(welcome, {visible: true});
+      if (disabled) {
+        assert.equal(await page.evaluate(() => {
+          try { return localStorage === null; }
+          catch (error) { if (error.name !== 'SecurityError') throw error; return true; }
+        }), true);
+      } else {
+        await page.click('#opencode-agent-welcome-dismiss');
+        await page.waitForSelector(welcome, {hidden: true});
+        await page.reload({waitUntil: 'domcontentloaded'});
+        assert.equal(await page.$eval(welcome, element => element.hidden), true);
+        const existing = {
+          list: [{type: 'http', http: {url: 'http://other.example'}}],
+          projects: {
+            local: [{worktree: '/other-project', expanded: false}],
+            'http://other.example': [{worktree: '/remote-project', expanded: true}],
+          },
+          lastProject: {'http://other.example': '/remote-project'},
+        };
+        await page.evaluate(value => localStorage.setItem('opencode.global.dat:server', JSON.stringify(value)), existing);
+        await page.reload({waitUntil: 'domcontentloaded'});
+        await page.waitForFunction(workdir => {
+          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
+          return state.projects.local?.some(project => project.worktree === workdir);
+        }, {}, config.workdir);
+        const saved = await page.evaluate(() => JSON.parse(localStorage.getItem('opencode.global.dat:server')));
+        assert.deepEqual(saved.list, existing.list);
+        assert.deepEqual(saved.projects['http://other.example'], existing.projects['http://other.example']);
+        assert.equal(saved.lastProject['http://other.example'], '/remote-project');
+        const projects = saved.projects.local;
+        assert.ok(Array.isArray(projects), JSON.stringify({url: page.url(), saved, existing}));
+        assert.deepEqual(projects.find(project => project.worktree === '/other-project'), {worktree: '/other-project', expanded: false});
+        assert.equal(projects.filter(project => project.worktree === config.workdir).length, 1);
+        await page.reload({waitUntil: 'domcontentloaded'});
+        assert.equal(await page.evaluate(workdir => {
+          const state = JSON.parse(localStorage.getItem('opencode.global.dat:server'));
+          return state.projects.local.filter(project => project.worktree === workdir).length;
+        }, config.workdir), 1);
+
+        // Fill the actual browser quota; do not replace or intercept Storage methods.
+        const failure = await page.evaluate(() => {
+          localStorage.clear();
+          let low = 0, high = 8 * 1024 * 1024;
+          while (low < high) {
+            const size = Math.ceil((low + high) / 2);
+            try { localStorage.setItem('padding', 'x'.repeat(size)); low = size; }
+            catch (error) { if (error.name !== 'QuotaExceededError') throw error; high = size - 1; }
+          }
+          try { localStorage.setItem('extra', '1'); return null; }
+          catch (error) { return error.name; }
+        });
+        assert.equal(failure, 'QuotaExceededError');
+        await page.reload({waitUntil: 'domcontentloaded'});
+        await page.waitForSelector(welcome, {visible: true});
+      }
+      await page.click('#opencode-agent-welcome-dismiss');
+      await page.waitForSelector(welcome, {hidden: true});
+      assert.equal((await page.goto(new URL('/add/', config.url).href, {waitUntil: 'domcontentloaded'})).status(), 200);
+    } finally {
+      await browser.close();
+    }
+  }
+  console.log('AGENT_STORAGE_FAILURE_ISOLATED');
+})().catch(error => { console.error(error); process.exitCode = 1; });
+"""
+    result = subprocess.run(
+        [str(browser_runtime["node_binary"]), "-e", script],
+        input=json.dumps(
+            {
+                "chrome": str(browser_runtime["chrome_binary"]),
+                "url": f"{server_url}/admin/agent",
+                "workdir": str(data_dir.resolve()),
+            },
+        ),
+        env={**os.environ, "NODE_PATH": browser_runtime["node_path"]},
+        text=True,
+        capture_output=True,
+        timeout=120,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    assert "AGENT_STORAGE_FAILURE_ISOLATED" in result.stdout

archivebox/tests/test_permissions.py

@@ -177,6 +177,40 @@ def test_permission_repairs_avoid_recursive_collection_and_abxpkg_chown():
     assert "chown -R" not in abxpkg_repairs
 
 
+def test_docker_entrypoint_uses_non_root_functional_checks_before_metadata_repairs():
+    entrypoint = (Path(__file__).parents[2] / "bin" / "docker_entrypoint.sh").read_text(encoding="utf-8")
+    ensure_dir = entrypoint.partition("ensure_dir() {")[2].partition("\n}")[0]
+    writable_probe = entrypoint.partition("assert_writable_dir() {")[2].partition("\n}")[0]
+
+    assert 'run_as_archivebox mkdir -p "$path"' in ensure_dir
+    assert 'path_is_writable "$path" && return 0' in ensure_dir
+    assert ensure_dir.index('path_is_writable "$path" && return 0') < ensure_dir.index('chown_if_needed "$path"')
+    assert ensure_dir.index('path_is_writable "$path" && return 0') < ensure_dir.index('chmod_if_possible "$path"')
+    assert 'run_as_archivebox test -w "$path"' in entrypoint
+    assert 'run_as_archivebox test -x "$path"' in entrypoint
+    assert 'run_as_archivebox mktemp "$path/.permissions_test.XXXXXX"' in writable_probe
+    assert 'run_as_archivebox rm -f "$probe"' in writable_probe
+    assert "du -s" not in entrypoint
+    assert "du -sb" not in entrypoint
+    assert 'find "$DATA_DIR"' not in entrypoint
+    assert 'find "$PERSONAS_DIR"' not in entrypoint
+    for line in entrypoint.splitlines():
+        assert not ("chown -R" in line and ("$DATA_DIR" in line or "$DATA_DIR/archive" in line))
+        assert not ("chmod -R" in line and ("$DATA_DIR" in line or "$DATA_DIR/archive" in line))
+
+
+def test_docker_entrypoint_keeps_root_pgid_and_validates_it_in_real_docker():
+    entrypoint = (Path(__file__).parents[2] / "bin" / "docker_entrypoint.sh").read_text(encoding="utf-8")
+    docker_validator = (Path(__file__).parents[2] / "bin" / "validate_docker_uid_gid.sh").read_text(encoding="utf-8")
+
+    assert 'TARGET_UID="${PUID:-$DETECTED_UID}"' in entrypoint
+    assert 'TARGET_GID="${PGID:-$DETECTED_GID}"' in entrypoint
+    assert '[[ "$TARGET_UID" == "0" ]]' in entrypoint
+    assert '[[ "$TARGET_GID" == "0" ]]' not in entrypoint.partition('[[ "$TARGET_UID" == "0" ]]')[2].partition("fi")[0]
+    assert 'run_case "explicit non-root PUID with PGID zero is supported"' in docker_validator
+    assert '"PUID=1201 PGID=0" "-" pass 1201 0' in docker_validator
+
+
 def test_root_handoff_never_selects_filesystem_root():
     from archivebox.config.permissions import root_data_dir_handoff_paths
 

archivebox/tests/test_recursive_crawl.py

@@ -635,7 +635,22 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
         for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
         if not (status in allowed_statuses or (plugin == "archivedotorg" and status == ArchiveResult.StatusChoices.FAILED))
     ]
-    assert not unexpected_results
+    # This fixture serves HTTP. TLSNotary must reject every plaintext document;
+    # a setup, timeout, or verifier error must not count as that expected refusal.
+    tlsnotary_results = [row for row in archive_results if row[3] == "tlsnotary"]
+    assert {row[1] for row in tlsnotary_results} == expected_urls
+    assert len(tlsnotary_results) == len(expected_urls)
+    with use_archivebox_db(initialized_archive):
+        for snapshot_id, _url, _depth, _plugin, hook_name, status, files, size, _output in tlsnotary_results:
+            assert status == ArchiveResult.StatusChoices.FAILED
+            assert not files
+            assert size == 0
+            process = ArchiveResult.objects.get(snapshot_id=snapshot_id, plugin="tlsnotary", hook_name=hook_name).process
+            assert process is not None
+            assert process.status == Process.StatusChoices.EXITED
+            assert process.exit_code == 1
+            assert process.stderr.strip() == "[tlsnotary] TLSNotary requires an HTTPS document"
+    assert not [result for result in unexpected_results if result["plugin"] != "tlsnotary"]
     ytdlp_results = [
         (url, status, output_str)
         for _snapshot_id, url, _depth, plugin, _hook_name, status, _files, _size, output_str in archive_results
@@ -682,19 +697,6 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(
     assert set(recursive_test_site["deep_urls"]).issubset(parsed_urls)
 
     assert processes
-    failed_hook_results = [
-        {
-            "url": url,
-            "depth": depth,
-            "plugin": plugin,
-            "hook_name": hook_name,
-            "status": status,
-            "output_str": output_str,
-        }
-        for _snapshot_id, url, depth, plugin, hook_name, status, _files, _size, output_str in archive_results
-        if status == ArchiveResult.StatusChoices.FAILED and plugin != "archivedotorg"
-    ]
-    assert not failed_hook_results
     assert all(status == Process.StatusChoices.EXITED for _id, _pwd, _cmd, status, _exit_code, _started_at, _ended_at in processes)
 
     intervals = []

archivebox/tests/test_resource_admission.py

@@ -0,0 +1,30 @@
+import os
+
+from archivebox.services.resource_admission import ResourceAdmission
+
+
+def test_cpu_admission_respects_process_affinity():
+    admission = ResourceAdmission()
+    available = len(os.sched_getaffinity(0)) if hasattr(os, "sched_getaffinity") else os.cpu_count()
+    assert 1 <= admission.snapshot_limit(100000) <= available
+    assert admission.snapshot_limit(1) == 1
+
+
+def test_memory_stall_counter_recovers_without_resetting_monitor():
+    # Real kernel PSI units: cumulative microseconds and ten-second average.
+    # Decision samples exercise the policy, not simulated hooks/processes.
+    admission = ResourceAdmission()
+    assert admission.observe_memory_stalls("host", 100, 0.0, 0.0) is False
+    assert admission.observe_memory_stalls("host", 400100, 20.0, 2.0) is True
+    assert admission.observe_memory_stalls("host", 400100, 20.0, 4.0) is False
+    assert admission.observe_memory_stalls("new-cgroup", 900, 20.0, 0.0) is True
+    assert admission.observe_memory_stalls("new-cgroup", 900, 20.0, 2.0) is False
+
+
+def test_transient_and_low_memory_stalls_do_not_starve_admission():
+    admission = ResourceAdmission()
+    for sample in range(20):
+        assert admission.observe_memory_stalls("host", sample * 2600, 0.13, sample * 2.0) is False
+    # One large interval alone is not sustained pressure; the rolling average
+    # must agree before deferring work.
+    assert admission.observe_memory_stalls("host", 1_000_000, 0.13, 40.0) is False

archivebox/tests/test_search.py

@@ -1,5 +1,6 @@
 import os
 import re
+import shutil
 import time
 from datetime import timedelta
 from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
@@ -745,6 +746,14 @@ class TestSearchBackendsE2E:
         )
         assert Path(binary_env["RIPGREP_BINARY"]).is_file()
         assert Path(binary_env["SONIC_BINARY"]).is_file()
+        # abxpkg may replace both env/bin and provider install paths on later
+        # hydration. Keep a real executable outside its managed lib tree so
+        # every index/query process uses the same on-disk KV format.
+        pinned_sonic_dir = initialized_archive / "pinned-sonic"
+        pinned_sonic_dir.mkdir()
+        sonic_binary = pinned_sonic_dir / "sonic"
+        shutil.copy2(Path(binary_env["SONIC_BINARY"]).resolve(strict=True), sonic_binary)
+        binary_env["SONIC_BINARY"] = str(sonic_binary)
 
         page_count = 23
         first_batch_count = 12
@@ -939,7 +948,7 @@ class TestSearchBackendsE2E:
                 )
                 assert create_result.returncode == 0, create_result.stderr or create_result.stdout
                 metadata_create_outputs.append(create_result.stdout)
-            from archivebox.core.models import Snapshot
+            from archivebox.core.models import ArchiveResult, Snapshot
             from archivebox.tests.test_orm_helpers import use_archivebox_db
 
             with use_archivebox_db(initialized_archive):
@@ -972,6 +981,17 @@ class TestSearchBackendsE2E:
                 timeout=180,
             )
             assert index_update.returncode == 0, index_update.stderr or index_update.stdout
+            with use_archivebox_db(initialized_archive):
+                sonic_results = list(
+                    ArchiveResult.objects.filter(
+                        plugin="search_backend_sonic",
+                        snapshot__url__in=[*first_wget_urls[:3], *second_wget_urls[:2]],
+                    ).values_list("snapshot__url", "status", "output_str", "process__stderr"),
+                )
+            assert {url for url, status, _, _ in sonic_results if status == "succeeded"} == {
+                *first_wget_urls[:3],
+                *second_wget_urls[:2],
+            }, (sonic_results, index_update.stdout, index_update.stderr)
 
             cli_backend_expectations = (
                 ("ripgrep", shared_content_needle, matrix_urls),
@@ -986,7 +1006,12 @@ class TestSearchBackendsE2E:
                     env={**env, "SEARCH_BACKEND_ENGINE": backend_name},
                     timeout=60,
                 )
-                assert backend_result.returncode == 0, backend_result.stderr or backend_result.stdout
+                sonic_log = initialized_archive / "logs" / "worker_sonic.log"
+                assert backend_result.returncode == 0, (
+                    backend_result.stderr or backend_result.stdout,
+                    sonic_log.read_text(encoding="utf-8", errors="replace") if backend_name == "sonic" and sonic_log.is_file() else "",
+                    str(sonic_binary),
+                )
                 backend_urls = [line.strip().strip('"') for line in backend_result.stdout.splitlines() if line.strip()]
                 assert set(backend_urls) == set(expected_urls), (backend_name, query, backend_result.stdout)
 
@@ -1264,3 +1289,75 @@ class TestSearchBackendsE2E:
             fixture_server.shutdown()
             fixture_server.server_close()
             fixture_thread.join(timeout=5)
+
+
+@pytest.mark.timeout(90)
+def test_live_public_search_disconnect_reaps_backend(initialized_archive):
+    """A real HTTP disconnect cancels a silent ripgrep process and its wrapper."""
+    import signal
+    from uuid import uuid4
+
+    import psutil
+
+    from archivebox.core.models import Snapshot
+    from archivebox.tests.test_orm_helpers import use_archivebox_db
+
+    rg = shutil.which("rg")
+    assert rg
+    query = "disconnect" + uuid4().hex
+    port = get_free_port()
+    env = cli_env(
+        live=True,
+        PUBLIC_INDEX="True",
+        PERMISSIONS="public",
+        SEARCH_BACKEND_ENGINE="ripgrep",
+        SEARCH_BACKEND_RIPGREP_ENABLED="True",
+        RIPGREP_BINARY=rg,
+        RIPGREP_ARGS_EXTRA='["--no-mmap", "--text", "--sort", "path"]',
+        RIPGREP_TIMEOUT="30",
+        ALLOWED_HOSTS="*",
+    )
+    created = run_archivebox_cmd(
+        ["snapshot", "create", "--status=sealed", "https://example.com/disconnect-test"],
+        cwd=initialized_archive,
+        env=env,
+    )
+    assert created.returncode == 0, created.stderr
+    with use_archivebox_db(initialized_archive):
+        snapshot = Snapshot.objects.get(url="https://example.com/disconnect-test")
+        directory = snapshot.output_dir / "dom"
+        directory.mkdir(parents=True, exist_ok=True)
+        (directory / "a.txt").write_text(query + "\n")
+        with (directory / "z.txt").open("wb") as large:
+            large.truncate(8 * 1024**3)
+            for offset in range(0, 8 * 1024**3, 1024**2):
+                large.seek(offset)
+                large.write(b"\n")
+    server = start_archivebox_server(initialized_archive, port=port, env=env, daemonize=False, log_name="disconnect-server.log")
+    children = []
+    try:
+        with requests.get(
+            f"http://127.0.0.1:{port}/public/search-stream/",
+            params={"q": query, "search_mode": "deep:ripgrep"},
+            headers={"Host": f"web.archivebox.localhost:{port}"},
+            stream=True,
+            timeout=(5, 10),
+        ) as response:
+            assert response.status_code == 200, response.text
+            lines = response.iter_lines()
+            assert int(next(lines).strip()) == 0
+            assert int(next(lines).strip()) == 1
+            children = [child for child in psutil.Process(server.pid).children(recursive=True) if query in " ".join(child.cmdline())]
+            assert len(children) == 2
+            for child in children:
+                if Path(child.exe()).name == Path(rg).name:
+                    child.send_signal(signal.SIGSTOP)
+        _, alive = psutil.wait_procs(children, timeout=5)
+        assert not alive, [(child.pid, child.status()) for child in alive]
+    finally:
+        for child in children:
+            try:
+                child.kill()
+            except psutil.NoSuchProcess:
+                pass
+        stop_archivebox_process(server)

archivebox/tests/test_serve_static.py

@@ -1,10 +1,26 @@
 from pathlib import Path
 
+import pytest
 from django.test import RequestFactory
 
 from archivebox.misc.serve_static import serve_static_with_byterange_support
 
 
+@pytest.mark.parametrize("filename", ["output.log", "hook.sh"])
+@pytest.mark.parametrize("byte_range", [None, "bytes=2-5"])
+def test_logs_and_shell_scripts_are_served_as_plain_text(tmp_path: Path, filename: str, byte_range: str | None):
+    content = b"# Heading\n\n- item\n- another item\n\n<literal>\n"
+    (tmp_path / filename).write_bytes(content)
+    request = RequestFactory().get(f"/{filename}", **({"HTTP_RANGE": byte_range} if byte_range else {}))
+
+    response = serve_static_with_byterange_support(request, filename, document_root=tmp_path)
+
+    assert response.status_code == (206 if byte_range else 200)
+    assert response["Content-Type"] == "text/plain; charset=utf-8"
+    assert response["Content-Disposition"] == f'inline; filename="{filename}"'
+    assert b"".join(response.streaming_content) == (content[2:6] if byte_range else content)
+
+
 def test_archive_file_response_uses_async_iterator_under_asgi(tmp_path: Path):
     output = tmp_path / "screenshot" / "output.png"
     output.parent.mkdir()

archivebox/tests/test_server_automatic_url.py

@@ -0,0 +1,61 @@
+"""Exercise automatic server addressing through real HTTP login pages."""
+
+import re
+
+import requests
+
+from .conftest import cli_env, get_free_port, run_archivebox_cmd, start_archivebox_server, stop_archivebox_process
+
+
+def test_safe_single_domain_login_does_not_require_base_url(tmp_path):
+    port = get_free_port()
+    env = cli_env(
+        disable_extractors=True,
+        BASE_URL="",
+        SERVER_SECURITY_MODE="safe-onedomain-nojsreplay",
+        ADMIN_USERNAME="automatic-url-test",
+        ADMIN_PASSWORD="Automatic-url-test-password-93!",
+        ALLOWED_HOSTS="*",
+    )
+    result = run_archivebox_cmd(["init", "--quick"], cwd=tmp_path, env=env, timeout=60)
+    assert result.returncode == 0, result.stderr or result.stdout
+    process = start_archivebox_server(tmp_path, port=port, env=env, daemonize=False, log_name="server.log")
+    try:
+        for hostname in ("localhost", "archivebox.localhost", "admin.archivebox.localhost", "192.168.1.20", "100.100.10.20"):
+            session = requests.Session()
+            response = session.get(
+                f"http://127.0.0.1:{port}/admin/login/",
+                headers={"Host": f"{hostname}:{port}"},
+                timeout=15,
+                allow_redirects=False,
+            )
+            assert response.status_code == 200, (hostname, response.status_code)
+            assert 'name="password"' in response.text, hostname
+            assert "base_url not set" not in response.text.lower(), hostname
+            assert "archivebox-setup-wizard" not in response.text, hostname
+            csrf = re.search(r'name="csrfmiddlewaretoken" value="([^"]+)"', response.text)
+            assert csrf, hostname
+            login = session.post(
+                f"http://127.0.0.1:{port}/admin/login/",
+                headers={"Host": f"{hostname}:{port}"},
+                data={
+                    "username": env["ADMIN_USERNAME"],
+                    "password": env["ADMIN_PASSWORD"],
+                    "csrfmiddlewaretoken": csrf[1],
+                    "next": "/admin/",
+                },
+                timeout=15,
+                allow_redirects=False,
+            )
+            assert login.status_code == 302, (hostname, login.status_code)
+            admin = session.get(
+                f"http://127.0.0.1:{port}/admin/",
+                headers={"Host": f"{hostname}:{port}"},
+                timeout=15,
+                allow_redirects=False,
+            )
+            assert admin.status_code == 200, (hostname, admin.status_code)
+            assert "base_url not set" not in admin.text.lower(), hostname
+            assert "archivebox-setup-wizard" not in admin.text, hostname
+    finally:
+        stop_archivebox_process(process)

archivebox/tests/test_server_security_browser.py

@@ -963,14 +963,13 @@ def test_archivewebpage_wacz_preview_serves_real_capture_frame(initialized_archi
             log_name="archivewebpage_server.log",
         )
         get_http_response(port, host=f"archivebox.localhost:{port}", path="/")
-        _cmd_result = run_archivebox_cmd(
+        command_result = run_archivebox_cmd(
             ["add", "--depth=0", "--max-urls=1", "--plugins=archivewebpage", url],
             cwd=initialized_archive,
             env=env,
             timeout=120,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, f"archivebox add failed:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
+        assert command_result.returncode == 0, f"archivebox add failed:\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
 
         capture = _get_archivewebpage_capture(initialized_archive, url)
         snapshot_host = f"{get_snapshot_subdomain(capture['snapshot_id'])}.archivebox.localhost:{port}"

archivebox/tests/test_snapshot_service.py

@@ -163,14 +163,15 @@ def test_snapshot_service_cli_add_seals_snapshot_and_writes_indexes(tmp_path, re
 
     port = get_free_port()
     env = cli_env(port=port, server=True, PLUGINS="wget", SAVE_WGET="True")
-    _cmd_result = run_archivebox_cmd(
+    command_result = run_archivebox_cmd(
         ["add", "--depth=0", "--plugins=wget", recursive_test_site["root_url"]],
         cwd=tmp_path,
         env=env,
         timeout=180,
     )
-    stdout, stderr, code = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-    assert code == 0, f"archivebox add failed with code {code}\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
+    assert command_result.returncode == 0, (
+        f"archivebox add failed with code {command_result.returncode}\nSTDOUT:\n{command_result.stdout}\nSTDERR:\n{command_result.stderr}"
+    )
 
     state = _snapshot_state(tmp_path, recursive_test_site["root_url"])
     snapshot_dir = state["snapshot_dir"]

archivebox/tests/test_takeover_util.py

@@ -3,6 +3,9 @@
 
 import os
 import json
+import re
+import shlex
+import shutil
 import signal
 import subprocess
 import sys
@@ -77,6 +80,78 @@ def _resolve_sonic_env(env: dict[str, str]) -> dict[str, str]:
     return payload
 
 
+def test_explicit_sonic_binary_is_used_by_real_search_worker(initialized_archive):
+    sonic_port = get_free_port()
+    lib_dir = initialized_archive / "lib"
+    env = cli_env(
+        live=True,
+        ABXPKG_LIB_DIR=str(lib_dir),
+        PLUGINS="search_backend_sonic",
+        SEARCH_BACKEND_ENGINE="sonic",
+        SEARCH_BACKEND_SONIC_PORT=str(sonic_port),
+    )
+    env.update(_resolve_sonic_env(env))
+    env["PATH"] = os.pathsep.join((str(Path(sys.executable).parent), env["PATH"]))
+    installed_binary = Path(env["SONIC_BINARY"]).resolve(strict=True)
+    managed_sonic_dir = lib_dir / "bash" / "bin"
+    managed_sonic_dir.mkdir(parents=True, exist_ok=True)
+    managed_binary = managed_sonic_dir / "sonic"
+    if installed_binary != managed_binary:
+        shutil.copy2(installed_binary, managed_binary)
+    config_result = run_archivebox_cmd(
+        ["config", "--set", f"SONIC_BINARY={managed_binary}"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert config_result.returncode == 0, config_result.stderr or config_result.stdout
+    pinned_sonic_dir = initialized_archive / "pinned-sonic"
+    pinned_sonic_dir.mkdir()
+    sonic_binary = pinned_sonic_dir / "sonic"
+    shutil.copy2(installed_binary, sonic_binary)
+    env["SONIC_BINARY"] = str(sonic_binary)
+
+    result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=env,
+        timeout=60,
+    )
+    assert result.returncode == 0, result.stderr or result.stdout
+    config_match = re.search(r"Using supervisord config file: (\S+)", result.stderr)
+    assert config_match, result.stderr
+    worker_config = Path(config_match.group(1)).parent / "workers" / "worker_sonic.conf"
+    worker_text = worker_config.read_text(encoding="utf-8")
+    command = next(line.partition("=")[2] for line in worker_text.splitlines() if line.startswith("command="))
+    assert shlex.split(command)[0] == str(sonic_binary), worker_text
+    from archivebox.workers.supervisord_util import _sonic_worker_bind_target
+
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": command}) == ("127.0.0.1", sonic_port)
+    renamed_binary = initialized_archive / "renamed-sonic-binary"
+    shutil.copy2(sonic_binary, renamed_binary)
+    renamed_command = shlex.join([str(renamed_binary), *shlex.split(command)[1:]])
+    assert _sonic_worker_bind_target({"name": "worker_sonic", "command": renamed_command}) == ("127.0.0.1", sonic_port)
+
+    relative_binary = pinned_sonic_dir / "sonic-alt"
+    shutil.copy2(sonic_binary, relative_binary)
+    relative_env = {
+        **env,
+        "SONIC_BINARY": relative_binary.name,
+        "PATH": os.pathsep.join((str(pinned_sonic_dir), env["PATH"])),
+        "SEARCH_BACKEND_SONIC_PORT": str(get_free_port()),
+    }
+    relative_result = run_archivebox_cmd(
+        ["list", "--search=contents", "--csv=url", "not-indexed"],
+        cwd=initialized_archive,
+        env=relative_env,
+        timeout=60,
+    )
+    assert relative_result.returncode == 0, relative_result.stderr or relative_result.stdout
+    relative_worker_text = worker_config.read_text(encoding="utf-8")
+    relative_command = next(line.partition("=")[2] for line in relative_worker_text.splitlines() if line.startswith("command="))
+    assert Path(shlex.split(relative_command)[0]).name == relative_binary.name, relative_worker_text
+
+
 def _archive_pages_for_sqlite_reindexing(data_dir: Path, env: dict[str, str], root_url: str) -> None:
     add_env = dict(env)
     add_env["SEARCH_BACKEND_ENGINE"] = "ripgrep"
@@ -312,14 +387,13 @@ def test_live_second_server_takes_over_existing_server_process(tmp_path, initial
         assert "A newer archivebox process took over the orchestrator, server" in first_text
         assert "Starting orchestrator, server" in second_text
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["status"],
             cwd=tmp_path,
             env=env,
             timeout=60,
         )
-        stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert returncode == 0, stderr or stdout
+        assert result.returncode == 0, result.stderr or result.stdout
 
         first_resumes = first_log.read_text(encoding="utf-8", errors="replace").count("Other newer archivebox process")
         stop_archivebox_process(second, signal.SIGTERM)
@@ -347,15 +421,14 @@ def test_live_update_index_only_does_not_take_over_server_runtime(tmp_path, init
         supervisor_pid_before = supervisor_pid_from_log(server_log)
         daphne_pid_before = worker_pid_from_log(server_log, "worker_daphne")
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--before=0"],
             cwd=tmp_path,
             env=env,
             timeout=90,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert pid_is_alive(supervisor_pid_before)
         assert pid_is_alive(daphne_pid_before)
@@ -396,15 +469,14 @@ def test_live_server_keeps_http_runtime_while_update_runs_real_sqlite_indexer(tm
         runner_spawn_text = "spawned: 'worker_runner' with pid"
         runner_spawn_count = supervisord_log.read_text(encoding="utf-8", errors="replace").count(runner_spawn_text)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--batch-size=1"],
             cwd=tmp_path,
             env=env,
             timeout=180,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
 
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert supervisor_pid_from_log(server_log) == supervisor_pid_before
         assert pid_is_alive(daphne_pid_before)
@@ -553,14 +625,13 @@ def test_live_repeated_server_startups_take_over_cleanly(tmp_path, initialized_a
                 wait_for_pid_to_disappear(daphne_pids[index - 1], timeout=15)
                 wait_for_pid_to_disappear(runner_pids[index - 1], timeout=15)
 
-            _cmd_result = run_archivebox_cmd(
+            result = run_archivebox_cmd(
                 ["status"],
                 cwd=tmp_path,
                 env=env,
                 timeout=60,
             )
-            stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-            assert returncode == 0, stderr or stdout
+            assert result.returncode == 0, result.stderr or result.stdout
 
         assert pid_is_alive(servers[-1].pid)
         assert all(pid_is_alive(server.pid) for server in servers)
@@ -598,14 +669,13 @@ def test_live_background_add_survives_server_exit_and_foreground_run_reclaims(tm
         server_log = server.log_path
         supervisor_pid_before = supervisor_pid_from_log(server_log)
 
-        _cmd_result = run_archivebox_cmd(
+        result = run_archivebox_cmd(
             ["update", "--index-only", "--batch-size=10"],
             cwd=tmp_path,
             env=env,
             timeout=90,
         )
-        update_stdout, update_stderr, update_returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
-        assert update_returncode == 0, update_stderr or update_stdout
+        assert result.returncode == 0, result.stderr or result.stdout
         assert pid_is_alive(server.pid)
         assert pid_is_alive(supervisor_pid_before)
         assert supervisor_pid_from_log(server_log) == supervisor_pid_before

archivebox/tests/test_ui_add_view.py

@@ -20,6 +20,23 @@ WEB_HOST = "web.archivebox.localhost:8000"
 ADMIN_HOST = "admin.archivebox.localhost:8000"
 
 
+ADD_FORM_DEFAULTS = {
+    "tag": "",
+    "depth": "0",
+    "max_urls": "0",
+    "crawl_max_size": "0",
+    "snapshot_max_size": "0",
+    "url_filters_allowlist": "",
+    "url_filters_denylist": "",
+    "notes": "",
+    "schedule": "",
+    "persona": "Default",
+    "permissions": "public",
+    "start_paused": "",
+    "config": "{}",
+}
+
+
 @pytest.fixture
 def admin_user(db):
     return User.objects.create_superuser(
@@ -140,22 +157,13 @@ def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/staff-config",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
             "crawl_timeout": "0",
             "timeout": "60",
-            "snapshot_max_size": "0",
             "delete_after": "0",
             "crawl_max_concurrent_snapshots": "1",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
-            "notes": "",
             "schedule": "daily",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
             "main_plugins": ["wget"],
             "plugin_config__wget__WGET_TIMEOUT": "77",
@@ -246,6 +254,7 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com\nhttps://cdn.example.com/asset.js",
             "tag": "alpha,beta",
             "depth": "1",
@@ -260,11 +269,6 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
             "url_filters_denylist": "cdn.example.com",
             "url_filters_only_new": "1",
             "notes": "Created from /add/",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -290,24 +294,12 @@ def test_add_view_creates_crawl_with_tag_and_url_filter_overrides(client, admin_
 
 def _csrf_test_add_data(url: str) -> dict[str, str]:
     return {
+        **ADD_FORM_DEFAULTS,
         "url": url,
-        "tag": "",
-        "depth": "0",
-        "max_urls": "0",
-        "crawl_max_size": "0",
         "crawl_timeout": "0",
         "timeout": "",
-        "snapshot_max_size": "0",
         "delete_after": "0",
         "crawl_max_concurrent_snapshots": "1",
-        "url_filters_allowlist": "",
-        "url_filters_denylist": "",
-        "notes": "",
-        "schedule": "",
-        "persona": "Default",
-        "permissions": "public",
-        "start_paused": "",
-        "config": "{}",
     }
 
 
@@ -337,24 +329,14 @@ def test_add_view_sanitizes_crawl_notes_before_safe_update(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/notes-xss",
-            "tag": "",
-            "depth": "0",
             "max_urls": "1",
-            "crawl_max_size": "0",
             "crawl_timeout": "0",
             "timeout": "",
-            "snapshot_max_size": "0",
             "delete_after": "0",
             "crawl_max_concurrent_snapshots": "1",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "notes": malicious_notes,
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -373,20 +355,8 @@ def test_add_view_unchecked_only_new_sets_crawl_override(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/rearchive",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -407,20 +377,10 @@ def test_add_view_selected_persona_wins_over_stale_config_override(client, admin
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/private",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
             "persona": "Private",
-            "permissions": "public",
-            "start_paused": "",
             "config": '{"DEFAULT_PERSONA": "Default"}',
         },
         HTTP_HOST=ADMIN_HOST,
@@ -447,20 +407,9 @@ def test_add_view_applies_plugin_config_overrides(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/plugin-config",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
             "main_plugins": ["wget"],
             "plugin_config__wget__WGET_ENABLED": "false",
             "plugin_config__wget__WGET_TIMEOUT": "77",
@@ -488,9 +437,8 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/public-safe",
-            "tag": "",
-            "depth": "0",
             "max_urls": "10",
             "crawl_max_size": "45mb",
             "crawl_timeout": "120",
@@ -503,8 +451,6 @@ def test_add_view_public_submission_ignores_plugin_and_custom_config(client, adm
             "url_filters_only_new": "1",
             "notes": "public add",
             "schedule": "daily",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
             "main_plugins": ["wget"],
             "plugin_config__twocaptcha__TWOCAPTCHA_API_KEY": "posted-token",
@@ -541,21 +487,9 @@ def test_add_view_queues_crawl_for_background_runner(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com",
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -575,21 +509,11 @@ def test_add_view_start_paused_creates_paused_crawl_without_snapshots(client, ad
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": "https://example.com/paused",
-            "tag": "",
             "depth": "1",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
             "start_paused": "on",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -610,6 +534,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": (
                 "https://sweeting.me,https://google.com\n"
                 "Notes: [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox), https://news.ycombinator.com\n"
@@ -617,20 +542,7 @@ def test_add_view_extracts_urls_from_mixed_text_input(client, admin_user):
                 '{"items":["https://example.com/three"]}\n'
                 "csv,https://example.com/four"
             ),
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )
@@ -656,21 +568,9 @@ def test_add_view_trims_trailing_punctuation_from_markdown_urls(client, admin_us
     response = client.post(
         reverse("add"),
         data={
+            **ADD_FORM_DEFAULTS,
             "url": ("Docs: https://github.com/ArchiveBox/ArchiveBox.\nIssue: https://github.com/abc?abc#234234?."),
-            "tag": "",
-            "depth": "0",
-            "max_urls": "0",
-            "crawl_max_size": "0",
-            "snapshot_max_size": "0",
-            "url_filters_allowlist": "",
-            "url_filters_denylist": "",
             "url_filters_only_new": "1",
-            "notes": "",
-            "schedule": "",
-            "persona": "Default",
-            "permissions": "public",
-            "start_paused": "",
-            "config": "{}",
         },
         HTTP_HOST=ADMIN_HOST,
     )

archivebox/tests/test_ui_add_view_runtime.py

@@ -431,6 +431,10 @@ def test_public_add_view_import_text_formats_preserve_metadata_and_resume_withou
         URL_ALLOWLIST=r"127\.0\.0\.1[:/].*",
     )
     create_admin_and_token(tmp_path)
+    # These tests deliberately stop the server while crawls are queued. Finish
+    # dependency installation first so shutdown cannot leave a half-built venv.
+    install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
+    assert install_result.returncode == 0, install_result.stderr or install_result.stdout
 
     try:
         start_archivebox_server(tmp_path, env=env, port=port)
@@ -536,6 +540,10 @@ def test_public_add_view_rejects_file_path_and_shell_injection_payloads(tmp_path
         URL_ALLOWLIST=r"example\.com|example\.org|iana\.org|www\.iana\.org",
     )
     create_admin_and_token(tmp_path)
+    # These tests deliberately stop the server while crawls are queued. Finish
+    # dependency installation first so shutdown cannot leave a half-built venv.
+    install_result = run_archivebox_cmd(["install", *env["PLUGINS"].split(",")], cwd=tmp_path, env=env, timeout=180)
+    assert install_result.returncode == 0, install_result.stderr or install_result.stdout
 
     try:
         start_archivebox_server(tmp_path, env=env, port=port)

archivebox/tests/test_ui_admin_config_widget.py

@@ -368,6 +368,19 @@ def test_configured_onedomain_base_url_warns_on_admin_alias():
     assert get_base_url_mismatch_context(request, config)["mode"] == "base_url_mismatch"
 
 
+def test_configured_base_url_mismatch_banner_hidden_on_public_web_alias():
+    config = get_config(include_machine=False).model_copy(
+        update={"BASE_URL": "https://archivebox.example.test", "SERVER_SECURITY_MODE": "safe-onedomain-nojsreplay"},
+    )
+    request = RequestFactory().get("/public/", secure=True, HTTP_HOST="web.archivebox.example.test")
+    request.user = AnonymousUser()
+
+    # The underlying mismatch remains visible to routing/security code; only
+    # this banner is suppressed on the conventional read-only web alias.
+    assert get_base_url_mismatch_context(request, config)["mode"] == "base_url_mismatch"
+    assert system_warnings_banner({"CONFIG": config, "request": request})["mode"] != "base_url_mismatch"
+
+
 def test_auto_onedomain_mode_is_not_reported_as_unsafe():
     config = get_config(include_machine=False).model_copy(
         update={"BASE_URL": "http://archivebox.example.test", "SERVER_SECURITY_MODE": "auto"},

archivebox/tests/test_ui_admin_crawl.py

@@ -14,6 +14,7 @@ pytestmark = pytest.mark.django_db
 
 
 class TestCrawlScheduleAdmin:
+    @pytest.mark.django_db(transaction=True)
     def test_crawlschedule_change_view_renders_and_saves(self, client, admin_user, crawl):
         from archivebox.crawls.models import CrawlSchedule
 
@@ -35,24 +36,33 @@ class TestCrawlScheduleAdmin:
         assert b"No Crawls yet..." not in get_response.content
         assert b"No Snapshots yet..." not in get_response.content
 
-        post_response = client.post(
-            change_url,
-            {
-                "label": "Morning crawl",
-                "notes": "updated",
-                "schedule": "0 8 * * *",
-                "template": str(crawl.pk),
-                "created_by": str(admin_user.pk),
-                "_save": "Save",
-            },
-            HTTP_HOST=ADMIN_TEST_HOST,
-        )
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext
+
+        previous_mode = getattr(connection, "transaction_mode", None)
+        with CaptureQueriesContext(connection) as queries:
+            post_response = client.post(
+                change_url,
+                {
+                    "label": "Morning crawl",
+                    "notes": "updated",
+                    "schedule": "0 8 * * *",
+                    "template": str(crawl.pk),
+                    "created_by": str(admin_user.pk),
+                    "_save": "Save",
+                },
+                HTTP_HOST=ADMIN_TEST_HOST,
+            )
+        if connection.vendor == "sqlite":
+            assert any(query["sql"] == "BEGIN IMMEDIATE" for query in queries)
+            assert connection.transaction_mode == previous_mode
 
         assert post_response.status_code == 302
         schedule.refresh_from_db()
         assert schedule.label == "Morning crawl"
         assert schedule.notes == "updated"
         assert schedule.schedule == "0 8 * * *"
+        assert schedule.is_enabled is False
         assert schedule.template_id == crawl.pk
         assert schedule.created_by_id == admin_user.pk
 

archivebox/tests/test_ui_admin_persona.py

@@ -127,6 +127,10 @@ def test_persona_admin_add_view_renders_import_ui(admin_client):
     assert b"auth.json" in response.content
     assert b"Plugin Config" in response.content
     assert b'name="plugin_config__wget__WGET_TIMEOUT"' in response.content
+    assert b'<fieldset id="plugin-config" class="module aligned wide persona-plugin-config">' in response.content
+    assert b'href="https://archivebox.github.io/archivebox-browser-extension/"' in response.content
+    assert b"import cookies into this persona from another browser" in response.content
+    assert "wide" in response.context["adminform"].fieldsets[0][1]["classes"]
 
 
 def test_import_persona_from_source_copies_profile_without_browser_export(admin_user):
@@ -179,6 +183,42 @@ def test_persona_admin_add_post_runs_shared_importer(admin_client, admin_user):
     assert not persona.AUTH_STORAGE_FILE
 
 
+def test_persona_admin_unset_cookie_paths_stay_unset(admin_client, admin_user):
+    from html.parser import HTMLParser
+
+    paths = {}
+
+    class CookiePathInputs(HTMLParser):
+        def handle_starttag(self, tag, attrs):
+            field = dict(attrs)
+            name = field.get("name", "")
+            if tag == "input" and name.startswith("plugin_config__") and name.endswith(("__COOKIES_FILE", "__AUTH_STORAGE_FILE")):
+                paths[name] = field.get("value", "")
+
+    url = reverse("admin:personas_persona_add")
+    page = admin_client.get(url, HTTP_HOST=ADMIN_TEST_HOST)
+    CookiePathInputs().feed(page.content.decode())
+    assert paths
+    assert all(value == "" for value in paths.values())
+    response = admin_client.post(
+        url,
+        {
+            "name": "UnsetCookiePaths",
+            "created_by": str(admin_user.pk),
+            "permissions": "private",
+            "config": "{}",
+            "import_mode": "none",
+            "_save": "Save",
+            **paths,
+        },
+        HTTP_HOST=ADMIN_TEST_HOST,
+    )
+    assert response.status_code == 302
+    config = Persona.objects.get(name="UnsetCookiePaths").config
+    assert "COOKIES_FILE" not in config
+    assert "AUTH_STORAGE_FILE" not in config
+
+
 def test_persona_admin_saves_typed_plugin_config(admin_client, admin_user):
     add_response = admin_client.get(reverse("admin:personas_persona_add"), HTTP_HOST=ADMIN_TEST_HOST)
     add_form = add_response.context["adminform"].form

archivebox/tests/test_ui_admin_snapshot.py

@@ -607,7 +607,14 @@ class TestSnapshotProgressStats:
 
         assert result.embed_path_db() is None
 
-    def test_embed_path_db_prefers_valid_output_str_over_first_output_file(self, snapshot, real_hash_projection):
+    @pytest.mark.parametrize(
+        "output_str",
+        [
+            ("wget/example.com/index.html"),
+            (""),
+        ],
+    )
+    def test_embed_path_db_selects_html_output(self, snapshot, real_hash_projection, output_str):
         from archivebox.core.models import ArchiveResult
 
         output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
@@ -619,32 +626,7 @@ class TestSnapshotProgressStats:
         _process, result = real_hash_projection
         ArchiveResult.objects.filter(pk=result.pk).update(
             plugin="wget",
-            output_str="wget/example.com/index.html",
-            output_files={
-                "example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
-                "example.com/index.html": {
-                    "size": (Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").stat().st_size,
-                    "mimetype": "text/html",
-                },
-            },
-        )
-        result.refresh_from_db()
-
-        assert result.embed_path_db() == "wget/example.com/index.html"
-
-    def test_embed_path_db_scores_output_files_instead_of_using_first_entry(self, snapshot, real_hash_projection):
-        from archivebox.core.models import ArchiveResult
-
-        output_dir = Path(snapshot.output_dir) / "wget" / "example.com" / "assets" / "css"
-        output_dir.mkdir(parents=True, exist_ok=True)
-        (Path(snapshot.output_dir) / "wget" / "example.com" / "index.html").parent.mkdir(parents=True, exist_ok=True)
-        shutil.copyfile(REPO_ROOT / "README.md", Path(snapshot.output_dir) / "wget" / "example.com" / "index.html")
-        shutil.copyfile(REPO_ROOT / "archivebox" / "templates" / "static" / "bootstrap.min.css", output_dir / "mobile.css")
-
-        _process, result = real_hash_projection
-        ArchiveResult.objects.filter(pk=result.pk).update(
-            plugin="wget",
-            output_str="",
+            output_str=output_str,
             output_files={
                 "example.com/assets/css/mobile.css": {"size": (output_dir / "mobile.css").stat().st_size, "mimetype": "text/css"},
                 "example.com/index.html": {

archivebox/tests/test_ui_live_progress.py

@@ -104,6 +104,74 @@ def real_crawl_setup_process(snapshot, hermetic_lib_dir):
 
 
 class TestLiveProgressView:
+    def test_collection_summary_reports_stored_sizes(self, client, admin_user, snapshot):
+        import time
+        from django.core.cache import cache
+        from django.db.models import Count, Sum
+        from archivebox.core.models import Snapshot
+
+        Snapshot.objects.filter(pk=snapshot.pk).update(output_size=12345)
+        expected = Snapshot.objects.aggregate(snapshots=Count("*"), bytes=Sum("output_size"))
+        cache.delete_many(["progress-collection:all", "progress-collection:all:refresh"])
+        client.force_login(admin_user)
+        deadline = time.monotonic() + 5
+        while True:
+            response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+            assert response.status_code == 200
+            summary = response.json().get("collection")
+            if summary is not None or time.monotonic() >= deadline:
+                break
+            time.sleep(0.05)
+        assert summary is not None
+        assert summary["snapshots"] == expected["snapshots"]
+        assert summary["bytes"] == expected["bytes"]
+        from django.db import connection
+        from django.test.utils import CaptureQueriesContext
+
+        if connection.vendor == "sqlite":
+            with connection.cursor() as cursor:
+                cursor.execute("EXPLAIN QUERY PLAN SELECT COUNT(*), SUM(output_size) FROM core_snapshot")
+                plan = " ".join(str(row) for row in cursor.fetchall())
+            assert "COVERING INDEX" in plan, plan
+
+        timings = []
+        with CaptureQueriesContext(connection) as queries:
+            for _ in range(20):
+                start = time.perf_counter()
+                response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+                timings.append((time.perf_counter() - start) * 1000)
+                assert response.status_code == 200
+                assert response.json()["collection"] == summary
+        assert not any('AS "bytes"' in query["sql"] for query in queries)
+        print(f"cached progress response: median={sorted(timings)[10]:.1f}ms max={max(timings):.1f}ms")
+
+    def test_collection_summary_is_scoped_to_staff_owner(self, client, admin_user, snapshot):
+        import time
+        from django.core.cache import cache
+        from archivebox.core.models import Snapshot
+
+        admin_user.is_superuser = False
+        admin_user.save(update_fields=["is_superuser"])
+        from django.contrib.auth import get_user_model
+
+        # Give the snapshot another owner; staff must not learn its size/count.
+        snapshot.crawl.created_by = get_user_model().objects.create_user(username="other-summary-owner")
+        snapshot.crawl.save(update_fields=["created_by"])
+        Snapshot.objects.filter(pk=snapshot.pk).update(output_size=12345)
+        cache.delete_many([f"progress-collection:{admin_user.pk}", f"progress-collection:{admin_user.pk}:refresh"])
+        client.force_login(admin_user)
+        deadline = time.monotonic() + 5
+        while True:
+            response = client.get(reverse("live_progress"), {"collection": "1"}, HTTP_HOST=ADMIN_TEST_HOST)
+            assert response.status_code == 200
+            summary = response.json().get("collection")
+            if summary is not None or time.monotonic() >= deadline:
+                break
+            time.sleep(0.05)
+        assert summary is not None
+        assert summary["snapshots"] == 0
+        assert summary["bytes"] == 0
+
     def test_live_progress_rejects_unauthenticated_unscoped_request(self, client):
         response = client.get(reverse("live_progress"), HTTP_HOST=ADMIN_TEST_HOST)
 

archivebox/tests/test_urls.py

@@ -1431,7 +1431,7 @@ class TestUrlRouting:
             [
                 "add",
                 "--depth=0",
-                "--plugins=consolelog,screenshot,chrome_mhtml",
+                "--plugins=consolelog,screenshot,pdf,chrome_mhtml",
                 console_source_url,
             ],
             cwd=self.data_dir,
@@ -1475,6 +1475,14 @@ class TestUrlRouting:
             resp = client.get(f"/{screenshot_rel}?preview=1", HTTP_HOST=snapshot_host)
             assert resp.status_code == 200
             assert resp["Content-Type"].startswith("text/html")
+            assert f"frame-ancestors 'self' http://{web_host}" in resp["Content-Security-Policy"]
+
+            resp = client.get("/pdf/output.pdf?preview=1", HTTP_HOST=snapshot_host)
+            assert resp.status_code == 200
+            pdf_preview = response_body(resp).decode("utf-8")
+            assert '<iframe ' in pdf_preview
+            assert '<embed ' not in pdf_preview
+            assert "object-src 'none'" in resp["Content-Security-Policy"]
 
             root_screenshot = screenshot_file.read_bytes()
             import shutil
@@ -1507,6 +1515,58 @@ class TestUrlRouting:
             """.replace("__CONSOLE_MARKER__", console_marker),
         )
 
+        # Exercise the same real captured files with each request-time posture,
+        # including both auto branches and an HTTPS canonical reverse-proxy URL.
+        for mode, base in (
+            ("auto", "http://archivebox.localhost:8000"),
+            ("auto", "http://archivebox.example:8000"),
+            ("safe-subdomains-fullreplay", "https://archivebox.example"),
+            ("safe-onedomain-nojsreplay", "http://archivebox.example:8000"),
+            ("unsafe-onedomain-noadmin", "http://archivebox.example:8000"),
+            ("danger-onedomain-fullreplay", "http://archivebox.example:8000"),
+        ):
+            self._run(
+                """
+                from urllib.parse import urlsplit
+                from archivebox.config.common import get_request_config
+                snapshot = get_snapshot()
+                client = Client()
+                for path in ('screenshot/screenshot.png', 'pdf/output.pdf', 'chrome_mhtml/snapshot.mhtml'):
+                    target = urlsplit(build_snapshot_url(str(snapshot.id), path))
+                    response = client.get(target.path + '?preview=1', HTTP_HOST=target.netloc)
+                    assert response.status_code == 200, (target, response.status_code)
+                    csp = response['Content-Security-Policy']
+                    config = get_request_config(response.wsgi_request)
+                    origins = tuple(dict.fromkeys((get_web_base_url(config=config), get_admin_base_url(config=config))))
+                    ancestors = next(d.strip() for d in csp.split(';') if d.strip().startswith('frame-ancestors'))
+                    assert ancestors == "frame-ancestors 'self' " + ' '.join(origins), ancestors
+                    assert '*' not in ancestors  # no unrelated snapshots/sites
+                    assert "object-src 'none'" in csp
+                    assert "form-action 'none'" in csp
+                    assert "base-uri 'none'" in csp
+                    assert "script-src 'self'" in csp  # trusted viewer can run
+                    if path.endswith('.mhtml'):
+                        assert 'sandbox="allow-popups allow-popups-to-escape-sandbox allow-same-origin"' in response_body(response).decode()
+
+                dangerous, _, _ = write_replay_fixtures(snapshot)
+                target = urlsplit(build_snapshot_url(str(snapshot.id), dangerous))
+                response = client.get(target.path, HTTP_HOST=target.netloc)
+                assert response.status_code == 200
+                config = get_request_config(response.wsgi_request)
+                csp = response.headers.get('Content-Security-Policy', '')
+                assert ("script-src 'none'" in csp) == config.SHOULD_NEUTER_RISKY_REPLAY
+                assert ('sandbox;' in csp) == config.SHOULD_NEUTER_RISKY_REPLAY
+                admin = client.get('/admin/login/', HTTP_HOST=get_admin_host())
+                assert admin.status_code == (200 if config.CONTROL_PLANE_ENABLED else 403)
+                if not config.CONTROL_PLANE_ENABLED:
+                    assert client.get('/api/v1/docs', HTTP_HOST=get_api_host()).status_code == 403
+                    assert client.post('/public/', HTTP_HOST=get_web_host()).status_code == 403
+                print('OK')
+                """,
+                mode=mode,
+                env_overrides={"BASE_URL": base},
+            )
+
     def test_api_available_on_admin_and_api_hosts(self) -> None:
         self._run(
             """

archivebox/workers/supervisord_util.py

@@ -47,7 +47,6 @@ _SUPERVISORD_ERRORS = (XmlRpcError, OSError, RuntimeError, TimeoutError)
 _PROCESS_STATE_ERRORS = (DatabaseError, OSError, RuntimeError, ValueError, psutil.Error)
 MIN_SERVER_WORKER_AVAILABLE_MEMORY_BYTES = 256 * 1024 * 1024
 MIN_DEPENDENCY_INSTALL_AVAILABLE_MEMORY_BYTES = MIN_SERVER_WORKER_AVAILABLE_MEMORY_BYTES
-MIN_CRAWL_AVAILABLE_MEMORY_BYTES = 512 * 1024 * 1024
 
 
 def _shell_join(args: list[str]) -> str:
@@ -118,16 +117,6 @@ def require_dependency_install_memory(available_bytes: int | None = None) -> Non
     )
 
 
-def require_crawl_memory(available_bytes: int | None = None) -> None:
-    available_bytes = effective_available_memory_bytes() if available_bytes is None else available_bytes
-    _require_available_memory(
-        "archive a crawl",
-        "    No crawl, runner, or Sonic workers were started.",
-        available_bytes,
-        MIN_CRAWL_AVAILABLE_MEMORY_BYTES,
-    )
-
-
 def archivebox_cmd(*args: str) -> list[str]:
     return [str(resolve_env_binary("archivebox")), *args]
 
@@ -405,7 +394,7 @@ def is_port_in_use(host: str, port: int) -> bool:
 def _sonic_worker_bind_target(worker: dict[str, str]) -> tuple[str, int] | None:
     """Read the plugin-owned Sonic config before starting its supervisord worker."""
     command = shlex.split(worker.get("command") or "")
-    if not command or Path(command[0]).name != "sonic" or "-c" not in command:
+    if worker.get("name") != "worker_sonic" or not command or "-c" not in command:
         return None
     config_index = command.index("-c") + 1
     if config_index >= len(command):
@@ -1189,6 +1178,25 @@ def build_server_worker_plan(*, config, host: str, port: str, debug: bool, reloa
             log_files.append(str(sonic_worker["stdout_logfile"]))
 
     workers = [(server_worker, False), *bg_workers]
+    if os.environ.get("DISPLAY") and (config.IN_DOCKER or os.environ.get("ARCHIVEBOX_VNC_PERSONA")):
+        vnc_persona = os.environ.get("ARCHIVEBOX_VNC_PERSONA") or config.DEFAULT_PERSONA
+        workers.append(
+            (
+                {
+                    "name": "worker_vnc_browser",
+                    "command": _shell_join(archivebox_cmd("persona", "open", vnc_persona)),
+                    "autostart": "false",
+                    "autorestart": "false",
+                    "stopasgroup": "true",
+                    "killasgroup": "true",
+                    "stopwaitsecs": "30",
+                    "stdout_logfile": "logs/worker_vnc_browser.log",
+                    "redirect_stderr": "true",
+                },
+                False,
+            ),
+        )
+        log_files.append("logs/worker_vnc_browser.log")
     components = runtime_components_for_worker_names([worker["name"] for worker, _lazy in workers])
     return workers, log_files, components
 
@@ -1261,11 +1269,18 @@ def tail_multiple_worker_logs(log_files: list[str], follow=True, proc=None, keep
 
     print()
 
+    # Log display needs a short refresh interval, but each ownership check
+    # queries the database and validates OS processes. Check leadership once
+    # per second even when workers continuously produce log output.
+    next_ownership_check = 0.0
     try:
         while follow:
-            if keep_running is not None and not keep_running():
-                print("\n[newer ArchiveBox process is now running the orchestrator and server]")
-                return "transferred"
+            now = time.monotonic()
+            if keep_running is not None and now >= next_ownership_check:
+                if not keep_running():
+                    print("\n[newer ArchiveBox process is now running the orchestrator and server]")
+                    return "transferred"
+                next_ownership_check = time.monotonic() + 1.0
 
             # Check if the monitored process has exited
             if proc is not None and proc.poll() is not None:
@@ -1311,7 +1326,22 @@ def get_sonic_supervisord_worker_from_plugin(config) -> dict[str, str] | None:
             raise
         return None
 
+    explicit_binary = os.environ.get("SONIC_BINARY")
+    requested_binary = Path(explicit_binary or str(config.SONIC_BINARY)).expanduser()
+    if explicit_binary:
+        # Machine.config can outrank the process environment; an explicit
+        # executable must win before the plugin hydrates its binary provider.
+        config = config.model_copy(update={"SONIC_BINARY": str(requested_binary)})
+
     worker = get_sonic_supervisord_worker(config)
+    if worker is not None:
+        if requested_binary.is_absolute():
+            # The plugin may project an explicit binary through env/bin/sonic.
+            # That symlink can be retargeted to a different Sonic release,
+            # whose on-disk index format may be incompatible with this one.
+            command = shlex.split(worker["command"])
+            command[0] = str(requested_binary.resolve(strict=True))
+            worker["command"] = shlex.join(command)
     return cast(dict[str, str] | None, worker)
 
 

bin/collect_ui_screenshots.sh

@@ -193,6 +193,22 @@ echo "[*] Creating a temporary screenshot admin"
 )
 CREATED_TEMP_USER=1
 
+# Configure the optional UI before the server snapshots its startup config.
+OPENCODE_PORT="$(uv run --no-cache --project "$REPO_DIR" python - <<'PYPORT'
+import socket
+
+with socket.socket() as sock:
+    sock.bind(("127.0.0.1", 0))
+    print(sock.getsockname()[1])
+PYPORT
+)"
+(
+    cd "$DATA_DIR"
+    uv run --no-cache --project "$REPO_DIR" archivebox config --set \
+        OPENCODE_ENABLED=True "OPENCODE_PORT=$OPENCODE_PORT" TLSNOTARY_ENABLED=True
+    uv run --no-cache --project "$REPO_DIR" archivebox install opencode --binproviders=env,pnpm
+)
+
 echo "[*] Starting ArchiveBox on port $PORT"
 (
     cd "$DATA_DIR"
@@ -248,6 +264,10 @@ while [[ "$capture_index" -lt "${#VIEWS[@]}" ]]; do
     capture_index=$((capture_index + 1))
     slug="$(printf '%s' "$name" | tr '[:upper:]' '[:lower:]' | tr -cs '[:alnum:]' '-' | sed 's/^-//; s/-$//')"
     expected_plugin=""
+    expected_frame_text=""
+    if [[ "$capture_mode" == wait-frame-text:* ]]; then
+        expected_frame_text="${capture_mode#wait-frame-text:}"
+    fi
     if [[ "$name" == "Snapshot View ("*")" && "$capture_mode" != "live-progress" && "$capture_mode" != "snapshot-collapsed" ]]; then
         expected_plugin="${name#Snapshot View (}"
         expected_plugin="${expected_plugin%)}"
@@ -407,6 +427,7 @@ PY
                     SCREENSHOT_VARIANTS_JSON="$output_variants" \
                     SCREENSHOT_COLLAPSE_FILTERS=1 \
                     SCREENSHOT_EXPECT_PLUGIN="$expected_plugin" \
+                    SCREENSHOT_EXPECT_FRAME_TEXT="$expected_frame_text" \
                     node "$REPO_DIR/bin/take_screenshot.js" "$url" "$screenshot_path" >"$capture_dir/report.json"
                 view_timing_report="$capture_dir/report.json"
                 timing_report_path="$view_timing_report"
@@ -494,21 +515,6 @@ PY
             CREATE_WEBHOOK="$CREATE_WEBHOOK" \
             node "$REPO_DIR/bin/setup_ui_screenshot_data.js" "$ADMIN_BASE_URL" "$USERNAME"
 
-        OPENCODE_PORT="$(uv run --no-cache --project "$REPO_DIR" python - <<'PY'
-import socket
-
-with socket.socket() as sock:
-    sock.bind(("127.0.0.1", 0))
-    print(sock.getsockname()[1])
-PY
-)"
-        (
-            cd "$DATA_DIR"
-            OPENCODE_PORT="$OPENCODE_PORT" uv run --no-cache --project "$REPO_DIR" archivebox manage shell --no-imports -c \
-                'import os; from archivebox.machine.models import Machine; Machine.from_json({"config": {"OPENCODE_ENABLED": True, "OPENCODE_PORT": int(os.environ["OPENCODE_PORT"])}})'
-            uv run --no-cache --project "$REPO_DIR" archivebox install opencode --binproviders=env,pnpm
-        )
-
         SWEETING_CAPTURE_STARTED_AT="$( (
             cd "$DATA_DIR"
             uv run --no-cache --project "$REPO_DIR" archivebox manage shell --no-imports -c \
@@ -561,7 +567,7 @@ PY
         VIEWS+=(
             "Add URLs|$ADMIN_BASE_URL/add/|/add/|archivebox/core/views.py"
             "Admin dashboard|$ADMIN_BASE_URL/admin/|/admin/|archivebox/core/admin_site.py"
-            "AI agent|$ADMIN_BASE_URL/admin/agent/|/admin/agent/|abx_plugins/plugins/opencode/views.py|wait-text:ArchiveBox AI Agent"
+            "AI agent|$ADMIN_BASE_URL/admin/agent/|/admin/agent/|archivebox/opencode/views.py|wait-text:ArchiveBox AI Agent"
             "Snapshots table|$ADMIN_BASE_URL/admin/core/snapshot/|/admin/core/snapshot/|archivebox/core/admin_snapshots.py"
             "Snapshots grid|$ADMIN_BASE_URL/admin/core/snapshot/grid/|/admin/core/snapshot/grid/|archivebox/templates/admin/snapshots_grid.html"
             "Snapshot admin detail|$ADMIN_BASE_URL/admin/core/snapshot/$SNAPSHOT_ID/change/|/admin/core/snapshot/$SNAPSHOT_ID/change/|archivebox/core/admin_snapshots.py"
@@ -625,11 +631,17 @@ PY
             SCREENSHOT_HEIGHT=1000 \
             node "$REPO_DIR/bin/take_screenshot.js" "$LIVE_SNAPSHOT_VIEW_URL" "$CAPTURE_ROOT/snapshot-output-discovery.png" >"$SNAPSHOT_DISCOVERY_REPORT"
         SNAPSHOT_OUTPUT_PLUGINS="$(UI_SCREENSHOT_DISCOVERY_REPORT="$SNAPSHOT_DISCOVERY_REPORT" uv run --no-cache --project "$REPO_DIR" python -c \
-            'import json, os; from urllib.parse import urlsplit; report=json.load(open(os.environ["UI_SCREENSHOT_DISCOVERY_REPORT"])); print("\n".join("{}\t{}".format(output["plugin"], "wait-replay:Nick Sweeting" if urlsplit(output["previewUrl"]).path.endswith(".wacz") else "") for output in report["checks"]["snapshotOutputs"]))')"
+            'import json, os; from urllib.parse import urlsplit; report=json.load(open(os.environ["UI_SCREENSHOT_DISCOVERY_REPORT"])); print("\n".join("{}\t{}".format(output["plugin"], "wait-frame-text:Verified · signature and archived response match" if output["plugin"] == "tlsnotary" else "wait-replay:Nick Sweeting" if urlsplit(output["previewUrl"]).path.endswith(".wacz") else "") for output in report["checks"]["snapshotOutputs"]))')"
         if [[ -z "$SNAPSHOT_OUTPUT_PLUGINS" ]]; then
             echo "[!] The Sweeting.me snapshot detail page exposed no selectable outputs" >&2
             exit 1
         fi
+        if ! printf '%s\n' "$SNAPSHOT_OUTPUT_PLUGINS" | cut -f1 | grep -qx tlsnotary; then
+            echo "[!] Live capture did not expose the required TLSNotary verifier output" >&2
+            tail -200 "$CAPTURE_ROOT/sweeting-live-capture.log" >&2
+            cat "$SNAPSHOT_DISCOVERY_REPORT" >&2
+            exit 1
+        fi
         while IFS=$'\t' read -r plugin_name output_capture_mode; do
             [[ -z "$plugin_name" ]] && continue
             VIEWS+=("Snapshot View ($plugin_name)|$LIVE_SNAPSHOT_VIEW_URL#$plugin_name|/|archivebox/templates/core/snapshot.html|$output_capture_mode")

bin/docker_entrypoint.sh

@@ -4,8 +4,9 @@
 # It takes a CMD as $* shell arguments and runs it following these setup steps:
 
 # - Set the archivebox user to match the existing /data owner when possible
-#     1. use the first non-root owner detected from existing collection files
-#     2. fall back to the image's default archivebox uid/gid when /data is root-owned
+#     1. use explicit PUID/PGID overrides when provided
+#     2. otherwise use the first non-root owner detected from collection files
+#     3. fall back to the image's default archivebox uid/gid when /data is root-owned
 # - Create a new /data dir if necessary and set the correct ownership on it
 # - Create a new /browsers dir if necessary and set the correct ownership on it
 # - Check whether we're running inside QEMU emulation and show a warning if so.
@@ -34,7 +35,8 @@ export XDG_CACHE_HOME="${XDG_CACHE_HOME:-$ABXPKG_LIB_DIR/cache}"
 export ABXBUS_CACHE_DIR="${ABXBUS_CACHE_DIR:-$XDG_CACHE_HOME/abxbus}"
 export UV_CACHE_DIR="${UV_CACHE_DIR:-$XDG_CACHE_HOME/uv}"
 
-# Global default uid/gid used when /data is empty or root-owned.
+# Global default uid/gid used when /data is empty or root-owned. PUID/PGID may
+# override the detected values for mounts with fixed server-side identities.
 export DEFAULT_ARCHIVEBOX_UID="${DEFAULT_ARCHIVEBOX_UID:-911}"
 export DEFAULT_ARCHIVEBOX_GID="${DEFAULT_ARCHIVEBOX_GID:-911}"
 
@@ -44,7 +46,7 @@ detect_data_owner() {
         if [[ -e "$path" ]]; then
             uid="$(stat -c '%u' "$path" 2>/dev/null || echo "$DEFAULT_ARCHIVEBOX_UID")"
             gid="$(stat -c '%g' "$path" 2>/dev/null || echo "$DEFAULT_ARCHIVEBOX_GID")"
-            if [[ "$uid" != "0" && "$gid" != "0" ]]; then
+            if [[ "$uid" != "0" ]]; then
                 echo "$uid:$gid"
                 return
             fi
@@ -54,8 +56,23 @@ detect_data_owner() {
 }
 
 export DETECTED_OWNER="$(detect_data_owner)"
-export TARGET_UID="${DETECTED_OWNER%%:*}"
-export TARGET_GID="${DETECTED_OWNER##*:}"
+export DETECTED_UID="${DETECTED_OWNER%%:*}"
+export DETECTED_GID="${DETECTED_OWNER##*:}"
+export TARGET_UID="${PUID:-$DETECTED_UID}"
+export TARGET_GID="${PGID:-$DETECTED_GID}"
+
+if [[ ! "$TARGET_UID" =~ ^[0-9]+$ || ! "$TARGET_GID" =~ ^[0-9]+$ ]]; then
+    echo -e "\n[X] Error: PUID and PGID must be numeric when set (got PUID=${PUID:-unset} PGID=${PGID:-unset})." > /dev/stderr
+    exit 3
+fi
+
+# The entrypoint needs root for mount preparation, but ArchiveBox and Chrome
+# must always run as a non-root user. Ignore PUID=0 injected by rootless
+# runtimes/providers and retain the detected/default non-root UID instead.
+# GID 0 remains valid for group-writable mounts.
+if [[ "$TARGET_UID" == "0" ]]; then
+    export TARGET_UID="$DETECTED_UID"
+fi
 
 if [[ "$(id -u)" == "0" ]]; then
     # Root is only used for startup permission repair. ArchiveBox/Chrome run as
@@ -77,6 +94,14 @@ else
     export TARGET_GID="$(id -g)"
 fi
 
+run_as_archivebox() {
+    if [[ "$(id -u)" == "0" ]]; then
+        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$@"
+    else
+        "$@"
+    fi
+}
+
 # Check if user attempted to run it in the root of their home folder or hard drive (common mistake)
 if [[ -d "$DATA_DIR/Documents" || -d "$DATA_DIR/.config" || -d "$DATA_DIR/usr" || -f "$DATA_DIR/.bashrc" || -f "$DATA_DIR/.zshrc" ]]; then
     echo -e "\n[X] ERROR: ArchiveBox was run from inside a home folder"
@@ -98,59 +123,87 @@ chmod_if_possible() {
     chmod u+rwX,g+rwX "$path" 2>/dev/null || true
 }
 
+path_is_writable() {
+    local path="$1"
+    [[ -d "$path" ]] \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        && run_as_archivebox test -x "$path" 2>/dev/null
+}
+
 ensure_dir() {
     local path="$1"
+
+    # Functional access is authoritative for NFS/CIFS/FUSE and Docker Desktop:
+    # they may report synthetic ownership or reject chown while writes work.
+    path_is_writable "$path" && return 0
+
+    # Create missing paths as the eventual application user first. This is the
+    # only reliable path on root-squashed mounts where container root is mapped
+    # to an unprivileged identity but the configured PUID can write.
+    if [[ ! -e "$path" ]] && run_as_archivebox mkdir -p "$path" 2>/dev/null; then
+        path_is_writable "$path" && return 0
+    fi
+
+    # Fall back to bounded, shallow root repair. Never walk the collection or
+    # archive tree during startup.
     mkdir -p "$path" 2>/dev/null || true
     chown_if_needed "$path"
     chmod_if_possible "$path"
+    path_is_writable "$path" || permission_error "$path"
 }
 
 ensure_file_owner() {
     local path="$1"
     [[ -e "$path" ]] || return 0
+    run_as_archivebox test -r "$path" 2>/dev/null \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        && return 0
     chown_if_needed "$path"
     chmod_if_possible "$path"
+    run_as_archivebox test -r "$path" 2>/dev/null \
+        && run_as_archivebox test -w "$path" 2>/dev/null \
+        || permission_error "$path"
 }
 
 ensure_runtime_tmp_tree() {
-    mkdir -p "$TMP_DIR" 2>/dev/null || true
-    [[ -e "$TMP_DIR" ]] || return 0
+    if [[ ! -e "$TMP_DIR" ]]; then
+        ensure_dir "$TMP_DIR"
+        return
+    fi
+    path_is_writable "$TMP_DIR" && return 0
     if [[ "$(id -u)" == "0" ]]; then
         chown -R "$TARGET_UID:$TARGET_GID" "$TMP_DIR" 2>/dev/null || true
     fi
     chmod_if_possible "$TMP_DIR"
+    path_is_writable "$TMP_DIR" || permission_error "$TMP_DIR"
 }
 
 ensure_small_runtime_tree() {
     local path="$1"
-    mkdir -p "$path" 2>/dev/null || true
-    [[ -e "$path" ]] || return 0
+    if [[ ! -e "$path" ]]; then
+        ensure_dir "$path"
+        return
+    fi
+    path_is_writable "$path" && return 0
     if [[ "$(id -u)" == "0" ]]; then
         chown -R "$TARGET_UID:$TARGET_GID" "$path" 2>/dev/null || true
     fi
     chmod -R u+rwX,g+rwX "$path" 2>/dev/null || true
-}
-
-run_as_archivebox() {
-    if [[ "$(id -u)" == "0" ]]; then
-        setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups "$@"
-    else
-        "$@"
-    fi
+    path_is_writable "$path" || permission_error "$path"
 }
 
 permission_error() {
     local path="$1"
     echo -e "\n[X] Error: archivebox user (uid=$TARGET_UID gid=$TARGET_GID) cannot write to $path." > /dev/stderr
     echo -e "    Current owner is $(stat -c '%u:%g' "$path" 2>/dev/null || echo 'unknown')." > /dev/stderr
-    echo -e "    Fix ownership of the matching host path for the intended archivebox user, e.g.:" > /dev/stderr
-    echo -e "       sudo chown $TARGET_UID:$TARGET_GID ./data" > /dev/stderr
-    echo -e "    Repair any nested path named above explicitly; never recursively chown a large archive." > /dev/stderr
+    echo -e "    The entrypoint tried non-root creation first, then bounded root ownership/mode repair." > /dev/stderr
+    echo -e "    The mount is read-only or its server-side identity/ACL does not allow uid=$TARGET_UID gid=$TARGET_GID." > /dev/stderr
     exit 3
 }
 
-# Create and repair only the small set of top-level writable paths. Do not recurse
-# through /data/archive; large collections can take days to recursively chown/chmod.
+# Create and repair only the small set of required paths. Functional checks run
+# as the final non-root identity first, so writable NFS/CIFS/FUSE mounts never
+# receive repeated chown attempts. Never recursively walk /data/archive here.
 ensure_dir "$DATA_DIR"
 ensure_dir "$CONFIG_DIR"
 ensure_dir "$DATA_DIR/logs"
@@ -166,34 +219,43 @@ ensure_file_owner "$DATA_DIR/ArchiveBox.conf"
 ensure_file_owner "$CONFIG_DIR/config.env"
 ensure_file_owner "$CONFIG_DIR/derived.env"
 
-run_as_archivebox touch "$DATA_DIR/logs/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$DATA_DIR/logs"
-rm -f "$DATA_DIR/logs/.permissions_test_safe_to_delete"
-run_as_archivebox touch "$DATA_DIR/archive/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$DATA_DIR/archive"
-rm -f "$DATA_DIR/archive/.permissions_test_safe_to_delete"
-run_as_archivebox touch "$PERSONAS_DIR/Default/chrome_profile/.permissions_test_safe_to_delete" 2>/dev/null || permission_error "$PERSONAS_DIR/Default/chrome_profile"
-rm -f "$PERSONAS_DIR/Default/chrome_profile/.permissions_test_safe_to_delete"
+assert_writable_dir() {
+    local path="$1"
+    local probe
+    probe="$(run_as_archivebox mktemp "$path/.permissions_test.XXXXXX" 2>/dev/null)" || permission_error "$path"
+    run_as_archivebox rm -f "$probe" 2>/dev/null || permission_error "$path"
+}
 
-# check if novnc x11 $DISPLAY is available
-export DISPLAY="${DISPLAY:-"novnc:0.0"}"
-if ! xdpyinfo > /dev/null 2>&1; then
-    # cant connect to x11 display, unset it so that chrome doesn't try to connect to it and hang indefinitely
-    unset DISPLAY
+assert_writable_dir "$DATA_DIR/logs"
+assert_writable_dir "$DATA_DIR/archive"
+assert_writable_dir "$PERSONAS_DIR/Default/chrome_profile"
+
+# Preserve an explicit DISPLAY (including an empty value to disable X11).
+# Only probe the implicit noVNC default: a missing probe or unavailable server
+# must not silently discard the user's display configuration.
+if [[ -z "${DISPLAY+x}" ]]; then
+    export DISPLAY="novnc:0.0"
+    if ! xdpyinfo > /dev/null 2>&1; then
+        unset DISPLAY
+    fi
 fi
 
 # Active browser processes do not survive container restarts, but their lock
-# files can. Clear stale browser state before dropping privileges.
-find "$PERSONAS_DIR" -type f \( \
-    -name "SingletonLock" \
-    -o -name "SingletonSocket" \
-    -o -name "SingletonCookie" \
-    -o -name "DevToolsActivePort" \
-    -o -name ".launch.lock" \
-    -o -name ".target.lock" \
-\) -delete >/dev/null 2>&1 || true
+# files can. Chromium keeps these at the user-data root; only inspect the
+# known root of each persona, never walk its potentially large profile tree.
+for persona_profile in "$PERSONAS_DIR"/*/chrome_profile; do
+    [[ -d "$persona_profile" ]] || continue
+    for lock_name in SingletonLock SingletonSocket SingletonCookie DevToolsActivePort .launch.lock .target.lock; do
+        rm -f "$persona_profile/$lock_name" 2>/dev/null || true
+    done
+done
 find /tmp "$TMP_DIR" -maxdepth 1 -type d -name "archivebox-chrome-profile.*" -mmin +30 -exec rm -rf {} + >/dev/null 2>&1 || true
     
 
 ensure_dir "/home/$ARCHIVEBOX_USER"
+# A nested read-only browser-profile mount can create this parent as root.
+# OpenCode's installer needs to create a sibling config directory as archivebox.
+ensure_dir "${XDG_CONFIG_HOME:-/home/$ARCHIVEBOX_USER/.config}"
 ensure_small_runtime_tree "$ABXBUS_CACHE_DIR"
 ensure_small_runtime_tree "$ABXBUS_CACHE_DIR/semaphores"
 ensure_small_runtime_tree "$UV_CACHE_DIR"

bin/generate_ui_screenshot_gallery.py

@@ -4,6 +4,7 @@ import hashlib
 import html
 import json
 import os
+import re
 import subprocess
 import struct
 import sys
@@ -123,6 +124,53 @@ def append_manifest(manifest_path: Path, screenshot_path: Path) -> None:
         manifest.write(json.dumps(item) + "\n")
 
 
+def render_gallery(gallery_header: str, gallery_sections: str) -> str:
+    return (
+        '<!doctype html><html lang="en"><head><meta charset="utf-8">'
+        '<meta name="viewport" content="width=device-width, initial-scale=1">'
+        "<title>ArchiveBox UI Screenshots</title><style>"
+        ":root{--ink:#28252a;--muted:#716c74;--accent:#9b2854;--paper:#fcfaf7;--line:#e9e2e3}"
+        "*{box-sizing:border-box}body{margin:0;font:16px/1.75 system-ui,sans-serif;"
+        "background:var(--paper);color:var(--ink)}.screenshot-content{max-width:1100px;margin:0 auto;padding:28px}"
+        "a{color:var(--accent)}h1,h2{line-height:1.15;letter-spacing:-.045em}"
+        "article{margin:40px 0 65px}article h2{font-size:29px;margin:0 0 10px}article p{font-size:13px;color:var(--muted);margin:0 0 20px}"
+        ".profile-switch{display:flex;flex-wrap:wrap;gap:4px;background:#f0e8eb;border:1px solid var(--line);padding:5px;"
+        "border-radius:28px;width:max-content;max-width:100%;margin:30px 0}.profile-switch button{font:600 13px system-ui;"
+        "cursor:pointer;border:0;border-radius:22px;background:transparent;color:var(--muted);padding:10px 20px}"
+        ".profile-switch button[aria-pressed=true]{background:var(--accent);color:white;box-shadow:0 3px 10px #9b285425}"
+        "a:focus-visible,button:focus-visible{outline:3px solid var(--accent);outline-offset:4px}"
+        "figure{min-width:0;margin:0 0 20px;padding:18px;border:1px solid var(--line);border-radius:20px;"
+        "background:linear-gradient(140deg,#fff,#f8edf2)}figure[hidden]{display:none}"
+        ".shot-mobile{max-width:440px;margin-inline:auto}.shot-tablet{max-width:800px;margin-inline:auto}"
+        "figcaption{font-size:12px;color:var(--muted);text-align:center;margin:0 0 12px}"
+        "img{display:block;width:100%;height:auto;border-radius:8px}code{overflow-wrap:anywhere}"
+        "@media(max-width:550px){.screenshot-content{padding:20px}figure{padding:8px;border-radius:12px}.profile-switch button{padding:9px 16px}}"
+        '</style><link rel="stylesheet" href="../site-chrome.css"></head><body>'
+        + (REPO_DIR / "bin/templates/screenshots-header.html").read_text(encoding="utf-8")
+        + '<div class="screenshot-content">'
+        + gallery_header
+        + '<div class="profile-switch" role="group" aria-label="Screenshot viewport">'
+        + "".join(
+            f'<button type="button" data-viewport="{profile}" aria-pressed="{str(profile == "desktop").lower()}">{profile.title()}</button>'
+            for profile in CAPTURE_PROFILES
+        )
+        + "</div><main>"
+        + gallery_sections
+        + """</main><script>
+const figures = document.querySelectorAll('.shot');
+const buttons = document.querySelectorAll('[data-viewport]');
+function selectProfile(profile) {
+  for (const figure of figures) figure.hidden = figure.dataset.profile !== profile;
+  for (const button of buttons) button.setAttribute('aria-pressed', String(button.dataset.viewport === profile));
+}
+for (const button of buttons) button.addEventListener('click', () => selectProfile(button.dataset.viewport));
+selectProfile('desktop');
+</script></div>"""
+        + (REPO_DIR / "bin/templates/screenshots-footer.html").read_text(encoding="utf-8")
+        + "</body></html>\n"
+    )
+
+
 def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -> None:
     provenance = build_provenance()
     source_base_url = f"https://github.com/ArchiveBox/ArchiveBox/blob/{provenance['revision']}/"
@@ -205,7 +253,7 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
                 f'alt="{html.escape(str(capture["name"]))} — {profile}" width="{width}"></td>',
             )
             html_figures.append(
-                f'<figure class="shot shot-{profile}"><figcaption>{label}</figcaption>'
+                f'<figure class="shot shot-{profile}" data-profile="{profile}"><figcaption>{label}</figcaption>'
                 f'<a href="./{versioned_filename}">'
                 f'<img src="./{versioned_filename}" width="{width}" height="{height}" loading="lazy" '
                 f'alt="{html.escape(str(capture["name"]))} — {profile}"></a></figure>',
@@ -252,25 +300,15 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
         encoding="utf-8",
     )
 
-    html_path.write_text(
-        '<!doctype html><html lang="en"><head><meta charset="utf-8">'
-        '<meta name="viewport" content="width=device-width, initial-scale=1">'
-        "<title>ArchiveBox UI Screenshots</title><style>"
-        "body{max-width:1900px;margin:0 auto;padding:20px;font:16px system-ui,sans-serif;background:#f6f7f9;color:#18202a}"
-        "a{color:#2563eb}article{margin:32px 0 64px}.shots{display:grid;grid-template-columns:2fr 1.35fr .8fr;gap:14px;align-items:start}"
-        "figure{min-width:0;margin:0}figcaption{font-weight:650;margin:0 0 8px}img{display:block;width:100%;height:auto;"
-        "border:1px solid #cbd5e1;border-radius:8px;background:white;box-shadow:0 8px 24px #0f172a18}code{overflow-wrap:anywhere}"
-        "@media(max-width:900px){.shots{grid-template-columns:1fr}body{padding:12px}}"
-        '</style></head><body><header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
-        f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
+    gallery_header = (
+        '<header><p><a href="../">← ArchiveBox</a></p><h1>ArchiveBox UI Screenshots</h1>'
+        + f"<p>Generated from ArchiveBox <code>{html.escape(provenance['version'])}</code> at "
         f'<a href="https://github.com/ArchiveBox/ArchiveBox/commit/{html.escape(provenance["revision"])}">'
         f"<code>{html.escape(provenance['revision'][:12])}</code></a> on "
         f'<time datetime="{html.escape(provenance["generated_at"])}">{html.escape(provenance["generated_at"])}</time>. '
-        "Desktop, tablet, and mobile viewports are captured from the same build.</p></header><main>"
-        + "".join(html_sections)
-        + "</main></body></html>\n",
-        encoding="utf-8",
+        "Desktop, tablet, and mobile viewports are captured from the same build.</p></header>"
     )
+    html_path.write_text(render_gallery(gallery_header, "".join(html_sections)), encoding="utf-8")
 
     file_hashes = {
         capture["filename"]: hashlib.sha256((html_path.parent / capture["filename"]).read_bytes()).hexdigest() for capture in captures
@@ -297,7 +335,27 @@ def build_galleries(manifest_path: Path, markdown_path: Path, html_path: Path) -
                     screenshot_path.unlink()
 
 
+def refresh_gallery(html_path: Path) -> None:
+    """Apply current presentation to published capture content without changing provenance."""
+    document = html_path.read_text(encoding="utf-8")
+    header = re.search(r"<header>.*?</header>", document, re.DOTALL)
+    sections = re.search(r"<main>(.*?)</main>", document, re.DOTALL)
+    if not header or not sections:
+        raise SystemExit("Published screenshot gallery is missing its capture header or content")
+    content = sections[1]
+    for profile in CAPTURE_PROFILES:
+        # Galleries published before the switcher used classes alone.
+        content = content.replace(
+            f'<figure class="shot shot-{profile}">',
+            f'<figure class="shot shot-{profile}" data-profile="{profile}">',
+        )
+    html_path.write_text(render_gallery(header[0], content), encoding="utf-8")
+
+
 def main() -> None:
+    if len(sys.argv) == 3 and sys.argv[1] == "refresh":
+        refresh_gallery(Path(sys.argv[2]))
+        return
     if len(sys.argv) == 4 and sys.argv[1] == "validate":
         validate_navigation(Path(sys.argv[2]), sys.argv[3])
         return

bin/restore_site_screenshots.py

@@ -0,0 +1,83 @@
+#!/usr/bin/env python3
+"""Restore validated capture output without running application CI."""
+
+import argparse
+from concurrent.futures import ThreadPoolExecutor
+import hashlib
+import json
+import os
+from pathlib import Path, PurePosixPath
+import subprocess
+from urllib.error import HTTPError
+from urllib.request import urlopen
+
+parser = argparse.ArgumentParser(description=__doc__)
+parser.add_argument("kind", choices=["archivebox", "ios", "extension"])
+parser.add_argument("destination", type=Path)
+args = parser.parse_args()
+args.destination.mkdir(parents=True, exist_ok=True)
+# Only successful default-branch capture runs are eligible, never PR artifacts.
+if os.environ.get("GH_TOKEN"):
+    repo = os.environ["GITHUB_REPOSITORY"]
+    branch = "dev" if args.kind == "archivebox" else "main"
+    runs = json.loads(
+        subprocess.check_output(
+            ["gh", "api", f"repos/{repo}/actions/workflows/screenshots.yml/runs?branch={branch}&status=success&per_page=100"],
+        ),
+    )
+    for run in runs["workflow_runs"]:
+        if run["event"] not in ("push", "workflow_dispatch"):
+            continue
+        artifacts = json.loads(subprocess.check_output(["gh", "api", f"repos/{repo}/actions/runs/{run['id']}/artifacts"]))
+        if any(a["name"] == "site-screenshots" and not a["expired"] for a in artifacts["artifacts"]):
+            subprocess.run(
+                ["gh", "run", "download", str(run["id"]), "--repo", repo, "--name", "site-screenshots", "--dir", str(args.destination)],
+                check=True,
+            )
+            print(f"Restored validated captures from run {run['id']} ({run['head_sha']})")
+            raise SystemExit(0)
+# Published captures remain available even after Actions artifacts expire.
+base, manifest_name = {
+    "archivebox": ("https://archivebox.io/screenshots/", "build.json"),
+    "ios": ("https://archivebox.github.io/ios-archivebox/screenshots/", "manifest.json"),
+    "extension": ("https://archivebox.github.io/archivebox-browser-extension/screenshots/", "manifest.json"),
+}[args.kind]
+
+
+def fetch(name):
+    path = PurePosixPath(name)
+    if path.is_absolute() or ".." in path.parts or ":" in name or "\\" in name:
+        raise ValueError(f"Unsafe capture path: {name}")
+    with urlopen(base + name, timeout=60) as response:
+        data = response.read()
+    expected = hashes.get(name)
+    if expected and hashlib.sha256(data).hexdigest() != expected:
+        raise ValueError(f"Capture checksum mismatch: {name}")
+    target = args.destination / name
+    target.parent.mkdir(parents=True, exist_ok=True)
+    target.write_bytes(data)
+
+
+try:
+    with urlopen(base + manifest_name, timeout=60) as response:
+        raw = response.read()
+except HTTPError as error:
+    if error.code == 404 and args.kind == "ios":
+        print("No complete native gallery published yet; use the curated README screenshots.")
+        raise SystemExit(0)
+    raise
+manifest = json.loads(raw)
+hashes = {}
+if args.kind == "archivebox":
+    hashes = manifest["files"]
+    names = [*hashes, "index.html"]
+elif args.kind == "extension":
+    names = [image["file"] for capture in manifest["screenshots"] for image in capture["images"]]
+else:
+    if manifest.get("complete") is not True:
+        raise ValueError("Published native gallery must be complete")
+    names = [capture["path"] for capture in manifest["captures"]]
+with ThreadPoolExecutor(max_workers=8) as pool:
+    list(pool.map(fetch, names))
+(args.destination / manifest_name).write_bytes(raw)
+print(f"Restored {len(names)} published capture files, retaining revision {manifest['revision']}")

bin/take_screenshot.js

@@ -32,6 +32,7 @@ Environment:
   SCREENSHOT_SNAPSHOT_VIEW   Set to list or grid before loading the page
   SCREENSHOT_SNAPSHOT_HEADER Set to expanded or collapsed before loading a snapshot detail page
   SCREENSHOT_EXPECT_PLUGIN   Require this snapshot output plugin to be selected
+  SCREENSHOT_EXPECT_FRAME_TEXT  Require this text in the selected output frame
   SCREENSHOT_EXPECT_LIVE_PROGRESS  Require real progress bars and a loaded screencast frame
   SCREENSHOT_COLLAPSE_FILTERS Set to 1 to keep admin filters out of screenshots
   SCREENSHOT_RESET_FILTERS   Set to 1 to clear the admin filter collapsed preference
@@ -231,6 +232,12 @@ async function main() {
           }
         });
       }, { timeout: 45000, polling: 250 }).catch(() => {});
+      if (process.env.SCREENSHOT_EXPECT_FRAME_TEXT) {
+        const frameElement = await page.$('#main-frame');
+        const frame = await frameElement.contentFrame();
+        await frame.waitForFunction((text) => document.body.innerText.includes(text),
+          { timeout: 45000 }, process.env.SCREENSHOT_EXPECT_FRAME_TEXT);
+      }
     }
 
     if (process.env.SCREENSHOT_EXPECT_LIVE_PROGRESS === '1') {

bin/templates/screenshots-footer.html

@@ -0,0 +1,26 @@
+<footer class="abx-footer" aria-label="ArchiveBox ecosystem">
+  <div class="abx-footer-intro"><div><a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span></a><p class="abx-tagline">Open-source tools for keeping a lasting copy of the web.<br>Your data. Your devices. Your archive.</p></div><p class="abx-footer-motto">A home for the web<br>you want to keep.</p></div>
+  <div class="abx-footer-grid"><section class="abx-footer-column"><h2>Apps &amp; tools</h2><ul><li><a href="https://archivebox.io/">ArchiveBox Server</a><a class="abx-source" href="https://github.com/ArchiveBox/ArchiveBox" aria-label="ArchiveBox Server source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#get-started">ArchiveBox for macOS &amp; iOS</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox for macOS &amp; iOS source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a class="abx-source" href="https://github.com/ArchiveBox/archivebox-browser-extension" aria-label="ArchiveBox Browser Extension source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-plugins" aria-label="ArchiveBox Plugin Library source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/abx-dl/">One-shot abx-dl CLI</a><a class="abx-source" href="https://github.com/ArchiveBox/abx-dl" aria-label="One-shot abx-dl CLI source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 12 .75Z"/></svg></a></li><li><a href="https://archivebox.github.io/ios-archivebox/#your-server-your-choice">ArchiveBox Server.app</a><a class="abx-source" href="https://github.com/ArchiveBox/ios-archivebox" aria-label="ArchiveBox Server.app source on GitHub" title="View source on GitHub"><svg viewBox="0 0 24 24" aria-hidden="true"><path d="M12 .75a11.25 11.25 0 0 0-3.56 21.92c.56.1.77-.24.77-.54v-2.1c-3.14.68-3.8-1.33-3.8-1.33-.51-1.3-1.25-1.65-1.25-1.65-1.02-.7.08-.69.08-.69 1.13.08 1.73 1.16 1.73 1.16 1 1.72 2.63 1.22 3.27.93.1-.73.39-1.22.71-1.5-2.5-.28-5.13-1.25-5.13-5.56 0-1.23.44-2.23 1.16-3.02-.12-.28-.5-1.43.11-2.98 0 0 .95-.3 3.1 1.15a10.8 10.8 0 0 1 5.64 0c2.15-1.46 3.1-1.15 3.1-1.15.62 1.55.23 2.7.12 2.98.72.79 1.16 1.8 1.16 3.02 0 4.32-2.64 5.27-5.15 5.55.4.35.76 1.03.76 2.08v3.11c0 .3.2.65.77.54A11.25 11.25 0 0 0 1Line truncated
+  
+  <div class="abx-footer-bottom"><p>Built in the open. Supported by our community.</p><p><a href="https://github.com/ArchiveBox">GitHub organization</a> · <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Security &amp; privacy</a> · <a href="https://archivebox.io">ArchiveBox.io ↗</a></p></div>
+</footer>
+<script>
+(() => {
+  const menu = document.querySelector('.abx-apps');
+  document.addEventListener('keydown', event => {
+    if (event.key === 'Escape' && menu?.open) { menu.open = false; menu.querySelector('summary').focus(); }
+  });
+  document.addEventListener('click', event => { if (menu?.open && !menu.contains(event.target)) menu.open = false; });
+  function revealAnchor() {
+    let target;
+    try { target = document.getElementById(decodeURIComponent(location.hash.slice(1))); } catch { return; }
+    if (!target) return;
+    let details = target.closest('details');
+    let opened = false;
+    while (details) { if (!details.open) { details.open = true; opened = true; } details = details.parentElement.closest('details'); }
+    if (opened) requestAnimationFrame(() => target.scrollIntoView());
+  }
+  addEventListener('hashchange', revealAnchor);
+  revealAnchor();
+})();
+</script>

bin/templates/screenshots-header.html

@@ -0,0 +1,9 @@
+<header class="abx-header">
+  <a class="abx-brand" href="https://archivebox.io/"><svg class="abx-logo" viewBox="0 0 40 40" aria-hidden="true"><rect width="40" height="40" rx="10" fill="#9b2854"/><path d="M10 14h20v17H10z" fill="#fff4ed"/><path d="M8 9h24v6H8z" fill="#fff4ed"/><path d="M16 19h8v3h-8z" fill="#9b2854"/><path d="M12 31V17" stroke="#9b2854" stroke-width="2"/></svg><span>ArchiveBox</span><span class="abx-subsite">UI Screenshots</span></a>
+  <nav class="abx-nav" aria-label="Main navigation">
+    <details class="abx-apps"><summary>Apps</summary><div class="abx-app-links"><a href="https://archivebox.io/">ArchiveBox Server</a><a href="https://archivebox.github.io/ios-archivebox/">ArchiveBox for macOS &amp; iOS</a><a href="https://archivebox.github.io/archivebox-browser-extension/">ArchiveBox Browser Extension</a><a href="https://archivebox.github.io/abx-plugins/">ArchiveBox Plugin Library</a><a href="https://archivebox.github.io/abx-dl/"><span>One-shot <code>abx-dl</code> CLI</span></a><a href="https://github.com/ArchiveBox">More on Github...</a></div></details>
+    <a href="https://archivebox.io/screenshots/">Screenshots</a><a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Docs</a>
+    <a href="https://github.com/ArchiveBox/ArchiveBox">GitHub ↗</a>
+    <a class="abx-cta" href="https://archivebox.io/#install">Get started</a>
+  </nav>
+</header>

bin/validate_docker_display.sh

@@ -0,0 +1,69 @@
+#!/usr/bin/env bash
+# Real Compose regression checks. Build the image under test first, then run:
+# IMAGE=archivebox/archivebox:issue-1872 ./bin/validate_docker_display.sh
+set -Eeuo pipefail
+
+export IMAGE="${IMAGE:-archivebox/archivebox:dev}"
+validation_dir="$(mktemp -d "${TMPDIR:-/tmp}/archivebox-display.XXXXXX")"
+compose=(docker compose --project-name "archivebox-display-$$" --file "$validation_dir/compose.yml")
+cleanup() {
+    "${compose[@]}" down --volumes --remove-orphans
+    echo "Validation files: $validation_dir"
+}
+trap cleanup EXIT
+
+cat > "$validation_dir/compose.yml" <<'YAML'
+services:
+  archivebox:
+    image: ${IMAGE}
+    volumes:
+      - ./data:/data
+      - ./data/personas:/data/personas
+      - tmp:/tmp/archivebox
+    shm_size: 1gb
+  novnc:
+    image: theasp/novnc:latest
+    environment:
+      DISPLAY_WIDTH: 1280
+      DISPLAY_HEIGHT: 800
+      RUN_XTERM: "no"
+    healthcheck:
+      test: ["CMD", "xdpyinfo", "-display", ":0"]
+      interval: 2s
+      timeout: 2s
+      retries: 10
+volumes:
+  tmp:
+YAML
+
+"${compose[@]}" up -d --wait novnc
+
+# Check the real runtime dependency as well as successful X11 connectivity.
+"${compose[@]}" run --rm archivebox sh -ec '
+    command -v xdpyinfo
+    test "$DISPLAY" = novnc:0.0
+    xdpyinfo >/dev/null
+'
+"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
+    test "$DISPLAY" = novnc:0.0
+    xdpyinfo >/dev/null
+'
+
+# Explicit values belong to the caller, even when X11 is unavailable.
+"${compose[@]}" run --rm -e DISPLAY=unavailable:0 archivebox sh -ec '
+    test "$DISPLAY" = unavailable:0
+'
+# An explicit empty value disables automatic display discovery.
+"${compose[@]}" run --rm -e DISPLAY= archivebox sh -ec '
+    test "${DISPLAY+x}" = x
+    test -z "$DISPLAY"
+'
+
+"${compose[@]}" stop novnc
+"${compose[@]}" run --rm archivebox sh -ec '
+    test "${DISPLAY+x}" != x
+'
+"${compose[@]}" run --rm -e DISPLAY=novnc:0.0 archivebox sh -ec '
+    test "$DISPLAY" = novnc:0.0
+'
+echo 'PASS: Compose DISPLAY discovery, explicit override, empty opt-out, and headless fallback'

bin/validate_docker_uid_gid.sh

@@ -246,6 +246,10 @@ run_case() {
         local nested_stat
         nested_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data/archive/existing/file" 2>/dev/null || true)"
         [[ "$nested_stat" == "0:0" ]] || ok=0
+    elif [[ "$post_assert" == "data-owner-stays" ]]; then
+        local data_stat
+        data_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data" 2>/dev/null || true)"
+        [[ "$data_stat" == "1000:1000" ]] || ok=0
     elif [[ "$post_assert" == runtime-nested-root-stays ]]; then
         local lib_stat browsers_stat chrome_rules_stat chrome_metadata_stat
         lib_stat="$("${docker_base[@]}" -v "$case_dir/lib:/libdir" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /libdir/existing/file" 2>/dev/null || true)"
@@ -256,6 +260,11 @@ run_case() {
             && "$browsers_stat" == "0:0" \
             && "$chrome_rules_stat" == "$expected_uid:$expected_gid" \
             && "$chrome_metadata_stat" == "$expected_uid:$expected_gid" ]] || ok=0
+    elif [[ "$post_assert" == cache-and-profile-shallow ]]; then
+        local cache_stat profile_state
+        cache_stat="$("${docker_base[@]}" -v "$case_dir/lib:/libdir" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /libdir/cache/uv/nested/marker" 2>/dev/null || true)"
+        profile_state="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "test ! -e /data/personas/Default/chrome_profile/SingletonLock && test -e /data/personas/Default/chrome_profile/nested/SingletonLock && echo shallow" 2>/dev/null || true)"
+        [[ "$cache_stat" == "0:0" && "$profile_state" == shallow ]] || ok=0
     elif [[ "$post_assert" == users-dir-repaired ]]; then
         local users_stat
         users_stat="$("${docker_base[@]}" -v "$case_dir/data:/data" --entrypoint /bin/bash "$IMAGE" -lc "stat -c '%u:%g' /data/users" 2>/dev/null || true)"
@@ -419,6 +428,62 @@ run_mount_case() {
     VALIDATION_ROOT="$previous_root"
 }
 
+run_nfs_root_squash_case() {
+    local mount_dir="$1"
+
+    [[ -n "$mount_dir" ]] || { log "FAIL NFS root_squash case: mount dir not provided"; exit 1; }
+    [[ -d "$mount_dir" ]] || { log "FAIL NFS root_squash case: $mount_dir does not exist"; exit 1; }
+    [[ -w "$mount_dir" ]] || { log "FAIL NFS root_squash case: $mount_dir is not writable by host user"; exit 1; }
+
+    total=$((total + 1))
+    local case_dir="$mount_dir/archivebox-root-squash-validation-$RUN_ID"
+    local log_file="$case_dir/output.log"
+    local owner uid gid status ok root_is_squashed=0
+    mkdir -p "$case_dir/data"
+    owner="$(stat -c '%u:%g' "$case_dir/data")"
+    uid="${owner%%:*}"
+    gid="${owner##*:}"
+
+    if ! "${docker_base[@]}" \
+        --user 0:0 \
+        -v "$case_dir/data:/data" \
+        --entrypoint /bin/touch \
+        "$IMAGE" \
+        /data/root-must-not-write 2>/dev/null; then
+        root_is_squashed=1
+    fi
+    rm -f "$case_dir/data/root-must-not-write"
+
+    set +e
+    "${docker_base[@]}" \
+        -e "PUID=$uid" \
+        -e "PGID=$gid" \
+        -v "$ENTRYPOINT_PATH:/app/bin/docker_entrypoint.sh:ro" \
+        -v "$case_dir/data:/data" \
+        --entrypoint /app/bin/docker_entrypoint.sh \
+        "$IMAGE" \
+        sh -c "$default_cmd" >"$log_file" 2>&1
+    status=$?
+    set -e
+
+    ok=1
+    [[ "$root_is_squashed" == "1" ]] || ok=0
+    [[ "$uid" != "0" ]] || ok=0
+    [[ "$status" == "0" ]] || ok=0
+    grep -q "^ABX_UID=$uid$" "$log_file" || ok=0
+    grep -q "^ABX_GID=$gid$" "$log_file" || ok=0
+    grep -q '^ABX_OK$' "$log_file" || ok=0
+
+    if [[ "$ok" == "1" ]]; then
+        passed=$((passed + 1))
+        log "PASS NFS root_squash mount uses the writable non-root identity without root repair"
+    else
+        failed=$((failed + 1))
+        log "FAIL NFS root_squash mount (status=$status owner=$owner log=$log_file)"
+        sed -n '1,180p' "$log_file"
+    fi
+}
+
 run_compose_personas_case() {
     total=$((total + 1))
     local case_dir="$VALIDATION_ROOT/compose-personas-persist"
@@ -451,6 +516,54 @@ run_compose_personas_case() {
     fi
 }
 
+run_compose_fresh_up_case() {
+    total=$((total + 1))
+    local case_dir="$VALIDATION_ROOT/compose-fresh-up"
+    local log_file="$case_dir/output.log"
+    local status=0
+    mkdir -p "$case_dir"
+    local ARCHIVEBOX_IMAGE="$IMAGE"
+    local ARCHIVEBOX_PORT=0
+    export ARCHIVEBOX_IMAGE ARCHIVEBOX_PORT
+
+    set +e
+    "$DOCKER_BINARY" compose \
+        --project-directory "$case_dir" \
+        -f "$COMPOSE_PATH" \
+        up -d --wait >"$log_file" 2>&1
+    status=$?
+    if [[ "$status" == "0" ]]; then
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" \
+            exec -T archivebox sh -c \
+            'server_identity="$(ps -eo uid=,gid=,args= | awk '\''/\/archivebox server --init/ && !/awk/ {print $1 ":" $2; exit}'\'')"; printf "ABX_SERVER_ID=%s\n" "$server_identity"; test "$server_identity" = 911:911; setpriv --reuid=911 --regid=911 --clear-groups sh -c '\''touch /data/logs/fresh-up-probe /data/archive/fresh-up-probe "$PERSONAS_DIR/Default/chrome_profile/fresh-up-probe" && rm -f /data/logs/fresh-up-probe /data/archive/fresh-up-probe "$PERSONAS_DIR/Default/chrome_profile/fresh-up-probe"'\''' \
+            >>"$log_file" 2>&1 || status=$?
+    fi
+    if [[ "$status" == "0" ]]; then
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" restart archivebox \
+            >>"$log_file" 2>&1 || status=$?
+        "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" up -d --wait \
+            >>"$log_file" 2>&1 || status=$?
+    fi
+    "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" logs --no-color archivebox \
+        >>"$log_file" 2>&1 || true
+    "$DOCKER_BINARY" compose --project-directory "$case_dir" -f "$COMPOSE_PATH" down --volumes --remove-orphans \
+        >>"$log_file" 2>&1 || status=$?
+    set -e
+
+    if [[ "$status" == "0" ]] \
+        && [[ -d "$case_dir/data/logs" ]] \
+        && [[ -d "$case_dir/data/personas/Default/chrome_profile" ]] \
+        && grep -q '^ABX_SERVER_ID=911:911$' "$log_file" \
+        && ! grep -q 'cannot write to /data' "$log_file"; then
+        passed=$((passed + 1))
+        log "PASS fresh compose up creates writable data and personas paths and survives restart"
+    else
+        failed=$((failed + 1))
+        log "FAIL fresh compose up permission initialization (status=$status log=$log_file)"
+        sed -n '1,220p' "$log_file"
+    fi
+}
+
 log "Running UID/GID validation on ${HOSTNAME:-unknown-host} using image=$IMAGE entrypoint=$ENTRYPOINT_PATH root=$VALIDATION_ROOT platform=${DOCKER_PLATFORM:-native}"
 
 run_case "root-owned empty data auto-detect default" \
@@ -477,6 +590,42 @@ run_case "nested root-owned archive content is not recursively chowned" \
     "chown 0:0 /case/data && chmod 755 /case/data && mkdir -p /case/data/archive/existing && touch /case/data/archive/existing/file && chown -R 0:0 /case/data/archive/existing" \
     "-" "-" pass 911 911 "$default_cmd" nested-root-stays
 
+run_case "explicit PUID and PGID override root-owned data" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=1234 PGID=2345" "-" pass 1234 2345
+
+run_case "PUID-only override retains detected group" \
+    "chown 501:20 /case/data && chmod 755 /case/data" \
+    "PUID=1234" "-" pass 1234 20
+
+run_case "PGID-only override retains detected user" \
+    "chown 501:20 /case/data && chmod 755 /case/data" \
+    "PGID=2345" "-" pass 501 2345
+
+run_case "PUID zero falls back to the non-root default" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=0 PGID=911" "-" pass 911 911
+
+run_case "nonnumeric PUID is rejected" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=archivebox PGID=911" "-" fail "" ""
+
+run_case "non-root UID with root group is supported" \
+    "chown 1201:0 /case/data && chmod 775 /case/data" \
+    "-" "-" pass 1201 0
+
+run_case "explicit non-root PUID with PGID zero is supported" \
+    "chown 0:0 /case/data && chmod 755 /case/data" \
+    "PUID=1201 PGID=0" "-" pass 1201 0
+
+run_case "writable cache and persona profile skip recursive startup scans" \
+    "chown 911:911 /case/data && mkdir -p /case/lib/cache/uv/nested /case/data/personas/Default/chrome_profile/nested && touch /case/lib/cache/uv/nested/marker /case/data/personas/Default/chrome_profile/SingletonLock /case/data/personas/Default/chrome_profile/nested/SingletonLock && chown 911:911 /case/lib/cache/uv /case/data/personas/Default/chrome_profile && chown 0:0 /case/lib/cache/uv/nested/marker" \
+    "-" "-" pass 911 911 "$default_cmd" cache-and-profile-shallow
+
+run_case "writable forced-owner mount skips metadata changes" \
+    "chown 1000:1000 /case/data && chmod 777 /case/data" \
+    "PUID=911 PGID=911" "-" pass 911 911 "$default_cmd" data-owner-stays
+
 run_case "only the writable Chrome extension cache is recursively repaired" \
     "chown 501:20 /case/data && mkdir -p /case/lib/existing /case/lib/chromewebstore/extensions/cookie /case/browsers/existing && touch /case/lib/existing/file /case/lib/chromewebstore/extensions/cookie/rules.json /case/browsers/existing/file && chown -R 0:0 /case/lib/existing /case/lib/chromewebstore /case/browsers/existing" \
     "-" "-" pass 501 20 "$runtime_dependency_cmd" runtime-nested-root-stays
@@ -506,8 +655,9 @@ run_case "root-owned ArchiveBox.conf only is repaired" \
     "-" "-" pass 911 911 "$default_cmd" config-files-repaired
 
 run_compose_personas_case
+run_compose_fresh_up_case
 
-run_mount_case "NFS" "${NFS_TEST_DIR:-}"
+run_nfs_root_squash_case "${NFS_TEST_DIR:-}"
 run_mount_case "SMB" "${SMB_TEST_DIR:-}"
 
 log "SUMMARY passed=$passed failed=$failed total=$total"

docker-compose.yml

@@ -15,6 +15,10 @@ services:
     archivebox:
         image: ${ARCHIVEBOX_IMAGE:-archivebox/archivebox:dev}
         restart: unless-stopped
+        # Uncomment with the novnc service below to open the browser automatically.
+        # depends_on:
+        #     novnc:
+        #         condition: service_healthy
         ports:
             - "${ARCHIVEBOX_PORT:-8000}:8000"
         volumes:
@@ -25,8 +29,11 @@ services:
             - tmp:/tmp/archivebox
         pids_limit: 2048                       # avoids low systemd task caps with Chrome and parallel extractors
         environment:
+            - PUID                                                                   # optional numeric UID for NFS/CIFS/FUSE mounts (defaults to collection owner or 911)
+            - PGID                                                                   # optional numeric GID for NFS/CIFS/FUSE mounts (defaults to collection group or 911)
             # - ADMIN_USERNAME=admin                             # creates an admin user on first run with the given user/pass combo
             # - ADMIN_PASSWORD=SomeSecretPassword
+            # - ARCHIVEBOX_VNC_PERSONA=personal                    # persona to open in VNC (defaults to DEFAULT_PERSONA)
             - BASE_URL                                                               # optional canonical URL; the first-run admin wizard fills it from the browser
             - SERVER_SECURITY_MODE                                                   # optional override; defaults to auto and can be set by the first-run wizard
             - PUBLIC_ADD_VIEW=False                              # set to True to allow anonymous users to submit new URLs to archive
@@ -64,12 +71,15 @@ services:
 
     # novnc:
     #     image: theasp/novnc:latest
-    #     profiles:
-    #         - novnc
     #     environment:
     #         - DISPLAY_WIDTH=1920
     #         - DISPLAY_HEIGHT=1080
     #         - RUN_XTERM=no
+    #     healthcheck:
+    #         test: ["CMD", "xdpyinfo", "-display", ":0"]
+    #         interval: 2s
+    #         timeout: 2s
+    #         retries: 10
     #     ports:
     #         # to view/control ArchiveBox's browser, visit: http://127.0.0.1:8080/vnc.html
     #         # restricted to access from localhost by default because it has no authentication

docs/Chromium-Install.md

@@ -1,151 +1,174 @@
 # Chrome / Chromium Setup
 
-ArchiveBox resolves Chrome through `abxpkg`, just like every other runtime binary. It checks compatible browsers already installed on the host first. When it finds one, it projects that exact browser into the managed runtime environment; otherwise it installs a compatible managed Chromium build.
+ArchiveBox uses a Chromium-based browser to load pages and save screenshots, PDFs, and HTML.
+
+## Install ArchiveBox's browser
+
+Docker images include the browser. For an installation without Docker, run these commands from your ArchiveBox data folder:
 
 ```bash
 archivebox install chrome
 archivebox version
 ```
 
-The resolved browser is always available through `./lib/env/bin/chromium` inside the collection. `archivebox version` shows whether it came from the host or a managed provider, along with the exact version and path.
+ArchiveBox uses a suitable installed browser or downloads one if needed. `archivebox version` shows the browser it selected.
 
-If you need to select a specific compatible browser already installed on the host, set `CHROME_BINARY` and let the same installer validate and project it:
+<details>
+<summary>Choose a specific installed browser</summary>
+
+Set `CHROME_BINARY` to the browser's command or executable path, then run the installer:
 
 ```bash
 archivebox config --set CHROME_BINARY=google-chrome
 archivebox install chrome
-archivebox version
 ```
 
-## Troubleshooting Chromium Install
-
-If you encounter problems setting up Google Chrome or Chromium, see the [Troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#chromiumgoogle-chrome) page.
-
----
+</details>
 
 # Setting Up a Chromium User Profile
 
-You may choose to set up a Chrome/Chromium user profile in order to use your cookies/sessions to log into sites behind authentication/paywall during archiving.
+ArchiveBox uses a browser to load pages and save their content. If a page requires you to log in, ArchiveBox needs to be logged in too. Otherwise, it may save the login screen instead of the page.
 
-*Note: not all extractors use Chrome (e.g. `wget`, `mercury`, `media`). Importing a dedicated host browser profile into a persona also exports its cookies for those extractors; directly logging in through a new ArchiveBox Chrome profile does not.*
-
-> [!WARNING]
-> **We strongly recommend you use [separate burner credentials dedicated to archiving](https://docs.sweeting.me/s/cookie-dilemma),** e.g. don't provide cookies for your normal daily Facebook/Instagram/Google/etc. accounts as server responses and page content will often contain your name/email/PII, session cookies, private tokens, etc. which then get preserved in your snapshots for eternity.
->  
-> Future viewers of your archive may be able to use any reflected archived session tokens to log in as you, or at the very least, associate the content with your real identity. Even if this tradeoff seems acceptable now or you plan to keep your archive data private, you may want to share a snapshot with others in the future, and snapshots are very hard to sanitize/anonymize after-the-fact!
->
-> For this reason, it's best to set up dedicated fake profile accounts for each site you want to archive, and consider them burned if you ever share any of your archived snapshots of those sites with untrusted people.
+A **browser profile** stores your logins and browser settings. Importing it into ArchiveBox lets ArchiveBox view pages using your account. ArchiveBox calls its copy a **persona**. Select that persona when saving a page.
 
+Profile importing is for **Chromium-based browsers**. The examples below use Chrome; the same workflow applies to other browsers in that family. You can also [log in using ArchiveBox's own browser](#setting-up-a-new-profile) instead of importing a profile.
 <a name="docker-setup"></a>
 <a name="Docker-Setup"></a>
 
-### Docker VNC Setup
+## Import an existing browser profile
 
-If using ArchiveBox in Docker, the easiest way to set up session credentials is by remote controlling the ArchiveBox Chrome browser over VNC, and using it to log in to the sites you want to save.
+These steps cover the common setup where your browser is on your computer and ArchiveBox runs in Docker on a server. You will export the profile on your computer, transfer the exported persona directly into the server's ArchiveBox data folder with `rsync`.
 
-1. Enable the `novnc` server using these settings in your `docker-compose.yml`:
+You need the [ArchiveBox CLI](Install.md) on your computer and SSH access to your server. The temporary folder below is only for exporting the profile; you do not need to download your server's archive.
 
-`docker-compose.yml`:
-```yaml
-services:
-    archivebox:
-        ...
-        volumes:
-            ...
-        environment:
-            - DISPLAY=novnc:0.0
-            
-    novnc:
-        image: theasp/novnc:latest
-        environment:
-            - DISPLAY_WIDTH=1920
-            - DISPLAY_HEIGHT=1080
-            - RUN_XTERM=no
-        ports:
-            - "8080:8080"
-```
+If ArchiveBox runs on the same computer as your browser, use your existing ArchiveBox data folder for step 1, skip step 2, and run `archivebox add --persona=personal URL` in step 3.
 
-2. Start the `novnc` window server container
-```bash
-docker compose up -d novnc
-# wait a few seconds for novnc to start...
-```
+### 1. Export the profile on your computer
 
-3. Start ArchiveBox's Chrome inside Docker
-```bash
-docker compose run --rm archivebox persona create personal
-docker compose run --rm archivebox /data/lib/env/bin/chromium --user-data-dir=/data/personas/personal/chrome_profile --profile-directory=Default --disable-gpu --disable-features=dbus --disable-dev-shm-usage --start-maximized --no-sandbox --disable-setuid-sandbox --no-zygote --disable-sync --no-first-run
-```
-<small>(make sure you set `DISPLAY` and keep the normal persistent `/data` volume from the Compose setup!)</small>
-
-4. Open [`http://localhost:8080/vnc.html`](http://localhost:8080/vnc.html) in your browser. You should see a remote linux desktop shown with Chrome open, allowing you to remote-control ArchiveBox's browser. Use it to log into any sites where you want to save credentials.
-
-5. ✅ Close the browser, stop & remove novnc, and then select the `personal` persona when archiving. Chrome-based extractors will use the saved profile and should see the sites as logged in.
+Open a page you want to save and check that you can view it while logged in. Close the browser, then create a temporary export folder:
 
 ```bash
-# stop the archivebox and novnc containers
-docker compose down
-docker compose down --remove-orphans
-# edit docker-compose.yml to remove/comment out the novnc: section
-
-# test it all out by archiving something hosted on one of the domains you logged in to
-docker compose run --rm archivebox add --persona=personal 'https://private.example.com/some/site/requiring/login.html'
-# check the SingleFile, Screenshot, DOM, or PDF snapshot output (only these use the Chrome profile)
-# make sure the content appears as your logged-in user would see it
+mkdir -p ~/archivebox-profile-export
+cd ~/archivebox-profile-export
+archivebox init
 ```
 
-Under the hood this uses [Xvfb](https://www.x.org/releases/X11R7.6/doc/man/man1/Xvfb.1.xhtml) + [Fluxbox](http://www.fluxbox.org/) + [`novnc`](https://github.com/theasp/docker-novnc) to provide a virtual display, window manager, and VNC server + novnc websocket viewer.
-<br/>
+**Using the CLI:**
+
+```bash
+archivebox persona create --import=chrome personal
+```
+
+`personal` is a name you choose for the persona. Replace `chrome` with your browser's import name: `chromium`, `brave`, or `edge` for those browsers. The command chooses a profile automatically; use `--profile=Default` or `--profile='Profile 1'` to select a specific one.
+
+**Using the UI instead:** run `archivebox server` from the export folder and follow the [web UI setup](Usage.md#ui-usage). Choose **Admin → Personas & Configs → Add persona → Use a detected profile**, name it `personal`, select the browser profile, and save. You can stop this local ArchiveBox server afterward.
+
+Both methods create `~/archivebox-profile-export/personas/personal`, ready to transfer.
+
+<details>
+<summary>My browser or profile is not detected</summary>
+
+Check the **Profile Path** on your browser's version page (`chrome://version` in Chrome). The final folder name identifies the profile, such as `Default` or `Profile 1`.
+
+For a browser installed in a different location, the CLI accepts `--source` for its profile path and `--browser-binary` for its executable. Run `archivebox persona create --help` for the available options. Automatic detection currently lists Chrome, Chromium, Brave, and Edge; other Chromium-based browsers need their paths supplied explicitly.
+
+</details>
+
+### 2. Transfer the persona to your server
+
+Run these commands **on your computer**. Replace `user@your-server` with your SSH login and `~/archivebox/data` with your server's ArchiveBox data folder (the folder mounted at `/data` in Docker):
+
+```bash
+ssh user@your-server 'mkdir -p ~/archivebox/data/personas/personal'
+rsync -av ~/archivebox-profile-export/personas/personal/ user@your-server:~/archivebox/data/personas/personal/
+```
+
+The persona is now in place. Docker uses the existing data mount; no Compose changes are needed.
+
+<a name="non-docker-setup-remote-host"></a>
+
+For a server without Docker, transfer to `personas/personal/` inside its ArchiveBox data folder in the same way.
+
+### 3. Save and check the page
+
+Connect to your server and enter the folder containing its Docker Compose file:
+
+```bash
+ssh user@your-server
+cd ~/archivebox
+```
+
+Replace the example URL with the page you want to save:
+
+```bash
+docker compose exec archivebox archivebox add --persona=personal 'https://example.com/page'
+```
+
+For a server without Docker, run `archivebox add --persona=personal URL` from its data folder.
+
+**Using the UI instead:** if `personal` is not listed yet, choose **Admin → Personas & Configs → Add persona → Blank Persona**, name it `personal`, and save. This registers the transferred folder without changing its contents. Then open **Add URL**, paste the URL, and select `personal` before starting the capture.
+
+Open the saved screenshot or HTML. Check that it contains the content you wanted, rather than a login screen. If the site requires you to log in again later, sign in in your browser and repeat the export and transfer.
+
+Treat imported personas and logged-in snapshots as private: they can contain account information. Review the [guidance on sharing archives](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing) before publishing them.
+
+## Setting up a new profile
+
+Use this option if you prefer to log in through ArchiveBox's browser instead of importing a browser profile. Log in once, then select the same persona for future captures.
 
 ### Non-Docker Setup (Local Host)
 
-If running ArchiveBox on your local machine without Docker, this process is fairly easy.
-
-First, create a persona to hold the dedicated Chrome profile.
+From your ArchiveBox data folder:
 
 ```bash
 archivebox persona create personal
+archivebox persona open personal
 ```
 
-Then install/resolve Chrome and launch the projected browser with that profile dir:
+Log in to the sites you want to archive, then close the browser. Save a page with `archivebox add --persona=personal URL` and check the resulting Screenshot, DOM, PDF, or SingleFile output.
+
+### Docker VNC Setup
+
+This opens the selected persona in a browser you can control remotely. It also works with an [imported profile](#import-an-existing-browser-profile).
+
+1. In your [Docker Compose file](https://docker-compose.archivebox.io), uncomment the `novnc` service and the `depends_on` block under `archivebox`. Uncomment `ARCHIVEBOX_VNC_PERSONA=personal` to select your persona.
+
+2. From the Compose folder, run:
 
 ```bash
-archivebox install chrome
-./lib/env/bin/chromium --user-data-dir="$PWD/personas/personal/chrome_profile"
+docker compose up -d
 ```
 
-Once it's open, log in to all the sites you want to be logged in to for archiving, then close/quit Chrome.
+ArchiveBox waits for VNC to be ready, then opens the persona's browser with its imported logins. If an ArchiveBox browser is already open on that display, it waits for that browser to close first.
 
-✅ Chrome-based extractors (e.g. Screenshot, PDF, DOM, Singlefile) use that profile whenever you archive with `--persona=personal`.
+3. Visit <http://localhost:8080/vnc.html> and click **Connect**. For a remote server, first run this on your computer and keep the connection open:
 
-Directly logging in through this profile does not generate a `cookies.txt` for non-Chrome extractors. If those extractors need the same login state, use the recommended [`archivebox persona create --import=chrome personal`](https://github.com/ArchiveBox/ArchiveBox/wiki/Personas) workflow with a dedicated host browser profile instead; the import copies the Chrome profile and exports its cookies together.
+```bash
+ssh -N -L 8080:127.0.0.1:8080 user@your-server
+```
 
-<br/>
+4. Open a page and confirm you can view it while logged in. For a new persona, log in to the sites you want to archive. Then follow [Save and check the page](#3-save-and-check-the-page), selecting the same persona.
 
-### Non-Docker Setup (Remote Host)
+#### One-off Compose setup
 
-You must set up the profile using the exact same version of Chrome that ArchiveBox is running. Run `archivebox install chrome` and `archivebox version` on each machine so `abxpkg` selects and validates the browser.
+To set up a persona without starting the ArchiveBox server, enable the same `novnc` service and healthy `depends_on` block, then run:
 
-**General steps:**
+```bash
+docker compose up -d --wait novnc
+docker compose run --rm archivebox persona create personal
+docker compose run --rm -e DISPLAY=novnc:0.0 archivebox persona open personal
+```
 
-1. Make sure you are running the same OS and have the same version of Chrome installed as the host running ArchiveBox
-2. Follow the `Non-Docker Setup (Local Host)` steps above to create the `personal` persona and Chrome profile locally
-3. Create the same persona from the ArchiveBox data directory on the remote host: `archivebox persona create personal`
-4. Rsync the persona's Chrome profile from your local collection into the matching remote persona: `rsync --archive ~/archivebox/data/personas/personal/chrome_profile/ remotehost:~/archivebox/data/personas/personal/chrome_profile/`
+Connect at <http://localhost:8080/vnc.html>, log in, and close the browser when finished. The persona remains in `data/personas/personal` across one-off containers.
 
-You may need to run `chown -R archivebox ~/archivebox/data/personas/personal/chrome_profile` on the remote host to make the profile editable by the `archivebox` user on that machine.
+`persona open` resolves the installed Chromium binary automatically. Do not hard-code `/data/lib/env/bin/chromium` or a versioned Playwright cache path: the browser location varies by image version and architecture.
 
-✅ Chrome-based extractors (e.g. Screenshot, PDF, DOM, Singlefile) use that profile whenever you archive with `--persona=personal`.
+An explicitly set `DISPLAY` is preserved. When it is unset, ArchiveBox probes `novnc:0.0` and falls back to headless operation if unavailable. Set `DISPLAY=` to disable this automatic display selection.
 
-If non-Chrome extractors need the same login state, prefer importing a dedicated host browser profile with `archivebox persona create --import=chrome personal` so the persona receives both the Chrome profile and an exported `cookies.txt`.
+## Troubleshooting Chromium Install
 
----
+<a name="more-info--troubleshooting"></a>
 
-## More Info & Troubleshooting
-
-- https://github.com/ArchiveBox/ArchiveBox/issues/952
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#archiving-private-content
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#%EF%B8%8F-things-to-watch-out-for-%EF%B8%8F
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing
-- https://archivebox.github.io/abx-plugins/#chrome (CHROME_USER_DATA_DIR, CHROME_BINARY, etc.)
-- https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file
+- **Browser missing or unable to start:** see [Chromium troubleshooting](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting#chromiumgoogle-chrome).
+- **Saved a login screen:** confirm you can view the page in the source browser, re-import the correct profile, and select that persona when saving.
+- **Need browser settings:** see the [Chrome plugin configuration](https://archivebox.github.io/abx-plugins/#chrome).

docs/Configuration.md

@@ -411,6 +411,8 @@ ArchiveBox splits its surfaces across three logical hosts: `admin.*` (Django adm
 
 SingleFile output is served as ordinary HTML in every mode; it remains usable in no-JS modes because SingleFile removes the page's scripts during capture. ArchiveWeb.page/ReplayWeb.page and MHTML use their existing trusted preview templates. ArchiveBox always attempts to load these viewers, including when the incoming request is plain HTTP, because HTTPS may be terminated by an upstream proxy. Browser service-worker rules still require ReplayWeb.page to be reached through HTTPS or localhost for replay to initialize.
 
+Trusted preview wrappers allow embedding by their own origin and the configured web/admin origins, so the collection UI can display snapshot-host previews without opening them to arbitrary embedding sites. PDF previews use a document frame rather than an object/embed, retaining `object-src 'none'` on the wrapper. Allowing a trusted viewer's scripts does not exempt the archived documents inside it from their replay policy; the MHTML viewer keeps its archived child document sandboxed without scripts.
+
 > [!WARNING]
 > Switching to any mode whose name starts with `unsafe-` or `danger-` is logged at startup and surfaces a banner in the UI. **Don't use these modes on a public hostname** — archived JavaScript will run on the same origin as your admin session.
 
@@ -585,6 +587,9 @@ When `True`, every LDAP user who successfully authenticates is auto-promoted to
 
 *Options for the on-disk layout, file permissions, and temp/lib directories that ArchiveBox reads and writes during archiving.*
 
+> [!NOTE]
+> Docker-only [`PUID` and `PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid) are entrypoint environment variables, not ArchiveBox configuration keys.
+
 ---
 <a id="dir_output_permissions"></a>
 #### `OUTPUT_PERMISSIONS`

docs/Docker.md

@@ -1,5 +1,7 @@
 # Docker
 
+For desktop browser cookies and bookmarks, see [Chrome profile setup](Chromium-Install.md#import-an-existing-browser-profile).
+
 ## Overview
 
 Running ArchiveBox with Docker allows you to manage it in a container without exposing it to the rest of your system. ArchiveBox generally works the same in Docker as it does outside Docker. You can even use `uv`-installed ArchiveBox and Docker ArchiveBox in tandem, as they both share the same data directory format.
@@ -126,7 +128,18 @@ docker compose exec archivebox archivebox add --depth=1 'https://example.com/som
 
 ### Accessing the data
 
-The outputted archive data is stored in `data/` (relative to the project root), or whatever folder path you specified in the `docker-compose.yml` `volumes:` section. The mounted directory must be writable by its current owner; the entrypoint detects that non-root owner and runs ArchiveBox with matching permissions.
+The outputted archive data is stored in `data/` (relative to the project root), or whatever folder path you specified in the `docker-compose.yml` `volumes:` section. The mounted directory must be writable by its current owner; by default the entrypoint detects that non-root owner and runs ArchiveBox with matching permissions.
+
+<a id="puid--pgid"></a>
+#### Docker `PUID` / `PGID`
+
+`PUID` and `PGID` are optional Docker-entrypoint environment variables for mounts with a fixed numeric identity, such as NFS/CIFS/FUSE. They override owner autodetection and are not ArchiveBox config keys: do not put them in `ArchiveBox.conf` or use `archivebox config --set`. Both must be numeric. `PUID=0` is ignored in favor of the detected/default non-root UID so ArchiveBox and Chrome never run as root; `PGID=0` is permitted for a group-writable mount.
+
+```bash
+PUID=1000 PGID=1000 docker compose up -d --wait
+```
+
+Leave them unset for normal local bind mounts. When you do set them, match the effective numeric owner/group or ACL configured by the storage server; container root cannot override server-side permission enforcement such as NFS `root_squash`.
 
 To access a result directly via the filesystem, follow its backwards-compatible `./data/archive/<timestamp>` symlink, or browse the canonical `./data/archive/users/<user>/snapshots/<date>/<domain>/<uuid>/` tree.
 
@@ -205,7 +218,7 @@ Then open `/admin/` on the hostname or IP used to reach ArchiveBox (local exampl
 
 *(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
 
-If you encounter permissions issues, make sure the mounted data directory is writable by its intended owner. Docker startup automatically uses the first non-root owner detected from the existing collection, or the default `archivebox` user when the data directory is root-owned.
+If you encounter permissions issues, make sure the mounted data directory is writable by its intended owner. Docker startup uses explicit [`PUID`/`PGID`](#puid--pgid) values when supplied, otherwise the first non-root owner detected from the existing collection or the default `archivebox` user when the data directory is root-owned.
 
 <br/>
 

docs/Release-Notes-v0.9.md

@@ -0,0 +1,157 @@
+# ArchiveBox v0.9.x: Safer, Faster, and Much More Powerful Web Archiving
+
+ArchiveBox v0.9 is our biggest release since the 0.7.x series. It makes everyday archiving easier to set up, easier to operate, more reliable at scale, and substantially safer when browsing saved pages.
+
+> [!IMPORTANT]
+> Back up your ArchiveBox data directory before upgrading. v0.9 migrates both the database and on-disk archive layout. Test the upgrade on a copy first, especially for large or customized installations.
+
+## ✨ What's New in v0.9.x
+
+- 🧙 **A much easier first-run experience.** A new browser-based setup wizard guides you through creating an admin account, choosing collection settings, and getting your first archive running without hand-editing configuration files.
+
+- 📱 **A faster, richer interface on every device.** Snapshot cards, previews, action menus, archive-result views, and nearly every other page have been redesigned to work comfortably across phones, tablets, and desktops. Almost all UI pages are served in ~100ms or less—even with 1 million Snapshots in the database.
+
+- 📡 **Live progress you can actually follow.** Crawl, Snapshot, process, and extractor status now update while work is running, with clearer visibility into what succeeded, what failed, and what is still queued.
+
+- ⏯️ **Durable crawls that survive interruptions.** Crawls can be paused, resumed, aborted, retried, and recovered after restarts instead of existing only as a single fragile command. ArchiveBox also prevents multiple competing runners from processing the same collection.
+
+- 🗓️ **Scheduling is built into ArchiveBox.** Recurring imports and crawls are stored in the database and run by the server, removing the need to maintain a separate cron process or Docker sidecar.
+
+- 👤 **Personas preserve real browser identities.** Import a Chrome profile—including cookies and authenticated sessions—into a named Persona, then choose that Persona when archiving sites that require a login or personalized browser state.
+
+  ```bash
+  archivebox persona create --import=chrome personal
+  archivebox add --persona=personal https://example.com
+  ```
+
+- 🎯 **More consistent, reliable captures.** Extractors now share clearer configuration and output rules, dependencies can be installed automatically, and each extractor writes to a predictable namespaced folder. This makes failures easier to understand and custom capture workflows easier to build.
+
+- 🌐 **Better browser isolation between crawls.** Chrome processes, profiles, tabs, and sessions are tracked per crawl so simultaneous jobs interfere with each other less and browser resources are cleaned up more reliably.
+
+- 🛡️ **Safer replay of archived pages.** ArchiveBox now separates the admin, API, web UI, and archived-page origins. Full interactive replay can use isolated `*.localhost` subdomains, while normal hosts use a safer no-JavaScript replay mode to reduce the risk from untrusted archived content.
+
+- 🔐 **More precise privacy and stronger request security.** Individual Snapshots can be public, unlisted, or private, and authentication, cookie, CSRF, redirect, and webhook handling have been hardened across the web and API surfaces.
+
+- 🔎 **Much better search choices.** Collections can use SQLite full-text search, Sonic, or ripgrep depending on size and deployment needs, with improved indexing and search behavior across titles, URLs, tags, and captured text.
+
+- 🐘 **Optional PostgreSQL support.** SQLite remains the simple default, while larger or more concurrent deployments can now use PostgreSQL. SQLite locking and Docker bind-mount behavior have also been improved.
+
+- 🔌 **A substantially expanded automation surface.** The Django Ninja REST API now exposes more collection, crawl, Snapshot, result, tag, user, token, and operational workflows. API tokens, outbound webhooks, and browser-extension uploads make ArchiveBox easier to connect to other tools.
+
+- 🤖 **Optional AI agents and Claude-powered archiving.** Disabled by default, the built-in `archivebox mcp` server lets a dedicated AI agent perform crawls, search and manage your archive, prune duplicate results, and more. New optional plugins add [Claude-powered browser interaction](https://archivebox.github.io/abx-plugins/#claudechrome), [custom content extraction](https://archivebox.github.io/abx-plugins/#claudecodeextract), and [duplicate-result cleanup](https://archivebox.github.io/abx-plugins/#claudecodecleanup).
+
+- 📦 **Simpler dependency and installation management.** `archivebox install` can discover and install required browser and command-line dependencies, and `uv tool install archivebox` provides a cleaner native installation path with isolated Python dependencies.
+
+- 🐳 **More dependable deployments and packages.** Docker startup, health checks, configuration, permissions, Chromium behavior, and multi-container operation have been tightened up. Homebrew and Debian packaging are available again as thin wrappers around the same supported ArchiveBox runtime.
+
+- 🗂️ **A scalable, inspectable archive layout.** New archives use human-readable date/domain paths with a stable Snapshot UUID and JSONL metadata. Static exports, large-collection traversal, and moving or inspecting archives outside ArchiveBox are more reliable.
+
+- 🚦 **Better controls for large and unattended crawls.** New crawl limits and retention settings can bound runtime, depth, size, and stored output, while improved process and machine visibility makes it easier to diagnose stuck or resource-heavy jobs.
+
+## ⬆️ Upgrading
+
+Upgrade from a backup and allow time for both database and filesystem migrations.
+
+### Docker Compose
+
+```bash
+cd ~/archivebox
+docker compose down
+cp -a data "data.backup-$(date +%Y%m%d)"
+docker compose pull
+docker compose run --rm archivebox init
+docker compose run --rm archivebox update --migrate-only
+docker compose up -d
+docker compose exec archivebox archivebox status
+```
+
+### Native / uv
+
+```bash
+uv tool install --python 3.13 --upgrade archivebox
+cd ~/archivebox/data
+archivebox init
+archivebox install
+archivebox update --migrate-only
+archivebox status
+```
+
+<details>
+<summary><strong>🏗️ Internal architecture and package changes</strong></summary>
+
+ArchiveBox's capture stack is now split into smaller reusable projects:
+
+- [abxbus](https://github.com/ArchiveBox/abxbus) provides typed events and process coordination.
+- [abxpkg](https://github.com/ArchiveBox/abxpkg) discovers, installs, and runs external binary dependencies.
+- [abx-plugins](https://github.com/ArchiveBox/abx-plugins) contains extractor and plugin definitions.
+- [abx-dl](https://github.com/ArchiveBox/abx-dl) provides the standalone downloader and extraction pipeline.
+- [ArchiveBox](https://github.com/ArchiveBox/ArchiveBox) adds the database, web UI, API, scheduling, access control, and collection management.
+
+The application now records crawls, processes, machines, workers, configurations, binaries, and archive results as durable database objects. This supports live status, recovery, scheduling, auditing, and clearer failure reporting without making those implementation details part of the normal archiving workflow.
+
+Release builds are versioned through this dependency chain, so final package and container releases must use mutually compatible versions.
+
+</details>
+
+<details>
+<summary><strong>🔄 Migration and filesystem details</strong></summary>
+
+The v0.9 migration preserves existing Snapshots, tags, users, ArchiveResults, and archive files while upgrading the database and converting legacy archive directories to the new layout:
+
+```text
+data/archive/users/<username>/snapshots/<date>/<domain>/<uuid>/index.jsonl
+```
+
+Run `archivebox init` first to apply database migrations, then `archivebox update --migrate-only` to migrate archive directories. The migration is designed to be resumable and idempotent, but large collections can take significant time and should always be upgraded from a tested backup.
+
+The 0.7.x stable line and the 0.9 development line diverged historically, so these notes summarize the user-visible change between the release families rather than a single linear commit range.
+
+</details>
+
+<details>
+<summary><strong>⚠️ Compatibility notes</strong></summary>
+
+- Python 3.13 or newer is required for native installs.
+- Custom plugins and extractors must use the current executable plugin contract and namespaced output directories.
+- Access control is now evaluated per Snapshot; review public, unlisted, and private defaults after migration.
+- Only one active runner should operate on a given ArchiveBox data directory.
+- Native installations should run `archivebox install` after upgrading to resolve external dependencies.
+- PostgreSQL is optional, but ArchiveBox does not provide an automatic SQLite-to-PostgreSQL data conversion.
+- Test customized Docker Compose files, reverse proxies, replay hostnames, and webhook integrations against a copied collection before production rollout.
+
+</details>
+
+<details>
+<summary><strong>🙌 Contributors and reporters</strong></summary>
+
+Thank you to everyone who contributed code, documentation, testing, bug reports, and responsible security reports during the long road from 0.7.x to 0.9.x.
+
+**Code and documentation contributors**
+Sorted approximately by the size and quantity of accepted contributions:
+
+@Brandl, @jimwins, @benmuth, @FellowTraveler, @pcrockett, @tqobqbq, @pellaeon, @sclu1034, @vladimirdulov, @andrew-d, @n-hebert, @gnattu, @danielalanbates, @boehs, @pyrox0, @ckiee, @neel-suthar, @dicnunz, @zkksdk, @1over137, @agowa, @benharri, @ckcr4lyf, @jasongodev, @naoph, @rdela, @slmingol, @ssoel, @TrisSherliker, @NelsonMinar, and @mpgirro.
+
+**Issue reporters whose reports led to fixes**
+This list excludes people already credited above for accepted code or documentation:
+
+@amy-r-oss, @saiarcot895, @cyberproaustin, @BenCzaczkes, @larsony99, @tztzz, @JitteryDoodle, @Finkregh, @philippemilink, @sbutcher, @melyux, @m0nhawk, @danst0, @KagurazakaShirosatosu, @s7x, @mawmawmawm, and @cdzombak.
+
+**Security advisory reporters explicitly promised release-note credit**
+This is intentionally limited to advisory threads where the maintainer explicitly said the reporter would be credited:
+
+@DavidCarliez, @FUNFACTOR1, @g4nkd, @geo-chen, @iaohkut-from-NightWolf-Team, and @Vasco0x4.
+
+</details>
+
+<details>
+<summary><strong>🔗 Related projects, documentation, and full changelog</strong></summary>
+
+- [ArchiveBox documentation](https://docs.archivebox.io/)
+- [ArchiveBox browser extension](https://github.com/ArchiveBox/archivebox-browser-extension)
+- [ArchiveBox REST API documentation](https://docs.archivebox.io/dev/#rest-api)
+- [ArchiveBox releases](https://github.com/ArchiveBox/ArchiveBox/releases)
+- [Full source comparison from v0.7.4 to the current v0.9 release candidate](https://github.com/ArchiveBox/ArchiveBox/compare/v0.7.4...v0.9.35rc410)
+
+Before publishing, replace the release-candidate version in the heading and comparison link with the final v0.9 tag.
+
+</details>

docs/Security-Overview.md

@@ -150,7 +150,7 @@ How much are you planning to archive? Only a few bookmarked articles, or thousan
 
 Consider what permissioning to apply to your archive folder carefully. Limit access to the fewest possible users by checking folder ownership and setting [`OUTPUT_PERMISSIONS`](https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#output_permissions) accordingly. Generally the `index.sqlite3` file, `archive/` folder, and `ArchiveBox.conf` file must all be owned and writable by the `archivebox` user or a dedicated non-root user.
 
-When running with Docker, the entrypoint uses the existing non-root owner of the mounted data directory when possible, otherwise it falls back to the image's `archivebox` user.
+When running with Docker, the entrypoint uses explicit [`PUID`/`PGID`](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#puid--pgid) values when supplied, otherwise the existing non-root owner of the mounted data directory or the image's `archivebox` user.
 
 More info:
 - https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout

docs/Setting-Up-Storage.md

@@ -127,6 +127,8 @@ The `data/archive/` subfolder contains the bulk archived content, and it support
 
 ### `NFS` (Docker Driver)
 
+Configure the export's ownership or ACL so a non-root numeric identity can create and remove files, then set Docker `PUID`/`PGID` to that identity if it differs from the local collection owner. NFS `root_squash` is compatible; ArchiveBox does not rely on container root to write the export.
+
 `docker-compose.yml`:
 ```yaml
 services:
@@ -148,6 +150,8 @@ volumes:
 
 ### `SMB` / `Ceph` (Docker CIFS Driver)
 
+The example below presents the mount as the image default `911:911`. If you choose another forced `uid`/`gid`, set the ArchiveBox service's `PUID`/`PGID` to the same values.
+
 `docker-compose.yml`:
 ```yaml
 services:
@@ -331,7 +335,8 @@ volumes:
             allow_other: 'true'
             vfs_cache_mode: full
             vfs_links: 'true'
-            # Match these to the numeric owner of ./data; 911:911 is the image default.
+            # Match these and the ArchiveBox PUID/PGID to the effective mount owner;
+            # 911:911 is the image default.
             uid: 911
             gid: 911
             transfers: 16

docs/Troubleshooting.md

@@ -134,7 +134,23 @@ if you have problem with a particular nginx config.
 
 #### Docker Permissions issues
 
-Make sure the mounted data directory is writable by its intended non-root owner. The current Docker entrypoint detects the first non-root collection owner and runs ArchiveBox with matching numeric UID/GID; a new root-owned collection falls back to the image's `archivebox` user. Check the host directory's numeric ownership and the entrypoint's startup output before changing permissions.
+The Docker entrypoint starts as root for mount and runtime-cache setup, then runs ArchiveBox and Chrome as a non-root user. It checks required paths as that user first, so normal starts do not traverse cache trees; an inaccessible runtime/cache tree may need a one-time recursive repair. Browser-lock cleanup checks only known files at each persona profile root, not the contents of large profiles. By default the entrypoint uses the first non-root numeric owner found in the collection, falling back to `911:911` for a new or root-owned collection. Set `PUID` and `PGID` when an NFS/CIFS/FUSE server or host requires a specific numeric identity.
+
+Required collection directories get a non-root access check or creation attempt first, followed by real create/delete probes on the critical output paths. Only a failed check triggers a shallow root `chown`/`chmod` of the exact collection path. This avoids repeated metadata changes on writable `root_squash` or synthetic-permission mounts. Startup never recursively scans or changes `data/archive`: byte size is not a safe proxy for traversal cost, and a large or metadata-heavy archive can take hours or days to walk.
+
+With SQLite, `index.sqlite3`, configuration, logs, temp/runtime files, and sidecar databases must remain on reliable local storage; only `data/archive/` may be remote. PostgreSQL is the supported exception for the main index.
+
+The default `restart: unless-stopped` policy repeats a startup failure but does not create it and should not be removed. If the entrypoint still cannot create and delete its probe file, stop the noisy loop, inspect the resolved volume sources and logs, then retry once after fixing the server-side UID/GID mapping or ACL:
+
+```bash
+docker compose stop archivebox
+docker compose config
+docker compose logs --tail=200 archivebox
+docker compose run --rm archivebox init
+docker compose up -d --wait
+```
+
+A read-only mount or NFS export that denies both the selected user and root cannot be repaired from inside the container.
 
 <br/>
 

docs/Usage.md

@@ -125,16 +125,7 @@ archivebox add < safari_history.json
 
 ### Import browser cookies into a persona
 
-To archive logged-in sites, you can import cookies from your browser into a persona. This generates a `cookies.txt` file in the persona directory (used by wget/curl/yt-dlp, etc.) and, for Chromium-based browsers, also copies the profile into the persona so Chrome-based extractors can reuse it.
-
-```bash
-archivebox persona create --import=chrome personal
-# supported: chrome/chromium/brave/edge (Chromium-based only)
-# use --profile to target a specific profile (e.g. Default, Profile 1)
-# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
-```
-
-If cookie extraction fails, you can still export a Netscape-format `cookies.txt` using a browser extension and place it at `data/personas/<NAME>/cookies.txt`.
+See [Chrome profile setup](Chromium-Install.md#import-an-existing-browser-profile) for cookies, bookmarks, and host/Docker imports. Select the imported identity with `archivebox add --persona=personal URL`.
 
 <br/>
 

docs/codeblocks.toml

@@ -21,7 +21,7 @@ version = 2
 "8fe4a2c15abb0ab2-1" = "run"
 "2b08779e4d24d305-1" = "run"
 "588be84a1f0fda9a-3" = "illustration"
-"1b9438e093866ff0-1" = "illustration"
+"9ab59dbbf9d5cedb-1" = "illustration"
 "6392c7b5f3593d07-1" = "run"
 "fe137daf1317bcd6-1" = "run"
 "724cf9fe3251b66f-1" = "illustration"
@@ -83,13 +83,15 @@ version = 2
 
 # docs/Chromium-Install.md
 "201988374bff6d14-1" = "run"
-"d85a11895d69713a-1" = "run"
-"d8061ea425363f8c-1" = "illustration"
-"bd5e6d3fc3193a25-1" = "illustration"
-"e41aab6197f32cb4-1" = "illustration"
-"c5c46f8a1c693a4c-1" = "illustration"
-"ce2edd9be87280fc-1" = "run"
-"7890998169008ad0-1" = "illustration"
+"e305c234db136566-1" = "run"
+"56056d409ad85f85-1" = "illustration"
+"baaf307bd7ee0f36-1" = "illustration"
+"b274b9afd4304207-1" = "illustration"
+"3bc4ad1dc41c32c6-1" = "illustration"
+"3dcfedd701e87748-1" = "illustration"
+"6c64433356e03458-1" = "illustration"
+"e9088da849bc5755-1" = "illustration"
+"b32c747a730e6f8b-1" = "illustration"
 
 # docs/Configuration.md
 "129eb9ca56f17500-1" = "illustration"
@@ -113,6 +115,7 @@ version = 2
 "746a3e228d29f9d2-1" = "run"
 "66a7d54ea81d64dc-1" = "illustration"
 "aa4884ebe3101cf6-1" = "illustration"
+"0b7277780e3efb1a-1" = "illustration"
 "150ae2c7f6d012ab-1" = "illustration"
 "8d86727108f1dbad-1" = "run"
 "6bcb6c2e8770907a-1" = "illustration"
@@ -166,6 +169,12 @@ version = 2
 "16a101113c74c070-1" = "illustration"
 "c84835e713d1c625-1" = "run"
 
+# docs/Release-Notes-v0.9.md
+"7f9cf63fad110997-1" = "illustration"
+"22c61747fad47c75-1" = "illustration"
+"693b5aec2007eea7-1" = "illustration"
+"c28c2cb9ba1f19ae-1" = "illustration"
+
 # docs/Scheduled-Archiving.md
 "e6102e50b4491a0d-1" = "illustration"
 "e256179d1504c37d-1" = "illustration"
@@ -195,7 +204,7 @@ version = 2
 "0fdcf228bb957933-1" = "run"
 "d76a2cce93150387-1" = "illustration"
 "acb33c3336d33ac5-1" = "illustration"
-"8bc58240c72e3eaa-1" = "illustration"
+"64a482f8906997f7-1" = "illustration"
 "74e75b4a84bbb636-1" = "run"
 
 # docs/Setting-up-Authentication.md
@@ -224,6 +233,7 @@ version = 2
 "201988374bff6d14-2" = "run"
 "725a7a8c6f829ece-1" = "run"
 "7658fabf90ed3c2e-1" = "run"
+"1192620577a0dc5a-1" = "illustration"
 "9612e7b40b348ab1-1" = "illustration"
 "2f491912a15ecd93-1" = "output"
 "e0cbb64ddf1d7135-1" = "run"
@@ -246,7 +256,6 @@ version = 2
 "6611c2dee47b1092-1" = "illustration"
 "2560680733c8e908-1" = "illustration"
 "debc2548308dec0c-1" = "illustration"
-"d29b422c54745a7a-1" = "illustration"
 "f6a482c5e0582b05-1" = "illustration"
 "5cd641727f4b04bb-1" = "output"
 "f0e895e5b375c198-1" = "illustration"
@@ -284,8 +293,7 @@ version = 2
 "46fe9d81940ea385-1" = "collection"
 "396490664ce459e9-2" = "project"
 "201988374bff6d14-1" = "collection"
-"d85a11895d69713a-1" = "collection"
-"ce2edd9be87280fc-1" = "collection"
+"e305c234db136566-1" = "collection"
 "798793d9d6b2e12e-1" = "collection"
 "ce0de8ed0d7cd552-1" = "collection"
 "83cb9be168d203ca-1" = "collection"
@@ -364,8 +372,7 @@ version = 2
 "46fe9d81940ea385-1" = "ubuntu"
 "396490664ce459e9-2" = "ubuntu"
 "201988374bff6d14-1" = "ubuntu"
-"d85a11895d69713a-1" = "ubuntu"
-"ce2edd9be87280fc-1" = "ubuntu"
+"e305c234db136566-1" = "ubuntu"
 "798793d9d6b2e12e-1" = "ubuntu"
 "ce0de8ed0d7cd552-1" = "ubuntu"
 "83cb9be168d203ca-1" = "ubuntu"