Keep macOS process inspection failures out of SQLite recovery

Machine.save intentionally invalidates the process cache so config edits are visible immediately. Keep a local instance through host refresh, sanitation, and file reconciliation, then publish the completed instance to the cache. Explain why invalidation must remain and why reading the global after save can return None.

Add a real CLI regression that ages persisted machine rows, runs archivebox status, and verifies successful refresh without changing machine identities. Confirmed failing before the fix; 13 targeted tests and Ruff checks pass.

.dockerignore

@@ -75,6 +75,15 @@ docker/
 website/
 typings/
 
+# The runtime COPY needs ArchiveBox's package and bin/ entrypoint, not the
+# development UI screenshots or test trees. PNGs are already compressed, so
+# including them creates a large image layer that later cleanup cannot remove.
+publicsite/
+old/
+test/
+tests/
+archivebox/tests/
+
 tmp/
 .tmp/
 data/

.github/ISSUE_TEMPLATE/1-bug_report.yml

@@ -191,7 +191,6 @@ body:
         In the meantime please consider:  
           
           - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 👨‍✈️ [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
           - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
           - 📚 Reading the [Troubleshooting Guide](https://github.com/ArchiveBox/ArchiveBox/wiki)
           - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)

.github/ISSUE_TEMPLATE/2-feature_request.yml

@@ -109,9 +109,6 @@ body:
           required: false
         - label: "I've had a lot of difficulty getting ArchiveBox set up"
           required: false
-        - label: "I would pay $10/mo for a hosted version of ArchiveBox if it had this feature"
-          required: false
-
   - type: markdown
     attributes:
       value: |
@@ -123,6 +120,5 @@ body:
         In the meantime please consider:  
           
           - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 📊 [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
           - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
           - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)

.github/ISSUE_TEMPLATE/3-documentation_change.yml

@@ -47,6 +47,5 @@ body:
         In the meantime please consider:  
           
           - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 👨‍✈️ [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
           - 🔍 [Checking out the new ReadTheDocs Documentation](https://docs.archivebox.io/)
           - ✨ Helping us test a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases)

.github/ISSUE_TEMPLATE/config.yml

@@ -6,6 +6,3 @@ contact_links:
   - name: 💬 Chat with the dev team & community on Zulip
     url: https://zulip.archivebox.io
     about: "Join us on our Zulip forum to chat with the developers and other users (it's similar to Discord but self-hosted)."
-  - name: 💁‍♂️ Hire us for professional support with fast response times
-    url: https://docs.monadical.com/s/archivebox-consulting-services
-    about: "We provide hosting, development, and support, including on-prem/cloud w/ SSO & storage, CAPTCHA-solving, proxies, etc."

.github/SECURITY.md

@@ -20,15 +20,9 @@ Please start by reading these links to understand our existing security model, k
 
 ## Reporting a Vulnerability
 
-We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.
+~We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.~ We no longer offer CVEs or accept private reporting, 95% of the reports we received were low quality AI slop, so now we just ask for normal Github Issues or PRs.
 
-1. Go to the Security tab on our Github repo: https://github.com/ArchiveBox/ArchiveBox/security
-
-2. Click the ["Report a Vulnerability"](https://github.com/ArchiveBox/ArchiveBox/security/advisories/new) button
-
-3. Fill out the form to submit the details of the report and it will be securely sent to the maintainers
-
-You can also contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp).
+You can contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) if you have a truly urgent security report, but we do not issue public CVEs.
 
 ---
 
@@ -39,7 +33,7 @@ You can also contact the maintainers via our public [Zulip Chat Server zulip.arc
 > Reporters **will still receive credit in release notes** however.
 
 > [!TIP]
-> Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.
+> ~Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.~
 
 Please read our existing published [security advisories](https://github.com/ArchiveBox/ArchiveBox/security/advisories), [Security Overview Docs](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview), and [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and avoid creating duplicates.
 
@@ -70,5 +64,5 @@ Non-logged in users should be tightly limited to prevent attacks like SQL/shell
 
 ### SSRF / URL Validation
 
-Note archivebox is **designed to be able to archive any URL the server can reach**, including internal intranet URLs, private IPs, and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` accordingly if they need to limit what the server should archive.
+Note archivebox is **designed to be able to archive any URL the server can reach**, including internal intranet URLs, private IPs, and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` or proper firewall/ACL-level blocking at the network level if they need to limit what the server should see.
 It does a basic check to try and avoid recursively archiving snapshots hosted by the archivebox server itself, but it's not a strict security check, it's just to prevent wasted CPU/disk from recursively archiving snapshots in a browser that has the ArchiveBox Browser Extension turned on.

.github/configs/ci-tooling.json

@@ -88,7 +88,7 @@
   "ci_binaries": [
     {
       "name": "{BASH_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{PYTHON_BINARY}",
@@ -96,7 +96,7 @@
     },
     {
       "name": "{JQ_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{UV_BINARY}",
@@ -122,7 +122,7 @@
     },
     {
       "name": "{GIT_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{PYTHON_BINARY}",
@@ -136,7 +136,7 @@
   "deploy_binaries": [
     {
       "name": "{GIT_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{PYTHON_BINARY}",
@@ -154,15 +154,15 @@
     },
     {
       "name": "{PV_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{TAR_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{TREE_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     }
   ],
   "docker_validation_binaries": [
@@ -170,6 +170,10 @@
       "name": "{DOCKER_BINARY}",
       "binproviders": "env"
     },
+    {
+      "name": "{SUDO_BINARY}",
+      "binproviders": "env"
+    },
     {
       "name": "{GREP_BINARY}",
       "binproviders": "env"
@@ -278,7 +282,7 @@
   "docs_binaries": [
     {
       "name": "{CURL_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{SUDO_BINARY}",
@@ -288,13 +292,13 @@
   "git_binaries": [
     {
       "name": "{GIT_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     }
   ],
   "github_binaries": [
     {
       "name": "{GH_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     }
   ],
   "node_binaries": [
@@ -323,19 +327,19 @@
     },
     {
       "name": "{GH_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{GIT_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{JQ_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     },
     {
       "name": "{CURL_BINARY}",
-      "binproviders": "env,apt,brew"
+      "binproviders": "env,brew,apt"
     }
   ]
 }

.github/workflows/ci.yml

@@ -2,9 +2,9 @@ name: CI
 
 on:
   pull_request:
-    branches: [dev]
+    branches: [dev, main]
   push:
-    branches: [dev]
+    branches: [dev, main]
   workflow_dispatch:
 
 permissions:
@@ -52,7 +52,7 @@ jobs:
     name: Tested Docker images
     uses: ./.github/workflows/docker.yml
     with:
-      push_digests: ${{ github.event_name == 'push' }}
+      push_digests: false
     secrets: inherit
 
   required:

.github/workflows/deploy-publicsite.yml

@@ -10,7 +10,6 @@ on:
       - bin/collect_ui_screenshots.*
       - bin/generate_ui_screenshot_gallery.py
       - bin/take_screenshot.js
-      - docs/Screenshots.md
       - pyproject.toml
       - uv.lock
       - .github/workflows/deploy-publicsite.yml
@@ -37,12 +36,6 @@ jobs:
         with:
           fetch-depth: 1
 
-      - name: Checkout screenshot plugin
-        uses: actions/checkout@v4
-        with:
-          repository: ArchiveBox/abx-plugins
-          path: .ui-screenshot-deps/abx-plugins
-
       - name: Setup Python and uv
         uses: astral-sh/setup-uv@v6
         with:
@@ -54,9 +47,24 @@ jobs:
       - name: Build UI screenshot gallery
         run: bin/collect_ui_screenshots.sh
         env:
-          ABX_PLUGINS_DIR: ${{ github.workspace }}/.ui-screenshot-deps/abx-plugins/abx_plugins/plugins
           BASE_URL: http://archivebox.localhost:8000
 
+      - name: Verify gallery was generated from this exact revision
+        run: |
+          uv run --no-cache --no-sync python - <<'PY'
+          import json
+          import os
+          import tomllib
+          from pathlib import Path
+
+          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
+          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
+          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
+          assert provenance["version"] == version, provenance
+          assert provenance["capture_count"] >= 3 * 35, provenance
+          assert len(provenance["files"]) == provenance["capture_count"], provenance
+          PY
+
       - name: Setup Pages
         uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
 

.github/workflows/docker.yml

@@ -8,6 +8,11 @@ on:
         required: false
         default: false
         type: boolean
+      full_tests:
+        description: Run storage, CLI, and size validation in addition to release smoke checks
+        required: false
+        default: true
+        type: boolean
 
 env:
   DOCKERHUB_IMAGE: archivebox/archivebox
@@ -28,12 +33,10 @@ jobs:
         include:
           - platform: linux/amd64
             runner: ubuntu-24.04
-            cache_scope: docker-amd64
             artifact_name: digest-linux-amd64
             local_tag: archivebox-ci:amd64
           - platform: linux/arm64
             runner: ubuntu-24.04-arm
-            cache_scope: docker-arm64
             artifact_name: digest-linux-arm64
             local_tag: archivebox-ci:arm64
     steps:
@@ -58,26 +61,27 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python - <<'PY'
+          ABXPKG_SPEC="$(uv run --no-cache --no-project python - <<'PY'
           import tomllib
           from pathlib import Path
 
           packages = tomllib.loads(Path("uv.lock").read_text())["package"]
-          matches = [package["version"] for package in packages if package["name"] == "abxpkg"]
+          matches = [package for package in packages if package["name"] == "abxpkg"]
           if len(matches) != 1:
               raise SystemExit(f"Expected one locked abxpkg package, found {len(matches)}")
-          print(matches[0])
+          wheel = matches[0]["wheels"][0]
+          print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))
           PY
           )"
-          test -n "$ABXPKG_VERSION"
+          test -n "$ABXPKG_SPEC"
           mkdir -p "$ABXPKG_LIB_DIR/env/bin"
           export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
           echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_VERSION=$ABXPKG_VERSION" >> "$GITHUB_ENV"
+          echo "ABXPKG_SPEC=$ABXPKG_SPEC" >> "$GITHUB_ENV"
           echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
 
           docker_env="$(
-            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+            uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
               --install \
               --json \
               --lib="$ABXPKG_LIB_DIR" \
@@ -202,32 +206,48 @@ jobs:
             ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
           pull: true
           platforms: ${{ matrix.platform }}
+          # Keep the released image as the one cache authority. A second GHA
+          # cache can hold gzip variants of identical layers, and BuildKit may
+          # prefer those over :dev's zstd blobs. That counter-intuitively makes
+          # a cache hit larger than the released image. :dev advances only
+          # after both architectures pass the release checks.
+          cache-from: type=registry,ref=${{ env.DOCKERHUB_IMAGE }}:dev
+          # Inline metadata reuses the released compressed blobs directly;
+          # force-compression would recompress the whole ~740 MiB image. The
+          # compressed-size gate below remains the regression guard.
+          cache-to: |
+            ${{ inputs.push_digests && 'type=inline' || '' }}
           outputs: |
-            type=docker,name=${{ matrix.local_tag }}
-            type=oci,dest=${{ runner.temp }}/archivebox-image.tar
-            type=image,"name=${{ env.DOCKERHUB_IMAGE }},${{ env.GHCR_IMAGE }}",push-by-digest=true,name-canonical=true,push=true
+            ${{ inputs.full_tests && format('type=oci,dest={0}/archivebox-image.tar,compression=zstd,compression-level=3', runner.temp) || '' }}
+            ${{ inputs.push_digests && format('type=image,"name={0},{1}",push-by-digest=true,name-canonical=true,push=true,compression=zstd,compression-level=3', env.DOCKERHUB_IMAGE, env.GHCR_IMAGE) || '' }}
 
       - name: Build pull request image
+        id: docker_build_pr
         if: ${{ !inputs.push_digests }}
         uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
         with:
           context: ./
           file: ./Dockerfile
           builder: ${{ steps.buildx.outputs.name }}
-          load: true
-          tags: ${{ matrix.local_tag }}
           labels: ${{ steps.docker_meta.outputs.labels }}
           build-args: |
             ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
             ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
           pull: true
           platforms: ${{ matrix.platform }}
+          # Test the same layer lineage users receive. Reusing :dev avoids a
+          # second cache whose compression can drift from the released image.
+          cache-from: type=registry,ref=${{ env.DOCKERHUB_IMAGE }}:dev
+          outputs: |
+            type=docker,name=${{ matrix.local_tag }}
+            type=oci,dest=${{ runner.temp }}/archivebox-image.tar
 
       - name: Image digest
         if: inputs.push_digests
         run: echo ${{ steps.docker_build.outputs.digest }}
 
       - name: Validate exact built image version and commit
+        if: ${{ !inputs.push_digests }}
         shell: bash
         run: |
           set -Eeuo pipefail
@@ -258,13 +278,35 @@ jobs:
               raise SystemExit(f"Missing exact commit marker: {sys.argv[3]}")
           PY
 
+          "$DOCKER_BINARY" run --rm --user 911:911 --platform '${{ matrix.platform }}' --entrypoint /venv/bin/python3 "$IMAGE" -c \
+            'import json, pathlib, stat; cache_file = "/opt/archivebox/lib/env/derived.env"; raw = open(cache_file).read().strip(); records = json.loads(raw[len("ABXPKG_DERIVED_CACHE=\x27"):-1]); projections = [projection for record in records.values() for projection in record.get("request_exec_projections", {}).values()]; plans = [projection["validation"] for projection in projections]; fingerprints = [fingerprint for plan in plans for fingerprint in plan["fingerprint"]]; actual = lambda fingerprint: (lambda path, value: {"path": str(path.resolve(strict=False)), "size": value.st_size, "mtime_ns": value.st_mtime_ns, "mode": stat.S_IMODE(value.st_mode), "euid": value.st_uid})(pathlib.Path(fingerprint["path"]), pathlib.Path(fingerprint["path"]).stat()); mismatches = [(fingerprint, actual(fingerprint)) for fingerprint in fingerprints if fingerprint != actual(fingerprint)]; assert projections and {plan.get("euid") for plan in plans} == {911}; assert all(not record.get("script_exec_plans") for record in records.values()); assert "ABXPKG_TMP_CACHE_DIR" not in json.dumps(projections); assert not mismatches, mismatches[:3]'
+
+      - name: Validate compressed candidate size
+        if: inputs.push_digests
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          DIGEST='${{ steps.docker_build.outputs.digest }}'
+          [[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
+          MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${DIGEST}")"
+          # Provenance wraps the real platform image and its attestation in an
+          # OCI index. Measure the image manifest, not the index/attestation.
+          IMAGE_DIGEST="$(MANIFEST="$MANIFEST" PLATFORM='${{ matrix.platform }}' "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); platform_os, platform_arch = os.environ["PLATFORM"].split("/")[:2]; print(next((item["digest"] for item in manifest.get("manifests", []) if item.get("platform", {}).get("os") == platform_os and item.get("platform", {}).get("architecture") == platform_arch), ""))')"
+          if [[ -n "$IMAGE_DIGEST" ]]; then
+            MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
+          fi
+          COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
+          LIMIT=$((750 * 1024 * 1024))
+          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
+          (( COMPRESSED_SIZE < LIMIT ))
+
       - name: Resolve real Docker CLI test dependencies through abxpkg
-        if: matrix.platform == 'linux/amd64'
+        if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         run: |
           set -Eeuo pipefail
           mount_env="$(
-            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+            uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
               --install \
               --json \
               --lib="$ABXPKG_LIB_DIR" \
@@ -281,7 +323,7 @@ jobs:
           done
 
       - name: Validate Docker UID and GID behavior on real NFS and SMB mounts
-        if: matrix.platform == 'linux/amd64'
+        if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         run: |
           set -Eeuo pipefail
@@ -315,7 +357,7 @@ jobs:
           "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
           "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+            "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
           "$SUDO_BINARY" "$BASH_BINARY" -c \
             "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
 
@@ -337,7 +379,7 @@ jobs:
             "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
 
       - name: Validate Docker CLI add/list and compose behavior
-        if: matrix.platform == 'linux/amd64'
+        if: inputs.full_tests && matrix.platform == 'linux/amd64'
         shell: bash
         env:
           ARCHIVEBOX_IMAGE: ${{ matrix.local_tag }}
@@ -359,29 +401,29 @@ jobs:
           "$DOCKER_BINARY" compose run --rm archivebox init
           "$DOCKER_BINARY" compose up -d --wait
           "$DOCKER_BINARY" network connect archivebox_default www-nginx
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000')"
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
           admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
           [[ "$index_html" == *ArchiveBox* ]]
           [[ "$admin_js" == *window.django* ]]
 
           "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000')"
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
           [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
           "$DOCKER_BINARY" compose down
 
       - name: Validate compressed image size
-        if: inputs.push_digests
+        if: inputs.full_tests && !inputs.push_digests
         env:
-          BUILD_METADATA: ${{ steps.docker_build.outputs.metadata }}
+          BUILD_METADATA: ${{ steps.docker_build_pr.outputs.metadata }}
         shell: bash
         run: |
           set -Eeuo pipefail
           LOCAL_IMAGE_ID="$("$DOCKER_BINARY" image inspect '${{ matrix.local_tag }}' --format '{{.Id}}')"
           "$UV_BINARY" run --no-cache --no-project python - \
             '${{ runner.temp }}/archivebox-image.tar' \
-            '${{ steps.docker_build.outputs.digest }}' \
+            '${{ steps.docker_build_pr.outputs.digest }}' \
             "$LOCAL_IMAGE_ID" \
-            "$((780 * 1024 * 1024))" <<'PY'
+            "$((750 * 1024 * 1024))" <<'PY'
           from hashlib import sha256
           import json
           import os

.github/workflows/docs.yml

@@ -67,6 +67,7 @@ jobs:
     timeout-minutes: 15
     strategy:
       fail-fast: false
+      max-parallel: 4
       matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
     steps:
       - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -98,9 +99,10 @@ jobs:
           if [[ '${{ matrix.environment }}' != macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries")
           fi
-          if [[ '${{ matrix.environment }}' == ubuntu ]]; then
+          if [[ '${{ matrix.name }}' == ubuntu-AGENTS || '${{ matrix.name }}' == ubuntu-skills-archivebox-SKILL || '${{ matrix.name }}' == ubuntu-docs-Configuration || '${{ matrix.name }}' == ubuntu-docs-Setting-up-Authentication ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
-          elif [[ '${{ matrix.environment }}' == docker ]]; then
+          fi
+          if [[ '${{ matrix.environment }}' == docker ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries")
           elif [[ '${{ matrix.environment }}' == macos ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries")

.github/workflows/duplicate-issue-detection.yml

@@ -1,96 +0,0 @@
-name: Duplicate Issue Detection
-
-on:
-  issues:
-    types: [opened]
-
-jobs:
-  check-duplicates:
-    runs-on: ubuntu-24.04
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-
-      - name: Resolve OpenCode and workflow tools through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          uv sync --no-cache --locked --no-sources --no-install-project
-          OPENCODE_CONFIG="$(uv run --no-cache --no-sync --no-sources python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "opencode" / "config.json")')"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
-          opencode_env="$(
-            uv run --no-cache --no-sync --no-sources abxpkg env \
-              --install \
-              --json \
-              --lib="$ABXPKG_LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:github_binaries" \
-              --deps-from="$OPENCODE_CONFIG:required_binaries"
-          )"
-          jq_binary="$ABXPKG_LIB_DIR/env/bin/jq"
-          "$jq_binary" -r 'to_entries[] | "\(.key)=\(.value)"' \
-            <<< "$opencode_env" >> "$GITHUB_ENV"
-
-          opencode_binary="$ABXPKG_LIB_DIR/env/bin/opencode"
-          echo "OPENCODE_BINARY=$opencode_binary" >> "$GITHUB_ENV"
-          test -L "$opencode_binary"
-          test -x "$opencode_binary"
-
-      - name: Check for duplicate issues
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-          OPENCODE_PERMISSION: |
-            {
-              "bash": {
-                "gh issue*": "allow",
-                "gh pr*": "allow",
-                "*": "deny"
-              },
-              "webfetch": "allow"
-            }
-        run: |
-          "$OPENCODE_BINARY" run -m anthropic/claude-haiku-4-5 "A new issue has been created:
-
-          Issue number:
-          ${{ github.event.issue.number }}
-
-          Lookup this issue and search through existing issues and PRs (excluding #${{ github.event.issue.number }}) in this repository to find any potential duplicates of this new issue.
-          Consider:
-          1. Similar titles or descriptions
-          2. Same error messages or symptoms
-          3. Related functionality or components
-          4. Similar feature requests
-
-          If you find any potential duplicates, please comment on the new issue with:
-          - A brief explanation of why it might be a duplicate
-          - Links to the potentially duplicate issues or PRs
-          - A suggestion to check those issues first
-
-          Use this format for the comment:
-          'This issue might be a duplicate of existing issues. Please check:
-          - #[issue_number]: [brief description of similarity]
-
-          Feel free to ignore if none of these address your specific case.'
-
-          If no clear duplicates are found, do not comment."

.github/workflows/pip.yml

@@ -2,6 +2,12 @@ name: Build Pip package
 
 on:
   workflow_call:
+    inputs:
+      full_tests:
+        description: Run the full package compatibility matrix instead of one release smoke check
+        required: false
+        default: true
+        type: boolean
 
 env:
   UV_VERSION: "0.11.3"
@@ -58,6 +64,21 @@ jobs:
           Path("dist/SHA256SUMS").write_text("\n".join(checksum_lines) + "\n")
           PY
 
+      - name: Release wheel import and CLI smoke
+        if: ${{ !inputs.full_tests }}
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          shopt -s nullglob
+          wheels=(dist/archivebox-*.whl)
+          [[ "${#wheels[@]}" -eq 1 ]]
+          uv sync --locked --no-cache --no-sources --no-dev --no-install-project
+          uv pip install --no-cache --no-deps --python .venv/bin/python "${wheels[0]}"
+          unset PYTHONPATH
+          cd "$(mktemp -d)"
+          VIRTUAL_ENV="$GITHUB_WORKSPACE/.venv" uv run --no-cache --active --no-project python -c 'import archivebox'
+          VIRTUAL_ENV="$GITHUB_WORKSPACE/.venv" uv run --no-cache --active --no-project archivebox version
+
       - name: Upload every distribution
         uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         with:
@@ -68,6 +89,7 @@ jobs:
 
   install-smoke:
     name: install ${{ matrix.artifact }} / Python ${{ matrix.python }} / ${{ matrix.os }}
+    if: inputs.full_tests
     needs: build
     runs-on: ${{ matrix.os }}
     strategy:
@@ -164,3 +186,58 @@ jobs:
           VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox version
           VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox init
           VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox status
+
+          if [[ "${{ runner.os }}" == "Linux" ]]; then
+            root_tool_dir="/root/archivebox-package-smoke-${{ matrix.artifact }}-${{ matrix.python }}"
+            uv_binary="$(command -v uv)"
+            sudo rm -rf "$root_tool_dir"
+            sudo mkdir -p "$root_tool_dir/data"
+
+            root_install_args=()
+            for dependency_artifact in "${dependency_artifacts[@]}"; do
+              root_install_args+=(--with "$dependency_artifact")
+            done
+            sudo UV_TOOL_DIR="$root_tool_dir/tools" UV_TOOL_BIN_DIR="$root_tool_dir/bin" \
+              "$uv_binary" tool install --no-cache --python "${{ matrix.python }}" \
+              "${root_install_args[@]}" "$artifact"
+
+            sudo useradd --system --create-home --home-dir /var/lib/archivebox --shell /bin/bash archivebox 2>/dev/null || true
+            sudo rm -rf /var/lib/archivebox
+            sudo env PATH="$root_tool_dir/bin:$PATH" \
+              bash -c "cd '$root_tool_dir/data' && archivebox init"
+            sudo stat -c %U /var/lib/archivebox | grep -qx archivebox
+            sudo stat -c %U "$root_tool_dir/data" | grep -qx archivebox
+            if sudo find "$root_tool_dir/data" -xdev -user root -print -quit | grep -q .; then
+              echo "Root-owned files remain in $root_tool_dir/data" >&2
+              exit 1
+            fi
+
+            sudo mkdir "$root_tool_dir/external"
+            sudo ln -s "$root_tool_dir/external" "$root_tool_dir/data/external-link"
+            sudo env PATH="$root_tool_dir/bin:$PATH" \
+              bash -c "cd '$root_tool_dir/data' && archivebox status"
+            sudo stat -c %U "$root_tool_dir/external" | grep -qx root
+
+            sudo mkdir "$root_tool_dir/data-universal-init"
+            sudo env PATH="$root_tool_dir/bin:$PATH" \
+              bash -c "cd '$root_tool_dir/data-universal-init' && archivebox status --init"
+            sudo stat -c %U "$root_tool_dir/data-universal-init" | grep -qx archivebox
+
+            sudo mkdir "$root_tool_dir/data-install"
+            sudo env PATH="$root_tool_dir/bin:$PATH" \
+              bash -c "cd '$root_tool_dir/data-install' && archivebox install --dry-run"
+            sudo stat -c %U "$root_tool_dir/data-install" | grep -qx archivebox
+
+            sudo useradd --create-home archivebox-smoke-user
+            smoke_user_group="$(id -gn archivebox-smoke-user)"
+            sudo mkdir "$root_tool_dir/data-mixed-owner"
+            sudo chown archivebox-smoke-user:root "$root_tool_dir/data-mixed-owner"
+            sudo env PATH="$root_tool_dir/bin:$PATH" \
+              bash -c "cd '$root_tool_dir/data-mixed-owner' && archivebox init"
+            sudo stat -c %U:%G "$root_tool_dir/data-mixed-owner" | grep -qx "archivebox-smoke-user:$smoke_user_group"
+            if sudo find "$root_tool_dir/data-mixed-owner" -xdev -user root -print -quit | grep -q .; then
+              echo "Root-owned files remain in $root_tool_dir/data-mixed-owner" >&2
+              exit 1
+            fi
+            sudo rm -rf "$root_tool_dir"
+          fi

.github/workflows/release-candidate.yml

@@ -0,0 +1,51 @@
+name: Release candidate
+
+on:
+  push:
+    branches: [dev]
+
+permissions:
+  actions: read
+  contents: write
+  packages: write
+
+concurrency:
+  group: release-candidate-dev
+  cancel-in-progress: true
+
+jobs:
+  prepare:
+    runs-on: ubuntu-24.04
+    outputs:
+      action: ${{ steps.version.outputs.action }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          submodules: true
+          token: ${{ secrets.RELEASE_GH_TOKEN }}
+
+      - id: version
+        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
+        with:
+          package: archivebox
+          branch: dev
+          tag-prefix: v
+          version-scheme: rc
+
+  python-artifacts:
+    needs: prepare
+    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
+    uses: ./.github/workflows/pip.yml
+    with:
+      full_tests: false
+    secrets: inherit
+
+  docker-digests:
+    needs: prepare
+    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
+    uses: ./.github/workflows/docker.yml
+    with:
+      push_digests: true
+      full_tests: false
+    secrets: inherit

.github/workflows/release.yml

@@ -2,15 +2,11 @@ name: Release
 
 on:
   workflow_run:
-    workflows: [CI]
+    workflows: [CI, Release candidate]
     types: [completed]
     branches: [dev, main]
 
-permissions:
-  actions: read
-  contents: write
-  id-token: write
-  packages: write
+permissions: {}
 
 concurrency:
   group: release-${{ github.repository }}-${{ github.event.workflow_run.head_branch }}
@@ -24,14 +20,45 @@ env:
   CI_RUN_ID: ${{ github.event.workflow_run.id }}
 
 jobs:
+  candidate:
+    if: >-
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.event == 'push' &&
+      ((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
+       (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
+    runs-on: ubuntu-24.04
+    outputs:
+      ready: ${{ steps.owner.outputs.ready }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          ref: ${{ env.RELEASE_SHA }}
+      - id: owner
+        shell: bash
+        run: |
+          if [[ "$RELEASE_BRANCH" == main ]]; then
+            echo 'ready=true' >> "$GITHUB_OUTPUT"
+            exit 0
+          fi
+          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
+          TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
+          [[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
+
   python-release:
     name: Publish tested Python artifacts
-    if: github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push'
+    needs: candidate
+    if: needs.candidate.outputs.ready == 'true'
     runs-on: ubuntu-24.04
     environment: pypi
+    permissions:
+      actions: read
+      contents: write
+      id-token: write
     outputs:
       version: ${{ steps.version.outputs.version }}
       tag: ${{ steps.version.outputs.tag }}
+      release_ready: ${{ steps.release-state.outputs.ready }}
     steps:
       - name: Checkout the tested commit
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -56,9 +83,9 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
           mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env --install \
+          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env --install \
             --lib="$ABXPKG_LIB_DIR" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" >/dev/null
           for name in uv gh git jq curl; do
@@ -117,11 +144,32 @@ jobs:
           RELEASE_DISTRIBUTIONS_DIR: ${{ runner.temp }}/python-distributions
         run: ./bin/release.sh
 
+      - name: Confirm this tested commit owns the release tag
+        id: release-state
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          TARGET=""
+          while read -r sha _; do
+            TARGET="$sha"
+          done < <("$GIT_BINARY" ls-remote origin "refs/tags/${{ steps.version.outputs.tag }}")
+          if [[ "$TARGET" == "$RELEASE_SHA" ]]; then
+            echo "ready=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "ready=false" >> "$GITHUB_OUTPUT"
+          fi
+
   docker-release:
     name: Publish tested multiarch image
-    if: github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push'
+    if: >-
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.event == 'push'
     needs: python-release
     runs-on: ubuntu-24.04
+    permissions:
+      actions: read
+      contents: read
+      packages: write
     steps:
       - name: Checkout the tested commit
         uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
@@ -144,9 +192,9 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
           mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env --install \
+          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env --install \
             --lib="$ABXPKG_LIB_DIR" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries" >/dev/null
@@ -246,6 +294,18 @@ jobs:
           for digest in "${DIGESTS[@]}"; do REFS+=("${GHCR_IMAGE}@sha256:${digest}"); done
           $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
 
+      - name: Verify published Docker images run
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION='${{ needs.python-release.outputs.version }}'
+          SHORT_SHA="${RELEASE_SHA::12}"
+          for image in "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}" "${GHCR_IMAGE}:sha-${SHORT_SHA}"; do
+            "$DOCKER_BINARY" buildx imagetools inspect "$image" >/dev/null
+            INSTALLED_VERSION="$("$DOCKER_BINARY" run --rm "$image" --version)"
+            test "$INSTALLED_VERSION" = "$VERSION"
+          done
+
       - name: Update Docker Hub README
         uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
         with:
@@ -253,13 +313,19 @@ jobs:
           password: ${{ secrets.DOCKER_PASSWORD }}
           repository: archivebox/archivebox
 
-  downstream-packages:
-    name: Verify downstream packages
+  cascade:
+    name: Cascade verified release
+    if: needs.python-release.outputs.release_ready == 'true'
     needs:
       - python-release
       - docker-release
-    uses: ./.github/workflows/update-homebrew-tap.yml
+    permissions:
+      actions: read
+      contents: read
+    uses: ArchiveBox/monorepo/.github/workflows/cascade-release.yml@main
     with:
-      release_tag: ${{ needs.python-release.outputs.tag }}
-      release_sha: ${{ github.event.workflow_run.head_sha }}
-    secrets: inherit
+      upstream_package: archivebox
+      upstream_version: ${{ needs.python-release.outputs.version }}
+      upstream_artifact_name: python-distributions
+    secrets:
+      release_token: ${{ secrets.RELEASE_GH_TOKEN }}

.github/workflows/test-parallel.yml

@@ -38,12 +38,12 @@ jobs:
           ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
         run: |
           set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
-          test -n "$ABXPKG_VERSION"
+          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
+          test -n "$ABXPKG_SPEC"
           mkdir -p "$ABXPKG_LIB_DIR/env/bin"
           export PYTHON_BINARY="$pythonLocation/bin/python"
           test -x "$PYTHON_BINARY"
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
             --install \
             --lib="$ABXPKG_LIB_DIR" \
             --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
@@ -106,8 +106,8 @@ jobs:
           TEST_PATHS: ${{ matrix.test.paths_arg }}
         run: |
           set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
-          test -n "$ABXPKG_VERSION"
+          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
+          test -n "$ABXPKG_SPEC"
           dependency_sources=()
           if [[ -n "$TEST_EXTRA" ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
@@ -115,7 +115,7 @@ jobs:
           if [[ "$TEST_PATHS" == *test_postgres_backend* ]]; then
             dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:postgres_binaries")
           fi
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
             --install \
             --no-cache \
             --json \

.github/workflows/test.yml

@@ -49,124 +49,8 @@ jobs:
           enable-cache: false
           cache-dependency-glob: uv.lock
 
-      - name: Prepare abxpkg environment
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
-          test -n "$ABXPKG_VERSION"
-          mkdir -p "$LIB_DIR/env/bin"
-          {
-            echo "ABXPKG_VERSION=$ABXPKG_VERSION"
-            echo "LIB_DIR=$LIB_DIR"
-            echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR"
-          } >> "$GITHUB_ENV"
-          echo "$LIB_DIR/env/bin" >> "$GITHUB_PATH"
-
-      - name: Resolve CI tools through abxpkg
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          export PATH="$LIB_DIR/env/bin:$PATH"
-          tooling_env="$(
-            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-              --install \
-              --json \
-              --lib="$LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
-          )"
-          JQ_BINARY="$LIB_DIR/env/bin/jq"
-          "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
-            <<< "$tooling_env" >> "$GITHUB_ENV"
-          echo "JQ_BINARY=$JQ_BINARY" >> "$GITHUB_ENV"
-          test -L "$JQ_BINARY"
-          test -x "$JQ_BINARY"
-
-      - name: Resolve Linux build dependencies through abxpkg
-        if: runner.os == 'Linux'
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-            --install \
-            --no-cache \
-            --json \
-            --lib="$LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries"
-
-      - name: Resolve host Homebrew through abxpkg
-        if: runner.os == 'macOS'
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          brew_env="$(
-            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-              --install \
-              --json \
-              --lib="$LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries"
-          )"
-          brew_binary="$("$JQ_BINARY" -r '.BREW_BINARY' <<< "$brew_env")"
-          test -L "$brew_binary"
-          echo "BREW_BINARY=$brew_binary" >> "$GITHUB_ENV"
-
-      - name: Resolve macOS build dependencies through abxpkg
-        if: runner.os == 'macOS'
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          brew_target="$(readlink "$BREW_BINARY")"
-          test -x "$brew_target"
-          brew_root="$(dirname "$(dirname "$brew_target")")"
-          export ABXPKG_BREW_ROOT="$brew_root"
-
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-            --install \
-            --json \
-            --lib="$LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-build.json:required_binaries"
-
-          PATH="$brew_root/opt/openldap/bin:$PATH" \
-            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-              --install \
-              --json \
-              --lib="$LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-openldap-env.json:required_binaries" \
-            | "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
-              >> "$GITHUB_ENV"
-
-          ldapvc_target="$(readlink "$LIB_DIR/env/bin/ldapvc")"
-          while [[ -L "$ldapvc_target" ]]; do
-            next_ldapvc_target="$(readlink "$ldapvc_target")"
-            if [[ "$next_ldapvc_target" == /* ]]; then
-              ldapvc_target="$next_ldapvc_target"
-            else
-              ldapvc_target="$(dirname "$ldapvc_target")/$next_ldapvc_target"
-            fi
-          done
-          test -x "$ldapvc_target"
-          openldap_prefix="$(dirname "$(dirname "$ldapvc_target")")"
-          test -f "$openldap_prefix/include/ldap.h"
-          test -f "$openldap_prefix/lib/libldap.dylib"
-          {
-            echo "ABXPKG_BREW_ROOT=$brew_root"
-            echo "CPPFLAGS=-I$openldap_prefix/include${CPPFLAGS:+ $CPPFLAGS}"
-            echo "LDFLAGS=-L$openldap_prefix/lib${LDFLAGS:+ $LDFLAGS}"
-            echo "PKG_CONFIG_PATH=$openldap_prefix/lib/pkgconfig${PKG_CONFIG_PATH:+:$PKG_CONFIG_PATH}"
-          } >> "$GITHUB_ENV"
-
       - name: Install dependencies with uv
-        run: uv sync --no-cache --locked --dev --all-extras
+        run: uv sync --no-cache --locked --dev
 
       - name: ArchiveBox full install check
         run: |

.github/workflows/update-homebrew-tap.yml

@@ -1,172 +0,0 @@
-name: Update downstream packages
-
-on:
-  workflow_call:
-    inputs:
-      release_tag:
-        required: true
-        type: string
-      release_sha:
-        required: true
-        type: string
-  workflow_dispatch:
-    inputs:
-      release_tag:
-        description: Exact ArchiveBox release tag
-        required: true
-        type: string
-      release_sha:
-        description: Full commit SHA owned by the release tag
-        required: true
-        type: string
-
-permissions:
-  actions: read
-  contents: read
-
-concurrency:
-  group: update-downstream-packages-${{ inputs.release_tag || github.ref_name }}
-  cancel-in-progress: false
-
-jobs:
-  dispatch:
-    runs-on: ubuntu-24.04
-    steps:
-      - name: Checkout release source
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          ref: ${{ inputs.release_sha || github.sha }}
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-
-      - name: Resolve release binaries through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_VERSION="$(uv run --no-cache --no-project python - <<'PY'
-          import tomllib
-          from pathlib import Path
-
-          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
-          matches = [package["version"] for package in packages if package["name"] == "abxpkg"]
-          if len(matches) != 1:
-              raise SystemExit(f"Expected one locked abxpkg package, found {len(matches)}")
-          print(matches[0])
-          PY
-          )"
-          test -n "$ABXPKG_VERSION"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-
-          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
-            --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
-            >/dev/null
-
-          for binary_name in uv git jq curl; do
-            binary="$ABXPKG_LIB_DIR/env/bin/$binary_name"
-            test -L "$binary"
-            test -x "$binary"
-          done
-          {
-            echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
-            echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
-            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
-            echo "CURL_BINARY=$ABXPKG_LIB_DIR/env/bin/curl"
-          } >> "$GITHUB_ENV"
-
-      - name: Verify the exact release and downstream builds
-        env:
-          DOWNSTREAM_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || secrets.HOMEBREW_TAP_TOKEN }}
-          INPUT_RELEASE_TAG: ${{ inputs.release_tag }}
-          INPUT_RELEASE_SHA: ${{ inputs.release_sha }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          [[ -n "$DOWNSTREAM_TOKEN" ]] || { echo "RELEASE_GH_TOKEN/HOMEBREW_TAP_TOKEN is not configured" >&2; exit 1; }
-
-          RELEASE_TAG="${INPUT_RELEASE_TAG:-${GITHUB_REF_NAME}}"
-          [[ "$RELEASE_TAG" == v* ]]
-          VERSION="${RELEASE_TAG#v}"
-
-          TAG_TARGET="$("$GIT_BINARY" ls-remote "https://github.com/${GITHUB_REPOSITORY}.git" "refs/tags/${RELEASE_TAG}^{}" | "$UV_BINARY" run --no-cache --no-project python -c 'import sys; print(next((line.split()[0] for line in sys.stdin if line.strip()), ""))')"
-          if [[ -z "$TAG_TARGET" ]]; then
-            TAG_TARGET="$("$GIT_BINARY" ls-remote "https://github.com/${GITHUB_REPOSITORY}.git" "refs/tags/${RELEASE_TAG}" | "$UV_BINARY" run --no-cache --no-project python -c 'import sys; print(next((line.split()[0] for line in sys.stdin if line.strip()), ""))')"
-          fi
-          RELEASE_SHA="${INPUT_RELEASE_SHA:-${TAG_TARGET}}"
-          [[ "$RELEASE_SHA" =~ ^[0-9a-f]{40}$ ]]
-          [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] || { echo "${RELEASE_TAG} points to ${TAG_TARGET}, not ${RELEASE_SHA}" >&2; exit 1; }
-
-          PAYLOAD_FILTER="{event_type: \"archivebox-dev-updated\", client_payload: {ref: \$ref, sha: \$sha, version: \$version}}"
-          PAYLOAD="$("$JQ_BINARY" -nc --arg ref "$RELEASE_TAG" --arg sha "$RELEASE_SHA" --arg version "$VERSION" "$PAYLOAD_FILTER")"
-
-          dispatch_and_wait() {
-            local repo="$1"
-            local workflow_name="$2"
-            local dispatched_at run_json status conclusion url
-
-            dispatched_at="$(date -u +"%Y-%m-%dT%H:%M:%SZ")"
-            "$CURL_BINARY" --fail-with-body \
-              -X POST \
-              -H "Accept: application/vnd.github+json" \
-              -H "Authorization: Bearer ${DOWNSTREAM_TOKEN}" \
-              -H "X-GitHub-Api-Version: 2022-11-28" \
-              "https://api.github.com/repos/ArchiveBox/${repo}/dispatches" \
-              --data "$PAYLOAD"
-
-            local deadline=$((SECONDS + 1800))
-            while (( SECONDS < deadline )); do
-              run_json="$("$CURL_BINARY" --fail-with-body \
-                -H "Accept: application/vnd.github+json" \
-                -H "Authorization: Bearer ${DOWNSTREAM_TOKEN}" \
-                -H "X-GitHub-Api-Version: 2022-11-28" \
-                "https://api.github.com/repos/ArchiveBox/${repo}/actions/runs?event=repository_dispatch&branch=main&per_page=20" \
-                | "$JQ_BINARY" -c \
-                  --arg workflow_name "$workflow_name" \
-                  --arg dispatched_at "$dispatched_at" \
-                  '.workflow_runs
-                    | map(select(.name == $workflow_name and .created_at >= $dispatched_at))
-                    | sort_by(.created_at)
-                    | reverse
-                    | .[0] // empty')"
-
-              if [[ -n "$run_json" ]]; then
-                status="$("$JQ_BINARY" -r '.status' <<< "$run_json")"
-                conclusion="$("$JQ_BINARY" -r '.conclusion // ""' <<< "$run_json")"
-                url="$("$JQ_BINARY" -r '.html_url' <<< "$run_json")"
-                echo "${repo}: ${status:-unknown} ${conclusion:-pending} ${url}"
-                if [[ "$status" == "completed" ]]; then
-                  [[ "$conclusion" == "success" ]] || {
-                    echo "${repo} downstream workflow failed: ${url}" >&2
-                    return 1
-                  }
-                  return 0
-                fi
-              else
-                echo "${repo}: waiting for ${workflow_name} repository_dispatch run"
-              fi
-              sleep 30
-            done
-
-            echo "${repo} downstream workflow did not complete within 30 minutes" >&2
-            return 1
-          }
-
-          for repo_and_workflow in \
-            "homebrew-archivebox|Update ArchiveBox dev formula" \
-            "debian-archivebox|Build Debian package"
-          do
-            dispatch_and_wait "${repo_and_workflow%%|*}" "${repo_and_workflow#*|}"
-          done

.pre-commit-config.yaml

@@ -64,4 +64,4 @@ repos:
         args: ["--maxkb=600"]
       - id: name-tests-test
         args: ["--pytest-test-first"]
-        exclude: ^archivebox/tests/(data/|fixtures\.py$|migrations_helpers\.py$)
+        exclude: ^archivebox/tests/(data/|fixtures\.py$|migrations_helpers\.py$|import_helpers\.py$)

AGENTS.md

@@ -15,6 +15,13 @@ ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev
 - Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
 - Read `README.md` for the full setup, CLI, Docker, API, and release surface.
 
+## Concurrency Contract
+
+- A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
+- SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
+- Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
+- PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.
+
 ## Development Setup
 
 ```bash
@@ -38,7 +45,7 @@ uv run --project .. archivebox run
 Recommended CLI install:
 
 ```bash
-uv tool install --python 3.13 --prerelease allow --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
+uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
 mkdir -p ~/archivebox/data
 cd ~/archivebox/data
 archivebox init --install

Dockerfile

@@ -10,10 +10,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.11.272 \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.274 \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:1.11.272
+ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.274
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -85,8 +85,8 @@ RUN cp /VERSION.txt /ABX-DL-VERSION.txt \
     && echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
     && uname -a \
     && sed -n '1,7p' /etc/os-release \
-    && abxpkg load --binproviders=env node \
-    && abxpkg load --binproviders=env uv \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env node \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env uv \
     ) | tee -a /VERSION.txt
 
 FROM archivebox-runtime-base AS archivebox-builder
@@ -102,8 +102,10 @@ echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
 rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
 apt-get install -qq -y --no-install-recommends \
-    build-essential gcc libldap2-dev libsasl2-dev libssl-dev
-/usr/bin/uv venv --no-cache --clear /venv --python "${PYTHON_VERSION}"
+    build-essential gcc libldap2-dev libsasl2-dev libssl-dev x11-utils
+# Extend abx-dl's existing venv instead of clearing it. Clearing and later
+# copying a complete replacement over the base duplicates every venv byte in
+# the final overlay history even when most packages are unchanged.
 /usr/bin/uv pip install --no-cache setuptools pip wheel
 
 mkdir -p /tmp/archivebox-uv-project
@@ -154,23 +156,30 @@ PY
     --no-install-project \
     --no-install-workspace \
     --no-sources
-ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$ABXPKG_LIB_DIR" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
-"$ABXPKG_LIB_DIR/env/bin/find" /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
+builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
+ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
+/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
 while IFS= read -r -d '' native_library; do
     magic=''
     if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
-        "$ABXPKG_LIB_DIR/env/bin/strip" --strip-unneeded "$native_library" || exit $?
+        "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
     fi
 done < /tmp/archivebox-native-libraries
 rm -f /tmp/archivebox-native-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$ABXPKG_LIB_DIR" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
-abxpkg run --binproviders=env --lib="$ABXPKG_LIB_DIR" apt-get autoremove -y
-"$ABXPKG_LIB_DIR/env/bin/find" "$ABXPKG_LIB_DIR/env/bin" -maxdepth 1 -type l -name strip -delete
-rm -rf /var/lib/apt/lists/*
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
+rm -rf "$builder_abxpkg_lib_dir"
+rm -rf /venv/lib/python3.*/site-packages/pip* \
+    /venv/lib/python3.*/site-packages/wheel* \
+    /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
+echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
+rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
 EOF
 
 COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
+# Compile only ArchiveBox's installed package. Touching all of /venv here would
+# copy every inherited abx-dl file into a new layer merely to change its mtime.
 RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && COMMIT_HASH="$( \
         if [[ "$ARCHIVEBOX_COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then \
@@ -187,9 +196,17 @@ RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && if [[ "$COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "COMMIT_HASH=$COMMIT_HASH" | tee -a /VERSION.txt; fi \
     && /usr/bin/uv pip install --no-cache --no-deps "$CODE_DIR" \
     && rm -f /venv/bin/uv /venv/bin/uvx \
+    && cd / \
+    && ARCHIVEBOX_PY_DIR="$(/venv/bin/python -c 'import pathlib, archivebox; print(pathlib.Path(archivebox.__file__).parent)')" \
+    && /venv/bin/python -m compileall --invalidation-mode checked-hash -q "$ARCHIVEBOX_PY_DIR" \
+    && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
-FROM archivebox-runtime-base
+# The builder installs and purges compilers in one layer, so its final
+# filesystem is already runtime-clean. Preserve that ancestry: starting again
+# from archivebox-runtime-base and COPYing /venv would bake a second full venv
+# over the inherited one instead of recording only ArchiveBox's package delta.
+FROM archivebox-builder
 
 LABEL name="archivebox" \
     maintainer="Nick Sweeting <dockerfile@archivebox.io>" \
@@ -205,10 +222,9 @@ LABEL name="archivebox" \
 COPY --from=sonic /usr/local/bin/sonic /usr/local/bin/sonic
 COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 
-COPY --from=archivebox-builder /venv /venv
-COPY --from=archivebox-builder /app /app
-COPY --from=archivebox-builder /VERSION.txt /VERSION.txt
-
+# The builder invokes abxpkg as root for temporary ELF stripping, which can
+# rewrite derived state ownership. Restore UID 911 before resolving binaries
+# as the runtime user; the old copy-based final stage hid this dependency.
 RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
     && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
     && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
@@ -220,11 +236,11 @@ RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..."
     && usermod --append --groups audio,video "$ARCHIVEBOX_USER" \
     && [[ "$(id -u "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_UID" ]] || usermod -u "$DEFAULT_ARCHIVEBOX_UID" "$ARCHIVEBOX_USER" \
     && [[ "$(id -g "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_GID" ]] || groupmod -g "$DEFAULT_ARCHIVEBOX_GID" "$ARCHIVEBOX_USER" \
-    && abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
     && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
     && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
     && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
     && openssl rand -hex 16 > /etc/machine-id \
     && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
     && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
@@ -246,14 +262,12 @@ RUN echo "[+] Initializing image collection..." \
 RUN chmod +x "$CODE_DIR"/bin/*.sh \
     && chmod g+w "$TMP_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
 
-RUN --mount=type=cache,target=/tmp/abxpkg-cache,sharing=locked,mode=1777 \
-    chmod 1777 /tmp/abxpkg-cache \
-    && for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
+RUN for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
     && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$CONFIG_DIR" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache ABXPKG_NO_CACHE=True ABXPKG_TMP_CACHE_DIR=/tmp/abxpkg-cache archivebox install \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache archivebox version 2>&1 | tee -a /VERSION.txt \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox install \
+    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox version 2>&1 | tee -a /VERSION.txt \
     && rm -rf /root/.cache /var/cache/apt/* /var/lib/apt/lists/*
 
 RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully." \

README.md

@@ -1,5 +1,5 @@
 <div align="center" style="text-align: center; width: 100%">
-<img src="https://archivebox.io/icon.png" height="90px"/>
+<img style="height: 90px" src="https://archivebox.io/icon.png"/>
 <h1>ArchiveBox<br/><sub>Open-source self-hosted web archiving.</sub></h1>
 
 <br/>
@@ -23,7 +23,7 @@ Without active preservation effort, everything on the internet eventually disapp
 *ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
 <br/>
 
-> ➡️ Get ArchiveBox with `uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
+> ➡️ Get ArchiveBox with `pip install archivebox>=0.9.0rc0'` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on Linux/macOS.
 
 *Once installed, you can interact with it through the: [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), [CLI](#usage), [self-hosted web interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python API](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [filesystem](#static-archive-exporting).*
 
@@ -70,23 +70,24 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
 <br/>
 <pre lang="bash"><code style="white-space: pre-line"># Option A: Get ArchiveBox with Docker Compose (recommended):
 mkdir -p ~/archivebox/data && cd ~/archivebox
-curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml   # edit options in this file as-needed
-docker compose run archivebox init
-# docker compose run archivebox add 'https://example.com'
-# docker compose run archivebox help
-# docker compose up
+curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
+docker compose pull
+docker compose up -d --wait                                                # initializes new collections automatically
+# open http://admin.archivebox.localhost:8000 to finish setup
+# docker compose run --rm archivebox add 'https://example.com'
+# docker compose run --rm archivebox help
 <br/>
 <br/>
 # Option B: Or use it as a plain Docker container:
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -it -v $PWD:/data archivebox/archivebox:dev init
+docker run -d --name archivebox -v "$PWD:/data" -p 8000:8000 archivebox/archivebox:dev
+# open http://admin.archivebox.localhost:8000 to finish setup
 # docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
 # docker run -it -v $PWD:/data archivebox/archivebox:dev help
-# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
 <br/>
 <br/>
 # Option C: Or install it with uv (see Quickstart below for apt, brew, and more)
-uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
+uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
 archivebox init
 archivebox install
@@ -95,19 +96,19 @@ archivebox install
 # archivebox server 0.0.0.0:8000
 <br/>
 <br/>
-# Option D: Or use the optional auto setup script to install it
+# Option D: Or use the uv install shortcut for Option C
 curl -fsSL 'https://get.archivebox.io' | bash
 </code></pre>
 <br/>
 <sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
-<sub>Set <code>BASE_URL</code> to change the public base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically. <code>BIND_ADDR</code> only controls the local listen address.</sub>
+<sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
 </details>
 <br/>
 
 
 <div align="center" style="text-align: center">
 <br/><br/>
-<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic"> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" height="75px" alt="logo" align="top"/> &nbsp; <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" height="70px" alt="bookshelf graphic">
+<img style="height: 70px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" alt="bookshelf graphic"> &nbsp; <img style="height: 75px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/b2765a33-0d1e-4019-a1db-920c7e00e20e" alt="logo" align="top"/> &nbsp; <img style="height: 70px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/5a7d95f2-6977-4de6-9f08-42851a1fe1d2" alt="bookshelf graphic">
 <br/><br/>
 <small><a href="https://demo.archivebox.io">Demo</a> | <a href="#screenshots">Screenshots</a> | <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage">Usage</a></small>
 <br/>
@@ -135,27 +136,6 @@ curl -fsSL 'https://get.archivebox.io' | bash
 
 <br/>
 
-## 🤝 Professional Integration
-
-ArchiveBox is free for everyone to self-host, but we also provide support, security review, and custom integrations to help NGOs, governments, and other organizations [run ArchiveBox professionally](https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102):
-
-- **Journalists:**
-  `crawling during research`, `preserving cited pages`, `fact-checking & review`  
-- **Lawyers:**
-  `collecting & preserving evidence`, `detecting changes`, `tagging & review`  
-- **Researchers:**
-  `analyzing social media trends`, `getting LLM training data`, `crawling pipelines`
-- **Individuals:**
-  `saving bookmarks`, `preserving portfolio content`, `legacy / memoirs archival`
-- **Governments:**
-  `snapshotting public service sites`, `recordkeeping compliance`
-
-> ***[Contact us](https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102)** if your org wants help using ArchiveBox professionally.*  
-> We offer: setup & support, CAPTCHA/ratelimit unblocking, SSO, audit logging/chain-of-custody, and more  
-> *ArchiveBox is a 🏛️ 501(c)(3) [nonprofit FSP](https://hackclub.com/hcb/) and all our work supports open-source development.* 
-
-<br/>
-
 <div align="center" style="text-align: center">
 <br/>
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0db52ea7-4a2c-441d-b47f-5553a5d8fe96" width="49%" alt="grass"/><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0db52ea7-4a2c-441d-b47f-5553a5d8fe96" width="49%" alt="grass"/>
@@ -165,14 +145,14 @@ ArchiveBox is free for everyone to self-host, but we also provide support, secur
 
 # Quickstart
 
-**🖥&nbsp; [Supported OSs](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#supported-systems):** Linux/BSD, macOS, Windows (Docker) &nbsp; **👾&nbsp; CPUs:** `amd64` (`x86_64`), `arm64`, `arm7` <sup>(raspi>=3)</sup><br/>
+**🖥&nbsp; [Supported OSs](https://github.com/ArchiveBox/ArchiveBox/wiki/Install#supported-systems):** Ubuntu, macOS, Docker &nbsp; **👾&nbsp; CPUs:** `amd64` (`x86_64`), `arm64`<br/>
 
 <br/>
 
 #### ✳️&nbsp; Easy Setup
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker-compose</code></b>  (macOS/Linux/Windows) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" align="top"/> <code>docker-compose</code></b>  (macOS/Linux) &nbsp; <b>👈&nbsp; recommended</b> &nbsp; <i>(click to expand)</i></summary>
 <br/>
 <i>👍 Docker Compose is recommended for the easiest install/update UX + best security + all <a href="#dependencies">extras</a> out-of-the-box.</i>
 <br/><br/>
@@ -180,18 +160,17 @@ ArchiveBox is free for everyone to self-host, but we also provide support, secur
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
 <li>Download the <a href="https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml" download><code>docker-compose.yml</code></a> file into a new empty directory (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox
-# Read and edit docker-compose.yml options as-needed after downloading
 curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
+docker compose pull
 </code></pre></li>
-<li>Run the initial setup to create an admin user (or set ADMIN_USER/PASS in docker-compose.yml)
-<pre lang="bash"><code style="white-space: pre-line">docker compose run archivebox init
+<li>Start the server, which initializes a new collection automatically.
+<pre lang="bash"><code style="white-space: pre-line">docker compose up -d --wait
 </code></pre></li>
-<li>Next steps: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">docker compose up
-# completely optional, CLI can always be used without running a server
-# docker compose run [-T] archivebox [subcommand] [--help]
-docker compose run archivebox add 'https://example.com'
-docker compose run archivebox help
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<pre lang="bash"><code style="white-space: pre-line">
+# run CLI commands inside the server container started above
+docker compose exec archivebox archivebox add 'https://example.com'
+docker compose exec archivebox archivebox help
 </code></pre>
 <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
 </li>
@@ -202,20 +181,20 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 </details>
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker run</code></b>  (macOS/Linux/Windows)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" align="top"/> <code>docker run</code></b>  (macOS/Linux)</summary>
 <br/>
 <ol>
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
-<li>Create a new empty directory and initialize your collection (can be anywhere).
+<li>Create a new empty directory and start the server, which initializes the collection automatically (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -v $PWD:/data -it archivebox/archivebox:dev init
+docker run -d --name archivebox -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
+<li>Open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the web setup wizard.
+<pre lang="bash"><code style="white-space: pre-line">
 # completely optional, CLI can always be used without running a server
-# docker run -v $PWD:/data -it [subcommand] [--help]
-docker run -v $PWD:/data -it archivebox/archivebox:dev help
+# docker exec archivebox archivebox [subcommand] [--help]
+docker exec archivebox archivebox help
 </code></pre>
 <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
 </li>
@@ -226,18 +205,17 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 </details>
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="28px" align="top"/> <code>bash</code> auto-setup script</b>  (macOS/Linux)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh uv install shortcut" align="top"/> <code>bash</code> uv install shortcut</b>  (macOS/Linux/BSD)</summary>
 <br/>
 <ol>
-<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (optional, highly recommended but not required).</li>
-<li>Run the automatic setup script.
+<li>Run the shortcut for the <code>uv</code> install method. It installs <code>uv</code> first when needed.
 <pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | bash</code></pre>
 <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-automatic-setup-script">Install: Bare Metal</a> in the Wiki. ➡️</i>
 </li>
 </ol>
 
 See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
-See <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/setup.sh"><code>setup.sh</code></a> for the source code of the auto-install script.<br/>
+See <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/bin/setup.sh"><code>setup.sh</code></a> for the source code of the <code>uv</code> install shortcut.<br/>
 See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as an install method"</a> blog post for my thoughts on the shortcomings of this install method.
 <br/><br/>
 </details>
@@ -250,26 +228,24 @@ See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as a
 
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" height="28px" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
 <br/>
 <ol>
 
 <li>Install <a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> on your system (if not already installed).</li>
 <li>Install the ArchiveBox package using <code>uv</code>.
-<pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
+<pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --prerelease explicit --upgrade 'archivebox>=0.9.0rc0,<0.10'
 archivebox version
-# install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions
-#    python@3.13 node curl wget git ripgrep ...
 </code></pre>
 <i>See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install">Install: Bare Metal</a> Wiki for full install instructions for each OS...</i>
 </li>
 <li>Create a new empty directory and initialize your collection (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data   # for example
-archivebox init     # instantialize a new collection
+archivebox init     # initialize a new collection
 archivebox install  # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
 </code></pre>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
 <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
@@ -286,7 +262,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117448075-49597580-af0c-11eb-91ba-f34fff10096b.png" alt="aptitude" height="28px" align="top"/> <code>apt</code></b> (Ubuntu/Debian/etc.)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117448075-49597580-af0c-11eb-91ba-f34fff10096b.png" alt="aptitude" align="top"/> <code>apt</code></b> (Ubuntu/Debian/etc.)</summary>
 <br/>
 <ol>
 <li>Add the third-party ArchiveBox apt repo and install <code>archivebox</code>.
@@ -297,16 +273,15 @@ sudo apt install archivebox
 </code></pre>
 </li>
 <li>Create a new empty directory and initialize your collection (can be anywhere).
-<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox
-sudo install -d -o archivebox -g archivebox ~/archivebox/data
+<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data
 cd ~/archivebox/data
 archivebox init
-archivebox install
+sudo archivebox install
 archivebox add 'https://example.com'
 </code></pre>
 <br/>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
 <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
@@ -315,20 +290,21 @@ archivebox help
 </li>
 </ol>
 See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
-<sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details, and run <code>sudo archivebox install</code> only if you want it to install missing system packages via apt.</sub>
+<sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. <code>sudo archivebox install</code> uses apt for missing system dependencies while preserving ownership of the user-owned collection. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details.</sub>
 <br/><br/>
 </details>
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" height="28px" align="top"/> <code>brew</code></b> (macOS only)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" align="top"/> <code>brew</code></b> (macOS and Linux)</summary>
 <br/>
+Run Homebrew as your normal non-root user on both macOS and Linux; do not use <code>sudo brew</code>.
 <ol>
 <li>Install <a href="https://brew.sh/#install">Homebrew</a> on your system (if not already installed).</li>
 <li>Install the ArchiveBox package using <code>brew</code>.
 <pre lang="bash"><code style="white-space: pre-line">brew tap archivebox/archivebox
 brew trust archivebox/archivebox
 brew install archivebox
-archivebox version                         # make sure all dependencies are installed
+archivebox version                         # verify the installed version
 </code></pre>
 <i>See the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-c-bare-metal-setup">Install: Bare Metal</a> Wiki for more granular instructions for macOS... ➡️</i>
 </li>
@@ -338,7 +314,7 @@ archivebox init
 archivebox install
 </code></pre>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<li>Optionally start the server, then open <code>/admin/</code> on the hostname or IP used to reach ArchiveBox (local example: <a href="http://admin.archivebox.localhost:8000/admin/">http://admin.archivebox.localhost:8000/admin/</a>) to create the first admin. If <code>BASE_URL</code> is not configured yet, continue through the same web setup wizard used by Docker installs.
 <pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
@@ -353,14 +329,14 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 </details>
 
 <details>
-<summary><img src="https://user-images.githubusercontent.com/511499/118077361-f0616580-b381-11eb-973c-ee894a3349fb.png" alt="Arch" height="28px" align="top"/> <code>pacman</code> / <img src="https://user-images.githubusercontent.com/511499/118077946-29e6a080-b383-11eb-94f0-d4871da08c3f.png" alt="FreeBSD" height="28px" align="top"/> <code>pkg</code> / <img src="https://user-images.githubusercontent.com/511499/118077861-002d7980-b383-11eb-86a7-5936fad9190f.png" alt="Nix" height="28px" align="top"/> <code>nix</code> (Arch/FreeBSD/NixOS/more)</summary>
+<summary><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/118077361-f0616580-b381-11eb-973c-ee894a3349fb.png" alt="Arch" align="top"/> <code>pacman</code> / <img style="height: 28px" src="https://user-images.githubusercontent.com/511499/118077946-29e6a080-b383-11eb-94f0-d4871da08c3f.png" alt="FreeBSD" align="top"/> <code>pkg</code> / <img style="height: 28px" src="https://user-images.githubusercontent.com/511499/118077861-002d7980-b383-11eb-86a7-5936fad9190f.png" alt="Nix" align="top"/> <code>nix</code> (Arch/FreeBSD/NixOS/more)</summary>
 <br/>
 
 > *Warning: These are contributed by external volunteers and may lag behind the official `uv` and Docker channels.*
 
 <ul>
 <li>Arch: <a href="https://aur.archlinux.org/packages/archivebox/"><code>yay -S archivebox</code></a> (contributed by <a href="https://github.com/imlonghao"><code>@imlonghao</code></a>, maintained by <a href="https://github.com/jasongodev"><code>@jasongodev</code></a>)</li>
-<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>pkg</code> + <code>uv</code> under-the-hood)</li>
+<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>uv</code>)</li>
 <li>Nix: <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/archivebox/default.nix"><code>nix-env --install archivebox</code></a> (contributed by <a href="https://github.com/siraben"><code>@siraben</code></a>)</li>
 <li>Guix: <a href="https://packages.guix.gnu.org/packages/archivebox/"><code>guix install archivebox</code></a> (contributed by <a href="https://github.com/rakino"><code>@rakino</code></a>)</li>
 <li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new"><i>contribute another distribution...!</i></a></li>
@@ -374,7 +350,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 #### 🎗&nbsp; Other Options
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="28px" align="top"/> <code>docker</code> + <img src="https://user-images.githubusercontent.com/511499/117447263-4316c980-af0b-11eb-928d-eaf1292ac646.png" alt="Electron" height="28px" align="top"/> <code>electron</code> Desktop App</b> (macOS/Linux/Windows)</summary>
+<summary><b><img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" align="top"/> <code>docker</code> + <img style="height: 28px" src="https://user-images.githubusercontent.com/511499/117447263-4316c980-af0b-11eb-928d-eaf1292ac646.png" alt="Electron" align="top"/> <code>electron</code> Desktop App</b> (macOS/Linux/Windows)</summary>
 <br/>
 <ol>
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
@@ -393,10 +369,10 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 </details>
 
 <details>
-<summary><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0c46e949-00fe-49c8-a613-ee14501c014c" alt="Self-hosting Platforms" height="28px" align="top"/><b> TrueNAS / UNRAID / YunoHost / Cloudron / etc.</b> (self-hosting solutions)</summary>
+<summary><img style="height: 28px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/0c46e949-00fe-49c8-a613-ee14501c014c" alt="Self-hosting Platforms" align="top"/><b> TrueNAS / UNRAID / YunoHost / Cloudron / etc.</b> (self-hosting solutions)</summary>
 <br/>
 
-> *Warning: These are contributed by external volunteers and may lag behind the official `pip` channel.*
+> *Warning: These are contributed by external volunteers and may lag behind the official Docker and `uv` channels.*
 
 <ul>
 <li><s>TrueNAS: <a href="https://truecharts.org/charts/stable/archivebox/">Official ArchiveBox TrueChart</a> / <a href="https://dev.to/finloop/setting-up-archivebox-on-truenas-scale-1788">Custom App Guide</a></s> (<a href="https://truecharts.org/news/scale-deprecation/">TrueCharts is discontinued</a>, wait for <a href="https://forums.truenas.com/t/the-future-of-electric-eel-and-apps/5409/">Electric Eel</a>)</li>
@@ -417,40 +393,31 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for usage examples using the CLI, W
 </details>
 
 <details>
-<summary><img src="https://user-images.githubusercontent.com/511499/117448723-1663b180-af0d-11eb-837f-d43959227810.png" alt="paid" height="27px" align="top"/> Paid hosting solutions (cloud VPS)</summary>
+<summary><img style="height: 27px" src="https://user-images.githubusercontent.com/511499/117448723-1663b180-af0d-11eb-837f-d43959227810.png" alt="paid" align="top"/> Third-party paid hosting options (cloud VPS)</summary>
 <br/>
 <ul>
-<li><a href="https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102">
- <img src="https://img.shields.io/badge/Custom_Development-ArchiveBox.io-%231a1a1a.svg?style=flat" height="22px"/>
-</a> (<a href="https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102">get hosting, support, and feature customization directly from us</a>)</li>
-<li><a href="https://monadical.com">
- <img src="https://img.shields.io/badge/General_Dev_Consulting-Monadical.com-%231a1a1a.svg?style=flat" height="22px"/>
-</a> (<a href="https://monadical.com/contact-us.html">generalist consultancy that has ArchiveBox experience</a>)</li>
-<br/>
-Other providers of paid ArchiveBox hosting (not officially endorsed):<br/>
-<br/><br/>
-<li><a href="https://elest.io/open-source/archivebox"><img src="https://img.shields.io/badge/Managed_Hosting-Elest.io-%23193f7e.svg?style=flat" height="22px"/></a></li>
-<li><a href="https://www.stellarhosted.com/archivebox/"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-StellarHosted.com-%23193f7e.svg?style=flat" height="22px"/></a> (USD $29-250/mo, <a href="https://www.stellarhosted.com/archivebox/#pricing">pricing</a>)</li>
-<li><a href="https://www.pikapods.com/pods?run=archivebox"><img src="https://img.shields.io/badge/Semi_Managed_Hosting-PikaPods.com-%2343a047.svg?style=flat" height="22px"/></a> (from USD $2.6/mo)</li>
+<li><a href="https://elest.io/open-source/archivebox"><img style="height: 22px" src="https://img.shields.io/badge/Managed_Hosting-Elest.io-%23193f7e.svg?style=flat"/></a></li>
+<li><a href="https://www.stellarhosted.com/archivebox/"><img style="height: 22px" src="https://img.shields.io/badge/Semi_Managed_Hosting-StellarHosted.com-%23193f7e.svg?style=flat"/></a> (USD $29-250/mo, <a href="https://www.stellarhosted.com/archivebox/#pricing">pricing</a>)</li>
+<li><a href="https://www.pikapods.com/pods?run=archivebox"><img style="height: 22px" src="https://img.shields.io/badge/Semi_Managed_Hosting-PikaPods.com-%2343a047.svg?style=flat"/></a> (from USD $2.6/mo)</li>
 <li><a href="https://m.do.co/c/cbc4c0c17840">
- <img src="https://img.shields.io/badge/Unmanaged_VPS-DigitalOcean.com-%232f7cf7.svg?style=flat" height="22px"/>
+ <img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_VPS-DigitalOcean.com-%232f7cf7.svg?style=flat"/>
 </a> (USD $5-50+/mo, <a href="https://m.do.co/c/cbc4c0c17840">🎗&nbsp; referral link</a>, <a href="https://www.digitalocean.com/community/tutorials/how-to-install-and-use-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
 <li><a href="https://www.vultr.com/?ref=7130289">
- <img src="https://img.shields.io/badge/Unmanaged_VPS-Vultr.com-%232337a8.svg?style=flat" height="22px"/>
+ <img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_VPS-Vultr.com-%232337a8.svg?style=flat"/>
 </a> (USD $2.5-50+/mo, <a href="https://www.vultr.com/?ref=7130289">🎗&nbsp; referral link</a>, <a href="https://www.vultr.com/docs/install-docker-compose-on-ubuntu-20-04">instructions</a>)</li>
 <li><a href="https://fly.io/">
- <img src="https://img.shields.io/badge/Unmanaged_App-Fly.io-%239a2de6.svg?style=flat" height="22px"/>
+ <img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_App-Fly.io-%239a2de6.svg?style=flat"/>
 </a> (USD $10-50+/mo, <a href="https://fly.io/docs/hands-on/start/">instructions</a>)</li>
 <li><a href="https://railway.app/template/2Vvhmy">
- <img src="https://img.shields.io/badge/Unmanaged_App-Railway-%23A11BE6.svg?style=flat" height="22px"/>
+ <img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_App-Railway-%23A11BE6.svg?style=flat"/>
 </a> (USD $0-5+/mo)</li>
-<li><a href="https://aws.amazon.com/marketplace/pp/Linnovate-Open-Source-Innovation-Support-For-Archi/B08RVW6MJ2"><img src="https://img.shields.io/badge/Unmanaged_VPS-AWS-%23ee8135.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
-<li><a href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/meanio.archivebox?ocid=gtmrewards_whatsnewblog_archivebox_vol118"><img src="https://img.shields.io/badge/Unmanaged_VPS-Azure-%237cb300.svg?style=flat" height="22px"/></a> (USD $60-200+/mo)</li>
+<li><a href="https://aws.amazon.com/marketplace/pp/Linnovate-Open-Source-Innovation-Support-For-Archi/B08RVW6MJ2"><img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_VPS-AWS-%23ee8135.svg?style=flat"/></a> (USD $60-200+/mo)</li>
+<li><a href="https://azuremarketplace.microsoft.com/en-us/marketplace/apps/meanio.archivebox?ocid=gtmrewards_whatsnewblog_archivebox_vol118"><img style="height: 22px" src="https://img.shields.io/badge/Unmanaged_VPS-Azure-%237cb300.svg?style=flat"/></a> (USD $60-200+/mo)</li>
 <br/>
 <sub><i>Referral links marked 🎗 provide $5-10 of free credit for new users and help pay for our <a href="https://demo.archivebox.io">demo server</a> hosting costs.</i></sub>
 </ul>
 
-For more discussion on managed and paid hosting options see here: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/531">Issue #531</a>.
+For more discussion on third-party hosting options see here: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/531">Issue #531</a>.
 
 </details>
 
@@ -459,7 +426,7 @@ For more discussion on managed and paid hosting options see here: <a href="https
 #### ➡️&nbsp; Next Steps
 
 - Import URLs from some of the supported [Input Formats](#input-formats) or view the supported [Output Formats](#output-formats)...
-- (Optional) Create a persona and import browser cookies to archive logged-in sites: `archivebox persona create --import=chrome personal`
+- (Optional) [Import browser cookies and settings into a persona](docs/Chromium-Install.md#import-an-existing-browser-profile).
 - Tweak your UI or archiving behavior [Configuration](#configuration), read about some of the [Caveats](#caveats), or [Troubleshoot](https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting)
 - Read about the [Dependencies](#dependencies) used for archiving, the [Upgrading Process](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives), or the [Archive Layout](#archive-layout) on disk...
 - Or check out our full [Documentation](#documentation) or [Community Wiki](#internet-archiving-ecosystem)...
@@ -481,17 +448,12 @@ cd ~/archivebox/data         # IMPORTANT: cd into the directory
 archivebox version
 archivebox help
 
-# equivalent: docker compose run archivebox [subcommand] [--help]
-docker compose run archivebox help
+# equivalent: docker compose run --rm archivebox [subcommand] [--help]
+docker compose run --rm archivebox help
 
 # equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
 docker run -it -v $PWD:/data archivebox/archivebox:dev help
 
-# optional: import your browser cookies into a persona for logged-in archiving
-archivebox persona create --import=chrome personal
-# supported: chrome/chromium/brave/edge (Chromium-based only)
-# use --profile to target a specific profile (e.g. Default, Profile 1)
-# re-running import merges/dedupes cookies.txt (by domain/path/name) but replaces chrome_user_data
 ```
 
 #### ArchiveBox Subcommands
@@ -503,7 +465,7 @@ archivebox persona create --import=chrome personal
 
 <br/>
 <details>
-<summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
+<summary><img style="height: 22px" src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
 <br/>
 <pre lang="bash"><code style="white-space: pre-line">
 # make sure you have uv-installed ArchiveBox and it's available in your $PATH first
@@ -522,17 +484,15 @@ archivebox help              # get list of archivebox subcommands that can be ru
 <br/>
 
 <details>
-<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> <b>CLI Usage Examples: Docker Compose</b></summary>
+<summary><img style="height: 22px" src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" align="top"/> <b>CLI Usage Examples: Docker Compose</b></summary>
 <br/>
 <pre lang="bash"><code style="white-space: pre-line">
 # make sure you have `docker-compose.yml` from the Quickstart instructions first
 <br/>
-# docker compose run archivebox [subcommand] [--help]
-docker compose run archivebox init
-docker compose run archivebox install
-docker compose run archivebox version
-docker compose run archivebox help
-docker compose run archivebox add 'https://example.com'
+# docker compose exec archivebox archivebox [subcommand] [--help]
+docker compose exec archivebox archivebox version
+docker compose exec archivebox archivebox help
+docker compose exec archivebox archivebox add 'https://example.com'
 # to start webserver: docker compose up
 </code></pre>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage">Usage: Docker Compose CLI</a> wiki. ➡️</i>
@@ -541,18 +501,15 @@ docker compose run archivebox add 'https://example.com'
 <br/>
 
 <details>
-<summary><img src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" height="22px" align="top"/> <b>CLI Usage Examples: Docker</b></summary>
+<summary><img style="height: 22px" src="https://user-images.githubusercontent.com/511499/117447182-29758200-af0b-11eb-97bd-58723fee62ab.png" alt="Docker" align="top"/> <b>CLI Usage Examples: Docker</b></summary>
 <br/>
 <pre lang="bash"><code style="white-space: pre-line">
-# make sure you create and cd into in a new empty directory first  
+# make sure the `archivebox` server container from the Quickstart is running first
 <br/>
-# docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
-docker run -v $PWD:/data -it archivebox/archivebox:dev init
-docker run -v $PWD:/data -it archivebox/archivebox:dev install
-docker run -v $PWD:/data -it archivebox/archivebox:dev version
-docker run -v $PWD:/data -it archivebox/archivebox:dev help
-docker run -v $PWD:/data -it archivebox/archivebox:dev add 'https://example.com'
-# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
+# docker exec archivebox archivebox [subcommand] [--help]
+docker exec archivebox archivebox version
+docker exec archivebox archivebox help
+docker exec archivebox archivebox add 'https://example.com'
 </code></pre>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️</i>
 </details>
@@ -575,21 +532,19 @@ find ./archive/users -path '*/snapshots/*/*/*/index.html'  # inspect snapshot da
 <details>
 <summary><b>🖥&nbsp; Web UI & API Usage</b></summary>
 <pre lang="bash"><code style="white-space: pre-line">
-# Start the server on bare metal (pip/apt/brew/etc):
-archivebox manage createsuperuser              # create a new admin user via CLI
+# Start the server on bare metal (uv/apt/brew):
 archivebox server 0.0.0.0:8000                 # start the server
 <br/>
 # Or with Docker Compose:
-nano docker-compose.yml                        # setup initial ADMIN_USERNAME & ADMIN_PASSWORD
 docker compose up                              # start the server
 <br/>
 # Or with a Docker container:
-docker run -v $PWD:/data -it archivebox/archivebox:dev archivebox manage createsuperuser
 docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 
-<sup>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sup><br/>
-<sup>Set <code>BASE_URL</code> to change the public base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically. <code>BIND_ADDR</code> only controls the local listen address.</sup>
+<sup>Open <a href="http://admin.archivebox.localhost:8000/admin/"><code>http://admin.archivebox.localhost:8000/admin/</code></a> to create the first admin and finish web setup. Use <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI. ➡️</sup><br/>
+<sup>Advanced: <code>archivebox manage createsuperuser</code> remains available for creating accounts from the CLI.</sup><br/>
+<sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
 <br/><br/>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage">Usage: Web UI</a> wiki. ➡️</i>
 <br/><br/>
@@ -600,7 +555,7 @@ archivebox config --set PUBLIC_ADD_VIEW=True   # allow guests to submit URLs
 archivebox config --set PERMISSIONS=public     # make newly added snapshots public
 archivebox config --set PUBLIC_INDEX=True      # allow guests to see list of all snapshots
 # or
-docker compose run archivebox config --set PERMISSIONS=public
+docker compose run --rm archivebox config --set PERMISSIONS=public
 
 # restart the server to apply any config changes
 </code></pre>
@@ -618,7 +573,7 @@ docker compose run archivebox config --set PERMISSIONS=public
 
 <pre lang="bash"><code style="white-space: pre-line">
 archivebox add --depth=1 'https://example.com'                     # add a URL with uv-installed archivebox on the host
-docker compose run archivebox add --depth=1 'https://example.com'                       # or w/ Docker Compose
+docker compose run --rm archivebox add --depth=1 'https://example.com'                  # or w/ Docker Compose
 docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com'  # or w/ Docker, all equivalent
 </code></pre>
 
@@ -658,19 +613,19 @@ docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://ex
 ##  Input Formats: How to pass URLs into ArchiveBox for saving
 
 
-- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e" height="28px"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b>  
+- <img style="height: 28px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/ff20d251-5347-4b85-ae9b-83037d0ac01e"/> <b>From the official <a href="https://github.com/ArchiveBox/archivebox-extension">ArchiveBox Browser Extension</a></b><br>
   <i>Provides realtime archiving of browsing history or selected pages from Chrome/Chromium/Firefox browsers.</i>
 
-- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8" height="22px"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files  
+- <img style="height: 22px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/64078483-21d7-4eb1-aa6e-9ad55afe45b8"/> From manual imports of URLs from RSS, JSON, CSV, TXT, SQL, HTML, Markdown, etc. files<br>
   <i>ArchiveBox supports injesting URLs in [any text-based format](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Import-a-list-of-URLs-from-a-text-file).</i>
 
-- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34" height="22px"/> From manually exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (in Netscape format)  
+- <img style="height: 22px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/32b494e6-4de1-4984-8d88-dc02f18e5c34"/> From manually exported [browser history](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) or [browser bookmarks](https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive) (in Netscape format)<br>
   <i>Instructions: <a href="https://support.google.com/chrome/answer/96816?hl=en">Chrome</a>, <a href="https://support.mozilla.org/en-US/kb/export-firefox-bookmarks-to-backup-or-transfer">Firefox</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/assets/511499/24ad068e-0fa6-41f4-a7ff-4c26fc91f71a">Safari</a>, <a href="https://support.microsoft.com/en-us/help/211089/how-to-import-and-export-the-internet-explorer-favorites-folder-to-a-32-bit-version-of-windows">IE</a>, <a href="https://help.opera.com/en/latest/features/#bookmarks:~:text=Click%20the%20import/-,export%20button,-on%20the%20bottom">Opera</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></i>
 
-- <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12" height="22px"/> From URLs visited through a [MITM Proxy](https://mitmproxy.org/) with [`archivebox-proxy`](https://github.com/ArchiveBox/archivebox-proxy)  
+- <img style="height: 22px" src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/4f7bd318-265c-4235-ad25-38be89946b12"/> From URLs visited through a [MITM Proxy](https://mitmproxy.org/) with [`archivebox-proxy`](https://github.com/ArchiveBox/archivebox-proxy)<br>
   <i>Provides [realtime archiving](https://github.com/ArchiveBox/ArchiveBox/issues/577) of all traffic from any device going through the proxy.</i>
 
-- <img src="https://getpocket.com/favicon.ico" height="22px"/> From bookmarking services or social media (e.g. Twitter bookmarks, Reddit saved posts, etc.)  
+- <img style="height: 22px" src="https://getpocket.com/favicon.ico"/> From bookmarking services or social media (e.g. Twitter bookmarks, Reddit saved posts, etc.)<br>
   <i>Instructions: <a href="https://getpocket.com/export">Pocket</a>, <a href="https://pinboard.in/export/">Pinboard</a>, <a href="https://www.instapaper.com/user">Instapaper</a>, <a href="https://shaarli.readthedocs.io/en/master/Usage/#importexport">Shaarli</a>, <a href="https://www.groovypost.com/howto/howto/export-delicious-bookmarks-xml/">Delicious</a>, <a href="https://github.com/csu/export-saved-reddit">Reddit Saved</a>, <a href="https://doc.wallabag.org/en/user/import/wallabagv2.html">Wallabag</a>, <a href="http://help.unmark.it/import-export">Unmark.it</a>, <a href="https://www.addictivetips.com/web/onetab-save-close-all-chrome-tabs-to-restore-export-or-import/">OneTab</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/issues/648">Firefox Sync</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Quickstart#2-get-your-list-of-urls-to-archive">and more...</a></i>
 
 
@@ -688,7 +643,7 @@ echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox a
 # if using Docker, add -i when piping stdin:
 # echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add
 # if using Docker Compose, add -T when piping stdin / stdout:
-# echo 'https://example.com' | docker compose run -T archivebox add
+# echo 'https://example.com' | docker compose run --rm -T archivebox add
 ```
 
 See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
@@ -714,7 +669,7 @@ It uses all available methods out-of-the-box, but you can disable extractors and
 <summary><i>Expand to see the full list of ways it saves each page...</i></summary>
 
 
-<code>data/archive/{Snapshot.id}/</code><br/>
+<code>data/archive/users/{username}/snapshots/{YYYYMMDD}/{domain}/{Snapshot.id}/</code><br/>
 <ul>
 <li><strong>Index:</strong> <code>index.html</code> &amp; <code>index.json</code> HTML and JSON index files containing metadata and details</li>
 <li><strong>Title</strong>, <strong>Favicon</strong>, <strong>Headers</strong> Response headers, site favicon, and parsed site title</li>
@@ -761,7 +716,7 @@ The configuration is documented here: **[Configuration Wiki](https://github.com/
 <summary><i>Expand to see the most common options to tweak...</i></summary>
 <pre lang="bash"><code style="white-space: pre-line">
 # e.g. archivebox config --set TIMEOUT=120
-# or   docker compose run archivebox config --set TIMEOUT=120
+# or   docker compose run --rm archivebox config --set TIMEOUT=120
 <br/>
 TIMEOUT=240                # default: 60    add more seconds on slower networks
 CHECK_SSL_VALIDITY=False   # default: True  False = allow saving URLs w/ bad SSL
@@ -881,7 +836,7 @@ The on-disk layout is optimized to be easy to browse by hand and durable long-te
                                 ...
 </code></pre>
 
-Each snapshot subfolder includes static metadata and plain extractor output files. ArchiveBox also maintains a backwards-compatible <code>data/archive/TIMESTAMP</code> symlink for each snapshot.
+Each snapshot subfolder includes static metadata and plain extractor output files. Current releases do not create top-level <code>data/archive/TIMESTAMP</code> projections; legacy timestamp directories are migrated into the user-scoped tree by <code>archivebox update --migrate-only</code>.
 
 <h4>Learn More</h4>
 <ul>
@@ -915,7 +870,7 @@ archivebox list --json --with-headers > index.json     # export to json blob
 archivebox list --csv=timestamp,url,title > index.csv  # export to csv spreadsheet
 
 # (if using Docker Compose, add the -T flag when piping)
-# docker compose run -T archivebox list --html 'https://example.com' > index.html
+# docker compose run --rm -T archivebox list --html 'https://example.com' > index.html
 </code></pre>
 
 The paths in the static exports are relative, make sure to keep them next to your `./archive` folder when backing them up or viewing them.
@@ -972,7 +927,7 @@ archivebox manage createsuperuser
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">Wiki: Chrome profiles and cookies</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
 </ul>
 
@@ -982,7 +937,7 @@ archivebox manage createsuperuser
 
 ### Security Risks of Viewing Archived JS
 
-Archived JavaScript is untrusted content. The default <code>SERVER_SECURITY_MODE=safe-subdomains-fullreplay</code> serves replay content on isolated snapshot subdomains so it cannot share the admin UI's cookies or origin. If your deployment cannot use wildcard <code>*.archivebox.localhost</code> subdomains, use <code>safe-onedomain-nojsreplay</code>, which keeps one origin but disables JavaScript replay. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) and [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239) for details.
+Archived JavaScript is untrusted content. The default <code>SERVER_SECURITY_MODE=auto</code> uses isolated subdomains with full replay on <code>*.localhost</code>, and a one-domain no-JS replay policy on ordinary public or LAN hostnames. Choose <code>safe-subdomains-fullreplay</code> only when wildcard DNS and TLS are configured. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) and [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239) for details.
 
 
 <br/>
@@ -990,7 +945,7 @@ Archived JavaScript is untrusted content. The default <code>SERVER_SECURITY_MODE
 <summary><i>Expand to see risks and mitigations...</i></summary>
 
 
-<pre lang="bash"><code style="white-space: pre-line"># Default: full replay on isolated snapshot subdomains
+<pre lang="bash"><code style="white-space: pre-line"># Explicit wildcard mode: full replay on isolated snapshot subdomains
 archivebox config --set SERVER_SECURITY_MODE=safe-subdomains-fullreplay
 
 # Alternative for deployments without wildcard subdomains: disable JS replay
@@ -1016,7 +971,7 @@ archivebox config --set SERVER_SECURITY_MODE=safe-onedomain-nojsreplay
 
 ### Working Around Sites that Block Archiving
 
-For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) actively block archiving or bots in general. There are a number of approaches to work around this, and we also provide <a href="https://docs.monadical.com/s/archivebox-consulting-services">consulting services</a> to help here.
+For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) actively block archiving or bots in general. There are a number of approaches to work around this.
 
 <br/>
 <details>
@@ -1026,7 +981,7 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
 
 <ul>
 <li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
-<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
+<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#import-an-existing-browser-profile">importing a dedicated browser profile into a persona</a></li>
 <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
 <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
 </ul>
@@ -1054,7 +1009,7 @@ Each re-archive creates a distinct Snapshot row for the same URL:
 archivebox add --no-only-new 'https://example.com'
 </code></pre>
 
-The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI performs the same explicit re-archive.
+The <img style="height: 24px" src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button"/> button in the Admin UI performs the same explicit re-archive.
 
 <h4>Learn More</h4>
 
@@ -1081,8 +1036,8 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 </li>
 <li><strong>Don't store large collections on older filesystems like EXT3/FAT</strong> as they may not be able to handle more than 50k directory entries in the <code>data/archive/</code> folder.
 </li>
-<li><strong>Try to keep the <code>data/index.sqlite3</code> file on local drive (not a network mount)</strong> or SSD for maximum performance, however the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
-<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, make sure the mounted data directory is writable by its intended owner and consider disabling <a href="https://github.com/ArchiveBox/ArchiveBox/issues/1304"><code>root_squash</code></a> on your fileshare server.
+<li><strong>Keep the <code>data/index.sqlite3</code> file on a reliable local drive (not a network mount)</strong> or SSD when using SQLite; the <code>data/archive/</code> folder can be on a network mount or slower HDD.</li>
+<li>If using Docker or NFS/SMB/FUSE for the <code>data/archive/</code> folder, configure the server-side UID/GID mapping or ACL so ArchiveBox's non-root user can create and remove files. <code>root_squash</code> is compatible when the selected non-root identity has write access; do not rely on container root to override server-side permissions.
 </li>
 </ul>
 
@@ -1092,7 +1047,7 @@ Because ArchiveBox is designed to ingest a large volume of URLs with multiple co
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#Disk-Layout">Wiki: Usage (Disk Layout)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#output-folder">Wiki: Security Overview (Output-Folder)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#large-archives">Wiki: Usage (Large Archives)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#do-not-run-as-root">Wiki: Security Overview (Do Not Run as Root)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#run-archivebox-as-an-unprivileged-user">Wiki: Security Overview (Privilege Dropping)</a></li>
 </ul>
 
 
@@ -1256,13 +1211,6 @@ ArchiveBox is neither the highest fidelity nor the simplest tool available for s
 
 <br/>
 
-**Need help building a custom archiving solution?**
-
-> ✨ **[Hire the team that built Archivebox](https://zulip.archivebox.io/#narrow/stream/167-enterprise/topic/welcome/near/1191102) to solve archiving for your org.** ([@ArchiveBoxApp](https://twitter.com/ArchiveBoxApp))
-
-<br/>
-
-
 <div align="center" style="text-align: center">
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/897f7a88-1265-4aab-b80c-b1640afaad1f" width="100%" alt="documentation graphic">
 </div>
@@ -1380,18 +1328,15 @@ archivebox server 0.0.0.0:8000
 
 ```bash
 # Optional: develop via docker by mounting the code dir into the container
-# if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
+# if you edit e.g. ./archivebox/core/models/snapshots.py on the docker host, runserver
 # inside the container will reload and pick up your changes
 ./bin/build_docker.sh dev
 
-docker run -it -v $PWD/data:/data archivebox/archivebox:dev init
-docker run -it -v $PWD/data:/data archivebox/archivebox:dev install
-
 # Run the development server w/ autoreloading (but no bg workers)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init --debug --reload 0.0.0.0:8000
 
 # Run the production server (with bg workers but no autoreloading)
-docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server
+docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --init
 
 # (remove the --reload flag and add the --nothreading flag when profiling with the django debug toolbar)
 # When using --reload, make sure any files you create can be read by the user in the Docker container, eg with 'chmod a+rX'.
@@ -1452,7 +1397,7 @@ services:
 
 # or with plain Docker:
 docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
-docker run -it -v $PWD:/data archivebox:dev init
+docker run -it -v $PWD:/data -p 8000:8000 archivebox:dev
 
 # or with uv:
 uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
@@ -1565,8 +1510,6 @@ Copy a similar plugin as a template to modify, then open a new PR to add it in t
 
 <br/>
 <div align="center" style="text-align: center">
-<b><a href="https://docs.sweeting.me/s/archivebox-consulting-services">🏛️ Contact us for professional support 💬</a></b><br/>
-<br/><br/>
 <a href="https://hcb.hackclub.com/donations/start/archivebox"><img src="https://img.shields.io/badge/Donate-Directly-%13DE5D26.svg"/></a> &nbsp;
 <a href="https://github.com/sponsors/pirate"><img src="https://img.shields.io/badge/Github_Sponsors-%23B7CDFE.svg"/></a> &nbsp;
 <a href="https://www.patreon.com/theSquashSH"><img src="https://img.shields.io/badge/Patreon-%23DD5D76.svg"/></a> &nbsp;

archivebox/api/admin.py

@@ -8,7 +8,7 @@ from signal_webhooks.admin import WebhookAdmin, WebhookModelForm
 from signal_webhooks.settings import webhook_settings
 from signal_webhooks.utils import get_webhook_model, model_from_reference
 
-from archivebox.base_models.admin import BaseModelAdmin
+from archivebox.base_models.admin import card_fieldset, BaseModelAdmin
 
 from archivebox.api.models import APIToken
 
@@ -25,27 +25,9 @@ class APITokenAdmin(BaseModelAdmin):
     search_fields = ("id", "created_by__username", "token")
 
     fieldsets = (
-        (
-            "Token",
-            {
-                "fields": ("token", "expires"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Owner",
-            {
-                "fields": ("created_by",),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Timestamps",
-            {
-                "fields": ("created_at", "modified_at"),
-                "classes": ("card",),
-            },
-        ),
+        card_fieldset("Token", ("token", "expires")),
+        card_fieldset("Owner", ("created_by",)),
+        card_fieldset("Timestamps", ("created_at", "modified_at")),
     )
 
     list_filter = ("created_by",)
@@ -73,41 +55,11 @@ class CustomWebhookAdmin(WebhookAdmin, BaseModelAdmin):
     readonly_fields = _webhook_fields("created_at", "modified_at", *WebhookAdmin.readonly_fields)
 
     fieldsets = (
-        (
-            "Webhook",
-            {
-                "fields": _webhook_fields("name", "signal", "ref", "endpoint", "headers", "keep_last_response"),
-                "classes": ("card", "wide"),
-            },
-        ),
-        (
-            "Authentication",
-            {
-                "fields": _webhook_fields("auth_token"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Status",
-            {
-                "fields": _webhook_fields("enabled", "last_success", "last_failure", "last_response"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Owner",
-            {
-                "fields": _webhook_fields("created_by"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Timestamps",
-            {
-                "fields": _webhook_fields("created_at", "modified_at"),
-                "classes": ("card",),
-            },
-        ),
+        card_fieldset("Webhook", _webhook_fields("name", "signal", "ref", "endpoint", "headers", "keep_last_response"), wide=True),
+        card_fieldset("Authentication", _webhook_fields("auth_token")),
+        card_fieldset("Status", _webhook_fields("enabled", "last_success", "last_failure", "last_response")),
+        card_fieldset("Owner", _webhook_fields("created_by")),
+        card_fieldset("Timestamps", _webhook_fields("created_at", "modified_at")),
     )
 
     def lookup_allowed(self, lookup: str, value: str, request: HttpRequest | None = None) -> bool:

archivebox/api/schemas.py

@@ -0,0 +1,17 @@
+"""Fields shared by responses for user-owned model objects."""
+
+from ninja import Schema
+
+
+class OwnedObjectSchema(Schema):
+    created_by_id: str
+    created_by_username: str
+
+    @staticmethod
+    def resolve_created_by_id(obj):
+        return str(obj.created_by.pk)
+
+    @staticmethod
+    def resolve_created_by_username(obj) -> str:
+        user = obj.created_by
+        return user.username if isinstance(user.username, str) else str(user)

archivebox/api/v1_cli.py

@@ -117,38 +117,48 @@ def snapshot_filter_kwargs(args: SnapshotFilterCommandSchema, *, default_filter_
     return kwargs
 
 
+def cli_response(request, result, *, result_format="json", success=True, errors=()):
+    """Use one response envelope for CLI-backed endpoints, including captured output."""
+    payload = {"success": success, "errors": list(errors), "result": result, "result_format": result_format}
+    for stream_name in ("stdout", "stderr"):
+        stream = request.__dict__.get(stream_name)
+        payload[stream_name] = ansi_to_html(stream.getvalue().strip()) if isinstance(stream, StringIO) else ""
+    return payload
+
+
+def command_config(args) -> dict[str, object] | None:
+    """Translate shared legacy add/schedule flags into runtime config overrides."""
+    if args.update or args.overwrite:
+        return {"ONLY_NEW": False}
+    return {"ONLY_NEW": bool(args.only_new)} if args.only_new is not None else None
+
+
 @router.post("/add", response=CLICommandResponseSchema, summary="archivebox add [args] [urls]")
 def cli_add(request: HttpRequest, args: AddCommandSchema):
     from archivebox.cli.archivebox_add import add
     from archivebox.misc.util import validate_url
 
-    config_overrides: dict[str, object] = {}
-    if args.only_new is not None:
-        config_overrides["ONLY_NEW"] = bool(args.only_new)
-    if args.update or args.overwrite:
-        config_overrides["ONLY_NEW"] = False
     submitted_urls: str | list[str] = args.urls
     if len(args.urls) == 1:
         try:
             validate_url(args.urls[0])
         except ValueError:
             submitted_urls = args.urls[0]
+    else:
+        submitted_urls = []
+        for url in args.urls:
+            try:
+                submitted_urls.append(validate_url(url))
+            except ValueError:
+                continue
+        if not submitted_urls:
+            raise HttpError(400, "No valid URLs were submitted")
     crawl, snapshots = add(
         urls=submitted_urls,
-        snapshot_ids=args.snapshot_ids,
-        tag=args.tag,
-        depth=args.depth,
-        max_urls=args.max_urls,
-        crawl_max_size=args.crawl_max_size,
-        crawl_timeout=args.crawl_timeout,
-        snapshot_max_size=args.snapshot_max_size,
-        index_only=args.index_only,
-        plugins=args.plugins,
-        persona=args.persona,
-        parser=args.parser,
+        **args.model_dump(exclude={"urls", "only_new", "update", "overwrite"}),
         bg=True,  # Always run in background for API calls
         created_by_id=request.user.pk,
-        config=config_overrides or None,
+        config=command_config(args),
     )
 
     snapshot_ids = [str(snapshot_id) for snapshot_id in snapshots.values_list("id", flat=True)]
@@ -156,19 +166,9 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
         "crawl_id": str(crawl.id),
         "num_snapshots": len(snapshot_ids),
         "snapshot_ids": snapshot_ids,
-        "queued_urls": args.urls,
-    }
-    stdout = request.__dict__.get("stdout")
-    stderr = request.__dict__.get("stderr")
-
-    return {
-        "success": True,
-        "errors": [],
-        "result": result_payload,
-        "result_format": "json",
-        "stdout": ansi_to_html(stdout.getvalue().strip()) if isinstance(stdout, StringIO) else "",
-        "stderr": ansi_to_html(stderr.getvalue().strip()) if isinstance(stderr, StringIO) else "",
+        "queued_urls": args.urls if isinstance(submitted_urls, str) else submitted_urls,
     }
+    return cli_response(request, result_payload, result_format="json")
 
 
 @router.post("/update", response=CLICommandResponseSchema, summary="archivebox update [args] [filter_patterns]")
@@ -195,55 +195,20 @@ def cli_update(request: HttpRequest, args: UpdateCommandSchema):
         ]
 
     update(**update_kwargs)
-    stdout = request.__dict__.get("stdout")
-    stderr = request.__dict__.get("stderr")
-    return {
-        "success": True,
-        "errors": [],
-        "result": {
-            "matched_count": len(matched_snapshot_ids),
-            "snapshot_ids": matched_snapshot_ids,
-        }
-        if is_filtered_update
-        else None,
-        "stdout": ansi_to_html(stdout.getvalue().strip()) if isinstance(stdout, StringIO) else "",
-        "stderr": ansi_to_html(stderr.getvalue().strip()) if isinstance(stderr, StringIO) else "",
-    }
+    return cli_response(
+        request,
+        {"matched_count": len(matched_snapshot_ids), "snapshot_ids": matched_snapshot_ids} if is_filtered_update else None,
+        result_format="str",
+    )
 
 
 @router.post("/schedule", response=CLICommandResponseSchema, summary="archivebox schedule [args] [import_path]")
 def cli_schedule(request: HttpRequest, args: ScheduleCommandSchema):
     from archivebox.cli.archivebox_schedule import schedule
 
-    config_overrides: dict[str, object] = {}
-    if args.only_new is not None:
-        config_overrides["ONLY_NEW"] = bool(args.only_new)
-    if args.update or args.overwrite:
-        config_overrides["ONLY_NEW"] = False
-    result = schedule(
-        import_path=args.import_path,
-        add=args.add,
-        show=args.show,
-        foreground=args.foreground,
-        run_all=args.run_all,
-        quiet=args.quiet,
-        clear=args.clear,
-        every=args.every,
-        tag=args.tag,
-        depth=args.depth,
-        config=config_overrides or None,
-    )
+    result = schedule(**args.model_dump(exclude={"only_new", "update", "overwrite"}), config=command_config(args))
 
-    stdout = request.__dict__.get("stdout")
-    stderr = request.__dict__.get("stderr")
-    return {
-        "success": True,
-        "errors": [],
-        "result": result,
-        "result_format": "json",
-        "stdout": ansi_to_html(stdout.getvalue().strip()) if isinstance(stdout, StringIO) else "",
-        "stderr": ansi_to_html(stderr.getvalue().strip()) if isinstance(stderr, StringIO) else "",
-    }
+    return cli_response(request, result, result_format="json")
 
 
 @router.post("/search", response=CLICommandResponseSchema, summary="archivebox search [args] [filter_patterns]")
@@ -278,16 +243,7 @@ def cli_search(request: HttpRequest, args: ListCommandSchema):
     else:
         result = "\n".join(snapshot.url for snapshot in snapshots.iterator(chunk_size=500))
 
-    stdout = request.__dict__.get("stdout")
-    stderr = request.__dict__.get("stderr")
-    return {
-        "success": True,
-        "errors": [],
-        "result": result,
-        "result_format": result_format,
-        "stdout": ansi_to_html(stdout.getvalue().strip()) if isinstance(stdout, StringIO) else "",
-        "stderr": ansi_to_html(stderr.getvalue().strip()) if isinstance(stderr, StringIO) else "",
-    }
+    return cli_response(request, result, result_format=result_format)
 
 
 @router.post("/remove", response=CLICommandResponseSchema, summary="archivebox remove [args] [filter_patterns]")
@@ -308,13 +264,10 @@ def cli_remove(request: HttpRequest, args: RemoveCommandSchema):
         snapshots=snapshots_to_remove,
         timeout=timeout,
     )
-    stdout = request.__dict__.get("stdout")
-    stderr = request.__dict__.get("stderr")
-    return {
-        "success": bool(result["success"]),
-        "errors": [str(result["error"])] if result["error"] else [],
-        "result": result,
-        "result_format": "json",
-        "stdout": ansi_to_html(stdout.getvalue().strip()) if isinstance(stdout, StringIO) else "",
-        "stderr": ansi_to_html(stderr.getvalue().strip()) if isinstance(stderr, StringIO) else "",
-    }
+    return cli_response(
+        request,
+        result,
+        result_format="json",
+        success=bool(result["success"]),
+        errors=[str(result["error"])] if result["error"] else [],
+    )

archivebox/api/v1_core/__init__.py

@@ -0,0 +1,47 @@
+"""Core REST routes, composed from entity-specific routers."""
+
+from typing import Union
+
+from django.db.models import Model
+from django.http import HttpRequest
+from django.shortcuts import redirect
+from ninja import Router
+from ninja.errors import HttpError
+
+from archivebox.api.v1_crawls import CrawlSchema, get_crawl
+
+from . import archiveresults, snapshots, tags
+from .archiveresults import get_archiveresult
+from .pagination import CustomPagination as CustomPagination
+from .schemas import ArchiveResultSchema, SnapshotSchema, TagSchema
+from .snapshots import get_snapshot
+from .tags import get_tag
+
+router = Router(tags=["Core Models"])
+router.add_router("", archiveresults.router)
+router.add_router("", snapshots.router)
+router.add_router("", tags.router)
+
+
+@router.get(
+    "/any/{id}",
+    response=Union[SnapshotSchema, ArchiveResultSchema, TagSchema, CrawlSchema],
+    url_name="get_any",
+    summary="Get any object by its ID",
+)
+def get_any(request: HttpRequest, id: str):
+    """Get any object by its ID (e.g. snapshot, archiveresult, tag, crawl, etc.)."""
+    setattr(request, "with_snapshots", False)
+    setattr(request, "with_archiveresults", False)
+
+    for getter in (get_snapshot, get_archiveresult, get_tag, get_crawl):
+        try:
+            response = getter(request, id)
+            if isinstance(response, Model):
+                return redirect(
+                    f"/api/v1/{response._meta.app_label}/{response._meta.model_name}/{response.pk}?{request.META['QUERY_STRING']}",
+                )
+        except Exception:
+            pass
+
+    raise HttpError(404, "Object with given ID not found")

archivebox/api/v1_core/archiveresults.py

@@ -0,0 +1,310 @@
+import json
+import mimetypes
+import re
+from pathlib import Path, PurePosixPath
+from typing import Any
+
+from django.core.files.storage import FileSystemStorage
+from django.http import HttpRequest
+from django.http.multipartparser import MultiPartParser, MultiPartParserError
+from ninja import Form, Query, Router, UploadedFile
+from ninja.errors import HttpError
+from ninja.pagination import paginate
+
+from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.misc.db import uuid_ref_query as _uuid_ref_query
+
+from .lookup import _get_snapshot_by_ref
+from .pagination import CustomPagination
+from .schemas import ArchiveResultFilterSchema, ArchiveResultSchema
+
+router = Router(tags=["Core Models"])
+
+ARCHIVERESULT_UPLOAD_HOOK_NAME = Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME
+ARCHIVERESULT_UPLOAD_PLUGIN_RE = re.compile(r"^[A-Za-z0-9_.-]{1,32}$")
+
+
+@router.get("/archiveresults", response=list[ArchiveResultSchema], url_name="get_archiveresult")
+@paginate(CustomPagination)
+def get_archiveresults(request: HttpRequest, filters: Query[ArchiveResultFilterSchema]):
+    """List all ArchiveResult entries matching these filters."""
+    queryset = filters.filter(ArchiveResult.objects.all())
+    if filters.search or filters.snapshot_tag:
+        return queryset.distinct()
+    return queryset
+
+
+@router.get("/archiveresult/{archiveresult_id}", response=ArchiveResultSchema, url_name="get_archiveresult")
+def get_archiveresult(request: HttpRequest, archiveresult_id: str):
+    """Get a specific ArchiveResult by id."""
+    return ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
+
+
+def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
+    normalized = str(plugin or "").strip().strip("/")
+    if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
+        raise HttpError(400, "Invalid ArchiveResult plugin name")
+    return normalized
+
+
+def _normalize_uploaded_archiveresult_output_path(output_path: str, *, filename: str) -> str:
+    raw_path = str(output_path or filename or "").strip().replace("\\", "/")
+    if not raw_path:
+        raise HttpError(400, "ArchiveResult output path is required")
+
+    path = PurePosixPath(raw_path)
+    if path.is_absolute() or any(part in ("", ".", "..") for part in path.parts):
+        raise HttpError(400, "Invalid ArchiveResult output path")
+
+    return str(path)
+
+
+def _parse_archiveresult_output_json(output_json: str | None) -> dict[str, Any] | None:
+    if not output_json:
+        return None
+    try:
+        parsed = json.loads(output_json)
+    except json.JSONDecodeError as err:
+        raise HttpError(400, "ArchiveResult output_json must be valid JSON") from err
+    if parsed is None:
+        return None
+    if not isinstance(parsed, dict):
+        raise HttpError(400, "ArchiveResult output_json must be a JSON object")
+    return parsed
+
+
+def _get_archiveresult_upload_data(request: HttpRequest):
+    cached = request.__dict__.get("_archiveresult_upload_data")
+    if cached is not None:
+        return cached
+
+    if request.method.upper() == "PATCH" and request.content_type.startswith("multipart/"):
+        try:
+            data = MultiPartParser(request.META, request, request.upload_handlers, request.encoding).parse()
+        except MultiPartParserError as err:
+            raise HttpError(400, f"Invalid ArchiveResult multipart upload: {err}") from err
+    else:
+        data = (request.POST, request.FILES)
+
+    setattr(request, "_archiveresult_upload_data", data)
+    return data
+
+
+def _get_archiveresult_upload_files(request: HttpRequest, *, allow_empty: bool = False) -> list[UploadedFile]:
+    _post, request_files = _get_archiveresult_upload_data(request)
+    files = [*request_files.getlist("files"), *request_files.getlist("file")]
+    if not files and not allow_empty:
+        raise HttpError(400, "At least one ArchiveResult file is required")
+    return files
+
+
+def _get_archiveresult_upload_form_values(request: HttpRequest, *field_names: str) -> list[str]:
+    request_post, _files = _get_archiveresult_upload_data(request)
+    values: list[str] = []
+    for field_name in field_names:
+        values.extend(str(value) for value in request_post.getlist(field_name))
+    if len(values) == 1:
+        value = values[0].strip()
+        if value.startswith("["):
+            try:
+                parsed = json.loads(value)
+            except json.JSONDecodeError:
+                parsed = None
+            if isinstance(parsed, list):
+                return [str(item) for item in parsed]
+    return values
+
+
+def _get_archiveresult_upload_form_value(request: HttpRequest, *field_names: str) -> str:
+    request_post, _files = _get_archiveresult_upload_data(request)
+    for field_name in field_names:
+        value = request_post.get(field_name)
+        if value is not None:
+            return str(value)
+    return ""
+
+
+def _parse_archiveresult_upload_int(value: str, field_name: str, *, default: int | None = None) -> int:
+    if value == "" and default is not None:
+        return default
+    try:
+        parsed = int(value)
+    except (TypeError, ValueError) as err:
+        raise HttpError(400, f"ArchiveResult {field_name} must be an integer") from err
+    if parsed < 0:
+        raise HttpError(400, f"ArchiveResult {field_name} must be non-negative")
+    return parsed
+
+
+def _write_archiveresult_files(
+    request: HttpRequest,
+    snapshot: Snapshot,
+    plugin_name: str,
+    *,
+    existing_output_files: dict[str, dict[str, Any]] | None = None,
+    allow_empty: bool = False,
+) -> dict[str, dict[str, Any]]:
+    files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
+    output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
+    mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
+    chunk_output_path = _get_archiveresult_upload_form_value(request, "chunk_output_path")
+
+    snapshot_dir = snapshot.output_dir
+    plugin_dir = snapshot_dir / plugin_name
+    storage = FileSystemStorage(location=str(plugin_dir))
+    output_files = dict(existing_output_files or {})
+
+    if not files:
+        return output_files
+
+    if chunk_output_path:
+        if len(files) != 1:
+            raise HttpError(400, "Exactly one ArchiveResult file chunk is required")
+
+        uploaded_file = files[0]
+        relative_output_path = _normalize_uploaded_archiveresult_output_path(
+            chunk_output_path,
+            filename=uploaded_file.name,
+        )
+        chunk_index = _parse_archiveresult_upload_int(
+            _get_archiveresult_upload_form_value(request, "chunk_index"),
+            "chunk_index",
+        )
+        chunk_count = _parse_archiveresult_upload_int(
+            _get_archiveresult_upload_form_value(request, "chunk_count"),
+            "chunk_count",
+        )
+        chunk_offset = _parse_archiveresult_upload_int(
+            _get_archiveresult_upload_form_value(request, "chunk_offset"),
+            "chunk_offset",
+        )
+        chunk_total_size = _parse_archiveresult_upload_int(
+            _get_archiveresult_upload_form_value(request, "chunk_total_size"),
+            "chunk_total_size",
+        )
+
+        if chunk_count < 1:
+            raise HttpError(400, "ArchiveResult chunk_count must be at least 1")
+        if chunk_index >= chunk_count:
+            raise HttpError(400, "ArchiveResult chunk_index must be less than chunk_count")
+        if chunk_total_size and chunk_offset > chunk_total_size:
+            raise HttpError(400, "ArchiveResult chunk_offset cannot exceed chunk_total_size")
+
+        if chunk_index == 0 and chunk_offset == 0 and storage.exists(relative_output_path):
+            storage.delete(relative_output_path)
+
+        current_size = storage.size(relative_output_path) if storage.exists(relative_output_path) else 0
+        if current_size != chunk_offset:
+            raise HttpError(
+                409,
+                f"ArchiveResult chunk offset mismatch for {relative_output_path}: expected {current_size}, got {chunk_offset}",
+            )
+
+        Path(storage.path(relative_output_path)).parent.mkdir(parents=True, exist_ok=True)
+        with storage.open(relative_output_path, "ab") as destination:
+            for chunk in uploaded_file.chunks():
+                destination.write(chunk)
+
+        size = storage.size(relative_output_path)
+        upload_complete = chunk_index + 1 == chunk_count
+        if upload_complete and size != chunk_total_size:
+            raise HttpError(
+                409,
+                f"ArchiveResult chunk size mismatch for {relative_output_path}: expected {chunk_total_size}, got {size}",
+            )
+
+        guessed_mime = mimetypes.guess_type(relative_output_path)[0]
+        output_mime_type = (mime_types[0] if mime_types else "") or uploaded_file.content_type or guessed_mime or "application/octet-stream"
+        output_files[relative_output_path] = {
+            "extension": PurePosixPath(relative_output_path).suffix.lower().lstrip("."),
+            "mimetype": output_mime_type,
+            "size": size,
+            "upload": {
+                "chunked": True,
+                "chunk_index": chunk_index,
+                "chunk_count": chunk_count,
+                "chunks_received": chunk_index + 1,
+                "complete": upload_complete,
+            },
+        }
+
+        return output_files
+
+    for index, uploaded_file in enumerate(files):
+        relative_output_path = _normalize_uploaded_archiveresult_output_path(
+            output_paths[index] if index < len(output_paths) else "",
+            filename=uploaded_file.name,
+        )
+        if storage.exists(relative_output_path):
+            storage.delete(relative_output_path)
+        saved_output_path = storage.save(relative_output_path, uploaded_file)
+        size = storage.size(saved_output_path)
+        guessed_mime = mimetypes.guess_type(saved_output_path)[0]
+        output_mime_type = (
+            (mime_types[index] if index < len(mime_types) else "")
+            or uploaded_file.content_type
+            or guessed_mime
+            or "application/octet-stream"
+        )
+        output_files[saved_output_path] = {
+            "extension": PurePosixPath(saved_output_path).suffix.lower().lstrip("."),
+            "mimetype": output_mime_type,
+            "size": size,
+        }
+
+    return output_files
+
+
+@router.post(
+    "/archiveresults",
+    response=ArchiveResultSchema,
+    url_name="create_archiveresult",
+)
+def create_archiveresult(
+    request: HttpRequest,
+    snapshot_id: str = Form(...),
+    plugin: str = Form(...),
+    output_str: str = Form(""),
+    hook_name: str = Form(ARCHIVERESULT_UPLOAD_HOOK_NAME),
+    status: str = Form(str(ArchiveResult.StatusChoices.SUCCEEDED)),
+    output_json: str = Form(""),
+):
+    """Create or update an ArchiveResult with one or more output files."""
+    snapshot = _get_snapshot_by_ref(snapshot_id)
+    plugin_name = _normalize_uploaded_archiveresult_plugin(plugin)
+    result = ArchiveResult.objects.filter(
+        snapshot=snapshot,
+        plugin=plugin_name,
+        hook_name=hook_name or ARCHIVERESULT_UPLOAD_HOOK_NAME,
+    ).first()
+    result = result or ArchiveResult(snapshot=snapshot, plugin=plugin_name, hook_name=hook_name or ARCHIVERESULT_UPLOAD_HOOK_NAME)
+    metadata = {"output_str": output_str, "status": status, "output_json": _parse_archiveresult_output_json(output_json)}
+    files = _write_archiveresult_files(request, snapshot, plugin_name, allow_empty=True)
+    try:
+        return result.apply_upload(files, metadata=metadata, replace_metadata=True)
+    except ArchiveResult.UploadConflict as err:
+        raise HttpError(409, str(err)) from err
+
+
+@router.patch("/archiveresult/{archiveresult_id}", response=ArchiveResultSchema, url_name="patch_archiveresult")
+def patch_archiveresult(
+    request: HttpRequest,
+    archiveresult_id: str,
+):
+    """Append or replace files on an existing ArchiveResult."""
+    result = ArchiveResult.objects.select_related("snapshot__crawl__created_by").get(_uuid_ref_query("id", archiveresult_id))
+    uploaded_output_files = _write_archiveresult_files(
+        request,
+        result.snapshot,
+        result.plugin,
+    )
+    output_str = _get_archiveresult_upload_form_value(request, "output_str")
+    status = _get_archiveresult_upload_form_value(request, "status")
+    output_json = _get_archiveresult_upload_form_value(request, "output_json")
+    metadata = {key: value for key, value in {"output_str": output_str, "status": status}.items() if value}
+    if output_json:
+        metadata["output_json"] = _parse_archiveresult_output_json(output_json)
+    try:
+        return result.apply_upload(uploaded_output_files, metadata=metadata)
+    except ArchiveResult.UploadConflict as err:
+        raise HttpError(409, str(err)) from err

archivebox/api/v1_core/lookup.py

@@ -0,0 +1,12 @@
+from django.db.models import Q
+
+from archivebox.core.models import Snapshot
+from archivebox.misc.db import uuid_ref_query as _uuid_ref_query
+
+
+def _get_snapshot_by_ref(snapshot_id: str):
+    queryset = Snapshot.objects.select_related("crawl__created_by")
+    try:
+        return queryset.get(_uuid_ref_query("id", snapshot_id) | Q(timestamp__startswith=snapshot_id))
+    except Snapshot.DoesNotExist:
+        return queryset.get(_uuid_ref_query("id", snapshot_id))

archivebox/api/v1_core/pagination.py

@@ -0,0 +1,40 @@
+import math
+from typing import Any
+
+from django.http import HttpRequest
+from ninja.pagination import PaginationBase
+
+
+class CustomPagination(PaginationBase):
+    class Input(PaginationBase.Input):
+        limit: int = 200
+        offset: int = 0
+        page: int = 0
+
+    class Output(PaginationBase.Output):
+        count: int
+        total_items: int
+        total_pages: int
+        page: int
+        limit: int
+        offset: int
+        num_items: int
+        items: list[Any]
+
+    def paginate_queryset(self, queryset, pagination: Input, request: HttpRequest, **params):
+        limit = min(pagination.limit, 500)
+        offset = pagination.offset or (pagination.page * limit)
+        total = queryset.values("pk").distinct().count() if queryset.query.distinct else queryset.count()
+        total_pages = math.ceil(total / limit)
+        current_page = math.ceil(offset / (limit + 1))
+        items = queryset[offset : offset + limit]
+        return {
+            "count": total,
+            "total_items": total,
+            "total_pages": total_pages,
+            "page": current_page,
+            "limit": limit,
+            "offset": offset,
+            "num_items": len(items),
+            "items": items,
+        }

archivebox/api/v1_core/schemas.py

@@ -0,0 +1,301 @@
+from collections import defaultdict
+from datetime import datetime
+from typing import Annotated, Any
+from uuid import UUID
+
+from django.db.models import Q
+from archivebox.api.schemas import OwnedObjectSchema
+
+from ninja import FilterLookup, FilterSchema, Schema
+
+from archivebox.core.models import ArchiveResult, Snapshot
+
+
+class MinimalArchiveResultSchema(OwnedObjectSchema):
+    TYPE: str = "core.models.ArchiveResult"
+    id: UUID
+    created_at: datetime | None
+    modified_at: datetime | None
+    status: str
+    retry_at: datetime | None = None
+    plugin: str
+    hook_name: str
+    process_id: UUID | None
+    cmd_version: str | None
+    cmd: list[str] | None
+    pwd: str | None
+    output_str: str
+    output_json: dict[str, Any] | None
+    output_files: dict[str, dict[str, Any]] | None
+    output_size: int
+    output_mimetypes: str
+    start_ts: datetime | None
+    end_ts: datetime | None
+
+    @staticmethod
+    def resolve_output_files(obj):
+        return obj.output_file_map()
+
+    @staticmethod
+    def resolve_output_mimetypes(obj) -> str:
+        mime_sizes: dict[str, int] = defaultdict(int)
+        for metadata in obj.output_file_map().values():
+            if not isinstance(metadata, dict):
+                continue
+            mimetype = str(metadata.get("mimetype") or "").strip()
+            try:
+                size = max(int(metadata.get("size") or 0), 0)
+            except (TypeError, ValueError):
+                size = 0
+            if mimetype and size:
+                mime_sizes[mimetype] += size
+        if mime_sizes:
+            return ",".join(mime for mime, _size in sorted(mime_sizes.items(), key=lambda item: item[1], reverse=True))
+        return obj.output_mimetypes or ""
+
+
+class ArchiveResultSchema(MinimalArchiveResultSchema):
+    TYPE: str = "core.models.ArchiveResult"
+    snapshot_id: UUID
+    snapshot_timestamp: str
+    snapshot_url: str
+    snapshot_tags: list[str]
+
+    @staticmethod
+    def resolve_snapshot_timestamp(obj):
+        return obj.snapshot.timestamp
+
+    @staticmethod
+    def resolve_snapshot_url(obj):
+        return obj.snapshot.url
+
+    @staticmethod
+    def resolve_snapshot_id(obj):
+        return obj.snapshot_id
+
+    @staticmethod
+    def resolve_snapshot_tags(obj):
+        return sorted(tag.name for tag in obj.snapshot.tags.all())
+
+
+class ArchiveResultFilterSchema(FilterSchema):
+    id: Annotated[str | None, FilterLookup(["id__startswith", "snapshot__id__startswith", "snapshot__timestamp__startswith"])] = None
+    search: Annotated[
+        str | None,
+        FilterLookup(
+            [
+                "snapshot__url__icontains",
+                "snapshot__title__icontains",
+                "snapshot__tags__name__icontains",
+                "plugin",
+                "output_str__icontains",
+                "id__startswith",
+                "snapshot__id__startswith",
+                "snapshot__timestamp__startswith",
+            ],
+        ),
+    ] = None
+    snapshot_id: Annotated[str | None, FilterLookup(["snapshot__id__startswith", "snapshot__timestamp__startswith"])] = None
+    snapshot_url: Annotated[str | None, FilterLookup("snapshot__url__icontains")] = None
+    snapshot_tag: Annotated[str | None, FilterLookup("snapshot__tags__name__icontains")] = None
+    status: Annotated[str | None, FilterLookup("status")] = None
+    output_str: Annotated[str | None, FilterLookup("output_str__icontains")] = None
+    plugin: Annotated[str | None, FilterLookup("plugin__icontains")] = None
+    hook_name: Annotated[str | None, FilterLookup("hook_name__icontains")] = None
+    process_id: Annotated[str | None, FilterLookup("process__id__startswith")] = None
+    cmd: Annotated[str | None, FilterLookup("cmd__0__icontains")] = None
+    pwd: Annotated[str | None, FilterLookup("pwd__icontains")] = None
+    cmd_version: Annotated[str | None, FilterLookup("cmd_version")] = None
+    created_at: Annotated[datetime | None, FilterLookup("created_at")] = None
+    created_at__gte: Annotated[datetime | None, FilterLookup("created_at__gte")] = None
+    created_at__lt: Annotated[datetime | None, FilterLookup("created_at__lt")] = None
+
+
+class SnapshotSchema(OwnedObjectSchema):
+    TYPE: str = "core.models.Snapshot"
+    id: UUID
+    created_at: datetime
+    modified_at: datetime
+    status: str
+    retry_at: datetime | None
+    bookmarked_at: datetime
+    downloaded_at: datetime | None
+    url: str
+    tags: list[str]
+    title: str | None
+    timestamp: str
+    archive_path: str
+    archive_size: int
+    output_size: int
+    num_archiveresults: int
+    archiveresults: list[MinimalArchiveResultSchema]
+
+    @staticmethod
+    def resolve_tags(obj):
+        return sorted(tag.name for tag in obj.tags.all())
+
+    @staticmethod
+    def resolve_archive_size(obj):
+        return int(obj.archive_size or 0)
+
+    @staticmethod
+    def resolve_output_size(obj):
+        return SnapshotSchema.resolve_archive_size(obj)
+
+    @staticmethod
+    def resolve_num_archiveresults(obj, context):
+        return obj.archiveresult_set.all().distinct().count()
+
+    @staticmethod
+    def resolve_archiveresults(obj, context):
+        if bool(context["request"].__dict__.get("with_archiveresults", False)):
+            return obj.archiveresult_set.all().distinct()
+        return ArchiveResult.objects.none()
+
+
+class SnapshotUpdateSchema(Schema):
+    action: str | None = None
+    status: str | None = None
+    retry_at: datetime | None = None
+    tags: list[str] | None = None
+
+
+class SnapshotCreateSchema(Schema):
+    url: str
+    crawl_id: str | None = None
+    depth: int = 0
+    title: str | None = None
+    tags: list[str] | None = None
+    status: str | None = None
+
+
+class SnapshotDeleteResponseSchema(Schema):
+    success: bool
+    snapshot_id: str
+    crawl_id: str
+    deleted_count: int
+
+
+class SnapshotFilterSchema(FilterSchema):
+    id: Annotated[str | None, FilterLookup(["id__istartswith", "id__iendswith", "timestamp__startswith"])] = None
+    created_by_id: Annotated[str | None, FilterLookup("crawl__created_by_id")] = None
+    created_by_username: Annotated[str | None, FilterLookup("crawl__created_by__username__icontains")] = None
+    created_at__gte: Annotated[datetime | None, FilterLookup("created_at__gte")] = None
+    created_at__lt: Annotated[datetime | None, FilterLookup("created_at__lt")] = None
+    created_at: Annotated[datetime | None, FilterLookup("created_at")] = None
+    modified_at: Annotated[datetime | None, FilterLookup("modified_at")] = None
+    modified_at__gte: Annotated[datetime | None, FilterLookup("modified_at__gte")] = None
+    modified_at__lt: Annotated[datetime | None, FilterLookup("modified_at__lt")] = None
+    search: str | None = None
+    search_mode: str | None = None
+    status: str | None = None
+    url: Annotated[str | None, FilterLookup("url")] = None
+    tag: Annotated[str | None, FilterLookup("tags__name")] = None
+    title: Annotated[str | None, FilterLookup("title__icontains")] = None
+    timestamp: Annotated[str | None, FilterLookup("timestamp__startswith")] = None
+    bookmarked_at__gte: Annotated[datetime | None, FilterLookup("bookmarked_at__gte")] = None
+    bookmarked_at__lt: Annotated[datetime | None, FilterLookup("bookmarked_at__lt")] = None
+
+    def filter_search(self, value: str | None) -> Q:
+        return Q()
+
+    def filter_search_mode(self, value: str | None) -> Q:
+        return Q()
+
+    def filter_status(self, value: str | None) -> Q:
+        return Q()
+
+
+class TagSchema(OwnedObjectSchema):
+    TYPE: str = "core.models.Tag"
+    id: int
+    modified_at: datetime
+    created_at: datetime
+    name: str
+    num_snapshots: int
+    snapshots: list[SnapshotSchema]
+
+    @staticmethod
+    def resolve_num_snapshots(obj, context):
+        return obj.snapshot_set.all().distinct().count()
+
+    @staticmethod
+    def resolve_snapshots(obj, context):
+        if bool(context["request"].__dict__.get("with_snapshots", False)):
+            return obj.snapshot_set.all().distinct()
+        return Snapshot.objects.none()
+
+
+class TagAutocompleteSchema(Schema):
+    tags: list[dict]
+
+
+class TagCreateSchema(Schema):
+    name: str
+
+
+class TagCreateResponseSchema(Schema):
+    success: bool
+    tag_id: int
+    tag_name: str
+    created: bool
+
+
+class TagSearchSnapshotSchema(Schema):
+    id: str
+    title: str
+    url: str
+    favicon_url: str
+    admin_url: str
+    archive_url: str
+    downloaded_at: str | None = None
+
+
+class TagSearchCardSchema(Schema):
+    id: int
+    name: str
+    slug: str
+    num_snapshots: int
+    filter_url: str
+    edit_url: str
+    export_urls_url: str
+    export_jsonl_url: str
+    rename_url: str
+    delete_url: str
+    snapshots: list[TagSearchSnapshotSchema]
+
+
+class TagSearchResponseSchema(Schema):
+    tags: list[TagSearchCardSchema]
+    sort: str
+    created_by: str
+    year: str
+    has_snapshots: str
+
+
+class TagUpdateSchema(Schema):
+    name: str
+
+
+class TagUpdateResponseSchema(Schema):
+    success: bool
+    tag_id: int
+    tag_name: str
+
+
+class TagDeleteResponseSchema(Schema):
+    success: bool
+    tag_id: int
+    deleted_count: int
+
+
+class TagSnapshotRequestSchema(Schema):
+    snapshot_id: str
+    tag_name: str | None = None
+    tag_id: int | None = None
+
+
+class TagSnapshotResponseSchema(Schema):
+    success: bool
+    tag_id: int
+    tag_name: str

archivebox/api/v1_core/snapshots.py

@@ -0,0 +1,356 @@
+from datetime import datetime, time
+from html import unescape
+from typing import Any
+
+from django.contrib.auth import get_user_model
+from django.contrib.auth.models import User
+from django.core.exceptions import ValidationError
+from django.db import IntegrityError
+from django.db.models import Q
+from django.http import HttpRequest, HttpResponse
+from django.utils import timezone
+from django.utils.dateparse import parse_date, parse_datetime
+from django.utils.feedgenerator import Rss201rev2Feed
+from ninja import Query, Router
+from ninja.errors import HttpError
+from ninja.pagination import paginate
+
+from archivebox.api.v1_crawls import get_crawl_by_ref
+from archivebox.core.models import Snapshot
+from archivebox.core.routes_util import build_web_url
+from archivebox.core.snapshot_status import filter_snapshots_by_status, normalize_snapshot_status
+from archivebox.crawls.locks import crawl_lifecycle_lock
+from archivebox.crawls.models import Crawl
+from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url_length
+from archivebox.search.config import get_search_mode, get_search_mode_backend
+from archivebox.search.query import apply_snapshot_search
+
+from .lookup import _get_snapshot_by_ref
+from .pagination import CustomPagination
+from .schemas import SnapshotCreateSchema, SnapshotDeleteResponseSchema, SnapshotFilterSchema, SnapshotSchema, SnapshotUpdateSchema
+
+router = Router(tags=["Core Models"])
+
+
+def normalize_tag_list(tags: list[str] | None = None) -> list[str]:
+    return [tag.strip() for tag in (tags or []) if tag and tag.strip()]
+
+
+def _parse_rss_before(before: str | None) -> datetime:
+    if not before:
+        return timezone.now()
+
+    value = before.strip()
+    parsed_dt = None
+
+    if len(value) == 8 and value.isdigit():
+        parsed_date = datetime.strptime(value, "%Y%m%d").date()
+    else:
+        parsed_dt = parse_datetime(value)
+        parsed_date = None if parsed_dt else parse_date(value)
+
+    if parsed_dt is None:
+        if parsed_date is None:
+            raise HttpError(400, "before must be an ISO datetime, YYYY-MM-DD, or YYYYMMDD")
+        parsed_dt = datetime.combine(parsed_date, time.max)
+
+    if timezone.is_naive(parsed_dt):
+        parsed_dt = timezone.make_aware(parsed_dt, timezone.get_current_timezone())
+    return parsed_dt
+
+
+def _filter_snapshots_for_rss(
+    *,
+    crawl_id: str = "",
+    created_by: str = "",
+    before: str | None = None,
+    limit: int = 50,
+):
+    limit = max(1, min(int(limit or 50), 500))
+    before_dt = _parse_rss_before(before)
+    queryset = (
+        Snapshot.objects.select_related("crawl__created_by")
+        .prefetch_related("tags")
+        .only(
+            "id",
+            "url",
+            "title",
+            "timestamp",
+            "bookmarked_at",
+            "created_at",
+            "modified_at",
+            "fs_version",
+            "crawl_id",
+            "crawl__id",
+            "crawl__created_by_id",
+            "crawl__created_by__id",
+            "crawl__created_by__username",
+        )
+        .filter(bookmarked_at__lte=before_dt)
+    )
+    crawl_id = crawl_id.strip()
+    if crawl_id:
+        matching_crawl_pks = list(filter_queryset_by_uuid_substring(Crawl.objects.all(), crawl_id).values_list("pk", flat=True)[:100])
+        queryset = queryset.filter(crawl_id__in=matching_crawl_pks)
+
+    created_by = created_by.strip()
+    if created_by:
+        created_by_query = Q(crawl__created_by__username__iexact=created_by)
+        user_model = get_user_model()
+        try:
+            prepared_pk = user_model._meta.pk.get_prep_value(created_by)
+        except (TypeError, ValueError, ValidationError):
+            prepared_pk = None
+        if prepared_pk not in (None, ""):
+            created_by_query |= Q(crawl__created_by_id=prepared_pk)
+        queryset = queryset.filter(created_by_query)
+
+    return queryset.order_by("-bookmarked_at", "-created_at", "-id")[:limit]
+
+
+def _snapshots_rss_response(
+    request: HttpRequest,
+    *,
+    snapshots,
+    title: str = "ArchiveBox Snapshots",
+) -> HttpResponse:
+    web_base_url = build_web_url("/", request=request).rstrip("/")
+    feed_query = request.GET.copy()
+    for sensitive_param in ("api_key", "token", "password"):
+        feed_query.pop(sensitive_param, None)
+    feed_path = request.path
+    feed_url = request.build_absolute_uri(f"{feed_path}?{feed_query.urlencode()}" if feed_query else feed_path)
+
+    feed = Rss201rev2Feed(
+        title=title,
+        link=build_web_url("/public/", request=request),
+        description="Recently added ArchiveBox snapshots.",
+        language="en",
+        feed_url=feed_url,
+    )
+
+    for snapshot in snapshots:
+        archived_url = build_web_url(f"/{snapshot.archive_path_from_db}", request=request)
+        tags = [tag.name for tag in snapshot.tags.all()]
+        crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
+        description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
+        feed.add_item(
+            title=unescape(snapshot.title or snapshot.url),
+            link=archived_url or web_base_url,
+            description=description,
+            unique_id=str(snapshot.id),
+            unique_id_is_permalink=False,
+            pubdate=snapshot.bookmarked_at or snapshot.created_at,
+            updateddate=snapshot.modified_at,
+            author_name=crawl_user.username if crawl_user else None,
+            categories=tags,
+        )
+
+    return HttpResponse(feed.writeString("utf-8"), content_type="application/rss+xml; charset=utf-8")
+
+
+@router.get("/snapshots", response=list[SnapshotSchema], url_name="get_snapshots")
+@paginate(CustomPagination)
+def get_snapshots(request: HttpRequest, filters: Query[SnapshotFilterSchema], with_archiveresults: bool = False):
+    """List all Snapshot entries matching these filters."""
+    setattr(request, "with_archiveresults", with_archiveresults)
+    try:
+        queryset = filter_snapshots_by_status(Snapshot.objects.all(), filters.status)
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
+    queryset = filters.filter(queryset).distinct()
+    query = (filters.search or "").strip()
+    if not query:
+        return queryset
+
+    runtime_config = request.archivebox_config
+    search_mode = get_search_mode(filters.search_mode, config=runtime_config)
+    try:
+        return apply_snapshot_search(
+            queryset,
+            query,
+            search_mode=search_mode,
+            config=runtime_config,
+            include_id_matches=True,
+        )
+    except Exception:
+        if get_search_mode_backend(search_mode, config=runtime_config):
+            return queryset.none()
+        return apply_snapshot_search(queryset, query, search_mode="meta", config=runtime_config, include_id_matches=True)
+
+
+@router.get("/snapshots.rss", url_name="get_snapshots_rss")
+def get_snapshots_rss(
+    request: HttpRequest,
+    crawl_id: str = "",
+    created_by: str = "",
+    limit: int = 50,
+    before: str | None = None,
+):
+    """Return matching snapshots as an RSS feed, newest first."""
+    snapshots = _filter_snapshots_for_rss(
+        crawl_id=crawl_id,
+        created_by=created_by,
+        limit=limit,
+        before=before,
+    )
+    return _snapshots_rss_response(request, snapshots=snapshots)
+
+
+@router.get("/snapshot/{snapshot_id}", response=SnapshotSchema, url_name="get_snapshot")
+def get_snapshot(request: HttpRequest, snapshot_id: str, with_archiveresults: bool = True):
+    """Get a specific Snapshot by id."""
+    setattr(request, "with_archiveresults", with_archiveresults)
+    return _get_snapshot_by_ref(snapshot_id)
+
+
+@router.post("/snapshots", response=SnapshotSchema, url_name="create_snapshot")
+def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
+    tags = normalize_tag_list(data.tags)
+    try:
+        status = normalize_snapshot_status(data.status)
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
+    if not data.url.strip():
+        raise HttpError(400, "URL is required")
+    try:
+        validate_url_length(data.url.strip())
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
+    if data.depth not in (0, 1, 2, 3, 4):
+        raise HttpError(400, "depth must be between 0 and 4")
+
+    if data.crawl_id:
+        crawl = get_crawl_by_ref(data.crawl_id)
+        crawl_tags = normalize_tag_list(crawl.tags_str.split(","))
+        tags = tags or crawl_tags
+    else:
+        crawl = Crawl.objects.create(
+            urls=data.url,
+            max_depth=max(data.depth, 0),
+            tags_str=",".join(tags),
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=timezone.now(),
+            created_by=request.user if isinstance(request.user, User) else None,
+        )
+
+    # Browser uploads must not wait behind the runner's crawl-wide lifecycle
+    # lock. The unique insert recovery and CAS update below are the request-side
+    # coordination boundary for this idempotent metadata sync.
+    snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
+    if snapshot is None:
+        try:
+            snapshot = Snapshot.objects.create(
+                url=data.url,
+                crawl=crawl,
+                depth=data.depth,
+                title=data.title,
+                timestamp=str(timezone.now().timestamp()),
+                status=status or Snapshot.StatusChoices.QUEUED,
+                retry_at=timezone.now(),
+            )
+        except IntegrityError:
+            snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
+            if snapshot is None:
+                raise
+
+    for _attempt in range(3):
+        updates: dict[str, Any] = {}
+        if data.title is not None and snapshot.title != data.title:
+            updates["title"] = data.title
+        if status is not None and snapshot.status != status:
+            updates["status"] = status
+        if not updates or snapshot.safe_update(updates, extra_filter={"modified_at": snapshot.modified_at}):
+            break
+    else:
+        raise HttpError(409, "Snapshot changed while metadata was being updated")
+
+    if tags:
+        snapshot.save_tags(
+            tags,
+            created_by=request.user if isinstance(request.user, User) else None,
+        )
+
+    try:
+        snapshot.ensure_crawl_symlink()
+    except Exception:
+        pass
+
+    setattr(request, "with_archiveresults", False)
+    return snapshot
+
+
+@router.patch("/snapshot/{snapshot_id}", response=SnapshotSchema, url_name="patch_snapshot")
+def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateSchema):
+    """Update a snapshot (e.g., set status=sealed to cancel queued work)."""
+    snapshot = _get_snapshot_by_ref(snapshot_id)
+    crawl_id = str(snapshot.crawl_id)
+
+    payload = data.dict(exclude_unset=True)
+    update_fields = ["modified_at"]
+    action = payload.pop("action", None)
+    tags = payload.pop("tags", None)
+
+    if action:
+        if action == "pause":
+            snapshot.pause()
+            setattr(request, "with_archiveresults", False)
+            return snapshot
+        if action in ("resume", "unpause"):
+            snapshot.resume()
+            setattr(request, "with_archiveresults", False)
+            return snapshot
+        if action == "cancel":
+            snapshot.cancel()
+            setattr(request, "with_archiveresults", False)
+            return snapshot
+        raise HttpError(400, f"Invalid action: {action}")
+
+    with crawl_lifecycle_lock(crawl_id):
+        snapshot = _get_snapshot_by_ref(snapshot_id)
+        if "status" in payload:
+            try:
+                snapshot.status = normalize_snapshot_status(payload["status"])
+            except ValueError as err:
+                raise HttpError(400, str(err)) from err
+            if snapshot.status == Snapshot.StatusChoices.SEALED and "retry_at" not in payload:
+                snapshot.retry_at = None
+            update_fields.append("status")
+
+        if "retry_at" in payload:
+            snapshot.retry_at = payload["retry_at"]
+            update_fields.append("retry_at")
+
+        if tags is not None:
+            snapshot.save_tags(
+                normalize_tag_list(tags),
+                created_by=request.user if isinstance(request.user, User) else None,
+            )
+
+        if payload.get("status") == Snapshot.StatusChoices.SEALED:
+            snapshot.cancel()
+        else:
+            snapshot.save(update_fields=update_fields)
+    setattr(request, "with_archiveresults", False)
+    return snapshot
+
+
+@router.delete("/snapshot/{snapshot_id}", response=SnapshotDeleteResponseSchema, url_name="delete_snapshot")
+def delete_snapshot(request: HttpRequest, snapshot_id: str):
+    snapshot = get_snapshot(request, snapshot_id, with_archiveresults=False)
+    snapshot_id_str = str(snapshot.id)
+    crawl_id_str = str(snapshot.crawl.pk)
+    snapshot.cancel()
+
+    from archivebox.services.runner import run_pending_crawls
+
+    with crawl_lifecycle_lock(crawl_id_str):
+        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
+        snapshot = get_snapshot(request, snapshot_id_str, with_archiveresults=False)
+        deleted_count, _ = snapshot.delete()
+    return {
+        "success": True,
+        "snapshot_id": snapshot_id_str,
+        "crawl_id": crawl_id_str,
+        "deleted_count": deleted_count,
+    }

archivebox/api/v1_core/tags.py

@@ -0,0 +1,285 @@
+from django.core.exceptions import ValidationError
+from django.db.models import Q
+from django.http import HttpRequest, HttpResponse
+from ninja import Router
+from ninja.errors import HttpError
+from ninja.pagination import paginate
+
+from archivebox.api.auth import authenticated_user_from_request
+from archivebox.config.common import get_config
+from archivebox.core.models import Snapshot, Tag
+from archivebox.core.permissions import public_snapshots_queryset
+from archivebox.core.tag_util import (
+    add_snapshot_counts,
+    build_tag_cards,
+    export_tag_snapshots_jsonl,
+    export_tag_urls,
+    get_matching_tags,
+    get_tag_by_ref,
+    normalize_created_by_filter,
+    normalize_created_year_filter,
+    normalize_has_snapshots_filter,
+    normalize_tag_sort,
+)
+
+from .pagination import CustomPagination
+from .schemas import (
+    TagAutocompleteSchema,
+    TagCreateResponseSchema,
+    TagCreateSchema,
+    TagDeleteResponseSchema,
+    TagSchema,
+    TagSearchResponseSchema,
+    TagSnapshotRequestSchema,
+    TagSnapshotResponseSchema,
+    TagUpdateResponseSchema,
+    TagUpdateSchema,
+)
+
+router = Router(tags=["Core Models"])
+
+
+@router.get("/tags", response=list[TagSchema], url_name="get_tags")
+@paginate(CustomPagination)
+def get_tags(request: HttpRequest):
+    setattr(request, "with_snapshots", False)
+    setattr(request, "with_archiveresults", False)
+    return get_matching_tags()
+
+
+@router.get("/tag/{tag_id}", response=TagSchema, url_name="get_tag")
+def get_tag(request: HttpRequest, tag_id: str, with_snapshots: bool = True):
+    setattr(request, "with_snapshots", with_snapshots)
+    setattr(request, "with_archiveresults", False)
+    try:
+        return get_tag_by_ref(tag_id)
+    except (Tag.DoesNotExist, ValidationError):
+        raise HttpError(404, "Tag not found")
+
+
+def _get_snapshot_for_tag_edit(snapshot_ref: str) -> Snapshot:
+    snapshot_ref = str(snapshot_ref or "").strip().lower()
+    if not snapshot_ref:
+        raise HttpError(400, "Snapshot id is required")
+
+    snapshot_qs = Snapshot.objects.only("id")
+    is_full_uuid = len(snapshot_ref.replace("-", "")) == 32 and all(char in "0123456789abcdef-" for char in snapshot_ref)
+    if is_full_uuid:
+        try:
+            return snapshot_qs.get(pk=snapshot_ref.replace("-", ""))
+        except (Snapshot.DoesNotExist, ValueError):
+            pass
+
+    if len(snapshot_ref) >= 14:
+        try:
+            return snapshot_qs.get(timestamp=snapshot_ref)
+        except Snapshot.DoesNotExist:
+            pass
+        except Snapshot.MultipleObjectsReturned:
+            snapshot = snapshot_qs.filter(timestamp=snapshot_ref).first()
+            if snapshot is not None:
+                return snapshot
+
+    try:
+        return snapshot_qs.get(Q(id__startswith=snapshot_ref) | Q(timestamp__startswith=snapshot_ref))
+    except Snapshot.DoesNotExist:
+        raise HttpError(404, "Snapshot not found") from None
+    except Snapshot.MultipleObjectsReturned:
+        snapshot = snapshot_qs.filter(Q(id__startswith=snapshot_ref) | Q(timestamp__startswith=snapshot_ref)).first()
+        if snapshot is None:
+            raise HttpError(404, "Snapshot not found")
+        return snapshot
+
+
+@router.get("/tags/search/", response=TagSearchResponseSchema, url_name="search_tags")
+def search_tags(
+    request: HttpRequest,
+    q: str = "",
+    sort: str = "created_desc",
+    created_by: str = "",
+    year: str = "",
+    has_snapshots: str = "all",
+):
+    """Return detailed tag cards for admin/live-search UIs."""
+    normalized_sort = normalize_tag_sort(sort)
+    normalized_created_by = normalize_created_by_filter(created_by)
+    normalized_year = normalize_created_year_filter(year)
+    normalized_has_snapshots = normalize_has_snapshots_filter(has_snapshots)
+    return {
+        "tags": build_tag_cards(
+            query=q,
+            request=request,
+            preview_limit=0,
+            sort=normalized_sort,
+            created_by=normalized_created_by,
+            year=normalized_year,
+            has_snapshots=normalized_has_snapshots,
+        ),
+        "sort": normalized_sort,
+        "created_by": normalized_created_by,
+        "year": normalized_year,
+        "has_snapshots": normalized_has_snapshots,
+    }
+
+
+def _public_tag_listing_enabled() -> bool:
+    return get_config().PUBLIC_INDEX
+
+
+def _request_has_tag_autocomplete_access(request: HttpRequest) -> bool:
+    if authenticated_user_from_request(request):
+        return True
+
+    return _public_tag_listing_enabled()
+
+
+@router.get("/tags/autocomplete/", response=TagAutocompleteSchema, url_name="tags_autocomplete", auth=None)
+def tags_autocomplete(request: HttpRequest, q: str = ""):
+    """Return tags matching the query for autocomplete."""
+    if not _request_has_tag_autocomplete_access(request):
+        raise HttpError(401, "Authentication required")
+
+    public_only = not request.user.is_authenticated and not request.__dict__.get("_api_token")
+    queryset = get_matching_tags(q)
+    public_snapshots = public_snapshots_queryset(Snapshot.objects.all())
+    if public_only:
+        queryset = queryset.filter(snapshot_set__id__in=public_snapshots.values("id")).distinct()
+    tags = list(queryset[: 50 if not q else 20])
+    add_snapshot_counts(tags, snapshot_queryset=public_snapshots if public_only else None)
+
+    return {
+        "tags": [{"id": tag.pk, "name": tag.name, "num_snapshots": tag.__dict__.get("num_snapshots", 0)} for tag in tags],
+    }
+
+
+@router.post("/tags/create/", response=TagCreateResponseSchema, url_name="tags_create")
+def tags_create(request: HttpRequest, data: TagCreateSchema):
+    """Create a new tag or return existing one."""
+    try:
+        tag, created = Tag.get_or_create_by_name(
+            data.name,
+            created_by=request.user if request.user.is_authenticated else None,
+        )
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
+
+    return {
+        "success": True,
+        "tag_id": tag.pk,
+        "tag_name": tag.name,
+        "created": created,
+    }
+
+
+@router.post("/tag/{tag_id}/rename", response=TagUpdateResponseSchema, url_name="rename_tag")
+def rename_tag(request: HttpRequest, tag_id: int, data: TagUpdateSchema):
+    try:
+        tag = get_tag_by_ref(tag_id).rename(data.name)
+    except Tag.DoesNotExist as err:
+        raise HttpError(404, "Tag not found") from err
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
+
+    return {
+        "success": True,
+        "tag_id": tag.pk,
+        "tag_name": tag.name,
+    }
+
+
+@router.delete("/tag/{tag_id}", response=TagDeleteResponseSchema, url_name="delete_tag")
+def delete_tag(request: HttpRequest, tag_id: int):
+    try:
+        tag = get_tag_by_ref(tag_id)
+    except Tag.DoesNotExist as err:
+        raise HttpError(404, "Tag not found") from err
+
+    deleted_count, _ = tag.delete()
+    return {
+        "success": True,
+        "tag_id": int(tag_id),
+        "deleted_count": deleted_count,
+    }
+
+
+@router.get("/tag/{tag_id}/urls.txt", url_name="tag_urls_export")
+def tag_urls_export(request: HttpRequest, tag_id: int):
+    try:
+        tag = get_tag_by_ref(tag_id)
+    except Tag.DoesNotExist as err:
+        raise HttpError(404, "Tag not found") from err
+
+    response = HttpResponse(export_tag_urls(tag), content_type="text/plain; charset=utf-8")
+    response["Content-Disposition"] = f'attachment; filename="tag-{tag.slug}-urls.txt"'
+    return response
+
+
+@router.get("/tag/{tag_id}/snapshots.jsonl", url_name="tag_snapshots_export")
+def tag_snapshots_export(request: HttpRequest, tag_id: int):
+    try:
+        tag = get_tag_by_ref(tag_id)
+    except Tag.DoesNotExist as err:
+        raise HttpError(404, "Tag not found") from err
+
+    response = HttpResponse(export_tag_snapshots_jsonl(tag), content_type="application/x-ndjson; charset=utf-8")
+    response["Content-Disposition"] = f'attachment; filename="tag-{tag.slug}-snapshots.jsonl"'
+    return response
+
+
+@router.post("/tags/add-to-snapshot/", response=TagSnapshotResponseSchema, url_name="tags_add_to_snapshot")
+def tags_add_to_snapshot(request: HttpRequest, data: TagSnapshotRequestSchema):
+    """Add a tag to a snapshot. Creates the tag if it doesn't exist."""
+    snapshot = _get_snapshot_for_tag_edit(data.snapshot_id)
+
+    # Get or create the tag
+    if data.tag_name:
+        try:
+            tag, _ = Tag.get_or_create_by_name(
+                data.tag_name,
+                created_by=request.user if request.user.is_authenticated else None,
+            )
+        except ValueError as err:
+            raise HttpError(400, str(err)) from err
+    elif data.tag_id:
+        try:
+            tag = get_tag_by_ref(data.tag_id)
+        except Tag.DoesNotExist:
+            raise HttpError(404, "Tag not found")
+    else:
+        raise HttpError(400, "Either tag_name or tag_id is required")
+
+    snapshot.add_tag_ids([tag.pk])
+
+    return {
+        "success": True,
+        "tag_id": tag.pk,
+        "tag_name": tag.name,
+    }
+
+
+@router.post("/tags/remove-from-snapshot/", response=TagSnapshotResponseSchema, url_name="tags_remove_from_snapshot")
+def tags_remove_from_snapshot(request: HttpRequest, data: TagSnapshotRequestSchema):
+    """Remove a tag from a snapshot."""
+    snapshot = _get_snapshot_for_tag_edit(data.snapshot_id)
+
+    # Get the tag
+    if data.tag_id:
+        try:
+            tag = Tag.objects.get(pk=data.tag_id)
+        except Tag.DoesNotExist:
+            raise HttpError(404, "Tag not found")
+    elif data.tag_name:
+        try:
+            tag = Tag.objects.get(name__iexact=data.tag_name.strip())
+        except Tag.DoesNotExist:
+            raise HttpError(404, "Tag not found")
+    else:
+        raise HttpError(400, "Either tag_name or tag_id is required")
+
+    snapshot.remove_tag_ids([tag.pk])
+
+    return {
+        "success": True,
+        "tag_id": tag.pk,
+        "tag_name": tag.name,
+    }

archivebox/api/v1_crawls.py

@@ -1,6 +1,5 @@
 __package__ = "archivebox.api"
 
-import json
 from pathlib import Path
 from uuid import UUID
 from datetime import datetime
@@ -8,9 +7,10 @@ from django.http import FileResponse, HttpRequest
 from django.shortcuts import redirect
 from django.utils import timezone
 
-from django.contrib.auth import get_user_model
 from django.contrib.auth.models import User
 
+from archivebox.api.schemas import OwnedObjectSchema
+
 from ninja import Router, Schema
 from ninja.errors import HttpError
 
@@ -23,22 +23,20 @@ from archivebox.core.permissions import (
 )
 from archivebox.config.common import get_config
 from archivebox.crawls.models import Crawl
-from archivebox.misc.util import filter_queryset_by_uuid_substring
+from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url
 
 from .auth import API_AUTH_METHODS, authenticated_user_from_request
 
 router = Router(tags=["Crawl Models"], auth=API_AUTH_METHODS)
 
 
-class CrawlSchema(Schema):
+class CrawlSchema(OwnedObjectSchema):
     TYPE: str = "crawls.models.Crawl"
 
     id: UUID
 
     modified_at: datetime
     created_at: datetime
-    created_by_id: str
-    created_by_username: str
 
     status: str
     retry_at: datetime | None
@@ -49,17 +47,6 @@ class CrawlSchema(Schema):
     tags_str: str
     config: dict
 
-    @staticmethod
-    def resolve_created_by_id(obj):
-        return str(obj.created_by_id)
-
-    @staticmethod
-    def resolve_created_by_username(obj):
-        user_model = get_user_model()
-        user = user_model.objects.get(id=obj.created_by_id)
-        username = user.username
-        return username if isinstance(username, str) else str(user)
-
     @staticmethod
     def resolve_config(obj):
         # Redact credential values so REST responses can never leak the raw
@@ -113,7 +100,10 @@ def get_crawls(request: HttpRequest):
 
 @router.post("/crawls", response=CrawlSchema, url_name="create_crawl")
 def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
-    urls = [url.strip() for url in data.urls if url and url.strip()]
+    try:
+        urls = [validate_url(url) for url in data.urls if url and url.strip()]
+    except ValueError as err:
+        raise HttpError(400, str(err)) from err
     if not urls:
         raise HttpError(400, "At least one URL is required")
     if data.max_depth not in (0, 1, 2, 3, 4):
@@ -121,9 +111,10 @@ def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
 
     tags = normalize_tag_list(data.tags, data.tags_str)
     config = dict(data.config or {})
+    config.setdefault("PARSER", "url_list")
     config.setdefault("PERMISSIONS", str(get_config().PERMISSIONS))
     crawl = Crawl.objects.create(
-        urls="\n".join(json.dumps({"type": "CrawlSeed", "url": url, "depth": 0}, separators=(",", ":")) for url in urls),
+        urls="\n".join(urls),
         max_depth=data.max_depth,
         tags_str=",".join(tags),
         label=data.label,

archivebox/api/v1_personas.py

@@ -7,6 +7,8 @@ from uuid import UUID
 
 from django.db.models import Q
 from django.http import HttpRequest
+from archivebox.api.schemas import OwnedObjectSchema
+
 from ninja import Router, Schema
 from ninja.pagination import paginate
 from pydantic import Field
@@ -36,23 +38,13 @@ class PersonaSyncSchema(Schema):
     auth_json: dict[str, Any] = Field(default_factory=dict)
 
 
-class PersonaSchema(Schema):
+class PersonaSchema(OwnedObjectSchema):
     TYPE: str = "personas.models.Persona"
     id: UUID
     name: str
     created_at: datetime
-    created_by_id: str
-    created_by_username: str
     config: dict[str, Any] | None
 
-    @staticmethod
-    def resolve_created_by_id(obj):
-        return str(obj.created_by.pk)
-
-    @staticmethod
-    def resolve_created_by_username(obj) -> str:
-        return obj.created_by.username
-
     @staticmethod
     def resolve_config(obj):
         # Redact credential values so REST responses don't leak the raw

archivebox/base_models/admin.py

@@ -13,10 +13,16 @@ from django.db import DatabaseError, models
 from django.forms.renderers import BaseRenderer
 from django.http import HttpRequest, QueryDict
 from django.urls import path, register_converter
+from django.utils.html import format_html
 from django.utils.safestring import SafeString, mark_safe
 from django_object_actions import DjangoObjectActions
 
 
+def card_fieldset(title: str | None, fields: tuple, *, wide: bool = False, **options):
+    """Build a Django fieldset using the shared card layout."""
+    return title, {"fields": fields, "classes": ("card", "wide") if wide else ("card",), **options}
+
+
 class HexUUIDConverter:
     """URL path converter that canonicalizes UUIDs to their 32-char hex form.
 
@@ -81,6 +87,11 @@ class KeyValueWidget(forms.Widget):
             options: dict[str, ConfigOption] = {}
             for key, metadata in config_field_metadata().items():
                 option_type = metadata.get("type", "string")
+                # Core fields expose Python type names; plugins use JSON Schema.
+                # The editor should apply the same typed validation to both.
+                type_names = {"bool": "boolean", "int": "integer", "float": "number", "str": "string", "dict": "object", "list": "array"}
+                if isinstance(option_type, str):
+                    option_type = type_names.get(option_type, option_type)
                 option: ConfigOption = {
                     "plugin": str(metadata.get("plugin", "archivebox")),
                     "type": cast(str | list[str], option_type if isinstance(option_type, (str, list)) else str(option_type)),
@@ -118,650 +129,36 @@ class KeyValueWidget(forms.Widget):
         attrs: Mapping[str, str] | None = None,
         renderer: BaseRenderer | None = None,
     ) -> SafeString:
-        data = self._parse_value(value)
+        from django.template.loader import render_to_string
 
         widget_id = attrs.get("id", name) if attrs else name
-        config_options = self._get_config_options()
-
-        # Build datalist options
-        datalist_options = "\n".join(
-            f'<option value="{self._escape(key)}">{self._escape(opt["description"][:60] or opt["type"])}</option>'
-            for key, opt in sorted(config_options.items())
+        options = self._get_config_options()
+        rows = [
+            mark_safe(self._render_row(widget_id, key, val if isinstance(val, str) else json.dumps(val)))
+            for key, val in self._parse_value(value).items()
+        ]
+        empty_row = mark_safe(self._render_row(widget_id, "", ""))
+        # JSON is embedded in a script, so escape HTML delimiters without changing data.
+        metadata = json.dumps(options).translate({ord("<"): r"\u003C", ord(">"): r"\u003E", ord("&"): r"\u0026"})
+        return mark_safe(
+            render_to_string(
+                "admin/widgets/config.html",
+                {
+                    "name": name,
+                    "widget_id": widget_id,
+                    "datalist_options": mark_safe(
+                        "\n".join(
+                            f'<option value="{self._escape(key)}">{self._escape(opt["description"][:60] or opt["type"])}</option>'
+                            for key, opt in sorted(options.items())
+                        ),
+                    ),
+                    "config_meta_json": metadata,
+                    "rows": [*rows, empty_row],
+                    "empty_row": empty_row,
+                },
+            ),
         )
 
-        # Build config metadata as JSON for JS
-        config_meta_json = json.dumps(config_options)
-
-        html = f'''
-        <div id="{widget_id}_container" class="key-value-editor" style="width: 100%; max-width: none;">
-            <datalist id="{widget_id}_keys">
-                {datalist_options}
-            </datalist>
-            <div id="{widget_id}_rows" class="key-value-rows">
-        '''
-
-        # Render existing key-value pairs
-        for key, val in data.items():
-            val_str = json.dumps(val) if not isinstance(val, str) else val
-            html += self._render_row(widget_id, key, val_str)
-
-        # Always add one empty row for new entries
-        html += self._render_row(widget_id, "", "")
-
-        html += f'''
-            </div>
-            <div style="display: flex; gap: 8px; align-items: center; margin-top: 8px;">
-                <button type="button" onclick="addKeyValueRow_{widget_id}()"
-                        style="padding: 4px 12px; cursor: pointer; background: #417690; color: white; border: none; border-radius: 4px;">
-                    + Add Row
-                </button>
-            </div>
-            <input type="hidden" name="{name}" id="{widget_id}" value="">
-            <script>
-                (function() {{
-                    var configMeta_{widget_id} = {config_meta_json};
-                    var rowCounter_{widget_id} = 0;
-
-                    function stringifyValue_{widget_id}(value) {{
-                        return typeof value === 'string' ? value : JSON.stringify(value);
-                    }}
-
-                    function getTypes_{widget_id}(meta) {{
-                        if (!meta || meta.type === undefined || meta.type === null) {{
-                            return [];
-                        }}
-                        return Array.isArray(meta.type) ? meta.type : [meta.type];
-                    }}
-
-                    function getMetaForKey_{widget_id}(key) {{
-                        if (!key) {{
-                            return null;
-                        }}
-
-                        var explicitMeta = configMeta_{widget_id}[key];
-                        if (explicitMeta) {{
-                            return Object.assign({{ key: key }}, explicitMeta);
-                        }}
-
-                        if (key.endsWith('_BINARY')) {{
-                            return {{
-                                key: key,
-                                plugin: 'custom',
-                                type: 'string',
-                                default: '',
-                                description: 'Path to binary executable',
-                            }};
-                        }}
-
-                        if (isRegexConfigKey_{widget_id}(key)) {{
-                            return {{
-                                key: key,
-                                plugin: 'custom',
-                                type: 'string',
-                                default: '',
-                                description: 'Regex pattern list',
-                            }};
-                        }}
-
-                        return null;
-                    }}
-
-                    function describeMeta_{widget_id}(meta) {{
-                        if (!meta) {{
-                            return '';
-                        }}
-
-                        var details = '';
-                        if (Array.isArray(meta.enum) && meta.enum.length) {{
-                            details = 'Allowed: ' + meta.enum.map(stringifyValue_{widget_id}).join(', ');
-                        }} else {{
-                            var types = getTypes_{widget_id}(meta);
-                            if (types.length) {{
-                                details = 'Expected: ' + types.join(' or ');
-                            }}
-                        }}
-
-                        if (meta.minimum !== undefined || meta.maximum !== undefined) {{
-                            var bounds = [];
-                            if (meta.minimum !== undefined) bounds.push('min ' + meta.minimum);
-                            if (meta.maximum !== undefined) bounds.push('max ' + meta.maximum);
-                            details += (details ? ' ' : '') + '(' + bounds.join(', ') + ')';
-                        }}
-
-                        return [meta.description || '', details].filter(Boolean).join(' ');
-                    }}
-
-                    function getExampleInput_{widget_id}(key, meta) {{
-                        var types = getTypes_{widget_id}(meta);
-                        if (key.endsWith('_BINARY')) {{
-                            return 'Example: wget or /usr/bin/wget';
-                        }}
-                        if (key.endsWith('_ARGS_EXTRA') || key.endsWith('_ARGS')) {{
-                            return 'Example: ["--extra-arg"]';
-                        }}
-                        if (types.includes('array')) {{
-                            return 'Example: ["value"]';
-                        }}
-                        if (types.includes('object')) {{
-                            return 'Example: {{"key": "value"}}';
-                        }}
-                        return '';
-                    }}
-
-                    function isRegexConfigKey_{widget_id}(key) {{
-                        return key === 'URL_ALLOWLIST' ||
-                            key === 'URL_DENYLIST' ||
-                            key.endsWith('_PATTERN') ||
-                            key.includes('REGEX');
-                    }}
-
-                    function isSimpleFilterPattern_{widget_id}(pattern) {{
-                        return /^[\\w.*:-]+$/.test(pattern);
-                    }}
-
-                    function validateRegexPattern_{widget_id}(pattern) {{
-                        if (!pattern || isSimpleFilterPattern_{widget_id}(pattern)) {{
-                            return '';
-                        }}
-
-                        try {{
-                            new RegExp(pattern);
-                        }} catch (error) {{
-                            return error && error.message ? error.message : 'Invalid regex';
-                        }}
-                        return '';
-                    }}
-
-                    function validateRegexConfig_{widget_id}(key, raw, typeName) {{
-                        if (typeName === 'object') {{
-                            var parsed;
-                            try {{
-                                parsed = JSON.parse(raw);
-                            }} catch (error) {{
-                                return {{ ok: false, value: raw, message: 'Must be valid JSON' }};
-                            }}
-                            if (!parsed || typeof parsed !== 'object' || Array.isArray(parsed)) {{
-                                return {{ ok: false, value: parsed, message: 'Must be a JSON object' }};
-                            }}
-                            for (var regexKey in parsed) {{
-                                var objectRegexError = validateRegexPattern_{widget_id}(regexKey);
-                                if (objectRegexError) {{
-                                    return {{ ok: false, value: parsed, message: 'Invalid regex key "' + regexKey + '": ' + objectRegexError }};
-                                }}
-                            }}
-                            return {{ ok: true, value: parsed, message: '' }};
-                        }}
-
-                        var patterns = raw.split(/[\\n,]+/).map(function(pattern) {{
-                            return pattern.trim();
-                        }}).filter(Boolean);
-                        for (var i = 0; i < patterns.length; i++) {{
-                            var regexError = validateRegexPattern_{widget_id}(patterns[i]);
-                            if (regexError) {{
-                                return {{ ok: false, value: raw, message: 'Invalid regex "' + patterns[i] + '": ' + regexError }};
-                            }}
-                        }}
-                        return {{ ok: true, value: raw, message: '' }};
-                    }}
-
-                    function validateBinaryValue_{widget_id}(raw) {{
-                        if (!raw) {{
-                            return {{ ok: true, value: raw, message: '' }};
-                        }}
-
-                        if (/['"`]/.test(raw)) {{
-                            return {{ ok: false, value: raw, message: 'Binary paths cannot contain quotes' }};
-                        }}
-
-                        if (/[;&|<>$(){{}}\\[\\]!]/.test(raw)) {{
-                            return {{ ok: false, value: raw, message: 'Binary paths can only be a binary name or absolute path' }};
-                        }}
-
-                        if (raw.startsWith('/')) {{
-                            if (/^[A-Za-z0-9_./+\\- ]+$/.test(raw)) {{
-                                return {{ ok: true, value: raw, message: '' }};
-                            }}
-                            return {{ ok: false, value: raw, message: 'Absolute paths may only contain path-safe characters' }};
-                        }}
-
-                        if (/^[A-Za-z0-9_.+-]+$/.test(raw)) {{
-                            return {{ ok: true, value: raw, message: '' }};
-                        }}
-
-                        return {{ ok: false, value: raw, message: 'Enter a binary name like wget or an absolute path like /usr/bin/wget' }};
-                    }}
-
-                    function parseValue_{widget_id}(raw) {{
-                        try {{
-                            if (raw === 'true') return true;
-                            if (raw === 'false') return false;
-                            if (raw === 'null') return null;
-                            if (raw !== '' && !isNaN(raw)) return Number(raw);
-                            if ((raw.startsWith('{{') && raw.endsWith('}}')) ||
-                                (raw.startsWith('[') && raw.endsWith(']')) ||
-                                (raw.startsWith('"') && raw.endsWith('"'))) {{
-                                return JSON.parse(raw);
-                            }}
-                        }} catch (error) {{
-                            return raw;
-                        }}
-                        return raw;
-                    }}
-
-                    function sameValue_{widget_id}(left, right) {{
-                        return left === right || JSON.stringify(left) === JSON.stringify(right);
-                    }}
-
-                    function parseTypedValue_{widget_id}(raw, typeName, meta) {{
-                        var numberValue;
-                        var parsed;
-
-                        if (typeName && meta && meta.key && isRegexConfigKey_{widget_id}(meta.key)) {{
-                            return validateRegexConfig_{widget_id}(meta.key, raw, typeName);
-                        }}
-
-                        if (typeName === 'string' && meta && meta.key && meta.key.endsWith('_BINARY')) {{
-                            return validateBinaryValue_{widget_id}(raw);
-                        }}
-
-                        if (typeName === 'string') {{
-                            if (meta.pattern) {{
-                                try {{
-                                    if (!(new RegExp(meta.pattern)).test(raw)) {{
-                                        return {{ ok: false, value: raw, message: 'Must match pattern ' + meta.pattern }};
-                                    }}
-                                }} catch (error) {{}}
-                            }}
-                            return {{ ok: true, value: raw, message: '' }};
-                        }}
-
-                        if (typeName === 'integer') {{
-                            if (!/^-?\\d+$/.test(raw)) {{
-                                return {{ ok: false, value: raw, message: 'Must be an integer' }};
-                            }}
-                            numberValue = Number(raw);
-                            if (meta.minimum !== undefined && numberValue < meta.minimum) {{
-                                return {{ ok: false, value: numberValue, message: 'Must be at least ' + meta.minimum }};
-                            }}
-                            if (meta.maximum !== undefined && numberValue > meta.maximum) {{
-                                return {{ ok: false, value: numberValue, message: 'Must be at most ' + meta.maximum }};
-                            }}
-                            return {{ ok: true, value: numberValue, message: '' }};
-                        }}
-
-                        if (typeName === 'number') {{
-                            if (raw === '' || isNaN(raw)) {{
-                                return {{ ok: false, value: raw, message: 'Must be a number' }};
-                            }}
-                            numberValue = Number(raw);
-                            if (meta.minimum !== undefined && numberValue < meta.minimum) {{
-                                return {{ ok: false, value: numberValue, message: 'Must be at least ' + meta.minimum }};
-                            }}
-                            if (meta.maximum !== undefined && numberValue > meta.maximum) {{
-                                return {{ ok: false, value: numberValue, message: 'Must be at most ' + meta.maximum }};
-                            }}
-                            return {{ ok: true, value: numberValue, message: '' }};
-                        }}
-
-                        if (typeName === 'boolean') {{
-                            var lowered = raw.toLowerCase();
-                            if (lowered === 'true' || raw === '1') return {{ ok: true, value: true, message: '' }};
-                            if (lowered === 'false' || raw === '0') return {{ ok: true, value: false, message: '' }};
-                            return {{ ok: false, value: raw, message: 'Must be true or false' }};
-                        }}
-
-                        if (typeName === 'null') {{
-                            return raw === 'null'
-                                ? {{ ok: true, value: null, message: '' }}
-                                : {{ ok: false, value: raw, message: 'Must be null' }};
-                        }}
-
-                        if (typeName === 'array' || typeName === 'object') {{
-                            try {{
-                                parsed = JSON.parse(raw);
-                            }} catch (error) {{
-                                return {{ ok: false, value: raw, message: 'Must be valid JSON' }};
-                            }}
-
-                            if (typeName === 'array' && Array.isArray(parsed)) {{
-                                return {{ ok: true, value: parsed, message: '' }};
-                            }}
-                            if (typeName === 'object' && parsed && typeof parsed === 'object' && !Array.isArray(parsed)) {{
-                                return {{ ok: true, value: parsed, message: '' }};
-                            }}
-
-                            return {{
-                                ok: false,
-                                value: parsed,
-                                message: typeName === 'array' ? 'Must be a JSON array' : 'Must be a JSON object',
-                            }};
-                        }}
-
-                        return {{ ok: true, value: parseValue_{widget_id}(raw), message: '' }};
-                    }}
-
-                    function validateValueAgainstMeta_{widget_id}(raw, meta) {{
-                        if (!meta || raw === '') {{
-                            return {{ state: 'neutral', value: raw, message: '' }};
-                        }}
-
-                        var enumValues = Array.isArray(meta.enum) ? meta.enum : [];
-                        var types = getTypes_{widget_id}(meta);
-                        if (!types.length) {{
-                            types = ['string'];
-                        }}
-
-                        var error = 'Invalid value';
-                        for (var i = 0; i < types.length; i++) {{
-                            var candidate = parseTypedValue_{widget_id}(raw, types[i], meta);
-                            if (!candidate.ok) {{
-                                error = candidate.message || error;
-                                continue;
-                            }}
-                            if (enumValues.length && !enumValues.some(function(enumValue) {{
-                                return sameValue_{widget_id}(enumValue, candidate.value) || stringifyValue_{widget_id}(enumValue) === raw;
-                            }})) {{
-                                error = 'Must be one of: ' + enumValues.map(stringifyValue_{widget_id}).join(', ');
-                                continue;
-                            }}
-                            return {{ state: 'valid', value: candidate.value, message: '' }};
-                        }}
-
-                        return {{ state: 'invalid', value: raw, message: error }};
-                    }}
-
-                    function ensureRowId_{widget_id}(row) {{
-                        if (!row.dataset.rowId) {{
-                            row.dataset.rowId = String(rowCounter_{widget_id}++);
-                        }}
-                        return row.dataset.rowId;
-                    }}
-
-                    function setRowHelp_{widget_id}(row) {{
-                        var keyInput = row.querySelector('.kv-key');
-                        var help = row.querySelector('.kv-help');
-                        if (!keyInput || !help) {{
-                            return;
-                        }}
-
-                        var key = keyInput.value.trim();
-                        if (!key) {{
-                            help.textContent = '';
-                            return;
-                        }}
-
-                        var meta = getMetaForKey_{widget_id}(key);
-                        if (meta) {{
-                            var extra = isRegexConfigKey_{widget_id}(key)
-                                ? ((meta.type === 'object' || (Array.isArray(meta.type) && meta.type.includes('object')))
-                                    ? ' Expected: JSON object with regex keys.'
-                                    : ' Expected: valid regex.')
-                                : '';
-                            var example = getExampleInput_{widget_id}(key, meta);
-                            help.textContent = [describeMeta_{widget_id}(meta) + extra, example].filter(Boolean).join(' ');
-                        }} else {{
-                            help.textContent = 'Custom key';
-                        }}
-                    }}
-
-                    function configureValueInput_{widget_id}(row) {{
-                        var keyInput = row.querySelector('.kv-key');
-                        var valueInput = row.querySelector('.kv-value');
-                        var datalist = row.querySelector('.kv-value-options');
-                        if (!keyInput || !valueInput || !datalist) {{
-                            return;
-                        }}
-
-                        var rowId = ensureRowId_{widget_id}(row);
-                        datalist.id = '{widget_id}_value_options_' + rowId;
-
-                        var meta = getMetaForKey_{widget_id}(keyInput.value.trim());
-                        var enumValues = Array.isArray(meta && meta.enum) ? meta.enum : [];
-                        var types = getTypes_{widget_id}(meta);
-                        if (!enumValues.length && types.includes('boolean')) {{
-                            enumValues = ['True', 'False'];
-                        }}
-                        if (enumValues.length) {{
-                            datalist.innerHTML = enumValues.map(function(enumValue) {{
-                                return '<option value="' + stringifyValue_{widget_id}(enumValue).replace(/"/g, '&quot;') + '"></option>';
-                            }}).join('');
-                            valueInput.setAttribute('list', datalist.id);
-                        }} else {{
-                            datalist.innerHTML = '';
-                            valueInput.removeAttribute('list');
-                        }}
-                    }}
-
-                    function setValueValidationState_{widget_id}(input, state, message) {{
-                        if (!input) {{
-                            return;
-                        }}
-
-                        if (state === 'valid') {{
-                            input.style.borderColor = '#2da44e';
-                            input.style.boxShadow = '0 0 0 1px rgba(45, 164, 78, 0.18)';
-                            input.style.backgroundColor = '#f6ffed';
-                        }} else if (state === 'invalid') {{
-                            input.style.borderColor = '#cf222e';
-                            input.style.boxShadow = '0 0 0 1px rgba(207, 34, 46, 0.18)';
-                            input.style.backgroundColor = '#fff8f8';
-                        }} else {{
-                            input.style.borderColor = '#ccc';
-                            input.style.boxShadow = 'none';
-                            input.style.backgroundColor = '';
-                        }}
-                        input.title = message || '';
-                    }}
-
-                    function applyValueValidation_{widget_id}(row) {{
-                        var keyInput = row.querySelector('.kv-key');
-                        var valueInput = row.querySelector('.kv-value');
-                        if (!keyInput || !valueInput) {{
-                            return;
-                        }}
-
-                        var key = keyInput.value.trim();
-                        if (!key) {{
-                            setValueValidationState_{widget_id}(valueInput, 'neutral', '');
-                            return;
-                        }}
-
-                        var meta = getMetaForKey_{widget_id}(key);
-                        if (!meta) {{
-                            setValueValidationState_{widget_id}(valueInput, 'neutral', '');
-                            return;
-                        }}
-
-                        var validation = validateValueAgainstMeta_{widget_id}(valueInput.value.trim(), meta);
-                        setValueValidationState_{widget_id}(valueInput, validation.state, validation.message);
-                    }}
-
-                    function coerceValueForStorage_{widget_id}(key, raw) {{
-                        var meta = getMetaForKey_{widget_id}(key);
-                        if (!meta) {{
-                            return parseValue_{widget_id}(raw);
-                        }}
-
-                        var validation = validateValueAgainstMeta_{widget_id}(raw, meta);
-                        return validation.state === 'valid' ? validation.value : raw;
-                    }}
-
-                    function initializeRows_{widget_id}() {{
-                        var container = document.getElementById('{widget_id}_rows');
-                        container.querySelectorAll('.key-value-row').forEach(function(row) {{
-                            ensureRowId_{widget_id}(row);
-                            configureValueInput_{widget_id}(row);
-                            setRowHelp_{widget_id}(row);
-                            applyValueValidation_{widget_id}(row);
-                        }});
-                    }}
-
-                    function updateHiddenField_{widget_id}() {{
-                        var container = document.getElementById('{widget_id}_rows');
-                        var rows = container.querySelectorAll('.key-value-row');
-                        var result = {{}};
-                        rows.forEach(function(row) {{
-                            var keyInput = row.querySelector('.kv-key');
-                            var valInput = row.querySelector('.kv-value');
-                            if (keyInput && valInput && keyInput.value.trim()) {{
-                                var key = keyInput.value.trim();
-                                var val = valInput.value.trim();
-                                result[key] = coerceValueForStorage_{widget_id}(key, val);
-                            }}
-                        }});
-                        document.getElementById('{widget_id}').value = JSON.stringify(result);
-                    }}
-
-                    window.addKeyValueRow_{widget_id} = function() {{
-                        var container = document.getElementById('{widget_id}_rows');
-                        var newRow = document.createElement('div');
-                        newRow.className = 'key-value-row';
-                        newRow.style.cssText = 'margin-bottom: 6px;';
-                        newRow.innerHTML = '<div class="kv-inputs" style="display: flex; gap: 8px; align-items: center;">' +
-                            '<input type="text" class="kv-key" placeholder="KEY" list="{widget_id}_keys" ' +
-                            'style="flex: 1; padding: 6px 8px; border: 1px solid #ccc; border-radius: 4px; font-family: monospace; font-size: 12px;">' +
-                            '<input type="text" class="kv-value" placeholder="value" ' +
-                            'style="flex: 2; padding: 6px 8px; border: 1px solid #ccc; border-radius: 4px; font-family: monospace; font-size: 12px;">' +
-                            '<datalist class="kv-value-options"></datalist>' +
-                            '<button type="button" onclick="removeKeyValueRow_{widget_id}(this)" ' +
-                            'style="padding: 4px 10px; cursor: pointer; background: #ba2121; color: white; border: none; border-radius: 4px; font-weight: bold;">−</button>' +
-                            '</div>' +
-                            '<div class="kv-help" style="margin-top: 4px; font-size: 11px; color: #666; font-style: italic;"></div>';
-                        container.appendChild(newRow);
-                        ensureRowId_{widget_id}(newRow);
-                        configureValueInput_{widget_id}(newRow);
-                        setRowHelp_{widget_id}(newRow);
-                        applyValueValidation_{widget_id}(newRow);
-                        updateHiddenField_{widget_id}();
-                        newRow.querySelector('.kv-key').focus();
-                    }};
-
-                    window.removeKeyValueRow_{widget_id} = function(btn) {{
-                        var row = btn.closest('.key-value-row');
-                        row.remove();
-                        updateHiddenField_{widget_id}();
-                    }};
-
-                    window.updateHiddenField_{widget_id} = updateHiddenField_{widget_id};
-
-                    function configRowForKey_{widget_id}(key) {{
-                        var container = document.getElementById('{widget_id}_rows');
-                        if (!container) {{
-                            return null;
-                        }}
-                        var match = null;
-                        container.querySelectorAll('.key-value-row').forEach(function(row) {{
-                            if (match) {{ return; }}
-                            var keyInput = row.querySelector('.kv-key');
-                            if (keyInput && keyInput.value.trim() === key) {{
-                                match = row;
-                            }}
-                        }});
-                        if (!match) {{
-                            window.addKeyValueRow_{widget_id}();
-                            var rows = container.querySelectorAll('.key-value-row');
-                            match = rows[rows.length - 1];
-                            var keyInput = match.querySelector('.kv-key');
-                            if (keyInput) {{
-                                keyInput.value = key;
-                                keyInput.dispatchEvent(new Event('input', {{ bubbles: true }}));
-                            }}
-                        }}
-                        return match;
-                    }}
-
-                    function prefillConfigFromQuery_{widget_id}() {{
-                        var params = new URLSearchParams(window.location.search);
-                        var consumedConfigKeys = [];
-                        params.forEach(function(value, key) {{
-                            if (!/^[A-Z][A-Z0-9_]*$/.test(key) || !configMeta_{widget_id}[key]) {{
-                                return;
-                            }}
-                            consumedConfigKeys.push(key);
-                            var match = configRowForKey_{widget_id}(key);
-                            var valueInput = match && match.querySelector('.kv-value');
-                            if (valueInput) {{
-                                valueInput.value = value;
-                                valueInput.dispatchEvent(new Event('input', {{ bubbles: true }}));
-                            }}
-                        }});
-                        updateHiddenField_{widget_id}();
-                        if (consumedConfigKeys.length) {{
-                            var cleanUrl = new URL(window.location.href);
-                            consumedConfigKeys.forEach(function(key) {{ cleanUrl.searchParams.delete(key); }});
-                            window.history.replaceState(null, '', cleanUrl.pathname + cleanUrl.search + cleanUrl.hash);
-                        }}
-                    }}
-
-                    function focusConfigKeyFromHash_{widget_id}() {{
-                        // Deep-link affordance: ``…/change/#SOME_KEY`` jumps directly
-                        // to (or creates) the matching row in this editor. Used by
-                        // the setup wizard and live-config detail pages.
-                        var hash = (window.location.hash || '').replace(/^#/, '').trim();
-                        if (!hash || !/^[A-Z][A-Z0-9_]*$/.test(hash)) {{
-                            return;
-                        }}
-                        var match = configRowForKey_{widget_id}(hash);
-                        if (!match) {{
-                            return;
-                        }}
-                        match.scrollIntoView({{ behavior: 'smooth', block: 'center' }});
-                        var prevOutline = match.style.outline;
-                        match.style.outline = '2px solid #f59e0b';
-                        match.style.outlineOffset = '2px';
-                        match.style.transition = 'outline 1.2s ease-out';
-                        setTimeout(function() {{
-                            match.style.outline = prevOutline || 'none';
-                        }}, 1400);
-                        var valueInput = match.querySelector('.kv-value');
-                        if (valueInput) {{
-                            valueInput.focus();
-                            try {{ valueInput.setSelectionRange(valueInput.value.length, valueInput.value.length); }} catch (e) {{}}
-                        }}
-                    }}
-
-                    // Initialize on load
-                    document.addEventListener('DOMContentLoaded', function() {{
-                        initializeRows_{widget_id}();
-                        prefillConfigFromQuery_{widget_id}();
-                        updateHiddenField_{widget_id}();
-                        focusConfigKeyFromHash_{widget_id}();
-                    }});
-                    // Also run immediately in case DOM is already ready
-                    if (document.readyState !== 'loading') {{
-                        initializeRows_{widget_id}();
-                        prefillConfigFromQuery_{widget_id}();
-                        updateHiddenField_{widget_id}();
-                        focusConfigKeyFromHash_{widget_id}();
-                    }}
-
-                    window.addEventListener('hashchange', focusConfigKeyFromHash_{widget_id});
-
-                    // Update on any input change
-                    var rowsEl_{widget_id} = document.getElementById('{widget_id}_rows');
-
-                    rowsEl_{widget_id}.addEventListener('input', function(event) {{
-                        var row = event.target.closest('.key-value-row');
-                        if (!row) {{
-                            return;
-                        }}
-
-                        if (event.target.classList.contains('kv-key')) {{
-                            configureValueInput_{widget_id}(row);
-                            setRowHelp_{widget_id}(row);
-                        }}
-
-                        if (event.target.classList.contains('kv-key') || event.target.classList.contains('kv-value')) {{
-                            applyValueValidation_{widget_id}(row);
-                            updateHiddenField_{widget_id}();
-                        }}
-                    }});
-                }})();
-            </script>
-        </div>
-        '''
-        return mark_safe(html)
-
     def _render_row(self, widget_id: str, key: str, value: str) -> str:
         from archivebox.config.common import is_sensitive_config_key
 
@@ -872,6 +269,28 @@ class BaseModelAdmin(DjangoObjectActions, admin.ModelAdmin):
     show_search_mode_selector = False
     change_form_template = "admin/archivebox_change_form.html"
 
+    @admin.display(description="Health", ordering="health")
+    def health_display(self, obj):
+        h = obj.health
+        color = "green" if h >= 80 else "orange" if h >= 50 else "red"
+        return format_html('<span style="color: {};">{}</span>', color, h)
+
+    def get_ordering_fields(self, request):
+        ordering = request.GET.get("o")
+        if not ordering:
+            return set()
+        fields = set()
+        for part in ordering.split("."):
+            if not part:
+                continue
+            try:
+                idx = abs(int(part)) - 1
+            except ValueError:
+                continue
+            if 0 <= idx < len(self.list_display):
+                fields.add(self.list_display[idx])
+        return fields
+
     def get_admin_toolbar_actions(self, request, obj):
         """Return extra action button dicts for the shared change-form toolbar.
 

archivebox/base_models/models.py

@@ -4,7 +4,7 @@ __package__ = "archivebox.base_models"
 
 import json
 import shutil
-from typing import Any
+from typing import TYPE_CHECKING, Any
 
 from archivebox.uuid_compat import CompactUUIDField, uuid7
 from pathlib import Path
@@ -23,6 +23,10 @@ from django_stubs_ext.db.models import TypedModelMeta
 from archivebox.config import CONSTANTS
 
 
+if TYPE_CHECKING:
+    from archivebox.config.common import ArchiveBoxBaseConfig
+
+
 def normalize_config_json_values(config: Any) -> Any:
     if not isinstance(config, dict):
         return config
@@ -55,17 +59,6 @@ def get_or_create_system_user_pk(username="system"):
     return user.pk
 
 
-class AutoDateTimeField(models.DateTimeField):
-    """DateTimeField that automatically updates on save (legacy compatibility)."""
-
-    def pre_save(self, model_instance, add):
-        if add or self.attname not in model_instance.__dict__ or not model_instance.__dict__[self.attname]:
-            value = timezone.now()
-            setattr(model_instance, self.attname, value)
-            return value
-        return super().pre_save(model_instance, add)
-
-
 class ModelWithUUID(models.Model):
     id = CompactUUIDField(primary_key=True, default=uuid7, editable=False, unique=True)
     created_at = models.DateTimeField(default=timezone.now, db_index=True)
@@ -138,6 +131,7 @@ class ModelWithHealthStats(models.Model):
 class ModelWithConfig(models.Model):
     """Mixin for models with a JSON config field."""
 
+    _runtime_config: "ArchiveBoxBaseConfig"
     config = models.JSONField(default=dict, null=True, blank=True, editable=True)
 
     class Meta(TypedModelMeta):
@@ -284,16 +278,20 @@ class ModelWithOutputDir(ModelWithUUID):
             elif path.is_dir():
                 shutil.rmtree(path, ignore_errors=True)
 
+    def schedule_delete_cleanup(self, *, using: str | None = None) -> None:
+        """Capture output paths before DB deletion and remove them after commit."""
+        paths = self.validate_output_paths_for_delete(self.output_paths_for_delete())
+        transaction.on_commit(lambda: self.delete_output_paths(paths), using=using)
+
     @classmethod
     def register_delete_signal(cls) -> None:
         if cls._delete_signal_registered:
             return
 
-        def schedule_output_dir_cleanup(sender, instance, **kwargs):
+        def schedule_output_dir_cleanup(sender, instance, using, **kwargs):
             if not isinstance(instance, ModelWithOutputDir):
                 return
-            paths = instance.validate_output_paths_for_delete(instance.output_paths_for_delete())
-            transaction.on_commit(lambda paths=paths: instance.delete_output_paths(paths))
+            instance.schedule_delete_cleanup(using=using)
 
         pre_delete.connect(
             schedule_output_dir_cleanup,

archivebox/cli/__init__.py

@@ -186,6 +186,10 @@ def cli(ctx, help=False):
 
 
 def main(args=None, prog_name=None):
+    from archivebox.config.constants import CONSTANTS
+
+    os.environ.setdefault("ABX_PLUGINS_DIR", str(CONSTANTS.USER_PLUGINS_DIR))
+
     # show `docker run archivebox xyz` in help messages if running in docker
     IN_DOCKER = os.environ.get("IN_DOCKER", False) in ("1", "true", "True", "TRUE", "yes")
     IS_TTY = sys.stdin.isatty()

archivebox/cli/archivebox_add.py

@@ -6,7 +6,6 @@ __command__ = "archivebox add"
 
 import sys
 import os
-import json
 from pathlib import Path
 
 from typing import Any, TYPE_CHECKING
@@ -71,17 +70,7 @@ def add(
     created_by_id: int | None = None,
     config: dict[str, Any] | None = None,
 ) -> tuple[Crawl, QuerySet[Snapshot]]:
-    """Add a new URL or list of URLs to your archive.
-
-    The flow is:
-    1. Save URLs to sources file
-    2. Create Crawl with URLs and max_depth
-    3. Crawl runner creates Snapshots from Crawl URLs (depth=0)
-    4. Crawl runner runs parser extractors on root snapshots
-    5. Parser extractors output to urls.jsonl
-    6. URLs are added to Crawl.urls and child Snapshots are created
-    7. Repeat until max_depth is reached
-    """
+    """Add a URL list or imported URL document to a new Crawl."""
 
     from rich import print
 
@@ -125,14 +114,25 @@ def add(
     created_by_id = created_by_id or get_or_create_system_user_pk()
     started_at = timezone.now()
 
-    use_internal_input_root = isinstance(urls, str)
-    source_text = (
-        urls
-        if use_internal_input_root
-        else "\n".join(json.dumps({"type": "CrawlSeed", "url": str(url), "depth": 0}, separators=(",", ":")) for url in urls)
-    )
+    if isinstance(urls, str):
+        source_text = urls
+    else:
+        from archivebox.misc.util import validate_url
+
+        source_text = "\n".join(validate_url(str(url)) for url in urls)
 
     # 2. Create a new Crawl with inline URLs
+    # Foreground add must claim runner ownership before publishing runnable
+    # work. Otherwise an existing server runner can lease the new Crawl in the
+    # gap between Crawl.objects.create() and current_command(), then be killed
+    # by the newer add process halfway through its first Snapshot hook.
+    command = None
+    if not bg and not index_only:
+        from archivebox.machine.models import Process
+        from archivebox.core.takeover_util import current_command
+
+        command = current_command(Process.TypeChoices.ADD, data_dir=CONSTANTS.DATA_DIR)
+
     cli_args = [*sys.argv]
     if cli_args[0].lower().endswith("archivebox"):
         cli_args[0] = "archivebox"
@@ -173,30 +173,32 @@ def add(
     # this add path. Keeping it byte-for-byte readable is what lets API/UI/CLI
     # callers audit or resume imports without losing RSS/Netscape/JSON metadata
     # that is not representable as one plain URL per line.
-    crawl = Crawl.objects.create(
-        urls=source_text,
-        # Stdin/import text gets an extra hop because the synthetic
-        # archivebox://internal root lives at depth 0 and parser-discovered
-        # URLs land at depth 1; direct URL args become the depth=0 input
-        # snapshots themselves so --depth=N matches the deepest hop the user
-        # asked for.
-        max_depth=depth + 1 if use_internal_input_root else depth,
-        tags_str=tag,
-        persona_id=persona_obj.id,
-        label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
-        created_by_id=created_by_id,
-        status=Crawl.StatusChoices.QUEUED,
-        retry_at=None if index_only else timezone.now(),
-        config=crawl_config,
-    )
+    try:
+        crawl = Crawl.objects.create(
+            urls=source_text,
+            max_depth=depth,
+            tags_str=tag,
+            persona_id=persona_obj.id,
+            label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
+            created_by_id=created_by_id,
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=None if index_only else timezone.now(),
+            config=crawl_config,
+        )
+        first_url = crawl.get_urls_list()[0] if crawl.get_urls_list() else ""
+    except BaseException:
+        if command is not None:
+            command.mark_exited(exit_code=1)
+        raise
+    if command is not None:
+        command.url = first_url
+        command.save(update_fields=["url", "modified_at"])
 
     print(f"[green]\\[+] Created Crawl {crawl.id} with max_depth={depth}[/green]")
-    first_url = crawl.get_urls_list()[0] if crawl.get_urls_list() else ""
     print(f"    [dim]First URL: {first_url}[/dim]")
 
-    # 3. The CrawlMachine will create Snapshots from all URLs when started
-    #    Parser extractors run on snapshots and discover more URLs
-    #    Discovered URLs become child Snapshots (depth+1)
+    # The runner parses Crawl.urls after claiming the Crawl, then persists the
+    # resulting depth-0 Snapshot facts. Recursive discoveries start at depth 1.
 
     if index_only:
         print("[yellow]\\[*] Index-only mode - URLs queued, runner not started[/yellow]")
@@ -220,17 +222,27 @@ def add(
     else:
         # Foreground mode: run full crawl runner until all work is done
         print("[green]\\[*] Starting crawl runner to process crawl...[/green]")
-        from archivebox.machine.models import Process
-        from archivebox.core.takeover_util import command_owns_foreground_runner, current_command, standby_until_foreground_runner_needed
+        from archivebox.core.takeover_util import command_owns_foreground_runner, standby_until_foreground_runner_needed
         from archivebox.workers.supervisord_util import get_existing_supervisord_process, run_runner_worker, stop_own_supervisord_process
 
-        command = current_command(Process.TypeChoices.ADD, data_dir=CONSTANTS.DATA_DIR, url=first_url)
+        assert command is not None
         exit_code = 0
+
+        def crawl_is_complete() -> bool:
+            crawl.refresh_from_db(fields=["status"])
+            return crawl.status not in crawl.RUNNABLE_STATES
+
         try:
             try:
                 with foreground_shutdown_signals(first_signal_message=None), foreground_parent_watchdog():
                     while True:
-                        standby_until_foreground_runner_needed(command, data_dir=CONSTANTS.DATA_DIR)
+                        standby = standby_until_foreground_runner_needed(
+                            command,
+                            data_dir=CONSTANTS.DATA_DIR,
+                            work_is_complete=crawl_is_complete,
+                        )
+                        if standby["work_completed"]:
+                            break
                         exit_code = run_runner_worker(
                             ["--crawl-id", str(crawl.id)],
                             name=f"worker_runner_add_{os.getpid()}",
@@ -376,6 +388,10 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
+        if not kwargs.get("index_only"):
+            from archivebox.workers.supervisord_util import require_crawl_memory
+
+            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.
@@ -388,13 +404,13 @@ def main(**kwargs):
         if only_new is not None:
             kwargs["config"] = {"ONLY_NEW": bool(only_new)}
         if extract:
-            from abx_dl.models import discover_plugins, plugins_matching_output
+            from archivebox.plugins.discovery import get_plugin_catalog
 
-            all_plugins = discover_plugins()
+            catalog = get_plugin_catalog()
             tokens = [token.strip() for token in extract.split(",") if token.strip()]
-            plugin_names = {name.lower(): name for name in all_plugins}
+            plugin_names = {name.lower(): name for name in catalog}
             selected = [plugin_names[token.lower()] for token in tokens if token.lower() in plugin_names]
-            selected += plugins_matching_output(all_plugins, tokens)
+            selected += catalog.matching_output(tokens)
             if not selected:
                 raise click.UsageError(f"No plugins found matching extract types: {extract}")
             existing = [token.strip() for token in (kwargs.get("plugins") or "").split(",") if token.strip()]

archivebox/cli/archivebox_archiveresult.py

@@ -6,13 +6,13 @@ archivebox archiveresult <action> [args...] [--filters]
 Manage ArchiveResult records (plugin extraction results).
 
 Actions:
-    create  - Create ArchiveResults for Snapshots (queue extractions)
+    create  - Emit plugin extraction request records for Snapshots
     list    - List ArchiveResults as JSONL (with optional filters)
     update  - Update ArchiveResults from stdin JSONL
     delete  - Delete ArchiveResults from stdin JSONL
 
 Examples:
-    # Create ArchiveResults for snapshots (queue for extraction)
+    # Emit extraction requests; `archivebox run` schedules their parent snapshots
     archivebox snapshot list --status=queued | archivebox archiveresult create
     archivebox archiveresult create --plugin=screenshot --snapshot-id=<uuid>
 
@@ -20,9 +20,6 @@ Examples:
     archivebox archiveresult list --status=failed
     archivebox archiveresult list --plugin=screenshot --status=succeeded
 
-    # Update (reset failed extractions to queued)
-    archivebox archiveresult list --status=failed | archivebox archiveresult update --status=queued
-
     # Delete
     archivebox archiveresult list --plugin=singlefile | archivebox archiveresult delete --yes
 
@@ -75,7 +72,7 @@ def create_archiveresults(
     """
     from archivebox.config.common import get_config
     from archivebox.plugins.hooks import discover_hooks
-    from archivebox.misc.jsonl import read_stdin, write_record, TYPE_SNAPSHOT, TYPE_ARCHIVERESULT
+    from archivebox.misc.jsonl import read_stdin, write_record, TYPE_SNAPSHOT
     from archivebox.core.models import Snapshot
 
     is_tty = sys.stdout.isatty()
@@ -102,26 +99,9 @@ def create_archiveresults(
         for record in records:
             record_type = record.get("type", "")
 
-            if record_type == TYPE_SNAPSHOT:
-                # Pass through the Snapshot record itself
+            if record_type:
                 pass_through_records.append(record)
-                if record.get("id"):
-                    snapshot_ids.append(record["id"])
-
-            elif record_type == TYPE_ARCHIVERESULT:
-                # ArchiveResult records: pass through if they have an id
-                if record.get("id"):
-                    pass_through_records.append(record)
-                # If no id, we could create it, but for now just pass through
-                else:
-                    pass_through_records.append(record)
-
-            elif record_type:
-                # Other typed records (Crawl, Tag, etc): pass through
-                pass_through_records.append(record)
-
-            elif record.get("id"):
-                # Untyped record with id - assume it's a snapshot ID
+            if record.get("id") and record_type in ("", TYPE_SNAPSHOT):
                 snapshot_ids.append(record["id"])
 
         # Output pass-through records first
@@ -144,21 +124,20 @@ def create_archiveresults(
 
     created_count = 0
     for snapshot in snapshots:
-        if plugin:
+        config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
+        hooks = [
+            hook
+            for hook in discover_hooks("Snapshot", filter_disabled=not plugin, config=config)
+            if not plugin or hook.parent.name == plugin
+        ]
+        for hook_path in hooks:
+            hook_name = hook_path.stem
+            plugin_name = hook_path.parent.name
             if not is_tty:
-                write_record(build_archiveresult_request(snapshot.id, plugin, status=status))
+                write_record(build_archiveresult_request(snapshot.id, plugin_name, hook_name=hook_name, status=status))
             created_count += 1
-        else:
-            config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
-            hooks = discover_hooks("Snapshot", config=config)
-            for hook_path in hooks:
-                hook_name = hook_path.stem
-                plugin_name = hook_path.parent.name
-                if not is_tty:
-                    write_record(build_archiveresult_request(snapshot.id, plugin_name, hook_name=hook_name, status=status))
-                created_count += 1
 
-    rprint(f"[green]Created {created_count} archive result request records[/green]", file=sys.stderr)
+    rprint(f"[green]Created {created_count} extraction request records[/green]", file=sys.stderr)
     return 0
 
 
@@ -173,48 +152,20 @@ def list_archiveresults(
     snapshot_id: str | None = None,
     limit: int | None = None,
 ) -> int:
-    """
-    List ArchiveResults as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+    """List archive results as JSONL, or formatted rows in a terminal."""
     from archivebox.core.models import ArchiveResult
+    from archivebox.cli.cli_util import list_records, format_status
 
-    is_tty = sys.stdout.isatty()
-
-    queryset = ArchiveResult.objects.all().order_by("-start_ts")
-
-    # Apply filters
-    filter_kwargs = {
-        "status": status,
-        "plugin": plugin,
-        "snapshot_id": snapshot_id,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for result in queryset:
-        if is_tty:
-            status_color = {
-                "queued": "yellow",
-                "started": "blue",
-                "succeeded": "green",
-                "failed": "red",
-                "skipped": "dim",
-                "noresults": "dim",
-                "backoff": "magenta",
-            }.get(result.status, "dim")
-            rprint(
-                f"[{status_color}]{result.status:10}[/{status_color}] {result.plugin:15} [dim]{result.id}[/dim] {result.snapshot.url[:40]}",
-            )
-        else:
-            write_record(result.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} archive results[/dim]", file=sys.stderr)
-    return 0
+    queryset = apply_filters(
+        ArchiveResult.objects.order_by("-start_ts"),
+        {"status": status, "plugin": plugin, "snapshot_id": snapshot_id},
+        limit=limit,
+    )
+    return list_records(
+        queryset,
+        plural="archive results",
+        render=lambda result: f"{format_status(result.status, 10)} {result.plugin:15} [dim]{result.id}[/dim] {result.snapshot.url[:40]}",
+    )
 
 
 # =============================================================================
@@ -222,54 +173,17 @@ def list_archiveresults(
 # =============================================================================
 
 
-def update_archiveresults(
-    status: str | None = None,
-) -> int:
-    """
-    Update ArchiveResults from stdin JSONL.
-
-    Reads ArchiveResult records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
-    from archivebox.misc.jsonl import read_stdin, write_record
+def update_archiveresults(status: str | None = None) -> int:
+    """Apply supplied fields to each JSONL-selected ArchiveResult."""
     from archivebox.core.models import ArchiveResult
+    from archivebox.cli.cli_util import update_records
 
-    is_tty = sys.stdout.isatty()
+    def update(archiveresult):
+        if status:
+            archiveresult.status = status
+        archiveresult.save()
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    updated_count = 0
-    for record in records:
-        result_id = record.get("id")
-        if not result_id:
-            continue
-
-        try:
-            result = ArchiveResult.objects.get(id=result_id)
-
-            # Apply updates from CLI flags
-            if status:
-                result.status = status
-
-            result.save()
-            updated_count += 1
-
-            if not is_tty:
-                write_record(result.to_json())
-
-        except ArchiveResult.DoesNotExist:
-            rprint(f"[yellow]ArchiveResult not found: {result_id}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} archive results[/green]", file=sys.stderr)
-    return 0
+    return update_records(ArchiveResult, update, plural="archive results")
 
 
 # =============================================================================
@@ -278,52 +192,19 @@ def update_archiveresults(
 
 
 def delete_archiveresults(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete ArchiveResults from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
+    """Delete archive results selected by stdin JSONL; --yes confirms, --dry-run previews."""
+    from archivebox.cli.cli_util import delete_records
     from archivebox.core.models import ArchiveResult
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    result_ids = [r.get("id") for r in records if r.get("id")]
-
-    if not result_ids:
-        rprint("[yellow]No valid archive result IDs in input[/yellow]", file=sys.stderr)
-        return 1
-
-    results = ArchiveResult.objects.filter(id__in=result_ids)
-    count = results.count()
-
-    if count == 0:
-        rprint("[yellow]No matching archive results found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} archive results (dry run)[/yellow]", file=sys.stderr)
-        for result in results[:10]:
-            rprint(f"  [dim]{result.id}[/dim] {result.plugin} {result.snapshot.url[:40]}", file=sys.stderr)
-        if count > 10:
-            rprint(f"  ... and {count - 10} more", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Perform deletion
-    deleted_count, _ = results.delete()
-    rprint(f"[green]Deleted {deleted_count} archive results[/green]", file=sys.stderr)
-    return 0
+    return delete_records(
+        ArchiveResult,
+        label="archive result",
+        plural="archive results",
+        preview=lambda obj: f"[dim]{obj.id}[/dim] {obj.plugin} {obj.snapshot.url[:40]}",
+        yes=yes,
+        dry_run=dry_run,
+        preview_limit=10,
+    )
 
 
 # =============================================================================
@@ -341,9 +222,9 @@ def main():
 @click.option("--snapshot-id", help="Snapshot ID to create results for")
 @click.option("--plugin", "-p", help="Plugin name (e.g., screenshot, singlefile)")
 @click.option("--status", "-s", default="queued", help="Initial status (default: queued)")
-def create_cmd(snapshot_id: str | None, plugin: str | None, status: str):
-    """Create ArchiveResults for Snapshots from stdin JSONL."""
-    sys.exit(create_archiveresults(snapshot_id=snapshot_id, plugin=plugin, status=status))
+def create_cmd(**kwargs):
+    """Emit Snapshot plugin extraction requests as JSONL."""
+    sys.exit(create_archiveresults(**kwargs))
 
 
 @main.command("list")
@@ -351,36 +232,24 @@ def create_cmd(snapshot_id: str | None, plugin: str | None, status: str):
 @click.option("--plugin", "-p", help="Filter by plugin name")
 @click.option("--snapshot-id", help="Filter by snapshot ID")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(
-    status: str | None,
-    plugin: str | None,
-    snapshot_id: str | None,
-    limit: int | None,
-):
+def list_cmd(**kwargs):
     """List ArchiveResults as JSONL."""
-    sys.exit(
-        list_archiveresults(
-            status=status,
-            plugin=plugin,
-            snapshot_id=snapshot_id,
-            limit=limit,
-        ),
-    )
+    sys.exit(list_archiveresults(**kwargs))
 
 
 @main.command("update")
 @click.option("--status", "-s", help="Set status")
-def update_cmd(status: str | None):
+def update_cmd(**kwargs):
     """Update ArchiveResults from stdin JSONL."""
-    sys.exit(update_archiveresults(status=status))
+    sys.exit(update_archiveresults(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete ArchiveResults from stdin JSONL."""
-    sys.exit(delete_archiveresults(yes=yes, dry_run=dry_run))
+    sys.exit(delete_archiveresults(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_binary.py

@@ -114,37 +114,20 @@ def list_binaries(
     version__icontains: str | None = None,
     limit: int | None = None,
 ) -> int:
-    """
-    List Binaries as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+    """List binaries as JSONL, or formatted rows in a terminal."""
     from archivebox.machine.models import Binary
+    from archivebox.cli.cli_util import list_records
 
-    is_tty = sys.stdout.isatty()
-
-    queryset = Binary.objects.all().order_by("name", "-modified_at", "-created_at")
-
-    # Apply filters
-    filter_kwargs = {
-        "name": name,
-        "abspath__icontains": abspath__icontains,
-        "version__icontains": version__icontains,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for binary in queryset:
-        if is_tty:
-            rprint(f"[cyan]{binary.name:20}[/cyan] [dim]{binary.version:15}[/dim] {binary.abspath}")
-        else:
-            write_record(binary.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} binaries[/dim]", file=sys.stderr)
-    return 0
+    queryset = apply_filters(
+        Binary.objects.order_by("name", "-modified_at", "-created_at"),
+        {"name": name, "abspath__icontains": abspath__icontains, "version__icontains": version__icontains},
+        limit=limit,
+    )
+    return list_records(
+        queryset,
+        plural="binaries",
+        render=lambda binary: f"[cyan]{binary.name:20}[/cyan] [dim]{binary.version:15}[/dim] {binary.abspath}",
+    )
 
 
 # =============================================================================
@@ -152,57 +135,19 @@ def list_binaries(
 # =============================================================================
 
 
-def update_binaries(
-    version: str | None = None,
-    abspath: str | None = None,
-) -> int:
-    """
-    Update Binaries from stdin JSONL.
-
-    Reads Binary records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
-    from archivebox.misc.jsonl import read_stdin, write_record
+def update_binaries(version: str | None = None, abspath: str | None = None) -> int:
+    """Apply supplied fields to each JSONL-selected Binary."""
     from archivebox.machine.models import Binary
+    from archivebox.cli.cli_util import update_records
 
-    is_tty = sys.stdout.isatty()
+    def update(binary):
+        if version:
+            binary.version = version
+        if abspath:
+            binary.abspath = abspath
+        binary.save()
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    updated_count = 0
-    for record in records:
-        binary_id = record.get("id")
-        if not binary_id:
-            continue
-
-        try:
-            binary = Binary.objects.get(id=binary_id)
-
-            # Apply updates from CLI flags
-            if version:
-                binary.version = version
-            if abspath:
-                binary.abspath = abspath
-
-            binary.save()
-            updated_count += 1
-
-            if not is_tty:
-                write_record(binary.to_json())
-
-        except Binary.DoesNotExist:
-            rprint(f"[yellow]Binary not found: {binary_id}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} binaries[/green]", file=sys.stderr)
-    return 0
+    return update_records(Binary, update, plural="binaries")
 
 
 # =============================================================================
@@ -211,50 +156,18 @@ def update_binaries(
 
 
 def delete_binaries(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete Binaries from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
+    """Delete binaries selected by stdin JSONL; --yes confirms, --dry-run previews."""
+    from archivebox.cli.cli_util import delete_records
     from archivebox.machine.models import Binary
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    binary_ids = [r.get("id") for r in records if r.get("id")]
-
-    if not binary_ids:
-        rprint("[yellow]No valid binary IDs in input[/yellow]", file=sys.stderr)
-        return 1
-
-    binaries = Binary.objects.filter(id__in=binary_ids)
-    count = binaries.count()
-
-    if count == 0:
-        rprint("[yellow]No matching binaries found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} binaries (dry run)[/yellow]", file=sys.stderr)
-        for binary in binaries:
-            rprint(f"  {binary.name} {binary.abspath}", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Perform deletion
-    deleted_count, _ = binaries.delete()
-    rprint(f"[green]Deleted {deleted_count} binaries[/green]", file=sys.stderr)
-    return 0
+    return delete_records(
+        Binary,
+        label="binary",
+        plural="binaries",
+        preview=lambda obj: f"{obj.name} {obj.abspath}",
+        yes=yes,
+        dry_run=dry_run,
+    )
 
 
 # =============================================================================
@@ -272,9 +185,9 @@ def main():
 @click.option("--name", "-n", required=True, help="Binary name (e.g., chrome, wget)")
 @click.option("--abspath", "-p", required=True, help="Absolute path to binary")
 @click.option("--version", "-v", default="", help="Binary version")
-def create_cmd(name: str, abspath: str, version: str):
+def create_cmd(**kwargs):
     """Create/register a Binary."""
-    sys.exit(create_binary(name=name, abspath=abspath, version=version))
+    sys.exit(create_binary(**kwargs))
 
 
 @main.command("list")
@@ -282,37 +195,25 @@ def create_cmd(name: str, abspath: str, version: str):
 @click.option("--abspath__icontains", help="Filter by path contains")
 @click.option("--version__icontains", help="Filter by version contains")
 @click.option("--limit", type=int, help="Limit number of results")
-def list_cmd(
-    name: str | None,
-    abspath__icontains: str | None,
-    version__icontains: str | None,
-    limit: int | None,
-):
+def list_cmd(**kwargs):
     """List Binaries as JSONL."""
-    sys.exit(
-        list_binaries(
-            name=name,
-            abspath__icontains=abspath__icontains,
-            version__icontains=version__icontains,
-            limit=limit,
-        ),
-    )
+    sys.exit(list_binaries(**kwargs))
 
 
 @main.command("update")
 @click.option("--version", "-v", help="Set version")
 @click.option("--abspath", "-p", help="Set path")
-def update_cmd(version: str | None, abspath: str | None):
+def update_cmd(**kwargs):
     """Update Binaries from stdin JSONL."""
-    sys.exit(update_binaries(version=version, abspath=abspath))
+    sys.exit(update_binaries(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete Binaries from stdin JSONL."""
-    sys.exit(delete_binaries(yes=yes, dry_run=dry_run))
+    sys.exit(delete_binaries(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_crawl.py

@@ -177,43 +177,20 @@ def list_crawls(
     max_depth: int | None = None,
     limit: int | None = None,
 ) -> int:
-    """
-    List Crawls as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+    """List crawls as JSONL, or formatted rows in a terminal."""
     from archivebox.crawls.models import Crawl
+    from archivebox.cli.cli_util import list_records, format_status
 
-    is_tty = sys.stdout.isatty()
-
-    queryset = Crawl.objects.all().order_by("-created_at")
-
-    # Apply filters
-    filter_kwargs = {
-        "status": status,
-        "urls__icontains": urls__icontains,
-        "max_depth": max_depth,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for crawl in queryset:
-        if is_tty:
-            status_color = {
-                "queued": "yellow",
-                "started": "blue",
-                "sealed": "green",
-            }.get(crawl.status, "dim")
-            url_preview = crawl.urls[:50].replace("\n", " ")
-            rprint(f"[{status_color}]{crawl.status:8}[/{status_color}] [dim]{crawl.id}[/dim] {url_preview}...")
-        else:
-            write_record(crawl.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} crawls[/dim]", file=sys.stderr)
-    return 0
+    queryset = apply_filters(
+        Crawl.objects.order_by("-created_at"),
+        {"status": status, "urls__icontains": urls__icontains, "max_depth": max_depth},
+        limit=limit,
+    )
+    return list_records(
+        queryset,
+        plural="crawls",
+        render=lambda crawl: f"{format_status(crawl.status, 8)} [dim]{crawl.id}[/dim] {crawl.urls[:50].replace(chr(10), chr(32))}...",
+    )
 
 
 # =============================================================================
@@ -221,73 +198,26 @@ def list_crawls(
 # =============================================================================
 
 
-def update_crawls(
-    status: str | None = None,
-    max_depth: int | None = None,
-) -> int:
-    """
-    Update Crawls from stdin JSONL.
-
-    Reads Crawl records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
+def update_crawls(status: str | None = None, max_depth: int | None = None) -> int:
+    """Update JSONL-selected crawls through their lifecycle operations."""
     from django.utils import timezone
-
-    from archivebox.misc.jsonl import read_stdin, write_record
     from archivebox.crawls.models import Crawl
+    from archivebox.cli.cli_util import update_records, update_record_status
 
-    is_tty = sys.stdout.isatty()
+    def update(crawl):
+        if status:
+            try:
+                update_record_status(crawl, status)
+            except ValueError as err:
+                rprint(f"[red]{err}[/red]", file=sys.stderr)
+                return False
+        if max_depth is not None:
+            crawl.safe_update({"max_depth": max_depth, "modified_at": timezone.now()}, refresh=False)
+            crawl.max_depth = max_depth
+        elif not status:
+            crawl.safe_update({"modified_at": timezone.now()}, refresh=False)
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    updated_count = 0
-    for record in records:
-        crawl_id = record.get("id")
-        if not crawl_id:
-            continue
-
-        try:
-            crawl = Crawl.objects.get(id=crawl_id)
-
-            if status:
-                if status not in Crawl.StatusChoices.values:
-                    rprint(f"[red]Invalid crawl status: {status}[/red]", file=sys.stderr)
-                    continue
-                if status == Crawl.StatusChoices.SEALED:
-                    crawl.cancel()
-                elif status == Crawl.StatusChoices.PAUSED:
-                    crawl.pause()
-                elif status == Crawl.StatusChoices.QUEUED:
-                    if crawl.status == Crawl.StatusChoices.PAUSED:
-                        crawl.resume()
-                    else:
-                        crawl.update_and_requeue(status=Crawl.StatusChoices.QUEUED, retry_at=timezone.now())
-                elif status == Crawl.StatusChoices.STARTED:
-                    crawl.update_and_requeue(status=Crawl.StatusChoices.STARTED, retry_at=timezone.now())
-            if max_depth is not None:
-                crawl.safe_update({"max_depth": max_depth, "modified_at": timezone.now()}, refresh=False)
-                crawl.max_depth = max_depth
-            elif not status:
-                crawl.safe_update({"modified_at": timezone.now()}, refresh=False)
-            updated_count += 1
-
-            if not is_tty:
-                crawl.refresh_from_db()
-                write_record(crawl.to_json())
-
-        except Crawl.DoesNotExist:
-            rprint(f"[yellow]Crawl not found: {crawl_id}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} crawls[/green]", file=sys.stderr)
-    return 0
+    return update_records(Crawl, update, plural="crawls", refresh=True)
 
 
 # =============================================================================
@@ -296,51 +226,18 @@ def update_crawls(
 
 
 def delete_crawls(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete Crawls from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
+    """Delete crawls selected by stdin JSONL; --yes confirms, --dry-run previews."""
+    from archivebox.cli.cli_util import delete_records
     from archivebox.crawls.models import Crawl
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    crawl_ids = [r.get("id") for r in records if r.get("id")]
-
-    if not crawl_ids:
-        rprint("[yellow]No valid crawl IDs in input[/yellow]", file=sys.stderr)
-        return 1
-
-    crawls = Crawl.objects.filter(id__in=crawl_ids)
-    count = crawls.count()
-
-    if count == 0:
-        rprint("[yellow]No matching crawls found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} crawls (dry run)[/yellow]", file=sys.stderr)
-        for crawl in crawls:
-            url_preview = crawl.urls[:50].replace("\n", " ")
-            rprint(f"  [dim]{crawl.id}[/dim] {url_preview}...", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Perform deletion
-    deleted_count, _ = crawls.delete()
-    rprint(f"[green]Deleted {deleted_count} crawls[/green]", file=sys.stderr)
-    return 0
+    return delete_records(
+        Crawl,
+        label="crawl",
+        plural="crawls",
+        preview=lambda obj: f"[dim]{obj.id}[/dim] {obj.urls[:50].replace(chr(10), chr(32))}...",
+        yes=yes,
+        dry_run=dry_run,
+    )
 
 
 # =============================================================================
@@ -369,37 +266,25 @@ def create_cmd(urls: tuple, depth: int, tag: str, status: str):
 @click.option("--urls__icontains", help="Filter by URLs contains")
 @click.option("--max-depth", type=int, help="Filter by max depth")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(
-    status: str | None,
-    urls__icontains: str | None,
-    max_depth: int | None,
-    limit: int | None,
-):
+def list_cmd(**kwargs):
     """List Crawls as JSONL."""
-    sys.exit(
-        list_crawls(
-            status=status,
-            urls__icontains=urls__icontains,
-            max_depth=max_depth,
-            limit=limit,
-        ),
-    )
+    sys.exit(list_crawls(**kwargs))
 
 
 @main.command("update")
 @click.option("--status", "-s", help="Set status")
 @click.option("--max-depth", type=int, help="Set max depth")
-def update_cmd(status: str | None, max_depth: int | None):
+def update_cmd(**kwargs):
     """Update Crawls from stdin JSONL."""
-    sys.exit(update_crawls(status=status, max_depth=max_depth))
+    sys.exit(update_crawls(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete Crawls from stdin JSONL."""
-    sys.exit(delete_crawls(yes=yes, dry_run=dry_run))
+    sys.exit(delete_crawls(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_extract.py

@@ -1,95 +1,135 @@
 #!/usr/bin/env python3
 
-"""
-archivebox extract [snapshot_ids...] [--plugins=NAMES]
-
-Run plugins on Snapshots. Accepts snapshot IDs as arguments, from stdin, or via JSONL.
-
-Input formats:
-    - Snapshot UUIDs (one per line)
-    - JSONL: {"type": "Snapshot", "id": "...", "url": "..."}
-    - JSONL: {"type": "ArchiveResult", "snapshot_id": "...", "plugin": "..."}
-
-Output (JSONL):
-    {"type": "ArchiveResult", "id": "...", "snapshot_id": "...", "plugin": "...", "status": "..."}
-
-Examples:
-    # Extract specific snapshot
-    archivebox extract 01234567-89ab-cdef-0123-456789abcdef
-
-    # Pipe from snapshot command
-    archivebox snapshot https://example.com | archivebox extract
-
-    # Run specific plugins only
-    archivebox extract --plugins=screenshot,singlefile 01234567-89ab-cdef-0123-456789abcdef
-
-    # Chain commands
-    archivebox crawl https://example.com | archivebox snapshot | archivebox extract
-"""
+"""Run abx-dl snapshot hooks for existing ArchiveBox snapshots."""
 
 __package__ = "archivebox.cli"
 __command__ = "archivebox extract"
 
 import sys
 from collections import defaultdict
-from itertools import product
+from contextlib import redirect_stdout
 
 import rich_click as click
 
 
-def process_archiveresult_by_id(archiveresult_id: str) -> int:
-    """
-    Re-run extraction for a single ArchiveResult by ID.
+def _resolve_requests(records: list[dict], plugins: str) -> tuple[dict[str, set[str]], set[str]]:
+    """Resolve CLI records to snapshot-level execution requests.
 
-    ArchiveResults are projected status rows, not queued work items. Re-running
-    a single result means resetting that row and queueing its parent snapshot
-    through the shared crawl runner with the corresponding plugin selected.
+    ArchiveResult input is accepted as a convenient reference to its parent
+    Snapshot and plugin. It is never reset or converted into queued work.
     """
+    from archivebox.misc.db import uuid_ref_query
+    from archivebox.core.models import ArchiveResult, Snapshot
+    from archivebox.misc.jsonl import TYPE_ARCHIVERESULT
+
+    explicit_plugins = {name.strip() for name in plugins.split(",") if name.strip()}
+    requested: dict[str, set[str]] = defaultdict(set)
+    missing: set[str] = set()
+
+    for record in records:
+        record_type = record.get("type")
+        record_id = str(record.get("id") or "")
+
+        if record_type == TYPE_ARCHIVERESULT and record.get("snapshot_id"):
+            requested[str(record["snapshot_id"])].update(explicit_plugins or {str(record.get("plugin") or "")})
+            requested[str(record["snapshot_id"])].discard("")
+            continue
+
+        if record_type == TYPE_ARCHIVERESULT:
+            result = ArchiveResult.objects.filter(uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
+            if result is not None:
+                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
+                continue
+
+        snapshot_id = str(record.get("snapshot_id") or record_id or "")
+        snapshot = Snapshot.objects.filter(id=snapshot_id).only("id").first() if snapshot_id else None
+        if snapshot is None and record.get("url"):
+            snapshot = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").only("id").first()
+        # Bare UUID CLI arguments are parsed as Snapshot records because the
+        # input layer cannot know which model owns them. Preserve the public
+        # convenience of passing an ArchiveResult ID by trying that reference
+        # only after the Snapshot lookup misses.
+        if snapshot is None and record_id and not record.get("url"):
+            result = ArchiveResult.objects.filter(uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
+            if result is not None:
+                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
+                continue
+        if snapshot is None:
+            missing.add(snapshot_id or str(record.get("url") or ""))
+            continue
+        requested[str(snapshot.id)].update(explicit_plugins)
+
+    return requested, missing
+
+
+def _run_snapshot_requests(requested: dict[str, set[str]], *, wait: bool, show_progress: bool) -> int:
+    from django.utils import timezone
     from rich import print as rprint
-    from archivebox.core.models import ArchiveResult
-    from archivebox.api.v1_core import _uuid_ref_query
+
+    from archivebox.core.models import Snapshot
     from archivebox.services.runner import run_crawl
 
-    try:
-        archiveresult = ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
-    except ArchiveResult.DoesNotExist:
-        rprint(f"[red]ArchiveResult {archiveresult_id} not found[/red]", file=sys.stderr)
+    snapshots = {str(snapshot.id): snapshot for snapshot in Snapshot.objects.filter(id__in=requested).select_related("crawl")}
+    if not snapshots:
+        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
         return 1
 
-    rprint(f"[blue]Extracting {archiveresult.plugin} for {archiveresult.snapshot.url}[/blue]", file=sys.stderr)
+    from archivebox.config.common import get_config
+    from archivebox.plugins.discovery import get_enabled_plugins
 
-    try:
-        archiveresult.reset_for_retry()
-        snapshot = archiveresult.snapshot
-        snapshot.queue_for_extraction()
-        crawl = snapshot.crawl
-        if not crawl.claim_processing_lock(lock_seconds=10):
-            rprint(
-                f"[yellow]Crawl {crawl.id} is already owned by another runner[/yellow]",
-                file=sys.stderr,
-            )
-            return 1
+    for snapshot_id, plugin_names in requested.items():
+        snapshot = snapshots.get(snapshot_id)
+        if snapshot is not None and not plugin_names:
+            plugin_names.update(get_enabled_plugins(config=get_config(crawl=snapshot.crawl, snapshot=snapshot)))
 
-        run_crawl(str(snapshot.crawl_id), snapshot_ids=[str(snapshot.id)], selected_plugins=[archiveresult.plugin])
-        archiveresult.refresh_from_db()
+    if wait and any("search_backend_sonic" in plugin_names for plugin_names in requested.values()):
+        from archivebox.core.takeover_util import ensure_daemon_stack
 
-        if archiveresult.status == ArchiveResult.StatusChoices.SUCCEEDED:
-            print(f"[green]Extraction succeeded: {archiveresult.output_str}[/green]")
-            return 0
-        elif archiveresult.status == ArchiveResult.StatusChoices.NORESULTS:
-            print(f"[dim]Extraction completed with no results: {archiveresult.output_str}[/dim]")
-            return 0
-        elif archiveresult.status == ArchiveResult.StatusChoices.FAILED:
-            print(f"[red]Extraction failed: {archiveresult.output_str}[/red]", file=sys.stderr)
-            return 1
-        else:
-            # Still in progress or backoff - not a failure
-            print(f"[yellow]Extraction status: {archiveresult.status}[/yellow]")
-            return 0
+        with redirect_stdout(sys.stderr):
+            ensure_daemon_stack(reason="Sonic snapshot indexing")
 
-    except Exception as e:
-        print(f"[red]Extraction error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-        return 1
+    # Explicit extraction resumes open/paused snapshots at the snapshot level.
+    # Sealed snapshots stay sealed during targeted maintenance backfills.
+    if wait:
+        for snapshot in snapshots.values():
+            if snapshot.status != Snapshot.StatusChoices.SEALED:
+                snapshot.update_and_requeue(
+                    status=Snapshot.StatusChoices.QUEUED,
+                    retry_at=timezone.now(),
+                )
+
+    if not wait:
+        for snapshot_id, plugin_names in requested.items():
+            snapshot = snapshots.get(snapshot_id)
+            if snapshot is None:
+                continue
+            if plugin_names:
+                snapshot.schedule_plugin_run(plugin_names, when=timezone.now())
+            elif snapshot.status == Snapshot.StatusChoices.SEALED:
+                snapshot.update_and_requeue(retry_at=timezone.now())
+            else:
+                snapshot.update_and_requeue(
+                    status=Snapshot.StatusChoices.QUEUED,
+                    retry_at=timezone.now(),
+                )
+        if show_progress:
+            rprint(f"[blue]Queued {len(snapshots)} snapshots for extraction[/blue]", file=sys.stderr)
+        return 0
+
+    grouped: dict[tuple[str, tuple[str, ...]], list[str]] = defaultdict(list)
+    for snapshot_id, plugin_names in requested.items():
+        snapshot = snapshots.get(snapshot_id)
+        if snapshot is not None:
+            grouped[(str(snapshot.crawl_id), tuple(sorted(plugin_names)))].append(snapshot_id)
+
+    for (crawl_id, plugin_names), snapshot_ids in grouped.items():
+        run_crawl(
+            crawl_id,
+            snapshot_ids=sorted(snapshot_ids),
+            selected_plugins=list(plugin_names) or None,
+            show_progress=show_progress,
+        )
+    return 0
 
 
 def run_plugins(
@@ -99,366 +139,59 @@ def run_plugins(
     wait: bool = True,
     emit_results: bool = True,
     show_progress: bool = True,
-    preserve_queued: bool = False,
 ) -> int:
-    """
-    Run plugins on Snapshots from input.
-
-    Reads Snapshot IDs or JSONL from args/stdin, runs plugins, outputs JSONL.
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
+    """Execute selected plugins through the snapshot-level runner."""
     from rich import print as rprint
-    from django.utils import timezone
 
-    from archivebox.misc.jsonl import (
-        read_args_or_stdin,
-        write_record,
-        TYPE_SNAPSHOT,
-        TYPE_ARCHIVERESULT,
-    )
     from archivebox.core.models import Snapshot
-    from archivebox.core.models import ArchiveResult
-    from archivebox.services.runner import run_crawl
-    from abx_dl.models import discover_plugins
+    from archivebox.misc.jsonl import read_args_or_stdin, write_record
 
-    is_tty = sys.stdout.isatty()
-
-    # Parse comma-separated plugins list once (reused in creation and filtering)
-    plugins_list = [p.strip() for p in plugins.split(",") if p.strip()] if plugins else []
-
-    # Parse stdin/args exactly once per CLI invocation.
-    # `main()` may already have consumed stdin to distinguish Snapshot input from
-    # ArchiveResult IDs; if so, it must pass the parsed records through here
-    # instead of asking this helper to reread an already-drained pipe.
     if records is None:
         records = list(read_args_or_stdin(args))
-
     if not records:
         rprint("[yellow]No snapshots provided. Pass snapshot IDs as arguments or via stdin.[/yellow]", file=sys.stderr)
         return 1
 
-    # Gather snapshot IDs and optional plugin constraints to process
-    snapshot_ids = set()
-    requested_plugins_by_snapshot: dict[str, set[str]] = defaultdict(set)
-    for record in records:
-        record_type = record.get("type")
-
-        if record_type == TYPE_SNAPSHOT:
-            snapshot_id = record.get("id")
-            if snapshot_id:
-                snapshot_ids.add(str(snapshot_id))
-            elif record.get("url"):
-                # Look up by URL (get most recent if multiple exist)
-                snap = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").first()
-                if snap:
-                    snapshot_ids.add(str(snap.id))
-                else:
-                    rprint(f"[yellow]Snapshot not found for URL: {record['url']}[/yellow]", file=sys.stderr)
-
-        elif record_type == TYPE_ARCHIVERESULT:
-            snapshot_id = record.get("snapshot_id")
-            if snapshot_id:
-                snapshot_ids.add(str(snapshot_id))
-                plugin_name = record.get("plugin")
-                if plugin_name and not plugins_list:
-                    requested_plugins_by_snapshot[str(snapshot_id)].add(str(plugin_name))
-
-        elif "id" in record:
-            # Assume it's a snapshot ID
-            snapshot_ids.add(str(record["id"]))
-
-    if not snapshot_ids:
-        rprint("[red]No valid snapshot IDs found in input[/red]", file=sys.stderr)
+    requested, missing = _resolve_requests(records, plugins)
+    for value in sorted(missing):
+        rprint(f"[yellow]Snapshot or ArchiveResult not found: {value}[/yellow]", file=sys.stderr)
+    if not requested:
         return 1
 
-    existing_snapshots = list(Snapshot.objects.filter(id__in=snapshot_ids).values_list("id", "crawl_id"))
-    existing_snapshot_ids = {str(snapshot_id) for snapshot_id, _crawl_id in existing_snapshots}
-    existing_crawl_ids = {str(crawl_id) for _snapshot_id, crawl_id in existing_snapshots}
-    missing_snapshot_ids = sorted(str(snapshot_id) for snapshot_id in snapshot_ids - existing_snapshot_ids)
-    for snapshot_id in missing_snapshot_ids:
-        rprint(f"[yellow]Snapshot {snapshot_id} not found[/yellow]", file=sys.stderr)
-
-    # Queue only the target plugin rows. Bulk updates keep large reindex runs
-    # from doing one SELECT+UPDATE per snapshot/plugin before hooks even start.
-    requested_pairs: set[tuple[str, str]] = set()
-    if plugins_list:
-        requested_pairs.update((snapshot_id, plugin_name) for snapshot_id, plugin_name in product(existing_snapshot_ids, plugins_list))
-    else:
-        requested_pairs.update(
-            (snapshot_id, plugin_name)
-            for snapshot_id, plugin_names in requested_plugins_by_snapshot.items()
-            if snapshot_id in existing_snapshot_ids
-            for plugin_name in plugin_names
-        )
-    plugins_by_name = discover_plugins(runtime="archivebox")
-    requested_rows: set[tuple[str, str, str]] = set()
-    for snapshot_id, plugin_name in requested_pairs:
-        plugin = plugins_by_name.get(plugin_name)
-        hooks = plugin.filter_hooks("Snapshot") if plugin is not None else []
-        if hooks:
-            requested_rows.update((snapshot_id, plugin_name, hook.name) for hook in hooks)
-        else:
-            requested_rows.add((snapshot_id, plugin_name, ""))
-
-    queued_rows: set[tuple[str, str, str]] = set()
-    if preserve_queued and requested_rows:
-        queued_rows = {
-            (str(snapshot_id), plugin_name, hook_name)
-            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
-                snapshot_id__in=existing_snapshot_ids,
-                plugin__in={plugin_name for _snapshot_id, plugin_name, _hook_name in requested_rows},
-                status=ArchiveResult.StatusChoices.QUEUED,
-            ).values_list("snapshot_id", "plugin", "hook_name")
-        }
-    rows_to_queue = requested_rows - queued_rows
-
-    reset_fields = {
-        "status": ArchiveResult.StatusChoices.QUEUED,
-        "output_str": "",
-        "output_json": None,
-        "output_files": {},
-        "output_size": 0,
-        "output_mimetypes": "",
-        "start_ts": None,
-        "end_ts": None,
-        "modified_at": timezone.now(),
-    }
-    if rows_to_queue and plugins_list:
-        rows_to_reset_by_hook: dict[tuple[str, str], set[str]] = defaultdict(set)
-        for snapshot_id, plugin_name, hook_name in rows_to_queue:
-            rows_to_reset_by_hook[(plugin_name, hook_name)].add(snapshot_id)
-        for (plugin_name, hook_name), plugin_snapshot_ids in rows_to_reset_by_hook.items():
-            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name, hook_name=hook_name).update(
-                **reset_fields,
-            )
-    elif rows_to_queue and requested_plugins_by_snapshot:
-        snapshot_ids_by_hook: dict[tuple[str, str], set[str]] = defaultdict(set)
-        for snapshot_id, plugin_name, hook_name in rows_to_queue:
-            snapshot_ids_by_hook[(plugin_name, hook_name)].add(snapshot_id)
-        for (plugin_name, hook_name), plugin_snapshot_ids in snapshot_ids_by_hook.items():
-            ArchiveResult.objects.filter(snapshot_id__in=plugin_snapshot_ids, plugin=plugin_name, hook_name=hook_name).update(
-                **reset_fields,
-            )
-    existing_rows = (
-        {
-            (str(snapshot_id), plugin_name, hook_name)
-            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
-                snapshot_id__in=existing_snapshot_ids,
-                plugin__in={plugin_name for _snapshot_id, plugin_name, _hook_name in rows_to_queue},
-            ).values_list("snapshot_id", "plugin", "hook_name")
-        }
-        if rows_to_queue
-        else set()
-    )
-    missing_rows = rows_to_queue - existing_rows
-    if missing_rows:
-        ArchiveResult.objects.bulk_create(
-            [
-                ArchiveResult(
-                    snapshot_id=snapshot_id,
-                    plugin=plugin_name,
-                    hook_name=hook_name,
-                    status=ArchiveResult.StatusChoices.QUEUED,
-                )
-                for snapshot_id, plugin_name, hook_name in sorted(missing_rows)
-            ],
-            batch_size=500,
-        )
-
-    processed_count = len(existing_snapshot_ids)
-    queue_at = timezone.now()
-    if existing_snapshot_ids:
-        if requested_rows:
-            # Search indexing on a sealed Snapshot is the only targeted hook
-            # allowed to bypass the normal lifecycle. Every other requested
-            # plugin requeues its Snapshot through the unified state machine.
-            affected_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in rows_to_queue}
-            if preserve_queued and queued_rows:
-                queued_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in queued_rows}
-                affected_snapshot_ids.update(
-                    str(snapshot_id)
-                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids)
-                    .filter(retry_at__gt=queue_at)
-                    .values_list("id", flat=True)
-                )
-                affected_snapshot_ids.update(
-                    str(snapshot_id)
-                    for snapshot_id in Snapshot.objects.filter(id__in=queued_snapshot_ids, retry_at__isnull=True).values_list(
-                        "id",
-                        flat=True,
-                    )
-                )
-            requested_plugins_by_id: dict[str, set[str]] = defaultdict(set)
-            for snapshot_id, plugin_name, _hook_name in requested_rows:
-                requested_plugins_by_id[snapshot_id].add(plugin_name)
-            for snapshot in Snapshot.objects.filter(id__in=affected_snapshot_ids).only("id", "status", "modified_at"):
-                # Guard the read-time status so we never bump retry_at on a
-                # row that's been re-queued / started by a concurrent runner.
-                plugin_names = requested_plugins_by_id.get(str(snapshot.id), set())
-                sealed_search_backfill = (
-                    snapshot.status == Snapshot.StatusChoices.SEALED
-                    and plugin_names
-                    and all(plugin_name.startswith("search_backend_") for plugin_name in plugin_names)
-                )
-                if sealed_search_backfill:
-                    snapshot.safe_update(
-                        {"retry_at": queue_at, "modified_at": queue_at},
-                        refresh=False,
-                        extra_filter={"status": snapshot.status},
-                    )
-                else:
-                    snapshot.update_and_requeue(
-                        status=Snapshot.StatusChoices.QUEUED,
-                        retry_at=queue_at,
-                        current_step=0,
-                    )
-        else:
-            # No plugin rows were requested, so this is a full snapshot retry.
-            for snapshot in Snapshot.objects.filter(id__in=existing_snapshot_ids).only("id", "status", "retry_at", "modified_at"):
-                snapshot.safe_update(
-                    {
-                        "status": Snapshot.StatusChoices.QUEUED,
-                        "retry_at": queue_at,
-                        "current_step": 0,
-                        "modified_at": queue_at,
-                    },
-                    refresh=False,
-                    extra_filter={"status": snapshot.status},
-                )
-    if existing_crawl_ids and not requested_rows:
-        from archivebox.crawls.models import Crawl
-
-        for crawl in Crawl.objects.filter(id__in=existing_crawl_ids).only("id", "status", "retry_at", "modified_at"):
-            update_fields = {
-                "retry_at": queue_at,
-                "modified_at": queue_at,
-            }
-            if crawl.status != Crawl.StatusChoices.STARTED:
-                update_fields["status"] = Crawl.StatusChoices.QUEUED
-            crawl.safe_update(
-                update_fields,
-                refresh=False,
-                extra_filter={"status": crawl.status},
-            )
-
-    if processed_count == 0:
-        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
-        return 1
-
-    if show_progress:
-        rprint(f"[blue]Queued {processed_count} snapshots for extraction[/blue]", file=sys.stderr)
-
-    # Run orchestrator if --wait (default)
-    if wait:
-        if show_progress:
-            rprint("[blue]Running plugins...[/blue]", file=sys.stderr)
-        snapshot_ids_by_crawl: dict[str, set[str]] = defaultdict(set)
-        for snapshot_id, crawl_id in existing_snapshots:
-            snapshot_ids_by_crawl[str(crawl_id)].add(str(snapshot_id))
-
-        for crawl_id, crawl_snapshot_ids in snapshot_ids_by_crawl.items():
-            from archivebox.crawls.models import Crawl
-
-            crawl = Crawl.objects.get(id=crawl_id)
-            if not crawl.claim_processing_lock(lock_seconds=10):
-                rprint(
-                    f"[yellow]Crawl {crawl_id} is already owned by another runner[/yellow]",
-                    file=sys.stderr,
-                )
-                return 1
-            selected_plugins = (
-                plugins_list
-                or sorted(
-                    {plugin for snapshot_id in crawl_snapshot_ids for plugin in requested_plugins_by_snapshot.get(str(snapshot_id), set())},
-                )
-                or None
-            )
-            run_crawl(
-                crawl_id,
-                snapshot_ids=sorted(crawl_snapshot_ids),
-                selected_plugins=selected_plugins,
-                show_progress=show_progress,
-            )
-
-    if not emit_results:
-        return 0
-
-    # Output results as JSONL (when piped) or human-readable (when TTY)
-    for snapshot_id in snapshot_ids:
-        try:
-            snapshot = Snapshot.objects.get(id=snapshot_id)
-            results = snapshot.archiveresult_set.all()
-            if plugins_list:
-                results = results.filter(plugin__in=plugins_list)
-
-            for result in results:
-                if is_tty:
-                    status_color = {
-                        "succeeded": "green",
-                        "failed": "red",
-                        "skipped": "yellow",
-                    }.get(result.status, "dim")
-                    rprint(
-                        f"  [{status_color}]{result.status}[/{status_color}] {result.plugin} → {result.output_str or ''}",
-                        file=sys.stderr,
-                    )
-                else:
-                    write_record(result.to_json())
-        except Snapshot.DoesNotExist:
-            continue
+    exit_code = _run_snapshot_requests(requested, wait=wait, show_progress=show_progress)
+    if exit_code or not emit_results:
+        return exit_code
 
+    is_tty = sys.stdout.isatty()
+    for snapshot in Snapshot.objects.filter(id__in=requested):
+        results = snapshot.archiveresult_set.all()
+        requested_plugins = requested[str(snapshot.id)]
+        if requested_plugins:
+            results = results.filter(plugin__in=requested_plugins)
+        for result in results:
+            if is_tty:
+                color = {"succeeded": "green", "failed": "red", "skipped": "yellow"}.get(result.status, "dim")
+                rprint(f"  [{color}]{result.status}[/{color}] {result.plugin} → {result.output_str or ''}", file=sys.stderr)
+            else:
+                write_record(result.to_json())
     return 0
 
 
-def is_archiveresult_id(value: str) -> bool:
-    """Check if value resolves to an ArchiveResult ID."""
-    from archivebox.core.models import ArchiveResult
-    from archivebox.api.v1_core import _uuid_ref_query
-
-    return ArchiveResult.objects.filter(_uuid_ref_query("id", value)).exists()
-
-
 @click.command()
-@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run (e.g., screenshot,singlefile)")
+@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run")
 @click.option("--wait/--no-wait", default=True, help="Wait for plugins to complete (default: wait)")
 @click.argument("args", nargs=-1)
 def main(plugins: str, wait: bool, args: tuple):
-    """Run plugins on Snapshots, or process existing ArchiveResults by ID"""
+    """Run plugins on Snapshots; ArchiveResult IDs select their parent plugin."""
     from archivebox.misc.jsonl import read_args_or_stdin
 
-    # Read all input
     records = list(read_args_or_stdin(args))
-
     if not records:
         from rich import print as rprint
 
         rprint("[yellow]No Snapshot IDs or ArchiveResult IDs provided. Pass as arguments or via stdin.[/yellow]", file=sys.stderr)
         sys.exit(1)
-
-    # Check if input looks like existing ArchiveResult IDs to process
-    all_are_archiveresult_ids = all(is_archiveresult_id(r.get("id") or r.get("url", "")) for r in records)
-
-    if all_are_archiveresult_ids:
-        # Process existing ArchiveResults by ID
-        from rich import print as rprint
-
-        exit_code = 0
-        for record in records:
-            archiveresult_id = record.get("id") or record.get("url")
-            if not isinstance(archiveresult_id, str):
-                rprint(f"[red]Invalid ArchiveResult input: {record}[/red]", file=sys.stderr)
-                exit_code = 1
-                continue
-            result = process_archiveresult_by_id(archiveresult_id)
-            if result != 0:
-                exit_code = result
-        sys.exit(exit_code)
-    else:
-        # Default behavior: run plugins on Snapshots from input
-        sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
+    sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_help.py

@@ -5,6 +5,8 @@ __command__ = "archivebox help"
 import ast
 import importlib.util
 import os
+import sys
+from datetime import UTC, datetime
 from pathlib import Path
 
 import click
@@ -40,11 +42,14 @@ def help() -> None:
 
     from archivebox.cli import ArchiveBoxGroup
     from archivebox.config import CONSTANTS
-    from archivebox.config.common import get_config
     from archivebox.config.permissions import IN_DOCKER
-    from archivebox.misc.logging_util import log_cli_command
 
-    log_cli_command("help", [], None, ".")
+    version_msg = (
+        f"[dark_magenta]\\[{datetime.now(UTC):%Y-%m-%d %H:%M:%S}][/dark_magenta] "
+        f"[dark_red]ArchiveBox[/dark_red] [dark_goldenrod]v{CONSTANTS.VERSION}[/dark_goldenrod]: "
+        "[green4]archivebox [green3]help"
+    )
+    print(Panel(version_msg), file=sys.stderr)
 
     COMMANDS_HELP_TEXT = (
         "\n    ".join(f"[green]{cmd.ljust(20)}[/green] {_command_doc(cmd, path)}" for cmd, path in ArchiveBoxGroup.meta_commands.items())
@@ -89,7 +94,6 @@ def help() -> None:
     [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration[/link]
 """)
 
-    get_config()
     if os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and CONSTANTS.ARCHIVE_DIR.is_dir():
         pretty_out_dir = str(CONSTANTS.DATA_DIR).replace(str(Path("~").expanduser()), "~")
         EXAMPLE_USAGE = f"""

archivebox/cli/archivebox_init.py

@@ -176,10 +176,13 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
         install_method()
 
     if Snapshot.objects.count() < 25:  # hide the hints for experienced users
+        from archivebox.core.routes_util import build_admin_url
+
+        admin_url = build_admin_url("/admin/", config=config)
         print()
-        print("    [violet]Hint:[/violet] To view your archive index, run:")
+        print("    [violet]Hint:[/violet] To finish web setup, run:")
         print(
-            "        archivebox server  # then visit [deep_sky_blue4][link=http://127.0.0.1:8000]http://127.0.0.1:8000[/link][/deep_sky_blue4]",
+            f"        archivebox server  # then visit [deep_sky_blue4][link={admin_url}]{admin_url}[/link][/deep_sky_blue4]",
         )
         print()
         print("    To add new links, you can run:")

archivebox/cli/archivebox_install.py

@@ -3,8 +3,8 @@
 __package__ = "archivebox.cli"
 
 import os
-import re
 import shutil
+import re
 from pathlib import Path
 
 import rich_click as click
@@ -13,93 +13,48 @@ from rich import print
 from archivebox.misc.util import docstring, enforce_types
 
 
-def _package_name_candidates(install_arg: object) -> set[str]:
-    """Return user-facing package tokens represented by a provider install arg."""
-    value = str(install_arg).strip()
-    if not value or value.startswith("-"):
-        return set()
-
-    candidates = {value}
-    package_name = re.split(r"==|>=|<=|~=|!=|>|<|\[", value, maxsplit=1)[0]
-    if "@" in package_name and not package_name.startswith("@"):
-        package_name = package_name.split("@", 1)[0]
-    if package_name:
-        candidates.add(package_name)
-        candidates.add(Path(package_name).name)
-    return candidates
-
-
-def _binary_record_candidates(binary_record: dict) -> set[str]:
-    """Return install-name tokens represented by an abx-dl binary request record."""
-    name = str(binary_record.get("name") or "")
-    display_name = Path(name).expanduser().name if ("/" in name or name.startswith("~")) else name
-    candidates = {name, display_name}
-    overrides = binary_record.get("overrides") or {}
-    if isinstance(overrides, dict):
-        for provider_overrides in overrides.values():
-            if not isinstance(provider_overrides, dict):
-                continue
-            install_args = provider_overrides.get("install_args") or ()
-            if isinstance(install_args, str):
-                install_args = (install_args,)
-            for install_arg in install_args:
-                candidates.update(_package_name_candidates(install_arg))
-    return {candidate for candidate in candidates if candidate}
-
-
 def _resolve_install_targets(
     requested_names: tuple[str, ...],
 ) -> tuple[list[str], list[str]]:
-    """Resolve `archivebox install <name>` tokens to plugin names plus raw binary names."""
-    from archivebox.config import CONSTANTS
-    from archivebox.config.common import get_config, normalize_runtime_config
-    from archivebox.machine.models import Machine
-    from abx_dl.config import get_required_binary_requests
-    from abx_dl.models import discover_plugins, filter_plugins
+    """Resolve plugin names and declared binary aliases, leaving unknown tokens raw."""
+    from archivebox.plugins.discovery import get_plugin_catalog
 
-    plugins = discover_plugins(runtime="archivebox")
-    config = get_config(include_machine=False)
-    runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
-    derived_config = normalize_runtime_config(Machine.current().config, json_safe=False)
-    selected_plugins = dict(filter_plugins(plugins, list(requested_names), include_providers=True))
+    catalog = get_plugin_catalog()
+    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in catalog}
+    plugin_names_by_binary_alias: dict[str, set[str]] = {}
+    for plugin_name, plugin in catalog.items():
+        for required_binary in plugin.config.required_binaries:
+            aliases = {required_binary.name.lower()}
+            template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", required_binary.name)
+            if template_match:
+                config_key = template_match.group(1)
+                aliases.add(config_key.lower())
+                aliases.add(config_key.removesuffix("_BINARY").lower())
+                default = plugin.config.properties.get(config_key, {}).get("default")
+                if default:
+                    aliases.add(str(default).lower())
+            for alias in aliases:
+                plugin_names_by_binary_alias.setdefault(alias, set()).add(plugin_name)
+
+    requested_plugins: list[str] = []
     raw_binary_names: list[str] = []
-
-    for requested_name in requested_names:
-        if requested_name in selected_plugins:
-            continue
-
-        matched_plugin_names: list[str] = []
-        for plugin_name, plugin in plugins.items():
-            logical_records = get_required_binary_requests(
-                plugin,
-                plugin.config.required_binaries,
-                overrides=runtime_config,
-                derived_overrides=derived_config,
-                run_output_dir=CONSTANTS.DATA_DIR,
-            )
-            actual_records = get_required_binary_requests(
-                plugin,
-                plugin.config.required_binaries,
-                overrides=runtime_config,
-                derived_overrides=derived_config,
-                run_output_dir=CONSTANTS.DATA_DIR,
-                logical_names=False,
-            )
-            for logical_record, actual_record in zip(logical_records, actual_records, strict=False):
-                if requested_name in _binary_record_candidates(logical_record) | _binary_record_candidates(actual_record):
-                    matched_plugin_names.append(plugin_name)
-                    break
-
-        if matched_plugin_names:
-            selected_plugins.update(filter_plugins(plugins, matched_plugin_names, include_providers=True))
+    for name in requested_names:
+        normalized_name = name.lower()
+        if normalized_name in plugin_names_by_lower:
+            requested_plugins.append(plugin_names_by_lower[normalized_name])
+        elif normalized_name in plugin_names_by_binary_alias:
+            requested_plugins.extend(sorted(plugin_names_by_binary_alias[normalized_name]))
         else:
-            raw_binary_names.append(requested_name)
+            raw_binary_names.append(name)
 
+    selected_plugins = catalog.select(requested_plugins) if requested_plugins else catalog.select([])
+    selected_plugin_names = {name.lower() for name in selected_plugins}
+    raw_binary_names = [name for name in raw_binary_names if name.lower() not in selected_plugin_names]
     return sorted(selected_plugins), sorted(set(raw_binary_names))
 
 
 def _install_raw_binary_names(binary_names: list[str], binproviders: str) -> None:
-    """Install user-requested standalone binaries through the Binary state machine."""
+    """Install user-requested standalone binaries through the Binary lifecycle."""
     from django.utils import timezone
 
     from archivebox.machine.models import Binary, Machine, _canonical_binary_name
@@ -212,6 +167,10 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
         install_plugin_names.extend(provider.strip() for provider in binproviders.split(",") if provider.strip())
 
     if install_plugin_names or not binaries:
+        if not binaries:
+            from archivebox.workers.supervisord_util import require_dependency_install_memory
+
+            require_dependency_install_memory()
         print("[+] Running plugin installer via abx-dl bus...")
         print()
 
@@ -220,7 +179,7 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
         run_install(plugin_names=install_plugin_names or None)
 
     if raw_binary_names:
-        print(f"[+] Running direct binary installer via ArchiveBox binary state machine: {', '.join(raw_binary_names)}")
+        print(f"[+] Running direct binary installer via ArchiveBox binary lifecycle: {', '.join(raw_binary_names)}")
         print()
         _install_raw_binary_names(raw_binary_names, binproviders)
 
@@ -237,8 +196,10 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
     User = get_user_model()
 
     if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
-        stderr("\n[+] Don't forget to create a new admin user for the Web UI...", color="green")
-        stderr("    archivebox manage createsuperuser")
+        from archivebox.core.routes_util import build_admin_url
+
+        stderr("\n[+] Open the Admin UI to create the first admin and finish web setup:", color="green")
+        stderr(f"    {build_admin_url('/admin/', config=get_config(resolve_plugins=False))}")
 
     print()
 

archivebox/cli/archivebox_machine.py

@@ -25,7 +25,6 @@ __command__ = "archivebox machine"
 import sys
 
 import rich_click as click
-from rich import print as rprint
 
 from archivebox.cli.cli_util import apply_filters
 
@@ -35,41 +34,21 @@ from archivebox.cli.cli_util import apply_filters
 # =============================================================================
 
 
-def list_machines(
-    hostname__icontains: str | None = None,
-    os_platform: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """
-    List Machines as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+def list_machines(hostname__icontains: str | None = None, os_platform: str | None = None, limit: int | None = None) -> int:
+    """List machines as JSONL, or formatted rows in a terminal."""
     from archivebox.machine.models import Machine
+    from archivebox.cli.cli_util import list_records
 
-    is_tty = sys.stdout.isatty()
-
-    queryset = Machine.objects.all().order_by("-created_at")
-
-    # Apply filters
-    filter_kwargs = {
-        "hostname__icontains": hostname__icontains,
-        "os_platform": os_platform,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for machine in queryset:
-        if is_tty:
-            rprint(f"[cyan]{machine.hostname:30}[/cyan] [dim]{machine.os_platform:10}[/dim] {machine.id}")
-        else:
-            write_record(machine.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} machines[/dim]", file=sys.stderr)
-    return 0
+    queryset = apply_filters(
+        Machine.objects.order_by("-created_at"),
+        {"hostname__icontains": hostname__icontains, "os_platform": os_platform},
+        limit=limit,
+    )
+    return list_records(
+        queryset,
+        plural="machines",
+        render=lambda machine: f"[cyan]{machine.hostname:30}[/cyan] [dim]{machine.os_platform:10}[/dim] {machine.id}",
+    )
 
 
 # =============================================================================
@@ -87,15 +66,9 @@ def main():
 @click.option("--hostname__icontains", help="Filter by hostname contains")
 @click.option("--os-platform", help="Filter by OS platform")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(hostname__icontains: str | None, os_platform: str | None, limit: int | None):
+def list_cmd(**kwargs):
     """List Machines as JSONL."""
-    sys.exit(
-        list_machines(
-            hostname__icontains=hostname__icontains,
-            os_platform=os_platform,
-            limit=limit,
-        ),
-    )
+    sys.exit(list_machines(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_manage.py

@@ -14,7 +14,7 @@ def manage(args: list[str] | None = None) -> None:
     from archivebox.misc.logging import stderr
 
     config = get_config()
-    if (args and "createsuperuser" in args) and (config.IN_DOCKER and not config.IS_TTY):
+    if (args and "createsuperuser" in args and "--noinput" not in args) and (config.IN_DOCKER and not config.IS_TTY):
         stderr("[!] Warning: you need to pass -it to use interactive commands in docker", color="lightyellow")
         stderr("    docker run -it archivebox manage {}".format(" ".join(args or ["..."])), color="lightyellow")
         stderr("")

archivebox/cli/archivebox_persona.py

@@ -27,164 +27,19 @@ Examples:
 __package__ = "archivebox.cli"
 __command__ = "archivebox persona"
 
-import os
 import sys
 import shutil
-import platform
 from pathlib import Path
+from dataclasses import replace
 from collections.abc import Iterable
 
 import rich_click as click
 from rich import print as rprint
 
-from archivebox.cli.cli_util import apply_filters
+from archivebox.cli.cli_util import apply_filters, list_records
 from archivebox.personas import importers as persona_importers
 
 
-# =============================================================================
-# Browser Profile Locations
-# =============================================================================
-
-
-def get_chrome_user_data_dir() -> Path | None:
-    """Get the default Chrome user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":  # macOS
-        candidates = [
-            home / "Library" / "Application Support" / "Google" / "Chrome",
-            home / "Library" / "Application Support" / "Chromium",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "google-chrome",
-            home / ".config" / "chromium",
-            home / ".config" / "chrome",
-            home / "snap" / "chromium" / "common" / "chromium",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Google" / "Chrome" / "User Data",
-            local_app_data / "Chromium" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_brave_user_data_dir() -> Path | None:
-    """Get the default Brave user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "BraveSoftware" / "Brave-Browser",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "BraveSoftware" / "Brave-Browser" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-def get_edge_user_data_dir() -> Path | None:
-    """Get the default Edge user data directory for the current platform."""
-    system = platform.system()
-    home = Path.home()
-
-    if system == "Darwin":
-        candidates = [
-            home / "Library" / "Application Support" / "Microsoft Edge",
-        ]
-    elif system == "Linux":
-        candidates = [
-            home / ".config" / "microsoft-edge",
-            home / ".config" / "microsoft-edge-beta",
-            home / ".config" / "microsoft-edge-dev",
-        ]
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = [
-            local_app_data / "Microsoft" / "Edge" / "User Data",
-        ]
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate.exists() and (candidate / "Default").exists():
-            return candidate
-
-    return None
-
-
-BROWSER_PROFILE_FINDERS = {
-    "chrome": get_chrome_user_data_dir,
-    "chromium": get_chrome_user_data_dir,  # Same locations
-    "brave": get_brave_user_data_dir,
-    "edge": get_edge_user_data_dir,
-}
-
-CHROMIUM_BROWSERS = {"chrome", "chromium", "brave", "edge"}
-
-
-# =============================================================================
-# Cookie Extraction via CDP
-# =============================================================================
-
-# =============================================================================
-# Validation Helpers
-# =============================================================================
-
-
-def validate_persona_name(name: str) -> tuple[bool, str]:
-    """
-    Validate persona name to prevent path traversal attacks.
-
-    Returns:
-        (is_valid, error_message): tuple indicating if name is valid
-    """
-    if not name or not name.strip():
-        return False, "Persona name cannot be empty"
-
-    # Check for path separators
-    if "/" in name or "\\" in name:
-        return False, "Persona name cannot contain path separators (/ or \\)"
-
-    # Check for parent directory references
-    if ".." in name:
-        return False, "Persona name cannot contain parent directory references (..)"
-
-    # Check for hidden files/directories
-    if name.startswith("."):
-        return False, "Persona name cannot start with a dot (.)"
-
-    # Ensure name doesn't contain null bytes or other dangerous chars
-    if "\x00" in name or "\n" in name or "\r" in name:
-        return False, "Persona name contains invalid characters"
-
-    return True, ""
-
-
 def ensure_path_within_personas_dir(persona_path: Path) -> bool:
     """
     Verify that a persona path is within PERSONAS_DIR.
@@ -218,6 +73,8 @@ def create_personas(
     names: Iterable[str],
     import_from: str | None = None,
     profile: str | None = None,
+    source: str | None = None,
+    browser_binary: str | None = None,
 ) -> int:
     """
     Create Personas from names.
@@ -239,25 +96,26 @@ def create_personas(
         rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
         return 1
 
-    # Validate import source if specified
-    source_profile_dir = None
+    import_source = None
+    if source and not import_from:
+        rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
+        return 1
     if import_from:
-        import_from = import_from.lower()
-        if import_from not in BROWSER_PROFILE_FINDERS:
-            rprint(f"[red]Unknown browser: {import_from}[/red]", file=sys.stderr)
-            rprint(f"[dim]Supported browsers: {', '.join(BROWSER_PROFILE_FINDERS.keys())}[/dim]", file=sys.stderr)
+        try:
+            if source:
+                import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
+                if import_source.browser != "persona":
+                    import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
+            elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
+                import_source = persona_importers.resolve_custom_import_source(import_from)
+            else:
+                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+            if browser_binary:
+                import_source = replace(import_source, browser_binary=browser_binary)
+        except ValueError as err:
+            rprint(f"[red]{err}[/red]", file=sys.stderr)
             return 1
 
-        source_profile_dir = BROWSER_PROFILE_FINDERS[import_from]()
-        if not source_profile_dir:
-            rprint(f"[red]Could not find {import_from} profile directory[/red]", file=sys.stderr)
-            return 1
-
-        rprint(f"[dim]Found {import_from} profile: {source_profile_dir}[/dim]", file=sys.stderr)
-
-        if profile is None and (source_profile_dir / "Default").exists():
-            profile = "Default"
-
     created_count = 0
     for name in name_list:
         name = name.strip()
@@ -275,16 +133,16 @@ def create_personas(
         if created:
             persona.ensure_dirs()
             created_count += 1
-            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
+
         else:
             rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
 
         cookies_file = Path(persona.path) / "cookies.txt"
 
         # Import browser profile if requested
-        if import_from in CHROMIUM_BROWSERS and source_profile_dir is not None:
+        if import_source is not None:
             try:
-                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
+                rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
                 import_result = persona_importers.import_persona_from_source(
                     persona,
                     import_source,
@@ -292,30 +150,27 @@ def create_personas(
                     import_cookies=True,
                     capture_storage=False,
                 )
-            except Exception as e:
+            except (Exception, KeyboardInterrupt) as e:
+                if created:
+                    shutil.rmtree(persona.path, ignore_errors=True)
+                    persona.delete()
                 rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
                 return 1
 
             if import_result.profile_copied:
                 rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
             if import_result.cookies_imported:
-                rprint(f"[green]Extracted cookies to {cookies_file}[/green]", file=sys.stderr)
+                rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
             elif not import_result.profile_copied:
                 rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
 
             for warning in import_result.warnings:
                 rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
 
+        if created:
+            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
         if not is_tty:
-            write_record(
-                {
-                    "id": str(persona.id),
-                    "name": persona.name,
-                    "path": str(persona.path),
-                    "CHROME_USER_DATA_DIR": persona.CHROME_USER_DATA_DIR,
-                    "COOKIES_FILE": persona.COOKIES_FILE,
-                },
-            )
+            write_record(persona.to_json())
 
     rprint(f"[green]Created {created_count} new persona(s)[/green]", file=sys.stderr)
     return 0
@@ -337,41 +192,20 @@ def list_personas(
     Exit codes:
         0: Success (even if no results)
     """
-    from archivebox.misc.jsonl import write_record
     from archivebox.personas.models import Persona
 
-    is_tty = sys.stdout.isatty()
+    queryset = apply_filters(
+        Persona.objects.order_by("name"),
+        {"name": name, "name__icontains": name__icontains},
+        limit=limit,
+    )
 
-    queryset = Persona.objects.all().order_by("name")
+    def render(persona):
+        cookies = "[green]✓[/green]" if persona.COOKIES_FILE else "[dim]✗[/dim]"
+        chrome = "[green]✓[/green]" if Path(persona.CHROME_USER_DATA_DIR).exists() else "[dim]✗[/dim]"
+        return f"[cyan]{persona.name:20}[/cyan] cookies:{cookies} chrome:{chrome} [dim]{persona.path}[/dim]"
 
-    # Apply filters
-    filter_kwargs = {
-        "name": name,
-        "name__icontains": name__icontains,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for persona in queryset:
-        cookies_status = "[green]✓[/green]" if persona.COOKIES_FILE else "[dim]✗[/dim]"
-        chrome_status = "[green]✓[/green]" if Path(persona.CHROME_USER_DATA_DIR).exists() else "[dim]✗[/dim]"
-
-        if is_tty:
-            rprint(f"[cyan]{persona.name:20}[/cyan] cookies:{cookies_status} chrome:{chrome_status} [dim]{persona.path}[/dim]")
-        else:
-            write_record(
-                {
-                    "id": str(persona.id),
-                    "name": persona.name,
-                    "path": str(persona.path),
-                    "CHROME_USER_DATA_DIR": persona.CHROME_USER_DATA_DIR,
-                    "COOKIES_FILE": persona.COOKIES_FILE,
-                },
-            )
-        count += 1
-
-    rprint(f"[dim]Listed {count} persona(s)[/dim]", file=sys.stderr)
-    return 0
+    return list_records(queryset, plural="persona(s)", render=render)
 
 
 # =============================================================================
@@ -435,13 +269,7 @@ def update_personas(name: str | None = None) -> int:
             updated_count += 1
 
             if not is_tty:
-                write_record(
-                    {
-                        "id": str(persona.id),
-                        "name": persona.name,
-                        "path": str(persona.path),
-                    },
-                )
+                write_record(persona.to_json())
 
         except Persona.DoesNotExist:
             rprint(f"[yellow]Persona not found: {persona_id or old_name}[/yellow]", file=sys.stderr)
@@ -544,35 +372,119 @@ def main():
 
 @main.command("create")
 @click.argument("names", nargs=-1)
-@click.option("--import", "import_from", help="Import profile from browser (chrome, chromium, brave, edge)")
+@click.option("--import", "import_from", help="Import from chrome, chromium, brave, edge, a persona directory, or a live CDP URL")
+@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
+@click.option(
+    "--browser-binary",
+    type=click.Path(exists=True, dir_okay=False),
+    help="Source Chromium browser executable; required for other Chromium-based browsers",
+)
 @click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
-def create_cmd(names: tuple, import_from: str | None, profile: str | None):
+def create_cmd(names: tuple, import_from: str | None, profile: str | None, source: str | None, browser_binary: str | None):
     """Create Personas, optionally importing from a browser profile."""
-    sys.exit(create_personas(names, import_from=import_from, profile=profile))
+    sys.exit(create_personas(names, import_from=import_from, profile=profile, source=source, browser_binary=browser_binary))
 
 
 @main.command("list")
 @click.option("--name", help="Filter by exact name")
 @click.option("--name__icontains", help="Filter by name contains")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
+def list_cmd(**kwargs):
     """List Personas as JSONL."""
-    sys.exit(list_personas(name=name, name__icontains=name__icontains, limit=limit))
+    sys.exit(list_personas(**kwargs))
+
+
+@main.command("open")
+@click.argument("name", default="Default")
+def open_cmd(name: str):
+    """Open a persona's browser, including its imported logins."""
+    import json
+    import os
+    import subprocess
+
+    import psutil
+
+    from abx_plugins import get_plugins_dir
+    from archivebox.config.common import get_config
+    from archivebox.personas.models import Persona
+
+    valid, reason = persona_importers.validate_persona_name(name)
+    if not valid:
+        raise click.ClickException(reason)
+    persona = Persona.get_or_create_named(name)
+    persona.ensure_dirs()
+    config = get_config(persona=persona)
+    if os.environ.get("DISPLAY"):
+        from archivebox.config import CONSTANTS
+
+        for process in psutil.process_iter():
+            try:
+                args = process.cmdline()
+                profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
+                if (
+                    not profile
+                    or any(arg.startswith(("--type=", "--headless")) for arg in args)
+                    or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
+                    or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
+                ):
+                    continue
+                rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
+                process.wait()
+            except (psutil.NoSuchProcess, psutil.AccessDenied):
+                continue
+    plugins_dir = Path(get_plugins_dir()).resolve()
+    env = os.environ.copy()
+    dependencies = subprocess.run(
+        [
+            str(Path(sys.executable).with_name("abxpkg")),
+            "env",
+            "--install",
+            "--json",
+            f"--lib={config.ABXPKG_LIB_DIR}",
+            f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
+        ],
+        capture_output=True,
+        text=True,
+        env=env,
+    )
+    if dependencies.returncode:
+        raise click.ClickException(dependencies.stderr.strip())
+    env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
+    env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
+    env["ACTIVE_PERSONA"] = persona.name
+    env["PERSONAS_DIR"] = str(persona.path.parent)
+    env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
+    payload = config.model_dump(mode="json")
+    payload.update(
+        ACTIVE_PERSONA=persona.name,
+        CHROME_HEADLESS=False,
+        CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
+        AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
+        COOKIES_FILE=str(persona.COOKIES_FILE or ""),
+    )
+    script = Path(__file__).parent.parent / "personas" / "open_browser.js"
+    result = subprocess.run(
+        [env.get("NODE_BINARY") or "node", str(script)],
+        input=json.dumps(payload),
+        text=True,
+        env=env,
+    )
+    sys.exit(result.returncode)
 
 
 @main.command("update")
 @click.option("--name", "-n", help="Set new name")
-def update_cmd(name: str | None):
+def update_cmd(**kwargs):
     """Update Personas from stdin JSONL."""
-    sys.exit(update_personas(name=name))
+    sys.exit(update_personas(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete Personas from stdin JSONL."""
-    sys.exit(delete_personas(yes=yes, dry_run=dry_run))
+    sys.exit(delete_personas(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_pluginmap.py

@@ -18,7 +18,7 @@ EVENT_FLOW_DIAGRAM = """
 │         └─ BinaryRequestEvent                                               │
 │              └─ abxpkg BinaryService builtin providers                      │
 │                   └─ BinaryEvent                                            │
-│                        └─ BinaryCacheService / project cache backend        │
+│                        └─ ArchiveBox DB history projection                  │
 │                                                                             │
 │  CrawlEvent                                                                 │
 │    └─ CrawlSetupEvent                                                       │
@@ -48,7 +48,7 @@ def pluginmap(
     Show the current abx-dl event phases and their associated plugin hooks.
 
     This command reflects the new bus-driven runtime, not the legacy ArchiveBox
-    state-machine executor. Event names are normalized to hook prefixes by
+    event runtime. Event names are normalized to hook prefixes by
     stripping a trailing `Event`, then ArchiveBox checks whether any matching
     `on_{EventFamily}__*` scripts actually exist.
     """

archivebox/cli/archivebox_process.py

@@ -28,7 +28,6 @@ __command__ = "archivebox process"
 import sys
 
 import rich_click as click
-from rich import print as rprint
 
 from archivebox.cli.cli_util import apply_filters
 
@@ -38,46 +37,20 @@ from archivebox.cli.cli_util import apply_filters
 # =============================================================================
 
 
-def list_processes(
-    binary_name: str | None = None,
-    machine_id: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """
-    List Processes as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+def list_processes(binary_name: str | None = None, machine_id: str | None = None, limit: int | None = None) -> int:
+    """List processes as JSONL, or exit codes and binary names in a terminal."""
     from archivebox.machine.models import Process
+    from archivebox.cli.cli_util import list_records
 
-    is_tty = sys.stdout.isatty()
+    def render(process):
+        name = process.binary.name if process.binary else "unknown"
+        exit_code = process.exit_code if process.exit_code is not None else "?"
+        color = "green" if process.exit_code == 0 else "red" if process.exit_code else "yellow"
+        return f"[{color}]exit={exit_code:3}[/{color}] [cyan]{name:15}[/cyan] [dim]{process.id}[/dim]"
 
-    queryset = Process.objects.all().select_related("binary", "machine").order_by("-started_at", "-created_at")
-
-    # Apply filters
-    filter_kwargs = {}
-    if binary_name:
-        filter_kwargs["binary__name"] = binary_name
-    if machine_id:
-        filter_kwargs["machine_id"] = machine_id
-
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for process in queryset:
-        if is_tty:
-            binary_name_str = process.binary.name if process.binary else "unknown"
-            exit_code = process.exit_code if process.exit_code is not None else "?"
-            status_color = "green" if process.exit_code == 0 else "red" if process.exit_code else "yellow"
-            rprint(f"[{status_color}]exit={exit_code:3}[/{status_color}] [cyan]{binary_name_str:15}[/cyan] [dim]{process.id}[/dim]")
-        else:
-            write_record(process.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} processes[/dim]", file=sys.stderr)
-    return 0
+    queryset = Process.objects.select_related("binary", "machine").order_by("-started_at", "-created_at")
+    filters = {"binary__name": binary_name or None, "machine_id": machine_id or None}
+    return list_records(apply_filters(queryset, filters, limit=limit), plural="processes", render=render)
 
 
 # =============================================================================
@@ -95,15 +68,9 @@ def main():
 @click.option("--binary-name", "-b", help="Filter by binary name")
 @click.option("--machine-id", "-m", help="Filter by machine ID")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(binary_name: str | None, machine_id: str | None, limit: int | None):
+def list_cmd(**kwargs):
     """List Processes as JSONL."""
-    sys.exit(
-        list_processes(
-            binary_name=binary_name,
-            machine_id=machine_id,
-            limit=limit,
-        ),
-    )
+    sys.exit(list_processes(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_remove.py

@@ -4,14 +4,17 @@ __package__ = "archivebox.cli"
 __command__ = "archivebox remove"
 
 import time
+from contextlib import nullcontext
 from pathlib import Path
 from collections.abc import Iterable
 
 import rich_click as click
+from rich.console import Console
 
 from django.db.models import QuerySet
 
 from archivebox.config import CONSTANTS
+from archivebox.config.common import get_config
 from archivebox.config.django import setup_django
 from archivebox.misc.util import enforce_types, docstring
 from archivebox.misc.checks import check_data_folder
@@ -20,7 +23,6 @@ from archivebox.misc.logging_util import (
     log_list_finished,
     log_removal_started,
     log_removal_finished,
-    TimedProgress,
 )
 from archivebox.cli.archivebox_snapshot import snapshot_filter_options
 
@@ -56,16 +58,14 @@ def remove(
     pattern_list = list(filter_patterns)
 
     log_list_started(pattern_list or None, filter_type)
-    timer = TimedProgress(360, prefix="      ")
-    try:
+    progress = Console().status("Finding matching snapshots...") if get_config().SHOW_PROGRESS else nullcontext()
+    with progress:
         if snapshots is None:
             snapshots = Snapshot.objects.order_by("-created_at").search(**filter_kwargs)
         # Freeze the target set up-front so a concurrent daemon writing new
         # snapshots can't extend the deletion under us, and so the cursor isn't
         # held open across the per-row deletes below.
         snapshot_pks = list(snapshots.values_list("pk", flat=True))
-    finally:
-        timer.end()
 
     if not snapshot_pks:
         log_removal_finished(0, 0)

archivebox/cli/archivebox_run.py

@@ -54,6 +54,11 @@ RUNNER_DAEMON_ENV = "ARCHIVEBOX_RUNNER_DAEMON"
 
 
 def _exit_daemon_runner_on_signal(sig: signal.Signals) -> None:
+    # A supervised `archivebox run --daemon` is intentionally a disposable
+    # child. If it receives SIGINT/SIGTERM directly, exit with the conventional
+    # signal status so supervisord treats it as an unexpected worker death and
+    # restarts only the runner. The parent `archivebox server` owns supervisord
+    # shutdown and must not be pulled down by a killed daemon worker.
     os._exit(128 + int(sig))
 
 
@@ -67,8 +72,8 @@ def process_stdin_records() -> int:
 
     Outputs JSONL of all processed records (for chaining).
 
-    Handles any record type: Crawl, Snapshot, ArchiveResult.
-    Auto-cascades: Crawl → Snapshots → ArchiveResults.
+    Handles Crawl and Snapshot work records. ArchiveResult records are accepted
+    as references to their parent Snapshot and plugin.
 
     Returns exit code (0 = success, 1 = error).
     """
@@ -85,6 +90,7 @@ def process_stdin_records() -> int:
     )
     from archivebox.base_models.models import get_or_create_system_user_pk
     from archivebox.core.models import Snapshot, ArchiveResult
+    from archivebox.misc.db import uuid_ref_query
     from archivebox.crawls.models import Crawl
     from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
     from archivebox.machine.models import Binary
@@ -111,15 +117,7 @@ def process_stdin_records() -> int:
 
         try:
             if record_type == TYPE_CRAWL:
-                if record_id:
-                    # Existing crawl - re-queue
-                    try:
-                        crawl = Crawl.objects.get(id=record_id)
-                    except Crawl.DoesNotExist:
-                        crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
-                else:
-                    # New crawl - create it
-                    crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
+                crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
 
                 if crawl:
                     crawl.update_and_requeue(
@@ -132,14 +130,8 @@ def process_stdin_records() -> int:
                     queued_count += 1
 
             elif record_type == TYPE_SNAPSHOT or (record.get("url") and not record_type):
-                if record_id:
-                    # Existing snapshot - re-queue
-                    try:
-                        snapshot = Snapshot.objects.get(id=record_id)
-                    except Snapshot.DoesNotExist:
-                        snapshot = Snapshot.from_json(record, overrides={"created_by_id": created_by_id})
-                else:
-                    # New snapshot - create it
+                snapshot = Snapshot.objects.filter(id=record_id).first() if record_id else None
+                if snapshot is None:
                     snapshot = Snapshot.from_json(record, overrides={"created_by_id": created_by_id})
 
                 if snapshot:
@@ -151,33 +143,15 @@ def process_stdin_records() -> int:
                     queued_count += 1
 
             elif record_type == TYPE_ARCHIVERESULT:
-                if record_id:
-                    # Existing archiveresult - re-queue
-                    try:
-                        archiveresult = ArchiveResult.objects.get(id=record_id)
-                    except ArchiveResult.DoesNotExist:
-                        archiveresult = None
-                else:
-                    archiveresult = None
-
-                snapshot_id = record.get("snapshot_id")
-                plugin_name = record.get("plugin")
-                snapshot = None
-                if archiveresult:
-                    if archiveresult.status in [
-                        ArchiveResult.StatusChoices.FAILED,
-                        ArchiveResult.StatusChoices.SKIPPED,
-                        ArchiveResult.StatusChoices.NORESULTS,
-                        ArchiveResult.StatusChoices.BACKOFF,
-                    ]:
-                        archiveresult.reset_for_retry()
-                    snapshot = archiveresult.snapshot
-                    plugin_name = plugin_name or archiveresult.plugin
-                elif snapshot_id:
-                    try:
-                        snapshot = Snapshot.objects.get(id=snapshot_id)
-                    except Snapshot.DoesNotExist:
-                        snapshot = None
+                snapshot_id = str(record.get("snapshot_id") or "")
+                plugin_name = str(record.get("plugin") or "")
+                archiveresult = None
+                if not snapshot_id and record_id:
+                    archiveresult = ArchiveResult.objects.filter(uuid_ref_query("id", str(record_id))).select_related("snapshot").first()
+                    if archiveresult:
+                        snapshot_id = str(archiveresult.snapshot_id)
+                        plugin_name = plugin_name or archiveresult.plugin
+                snapshot = Snapshot.objects.filter(id=snapshot_id).first() if snapshot_id else None
 
                 if snapshot:
                     snapshot.queue_for_extraction()
@@ -185,16 +159,12 @@ def process_stdin_records() -> int:
                     snapshot_ids_by_crawl[crawl_id].add(str(snapshot.id))
                     if plugin_name:
                         plugin_names_by_crawl[crawl_id].add(str(plugin_name))
-                    output_records.append(record if not archiveresult else archiveresult.to_json())
+                    output_records.append(archiveresult.to_json() if archiveresult else record)
                     queued_count += 1
 
             elif record_type in {TYPE_BINARYREQUEST, TYPE_BINARY}:
-                if record_id:
-                    try:
-                        binary = Binary.objects.get(id=record_id)
-                    except Binary.DoesNotExist:
-                        binary = Binary.from_json(record)
-                else:
+                binary = Binary.objects.filter(id=record_id).first() if record_id else None
+                if binary is None:
                     binary = Binary.from_json(record)
 
                 if binary:
@@ -243,7 +213,6 @@ def process_stdin_records() -> int:
                     crawl_id,
                     snapshot_ids=None if crawl_id in full_crawl_ids else sorted(snapshot_ids_by_crawl[crawl_id]),
                     selected_plugins=None if crawl_id in run_all_plugins_for_crawl else sorted(plugin_names_by_crawl[crawl_id]),
-                    selected_plugins_are_explicit=False,
                 )
     return 0
 
@@ -252,7 +221,6 @@ def run_runner(
     daemon: bool = False,
     crawl_id: str | None = None,
     maintenance_only: bool = False,
-    maintenance_batch_size: int | None = None,
 ) -> int:
     """
     Run the background runner loop.
@@ -305,6 +273,7 @@ def run_runner(
     interactive_interrupts = current.root.process_type == Process.TypeChoices.ADD
     if daemon:
         os.environ[RUNNER_DAEMON_ENV] = "1"
+
     try:
         with (
             foreground_shutdown_signals(
@@ -318,7 +287,6 @@ def run_runner(
                 crawl_id=crawl_id,
                 maintenance_only=maintenance_only,
                 interactive_interrupts=interactive_interrupts,
-                **({"maintenance_batch_size": maintenance_batch_size} if maintenance_batch_size else {}),
             )
         return 0
     except KeyboardInterrupt:
@@ -343,13 +311,6 @@ def run_runner(
 @click.option("--snapshot-id", help="Run one snapshot through its crawl")
 @click.option("--binary-id", help="Run one queued binary install directly on the bus")
 @click.option("--maintenance-only", is_flag=True, help="Only process sealed Snapshot maintenance and search-index backfills")
-@click.option(
-    "--maintenance-batch-size",
-    type=int,
-    default=None,
-    hidden=True,
-    help="Limit queued maintenance snapshots claimed per scheduler tick",
-)
 @click.option("--no-stdin", is_flag=True, hidden=True, help="Run the scheduler even when stdin is not a TTY")
 def main(
     daemon: bool,
@@ -357,7 +318,6 @@ def main(
     snapshot_id: str,
     binary_id: str,
     maintenance_only: bool,
-    maintenance_batch_size: int | None,
     no_stdin: bool,
 ):
     """
@@ -382,7 +342,7 @@ def main(
                 ),
                 foreground_parent_watchdog(enabled=False),
             ):
-                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
+                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only))
         except KeyboardInterrupt:
             sys.exit(0)
 
@@ -411,17 +371,16 @@ def main(
                     daemon=False,
                     crawl_id=crawl_id,
                     maintenance_only=maintenance_only,
-                    maintenance_batch_size=maintenance_batch_size,
                 ),
             )
 
         if maintenance_only:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=True, maintenance_batch_size=maintenance_batch_size))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=True))
 
         if not no_stdin and not sys.stdin.isatty():
             sys.exit(process_stdin_records())
         else:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only, maintenance_batch_size=maintenance_batch_size))
+            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only))
 
 
 def run_snapshot_worker(snapshot_id: str) -> int:

archivebox/cli/archivebox_schedule.py

@@ -37,6 +37,7 @@ def schedule(
         "created_schedule_ids": [],
         "disabled_count": 0,
         "run_all_enqueued": 0,
+        "run_all_maintained": 0,
         "active_schedule_ids": [],
     }
 
@@ -57,12 +58,12 @@ def schedule(
 
         created_by_id = get_or_create_system_user_pk()
         is_update_schedule = not import_path
-        template_urls = import_path or "archivebox://update"
+        template_urls = import_path or ""
         template_label = (f"Scheduled import: {template_urls}" if import_path else "Scheduled ArchiveBox update")[:64]
         template_notes = (
             f"Created by archivebox schedule for {template_urls}"
             if import_path
-            else "Created by archivebox schedule to queue recurring archivebox://update maintenance crawls."
+            else "Created by archivebox schedule to run recurring ArchiveBox maintenance."
         )
 
         template = Crawl.objects.create(
@@ -76,7 +77,6 @@ def schedule(
             retry_at=None,
             config={
                 "DEPTH": 0 if is_update_schedule else depth,
-                "SCHEDULE_KIND": "update" if is_update_schedule else "crawl",
                 # Caller-supplied overrides (e.g. {"ONLY_NEW": False}) win over the
                 # template defaults. Anything left unset falls through to the
                 # standard config stack at crawl-resolution time.
@@ -86,6 +86,7 @@ def schedule(
         crawl_schedule = CrawlSchedule.objects.create(
             template=template,
             schedule=schedule_str,
+            config={**template.config, "SCHEDULE_KIND": "update" if is_update_schedule else "crawl"},
             is_enabled=True,
             label=template_label,
             notes=template_notes,
@@ -119,12 +120,19 @@ def schedule(
 
     if run_all:
         enqueued = 0
+        maintained = 0
         now = timezone.now()
         for scheduled_crawl in schedules:
-            scheduled_crawl.enqueue(queued_at=now)
-            enqueued += 1
+            if scheduled_crawl.dispatch(queued_at=now) is None:
+                maintained += 1
+            else:
+                enqueued += 1
         result["run_all_enqueued"] = enqueued
+        result["run_all_maintained"] = maintained
         print(f"[green]\\[*] Enqueued {enqueued} scheduled crawl(s) immediately.[/green]")
+        if maintained:
+            run_pending_crawls(maintenance_only=True)
+            print(f"[green]\\[*] Ran {maintained} scheduled maintenance update(s).[/green]")
         if enqueued:
             print(
                 "[yellow]\\[*] Start `archivebox server`, `archivebox run --daemon`, or `archivebox schedule --foreground` to process the queued crawls.[/yellow]",
@@ -171,7 +179,7 @@ def schedule(
 @click.option("--clear", is_flag=True, help="Disable all currently enabled schedules")
 @click.option("--show", is_flag=True, help="Print all currently enabled schedules")
 @click.option("--foreground", "-f", is_flag=True, help="Run the global crawl runner in the foreground (no crontab required)")
-@click.option("--run-all", is_flag=True, help="Enqueue all enabled schedules immediately and process them once")
+@click.option("--run-all", is_flag=True, help="Dispatch all enabled schedules immediately")
 @click.argument("import_path", required=False)
 @docstring(schedule.__doc__)
 def main(**kwargs):

archivebox/cli/archivebox_server.py

@@ -11,6 +11,8 @@ from collections.abc import Iterable
 
 import rich_click as click
 from rich import print
+from rich.style import Style
+from rich.text import Text
 
 from archivebox.config import CONSTANTS
 from archivebox.misc.util import docstring, enforce_types
@@ -251,14 +253,6 @@ def server(
         manage(args=["createsuperuser"])
         print()
 
-    if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
-        print()
-        print(
-            "[violet]Hint:[/violet] To create an [bold]admin username & password[/bold] for the [deep_sky_blue3][underline][link=http://{host}:{port}/admin]Admin UI[/link][/underline][/deep_sky_blue3], run:",
-        )
-        print("      [green]archivebox manage createsuperuser[/green]")
-        print()
-
     # First non-empty positional arg is the bind spec; otherwise inherit from
     # config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
     # hard-errors on hostnames so the rest of the server can assume a numeric
@@ -266,6 +260,17 @@ def server(
     bind_spec = next((arg for arg in runserver_args if arg), "")
     host, port = _parse_and_validate_bind_spec(bind_spec)
 
+    if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
+        from archivebox.core.routes_util import build_admin_url
+
+        print()
+        print("[violet]Hint:[/violet] Open the Admin UI to create the first admin and finish web setup:")
+        runtime_config = config.model_copy(update={"BIND_ADDR": f"{host}:{port}"})
+        print(f"      [green]{build_admin_url('/admin/', config=runtime_config)}[/green]")
+        if not config.BASE_URL and host not in ("127.0.0.1", "localhost"):
+            print("      (When running remotely, replace localhost with this server's IP address or hostname.)")
+        print()
+
     if daemonize and os.environ.get("ARCHIVEBOX_SERVER_DAEMON_CHILD") != "1":
         from archivebox.workers.supervisord_util import resolve_env_binary
 
@@ -302,7 +307,14 @@ def server(
             except OSError:
                 time.sleep(0.1)
         else:
-            print(f"[yellow][!] ArchiveBox daemon server pid={proc.pid} is still starting. See {log_path}[/yellow]")
+            print(f"[red][X] ArchiveBox daemon server pid={proc.pid} did not become ready. See {log_path}[/red]")
+            proc.terminate()
+            try:
+                proc.wait(timeout=10)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+                proc.wait(timeout=5)
+            sys.exit(1)
         return
 
     os.environ["BIND_ADDR"] = f"{host}:{port}"
@@ -332,12 +344,15 @@ def server(
         print("[green][+] Starting ArchiveBox webserver in DEBUG mode...[/green]")
     else:
         print("[green][+] Starting ArchiveBox webserver...[/green]")
-    print(
-        f"    [blink][green]>[/green][/blink] Starting ArchiveBox webserver on [dim]BIND_ADDR[/dim] [deep_sky_blue4][link=http://{host}:{port}]http://{host}:{port}[/link][/deep_sky_blue4]",
-    )
-    print(
-        f"    [green]>[/green] Log in to ArchiveBox Admin UI on [dim]BASE_URL [/dim] [deep_sky_blue3][link={admin_url}]{admin_url}[/link][/deep_sky_blue3]",
+    bind_url = f"http://{host}:{port}"
+    bind_message = Text.from_markup(
+        "    [blink][green]>[/green][/blink] Starting ArchiveBox webserver on [dim]BIND_ADDR[/dim] ",
     )
+    bind_message.append(bind_url, style=Style(color="deep_sky_blue4", link=bind_url))
+    print(bind_message)
+    admin_message = Text.from_markup("    [green]>[/green] Log in to ArchiveBox Admin UI on [dim]BASE_URL [/dim] ")
+    admin_message.append(admin_url, style=Style(color="deep_sky_blue3", link=admin_url))
+    print(admin_message)
     print("    > Writing ArchiveBox error log to ./logs/errors.log")
     print()
 
@@ -345,7 +360,6 @@ def server(
     # security-mode + base-url warnings see the effective values.
     runtime_config = get_config()
     _print_server_startup_warnings(runtime_config, host, port)
-    bind_url = f"http://{host}:{port}"
     command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
 
     def still_owns_runtime_stack() -> bool:
@@ -368,7 +382,7 @@ def server(
                     data_dir=CONSTANTS.DATA_DIR,
                     exclude_id=command.id,
                 )
-                takeover_components = active_supervisord_runtime_components(config=config)
+                takeover_components = active_supervisord_runtime_components()
                 if older_owner and takeover_components:
                     print(
                         "[yellow][*] Taking over "

archivebox/cli/archivebox_snapshot.py

@@ -144,15 +144,7 @@ def create_snapshots(
                     write_record(record)
 
                 # Input is a Crawl - get or create it, then create Snapshots for its URLs
-                crawl = None
-                crawl_id = record.get("id")
-                if crawl_id:
-                    try:
-                        crawl = Crawl.objects.get(id=crawl_id)
-                    except Crawl.DoesNotExist:
-                        crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
-                else:
-                    crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
+                crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
 
                 if not crawl:
                     continue
@@ -174,7 +166,7 @@ def create_snapshots(
                         "depth": depth,
                         "status": status,
                     }
-                    snapshot = Snapshot.from_json(snapshot_record, overrides={"created_by_id": created_by_id})
+                    snapshot = Snapshot.from_json(snapshot_record, overrides={"crawl": crawl, "created_by_id": created_by_id})
                     if snapshot:
                         created_snapshots.append(snapshot)
                         if not is_tty:
@@ -368,77 +360,29 @@ def list_snapshots(
 # =============================================================================
 
 
-def update_snapshots(
-    status: str | None = None,
-    tag: str | None = None,
-) -> int:
-    """
-    Update Snapshots from stdin JSONL.
-
-    Reads Snapshot records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
+def update_snapshots(status: str | None = None, tag: str | None = None) -> int:
+    """Update JSONL-selected snapshots through their lifecycle operations."""
     from django.utils import timezone
-
-    from archivebox.misc.jsonl import read_stdin, write_record
     from archivebox.core.models import Snapshot
+    from archivebox.cli.cli_util import update_records, update_record_status
 
-    is_tty = sys.stdout.isatty()
+    def update(snapshot):
+        if status:
+            try:
+                update_record_status(snapshot, status)
+            except ValueError as err:
+                rprint(f"[red]{err}[/red]", file=sys.stderr)
+                return False
+        if tag:
+            from archivebox.core.models import Tag
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
+            tag_obj, _ = Tag.objects.get_or_create(name=tag)
+            snapshot.tags.add(tag_obj)
+            snapshot.safe_update({"modified_at": timezone.now()}, refresh=False)
+        if not status and not tag:
+            snapshot.safe_update({"modified_at": timezone.now()}, refresh=False)
 
-    updated_count = 0
-    for record in records:
-        snapshot_id = record.get("id")
-        if not snapshot_id:
-            continue
-
-        try:
-            snapshot = Snapshot.objects.get(id=snapshot_id)
-
-            if status:
-                if status not in Snapshot.StatusChoices.values:
-                    rprint(f"[red]Invalid snapshot status: {status}[/red]", file=sys.stderr)
-                    continue
-                if status == Snapshot.StatusChoices.SEALED:
-                    snapshot.cancel()
-                elif status == Snapshot.StatusChoices.PAUSED:
-                    snapshot.pause()
-                elif status == Snapshot.StatusChoices.QUEUED:
-                    if snapshot.status == Snapshot.StatusChoices.PAUSED:
-                        snapshot.resume()
-                    else:
-                        snapshot.update_and_requeue(status=Snapshot.StatusChoices.QUEUED, retry_at=timezone.now())
-                elif status == Snapshot.StatusChoices.STARTED:
-                    snapshot.update_and_requeue(status=Snapshot.StatusChoices.STARTED, retry_at=timezone.now())
-            if tag:
-                from archivebox.core.models import Tag
-
-                tag_obj, _ = Tag.objects.get_or_create(name=tag)
-                snapshot.tags.add(tag_obj)
-                snapshot.safe_update({"modified_at": timezone.now()}, refresh=False)
-
-            if not status and not tag:
-                snapshot.safe_update({"modified_at": timezone.now()}, refresh=False)
-            updated_count += 1
-
-            if not is_tty:
-                snapshot.refresh_from_db()
-                write_record(snapshot.to_json())
-
-        except Snapshot.DoesNotExist:
-            rprint(f"[yellow]Snapshot not found: {snapshot_id}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} snapshots[/green]", file=sys.stderr)
-    return 0
+    return update_records(Snapshot, update, plural="snapshots", refresh=True)
 
 
 # =============================================================================
@@ -447,50 +391,18 @@ def update_snapshots(
 
 
 def delete_snapshots(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete Snapshots from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
+    """Delete snapshots selected by stdin JSONL; --yes confirms, --dry-run previews."""
+    from archivebox.cli.cli_util import delete_records
     from archivebox.core.models import Snapshot
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    snapshot_ids = [r.get("id") for r in records if r.get("id")]
-
-    if not snapshot_ids:
-        rprint("[yellow]No valid snapshot IDs in input[/yellow]", file=sys.stderr)
-        return 1
-
-    snapshots = Snapshot.objects.filter(id__in=snapshot_ids)
-    count = snapshots.count()
-
-    if count == 0:
-        rprint("[yellow]No matching snapshots found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} snapshots (dry run)[/yellow]", file=sys.stderr)
-        for snapshot in snapshots:
-            rprint(f"  [dim]{snapshot.id}[/dim] {snapshot.url[:60]}", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Perform deletion
-    deleted_count, _ = snapshots.delete()
-    rprint(f"[green]Deleted {deleted_count} snapshots[/green]", file=sys.stderr)
-    return 0
+    return delete_records(
+        Snapshot,
+        label="snapshot",
+        plural="snapshots",
+        preview=lambda obj: f"[dim]{obj.id}[/dim] {obj.url[:60]}",
+        yes=yes,
+        dry_run=dry_run,
+    )
 
 
 # =============================================================================
@@ -525,17 +437,17 @@ def list_cmd(**kwargs):
 @main.command("update")
 @click.option("--status", "-s", help="Set status")
 @click.option("--tag", "-t", help="Add tag")
-def update_cmd(status: str | None, tag: str | None):
+def update_cmd(**kwargs):
     """Update Snapshots from stdin JSONL."""
-    sys.exit(update_snapshots(status=status, tag=tag))
+    sys.exit(update_snapshots(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete Snapshots from stdin JSONL."""
-    sys.exit(delete_snapshots(yes=yes, dry_run=dry_run))
+    sys.exit(delete_snapshots(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_status.py

@@ -52,7 +52,10 @@ def status(out_dir: Path = CONSTANTS.DATA_DIR) -> None:
     print()
     print("[green]\\[*] Scanning archive data directories...[/green]")
     users_dir = CONSTANTS.USERS_DIR
-    scan_roots = [root for root in (archive_dir, users_dir) if root.exists()]
+    # The current users layout is nested below archive/, so scanning both
+    # archive/ and archive/users/ recursively counts every current snapshot
+    # twice. A single archive root includes both legacy and current layouts.
+    scan_roots = [archive_dir] if archive_dir.exists() else []
     scan_roots_display = ", ".join(str(root) for root in scan_roots) if scan_roots else str(archive_dir)
     print(f"[yellow]   {scan_roots_display}[/yellow]")
     do_precise_fs_scan = num_sql_links <= MAX_STATUS_FS_DIR_SCAN
@@ -112,7 +115,7 @@ def status(out_dir: Path = CONSTANTS.DATA_DIR) -> None:
 
     if orphaned_dirs:
         print("    [violet]Hint:[/violet] To automatically import orphaned data directories into the main index, run:")
-        print("        [green]archivebox init[/green]")
+        print("        [green]archivebox update[/green]")
 
     print()
     print("[green]\\[*] Scanning recent archive changes and user logins:[/green]")
@@ -128,9 +131,11 @@ def status(out_dir: Path = CONSTANTS.DATA_DIR) -> None:
         print(f"    Last changes: {str(last_downloaded.downloaded_at)[:16]}")
 
     if not users:
+        from archivebox.core.routes_util import build_admin_url
+
         print()
-        print("    [violet]Hint:[/violet] You can create an admin user by running:")
-        print("        [green]archivebox manage createsuperuser[/green]")
+        print("    [violet]Hint:[/violet] Open the Admin UI to create the first admin and finish web setup:")
+        print(f"        [green]{build_admin_url('/admin/', config=config)}[/green]")
 
     print()
     recent_snapshots = snapshots_qs.order_by(

archivebox/cli/archivebox_tag.py

@@ -70,7 +70,7 @@ def create_tags(names: Iterable[str]) -> int:
         if not name:
             continue
 
-        tag, created = Tag.objects.get_or_create(name=name)
+        tag, created = Tag.get_or_create_by_name(name)
 
         if not is_tty:
             write_record(tag.to_json())
@@ -90,42 +90,17 @@ def create_tags(names: Iterable[str]) -> int:
 # =============================================================================
 
 
-def list_tags(
-    name: str | None = None,
-    name__icontains: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """
-    List Tags as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.misc.jsonl import write_record
+def list_tags(name: str | None = None, name__icontains: str | None = None, limit: int | None = None) -> int:
+    """List tags as JSONL, or formatted rows in a terminal."""
     from archivebox.core.models import Tag
+    from archivebox.cli.cli_util import list_records
 
-    is_tty = sys.stdout.isatty()
-
-    queryset = Tag.objects.all().order_by("name")
-
-    # Apply filters
-    filter_kwargs = {
-        "name": name,
-        "name__icontains": name__icontains,
-    }
-    queryset = apply_filters(queryset, filter_kwargs, limit=limit)
-
-    count = 0
-    for tag in queryset:
-        snapshot_count = tag.snapshot_set.count()
-        if is_tty:
-            rprint(f"[cyan]{tag.name:30}[/cyan] [dim]({snapshot_count} snapshots)[/dim]")
-        else:
-            write_record(tag.to_json())
-        count += 1
-
-    rprint(f"[dim]Listed {count} tags[/dim]", file=sys.stderr)
-    return 0
+    queryset = apply_filters(Tag.objects.order_by("name"), {"name": name, "name__icontains": name__icontains}, limit=limit)
+    return list_records(
+        queryset,
+        plural="tags",
+        render=lambda tag: f"[cyan]{tag.name:30}[/cyan] [dim]({tag.snapshot_set.count()} snapshots)[/dim]",
+    )
 
 
 # =============================================================================
@@ -134,56 +109,15 @@ def list_tags(
 
 
 def update_tags(name: str | None = None) -> int:
-    """
-    Update Tags from stdin JSONL.
-
-    Reads Tag records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
-    from archivebox.misc.jsonl import read_stdin, write_record
+    """Apply supplied fields to each JSONL-selected Tag."""
     from archivebox.core.models import Tag
+    from archivebox.cli.cli_util import update_records
 
-    is_tty = sys.stdout.isatty()
+    def update(tag):
+        if name:
+            tag.rename(name)
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    updated_count = 0
-    for record in records:
-        tag_id = record.get("id")
-        old_name = record.get("name")
-
-        if not tag_id and not old_name:
-            continue
-
-        try:
-            if tag_id:
-                tag = Tag.objects.get(id=tag_id)
-            else:
-                tag = Tag.objects.get(name=old_name)
-
-            # Apply updates from CLI flags
-            if name:
-                tag.name = name
-                tag.save()
-
-            updated_count += 1
-
-            if not is_tty:
-                write_record(tag.to_json())
-
-        except Tag.DoesNotExist:
-            rprint(f"[yellow]Tag not found: {tag_id or old_name}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} tags[/green]", file=sys.stderr)
-    return 0
+    return update_records(Tag, update, plural="tags", by_name=True)
 
 
 # =============================================================================
@@ -192,65 +126,19 @@ def update_tags(name: str | None = None) -> int:
 
 
 def delete_tags(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete Tags from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
+    """Delete tags selected by stdin JSONL; --yes confirms, --dry-run previews."""
+    from archivebox.cli.cli_util import delete_records
     from archivebox.core.models import Tag
 
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    # Collect tag IDs or names
-    tag_ids = []
-    tag_names = []
-    for r in records:
-        if r.get("id"):
-            tag_ids.append(r["id"])
-        elif r.get("name"):
-            tag_names.append(r["name"])
-
-    if not tag_ids and not tag_names:
-        rprint("[yellow]No valid tag IDs or names in input[/yellow]", file=sys.stderr)
-        return 1
-
-    from django.db.models import Q
-
-    query = Q()
-    if tag_ids:
-        query |= Q(id__in=tag_ids)
-    if tag_names:
-        query |= Q(name__in=tag_names)
-
-    tags = Tag.objects.filter(query)
-    count = tags.count()
-
-    if count == 0:
-        rprint("[yellow]No matching tags found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} tags (dry run)[/yellow]", file=sys.stderr)
-        for tag in tags:
-            rprint(f"  {tag.name}", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Perform deletion
-    deleted_count, _ = tags.delete()
-    rprint(f"[green]Deleted {deleted_count} tags[/green]", file=sys.stderr)
-    return 0
+    return delete_records(
+        Tag,
+        label="tag",
+        plural="tags",
+        preview=lambda obj: obj.name,
+        yes=yes,
+        dry_run=dry_run,
+        by_name=True,
+    )
 
 
 # =============================================================================
@@ -275,24 +163,24 @@ def create_cmd(names: tuple):
 @click.option("--name", help="Filter by exact name")
 @click.option("--name__icontains", help="Filter by name contains")
 @click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(name: str | None, name__icontains: str | None, limit: int | None):
+def list_cmd(**kwargs):
     """List Tags as JSONL."""
-    sys.exit(list_tags(name=name, name__icontains=name__icontains, limit=limit))
+    sys.exit(list_tags(**kwargs))
 
 
 @main.command("update")
 @click.option("--name", "-n", help="Set new name")
-def update_cmd(name: str | None):
+def update_cmd(**kwargs):
     """Update Tags from stdin JSONL."""
-    sys.exit(update_tags(name=name))
+    sys.exit(update_tags(**kwargs))
 
 
 @main.command("delete")
 @click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
 @click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(yes: bool, dry_run: bool):
+def delete_cmd(**kwargs):
     """Delete Tags from stdin JSONL."""
-    sys.exit(delete_tags(yes=yes, dry_run=dry_run))
+    sys.exit(delete_tags(**kwargs))
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_update.py

@@ -77,107 +77,71 @@ def reindex_snapshots(
 ) -> dict[str, Any]:
     from archivebox.cli.archivebox_extract import run_plugins
     from archivebox.core.models import ArchiveResult, Snapshot
-    from abx_dl.models import discover_plugins
+    from django.db.models import Exists, OuterRef
 
     # Search backfill is the one maintenance hook allowed to execute without
     # reopening a Snapshot. Restrict that exception to already-sealed rows;
-    # every open lifecycle state remains owned by the normal state machine.
+    # every open lifecycle state remains owned by the normal runner lifecycle.
     snapshots = snapshots.filter(status=Snapshot.StatusChoices.SEALED)
 
-    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "skipped_queued": 0, "reindexed": 0, "snapshot_ids": []}
-    records: list[dict[str, str]] = []
-    plugins_by_name = discover_plugins(runtime="archivebox")
-    required_hooks_by_plugin = {
-        plugin_name: frozenset(hook.name for hook in plugins_by_name[plugin_name].filter_hooks("Snapshot"))
-        for plugin_name in search_plugins
-        if plugin_name in plugins_by_name
-    }
+    stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
+    print(f"[*] Reindexing missing search indexes with: {', '.join(search_plugins)}")
 
-    total = snapshots.count()
-    print(f"[*] Reindexing {total} snapshots with search plugins: {', '.join(search_plugins)}")
-
-    def run_batch() -> None:
-        if not records:
-            return
-        if wait_for_turn:
-            wait_for_turn()
-        batch_records = list(records)
-        snapshot_ids = {record["snapshot_id"] for record in batch_records}
-        plugin_names = {record["plugin"] for record in batch_records}
-        queued_rows = {
-            (str(snapshot_id), plugin_name, hook_name)
-            for snapshot_id, plugin_name, hook_name in ArchiveResult.objects.filter(
-                snapshot_id__in=snapshot_ids,
-                plugin__in=plugin_names,
-                status=ArchiveResult.StatusChoices.QUEUED,
-            ).values_list("snapshot_id", "plugin", "hook_name")
-        }
-        records_to_queue = []
-        for record in batch_records:
-            snapshot_id = record["snapshot_id"]
-            plugin_name = record["plugin"]
-            required_hooks = required_hooks_by_plugin.get(plugin_name, frozenset())
-            if required_hooks and all((snapshot_id, plugin_name, hook_name) in queued_rows for hook_name in required_hooks):
-                stats["skipped_queued"] += 1
-                continue
-            records_to_queue.append(record)
-        if not records_to_queue:
-            print(
-                f"    [{stats['processed']}/{total}] Already queued {len(batch_records)} index jobs",
-            )
-            records.clear()
-            return
-        # `archivebox update --index-only` intentionally breaks the usual
-        # "runner discovers work" rule by inserting synthetic queued
-        # ArchiveResult rows for search backends. run_plugins() keeps this as
-        # statement-sized UPDATE/bulk_create work, then bumps Snapshot.retry_at
-        # so the orchestrator owns actual hook execution. Paused snapshots stay
-        # PAUSED; run_due_snapshot restores retry_at=MAX after targeted rows
-        # finish.
-        exit_code = run_plugins(
-            args=(),
-            records=records_to_queue,
-            wait=False,
-            emit_results=False,
-            show_progress=False,
-            preserve_queued=True,
+    completed_statuses = [ArchiveResult.StatusChoices.SUCCEEDED, ArchiveResult.StatusChoices.NORESULTS]
+    for plugin_name in search_plugins:
+        completed_result = ArchiveResult.objects.filter(
+            snapshot_id=OuterRef("pk"),
+            plugin=plugin_name,
+            status__in=completed_statuses,
         )
-        if exit_code != 0:
-            raise SystemExit(exit_code)
-        stats["queued"] += len(records_to_queue)
-        print(
-            f"    [{stats['processed']}/{total}] Queued {len(records_to_queue)} index jobs for orchestrator",
-        )
-        records.clear()
-
-    for snapshot in snapshots.select_related("crawl").paged_iterator(chunk_size=batch_size):
-        try:
-            stats["processed"] += 1
-
-            if _get_snapshot_crawl(snapshot) is None:
-                continue
-
+        candidates = snapshots.annotate(has_completed_index=Exists(completed_result)).filter(has_completed_index=False).order_by("id")
+        after_id = None
+        while True:
+            if wait_for_turn:
+                wait_for_turn()
+            page = candidates.filter(id__gt=after_id) if after_id is not None else candidates
+            batch = list(page.select_related(None).only("id", "timestamp")[:batch_size])
+            if not batch:
+                break
+            after_id = batch[-1].id
+            records = [{"type": "Snapshot", "id": str(snapshot.id)} for snapshot in batch]
+            stats["processed"] += len(batch)
+            stats["requested"] += len(batch)
             if collect_ids:
-                stats["snapshot_ids"].append(str(snapshot.id))
-            for plugin_name in search_plugins:
-                records.append(
-                    {
-                        "type": "ArchiveResult",
-                        "snapshot_id": str(snapshot.id),
-                        "plugin": plugin_name,
-                    },
-                )
-                stats["requested"] += 1
-            if len(records) >= batch_size:
-                run_batch()
-        except KeyboardInterrupt as err:
-            err.archivebox_resume = snapshot.timestamp
-            raise
-
-    run_batch()
+                stats["snapshot_ids"].extend(str(snapshot.id) for snapshot in batch)
+            exit_code = run_plugins(
+                args=(),
+                records=records,
+                plugins=plugin_name,
+                wait=True,
+                emit_results=False,
+                show_progress=False,
+            )
+            if exit_code != 0:
+                raise SystemExit(exit_code)
+            stats["reindexed"] += len(batch)
+            print(f"    [{plugin_name}] indexed {stats['reindexed']} missing snapshots")
     return stats
 
 
+def run_scheduled_maintenance(*, batch_size: int = 500) -> dict[str, Any]:
+    """Queue stale filesystem rows and backfill missing search facts."""
+    from archivebox.core.models import Snapshot
+
+    filesystem_stats = process_all_db_snapshots(batch_size=batch_size)
+    search_plugins = _get_search_indexing_plugins()
+    search_stats = (
+        reindex_snapshots(
+            Snapshot.objects.filter(fs_version=Snapshot._fs_current_version()),
+            search_plugins=search_plugins,
+            batch_size=batch_size,
+        )
+        if search_plugins
+        else {"processed": 0, "requested": 0, "queued": 0, "reindexed": 0, "snapshot_ids": []}
+    )
+    return {"filesystem": filesystem_stats, "search": search_stats}
+
+
 @enforce_types
 def update(
     filter_patterns: Iterable[str] = (),
@@ -203,9 +167,9 @@ def update(
     Update snapshots: migrate old dirs, reconcile DB, and re-queue for archiving.
 
     Three-phase operation (without filters):
-    - Phase 1: Drain old archive/ dirs by moving to new fs location (0.8.x → 0.9.x)
-    - Phase 2: O(n) scan over entire DB from most recent to least recent
-    - No orphan scans needed (trust 1:1 mapping between DB and filesystem after phase 1)
+    - Phase 1: Drain legacy archive/ directories into the current layout
+    - Phase 2: Select only stale fs_version rows through the indexed column
+    - Phase 3: Run queued snapshot-level filesystem maintenance until idle
 
     With filters: Only phase 2 (DB query), no filesystem operations.
     Without filters: All phases (full update).
@@ -232,7 +196,6 @@ def update(
     from archivebox.core.takeover_util import (
         command_owns_foreground_runner,
         current_command,
-        foreground_runner_owner,
         standby_until_foreground_runner_needed,
     )
     from archivebox.workers.supervisord_util import run_runner_worker, stop_own_supervisord_process
@@ -266,21 +229,20 @@ def update(
                 continue
             raise SystemExit(exit_code)
 
-    is_filtered_update = any(
-        (
-            filter_patterns,
-            status,
-            url__icontains,
-            url__istartswith,
-            tag,
-            crawl_id,
-            limit,
-            sort,
-            search,
-            before,
-            after,
-        ),
-    )
+    filters = {
+        "filter_patterns": filter_patterns,
+        "status": status,
+        "url__icontains": url__icontains,
+        "url__istartswith": url__istartswith,
+        "tag": tag,
+        "crawl_id": crawl_id,
+        "limit": limit,
+        "sort": sort,
+        "search": search,
+        "before": before,
+        "after": after,
+    }
+    is_filtered_update = any(filters.values())
     touched_snapshot_ids: set[str] = set()
     exit_code = 0
 
@@ -291,49 +253,20 @@ def update(
             while True:
                 do_migrate = migrate_only or not index_only
                 do_index = index_only or not migrate_only
-                do_run_until_idle = do_migrate or do_index
-                ran_post_migrate_runner = False
-                full_update_empty = False
-                maintenance_work_queued = False
-                runner_work_queued = False
 
                 if do_migrate:
-                    if (
-                        filter_patterns
-                        or status
-                        or url__icontains
-                        or url__istartswith
-                        or tag
-                        or crawl_id
-                        or limit
-                        or sort
-                        or search
-                        or before
-                        or after
-                    ):
+                    if is_filtered_update:
                         print("[*] Processing filtered snapshots from database...")
                         stats = process_filtered_snapshots(
-                            filter_patterns=filter_patterns,
+                            **filters,
                             filter_type=filter_type,
-                            status=status,
-                            url__icontains=url__icontains,
-                            url__istartswith=url__istartswith,
-                            tag=tag,
-                            crawl_id=crawl_id,
-                            limit=limit,
-                            sort=sort,
-                            search=search,
-                            before=before,
-                            after=after,
                             resume=resume,
                             batch_size=batch_size,
-                            queue_for_archiving=do_run_until_idle,
+                            queue_for_archiving=True,
                             wait_for_turn=wait_for_turn,
                         )
                         print_stats(stats)
                         touched_snapshot_ids.update(stats.get("snapshot_ids", []))
-                        maintenance_work_queued = stats.get("queued", 0) > 0
-                        runner_work_queued = runner_work_queued or maintenance_work_queued
                     else:
                         stats_combined = {"phase1": {}, "phase2": {}}
 
@@ -343,162 +276,44 @@ def update(
                             batch_size=batch_size,
                         )
 
-                        print("[*] Phase 2: Processing all database snapshots (most recent first)...")
+                        print("[*] Phase 2: Selecting database snapshots with stale filesystem versions...")
                         stats_combined["phase2"] = process_all_db_snapshots(
                             batch_size=batch_size,
                             resume=resume,
                             wait_for_turn=wait_for_turn,
                         )
                         print_combined_stats(stats_combined)
-                        full_update_empty = (
-                            stats_combined["phase1"].get("processed", 0) == 0 and stats_combined["phase2"].get("snapshots", 0) == 0
-                        )
-                        maintenance_work_queued = any(
-                            (
-                                stats_combined["phase1"].get("queued", 0),
-                                stats_combined["phase2"].get("queued", 0),
-                                stats_combined["phase2"].get("crawls_sealed", 0),
-                            ),
-                        )
-                        runner_work_queued = runner_work_queued or maintenance_work_queued
-
-                    if do_run_until_idle:
-                        # Filesystem migration is maintenance on existing
-                        # Snapshot rows: Snapshot.save() moves archive/<ts> to
-                        # the current output_dir and preserves the lifecycle
-                        # status. Drain those retry_at ticks before queuing
-                        # search backfill below. Otherwise the sealed search
-                        # runner branch correctly sees queued ArchiveResult
-                        # rows first, runs the targeted plugins, and may leave
-                        # the fs_version maintenance tick hidden behind that
-                        # plugin work until another update pass.
-                        if full_update_empty:
-                            print("[*] No snapshots or legacy archive directories found; skipping filesystem maintenance runner.")
-                        elif not maintenance_work_queued:
-                            print("[*] No filesystem maintenance work queued; skipping filesystem maintenance runner.")
-                        else:
-                            print("[*] Phase 3: Running filesystem maintenance until idle...")
-                        if full_update_empty:
-                            pass
-                        elif not maintenance_work_queued:
-                            pass
-                        elif is_filtered_update:
-                            if not touched_snapshot_ids:
-                                print("[*] No matching snapshots queued work for the runner.")
-                            for snapshot_id in sorted(touched_snapshot_ids):
-                                run_scoped_runner("--snapshot-id", snapshot_id)
-                        else:
-                            run_scoped_runner("--maintenance-only", "--maintenance-batch-size", str(batch_size))
-                        ran_post_migrate_runner = True
-
-                if do_index:
-                    if full_update_empty:
-                        print("[*] No snapshots found; skipping search indexing backfill.")
-                    else:
-                        search_plugins = _get_search_indexing_plugins()
-                        if not search_plugins:
-                            print("[*] No search indexing plugins are available, nothing to backfill.")
-                        else:
-                            snapshots = _build_filtered_snapshots_queryset(
-                                filter_patterns=filter_patterns,
-                                filter_type=filter_type,
-                                status=status,
-                                url__icontains=url__icontains,
-                                url__istartswith=url__istartswith,
-                                tag=tag,
-                                crawl_id=crawl_id,
-                                limit=limit,
-                                sort=sort,
-                                search=search,
-                                before=before,
-                                after=after,
-                                resume=resume,
-                            )
-                            from django.db.models import Exists, OuterRef, Q
-                            from django.utils import timezone
-                            from archivebox.core.models import ArchiveResult, Snapshot
-
-                            scoped_snapshot_ids = snapshots.order_by().values("id") if is_filtered_update else None
-                            queued_index_results = ArchiveResult.objects.filter(
-                                status=ArchiveResult.StatusChoices.QUEUED,
-                                plugin__in=search_plugins,
-                            )
-                            if scoped_snapshot_ids is not None:
-                                queued_index_results = queued_index_results.filter(snapshot_id__in=scoped_snapshot_ids)
-
-                            if queued_index_results.exists():
-                                runner_work_queued = True
-                                now = timezone.now()
-                                queued_result_for_snapshot = queued_index_results.filter(snapshot_id=OuterRef("pk"))
-                                snapshots_to_wake = (
-                                    Snapshot.objects.filter(status=Snapshot.StatusChoices.SEALED)
-                                    .annotate(
-                                        has_queued_index_result=Exists(queued_result_for_snapshot),
-                                    )
-                                    .filter(
-                                        has_queued_index_result=True,
-                                    )
-                                    .filter(
-                                        Q(retry_at__isnull=True) | Q(retry_at__gt=now),
-                                    )
-                                )
-                                if scoped_snapshot_ids is not None:
-                                    snapshots_to_wake = snapshots_to_wake.filter(id__in=scoped_snapshot_ids)
-                                woken_count = snapshots_to_wake.update(
-                                    retry_at=now,
-                                    modified_at=now,
-                                )
-                                print(
-                                    "[*] Existing queued search index jobs found; "
-                                    f"skipping backfill scan and waking {woken_count} snapshot(s) for the runner.",
-                                )
-                            else:
-                                collect_index_ids = (
-                                    is_filtered_update
-                                    or foreground_runner_owner(
-                                        data_dir=CONSTANTS.DATA_DIR,
-                                        exclude_id=command.id,
-                                    )
-                                    is not None
-                                )
-                                stats = reindex_snapshots(
-                                    snapshots,
-                                    search_plugins=search_plugins,
-                                    batch_size=batch_size,
-                                    collect_ids=collect_index_ids,
-                                    wait_for_turn=wait_for_turn,
-                                )
-                                print_index_stats(stats)
-                                touched_snapshot_ids.update(stats.get("snapshot_ids", []))
-                                runner_work_queued = runner_work_queued or stats["queued"] > 0
-
-                if do_run_until_idle and (do_index or not ran_post_migrate_runner):
-                    # Search/index backfill intentionally queues targeted
-                    # ArchiveResult rows without reopening sealed snapshots.
-                    # This second runner pass drains those plugin
-                    # rows after filesystem maintenance has had its own turn.
-                    # For a normal unfiltered `archivebox update`, keep the
-                    # historical final pass broad enough to resume genuinely
-                    # queued/interrupted crawl work after maintenance is done.
-                    if full_update_empty:
-                        print("[*] No snapshots found; skipping queued/interrupted crawl runner.")
-                    elif not runner_work_queued:
-                        print("[*] No queued/interrupted crawl work found; skipping queued/interrupted crawl runner.")
-                    else:
-                        print("[*] Phase 3: Running queued/interrupted crawl work until idle...")
-                    if full_update_empty:
-                        pass
-                    elif not runner_work_queued:
-                        pass
-                    elif touched_snapshot_ids and is_filtered_update:
-                        if not touched_snapshot_ids:
-                            print("[*] No matching snapshots queued work for the runner.")
+                    # The due selectors are indexed and cheap when empty, so
+                    # always drain them instead of preceding the runner with
+                    # whole-table counts merely to decide whether to call it.
+                    print("[*] Phase 3: Running filesystem maintenance until idle...")
+                    if is_filtered_update:
                         for snapshot_id in sorted(touched_snapshot_ids):
                             run_scoped_runner("--snapshot-id", snapshot_id)
+                    elif migrate_only:
+                        run_scoped_runner("--maintenance-only")
                     else:
-                        run_scoped_runner(
-                            *(["--maintenance-only", "--maintenance-batch-size", str(batch_size)] if index_only or migrate_only else []),
+                        run_scoped_runner()
+
+                if do_index:
+                    search_plugins = _get_search_indexing_plugins()
+                    if not search_plugins:
+                        print("[*] No search indexing plugins are available, nothing to backfill.")
+                    else:
+                        snapshots = _build_filtered_snapshots_queryset(
+                            **filters,
+                            filter_type=filter_type,
+                            resume=resume,
                         )
+                        stats = reindex_snapshots(
+                            snapshots,
+                            search_plugins=search_plugins,
+                            batch_size=batch_size,
+                            collect_ids=is_filtered_update,
+                            wait_for_turn=wait_for_turn,
+                        )
+                        print_index_stats(stats)
+                        touched_snapshot_ids.update(stats.get("snapshot_ids", []))
 
                 if not continuous:
                     break
@@ -514,32 +329,17 @@ def update(
             resume_cmd.append("--migrate-only")
         if index_only:
             resume_cmd.append("--index-only")
-        if batch_size != 500:
-            resume_cmd.extend(["--batch-size", str(batch_size)])
-        if exact_resume or resume:
-            resume_cmd.extend(["--resume", str(exact_resume or resume)])
-        if before is not None:
-            resume_cmd.extend(["--before", str(before)])
-        if after is not None:
-            resume_cmd.extend(["--after", str(after)])
-        if filter_type != "exact":
-            resume_cmd.extend(["--filter-type", filter_type])
-        if status:
-            resume_cmd.extend(["--status", status])
-        if url__icontains:
-            resume_cmd.extend(["--url__icontains", url__icontains])
-        if url__istartswith:
-            resume_cmd.extend(["--url__istartswith", url__istartswith])
-        if tag:
-            resume_cmd.extend(["--tag", tag])
-        if crawl_id:
-            resume_cmd.extend(["--crawl-id", crawl_id])
-        if limit:
-            resume_cmd.extend(["--limit", str(limit)])
-        if sort:
-            resume_cmd.extend(["--sort", sort])
-        if search:
-            resume_cmd.extend(["--search", search])
+        resume_options = {
+            "batch-size": batch_size if batch_size != 500 else None,
+            "resume": exact_resume or resume,
+            "filter-type": filter_type if filter_type != "exact" else None,
+            **{key: value for key, value in filters.items() if key != "filter_patterns"},
+        }
+        for name, value in resume_options.items():
+            # Zero is a meaningful date boundary, but remains an unset limit.
+            if value or (name in {"before", "after"} and value is not None):
+                option = "crawl-id" if name == "crawl_id" else name
+                resume_cmd.extend([f"--{option}", str(value)])
         resume_cmd.extend(str(pattern) for pattern in filter_patterns)
         print("\n[red][X] archivebox update interrupted.[/red]")
         print("[yellow]Hint: resume this idempotent update with:[/yellow]")
@@ -561,21 +361,20 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
     """
     Drain old archive/ directories (0.8.x → 0.9.x migration).
 
-    Only processes real directories (skips symlinks - those are already migrated).
+    Removes obsolete timestamp symlinks and processes real legacy directories.
     For each old dir found in archive/:
       1. Load or create DB snapshot
       2. Trigger fs migration on save() to move to data/archive/users/{user}/...
-      3. Leave symlink in archive/ pointing to new location
+      3. Remove the old timestamp path after the verified migration commits
 
-    After this drains, archive/ should only contain symlinks and we can trust
-    1:1 mapping between DB and filesystem.
+    After this drains, current snapshot data exists only under archive/users/.
     """
     from archivebox.core.models import Snapshot
     from archivebox.config import CONSTANTS
     from archivebox.crawls.models import Crawl
     from django.utils import timezone
 
-    stats = {"processed": 0, "migrated": 0, "queued": 0, "skipped": 0, "invalid": 0}
+    stats = {"processed": 0, "migrated": 0, "queued": 0, "skipped": 0, "invalid": 0, "removed_symlinks": 0}
     crawl_url_lines: dict[str, list[str]] = {}
     crawl_url_sets: dict[str, set[str]] = {}
     dirty_crawl_ids: set[str] = set()
@@ -606,8 +405,23 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
             if changed:
                 Crawl.objects.filter(pk=crawl.pk).update(urls="\n".join(lines), modified_at=timezone.now())
 
-    # Scan for real directories only (skip symlinks - they're already migrated)
+    # Compatibility timestamp projections are harmful in portable exports and
+    # duplicate the obsolete 0.7.x-looking namespace without containing data.
     all_entries = list(os.scandir(archive_dir))
+    for entry in all_entries:
+        entry_path = Path(entry.path)
+        if entry.is_symlink() and Snapshot.is_legacy_archive_dir(entry_path):
+            try:
+                target_path = entry_path.resolve(strict=True)
+                points_into_current_layout = target_path.is_relative_to(CONSTANTS.USERS_DIR.resolve(strict=True))
+            except OSError:
+                points_into_current_layout = False
+
+            if points_into_current_layout:
+                entry_path.unlink(missing_ok=True)
+                stats["removed_symlinks"] += 1
+
+    # Scan real legacy directories only; these still contain data to migrate.
     entries = [
         (e.stat().st_mtime, e.path)
         for e in all_entries
@@ -723,18 +537,8 @@ def drain_old_archive_dirs(resume_from: str | None = None, batch_size: int = 500
 
 
 def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, wait_for_turn=None) -> dict[str, int]:
-    """
-    O(n) scan over entire DB from most recent to least recent.
-
-    For each snapshot:
-      1. Reconcile index.json with DB (merge titles, tags, archive results)
-      2. Mark migrated snapshots sealed unless explicitly re-queued elsewhere
-
-    No orphan detection needed - we trust 1:1 mapping between DB and filesystem
-    after Phase 1 has drained all old archive/ directories.
-    """
+    """Queue only snapshots whose indexed filesystem version is stale."""
     from archivebox.core.models import Snapshot
-    from archivebox.crawls.models import Crawl
     from django.db.models import Q
     from django.utils import timezone
 
@@ -747,128 +551,35 @@ def process_all_db_snapshots(batch_size: int = 500, resume: str | None = None, w
         "sealed": 0,
         "crawls_sealed": 0,
     }
-    current_fs_version = Snapshot._fs_current_version()
-
-    queryset = Snapshot.objects.all()
+    queryset = Snapshot.objects.filter(fs_version__in=Snapshot._FS_VERSION_MIGRATION_PATHS)
     if resume:
         queryset = queryset.filter(timestamp__lte=resume)
-    total = queryset.count()
-    stats["snapshots"] = total
-    print(f"[*] Processing {total} snapshots from database (most recent first)...")
-
-    def update_in_batches(rows, *, label: str, **updates) -> int:
-        updated = 0
-        checked = 0
-        while True:
-            if wait_for_turn:
-                wait_for_turn()
-            batch = list(rows.only("id", "modified_at").order_by("-timestamp")[:batch_size])
-            if not batch:
-                if updated:
-                    print(f"    [{label}] complete: {updated} rows updated")
-                return updated
-            checked += len(batch)
-            print(f"    [{label}] updating next {len(batch)} rows (seen {checked})...")
-            for snapshot in batch:
-                # This maintenance scan intentionally bypasses save(); it is
-                # only normalizing scheduler fields, and Snapshot.save() may
-                # do filesystem migration work that belongs in the runner.
-                # Guard each single-row UPDATE with modified_at so stale scan
-                # pages cannot overwrite newer runner/admin writes.
-                updated += int(
-                    snapshot.safe_update(
-                        updates,
-                        refresh=False,
-                        extra_filter={"modified_at": snapshot.modified_at},
-                    ),
-                )
-            print(f"    [{label}] updated {updated} rows so far")
-
-    now = timezone.now()
-    updated_rows = update_in_batches(
-        queryset.exclude(
-            status__in=[
-                Snapshot.StatusChoices.QUEUED,
-                Snapshot.StatusChoices.STARTED,
-                Snapshot.StatusChoices.PAUSED,
-                Snapshot.StatusChoices.SEALED,
-            ],
-        ),
-        label="snapshot status normalization",
-        status=Snapshot.StatusChoices.SEALED,
-        retry_at=None,
-        modified_at=now,
-    )
-    stats["sealed"] += updated_rows
-    stats["updated_db"] += updated_rows
-
-    fs_version_rows = queryset.exclude(fs_version=current_fs_version).filter(Q(retry_at__isnull=True) | Q(retry_at__gt=now))
-    stale_batch = []
-
-    def queue_stale_fs_batch() -> None:
-        if not stale_batch:
-            return
+    initial_now = timezone.now()
+    rows_to_wake = queryset.filter(Q(retry_at__isnull=True) | Q(retry_at__gt=initial_now))
+    after_id = None
+    while True:
         if wait_for_turn:
             wait_for_turn()
+        page = rows_to_wake.filter(id__gt=after_id) if after_id is not None else rows_to_wake
+        batch = list(page.only("id", "fs_version", "modified_at").order_by("id")[:batch_size])
+        if not batch:
+            break
+        after_id = batch[-1].id
         now = timezone.now()
-        # Do not bump fs_version here. The orchestrator calls Snapshot.save(),
-        # which performs the idempotent filesystem migration and commits the new
-        # fs_version in the same serialized worker path as normal crawls.
         updated = 0
-        for snapshot in stale_batch:
-            # Each row gets its own short autocommit UPDATE because this scan
-            # can touch millions of snapshots while a server is also alive.
-            # The modified_at predicate is the CAS guard: if the runner or
-            # admin changed the snapshot after paged_iterator read it, skip it
-            # and let the newer state decide whether migration is still due.
+        for snapshot in batch:
             updated += int(
                 snapshot.safe_update(
-                    {
-                        "retry_at": now,
-                        "modified_at": now,
-                    },
+                    {"retry_at": now, "modified_at": now},
                     refresh=False,
                     extra_filter={"fs_version": snapshot.fs_version},
                 ),
             )
-        stats["processed"] += len(stale_batch)
+        stats["processed"] += len(batch)
         stats["updated_db"] += updated
         stats["queued"] += updated
-        print(f"    [{stats['processed']}/{total}] Queued {updated} filesystem migrations for orchestrator...")
-        stale_batch.clear()
-
-    for snapshot in (
-        fs_version_rows.only("id", "crawl_id", "timestamp", "fs_version", "modified_at")
-        .order_by("-timestamp")
-        .paged_iterator(chunk_size=batch_size)
-    ):
-        try:
-            stale_batch.append(snapshot)
-            if len(stale_batch) >= batch_size:
-                queue_stale_fs_batch()
-        except KeyboardInterrupt as err:
-            err.archivebox_resume = snapshot.timestamp
-            raise
-    queue_stale_fs_batch()
-
-    now = timezone.now()
-    # Crawls with no open child snapshots are already finished. Seal them here
-    # instead of waking the foreground runner; otherwise migration/update can
-    # accidentally re-enter full crawl execution for historical rows.
-    stats["crawls_sealed"] = (
-        Crawl.objects.filter(
-            status__in=Crawl.RUNNABLE_STATES,
-        )
-        .exclude(
-            snapshot_set__status__in=Snapshot.OPEN_STATES,
-        )
-        .update(
-            status=Crawl.StatusChoices.SEALED,
-            retry_at=None,
-            modified_at=now,
-        )
-    )
-    stats["updated_db"] += stats["crawls_sealed"]
+        print(f"    Queued {stats['queued']} stale filesystem snapshots so far...")
+    stats["snapshots"] = stats["processed"]
     return stats
 
 
@@ -928,8 +639,6 @@ def process_filtered_snapshots(
             stats["snapshot_ids"].append(str(snapshot.id))
             update_values = {}
             updated = 0
-            if not isinstance(snapshot.current_step, int):
-                update_values["current_step"] = 0
             if queue_for_archiving:
                 update_values.update(
                     {
@@ -1015,10 +724,9 @@ def print_index_stats(stats: dict[str, Any]) -> None:
 
     print(f"""
 [green]Search Reindex Complete[/green]
-  Scanned rows:      {stats["processed"]}
-  Requested jobs:    {stats.get("requested", stats["queued"])}
-  Queued index jobs: {stats["queued"]}
-  Already queued:    {stats.get("skipped_queued", 0)}
+  Missing index runs: {stats["processed"]}
+  Requested runs:    {stats.get("requested", 0)}
+  Indexed snapshots: {stats.get("reindexed", 0)}
 """)
 
 

archivebox/cli/archivebox_version.py

@@ -7,6 +7,7 @@ import os
 import json
 import asyncio
 import platform
+import re
 from pathlib import Path
 from collections.abc import Iterable
 
@@ -46,13 +47,7 @@ def _format_binary_abspath(
                 continue
 
             relative_str = relative.as_posix()
-            if prefix == "./":
-                return "." if not relative_str else f"./{relative_str}"
-            if prefix == "ABXPKG_LIB_DIR/":
-                return "ABXPKG_LIB_DIR" if not relative_str else f"ABXPKG_LIB_DIR/{relative_str}"
-            if prefix == "PERSONAS_DIR/":
-                return "PERSONAS_DIR" if not relative_str else f"PERSONAS_DIR/{relative_str}"
-            return "~" if not relative_str else f"~/{relative_str}"
+            return prefix + relative_str if relative_str else prefix.rstrip("/")
 
     return normalized.as_posix()
 
@@ -60,22 +55,9 @@ def _format_binary_abspath(
 def _render_binary_abspath(abspath: str):
     from rich.text import Text
 
-    if abspath.startswith("ABXPKG_LIB_DIR/"):
-        return Text.assemble(("ABXPKG_LIB_DIR", "bright_blue"), (abspath.removeprefix("ABXPKG_LIB_DIR"), "green"))
-    if abspath == "ABXPKG_LIB_DIR":
-        return Text("ABXPKG_LIB_DIR", style="bright_blue")
-    if abspath.startswith("PERSONAS_DIR/"):
-        return Text.assemble(("PERSONAS_DIR", "medium_purple"), (abspath.removeprefix("PERSONAS_DIR"), "green"))
-    if abspath == "PERSONAS_DIR":
-        return Text("PERSONAS_DIR", style="medium_purple")
-    if abspath.startswith("~/"):
-        return Text.assemble(("~", "cyan"), (abspath.removeprefix("~"), "green"))
-    if abspath == "~":
-        return Text("~", style="cyan")
-    if abspath.startswith("./"):
-        return Text.assemble((".", "cyan"), (abspath.removeprefix("."), "green"))
-    if abspath == ".":
-        return Text(".", style="cyan")
+    for prefix, color in (("ABXPKG_LIB_DIR", "bright_blue"), ("PERSONAS_DIR", "medium_purple"), ("~", "cyan"), (".", "cyan")):
+        if abspath == prefix or abspath.startswith(prefix + "/"):
+            return Text.assemble((prefix, color), (abspath.removeprefix(prefix), "green"))
     return Text(abspath, style="green")
 
 
@@ -161,7 +143,8 @@ def version(
     )
 
     try:
-        OUTPUT_IS_REMOTE_FS = get_data_locations().DATA_DIR.is_mount or get_data_locations().ARCHIVE_DIR.is_mount
+        data_locations = get_data_locations(config=config)
+        OUTPUT_IS_REMOTE_FS = data_locations.DATA_DIR.is_mount or data_locations.ARCHIVE_DIR.is_mount
     except Exception:
         OUTPUT_IS_REMOTE_FS = False
 
@@ -199,8 +182,23 @@ def version(
     else:
         requested_names = {name for name in (binaries or ()) if name}
 
-    def binary_is_requested(logical_name: str, actual_name: str, display_name: str) -> bool:
-        return not requested_names or bool({logical_name, actual_name, display_name} & requested_names)
+    def plugin_may_have_requested_binary(plugin) -> bool:
+        if not requested_names:
+            return True
+        aliases = {plugin.name}
+        for required_binary in plugin.config.required_binaries:
+            raw_name = str(required_binary.name)
+            aliases.add(raw_name)
+            template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", raw_name)
+            if template_match:
+                config_key = template_match.group(1)
+                aliases.add(config_key)
+                aliases.add(config_key.removesuffix("_BINARY"))
+                default = plugin.config.properties.get(config_key, {}).get("default")
+                if default:
+                    aliases.add(str(default))
+        normalized_aliases = {alias.lower() for alias in aliases}
+        return bool({name.lower() for name in requested_names} & normalized_aliases)
 
     if not in_data_dir:
         PANEL_TEXT = "\n".join(
@@ -230,7 +228,7 @@ def version(
         prnt()
         prnt("[deep_sky_blue3][i] Code locations:[/deep_sky_blue3]")
         try:
-            for name, path in get_code_locations().items():
+            for name, path in get_code_locations(config=config).items():
                 if isinstance(name, str) and isinstance(path, dict):
                     prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
         except Exception as e:
@@ -242,18 +240,16 @@ def version(
 
     prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
     failures = []
-    seen_failures: set[str] = set()
     seen_rows: set[tuple[str, str, str, str]] = set()
 
-    from archivebox.plugins.discovery import get_enabled_plugins
+    from archivebox.plugins.discovery import get_enabled_plugins, get_plugin_catalog
     from abx_dl.config import get_required_binary_requests
     from abx_dl.dependencies import resolve_binary_requests
-    from abx_dl.models import discover_plugins, filter_plugins
     from abx_dl.orchestrator import create_bus
     from abxpkg.binary_service import BinaryEvent, BinaryService
 
-    plugins = discover_plugins(runtime="archivebox")
-    enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
+    plugins = get_plugin_catalog()
+    enabled_plugins = plugins.select(get_enabled_plugins(config=config))
     enabled_plugin_names = set(enabled_plugins)
     runtime_config = normalize_runtime_config(config.for_crawl(), json_safe=False)
     derived_config: dict[str, object] = {}
@@ -279,33 +275,33 @@ def version(
             prnt("", f"[yellow]Warning: Could not query collection binary records; resolving through abxpkg: {e}[/yellow]")
 
     declared_binary_specs: dict[str, dict[str, object]] = {}
+    plugin_binaries = []
     for plugin_name, plugin in plugins.items():
+        if not plugin_may_have_requested_binary(plugin):
+            continue
         plugin_enabled = plugin_name in enabled_plugin_names
-        logical_records = get_required_binary_requests(
+        binary_records = get_required_binary_requests(
             plugin,
             plugin.config.required_binaries,
             overrides=runtime_config,
             derived_overrides=derived_config,
             run_output_dir=CONSTANTS.DATA_DIR,
         )
-        actual_records = get_required_binary_requests(
-            plugin,
-            plugin.config.required_binaries,
-            overrides=runtime_config,
-            derived_overrides=derived_config,
-            run_output_dir=CONSTANTS.DATA_DIR,
-            logical_names=False,
-        )
-        for logical_record, actual_record in zip(logical_records, actual_records, strict=False):
-            logical_name = str(logical_record["name"])
-            actual_name = str(actual_record["name"])
-            display_name = Path(actual_name).expanduser().name if ("/" in actual_name or actual_name.startswith("~")) else logical_name
-            if not binary_is_requested(logical_name, actual_name, display_name):
-                continue
+        for binary_record in binary_records:
+            actual_name = str(binary_record["name"])
+            logical_name = (
+                Path(actual_name).expanduser().name
+                if ("/" in actual_name or "\\" in actual_name or actual_name.startswith("~"))
+                else actual_name
+            )
+            display_name = logical_name
+            plugin_binaries.append((plugin_name, plugin_enabled, display_name, binary_record))
             if not plugin_enabled and not requested_names:
                 continue
-            signature = json.dumps(actual_record, sort_keys=True, default=str)
-            declared_binary_specs.setdefault(signature, actual_record)
+            if _binary_record_matches_runtime(db_binaries.get(logical_name), config.ABXPKG_LIB_DIR):
+                continue
+            signature = json.dumps(binary_record, sort_keys=True, default=str)
+            declared_binary_specs.setdefault(signature, binary_record)
 
     loaded_binaries: dict[str, BinaryEvent | None] = {}
     if declared_binary_specs:
@@ -322,157 +318,81 @@ def version(
         loaded_binaries = asyncio.run(resolve_declared_binaries())
 
     rows: list[dict[str, object]] = []
-    any_rows = False
     any_available = False
     compact_paths = console.is_terminal
-    live_enabled = console.is_terminal
-    live_cm = Live(binary_dependency_table(rows), console=console, refresh_per_second=8) if live_enabled else None
+    live_cm = Live(binary_dependency_table(rows), console=console, refresh_per_second=8) if console.is_terminal else None
 
-    def emit_row(row: dict[str, object]) -> None:
-        rows.append(row)
+    def emit_binary(plugin_name, plugin_enabled, display_name, binary, *, dim=False):
+        nonlocal any_available
+        valid = binary is not None
+        abspath = binary.abspath if valid else ""
+        version_str = str(binary.version or "unknown")[:15] if valid else "-"
+        provider = str(binary.binprovider or "env")[:8] if valid else "-"
+        any_available |= valid
+        if not valid and plugin_enabled and display_name not in failures:
+            failures.append(display_name)
+        row_key = _binary_row_dedupe_key(
+            display_name=display_name,
+            valid=valid,
+            version=version_str,
+            provider=provider,
+            abspath=abspath,
+        )
+        if row_key in seen_rows:
+            return
+        seen_rows.add(row_key)
+        rendered_path = "[grey53]not installed[/grey53]"
+        if valid:
+            display_path = (
+                _format_binary_abspath(
+                    abspath,
+                    pwd=Path.cwd(),
+                    lib_dir=config.ABXPKG_LIB_DIR,
+                    personas_dir=CONSTANTS.PERSONAS_DIR,
+                    home=Path.home(),
+                )
+                if compact_paths
+                else abspath
+            )
+            rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
+        rows.append(
+            {
+                "plugin": plugin_name,
+                "status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
+                "binary": display_name,
+                "version": version_str,
+                "provider": provider,
+                "path": rendered_path,
+                "style": "dim" if dim or not plugin_enabled else "",
+            },
+        )
         if live_cm is not None:
             live_cm.update(binary_dependency_table(rows), refresh=True)
 
     if live_cm is not None:
         live_cm.start()
     try:
-        for plugin_name, plugin in plugins.items():
-            plugin_enabled = plugin_name in enabled_plugin_names
-            logical_records = get_required_binary_requests(
-                plugin,
-                plugin.config.required_binaries,
-                overrides=runtime_config,
-                derived_overrides=derived_config,
-                run_output_dir=CONSTANTS.DATA_DIR,
-            )
-            actual_records = get_required_binary_requests(
-                plugin,
-                plugin.config.required_binaries,
-                overrides=runtime_config,
-                derived_overrides=derived_config,
-                run_output_dir=CONSTANTS.DATA_DIR,
-                logical_names=False,
-            )
-            for logical_record, actual_record in zip(logical_records, actual_records, strict=False):
-                logical_name = str(logical_record["name"])
-                actual_name = str(actual_record["name"])
-                display_name = Path(actual_name).expanduser().name if ("/" in actual_name or actual_name.startswith("~")) else logical_name
-                if not binary_is_requested(logical_name, actual_name, display_name):
+        for plugin_name, plugin_enabled, display_name, binary_record in plugin_binaries:
+            installed = db_binaries.get(display_name) if db_available else None
+            if not _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
+                # Resolve only enabled or explicitly requested dependencies. Cold-loading
+                # every optional provider makes status and migration checks needlessly slow.
+                if not plugin_enabled and not requested_names:
                     continue
-
-                installed = db_binaries.get(logical_name) if db_available else None
-                if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
-                    abspath = installed.abspath
-                    version_str = (installed.version or "unknown")[:15]
-                    provider = (installed.binprovider or "env")[:8]
-                    valid = True
-                elif not plugin_enabled and not requested_names:
-                    # `archivebox version` is expected to verify the active runtime, not
-                    # cold-load every optional plugin provider. Migration and status
-                    # checks often run with PLUGINS narrowed to a tiny set; resolving
-                    # disabled plugin binaries there can spend most of the command on
-                    # providers the current collection will never execute.
-                    continue
-                else:
-                    loaded = loaded_binaries[json.dumps(actual_record, sort_keys=True, default=str)]
-                    abspath = loaded.abspath if loaded is not None else ""
-                    version_str = str(loaded.version or "unknown")[:15] if loaded is not None else "unknown"
-                    provider = str(loaded.binprovider or "env")[:8] if loaded is not None else "env"
-                    valid = loaded is not None
-
-                any_rows = True
-                if valid:
-                    display_path = (
-                        _format_binary_abspath(
-                            abspath,
-                            pwd=Path.cwd(),
-                            lib_dir=config.ABXPKG_LIB_DIR,
-                            personas_dir=CONSTANTS.PERSONAS_DIR,
-                            home=Path.home(),
-                        )
-                        if compact_paths
-                        else abspath
-                    )
-                    rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
-                    any_available = True
-                else:
-                    rendered_path = "[grey53]not installed[/grey53]"
-                    if plugin_enabled and display_name not in seen_failures:
-                        failures.append(display_name)
-                        seen_failures.add(display_name)
-
-                row_key = _binary_row_dedupe_key(
-                    display_name=display_name,
-                    valid=valid,
-                    version=version_str if valid else "-",
-                    provider=provider if valid else "-",
-                    abspath=abspath,
-                )
-                if row_key in seen_rows:
-                    continue
-                seen_rows.add(row_key)
-
-                emit_row(
-                    {
-                        "plugin": plugin_name,
-                        "status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
-                        "binary": display_name,
-                        "version": version_str if valid else "-",
-                        "provider": provider if valid else "-",
-                        "path": rendered_path,
-                        "style": "" if plugin_enabled else "dim",
-                    },
-                )
+                installed = loaded_binaries[json.dumps(binary_record, sort_keys=True, default=str)]
+            emit_binary(plugin_name, plugin_enabled, display_name, installed)
 
         if db_available:
             for binary_name, installed in db_binaries.items():
-                if requested_names and binary_name not in requested_names:
-                    continue
-                abspath = installed.abspath
-                version_str = (installed.version or "unknown")[:15]
-                provider = (installed.binprovider or "env")[:8]
-                row_key = _binary_row_dedupe_key(
-                    display_name=binary_name,
-                    valid=True,
-                    version=version_str,
-                    provider=provider,
-                    abspath=abspath,
-                )
-                if row_key in seen_rows:
-                    continue
-                seen_rows.add(row_key)
-
-                display_path = (
-                    _format_binary_abspath(
-                        abspath,
-                        pwd=Path.cwd(),
-                        lib_dir=config.ABXPKG_LIB_DIR,
-                        personas_dir=CONSTANTS.PERSONAS_DIR,
-                        home=Path.home(),
-                    )
-                    if compact_paths
-                    else abspath
-                )
-                emit_row(
-                    {
-                        "plugin": "(database)",
-                        "status": binary_dependency_status(enabled=True, valid=True),
-                        "binary": binary_name,
-                        "version": version_str,
-                        "provider": provider,
-                        "path": _render_binary_abspath(display_path) if compact_paths else display_path,
-                        "style": "dim",
-                    },
-                )
-                any_rows = True
-                any_available = True
+                if not requested_names or binary_name in requested_names:
+                    emit_binary("(database)", True, binary_name, installed, dim=True)
     finally:
         if live_cm is not None:
             live_cm.stop()
         else:
             prnt(binary_dependency_table(rows))
 
-    if not any_rows:
+    if not rows:
         prnt("", "[grey53]No required binaries declared for discovered plugins.[/grey53]")
     elif not any_available:
         prnt("", "[grey53]No binaries detected. Run [green]archivebox install[/green] to detect dependencies.[/grey53]")
@@ -482,7 +402,7 @@ def version(
         prnt()
         prnt("[deep_sky_blue3][i] Code locations:[/deep_sky_blue3]")
         try:
-            for name, path in get_code_locations().items():
+            for name, path in get_code_locations(config=config).items():
                 if isinstance(name, str) and isinstance(path, dict):
                     prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
         except Exception as e:
@@ -492,7 +412,7 @@ def version(
         if os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) or os.access(CONSTANTS.CONFIG_FILE, os.R_OK):
             prnt("[bright_yellow][i] Data locations:[/bright_yellow]")
             try:
-                for name, path in get_data_locations().items():
+                for name, path in get_data_locations(config=config).items():
                     if isinstance(name, str) and isinstance(path, dict):
                         prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
             except Exception as e:

archivebox/cli/cli_util.py

@@ -42,3 +42,139 @@ def apply_filters(queryset, filter_kwargs: dict, limit: int | None = None):
         queryset = queryset[:limit]
 
     return queryset
+
+
+def delete_records(
+    model,
+    *,
+    label: str,
+    plural: str,
+    preview,
+    yes: bool,
+    dry_run: bool,
+    preview_limit: int | None = None,
+    by_name: bool = False,
+) -> int:
+    """Delete a JSONL selection with one confirmation and dry-run contract.
+
+    Keep queryset deletion so Django's cascades and model cleanup signals run.
+    Persona deletion has additional filesystem policy and stays with that command.
+    """
+    import sys
+
+    from django.db.models import Q
+    from rich import print as rprint
+
+    from archivebox.misc.jsonl import read_stdin
+
+    records = list(read_stdin())
+    if not records:
+        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
+        return 1
+    ids = [record["id"] for record in records if record.get("id")]
+    names = [record["name"] for record in records if by_name and not record.get("id") and record.get("name")]
+    if not ids and not names:
+        rprint(f"[yellow]No valid {label} IDs{' or names' if by_name else ''} in input[/yellow]", file=sys.stderr)
+        return 1
+    query = Q(id__in=ids)
+    if names:
+        query |= Q(name__in=names)
+    queryset = model.objects.filter(query)
+    count = queryset.count()
+    if not count:
+        rprint(f"[yellow]No matching {plural} found[/yellow]", file=sys.stderr)
+        return 0
+    if dry_run:
+        rprint(f"[yellow]Would delete {count} {plural} (dry run)[/yellow]", file=sys.stderr)
+        for obj in queryset[:preview_limit] if preview_limit is not None else queryset:
+            rprint(f"  {preview(obj)}", file=sys.stderr)
+        if preview_limit is not None and count > preview_limit:
+            rprint(f"  ... and {count - preview_limit} more", file=sys.stderr)
+        return 0
+    if not yes:
+        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
+        return 1
+    deleted_count, _ = queryset.delete()
+    rprint(f"[green]Deleted {deleted_count} {plural}[/green]", file=sys.stderr)
+    return 0
+
+
+def update_record_status(record, status: str) -> None:
+    """Route CLI status changes through the model's lifecycle operations."""
+    from django.utils import timezone
+
+    if status not in record.StatusChoices.values:
+        raise ValueError(f"Invalid {record._meta.model_name} status: {status}")
+    transitions = {"sealed": record.cancel, "paused": record.pause}
+    if status == "queued" and record.is_paused:
+        record.resume()
+    elif transition := transitions.get(status):
+        transition()
+    else:
+        record.update_and_requeue(status=status, retry_at=timezone.now())
+
+
+def list_records(queryset, *, plural: str, render) -> int:
+    """Stream JSONL when piped; render rows and a count in a terminal."""
+    import sys
+    from rich import print as rprint
+    from archivebox.misc.jsonl import write_record
+
+    is_tty = sys.stdout.isatty()
+    count = 0
+    for record in queryset:
+        if is_tty:
+            rprint(render(record))
+        else:
+            write_record(record.to_json())
+        count += 1
+    rprint(f"[dim]Listed {count} {plural}[/dim]", file=sys.stderr)
+    return 0
+
+
+def format_status(status: str, width: int) -> str:
+    colors = {
+        "queued": "yellow",
+        "started": "blue",
+        "sealed": "green",
+        "succeeded": "green",
+        "failed": "red",
+        "backoff": "magenta",
+    }
+    color = colors.get(status, "dim")
+    return f"[{color}]{status:{width}}[/{color}]"
+
+
+def update_records(model, update, *, plural: str, by_name: bool = False, refresh: bool = False) -> int:
+    """Apply a command's PATCH operation to each JSONL-selected model.
+
+    Returning False from the operation skips that row. Model operations stay in
+    their commands so lifecycle changes and ordinary field saves remain explicit.
+    """
+    import sys
+    from rich import print as rprint
+    from archivebox.misc.jsonl import read_stdin, write_record
+
+    records = list(read_stdin())
+    if not records:
+        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
+        return 1
+    is_tty = sys.stdout.isatty()
+    count = 0
+    for record in records:
+        key = "id" if record.get("id") else "name" if by_name and record.get("name") else None
+        if key is None:
+            continue
+        try:
+            obj = model.objects.get(**{key: record[key]})
+            if update(obj) is False:
+                continue
+            count += 1
+            if not is_tty:
+                if refresh:
+                    obj.refresh_from_db()
+                write_record(obj.to_json())
+        except model.DoesNotExist:
+            rprint(f"[yellow]{model.__name__} not found: {record[key]}[/yellow]", file=sys.stderr)
+    rprint(f"[green]Updated {count} {plural}[/green]", file=sys.stderr)
+    return 0

archivebox/config/collection.py

@@ -7,7 +7,7 @@ from collections.abc import Mapping
 from typing import Any
 
 from archivebox.config.constants import CONSTANTS
-from archivebox.config.configset import CaseConfigParser
+from archivebox.config.configset import CaseConfigParser, decode_config_inputs
 from archivebox.misc.logging import AttrDict
 
 
@@ -17,7 +17,7 @@ CONFIG_FILE_HEADER = (
     "# You can add options here manually in INI format, or automatically by running:\n"
     "#    archivebox config --set KEY=VALUE\n"
     "#\n"
-    "# This file is kept in sync 1:1 with Machine.config in the index DB —\n"
+    "# This file is kept in sync 1:1 with the current Machine.config in the DB —\n"
     "# editing either side propagates to the other. ``archivebox init`` reads\n"
     "# this file on startup; the admin Machine.config editor writes both.\n"
     "#\n"
@@ -160,7 +160,7 @@ def _coerce_from_str_dict(file_config: dict[str, str]) -> dict[str, Any]:
     so they round-trip through INI's string-only storage. When reading the
     file back into ``Machine.config`` (a JSONField that holds native types)
     those strings have to be decoded — otherwise downstream consumers like
-    ``_emit_machine_config`` → ``MachineEvent`` → abx-dl see a JSON string
+    ``MachineEvent`` → abx-dl see a JSON string
     where they expect a dict and raise ``TypeError``.
     Declared fields go through pydantic-settings' own ``field_is_complex`` /
     ``prepare_field_value`` so they're decoded per annotation. Undeclared
@@ -169,31 +169,8 @@ def _coerce_from_str_dict(file_config: dict[str, str]) -> dict[str, Any]:
     shape ``_coerce_to_str_dict`` writes them as.
     """
     from archivebox.config.common import ArchiveBoxConfig
-    from archivebox.config.configset import IniConfigSettingsSource
 
-    decoder = IniConfigSettingsSource(ArchiveBoxConfig)
-    decoded: dict[str, Any] = dict(file_config)
-    declared_fields = set(ArchiveBoxConfig.model_fields)
-    for field_name, field in ArchiveBoxConfig.model_fields.items():
-        if field_name not in decoded:
-            continue
-        raw = decoded[field_name]
-        if not isinstance(raw, str) or not raw:
-            continue
-        if decoder.field_is_complex(field):
-            decoded[field_name] = decoder.prepare_field_value(field_name, field, raw, True)
-    for key, raw in list(decoded.items()):
-        if key in declared_fields:
-            continue
-        if not isinstance(raw, str) or not raw:
-            continue
-        first = raw[:1]
-        if first not in ("{", "["):
-            continue
-        try:
-            decoded[key] = json.loads(raw)
-        except (TypeError, ValueError):
-            continue
+    decoded = decode_config_inputs(ArchiveBoxConfig, file_config, decode_unknown_json=True)
     if not str(decoded.get("BASE_URL") or "").strip():
         from archivebox.config.common import base_url_from_legacy_server_config
 
@@ -326,7 +303,7 @@ def write_config_file(config: dict[str, str]) -> AttrDict:
         # validate the updated_config by attempting to re-parse it
         from archivebox.config.common import get_config
 
-        updated_config = get_config().as_dict()
+        updated_config = get_config(include_machine=False).as_dict()
     except BaseException:  # lgtm [py/catch-base-exception]
         # something went horribly wrong, revert to the previous version
         with open(f"{config_path}.bak", encoding="utf-8") as old:

archivebox/config/common.py

@@ -16,13 +16,13 @@ from pathlib import Path
 from typing import Any, ClassVar, cast
 from urllib.parse import quote, urlparse
 
-from abx_plugins.plugins.base.utils import BASE_CONFIG_PATH, build_config_model, resolve_plugin_configs
+from abx_plugins.plugins.base.utils import build_config_model
 from django.db import DatabaseError
-from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
-from pydantic_settings import SettingsConfigDict
+from pydantic import BaseModel, Field, PrivateAttr, TypeAdapter, create_model, field_validator, model_validator
+from pydantic_settings import BaseSettings, SettingsConfigDict
 from rich.console import Console
 
-from archivebox.config.configset import COMPUTED_CONFIG_KEYS, BaseConfigSet, IniConfigSettingsSource
+from archivebox.config.configset import COMPUTED_CONFIG_KEYS, BaseConfigSet, decode_config_inputs
 
 from .constants import CONSTANTS
 from .ldap import LDAPConfig
@@ -406,7 +406,11 @@ class DatabaseConfig(BaseConfigSet):
     DATABASE_USER: str = Field(default="archivebox", alias="ARCHIVEBOX_DATABASE_USER")
     DATABASE_PASSWORD: str = Field(default="", alias="ARCHIVEBOX_DATABASE_PASSWORD")
     SQLITE_JOURNAL_MODE: str = Field(
-        default="WAL",
+        # Docker collections commonly live on a host bind mount. WAL's -shm
+        # locking is only safe when every SQLite process is on the same host;
+        # Docker Desktop/OrbStack place the container and host in different
+        # locking domains and a host-side reader can corrupt the live DB.
+        default="DELETE" if IN_DOCKER else "WAL",
         alias="ARCHIVEBOX_SQLITE_JOURNAL_MODE",
         pattern=r"(?i)^(DELETE|TRUNCATE|PERSIST|MEMORY|WAL|OFF)$",
     )
@@ -419,6 +423,15 @@ class DatabaseConfig(BaseConfigSet):
     SQLITE_LOCK_RETRY_TIMEOUT: float = Field(default=60.0, alias="ARCHIVEBOX_SQLITE_LOCK_RETRY_TIMEOUT", ge=0)
     SQLITE_LOCK_RETRY_INTERVAL: float = Field(default=5.0, alias="ARCHIVEBOX_SQLITE_LOCK_RETRY_INTERVAL", gt=0)
 
+    @model_validator(mode="after")
+    def reject_docker_sqlite_wal(self):
+        if IN_DOCKER and self.DATABASE_ENGINE.lower() == "sqlite" and self.SQLITE_JOURNAL_MODE.upper() == "WAL":
+            raise ValueError(
+                "SQLITE_JOURNAL_MODE=WAL is unsafe for Docker collections because host bind mounts cross SQLite "
+                "locking domains; use DELETE (the Docker default) or PostgreSQL",
+            )
+        return self
+
 
 class ArchivingConfig(BaseConfigSet):
     toml_section_header: str = "ARCHIVING_CONFIG"
@@ -428,6 +441,7 @@ class ArchivingConfig(BaseConfigSet):
         default="",
         description="Comma-separated plugin selection for this run. Empty means use enabled plugin defaults.",
     )
+    PARSER: str = Field(default="auto", description="Parser used to read the submitted Crawl input document.")
 
     ONLY_NEW: bool = Field(default=True)
     INDEX_ONLY: bool = Field(default=False)
@@ -447,7 +461,8 @@ class ArchivingConfig(BaseConfigSet):
     USER_AGENT: str = Field(
         default=f"Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)",
     )
-    COOKIES_FILE: Path | None = Field(default=None)
+    COOKIES_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    AUTH_STORAGE_FILE: Path | None = Field(default=None, json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
     URL_DENYLIST: str = Field(default=r"\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$", alias="URL_BLACKLIST")
     URL_ALLOWLIST: str | None = Field(default=None, alias="URL_WHITELIST")
@@ -546,7 +561,7 @@ class SearchBackendConfig(BaseConfigSet):
     toml_section_header: str = "SEARCH_BACKEND_CONFIG"
     _scope: str = PrivateAttr(default=_SCOPE_SERVER)
 
-    SEARCH_BACKEND_ENGINE: str = Field(default="ripgrep", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    SEARCH_BACKEND_ENGINE: str = Field(default="sonic", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
 
 def _plugin_user_config_value(value: Any) -> str:
@@ -588,13 +603,9 @@ def _explicit_plugin_enabled_keys(config: Mapping[str, object]) -> set[str]:
 
 
 def _discover_plugin_config_schemas() -> PluginSchemaDocuments:
-    from archivebox.plugins.discovery import discover_plugin_configs
+    from archivebox.plugins.discovery import get_plugin_config_resolver
 
-    schemas: PluginSchemaDocuments = {}
-    if BASE_CONFIG_PATH.exists():
-        schemas["base"] = json.loads(BASE_CONFIG_PATH.read_text())
-    schemas.update(discover_plugin_configs())
-    return schemas
+    return get_plugin_config_resolver().schemas
 
 
 def _plugin_config_properties(plugin_schemas: PluginSchemaDocuments) -> dict[str, dict[str, Any]]:
@@ -843,13 +854,11 @@ class ArchiveBoxBaseConfig(
             disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
         selected_plugin_roots = plugin_names
         if selected_plugin_roots:
-            from abx_dl.models import discover_plugins, filter_plugins
+            from archivebox.plugins.discovery import get_plugin_catalog
 
             selected_plugins = set(
-                filter_plugins(
-                    discover_plugins(runtime="archivebox"),
+                get_plugin_catalog().select(
                     sorted(selected_plugin_roots),
-                    include_providers=True,
                     disabled_names=disabled_plugins,
                 ),
             )
@@ -858,6 +867,8 @@ class ArchiveBoxBaseConfig(
         search_backend = self.SEARCH_BACKEND_ENGINE.strip().lower()
         if search_backend:
             selected_plugins.add(f"search_backend_{search_backend}")
+        if respect_current_enabled:
+            selected_plugins.difference_update(disabled_plugins)
         for plugin_name, enabled_key in enabled_config_keys.items():
             if plugin_names or plugin_name in selected_plugins:
                 setattr(self, enabled_key, plugin_name in selected_plugins)
@@ -883,6 +894,12 @@ PLUGIN_CONFIG_SCHEMAS = _discover_plugin_config_schemas()
 ArchiveBoxConfig = _build_archivebox_config_model(PLUGIN_CONFIG_SCHEMAS)
 
 
+class ArchiveBoxSourceSettings(ArchiveBoxConfig, BaseSettings):
+    """Environment and INI source loader for the pure runtime config model."""
+
+    model_config = SettingsConfigDict(**{**ArchiveBoxConfig.model_config, "extra": "ignore"})
+
+
 def _normalize_plugins_config_value(value: Any) -> set[str]:
     if value is None:
         return set()
@@ -1039,7 +1056,9 @@ def get_request_config(request: Any, *, resolve_plugins: bool = False) -> Archiv
     request_config = request_state.get("archivebox_config")
     request_config_resolves_plugins = bool(request_state.get("_archivebox_config_resolves_plugins", False))
     if request_config is None or (resolve_plugins and not request_config_resolves_plugins):
-        request_config = get_config(resolve_plugins=resolve_plugins)
+        from django.conf import settings
+
+        request_config = get_config(base_config=settings.CONFIG, resolve_plugins=resolve_plugins)
         request._archivebox_config_resolves_plugins = resolve_plugins
     if request_config.SERVER_SECURITY_MODE == "auto":
         request_host = (urlparse(f"//{request.get_host()}").hostname or "").lower().rstrip(".")
@@ -1094,9 +1113,11 @@ def get_config(
     config_data: ConfigPayload = dict(defaults or {})
     config_data["PERSONAS_DIR"] = str(CONSTANTS.PERSONAS_DIR)
     base_config_payload: ConfigPayload = {}
-    base_config_model = ArchiveBoxConfig()
+    file_config: dict[str, Any] | None = None
+    base_config_model = ArchiveBoxSourceSettings() if base_config is None else None
 
     if crawl_config_base:
+        assert base_config_model is not None
         config_data.update(
             normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
         )
@@ -1108,10 +1129,12 @@ def get_config(
             base_config_payload.update(dict(base_config))
         config_data.update(normalize_runtime_config(base_config_payload, exclude_runtime_derived=True, json_safe=False))
     else:
+        assert base_config_model is not None
         config_data.update(
             normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
         )
-        legacy_config = {**BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE), **os.environ}
+        file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
+        legacy_config = {**file_config, **os.environ}
         legacy_permissions = permissions_from_legacy_public_flags(legacy_config)
         if legacy_permissions:
             config_data["PERMISSIONS"] = legacy_permissions
@@ -1168,7 +1191,6 @@ def get_config(
 
     explicit_plugin_enabled_keys: set[str] = set()
     if resolve_plugins:
-        plugin_schemas = {plugin_name: schema for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items() if isinstance(schema, dict)}
         plugin_global_config = {key: str(value) if isinstance(value, Path) else value for key, value in config_data.items()}
         crawl_selected_plugins = crawl_config_base and bool(_normalize_plugins_config_value(dict(crawl.config or {}).get("PLUGINS")))
         # A frozen crawl-level PLUGINS selector is the exact extractor set for
@@ -1185,14 +1207,16 @@ def get_config(
             },
         )
         if not crawl_config_base:
-            file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
+            if file_config is None:
+                file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
             explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(file_config))
             plugin_user_config = {
                 **_plugin_user_config(_plugin_input_config(file_config)),
                 **plugin_user_config,
             }
-        plugin_sections = resolve_plugin_configs(
-            plugin_schemas,
+        from archivebox.plugins.discovery import get_plugin_config_resolver
+
+        plugin_sections = get_plugin_config_resolver().resolve(
             global_config=plugin_global_config,
             user_config=plugin_user_config,
             environ={},
@@ -1221,28 +1245,7 @@ def get_config(
             for enabled_key in _plugin_enabled_config_keys().values():
                 config_data.pop(enabled_key, None)
 
-    # Decode JSON-encoded complex values (dict/list fields) that came from
-    # string-only sources before validation. ``IniConfigSettingsSource`` does
-    # this for the ArchiveBox.conf path, but Machine.config (mirrored from the
-    # INI via ``_coerce_to_str_dict``) and plugin/env scope overrides bypass
-    # pydantic-settings sources entirely — they feed JSON strings directly
-    # into ``model_validate``, which rejects ``"{...}"`` for a ``dict[str, str]``
-    # field. Run pydantic-settings' own complex-value decoder here so every
-    # source converges on the same shape before validation.
-    _complex_decoder = IniConfigSettingsSource(ArchiveBoxConfig)
-    for _field_name, _field in ArchiveBoxConfig.model_fields.items():
-        if _field_name not in config_data:
-            continue
-        _raw = config_data[_field_name]
-        if not isinstance(_raw, str) or not _raw:
-            continue
-        if _complex_decoder.field_is_complex(_field):
-            config_data[_field_name] = _complex_decoder.prepare_field_value(
-                _field_name,
-                _field,
-                _raw,
-                True,
-            )
+    config_data = decode_config_inputs(ArchiveBoxConfig, config_data)
 
     config = ArchiveBoxConfig.model_validate(config_data)
     if config.SERVER_SECURITY_MODE == "auto":
@@ -1251,7 +1254,7 @@ def get_config(
             config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
     for key in explicit_plugin_enabled_keys:
         if key in config_data:
-            setattr(config, key, config_data[key])
+            setattr(config, key, TypeAdapter(bool).validate_python(config_data[key]))
     if config.PLUGINS:
         config._derive_plugin_enabled_config(respect_current_enabled=True)
     if redact_sensitive:
@@ -1259,7 +1262,6 @@ def get_config(
             value = config[key]
             if is_sensitive_config_key(key) and value not in (None, ""):
                 setattr(config, key, SENSITIVE_CONFIG_VALUE_REDACTED)
-    os.environ["ABXPKG_LIB_DIR"] = str(config.ABXPKG_LIB_DIR)
     archiving_warning_key = (config.TIMEOUT, config.USE_COLOR)
     if archiving_warning_key not in _WARNED_ARCHIVING_CONFIGS:
         config.warn_if_invalid()
@@ -1269,13 +1271,14 @@ def get_config(
 
 def get_all_configs() -> dict[str, BaseConfigSet]:
     """Get all config section objects as a dictionary."""
+    resolved = ArchiveBoxSourceSettings().model_dump(mode="json")
     return {
-        "SHELL_CONFIG": ShellConfig(),
-        "STORAGE_CONFIG": StorageConfig(),
-        "GENERAL_CONFIG": GeneralConfig(),
-        "SERVER_CONFIG": ServerConfig(),
-        "DATABASE_CONFIG": DatabaseConfig(),
-        "ARCHIVING_CONFIG": ArchivingConfig(),
-        "SEARCH_BACKEND_CONFIG": SearchBackendConfig(),
-        "LDAP_CONFIG": LDAPConfig(),
+        "SHELL_CONFIG": ShellConfig.model_validate(resolved),
+        "STORAGE_CONFIG": StorageConfig.model_validate(resolved),
+        "GENERAL_CONFIG": GeneralConfig.model_validate(resolved),
+        "SERVER_CONFIG": ServerConfig.model_validate(resolved),
+        "DATABASE_CONFIG": DatabaseConfig.model_validate(resolved),
+        "ARCHIVING_CONFIG": ArchivingConfig.model_validate(resolved),
+        "SEARCH_BACKEND_CONFIG": SearchBackendConfig.model_validate(resolved),
+        "LDAP_CONFIG": LDAPConfig.model_validate(resolved),
     }

archivebox/config/configset.py

@@ -2,10 +2,13 @@
 
 __package__ = "archivebox.config"
 
+import json
+from collections.abc import Mapping
+from configparser import ConfigParser
 from pathlib import Path
 from typing import Any, ClassVar
-from configparser import ConfigParser
 
+from pydantic import BaseModel
 from pydantic_settings import BaseSettings, PydanticBaseSettingsSource, SettingsConfigDict
 
 COMPUTED_CONFIG_KEYS = (
@@ -30,31 +33,17 @@ class CaseConfigParser(ConfigParser):
         return optionstr
 
 
-# ``IniConfigSettingsSource.get_field_value`` is called once per pydantic field
-# (hundreds of fields across the GlobalConfig + plugin configs). Without
-# caching, every field re-opens and re-parses ``ArchiveBox.conf`` from disk —
-# that's the dominant cost of ``get_config()`` (each call was ~130ms; profile
-# showed 632 ``parser.read(config_path)`` invocations per call). The cache is
-# keyed on (path, mtime) so external edits to the file still get picked up.
-_INI_CACHE: dict[tuple[str, float], dict[str, Any]] = {}
-
-
-def _read_ini_config_cached(config_path_str: str) -> dict[str, Any]:
-    config_path = Path(config_path_str)
+def read_ini_config(config_path: str | Path) -> dict[str, Any]:
+    """Read and flatten an ArchiveBox INI config file."""
+    config_path = Path(config_path)
     try:
-        mtime = config_path.stat().st_mtime
-    except FileNotFoundError:
+        if not config_path.exists():
+            return {}
+        parser = CaseConfigParser(interpolation=None)
+        parser.read(config_path)
+    except OSError:
         return {}
-    cache_key = (str(config_path), mtime)
-    cached = _INI_CACHE.get(cache_key)
-    if cached is not None:
-        return cached
-    parser = CaseConfigParser()
-    parser.read(config_path)
-    flat = {key.upper(): value for section in parser.sections() for key, value in parser.items(section)}
-    _INI_CACHE.clear()
-    _INI_CACHE[cache_key] = flat
-    return flat
+    return {key.upper(): value for section in parser.sections() for key, value in parser.items(section)}
 
 
 class IniConfigSettingsSource(PydanticBaseSettingsSource):
@@ -75,18 +64,7 @@ class IniConfigSettingsSource(PydanticBaseSettingsSource):
         return field_value, field_name, value_is_complex
 
     def __call__(self) -> dict[str, Any]:
-        # Use the per-field path (``get_field_value`` + ``prepare_field_value``)
-        # so complex types get JSON-decoded. The previous flat-dict return
-        # skipped pydantic-settings' complex-value handling entirely.
-        result: dict[str, Any] = {}
-        for field_name, field in self.settings_cls.model_fields.items():
-            value, key, value_is_complex = self.get_field_value(field, field_name)
-            if value is None:
-                continue
-            prepared = self.prepare_field_value(field_name, field, value, value_is_complex)
-            if prepared is not None:
-                result[key] = prepared
-        return result
+        return decode_config_inputs(self.settings_cls, self._load_config_file())
 
     def _load_config_file(self) -> dict[str, Any]:
         try:
@@ -96,14 +74,42 @@ class IniConfigSettingsSource(PydanticBaseSettingsSource):
         except ImportError:
             return {}
 
-        return _read_ini_config_cached(str(config_path))
+        return read_ini_config(config_path)
 
 
-class BaseConfigSet(BaseSettings):
+def decode_config_inputs(
+    settings_cls: type[BaseSettings],
+    config: Mapping[str, Any],
+    *,
+    decode_unknown_json: bool = False,
+) -> dict[str, Any]:
+    """Decode string-only config values once at the boundary to typed config."""
+    decoder = IniConfigSettingsSource(settings_cls)
+    decoded: dict[str, Any] = dict(config)
+    for source_key, raw in list(decoded.items()):
+        if not isinstance(raw, str) or not raw:
+            continue
+        field_name = source_key if source_key in settings_cls.model_fields else source_key.upper()
+        field = settings_cls.model_fields.get(field_name)
+        if field is None:
+            if decode_unknown_json and raw[:1] in ("{", "["):
+                try:
+                    decoded[source_key] = json.loads(raw)
+                except (TypeError, ValueError):
+                    pass
+            continue
+        if decoder.field_is_complex(field):
+            decoded[field_name] = decoder.prepare_field_value(field_name, field, raw, True)
+            if source_key != field_name:
+                decoded.pop(source_key, None)
+    return decoded
+
+
+class BaseConfigSet(BaseModel):
     """
-    Base class for config sections.
+    Pure typed runtime model for config sections.
 
-    Automatically loads values from (highest to lowest priority):
+    Source-loading subclasses combine this model with ``BaseSettings`` and load:
     1. Environment variables
     2. ArchiveBox.conf file (INI format, flattened)
     3. Default values
@@ -146,14 +152,7 @@ class BaseConfigSet(BaseSettings):
     @classmethod
     def load_from_file(cls, config_path: Path) -> dict[str, str]:
         """Load config values from INI file."""
-        if not config_path.exists():
-            return {}
-
-        parser = CaseConfigParser()
-        parser.read(config_path)
-
-        # Flatten all sections into single namespace
-        return {key.upper(): value for section in parser.sections() for key, value in parser.items(section)}
+        return read_ini_config(config_path)
 
     def __getitem__(self, key: str) -> Any:
         if key in type(self).model_fields:

archivebox/config/django.py

@@ -2,20 +2,20 @@ __package__ = "archivebox.config"
 
 import os
 import sys
-
-from datetime import datetime, timezone
-
-from rich.console import Console
+from datetime import UTC, datetime
 
 import django
 import django.db
+from django.core.exceptions import ImproperlyConfigured
+from rich.console import Console
 
 from archivebox.misc import logging
 
-from .constants import CONSTANTS
 from .common import get_config
+from .constants import CONSTANTS
 
 CONFIG = get_config()
+os.environ.setdefault("ABXPKG_LIB_DIR", str(CONFIG.ABXPKG_LIB_DIR))
 
 if not CONFIG.USE_COLOR:
     os.environ["NO_COLOR"] = "1"
@@ -30,8 +30,9 @@ logging.CONSOLE = CONSOLE
 DJANGO_SET_UP = False
 
 
-def setup_django(check_db=False, in_memory_db=False) -> None:
+def setup_django(check_db=False) -> None:
     from rich.panel import Panel
+
     from archivebox.misc.checks import check_not_inside_source_dir
 
     global DJANGO_SET_UP
@@ -52,18 +53,17 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
     # Keeping them out of archivebox.__init__ avoids paying Django/Daphne setup
     # cost for cheap CLI startup paths like `archivebox <cmd> --help`.
     import archivebox.misc.monkey_patches  # noqa: F401
-
-    from archivebox.config.permissions import IS_ROOT, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, SudoPermission
+    from archivebox.config.permissions import ARCHIVEBOX_GROUP, ARCHIVEBOX_USER, IS_ROOT, SudoPermission
 
     # if running as root, chown the data dir to the archivebox user to make sure it's accessible to the archivebox user
     if IS_ROOT and ARCHIVEBOX_USER != 0:
         with SudoPermission(uid=0):
             # running as root is a special case where it's ok to be a bit slower
             # make sure data dir is always owned by the correct user
-            os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP)
+            os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
             if CONSTANTS.DATA_DIR.exists():
                 for child in CONSTANTS.DATA_DIR.iterdir():
-                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP)
+                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, follow_symlinks=False)
 
     # Suppress the "database access during app initialization" warning
     # This warning can be triggered during django.setup() but is safe to ignore
@@ -79,40 +79,30 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
     try:
         from django.core.management import call_command
 
-        if in_memory_db:
-            raise Exception("dont use this anymore")
-
-            # some commands dont store a long-lived sqlite3 db file on disk.
-            # in those cases we create a temporary in-memory db and run the migrations
-            # immediately to get a usable in-memory-database at startup
-            os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:")
+        # Initialize the configured file-based database without running
+        # migrations automatically; `archivebox init` owns migrations.
+        try:
             django.setup()
+        # User config and import errors are reported through one CLI message.
+        except (ImproperlyConfigured, django.db.Error, ImportError, OSError, RuntimeError, ValueError) as e:
+            is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ("help", "version", "--help", "--version"))
+            if not is_using_meta_cmd:
+                # show error message to user only if they're not running a meta command / just trying to get help
+                STDERR.print()
+                STDERR.print(
+                    Panel(
+                        f"\n[red]{e.__class__.__name__}[/red]: [yellow]{e}[/yellow]\nPlease check your config and [blue]DATA_DIR[/blue] permissions.\n",
+                        title="\n\n[red][X] Error while trying to load database![/red]",
+                        subtitle="[grey53]NO WRITES CAN BE PERFORMED[/grey53]",
+                        expand=False,
+                        style="bold red",
+                    ),
+                )
+                STDERR.print()
+                import traceback
 
-            call_command("migrate", interactive=False, verbosity=0)
-        else:
-            # Otherwise use default sqlite3 file-based database and initialize django
-            # without running migrations automatically (user runs them manually by calling init)
-            try:
-                django.setup()
-            except Exception as e:
-                is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ("help", "version", "--help", "--version"))
-                if not is_using_meta_cmd:
-                    # show error message to user only if they're not running a meta command / just trying to get help
-                    STDERR.print()
-                    STDERR.print(
-                        Panel(
-                            f"\n[red]{e.__class__.__name__}[/red]: [yellow]{e}[/yellow]\nPlease check your config and [blue]DATA_DIR[/blue] permissions.\n",
-                            title="\n\n[red][X] Error while trying to load database![/red]",
-                            subtitle="[grey53]NO WRITES CAN BE PERFORMED[/grey53]",
-                            expand=False,
-                            style="bold red",
-                        ),
-                    )
-                    STDERR.print()
-                    import traceback
-
-                    traceback.print_exc()
-                return
+                traceback.print_exc()
+            return
 
         from archivebox.core.settings_logging import ERROR_LOG as DEFAULT_ERROR_LOG
 
@@ -120,13 +110,13 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
         error_log = DEFAULT_ERROR_LOG
         with open(error_log, "a", encoding="utf-8") as f:
             command = " ".join(sys.argv)
-            ts = datetime.now(timezone.utc).strftime("%Y-%m-%d__%H:%M:%S")
+            ts = datetime.now(UTC).strftime("%Y-%m-%d__%H:%M:%S")
             config = get_config()
             f.write(f"\n> {command}; TS={ts} VERSION={CONSTANTS.VERSION} IN_DOCKER={config.IN_DOCKER} IS_TTY={config.IS_TTY}\n")
 
         if check_db:
             # make sure the data dir is owned by a non-root user
-            if CONSTANTS.DATA_DIR.stat().st_uid == 0:
+            if CONSTANTS.DATA_DIR.stat().st_uid == 0 and not (IS_ROOT and ARCHIVEBOX_USER == 0):
                 STDERR.print("[red][X] Error: ArchiveBox DATA_DIR cannot be owned by root![/red]")
                 STDERR.print(f"    {CONSTANTS.DATA_DIR}")
                 STDERR.print()
@@ -159,6 +149,7 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
             )
 
     except KeyboardInterrupt:
+        DJANGO_SET_UP = False
         raise
 
     DJANGO_SET_UP = True

archivebox/config/paths.py

@@ -222,7 +222,7 @@ def get_or_create_working_tmp_dir(autofix=True, quiet=True, config: "ArchiveBoxC
             create_and_chown_dir(candidate)
         except Exception:
             pass
-        if check_tmp_dir(candidate, throw=False, quiet=True, must_exist=True):
+        if check_tmp_dir(candidate, throw=False, quiet=True, must_exist=True, config=config):
             if autofix and config.TMP_DIR != candidate:
                 os.environ["TMP_DIR"] = str(candidate)
             return candidate
@@ -263,7 +263,7 @@ def get_or_create_working_lib_dir(autofix=True, quiet=False, config: "ArchiveBox
             create_and_chown_dir(candidate)
         except Exception:
             pass
-        if check_lib_dir(candidate, throw=False, quiet=True, must_exist=True):
+        if check_lib_dir(candidate, throw=False, quiet=True, must_exist=True, config=config):
             if autofix and config.ABXPKG_LIB_DIR != candidate:
                 os.environ["ABXPKG_LIB_DIR"] = str(candidate)
             return candidate
@@ -272,10 +272,10 @@ def get_or_create_working_lib_dir(autofix=True, quiet=False, config: "ArchiveBox
         raise OSError(f"ArchiveBox is unable to find a writable ABXPKG_LIB_DIR, tried {CANDIDATES}!")
 
 
-def _sql_index_location() -> dict:
+def _sql_index_location(config: "ArchiveBoxConfig") -> dict:
     from archivebox.misc.db import database_display_location, database_exists, is_postgres
 
-    if is_postgres():
+    if is_postgres(config):
         return {
             "path": database_display_location(),
             "enabled": True,
@@ -316,7 +316,7 @@ def get_data_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs
                 and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
                 and os.access(CONSTANTS.CONFIG_FILE, os.W_OK),
             },
-            "SQL_INDEX": _sql_index_location(),
+            "SQL_INDEX": _sql_index_location(config),
             "ARCHIVE_DIR": {
                 "path": CONSTANTS.ARCHIVE_DIR.resolve(),
                 "enabled": True,

archivebox/config/permissions.py

@@ -1,19 +1,53 @@
 __package__ = "archivebox.config"
 
 import os
-import pwd
-import sys
-import socket
 import platform
+import pwd
+import socket
+import stat
+import subprocess
+import sys
+from contextlib import contextmanager, suppress
+from pathlib import Path
 from typing import cast
 
 from rich import print
 
-from pathlib import Path
-from contextlib import contextmanager
-
 #############################################################################################
 
+
+def is_root_identity(running_uid: int, effective_uid: int) -> bool:
+    return running_uid == 0 or effective_uid == 0
+
+
+def select_archivebox_user(
+    *,
+    running_uid: int,
+    running_gid: int,
+    effective_uid: int,
+    effective_gid: int,
+    sudo_uid: int,
+    sudo_gid: int,
+    data_dir_uid: int,
+    data_dir_gid: int,
+    account_uid: int | None,
+    account_gid: int | None,
+    data_dir_owner_gid: int | None = None,
+    data_dir_owner_exists: bool = True,
+) -> tuple[int, int]:
+    if running_uid == 0:
+        if data_dir_uid != 0 and data_dir_owner_exists:
+            owner_gid = data_dir_owner_gid if data_dir_gid == 0 and data_dir_owner_gid is not None else data_dir_gid
+            return data_dir_uid, owner_gid
+        if account_uid is not None and account_gid is not None:
+            return account_uid, account_gid
+
+    return (
+        effective_uid or running_uid or sudo_uid,
+        effective_gid or running_gid or sudo_gid,
+    )
+
+
 DATA_DIR = Path(os.getcwd())
 
 try:
@@ -24,42 +58,100 @@ except PermissionError:
     DATA_DIR_UID = 0
     DATA_DIR_GID = 0
 
+try:
+    DATA_DIR_OWNER_GID = pwd.getpwuid(DATA_DIR_UID).pw_gid
+    DATA_DIR_OWNER_EXISTS = True
+except KeyError:
+    DATA_DIR_OWNER_GID = None
+    DATA_DIR_OWNER_EXISTS = False
+
 DEFAULT_UID = 911
 DEFAULT_GID = 911
 RUNNING_AS_UID = os.getuid()
 RUNNING_AS_GID = os.getgid()
 EUID = os.geteuid()
 EGID = os.getegid()
-SUDO_UID = int(os.environ.get("SUDO_UID", 0))
-SUDO_GID = int(os.environ.get("SUDO_GID", 0))
+SUDO_UID = int(os.environ.get("SUDO_UID", "0"))
+SUDO_GID = int(os.environ.get("SUDO_GID", "0"))
 USER: str = Path("~").expanduser().resolve().name
 HOSTNAME: str = cast(str, max([socket.gethostname(), platform.node()], key=len))
 
-IS_ROOT = RUNNING_AS_UID == 0
-IN_DOCKER = os.environ.get("IN_DOCKER", False) in ("1", "true", "True", "TRUE", "yes")
+IS_ROOT = is_root_identity(RUNNING_AS_UID, EUID)
+IN_DOCKER = os.environ.get("IN_DOCKER", "") in ("1", "true", "True", "TRUE", "yes")
 
 FALLBACK_UID = RUNNING_AS_UID or SUDO_UID
 FALLBACK_GID = RUNNING_AS_GID or SUDO_GID
-try:
-    ARCHIVEBOX_ACCOUNT = pwd.getpwnam("archivebox")
-except KeyError:
-    ARCHIVEBOX_ACCOUNT = None
 
-if DATA_DIR_UID != 0:
-    ARCHIVEBOX_USER = DATA_DIR_UID
-    ARCHIVEBOX_GROUP = DATA_DIR_GID
-elif RUNNING_AS_UID == 0 and ARCHIVEBOX_ACCOUNT is not None:
-    ARCHIVEBOX_USER = ARCHIVEBOX_ACCOUNT.pw_uid
-    ARCHIVEBOX_GROUP = ARCHIVEBOX_ACCOUNT.pw_gid
-else:
-    ARCHIVEBOX_USER = EUID or RUNNING_AS_UID or FALLBACK_UID
-    ARCHIVEBOX_GROUP = EGID or RUNNING_AS_GID or FALLBACK_GID
+
+def get_or_create_archivebox_account():
+    try:
+        account = pwd.getpwnam("archivebox")
+    except KeyError:
+        account = None
+
+    if account is None and (RUNNING_AS_UID != 0 or platform.system() != "Linux"):
+        return None
+
+    if account is None:
+        useradd = "/usr/sbin/useradd" if Path("/usr/sbin/useradd").exists() else "useradd"
+        command = [
+            useradd,
+            "--system",
+            "--create-home",
+            "--home-dir",
+            "/var/lib/archivebox",
+            "--shell",
+            "/bin/bash",
+            "archivebox",
+        ]
+        try:
+            subprocess.run(command, check=True)
+        except (OSError, subprocess.CalledProcessError) as err:
+            # Another concurrently starting process may have created it first.
+            try:
+                account = pwd.getpwnam("archivebox")
+            except KeyError:
+                raise RuntimeError(f"Failed to create the archivebox system user with: {' '.join(command)}") from err
+        else:
+            account = pwd.getpwnam("archivebox")
+
+    if RUNNING_AS_UID == 0 and platform.system() == "Linux":
+        # Package removal may preserve the account but remove its home. Repair
+        # only this directory entry; never recursively chown existing data.
+        account_home = Path(account.pw_dir)
+        try:
+            account_home.mkdir(parents=True, exist_ok=True)
+            home_stat = account_home.stat()
+            if (home_stat.st_uid, home_stat.st_gid) != (account.pw_uid, account.pw_gid):
+                os.chown(account_home, account.pw_uid, account.pw_gid)
+        except OSError as err:
+            raise RuntimeError(f"Failed to prepare archivebox account home: {account_home}") from err
+
+    return account
+
+
+ARCHIVEBOX_ACCOUNT = get_or_create_archivebox_account()
+
+ARCHIVEBOX_USER, ARCHIVEBOX_GROUP = select_archivebox_user(
+    running_uid=RUNNING_AS_UID,
+    running_gid=RUNNING_AS_GID,
+    effective_uid=EUID,
+    effective_gid=EGID,
+    sudo_uid=SUDO_UID,
+    sudo_gid=SUDO_GID,
+    data_dir_uid=DATA_DIR_UID,
+    data_dir_gid=DATA_DIR_GID,
+    data_dir_owner_gid=DATA_DIR_OWNER_GID,
+    account_uid=ARCHIVEBOX_ACCOUNT.pw_uid if ARCHIVEBOX_ACCOUNT is not None else None,
+    account_gid=ARCHIVEBOX_ACCOUNT.pw_gid if ARCHIVEBOX_ACCOUNT is not None else None,
+    data_dir_owner_exists=DATA_DIR_OWNER_EXISTS,
+)
 if not USER:
     try:
         # alternative method 1 to get username
         USER = pwd.getpwuid(ARCHIVEBOX_USER).pw_name
-    except Exception:
-        pass
+    except (KeyError, OSError):
+        USER = ""
 
 if not USER:
     try:
@@ -67,56 +159,172 @@ if not USER:
         import getpass
 
         USER = getpass.getuser()
-    except Exception:
-        pass
+    except OSError:
+        USER = ""
 
 if not USER:
     try:
         # alternative method 3 to get username
         USER = os.getlogin() or "archivebox"
-    except Exception:
+    except OSError:
         USER = "archivebox"
 
 ARCHIVEBOX_USER_EXISTS = False
 try:
     pwd.getpwuid(ARCHIVEBOX_USER)
     ARCHIVEBOX_USER_EXISTS = True
-except Exception:
+except KeyError:
     ARCHIVEBOX_USER_EXISTS = False
 
 
+ROOT_HANDOFF_NAMES = (
+    ".archivebox_id",
+    "ArchiveBox.conf",
+    "index.sqlite3",
+    "index.sqlite3-shm",
+    "index.sqlite3-wal",
+    "archive",
+    "cache",
+    "custom_plugins",
+    "custom_templates",
+    "lib",
+    "logs",
+    "personas",
+    "sonic",
+    "sources",
+    "tmp",
+    "users",
+)
+
+
+def root_data_dir_handoff_paths(data_dir: Path, argv: list[str]) -> tuple[Path, ...]:
+    """Return bounded top-level paths safe to hand from root to archivebox."""
+
+    data_dir = data_dir.resolve()
+    if data_dir == Path("/"):
+        return ()
+
+    try:
+        children = tuple(data_dir.iterdir())
+    except (FileNotFoundError, PermissionError):
+        return ()
+
+    is_init = any(arg in ("init", "install", "--init", "--quick-init") for arg in argv[1:])
+    collection_exists = any((data_dir / marker).exists() for marker in (".archivebox_id", "ArchiveBox.conf", "index.sqlite3"))
+    if not collection_exists and not (is_init and not children):
+        return ()
+
+    top_level_paths = tuple(data_dir / name for name in ROOT_HANDOFF_NAMES if (data_dir / name).exists())
+    log_files = tuple((data_dir / "logs").glob("*.log"))
+    return (data_dir, *top_level_paths, *log_files)
+
+
+def root_should_handoff_data_dir(
+    *,
+    is_root: bool,
+    data_dir_uid: int,
+    account_uid: int | None,
+    data_dir_owner_exists: bool = True,
+) -> bool:
+    """Allow bounded handoff only for root or archivebox-owned collection roots."""
+
+    return is_root and account_uid is not None and (not data_dir_owner_exists or data_dir_uid in (0, account_uid))
+
+
+def root_parent_can_grant_group_traversal(*, parent_uid: int, parent_gid: int, parent_mode: int, account_gid: int) -> bool:
+    """Return whether adding archivebox traversal preserves existing parent access."""
+
+    if parent_uid != 0 or parent_mode & stat.S_IXOTH:
+        return False
+    if parent_gid == account_gid:
+        return not bool(parent_mode & stat.S_IXGRP)
+    return not bool(parent_mode & stat.S_IRWXG)
+
+
+def grant_archivebox_group_traversal(path: Path) -> None:
+    """Let the archivebox account traverse private root-owned parents."""
+
+    if not IS_ROOT or ARCHIVEBOX_ACCOUNT is None:
+        return
+
+    for parent in path.resolve().parents:
+        if parent == Path("/"):
+            continue
+        parent_stat = parent.stat(follow_symlinks=False)
+        if root_parent_can_grant_group_traversal(
+            parent_uid=parent_stat.st_uid,
+            parent_gid=parent_stat.st_gid,
+            parent_mode=parent_stat.st_mode,
+            account_gid=ARCHIVEBOX_ACCOUNT.pw_gid,
+        ):
+            os.chown(parent, -1, ARCHIVEBOX_ACCOUNT.pw_gid, follow_symlinks=False)
+            os.chmod(parent, stat.S_IMODE(parent_stat.st_mode) | stat.S_IXGRP, follow_symlinks=False)
+
+
+def handoff_root_owned_data_dir() -> None:
+    account_uid = ARCHIVEBOX_ACCOUNT.pw_uid if ARCHIVEBOX_ACCOUNT is not None else None
+    if not root_should_handoff_data_dir(
+        is_root=IS_ROOT,
+        data_dir_uid=DATA_DIR_UID,
+        account_uid=account_uid,
+        data_dir_owner_exists=DATA_DIR_OWNER_EXISTS,
+    ):
+        return
+
+    handoff_paths = root_data_dir_handoff_paths(DATA_DIR, sys.argv)
+    if not handoff_paths:
+        return
+
+    # Never walk collection contents. This only grants execute-only traversal
+    # on private parents such as /root so the handed-off data remains reachable.
+    grant_archivebox_group_traversal(DATA_DIR)
+
+    for path in handoff_paths:
+        try:
+            os.chown(path, ARCHIVEBOX_ACCOUNT.pw_uid, ARCHIVEBOX_ACCOUNT.pw_gid, follow_symlinks=False)
+        except (FileNotFoundError, PermissionError):
+            pass
+
+
 #############################################################################################
 
 
 def drop_privileges():
     """If running as root, drop privileges to the data dir owner or archivebox user."""
 
+    # Root-owned uv tools commonly live below /root. Keep the installed package
+    # importable after dropping EUID without changing ownership of the tool env.
+    grant_archivebox_group_traversal(Path(__file__))
+    handoff_root_owned_data_dir()
+
     # Always run ArchiveBox as the user that owns the data dir, or as the
     # archivebox service account when the data dir is root-owned.
-    if os.getuid() == 0:
-        if os.geteuid() != ARCHIVEBOX_USER and ARCHIVEBOX_USER != 0 and ARCHIVEBOX_USER_EXISTS:
+    if os.geteuid() == 0 and ARCHIVEBOX_USER != 0 and ARCHIVEBOX_USER_EXISTS:
+        pw_record = pwd.getpwuid(ARCHIVEBOX_USER)
+        supplemental_gid = ARCHIVEBOX_ACCOUNT.pw_gid if ARCHIVEBOX_ACCOUNT is not None else ARCHIVEBOX_GROUP
+        os.initgroups(pw_record.pw_name, supplemental_gid)
+        if os.getegid() != ARCHIVEBOX_GROUP:
+            os.setegid(ARCHIVEBOX_GROUP)
+        if os.geteuid() != ARCHIVEBOX_USER:
             os.seteuid(ARCHIVEBOX_USER)
 
-            # update environment variables so that subprocesses dont try to write to /root
-            pw_record = pwd.getpwuid(ARCHIVEBOX_USER)
-            os.environ["HOME"] = pw_record.pw_dir
-            os.environ["LOGNAME"] = pw_record.pw_name
-            os.environ["USER"] = pw_record.pw_name
-            os.environ["XDG_CACHE_HOME"] = str(Path(pw_record.pw_dir) / ".cache")
-            os.environ["XDG_CONFIG_HOME"] = str(Path(pw_record.pw_dir) / ".config")
-            os.environ["XDG_DATA_HOME"] = str(Path(pw_record.pw_dir) / ".local" / "share")
-            os.environ.pop("XDG_RUNTIME_DIR", None)
-            os.environ.pop("ABXBUS_MULTIPROCESS_SEMAPHORE_DIR", None)
+        # update environment variables so that subprocesses dont try to write to /root
+        os.environ["HOME"] = pw_record.pw_dir
+        os.environ["LOGNAME"] = pw_record.pw_name
+        os.environ["USER"] = pw_record.pw_name
+        os.environ["XDG_CACHE_HOME"] = str(Path(pw_record.pw_dir) / ".cache")
+        os.environ["XDG_CONFIG_HOME"] = str(Path(pw_record.pw_dir) / ".config")
+        os.environ["XDG_DATA_HOME"] = str(Path(pw_record.pw_dir) / ".local" / "share")
+        os.environ.pop("XDG_RUNTIME_DIR", None)
+        os.environ.pop("ABXBUS_MULTIPROCESS_SEMAPHORE_DIR", None)
 
-            semaphore_dir = Path(pw_record.pw_dir) / ".cache" / "abxbus" / "semaphores"
-            os.environ["ABXBUS_MULTIPROCESS_SEMAPHORE_DIR"] = str(semaphore_dir)
+        semaphore_dir = Path(pw_record.pw_dir) / ".cache" / "abxbus" / "semaphores"
+        os.environ["ABXBUS_MULTIPROCESS_SEMAPHORE_DIR"] = str(semaphore_dir)
 
-            try:
-                from abxbus import retry as abxbus_retry
-            except Exception:
-                pass
-            else:
-                abxbus_retry.MULTIPROCESS_SEMAPHORE_DIR = semaphore_dir
+        with suppress(ImportError):
+            from abxbus import retry as abxbus_retry
+
+            abxbus_retry.MULTIPROCESS_SEMAPHORE_DIR = semaphore_dir
 
     if ARCHIVEBOX_USER == 0 or not ARCHIVEBOX_USER_EXISTS:
         print(

archivebox/core/admin_archiveresults.py

@@ -1,7 +1,5 @@
 __package__ = "archivebox.core"
 
-import html
-import json
 import os
 import shlex
 from functools import reduce
@@ -10,20 +8,24 @@ from pathlib import Path
 from urllib.parse import quote
 
 from django.contrib import admin
-from django.core.exceptions import ValidationError
-from django.db.models import Min, Prefetch, Q, TextField
+from django.contrib.admin.actions import delete_selected
+from django.contrib.admin.helpers import ACTION_CHECKBOX_NAME
+from django.core.exceptions import PermissionDenied, SuspiciousOperation, ValidationError
+from django.db.models import Count, Min, Prefetch, Q, Subquery, TextField, Window
 from django.db.models.functions import Cast
-from django.urls import resolve, reverse
-from django.utils import timezone
+from django.shortcuts import redirect
+from django.template.loader import render_to_string
+from django.urls import reverse
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.utils.text import smart_split
 
-from archivebox.base_models.admin import BaseModelAdmin
+from archivebox.base_models.admin import card_fieldset, BaseModelAdmin
 from archivebox.core.models import ArchiveResult, Snapshot
 from archivebox.core.routes_util import build_snapshot_url
-from archivebox.core.widgets import InlineTagEditorWidget
+from archivebox.core.widgets import InlineTagEditorWidget, render_copy_block
 from archivebox.machine.env_util import env_to_shell_exports
+from archivebox.misc.logging_util import printable_filesize
 from archivebox.misc.paginators import AcceleratedPaginator
 from archivebox.plugins.discovery import get_plugin_icon
 from archivebox.plugins.views import LIVE_PLUGIN_BASE_URL
@@ -79,24 +81,22 @@ def get_process_link_label(process) -> str:
     return str(process.id)[-8:]
 
 
-def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
+def render_archiveresults_list(archiveresults_qs, limit=50, config=None, can_delete=False):
     """Render a nice inline list view of archive results with status, plugin, output, and actions."""
 
-    result_ids = list(archiveresults_qs.order_by("plugin").values_list("pk", flat=True)[:limit])
-    if not result_ids:
-        return mark_safe('<div style="color: #64748b; font-style: italic; padding: 16px 0;">No Archive Results yet...</div>')
-
-    results_by_id = {
-        result.pk: result
-        for result in ArchiveResult.objects.filter(pk__in=result_ids).select_related(
+    results = list(
+        ArchiveResult.objects.filter(pk__in=Subquery(archiveresults_qs.order_by().values("pk")))
+        .order_by("plugin")
+        .annotate(_inline_total_count=Window(expression=Count("pk")))
+        .select_related(
             "snapshot",
             "snapshot__crawl",
             "snapshot__crawl__created_by",
             "process",
             "process__machine",
         )
-    }
-    results = [results_by_id[result_id] for result_id in result_ids if result_id in results_by_id]
+        .all()[:limit],
+    )
 
     if not results:
         return mark_safe('<div style="color: #64748b; font-style: italic; padding: 16px 0;">No Archive Results yet...</div>')
@@ -117,278 +117,55 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
     for idx, result in enumerate(results):
         status = result.status or "queued"
         color, bg = status_colors.get(status, ("#6b7280", "#f3f4f6"))
-        output_files = result.output_files or {}
-        if isinstance(output_files, (dict, list, tuple, set)):
-            output_file_count = len(output_files)
-        elif isinstance(output_files, str):
-            try:
-                parsed = json.loads(output_files)
-                output_file_count = len(parsed) if isinstance(parsed, (dict, list, tuple, set)) else 0
-            except (TypeError, ValueError):
-                output_file_count = 0
-        else:
-            output_file_count = 0
-
-        # Get plugin icon
-        icon = get_plugin_icon(result.plugin)
-
-        # Format timestamp
-        end_time = result.end_ts.strftime("%Y-%m-%d %H:%M:%S") if result.end_ts else "-"
-
         process = result.process_record
-        process_display = "-"
-        if process:
-            process_url = html.escape(reverse("admin:machine_process_change", args=[process.id]), quote=True)
-            process_label = html.escape(get_process_link_label(process), quote=True)
-            process_display = f'''
-                <a href="{process_url}"
-                   style="color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 12px;"
-                   title="View process">{process_label}</a>
-            '''
-
-        machine_display = "-"
-        if process and process.machine_id:
-            machine_url = html.escape(reverse("admin:machine_machine_change", args=[process.machine_id]), quote=True)
-            machine_label = html.escape(str(process.machine.hostname or ""), quote=True)
-            machine_display = f'''
-                <a href="{machine_url}"
-                   style="color: #2563eb; text-decoration: none; font-size: 12px;"
-                   title="View machine">{machine_label}</a>
-            '''
-
-        # Truncate output for display
-        full_output_raw = result.output_str_for_display() or "-"
-        output_display_raw = full_output_raw[:60]
-        if len(full_output_raw) > 60:
-            output_display_raw += "..."
-        full_output = html.escape(full_output_raw)
-        output_display = html.escape(output_display_raw)
-
-        display_cmd = build_abx_dl_display_command(result)
-        replay_cmd = build_abx_dl_replay_command(result, config=config)
-        cmd_str_escaped = html.escape(display_cmd)
-        cmd_attr = html.escape(replay_cmd, quote=True)
-
-        # Build output link - use embed_path() which checks output_files first
+        output = result.output_str_for_display() or "-"
         embed_path = result.embed_path()
-        snapshot_id = str(result.snapshot_id)
-        if embed_path and result.status == "succeeded":
-            output_link = build_snapshot_url(snapshot_id, embed_path, config=config)
-        else:
-            output_link = build_snapshot_url(snapshot_id, "", config=config)
-        output_link_attr = html.escape(output_link, quote=True)
-
-        # Get version - try cmd_version field
-        version = html.escape(str(result.cmd_version if result.cmd_version else "-"), quote=True)
-        plugin_text = html.escape(str(result.plugin or ""), quote=True)
-        status_text = html.escape(str(status), quote=True)
-        pwd_text = html.escape(str(result.pwd or "-"), quote=True)
-
-        # Unique ID for this row's expandable output
-        row_id = f"output_{idx}_{str(result.id)[:8]}"
-
-        rows.append(f'''
-            <tr style="border-bottom: 1px solid #f1f5f9; transition: background 0.15s;" onmouseover="this.style.background='#f8fafc'" onmouseout="this.style.background='transparent'">
-                <td style="padding: 10px 12px; white-space: nowrap;">
-                    <a href="{reverse("admin:core_archiveresult_change", args=[result.id])}"
-                       style="color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 11px;"
-                       title="View/edit archive result">
-                        <code>{str(result.id)[-8:]}</code>
-                    </a>
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap;">
-                    <span style="display: inline-block; padding: 3px 10px; border-radius: 12px;
-                                 font-size: 11px; font-weight: 600; text-transform: uppercase;
-                                 color: {color}; background: {bg};">{status_text}</span>
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap; font-size: 20px;" title="{plugin_text}">
-                    {icon}
-                </td>
-                <td style="padding: 10px 12px; font-weight: 500; color: #334155;">
-                        <a href="{output_link_attr}" target="_blank"
-                           style="color: #334155; text-decoration: none;"
-                       title="View output fullscreen"
-                       onmouseover="this.style.color='#2563eb'; this.style.textDecoration='underline';"
-                       onmouseout="this.style.color='#334155'; this.style.textDecoration='none';">
-                        {plugin_text}
-                    </a>
-                </td>
-                <td style="padding: 10px 12px; max-width: 280px;">
-                    <span onclick="document.getElementById('{row_id}').open = !document.getElementById('{row_id}').open"
-                          style="color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 12px; cursor: pointer;"
-                          title="Click to expand full output">
-                        {output_display}
-                    </span>
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap; color: #64748b; font-size: 12px; text-align: right;">
-                    {output_file_count}
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap; color: #64748b; font-size: 12px;">
-                    {end_time}
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap;">
-                    {process_display}
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap;">
-                    {machine_display}
-                </td>
-                <td style="padding: 10px 12px; white-space: nowrap; font-family: ui-monospace, monospace; font-size: 11px; color: #64748b;">
-                    {version}
-                </td>
-                <td style="padding: 10px 8px; white-space: nowrap;">
-                    <div style="display: flex; gap: 4px;">
-                        <a href="{output_link_attr}" target="_blank"
-                           style="padding: 4px 8px; background: #f1f5f9; border-radius: 4px; color: #475569; text-decoration: none; font-size: 11px;"
-                           title="View output">📄</a>
-                        <a href="{reverse("admin:core_archiveresult_change", args=[result.id])}"
-                           style="padding: 4px 8px; background: #f1f5f9; border-radius: 4px; color: #475569; text-decoration: none; font-size: 11px;"
-                           title="Edit">✏️</a>
-                    </div>
-                </td>
-            </tr>
-            <tr style="border-bottom: 1px solid #e2e8f0;">
-                <td colspan="11" style="padding: 0 12px 10px 12px;">
-                    <details id="{row_id}" style="margin: 0;">
-                        <summary style="cursor: pointer; font-size: 11px; color: #94a3b8; user-select: none;">
-                            Details &amp; Output
-                        </summary>
-                        <div style="margin-top: 8px; padding: 10px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 6px; max-height: 200px; overflow: auto;">
-                            <div style="font-size: 11px; color: #64748b; margin-bottom: 8px;">
-                                <span style="margin-right: 16px;"><b>ID:</b> <code>{result.id!s}</code></span>
-                                <span style="margin-right: 16px;"><b>Version:</b> <code>{version}</code></span>
-                                <span style="margin-right: 16px;"><b>PWD:</b> <code>{pwd_text}</code></span>
-                            </div>
-                            <div style="font-size: 11px; color: #64748b; margin-bottom: 8px;">
-                                <b>Output:</b>
-                            </div>
-                            <pre style="margin: 0; padding: 8px; background: #1e293b; border-radius: 4px; color: #e2e8f0; font-size: 12px; white-space: pre-wrap; word-break: break-all; max-height: 120px; overflow: auto;">{full_output}</pre>
-                            <div style="font-size: 11px; color: #64748b; margin-top: 8px;">
-                                <b>Command:</b>
-                            </div>
-                            <div style="position: relative; margin: 0; padding: 8px 56px 8px 8px; background: #1e293b; border-radius: 4px;">
-                                <button type="button"
-                                        data-command="{cmd_attr}"
-                                        onclick="(function(btn){{var text=btn.dataset.command||''; if(navigator.clipboard&&navigator.clipboard.writeText){{navigator.clipboard.writeText(text);}} else {{var ta=document.createElement('textarea'); ta.value=text; document.body.appendChild(ta); ta.select(); document.execCommand('copy'); document.body.removeChild(ta);}}}})(this); return false;"
-                                        style="position: absolute; top: 6px; right: 6px; padding: 2px 8px; border: 0; border-radius: 4px; background: #334155; color: #e2e8f0; font-size: 11px; cursor: pointer;">
-                                    Copy
-                                </button>
-                                <code title="{cmd_attr}" style="display: block; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; color: #e2e8f0; font-size: 11px;">{cmd_str_escaped}</code>
-                            </div>
-                        </div>
-                    </details>
-                </td>
-            </tr>
-        ''')
-
-    total_count = archiveresults_qs.count()
-    footer = ""
-    if total_count > limit:
-        footer = f"""
-            <tr>
-                <td colspan="11" style="padding: 12px; text-align: center; color: #64748b; font-size: 13px; background: #f8fafc;">
-                    Showing {limit} of {total_count} results &nbsp;
-                    <a href="/admin/core/archiveresult/?snapshot__id__exact={results[0].snapshot_id if results else ""}"
-                       style="color: #2563eb;">View all →</a>
-                </td>
-            </tr>
-        """
-
-    return mark_safe(f"""
-        <div style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden; background: #fff; width: 100%;">
-            <table style="width: 100%; border-collapse: collapse; font-size: 14px;">
-                <thead>
-                    <tr style="background: #f8fafc; border-bottom: 2px solid #e2e8f0;">
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Details</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Status</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; width: 32px;"></th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Plugin</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Output</th>
-                        <th style="padding: 10px 12px; text-align: right; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Files</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Completed</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Process</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Machine</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Version</th>
-                        <th style="padding: 10px 8px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Actions</th>
-                    </tr>
-                </thead>
-                <tbody>
-                    {"".join(rows)}
-                    {footer}
-                </tbody>
-            </table>
-        </div>
-    """)
-
-
-class ArchiveResultInline(admin.TabularInline):
-    name = "Archive Results Log"
-    model = ArchiveResult
-    parent_model = Snapshot
-    extra = 0
-    sort_fields = ("end_ts", "plugin", "output_str", "status", "cmd_version")
-    readonly_fields = ("id", "result_id", "completed", "command", "version")
-    fields = ("start_ts", "end_ts", *readonly_fields, "plugin", "cmd", "cmd_version", "pwd", "status", "output_str")
-    ordering = ("end_ts",)
-    show_change_link = True
-
-    def get_parent_object_from_request(self, request):
-        resolved = resolve(request.path_info)
-        try:
-            return self.parent_model.objects.get(pk=resolved.kwargs["object_id"])
-        except (self.parent_model.DoesNotExist, ValidationError):
-            return None
-
-    @admin.display(
-        description="Completed",
-        ordering="end_ts",
-    )
-    def completed(self, obj):
-        return format_html('<p style="white-space: nowrap">{}</p>', obj.end_ts.strftime("%Y-%m-%d %H:%M:%S"))
-
-    def result_id(self, obj):
-        return format_html(
-            '<a href="{}"><code style="font-size: 10px">[{}]</code></a>',
-            reverse("admin:core_archiveresult_change", args=(obj.id,)),
-            str(obj.id)[:8],
+        rows.append(
+            {
+                "result": result,
+                "status": status,
+                "color": color,
+                "bg": bg,
+                "icon": get_plugin_icon(result.plugin),
+                "file_count": result.output_file_stats()[0],
+                "output_size": int(result.output_size or 0),
+                "output_size_display": printable_filesize(int(result.output_size or 0)),
+                "end_date": result.end_ts.strftime("%Y-%m-%d") if result.end_ts else "",
+                "end_time": result.end_ts.strftime("%H:%M:%S") if result.end_ts else "",
+                "process": process,
+                "process_url": reverse("admin:machine_process_change", args=[process.id]) if process else "",
+                "process_label": get_process_link_label(process) if process else "",
+                "machine_url": reverse("admin:machine_machine_change", args=[process.machine_id]) if process and process.machine_id else "",
+                "output": output,
+                "output_preview": output[:60] + ("..." if len(output) > 60 else ""),
+                "display_command": build_abx_dl_display_command(result),
+                "replay_command": build_abx_dl_replay_command(result, config=config),
+                "output_url": build_snapshot_url(
+                    str(result.snapshot_id),
+                    embed_path if embed_path and status == "succeeded" else "",
+                    config=config,
+                ),
+                "change_url": reverse("admin:core_archiveresult_change", args=[result.id]),
+                "version": result.cmd_version or "-",
+                "pwd": str(result.pwd or "-"),
+                "dom_id": f"output_{idx}_{str(result.id)[:8]}",
+                "short_id": str(result.id)[-8:],
+            },
         )
 
-    def command(self, obj):
-        return format_html("<small><code>{}</code></small>", " ".join(obj.cmd or []))
-
-    def version(self, obj):
-        return format_html("<small><code>{}</code></small>", obj.cmd_version or "-")
-
-    def get_formset(self, request, obj=None, **kwargs):
-        formset = super().get_formset(request, obj, **kwargs)
-        snapshot = self.get_parent_object_from_request(request)
-        base_fields = formset.form.base_fields
-        snapshot_output_dir = str(snapshot.output_dir) if snapshot else ""
-
-        # formset.form.base_fields['id'].widget = formset.form.base_fields['id'].hidden_widget()
-
-        # default values for new entries
-        base_fields["status"].initial = "succeeded"
-        base_fields["start_ts"].initial = timezone.now()
-        base_fields["end_ts"].initial = timezone.now()
-        base_fields["cmd_version"].initial = "-"
-        base_fields["pwd"].initial = snapshot_output_dir
-        base_fields["cmd"].initial = '["-"]'
-        base_fields["output_str"].initial = "Manually recorded cmd output..."
-
-        if obj is not None:
-            # hidden values for existing entries and new entries
-            base_fields["start_ts"].widget = base_fields["start_ts"].hidden_widget()
-            base_fields["end_ts"].widget = base_fields["end_ts"].hidden_widget()
-            base_fields["cmd"].widget = base_fields["cmd"].hidden_widget()
-            base_fields["pwd"].widget = base_fields["pwd"].hidden_widget()
-            base_fields["cmd_version"].widget = base_fields["cmd_version"].hidden_widget()
-        return formset
-
-    def get_readonly_fields(self, request, obj=None):
-        if obj is not None:
-            return self.readonly_fields
-        else:
-            return []
+    return mark_safe(
+        render_to_string(
+            "admin/core/archiveresult/inline.html",
+            {
+                "rows": rows,
+                "limit": limit,
+                "total_count": results[0]._inline_total_count,
+                "snapshot_id": results[0].snapshot_id,
+                "can_delete": can_delete,
+                "delete_url": reverse("admin:core_archiveresult_changelist"),
+            },
+        ),
+    )
 
 
 class ArchiveResultAdmin(BaseModelAdmin):
@@ -436,40 +213,14 @@ class ArchiveResultAdmin(BaseModelAdmin):
     autocomplete_fields = ("snapshot",)
 
     fieldsets = (
-        (
-            "Snapshot",
-            {
-                "fields": ("snapshot", "snapshot_info", "tags_str", "admin_actions"),
-                "classes": ("card", "wide"),
-            },
-        ),
-        (
-            "Plugin",
-            {
-                "fields": ("plugin_with_icon", "process_link", "status"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Timing",
-            {
-                "fields": ("start_ts", "end_ts", "created_at", "modified_at"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Command",
-            {
-                "fields": ("cmd", "cmd_str", "cmd_version", "pwd"),
-                "classes": ("card",),
-            },
-        ),
-        (
+        card_fieldset("Snapshot", ("snapshot", "snapshot_info", "tags_str", "admin_actions"), wide=True),
+        card_fieldset("Plugin", ("plugin_with_icon", "process_link", "status")),
+        card_fieldset("Timing", ("start_ts", "end_ts", "created_at", "modified_at")),
+        card_fieldset("Command", ("cmd", "cmd_str", "cmd_version", "pwd")),
+        card_fieldset(
             "Output",
-            {
-                "fields": ("output_str", "output_json", "output_files", "output_size", "output_mimetypes", "output_summary"),
-                "classes": ("card", "wide"),
-            },
+            ("output_str", "output_json", "output_files", "output_size", "output_mimetypes", "output_summary"),
+            wide=True,
         ),
     )
 
@@ -523,34 +274,53 @@ class ArchiveResultAdmin(BaseModelAdmin):
 
     def changelist_view(self, request, extra_context=None):
         self.request = request
+        selected = request.GET.getlist(ACTION_CHECKBOX_NAME)
+        if request.method == "GET" and request.GET.get("action") == "delete_selected" and selected:
+            if not request.user.is_superuser:
+                raise PermissionDenied
+            if len(selected) > 100:
+                raise SuspiciousOperation("Too many ArchiveResults selected for deletion")
+            try:
+                queryset = self.get_queryset(request).filter(pk__in=selected)
+                if not queryset.exists():
+                    snapshot = Snapshot.objects.only("id").filter(pk=request.GET.get("snapshot")).first()
+                    return redirect(build_snapshot_url(str(snapshot.id), "index.html", request=request) if snapshot else request.path)
+            except (ValidationError, ValueError):
+                return redirect(request.path)
+            return delete_selected(self, request, queryset)
+        handoff_snapshot = (
+            request.GET.get("snapshot") if request.method == "POST" and request.GET.get("action") == "delete_selected" else None
+        )
+        if handoff_snapshot:
+            request.GET = request.GET.copy()
+            request.GET.clear()
+            request.META["QUERY_STRING"] = ""
+            try:
+                handoff_snapshot = Snapshot.objects.only("id").filter(pk=handoff_snapshot).first()
+            except (ValidationError, ValueError):
+                handoff_snapshot = None
         saved_list_per_page = self.list_per_page
         self.list_per_page = request.archivebox_config.SNAPSHOTS_PER_PAGE
         try:
-            return super().changelist_view(request, extra_context)
+            response = super().changelist_view(request, extra_context)
+            if (
+                handoff_snapshot
+                and response.status_code in (301, 302)
+                and not ArchiveResult.objects.filter(
+                    pk__in=request.POST.getlist(ACTION_CHECKBOX_NAME),
+                ).exists()
+            ):
+                return redirect(build_snapshot_url(str(handoff_snapshot.id), "index.html", request=request))
+            return response
         finally:
             self.list_per_page = saved_list_per_page
 
     def get_queryset(self, request):
-        ordering = request.GET.get("o")
-        ordering_fields = set()
-        if ordering:
-            for part in ordering.split("."):
-                if not part:
-                    continue
-                try:
-                    idx = abs(int(part)) - 1
-                except ValueError:
-                    continue
-                if 0 <= idx < len(self.list_display):
-                    ordering_fields.add(self.list_display[idx])
+        ordering_fields = self.get_ordering_fields(request)
 
         qs = (
             super()
             .get_queryset(request)
-            .defer(
-                "notes",
-                "output_json",
-            )
             .prefetch_related(
                 Prefetch(
                     "snapshot",
@@ -559,6 +329,10 @@ class ArchiveResultAdmin(BaseModelAdmin):
                 "process__machine",
             )
         )
+        if request.resolver_match.url_name == "core_archiveresult_change":
+            qs = qs.defer("notes")
+        else:
+            qs = qs.defer("notes", "output_json")
         if "tags_inline" in ordering_fields:
             qs = qs.annotate(snapshot_first_tag=Min("snapshot__tags__name"))
         return qs
@@ -743,26 +517,9 @@ class ArchiveResultAdmin(BaseModelAdmin):
 
     @admin.display(description="Command")
     def cmd_str(self, result):
-        request = self.request
-        display_cmd = build_abx_dl_display_command(result)
-        replay_cmd = build_abx_dl_replay_command(result, config=request.archivebox_config)
-        return format_html(
-            """
-            <div style="position: relative; width: 100%; max-width: 100%; overflow: hidden; box-sizing: border-box;">
-                <button type="button"
-                        data-command="{}"
-                        onclick="(function(btn){{var text=btn.dataset.command||''; if(navigator.clipboard&&navigator.clipboard.writeText){{navigator.clipboard.writeText(text);}} else {{var ta=document.createElement('textarea'); ta.value=text; document.body.appendChild(ta); ta.select(); document.execCommand('copy'); document.body.removeChild(ta);}}}})(this); return false;"
-                        style="position: absolute; top: 6px; right: 6px; z-index: 1; padding: 2px 8px; border: 0; border-radius: 4px; background: #e2e8f0; color: #334155; font-size: 11px; cursor: pointer;">
-                    Copy
-                </button>
-                <code title="{}" style="display: block; width: 100%; max-width: 100%; overflow: hidden; text-overflow: ellipsis; white-space: nowrap; padding: 8px 56px 8px 8px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 6px; font-size: 11px; box-sizing: border-box;">
-                    {}
-                </code>
-            </div>
-            """,
-            replay_cmd,
-            replay_cmd,
-            display_cmd,
+        return render_copy_block(
+            build_abx_dl_display_command(result),
+            copy_text=build_abx_dl_replay_command(result, config=self.request.archivebox_config),
         )
 
     def output_display(self, result):

archivebox/core/admin_site.py

@@ -12,11 +12,13 @@ from admin_data_views.admin import (
     get_app_list as adv_get_app_list,
 )
 from django.contrib import admin
-from django.contrib.auth import REDIRECT_FIELD_NAME
+from django.contrib.auth import REDIRECT_FIELD_NAME, get_user_model, login as auth_login
 from django.contrib.auth.decorators import login_not_required
+from django.contrib.auth.forms import UserCreationForm
 from django.contrib.auth.views import LoginView
-from django.db import DatabaseError, connection
+from django.db import DatabaseError, connection, transaction
 from django.http import HttpResponseRedirect
+from django.template.response import TemplateResponse
 from django.urls import reverse
 from django.utils.decorators import method_decorator
 from django.utils.translation import gettext as _
@@ -103,12 +105,15 @@ class ArchiveBoxAdmin(admin.AdminSite):
 
         from django.contrib.admin.forms import AdminAuthenticationForm
 
+        User = get_user_model()
+        first_admin_setup = not User.objects.filter(is_superuser=True).exclude(username="system").exists()
         context = {
             **self.each_context(request),
-            "title": _("Log in"),
+            "title": _("Set up ArchiveBox") if first_admin_setup else _("Log in"),
             "subtitle": None,
             "app_path": request.get_full_path(),
             "username": request.user.get_username(),
+            "first_admin_setup": first_admin_setup,
         }
         if REDIRECT_FIELD_NAME not in request.GET and REDIRECT_FIELD_NAME not in request.POST:
             context[REDIRECT_FIELD_NAME] = reverse("admin:index", current_app=self.name)
@@ -116,6 +121,30 @@ class ArchiveBoxAdmin(admin.AdminSite):
 
         index_path = reverse("admin:index", current_app=self.name)
         request.current_app = self.name
+        if first_admin_setup:
+            form = UserCreationForm(request.POST or None)
+            if request.method == "POST" and form.is_valid():
+                with transaction.atomic():
+                    # Fresh collections contain the system superuser, which also
+                    # provides a row lock while the first real admin is created.
+                    list(User.objects.select_for_update().filter(is_superuser=True).values_list("pk", flat=True))
+                    if User.objects.filter(is_superuser=True).exclude(username="system").exists():
+                        form.add_error(None, _("An admin account was already created. Log in instead."))
+                    else:
+                        user = form.save(commit=False)
+                        user.is_staff = True
+                        user.is_superuser = True
+                        user.save()
+                        auth_login(request, user, backend="django.contrib.auth.backends.ModelBackend")
+                        return HttpResponseRedirect(index_path)
+
+            context["form"] = form
+            return TemplateResponse(
+                request,
+                self.login_template or "admin/login.html",
+                context,
+            )
+
         return ArchiveBoxLoginView.as_view(
             extra_context=context,
             authentication_form=self.login_form or AdminAuthenticationForm,

archivebox/core/admin_snapshots.py

@@ -1,6 +1,8 @@
 __package__ = "archivebox.core"
 
 import json
+from copy import copy
+from urllib.parse import urlencode
 from functools import lru_cache
 from types import SimpleNamespace
 
@@ -9,11 +11,12 @@ from django.urls import path, reverse
 from django.shortcuts import get_object_or_404, redirect
 from django.http import JsonResponse, HttpResponseBadRequest, HttpResponseNotAllowed
 from django.utils import timezone
-from django.utils.html import format_html, format_html_join
+from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 from django.db.models import Q, Count, Exists, F, OuterRef, Prefetch
 from django import forms
 from django.template import Template, RequestContext
+from django.template.loader import render_to_string
 from django.contrib.admin.helpers import ActionForm
 
 from archivebox.config.common import get_config
@@ -23,11 +26,11 @@ from archivebox.misc.logging_util import printable_filesize
 from archivebox.search.admin import SearchResultsAdminMixin, SearchResultsChangeList
 from archivebox.search.views import admin_snapshot_search_stream_view
 from archivebox.core.routes_util import build_snapshot_url, build_web_url
-from archivebox.core.tag_util import get_or_create_tag
 from archivebox.plugins.hooks import discover_hooks
 from archivebox.plugins.discovery import get_plugin_icon, get_plugin_name, get_plugins
 
-from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
+from archivebox.core.widgets import render_permissions_badge, render_snapshot_progress
+from archivebox.base_models.admin import card_fieldset, BaseModelAdmin, ConfigEditorMixin
 
 from archivebox.core.models import Tag, Snapshot, ArchiveResult
 from archivebox.crawls.models import Crawl
@@ -38,7 +41,6 @@ from archivebox.core.permissions import (
     PERMISSIONS_CHOICES,
     PERMISSIONS_META,
     get_snapshot_permissions,
-    normalize_permissions,
 )
 from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
 
@@ -225,19 +227,17 @@ class SnapshotResultHealthListFilter(admin.SimpleListFilter):
 
                 # Rare statuses are faster status-first: use the
                 # (status, snapshot_id) index to find candidate snapshots.
-                snapshot_ids = ArchiveResult.cached_snapshot_ids_with_majority_status(status)
-                queryset = queryset.filter(pk__in=snapshot_ids)
-                queryset._archivebox_count_hint = len(snapshot_ids)
-                queryset.query._archivebox_count_hint = queryset._archivebox_count_hint
-                return queryset
+                return queryset.filter(pk__in=ArchiveResult.snapshot_ids_with_majority_status(status))
         return queryset
 
 
 class SnapshotChangeList(SearchResultsChangeList):
+    snapshot_status_choices = Snapshot.StatusChoices.choices
+
     def __init__(self, request, *args, **kwargs):
         super().__init__(request, *args, **kwargs)
         resolver_name = request.resolver_match.url_name
-        self.embedded_changelist = request.GET.get("_embedded") == "crawl"
+        self.embedded_changelist = request.GET.get("_embedded") in {"crawl", "snapshot"}
         self.snapshot_is_grid_view = not self.embedded_changelist and (
             resolver_name == "grid" or request.path.rstrip("/").endswith("/grid")
         )
@@ -247,7 +247,21 @@ class SnapshotChangeList(SearchResultsChangeList):
         if not snapshot_ids:
             return
 
-        results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
+        status_counts_by_snapshot = {
+            row["snapshot_id"]: row
+            for row in ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids)
+            .values("snapshot_id")
+            .annotate(
+                total=Count("pk"),
+                succeeded=Count("pk", filter=Q(status=ArchiveResult.StatusChoices.SUCCEEDED)),
+                failed=Count("pk", filter=Q(status=ArchiveResult.StatusChoices.FAILED)),
+                running=Count("pk", filter=Q(status=ArchiveResult.StatusChoices.STARTED)),
+                skipped=Count("pk", filter=Q(status=ArchiveResult.StatusChoices.SKIPPED)),
+                noresults=Count("pk", filter=Q(status=ArchiveResult.StatusChoices.NORESULTS)),
+            )
+        }
+
+        output_results_by_snapshot = {snapshot_id: [] for snapshot_id in snapshot_ids}
         seen_plugins = {snapshot_id: set() for snapshot_id in snapshot_ids}
         rows = (
             ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
@@ -258,16 +272,36 @@ class SnapshotChangeList(SearchResultsChangeList):
             if plugin in seen_plugins[snapshot_id]:
                 continue
             seen_plugins[snapshot_id].add(plugin)
-            results_by_snapshot[snapshot_id].append(
+            output_results_by_snapshot[snapshot_id].append(
                 SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
             )
 
         for obj in self.result_list:
-            obj.__dict__["_admin_archiveresults"] = results_by_snapshot[obj.pk]
+            counts = status_counts_by_snapshot.get(obj.pk, {})
+            total = int(counts.get("total") or 0)
+            succeeded = int(counts.get("succeeded") or 0)
+            failed = int(counts.get("failed") or 0)
+            running = int(counts.get("running") or 0)
+            skipped = int(counts.get("skipped") or 0)
+            noresults = int(counts.get("noresults") or 0)
+            completed = succeeded + failed + skipped + noresults
+            obj.__dict__["_admin_progress_stats"] = {
+                "total": total,
+                "succeeded": succeeded,
+                "failed": failed,
+                "running": running,
+                "pending": max(total - completed - running, 0),
+                "skipped": skipped,
+                "noresults": noresults,
+                "percent": int((completed / total * 100) if total else 0),
+                "output_size": obj.output_size or 0,
+                "is_sealed": obj.status not in (obj.StatusChoices.QUEUED, obj.StatusChoices.STARTED, obj.StatusChoices.PAUSED),
+            }
+            obj.__dict__["_admin_output_results"] = output_results_by_snapshot[obj.pk]
 
     def get_results(self, request):
         super().get_results(request)
-        if request.GET.get("_embedded") == "crawl":
+        if request.GET.get("_embedded") in {"crawl", "snapshot"}:
             self.full_result_count = self.result_count
             self.show_full_result_count = True
         self._attach_archiveresult_summaries()
@@ -316,19 +350,8 @@ class SnapshotAdminForm(forms.ModelForm):
 
             # Parse and save tags from tags_editor
             tags_str = self.cleaned_data.get("tags_editor", "")
-            if tags_str:
-                tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
-                tags = []
-                for name in tag_names:
-                    tag, _ = Tag.objects.get_or_create(
-                        name__iexact=name,
-                        defaults={"name": name},
-                    )
-                    tag = Tag.objects.filter(name__iexact=name).first() or tag
-                    tags.append(tag)
-                instance.tags.set(tags)
-            else:
-                instance.tags.clear()
+            tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
+            instance.save_tags(tag_names)
 
         return instance
 
@@ -337,6 +360,16 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
     form = SnapshotAdminForm
     raw_id_fields = ("crawl", "parent_snapshot")
     list_select_related = ()
+
+    def save_related(self, request, form, formsets, change):
+        super().save_related(request, form, formsets, change)
+        tags_str = form.cleaned_data.get("tags_editor", "")
+        tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
+        form.instance.save_tags(
+            tag_names,
+            created_by=request.user if request.user.is_authenticated else None,
+        )
+
     list_display = (
         "permissions_badge",
         "created_at",
@@ -382,70 +415,19 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
                 "classes": ("card", "actions-card"),
             },
         ),
-        (
-            "Snapshot",
-            {
-                "fields": ("snapshot_summary",),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "URL",
-            {
-                "fields": (("url_favicon", "url"), ("title", "tags_badges")),
-                "classes": ("card", "wide"),
-            },
-        ),
-        (
-            "Tags",
-            {
-                "fields": ("tags_editor", "permissions_config"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Status",
-            {
-                "fields": ("status", "retry_at"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Timestamps",
-            {
-                "fields": ("bookmarked_at", "created_at", "modified_at", "downloaded_at"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Relations",
-            {
-                "fields": ("crawl",),
-                "classes": ("card",),
-            },
-        ),
-        (
+        card_fieldset("Snapshot", ("snapshot_summary",)),
+        card_fieldset("URL", (("url_favicon", "url"), ("title", "tags_badges")), wide=True),
+        card_fieldset("Tags", ("tags_editor", "permissions_config")),
+        card_fieldset("Status", ("status", "retry_at")),
+        card_fieldset("Timestamps", ("bookmarked_at", "created_at", "modified_at", "downloaded_at")),
+        card_fieldset("Relations", ("crawl",)),
+        card_fieldset(
             "Config",
-            {
-                "fields": ("config",),
-                "description": '<span style="display:block; margin:-4px 0 6px; font-size:11px; line-height:1.35; color:#94a3b8;">Uses <code>Crawl.config</code> by default. Only set per-snapshot overrides here when needed.</span>',
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Files",
-            {
-                "fields": ("output_dir",),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Archive Results",
-            {
-                "fields": ("archiveresults_list",),
-                "classes": ("card", "wide"),
-            },
+            ("config",),
+            description='<span style="display:block; margin:-4px 0 6px; font-size:11px; line-height:1.35; color:#94a3b8;">Uses <code>Crawl.config</code> by default. Only set per-snapshot overrides here when needed.</span>',
         ),
+        card_fieldset("Files", ("output_dir",)),
+        card_fieldset("Archive Results", ("archiveresults_list",), wide=True),
     )
 
     ordering = ["-created_at"]
@@ -475,24 +457,23 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             return []
         return super().get_ordering(request)
 
-    def change_view(self, request, object_id, form_url="", extra_context=None):
+    def render_change_form(self, request, context, add=False, change=False, form_url="", obj=None):
         self.request = request
-        extra_context = extra_context or {}
-        extra_context["CONFIG"] = request.archivebox_config
-        snapshot = self.get_object(request, object_id)
-        if snapshot and snapshot.status in {
+        context["CONFIG"] = request.archivebox_config
+        if obj and obj.status in {
             Snapshot.StatusChoices.QUEUED,
             Snapshot.StatusChoices.STARTED,
             Snapshot.StatusChoices.PAUSED,
         }:
-            extra_context["progress_auto_expand"] = True
-            extra_context["progress_endpoint"] = progress_endpoint("snapshot", snapshot.id)
-        return super().change_view(request, object_id, form_url, extra_context | GLOBAL_CONTEXT)
+            context["progress_auto_expand"] = True
+            context["progress_endpoint"] = progress_endpoint("snapshot", obj.id)
+        context.update(GLOBAL_CONTEXT)
+        return super().render_change_form(request, context, add=add, change=change, form_url=form_url, obj=obj)
 
     def changelist_view(self, request, extra_context=None):
         self.request = request
         saved_list_per_page = self.list_per_page
-        embedded_changelist = request.GET.get("_embedded") == "crawl"
+        embedded_changelist = request.GET.get("_embedded") in {"crawl", "snapshot"}
         if embedded_changelist:
             try:
                 requested_per_page = int(request.GET.get("per_page", "200"))
@@ -513,6 +494,27 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         finally:
             self.list_per_page = saved_list_per_page
 
+    def render_embedded_changelist(self, request, *, filters, title, default_search_mode=None):
+        """Reuse snapshot search, rows, pagination, and actions in parent admin pages."""
+        changelist_path = reverse(f"{self.admin_site.name}:core_snapshot_changelist")
+        full_url = f"{changelist_path}?{urlencode(filters)}"
+        changelist_request = copy(request)
+        changelist_request.method = "GET"
+        changelist_request.path = changelist_path
+        changelist_request.GET = request.GET.copy()
+        if default_search_mode:
+            changelist_request.GET.setdefault("search_mode", default_search_mode)
+        changelist_request.GET.update({**filters, "_embedded": "snapshot", "per_page": "200"})
+        changelist_request.POST = request.POST.copy()
+        changelist_request.POST.clear()
+        response = self.changelist_view(changelist_request)
+        context = {
+            **response.context_data,
+            "snapshot_changelist_url": full_url,
+            "snapshot_changelist_title": title,
+        }
+        return mark_safe(render_to_string("admin/core/snapshot/embedded_changelist.html", context, request=request))
+
     def get_actions(self, request):
         actions = super().get_actions(request)
         if not actions:
@@ -521,7 +523,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         return actions
 
     def lookup_allowed(self, lookup, value, request=None):
-        if lookup in {"crawl__id__exact", "crawl_id__exact", "crawl_id"}:
+        if lookup in {"crawl__id__exact", "crawl_id__exact", "crawl_id", "crawl__schedule__id__exact"}:
             return True
         return super().lookup_allowed(lookup, value, request=request)
 
@@ -610,19 +612,19 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             if retried:
                 messages.success(
                     request,
-                    f"Queued {retried} failed/skipped extractors for retry on this snapshot.",
+                    f"Queued {retried} failed extractors for retry on this snapshot.",
                 )
             else:
                 messages.info(
                     request,
-                    "No failed/skipped extractors were found on this snapshot.",
+                    "No failed extractors were found on this snapshot.",
                 )
 
         return redirect(snapshot.admin_change_url)
 
     def get_queryset(self, request):
         self.request = request
-        ordering_fields = self._get_ordering_fields(request)
+        ordering_fields = self.get_ordering_fields(request)
         needs_files_sort = "files" in ordering_fields
         needs_tags_sort = "tags_inline" in ordering_fields
         is_change_view = request.resolver_match.url_name == "core_snapshot_change"
@@ -686,45 +688,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         permissions = obj.__dict__.get("snapshot_permissions")
         if permissions is None:
             permissions = obj.permissions
-        permissions = normalize_permissions(permissions)
-        icon, label, fg, bg = SNAPSHOT_PERMISSION_META[permissions]
-        menu_items = format_html_join(
-            "",
-            (
-                '<button type="button" class="snapshot-permissions-menu-item{}" data-permissions="{}">'
-                '<span class="snapshot-permissions-icon" aria-hidden="true" style="color:{}; background:{};">{}</span>'
-                "<span>{}</span>"
-                "</button>"
-            ),
-            (
-                (
-                    " is-active" if choice_value == permissions else "",
-                    choice_value,
-                    choice_fg,
-                    choice_bg,
-                    choice_icon,
-                    choice_label,
-                )
-                for choice_value, choice_label in PERMISSIONS_CHOICES
-                for choice_icon, _choice_title, choice_fg, choice_bg in [SNAPSHOT_PERMISSION_META[choice_value]]
-            ),
-        )
-        return format_html(
-            '<span class="snapshot-permissions-quick" data-current-permissions="{}" data-permissions-url="{}">'
-            '<button type="button" class="snapshot-permissions-button snapshot-permissions-{}" title="{}" aria-label="Change snapshot permissions: {}" aria-expanded="false">'
-            '<span class="snapshot-permissions-icon" aria-hidden="true" style="color:{}; background:{};">{}</span>'
-            "</button>"
-            '<span class="snapshot-permissions-menu" role="menu" hidden>{}</span>'
-            "</span>",
+        return render_permissions_badge(
             permissions,
-            reverse(f"{self.admin_site.name}:core_snapshot_set_permissions", args=[obj.pk]),
-            permissions,
-            label,
-            label,
-            fg,
-            bg,
-            icon,
-            menu_items,
+            url=reverse(f"{self.admin_site.name}:core_snapshot_set_permissions", args=[obj.pk]),
+            object_name="snapshot",
         )
 
     @admin.display(description="Imported Timestamp")
@@ -824,27 +791,15 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             obj.pk,
         )
 
-    def status_info(self, obj):
-        request = self.request
-        config = request.archivebox_config
-        favicon_url = build_snapshot_url(str(obj.id), "favicon.ico", request=request, config=config)
-        return format_html(
-            """
-            Archived: {} ({} files {}) &nbsp; &nbsp;
-            Favicon: <img src="{}" style="height: 20px"/> &nbsp; &nbsp;
-            Extension: {} &nbsp; &nbsp;
-            """,
-            "✅" if obj.is_archived else "❌",
-            obj.num_outputs,
-            self.size(obj) or "0kb",
-            favicon_url,
-            obj.extension or "-",
-        )
-
     @admin.display(description="Archive Results")
     def archiveresults_list(self, obj):
         request = self.request
-        return render_archiveresults_list(obj.archiveresult_set.all(), limit=8, config=request.archivebox_config)
+        return render_archiveresults_list(
+            obj.archiveresult_set.all(),
+            limit=8,
+            config=request.archivebox_config,
+            can_delete=request.user.is_superuser,
+        )
 
     @admin.display(
         description="Title",
@@ -1156,36 +1111,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
 
         # For started snapshots, show progress bar
         if obj.status == "started" and stats["total"] > 0:
-            percent = stats["percent"]
-            running = stats["running"]
-            succeeded = stats["succeeded"]
-            failed = stats["failed"]
-
-            return format_html(
-                """<div style="min-width: 90px;">
-                    <div style="display: flex; align-items: center; gap: 6px; margin-bottom: 4px;">
-                        <span class="snapshot-progress-spinner"></span>
-                        <span style="font-size: 11px; color: #64748b;">{}/{} hooks</span>
-                    </div>
-                    <div style="background: #e2e8f0; border-radius: 4px; height: 6px; overflow: hidden;">
-                        <div style="background: linear-gradient(90deg, #10b981 0%, #10b981 {}%, #ef4444 {}%, #ef4444 {}%, #3b82f6 {}%, #3b82f6 100%);
-                                    width: {}%; height: 100%; transition: width 0.3s;"></div>
-                    </div>
-                    <div style="font-size: 10px; color: #94a3b8; margin-top: 2px;">
-                        ✓{} ✗{} ⏳{}
-                    </div>
-                </div>""",
-                succeeded + failed + stats["skipped"],
-                stats["total"],
-                int(succeeded / stats["total"] * 100) if stats["total"] else 0,
-                int(succeeded / stats["total"] * 100) if stats["total"] else 0,
-                int((succeeded + failed) / stats["total"] * 100) if stats["total"] else 0,
-                int((succeeded + failed) / stats["total"] * 100) if stats["total"] else 0,
-                percent,
-                succeeded,
-                failed,
-                running,
-            )
+            return render_snapshot_progress(stats)
 
         # For other statuses, show simple badge
         return format_html(
@@ -1233,50 +1159,16 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         )
 
     def _get_progress_stats(self, obj):
-        cached_stats = obj.__dict__.get("_admin_progress_stats")
-        if cached_stats is not None:
-            return cached_stats
-
-        results = self._get_prefetched_results(obj)
-        if results is None:
-            stats = obj.get_progress_stats()
-            expected_total = self._get_expected_hook_total(obj)
-            total = max(stats["total"], expected_total)
-            completed = stats["succeeded"] + stats["failed"] + stats.get("skipped", 0) + stats.get("noresults", 0)
-            stats["total"] = total
-            stats["pending"] = max(total - completed - stats["running"], 0)
-            stats["percent"] = int((completed / total * 100) if total > 0 else 0)
-            obj._admin_progress_stats = stats
-            return stats
-
-        expected_total = self._get_expected_hook_total(obj)
-        observed_total = len(results)
-        total = max(observed_total, expected_total)
-        succeeded = sum(1 for r in results if r.status == "succeeded")
-        failed = sum(1 for r in results if r.status == "failed")
-        running = sum(1 for r in results if r.status == "started")
-        skipped = sum(1 for r in results if r.status == "skipped")
-        noresults = sum(1 for r in results if r.status == "noresults")
-        pending = max(total - succeeded - failed - running - skipped - noresults, 0)
-        completed = succeeded + failed + skipped + noresults
-        percent = int((completed / total * 100) if total > 0 else 0)
-        is_sealed = obj.status not in (obj.StatusChoices.QUEUED, obj.StatusChoices.STARTED, obj.StatusChoices.PAUSED)
-        stats = {
-            "total": total,
-            "succeeded": succeeded,
-            "failed": failed,
-            "running": running,
-            "pending": pending,
-            "skipped": skipped,
-            "noresults": noresults,
-            "percent": percent,
-            "output_size": obj.output_size or 0,
-            "is_sealed": is_sealed,
-        }
-        obj._admin_progress_stats = stats
-        return stats
+        if "_admin_progress_stats" not in obj.__dict__:
+            obj._admin_progress_stats = obj.get_progress_stats(
+                results=self._get_prefetched_results(obj),
+                expected_total=self._get_expected_hook_total(obj),
+            )
+        return obj._admin_progress_stats
 
     def _get_prefetched_results(self, obj):
+        if "_admin_output_results" in obj.__dict__:
+            return obj.__dict__["_admin_output_results"]
         if "_admin_archiveresults" in obj.__dict__:
             return obj.__dict__["_admin_archiveresults"]
         if "archiveresult_set" in obj.__dict__.get("_prefetched_objects_cache", {}):
@@ -1328,22 +1220,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             return list(prefetched_cache["tags"])
         return None
 
-    def _get_ordering_fields(self, request):
-        ordering = request.GET.get("o")
-        if not ordering:
-            return set()
-        fields = set()
-        for part in ordering.split("."):
-            if not part:
-                continue
-            try:
-                idx = abs(int(part)) - 1
-            except ValueError:
-                continue
-            if 0 <= idx < len(self.list_display):
-                fields.add(self.list_display[idx])
-        return fields
-
     @admin.display(
         description="Original URL",
         ordering="url",
@@ -1355,12 +1231,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             obj.url[:128],
         )
 
-    @admin.display(description="Health", ordering="health")
-    def health_display(self, obj):
-        h = obj.health
-        color = "green" if h >= 80 else "orange" if h >= 50 else "red"
-        return format_html('<span style="color: {};">{}</span>', color, h)
-
     def grid_view(self, request, extra_context=None):
         extra_context = extra_context or {}
         extra_context["snapshot_is_grid_view"] = True
@@ -1377,10 +1247,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         if queued:
             messages.success(
                 request,
-                f"Queued {queued} failed/skipped extractors for retry. The background runner will process them.",
+                f"Queued {queued} failed extractors for retry. The background runner will process them.",
             )
         else:
-            messages.info(request, "No failed/skipped extractors were found in the selected snapshots.")
+            messages.info(request, "No failed extractors were found in the selected snapshots.")
 
     @admin.action(
         description="🆕 Archive Now",
@@ -1464,7 +1334,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         tag_names = [name.strip() for name in tags_str.split(",") if name.strip()]
         tags = []
         for name in tag_names:
-            tag, _ = get_or_create_tag(
+            tag, _ = Tag.get_or_create_by_name(
                 name,
                 created_by=request.user if request.user.is_authenticated else None,
             )

archivebox/core/admin_tags.py

@@ -11,7 +11,7 @@ from django.urls import reverse
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 
-from archivebox.base_models.admin import BaseModelAdmin
+from archivebox.base_models.admin import card_fieldset, BaseModelAdmin
 from archivebox.core.models import SnapshotTag, Tag
 from archivebox.core.tag_util import (
     TAG_HAS_SNAPSHOTS_CHOICES,
@@ -69,44 +69,14 @@ class TagAdmin(BaseModelAdmin):
     ordering = ["name", "id"]
 
     fieldsets = (
-        (
-            "Tag",
-            {
-                "fields": ("name",),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Metadata",
-            {
-                "fields": ("id", "created_by", "created_at", "modified_at"),
-                "classes": ("card",),
-            },
-        ),
-        (
-            "Recent Snapshots",
-            {
-                "fields": ("snapshots",),
-                "classes": ("card", "wide"),
-            },
-        ),
+        card_fieldset("Tag", ("name",)),
+        card_fieldset("Metadata", ("id", "created_by", "created_at", "modified_at")),
+        card_fieldset("Recent Snapshots", ("snapshots",), wide=True),
     )
 
     add_fieldsets = (
-        (
-            "Tag",
-            {
-                "fields": ("name",),
-                "classes": ("card", "wide"),
-            },
-        ),
-        (
-            "Metadata",
-            {
-                "fields": ("created_by",),
-                "classes": ("card",),
-            },
-        ),
+        card_fieldset("Tag", ("name",), wide=True),
+        card_fieldset("Metadata", ("created_by",)),
     )
 
     def get_fieldsets(self, request: HttpRequest, obj: Tag | None = None):

archivebox/core/admin_users.py

@@ -96,7 +96,7 @@ class CustomUserAdmin(UserAdmin):
             + f'<br/><a href="/admin/core/snapshot/?created_by__id__exact={obj.pk}">{total_count} total records...<a>',
         )
 
-    @admin.display(description="Archive Result Logs")
+    @admin.display(description="Archive Results")
     def archiveresult_set(self, obj):
         total_count = obj.archiveresult_set.count()
         return mark_safe(

archivebox/core/apps.py

@@ -10,7 +10,6 @@ class CoreConfig(AppConfig):
 
     def ready(self):
         """Register the archivebox.core.admin_site as the main django admin site"""
-        import sys
         from django.utils.autoreload import DJANGO_AUTORELOAD_ENV
 
         from archivebox.core.admin_site import register_admin_site
@@ -28,11 +27,6 @@ class CoreConfig(AppConfig):
 
         pre_save.connect(truncate_overlong_charfields, dispatch_uid="archivebox_truncate_overlong_charfields")
 
-        # Import models to register state machines with the registry
-        # Skip during makemigrations to avoid premature state machine access
-        if "makemigrations" not in sys.argv:
-            from archivebox.core import models  # noqa: F401
-
         def _should_prepare_runtime() -> bool:
             if os.environ.get("ARCHIVEBOX_RUNSERVER") == "1":
                 if os.environ.get("ARCHIVEBOX_AUTORELOAD") == "1":

archivebox/core/asgi.py

@@ -7,69 +7,18 @@ For more information on this file, see
 https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
 """
 
-from archivebox.config.django import setup_django
 from django.core.asgi import get_asgi_application
 
-setup_django(in_memory_db=False, check_db=True)
+from archivebox.config.django import setup_django
+
+setup_django(check_db=True)
+
+django_application = get_asgi_application()
 
 
-def _patch_thread_sensitive_context_shutdown() -> None:
-    """Stop ``ThreadSensitiveContext.__aexit__`` from blocking the daphne loop.
+async def application(scope, receive, send):
+    if scope["type"] == "websocket":
+        from archivebox.opencode.views import websocket_view
 
-    Django 6.0's ASGIHandler wraps every request in ``async with
-    ThreadSensitiveContext():`` (django/core/handlers/asgi.py:169). On exit
-    asgiref calls ``executor.shutdown()`` with the default ``wait=True``
-    (asgiref/sync.py:148), which is a *synchronous* ``Thread.join()`` inside
-    an async function — so it blocks the daphne event loop until the
-    executor's worker thread exits.
-
-    That's normally fine because the request handler has already awaited
-    every ``sync_to_async`` it submitted, so the worker is idle and dies as
-    soon as the shutdown sentinel reaches it. The blocking turns into a
-    problem when a client disconnects mid-request:
-
-    * ``SyncToAsync.__call__`` shields the executor work with
-      ``await asyncio.shield(exec_coro)`` (asgiref/sync.py:506) so that the
-      sync DB call doesn't get torn down halfway through.
-    * On cancellation it calls ``exec_coro.cancel()`` (line 522) which only
-      flips the asyncio ``Future`` to cancelled — the underlying thread
-      keeps running the SQL query.
-    * Control unwinds to ``__aexit__`` while the orphaned thread is still
-      mid-query. ``shutdown(wait=True)`` then blocks the event loop until
-      that orphan finishes.
-
-    Under heavy SQLite contention (the load-test scenario that surfaced
-    this on cabbage) those orphan threads can take 30 seconds each waiting
-    for write locks, and the daphne loop is single-threaded — so every
-    such orphan stalls every other in-flight request, healthchecks time
-    out, and the container goes ``unhealthy``.
-
-    Switching to ``shutdown(wait=False)`` queues the sentinel and returns
-    immediately; the worker thread still exits cleanly once its current
-    task finishes, and asgiref's ``WeakKeyDictionary`` releases the
-    executor as soon as the request's context is GC'd. No per-request
-    teardown guarantee is lost — there was no caller relying on it.
-    """
-    from asgiref import sync as _asgiref_sync
-
-    original_aexit = _asgiref_sync.ThreadSensitiveContext.__aexit__
-
-    async def __aexit__(self, exc, value, tb):  # type: ignore[no-redef]
-        if not self.token:
-            return
-        executor = _asgiref_sync.SyncToAsync.context_to_thread_executor.pop(self, None)
-        if executor is not None:
-            executor.shutdown(wait=False)
-        _asgiref_sync.SyncToAsync.thread_sensitive_context.reset(self.token)
-
-    # Idempotent: only patch once even if asgi.py is reloaded.
-    if getattr(original_aexit, "_archivebox_patched", False):
-        return
-    __aexit__._archivebox_patched = True  # type: ignore[attr-defined]
-    _asgiref_sync.ThreadSensitiveContext.__aexit__ = __aexit__
-
-
-_patch_thread_sensitive_context_shutdown()
-
-# Standard Django ASGI application (no websockets/channels needed)
-application = get_asgi_application()
+        return await websocket_view(scope, receive, send)
+    return await django_application(scope, receive, send)

archivebox/core/forms.py

@@ -9,12 +9,13 @@ from archivebox.base_models.admin import KeyValueWidget
 from archivebox.config.common import get_config, parse_delete_after
 from archivebox.core.permissions import PERMISSIONS_CHOICES, PERMISSIONS_PUBLIC, filter_personas_by_permissions
 from archivebox.core.widgets import TagEditorWidget, URLFiltersWidget
+from archivebox.crawls.forms import URLFiltersField
 from archivebox.crawls.schedule_util import validate_schedule
 from archivebox.misc.util import URL_REGEX, find_all_urls, parse_filesize_to_bytes
 from archivebox.personas.models import Persona
-from archivebox.plugins.discovery import get_plugins
+from archivebox.plugins.discovery import get_plugin_catalog
 from archivebox.plugins.forms import (
-    PLUGIN_GROUP_DEFINITIONS,
+    PLUGIN_GROUPS,
     TIMEOUT_INPUT_PATTERN,
     PluginConfigFormMixin,
     get_choice_field,
@@ -90,6 +91,18 @@ def edit_string_for_tag_names(tags) -> str:
     return ", ".join(sorted(names))
 
 
+class FileSizeField(forms.CharField):
+    def to_python(self, value):
+        raw_value = super().to_python(value)
+        try:
+            size = parse_filesize_to_bytes(raw_value) if raw_value else 0
+        except ValueError as err:
+            raise forms.ValidationError(str(err)) from err
+        if size < 0:
+            raise forms.ValidationError(f"{self.label} must be 0 or a positive number of bytes.")
+        return size
+
+
 class AddLinkForm(PluginConfigFormMixin, forms.Form):
     allow_crawl_execution_config_fields = False
 
@@ -136,7 +149,7 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             },
         ),
     )
-    crawl_max_size = forms.CharField(
+    crawl_max_size = FileSizeField(
         label="Max crawl size",
         required=False,
         initial="0",
@@ -169,7 +182,7 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             },
         ),
     )
-    snapshot_max_size = forms.CharField(
+    snapshot_max_size = FileSizeField(
         label="Max snapshot size",
         required=False,
         initial="0",
@@ -210,7 +223,7 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             },
         ),
     )
-    url_filters = forms.Field(
+    url_filters = URLFiltersField(
         label="URL allowlist / denylist",
         required=False,
         widget=URLFiltersWidget(source_selector='textarea[name="url"]'),
@@ -333,10 +346,9 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
         if self.can_override_crawl_config:
             self.build_plugin_groups(get_config(persona=selected_persona) if selected_persona else get_config())
         else:
-            all_plugins = get_plugins()
-            for field_name, *_rest, plugin_names in PLUGIN_GROUP_DEFINITIONS:
-                get_choice_field(self, field_name).choices = [(p, p) for p in all_plugins if p in plugin_names]
-            get_choice_field(self, "other_plugins").choices = [(p, p) for p in all_plugins]
+            grouped_plugins = get_plugin_catalog().groups()
+            for category, field_name, _title in PLUGIN_GROUPS:
+                get_choice_field(self, field_name).choices = [(plugin.name, plugin.name) for plugin in grouped_plugins.get(category, [])]
             self.plugin_groups = []
 
     def clean(self):
@@ -348,23 +360,7 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             cleaned_data["config"] = {}
             return cleaned_data
 
-        # Combine all plugin groups into single list
-        all_selected_plugins = []
-        for field in [
-            "main_plugins",
-            "page_setup_plugins",
-            "media_plugins",
-            "text_plugins",
-            "metadata_plugins",
-            "postprocessing_plugins",
-            "other_plugins",
-        ]:
-            selected = cleaned_data.get(field)
-            if isinstance(selected, list):
-                all_selected_plugins.extend(selected)
-
-        # Store combined list for easy access
-        cleaned_data["plugins"] = all_selected_plugins
+        cleaned_data["plugins"] = [plugin for _category, field, _title in PLUGIN_GROUPS for plugin in cleaned_data.get(field, [])]
 
         plugin_config_overrides = self.clean_plugin_config_overrides(get_config(persona=cleaned_data.get("persona")))
         custom_config = cleaned_data.get("config") or {}
@@ -381,34 +377,10 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             raise forms.ValidationError("Enter at least one valid URL.")
         return value
 
-    def clean_url_filters(self):
-        from archivebox.crawls.models import Crawl
-
-        value = self.cleaned_data.get("url_filters") or {}
-        return {
-            "allowlist": "\n".join(Crawl.split_filter_patterns(value.get("allowlist", ""))),
-            "denylist": "\n".join(Crawl.split_filter_patterns(value.get("denylist", ""))),
-            "same_domain_only": bool(value.get("same_domain_only")),
-            "subpaths_only": bool(value.get("subpaths_only")),
-            "only_new": bool(value.get("only_new")),
-        }
-
     def clean_max_urls(self):
         value = self.cleaned_data.get("max_urls")
         return int(value or 0)
 
-    def clean_crawl_max_size(self):
-        raw_value = str(self.cleaned_data.get("crawl_max_size") or "").strip()
-        if not raw_value:
-            return 0
-        try:
-            value = parse_filesize_to_bytes(raw_value)
-        except ValueError as err:
-            raise forms.ValidationError(str(err))
-        if value < 0:
-            raise forms.ValidationError("Max crawl size must be 0 or a positive number of bytes.")
-        return value
-
     def clean_crawl_timeout(self):
         return self._clean_timeout_seconds(self.cleaned_data.get("crawl_timeout"), "Max crawl time", blank_value=0)
 
@@ -440,18 +412,6 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             raise forms.ValidationError(f"{field_label} must be 0 or greater than 10 seconds.")
         return value
 
-    def clean_snapshot_max_size(self):
-        raw_value = str(self.cleaned_data.get("snapshot_max_size") or "").strip()
-        if not raw_value:
-            return 0
-        try:
-            value = parse_filesize_to_bytes(raw_value)
-        except ValueError as err:
-            raise forms.ValidationError(str(err))
-        if value < 0:
-            raise forms.ValidationError("Max snapshot size must be 0 or a positive number of bytes.")
-        return value
-
     def clean_delete_after(self):
         raw_value = str(self.cleaned_data.get("delete_after") or "0").strip() or "0"
         try:

archivebox/core/middleware.py

@@ -15,7 +15,7 @@ from django.utils import timezone
 from django.utils.http import http_date
 
 from archivebox.config import VERSION
-from archivebox.config.common import get_config
+from archivebox.config.common import get_config, get_request_config
 from archivebox.config.version import get_COMMIT_HASH
 from archivebox.core.routes_util import (
     build_admin_url,
@@ -39,8 +39,8 @@ ADMIN_LOGIN_HINT_COOKIE = "archivebox_admin_logged_in"
 def _admin_login_hint_cookie_domain(config) -> str | None:
     """Resolve the parent domain to scope the cross-subdomain login hint.
 
-    NOTE: this cookie carries only the single bit "user is logged in on
-    admin somewhere"; it MUST NOT be confused with the session cookie,
+    NOTE: this cookie carries only the single bit "a superuser is logged in
+    on admin somewhere"; it MUST NOT be confused with the session cookie,
     which stays admin-host-scoped (see core/settings.py
     SESSION_COOKIE_DOMAIN comment — admin/web is a security boundary).
 
@@ -88,6 +88,11 @@ def AdminCookieIsolationMiddleware(get_response):
     def middleware(request):
         response = get_response(request)
 
+        if request.path == "/admin" or request.path.startswith("/admin/"):
+            if not getattr(response, "xframe_options_exempt", False):
+                response.headers["X-Frame-Options"] = "DENY"
+                response.headers["Content-Security-Policy"] = "frame-ancestors 'none'"
+
         config = request.__dict__.get("archivebox_config")
         if config is None or config.SERVER_SECURITY_MODE == "auto":
             from archivebox.config.common import get_request_config
@@ -178,13 +183,40 @@ def ServerSecurityModeMiddleware(get_response):
                 api_host, _api_port = split_host_port(get_api_host(config=base_config))
                 web_host, _web_port = split_host_port(get_web_host(config=base_config))
                 control_hosts = {host for host in (base_host, admin_host, api_host, web_host) if host}
-                if control_hosts and request_host not in control_hosts:
+                credentialed_api_request = request.path == "/api/v1/auth/check_api_token" or bool(
+                    request.META.get("HTTP_X_ARCHIVEBOX_API_KEY") or request.META.get("HTTP_AUTHORIZATION"),
+                )
+                if not base_config.BASE_URL and credentialed_api_request:
+                    control_hosts.add(request_host)
+                first_run_setup_request = not base_config.BASE_URL and (
+                    request.path == "/admin/login/"
+                    or (
+                        bool(getattr(request.user, "is_superuser", False))
+                        and re.fullmatch(r"/admin/machine/machine/[0-9a-f-]{32,36}/change/", request.path)
+                    )
+                )
+                if control_hosts and request_host not in control_hosts and not first_run_setup_request:
                     return HttpResponseForbidden("ArchiveBox is running with the control plane disabled on this host.")
 
             from archivebox.config.common import get_request_config
 
             config = get_request_config(request, resolve_plugins=False)
 
+        if config.USES_SUBDOMAIN_ROUTING and config.BASE_URL and request.method.upper() not in allowed_methods:
+            request_host, _request_port = split_host_port((request.get_host() or "").lower())
+            control_hosts = {
+                split_host_port(host)[0]
+                for host in (
+                    get_base_host(config=config),
+                    get_admin_host(config=config),
+                    get_api_host(config=config),
+                    get_web_host(config=config),
+                )
+                if host
+            }
+            if request_host not in control_hosts:
+                return HttpResponseForbidden("ArchiveBox is running with the control plane disabled on this host.")
+
         if config.CONTROL_PLANE_ENABLED:
             return get_response(request)
 
@@ -207,6 +239,10 @@ def HostRoutingMiddleware(get_response):
     snapshot_path_re = re.compile(
         r"^/(?P<username>[^/]+)/(?P<date>\d{4}(?:\d{2})?(?:\d{2})?)/(?P<domain>[^/]+)/(?P<snapshot_id>[0-9a-fA-F-]{8,36})(?:/(?P<path>.*))?$",
     )
+    snapshot_replay_path_re = re.compile(
+        r"^/snapshot/(?P<snapshot_id>[0-9a-fA-F-]{8,36})(?:/(?P<path>.*))?$",
+    )
+    original_replay_path_re = re.compile(r"^/original/(?P<domain>[^/]+)(?:/(?P<path>.*))?$")
 
     def middleware(request):
         if request.path in {"/health", "/health/"}:
@@ -214,14 +250,13 @@ def HostRoutingMiddleware(get_response):
 
         request_host = (request.get_host() or "").lower()
         config = request.__dict__.get("archivebox_config")
-        if config is None:
-            config = get_config(resolve_plugins=False)
-            request.archivebox_config = config
-        admin_host = get_admin_host(config=config)
-        web_host = get_web_host(config=config)
-        api_host = get_api_host(config=config)
+        if config is None or config.SERVER_SECURITY_MODE == "auto":
+            config = get_request_config(request, resolve_plugins=False)
+        admin_host = get_admin_host(config=config, request=request)
+        web_host = get_web_host(config=config, request=request)
+        api_host = get_api_host(config=config, request=request)
         listen_host = get_listen_host(config=config)
-        subdomain = get_listen_subdomain(request_host, config=config)
+        subdomain = get_listen_subdomain(request_host, config=config, request=request)
 
         # Framework-owned assets must bypass snapshot/original-domain replay routing.
         # Otherwise pages on snapshot subdomains can receive HTML for JS/CSS requests.
@@ -256,6 +291,27 @@ def HostRoutingMiddleware(get_response):
         if config.USES_SUBDOMAIN_ROUTING and not config.BASE_URL:
             return get_response(request)
 
+        if config.USES_SUBDOMAIN_ROUTING:
+            snapshot_replay_match = snapshot_replay_path_re.match(request.path)
+            if snapshot_replay_match:
+                target = build_snapshot_url(
+                    snapshot_replay_match.group("snapshot_id"),
+                    (snapshot_replay_match.group("path") or "").strip("/"),
+                    request=request,
+                )
+                if request.META.get("QUERY_STRING"):
+                    target = f"{target}?{request.META['QUERY_STRING']}"
+                return redirect(target)
+
+            original_replay_match = original_replay_path_re.match(request.path)
+            if original_replay_match:
+                view = OriginalDomainHostView.as_view()
+                return view(
+                    request,
+                    domain=original_replay_match.group("domain"),
+                    path=(original_replay_match.group("path") or "").strip("/"),
+                )
+
         if not config.USES_SUBDOMAIN_ROUTING:
             if host_matches(request_host, listen_host):
                 return get_response(request)
@@ -283,7 +339,12 @@ def HostRoutingMiddleware(get_response):
                 return redirect(target)
             response = get_response(request)
             hint_cookie_domain = _admin_login_hint_cookie_domain(config)
-            if request.user.is_authenticated and not request.path.startswith("/admin/logout"):
+            if (
+                request.user.is_authenticated
+                and request.user.is_active
+                and request.user.is_superuser
+                and not request.path.startswith("/admin/logout")
+            ):
                 response.set_cookie(
                     ADMIN_LOGIN_HINT_COOKIE,
                     "1",
@@ -354,7 +415,7 @@ class ReverseProxyAuthMiddleware(RemoteUserMiddleware):
     def process_request(self, request):
         config = request.__dict__.get("archivebox_config")
         if config is None:
-            config = get_config(resolve_plugins=False)
+            config = get_config(base_config=settings.CONFIG, resolve_plugins=False)
             request.archivebox_config = config
         self.header = "HTTP_{normalized}".format(normalized=config.REVERSE_PROXY_USER_HEADER.replace("-", "_").upper())
         if config.REVERSE_PROXY_WHITELIST == "":

archivebox/core/migrations/0004_auto_20200713_1552.py

@@ -9,6 +9,12 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
+        # The original schema allowed NULL timestamps. Use the already-unique
+        # 32-character snapshot ID before enforcing the later NOT NULL field.
+        migrations.RunSQL(
+            sql="UPDATE core_snapshot SET timestamp = id WHERE timestamp IS NULL",
+            reverse_sql=migrations.RunSQL.noop,
+        ),
         migrations.AlterField(
             model_name="snapshot",
             name="timestamp",

archivebox/core/migrations/0007_archiveresult.py

@@ -1,9 +1,11 @@
 # Generated by Django 3.0.8 on 2020-11-04 12:25
 
 import json
+from datetime import timezone
 from pathlib import Path
 
 from django.db import migrations, models
+from django.utils.dateparse import parse_datetime
 import django.db.models.deletion
 
 # Handle old vs new import paths
@@ -30,6 +32,13 @@ except AttributeError:
     JSONField = jsonfield.JSONField
 
 
+def parse_legacy_datetime(value):
+    parsed = parse_datetime(value) if isinstance(value, str) else value
+    if parsed is not None and parsed.tzinfo is None:
+        return parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
 def forwards_func(apps, schema_editor):
     Snapshot = apps.get_model("core", "Snapshot")
     ArchiveResult = apps.get_model("core", "ArchiveResult")
@@ -55,8 +64,8 @@ def forwards_func(apps, schema_editor):
                         pwd=result["pwd"],
                         cmd=result.get("cmd") or [],
                         cmd_version=result.get("cmd_version") or "unknown",
-                        start_ts=result["start_ts"],
-                        end_ts=result["end_ts"],
+                        start_ts=parse_legacy_datetime(result["start_ts"]),
+                        end_ts=parse_legacy_datetime(result["end_ts"]),
                         status=result["status"],
                         output=result.get("output") or "null",
                     )

archivebox/core/migrations/0023_upgrade_to_0_9_0.py

@@ -19,6 +19,19 @@ def get_table_columns(table_name):
     return {row[1] for row in cursor.fetchall()}
 
 
+def assert_rebuild_row_count(cursor, source_table, destination_table, expected_count=None):
+    """Refuse to drop a legacy table unless every source row was copied."""
+    if expected_count is None:
+        cursor.execute(f"SELECT COUNT(*) FROM {source_table}")
+        expected_count = cursor.fetchone()[0]
+    cursor.execute(f"SELECT COUNT(*) FROM {destination_table}")
+    destination_count = cursor.fetchone()[0]
+    if destination_count != expected_count:
+        raise RuntimeError(
+            f"Refusing to replace {source_table}: copied {destination_count} of {expected_count} rows",
+        )
+
+
 def normalize_cmd(cmd):
     if not cmd:
         return "[]"
@@ -99,6 +112,7 @@ def upgrade_core_tables(apps, schema_editor):
             output VARCHAR(1024),
             created_at DATETIME,
             modified_at DATETIME,
+            notes TEXT NOT NULL DEFAULT '',
 
             FOREIGN KEY (snapshot_id) REFERENCES core_snapshot(id) ON DELETE CASCADE
         );
@@ -123,6 +137,8 @@ def upgrade_core_tables(apps, schema_editor):
             archiveresult_select_cols.append("created_at")
         if has_archiveresult_modified_at:
             archiveresult_select_cols.append("modified_at")
+        if "notes" in archiveresult_cols:
+            archiveresult_select_cols.append("notes")
 
         if has_uuid and not has_abid:
             # Migrating from v0.7.2+ (has uuid column)
@@ -132,6 +148,8 @@ def upgrade_core_tables(apps, schema_editor):
                 select_cols.append("created_at")
             if has_archiveresult_modified_at:
                 select_cols.append("modified_at")
+            if "notes" in archiveresult_cols:
+                select_cols.append("notes")
             cursor.execute(f"SELECT {', '.join(select_cols)} FROM core_archiveresult")
             old_records = cursor.fetchall()
             for i, record in enumerate(old_records, start=1):
@@ -147,8 +165,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         id, uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         values["id"],
@@ -164,6 +182,7 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
@@ -186,8 +205,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         new_uuid,
@@ -202,6 +221,7 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
@@ -221,8 +241,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         id, uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         values["id"],
@@ -238,12 +258,14 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
                     print(f"        copied {i}/{len(old_records)} ArchiveResults...")
         print(f"      copied {len(old_records)} ArchiveResults")
 
+    assert_rebuild_row_count(cursor, "core_archiveresult", "core_archiveresult_new", row_count)
     cursor.execute("DROP TABLE IF EXISTS core_archiveresult;")
     cursor.execute("ALTER TABLE core_archiveresult_new RENAME TO core_archiveresult;")
 
@@ -296,13 +318,29 @@ def upgrade_core_tables(apps, schema_editor):
             snapshot_cols = get_table_columns("core_snapshot")
             has_added = "added" in snapshot_cols
             has_bookmarked_at = "bookmarked_at" in snapshot_cols
+            inferred_status_sql = """
+                CASE
+                    WHEN EXISTS (
+                        SELECT 1 FROM core_archiveresult
+                        WHERE core_archiveresult.snapshot_id = core_snapshot.id
+                        AND core_archiveresult.status IN ('queued', 'started', 'backoff')
+                    )
+                    THEN 'queued'
+                    WHEN EXISTS (
+                        SELECT 1 FROM core_archiveresult
+                        WHERE core_archiveresult.snapshot_id = core_snapshot.id
+                    )
+                    THEN 'sealed'
+                    ELSE 'queued'
+                END
+            """
 
             if has_added and not has_bookmarked_at:
                 # Migrating from v0.7.2 (has added/updated fields)
                 print("      copying Snapshots from 0.7.x schema...")
                 # timestamp is the legacy bookmark/import timestamp and archive/{timestamp} identity.
                 # added is the DB row creation/import time, and updated was renamed to downloaded_at in 0.8.x.
-                cursor.execute("""
+                cursor.execute(f"""
                     INSERT OR IGNORE INTO core_snapshot_new (
                         id, url, timestamp, title,
                         bookmarked_at, created_at, modified_at, downloaded_at,
@@ -321,20 +359,7 @@ def upgrade_core_tables(apps, schema_editor):
                         COALESCE(added, CURRENT_TIMESTAMP) as created_at,
                         COALESCE(updated, added, CURRENT_TIMESTAMP) as modified_at,
                         updated as downloaded_at,
-                        CASE
-                            WHEN EXISTS (
-                                SELECT 1 FROM core_archiveresult
-                                WHERE core_archiveresult.snapshot_id = core_snapshot.id
-                                AND core_archiveresult.status IN ('queued', 'started', 'backoff')
-                            )
-                            THEN 'queued'
-                            WHEN EXISTS (
-                                SELECT 1 FROM core_archiveresult
-                                WHERE core_archiveresult.snapshot_id = core_snapshot.id
-                            )
-                            THEN 'sealed'
-                            ELSE 'queued'
-                        END as status
+                        {inferred_status_sql} as status
                     FROM core_snapshot;
                 """)
                 print(f"      copied {cursor.rowcount} Snapshots")
@@ -352,6 +377,9 @@ def upgrade_core_tables(apps, schema_editor):
                 if has_crawl_id:
                     insert_cols.append("crawl_id")
                     select_cols.append("REPLACE(crawl_id, '-', '')")
+                if "parent_snapshot_id" in snapshot_cols:
+                    insert_cols.append("parent_snapshot_id")
+                    select_cols.append("REPLACE(parent_snapshot_id, '-', '')")
                 if has_status:
                     insert_cols.append("status")
                     select_cols.append(
@@ -373,9 +401,24 @@ def upgrade_core_tables(apps, schema_editor):
                         END
                         """,
                     )
+                else:
+                    insert_cols.append("status")
+                    select_cols.append(inferred_status_sql)
                 if has_retry_at:
                     insert_cols.append("retry_at")
                     select_cols.append("retry_at")
+                for field_name in (
+                    "depth",
+                    "fs_version",
+                    "config",
+                    "notes",
+                    "num_uses_succeeded",
+                    "num_uses_failed",
+                    "current_step",
+                ):
+                    if field_name in snapshot_cols:
+                        insert_cols.append(field_name)
+                        select_cols.append(field_name)
 
                 cursor.execute(f"""
                     INSERT OR IGNORE INTO core_snapshot_new ({", ".join(insert_cols)})
@@ -386,6 +429,7 @@ def upgrade_core_tables(apps, schema_editor):
             else:
                 print(f"Warning: Unexpected Snapshot schema - has_added={has_added}, has_bookmarked_at={has_bookmarked_at}")
 
+    assert_rebuild_row_count(cursor, "core_snapshot", "core_snapshot_new")
     cursor.execute("DROP TABLE IF EXISTS core_snapshot;")
     cursor.execute("ALTER TABLE core_snapshot_new RENAME TO core_snapshot;")
 
@@ -424,6 +468,7 @@ def upgrade_core_tables(apps, schema_editor):
         tag_has_data = cursor.fetchone()[0] > 0
 
         if tag_has_data:
+            tag_cols = get_table_columns("core_tag")
             cursor.execute("PRAGMA table_info(core_tag)")
             tag_id_type = None
             for row in cursor.fetchall():
@@ -435,22 +480,23 @@ def upgrade_core_tables(apps, schema_editor):
                 # v0.8.6rc0: Tag IDs are UUIDs, need to convert to INTEGER
                 print("      converting Tag IDs from UUID to integers...")
 
-                # Get all tags with their UUIDs
-                cursor.execute("SELECT id, name, slug, created_at, modified_at, created_by_id FROM core_tag ORDER BY name")
+                # Get all tags with their UUIDs and any available audit fields.
+                tag_select_cols = ["id", "name", "slug"]
+                tag_select_cols.extend(field for field in ("created_at", "modified_at", "created_by_id") if field in tag_cols)
+                cursor.execute(f"SELECT {', '.join(tag_select_cols)} FROM core_tag ORDER BY name")
                 tags = cursor.fetchall()
 
                 # Create mapping from old UUID to new INTEGER ID
                 uuid_to_int_map = {}
                 for i, tag in enumerate(tags, start=1):
-                    old_id, name, slug, created_at, modified_at, created_by_id = tag
+                    values = dict(zip(tag_select_cols, tag))
+                    old_id = values.pop("id")
                     uuid_to_int_map[old_id] = i
                     # Insert with new INTEGER ID
+                    insert_values = {"id": i, **values}
                     cursor.execute(
-                        """
-                        INSERT OR IGNORE INTO core_tag_new (id, name, slug, created_at, modified_at, created_by_id)
-                        VALUES (?, ?, ?, ?, ?, ?)
-                    """,
-                        (i, name, slug, created_at, modified_at, created_by_id),
+                        f"INSERT OR IGNORE INTO core_tag_new ({', '.join(insert_values)}) VALUES ({', '.join('%s' for _ in insert_values)})",
+                        tuple(insert_values.values()),
                     )
                     if i % PROGRESS_EVERY == 0:
                         print(f"        copied {i}/{len(tags)} Tags...")
@@ -471,23 +517,33 @@ def upgrade_core_tables(apps, schema_editor):
                             cursor.execute(
                                 """
                                 INSERT OR IGNORE INTO core_snapshot_tags (id, snapshot_id, tag_id)
-                                VALUES (?, ?, ?)
+                                VALUES (%s, %s, %s)
                             """,
                                 (st_id, snapshot_id, new_tag_id),
                             )
                         if i % PROGRESS_EVERY == 0:
                             print(f"        copied {i}/{len(snapshot_tags)} SnapshotTag rows...")
+                    cursor.execute("SELECT COUNT(*) FROM core_snapshot_tags")
+                    copied_snapshot_tags = cursor.fetchone()[0]
+                    if copied_snapshot_tags != len(snapshot_tags):
+                        raise RuntimeError(
+                            f"Refusing to replace Tag IDs: copied {copied_snapshot_tags} of {len(snapshot_tags)} SnapshotTag rows",
+                        )
                 print(f"      copied {len(tags)} Tags")
             else:
-                # v0.7.2: Tag IDs are already INTEGER
-                print("      copying Tags from 0.7.x schema...")
-                cursor.execute("""
-                    INSERT OR IGNORE INTO core_tag_new (id, name, slug)
-                    SELECT id, name, slug
+                # v0.7.x and early v0.8.x use INTEGER IDs. Preserve audit
+                # metadata when the source schema has it.
+                print("      copying Tags with integer IDs...")
+                copy_cols = ["id", "name", "slug"]
+                copy_cols.extend(field for field in ("created_at", "modified_at", "created_by_id") if field in tag_cols)
+                cursor.execute(f"""
+                    INSERT OR IGNORE INTO core_tag_new ({", ".join(copy_cols)})
+                    SELECT {", ".join(copy_cols)}
                     FROM core_tag;
                 """)
                 print(f"      copied {cursor.rowcount} Tags")
 
+    assert_rebuild_row_count(cursor, "core_tag", "core_tag_new")
     cursor.execute("DROP TABLE IF EXISTS core_tag;")
     cursor.execute("ALTER TABLE core_tag_new RENAME TO core_tag;")
 

archivebox/core/migrations/0024_assign_default_crawl.py

@@ -94,7 +94,7 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
     Create a default crawl for migrated snapshots and assign all snapshots without a crawl to it.
     Uses raw SQL because the app registry isn't fully populated during migrations.
     """
-    # Legacy-data-only, sqlite-specific (PRAGMA + '?' placeholders). On a fresh
+    # Legacy-data-only and sqlite-specific because it uses PRAGMA introspection.
     # postgres install core_snapshot has no crawl_id column yet and there are no
     # unassigned snapshots, so gate before touching any SQL.
     if schema_editor.connection.vendor != "sqlite":
@@ -123,7 +123,7 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
         cursor.execute(
             """
             INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
-            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, ?)
+            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
         """,
             [datetime.now().isoformat()],
         )
@@ -153,15 +153,15 @@ def create_default_crawl_and_assign_snapshots(apps, schema_editor):
             id, created_at, modified_at, num_uses_succeeded, num_uses_failed,
             urls, max_depth, tags_str, label, notes, {default_columns_sql}
             status, retry_at, created_by_id, schedule_id, config, persona_id
-        ) VALUES (?, ?, ?, 0, 0, ?, 0, '', 'Migrated from v0.7.2/v0.8.6',
+        ) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
                   'Auto-created crawl for migrated snapshots', {default_values_sql}
-                  'sealed', ?, 1, NULL, '{{}}', NULL)
+                  'sealed', %s, 1, NULL, '{{}}', NULL)
     """,
         [crawl_id, now, now, crawl_urls, now],
     )
 
     # Assign all snapshots without a crawl to the default crawl
-    cursor.execute("UPDATE core_snapshot SET crawl_id = ? WHERE crawl_id IS NULL", [crawl_id])
+    cursor.execute("UPDATE core_snapshot SET crawl_id = %s WHERE crawl_id IS NULL", [crawl_id])
 
     print(f"✓ Assigned {snapshots_without_crawl} snapshots to default crawl {crawl_id}")
 

archivebox/core/migrations/0025_alter_archiveresult_options_alter_snapshot_options_and_more.py

@@ -55,11 +55,12 @@ def _pg_sync_schema(apps, schema_editor):
     # retry_at/status columns and archiveresult created_at/modified_at were
     # created on sqlite by 0023's raw rebuild, so here they are state-only).
     # On postgres those columns would never land, desyncing the real schema.
-    # Resync to this migration's final state (empty tables on pg). Snapshot is
-    # referenced by SnapshotTag and ArchiveResult, so rebuild those too.
+    # Resync to this migration's final state (empty tables on pg). Tag and
+    # Snapshot both gain state-only fields; Snapshot is referenced by
+    # SnapshotTag and ArchiveResult, so rebuild those too.
     from archivebox.misc.db import rebuild_models_from_migration_state
 
-    rebuild_models_from_migration_state(apps, schema_editor, "core", ["Snapshot", "SnapshotTag", "ArchiveResult"])
+    rebuild_models_from_migration_state(apps, schema_editor, "core", ["Tag", "Snapshot", "SnapshotTag", "ArchiveResult"])
 
 
 class Migration(migrations.Migration):
@@ -80,8 +81,8 @@ class Migration(migrations.Migration):
         ),
         # NOTE: RemoveField for cmd, cmd_version, pwd moved to migration 0027
         # to allow data migration to Process records first
-        # NOTE: created_at/modified_at are created by migration 0023 so they can
-        # preserve old ArchiveResult row metadata before SQLite table rebuilds.
+        # NOTE: created_at/modified_at/notes are created by migration 0023 so
+        # they can preserve old ArchiveResult metadata before SQLite rebuilds.
         # Update Django's state here before the AddField operations below.
         migrations.SeparateDatabaseAndState(
             database_operations=[],
@@ -96,6 +97,11 @@ class Migration(migrations.Migration):
                     name="modified_at",
                     field=models.DateTimeField(auto_now=True),
                 ),
+                migrations.AddField(
+                    model_name="archiveresult",
+                    name="notes",
+                    field=models.TextField(blank=True, default=""),
+                ),
             ],
         ),
         migrations.AddField(
@@ -114,11 +120,6 @@ class Migration(migrations.Migration):
                 max_length=255,
             ),
         ),
-        migrations.AddField(
-            model_name="archiveresult",
-            name="notes",
-            field=models.TextField(blank=True, default=""),
-        ),
         migrations.AddField(
             model_name="archiveresult",
             name="num_uses_failed",
@@ -237,26 +238,33 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
-        migrations.AddField(
-            model_name="tag",
-            name="created_at",
-            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now, null=True),
-        ),
-        migrations.AddField(
-            model_name="tag",
-            name="created_by",
-            field=models.ForeignKey(
-                default=archivebox.base_models.models.get_or_create_system_user_pk,
-                null=True,
-                on_delete=django.db.models.deletion.CASCADE,
-                related_name="tag_set",
-                to=settings.AUTH_USER_MODEL,
-            ),
-        ),
-        migrations.AddField(
-            model_name="tag",
-            name="modified_at",
-            field=models.DateTimeField(auto_now=True),
+        # 0023 creates these columns before rebuilding the legacy Tag table so
+        # their values survive. Declare them in state without rebuilding it.
+        migrations.SeparateDatabaseAndState(
+            database_operations=[],
+            state_operations=[
+                migrations.AddField(
+                    model_name="tag",
+                    name="created_at",
+                    field=models.DateTimeField(db_index=True, default=django.utils.timezone.now, null=True),
+                ),
+                migrations.AddField(
+                    model_name="tag",
+                    name="created_by",
+                    field=models.ForeignKey(
+                        default=archivebox.base_models.models.get_or_create_system_user_pk,
+                        null=True,
+                        on_delete=django.db.models.deletion.CASCADE,
+                        related_name="tag_set",
+                        to=settings.AUTH_USER_MODEL,
+                    ),
+                ),
+                migrations.AddField(
+                    model_name="tag",
+                    name="modified_at",
+                    field=models.DateTimeField(auto_now=True),
+                ),
+            ],
         ),
         # Copy data from old field names to new field names after AddField operations
         migrations.RunPython(

archivebox/core/migrations/0027_copy_archiveresult_to_process.py

@@ -1,7 +1,7 @@
 # Generated by hand on 2026-01-01
 # Copies ArchiveResult cmd/pwd/cmd_version data to Process records before removing old fields
 
-from django.db import migrations, connection
+from django.db import migrations
 import json
 from pathlib import Path
 from archivebox.uuid_compat import uuid7
@@ -51,7 +51,7 @@ def get_or_create_current_machine(cursor):
     guid = f"host_{hostname}"  # Simple but stable identifier
 
     # Check if machine exists
-    cursor.execute("SELECT id FROM machine_machine WHERE guid = ?", [guid])
+    cursor.execute("SELECT id FROM machine_machine WHERE guid = %s", [guid])
     row = cursor.fetchone()
 
     if row:
@@ -76,7 +76,7 @@ def get_or_create_current_machine(cursor):
                 hw_in_docker, hw_in_vm, hw_manufacturer, hw_product, hw_uuid,
                 os_arch, os_family, os_platform, os_release, os_kernel,
                 stats, config, num_uses_failed, num_uses_succeeded
-            ) VALUES (?, ?, ?, ?, ?, 0, 0, '', '', '',
+            ) VALUES (%s, %s, %s, %s, %s, 0, 0, '', '', '',
                       '', '', '', '', '', '{}', '{}', 0, 0)
         """,
             [machine_id, now, now, guid, hostname],
@@ -90,7 +90,7 @@ def get_or_create_current_machine(cursor):
                 hw_in_docker, hw_in_vm, hw_manufacturer, hw_product, hw_uuid,
                 os_arch, os_family, os_platform, os_release, os_kernel,
                 stats, num_uses_failed, num_uses_succeeded
-            ) VALUES (?, ?, ?, ?, ?, 0, 0, '', '', '',
+            ) VALUES (%s, %s, %s, %s, %s, 0, 0, '', '', '',
                       '', '', '', '', '', '{}', 0, 0)
         """,
             [machine_id, now, now, guid, hostname],
@@ -125,7 +125,7 @@ def get_or_create_binary(cursor, machine_id, name, abspath, version):
     cursor.execute(
         """
         SELECT id FROM machine_binary
-        WHERE machine_id = ? AND name = ? AND abspath = ? AND version = ?
+        WHERE machine_id = %s AND name = %s AND abspath = %s AND version = %s
     """,
         [machine_id, name, abspath, version],
     )
@@ -162,7 +162,7 @@ def get_or_create_binary(cursor, machine_id, name, abspath, version):
         "num_uses_succeeded": 0,
     }
     insert_cols = [col for col in values_by_col if col in binary_cols]
-    placeholders = ", ".join(["?"] * len(insert_cols))
+    placeholders = ", ".join(["%s"] * len(insert_cols))
     cursor.execute(
         f"""
         INSERT INTO machine_binary ({", ".join(insert_cols)})
@@ -244,7 +244,7 @@ def create_process(cursor, machine_id, pwd, cmd, status, exit_code, started_at,
         "num_uses_succeeded": 0,
     }
     insert_cols = [col for col in values_by_col if col in process_cols]
-    placeholders = ", ".join(["?"] * len(insert_cols))
+    placeholders = ", ".join(["%s"] * len(insert_cols))
     cursor.execute(
         f"""
         INSERT INTO machine_process ({", ".join(insert_cols)})
@@ -275,7 +275,7 @@ def copy_archiveresult_data_to_process(apps, schema_editor):
     - failed → exited (exit_code=1)
     - skipped → exited (exit_code=None)
     """
-    # sqlite-only legacy data copy (PRAGMA introspection, '?' placeholders).
+    # sqlite-only legacy data copy because it uses PRAGMA introspection.
     # A postgres install can never contain legacy ArchiveResult cmd/pwd data at
     # this point, so there is nothing to copy into machine_process. The
     # RemoveField ops below are real (non-state-only) and drop the now-empty
@@ -283,7 +283,7 @@ def copy_archiveresult_data_to_process(apps, schema_editor):
     if schema_editor.connection.vendor != "sqlite":
         return
 
-    cursor = connection.cursor()
+    cursor = schema_editor.connection.cursor()
 
     # Check if old fields still exist (skip if fresh install or already migrated)
     cursor.execute("PRAGMA table_info(core_archiveresult)")
@@ -295,96 +295,101 @@ def copy_archiveresult_data_to_process(apps, schema_editor):
 
     # Check if process_id field exists (should exist from 0026)
     if "process_id" not in cols:
-        print("✗ ERROR: process_id field not found. Migration 0026 must run first.")
-        return
+        raise RuntimeError("process_id field not found. Migration 0026 must run first.")
 
     # Get or create Machine.current()
     machine_id = get_or_create_current_machine(cursor)
 
-    # Get ArchiveResults without process_id that have cmd data
+    # Get every ArchiveResult without a Process. Rows with empty execution
+    # metadata still need a Process before the legacy columns can be removed.
     # Use plugin (extractor was renamed to plugin in migration 0025)
     cursor.execute("""
         SELECT id, snapshot_id, plugin, cmd, pwd, cmd_version,
                status, start_ts, end_ts, created_at
         FROM core_archiveresult
         WHERE process_id IS NULL
-        AND (cmd IS NOT NULL OR pwd IS NOT NULL)
     """)
 
     results = cursor.fetchall()
 
     if not results:
         print("    ✓ No ArchiveResults need Process migration")
-        return
-
-    print(f"    - Migrating {len(results)} ArchiveResults to Process rows...")
+    else:
+        print(f"    - Migrating {len(results)} ArchiveResults to Process rows...")
 
     migrated_count = 0
-    skipped_count = 0
-    error_count = 0
-
     for i, row in enumerate(results):
         ar_id, snapshot_id, plugin, cmd_raw, pwd, cmd_version, status, start_ts, end_ts, created_at = row
 
-        try:
-            # Parse cmd field
-            cmd_array = parse_cmd_field(cmd_raw)
+        # Parse cmd field
+        cmd_array = parse_cmd_field(cmd_raw)
 
-            # Extract binary info from cmd[0] if available
-            binary_id = None
-            if cmd_array and cmd_array[0]:
-                binary_name = Path(cmd_array[0]).name or plugin  # Fallback to plugin name
-                binary_abspath = cmd_array[0]
-                binary_version = cmd_version or ""
+        # Extract binary info from cmd[0] if available
+        binary_id = None
+        if cmd_array and cmd_array[0]:
+            binary_name = Path(cmd_array[0]).name or plugin  # Fallback to plugin name
+            binary_abspath = cmd_array[0]
+            binary_version = cmd_version or ""
 
-                # Get or create Binary record
-                binary_id = get_or_create_binary(
-                    cursor,
-                    machine_id,
-                    binary_name,
-                    binary_abspath,
-                    binary_version,
-                )
-
-            # Map status
-            process_status, exit_code = map_status(status)
-
-            # Set timestamps
-            started_at = start_ts or created_at
-            ended_at = end_ts if process_status == "exited" else None
-
-            # Create Process record
-            process_id = create_process(
-                cursor=cursor,
-                machine_id=machine_id,
-                pwd=pwd or "",
-                cmd=cmd_array,
-                status=process_status,
-                exit_code=exit_code,
-                started_at=started_at,
-                ended_at=ended_at,
-                binary_id=binary_id,
+            # Get or create Binary record
+            binary_id = get_or_create_binary(
+                cursor,
+                machine_id,
+                binary_name,
+                binary_abspath,
+                binary_version,
             )
+        elif cmd_version:
+            raise RuntimeError(f"ArchiveResult {ar_id} has cmd_version metadata but no command")
 
-            # Link ArchiveResult to Process
-            cursor.execute(
-                "UPDATE core_archiveresult SET process_id = ? WHERE id = ?",
-                [process_id, ar_id],
-            )
+        # Map status
+        process_status, exit_code = map_status(status)
 
-            migrated_count += 1
-            if migrated_count % PROGRESS_EVERY == 0:
-                print(f"      migrated {migrated_count}/{len(results)} ArchiveResults...")
+        # Set timestamps
+        started_at = start_ts or created_at
+        ended_at = end_ts if process_status == "exited" else None
 
-        except Exception as e:
-            print(f"✗ Error migrating ArchiveResult {ar_id}: {e}")
-            import traceback
+        # Create Process record
+        process_id = create_process(
+            cursor=cursor,
+            machine_id=machine_id,
+            pwd=pwd or "",
+            cmd=cmd_array,
+            status=process_status,
+            exit_code=exit_code,
+            started_at=started_at,
+            ended_at=ended_at,
+            binary_id=binary_id,
+        )
 
-            traceback.print_exc()
-            error_count += 1
-            continue
+        # Link ArchiveResult to Process
+        cursor.execute(
+            "UPDATE core_archiveresult SET process_id = %s WHERE id = %s",
+            [process_id, ar_id],
+        )
+        if cursor.rowcount != 1:
+            raise RuntimeError(f"ArchiveResult {ar_id} was not linked to Process {process_id}")
 
-    print(f"    ✓ Process migration complete: {migrated_count} migrated, {skipped_count} skipped, {error_count} errors")
+        migrated_count += 1
+        if migrated_count % PROGRESS_EVERY == 0:
+            print(f"      migrated {migrated_count}/{len(results)} ArchiveResults...")
+
+    cursor.execute("""
+        SELECT ar.id, ar.cmd, ar.pwd, ar.cmd_version, ar.process_id,
+               process.cmd, process.pwd, binary.version
+        FROM core_archiveresult AS ar
+        LEFT JOIN machine_process AS process ON process.id = ar.process_id
+        LEFT JOIN machine_binary AS binary ON binary.id = process.binary_id
+    """)
+    for ar_id, cmd_raw, pwd, cmd_version, process_id, process_cmd_raw, process_pwd, binary_version in cursor.fetchall():
+        if not process_id:
+            raise RuntimeError(f"ArchiveResult {ar_id} is not linked to a Process")
+        if parse_cmd_field(process_cmd_raw) != parse_cmd_field(cmd_raw) or process_pwd != (pwd or ""):
+            raise RuntimeError(f"ArchiveResult {ar_id} execution metadata was not preserved")
+        if cmd_version and binary_version != cmd_version:
+            raise RuntimeError(f"ArchiveResult {ar_id} command version was not preserved")
+
+    print(f"    ✓ Process migration complete: {migrated_count} migrated")
 
 
 class Migration(migrations.Migration):

archivebox/core/migrations/0029_migrate_archiveresult_to_uuid_pk.py

@@ -1,7 +1,7 @@
 # Generated by hand on 2026-01-02
 # Migrate ArchiveResult from integer PK to UUID PK (matching Snapshot)
 
-from django.db import migrations, models, connection
+from django.db import migrations, models
 from uuid import UUID
 from archivebox.uuid_compat import uuid7
 
@@ -32,7 +32,7 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
     if schema_editor.connection.vendor != "sqlite":
         return
 
-    cursor = connection.cursor()
+    cursor = schema_editor.connection.cursor()
 
     # Check if table exists and has data
     cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='core_archiveresult'")
@@ -101,23 +101,35 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
     col_names = [col[1] for col in columns]
     has_uuid_column = "uuid" in col_names
 
+    used_uuids = set()
+
+    def fresh_uuid():
+        while True:
+            candidate = uuid7().hex
+            if candidate not in used_uuids:
+                used_uuids.add(candidate)
+                return candidate
+
+    def unique_uuid(existing_uuid):
+        try:
+            candidate = UUID(str(existing_uuid)).hex if existing_uuid else None
+        except (TypeError, ValueError, AttributeError):
+            candidate = None
+
+        if candidate and candidate not in used_uuids:
+            used_uuids.add(candidate)
+            return candidate
+        return fresh_uuid()
+
     if has_uuid_column:
         cursor.execute("SELECT id, uuid FROM core_archiveresult")
         records = cursor.fetchall()
-        id_to_uuid = {}
-        for old_id, existing_uuid in records:
-            if existing_uuid:
-                # Normalize existing UUID to 32-char hex format (Django SQLite UUIDField format)
-                # (existing UUIDs might be stored with or without dashes in old schema)
-                id_to_uuid[old_id] = UUID(existing_uuid).hex
-            else:
-                # Generate new UUIDv7 (time-ordered) as 32-char hex
-                id_to_uuid[old_id] = uuid7().hex
+        id_to_uuid = {old_id: unique_uuid(existing_uuid) for old_id, existing_uuid in records}
     else:
         # 0.7.x path: no uuid column, generate new UUIDs for all records
         cursor.execute("SELECT id FROM core_archiveresult")
         records = cursor.fetchall()
-        id_to_uuid = {old_id: uuid7().hex for (old_id,) in records}
+        id_to_uuid = {old_id: fresh_uuid() for (old_id,) in records}
 
     # Step 3: Copy data with UUIDs as new primary key
     cursor.execute("SELECT * FROM core_archiveresult")
@@ -125,7 +137,7 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
 
     # col_names already fetched in Step 2
     inserted_count = 0
-    for i, record in enumerate(old_records):
+    for record in old_records:
         old_id = record[col_names.index("id")]
         new_uuid = id_to_uuid[old_id]
 
@@ -158,29 +170,34 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
         # Build INSERT statement (only copy fields that exist in source)
         existing_fields = [f for f in fields_to_copy if f in values]
 
-        placeholders = ", ".join(["?"] * (len(existing_fields) + 1))  # +1 for id
+        placeholders = ", ".join(["%s"] * (len(existing_fields) + 1))  # +1 for id
         field_list = "id, " + ", ".join(existing_fields)
 
         insert_values = [new_uuid] + [values.get(f) for f in existing_fields]
 
-        try:
-            cursor.execute(
-                f"INSERT INTO core_archiveresult_new ({field_list}) VALUES ({placeholders})",
-                insert_values,
-            )
-            inserted_count += 1
-            if inserted_count % PROGRESS_EVERY == 0:
-                print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults...")
-        except Exception as e:
-            print(f"[0029] ERROR inserting record {old_id}: {e}")
-            if i == 0:
-                print(f"[0029] First record values: {insert_values[:5]}...")
-                raise
+        cursor.execute(
+            f"INSERT INTO core_archiveresult_new ({field_list}) VALUES ({placeholders})",
+            insert_values,
+        )
+        inserted_count += 1
+        if inserted_count % PROGRESS_EVERY == 0:
+            print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults...")
 
     if old_records:
         print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults")
 
-    # Step 4: Replace old table with new table
+    cursor.execute("SELECT COUNT(*) FROM core_archiveresult")
+    source_count = cursor.fetchone()[0]
+    cursor.execute("SELECT COUNT(*) FROM core_archiveresult_new")
+    destination_count = cursor.fetchone()[0]
+    if source_count != row_count or destination_count != row_count or inserted_count != row_count:
+        raise RuntimeError(
+            "ArchiveResult UUID migration row-count mismatch: "
+            f"expected={row_count}, source={source_count}, destination={destination_count}, inserted={inserted_count}",
+        )
+
+    # Step 4: Replace old table with new table. Django runs this migration in
+    # one transaction, so any failure above leaves the source table untouched.
     cursor.execute("DROP TABLE core_archiveresult")
     cursor.execute("ALTER TABLE core_archiveresult_new RENAME TO core_archiveresult")
 

archivebox/core/migrations/0045_archiveresult_unique_hook.py

@@ -2,13 +2,13 @@ from django.db import migrations, models
 
 
 def deduplicate_archiveresults_per_hook(apps, schema_editor):
-    """Drop duplicate ArchiveResult rows per (snapshot, plugin, hook_name).
+    """Give historical duplicate rows unique hook identities.
 
     Real long-lived collections (cabbage's demo, beta-tester DBs) accumulated
-    multiple rows per hook over the dev rc chain. The next operation adds a
-    UniqueConstraint on that tuple; without this cleanup pass the constraint
-    fails with ``UNIQUE constraint failed`` mid-migration and bricks startup.
-    Keep the row with the highest id (most recent) for each tuple.
+    multiple rows per hook. Preserve every row while keeping the newest row on
+    the canonical hook identity expected by the current scheduler. Older rows
+    receive deterministic historical identities before the existing unique
+    constraint is added.
     """
     ArchiveResult = apps.get_model("core", "ArchiveResult")
     duplicate_groups = (
@@ -20,9 +20,10 @@ def deduplicate_archiveresults_per_hook(apps, schema_editor):
             "plugin": group["plugin"],
             "hook_name": group["hook_name"],
         }
-        keep = ArchiveResult.objects.filter(**lookup).order_by("-id").values_list("id", flat=True).first()
-        if keep is not None:
-            ArchiveResult.objects.filter(**lookup).exclude(id=keep).delete()
+        rows = list(ArchiveResult.objects.filter(**lookup).order_by("-id").values_list("id", flat=True))
+        for row_id in rows[1:]:
+            historical_name = f"__legacy_history__{row_id}"[:255]
+            ArchiveResult.objects.filter(id=row_id).update(hook_name=historical_name)
 
 
 class Migration(migrations.Migration):

archivebox/core/migrations/0053_alter_archiveresult_options.py

@@ -0,0 +1,16 @@
+# Generated by Django 6.1 on 2026-08-30 18:20
+
+from django.db import migrations
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0051_postgres_url_pattern_ops_index"),
+    ]
+
+    operations = [
+        migrations.AlterModelOptions(
+            name="archiveresult",
+            options={"verbose_name": "Archive Result", "verbose_name_plural": "Archive Results"},
+        ),
+    ]

archivebox/core/migrations/0054_snapshot_fs_version_index.py

@@ -0,0 +1,26 @@
+# Generated by Django 6.1 on 2026-09-02
+
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0053_alter_archiveresult_options"),
+    ]
+
+    operations = [
+        migrations.RemoveField(
+            model_name="snapshot",
+            name="current_step",
+        ),
+        migrations.AlterField(
+            model_name="snapshot",
+            name="fs_version",
+            field=models.CharField(
+                db_index=True,
+                default="0.9.0",
+                help_text='Filesystem version of this snapshot (e.g., "0.7.0", "0.8.0", "0.9.4").',
+                max_length=10,
+            ),
+        ),
+    ]

archivebox/core/models/__init__.py

@@ -0,0 +1,17 @@
+"""Core persistence models, grouped by entity and queryset responsibility."""
+
+from .archiveresults import ArchiveResult
+from .querysets import SnapshotManager, SnapshotQuerySet, UngroupedSubquery
+from .snapshots import Snapshot, SnapshotMigrationError
+from .tags import SnapshotTag, Tag
+
+__all__ = [
+    "ArchiveResult",
+    "Snapshot",
+    "SnapshotManager",
+    "SnapshotMigrationError",
+    "SnapshotQuerySet",
+    "SnapshotTag",
+    "Tag",
+    "UngroupedSubquery",
+]

archivebox/core/models/querysets.py

@@ -0,0 +1,361 @@
+from __future__ import annotations
+
+from datetime import UTC, datetime
+from typing import Any, ClassVar
+
+from django.core.exceptions import FieldDoesNotExist
+from django.db import models
+
+from archivebox.config import CONSTANTS
+from archivebox.config.common import get_config
+from archivebox.crawls.models import Crawl
+from archivebox.misc.system import atomic_write
+from archivebox.misc.util import (
+    to_json,
+)
+
+
+class UngroupedSubquery(models.Subquery):
+    """Scalar subquery that should not be copied into the outer GROUP BY."""
+
+    def get_group_by_cols(self):
+        return []
+
+
+class SnapshotQuerySet(models.QuerySet):
+    """Custom QuerySet for Snapshot model with export methods that persist through .filter() etc."""
+
+    def bulk_create(self, objs, *args, **kwargs):
+        objs = list(objs)
+        missing_crawl_ids = set()
+        from archivebox.core.permissions import PERMISSIONS_VALUES
+
+        for obj in objs:
+            if isinstance(obj, self.model):
+                config = dict(obj.config or {})
+                permission = str(config.get("PERMISSIONS") or "").strip().lower()
+                if permission not in PERMISSIONS_VALUES and obj.crawl_id:
+                    crawl = getattr(obj, "crawl", None)
+                    if not getattr(crawl, "permissions", None):
+                        missing_crawl_ids.add(str(obj.crawl_id))
+
+        crawl_permissions_by_id = {}
+        if missing_crawl_ids:
+            crawl_permissions_by_id = {
+                str(crawl_id): permissions
+                for crawl_id, permissions in Crawl.objects.filter(pk__in=missing_crawl_ids).values_list("pk", "permissions")
+            }
+
+        from archivebox.misc.db import truncate_overlong_charfields
+
+        for obj in objs:
+            if isinstance(obj, self.model):
+                obj.ensure_permissions_config(crawl_permissions=crawl_permissions_by_id.get(str(obj.crawl_id)))
+                # bulk_create bypasses pre_save, so clamp CharFields here too
+                # (e.g. page titles) to stay within postgres VARCHAR limits.
+                truncate_overlong_charfields(obj)
+        return super().bulk_create(objs, *args, **kwargs)
+
+    def paged_iterator(self, chunk_size: int = 500):
+        """
+        Iterate snapshots using bounded keyset pages instead of one streaming cursor.
+
+        Django's iterator(chunk_size=...) still keeps a single SQLite SELECT
+        cursor open until the full queryset is exhausted. That is fine for
+        read-only exports, but update/migration code does filesystem work and
+        writes while iterating; a long-lived read cursor there can stretch lock
+        waits across thousands of rows. This respects the queryset's existing
+        filters, order_by(), select_related(), and prefetch_related() state; if
+        no ordering is defined, it falls back to primary-key order.
+        """
+        pk_field = self.model._meta.pk.name
+        raw_ordering = tuple(self.query.order_by or self.model._meta.ordering or (pk_field,))
+
+        ordering = []
+        for term in raw_ordering:
+            if not isinstance(term, str) or term == "?":
+                ordering = []
+                break
+            field_name = term.removeprefix("-")
+            field_name = pk_field if field_name == "pk" else field_name
+            ordering.append(f"-{field_name}" if term.startswith("-") else field_name)
+
+        ordered_field_names = [term.removeprefix("-") for term in ordering]
+        unique_field_names = {pk_field, *(field.name for field in self.model._meta.fields if field.unique)}
+        try:
+            use_keyset = any(name in unique_field_names for name in ordered_field_names) and not any(
+                self.model._meta.get_field(name).null for name in ordered_field_names
+            )
+        except (AttributeError, FieldDoesNotExist):
+            use_keyset = False
+        # Expressions, nullable/related fields and non-unique orderings retain
+        # Django's ordering through bounded offset pages.
+        if not use_keyset:
+            offset = 0
+            while batch := list(self[offset : offset + chunk_size]):
+                yield from batch
+                offset += chunk_size
+            return
+
+        last_values = None
+        value_field_names = tuple(dict.fromkeys([*ordered_field_names, pk_field]))
+        while True:
+            batch_qs = self.order_by(*ordering)
+            if last_values is not None:
+                page_filter = models.Q()
+                for idx, term in enumerate(ordering):
+                    descending = term.startswith("-")
+                    field_name = term[1:] if descending else term
+                    prefix = {ordered_field_names[i]: last_values[i] for i in range(idx)}
+                    comparison = "lt" if descending else "gt"
+                    page_filter |= models.Q(**prefix, **{f"{field_name}__{comparison}": last_values[idx]})
+                batch_qs = batch_qs.filter(page_filter)
+
+            batch_rows = list(batch_qs.values_list(*value_field_names)[:chunk_size])
+            if not batch_rows:
+                break
+
+            pk_idx = value_field_names.index(pk_field)
+            snapshot_ids = [row[pk_idx] for row in batch_rows]
+            snapshots_by_id = {snapshot.pk: snapshot for snapshot in self.filter(pk__in=snapshot_ids).order_by()}
+
+            for row in batch_rows:
+                snapshot_id = row[pk_idx]
+                snapshot = snapshots_by_id.get(snapshot_id)
+                if snapshot is not None:
+                    yield snapshot
+
+            last_values = batch_rows[-1][: len(ordered_field_names)]
+
+    # =========================================================================
+    # Filtering Methods
+    # =========================================================================
+
+    FILTER_TYPES: ClassVar[dict[str, Any]] = {
+        "exact": lambda pattern: models.Q(url=pattern),
+        "substring": lambda pattern: models.Q(url__icontains=pattern),
+        "regex": lambda pattern: models.Q(url__iregex=pattern),
+        "domain": lambda pattern: (
+            models.Q(url__istartswith=f"http://{pattern}")
+            | models.Q(url__istartswith=f"https://{pattern}")
+            | models.Q(url__istartswith=f"ftp://{pattern}")
+        ),
+        "tag": lambda pattern: models.Q(tags__name=pattern),
+        "timestamp": lambda pattern: models.Q(timestamp=pattern),
+    }
+    FILTER_TYPE_CHOICES = tuple(FILTER_TYPES)
+    FILTER_ARG_KEYS = (
+        "after",
+        "before",
+        "filter_type",
+        "filter_patterns",
+        "status",
+        "url__icontains",
+        "url__istartswith",
+        "tag",
+        "crawl_id",
+        "limit",
+        "sort",
+        "search",
+    )
+    SPECIAL_FILTER_ARG_KEYS = frozenset({"filter_patterns", "filter_type", "query", "search", "tag", "before", "after", "limit", "sort"})
+
+    def filter_by_patterns(self, patterns: list[str], filter_type: str = "exact") -> SnapshotQuerySet:
+        """Filter snapshots by URL patterns using specified filter type"""
+        from archivebox.misc.logging import stderr
+
+        q_filter = models.Q()
+        for pattern in patterns:
+            try:
+                q_filter = q_filter | self.FILTER_TYPES[filter_type](pattern)
+            except KeyError:
+                stderr()
+                stderr(f"[X] Got invalid pattern for --filter-type={filter_type}:", color="red")
+                stderr(f"    {pattern}")
+                raise SystemExit(2)
+        return self.filter(q_filter)
+
+    def search(self, **kwargs) -> SnapshotQuerySet:
+
+        from archivebox.core.snapshot_status import filter_snapshots_by_status
+        from archivebox.search.query import apply_snapshot_search
+
+        queryset = self
+        filter_patterns = tuple(str(pattern) for pattern in kwargs.get("filter_patterns") or ())
+        filter_type = kwargs.get("filter_type") or "substring"
+        query = kwargs.get("query")
+        if isinstance(query, (list, tuple)):
+            query = " ".join(str(part) for part in query)
+        query = (query or (" ".join(filter_patterns) if kwargs.get("search") else "")).strip()
+
+        field_names = {field.name for field in self.model._meta.get_fields()}
+        field_names.update(field.attname for field in self.model._meta.fields)
+        field_filters = {
+            key: value
+            for key, value in kwargs.items()
+            if value is not None and key not in self.SPECIAL_FILTER_ARG_KEYS and key.split("__", 1)[0] in field_names
+        }
+        status = field_filters.pop("status", None)
+        queryset = filter_snapshots_by_status(queryset, status)
+        if field_filters:
+            queryset = queryset.filter(**field_filters)
+        if kwargs.get("tag"):
+            queryset = queryset.filter(tags__name__iexact=kwargs["tag"])
+        if kwargs.get("before") is not None:
+            queryset = queryset.filter(bookmarked_at__lt=datetime.fromtimestamp(float(kwargs["before"]), tz=UTC))
+        if kwargs.get("after") is not None:
+            queryset = queryset.filter(bookmarked_at__gt=datetime.fromtimestamp(float(kwargs["after"]), tz=UTC))
+
+        if query:
+            queryset = apply_snapshot_search(
+                queryset,
+                query,
+                search_mode=kwargs.get("search"),
+                ordering=("-created_at",) if not kwargs.get("sort") else None,
+                max_results=kwargs.get("limit"),
+                skip_backend_when_metadata_satisfies_limit=True,
+                include_metadata_for_forced_backend=True,
+            )
+        elif filter_patterns:
+            queryset = queryset.filter_by_patterns(list(filter_patterns), filter_type)
+
+        if kwargs.get("sort"):
+            queryset = queryset.order_by(kwargs["sort"])
+        elif not queryset.query.order_by:
+            queryset = queryset.order_by("-created_at")
+
+        limit = kwargs.get("limit")
+        if limit is not None and limit > 0:
+            queryset = queryset[:limit]
+
+        return queryset
+
+    # =========================================================================
+    # Export Methods
+    # =========================================================================
+
+    def to_json(self, with_headers: bool = False) -> str:
+        """Generate JSON index from snapshots"""
+        import sys
+        from datetime import datetime
+
+        from archivebox.config import VERSION
+
+        config = get_config()
+
+        MAIN_INDEX_HEADER = (
+            {
+                "info": "This is an index of site data archived by ArchiveBox: The self-hosted web archive.",
+                "schema": "archivebox.index.json",
+                "copyright_info": config.FOOTER_INFO,
+                "meta": {
+                    "project": "ArchiveBox",
+                    "version": VERSION,
+                    "git_sha": VERSION,
+                    "website": "https://ArchiveBox.io",
+                    "docs": "https://github.com/ArchiveBox/ArchiveBox/wiki",
+                    "source": "https://github.com/ArchiveBox/ArchiveBox",
+                    "issues": "https://github.com/ArchiveBox/ArchiveBox/issues",
+                    "dependencies": {},
+                },
+            }
+            if with_headers
+            else {}
+        )
+
+        snapshot_dicts = [s.to_dict(extended=True, static_export=True) for s in self.iterator(chunk_size=500)]
+
+        if with_headers:
+            output = {
+                **MAIN_INDEX_HEADER,
+                "num_links": len(snapshot_dicts),
+                "updated": datetime.now(UTC),
+                "last_run_cmd": sys.argv,
+                "links": snapshot_dicts,
+            }
+        else:
+            output = snapshot_dicts
+        return to_json(output, indent=4, sort_keys=True)
+
+    def to_csv(self, cols: list[str] | None = None, header: bool = True, separator: str = ",", ljust: int = 0) -> str:
+        """Generate CSV output from snapshots"""
+        cols = cols or ["timestamp", "is_archived", "url"]
+        header_str = separator.join(col.ljust(ljust) for col in cols) if header else ""
+        row_strs = (s.to_csv(cols=cols, ljust=ljust, separator=separator) for s in self.iterator(chunk_size=500))
+        return "\n".join((header_str, *row_strs))
+
+    def to_html(self, with_headers: bool = True) -> str:
+        """Generate main index HTML from snapshots"""
+        from datetime import datetime
+
+        from django.template.loader import render_to_string
+
+        from archivebox.config import VERSION
+        from archivebox.config.version import get_COMMIT_HASH
+
+        config = get_config()
+
+        template = "static_index.html" if with_headers else "minimal_index.html"
+        snapshot_list = list(self.iterator(chunk_size=500))
+        manifest_records = []
+        for snapshot in snapshot_list:
+            outputs = snapshot.discover_outputs(include_filesystem_fallback=True)
+            output_paths = [str(output.get("path") or "") for output in outputs]
+            snapshot._public_preview_paths = [
+                path for preferred in ("screenshot/screenshot.png", "screenshot.png") for path in output_paths if path == preferred
+            ]
+            snapshot._public_favicon_paths = [path for path in output_paths if path in ("favicon/favicon.ico", "favicon.ico")]
+            snapshot.write_html_details()
+            if with_headers:
+                # Use the same portable schema as the JSON export. Rendering
+                # above has already populated result-count caches, archive_size
+                # reuses the sealed output_size field, and tags are prefetched.
+                manifest_records.append(snapshot.to_dict(extended=True, static_export=True))
+
+        if with_headers:
+            manifest = "".join(f"{to_json(record, indent=None, sort_keys=True)}\n" for record in manifest_records)
+            atomic_write(str(CONSTANTS.DATA_DIR / CONSTANTS.JSONL_INDEX_FILENAME), manifest)
+
+        return render_to_string(
+            template,
+            {
+                "version": VERSION,
+                "git_sha": get_COMMIT_HASH() or VERSION,
+                "num_links": str(len(snapshot_list)),
+                "date_updated": datetime.now(UTC).strftime("%Y-%m-%d"),
+                "time_updated": datetime.now(UTC).strftime("%Y-%m-%d %H:%M"),
+                "links": snapshot_list,
+                "FOOTER_INFO": config.FOOTER_INFO,
+                "STATIC_EXPORT": True,
+                "STATIC_EXPORT_DIR": CONSTANTS.DATA_DIR,
+            },
+        )
+
+
+class SnapshotManager(models.Manager.from_queryset(SnapshotQuerySet)):  # ty: ignore[unsupported-base]
+    """Manager for Snapshot model - uses SnapshotQuerySet for chainable methods"""
+
+    def filter(self, *args, **kwargs):
+        domain = kwargs.pop("domain", None)
+        qs = super().filter(*args, **kwargs)
+        if domain:
+            qs = qs.filter(url__icontains=f"://{domain}")
+        return qs
+
+    def get_queryset(self):
+        # Don't prefetch by default - it causes "too many open files" during bulk operations
+        # Views/templates can add .prefetch_related('tags', 'archiveresult_set') where needed
+        return super().get_queryset()
+
+    # =========================================================================
+    # Import Methods
+    # =========================================================================
+
+    def remove(self, atomic: bool = False) -> tuple:
+        """Remove snapshots from the database"""
+        from django.db import transaction
+
+        if atomic:
+            with transaction.atomic():
+                return self.get_queryset().delete()
+        return self.get_queryset().delete()

archivebox/core/models/tags.py

@@ -0,0 +1,150 @@
+from __future__ import annotations
+
+from collections.abc import Mapping
+from typing import TYPE_CHECKING, Any, ClassVar
+
+from django.conf import settings
+from django.db import IntegrityError, models
+from django.urls import reverse_lazy
+from django.utils import timezone
+from django.utils.text import slugify
+
+from archivebox.base_models.models import (
+    ModelWithUUID,
+    get_or_create_system_user_pk,
+)
+from archivebox.misc.util import (
+    sanitize_html_text,
+)
+
+if TYPE_CHECKING:
+    from .snapshots import Snapshot
+
+
+class Tag(ModelWithUUID):
+    # Keep AutoField for compatibility with main branch migrations
+    # Don't use UUIDField here - requires complex FK transformation
+    id = models.AutoField(primary_key=True, serialize=False, verbose_name="ID")
+    created_by = models.ForeignKey(
+        settings.AUTH_USER_MODEL,
+        on_delete=models.CASCADE,
+        default=get_or_create_system_user_pk,
+        null=True,
+        related_name="tag_set",
+    )
+    created_at = models.DateTimeField(default=timezone.now, db_index=True, null=True)
+    modified_at = models.DateTimeField(auto_now=True)
+    name = models.CharField(unique=True, blank=False, max_length=100)
+
+    @classmethod
+    def get_or_create_by_name(
+        cls,
+        name: str,
+        *,
+        defaults: Mapping[str, Any] | None = None,
+        created_by=None,
+    ) -> tuple[Tag, bool]:
+        name = cls.normalize_name(name)
+        defaults = dict(defaults or {})
+        if created_by is not None:
+            defaults["created_by"] = created_by
+        tag = cls.objects.filter(name__iexact=name).first()
+        if tag:
+            return tag, False
+        try:
+            return cls.objects.create(name=name, **(defaults or {})), True
+        except IntegrityError:
+            tag = cls.objects.filter(name__iexact=name).first()
+            if tag is None:
+                raise
+            return tag, False
+
+    @staticmethod
+    def normalize_name(name: str) -> str:
+        name = sanitize_html_text(name).strip()
+        if not name:
+            raise ValueError("Tag name is required")
+        return name
+
+    def rename(self, name: str) -> Tag:
+        name = self.normalize_name(name)
+        existing = type(self).objects.filter(name__iexact=name).exclude(pk=self.pk).first()
+        if existing:
+            raise ValueError(f'Tag "{existing.name}" already exists')
+        if self.name != name:
+            self.name = name
+            self.save()
+        return self
+
+    snapshot_set: models.Manager[Snapshot]
+
+    class Meta(ModelWithUUID.Meta):
+        app_label = "core"
+        verbose_name = "Tag"
+        verbose_name_plural = "Tags"
+
+    def __str__(self):
+        return self.name
+
+    def save(self, *args, **kwargs):
+        update_fields = kwargs.get("update_fields")
+        if update_fields is None or "name" in update_fields:
+            self.name = sanitize_html_text(self.name).strip()
+        super().save(*args, **kwargs)
+
+    @property
+    def slug(self) -> str:
+        """ASCII-safe slugified form of the tag name (derived, not stored)."""
+        return slugify(self.name or "") or "tag"
+
+    @property
+    def api_url(self) -> str:
+        return str(reverse_lazy("api-1:get_tag", args=[self.id]))
+
+    def to_json(self) -> dict:
+        """
+        Convert Tag model instance to a JSON-serializable dict.
+        """
+        from archivebox.config import VERSION
+
+        return {
+            "type": "Tag",
+            "schema_version": VERSION,
+            "id": str(self.id),
+            "name": self.name,
+        }
+
+    @staticmethod
+    def from_json(record: dict[str, Any], overrides: dict[str, Any] | None = None):
+        """
+        Create/update Tag from JSON dict.
+
+        Args:
+            record: JSON dict with 'name' field
+            overrides: Optional dict with 'snapshot' to auto-attach tag
+
+        Returns:
+            Tag instance or None
+        """
+        name = record.get("name")
+        if not name:
+            return None
+
+        tag, _ = Tag.get_or_create_by_name(name)
+
+        # Auto-attach to snapshot if in overrides
+        if overrides and "snapshot" in overrides and tag:
+            overrides["snapshot"].add_tag_ids([tag.pk])
+
+        return tag
+
+
+class SnapshotTag(models.Model):
+    id = models.AutoField(primary_key=True)
+    snapshot = models.ForeignKey("Snapshot", db_column="snapshot_id", on_delete=models.CASCADE, to_field="id")
+    tag = models.ForeignKey(Tag, db_column="tag_id", on_delete=models.CASCADE, to_field="id")
+
+    class Meta:
+        app_label = "core"
+        db_table = "core_snapshot_tags"
+        unique_together: ClassVar[list[tuple[str, str]]] = [("snapshot", "tag")]

archivebox/core/recovery_util.py

@@ -10,10 +10,15 @@ def _is_signal_interrupted_exit(exit_code: int | None) -> bool:
     return exit_code is not None and (exit_code < 0 or exit_code >= 128)
 
 
+def _canonical_hook_name(hook_name: str) -> str:
+    hook_name = Path(hook_name).name
+    return Path(hook_name).stem if Path(hook_name).suffix in {".py", ".js", ".sh"} else hook_name
+
+
 def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str | None = None) -> dict[str, int]:
     from archivebox.crawls.models import Crawl
     from archivebox.core.models import ArchiveResult, Snapshot
-    from archivebox.services.archive_result_service import _collect_output_metadata
+    from abx_dl.output_files import OutputManifest
     from archivebox.machine.models import Process
     from django.core.exceptions import ValidationError
     from django.db.models import Exists, OuterRef, Q, Subquery, Value
@@ -30,9 +35,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "chrome_processes_orphaned": Process.cleanup_orphaned_chrome() if include_chrome and not crawl_id else 0,
         "crawls_queued_without_retry_at": 0,
         "snapshots_queued_without_retry_at": 0,
-        "archiveresults_backoff": 0,
-        "snapshots_queued_plugin_rows_waiting_on_stale_lease": 0,
-        "archiveresults_started_without_running_process": 0,
+        "snapshots_sealed_with_extension_uploads_only": 0,
+        "archiveresults_interrupted_without_running_process": 0,
         "archiveresults_missing_for_orphaned_hook_processes": 0,
         "snapshots_started_without_running_results": 0,
         "crawls_started_with_due_snapshots": 0,
@@ -40,10 +44,10 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         "crawls_started_without_active_snapshots": 0,
     }
 
-    running_archiveresults = ArchiveResult.objects.filter(
-        snapshot_id=OuterRef("pk"),
-        status=ArchiveResult.StatusChoices.STARTED,
-        process__status=Process.StatusChoices.RUNNING,
+    running_hook_processes = Process.objects.filter(
+        archiveresult__snapshot_id=OuterRef("pk"),
+        process_type=Process.TypeChoices.HOOK,
+        status=Process.StatusChoices.RUNNING,
     )
     active_child_snapshots = Snapshot.objects.filter(
         crawl_id=OuterRef("pk"),
@@ -69,49 +73,52 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
         crawl__status__in=Crawl.RUNNABLE_STATES,
         **snapshot_filter,
     ).update(retry_at=now, modified_at=now)
-    backoff_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.BACKOFF, **result_filter)
+
+    extension_upload_results = ArchiveResult.objects.filter(
+        snapshot_id=OuterRef("pk"),
+        hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
+    )
+    server_results = ArchiveResult.objects.filter(snapshot_id=OuterRef("pk")).exclude(
+        hook_name=Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME,
+    )
+    extension_only_snapshots = (
+        Snapshot.objects.filter(
+            status=Snapshot.StatusChoices.SEALED,
+            crawl__status__in=[Crawl.StatusChoices.QUEUED, Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED],
+            **snapshot_filter,
+        )
+        .annotate(
+            has_extension_upload=Exists(extension_upload_results),
+            has_server_result=Exists(server_results),
+        )
+        .filter(has_extension_upload=True, has_server_result=False)
+    )
+    # Older browser-extension uploads could win a race with runner startup:
+    # their successful external rows made the fresh Snapshot look finished
+    # before its configured server-hook workset was materialized. Reopen only
+    # that exact, recognizable state so the normal runner adds the missing
+    # server rows to the same Snapshot and preserves the uploaded outputs.
+    Crawl.objects.filter(
+        id__in=Subquery(extension_only_snapshots.values("crawl_id")),
+        status=Crawl.StatusChoices.SEALED,
+    ).update(status=Crawl.StatusChoices.STARTED, retry_at=now, modified_at=now)
+    cleaned["snapshots_sealed_with_extension_uploads_only"] = extension_only_snapshots.update(
+        status=Snapshot.StatusChoices.QUEUED,
+        retry_at=now,
+        modified_at=now,
+    )
     orphaned_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.STARTED, **result_filter).exclude(
         process__status=Process.StatusChoices.RUNNING,
     )
-    # ArchiveResult has no retry_at scheduler. Wake only the parent Snapshots
-    # for result rows we are about to repair, then requeue those exact rows.
-    # Using subqueries keeps million-row recovery in SQLite instead of building
-    # Python ID lists or scanning all sealed snapshots.
-    Snapshot.objects.filter(
-        id__in=backoff_results.values("snapshot_id"),
-        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        retry_at__isnull=True,
-    ).update(retry_at=now, modified_at=now)
+    # ArchiveResult rows are projections, never work items. Close interrupted
+    # projections as failed and wake their parent Snapshot so abx-dl can replay
+    # the snapshot-level sequence. Indexed subqueries keep this bounded.
     Snapshot.objects.filter(
         id__in=orphaned_results.values("snapshot_id"),
-        status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        retry_at__isnull=True,
+        status=Snapshot.StatusChoices.STARTED,
     ).update(retry_at=now, modified_at=now)
-    cleaned["archiveresults_backoff"] = backoff_results.update(status=ArchiveResult.StatusChoices.QUEUED, modified_at=now)
-    # Targeted plugin rows on final/paused Snapshots are scheduled through the
-    # parent Snapshot.retry_at. retry_at=NULL is the normal idle marker for a
-    # sealed Snapshot and must not be interpreted as queued work just because
-    # old/synthetic ArchiveResult rows exist. If takeover kills the runner
-    # after it leases the Snapshot but before queued ArchiveResult rows finish,
-    # the rows remain QUEUED while retry_at sits in the future. Recovery runs
-    # only after this runner has won the single-runner gate, so it can safely
-    # unlock those stale plugin leases for immediate processing instead of
-    # waiting out the previous owner's full lock timeout.
-    queued_plugin_results = ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.QUEUED, **result_filter)
-    cleaned["snapshots_queued_plugin_rows_waiting_on_stale_lease"] = (
-        Snapshot.objects.filter(
-            id__in=queued_plugin_results.values("snapshot_id"),
-            status__in=[Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED],
-        )
-        .filter(retry_at__gt=now)
-        .update(retry_at=now, modified_at=now)
-    )
-    # Impossible state repair: STARTED ArchiveResults without a live Process
-    # have no owner left to emit completion. Requeue only the result row; the
-    # snapshot/crawl schedulers will pick up normal retry processing.
-    cleaned["archiveresults_started_without_running_process"] = orphaned_results.update(
-        status=ArchiveResult.StatusChoices.QUEUED,
-        process=None,
+    cleaned["archiveresults_interrupted_without_running_process"] = orphaned_results.update(
+        status=ArchiveResult.StatusChoices.FAILED,
         modified_at=now,
     )
     orphaned_hook_processes = Process.objects.filter(
@@ -133,11 +140,23 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             orphaned_hook_processes = orphaned_hook_processes.filter(snapshot_pwd_filter)
         else:
             orphaned_hook_processes = orphaned_hook_processes.none()
-    for process in orphaned_hook_processes.only("id", "pwd", "cmd", "process_type", "status"):
+    for process in orphaned_hook_processes.only(
+        "id",
+        "pwd",
+        "cmd",
+        "process_type",
+        "status",
+        "exit_code",
+        "stdout",
+        "stderr",
+        "started_at",
+        "ended_at",
+    ).order_by("-started_at", "-id"):
         hook_script_name = process.hook_script_name
         if not hook_script_name or not process.pwd:
             continue
         plugin_dir = Path(process.pwd)
+        hook_name = _canonical_hook_name(hook_script_name)
         if crawl_snapshot_ids is not None and plugin_dir.parent.name not in crawl_snapshot_ids:
             continue
         try:
@@ -149,49 +168,75 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             continue
         if snapshot is None:
             continue
-        result, created = ArchiveResult.objects.get_or_create(
-            snapshot=snapshot,
-            plugin=plugin_dir.name,
-            hook_name=Path(hook_script_name).stem,
+        result, created = ArchiveResult.get_or_create_by_hook(
+            snapshot,
+            plugin_dir.name,
+            hook_name,
             defaults={
-                "status": ArchiveResult.StatusChoices.QUEUED,
+                "status": ArchiveResult.StatusChoices.FAILED,
             },
         )
-        if result.status == ArchiveResult.StatusChoices.QUEUED:
+        process_is_newer = bool(process.started_at and (result.start_ts is None or process.started_at >= result.start_ts))
+        if created or process_is_newer:
             requeue_snapshot = False
             # A runner can die after the hook Process exits but before the
             # ProcessCompletedEvent projector links/finalizes ArchiveResult.
             # Reconstruct only that exact hook row from the durable Process row.
-            output_files, output_size, output_mimetypes = _collect_output_metadata(plugin_dir)
+            manifest = OutputManifest.scan(plugin_dir, containment_root=snapshot.output_dir)
+            output_files = manifest.as_mapping()
+            output_size = manifest.total_size
+            output_mimetypes = ",".join(manifest.mimetypes)
+            emitted_records = [
+                record
+                for record in Process.parse_records_from_text(process.stdout or "")
+                if record.get("type") == "ArchiveResult"
+                and (record.get("plugin") or plugin_dir.name) == plugin_dir.name
+                and _canonical_hook_name(str(record.get("hook_name") or hook_name)) == hook_name
+            ]
+            emitted_result = emitted_records[-1] if emitted_records else {}
             result.process = process
+            result.start_ts = process.started_at
+            result.end_ts = process.ended_at
             if _is_signal_interrupted_exit(process.exit_code):
-                # The owning runner died or was asked to stop while the hook was
-                # still active. Keep the work item queued so takeover retries the
-                # same hook; treating an unknown signal exit as success would
-                # silently skip unfinished side effects.
+                # The Process is a durable fact; record the interruption as a
+                # failure and wake the parent Snapshot for replay.
                 result.output_files = {}
                 result.output_size = 0
                 result.output_mimetypes = ""
                 result.output_str = ""
-                result.status = ArchiveResult.StatusChoices.QUEUED
+                result.output_json = None
+                result.status = ArchiveResult.StatusChoices.FAILED
                 requeue_snapshot = True
             else:
                 result.output_files = output_files
                 result.output_size = output_size
                 result.output_mimetypes = output_mimetypes
-                result.output_str = process.stderr if process.exit_code not in (0, None) else ""
+                result.output_str = (
+                    emitted_result.get("output_str")
+                    or emitted_result.get("output")
+                    or (process.stderr if process.exit_code not in (0, None) else "")
+                )
+                result.output_json = emitted_result.get("output_json") if isinstance(emitted_result.get("output_json"), dict) else None
+                emitted_status = emitted_result.get("status")
                 result.status = (
-                    ArchiveResult.StatusChoices.FAILED
-                    if process.exit_code not in (0, None)
-                    else (ArchiveResult.StatusChoices.SUCCEEDED if output_files else ArchiveResult.StatusChoices.NORESULTS)
+                    emitted_status
+                    if emitted_status in ArchiveResult.StatusChoices.values
+                    else (
+                        ArchiveResult.StatusChoices.FAILED
+                        if process.exit_code not in (0, None)
+                        else (ArchiveResult.StatusChoices.SUCCEEDED if output_files else ArchiveResult.StatusChoices.NORESULTS)
+                    )
                 )
             result.save(
                 update_fields=[
                     "process",
+                    "start_ts",
+                    "end_ts",
                     "output_files",
                     "output_size",
                     "output_mimetypes",
                     "output_str",
+                    "output_json",
                     "status",
                     "modified_at",
                 ],
@@ -200,7 +245,6 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
                 Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
         if created:
             cleaned["archiveresults_missing_for_orphaned_hook_processes"] += 1
-            Snapshot.objects.filter(id=snapshot.id).update(retry_at=now, modified_at=now)
     started_snapshots = Snapshot.objects.filter(status=Snapshot.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
         **snapshot_filter,
@@ -214,8 +258,8 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
     # We only unlock scheduling; normal Snapshot runner code owns the next
     # transition and side effects.
     cleaned["snapshots_started_without_running_results"] = (
-        started_snapshots.annotate(has_running_results=Exists(running_archiveresults))
-        .filter(has_running_results=False)
+        started_snapshots.annotate(has_running_process=Exists(running_hook_processes))
+        .filter(has_running_process=False)
         .update(
             retry_at=now,
             modified_at=now,
@@ -224,7 +268,7 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
 
     # Broken lock repair: STARTED + retry_at=NULL is an orphaned ownership
     # lease. Recovery only unlocks scheduling; the runner owns any subsequent
-    # state-machine transition, including sealing rows whose children/results
+    # lifecycle transition, including sealing rows whose children/results
     # are already final.
     recoverable_started_crawls = Crawl.objects.filter(status=Crawl.StatusChoices.STARTED).filter(
         Q(retry_at__isnull=True) | Q(retry_at__gt=now),
@@ -268,13 +312,13 @@ def recover_orchestrator_state(*, include_chrome: bool = False, crawl_id: str |
             "Starting {count} Snapshot(s) that were queued but never started "
             "(ArchiveBox may have been interrupted before it was able to archive those URLs)."
         ),
-        "archiveresults_backoff": (
-            "Retrying {count} extractor result(s) that were waiting to retry "
-            "(ArchiveBox may have been interrupted before it was able to try them again; affected outputs will be retried)."
+        "snapshots_sealed_with_extension_uploads_only": (
+            "Finishing {count} browser-extension Snapshot(s) that received uploaded files before server extractors started "
+            "(uploaded and server-created results will remain together on the same Snapshot)."
         ),
-        "archiveresults_started_without_running_process": (
-            "Retrying {count} extractor result(s) that were interrupted before finishing "
-            "(ArchiveBox may have been interrupted before it was able to save their final status; partial files will be overwritten with fresh results upon retry)."
+        "archiveresults_interrupted_without_running_process": (
+            "Closing {count} interrupted extractor result projection(s) "
+            "(their parent Snapshots are resumed through the normal snapshot-level runner)."
         ),
         "snapshots_started_without_running_results": (
             "Resuming {count} Snapshot(s) that were interrupted before finishing "

archivebox/core/routes_util.py

@@ -60,13 +60,6 @@ def _csrf_trusted_origins(config) -> list[str]:
     return seen
 
 
-def _allowed_hosts(config) -> set[str]:
-    raw = (config.ALLOWED_HOSTS or "").strip()
-    if not raw:
-        return set()
-    return {entry.strip().lower() for entry in raw.split(",") if entry.strip() and entry.strip() != "*"}
-
-
 def derive_base_url_from_csrf(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
     """Pick a single CSRF_TRUSTED_ORIGINS entry to act as the implicit BASE_URL.
 
@@ -199,25 +192,25 @@ def _build_base_host(subdomain: str | None, request=None, config: dict[str, Any]
     return _with_port(full_host, port)
 
 
-def get_admin_host(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_admin_host(config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
-        return get_base_host(config=config)
-    return _build_base_host("admin", config=config)
+        return get_base_host(request=request, config=config)
+    return _build_base_host("admin", request=request, config=config)
 
 
-def get_web_host(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_web_host(config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
-        return get_base_host(config=config)
-    return _build_base_host("web", config=config)
+        return get_base_host(request=request, config=config)
+    return _build_base_host("web", request=request, config=config)
 
 
-def get_api_host(config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_api_host(config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
-        return get_base_host(config=config)
-    return _build_base_host("api", config=config)
+        return get_base_host(request=request, config=config)
+    return _build_base_host("api", request=request, config=config)
 
 
 def get_snapshot_subdomain(snapshot_id: str) -> str:
@@ -226,18 +219,18 @@ def get_snapshot_subdomain(snapshot_id: str) -> str:
     return f"snap-{suffix}"
 
 
-def get_snapshot_host(snapshot_id: str, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_snapshot_host(snapshot_id: str, config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
-        return get_base_host(config=config)
-    return _build_base_host(get_snapshot_subdomain(snapshot_id), config=config)
+        return get_base_host(request=request, config=config)
+    return _build_base_host(get_snapshot_subdomain(snapshot_id), request=request, config=config)
 
 
-def get_original_host(domain: str, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_original_host(domain: str, config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
-        return get_base_host(config=config)
-    return _build_base_host(domain, config=config)
+        return get_base_host(request=request, config=config)
+    return _build_base_host(domain, request=request, config=config)
 
 
 def is_snapshot_subdomain(subdomain: str) -> bool:
@@ -255,12 +248,12 @@ def get_snapshot_lookup_key(snapshot_ref: str) -> str:
     return value
 
 
-def get_listen_subdomain(request_host: str, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+def get_listen_subdomain(request_host: str, config: dict[str, Any] | None = None, request=None, **config_kwargs: Any) -> str:
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return ""
     req_host, req_port = split_host_port(request_host)
-    base_host, base_port = split_host_port(get_base_host(config=config))
+    base_host, base_port = split_host_port(get_base_host(request=request, config=config))
     if not base_host:
         return ""
     if base_port and req_port and base_port != req_port:
@@ -382,9 +375,7 @@ def get_snapshot_base_url(snapshot_id: str, request=None, config: dict[str, Any]
 
 def get_original_base_url(domain: str, request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
     config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
-    if not config.USES_SUBDOMAIN_ROUTING:
-        return _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
-    return _build_base_url_for_host(_build_base_host(domain, request=request, config=config), request=request, config=config)
+    return _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
 
 
 def build_admin_url(path: str = "", request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:

archivebox/core/settings.py

@@ -5,12 +5,13 @@ import inspect
 import logging
 import os
 import sys
+from importlib.util import find_spec
 from pathlib import Path
 
 from django.conf.locale.en import formats as en_formats  # type: ignore
 
 import archivebox
-from archivebox.config.common import get_config
+from archivebox.config.django import CONFIG
 from archivebox.config.constants import CONSTANTS
 from archivebox.core.routes_util import get_admin_base_url, get_api_base_url, get_base_url, normalize_base_url
 
@@ -24,7 +25,6 @@ IS_MIGRATING = "makemigrations" in sys.argv[:3] or "migrate" in sys.argv[:3]
 IS_TESTING = "test" in sys.argv[:3]
 IS_SHELL = "shell" in sys.argv[:3] or "shell_plus" in sys.argv[:3]
 IS_GETTING_VERSION_OR_HELP = "version" in sys.argv or "help" in sys.argv or "--version" in sys.argv or "--help" in sys.argv
-CONFIG = get_config()
 PACKAGE_DIR = CONSTANTS.PACKAGE_DIR
 logger = logging.getLogger(__name__)
 
@@ -76,7 +76,6 @@ INSTALLED_APPS = [
     "archivebox.crawls",  # handles Crawl and CrawlSchedule models and management (depends on core)
     "archivebox.progressmonitor",  # live progress endpoint and admin monitor template
     "archivebox.api",  # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.
-    "archivebox.opencode",
     # 3rd-party apps from PyPI that need to be loaded last
     "admin_data_views",  # handles rendering some convenient automatic read-only views of data in Django admin
     "django_extensions",  # provides Django Debug Toolbar (and other non-debug helpers)
@@ -249,7 +248,7 @@ SQLITE_CONNECTION_OPTIONS = {
     },
 }
 
-if is_postgres():
+if is_postgres(CONFIG):
     DATABASES = {
         "default": {
             "ENGINE": "django.db.backends.postgresql",
@@ -382,8 +381,8 @@ SECURE_CONTENT_TYPE_NOSNIFF = True
 SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
 
 # When BASE_URL is an https:// URL the deployment is HTTPS end-to-end, typically
-# behind a TLS-terminating proxy/tunnel (the bundled traefik/cloudflared profiles,
-# or your own caddy/traefik/nginx) where the proxy -> archivebox hop is plain HTTP, so
+# behind a TLS-terminating proxy/tunnel (such as Cloudflare, Caddy, Traefik, or nginx)
+# where the proxy -> archivebox hop is plain HTTP, so
 # request.is_secure() / request.scheme would otherwise report http. Honour the
 # proxy's X-Forwarded-Proto so first-run URL detection and CSRF origin checks are
 # correct. Mark auth cookies Secure once the saved BASE_URL confirms HTTPS.
@@ -603,8 +602,7 @@ if DEBUG:
 # https://github.com/bensi94/Django-Requests-Tracker (improved version of django-debug-toolbar)
 # Must delete archivebox/templates/admin to use because it relies on some things we override
 # visit /__requests_tracker__/ to access
-DEBUG_REQUESTS_TRACKER = True
-DEBUG_REQUESTS_TRACKER = DEBUG_REQUESTS_TRACKER and DEBUG
+DEBUG_REQUESTS_TRACKER = DEBUG and find_spec("requests_tracker") is not None
 if DEBUG_REQUESTS_TRACKER:
     import requests_tracker
 

archivebox/core/tag_util.py

@@ -5,7 +5,6 @@ from collections import defaultdict
 from typing import Any
 from urllib.parse import unquote
 
-from django.contrib.auth.models import User
 from django.db.models import Count, Exists, F, OuterRef, QuerySet
 from django.db.models.functions import Lower
 from django.http import HttpRequest
@@ -147,41 +146,6 @@ def get_tag_by_ref(tag_ref: str | int) -> Tag:
     return Tag.objects.get(name__iexact=decoded)
 
 
-def get_or_create_tag(name: str, created_by: User | None = None) -> tuple[Tag, bool]:
-    normalized_name = normalize_tag_name(name)
-    if not normalized_name:
-        raise ValueError("Tag name is required")
-
-    existing = Tag.objects.filter(name__iexact=normalized_name).first()
-    if existing:
-        return existing, False
-
-    tag = Tag.objects.create(
-        name=normalized_name,
-        created_by=created_by,
-    )
-    return tag, True
-
-
-def rename_tag(tag: Tag, name: str) -> Tag:
-    normalized_name = normalize_tag_name(name)
-    if not normalized_name:
-        raise ValueError("Tag name is required")
-
-    existing = Tag.objects.filter(name__iexact=normalized_name).exclude(pk=tag.pk).first()
-    if existing:
-        raise ValueError(f'Tag "{existing.name}" already exists')
-
-    if tag.name != normalized_name:
-        tag.name = normalized_name
-        tag.save()
-    return tag
-
-
-def delete_tag(tag: Tag) -> tuple[int, dict[str, int]]:
-    return tag.delete()
-
-
 def export_tag_urls(tag: Tag) -> str:
     urls = tag.snapshot_set.order_by("-downloaded_at", "-created_at", "-pk").values_list("url", flat=True)
     return "\n".join(urls)
@@ -257,10 +221,10 @@ def build_tag_card(tag: Tag, snapshot_previews: list[dict[str, Any]] | None = No
         "num_snapshots": count,
         "filter_url": f"/admin/core/snapshot/?tags__id__exact={tag.pk}",
         "edit_url": f"/admin/core/tag/{tag.pk}/change/",
-        "export_urls_url": f"/api/v1/core/tag/{tag.pk}/urls.txt",
-        "export_jsonl_url": f"/api/v1/core/tag/{tag.pk}/snapshots.jsonl",
-        "rename_url": f"/api/v1/core/tag/{tag.pk}/rename",
-        "delete_url": f"/api/v1/core/tag/{tag.pk}/",
+        "export_urls_url": reverse("api-1:tag_urls_export", args=[tag.pk]),
+        "export_jsonl_url": reverse("api-1:tag_snapshots_export", args=[tag.pk]),
+        "rename_url": reverse("api-1:rename_tag", args=[tag.pk]),
+        "delete_url": reverse("api-1:delete_tag", args=[tag.pk]),
         "snapshots": snapshot_previews or [],
     }
 

archivebox/core/takeover_util.py

@@ -1,7 +1,22 @@
+"""Coordinate local foreground processes without pretending to provide a distributed lock.
+
+ArchiveBox wants one orchestrator per collection, but Process rows can only prove
+liveness for PIDs visible on the current machine/PID namespace. We therefore
+enforce one active runner per ``(Machine, DATA_DIR)`` locally, retire stale rows
+from sequential containers on that machine, and only warn about rows owned by a
+different machine. Foreign-machine rows must never block progress or be killed:
+multi-machine coordination belongs in the Crawl/Snapshot CAS claim layer, not in
+process takeover.
+
+These helpers only hand local supervisord/runner ownership between CLI parents.
+They must not hold database transactions or filesystem locks while work runs.
+"""
+
 from __future__ import annotations
 
 import time
 import sys
+from collections.abc import Callable
 from pathlib import Path
 
 from django.db import IntegrityError
@@ -14,25 +29,6 @@ RUNNER_WAITING_WORKER_TYPE = "runner_waiting"
 RUNNER_GATE_WORKER_TYPES = (RUNNER_ACTIVE_WORKER_TYPE, RUNNER_WAITING_WORKER_TYPE, "")
 
 
-def runtime_stack_owner_types():
-    from archivebox.machine.models import Process
-
-    return (
-        Process.TypeChoices.SERVER,
-        Process.TypeChoices.ORCHESTRATOR,
-    )
-
-
-def foreground_runner_owner_types():
-    from archivebox.machine.models import Process
-
-    return (
-        Process.TypeChoices.SERVER,
-        Process.TypeChoices.ADD,
-        Process.TypeChoices.UPDATE,
-    )
-
-
 def current_command(process_type: str, *, data_dir: str | Path, url: str | None = None):
     from archivebox.machine.models import Process
 
@@ -41,31 +37,8 @@ def current_command(process_type: str, *, data_dir: str | Path, url: str | None
     return proc
 
 
-def live_processes(*, process_type: str, data_dir: str | Path, url: str | None = None):
-    from archivebox.machine.models import Machine, Process
-
-    qs = Process.objects.filter(
-        machine=Machine.current(),
-        process_type=process_type,
-        status=Process.StatusChoices.RUNNING,
-        pwd=str(data_dir),
-    )
-    if url is not None:
-        qs = qs.filter(url=url)
-    return [proc for proc in qs.order_by("-created_at", "-modified_at").iterator(chunk_size=50) if proc.is_running]
-
-
-def newest_live_process(*, process_type: str, data_dir: str | Path, url: str | None = None):
-    processes = live_processes(process_type=process_type, data_dir=data_dir, url=url)
-    return processes[0] if processes else None
-
-
-def command_is_newest(command, *, process_type: str, data_dir: str | Path, url: str | None = None) -> bool:
-    leader = newest_live_process(process_type=process_type, data_dir=data_dir, url=url)
-    return bool(leader and leader.id == command.id)
-
-
 def runtime_stack_owner(*, data_dir: str | Path, exclude_id=None):
+    """Return the live local parent allowed to own the server runtime stack."""
     from archivebox.machine.models import Machine, Process
 
     machine = Machine.current()
@@ -73,7 +46,7 @@ def runtime_stack_owner(*, data_dir: str | Path, exclude_id=None):
         machine=machine,
         status=Process.StatusChoices.RUNNING,
         pwd=str(data_dir),
-        process_type__in=runtime_stack_owner_types(),
+        process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ORCHESTRATOR),
     )
     if exclude_id is not None:
         base_qs = base_qs.exclude(id=exclude_id)
@@ -102,6 +75,7 @@ def command_owns_runtime_stack(command, *, data_dir: str | Path) -> bool:
 
 
 def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
+    """Return the newest live local parent allowed to borrow runner/sonic."""
     from archivebox.machine.models import Machine, Process
 
     machine = Machine.current()
@@ -109,7 +83,7 @@ def foreground_runner_owner(*, data_dir: str | Path, exclude_id=None):
         machine=machine,
         status=Process.StatusChoices.RUNNING,
         pwd=str(data_dir),
-        process_type__in=foreground_runner_owner_types(),
+        process_type__in=(Process.TypeChoices.SERVER, Process.TypeChoices.ADD, Process.TypeChoices.UPDATE),
     )
     if exclude_id is not None:
         qs = qs.exclude(id=exclude_id)
@@ -125,27 +99,6 @@ def command_owns_foreground_runner(command, *, data_dir: str | Path) -> bool:
     return bool(owner and owner.id == command.id)
 
 
-def runtime_stack_component_label(*, owner=None, data_dir: str | Path) -> str:
-    try:
-        from archivebox.config.common import get_config
-        from archivebox.workers.supervisord_util import active_supervisord_runtime_components
-
-        components = active_supervisord_runtime_components(config=get_config())
-    except Exception:
-        components = []
-
-    names = list(components)
-    if not names and owner is not None:
-        from archivebox.machine.models import Process
-
-        if owner.process_type == Process.TypeChoices.SERVER:
-            names = ["orchestrator", "server"]
-        elif owner.process_type == Process.TypeChoices.ORCHESTRATOR:
-            names = ["orchestrator"]
-
-    return ", ".join(dict.fromkeys(names)) or "runtime stack"
-
-
 def ensure_daemon_stack(*, reason: str = ""):
     from archivebox.config.common import get_config
     from archivebox.workers.supervisord_util import (
@@ -173,64 +126,76 @@ def ensure_daemon_stack(*, reason: str = ""):
 
     supervisor = get_existing_supervisord_process() or get_or_create_supervisord_process(daemonize=False)
     worker = get_worker(supervisor, sonic_worker["name"])
-    if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
-        return worker
+    if not (isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING")):
+        if reason:
+            rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
+        worker = start_worker(supervisor, sonic_worker)
 
-    if reason:
-        rprint(f"[yellow][*] Starting daemon stack for {reason}...[/yellow]")
-    return start_worker(supervisor, sonic_worker)
+    from archivebox.search.sonic_daemon import wait_for_sonic_daemon
+
+    wait_for_sonic_daemon(sonic_event)
+    return worker
 
 
-def healthy_orchestrator(*, data_dir: str | Path):
-    from archivebox.machine.models import Machine, Process
-    from archivebox.workers.supervisord_util import get_existing_supervisord_process, get_worker
+def live_runner_processes(*, data_dir: str | Path):
+    """Return locally verifiable runners and warn about unsupported overlap.
 
-    supervisor = get_existing_supervisord_process()
-    worker = get_worker(supervisor, "worker_runner") if supervisor else None
-    if isinstance(worker, dict) and worker.get("statename") in ("STARTING", "RUNNING"):
-        return worker
-
-    for proc in Process.objects.filter(
-        machine=Machine.current(),
-        process_type=Process.TypeChoices.ORCHESTRATOR,
-        status=Process.StatusChoices.RUNNING,
-        pwd=str(data_dir),
-    ).order_by("-created_at"):
-        if proc.is_running:
-            return proc
-    return None
-
-
-def _runner_sort_key(process):
-    return (process.started_at or process.created_at, process.created_at, str(process.id))
-
-
-def live_runner_processes(*, data_dir: str | Path, exclude_id=None):
+    A Process row from another machine is observability only: its PID cannot be
+    checked or signalled here, so it neither joins the local election nor gets
+    mutated. A row for this same Machine from another PID namespace represents
+    a previous sequential container under the supported model; warn, retire the
+    unreachable row, and let the new container continue.
+    """
     from archivebox.machine.models import Machine, Process
 
     machine = Machine.current()
     Process.cleanup_stale_running(machine=machine)
     qs = Process.objects.filter(
-        machine=machine,
         status=Process.StatusChoices.RUNNING,
         process_type=Process.TypeChoices.ORCHESTRATOR,
         worker_type__in=RUNNER_GATE_WORKER_TYPES,
         pwd=str(data_dir),
     )
-    if exclude_id is not None:
-        qs = qs.exclude(id=exclude_id)
-    return [process for process in qs.order_by("started_at", "created_at").iterator(chunk_size=20) if process.is_running]
+    foreign_machine_exists = qs.exclude(machine=machine).exists()
+    qs = qs.filter(machine=machine)
+    live = []
+    foreign_namespace_ids = []
+    for process in qs.order_by("started_at", "created_at").iterator(chunk_size=20):
+        if not process.shares_pid_namespace:
+            foreign_namespace_ids.append(process.id)
+            continue
+        if process.is_running:
+            live.append(process)
+    if foreign_machine_exists or foreign_namespace_ids:
+        rprint(
+            "[bold yellow]WARNING: Multiple orchestrators sharing a single collection is not officially supported! "
+            "Corruption may occur if you run two ArchiveBox workers on the same collection at once.[/bold yellow]",
+            file=sys.stderr,
+            soft_wrap=True,
+        )
+    if foreign_namespace_ids:
+        now = timezone.now()
+        Process.objects.filter(id__in=foreign_namespace_ids, status=Process.StatusChoices.RUNNING).update(
+            status=Process.StatusChoices.EXITED,
+            exit_code=0,
+            ended_at=now,
+            retry_at=None,
+            modified_at=now,
+        )
+    return live
 
 
 def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout: float = 5.0) -> bool:
     """
-    Admit exactly one active runner for this DATA_DIR using Process rows.
+    Admit one active runner for this Machine and DATA_DIR using Process rows.
 
     The current process is a real OS process while it waits, so we keep its
     Process row RUNNING but mark worker_type=runner_waiting. Only the process
     that wins takeover is promoted to worker_type=worker_runner, which is
-    protected by a partial unique DB constraint. Older runners are terminated
-    and fully waited out before promotion, so the runner work loop never overlaps.
+    protected by a partial unique DB constraint scoped to (Machine, DATA_DIR).
+    Older locally verifiable runners are terminated and fully waited out before
+    promotion, so runner work never overlaps on one machine. Foreign machines
+    are intentionally outside this gate and only produce a warning above.
     """
     from archivebox.machine.models import Process
 
@@ -252,7 +217,7 @@ def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout:
             )
             runners = live_runner_processes(data_dir=data_dir)
 
-        newest = max(runners, key=_runner_sort_key)
+        newest = max(runners, key=lambda process: (process.started_at or process.created_at, process.created_at, str(process.id)))
         if newest.id != command.id:
             rprint(
                 f"[yellow][*] Newer ArchiveBox runner pid={newest.pid} is taking over; exiting this runner.[/yellow]",
@@ -284,24 +249,9 @@ def enter_single_runner_gate(command, *, data_dir: str | Path, graceful_timeout:
             time.sleep(0.1)
 
 
-def standby_until_leader_needed(command, *, process_type: str, data_dir: str | Path, url: str | None = None, interval: float = 2.0) -> None:
-    from archivebox.workers.supervisord_util import reap_foreground_supervisord_process
-
-    announced = False
-    while not command_is_newest(command, process_type=process_type, data_dir=data_dir, url=url):
-        reap_foreground_supervisord_process()
-        if not announced:
-            leader = newest_live_process(process_type=process_type, data_dir=data_dir, url=url)
-            leader_pid = leader.pid if leader else "unknown"
-            rprint(f"[yellow][*] Standing by; newer ArchiveBox process pid={leader_pid} is running the orchestrator and server.[/yellow]")
-            announced = True
-        time.sleep(interval)
-    command.modified_at = timezone.now()
-    command.save(update_fields=["modified_at"])
-
-
 def standby_until_runtime_stack_needed(command, *, data_dir: str | Path, interval: float = 2.0) -> dict[str, object]:
-    from archivebox.workers.supervisord_util import reap_foreground_supervisord_process
+    from archivebox.machine.models import Process
+    from archivebox.workers.supervisord_util import active_supervisord_runtime_components, reap_foreground_supervisord_process
 
     announced = False
     previous_owner_pid = None
@@ -310,7 +260,16 @@ def standby_until_runtime_stack_needed(command, *, data_dir: str | Path, interva
         if not announced:
             owner = runtime_stack_owner(data_dir=data_dir)
             owner_pid = owner.pid if owner else "unknown"
-            components = runtime_stack_component_label(owner=owner, data_dir=data_dir)
+            try:
+                component_names = list(active_supervisord_runtime_components())
+            except Exception:
+                component_names = []
+            if not component_names and owner is not None:
+                if owner.process_type == Process.TypeChoices.SERVER:
+                    component_names = ["orchestrator", "server"]
+                elif owner.process_type == Process.TypeChoices.ORCHESTRATOR:
+                    component_names = ["orchestrator"]
+            components = ", ".join(dict.fromkeys(component_names)) or "runtime stack"
             previous_owner_pid = owner_pid
             rprint(
                 f"[yellow][*] A newer archivebox process took over the {components} "
@@ -324,12 +283,22 @@ def standby_until_runtime_stack_needed(command, *, data_dir: str | Path, interva
     return {"resumed": announced, "previous_owner_pid": previous_owner_pid}
 
 
-def standby_until_foreground_runner_needed(command, *, data_dir: str | Path, interval: float = 2.0) -> dict[str, object]:
+def standby_until_foreground_runner_needed(
+    command,
+    *,
+    data_dir: str | Path,
+    interval: float = 2.0,
+    work_is_complete: Callable[[], bool] | None = None,
+) -> dict[str, object]:
     from archivebox.workers.supervisord_util import reap_foreground_supervisord_process
 
     announced = False
     previous_owner_pid = None
-    while not command_owns_foreground_runner(command, data_dir=data_dir):
+    while True:
+        if work_is_complete is not None and work_is_complete():
+            return {"resumed": announced, "previous_owner_pid": previous_owner_pid, "work_completed": True}
+        if command_owns_foreground_runner(command, data_dir=data_dir):
+            break
         reap_foreground_supervisord_process()
         if not announced:
             owner = foreground_runner_owner(data_dir=data_dir)
@@ -344,4 +313,4 @@ def standby_until_foreground_runner_needed(command, *, data_dir: str | Path, int
         time.sleep(interval)
     command.modified_at = timezone.now()
     command.save(update_fields=["modified_at"])
-    return {"resumed": announced, "previous_owner_pid": previous_owner_pid}
+    return {"resumed": announced, "previous_owner_pid": previous_owner_pid, "work_completed": False}

archivebox/core/templatetags/core_tags.py

@@ -1,16 +1,19 @@
+import logging
 import os
+from html import unescape
 from pathlib import Path
 from typing import Any
+from urllib.parse import quote, urlparse
 
 from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
 from django import template
-from django.contrib.admin.templatetags.base import InclusionAdminNode
 from django.templatetags.static import static
 from django.utils import timezone
 from django.utils.html import escape
 from django.utils.safestring import mark_safe
 from django.utils.text import Truncator
 
+from archivebox.config import CONSTANTS
 from archivebox.core.routes_util import (
     build_snapshot_url,
     get_admin_base_url,
@@ -26,53 +29,38 @@ from archivebox.plugins.discovery import (
 
 register = template.Library()
 
+
+@register.simple_tag(takes_context=True)
+def plugin_ui(context, plugin: str, slot: str):
+    """Optional plugin UI must never prevent the host page from rendering."""
+    try:
+        source = get_plugin_template(plugin, slot, fallback=False)
+        if not source:
+            return ""
+        return template.engines["django"].from_string(source).render(context.flatten())
+    except Exception:
+        logging.getLogger(__name__).exception("Unable to render optional plugin UI: %s/%s", plugin, slot)
+        return ""
+
+
 _TEXT_PREVIEW_EXTS = (".json", ".jsonl", ".txt", ".csv", ".tsv", ".xml", ".yml", ".yaml", ".md", ".log")
 _IMAGE_PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".ico", ".avif")
-
-_MEDIA_FILE_EXTS = {
-    ".mp4",
-    ".webm",
-    ".mkv",
-    ".avi",
-    ".mov",
-    ".flv",
-    ".wmv",
-    ".m4v",
-    ".mpg",
-    ".mpeg",
-    ".ts",
-    ".m2ts",
-    ".mts",
-    ".3gp",
-    ".3g2",
-    ".ogv",
-    ".mp3",
-    ".m4a",
-    ".aac",
-    ".ogg",
-    ".oga",
-    ".opus",
-    ".wav",
-    ".flac",
-    ".alac",
-    ".aiff",
-    ".wma",
-    ".mka",
-    ".ac3",
-    ".eac3",
-    ".dts",
-}
+_STATIC_URL_SAFE = "/@-._~!$&'()*+,;="
 
 
-def _normalize_output_files(output_files: Any) -> dict[str, dict[str, Any]]:
-    if isinstance(output_files, dict):
-        normalized: dict[str, dict[str, Any]] = {}
-        for path, metadata in output_files.items():
-            if not path:
-                continue
-            normalized[str(path)] = dict(metadata) if isinstance(metadata, dict) else {}
-        return normalized
-    return {}
+def _is_web_subdomain_alias(browser_url: str, configured_base_url: str) -> bool:
+    """Hide the informational mismatch banner on the conventional public web alias.
+
+    This is deliberately presentation-only: request routing and the restriction
+    on state-changing requests continue to use the configured BASE_URL.
+    """
+    browser = urlparse(browser_url)
+    configured = urlparse(configured_base_url)
+    return (
+        browser.scheme.lower() == configured.scheme.lower()
+        and browser.hostname == f"web.{configured.hostname}"
+        and browser.port == configured.port
+    )
 
 
 def _snapshot_id(value: Any) -> Any:
@@ -81,93 +69,6 @@ def _snapshot_id(value: Any) -> Any:
     return value.id if isinstance(value, Snapshot) else value
 
 
-def _coerce_output_file_size(value: Any) -> int | None:
-    try:
-        return max(int(value or 0), 0)
-    except (TypeError, ValueError):
-        return None
-
-
-def _count_media_files(result) -> int:
-    try:
-        output_files = _normalize_output_files(result.output_files or {})
-    except (AttributeError, TypeError, ValueError):
-        output_files = {}
-
-    if output_files:
-        return sum(1 for path in output_files if Path(path).suffix.lower() in _MEDIA_FILE_EXTS)
-
-    try:
-        plugin_dir = Path(result.snapshot_dir) / result.plugin
-    except (AttributeError, TypeError, ValueError):
-        return 0
-
-    if not plugin_dir.exists():
-        return 0
-
-    count = 0
-    scanned = 0
-    max_scan = 500
-    for file_path in plugin_dir.rglob("*"):
-        if scanned >= max_scan:
-            break
-        scanned += 1
-        if not file_path.is_file():
-            continue
-        if file_path.suffix.lower() in _MEDIA_FILE_EXTS:
-            count += 1
-    return count
-
-
-def _list_media_files(result) -> list[dict]:
-    media_files: list[dict] = []
-    try:
-        plugin_dir = Path(result.snapshot_dir) / result.plugin
-    except (AttributeError, TypeError, ValueError):
-        return media_files
-
-    output_files = _normalize_output_files(result.output_files or {})
-    candidates: list[tuple[Path, int | None]] = []
-    if output_files:
-        for path, metadata in output_files.items():
-            rel_path = Path(path)
-            if rel_path.suffix.lower() in _MEDIA_FILE_EXTS:
-                candidates.append((rel_path, _coerce_output_file_size(metadata.get("size"))))
-
-    if not candidates and plugin_dir.exists():
-        scanned = 0
-        max_scan = 2000
-        for file_path in plugin_dir.rglob("*"):
-            if scanned >= max_scan:
-                break
-            scanned += 1
-            if not file_path.is_file():
-                continue
-            if file_path.suffix.lower() in _MEDIA_FILE_EXTS:
-                try:
-                    rel_path = file_path.relative_to(plugin_dir)
-                except ValueError:
-                    continue
-                try:
-                    size = file_path.stat().st_size
-                except OSError:
-                    size = None
-                candidates.append((rel_path, size))
-
-    for rel_path, size in candidates:
-        href = str(Path(result.plugin) / rel_path)
-        media_files.append(
-            {
-                "name": rel_path.name,
-                "path": href,
-                "size": size,
-            },
-        )
-
-    media_files.sort(key=lambda item: item["name"].lower())
-    return media_files
-
-
 def _resolve_snapshot_output_file(snapshot_dir: str | Path | None, raw_output_path: str | None) -> Path | None:
     if not snapshot_dir or not raw_output_path or str(raw_output_path).strip() in (".", "/", "./"):
         return None
@@ -202,16 +103,70 @@ def _is_root_snapshot_output_path(raw_output_path: str | None) -> bool:
     return normalized in ("", ".", "./", "/", "index.html", "index.json")
 
 
-def _build_snapshot_files_url(snapshot_id: str, request=None, config=None) -> str:
-    return build_snapshot_url(str(snapshot_id), "/?files=1", request=request, config=config)
+def _static_snapshot_base_url(context, snapshot) -> str:
+    start_dir = Path(context["STATIC_EXPORT_DIR"])
+    relative_path = os.path.relpath(Path(snapshot.output_dir), start=start_dir).replace(os.sep, "/")
+    if relative_path == ".":
+        return "."
+    return f"./{quote(relative_path, safe=_STATIC_URL_SAFE)}"
 
 
-def _build_snapshot_preview_url(snapshot_id: str, path: str = "", request=None, config=None, plugin: str = "") -> str:
+def _snapshot_base_url_for_context(context, snapshot) -> str:
+    if context.get("STATIC_EXPORT"):
+        return _static_snapshot_base_url(context, snapshot)
+    return get_snapshot_base_url(
+        str(_snapshot_id(snapshot)),
+        request=context.get("request"),
+        config=context.get("CONFIG"),
+    )
+
+
+def _snapshot_url_for_context(context, snapshot, path: str = "") -> str:
+    if not context.get("STATIC_EXPORT"):
+        return build_snapshot_url(
+            str(_snapshot_id(snapshot)),
+            path,
+            request=context.get("request"),
+            config=context.get("CONFIG"),
+        )
+
+    base_url = _static_snapshot_base_url(context, snapshot)
+    raw_path = str(path or "")
+    if not raw_path:
+        return base_url
+    path_part, separator, query = raw_path.lstrip("/").partition("?")
+    quoted_path = quote(path_part, safe=_STATIC_URL_SAFE)
+    suffix = f"?{query}" if separator else ""
+    return f"{base_url.rstrip('/')}/{quoted_path}{suffix}"
+
+
+def _build_snapshot_files_url(snapshot_id: str, request=None, config=None, base_url: str | None = None) -> str:
+    return (
+        f"{base_url.rstrip('/')}/index.jsonl"
+        if base_url
+        else build_snapshot_url(str(snapshot_id), "/?files=1", request=request, config=config)
+    )
+
+
+def _build_snapshot_preview_url(
+    snapshot_id: str,
+    path: str = "",
+    request=None,
+    config=None,
+    plugin: str = "",
+    base_url: str | None = None,
+) -> str:
     if path == "about:blank":
         return path
     if _is_root_snapshot_output_path(path):
-        return _build_snapshot_files_url(snapshot_id, request=request, config=config)
-    url = build_snapshot_url(str(snapshot_id), path, request=request, config=config)
+        return _build_snapshot_files_url(snapshot_id, request=request, config=config, base_url=base_url)
+    if base_url:
+        path_part, separator, query = str(path).lstrip("/").partition("?")
+        url = f"{base_url.rstrip('/')}/{quote(path_part, safe=_STATIC_URL_SAFE)}"
+        if separator:
+            url = f"{url}?{query}"
+    else:
+        url = build_snapshot_url(str(snapshot_id), path, request=request, config=config)
     path_parts = Path(path).parts
     plugin = get_plugin_name(plugin) if plugin else (path_parts[0] if len(path_parts) > 1 else "")
     has_plugin_preview = bool(plugin and get_plugin_template(plugin, "full", fallback=False))
@@ -299,6 +254,7 @@ def intcomma(value: int | str | None) -> str:
         return str(value or "")
 
 
+@register.inclusion_tag("admin/snapshots_grid.html", takes_context=True, name="snapshots_grid")
 def result_list(context, cl):
     """
     Monkey patched result
@@ -319,17 +275,6 @@ def result_list(context, cl):
     }
 
 
-@register.tag(name="snapshots_grid")
-def result_list_tag(parser, token):
-    return InclusionAdminNode(
-        parser,
-        token,
-        func=result_list,
-        template_name="snapshots_grid.html",
-        takes_context=True,
-    )
-
-
 _LOW_DISK_THRESHOLD_GB = 1.0
 _HIGH_MEMORY_THRESHOLD_PCT = 95.0
 _HIGH_LOAD_MULTIPLE = 3  # 15-min loadavg > 3 * cpu_count
@@ -389,6 +334,9 @@ def system_warnings_banner(context):
     after and reuse ``machine.detect.get_host_stats`` (the same function that
     populates the Machine admin page), cached for 30s.
     """
+    if context.get("first_admin_setup"):
+        return {"mode": ""}
+
     config = context.get("CONFIG")
     if config is None:
         from archivebox.config.common import get_config
@@ -398,7 +346,7 @@ def system_warnings_banner(context):
     if not config.BASE_URL:
         return get_setup_wizard_context(context.get("request"), config)
     mismatch = get_base_url_mismatch_context(context.get("request"), config)
-    if mismatch:
+    if mismatch and not _is_web_subdomain_alias(mismatch["browser_url"], mismatch["configured_base_url"]):
         return mismatch
     if config.IS_LOWER_SECURITY_MODE:
         return {"mode": "unsafe"}
@@ -438,16 +386,6 @@ def url_replace(context, **kwargs):
     return dict_.urlencode()
 
 
-@register.simple_tag
-def has_real_admin_users() -> bool:
-    """True if any non-``system`` superuser exists. Used by the login page to
-    only show the bootstrap hint (createsuperuser / ADMIN_USERNAME env vars)
-    when the collection still has no real admin."""
-    from django.contrib.auth.models import User
-
-    return User.objects.filter(is_superuser=True).exclude(username="system").exists()
-
-
 @register.simple_tag(takes_context=True)
 def admin_base_url(context) -> str:
     return get_admin_base_url(request=context.get("request"), config=context.get("CONFIG"))
@@ -458,23 +396,35 @@ def web_base_url(context) -> str:
     return get_web_base_url(request=context.get("request"), config=context.get("CONFIG"))
 
 
+@register.simple_tag(takes_context=True)
+def static_export_root_url(context) -> str:
+    if not context.get("STATIC_EXPORT"):
+        return ""
+    relative_path = os.path.relpath(CONSTANTS.DATA_DIR, start=Path(context["STATIC_EXPORT_DIR"]))
+    relative_path = relative_path.replace(os.sep, "/")
+    return "./" if relative_path == "." else f"{quote(relative_path, safe=_STATIC_URL_SAFE)}/"
+
+
 @register.simple_tag(takes_context=True)
 def snapshot_base_url(context, snapshot) -> str:
-    snapshot_id = _snapshot_id(snapshot)
-    return get_snapshot_base_url(str(snapshot_id), request=context.get("request"), config=context.get("CONFIG"))
+    return _snapshot_base_url_for_context(context, snapshot)
 
 
 @register.simple_tag(takes_context=True)
 def snapshot_url(context, snapshot, path: str = "") -> str:
-    snapshot_id = _snapshot_id(snapshot)
-    return build_snapshot_url(str(snapshot_id), path, request=context.get("request"), config=context.get("CONFIG"))
+    return _snapshot_url_for_context(context, snapshot, path)
 
 
 @register.simple_tag(takes_context=True)
 def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) -> str:
     snapshot_id = str(_snapshot_id(snapshot))
     url_cache = snapshot.__dict__.setdefault("_snapshot_archiveresult_url_cache", {})
-    cache_key = (plugin, filename)
+    cache_key = (
+        plugin,
+        filename,
+        bool(context.get("STATIC_EXPORT")),
+        str(context.get("STATIC_EXPORT_DIR") or ""),
+    )
     if cache_key in url_cache:
         return url_cache[cache_key]
 
@@ -510,7 +460,7 @@ def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) ->
         if not isinstance(file_info, dict) or int(file_info.get("size") or 0) <= 0:
             continue
         output_path = filename if file_info.get("root_relative") else f"{plugin}/{filename}"
-        url_cache[cache_key] = build_snapshot_url(snapshot_id, output_path, request=context.get("request"), config=context.get("CONFIG"))
+        url_cache[cache_key] = _snapshot_url_for_context(context, snapshot, output_path)
         return url_cache[cache_key]
 
     url_cache[cache_key] = ""
@@ -519,10 +469,7 @@ def snapshot_archiveresult_url(context, snapshot, plugin: str, filename: str) ->
 
 @register.simple_tag(takes_context=True)
 def snapshot_index_row(context, link) -> str:
-    snapshot_id = str(_snapshot_id(link))
-    request = context.get("request")
-    config = context.get("CONFIG")
-    snapshot_base = get_snapshot_base_url(snapshot_id, request=request, config=config)
+    snapshot_base = _snapshot_base_url_for_context(context, link)
 
     status = getattr(link, "status", None) or "unknown"
     bookmarked_at = getattr(link, "bookmarked_at", None)
@@ -540,7 +487,7 @@ def snapshot_index_row(context, link) -> str:
         title_time = f"Bookmarked: ({timestamp})"
 
     url = getattr(link, "url", "") or ""
-    title = getattr(link, "title", "") or ""
+    title = unescape(getattr(link, "title", "") or "")
     is_pending = status in {"queued", "started", "backoff"}
     title_text = title or url
     tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
@@ -551,22 +498,28 @@ def snapshot_index_row(context, link) -> str:
         tag_cell = '<span class="empty-value">...</span>'
 
     num_outputs = int(getattr(link, "num_outputs", 0) or 0)
-    icons = link.icons() if callable(getattr(link, "icons", None)) else getattr(link, "icons", "")
+    if context.get("STATIC_EXPORT") and callable(getattr(link, "icons", None)):
+        icons = link.icons(path=quote(link.static_archive_path, safe=_STATIC_URL_SAFE), prefix="./", quote_paths=True)
+    else:
+        icons = link.icons() if callable(getattr(link, "icons", None)) else getattr(link, "icons", "")
     icons_cell = str(icons) if icons else '<span class="empty-value">...</span>'
     archive_size = int(getattr(link, "archive_size", 0) or 0)
     size_cell = file_size(archive_size) if archive_size else '<span class="empty-value">...</span>'
     output_plural = "" if num_outputs == 1 else "s"
+    files_url = _snapshot_url_for_context(context, link, "index.jsonl") if context.get("STATIC_EXPORT") else f"{snapshot_base}/?files=1"
 
     if is_pending:
-        preview_html = (
-            '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress">'
-            f'<img src="{escape(static("spinner.gif"))}" alt="" decoding="async" loading="lazy">'
-            "</span>"
-        )
+        preview_html = '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress"></span>'
+        if not context.get("STATIC_EXPORT"):
+            preview_html = (
+                '<span class="snapshot-preview snapshot-preview-spinner" aria-label="Archiving in progress">'
+                f'<img src="{escape(static("spinner.gif"))}" alt="" decoding="async" loading="lazy">'
+                "</span>"
+            )
     elif "_public_preview_paths" in link.__dict__:
         preview_paths = list(getattr(link, "_public_preview_paths", []) or [])
         if preview_paths:
-            preview_urls = [build_snapshot_url(snapshot_id, path, request=request, config=config) for path in preview_paths]
+            preview_urls = [_snapshot_url_for_context(context, link, path) for path in preview_paths]
             preview_html = (
                 f'<img src="{escape(preview_urls[0])}" '
                 f'data-fallbacks="{escape(",".join(preview_urls[1:]))}" '
@@ -596,7 +549,7 @@ def snapshot_index_row(context, link) -> str:
     if "_public_favicon_paths" in link.__dict__:
         favicon_paths = list(getattr(link, "_public_favicon_paths", []) or [])
         if favicon_paths:
-            favicon_urls = [build_snapshot_url(snapshot_id, path, request=request, config=config) for path in favicon_paths]
+            favicon_urls = [_snapshot_url_for_context(context, link, path) for path in favicon_paths]
             favicon_html = (
                 f'<img src="{escape(favicon_urls[0])}" '
                 f'data-fallbacks="{escape(",".join(favicon_urls[1:]))}" '
@@ -657,7 +610,7 @@ def snapshot_index_row(context, link) -> str:
         </span>
     </td>
     <td class="snapshot-size-cell">
-        <a href="{escape(snapshot_base)}/?files=1" title="View archived files">
+        <a href="{escape(files_url)}" title="View archived file manifest">
             {size_cell}
         </a>
         <small>{num_outputs} output{output_plural}</small>
@@ -677,6 +630,7 @@ def snapshot_preview_url(context, snapshot, path: str = "", result=None) -> str:
         request=context.get("request"),
         config=context.get("CONFIG"),
         plugin=plugin,
+        base_url=_snapshot_base_url_for_context(context, snapshot) if context.get("STATIC_EXPORT") else None,
     )
 
 
@@ -716,24 +670,24 @@ def plugin_card(context, result) -> str:
 
     # Use embed_path() for the display path
     raw_output_path = result.embed_path() or ""
-    output_url = build_snapshot_url(
-        str(result.snapshot_id),
-        raw_output_path or "",
-        request=context.get("request"),
-        config=context.get("CONFIG"),
-    )
+    output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path or "")
 
     icon_html = get_plugin_icon(plugin)
     plugin_lower = (plugin or "").lower()
-    media_file_count = _count_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else 0
-    media_files = _list_media_files(result) if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else []
+    media_files = result.media_files() if plugin_lower in ("ytdlp", "yt-dlp", "youtube-dl") else []
+    if context.get("STATIC_EXPORT") and media_files:
+        media_files = [
+            item
+            for item in media_files
+            if (media_path := Path(str(item.get("path") or "")))
+            and not media_path.is_absolute()
+            and ".." not in media_path.parts
+            and (Path(result.snapshot_dir) / media_path).is_file()
+        ]
     if media_files:
-        snapshot_id = str(result.snapshot_id)
-        request = context.get("request")
-        config = context.get("CONFIG")
         for item in media_files:
             path = item.get("path") or ""
-            item["url"] = build_snapshot_url(snapshot_id, path, request=request, config=config) if path else ""
+            item["url"] = _snapshot_url_for_context(context, result.snapshot, path) if path else ""
 
     output_lower = (raw_output_path or "").lower()
     force_text_preview = output_lower.endswith(_TEXT_PREVIEW_EXTS)
@@ -750,7 +704,7 @@ def plugin_card(context, result) -> str:
                     "output_path_raw": raw_output_path,
                     "plugin": plugin,
                     "plugin_icon": icon_html,
-                    "media_file_count": media_file_count,
+                    "media_file_count": len(media_files),
                     "media_files": media_files,
                 },
             )
@@ -811,12 +765,7 @@ def plugin_full(context, result) -> str:
         raw_output_path = result.embed_path() or ""
     if _is_root_snapshot_output_path(raw_output_path):
         return ""
-    output_url = build_snapshot_url(
-        str(result.snapshot_id),
-        raw_output_path,
-        request=context.get("request"),
-        config=context.get("CONFIG"),
-    )
+    output_url = _snapshot_url_for_context(context, result.snapshot, raw_output_path)
 
     try:
         tpl = template.Template(template_str)

archivebox/core/urls.py

@@ -1,6 +1,5 @@
 __package__ = "archivebox.core"
 
-import sys
 from importlib.util import find_spec
 
 from django.conf import settings
@@ -10,7 +9,6 @@ from django.views.generic.base import RedirectView
 from django.http import HttpRequest
 
 from archivebox.config.constants import CONSTANTS
-from archivebox.config.common import get_config
 from archivebox.misc.serve_static import serve_static
 
 from archivebox.core.admin_site import archivebox_admin
@@ -30,9 +28,6 @@ from archivebox.progressmonitor.views import live_progress_view
 from archivebox.search.views import public_snapshot_search_stream_view
 from archivebox.opencode.views import opencode_proxy_view
 
-CONFIG = get_config()
-DEBUG = CONFIG.DEBUG or ("--debug" in sys.argv)
-
 urlpatterns = [
     re_path(r"^static/(?P<path>.*)$", serve_static),
     path("robots.txt", static.serve, {"document_root": CONSTANTS.STATIC_DIR, "path": "robots.txt"}),