Fix setup wizard reachability gate

Updated the note on URL archiving to include firewall/ACL-level blocking recommendations.

.github/SECURITY.md

@@ -20,15 +20,9 @@ Please start by reading these links to understand our existing security model, k
 
 ## Reporting a Vulnerability
 
-We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.
+~We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.~ We no longer offer CVEs or accept private reporting, 95% of the reports we received were low quality AI slop, so now we just ask for normal Github Issues or PRs.
 
-1. Go to the Security tab on our Github repo: https://github.com/ArchiveBox/ArchiveBox/security
-
-2. Click the ["Report a Vulnerability"](https://github.com/ArchiveBox/ArchiveBox/security/advisories/new) button
-
-3. Fill out the form to submit the details of the report and it will be securely sent to the maintainers
-
-You can also contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp).
+You can contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) if you have a truly urgent security report, but we do not issue public CVEs.
 
 ---
 
@@ -39,7 +33,7 @@ You can also contact the maintainers via our public [Zulip Chat Server zulip.arc
 > Reporters **will still receive credit in release notes** however.
 
 > [!TIP]
-> Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.
+> ~Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.~
 
 Please read our existing published [security advisories](https://github.com/ArchiveBox/ArchiveBox/security/advisories), [Security Overview Docs](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview), and [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and avoid creating duplicates.
 
@@ -68,4 +62,7 @@ Non-logged in users should be tightly limited to prevent attacks like SQL/shell
 
 [`SERVER_SECURITY_MODE`](https://github.com/archiveBox/archiveBox/wiki/configuration#server_security_mode) + CSRF/XSS defenses were only added in the new 0.9.0 `dev` work and are planned to land in `main` in mid-2026.
 
-Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs. The only URLs that are blocked are URLs hosted by the archivebox server itself.
+### SSRF / URL Validation
+
+Note archivebox is **designed to be able to archive any URL the server can reach**, including internal intranet URLs, private IPs, and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` or proper firewall/ACL-level blocking at the network level if they need to limit what the server should see.
+It does a basic check to try and avoid recursively archiving snapshots hosted by the archivebox server itself, but it's not a strict security check, it's just to prevent wasted CPU/disk from recursively archiving snapshots in a browser that has the ArchiveBox Browser Extension turned on.

.github/configs/ci-linux-build.json

@@ -0,0 +1,81 @@
+{
+  "properties": {
+    "CC_BINARY": {
+      "default": "cc"
+    },
+    "LDAPSEARCH_BINARY": {
+      "default": "ldapsearch"
+    },
+    "INITDB_BINARY": {
+      "default": "initdb"
+    },
+    "PG_CTL_BINARY": {
+      "default": "pg_ctl"
+    }
+  },
+  "required_binaries": [
+    {
+      "name": "{CC_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "build-essential"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{LDAPSEARCH_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "env": {
+          "version": [
+            "ldapsearch",
+            "-VV"
+          ]
+        },
+        "apt": {
+          "install_args": [
+            "ldap-utils",
+            "python3-dev",
+            "python3-setuptools",
+            "libssl-dev",
+            "libldap2-dev",
+            "libsasl2-dev",
+            "zlib1g-dev",
+            "libatomic1"
+          ],
+          "version": [
+            "ldapsearch",
+            "-VV"
+          ]
+        }
+      }
+    }
+  ],
+  "postgres_binaries": [
+    {
+      "name": "{INITDB_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "postgresql"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{PG_CTL_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "postgresql"
+          ]
+        }
+      }
+    }
+  ]
+}

.github/configs/ci-macos-brew.json

@@ -0,0 +1,13 @@
+{
+  "properties": {
+    "BREW_BINARY": {
+      "default": "brew"
+    }
+  },
+  "required_binaries": [
+    {
+      "name": "{BREW_BINARY}",
+      "binproviders": "env"
+    }
+  ]
+}

.github/configs/ci-macos-build.json

@@ -0,0 +1,44 @@
+{
+  "properties": {
+    "CLANG_BINARY": {
+      "default": "clang"
+    },
+    "LDAPVC_BINARY": {
+      "default": "ldapvc"
+    }
+  },
+  "required_binaries": [
+    {
+      "name": "{CLANG_BINARY}",
+      "binproviders": "env,brew",
+      "overrides": {
+        "brew": {
+          "install_args": [
+            "llvm"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{LDAPVC_BINARY}",
+      "binproviders": "env,brew",
+      "overrides": {
+        "env": {
+          "version": [
+            "ldapvc",
+            "-VV"
+          ]
+        },
+        "brew": {
+          "install_args": [
+            "openldap"
+          ],
+          "version": [
+            "ldapvc",
+            "-VV"
+          ]
+        }
+      }
+    }
+  ]
+}

.github/configs/ci-macos-openldap-env.json

@@ -0,0 +1,21 @@
+{
+  "properties": {
+    "LDAPVC_BINARY": {
+      "default": "ldapvc"
+    }
+  },
+  "required_binaries": [
+    {
+      "name": "{LDAPVC_BINARY}",
+      "binproviders": "env",
+      "overrides": {
+        "env": {
+          "version": [
+            "ldapvc",
+            "-VV"
+          ]
+        }
+      }
+    }
+  ]
+}

.github/configs/ci-tooling.json

@@ -0,0 +1,341 @@
+{
+  "properties": {
+    "CURL_BINARY": {
+      "default": "curl"
+    },
+    "BASH_BINARY": {
+      "default": "bash"
+    },
+    "CHMOD_BINARY": {
+      "default": "chmod"
+    },
+    "DOCKER_BINARY": {
+      "default": "docker"
+    },
+    "GIT_BINARY": {
+      "default": "git"
+    },
+    "GREP_BINARY": {
+      "default": "grep"
+    },
+    "GH_BINARY": {
+      "default": "gh"
+    },
+    "EXPORTFS_BINARY": {
+      "default": "exportfs"
+    },
+    "JQ_BINARY": {
+      "default": "jq"
+    },
+    "MKDIR_BINARY": {
+      "default": "mkdir"
+    },
+    "MOUNT_CIFS_BINARY": {
+      "default": "mount.cifs"
+    },
+    "MOUNT_NFS_BINARY": {
+      "default": "mount.nfs"
+    },
+    "NODE_BINARY": {
+      "default": "node"
+    },
+    "PS_BINARY": {
+      "default": "ps"
+    },
+    "PYTHON_BINARY": {
+      "default": "python"
+    },
+    "PV_BINARY": {
+      "default": "pv"
+    },
+    "SSH_BINARY": {
+      "default": "ssh"
+    },
+    "SED_BINARY": {
+      "default": "sed"
+    },
+    "SMBD_BINARY": {
+      "default": "smbd"
+    },
+    "SUDO_BINARY": {
+      "default": "sudo"
+    },
+    "SYSTEMCTL_BINARY": {
+      "default": "systemctl"
+    },
+    "TAR_BINARY": {
+      "default": "tar"
+    },
+    "TREE_BINARY": {
+      "default": "tree"
+    },
+    "TR_BINARY": {
+      "default": "tr"
+    },
+    "RM_BINARY": {
+      "default": "rm"
+    },
+    "UMOUNT_BINARY": {
+      "default": "umount"
+    },
+    "UV_BINARY": {
+      "default": "uv"
+    },
+    "UNAME_BINARY": {
+      "default": "uname"
+    }
+  },
+  "ci_binaries": [
+    {
+      "name": "{BASH_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{PYTHON_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{JQ_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{UV_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "bsd_binaries": [
+    {
+      "name": "{BASH_BINARY}",
+      "binproviders": "env,pyinfra"
+    }
+  ],
+  "freebsd_binaries": [
+    {
+      "name": "{SUDO_BINARY}",
+      "binproviders": "env,pyinfra"
+    }
+  ],
+  "docker_binaries": [
+    {
+      "name": "{DOCKER_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{GIT_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{PYTHON_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{UNAME_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "deploy_binaries": [
+    {
+      "name": "{GIT_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{PYTHON_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{SSH_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "docker_debug_binaries": [
+    {
+      "name": "{DOCKER_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{PV_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{TAR_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{TREE_BINARY}",
+      "binproviders": "env,brew,apt"
+    }
+  ],
+  "docker_validation_binaries": [
+    {
+      "name": "{DOCKER_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{GREP_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{MKDIR_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{RM_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{SED_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{SSH_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{TR_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{UNAME_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "mount_validation_binaries": [
+    {
+      "name": "{BASH_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{CHMOD_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{EXPORTFS_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "nfs-kernel-server"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{MKDIR_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{MOUNT_CIFS_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "cifs-utils"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{MOUNT_NFS_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "nfs-common"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{SMBD_BINARY}",
+      "binproviders": "env,apt",
+      "overrides": {
+        "apt": {
+          "install_args": [
+            "samba"
+          ]
+        }
+      }
+    },
+    {
+      "name": "{SUDO_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{SYSTEMCTL_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{UMOUNT_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "process_binaries": [
+    {
+      "name": "{PS_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "docs_binaries": [
+    {
+      "name": "{CURL_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{SUDO_BINARY}",
+      "binproviders": "env"
+    }
+  ],
+  "git_binaries": [
+    {
+      "name": "{GIT_BINARY}",
+      "binproviders": "env,brew,apt"
+    }
+  ],
+  "github_binaries": [
+    {
+      "name": "{GH_BINARY}",
+      "binproviders": "env,brew,apt"
+    }
+  ],
+  "node_binaries": [
+    {
+      "name": "{NODE_BINARY}",
+      "binproviders": "env,npm",
+      "min_version": "22.23.1",
+      "overrides": {
+        "npm": {
+          "install_args": [
+            "node@22.23.1"
+          ],
+          "postinstall_scripts": true
+        }
+      }
+    }
+  ],
+  "release_binaries": [
+    {
+      "name": "{UV_BINARY}",
+      "binproviders": "env,pip,brew"
+    },
+    {
+      "name": "{PYTHON_BINARY}",
+      "binproviders": "env"
+    },
+    {
+      "name": "{GH_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{GIT_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{JQ_BINARY}",
+      "binproviders": "env,brew,apt"
+    },
+    {
+      "name": "{CURL_BINARY}",
+      "binproviders": "env,brew,apt"
+    }
+  ]
+}

.github/scripts/clone_abx_repo.sh

@@ -1,37 +0,0 @@
-#!/usr/bin/env bash
-set -Eeuo pipefail
-
-repo_name="$1"
-target_dir="${2:-$repo_name}"
-
-case "$repo_name" in
-    abxpkg)
-        version="1.11.229"
-        ;;
-    abx-plugins)
-        version="1.11.236"
-        ;;
-    *)
-version="$(
-python3 - "$repo_name" <<'PY'
-import re
-import sys
-from pathlib import Path
-
-repo_name = sys.argv[1]
-lock_text = Path("uv.lock").read_text()
-match = re.search(
-    rf'^\[\[package\]\]\s*\nname = "{re.escape(repo_name)}"\s*\nversion = "([^"]+)"',
-    lock_text,
-    re.MULTILINE,
-)
-if not match:
-    raise SystemExit(f"Could not find {repo_name} in uv.lock")
-print(match.group(1))
-PY
-)"
-        ;;
-esac
-
-echo "Cloning ArchiveBox/${repo_name}@v${version} into ${target_dir}"
-git clone --depth=1 --branch "v${version}" "https://github.com/ArchiveBox/${repo_name}.git" "$target_dir"

.github/scripts/discover_test_matrix.py

@@ -0,0 +1,39 @@
+#!/usr/bin/env python3
+"""Discover every ArchiveBox test file and expose it as one CI job."""
+
+import json
+from pathlib import Path
+
+
+def main() -> None:
+    root = Path.cwd().resolve()
+    archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))
+
+    if not archivebox_tests:
+        raise SystemExit("No ArchiveBox tests discovered")
+
+    matrix: list[dict[str, object]] = []
+    for path in archivebox_tests:
+        test_path = path.relative_to(root).as_posix()
+        matrix.append(
+            {
+                "name": f"main/{path.stem}",
+                "paths": [test_path],
+                "paths_arg": test_path,
+                "extra": "ldap" if test_path.endswith("/test_auth_ldap.py") else "",
+                "count": 1,
+            },
+        )
+
+    discovered_paths = [path for entry in matrix for path in entry["paths"]]
+    if len(discovered_paths) != len(set(discovered_paths)):
+        raise SystemExit("Tests were not discovered exactly once")
+    if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
+        raise SystemExit("Discovered test shard coverage does not match test files")
+
+    print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
+    print(json.dumps(matrix, separators=(",", ":")))
+
+
+if __name__ == "__main__":
+    main()

.github/workflows/ci.yml

@@ -0,0 +1,92 @@
+name: CI
+
+on:
+  pull_request:
+    branches: [dev, main]
+  push:
+    branches: [dev, main]
+  workflow_dispatch:
+
+permissions:
+  actions: read
+  contents: read
+  packages: write
+  security-events: write
+
+concurrency:
+  group: ci-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
+
+jobs:
+  lint:
+    name: Linters
+    uses: ./.github/workflows/lint.yml
+    secrets: inherit
+
+  install-cli-platform:
+    name: Install and CLI platform compatibility
+    uses: ./.github/workflows/test.yml
+    secrets: inherit
+
+  tests:
+    name: Discovered test matrix
+    uses: ./.github/workflows/test-parallel.yml
+    secrets: inherit
+
+  documentation:
+    name: Documentation and root tests
+    uses: ./.github/workflows/docs.yml
+    secrets: inherit
+
+  codeql:
+    name: CodeQL
+    uses: ./.github/workflows/codeql.yml
+    secrets: inherit
+
+  python-artifacts:
+    name: Tested Python artifacts
+    uses: ./.github/workflows/pip.yml
+    secrets: inherit
+
+  docker-digests:
+    name: Tested Docker images
+    uses: ./.github/workflows/docker.yml
+    with:
+      push_digests: ${{ github.event_name == 'push' }}
+    secrets: inherit
+
+  required:
+    name: All required CI lanes
+    if: ${{ always() }}
+    needs:
+      - lint
+      - install-cli-platform
+      - tests
+      - documentation
+      - codeql
+      - python-artifacts
+      - docker-digests
+    runs-on: ubuntu-24.04
+    steps:
+      - name: Verify every required lane succeeded
+        env:
+          LINT_RESULT: ${{ needs.lint.result }}
+          INSTALL_CLI_PLATFORM_RESULT: ${{ needs.install-cli-platform.result }}
+          TESTS_RESULT: ${{ needs.tests.result }}
+          DOCUMENTATION_RESULT: ${{ needs.documentation.result }}
+          CODEQL_RESULT: ${{ needs.codeql.result }}
+          PYTHON_ARTIFACTS_RESULT: ${{ needs.python-artifacts.result }}
+          DOCKER_DIGESTS_RESULT: ${{ needs.docker-digests.result }}
+        run: |
+          for result in \
+            "$LINT_RESULT" \
+            "$INSTALL_CLI_PLATFORM_RESULT" \
+            "$TESTS_RESULT" \
+            "$DOCUMENTATION_RESULT" \
+            "$CODEQL_RESULT" \
+            "$PYTHON_ARTIFACTS_RESULT" \
+            "$DOCKER_DIGESTS_RESULT"
+          do
+            [[ "$result" == success ]]
+          done
+          echo 'All required CI lanes and tested release artifacts succeeded.'

.github/workflows/claude.yml

@@ -17,33 +17,24 @@ jobs:
       (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
       (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
       (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     permissions:
       contents: read
       pull-requests: write
       issues: write
       id-token: write
-      actions: read # Required for Claude to read CI results on PRs
+      actions: read
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 1
 
       - name: Run Claude Code
         id: claude
-        uses: anthropics/claude-code-action@v1
+        uses: anthropics/claude-code-action@b76a0776ae74036e77cd11018083743453d7ad35 # v1
         with:
           anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-
-          # This is an optional setting that allows Claude to read CI results on PRs
           additional_permissions: |
             actions: read
-
-          # Optional: Give a custom prompt to Claude. If this is not specified, Claude will perform the instructions specified in the comment that tagged it.
-          # prompt: 'Update the pull request description to include a summary of changes.'
-
-          # Optional: Add claude_args to customize behavior and configuration
-          # See https://github.com/anthropics/claude-code-action/blob/main/docs/usage.md
-          # or https://code.claude.com/docs/en/cli-reference for available options
           claude_args: '--allowed-tools Bash(gh pr:*)'

.github/workflows/codeql.yml

@@ -1,46 +1,21 @@
-# For most projects, this workflow file will not need changing; you simply need
-# to commit it to your repository.
-#
-# You may wish to alter this file to override the set of languages analyzed,
-# or to provide custom queries or build logic.
-#
-# ******** NOTE ********
-# We have attempted to detect the languages in your repository. Please check
-# the `language` matrix defined below to confirm you have the correct set of
-# supported CodeQL languages.
-#
 name: "CodeQL"
 
 on:
-  push:
-    branches: [ "dev" ]
-  pull_request:
-    branches: [ "dev" ]
+  workflow_call:
   schedule:
     - cron: '33 17 * * 6'
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  group: codeql-${{ github.repository }}-${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}
   cancel-in-progress: true
 
 jobs:
   analyze:
     name: Analyze (${{ matrix.language }})
-    # Runner size impacts CodeQL analysis time. To learn more, please see:
-    #   - https://gh.io/recommended-hardware-resources-for-running-codeql
-    #   - https://gh.io/supported-runners-and-hardware-resources
-    #   - https://gh.io/using-larger-runners (GitHub.com only)
-    # Consider using larger runners or machines with greater resources for possible analysis time improvements.
-    runs-on: ${{ (matrix.language == 'swift' && 'macos-latest') || 'ubuntu-latest' }}
-    timeout-minutes: ${{ (matrix.language == 'swift' && 120) || 360 }}
+    runs-on: ubuntu-24.04
     permissions:
-      # required for all workflows
       security-events: write
-
-      # required to fetch internal or private CodeQL packs
       packages: read
-
-      # only required for workflows in private repositories
       actions: read
       contents: read
 
@@ -50,47 +25,18 @@ jobs:
         include:
         - language: python
           build-mode: none
-        # CodeQL supports the following values keywords for 'language': 'c-cpp', 'csharp', 'go', 'java-kotlin', 'javascript-typescript', 'python', 'ruby', 'swift'
-        # Use `c-cpp` to analyze code written in C, C++ or both
-        # Use 'java-kotlin' to analyze code written in Java, Kotlin or both
-        # Use 'javascript-typescript' to analyze code written in JavaScript, TypeScript or both
-        # To learn more about changing the languages that are analyzed or customizing the build mode for your analysis,
-        # see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/customizing-your-advanced-setup-for-code-scanning.
-        # If you are analyzing a compiled language, you can modify the 'build-mode' for that language to customize how
-        # your codebase is analyzed, see https://docs.github.com/en/code-security/code-scanning/creating-an-advanced-setup-for-code-scanning/codeql-code-scanning-for-compiled-languages
+        - language: javascript-typescript
+          build-mode: none
     steps:
     - name: Checkout repository
-      uses: actions/checkout@v4
+      uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
 
-    # Initializes the CodeQL tools for scanning.
     - name: Initialize CodeQL
-      uses: github/codeql-action/init@v3
+      uses: github/codeql-action/init@b7351df727350dca84cb9d725d57dcf5bc82ba26 # v3
       with:
         languages: ${{ matrix.language }}
         build-mode: ${{ matrix.build-mode }}
-        # If you wish to specify custom queries, you can do so here or in a config file.
-        # By default, queries listed here will override any specified in a config file.
-        # Prefix the list here with "+" to use these queries and those in the config file.
-
-        # For more details on CodeQL's query packs, refer to: https://docs.github.com/en/code-security/code-scanning/automatically-scanning-your-code-for-vulnerabilities-and-errors/configuring-code-scanning#using-queries-in-ql-packs
-        # queries: security-extended,security-and-quality
-
-    # If the analyze step fails for one of the languages you are analyzing with
-    # "We were unable to automatically build your code", modify the matrix above
-    # to set the build mode to "manual" for that language. Then modify this step
-    # to build your code.
-    # ℹ️ Command-line programs to run using the OS shell.
-    # 📚 See https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#jobsjob_idstepsrun
-    - if: matrix.build-mode == 'manual'
-      run: |
-        echo 'If you are using a "manual" build mode for one or more of the' \
-          'languages you are analyzing, replace this with the commands to build' \
-          'your code, for example:'
-        echo '  make bootstrap'
-        echo '  make release'
-        exit 1
-
     - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@v3
+      uses: github/codeql-action/analyze@b7351df727350dca84cb9d725d57dcf5bc82ba26 # v3
       with:
         category: "/language:${{matrix.language}}"

.github/workflows/deploy-publicsite.yml

@@ -6,6 +6,13 @@ on:
       - dev
     paths:
       - publicsite/**
+      - archivebox/**
+      - bin/collect_ui_screenshots.*
+      - bin/generate_ui_screenshot_gallery.py
+      - bin/take_screenshot.js
+      - docs/Screenshots.md
+      - pyproject.toml
+      - uv.lock
       - .github/workflows/deploy-publicsite.yml
   workflow_dispatch:
 
@@ -15,29 +22,42 @@ permissions:
   id-token: write
 
 concurrency:
-  group: github-pages-publicsite
-  cancel-in-progress: true
+  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
+  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
 
 jobs:
   deploy:
     environment:
       name: github-pages
       url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 1
 
+      - name: Setup Python and uv
+        uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: false
+
+      - name: Install ArchiveBox and screenshot dependencies
+        run: uv sync --no-cache --frozen
+
+      - name: Build UI screenshot gallery
+        run: bin/collect_ui_screenshots.sh
+        env:
+          BASE_URL: http://archivebox.localhost:8000
+
       - name: Setup Pages
-        uses: actions/configure-pages@v5
+        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
 
       - name: Upload artifact
-        uses: actions/upload-pages-artifact@v3
+        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
         with:
           path: ./publicsite
 
       - name: Deploy to GitHub Pages
         id: deployment
-        uses: actions/deploy-pages@v4
+        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4

.github/workflows/docker.yml

@@ -1,27 +1,23 @@
 name: Build Docker image
 
 on:
-  workflow_dispatch:
   workflow_call:
-  push:
-    branches:
-      - "**"
-    tags:
-      - 'v*'
-  # pull_request:
+    inputs:
+      push_digests:
+        description: Push validated per-platform images by digest for the release workflow
+        required: false
+        default: false
+        type: boolean
 
 env:
   DOCKERHUB_IMAGE: archivebox/archivebox
   GHCR_IMAGE: ghcr.io/archivebox/archivebox
 
 permissions:
+  actions: read
   contents: read
   packages: write
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
 jobs:
   build:
     name: build ${{ matrix.platform }}
@@ -34,63 +30,81 @@ jobs:
             runner: ubuntu-24.04
             cache_scope: docker-amd64
             artifact_name: digest-linux-amd64
+            local_tag: archivebox-ci:amd64
           - platform: linux/arm64
             runner: ubuntu-24.04-arm
             cache_scope: docker-arm64
             artifact_name: digest-linux-arm64
+            local_tag: archivebox-ci:arm64
     steps:
       - name: Checkout
-        uses: actions/checkout@v4
-        # with:
-        #   submodules: true
-        #   fetch-depth: 1
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
 
-      - name: Wait for released ArchiveBox deps on PyPI
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
+
+      - name: Resolve Docker through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
+        shell: bash
         run: |
-          python3 - <<'PY'
-          import json
-          import re
-          import sys
-          import time
+          set -Eeuo pipefail
+          ABXPKG_VERSION="$(uv run --no-cache --no-project python - <<'PY'
           import tomllib
-          import urllib.request
+          from pathlib import Path
 
-          watched = {"abxbus", "abxpkg", "abx-plugins", "abx-dl"}
-          deps = tomllib.loads(open("pyproject.toml", "rb").read().decode())["project"]["dependencies"]
-          required = {}
-          for dep in deps:
-              for name in watched:
-                  match = re.match(rf"{re.escape(name)}\s*(==|>=)\s*([^,;\s]+)", dep)
-                  if match:
-                      required[name] = match.group(2)
-
-          deadline = time.monotonic() + 300
-          missing = required.copy()
-          while missing and time.monotonic() < deadline:
-              for name, version in list(missing.items()):
-                  try:
-                      with urllib.request.urlopen(f"https://pypi.org/pypi/{name}/{version}/json", timeout=20):
-                          available = True
-                  except Exception:
-                      available = False
-                  if available:
-                      print(f"{name} {version} is available on PyPI")
-                      missing.pop(name)
-                  else:
-                      print(f"{name} {version} is not available on PyPI yet")
-              if missing:
-                  time.sleep(10)
-
-          if missing:
-              print(f"Missing PyPI releases after wait: {missing}", file=sys.stderr)
-              sys.exit(1)
+          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
+          matches = [package["version"] for package in packages if package["name"] == "abxpkg"]
+          if len(matches) != 1:
+              raise SystemExit(f"Expected one locked abxpkg package, found {len(matches)}")
+          print(matches[0])
           PY
+          )"
+          test -n "$ABXPKG_VERSION"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
+          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
+          echo "ABXPKG_VERSION=$ABXPKG_VERSION" >> "$GITHUB_ENV"
+          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
+
+          docker_env="$(
+            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+              --install \
+              --json \
+              --lib="$ABXPKG_LIB_DIR" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries"
+          )"
+          JQ_BINARY="$ABXPKG_LIB_DIR/env/bin/jq"
+          "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
+            <<< "$docker_env" >> "$GITHUB_ENV"
+
+          docker_binary="$ABXPKG_LIB_DIR/env/bin/docker"
+          UV_BINARY="$ABXPKG_LIB_DIR/env/bin/uv"
+          {
+            echo "DOCKER_BINARY=$docker_binary"
+            echo "JQ_BINARY=$JQ_BINARY"
+            echo "UV_BINARY=$UV_BINARY"
+          } >> "$GITHUB_ENV"
+          for binary in "$docker_binary" "$JQ_BINARY" "$UV_BINARY"; do
+            test -L "$binary"
+            test -x "$binary"
+          done
+          "$docker_binary" --version
 
       - name: Set up Docker Buildx
         id: buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
         with:
-          version: latest
           install: true
 
       - name: Builder instance name
@@ -99,50 +113,50 @@ jobs:
       - name: Available platforms
         run: echo ${{ steps.buildx.outputs.platforms }}
 
-      - name: Wait for published abx-dl image
+      - name: Verify published abx-dl image
         id: abx_dl_image
         shell: bash
         run: |
           set -Eeuo pipefail
-          ABX_DL_VERSION="$(python3 - <<'PY'
-          import re
+          ABX_DL_VERSION="$("$UV_BINARY" run --no-cache --no-project python - <<'PY'
           import tomllib
+          from pathlib import Path
 
-          with open("pyproject.toml", "rb") as f:
-              deps = tomllib.load(f)["project"]["dependencies"]
-
-          for dep in deps:
-              match = re.match(r"abx-dl\s*(?:==|>=)\s*([^,;\s]+)", dep)
-              if match:
-                  print(match.group(1))
-                  break
-          else:
-              raise SystemExit("Missing abx-dl dependency in pyproject.toml")
+          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
+          matches = [package["version"] for package in packages if package["name"] == "abx-dl"]
+          if len(matches) != 1:
+              raise SystemExit(f"Expected one locked abx-dl package, found {len(matches)}")
+          print(matches[0])
           PY
           )"
-          ABX_DL_IMAGE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
-          deadline=$((SECONDS + 1800))
-          until docker buildx imagetools inspect "${ABX_DL_IMAGE}" >/tmp/abx-dl-image.json; do
-            if (( SECONDS >= deadline )); then
-              echo "Timed out waiting for published ${ABX_DL_IMAGE}" >&2
-              exit 1
-            fi
-            echo "${ABX_DL_IMAGE} is not published yet; waiting..."
-            sleep 30
-          done
+          test -n "$ABX_DL_VERSION"
+          ABX_DL_REFERENCE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
+          INSPECT_OUTPUT="$("$DOCKER_BINARY" buildx imagetools inspect "$ABX_DL_REFERENCE")"
+          printf '%s\n' "$INSPECT_OUTPUT"
+          ABX_DL_DIGEST="$(INSPECT_OUTPUT="$INSPECT_OUTPUT" "$UV_BINARY" run --no-cache --no-project python - <<'PY'
+          import os
 
+          digest = next(
+              (fields[1] for line in os.environ["INSPECT_OUTPUT"].splitlines() if (fields := line.split()) and fields[0] == "Digest:"),
+              "",
+          )
+          print(digest)
+          PY
+          )"
+          [[ "$ABX_DL_DIGEST" == sha256:* ]]
+          ABX_DL_IMAGE="${ABX_DL_REFERENCE%%@*}@${ABX_DL_DIGEST}"
           echo "image=${ABX_DL_IMAGE}" >> "$GITHUB_OUTPUT"
-          docker buildx imagetools inspect "${ABX_DL_IMAGE}"
 
       - name: Login to Docker Hub
-        uses: docker/login-action@v3
-        if: github.event_name != 'pull_request'
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        if: inputs.push_digests
         with:
           username: ${{ secrets.DOCKER_USERNAME }}
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        if: inputs.push_digests
         with:
           registry: ghcr.io
           username: ${{ github.actor }}
@@ -153,13 +167,14 @@ jobs:
         shell: bash
         run: |
           set -Eeuo pipefail
-          VERSION="$(python3 - <<'PY'
+          VERSION="$("$UV_BINARY" run --no-cache --no-project python - <<'PY'
           import tomllib
+          from pathlib import Path
 
-          with open("pyproject.toml", "rb") as f:
-              print(tomllib.load(f)["project"]["version"])
+          print(tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"])
           PY
           )"
+          test -n "$VERSION"
 
           {
             echo 'labels<<EOF'
@@ -175,59 +190,271 @@ jobs:
 
       - name: Build and push digest
         id: docker_build
-        uses: docker/build-push-action@v6
+        if: inputs.push_digests
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
         with:
           context: ./
           file: ./Dockerfile
           builder: ${{ steps.buildx.outputs.name }}
-          push: ${{ github.event_name != 'pull_request' }}
-          tags: |
-            ${{ env.DOCKERHUB_IMAGE }}
-            ${{ env.GHCR_IMAGE }}
           labels: ${{ steps.docker_meta.outputs.labels }}
           build-args: |
             ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
             ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
-          cache-from: type=gha,scope=${{ matrix.cache_scope }}
-          cache-to: type=gha,mode=max,scope=${{ matrix.cache_scope }}
           pull: true
           platforms: ${{ matrix.platform }}
-          outputs: type=image,push-by-digest=true,name-canonical=true,push=true
+          outputs: |
+            type=docker,name=${{ matrix.local_tag }}
+            type=oci,dest=${{ runner.temp }}/archivebox-image.tar
+            type=image,"name=${{ env.DOCKERHUB_IMAGE }},${{ env.GHCR_IMAGE }}",push-by-digest=true,name-canonical=true,push=true
+
+      - name: Build pull request image
+        if: ${{ !inputs.push_digests }}
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
+        with:
+          context: ./
+          file: ./Dockerfile
+          builder: ${{ steps.buildx.outputs.name }}
+          load: true
+          tags: ${{ matrix.local_tag }}
+          labels: ${{ steps.docker_meta.outputs.labels }}
+          build-args: |
+            ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
+            ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
+          pull: true
+          platforms: ${{ matrix.platform }}
 
       - name: Image digest
+        if: inputs.push_digests
         run: echo ${{ steps.docker_build.outputs.digest }}
 
-      - name: Validate pushed image version, commit, and size
+      - name: Validate exact built image version and commit
         shell: bash
         run: |
           set -Eeuo pipefail
-          IMAGE="${DOCKERHUB_IMAGE}@${{ steps.docker_build.outputs.digest }}"
+          IMAGE="${{ matrix.local_tag }}"
           SHORT_SHA="${GITHUB_SHA::7}"
 
           DATA_DIR="$(mktemp -d)"
-          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
-          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox version | tee /tmp/archivebox-version.out
-          docker run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox status
-          docker run --rm --platform '${{ matrix.platform }}' --entrypoint cat "$IMAGE" /VERSION.txt > /tmp/archivebox-image-version.txt
-          grep -F "ArchiveBox v${{ steps.docker_meta.outputs.version }}" /tmp/archivebox-version.out
-          grep -F "COMMIT_HASH=${SHORT_SHA}" /tmp/archivebox-version.out
-          grep -Fx "COMMIT_HASH=${GITHUB_SHA}" /tmp/archivebox-image-version.txt
-
-          python3 - "$IMAGE" <<'PY'
-          import json
-          import subprocess
+          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
+          ARCHIVEBOX_VERSION_OUTPUT="$("$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox version)"
+          printf '%s\n' "$ARCHIVEBOX_VERSION_OUTPUT"
+          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox status
+          IMAGE_VERSION_OUTPUT="$("$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' --entrypoint cat "$IMAGE" /VERSION.txt)"
+          ARCHIVEBOX_VERSION_OUTPUT="$ARCHIVEBOX_VERSION_OUTPUT" IMAGE_VERSION_OUTPUT="$IMAGE_VERSION_OUTPUT" \
+            "$UV_BINARY" run --no-cache --no-project python - \
+            "ArchiveBox v${{ steps.docker_meta.outputs.version }}" \
+            "COMMIT_HASH=${SHORT_SHA}" \
+            "COMMIT_HASH=${GITHUB_SHA}" <<'PY'
+          import os
           import sys
 
-          image = sys.argv[1]
-          limit = 780 * 1024 * 1024
-          manifest = json.loads(subprocess.check_output(["docker", "manifest", "inspect", image]))
-          total = manifest.get("config", {}).get("size", 0) + sum(layer.get("size", 0) for layer in manifest.get("layers", []))
-          print(f"{image} compressed_size={total / 1024 / 1024:.2f} MiB limit={limit / 1024 / 1024:.2f} MiB")
+          version_output = os.environ["ARCHIVEBOX_VERSION_OUTPUT"]
+          image_version_lines = os.environ["IMAGE_VERSION_OUTPUT"].splitlines()
+          if sys.argv[1] not in version_output:
+              raise SystemExit(f"Missing version marker: {sys.argv[1]}")
+          if sys.argv[2] not in version_output:
+              raise SystemExit(f"Missing short commit marker: {sys.argv[2]}")
+          if sys.argv[3] not in image_version_lines:
+              raise SystemExit(f"Missing exact commit marker: {sys.argv[3]}")
+          PY
+
+      - name: Resolve real Docker CLI test dependencies through abxpkg
+        if: matrix.platform == 'linux/amd64'
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          mount_env="$(
+            uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+              --install \
+              --json \
+              --lib="$ABXPKG_LIB_DIR" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:mount_validation_binaries"
+          )"
+          "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
+            <<< "$mount_env" >> "$GITHUB_ENV"
+
+          for binary_name in bash chmod curl exportfs mkdir mount.cifs mount.nfs smbd sudo systemctl umount; do
+            binary="$ABXPKG_LIB_DIR/env/bin/$binary_name"
+            test -L "$binary"
+            test -x "$binary"
+          done
+
+      - name: Validate Docker UID and GID behavior on real NFS and SMB mounts
+        if: matrix.platform == 'linux/amd64'
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          NFS_SOURCE="/srv/archivebox-ci-nfs"
+          SMB_SOURCE="/srv/archivebox-ci-smb"
+          NFS_MOUNT="$RUNNER_TEMP/archivebox-nfs-mount"
+          SMB_MOUNT="$RUNNER_TEMP/archivebox-smb-mount"
+          mounted_nfs=0
+          mounted_smb=0
+
+          cleanup() {
+            local original_status=$? cleanup_status=0
+            trap - EXIT
+            if [[ "$mounted_smb" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$SMB_MOUNT"; then
+              cleanup_status=1
+            fi
+            if [[ "$mounted_nfs" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$NFS_MOUNT"; then
+              cleanup_status=1
+            fi
+            if ! "$SUDO_BINARY" "$SYSTEMCTL_BINARY" stop smbd.service nfs-kernel-server.service; then
+              cleanup_status=1
+            fi
+            if [[ "$original_status" != "0" ]]; then
+              exit "$original_status"
+            fi
+            exit "$cleanup_status"
+          }
+          trap cleanup EXIT
+
+          "$SUDO_BINARY" "$MKDIR_BINARY" -p "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
+          "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
+          "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
+          "$SUDO_BINARY" "$BASH_BINARY" -c \
+            "printf '%s *(rw,sync,no_subtree_check,no_root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
+          "$SUDO_BINARY" "$BASH_BINARY" -c \
+            "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
+
+          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart nfs-kernel-server.service
+          "$SUDO_BINARY" "$EXPORTFS_BINARY" -ra
+          "$SYSTEMCTL_BINARY" is-active --quiet nfs-kernel-server.service
+          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart smbd.service
+          "$SYSTEMCTL_BINARY" is-active --quiet smbd.service
+
+          "$SUDO_BINARY" "$MOUNT_NFS_BINARY" -o vers=3,nolock "127.0.0.1:$NFS_SOURCE" "$NFS_MOUNT"
+          mounted_nfs=1
+          "$SUDO_BINARY" "$MOUNT_CIFS_BINARY" //127.0.0.1/archivebox-ci "$SMB_MOUNT" \
+            -o "guest,vers=3.0,uid=0,gid=0,forceuid,forcegid,file_mode=0777,dir_mode=0777,nounix"
+          mounted_smb=1
+
+          NFS_TEST_DIR="$NFS_MOUNT" \
+          SMB_TEST_DIR="$SMB_MOUNT" \
+          IMAGE="${{ matrix.local_tag }}" \
+            "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
+
+      - name: Validate Docker CLI add/list and compose behavior
+        if: matrix.platform == 'linux/amd64'
+        shell: bash
+        env:
+          ARCHIVEBOX_IMAGE: ${{ matrix.local_tag }}
+        run: |
+          set -Eeuo pipefail
+          "$MKDIR_BINARY" "${{ github.workspace }}/data"
+          "$DOCKER_BINARY" run -v "${{ github.workspace }}/data":/data "$ARCHIVEBOX_IMAGE" init
+
+          "$SUDO_BINARY" "$BASH_BINARY" -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
+          "$DOCKER_BINARY" run --name www-nginx -p 80:80 -d nginx:1.29.5@sha256:0236ee02dcbce00b9bd83e0f5fbc51069e7e1161bd59d99885b3ae1734f3392e
+
+          "$DOCKER_BINARY" run -v "$PWD"/data:/data --network host "$ARCHIVEBOX_IMAGE" add http://www.test-nginx-1.local
+          echo "http://www.test-nginx-2.local" | "$DOCKER_BINARY" run -i --network host -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" add
+
+          archive_list="$("$DOCKER_BINARY" run -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" list)"
+          [[ "$archive_list" == *"www.test-nginx-1.local"* ]] || { echo "The site 1 isn't in the list"; exit 1; }
+          [[ "$archive_list" == *"www.test-nginx-2.local"* ]] || { echo "The site 2 isn't in the list"; exit 1; }
+
+          "$DOCKER_BINARY" compose run --rm archivebox init
+          "$DOCKER_BINARY" compose up -d --wait
+          "$DOCKER_BINARY" network connect archivebox_default www-nginx
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000')"
+          admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
+          [[ "$index_html" == *ArchiveBox* ]]
+          [[ "$admin_js" == *window.django* ]]
+
+          "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
+          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000')"
+          [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
+          "$DOCKER_BINARY" compose down
+
+      - name: Validate compressed image size
+        if: inputs.push_digests
+        env:
+          BUILD_METADATA: ${{ steps.docker_build.outputs.metadata }}
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          LOCAL_IMAGE_ID="$("$DOCKER_BINARY" image inspect '${{ matrix.local_tag }}' --format '{{.Id}}')"
+          "$UV_BINARY" run --no-cache --no-project python - \
+            '${{ runner.temp }}/archivebox-image.tar' \
+            '${{ steps.docker_build.outputs.digest }}' \
+            "$LOCAL_IMAGE_ID" \
+            "$((780 * 1024 * 1024))" <<'PY'
+          from hashlib import sha256
+          import json
+          import os
+          import sys
+          import tarfile
+
+          archive_path, pushed_digest, local_image_id, limit_text = sys.argv[1:]
+          limit = int(limit_text)
+          metadata = json.loads(os.environ["BUILD_METADATA"])
+          if metadata["containerimage.digest"] != pushed_digest:
+              raise SystemExit("Build push transaction digest does not match the action output")
+          if not pushed_digest.startswith("sha256:") or len(pushed_digest) != 71:
+              raise SystemExit(f"Invalid pushed digest: {pushed_digest}")
+
+          with tarfile.open(archive_path) as archive:
+              def read_blob(descriptor):
+                  algorithm, digest = descriptor["digest"].split(":", 1)
+                  if algorithm != "sha256":
+                      raise SystemExit(f"Unsupported OCI digest: {descriptor['digest']}")
+                  member = archive.extractfile(f"blobs/sha256/{digest}")
+                  if member is None:
+                      raise SystemExit(f"Missing OCI blob: {descriptor['digest']}")
+                  blob = member.read()
+                  if len(blob) != descriptor["size"]:
+                      raise SystemExit(f"OCI size mismatch: {descriptor['digest']}")
+                  if sha256(blob).hexdigest() != digest:
+                      raise SystemExit(f"OCI digest mismatch: {descriptor['digest']}")
+                  return blob
+
+              index_member = archive.extractfile("index.json")
+              if index_member is None:
+                  raise SystemExit("OCI export is missing index.json")
+              index = json.load(index_member)
+
+              manifests = []
+
+              def collect(descriptor):
+                  document = json.loads(read_blob(descriptor))
+                  media_type = descriptor["mediaType"]
+                  if "image.index" in media_type or "manifest.list" in media_type:
+                      for child in document["manifests"]:
+                          collect(child)
+                  elif "image.manifest" in media_type:
+                      config_type = document["config"]["mediaType"]
+                      if "image.config" in config_type or "container.image" in config_type:
+                          manifests.append(document)
+
+              for root in index["manifests"]:
+                  collect(root)
+              if len(manifests) != 1:
+                  raise SystemExit(f"Expected one runnable image manifest, found {len(manifests)}")
+
+              manifest = manifests[0]
+              config_digest = manifest["config"]["digest"]
+              if metadata["containerimage.config.digest"] != config_digest:
+                  raise SystemExit("OCI config digest does not match the pushed build result")
+              if local_image_id != config_digest:
+                  raise SystemExit("Locally tested image does not match the OCI build result")
+              descriptors = [manifest["config"], *manifest["layers"]]
+              for descriptor in descriptors:
+                  read_blob(descriptor)
+              total = sum(descriptor["size"] for descriptor in descriptors)
+
+          print(
+              f"{pushed_digest} compressed_size={total // 1024 // 1024} MiB "
+              f"limit={limit // 1024 // 1024} MiB",
+          )
           if total > limit:
-              raise SystemExit(f"{image} is over the compressed size limit")
+              raise SystemExit(f"{pushed_digest} is over the compressed size limit")
           PY
 
       - name: Export digest
+        if: inputs.push_digests
         shell: bash
         run: |
           set -Eeuo pipefail
@@ -236,151 +463,10 @@ jobs:
           touch "/tmp/digests/${digest#sha256:}"
 
       - name: Upload digest
-        uses: actions/upload-artifact@v4
+        if: inputs.push_digests
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
         with:
           name: ${{ matrix.artifact_name }}
           path: /tmp/digests/*
           if-no-files-found: error
           retention-days: 1
-
-  publish:
-    name: publish multiarch tags
-    runs-on: ubuntu-24.04
-    needs:
-      - build
-    steps:
-      - name: Checkout
-        uses: actions/checkout@v4
-
-      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
-        with:
-          version: latest
-          install: true
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@v3
-        if: github.event_name != 'pull_request'
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Download digests
-        uses: actions/download-artifact@v4
-        with:
-          path: /tmp/digests
-          pattern: digest-*
-          merge-multiple: true
-
-      - name: Collect Docker tags
-        id: docker_meta
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$(python3 - <<'PY'
-          import tomllib
-
-          with open("pyproject.toml", "rb") as f:
-              print(tomllib.load(f)["project"]["version"])
-          PY
-          )"
-          BRANCH_TAG="$(printf '%s' "${GITHUB_REF_NAME}" | tr -c 'A-Za-z0-9_.-' '-' | sed -E 's/^-+//; s/-+$//; s/-+/-/g' | cut -c1-128)"
-          SHORT_SHA="${GITHUB_SHA::12}"
-          test -n "$BRANCH_TAG"
-          test -n "$SHORT_SHA"
-
-          {
-            echo 'dockerhub_tags<<EOF'
-            if [[ "${GITHUB_REF_NAME}" == "main" ]]; then
-              echo "${DOCKERHUB_IMAGE}:latest"
-            fi
-            echo "${DOCKERHUB_IMAGE}:${BRANCH_TAG}"
-            echo "${DOCKERHUB_IMAGE}:${VERSION}"
-            echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
-            echo 'EOF'
-            echo 'ghcr_tags<<EOF'
-            if [[ "${GITHUB_REF_NAME}" == "main" ]]; then
-              echo "${GHCR_IMAGE}:latest"
-            fi
-            echo "${GHCR_IMAGE}:${BRANCH_TAG}"
-            echo "${GHCR_IMAGE}:${VERSION}"
-            echo "${GHCR_IMAGE}:sha-${SHORT_SHA}"
-            echo 'EOF'
-            echo "version=${VERSION}"
-            echo "branch_tag=${BRANCH_TAG}"
-            echo "short_sha=${SHORT_SHA}"
-          } >> "$GITHUB_OUTPUT"
-
-          echo "[+] Publishing Docker Hub tags:"
-          if [[ "${GITHUB_REF_NAME}" == "main" ]]; then
-            printf '%s\n' "${DOCKERHUB_IMAGE}:latest"
-          fi
-          printf '%s\n' "${DOCKERHUB_IMAGE}:${BRANCH_TAG}" "${DOCKERHUB_IMAGE}:${VERSION}" "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
-          echo "[+] Publishing GHCR tags:"
-          if [[ "${GITHUB_REF_NAME}" == "main" ]]; then
-            printf '%s\n' "${GHCR_IMAGE}:latest"
-          fi
-          printf '%s\n' "${GHCR_IMAGE}:${BRANCH_TAG}" "${GHCR_IMAGE}:${VERSION}" "${GHCR_IMAGE}:sha-${SHORT_SHA}"
-
-      - name: Create Docker Hub manifest
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mapfile -t DIGESTS < <(find /tmp/digests -maxdepth 1 -type f -printf '%f\n' | sort)
-          [[ "${#DIGESTS[@]}" -gt 0 ]]
-
-          TAG_ARGS=()
-          while IFS= read -r tag; do
-            [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag")
-          done <<< "${{ steps.docker_meta.outputs.dockerhub_tags }}"
-
-          REFS=()
-          for digest in "${DIGESTS[@]}"; do
-            REFS+=("${DOCKERHUB_IMAGE}@sha256:${digest}")
-          done
-
-          docker buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
-
-      - name: Create GHCR manifest
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mapfile -t DIGESTS < <(find /tmp/digests -maxdepth 1 -type f -printf '%f\n' | sort)
-          [[ "${#DIGESTS[@]}" -gt 0 ]]
-
-          TAG_ARGS=()
-          while IFS= read -r tag; do
-            [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag")
-          done <<< "${{ steps.docker_meta.outputs.ghcr_tags }}"
-
-          REFS=()
-          for digest in "${DIGESTS[@]}"; do
-            REFS+=("${GHCR_IMAGE}@sha256:${digest}")
-          done
-
-          docker buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
-
-      - name: Inspect published images
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          while IFS= read -r tag; do
-            [[ -n "$tag" ]] && docker buildx imagetools inspect "$tag"
-          done <<< "${{ steps.docker_meta.outputs.dockerhub_tags }}"
-          while IFS= read -r tag; do
-            [[ -n "$tag" ]] && docker buildx imagetools inspect "$tag"
-          done <<< "${{ steps.docker_meta.outputs.ghcr_tags }}"
-
-      - name: Update README
-        uses: peter-evans/dockerhub-description@v4
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-          repository: archivebox/archivebox

.github/workflows/docs.yml

@@ -0,0 +1,156 @@
+name: Documentation tests
+
+on:
+  workflow_call:
+
+env:
+  PYTHONIOENCODING: utf-8
+  USE_COLOR: "False"
+  SHOW_PROGRESS: "False"
+
+jobs:
+  discover:
+    name: Discover documentation examples
+    runs-on: ubuntu-24.04
+    timeout-minutes: 5
+    outputs:
+      matrix: ${{ steps.inventory.outputs.matrix }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          submodules: true
+          fetch-depth: 1
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
+      - name: Install ArchiveBox development dependencies
+        run: uv sync --no-cache --locked --dev
+      - name: Resolve documentation tools through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/archivebox-docs-lib
+        run: |
+          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
+          uv run --no-cache --no-sync abxpkg env \
+            --install \
+            --lib="$ABXPKG_LIB_DIR" \
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
+            --deps-from="$GITHUB_WORKSPACE/docs/mermaid-binary.json:required_binaries" \
+            --deps-from="$GITHUB_WORKSPACE/docs/nginx-binary.json:required_binaries"
+          bash_binary="$ABXPKG_LIB_DIR/env/bin/bash"
+          mmdc_binary="$ABXPKG_LIB_DIR/env/bin/mmdc"
+          nginx_binary="$ABXPKG_LIB_DIR/env/bin/nginx"
+          test -L "$bash_binary"
+          test -L "$mmdc_binary"
+          test -L "$nginx_binary"
+          test -x "$bash_binary"
+          test -x "$mmdc_binary"
+          test -x "$nginx_binary"
+          {
+            echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR"
+            echo "BASH_BINARY=$bash_binary"
+          } >> "$GITHUB_ENV"
+      - id: inventory
+        name: Validate the complete authored documentation inventory
+        run: |
+          matrix="$(uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py matrix)"
+          echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
+
+  examples:
+    name: docs/${{ matrix.name }}
+    needs: discover
+    runs-on: ${{ matrix.runner }}
+    timeout-minutes: 15
+    strategy:
+      fail-fast: false
+      matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          submodules: true
+          fetch-depth: 1
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
+        with:
+          python-version: "3.13"
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
+        with:
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
+      - name: Install ArchiveBox
+        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
+        run: uv sync --no-cache --locked --dev
+      - name: Resolve documentation runtime through abxpkg
+        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/archivebox-docs-lib
+        run: |
+          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
+          dependency_sources=(
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
+          )
+          if [[ '${{ matrix.environment }}' != macos ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries")
+          fi
+          if [[ '${{ matrix.environment }}' == ubuntu ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
+          elif [[ '${{ matrix.environment }}' == docker ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries")
+          elif [[ '${{ matrix.environment }}' == macos ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries")
+          fi
+          uv run --no-cache --no-sync abxpkg env \
+            --install \
+            --lib="$ABXPKG_LIB_DIR" \
+            "${dependency_sources[@]}"
+          bash_binary="$ABXPKG_LIB_DIR/env/bin/bash"
+          python_binary="$ABXPKG_LIB_DIR/env/bin/python"
+          uv_binary="$ABXPKG_LIB_DIR/env/bin/uv"
+          test -L "$bash_binary"
+          test -x "$bash_binary"
+          test -L "$python_binary"
+          test -x "$python_binary"
+          test -L "$uv_binary"
+          test -x "$uv_binary"
+          {
+            echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR"
+            echo "BASH_BINARY=$bash_binary"
+            echo "PYTHON_BINARY=$python_binary"
+            echo "UV_BINARY=$uv_binary"
+          } >> "$GITHUB_ENV"
+          if [[ '${{ matrix.environment }}' != macos ]]; then
+            sudo_binary="$ABXPKG_LIB_DIR/env/bin/sudo"
+            test -L "$sudo_binary"
+            test -x "$sudo_binary"
+            echo "SUDO_BINARY=$sudo_binary" >> "$GITHUB_ENV"
+          fi
+          if [[ '${{ matrix.environment }}' != macos ]]; then
+            echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
+          fi
+          if [[ '${{ matrix.environment }}' == docker ]]; then
+            docker_binary="$ABXPKG_LIB_DIR/env/bin/docker"
+            test -L "$docker_binary"
+            test -x "$docker_binary"
+            echo "DOCKER_BINARY=$docker_binary" >> "$GITHUB_ENV"
+          elif [[ '${{ matrix.environment }}' == macos ]]; then
+            test -L "$ABXPKG_LIB_DIR/env/bin/brew"
+            test -x "$ABXPKG_LIB_DIR/env/bin/brew"
+          fi
+      - name: Build the exact checkout's documentation image
+        if: ${{ matrix.environment == 'docker' }}
+        run: '"$DOCKER_BINARY" build --tag archivebox/archivebox:dev .'
+      - name: Run the Ubuntu, Docker, or macOS documentation lane
+        if: ${{ !contains(fromJSON('["root", "freebsd", "openbsd"]'), matrix.environment) }}
+        run: uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
+      - name: Run the root documentation lane
+        if: ${{ matrix.environment == 'root' }}
+        run: |
+          "$SUDO_BINARY" \
+            --preserve-env=PATH,ABXPKG_LIB_DIR,BASH_BINARY,PYTHON_BINARY,SUDO_BINARY,UV_BINARY \
+            "$UV_BINARY" run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'

.github/workflows/duplicate-issue-detection.yml

@@ -6,18 +6,55 @@ on:
 
 jobs:
   check-duplicates:
-    runs-on: ubuntu-latest
+    runs-on: ubuntu-24.04
     permissions:
       contents: read
       issues: write
     steps:
       - name: Checkout repository
-        uses: actions/checkout@v4
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           fetch-depth: 1
 
-      - name: Install opencode
-        run: curl -fsSL https://opencode.ai/install | bash
+      - name: Set up Python
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+
+      - name: Resolve OpenCode and workflow tools through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
+        run: |
+          set -Eeuo pipefail
+          uv sync --no-cache --locked --no-sources --no-install-project
+          OPENCODE_CONFIG="$(uv run --no-cache --no-sync --no-sources python -c 'from abx_plugins import get_plugins_dir; print(get_plugins_dir() / "opencode" / "config.json")')"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
+          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
+          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
+          opencode_env="$(
+            uv run --no-cache --no-sync --no-sources abxpkg env \
+              --install \
+              --json \
+              --lib="$ABXPKG_LIB_DIR" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
+              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:github_binaries" \
+              --deps-from="$OPENCODE_CONFIG:required_binaries"
+          )"
+          jq_binary="$ABXPKG_LIB_DIR/env/bin/jq"
+          "$jq_binary" -r 'to_entries[] | "\(.key)=\(.value)"' \
+            <<< "$opencode_env" >> "$GITHUB_ENV"
+
+          opencode_binary="$ABXPKG_LIB_DIR/env/bin/opencode"
+          echo "OPENCODE_BINARY=$opencode_binary" >> "$GITHUB_ENV"
+          test -L "$opencode_binary"
+          test -x "$opencode_binary"
 
       - name: Check for duplicate issues
         env:
@@ -33,7 +70,7 @@ jobs:
               "webfetch": "allow"
             }
         run: |
-          opencode run -m anthropic/claude-haiku-4-5 "A new issue has been created:
+          "$OPENCODE_BINARY" run -m anthropic/claude-haiku-4-5 "A new issue has been created:
 
           Issue number:
           ${{ github.event.issue.number }}

.github/workflows/lint.yml

@@ -1,51 +1,38 @@
 name: Run linters
 
 on:
-  workflow_dispatch:
-  push:
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+  workflow_call:
 
 env:
   UV_NO_SOURCES: "1"
-  PYTHONPATH: ${{ github.workspace }}/abxpkg:${{ github.workspace }}/abx-plugins:${{ github.workspace }}/abx-dl
 
 jobs:
   lint:
     runs-on: ubuntu-22.04
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
           fetch-depth: 1
 
-      - name: Clone abxpkg
-        run: bash .github/scripts/clone_abx_repo.sh abxpkg
-
-      - name: Clone abx-plugins
-        run: bash .github/scripts/clone_abx_repo.sh abx-plugins
-
-      - name: Clone abx-dl
-        run: bash .github/scripts/clone_abx_repo.sh abx-dl
-
       - name: Set up Python
-        uses: actions/setup-python@v5
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
           python-version: "3.13"
           architecture: x64
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v6
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
         with:
-          version: "0.10.6"
+          version: "0.11.3"
           enable-cache: false
+          cache-dependency-glob: uv.lock
+
+      - name: Verify checked dependency lock
+        run: uv lock --no-cache --check
 
       - name: Install dependencies with uv
-        run: |
-          uv venv
-          uv pip install --all-groups -e ./abxpkg -e ./abx-plugins -e ./abx-dl -e ".[sonic,debug]"
+        run: uv sync --no-cache --locked --dev --extra sonic --extra debug
 
       - name: Run prek
-        run: uv run --no-sync --no-sources prek run --all-files
+        run: uv run --no-cache --no-sync --no-sources prek run --all-files

.github/workflows/pip.yml

@@ -1,157 +1,166 @@
 name: Build Pip package
 
 on:
-  workflow_dispatch:
   workflow_call:
-  push:
-    branches:
-      - '**'
-    tags:
-      - 'v*'
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
 
 env:
-  PYTHON_VERSION: "3.13"
+  UV_VERSION: "0.11.3"
 
 jobs:
   build:
-    permissions:
-      id-token: write
-
+    name: build distributions
     runs-on: ubuntu-24.04
-    environment: pypi
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
 
-      - name: Install uv
-        uses: astral-sh/setup-uv@v6
+      - name: Set up Python 3.13
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
-          version: "0.10.6"
-          enable-cache: false
-
-      - name: Set up Python
-        uses: actions/setup-python@v5
-        with:
-          python-version: ${{ env.PYTHON_VERSION }}
+          python-version: "3.13"
           architecture: x64
 
-      - name: APT install archivebox dev + run dependencies
-        uses: awalsh128/cache-apt-pkgs-action@latest
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
         with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
+          version: ${{ env.UV_VERSION }}
+          enable-cache: false
+          cache-dependency-glob: uv.lock
 
-      - name: Wait for released ArchiveBox deps on PyPI
-        run: |
-          python - <<'PY'
-          import re
-          import sys
-          import time
-          import tomllib
-          import urllib.request
-
-          watched = {"abxbus", "abxpkg", "abx-plugins", "abx-dl"}
-          deps = tomllib.loads(open("pyproject.toml", "rb").read().decode())["project"]["dependencies"]
-          required = {}
-          for dep in deps:
-              for name in watched:
-                  match = re.match(rf"{re.escape(name)}\s*(==|>=)\s*([^,;\s]+)", dep)
-                  if match:
-                      required[name] = match.group(2)
-
-          deadline = time.monotonic() + 300
-          missing = required.copy()
-          while missing and time.monotonic() < deadline:
-              for name, version in list(missing.items()):
-                  try:
-                      with urllib.request.urlopen(f"https://pypi.org/pypi/{name}/{version}/json", timeout=20):
-                          available = True
-                  except Exception:
-                      available = False
-                  if available:
-                      print(f"{name} {version} is available on PyPI")
-                      missing.pop(name)
-                  else:
-                      print(f"{name} {version} is not available on PyPI yet")
-              if missing:
-                  time.sleep(10)
-
-          if missing:
-              print(f"Missing PyPI releases after wait: {missing}", file=sys.stderr)
-              sys.exit(1)
-          PY
-
-      - name: Prepare released dependency sync project
-        run: |
-          python - <<'PY'
-          from pathlib import Path
-          import json
-          import re
-          import urllib.request
-
-          source = Path("pyproject.toml")
-          target_dir = Path(".ci-uv-project")
-          target_dir.mkdir(exist_ok=True)
-          text = source.read_text()
-          text = text.replace(
-              'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
-              'environments = ["sys_platform == \'linux\'"]',
-          )
-
-          # CI runs immediately after publishing internal abx packages. PyPI's
-          # simple index can lag the version JSON endpoint, so sync against the
-          # exact wheel URLs from the version endpoint while leaving package
-          # metadata in pyproject.toml unchanged for the actual build.
-          for package in ("abxpkg", "abx-plugins", "abx-dl"):
-              match = re.search(rf'"{re.escape(package)}>=(?P<version>[^"]+)"', text)
-              if not match:
-                  continue
-              version = match.group("version")
-              with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
-                  data = json.load(response)
-              wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
-              text = re.sub(
-                  rf'"{re.escape(package)}>=[^"]+"',
-                  f'"{package} @ {wheel_url}"',
-                  text,
-                  count=1,
-              )
-
-          (target_dir / "pyproject.toml").write_text(text)
-          PY
-
-      - name: UV install archivebox dev + run sub-dependencies
-        env:
-          UV_PROJECT_ENVIRONMENT: ${{ github.workspace }}/.venv
-        run: uv sync --project .ci-uv-project --all-extras --no-install-project --no-install-workspace --no-sources --no-cache
-
-      - name: UV build archivebox and archivebox/pkgs/* packages
-        run: |
-          uv build --all
-
-      - name: Publish new package wheels and sdists to PyPI
-        uses: pypa/gh-action-pypi-publish@release/v1
-        # ignore when publish to PyPI fails due to duplicate tag
-        continue-on-error: true
-
-      - name: UV install archivebox and archivebox/pkgs/* locally for tests
-        env:
-          UV_PROJECT_ENVIRONMENT: ${{ github.workspace }}/.venv
-        run: |
-          uv sync --project .ci-uv-project --all-extras --no-install-project --no-install-workspace --no-sources --no-cache
-          uv pip install --python .venv/bin/python --no-deps dist/archivebox-*.whl
-
-      - name: Verify built package full install
-        env:
-          UV_PROJECT_ENVIRONMENT: ${{ github.workspace }}/.venv
+      - name: Build every distribution
         run: |
           set -Eeuo pipefail
-          DATA_DIR="$(mktemp -d)"
-          cd "$DATA_DIR"
-          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox init
-          timeout 30m uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox install
-          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox version
-          uv run --project "$GITHUB_WORKSPACE/.ci-uv-project" --no-sync --no-sources archivebox status
+          uv build --no-cache --all-packages --no-sources --out-dir dist --clear
+
+          shopt -s nullglob
+          artifacts=(dist/*)
+          wheels=(dist/archivebox-*.whl)
+          sdists=(dist/archivebox-*.tar.gz)
+          [[ "${#artifacts[@]}" -gt 0 ]]
+          [[ "${#wheels[@]}" -eq 1 ]]
+          [[ "${#sdists[@]}" -eq 1 ]]
+          printf '%s\n' "$GITHUB_SHA" > dist/COMMIT_SHA
+          uv run --no-cache --no-project python - <<'PY'
+          import hashlib
+          from pathlib import Path
+
+          root = Path("dist")
+          wheels = list(root.glob("archivebox-*.whl"))
+          sdists = list(root.glob("archivebox-*.tar.gz"))
+          if len(wheels) != 1 or len(sdists) != 1:
+              raise SystemExit("Expected exactly one ArchiveBox wheel and one sdist")
+          artifacts = [root / "COMMIT_SHA", wheels[0], sdists[0]]
+          checksum_lines = []
+          for artifact in artifacts:
+              digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
+              line = f"{digest}  {artifact.name}"
+              checksum_lines.append(line)
+              print(line)
+          Path("dist/SHA256SUMS").write_text("\n".join(checksum_lines) + "\n")
+          PY
+
+      - name: Upload every distribution
+        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
+        with:
+          name: python-distributions
+          path: dist/*
+          if-no-files-found: error
+          retention-days: 7
+
+  install-smoke:
+    name: install ${{ matrix.artifact }} / Python ${{ matrix.python }} / ${{ matrix.os }}
+    needs: build
+    runs-on: ${{ matrix.os }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-24.04, macos-15]
+        python: ["3.13", "3.14"]
+        artifact: [wheel, sdist]
+
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+
+      - name: Set up Python ${{ matrix.python }}
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ matrix.python }}
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: ${{ env.UV_VERSION }}
+          enable-cache: false
+          cache-dependency-glob: uv.lock
+
+      - name: Download distributions
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: python-distributions
+          path: ${{ runner.temp }}/python-distributions
+
+      - name: Install and smoke-test the built ${{ matrix.artifact }}
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          shopt -s nullglob
+
+          case "${{ matrix.artifact }}" in
+            wheel) candidates=("$RUNNER_TEMP"/python-distributions/archivebox-*.whl) ;;
+            sdist) candidates=("$RUNNER_TEMP"/python-distributions/archivebox-*.tar.gz) ;;
+            *) exit 2 ;;
+          esac
+          [[ "${#candidates[@]}" -eq 1 ]]
+          artifact="${candidates[0]}"
+
+          smoke_root="$(mktemp -d "$RUNNER_TEMP/archivebox-package-smoke.XXXXXX")"
+          trap 'rm -rf "$smoke_root"' EXIT
+          smoke_env="$smoke_root/env"
+          data_dir="$smoke_root/data"
+          mkdir -p "$data_dir"
+
+          dependency_artifacts=()
+          while IFS= read -r dependency_artifact; do
+            dependency_artifacts+=("$dependency_artifact")
+          done < <(uv run --no-cache --no-project python - <<'PY'
+          import tomllib
+
+          locked = {
+              package["name"]: package
+              for package in tomllib.load(open("uv.lock", "rb"))["package"]
+          }
+          for package_name in ("abx-dl", "abx-plugins", "abxpkg", "abxbus"):
+              package = locked[package_name]
+              wheels = package.get("wheels") or []
+              if len(wheels) != 1:
+                  raise SystemExit(f"Expected exactly one locked wheel for {package_name}")
+              print(wheels[0]["url"])
+          PY
+          )
+
+          uv venv --python "${{ matrix.python }}" "$smoke_env"
+          uv pip install --no-cache \
+            --python "$smoke_env" \
+            "${dependency_artifacts[@]}" \
+            "$artifact"
+
+          unset PYTHONPATH
+          cd "$data_dir"
+          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync python - <<'PY'
+          import os
+          from pathlib import Path
+
+          import archivebox
+
+          installed_path = Path(archivebox.__file__).resolve()
+          environment_path = Path(os.environ["VIRTUAL_ENV"]).resolve()
+          workspace_path = Path(os.environ["GITHUB_WORKSPACE"]).resolve()
+          assert installed_path.is_relative_to(environment_path), (installed_path, environment_path)
+          assert not installed_path.is_relative_to(workspace_path), (installed_path, workspace_path)
+          print(installed_path)
+          PY
+          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox version
+          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox init
+          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox status

.github/workflows/release-runner.yml

@@ -1,60 +0,0 @@
-name: Release State
-
-on:
-  push:
-    branches:
-      - dev
-  workflow_dispatch:
-
-permissions:
-  actions: write
-  contents: write
-  id-token: write
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
-jobs:
-  release-state:
-    if: github.event_name != 'push' || !startsWith(github.event.head_commit.message, 'release:')
-    runs-on: ubuntu-latest
-    environment: pypi
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          ref: ${{ github.ref_name }}
-          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-
-      - uses: actions/setup-python@v5
-        with:
-          python-version: "3.13"
-
-      - uses: astral-sh/setup-uv@v6
-        with:
-          version: "0.10.6"
-          enable-cache: false
-
-      - uses: actions/setup-node@v4
-        with:
-          node-version: 22
-
-      - name: Configure git identity
-        run: |
-          git config user.name "github-actions[bot]"
-          git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
-
-      - name: Run release script
-        env:
-          DEFAULT_BRANCH: dev
-          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-        run: ./bin/release.sh
-
-      - name: Dispatch image and tap updates
-        env:
-          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-        run: |
-          gh workflow run docker.yml --ref dev
-          gh workflow run update-homebrew-tap.yml --ref dev

.github/workflows/release.yml

@@ -1,41 +1,316 @@
 name: Release
 
-# Orchestrates the full release pipeline:
-#   1. Build and publish pip package to PyPI
-#   2. Build Homebrew formula after pip
-#   3. Build Docker images in parallel
-#
-# Individual workflows also run on push for CI (see their own triggers).
-# Manual fallback for the hosted release pipeline. The normal prerelease loop
-# is run by bin/release.sh locally so published GitHub releases do not launch a
-# duplicate release job.
-
 on:
-  workflow_dispatch:
+  workflow_run:
+    workflows: [CI]
+    types: [completed]
+    branches: [dev, main]
 
-permissions:
-  contents: write
-  packages: write
-  id-token: write
+permissions: {}
 
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+  group: release-${{ github.repository }}-${{ github.event.workflow_run.head_branch }}
+  cancel-in-progress: false
+
+env:
+  DOCKERHUB_IMAGE: archivebox/archivebox
+  GHCR_IMAGE: ghcr.io/archivebox/archivebox
+  RELEASE_BRANCH: ${{ github.event.workflow_run.head_branch }}
+  RELEASE_SHA: ${{ github.event.workflow_run.head_sha }}
+  CI_RUN_ID: ${{ github.event.workflow_run.id }}
 
 jobs:
-  pip:
-    name: Publish to PyPI
-    uses: ./.github/workflows/pip.yml
-    secrets: inherit
+  python-release:
+    name: Publish tested Python artifacts
+    if: github.event.workflow_run.conclusion == 'success' && github.event.workflow_run.event == 'push'
+    runs-on: ubuntu-24.04
+    environment: pypi
+    permissions:
+      actions: read
+      contents: write
+      id-token: write
+    outputs:
+      version: ${{ steps.version.outputs.version }}
+      tag: ${{ steps.version.outputs.tag }}
+      release_ready: ${{ steps.release-state.outputs.ready }}
+    steps:
+      - name: Checkout the tested commit
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 0
+          submodules: true
+          ref: ${{ env.RELEASE_SHA }}
+          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
 
-  homebrew:
-    name: Update Homebrew formula
-    needs: pip
-    uses: ./.github/workflows/homebrew.yml
-    secrets: inherit
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
 
-  docker:
-    name: Build Docker images
-    needs: pip
-    uses: ./.github/workflows/docker.yml
-    secrets: inherit
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+
+      - name: Resolve release binaries through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env --install \
+            --lib="$ABXPKG_LIB_DIR" \
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" >/dev/null
+          for name in uv gh git jq curl; do
+            test -L "$ABXPKG_LIB_DIR/env/bin/$name"
+            test -x "$ABXPKG_LIB_DIR/env/bin/$name"
+          done
+          {
+            echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
+            echo "GH_BINARY=$ABXPKG_LIB_DIR/env/bin/gh"
+            echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
+            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
+            echo "CURL_BINARY=$ABXPKG_LIB_DIR/env/bin/curl"
+          } >> "$GITHUB_ENV"
+
+      - name: Download exact tested distributions
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          name: python-distributions
+          path: ${{ runner.temp }}/python-distributions
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ env.CI_RUN_ID }}
+
+      - name: Record release version
+        id: version
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION="$($UV_BINARY run --no-cache --no-project python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
+          echo "version=$VERSION" >> "$GITHUB_OUTPUT"
+          echo "tag=v$VERSION" >> "$GITHUB_OUTPUT"
+
+      - name: Validate release branch and version
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION='${{ steps.version.outputs.version }}'
+          case "$RELEASE_BRANCH" in
+            main)
+              ;;
+            dev)
+              [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] || {
+                echo "Refusing to publish ArchiveBox $VERSION from dev; only archivebox:main may publish non-prerelease versions." >&2
+                exit 1
+              }
+              ;;
+            *)
+              echo "Refusing to publish ArchiveBox $VERSION from unsupported branch $RELEASE_BRANCH; only archivebox:dev and archivebox:main may publish." >&2
+              exit 1
+              ;;
+          esac
+
+      - name: Publish the exact tested distributions
+        env:
+          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
+          RELEASE_DISTRIBUTIONS_DIR: ${{ runner.temp }}/python-distributions
+        run: ./bin/release.sh
+
+      - name: Confirm this tested commit owns the release tag
+        id: release-state
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          TARGET=""
+          while read -r sha _; do
+            TARGET="$sha"
+          done < <("$GIT_BINARY" ls-remote origin "refs/tags/${{ steps.version.outputs.tag }}")
+          if [[ "$TARGET" == "$RELEASE_SHA" ]]; then
+            echo "ready=true" >> "$GITHUB_OUTPUT"
+          else
+            echo "ready=false" >> "$GITHUB_OUTPUT"
+          fi
+
+      - name: Verify published PyPI package installs and runs
+        if: steps.release-state.outputs.ready == 'true'
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          sleep 60
+          VERSION='${{ steps.version.outputs.version }}'
+          TOOL_ROOT="$(mktemp -d)"
+          export UV_TOOL_DIR="$TOOL_ROOT/tools"
+          export UV_TOOL_BIN_DIR="$TOOL_ROOT/bin"
+          "$UV_BINARY" tool install --no-cache --prerelease allow --force "archivebox==$VERSION"
+          INSTALLED_VERSION="$("$UV_TOOL_BIN_DIR/archivebox" --version)"
+          test "$INSTALLED_VERSION" = "$VERSION"
+
+  docker-release:
+    name: Publish tested multiarch image
+    if: >-
+      github.event.workflow_run.conclusion == 'success' &&
+      github.event.workflow_run.event == 'push' &&
+      needs.python-release.outputs.release_ready == 'true'
+    needs: python-release
+    runs-on: ubuntu-24.04
+    permissions:
+      actions: read
+      contents: read
+      packages: write
+    steps:
+      - name: Checkout the tested commit
+        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+        with:
+          fetch-depth: 1
+          ref: ${{ env.RELEASE_SHA }}
+
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: "3.13"
+
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+
+      - name: Resolve release and Docker binaries through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env --install \
+            --lib="$ABXPKG_LIB_DIR" \
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries" >/dev/null
+          for name in uv git jq docker; do
+            test -L "$ABXPKG_LIB_DIR/env/bin/$name"
+            test -x "$ABXPKG_LIB_DIR/env/bin/$name"
+          done
+          {
+            echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
+            echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
+            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
+            echo "DOCKER_BINARY=$ABXPKG_LIB_DIR/env/bin/docker"
+          } >> "$GITHUB_ENV"
+
+      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+        with:
+          install: true
+
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ github.token }}
+
+      - name: Download exact tested image digests
+        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
+        with:
+          path: /tmp/digests
+          pattern: digest-*
+          merge-multiple: true
+          github-token: ${{ github.token }}
+          repository: ${{ github.repository }}
+          run-id: ${{ env.CI_RUN_ID }}
+
+      - name: Verify digest set and collect tags
+        id: docker_meta
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
+          [[ "${#DIGESTS[@]}" -eq 2 ]]
+          for digest in "${DIGESTS[@]}"; do [[ "$digest" =~ ^[0-9a-f]{64}$ ]]; done
+
+          VERSION='${{ needs.python-release.outputs.version }}'
+          SHORT_SHA="${RELEASE_SHA::12}"
+          TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}^{}")"
+          TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
+          if [[ -z "$TAG_TARGET" ]]; then
+            TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}")"
+            TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
+          fi
+          {
+            echo 'dockerhub_tags<<EOF'
+            if [[ "$RELEASE_BRANCH" == dev ]]; then
+              echo "${DOCKERHUB_IMAGE}:dev"
+            elif [[ "$RELEASE_BRANCH" == main ]]; then
+              echo "${DOCKERHUB_IMAGE}:latest"
+            fi
+            echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
+            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${DOCKERHUB_IMAGE}:${VERSION}"
+            echo 'EOF'
+            echo 'ghcr_tags<<EOF'
+            if [[ "$RELEASE_BRANCH" == dev ]]; then
+              echo "${GHCR_IMAGE}:dev"
+            elif [[ "$RELEASE_BRANCH" == main ]]; then
+              echo "${GHCR_IMAGE}:latest"
+            fi
+            echo "${GHCR_IMAGE}:sha-${SHORT_SHA}"
+            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${GHCR_IMAGE}:${VERSION}"
+            echo 'EOF'
+          } >> "$GITHUB_OUTPUT"
+
+      - name: Create Docker Hub manifest
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
+          TAG_ARGS=()
+          while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.dockerhub_tags }}'
+          REFS=()
+          for digest in "${DIGESTS[@]}"; do REFS+=("${DOCKERHUB_IMAGE}@sha256:${digest}"); done
+          $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
+
+      - name: Create GHCR manifest
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
+          TAG_ARGS=()
+          while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.ghcr_tags }}'
+          REFS=()
+          for digest in "${DIGESTS[@]}"; do REFS+=("${GHCR_IMAGE}@sha256:${digest}"); done
+          $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
+
+      - name: Verify published Docker images run
+        shell: bash
+        run: |
+          set -Eeuo pipefail
+          VERSION='${{ needs.python-release.outputs.version }}'
+          for image in "${DOCKERHUB_IMAGE}:${VERSION}" "${GHCR_IMAGE}:${VERSION}"; do
+            "$DOCKER_BINARY" buildx imagetools inspect "$image" >/dev/null
+            INSTALLED_VERSION="$("$DOCKER_BINARY" run --rm "$image" --version)"
+            test "$INSTALLED_VERSION" = "$VERSION"
+          done
+
+      - name: Update Docker Hub README
+        uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+          repository: archivebox/archivebox
+
+  cascade:
+    name: Cascade verified release
+    needs:
+      - python-release
+      - docker-release
+    permissions:
+      contents: read
+    uses: ArchiveBox/monorepo/.github/workflows/cascade-release.yml@main
+    with:
+      upstream_package: archivebox
+      upstream_version: ${{ needs.python-release.outputs.version }}
+    secrets:
+      release_token: ${{ secrets.RELEASE_GH_TOKEN }}

.github/workflows/test-parallel.yml

@@ -1,382 +1,154 @@
 name: Parallel Tests
 
 on:
-  workflow_dispatch:
-  pull_request:
-    branches: [dev, main, master]
-  push:
-    branches: [dev]
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+  workflow_call:
 
 env:
   PYTHONIOENCODING: utf-8
   PYTHONLEGACYWINDOWSSTDIO: utf-8
   USE_COLOR: False
   UV_NO_SOURCES: "1"
-  ABXPKG_MIN_RELEASE_AGE: "0"
+  CI_PYTHON_VERSION: "3.13"
 
 jobs:
   discover-tests:
-    name: Discover test files
+    name: Discover every test
     runs-on: ubuntu-24.04
     outputs:
-      test-files: ${{ steps.set-matrix.outputs.test-files }}
-      plugin-tests: ${{ steps.set-plugin-matrix.outputs.plugin-tests }}
+      tests: ${{ steps.discover.outputs.tests }}
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
           fetch-depth: 1
 
-      - name: Discover test files
-        id: set-matrix
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
+        with:
+          python-version: ${{ env.CI_PYTHON_VERSION }}
+
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
+
+      - name: Resolve discovery tools through abxpkg
+        env:
+          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
         run: |
-          # Find all main test files
-          all_tests=$(find archivebox/tests -maxdepth 1 -name "test_*.py" -type f | sort)
+          set -Eeuo pipefail
+          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          test -n "$ABXPKG_VERSION"
+          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
+          export PYTHON_BINARY="$pythonLocation/bin/python"
+          test -x "$PYTHON_BINARY"
+          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+            --install \
+            --lib="$ABXPKG_LIB_DIR" \
+            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
+          {
+            echo "PYTHON_BINARY=$ABXPKG_LIB_DIR/env/bin/python"
+            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
+          } >> "$GITHUB_ENV"
+          test -L "$ABXPKG_LIB_DIR/env/bin/python"
+          test -x "$ABXPKG_LIB_DIR/env/bin/python"
+          test -L "$ABXPKG_LIB_DIR/env/bin/jq"
+          test -x "$ABXPKG_LIB_DIR/env/bin/jq"
 
-          # Create JSON array with test file info
-          json_array="["
-          first=true
-          for test_file in $all_tests; do
-            if [ "$first" = true ]; then
-              first=false
-            else
-              json_array+=","
-            fi
-
-            # Extract a display name for the test
-            name="main/$(basename $test_file .py | sed 's/^test_//')"
-            needs_chromium=false
-            if grep -Eiq '(chrom|archivewebpage|PLUGINS=.*title|--plugins=.*title|SAVE_TITLE.*[Tt]rue)' "$test_file"; then
-              needs_chromium=true
-            fi
-            needs_sonic=false
-            if grep -Eiq "shutil\\.which\\([\"']sonic|SEARCH_BACKEND_ENGINE=.*sonic|worker_sonic" "$test_file"; then
-              needs_sonic=true
-            fi
-
-            json_array+="{\"path\":\"$test_file\",\"name\":\"$name\",\"needs_chromium\":$needs_chromium,\"needs_sonic\":$needs_sonic}"
-          done
-          json_array+="]"
-
-          echo "test-files=$json_array" >> $GITHUB_OUTPUT
-          echo "Found $(echo $all_tests | wc -w) test files"
-          echo "$json_array" | jq '.'
-
-      - name: Clone abx-plugins
-        run: bash .github/scripts/clone_abx_repo.sh abx-plugins
-
-      - name: Discover plugin tests
-        id: set-plugin-matrix
+      - name: Discover every test exactly once
+        id: discover
         run: |
-          plugin_tests=$(find abx-plugins/abx_plugins/plugins -maxdepth 2 -type d -name tests | sed 's#abx-plugins/abx_plugins/plugins/##; s#/tests##' | sort)
+          set -Eeuo pipefail
+          mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
+          test "${#output[@]}" -eq 2
+          echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
+          echo "${output[0]}"
+          "$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
 
-          json_array="["
-          first=true
-          for plugin_name in $plugin_tests; do
-            if [ "$first" = true ]; then
-              first=false
-            else
-              json_array+=","
-            fi
-
-            needs_chromium=false
-            if grep -Riq "chrom" "abx-plugins/abx_plugins/plugins/$plugin_name"; then
-              needs_chromium=true
-            fi
-            needs_sonic=false
-            if [ "$plugin_name" = "search_backend_sonic" ]; then
-              needs_sonic=true
-            fi
-
-            json_array+="{\"plugin\":\"$plugin_name\",\"name\":\"plugin/$plugin_name\",\"needs_chromium\":$needs_chromium,\"needs_sonic\":$needs_sonic}"
-          done
-          json_array+="]"
-
-          echo "plugin-tests=$json_array" >> $GITHUB_OUTPUT
-          echo "Found $(echo $plugin_tests | wc -w) plugin test suites"
-          echo "$json_array" | jq '.'
-
-  run-tests:
+  tests:
     name: ${{ matrix.test.name }}
-    runs-on: ubuntu-24.04
     needs: discover-tests
+    runs-on: ubuntu-24.04
     env:
-      PYTHONPATH: ${{ github.workspace }}/abxpkg:${{ github.workspace }}/abx-plugins:${{ github.workspace }}/abx-dl
+      ABXPKG_LIB_DIR: /tmp/archivebox-lib
       CHROME_HEADLESS: "true"
-      CHROME_SANDBOX: "false"
+      DATA_DIR: /tmp/archivebox-test-data
       PERSONAS_DIR: /tmp/abx-personas
       CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile
 
     strategy:
       fail-fast: false
       matrix:
-        test: ${{ fromJson(needs.discover-tests.outputs.test-files) }}
-        python: ["3.13"]
+        test: ${{ fromJson(needs.discover-tests.outputs.tests) }}
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
           fetch-depth: 1
 
-      - name: Clone abxpkg
-        run: bash .github/scripts/clone_abx_repo.sh abxpkg
-
-      - name: Clone abx-plugins
-        run: bash .github/scripts/clone_abx_repo.sh abx-plugins
-
-      - name: Clone abx-dl
-        run: bash .github/scripts/clone_abx_repo.sh abx-dl
-
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v5
+      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
-          python-version: ${{ matrix.python }}
+          python-version: ${{ env.CI_PYTHON_VERSION }}
           architecture: x64
 
-      - name: Install uv
-        uses: astral-sh/setup-uv@v6
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
         with:
-          version: "0.10.6"
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
 
-      - name: Set up Node JS
-        uses: actions/setup-node@v4
-        with:
-          node-version: 22
-
-      - name: Cache uv
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/uv
-          key: ${{ runner.os }}-${{ matrix.python }}-uv-${{ hashFiles('pyproject.toml') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.python }}-uv-
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: git ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 python3-minimal gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.1
-
-      - name: Install dependencies with uv
-        run: |
-          uv venv
-          uv pip install --group dev -e ./abxpkg -e ./abx-plugins -e ./abx-dl -e ".[all]"
-
-      - name: Install Chrome through ArchiveBox
-        if: ${{ matrix.test.needs_chromium }}
+      - name: Resolve optional build dependencies through abxpkg
+        if: matrix.test.extra != '' || contains(matrix.test.paths_arg, 'test_postgres_backend')
         env:
-          MIN_CHROMIUM_MAJOR: "149"
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_INSTALL_TIMEOUT: "900"
-          ABX_CI_DATA_DIR: ${{ runner.temp }}/archivebox-ci-data
+          TEST_EXTRA: ${{ matrix.test.extra }}
+          TEST_PATHS: ${{ matrix.test.paths_arg }}
         run: |
-          set -euo pipefail
-          mkdir -p "$LIB_DIR" "$ABX_CI_DATA_DIR"
-          echo "LIB_DIR=$LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-
-          uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox install chrome
-
-          candidate="$(uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox shell -c 'from archivebox.machine.models import Binary; binary = Binary.objects.filter(name="chromium", status="installed").order_by("-modified_at").first(); print(binary.abspath if binary else "")' | tail -n 1)"
-          if [ ! -x "$candidate" ]; then
-            echo "ArchiveBox did not install an executable Chromium binary: ${candidate:-not found}" >&2
-            exit 1
+          set -Eeuo pipefail
+          ABXPKG_VERSION="$(uv run --no-cache --no-project python -c 'import tomllib; print(next(package["version"] for package in tomllib.load(open("uv.lock", "rb"))["package"] if package["name"] == "abxpkg"))')"
+          test -n "$ABXPKG_VERSION"
+          dependency_sources=()
+          if [[ -n "$TEST_EXTRA" ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
           fi
-          version="$("$candidate" --version || true)"
-          major="$(printf '%s\n' "$version" | sed -E 's/.* ([0-9]+)\..*/\1/' | head -1)"
-          case "$major" in
-            ''|*[!0-9]*) major=0 ;;
-          esac
-          if [ "$major" -lt "$MIN_CHROMIUM_MAJOR" ]; then
-            echo "ArchiveBox installed Chromium is too old: ${version:-unknown}" >&2
-            exit 1
+          if [[ "$TEST_PATHS" == *test_postgres_backend* ]]; then
+            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:postgres_binaries")
           fi
-          echo "Using ArchiveBox-installed Chromium: $candidate ($version)"
-          echo "CHROME_BINARY=$candidate" >> "$GITHUB_ENV"
-          echo "DATA_DIR=$ABX_CI_DATA_DIR" >> "$GITHUB_ENV"
-          echo "NODE_MODULES_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules" >> "$GITHUB_ENV"
-          echo "NODE_PATH=$LIB_DIR/pnpm/packages/chrome/node_modules" >> "$GITHUB_ENV"
-          echo "PNPM_BIN_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules/.bin" >> "$GITHUB_ENV"
-          echo "NPM_BIN_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules/.bin" >> "$GITHUB_ENV"
-          echo "CHROMEWEBSTORE_EXTENSIONS_DIR=$LIB_DIR/chromewebstore/extensions" >> "$GITHUB_ENV"
+          uv run --no-cache --no-project --with "abxpkg==$ABXPKG_VERSION" abxpkg env \
+            --install \
+            --no-cache \
+            --json \
+            --lib="$ABXPKG_LIB_DIR" \
+            "${dependency_sources[@]}"
 
-      - name: Install Sonic through ArchiveBox
-        if: ${{ matrix.test.needs_sonic }}
+      - name: Install test dependencies
         env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_INSTALL_TIMEOUT: "900"
-          ABX_CI_DATA_DIR: ${{ runner.temp }}/archivebox-ci-data
+          TEST_EXTRA: ${{ matrix.test.extra }}
         run: |
-          set -euo pipefail
-          mkdir -p "$LIB_DIR" "$ABX_CI_DATA_DIR"
-          echo "LIB_DIR=$LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-
-          uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox install search_backend_sonic
-
-          candidate="$(uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox shell -c 'from archivebox.machine.models import Binary; binary = Binary.objects.filter(name="sonic", status="installed").order_by("-modified_at").first(); print(binary.abspath if binary else "")' | tail -n 1)"
-          if [ ! -x "$candidate" ]; then
-            echo "ArchiveBox did not install an executable Sonic binary: ${candidate:-not found}" >&2
-            exit 1
+          set -Eeuo pipefail
+          extra_args=()
+          if [[ -n "$TEST_EXTRA" ]]; then
+            extra_args=(--extra "$TEST_EXTRA")
           fi
-          version="$("$candidate" --version || true)"
-          echo "Using ArchiveBox-installed Sonic: $candidate (${version:-version unknown})"
-          echo "SONIC_BINARY=$candidate" >> "$GITHUB_ENV"
-          echo "$(dirname "$candidate")" >> "$GITHUB_PATH"
+          uv sync --no-cache --locked --dev "${extra_args[@]}"
 
-      - name: Run test - ${{ matrix.test.name }}
-        run: |
-          mkdir -p tests/out
-          set +e
-          uv run --no-sync --no-sources pytest -xvs "${{ matrix.test.path }}" --basetemp=tests/out --ignore=archivebox/pkgs
-          status=$?
-          set -e
-          if [ "$status" -eq 5 ]; then
-            echo "No tests collected from ${{ matrix.test.path }}; treating as an empty test module."
-            exit 0
-          fi
-          exit "$status"
-
-  plugin-tests:
-    name: ${{ matrix.plugin.name }}
-    runs-on: ubuntu-24.04
-    needs: discover-tests
-    env:
-      PYTHONPATH: ${{ github.workspace }}/abxpkg:${{ github.workspace }}/abx-plugins:${{ github.workspace }}/abx-dl
-      CHROME_HEADLESS: "true"
-      CHROME_SANDBOX: "false"
-      PERSONAS_DIR: /tmp/abx-personas
-      CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile
-
-    strategy:
-      fail-fast: false
-      matrix:
-        plugin: ${{ fromJson(needs.discover-tests.outputs.plugin-tests) }}
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Clone abxpkg
-        run: bash .github/scripts/clone_abx_repo.sh abxpkg
-
-      - name: Clone abx-plugins
-        run: bash .github/scripts/clone_abx_repo.sh abx-plugins
-
-      - name: Clone abx-dl
-        run: bash .github/scripts/clone_abx_repo.sh abx-dl
-
-      - name: Set up Python 3.13
-        uses: actions/setup-python@v5
-        with:
-          python-version: "3.13"
-          architecture: x64
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          version: "0.10.6"
-
-      - name: Set up Node JS
-        uses: actions/setup-node@v4
-        with:
-          node-version: 22
-
-      - name: Cache uv
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/uv
-          key: ${{ runner.os }}-3.13-uv-${{ hashFiles('pyproject.toml') }}
-          restore-keys: |
-            ${{ runner.os }}-3.13-uv-
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: git ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 python3-minimal gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.1
-
-      - name: Install dependencies with uv
-        run: |
-          uv venv
-          uv pip install --group dev -e ./abxpkg -e ./abx-plugins -e ./abx-dl -e ".[all]"
-
-      - name: Install Chrome through ArchiveBox
-        if: ${{ matrix.plugin.needs_chromium }}
+      - name: Run ${{ matrix.test.name }}
         env:
-          MIN_CHROMIUM_MAJOR: "149"
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_INSTALL_TIMEOUT: "900"
-          ABX_CI_DATA_DIR: ${{ runner.temp }}/archivebox-ci-data
+          TEST_PATHS: ${{ matrix.test.paths_arg }}
         run: |
-          set -euo pipefail
-          mkdir -p "$LIB_DIR" "$ABX_CI_DATA_DIR"
-          echo "LIB_DIR=$LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-
-          uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox install chrome
-
-          candidate="$(uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox shell -c 'from archivebox.machine.models import Binary; binary = Binary.objects.filter(name="chromium", status="installed").order_by("-modified_at").first(); print(binary.abspath if binary else "")' | tail -n 1)"
-          if [ ! -x "$candidate" ]; then
-            echo "ArchiveBox did not install an executable Chromium binary: ${candidate:-not found}" >&2
-            exit 1
-          fi
-          version="$("$candidate" --version || true)"
-          major="$(printf '%s\n' "$version" | sed -E 's/.* ([0-9]+)\..*/\1/' | head -1)"
-          case "$major" in
-            ''|*[!0-9]*) major=0 ;;
-          esac
-          if [ "$major" -lt "$MIN_CHROMIUM_MAJOR" ]; then
-            echo "ArchiveBox installed Chromium is too old: ${version:-unknown}" >&2
-            exit 1
-          fi
-          echo "Using ArchiveBox-installed Chromium: $candidate ($version)"
-          echo "CHROME_BINARY=$candidate" >> "$GITHUB_ENV"
-          echo "DATA_DIR=$ABX_CI_DATA_DIR" >> "$GITHUB_ENV"
-          echo "NODE_MODULES_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules" >> "$GITHUB_ENV"
-          echo "NODE_PATH=$LIB_DIR/pnpm/packages/chrome/node_modules" >> "$GITHUB_ENV"
-          echo "PNPM_BIN_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules/.bin" >> "$GITHUB_ENV"
-          echo "NPM_BIN_DIR=$LIB_DIR/pnpm/packages/chrome/node_modules/.bin" >> "$GITHUB_ENV"
-          echo "CHROMEWEBSTORE_EXTENSIONS_DIR=$LIB_DIR/chromewebstore/extensions" >> "$GITHUB_ENV"
-
-      - name: Install Sonic through ArchiveBox
-        if: ${{ matrix.plugin.needs_sonic }}
-        env:
-          LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-          ABXPKG_INSTALL_TIMEOUT: "900"
-          ABX_CI_DATA_DIR: ${{ runner.temp }}/archivebox-ci-data
-        run: |
-          set -euo pipefail
-          mkdir -p "$LIB_DIR" "$ABX_CI_DATA_DIR"
-          echo "LIB_DIR=$LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-
-          uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox install search_backend_sonic
-
-          candidate="$(uv run --project "$GITHUB_WORKSPACE" --directory "$ABX_CI_DATA_DIR" --no-sync --no-sources archivebox shell -c 'from archivebox.machine.models import Binary; binary = Binary.objects.filter(name="sonic", status="installed").order_by("-modified_at").first(); print(binary.abspath if binary else "")' | tail -n 1)"
-          if [ ! -x "$candidate" ]; then
-            echo "ArchiveBox did not install an executable Sonic binary: ${candidate:-not found}" >&2
-            exit 1
-          fi
-          version="$("$candidate" --version || true)"
-          echo "Using ArchiveBox-installed Sonic: $candidate (${version:-version unknown})"
-          echo "SONIC_BINARY=$candidate" >> "$GITHUB_ENV"
-          echo "$(dirname "$candidate")" >> "$GITHUB_PATH"
-
-      - name: Run plugin tests - ${{ matrix.plugin.name }}
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          TWOCAPTCHA_API_KEY: ${{ secrets.TWOCAPTCHA_API_KEY }}
-          API_KEY_2CAPTCHA: ${{ secrets.TWOCAPTCHA_API_KEY }}
-        run: |
-          uv run --no-sync --no-sources bash ./bin/test_plugins.sh "${{ matrix.plugin.plugin }}" --no-coverage
+          set -Eeuo pipefail
+          read -r -a test_paths <<< "$TEST_PATHS"
+          for test_path in "${test_paths[@]}"; do
+            test -f "$test_path"
+          done
+          for test_path in "${test_paths[@]}"; do
+            test_name="$(basename "$test_path" .py)"
+            test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
+            export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
+            export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
+            export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
+            mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
+            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
+              --basetemp="$RUNNER_TEMP/pytest/$test_slug"
+          done

.github/workflows/test.yml

@@ -1,161 +1,63 @@
-name: Run tests
-on: [push]
+name: Install and CLI Compatibility
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+on:
+  workflow_call:
 
 env:
   DOCKER_IMAGE: archivebox-ci
+  ARCHIVEBOX_IMAGE: archivebox-ci
   PYTHONIOENCODING: utf-8
   PYTHONLEGACYWINDOWSSTDIO: utf-8
   USE_COLOR: False
   UV_NO_SOURCES: "1"
 
 jobs:
-  python_tests:
+  install_cli_platform:
+    name: CLI install / ${{ matrix.os_name }} / Python ${{ matrix.python }}
     runs-on: ${{ matrix.os }}
-    env:
-      PYTHONPATH: ${{ github.workspace }}/abxpkg:${{ github.workspace }}/abx-plugins:${{ github.workspace }}/abx-dl
 
     strategy:
+      fail-fast: false
       matrix:
-        os: [ubuntu-22.04]
-        # os: [ubuntu-22.04, macos-latest, windows-latest]
-        python: ["3.13"]
+        include:
+          - os: ubuntu-24.04
+            os_name: linux
+            python: "3.14"
+          - os: macos-15
+            os_name: macOS
+            python: "3.13"
+          - os: macos-15
+            os_name: macOS
+            python: "3.14"
 
     steps:
-      - uses: actions/checkout@v4
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
         with:
           submodules: true
           fetch-depth: 1
 
-      - name: Clone abxpkg
-        run: bash .github/scripts/clone_abx_repo.sh abxpkg
-
-      - name: Clone abx-plugins
-        run: bash .github/scripts/clone_abx_repo.sh abx-plugins
-
-      - name: Clone abx-dl
-        run: bash .github/scripts/clone_abx_repo.sh abx-dl
-
-      ### Setup Python & JS Languages
+      ### Set up Python
       - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@v4
+        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
         with:
           python-version: ${{ matrix.python }}
-          architecture: x64
 
       - name: Install uv
-        uses: astral-sh/setup-uv@v4
+        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
         with:
-          version: "0.10.6"
-
-      - name: Set up Node JS
-        uses: actions/setup-node@v4
-        with:
-          node-version: 22
-
-      ### Install Python & JS Dependencies
-      - name: Cache uv
-        uses: actions/cache@v3
-        with:
-          path: ~/.cache/uv
-          key: ${{ runner.os }}-${{ matrix.python }}-uv-${{ hashFiles('pyproject.toml') }}
-          restore-keys: |
-            ${{ runner.os }}-${{ matrix.python }}-uv-
-
-      - uses: awalsh128/cache-apt-pkgs-action@latest
-        with:
-          packages: ripgrep build-essential python3-dev python3-setuptools libssl-dev libldap2-dev libsasl2-dev zlib1g-dev libatomic1 python3-minimal gnupg2 curl wget python3-ldap python3-msgpack python3-mutagen python3-regex python3-pycryptodome procps
-          version: 1.0
+          version: "0.11.3"
+          enable-cache: false
+          cache-dependency-glob: uv.lock
 
       - name: Install dependencies with uv
-        run: |
-          uv venv
-          uv pip install --group dev -e ./abxpkg -e ./abx-plugins -e ./abx-dl -e ".[all]"
-
-      ### Run the tests
-      - name: Directory listing for debugging
-        run: |
-          pwd
-          ls
+        run: uv sync --no-cache --locked --dev
 
       - name: ArchiveBox full install check
         run: |
           set -Eeuo pipefail
           DATA_DIR="$PWD/tests/out/data"
           mkdir -p "$DATA_DIR"
-          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox init
-          timeout 30m uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox install
-          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox version
-          uv run --directory "$DATA_DIR" --no-sync --no-sources archivebox status
-
-      - name: Test built package with pytest
-        # TODO: remove this exception for windows once we get tests passing on that platform
-        if: ${{ !contains(matrix.os, 'windows') }}
-        run: |
-          mkdir -p tests/out
-          uv run --no-sync --no-sources pytest -s archivebox/tests --basetemp=tests/out --ignore=archivebox/pkgs
-
-      - name: Run plugin tests
-        if: ${{ !contains(matrix.os, 'windows') }}
-        env:
-          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
-          TWOCAPTCHA_API_KEY: ${{ secrets.TWOCAPTCHA_API_KEY }}
-          API_KEY_2CAPTCHA: ${{ secrets.TWOCAPTCHA_API_KEY }}
-        run: |
-          uv run --no-sync --no-sources bash ./bin/test_plugins.sh --no-coverage
-
-  docker_tests:
-    runs-on: ubuntu-latest
-
-    steps:
-      - uses: actions/checkout@v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      # TODO: as of 2020-11 this helper layer broke, upgrade and re-enable this once it's usable again
-      # - uses: satackey/action-docker-layer-caching@v0.0.8
-
-      - name: Build image
-        run: |
-          docker build . -t "$DOCKER_IMAGE"
-
-      - name: Init data dir
-        run: |
-          mkdir "${{ github.workspace }}/data"
-          docker run -v "${{ github.workspace }}/data":/data "$DOCKER_IMAGE" init
-
-      - name: Run test server
-        run: |
-          sudo bash -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
-          docker run --name www-nginx -p 80:80 -d nginx
-
-      - name: Add link
-        run: |
-          docker run -v "$PWD"/data:/data --network host "$DOCKER_IMAGE" add http://www.test-nginx-1.local
-
-      - name: Add stdin link
-        run: |
-          echo "http://www.test-nginx-2.local" | docker run -i --network host -v "$PWD"/data:/data "$DOCKER_IMAGE" add
-
-      - name: List links
-        run: |
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-1.local" || { echo "The site 1 isn't in the list"; exit 1; }
-          docker run -v "$PWD"/data:/data "$DOCKER_IMAGE" list | grep -q "www.test-nginx-2.local" || { echo "The site 2 isn't in the list"; exit 1; }
-
-      - name: Start docker-compose stack
-        run: |
-          docker-compose run archivebox init
-          docker-compose up -d
-          sleep 5
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'ArchiveBox'
-          curl --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js' | grep 'window.django'
-
-      - name: Check added urls show up in index
-        run: |
-          docker-compose run archivebox add 'http://example.com/#test_docker' --index-only
-          curl --silent --location 'http://127.0.0.1:8000' | grep 'http://example.com/#test_docker'
-          docker-compose down || true
+          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox init
+          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox install
+          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox version
+          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox status

.github/workflows/update-homebrew-tap.yml

@@ -1,37 +0,0 @@
-name: Update downstream packages
-
-on:
-  push:
-    branches: [dev]
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: update-downstream-packages-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  dispatch:
-    runs-on: ubuntu-latest
-
-    steps:
-      - name: Dispatch downstream package updates
-        env:
-          DOWNSTREAM_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || secrets.HOMEBREW_TAP_TOKEN }}
-        run: |
-          if [ -z "$DOWNSTREAM_TOKEN" ]; then
-            echo "RELEASE_GH_TOKEN/HOMEBREW_TAP_TOKEN is not configured; downstream schedules will update packages."
-            exit 0
-          fi
-
-          for repo in homebrew-archivebox debian-archivebox; do
-            curl --fail-with-body \
-              -X POST \
-              -H "Accept: application/vnd.github+json" \
-              -H "Authorization: Bearer ${DOWNSTREAM_TOKEN}" \
-              -H "X-GitHub-Api-Version: 2022-11-28" \
-              "https://api.github.com/repos/ArchiveBox/${repo}/dispatches" \
-              -d '{"event_type":"archivebox-dev-updated","client_payload":{"ref":"dev"}}'
-          done

.gitignore

@@ -8,6 +8,8 @@ __pycache__/
 .mypy_cache/
 .eggs/
 tests/out/
+/docs/screenshots/
+/publicsite/screenshots/
 
 # Coverage
 .coverage

AGENTS.md

@@ -29,7 +29,7 @@ Run collection commands from inside an initialized data directory:
 ```bash
 cd data
 uv run --project .. archivebox status
-uv run --project .. archivebox add 'https://example.com'
+uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
 uv run --project .. archivebox run
 ```
 
@@ -38,12 +38,11 @@ uv run --project .. archivebox run
 Recommended CLI install:
 
 ```bash
-uv tool install archivebox
+uv tool install --python 3.13 --prerelease allow --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
 mkdir -p ~/archivebox/data
 cd ~/archivebox/data
 archivebox init --install
-archivebox add 'https://example.com'
-archivebox server 0.0.0.0:8000
+archivebox add --plugins=parse_txt_urls 'https://example.com/'
 ```
 
 Alternative install methods:
@@ -56,16 +55,16 @@ Alternative install methods:
 ## Basic Usage
 
 ```bash
+cd ~/archivebox/data
 archivebox version
 archivebox help
 archivebox status
 archivebox install
-archivebox add 'https://example.com'
-archivebox add --extract=title,screenshot,pdf 'https://example.com'
+archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
 archivebox list --json --with-headers
 archivebox search 'example'
 archivebox update --filter-type=domain example.com
-archivebox remove --filter-type=exact 'https://example.com'
+archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
 archivebox run
 ```
 
@@ -73,15 +72,9 @@ archivebox run
 
 Use targeted tests for focused work:
 
-<!--pytest.mark.skip(reason="pytest invocation")-->
 ```bash
-uv run pytest archivebox/tests/test_cli_add.py -q
+uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
 uv run prek run --all-files
 ```
 
-Use the full release/deploy loop only when requested:
-
-<!--pytest.mark.skip(reason="release/deploy script")-->
-```bash
-./bin/release_dev_stack.sh
-```
+Releases are published only by `.github/workflows/release.yml` after the complete `dev` CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.

Dockerfile

@@ -10,10 +10,10 @@
 #       --build-context abx-plugins=../abx-plugins \
 #       -t archivebox/abx-dl:dev
 #   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:latest \
+#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.11.272 \
 #       -t archivebox:multistage
 
-ARG ABX_DL_IMAGE=archivebox/abx-dl:latest
+ARG ABX_DL_IMAGE=archivebox/abx-dl:1.11.272
 
 FROM archivebox/sonic:1.4.9 AS sonic
 FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
@@ -66,9 +66,6 @@ ENV TMP_DIR=/tmp/archivebox \
 ENV HOME=/home/archivebox \
     XDG_CONFIG_HOME=/home/archivebox/.config \
     XDG_CACHE_HOME=/opt/archivebox/lib/cache \
-    ABXPKG_INSTALL_TIMEOUT=600 \
-    ABXPKG_POSTINSTALL_SCRIPTS=True \
-    ABXPKG_MIN_RELEASE_AGE=0 \
     TIMEOUT=600
 
 ENV UV_COMPILE_BYTECODE=false \
@@ -96,7 +93,6 @@ FROM archivebox-runtime-base AS archivebox-builder
 
 WORKDIR "$CODE_DIR"
 RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
-    --mount=type=cache,target=/root/.cache/uv,sharing=locked,id=uv-$TARGETARCH$TARGETVARIANT \
     --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
     <<'EOF'
 echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
@@ -107,8 +103,8 @@ rm -f /etc/apt/apt.conf.d/docker-clean
 apt-get update -qq
 apt-get install -qq -y --no-install-recommends \
     build-essential gcc libldap2-dev libsasl2-dev libssl-dev
-/usr/bin/uv venv --clear /venv --python "${PYTHON_VERSION}"
-/usr/bin/uv pip install setuptools pip wheel
+/usr/bin/uv venv --no-cache --clear /venv --python "${PYTHON_VERSION}"
+/usr/bin/uv pip install --no-cache setuptools pip wheel
 
 mkdir -p /tmp/archivebox-uv-project
 /venv/bin/python - <<'PY'
@@ -129,8 +125,11 @@ text = text.replace(
 # PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
 # Docker-only dependency view from the version JSON so the published package
 # metadata stays normal while image builds remain resumable after a release.
-for package in ("abxpkg", "abx-plugins", "abx-dl"):
-    match = re.search(rf'"{re.escape(package)}>=(?P<version>[^"]+)"', text)
+for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
+    match = re.search(
+        rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
+        text,
+    )
     if not match:
         continue
     version = match.group("version")
@@ -138,7 +137,7 @@ for package in ("abxpkg", "abx-plugins", "abx-dl"):
         data = json.load(response)
     wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
     text = re.sub(
-        rf'"{re.escape(package)}>=[^"]+"',
+        rf'"{re.escape(package)}(?:==|>=)[^"]+"',
         f'"{package} @ {wheel_url}"',
         text,
         count=1,
@@ -149,22 +148,30 @@ PY
 
 /usr/bin/uv sync \
     --project /tmp/archivebox-uv-project \
-    --refresh \
+    --no-cache \
     --no-dev \
     --inexact \
     --no-install-project \
     --no-install-workspace \
     --no-sources
-(find /venv/lib/python3.*/site-packages -type f -name '*.so' -exec strip --strip-unneeded {} + 2>/dev/null || true)
+ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$ABXPKG_LIB_DIR" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
+"$ABXPKG_LIB_DIR/env/bin/find" /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
+while IFS= read -r -d '' native_library; do
+    magic=''
+    if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
+        "$ABXPKG_LIB_DIR/env/bin/strip" --strip-unneeded "$native_library" || exit $?
+    fi
+done < /tmp/archivebox-native-libraries
+rm -f /tmp/archivebox-native-libraries
 rm -f /venv/bin/uv /venv/bin/uvx
-apt-get purge -y build-essential gcc libldap2-dev libsasl2-dev libssl-dev
-apt-get autoremove -y
+abxpkg run --binproviders=env --lib="$ABXPKG_LIB_DIR" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
+abxpkg run --binproviders=env --lib="$ABXPKG_LIB_DIR" apt-get autoremove -y
+"$ABXPKG_LIB_DIR/env/bin/find" "$ABXPKG_LIB_DIR/env/bin" -maxdepth 1 -type l -name strip -delete
 rm -rf /var/lib/apt/lists/*
 EOF
 
 COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
-RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked,id=uv-$TARGETARCH$TARGETVARIANT \
-    echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
+RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
     && COMMIT_HASH="$( \
         if [[ "$ARCHIVEBOX_COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then \
             echo "$ARCHIVEBOX_COMMIT_HASH"; \
@@ -178,7 +185,7 @@ RUN --mount=type=cache,target=/root/.cache/uv,sharing=locked,id=uv-$TARGETARCH$T
             fi; \
         fi)" \
     && if [[ "$COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "COMMIT_HASH=$COMMIT_HASH" | tee -a /VERSION.txt; fi \
-    && /usr/bin/uv pip install --no-deps "$CODE_DIR" \
+    && /usr/bin/uv pip install --no-cache --no-deps "$CODE_DIR" \
     && rm -f /venv/bin/uv /venv/bin/uvx \
     && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
 
@@ -198,7 +205,6 @@ LABEL name="archivebox" \
 COPY --from=sonic /usr/local/bin/sonic /usr/local/bin/sonic
 COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
 
-COPY --from=archivebox-builder /opt/uv/python /opt/uv/python
 COPY --from=archivebox-builder /venv /venv
 COPY --from=archivebox-builder /app /app
 COPY --from=archivebox-builder /VERSION.txt /VERSION.txt
@@ -241,13 +247,11 @@ RUN chmod +x "$CODE_DIR"/bin/*.sh \
     && chmod g+w "$TMP_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
 
 RUN --mount=type=cache,target=/tmp/abxpkg-cache,sharing=locked,mode=1777 \
-    for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
+    chmod 1777 /tmp/abxpkg-cache \
+    && for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
     && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$CONFIG_DIR" \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$ABXPKG_LIB_DIR" \
-    && python3 -c 'from abx_dl.models import discover_plugins; [print(f"export {plugin.enabled_key}=True") for plugin in discover_plugins(runtime="archivebox").values() if plugin.enabled_key in plugin.config.properties]' > /tmp/archivebox-enable-plugins.env \
-    && sort /tmp/archivebox-enable-plugins.env | tee -a /VERSION.txt \
-    && source /tmp/archivebox-enable-plugins.env \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache ABXPKG_NO_CACHE=True ABXPKG_TMP_CACHE_DIR=/tmp/abxpkg-cache archivebox install \
     && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups env HOME=/tmp/abxpkg-cache archivebox version 2>&1 | tee -a /VERSION.txt \
     && rm -rf /root/.cache /var/cache/apt/* /var/lib/apt/lists/*

README.md

@@ -8,14 +8,8 @@
 
 <br/>
 
-<!--<a href="http://webchat.freenode.net?channels=ArchiveBox&uio=d4"><img src="https://img.shields.io/badge/Community_chat-IRC-%2328A745.svg"/></a>-->
-
 <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a> <a href="https://github.com/ArchiveBox/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/ArchiveBox/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a> &nbsp; <a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a> &nbsp; <a href="https://hub.docker.com/r/archivebox/archivebox"><img src="https://img.shields.io/docker/pulls/archivebox/archivebox.svg?label=Docker+Pulls"/></a> <a href="https://pypi.org/project/archivebox/"><img src="https://img.shields.io/pypi/dm/archivebox?label=PyPI%20Installs&color=%235f7dae"/></a> <a href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj"><img src="https://img.shields.io/chrome-web-store/users/habonpimjphpdnmcfkaockjnffodikoj?label=Chrome%20Store&color=%231973e8"/></a>
 
-<!--<pre lang="bash" align="left"><code style="white-space: pre-line; text-align: left" align="left">
-curl -fsSL 'https://get.archivebox.io' | bash    # (or see pip/brew/Docker instructions below)
-</code></pre>-->
-
 </div>
 <hr/>
 <br/>
@@ -29,7 +23,7 @@ Without active preservation effort, everything on the internet eventually disapp
 *ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
 <br/>
 
-> ➡️ Get ArchiveBox with `pip install archivebox>=0.9.5rc1` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.  
+> ➡️ Get ArchiveBox with `uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
 
 *Once installed, you can interact with it through the: [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), [CLI](#usage), [self-hosted web interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python API](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [filesystem](#static-archive-exporting).*
 
@@ -52,7 +46,7 @@ It also detects any content featured *inside* pages & extracts it out into a fol
 - 💾 **Github**/**Gitlab**/etc. links ➡️ `clone of GIT source code`, `README`, `images`, ...
 - ✨ *and more, see [Output Formats](#output-formats) below...*
 
-You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `pip`-installed CLI, Python API, and SQLite3 APIs. 
+You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `uv`-installed CLI, Python API, and SQLite3 APIs.
 All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...
 
 <br/>
@@ -68,7 +62,7 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
 <br/>
 
 
-**📦&nbsp; Install ArchiveBox using your preferred method: `docker` / `pip` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
+**📦&nbsp; Install ArchiveBox using your preferred method: `docker` / `uv` / `brew` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
 
 
 <details>
@@ -77,7 +71,10 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
 <pre lang="bash"><code style="white-space: pre-line"># Option A: Get ArchiveBox with Docker Compose (recommended):
 mkdir -p ~/archivebox/data && cd ~/archivebox
 curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml   # edit options in this file as-needed
+docker compose pull
 docker compose run archivebox init
+docker compose run archivebox install
+docker compose run archivebox manage createsuperuser
 # docker compose run archivebox add 'https://example.com'
 # docker compose run archivebox help
 # docker compose up
@@ -85,14 +82,15 @@ docker compose run archivebox init
 <br/>
 # Option B: Or use it as a plain Docker container:
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -it -v $PWD:/data archivebox/archivebox init
-# docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
-# docker run -it -v $PWD:/data archivebox/archivebox help
-# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox
+docker run -it -v $PWD:/data archivebox/archivebox:dev init
+docker run -it -v $PWD:/data archivebox/archivebox:dev install
+# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
+# docker run -it -v $PWD:/data archivebox/archivebox:dev help
+# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
 <br/>
 <br/>
-# Option C: Or install it with your preferred pkg manager (see Quickstart below for apt, brew, and more)
-pip install 'archivebox>=0.9.5rc1'
+# Option C: Or install it with uv (see Quickstart below for apt, brew, and more)
+uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
 archivebox init
 archivebox install
@@ -106,7 +104,7 @@ curl -fsSL 'https://get.archivebox.io' | bash
 </code></pre>
 <br/>
 <sub>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sub><br/>
-<sub>Set <code>BIND_ADDR</code> to change the base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically.</sub>
+<sub>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sub>
 </details>
 <br/>
 
@@ -188,9 +186,12 @@ ArchiveBox is free for everyone to self-host, but we also provide support, secur
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox
 # Read and edit docker-compose.yml options as-needed after downloading
 curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
+docker compose pull
 </code></pre></li>
-<li>Run the initial setup to create an admin user (or set ADMIN_USER/PASS in docker-compose.yml)
+<li>Initialize the collection, then create an admin user (or set ADMIN_USERNAME/ADMIN_PASSWORD in docker-compose.yml)
 <pre lang="bash"><code style="white-space: pre-line">docker compose run archivebox init
+docker compose run archivebox install
+docker compose run archivebox manage createsuperuser
 </code></pre></li>
 <li>Next steps: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
 <pre lang="bash"><code style="white-space: pre-line">docker compose up
@@ -214,14 +215,15 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
 <li>Create a new empty directory and initialize your collection (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
-docker run -v $PWD:/data -it archivebox/archivebox init
+docker run -v $PWD:/data -it archivebox/archivebox:dev init
+docker run -v $PWD:/data -it archivebox/archivebox:dev install
 </code></pre>
 </li>
 <li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox
+<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
 # completely optional, CLI can always be used without running a server
 # docker run -v $PWD:/data -it [subcommand] [--help]
-docker run -v $PWD:/data -it archivebox/archivebox help
+docker run -v $PWD:/data -it archivebox/archivebox:dev help
 </code></pre>
 <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
 </li>
@@ -256,13 +258,13 @@ See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as a
 
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="Pip" height="28px" align="top"/> <code>pip</code></b> (macOS/Linux/BSD)</summary>
+<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" height="28px" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
 <br/>
 <ol>
 
-<li>Install <a href="https://realpython.com/installing-python/">Python >= v3.13</a> and <a href="https://nodejs.org/en/download/package-manager/">Node >= v22</a> on your system (if not already installed).</li>
-<li>Install the ArchiveBox package using <code>pip3</code> (or <a href="https://docs.astral.sh/uv/guides/tools/#running-tools"><code>uvx</code></a>).
-<pre lang="bash"><code style="white-space: pre-line">pip3 install --upgrade archivebox
+<li>Install <a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> on your system (if not already installed).</li>
+<li>Install the ArchiveBox package using <code>uv</code>.
+<pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
 archivebox version
 # install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions
 #    python@3.13 node curl wget git ripgrep ...
@@ -271,12 +273,13 @@ archivebox version
 </li>
 <li>Create a new empty directory and initialize your collection (can be anywhere).
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data   # for example
-archivebox init     # instantialize a new collection
+archivebox init     # initialize a new collection
 archivebox install  # install all the runtime dependencies (e.g. chrome, single-file, yt-dlp, etc.)
 </code></pre>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Create an admin account, then optionally start the server and log in to the Web UI at <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<pre lang="bash"><code style="white-space: pre-line">archivebox manage createsuperuser
+archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -286,7 +289,7 @@ archivebox help
 
 See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
 <br/>
-<sub>See the <a href="https://github.com/ArchiveBox/pip-archivebox"><code>pip-archivebox</code></a> repo for more details about this distribution.</sub>
+<sub>See the <a href="https://docs.astral.sh/uv/guides/tools/"><code>uv tool</code> documentation</a> for more details about this installation method.</sub>
 <br/><br/>
 </details>
 
@@ -299,19 +302,21 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <pre lang="bash"><code style="white-space: pre-line">echo 'deb [trusted=yes] https://archivebox.github.io/debian-archivebox dev main' | sudo tee /etc/apt/sources.list.d/archivebox.list
 sudo apt update
 sudo apt install archivebox
-archivebox version                         # make sure the package is installed
+(cd /tmp && archivebox version)            # make sure the package is installed
 </code></pre>
 </li>
 <li>Create a new empty directory and initialize your collection (can be anywhere).
-<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
+<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data
+cd ~/archivebox/data
 archivebox init
-sudo archivebox install
+archivebox install
 archivebox add 'https://example.com'
 </code></pre>
 <br/>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Create an admin account, then optionally start the server and log in to the Web UI at <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<pre lang="bash"><code style="white-space: pre-line">archivebox manage createsuperuser
+archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -319,17 +324,18 @@ archivebox help
 </li>
 </ol>
 See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
-<sub>See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for more details about this distribution.</sub>
+<sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details, and run <code>sudo archivebox install</code> only if you want it to install missing system packages via apt.</sub>
 <br/><br/>
 </details>
 
 <details>
-<summary><b><img src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" height="28px" align="top"/> <code>brew</code></b> (macOS only)</summary>
+<summary><b><img src="https://user-images.githubusercontent.com/511499/117447803-f2ec3700-af0b-11eb-87d3-671d114f011d.png" alt="homebrew" height="28px" align="top"/> <code>brew</code></b> (macOS and Linux)</summary>
 <br/>
 <ol>
 <li>Install <a href="https://brew.sh/#install">Homebrew</a> on your system (if not already installed).</li>
 <li>Install the ArchiveBox package using <code>brew</code>.
 <pre lang="bash"><code style="white-space: pre-line">brew tap archivebox/archivebox
+brew trust archivebox/archivebox
 brew install archivebox
 archivebox version                         # make sure all dependencies are installed
 </code></pre>
@@ -341,8 +347,9 @@ archivebox init
 archivebox install
 </code></pre>
 </li>
-<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
-<pre lang="bash"><code style="white-space: pre-line">archivebox server 0.0.0.0:8000
+<li>Create an admin account, then optionally start the server and log in to the Web UI at <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
+<pre lang="bash"><code style="white-space: pre-line">archivebox manage createsuperuser
+archivebox server 0.0.0.0:8000
 # completely optional, CLI can always be used without running a server
 # archivebox [subcommand] [--help]
 archivebox help
@@ -359,11 +366,11 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <summary><img src="https://user-images.githubusercontent.com/511499/118077361-f0616580-b381-11eb-973c-ee894a3349fb.png" alt="Arch" height="28px" align="top"/> <code>pacman</code> / <img src="https://user-images.githubusercontent.com/511499/118077946-29e6a080-b383-11eb-94f0-d4871da08c3f.png" alt="FreeBSD" height="28px" align="top"/> <code>pkg</code> / <img src="https://user-images.githubusercontent.com/511499/118077861-002d7980-b383-11eb-86a7-5936fad9190f.png" alt="Nix" height="28px" align="top"/> <code>nix</code> (Arch/FreeBSD/NixOS/more)</summary>
 <br/>
 
-> *Warning: These are contributed by external volunteers and may lag behind the official `pip` channel.*
+> *Warning: These are contributed by external volunteers and may lag behind the official `uv` and Docker channels.*
 
 <ul>
 <li>Arch: <a href="https://aur.archlinux.org/packages/archivebox/"><code>yay -S archivebox</code></a> (contributed by <a href="https://github.com/imlonghao"><code>@imlonghao</code></a>, maintained by <a href="https://github.com/jasongodev"><code>@jasongodev</code></a>)</li>
-<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>pkg</code> + <code>pip3</code> under-the-hood)</li>
+<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>pkg</code> + <code>uv</code> under-the-hood)</li>
 <li>Nix: <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/archivebox/default.nix"><code>nix-env --install archivebox</code></a> (contributed by <a href="https://github.com/siraben"><code>@siraben</code></a>)</li>
 <li>Guix: <a href="https://packages.guix.gnu.org/packages/archivebox/"><code>guix install archivebox</code></a> (contributed by <a href="https://github.com/rakino"><code>@rakino</code></a>)</li>
 <li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new"><i>contribute another distribution...!</i></a></li>
@@ -476,22 +483,6 @@ For more discussion on managed and paid hosting options see here: <a href="https
 ArchiveBox commands can be run in a terminal [directly on your host](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage), or via [Docker](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1)/[Docker Compose](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage).  
 <sup>(depending on how you chose to install it above)</sup>
 
-<!--
-```bash
-set -euo pipefail
-__archivebox_docs_home="$(mktemp -d)"
-export HOME="$__archivebox_docs_home"
-mkdir -p ~/archivebox/data
-cd ~/archivebox/data
-archivebox init
-archivebox version
-archivebox help
-test -f index.sqlite3
-test -d archive
-rm -rf "$__archivebox_docs_home"
-```
--->
-<!--pytest.mark.skip(reason='Mixed host/Docker alternatives include interactive Docker commands')-->
 ```bash
 mkdir -p ~/archivebox/data   # create a new data dir anywhere
 cd ~/archivebox/data         # IMPORTANT: cd into the directory
@@ -503,8 +494,8 @@ archivebox help
 # equivalent: docker compose run archivebox [subcommand] [--help]
 docker compose run archivebox help
 
-# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
-docker run -it -v $PWD:/data archivebox/archivebox help
+# equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
+docker run -it -v $PWD:/data archivebox/archivebox:dev help
 
 # optional: import your browser cookies into a persona for logged-in archiving
 archivebox persona create --import=chrome personal
@@ -525,12 +516,12 @@ archivebox persona create --import=chrome personal
 <summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
 <br/>
 <pre lang="bash"><code style="white-space: pre-line">
-# make sure you have pip-installed ArchiveBox and it's available in your $PATH first  
+# make sure you have uv-installed ArchiveBox and it's available in your $PATH first
 <br/>
 # archivebox [subcommand] [--help]
 mkdir -p ~/archivebox/data && cd ~/archivebox/data
 archivebox init
-sudo archivebox install
+archivebox install
 archivebox add 'https://example.com'
 archivebox version           # get archivebox version info + check dependencies
 archivebox help              # get list of archivebox subcommands that can be run
@@ -565,13 +556,13 @@ docker compose run archivebox add 'https://example.com'
 <pre lang="bash"><code style="white-space: pre-line">
 # make sure you create and cd into in a new empty directory first  
 <br/>
-# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
-docker run -v $PWD:/data -it archivebox/archivebox init
-docker run -v $PWD:/data -it archivebox/archivebox install
-docker run -v $PWD:/data -it archivebox/archivebox version
-docker run -v $PWD:/data -it archivebox/archivebox help
-docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
-# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
+# docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
+docker run -v $PWD:/data -it archivebox/archivebox:dev init
+docker run -v $PWD:/data -it archivebox/archivebox:dev install
+docker run -v $PWD:/data -it archivebox/archivebox:dev version
+docker run -v $PWD:/data -it archivebox/archivebox:dev help
+docker run -v $PWD:/data -it archivebox/archivebox:dev add 'https://example.com'
+# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️</i>
 </details>
@@ -583,7 +574,7 @@ docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
 <pre lang="bash"><code style="white-space: pre-line">
 archivebox shell           # explore the Python library API in a REPL
 sqlite3 ./index.sqlite3    # run SQL queries directly on your index
-ls ./archive/*/index.html  # or inspect snapshot data directly on the filesystem
+find ./archive/users -path '*/snapshots/*/*/*/index.html'  # inspect snapshot data directly
 </code></pre>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage">Python Shell</a>, <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#sql-shell-usage">SQL API</a>, and <a href="https://github.com/ArchiveBox/ArchiveBox#archive-layout">Disk Layout</a> wikis. ➡️</i>
 </details>
@@ -603,12 +594,12 @@ nano docker-compose.yml                        # setup initial ADMIN_USERNAME &
 docker compose up                              # start the server
 <br/>
 # Or with a Docker container:
-docker run -v $PWD:/data -it archivebox/archivebox archivebox manage createsuperuser
-docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
+docker run -v $PWD:/data -it archivebox/archivebox:dev manage createsuperuser
+docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
 </code></pre>
 
 <sup>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sup><br/>
-<sup>Set <code>BIND_ADDR</code> to change the base domain; <code>web.</code> and <code>admin.</code> subdomains are used automatically.</sup>
+<sup>Set <code>BASE_URL</code> to change the public base domain. The default <code>auto</code> mode uses <code>web.</code> and <code>admin.</code> subdomains on <code>*.localhost</code>, but one host for ordinary DNS names. <code>BIND_ADDR</code> only controls the local listen address.</sup>
 <br/><br/>
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#ui-usage">Usage: Web UI</a> wiki. ➡️</i>
 <br/><br/>
@@ -616,10 +607,10 @@ docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
 
 <pre lang="bash"><code style="white-space: pre-line">
 archivebox config --set PUBLIC_ADD_VIEW=True   # allow guests to submit URLs 
-archivebox config --set PUBLIC_SNAPSHOTS=True  # allow guests to see snapshot content
+archivebox config --set PERMISSIONS=public     # make newly added snapshots public
 archivebox config --set PUBLIC_INDEX=True      # allow guests to see list of all snapshots
 # or
-docker compose run archivebox config --set ...
+docker compose run archivebox config --set PERMISSIONS=public
 
 # restart the server to apply any config changes
 </code></pre>
@@ -630,15 +621,15 @@ docker compose run archivebox config --set ...
 
 > [!TIP]
 > Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.
-> For example, you could run the Web UI in Docker Compose, and run one-off commands with `pip`-installed ArchiveBox.
+> For example, you could run the Web UI in Docker Compose, and run one-off commands with `uv`-installed ArchiveBox.
 
 <details>
 <summary><i>Expand to show comparison...</i></summary><br/>
 
 <pre lang="bash"><code style="white-space: pre-line">
-archivebox add --depth=1 'https://example.com'                     # add a URL with pip-installed archivebox on the host
+archivebox add --depth=1 'https://example.com'                     # add a URL with uv-installed archivebox on the host
 docker compose run archivebox add --depth=1 'https://example.com'                       # or w/ Docker Compose
-docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com'  # or w/ Docker, all equivalent
+docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com'  # or w/ Docker, all equivalent
 </code></pre>
 
 <i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">Docker</a> wiki. ➡️</i>
@@ -696,32 +687,6 @@ docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://exampl
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/e1e5bd78-b0b6-45dc-914c-e1046fee4bc4" width="330px" align="right" style="float: right"/>
 
 
-<!--
-```bash
-set -euo pipefail
-__archivebox_docs_home="$(mktemp -d)"
-export HOME="$__archivebox_docs_home"
-mkdir -p ~/archivebox/data ~/Downloads
-cd ~/archivebox/data
-archivebox init
-cat > ~/Downloads/some_feed.xml <<'EOF'
-<?xml version="1.0" encoding="UTF-8"?>
-<rss version="2.0">
-  <channel>
-    <title>ArchiveBox docs test feed</title>
-    <link>https://example.com/</link>
-    <description>ArchiveBox docs test feed</description>
-    <item>
-      <title>Feed item</title>
-      <link>https://example.com/from-feed</link>
-      <guid>https://example.com/from-feed</guid>
-    </item>
-  </channel>
-</rss>
-EOF
-```
--->
-<!--pytest-codeblocks:cont-->
 ```bash
 # archivebox add --help
 archivebox add 'https://example.com/some/page'
@@ -731,23 +696,10 @@ echo 'http://example.com' | archivebox add
 echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add
 
 # if using Docker, add -i when piping stdin:
-# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox add
+# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add
 # if using Docker Compose, add -T when piping stdin / stdout:
 # echo 'https://example.com' | docker compose run -T archivebox add
 ```
-<!--pytest-codeblocks:cont-->
-<!--
-```bash
-archivebox list --json > snapshots.json
-grep -q 'https://example.com/some/page' snapshots.json
-grep -q 'https://example.com/from-feed' snapshots.json
-grep -q 'https://news.ycombinator.com#2020-12-12' snapshots.json
-grep -q 'http://example.com' snapshots.json
-test -d archive
-test "$(find archive -mindepth 1 -maxdepth 1 -type d | wc -l | tr -d ' ')" -ge 4
-rm -rf "$__archivebox_docs_home"
-```
--->
 
 See the [Usage: CLI](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#CLI-Usage) page for documentation and examples.
 
@@ -804,10 +756,10 @@ ArchiveBox can be configured via environment variables, by using the `archivebox
 archivebox config --get CHROME_BINARY           # view a specific value
 <br/>
 archivebox config --set CHROME_BINARY=chromium  # persist a config using CLI
+# OR edit ArchiveBox.conf and add this under its existing [ARCHIVING_CONFIG] section:
+CHROME_BINARY=chromium
 # OR
-echo CHROME_BINARY=chromium >> ArchiveBox.conf  # persist a config using file
-# OR
-env CHROME_BINARY=chromium archivebox ...       # run with a one-off config
+env CHROME_BINARY=chromium archivebox version   # run with a one-off config
 </code></pre>
 <sub>These methods also work the same way when run inside Docker, see the <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration">Docker Configuration</a> wiki page for details.</sub>
 </details><br/>
@@ -825,7 +777,7 @@ TIMEOUT=240                # default: 60    add more seconds on slower networks
 CHECK_SSL_VALIDITY=False   # default: True  False = allow saving URLs w/ bad SSL
 <br/>
 PUBLIC_INDEX=True          # default: True  whether anon users can view index
-PUBLIC_SNAPSHOTS=True      # default: True  whether anon users can view pages
+PERMISSIONS=public         # default: public visibility for newly added snapshots
 PUBLIC_ADD_VIEW=False      # default: False whether anon users can add new URLs
 <br/>
 USER_AGENT="Mozilla/5.0 ..."  # change this to get around bot blocking
@@ -878,14 +830,10 @@ These optional subdependencies used for archiving sites include:
 <li>and more as we grow...</li>
 </ul>
 
-You don't need to install every dependency to use ArchiveBox. ArchiveBox will automatically disable extractors that rely on dependencies that aren't installed, based on what is configured and available in your <code>$PATH</code>.
-  
-If not using Docker, make sure to keep the dependencies up-to-date yourself and check that ArchiveBox isn't reporting any incompatibility with the versions you install.
+You don't need to install every dependency by hand. ArchiveBox resolves every extractor dependency through <code>abxpkg</code>: it uses a compatible host installation when one is already available, and otherwise installs and manages the dependency for you.
 
-<pre lang="bash"><code style="white-space: pre-line">#install python3 and archivebox with your system package manager
-# apt/brew/pip/etc install ... (see Quickstart instructions above)
+<pre lang="bash"><code style="white-space: pre-line"># install uv + archivebox first (see Quickstart instructions above)
 <br/>
-which -a archivebox    # see where you have installed archivebox
 archivebox install     # auto install all the extractors and extras
 archivebox --version   # see info and check validity of installed dependencies
 </code></pre>
@@ -918,7 +866,7 @@ All <code>archivebox</code> CLI commands are designed to be run from inside an A
 <pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data   # just an example, can be anywhere
 archivebox init</code></pre>
 
-The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard <code>index.sqlite3</code> database in the root of the data folder (it can also be <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">exported as static JSON/HTML</a>), and the archive snapshots are organized by date-added timestamp in the <code>data/archive/</code> subfolder.
+The on-disk layout is optimized to be easy to browse by hand and durable long-term. The main index is a standard <code>index.sqlite3</code> database in the root of the data folder (it can also be <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">exported as static JSON/HTML</a>). Snapshot data is organized by user, date, domain, and UUID under <code>data/archive/users/</code>.
 
 <img src="https://user-images.githubusercontent.com/511499/117453293-c7b91600-af12-11eb-8a3f-aa48b0f9da3c.png" width="400px" align="right" style="float: right"/>
 
@@ -927,18 +875,23 @@ The on-disk layout is optimized to be easy to browse by hand and durable long-te
     index.sqlite3
     ArchiveBox.conf
     archive/
-        ...
-        1617687755/
-            index.html
-            index.json
-            screenshot.png
-            media/some_video.mp4
-            warc/1617687755.warc.gz
-            git/somerepo.git
-            ...
+        1617687755 -> users/admin/snapshots/20210406/example.com/SNAPSHOT_UUID/
+        users/
+            admin/
+                snapshots/
+                    20210406/
+                        example.com/
+                            SNAPSHOT_UUID/
+                                index.html
+                                index.jsonl
+                                screenshot/screenshot.png
+                                ytdlp/media/some_video.mp4
+                                wget/warc/example.com.warc.gz
+                                git/somerepo.git
+                                ...
 </code></pre>
 
-Each snapshot subfolder <code>data/archive/TIMESTAMP/</code> includes a static <code>index.json</code> and <code>index.html</code> describing its contents, and the snapshot extractor outputs are plain files within the folder.
+Each snapshot subfolder includes static metadata and plain extractor output files. ArchiveBox also maintains a backwards-compatible <code>data/archive/TIMESTAMP</code> symlink for each snapshot.
 
 <h4>Learn More</h4>
 <ul>
@@ -972,7 +925,7 @@ archivebox list --json --with-headers > index.json     # export to json blob
 archivebox list --csv=timestamp,url,title > index.csv  # export to csv spreadsheet
 
 # (if using Docker Compose, add the -T flag when piping)
-# docker compose run -T archivebox list --html 'https://example.com' > index.json
+# docker compose run -T archivebox list --html 'https://example.com' > index.html
 </code></pre>
 
 The paths in the static exports are relative, make sure to keep them next to your `./archive` folder when backing them up or viewing them.
@@ -982,7 +935,7 @@ The paths in the static exports are relative, make sure to keep them next to you
 <ul>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#2-export-and-host-it-as-static-html">Wiki: Publishing Your Archive (Exporting as Static HTML)</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#publishing">Wiki: Security Overview (Publishing)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#public_index--public_snapshots--public_add_view">Wiki: Configuration (<code>PUBLIC_INDEX</code>, <code>PUBLIC_SNAPSHOTS</code>, <code>PUBLIC_ADD_VIEW</code>)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#permissions">Wiki: Configuration (<code>PERMISSIONS</code>, <code>PUBLIC_INDEX</code>, <code>PUBLIC_ADD_VIEW</code>)</a></li>
 </ul>
 
 </details>
@@ -1013,7 +966,7 @@ archivebox add 'https://vimeo.com/somePrivateVideo'
 
 # restrict the main index, Snapshot content, and Add Page to authenticated users as-needed:
 archivebox config --set PUBLIC_INDEX=False
-archivebox config --set PUBLIC_SNAPSHOTS=False
+archivebox config --set PERMISSIONS=private
 archivebox config --set PUBLIC_ADD_VIEW=False
 archivebox manage createsuperuser
 </code></pre>
@@ -1029,8 +982,8 @@ archivebox manage createsuperuser
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive">Wiki: Publishing Your Archive</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview">Wiki: Security Overview</a></li>
 <li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile">Wiki: Chromium Install (Setting Up a User Profile)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#chrome_user_data_dir">Wiki: Configuration (<code>CHROME_USER_DATA_DIR</code>)</a></li>
-<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#cookies_file">Wiki: Configuration (<code>COOKIES_FILE</code>)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">Wiki: Personas (browser profiles and cookies)</a></li>
+<li><a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#default_persona">Wiki: Configuration (<code>DEFAULT_PERSONA</code>)</a></li>
 </ul>
 
 </details>
@@ -1039,7 +992,7 @@ archivebox manage createsuperuser
 
 ### Security Risks of Viewing Archived JS
 
-Be aware that malicious archived JS can access the contents of other pages in your archive when viewed. Because the Web UI serves all viewed snapshots from a single domain, they share a request context and **typical CSRF/CORS/XSS/CSP protections do not work to prevent cross-site request attacks**. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview#stealth-mode) page and [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239) for more details.
+Archived JavaScript is untrusted content. The default <code>SERVER_SECURITY_MODE=auto</code> uses isolated subdomains with full replay on <code>*.localhost</code>, and a one-domain no-JS replay policy on ordinary public or LAN hostnames. Choose <code>safe-subdomains-fullreplay</code> only when wildcard DNS and TLS are configured. See the [Security Overview](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview) and [Issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239) for details.
 
 
 <br/>
@@ -1047,18 +1000,16 @@ Be aware that malicious archived JS can access the contents of other pages in yo
 <summary><i>Expand to see risks and mitigations...</i></summary>
 
 
-<pre lang="bash"><code style="white-space: pre-line"># visiting an archived page with malicious JS:
-https://127.0.0.1:8000/archive/1602401954/example.com/index.html
+<pre lang="bash"><code style="white-space: pre-line"># Explicit wildcard mode: full replay on isolated snapshot subdomains
+archivebox config --set SERVER_SECURITY_MODE=safe-subdomains-fullreplay
 
-# example.com/index.js can now make a request to read everything from:
-https://127.0.0.1:8000/index.html
-https://127.0.0.1:8000/archive/*
-# then example.com/index.js can send it off to some evil server
+# Alternative for deployments without wildcard subdomains: disable JS replay
+archivebox config --set SERVER_SECURITY_MODE=safe-onedomain-nojsreplay
 </code></pre>
 
 <blockquote>
 <p><em>NOTE: Only the <code>wget</code> &amp; <code>dom</code> extractor methods execute archived JS when viewing snapshots, all other archive methods produce static output that does not execute JS on viewing.</em><br/>
-<em>If you are worried about these issues ^ you should disable these extractors using:<br/> <code>archivebox config --set SAVE_WGET=False SAVE_DOM=False</code>.</em></p>
+<em>If you do not need JavaScript-capable replay at all, you can also disable those extractors with:<br/> <code>archivebox config --set WGET_ENABLED=False DOM_ENABLED=False</code>.</em></p>
 </blockquote>
 
 <h4>Learn More</h4>
@@ -1085,7 +1036,7 @@ For various reasons, many large sites (Reddit, Twitter, Cloudflare, etc.) active
 
 <ul>
 <li>Set <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#curl_user_agent"><code>CHROME_USER_AGENT</code>, <code>WGET_USER_AGENT</code>, <code>CURL_USER_AGENT</code></a> to impersonate a real browser (by default, ArchiveBox reveals that it's a bot when using the default user agent settings)</li>
-<li>Set up a logged-in browser session for archiving using <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Chromium-Install#setting-up-a-chromium-user-profile"><code>CHROME_USER_DATA_DIR</code> &amp; <code>COOKIES_FILE</code></a></li>
+<li>Set up a logged-in browser session for archiving by <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Personas">importing a dedicated browser profile into a persona</a></li>
 <li>Rewrite your URLs before archiving to swap in alternative frontends that are more bot-friendly e.g.<br>
 <code>reddit.com/some/url</code> -&gt; <code>teddit.net/some/url</code>: <a href="https://github.com/mendel5/alternative-front-ends">https://github.com/mendel5/alternative-front-ends</a></li>
 </ul>
@@ -1098,7 +1049,7 @@ In the future we plan on adding support for running JS scripts during archiving
 
 ### Saving Multiple Snapshots of a Single URL
 
-ArchiveBox appends a hash with the current date `https://example.com#2020-10-24` to differentiate when a single URL is archived multiple times.
+ArchiveBox can preserve multiple snapshots of the same URL. The default <code>ONLY_NEW=True</code> skips URLs already in the collection; use <code>--no-only-new</code> when you intentionally want another snapshot.
 
 
 <br/>
@@ -1107,16 +1058,13 @@ ArchiveBox appends a hash with the current date `https://example.com#2020-10-24`
 <br/>
 
 
-Because ArchiveBox uniquely identifies snapshots by URL, it must use a workaround to take multiple snapshots of the same URL (otherwise they would show up as a single Snapshot entry). It makes the URLs of repeated snapshots unique by adding a hash with the archive date at the end:
+Each re-archive creates a distinct Snapshot row for the same URL:
 
-<pre lang="bash"><code style="white-space: pre-line">archivebox add 'https://example.com#2020-10-24'
-...
-archivebox add 'https://example.com#2020-10-25'
+<pre lang="bash"><code style="white-space: pre-line">archivebox add 'https://example.com'
+archivebox add --no-only-new 'https://example.com'
 </code></pre>
 
-The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI is a shortcut for this hash-date multi-snapshotting workaround.
-
-Improved support for saving multiple snapshots of a single URL without this hash-date workaround will be <a href="https://github.com/ArchiveBox/ArchiveBox/issues/179">added eventually</a> (along with the ability to view diffs of the changes between runs).
+The <img src="https://user-images.githubusercontent.com/511499/115942091-73c02300-a476-11eb-958e-5c1fc04da488.png" alt="Re-Snapshot Button" height="24px"/> button in the Admin UI performs the same explicit re-archive.
 
 <h4>Learn More</h4>
 
@@ -1243,7 +1191,7 @@ ArchiveBox's stance is that duplication of other people's content is only ethica
 
 In the U.S., <a href="https://guides.library.oregonstate.edu/copyright/libraries">libraries, researchers, and archivists</a> are allowed to duplicate copyrighted materials under <a href="https://libguides.ala.org/copyright/fairuse">"fair use"</a> for <a href="https://guides.cuny.edu/cunyfairuse/librarians#:~:text=One%20of%20these%20specified%20conditions,may%20be%20liable%20for%20copyright">private study, scholarship, or research</a>. Archive.org's non-profit preservation work is <a href="https://blog.archive.org/2024/03/01/fair-use-in-action-at-the-internet-archive/">covered under fair use</a> in the US, and they properly handle <a href="https://cardozoaelj.com/2015/03/20/use-of-copyright-law-to-take-down-revenge-porn/">unethical content</a>/<a href="https://help.archive.org/help/rights/">DMCA</a>/<a href="https://gdpr.eu/right-to-be-forgotten/#:~:text=An%20individual%20has%20the%20right,that%20individual%20withdraws%20their%20consent.">GDPR</a> removal requests to maintain good standing in the eyes of the law.
 
-As long as you A. don't try to profit off pirating copyrighted content and B. have processes in place to respond to removal requests, many countries allow you to use software like ArchiveBox to ethically and responsibly archive any web content you can view. That being said, ArchiveBox is not liable for how you choose to operate the software. You must research your own local laws and regulations, and get proper legal council if you plan to host a public instance (start by putting your DMCA/GDPR contact info in <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#footer_info"><code>FOOTER_INFO</code></a> and changing your instance's branding using <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#custom_templates_dir"><code>CUSTOM_TEMPLATES_DIR</code></a>).
+As long as you A. don't try to profit off pirating copyrighted content and B. have processes in place to respond to removal requests, many countries allow you to use software like ArchiveBox to ethically and responsibly archive any web content you can view. That being said, ArchiveBox is not liable for how you choose to operate the software. You must research your own local laws and regulations, and get proper legal counsel if you plan to host a public instance (start by putting your DMCA/GDPR contact info in <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#footer_info"><code>FOOTER_INFO</code></a> and placing branding overrides in your collection's fixed <code>custom_templates/</code> directory).
 
 </details>
 <br/>
@@ -1293,8 +1241,6 @@ ArchiveBox is neither the highest fidelity nor the simplest tool available for s
 
 <br/>
 
-<!--<div align="center" style="text-align: center"><br/><img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/04808ac2-3133-44fd-8703-3387e06dc851" width="100%" alt="dependencies graphic"></div>-->
-
 ## Internet Archiving Ecosystem
 
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/78d8a725-97f4-47f5-b983-1f62843ddc51" width="14%" align="right" style="float: right"/>
@@ -1393,7 +1339,7 @@ All contributions to ArchiveBox are welcomed! Check our [issues](https://github.
 
 For low hanging fruit / easy first tickets, see: <a href="https://github.com/ArchiveBox/ArchiveBox/issues?q=is%3Aissue+is%3Aopen+sort%3Aupdated-desc+label%3A%22help+wanted%22">ArchiveBox/Issues `#good first ticket` `#help wanted`</a>.
 
-**Python API Documentation:** https://docs.archivebox.io/en/dev/archivebox.html#module-archivebox.main
+**Python API Documentation:** https://docs.archivebox.io/dev/apidocs/
 
 **Internal Architecture Diagrams:** https://github.com/ArchiveBox/ArchiveBox/wiki/ArchiveBox-Architecture-Diagrams
 
@@ -1406,7 +1352,6 @@ For low hanging fruit / easy first tickets, see: <a href="https://github.com/Arc
 
 First make sure you have `uv` installed: https://docs.astral.sh/uv/getting-started/installation/
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 git clone https://github.com/ArchiveBox/monorepo
 cd monorepo
@@ -1428,7 +1373,6 @@ Repos included in monorepo setup:
 
 #### 2. Option A: Install the Python, JS, and system dependencies directly on your machine
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 # Install ArchiveBox runtime dependencies
 mkdir -p data && cd data
@@ -1444,14 +1388,14 @@ archivebox server 0.0.0.0:8000
 
 #### 2. Option B: Build the docker container and use that for development instead
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 # Optional: develop via docker by mounting the code dir into the container
 # if you edit e.g. ./archivebox/core/models.py on the docker host, runserver
 # inside the container will reload and pick up your changes
 ./bin/build_docker.sh dev
 
-docker run -it -v $PWD/data:/data archivebox/archivebox:dev init --install
+docker run -it -v $PWD/data:/data archivebox/archivebox:dev init
+docker run -it -v $PWD/data:/data archivebox/archivebox:dev install
 
 # Run the development server w/ autoreloading (but no bg workers)
 docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
@@ -1474,7 +1418,6 @@ You can also run all these in Docker. For more examples see the GitHub Actions C
 
 <details><summary><i>Click to expand...</i></summary>
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 # set up persistent DEBUG=True for all runs
 archivebox config --set DEBUG=True
@@ -1499,7 +1442,6 @@ https://stackoverflow.com/questions/1074212/how-can-i-see-the-raw-sql-queries-dj
 
 If you're looking for the latest `dev` Docker image, it's often available pre-built on Docker Hub, simply pull and use `archivebox/archivebox:dev`.
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 docker pull archivebox/archivebox:dev
 docker run archivebox/archivebox:dev version
@@ -1510,7 +1452,6 @@ docker run archivebox/archivebox:dev version
 
 You can also build and run any branch yourself from source, for example to build & use `dev` locally:
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 # docker-compose.yml:
 services:
@@ -1523,9 +1464,8 @@ services:
 docker build -t archivebox:dev https://github.com/ArchiveBox/ArchiveBox.git#dev
 docker run -it -v $PWD:/data archivebox:dev init
 
-# or with pip:
-pip install 'git+https://github.com/pirate/ArchiveBox@dev'
-npm install 'git+https://github.com/ArchiveBox/ArchiveBox.git#dev'
+# or with uv:
+uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
 archivebox install
 ```
 
@@ -1535,7 +1475,6 @@ archivebox install
 
 <details><summary><i>Click to expand...</i></summary>
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 ./bin/lint.sh
 ./bin/test.sh
@@ -1549,11 +1488,9 @@ archivebox install
 
 <details><summary><i>Click to expand...</i></summary>
 
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 # generate the database migrations after changes to models.py
-cd archivebox/
-./manage.py makemigrations
+archivebox manage makemigrations
 
 # enter a python shell or a SQL shell
 cd path/to/test/data/
@@ -1562,8 +1499,7 @@ archivebox manage dbshell
 
 # generate a graph of the ORM models
 brew install graphviz
-pip install pydot graphviz
-archivebox manage graph_models -a -o orm.png
+uv run --with pydot --with graphviz archivebox manage graph_models -a -o orm.png
 open orm.png
 
 # list all models with field db info and methods
@@ -1571,7 +1507,7 @@ archivebox manage list_model_info --all --signature --db-type --field-class
 
 # print all django settings
 archivebox manage print_settings
-archivebox manage print_settings --format=yaml    # pip install pyyaml
+uv run --with pyyaml archivebox manage print_settings --format=yaml
 
 # autogenerate an admin.py from given app models
 archivebox manage admin_generator core > core/admin.py
@@ -1585,9 +1521,8 @@ archivebox manage runscript testdata
 archivebox manage reset_db
 
 # use django-tui to interactively explore commands
-uv pip install django-tui
 # ensure django-tui is in INSTALLED_APPS: core/settings.py
-archivebox manage tui
+uv run --with django-tui archivebox manage tui
 ```
 
 <img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/dc3e9f8c-9544-46e0-a7f0-30f571b72022" width="600px" alt="ArchiveBox ORM models relatinoship graph"/>
@@ -1614,7 +1549,6 @@ Copy a similar plugin as a template to modify, then open a new PR to add it in t
 <details><summary><i>Click to expand...</i></summary>
 
 (Normally CI takes care of this, but these scripts can be run to do it manually)
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
 ```bash
 ./bin/build.sh
 
@@ -1626,23 +1560,6 @@ Copy a similar plugin as a template to modify, then open a new PR to add it in t
 
 </details>
 
-#### Roll a release
-
-<details><summary><i>Click to expand...</i></summary>
-
-(Normally CI takes care of this, but these scripts can be run to do it manually)
-<!--pytest.mark.skip(reason='Development workflow snippets are not part of docs codeblock tests')-->
-```bash
-./bin/release.sh
-
-# or individually:
-./bin/release_docs.sh
-./bin/release_pip.sh
-./bin/release_docker.sh
-```
-
-</details>
-
 ---
 
 ## Further Reading

archivebox/__init__.py

@@ -38,11 +38,6 @@ ASCII_LOGO = """
 
 PACKAGE_DIR = Path(__file__).resolve().parent
 
-# # Add PACKAGE_DIR to sys.path - required for Django migrations to import models
-# # Migrations reference models like 'machine.Binary' which need to be importable
-# if str(PACKAGE_DIR) not in sys.path:
-#     sys.path.append(str(PACKAGE_DIR))
-
 os.environ["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
 os.environ["TZ"] = "UTC"
 
@@ -51,10 +46,9 @@ from .config.permissions import drop_privileges  # noqa
 
 drop_privileges()
 
-from .misc.checks import check_not_root, check_not_inside_source_dir, check_io_encoding  # noqa
+from .misc.checks import check_not_root, check_io_encoding  # noqa
 
 check_not_root()
-check_not_inside_source_dir()
 check_io_encoding()
 
 from .config.version import VERSION  # noqa

archivebox/api/migrations/0001_initial.py

@@ -12,19 +12,9 @@ import signal_webhooks.fields
 import signal_webhooks.utils
 
 
-class Migration(migrations.Migration):
-    initial = True
-
-    dependencies = [
-        ("auth", "0012_alter_user_first_name_max_length"),
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-    ]
-
-    operations = [
-        migrations.SeparateDatabaseAndState(
-            database_operations=[
-                migrations.RunSQL(
-                    sql="""
+# Raw sqlite DDL kept EXACTLY as-is; executed only on sqlite via Django's
+# RunSQL (identical statement splitting → byte-for-byte identical behavior).
+INITIAL_SQL = """
                 -- Create api_apitoken table
                 CREATE TABLE IF NOT EXISTS api_apitoken (
                     id TEXT PRIMARY KEY NOT NULL,
@@ -70,12 +60,60 @@ class Migration(migrations.Migration):
                 CREATE INDEX IF NOT EXISTS api_outboundwebhook_created_at_idx ON api_outboundwebhook(created_at);
                 CREATE INDEX IF NOT EXISTS api_outboundwebhook_name_idx ON api_outboundwebhook(name);
                 CREATE INDEX IF NOT EXISTS api_outboundwebhook_ref_idx ON api_outboundwebhook(ref);
-                    """,
-                    reverse_sql="""
+                    """
+
+INITIAL_REVERSE_SQL = """
                 DROP TABLE IF EXISTS api_outboundwebhook;
                 DROP TABLE IF EXISTS api_apitoken;
-                    """,
-                ),
+                    """
+
+
+def _run_sqlite_only_sql(apps, schema_editor):
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_forwards(
+        "api",
+        schema_editor,
+        None,
+        None,
+    )
+
+
+def _run_sqlite_only_sql_reverse(apps, schema_editor):
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_backwards(
+        "api",
+        schema_editor,
+        None,
+        None,
+    )
+
+
+def _pg_sync_schema(apps, schema_editor):
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "api", ["APIToken", "OutboundWebhook"])
+
+
+def _pg_drop_schema(apps, schema_editor):
+    from archivebox.misc.db import drop_models_on_postgres
+
+    drop_models_on_postgres(apps, schema_editor, "api", ["OutboundWebhook", "APIToken"])
+
+
+class Migration(migrations.Migration):
+    initial = True
+
+    dependencies = [
+        ("auth", "0012_alter_user_first_name_max_length"),
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.SeparateDatabaseAndState(
+            database_operations=[
+                migrations.RunPython(_run_sqlite_only_sql, reverse_code=_run_sqlite_only_sql_reverse),
             ],
             state_operations=[
                 migrations.CreateModel(
@@ -236,4 +274,8 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        # On non-sqlite backends the raw DDL above is skipped, so the real
+        # schema diverges from Django state. Rebuild both tables (including the
+        # unique constraint) from the post-migration state (no-op on sqlite).
+        migrations.RunPython(_pg_sync_schema, reverse_code=_pg_drop_schema),
     ]

archivebox/api/urls.py

@@ -23,7 +23,4 @@ urlpatterns = [
     path("v1/", RedirectView.as_view(url="/api/v1/docs")),
     path("v1/", v1_api_urls),
     path("v1", RedirectView.as_view(url="/api/v1/docs")),
-    # ... v2 can be added here ...
-    # path("v2/",              v2_api_urls),
-    # path("v2",               RedirectView.as_view(url='/api/v2/docs')),
 ]

archivebox/api/v1_api.py

@@ -78,9 +78,6 @@ class NinjaAPIWithIOCapture(NinjaAPI):
         response["X-ArchiveBox-Auth-User-Id"] = str(request.user.pk) if request.user.pk else "None"
         response["X-ArchiveBox-Auth-User-Username"] = request.user.username if isinstance(request.user, User) else "None"
 
-        # import ipdb; ipdb.set_trace()
-        # print('RESPONDING NOW', response)
-
         return response
 
 
@@ -91,8 +88,6 @@ api = NinjaAPIWithIOCapture(
     auth=API_AUTH_METHODS,
     urls_namespace="api-1",
     docs=Swagger(settings={"persistAuthorization": True}),
-    # docs_decorator=login_required,
-    # renderer=ORJSONRenderer(),
 )
 api = register_urls(api)
 urls = api.urls
@@ -113,24 +108,7 @@ def generic_exception_handler(request, err):
             "message": f"{err.__class__.__name__}: {err}",
             "errors": [
                 "".join(format_exception(err)),
-                # or send simpler parent-only traceback:
-                # *([str(err.__context__)] if getattr(err, '__context__', None) else []),
             ],
         },
         status=status,
     )
-
-
-# import orjson
-# from ninja.renderers import BaseRenderer
-# class ORJSONRenderer(BaseRenderer):
-#     media_type = "application/json"
-#     def render(self, request, data, *, response_status):
-#         return {
-#             "success": True,
-#             "errors": [],
-#             "result": data,
-#             "stdout": ansi_to_html(stdout.getvalue().strip()),
-#             "stderr": ansi_to_html(stderr.getvalue().strip()),
-#         }
-#         return orjson.dumps(data)

archivebox/api/v1_cli.py

@@ -15,8 +15,6 @@ from archivebox.misc.util import ansi_to_html
 from archivebox.core.models import SnapshotQuerySet
 
 
-# from .auth import API_AUTH_METHODS
-
 # router for API that exposes archivebox cli subcommands as REST endpoints
 router = Router(tags=["ArchiveBox CLI Sub-Commands"])
 
@@ -54,6 +52,7 @@ class AddCommandSchema(Schema):
     snapshot_max_size: int = 0
     parser: str = "auto"
     plugins: str = ""
+    persona: str = "Default"
     only_new: bool | None = None
     update: bool = False
     overwrite: bool = False
@@ -145,6 +144,7 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
         snapshot_max_size=args.snapshot_max_size,
         index_only=args.index_only,
         plugins=args.plugins,
+        persona=args.persona,
         parser=args.parser,
         bg=True,  # Always run in background for API calls
         created_by_id=request.user.pk,

archivebox/api/v1_core.py

@@ -4,6 +4,7 @@ import math
 import json
 import mimetypes
 import re
+from html import unescape
 from collections import defaultdict
 from pathlib import Path, PurePosixPath
 from uuid import UUID
@@ -49,6 +50,7 @@ from archivebox.core.tag_util import (
     rename_tag as rename_tag_record,
 )
 from archivebox.crawls.models import Crawl
+from archivebox.crawls.locks import crawl_lifecycle_lock
 from archivebox.api.v1_crawls import CrawlSchema, get_crawl_by_ref
 from archivebox.search.config import get_search_mode, get_search_mode_backend
 from archivebox.search.query import apply_snapshot_search
@@ -832,7 +834,7 @@ def _snapshots_rss_response(
         crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
         description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
         feed.add_item(
-            title=snapshot.title or snapshot.url,
+            title=unescape(snapshot.title or snapshot.url),
             link=archived_url or web_base_url,
             description=description,
             unique_id=str(snapshot.id),
@@ -961,37 +963,38 @@ def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
             created_by=request.user if isinstance(request.user, User) else None,
         )
 
-    snapshot_defaults = {
-        "depth": data.depth,
-        "title": data.title,
-        "timestamp": str(timezone.now().timestamp()),
-        "status": status or Snapshot.StatusChoices.QUEUED,
-        "retry_at": timezone.now(),
-    }
-    snapshot, _ = Snapshot.objects.get_or_create(
-        url=data.url,
-        crawl=crawl,
-        defaults=snapshot_defaults,
-    )
+    with crawl_lifecycle_lock(str(crawl.id)):
+        snapshot_defaults = {
+            "depth": data.depth,
+            "title": data.title,
+            "timestamp": str(timezone.now().timestamp()),
+            "status": status or Snapshot.StatusChoices.QUEUED,
+            "retry_at": timezone.now(),
+        }
+        snapshot, _ = Snapshot.objects.get_or_create(
+            url=data.url,
+            crawl=crawl,
+            defaults=snapshot_defaults,
+        )
 
-    update_fields: list[str] = []
-    if data.title is not None and snapshot.title != data.title:
-        snapshot.title = data.title
-        update_fields.append("title")
-    if status is not None and snapshot.status != status:
-        snapshot.status = status
-        update_fields.append("status")
-    if update_fields:
-        update_fields.append("modified_at")
-        snapshot.save(update_fields=update_fields)
+        update_fields: list[str] = []
+        if data.title is not None and snapshot.title != data.title:
+            snapshot.title = data.title
+            update_fields.append("title")
+        if status is not None and snapshot.status != status:
+            snapshot.status = status
+            update_fields.append("status")
+        if update_fields:
+            update_fields.append("modified_at")
+            snapshot.save(update_fields=update_fields)
 
-    if tags:
-        snapshot.save_tags(tags)
+        if tags:
+            snapshot.save_tags(tags)
 
-    try:
-        snapshot.ensure_crawl_symlink()
-    except Exception:
-        pass
+        try:
+            snapshot.ensure_crawl_symlink()
+        except Exception:
+            pass
 
     setattr(request, "with_archiveresults", False)
     return snapshot
@@ -1001,6 +1004,7 @@ def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
 def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateSchema):
     """Update a snapshot (e.g., set status=sealed to cancel queued work)."""
     snapshot = _get_snapshot_by_ref(snapshot_id)
+    crawl_id = str(snapshot.crawl_id)
 
     payload = data.dict(exclude_unset=True)
     update_fields = ["modified_at"]
@@ -1022,26 +1026,28 @@ def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateS
             return snapshot
         raise HttpError(400, f"Invalid action: {action}")
 
-    if "status" in payload:
-        try:
-            snapshot.status = normalize_snapshot_status(payload["status"])
-        except ValueError as err:
-            raise HttpError(400, str(err)) from err
-        if snapshot.status == Snapshot.StatusChoices.SEALED and "retry_at" not in payload:
-            snapshot.retry_at = None
-        update_fields.append("status")
+    with crawl_lifecycle_lock(crawl_id):
+        snapshot = _get_snapshot_by_ref(snapshot_id)
+        if "status" in payload:
+            try:
+                snapshot.status = normalize_snapshot_status(payload["status"])
+            except ValueError as err:
+                raise HttpError(400, str(err)) from err
+            if snapshot.status == Snapshot.StatusChoices.SEALED and "retry_at" not in payload:
+                snapshot.retry_at = None
+            update_fields.append("status")
 
-    if "retry_at" in payload:
-        snapshot.retry_at = payload["retry_at"]
-        update_fields.append("retry_at")
+        if "retry_at" in payload:
+            snapshot.retry_at = payload["retry_at"]
+            update_fields.append("retry_at")
 
-    if tags is not None:
-        snapshot.save_tags(normalize_tag_list(tags))
+        if tags is not None:
+            snapshot.save_tags(normalize_tag_list(tags))
 
-    if payload.get("status") == Snapshot.StatusChoices.SEALED:
-        snapshot.cancel()
-    else:
-        snapshot.save(update_fields=update_fields)
+        if payload.get("status") == Snapshot.StatusChoices.SEALED:
+            snapshot.cancel()
+        else:
+            snapshot.save(update_fields=update_fields)
     setattr(request, "with_archiveresults", False)
     return snapshot
 
@@ -1051,7 +1057,14 @@ def delete_snapshot(request: HttpRequest, snapshot_id: str):
     snapshot = get_snapshot(request, snapshot_id, with_archiveresults=False)
     snapshot_id_str = str(snapshot.id)
     crawl_id_str = str(snapshot.crawl.pk)
-    deleted_count, _ = snapshot.delete()
+    snapshot.cancel()
+
+    from archivebox.services.runner import run_pending_crawls
+
+    with crawl_lifecycle_lock(crawl_id_str):
+        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
+        snapshot = get_snapshot(request, snapshot_id_str, with_archiveresults=False)
+        deleted_count, _ = snapshot.delete()
     return {
         "success": True,
         "snapshot_id": snapshot_id_str,

archivebox/api/v1_crawls.py

@@ -49,8 +49,6 @@ class CrawlSchema(Schema):
     tags_str: str
     config: dict
 
-    # snapshots: List[SnapshotSchema]
-
     @staticmethod
     def resolve_created_by_id(obj):
         return str(obj.created_by_id)
@@ -251,8 +249,16 @@ def patch_crawl(request: HttpRequest, crawl_id: str, data: CrawlUpdateSchema):
 def delete_crawl(request: HttpRequest, crawl_id: str):
     crawl = get_crawl_by_ref(crawl_id)
     crawl_id_str = str(crawl.id)
-    snapshot_count = crawl.snapshot_set.count()
-    deleted_count, _ = crawl.delete()
+    crawl.cancel()
+
+    from archivebox.crawls.locks import crawl_lifecycle_lock
+    from archivebox.services.runner import run_pending_crawls
+
+    with crawl_lifecycle_lock(crawl_id_str):
+        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
+        crawl = get_crawl_by_ref(crawl_id_str)
+        snapshot_count = crawl.snapshot_set.count()
+        deleted_count, _ = crawl.delete()
     return {
         "success": True,
         "crawl_id": crawl_id_str,

archivebox/base_models/admin.py

@@ -5,11 +5,11 @@ __package__ = "archivebox.base_models"
 import json
 import uuid
 from collections.abc import Mapping
-from typing import NotRequired, TypedDict, cast
+from typing import ClassVar, NotRequired, TypedDict, cast
 
 from django import forms
 from django.contrib import admin
-from django.db import models
+from django.db import DatabaseError, models
 from django.forms.renderers import BaseRenderer
 from django.http import HttpRequest, QueryDict
 from django.urls import path, register_converter
@@ -68,10 +68,10 @@ class KeyValueWidget(forms.Widget):
     template_name = ""  # We render manually
 
     class Media:
-        css = {
+        css: ClassVar[dict[str, list[str]]] = {
             "all": [],
         }
-        js = []
+        js: ClassVar[list[str]] = []
 
     def _get_config_options(self) -> dict[str, ConfigOption]:
         """Get available config options from plugins."""
@@ -94,7 +94,7 @@ class KeyValueWidget(forms.Widget):
                             option[schema_key] = schema[schema_key]
                 options[key] = option
             return options
-        except Exception:
+        except (ImportError, KeyError, TypeError, ValueError):
             return {}
 
     def _parse_value(self, value: object) -> dict[str, object]:
@@ -644,41 +644,64 @@ class KeyValueWidget(forms.Widget):
 
                     window.updateHiddenField_{widget_id} = updateHiddenField_{widget_id};
 
-                    function focusConfigKeyFromHash_{widget_id}() {{
-                        // Deep-link affordance: ``…/change/#SOME_KEY`` jumps directly
-                        // to (or creates) the matching row in this editor. Used by
-                        // the in-banner "pin via admin" link and the
-                        // ``→ Edit <KEY> in Machine.config`` shortcut on the live
-                        // config detail page.
-                        var hash = (window.location.hash || '').replace(/^#/, '').trim();
-                        if (!hash || !/^[A-Z][A-Z0-9_]*$/.test(hash)) {{
-                            return;
-                        }}
+                    function configRowForKey_{widget_id}(key) {{
                         var container = document.getElementById('{widget_id}_rows');
                         if (!container) {{
-                            return;
+                            return null;
                         }}
                         var match = null;
                         container.querySelectorAll('.key-value-row').forEach(function(row) {{
                             if (match) {{ return; }}
                             var keyInput = row.querySelector('.kv-key');
-                            if (keyInput && keyInput.value.trim() === hash) {{
+                            if (keyInput && keyInput.value.trim() === key) {{
                                 match = row;
                             }}
                         }});
                         if (!match) {{
-                            // No existing row for this key — prepopulate one with the
-                            // key filled in but value left blank so the operator just
-                            // types/pastes the value and hits save.
                             window.addKeyValueRow_{widget_id}();
                             var rows = container.querySelectorAll('.key-value-row');
                             match = rows[rows.length - 1];
                             var keyInput = match.querySelector('.kv-key');
                             if (keyInput) {{
-                                keyInput.value = hash;
+                                keyInput.value = key;
                                 keyInput.dispatchEvent(new Event('input', {{ bubbles: true }}));
                             }}
                         }}
+                        return match;
+                    }}
+
+                    function prefillConfigFromQuery_{widget_id}() {{
+                        var params = new URLSearchParams(window.location.search);
+                        var consumedConfigKeys = [];
+                        params.forEach(function(value, key) {{
+                            if (!/^[A-Z][A-Z0-9_]*$/.test(key) || !configMeta_{widget_id}[key]) {{
+                                return;
+                            }}
+                            consumedConfigKeys.push(key);
+                            var match = configRowForKey_{widget_id}(key);
+                            var valueInput = match && match.querySelector('.kv-value');
+                            if (valueInput) {{
+                                valueInput.value = value;
+                                valueInput.dispatchEvent(new Event('input', {{ bubbles: true }}));
+                            }}
+                        }});
+                        updateHiddenField_{widget_id}();
+                        if (consumedConfigKeys.length) {{
+                            var cleanUrl = new URL(window.location.href);
+                            consumedConfigKeys.forEach(function(key) {{ cleanUrl.searchParams.delete(key); }});
+                            window.history.replaceState(null, '', cleanUrl.pathname + cleanUrl.search + cleanUrl.hash);
+                        }}
+                    }}
+
+                    function focusConfigKeyFromHash_{widget_id}() {{
+                        // Deep-link affordance: ``…/change/#SOME_KEY`` jumps directly
+                        // to (or creates) the matching row in this editor. Used by
+                        // the setup wizard and live-config detail pages.
+                        var hash = (window.location.hash || '').replace(/^#/, '').trim();
+                        if (!hash || !/^[A-Z][A-Z0-9_]*$/.test(hash)) {{
+                            return;
+                        }}
+                        var match = configRowForKey_{widget_id}(hash);
                         if (!match) {{
                             return;
                         }}
@@ -700,12 +723,14 @@ class KeyValueWidget(forms.Widget):
                     // Initialize on load
                     document.addEventListener('DOMContentLoaded', function() {{
                         initializeRows_{widget_id}();
+                        prefillConfigFromQuery_{widget_id}();
                         updateHiddenField_{widget_id}();
                         focusConfigKeyFromHash_{widget_id}();
                     }});
                     // Also run immediately in case DOM is already ready
                     if (document.readyState !== 'loading') {{
                         initializeRows_{widget_id}();
+                        prefillConfigFromQuery_{widget_id}();
                         updateHiddenField_{widget_id}();
                         focusConfigKeyFromHash_{widget_id}();
                     }}
@@ -826,7 +851,7 @@ class ConfigEditorMixin(admin.ModelAdmin):
         if change and obj.pk and obj.config is not None:
             try:
                 stored = type(obj).objects.filter(pk=obj.pk).values_list("config", flat=True).first() or {}
-            except Exception:
+            except (AttributeError, DatabaseError, TypeError, ValueError):
                 stored = {}
             if isinstance(stored, dict):
                 new_config = dict(obj.config or {})
@@ -845,6 +870,20 @@ class BaseModelAdmin(DjangoObjectActions, admin.ModelAdmin):
     list_display = ("id", "created_at", "created_by")
     readonly_fields = ("id", "created_at", "modified_at")
     show_search_mode_selector = False
+    change_form_template = "admin/archivebox_change_form.html"
+
+    def get_admin_toolbar_actions(self, request, obj):
+        """Return extra action button dicts for the shared change-form toolbar.
+
+        See ``templates/admin/includes/archivebox_toolbar.html`` for the
+        accepted keys. Default: no extras (toolbar renders Save/History/Delete
+        plus any ``django-object-actions`` change_actions).
+        """
+        return []
+
+    def render_change_form(self, request, context, add=False, change=False, form_url="", obj=None):
+        context.setdefault("archivebox_admin_actions", self.get_admin_toolbar_actions(request, obj))
+        return super().render_change_form(request, context, add=add, change=change, form_url=form_url, obj=obj)
 
     def get_default_search_mode(self) -> str:
         # The shared changelist template always asks every admin for a default
@@ -877,9 +916,9 @@ class BaseModelAdmin(DjangoObjectActions, admin.ModelAdmin):
         """
         info = self.opts.app_label, self.opts.model_name
         object_routes = [
-            path("<hexuuid:object_id>/history/", self.admin_site.admin_view(self.history_view), name="%s_%s_history" % info),
-            path("<hexuuid:object_id>/delete/", self.admin_site.admin_view(self.delete_view), name="%s_%s_delete" % info),
-            path("<hexuuid:object_id>/change/", self.admin_site.admin_view(self.change_view), name="%s_%s_change" % info),
+            path("<hexuuid:object_id>/history/", self.admin_site.admin_view(self.history_view), name="{}_{}_history".format(*info)),
+            path("<hexuuid:object_id>/delete/", self.admin_site.admin_view(self.delete_view), name="{}_{}_delete".format(*info)),
+            path("<hexuuid:object_id>/change/", self.admin_site.admin_view(self.change_view), name="{}_{}_change".format(*info)),
         ]
         # Append after super().get_urls() so our patterns are the
         # *last-registered* ones with the canonical admin URL names — Django's

archivebox/base_models/apps.py

@@ -1,7 +0,0 @@
-# from django.apps import AppConfig
-
-
-# class BaseModelsConfig(AppConfig):
-#     default_auto_field = 'django.db.models.BigAutoField'
-
-#     name = 'base_models'

archivebox/cli/__init__.py

@@ -128,16 +128,12 @@ class ArchiveBoxGroup(click.Group):
             import_path = cmd_name_or_path
         modname, funcname = import_path.rsplit(".", 1)
 
-        # print(f'LAZY LOADING {import_path}')
         mod = import_module(modname)
         func = vars(mod)[funcname]
 
         if func.__doc__ is None:
             raise ValueError(f"lazy loading of {import_path} failed - no docstring found on method")
 
-        # if not isinstance(cmd, click.BaseCommand):
-        # raise ValueError(f'lazy loading of {import_path} failed - not a click command')
-
         return func
 
 
@@ -158,7 +154,6 @@ def cli(ctx, help=False):
     # then we need to set up the django environment and check that we're in a valid data folder
     wants_help = any(arg in ("-h", "--help", "--version") for arg in sys.argv[1:])
     if not wants_help and (subcommand in ArchiveBoxGroup.archive_commands or subcommand in ArchiveBoxGroup.model_commands):
-        # print('SETUP DJANGO AND CHECK DATA FOLDER')
         try:
             if subcommand == "server":
                 run_in_debug = "--reload" in sys.argv or os.environ.get("DEBUG") in ("1", "true", "True", "TRUE", "yes")

archivebox/cli/archivebox_add.py

@@ -133,6 +133,17 @@ def add(
     )
 
     # 2. Create a new Crawl with inline URLs
+    # Foreground add must claim runner ownership before publishing runnable
+    # work. Otherwise an existing server runner can lease the new Crawl in the
+    # gap between Crawl.objects.create() and current_command(), then be killed
+    # by the newer add process halfway through its first Snapshot hook.
+    command = None
+    if not bg and not index_only:
+        from archivebox.machine.models import Process
+        from archivebox.core.takeover_util import current_command
+
+        command = current_command(Process.TypeChoices.ADD, data_dir=CONSTANTS.DATA_DIR)
+
     cli_args = [*sys.argv]
     if cli_args[0].lower().endswith("archivebox"):
         cli_args[0] = "archivebox"
@@ -173,25 +184,33 @@ def add(
     # this add path. Keeping it byte-for-byte readable is what lets API/UI/CLI
     # callers audit or resume imports without losing RSS/Netscape/JSON metadata
     # that is not representable as one plain URL per line.
-    crawl = Crawl.objects.create(
-        urls=source_text,
-        # Stdin/import text gets an extra hop because the synthetic
-        # archivebox://internal root lives at depth 0 and parser-discovered
-        # URLs land at depth 1; direct URL args become the depth=0 input
-        # snapshots themselves so --depth=N matches the deepest hop the user
-        # asked for.
-        max_depth=depth + 1 if use_internal_input_root else depth,
-        tags_str=tag,
-        persona_id=persona_obj.id,
-        label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
-        created_by_id=created_by_id,
-        status=Crawl.StatusChoices.QUEUED,
-        retry_at=None if index_only else timezone.now(),
-        config=crawl_config,
-    )
+    try:
+        crawl = Crawl.objects.create(
+            urls=source_text,
+            # Stdin/import text gets an extra hop because the synthetic
+            # archivebox://internal root lives at depth 0 and parser-discovered
+            # URLs land at depth 1; direct URL args become the depth=0 input
+            # snapshots themselves so --depth=N matches the deepest hop the user
+            # asked for.
+            max_depth=depth + 1 if use_internal_input_root else depth,
+            tags_str=tag,
+            persona_id=persona_obj.id,
+            label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
+            created_by_id=created_by_id,
+            status=Crawl.StatusChoices.QUEUED,
+            retry_at=None if index_only else timezone.now(),
+            config=crawl_config,
+        )
+        first_url = crawl.get_urls_list()[0] if crawl.get_urls_list() else ""
+    except BaseException:
+        if command is not None:
+            command.mark_exited(exit_code=1)
+        raise
+    if command is not None:
+        command.url = first_url
+        command.save(update_fields=["url", "modified_at"])
 
     print(f"[green]\\[+] Created Crawl {crawl.id} with max_depth={depth}[/green]")
-    first_url = crawl.get_urls_list()[0] if crawl.get_urls_list() else ""
     print(f"    [dim]First URL: {first_url}[/dim]")
 
     # 3. The CrawlMachine will create Snapshots from all URLs when started
@@ -216,15 +235,14 @@ def add(
         )
         from archivebox.services.runner import ensure_background_runner
 
-        ensure_background_runner(allow_under_pytest=True)
+        ensure_background_runner()
     else:
         # Foreground mode: run full crawl runner until all work is done
         print("[green]\\[*] Starting crawl runner to process crawl...[/green]")
-        from archivebox.machine.models import Process
-        from archivebox.core.takeover_util import command_owns_foreground_runner, current_command, standby_until_foreground_runner_needed
+        from archivebox.core.takeover_util import command_owns_foreground_runner, standby_until_foreground_runner_needed
         from archivebox.workers.supervisord_util import get_existing_supervisord_process, run_runner_worker, stop_own_supervisord_process
 
-        command = current_command(Process.TypeChoices.ADD, data_dir=CONSTANTS.DATA_DIR, url=first_url)
+        assert command is not None
         exit_code = 0
         try:
             try:
@@ -235,6 +253,7 @@ def add(
                             ["--crawl-id", str(crawl.id)],
                             name=f"worker_runner_add_{os.getpid()}",
                             interactive_interrupts=True,
+                            config=get_config(crawl=crawl),
                         )
                         crawl.refresh_from_db(fields=["status", "retry_at"])
                         if exit_code == 0 and crawl.status == crawl.StatusChoices.SEALED:
@@ -375,6 +394,10 @@ def main(**kwargs):
             raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
         if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
             raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
+        if not kwargs.get("index_only"):
+            from archivebox.workers.supervisord_util import require_crawl_memory
+
+            require_crawl_memory()
 
         # Translate --only-new/--no-only-new into a crawl config override.
         # add() takes config overrides as a dict; no per-flag kwargs.

archivebox/cli/archivebox_config.py

@@ -59,28 +59,20 @@ def config(
     matching_config = {}
     if search:
         if config_options:
-            config_options = [
-                core_config_aliases.get(key.upper().strip()) or resolve_alias(key.upper().strip(), plugin_schemas) for key in config_options
-            ]
-            matching_config = {key: readable_config[key] for key in config_options if key in readable_config}
-            for config_section in CONFIGS.values():
-                aliases = {str(field.alias): field_name for field_name, field in type(config_section).model_fields.items() if field.alias}
+            search_terms = [key.strip().lower() for key in config_options]
 
-                for search_key in config_options:
-                    # search all aliases in the section
-                    for alias_key, key in aliases.items():
-                        if key in readable_config and search_key.lower() in alias_key.lower():
-                            matching_config[key] = dict(config_section)[key]
+            for existing_key, value in readable_config.items():
+                if any(term in existing_key.lower() or term in str(value).lower() for term in search_terms):
+                    matching_config[existing_key] = value
 
-                    # search all keys and values in the section
-                    for existing_key, value in dict(config_section).items():
-                        if existing_key in readable_config and (
-                            search_key.lower() in existing_key.lower() or search_key.lower() in str(value).lower()
-                        ):
-                            matching_config[existing_key] = value
-                    for existing_key, value in readonly_config.items():
-                        if search_key.lower() in existing_key.lower() or search_key.lower() in str(value).lower():
-                            matching_config[existing_key] = value
+            for alias, key in core_config_aliases.items():
+                if key in readable_config and any(term in alias.lower() for term in search_terms):
+                    matching_config[key] = readable_config[key]
+
+            for schema in plugin_schemas.values():
+                for key, metadata in schema.items():
+                    if key in readable_config and any(term in key.lower() or term in str(metadata).lower() for term in search_terms):
+                        matching_config[key] = readable_config[key]
 
         print(printable_config(matching_config))
         raise SystemExit(not matching_config)

archivebox/cli/archivebox_extract.py

@@ -46,7 +46,6 @@ def process_archiveresult_by_id(archiveresult_id: str) -> int:
     through the shared crawl runner with the corresponding plugin selected.
     """
     from rich import print as rprint
-    from django.utils import timezone
     from archivebox.core.models import ArchiveResult
     from archivebox.api.v1_core import _uuid_ref_query
     from archivebox.services.runner import run_crawl
@@ -60,22 +59,9 @@ def process_archiveresult_by_id(archiveresult_id: str) -> int:
     rprint(f"[blue]Extracting {archiveresult.plugin} for {archiveresult.snapshot.url}[/blue]", file=sys.stderr)
 
     try:
-        was_paused = archiveresult.snapshot.is_paused
         archiveresult.reset_for_retry()
         snapshot = archiveresult.snapshot
-        if not was_paused:
-            snapshot.queue_for_extraction()
-        else:
-            # A paused snapshot may still accept explicit maintenance for one
-            # ArchiveResult, but this path must not transition it back to
-            # queued/startable work. Guard: only set retry_at while the row is
-            # still paused — concurrent resume would otherwise see a stale
-            # retry_at marker.
-            snapshot.safe_update(
-                {"retry_at": timezone.now()},
-                refresh=False,
-                extra_filter={"status": snapshot.StatusChoices.PAUSED},
-            )
+        snapshot.queue_for_extraction()
         crawl = snapshot.crawl
         if not crawl.claim_processing_lock(lock_seconds=10):
             rprint(
@@ -84,11 +70,7 @@ def process_archiveresult_by_id(archiveresult_id: str) -> int:
             )
             return 1
 
-        try:
-            run_crawl(str(snapshot.crawl_id), snapshot_ids=[str(snapshot.id)], selected_plugins=[archiveresult.plugin])
-        finally:
-            if was_paused:
-                snapshot.restore_paused_scheduler_marker()
+        run_crawl(str(snapshot.crawl_id), snapshot_ids=[str(snapshot.id)], selected_plugins=[archiveresult.plugin])
         archiveresult.refresh_from_db()
 
         if archiveresult.status == ArchiveResult.StatusChoices.SUCCEEDED:
@@ -290,12 +272,9 @@ def run_plugins(
     queue_at = timezone.now()
     if existing_snapshot_ids:
         if requested_rows:
-            # Targeted ArchiveResult retries use retry_at as the scheduling
-            # signal and keep sealed snapshots sealed so extractors are not
-            # re-run outside the explicitly queued plugin rows. Paused snapshots
-            # also keep status=paused here: `retry_at` only asks the orchestrator
-            # to process the queued plugin rows, and run_due_snapshot restores
-            # retry_at=MAX afterward instead of resuming the snapshot lifecycle.
+            # Search indexing on a sealed Snapshot is the only targeted hook
+            # allowed to bypass the normal lifecycle. Every other requested
+            # plugin requeues its Snapshot through the unified state machine.
             affected_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in rows_to_queue}
             if preserve_queued and queued_rows:
                 queued_snapshot_ids = {snapshot_id for snapshot_id, _plugin_name, _hook_name in queued_rows}
@@ -312,14 +291,30 @@ def run_plugins(
                         flat=True,
                     )
                 )
+            requested_plugins_by_id: dict[str, set[str]] = defaultdict(set)
+            for snapshot_id, plugin_name, _hook_name in requested_rows:
+                requested_plugins_by_id[snapshot_id].add(plugin_name)
             for snapshot in Snapshot.objects.filter(id__in=affected_snapshot_ids).only("id", "status", "modified_at"):
                 # Guard the read-time status so we never bump retry_at on a
                 # row that's been re-queued / started by a concurrent runner.
-                snapshot.safe_update(
-                    {"retry_at": queue_at, "modified_at": queue_at},
-                    refresh=False,
-                    extra_filter={"status": snapshot.status},
+                plugin_names = requested_plugins_by_id.get(str(snapshot.id), set())
+                sealed_search_backfill = (
+                    snapshot.status == Snapshot.StatusChoices.SEALED
+                    and plugin_names
+                    and all(plugin_name.startswith("search_backend_") for plugin_name in plugin_names)
                 )
+                if sealed_search_backfill:
+                    snapshot.safe_update(
+                        {"retry_at": queue_at, "modified_at": queue_at},
+                        refresh=False,
+                        extra_filter={"status": snapshot.status},
+                    )
+                else:
+                    snapshot.update_and_requeue(
+                        status=Snapshot.StatusChoices.QUEUED,
+                        retry_at=queue_at,
+                        current_step=0,
+                    )
         else:
             # No plugin rows were requested, so this is a full snapshot retry.
             for snapshot in Snapshot.objects.filter(id__in=existing_snapshot_ids).only("id", "status", "retry_at", "modified_at"):

archivebox/cli/archivebox_init.py

@@ -25,20 +25,20 @@ def _display_data_path(path: Path, data_dir: Path) -> str:
 def init(force: bool = False, quick: bool = False, install: bool = False) -> None:
     """Initialize a new ArchiveBox collection in the current directory"""
 
+    from archivebox.misc.checks import check_not_inside_source_dir
+
+    check_not_inside_source_dir()
+
     from archivebox.config import CONSTANTS, VERSION
     from archivebox.config.common import get_config
     from archivebox.config.collection import write_config_file
-    from archivebox.misc.db import apply_migrations
+    from archivebox.misc.db import apply_migrations, database_exists, ensure_database_ready
     from archivebox.misc.checks import check_migrations
 
     config = get_config()
 
-    # if os.access(out_dir / CONSTANTS.JSON_INDEX_FILENAME, os.F_OK):
-    #     print("[red]:warning: This folder contains a JSON index. It is deprecated, and will no longer be kept up to date automatically.[/red]", file=sys.stderr)
-    #     print("[red]    You can run `archivebox list --json --with-headers > static_index.json` to manually generate it.[/red]", file=sys.stderr)
-
     is_empty = not len(set(os.listdir(CONSTANTS.DATA_DIR)) - CONSTANTS.ALLOWED_IN_DATA_DIR)
-    existing_index = os.path.isfile(CONSTANTS.DATABASE_FILE)
+    existing_index = database_exists()
     if is_empty and not existing_index:
         print(f"[turquoise4][+] Initializing a new ArchiveBox v{VERSION} collection...[/turquoise4]")
         print("[green]----------------------------------------------------------------------[/green]")
@@ -86,11 +86,15 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
     # create the ArchiveBox.conf file
     write_config_file({"SECRET_KEY": config.SECRET_KEY})
 
-    if os.access(CONSTANTS.DATABASE_FILE, os.F_OK):
+    if existing_index:
         print("\n[green][*] Verifying main SQL index and running any migrations needed...[/green]")
     else:
         print("\n[green][+] Building main SQL index and running initial migrations...[/green]")
 
+    # For postgres, make sure the server is reachable and create the database
+    # if it doesn't exist yet (sqlite creates its file automatically).
+    ensure_database_ready()
+
     from archivebox.config.django import setup_django
 
     setup_django()
@@ -107,12 +111,14 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
         else:
             os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init
 
-    assert os.path.isfile(CONSTANTS.DATABASE_FILE) and os.access(CONSTANTS.DATABASE_FILE, os.R_OK)
-    print()
-    print(f"    √ {_display_data_path(CONSTANTS.DATABASE_FILE, CONSTANTS.DATA_DIR)}")
+    from archivebox.misc.db import database_display_location, is_postgres
 
-    # from django.contrib.auth.models import User
-    #     call_command("createsuperuser", interactive=True)
+    assert database_exists()
+    print()
+    if is_postgres():
+        print(f"    √ {database_display_location()}")
+    else:
+        print(f"    √ {_display_data_path(CONSTANTS.DATABASE_FILE, CONSTANTS.DATA_DIR)}")
 
     print()
     print("[dodger_blue3][*] Checking links from indexes and archive folders (safe to Ctrl+C)...[/dodger_blue3]")
@@ -149,12 +155,12 @@ def init(force: bool = False, quick: bool = False, install: bool = False) -> Non
     CONSTANTS.PERSONAS_DIR.mkdir(parents=True, exist_ok=True)
     CONSTANTS.DEFAULT_TMP_DIR.mkdir(parents=True, exist_ok=True)
 
-    from archivebox.config.paths import get_or_create_working_tmp_dir, get_or_create_working_lib_dir
+    from archivebox.config.paths import create_and_chown_dir, get_or_create_working_tmp_dir, get_or_create_working_lib_dir
 
     config = get_config()
-    config.TMP_DIR.mkdir(parents=True, exist_ok=True)
-    config.ABXPKG_LIB_DIR.mkdir(parents=True, exist_ok=True)
-    (config.ABXPKG_LIB_DIR / "bin").mkdir(parents=True, exist_ok=True)
+    create_and_chown_dir(config.TMP_DIR)
+    create_and_chown_dir(config.ABXPKG_LIB_DIR)
+    create_and_chown_dir(config.ABXPKG_LIB_DIR / "bin")
 
     working_tmp_dir = get_or_create_working_tmp_dir(autofix=True, quiet=True)
     if working_tmp_dir:

archivebox/cli/archivebox_install.py

@@ -4,6 +4,7 @@ __package__ = "archivebox.cli"
 
 import os
 import shutil
+import re
 from pathlib import Path
 
 import rich_click as click
@@ -12,6 +13,76 @@ from rich import print
 from archivebox.misc.util import docstring, enforce_types
 
 
+def _resolve_install_targets(
+    requested_names: tuple[str, ...],
+) -> tuple[list[str], list[str]]:
+    """Resolve plugin names and declared binary aliases, leaving unknown tokens raw."""
+    from abx_dl.models import discover_plugins, filter_plugins
+
+    plugins = discover_plugins(runtime="archivebox")
+    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in plugins}
+    plugin_names_by_binary_alias: dict[str, set[str]] = {}
+    for plugin_name, plugin in plugins.items():
+        for required_binary in plugin.config.required_binaries:
+            aliases = {required_binary.name.lower()}
+            template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", required_binary.name)
+            if template_match:
+                config_key = template_match.group(1)
+                aliases.add(config_key.lower())
+                aliases.add(config_key.removesuffix("_BINARY").lower())
+                default = plugin.config.properties.get(config_key, {}).get("default")
+                if default:
+                    aliases.add(str(default).lower())
+            for alias in aliases:
+                plugin_names_by_binary_alias.setdefault(alias, set()).add(plugin_name)
+
+    requested_plugins: list[str] = []
+    raw_binary_names: list[str] = []
+    for name in requested_names:
+        normalized_name = name.lower()
+        if normalized_name in plugin_names_by_lower:
+            requested_plugins.append(plugin_names_by_lower[normalized_name])
+        elif normalized_name in plugin_names_by_binary_alias:
+            requested_plugins.extend(sorted(plugin_names_by_binary_alias[normalized_name]))
+        else:
+            raw_binary_names.append(name)
+
+    selected_plugins = filter_plugins(plugins, requested_plugins, include_providers=True) if requested_plugins else {}
+    selected_plugin_names = {name.lower() for name in selected_plugins}
+    raw_binary_names = [name for name in raw_binary_names if name.lower() not in selected_plugin_names]
+    return sorted(selected_plugins), sorted(set(raw_binary_names))
+
+
+def _install_raw_binary_names(binary_names: list[str], binproviders: str) -> None:
+    """Install user-requested standalone binaries through the Binary state machine."""
+    from django.utils import timezone
+
+    from archivebox.machine.models import Binary, Machine, _canonical_binary_name
+    from archivebox.services.runner import run_due_binary
+
+    machine = Machine.current()
+    for requested_name in binary_names:
+        binary_name = _canonical_binary_name(requested_name)
+        if not binary_name:
+            raise click.UsageError(f"Invalid binary name: {requested_name}")
+        binary = Binary.objects.filter(machine=machine, name=binary_name).order_by("-modified_at").first()
+        if binary is None:
+            binary = Binary.objects.create(
+                machine=machine,
+                name=binary_name,
+                binproviders=binproviders,
+                overrides={},
+                status=Binary.StatusChoices.QUEUED,
+            )
+        elif not binary.is_valid or binary.binproviders != binproviders:
+            binary.binproviders = binproviders
+            binary.overrides = {}
+            binary.status = Binary.StatusChoices.QUEUED
+            binary.retry_at = timezone.now()
+            binary.save(update_fields=["binproviders", "overrides", "status", "retry_at", "modified_at"])
+        run_due_binary(binary, lock_seconds=60)
+
+
 def ensure_data_dir_lib_symlink(data_dir: Path, abxpkg_lib_dir: Path) -> Path | None:
     """Expose the shared abxpkg lib dir at DATA_DIR/lib after install."""
     link_path = data_dir / "lib"
@@ -90,16 +161,27 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
 
     setup_django()
 
-    plugin_names = list(binaries)
+    plugin_names, raw_binary_names = _resolve_install_targets(binaries) if binaries else ([], [])
+    install_plugin_names = list(plugin_names)
     if binproviders != "*":
-        plugin_names.extend(provider.strip() for provider in binproviders.split(",") if provider.strip())
+        install_plugin_names.extend(provider.strip() for provider in binproviders.split(",") if provider.strip())
 
-    print("[+] Running installer via abx-dl bus...")
-    print()
+    if install_plugin_names or not binaries:
+        if not binaries:
+            from archivebox.workers.supervisord_util import require_dependency_install_memory
 
-    from archivebox.services.runner import run_install
+            require_dependency_install_memory()
+        print("[+] Running plugin installer via abx-dl bus...")
+        print()
 
-    run_install(plugin_names=plugin_names or None)
+        from archivebox.services.runner import run_install
+
+        run_install(plugin_names=install_plugin_names or None)
+
+    if raw_binary_names:
+        print(f"[+] Running direct binary installer via ArchiveBox binary state machine: {', '.join(raw_binary_names)}")
+        print()
+        _install_raw_binary_names(raw_binary_names, binproviders)
 
     from archivebox.config.common import get_config
 
@@ -123,7 +205,7 @@ def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bo
     # Django is already loaded, so just import and call the function directly
     from archivebox.cli.archivebox_version import version as show_version
 
-    show_version(quiet=False)
+    show_version(quiet=False, binaries=binaries)
 
 
 @click.command()

archivebox/cli/archivebox_oneshot.py

@@ -4,23 +4,45 @@ __package__ = "archivebox.cli"
 __command__ = "archivebox oneshot"
 
 import subprocess
+import sys
 from pathlib import Path
 
 import rich_click as click
 
 from archivebox.config import CONSTANTS
+from archivebox.config.common import get_config
 
 
 @click.command(add_help_option=False, context_settings=dict(ignore_unknown_options=True))
 @click.argument("args", nargs=-1)
 def main(args: tuple[str, ...] = ()) -> None:
     """Download URLs using abx-dl"""
+    from archivebox.misc.checks import check_not_inside_source_dir
+
+    check_not_inside_source_dir()
+
     cwd = Path.cwd()
     if any((path / CONSTANTS.SQL_INDEX_FILENAME).exists() for path in (cwd, *cwd.parents)):
         raise click.ClickException(
             "Refusing to run `archivebox oneshot` inside an ArchiveBox DATA_DIR. Use `archivebox add` here, or run oneshot from another directory.",
         )
-    raise SystemExit(subprocess.run(["abx-dl", *args]).returncode)
+    abxpkg_binary = Path(sys.executable).with_name("abxpkg")
+    if not abxpkg_binary.is_file():
+        raise click.ClickException(f"abxpkg executable is missing from the ArchiveBox environment: {abxpkg_binary}")
+    abxpkg_lib_dir = get_config(include_machine=False).ABXPKG_LIB_DIR
+    raise SystemExit(
+        subprocess.run(
+            [
+                str(abxpkg_binary),
+                f"--lib={abxpkg_lib_dir}",
+                "--binproviders=env",
+                "--install",
+                "run",
+                "abx-dl",
+                *args,
+            ],
+        ).returncode,
+    )
 
 
 if __name__ == "__main__":

archivebox/cli/archivebox_persona.py

@@ -137,65 +137,6 @@ def get_edge_user_data_dir() -> Path | None:
     return None
 
 
-def get_browser_binary(browser: str) -> str | None:
-    system = platform.system()
-    home = Path.home()
-    browser = browser.lower()
-
-    if system == "Darwin":
-        candidates = {
-            "chrome": ["/Applications/Google Chrome.app/Contents/MacOS/Google Chrome"],
-            "chromium": ["/Applications/Chromium.app/Contents/MacOS/Chromium"],
-            "brave": ["/Applications/Brave Browser.app/Contents/MacOS/Brave Browser"],
-            "edge": ["/Applications/Microsoft Edge.app/Contents/MacOS/Microsoft Edge"],
-        }.get(browser, [])
-    elif system == "Linux":
-        candidates = {
-            "chrome": [
-                "/usr/bin/google-chrome",
-                "/usr/bin/google-chrome-stable",
-                "/usr/bin/google-chrome-beta",
-                "/usr/bin/google-chrome-unstable",
-            ],
-            "chromium": ["/usr/bin/chromium", "/usr/bin/chromium-browser"],
-            "brave": ["/usr/bin/brave-browser", "/usr/bin/brave-browser-beta", "/usr/bin/brave-browser-nightly"],
-            "edge": [
-                "/usr/bin/microsoft-edge",
-                "/usr/bin/microsoft-edge-stable",
-                "/usr/bin/microsoft-edge-beta",
-                "/usr/bin/microsoft-edge-dev",
-            ],
-        }.get(browser, [])
-    elif system == "Windows":
-        local_app_data = Path(os.environ.get("LOCALAPPDATA", home / "AppData" / "Local"))
-        candidates = {
-            "chrome": [
-                str(local_app_data / "Google" / "Chrome" / "Application" / "chrome.exe"),
-                "C:\\Program Files\\Google\\Chrome\\Application\\chrome.exe",
-                "C:\\Program Files (x86)\\Google\\Chrome\\Application\\chrome.exe",
-            ],
-            "chromium": [str(local_app_data / "Chromium" / "Application" / "chrome.exe")],
-            "brave": [
-                str(local_app_data / "BraveSoftware" / "Brave-Browser" / "Application" / "brave.exe"),
-                "C:\\Program Files\\BraveSoftware\\Brave-Browser\\Application\\brave.exe",
-                "C:\\Program Files (x86)\\BraveSoftware\\Brave-Browser\\Application\\brave.exe",
-            ],
-            "edge": [
-                str(local_app_data / "Microsoft" / "Edge" / "Application" / "msedge.exe"),
-                "C:\\Program Files\\Microsoft\\Edge\\Application\\msedge.exe",
-                "C:\\Program Files (x86)\\Microsoft\\Edge\\Application\\msedge.exe",
-            ],
-        }.get(browser, [])
-    else:
-        candidates = []
-
-    for candidate in candidates:
-        if candidate and Path(candidate).exists():
-            return candidate
-
-    return None
-
-
 BROWSER_PROFILE_FINDERS = {
     "chrome": get_chrome_user_data_dir,
     "chromium": get_chrome_user_data_dir,  # Same locations
@@ -317,10 +258,6 @@ def create_personas(
         if profile is None and (source_profile_dir / "Default").exists():
             profile = "Default"
 
-        browser_binary = get_browser_binary(import_from)
-        if browser_binary:
-            rprint(f"[dim]Using {import_from} binary: {browser_binary}[/dim]", file=sys.stderr)
-
     created_count = 0
     for name in name_list:
         name = name.strip()

archivebox/cli/archivebox_remove.py

@@ -9,7 +9,6 @@ from collections.abc import Iterable
 
 import rich_click as click
 
-from django.db import OperationalError
 from django.db.models import QuerySet
 
 from archivebox.config import CONSTANTS
@@ -92,12 +91,6 @@ def remove(
     # in ``base_models/models.py`` and runs AFTER its row's tx commits — so
     # rmtree doesn't hold the lock either.
     #
-    # The SQLite retry wrapper in core/sqlite_backend/base.py re-raises lock
-    # errors when called inside an atomic block (because it can't safely
-    # release+reacquire a transaction), so we wrap each row's delete in our
-    # own retry loop at this outer (non-atomic) level. Each attempt is a
-    # fresh atomic; an exception cleanly rolls it back before we sleep.
-    retry_interval = 1.0
     deleted_snapshot_pks = []
     timed_out = False
     timeout_error = ""
@@ -106,23 +99,9 @@ def remove(
             timed_out = True
             timeout_error = f"Remove timed out after {timeout:g}s with {len(snapshot_pks) - index} snapshots remaining."
             break
-        while True:
-            try:
-                deleted_count, _ = Snapshot.objects.filter(pk=pk).delete()
-                if deleted_count:
-                    deleted_snapshot_pks.append(pk)
-                break
-            except OperationalError as err:
-                if "database is locked" not in str(err):
-                    raise
-                remaining_time = deadline - time.monotonic() if deadline is not None else None
-                if remaining_time is not None and remaining_time <= 0:
-                    timed_out = True
-                    timeout_error = f"Remove timed out after {timeout:g}s while waiting for the database lock."
-                    break
-                time.sleep(min(retry_interval, remaining_time) if remaining_time is not None else retry_interval)
-        if timed_out:
-            break
+        deleted_count, _ = Snapshot.objects.filter(pk=pk).delete()
+        if deleted_count:
+            deleted_snapshot_pks.append(pk)
 
     all_snapshots = Snapshot.objects.all()
     remaining_count = all_snapshots.count()

archivebox/cli/archivebox_run.py

@@ -294,17 +294,11 @@ def run_runner(
 
         crawl = Crawl.objects.filter(id=crawl_id, status__in=Crawl.RUNNABLE_STATES).first()
         now = timezone.now()
-        # Only re-lease when the row is unscheduled (retry_at IS NULL) or its
-        # existing lease has already expired. A future retry_at means another
-        # worker is already scheduled — don't clobber.
-        if crawl is not None and (crawl.retry_at is None or crawl.retry_at <= now):
-            # extra_filter pins the read-time retry_at so a concurrent worker
-            # that grabbed the lease between our SELECT and UPDATE wins.
-            crawl.safe_update(
-                {"retry_at": now},
-                refresh=False,
-                extra_filter={"retry_at": crawl.retry_at},
-            )
+        # Winning the single-runner gate terminates and waits for every older
+        # runner. Requeue the explicitly requested crawl so an abandoned
+        # future ownership lease cannot hide it from the unified scheduler.
+        if crawl is not None:
+            crawl.update_and_requeue(retry_at=now, refresh=False)
     # Only a foreground `archivebox add` gets the interactive "abort current
     # hook, continue/retry, second Ctrl+C exits" flow. Server/update/run owned
     # orchestrators should shut down immediately and cleanly on the first signal.
@@ -348,7 +342,7 @@ def run_runner(
 @click.option("--crawl-id", help="Run the crawl runner for a specific crawl only")
 @click.option("--snapshot-id", help="Run one snapshot through its crawl")
 @click.option("--binary-id", help="Run one queued binary install directly on the bus")
-@click.option("--maintenance-only", is_flag=True, help="Only process due maintenance ticks on sealed/paused snapshots")
+@click.option("--maintenance-only", is_flag=True, help="Only process sealed Snapshot maintenance and search-index backfills")
 @click.option(
     "--maintenance-batch-size",
     type=int,

archivebox/cli/archivebox_server.py

@@ -267,11 +267,13 @@ def server(
     host, port = _parse_and_validate_bind_spec(bind_spec)
 
     if daemonize and os.environ.get("ARCHIVEBOX_SERVER_DAEMON_CHILD") != "1":
+        from archivebox.workers.supervisord_util import resolve_env_binary
+
         log_path = CONSTANTS.LOGS_DIR / "server.log"
         log_path.parent.mkdir(parents=True, exist_ok=True)
         daemon_env = os.environ.copy()
         daemon_env["ARCHIVEBOX_SERVER_DAEMON_CHILD"] = "1"
-        daemon_cmd = [sys.executable, "-m", "archivebox", "server"]
+        daemon_cmd = [str(resolve_env_binary("archivebox")), "server"]
         if debug:
             daemon_cmd.append("--debug")
         if reload:
@@ -300,7 +302,14 @@ def server(
             except OSError:
                 time.sleep(0.1)
         else:
-            print(f"[yellow][!] ArchiveBox daemon server pid={proc.pid} is still starting. See {log_path}[/yellow]")
+            print(f"[red][X] ArchiveBox daemon server pid={proc.pid} did not become ready. See {log_path}[/red]")
+            proc.terminate()
+            try:
+                proc.wait(timeout=10)
+            except subprocess.TimeoutExpired:
+                proc.kill()
+                proc.wait(timeout=5)
+            sys.exit(1)
         return
 
     os.environ["BIND_ADDR"] = f"{host}:{port}"
@@ -366,7 +375,7 @@ def server(
                     data_dir=CONSTANTS.DATA_DIR,
                     exclude_id=command.id,
                 )
-                takeover_components = active_supervisord_runtime_components(config=config)
+                takeover_components = active_supervisord_runtime_components()
                 if older_owner and takeover_components:
                     print(
                         "[yellow][*] Taking over "

archivebox/cli/archivebox_snapshot.py

@@ -31,14 +31,62 @@ __package__ = "archivebox.cli"
 __command__ = "archivebox snapshot"
 
 import sys
-from collections.abc import Iterable
+from collections.abc import Iterable, Iterator
+from itertools import islice
 
 import rich_click as click
 from rich import print as rprint
 from django.db.models import QuerySet
 
 SNAPSHOT_FILTER_TYPE_CHOICES = ("exact", "substring", "regex", "domain", "tag", "timestamp")
-SNAPSHOT_LIST_CHUNK_SIZE = 100
+SNAPSHOT_LIST_CHUNK_SIZE = 5000
+
+
+def iter_snapshot_json(queryset: QuerySet) -> Iterator[dict[str, object]]:
+    from archivebox.config import VERSION
+    from archivebox.core.models import SnapshotTag
+
+    fields = (
+        "id",
+        "crawl_id",
+        "url",
+        "title",
+        "bookmarked_at",
+        "created_at",
+        "timestamp",
+        "depth",
+        "status",
+        "fs_version",
+        "output_size",
+    )
+    rows = queryset.values(*fields).iterator(chunk_size=SNAPSHOT_LIST_CHUNK_SIZE)
+    while batch := list(islice(rows, SNAPSHOT_LIST_CHUNK_SIZE)):
+        tags_by_snapshot = {row["id"]: [] for row in batch}
+        tag_rows = (
+            SnapshotTag.objects.filter(snapshot_id__in=tags_by_snapshot).order_by("tag__name").values_list("snapshot_id", "tag__name")
+        )
+        for snapshot_id, tag_name in tag_rows:
+            tags_by_snapshot[snapshot_id].append(tag_name)
+
+        for row in batch:
+            archive_size = int(row["output_size"] or 0)
+            yield {
+                "type": "Snapshot",
+                "schema_version": VERSION,
+                "id": str(row["id"]),
+                "crawl_id": str(row["crawl_id"]),
+                "url": row["url"],
+                "title": row["title"],
+                "tags": ",".join(sorted(tags_by_snapshot[row["id"]])),
+                "bookmarked_at": row["bookmarked_at"].isoformat() if row["bookmarked_at"] else None,
+                "created_at": row["created_at"].isoformat() if row["created_at"] else None,
+                "timestamp": row["timestamp"],
+                "depth": row["depth"],
+                "status": row["status"],
+                "fs_version": row["fs_version"],
+                "archive_size": archive_size,
+                "output_size": archive_size,
+            }
 
 
 # =============================================================================
@@ -296,8 +344,8 @@ def list_snapshots(
         return 0
 
     if not is_tty:
-        for snapshot in queryset.prefetch_related("tags").iterator(chunk_size=SNAPSHOT_LIST_CHUNK_SIZE):
-            write_record(snapshot.to_json())
+        for snapshot_json in iter_snapshot_json(queryset):
+            write_record(snapshot_json)
             count += 1
         rprint(f"[dim]Listed {count} snapshots[/dim]", file=sys.stderr)
         return 0

archivebox/cli/archivebox_status.py

@@ -44,12 +44,18 @@ def status(out_dir: Path = CONSTANTS.DATA_DIR) -> None:
         legacy_snapshot_dirs = [
             entry for entry in archive_dir.iterdir() if entry.is_dir() and not entry.is_symlink() and Snapshot.is_legacy_archive_dir(entry)
         ]
-    print(f"    > SQL Main Index: {num_sql_links} links".ljust(36), f"(found in {CONSTANTS.SQL_INDEX_FILENAME})")
+    from archivebox.misc.db import database_display_location, is_postgres
+
+    index_location = database_display_location() if is_postgres() else CONSTANTS.SQL_INDEX_FILENAME
+    print(f"    > SQL Main Index: {num_sql_links} links".ljust(36), f"(found in {index_location})")
     print(f"    > JSON Link Details: {len(legacy_snapshot_dirs)} links".ljust(36), f"(found in {archive_dir.name}/*/index.json)")
     print()
     print("[green]\\[*] Scanning archive data directories...[/green]")
     users_dir = CONSTANTS.USERS_DIR
-    scan_roots = [root for root in (archive_dir, users_dir) if root.exists()]
+    # The current users layout is nested below archive/, so scanning both
+    # archive/ and archive/users/ recursively counts every current snapshot
+    # twice. A single archive root includes both legacy and current layouts.
+    scan_roots = [archive_dir] if archive_dir.exists() else []
     scan_roots_display = ", ".join(str(root) for root in scan_roots) if scan_roots else str(archive_dir)
     print(f"[yellow]   {scan_roots_display}[/yellow]")
     do_precise_fs_scan = num_sql_links <= MAX_STATUS_FS_DIR_SCAN

archivebox/cli/archivebox_update.py

@@ -76,9 +76,14 @@ def reindex_snapshots(
     wait_for_turn=None,
 ) -> dict[str, Any]:
     from archivebox.cli.archivebox_extract import run_plugins
-    from archivebox.core.models import ArchiveResult
+    from archivebox.core.models import ArchiveResult, Snapshot
     from abx_dl.models import discover_plugins
 
+    # Search backfill is the one maintenance hook allowed to execute without
+    # reopening a Snapshot. Restrict that exception to already-sealed rows;
+    # every open lifecycle state remains owned by the normal state machine.
+    snapshots = snapshots.filter(status=Snapshot.StatusChoices.SEALED)
+
     stats: dict[str, Any] = {"processed": 0, "requested": 0, "queued": 0, "skipped_queued": 0, "reindexed": 0, "snapshot_ids": []}
     records: list[dict[str, str]] = []
     plugins_by_name = discover_plugins(runtime="archivebox")
@@ -227,7 +232,6 @@ def update(
     from archivebox.core.takeover_util import (
         command_owns_foreground_runner,
         current_command,
-        ensure_daemon_stack,
         foreground_runner_owner,
         standby_until_foreground_runner_needed,
     )
@@ -248,11 +252,9 @@ def update(
         standby_until_foreground_runner_needed(command, data_dir=CONSTANTS.DATA_DIR)
         raise_if_shutdown_requested()
 
-    def run_scoped_runner(*args: str, ensure_daemon_reason: str | None = None) -> None:
+    def run_scoped_runner(*args: str) -> None:
         while True:
             wait_for_turn()
-            if ensure_daemon_reason:
-                ensure_daemon_stack(reason=ensure_daemon_reason)
             exit_code = run_runner_worker(
                 list(args),
                 name=f"worker_runner_update_{os.getpid()}",
@@ -365,7 +367,7 @@ def update(
                         # Snapshot rows: Snapshot.save() moves archive/<ts> to
                         # the current output_dir and preserves the lifecycle
                         # status. Drain those retry_at ticks before queuing
-                        # search backfill below. Otherwise the sealed/paused
+                        # search backfill below. Otherwise the sealed search
                         # runner branch correctly sees queued ArchiveResult
                         # rows first, runs the targeted plugins, and may leave
                         # the fs_version maintenance tick hidden behind that
@@ -393,7 +395,6 @@ def update(
                     if full_update_empty:
                         print("[*] No snapshots found; skipping search indexing backfill.")
                     else:
-                        ensure_daemon_stack(reason="search indexing")
                         search_plugins = _get_search_indexing_plugins()
                         if not search_plugins:
                             print("[*] No search indexing plugins are available, nothing to backfill.")
@@ -430,9 +431,7 @@ def update(
                                 now = timezone.now()
                                 queued_result_for_snapshot = queued_index_results.filter(snapshot_id=OuterRef("pk"))
                                 snapshots_to_wake = (
-                                    Snapshot.objects.filter(
-                                        status__in=(Snapshot.StatusChoices.SEALED, Snapshot.StatusChoices.PAUSED),
-                                    )
+                                    Snapshot.objects.filter(status=Snapshot.StatusChoices.SEALED)
                                     .annotate(
                                         has_queued_index_result=Exists(queued_result_for_snapshot),
                                     )
@@ -475,8 +474,8 @@ def update(
 
                 if do_run_until_idle and (do_index or not ran_post_migrate_runner):
                     # Search/index backfill intentionally queues targeted
-                    # ArchiveResult rows without reopening sealed/paused
-                    # snapshots. This second runner pass drains those plugin
+                    # ArchiveResult rows without reopening sealed snapshots.
+                    # This second runner pass drains those plugin
                     # rows after filesystem maintenance has had its own turn.
                     # For a normal unfiltered `archivebox update`, keep the
                     # historical final pass broad enough to resume genuinely
@@ -499,7 +498,6 @@ def update(
                     else:
                         run_scoped_runner(
                             *(["--maintenance-only", "--maintenance-batch-size", str(batch_size)] if index_only or migrate_only else []),
-                            ensure_daemon_reason="search indexing" if do_index else None,
                         )
 
                 if not continuous:

archivebox/cli/archivebox_version.py

@@ -4,7 +4,10 @@ __package__ = "archivebox.cli"
 
 import sys
 import os
+import json
+import asyncio
 import platform
+import re
 from pathlib import Path
 from collections.abc import Iterable
 
@@ -77,48 +80,6 @@ def _render_binary_abspath(abspath: str):
     return Text(abspath, style="green")
 
 
-def _build_binary_table(rows: list[dict[str, object]]):
-    from rich import box
-    from rich.table import Table
-
-    table = Table(title="Binary Dependencies", box=box.SIMPLE_HEAVY, expand=True)
-    table.add_column("Binary", no_wrap=True, max_width=28)
-    table.add_column("Plugin", no_wrap=True, max_width=24)
-    table.add_column("State", no_wrap=True, width=8)
-    table.add_column("Status", justify="center", no_wrap=True, width=6)
-    table.add_column("Version", no_wrap=True, width=16)
-    table.add_column("Provider", no_wrap=True, width=8)
-    table.add_column("Path", overflow="fold", ratio=1)
-    for row in rows:
-        table.add_row(
-            str(row["binary"]),
-            str(row["plugin"]),
-            str(row["state"]),
-            str(row["status"]),
-            str(row["version"]),
-            str(row["provider"]),
-            row["path"],
-            style=str(row.get("style") or ""),
-        )
-    return table
-
-
-def _print_binary_row(prnt, row: dict[str, object]) -> None:
-    prnt(
-        "",
-        str(row["binary"]).ljust(28),
-        row["status"],
-        str(row["plugin"]).ljust(24),
-        str(row["state"]).ljust(8),
-        str(row["version"]).ljust(16),
-        str(row["provider"]).ljust(8),
-        row["path"],
-        overflow="ignore",
-        crop=False,
-        style=str(row.get("style") or ""),
-    )
-
-
 def _binary_record_matches_runtime(installed, lib_dir: Path) -> bool:
     if not installed or not installed.is_valid or not installed.version:
         return False
@@ -167,11 +128,13 @@ def version(
     from rich.panel import Panel
     from rich.console import Console
     from rich.live import Live
+    from abx_dl.tables import binary_dependency_status, binary_dependency_table
 
     from archivebox.config import CONSTANTS
     from archivebox.config.version import get_COMMIT_HASH, get_BUILD_TIME
     from archivebox.config.permissions import ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, RUNNING_AS_UID, RUNNING_AS_GID, IN_DOCKER
     from archivebox.config.paths import get_data_locations, get_code_locations
+    from archivebox.misc.checks import is_archivebox_source_root
     from archivebox.misc.logging_util import printable_folder_status
     from archivebox.config.common import get_config, normalize_runtime_config
 
@@ -227,7 +190,37 @@ def version(
     )
     prnt()
 
-    in_data_dir = os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
+    in_data_dir = (
+        not is_archivebox_source_root(CONSTANTS.DATA_DIR)
+        and os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK)
+        and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
+    )
+    if isinstance(binaries, str):
+        requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
+    else:
+        requested_names = {name for name in (binaries or ()) if name}
+
+    def binary_is_requested(logical_name: str, actual_name: str, display_name: str) -> bool:
+        return not requested_names or bool({logical_name, actual_name, display_name} & requested_names)
+
+    def plugin_may_have_requested_binary(plugin) -> bool:
+        if not requested_names:
+            return True
+        aliases = {plugin.name}
+        for required_binary in plugin.config.required_binaries:
+            raw_name = str(required_binary.name)
+            aliases.add(raw_name)
+            template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", raw_name)
+            if template_match:
+                config_key = template_match.group(1)
+                aliases.add(config_key)
+                aliases.add(config_key.removesuffix("_BINARY"))
+                default = plugin.config.properties.get(config_key, {}).get("default")
+                if default:
+                    aliases.add(str(default))
+        normalized_aliases = {alias.lower() for alias in aliases}
+        return bool({name.lower() for name in requested_names} & normalized_aliases)
+
     if not in_data_dir:
         PANEL_TEXT = "\n".join(
             (
@@ -250,6 +243,22 @@ def version(
         )
     prnt()
 
+    if not in_data_dir and not requested_names:
+        prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
+        prnt("", "[grey53]Dependency checks require an initialized collection DATA_DIR.[/grey53]")
+        prnt()
+        prnt("[deep_sky_blue3][i] Code locations:[/deep_sky_blue3]")
+        try:
+            for name, path in get_code_locations().items():
+                if isinstance(name, str) and isinstance(path, dict):
+                    prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
+        except Exception as e:
+            prnt(f"  [red]Error getting code locations: {e}[/red]")
+        prnt()
+        prnt("[red][i] Data locations:[/red] (not in a data directory)")
+        prnt()
+        return []
+
     prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
     failures = []
     seen_failures: set[str] = set()
@@ -257,13 +266,10 @@ def version(
 
     from archivebox.plugins.discovery import get_enabled_plugins
     from abx_dl.config import get_required_binary_requests
-    from abx_dl.dependencies import load_binary
+    from abx_dl.dependencies import resolve_binary_requests
     from abx_dl.models import discover_plugins, filter_plugins
-
-    if isinstance(binaries, str):
-        requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
-    else:
-        requested_names = {name for name in (binaries or ()) if name}
+    from abx_dl.orchestrator import create_bus
+    from abxpkg.binary_service import BinaryEvent, BinaryService
 
     plugins = discover_plugins(runtime="archivebox")
     enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
@@ -289,28 +295,75 @@ def version(
 
         except Exception as e:
             prnt()
-            prnt("", f"[yellow]Warning: Could not query collection binary records, falling back to abxpkg state: {e}[/yellow]")
+            prnt("", f"[yellow]Warning: Could not query collection binary records; resolving through abxpkg: {e}[/yellow]")
+
+    declared_binary_specs: dict[str, dict[str, object]] = {}
+    for plugin_name, plugin in plugins.items():
+        if not plugin_may_have_requested_binary(plugin):
+            continue
+        plugin_requested = bool(requested_names)
+        plugin_enabled = plugin_name in enabled_plugin_names
+        logical_records = get_required_binary_requests(
+            plugin,
+            plugin.config.required_binaries,
+            overrides=runtime_config,
+            derived_overrides=derived_config,
+            run_output_dir=CONSTANTS.DATA_DIR,
+        )
+        actual_records = get_required_binary_requests(
+            plugin,
+            plugin.config.required_binaries,
+            overrides=runtime_config,
+            derived_overrides=derived_config,
+            run_output_dir=CONSTANTS.DATA_DIR,
+            logical_names=False,
+        )
+        for logical_record, actual_record in zip(logical_records, actual_records, strict=False):
+            logical_name = str(logical_record["name"])
+            actual_name = str(actual_record["name"])
+            display_name = Path(actual_name).expanduser().name if ("/" in actual_name or actual_name.startswith("~")) else logical_name
+            if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
+                continue
+            if not plugin_enabled and not requested_names:
+                continue
+            if _binary_record_matches_runtime(db_binaries.get(logical_name), config.ABXPKG_LIB_DIR):
+                continue
+            signature = json.dumps(actual_record, sort_keys=True, default=str)
+            declared_binary_specs.setdefault(signature, actual_record)
+
+    loaded_binaries: dict[str, BinaryEvent | None] = {}
+    if declared_binary_specs:
+        binary_bus = create_bus(name="ArchiveBoxVersionBinaryCheck")
+        BinaryService(binary_bus, auto_install=False, lib_dir=config.ABXPKG_LIB_DIR)
+
+        async def resolve_declared_binaries() -> dict[str, BinaryEvent | None]:
+            try:
+                return await resolve_binary_requests(binary_bus, declared_binary_specs)
+            finally:
+                await binary_bus.wait_until_idle()
+                await binary_bus.destroy(clear=False)
+
+        loaded_binaries = asyncio.run(resolve_declared_binaries())
 
     rows: list[dict[str, object]] = []
     any_rows = False
     any_available = False
     compact_paths = console.is_terminal
     live_enabled = console.is_terminal
-    live_cm = Live(_build_binary_table(rows), console=console, refresh_per_second=8) if live_enabled else None
-    if not live_enabled:
-        prnt("", "Binary".ljust(28), "Status", "Plugin".ljust(24), "State".ljust(8), "Version".ljust(16), "Provider".ljust(8), "Path")
+    live_cm = Live(binary_dependency_table(rows), console=console, refresh_per_second=8) if live_enabled else None
 
     def emit_row(row: dict[str, object]) -> None:
         rows.append(row)
         if live_cm is not None:
-            live_cm.update(_build_binary_table(rows), refresh=True)
-        else:
-            _print_binary_row(prnt, row)
+            live_cm.update(binary_dependency_table(rows), refresh=True)
 
     if live_cm is not None:
         live_cm.start()
     try:
         for plugin_name, plugin in plugins.items():
+            if not plugin_may_have_requested_binary(plugin):
+                continue
+            plugin_requested = bool(requested_names)
             plugin_enabled = plugin_name in enabled_plugin_names
             logical_records = get_required_binary_requests(
                 plugin,
@@ -331,28 +384,30 @@ def version(
                 logical_name = str(logical_record["name"])
                 actual_name = str(actual_record["name"])
                 display_name = Path(actual_name).expanduser().name if ("/" in actual_name or actual_name.startswith("~")) else logical_name
-                if (
-                    requested_names
-                    and logical_name not in requested_names
-                    and actual_name not in requested_names
-                    and display_name not in requested_names
-                ):
+                if not plugin_requested and not binary_is_requested(logical_name, actual_name, display_name):
                     continue
 
-                any_rows = True
                 installed = db_binaries.get(logical_name) if db_available else None
                 if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
                     abspath = installed.abspath
                     version_str = (installed.version or "unknown")[:15]
                     provider = (installed.binprovider or "env")[:8]
                     valid = True
+                elif not plugin_enabled and not requested_names:
+                    # `archivebox version` is expected to verify the active runtime, not
+                    # cold-load every optional plugin provider. Migration and status
+                    # checks often run with PLUGINS narrowed to a tiny set; resolving
+                    # disabled plugin binaries there can spend most of the command on
+                    # providers the current collection will never execute.
+                    continue
                 else:
-                    loaded = load_binary(actual_record)
-                    abspath = str(loaded.loaded_abspath or "")
-                    version_str = str(loaded.loaded_version or "unknown")[:15]
-                    provider = (loaded.loaded_binprovider.name if loaded.loaded_binprovider else "env")[:8]
-                    valid = loaded.is_valid
+                    loaded = loaded_binaries[json.dumps(actual_record, sort_keys=True, default=str)]
+                    abspath = loaded.abspath if loaded is not None else ""
+                    version_str = str(loaded.version or "unknown")[:15] if loaded is not None else "unknown"
+                    provider = str(loaded.binprovider or "env")[:8] if loaded is not None else "env"
+                    valid = loaded is not None
 
+                any_rows = True
                 if valid:
                     display_path = (
                         _format_binary_abspath(
@@ -366,11 +421,9 @@ def version(
                         else abspath
                     )
                     rendered_path = _render_binary_abspath(display_path) if compact_paths else display_path
-                    status = "[green]√[/green]" if plugin_enabled else "[grey53]-[/grey53]"
                     any_available = True
                 else:
                     rendered_path = "[grey53]not installed[/grey53]"
-                    status = "[red]X[/red]" if plugin_enabled else "[grey53]-[/grey53]"
                     if plugin_enabled and display_name not in seen_failures:
                         failures.append(display_name)
                         seen_failures.add(display_name)
@@ -389,8 +442,7 @@ def version(
                 emit_row(
                     {
                         "plugin": plugin_name,
-                        "state": "enabled" if plugin_enabled else "disabled",
-                        "status": status,
+                        "status": binary_dependency_status(enabled=plugin_enabled, valid=valid),
                         "binary": display_name,
                         "version": version_str if valid else "-",
                         "provider": provider if valid else "-",
@@ -398,9 +450,54 @@ def version(
                         "style": "" if plugin_enabled else "dim",
                     },
                 )
+
+        if db_available:
+            for binary_name, installed in db_binaries.items():
+                if requested_names and binary_name not in requested_names:
+                    continue
+                abspath = installed.abspath
+                version_str = (installed.version or "unknown")[:15]
+                provider = (installed.binprovider or "env")[:8]
+                row_key = _binary_row_dedupe_key(
+                    display_name=binary_name,
+                    valid=True,
+                    version=version_str,
+                    provider=provider,
+                    abspath=abspath,
+                )
+                if row_key in seen_rows:
+                    continue
+                seen_rows.add(row_key)
+
+                display_path = (
+                    _format_binary_abspath(
+                        abspath,
+                        pwd=Path.cwd(),
+                        lib_dir=config.ABXPKG_LIB_DIR,
+                        personas_dir=CONSTANTS.PERSONAS_DIR,
+                        home=Path.home(),
+                    )
+                    if compact_paths
+                    else abspath
+                )
+                emit_row(
+                    {
+                        "plugin": "(database)",
+                        "status": binary_dependency_status(enabled=True, valid=True),
+                        "binary": binary_name,
+                        "version": version_str,
+                        "provider": provider,
+                        "path": _render_binary_abspath(display_path) if compact_paths else display_path,
+                        "style": "dim",
+                    },
+                )
+                any_rows = True
+                any_available = True
     finally:
         if live_cm is not None:
             live_cm.stop()
+        else:
+            prnt(binary_dependency_table(rows))
 
     if not any_rows:
         prnt("", "[grey53]No required binaries declared for discovered plugins.[/grey53]")

archivebox/config/collection.py

@@ -194,6 +194,12 @@ def _coerce_from_str_dict(file_config: dict[str, str]) -> dict[str, Any]:
             decoded[key] = json.loads(raw)
         except (TypeError, ValueError):
             continue
+    if not str(decoded.get("BASE_URL") or "").strip():
+        from archivebox.config.common import base_url_from_legacy_server_config
+
+        legacy_base_url = base_url_from_legacy_server_config(decoded)
+        if legacy_base_url:
+            decoded["BASE_URL"] = legacy_base_url
     return decoded
 
 

archivebox/config/common.py

@@ -2,32 +2,32 @@ from __future__ import annotations
 
 __package__ = "archivebox.config"
 
+import inspect
 import json
 import os
 import re
 import secrets
-import sys
 import shutil
-import inspect
-from functools import lru_cache
+import sys
 from collections.abc import Mapping
 from datetime import timedelta
-from typing import Any, ClassVar, cast
+from functools import cache, lru_cache
 from pathlib import Path
-from urllib.parse import quote
+from typing import Any, ClassVar, cast
+from urllib.parse import quote, urlparse
 
-from rich.console import Console
+from abx_plugins.plugins.base.utils import BASE_CONFIG_PATH, build_config_model, resolve_plugin_configs
+from django.db import DatabaseError
 from pydantic import BaseModel, Field, PrivateAttr, create_model, field_validator, model_validator
 from pydantic_settings import SettingsConfigDict
-from abx_plugins.plugins.base.utils import BASE_CONFIG_PATH, build_config_model, resolve_plugin_configs
+from rich.console import Console
 
-from archivebox.config.configset import BaseConfigSet, IniConfigSettingsSource
-from archivebox.config.configset import COMPUTED_CONFIG_KEYS
+from archivebox.config.configset import COMPUTED_CONFIG_KEYS, BaseConfigSet, IniConfigSettingsSource
 
 from .constants import CONSTANTS
 from .ldap import LDAPConfig
-from .version import get_COMMIT_HASH, get_BUILD_TIME, VERSION
 from .permissions import IN_DOCKER
+from .version import VERSION, get_BUILD_TIME, get_COMMIT_HASH
 
 ConfigOverrides = Mapping[str, object]
 ConfigPayload = dict[str, object]
@@ -67,6 +67,40 @@ def permissions_from_legacy_public_flags(raw_config: Mapping[str, object]) -> st
     return None
 
 
+def base_url_from_legacy_server_config(raw_config: Mapping[str, object]) -> str:
+    """Derive the current canonical URL from the pre-0.9 server settings."""
+
+    def normalize(value: object) -> str:
+        raw = str(value or "").strip()
+        if not raw:
+            return ""
+        parsed = urlparse(raw if "://" in raw else f"http://{raw}")
+        return f"{parsed.scheme}://{parsed.netloc}" if parsed.netloc else ""
+
+    for key in ("ARCHIVE_BASE_URL", "ADMIN_BASE_URL"):
+        candidate = normalize(raw_config.get(key))
+        if candidate:
+            return candidate
+
+    listen_host = str(raw_config.get("LISTEN_HOST") or "").strip()
+    if listen_host and listen_host.lower() != "archivebox.localhost:8000":
+        return normalize(listen_host)
+
+    csrf_origins = [normalize(origin) for origin in str(raw_config.get("CSRF_TRUSTED_ORIGINS") or "").split(",")]
+    csrf_origins = [origin for origin in csrf_origins if origin]
+    nondefault_csrf_origins = [origin for origin in csrf_origins if origin != "http://admin.archivebox.localhost:8000"]
+    if len(nondefault_csrf_origins) == 1:
+        return nondefault_csrf_origins[0]
+
+    allowed_hosts = [host.strip() for host in str(raw_config.get("ALLOWED_HOSTS") or "").split(",") if host.strip() != "*"]
+    if len(allowed_hosts) == 1:
+        return normalize(allowed_hosts[0])
+
+    if len(csrf_origins) == 1:
+        return csrf_origins[0]
+    return normalize(listen_host)
+
+
 def resolve_delete_after_config_value(*configs: Mapping[str, Any] | None) -> str:
     for config in configs:
         if config is None:
@@ -272,6 +306,7 @@ class ServerConfig(BaseConfigSet):
     _scope: str = PrivateAttr(default=_SCOPE_SERVER)
 
     SERVER_SECURITY_MODES: ClassVar[tuple[str, ...]] = (
+        "auto",
         "safe-subdomains-fullreplay",
         "safe-onedomain-nojsreplay",
         "unsafe-onedomain-noadmin",
@@ -283,7 +318,7 @@ class ServerConfig(BaseConfigSet):
     BASE_URL: str = Field(default="")
     ALLOWED_HOSTS: str = Field(default="*")
     CSRF_TRUSTED_ORIGINS: str = Field(default="")
-    SERVER_SECURITY_MODE: str = Field(default="safe-subdomains-fullreplay")
+    SERVER_SECURITY_MODE: str = Field(default="auto")
 
     SNAPSHOTS_PER_PAGE: int = Field(default=50, ge=1)
     FOOTER_INFO: str = Field(
@@ -308,11 +343,21 @@ class ServerConfig(BaseConfigSet):
 
     @property
     def USES_SUBDOMAIN_ROUTING(self) -> bool:
-        return self.SERVER_SECURITY_MODE == "safe-subdomains-fullreplay"
+        if self.SERVER_SECURITY_MODE == "safe-subdomains-fullreplay":
+            return True
+        if self.SERVER_SECURITY_MODE != "auto":
+            return False
+
+        base_host = (urlparse(self.BASE_URL if "://" in self.BASE_URL else f"//{self.BASE_URL}").hostname or "").lower()
+        if base_host:
+            return base_host.endswith(".localhost")
+
+        bind_host = (urlparse(f"//{self.BIND_ADDR}").hostname or "").lower()
+        return bind_host in {"", "0.0.0.0", "127.0.0.1", "::", "::1", "localhost"}
 
     @property
     def ENABLES_FULL_JS_REPLAY(self) -> bool:
-        return self.SERVER_SECURITY_MODE in (
+        return self.USES_SUBDOMAIN_ROUTING or self.SERVER_SECURITY_MODE in (
             "safe-subdomains-fullreplay",
             "unsafe-onedomain-noadmin",
             "danger-onedomain-fullreplay",
@@ -328,7 +373,7 @@ class ServerConfig(BaseConfigSet):
 
     @property
     def SHOULD_NEUTER_RISKY_REPLAY(self) -> bool:
-        return self.SERVER_SECURITY_MODE == "safe-onedomain-nojsreplay"
+        return self.SERVER_SECURITY_MODE in ("auto", "safe-onedomain-nojsreplay")
 
     @property
     def IS_UNSAFE_MODE(self) -> bool:
@@ -350,7 +395,16 @@ class DatabaseConfig(BaseConfigSet):
     toml_section_header: str = "DATABASE_CONFIG"
     _scope: str = PrivateAttr(default=_SCOPE_SERVER)
 
+    DATABASE_ENGINE: str = Field(
+        default="sqlite",
+        alias="ARCHIVEBOX_DATABASE_ENGINE",
+        pattern=r"(?i)^(sqlite|postgres|postgresql)$",
+    )
     DATABASE_NAME: str = Field(default=str(CONSTANTS.DATABASE_FILE), alias="ARCHIVEBOX_DATABASE_NAME")
+    DATABASE_HOST: str = Field(default="127.0.0.1", alias="ARCHIVEBOX_DATABASE_HOST")
+    DATABASE_PORT: int = Field(default=5432, alias="ARCHIVEBOX_DATABASE_PORT", ge=1, le=65535)
+    DATABASE_USER: str = Field(default="archivebox", alias="ARCHIVEBOX_DATABASE_USER")
+    DATABASE_PASSWORD: str = Field(default="", alias="ARCHIVEBOX_DATABASE_PASSWORD")
     SQLITE_JOURNAL_MODE: str = Field(
         default="WAL",
         alias="ARCHIVEBOX_SQLITE_JOURNAL_MODE",
@@ -492,7 +546,7 @@ class SearchBackendConfig(BaseConfigSet):
     toml_section_header: str = "SEARCH_BACKEND_CONFIG"
     _scope: str = PrivateAttr(default=_SCOPE_SERVER)
 
-    SEARCH_BACKEND_ENGINE: str = Field(default="ripgrep", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
+    SEARCH_BACKEND_ENGINE: str = Field(default="sonic", json_schema_extra={"scope": _SCOPE_CRAWL_EXECUTION})
 
 
 def _plugin_user_config_value(value: Any) -> str:
@@ -507,6 +561,32 @@ def _plugin_user_config(config: Mapping[str, object]) -> dict[str, str]:
     return {key: _plugin_user_config_value(value) for key, value in config.items()}
 
 
+def _plugin_input_config(config: Mapping[str, object]) -> dict[str, object]:
+    """Return raw config inputs understood by core or plugin schemas."""
+    input_names = _archivebox_config_input_names()
+    return {str(key): value for key, value in config.items() if str(key) in input_names}
+
+
+def _explicit_plugin_enabled_keys(config: Mapping[str, object]) -> set[str]:
+    """Return canonical ``*_ENABLED`` keys explicitly provided by the user.
+
+    ``PLUGINS`` is a selector, not an override for concrete plugin enable flags.
+    If a user passes ``PLUGINS=wget`` and ``SAVE_WGET=False`` together, the
+    alias-resolved ``WGET_ENABLED=False`` must survive the model validator that
+    expands ``PLUGINS`` into concrete booleans.
+    """
+    provided = {str(key) for key in config}
+    explicit: set[str] = set()
+    for key, prop in _plugin_config_properties(PLUGIN_CONFIG_SCHEMAS).items():
+        if not key.endswith("_ENABLED") or not isinstance(prop, Mapping):
+            continue
+        aliases = prop.get("x-aliases") if isinstance(prop, Mapping) else None
+        input_names = {key, *(str(alias) for alias in aliases if str(alias).strip())} if isinstance(aliases, list) else {key}
+        if provided & input_names:
+            explicit.add(key)
+    return explicit
+
+
 def _discover_plugin_config_schemas() -> PluginSchemaDocuments:
     from archivebox.plugins.discovery import discover_plugin_configs
 
@@ -608,7 +688,7 @@ class ArchiveBoxBaseConfig(
     @classmethod
     def _plugin_field_scope(cls, key: str) -> str | None:
         scope = None
-        for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items():
+        for schema in PLUGIN_CONFIG_SCHEMAS.values():
             properties = schema.get("properties") if isinstance(schema, dict) else None
             if not isinstance(properties, dict) or key not in properties:
                 continue
@@ -620,7 +700,7 @@ class ArchiveBoxBaseConfig(
         return scope
 
     @classmethod
-    @lru_cache(maxsize=None)
+    @cache
     def scope_for_key(cls, key: str) -> str:
         for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items():
             properties = schema.get("properties") if isinstance(schema, dict) else None
@@ -664,7 +744,7 @@ class ArchiveBoxBaseConfig(
     def _scoped_config(self, *, include_execution: bool) -> dict[str, Any]:
         keys = type(self)._crawl_runtime_keys() if include_execution else type(self)._crawl_frozen_keys()
         payload = self.model_dump(mode="json")
-        return {key: payload[key] for key in keys if payload.get(key) is not None}
+        return {key: payload[key] for key in keys if payload.get(key, None) is not None}
 
     def for_crawl(self) -> dict[str, Any]:
         """Config scoped to crawl execution, without runtime object overlays."""
@@ -752,15 +832,37 @@ class ArchiveBoxBaseConfig(
 
     @model_validator(mode="after")
     def derive_plugin_enabled_config(self):
+        self._derive_plugin_enabled_config(respect_current_enabled=False)
+        return self
+
+    def _derive_plugin_enabled_config(self, *, respect_current_enabled: bool) -> None:
         plugin_names = _normalize_plugins_config_value(self.PLUGINS)
-        selected_plugins = _plugins_with_required_plugins(plugin_names) if plugin_names else set()
+        enabled_config_keys = _plugin_enabled_config_keys()
+        disabled_plugins = []
+        if respect_current_enabled:
+            disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
+        selected_plugin_roots = plugin_names
+        if selected_plugin_roots:
+            from abx_dl.models import discover_plugins, filter_plugins
+
+            selected_plugins = set(
+                filter_plugins(
+                    discover_plugins(runtime="archivebox"),
+                    sorted(selected_plugin_roots),
+                    include_providers=True,
+                    disabled_names=disabled_plugins,
+                ),
+            )
+        else:
+            selected_plugins = set()
         search_backend = self.SEARCH_BACKEND_ENGINE.strip().lower()
         if search_backend:
             selected_plugins.add(f"search_backend_{search_backend}")
-        for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
+        if respect_current_enabled:
+            selected_plugins.difference_update(disabled_plugins)
+        for plugin_name, enabled_key in enabled_config_keys.items():
             if plugin_names or plugin_name in selected_plugins:
                 setattr(self, enabled_key, plugin_name in selected_plugins)
-        return self
 
 
 def _build_archivebox_config_model(plugin_schemas: PluginSchemaDocuments) -> type[ArchiveBoxBaseConfig]:
@@ -817,23 +919,6 @@ def _plugin_enabled_config_keys() -> dict[str, str]:
     return enabled_keys
 
 
-def _plugins_with_required_plugins(plugin_names: set[str]) -> set[str]:
-    selected = set(plugin_names)
-    pending = list(selected)
-    while pending:
-        plugin_name = pending.pop()
-        schema = PLUGIN_CONFIG_SCHEMAS.get(plugin_name, {})
-        required_plugins = schema.get("required_plugins") if isinstance(schema, dict) else None
-        if not isinstance(required_plugins, list):
-            continue
-        for required_plugin in required_plugins:
-            required_plugin_name = str(required_plugin).strip().lower()
-            if required_plugin_name and required_plugin_name not in selected:
-                selected.add(required_plugin_name)
-                pending.append(required_plugin_name)
-    return selected
-
-
 def get_live_config_url(key: str) -> str:
     return f"{LIVE_CONFIG_BASE_URL}{quote(key)}/"
 
@@ -957,8 +1042,12 @@ def get_request_config(request: Any, *, resolve_plugins: bool = False) -> Archiv
     request_config_resolves_plugins = bool(request_state.get("_archivebox_config_resolves_plugins", False))
     if request_config is None or (resolve_plugins and not request_config_resolves_plugins):
         request_config = get_config(resolve_plugins=resolve_plugins)
-        request.archivebox_config = request_config
         request._archivebox_config_resolves_plugins = resolve_plugins
+    if request_config.SERVER_SECURITY_MODE == "auto":
+        request_host = (urlparse(f"//{request.get_host()}").hostname or "").lower().rstrip(".")
+        effective_mode = "safe-subdomains-fullreplay" if request_host.endswith(".localhost") else "safe-onedomain-nojsreplay"
+        request_config = request_config.model_copy(update={"SERVER_SECURITY_MODE": effective_mode})
+    request.archivebox_config = request_config
     return request_config
 
 
@@ -998,7 +1087,7 @@ def get_config(
                 from archivebox.machine.models import Machine
 
                 machine = Machine.current()
-        except Exception:
+        except (DatabaseError, ImportError, RuntimeError, TypeError, ValueError):
             machine = None
 
     if persona is None and crawl is not None:
@@ -1007,9 +1096,11 @@ def get_config(
     config_data: ConfigPayload = dict(defaults or {})
     config_data["PERSONAS_DIR"] = str(CONSTANTS.PERSONAS_DIR)
     base_config_payload: ConfigPayload = {}
+    base_config_model = ArchiveBoxConfig()
+
     if crawl_config_base:
         config_data.update(
-            normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
+            normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
         )
         config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
     elif base_config is not None:
@@ -1020,11 +1111,16 @@ def get_config(
         config_data.update(normalize_runtime_config(base_config_payload, exclude_runtime_derived=True, json_safe=False))
     else:
         config_data.update(
-            normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
+            normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
         )
-        legacy_permissions = permissions_from_legacy_public_flags({**BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE), **os.environ})
+        legacy_config = {**BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE), **os.environ}
+        legacy_permissions = permissions_from_legacy_public_flags(legacy_config)
         if legacy_permissions:
             config_data["PERMISSIONS"] = legacy_permissions
+        if not str(config_data.get("BASE_URL") or "").strip():
+            legacy_base_url = base_url_from_legacy_server_config(legacy_config)
+            if legacy_base_url:
+                config_data["BASE_URL"] = legacy_base_url
 
     scope_overrides: ConfigPayload = {}
 
@@ -1072,22 +1168,29 @@ def get_config(
     archivebox_scope_overrides = {key: value for key, value in scope_overrides.items() if key in _archivebox_config_input_names()}
     config_data.update(archivebox_scope_overrides)
 
+    explicit_plugin_enabled_keys: set[str] = set()
     if resolve_plugins:
         plugin_schemas = {plugin_name: schema for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items() if isinstance(schema, dict)}
         plugin_global_config = {key: str(value) if isinstance(value, Path) else value for key, value in config_data.items()}
+        crawl_selected_plugins = crawl_config_base and bool(_normalize_plugins_config_value(dict(crawl.config or {}).get("PLUGINS")))
+        # A frozen crawl-level PLUGINS selector is the exact extractor set for
+        # this crawl. Env/file *_ENABLED values are ambient defaults and must
+        # not be re-applied after validation, or a later process env can change
+        # which hooks an already-queued crawl runs.
+        if not crawl_selected_plugins:
+            explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(os.environ))
+        explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(scope_overrides))
         plugin_user_config = _plugin_user_config(
             {
-                **normalize_runtime_config(config_data, only_crawl_execution=True, json_safe=False),
+                **_plugin_input_config(os.environ),
                 **scope_overrides,
             },
         )
         if not crawl_config_base:
+            file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
+            explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(file_config))
             plugin_user_config = {
-                **normalize_runtime_config(
-                    BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE),
-                    exclude_runtime_derived=True,
-                    json_safe=False,
-                ),
+                **_plugin_user_config(_plugin_input_config(file_config)),
                 **plugin_user_config,
             }
         plugin_sections = resolve_plugin_configs(
@@ -1112,6 +1215,13 @@ def get_config(
         if crawl_config_base:
             config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
         config_data.update(archivebox_scope_overrides)
+        if crawl_selected_plugins:
+            # Crawl PLUGINS is a frozen selector. Drop ambient enable flags from
+            # env/file/defaults before model validation so the selected plugin
+            # set is derived from the crawl, not from the worker process that
+            # happens to resume it.
+            for enabled_key in _plugin_enabled_config_keys().values():
+                config_data.pop(enabled_key, None)
 
     # Decode JSON-encoded complex values (dict/list fields) that came from
     # string-only sources before validation. ``IniConfigSettingsSource`` does
@@ -1137,6 +1247,15 @@ def get_config(
             )
 
     config = ArchiveBoxConfig.model_validate(config_data)
+    if config.SERVER_SECURITY_MODE == "auto":
+        base_host = (urlparse(config.BASE_URL if "://" in config.BASE_URL else f"//{config.BASE_URL}").hostname or "").lower()
+        if base_host.endswith(".localhost"):
+            config = config.model_copy(update={"SERVER_SECURITY_MODE": "safe-subdomains-fullreplay"})
+    for key in explicit_plugin_enabled_keys:
+        if key in config_data:
+            setattr(config, key, config_data[key])
+    if config.PLUGINS:
+        config._derive_plugin_enabled_config(respect_current_enabled=True)
     if redact_sensitive:
         for key in type(config).model_fields:
             value = config[key]

archivebox/config/constants.py

@@ -293,5 +293,3 @@ CONSTANTS = ConstantsDict
 CONSTANTS_CONFIG = AttrDict({key: value for key, value in CONSTANTS.__dict__.items() if key.isupper() and not key.startswith("_")})
 
 # add all key: values to globals() for easier importing, e.g.:
-# from archivebox.config.constants import IS_ROOT, PERSONAS_DIR, ...
-# globals().update(CONSTANTS)

archivebox/config/django.py

@@ -2,19 +2,17 @@ __package__ = "archivebox.config"
 
 import os
 import sys
-import subprocess
-
-from datetime import datetime, timezone
-
-from rich.console import Console
+from datetime import UTC, datetime
 
 import django
 import django.db
+from django.core.exceptions import ImproperlyConfigured
+from rich.console import Console
 
 from archivebox.misc import logging
 
-from .constants import CONSTANTS
 from .common import get_config
+from .constants import CONSTANTS
 
 CONFIG = get_config()
 
@@ -31,16 +29,19 @@ logging.CONSOLE = CONSOLE
 DJANGO_SET_UP = False
 
 
-def setup_django(check_db=False, in_memory_db=False) -> None:
+def setup_django(check_db=False) -> None:
     from rich.panel import Panel
 
+    from archivebox.misc.checks import check_not_inside_source_dir
+
     global DJANGO_SET_UP
 
     if DJANGO_SET_UP:
-        # raise Exception('django is already set up!')
         # TODO: figure out why CLI entrypoints with init_pending are running this twice sometimes
         return
 
+    check_not_inside_source_dir(CONSTANTS.DATA_DIR)
+
     # SQLite creates index.sqlite3 during django.setup()/migrate. Apply the
     # ArchiveBox file-mode policy before any DB connection can create the file,
     # otherwise a permissive parent umask can expose a just-created DB until a
@@ -51,18 +52,17 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
     # Keeping them out of archivebox.__init__ avoids paying Django/Daphne setup
     # cost for cheap CLI startup paths like `archivebox <cmd> --help`.
     import archivebox.misc.monkey_patches  # noqa: F401
-
-    from archivebox.config.permissions import IS_ROOT, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, SudoPermission
+    from archivebox.config.permissions import ARCHIVEBOX_GROUP, ARCHIVEBOX_USER, IS_ROOT, SudoPermission
 
     # if running as root, chown the data dir to the archivebox user to make sure it's accessible to the archivebox user
     if IS_ROOT and ARCHIVEBOX_USER != 0:
         with SudoPermission(uid=0):
             # running as root is a special case where it's ok to be a bit slower
             # make sure data dir is always owned by the correct user
-            subprocess.run(["chown", f"{ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP}", str(CONSTANTS.DATA_DIR)], stderr=subprocess.DEVNULL)
+            os.chown(CONSTANTS.DATA_DIR, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP)
             if CONSTANTS.DATA_DIR.exists():
                 for child in CONSTANTS.DATA_DIR.iterdir():
-                    subprocess.run(["chown", f"{ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP}", str(child)], stderr=subprocess.DEVNULL)
+                    os.chown(child, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP)
 
     # Suppress the "database access during app initialization" warning
     # This warning can be triggered during django.setup() but is safe to ignore
@@ -78,40 +78,30 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
     try:
         from django.core.management import call_command
 
-        if in_memory_db:
-            raise Exception("dont use this anymore")
-
-            # some commands dont store a long-lived sqlite3 db file on disk.
-            # in those cases we create a temporary in-memory db and run the migrations
-            # immediately to get a usable in-memory-database at startup
-            os.environ.setdefault("ARCHIVEBOX_DATABASE_NAME", ":memory:")
+        # Initialize the configured file-based database without running
+        # migrations automatically; `archivebox init` owns migrations.
+        try:
             django.setup()
+        # User config and import errors are reported through one CLI message.
+        except (ImproperlyConfigured, django.db.Error, ImportError, OSError, RuntimeError, ValueError) as e:
+            is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ("help", "version", "--help", "--version"))
+            if not is_using_meta_cmd:
+                # show error message to user only if they're not running a meta command / just trying to get help
+                STDERR.print()
+                STDERR.print(
+                    Panel(
+                        f"\n[red]{e.__class__.__name__}[/red]: [yellow]{e}[/yellow]\nPlease check your config and [blue]DATA_DIR[/blue] permissions.\n",
+                        title="\n\n[red][X] Error while trying to load database![/red]",
+                        subtitle="[grey53]NO WRITES CAN BE PERFORMED[/grey53]",
+                        expand=False,
+                        style="bold red",
+                    ),
+                )
+                STDERR.print()
+                import traceback
 
-            call_command("migrate", interactive=False, verbosity=0)
-        else:
-            # Otherwise use default sqlite3 file-based database and initialize django
-            # without running migrations automatically (user runs them manually by calling init)
-            try:
-                django.setup()
-            except Exception as e:
-                is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ("help", "version", "--help", "--version"))
-                if not is_using_meta_cmd:
-                    # show error message to user only if they're not running a meta command / just trying to get help
-                    STDERR.print()
-                    STDERR.print(
-                        Panel(
-                            f"\n[red]{e.__class__.__name__}[/red]: [yellow]{e}[/yellow]\nPlease check your config and [blue]DATA_DIR[/blue] permissions.\n",
-                            title="\n\n[red][X] Error while trying to load database![/red]",
-                            subtitle="[grey53]NO WRITES CAN BE PERFORMED[/grey53]",
-                            expand=False,
-                            style="bold red",
-                        ),
-                    )
-                    STDERR.print()
-                    import traceback
-
-                    traceback.print_exc()
-                return
+                traceback.print_exc()
+            return
 
         from archivebox.core.settings_logging import ERROR_LOG as DEFAULT_ERROR_LOG
 
@@ -119,13 +109,13 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
         error_log = DEFAULT_ERROR_LOG
         with open(error_log, "a", encoding="utf-8") as f:
             command = " ".join(sys.argv)
-            ts = datetime.now(timezone.utc).strftime("%Y-%m-%d__%H:%M:%S")
+            ts = datetime.now(UTC).strftime("%Y-%m-%d__%H:%M:%S")
             config = get_config()
             f.write(f"\n> {command}; TS={ts} VERSION={CONSTANTS.VERSION} IN_DOCKER={config.IN_DOCKER} IS_TTY={config.IS_TTY}\n")
 
         if check_db:
             # make sure the data dir is owned by a non-root user
-            if CONSTANTS.DATA_DIR.stat().st_uid == 0:
+            if CONSTANTS.DATA_DIR.stat().st_uid == 0 and not (IS_ROOT and ARCHIVEBOX_USER == 0):
                 STDERR.print("[red][X] Error: ArchiveBox DATA_DIR cannot be owned by root![/red]")
                 STDERR.print(f"    {CONSTANTS.DATA_DIR}")
                 STDERR.print()
@@ -151,23 +141,14 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
             for conn in connections.all():
                 conn.close_if_unusable_or_obsolete()
 
-            sql_index_path = CONSTANTS.DATABASE_FILE
-            assert os.access(sql_index_path, os.F_OK), (
-                f"No database file {sql_index_path} found in: {CONSTANTS.DATA_DIR} (Are you in an ArchiveBox collection directory?)"
+            from archivebox.misc.db import database_display_location, database_exists
+
+            assert database_exists(), (
+                f"No database {database_display_location()} found for: {CONSTANTS.DATA_DIR} (Are you in an ArchiveBox collection directory?)"
             )
 
-            # https://docs.pydantic.dev/logfire/integrations/django/ Logfire Debugging
-            # if settings.DEBUG_LOGFIRE:
-            #     from opentelemetry.instrumentation.sqlite3 import SQLite3Instrumentor
-            #     SQLite3Instrumentor().instrument()
-
-            #     import logfire
-
-            #     logfire.configure()
-            #     logfire.instrument_django(is_sql_commentor_enabled=True)
-            #     logfire.info(f'Started ArchiveBox v{CONSTANTS.VERSION}', argv=sys.argv)
-
     except KeyboardInterrupt:
+        DJANGO_SET_UP = False
         raise
 
     DJANGO_SET_UP = True

archivebox/config/paths.py

@@ -5,7 +5,6 @@ import socket
 import hashlib
 import tempfile
 import platform
-import subprocess
 from pathlib import Path
 from functools import cache
 from datetime import datetime
@@ -59,9 +58,11 @@ def _get_collection_id(DATA_DIR=DATA_DIR, force_create=False) -> str:
     collection_id = hashlib.sha256(f"{machine_id}:{collection_path}@{creation_date}".encode()).hexdigest()[:8]
 
     try:
-        # only persist collection_id file if we already have an index.sqlite3 file present
+        # only persist collection_id file if this dir already looks like a real collection
+        # (has an index.sqlite3, or an ArchiveBox.conf when the DB lives in postgres),
         # otherwise we might be running in a directory that is not a collection, no point creating cruft files
-        collection_is_active = os.path.isfile(DATABASE_FILE) and os.path.isdir(ARCHIVE_DIR) and os.access(DATA_DIR, os.W_OK)
+        collection_marker = os.path.isfile(DATABASE_FILE) or os.path.isfile(DATA_DIR / "ArchiveBox.conf")
+        collection_is_active = collection_marker and os.path.isdir(ARCHIVE_DIR) and os.access(DATA_DIR, os.W_OK)
         if collection_is_active or force_create:
             collection_id_file.write_text(collection_id)
 
@@ -69,9 +70,9 @@ def _get_collection_id(DATA_DIR=DATA_DIR, force_create=False) -> str:
             if IS_ROOT:
                 with SudoPermission(uid=0):
                     if ARCHIVEBOX_USER == 0:
-                        subprocess.run(["chmod", "777", str(collection_id_file)])
+                        collection_id_file.chmod(0o777)
                     else:
-                        subprocess.run(["chown", str(ARCHIVEBOX_USER), str(collection_id_file)])
+                        os.chown(collection_id_file, ARCHIVEBOX_USER, -1)
     except (OSError, FileNotFoundError, PermissionError):
         pass
     return collection_id
@@ -129,7 +130,7 @@ def dir_is_writable(dir_path: Path, uid: int | None = None, gid: int | None = No
         if chown:
             # try fixing it using sudo permissions
             with SudoPermission(uid=uid, fallback=fallback):
-                subprocess.run(["chown", f"{uid}:{gid}", str(dir_path)], stderr=subprocess.DEVNULL)
+                os.chown(dir_path, uid, gid)
             return dir_is_writable(dir_path, uid=uid, gid=gid, fallback=fallback, chown=False)
     return False
 
@@ -271,6 +272,24 @@ def get_or_create_working_lib_dir(autofix=True, quiet=False, config: "ArchiveBox
         raise OSError(f"ArchiveBox is unable to find a writable ABXPKG_LIB_DIR, tried {CANDIDATES}!")
 
 
+def _sql_index_location() -> dict:
+    from archivebox.misc.db import database_display_location, database_exists, is_postgres
+
+    if is_postgres():
+        return {
+            "path": database_display_location(),
+            "enabled": True,
+            "is_valid": database_exists(),
+            "is_mount": False,
+        }
+    return {
+        "path": DATABASE_FILE.resolve(),
+        "enabled": True,
+        "is_valid": os.path.isfile(DATABASE_FILE) and os.access(DATABASE_FILE, os.R_OK) and os.access(DATABASE_FILE, os.W_OK),
+        "is_mount": os.path.ismount(DATABASE_FILE.resolve()),
+    }
+
+
 def get_data_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs):
     from archivebox.config.constants import CONSTANTS
     from archivebox.config.common import get_config
@@ -297,12 +316,7 @@ def get_data_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs
                 and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
                 and os.access(CONSTANTS.CONFIG_FILE, os.W_OK),
             },
-            "SQL_INDEX": {
-                "path": DATABASE_FILE.resolve(),
-                "enabled": True,
-                "is_valid": os.path.isfile(DATABASE_FILE) and os.access(DATABASE_FILE, os.R_OK) and os.access(DATABASE_FILE, os.W_OK),
-                "is_mount": os.path.ismount(DATABASE_FILE.resolve()),
-            },
+            "SQL_INDEX": _sql_index_location(),
             "ARCHIVE_DIR": {
                 "path": CONSTANTS.ARCHIVE_DIR.resolve(),
                 "enabled": True,
@@ -348,11 +362,6 @@ def get_data_locations(config: "ArchiveBoxConfig | None" = None, **config_kwargs
                 and os.access(tmp_dir, os.W_OK)
                 and tmp_dir_socket_path_is_short_enough(tmp_dir),
             },
-            # "CACHE_DIR": {
-            #     "path": CACHE_DIR.resolve(),
-            #     "enabled": True,
-            #     "is_valid": os.access(CACHE_DIR, os.R_OK) and os.access(CACHE_DIR, os.W_OK),                        # read + write
-            # },
         },
     )
 

archivebox/config/permissions.py

@@ -1,19 +1,51 @@
 __package__ = "archivebox.config"
 
 import os
-import pwd
-import sys
-import socket
 import platform
+import pwd
+import socket
+import stat
+import subprocess
+import sys
+from contextlib import contextmanager, suppress
+from pathlib import Path
 from typing import cast
 
 from rich import print
 
-from pathlib import Path
-from contextlib import contextmanager
-
 #############################################################################################
 
+
+def is_root_identity(running_uid: int, effective_uid: int) -> bool:
+    return running_uid == 0 or effective_uid == 0
+
+
+def select_archivebox_user(
+    *,
+    running_uid: int,
+    running_gid: int,
+    effective_uid: int,
+    effective_gid: int,
+    sudo_uid: int,
+    sudo_gid: int,
+    data_dir_uid: int,
+    data_dir_gid: int,
+    account_uid: int | None,
+    account_gid: int | None,
+    data_dir_owner_exists: bool = True,
+) -> tuple[int, int]:
+    if running_uid == 0:
+        if data_dir_uid != 0 and data_dir_owner_exists:
+            return data_dir_uid, data_dir_gid
+        if account_uid is not None and account_gid is not None:
+            return account_uid, account_gid
+
+    return (
+        effective_uid or running_uid or sudo_uid,
+        effective_gid or running_gid or sudo_gid,
+    )
+
+
 DATA_DIR = Path(os.getcwd())
 
 try:
@@ -24,44 +56,82 @@ except PermissionError:
     DATA_DIR_UID = 0
     DATA_DIR_GID = 0
 
+try:
+    pwd.getpwuid(DATA_DIR_UID)
+    DATA_DIR_OWNER_EXISTS = True
+except KeyError:
+    DATA_DIR_OWNER_EXISTS = False
+
 DEFAULT_UID = 911
 DEFAULT_GID = 911
 RUNNING_AS_UID = os.getuid()
 RUNNING_AS_GID = os.getgid()
 EUID = os.geteuid()
 EGID = os.getegid()
-SUDO_UID = int(os.environ.get("SUDO_UID", 0))
-SUDO_GID = int(os.environ.get("SUDO_GID", 0))
+SUDO_UID = int(os.environ.get("SUDO_UID", "0"))
+SUDO_GID = int(os.environ.get("SUDO_GID", "0"))
 USER: str = Path("~").expanduser().resolve().name
 HOSTNAME: str = cast(str, max([socket.gethostname(), platform.node()], key=len))
 
-IS_ROOT = RUNNING_AS_UID == 0
-IN_DOCKER = os.environ.get("IN_DOCKER", False) in ("1", "true", "True", "TRUE", "yes")
-# IN_DOCKER_COMPOSE =  # TODO: figure out a way to detect if running in docker compose
-
+IS_ROOT = is_root_identity(RUNNING_AS_UID, EUID)
+IN_DOCKER = os.environ.get("IN_DOCKER", "") in ("1", "true", "True", "TRUE", "yes")
 
 FALLBACK_UID = RUNNING_AS_UID or SUDO_UID
 FALLBACK_GID = RUNNING_AS_GID or SUDO_GID
-try:
-    ARCHIVEBOX_ACCOUNT = pwd.getpwnam("archivebox")
-except KeyError:
-    ARCHIVEBOX_ACCOUNT = None
 
-if DATA_DIR_UID != 0:
-    ARCHIVEBOX_USER = DATA_DIR_UID
-    ARCHIVEBOX_GROUP = DATA_DIR_GID
-elif RUNNING_AS_UID == 0 and ARCHIVEBOX_ACCOUNT is not None:
-    ARCHIVEBOX_USER = ARCHIVEBOX_ACCOUNT.pw_uid
-    ARCHIVEBOX_GROUP = ARCHIVEBOX_ACCOUNT.pw_gid
-else:
-    ARCHIVEBOX_USER = EUID or RUNNING_AS_UID or FALLBACK_UID
-    ARCHIVEBOX_GROUP = EGID or RUNNING_AS_GID or FALLBACK_GID
+
+def get_or_create_archivebox_account():
+    try:
+        return pwd.getpwnam("archivebox")
+    except KeyError:
+        pass
+
+    if RUNNING_AS_UID != 0 or platform.system() != "Linux":
+        return None
+
+    useradd = "/usr/sbin/useradd" if Path("/usr/sbin/useradd").exists() else "useradd"
+    command = [
+        useradd,
+        "--system",
+        "--create-home",
+        "--home-dir",
+        "/var/lib/archivebox",
+        "--shell",
+        "/bin/bash",
+        "archivebox",
+    ]
+    try:
+        subprocess.run(command, check=True)
+    except (OSError, subprocess.CalledProcessError) as err:
+        # Another concurrently starting process may have created it first.
+        try:
+            return pwd.getpwnam("archivebox")
+        except KeyError:
+            raise RuntimeError(f"Failed to create the archivebox system user with: {' '.join(command)}") from err
+    return pwd.getpwnam("archivebox")
+
+
+ARCHIVEBOX_ACCOUNT = get_or_create_archivebox_account()
+
+ARCHIVEBOX_USER, ARCHIVEBOX_GROUP = select_archivebox_user(
+    running_uid=RUNNING_AS_UID,
+    running_gid=RUNNING_AS_GID,
+    effective_uid=EUID,
+    effective_gid=EGID,
+    sudo_uid=SUDO_UID,
+    sudo_gid=SUDO_GID,
+    data_dir_uid=DATA_DIR_UID,
+    data_dir_gid=DATA_DIR_GID,
+    account_uid=ARCHIVEBOX_ACCOUNT.pw_uid if ARCHIVEBOX_ACCOUNT is not None else None,
+    account_gid=ARCHIVEBOX_ACCOUNT.pw_gid if ARCHIVEBOX_ACCOUNT is not None else None,
+    data_dir_owner_exists=DATA_DIR_OWNER_EXISTS,
+)
 if not USER:
     try:
         # alternative method 1 to get username
         USER = pwd.getpwuid(ARCHIVEBOX_USER).pw_name
-    except Exception:
-        pass
+    except (KeyError, OSError):
+        USER = ""
 
 if not USER:
     try:
@@ -69,41 +139,160 @@ if not USER:
         import getpass
 
         USER = getpass.getuser()
-    except Exception:
-        pass
+    except OSError:
+        USER = ""
 
 if not USER:
     try:
         # alternative method 3 to get username
         USER = os.getlogin() or "archivebox"
-    except Exception:
+    except OSError:
         USER = "archivebox"
 
 ARCHIVEBOX_USER_EXISTS = False
 try:
     pwd.getpwuid(ARCHIVEBOX_USER)
     ARCHIVEBOX_USER_EXISTS = True
-except Exception:
+except KeyError:
     ARCHIVEBOX_USER_EXISTS = False
 
 
+ROOT_HANDOFF_NAMES = (
+    ".archivebox_id",
+    "ArchiveBox.conf",
+    "index.sqlite3",
+    "index.sqlite3-shm",
+    "index.sqlite3-wal",
+    "archive",
+    "cache",
+    "custom_plugins",
+    "custom_templates",
+    "lib",
+    "logs",
+    "personas",
+    "sonic",
+    "sources",
+    "tmp",
+    "users",
+)
+
+
+def root_data_dir_handoff_paths(data_dir: Path, argv: list[str]) -> tuple[Path, ...]:
+    """Return bounded top-level paths safe to hand from root to archivebox."""
+
+    data_dir = data_dir.resolve()
+    if data_dir == Path("/"):
+        return ()
+
+    try:
+        children = tuple(data_dir.iterdir())
+    except (FileNotFoundError, PermissionError):
+        return ()
+
+    is_init = "init" in argv[1:]
+    collection_exists = any((data_dir / marker).exists() for marker in (".archivebox_id", "ArchiveBox.conf", "index.sqlite3"))
+    if not collection_exists and not (is_init and not children):
+        return ()
+
+    top_level_paths = tuple(data_dir / name for name in ROOT_HANDOFF_NAMES if (data_dir / name).exists())
+    log_files = tuple((data_dir / "logs").glob("*.log"))
+    return (data_dir, *top_level_paths, *log_files)
+
+
+def root_should_handoff_data_dir(
+    *,
+    is_root: bool,
+    data_dir_uid: int,
+    account_uid: int | None,
+    data_dir_owner_exists: bool = True,
+) -> bool:
+    """Allow bounded handoff only for root or archivebox-owned collection roots."""
+
+    return is_root and account_uid is not None and (not data_dir_owner_exists or data_dir_uid in (0, account_uid))
+
+
+def root_parent_can_grant_group_traversal(*, parent_uid: int, parent_gid: int, parent_mode: int, account_gid: int) -> bool:
+    """Return whether adding archivebox traversal preserves existing parent access."""
+
+    if parent_uid != 0 or parent_mode & stat.S_IXOTH:
+        return False
+    if parent_gid == account_gid:
+        return not bool(parent_mode & stat.S_IXGRP)
+    return not bool(parent_mode & stat.S_IRWXG)
+
+
+def handoff_root_owned_data_dir() -> None:
+    account_uid = ARCHIVEBOX_ACCOUNT.pw_uid if ARCHIVEBOX_ACCOUNT is not None else None
+    if not root_should_handoff_data_dir(
+        is_root=IS_ROOT,
+        data_dir_uid=DATA_DIR_UID,
+        account_uid=account_uid,
+        data_dir_owner_exists=DATA_DIR_OWNER_EXISTS,
+    ):
+        return
+
+    handoff_paths = root_data_dir_handoff_paths(DATA_DIR, sys.argv)
+    if not handoff_paths:
+        return
+
+    # A root user's collection commonly lives below /root, which the archivebox
+    # account cannot traverse after EUID is dropped. Grant only that group the
+    # missing execute bit on root-owned parents; never walk collection contents.
+    for parent in DATA_DIR.resolve().parents:
+        if parent == Path("/"):
+            continue
+        parent_stat = parent.stat(follow_symlinks=False)
+        if root_parent_can_grant_group_traversal(
+            parent_uid=parent_stat.st_uid,
+            parent_gid=parent_stat.st_gid,
+            parent_mode=parent_stat.st_mode,
+            account_gid=ARCHIVEBOX_ACCOUNT.pw_gid,
+        ):
+            os.chown(parent, -1, ARCHIVEBOX_ACCOUNT.pw_gid, follow_symlinks=False)
+            os.chmod(parent, stat.S_IMODE(parent_stat.st_mode) | stat.S_IXGRP, follow_symlinks=False)
+
+    for path in handoff_paths:
+        try:
+            os.chown(path, ARCHIVEBOX_ACCOUNT.pw_uid, ARCHIVEBOX_ACCOUNT.pw_gid, follow_symlinks=False)
+        except (FileNotFoundError, PermissionError):
+            pass
+
+
 #############################################################################################
 
 
 def drop_privileges():
     """If running as root, drop privileges to the data dir owner or archivebox user."""
 
+    handoff_root_owned_data_dir()
+
     # Always run ArchiveBox as the user that owns the data dir, or as the
     # archivebox service account when the data dir is root-owned.
-    if os.getuid() == 0:
-        if os.geteuid() != ARCHIVEBOX_USER and ARCHIVEBOX_USER != 0 and ARCHIVEBOX_USER_EXISTS:
+    if os.geteuid() == 0 and ARCHIVEBOX_USER != 0 and ARCHIVEBOX_USER_EXISTS:
+        pw_record = pwd.getpwuid(ARCHIVEBOX_USER)
+        os.initgroups(pw_record.pw_name, ARCHIVEBOX_GROUP)
+        if os.getegid() != ARCHIVEBOX_GROUP:
+            os.setegid(ARCHIVEBOX_GROUP)
+        if os.geteuid() != ARCHIVEBOX_USER:
             os.seteuid(ARCHIVEBOX_USER)
 
-            # update environment variables so that subprocesses dont try to write to /root
-            pw_record = pwd.getpwuid(ARCHIVEBOX_USER)
-            os.environ["HOME"] = pw_record.pw_dir
-            os.environ["LOGNAME"] = pw_record.pw_name
-            os.environ["USER"] = pw_record.pw_name
+        # update environment variables so that subprocesses dont try to write to /root
+        os.environ["HOME"] = pw_record.pw_dir
+        os.environ["LOGNAME"] = pw_record.pw_name
+        os.environ["USER"] = pw_record.pw_name
+        os.environ["XDG_CACHE_HOME"] = str(Path(pw_record.pw_dir) / ".cache")
+        os.environ["XDG_CONFIG_HOME"] = str(Path(pw_record.pw_dir) / ".config")
+        os.environ["XDG_DATA_HOME"] = str(Path(pw_record.pw_dir) / ".local" / "share")
+        os.environ.pop("XDG_RUNTIME_DIR", None)
+        os.environ.pop("ABXBUS_MULTIPROCESS_SEMAPHORE_DIR", None)
+
+        semaphore_dir = Path(pw_record.pw_dir) / ".cache" / "abxbus" / "semaphores"
+        os.environ["ABXBUS_MULTIPROCESS_SEMAPHORE_DIR"] = str(semaphore_dir)
+
+        with suppress(ImportError):
+            from abxbus import retry as abxbus_retry
+
+            abxbus_retry.MULTIPROCESS_SEMAPHORE_DIR = semaphore_dir
 
     if ARCHIVEBOX_USER == 0 or not ARCHIVEBOX_USER_EXISTS:
         print(

archivebox/config/version.py

@@ -36,13 +36,12 @@ def detect_installed_version(PACKAGE_DIR: Path = PACKAGE_DIR):
         # building docs, pyproject.toml is not available
         pass
 
-    # raise Exception('Failed to detect installed archivebox version!')
     return "dev"
 
 
 @cache
 def get_COMMIT_HASH() -> str | None:
-    for env_var in ("ARCHIVEBOX_COMMIT_HASH", "COMMIT_HASH"):
+    for env_var in ("ARCHIVEBOX_COMMIT_HASH",):
         env_commit_hash = os.environ.get(env_var, "").strip()
         if re.fullmatch(r"[0-9a-fA-F]{40}", env_commit_hash):
             return env_commit_hash
@@ -56,6 +55,13 @@ def get_COMMIT_HASH() -> str | None:
         except Exception:
             pass
 
+    try:
+        packaged_commit_hash = (PACKAGE_DIR / "COMMIT_SHA").read_text().strip()
+        if re.fullmatch(r"[0-9a-fA-F]{40}", packaged_commit_hash):
+            return packaged_commit_hash
+    except Exception:
+        pass
+
     def _read_git_file(git_dir: Path, ref: str) -> str | None:
         try:
             return git_dir.joinpath(ref).read_text().strip()
@@ -77,6 +83,13 @@ def get_COMMIT_HASH() -> str | None:
         return None
 
     try:
+        try:
+            pyproject_text = (PACKAGE_DIR.parent / "pyproject.toml").read_text()
+        except FileNotFoundError:
+            pyproject_text = ""
+        if not re.search(r'^name = "archivebox"$', pyproject_text, re.MULTILINE):
+            return None
+
         git_dir = PACKAGE_DIR.parent / ".git"
         if git_dir.is_file():
             gitdir_line = git_dir.read_text().strip()
@@ -96,11 +109,6 @@ def get_COMMIT_HASH() -> str | None:
     except Exception:
         pass
 
-    try:
-        return list((PACKAGE_DIR.parent / ".git/refs/heads/").glob("*"))[0].read_text().strip()
-    except Exception:
-        pass
-
     return None
 
 
@@ -115,53 +123,4 @@ def get_BUILD_TIME() -> str:
     return datetime.fromtimestamp(src_last_modified_unix_timestamp).strftime("%Y-%m-%d %H:%M:%S %s")
 
 
-# def get_versions_available_on_github(config):
-#     """
-#     returns a dictionary containing the ArchiveBox GitHub release info for
-#     the recommended upgrade version and the currently installed version
-#     """
-
-#     # we only want to perform the (relatively expensive) check for new versions
-#     # when its most relevant, e.g. when the user runs a long-running command
-#     subcommand_run_by_user = sys.argv[3] if len(sys.argv) > 3 else 'help'
-#     long_running_commands = ('add', 'schedule', 'update', 'status', 'server')
-#     if subcommand_run_by_user not in long_running_commands:
-#         return None
-
-#     github_releases_api = "https://api.github.com/repos/ArchiveBox/ArchiveBox/releases"
-#     response = requests.get(github_releases_api)
-#     if response.status_code != 200:
-#         stderr(f'[!] Warning: GitHub API call to check for new ArchiveBox version failed! (status={response.status_code})', color='lightyellow', config=config)
-#         return None
-#     all_releases = response.json()
-
-#     installed_version = parse_version_string(config['VERSION'])
-
-#     # find current version or nearest older version (to link to)
-#     current_version = None
-#     for idx, release in enumerate(all_releases):
-#         release_version = parse_version_string(release['tag_name'])
-#         if release_version <= installed_version:
-#             current_version = release
-#             break
-
-#     current_version = current_version or all_releases[-1]
-
-#     # recommended version is whatever comes after current_version in the release list
-#     # (perhaps too conservative to only recommend upgrading one version at a time, but it's safest)
-#     try:
-#         recommended_version = all_releases[idx+1]
-#     except IndexError:
-#         recommended_version = None
-
-#     return {'recommended_version': recommended_version, 'current_version': current_version}
-
-# def can_upgrade(config):
-#     if config['VERSIONS_AVAILABLE'] and config['VERSIONS_AVAILABLE']['recommended_version']:
-#         recommended_version = parse_version_string(config['VERSIONS_AVAILABLE']['recommended_version']['tag_name'])
-#         current_version = parse_version_string(config['VERSIONS_AVAILABLE']['current_version']['tag_name'])
-#         return recommended_version > current_version
-#     return False
-
-
 VERSION: str = detect_installed_version()

archivebox/config/views.py

@@ -1,25 +1,32 @@
 __package__ = "archivebox.config"
 
-import os
 import inspect
+import os
 from pathlib import Path
 from typing import Any
 from urllib.parse import quote, urlencode
+
+from admin_data_views.typing import ItemContext, SectionData, TableContext
+from admin_data_views.utils import ItemLink, render_with_item_view, render_with_table_view
 from django.http import HttpRequest
 from django.utils import timezone
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
 
-from admin_data_views.typing import TableContext, ItemContext, SectionData
-from admin_data_views.utils import render_with_table_view, render_with_item_view, ItemLink
-
 from archivebox.config import CONSTANTS
-from archivebox.misc.util import parse_date
-
 from archivebox.machine.models import Binary
+from archivebox.misc.util import parse_date
 
 ENVIRONMENT_BINARIES_BASE_URL = "/admin/environment/binaries/"
 INSTALLED_BINARIES_BASE_URL = "/admin/machine/binary/"
+LOG_DETAIL_TAIL_BYTES = 100_000
+
+
+def _read_text_tail(path: Path, max_bytes: int = LOG_DETAIL_TAIL_BYTES) -> str:
+    with path.open("rb") as log_file:
+        size = path.stat().st_size
+        log_file.seek(max(size - max_bytes, 0))
+        return log_file.read(max_bytes).decode("utf-8", errors="replace")
 
 
 def is_superuser(request: HttpRequest) -> bool:
@@ -87,7 +94,7 @@ def obj_to_yaml(obj: Any, indent: int = 0) -> str:
             return f" {obj}"
 
     elif isinstance(obj, (int, float, bool)):
-        return f" {str(obj)}"
+        return f" {obj!s}"
 
     elif callable(obj):
         source = (
@@ -98,7 +105,7 @@ def obj_to_yaml(obj: Any, indent: int = 0) -> str:
         return f" {indent_str}  " + source.replace("\n", f"\n{indent_str}  ")
 
     else:
-        return f" {str(obj)}"
+        return f" {obj!s}"
 
 
 def _binary_sort_key(binary: Binary) -> tuple[int, int, int, Any]:
@@ -300,7 +307,7 @@ def worker_list_view(request: HttpRequest, **kwargs) -> TableContext:
                 pid_to_process_id[row.pid] = str(row.id)
                 pid_to_process_type[row.pid] = row.process_type
 
-        except Exception:
+        except (ImportError, RuntimeError, TypeError, ValueError):
             pass
 
     def _pid_cell(pid_value: int | None, uptime_str: str = ""):
@@ -341,7 +348,7 @@ def worker_list_view(request: HttpRequest, **kwargs) -> TableContext:
         hours, remainder = divmod(seconds, 3600)
         minutes, secs = divmod(remainder, 60)
         supervisor_uptime = f"{hours}:{minutes:02d}:{secs:02d}"
-    except Exception:
+    except (ImportError, RuntimeError, TypeError, ValueError):
         try:
             from archivebox.machine.models import Machine, Process
 
@@ -360,7 +367,7 @@ def worker_list_view(request: HttpRequest, **kwargs) -> TableContext:
                 hours, remainder = divmod(seconds, 3600)
                 minutes, secs = divmod(remainder, 60)
                 supervisor_uptime = f"{hours}:{minutes:02d}:{secs:02d}"
-        except Exception:
+        except (RuntimeError, TypeError, ValueError):
             pass
 
     rows["PID"].append(_pid_cell(supervisor_pid if isinstance(supervisor_pid, int) else None, supervisor_uptime))
@@ -415,7 +422,7 @@ def worker_list_view(request: HttpRequest, **kwargs) -> TableContext:
 def worker_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
     assert is_superuser(request), "Must be a superuser to view configuration settings."
 
-    from archivebox.workers.supervisord_util import get_existing_supervisord_process, get_worker, get_sock_file, CONFIG_FILE_NAME
+    from archivebox.workers.supervisord_util import CONFIG_FILE_NAME, get_existing_supervisord_process, get_sock_file, get_worker
 
     SOCK_FILE = get_sock_file()
     CONFIG_FILE = SOCK_FILE.parent / CONFIG_FILE_NAME
@@ -438,8 +445,12 @@ def worker_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
 
     if key == "supervisord":
         relevant_config = CONFIG_FILE.read_text()
-        relevant_logs = str(supervisor.readLog(0, 10_000_000))
-        start_ts = [line for line in relevant_logs.split("\n") if "RPC interface 'supervisor' initialized" in line][-1].split(",", 1)[0]
+        supervisor_log = CONSTANTS.LOGS_DIR / "supervisord.log"
+        with supervisor_log.open("rb") as log_file:
+            startup_logs = log_file.read(LOG_DETAIL_TAIL_BYTES).decode("utf-8", errors="replace")
+        relevant_logs = _read_text_tail(supervisor_log)
+        startup_lines = [line for line in startup_logs.split("\n") if "RPC interface 'supervisor' initialized" in line]
+        start_ts = startup_lines[-1].split(",", 1)[0] if startup_lines else ""
         start_dt = parse_date(start_ts)
         uptime = str(timezone.now() - start_dt).split(".")[0] if start_dt else ""
         supervisor_state = supervisor.getState()
@@ -458,7 +469,7 @@ def worker_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
         worker_data = get_worker(supervisor, key)
         proc = worker_data if isinstance(worker_data, dict) else {}
         relevant_config = next((config for config in all_config if config.get("name") == key), {})
-        log_result = supervisor.tailProcessStdoutLog(key, 0, 10_000_000)
+        log_result = supervisor.tailProcessStdoutLog(key, -LOG_DETAIL_TAIL_BYTES, LOG_DETAIL_TAIL_BYTES)
         relevant_logs = str(log_result[0] if isinstance(log_result, tuple) else log_result)
 
     section: SectionData = {
@@ -474,7 +485,7 @@ def worker_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
             "Logfile": str(proc.get("stdout_logfile") or ""),
             "Uptime": str(str(proc.get("description") or "").split("uptime ", 1)[-1]),
             "Config": obj_to_yaml(relevant_config) if isinstance(relevant_config, dict) else str(relevant_config),
-            "Logs": relevant_logs,
+            "Recent Logs (last 100 KB)": relevant_logs,
         },
         "help_texts": {"Uptime": "How long the process has been running ([days:]hours:minutes:seconds)"},
     }
@@ -528,9 +539,9 @@ def log_list_view(request: HttpRequest, **kwargs) -> TableContext:
 def log_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
     assert is_superuser(request), "Must be a superuser to view configuration settings."
 
-    log_file = [logfile for logfile in CONSTANTS.LOGS_DIR.glob("*.log") if key in logfile.name][0]
+    log_file = next(logfile for logfile in CONSTANTS.LOGS_DIR.glob("*.log") if key in logfile.name)
 
-    log_text = log_file.read_text()
+    log_text = _read_text_tail(log_file)
     log_stat = log_file.stat()
 
     section: SectionData = {
@@ -540,8 +551,8 @@ def log_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
             "Path": str(log_file),
             "Size": f"{log_stat.st_size // 1000} kb",
             "Last Updated": format_parsed_datetime(log_stat.st_mtime),
-            "Tail": "\n".join(log_text[-10_000:].split("\n")[-20:]),
-            "Full Log": log_text,
+            "Tail": "\n".join(log_text.split("\n")[-20:]),
+            "Recent Log (last 100 KB)": log_text,
         },
     }
 

archivebox/core/admin_archiveresults.py

@@ -4,31 +4,29 @@ import html
 import json
 import os
 import shlex
-from pathlib import Path
-from urllib.parse import quote
 from functools import reduce
 from operator import and_
+from pathlib import Path
+from urllib.parse import quote
 
 from django.contrib import admin
+from django.core.exceptions import ValidationError
 from django.db.models import Min, Prefetch, Q, TextField
 from django.db.models.functions import Cast
+from django.urls import resolve, reverse
+from django.utils import timezone
 from django.utils.html import format_html
 from django.utils.safestring import mark_safe
-from django.core.exceptions import ValidationError
-from django.urls import reverse, resolve
-from django.utils import timezone
 from django.utils.text import smart_split
 
-from archivebox.misc.paginators import AcceleratedPaginator
 from archivebox.base_models.admin import BaseModelAdmin
-from archivebox.plugins.discovery import get_plugin_icon
-from archivebox.plugins.views import LIVE_PLUGIN_BASE_URL
+from archivebox.core.models import ArchiveResult, Snapshot
 from archivebox.core.routes_util import build_snapshot_url
 from archivebox.core.widgets import InlineTagEditorWidget
 from archivebox.machine.env_util import env_to_shell_exports
-
-
-from archivebox.core.models import ArchiveResult, Snapshot
+from archivebox.misc.paginators import AcceleratedPaginator
+from archivebox.plugins.discovery import get_plugin_icon
+from archivebox.plugins.views import LIVE_PLUGIN_BASE_URL
 
 
 def _get_replay_source_url(result: ArchiveResult) -> str:
@@ -120,15 +118,13 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
         status = result.status or "queued"
         color, bg = status_colors.get(status, ("#6b7280", "#f3f4f6"))
         output_files = result.output_files or {}
-        if isinstance(output_files, dict):
-            output_file_count = len(output_files)
-        elif isinstance(output_files, (list, tuple, set)):
+        if isinstance(output_files, (dict, list, tuple, set)):
             output_file_count = len(output_files)
         elif isinstance(output_files, str):
             try:
                 parsed = json.loads(output_files)
                 output_file_count = len(parsed) if isinstance(parsed, (dict, list, tuple, set)) else 0
-            except Exception:
+            except (TypeError, ValueError):
                 output_file_count = 0
         else:
             output_file_count = 0
@@ -136,8 +132,14 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
         # Get plugin icon
         icon = get_plugin_icon(result.plugin)
 
-        # Format timestamp
-        end_time = result.end_ts.strftime("%Y-%m-%d %H:%M:%S") if result.end_ts else "-"
+        # Keep each timestamp component intact while allowing one wrap between them.
+        if result.end_ts:
+            end_time = (
+                f'<span style="white-space: nowrap;">{result.end_ts:%Y-%m-%d}</span>'
+                f'<wbr> <span style="white-space: nowrap;">{result.end_ts:%H:%M:%S}</span>'
+            )
+        else:
+            end_time = "-"
 
         process = result.process_record
         process_display = "-"
@@ -208,7 +210,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                 <td style="padding: 10px 12px; white-space: nowrap; font-size: 20px;" title="{plugin_text}">
                     {icon}
                 </td>
-                <td style="padding: 10px 12px; font-weight: 500; color: #334155;">
+                <td class="archive-results-plugin" style="padding: 10px 12px; font-weight: 500; color: #334155;">
                         <a href="{output_link_attr}" target="_blank"
                            style="color: #334155; text-decoration: none;"
                        title="View output fullscreen"
@@ -217,17 +219,18 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                         {plugin_text}
                     </a>
                 </td>
-                <td style="padding: 10px 12px; max-width: 280px;">
+                <td class="archive-results-output" style="padding: 10px 12px; max-width: 280px;">
                     <span onclick="document.getElementById('{row_id}').open = !document.getElementById('{row_id}').open"
-                          style="color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 12px; cursor: pointer;"
+                          style="display: block; max-width: 280px; overflow: hidden; text-overflow: ellipsis; white-space: nowrap;
+                                 color: #2563eb; text-decoration: none; font-family: ui-monospace, monospace; font-size: 12px; cursor: pointer;"
                           title="Click to expand full output">
                         {output_display}
                     </span>
                 </td>
-                <td style="padding: 10px 12px; white-space: nowrap; color: #64748b; font-size: 12px; text-align: right;">
+                <td class="archive-results-files" style="padding: 10px 12px; color: #64748b; font-size: 12px; text-align: right;">
                     {output_file_count}
                 </td>
-                <td style="padding: 10px 12px; white-space: nowrap; color: #64748b; font-size: 12px;">
+                <td class="archive-results-completed" style="padding: 10px 12px; color: #64748b; font-size: 12px;">
                     {end_time}
                 </td>
                 <td style="padding: 10px 12px; white-space: nowrap;">
@@ -258,7 +261,7 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
                         </summary>
                         <div style="margin-top: 8px; padding: 10px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 6px; max-height: 200px; overflow: auto;">
                             <div style="font-size: 11px; color: #64748b; margin-bottom: 8px;">
-                                <span style="margin-right: 16px;"><b>ID:</b> <code>{str(result.id)}</code></span>
+                                <span style="margin-right: 16px;"><b>ID:</b> <code>{result.id!s}</code></span>
                                 <span style="margin-right: 16px;"><b>Version:</b> <code>{version}</code></span>
                                 <span style="margin-right: 16px;"><b>PWD:</b> <code>{pwd_text}</code></span>
                             </div>
@@ -298,17 +301,17 @@ def render_archiveresults_list(archiveresults_qs, limit=50, config=None):
         """
 
     return mark_safe(f"""
-        <div style="border: 1px solid #e2e8f0; border-radius: 8px; overflow: hidden; background: #fff; width: 100%;">
-            <table style="width: 100%; border-collapse: collapse; font-size: 14px;">
+        <div style="border: 1px solid #e2e8f0; border-radius: 8px; overflow-x: auto; overflow-y: hidden; background: #fff; width: 100%;">
+            <table class="archive-results-table" style="width: 100%; min-width: 1100px; border-collapse: collapse; font-size: 14px;">
                 <thead>
                     <tr style="background: #f8fafc; border-bottom: 2px solid #e2e8f0;">
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Details</th>
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Status</th>
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; width: 32px;"></th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Plugin</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Output</th>
-                        <th style="padding: 10px 12px; text-align: right; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Files</th>
-                        <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Completed</th>
+                        <th class="archive-results-plugin" style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Plugin</th>
+                        <th class="archive-results-output" style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Output</th>
+                        <th class="archive-results-files" style="padding: 10px 12px; text-align: right; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Files</th>
+                        <th class="archive-results-completed" style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Completed</th>
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Process</th>
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Machine</th>
                         <th style="padding: 10px 12px; text-align: left; font-weight: 600; color: #475569; font-size: 12px; text-transform: uppercase; letter-spacing: 0.05em;">Version</th>
@@ -328,15 +331,12 @@ class ArchiveResultInline(admin.TabularInline):
     name = "Archive Results Log"
     model = ArchiveResult
     parent_model = Snapshot
-    # fk_name = 'snapshot'
     extra = 0
     sort_fields = ("end_ts", "plugin", "output_str", "status", "cmd_version")
     readonly_fields = ("id", "result_id", "completed", "command", "version")
     fields = ("start_ts", "end_ts", *readonly_fields, "plugin", "cmd", "cmd_version", "pwd", "status", "output_str")
-    # exclude = ('id',)
     ordering = ("end_ts",)
     show_change_link = True
-    # # classes = ['collapse']
 
     def get_parent_object_from_request(self, request):
         resolved = resolve(request.path_info)
@@ -371,7 +371,6 @@ class ArchiveResultInline(admin.TabularInline):
         base_fields = formset.form.base_fields
         snapshot_output_dir = str(snapshot.output_dir) if snapshot else ""
 
-        # import ipdb; ipdb.set_trace()
         # formset.form.base_fields['id'].widget = formset.form.base_fields['id'].hidden_widget()
 
         # default values for new entries
@@ -417,11 +416,11 @@ class ArchiveResultAdmin(BaseModelAdmin):
     list_display_links = None
     sort_fields = ("id", "created_at", "plugin", "status")
     readonly_fields = (
-        "admin_actions",
         "cmd",
         "cmd_version",
         "pwd",
         "cmd_str",
+        "admin_actions",
         "snapshot_info",
         "tags_str",
         "created_at",
@@ -441,20 +440,13 @@ class ArchiveResultAdmin(BaseModelAdmin):
         "output_json",
         "process__cmd",
     )
-    autocomplete_fields = ["snapshot"]
+    autocomplete_fields = ("snapshot",)
 
     fieldsets = (
-        (
-            "Actions",
-            {
-                "fields": ("admin_actions",),
-                "classes": ("card", "wide"),
-            },
-        ),
         (
             "Snapshot",
             {
-                "fields": ("snapshot", "snapshot_info", "tags_str"),
+                "fields": ("snapshot", "snapshot_info", "tags_str", "admin_actions"),
                 "classes": ("card", "wide"),
             },
         ),
@@ -489,14 +481,14 @@ class ArchiveResultAdmin(BaseModelAdmin):
     )
 
     list_filter = ("status", "plugin", "start_ts")
-    ordering = ["-start_ts"]
+    ordering = ("-start_ts",)
     list_per_page = 50
 
     paginator = AcceleratedPaginator
     save_on_top = True
     show_full_result_count = False
 
-    actions = ["delete_selected"]
+    actions = ("delete_selected",)
 
     class Meta:
         verbose_name = "Archive Result"
@@ -506,6 +498,36 @@ class ArchiveResultAdmin(BaseModelAdmin):
         self.request = request
         return super().change_view(request, object_id, form_url, extra_context)
 
+    def get_admin_toolbar_actions(self, request, obj):
+        if obj is None:
+            return []
+        self.request = request
+        return [
+            {
+                "label": "View Output",
+                "icon": "📄",
+                "url": self.get_output_view_url(obj),
+                "title": "Open the archived output for this result",
+            },
+            {
+                "label": "Output files",
+                "icon": "📁",
+                "url": self.get_output_files_url(obj),
+                "title": "Browse the output files for this result",
+            },
+            {
+                "label": "Download Zip",
+                "icon": "⬇",
+                "url": self.get_output_zip_url(obj),
+                "kind": "accent",
+                "css_classes": "archivebox-zip-button",
+                "onclick": "return window.archiveboxHandleZipClick(this, event);",
+                "extra_attrs": [("data-loading-label", "Preparing...")],
+                "title": "Download all output files as a zip",
+            },
+            {"label": "Snapshot", "icon": "🗂", "url": self.get_snapshot_view_url(obj), "title": "Open the parent snapshot view"},
+        ]
+
     def changelist_view(self, request, extra_context=None):
         self.request = request
         saved_list_per_page = self.list_per_page
@@ -616,6 +638,38 @@ class ArchiveResultAdmin(BaseModelAdmin):
             self.get_output_zip_url(result),
         )
 
+    @admin.display(description="")
+    def admin_actions(self, result):
+        return format_html(
+            """
+            <div style="display: flex; flex-wrap: wrap; gap: 12px; align-items: center;">
+                <a class="btn" style="display: inline-flex; align-items: center; gap: 6px; padding: 10px 16px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 8px; color: #334155; text-decoration: none; font-size: 14px; font-weight: 500;"
+                   href="{}">
+                    📄 View Output
+                </a>
+                <a class="btn" style="display: inline-flex; align-items: center; gap: 6px; padding: 10px 16px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 8px; color: #334155; text-decoration: none; font-size: 14px; font-weight: 500;"
+                   href="{}">
+                    📁 Output files
+                </a>
+                <a class="btn archivebox-zip-button" style="display: inline-flex; align-items: center; gap: 6px; padding: 10px 16px; background: #eff6ff; border: 1px solid #bfdbfe; border-radius: 8px; color: #1d4ed8; text-decoration: none; font-size: 14px; font-weight: 500;"
+                   href="{}"
+                   data-loading-label="Preparing..."
+                   onclick="return window.archiveboxHandleZipClick(this, event);">
+                    <span class="archivebox-zip-spinner" aria-hidden="true"></span>
+                    <span class="archivebox-zip-label">⬇ Download Zip</span>
+                </a>
+                <a class="btn" style="display: inline-flex; align-items: center; gap: 6px; padding: 10px 16px; background: #f8fafc; border: 1px solid #e2e8f0; border-radius: 8px; color: #334155; text-decoration: none; font-size: 14px; font-weight: 500;"
+                   href="{}">
+                    🗂 Snapshot
+                </a>
+            </div>
+            """,
+            self.get_output_view_url(result),
+            self.get_output_files_url(result),
+            self.get_output_zip_url(result),
+            self.get_snapshot_view_url(result),
+        )
+
     @admin.display(
         description="Snapshot",
         ordering="snapshot__url",
@@ -753,46 +807,6 @@ class ArchiveResultAdmin(BaseModelAdmin):
             output_text,
         )
 
-    @admin.display(description="")
-    def admin_actions(self, result):
-        return format_html(
-            """
-            <div style="display:flex; flex-wrap:wrap; gap:12px; align-items:center;">
-                <a class="btn" style="display:inline-flex; align-items:center; gap:6px; padding:10px 16px; background:#f8fafc; border:1px solid #e2e8f0; border-radius:8px; color:#334155; text-decoration:none; font-size:14px; font-weight:500; transition:all 0.15s;"
-                   href="{}"
-                   onmouseover="this.style.background='#f1f5f9'; this.style.borderColor='#cbd5e1';"
-                   onmouseout="this.style.background='#f8fafc'; this.style.borderColor='#e2e8f0';">
-                    📄 View Output
-                </a>
-                <a class="btn" style="display:inline-flex; align-items:center; gap:6px; padding:10px 16px; background:#f8fafc; border:1px solid #e2e8f0; border-radius:8px; color:#334155; text-decoration:none; font-size:14px; font-weight:500; transition:all 0.15s;"
-                   href="{}"
-                   onmouseover="this.style.background='#f1f5f9'; this.style.borderColor='#cbd5e1';"
-                   onmouseout="this.style.background='#f8fafc'; this.style.borderColor='#e2e8f0';">
-                    📁 Output files
-                </a>
-                <a class="btn archivebox-zip-button" style="display:inline-flex; align-items:center; gap:6px; padding:10px 16px; background:#eff6ff; border:1px solid #bfdbfe; border-radius:8px; color:#1d4ed8; text-decoration:none; font-size:14px; font-weight:500; transition:all 0.15s;"
-                   href="{}"
-                   data-loading-label="Preparing..."
-                   onclick="return window.archiveboxHandleZipClick(this, event);"
-                   onmouseover="this.style.background='#dbeafe'; this.style.borderColor='#93c5fd';"
-                   onmouseout="this.style.background='#eff6ff'; this.style.borderColor='#bfdbfe';">
-                    <span class="archivebox-zip-spinner" aria-hidden="true"></span>
-                    <span class="archivebox-zip-label">⬇ Download Zip</span>
-                </a>
-                <a class="btn" style="display:inline-flex; align-items:center; gap:6px; padding:10px 16px; background:#f8fafc; border:1px solid #e2e8f0; border-radius:8px; color:#334155; text-decoration:none; font-size:14px; font-weight:500; transition:all 0.15s;"
-                   href="{}"
-                   onmouseover="this.style.background='#f1f5f9'; this.style.borderColor='#cbd5e1';"
-                   onmouseout="this.style.background='#f8fafc'; this.style.borderColor='#e2e8f0';">
-                    🗂 Snapshot
-                </a>
-            </div>
-            """,
-            self.get_output_view_url(result),
-            self.get_output_files_url(result),
-            self.get_output_zip_url(result),
-            self.get_snapshot_view_url(result),
-        )
-
     def output_summary(self, result):
         snapshot_dir = Path(result.snapshot.output_dir)
         output_html = format_html(
@@ -817,8 +831,6 @@ class ArchiveResultAdmin(BaseModelAdmin):
         else:
             root_dir = str(snapshot_dir)
 
-        # print(root_dir, str(list(os.walk(root_dir))))
-
         for root, dirs, files in os.walk(root_dir):
             depth = root.replace(root_dir, "").count(os.sep) + 1
             if depth > 2:

archivebox/core/admin_site.py

@@ -2,6 +2,15 @@ __package__ = "archivebox.core"
 
 from typing import TYPE_CHECKING, Any
 
+from admin_data_views.admin import (
+    admin_data_index_view as adv_admin_data_index_view,
+)
+from admin_data_views.admin import (
+    get_admin_data_urls as adv_get_admin_data_urls,
+)
+from admin_data_views.admin import (
+    get_app_list as adv_get_app_list,
+)
 from django.contrib import admin
 from django.contrib.auth import REDIRECT_FIELD_NAME
 from django.contrib.auth.decorators import login_not_required
@@ -12,23 +21,17 @@ from django.urls import reverse
 from django.utils.decorators import method_decorator
 from django.utils.translation import gettext as _
 from django.views.decorators.cache import never_cache
-from admin_data_views.admin import (
-    admin_data_index_view as adv_admin_data_index_view,
-    get_admin_data_urls as adv_get_admin_data_urls,
-    get_app_list as adv_get_app_list,
-)
 
 from archivebox.config import VERSION
-from archivebox.config.version import get_COMMIT_HASH
+from archivebox.core.context_processors import get_static_cache_key
 from archivebox.core.routes_util import is_allowed_archivebox_redirect_url
 
 if TYPE_CHECKING:
+    from admin_data_views.typing import AppDict
     from django.http import HttpRequest
     from django.template.response import TemplateResponse
     from django.urls import URLPattern, URLResolver
 
-    from admin_data_views.typing import AppDict
-
 
 class ArchiveBoxLoginView(LoginView):
     def get_redirect_url(self) -> str:
@@ -50,7 +53,7 @@ class ArchiveBoxAdmin(admin.AdminSite):
     def each_context(self, request: "HttpRequest") -> dict[str, Any]:
         context = super().each_context(request)
         context["VERSION"] = VERSION
-        context["STATIC_CACHE_KEY"] = (get_COMMIT_HASH() or VERSION or "dev").strip()
+        context["STATIC_CACHE_KEY"] = get_static_cache_key()
         return context
 
     @staticmethod
@@ -122,8 +125,6 @@ class ArchiveBoxAdmin(admin.AdminSite):
 
     def index(self, request: "HttpRequest", extra_context: dict[str, Any] | None = None) -> "TemplateResponse":
         response = super().index(request, extra_context)
-        if connection.vendor != "sqlite":
-            return response
 
         models_by_table: dict[str, list[dict[str, Any]]] = {}
         for app in response.context_data.get("app_list", []):
@@ -136,23 +137,18 @@ class ArchiveBoxAdmin(admin.AdminSite):
         if not models_by_table:
             return response
 
-        try:
-            with connection.cursor() as cursor:
-                cursor.execute("SELECT tbl, stat FROM sqlite_stat1")
-                for table, stat in cursor.fetchall():
-                    try:
-                        count = int(str(stat).split()[0])
-                    except (IndexError, TypeError, ValueError):
-                        continue
-                    self._set_model_object_count(
-                        models_by_table,
-                        table,
-                        count,
-                        title=f"Approximate count from SQLite stats: {count:,}",
-                    )
-                    models_by_table.pop(table, None)
-        except DatabaseError:
-            pass
+        from archivebox.misc.db import approximate_row_counts
+
+        for table, count in approximate_row_counts(connection).items():
+            if table not in models_by_table:
+                continue
+            self._set_model_object_count(
+                models_by_table,
+                table,
+                count,
+                title=f"Approximate count from database stats: {count:,}",
+            )
+            models_by_table.pop(table, None)
 
         for table in list(models_by_table):
             try:
@@ -190,9 +186,9 @@ def register_admin_site():
 
     # Register admin views for each app
     # (Previously handled by ABX plugin system, now called directly)
+    from archivebox.api.admin import register_admin as register_api_admin
     from archivebox.core.admin import register_admin as register_core_admin
     from archivebox.crawls.admin import register_admin as register_crawls_admin
-    from archivebox.api.admin import register_admin as register_api_admin
     from archivebox.machine.admin import register_admin as register_machine_admin
     from archivebox.personas.admin import register_admin as register_personas_admin
     from archivebox.workers.admin import register_admin as register_workers_admin

archivebox/core/admin_snapshots.py

@@ -43,7 +43,6 @@ from archivebox.core.permissions import (
 from archivebox.core.widgets import TagEditorWidget, InlineTagEditorWidget
 
 
-# GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': [], 'CAN_UPGRADE': False}
 GLOBAL_CONTEXT = {}
 
 SNAPSHOT_PERMISSION_META = PERMISSIONS_META
@@ -72,13 +71,6 @@ class SnapshotActionForm(ActionForm):
 
         return [name.strip() for name in tags_str.split(",") if name.strip()]
 
-    # TODO: allow selecting actions for specific extractor plugins? is this useful?
-    # plugin = forms.ChoiceField(
-    #     choices=ArchiveResult.PLUGIN_CHOICES,
-    #     required=False,
-    #     widget=forms.MultileChoiceField(attrs={'class': "form-control"})
-    # )
-
 
 class TagNameListFilter(admin.SimpleListFilter):
     title = "By tag name"
@@ -260,13 +252,15 @@ class SnapshotChangeList(SearchResultsChangeList):
         rows = (
             ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
             .order_by("snapshot_id", "plugin")
-            .values_list("snapshot_id", "plugin", "status", "output_size")
+            .values_list("snapshot_id", "plugin", "status", "output_size", "output_files")
         )
-        for snapshot_id, plugin, status, output_size in rows.iterator(chunk_size=1000):
+        for snapshot_id, plugin, status, output_size, output_files in rows.iterator(chunk_size=1000):
             if plugin in seen_plugins[snapshot_id]:
                 continue
             seen_plugins[snapshot_id].add(plugin)
-            results_by_snapshot[snapshot_id].append(SimpleNamespace(plugin=plugin, status=status, output_size=output_size))
+            results_by_snapshot[snapshot_id].append(
+                SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
+            )
 
         for obj in self.result_list:
             obj.__dict__["_admin_archiveresults"] = results_by_snapshot[obj.pk]
@@ -616,12 +610,12 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             if retried:
                 messages.success(
                     request,
-                    f"Queued {retried} failed/skipped extractors for retry on this snapshot.",
+                    f"Queued {retried} failed extractors for retry on this snapshot.",
                 )
             else:
                 messages.info(
                     request,
-                    "No failed/skipped extractors were found on this snapshot.",
+                    "No failed extractors were found on this snapshot.",
                 )
 
         return redirect(snapshot.admin_change_url)
@@ -746,31 +740,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         html = Template("""{{bookmarked_date}} (<code>{{timestamp}}</code>)""")
         return mark_safe(html.render(context))
 
-        # pretty_time = obj.bookmarked.strftime('%Y-%m-%d %H:%M:%S')
-        # return f'{pretty_time} ({obj.timestamp})'
-
-    # TODO: figure out a different way to do this, you cant nest forms so this doenst work
-    # def action(self, obj):
-    #     # csrfmiddlewaretoken: Wa8UcQ4fD3FJibzxqHN3IYrrjLo4VguWynmbzzcPYoebfVUnDovon7GEMYFRgsh0
-    #     # action: update_snapshots
-    #     # select_across: 0
-    #     # _selected_action: 76d29b26-2a88-439e-877c-a7cca1b72bb3
-    #     return format_html(
-    #         '''
-    #             <form action="/admin/core/snapshot/" method="post" onsubmit="e => e.stopPropagation()">
-    #                 <input type="hidden" name="csrfmiddlewaretoken" value="{}">
-    #                 <input type="hidden" name="_selected_action" value="{}">
-    #                 <button name="update_snapshots">Check</button>
-    #                 <button name="update_titles">Pull title + favicon</button>
-    #                 <button name="update_snapshots">Update</button>
-    #                 <button name="overwrite_snapshots">Re-Archive (overwrite)</button>
-    #                 <button name="delete_snapshots">Permanently delete</button>
-    #             </form>
-    #         ''',
-    #         csrf.get_token(self.request),
-    #         obj.pk,
-    #     )
-
     @admin.display(description="")
     def admin_actions(self, obj):
         summary_url = self.get_snapshot_view_url(obj)
@@ -1147,9 +1116,7 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
             mark_safe("".join(output)),
         )
 
-    @admin.display(
-        # ordering='archiveresult_count'
-    )
+    @admin.display()
     def size(self, obj):
         request = self.request
         config = request.archivebox_config
@@ -1399,11 +1366,6 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         extra_context["snapshot_is_grid_view"] = True
         return self.changelist_view(request, extra_context=extra_context)
 
-    # for debugging, uncomment this to print all requests:
-    # def changelist_view(self, request, extra_context=None):
-    #     print('[*] Got request', request.method, request.POST)
-    #     return super().changelist_view(request, extra_context=None)
-
     @admin.action(
         description="🔁 Redo Failed",
     )
@@ -1415,10 +1377,10 @@ class SnapshotAdmin(SearchResultsAdminMixin, ConfigEditorMixin, BaseModelAdmin):
         if queued:
             messages.success(
                 request,
-                f"Queued {queued} failed/skipped extractors for retry. The background runner will process them.",
+                f"Queued {queued} failed extractors for retry. The background runner will process them.",
             )
         else:
-            messages.info(request, "No failed/skipped extractors were found in the selected snapshots.")
+            messages.info(request, "No failed extractors were found in the selected snapshots.")
 
     @admin.action(
         description="🆕 Archive Now",

archivebox/core/apps.py

@@ -20,6 +20,14 @@ class CoreConfig(AppConfig):
 
         ModelWithOutputDir.register_delete_signal()
 
+        # SQLite ignores VARCHAR(n) limits but PostgreSQL enforces them; clamp
+        # CharField values on save so writes behave the same on both backends.
+        from django.db.models.signals import pre_save
+
+        from archivebox.misc.db import truncate_overlong_charfields
+
+        pre_save.connect(truncate_overlong_charfields, dispatch_uid="archivebox_truncate_overlong_charfields")
+
         # Import models to register state machines with the registry
         # Skip during makemigrations to avoid premature state machine access
         if "makemigrations" not in sys.argv:

archivebox/core/asgi.py

@@ -7,10 +7,11 @@ For more information on this file, see
 https://docs.djangoproject.com/en/stable/howto/deployment/asgi/
 """
 
-from archivebox.config.django import setup_django
 from django.core.asgi import get_asgi_application
 
-setup_django(in_memory_db=False, check_db=True)
+from archivebox.config.django import setup_django
+
+setup_django(check_db=True)
 
 
 def _patch_thread_sensitive_context_shutdown() -> None:
@@ -73,16 +74,3 @@ _patch_thread_sensitive_context_shutdown()
 
 # Standard Django ASGI application (no websockets/channels needed)
 application = get_asgi_application()
-
-# If websocket support is needed later, install channels and use:
-# from channels.routing import ProtocolTypeRouter, URLRouter
-# from channels.auth import AuthMiddlewareStack
-# from channels.security.websocket import AllowedHostsOriginValidator
-# from archivebox.core.routing import websocket_urlpatterns
-#
-# application = ProtocolTypeRouter({
-#     "http": get_asgi_application(),
-#     "websocket": AllowedHostsOriginValidator(
-#         AuthMiddlewareStack(URLRouter(websocket_urlpatterns))
-#     ),
-# })

archivebox/core/context_processors.py

@@ -1,9 +1,21 @@
+from pathlib import Path
+
 from archivebox.config import VERSION
 from archivebox.config.version import get_COMMIT_HASH
 
 
+def get_static_cache_key() -> str:
+    """Version the admin stylesheet even when the checkout has uncommitted edits."""
+    base_key = (get_COMMIT_HASH() or VERSION or "dev").strip()
+    admin_css_path = Path(__file__).resolve().parent.parent / "templates" / "static" / "admin.css"
+    try:
+        return f"{base_key}-{admin_css_path.stat().st_mtime_ns}"
+    except OSError:
+        return base_key
+
+
 def archivebox_globals(request):
     return {
         "VERSION": VERSION,
-        "STATIC_CACHE_KEY": (get_COMMIT_HASH() or VERSION or "dev").strip(),
+        "STATIC_CACHE_KEY": get_static_cache_key(),
     }

archivebox/core/forms.py

@@ -1,16 +1,17 @@
 __package__ = "archivebox.core"
 
 import re
-from decimal import Decimal, InvalidOperation, ROUND_CEILING
+from decimal import ROUND_CEILING, Decimal, InvalidOperation
 
 from django import forms
 
-from archivebox.misc.util import URL_REGEX, find_all_urls, parse_filesize_to_bytes
 from archivebox.base_models.admin import KeyValueWidget
-from archivebox.crawls.schedule_util import validate_schedule
 from archivebox.config.common import get_config, parse_delete_after
 from archivebox.core.permissions import PERMISSIONS_CHOICES, PERMISSIONS_PUBLIC, filter_personas_by_permissions
 from archivebox.core.widgets import TagEditorWidget, URLFiltersWidget
+from archivebox.crawls.schedule_util import validate_schedule
+from archivebox.misc.util import URL_REGEX, find_all_urls, parse_filesize_to_bytes
+from archivebox.personas.models import Persona
 from archivebox.plugins.discovery import get_plugins
 from archivebox.plugins.forms import (
     PLUGIN_GROUP_DEFINITIONS,
@@ -18,7 +19,6 @@ from archivebox.plugins.forms import (
     PluginConfigFormMixin,
     get_choice_field,
 )
-from archivebox.personas.models import Persona
 
 DEPTH_CHOICES = (
     ("0", "depth = 0 (archive just these URLs)"),
@@ -101,12 +101,11 @@ class AddLinkForm(PluginConfigFormMixin, forms.Form):
             attrs={
                 "data-url-regex": URL_REGEX.pattern,
                 "placeholder": (
-                    "\n",
-                    "Enter URL(s) to archive. Any format is ok: one per line, CSV, JSON, embedded in text, etc."
+                    "Enter URL(s) to archive. Any format is ok: one per line, CSV, JSON, RSS, embedded in text, etc.\n\n"
                     "Examples:\n\n"
                     "https://example.com\n\n"
                     "https://news.ycombinator.com,https://news.google.com\n\n"
-                    "Or any text-based content [containing URLs](https://github.com/ArchiveBox/ArchiveBox)...",
+                    "Or any text-based format [containing URLs](https://github.com/ArchiveBox/ArchiveBox)..."
                 ),
             },
         ),

archivebox/core/middleware.py

@@ -3,25 +3,27 @@ __package__ = "archivebox.core"
 import ipaddress
 import re
 from pathlib import Path
+
 from django.conf import settings
-from django.utils import timezone
 from django.contrib.auth.middleware import RemoteUserMiddleware
 from django.contrib.auth.models import AnonymousUser
-from django.core.exceptions import ImproperlyConfigured
-from django.shortcuts import redirect
 from django.contrib.staticfiles import finders
-from django.utils.http import http_date
+from django.core.exceptions import ImproperlyConfigured
 from django.http import HttpResponseForbidden, HttpResponseNotModified
+from django.shortcuts import redirect
+from django.utils import timezone
+from django.utils.http import http_date
 
-from archivebox.config.common import get_config
 from archivebox.config import VERSION
+from archivebox.config.common import get_config
 from archivebox.config.version import get_COMMIT_HASH
 from archivebox.core.routes_util import (
-    build_snapshot_url,
     build_admin_url,
+    build_original_url,
+    build_snapshot_url,
     build_web_url,
-    get_api_host,
     get_admin_host,
+    get_api_host,
     get_base_host,
     get_listen_host,
     get_listen_subdomain,
@@ -30,8 +32,7 @@ from archivebox.core.routes_util import (
     is_snapshot_subdomain,
     split_host_port,
 )
-from archivebox.core.views import SnapshotHostView, OriginalDomainHostView
-
+from archivebox.core.views import OriginalDomainHostView, SnapshotHostView
 
 ADMIN_LOGIN_HINT_COOKIE = "archivebox_admin_logged_in"
 
@@ -73,7 +74,6 @@ def detect_timezone(request, activate: bool = True):
         tz = timezone.get_fixed_timezone(int(gmt_offset))
         if activate:
             timezone.activate(tz)
-    # print('GMT_OFFSET', gmt_offset, tz)
     return tz
 
 
@@ -90,17 +90,22 @@ def AdminCookieIsolationMiddleware(get_response):
         response = get_response(request)
 
         config = request.__dict__.get("archivebox_config")
-        if config is None:
-            config = get_config(resolve_plugins=False)
-            request.archivebox_config = config
+        if config is None or config.SERVER_SECURITY_MODE == "auto":
+            from archivebox.config.common import get_request_config
+
+            config = get_request_config(request, resolve_plugins=False)
         if not config.USES_SUBDOMAIN_ROUTING:
             return response
 
-        request_host = (request.get_host() or "").lower()
-        if host_matches(request_host, get_admin_host(config=config)):
+        if not config.BASE_URL and request.path.startswith("/admin/"):
             return response
 
-        if host_matches(request_host, get_web_host(config=config)):
+        request_host = (request.get_host() or "").lower()
+        request_hostname, _request_port = split_host_port(request_host)
+        if host_matches(request_hostname, get_admin_host(config=config)):
+            return response
+
+        if host_matches(request_hostname, get_web_host(config=config)):
             for cookie_name in tuple(response.cookies.keys()):
                 if cookie_name != ADMIN_LOGIN_HINT_COOKIE:
                     response.cookies.pop(cookie_name, None)
@@ -145,15 +150,15 @@ def CacheControlMiddleware(get_response):
                     response.headers["Last-Modified"] = http_date(mtime)
                 return response
 
-        if "/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path):
-            if not response.get("Cache-Control"):
-                config = request.__dict__.get("archivebox_config")
-                if config is None:
-                    config = get_config(resolve_plugins=False)
-                    request.archivebox_config = config
-                policy = "private" if config.PERMISSIONS == "private" else "public"
-                response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
-                # print('Set Cache-Control header to', response['Cache-Control'])
+        if ("/archive/" in request.path or "/static/" in request.path or snapshot_path_re.match(request.path)) and not response.get(
+            "Cache-Control",
+        ):
+            config = request.__dict__.get("archivebox_config")
+            if config is None:
+                config = get_config(resolve_plugins=False)
+                request.archivebox_config = config
+            policy = "private" if config.PERMISSIONS == "private" else "public"
+            response["Cache-Control"] = f"{policy}, max-age=60, stale-while-revalidate=300"
         return response
 
     return middleware
@@ -165,9 +170,28 @@ def ServerSecurityModeMiddleware(get_response):
 
     def middleware(request):
         config = request.__dict__.get("archivebox_config")
-        if config is None:
-            config = get_config(resolve_plugins=False)
-            request.archivebox_config = config
+        if config is None or config.SERVER_SECURITY_MODE == "auto":
+            base_config = config or get_config(resolve_plugins=False)
+            if base_config.SERVER_SECURITY_MODE == "auto" and request.method.upper() not in allowed_methods:
+                request_host, _request_port = split_host_port((request.get_host() or "").lower())
+                base_host, _base_port = split_host_port(get_base_host(config=base_config))
+                admin_host, _admin_port = split_host_port(get_admin_host(config=base_config))
+                api_host, _api_port = split_host_port(get_api_host(config=base_config))
+                web_host, _web_port = split_host_port(get_web_host(config=base_config))
+                control_hosts = {host for host in (base_host, admin_host, api_host, web_host) if host}
+                first_run_setup_request = not base_config.BASE_URL and (
+                    request.path == "/admin/login/"
+                    or (
+                        bool(getattr(request.user, "is_superuser", False))
+                        and re.fullmatch(r"/admin/machine/machine/[0-9a-f-]{32,36}/change/", request.path)
+                    )
+                )
+                if control_hosts and request_host not in control_hosts and not first_run_setup_request:
+                    return HttpResponseForbidden("ArchiveBox is running with the control plane disabled on this host.")
+
+            from archivebox.config.common import get_request_config
+
+            config = get_request_config(request, resolve_plugins=False)
 
         if config.CONTROL_PLANE_ENABLED:
             return get_response(request)
@@ -191,6 +215,10 @@ def HostRoutingMiddleware(get_response):
     snapshot_path_re = re.compile(
         r"^/(?P<username>[^/]+)/(?P<date>\d{4}(?:\d{2})?(?:\d{2})?)/(?P<domain>[^/]+)/(?P<snapshot_id>[0-9a-fA-F-]{8,36})(?:/(?P<path>.*))?$",
     )
+    snapshot_replay_path_re = re.compile(
+        r"^/snapshot/(?P<snapshot_id>[0-9a-fA-F-]{8,36})(?:/(?P<path>.*))?$",
+    )
+    original_replay_path_re = re.compile(r"^/original/(?P<domain>[^/]+)(?:/(?P<path>.*))?$")
 
     def middleware(request):
         if request.path in {"/health", "/health/"}:
@@ -240,18 +268,40 @@ def HostRoutingMiddleware(get_response):
         if config.USES_SUBDOMAIN_ROUTING and not config.BASE_URL:
             return get_response(request)
 
+        if config.USES_SUBDOMAIN_ROUTING:
+            snapshot_replay_match = snapshot_replay_path_re.match(request.path)
+            if snapshot_replay_match:
+                target = build_snapshot_url(
+                    snapshot_replay_match.group("snapshot_id"),
+                    (snapshot_replay_match.group("path") or "").strip("/"),
+                    request=request,
+                )
+                if request.META.get("QUERY_STRING"):
+                    target = f"{target}?{request.META['QUERY_STRING']}"
+                return redirect(target)
+
+            original_replay_match = original_replay_path_re.match(request.path)
+            if original_replay_match:
+                target = build_original_url(
+                    original_replay_match.group("domain"),
+                    (original_replay_match.group("path") or "").strip("/"),
+                    request=request,
+                )
+                if request.META.get("QUERY_STRING"):
+                    target = f"{target}?{request.META['QUERY_STRING']}"
+                return redirect(target)
+
         if not config.USES_SUBDOMAIN_ROUTING:
             if host_matches(request_host, listen_host):
                 return get_response(request)
 
             req_host, req_port = split_host_port(request_host)
             listen_host_only, listen_port = split_host_port(listen_host)
-            if req_host.endswith(f".{listen_host_only}"):
-                if not listen_port or not req_port or listen_port == req_port:
-                    target = build_web_url(request.path, request=request)
-                    if request.META.get("QUERY_STRING"):
-                        target = f"{target}?{request.META['QUERY_STRING']}"
-                    return redirect(target)
+            if req_host.endswith(f".{listen_host_only}") and (not listen_port or not req_port or listen_port == req_port):
+                target = build_web_url(request.path, request=request)
+                if request.META.get("QUERY_STRING"):
+                    target = f"{target}?{request.META['QUERY_STRING']}"
+                return redirect(target)
 
             return get_response(request)
 

archivebox/core/migrations/0004_auto_20200713_1552.py

@@ -9,6 +9,12 @@ class Migration(migrations.Migration):
     ]
 
     operations = [
+        # The original schema allowed NULL timestamps. Use the already-unique
+        # 32-character snapshot ID before enforcing the later NOT NULL field.
+        migrations.RunSQL(
+            sql="UPDATE core_snapshot SET timestamp = id WHERE timestamp IS NULL",
+            reverse_sql=migrations.RunSQL.noop,
+        ),
         migrations.AlterField(
             model_name="snapshot",
             name="timestamp",

archivebox/core/migrations/0007_archiveresult.py

@@ -1,9 +1,11 @@
 # Generated by Django 3.0.8 on 2020-11-04 12:25
 
 import json
+from datetime import timezone
 from pathlib import Path
 
 from django.db import migrations, models
+from django.utils.dateparse import parse_datetime
 import django.db.models.deletion
 
 # Handle old vs new import paths
@@ -30,6 +32,13 @@ except AttributeError:
     JSONField = jsonfield.JSONField
 
 
+def parse_legacy_datetime(value):
+    parsed = parse_datetime(value) if isinstance(value, str) else value
+    if parsed is not None and parsed.tzinfo is None:
+        return parsed.replace(tzinfo=timezone.utc)
+    return parsed
+
+
 def forwards_func(apps, schema_editor):
     Snapshot = apps.get_model("core", "Snapshot")
     ArchiveResult = apps.get_model("core", "ArchiveResult")
@@ -55,8 +64,8 @@ def forwards_func(apps, schema_editor):
                         pwd=result["pwd"],
                         cmd=result.get("cmd") or [],
                         cmd_version=result.get("cmd_version") or "unknown",
-                        start_ts=result["start_ts"],
-                        end_ts=result["end_ts"],
+                        start_ts=parse_legacy_datetime(result["start_ts"]),
+                        end_ts=parse_legacy_datetime(result["end_ts"]),
                         status=result["status"],
                         output=result.get("output") or "null",
                     )

archivebox/core/migrations/0023_upgrade_to_0_9_0.py

@@ -19,6 +19,19 @@ def get_table_columns(table_name):
     return {row[1] for row in cursor.fetchall()}
 
 
+def assert_rebuild_row_count(cursor, source_table, destination_table, expected_count=None):
+    """Refuse to drop a legacy table unless every source row was copied."""
+    if expected_count is None:
+        cursor.execute(f"SELECT COUNT(*) FROM {source_table}")
+        expected_count = cursor.fetchone()[0]
+    cursor.execute(f"SELECT COUNT(*) FROM {destination_table}")
+    destination_count = cursor.fetchone()[0]
+    if destination_count != expected_count:
+        raise RuntimeError(
+            f"Refusing to replace {source_table}: copied {destination_count} of {expected_count} rows",
+        )
+
+
 def normalize_cmd(cmd):
     if not cmd:
         return "[]"
@@ -47,6 +60,14 @@ def normalize_status(status):
 
 def upgrade_core_tables(apps, schema_editor):
     """Upgrade core tables from v0.7.2 or v0.8.6rc0 to v0.9.0."""
+    # sqlite-only legacy repair/rebuild. On postgres this raw SQL (PRAGMA,
+    # sqlite_master, INSERT OR IGNORE, DATETIME, table-rebuild dance) is both
+    # invalid and unnecessary: a postgres database can never contain legacy
+    # data at this point. The final _pg_sync_schema op resyncs the real
+    # postgres schema to migration state instead.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
     from archivebox.uuid_compat import uuid7
 
     cursor = connection.cursor()
@@ -91,6 +112,7 @@ def upgrade_core_tables(apps, schema_editor):
             output VARCHAR(1024),
             created_at DATETIME,
             modified_at DATETIME,
+            notes TEXT NOT NULL DEFAULT '',
 
             FOREIGN KEY (snapshot_id) REFERENCES core_snapshot(id) ON DELETE CASCADE
         );
@@ -115,6 +137,8 @@ def upgrade_core_tables(apps, schema_editor):
             archiveresult_select_cols.append("created_at")
         if has_archiveresult_modified_at:
             archiveresult_select_cols.append("modified_at")
+        if "notes" in archiveresult_cols:
+            archiveresult_select_cols.append("notes")
 
         if has_uuid and not has_abid:
             # Migrating from v0.7.2+ (has uuid column)
@@ -124,6 +148,8 @@ def upgrade_core_tables(apps, schema_editor):
                 select_cols.append("created_at")
             if has_archiveresult_modified_at:
                 select_cols.append("modified_at")
+            if "notes" in archiveresult_cols:
+                select_cols.append("notes")
             cursor.execute(f"SELECT {', '.join(select_cols)} FROM core_archiveresult")
             old_records = cursor.fetchall()
             for i, record in enumerate(old_records, start=1):
@@ -139,8 +165,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         id, uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         values["id"],
@@ -156,6 +182,7 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
@@ -178,8 +205,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         new_uuid,
@@ -194,6 +221,7 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
@@ -213,8 +241,8 @@ def upgrade_core_tables(apps, schema_editor):
                     INSERT OR IGNORE INTO core_archiveresult_new (
                         id, uuid, snapshot_id, cmd, pwd, cmd_version,
                         start_ts, end_ts, status, extractor, output,
-                        created_at, modified_at
-                    ) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
+                        created_at, modified_at, notes
+                    ) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s, %s)
                 """,
                     (
                         values["id"],
@@ -230,12 +258,14 @@ def upgrade_core_tables(apps, schema_editor):
                         values["output"] or "",
                         values.get("created_at") or start_ts,
                         values.get("modified_at") or end_ts,
+                        values.get("notes") or "",
                     ),
                 )
                 if i % PROGRESS_EVERY == 0:
                     print(f"        copied {i}/{len(old_records)} ArchiveResults...")
         print(f"      copied {len(old_records)} ArchiveResults")
 
+    assert_rebuild_row_count(cursor, "core_archiveresult", "core_archiveresult_new", row_count)
     cursor.execute("DROP TABLE IF EXISTS core_archiveresult;")
     cursor.execute("ALTER TABLE core_archiveresult_new RENAME TO core_archiveresult;")
 
@@ -344,6 +374,9 @@ def upgrade_core_tables(apps, schema_editor):
                 if has_crawl_id:
                     insert_cols.append("crawl_id")
                     select_cols.append("REPLACE(crawl_id, '-', '')")
+                if "parent_snapshot_id" in snapshot_cols:
+                    insert_cols.append("parent_snapshot_id")
+                    select_cols.append("REPLACE(parent_snapshot_id, '-', '')")
                 if has_status:
                     insert_cols.append("status")
                     select_cols.append(
@@ -368,6 +401,18 @@ def upgrade_core_tables(apps, schema_editor):
                 if has_retry_at:
                     insert_cols.append("retry_at")
                     select_cols.append("retry_at")
+                for field_name in (
+                    "depth",
+                    "fs_version",
+                    "config",
+                    "notes",
+                    "num_uses_succeeded",
+                    "num_uses_failed",
+                    "current_step",
+                ):
+                    if field_name in snapshot_cols:
+                        insert_cols.append(field_name)
+                        select_cols.append(field_name)
 
                 cursor.execute(f"""
                     INSERT OR IGNORE INTO core_snapshot_new ({", ".join(insert_cols)})
@@ -378,6 +423,7 @@ def upgrade_core_tables(apps, schema_editor):
             else:
                 print(f"Warning: Unexpected Snapshot schema - has_added={has_added}, has_bookmarked_at={has_bookmarked_at}")
 
+    assert_rebuild_row_count(cursor, "core_snapshot", "core_snapshot_new")
     cursor.execute("DROP TABLE IF EXISTS core_snapshot;")
     cursor.execute("ALTER TABLE core_snapshot_new RENAME TO core_snapshot;")
 
@@ -416,6 +462,7 @@ def upgrade_core_tables(apps, schema_editor):
         tag_has_data = cursor.fetchone()[0] > 0
 
         if tag_has_data:
+            tag_cols = get_table_columns("core_tag")
             cursor.execute("PRAGMA table_info(core_tag)")
             tag_id_type = None
             for row in cursor.fetchall():
@@ -427,22 +474,23 @@ def upgrade_core_tables(apps, schema_editor):
                 # v0.8.6rc0: Tag IDs are UUIDs, need to convert to INTEGER
                 print("      converting Tag IDs from UUID to integers...")
 
-                # Get all tags with their UUIDs
-                cursor.execute("SELECT id, name, slug, created_at, modified_at, created_by_id FROM core_tag ORDER BY name")
+                # Get all tags with their UUIDs and any available audit fields.
+                tag_select_cols = ["id", "name", "slug"]
+                tag_select_cols.extend(field for field in ("created_at", "modified_at", "created_by_id") if field in tag_cols)
+                cursor.execute(f"SELECT {', '.join(tag_select_cols)} FROM core_tag ORDER BY name")
                 tags = cursor.fetchall()
 
                 # Create mapping from old UUID to new INTEGER ID
                 uuid_to_int_map = {}
                 for i, tag in enumerate(tags, start=1):
-                    old_id, name, slug, created_at, modified_at, created_by_id = tag
+                    values = dict(zip(tag_select_cols, tag))
+                    old_id = values.pop("id")
                     uuid_to_int_map[old_id] = i
                     # Insert with new INTEGER ID
+                    insert_values = {"id": i, **values}
                     cursor.execute(
-                        """
-                        INSERT OR IGNORE INTO core_tag_new (id, name, slug, created_at, modified_at, created_by_id)
-                        VALUES (?, ?, ?, ?, ?, ?)
-                    """,
-                        (i, name, slug, created_at, modified_at, created_by_id),
+                        f"INSERT OR IGNORE INTO core_tag_new ({', '.join(insert_values)}) VALUES ({', '.join('%s' for _ in insert_values)})",
+                        tuple(insert_values.values()),
                     )
                     if i % PROGRESS_EVERY == 0:
                         print(f"        copied {i}/{len(tags)} Tags...")
@@ -463,23 +511,33 @@ def upgrade_core_tables(apps, schema_editor):
                             cursor.execute(
                                 """
                                 INSERT OR IGNORE INTO core_snapshot_tags (id, snapshot_id, tag_id)
-                                VALUES (?, ?, ?)
+                                VALUES (%s, %s, %s)
                             """,
                                 (st_id, snapshot_id, new_tag_id),
                             )
                         if i % PROGRESS_EVERY == 0:
                             print(f"        copied {i}/{len(snapshot_tags)} SnapshotTag rows...")
+                    cursor.execute("SELECT COUNT(*) FROM core_snapshot_tags")
+                    copied_snapshot_tags = cursor.fetchone()[0]
+                    if copied_snapshot_tags != len(snapshot_tags):
+                        raise RuntimeError(
+                            f"Refusing to replace Tag IDs: copied {copied_snapshot_tags} of {len(snapshot_tags)} SnapshotTag rows",
+                        )
                 print(f"      copied {len(tags)} Tags")
             else:
-                # v0.7.2: Tag IDs are already INTEGER
-                print("      copying Tags from 0.7.x schema...")
-                cursor.execute("""
-                    INSERT OR IGNORE INTO core_tag_new (id, name, slug)
-                    SELECT id, name, slug
+                # v0.7.x and early v0.8.x use INTEGER IDs. Preserve audit
+                # metadata when the source schema has it.
+                print("      copying Tags with integer IDs...")
+                copy_cols = ["id", "name", "slug"]
+                copy_cols.extend(field for field in ("created_at", "modified_at", "created_by_id") if field in tag_cols)
+                cursor.execute(f"""
+                    INSERT OR IGNORE INTO core_tag_new ({", ".join(copy_cols)})
+                    SELECT {", ".join(copy_cols)}
                     FROM core_tag;
                 """)
                 print(f"      copied {cursor.rowcount} Tags")
 
+    assert_rebuild_row_count(cursor, "core_tag", "core_tag_new")
     cursor.execute("DROP TABLE IF EXISTS core_tag;")
     cursor.execute("ALTER TABLE core_tag_new RENAME TO core_tag;")
 
@@ -491,6 +549,15 @@ def upgrade_core_tables(apps, schema_editor):
         print("    ✓ Core table rebuild complete")
 
 
+def _pg_sync_schema(apps, schema_editor):
+    # On postgres, upgrade_core_tables is a no-op above, so the raw sqlite
+    # rebuilds of core_tag/core_snapshot/core_archiveresult never ran. Resync
+    # the real schema to this migration's end-state (tables are empty on pg).
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "core", ["Tag", "Snapshot", "SnapshotTag", "ArchiveResult"])
+
+
 class Migration(migrations.Migration):
     dependencies = [
         ("core", "0022_auto_20231023_2008"),
@@ -571,4 +638,5 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        migrations.RunPython(_pg_sync_schema, reverse_code=migrations.RunPython.noop),
     ]

archivebox/core/migrations/0024_assign_default_crawl.py

@@ -4,94 +4,9 @@
 from django.db import migrations, models
 
 
-def create_default_crawl_and_assign_snapshots(apps, schema_editor):
-    """
-    Create a default crawl for migrated snapshots and assign all snapshots without a crawl to it.
-    Uses raw SQL because the app registry isn't fully populated during migrations.
-    """
-    from django.db import connection
-    import uuid as uuid_lib
-    from datetime import datetime
-
-    cursor = connection.cursor()
-
-    # Check if there are any snapshots without a crawl
-    cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
-    snapshots_without_crawl = cursor.fetchone()[0]
-
-    if snapshots_without_crawl == 0:
-        print("✓ Fresh install or all snapshots already have crawls")
-        return
-
-    cursor.execute("SELECT url FROM core_snapshot WHERE crawl_id IS NULL ORDER BY bookmarked_at, timestamp")
-    crawl_urls = "\n".join(url for (url,) in cursor.fetchall() if url)
-
-    # Get or create system user (pk=1)
-    cursor.execute("SELECT id FROM auth_user WHERE id = 1")
-    if not cursor.fetchone():
-        cursor.execute(
-            """
-            INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
-            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, ?)
-        """,
-            [datetime.now().isoformat()],
-        )
-
-    # Create a default crawl for migrated snapshots.
-    # Depending on migration graph order, later crawls migrations may already
-    # have removed output_dir by the time this data migration runs.
-    crawl_id = uuid_lib.uuid4().hex
-    now = datetime.now().isoformat()
-    cursor.execute("PRAGMA table_info(crawls_crawl)")
-    crawl_columns = {row[1] for row in cursor.fetchall()}
-    default_columns: list[str] = []
-    default_values: list[str] = []
-    if "output_dir" in crawl_columns:
-        default_columns.append("output_dir")
-        default_values.append("''")
-    for column in ("max_size", "max_urls", "crawl_max_size", "snapshot_max_size"):
-        if column in crawl_columns:
-            default_columns.append(column)
-            default_values.append("0")
-    default_columns_sql = f"{', '.join(default_columns)}, " if default_columns else ""
-    default_values_sql = f"{', '.join(default_values)}, " if default_values else ""
-
-    cursor.execute(
-        f"""
-        INSERT INTO crawls_crawl (
-            id, created_at, modified_at, num_uses_succeeded, num_uses_failed,
-            urls, max_depth, tags_str, label, notes, {default_columns_sql}
-            status, retry_at, created_by_id, schedule_id, config, persona_id
-        ) VALUES (?, ?, ?, 0, 0, ?, 0, '', 'Migrated from v0.7.2/v0.8.6',
-                  'Auto-created crawl for migrated snapshots', {default_values_sql}
-                  'sealed', ?, 1, NULL, '{{}}', NULL)
-    """,
-        [crawl_id, now, now, crawl_urls, now],
-    )
-
-    # Assign all snapshots without a crawl to the default crawl
-    cursor.execute("UPDATE core_snapshot SET crawl_id = ? WHERE crawl_id IS NULL", [crawl_id])
-
-    print(f"✓ Assigned {snapshots_without_crawl} snapshots to default crawl {crawl_id}")
-
-
-class Migration(migrations.Migration):
-    dependencies = [
-        ("core", "0023_upgrade_to_0_9_0"),
-        ("crawls", "0002_upgrade_from_0_8_6"),
-        ("auth", "0012_alter_user_first_name_max_length"),
-    ]
-
-    operations = [
-        migrations.RunPython(
-            create_default_crawl_and_assign_snapshots,
-            reverse_code=migrations.RunPython.noop,
-        ),
-        migrations.SeparateDatabaseAndState(
-            database_operations=[
-                # Now make crawl_id NOT NULL
-                migrations.RunSQL(
-                    sql="""
+# Raw sqlite table-rebuild that makes core_snapshot.crawl_id NOT NULL. Kept
+# byte-for-byte and replayed through Django's own RunSQL only on sqlite.
+_MAKE_CRAWL_ID_NOT_NULL_SQL = """
                         -- Rebuild snapshot table with NOT NULL crawl_id
                         CREATE TABLE core_snapshot_final (
                             id TEXT PRIMARY KEY NOT NULL,
@@ -153,8 +68,122 @@ class Migration(migrations.Migration):
                         CREATE INDEX core_snapshot_retry_at_idx ON core_snapshot(retry_at);
                         CREATE INDEX core_snapshot_created_at_idx ON core_snapshot(created_at);
                         CREATE UNIQUE INDEX core_snapshot_url_crawl_unique ON core_snapshot(url, crawl_id);
-                    """,
-                    reverse_sql=migrations.RunSQL.noop,
+                    """
+
+
+def _make_crawl_id_not_null(apps, schema_editor):
+    # sqlite-only table rebuild. Reuse Django's own RunSQL statement splitting so
+    # the sqlite behavior is byte-for-byte identical to the original RunSQL op.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(sql=_MAKE_CRAWL_ID_NOT_NULL_SQL).database_forwards("core", schema_editor, None, None)
+
+
+def _pg_sync_schema(apps, schema_editor):
+    # On postgres the raw sqlite rebuilds above are skipped and crawl was only
+    # added to migration state; resync the real schema to this migration's
+    # end-state (empty tables). Snapshot is referenced by SnapshotTag and
+    # ArchiveResult, so those are rebuilt too to restore FK constraints.
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "core", ["Snapshot", "SnapshotTag", "ArchiveResult"])
+
+
+def create_default_crawl_and_assign_snapshots(apps, schema_editor):
+    """
+    Create a default crawl for migrated snapshots and assign all snapshots without a crawl to it.
+    Uses raw SQL because the app registry isn't fully populated during migrations.
+    """
+    # Legacy-data-only and sqlite-specific because it uses PRAGMA introspection.
+    # postgres install core_snapshot has no crawl_id column yet and there are no
+    # unassigned snapshots, so gate before touching any SQL.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
+    from django.db import connection
+    import uuid as uuid_lib
+    from datetime import datetime
+
+    cursor = connection.cursor()
+
+    # Check if there are any snapshots without a crawl
+    cursor.execute("SELECT COUNT(*) FROM core_snapshot WHERE crawl_id IS NULL")
+    snapshots_without_crawl = cursor.fetchone()[0]
+
+    if snapshots_without_crawl == 0:
+        print("✓ Fresh install or all snapshots already have crawls")
+        return
+
+    cursor.execute("SELECT url FROM core_snapshot WHERE crawl_id IS NULL ORDER BY bookmarked_at, timestamp")
+    crawl_urls = "\n".join(url for (url,) in cursor.fetchall() if url)
+
+    # Get or create system user (pk=1)
+    cursor.execute("SELECT id FROM auth_user WHERE id = 1")
+    if not cursor.fetchone():
+        cursor.execute(
+            """
+            INSERT INTO auth_user (id, password, is_superuser, username, first_name, last_name, email, is_staff, is_active, date_joined)
+            VALUES (1, '!', 1, 'system', '', '', '', 1, 1, %s)
+        """,
+            [datetime.now().isoformat()],
+        )
+
+    # Create a default crawl for migrated snapshots.
+    # Depending on migration graph order, later crawls migrations may already
+    # have removed output_dir by the time this data migration runs.
+    crawl_id = uuid_lib.uuid4().hex
+    now = datetime.now().isoformat()
+    cursor.execute("PRAGMA table_info(crawls_crawl)")
+    crawl_columns = {row[1] for row in cursor.fetchall()}
+    default_columns: list[str] = []
+    default_values: list[str] = []
+    if "output_dir" in crawl_columns:
+        default_columns.append("output_dir")
+        default_values.append("''")
+    for column in ("max_size", "max_urls", "crawl_max_size", "snapshot_max_size"):
+        if column in crawl_columns:
+            default_columns.append(column)
+            default_values.append("0")
+    default_columns_sql = f"{', '.join(default_columns)}, " if default_columns else ""
+    default_values_sql = f"{', '.join(default_values)}, " if default_values else ""
+
+    cursor.execute(
+        f"""
+        INSERT INTO crawls_crawl (
+            id, created_at, modified_at, num_uses_succeeded, num_uses_failed,
+            urls, max_depth, tags_str, label, notes, {default_columns_sql}
+            status, retry_at, created_by_id, schedule_id, config, persona_id
+        ) VALUES (%s, %s, %s, 0, 0, %s, 0, '', 'Migrated from v0.7.2/v0.8.6',
+                  'Auto-created crawl for migrated snapshots', {default_values_sql}
+                  'sealed', %s, 1, NULL, '{{}}', NULL)
+    """,
+        [crawl_id, now, now, crawl_urls, now],
+    )
+
+    # Assign all snapshots without a crawl to the default crawl
+    cursor.execute("UPDATE core_snapshot SET crawl_id = %s WHERE crawl_id IS NULL", [crawl_id])
+
+    print(f"✓ Assigned {snapshots_without_crawl} snapshots to default crawl {crawl_id}")
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0023_upgrade_to_0_9_0"),
+        ("crawls", "0002_upgrade_from_0_8_6"),
+        ("auth", "0012_alter_user_first_name_max_length"),
+    ]
+
+    operations = [
+        migrations.RunPython(
+            create_default_crawl_and_assign_snapshots,
+            reverse_code=migrations.RunPython.noop,
+        ),
+        migrations.SeparateDatabaseAndState(
+            database_operations=[
+                # Now make crawl_id NOT NULL (sqlite-only table rebuild)
+                migrations.RunPython(
+                    _make_crawl_id_not_null,
+                    reverse_code=migrations.RunPython.noop,
                 ),
             ],
             state_operations=[
@@ -169,4 +198,5 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        migrations.RunPython(_pg_sync_schema, reverse_code=migrations.RunPython.noop),
     ]

archivebox/core/migrations/0025_alter_archiveresult_options_alter_snapshot_options_and_more.py

@@ -11,6 +11,12 @@ from archivebox.uuid_compat import uuid7
 
 def copy_old_fields_to_new(apps, schema_editor):
     """Copy data from old field names to new field names after AddField operations."""
+    # sqlite-only legacy data copy (PRAGMA introspection). On postgres there is
+    # no legacy data and the state-only AddFields below are resynced by
+    # _pg_sync_schema at the end of this migration.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
     cursor = connection.cursor()
 
     # Check if old fields still exist
@@ -43,6 +49,20 @@ def copy_old_fields_to_new(apps, schema_editor):
     # NOTE: UUIDs are already populated by migration 0023 for all migration paths.
 
 
+def _pg_sync_schema(apps, schema_editor):
+    # This migration mixes real ORM AddFields with state-only AddFields (the
+    # snapshot config/current_step/depth/notes/num_uses_*/parent_snapshot/
+    # retry_at/status columns and archiveresult created_at/modified_at were
+    # created on sqlite by 0023's raw rebuild, so here they are state-only).
+    # On postgres those columns would never land, desyncing the real schema.
+    # Resync to this migration's final state (empty tables on pg). Tag and
+    # Snapshot both gain state-only fields; Snapshot is referenced by
+    # SnapshotTag and ArchiveResult, so rebuild those too.
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "core", ["Tag", "Snapshot", "SnapshotTag", "ArchiveResult"])
+
+
 class Migration(migrations.Migration):
     dependencies = [
         ("core", "0024_assign_default_crawl"),
@@ -61,8 +81,8 @@ class Migration(migrations.Migration):
         ),
         # NOTE: RemoveField for cmd, cmd_version, pwd moved to migration 0027
         # to allow data migration to Process records first
-        # NOTE: created_at/modified_at are created by migration 0023 so they can
-        # preserve old ArchiveResult row metadata before SQLite table rebuilds.
+        # NOTE: created_at/modified_at/notes are created by migration 0023 so
+        # they can preserve old ArchiveResult metadata before SQLite rebuilds.
         # Update Django's state here before the AddField operations below.
         migrations.SeparateDatabaseAndState(
             database_operations=[],
@@ -77,6 +97,11 @@ class Migration(migrations.Migration):
                     name="modified_at",
                     field=models.DateTimeField(auto_now=True),
                 ),
+                migrations.AddField(
+                    model_name="archiveresult",
+                    name="notes",
+                    field=models.TextField(blank=True, default=""),
+                ),
             ],
         ),
         migrations.AddField(
@@ -95,11 +120,6 @@ class Migration(migrations.Migration):
                 max_length=255,
             ),
         ),
-        migrations.AddField(
-            model_name="archiveresult",
-            name="notes",
-            field=models.TextField(blank=True, default=""),
-        ),
         migrations.AddField(
             model_name="archiveresult",
             name="num_uses_failed",
@@ -218,26 +238,33 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
-        migrations.AddField(
-            model_name="tag",
-            name="created_at",
-            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now, null=True),
-        ),
-        migrations.AddField(
-            model_name="tag",
-            name="created_by",
-            field=models.ForeignKey(
-                default=archivebox.base_models.models.get_or_create_system_user_pk,
-                null=True,
-                on_delete=django.db.models.deletion.CASCADE,
-                related_name="tag_set",
-                to=settings.AUTH_USER_MODEL,
-            ),
-        ),
-        migrations.AddField(
-            model_name="tag",
-            name="modified_at",
-            field=models.DateTimeField(auto_now=True),
+        # 0023 creates these columns before rebuilding the legacy Tag table so
+        # their values survive. Declare them in state without rebuilding it.
+        migrations.SeparateDatabaseAndState(
+            database_operations=[],
+            state_operations=[
+                migrations.AddField(
+                    model_name="tag",
+                    name="created_at",
+                    field=models.DateTimeField(db_index=True, default=django.utils.timezone.now, null=True),
+                ),
+                migrations.AddField(
+                    model_name="tag",
+                    name="created_by",
+                    field=models.ForeignKey(
+                        default=archivebox.base_models.models.get_or_create_system_user_pk,
+                        null=True,
+                        on_delete=django.db.models.deletion.CASCADE,
+                        related_name="tag_set",
+                        to=settings.AUTH_USER_MODEL,
+                    ),
+                ),
+                migrations.AddField(
+                    model_name="tag",
+                    name="modified_at",
+                    field=models.DateTimeField(auto_now=True),
+                ),
+            ],
         ),
         # Copy data from old field names to new field names after AddField operations
         migrations.RunPython(
@@ -335,4 +362,5 @@ class Migration(migrations.Migration):
             model_name="snapshot",
             constraint=models.UniqueConstraint(fields=("timestamp",), name="unique_timestamp"),
         ),
+        migrations.RunPython(_pg_sync_schema, reverse_code=migrations.RunPython.noop),
     ]

archivebox/core/migrations/0027_copy_archiveresult_to_process.py

@@ -1,7 +1,7 @@
 # Generated by hand on 2026-01-01
 # Copies ArchiveResult cmd/pwd/cmd_version data to Process records before removing old fields
 
-from django.db import migrations, connection
+from django.db import migrations
 import json
 from pathlib import Path
 from archivebox.uuid_compat import uuid7
@@ -51,7 +51,7 @@ def get_or_create_current_machine(cursor):
     guid = f"host_{hostname}"  # Simple but stable identifier
 
     # Check if machine exists
-    cursor.execute("SELECT id FROM machine_machine WHERE guid = ?", [guid])
+    cursor.execute("SELECT id FROM machine_machine WHERE guid = %s", [guid])
     row = cursor.fetchone()
 
     if row:
@@ -76,7 +76,7 @@ def get_or_create_current_machine(cursor):
                 hw_in_docker, hw_in_vm, hw_manufacturer, hw_product, hw_uuid,
                 os_arch, os_family, os_platform, os_release, os_kernel,
                 stats, config, num_uses_failed, num_uses_succeeded
-            ) VALUES (?, ?, ?, ?, ?, 0, 0, '', '', '',
+            ) VALUES (%s, %s, %s, %s, %s, 0, 0, '', '', '',
                       '', '', '', '', '', '{}', '{}', 0, 0)
         """,
             [machine_id, now, now, guid, hostname],
@@ -90,7 +90,7 @@ def get_or_create_current_machine(cursor):
                 hw_in_docker, hw_in_vm, hw_manufacturer, hw_product, hw_uuid,
                 os_arch, os_family, os_platform, os_release, os_kernel,
                 stats, num_uses_failed, num_uses_succeeded
-            ) VALUES (?, ?, ?, ?, ?, 0, 0, '', '', '',
+            ) VALUES (%s, %s, %s, %s, %s, 0, 0, '', '', '',
                       '', '', '', '', '', '{}', 0, 0)
         """,
             [machine_id, now, now, guid, hostname],
@@ -125,7 +125,7 @@ def get_or_create_binary(cursor, machine_id, name, abspath, version):
     cursor.execute(
         """
         SELECT id FROM machine_binary
-        WHERE machine_id = ? AND name = ? AND abspath = ? AND version = ?
+        WHERE machine_id = %s AND name = %s AND abspath = %s AND version = %s
     """,
         [machine_id, name, abspath, version],
     )
@@ -162,7 +162,7 @@ def get_or_create_binary(cursor, machine_id, name, abspath, version):
         "num_uses_succeeded": 0,
     }
     insert_cols = [col for col in values_by_col if col in binary_cols]
-    placeholders = ", ".join(["?"] * len(insert_cols))
+    placeholders = ", ".join(["%s"] * len(insert_cols))
     cursor.execute(
         f"""
         INSERT INTO machine_binary ({", ".join(insert_cols)})
@@ -244,7 +244,7 @@ def create_process(cursor, machine_id, pwd, cmd, status, exit_code, started_at,
         "num_uses_succeeded": 0,
     }
     insert_cols = [col for col in values_by_col if col in process_cols]
-    placeholders = ", ".join(["?"] * len(insert_cols))
+    placeholders = ", ".join(["%s"] * len(insert_cols))
     cursor.execute(
         f"""
         INSERT INTO machine_process ({", ".join(insert_cols)})
@@ -275,7 +275,15 @@ def copy_archiveresult_data_to_process(apps, schema_editor):
     - failed → exited (exit_code=1)
     - skipped → exited (exit_code=None)
     """
-    cursor = connection.cursor()
+    # sqlite-only legacy data copy because it uses PRAGMA introspection.
+    # A postgres install can never contain legacy ArchiveResult cmd/pwd data at
+    # this point, so there is nothing to copy into machine_process. The
+    # RemoveField ops below are real (non-state-only) and drop the now-empty
+    # cmd/pwd/cmd_version columns on both vendors, so no resync is needed.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
+    cursor = schema_editor.connection.cursor()
 
     # Check if old fields still exist (skip if fresh install or already migrated)
     cursor.execute("PRAGMA table_info(core_archiveresult)")
@@ -287,96 +295,101 @@ def copy_archiveresult_data_to_process(apps, schema_editor):
 
     # Check if process_id field exists (should exist from 0026)
     if "process_id" not in cols:
-        print("✗ ERROR: process_id field not found. Migration 0026 must run first.")
-        return
+        raise RuntimeError("process_id field not found. Migration 0026 must run first.")
 
     # Get or create Machine.current()
     machine_id = get_or_create_current_machine(cursor)
 
-    # Get ArchiveResults without process_id that have cmd data
+    # Get every ArchiveResult without a Process. Rows with empty execution
+    # metadata still need a Process before the legacy columns can be removed.
     # Use plugin (extractor was renamed to plugin in migration 0025)
     cursor.execute("""
         SELECT id, snapshot_id, plugin, cmd, pwd, cmd_version,
                status, start_ts, end_ts, created_at
         FROM core_archiveresult
         WHERE process_id IS NULL
-        AND (cmd IS NOT NULL OR pwd IS NOT NULL)
     """)
 
     results = cursor.fetchall()
 
     if not results:
         print("    ✓ No ArchiveResults need Process migration")
-        return
-
-    print(f"    - Migrating {len(results)} ArchiveResults to Process rows...")
+    else:
+        print(f"    - Migrating {len(results)} ArchiveResults to Process rows...")
 
     migrated_count = 0
-    skipped_count = 0
-    error_count = 0
-
     for i, row in enumerate(results):
         ar_id, snapshot_id, plugin, cmd_raw, pwd, cmd_version, status, start_ts, end_ts, created_at = row
 
-        try:
-            # Parse cmd field
-            cmd_array = parse_cmd_field(cmd_raw)
+        # Parse cmd field
+        cmd_array = parse_cmd_field(cmd_raw)
 
-            # Extract binary info from cmd[0] if available
-            binary_id = None
-            if cmd_array and cmd_array[0]:
-                binary_name = Path(cmd_array[0]).name or plugin  # Fallback to plugin name
-                binary_abspath = cmd_array[0]
-                binary_version = cmd_version or ""
+        # Extract binary info from cmd[0] if available
+        binary_id = None
+        if cmd_array and cmd_array[0]:
+            binary_name = Path(cmd_array[0]).name or plugin  # Fallback to plugin name
+            binary_abspath = cmd_array[0]
+            binary_version = cmd_version or ""
 
-                # Get or create Binary record
-                binary_id = get_or_create_binary(
-                    cursor,
-                    machine_id,
-                    binary_name,
-                    binary_abspath,
-                    binary_version,
-                )
-
-            # Map status
-            process_status, exit_code = map_status(status)
-
-            # Set timestamps
-            started_at = start_ts or created_at
-            ended_at = end_ts if process_status == "exited" else None
-
-            # Create Process record
-            process_id = create_process(
-                cursor=cursor,
-                machine_id=machine_id,
-                pwd=pwd or "",
-                cmd=cmd_array,
-                status=process_status,
-                exit_code=exit_code,
-                started_at=started_at,
-                ended_at=ended_at,
-                binary_id=binary_id,
+            # Get or create Binary record
+            binary_id = get_or_create_binary(
+                cursor,
+                machine_id,
+                binary_name,
+                binary_abspath,
+                binary_version,
             )
+        elif cmd_version:
+            raise RuntimeError(f"ArchiveResult {ar_id} has cmd_version metadata but no command")
 
-            # Link ArchiveResult to Process
-            cursor.execute(
-                "UPDATE core_archiveresult SET process_id = ? WHERE id = ?",
-                [process_id, ar_id],
-            )
+        # Map status
+        process_status, exit_code = map_status(status)
 
-            migrated_count += 1
-            if migrated_count % PROGRESS_EVERY == 0:
-                print(f"      migrated {migrated_count}/{len(results)} ArchiveResults...")
+        # Set timestamps
+        started_at = start_ts or created_at
+        ended_at = end_ts if process_status == "exited" else None
 
-        except Exception as e:
-            print(f"✗ Error migrating ArchiveResult {ar_id}: {e}")
-            import traceback
+        # Create Process record
+        process_id = create_process(
+            cursor=cursor,
+            machine_id=machine_id,
+            pwd=pwd or "",
+            cmd=cmd_array,
+            status=process_status,
+            exit_code=exit_code,
+            started_at=started_at,
+            ended_at=ended_at,
+            binary_id=binary_id,
+        )
 
-            traceback.print_exc()
-            error_count += 1
-            continue
+        # Link ArchiveResult to Process
+        cursor.execute(
+            "UPDATE core_archiveresult SET process_id = %s WHERE id = %s",
+            [process_id, ar_id],
+        )
+        if cursor.rowcount != 1:
+            raise RuntimeError(f"ArchiveResult {ar_id} was not linked to Process {process_id}")
 
-    print(f"    ✓ Process migration complete: {migrated_count} migrated, {skipped_count} skipped, {error_count} errors")
+        migrated_count += 1
+        if migrated_count % PROGRESS_EVERY == 0:
+            print(f"      migrated {migrated_count}/{len(results)} ArchiveResults...")
+
+    cursor.execute("""
+        SELECT ar.id, ar.cmd, ar.pwd, ar.cmd_version, ar.process_id,
+               process.cmd, process.pwd, binary.version
+        FROM core_archiveresult AS ar
+        LEFT JOIN machine_process AS process ON process.id = ar.process_id
+        LEFT JOIN machine_binary AS binary ON binary.id = process.binary_id
+    """)
+    for ar_id, cmd_raw, pwd, cmd_version, process_id, process_cmd_raw, process_pwd, binary_version in cursor.fetchall():
+        if not process_id:
+            raise RuntimeError(f"ArchiveResult {ar_id} is not linked to a Process")
+        if parse_cmd_field(process_cmd_raw) != parse_cmd_field(cmd_raw) or process_pwd != (pwd or ""):
+            raise RuntimeError(f"ArchiveResult {ar_id} execution metadata was not preserved")
+        if cmd_version and binary_version != cmd_version:
+            raise RuntimeError(f"ArchiveResult {ar_id} command version was not preserved")
+
+    print(f"    ✓ Process migration complete: {migrated_count} migrated")
 
 
 class Migration(migrations.Migration):

archivebox/core/migrations/0029_migrate_archiveresult_to_uuid_pk.py

@@ -1,7 +1,7 @@
 # Generated by hand on 2026-01-02
 # Migrate ArchiveResult from integer PK to UUID PK (matching Snapshot)
 
-from django.db import migrations, models, connection
+from django.db import migrations, models
 from uuid import UUID
 from archivebox.uuid_compat import uuid7
 
@@ -26,7 +26,13 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
 
     Result: Clean schema with ONLY id as UUIDField (no old_id, no uuid)
     """
-    cursor = connection.cursor()
+    # sqlite-only table rebuild (sqlite_master, PRAGMA, table copy/rename). On
+    # postgres this is unnecessary and invalid: the table is empty and the
+    # state flip of id -> UUIDField is applied by _pg_sync_schema below.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
+    cursor = schema_editor.connection.cursor()
 
     # Check if table exists and has data
     cursor.execute("SELECT name FROM sqlite_master WHERE type='table' AND name='core_archiveresult'")
@@ -95,23 +101,35 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
     col_names = [col[1] for col in columns]
     has_uuid_column = "uuid" in col_names
 
+    used_uuids = set()
+
+    def fresh_uuid():
+        while True:
+            candidate = uuid7().hex
+            if candidate not in used_uuids:
+                used_uuids.add(candidate)
+                return candidate
+
+    def unique_uuid(existing_uuid):
+        try:
+            candidate = UUID(str(existing_uuid)).hex if existing_uuid else None
+        except (TypeError, ValueError, AttributeError):
+            candidate = None
+
+        if candidate and candidate not in used_uuids:
+            used_uuids.add(candidate)
+            return candidate
+        return fresh_uuid()
+
     if has_uuid_column:
         cursor.execute("SELECT id, uuid FROM core_archiveresult")
         records = cursor.fetchall()
-        id_to_uuid = {}
-        for old_id, existing_uuid in records:
-            if existing_uuid:
-                # Normalize existing UUID to 32-char hex format (Django SQLite UUIDField format)
-                # (existing UUIDs might be stored with or without dashes in old schema)
-                id_to_uuid[old_id] = UUID(existing_uuid).hex
-            else:
-                # Generate new UUIDv7 (time-ordered) as 32-char hex
-                id_to_uuid[old_id] = uuid7().hex
+        id_to_uuid = {old_id: unique_uuid(existing_uuid) for old_id, existing_uuid in records}
     else:
         # 0.7.x path: no uuid column, generate new UUIDs for all records
         cursor.execute("SELECT id FROM core_archiveresult")
         records = cursor.fetchall()
-        id_to_uuid = {old_id: uuid7().hex for (old_id,) in records}
+        id_to_uuid = {old_id: fresh_uuid() for (old_id,) in records}
 
     # Step 3: Copy data with UUIDs as new primary key
     cursor.execute("SELECT * FROM core_archiveresult")
@@ -119,7 +137,7 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
 
     # col_names already fetched in Step 2
     inserted_count = 0
-    for i, record in enumerate(old_records):
+    for record in old_records:
         old_id = record[col_names.index("id")]
         new_uuid = id_to_uuid[old_id]
 
@@ -152,29 +170,34 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
         # Build INSERT statement (only copy fields that exist in source)
         existing_fields = [f for f in fields_to_copy if f in values]
 
-        placeholders = ", ".join(["?"] * (len(existing_fields) + 1))  # +1 for id
+        placeholders = ", ".join(["%s"] * (len(existing_fields) + 1))  # +1 for id
         field_list = "id, " + ", ".join(existing_fields)
 
         insert_values = [new_uuid] + [values.get(f) for f in existing_fields]
 
-        try:
-            cursor.execute(
-                f"INSERT INTO core_archiveresult_new ({field_list}) VALUES ({placeholders})",
-                insert_values,
-            )
-            inserted_count += 1
-            if inserted_count % PROGRESS_EVERY == 0:
-                print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults...")
-        except Exception as e:
-            print(f"[0029] ERROR inserting record {old_id}: {e}")
-            if i == 0:
-                print(f"[0029] First record values: {insert_values[:5]}...")
-                raise
+        cursor.execute(
+            f"INSERT INTO core_archiveresult_new ({field_list}) VALUES ({placeholders})",
+            insert_values,
+        )
+        inserted_count += 1
+        if inserted_count % PROGRESS_EVERY == 0:
+            print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults...")
 
     if old_records:
         print(f"      copied {inserted_count}/{len(old_records)} ArchiveResults")
 
-    # Step 4: Replace old table with new table
+    cursor.execute("SELECT COUNT(*) FROM core_archiveresult")
+    source_count = cursor.fetchone()[0]
+    cursor.execute("SELECT COUNT(*) FROM core_archiveresult_new")
+    destination_count = cursor.fetchone()[0]
+    if source_count != row_count or destination_count != row_count or inserted_count != row_count:
+        raise RuntimeError(
+            "ArchiveResult UUID migration row-count mismatch: "
+            f"expected={row_count}, source={source_count}, destination={destination_count}, inserted={inserted_count}",
+        )
+
+    # Step 4: Replace old table with new table. Django runs this migration in
+    # one transaction, so any failure above leaves the source table untouched.
     cursor.execute("DROP TABLE core_archiveresult")
     cursor.execute("ALTER TABLE core_archiveresult_new RENAME TO core_archiveresult")
 
@@ -190,6 +213,16 @@ def migrate_archiveresult_id_to_uuid(apps, schema_editor):
     print(f"    ✓ ArchiveResult UUID primary key migration complete ({row_count} records)")
 
 
+def _pg_sync_schema(apps, schema_editor):
+    # On postgres the sqlite table rebuild above is skipped; the id -> UUIDField
+    # flip and uuid-field removal only reach migration state. Resync the real
+    # (empty) core_archiveresult table to this migration's end-state. Nothing in
+    # core references ArchiveResult, so no other models need rebuilding.
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "core", ["ArchiveResult"])
+
+
 class Migration(migrations.Migration):
     dependencies = [
         ("core", "0028_alter_snapshot_fs_version"),
@@ -217,4 +250,5 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        migrations.RunPython(_pg_sync_schema, reverse_code=migrations.RunPython.noop),
     ]

archivebox/core/migrations/0045_archiveresult_unique_hook.py

@@ -2,13 +2,13 @@ from django.db import migrations, models
 
 
 def deduplicate_archiveresults_per_hook(apps, schema_editor):
-    """Drop duplicate ArchiveResult rows per (snapshot, plugin, hook_name).
+    """Give historical duplicate rows unique hook identities.
 
     Real long-lived collections (cabbage's demo, beta-tester DBs) accumulated
-    multiple rows per hook over the dev rc chain. The next operation adds a
-    UniqueConstraint on that tuple; without this cleanup pass the constraint
-    fails with ``UNIQUE constraint failed`` mid-migration and bricks startup.
-    Keep the row with the highest id (most recent) for each tuple.
+    multiple rows per hook. Preserve every row while keeping the newest row on
+    the canonical hook identity expected by the current scheduler. Older rows
+    receive deterministic historical identities before the existing unique
+    constraint is added.
     """
     ArchiveResult = apps.get_model("core", "ArchiveResult")
     duplicate_groups = (
@@ -20,9 +20,10 @@ def deduplicate_archiveresults_per_hook(apps, schema_editor):
             "plugin": group["plugin"],
             "hook_name": group["hook_name"],
         }
-        keep = ArchiveResult.objects.filter(**lookup).order_by("-id").values_list("id", flat=True).first()
-        if keep is not None:
-            ArchiveResult.objects.filter(**lookup).exclude(id=keep).delete()
+        rows = list(ArchiveResult.objects.filter(**lookup).order_by("-id").values_list("id", flat=True))
+        for row_id in rows[1:]:
+            historical_name = f"__legacy_history__{row_id}"[:255]
+            ArchiveResult.objects.filter(id=row_id).update(hook_name=historical_name)
 
 
 class Migration(migrations.Migration):

archivebox/core/migrations/0046_repair_snapshot_permissions.py

@@ -13,6 +13,13 @@ def _repair_snapshot_permissions(apps, schema_editor):
     no underlying column. Fresh installs added the column via 0041 and
     this is a no-op.
     """
+    # sqlite-only legacy repair (PRAGMA table_xinfo + sqlite VIRTUAL generated
+    # column syntax). On postgres the permissions column was created by 0041's
+    # portable GeneratedField AddField (STORED generated column), so there is
+    # nothing to repair.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
     cursor = schema_editor.connection.cursor()
     # ``table_xinfo`` lists STORED/VIRTUAL generated columns; ``table_info``
     # silently drops them, so a prior 0041 that landed the STORED column

archivebox/core/migrations/0051_postgres_url_pattern_ops_index.py

@@ -0,0 +1,30 @@
+# Postgres-only: btree pattern-ops index on core_snapshot.url so LIKE 'prefix%'
+# queries (url__startswith and the URL prefix search) stay index scans under any
+# database collation. SQLite needs nothing here: its plain url index already
+# serves the bytewise range comparisons used on that backend.
+
+from django.db import migrations
+
+
+def add_pg_url_pattern_index(apps, schema_editor):
+    if schema_editor.connection.vendor != "postgresql":
+        return
+    schema_editor.execute(
+        "CREATE INDEX IF NOT EXISTS core_snapshot_url_pattern_ops_idx ON core_snapshot (url text_pattern_ops)",
+    )
+
+
+def remove_pg_url_pattern_index(apps, schema_editor):
+    if schema_editor.connection.vendor != "postgresql":
+        return
+    schema_editor.execute("DROP INDEX IF EXISTS core_snapshot_url_pattern_ops_idx")
+
+
+class Migration(migrations.Migration):
+    dependencies = [
+        ("core", "0050_snapshot_permissions_not_null"),
+    ]
+
+    operations = [
+        migrations.RunPython(add_pg_url_pattern_index, remove_pg_url_pattern_index),
+    ]

archivebox/core/models.py

@@ -1,68 +1,72 @@
 __package__ = "archivebox.core"
 
-from typing import TYPE_CHECKING, Optional, Any
-from collections.abc import Iterable, Mapping, Sequence
-import uuid
-from archivebox.uuid_compat import CompactUUIDField, uuid7
-from datetime import datetime, timedelta
-
-import os
 import json
+import os
+import uuid
+from collections.abc import Iterable, Mapping, Sequence
+from datetime import UTC, datetime, timedelta
 from pathlib import Path
+from typing import TYPE_CHECKING, Any, ClassVar, Optional
 from urllib.parse import urlparse
 
-from statemachine import State, registry
-
+from django.conf import settings
+from django.contrib import admin
+from django.core.cache import cache
+from django.core.exceptions import FieldDoesNotExist, ObjectDoesNotExist, ValidationError
 from django.db import models, transaction
 from django.db.models import Case, F, Q, QuerySet, Sum, Value, When
-from django.db.models.functions import Coalesce, Concat
 from django.db.models.fields.json import KT
-from django.utils.functional import cached_property
-from django.utils.text import slugify
-from django.utils import timezone
-from django.core.cache import cache
+from django.db.models.functions import Coalesce, Concat
 from django.urls import reverse_lazy
-from django.contrib import admin
-from django.conf import settings
-from django.core.exceptions import ObjectDoesNotExist, ValidationError
+from django.utils import timezone
+from django.utils.functional import cached_property
 from django.utils.safestring import mark_safe
+from django.utils.text import slugify
+from statemachine import State, registry
 
+from archivebox.base_models.models import (
+    ModelWithConfig,
+    ModelWithDeleteAfter,
+    ModelWithHealthStats,
+    ModelWithNotes,
+    ModelWithOutputDir,
+    ModelWithUUID,
+    get_or_create_system_user_pk,
+)
 from archivebox.config import CONSTANTS
 from archivebox.config.common import get_config, rprint
+from archivebox.crawls.models import Crawl
+from archivebox.machine.models import Binary
 from archivebox.misc.system import atomic_write
 from archivebox.misc.util import (
-    parse_date,
     domain as url_domain,
+)
+from archivebox.misc.util import (
+    htmldecode,
+    parse_date,
+    sanitize_html_text,
     to_json,
     ts_to_date_str,
-    urlencode,
-    htmldecode,
-    sanitize_html_text,
     urldecode,
+    urlencode,
     validate_url,
 )
 from archivebox.plugins.discovery import (
-    get_plugins,
-    get_plugin_name,
     get_plugin_icon,
+    get_plugin_name,
+    get_plugins,
 )
-from archivebox.base_models.models import (
-    ModelWithUUID,
-    ModelWithDeleteAfter,
-    ModelWithOutputDir,
-    ModelWithConfig,
-    ModelWithNotes,
-    ModelWithHealthStats,
-    get_or_create_system_user_pk,
-)
-from archivebox.workers.models import ACTIVE_STATE_LEASE_SECONDS, RETRY_AT_MAX, ModelWithStateMachine, BaseStateMachine
-from archivebox.crawls.models import Crawl
-from archivebox.machine.models import Binary
+from archivebox.uuid_compat import CompactUUIDField, uuid7
+from archivebox.workers.models import ACTIVE_STATE_LEASE_SECONDS, RETRY_AT_MAX, BaseStateMachine, ModelWithStateMachine
 
 if TYPE_CHECKING:
     from archivebox.config.common import ArchiveBoxBaseConfig
 
 
+class SnapshotMigrationError(RuntimeError):
+    """Raised when a snapshot filesystem migration fails validation."""
+
+
 class UngroupedSubquery(models.Subquery):
     """Scalar subquery that should not be copied into the outer GROUP BY."""
 
@@ -156,7 +160,7 @@ class SnapshotTag(models.Model):
     class Meta:
         app_label = "core"
         db_table = "core_snapshot_tags"
-        unique_together = [("snapshot", "tag")]
+        unique_together: ClassVar[list[tuple[str, str]]] = [("snapshot", "tag")]
 
 
 class SnapshotQuerySet(models.QuerySet):
@@ -183,9 +187,14 @@ class SnapshotQuerySet(models.QuerySet):
                 for crawl_id, permissions in Crawl.objects.filter(pk__in=missing_crawl_ids).values_list("pk", "permissions")
             }
 
+        from archivebox.misc.db import truncate_overlong_charfields
+
         for obj in objs:
             if isinstance(obj, self.model):
                 obj.ensure_permissions_config(crawl_permissions=crawl_permissions_by_id.get(str(obj.crawl_id)))
+                # bulk_create bypasses pre_save, so clamp CharFields here too
+                # (e.g. page titles) to stay within postgres VARCHAR limits.
+                truncate_overlong_charfields(obj)
         return super().bulk_create(objs, *args, **kwargs)
 
     def paged_iterator(self, chunk_size: int = 500):
@@ -221,7 +230,7 @@ class SnapshotQuerySet(models.QuerySet):
                 field_name = pk_field
             ordering.append(f"-{field_name}" if descending else field_name)
 
-        ordered_field_names = [term[1:] if term.startswith("-") else term for term in ordering]
+        ordered_field_names = [term.removeprefix("-") for term in ordering]
         try:
             if any(self.model._meta.get_field(field_name).null for field_name in ordered_field_names):
                 offset = 0
@@ -232,7 +241,7 @@ class SnapshotQuerySet(models.QuerySet):
                     yield from batch
                     offset += chunk_size
                 return
-        except Exception:
+        except (AttributeError, FieldDoesNotExist):
             offset = 0
             while True:
                 batch = list(self[offset : offset + chunk_size])
@@ -287,7 +296,7 @@ class SnapshotQuerySet(models.QuerySet):
     # Filtering Methods
     # =========================================================================
 
-    FILTER_TYPES = {
+    FILTER_TYPES: ClassVar[dict[str, Any]] = {
         "exact": lambda pattern: models.Q(url=pattern),
         "substring": lambda pattern: models.Q(url__icontains=pattern),
         "regex": lambda pattern: models.Q(url__iregex=pattern),
@@ -332,7 +341,6 @@ class SnapshotQuerySet(models.QuerySet):
         return self.filter(q_filter)
 
     def search(self, **kwargs) -> "SnapshotQuerySet":
-        from datetime import timezone as dt_timezone
 
         from archivebox.core.snapshot_status import filter_snapshots_by_status
         from archivebox.search.query import apply_snapshot_search
@@ -359,9 +367,9 @@ class SnapshotQuerySet(models.QuerySet):
         if kwargs.get("tag"):
             queryset = queryset.filter(tags__name__iexact=kwargs["tag"])
         if kwargs.get("before") is not None:
-            queryset = queryset.filter(bookmarked_at__lt=datetime.fromtimestamp(float(kwargs["before"]), tz=dt_timezone.utc))
+            queryset = queryset.filter(bookmarked_at__lt=datetime.fromtimestamp(float(kwargs["before"]), tz=UTC))
         if kwargs.get("after") is not None:
-            queryset = queryset.filter(bookmarked_at__gt=datetime.fromtimestamp(float(kwargs["after"]), tz=dt_timezone.utc))
+            queryset = queryset.filter(bookmarked_at__gt=datetime.fromtimestamp(float(kwargs["after"]), tz=UTC))
 
         if query:
             queryset = apply_snapshot_search(
@@ -394,7 +402,8 @@ class SnapshotQuerySet(models.QuerySet):
     def to_json(self, with_headers: bool = False) -> str:
         """Generate JSON index from snapshots"""
         import sys
-        from datetime import datetime, timezone as tz
+        from datetime import datetime
+
         from archivebox.config import VERSION
 
         config = get_config()
@@ -425,7 +434,7 @@ class SnapshotQuerySet(models.QuerySet):
             output = {
                 **MAIN_INDEX_HEADER,
                 "num_links": len(snapshot_dicts),
-                "updated": datetime.now(tz.utc),
+                "updated": datetime.now(UTC),
                 "last_run_cmd": sys.argv,
                 "links": snapshot_dicts,
             }
@@ -442,8 +451,10 @@ class SnapshotQuerySet(models.QuerySet):
 
     def to_html(self, with_headers: bool = True) -> str:
         """Generate main index HTML from snapshots"""
-        from datetime import datetime, timezone as tz
+        from datetime import datetime
+
         from django.template.loader import render_to_string
+
         from archivebox.config import VERSION
         from archivebox.config.version import get_COMMIT_HASH
 
@@ -458,8 +469,8 @@ class SnapshotQuerySet(models.QuerySet):
                 "version": VERSION,
                 "git_sha": get_COMMIT_HASH() or VERSION,
                 "num_links": str(len(snapshot_list)),
-                "date_updated": datetime.now(tz.utc).strftime("%Y-%m-%d"),
-                "time_updated": datetime.now(tz.utc).strftime("%Y-%m-%d %H:%M"),
+                "date_updated": datetime.now(UTC).strftime("%Y-%m-%d"),
+                "time_updated": datetime.now(UTC).strftime("%Y-%m-%d %H:%M"),
                 "links": snapshot_list,
                 "FOOTER_INFO": config.FOOTER_INFO,
             },
@@ -598,11 +609,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         app_label = "core"
         verbose_name = "Snapshot"
         verbose_name_plural = "Snapshots"
-        indexes = [
+        indexes: ClassVar[list[models.Index]] = [
             models.Index(fields=["-bookmarked_at", "-created_at"], name="snapshot_public_order_idx"),
             models.Index(fields=["crawl", "status", "modified_at"], name="snapshot_progress_idx"),
         ]
-        constraints = [
+        constraints: ClassVar[list[models.BaseConstraint]] = [
             # Allow same URL in different crawls, but not duplicates within same crawl
             models.UniqueConstraint(fields=["url", "crawl"], name="unique_url_per_crawl"),
             # Global timestamp uniqueness for 1:1 symlink mapping
@@ -695,12 +706,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
     def restore_paused_scheduler_marker(self) -> None:
         """
-        Keep explicit maintenance from accidentally resuming paused snapshots.
-
-        Targeted jobs such as `archivebox update --index-only` may bump
-        retry_at so the orchestrator can run only queued search ArchiveResult
-        rows. After that maintenance pass, the lifecycle must remain PAUSED and
-        retry_at must go back to MAX until a real resume transition happens.
+        Restore the indefinite scheduler marker owned by the PAUSED lifecycle.
         """
         type(self).objects.filter(
             pk=self.pk,
@@ -835,6 +841,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         protected_subdomains = {"admin", "web", "api"}
         protected_hosts: set[tuple[str, str | None]] = set()
         protected_roots: set[tuple[str, str | None]] = set()
+        protected_local_roots: set[tuple[str, str | None]] = set()
         for host_value in (
             get_listen_host(config=config),
             get_base_host(config=config),
@@ -852,6 +859,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             if protected_host in {"", "0.0.0.0", "::", "127.0.0.1", "::1", "localhost"}:
                 for local_alias in ("127.0.0.1", "localhost"):
                     protected_hosts.add((local_alias, protected_port))
+                protected_local_roots.add(("archivebox.localhost", protected_port))
+            elif protected_host == "archivebox.localhost":
+                protected_local_roots.add((protected_host, protected_port))
             parts = protected_host.split(".", 1)
             if len(parts) == 2 and (parts[0] in protected_subdomains or parts[0].startswith("snap-")):
                 protected_roots.add((parts[1], protected_port))
@@ -862,15 +872,15 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             if host == protected_host and (protected_port is None or port == protected_port):
                 return True
 
-        if config.USES_SUBDOMAIN_ROUTING:
-            for protected_root, protected_port in protected_roots:
-                if protected_port is not None and port != protected_port:
-                    continue
-                if not protected_root or not host.endswith(f".{protected_root}"):
-                    continue
-                subdomain = host[: -(len(protected_root) + 1)]
-                if subdomain in protected_subdomains or subdomain.startswith("snap-"):
-                    return True
+        role_roots = protected_roots if config.USES_SUBDOMAIN_ROUTING else protected_local_roots
+        for protected_root, protected_port in role_roots:
+            if protected_port is not None and port != protected_port:
+                continue
+            if not protected_root or not host.endswith(f".{protected_root}"):
+                continue
+            subdomain = host[: -(len(protected_root) + 1)]
+            if subdomain in protected_subdomains or subdomain.startswith("snap-"):
+                return True
 
         return False
 
@@ -899,9 +909,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         from archivebox.core.permissions import PERMISSIONS_PUBLIC, PERMISSIONS_VALUES, normalize_permissions
 
         if permission not in PERMISSIONS_VALUES:
-            if self.crawl_id:
-                if not crawl_permissions:
-                    crawl_permissions = Crawl.objects.filter(pk=self.crawl_id).values_list("permissions", flat=True).first()
+            if self.crawl_id and not crawl_permissions:
+                crawl_permissions = Crawl.objects.filter(pk=self.crawl_id).values_list("permissions", flat=True).first()
             config["PERMISSIONS"] = normalize_permissions(
                 crawl_permissions,
                 default=PERMISSIONS_PUBLIC,
@@ -940,9 +949,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                 crawl_config_for_save = crawl_row.get("config") or {}
                 crawl_permissions_for_save = crawl_row.get("permissions")
 
-        if self.ensure_permissions_config(crawl_permissions=crawl_permissions_for_save):
-            if update_fields is not None:
-                kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))
+        if self.ensure_permissions_config(crawl_permissions=crawl_permissions_for_save) and update_fields is not None:
+            kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))
 
         if validate_url_field:
             self.validate_url_for_archiving(config=crawl_config_for_save if self.crawl_id else None)
@@ -1037,21 +1045,6 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             transaction.on_commit(lambda: self._cleanup_old_migration_dir(old_dir, new_dir))
             delattr(self, "_pending_fs_migration_cleanup")
 
-        # if is_new:
-        #     from archivebox.misc.logging_util import log_worker_event
-        #     log_worker_event(
-        #         worker_type='DB',
-        #         event='Created Snapshot',
-        #         indent_level=2,
-        #         url=self.url,
-        #         metadata={
-        #             'id': str(self.id),
-        #             'crawl_id': str(self.crawl_id),
-        #             'depth': self.depth,
-        #             'status': self.status,
-        #         },
-        #     )
-
     # =========================================================================
     # Filesystem Migration Methods
     # =========================================================================
@@ -1061,19 +1054,24 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         """Get current ArchiveBox filesystem layout version."""
         return "0.9.4"
 
+    _FS_VERSION_MIGRATION_PATHS: ClassVar[dict[str, str]] = {
+        "0.7.0": "0.9.0",
+        "0.8.0": "0.9.0",
+        "0.8.5": "0.9.0",
+        "0.9.0": "0.9.4",
+        "0.9.1": "0.9.4",
+        "0.9.2": "0.9.4",
+        "0.9.3": "0.9.4",
+    }
+
     @property
     def fs_migration_needed(self) -> bool:
         """Check if snapshot needs filesystem migration"""
         return self.fs_version != self._fs_current_version()
 
     def _fs_next_version(self, version: str) -> str:
-        """Get next version in migration chain (0.7/0.8 had same layout, only 0.8→0.9 migration needed)"""
-        # Treat 0.7.0 and 0.8.0 as equivalent (both used archive/{timestamp})
-        if version in ("0.7.0", "0.8.0"):
-            return "0.9.0"
-        if version in ("0.9.0", "0.9.1", "0.9.2", "0.9.3"):
-            return "0.9.4"
-        return self._fs_current_version()
+        """Get the next declared version in the filesystem migration chain."""
+        return self._FS_VERSION_MIGRATION_PATHS.get(version, self._fs_current_version())
 
     @staticmethod
     def is_legacy_archive_dir(path: Path) -> bool:
@@ -1119,6 +1117,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             migrations = {
                 ("0.7.0", "0.9.0"): self._fs_migrate_from_0_7_0_to_0_9_0,
                 ("0.8.0", "0.9.0"): self._fs_migrate_from_0_8_0_to_0_9_0,
+                ("0.8.5", "0.9.0"): self._fs_migrate_from_0_8_0_to_0_9_0,
                 ("0.9.0", "0.9.4"): self._fs_migrate_from_0_9_0_to_0_9_4,
                 ("0.9.1", "0.9.4"): self._fs_migrate_from_0_9_0_to_0_9_4,
                 ("0.9.2", "0.9.4"): self._fs_migrate_from_0_9_0_to_0_9_4,
@@ -1128,7 +1127,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             migration = migrations.get((current, next_ver))
             if migration is None:
                 raise ValueError(f"No filesystem migration path from {current} to {next_ver}")
-            cleanup = migration(source_dir=source_dir, config=runtime_config)
+            cleanup = migration(source_dir=source_dir, config=runtime_config) or cleanup
 
             current = next_ver
             source_dir = None
@@ -1154,6 +1153,13 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             old_crawl_dir.rename(crawl_dir)
         return cleanup
 
+    def hydrate_archiveresult_output_metadata(self, snapshot_dir: Path | None = None) -> int:
+        """Populate missing ArchiveResult file metadata from existing outputs."""
+        hydrated = 0
+        for result in self.archiveresult_set.filter(output_files={}).iterator():
+            hydrated += int(result.update_output_metadata_from_filesystem(snapshot_dir=snapshot_dir))
+        return hydrated
+
     def _fs_migrate_legacy_to_0_9_0(
         self,
         source_dir: Path | None = None,
@@ -1176,11 +1182,20 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             return None
 
         if old_dir.is_symlink():
+            try:
+                points_to_target = new_dir.exists() and old_dir.resolve() == new_dir.resolve()
+            except OSError:
+                points_to_target = False
+            if not points_to_target:
+                raise SnapshotMigrationError(f"Legacy output symlink does not point to the expected target: {old_dir}")
+            self.convert_index_json_to_jsonl(output_dir=new_dir)
+            self.hydrate_archiveresult_output_metadata(snapshot_dir=new_dir)
             return None
 
         if not old_dir.exists():
             if new_dir.exists():
                 self.convert_index_json_to_jsonl(output_dir=new_dir)
+                self.hydrate_archiveresult_output_metadata(snapshot_dir=new_dir)
                 return None
             return None
 
@@ -1188,51 +1203,57 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             new_dir.parent.mkdir(parents=True, exist_ok=True)
             try:
                 old_dir.rename(new_dir)
-                self.convert_index_json_to_jsonl(output_dir=new_dir)
-                return (old_dir, new_dir)
             except OSError:
                 pass
+            else:
+                self.convert_index_json_to_jsonl(output_dir=new_dir)
+                self.hydrate_archiveresult_output_metadata(snapshot_dir=new_dir)
+                return (old_dir, new_dir)
 
-        new_dir.mkdir(parents=True, exist_ok=True)
+        def copy_file_without_overwriting(source: str, destination: str):
+            destination_path = Path(destination)
+            if os.path.lexists(destination_path):
+                if not destination_path.is_symlink() and destination_path.is_file() and filecmp.cmp(source, destination, shallow=False):
+                    return destination
+                raise SnapshotMigrationError(f"Migration would overwrite a different output: {destination_path}")
+            return shutil.copy2(source, destination)
 
-        # Copy all files idempotently. If a previous attempt already converted
-        # index.json to index.jsonl, recopying index.json is harmless; conversion
-        # below removes it again after ensuring index.jsonl exists.
-        for old_file in old_dir.rglob("*"):
-            if not old_file.is_file():
+        # copytree preserves unknown directories and symlinks. Remove only
+        # already-copied identical symlinks so interrupted migrations can retry.
+        for source in old_dir.rglob("*"):
+            if not source.is_symlink():
                 continue
+            destination = new_dir / source.relative_to(old_dir)
+            if destination.is_symlink() and destination.readlink() == source.readlink():
+                destination.unlink()
+            elif os.path.lexists(destination):
+                raise SnapshotMigrationError(f"Migration would overwrite a different output: {destination}")
 
-            rel_path = old_file.relative_to(old_dir)
-            new_file = new_dir / rel_path
+        shutil.copytree(
+            old_dir,
+            new_dir,
+            copy_function=copy_file_without_overwriting,
+            dirs_exist_ok=True,
+            symlinks=True,
+        )
 
-            # Skip if already copied
-            if new_file.exists():
-                if new_file.stat().st_size == old_file.stat().st_size and filecmp.cmp(old_file, new_file, shallow=False):
-                    continue
-
-            new_file.parent.mkdir(parents=True, exist_ok=True)
-            shutil.copy2(old_file, new_file)
-
-        # Verify all copied
-        old_files = {f.relative_to(old_dir): f.stat().st_size for f in old_dir.rglob("*") if f.is_file()}
-        new_files = {f.relative_to(new_dir): f.stat().st_size for f in new_dir.rglob("*") if f.is_file()}
-
-        if old_files.keys() != new_files.keys():
-            missing = old_files.keys() - new_files.keys()
-            missing.discard(Path(CONSTANTS.JSON_INDEX_FILENAME))
-            if missing:
-                raise Exception(f"Migration incomplete: missing {missing}")
-
-        for rel_path, old_size in old_files.items():
-            if rel_path == Path(CONSTANTS.JSON_INDEX_FILENAME):
-                continue
-            if new_files.get(rel_path) != old_size:
-                raise Exception(f"Migration incomplete: size mismatch for {rel_path}")
-            if not filecmp.cmp(old_dir / rel_path, new_dir / rel_path, shallow=False):
-                raise Exception(f"Migration incomplete: content mismatch for {rel_path}")
+        # Verify every source entry before the old tree is eligible for cleanup.
+        for source in old_dir.rglob("*"):
+            destination = new_dir / source.relative_to(old_dir)
+            if source.is_symlink():
+                copied = destination.is_symlink() and destination.readlink() == source.readlink()
+            elif source.is_dir():
+                copied = destination.is_dir() and not destination.is_symlink()
+            elif source.is_file():
+                copied = destination.is_file() and not destination.is_symlink() and filecmp.cmp(source, destination, shallow=False)
+            else:
+                raise SnapshotMigrationError(f"Migration cannot safely copy special output: {source}")
+            if not copied:
+                raise SnapshotMigrationError(f"Migration incomplete: {source.relative_to(old_dir)}")
 
         # Convert index.json to index.jsonl in the new directory.
         self.convert_index_json_to_jsonl(output_dir=new_dir)
+        self.hydrate_archiveresult_output_metadata(snapshot_dir=new_dir)
 
         return (old_dir, new_dir)
 
@@ -1240,14 +1261,19 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         """
         Delete old directory and create symlink after successful migration.
         """
-        import shutil
         import logging
+        import shutil
+
+        from archivebox.config.permissions import SudoPermission
 
         # Delete old directory
         if old_dir.exists() and not old_dir.is_symlink():
             try:
-                shutil.rmtree(old_dir)
-            except Exception as e:
+                # Root-launched commands retain a saved root EUID after dropping
+                # privileges, which is needed for legacy trees owned by old UIDs.
+                with SudoPermission(uid=0, fallback=True):
+                    shutil.rmtree(old_dir)
+            except OSError as e:
                 logging.getLogger("archivebox.migration").warning(
                     f"Could not remove old migration directory {old_dir}: {e}",
                 )
@@ -1261,7 +1287,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         if not symlink_path.exists():
             try:
                 symlink_path.symlink_to(new_dir, target_is_directory=True)
-            except Exception as e:
+            except OSError as e:
                 logging.getLogger("archivebox.migration").warning(
                     f"Could not create symlink from {symlink_path} to {new_dir}: {e}",
                 )
@@ -1297,7 +1323,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                 return parsed.scheme
             else:
                 return "unknown"
-        except Exception:
+        except (TypeError, ValueError):
             return "unknown"
 
     def get_storage_path_for_version(self, version: str) -> Path:
@@ -1308,7 +1334,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         0.7.x/0.8.x: archive/{timestamp}
         0.9.x: archive/users/{username}/snapshots/YYYYMMDD/{domain}/{uuid}/
         """
-        if version in ("0.7.0", "0.8.0"):
+        if version in ("0.7.0", "0.8.0", "0.8.5"):
             return CONSTANTS.ARCHIVE_DIR / self.timestamp
 
         elif version in ("0.9.0", "0.9.1", "0.9.2", "0.9.3", "0.9.4", "1.0.0"):
@@ -1355,7 +1381,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                         break
             except OSError:
                 pass
-        elif json_path.exists():
+        if data is None and json_path.exists():
             try:
                 with open(json_path) as f:
                     data = json.load(f)
@@ -1454,7 +1480,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                         break
             except OSError:
                 pass
-        elif json_path.exists():
+        if data is None and json_path.exists():
             try:
                 with open(json_path) as f:
                     data = json.load(f)
@@ -1737,6 +1763,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         """Create ArchiveResult if not already in DB."""
         from dateutil import parser
         from django.db import transaction
+
         from archivebox.machine.models import Machine, Process
 
         # Support both old 'extractor' and new 'plugin' keys for backwards compat
@@ -1924,11 +1951,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         """
         from archivebox.machine.models import Process
         from archivebox.misc.jsonl import (
-            TYPE_SNAPSHOT,
             TYPE_ARCHIVERESULT,
-            TYPE_BINARYREQUEST,
             TYPE_BINARY,
+            TYPE_BINARYREQUEST,
             TYPE_PROCESS,
+            TYPE_SNAPSHOT,
         )
 
         output_dir = Path(output_dir) if output_dir is not None else Path(self.output_dir)
@@ -1961,7 +1988,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         """
         Convert index.json to index.jsonl format.
 
-        Reads existing index.json, creates index.jsonl, and removes index.json.
+        Reads existing index.json and creates index.jsonl while preserving the
+        original JSON byte-for-byte for unknown legacy metadata.
         Returns True if conversion was performed, False if no conversion needed.
         """
         import json
@@ -1970,9 +1998,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         json_path = output_dir / CONSTANTS.JSON_INDEX_FILENAME
         jsonl_path = output_dir / CONSTANTS.JSONL_INDEX_FILENAME
 
-        # Skip if already converted or no json file exists
+        # Skip if already converted or no json file exists. Keep a divergent
+        # legacy JSON index intact instead of silently discarding its metadata.
         if jsonl_path.exists():
-            json_path.unlink(missing_ok=True)
             return False
         if not json_path.exists():
             return False
@@ -2055,12 +2083,6 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             f.write("".join(json.dumps(record) + "\n" for record in records))
         os.replace(tmp_jsonl_path, jsonl_path)
 
-        # Remove old index.json after successful conversion
-        try:
-            json_path.unlink()
-        except OSError:
-            pass
-
         return True
 
     # =========================================================================
@@ -2074,10 +2096,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
         Used by: archivebox update (when encountering invalid directories)
         """
-        from datetime import datetime
         import shutil
 
-        invalid_dir = CONSTANTS.DATA_DIR / "invalid" / datetime.now().strftime("%Y%m%d")
+        invalid_dir = CONSTANTS.DATA_DIR / "invalid" / datetime.now(UTC).strftime("%Y%m%d")
         invalid_dir.mkdir(parents=True, exist_ok=True)
 
         dest = invalid_dir / snapshot_dir.name
@@ -2088,8 +2109,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
         try:
             shutil.move(str(snapshot_dir), str(dest))
-        except Exception:
-            pass
+        except OSError:
+            return
 
     @classmethod
     def find_and_merge_duplicates(cls) -> int:
@@ -2113,8 +2134,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                 try:
                     cls._merge_snapshots(snapshots)
                     merged += 1
-                except Exception:
-                    pass
+                except OSError:
+                    continue
 
         return merged
 
@@ -2149,8 +2170,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
                 try:
                     shutil.rmtree(dup_dir)
-                except Exception:
-                    pass
+                except OSError:
+                    continue
 
             # Merge tags
             for tag in dup.tags.all():
@@ -2215,6 +2236,29 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                 running = int(progress_stats.get("running") or 0)
                 completed = succeeded + failed + skipped + noresults
                 percent = int((completed / total * 100) if total > 0 else 0)
+                successful_plugins = sorted(self.__dict__.get("_icons_archive_results") or ())
+                visible_plugins = successful_plugins[:8]
+                successful_icons = format_html(
+                    '<div class="snapshot-successful-plugin-icons" style="display:flex; flex-wrap:wrap; gap:2px; margin-top:4px;">{}</div>',
+                    mark_safe(
+                        "".join(
+                            str(format_html('<span title="{}">{}</span>', plugin, mark_safe(get_plugin_icon(plugin))))
+                            for plugin in visible_plugins
+                            if str(get_plugin_icon(plugin)).strip()
+                        )
+                        + (
+                            str(
+                                format_html(
+                                    '<span title="{} more successful plugins">+{}</span>',
+                                    len(successful_plugins) - 8,
+                                    len(successful_plugins) - 8,
+                                ),
+                            )
+                            if len(successful_plugins) > 8
+                            else ""
+                        ),
+                    ),
+                )
                 return format_html(
                     '<div class="snapshot-files-progress" title="{} of {} hooks complete" style="min-width: 96px;">'
                     '<div style="display: flex; align-items: center; gap: 6px; margin-bottom: 4px;">'
@@ -2227,6 +2271,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                     '<div style="font-size: 10px; color: #94a3b8; margin-top: 2px;">'
                     "✓{} ✗{} ⏳{}"
                     "</div>"
+                    "{}"
                     "</div>",
                     completed,
                     total,
@@ -2236,6 +2281,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                     succeeded,
                     failed,
                     running,
+                    successful_icons,
                 )
 
             precomputed_archive_results = self.__dict__.get("_icons_archive_results")
@@ -2378,7 +2424,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
         try:
             data = json.loads(hashes_path.read_text(encoding="utf-8"))
-        except Exception:
+        except (json.JSONDecodeError, OSError, TypeError, ValueError):
             return {}
 
         index: dict[str, dict[str, Any]] = {}
@@ -2539,7 +2585,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         output_dir = Path(self.output_dir).resolve()
         try:
             rel_users_path = output_dir.relative_to(CONSTANTS.USERS_DIR)
-        except Exception:
+        except ValueError:
             rel_users_path = None
 
         if rel_users_path:
@@ -2556,7 +2602,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
         try:
             rel_path = output_dir.relative_to(CONSTANTS.DATA_DIR)
-        except Exception:
+        except ValueError:
             return self.legacy_archive_path
 
         parts = rel_path.parts
@@ -2621,7 +2667,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         Clean up background ArchiveResult hooks and empty results.
 
         Called by the state machine when entering the 'sealed' state.
-        Deletes empty ArchiveResults after the abx-dl cleanup phase has finished.
+        Reconcile late background outputs and hydrate result metadata.
         """
         # Clean up .pid files from output directory.
         output_dir = Path(self.output_dir)
@@ -2638,16 +2684,11 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
         else:
             return
 
-        # Delete ArchiveResults that produced no output files
-        empty_ars = self.archiveresult_set.filter(
-            output_files={},  # No output files
-        ).filter(
-            status__in=ArchiveResult.FINAL_STATES,  # Only delete finished ones
-        )
+        self.hydrate_archiveresult_output_metadata(snapshot_dir=output_dir)
 
-        if empty_ars.exists():
-            deleted_count, _ = empty_ars.delete()
-            rprint(f"[yellow]🗑️  Deleted {deleted_count} empty ArchiveResults for {self.url}[/yellow]")
+        # Keep empty historical rows. ArchiveResults for the same plugin share
+        # one output directory, so deleting any row can remove another row's
+        # files, including log-only and otherwise unrecognized outputs.
 
     def to_json(self) -> dict:
         """
@@ -2696,7 +2737,9 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             Snapshot instance or None
         """
         import re
+
         from django.utils import timezone
+
         from archivebox.base_models.models import get_or_create_system_user_pk
 
         config = get_config()
@@ -2721,9 +2764,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                         continue
 
                     # Special parsing for date fields
-                    if field_name in ("bookmarked_at", "retry_at", "created_at", "modified_at"):
-                        if value and isinstance(value, str):
-                            value = parse_date(value)
+                    if field_name in ("bookmarked_at", "retry_at", "created_at", "modified_at") and value and isinstance(value, str):
+                        value = parse_date(value)
 
                     # Update field if value is provided and different
                     if value is not None and getattr(snapshot, field_name) != value:
@@ -2767,8 +2809,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
                 crawl = parent_snapshot.crawl
             else:
                 # Auto-create a single-URL crawl
-                from archivebox.crawls.models import Crawl
                 from archivebox.config import CONSTANTS
+                from archivebox.crawls.models import Crawl
 
                 timestamp_str = timezone.now().strftime("%Y-%m-%d__%H-%M-%S")
                 sources_file = CONSTANTS.SOURCES_DIR / f"{timestamp_str}__auto_crawl.txt"
@@ -2885,8 +2927,8 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             return []
 
         if hooks is None:
-            from archivebox.plugins.hooks import discover_hooks
             from archivebox.config.common import get_config
+            from archivebox.plugins.hooks import discover_hooks
 
             # Compatibility path for direct model callers. The runner passes its
             # abx-dl hook inventory explicitly so queued rows match execution.
@@ -2985,17 +3027,13 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
     def retry_failed_archiveresults(self) -> int:
         """
-        Reset failed/skipped ArchiveResults to queued for retry.
+        Reset failed ArchiveResults to queued for retry.
 
         Returns count of ArchiveResults reset.
         """
         retryable_results = ArchiveResult.objects.filter(
             snapshot=self,
-            status__in=[
-                ArchiveResult.StatusChoices.FAILED,
-                ArchiveResult.StatusChoices.SKIPPED,
-                ArchiveResult.StatusChoices.NORESULTS,
-            ],
+            status=ArchiveResult.StatusChoices.FAILED,
         )
         legacy_result_count = retryable_results.filter(hook_name="").count()
         now = timezone.now()
@@ -3136,9 +3174,10 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
 
     def latest_outputs(self, status: str | None = None) -> dict[str, Any]:
         """Get the latest output that each plugin produced"""
-        from archivebox.plugins.discovery import get_plugins
         from django.db.models import Q
 
+        from archivebox.plugins.discovery import get_plugins
+
         latest: dict[str, Any] = {}
         for plugin in get_plugins():
             results = self.archiveresult_set.filter(plugin=plugin)
@@ -3397,6 +3436,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
     def write_html_details(self, out_dir: Path | str | None = None) -> None:
         """Write HTML detail page for this snapshot to its output directory"""
         from django.template.loader import render_to_string
+
         from archivebox.core.widgets import TagEditorWidget
         from archivebox.misc.logging_util import printable_filesize
 
@@ -3436,7 +3476,7 @@ class Snapshot(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelW
             "snapshot": self,
             "title": htmldecode(self.resolved_title or (self.base_url if is_archived else TITLE_LOADING_MSG)),
             "url_str": htmldecode(urldecode(self.base_url)),
-            "archive_url": urlencode(f"warc/{self.timestamp}" or (self.domain if is_archived else "")) or "about:blank",
+            "archive_url": urlencode(f"warc/{self.timestamp}") or "about:blank",
             "extension": self.extension or "html",
             "tags": self.tags_str() or "untagged",
             "size": printable_filesize(output_size) if output_size else "pending",
@@ -3610,9 +3650,12 @@ class SnapshotMachine(BaseStateMachine):
 
     @paused.enter
     def enter_paused(self):
-        self.snapshot.update_and_requeue(
-            retry_at=RETRY_AT_MAX,
-            status=Snapshot.StatusChoices.PAUSED,
+        self.snapshot.safe_update(
+            {
+                "retry_at": RETRY_AT_MAX,
+                "status": Snapshot.StatusChoices.PAUSED,
+            },
+            extra_filter={"status__in": Snapshot.RUNNABLE_STATES},
         )
 
     @started.enter
@@ -3870,12 +3913,12 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
         app_label = "core"
         verbose_name = "Archive Result"
         verbose_name_plural = "Archive Results Log"
-        indexes = [
+        indexes: ClassVar[list[models.Index]] = [
             models.Index(fields=["snapshot", "status"], name="archiveresult_snap_status_idx"),
             models.Index(fields=["status", "snapshot"], name="archiveresult_status_snap_idx"),
             models.Index(fields=["-start_ts", "-id"], name="archiveresult_start_idx"),
         ]
-        constraints = [
+        constraints: ClassVar[list[models.BaseConstraint]] = [
             models.UniqueConstraint(fields=["snapshot", "plugin", "hook_name"], name="unique_archiveresult_per_snapshot_hook"),
         ]
 
@@ -4062,21 +4105,6 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
         if is_new or update_fields is None or "status" in update_fields or "snapshot" in update_fields or "snapshot_id" in update_fields:
             transaction.on_commit(type(self).clear_majority_status_cache)
 
-        # if is_new:
-        #     from archivebox.misc.logging_util import log_worker_event
-        #     log_worker_event(
-        #         worker_type='DB',
-        #         event='Created ArchiveResult',
-        #         indent_level=3,
-        #         plugin=self.plugin,
-        #         metadata={
-        #             'id': str(self.id),
-        #             'snapshot_id': str(self.snapshot_id),
-        #             'snapshot_url': str(self.snapshot.url)[:64],
-        #             'status': self.status,
-        #         },
-        #     )
-
     def delete(self, *args, **kwargs):
         snapshot_id = self.snapshot_id
         deleted = super().delete(*args, **kwargs)
@@ -4252,6 +4280,7 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
 
     def update_output_metadata_from_filesystem(self, snapshot_dir: Path | None = None, save: bool = True) -> bool:
         from collections import defaultdict
+
         from abx_dl.output_files import guess_mimetype
 
         if self.plugin == "title":
@@ -4606,10 +4635,12 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
         """
         from collections import defaultdict
         from pathlib import Path
-        from django.utils import timezone
+
         from abx_dl.output_files import guess_mimetype
-        from archivebox.plugins.hooks import process_hook_records, extract_records_from_process
+        from django.utils import timezone
+
         from archivebox.machine.models import Process
+        from archivebox.plugins.hooks import extract_records_from_process, process_hook_records
 
         plugin_dir = Path(self.pwd) if self.pwd else None
         if not plugin_dir or not plugin_dir.exists():
@@ -4661,8 +4692,8 @@ class ArchiveResult(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithNotes):
                 from archivebox.plugins.hooks import is_background_hook
 
                 is_background = bool(self.hook_name and is_background_hook(self.hook_name))
-            except Exception:
-                pass
+            except (ImportError, TypeError, ValueError):
+                is_background = False
 
             if is_background or (process and process.exit_code == 0):
                 self.status = self.StatusChoices.SKIPPED

archivebox/core/routes_util.py

@@ -6,8 +6,7 @@ from urllib.parse import urlparse
 
 from django.utils.http import url_has_allowed_host_and_scheme
 
-from archivebox.config.common import get_config
-
+from archivebox.config.common import get_config, get_request_config
 
 _SNAPSHOT_ID_RE = re.compile(r"^[0-9a-fA-F-]{8,36}$")
 _SNAPSHOT_SUBDOMAIN_RE = re.compile(r"^snap-(?P<suffix>[0-9a-fA-F]{12})$")
@@ -153,7 +152,7 @@ def _root_host_from_listen(config: dict[str, Any] | None = None, **config_kwargs
 
 
 def get_base_url(request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     override = _normalize_base_url(config.BASE_URL)
     if override:
         return override
@@ -281,9 +280,7 @@ def host_matches(request_host: str, target_host: str) -> bool:
     target_host_only, target_port = split_host_port(target_host)
     if req_host != target_host_only:
         return False
-    if target_port and req_port and target_port != req_port:
-        return False
-    return True
+    return not (target_port and req_port and target_port != req_port)
 
 
 def is_allowed_archivebox_redirect_url(url: str | None, request=None, config: dict[str, Any] | None = None) -> bool:
@@ -352,28 +349,28 @@ def _build_base_url_for_host(host: str, request=None, config: dict[str, Any] | N
 
 
 def get_admin_base_url(request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return get_base_url(request=request, config=config)
     return _build_base_url_for_host(_build_base_host("admin", request=request, config=config), request=request, config=config)
 
 
 def get_web_base_url(request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return get_base_url(request=request, config=config)
     return _build_base_url_for_host(_build_base_host("web", request=request, config=config), request=request, config=config)
 
 
 def get_api_base_url(request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return get_base_url(request=request, config=config)
     return _build_base_url_for_host(_build_base_host("api", request=request, config=config), request=request, config=config)
 
 
 def get_snapshot_base_url(snapshot_id: str, request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return _build_url(get_web_base_url(request=request, config=config), f"/snapshot/{str(snapshot_id).replace('-', '')}")
     return _build_base_url_for_host(
@@ -384,7 +381,7 @@ def get_snapshot_base_url(snapshot_id: str, request=None, config: dict[str, Any]
 
 
 def get_original_base_url(domain: str, request=None, config: dict[str, Any] | None = None, **config_kwargs: Any) -> str:
-    config = config or get_config(**config_kwargs)
+    config = config or (get_request_config(request) if request is not None else get_config(**config_kwargs))
     if not config.USES_SUBDOMAIN_ROUTING:
         return _build_url(get_web_base_url(request=request, config=config), f"/original/{domain}")
     return _build_base_url_for_host(_build_base_host(domain, request=request, config=config), request=request, config=config)

archivebox/core/settings.py

@@ -1,28 +1,33 @@
 __package__ = "archivebox.core"
 
+import importlib
+import inspect
+import logging
 import os
 import sys
-import inspect
-import importlib
-
+from importlib.util import find_spec
 from pathlib import Path
 
 from django.conf.locale.en import formats as en_formats  # type: ignore
 
 import archivebox
-
-from archivebox.config.constants import CONSTANTS
 from archivebox.config.common import get_config
-from archivebox.core.routes_util import get_api_base_url, get_admin_base_url, get_base_url, normalize_base_url
+from archivebox.config.constants import CONSTANTS
+from archivebox.core.routes_util import get_admin_base_url, get_api_base_url, get_base_url, normalize_base_url
+
+# DATABASE_ENGINE config selects the backend (sqlite by default); the
+# sqlite-vs-postgres helpers live in archivebox.misc.db.
+from archivebox.misc.db import is_postgres, postgres_db_params
+
 from .settings_logging import SETTINGS_LOGGING
 
-
 IS_MIGRATING = "makemigrations" in sys.argv[:3] or "migrate" in sys.argv[:3]
-IS_TESTING = "test" in sys.argv[:3] or "PYTEST_CURRENT_TEST" in os.environ
+IS_TESTING = "test" in sys.argv[:3]
 IS_SHELL = "shell" in sys.argv[:3] or "shell_plus" in sys.argv[:3]
 IS_GETTING_VERSION_OR_HELP = "version" in sys.argv or "help" in sys.argv or "--version" in sys.argv or "--help" in sys.argv
 CONFIG = get_config()
 PACKAGE_DIR = CONSTANTS.PACKAGE_DIR
+logger = logging.getLogger(__name__)
 
 ################################################################################
 ### ArchiveBox Plugin Settings
@@ -72,7 +77,7 @@ INSTALLED_APPS = [
     "archivebox.crawls",  # handles Crawl and CrawlSchedule models and management (depends on core)
     "archivebox.progressmonitor",  # live progress endpoint and admin monitor template
     "archivebox.api",  # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.
-    "abx_plugins.plugins.opencode",
+    "archivebox.opencode",
     # 3rd-party apps from PyPI that need to be loaded last
     "admin_data_views",  # handles rendering some convenient automatic read-only views of data in Django admin
     "django_extensions",  # provides Django Debug Toolbar (and other non-debug helpers)
@@ -103,7 +108,6 @@ MIDDLEWARE = [
 ### Authentication Settings
 ################################################################################
 
-# AUTH_USER_MODEL = 'auth.User'   # cannot be easily changed unfortunately
 
 AUTHENTICATION_BACKENDS = [
     "django.contrib.auth.backends.RemoteUserBackend",
@@ -181,23 +185,11 @@ TEMPLATES_DIR_NAME = "templates"
 CUSTOM_TEMPLATES_ENABLED = os.path.isdir(CONSTANTS.CUSTOM_TEMPLATES_DIR) and os.access(CONSTANTS.CUSTOM_TEMPLATES_DIR, os.R_OK)
 STATICFILES_DIRS = [
     *([str(CONSTANTS.CUSTOM_TEMPLATES_DIR / "static")] if CUSTOM_TEMPLATES_ENABLED else []),
-    # *[
-    #     str(plugin_dir / 'static')
-    #     for plugin_dir in PLUGIN_DIRS.values()
-    #     if (plugin_dir / 'static').is_dir()
-    # ],
-    # Additional static file dirs from plugins
     str(PACKAGE_DIR / TEMPLATES_DIR_NAME / "static"),
 ]
 
 TEMPLATE_DIRS = [
     *([str(CONSTANTS.CUSTOM_TEMPLATES_DIR)] if CUSTOM_TEMPLATES_ENABLED else []),
-    # *[
-    #     str(plugin_dir / 'templates')
-    #     for plugin_dir in PLUGIN_DIRS.values()
-    #     if (plugin_dir / 'templates').is_dir()
-    # ],
-    # Additional template dirs from plugins
     str(PACKAGE_DIR / TEMPLATES_DIR_NAME / "core"),
     str(PACKAGE_DIR / TEMPLATES_DIR_NAME / "admin"),
     str(PACKAGE_DIR / TEMPLATES_DIR_NAME),
@@ -225,10 +217,6 @@ TEMPLATES = [
 ### External Service Settings
 ################################################################################
 
-# CACHE_DB_FILENAME = 'cache.sqlite3'
-# CACHE_DB_PATH = CONSTANTS.CACHE_DIR / CACHE_DB_FILENAME
-# CACHE_DB_TABLE = 'django_cache'
-
 DATABASE_NAME = CONFIG.DATABASE_NAME
 SQLITE_JOURNAL_MODE = CONFIG.SQLITE_JOURNAL_MODE
 SQLITE_MMAP_SIZE = CONFIG.SQLITE_MMAP_SIZE
@@ -262,20 +250,21 @@ SQLITE_CONNECTION_OPTIONS = {
     },
 }
 
-DATABASES = {
-    "default": {
-        "NAME": DATABASE_NAME,
-        **SQLITE_CONNECTION_OPTIONS,
-    },
-    # "filestore": {
-    #     "NAME": CONSTANTS.FILESTORE_DATABASE_FILE,
-    #     **SQLITE_CONNECTION_OPTIONS,
-    # },
-    # 'cache': {
-    #     'NAME': CACHE_DB_PATH,
-    #     **SQLITE_CONNECTION_OPTIONS,
-    # },
-}
+if is_postgres():
+    DATABASES = {
+        "default": {
+            "ENGINE": "django.db.backends.postgresql",
+            **postgres_db_params(),
+            "OPTIONS": {"connect_timeout": 10},
+        },
+    }
+else:
+    DATABASES = {
+        "default": {
+            "NAME": DATABASE_NAME,
+            **SQLITE_CONNECTION_OPTIONS,
+        },
+    }
 MIGRATION_MODULES = {"signal_webhooks": None}
 
 # Django requires DEFAULT_AUTO_FIELD to subclass AutoField (BigAutoField, SmallAutoField, etc.)
@@ -285,43 +274,10 @@ MIGRATION_MODULES = {"signal_webhooks": None}
 DEFAULT_AUTO_FIELD = "django.db.models.BigAutoField"
 
 
-# class FilestoreDBRouter:
-#     """
-#     A router to store all the File models in the filestore.sqlite3 database.
-#     This data just mirrors what is in the file system, so we want to keep it in a separate database
-#     from the main index database to avoid contention.
-#     """
-
-#     route_app_labels = {"filestore"}
-#     db_name = "filestore"
-
-#     def db_for_read(self, model, **hints):
-#         if model._meta.app_label in self.route_app_labels:
-#             return self.db_name
-#         return 'default'
-
-#     def db_for_write(self, model, **hints):
-#         if model._meta.app_label in self.route_app_labels:
-#             return self.db_name
-#         return 'default'
-
-#     def allow_relation(self, obj1, obj2, **hints):
-#         if obj1._meta.app_label in self.route_app_labels or obj2._meta.app_label in self.route_app_labels:
-#             return obj1._meta.app_label == obj2._meta.app_label
-#         return None
-
-#     def allow_migrate(self, db, app_label, model_name=None, **hints):
-#         if app_label in self.route_app_labels:
-#             return db == self.db_name
-#         return db == "default"
-
 DATABASE_ROUTERS = []
 
 CACHES = {
     "default": {"BACKEND": "django.core.cache.backends.locmem.LocMemCache"},
-    # 'sqlite': {'BACKEND': 'django.core.cache.backends.db.DatabaseCache', 'LOCATION': 'cache'},
-    # 'dummy': {'BACKEND': 'django.core.cache.backends.dummy.DummyCache'},
-    # 'filebased': {"BACKEND": "django.core.cache.backends.filebased.FileBasedCache", "LOCATION": CACHE_DIR / 'cache_filebased'},
 }
 
 EMAIL_BACKEND = "django.core.mail.backends.console.EmailBackend"
@@ -379,20 +335,26 @@ try:
 
     _persisted_keys = _BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
     _secret_persisted = bool((_persisted_keys.get("SECRET_KEY") or "").strip())
-except Exception:
+except (OSError, RuntimeError, ValueError):
     _secret_persisted = True  # err on the side of NOT touching disk
 if not _secret_persisted:
     try:
         from archivebox.config.collection import write_config_file
 
         write_config_file({"SECRET_KEY": SECRET_KEY})
-    except Exception:
+    except (OSError, RuntimeError, ValueError):
         # Read-only mount, missing data dir, mid-init race — fall back to the
         # in-memory random key. The user will get logged out on the next boot
         # but the server still comes up.
-        pass
+        logger.debug("Unable to persist generated SECRET_KEY", exc_info=True)
 
 ALLOWED_HOSTS = [host.strip() for host in CONFIG.ALLOWED_HOSTS.split(",") if host.strip()]
+if (CONFIG.SERVER_SECURITY_MODE == "auto" or CONFIG.USES_SUBDOMAIN_ROUTING) and "*" not in ALLOWED_HOSTS:
+    # ArchiveBox owns the effective host policy at request time in auto and
+    # subdomain modes: canonical hosts keep the normal app surface, while
+    # alternate hosts are downgraded/rejected by get_request_config()/middleware.
+    # Django has to admit the Host header first for that existing boundary to run.
+    ALLOWED_HOSTS.append("*")
 CSRF_TRUSTED_ORIGINS = list({origin.strip() for origin in CONFIG.CSRF_TRUSTED_ORIGINS.split(",") if origin.strip()})
 
 admin_base_url = normalize_base_url(get_admin_base_url())
@@ -424,15 +386,15 @@ SECURE_REFERRER_POLICY = "strict-origin-when-cross-origin"
 # behind a TLS-terminating proxy/tunnel (the bundled traefik/cloudflared profiles,
 # or your own caddy/traefik/nginx) where the proxy -> archivebox hop is plain HTTP, so
 # request.is_secure() / request.scheme would otherwise report http. Honour the
-# proxy's X-Forwarded-Proto so request-derived schemes are correct, and mark the
-# admin session + CSRF cookies Secure so auth cookies are never sent in cleartext.
+# proxy's X-Forwarded-Proto so first-run URL detection and CSRF origin checks are
+# correct. Mark auth cookies Secure once the saved BASE_URL confirms HTTPS.
 # Derived from the RESOLVED base URL's scheme — no separate flag to keep in sync.
 # get_base_url() also covers deployments that only set CSRF_TRUSTED_ORIGINS (the
 # implicit-BASE_URL fallback used on 0.7.x->0.9.x upgrades), so HTTPS hardening
-# isn't lost until BASE_URL is migrated. A plain-http base (e.g. local
-# http://archivebox.localhost:8000) keeps the defaults below.
+# isn't lost until BASE_URL is migrated. An explicit plain-http base (e.g. local
+# http://archivebox.localhost:8000) disables proxy HTTPS handling.
 BASE_URL_IS_HTTPS = get_base_url(config=CONFIG).strip().lower().startswith("https://")
-if BASE_URL_IS_HTTPS:
+if BASE_URL_IS_HTTPS or not CONFIG.BASE_URL:
     SECURE_PROXY_SSL_HEADER = ("HTTP_X_FORWARDED_PROTO", "https")
 
 CSRF_COOKIE_SECURE = BASE_URL_IS_HTTPS
@@ -642,8 +604,7 @@ if DEBUG:
 # https://github.com/bensi94/Django-Requests-Tracker (improved version of django-debug-toolbar)
 # Must delete archivebox/templates/admin to use because it relies on some things we override
 # visit /__requests_tracker__/ to access
-DEBUG_REQUESTS_TRACKER = True
-DEBUG_REQUESTS_TRACKER = DEBUG_REQUESTS_TRACKER and DEBUG
+DEBUG_REQUESTS_TRACKER = DEBUG and find_spec("requests_tracker") is not None
 if DEBUG_REQUESTS_TRACKER:
     import requests_tracker
 
@@ -666,16 +627,3 @@ if DEBUG_REQUESTS_TRACKER:
             r"^SELECT .* FROM django_migrations WHERE app = 'auth'",
         ),
     }
-
-# # https://docs.pydantic.dev/logfire/integrations/django/ (similar to DataDog / NewRelic / etc.)
-# DEBUG_LOGFIRE = False
-# DEBUG_LOGFIRE = DEBUG_LOGFIRE and os.access(DATA_DIR / '.logfire', os.W_OK) and (DATA_DIR / '.logfire').is_dir()
-
-
-# For usage with https://www.jetadmin.io/integrations/django
-# INSTALLED_APPS += ['jet_django']
-# JET_PROJECT = 'archivebox'
-# JET_TOKEN = 'some-api-token-here'
-
-
-# import ipdb; ipdb.set_trace()

archivebox/core/settings_logging.py

@@ -130,8 +130,6 @@ if os.access(LOGS_DIR, os.W_OK) and LOGS_DIR.is_dir():
     ERROR_LOG = LOGS_DIR / "errors.log"
 else:
     # historically too many edge cases here around creating log dir w/ correct permissions early on
-    # if there's an issue on startup, we trash the log and let user figure it out via stdout/stderr
-    # print(f'[!] WARNING: data/logs dir does not exist. Logging to temp file: {ERROR_LOG}')
     pass
 
 LOG_LEVEL_DATABASE = "WARNING"  # change to DEBUG to log all SQL queries

archivebox/core/setup_wizard.py

@@ -0,0 +1,75 @@
+from typing import Any
+
+from archivebox.core.routes_util import canonical_base_host_for_request, get_base_url, split_host_port
+
+
+def _request_scheme(request) -> str:
+    forwarded_scheme = (request.META.get("HTTP_X_FORWARDED_PROTO") or "").split(",", 1)[0].strip().lower()
+    if forwarded_scheme in {"http", "https"}:
+        return forwarded_scheme
+    return request.scheme or "http"
+
+
+def get_setup_wizard_context(request, config) -> dict[str, Any]:
+    """Build the first-run setup context when ``BASE_URL`` is unset."""
+    context = {
+        "mode": "unconfigured",
+        "canonical_host": "",
+        "display_host": "",
+        "suggested_base_url": "",
+        "machine_admin_url": "",
+        "can_configure": False,
+        "public_index": config.PUBLIC_INDEX,
+        "public_add_view": config.PUBLIC_ADD_VIEW,
+        "permissions": config.PERMISSIONS,
+    }
+    if request is None:
+        return context
+
+    scheme = _request_scheme(request)
+    canonical_host = canonical_base_host_for_request(request.get_host() or "")
+    display_hostname, display_port = split_host_port(canonical_host)
+    display_host = canonical_host
+    if (scheme, display_port) in (("http", "80"), ("https", "443")):
+        display_host = display_hostname
+
+    user = request.user
+    is_superuser = bool(user and user.is_authenticated and user.is_superuser)
+    machine_admin_url = ""
+    if is_superuser:
+        try:
+            from archivebox.machine.models import Machine
+
+            machine_admin_url = f"/admin/machine/machine/{Machine.current().id}/change/"
+        except (ImportError, RuntimeError, TypeError, ValueError):
+            machine_admin_url = ""
+
+    context.update(
+        canonical_host=canonical_host,
+        display_host=display_host,
+        suggested_base_url=f"{scheme}://{canonical_host}" if canonical_host else "",
+        machine_admin_url=machine_admin_url,
+        can_configure=is_superuser,
+    )
+    return context
+
+
+def get_base_url_mismatch_context(request, config) -> dict[str, str] | None:
+    """Describe a request origin that does not resolve to the configured base."""
+    if request is None or not config.BASE_URL:
+        return None
+
+    scheme = _request_scheme(request)
+    browser_url = f"{scheme}://{request.get_host()}".rstrip("/")
+    browser_base_url = browser_url
+    if config.USES_SUBDOMAIN_ROUTING:
+        browser_base_url = f"{scheme}://{canonical_base_host_for_request(request.get_host())}".rstrip("/")
+    configured_base_url = get_base_url(config=config).rstrip("/")
+    if browser_base_url.lower() == configured_base_url.lower():
+        return None
+
+    return {
+        "mode": "base_url_mismatch",
+        "browser_url": browser_url,
+        "configured_base_url": configured_base_url,
+    }

archivebox/core/takeover_util.py

@@ -127,10 +127,9 @@ def command_owns_foreground_runner(command, *, data_dir: str | Path) -> bool:
 
 def runtime_stack_component_label(*, owner=None, data_dir: str | Path) -> str:
     try:
-        from archivebox.config.common import get_config
         from archivebox.workers.supervisord_util import active_supervisord_runtime_components
 
-        components = active_supervisord_runtime_components(config=get_config())
+        components = active_supervisord_runtime_components()
     except Exception:
         components = []
 

archivebox/core/templatetags/core_tags.py

@@ -1,36 +1,33 @@
+import os
+from pathlib import Path
 from typing import Any
 
+from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
 from django import template
 from django.contrib.admin.templatetags.base import InclusionAdminNode
-from django.utils.safestring import mark_safe
-from django.utils.html import escape
 from django.templatetags.static import static
 from django.utils import timezone
+from django.utils.html import escape
+from django.utils.safestring import mark_safe
 from django.utils.text import Truncator
 
-from pathlib import Path
-
-from abx_plugins.plugins.archivewebpage.replay_preview import is_replay_target as is_archivewebpage_replay_target
-
+from archivebox.core.routes_util import (
+    build_snapshot_url,
+    get_admin_base_url,
+    get_snapshot_base_url,
+    get_web_base_url,
+)
+from archivebox.core.setup_wizard import get_base_url_mismatch_context, get_setup_wizard_context
 from archivebox.plugins.discovery import (
     get_plugin_icon,
-    get_plugin_template,
     get_plugin_name,
+    get_plugin_template,
 )
-from archivebox.core.routes_util import (
-    canonical_base_host_for_request,
-    get_admin_base_url,
-    get_web_base_url,
-    get_snapshot_base_url,
-    build_snapshot_url,
-)
-
 
 register = template.Library()
 
 _TEXT_PREVIEW_EXTS = (".json", ".jsonl", ".txt", ".csv", ".tsv", ".xml", ".yml", ".yaml", ".md", ".log")
 _IMAGE_PREVIEW_EXTS = (".png", ".jpg", ".jpeg", ".gif", ".webp", ".bmp", ".ico", ".avif")
-_MHTML_PREVIEW_EXTS = (".mhtml", ".mht")
 
 _MEDIA_FILE_EXTS = {
     ".mp4",
@@ -94,15 +91,15 @@ def _coerce_output_file_size(value: Any) -> int | None:
 def _count_media_files(result) -> int:
     try:
         output_files = _normalize_output_files(result.output_files or {})
-    except Exception:
+    except (AttributeError, TypeError, ValueError):
         output_files = {}
 
     if output_files:
-        return sum(1 for path in output_files.keys() if Path(path).suffix.lower() in _MEDIA_FILE_EXTS)
+        return sum(1 for path in output_files if Path(path).suffix.lower() in _MEDIA_FILE_EXTS)
 
     try:
         plugin_dir = Path(result.snapshot_dir) / result.plugin
-    except Exception:
+    except (AttributeError, TypeError, ValueError):
         return 0
 
     if not plugin_dir.exists():
@@ -126,7 +123,7 @@ def _list_media_files(result) -> list[dict]:
     media_files: list[dict] = []
     try:
         plugin_dir = Path(result.snapshot_dir) / result.plugin
-    except Exception:
+    except (AttributeError, TypeError, ValueError):
         return media_files
 
     output_files = _normalize_output_files(result.output_files or {})
@@ -184,7 +181,7 @@ def _resolve_snapshot_output_file(snapshot_dir: str | Path | None, raw_output_pa
         snap_dir = Path(snapshot_dir).resolve()
         if snap_dir not in output_file.parents and output_file != snap_dir:
             return None
-    except Exception:
+    except (OSError, RuntimeError, TypeError, ValueError):
         return None
 
     if output_file.exists() and output_file.is_file():
@@ -209,17 +206,17 @@ def _build_snapshot_files_url(snapshot_id: str, request=None, config=None) -> st
     return build_snapshot_url(str(snapshot_id), "/?files=1", request=request, config=config)
 
 
-def _build_snapshot_preview_url(snapshot_id: str, path: str = "", request=None, config=None) -> str:
+def _build_snapshot_preview_url(snapshot_id: str, path: str = "", request=None, config=None, plugin: str = "") -> str:
     if path == "about:blank":
         return path
     if _is_root_snapshot_output_path(path):
         return _build_snapshot_files_url(snapshot_id, request=request, config=config)
     url = build_snapshot_url(str(snapshot_id), path, request=request, config=config)
+    path_parts = Path(path).parts
+    plugin = get_plugin_name(plugin) if plugin else (path_parts[0] if len(path_parts) > 1 else "")
+    has_plugin_preview = bool(plugin and get_plugin_template(plugin, "full", fallback=False))
     if not (
-        _is_text_preview_path(path)
-        or _is_image_preview_path(path)
-        or (path or "").lower().endswith(_MHTML_PREVIEW_EXTS)
-        or is_archivewebpage_replay_target(path or "")
+        _is_text_preview_path(path) or _is_image_preview_path(path) or has_plugin_preview or is_archivewebpage_replay_target(path or "")
     ):
         return url
     separator = "&" if "?" in url else "?"
@@ -268,7 +265,7 @@ def _render_text_file_preview(snapshot_dir: str | Path | None, raw_output_path:
         lines = text.splitlines()[:6]
         snippet = "\n".join(lines)
         return _render_text_preview(plugin, icon_html, snippet)
-    except Exception:
+    except (OSError, UnicodeDecodeError, ValueError):
         return None
 
 
@@ -281,12 +278,12 @@ def split(value, separator: str = ","):
 def index(value, position):
     try:
         return value[int(position)]
-    except Exception:
+    except (IndexError, TypeError, ValueError):
         return None
 
 
 @register.filter
-def file_size(num_bytes: int | float) -> str:
+def file_size(num_bytes: float) -> str:
     for count in ["Bytes", "KB", "MB", "GB"]:
         if num_bytes > -1024.0 and num_bytes < 1024.0:
             return f"{num_bytes:3.1f} {count}"
@@ -359,7 +356,7 @@ def _machine_health_stats() -> dict:
         from archivebox.machine.detect import get_host_stats
 
         stats = get_host_stats() or {}
-    except Exception:
+    except (ImportError, RuntimeError, TypeError, ValueError):
         stats = {}
 
     _health_cache["checked_at"] = now
@@ -375,13 +372,15 @@ def system_warnings_banner(context):
     1. ``mode="unconfigured"``— ``BASE_URL`` is empty. Security/correctness
        issue: until it's pinned, generated URLs can echo any Host the client
        sends, and admin/web/api routing has no canonical anchor.
-    2. ``mode="unsafe"``      — ``SERVER_SECURITY_MODE`` is a non-subdomain
-       mode. Archived pages share an origin with privileged routes.
-    3. ``mode="low_disk"``    — ``DATA_DIR`` has <1 GiB free; new archive
+    2. ``mode="base_url_mismatch"`` — the browser reached ArchiveBox through
+       an origin that differs from the configured canonical ``BASE_URL``.
+    3. ``mode="unsafe"``      — ``SERVER_SECURITY_MODE`` explicitly enables
+       a lower-security replay mode.
+    4. ``mode="low_disk"``    — ``DATA_DIR`` has <1 GiB free; new archive
        jobs will start failing on ENOSPC.
-    4. ``mode="high_memory"`` — virtual memory utilization at/above 95%; the
+    5. ``mode="high_memory"`` — virtual memory utilization at/above 95%; the
        host is one OOM-kill from a crash.
-    5. ``mode="high_load"``   — 15-minute load average exceeds 3 × CPU count
+    6. ``mode="high_load"``   — 15-minute load average exceeds 3 × CPU count
        (the kernel's own sustained-load EMA, so no rolling buffer of ours is
        needed).
 
@@ -397,8 +396,11 @@ def system_warnings_banner(context):
         config = get_config(resolve_plugins=False)
 
     if not config.BASE_URL:
-        return _unconfigured_banner_context(context.get("request"))
-    if not config.USES_SUBDOMAIN_ROUTING:
+        return get_setup_wizard_context(context.get("request"), config)
+    mismatch = get_base_url_mismatch_context(context.get("request"), config)
+    if mismatch:
+        return mismatch
+    if config.IS_LOWER_SECURITY_MODE:
         return {"mode": "unsafe"}
 
     stats = _machine_health_stats()
@@ -417,6 +419,8 @@ def system_warnings_banner(context):
     # "sustained for 15min" and the kernel already maintains that EMA.
     load_15 = cpu_load[2] if isinstance(cpu_load, (list, tuple)) and len(cpu_load) >= 3 else None
     if isinstance(load_15, (int, float)) and load_15 > _HIGH_LOAD_MULTIPLE * cpu_count:
+        if os.environ.get("UI_SCREENSHOT_HIDE_HIGH_LOAD_WARNING") == "1":
+            return {"mode": ""}
         return {
             "mode": "high_load",
             "load_15": f"{load_15:.2f}",
@@ -427,49 +431,6 @@ def system_warnings_banner(context):
     return {"mode": ""}
 
 
-def _unconfigured_banner_context(request) -> dict:
-    """Build the banner payload for the unset-BASE_URL case.
-
-    Always returns ``mode="unconfigured"`` — the user explicitly asked for
-    the banner to render whenever ``BASE_URL`` is empty, regardless of
-    whether the request host happens to match a CSRF-derived value. The
-    ``suggested_base_url`` is derived from the current request when one is
-    available so the user can copy/paste the right value straight into
-    their config.
-    """
-    if request is None:
-        return {
-            "mode": "unconfigured",
-            "actual_host": "",
-            "suggested_base_url": "",
-            "machine_admin_url": "",
-        }
-    scheme = request.scheme or "http"
-    actual_full_host = request.get_host() or ""
-    canonical_host = canonical_base_host_for_request(actual_full_host)
-    # Suggest the wildcard form ``http://*.<host>`` so the value lands in the
-    # operator's clipboard already aligned with subdomain routing. The config
-    # parser strips the leading ``*.`` so users can paste it verbatim.
-    suggested_base_url = f"{scheme}://*.{canonical_host}" if canonical_host else ""
-    user = request.user
-    is_superuser = bool(user and user.is_authenticated and user.is_superuser)
-    machine_admin_url = ""
-    if is_superuser:
-        try:
-            from archivebox.machine.models import Machine
-
-            machine = Machine.current()
-            machine_admin_url = f"/admin/machine/machine/{machine.id}/change/"
-        except Exception:
-            machine_admin_url = ""
-    return {
-        "mode": "unconfigured",
-        "actual_host": actual_full_host,
-        "suggested_base_url": suggested_base_url,
-        "machine_admin_url": machine_admin_url,
-    }
-
-
 @register.simple_tag(takes_context=True)
 def url_replace(context, **kwargs):
     dict_ = context["request"].GET.copy()
@@ -581,7 +542,7 @@ def snapshot_index_row(context, link) -> str:
     url = getattr(link, "url", "") or ""
     title = getattr(link, "title", "") or ""
     is_pending = status in {"queued", "started", "backoff"}
-    title_text = title or ("Loading..." if is_pending else url)
+    title_text = title or url
     tags_str = link.tags_str() if callable(getattr(link, "tags_str", None)) else getattr(link, "tags_str", "")
     tag_html = "".join(f'<span class="snapshot-tag">{escape(tag)}</span>' for tag in (tags_str or "").split(",") if tag)
     if tag_html:
@@ -682,6 +643,7 @@ def snapshot_index_row(context, link) -> str:
         <a href="{escape(url)}" class="snapshot-url" title="{escape(url)}" target="_blank" rel="noopener noreferrer">
             {escape(url)}
         </a>
+        <span class="snapshot-mobile-saved">Saved {escape(date_text)} at {escape(time_text)}</span>
     </td>
     <td class="snapshot-tags-cell">
         {tag_cell}
@@ -706,9 +668,16 @@ def snapshot_index_row(context, link) -> str:
 
 
 @register.simple_tag(takes_context=True)
-def snapshot_preview_url(context, snapshot, path: str = "") -> str:
+def snapshot_preview_url(context, snapshot, path: str = "", result=None) -> str:
     snapshot_id = _snapshot_id(snapshot)
-    return _build_snapshot_preview_url(str(snapshot_id), path, request=context.get("request"), config=context.get("CONFIG"))
+    plugin = getattr(result, "plugin", "") if result else ""
+    return _build_snapshot_preview_url(
+        str(snapshot_id),
+        path,
+        request=context.get("request"),
+        config=context.get("CONFIG"),
+        plugin=plugin,
+    )
 
 
 @register.simple_tag
@@ -789,8 +758,8 @@ def plugin_card(context, result) -> str:
             # Only return non-empty content (strip whitespace to check)
             if rendered.strip():
                 return mark_safe(rendered)
-    except Exception:
-        pass
+    except (template.TemplateSyntaxError, AttributeError, TypeError, ValueError):
+        rendered = ""
 
     if force_text_preview:
         preview = _render_text_file_preview(result.snapshot_dir, raw_output_path, plugin, icon_html)
@@ -865,7 +834,7 @@ def plugin_full(context, result) -> str:
         if rendered.strip():
             return mark_safe(rendered)
         return ""
-    except Exception:
+    except (template.TemplateSyntaxError, AttributeError, TypeError, ValueError):
         return ""
 
 

archivebox/core/urls.py

@@ -28,13 +28,7 @@ from archivebox.core.views import (
 )
 from archivebox.progressmonitor.views import live_progress_view
 from archivebox.search.views import public_snapshot_search_stream_view
-from abx_plugins.plugins.opencode.views import opencode_proxy_view
-
-
-# GLOBAL_CONTEXT doesn't work as-is, disabled for now: https://github.com/ArchiveBox/ArchiveBox/discussions/1306
-# from archivebox.config import VERSION, VERSIONS_AVAILABLE, CAN_UPGRADE
-# GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': VERSIONS_AVAILABLE, 'CAN_UPGRADE': CAN_UPGRADE}
-
+from archivebox.opencode.views import opencode_proxy_view
 
 CONFIG = get_config()
 DEBUG = CONFIG.DEBUG or ("--debug" in sys.argv)
@@ -44,7 +38,7 @@ urlpatterns = [
     path("robots.txt", static.serve, {"document_root": CONSTANTS.STATIC_DIR, "path": "robots.txt"}),
     path("favicon.ico", static.serve, {"document_root": CONSTANTS.STATIC_DIR, "path": "favicon.ico"}),
     path("docs/", RedirectView.as_view(url="https://github.com/ArchiveBox/ArchiveBox/wiki"), name="Docs"),
-    re_path(r"^admin/agent/?(?=$|opencode)", include("abx_plugins.plugins.opencode.urls")),
+    re_path(r"^admin/agent/?(?=$|opencode)", include("archivebox.opencode.urls")),
     re_path(r"^(?P<path>assets/.*)$", opencode_proxy_view, name="opencode-assets"),
     path("public/search-stream/", public_snapshot_search_stream_view, name="public-search-stream"),
     path("public/", PublicIndexView.as_view(), name="public-index"),
@@ -86,7 +80,6 @@ urlpatterns = [
     path("api/", include("archivebox.api.urls"), name="api"),
     path("health/", HealthCheckView.as_view(), name="healthcheck"),
     path("error/", lambda request: _raise_test_error(request)),
-    # path('jet_api/', include('jet_django.urls')),  Enable to use https://www.jetadmin.io/integrations/django
     path("index.html", RedirectView.as_view(url="/")),
     path("", HomepageView.as_view(), name="Home"),
 ]
@@ -101,36 +94,3 @@ if getattr(settings, "DEBUG_TOOLBAR", False):
 
 if getattr(settings, "DEBUG_REQUESTS_TRACKER", False) and find_spec("requests_tracker"):
     urlpatterns += [path("__requests_tracker__/", include("requests_tracker.urls"))]
-
-
-# # Proposed FUTURE URLs spec
-# path('',                 HomepageView)
-# path('/add',             AddView)
-# path('/public',          PublicIndexView)
-# path('/snapshot/:slug',  SnapshotView)
-
-# path('/admin',           admin.site.urls)
-# path('/accounts',        django.contrib.auth.urls)
-
-# # Proposed REST API spec
-# # :slugs can be uuid, short_uuid, or any of the unique index_fields
-# path('api/v1/'),
-# path('api/v1/core/'                      [GET])
-# path('api/v1/core/snapshot/',            [GET, POST, PUT]),
-# path('api/v1/core/snapshot/:slug',       [GET, PATCH, DELETE]),
-# path('api/v1/core/archiveresult',        [GET, POST, PUT]),
-# path('api/v1/core/archiveresult/:slug',  [GET, PATCH, DELETE]),
-# path('api/v1/core/tag/',                 [GET, POST, PUT]),
-# path('api/v1/core/tag/:slug',            [GET, PATCH, DELETE]),
-
-# path('api/v1/cli/',                      [GET])
-# path('api/v1/cli/{add,list,config,...}', [POST]),  # pass query as kwargs directly to `run_subcommand` and return stdout, stderr, exitcode
-
-# path('api/v1/extractors/',                    [GET])
-# path('api/v1/extractors/:extractor/',         [GET]),
-# path('api/v1/extractors/:extractor/:func',    [GET, POST]),  # pass query as args directly to chosen function
-
-# future, just an idea:
-# path('api/v1/scheduler/',                [GET])
-# path('api/v1/scheduler/task/',           [GET, POST, PUT]),
-# path('api/v1/scheduler/task/:slug',      [GET, PATCH, DELETE]),

archivebox/core/views.py

@@ -3,71 +3,50 @@ __package__ = "archivebox.core"
 import json
 import os
 import posixpath
-from glob import glob, escape
-from django.utils import timezone
-from typing import cast
+from glob import escape, glob
 from pathlib import Path
+from typing import ClassVar, cast
 from urllib.parse import quote, urlparse
 
-from django.shortcuts import render, redirect
-from django.http import HttpRequest, HttpResponse, Http404, HttpResponseForbidden, QueryDict
-from django.utils.html import format_html, format_html_join
-from django.utils.safestring import mark_safe
-from django.views import View
-from django.views.generic.list import ListView
-from django.views.generic import FormView
-from django.db.models import Case, IntegerField, Q, Value, When
-from django.core.paginator import InvalidPage
-from django.contrib import messages
+from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
+from admin_data_views.typing import ItemContext, SectionData, TableContext
+from admin_data_views.utils import ItemLink, render_with_item_view, render_with_table_view
+from django import template
 from django.conf import settings
+from django.contrib import messages
 from django.contrib.auth import HASH_SESSION_KEY, SESSION_KEY, get_user_model
 from django.contrib.auth.mixins import UserPassesTestMixin
 from django.contrib.sessions.models import Session
 from django.core import signing
-from django.views.decorators.csrf import csrf_exempt
+from django.core.paginator import InvalidPage
+from django.db.models import Case, IntegerField, Q, Value, When
+from django.http import Http404, HttpRequest, HttpResponse, HttpResponseForbidden, QueryDict
+from django.shortcuts import redirect, render
+from django.utils import timezone
 from django.utils.decorators import method_decorator
-
-from admin_data_views.typing import TableContext, ItemContext, SectionData
-from admin_data_views.utils import render_with_table_view, render_with_item_view, ItemLink
-
-from abx_plugins.plugins.archivewebpage import replay_preview as archivewebpage_replay
+from django.utils.html import format_html, format_html_join
+from django.utils.safestring import mark_safe
+from django.views import View
+from django.views.decorators.csrf import csrf_exempt
+from django.views.generic import FormView
+from django.views.generic.list import ListView
 
 from archivebox.config import CONSTANTS, CONSTANTS_CONFIG, VERSION
 from archivebox.config.common import (
+    PLUGIN_CONFIG_SCHEMAS,
     SENSITIVE_CONFIG_VALUE_REDACTED,
+    _plugin_config_properties,
     find_config_default,
     find_config_section,
     find_config_source,
     find_config_type,
-    get_config,
     get_all_configs,
+    get_config,
     get_request_config,
-    _plugin_config_properties,
     redact_sensitive_config,
 )
-from archivebox.config.common import PLUGIN_CONFIG_SCHEMAS
 from archivebox.config.configset import BaseConfigSet
-from archivebox.misc.paginators import AcceleratedPaginator
-from archivebox.misc.util import (
-    base_url,
-    filter_queryset_by_uuid_substring,
-    htmldecode,
-    sanitize_html_text,
-    ts_to_date_str,
-    urldecode,
-    validate_url,
-    without_fragment,
-)
-from archivebox.misc.serve_static import serve_static_with_byterange_support
-from archivebox.misc.logging_util import printable_filesize
-from archivebox.search.config import (
-    get_search_mode,
-    get_search_mode_backend,
-    get_search_mode_base,
-    get_search_mode_options,
-)
-from archivebox.search.views import get_cached_public_search_state
-
+from archivebox.core.forms import AddLinkForm
 from archivebox.core.models import ArchiveResult, Snapshot, SnapshotTag
 from archivebox.core.permissions import (
     PERMISSIONS_PRIVATE,
@@ -90,12 +69,31 @@ from archivebox.core.routes_util import (
     get_web_host,
     host_matches,
 )
-from archivebox.core.forms import AddLinkForm
-from archivebox.plugins.forms import get_plugin_config_binary_urls
 from archivebox.crawls.models import Crawl
-from archivebox.plugins.discovery import discover_plugin_configs
+from archivebox.misc.logging_util import printable_filesize
+from archivebox.misc.paginators import AcceleratedPaginator
+from archivebox.misc.serve_static import serve_static_with_byterange_support
+from archivebox.misc.util import (
+    base_url,
+    filter_queryset_by_uuid_substring,
+    htmldecode,
+    sanitize_html_text,
+    ts_to_date_str,
+    urldecode,
+    validate_url,
+    without_fragment,
+)
+from archivebox.plugins.discovery import get_plugin_name, get_plugin_template
+from archivebox.plugins.forms import get_plugin_config_binary_urls
 from archivebox.plugins.views import get_config_definition_link
 from archivebox.progressmonitor.views import live_progress_view, progress_endpoint
+from archivebox.search.config import (
+    get_search_mode,
+    get_search_mode_backend,
+    get_search_mode_base,
+    get_search_mode_options,
+)
+from archivebox.search.views import get_cached_public_search_state
 
 
 def _files_index_target(snapshot: Snapshot, archivefile: str | None) -> str:
@@ -170,7 +168,7 @@ def _replay_payload_is_valid(payload: dict, snapshot: Snapshot) -> bool:
         user_id = str(session_data.get(SESSION_KEY) or "")
         auth_hash = str(session_data.get(HASH_SESSION_KEY) or "")
         user = get_user_model().objects.get(pk=user_id)
-    except Exception:
+    except (Session.DoesNotExist, get_user_model().DoesNotExist, KeyError, TypeError, ValueError):
         return False
     return (
         str(payload.get("user_id")) == user_id
@@ -277,12 +275,20 @@ class SnapshotView(View):
         """
 
         def _fragmentless_url_query(url: str) -> Q:
-            # Use a range comparison (url >= 'canonical#' AND url < 'canonical#\U0010ffff')
-            # instead of LIKE/__startswith — SQLite's case-insensitive LIKE bypasses the
-            # url index and forces a full-table scan over ~1M rows (~250ms). The range
-            # form lets SQLite use a MULTI-INDEX OR and stays under 1ms.
+            from archivebox.misc.db import is_postgres
+
             canonical = without_fragment(url)
-            return Q(url=canonical) | (Q(url__gte=f"{canonical}#") & Q(url__lt=f"{canonical}#\U0010ffff"))
+            if not is_postgres():
+                # Use a range comparison (url >= 'canonical#' AND url < 'canonical#\U0010ffff')
+                # instead of LIKE/__startswith — SQLite's case-insensitive LIKE bypasses the
+                # url index and forces a full-table scan over ~1M rows (~250ms). The range
+                # form lets SQLite use a MULTI-INDEX OR and stays under 1ms.
+                return Q(url=canonical) | (Q(url__gte=f"{canonical}#") & Q(url__lt=f"{canonical}#\U0010ffff"))
+            # On postgres the range trick is unsafe: linguistic (ICU/libc) collations
+            # don't compare '#'-suffixed strings bytewise, so the range can miss rows.
+            # startswith compiles to LIKE 'prefix%' with wildcards escaped, which is
+            # correct under any collation and uses the url pattern-ops index.
+            return Q(url=canonical) | Q(url__startswith=f"{canonical}#")
 
         normalized = without_fragment(path)
         if path.startswith(("http://", "https://")):
@@ -336,7 +342,11 @@ class SnapshotView(View):
             for out in snapshot.discover_outputs(include_filesystem_fallback=True)
             if (out.get("size") or 0) > 0 and out.get("name") not in hidden_card_plugins
         ]
-        archiveresults = {out["name"]: out for out in outputs}
+        archiveresults = {}
+        for output in outputs:
+            current = archiveresults.get(output["name"])
+            if current is None or (output.get("size") or 0) > (current.get("size") or 0):
+                archiveresults[output["name"]] = output
         hash_index = snapshot.hashes_index
         accounted_entries: set[str] = set()
         for output in outputs:
@@ -439,6 +449,7 @@ class SnapshotView(View):
             "id": str(snapshot.id),
             "snapshot_id": str(snapshot.id),
             "progress_endpoint": progress_endpoint("snapshot", snapshot.id),
+            "progress_auto_expand": snapshot_status in {"queued", "started", "paused"},
             "url": snapshot.url,
             "archive_path": snapshot.archive_path_from_db,
             "title": htmldecode(snapshot.resolved_title or (snapshot.base_url if is_archived else TITLE_LOADING_MSG)),
@@ -625,10 +636,7 @@ class SnapshotView(View):
             return SnapshotView.find_snapshots_for_url(slug)
 
         try:
-            try:
-                snapshot = direct_snapshots_queryset(request, _resolve_snapshots_for_slug(path)).get()
-            except Snapshot.DoesNotExist:
-                raise
+            snapshot = direct_snapshots_queryset(request, _resolve_snapshots_for_slug(path)).get()
         except Snapshot.DoesNotExist:
             return HttpResponse(
                 format_html(
@@ -820,21 +828,23 @@ def _safe_archive_relpath(path: str) -> str | None:
     return cleaned
 
 
-def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[str, bool]:
+def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[str, ArchiveResult | None]:
     """Resolve plugin-relative output paths through ArchiveResult.output_files."""
     parts = Path(rel_path).parts
     if len(parts) < 2:
-        return rel_path, False
+        return rel_path, None
 
     plugin = parts[0]
     plugin_relpath = posixpath.join(*parts[1:])
     result = (
         ArchiveResult.objects.filter(snapshot=snapshot, plugin=plugin, status=ArchiveResult.StatusChoices.SUCCEEDED)
-        .only("output_files")
+        .only("plugin", "output_files")
         .first()
     )
-    if not result or not result.output_files:
-        return rel_path, False
+    if not result:
+        return rel_path, None
+    if not result.output_files:
+        return rel_path, result
 
     output_files = result.output_files or {}
     for candidate in (plugin_relpath, rel_path):
@@ -842,10 +852,79 @@ def _resolve_archiveresult_relpath(snapshot: Snapshot, rel_path: str) -> tuple[s
         if not isinstance(file_info, dict):
             continue
         if file_info.get("root_relative"):
-            return candidate, True
-        return rel_path, True
+            return candidate, result
+        return rel_path, result
 
-    return rel_path, True
+    return rel_path, result
+
+
+def _plugin_full_preview_response(
+    request: HttpRequest,
+    snapshot: Snapshot,
+    rel_path: str,
+    result: ArchiveResult | None,
+) -> HttpResponse | None:
+    """Render an explicit plugin full template as a trusted preview wrapper."""
+    if not request.GET.get("preview"):
+        return None
+
+    path_parts = Path(rel_path).parts
+    plugin = get_plugin_name(result.plugin) if result else (path_parts[0] if len(path_parts) > 1 else "")
+    if not plugin:
+        return None
+
+    # ReplayWeb.page needs plugin-owned WACZ inspection and service-worker
+    # context, so it remains the one narrow preview exception below.
+    if plugin == "archivewebpage" and archivewebpage_replay.is_replay_target(rel_path):
+        return None
+
+    template_str = get_plugin_template(plugin, "full", fallback=False)
+    if not template_str:
+        return None
+
+    raw_query = request.GET.copy()
+    raw_query.pop("preview", None)
+    output_url = request.path
+    if raw_query:
+        output_url = f"{output_url}?{raw_query.urlencode()}"
+
+    rendered = (
+        template.Engine(debug=False)
+        .from_string(template_str)
+        .render(
+            template.Context(
+                {
+                    "result": result,
+                    "snapshot": snapshot,
+                    "output_path": output_url,
+                    "output_path_raw": rel_path,
+                    "plugin": plugin,
+                    "preview_base": f"{request.path.rsplit('/', 1)[0]}/",
+                },
+            ),
+        )
+    )
+    response = HttpResponse(rendered, content_type="text/html; charset=utf-8")
+    response.headers["Content-Disposition"] = f'inline; filename="{Path(rel_path).stem}.html"'
+    response.headers["X-Content-Type-Options"] = "nosniff"
+    response.headers["X-ArchiveBox-Security-Mode"] = request.archivebox_config.SERVER_SECURITY_MODE
+    response.headers["Referrer-Policy"] = "no-referrer"
+    response.headers["Content-Security-Policy"] = (
+        "default-src 'self' data: blob:; "
+        "script-src 'self' 'unsafe-inline' 'unsafe-eval' data: blob:; "
+        "style-src 'unsafe-inline' data: blob: 'self'; "
+        "connect-src 'self' data: blob:; "
+        "img-src 'self' data: blob:; "
+        "media-src 'self' data: blob:; "
+        "font-src 'self' data: blob:; "
+        "frame-src 'self' data: blob:; "
+        "worker-src 'self' blob:; "
+        "object-src 'none'; "
+        "base-uri 'none'; "
+        "form-action 'none'; "
+        "frame-ancestors 'self';"
+    )
+    return response
 
 
 def _coerce_sort_timestamp(value: str | float | None) -> float:
@@ -865,7 +944,7 @@ def _snapshot_sort_key(match_path: str, cache: dict[str, float]) -> tuple[float,
         idx = parts.index("snapshots")
         date_str = parts[idx + 1]
         snapshot_id = parts[idx + 3]
-    except Exception:
+    except (IndexError, ValueError):
         return (_coerce_sort_timestamp(date_str), match_path)
 
     if snapshot_id not in cache:
@@ -1041,7 +1120,11 @@ def _serve_snapshot_replay(request: HttpRequest, snapshot: Snapshot, path: str =
     if rel_path is None:
         raise Http404
 
-    rel_path, _is_known_plugin_output = _resolve_archiveresult_relpath(snapshot, rel_path)
+    rel_path, archive_result = _resolve_archiveresult_relpath(snapshot, rel_path)
+
+    plugin_preview = _plugin_full_preview_response(request, snapshot, rel_path, archive_result)
+    if plugin_preview is not None:
+        return plugin_preview
 
     try:
         return serve_static_with_byterange_support(
@@ -1165,7 +1248,7 @@ class OriginalDomainReplayView(View):
 class PublicIndexView(ListView):
     template_name = "public_index.html"
     model = Snapshot
-    ordering = ["-bookmarked_at", "-created_at"]
+    ordering: ClassVar[list[str]] = ["-bookmarked_at", "-created_at"]
     paginator_class = AcceleratedPaginator
     public_page_scan_chunk_size = 50
 
@@ -1424,7 +1507,6 @@ class AddView(UserPassesTestMixin, FormView):
         request_config = get_request_config(self.request, resolve_plugins=True)
         required_search_plugin = f"search_backend_{request_config.SEARCH_BACKEND_ENGINE}".strip()
         can_override_crawl_config = self._can_override_crawl_config()
-        plugin_configs = discover_plugin_configs() if can_override_crawl_config else {}
         public_persona_config_keys = {
             "CRAWL_MAX_CONCURRENT_SNAPSHOTS",
             "DELETE_AFTER",
@@ -1453,17 +1535,6 @@ class AddView(UserPassesTestMixin, FormView):
                 "binary_urls": binary_urls,
             }
         recent_personas = list(persona_queryset.order_by("-created_at", "name")[:5])
-        plugin_dependency_map = {}
-        if can_override_crawl_config:
-            plugin_dependency_map = {
-                plugin_name: [
-                    str(required_plugin).strip()
-                    for required_plugin in (schema.get("required_plugins") or [])
-                    if str(required_plugin).strip()
-                ]
-                for plugin_name, schema in plugin_configs.items()
-                if isinstance(schema.get("required_plugins"), list) and schema.get("required_plugins")
-            }
         return {
             **context,
             "title": "Create Crawl",
@@ -1473,7 +1544,6 @@ class AddView(UserPassesTestMixin, FormView):
             "VERSION": VERSION,
             "FOOTER_INFO": request_config.FOOTER_INFO,
             "required_search_plugin": required_search_plugin,
-            "plugin_dependency_map_json": json.dumps(plugin_dependency_map, sort_keys=True),
             "persona_config_map_json": json.dumps(persona_config_map, sort_keys=True, default=str),
             "recent_personas": recent_personas,
             "can_override_crawl_config": can_override_crawl_config,
@@ -1640,13 +1710,16 @@ class WebAddView(AddView):
                 return redirect(f"/{snapshot.url_path}")
 
         request_host = (request.get_host() or "").lower()
-        if request.user.is_authenticated and not get_request_config(request).PUBLIC_ADD_VIEW and host_matches(request_host, get_web_host()):
+        request_config = get_request_config(request)
+        web_host = get_web_host(config=request_config)
+        admin_host = get_admin_host(config=request_config)
+        if request.user.is_authenticated and not request_config.PUBLIC_ADD_VIEW and host_matches(request_host, web_host):
             return redirect(build_admin_url(request.get_full_path(), request=request))
 
         if not self.test_func():
-            if host_matches(request_host, get_web_host()):
+            if host_matches(request_host, web_host):
                 return redirect(build_admin_url(request.get_full_path(), request=request))
-            if host_matches(request_host, get_admin_host()):
+            if host_matches(request_host, admin_host):
                 next_url = quote(request.get_full_path(), safe="/:?=&")
                 return redirect(f"{build_admin_url('/admin/login/', request=request)}?next={next_url}")
             return HttpResponse(
@@ -1739,7 +1812,7 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
     }
 
     for section_id, section in reversed(list(CONFIGS.items())):
-        for key in dict(section).keys():
+        for key in dict(section):
             rows["Section"].append(section_id)  # section.replace('_', ' ').title().replace(' Config', '')
             rows["Key"].append(ItemLink(key, key=key))
             rows["Type"].append(format_html("<code>{}</code>", find_config_type(key)))
@@ -1760,11 +1833,9 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
                     find_config_default(key) or "See here...",
                 ),
             )
-            # rows['Documentation'].append(mark_safe(f'Wiki: <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#{key.lower()}">{key}</a>'))
-            # rows['Aliases'].append(', '.join(find_config_aliases(key)))
 
     section = "CONSTANT"
-    for key in CONSTANTS_CONFIG.keys():
+    for key in CONSTANTS_CONFIG:
         rows["Section"].append(section)  # section.replace('_', ' ').title().replace(' Config', '')
         rows["Key"].append(ItemLink(key, key=key))
         rows["Type"].append(format_html("<code>{}</code>", type(CONSTANTS_CONFIG[key]).__name__))
@@ -1777,8 +1848,6 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
                 find_config_default(key) or "See here...",
             ),
         )
-        # rows['Documentation'].append(mark_safe(f'Wiki: <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#{key.lower()}">{key}</a>'))
-        # rows['Aliases'].append('')
 
     return TableContext(
         title="Computed Configuration Values",
@@ -1834,7 +1903,6 @@ def live_config_value_view(request: HttpRequest, key: str, **kwargs) -> ItemCont
         ((color, source, value) for source, value, color in sources_info),
     )
 
-    # aliases = USER_CONFIG.get(key, {}).get("aliases", [])
     aliases = []
 
     if key in CONSTANTS_CONFIG:

archivebox/core/wsgi.py

@@ -11,6 +11,6 @@ import archivebox  # noqa
 from archivebox.config.django import setup_django
 from django.core.wsgi import get_wsgi_application
 
-setup_django(in_memory_db=False, check_db=True)
+setup_django(check_db=True)
 
 application = get_wsgi_application()

archivebox/crawls/admin.py

@@ -1,27 +1,25 @@
 __package__ = "archivebox.crawls"
 
-from copy import copy
 import json
+from copy import copy
+from typing import ClassVar
 from urllib.parse import urlencode, urlparse
 
 from django import forms
-from django.core.paginator import Paginator
-from django.http import JsonResponse, HttpRequest, HttpResponseBadRequest, HttpResponseNotAllowed
-from django.shortcuts import get_object_or_404, redirect
-from django.template.response import TemplateResponse
-from django.template.loader import render_to_string
-from django.urls import path, reverse
-from django.utils.html import escape, format_html, format_html_join
-from django.utils import timezone
-from django.utils.safestring import mark_safe
 from django.contrib import admin, messages
+from django.core.paginator import Paginator
 from django.db.models import Count, F, Q
-
-
+from django.http import HttpRequest, HttpResponseBadRequest, HttpResponseNotAllowed, JsonResponse
+from django.shortcuts import get_object_or_404, redirect
+from django.template.loader import render_to_string
+from django.template.response import TemplateResponse
+from django.urls import path, reverse
+from django.utils import timezone
+from django.utils.html import escape, format_html, format_html_join
+from django.utils.safestring import mark_safe
 from django_object_actions import action
 
 from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
-
 from archivebox.core.models import ArchiveResult, Snapshot
 from archivebox.core.permissions import (
     PERMISSIONS_CHOICES,
@@ -397,7 +395,7 @@ class CrawlAdminForm(forms.ModelForm):
     class Meta:
         model = Crawl
         fields = "__all__"
-        widgets = {
+        widgets: ClassVar[dict[str, forms.Widget]] = {
             "urls": forms.Textarea(
                 attrs={
                     "rows": 8,
@@ -704,16 +702,16 @@ class CrawlAdmin(ConfigEditorMixin, BaseModelAdmin):
     )
 
     list_filter = (MaxDepthListFilter, "schedule", "created_by", "status", "retry_at")
-    ordering = ["-created_at", "-retry_at"]
+    ordering = ("-created_at", "-retry_at")
     list_per_page = 50
-    actions = [
+    actions = (
         "pause_selected_crawls",
         "resume_selected_crawls",
         "seal_selected_crawls",
         "delete_selected_batched",
         "set_crawl_permissions",
-    ]
-    change_actions = ["recrawl"]
+    )
+    change_actions = ("recrawl",)
 
     def __init__(self, model, admin_site):
         super().__init__(model, admin_site)
@@ -721,8 +719,8 @@ class CrawlAdmin(ConfigEditorMixin, BaseModelAdmin):
         self.stop_reason_cache = {}
 
     class Media:
-        css = {"all": ("admin/crawls/crawl_change.css",)}
-        js = ("admin/crawls/crawl_admin.js",)
+        css: ClassVar[dict[str, tuple[str, ...]]] = {"all": ("admin/crawls/crawl_change.css",)}
+        js: ClassVar[tuple[str, ...]] = ("admin/crawls/crawl_admin.js",)
 
     def changelist_view(self, request, extra_context=None):
         self.request = request
@@ -1219,6 +1217,7 @@ class CrawlScheduleAdmin(BaseModelAdmin):
     search_fields = ("id", "created_by__username", "label", "notes", "schedule_id", "template_id", "template__urls")
 
     readonly_fields = ("created_at", "modified_at", "crawls", "snapshots")
+    autocomplete_fields = ("template", "created_by")
 
     fieldsets = (
         (
@@ -1259,9 +1258,9 @@ class CrawlScheduleAdmin(BaseModelAdmin):
     )
 
     list_filter = ("created_by",)
-    ordering = ["-created_at"]
+    ordering = ("-created_at",)
     list_per_page = 100
-    actions = ["delete_selected"]
+    actions = ("delete_selected",)
 
     def get_queryset(self, request):
         self.request = request

archivebox/crawls/locks.py

@@ -0,0 +1,85 @@
+from __future__ import annotations
+
+import fcntl
+import os
+import threading
+from contextlib import contextmanager
+from pathlib import Path
+from typing import IO
+from collections.abc import Iterator
+
+from archivebox.config.constants import CONSTANTS
+
+
+class _LifecycleLockState:
+    def __init__(self) -> None:
+        self.thread_lock = threading.RLock()
+        self.users = 0
+        self.depth = 0
+        self.file: IO[str] | None = None
+
+
+_registry_lock = threading.Lock()
+_registry_pid = os.getpid()
+_registry: dict[str, _LifecycleLockState] = {}
+
+
+def crawl_lifecycle_lock_path(crawl_id: str) -> Path:
+    return CONSTANTS.DEFAULT_TMP_DIR / "crawl-locks" / f"{crawl_id}.lock"
+
+
+@contextmanager
+def _lifecycle_lock(key: str, lock_path: Path) -> Iterator[None]:
+    global _registry_pid
+
+    with _registry_lock:
+        if _registry_pid != os.getpid():
+            for inherited_state in _registry.values():
+                if inherited_state.file is not None:
+                    inherited_state.file.close()
+            _registry.clear()
+            _registry_pid = os.getpid()
+        state = _registry.setdefault(key, _LifecycleLockState())
+        state.users += 1
+
+    state.thread_lock.acquire()
+    try:
+        if state.depth == 0:
+            lock_path.parent.mkdir(parents=True, exist_ok=True)
+            state.file = lock_path.open("a+", encoding="utf-8")
+            try:
+                fcntl.flock(state.file.fileno(), fcntl.LOCK_EX)
+            except BaseException:
+                state.file.close()
+                state.file = None
+                raise
+        state.depth += 1
+        try:
+            yield
+        finally:
+            state.depth -= 1
+            if state.depth == 0 and state.file is not None:
+                fcntl.flock(state.file.fileno(), fcntl.LOCK_UN)
+                state.file.close()
+                state.file = None
+    finally:
+        state.thread_lock.release()
+        with _registry_lock:
+            state.users -= 1
+            if state.users == 0 and state.depth == 0 and _registry.get(key) is state:
+                _registry.pop(key, None)
+
+
+@contextmanager
+def crawl_lifecycle_lock(crawl_id: str) -> Iterator[None]:
+    key = str(crawl_id)
+    with _lifecycle_lock(f"crawl:{key}", crawl_lifecycle_lock_path(key)):
+        yield
+
+
+@contextmanager
+def binary_lifecycle_lock(binary_id: str) -> Iterator[None]:
+    key = str(binary_id)
+    lock_path = CONSTANTS.DEFAULT_TMP_DIR / "binary-locks" / f"{key}.lock"
+    with _lifecycle_lock(f"binary:{key}", lock_path):
+        yield

archivebox/crawls/migrations/0001_initial.py

@@ -10,19 +10,9 @@ from archivebox.uuid_compat import uuid7
 from archivebox.base_models.models import get_or_create_system_user_pk
 
 
-class Migration(migrations.Migration):
-    initial = True
-
-    dependencies = [
-        ("auth", "0012_alter_user_first_name_max_length"),
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-    ]
-
-    operations = [
-        migrations.SeparateDatabaseAndState(
-            database_operations=[
-                migrations.RunSQL(
-                    sql="""
+# Raw sqlite DDL kept EXACTLY as-is; executed only on sqlite via Django's
+# RunSQL (identical statement splitting → byte-for-byte identical behavior).
+INITIAL_SQL = """
                 -- Create crawls_crawlschedule table first (circular FK will be added later)
                 CREATE TABLE IF NOT EXISTS crawls_crawlschedule (
                     id TEXT PRIMARY KEY NOT NULL,
@@ -75,12 +65,60 @@ class Migration(migrations.Migration):
                 CREATE INDEX IF NOT EXISTS crawls_crawl_created_at_idx ON crawls_crawl(created_at);
                 CREATE INDEX IF NOT EXISTS crawls_crawl_created_by_id_idx ON crawls_crawl(created_by_id);
                 CREATE INDEX IF NOT EXISTS crawls_crawl_schedule_id_idx ON crawls_crawl(schedule_id);
-                    """,
-                    reverse_sql="""
+                    """
+
+INITIAL_REVERSE_SQL = """
                 DROP TABLE IF EXISTS crawls_crawl;
                 DROP TABLE IF EXISTS crawls_crawlschedule;
-                    """,
-                ),
+                    """
+
+
+def _run_sqlite_only_sql(apps, schema_editor):
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_forwards(
+        "crawls",
+        schema_editor,
+        None,
+        None,
+    )
+
+
+def _run_sqlite_only_sql_reverse(apps, schema_editor):
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_backwards(
+        "crawls",
+        schema_editor,
+        None,
+        None,
+    )
+
+
+def _pg_sync_schema(apps, schema_editor):
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(apps, schema_editor, "crawls", ["CrawlSchedule", "Crawl"])
+
+
+def _pg_drop_schema(apps, schema_editor):
+    from archivebox.misc.db import drop_models_on_postgres
+
+    drop_models_on_postgres(apps, schema_editor, "crawls", ["Crawl", "CrawlSchedule"])
+
+
+class Migration(migrations.Migration):
+    initial = True
+
+    dependencies = [
+        ("auth", "0012_alter_user_first_name_max_length"),
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.SeparateDatabaseAndState(
+            database_operations=[
+                migrations.RunPython(_run_sqlite_only_sql, reverse_code=_run_sqlite_only_sql_reverse),
             ],
             state_operations=[
                 migrations.CreateModel(
@@ -174,4 +212,9 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        # On non-sqlite backends the raw DDL above is skipped, so the real
+        # schema diverges from Django state. Rebuild both tables from the
+        # post-migration state (no-op on sqlite). Runs as a top-level op so
+        # ``apps`` reflects the full state including the circular template FK.
+        migrations.RunPython(_pg_sync_schema, reverse_code=_pg_drop_schema),
     ]

archivebox/crawls/migrations/0002_upgrade_from_0_8_6.py

@@ -6,6 +6,10 @@ from django.db import migrations, connection
 
 def upgrade_crawl_table_from_v086(apps, schema_editor):
     """Upgrade crawls_crawl table from v0.8.6rc0 schema to v0.9.0 schema."""
+    # sqlite-only legacy repair (sqlite_master/PRAGMA/GLOB). Postgres support
+    # postdates all legacy data, so this never has work to do there.
+    if schema_editor.connection.vendor != "sqlite":
+        return
     cursor = connection.cursor()
 
     # Check if crawls_crawl table exists
@@ -124,10 +128,13 @@ def upgrade_crawl_table_from_v086(apps, schema_editor):
                 status, retry_at, created_by_id, schedule_id
             )
             SELECT
-                REPLACE(id, '-', ''), created_at, modified_at, num_uses_succeeded, num_uses_failed,
-                '', config, max_depth, tags_str, REPLACE(persona_id, '-', ''), '', '', '',
-                status, retry_at, created_by_id, REPLACE(schedule_id, '-', '')
-            FROM crawls_crawl;
+                REPLACE(old_crawl.id, '-', ''), old_crawl.created_at, old_crawl.modified_at,
+                old_crawl.num_uses_succeeded, old_crawl.num_uses_failed,
+                COALESCE(old_seed.uri, ''), old_crawl.config, old_crawl.max_depth, old_crawl.tags_str,
+                NULL, '', '', '', old_crawl.status, old_crawl.retry_at,
+                old_crawl.created_by_id, old_crawl.schedule_id
+            FROM crawls_crawl AS old_crawl
+            LEFT JOIN seeds_seed AS old_seed ON old_seed.id = old_crawl.seed_id;
         """)
 
     cursor.execute("DROP TABLE crawls_crawl;")

archivebox/crawls/migrations/0016_hydrate_crawl_permissions.py

@@ -97,6 +97,12 @@ def _ensure_permissions_column(cursor):
 
 
 def hydrate_crawl_permissions(apps, schema_editor):
+    # sqlite-only legacy hydration + generated-column repair (PRAGMA
+    # table_xinfo / VIRTUAL generated column). On postgres the ``permissions``
+    # column is created portably by 0013 (AddField GeneratedField) and there is
+    # never legacy data to hydrate, so this is a no-op there.
+    if schema_editor.connection.vendor != "sqlite":
+        return
     Crawl = apps.get_model("crawls", "Crawl")
     base_config = raw_base_config(apps)
     default_permissions = resolve_permissions(base_config, "public")

archivebox/crawls/models.py

@@ -209,7 +209,13 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
         return resolve_delete_after_config_value(self.config)
 
     def pause(self, *, save: bool = True) -> bool:
-        return super().pause(save=save)
+        paused = super().pause(save=save)
+        if paused and save and self.pk:
+            from archivebox.core.models import Snapshot
+
+            for snapshot in self.snapshot_set.exclude(status__in=Snapshot.FINAL_STATES).iterator():
+                snapshot.pause()
+        return paused
 
     def resume(self, *, when=None, save: bool = True) -> bool:
         resumed = super().resume(when=when, save=save)
@@ -338,20 +344,6 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
                         removed_tag_names=removed_tag_names,
                     ),
                 )
-        # if is_new:
-        #     from archivebox.misc.logging_util import log_worker_event
-        #     first_url = self.get_urls_list()[0] if self.get_urls_list() else ''
-        #     log_worker_event(
-        #         worker_type='DB',
-        #         event='Created Crawl',
-        #         indent_level=1,
-        #         metadata={
-        #             'id': str(self.id),
-        #             'first_url': first_url[:64],
-        #             'max_depth': self.max_depth,
-        #             'status': self.status,
-        #         },
-        #     )
 
     def update_child_snapshot_permissions(self, old_permissions: str | None, new_permissions: str | None) -> int:
         from archivebox.core.models import Snapshot
@@ -1305,71 +1297,23 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
         return created_snapshots
 
     def install_declared_binaries(self, binary_names: set[str], machine=None) -> None:
-        """
-        Install crawl-declared Binary rows without violating the retry_at lock lifecycle.
-
-        Correct calling pattern:
-        1. Crawl hooks declare Binary records and queue them with retry_at <= now
-        2. Exactly one actor claims each Binary by moving retry_at into the future
-        3. Only that owner executes `.sm.tick()` and performs install side effects
-        4. Everyone else waits for the claimed owner to finish instead of launching
-           a second install against shared state such as the pip or npm trees
-
-        This helper follows that contract by claiming each Binary before ticking
-        it, and by waiting when another worker already owns the row. That keeps
-        synchronous crawl execution compatible with the shared background runner and
-        avoids duplicate installs of the same dependency.
-        """
-        import time
+        """Install crawl-declared binaries through their unified state machine."""
+        from archivebox.crawls.locks import binary_lifecycle_lock
         from archivebox.machine.models import Binary, Machine
 
         if not binary_names:
             return
 
         machine = machine or Machine.current()
-        lock_seconds = 600
-        deadline = time.monotonic() + max(lock_seconds, len(binary_names) * lock_seconds)
-
-        while time.monotonic() < deadline:
-            unresolved_binaries = list(
-                Binary.objects.filter(
-                    machine=machine,
-                    name__in=binary_names,
-                )
-                .exclude(
-                    status=Binary.StatusChoices.INSTALLED,
-                )
-                .order_by("name"),
-            )
-            if not unresolved_binaries:
-                return
-
-            claimed_any = False
-            waiting_on_existing_owner = False
-            now = timezone.now()
-
-            for binary in unresolved_binaries:
-                try:
-                    if binary.tick_claimed(lock_seconds=lock_seconds):
-                        claimed_any = True
-                        continue
-                except Exception:
-                    claimed_any = True
-                    continue
-
+        binaries = Binary.objects.filter(machine=machine, name__in=binary_names).order_by("name")
+        for binary in binaries:
+            with binary_lifecycle_lock(str(binary.id)):
                 binary.refresh_from_db()
                 if binary.status == Binary.StatusChoices.INSTALLED:
-                    claimed_any = True
                     continue
-                if binary.retry_at and binary.retry_at > now:
-                    waiting_on_existing_owner = True
-
-            if claimed_any:
-                continue
-            if waiting_on_existing_owner:
-                time.sleep(0.5)
-                continue
-            break
+                binary.update_and_requeue(retry_at=timezone.now())
+                binary.refresh_from_db()
+                binary.tick_claimed(lock_seconds=600)
 
         unresolved_binaries = list(
             Binary.objects.filter(
@@ -1401,7 +1345,7 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
         import time
         from archivebox.plugins.hooks import run_hook, discover_hooks, process_hook_records
         from archivebox.config.common import get_config
-        from archivebox.machine.models import Binary, Machine
+        from archivebox.machine.models import Machine
 
         def get_runtime_config():
             return get_config(crawl=self).for_crawl_runtime(
@@ -1429,10 +1373,7 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
                 chrome_binary=chrome_binary,
             )
 
-        executed_crawl_hooks: set[str] = set()
-
         def run_crawl_hook(hook: Path) -> set[str]:
-            executed_crawl_hooks.add(str(hook))
             primary_url = next(
                 (line.strip() for line in self.urls.splitlines() if line.strip()),
                 self.urls.strip(),
@@ -1493,53 +1434,12 @@ class Crawl(ModelWithDeleteAfter, ModelWithOutputDir, ModelWithConfig, ModelWith
                 declared_binary_names.update(hook_binary_names)
             return hook_binary_names
 
-        def resolve_provider_binaries(binary_names: set[str]) -> set[str]:
-            if not binary_names:
-                return set()
-
-            resolved_binary_names = set(binary_names)
-
-            while True:
-                unresolved_binaries = list(
-                    Binary.objects.filter(
-                        machine=machine,
-                        name__in=resolved_binary_names,
-                    )
-                    .exclude(
-                        status=Binary.StatusChoices.INSTALLED,
-                    )
-                    .order_by("name"),
-                )
-                if not unresolved_binaries:
-                    return resolved_binary_names
-
-                needed_provider_names: set[str] = set()
-                for binary in unresolved_binaries:
-                    allowed_binproviders = binary._allowed_binproviders()
-                    if allowed_binproviders is None:
-                        continue
-                    needed_provider_names.update(allowed_binproviders)
-
-                if not needed_provider_names:
-                    return resolved_binary_names
-
-                provider_hooks = [
-                    hook
-                    for hook in discover_hooks("Crawl", filter_disabled=False, config=get_runtime_config())
-                    if hook.parent.name in needed_provider_names and str(hook) not in executed_crawl_hooks
-                ]
-                if not provider_hooks:
-                    return resolved_binary_names
-
-                for hook in provider_hooks:
-                    resolved_binary_names.update(run_crawl_hook(hook))
-
         hooks = discover_hooks("Crawl", config=get_runtime_config())
 
         for hook in hooks:
             hook_binary_names = run_crawl_hook(hook)
             if hook_binary_names:
-                self.install_declared_binaries(resolve_provider_binaries(hook_binary_names), machine=machine)
+                self.install_declared_binaries(hook_binary_names, machine=machine)
 
         # Safety check: don't create snapshots if any crawl-declared dependency
         # is still unresolved after all crawl hooks have run.
@@ -1747,11 +1647,15 @@ class CrawlMachine(BaseStateMachine):
 
     @paused.enter
     def enter_paused(self):
-        self.crawl.update_and_requeue(
-            retry_at=RETRY_AT_MAX,
-            status=Crawl.StatusChoices.PAUSED,
+        paused = self.crawl.safe_update(
+            {
+                "retry_at": RETRY_AT_MAX,
+                "status": Crawl.StatusChoices.PAUSED,
+            },
+            extra_filter={"status__in": Crawl.RUNNABLE_STATES},
         )
-        self.crawl.schedule_child_snapshots_for_pause()
+        if paused:
+            self.crawl.schedule_child_snapshots_for_pause()
 
     @sealed.enter
     def enter_sealed(self):

archivebox/machine/admin.py

@@ -5,6 +5,7 @@ import shlex
 from pathlib import Path
 
 from django.contrib import admin, messages
+from django.db import DatabaseError
 from django.db.models import DurationField, ExpressionWrapper, F
 from django.db.models.functions import Coalesce, Now
 from django.shortcuts import redirect
@@ -14,9 +15,9 @@ from django.utils.safestring import mark_safe
 from django_object_actions import action
 
 from archivebox.base_models.admin import BaseModelAdmin, ConfigEditorMixin
-from archivebox.misc.logging_util import printable_filesize
 from archivebox.machine.env_util import env_to_dotenv_text
-from archivebox.machine.models import Machine, NetworkInterface, Binary, Process
+from archivebox.machine.models import Binary, Machine, NetworkInterface, Process
+from archivebox.misc.logging_util import printable_filesize
 
 
 def _render_copy_block(text: str, *, multiline: bool = False):
@@ -102,6 +103,21 @@ class MachineAdmin(ConfigEditorMixin, BaseModelAdmin):
         "os_release",
         "hw_uuid",
     )
+    search_fields = (
+        "id",
+        "guid",
+        "hostname",
+        "networkinterface__ip_public",
+        "networkinterface__ip_local",
+        "hw_manufacturer",
+        "hw_product",
+        "hw_uuid",
+        "os_platform",
+        "os_family",
+        "os_arch",
+        "os_release",
+        "os_kernel",
+    )
 
     readonly_fields = ("guid", "created_at", "modified_at", "ips")
 
@@ -162,9 +178,9 @@ class MachineAdmin(ConfigEditorMixin, BaseModelAdmin):
     )
 
     list_filter = ("hw_in_docker", "hw_in_vm", "os_arch", "os_family", "os_platform")
-    ordering = ["-created_at"]
+    ordering = ("-created_at",)
     list_per_page = 100
-    actions = ["delete_selected"]
+    actions = ("delete_selected",)
 
     @admin.display(description="Public IP", ordering="networkinterface__ip_public")
     def ips(self, machine):
@@ -192,7 +208,7 @@ class MachineAdmin(ConfigEditorMixin, BaseModelAdmin):
 
         try:
             current_id = str(Machine.current().pk)
-        except Exception:
+        except (DatabaseError, RuntimeError, TypeError, ValueError):
             current_id = None
 
         machine_id = str(machine.pk)
@@ -297,9 +313,9 @@ class NetworkInterfaceAdmin(BaseModelAdmin):
     )
 
     list_filter = ("isp", "country", "region")
-    ordering = ["-created_at"]
+    ordering = ("-created_at",)
     list_per_page = 100
-    actions = ["delete_selected"]
+    actions = ("delete_selected",)
 
     @admin.display(description="Machine", ordering="machine__id")
     def machine_info(self, iface):
@@ -370,9 +386,9 @@ class BinaryAdmin(BaseModelAdmin):
     )
 
     list_filter = ("name", "binprovider", "status", "machine_id")
-    ordering = ["-created_at"]
+    ordering = ("-created_at",)
     list_per_page = 100
-    actions = ["delete_selected"]
+    actions = ("delete_selected",)
 
     @admin.display(description="Machine", ordering="machine__id")
     def machine_info(self, binary):
@@ -492,10 +508,10 @@ class ProcessAdmin(BaseModelAdmin):
     )
 
     list_filter = ("status", "exit_code", "machine_id")
-    ordering = ["-created_at"]
+    ordering = ("-created_at",)
     list_per_page = 100
-    actions = ["kill_processes", "delete_selected"]
-    change_actions = ["kill_process"]
+    actions = ("kill_processes", "delete_selected")
+    change_actions = ("kill_process",)
 
     def get_queryset(self, request):
         return (
@@ -694,7 +710,7 @@ class ProcessAdmin(BaseModelAdmin):
         if isinstance(output_files, str):
             try:
                 output_files = json.loads(output_files)
-            except Exception:
+            except (json.JSONDecodeError, TypeError, ValueError):
                 output_files = {}
 
         file_count = 0

archivebox/machine/detect.py

@@ -18,6 +18,30 @@ PACKAGE_DIR = Path(__file__).parent
 DATA_DIR = Path(os.getcwd()).resolve()
 
 
+def _run_abxpkg_host_binary(name: str, *args: str) -> subprocess.CompletedProcess[str]:
+    from abxpkg import EnvProvider
+    from archivebox.config.common import get_config
+
+    env_root = get_config(include_machine=False).ABXPKG_LIB_DIR / "env"
+    provider = EnvProvider(install_root=env_root, PATH=os.environ["PATH"])
+    if name == "system_profiler":
+        provider = provider.get_provider_with_overrides(
+            overrides={name: {"version": platform.mac_ver()[0] or "0.0.0"}},
+        )
+    elif name == "lsb_release":
+        provider = provider.get_provider_with_overrides(
+            overrides={name: {"version": "1.0.0"}},
+        )
+    loaded = provider.load(name)
+    if loaded is None or loaded.loaded_abspath is None:
+        raise RuntimeError(f"abxpkg could not resolve {name}")
+    projection = Path(loaded.loaded_abspath)
+    expected_projection = env_root / "bin" / name
+    if projection != expected_projection or not projection.is_symlink() or not os.access(projection, os.X_OK):
+        raise RuntimeError(f"abxpkg did not project {name} into {expected_projection}")
+    return subprocess.run([str(projection), *args], capture_output=True, text=True, check=True)
+
+
 def get_vm_info():
     hw_in_docker = bool(os.getenv("IN_DOCKER", False) in ("1", "true", "True", "TRUE"))
     hw_in_vm = False
@@ -49,7 +73,7 @@ def get_vm_info():
             #       Serial Number (system): M230YYTD77
             #       Hardware UUID: 39A12B50-1972-5910-8BEE-235AD20C8EE3
             #       ...
-            result = subprocess.run(["system_profiler", "SPHardwareDataType"], capture_output=True, text=True, check=True)
+            result = _run_abxpkg_host_binary("system_profiler", "SPHardwareDataType")
             for line in result.stdout.split("\n"):
                 if "Model Name:" in line:
                     hw_product = line.split(":", 1)[-1].strip()
@@ -73,7 +97,7 @@ def get_vm_info():
             #         UUID: fb65f41c-ec24-4539-beaf-f941903bdb2c
             #         ...
             #         Family: DigitalOcean_Droplet
-            dmidecode = subprocess.run(["dmidecode", "-t", "system"], capture_output=True, text=True, check=True)
+            dmidecode = _run_abxpkg_host_binary("dmidecode", "-t", "system")
             for line in dmidecode.stdout.split("\n"):
                 if "Manufacturer:" in line:
                     hw_manufacturer = line.split(":", 1)[-1].strip()
@@ -90,7 +114,7 @@ def get_vm_info():
 
     # Check for QEMU explicitly in pmap output
     try:
-        result = subprocess.run(["pmap", "1"], capture_output=True, text=True, check=True)
+        result = _run_abxpkg_host_binary("pmap", "1")
         if "qemu" in result.stdout.lower():
             hw_in_vm = True
     except Exception:
@@ -174,7 +198,7 @@ def get_isp_info(ip=None):
     # Get system DNS resolver servers
     dns_server = None
     try:
-        result = subprocess.run(["dig", "example.com", "A"], capture_output=True, text=True, check=True).stdout
+        result = _run_abxpkg_host_binary("dig", "example.com", "A").stdout
         dns_server = result.split(";; SERVER: ", 1)[-1].split("\n")[0].split("#")[0].strip()
     except Exception:
         try:
@@ -183,10 +207,6 @@ def get_isp_info(ip=None):
             dns_server = "127.0.0.1"
             print(f"[red]:warning: WARNING: Could not determine DNS server, using {dns_server}[/red]")
 
-    # Get DNS resolver's ISP name
-    # url = f'https://ipapi.co/{dns_server}/json/'
-    # dns_isp = json.loads(urllib.request.urlopen(url).read().decode()).get('org', 'Unknown')
-
     return {
         "isp": isp,
         "city": city,
@@ -225,7 +245,7 @@ def get_os_info() -> dict[str, Any]:
         os_release = "macOS " + platform.mac_ver()[0]
     else:
         try:
-            os_release = subprocess.run(["lsb_release", "-ds"], capture_output=True, text=True, check=True).stdout.strip()
+            os_release = _run_abxpkg_host_binary("lsb_release", "-ds").stdout.strip()
         except Exception:
             pass
 

archivebox/machine/migrations/0001_initial.py

@@ -7,16 +7,45 @@ import django.utils.timezone
 from archivebox.uuid_compat import uuid7
 
 
-class Migration(migrations.Migration):
-    initial = True
+def _run_sqlite_initial_ddl(apps, schema_editor):
+    # sqlite-only: raw DDL creates the real tables byte-for-byte as historically
+    # shipped. On other vendors the tables are built from migration state by
+    # _pg_sync_schema below (a fresh non-sqlite DB never holds legacy data here).
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(
+        sql=_SQLITE_INITIAL_SQL,
+        reverse_sql=_SQLITE_INITIAL_REVERSE_SQL,
+    ).database_forwards("machine", schema_editor, None, None)
 
-    dependencies = []
 
-    operations = [
-        migrations.SeparateDatabaseAndState(
-            database_operations=[
-                migrations.RunSQL(
-                    sql="""
+def _reverse_sqlite_initial_ddl(apps, schema_editor):
+    if schema_editor.connection.vendor != "sqlite":
+        return
+    migrations.RunSQL(
+        sql=_SQLITE_INITIAL_SQL,
+        reverse_sql=_SQLITE_INITIAL_REVERSE_SQL,
+    ).database_backwards("machine", schema_editor, None, None)
+
+
+def _pg_sync_schema(apps, schema_editor):
+    from archivebox.misc.db import rebuild_models_from_migration_state
+
+    rebuild_models_from_migration_state(
+        apps,
+        schema_editor,
+        "machine",
+        ["Machine", "NetworkInterface", "Binary"],
+    )
+
+
+def _pg_drop_schema(apps, schema_editor):
+    from archivebox.misc.db import drop_models_on_postgres
+
+    drop_models_on_postgres(apps, schema_editor, "machine", ["Binary", "NetworkInterface", "Machine"])
+
+
+_SQLITE_INITIAL_SQL = """
                 -- Create machine_machine table
                 CREATE TABLE IF NOT EXISTS machine_machine (
                     id TEXT PRIMARY KEY NOT NULL,
@@ -98,13 +127,25 @@ class Migration(migrations.Migration):
                 CREATE INDEX IF NOT EXISTS machine_binary_status_idx ON machine_binary(status);
                 CREATE INDEX IF NOT EXISTS machine_binary_retry_at_idx ON machine_binary(retry_at);
 
-            """,
-                    reverse_sql="""
+            """
+
+
+_SQLITE_INITIAL_REVERSE_SQL = """
                         DROP TABLE IF EXISTS machine_binary;
                         DROP TABLE IF EXISTS machine_networkinterface;
                         DROP TABLE IF EXISTS machine_machine;
-                    """,
-                ),
+                    """
+
+
+class Migration(migrations.Migration):
+    initial = True
+
+    dependencies = []
+
+    operations = [
+        migrations.SeparateDatabaseAndState(
+            database_operations=[
+                migrations.RunPython(_run_sqlite_initial_ddl, _reverse_sqlite_initial_ddl),
             ],
             state_operations=[
                 migrations.CreateModel(
@@ -244,4 +285,5 @@ class Migration(migrations.Migration):
                 ),
             ],
         ),
+        migrations.RunPython(_pg_sync_schema, reverse_code=_pg_drop_schema),
     ]

archivebox/machine/migrations/0005_converge_binary_model.py

@@ -10,6 +10,13 @@ def converge_binary_table(apps, schema_editor):
     Drop machine_installedbinary if it exists (0.8.6rc0 path).
     Create machine_binary if it doesn't exist (needed by Process model).
     """
+    # sqlite-only legacy convergence: on a fresh install machine_binary already
+    # exists with the final column set (created by 0001), so every branch below
+    # is a no-op schema-wise. A non-sqlite DB never carries the 0.8.x legacy
+    # shapes this repairs, and its Binary table already matches migration state.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
     cursor = connection.cursor()
 
     # Check what tables exist

archivebox/machine/migrations/0011_remove_binary_output_dir.py

@@ -2,9 +2,12 @@ from django.db import migrations
 
 
 def remove_output_dir_if_exists(apps, schema_editor):
-    cursor = schema_editor.connection.cursor()
-    cursor.execute("PRAGMA table_info(machine_binary)")
-    columns = {row[1] for row in cursor.fetchall()}
+    # On non-sqlite the RemoveField below is state-only, so the real
+    # ALTER TABLE DROP COLUMN must run here too (a fresh DB has machine_binary
+    # with output_dir from 0001). Portable introspection works on both backends.
+    connection = schema_editor.connection
+    with connection.cursor() as cursor:
+        columns = {col.name for col in connection.introspection.get_table_description(cursor, "machine_binary")}
 
     if "output_dir" not in columns:
         return

archivebox/machine/migrations/0012_add_machine_config_if_missing.py

@@ -2,6 +2,12 @@ from django.db import migrations
 
 
 def add_machine_config_if_missing(apps, schema_editor):
+    # sqlite-only: uses PRAGMA introspection. On a fresh non-sqlite install the
+    # config column already exists (created from 0001 migration state), so there
+    # is nothing to add.
+    if schema_editor.connection.vendor != "sqlite":
+        return
+
     cursor = schema_editor.connection.cursor()
     cursor.execute("PRAGMA table_info(machine_machine)")
     columns = {row[1] for row in cursor.fetchall()}

archivebox/machine/models.py

@@ -238,9 +238,18 @@ class Machine(ModelWithHealthStats):
         try:
             _CURRENT_MACHINE = cls.objects.get(guid=host_guid)
         except cls.DoesNotExist:
+            config = {}
+            try:
+                from archivebox.config.collection import _coerce_from_str_dict, _load_file_config_dict
+
+                file_config, _file_mtime = _load_file_config_dict()
+                config = _coerce_from_str_dict(file_config)
+            except Exception:
+                config = {}
             _CURRENT_MACHINE = cls.objects.create(
                 guid=host_guid,
                 hostname=socket.gethostname(),
+                config=config,
                 **get_os_info(),
                 **get_vm_info(),
                 stats=get_host_stats(),
@@ -400,9 +409,6 @@ class NetworkInterface(ModelWithHealthStats):
     city = models.CharField(max_length=63, default="", null=False)
     region = models.CharField(max_length=63, default="", null=False)
     country = models.CharField(max_length=63, default="", null=False)
-    # num_uses_failed = models.PositiveIntegerField(default=0)  # from ModelWithHealthStats
-    # num_uses_succeeded = models.PositiveIntegerField(default=0)  # from ModelWithHealthStats
-
     objects = NetworkInterfaceManager()  # pyright: ignore[reportIncompatibleVariableOverride]
     machine_id: uuid.UUID
 
@@ -1493,15 +1499,11 @@ class Process(ModelWithDeleteAfter, models.Model):
         ppid = os.getppid()
         machine = machine or Machine.current()
 
-        # Debug logging
-        # print(f"DEBUG _find_parent_process: my_pid={os.getpid()}, ppid={ppid}", file=sys.stderr)
-
         # Get parent process start time from OS
         try:
             os_parent = psutil.Process(ppid)
             os_parent_start = os_parent.create_time()
         except (psutil.NoSuchProcess, psutil.AccessDenied, psutil.ZombieProcess):
-            # print(f"DEBUG _find_parent_process: Parent process {ppid} not accessible", file=sys.stderr)
             return None  # Parent process doesn't exist
 
         # Find matching Process record
@@ -1512,18 +1514,13 @@ class Process(ModelWithDeleteAfter, models.Model):
             started_at__gte=timezone.now() - PID_REUSE_WINDOW,
         ).order_by("-started_at")
 
-        # print(f"DEBUG _find_parent_process: Found {candidates.count()} candidates for ppid={ppid}", file=sys.stderr)
-
         for candidate in candidates:
             if candidate.started_at:
                 db_start_time = candidate.started_at.timestamp()
                 time_diff = abs(db_start_time - os_parent_start)
-                # print(f"DEBUG _find_parent_process: Checking candidate id={candidate.id} time_diff={time_diff:.2f}s tolerance={START_TIME_TOLERANCE}s", file=sys.stderr)
                 if time_diff < START_TIME_TOLERANCE:
-                    # print(f"DEBUG _find_parent_process: MATCH! Returning parent id={candidate.id} pid={candidate.pid}", file=sys.stderr)
                     return candidate
 
-        # print(f"DEBUG _find_parent_process: No matching parent found for ppid={ppid}", file=sys.stderr)
         return None  # No matching ArchiveBox parent process
 
     @classmethod
@@ -1575,7 +1572,7 @@ class Process(ModelWithDeleteAfter, models.Model):
 
         Processes are stale if:
         - Status is RUNNING but OS process no longer exists
-        - Status is RUNNING but exceeded its timeout plus a small grace margin
+        - A bounded HOOK or BINARY process exceeded its timeout plus a small grace margin
         - Status is RUNNING but started_at is older than PID_REUSE_WINDOW
 
         Returns count of processes cleaned up.
@@ -1595,7 +1592,7 @@ class Process(ModelWithDeleteAfter, models.Model):
 
             is_stale = False
 
-            if proc.started_at:
+            if proc.started_at and proc.process_type in (cls.TypeChoices.HOOK, cls.TypeChoices.BINARY):
                 timeout_seconds = max(int(proc.timeout or 0), 0)
                 timeout_deadline = proc.started_at + timedelta(seconds=timeout_seconds) + PROCESS_TIMEOUT_GRACE
                 if timeout_seconds > 0 and timezone.now() >= timeout_deadline:
@@ -2142,12 +2139,8 @@ class Process(ModelWithDeleteAfter, models.Model):
             # Process exited - read output and copy to DB
             if self.stdout_file and self.stdout_file.exists():
                 self.stdout = self.stdout_file.read_text(errors="replace")
-                # TODO: Uncomment to cleanup (keeping for debugging for now)
-                # self.stdout_file.unlink(missing_ok=True)
             if self.stderr_file and self.stderr_file.exists():
                 self.stderr = self.stderr_file.read_text(errors="replace")
-                # TODO: Uncomment to cleanup (keeping for debugging for now)
-                # self.stderr_file.unlink(missing_ok=True)
 
             self.exit_code = self.exit_code if self.exit_code is not None else _default_exit_code_for_unowned_process(self.process_type)
             if self.exit_code == -1:
@@ -2462,6 +2455,15 @@ class Process(ModelWithDeleteAfter, models.Model):
         if not chrome_utils.exists():
             return 0
 
+        node_binary = cast(BinaryManager, Binary.objects).get_valid_binary("node")
+        if node_binary is None or not node_binary.is_valid:
+            return 0
+        from archivebox.config.common import get_config
+
+        node_projection = get_config().ABXPKG_LIB_DIR / "env" / "bin" / "node"
+        if not node_projection.is_symlink() or not os.access(node_projection, os.X_OK):
+            return 0
+
         crawl_roots = [
             crawls_dir
             for user_dir in CONSTANTS.USERS_DIR.iterdir()
@@ -2476,7 +2478,7 @@ class Process(ModelWithDeleteAfter, models.Model):
         try:
             for crawl_root in crawl_roots:
                 result = subprocess.run(
-                    ["node", str(chrome_utils), "killZombieChrome", str(crawl_root)],
+                    [str(node_projection), str(chrome_utils), "killZombieChrome", str(crawl_root)],
                     capture_output=True,
                     timeout=30,
                     text=True,

archivebox/mcp/README.md

@@ -19,7 +19,7 @@ This is a lightweight, stateless MCP server that dynamically introspects Archive
 ### Start the MCP Server
 
 ```bash
-archivebox mcp
+printf '%s\n' '{"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}' | archivebox mcp
 ```
 
 The server runs in stdio mode, reading JSON-RPC 2.0 requests from stdin and writing responses to stdout.
@@ -27,25 +27,21 @@ The server runs in stdio mode, reading JSON-RPC 2.0 requests from stdin and writ
 ### Example Client
 
 ```python
-import subprocess
 import json
+import subprocess
 
-# Start MCP server
-proc = subprocess.Popen(
-    ['archivebox', 'mcp'],
-    stdin=subprocess.PIPE,
-    stdout=subprocess.PIPE,
-    text=True
-)
-
-# Send initialize request
 request = {"jsonrpc": "2.0", "id": 1, "method": "initialize", "params": {}}
-proc.stdin.write(json.dumps(request) + '\n')
-proc.stdin.flush()
-
-# Read response
-response = json.loads(proc.stdout.readline())
-print(response)
+completed = subprocess.run(
+    ["archivebox", "mcp"],
+    input=json.dumps(request) + "\n",
+    capture_output=True,
+    text=True,
+    check=True,
+    timeout=30,
+)
+response = json.loads(completed.stdout)
+assert response["id"] == 1
+assert response["result"]["serverInfo"]["name"] == "archivebox-mcp"
 ```
 
 ### Example Requests
@@ -98,16 +94,14 @@ The server exposes all ArchiveBox CLI commands:
 Instead of manually defining schemas, the server uses Click's introspection API to automatically generate MCP tool definitions:
 
 ```python
-# Auto-discover commands
-from archivebox.cli import ArchiveBoxGroup
-cli_group = ArchiveBoxGroup()
-all_commands = cli_group.all_subcommands
+from archivebox.mcp.server import MCPServer, click_command_to_mcp_tool
 
-# Auto-generate schemas from Click metadata
-for cmd_name in all_commands:
-    click_cmd = cli_group.get_command(None, cmd_name)
-    # Extract params, types, help text, etc.
-    tool_schema = click_command_to_mcp_tool(cmd_name, click_cmd)
+tools = []
+for discovered_tool in MCPServer().get_tools().values():
+    tools.append(click_command_to_mcp_tool(discovered_tool))
+
+assert {tool["name"] for tool in tools}
+assert all("inputSchema" in tool for tool in tools)
 ```
 
 ### Tool Execution
@@ -115,10 +109,14 @@ for cmd_name in all_commands:
 Commands are executed using Click's `CliRunner`:
 
 ```python
-from click.testing import CliRunner
+from archivebox.mcp.server import MCPServer
 
-runner = CliRunner()
-result = runner.invoke(click_command, args)
+response = MCPServer().handle_request(
+    {"jsonrpc": "2.0", "id": 2, "method": "tools/list", "params": {}},
+)
+assert response["id"] == 2
+assert response["result"]["tools"]
+assert all("name" in tool and "inputSchema" in tool for tool in response["result"]["tools"])
 ```
 
 ## Files