Preserve historical code blocks and pin the complete docs build

Restore historical documentation for the 0.0 release line
2026-09-17 23:59:31 -07:00 · 2026-09-17 23:37:27 -07:00
700 changed files with 2726 additions and 153712 deletions
@@ -1,104 +0,0 @@
-.DS_Store
-._*
-/.heartbeat.json
-/package.json
-/package-lock.json
-__pycache__
-**/__pycache__
-*.pyc
-*.pyo
-*.py[cod]
-*$py.class
-.mypy_cache/
-.pytest_cache/
-.ruff_cache/
-.uv-cache/
-.cache/
-.cache*/
-.github/
-.pdm-build/
-.pdm-build/**
-.pdm-python
-.eggs/
-*.egg-info/
-.git/
-!.git/
-.git/*
-.vscode/
-!.git/HEAD
-!.git/packed-refs
-!.git/refs/
-!.git/refs/heads/
-!.git/refs/heads/**
-
-venv/
-.venv/
-.venv-old/
-.docker_venv/
-.docker-venv/
-node_modules/
-abx-dl/
-abxpkg/
-abx-plugins/
-abxbus/
-chrome/
-chromeprofile/
-chrome_profile/
-lib/
-out/
-tests/out/
-users/
-archive/
-crawls/
-snapshots/
-logs/
-archivebox-docker-smoke*/
-archivebox-compose-smoke*/
-docker-test/
-docker-test*/
-core
-*.core
-
-pdm.dev.lock
-pdm.lock
-
-docs/
-build/
-dist/
-brew_dist/
-deb_dist/
-pip_dist/
-cache/
-caches/
-assets/
-docker/
-website/
-typings/
-
-# The runtime COPY needs ArchiveBox's package and bin/ entrypoint, not the
-# development UI screenshots or test trees. PNGs are already compressed, so
-# including them creates a large image layer that later cleanup cannot remove.
-publicsite/
-old/
-test/
-tests/
-archivebox/tests/
-
-tmp/
-.tmp/
-data/
-data*/
-
-personas/
-sources/
-output/
-index.sqlite3
-index.sqlite3-wal
-queue.sqlite3
-*.sqlite*
-data.*
-.archivebox_id
-ArchiveBox.conf
-*.stdout
-*.stderr
-*.log
@@ -1,26 +1,17 @@
-# Read the Docs config for https://docs.archivebox.io
-# https://docs.readthedocs.io/en/stable/config-file/v2.html
-
 version: 2
-
-submodules:
-  include: all
-  recursive: true
-
 build:
-  os: ubuntu-22.04
+  os: ubuntu-24.04
  tools:
-    python: "3.12"
-    #nodejs: "20"     # not needed unless we need the full archivebox to run while building docs for some reason
-
-sphinx:
-  configuration: docs/conf.py
-
-formats:
-  - pdf
-  - epub
-
-# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
+    python: "3.13"
+  jobs:
+    build:
+      html:
+        - uv run --no-project --with-requirements .github/docs/requirements.txt sphinx-build -W --keep-going -b html -c .github/docs . "$READTHEDOCS_OUTPUT/html"
 python:
  install:
-    - requirements: docs/requirements.txt
+    - method: uv
+      command: pip
+      requirements: .github/docs/requirements.txt
+sphinx:
+  configuration: .github/docs/conf.py
+  fail_on_warning: true
@@ -1,76 +0,0 @@
-# Contributor Covenant Code of Conduct
-
-## Our Pledge
-
-In the interest of fostering an open and welcoming environment, we as
-contributors and maintainers pledge to making participation in our project and
-our community a harassment-free experience for everyone, regardless of age, body
-size, disability, ethnicity, sex characteristics, gender identity and expression,
-level of experience, education, socio-economic status, nationality, personal
-appearance, race, religion, or sexual identity and orientation.
-
-## Our Standards
-
-Examples of behavior that contributes to creating a positive environment
-include:
-
-* Using welcoming and inclusive language
-* Being respectful of differing viewpoints and experiences
-* Gracefully accepting constructive criticism
-* Focusing on what is best for the community
-* Showing empathy towards other community members
-
-Examples of unacceptable behavior by participants include:
-
-* The use of sexualized language or imagery and unwelcome sexual attention or
- advances
-* Trolling, insulting/derogatory comments, and personal or political attacks
-* Public or private harassment
-* Publishing others' private information, such as a physical or electronic
- address, without explicit permission
-* Other conduct which could reasonably be considered inappropriate in a
- professional setting
-
-## Our Responsibilities
-
-Project maintainers are responsible for clarifying the standards of acceptable
-behavior and are expected to take appropriate and fair corrective action in
-response to any instances of unacceptable behavior.
-
-Project maintainers have the right and responsibility to remove, edit, or
-reject comments, commits, code, wiki edits, issues, and other contributions
-that are not aligned to this Code of Conduct, or to ban temporarily or
-permanently any contributor for other behaviors that they deem inappropriate,
-threatening, offensive, or harmful.
-
-## Scope
-
-This Code of Conduct applies both within project spaces and in public spaces
-when an individual is representing the project or its community. Examples of
-representing a project or community include using an official project e-mail
-address, posting via an official social media account, or acting as an appointed
-representative at an online or offline event. Representation of a project may be
-further defined and clarified by project maintainers.
-
-## Enforcement
-
-Instances of abusive, harassing, or otherwise unacceptable behavior may be
-reported by contacting the project team at archivebox-coc@sweeting.me. All
-complaints will be reviewed and investigated and will result in a response that
-is deemed necessary and appropriate to the circumstances. The project team is
-obligated to maintain confidentiality with regard to the reporter of an incident.
-Further details of specific enforcement policies may be posted separately.
-
-Project maintainers who do not follow or enforce the Code of Conduct in good
-faith may face temporary or permanent repercussions as determined by other
-members of the project's leadership.
-
-## Attribution
-
-This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4,
-available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html
-
-[homepage]: https://www.contributor-covenant.org
-
-For answers to common questions about this code of conduct, see
-https://www.contributor-covenant.org/faq
@@ -1,38 +0,0 @@
-# Contribution Process
-
-1. Confirm your desired features fit into our bigger project goals [Roadmap](https://github.com/pirate/ArchiveBox/wiki/Roadmap).
-2. Open an issue with your planned implementation to discuss
-3. Check in with me before starting development to make sure your work wont conflict with or duplicate existing work
-4. Setup your dev environment, make some changes, and test using the test input files
-5. Commit, push, and submit a PR and wait for review feedback
-6. Have patience, don't abandon your PR! We love contributors but we all have day jobs and don't always have time to respond to notifications instantly. If you want a faster response, ping @theSquashSH on twitter or Patreon.
-
-**Useful links:**
-
- https://github.com/ArchiveBox/ArchiveBox/issues
- https://github.com/ArchiveBox/ArchiveBox/pulls
- https://github.com/ArchiveBox/ArchiveBox/wiki/Roadmap
- https://github.com/ArchiveBox/ArchiveBox/wiki/Install#manual-setup
-
-### Development Setup
-
-```bash
-git clone https://github.com/ArchiveBox/ArchiveBox
-cd ArchiveBox
-# Ideally do this in a virtualenv
-pip install -e '.[dev]'  # or use: pipenv install --dev
-```
-
-### Running Tests
-
-```bash
-./bin/lint.sh
-./bin/test.sh
-./bin/build.sh
-```
-
-For more common tasks see the `Development` section at the bottom of the README.
-
-### Getting Help
-
-Open issues on Github or message me https://sweeting.me/#contact.
@@ -1,2 +0,0 @@
-github: ["ArchiveBox", "pirate"]
-custom: ["https://donate.archivebox.io", "https://swag.archivebox.io"]
@@ -1,197 +0,0 @@
-name: 🐞 Bug report
-description: Report a bug or error you encountered in ArchiveBox
-title: "Bug: ..."
-assignees:
-  - pirate
-type: 'Bug'
-body:
-  - type: markdown
-    attributes:
-      value: |
-        *Please note:* it is normal to see errors occasionally for some extractors on some URLs (not every extractor will work on every type of page).  
-        Please report archiving errors if you are seeing them *consistently across many URLs* or if they are *preventing you from using ArchiveBox*.
-
-  - type: textarea
-    id: description
-    attributes:
-      label: Provide a screenshot and describe the bug
-      description: |
-        Attach a screenshot and describe what the issue is, what you expected to happen, and if relevant, the *URLs you were trying to archive*.  
-      placeholder: |
-        Got a bunch of 'singlefile was unable to archive this page' errors when trying to archive URLs from this site: https://example.com/xyz ...
-        I also tried to archive the same URLs using `singlefile` directly and some of them worked but not all of them. etc. ...
-    validations:
-      required: true
-
-  - type: textarea
-    id: steps_to_reproduce
-    attributes:
-      label: Steps to reproduce
-      description: Please provide the exact steps you took to trigger the issue (including any shell commands run, URLs visited, buttons clicked, etc.).
-      render: markdown
-      placeholder: |
-        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox` in iTerm2
-        2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
-        3. Typed 'https://example.com/xyz' into the 'Add URL' input field
-        4. Clicked the 'Add+' button
-        5. Got a 500 error and saw the errors below in terminal
-    validations:
-      required: true
-
-  - type: textarea
-    id: logs
-    attributes:
-      label: Logs or errors
-      description: "Paste any terminal output, logs, or errors (check `data/logs/errors.log` as well)."
-      placeholder: |
-        ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
-        │ [2024-11-02 19:54:28] ArchiveBox v0.8.6rc0: archivebox add https://example.com#1234567                  │
-        ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
-        
-        [+] [2024-11-02 19:54:29] Adding 1 links to index (crawl depth=0)...
-            > Saved verbatim input to sources/1730577269-import.txt
-            > Parsed 1 URLs from input (Generic TXT)
-            ...
-      render: shell
-    validations:
-      required: false
-
-  - type: textarea
-    id: version
-    attributes:
-      label: ArchiveBox Version
-      description: |
-        **REQUIRED:** Run the `archivebox version` command inside your collection dir and paste the *full output* here (*not just the version number*).  
-        For Docker Compose run: `docker compose run archivebox version`  
-        For plain Docker run:   `docker run -v $PWD:/data archivebox/archivebox version`  
-      render: shell
-      placeholder: |
-        0.8.6
-        ArchiveBox v0.8.6rc0 COMMIT_HASH=721427a BUILD_TIME=2024-10-21 12:57:02 1729515422
-        IN_DOCKER=False IN_QEMU=False ARCH=arm64 OS=Darwin PLATFORM=macOS-15.1-arm64-arm-64bit PYTHON=Cpython (venv)
-        EUID=502:20 UID=502:20 FS_UID=502:20 FS_PERMS=644 FS_ATOMIC=True FS_REMOTE=False
-        DEBUG=False IS_TTY=True SUDO=False ID=dfa11485:aa78ad45 SEARCH_BACKEND=ripgrep LDAP=False
-        
-        Binary Dependencies:
-        √  python     3.14.0     venv_pip   ~/.venv/bin/python
-        √  django     6.0        venv_pip   ~/.venv/lib/python3.14/site-packages/django/__init__.py
-        √  sqlite     2.6.0      venv_pip   ~/.venv/lib/python3.14/site-packages/django/db/backends/sqlite3/base.py
-        √  pip        24.3.1     venv_pip   ~/.venv/bin/pip
-        ...
-    validations:
-      required: true
-
-  - type: dropdown
-    id: install_method
-    validations:
-      required: true
-    attributes:
-      label: How did you install the version of ArchiveBox you are using?
-      multiple: false
-      options:
-        - pip
-        - apt
-        - brew
-        - nix
-        - Docker (or Podman/LXC/K8s/TrueNAS/Proxmox/etc)
-        - Other
-
-  - type: dropdown
-    id: operating_system
-    validations:
-      required: true
-    attributes:
-      label: What operating system are you running on?
-      description: |
-        Please note we are *unable to provide support for Windows users* unless you are using [Docker on Windows](https://github.com/ArchiveBox/archivebox#:~:text=windows%20without%20docker).
-      multiple: false
-      options:
-        - Linux (Ubuntu/Debian/Arch/Alpine/etc.)
-        - macOS (including Docker on macOS)
-        - BSD (FreeBSD/OpenBSD/NetBSD/etc.)
-        - Windows (including WSL, WSL2, Docker Desktop on Windows)
-        - Other
-
-  - type: checkboxes
-    id: filesystem
-    attributes:
-      label: What type of drive are you using to store your ArchiveBox data?
-      description: Are you using a [remote filesystem](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#supported-remote-filesystems) or FUSE mount for `data/` OR `data/archive`?
-      options:
-        - label: "some of `data/` is on a local SSD or NVMe drive"
-          required: false
-        - label: "some of `data/` is on a spinning hard drive or external USB drive"
-          required: false
-        - label: "some of `data/` is on a network mount (e.g. NFS/SMB/Ceph/GlusterFS/etc.)"
-          required: false
-        - label: "some of `data/` is on a FUSE mount (e.g. SSHFS/RClone/S3/B2/Google Drive/Dropbox/etc.)"
-          required: false
-  
-
-  - type: textarea
-    id: docker_compose_yml
-    attributes:
-      label: Docker Compose Configuration
-      description: "If using Docker Compose, please share your full `docker-compose.yml` file. If using plain Docker, paste the `docker run ...` command you use."
-      placeholder: |
-        services:
-            archivebox:
-                image: archivebox/archivebox:latest
-                ports:
-                    - 8000:8000
-                volumes:
-                    - ./data:/data
-                environment:
-                    - ADMIN_USERNAME=admin
-                    - ADMIN_PASSWORD=****<redact any passwords>****
-                    - ALLOWED_HOSTS=*
-                    - CSRF_TRUSTED_ORIGINS=https://archivebox.example.com
-                    - PUBLIC_INDEX=True
-                    - PUBLIC_SNAPSHOTS=True
-                    - PUBLIC_ADD_VIEW=False
-                    ...
-
-            archivebox_scheduler:
-                image: archivebox/archivebox:latest
-                command: schedule --foreground --update --every=day
-                environment:
-                    ...
-      
-          ...
-      render: shell
-    validations:
-      required: false
-
-  - type: textarea
-    id: configuration
-    attributes:
-      label: ArchiveBox Configuration
-      description: "Please share your full `data/ArchiveBox.conf` file here."
-      render: shell
-      placeholder: |
-        [SERVER_CONFIG]
-        SECRET_KEY = "*********<redact any secrets/passwords>************"
-
-        WGET_RESTRICT_FILE_NAMES=windows
-        USE_SYSTEM_WGET=true
-        CHECK_SSL_VALIDITY=false
-        ...
-    validations:
-      required: false
-
-
-  - type: markdown
-    attributes:
-      value: |
-        ---
-          
-        We strive to answer issues as quickly as possible, it usually takes us *about a ~week* to respond.  
-        Make sure your `data/` is [**fully backed up**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout) before trying anything suggested here, **we are not responsible for data loss**.  
-          
-        In the meantime please consider:  
-          
-          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
-          - 📚 Reading the [Troubleshooting Guide](https://github.com/ArchiveBox/ArchiveBox/wiki)
-          - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)
-        
@@ -1,124 +0,0 @@
-name: 💡 Feature or enhancement request
-description: Suggest an idea or improvement for this project
-title: "Feature Request: ..."
-assignees:
-  - pirate
-type: 'Enhancement'
-labels: 'status: idea phase'
-body:
-  - type: dropdown
-    id: suggestion_type
-    validations:
-      required: true
-    attributes:
-      label: "What type of suggestion are you making?"
-      multiple: false
-      options:
-        - New extractor / type of content to save
-        - Proposing a new feature
-        - Modification of existing behavior
-        - Web UI or UX design improvement
-
-  - type: textarea
-    id: current_problem
-    attributes:
-      label: "What is the problem that your feature request solves?"
-      description: |
-        Describe the problem or need that your feature request solves, feel free to include any screenshots or examples.
-      placeholder: |
-        e.g. I need to be able to archive spanish and french subtitle files from a particular movie site https://example.com/somevideos that's going down soon.
-    validations:
-      required: true
-
-  - type: textarea
-    id: proposed_solution
-    attributes:
-      label: "What is your proposed solution?"
-      description: |
-        Describe the ideal specific solution you'd want, *and whether it fits into any broader scope of changes*.
-      placeholder: |
-        e.g. I specifically need a new archive method to look for multilingual subtitle files related to pages.  
-        The bigger picture solution is the ability for custom user scripts to be run in a puppeteer context during archiving.
-    validations:
-      required: true
-
-  - type: textarea
-    id: workarounds_tried
-    attributes:
-      label: "What hacks or alternative solutions have you tried to solve the problem?"
-      description: |
-        A description of any alternative approaches, workarounds, or other solutions you've considered to fix the problem.
-      placeholder: |
-        e.g. I wait for archivebox to finish archiving the page, then I manually run `yt-dlp --subs <URL>` inside 
-        the `data/archive/<timestamp>/` directory to download the subtitle files and add them to the snapshot folder.
-    validations:
-      required: true
-
-  - type: textarea
-    id: version
-    attributes:
-      label: Share the entire output of the `archivebox version` command for the current version you are using.
-      description: |
-        DO NOT JUST ENTER "the latest version" OR YOUR ISSUE WILL BE CLOSED.  
-        We need to know what version of ArchiveBox and what feature flags you're currently running with in order to contextualize your feature request.  
-        Sometimes we've already fixed the issues in newer BETA versions, sometimes features already exist but may not be available in your specific environment.
-          
-        Run the `archivebox version` command inside your current collection dir and paste the *full output* here (*not just the version number*).  
-        For Docker Compose run: `docker compose run archivebox version`  
-        For plain Docker run:   `docker run -v $PWD:/data archivebox/archivebox version`  
-      render: shell
-      placeholder: |
-        0.8.6
-        ArchiveBox v0.8.6rc0 COMMIT_HASH=721427a BUILD_TIME=2024-10-21 12:57:02 1729515422
-        IN_DOCKER=False IN_QEMU=False ARCH=arm64 OS=Darwin PLATFORM=macOS-15.1-arm64-arm-64bit PYTHON=Cpython (venv)
-        EUID=502:20 UID=502:20 FS_UID=502:20 FS_PERMS=644 FS_ATOMIC=True FS_REMOTE=False
-        DEBUG=False IS_TTY=True SUDO=False ID=dfa11485:aa78ad45 SEARCH_BACKEND=ripgrep LDAP=False
-        
-        Binary Dependencies:
-        √  python     3.14.0     venv_pip   ~/.venv/bin/python
-        √  django     6.0        venv_pip   ~/.venv/lib/python3.14/site-packages/django/__init__.py
-        √  sqlite     2.6.0      venv_pip   ~/.venv/lib/python3.14/site-packages/django/db/backends/sqlite3/base.py
-        √  pip        24.3.1     venv_pip   ~/.venv/bin/pip
-        ...
-    validations:
-      required: true
-
-  - type: checkboxes
-    id: priority
-    attributes:
-      label: "How badly do you want this new feature?"
-      options:
-        - label: "It's an urgent deal-breaker, I can't live without it"
-          required: false
-        - label: "It's important to add it in the near-mid term future"
-          required: false
-        - label: "It would be nice to have eventually"
-          required: false
-        - label: "I'm willing to [start a PR](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) to develop this myself"
-          required: false
-        - label: "I have [donated money](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) to go towards fixing this issue"
-          required: false
-  
-  - type: checkboxes
-    id: satisfaction_survey
-    attributes:
-      label: Mini Survey
-      description: How do you like ArchiveBox so far?
-      options:
-        - label: "I like ArchiveBox so far / would recommend it to a friend"
-          required: false
-        - label: "I've had a lot of difficulty getting ArchiveBox set up"
-          required: false
-  - type: markdown
-    attributes:
-      value: |
-        ---
-          
-        We strive to answer issues as quickly as possible, it usually takes us *about a ~week* to respond.  
-        Make sure your `data/` is [**fully backed up**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout) before trying any workarounds or BETAs suggested here, **we are not responsible for data loss**.  
-          
-        In the meantime please consider:  
-          
-          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
-          - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)
@@ -1,51 +0,0 @@
-name: 📑 Documentation improvement
-description: Submit an idea or correction for the Wiki documentation
-title: "Documentation: ..."
-labels: 'touches: docs'
-type: 'Enhancement'
-assignees:
-  - pirate
-body:
-  - type: markdown
-    attributes:
-      value: |
-        If you prefer, you can submit a [Pull Request](https://github.com/ArchiveBox/docs) on https://github.com/ArchiveBox/docs to edit the docs directly instead.
-
-  - type: input
-    id: page_url
-    validations:
-      required: true
-    attributes:
-      label: "What is the URL of the page you'd like to see improved?"
-      placeholder: e.g. https://github.com/ArchiveBox/docs/wiki/Install
-
-  - type: input
-    id: section_title
-    validations:
-      required: true
-    attributes:
-      label: "What is the title of the relevant section?"
-      placeholder: e.g. Option B. Automatic Setup Script
-
-  - type: textarea
-    id: suggested_edit
-    attributes:
-      label: "What is the suggested edit?"
-      placeholder: |
-        e.g. Please document how to run the automatic setup script for ArchiveBox on TempleOS.  
-        Attach images, screenshots, code snippets, etc. anything you think would help.
-    validations:
-      required: true
-
-  - type: markdown
-    attributes:
-      value: |
-        ---
-          
-        We strive to address issues as quickly as possible, it usually takes us *about a ~week* to respond.  
-          
-        In the meantime please consider:  
-          
-          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
-          - 🔍 [Checking out the new ReadTheDocs Documentation](https://docs.archivebox.io/)
-          - ✨ Helping us test a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases)
@@ -1,8 +0,0 @@
-blank_issues_enabled: false
-contact_links:
-  - name: ❓ Ask a question or start a discussion
-    url: https://github.com/ArchiveBox/ArchiveBox/discussions
-    about: "Ask a question, get support, or start a design discussion (to report a problem please use '🐞 Bug report' instead)"
-  - name: 💬 Chat with the dev team & community on Zulip
-    url: https://zulip.archivebox.io
-    about: "Join us on our Zulip forum to chat with the developers and other users (it's similar to Discord but self-hosted)."
@@ -1,18 +0,0 @@
-<!-- IMPORTANT: Do not submit PRs with only formatting / PEP8 / line length changes. -->
-
-# Summary
-
-<!--e.g. This PR fixes ABC or adds the ability to do XYZ...-->
-
-# Related issues
-
-<!-- e.g. #123 or Roadmap goal # https://github.com/pirate/ArchiveBox/wiki/Roadmap -->
-
-# Changes these areas
-
- [ ] Bugfixes
- [ ] Feature behavior
- [ ] Command line interface
- [ ] Configuration options
- [ ] Internal architecture
- [ ] Snapshot data layout on disk
@@ -1,68 +0,0 @@
-# Security Policy
-
---
-
-## Security Documentation
-
-Please start by reading these links to understand our existing security model, known issues, API surfaces, and setup steps:
-
- https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview
- https://github.com/ArchiveBox/ArchiveBox/security/advisories/
- https://github.com/ArchiveBox/ArchiveBox/tree/v0.7.4#security-risks-of-viewing-archived-js
- https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#security-concerns
- https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-up-Authentication
- https://github.com/ArchiveBox/ArchiveBox#archive-layout
- https://github.com/ArchiveBox/ArchiveBox#archivebox-development
- https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives
- https://github.com/ArchiveBox/ArchiveBox/wiki/Troubleshooting
-
---
-
-## Reporting a Vulnerability
-
-~We use Github's built-in [Private Reporting](https://docs.github.com/en/code-security/security-advisories/guidance-on-reporting-and-writing-information-about-vulnerabilities/privately-reporting-a-security-vulnerability) feature to accept vulnerability reports.~ We no longer offer CVEs or accept private reporting, 95% of the reports we received were low quality AI slop, so now we just ask for normal Github Issues or PRs.
-
-You can contact the maintainers via our public [Zulip Chat Server zulip.archivebox.io](https://zulip.archivebox.io) or [Twitter DMs @ArchiveBoxApp](https://twitter.com/ArchiveBoxApp) if you have a truly urgent security report, but we do not issue public CVEs.
-
---
-
-## CVEs and Report Credits
-
-> [!IMPORTANT]
-> **Reports that only affect `dev` or beta versions will be converted to normal Github issues and will not be issued CVEs.**  
-> Reporters **will still receive credit in release notes** however.
-
-> [!TIP]
-> ~Reports that affect **stable published versions will be issued CVEs** and published w/ full credit.~
-
-Please read our existing published [security advisories](https://github.com/ArchiveBox/ArchiveBox/security/advisories), [Security Overview Docs](https://github.com/ArchiveBox/ArchiveBox/wiki/Security-Overview), and [issues](https://github.com/ArchiveBox/ArchiveBox/issues) and avoid creating duplicates.
-
---
-
-## Threat Model
-
-### All admin users are assumed to be trusted
-
-ArchiveBox does not support granular user permissions, there are only two types of users available in the Admin UI & REST API:
-
- logged in admin superusers (which are assumed to have `root` on the ArchiveBox host machine already)
- non-logged in anonymous guest users
-
-Because admin / REST API / CLI / Python API users are already assumed to have `root` on the host machine, there are deliberately no protections against admins inputting malicious config values, attempts to attack other admin users, or destructive/malicious actions. Permissions for non-logged in users are managed via the [`PERMISSIONS`](https://github.com/archiveBox/archiveBox/wiki/configuration#permissions) + [`PUBLIC_INDEX` + `PUBLIC_ADD_VIEW`](https://github.com/archiveBox/archiveBox/wiki/configuration#public_index--public_add_view) + other config options.
-
-### Sanitization of Config Values, Filesystem Paths, DB Fields, etc.
-
-Non-logged in users can only choose between public Persona config presets on the `/add/` page, but they should never be allowed to directly set config options like `*_ARGS`, `*_BINARY` or DB fields like `url`, `title`, `tag.name`, `notes` without sanitization/filtering on save and/or use.
-
-Non-logged in users should be tightly limited to prevent attacks like SQL/shell injection, path traversal attacks, XSS, cookie leaks between admin/non-admin domains, etc. Any reports related to non-logged-in users will be prioritized as this is a surface we care a lot about hardening.
-
-### XSS, CSRF, CORS, CSP
-
-`< v0.9.x` versions do not include any XSS/CSRF protections at all, as documented in the [readme](https://github.com/ArchiveBox/ArchiveBox/tree/v0.7.4#security-risks-of-viewing-archived-js), [wiki](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive#security-concerns), [issue #239](https://github.com/ArchiveBox/ArchiveBox/issues/239), and [existing advisory](https://github.com/ArchiveBox/ArchiveBox/security/advisories/GHSA-cr45-98w9-gwqx). Do not open advisories related to XSS or CSRF for those versions, as they are missing fundamental architectural changes added in `v0.9.0` needed to support per-snapshot replay isolation.
-
-[`SERVER_SECURITY_MODE`](https://github.com/archiveBox/archiveBox/wiki/configuration#server_security_mode) + CSRF/XSS defenses were only added in the new 0.9.0 `dev` work and are planned to land in `main` in mid-2026.
-
-### SSRF / URL Validation
-
-Note archivebox is **designed to be able to archive any URL the server can reach**, including internal intranet URLs, private IPs, and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` or proper firewall/ACL-level blocking at the network level if they need to limit what the server should see.
-It does a basic check to try and avoid recursively archiving snapshots hosted by the archivebox server itself, but it's not a strict security check, it's just to prevent wasted CPU/disk from recursively archiving snapshots in a browser that has the ArchiveBox Browser Extension turned on.
@@ -1,81 +0,0 @@
-{
-  "properties": {
-    "CC_BINARY": {
-      "default": "cc"
-    },
-    "LDAPSEARCH_BINARY": {
-      "default": "ldapsearch"
-    },
-    "INITDB_BINARY": {
-      "default": "initdb"
-    },
-    "PG_CTL_BINARY": {
-      "default": "pg_ctl"
-    }
-  },
-  "required_binaries": [
-    {
-      "name": "{CC_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "build-essential"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{LDAPSEARCH_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "env": {
-          "version": [
-            "ldapsearch",
-            "-VV"
-          ]
-        },
-        "apt": {
-          "install_args": [
-            "ldap-utils",
-            "python3-dev",
-            "python3-setuptools",
-            "libssl-dev",
-            "libldap2-dev",
-            "libsasl2-dev",
-            "zlib1g-dev",
-            "libatomic1"
-          ],
-          "version": [
-            "ldapsearch",
-            "-VV"
-          ]
-        }
-      }
-    }
-  ],
-  "postgres_binaries": [
-    {
-      "name": "{INITDB_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "postgresql"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{PG_CTL_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "postgresql"
-          ]
-        }
-      }
-    }
-  ]
-}
@@ -1,13 +0,0 @@
-{
-  "properties": {
-    "BREW_BINARY": {
-      "default": "brew"
-    }
-  },
-  "required_binaries": [
-    {
-      "name": "{BREW_BINARY}",
-      "binproviders": "env"
-    }
-  ]
-}
@@ -1,44 +0,0 @@
-{
-  "properties": {
-    "CLANG_BINARY": {
-      "default": "clang"
-    },
-    "LDAPVC_BINARY": {
-      "default": "ldapvc"
-    }
-  },
-  "required_binaries": [
-    {
-      "name": "{CLANG_BINARY}",
-      "binproviders": "env,brew",
-      "overrides": {
-        "brew": {
-          "install_args": [
-            "llvm"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{LDAPVC_BINARY}",
-      "binproviders": "env,brew",
-      "overrides": {
-        "env": {
-          "version": [
-            "ldapvc",
-            "-VV"
-          ]
-        },
-        "brew": {
-          "install_args": [
-            "openldap"
-          ],
-          "version": [
-            "ldapvc",
-            "-VV"
-          ]
-        }
-      }
-    }
-  ]
-}
@@ -1,21 +0,0 @@
-{
-  "properties": {
-    "LDAPVC_BINARY": {
-      "default": "ldapvc"
-    }
-  },
-  "required_binaries": [
-    {
-      "name": "{LDAPVC_BINARY}",
-      "binproviders": "env",
-      "overrides": {
-        "env": {
-          "version": [
-            "ldapvc",
-            "-VV"
-          ]
-        }
-      }
-    }
-  ]
-}
@@ -1,345 +0,0 @@
-{
-  "properties": {
-    "CURL_BINARY": {
-      "default": "curl"
-    },
-    "BASH_BINARY": {
-      "default": "bash"
-    },
-    "CHMOD_BINARY": {
-      "default": "chmod"
-    },
-    "DOCKER_BINARY": {
-      "default": "docker"
-    },
-    "GIT_BINARY": {
-      "default": "git"
-    },
-    "GREP_BINARY": {
-      "default": "grep"
-    },
-    "GH_BINARY": {
-      "default": "gh"
-    },
-    "EXPORTFS_BINARY": {
-      "default": "exportfs"
-    },
-    "JQ_BINARY": {
-      "default": "jq"
-    },
-    "MKDIR_BINARY": {
-      "default": "mkdir"
-    },
-    "MOUNT_CIFS_BINARY": {
-      "default": "mount.cifs"
-    },
-    "MOUNT_NFS_BINARY": {
-      "default": "mount.nfs"
-    },
-    "NODE_BINARY": {
-      "default": "node"
-    },
-    "PS_BINARY": {
-      "default": "ps"
-    },
-    "PYTHON_BINARY": {
-      "default": "python"
-    },
-    "PV_BINARY": {
-      "default": "pv"
-    },
-    "SSH_BINARY": {
-      "default": "ssh"
-    },
-    "SED_BINARY": {
-      "default": "sed"
-    },
-    "SMBD_BINARY": {
-      "default": "smbd"
-    },
-    "SUDO_BINARY": {
-      "default": "sudo"
-    },
-    "SYSTEMCTL_BINARY": {
-      "default": "systemctl"
-    },
-    "TAR_BINARY": {
-      "default": "tar"
-    },
-    "TREE_BINARY": {
-      "default": "tree"
-    },
-    "TR_BINARY": {
-      "default": "tr"
-    },
-    "RM_BINARY": {
-      "default": "rm"
-    },
-    "UMOUNT_BINARY": {
-      "default": "umount"
-    },
-    "UV_BINARY": {
-      "default": "uv"
-    },
-    "UNAME_BINARY": {
-      "default": "uname"
-    }
-  },
-  "ci_binaries": [
-    {
-      "name": "{BASH_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{PYTHON_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{JQ_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{UV_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "bsd_binaries": [
-    {
-      "name": "{BASH_BINARY}",
-      "binproviders": "env,pyinfra"
-    }
-  ],
-  "freebsd_binaries": [
-    {
-      "name": "{SUDO_BINARY}",
-      "binproviders": "env,pyinfra"
-    }
-  ],
-  "docker_binaries": [
-    {
-      "name": "{DOCKER_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{GIT_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{PYTHON_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{UNAME_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "deploy_binaries": [
-    {
-      "name": "{GIT_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{PYTHON_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{SSH_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "docker_debug_binaries": [
-    {
-      "name": "{DOCKER_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{PV_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{TAR_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{TREE_BINARY}",
-      "binproviders": "env,brew,apt"
-    }
-  ],
-  "docker_validation_binaries": [
-    {
-      "name": "{DOCKER_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{SUDO_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{GREP_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{MKDIR_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{RM_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{SED_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{SSH_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{TR_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{UNAME_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "mount_validation_binaries": [
-    {
-      "name": "{BASH_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{CHMOD_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{EXPORTFS_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "nfs-kernel-server"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{MKDIR_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{MOUNT_CIFS_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "cifs-utils"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{MOUNT_NFS_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "nfs-common"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{SMBD_BINARY}",
-      "binproviders": "env,apt",
-      "overrides": {
-        "apt": {
-          "install_args": [
-            "samba"
-          ]
-        }
-      }
-    },
-    {
-      "name": "{SUDO_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{SYSTEMCTL_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{UMOUNT_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "process_binaries": [
-    {
-      "name": "{PS_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "docs_binaries": [
-    {
-      "name": "{CURL_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{SUDO_BINARY}",
-      "binproviders": "env"
-    }
-  ],
-  "git_binaries": [
-    {
-      "name": "{GIT_BINARY}",
-      "binproviders": "env,brew,apt"
-    }
-  ],
-  "github_binaries": [
-    {
-      "name": "{GH_BINARY}",
-      "binproviders": "env,brew,apt"
-    }
-  ],
-  "node_binaries": [
-    {
-      "name": "{NODE_BINARY}",
-      "binproviders": "env,npm",
-      "min_version": "22.23.1",
-      "overrides": {
-        "npm": {
-          "install_args": [
-            "node@22.23.1"
-          ],
-          "postinstall_scripts": true
-        }
-      }
-    }
-  ],
-  "release_binaries": [
-    {
-      "name": "{UV_BINARY}",
-      "binproviders": "env,pip,brew"
-    },
-    {
-      "name": "{PYTHON_BINARY}",
-      "binproviders": "env"
-    },
-    {
-      "name": "{GH_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{GIT_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{JQ_BINARY}",
-      "binproviders": "env,brew,apt"
-    },
-    {
-      "name": "{CURL_BINARY}",
-      "binproviders": "env,brew,apt"
-    }
-  ]
-}
@@ -1,25 +0,0 @@
-# To get started with Dependabot version updates, you'll need to specify which
-# package ecosystems to update and where the package manifests are located.
-# Please see the documentation for all configuration options:
-# https://docs.github.com/code-security/dependabot/dependabot-version-updates/configuration-options-for-the-dependabot.yml-file
-
-version: 2
-updates:
-  - package-ecosystem: "pip"
-    directory: "/"
-    target-branch: "dev"
-    schedule:
-      interval: "monthly"
-    groups:
-      pip:
-        patterns:
-          - "*"
-  - package-ecosystem: "npm"
-    directory: "/"
-    target-branch: "dev"
-    schedule:
-      interval: "monthly"
-    groups:
-      npm:
-        patterns:
-          - "*"
@@ -0,0 +1,25 @@
+# Historical 0.0.3 documentation
+
+This documentation-only branch starts at the immutable `v0.0.3` tag.
+The original README, application files, and pinned wiki (when present) remain unchanged.
+Sphinx reads them without installing or importing the historical application.
+The render-time adapter fixes historical link and heading syntax only; the visible warning
+explains that these old installation commands and external services may no longer work.
+
+Prerequisites: [uv](https://docs.astral.sh/uv/getting-started/installation/) 0.11.3 and Python 3.13, matching CI and Read the Docs. `uv` can install Python when needed.
+
+```sh
+READTHEDOCS=True uv run --no-project --python 3.13 --with-requirements .github/docs/requirements.txt \
+  sphinx-build -E -a -W --keep-going -b html -c .github/docs . /tmp/archivebox-historical-docs
+```
+
+Read the Docs must use `.github/.readthedocs.yaml`. The pinned theme supplies
+its version selector and listens for Read the Docs addon version data.
+
+`requirements.txt` pins the complete documentation dependency set. To update it,
+edit `requirements.in` and run:
+
+```sh
+uv pip compile --universal --python-version 3.13 .github/docs/requirements.in \
+  --output-file .github/docs/requirements.txt
+```
@@ -0,0 +1,124 @@
+"""Render this release's historical documentation without importing the application."""
+
+import os
+import re
+from pathlib import Path
+from urllib.parse import urlsplit
+
+from markdown_it import MarkdownIt
+
+ROOT = Path(__file__).resolve().parents[2]
+release = "0.0.3"
+version = release
+project = "ArchiveBox"
+author = "ArchiveBox contributors"
+copyright = "ArchiveBox contributors"
+extensions = ["myst_parser", "sphinx_rtd_theme"]
+source_suffix = {".md": "markdown"}
+root_doc = "index"
+exclude_patterns = [
+    p.name for p in ROOT.iterdir() if p.name not in {"index.md", "docs"}
+]
+exclude_patterns += ["docs/_Sidebar.md", "docs/_Footer.md", "docs/.git"]
+html_theme = "sphinx_rtd_theme"
+html_theme_options = {
+    "navigation_depth": 3,
+    "collapse_navigation": False,
+    "version_selector": True,
+    "language_selector": False,
+}
+html_context = {"current_version": os.environ.get("READTHEDOCS_VERSION", release)}
+html_title = f"ArchiveBox {release} historical documentation"
+html_show_sphinx = False
+myst_heading_anchors = 6
+
+
+def rewrite_prose(text, transform):
+    """Rewrite navigation outside fenced and indented code blocks."""
+    lines = text.splitlines(keepends=True)
+    output = []
+    start = 0
+    for token in MarkdownIt().parse(text):
+        if token.type not in {"fence", "code_block"}:
+            continue
+        first, last = token.map
+        output.append(transform("".join(lines[start:first])))
+        output.append("".join(lines[first:last]))
+        start = last
+    output.append(transform("".join(lines[start:])))
+    return "".join(output)
+
+
+def prepare_source(app, docname, source):
+    text = (ROOT / "README.md").read_text() if docname == "index" else source[0]
+    directory = ROOT if docname == "index" else ROOT / "docs"
+
+    def wiki(match):
+        parts = match.group(1).split("|", 1)
+        return f"[{parts[0]}]({parts[-1].replace(' ', '-')}.md)"
+
+    text = rewrite_prose(text, lambda prose: re.sub(r"\[\[([^\]\n]+)\]\]", wiki, prose))
+
+    def resource(target):
+        if urlsplit(target).scheme or target.startswith(("#", "//")):
+            return target
+        page, sep, anchor = target.partition("#")
+        path = directory / page
+        if path.suffix == ".md" and path.parent == ROOT / "docs":
+            return target
+        if path.is_file():
+            return (
+                f"https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/v{release}/{path.relative_to(ROOT)}"
+                + ("#" + anchor if sep else "")
+            )
+        return target
+
+    def rewrite_links(prose):
+        prose = re.sub(
+            r'((?:src|href)=["\'])([^"\']+)', lambda m: m[1] + resource(m[2]), prose
+        )
+        prose = re.sub(r"\]\(([^)\s]+)\)", lambda m: "](" + resource(m[1]) + ")", prose)
+        return prose
+
+    text = rewrite_prose(text, rewrite_links)
+    text = (
+        f"# ArchiveBox {release}"
+        + (
+            " historical guide"
+            if docname == "index"
+            else " — " + Path(docname).name.replace("-", " ")
+        )
+        + "\n\n"
+        + text
+    )
+    lines = text.splitlines()
+    stack = []
+    for token in MarkdownIt().parse(text):
+        if token.type != "heading_open":
+            continue
+        original = int(token.tag[1])
+        while stack and stack[-1] >= original:
+            stack.pop()
+        stack.append(original)
+        lines[token.map[0]] = re.sub(
+            r"^#{1,6}(?=\s)", "#" * len(stack), lines[token.map[0]]
+        )
+    text = "\n".join(lines) + "\n"
+    if docname == "index":
+        notice = f"\n```{{warning}}\nThis is the original guide shipped with ArchiveBox {release}, preserved for historical reference. Commands, dependencies, and external services describe that old release and may no longer work. Use the version selector for current documentation.\n```\n"
+        first, rest = text.split("\n", 1)
+        text = first + "\n" + notice + rest
+        pages = sorted(
+            p.stem for p in (ROOT / "docs").glob("*.md") if not p.name.startswith("_")
+        )
+        if pages:
+            text += (
+                "\n```{toctree}\n:maxdepth: 2\n\n"
+                + "\n".join("docs/" + p for p in pages)
+                + "\n```\n"
+            )
+    source[0] = text
+
+
+def setup(app):
+    app.connect("source-read", prepare_source)
@@ -0,0 +1,3 @@
+Sphinx==8.2.3
+myst-parser==4.0.1
+sphinx-rtd-theme==3.1.0
@@ -0,0 +1,75 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --universal --python-version 3.13 .github/docs/requirements.in --output-file .github/docs/requirements.txt
+alabaster==1.0.0
+    # via sphinx
+babel==2.18.0
+    # via sphinx
+certifi==2026.7.22
+    # via requests
+charset-normalizer==3.5.1
+    # via requests
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via sphinx
+docutils==0.21.2
+    # via
+    #   myst-parser
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.20
+    # via requests
+imagesize==2.0.1
+    # via sphinx
+jinja2==3.1.6
+    # via
+    #   myst-parser
+    #   sphinx
+markdown-it-py==3.0.0
+    # via
+    #   mdit-py-plugins
+    #   myst-parser
+markupsafe==3.0.3
+    # via jinja2
+mdit-py-plugins==0.6.1
+    # via myst-parser
+mdurl==0.1.2
+    # via markdown-it-py
+myst-parser==4.0.1
+    # via -r .github/docs/requirements.in
+packaging==26.3
+    # via sphinx
+pygments==2.21.0
+    # via sphinx
+pyyaml==6.0.3
+    # via myst-parser
+requests==2.34.2
+    # via sphinx
+roman-numerals==4.1.0
+    # via roman-numerals-py
+roman-numerals-py==4.1.0
+    # via sphinx
+snowballstemmer==3.1.1
+    # via sphinx
+sphinx==8.2.3
+    # via
+    #   -r .github/docs/requirements.in
+    #   myst-parser
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-jquery
+sphinx-rtd-theme==3.1.0
+    # via -r .github/docs/requirements.in
+sphinxcontrib-applehelp==2.0.0
+    # via sphinx
+sphinxcontrib-devhelp==2.0.0
+    # via sphinx
+sphinxcontrib-htmlhelp==2.1.0
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-qthelp==2.0.0
+    # via sphinx
+sphinxcontrib-serializinghtml==2.0.0
+    # via sphinx
+urllib3==2.8.0
+    # via requests
@@ -1,39 +0,0 @@
-#!/usr/bin/env python3
-"""Discover every ArchiveBox test file and expose it as one CI job."""
-
-import json
-from pathlib import Path
-
-
-def main() -> None:
-    root = Path.cwd().resolve()
-    archivebox_tests = sorted((root / "archivebox/tests").glob("test_*.py"))
-
-    if not archivebox_tests:
-        raise SystemExit("No ArchiveBox tests discovered")
-
-    matrix: list[dict[str, object]] = []
-    for path in archivebox_tests:
-        test_path = path.relative_to(root).as_posix()
-        matrix.append(
-            {
-                "name": f"main/{path.stem}",
-                "paths": [test_path],
-                "paths_arg": test_path,
-                "extra": "ldap" if test_path.endswith("/test_auth_ldap.py") else "",
-                "count": 1,
-            },
-        )
-
-    discovered_paths = [path for entry in matrix for path in entry["paths"]]
-    if len(discovered_paths) != len(set(discovered_paths)):
-        raise SystemExit("Tests were not discovered exactly once")
-    if sorted(discovered_paths) != [path.relative_to(root).as_posix() for path in archivebox_tests]:
-        raise SystemExit("Discovered test shard coverage does not match test files")
-
-    print(f"Discovered {len(discovered_paths)} test files exactly once across {len(matrix)} jobs")
-    print(json.dumps(matrix, separators=(",", ":")))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,92 +0,0 @@
-name: CI
-
-on:
-  pull_request:
-    branches: [dev, main]
-  push:
-    branches: [dev, main]
-  workflow_dispatch:
-
-permissions:
-  actions: read
-  contents: read
-  packages: write
-  security-events: write
-
-concurrency:
-  group: ci-${{ github.event_name == 'pull_request' && format('pr-{0}', github.event.pull_request.number) || github.event_name == 'workflow_dispatch' && format('manual-{0}', github.run_id) || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
-
-jobs:
-  lint:
-    name: Linters
-    uses: ./.github/workflows/lint.yml
-    secrets: inherit
-
-  install-cli-platform:
-    name: Install and CLI platform compatibility
-    uses: ./.github/workflows/test.yml
-    secrets: inherit
-
-  tests:
-    name: Discovered test matrix
-    uses: ./.github/workflows/test-parallel.yml
-    secrets: inherit
-
-  documentation:
-    name: Documentation and root tests
-    uses: ./.github/workflows/docs.yml
-    secrets: inherit
-
-  codeql:
-    name: CodeQL
-    uses: ./.github/workflows/codeql.yml
-    secrets: inherit
-
-  python-artifacts:
-    name: Tested Python artifacts
-    uses: ./.github/workflows/pip.yml
-    secrets: inherit
-
-  docker-digests:
-    name: Tested Docker images
-    uses: ./.github/workflows/docker.yml
-    with:
-      push_digests: false
-    secrets: inherit
-
-  required:
-    name: All required CI lanes
-    if: ${{ always() }}
-    needs:
-      - lint
-      - install-cli-platform
-      - tests
-      - documentation
-      - codeql
-      - python-artifacts
-      - docker-digests
-    runs-on: ubuntu-24.04
-    steps:
-      - name: Verify every required lane succeeded
-        env:
-          LINT_RESULT: ${{ needs.lint.result }}
-          INSTALL_CLI_PLATFORM_RESULT: ${{ needs.install-cli-platform.result }}
-          TESTS_RESULT: ${{ needs.tests.result }}
-          DOCUMENTATION_RESULT: ${{ needs.documentation.result }}
-          CODEQL_RESULT: ${{ needs.codeql.result }}
-          PYTHON_ARTIFACTS_RESULT: ${{ needs.python-artifacts.result }}
-          DOCKER_DIGESTS_RESULT: ${{ needs.docker-digests.result }}
-        run: |
-          for result in \
-            "$LINT_RESULT" \
-            "$INSTALL_CLI_PLATFORM_RESULT" \
-            "$TESTS_RESULT" \
-            "$DOCUMENTATION_RESULT" \
-            "$CODEQL_RESULT" \
-            "$PYTHON_ARTIFACTS_RESULT" \
-            "$DOCKER_DIGESTS_RESULT"
-          do
-            [[ "$result" == success ]]
-          done
-          echo 'All required CI lanes and tested release artifacts succeeded.'
@@ -1,40 +0,0 @@
-name: Claude Code
-
-on:
-  issue_comment:
-    types: [created]
-  pull_request_review_comment:
-    types: [created]
-  issues:
-    types: [opened, assigned]
-  pull_request_review:
-    types: [submitted]
-
-jobs:
-  claude:
-    if: |
-      (github.event_name == 'issue_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review_comment' && contains(github.event.comment.body, '@claude')) ||
-      (github.event_name == 'pull_request_review' && contains(github.event.review.body, '@claude')) ||
-      (github.event_name == 'issues' && (contains(github.event.issue.body, '@claude') || contains(github.event.issue.title, '@claude')))
-    runs-on: ubuntu-24.04
-    permissions:
-      contents: read
-      pull-requests: write
-      issues: write
-      id-token: write
-      actions: read
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Run Claude Code
-        id: claude
-        uses: anthropics/claude-code-action@b76a0776ae74036e77cd11018083743453d7ad35 # v1
-        with:
-          anthropic_api_key: ${{ secrets.ANTHROPIC_API_KEY }}
-          additional_permissions: |
-            actions: read
-          claude_args: '--allowed-tools Bash(gh pr:*)'
@@ -1,42 +0,0 @@
-name: "CodeQL"
-
-on:
-  workflow_call:
-  schedule:
-    - cron: '33 17 * * 6'
-
-concurrency:
-  group: codeql-${{ github.repository }}-${{ github.workflow }}-${{ github.event_name }}-${{ github.ref }}
-  cancel-in-progress: true
-
-jobs:
-  analyze:
-    name: Analyze (${{ matrix.language }})
-    runs-on: ubuntu-24.04
-    permissions:
-      security-events: write
-      packages: read
-      actions: read
-      contents: read
-
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-        - language: python
-          build-mode: none
-        - language: javascript-typescript
-          build-mode: none
-    steps:
-    - name: Checkout repository
-      uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-
-    - name: Initialize CodeQL
-      uses: github/codeql-action/init@b7351df727350dca84cb9d725d57dcf5bc82ba26 # v3
-      with:
-        languages: ${{ matrix.language }}
-        build-mode: ${{ matrix.build-mode }}
-    - name: Perform CodeQL Analysis
-      uses: github/codeql-action/analyze@b7351df727350dca84cb9d725d57dcf5bc82ba26 # v3
-      with:
-        category: "/language:${{matrix.language}}"
@@ -1,78 +0,0 @@
-name: Deploy Publicsite to GitHub Pages
-
-on:
-  push:
-    branches:
-      - dev
-    paths:
-      - publicsite/**
-      - archivebox/**
-      - bin/collect_ui_screenshots.*
-      - bin/generate_ui_screenshot_gallery.py
-      - bin/take_screenshot.js
-      - pyproject.toml
-      - uv.lock
-      - .github/workflows/deploy-publicsite.yml
-  workflow_dispatch:
-
-permissions:
-  contents: read
-  pages: write
-  id-token: write
-
-concurrency:
-  group: ${{ github.event_name == 'workflow_dispatch' && format('github-pages-publicsite-manual-{0}', github.run_id) || 'github-pages-publicsite' }}
-  cancel-in-progress: ${{ github.event_name != 'workflow_dispatch' }}
-
-jobs:
-  deploy:
-    environment:
-      name: github-pages
-      url: ${{ steps.deployment.outputs.page_url }}
-    runs-on: ubuntu-24.04
-    steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Setup Python and uv
-        uses: astral-sh/setup-uv@v6
-        with:
-          enable-cache: false
-
-      - name: Install ArchiveBox and screenshot dependencies
-        run: uv sync --no-cache --frozen
-
-      - name: Build UI screenshot gallery
-        run: bin/collect_ui_screenshots.sh
-        env:
-          BASE_URL: http://archivebox.localhost:8000
-
-      - name: Verify gallery was generated from this exact revision
-        run: |
-          uv run --no-cache --no-sync python - <<'PY'
-          import json
-          import os
-          import tomllib
-          from pathlib import Path
-
-          provenance = json.loads(Path("publicsite/screenshots/build.json").read_text())
-          version = tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"]
-          assert provenance["revision"] == os.environ["GITHUB_SHA"], provenance
-          assert provenance["version"] == version, provenance
-          assert provenance["capture_count"] >= 3 * 35, provenance
-          assert len(provenance["files"]) == provenance["capture_count"], provenance
-          PY
-
-      - name: Setup Pages
-        uses: actions/configure-pages@983d7736d9b0ae728b81ab479565c72886d7745b # v5
-
-      - name: Upload artifact
-        uses: actions/upload-pages-artifact@56afc609e74202658d3ffba0e8f6dda462b719fa # v3
-        with:
-          path: ./publicsite
-
-      - name: Deploy to GitHub Pages
-        id: deployment
-        uses: actions/deploy-pages@d6db90164ac5ed86f2b6aed7e0febac5b3c0c03e # v4
@@ -1,514 +0,0 @@
-name: Build Docker image
-
-on:
-  workflow_call:
-    inputs:
-      push_digests:
-        description: Push validated per-platform images by digest for the release workflow
-        required: false
-        default: false
-        type: boolean
-      full_tests:
-        description: Run storage, CLI, and size validation in addition to release smoke checks
-        required: false
-        default: true
-        type: boolean
-
-env:
-  DOCKERHUB_IMAGE: archivebox/archivebox
-  GHCR_IMAGE: ghcr.io/archivebox/archivebox
-
-permissions:
-  actions: read
-  contents: read
-  packages: write
-
-jobs:
-  build:
-    name: build ${{ matrix.platform }}
-    runs-on: ${{ matrix.runner }}
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - platform: linux/amd64
-            runner: ubuntu-24.04
-            artifact_name: digest-linux-amd64
-            local_tag: archivebox-ci:amd64
-          - platform: linux/arm64
-            runner: ubuntu-24.04-arm
-            artifact_name: digest-linux-arm64
-            local_tag: archivebox-ci:arm64
-    steps:
-      - name: Checkout
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Resolve Docker through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_SPEC="$(uv run --no-cache --no-project python - <<'PY'
-          import tomllib
-          from pathlib import Path
-
-          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
-          matches = [package for package in packages if package["name"] == "abxpkg"]
-          if len(matches) != 1:
-              raise SystemExit(f"Expected one locked abxpkg package, found {len(matches)}")
-          wheel = matches[0]["wheels"][0]
-          print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))
-          PY
-          )"
-          test -n "$ABXPKG_SPEC"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
-          echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR" >> "$GITHUB_ENV"
-          echo "ABXPKG_SPEC=$ABXPKG_SPEC" >> "$GITHUB_ENV"
-          echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
-
-          docker_env="$(
-            uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
-              --install \
-              --json \
-              --lib="$ABXPKG_LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries"
-          )"
-          JQ_BINARY="$ABXPKG_LIB_DIR/env/bin/jq"
-          "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
-            <<< "$docker_env" >> "$GITHUB_ENV"
-
-          docker_binary="$ABXPKG_LIB_DIR/env/bin/docker"
-          UV_BINARY="$ABXPKG_LIB_DIR/env/bin/uv"
-          {
-            echo "DOCKER_BINARY=$docker_binary"
-            echo "JQ_BINARY=$JQ_BINARY"
-            echo "UV_BINARY=$UV_BINARY"
-          } >> "$GITHUB_ENV"
-          for binary in "$docker_binary" "$JQ_BINARY" "$UV_BINARY"; do
-            test -L "$binary"
-            test -x "$binary"
-          done
-          "$docker_binary" --version
-
-      - name: Set up Docker Buildx
-        id: buildx
-        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
-        with:
-          install: true
-
-      - name: Builder instance name
-        run: echo ${{ steps.buildx.outputs.name }}
-
-      - name: Available platforms
-        run: echo ${{ steps.buildx.outputs.platforms }}
-
-      - name: Verify published abx-dl image
-        id: abx_dl_image
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          ABX_DL_VERSION="$("$UV_BINARY" run --no-cache --no-project python - <<'PY'
-          import tomllib
-          from pathlib import Path
-
-          packages = tomllib.loads(Path("uv.lock").read_text())["package"]
-          matches = [package["version"] for package in packages if package["name"] == "abx-dl"]
-          if len(matches) != 1:
-              raise SystemExit(f"Expected one locked abx-dl package, found {len(matches)}")
-          print(matches[0])
-          PY
-          )"
-          test -n "$ABX_DL_VERSION"
-          ABX_DL_REFERENCE="${ABX_DL_IMAGE:-archivebox/abx-dl:${ABX_DL_VERSION}}"
-          INSPECT_OUTPUT="$("$DOCKER_BINARY" buildx imagetools inspect "$ABX_DL_REFERENCE")"
-          printf '%s\n' "$INSPECT_OUTPUT"
-          ABX_DL_DIGEST="$(INSPECT_OUTPUT="$INSPECT_OUTPUT" "$UV_BINARY" run --no-cache --no-project python - <<'PY'
-          import os
-
-          digest = next(
-              (fields[1] for line in os.environ["INSPECT_OUTPUT"].splitlines() if (fields := line.split()) and fields[0] == "Digest:"),
-              "",
-          )
-          print(digest)
-          PY
-          )"
-          [[ "$ABX_DL_DIGEST" == sha256:* ]]
-          ABX_DL_IMAGE="${ABX_DL_REFERENCE%%@*}@${ABX_DL_DIGEST}"
-          echo "image=${ABX_DL_IMAGE}" >> "$GITHUB_OUTPUT"
-
-      - name: Login to Docker Hub
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
-        if: inputs.push_digests
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-
-      - name: Login to GitHub Container Registry
-        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
-        if: inputs.push_digests
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ secrets.GITHUB_TOKEN }}
-
-      - name: Collect Docker labels
-        id: docker_meta
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$("$UV_BINARY" run --no-cache --no-project python - <<'PY'
-          import tomllib
-          from pathlib import Path
-
-          print(tomllib.loads(Path("pyproject.toml").read_text())["project"]["version"])
-          PY
-          )"
-          test -n "$VERSION"
-
-          {
-            echo 'labels<<EOF'
-            echo "org.opencontainers.image.version=${VERSION}"
-            echo "org.opencontainers.image.revision=${GITHUB_SHA}"
-            echo 'org.opencontainers.image.source=https://github.com/ArchiveBox/ArchiveBox'
-            echo "io.archivebox.abx-dl.image=${{ steps.abx_dl_image.outputs.image }}"
-            echo 'EOF'
-            echo "version=${VERSION}"
-          } >> "$GITHUB_OUTPUT"
-
-          echo "[+] Building ${{ matrix.platform }} for ${VERSION} using ${{ steps.abx_dl_image.outputs.image }}"
-
-      - name: Build and push digest
-        id: docker_build
-        if: inputs.push_digests
-        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
-        with:
-          context: ./
-          file: ./Dockerfile
-          builder: ${{ steps.buildx.outputs.name }}
-          labels: ${{ steps.docker_meta.outputs.labels }}
-          build-args: |
-            ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
-            ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
-          pull: true
-          platforms: ${{ matrix.platform }}
-          # Keep the released image as the one cache authority. A second GHA
-          # cache can hold gzip variants of identical layers, and BuildKit may
-          # prefer those over :dev's zstd blobs. That counter-intuitively makes
-          # a cache hit larger than the released image. :dev advances only
-          # after both architectures pass the release checks.
-          cache-from: type=registry,ref=${{ env.DOCKERHUB_IMAGE }}:dev
-          # Inline metadata reuses the released compressed blobs directly;
-          # force-compression would recompress the whole ~740 MiB image. The
-          # compressed-size gate below remains the regression guard.
-          cache-to: |
-            ${{ inputs.push_digests && 'type=inline' || '' }}
-          outputs: |
-            ${{ inputs.full_tests && format('type=oci,dest={0}/archivebox-image.tar,compression=zstd,compression-level=3', runner.temp) || '' }}
-            ${{ inputs.push_digests && format('type=image,"name={0},{1}",push-by-digest=true,name-canonical=true,push=true,compression=zstd,compression-level=3', env.DOCKERHUB_IMAGE, env.GHCR_IMAGE) || '' }}
-
-      - name: Build pull request image
-        id: docker_build_pr
-        if: ${{ !inputs.push_digests }}
-        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
-        with:
-          context: ./
-          file: ./Dockerfile
-          builder: ${{ steps.buildx.outputs.name }}
-          labels: ${{ steps.docker_meta.outputs.labels }}
-          build-args: |
-            ABX_DL_IMAGE=${{ steps.abx_dl_image.outputs.image }}
-            ARCHIVEBOX_COMMIT_HASH=${{ github.sha }}
-          pull: true
-          platforms: ${{ matrix.platform }}
-          # Test the same layer lineage users receive. Reusing :dev avoids a
-          # second cache whose compression can drift from the released image.
-          cache-from: type=registry,ref=${{ env.DOCKERHUB_IMAGE }}:dev
-          outputs: |
-            type=docker,name=${{ matrix.local_tag }}
-            type=oci,dest=${{ runner.temp }}/archivebox-image.tar
-
-      - name: Image digest
-        if: inputs.push_digests
-        run: echo ${{ steps.docker_build.outputs.digest }}
-
-      - name: Validate exact built image version and commit
-        if: ${{ !inputs.push_digests }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          IMAGE="${{ matrix.local_tag }}"
-          SHORT_SHA="${GITHUB_SHA::7}"
-
-          DATA_DIR="$(mktemp -d)"
-          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox init
-          ARCHIVEBOX_VERSION_OUTPUT="$("$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox version)"
-          printf '%s\n' "$ARCHIVEBOX_VERSION_OUTPUT"
-          "$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' -v "$DATA_DIR":/data "$IMAGE" archivebox status
-          IMAGE_VERSION_OUTPUT="$("$DOCKER_BINARY" run --rm --platform '${{ matrix.platform }}' --entrypoint cat "$IMAGE" /VERSION.txt)"
-          ARCHIVEBOX_VERSION_OUTPUT="$ARCHIVEBOX_VERSION_OUTPUT" IMAGE_VERSION_OUTPUT="$IMAGE_VERSION_OUTPUT" \
-            "$UV_BINARY" run --no-cache --no-project python - \
-            "ArchiveBox v${{ steps.docker_meta.outputs.version }}" \
-            "COMMIT_HASH=${SHORT_SHA}" \
-            "COMMIT_HASH=${GITHUB_SHA}" <<'PY'
-          import os
-          import sys
-
-          version_output = os.environ["ARCHIVEBOX_VERSION_OUTPUT"]
-          image_version_lines = os.environ["IMAGE_VERSION_OUTPUT"].splitlines()
-          if sys.argv[1] not in version_output:
-              raise SystemExit(f"Missing version marker: {sys.argv[1]}")
-          if sys.argv[2] not in version_output:
-              raise SystemExit(f"Missing short commit marker: {sys.argv[2]}")
-          if sys.argv[3] not in image_version_lines:
-              raise SystemExit(f"Missing exact commit marker: {sys.argv[3]}")
-          PY
-
-          "$DOCKER_BINARY" run --rm --user 911:911 --platform '${{ matrix.platform }}' --entrypoint /venv/bin/python3 "$IMAGE" -c \
-            'import json, pathlib, stat; cache_file = "/opt/archivebox/lib/env/derived.env"; raw = open(cache_file).read().strip(); records = json.loads(raw[len("ABXPKG_DERIVED_CACHE=\x27"):-1]); projections = [projection for record in records.values() for projection in record.get("request_exec_projections", {}).values()]; plans = [projection["validation"] for projection in projections]; fingerprints = [fingerprint for plan in plans for fingerprint in plan["fingerprint"]]; actual = lambda fingerprint: (lambda path, value: {"path": str(path.resolve(strict=False)), "size": value.st_size, "mtime_ns": value.st_mtime_ns, "mode": stat.S_IMODE(value.st_mode), "euid": value.st_uid})(pathlib.Path(fingerprint["path"]), pathlib.Path(fingerprint["path"]).stat()); mismatches = [(fingerprint, actual(fingerprint)) for fingerprint in fingerprints if fingerprint != actual(fingerprint)]; assert projections and {plan.get("euid") for plan in plans} == {911}; assert all(not record.get("script_exec_plans") for record in records.values()); assert "ABXPKG_TMP_CACHE_DIR" not in json.dumps(projections); assert not mismatches, mismatches[:3]'
-
-      - name: Validate compressed candidate size
-        if: inputs.push_digests
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          DIGEST='${{ steps.docker_build.outputs.digest }}'
-          [[ "$DIGEST" =~ ^sha256:[0-9a-f]{64}$ ]]
-          MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${DIGEST}")"
-          # Provenance wraps the real platform image and its attestation in an
-          # OCI index. Measure the image manifest, not the index/attestation.
-          IMAGE_DIGEST="$(MANIFEST="$MANIFEST" PLATFORM='${{ matrix.platform }}' "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); platform_os, platform_arch = os.environ["PLATFORM"].split("/")[:2]; print(next((item["digest"] for item in manifest.get("manifests", []) if item.get("platform", {}).get("os") == platform_os and item.get("platform", {}).get("architecture") == platform_arch), ""))')"
-          if [[ -n "$IMAGE_DIGEST" ]]; then
-            MANIFEST="$("$DOCKER_BINARY" buildx imagetools inspect --raw "${DOCKERHUB_IMAGE}@${IMAGE_DIGEST}")"
-          fi
-          COMPRESSED_SIZE="$(MANIFEST="$MANIFEST" "$UV_BINARY" run --no-cache --no-project python -c 'import json, os; manifest = json.loads(os.environ["MANIFEST"]); print(manifest["config"]["size"] + sum(layer["size"] for layer in manifest["layers"]))')"
-          LIMIT=$((750 * 1024 * 1024))
-          echo "${DIGEST} compressed_size=$((COMPRESSED_SIZE / 1024 / 1024)) MiB limit=750 MiB"
-          (( COMPRESSED_SIZE < LIMIT ))
-
-      - name: Resolve real Docker CLI test dependencies through abxpkg
-        if: inputs.full_tests && matrix.platform == 'linux/amd64'
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mount_env="$(
-            uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
-              --install \
-              --json \
-              --lib="$ABXPKG_LIB_DIR" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries" \
-              --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:mount_validation_binaries"
-          )"
-          "$JQ_BINARY" -r 'to_entries[] | "\(.key)=\(.value)"' \
-            <<< "$mount_env" >> "$GITHUB_ENV"
-
-          for binary_name in bash chmod curl exportfs mkdir mount.cifs mount.nfs smbd sudo systemctl umount; do
-            binary="$ABXPKG_LIB_DIR/env/bin/$binary_name"
-            test -L "$binary"
-            test -x "$binary"
-          done
-
-      - name: Validate Docker UID and GID behavior on real NFS and SMB mounts
-        if: inputs.full_tests && matrix.platform == 'linux/amd64'
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          NFS_SOURCE="/srv/archivebox-ci-nfs"
-          SMB_SOURCE="/srv/archivebox-ci-smb"
-          NFS_MOUNT="$RUNNER_TEMP/archivebox-nfs-mount"
-          SMB_MOUNT="$RUNNER_TEMP/archivebox-smb-mount"
-          mounted_nfs=0
-          mounted_smb=0
-
-          cleanup() {
-            local original_status=$? cleanup_status=0
-            trap - EXIT
-            if [[ "$mounted_smb" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$SMB_MOUNT"; then
-              cleanup_status=1
-            fi
-            if [[ "$mounted_nfs" == "1" ]] && ! "$SUDO_BINARY" "$UMOUNT_BINARY" "$NFS_MOUNT"; then
-              cleanup_status=1
-            fi
-            if ! "$SUDO_BINARY" "$SYSTEMCTL_BINARY" stop smbd.service nfs-kernel-server.service; then
-              cleanup_status=1
-            fi
-            if [[ "$original_status" != "0" ]]; then
-              exit "$original_status"
-            fi
-            exit "$cleanup_status"
-          }
-          trap cleanup EXIT
-
-          "$SUDO_BINARY" "$MKDIR_BINARY" -p "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
-          "$SUDO_BINARY" "$MKDIR_BINARY" -p /etc/exports.d
-          "$SUDO_BINARY" "$CHMOD_BINARY" 0777 "$NFS_SOURCE" "$SMB_SOURCE" "$NFS_MOUNT" "$SMB_MOUNT"
-          "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '%s *(rw,sync,no_subtree_check,root_squash,insecure)\\n' '$NFS_SOURCE' > /etc/exports.d/archivebox-ci.exports"
-          "$SUDO_BINARY" "$BASH_BINARY" -c \
-            "printf '\\n[archivebox-ci]\\npath = %s\\nbrowseable = yes\\nread only = no\\nguest ok = yes\\nguest only = yes\\nforce user = nobody\\ncreate mask = 0777\\ndirectory mask = 0777\\n' '$SMB_SOURCE' >> /etc/samba/smb.conf"
-
-          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart nfs-kernel-server.service
-          "$SUDO_BINARY" "$EXPORTFS_BINARY" -ra
-          "$SYSTEMCTL_BINARY" is-active --quiet nfs-kernel-server.service
-          "$SUDO_BINARY" "$SYSTEMCTL_BINARY" restart smbd.service
-          "$SYSTEMCTL_BINARY" is-active --quiet smbd.service
-
-          "$SUDO_BINARY" "$MOUNT_NFS_BINARY" -o vers=3,nolock "127.0.0.1:$NFS_SOURCE" "$NFS_MOUNT"
-          mounted_nfs=1
-          "$SUDO_BINARY" "$MOUNT_CIFS_BINARY" //127.0.0.1/archivebox-ci "$SMB_MOUNT" \
-            -o "guest,vers=3.0,uid=0,gid=0,forceuid,forcegid,file_mode=0777,dir_mode=0777,nounix"
-          mounted_smb=1
-
-          NFS_TEST_DIR="$NFS_MOUNT" \
-          SMB_TEST_DIR="$SMB_MOUNT" \
-          IMAGE="${{ matrix.local_tag }}" \
-            "$BASH_BINARY" bin/validate_docker_uid_gid.sh --local-only
-
-      - name: Validate Docker CLI add/list and compose behavior
-        if: inputs.full_tests && matrix.platform == 'linux/amd64'
-        shell: bash
-        env:
-          ARCHIVEBOX_IMAGE: ${{ matrix.local_tag }}
-        run: |
-          set -Eeuo pipefail
-          "$MKDIR_BINARY" "${{ github.workspace }}/data"
-          "$DOCKER_BINARY" run -v "${{ github.workspace }}/data":/data "$ARCHIVEBOX_IMAGE" init
-
-          "$SUDO_BINARY" "$BASH_BINARY" -c 'echo "127.0.0.1  www.test-nginx-1.local www.test-nginx-2.local" >> /etc/hosts'
-          "$DOCKER_BINARY" run --name www-nginx -p 80:80 -d nginx:1.29.5@sha256:0236ee02dcbce00b9bd83e0f5fbc51069e7e1161bd59d99885b3ae1734f3392e
-
-          "$DOCKER_BINARY" run -v "$PWD"/data:/data --network host "$ARCHIVEBOX_IMAGE" add http://www.test-nginx-1.local
-          echo "http://www.test-nginx-2.local" | "$DOCKER_BINARY" run -i --network host -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" add
-
-          archive_list="$("$DOCKER_BINARY" run -v "$PWD"/data:/data "$ARCHIVEBOX_IMAGE" list)"
-          [[ "$archive_list" == *"www.test-nginx-1.local"* ]] || { echo "The site 1 isn't in the list"; exit 1; }
-          [[ "$archive_list" == *"www.test-nginx-2.local"* ]] || { echo "The site 2 isn't in the list"; exit 1; }
-
-          "$DOCKER_BINARY" compose run --rm archivebox init
-          "$DOCKER_BINARY" compose up -d --wait
-          "$DOCKER_BINARY" network connect archivebox_default www-nginx
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
-          admin_js="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/static/admin/js/jquery.init.js')"
-          [[ "$index_html" == *ArchiveBox* ]]
-          [[ "$admin_js" == *window.django* ]]
-
-          "$DOCKER_BINARY" compose run --rm archivebox add --plugins=parse_txt_urls 'http://www-nginx/#test_docker'
-          index_html="$("$CURL_BINARY" --silent --location 'http://127.0.0.1:8000/public/')"
-          [[ "$index_html" == *'http://www-nginx/#test_docker'* ]]
-          "$DOCKER_BINARY" compose down
-
-      - name: Validate compressed image size
-        if: inputs.full_tests && !inputs.push_digests
-        env:
-          BUILD_METADATA: ${{ steps.docker_build_pr.outputs.metadata }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          LOCAL_IMAGE_ID="$("$DOCKER_BINARY" image inspect '${{ matrix.local_tag }}' --format '{{.Id}}')"
-          "$UV_BINARY" run --no-cache --no-project python - \
-            '${{ runner.temp }}/archivebox-image.tar' \
-            '${{ steps.docker_build_pr.outputs.digest }}' \
-            "$LOCAL_IMAGE_ID" \
-            "$((750 * 1024 * 1024))" <<'PY'
-          from hashlib import sha256
-          import json
-          import os
-          import sys
-          import tarfile
-
-          archive_path, pushed_digest, local_image_id, limit_text = sys.argv[1:]
-          limit = int(limit_text)
-          metadata = json.loads(os.environ["BUILD_METADATA"])
-          if metadata["containerimage.digest"] != pushed_digest:
-              raise SystemExit("Build push transaction digest does not match the action output")
-          if not pushed_digest.startswith("sha256:") or len(pushed_digest) != 71:
-              raise SystemExit(f"Invalid pushed digest: {pushed_digest}")
-
-          with tarfile.open(archive_path) as archive:
-              def read_blob(descriptor):
-                  algorithm, digest = descriptor["digest"].split(":", 1)
-                  if algorithm != "sha256":
-                      raise SystemExit(f"Unsupported OCI digest: {descriptor['digest']}")
-                  member = archive.extractfile(f"blobs/sha256/{digest}")
-                  if member is None:
-                      raise SystemExit(f"Missing OCI blob: {descriptor['digest']}")
-                  blob = member.read()
-                  if len(blob) != descriptor["size"]:
-                      raise SystemExit(f"OCI size mismatch: {descriptor['digest']}")
-                  if sha256(blob).hexdigest() != digest:
-                      raise SystemExit(f"OCI digest mismatch: {descriptor['digest']}")
-                  return blob
-
-              index_member = archive.extractfile("index.json")
-              if index_member is None:
-                  raise SystemExit("OCI export is missing index.json")
-              index = json.load(index_member)
-
-              manifests = []
-
-              def collect(descriptor):
-                  document = json.loads(read_blob(descriptor))
-                  media_type = descriptor["mediaType"]
-                  if "image.index" in media_type or "manifest.list" in media_type:
-                      for child in document["manifests"]:
-                          collect(child)
-                  elif "image.manifest" in media_type:
-                      config_type = document["config"]["mediaType"]
-                      if "image.config" in config_type or "container.image" in config_type:
-                          manifests.append(document)
-
-              for root in index["manifests"]:
-                  collect(root)
-              if len(manifests) != 1:
-                  raise SystemExit(f"Expected one runnable image manifest, found {len(manifests)}")
-
-              manifest = manifests[0]
-              config_digest = manifest["config"]["digest"]
-              if metadata["containerimage.config.digest"] != config_digest:
-                  raise SystemExit("OCI config digest does not match the pushed build result")
-              if local_image_id != config_digest:
-                  raise SystemExit("Locally tested image does not match the OCI build result")
-              descriptors = [manifest["config"], *manifest["layers"]]
-              for descriptor in descriptors:
-                  read_blob(descriptor)
-              total = sum(descriptor["size"] for descriptor in descriptors)
-
-          print(
-              f"{pushed_digest} compressed_size={total // 1024 // 1024} MiB "
-              f"limit={limit // 1024 // 1024} MiB",
-          )
-          if total > limit:
-              raise SystemExit(f"{pushed_digest} is over the compressed size limit")
-          PY
-
-      - name: Export digest
-        if: inputs.push_digests
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mkdir -p /tmp/digests
-          digest="${{ steps.docker_build.outputs.digest }}"
-          touch "/tmp/digests/${digest#sha256:}"
-
-      - name: Upload digest
-        if: inputs.push_digests
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
-        with:
-          name: ${{ matrix.artifact_name }}
-          path: /tmp/digests/*
-          if-no-files-found: error
-          retention-days: 1
@@ -1,158 +0,0 @@
-name: Documentation tests
-
-on:
-  workflow_call:
-
-env:
-  PYTHONIOENCODING: utf-8
-  USE_COLOR: "False"
-  SHOW_PROGRESS: "False"
-
-jobs:
-  discover:
-    name: Discover documentation examples
-    runs-on: ubuntu-24.04
-    timeout-minutes: 5
-    outputs:
-      matrix: ${{ steps.inventory.outputs.matrix }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-      - name: Install ArchiveBox development dependencies
-        run: uv sync --no-cache --locked --dev
-      - name: Resolve documentation tools through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/archivebox-docs-lib
-        run: |
-          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
-          uv run --no-cache --no-sync abxpkg env \
-            --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries" \
-            --deps-from="$GITHUB_WORKSPACE/docs/mermaid-binary.json:required_binaries" \
-            --deps-from="$GITHUB_WORKSPACE/docs/nginx-binary.json:required_binaries"
-          bash_binary="$ABXPKG_LIB_DIR/env/bin/bash"
-          mmdc_binary="$ABXPKG_LIB_DIR/env/bin/mmdc"
-          nginx_binary="$ABXPKG_LIB_DIR/env/bin/nginx"
-          test -L "$bash_binary"
-          test -L "$mmdc_binary"
-          test -L "$nginx_binary"
-          test -x "$bash_binary"
-          test -x "$mmdc_binary"
-          test -x "$nginx_binary"
-          {
-            echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR"
-            echo "BASH_BINARY=$bash_binary"
-          } >> "$GITHUB_ENV"
-      - id: inventory
-        name: Validate the complete authored documentation inventory
-        run: |
-          matrix="$(uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py matrix)"
-          echo "matrix=$matrix" >> "$GITHUB_OUTPUT"
-
-  examples:
-    name: docs/${{ matrix.name }}
-    needs: discover
-    runs-on: ${{ matrix.runner }}
-    timeout-minutes: 15
-    strategy:
-      fail-fast: false
-      max-parallel: 4
-      matrix: ${{ fromJSON(needs.discover.outputs.matrix) }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
-        with:
-          python-version: "3.13"
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-      - name: Install ArchiveBox
-        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
-        run: uv sync --no-cache --locked --dev
-      - name: Resolve documentation runtime through abxpkg
-        if: ${{ !contains(fromJSON('["freebsd", "openbsd"]'), matrix.environment) }}
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/archivebox-docs-lib
-        run: |
-          export PATH="$ABXPKG_LIB_DIR/env/bin:$PATH"
-          dependency_sources=(
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
-          )
-          if [[ '${{ matrix.environment }}' != macos ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docs_binaries")
-          fi
-          if [[ '${{ matrix.name }}' == ubuntu-AGENTS || '${{ matrix.name }}' == ubuntu-skills-archivebox-SKILL || '${{ matrix.name }}' == ubuntu-docs-Configuration || '${{ matrix.name }}' == ubuntu-docs-Setting-up-Authentication ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
-          fi
-          if [[ '${{ matrix.environment }}' == docker ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries")
-          elif [[ '${{ matrix.environment }}' == macos ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-macos-brew.json:required_binaries")
-          fi
-          uv run --no-cache --no-sync abxpkg env \
-            --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            "${dependency_sources[@]}"
-          bash_binary="$ABXPKG_LIB_DIR/env/bin/bash"
-          python_binary="$ABXPKG_LIB_DIR/env/bin/python"
-          uv_binary="$ABXPKG_LIB_DIR/env/bin/uv"
-          test -L "$bash_binary"
-          test -x "$bash_binary"
-          test -L "$python_binary"
-          test -x "$python_binary"
-          test -L "$uv_binary"
-          test -x "$uv_binary"
-          {
-            echo "ABXPKG_LIB_DIR=$ABXPKG_LIB_DIR"
-            echo "BASH_BINARY=$bash_binary"
-            echo "PYTHON_BINARY=$python_binary"
-            echo "UV_BINARY=$uv_binary"
-          } >> "$GITHUB_ENV"
-          if [[ '${{ matrix.environment }}' != macos ]]; then
-            sudo_binary="$ABXPKG_LIB_DIR/env/bin/sudo"
-            test -L "$sudo_binary"
-            test -x "$sudo_binary"
-            echo "SUDO_BINARY=$sudo_binary" >> "$GITHUB_ENV"
-          fi
-          if [[ '${{ matrix.environment }}' != macos ]]; then
-            echo "$ABXPKG_LIB_DIR/env/bin" >> "$GITHUB_PATH"
-          fi
-          if [[ '${{ matrix.environment }}' == docker ]]; then
-            docker_binary="$ABXPKG_LIB_DIR/env/bin/docker"
-            test -L "$docker_binary"
-            test -x "$docker_binary"
-            echo "DOCKER_BINARY=$docker_binary" >> "$GITHUB_ENV"
-          elif [[ '${{ matrix.environment }}' == macos ]]; then
-            test -L "$ABXPKG_LIB_DIR/env/bin/brew"
-            test -x "$ABXPKG_LIB_DIR/env/bin/brew"
-          fi
-      - name: Build the exact checkout's documentation image
-        if: ${{ matrix.environment == 'docker' }}
-        run: '"$DOCKER_BINARY" build --tag archivebox/archivebox:dev .'
-      - name: Run the Ubuntu, Docker, or macOS documentation lane
-        if: ${{ !contains(fromJSON('["root", "freebsd", "openbsd"]'), matrix.environment) }}
-        run: uv run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
-      - name: Run the root documentation lane
-        if: ${{ matrix.environment == 'root' }}
-        run: |
-          "$SUDO_BINARY" \
-            --preserve-env=PATH,ABXPKG_LIB_DIR,BASH_BINARY,PYTHON_BINARY,SUDO_BINARY,UV_BINARY \
-            "$UV_BINARY" run --no-cache --no-sync python docs/test_codeblocks_manifest.py run-group '${{ matrix.snippet_ids }}'
@@ -1,38 +0,0 @@
-name: Run linters
-
-on:
-  workflow_call:
-
-env:
-  UV_NO_SOURCES: "1"
-
-jobs:
-  lint:
-    runs-on: ubuntu-22.04
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - name: Set up Python
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-          architecture: x64
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Verify checked dependency lock
-        run: uv lock --no-cache --check
-
-      - name: Install dependencies with uv
-        run: uv sync --no-cache --locked --dev --extra sonic --extra debug
-
-      - name: Run prek
-        run: uv run --no-cache --no-sync --no-sources prek run --all-files
@@ -1,243 +0,0 @@
-name: Build Pip package
-
-on:
-  workflow_call:
-    inputs:
-      full_tests:
-        description: Run the full package compatibility matrix instead of one release smoke check
-        required: false
-        default: true
-        type: boolean
-
-env:
-  UV_VERSION: "0.11.3"
-
-jobs:
-  build:
-    name: build distributions
-    runs-on: ubuntu-24.04
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-
-      - name: Set up Python 3.13
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-          architecture: x64
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: ${{ env.UV_VERSION }}
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Build every distribution
-        run: |
-          set -Eeuo pipefail
-          uv build --no-cache --all-packages --no-sources --out-dir dist --clear
-
-          shopt -s nullglob
-          artifacts=(dist/*)
-          wheels=(dist/archivebox-*.whl)
-          sdists=(dist/archivebox-*.tar.gz)
-          [[ "${#artifacts[@]}" -gt 0 ]]
-          [[ "${#wheels[@]}" -eq 1 ]]
-          [[ "${#sdists[@]}" -eq 1 ]]
-          printf '%s\n' "$GITHUB_SHA" > dist/COMMIT_SHA
-          uv run --no-cache --no-project python - <<'PY'
-          import hashlib
-          from pathlib import Path
-
-          root = Path("dist")
-          wheels = list(root.glob("archivebox-*.whl"))
-          sdists = list(root.glob("archivebox-*.tar.gz"))
-          if len(wheels) != 1 or len(sdists) != 1:
-              raise SystemExit("Expected exactly one ArchiveBox wheel and one sdist")
-          artifacts = [root / "COMMIT_SHA", wheels[0], sdists[0]]
-          checksum_lines = []
-          for artifact in artifacts:
-              digest = hashlib.sha256(artifact.read_bytes()).hexdigest()
-              line = f"{digest}  {artifact.name}"
-              checksum_lines.append(line)
-              print(line)
-          Path("dist/SHA256SUMS").write_text("\n".join(checksum_lines) + "\n")
-          PY
-
-      - name: Release wheel import and CLI smoke
-        if: ${{ !inputs.full_tests }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          shopt -s nullglob
-          wheels=(dist/archivebox-*.whl)
-          [[ "${#wheels[@]}" -eq 1 ]]
-          uv sync --locked --no-cache --no-sources --no-dev --no-install-project
-          uv pip install --no-cache --no-deps --python .venv/bin/python "${wheels[0]}"
-          unset PYTHONPATH
-          cd "$(mktemp -d)"
-          VIRTUAL_ENV="$GITHUB_WORKSPACE/.venv" uv run --no-cache --active --no-project python -c 'import archivebox'
-          VIRTUAL_ENV="$GITHUB_WORKSPACE/.venv" uv run --no-cache --active --no-project archivebox version
-
-      - name: Upload every distribution
-        uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4
-        with:
-          name: python-distributions
-          path: dist/*
-          if-no-files-found: error
-          retention-days: 7
-
-  install-smoke:
-    name: install ${{ matrix.artifact }} / Python ${{ matrix.python }} / ${{ matrix.os }}
-    if: inputs.full_tests
-    needs: build
-    runs-on: ${{ matrix.os }}
-    strategy:
-      fail-fast: false
-      matrix:
-        os: [ubuntu-24.04, macos-15]
-        python: ["3.13", "3.14"]
-        artifact: [wheel, sdist]
-
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: ${{ matrix.python }}
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: ${{ env.UV_VERSION }}
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Download distributions
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
-        with:
-          name: python-distributions
-          path: ${{ runner.temp }}/python-distributions
-
-      - name: Install and smoke-test the built ${{ matrix.artifact }}
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          shopt -s nullglob
-
-          case "${{ matrix.artifact }}" in
-            wheel) candidates=("$RUNNER_TEMP"/python-distributions/archivebox-*.whl) ;;
-            sdist) candidates=("$RUNNER_TEMP"/python-distributions/archivebox-*.tar.gz) ;;
-            *) exit 2 ;;
-          esac
-          [[ "${#candidates[@]}" -eq 1 ]]
-          artifact="${candidates[0]}"
-
-          smoke_root="$(mktemp -d "$RUNNER_TEMP/archivebox-package-smoke.XXXXXX")"
-          trap 'rm -rf "$smoke_root"' EXIT
-          smoke_env="$smoke_root/env"
-          data_dir="$smoke_root/data"
-          mkdir -p "$data_dir"
-
-          dependency_artifacts=()
-          while IFS= read -r dependency_artifact; do
-            dependency_artifacts+=("$dependency_artifact")
-          done < <(uv run --no-cache --no-project python - <<'PY'
-          import tomllib
-
-          locked = {
-              package["name"]: package
-              for package in tomllib.load(open("uv.lock", "rb"))["package"]
-          }
-          for package_name in ("abx-dl", "abx-plugins", "abxpkg", "abxbus"):
-              package = locked[package_name]
-              wheels = package.get("wheels") or []
-              if len(wheels) != 1:
-                  raise SystemExit(f"Expected exactly one locked wheel for {package_name}")
-              print(wheels[0]["url"])
-          PY
-          )
-
-          uv venv --python "${{ matrix.python }}" "$smoke_env"
-          uv pip install --no-cache \
-            --python "$smoke_env" \
-            "${dependency_artifacts[@]}" \
-            "$artifact"
-
-          unset PYTHONPATH
-          cd "$data_dir"
-          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync python - <<'PY'
-          import os
-          from pathlib import Path
-
-          import archivebox
-
-          installed_path = Path(archivebox.__file__).resolve()
-          environment_path = Path(os.environ["VIRTUAL_ENV"]).resolve()
-          workspace_path = Path(os.environ["GITHUB_WORKSPACE"]).resolve()
-          assert installed_path.is_relative_to(environment_path), (installed_path, environment_path)
-          assert not installed_path.is_relative_to(workspace_path), (installed_path, workspace_path)
-          print(installed_path)
-          PY
-          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox version
-          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox init
-          VIRTUAL_ENV="$smoke_env" uv run --no-cache --active --no-project --no-sync archivebox status
-
-          if [[ "${{ runner.os }}" == "Linux" ]]; then
-            root_tool_dir="/root/archivebox-package-smoke-${{ matrix.artifact }}-${{ matrix.python }}"
-            uv_binary="$(command -v uv)"
-            sudo rm -rf "$root_tool_dir"
-            sudo mkdir -p "$root_tool_dir/data"
-
-            root_install_args=()
-            for dependency_artifact in "${dependency_artifacts[@]}"; do
-              root_install_args+=(--with "$dependency_artifact")
-            done
-            sudo UV_TOOL_DIR="$root_tool_dir/tools" UV_TOOL_BIN_DIR="$root_tool_dir/bin" \
-              "$uv_binary" tool install --no-cache --python "${{ matrix.python }}" \
-              "${root_install_args[@]}" "$artifact"
-
-            sudo useradd --system --create-home --home-dir /var/lib/archivebox --shell /bin/bash archivebox 2>/dev/null || true
-            sudo rm -rf /var/lib/archivebox
-            sudo env PATH="$root_tool_dir/bin:$PATH" \
-              bash -c "cd '$root_tool_dir/data' && archivebox init"
-            sudo stat -c %U /var/lib/archivebox | grep -qx archivebox
-            sudo stat -c %U "$root_tool_dir/data" | grep -qx archivebox
-            if sudo find "$root_tool_dir/data" -xdev -user root -print -quit | grep -q .; then
-              echo "Root-owned files remain in $root_tool_dir/data" >&2
-              exit 1
-            fi
-
-            sudo mkdir "$root_tool_dir/external"
-            sudo ln -s "$root_tool_dir/external" "$root_tool_dir/data/external-link"
-            sudo env PATH="$root_tool_dir/bin:$PATH" \
-              bash -c "cd '$root_tool_dir/data' && archivebox status"
-            sudo stat -c %U "$root_tool_dir/external" | grep -qx root
-
-            sudo mkdir "$root_tool_dir/data-universal-init"
-            sudo env PATH="$root_tool_dir/bin:$PATH" \
-              bash -c "cd '$root_tool_dir/data-universal-init' && archivebox status --init"
-            sudo stat -c %U "$root_tool_dir/data-universal-init" | grep -qx archivebox
-
-            sudo mkdir "$root_tool_dir/data-install"
-            sudo env PATH="$root_tool_dir/bin:$PATH" \
-              bash -c "cd '$root_tool_dir/data-install' && archivebox install --dry-run"
-            sudo stat -c %U "$root_tool_dir/data-install" | grep -qx archivebox
-
-            sudo useradd --create-home archivebox-smoke-user
-            smoke_user_group="$(id -gn archivebox-smoke-user)"
-            sudo mkdir "$root_tool_dir/data-mixed-owner"
-            sudo chown archivebox-smoke-user:root "$root_tool_dir/data-mixed-owner"
-            sudo env PATH="$root_tool_dir/bin:$PATH" \
-              bash -c "cd '$root_tool_dir/data-mixed-owner' && archivebox init"
-            sudo stat -c %U:%G "$root_tool_dir/data-mixed-owner" | grep -qx "archivebox-smoke-user:$smoke_user_group"
-            if sudo find "$root_tool_dir/data-mixed-owner" -xdev -user root -print -quit | grep -q .; then
-              echo "Root-owned files remain in $root_tool_dir/data-mixed-owner" >&2
-              exit 1
-            fi
-            sudo rm -rf "$root_tool_dir"
-          fi
@@ -1,51 +0,0 @@
-name: Release candidate
-
-on:
-  push:
-    branches: [dev]
-
-permissions:
-  actions: read
-  contents: write
-  packages: write
-
-concurrency:
-  group: release-candidate-dev
-  cancel-in-progress: true
-
-jobs:
-  prepare:
-    runs-on: ubuntu-24.04
-    outputs:
-      action: ${{ steps.version.outputs.action }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          token: ${{ secrets.RELEASE_GH_TOKEN }}
-
-      - id: version
-        uses: ArchiveBox/monorepo/.github/actions/prepare-release-version@main
-        with:
-          package: archivebox
-          branch: dev
-          tag-prefix: v
-          version-scheme: rc
-
-  python-artifacts:
-    needs: prepare
-    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
-    uses: ./.github/workflows/pip.yml
-    with:
-      full_tests: false
-    secrets: inherit
-
-  docker-digests:
-    needs: prepare
-    if: needs.prepare.outputs.action == 'ready' || needs.prepare.outputs.action == 'reserved'
-    uses: ./.github/workflows/docker.yml
-    with:
-      push_digests: true
-      full_tests: false
-    secrets: inherit
@@ -1,331 +0,0 @@
-name: Release
-
-on:
-  workflow_run:
-    workflows: [CI, Release candidate]
-    types: [completed]
-    branches: [dev, main]
-
-permissions: {}
-
-concurrency:
-  group: release-${{ github.repository }}-${{ github.event.workflow_run.head_branch }}
-  cancel-in-progress: false
-
-env:
-  DOCKERHUB_IMAGE: archivebox/archivebox
-  GHCR_IMAGE: ghcr.io/archivebox/archivebox
-  RELEASE_BRANCH: ${{ github.event.workflow_run.head_branch }}
-  RELEASE_SHA: ${{ github.event.workflow_run.head_sha }}
-  CI_RUN_ID: ${{ github.event.workflow_run.id }}
-
-jobs:
-  candidate:
-    if: >-
-      github.event.workflow_run.conclusion == 'success' &&
-      github.event.workflow_run.event == 'push' &&
-      ((github.event.workflow_run.head_branch == 'dev' && github.event.workflow_run.name == 'Release candidate') ||
-       (github.event.workflow_run.head_branch == 'main' && github.event.workflow_run.name == 'CI'))
-    runs-on: ubuntu-24.04
-    outputs:
-      ready: ${{ steps.owner.outputs.ready }}
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          ref: ${{ env.RELEASE_SHA }}
-      - id: owner
-        shell: bash
-        run: |
-          if [[ "$RELEASE_BRANCH" == main ]]; then
-            echo 'ready=true' >> "$GITHUB_OUTPUT"
-            exit 0
-          fi
-          VERSION="$(sed -nE 's/^version = "([^"]+)".*/\1/p' pyproject.toml | head -n 1)"
-          TARGET="$(git ls-remote origin "refs/tags/release-candidate/$VERSION" | awk '{print $1}')"
-          [[ "$TARGET" == "$RELEASE_SHA" ]] && echo 'ready=true' >> "$GITHUB_OUTPUT" || echo 'ready=false' >> "$GITHUB_OUTPUT"
-
-  python-release:
-    name: Publish tested Python artifacts
-    needs: candidate
-    if: needs.candidate.outputs.ready == 'true'
-    runs-on: ubuntu-24.04
-    environment: pypi
-    permissions:
-      actions: read
-      contents: write
-      id-token: write
-    outputs:
-      version: ${{ steps.version.outputs.version }}
-      tag: ${{ steps.version.outputs.tag }}
-      release_ready: ${{ steps.release-state.outputs.ready }}
-    steps:
-      - name: Checkout the tested commit
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 0
-          submodules: true
-          ref: ${{ env.RELEASE_SHA }}
-          token: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-
-      - name: Resolve release binaries through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" >/dev/null
-          for name in uv gh git jq curl; do
-            test -L "$ABXPKG_LIB_DIR/env/bin/$name"
-            test -x "$ABXPKG_LIB_DIR/env/bin/$name"
-          done
-          {
-            echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
-            echo "GH_BINARY=$ABXPKG_LIB_DIR/env/bin/gh"
-            echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
-            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
-            echo "CURL_BINARY=$ABXPKG_LIB_DIR/env/bin/curl"
-          } >> "$GITHUB_ENV"
-
-      - name: Download exact tested distributions
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
-        with:
-          name: python-distributions
-          path: ${{ runner.temp }}/python-distributions
-          github-token: ${{ github.token }}
-          repository: ${{ github.repository }}
-          run-id: ${{ env.CI_RUN_ID }}
-
-      - name: Record release version
-        id: version
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION="$($UV_BINARY run --no-cache --no-project python -c 'import tomllib; print(tomllib.load(open("pyproject.toml", "rb"))["project"]["version"])')"
-          echo "version=$VERSION" >> "$GITHUB_OUTPUT"
-          echo "tag=v$VERSION" >> "$GITHUB_OUTPUT"
-
-      - name: Validate release branch and version
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION='${{ steps.version.outputs.version }}'
-          case "$RELEASE_BRANCH" in
-            main)
-              ;;
-            dev)
-              [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+rc[0-9]+$ ]] || {
-                echo "Refusing to publish ArchiveBox $VERSION from dev; only archivebox:main may publish non-prerelease versions." >&2
-                exit 1
-              }
-              ;;
-            *)
-              echo "Refusing to publish ArchiveBox $VERSION from unsupported branch $RELEASE_BRANCH; only archivebox:dev and archivebox:main may publish." >&2
-              exit 1
-              ;;
-          esac
-
-      - name: Publish the exact tested distributions
-        env:
-          GH_TOKEN: ${{ secrets.RELEASE_GH_TOKEN || github.token }}
-          RELEASE_DISTRIBUTIONS_DIR: ${{ runner.temp }}/python-distributions
-        run: ./bin/release.sh
-
-      - name: Confirm this tested commit owns the release tag
-        id: release-state
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          TARGET=""
-          while read -r sha _; do
-            TARGET="$sha"
-          done < <("$GIT_BINARY" ls-remote origin "refs/tags/${{ steps.version.outputs.tag }}")
-          if [[ "$TARGET" == "$RELEASE_SHA" ]]; then
-            echo "ready=true" >> "$GITHUB_OUTPUT"
-          else
-            echo "ready=false" >> "$GITHUB_OUTPUT"
-          fi
-
-  docker-release:
-    name: Publish tested multiarch image
-    if: >-
-      github.event.workflow_run.conclusion == 'success' &&
-      github.event.workflow_run.event == 'push'
-    needs: python-release
-    runs-on: ubuntu-24.04
-    permissions:
-      actions: read
-      contents: read
-      packages: write
-    steps:
-      - name: Checkout the tested commit
-        uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          fetch-depth: 1
-          ref: ${{ env.RELEASE_SHA }}
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: "3.13"
-
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-
-      - name: Resolve release and Docker binaries through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:release_binaries" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:docker_binaries" >/dev/null
-          for name in uv git jq docker; do
-            test -L "$ABXPKG_LIB_DIR/env/bin/$name"
-            test -x "$ABXPKG_LIB_DIR/env/bin/$name"
-          done
-          {
-            echo "UV_BINARY=$ABXPKG_LIB_DIR/env/bin/uv"
-            echo "GIT_BINARY=$ABXPKG_LIB_DIR/env/bin/git"
-            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
-            echo "DOCKER_BINARY=$ABXPKG_LIB_DIR/env/bin/docker"
-          } >> "$GITHUB_ENV"
-
-      - uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
-        with:
-          install: true
-
-      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-
-      - uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
-        with:
-          registry: ghcr.io
-          username: ${{ github.actor }}
-          password: ${{ github.token }}
-
-      - name: Download exact tested image digests
-        uses: actions/download-artifact@d3f86a106a0bac45b974a628896c90dbdf5c8093 # v4
-        with:
-          path: /tmp/digests
-          pattern: digest-*
-          merge-multiple: true
-          github-token: ${{ github.token }}
-          repository: ${{ github.repository }}
-          run-id: ${{ env.CI_RUN_ID }}
-
-      - name: Verify digest set and collect tags
-        id: docker_meta
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
-          [[ "${#DIGESTS[@]}" -eq 2 ]]
-          for digest in "${DIGESTS[@]}"; do [[ "$digest" =~ ^[0-9a-f]{64}$ ]]; done
-
-          VERSION='${{ needs.python-release.outputs.version }}'
-          SHORT_SHA="${RELEASE_SHA::12}"
-          TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}^{}")"
-          TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
-          if [[ -z "$TAG_TARGET" ]]; then
-            TAG_TARGET="$($GIT_BINARY ls-remote origin "refs/tags/v${VERSION}")"
-            TAG_TARGET="${TAG_TARGET%%[[:space:]]*}"
-          fi
-          {
-            echo 'dockerhub_tags<<EOF'
-            if [[ "$RELEASE_BRANCH" == dev ]]; then
-              echo "${DOCKERHUB_IMAGE}:dev"
-            elif [[ "$RELEASE_BRANCH" == main ]]; then
-              echo "${DOCKERHUB_IMAGE}:latest"
-            fi
-            echo "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}"
-            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${DOCKERHUB_IMAGE}:${VERSION}"
-            echo 'EOF'
-            echo 'ghcr_tags<<EOF'
-            if [[ "$RELEASE_BRANCH" == dev ]]; then
-              echo "${GHCR_IMAGE}:dev"
-            elif [[ "$RELEASE_BRANCH" == main ]]; then
-              echo "${GHCR_IMAGE}:latest"
-            fi
-            echo "${GHCR_IMAGE}:sha-${SHORT_SHA}"
-            [[ "$TAG_TARGET" == "$RELEASE_SHA" ]] && echo "${GHCR_IMAGE}:${VERSION}"
-            echo 'EOF'
-          } >> "$GITHUB_OUTPUT"
-
-      - name: Create Docker Hub manifest
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
-          TAG_ARGS=()
-          while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.dockerhub_tags }}'
-          REFS=()
-          for digest in "${DIGESTS[@]}"; do REFS+=("${DOCKERHUB_IMAGE}@sha256:${digest}"); done
-          $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
-
-      - name: Create GHCR manifest
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          mapfile -t DIGESTS < <($UV_BINARY run --no-cache --no-project python -c 'from pathlib import Path; print("\n".join(sorted(path.name for path in Path("/tmp/digests").iterdir() if path.is_file())))')
-          TAG_ARGS=()
-          while IFS= read -r tag; do [[ -n "$tag" ]] && TAG_ARGS+=(--tag "$tag"); done <<< '${{ steps.docker_meta.outputs.ghcr_tags }}'
-          REFS=()
-          for digest in "${DIGESTS[@]}"; do REFS+=("${GHCR_IMAGE}@sha256:${digest}"); done
-          $DOCKER_BINARY buildx imagetools create "${TAG_ARGS[@]}" "${REFS[@]}"
-
-      - name: Verify published Docker images run
-        shell: bash
-        run: |
-          set -Eeuo pipefail
-          VERSION='${{ needs.python-release.outputs.version }}'
-          SHORT_SHA="${RELEASE_SHA::12}"
-          for image in "${DOCKERHUB_IMAGE}:sha-${SHORT_SHA}" "${GHCR_IMAGE}:sha-${SHORT_SHA}"; do
-            "$DOCKER_BINARY" buildx imagetools inspect "$image" >/dev/null
-            INSTALLED_VERSION="$("$DOCKER_BINARY" run --rm "$image" --version)"
-            test "$INSTALLED_VERSION" = "$VERSION"
-          done
-
-      - name: Update Docker Hub README
-        uses: peter-evans/dockerhub-description@432a30c9e07499fd01da9f8a49f0faf9e0ca5b77 # v4
-        with:
-          username: ${{ secrets.DOCKER_USERNAME }}
-          password: ${{ secrets.DOCKER_PASSWORD }}
-          repository: archivebox/archivebox
-
-  cascade:
-    name: Cascade verified release
-    if: needs.python-release.outputs.release_ready == 'true'
-    needs:
-      - python-release
-      - docker-release
-    permissions:
-      actions: read
-      contents: read
-    uses: ArchiveBox/monorepo/.github/workflows/cascade-release.yml@main
-    with:
-      upstream_package: archivebox
-      upstream_version: ${{ needs.python-release.outputs.version }}
-      upstream_artifact_name: python-distributions
-    secrets:
-      release_token: ${{ secrets.RELEASE_GH_TOKEN }}
@@ -1,154 +0,0 @@
-name: Parallel Tests
-
-on:
-  workflow_call:
-
-env:
-  PYTHONIOENCODING: utf-8
-  PYTHONLEGACYWINDOWSSTDIO: utf-8
-  USE_COLOR: False
-  UV_NO_SOURCES: "1"
-  CI_PYTHON_VERSION: "3.13"
-
-jobs:
-  discover-tests:
-    name: Discover every test
-    runs-on: ubuntu-24.04
-    outputs:
-      tests: ${{ steps.discover.outputs.tests }}
-
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: ${{ env.CI_PYTHON_VERSION }}
-
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Resolve discovery tools through abxpkg
-        env:
-          ABXPKG_LIB_DIR: ${{ runner.temp }}/abx-lib
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
-          test -n "$ABXPKG_SPEC"
-          mkdir -p "$ABXPKG_LIB_DIR/env/bin"
-          export PYTHON_BINARY="$pythonLocation/bin/python"
-          test -x "$PYTHON_BINARY"
-          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
-            --install \
-            --lib="$ABXPKG_LIB_DIR" \
-            --deps-from="$GITHUB_WORKSPACE/.github/configs/ci-tooling.json:ci_binaries"
-          {
-            echo "PYTHON_BINARY=$ABXPKG_LIB_DIR/env/bin/python"
-            echo "JQ_BINARY=$ABXPKG_LIB_DIR/env/bin/jq"
-          } >> "$GITHUB_ENV"
-          test -L "$ABXPKG_LIB_DIR/env/bin/python"
-          test -x "$ABXPKG_LIB_DIR/env/bin/python"
-          test -L "$ABXPKG_LIB_DIR/env/bin/jq"
-          test -x "$ABXPKG_LIB_DIR/env/bin/jq"
-
-      - name: Discover every test exactly once
-        id: discover
-        run: |
-          set -Eeuo pipefail
-          mapfile -t output < <("$PYTHON_BINARY" .github/scripts/discover_test_matrix.py)
-          test "${#output[@]}" -eq 2
-          echo "tests=${output[1]}" >> "$GITHUB_OUTPUT"
-          echo "${output[0]}"
-          "$JQ_BINARY" -e 'length > 0 and length <= 256 and ([.[].paths[]] | length == (unique | length))' <<< "${output[1]}"
-
-  tests:
-    name: ${{ matrix.test.name }}
-    needs: discover-tests
-    runs-on: ubuntu-24.04
-    env:
-      ABXPKG_LIB_DIR: /tmp/archivebox-lib
-      CHROME_HEADLESS: "true"
-      DATA_DIR: /tmp/archivebox-test-data
-      PERSONAS_DIR: /tmp/abx-personas
-      CHROME_USER_DATA_DIR: /tmp/abx-personas/Default/chrome_profile
-
-    strategy:
-      fail-fast: false
-      matrix:
-        test: ${{ fromJson(needs.discover-tests.outputs.tests) }}
-
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      - uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: ${{ env.CI_PYTHON_VERSION }}
-          architecture: x64
-
-      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Resolve optional build dependencies through abxpkg
-        if: matrix.test.extra != '' || contains(matrix.test.paths_arg, 'test_postgres_backend')
-        env:
-          TEST_EXTRA: ${{ matrix.test.extra }}
-          TEST_PATHS: ${{ matrix.test.paths_arg }}
-        run: |
-          set -Eeuo pipefail
-          ABXPKG_SPEC="$(uv run --no-cache --no-project python -c 'import tomllib; package = next(item for item in tomllib.load(open("uv.lock", "rb"))["package"] if item["name"] == "abxpkg"); wheel = package["wheels"][0]; print("abxpkg @ {}#{}".format(wheel["url"], wheel["hash"].replace(":", "=")))')"
-          test -n "$ABXPKG_SPEC"
-          dependency_sources=()
-          if [[ -n "$TEST_EXTRA" ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:required_binaries")
-          fi
-          if [[ "$TEST_PATHS" == *test_postgres_backend* ]]; then
-            dependency_sources+=(--deps-from="$GITHUB_WORKSPACE/.github/configs/ci-linux-build.json:postgres_binaries")
-          fi
-          uv run --no-cache --no-project --with "$ABXPKG_SPEC" abxpkg env \
-            --install \
-            --no-cache \
-            --json \
-            --lib="$ABXPKG_LIB_DIR" \
-            "${dependency_sources[@]}"
-
-      - name: Install test dependencies
-        env:
-          TEST_EXTRA: ${{ matrix.test.extra }}
-        run: |
-          set -Eeuo pipefail
-          extra_args=()
-          if [[ -n "$TEST_EXTRA" ]]; then
-            extra_args=(--extra "$TEST_EXTRA")
-          fi
-          uv sync --no-cache --locked --dev "${extra_args[@]}"
-
-      - name: Run ${{ matrix.test.name }}
-        env:
-          TEST_PATHS: ${{ matrix.test.paths_arg }}
-        run: |
-          set -Eeuo pipefail
-          read -r -a test_paths <<< "$TEST_PATHS"
-          for test_path in "${test_paths[@]}"; do
-            test -f "$test_path"
-          done
-          for test_path in "${test_paths[@]}"; do
-            test_name="$(basename "$test_path" .py)"
-            test_slug="${test_name//[^A-Za-z0-9_.-]/_}"
-            export DATA_DIR="$RUNNER_TEMP/archivebox-test-data/$test_slug"
-            export PERSONAS_DIR="$RUNNER_TEMP/abx-personas/$test_slug"
-            export CHROME_USER_DATA_DIR="$PERSONAS_DIR/Default/chrome_profile"
-            mkdir -p "$DATA_DIR" "$PERSONAS_DIR" "$RUNNER_TEMP/pytest/$test_slug"
-            uv run --no-cache --no-sync --no-sources pytest -vs "$test_path" \
-              --basetemp="$RUNNER_TEMP/pytest/$test_slug"
-          done
@@ -1,63 +0,0 @@
-name: Install and CLI Compatibility
-
-on:
-  workflow_call:
-
-env:
-  DOCKER_IMAGE: archivebox-ci
-  ARCHIVEBOX_IMAGE: archivebox-ci
-  PYTHONIOENCODING: utf-8
-  PYTHONLEGACYWINDOWSSTDIO: utf-8
-  USE_COLOR: False
-  UV_NO_SOURCES: "1"
-
-jobs:
-  install_cli_platform:
-    name: CLI install / ${{ matrix.os_name }} / Python ${{ matrix.python }}
-    runs-on: ${{ matrix.os }}
-
-    strategy:
-      fail-fast: false
-      matrix:
-        include:
-          - os: ubuntu-24.04
-            os_name: linux
-            python: "3.14"
-          - os: macos-15
-            os_name: macOS
-            python: "3.13"
-          - os: macos-15
-            os_name: macOS
-            python: "3.14"
-
-    steps:
-      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
-        with:
-          submodules: true
-          fetch-depth: 1
-
-      ### Set up Python
-      - name: Set up Python ${{ matrix.python }}
-        uses: actions/setup-python@a26af69be951a213d495a4c3e4e4022e16d87065 # v5
-        with:
-          python-version: ${{ matrix.python }}
-
-      - name: Install uv
-        uses: astral-sh/setup-uv@37802adc94f370d6bfd71619e3f0bf239e1f3b78 # v7
-        with:
-          version: "0.11.3"
-          enable-cache: false
-          cache-dependency-glob: uv.lock
-
-      - name: Install dependencies with uv
-        run: uv sync --no-cache --locked --dev
-
-      - name: ArchiveBox full install check
-        run: |
-          set -Eeuo pipefail
-          DATA_DIR="$PWD/tests/out/data"
-          mkdir -p "$DATA_DIR"
-          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox init
-          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox install
-          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox version
-          uv run --no-cache --directory "$DATA_DIR" --no-sync --no-sources archivebox status
@@ -1,70 +1,19 @@
-.DS_Store
-._*
-/.heartbeat.json
-/package.json
-/package-lock.json
-*.pyc
+# Pocket archive output folder
+pocket/
+bookmarks/
+pinboard/
+html/
+downloads/
+
+# Byte-compiled / optimized / DLL files
 __pycache__/
-.mypy_cache/
-.eggs/
-tests/out/
-/docs/screenshots/
-/publicsite/screenshots/

-# Coverage
-.coverage
-.coverage.*
-coverage.json
-coverage/
-htmlcov/
-
-# Python and Node dependencies
+# Virtualenv
+env/
+ENV/
 venv/
+VENV/
 .venv/
-.docker-venv/
-node_modules/
-typings/
-
-# Ignore dev lockfiles (should always be built fresh)
-pdm.dev.lock
-requirements-dev.txt
-
-# Packaging artifacts
-requirements.txt
-.pdm-python
-.pdm-build
-archivebox.egg-info
-archivebox-*.tar.gz
-build/
-dist/
-
-# Data folders
-lib/
-out/
-tmp/
-data/
-data*/
-archivebox/tests/data/
-archive/
-output/
-logs/
-/-
-/personas/
-/sources/
-index.sqlite3
-queue.sqlite3
-*.sqlite*
-data.*
-.archivebox_id
-ArchiveBox.conf
-*.stdout
-*.stderr
-*.log
-.tmp/
-
-# vim
-*.sw?
-.vscode
-
-# Local Claude Code task locks (not project state)
-.claude/scheduled_tasks.lock
+.env/
+.python-version
+.env
@@ -1,67 +0,0 @@
-default_language_version:
-  python: python3.13
-
-repos:
-  - repo: https://github.com/asottile/yesqa
-    rev: v1.5.0
-    hooks:
-      - id: yesqa
-
-  - repo: https://github.com/codespell-project/codespell
-    rev: v2.4.1
-    hooks:
-      - id: codespell
-        additional_dependencies:
-          - tomli
-
-  - repo: https://github.com/asottile/pyupgrade
-    rev: v3.20.0
-    hooks:
-      - id: pyupgrade
-        args: [--py313-plus]
-
-  - repo: https://github.com/asottile/add-trailing-comma
-    rev: v3.1.0
-    hooks:
-      - id: add-trailing-comma
-
-  - repo: local
-    hooks:
-      - id: ruff-format
-        name: ruff-format
-        entry: uv run --active --no-sync --no-sources ruff format
-        language: system
-        types_or: [python, pyi]
-      - id: ruff-check
-        name: ruff-check
-        entry: uv run --active --no-sync --no-sources ruff check --fix
-        language: system
-        types_or: [python, pyi]
-
-  - repo: https://github.com/pre-commit/pre-commit-hooks
-    rev: v5.0.0
-    hooks:
-      - id: check-ast
-      - id: check-toml
-      - id: check-yaml
-        exclude: ^\.github/workflows/homebrew\.yml$
-      - id: check-json
-      - id: check-merge-conflict
-      - id: check-symlinks
-      - id: destroyed-symlinks
-      - id: check-case-conflict
-      - id: check-illegal-windows-names
-      - id: check-shebang-scripts-are-executable
-        exclude: ^(archivebox/.*\.py|archivebox/tests/.*\.py|archivebox/personas/export_browser_state\.js)$
-      - id: mixed-line-ending
-      - id: fix-byte-order-marker
-      - id: end-of-file-fixer
-      - id: detect-private-key
-      - id: debug-statements
-      - id: forbid-submodules
-        exclude: ^docs$
-      - id: check-added-large-files
-        args: ["--maxkb=600"]
-      - id: name-tests-test
-        args: ["--pytest-test-first"]
-        exclude: ^archivebox/tests/(data/|fixtures\.py$|migrations_helpers\.py$|import_helpers\.py$)
@@ -1,87 +0,0 @@
-# ArchiveBox Agent Guide
-
-ArchiveBox is the full self-hosted web archiving app. Keep this repo on the `dev` branch.
-
-## Shared Standards
-
- Use `uv` and `uv run` for Python commands. Do not use system `python`, direct `.venv/bin/python`, or `pip` commands.
- Prefer existing repo patterns, helper APIs, fixtures, scripts, and command surfaces.
- Keep edits focused and minimal. Do not add wrappers, shims, aliases, or extra abstraction layers unless the current code path requires them.
- Do not weaken assertions, skip tests, xfail tests, or accept flaky behavior.
- No mocks, monkeypatches, fakes, simulated handlers, fake binaries, fake hooks, fake buses, or direct shortcuts around user-facing flows.
- Tests and verification should use real CLI commands, REST/API calls, browser UI flows, real hooks, real installs, real subprocesses, real DB rows, real files, and existing fixtures.
- Assertions must verify real correctness: exit codes, returned values, DB state, filesystem contents, field values, rendered output, and side effects.
- Start behavior fixes with a red failing test when a test is requested or practical.
- Trace root causes from observed behavior. Do not paper over failures with retries, wider timeouts, broad fallbacks, or looser assertions.
- Read `README.md` for the full setup, CLI, Docker, API, and release surface.
-
-## Concurrency Contract
-
- A collection has one orchestrator at a time. Local PID/process checks may warn about obvious same-machine duplicates, but must not claim to enforce ownership across machines or shared filesystems.
- SQLite remains supported with concurrent short writes from CLI and server processes. Keep the SQLite database on a local filesystem, never NFS/SMB, and use short autocommit/CAS updates instead of long transactions or database locks.
- Network calls, hook execution, filesystem migrations, and other long work belong in the orchestrator. Never hold a database transaction or lock across that work.
- PostgreSQL and shared data directories are the path to future multi-machine scheduling. Coordinate that work through database-backed per-crawl/per-snapshot claims; do not introduce file leases or timer-based orchestrator election.
-
-## Development Setup
-
-```bash
-uv sync --dev --all-extras
-mkdir -p data
-cd data
-uv run --project .. archivebox init --install
-```
-
-Run collection commands from inside an initialized data directory:
-
-```bash
-cd data
-uv run --project .. archivebox status
-uv run --project .. archivebox add --plugins=parse_txt_urls 'https://example.com/'
-uv run --project .. archivebox run
-```
-
-## User-Facing Setup
-
-Recommended CLI install:
-
-```bash
-uv tool install --python 3.13 --prerelease explicit --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
-mkdir -p ~/archivebox/data
-cd ~/archivebox/data
-archivebox init --install
-archivebox add --plugins=parse_txt_urls 'https://example.com/'
-```
-
-Alternative install methods:
-
- Docker Compose / Docker
- Homebrew
- Debian package
- pip
-
-## Basic Usage
-
-```bash
-cd ~/archivebox/data
-archivebox version
-archivebox help
-archivebox status
-archivebox install
-archivebox add --plugins=parse_txt_urls 'https://example.com/docs-basic-usage'
-archivebox list --json --with-headers
-archivebox search 'example'
-archivebox update --filter-type=domain example.com
-archivebox remove --yes --delete --filter-type=exact 'https://example.com/docs-basic-usage'
-archivebox run
-```
-
-## Verification
-
-Use targeted tests for focused work:
-
-```bash
-uv run pytest archivebox/tests/test_cli_add.py::test_add_help_shows_depth_and_tag_options -q
-uv run prek run --all-files
-```
-
-Releases are published only by `.github/workflows/release.yml` after the complete `dev` CI workflow succeeds. Local development and deployment commands must not publish packages, images, tags, or GitHub releases.
@@ -0,0 +1,46 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at git@nicksweeting.com. The project team will review and investigate all complaints, and will respond in a way that it deems appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][homepage], version 1.4, available at [http://contributor-covenant.org/version/1/4][version]
+
+[homepage]: http://contributor-covenant.org
+[version]: http://contributor-covenant.org/version/1/4/
@@ -1,286 +0,0 @@
-# syntax=docker/dockerfile:1.7
-
-# Multistage ArchiveBox Dockerfile that consumes the abx-dl runtime image.
-# abx-dl owns Python, Node, Chromium, and downloader plugin runtimes.
-# ArchiveBox owns sonic, supervisor, Django, and the app runtime.
-# Build abx-dl first, then point this file at it:
-#   docker buildx build ../abx-dl -f ../abx-dl/Dockerfile \
-#       --build-context abxbus=../abxbus \
-#       --build-context abxpkg=../abxpkg \
-#       --build-context abx-plugins=../abx-plugins \
-#       -t archivebox/abx-dl:dev
-#   docker buildx build . -f Dockerfile \
-#       --build-arg ABX_DL_IMAGE=archivebox/abx-dl:1.12.274 \
-#       -t archivebox:multistage
-
-ARG ABX_DL_IMAGE=archivebox/abx-dl:1.12.274
-
-FROM archivebox/sonic:1.4.9 AS sonic
-FROM ${ABX_DL_IMAGE} AS archivebox-runtime-base
-
-ARG TARGETPLATFORM
-ARG TARGETOS
-ARG TARGETARCH
-ARG TARGETVARIANT
-ARG ARCHIVEBOX_COMMIT_HASH=""
-
-ENV TZ=UTC \
-    LANGUAGE=en_US:en \
-    LC_ALL=C.UTF-8 \
-    LANG=C.UTF-8 \
-    DEBIAN_FRONTEND=noninteractive \
-    APT_KEY_DONT_WARN_ON_DANGEROUS_USAGE=1 \
-    PYTHONIOENCODING=UTF-8 \
-    PYTHONUNBUFFERED=1 \
-    PYTHONDONTWRITEBYTECODE=1 \
-    PIP_DISABLE_PIP_VERSION_CHECK=1 \
-    PIP_NO_COMPILE=1 \
-    PIP_ONLY_BINARY=aiohttp \
-    npm_config_loglevel=error
-
-ENV PYTHON_VERSION=3.13 \
-    NODE_VERSION=24
-
-ENV ARCHIVEBOX_USER=archivebox \
-    DEFAULT_ARCHIVEBOX_UID=911 \
-    DEFAULT_ARCHIVEBOX_GID=911 \
-    IN_DOCKER=True
-
-ENV CODE_DIR=/app \
-    DATA_DIR=/data \
-    CONFIG_DIR=/opt/archivebox \
-    ABXPKG_LIB_DIR=/opt/archivebox/lib \
-    PLAYWRIGHT_BROWSERS_PATH=/opt/archivebox/lib/playwright/cache \
-    PERSONAS_DIR=/data/personas \
-    CHROME_HEADLESS=true \
-    CHROME_SANDBOX=false \
-    CHROME_ISOLATION=crawl \
-    CHROME_ARGS_EXTRA='["--disable-gpu","--disable-features=Translate,OptimizationGuideModelDownloading,MediaRouter"]'
-
-ENV TMP_DIR=/tmp/archivebox \
-    PIP_VENV_PYTHON=/venv/bin/python3 \
-    GOOGLE_API_KEY=no \
-    GOOGLE_DEFAULT_CLIENT_ID=no \
-    GOOGLE_DEFAULT_CLIENT_SECRET=no
-
-ENV HOME=/home/archivebox \
-    XDG_CONFIG_HOME=/home/archivebox/.config \
-    XDG_CACHE_HOME=/opt/archivebox/lib/cache \
-    TIMEOUT=600
-
-ENV UV_COMPILE_BYTECODE=false \
-    UV_PYTHON_PREFERENCE=managed \
-    UV_PYTHON_INSTALL_DIR=/opt/uv/python \
-    UV_LINK_MODE=copy \
-    UV_PROJECT_ENVIRONMENT=/venv \
-    VIRTUAL_ENV=/venv \
-    PATH="/venv/bin:/opt/node/bin:$PATH"
-
-SHELL ["/bin/bash", "-o", "pipefail", "-o", "errexit", "-o", "errtrace", "-o", "nounset", "-c"]
-WORKDIR "$CODE_DIR"
-
-RUN cp /VERSION.txt /ABX-DL-VERSION.txt \
-    && (echo "[i] Docker build for ArchiveBox multistage starting..." \
-    && echo "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) (${TARGETARCH} ${TARGETVARIANT})" \
-    && echo "BUILD_START_TIME=$(date +"%Y-%m-%d %H:%M:%S %s") TZ=${TZ} LANG=${LANG}" \
-    && uname -a \
-    && sed -n '1,7p' /etc/os-release \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env node \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env uv \
-    ) | tee -a /VERSION.txt
-
-FROM archivebox-runtime-base AS archivebox-builder
-
-WORKDIR "$CODE_DIR"
-RUN --mount=type=cache,target=/var/cache/apt,sharing=locked,id=apt-$TARGETARCH$TARGETVARIANT \
-    --mount=type=bind,source=pyproject.toml,target=/app/pyproject.toml \
-    <<'EOF'
-echo "[+] UV Installing ArchiveBox dependencies from pyproject.toml..."
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "1";' > /etc/apt/apt.conf.d/99keep-cache
-echo 'APT::Install-Recommends "0";' > /etc/apt/apt.conf.d/99no-install-recommends
-echo 'APT::Install-Suggests "0";' > /etc/apt/apt.conf.d/99no-install-suggests
-rm -f /etc/apt/apt.conf.d/docker-clean
-apt-get update -qq
-apt-get install -qq -y --no-install-recommends \
-    build-essential gcc libldap2-dev libsasl2-dev libssl-dev x11-utils
-# Extend abx-dl's existing venv instead of clearing it. Clearing and later
-# copying a complete replacement over the base duplicates every venv byte in
-# the final overlay history even when most packages are unchanged.
-/usr/bin/uv pip install --no-cache setuptools pip wheel
-
-mkdir -p /tmp/archivebox-uv-project
-/venv/bin/python - <<'PY'
-from pathlib import Path
-import json
-import re
-import urllib.request
-
-source = Path("/app/pyproject.toml")
-target = Path("/tmp/archivebox-uv-project/pyproject.toml")
-text = source.read_text()
-text = text.replace(
-    'environments = ["sys_platform == \'darwin\'", "sys_platform == \'linux\'"]',
-    'environments = ["sys_platform == \'linux\'"]',
-)
-
-# Docker builds need the just-published internal abx wheels immediately, but
-# PyPI simple can lag the version JSON endpoints by tens of minutes. Generate a
-# Docker-only dependency view from the version JSON so the published package
-# metadata stays normal while image builds remain resumable after a release.
-for package in ("abxbus", "abxpkg", "abx-plugins", "abx-dl"):
-    match = re.search(
-        rf'"{re.escape(package)}(?P<operator>==|>=)(?P<version>[^"]+)"',
-        text,
-    )
-    if not match:
-        continue
-    version = match.group("version")
-    with urllib.request.urlopen(f"https://pypi.org/pypi/{package}/{version}/json", timeout=20) as response:
-        data = json.load(response)
-    wheel_url = next(url["url"] for url in data["urls"] if url["filename"].endswith(".whl"))
-    text = re.sub(
-        rf'"{re.escape(package)}(?:==|>=)[^"]+"',
-        f'"{package} @ {wheel_url}"',
-        text,
-        count=1,
-    )
-
-target.write_text(text)
-PY
-
-/usr/bin/uv sync \
-    --project /tmp/archivebox-uv-project \
-    --no-cache \
-    --no-dev \
-    --inexact \
-    --no-install-project \
-    --no-install-workspace \
-    --no-sources
-builder_abxpkg_lib_dir=/tmp/archivebox-builder-abxpkg
-ABXPKG_NO_CACHE=True abxpkg env --install --binproviders=env,apt --lib="$builder_abxpkg_lib_dir" --overrides='{"apt":{"install_args":["binutils"]}}' strip >/dev/null
-/usr/bin/find /venv/lib/python3.*/site-packages -type f -name '*.so' -print0 > /tmp/archivebox-native-libraries
-while IFS= read -r -d '' native_library; do
-    magic=''
-    if IFS= read -r -N 4 magic < "$native_library" && [[ "$magic" == $'\x7fELF' ]]; then
-        "$builder_abxpkg_lib_dir/env/bin/strip" --strip-unneeded "$native_library" || exit $?
-    fi
-done < /tmp/archivebox-native-libraries
-rm -f /tmp/archivebox-native-libraries
-rm -f /venv/bin/uv /venv/bin/uvx
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get purge -y binutils build-essential gcc libldap2-dev libsasl2-dev libssl-dev
-abxpkg run --binproviders=env --lib="$builder_abxpkg_lib_dir" apt-get autoremove -y
-rm -rf "$builder_abxpkg_lib_dir"
-rm -rf /venv/lib/python3.*/site-packages/pip* \
-    /venv/lib/python3.*/site-packages/wheel* \
-    /venv/bin/pip /venv/bin/pip3 /venv/bin/pip3.* /venv/bin/wheel
-echo 'Binary::apt::APT::Keep-Downloaded-Packages "0";' > /etc/apt/apt.conf.d/99keep-cache
-rm -rf /var/lib/apt/lists/* /tmp/archivebox-uv-project
-EOF
-
-COPY --chown=root:root --chmod=755 "." "$CODE_DIR/"
-# Compile only ArchiveBox's installed package. Touching all of /venv here would
-# copy every inherited abx-dl file into a new layer merely to change its mtime.
-RUN echo "[*] Installing ArchiveBox Python source code from $CODE_DIR..." \
-    && COMMIT_HASH="$( \
-        if [[ "$ARCHIVEBOX_COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then \
-            echo "$ARCHIVEBOX_COMMIT_HASH"; \
-        elif [[ -f "$CODE_DIR/.git/HEAD" ]]; then \
-            HEAD_REF="$(cat "$CODE_DIR/.git/HEAD")"; \
-            if [[ "$HEAD_REF" =~ ^[0-9a-fA-F]{40}$ ]]; then \
-                echo "$HEAD_REF"; \
-            elif [[ "$HEAD_REF" == ref:\ * ]]; then \
-                REF_PATH="${HEAD_REF#ref: }"; \
-                cat "$CODE_DIR/.git/$REF_PATH" 2>/dev/null || awk -v ref="$REF_PATH" '$2 == ref {print $1}' "$CODE_DIR/.git/packed-refs" 2>/dev/null || true; \
-            fi; \
-        fi)" \
-    && if [[ "$COMMIT_HASH" =~ ^[0-9a-fA-F]{40}$ ]]; then echo "COMMIT_HASH=$COMMIT_HASH" | tee -a /VERSION.txt; fi \
-    && /usr/bin/uv pip install --no-cache --no-deps "$CODE_DIR" \
-    && rm -f /venv/bin/uv /venv/bin/uvx \
-    && cd / \
-    && ARCHIVEBOX_PY_DIR="$(/venv/bin/python -c 'import pathlib, archivebox; print(pathlib.Path(archivebox.__file__).parent)')" \
-    && /venv/bin/python -m compileall --invalidation-mode checked-hash -q "$ARCHIVEBOX_PY_DIR" \
-    && test -f "$(/venv/bin/python -c 'import archivebox; print(archivebox.__cached__)')" \
-    && /usr/bin/uv pip show archivebox | tee -a /VERSION.txt
-
-# The builder installs and purges compilers in one layer, so its final
-# filesystem is already runtime-clean. Preserve that ancestry: starting again
-# from archivebox-runtime-base and COPYing /venv would bake a second full venv
-# over the inherited one instead of recording only ArchiveBox's package delta.
-FROM archivebox-builder
-
-LABEL name="archivebox" \
-    maintainer="Nick Sweeting <dockerfile@archivebox.io>" \
-    description="All-in-one self-hosted internet archiving solution" \
-    homepage="https://github.com/ArchiveBox/ArchiveBox" \
-    documentation="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker" \
-    org.opencontainers.image.title="ArchiveBox" \
-    org.opencontainers.image.vendor="ArchiveBox" \
-    org.opencontainers.image.description="All-in-one self-hosted internet archiving solution" \
-    org.opencontainers.image.source="https://github.com/ArchiveBox/ArchiveBox" \
-    com.docker.image.source.entrypoint="Dockerfile"
-
-COPY --from=sonic /usr/local/bin/sonic /usr/local/bin/sonic
-COPY --chown=root:root --chmod=755 "etc/sonic.cfg" /etc/sonic.cfg
-
-# The builder invokes abxpkg as root for temporary ELF stripping, which can
-# rewrite derived state ownership. Restore UID 911 before resolving binaries
-# as the runtime user; the old copy-based final stage hid this dependency.
-RUN echo "[*] Setting up $ARCHIVEBOX_USER user uid=${DEFAULT_ARCHIVEBOX_UID}..." \
-    && printf 'export PATH="/venv/bin:/opt/node/bin:$PATH"\n' > /etc/profile.d/archivebox-path.sh \
-    && ln -sf /venv/bin/archivebox /usr/local/bin/archivebox \
-    && ln -sf /venv/bin/daphne /usr/local/bin/daphne \
-    && ln -sf /venv/bin/supervisord /usr/local/bin/supervisord \
-    && ln -sf /venv/bin/supervisorctl /usr/local/bin/supervisorctl \
-    && getent group "$ARCHIVEBOX_USER" >/dev/null || groupadd --system "$ARCHIVEBOX_USER" \
-    && id -u "$ARCHIVEBOX_USER" >/dev/null 2>&1 || useradd --system --create-home --gid "$ARCHIVEBOX_USER" --groups audio,video "$ARCHIVEBOX_USER" \
-    && usermod --append --groups audio,video "$ARCHIVEBOX_USER" \
-    && [[ "$(id -u "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_UID" ]] || usermod -u "$DEFAULT_ARCHIVEBOX_UID" "$ARCHIVEBOX_USER" \
-    && [[ "$(id -g "$ARCHIVEBOX_USER")" == "$DEFAULT_ARCHIVEBOX_GID" ]] || groupmod -g "$DEFAULT_ARCHIVEBOX_GID" "$ARCHIVEBOX_USER" \
-    && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
-    && install -d -o "$DEFAULT_ARCHIVEBOX_UID" -g "$DEFAULT_ARCHIVEBOX_GID" "/home/$ARCHIVEBOX_USER" \
-    && chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$DATA_DIR" "$TMP_DIR" \
-    && chown -R "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups abxpkg load --binproviders=env sonic | tee -a /VERSION.txt \
-    && openssl rand -hex 16 > /etc/machine-id \
-    && echo -e "\nARCHIVEBOX_USER=$ARCHIVEBOX_USER ARCHIVEBOX_UID=$(id -u "$ARCHIVEBOX_USER") ARCHIVEBOX_GID=$(id -g "$ARCHIVEBOX_USER")" | tee -a /VERSION.txt \
-    && echo -e "TMP_DIR=$TMP_DIR\nABXPKG_LIB_DIR=$ABXPKG_LIB_DIR\nPLAYWRIGHT_BROWSERS_PATH=$PLAYWRIGHT_BROWSERS_PATH\nMACHINE_ID=$(cat /etc/machine-id)\n" | tee -a /VERSION.txt
-
-WORKDIR "$DATA_DIR"
-RUN echo "[+] Initializing image collection..." \
-    && find "$DATA_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} + \
-    && HOME="$TMP_DIR" archivebox init \
-    && (chown "$DEFAULT_ARCHIVEBOX_UID:$DEFAULT_ARCHIVEBOX_GID" \
-        "$DATA_DIR" "$DATA_DIR"/.archivebox_id "$DATA_DIR"/ArchiveBox.conf "$DATA_DIR"/index.sqlite3 \
-        "$DATA_DIR"/logs "$DATA_DIR"/logs/* "$DATA_DIR"/sources \
-        "$DATA_DIR"/archive "$DATA_DIR"/archive/users "$DATA_DIR"/personas \
-        "$DATA_DIR"/tmp "$DATA_DIR"/tmp/* \
-        "$CONFIG_DIR" "$CONFIG_DIR"/config.env "$CONFIG_DIR"/derived.env \
-        "$TMP_DIR" "$ABXPKG_LIB_DIR" "$XDG_CACHE_HOME" "$PLAYWRIGHT_BROWSERS_PATH" \
-        2>/dev/null || true) \
-    && find "$TMP_DIR" -mindepth 1 -maxdepth 1 -exec rm -rf {} +
-
-RUN chmod +x "$CODE_DIR"/bin/*.sh \
-    && chmod g+w "$TMP_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH"
-
-RUN for forbidden_bin in gcc g++ make; do ! abxpkg load --binproviders=env "$forbidden_bin" >/dev/null 2>&1 || (echo "Unexpected build tool in runtime: $forbidden_bin" >&2 && exit 1); done \
-    && stat -c "%U:%G %a %n" "$CONFIG_DIR" "$ABXPKG_LIB_DIR" "$PLAYWRIGHT_BROWSERS_PATH" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$CONFIG_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups test -w "$ABXPKG_LIB_DIR" \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox install \
-    && setpriv --reuid="$ARCHIVEBOX_USER" --regid="$ARCHIVEBOX_USER" --init-groups archivebox version 2>&1 | tee -a /VERSION.txt \
-    && rm -rf /root/.cache /var/cache/apt/* /var/lib/apt/lists/*
-
-RUN (echo -e "\n\n[√] Finished ArchiveBox multistage Docker build successfully." \
-    && echo -e "PLATFORM=${TARGETPLATFORM} ARCH=$(uname -m) (${TARGETARCH} ${TARGETVARIANT})" \
-    && echo -e "BUILD_END_TIME=$(date +"%Y-%m-%d %H:%M:%S %s")\n\n" \
-    ) | tee -a /VERSION.txt
-
-WORKDIR "$DATA_DIR"
-VOLUME "$DATA_DIR"
-EXPOSE 8000
-
-HEALTHCHECK --interval=30s --timeout=20s --retries=15 \
-    CMD curl --fail --silent --show-error --max-time 5 --connect-timeout 2 'http://admin.archivebox.localhost:8000/health/' | grep -q 'OK'
-
-ENTRYPOINT ["dumb-init", "--", "/app/bin/docker_entrypoint.sh"]
-CMD ["archivebox", "server", "--init", "0.0.0.0:8000"]
@@ -1,6 +1,6 @@
 MIT License

-Copyright (c) 2026 Nick Sweeting
+Copyright (c) 2017 Nick Sweeting

 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal
@@ -0,0 +1,109 @@
+#!/usr/bin/env python3
+# Bookmark Archiver
+# Nick Sweeting 2017 | MIT License
+# https://github.com/pirate/bookmark-archiver
+
+import sys
+
+from datetime import datetime
+
+from parse import parse_links
+from links import validate_links
+from archive_methods import archive_links, _RESULTS_TOTALS
+from index import (
+    write_links_index,
+    write_link_index,
+    parse_json_links_index,
+    parse_json_link_index,
+)
+from config import (
+    ARCHIVE_PERMISSIONS,
+    HTML_FOLDER,
+    ARCHIVE_FOLDER,
+    ANSI,
+    TIMEOUT,
+)
+from util import (
+    download_url,
+    check_dependencies,
+    progress,
+    cleanup_archive,
+)
+
+__DESCRIPTION__ = 'Bookmark Archiver: Create a browsable html archive of a list of links.'
+__DOCUMENTATION__ = 'https://github.com/pirate/bookmark-archiver'
+
+
+def update_archive(export_path, links, resume=None, append=True):
+    """update or create index.html+json given a path to an export file containing new links"""
+
+    start_ts = datetime.now().timestamp()
+
+    # loop over links and archive them
+    archive_links(ARCHIVE_FOLDER, links, export_path, resume=resume)
+
+    # print timing information & summary
+    end_ts = datetime.now().timestamp()
+    seconds = round(end_ts - start_ts, 1)
+    duration = '{} min'.format(seconds / 60) if seconds > 60 else '{} sec'.format(seconds)
+    print('{}[√] [{}] Archive update complete ({}){}'.format(
+        ANSI['green'],
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        duration,
+        ANSI['reset'],
+    ))
+    print('    - {} entries skipped'.format(_RESULTS_TOTALS['skipped']))
+    print('    - {} entries updated'.format(_RESULTS_TOTALS['succeded']))
+    print('    - {} errors'.format(_RESULTS_TOTALS['failed']))
+
+
+def update_index(export_path, resume=None, append=True):
+    """handling parsing new links into the json index, returns a set of clean links"""
+
+    # parse an validate the export file
+    new_links = validate_links(parse_links(export_path))
+
+    # load existing links if archive folder is present
+    existing_links = []
+    if append:
+        existing_links = parse_json_links_index(HTML_FOLDER)
+        links = validate_links(existing_links + new_links)
+        
+
+    # merge existing links and new links
+    num_new_links = len(links) - len(existing_links)
+    print('[*] [{}] Adding {} new links from {} to index'.format(
+        datetime.now().strftime('%Y-%m-%d %H:%M:%S'),
+        num_new_links,
+        export_path,
+    ))
+
+    # write link index html & json
+    write_links_index(HTML_FOLDER, links)
+
+    return links
+
+
+if __name__ == '__main__':
+    argc = len(sys.argv)
+
+    if argc < 2 or sys.argv[1] in ('-h', '--help', 'help'):
+        print(__DESCRIPTION__)
+        print("Documentation:     {}".format(__DOCUMENTATION__))
+        print("")
+        print("Usage:")
+        print("    ./archive.py ~/Downloads/bookmarks_export.html")
+        print("")
+        raise SystemExit(0)
+
+    export_path = sys.argv[1]                        # path to export file
+    resume_from = sys.argv[2] if argc > 2 else None  # timestamp to resume dowloading from
+
+    if any(export_path.startswith(s) for s in ('http://', 'https://', 'ftp://')):
+        export_path = download_url(export_path)
+
+    links = update_index(export_path, resume=resume_from, append=True)
+
+    # make sure folder structure is sane
+    cleanup_archive(ARCHIVE_FOLDER, links)
+    update_archive(export_path, links, resume=resume_from, append=True)
@@ -0,0 +1,427 @@
+import os
+
+from functools import wraps
+from datetime import datetime
+from subprocess import run, PIPE, DEVNULL
+
+from index import html_appended_url, parse_json_link_index, write_link_index
+from links import links_after_timestamp
+from config import (
+    ARCHIVE_DIR,
+    CHROME_BINARY,
+    FETCH_WGET,
+    FETCH_WGET_REQUISITES,
+    FETCH_PDF,
+    FETCH_SCREENSHOT,
+    RESOLUTION,
+    SUBMIT_ARCHIVE_DOT_ORG,
+    FETCH_AUDIO,
+    FETCH_VIDEO,
+    FETCH_FAVICON,
+    WGET_USER_AGENT,
+    CHROME_USER_DATA_DIR,
+    TIMEOUT,
+    ANSI,
+)
+from util import (
+    check_dependencies,
+    progress,
+    chmod_file,
+)
+
+
+_RESULTS_TOTALS = {   # globals are bad, mmkay
+    'skipped': 0,
+    'succeded': 0,
+    'failed': 0,
+}
+
+def archive_links(out_dir, links, export_path, resume=None):
+    check_dependencies()
+
+    to_archive = links_after_timestamp(links, resume)
+    try:
+        for idx, link in enumerate(to_archive):
+            link_dir = os.path.join(out_dir, link['timestamp'])
+            archive_link(link_dir, link)
+    
+    except (KeyboardInterrupt, SystemExit, Exception) as e:
+        print('{red}[X] Archive update stopped on #{idx} out of {total} links{reset}'.format(
+            **ANSI,
+            idx=idx,
+            total=len(list(to_archive)),
+        ))
+        print('    Continue where you left off by running:')
+        print('       ./archive.py {} {}'.format(
+            export_path,
+            link['timestamp'],
+        ))
+        if not isinstance(e, KeyboardInterrupt):
+            raise e
+        raise SystemExit(1)
+
+
+def archive_link(out_dir, link, overwrite=False):
+    """download the DOM, PDF, and a screenshot into a folder named after the link's timestamp"""
+
+    link = {**parse_json_link_index(out_dir), **link}
+    log_link_archive(out_dir, link)
+
+    if FETCH_WGET:
+        link = fetch_wget(out_dir, link, overwrite=overwrite)
+
+    if FETCH_PDF:
+        link = fetch_pdf(out_dir, link, overwrite=overwrite)
+
+    if FETCH_SCREENSHOT:
+        link = fetch_screenshot(out_dir, link, overwrite=overwrite)
+
+    if SUBMIT_ARCHIVE_DOT_ORG:
+        link = archive_dot_org(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_AUDIO:
+    #     link = fetch_audio(out_dir, link, overwrite=overwrite)
+
+    # if FETCH_VIDEO:
+    #     link = fetch_video(out_dir, link, overwrite=overwrite)
+
+    if FETCH_FAVICON:
+        link = fetch_favicon(out_dir, link, overwrite=overwrite)
+
+    write_link_index(out_dir, link)
+    
+    return link
+
+def log_link_archive(out_dir, link):
+    update_existing = os.path.exists(out_dir)
+    if not update_existing:
+        os.makedirs(out_dir)
+    
+    print('[{symbol_color}{symbol}{reset}] [{timestamp}] "{title}": {blue}{base_url}{reset}'.format(
+        symbol='*' if update_existing else '+',
+        symbol_color=ANSI['black' if update_existing else 'green'],
+        **link,
+        **ANSI,
+    ))
+    if link['type']:
+        print('    i Type: {}'.format(link['type']))
+
+    print('    {} ({})'.format(out_dir, 'updating' if update_existing else 'creating'))
+
+
+
+def attach_result_to_link(method):
+    """
+    Instead of returning a result={output:'...', status:'success'} object,
+    attach that result to the links's history & latest fields, then return
+    the updated link object.
+    """
+    def decorator(fetch_func):
+        @wraps(fetch_func)
+        def timed_fetch_func(out_dir, link, overwrite=False, **kwargs):
+            # initialize methods and history json field on link
+            link['latest'] = link.get('latest') or {}
+            link['latest'][method] = link['latest'].get(method) or None
+            link['history'] = link.get('history') or {}
+            link['history'][method] = link['history'].get(method) or []
+
+            start_ts = datetime.now().timestamp()
+
+            # if a valid method output is already present, dont run the fetch function
+            if link['latest'][method] and not overwrite:
+                print('    √ Skipping: {}'.format(method))
+                result = None
+            else:
+                print('    - Fetching: {}'.format(method))
+                result = fetch_func(out_dir, link, **kwargs)
+
+            end_ts = datetime.now().timestamp()
+            duration = str(end_ts * 1000 - start_ts * 1000).split('.')[0]
+
+            # append a history item recording fail/success
+            history_entry = {
+                'timestamp': str(start_ts).split('.')[0],
+            }
+            if result is None:
+                history_entry['status'] = 'skipped'
+            elif isinstance(result.get('output'), Exception):
+                history_entry['status'] = 'failed'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+            else:
+                history_entry['status'] = 'succeded'
+                history_entry['duration'] = duration
+                history_entry.update(result or {})
+                link['history'][method].append(history_entry)
+                link['latest'][method] = result['output']
+            
+            _RESULTS_TOTALS[history_entry['status']] += 1
+            
+            return link
+        return timed_fetch_func
+    return decorator
+
+
+@attach_result_to_link('wget')
+def fetch_wget(out_dir, link, requisites=FETCH_WGET_REQUISITES, timeout=TIMEOUT):
+    """download full site using wget"""
+
+    if os.path.exists(os.path.join(out_dir, link['domain'])):
+        return {'output': html_appended_url(link), 'status': 'skipped'}
+
+    CMD = [
+        *'wget --timestamping --adjust-extension --no-parent'.split(' '),                # Docs: https://www.gnu.org/software/wget/manual/wget.html
+        *(('--page-requisites', '--convert-links') if FETCH_WGET_REQUISITES else ()),
+        *(('--user-agent="{}"'.format(WGET_USER_AGENT),) if WGET_USER_AGENT else ()),
+        link['url'],
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # index.html
+        end()
+        output = html_appended_url(link)
+        if result.returncode > 0:
+            print('       got wget response code {}:'.format(result.returncode))
+            print('\n'.join('         ' + line for line in (result.stderr or result.stdout).decode().rsplit('\n', 10)[-10:] if line.strip()))
+            # raise Exception('Failed to wget download')
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('pdf')
+def fetch_pdf(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR):
+    """print PDF of site to file using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'output.pdf')):
+        return {'output': 'output.pdf', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --print-to-pdf'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # output.pdf
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to print PDF')
+        output = 'output.pdf'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+
+@attach_result_to_link('screenshot')
+def fetch_screenshot(out_dir, link, timeout=TIMEOUT, user_data_dir=CHROME_USER_DATA_DIR, resolution=RESOLUTION):
+    """take screenshot of site using chrome --headless"""
+
+    if link['type'] in ('PDF', 'image'):
+        return {'output': html_appended_url(link)}
+    
+    if os.path.exists(os.path.join(out_dir, 'screenshot.png')):
+        return {'output': 'screenshot.png', 'status': 'skipped'}
+
+    CMD = [
+        CHROME_BINARY,
+        *'--headless --disable-gpu --screenshot'.split(' '),
+        *chrome_data_dir_args(user_data_dir),
+        '--window-size={}'.format(resolution),
+        link['url']
+    ]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=PIPE, cwd=out_dir, timeout=timeout + 1)  # sreenshot.png
+        end()
+        if result.returncode:
+            print('     ', (result.stderr or result.stdout).decode())
+            raise Exception('Failed to take screenshot')
+        chmod_file('screenshot.png', cwd=out_dir)
+        output = 'screenshot.png'
+    except Exception as e:
+        end()
+        print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+    
+
+@attach_result_to_link('archive_org')
+def archive_dot_org(out_dir, link, timeout=TIMEOUT):
+    """submit site to archive.org for archiving via their service, save returned archive url"""
+
+    path = os.path.join(out_dir, 'archive.org.txt')
+    if os.path.exists(path):
+        archive_org_url = open(path, 'r').read().strip()
+        return {'output': archive_org_url, 'status': 'skipped'}
+
+    submit_url = 'https://web.archive.org/save/{}'.format(link['url'].split('?', 1)[0])
+
+    success = False
+    CMD = ['curl', '-I', submit_url]
+    end = progress(timeout, prefix='      ')
+    try:
+        result = run(CMD, stdout=PIPE, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # archive.org.txt
+        end()
+
+        # Parse archive.org response headers
+        headers = result.stdout.splitlines()
+        content_location = [h for h in headers if b'Content-Location: ' in h]
+        errors = [h for h in headers if h and b'X-Archive-Wayback-Runtime-Error: ' in h]
+
+        if content_location:
+            archive_path = content_location[0].split(b'Content-Location: ', 1)[-1].decode('utf-8')
+            saved_url = 'https://web.archive.org{}'.format(archive_path)
+            success = True
+
+        elif len(errors) == 1 and b'RobotAccessControlException' in errors[0]:
+            output = submit_url
+            # raise Exception('Archive.org denied by {}/robots.txt'.format(link['domain']))
+        elif errors:
+            raise Exception(', '.join(e.decode() for e in errors))
+        else:
+            raise Exception('Failed to find "Content-Location" URL header in Archive.org response.')
+    except Exception as e:
+        end()
+        print('       Visit url to see output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    if success:
+        with open(os.path.join(out_dir, 'archive.org.txt'), 'w', encoding='utf-8') as f:
+            f.write(saved_url)
+        chmod_file('archive.org.txt', cwd=out_dir)
+        output = saved_url
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+@attach_result_to_link('favicon')
+def fetch_favicon(out_dir, link, timeout=TIMEOUT):
+    """download site favicon from google's favicon api"""
+
+    if os.path.exists(os.path.join(out_dir, 'favicon.ico')):
+        return {'output': 'favicon.ico', 'status': 'skipped'}
+
+    CMD = ['curl', 'https://www.google.com/s2/favicons?domain={domain}'.format(**link)]
+    fout = open('{}/favicon.ico'.format(out_dir), 'w')
+    end = progress(timeout, prefix='      ')
+    try:
+        run(CMD, stdout=fout, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # favicon.ico
+        fout.close()
+        end()
+        chmod_file('favicon.ico', cwd=out_dir)
+        output = 'favicon.ico'
+    except Exception as e:
+        fout.close()
+        end()
+        print('       Run to see full output:', ' '.join(CMD))
+        print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+        output = e
+
+    return {
+        'cmd': CMD,
+        'output': output,
+    }
+
+# @attach_result_to_link('audio')
+# def fetch_audio(out_dir, link, timeout=TIMEOUT):
+#     """Download audio rip using youtube-dl"""
+
+#     if link['type'] not in ('soundcloud',)\
+#        and 'audio' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'audio')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading audio')
+#         CMD = [
+#             "youtube-dl -x --audio-format mp3 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # audio/audio.mp3
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download audio')
+#             chmod_file('audio.mp3', cwd=out_dir)
+#             return 'audio.mp3'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping audio download')
+
+# @attach_result_to_link('video')
+# def fetch_video(out_dir, link, timeout=TIMEOUT):
+#     """Download video rip using youtube-dl"""
+
+#     if link['type'] not in ('youtube', 'youku', 'vimeo')\
+#        and 'video' not in link['tags']:
+#         return
+
+#     path = os.path.join(out_dir, 'video')
+
+#     if not os.path.exists(path) or overwrite:
+#         print('    - Downloading video')
+#         CMD = [
+#             "youtube-dl -x --video-format mp4 --audio-quality 0 -o '%(title)s.%(ext)s'",
+#             link['url'],
+#         ]
+#         end = progress(timeout, prefix='      ')
+#         try:
+#             result = run(CMD, stdout=DEVNULL, stderr=DEVNULL, cwd=out_dir, timeout=timeout + 1)  # video/movie.mp4
+#             end()
+#             if result.returncode:
+#                 print('     ', result.stderr.decode())
+#                 raise Exception('Failed to download video')
+#             chmod_file('video.mp4', cwd=out_dir)
+#             return 'video.mp4'
+#         except Exception as e:
+#             end()
+#             print('       Run to see full output:', 'cd {}; {}'.format(out_dir, ' '.join(CMD)))
+#             print('       {}Failed: {} {}{}'.format(ANSI['red'], e.__class__.__name__, e, ANSI['reset']))
+#             raise
+#     else:
+#         print('    √ Skipping video download')
+
+
+def chrome_data_dir_args(user_data_dir=CHROME_USER_DATA_DIR):
+    default = os.path.expanduser('~/Library/Application Support/Google/Chrome/Default')
+    if user_data_dir:
+        return ('--user-data-dir={}'.format(user_data_dir),)
+    elif os.path.exists(default):
+        return ('--user-data-dir={}'.format(default),)
+    return ()
@@ -1,6 +0,0 @@
-[flake8]
-ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E272,E701,E731,W293,W503,W291,W391
-select = F,E9,W
-max-line-length = 130
-max-complexity = 10
-exclude = migrations,tests,node_modules,vendor,venv,.venv,.venv2,.docker-venv,data,data*
@@ -1 +0,0 @@
-../README.md
@@ -1,150 +0,0 @@
-#!/usr/bin/env python3
-
-# Welcome to the ArchiveBox source code! Thanks for checking it out!
-#
-# "We are swimming upstream against a great torrent of disorganization.
-# In this, our main obligation is to establish arbitrary enclaves of order and system.
-# It is the greatest possible victory to be, to continue to be, and to have been.
-# No defeat can deprive us of the success of having existed for some moment of time
-# in a universe that seems indifferent to us."
-# --Norber Weiner
-
-__package__ = "archivebox"
-
-import os
-import sys
-from pathlib import Path
-from typing import Protocol, cast
-
-
-class _ReconfigurableStream(Protocol):
-    def reconfigure(self, *, line_buffering: bool) -> object: ...
-
-
-# Force unbuffered output for real-time logs
-if hasattr(sys.stdout, "reconfigure"):
-    cast(_ReconfigurableStream, sys.stdout).reconfigure(line_buffering=True)
-    cast(_ReconfigurableStream, sys.stderr).reconfigure(line_buffering=True)
-os.environ["PYTHONUNBUFFERED"] = "1"
-
-ASCII_LOGO = """
- █████╗ ██████╗  ██████╗██╗  ██╗██╗██╗   ██╗███████╗ ██████╗  ██████╗ ██╗  ██╗
-██╔══██╗██╔══██╗██╔════╝██║  ██║██║██║   ██║██╔════╝ ██╔══██╗██╔═══██╗╚██╗██╔╝
-███████║██████╔╝██║     ███████║██║██║   ██║█████╗   ██████╔╝██║   ██║ ╚███╔╝ 
-██╔══██║██╔══██╗██║     ██╔══██║██║╚██╗ ██╔╝██╔══╝   ██╔══██╗██║   ██║ ██╔██╗ 
-██║  ██║██║  ██║╚██████╗██║  ██║██║ ╚████╔╝ ███████╗ ██████╔╝╚██████╔╝██╔╝ ██╗
-╚═╝  ╚═╝╚═╝  ╚═╝ ╚═════╝╚═╝  ╚═╝╚═╝  ╚═══╝  ╚══════╝ ╚═════╝  ╚═════╝ ╚═╝  ╚═╝
-"""
-
-PACKAGE_DIR = Path(__file__).resolve().parent
-
-os.environ["DJANGO_SETTINGS_MODULE"] = "archivebox.core.settings"
-os.environ["TZ"] = "UTC"
-
-# detect ArchiveBox user's UID/GID based on data dir ownership
-from .config.permissions import drop_privileges  # noqa
-
-drop_privileges()
-
-from .misc.checks import check_not_root, check_io_encoding  # noqa
-
-check_not_root()
-check_io_encoding()
-
-from .config.version import VERSION  # noqa
-
-
-__version__ = VERSION
-__author__ = "ArchiveBox"
-__license__ = "MIT"
-
-
-def __getattr__(name: str):
-    if name == "CONSTANTS":
-        from .config.constants import CONSTANTS
-
-        os.environ.setdefault("MACHINE_ID", CONSTANTS.MACHINE_ID)
-        return CONSTANTS
-    if name == "DATA_DIR":
-        from .config.paths import DATA_DIR
-
-        return DATA_DIR
-    if name == "VERSION":
-        return VERSION
-    if name in ("BUILTIN_PLUGINS_DIR", "USER_PLUGINS_DIR", "ALL_PLUGINS", "LOADED_PLUGINS"):
-        from abx_plugins import get_plugins_dir
-        from .config.constants import CONSTANTS
-
-        builtin_plugins_dir = Path(get_plugins_dir()).resolve()
-        user_plugins_dir = CONSTANTS.USER_PLUGINS_DIR
-        plugins = {
-            "builtin": builtin_plugins_dir,
-            "user": user_plugins_dir,
-        }
-        values = {
-            "BUILTIN_PLUGINS_DIR": builtin_plugins_dir,
-            "USER_PLUGINS_DIR": user_plugins_dir,
-            "ALL_PLUGINS": plugins,
-            "LOADED_PLUGINS": plugins,
-        }
-        return values[name]
-    raise AttributeError(name)
-
-
-__all__ = (
-    "ASCII_LOGO",
-    "ASCII_ICON",
-    "PACKAGE_DIR",
-    "DATA_DIR",
-    "CONSTANTS",
-    "VERSION",
-    "BUILTIN_PLUGINS_DIR",
-    "USER_PLUGINS_DIR",
-    "ALL_PLUGINS",
-    "LOADED_PLUGINS",
-)
-
-ASCII_ICON = """
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-██████████████████████████████████████████████████████████████████████████████████████████████████ 
-         ██                                                                            ██          
-         ██                                                                            ██        
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                    ████████████████████████████████████                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ██       █████████████████████████ █                    ██          
-         ██                    ████████████████████████████████████                    ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                 ██████████████████████████████████████████                 ██          
-         ██                 ██████████████████████████████████████████                 ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██          
-         ██                                                                            ██        
-         ████████████████████████████████████████████████████████████████████████████████          
-"""
@@ -1,20 +0,0 @@
-#!/usr/bin/env python3
-"""This is the entrypoint for python -m archivebox ..."""
-
-__package__ = "archivebox"
-
-import archivebox  # noqa # make sure monkey patches are applied before anything else
-import sys
-
-from .cli import main
-
-ASCII_LOGO_MINI = r"""
-     _             _     _           ____
-    / \   _ __ ___| |__ (_)_   _____| __ )  _____  __
-   / _ \ | '__/ __| '_ \| \ \ / / _ \  _ \ / _ \ \/ /
-  / ___ \| | | (__| | | | |\ V /  __/ |_) | (_) >  <
- /_/   \_\_|  \___|_| |_|_| \_/ \___|____/ \___/_/\_\
-"""
-
-if __name__ == "__main__":
-    main(args=sys.argv[1:])
@@ -1 +0,0 @@
-__package__ = "archivebox.api"
@@ -1,72 +0,0 @@
-__package__ = "archivebox.api"
-
-from django import forms
-from django.contrib import admin
-from django.http import HttpRequest
-from django.utils.text import capfirst
-from signal_webhooks.admin import WebhookAdmin, WebhookModelForm
-from signal_webhooks.settings import webhook_settings
-from signal_webhooks.utils import get_webhook_model, model_from_reference
-
-from archivebox.base_models.admin import card_fieldset, BaseModelAdmin
-
-from archivebox.api.models import APIToken
-
-
-def _webhook_fields(*names: str) -> tuple[str, ...]:
-    model_fields = {field.name for field in get_webhook_model()._meta.fields}
-    return tuple(name for name in names if name in model_fields)
-
-
-class APITokenAdmin(BaseModelAdmin):
-    list_display = ("created_at", "id", "created_by", "token_redacted", "expires")
-    sort_fields = ("id", "created_at", "created_by", "expires")
-    readonly_fields = ("created_at", "modified_at")
-    search_fields = ("id", "created_by__username", "token")
-
-    fieldsets = (
-        card_fieldset("Token", ("token", "expires")),
-        card_fieldset("Owner", ("created_by",)),
-        card_fieldset("Timestamps", ("created_at", "modified_at")),
-    )
-
-    list_filter = ("created_by",)
-    ordering = ["-created_at"]
-    list_per_page = 100
-
-
-class OutboundWebhookAdminForm(WebhookModelForm):
-    def __init__(self, *args, **kwargs):
-        super().__init__(*args, **kwargs)
-        self.fields["ref"] = forms.ChoiceField(
-            label=self.fields["ref"].label,
-            help_text=self.fields["ref"].help_text,
-            choices=[
-                (ref, f"{capfirst(model_from_reference(ref, check_hooks=False)._meta.verbose_name_plural)} ({ref})")
-                for ref in sorted(webhook_settings.HOOKS)
-            ],
-        )
-
-
-class CustomWebhookAdmin(WebhookAdmin, BaseModelAdmin):
-    form = OutboundWebhookAdminForm
-    list_display = ("created_at", "created_by", "id", *WebhookAdmin.list_display)
-    sort_fields = _webhook_fields("created_at", "created_by", "id", "ref", "endpoint", "last_success", "last_failure")
-    readonly_fields = _webhook_fields("created_at", "modified_at", *WebhookAdmin.readonly_fields)
-
-    fieldsets = (
-        card_fieldset("Webhook", _webhook_fields("name", "signal", "ref", "endpoint", "headers", "keep_last_response"), wide=True),
-        card_fieldset("Authentication", _webhook_fields("auth_token")),
-        card_fieldset("Status", _webhook_fields("enabled", "last_success", "last_failure", "last_response")),
-        card_fieldset("Owner", _webhook_fields("created_by")),
-        card_fieldset("Timestamps", _webhook_fields("created_at", "modified_at")),
-    )
-
-    def lookup_allowed(self, lookup: str, value: str, request: HttpRequest | None = None) -> bool:
-        """Preserve WebhookAdmin's auth token filter with Django's current admin signature."""
-        return not lookup.startswith("auth_token") and admin.ModelAdmin.lookup_allowed(self, lookup, value, request)
-
-
-def register_admin(admin_site: admin.AdminSite) -> None:
-    admin_site.register(APIToken, APITokenAdmin)
-    admin_site.register(get_webhook_model(), CustomWebhookAdmin)
@@ -1,14 +0,0 @@
-__package__ = "archivebox.api"
-
-from django.apps import AppConfig
-
-
-class APIConfig(AppConfig):
-    name = "archivebox.api"
-    label = "api"
-
-
-def register_admin(admin_site):
-    from archivebox.api.admin import register_admin
-
-    register_admin(admin_site)
@@ -1,137 +0,0 @@
-__package__ = "archivebox.api"
-
-from datetime import timedelta
-
-from django.utils import timezone
-from django.http import HttpRequest
-from django.contrib.auth import authenticate
-from django.contrib.auth.models import User
-
-from ninja.security import HttpBearer, APIKeyQuery, APIKeyHeader
-from ninja.errors import HttpError
-
-
-def get_or_create_api_token(user: User | None):
-    from archivebox.api.models import APIToken
-
-    if user and user.is_superuser:
-        api_tokens = APIToken.objects.filter(created_by_id=user.pk, expires__gt=timezone.now())
-        if api_tokens.exists():
-            # unexpired token exists, use it
-            api_token = api_tokens.last()
-        else:
-            # does not exist, create a new one
-            api_token = APIToken.objects.create(created_by_id=user.pk, expires=timezone.now() + timedelta(days=30))
-
-        if api_token is None:
-            return None
-        assert api_token.is_valid(), f"API token is not valid {api_token}"
-
-        return api_token
-    return None
-
-
-def auth_using_token(token: str | None, request: HttpRequest | None = None) -> User | None:
-    """Given an API token string, check if a corresponding non-expired APIToken exists, and return its user"""
-    from archivebox.api.models import APIToken  # lazy import model to avoid loading it at urls.py import time
-
-    user: User | None = None
-
-    submitted_empty_form = str(token).strip() in ("string", "", "None", "null")
-    if not submitted_empty_form:
-        try:
-            api_token = APIToken.objects.get(token=token)
-            if api_token.is_valid() and isinstance(api_token.created_by, User):
-                user = api_token.created_by
-                if request is not None:
-                    setattr(request, "_api_token", api_token)
-        except APIToken.DoesNotExist:
-            pass
-
-    return user
-
-
-def token_from_request(request: HttpRequest) -> str:
-    token = request.GET.get("api_key") or request.headers.get("X-ArchiveBox-API-Key") or ""
-    auth_header = request.headers.get("Authorization", "")
-    if not token and auth_header.lower().startswith("bearer "):
-        token = auth_header.split(None, 1)[1].strip()
-    return token
-
-
-def authenticated_user_from_request(request: HttpRequest) -> User | None:
-    user = request.user
-    if user.is_authenticated and user.is_active:
-        return user
-
-    token = token_from_request(request)
-    token_user = auth_using_token(token=token, request=request) if token else None
-    if token_user and token_user.is_active:
-        request.user = token_user
-        return token_user
-    return None
-
-
-def auth_using_password(username: str | None, password: str | None, request: HttpRequest | None = None) -> User | None:
-    """Given a username and password, check if they are valid and return the corresponding user"""
-    user: User | None = None
-
-    submitted_empty_form = (username, password) in (("string", "string"), ("", ""), (None, None))
-    if not submitted_empty_form:
-        authenticated_user = authenticate(
-            username=username,
-            password=password,
-        )
-        if isinstance(authenticated_user, User):
-            user = authenticated_user
-    return user
-
-
-### Base Auth Types
-
-
-def _require_superuser(user: User | None, request: HttpRequest, auth_method: str) -> User | None:
-    if user and user.pk:
-        request.user = user
-        setattr(request, "_api_auth_method", auth_method)
-        if not user.is_superuser:
-            raise HttpError(403, "Valid credentials but User does not have permission (make sure user.is_superuser=True)")
-    return user
-
-
-### Django-Ninja-Provided Auth Methods
-
-
-class HeaderTokenAuth(APIKeyHeader):
-    """Allow authenticating by passing X-API-Key=xyz as a request header"""
-
-    param_name = "X-ArchiveBox-API-Key"
-
-    def authenticate(self, request: HttpRequest, key: str | None) -> User | None:
-        return _require_superuser(auth_using_token(token=key, request=request), request, self.__class__.__name__)
-
-
-class BearerTokenAuth(HttpBearer):
-    """Allow authenticating by passing Bearer=xyz as a request header"""
-
-    def authenticate(self, request: HttpRequest, token: str) -> User | None:
-        return _require_superuser(auth_using_token(token=token, request=request), request, self.__class__.__name__)
-
-
-class QueryParamTokenAuth(APIKeyQuery):
-    """Allow authenticating by passing api_key=xyz as a GET/POST query parameter"""
-
-    param_name = "api_key"
-
-    def authenticate(self, request: HttpRequest, key: str | None) -> User | None:
-        return _require_superuser(auth_using_token(token=key, request=request), request, self.__class__.__name__)
-
-
-### Enabled Auth Methods
-
-API_AUTH_METHODS = [
-    HeaderTokenAuth(),
-    BearerTokenAuth(),
-    QueryParamTokenAuth(),
-    # django_auth_superuser,       # django admin cookie auth, not secure to use with csrf=False
-]
@@ -1,32 +0,0 @@
-__package__ = "archivebox.api"
-
-from django.http import HttpResponse
-
-
-class ApiCorsMiddleware:
-    """Attach permissive CORS headers for API routes (token-based auth)."""
-
-    def __init__(self, get_response):
-        self.get_response = get_response
-
-    def __call__(self, request):
-        if request.path.startswith("/api/"):
-            if request.method == "OPTIONS" and request.META.get("HTTP_ACCESS_CONTROL_REQUEST_METHOD"):
-                response = HttpResponse(status=204)
-                return self._add_cors_headers(request, response)
-
-            response = self.get_response(request)
-            return self._add_cors_headers(request, response)
-
-        return self.get_response(request)
-
-    def _add_cors_headers(self, request, response):
-        origin = request.META.get("HTTP_ORIGIN")
-        if not origin:
-            return response
-
-        response["Access-Control-Allow-Origin"] = "*"
-        response["Access-Control-Allow-Methods"] = "GET, POST, PUT, PATCH, DELETE, OPTIONS"
-        response["Access-Control-Allow-Headers"] = "Authorization, X-ArchiveBox-API-Key, Content-Type, X-CSRFToken"
-        response["Access-Control-Max-Age"] = "600"
-        return response
@@ -1,281 +0,0 @@
-# Generated by hand on 2025-12-29
-# Creates APIToken and OutboundWebhook tables using raw SQL
-
-from django.db import migrations, models
-import django.db.models.deletion
-import django.utils.timezone
-from django.conf import settings
-from archivebox.uuid_compat import uuid7
-from archivebox.base_models.models import get_or_create_system_user_pk
-import archivebox.api.models
-import signal_webhooks.fields
-import signal_webhooks.utils
-
-
-# Raw sqlite DDL kept EXACTLY as-is; executed only on sqlite via Django's
-# RunSQL (identical statement splitting → byte-for-byte identical behavior).
-INITIAL_SQL = """
-                -- Create api_apitoken table
-                CREATE TABLE IF NOT EXISTS api_apitoken (
-                    id TEXT PRIMARY KEY NOT NULL,
-                    created_at DATETIME NOT NULL,
-                    modified_at DATETIME NOT NULL,
-
-                    token VARCHAR(32) NOT NULL UNIQUE,
-                    expires DATETIME,
-
-                    created_by_id INTEGER NOT NULL,
-
-                    FOREIGN KEY (created_by_id) REFERENCES auth_user(id) ON DELETE CASCADE
-                );
-                CREATE INDEX IF NOT EXISTS api_apitoken_created_by_id_idx ON api_apitoken(created_by_id);
-                CREATE INDEX IF NOT EXISTS api_apitoken_created_at_idx ON api_apitoken(created_at);
-                CREATE INDEX IF NOT EXISTS api_apitoken_token_idx ON api_apitoken(token);
-
-                -- Create api_outboundwebhook table
-                CREATE TABLE IF NOT EXISTS api_outboundwebhook (
-                    id TEXT PRIMARY KEY NOT NULL,
-                    created_at DATETIME NOT NULL,
-                    modified_at DATETIME NOT NULL,
-
-                    name VARCHAR(255) NOT NULL UNIQUE,
-                    signal VARCHAR(255) NOT NULL,
-                    ref VARCHAR(1024) NOT NULL,
-                    endpoint VARCHAR(2048) NOT NULL,
-                    headers TEXT NOT NULL DEFAULT '{}',
-                    auth_token TEXT NOT NULL DEFAULT '',
-                    enabled BOOLEAN NOT NULL DEFAULT 1,
-                    keep_last_response BOOLEAN NOT NULL DEFAULT 0,
-                    created DATETIME NOT NULL,
-                    updated DATETIME NOT NULL,
-                    last_response TEXT NOT NULL DEFAULT '',
-                    last_success DATETIME,
-                    last_failure DATETIME,
-
-                    created_by_id INTEGER NOT NULL,
-
-                    FOREIGN KEY (created_by_id) REFERENCES auth_user(id) ON DELETE CASCADE
-                );
-                CREATE INDEX IF NOT EXISTS api_outboundwebhook_created_by_id_idx ON api_outboundwebhook(created_by_id);
-                CREATE INDEX IF NOT EXISTS api_outboundwebhook_created_at_idx ON api_outboundwebhook(created_at);
-                CREATE INDEX IF NOT EXISTS api_outboundwebhook_name_idx ON api_outboundwebhook(name);
-                CREATE INDEX IF NOT EXISTS api_outboundwebhook_ref_idx ON api_outboundwebhook(ref);
-                    """
-
-INITIAL_REVERSE_SQL = """
-                DROP TABLE IF EXISTS api_outboundwebhook;
-                DROP TABLE IF EXISTS api_apitoken;
-                    """
-
-
-def _run_sqlite_only_sql(apps, schema_editor):
-    if schema_editor.connection.vendor != "sqlite":
-        return
-    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_forwards(
-        "api",
-        schema_editor,
-        None,
-        None,
-    )
-
-
-def _run_sqlite_only_sql_reverse(apps, schema_editor):
-    if schema_editor.connection.vendor != "sqlite":
-        return
-    migrations.RunSQL(sql=INITIAL_SQL, reverse_sql=INITIAL_REVERSE_SQL).database_backwards(
-        "api",
-        schema_editor,
-        None,
-        None,
-    )
-
-
-def _pg_sync_schema(apps, schema_editor):
-    from archivebox.misc.db import rebuild_models_from_migration_state
-
-    rebuild_models_from_migration_state(apps, schema_editor, "api", ["APIToken", "OutboundWebhook"])
-
-
-def _pg_drop_schema(apps, schema_editor):
-    from archivebox.misc.db import drop_models_on_postgres
-
-    drop_models_on_postgres(apps, schema_editor, "api", ["OutboundWebhook", "APIToken"])
-
-
-class Migration(migrations.Migration):
-    initial = True
-
-    dependencies = [
-        ("auth", "0012_alter_user_first_name_max_length"),
-        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
-    ]
-
-    operations = [
-        migrations.SeparateDatabaseAndState(
-            database_operations=[
-                migrations.RunPython(_run_sqlite_only_sql, reverse_code=_run_sqlite_only_sql_reverse),
-            ],
-            state_operations=[
-                migrations.CreateModel(
-                    name="APIToken",
-                    fields=[
-                        ("id", models.UUIDField(default=uuid7, editable=False, primary_key=True, serialize=False, unique=True)),
-                        ("created_at", models.DateTimeField(db_index=True, default=django.utils.timezone.now)),
-                        ("modified_at", models.DateTimeField(auto_now=True)),
-                        ("token", models.CharField(default=archivebox.api.models.generate_secret_token, max_length=32, unique=True)),
-                        ("expires", models.DateTimeField(blank=True, null=True)),
-                        (
-                            "created_by",
-                            models.ForeignKey(
-                                default=get_or_create_system_user_pk,
-                                on_delete=django.db.models.deletion.CASCADE,
-                                to=settings.AUTH_USER_MODEL,
-                            ),
-                        ),
-                    ],
-                    options={
-                        "verbose_name": "API Key",
-                        "verbose_name_plural": "API Keys",
-                        "app_label": "api",
-                    },
-                ),
-                migrations.CreateModel(
-                    name="OutboundWebhook",
-                    fields=[
-                        ("id", models.UUIDField(default=uuid7, editable=False, primary_key=True, serialize=False, unique=True)),
-                        ("created_at", models.DateTimeField(db_index=True, default=django.utils.timezone.now)),
-                        ("modified_at", models.DateTimeField(auto_now=True)),
-                        (
-                            "name",
-                            models.CharField(db_index=True, help_text="Webhook name.", max_length=255, unique=True, verbose_name="name"),
-                        ),
-                        (
-                            "signal",
-                            models.CharField(
-                                choices=[
-                                    ("CREATE", "Create"),
-                                    ("UPDATE", "Update"),
-                                    ("DELETE", "Delete"),
-                                    ("M2M", "M2M changed"),
-                                    ("CREATE_OR_UPDATE", "Create or Update"),
-                                    ("CREATE_OR_DELETE", "Create or Delete"),
-                                    ("CREATE_OR_M2M", "Create or M2M changed"),
-                                    ("UPDATE_OR_DELETE", "Update or Delete"),
-                                    ("UPDATE_OR_M2M", "Update or M2M changed"),
-                                    ("DELETE_OR_M2M", "Delete or M2M changed"),
-                                    ("CREATE_UPDATE_OR_DELETE", "Create, Update or Delete"),
-                                    ("CREATE_UPDATE_OR_M2M", "Create, Update or M2M changed"),
-                                    ("CREATE_DELETE_OR_M2M", "Create, Delete or M2M changed"),
-                                    ("UPDATE_DELETE_OR_M2M", "Update, Delete or M2M changed"),
-                                    ("CREATE_UPDATE_DELETE_OR_M2M", "Create, Update or Delete, or M2M changed"),
-                                ],
-                                help_text="Signal the webhook fires to.",
-                                max_length=255,
-                                verbose_name="signal",
-                            ),
-                        ),
-                        (
-                            "ref",
-                            models.CharField(
-                                db_index=True,
-                                help_text="Dot import notation to the model the webhook is for.",
-                                max_length=1023,
-                                validators=[signal_webhooks.utils.model_from_reference],
-                                verbose_name="referenced model",
-                            ),
-                        ),
-                        (
-                            "endpoint",
-                            models.URLField(help_text="Target endpoint for this webhook.", max_length=2047, verbose_name="endpoint"),
-                        ),
-                        (
-                            "headers",
-                            models.JSONField(
-                                blank=True,
-                                default=dict,
-                                help_text="Headers to send with the webhook request.",
-                                validators=[signal_webhooks.utils.is_dict],
-                                verbose_name="headers",
-                            ),
-                        ),
-                        (
-                            "auth_token",
-                            signal_webhooks.fields.TokenField(
-                                blank=True,
-                                default="",
-                                help_text="Authentication token to use in an Authorization header.",
-                                max_length=8000,
-                                validators=[signal_webhooks.utils.decode_cipher_key],
-                                verbose_name="authentication token",
-                            ),
-                        ),
-                        ("enabled", models.BooleanField(default=True, help_text="Is this webhook enabled?", verbose_name="enabled")),
-                        (
-                            "keep_last_response",
-                            models.BooleanField(
-                                default=False,
-                                help_text="Should the webhook keep a log of the latest response it got?",
-                                verbose_name="keep last response",
-                            ),
-                        ),
-                        (
-                            "created",
-                            models.DateTimeField(auto_now_add=True, help_text="When the webhook was created.", verbose_name="created"),
-                        ),
-                        (
-                            "updated",
-                            models.DateTimeField(auto_now=True, help_text="When the webhook was last updated.", verbose_name="updated"),
-                        ),
-                        (
-                            "last_response",
-                            models.CharField(
-                                blank=True,
-                                default="",
-                                help_text="Latest response to this webhook.",
-                                max_length=8000,
-                                verbose_name="last response",
-                            ),
-                        ),
-                        (
-                            "last_success",
-                            models.DateTimeField(
-                                default=None,
-                                help_text="When the webhook last succeeded.",
-                                null=True,
-                                verbose_name="last success",
-                            ),
-                        ),
-                        (
-                            "last_failure",
-                            models.DateTimeField(
-                                default=None,
-                                help_text="When the webhook last failed.",
-                                null=True,
-                                verbose_name="last failure",
-                            ),
-                        ),
-                        (
-                            "created_by",
-                            models.ForeignKey(
-                                default=get_or_create_system_user_pk,
-                                on_delete=django.db.models.deletion.CASCADE,
-                                to=settings.AUTH_USER_MODEL,
-                            ),
-                        ),
-                    ],
-                    options={
-                        "verbose_name": "API Outbound Webhook",
-                        "app_label": "api",
-                    },
-                ),
-                migrations.AddConstraint(
-                    model_name="outboundwebhook",
-                    constraint=models.UniqueConstraint(fields=["ref", "endpoint"], name="prevent_duplicate_hooks_api_outboundwebhook"),
-                ),
-            ],
-        ),
-        # On non-sqlite backends the raw DDL above is skipped, so the real
-        # schema diverges from Django state. Rebuild both tables (including the
-        # unique constraint) from the post-migration state (no-op on sqlite).
-        migrations.RunPython(_pg_sync_schema, reverse_code=_pg_drop_schema),
-    ]
@@ -1,54 +0,0 @@
-__package__ = "archivebox.api"
-
-import secrets
-from archivebox.uuid_compat import CompactUUIDField, uuid7
-
-from django.conf import settings
-from django.db import models
-from django.utils import timezone
-from django_stubs_ext.db.models import TypedModelMeta
-from signal_webhooks.models import WebhookBase
-
-from archivebox.base_models.models import get_or_create_system_user_pk
-
-
-def generate_secret_token() -> str:
-    return secrets.token_hex(16)
-
-
-class APIToken(models.Model):
-    id = CompactUUIDField(primary_key=True, default=uuid7, editable=False, unique=True)
-    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=get_or_create_system_user_pk, null=False)
-    created_at = models.DateTimeField(default=timezone.now, db_index=True)
-    modified_at = models.DateTimeField(auto_now=True)
-    token = models.CharField(max_length=32, default=generate_secret_token, unique=True)
-    expires = models.DateTimeField(null=True, blank=True)
-
-    class Meta(TypedModelMeta):
-        app_label = "api"
-        verbose_name = "API Key"
-        verbose_name_plural = "API Keys"
-
-    def __str__(self) -> str:
-        return self.token
-
-    @property
-    def token_redacted(self):
-        return f"************{self.token[-4:]}"
-
-    def is_valid(self, for_date=None):
-        return not self.expires or self.expires >= (for_date or timezone.now())
-
-
-class OutboundWebhook(WebhookBase):
-    id = CompactUUIDField(primary_key=True, default=uuid7, editable=False, unique=True)
-    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=get_or_create_system_user_pk, null=False)
-    created_at = models.DateTimeField(default=timezone.now, db_index=True)
-    modified_at = models.DateTimeField(auto_now=True)
-
-    class Meta(WebhookBase.Meta):
-        app_label = "api"
-        verbose_name = "API Outbound Webhook"
-
-    def __str__(self) -> str:
-        return f"[{self.id}] {self.ref} -> {self.endpoint}"
@@ -1,17 +0,0 @@
-"""Fields shared by responses for user-owned model objects."""
-
-from ninja import Schema
-
-
-class OwnedObjectSchema(Schema):
-    created_by_id: str
-    created_by_username: str
-
-    @staticmethod
-    def resolve_created_by_id(obj):
-        return str(obj.created_by.pk)
-
-    @staticmethod
-    def resolve_created_by_username(obj) -> str:
-        user = obj.created_by
-        return user.username if isinstance(user.username, str) else str(user)
@@ -1,26 +0,0 @@
-__package__ = "archivebox.api"
-
-from urllib.parse import quote
-
-from django.http import HttpRequest, HttpResponseRedirect
-from django.shortcuts import redirect
-from django.urls import path
-from django.views.generic.base import RedirectView
-
-from archivebox.core.routes_util import build_web_url
-from .v1_api import urls as v1_api_urls
-
-
-def archive_redirect_view(request: HttpRequest, url: str) -> HttpResponseRedirect:
-    if request.META.get("QUERY_STRING"):
-        url = f"{url}?{request.META['QUERY_STRING']}"
-    return redirect(build_web_url(f"/web/{quote(url, safe=':/')}", request=request))
-
-
-urlpatterns = [
-    path("", RedirectView.as_view(url="/api/v1/docs")),
-    path("archive/<path:url>", archive_redirect_view, name="api-archive-redirect"),
-    path("v1/", RedirectView.as_view(url="/api/v1/docs")),
-    path("v1/", v1_api_urls),
-    path("v1", RedirectView.as_view(url="/api/v1/docs")),
-]
@@ -1,114 +0,0 @@
-__package__ = "archivebox.api"
-
-
-from io import StringIO
-from traceback import format_exception
-from contextlib import redirect_stdout, redirect_stderr
-
-from django.http import HttpRequest, HttpResponse
-from django.core.exceptions import ObjectDoesNotExist, EmptyResultSet, PermissionDenied
-from django.contrib.auth.models import User
-
-from ninja import NinjaAPI, Swagger
-
-# TODO: explore adding https://eadwincode.github.io/django-ninja-extra/
-
-from archivebox.config import VERSION
-from archivebox.config.version import get_COMMIT_HASH
-
-from archivebox.api.auth import API_AUTH_METHODS
-from archivebox.api.models import APIToken
-
-
-COMMIT_HASH = get_COMMIT_HASH() or "unknown"
-
-html_description = f"""
-<h3>Welcome to your ArchiveBox server's REST API <code>[v1 ALPHA]</code> homepage!</h3>
-<br/>
-<i><b>WARNING: This API is still in an early development stage and may change!</b></i>
-<br/>
-<ul>
-<li>⬅️ Manage your server: <a href="/admin/api/"><b>Setup API Keys</b></a>, <a href="/admin/">Go to your Server Admin UI</a>, <a href="/">Go to your Snapshots list</a> 
-<li>💬 Ask questions and get help here: <a href="https://zulip.archivebox.io">ArchiveBox Chat Forum</a></li>
-<li>🐞 Report API bugs here: <a href="https://github.com/ArchiveBox/ArchiveBox/issues">Github Issues</a></li>
-<li>📚 ArchiveBox Documentation: <a href="https://github.com/ArchiveBox/ArchiveBox/wiki">Github Wiki</a></li>
-<li>📜 See the API source code: <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/archivebox/api"><code>archivebox/api/</code></a></li>
-</ul>
-<small>Served by ArchiveBox v{VERSION} (<a href="https://github.com/ArchiveBox/ArchiveBox/commit/{COMMIT_HASH}"><code>{COMMIT_HASH[:8]}</code></a>), API powered by <a href="https://django-ninja.dev/"><code>django-ninja</code></a>.</small>
-"""
-
-
-def register_urls(api: NinjaAPI) -> NinjaAPI:
-    api.add_router("/auth/", "archivebox.api.v1_auth.router")
-    api.add_router("/core/", "archivebox.api.v1_core.router")
-    api.add_router("/crawls/", "archivebox.api.v1_crawls.router")
-    api.add_router("/cli/", "archivebox.api.v1_cli.router")
-    api.add_router("/machine/", "archivebox.api.v1_machine.router")
-    api.add_router("/personas/", "archivebox.api.v1_personas.router")
-    return api
-
-
-class NinjaAPIWithIOCapture(NinjaAPI):
-    def create_temporal_response(self, request: HttpRequest) -> HttpResponse:
-        stdout, stderr = StringIO(), StringIO()
-
-        with redirect_stderr(stderr):
-            with redirect_stdout(stdout):
-                setattr(request, "stdout", stdout)
-                setattr(request, "stderr", stderr)
-
-                response = super().create_temporal_response(request)
-
-        # Disable caching of API responses entirely
-        response["Cache-Control"] = "no-store"
-
-        # Add debug stdout and stderr headers to response
-        response["X-ArchiveBox-Stdout"] = stdout.getvalue().replace("\n", "\\n")[:200]
-        response["X-ArchiveBox-Stderr"] = stderr.getvalue().replace("\n", "\\n")[:200]
-        # response['X-ArchiveBox-View'] = self.get_openapi_operation_id(request) or 'Unknown'
-
-        # Add Auth Headers to response
-        api_token_attr = request.__dict__.get("_api_token")
-        api_token = api_token_attr if isinstance(api_token_attr, APIToken) else None
-        token_expiry = api_token.expires.isoformat() if api_token and api_token.expires else "Never"
-
-        response["X-ArchiveBox-Auth-Method"] = str(request.__dict__.get("_api_auth_method", "None"))
-        response["X-ArchiveBox-Auth-Expires"] = token_expiry
-        response["X-ArchiveBox-Auth-Token-Id"] = str(api_token.id) if api_token else "None"
-        response["X-ArchiveBox-Auth-User-Id"] = str(request.user.pk) if request.user.pk else "None"
-        response["X-ArchiveBox-Auth-User-Username"] = request.user.username if isinstance(request.user, User) else "None"
-
-        return response
-
-
-api = NinjaAPIWithIOCapture(
-    title="ArchiveBox API",
-    description=html_description,
-    version=VERSION,
-    auth=API_AUTH_METHODS,
-    urls_namespace="api-1",
-    docs=Swagger(settings={"persistAuthorization": True}),
-)
-api = register_urls(api)
-urls = api.urls
-
-
-@api.exception_handler(Exception)
-def generic_exception_handler(request, err):
-    status = 503
-    if isinstance(err, (ObjectDoesNotExist, EmptyResultSet, PermissionDenied)):
-        status = 404
-
-    print("".join(format_exception(err)))
-
-    return api.create_response(
-        request,
-        {
-            "succeeded": False,
-            "message": f"{err.__class__.__name__}: {err}",
-            "errors": [
-                "".join(format_exception(err)),
-            ],
-        },
-        status=status,
-    )
@@ -1,65 +0,0 @@
-__package__ = "archivebox.api"
-
-from django.http import HttpRequest
-
-from ninja import Router, Schema
-
-from archivebox.api.auth import auth_using_token, auth_using_password, get_or_create_api_token
-
-
-router = Router(tags=["Authentication"], auth=None)
-
-
-class PasswordAuthSchema(Schema):
-    """Schema for a /get_api_token request"""
-
-    username: str | None = None
-    password: str | None = None
-
-
-@router.post(
-    "/get_api_token",
-    auth=None,
-    summary="Generate an API token for a given username & password (or currently logged-in user)",
-)  # auth=None because they are not authed yet
-def get_api_token(request: HttpRequest, auth_data: PasswordAuthSchema):
-    user = auth_using_password(
-        username=auth_data.username,
-        password=auth_data.password,
-        request=request,
-    )
-
-    if user and user.is_superuser:
-        api_token = get_or_create_api_token(user)
-        assert api_token is not None, "Failed to create API token"
-        return {
-            "success": True,
-            "user_id": str(user.pk),
-            "username": user.username,
-            "token": api_token.token,
-            "expires": api_token.expires.isoformat() if api_token.expires else None,
-        }
-
-    return {"success": False, "errors": ["Invalid credentials"]}
-
-
-class TokenAuthSchema(Schema):
-    """Schema for a /check_api_token request"""
-
-    token: str
-
-
-@router.post(
-    "/check_api_token",
-    auth=None,
-    summary="Validate an API token to make sure its valid and non-expired",
-)  # auth=None because they are not authed yet
-def check_api_token(request: HttpRequest, token_data: TokenAuthSchema):
-    user = auth_using_token(
-        token=token_data.token,
-        request=request,
-    )
-    if user:
-        return {"success": True, "user_id": str(user.pk)}
-
-    return {"success": False, "user_id": None}
@@ -1,273 +0,0 @@
-__package__ = "archivebox.api"
-
-import json
-from io import StringIO
-from typing import Any
-from enum import Enum
-
-from django.http import HttpRequest
-
-from ninja import Router, Schema
-from ninja.errors import HttpError
-from pydantic import Field
-
-from archivebox.misc.util import ansi_to_html
-from archivebox.core.models import SnapshotQuerySet
-
-
-# router for API that exposes archivebox cli subcommands as REST endpoints
-router = Router(tags=["ArchiveBox CLI Sub-Commands"])
-
-
-# Schemas
-
-JSONType = list[Any] | dict[str, Any] | bool | int | str | None
-FILTER_PATTERNS_EXAMPLES = [["https://example.com"]]
-
-
-class CLICommandResponseSchema(Schema):
-    success: bool
-    errors: list[str]
-    result: JSONType
-    result_format: str = "str"
-    stdout: str
-    stderr: str
-
-
-FilterTypeChoices = Enum(
-    "FilterTypeChoices",
-    {filter_type: filter_type for filter_type in SnapshotQuerySet.FILTER_TYPE_CHOICES},
-    type=str,
-)
-
-
-class AddCommandSchema(Schema):
-    urls: list[str]
-    snapshot_ids: list[str] | None = None
-    tag: str = ""
-    depth: int = 0
-    max_urls: int = 0
-    crawl_max_size: int = 0
-    crawl_timeout: int = 0
-    snapshot_max_size: int = 0
-    parser: str = "auto"
-    plugins: str = ""
-    persona: str = "Default"
-    only_new: bool | None = None
-    update: bool = False
-    overwrite: bool = False
-    index_only: bool = False
-
-
-class SnapshotFilterCommandSchema(Schema):
-    after: float | None = 0
-    before: float | None = None
-    filter_type: str | None = FilterTypeChoices.substring
-    filter_patterns: list[str] | None = Field(default=None, examples=FILTER_PATTERNS_EXAMPLES)
-    status: str | None = None
-    url__icontains: str | None = None
-    url__istartswith: str | None = None
-    tag: str | None = None
-    crawl_id: str | None = None
-    limit: int | None = None
-    sort: str | None = None
-    search: str | None = None
-
-
-class UpdateCommandSchema(SnapshotFilterCommandSchema):
-    resume: str | None = None
-    batch_size: int = 100
-    continuous: bool = False
-    index_only: bool = False
-    migrate_only: bool = False
-
-
-class ScheduleCommandSchema(Schema):
-    import_path: str | None = None
-    add: bool = False
-    show: bool = False
-    foreground: bool = False
-    run_all: bool = False
-    quiet: bool = False
-    every: str | None = None
-    tag: str = ""
-    depth: int = 0
-    only_new: bool | None = None
-    update: bool = False
-    overwrite: bool = False
-    clear: bool = False
-
-
-class ListCommandSchema(SnapshotFilterCommandSchema):
-    as_json: bool = True
-    as_html: bool = False
-    as_csv: str | None = "timestamp,url"
-    with_headers: bool = False
-
-
-class RemoveCommandSchema(SnapshotFilterCommandSchema):
-    filter_type: str = FilterTypeChoices.exact
-    timeout: float = 60.0
-
-
-def snapshot_filter_kwargs(args: SnapshotFilterCommandSchema, *, default_filter_type: str) -> dict[str, Any]:
-    kwargs = args.dict()
-    kwargs["filter_patterns"] = kwargs.get("filter_patterns") or []
-    kwargs["filter_type"] = kwargs.get("filter_type") or default_filter_type
-    return kwargs
-
-
-def cli_response(request, result, *, result_format="json", success=True, errors=()):
-    """Use one response envelope for CLI-backed endpoints, including captured output."""
-    payload = {"success": success, "errors": list(errors), "result": result, "result_format": result_format}
-    for stream_name in ("stdout", "stderr"):
-        stream = request.__dict__.get(stream_name)
-        payload[stream_name] = ansi_to_html(stream.getvalue().strip()) if isinstance(stream, StringIO) else ""
-    return payload
-
-
-def command_config(args) -> dict[str, object] | None:
-    """Translate shared legacy add/schedule flags into runtime config overrides."""
-    if args.update or args.overwrite:
-        return {"ONLY_NEW": False}
-    return {"ONLY_NEW": bool(args.only_new)} if args.only_new is not None else None
-
-
-@router.post("/add", response=CLICommandResponseSchema, summary="archivebox add [args] [urls]")
-def cli_add(request: HttpRequest, args: AddCommandSchema):
-    from archivebox.cli.archivebox_add import add
-    from archivebox.misc.util import validate_url
-
-    submitted_urls: str | list[str] = args.urls
-    if len(args.urls) == 1:
-        try:
-            validate_url(args.urls[0])
-        except ValueError:
-            submitted_urls = args.urls[0]
-    else:
-        submitted_urls = []
-        for url in args.urls:
-            try:
-                submitted_urls.append(validate_url(url))
-            except ValueError:
-                continue
-        if not submitted_urls:
-            raise HttpError(400, "No valid URLs were submitted")
-    crawl, snapshots = add(
-        urls=submitted_urls,
-        **args.model_dump(exclude={"urls", "only_new", "update", "overwrite"}),
-        bg=True,  # Always run in background for API calls
-        created_by_id=request.user.pk,
-        config=command_config(args),
-    )
-
-    snapshot_ids = [str(snapshot_id) for snapshot_id in snapshots.values_list("id", flat=True)]
-    result_payload = {
-        "crawl_id": str(crawl.id),
-        "num_snapshots": len(snapshot_ids),
-        "snapshot_ids": snapshot_ids,
-        "queued_urls": args.urls if isinstance(submitted_urls, str) else submitted_urls,
-    }
-    return cli_response(request, result_payload, result_format="json")
-
-
-@router.post("/update", response=CLICommandResponseSchema, summary="archivebox update [args] [filter_patterns]")
-def cli_update(request: HttpRequest, args: UpdateCommandSchema):
-    from archivebox.cli.archivebox_update import _build_filtered_snapshots_queryset, update
-    from archivebox.core.snapshot_status import normalize_snapshot_status
-
-    try:
-        status = normalize_snapshot_status(args.status)
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-
-    update_kwargs = snapshot_filter_kwargs(args, default_filter_type=FilterTypeChoices.substring)
-    update_kwargs["status"] = status
-    update_kwargs["stop_daemon_stack"] = False
-
-    is_filtered_update = any(
-        (update_kwargs.get(key) for key in (*SnapshotQuerySet.FILTER_ARG_KEYS, "resume") if key != "filter_type"),
-    )
-    matched_snapshot_ids = []
-    if is_filtered_update:
-        matched_snapshot_ids = [
-            str(snapshot_id) for snapshot_id in _build_filtered_snapshots_queryset(**update_kwargs).values_list("id", flat=True)
-        ]
-
-    update(**update_kwargs)
-    return cli_response(
-        request,
-        {"matched_count": len(matched_snapshot_ids), "snapshot_ids": matched_snapshot_ids} if is_filtered_update else None,
-        result_format="str",
-    )
-
-
-@router.post("/schedule", response=CLICommandResponseSchema, summary="archivebox schedule [args] [import_path]")
-def cli_schedule(request: HttpRequest, args: ScheduleCommandSchema):
-    from archivebox.cli.archivebox_schedule import schedule
-
-    result = schedule(**args.model_dump(exclude={"only_new", "update", "overwrite"}), config=command_config(args))
-
-    return cli_response(request, result, result_format="json")
-
-
-@router.post("/search", response=CLICommandResponseSchema, summary="archivebox search [args] [filter_patterns]")
-def cli_search(request: HttpRequest, args: ListCommandSchema):
-    from archivebox.cli.archivebox_snapshot import build_snapshot_queryset
-
-    search_kwargs = snapshot_filter_kwargs(args, default_filter_type=FilterTypeChoices.substring)
-    as_json = search_kwargs.pop("as_json")
-    as_html = search_kwargs.pop("as_html")
-    as_csv = search_kwargs.pop("as_csv")
-    with_headers = search_kwargs.pop("with_headers")
-    try:
-        snapshots = build_snapshot_queryset(**search_kwargs).select_related("crawl", "crawl__created_by")
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-
-    result_format = "txt"
-    if as_json:
-        result_format = "json"
-        result = [
-            json.loads(json.dumps(snapshot.to_dict(extended=True), default=str))
-            for snapshot in snapshots.prefetch_related("tags").iterator(chunk_size=500)
-        ]
-    elif as_html:
-        result_format = "html"
-        result = "\n".join(snapshot.url for snapshot in snapshots.iterator(chunk_size=500))
-    elif as_csv:
-        result_format = "csv"
-        cols = [col.strip() for col in as_csv.split(",") if col.strip()]
-        rows = [snapshot.to_csv(cols=cols, separator=",") for snapshot in snapshots.prefetch_related("tags").iterator(chunk_size=500)]
-        result = "\n".join((",".join(cols), *rows) if with_headers else rows)
-    else:
-        result = "\n".join(snapshot.url for snapshot in snapshots.iterator(chunk_size=500))
-
-    return cli_response(request, result, result_format=result_format)
-
-
-@router.post("/remove", response=CLICommandResponseSchema, summary="archivebox remove [args] [filter_patterns]")
-def cli_remove(request: HttpRequest, args: RemoveCommandSchema):
-    from archivebox.cli.archivebox_remove import remove
-    from archivebox.core.models import Snapshot
-
-    remove_kwargs = snapshot_filter_kwargs(args, default_filter_type=FilterTypeChoices.exact)
-    timeout_arg = remove_kwargs.pop("timeout")
-    timeout = min(float(timeout_arg if timeout_arg is not None else 60.0), 60.0)
-    try:
-        snapshots_to_remove = Snapshot.objects.order_by("-created_at").search(**remove_kwargs)
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-
-    result = remove(
-        yes=True,  # no way to interactively ask for confirmation via API, so we force yes
-        snapshots=snapshots_to_remove,
-        timeout=timeout,
-    )
-    return cli_response(
-        request,
-        result,
-        result_format="json",
-        success=bool(result["success"]),
-        errors=[str(result["error"])] if result["error"] else [],
-    )
@@ -1,47 +0,0 @@
-"""Core REST routes, composed from entity-specific routers."""
-
-from typing import Union
-
-from django.db.models import Model
-from django.http import HttpRequest
-from django.shortcuts import redirect
-from ninja import Router
-from ninja.errors import HttpError
-
-from archivebox.api.v1_crawls import CrawlSchema, get_crawl
-
-from . import archiveresults, snapshots, tags
-from .archiveresults import get_archiveresult
-from .pagination import CustomPagination as CustomPagination
-from .schemas import ArchiveResultSchema, SnapshotSchema, TagSchema
-from .snapshots import get_snapshot
-from .tags import get_tag
-
-router = Router(tags=["Core Models"])
-router.add_router("", archiveresults.router)
-router.add_router("", snapshots.router)
-router.add_router("", tags.router)
-
-
-@router.get(
-    "/any/{id}",
-    response=Union[SnapshotSchema, ArchiveResultSchema, TagSchema, CrawlSchema],
-    url_name="get_any",
-    summary="Get any object by its ID",
-)
-def get_any(request: HttpRequest, id: str):
-    """Get any object by its ID (e.g. snapshot, archiveresult, tag, crawl, etc.)."""
-    setattr(request, "with_snapshots", False)
-    setattr(request, "with_archiveresults", False)
-
-    for getter in (get_snapshot, get_archiveresult, get_tag, get_crawl):
-        try:
-            response = getter(request, id)
-            if isinstance(response, Model):
-                return redirect(
-                    f"/api/v1/{response._meta.app_label}/{response._meta.model_name}/{response.pk}?{request.META['QUERY_STRING']}",
-                )
-        except Exception:
-            pass
-
-    raise HttpError(404, "Object with given ID not found")
@@ -1,310 +0,0 @@
-import json
-import mimetypes
-import re
-from pathlib import Path, PurePosixPath
-from typing import Any
-
-from django.core.files.storage import FileSystemStorage
-from django.http import HttpRequest
-from django.http.multipartparser import MultiPartParser, MultiPartParserError
-from ninja import Form, Query, Router, UploadedFile
-from ninja.errors import HttpError
-from ninja.pagination import paginate
-
-from archivebox.core.models import ArchiveResult, Snapshot
-from archivebox.misc.db import uuid_ref_query as _uuid_ref_query
-
-from .lookup import _get_snapshot_by_ref
-from .pagination import CustomPagination
-from .schemas import ArchiveResultFilterSchema, ArchiveResultSchema
-
-router = Router(tags=["Core Models"])
-
-ARCHIVERESULT_UPLOAD_HOOK_NAME = Snapshot.BROWSER_EXTENSION_UPLOAD_HOOK_NAME
-ARCHIVERESULT_UPLOAD_PLUGIN_RE = re.compile(r"^[A-Za-z0-9_.-]{1,32}$")
-
-
-@router.get("/archiveresults", response=list[ArchiveResultSchema], url_name="get_archiveresult")
-@paginate(CustomPagination)
-def get_archiveresults(request: HttpRequest, filters: Query[ArchiveResultFilterSchema]):
-    """List all ArchiveResult entries matching these filters."""
-    queryset = filters.filter(ArchiveResult.objects.all())
-    if filters.search or filters.snapshot_tag:
-        return queryset.distinct()
-    return queryset
-
-
-@router.get("/archiveresult/{archiveresult_id}", response=ArchiveResultSchema, url_name="get_archiveresult")
-def get_archiveresult(request: HttpRequest, archiveresult_id: str):
-    """Get a specific ArchiveResult by id."""
-    return ArchiveResult.objects.get(_uuid_ref_query("id", archiveresult_id))
-
-
-def _normalize_uploaded_archiveresult_plugin(plugin: str) -> str:
-    normalized = str(plugin or "").strip().strip("/")
-    if not ARCHIVERESULT_UPLOAD_PLUGIN_RE.fullmatch(normalized):
-        raise HttpError(400, "Invalid ArchiveResult plugin name")
-    return normalized
-
-
-def _normalize_uploaded_archiveresult_output_path(output_path: str, *, filename: str) -> str:
-    raw_path = str(output_path or filename or "").strip().replace("\\", "/")
-    if not raw_path:
-        raise HttpError(400, "ArchiveResult output path is required")
-
-    path = PurePosixPath(raw_path)
-    if path.is_absolute() or any(part in ("", ".", "..") for part in path.parts):
-        raise HttpError(400, "Invalid ArchiveResult output path")
-
-    return str(path)
-
-
-def _parse_archiveresult_output_json(output_json: str | None) -> dict[str, Any] | None:
-    if not output_json:
-        return None
-    try:
-        parsed = json.loads(output_json)
-    except json.JSONDecodeError as err:
-        raise HttpError(400, "ArchiveResult output_json must be valid JSON") from err
-    if parsed is None:
-        return None
-    if not isinstance(parsed, dict):
-        raise HttpError(400, "ArchiveResult output_json must be a JSON object")
-    return parsed
-
-
-def _get_archiveresult_upload_data(request: HttpRequest):
-    cached = request.__dict__.get("_archiveresult_upload_data")
-    if cached is not None:
-        return cached
-
-    if request.method.upper() == "PATCH" and request.content_type.startswith("multipart/"):
-        try:
-            data = MultiPartParser(request.META, request, request.upload_handlers, request.encoding).parse()
-        except MultiPartParserError as err:
-            raise HttpError(400, f"Invalid ArchiveResult multipart upload: {err}") from err
-    else:
-        data = (request.POST, request.FILES)
-
-    setattr(request, "_archiveresult_upload_data", data)
-    return data
-
-
-def _get_archiveresult_upload_files(request: HttpRequest, *, allow_empty: bool = False) -> list[UploadedFile]:
-    _post, request_files = _get_archiveresult_upload_data(request)
-    files = [*request_files.getlist("files"), *request_files.getlist("file")]
-    if not files and not allow_empty:
-        raise HttpError(400, "At least one ArchiveResult file is required")
-    return files
-
-
-def _get_archiveresult_upload_form_values(request: HttpRequest, *field_names: str) -> list[str]:
-    request_post, _files = _get_archiveresult_upload_data(request)
-    values: list[str] = []
-    for field_name in field_names:
-        values.extend(str(value) for value in request_post.getlist(field_name))
-    if len(values) == 1:
-        value = values[0].strip()
-        if value.startswith("["):
-            try:
-                parsed = json.loads(value)
-            except json.JSONDecodeError:
-                parsed = None
-            if isinstance(parsed, list):
-                return [str(item) for item in parsed]
-    return values
-
-
-def _get_archiveresult_upload_form_value(request: HttpRequest, *field_names: str) -> str:
-    request_post, _files = _get_archiveresult_upload_data(request)
-    for field_name in field_names:
-        value = request_post.get(field_name)
-        if value is not None:
-            return str(value)
-    return ""
-
-
-def _parse_archiveresult_upload_int(value: str, field_name: str, *, default: int | None = None) -> int:
-    if value == "" and default is not None:
-        return default
-    try:
-        parsed = int(value)
-    except (TypeError, ValueError) as err:
-        raise HttpError(400, f"ArchiveResult {field_name} must be an integer") from err
-    if parsed < 0:
-        raise HttpError(400, f"ArchiveResult {field_name} must be non-negative")
-    return parsed
-
-
-def _write_archiveresult_files(
-    request: HttpRequest,
-    snapshot: Snapshot,
-    plugin_name: str,
-    *,
-    existing_output_files: dict[str, dict[str, Any]] | None = None,
-    allow_empty: bool = False,
-) -> dict[str, dict[str, Any]]:
-    files = _get_archiveresult_upload_files(request, allow_empty=allow_empty)
-    output_paths = _get_archiveresult_upload_form_values(request, "output_paths", "output_path")
-    mime_types = _get_archiveresult_upload_form_values(request, "mime_types", "mime_type")
-    chunk_output_path = _get_archiveresult_upload_form_value(request, "chunk_output_path")
-
-    snapshot_dir = snapshot.output_dir
-    plugin_dir = snapshot_dir / plugin_name
-    storage = FileSystemStorage(location=str(plugin_dir))
-    output_files = dict(existing_output_files or {})
-
-    if not files:
-        return output_files
-
-    if chunk_output_path:
-        if len(files) != 1:
-            raise HttpError(400, "Exactly one ArchiveResult file chunk is required")
-
-        uploaded_file = files[0]
-        relative_output_path = _normalize_uploaded_archiveresult_output_path(
-            chunk_output_path,
-            filename=uploaded_file.name,
-        )
-        chunk_index = _parse_archiveresult_upload_int(
-            _get_archiveresult_upload_form_value(request, "chunk_index"),
-            "chunk_index",
-        )
-        chunk_count = _parse_archiveresult_upload_int(
-            _get_archiveresult_upload_form_value(request, "chunk_count"),
-            "chunk_count",
-        )
-        chunk_offset = _parse_archiveresult_upload_int(
-            _get_archiveresult_upload_form_value(request, "chunk_offset"),
-            "chunk_offset",
-        )
-        chunk_total_size = _parse_archiveresult_upload_int(
-            _get_archiveresult_upload_form_value(request, "chunk_total_size"),
-            "chunk_total_size",
-        )
-
-        if chunk_count < 1:
-            raise HttpError(400, "ArchiveResult chunk_count must be at least 1")
-        if chunk_index >= chunk_count:
-            raise HttpError(400, "ArchiveResult chunk_index must be less than chunk_count")
-        if chunk_total_size and chunk_offset > chunk_total_size:
-            raise HttpError(400, "ArchiveResult chunk_offset cannot exceed chunk_total_size")
-
-        if chunk_index == 0 and chunk_offset == 0 and storage.exists(relative_output_path):
-            storage.delete(relative_output_path)
-
-        current_size = storage.size(relative_output_path) if storage.exists(relative_output_path) else 0
-        if current_size != chunk_offset:
-            raise HttpError(
-                409,
-                f"ArchiveResult chunk offset mismatch for {relative_output_path}: expected {current_size}, got {chunk_offset}",
-            )
-
-        Path(storage.path(relative_output_path)).parent.mkdir(parents=True, exist_ok=True)
-        with storage.open(relative_output_path, "ab") as destination:
-            for chunk in uploaded_file.chunks():
-                destination.write(chunk)
-
-        size = storage.size(relative_output_path)
-        upload_complete = chunk_index + 1 == chunk_count
-        if upload_complete and size != chunk_total_size:
-            raise HttpError(
-                409,
-                f"ArchiveResult chunk size mismatch for {relative_output_path}: expected {chunk_total_size}, got {size}",
-            )
-
-        guessed_mime = mimetypes.guess_type(relative_output_path)[0]
-        output_mime_type = (mime_types[0] if mime_types else "") or uploaded_file.content_type or guessed_mime or "application/octet-stream"
-        output_files[relative_output_path] = {
-            "extension": PurePosixPath(relative_output_path).suffix.lower().lstrip("."),
-            "mimetype": output_mime_type,
-            "size": size,
-            "upload": {
-                "chunked": True,
-                "chunk_index": chunk_index,
-                "chunk_count": chunk_count,
-                "chunks_received": chunk_index + 1,
-                "complete": upload_complete,
-            },
-        }
-
-        return output_files
-
-    for index, uploaded_file in enumerate(files):
-        relative_output_path = _normalize_uploaded_archiveresult_output_path(
-            output_paths[index] if index < len(output_paths) else "",
-            filename=uploaded_file.name,
-        )
-        if storage.exists(relative_output_path):
-            storage.delete(relative_output_path)
-        saved_output_path = storage.save(relative_output_path, uploaded_file)
-        size = storage.size(saved_output_path)
-        guessed_mime = mimetypes.guess_type(saved_output_path)[0]
-        output_mime_type = (
-            (mime_types[index] if index < len(mime_types) else "")
-            or uploaded_file.content_type
-            or guessed_mime
-            or "application/octet-stream"
-        )
-        output_files[saved_output_path] = {
-            "extension": PurePosixPath(saved_output_path).suffix.lower().lstrip("."),
-            "mimetype": output_mime_type,
-            "size": size,
-        }
-
-    return output_files
-
-
-@router.post(
-    "/archiveresults",
-    response=ArchiveResultSchema,
-    url_name="create_archiveresult",
-)
-def create_archiveresult(
-    request: HttpRequest,
-    snapshot_id: str = Form(...),
-    plugin: str = Form(...),
-    output_str: str = Form(""),
-    hook_name: str = Form(ARCHIVERESULT_UPLOAD_HOOK_NAME),
-    status: str = Form(str(ArchiveResult.StatusChoices.SUCCEEDED)),
-    output_json: str = Form(""),
-):
-    """Create or update an ArchiveResult with one or more output files."""
-    snapshot = _get_snapshot_by_ref(snapshot_id)
-    plugin_name = _normalize_uploaded_archiveresult_plugin(plugin)
-    result = ArchiveResult.objects.filter(
-        snapshot=snapshot,
-        plugin=plugin_name,
-        hook_name=hook_name or ARCHIVERESULT_UPLOAD_HOOK_NAME,
-    ).first()
-    result = result or ArchiveResult(snapshot=snapshot, plugin=plugin_name, hook_name=hook_name or ARCHIVERESULT_UPLOAD_HOOK_NAME)
-    metadata = {"output_str": output_str, "status": status, "output_json": _parse_archiveresult_output_json(output_json)}
-    files = _write_archiveresult_files(request, snapshot, plugin_name, allow_empty=True)
-    try:
-        return result.apply_upload(files, metadata=metadata, replace_metadata=True)
-    except ArchiveResult.UploadConflict as err:
-        raise HttpError(409, str(err)) from err
-
-
-@router.patch("/archiveresult/{archiveresult_id}", response=ArchiveResultSchema, url_name="patch_archiveresult")
-def patch_archiveresult(
-    request: HttpRequest,
-    archiveresult_id: str,
-):
-    """Append or replace files on an existing ArchiveResult."""
-    result = ArchiveResult.objects.select_related("snapshot__crawl__created_by").get(_uuid_ref_query("id", archiveresult_id))
-    uploaded_output_files = _write_archiveresult_files(
-        request,
-        result.snapshot,
-        result.plugin,
-    )
-    output_str = _get_archiveresult_upload_form_value(request, "output_str")
-    status = _get_archiveresult_upload_form_value(request, "status")
-    output_json = _get_archiveresult_upload_form_value(request, "output_json")
-    metadata = {key: value for key, value in {"output_str": output_str, "status": status}.items() if value}
-    if output_json:
-        metadata["output_json"] = _parse_archiveresult_output_json(output_json)
-    try:
-        return result.apply_upload(uploaded_output_files, metadata=metadata)
-    except ArchiveResult.UploadConflict as err:
-        raise HttpError(409, str(err)) from err
@@ -1,12 +0,0 @@
-from django.db.models import Q
-
-from archivebox.core.models import Snapshot
-from archivebox.misc.db import uuid_ref_query as _uuid_ref_query
-
-
-def _get_snapshot_by_ref(snapshot_id: str):
-    queryset = Snapshot.objects.select_related("crawl__created_by")
-    try:
-        return queryset.get(_uuid_ref_query("id", snapshot_id) | Q(timestamp__startswith=snapshot_id))
-    except Snapshot.DoesNotExist:
-        return queryset.get(_uuid_ref_query("id", snapshot_id))
@@ -1,40 +0,0 @@
-import math
-from typing import Any
-
-from django.http import HttpRequest
-from ninja.pagination import PaginationBase
-
-
-class CustomPagination(PaginationBase):
-    class Input(PaginationBase.Input):
-        limit: int = 200
-        offset: int = 0
-        page: int = 0
-
-    class Output(PaginationBase.Output):
-        count: int
-        total_items: int
-        total_pages: int
-        page: int
-        limit: int
-        offset: int
-        num_items: int
-        items: list[Any]
-
-    def paginate_queryset(self, queryset, pagination: Input, request: HttpRequest, **params):
-        limit = min(pagination.limit, 500)
-        offset = pagination.offset or (pagination.page * limit)
-        total = queryset.values("pk").distinct().count() if queryset.query.distinct else queryset.count()
-        total_pages = math.ceil(total / limit)
-        current_page = math.ceil(offset / (limit + 1))
-        items = queryset[offset : offset + limit]
-        return {
-            "count": total,
-            "total_items": total,
-            "total_pages": total_pages,
-            "page": current_page,
-            "limit": limit,
-            "offset": offset,
-            "num_items": len(items),
-            "items": items,
-        }
@@ -1,301 +0,0 @@
-from collections import defaultdict
-from datetime import datetime
-from typing import Annotated, Any
-from uuid import UUID
-
-from django.db.models import Q
-from archivebox.api.schemas import OwnedObjectSchema
-
-from ninja import FilterLookup, FilterSchema, Schema
-
-from archivebox.core.models import ArchiveResult, Snapshot
-
-
-class MinimalArchiveResultSchema(OwnedObjectSchema):
-    TYPE: str = "core.models.ArchiveResult"
-    id: UUID
-    created_at: datetime | None
-    modified_at: datetime | None
-    status: str
-    retry_at: datetime | None = None
-    plugin: str
-    hook_name: str
-    process_id: UUID | None
-    cmd_version: str | None
-    cmd: list[str] | None
-    pwd: str | None
-    output_str: str
-    output_json: dict[str, Any] | None
-    output_files: dict[str, dict[str, Any]] | None
-    output_size: int
-    output_mimetypes: str
-    start_ts: datetime | None
-    end_ts: datetime | None
-
-    @staticmethod
-    def resolve_output_files(obj):
-        return obj.output_file_map()
-
-    @staticmethod
-    def resolve_output_mimetypes(obj) -> str:
-        mime_sizes: dict[str, int] = defaultdict(int)
-        for metadata in obj.output_file_map().values():
-            if not isinstance(metadata, dict):
-                continue
-            mimetype = str(metadata.get("mimetype") or "").strip()
-            try:
-                size = max(int(metadata.get("size") or 0), 0)
-            except (TypeError, ValueError):
-                size = 0
-            if mimetype and size:
-                mime_sizes[mimetype] += size
-        if mime_sizes:
-            return ",".join(mime for mime, _size in sorted(mime_sizes.items(), key=lambda item: item[1], reverse=True))
-        return obj.output_mimetypes or ""
-
-
-class ArchiveResultSchema(MinimalArchiveResultSchema):
-    TYPE: str = "core.models.ArchiveResult"
-    snapshot_id: UUID
-    snapshot_timestamp: str
-    snapshot_url: str
-    snapshot_tags: list[str]
-
-    @staticmethod
-    def resolve_snapshot_timestamp(obj):
-        return obj.snapshot.timestamp
-
-    @staticmethod
-    def resolve_snapshot_url(obj):
-        return obj.snapshot.url
-
-    @staticmethod
-    def resolve_snapshot_id(obj):
-        return obj.snapshot_id
-
-    @staticmethod
-    def resolve_snapshot_tags(obj):
-        return sorted(tag.name for tag in obj.snapshot.tags.all())
-
-
-class ArchiveResultFilterSchema(FilterSchema):
-    id: Annotated[str | None, FilterLookup(["id__startswith", "snapshot__id__startswith", "snapshot__timestamp__startswith"])] = None
-    search: Annotated[
-        str | None,
-        FilterLookup(
-            [
-                "snapshot__url__icontains",
-                "snapshot__title__icontains",
-                "snapshot__tags__name__icontains",
-                "plugin",
-                "output_str__icontains",
-                "id__startswith",
-                "snapshot__id__startswith",
-                "snapshot__timestamp__startswith",
-            ],
-        ),
-    ] = None
-    snapshot_id: Annotated[str | None, FilterLookup(["snapshot__id__startswith", "snapshot__timestamp__startswith"])] = None
-    snapshot_url: Annotated[str | None, FilterLookup("snapshot__url__icontains")] = None
-    snapshot_tag: Annotated[str | None, FilterLookup("snapshot__tags__name__icontains")] = None
-    status: Annotated[str | None, FilterLookup("status")] = None
-    output_str: Annotated[str | None, FilterLookup("output_str__icontains")] = None
-    plugin: Annotated[str | None, FilterLookup("plugin__icontains")] = None
-    hook_name: Annotated[str | None, FilterLookup("hook_name__icontains")] = None
-    process_id: Annotated[str | None, FilterLookup("process__id__startswith")] = None
-    cmd: Annotated[str | None, FilterLookup("cmd__0__icontains")] = None
-    pwd: Annotated[str | None, FilterLookup("pwd__icontains")] = None
-    cmd_version: Annotated[str | None, FilterLookup("cmd_version")] = None
-    created_at: Annotated[datetime | None, FilterLookup("created_at")] = None
-    created_at__gte: Annotated[datetime | None, FilterLookup("created_at__gte")] = None
-    created_at__lt: Annotated[datetime | None, FilterLookup("created_at__lt")] = None
-
-
-class SnapshotSchema(OwnedObjectSchema):
-    TYPE: str = "core.models.Snapshot"
-    id: UUID
-    created_at: datetime
-    modified_at: datetime
-    status: str
-    retry_at: datetime | None
-    bookmarked_at: datetime
-    downloaded_at: datetime | None
-    url: str
-    tags: list[str]
-    title: str | None
-    timestamp: str
-    archive_path: str
-    archive_size: int
-    output_size: int
-    num_archiveresults: int
-    archiveresults: list[MinimalArchiveResultSchema]
-
-    @staticmethod
-    def resolve_tags(obj):
-        return sorted(tag.name for tag in obj.tags.all())
-
-    @staticmethod
-    def resolve_archive_size(obj):
-        return int(obj.archive_size or 0)
-
-    @staticmethod
-    def resolve_output_size(obj):
-        return SnapshotSchema.resolve_archive_size(obj)
-
-    @staticmethod
-    def resolve_num_archiveresults(obj, context):
-        return obj.archiveresult_set.all().distinct().count()
-
-    @staticmethod
-    def resolve_archiveresults(obj, context):
-        if bool(context["request"].__dict__.get("with_archiveresults", False)):
-            return obj.archiveresult_set.all().distinct()
-        return ArchiveResult.objects.none()
-
-
-class SnapshotUpdateSchema(Schema):
-    action: str | None = None
-    status: str | None = None
-    retry_at: datetime | None = None
-    tags: list[str] | None = None
-
-
-class SnapshotCreateSchema(Schema):
-    url: str
-    crawl_id: str | None = None
-    depth: int = 0
-    title: str | None = None
-    tags: list[str] | None = None
-    status: str | None = None
-
-
-class SnapshotDeleteResponseSchema(Schema):
-    success: bool
-    snapshot_id: str
-    crawl_id: str
-    deleted_count: int
-
-
-class SnapshotFilterSchema(FilterSchema):
-    id: Annotated[str | None, FilterLookup(["id__istartswith", "id__iendswith", "timestamp__startswith"])] = None
-    created_by_id: Annotated[str | None, FilterLookup("crawl__created_by_id")] = None
-    created_by_username: Annotated[str | None, FilterLookup("crawl__created_by__username__icontains")] = None
-    created_at__gte: Annotated[datetime | None, FilterLookup("created_at__gte")] = None
-    created_at__lt: Annotated[datetime | None, FilterLookup("created_at__lt")] = None
-    created_at: Annotated[datetime | None, FilterLookup("created_at")] = None
-    modified_at: Annotated[datetime | None, FilterLookup("modified_at")] = None
-    modified_at__gte: Annotated[datetime | None, FilterLookup("modified_at__gte")] = None
-    modified_at__lt: Annotated[datetime | None, FilterLookup("modified_at__lt")] = None
-    search: str | None = None
-    search_mode: str | None = None
-    status: str | None = None
-    url: Annotated[str | None, FilterLookup("url")] = None
-    tag: Annotated[str | None, FilterLookup("tags__name")] = None
-    title: Annotated[str | None, FilterLookup("title__icontains")] = None
-    timestamp: Annotated[str | None, FilterLookup("timestamp__startswith")] = None
-    bookmarked_at__gte: Annotated[datetime | None, FilterLookup("bookmarked_at__gte")] = None
-    bookmarked_at__lt: Annotated[datetime | None, FilterLookup("bookmarked_at__lt")] = None
-
-    def filter_search(self, value: str | None) -> Q:
-        return Q()
-
-    def filter_search_mode(self, value: str | None) -> Q:
-        return Q()
-
-    def filter_status(self, value: str | None) -> Q:
-        return Q()
-
-
-class TagSchema(OwnedObjectSchema):
-    TYPE: str = "core.models.Tag"
-    id: int
-    modified_at: datetime
-    created_at: datetime
-    name: str
-    num_snapshots: int
-    snapshots: list[SnapshotSchema]
-
-    @staticmethod
-    def resolve_num_snapshots(obj, context):
-        return obj.snapshot_set.all().distinct().count()
-
-    @staticmethod
-    def resolve_snapshots(obj, context):
-        if bool(context["request"].__dict__.get("with_snapshots", False)):
-            return obj.snapshot_set.all().distinct()
-        return Snapshot.objects.none()
-
-
-class TagAutocompleteSchema(Schema):
-    tags: list[dict]
-
-
-class TagCreateSchema(Schema):
-    name: str
-
-
-class TagCreateResponseSchema(Schema):
-    success: bool
-    tag_id: int
-    tag_name: str
-    created: bool
-
-
-class TagSearchSnapshotSchema(Schema):
-    id: str
-    title: str
-    url: str
-    favicon_url: str
-    admin_url: str
-    archive_url: str
-    downloaded_at: str | None = None
-
-
-class TagSearchCardSchema(Schema):
-    id: int
-    name: str
-    slug: str
-    num_snapshots: int
-    filter_url: str
-    edit_url: str
-    export_urls_url: str
-    export_jsonl_url: str
-    rename_url: str
-    delete_url: str
-    snapshots: list[TagSearchSnapshotSchema]
-
-
-class TagSearchResponseSchema(Schema):
-    tags: list[TagSearchCardSchema]
-    sort: str
-    created_by: str
-    year: str
-    has_snapshots: str
-
-
-class TagUpdateSchema(Schema):
-    name: str
-
-
-class TagUpdateResponseSchema(Schema):
-    success: bool
-    tag_id: int
-    tag_name: str
-
-
-class TagDeleteResponseSchema(Schema):
-    success: bool
-    tag_id: int
-    deleted_count: int
-
-
-class TagSnapshotRequestSchema(Schema):
-    snapshot_id: str
-    tag_name: str | None = None
-    tag_id: int | None = None
-
-
-class TagSnapshotResponseSchema(Schema):
-    success: bool
-    tag_id: int
-    tag_name: str
@@ -1,356 +0,0 @@
-from datetime import datetime, time
-from html import unescape
-from typing import Any
-
-from django.contrib.auth import get_user_model
-from django.contrib.auth.models import User
-from django.core.exceptions import ValidationError
-from django.db import IntegrityError
-from django.db.models import Q
-from django.http import HttpRequest, HttpResponse
-from django.utils import timezone
-from django.utils.dateparse import parse_date, parse_datetime
-from django.utils.feedgenerator import Rss201rev2Feed
-from ninja import Query, Router
-from ninja.errors import HttpError
-from ninja.pagination import paginate
-
-from archivebox.api.v1_crawls import get_crawl_by_ref
-from archivebox.core.models import Snapshot
-from archivebox.core.routes_util import build_web_url
-from archivebox.core.snapshot_status import filter_snapshots_by_status, normalize_snapshot_status
-from archivebox.crawls.locks import crawl_lifecycle_lock
-from archivebox.crawls.models import Crawl
-from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url_length
-from archivebox.search.config import get_search_mode, get_search_mode_backend
-from archivebox.search.query import apply_snapshot_search
-
-from .lookup import _get_snapshot_by_ref
-from .pagination import CustomPagination
-from .schemas import SnapshotCreateSchema, SnapshotDeleteResponseSchema, SnapshotFilterSchema, SnapshotSchema, SnapshotUpdateSchema
-
-router = Router(tags=["Core Models"])
-
-
-def normalize_tag_list(tags: list[str] | None = None) -> list[str]:
-    return [tag.strip() for tag in (tags or []) if tag and tag.strip()]
-
-
-def _parse_rss_before(before: str | None) -> datetime:
-    if not before:
-        return timezone.now()
-
-    value = before.strip()
-    parsed_dt = None
-
-    if len(value) == 8 and value.isdigit():
-        parsed_date = datetime.strptime(value, "%Y%m%d").date()
-    else:
-        parsed_dt = parse_datetime(value)
-        parsed_date = None if parsed_dt else parse_date(value)
-
-    if parsed_dt is None:
-        if parsed_date is None:
-            raise HttpError(400, "before must be an ISO datetime, YYYY-MM-DD, or YYYYMMDD")
-        parsed_dt = datetime.combine(parsed_date, time.max)
-
-    if timezone.is_naive(parsed_dt):
-        parsed_dt = timezone.make_aware(parsed_dt, timezone.get_current_timezone())
-    return parsed_dt
-
-
-def _filter_snapshots_for_rss(
-    *,
-    crawl_id: str = "",
-    created_by: str = "",
-    before: str | None = None,
-    limit: int = 50,
-):
-    limit = max(1, min(int(limit or 50), 500))
-    before_dt = _parse_rss_before(before)
-    queryset = (
-        Snapshot.objects.select_related("crawl__created_by")
-        .prefetch_related("tags")
-        .only(
-            "id",
-            "url",
-            "title",
-            "timestamp",
-            "bookmarked_at",
-            "created_at",
-            "modified_at",
-            "fs_version",
-            "crawl_id",
-            "crawl__id",
-            "crawl__created_by_id",
-            "crawl__created_by__id",
-            "crawl__created_by__username",
-        )
-        .filter(bookmarked_at__lte=before_dt)
-    )
-    crawl_id = crawl_id.strip()
-    if crawl_id:
-        matching_crawl_pks = list(filter_queryset_by_uuid_substring(Crawl.objects.all(), crawl_id).values_list("pk", flat=True)[:100])
-        queryset = queryset.filter(crawl_id__in=matching_crawl_pks)
-
-    created_by = created_by.strip()
-    if created_by:
-        created_by_query = Q(crawl__created_by__username__iexact=created_by)
-        user_model = get_user_model()
-        try:
-            prepared_pk = user_model._meta.pk.get_prep_value(created_by)
-        except (TypeError, ValueError, ValidationError):
-            prepared_pk = None
-        if prepared_pk not in (None, ""):
-            created_by_query |= Q(crawl__created_by_id=prepared_pk)
-        queryset = queryset.filter(created_by_query)
-
-    return queryset.order_by("-bookmarked_at", "-created_at", "-id")[:limit]
-
-
-def _snapshots_rss_response(
-    request: HttpRequest,
-    *,
-    snapshots,
-    title: str = "ArchiveBox Snapshots",
-) -> HttpResponse:
-    web_base_url = build_web_url("/", request=request).rstrip("/")
-    feed_query = request.GET.copy()
-    for sensitive_param in ("api_key", "token", "password"):
-        feed_query.pop(sensitive_param, None)
-    feed_path = request.path
-    feed_url = request.build_absolute_uri(f"{feed_path}?{feed_query.urlencode()}" if feed_query else feed_path)
-
-    feed = Rss201rev2Feed(
-        title=title,
-        link=build_web_url("/public/", request=request),
-        description="Recently added ArchiveBox snapshots.",
-        language="en",
-        feed_url=feed_url,
-    )
-
-    for snapshot in snapshots:
-        archived_url = build_web_url(f"/{snapshot.archive_path_from_db}", request=request)
-        tags = [tag.name for tag in snapshot.tags.all()]
-        crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
-        description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
-        feed.add_item(
-            title=unescape(snapshot.title or snapshot.url),
-            link=archived_url or web_base_url,
-            description=description,
-            unique_id=str(snapshot.id),
-            unique_id_is_permalink=False,
-            pubdate=snapshot.bookmarked_at or snapshot.created_at,
-            updateddate=snapshot.modified_at,
-            author_name=crawl_user.username if crawl_user else None,
-            categories=tags,
-        )
-
-    return HttpResponse(feed.writeString("utf-8"), content_type="application/rss+xml; charset=utf-8")
-
-
-@router.get("/snapshots", response=list[SnapshotSchema], url_name="get_snapshots")
-@paginate(CustomPagination)
-def get_snapshots(request: HttpRequest, filters: Query[SnapshotFilterSchema], with_archiveresults: bool = False):
-    """List all Snapshot entries matching these filters."""
-    setattr(request, "with_archiveresults", with_archiveresults)
-    try:
-        queryset = filter_snapshots_by_status(Snapshot.objects.all(), filters.status)
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-    queryset = filters.filter(queryset).distinct()
-    query = (filters.search or "").strip()
-    if not query:
-        return queryset
-
-    runtime_config = request.archivebox_config
-    search_mode = get_search_mode(filters.search_mode, config=runtime_config)
-    try:
-        return apply_snapshot_search(
-            queryset,
-            query,
-            search_mode=search_mode,
-            config=runtime_config,
-            include_id_matches=True,
-        )
-    except Exception:
-        if get_search_mode_backend(search_mode, config=runtime_config):
-            return queryset.none()
-        return apply_snapshot_search(queryset, query, search_mode="meta", config=runtime_config, include_id_matches=True)
-
-
-@router.get("/snapshots.rss", url_name="get_snapshots_rss")
-def get_snapshots_rss(
-    request: HttpRequest,
-    crawl_id: str = "",
-    created_by: str = "",
-    limit: int = 50,
-    before: str | None = None,
-):
-    """Return matching snapshots as an RSS feed, newest first."""
-    snapshots = _filter_snapshots_for_rss(
-        crawl_id=crawl_id,
-        created_by=created_by,
-        limit=limit,
-        before=before,
-    )
-    return _snapshots_rss_response(request, snapshots=snapshots)
-
-
-@router.get("/snapshot/{snapshot_id}", response=SnapshotSchema, url_name="get_snapshot")
-def get_snapshot(request: HttpRequest, snapshot_id: str, with_archiveresults: bool = True):
-    """Get a specific Snapshot by id."""
-    setattr(request, "with_archiveresults", with_archiveresults)
-    return _get_snapshot_by_ref(snapshot_id)
-
-
-@router.post("/snapshots", response=SnapshotSchema, url_name="create_snapshot")
-def create_snapshot(request: HttpRequest, data: SnapshotCreateSchema):
-    tags = normalize_tag_list(data.tags)
-    try:
-        status = normalize_snapshot_status(data.status)
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-    if not data.url.strip():
-        raise HttpError(400, "URL is required")
-    try:
-        validate_url_length(data.url.strip())
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-    if data.depth not in (0, 1, 2, 3, 4):
-        raise HttpError(400, "depth must be between 0 and 4")
-
-    if data.crawl_id:
-        crawl = get_crawl_by_ref(data.crawl_id)
-        crawl_tags = normalize_tag_list(crawl.tags_str.split(","))
-        tags = tags or crawl_tags
-    else:
-        crawl = Crawl.objects.create(
-            urls=data.url,
-            max_depth=max(data.depth, 0),
-            tags_str=",".join(tags),
-            status=Crawl.StatusChoices.QUEUED,
-            retry_at=timezone.now(),
-            created_by=request.user if isinstance(request.user, User) else None,
-        )
-
-    # Browser uploads must not wait behind the runner's crawl-wide lifecycle
-    # lock. The unique insert recovery and CAS update below are the request-side
-    # coordination boundary for this idempotent metadata sync.
-    snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
-    if snapshot is None:
-        try:
-            snapshot = Snapshot.objects.create(
-                url=data.url,
-                crawl=crawl,
-                depth=data.depth,
-                title=data.title,
-                timestamp=str(timezone.now().timestamp()),
-                status=status or Snapshot.StatusChoices.QUEUED,
-                retry_at=timezone.now(),
-            )
-        except IntegrityError:
-            snapshot = Snapshot.objects.filter(url=data.url, crawl=crawl).first()
-            if snapshot is None:
-                raise
-
-    for _attempt in range(3):
-        updates: dict[str, Any] = {}
-        if data.title is not None and snapshot.title != data.title:
-            updates["title"] = data.title
-        if status is not None and snapshot.status != status:
-            updates["status"] = status
-        if not updates or snapshot.safe_update(updates, extra_filter={"modified_at": snapshot.modified_at}):
-            break
-    else:
-        raise HttpError(409, "Snapshot changed while metadata was being updated")
-
-    if tags:
-        snapshot.save_tags(
-            tags,
-            created_by=request.user if isinstance(request.user, User) else None,
-        )
-
-    try:
-        snapshot.ensure_crawl_symlink()
-    except Exception:
-        pass
-
-    setattr(request, "with_archiveresults", False)
-    return snapshot
-
-
-@router.patch("/snapshot/{snapshot_id}", response=SnapshotSchema, url_name="patch_snapshot")
-def patch_snapshot(request: HttpRequest, snapshot_id: str, data: SnapshotUpdateSchema):
-    """Update a snapshot (e.g., set status=sealed to cancel queued work)."""
-    snapshot = _get_snapshot_by_ref(snapshot_id)
-    crawl_id = str(snapshot.crawl_id)
-
-    payload = data.dict(exclude_unset=True)
-    update_fields = ["modified_at"]
-    action = payload.pop("action", None)
-    tags = payload.pop("tags", None)
-
-    if action:
-        if action == "pause":
-            snapshot.pause()
-            setattr(request, "with_archiveresults", False)
-            return snapshot
-        if action in ("resume", "unpause"):
-            snapshot.resume()
-            setattr(request, "with_archiveresults", False)
-            return snapshot
-        if action == "cancel":
-            snapshot.cancel()
-            setattr(request, "with_archiveresults", False)
-            return snapshot
-        raise HttpError(400, f"Invalid action: {action}")
-
-    with crawl_lifecycle_lock(crawl_id):
-        snapshot = _get_snapshot_by_ref(snapshot_id)
-        if "status" in payload:
-            try:
-                snapshot.status = normalize_snapshot_status(payload["status"])
-            except ValueError as err:
-                raise HttpError(400, str(err)) from err
-            if snapshot.status == Snapshot.StatusChoices.SEALED and "retry_at" not in payload:
-                snapshot.retry_at = None
-            update_fields.append("status")
-
-        if "retry_at" in payload:
-            snapshot.retry_at = payload["retry_at"]
-            update_fields.append("retry_at")
-
-        if tags is not None:
-            snapshot.save_tags(
-                normalize_tag_list(tags),
-                created_by=request.user if isinstance(request.user, User) else None,
-            )
-
-        if payload.get("status") == Snapshot.StatusChoices.SEALED:
-            snapshot.cancel()
-        else:
-            snapshot.save(update_fields=update_fields)
-    setattr(request, "with_archiveresults", False)
-    return snapshot
-
-
-@router.delete("/snapshot/{snapshot_id}", response=SnapshotDeleteResponseSchema, url_name="delete_snapshot")
-def delete_snapshot(request: HttpRequest, snapshot_id: str):
-    snapshot = get_snapshot(request, snapshot_id, with_archiveresults=False)
-    snapshot_id_str = str(snapshot.id)
-    crawl_id_str = str(snapshot.crawl.pk)
-    snapshot.cancel()
-
-    from archivebox.services.runner import run_pending_crawls
-
-    with crawl_lifecycle_lock(crawl_id_str):
-        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
-        snapshot = get_snapshot(request, snapshot_id_str, with_archiveresults=False)
-        deleted_count, _ = snapshot.delete()
-    return {
-        "success": True,
-        "snapshot_id": snapshot_id_str,
-        "crawl_id": crawl_id_str,
-        "deleted_count": deleted_count,
-    }
@@ -1,285 +0,0 @@
-from django.core.exceptions import ValidationError
-from django.db.models import Q
-from django.http import HttpRequest, HttpResponse
-from ninja import Router
-from ninja.errors import HttpError
-from ninja.pagination import paginate
-
-from archivebox.api.auth import authenticated_user_from_request
-from archivebox.config.common import get_config
-from archivebox.core.models import Snapshot, Tag
-from archivebox.core.permissions import public_snapshots_queryset
-from archivebox.core.tag_util import (
-    add_snapshot_counts,
-    build_tag_cards,
-    export_tag_snapshots_jsonl,
-    export_tag_urls,
-    get_matching_tags,
-    get_tag_by_ref,
-    normalize_created_by_filter,
-    normalize_created_year_filter,
-    normalize_has_snapshots_filter,
-    normalize_tag_sort,
-)
-
-from .pagination import CustomPagination
-from .schemas import (
-    TagAutocompleteSchema,
-    TagCreateResponseSchema,
-    TagCreateSchema,
-    TagDeleteResponseSchema,
-    TagSchema,
-    TagSearchResponseSchema,
-    TagSnapshotRequestSchema,
-    TagSnapshotResponseSchema,
-    TagUpdateResponseSchema,
-    TagUpdateSchema,
-)
-
-router = Router(tags=["Core Models"])
-
-
-@router.get("/tags", response=list[TagSchema], url_name="get_tags")
-@paginate(CustomPagination)
-def get_tags(request: HttpRequest):
-    setattr(request, "with_snapshots", False)
-    setattr(request, "with_archiveresults", False)
-    return get_matching_tags()
-
-
-@router.get("/tag/{tag_id}", response=TagSchema, url_name="get_tag")
-def get_tag(request: HttpRequest, tag_id: str, with_snapshots: bool = True):
-    setattr(request, "with_snapshots", with_snapshots)
-    setattr(request, "with_archiveresults", False)
-    try:
-        return get_tag_by_ref(tag_id)
-    except (Tag.DoesNotExist, ValidationError):
-        raise HttpError(404, "Tag not found")
-
-
-def _get_snapshot_for_tag_edit(snapshot_ref: str) -> Snapshot:
-    snapshot_ref = str(snapshot_ref or "").strip().lower()
-    if not snapshot_ref:
-        raise HttpError(400, "Snapshot id is required")
-
-    snapshot_qs = Snapshot.objects.only("id")
-    is_full_uuid = len(snapshot_ref.replace("-", "")) == 32 and all(char in "0123456789abcdef-" for char in snapshot_ref)
-    if is_full_uuid:
-        try:
-            return snapshot_qs.get(pk=snapshot_ref.replace("-", ""))
-        except (Snapshot.DoesNotExist, ValueError):
-            pass
-
-    if len(snapshot_ref) >= 14:
-        try:
-            return snapshot_qs.get(timestamp=snapshot_ref)
-        except Snapshot.DoesNotExist:
-            pass
-        except Snapshot.MultipleObjectsReturned:
-            snapshot = snapshot_qs.filter(timestamp=snapshot_ref).first()
-            if snapshot is not None:
-                return snapshot
-
-    try:
-        return snapshot_qs.get(Q(id__startswith=snapshot_ref) | Q(timestamp__startswith=snapshot_ref))
-    except Snapshot.DoesNotExist:
-        raise HttpError(404, "Snapshot not found") from None
-    except Snapshot.MultipleObjectsReturned:
-        snapshot = snapshot_qs.filter(Q(id__startswith=snapshot_ref) | Q(timestamp__startswith=snapshot_ref)).first()
-        if snapshot is None:
-            raise HttpError(404, "Snapshot not found")
-        return snapshot
-
-
-@router.get("/tags/search/", response=TagSearchResponseSchema, url_name="search_tags")
-def search_tags(
-    request: HttpRequest,
-    q: str = "",
-    sort: str = "created_desc",
-    created_by: str = "",
-    year: str = "",
-    has_snapshots: str = "all",
-):
-    """Return detailed tag cards for admin/live-search UIs."""
-    normalized_sort = normalize_tag_sort(sort)
-    normalized_created_by = normalize_created_by_filter(created_by)
-    normalized_year = normalize_created_year_filter(year)
-    normalized_has_snapshots = normalize_has_snapshots_filter(has_snapshots)
-    return {
-        "tags": build_tag_cards(
-            query=q,
-            request=request,
-            preview_limit=0,
-            sort=normalized_sort,
-            created_by=normalized_created_by,
-            year=normalized_year,
-            has_snapshots=normalized_has_snapshots,
-        ),
-        "sort": normalized_sort,
-        "created_by": normalized_created_by,
-        "year": normalized_year,
-        "has_snapshots": normalized_has_snapshots,
-    }
-
-
-def _public_tag_listing_enabled() -> bool:
-    return get_config().PUBLIC_INDEX
-
-
-def _request_has_tag_autocomplete_access(request: HttpRequest) -> bool:
-    if authenticated_user_from_request(request):
-        return True
-
-    return _public_tag_listing_enabled()
-
-
-@router.get("/tags/autocomplete/", response=TagAutocompleteSchema, url_name="tags_autocomplete", auth=None)
-def tags_autocomplete(request: HttpRequest, q: str = ""):
-    """Return tags matching the query for autocomplete."""
-    if not _request_has_tag_autocomplete_access(request):
-        raise HttpError(401, "Authentication required")
-
-    public_only = not request.user.is_authenticated and not request.__dict__.get("_api_token")
-    queryset = get_matching_tags(q)
-    public_snapshots = public_snapshots_queryset(Snapshot.objects.all())
-    if public_only:
-        queryset = queryset.filter(snapshot_set__id__in=public_snapshots.values("id")).distinct()
-    tags = list(queryset[: 50 if not q else 20])
-    add_snapshot_counts(tags, snapshot_queryset=public_snapshots if public_only else None)
-
-    return {
-        "tags": [{"id": tag.pk, "name": tag.name, "num_snapshots": tag.__dict__.get("num_snapshots", 0)} for tag in tags],
-    }
-
-
-@router.post("/tags/create/", response=TagCreateResponseSchema, url_name="tags_create")
-def tags_create(request: HttpRequest, data: TagCreateSchema):
-    """Create a new tag or return existing one."""
-    try:
-        tag, created = Tag.get_or_create_by_name(
-            data.name,
-            created_by=request.user if request.user.is_authenticated else None,
-        )
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-
-    return {
-        "success": True,
-        "tag_id": tag.pk,
-        "tag_name": tag.name,
-        "created": created,
-    }
-
-
-@router.post("/tag/{tag_id}/rename", response=TagUpdateResponseSchema, url_name="rename_tag")
-def rename_tag(request: HttpRequest, tag_id: int, data: TagUpdateSchema):
-    try:
-        tag = get_tag_by_ref(tag_id).rename(data.name)
-    except Tag.DoesNotExist as err:
-        raise HttpError(404, "Tag not found") from err
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-
-    return {
-        "success": True,
-        "tag_id": tag.pk,
-        "tag_name": tag.name,
-    }
-
-
-@router.delete("/tag/{tag_id}", response=TagDeleteResponseSchema, url_name="delete_tag")
-def delete_tag(request: HttpRequest, tag_id: int):
-    try:
-        tag = get_tag_by_ref(tag_id)
-    except Tag.DoesNotExist as err:
-        raise HttpError(404, "Tag not found") from err
-
-    deleted_count, _ = tag.delete()
-    return {
-        "success": True,
-        "tag_id": int(tag_id),
-        "deleted_count": deleted_count,
-    }
-
-
-@router.get("/tag/{tag_id}/urls.txt", url_name="tag_urls_export")
-def tag_urls_export(request: HttpRequest, tag_id: int):
-    try:
-        tag = get_tag_by_ref(tag_id)
-    except Tag.DoesNotExist as err:
-        raise HttpError(404, "Tag not found") from err
-
-    response = HttpResponse(export_tag_urls(tag), content_type="text/plain; charset=utf-8")
-    response["Content-Disposition"] = f'attachment; filename="tag-{tag.slug}-urls.txt"'
-    return response
-
-
-@router.get("/tag/{tag_id}/snapshots.jsonl", url_name="tag_snapshots_export")
-def tag_snapshots_export(request: HttpRequest, tag_id: int):
-    try:
-        tag = get_tag_by_ref(tag_id)
-    except Tag.DoesNotExist as err:
-        raise HttpError(404, "Tag not found") from err
-
-    response = HttpResponse(export_tag_snapshots_jsonl(tag), content_type="application/x-ndjson; charset=utf-8")
-    response["Content-Disposition"] = f'attachment; filename="tag-{tag.slug}-snapshots.jsonl"'
-    return response
-
-
-@router.post("/tags/add-to-snapshot/", response=TagSnapshotResponseSchema, url_name="tags_add_to_snapshot")
-def tags_add_to_snapshot(request: HttpRequest, data: TagSnapshotRequestSchema):
-    """Add a tag to a snapshot. Creates the tag if it doesn't exist."""
-    snapshot = _get_snapshot_for_tag_edit(data.snapshot_id)
-
-    # Get or create the tag
-    if data.tag_name:
-        try:
-            tag, _ = Tag.get_or_create_by_name(
-                data.tag_name,
-                created_by=request.user if request.user.is_authenticated else None,
-            )
-        except ValueError as err:
-            raise HttpError(400, str(err)) from err
-    elif data.tag_id:
-        try:
-            tag = get_tag_by_ref(data.tag_id)
-        except Tag.DoesNotExist:
-            raise HttpError(404, "Tag not found")
-    else:
-        raise HttpError(400, "Either tag_name or tag_id is required")
-
-    snapshot.add_tag_ids([tag.pk])
-
-    return {
-        "success": True,
-        "tag_id": tag.pk,
-        "tag_name": tag.name,
-    }
-
-
-@router.post("/tags/remove-from-snapshot/", response=TagSnapshotResponseSchema, url_name="tags_remove_from_snapshot")
-def tags_remove_from_snapshot(request: HttpRequest, data: TagSnapshotRequestSchema):
-    """Remove a tag from a snapshot."""
-    snapshot = _get_snapshot_for_tag_edit(data.snapshot_id)
-
-    # Get the tag
-    if data.tag_id:
-        try:
-            tag = Tag.objects.get(pk=data.tag_id)
-        except Tag.DoesNotExist:
-            raise HttpError(404, "Tag not found")
-    elif data.tag_name:
-        try:
-            tag = Tag.objects.get(name__iexact=data.tag_name.strip())
-        except Tag.DoesNotExist:
-            raise HttpError(404, "Tag not found")
-    else:
-        raise HttpError(400, "Either tag_name or tag_id is required")
-
-    snapshot.remove_tag_ids([tag.pk])
-
-    return {
-        "success": True,
-        "tag_id": tag.pk,
-        "tag_name": tag.name,
-    }
@@ -1,258 +0,0 @@
-__package__ = "archivebox.api"
-
-from pathlib import Path
-from uuid import UUID
-from datetime import datetime
-from django.http import FileResponse, HttpRequest
-from django.shortcuts import redirect
-from django.utils import timezone
-
-from django.contrib.auth.models import User
-
-from archivebox.api.schemas import OwnedObjectSchema
-
-from ninja import Router, Schema
-from ninja.errors import HttpError
-
-from archivebox.core.models import Snapshot
-from archivebox.core.permissions import (
-    PERMISSIONS_PUBLIC,
-    PERMISSIONS_UNLISTED,
-    is_admin_user,
-    normalize_permissions,
-)
-from archivebox.config.common import get_config
-from archivebox.crawls.models import Crawl
-from archivebox.misc.util import filter_queryset_by_uuid_substring, validate_url
-
-from .auth import API_AUTH_METHODS, authenticated_user_from_request
-
-router = Router(tags=["Crawl Models"], auth=API_AUTH_METHODS)
-
-
-class CrawlSchema(OwnedObjectSchema):
-    TYPE: str = "crawls.models.Crawl"
-
-    id: UUID
-
-    modified_at: datetime
-    created_at: datetime
-
-    status: str
-    retry_at: datetime | None
-    is_paused: bool
-
-    urls: str
-    max_depth: int
-    tags_str: str
-    config: dict
-
-    @staticmethod
-    def resolve_config(obj):
-        # Redact credential values so REST responses can never leak the raw
-        # token/secret/api-key that the operator stored in Crawl.config.
-        from archivebox.config.common import redact_sensitive_config
-
-        return redact_sensitive_config(obj.config)
-
-    @staticmethod
-    def resolve_snapshots(obj, context):
-        if bool(context["request"].__dict__.get("with_snapshots", False)):
-            return obj.snapshot_set.all().distinct()
-        return Snapshot.objects.none()
-
-
-class CrawlUpdateSchema(Schema):
-    action: str | None = None
-    status: str | None = None
-    retry_at: datetime | None = None
-    tags: list[str] | None = None
-    tags_str: str | None = None
-
-
-class CrawlCreateSchema(Schema):
-    urls: list[str]
-    max_depth: int = 0
-    tags: list[str] | None = None
-    tags_str: str = ""
-    label: str = ""
-    notes: str = ""
-    config: dict = {}
-
-
-class CrawlDeleteResponseSchema(Schema):
-    success: bool
-    crawl_id: str
-    deleted_count: int
-    deleted_snapshots: int
-
-
-def normalize_tag_list(tags: list[str] | None = None, tags_str: str = "") -> list[str]:
-    if tags is not None:
-        return [tag.strip() for tag in tags if tag and tag.strip()]
-    return [tag.strip() for tag in tags_str.split(",") if tag.strip()]
-
-
-@router.get("/crawls", response=list[CrawlSchema], url_name="get_crawls")
-def get_crawls(request: HttpRequest):
-    return Crawl.objects.all().distinct()
-
-
-@router.post("/crawls", response=CrawlSchema, url_name="create_crawl")
-def create_crawl(request: HttpRequest, data: CrawlCreateSchema):
-    try:
-        urls = [validate_url(url) for url in data.urls if url and url.strip()]
-    except ValueError as err:
-        raise HttpError(400, str(err)) from err
-    if not urls:
-        raise HttpError(400, "At least one URL is required")
-    if data.max_depth not in (0, 1, 2, 3, 4):
-        raise HttpError(400, "max_depth must be between 0 and 4")
-
-    tags = normalize_tag_list(data.tags, data.tags_str)
-    config = dict(data.config or {})
-    config.setdefault("PARSER", "url_list")
-    config.setdefault("PERMISSIONS", str(get_config().PERMISSIONS))
-    crawl = Crawl.objects.create(
-        urls="\n".join(urls),
-        max_depth=data.max_depth,
-        tags_str=",".join(tags),
-        label=data.label,
-        notes=data.notes,
-        config=config,
-        status=Crawl.StatusChoices.QUEUED,
-        retry_at=timezone.now(),
-        created_by=request.user if isinstance(request.user, User) else None,
-    )
-    return crawl
-
-
-def get_crawl_by_ref(crawl_id: str):
-    return filter_queryset_by_uuid_substring(Crawl.objects.all(), crawl_id).get()
-
-
-@router.get("/crawl/{crawl_id}", response=CrawlSchema, url_name="get_crawl")
-def get_crawl(request: HttpRequest, crawl_id: str, as_rss: bool = False, with_snapshots: bool = False, with_archiveresults: bool = False):
-    """Get a specific Crawl by id."""
-    setattr(request, "with_snapshots", with_snapshots)
-    setattr(request, "with_archiveresults", with_archiveresults)
-    crawl = get_crawl_by_ref(crawl_id)
-
-    if crawl and as_rss:
-        query = request.GET.copy()
-        query.pop("as_rss", None)
-        query["crawl_id"] = str(crawl.id)
-        return redirect(f"/api/v1/core/snapshots.rss?{query.urlencode()}")
-
-    return crawl
-
-
-def crawl_file(request: HttpRequest, crawl_id: str, path: str):
-    # Try to resolve the crawl first; if it doesn't exist, return 404.
-    try:
-        crawl = get_crawl_by_ref(crawl_id)
-    except Crawl.DoesNotExist:
-        raise HttpError(404, "Crawl not found")
-
-    user = authenticated_user_from_request(request)
-
-    # Gate access using the same model as SnapshotView/can_view_snapshot:
-    # admins always pass; owners can see their own crawls; otherwise the crawl
-    # must be PUBLIC or UNLISTED. Don't disclose existence of private crawls.
-    if not is_admin_user(request):
-        permissions = normalize_permissions(crawl.permissions)
-        is_owner = bool(user and crawl.created_by_id == user.id)
-        if not is_owner and permissions not in {PERMISSIONS_PUBLIC, PERMISSIONS_UNLISTED}:
-            raise HttpError(404, "Crawl not found")
-
-    crawl_root = Path(crawl.output_dir).resolve()
-    file_path = (crawl_root / path).resolve()
-    if not file_path.is_file() or crawl_root not in file_path.parents:
-        raise HttpError(404, "Crawl file not found")
-
-    response = FileResponse(file_path.open("rb"))
-    response["Cache-Control"] = "no-store, no-cache, max-age=0, must-revalidate"
-    response["Pragma"] = "no-cache"
-    response["Expires"] = "0"
-    response["X-Content-Type-Options"] = "nosniff"
-    return response
-
-
-@router.get("/crawl/{crawl_id}/files/{filename}", auth=None, url_name="crawl_file_root")
-def crawl_file_root(request: HttpRequest, crawl_id: str, filename: str):
-    return crawl_file(request, crawl_id, filename)
-
-
-@router.get("/crawl/{crawl_id}/files/{folder}/{filename}", auth=None, url_name="crawl_file_nested_1")
-def crawl_file_nested_1(request: HttpRequest, crawl_id: str, folder: str, filename: str):
-    return crawl_file(request, crawl_id, f"{folder}/{filename}")
-
-
-@router.get("/crawl/{crawl_id}/files/{folder}/{subfolder}/{filename}", auth=None, url_name="crawl_file_nested_2")
-def crawl_file_nested_2(request: HttpRequest, crawl_id: str, folder: str, subfolder: str, filename: str):
-    return crawl_file(request, crawl_id, f"{folder}/{subfolder}/{filename}")
-
-
-@router.patch("/crawl/{crawl_id}", response=CrawlSchema, url_name="patch_crawl")
-def patch_crawl(request: HttpRequest, crawl_id: str, data: CrawlUpdateSchema):
-    """Update a crawl (e.g., set status=sealed to cancel queued work)."""
-    crawl = get_crawl_by_ref(crawl_id)
-    payload = data.dict(exclude_unset=True)
-    update_fields = ["modified_at"]
-
-    action = payload.pop("action", None)
-    if action:
-        if action == "pause":
-            crawl.pause()
-            return crawl
-        if action in ("resume", "unpause"):
-            crawl.resume()
-            return crawl
-        if action == "cancel":
-            crawl.cancel()
-            return crawl
-        raise HttpError(400, f"Invalid action: {action}")
-
-    tags = payload.pop("tags", None)
-    tags_str = payload.pop("tags_str", None)
-    if tags is not None or tags_str is not None:
-        crawl.tags_str = ",".join(normalize_tag_list(tags, tags_str or ""))
-        update_fields.append("tags_str")
-
-    if "status" in payload:
-        if payload["status"] not in Crawl.StatusChoices.values:
-            raise HttpError(400, f"Invalid status: {payload['status']}")
-        if payload["status"] == Crawl.StatusChoices.SEALED:
-            crawl.cancel()
-            return crawl
-        crawl.status = payload["status"]
-        update_fields.append("status")
-
-    if "retry_at" in payload:
-        crawl.retry_at = payload["retry_at"]
-        update_fields.append("retry_at")
-
-    crawl.save(update_fields=update_fields)
-    return crawl
-
-
-@router.delete("/crawl/{crawl_id}", response=CrawlDeleteResponseSchema, url_name="delete_crawl")
-def delete_crawl(request: HttpRequest, crawl_id: str):
-    crawl = get_crawl_by_ref(crawl_id)
-    crawl_id_str = str(crawl.id)
-    crawl.cancel()
-
-    from archivebox.crawls.locks import crawl_lifecycle_lock
-    from archivebox.services.runner import run_pending_crawls
-
-    with crawl_lifecycle_lock(crawl_id_str):
-        run_pending_crawls(crawl_id=crawl_id_str, daemon=False)
-        crawl = get_crawl_by_ref(crawl_id_str)
-        snapshot_count = crawl.snapshot_set.count()
-        deleted_count, _ = crawl.delete()
-    return {
-        "success": True,
-        "crawl_id": crawl_id_str,
-        "deleted_count": deleted_count,
-        "deleted_snapshots": snapshot_count,
-    }
@@ -1,161 +0,0 @@
-__package__ = "archivebox.api"
-
-from uuid import UUID
-from typing import Annotated
-from datetime import datetime
-
-from django.http import HttpRequest
-
-from ninja import FilterLookup, FilterSchema, Query, Router, Schema
-from ninja.pagination import paginate
-
-from archivebox.api.v1_core import CustomPagination
-
-
-router = Router(tags=["Machine and Dependencies"])
-
-
-# ============================================================================
-# Machine Schemas
-# ============================================================================
-
-
-class MachineSchema(Schema):
-    """Schema for Machine model."""
-
-    TYPE: str = "machine.Machine"
-    id: UUID
-    created_at: datetime
-    modified_at: datetime
-    guid: str
-    hostname: str
-    hw_in_docker: bool
-    hw_in_vm: bool
-    hw_manufacturer: str
-    hw_product: str
-    hw_uuid: str
-    os_arch: str
-    os_family: str
-    os_platform: str
-    os_release: str
-    os_kernel: str
-    stats: dict
-    num_uses_succeeded: int
-    num_uses_failed: int
-
-
-class MachineFilterSchema(FilterSchema):
-    id: Annotated[str | None, FilterLookup("id__startswith")] = None
-    hostname: Annotated[str | None, FilterLookup("hostname__icontains")] = None
-    os_platform: Annotated[str | None, FilterLookup("os_platform__icontains")] = None
-    os_arch: Annotated[str | None, FilterLookup("os_arch")] = None
-    hw_in_docker: Annotated[bool | None, FilterLookup("hw_in_docker")] = None
-    hw_in_vm: Annotated[bool | None, FilterLookup("hw_in_vm")] = None
-    bin_providers: Annotated[str | None, FilterLookup("bin_providers__icontains")] = None
-
-
-# ============================================================================
-# Binary Schemas
-# ============================================================================
-
-
-class BinarySchema(Schema):
-    """Schema for Binary model."""
-
-    TYPE: str = "machine.Binary"
-    id: UUID
-    created_at: datetime
-    modified_at: datetime
-    machine_id: UUID
-    machine_hostname: str
-    name: str
-    binproviders: str
-    binprovider: str
-    abspath: str
-    version: str
-    sha256: str
-    status: str
-    is_valid: bool
-    num_uses_succeeded: int
-    num_uses_failed: int
-
-    @staticmethod
-    def resolve_machine_hostname(obj) -> str:
-        return obj.machine.hostname
-
-    @staticmethod
-    def resolve_is_valid(obj) -> bool:
-        return obj.is_valid
-
-
-class BinaryFilterSchema(FilterSchema):
-    id: Annotated[str | None, FilterLookup("id__startswith")] = None
-    name: Annotated[str | None, FilterLookup("name__icontains")] = None
-    binprovider: Annotated[str | None, FilterLookup("binprovider")] = None
-    status: Annotated[str | None, FilterLookup("status")] = None
-    machine_id: Annotated[str | None, FilterLookup("machine_id__startswith")] = None
-    version: Annotated[str | None, FilterLookup("version__icontains")] = None
-
-
-# ============================================================================
-# Machine Endpoints
-# ============================================================================
-
-
-@router.get("/machines", response=list[MachineSchema], url_name="get_machines")
-@paginate(CustomPagination)
-def get_machines(request: HttpRequest, filters: Query[MachineFilterSchema]):
-    """List all machines."""
-    from archivebox.machine.models import Machine
-
-    return filters.filter(Machine.objects.all()).distinct()
-
-
-@router.get("/machine/current", response=MachineSchema, url_name="get_current_machine")
-def get_current_machine(request: HttpRequest):
-    """Get the current machine."""
-    from archivebox.machine.models import Machine
-
-    return Machine.current()
-
-
-@router.get("/machine/{machine_id}", response=MachineSchema, url_name="get_machine")
-def get_machine(request: HttpRequest, machine_id: str):
-    """Get a specific machine by ID."""
-    from archivebox.machine.models import Machine
-    from django.db.models import Q
-
-    return Machine.objects.get(Q(id__startswith=machine_id) | Q(hostname__iexact=machine_id))
-
-
-# ============================================================================
-
-
-# ============================================================================
-# Binary Endpoints
-# ============================================================================
-
-
-@router.get("/binaries", response=list[BinarySchema], url_name="get_binaries")
-@paginate(CustomPagination)
-def get_binaries(request: HttpRequest, filters: Query[BinaryFilterSchema]):
-    """List all binaries."""
-    from archivebox.machine.models import Binary
-
-    return filters.filter(Binary.objects.all().select_related("machine")).distinct()
-
-
-@router.get("/binary/{binary_id}", response=BinarySchema, url_name="get_binary")
-def get_binary(request: HttpRequest, binary_id: str):
-    """Get a specific binary by ID."""
-    from archivebox.machine.models import Binary
-
-    return Binary.objects.select_related("machine").get(id__startswith=binary_id)
-
-
-@router.get("/binary/by-name/{name}", response=list[BinarySchema], url_name="get_binaries_by_name")
-def get_binaries_by_name(request: HttpRequest, name: str):
-    """Get all binaries with the given name."""
-    from archivebox.machine.models import Binary
-
-    return list(Binary.objects.filter(name__iexact=name).select_related("machine"))
@@ -1,160 +0,0 @@
-__package__ = "archivebox.api"
-
-import json
-from datetime import datetime
-from typing import Any
-from uuid import UUID
-
-from django.db.models import Q
-from django.http import HttpRequest
-from archivebox.api.schemas import OwnedObjectSchema
-
-from ninja import Router, Schema
-from ninja.pagination import paginate
-from pydantic import Field
-
-from archivebox.api.v1_core import CustomPagination
-from archivebox.personas.importers import validate_persona_name
-from archivebox.personas.models import Persona
-
-
-router = Router(tags=["Personas"])
-
-
-class PersonaBrowserSettingsSchema(Schema):
-    user_agent: str = ""
-    viewport_size: str = ""
-    viewport_device_scale_factor: float | None = None
-    language: str = ""
-    timezone: str = ""
-    geolocation: dict[str, Any] | None = None
-
-
-class PersonaSyncSchema(Schema):
-    extension_persona_id: str
-    name: str
-    settings: PersonaBrowserSettingsSchema = Field(default_factory=PersonaBrowserSettingsSchema)
-    cookies_txt: str = ""
-    auth_json: dict[str, Any] = Field(default_factory=dict)
-
-
-class PersonaSchema(OwnedObjectSchema):
-    TYPE: str = "personas.models.Persona"
-    id: UUID
-    name: str
-    created_at: datetime
-    config: dict[str, Any] | None
-
-    @staticmethod
-    def resolve_config(obj):
-        # Redact credential values so REST responses don't leak the raw
-        # token/secret/api-key the operator stored in Persona.config.
-        from archivebox.config.common import redact_sensitive_config
-
-        return redact_sensitive_config(obj.config)
-
-
-class PersonaSyncResponseSchema(Schema):
-    success: bool
-    created: bool
-    persona: PersonaSchema
-    cookies_file_written: bool
-    auth_file_written: bool
-
-
-def browser_settings_to_config(extension_persona_id: str, settings: PersonaBrowserSettingsSchema) -> dict[str, Any]:
-    config: dict[str, Any] = {
-        "BROWSER_EXTENSION_PERSONA_ID": extension_persona_id,
-        "BROWSER_EXTENSION_SYNCED_AT": datetime.utcnow().isoformat() + "Z",
-    }
-
-    if settings.user_agent:
-        config.update(
-            {
-                "USER_AGENT": settings.user_agent,
-                "CHROME_USER_AGENT": settings.user_agent,
-                "WGET_USER_AGENT": settings.user_agent,
-                "CURL_USER_AGENT": settings.user_agent,
-            },
-        )
-    if settings.viewport_size:
-        config.update(
-            {
-                "RESOLUTION": settings.viewport_size,
-                "CHROME_RESOLUTION": settings.viewport_size,
-            },
-        )
-    if settings.viewport_device_scale_factor is not None:
-        config["BROWSER_DEVICE_SCALE_FACTOR"] = settings.viewport_device_scale_factor
-    if settings.language:
-        config["BROWSER_LANGUAGE"] = settings.language
-    if settings.timezone:
-        config["BROWSER_TIMEZONE"] = settings.timezone
-    if settings.geolocation:
-        config["BROWSER_GEOLOCATION"] = settings.geolocation
-
-    return config
-
-
-def find_persona(extension_persona_id: str, name: str) -> Persona | None:
-    return (
-        Persona.objects.filter(
-            Q(config__BROWSER_EXTENSION_PERSONA_ID=extension_persona_id) | Q(name=name),
-        )
-        .order_by("created_at")
-        .first()
-    )
-
-
-@router.get("/personas", response=list[PersonaSchema], url_name="get_personas")
-@paginate(CustomPagination)
-def get_personas(request: HttpRequest):
-    """List personas available on this ArchiveBox server."""
-    return Persona.objects.all().order_by("name")
-
-
-@router.post("/sync", response=PersonaSyncResponseSchema, url_name="sync_persona")
-def sync_persona(request: HttpRequest, payload: PersonaSyncSchema):
-    """
-    Create or update a Persona from a browser extension profile export.
-
-    The extension sends browser settings plus portable auth artifacts. The server
-    keeps browser override settings in Persona.config and writes cookies.txt /
-    auth.json into the persona directory for extractors to consume.
-    """
-    name = payload.name.strip()
-    is_valid, error_message = validate_persona_name(name)
-    if not is_valid:
-        raise ValueError(error_message)
-
-    persona = find_persona(payload.extension_persona_id, name)
-    created = persona is None
-    if persona is None:
-        persona = Persona(name=name)
-        if request.user.is_authenticated:
-            persona.created_by = request.user
-
-    persona.config = {
-        **(persona.config or {}),
-        **browser_settings_to_config(payload.extension_persona_id, payload.settings),
-    }
-    persona.save()
-    persona.ensure_dirs()
-
-    cookies_written = False
-    if payload.cookies_txt.strip():
-        (persona.path / "cookies.txt").write_text(payload.cookies_txt)
-        cookies_written = True
-
-    auth_written = False
-    if payload.auth_json:
-        (persona.path / "auth.json").write_text(json.dumps(payload.auth_json, indent=2, sort_keys=True) + "\n")
-        auth_written = True
-
-    return {
-        "success": True,
-        "created": created,
-        "persona": persona,
-        "cookies_file_written": cookies_written,
-        "auth_file_written": auth_written,
-    }
@@ -1,32 +0,0 @@
-__package__ = "archivebox.api"
-
-from typing import Any
-from collections.abc import Callable
-import logging
-
-from django.db import transaction
-from signal_webhooks.handlers import sync_task_handler
-
-
-logger = logging.getLogger(__name__)
-
-
-def warning_error_handler(hook: Any, error: Exception | None) -> None:
-    if error is not None:
-        logger.warning("Outbound webhook %r failed: %s", hook.name, error)
-        return
-
-    logger.warning("Outbound webhook %r returned a non-success response.", hook.name)
-
-
-def transaction_on_commit_task_handler(hook: Callable[..., None], **kwargs: Any) -> None:
-    def run_webhook() -> None:
-        try:
-            sync_task_handler(hook, **kwargs)
-        except Exception:
-            logger.warning("Outbound webhook failed after transaction commit.", exc_info=True)
-
-    try:
-        transaction.on_commit(run_webhook)
-    except Exception:
-        logger.warning("Could not schedule outbound webhook after transaction commit.", exc_info=True)
@@ -1 +0,0 @@
-__package__ = "archivebox.base_models"
@@ -1,349 +0,0 @@
-"""Base admin classes for models using UUIDv7."""
-
-__package__ = "archivebox.base_models"
-
-import json
-import uuid
-from collections.abc import Mapping
-from typing import ClassVar, NotRequired, TypedDict, cast
-
-from django import forms
-from django.contrib import admin
-from django.db import DatabaseError, models
-from django.forms.renderers import BaseRenderer
-from django.http import HttpRequest, QueryDict
-from django.urls import path, register_converter
-from django.utils.html import format_html
-from django.utils.safestring import SafeString, mark_safe
-from django_object_actions import DjangoObjectActions
-
-
-def card_fieldset(title: str | None, fields: tuple, *, wide: bool = False, **options):
-    """Build a Django fieldset using the shared card layout."""
-    return title, {"fields": fields, "classes": ("card", "wide") if wide else ("card",), **options}
-
-
-class HexUUIDConverter:
-    """URL path converter that canonicalizes UUIDs to their 32-char hex form.
-
-    Accepts both the hyphenated (``aaaaaaaa-bbbb-...``) and bare-hex
-    (``aaaaaaaabbbb...``) UUID strings on the way in (Django's UUIDField
-    parses either), but ``to_url`` always emits the bare-hex form. This is
-    what makes ``reverse("admin:app_model_change", args=[obj.pk])`` produce
-    ``/admin/app/model/06a1a8facb0d.../change/`` instead of the default
-    hyphenated rendering — admin links throughout the app reverse through
-    this converter once ``BaseModelAdmin.get_urls`` swaps in
-    ``<hexuuid:object_id>`` below.
-    """
-
-    regex = r"[0-9a-fA-F]{32}|[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}"
-
-    def to_python(self, value: str) -> str:
-        # Strip hyphens but stay as a string — Django admin treats the
-        # captured object_id as a string and calls ``model._meta.pk.to_python``
-        # itself, so we don't want to short-circuit that.
-        return value.replace("-", "")
-
-    def to_url(self, value) -> str:
-        if isinstance(value, uuid.UUID):
-            return value.hex
-        return str(value).replace("-", "")
-
-
-register_converter(HexUUIDConverter, "hexuuid")
-
-
-class ConfigOption(TypedDict):
-    plugin: str
-    type: str | list[str]
-    default: object
-    description: str
-    enum: NotRequired[list[object]]
-    pattern: NotRequired[str]
-    minimum: NotRequired[int | float]
-    maximum: NotRequired[int | float]
-
-
-class KeyValueWidget(forms.Widget):
-    """
-    A widget that renders JSON dict as editable key-value input fields
-    with + and - buttons to add/remove rows.
-    Includes autocomplete for available config keys from the plugin system.
-    """
-
-    template_name = ""  # We render manually
-
-    class Media:
-        css: ClassVar[dict[str, list[str]]] = {
-            "all": [],
-        }
-        js: ClassVar[list[str]] = []
-
-    def _get_config_options(self) -> dict[str, ConfigOption]:
-        """Get available config options from plugins."""
-        try:
-            from archivebox.config.common import config_field_metadata
-
-            options: dict[str, ConfigOption] = {}
-            for key, metadata in config_field_metadata().items():
-                option_type = metadata.get("type", "string")
-                # Core fields expose Python type names; plugins use JSON Schema.
-                # The editor should apply the same typed validation to both.
-                type_names = {"bool": "boolean", "int": "integer", "float": "number", "str": "string", "dict": "object", "list": "array"}
-                if isinstance(option_type, str):
-                    option_type = type_names.get(option_type, option_type)
-                option: ConfigOption = {
-                    "plugin": str(metadata.get("plugin", "archivebox")),
-                    "type": cast(str | list[str], option_type if isinstance(option_type, (str, list)) else str(option_type)),
-                    "default": metadata.get("default", ""),
-                    "description": str(metadata.get("description", "")),
-                }
-                schema = metadata.get("schema")
-                if isinstance(schema, Mapping):
-                    for schema_key in ("enum", "pattern", "minimum", "maximum"):
-                        if schema_key in schema:
-                            option[schema_key] = schema[schema_key]
-                options[key] = option
-            return options
-        except (ImportError, KeyError, TypeError, ValueError):
-            return {}
-
-    def _parse_value(self, value: object) -> dict[str, object]:
-        # Parse JSON value to dict
-        if value is None:
-            return {}
-        if isinstance(value, str):
-            try:
-                parsed = json.loads(value) if value else {}
-            except json.JSONDecodeError:
-                return {}
-            return parsed if isinstance(parsed, dict) else {}
-        if isinstance(value, Mapping):
-            return {str(key): item for key, item in value.items()}
-        return {}
-
-    def render(
-        self,
-        name: str,
-        value: object,
-        attrs: Mapping[str, str] | None = None,
-        renderer: BaseRenderer | None = None,
-    ) -> SafeString:
-        from django.template.loader import render_to_string
-
-        widget_id = attrs.get("id", name) if attrs else name
-        options = self._get_config_options()
-        rows = [
-            mark_safe(self._render_row(widget_id, key, val if isinstance(val, str) else json.dumps(val)))
-            for key, val in self._parse_value(value).items()
-        ]
-        empty_row = mark_safe(self._render_row(widget_id, "", ""))
-        # JSON is embedded in a script, so escape HTML delimiters without changing data.
-        metadata = json.dumps(options).translate({ord("<"): r"\u003C", ord(">"): r"\u003E", ord("&"): r"\u0026"})
-        return mark_safe(
-            render_to_string(
-                "admin/widgets/config.html",
-                {
-                    "name": name,
-                    "widget_id": widget_id,
-                    "datalist_options": mark_safe(
-                        "\n".join(
-                            f'<option value="{self._escape(key)}">{self._escape(opt["description"][:60] or opt["type"])}</option>'
-                            for key, opt in sorted(options.items())
-                        ),
-                    ),
-                    "config_meta_json": metadata,
-                    "rows": [*rows, empty_row],
-                    "empty_row": empty_row,
-                },
-            ),
-        )
-
-    def _render_row(self, widget_id: str, key: str, value: str) -> str:
-        from archivebox.config.common import is_sensitive_config_key
-
-        # Sensitive keys (``*TOKEN*``, ``*SECRET*``, ``*API_KEY*``, ``*APIKEY*``) are
-        # rendered write-only: the input is a password field with a placeholder
-        # showing the value is set, but the raw value is NEVER sent to the browser.
-        # When the user submits the form with the field left blank, the
-        # ``ConfigEditorMixin.save_model`` hook re-merges the previously-saved
-        # value so leaving it untouched is a no-op rather than a destructive clear.
-        is_sensitive = is_sensitive_config_key(key)
-        has_value = bool(value)
-        if is_sensitive:
-            input_type = "password"
-            rendered_value = ""
-            placeholder = (
-                "•••••• (saved — enter new value to replace, clear by deleting row)" if has_value else "value (will be saved write-only)"
-            )
-            extra_attrs = ' autocomplete="off" data-sensitive="1"' + (' data-had-value="1"' if has_value else "")
-        else:
-            input_type = "text"
-            rendered_value = self._escape(value)
-            placeholder = "value"
-            extra_attrs = ""
-        return f'''
-            <div class="key-value-row" style="margin-bottom: 6px;">
-                <div class="kv-inputs" style="display: flex; gap: 8px; align-items: center;">
-                    <input type="text" class="kv-key" value="{self._escape(key)}" placeholder="KEY" list="{widget_id}_keys"
-                           style="flex: 1; padding: 6px 8px; border: 1px solid #ccc; border-radius: 4px; font-family: monospace; font-size: 12px;">
-                    <input type="{input_type}" class="kv-value" value="{rendered_value}" placeholder="{self._escape(placeholder)}"{extra_attrs}
-                           style="flex: 2; padding: 6px 8px; border: 1px solid #ccc; border-radius: 4px; font-family: monospace; font-size: 12px;">
-                    <datalist class="kv-value-options"></datalist>
-                    <button type="button" onclick="removeKeyValueRow_{widget_id}(this)"
-                            style="padding: 4px 10px; cursor: pointer; background: #ba2121; color: white; border: none; border-radius: 4px; font-weight: bold;">−</button>
-                </div>
-                <div class="kv-help" style="margin-top: 4px; font-size: 11px; color: #666; font-style: italic;"></div>
-            </div>
-        '''
-
-    def _escape(self, s: object) -> str:
-        """Escape HTML special chars in attribute values."""
-        if not s:
-            return ""
-        return str(s).replace("&", "&amp;").replace("<", "&lt;").replace(">", "&gt;").replace('"', "&quot;")
-
-    def value_from_datadict(
-        self,
-        data: QueryDict | Mapping[str, object],
-        files: object,
-        name: str,
-    ) -> str:
-        value = data.get(name, "{}")
-        return value if isinstance(value, str) else "{}"
-
-
-class ConfigEditorMixin(admin.ModelAdmin):
-    """
-    Mixin for admin classes with a config JSON field.
-
-    Provides a key-value editor widget with autocomplete for available config keys.
-    """
-
-    def formfield_for_dbfield(
-        self,
-        db_field: models.Field,
-        request: HttpRequest,
-        **kwargs: object,
-    ) -> forms.Field | None:
-        """Use KeyValueWidget for the config JSON field."""
-        if db_field.name == "config":
-            kwargs["widget"] = KeyValueWidget()
-        return super().formfield_for_dbfield(db_field, request, **kwargs)
-
-    def save_model(self, request: HttpRequest, obj, form, change):
-        """Preserve write-only redacted credentials on save.
-
-        The KeyValueWidget renders sensitive keys (``*TOKEN*``, ``*SECRET*``,
-        ``*API_KEY*``, ``*APIKEY*``) with an empty value + password input —
-        the real value never leaves the server. On submit, an empty value
-        for a sensitive key that was previously set means "leave untouched",
-        not "clear it." We honor that here by re-merging the stored value
-        before the row is written. Explicitly removing the row in the UI
-        still clears it (the key is gone from the submitted JSON, so there's
-        nothing to merge over).
-        """
-        from archivebox.config.common import is_sensitive_config_key
-
-        if change and obj.pk and obj.config is not None:
-            try:
-                stored = type(obj).objects.filter(pk=obj.pk).values_list("config", flat=True).first() or {}
-            except (AttributeError, DatabaseError, TypeError, ValueError):
-                stored = {}
-            if isinstance(stored, dict):
-                new_config = dict(obj.config or {})
-                for key, new_value in list(new_config.items()):
-                    if not is_sensitive_config_key(key):
-                        continue
-                    if new_value not in (None, "") and new_value != "********":
-                        continue
-                    if key in stored:
-                        new_config[key] = stored[key]
-                obj.config = new_config
-        super().save_model(request, obj, form, change)
-
-
-class BaseModelAdmin(DjangoObjectActions, admin.ModelAdmin):
-    list_display = ("id", "created_at", "created_by")
-    readonly_fields = ("id", "created_at", "modified_at")
-    show_search_mode_selector = False
-    change_form_template = "admin/archivebox_change_form.html"
-
-    @admin.display(description="Health", ordering="health")
-    def health_display(self, obj):
-        h = obj.health
-        color = "green" if h >= 80 else "orange" if h >= 50 else "red"
-        return format_html('<span style="color: {};">{}</span>', color, h)
-
-    def get_ordering_fields(self, request):
-        ordering = request.GET.get("o")
-        if not ordering:
-            return set()
-        fields = set()
-        for part in ordering.split("."):
-            if not part:
-                continue
-            try:
-                idx = abs(int(part)) - 1
-            except ValueError:
-                continue
-            if 0 <= idx < len(self.list_display):
-                fields.add(self.list_display[idx])
-        return fields
-
-    def get_admin_toolbar_actions(self, request, obj):
-        """Return extra action button dicts for the shared change-form toolbar.
-
-        See ``templates/admin/includes/archivebox_toolbar.html`` for the
-        accepted keys. Default: no extras (toolbar renders Save/History/Delete
-        plus any ``django-object-actions`` change_actions).
-        """
-        return []
-
-    def render_change_form(self, request, context, add=False, change=False, form_url="", obj=None):
-        context.setdefault("archivebox_admin_actions", self.get_admin_toolbar_actions(request, obj))
-        return super().render_change_form(request, context, add=add, change=change, form_url=form_url, obj=obj)
-
-    def get_default_search_mode(self) -> str:
-        # The shared changelist template always asks every admin for a default
-        # search mode, even when the search-mode toggle is hidden.
-        return "meta"
-
-    def get_form(
-        self,
-        request: HttpRequest,
-        obj: models.Model | None = None,
-        change: bool = False,
-        **kwargs: object,
-    ):
-        form = super().get_form(request, obj, change=change, **kwargs)
-        if "created_by" in form.base_fields:
-            form.base_fields["created_by"].initial = request.user
-        return form
-
-    def get_urls(self):
-        """Swap the per-object admin URLs from ``<path:object_id>`` to
-        ``<hexuuid:object_id>`` so canonical change/delete/history URLs use the
-        32-char hex form. The hyphenated form still resolves because the
-        converter's regex accepts both — Django reverses through ``to_url``
-        which always emits hex, so links in templates / changelists / inline
-        formsets all canonicalize automatically.
-
-        Non-UUID PKs (an ``IntegerField`` PK on some legacy table, for example)
-        won't match the converter's regex and fall back to the default
-        ``<path:object_id>`` patterns we still include after our swap.
-        """
-        info = self.opts.app_label, self.opts.model_name
-        object_routes = [
-            path("<hexuuid:object_id>/history/", self.admin_site.admin_view(self.history_view), name="{}_{}_history".format(*info)),
-            path("<hexuuid:object_id>/delete/", self.admin_site.admin_view(self.delete_view), name="{}_{}_delete".format(*info)),
-            path("<hexuuid:object_id>/change/", self.admin_site.admin_view(self.change_view), name="{}_{}_change".format(*info)),
-        ]
-        # Append after super().get_urls() so our patterns are the
-        # *last-registered* ones with the canonical admin URL names — Django's
-        # reverse() picks the later registration when names collide, which is
-        # how we make ``reverse("admin:app_model_change", args=[obj.pk])``
-        # emit the hex form. The original ``<path:object_id>`` routes stay in
-        # place as a fallback for non-UUID PKs and for resolving inbound
-        # hyphenated URLs (the ``hexuuid`` regex accepts both forms anyway).
-        return super().get_urls() + object_routes
@@ -1,301 +0,0 @@
-"""Base models using UUIDv7 for all id fields."""
-
-__package__ = "archivebox.base_models"
-
-import json
-import shutil
-from typing import TYPE_CHECKING, Any
-
-from archivebox.uuid_compat import CompactUUIDField, uuid7
-from pathlib import Path
-
-from django.db import models
-from django.db.models import F
-from django.db import transaction
-from django.db.models.signals import pre_delete
-from django.utils import timezone
-from django.contrib.auth import get_user_model
-from django.urls import reverse_lazy
-from django.conf import settings
-
-from django_stubs_ext.db.models import TypedModelMeta
-
-from archivebox.config import CONSTANTS
-
-
-if TYPE_CHECKING:
-    from archivebox.config.common import ArchiveBoxBaseConfig
-
-
-def normalize_config_json_values(config: Any) -> Any:
-    if not isinstance(config, dict):
-        return config
-
-    normalized = dict(config)
-    for key, value in list(normalized.items()):
-        if not isinstance(value, str) or len(value) < 2:
-            continue
-        if value[:1] != '"' or value[-1:] != '"':
-            continue
-        try:
-            decoded = json.loads(value)
-        except ValueError:
-            continue
-        if isinstance(decoded, str):
-            normalized[key] = decoded
-    return normalized
-
-
-def get_or_create_system_user_pk(username="system"):
-    User = get_user_model()
-    # If there's exactly one superuser, use that for all system operations
-    if User.objects.filter(is_superuser=True).count() == 1:
-        return User.objects.filter(is_superuser=True).values_list("pk", flat=True)[0]
-    # Otherwise get or create the system user
-    user, _ = User.objects.get_or_create(
-        username=username,
-        defaults={"is_staff": True, "is_superuser": True, "email": "", "password": "!"},
-    )
-    return user.pk
-
-
-class ModelWithUUID(models.Model):
-    id = CompactUUIDField(primary_key=True, default=uuid7, editable=False, unique=True)
-    created_at = models.DateTimeField(default=timezone.now, db_index=True)
-    modified_at = models.DateTimeField(auto_now=True)
-    created_by = models.ForeignKey(
-        settings.AUTH_USER_MODEL,
-        on_delete=models.CASCADE,
-        default=get_or_create_system_user_pk,
-        null=False,
-        db_index=True,
-    )
-
-    class Meta(TypedModelMeta):
-        abstract = True
-
-    def __str__(self) -> str:
-        return f"[{self.id}] {self.__class__.__name__}"
-
-    @property
-    def admin_change_url(self) -> str:
-        return f"/admin/{self._meta.app_label}/{self._meta.model_name}/{self.pk}/change/"
-
-    @property
-    def api_url(self) -> str:
-        return str(reverse_lazy("api-1:get_any", args=[self.id]))
-
-    @property
-    def api_docs_url(self) -> str:
-        return f"/api/v1/docs#/{self._meta.app_label.title()}%20Models/api_v1_{self._meta.app_label}_get_{self._meta.db_table}"
-
-
-class ModelWithNotes(models.Model):
-    """Mixin for models with a notes field."""
-
-    notes = models.TextField(blank=True, null=False, default="")
-
-    class Meta(TypedModelMeta):
-        abstract = True
-
-
-class ModelWithHealthStats(models.Model):
-    """Mixin for models with health tracking fields."""
-
-    num_uses_failed = models.PositiveIntegerField(default=0)
-    num_uses_succeeded = models.PositiveIntegerField(default=0)
-
-    class Meta(TypedModelMeta):
-        abstract = True
-
-    @property
-    def admin_change_url(self) -> str:
-        return f"/admin/{self._meta.app_label}/{self._meta.model_name}/{self.pk}/change/"
-
-    @property
-    def health(self) -> int:
-        total = max(self.num_uses_failed + self.num_uses_succeeded, 1)
-        return round((self.num_uses_succeeded / total) * 100)
-
-    def increment_health_stats(self, success: bool):
-        """Atomically increment success or failure counter using F() expression."""
-        field = "num_uses_succeeded" if success else "num_uses_failed"
-        type(self).objects.filter(pk=self.pk).update(
-            **{
-                field: F(field) + 1,
-                "modified_at": timezone.now(),
-            },
-        )
-
-
-class ModelWithConfig(models.Model):
-    """Mixin for models with a JSON config field."""
-
-    _runtime_config: "ArchiveBoxBaseConfig"
-    config = models.JSONField(default=dict, null=True, blank=True, editable=True)
-
-    class Meta(TypedModelMeta):
-        abstract = True
-
-    def save(self, *args, **kwargs):
-        normalized_config = normalize_config_json_values(self.config)
-        if normalized_config != self.config:
-            self.config = normalized_config
-            update_fields = kwargs.get("update_fields")
-            if update_fields is not None:
-                kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "config"]))
-        super().save(*args, **kwargs)
-
-
-class ModelWithDeleteAfter(models.Model):
-    delete_after_final_statuses: tuple[str, ...] = ()
-    delete_at = models.DateTimeField(default=None, null=True, blank=True, db_index=True)
-
-    class Meta(TypedModelMeta):
-        abstract = True
-
-    def save(self, *args, **kwargs):
-        update_fields = kwargs.get("update_fields")
-        if self.delete_at is None:
-            self.set_delete_at_from_config()
-            if self.delete_at is not None and update_fields is not None:
-                kwargs["update_fields"] = tuple(dict.fromkeys([*update_fields, "delete_at"]))
-        super().save(*args, **kwargs)
-
-    def get_delete_after_config_value(self):
-        from archivebox.config.common import get_config
-
-        return get_config(include_machine=False, resolve_plugins=False).DELETE_AFTER
-
-    def set_delete_at_from_config(self, config_value=None) -> bool:
-        if self.delete_at is not None:
-            return False
-
-        from archivebox.config.common import parse_delete_after
-
-        duration = parse_delete_after(self.get_delete_after_config_value() if config_value is None else config_value)
-        if duration is None:
-            return False
-
-        self.delete_at = (self.created_at or timezone.now()) + duration
-        return True
-
-    @classmethod
-    def missing_delete_at_candidates(cls):
-        return cls.objects.none()
-
-    @classmethod
-    def delete_expired(cls, *, batch_size: int = 100, backfill_missing: bool = True) -> int:
-        if backfill_missing:
-            missing_delete_at = list(cls.missing_delete_at_candidates().order_by("created_at", "pk")[:batch_size])
-            for obj in missing_delete_at:
-                if obj.set_delete_at_from_config():
-                    cls.objects.filter(pk=obj.pk, delete_at__isnull=True).update(
-                        delete_at=obj.delete_at,
-                        modified_at=timezone.now(),
-                    )
-
-        # Keep the expiration sweep anchored on delete_at. Some large tables
-        # have millions of final-status rows but almost no retained rows; adding
-        # the status filter before SQLite has narrowed by delete_at can make the
-        # runner scan the hot status index before it claims new work.
-        due_pks = list(
-            cls.objects.filter(delete_at__isnull=False, delete_at__lte=timezone.now())
-            .order_by("delete_at", "pk")
-            .values_list("pk", flat=True)[:batch_size],
-        )
-        if not due_pks:
-            return 0
-
-        queryset = cls.objects.filter(pk__in=due_pks)
-        if cls.delete_after_final_statuses:
-            queryset = queryset.filter(status__in=cls.delete_after_final_statuses)
-
-        count = 0
-        expired = list(queryset.order_by("delete_at", "pk"))
-        for obj in expired:
-            obj.delete()
-            count += 1
-        return count
-
-
-class ModelWithOutputDir(ModelWithUUID):
-    class Meta(ModelWithUUID.Meta):
-        abstract = True
-
-    _delete_signal_registered = False
-
-    def save(self, *args, **kwargs):
-        super().save(*args, **kwargs)
-        output_dir = Path(self.output_dir)
-        # Avoid holding SQLite write transactions open across slow filesystem work.
-        transaction.on_commit(lambda: output_dir.mkdir(parents=True, exist_ok=True))
-        # Note: index.json is deprecated, models should use write_index_jsonl() for full data
-
-    @property
-    def output_dir_parent(self) -> str:
-        return f"{self._meta.model_name}s"
-
-    @property
-    def output_dir_name(self) -> str:
-        return str(self.id)
-
-    @property
-    def output_dir_str(self) -> str:
-        return f"{self.output_dir_parent}/{self.output_dir_name}"
-
-    @property
-    def output_dir(self) -> Path:
-        raise NotImplementedError(f"{self.__class__.__name__} must implement output_dir property")
-
-    def output_paths_for_delete(self) -> tuple[Path, ...]:
-        return (Path(self.output_dir),)
-
-    @classmethod
-    def validate_output_paths_for_delete(cls, paths) -> tuple[Path, ...]:
-        data_dir = CONSTANTS.DATA_DIR.resolve()
-        safe_paths = []
-        for raw_path in paths:
-            path = Path(raw_path)
-            is_safe = False
-            for candidate in (path.absolute(), path.resolve()):
-                try:
-                    candidate.relative_to(data_dir)
-                    is_safe = True
-                    break
-                except ValueError:
-                    continue
-            if not is_safe:
-                raise ValueError(f"Refusing to delete output path outside DATA_DIR: {path}")
-            safe_paths.append(path)
-        return tuple(safe_paths)
-
-    @classmethod
-    def delete_output_paths(cls, paths) -> None:
-        for path in cls.validate_output_paths_for_delete(paths):
-            if path.is_symlink() or path.is_file():
-                path.unlink(missing_ok=True)
-            elif path.is_dir():
-                shutil.rmtree(path, ignore_errors=True)
-
-    def schedule_delete_cleanup(self, *, using: str | None = None) -> None:
-        """Capture output paths before DB deletion and remove them after commit."""
-        paths = self.validate_output_paths_for_delete(self.output_paths_for_delete())
-        transaction.on_commit(lambda: self.delete_output_paths(paths), using=using)
-
-    @classmethod
-    def register_delete_signal(cls) -> None:
-        if cls._delete_signal_registered:
-            return
-
-        def schedule_output_dir_cleanup(sender, instance, using, **kwargs):
-            if not isinstance(instance, ModelWithOutputDir):
-                return
-            instance.schedule_delete_cleanup(using=using)
-
-        pre_delete.connect(
-            schedule_output_dir_cleanup,
-            dispatch_uid="archivebox.output_dir_cleanup_on_delete",
-            weak=False,
-        )
-        cls._delete_signal_registered = True
@@ -1,224 +0,0 @@
-__package__ = "archivebox.cli"
-__command__ = "archivebox"
-import os
-import sys
-from importlib import import_module
-
-import rich_click as click
-from rich.console import Console
-
-from archivebox.config.version import VERSION
-
-STDERR = Console(stderr=True)
-
-
-if "--debug" in sys.argv:
-    os.environ["DEBUG"] = "True"
-    sys.argv.remove("--debug")
-
-# Universal `--init` flag: when passed to ANY subcommand (e.g. `archivebox server --init`,
-# `archivebox add --init`, `archivebox shell --init`), run a `quick` archivebox init before
-# the subcommand executes. `--quick-init` is the old server spelling. Strip either from
-# argv here so each subcommand's own click parser never sees it. Ignored for `help` and
-# `init` themselves.
-if "--init" in sys.argv or "--quick-init" in sys.argv:
-    sys.argv = [arg for arg in sys.argv if arg not in ("--init", "--quick-init")]
-    os.environ["ARCHIVEBOX_WANTS_INIT"] = "1"
-
-
-class ArchiveBoxGroup(click.Group):
-    """lazy loading click group for archivebox commands"""
-
-    meta_commands = {
-        "help": "archivebox.cli.archivebox_help.main",
-        "version": "archivebox.cli.archivebox_version.main",
-        "mcp": "archivebox.cli.archivebox_mcp.main",
-        "oneshot": "archivebox.cli.archivebox_oneshot.main",
-    }
-    setup_commands = {
-        "init": "archivebox.cli.archivebox_init.main",
-        "install": "archivebox.cli.archivebox_install.main",
-    }
-    # Model commands (CRUD operations via subcommands)
-    model_commands = {
-        "crawl": "archivebox.cli.archivebox_crawl.main",
-        "snapshot": "archivebox.cli.archivebox_snapshot.main",
-        "archiveresult": "archivebox.cli.archivebox_archiveresult.main",
-        "tag": "archivebox.cli.archivebox_tag.main",
-        "binary": "archivebox.cli.archivebox_binary.main",
-        "process": "archivebox.cli.archivebox_process.main",
-        "machine": "archivebox.cli.archivebox_machine.main",
-        "persona": "archivebox.cli.archivebox_persona.main",
-    }
-    archive_commands = {
-        # High-level commands
-        "add": "archivebox.cli.archivebox_add.main",
-        "extract": "archivebox.cli.archivebox_extract.main",
-        "list": "archivebox.cli.archivebox_list.main",
-        "remove": "archivebox.cli.archivebox_remove.main",
-        "run": "archivebox.cli.archivebox_run.main",
-        "update": "archivebox.cli.archivebox_update.main",
-        "status": "archivebox.cli.archivebox_status.main",
-        "search": "archivebox.cli.archivebox_search.main",
-        "config": "archivebox.cli.archivebox_config.main",
-        "schedule": "archivebox.cli.archivebox_schedule.main",
-        "server": "archivebox.cli.archivebox_server.main",
-        "shell": "archivebox.cli.archivebox_shell.main",
-        "manage": "archivebox.cli.archivebox_manage.main",
-        # Introspection commands
-        "pluginmap": "archivebox.cli.archivebox_pluginmap.main",
-    }
-    all_subcommands = {
-        **meta_commands,
-        **setup_commands,
-        **model_commands,
-        **archive_commands,
-    }
-    renamed_commands = {
-        "setup": "install",
-        "import": "add",
-        "archive": "add",
-    }
-
-    @classmethod
-    def get_canonical_name(cls, cmd_name):
-        return cls.renamed_commands.get(cmd_name, cmd_name)
-
-    @classmethod
-    def _needs_django_for_lazy_import(cls, cmd_name: str) -> bool:
-        wants_help = any(arg in ("-h", "--help", "--version") for arg in sys.argv[1:])
-        return not wants_help and (cmd_name in cls.archive_commands or cmd_name in cls.model_commands)
-
-    @classmethod
-    def _setup_django_for_lazy_import(cls, cmd_name: str) -> None:
-        if not cls._needs_django_for_lazy_import(cmd_name):
-            return
-
-        from django.apps import apps
-
-        if apps.ready:
-            return
-
-        from archivebox.config.django import setup_django
-
-        setup_django()
-
-    def get_command(self, ctx, cmd_name):
-        # handle renamed commands
-        if cmd_name in self.renamed_commands:
-            new_name = self.renamed_commands[cmd_name]
-            STDERR.print(
-                f" [violet]Hint:[/violet] `archivebox {cmd_name}` has been renamed to `archivebox {new_name}`",
-            )
-            cmd_name = new_name
-            ctx.invoked_subcommand = cmd_name
-
-        # handle lazy loading of commands
-        if cmd_name in self.all_subcommands:
-            self._setup_django_for_lazy_import(cmd_name)
-            return self._lazy_load(cmd_name)
-
-        # fall-back to using click's default command lookup
-        return super().get_command(ctx, cmd_name)
-
-    @classmethod
-    def _lazy_load(cls, cmd_name_or_path):
-        import_path = cls.all_subcommands.get(cmd_name_or_path)
-        if import_path is None:
-            import_path = cmd_name_or_path
-        modname, funcname = import_path.rsplit(".", 1)
-
-        mod = import_module(modname)
-        func = vars(mod)[funcname]
-
-        if func.__doc__ is None:
-            raise ValueError(f"lazy loading of {import_path} failed - no docstring found on method")
-
-        return func
-
-
-@click.group(cls=ArchiveBoxGroup, invoke_without_command=True)
-@click.option("--help", "-h", is_flag=True, help="Show help")
-@click.version_option(VERSION, "-v", "--version", package_name="archivebox", message="%(version)s")
-@click.pass_context
-def cli(ctx, help=False):
-    """ArchiveBox: The self-hosted internet archive"""
-
-    subcommand = ArchiveBoxGroup.get_canonical_name(ctx.invoked_subcommand)
-
-    # if --help is passed or no subcommand is given, show custom help message
-    if help or ctx.invoked_subcommand is None:
-        ctx.invoke(ctx.command.get_command(ctx, "help"))
-
-    # if the subcommand is in archive_commands or model_commands,
-    # then we need to set up the django environment and check that we're in a valid data folder
-    wants_help = any(arg in ("-h", "--help", "--version") for arg in sys.argv[1:])
-    if not wants_help and (subcommand in ArchiveBoxGroup.archive_commands or subcommand in ArchiveBoxGroup.model_commands):
-        try:
-            if subcommand == "server":
-                run_in_debug = "--reload" in sys.argv or os.environ.get("DEBUG") in ("1", "true", "True", "TRUE", "yes")
-                if run_in_debug:
-                    os.environ["ARCHIVEBOX_RUNSERVER"] = "1"
-                    if "--reload" in sys.argv:
-                        os.environ["ARCHIVEBOX_AUTORELOAD"] = "1"
-
-            from archivebox.config.django import setup_django
-            from archivebox.misc.checks import check_data_folder, check_migrations
-
-            setup_django()
-            if os.environ.get("ARCHIVEBOX_WANTS_INIT") == "1" and subcommand not in ("init", "help"):
-                # Universal `--init` was passed: build/upgrade the data folder before
-                # the regular preflight runs, so it succeeds on a fresh dir and an
-                # out-of-date schema both. Drop the env var afterwards so spawned
-                # subprocesses (supervisord workers, daphne, runner, etc.) inherit
-                # a clean env and don't re-trigger init in every child.
-                from archivebox.cli.archivebox_init import init as archivebox_init
-
-                archivebox_init(quick=True)
-                os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
-            check_data_folder()
-            if subcommand != "update":
-                check_migrations(auto_apply=True)
-        except Exception as e:
-            STDERR.print(f"[red][X] Error setting up Django or checking data folder: {e}[/red]")
-            if subcommand not in ("manage", "shell"):  # not all management commands need django to be setup beforehand
-                raise
-
-
-def main(args=None, prog_name=None):
-    from archivebox.config.constants import CONSTANTS
-
-    os.environ.setdefault("ABX_PLUGINS_DIR", str(CONSTANTS.USER_PLUGINS_DIR))
-
-    # show `docker run archivebox xyz` in help messages if running in docker
-    IN_DOCKER = os.environ.get("IN_DOCKER", False) in ("1", "true", "True", "TRUE", "yes")
-    IS_TTY = sys.stdin.isatty()
-    prog_name = prog_name or (f"docker compose run{'' if IS_TTY else ' -T'} archivebox" if IN_DOCKER else "archivebox")
-
-    previous_unraisablehook = sys.unraisablehook
-
-    def ignore_shutdown_unraisable(unraisable):
-        if isinstance(unraisable.exc_value, (KeyboardInterrupt, SystemExit)):
-            return
-        previous_unraisablehook(unraisable)
-
-    sys.unraisablehook = ignore_shutdown_unraisable
-    try:
-        cli(args=args, prog_name=prog_name, standalone_mode=False)
-    except click.Abort:
-        STDERR.print("\n[red][X] Got CTRL+C. Exiting...[/red]")
-        raise SystemExit(130) from None
-    except click.ClickException as err:
-        err.show()
-        raise SystemExit(err.exit_code) from None
-    except click.exceptions.Exit as err:
-        raise SystemExit(err.exit_code) from None
-    except KeyboardInterrupt:
-        STDERR.print("\n[red][X] Got CTRL+C. Exiting...[/red]")
-        raise SystemExit(130) from None
-    finally:
-        sys.unraisablehook = previous_unraisablehook
-
-
-if __name__ == "__main__":
-    main()
@@ -1,423 +0,0 @@
-#!/usr/bin/env python3
-from __future__ import annotations
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox add"
-
-import sys
-import os
-from pathlib import Path
-
-from typing import Any, TYPE_CHECKING
-
-import rich_click as click
-
-from archivebox.config import CONSTANTS
-from archivebox.misc.util import enforce_types, docstring
-from archivebox.misc.util import parse_filesize_to_bytes
-
-
-if TYPE_CHECKING:
-    from django.db.models import QuerySet
-    from archivebox.crawls.models import Crawl
-    from archivebox.core.models import Snapshot
-
-
-def _collect_input_urls(args: tuple[str, ...], *, parser: str = "auto") -> list[str]:
-    from archivebox.misc.jsonl import read_args_or_stdin
-    from archivebox.misc.util import validate_url
-
-    urls: list[str] = []
-    for record in read_args_or_stdin(args):
-        url = record.get("url")
-        if isinstance(url, str) and url:
-            try:
-                urls.append(validate_url(url))
-            except ValueError as err:
-                raise click.BadParameter(str(err), param_hint="URL") from err
-
-        urls_field = record.get("urls")
-        if isinstance(urls_field, str):
-            for line in urls_field.splitlines():
-                line = line.strip()
-                if line and not line.startswith("#"):
-                    try:
-                        urls.append(validate_url(line))
-                    except ValueError as err:
-                        raise click.BadParameter(str(err), param_hint="URL") from err
-
-    return urls
-
-
-@enforce_types
-def add(
-    urls: str | list[str],
-    snapshot_ids: list[str] | None = None,
-    depth: int | str = 0,
-    max_urls: int = 0,
-    crawl_max_size: int | str = 0,
-    crawl_timeout: int = 0,
-    snapshot_max_size: int | str = 0,
-    crawl_max_concurrent_snapshots: int | None = None,
-    tag: str = "",
-    url_allowlist: str = "",
-    url_denylist: str = "",
-    parser: str = "auto",
-    plugins: str = "",
-    persona: str = "Default",
-    index_only: bool = False,
-    bg: bool = False,
-    created_by_id: int | None = None,
-    config: dict[str, Any] | None = None,
-) -> tuple[Crawl, QuerySet[Snapshot]]:
-    """Add a URL list or imported URL document to a new Crawl."""
-
-    from rich import print
-
-    depth = int(depth)
-    max_urls = int(max_urls or 0)
-    crawl_max_size = parse_filesize_to_bytes(crawl_max_size)
-    crawl_timeout = int(crawl_timeout or 0)
-    snapshot_max_size = parse_filesize_to_bytes(snapshot_max_size)
-    from archivebox.config.permissions import USER, HOSTNAME
-    from archivebox.config.common import get_config
-
-    config_overrides = dict(config or {})
-    runtime_config = get_config()
-    crawl_max_concurrent_snapshots_override = crawl_max_concurrent_snapshots is not None
-    if crawl_max_concurrent_snapshots is None:
-        crawl_max_concurrent_snapshots = runtime_config.CRAWL_MAX_CONCURRENT_SNAPSHOTS
-    crawl_max_concurrent_snapshots = int(crawl_max_concurrent_snapshots)
-
-    if depth not in (0, 1, 2, 3, 4):
-        raise ValueError("Depth must be 0-4")
-    if max_urls < 0:
-        raise ValueError("max_urls must be >= 0")
-    if crawl_max_size < 0:
-        raise ValueError("crawl_max_size must be >= 0")
-    if crawl_timeout < 0:
-        raise ValueError("crawl_timeout must be >= 0")
-    if snapshot_max_size < 0:
-        raise ValueError("snapshot_max_size must be >= 0")
-    if crawl_max_concurrent_snapshots < 1:
-        raise ValueError("crawl_max_concurrent_snapshots must be >= 1")
-
-    # import models once django is set up
-    from archivebox.crawls.models import Crawl
-    from archivebox.base_models.models import get_or_create_system_user_pk
-    from archivebox.personas.models import Persona
-    from archivebox.misc.logging_util import printable_filesize
-    from archivebox.misc.system import get_dir_size
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-    from django.utils import timezone
-
-    created_by_id = created_by_id or get_or_create_system_user_pk()
-    started_at = timezone.now()
-
-    if isinstance(urls, str):
-        source_text = urls
-    else:
-        from archivebox.misc.util import validate_url
-
-        source_text = "\n".join(validate_url(str(url)) for url in urls)
-
-    # 2. Create a new Crawl with inline URLs
-    # Foreground add must claim runner ownership before publishing runnable
-    # work. Otherwise an existing server runner can lease the new Crawl in the
-    # gap between Crawl.objects.create() and current_command(), then be killed
-    # by the newer add process halfway through its first Snapshot hook.
-    command = None
-    if not bg and not index_only:
-        from archivebox.machine.models import Process
-        from archivebox.core.takeover_util import current_command
-
-        command = current_command(Process.TypeChoices.ADD, data_dir=CONSTANTS.DATA_DIR)
-
-    cli_args = [*sys.argv]
-    if cli_args[0].lower().endswith("archivebox"):
-        cli_args[0] = "archivebox"
-    cmd_str = " ".join(cli_args)
-
-    timestamp = timezone.now().strftime("%Y-%m-%d__%H-%M-%S")
-
-    persona_name = (persona or "Default").strip() or "Default"
-    plugins = plugins or ""
-    persona_obj = Persona.get_or_create_named(persona_name)
-    persona_obj.ensure_dirs()
-    effective_persona_config = get_config(persona=persona_obj)
-
-    crawl_config = {
-        "PERMISSIONS": str(effective_persona_config.PERMISSIONS),
-        **({"INDEX_ONLY": True} if index_only else {}),
-        **({"PLUGINS": plugins} if plugins else {}),
-        **(
-            {"CRAWL_MAX_CONCURRENT_SNAPSHOTS": crawl_max_concurrent_snapshots}
-            if crawl_max_concurrent_snapshots_override
-            and crawl_max_concurrent_snapshots != int(effective_persona_config.CRAWL_MAX_CONCURRENT_SNAPSHOTS)
-            else {}
-        ),
-        **({"CRAWL_MAX_URLS": max_urls} if max_urls else {}),
-        **({"CRAWL_MAX_SIZE": crawl_max_size} if crawl_max_size else {}),
-        **({"CRAWL_TIMEOUT": crawl_timeout} if crawl_timeout else {}),
-        **({"SNAPSHOT_MAX_SIZE": snapshot_max_size} if snapshot_max_size else {}),
-        **({"PARSER": parser} if parser != "auto" else {}),
-        **({"URL_ALLOWLIST": url_allowlist} if url_allowlist else {}),
-        **({"URL_DENYLIST": url_denylist} if url_denylist else {}),
-    }
-    # Caller-supplied overrides (e.g. {"ONLY_NEW": False}) are the highest
-    # priority for crawl-frozen keys. Runtime-derived execution keys are
-    # stripped by Crawl.save() and rederived when hooks run.
-    crawl_config.update(config_overrides)
-
-    # Crawl.urls is the user's submitted source, not a derived work queue for
-    # this add path. Keeping it byte-for-byte readable is what lets API/UI/CLI
-    # callers audit or resume imports without losing RSS/Netscape/JSON metadata
-    # that is not representable as one plain URL per line.
-    try:
-        crawl = Crawl.objects.create(
-            urls=source_text,
-            max_depth=depth,
-            tags_str=tag,
-            persona_id=persona_obj.id,
-            label=f"{USER}@{HOSTNAME} $ {cmd_str} [{timestamp}]",
-            created_by_id=created_by_id,
-            status=Crawl.StatusChoices.QUEUED,
-            retry_at=None if index_only else timezone.now(),
-            config=crawl_config,
-        )
-        first_url = crawl.get_urls_list()[0] if crawl.get_urls_list() else ""
-    except BaseException:
-        if command is not None:
-            command.mark_exited(exit_code=1)
-        raise
-    if command is not None:
-        command.url = first_url
-        command.save(update_fields=["url", "modified_at"])
-
-    print(f"[green]\\[+] Created Crawl {crawl.id} with max_depth={depth}[/green]")
-    print(f"    [dim]First URL: {first_url}[/dim]")
-
-    # The runner parses Crawl.urls after claiming the Crawl, then persists the
-    # resulting depth-0 Snapshot facts. Recursive discoveries start at depth 1.
-
-    if index_only:
-        print("[yellow]\\[*] Index-only mode - URLs queued, runner not started[/yellow]")
-        return crawl, crawl.snapshot_set.none()
-
-    # 5. Start the crawl runner to process the queue
-    #    The runner will:
-    #    - Process Crawl -> create Snapshots from all URLs
-    #    - Process Snapshots -> run extractors
-    #    - Parser extractors discover new URLs -> create child Snapshots
-    #    - Repeat until max_depth reached
-
-    if bg:
-        # Background mode: just queue work and return (background runner via server will pick it up).
-        print(
-            "[yellow]\\[*] URLs queued. The background runner will process them (run `archivebox server` or `archivebox run --daemon` if not already running).[/yellow]",
-        )
-        from archivebox.services.runner import ensure_background_runner
-
-        ensure_background_runner()
-    else:
-        # Foreground mode: run full crawl runner until all work is done
-        print("[green]\\[*] Starting crawl runner to process crawl...[/green]")
-        from archivebox.core.takeover_util import command_owns_foreground_runner, standby_until_foreground_runner_needed
-        from archivebox.workers.supervisord_util import get_existing_supervisord_process, run_runner_worker, stop_own_supervisord_process
-
-        assert command is not None
-        exit_code = 0
-
-        def crawl_is_complete() -> bool:
-            crawl.refresh_from_db(fields=["status"])
-            return crawl.status not in crawl.RUNNABLE_STATES
-
-        try:
-            try:
-                with foreground_shutdown_signals(first_signal_message=None), foreground_parent_watchdog():
-                    while True:
-                        standby = standby_until_foreground_runner_needed(
-                            command,
-                            data_dir=CONSTANTS.DATA_DIR,
-                            work_is_complete=crawl_is_complete,
-                        )
-                        if standby["work_completed"]:
-                            break
-                        exit_code = run_runner_worker(
-                            ["--crawl-id", str(crawl.id)],
-                            name=f"worker_runner_add_{os.getpid()}",
-                            interactive_interrupts=True,
-                            config=get_config(crawl=crawl),
-                        )
-                        crawl.refresh_from_db(fields=["status", "retry_at"])
-                        if exit_code == 0 and crawl.status == crawl.StatusChoices.SEALED:
-                            break
-                        # A shared supervisord can be stopped by another
-                        # foreground owner (for example `archivebox server`
-                        # shutting down) and stop the one-shot add runner while
-                        # this add's crawl is still runnable. Keep the add
-                        # process alive so it can re-enter the normal
-                        # foreground-runner ownership loop and finish its crawl.
-                        supervisor_gone = exit_code == 1 and get_existing_supervisord_process(quiet=True) is None
-                        if crawl.status in crawl.RUNNABLE_STATES and (exit_code in (0, 130, 143) or supervisor_gone):
-                            continue
-                        if exit_code == 0:
-                            break
-                        if not command_owns_foreground_runner(command, data_dir=CONSTANTS.DATA_DIR):
-                            continue
-                        raise SystemExit(exit_code)
-            except KeyboardInterrupt:
-                exit_code = 130
-                print("\n[red][X] archivebox add interrupted.[/red]")
-                print("[yellow]Hint: resume this crawl with:[/yellow]")
-                print(f"    [green]archivebox run --crawl-id={crawl.id}[/green]")
-                raise SystemExit(exit_code)
-        finally:
-            command.mark_exited(exit_code=exit_code)
-            stop_own_supervisord_process()
-
-        # Print summary for foreground runs
-        try:
-            crawl.refresh_from_db()
-            try:
-                from django.db.models import Count, Sum
-
-                totals = crawl.snapshot_set.aggregate(snapshot_count=Count("id"), total_bytes=Sum("output_size"))
-                snapshots_count = int(totals["snapshot_count"] or 0)
-                total_bytes = int(totals["total_bytes"] or 0)
-            except Exception:
-                snapshots_count = crawl.snapshot_set.count()
-                total_bytes, _, _ = get_dir_size(crawl.output_dir)
-            total_size = printable_filesize(total_bytes)
-            total_time = timezone.now() - started_at
-            total_seconds = int(total_time.total_seconds())
-            mins, secs = divmod(total_seconds, 60)
-            hours, mins = divmod(mins, 60)
-            if hours:
-                duration_str = f"{hours}h {mins}m {secs}s"
-            elif mins:
-                duration_str = f"{mins}m {secs}s"
-            else:
-                duration_str = f"{secs}s"
-
-            # Output dir relative to DATA_DIR
-            try:
-                rel_output = Path(crawl.output_dir).relative_to(CONSTANTS.DATA_DIR)
-                rel_output_str = f"./{rel_output}"
-            except Exception:
-                rel_output_str = str(crawl.output_dir)
-
-            from archivebox.core.routes_util import build_admin_url
-
-            admin_url = build_admin_url(f"/admin/crawls/crawl/{crawl.id}/change/", config=config)
-
-            print("\n[bold]crawl output saved to:[/bold]")
-            print(f"  {rel_output_str}")
-            print(f"  {admin_url}")
-            print(f"\n[bold]total urls snapshotted:[/bold] {snapshots_count}")
-            print(f"[bold]total size:[/bold] {total_size}")
-            print(f"[bold]total time:[/bold] {duration_str}")
-        except Exception:
-            # Summary is best-effort; avoid failing the command if something goes wrong
-            pass
-
-    # 6. Return the list of Snapshots in this crawl
-    snapshots = crawl.snapshot_set.all()
-    return crawl, snapshots
-
-
-@click.command()
-@click.option(
-    "--depth",
-    "-d",
-    type=click.Choice([str(i) for i in range(5)]),
-    default="0",
-    help="Recursively archive linked pages up to N hops away",
-)
-@click.option("--max-urls", type=int, default=0, help="Maximum number of URLs to snapshot for this crawl (0 = unlimited)")
-@click.option("--crawl-max-size", default="0", help="Maximum total crawl size in bytes or units like 45mb / 1gb (0 = unlimited)")
-@click.option("--crawl-timeout", type=int, default=0, help="Maximum total crawl runtime in seconds (0 = unlimited)")
-@click.option("--snapshot-max-size", default="0", help="Maximum per-snapshot size in bytes or units like 45mb / 1gb (0 = unlimited)")
-@click.option("--crawl-max-concurrent-snapshots", type=int, default=None, help="Maximum snapshots to archive concurrently within one crawl")
-@click.option("--tag", "-t", default="", help="Comma-separated list of tags to add to each snapshot e.g. tag1,tag2,tag3")
-@click.option("--url-allowlist", "--domain-allowlist", default="", help="Comma-separated URL/domain allowlist for this crawl")
-@click.option("--url-denylist", "--domain-denylist", default="", help="Comma-separated URL/domain denylist for this crawl")
-@click.option("--parser", default="auto", help="Parser for reading input URLs (auto, txt, html, rss, json, jsonl, netscape, ...)")
-@click.option("--plugins", "-p", default="", help="Comma-separated list of plugins to run e.g. title,favicon,screenshot,singlefile,...")
-@click.option("--extract", default="", help="Comma-separated plugins or output types to extract e.g. title,pdf,text/html,image,...")
-@click.option("--persona", default="Default", help="Authentication profile to use when archiving")
-@click.option(
-    "--only-new/--no-only-new",
-    "only_new",
-    default=None,
-    help="Skip URLs that already have a snapshot (default: inherit from ONLY_NEW config). "
-    "Pass --no-only-new to force re-archive of URLs that already exist.",
-)
-@click.option("--index-only", is_flag=True, help="Just add the URLs to the index without archiving them now")
-@click.option("--overwrite", is_flag=True, help="Re-archive URLs even if they already exist (alias for --no-only-new)")
-@click.option("--update", is_flag=True, help="Re-archive URLs even if they already exist (alias for --no-only-new)")
-@click.option("--bg", is_flag=True, help="Run archiving in background (queue work and return immediately)")
-@click.argument("urls", nargs=-1, type=click.Path())
-@docstring(add.__doc__)
-def main(**kwargs):
-    """Add a new URL or list of URLs to your archive"""
-
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-
-    with foreground_shutdown_signals(), foreground_parent_watchdog():
-        raw_urls = kwargs.pop("urls")
-        if raw_urls:
-            urls = _collect_input_urls(raw_urls, parser=kwargs.get("parser", "auto"))
-        elif not sys.stdin.isatty():
-            urls = sys.stdin.read()
-        else:
-            urls = []
-        if not urls:
-            raise click.UsageError("No URLs provided. Pass URLs as arguments or via stdin.")
-        if int(kwargs.get("max_urls") or 0) < 0:
-            raise click.BadParameter("max_urls must be 0 or a positive integer.", param_hint="--max-urls")
-        if int(kwargs.get("crawl_timeout") or 0) < 0:
-            raise click.BadParameter("crawl_timeout must be 0 or a positive integer.", param_hint="--crawl-timeout")
-        try:
-            kwargs["crawl_max_size"] = parse_filesize_to_bytes(kwargs.get("crawl_max_size"))
-        except ValueError as err:
-            raise click.BadParameter(str(err), param_hint="--crawl-max-size") from err
-        try:
-            kwargs["snapshot_max_size"] = parse_filesize_to_bytes(kwargs.get("snapshot_max_size"))
-        except ValueError as err:
-            raise click.BadParameter(str(err), param_hint="--snapshot-max-size") from err
-        if kwargs.get("crawl_max_concurrent_snapshots") is not None and int(kwargs["crawl_max_concurrent_snapshots"]) < 1:
-            raise click.BadParameter("crawl_max_concurrent_snapshots must be at least 1.", param_hint="--crawl-max-concurrent-snapshots")
-        if not kwargs.get("index_only"):
-            from archivebox.workers.supervisord_util import require_crawl_memory
-
-            require_crawl_memory()
-
-        # Translate --only-new/--no-only-new into a crawl config override.
-        # add() takes config overrides as a dict; no per-flag kwargs.
-        overwrite = kwargs.pop("overwrite", False)
-        update = kwargs.pop("update", False)
-        only_new = kwargs.pop("only_new", None)
-        extract = kwargs.pop("extract", "")
-        if overwrite or update:
-            only_new = False
-        if only_new is not None:
-            kwargs["config"] = {"ONLY_NEW": bool(only_new)}
-        if extract:
-            from archivebox.plugins.discovery import get_plugin_catalog
-
-            catalog = get_plugin_catalog()
-            tokens = [token.strip() for token in extract.split(",") if token.strip()]
-            plugin_names = {name.lower(): name for name in catalog}
-            selected = [plugin_names[token.lower()] for token in tokens if token.lower() in plugin_names]
-            selected += catalog.matching_output(tokens)
-            if not selected:
-                raise click.UsageError(f"No plugins found matching extract types: {extract}")
-            existing = [token.strip() for token in (kwargs.get("plugins") or "").split(",") if token.strip()]
-            kwargs["plugins"] = ",".join(dict.fromkeys([*existing, *selected]))
-
-        add(urls=urls, **kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,256 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox archiveresult <action> [args...] [--filters]
-
-Manage ArchiveResult records (plugin extraction results).
-
-Actions:
-    create  - Emit plugin extraction request records for Snapshots
-    list    - List ArchiveResults as JSONL (with optional filters)
-    update  - Update ArchiveResults from stdin JSONL
-    delete  - Delete ArchiveResults from stdin JSONL
-
-Examples:
-    # Emit extraction requests; `archivebox run` schedules their parent snapshots
-    archivebox snapshot list --status=queued | archivebox archiveresult create
-    archivebox archiveresult create --plugin=screenshot --snapshot-id=<uuid>
-
-    # List with filters
-    archivebox archiveresult list --status=failed
-    archivebox archiveresult list --plugin=screenshot --status=succeeded
-
-    # Delete
-    archivebox archiveresult list --plugin=singlefile | archivebox archiveresult delete --yes
-
-    # Re-run failed extractions
-    archivebox archiveresult list --status=failed | archivebox run
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox archiveresult"
-
-import sys
-
-import rich_click as click
-from rich import print as rprint
-
-from archivebox.cli.cli_util import apply_filters
-
-
-def build_archiveresult_request(snapshot_id: str, plugin: str, hook_name: str = "", status: str = "queued") -> dict:
-    return {
-        "type": "ArchiveResult",
-        "snapshot_id": str(snapshot_id),
-        "plugin": plugin,
-        "hook_name": hook_name,
-        "status": status,
-    }
-
-
-# =============================================================================
-# CREATE
-# =============================================================================
-
-
-def create_archiveresults(
-    snapshot_id: str | None = None,
-    plugin: str | None = None,
-    status: str = "queued",
-) -> int:
-    """
-    Create ArchiveResult request records for Snapshots.
-
-    Reads Snapshot records from stdin and emits ArchiveResult request JSONL.
-    Pass-through: Non-Snapshot/ArchiveResult records are output unchanged.
-    If --plugin is specified, only emits requests for that plugin.
-    Otherwise, emits requests for all enabled snapshot hooks.
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
-    from archivebox.config.common import get_config
-    from archivebox.plugins.hooks import discover_hooks
-    from archivebox.misc.jsonl import read_stdin, write_record, TYPE_SNAPSHOT
-    from archivebox.core.models import Snapshot
-
-    is_tty = sys.stdout.isatty()
-
-    # If snapshot_id provided directly, use that
-    if snapshot_id:
-        try:
-            snapshots = [Snapshot.objects.get(id=snapshot_id)]
-            pass_through_records = []
-        except Snapshot.DoesNotExist:
-            rprint(f"[red]Snapshot not found: {snapshot_id}[/red]", file=sys.stderr)
-            return 1
-    else:
-        # Read from stdin
-        records = list(read_stdin())
-        if not records:
-            rprint("[yellow]No Snapshot records provided via stdin[/yellow]", file=sys.stderr)
-            return 1
-
-        # Separate snapshot records from pass-through records
-        snapshot_ids = []
-        pass_through_records = []
-
-        for record in records:
-            record_type = record.get("type", "")
-
-            if record_type:
-                pass_through_records.append(record)
-            if record.get("id") and record_type in ("", TYPE_SNAPSHOT):
-                snapshot_ids.append(record["id"])
-
-        # Output pass-through records first
-        if not is_tty:
-            for record in pass_through_records:
-                write_record(record)
-
-        if not snapshot_ids:
-            if pass_through_records:
-                rprint(f"[dim]Passed through {len(pass_through_records)} records, no new snapshots to process[/dim]", file=sys.stderr)
-                return 0
-            rprint("[yellow]No valid Snapshot IDs in input[/yellow]", file=sys.stderr)
-            return 1
-
-        snapshots = list(Snapshot.objects.filter(id__in=snapshot_ids))
-
-    if not snapshots:
-        rprint("[yellow]No matching snapshots found[/yellow]", file=sys.stderr)
-        return 0 if pass_through_records else 1
-
-    created_count = 0
-    for snapshot in snapshots:
-        config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
-        hooks = [
-            hook
-            for hook in discover_hooks("Snapshot", filter_disabled=not plugin, config=config)
-            if not plugin or hook.parent.name == plugin
-        ]
-        for hook_path in hooks:
-            hook_name = hook_path.stem
-            plugin_name = hook_path.parent.name
-            if not is_tty:
-                write_record(build_archiveresult_request(snapshot.id, plugin_name, hook_name=hook_name, status=status))
-            created_count += 1
-
-    rprint(f"[green]Created {created_count} extraction request records[/green]", file=sys.stderr)
-    return 0
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_archiveresults(
-    status: str | None = None,
-    plugin: str | None = None,
-    snapshot_id: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """List archive results as JSONL, or formatted rows in a terminal."""
-    from archivebox.core.models import ArchiveResult
-    from archivebox.cli.cli_util import list_records, format_status
-
-    queryset = apply_filters(
-        ArchiveResult.objects.order_by("-start_ts"),
-        {"status": status, "plugin": plugin, "snapshot_id": snapshot_id},
-        limit=limit,
-    )
-    return list_records(
-        queryset,
-        plural="archive results",
-        render=lambda result: f"{format_status(result.status, 10)} {result.plugin:15} [dim]{result.id}[/dim] {result.snapshot.url[:40]}",
-    )
-
-
-# =============================================================================
-# UPDATE
-# =============================================================================
-
-
-def update_archiveresults(status: str | None = None) -> int:
-    """Apply supplied fields to each JSONL-selected ArchiveResult."""
-    from archivebox.core.models import ArchiveResult
-    from archivebox.cli.cli_util import update_records
-
-    def update(archiveresult):
-        if status:
-            archiveresult.status = status
-        archiveresult.save()
-
-    return update_records(ArchiveResult, update, plural="archive results")
-
-
-# =============================================================================
-# DELETE
-# =============================================================================
-
-
-def delete_archiveresults(yes: bool = False, dry_run: bool = False) -> int:
-    """Delete archive results selected by stdin JSONL; --yes confirms, --dry-run previews."""
-    from archivebox.cli.cli_util import delete_records
-    from archivebox.core.models import ArchiveResult
-
-    return delete_records(
-        ArchiveResult,
-        label="archive result",
-        plural="archive results",
-        preview=lambda obj: f"[dim]{obj.id}[/dim] {obj.plugin} {obj.snapshot.url[:40]}",
-        yes=yes,
-        dry_run=dry_run,
-        preview_limit=10,
-    )
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage ArchiveResult records (plugin extraction results)."""
-    pass
-
-
-@main.command("create")
-@click.option("--snapshot-id", help="Snapshot ID to create results for")
-@click.option("--plugin", "-p", help="Plugin name (e.g., screenshot, singlefile)")
-@click.option("--status", "-s", default="queued", help="Initial status (default: queued)")
-def create_cmd(**kwargs):
-    """Emit Snapshot plugin extraction requests as JSONL."""
-    sys.exit(create_archiveresults(**kwargs))
-
-
-@main.command("list")
-@click.option("--status", "-s", help="Filter by status (queued, started, succeeded, failed, skipped)")
-@click.option("--plugin", "-p", help="Filter by plugin name")
-@click.option("--snapshot-id", help="Filter by snapshot ID")
-@click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List ArchiveResults as JSONL."""
-    sys.exit(list_archiveresults(**kwargs))
-
-
-@main.command("update")
-@click.option("--status", "-s", help="Set status")
-def update_cmd(**kwargs):
-    """Update ArchiveResults from stdin JSONL."""
-    sys.exit(update_archiveresults(**kwargs))
-
-
-@main.command("delete")
-@click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
-@click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(**kwargs):
-    """Delete ArchiveResults from stdin JSONL."""
-    sys.exit(delete_archiveresults(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,220 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox binary <action> [args...] [--filters]
-
-Manage Binary records (detected executables like chrome, wget, etc.).
-
-Actions:
-    create  - Create/register a Binary
-    list    - List Binaries as JSONL (with optional filters)
-    update  - Update Binaries from stdin JSONL
-    delete  - Delete Binaries from stdin JSONL
-
-Examples:
-    # List all binaries
-    archivebox binary list
-
-    # List specific binary
-    archivebox binary list --name=chrome
-
-    # List binaries with specific version
-    archivebox binary list --version__icontains=120
-
-    # Delete old binary entries
-    archivebox binary list --name=chrome | archivebox binary delete --yes
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox binary"
-
-import sys
-
-import rich_click as click
-from rich import print as rprint
-
-from archivebox.cli.cli_util import apply_filters
-
-
-# =============================================================================
-# CREATE
-# =============================================================================
-
-
-def create_binary(
-    name: str,
-    abspath: str,
-    version: str = "",
-) -> int:
-    """
-    Create/register a Binary.
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
-    from archivebox.misc.jsonl import write_record
-    from archivebox.machine.models import Binary
-
-    is_tty = sys.stdout.isatty()
-
-    if not name or not abspath:
-        rprint("[red]Both --name and --abspath are required[/red]", file=sys.stderr)
-        return 1
-
-    try:
-        from archivebox.machine.models import Machine
-
-        machine = Machine.current()
-        created = not Binary.objects.filter(
-            machine=machine,
-            name=name,
-            abspath=abspath,
-            version=version,
-        ).exists()
-
-        # Mirror the Binary model lifecycle used elsewhere in the system so CLI
-        # records are owned by the current machine and can be safely piped into
-        # `archivebox run` without creating invalid rows missing machine_id.
-        binary = Binary.from_json(
-            {
-                "name": name,
-                "abspath": abspath,
-                "version": version,
-                "binproviders": "env",
-                "binprovider": "env",
-            },
-        )
-        if binary is None:
-            raise ValueError("failed to create binary record")
-
-        if not is_tty:
-            write_record(binary.to_json())
-
-        if created:
-            rprint(f"[green]Created binary: {name} at {abspath}[/green]", file=sys.stderr)
-        else:
-            rprint(f"[dim]Binary already exists: {name} at {abspath}[/dim]", file=sys.stderr)
-
-        return 0
-
-    except Exception as e:
-        rprint(f"[red]Error creating binary: {e}[/red]", file=sys.stderr)
-        return 1
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_binaries(
-    name: str | None = None,
-    abspath__icontains: str | None = None,
-    version__icontains: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """List binaries as JSONL, or formatted rows in a terminal."""
-    from archivebox.machine.models import Binary
-    from archivebox.cli.cli_util import list_records
-
-    queryset = apply_filters(
-        Binary.objects.order_by("name", "-modified_at", "-created_at"),
-        {"name": name, "abspath__icontains": abspath__icontains, "version__icontains": version__icontains},
-        limit=limit,
-    )
-    return list_records(
-        queryset,
-        plural="binaries",
-        render=lambda binary: f"[cyan]{binary.name:20}[/cyan] [dim]{binary.version:15}[/dim] {binary.abspath}",
-    )
-
-
-# =============================================================================
-# UPDATE
-# =============================================================================
-
-
-def update_binaries(version: str | None = None, abspath: str | None = None) -> int:
-    """Apply supplied fields to each JSONL-selected Binary."""
-    from archivebox.machine.models import Binary
-    from archivebox.cli.cli_util import update_records
-
-    def update(binary):
-        if version:
-            binary.version = version
-        if abspath:
-            binary.abspath = abspath
-        binary.save()
-
-    return update_records(Binary, update, plural="binaries")
-
-
-# =============================================================================
-# DELETE
-# =============================================================================
-
-
-def delete_binaries(yes: bool = False, dry_run: bool = False) -> int:
-    """Delete binaries selected by stdin JSONL; --yes confirms, --dry-run previews."""
-    from archivebox.cli.cli_util import delete_records
-    from archivebox.machine.models import Binary
-
-    return delete_records(
-        Binary,
-        label="binary",
-        plural="binaries",
-        preview=lambda obj: f"{obj.name} {obj.abspath}",
-        yes=yes,
-        dry_run=dry_run,
-    )
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage Binary records (detected executables)."""
-    pass
-
-
-@main.command("create")
-@click.option("--name", "-n", required=True, help="Binary name (e.g., chrome, wget)")
-@click.option("--abspath", "-p", required=True, help="Absolute path to binary")
-@click.option("--version", "-v", default="", help="Binary version")
-def create_cmd(**kwargs):
-    """Create/register a Binary."""
-    sys.exit(create_binary(**kwargs))
-
-
-@main.command("list")
-@click.option("--name", "-n", help="Filter by name")
-@click.option("--abspath__icontains", help="Filter by path contains")
-@click.option("--version__icontains", help="Filter by version contains")
-@click.option("--limit", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List Binaries as JSONL."""
-    sys.exit(list_binaries(**kwargs))
-
-
-@main.command("update")
-@click.option("--version", "-v", help="Set version")
-@click.option("--abspath", "-p", help="Set path")
-def update_cmd(**kwargs):
-    """Update Binaries from stdin JSONL."""
-    sys.exit(update_binaries(**kwargs))
-
-
-@main.command("delete")
-@click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
-@click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(**kwargs):
-    """Delete Binaries from stdin JSONL."""
-    sys.exit(delete_binaries(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,199 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import sys
-import toml
-import rich_click as click
-from rich import print
-from pathlib import Path
-
-from archivebox.misc.util import docstring, enforce_types
-from archivebox.misc.toml_util import CustomTOMLEncoder
-
-
-def _format_toml(config: dict) -> str:
-    return toml.dumps(config, encoder=CustomTOMLEncoder()).strip().replace("\n\n", "\n")
-
-
-@enforce_types
-def config(
-    *keys,
-    get: bool = False,
-    set: bool = False,
-    search: bool = False,
-    reset: bool = False,
-    **kwargs,
-) -> None:
-    """Get and set your ArchiveBox project configuration values"""
-
-    from archivebox.misc.checks import check_data_folder
-    from archivebox.misc.logging_util import printable_config
-    from abx_plugins.plugins.base.utils import resolve_alias
-    from archivebox.config.collection import write_config_file
-    from archivebox.config import CONSTANTS_CONFIG
-    from archivebox.config.common import ArchiveBoxConfig, get_config, get_all_configs
-    from archivebox.plugins.discovery import discover_plugin_configs
-
-    check_data_folder()
-
-    FLAT_CONFIG = get_config().as_dict()
-    runtime_derived_keys = ArchiveBoxConfig.runtime_derived_config_keys()
-    readonly_config = {key: val for key, val in CONSTANTS_CONFIG.items() if key.isupper() and isinstance(val, Path)}
-    writable_config = {key: val for key, val in FLAT_CONFIG.items() if key not in runtime_derived_keys and key not in readonly_config}
-    readable_config = {**writable_config, **readonly_config}
-    CONFIGS = get_all_configs()
-    plugin_schemas = {
-        plugin_name: schema.get("properties", {}) for plugin_name, schema in discover_plugin_configs().items() if isinstance(schema, dict)
-    }
-    core_config_aliases = {
-        alias.upper(): field_name
-        for field_name, field in ArchiveBoxConfig.model_fields.items()
-        for alias in (field_name, str(field.alias or ""))
-        if alias
-    }
-
-    config_options: list[str] = list(kwargs.pop("key=value", []) or keys or [f"{key}={val}" for key, val in kwargs.items()])
-    no_args = not (get or set or reset or config_options)
-
-    matching_config = {}
-    if search:
-        if config_options:
-            search_terms = [key.strip().lower() for key in config_options]
-
-            for existing_key, value in readable_config.items():
-                if any(term in existing_key.lower() or term in str(value).lower() for term in search_terms):
-                    matching_config[existing_key] = value
-
-            for alias, key in core_config_aliases.items():
-                if key in readable_config and any(term in alias.lower() for term in search_terms):
-                    matching_config[key] = readable_config[key]
-
-            for schema in plugin_schemas.values():
-                for key, metadata in schema.items():
-                    if key in readable_config and any(term in key.lower() or term in str(metadata).lower() for term in search_terms):
-                        matching_config[key] = readable_config[key]
-
-        print(printable_config(matching_config))
-        raise SystemExit(not matching_config)
-
-    elif get or no_args:
-        if config_options:
-            config_options = [
-                core_config_aliases.get(key.upper().strip()) or resolve_alias(key.upper().strip(), plugin_schemas) for key in config_options
-            ]
-            matching_config = {key: readable_config[key] for key in config_options if key in readable_config}
-            failed_config = [key for key in config_options if key not in readable_config]
-            if failed_config:
-                print("\n[red][X] These options failed to get[/red]")
-                print("    {}".format("\n    ".join(config_options)))
-                raise SystemExit(1)
-        else:
-            matching_config = readable_config
-
-        # Display core config sections
-        for config_section in CONFIGS.values():
-            section_header = config_section.toml_section_header
-            if isinstance(section_header, str) and section_header:
-                print(f"[grey53]\\[{section_header}][/grey53]")
-            else:
-                print("[grey53]\\[CONSTANTS]                                        # (read-only)[/grey53]")
-
-            kv_in_section = {key: val for key, val in dict(config_section).items() if key in matching_config}
-            print(_format_toml(kv_in_section))
-            print("[grey53]################################################################[/grey53]")
-
-        readonly_keys = {key: val for key, val in readonly_config.items() if key in matching_config}
-        if readonly_keys:
-            print("[grey53]\\[CONSTANTS]                                        # (read-only)[/grey53]")
-            print(_format_toml(readonly_keys))
-            print("[grey53]################################################################[/grey53]")
-
-        plugin_keys = {}
-
-        # Collect all plugin config keys
-        for schema in plugin_schemas.values():
-            for key in schema.keys():
-                if key in matching_config and key in writable_config:
-                    plugin_keys[key] = matching_config[key]
-
-        # Display all plugin config in single [PLUGINS] section
-        if plugin_keys:
-            print("[grey53]\\[PLUGINS][/grey53]")
-            print(_format_toml(plugin_keys))
-            print("[grey53]################################################################[/grey53]")
-
-        raise SystemExit(not matching_config)
-
-    elif set:
-        new_config = {}
-        failed_options = []
-        for line in config_options:
-            if line.startswith("#") or not line.strip():
-                continue
-            if "=" not in line:
-                print("[red][X] Config KEY=VALUE must have an = sign in it[/red]")
-                print(f"    {line}")
-                raise SystemExit(2)
-
-            raw_key, val = line.split("=", 1)
-            raw_key = raw_key.upper().strip()
-            key = core_config_aliases.get(raw_key) or resolve_alias(raw_key, plugin_schemas)
-            if key != raw_key:
-                print(
-                    f"[yellow][i] Note: The config option {raw_key} has been renamed to {key}, please use the new name going forwards.[/yellow]",
-                )
-
-            if key in writable_config:
-                new_config[key] = val.strip()
-            else:
-                failed_options.append(line)
-
-        if new_config:
-            before = FLAT_CONFIG
-            matching_config = write_config_file(new_config)
-            after = get_config().as_dict()
-            print(printable_config(matching_config))
-
-            side_effect_changes = {}
-            for key, val in after.items():
-                if key in FLAT_CONFIG and (str(before[key]) != str(after[key])) and (key not in matching_config):
-                    side_effect_changes[key] = after[key]
-
-            if side_effect_changes:
-                print(file=sys.stderr)
-                print("[yellow][i] Note: This change also affected these other options that depended on it:[/yellow]", file=sys.stderr)
-                print("    {}".format(printable_config(side_effect_changes, prefix="    ")), file=sys.stderr)
-
-        if failed_options:
-            print()
-            print("[red][X] These options failed to set (check for typos):[/red]")
-            print("    {}".format("\n    ".join(failed_options)))
-            raise SystemExit(1)
-
-    elif reset:
-        print("[red][X] This command is not implemented yet.[/red]")
-        print("    Please manually remove the relevant lines from your config file:")
-        raise SystemExit(2)
-
-    else:
-        print("[red][X] You must pass either --get or --set, or no arguments to get the whole config.[/red]")
-        print("    archivebox config")
-        print("    archivebox config --get SOME_KEY")
-        print("    archivebox config --set SOME_KEY=SOME_VALUE")
-        raise SystemExit(2)
-
-
-@click.command()
-@click.option("--search", is_flag=True, help="Search config KEYs, VALUEs, and ALIASES for the given term")
-@click.option("--get", is_flag=True, help="Get the value for the given config KEYs")
-@click.option("--set", is_flag=True, help="Set the given KEY=VALUE config values")
-@click.option("--reset", is_flag=True, help="Reset the given KEY config values to their defaults")
-@click.argument("KEY=VALUE", nargs=-1, type=str)
-@docstring(config.__doc__)
-def main(**kwargs) -> None:
-    config(**kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,291 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox crawl <action> [args...] [--filters]
-
-Manage Crawl records.
-
-Actions:
-    create  - Create Crawl jobs from URLs
-    list    - List Crawls as JSONL (with optional filters)
-    update  - Update Crawls from stdin JSONL
-    delete  - Delete Crawls from stdin JSONL
-
-Examples:
-    # Create
-    archivebox crawl create https://example.com https://foo.com --depth=1
-    archivebox crawl create --tag=news https://example.com
-
-    # List with filters
-    archivebox crawl list --status=queued
-    archivebox crawl list --urls__icontains=example.com
-
-    # Update
-    archivebox crawl list --status=started | archivebox crawl update --status=queued
-
-    # Delete
-    archivebox crawl list --urls__icontains=spam.com | archivebox crawl delete --yes
-
-    # Full pipeline
-    archivebox crawl create https://example.com | archivebox snapshot create | archivebox run
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox crawl"
-
-import sys
-from collections.abc import Iterable
-
-import rich_click as click
-from rich import print as rprint
-
-from archivebox.cli.cli_util import apply_filters
-
-
-# =============================================================================
-# CREATE
-# =============================================================================
-
-
-def create_crawl(
-    urls: Iterable[str],
-    depth: int = 0,
-    tag: str = "",
-    status: str = "queued",
-    created_by_id: int | None = None,
-) -> int:
-    """
-    Create a Crawl job from URLs.
-
-    Takes URLs as args or stdin, creates one Crawl with all URLs, outputs JSONL.
-    Pass-through: Records that are not URLs are output unchanged (for piping).
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
-    from archivebox.misc.jsonl import read_args_or_stdin, write_record, TYPE_CRAWL
-    from archivebox.misc.util import validate_url
-    from archivebox.base_models.models import get_or_create_system_user_pk
-    from archivebox.crawls.models import Crawl
-
-    created_by_id = created_by_id or get_or_create_system_user_pk()
-    is_tty = sys.stdout.isatty()
-
-    # Collect all input records
-    records = list(read_args_or_stdin(urls))
-
-    if not records:
-        rprint("[yellow]No URLs provided. Pass URLs as arguments or via stdin.[/yellow]", file=sys.stderr)
-        return 1
-
-    # Separate pass-through records from URL records
-    url_list = []
-    pass_through_records = []
-
-    for record in records:
-        record_type = record.get("type", "")
-
-        # Pass-through: output records that aren't URL/Crawl types
-        if record_type and record_type != TYPE_CRAWL and not record.get("url") and not record.get("urls"):
-            pass_through_records.append(record)
-            continue
-
-        # Handle existing Crawl records (just pass through with id)
-        if record_type == TYPE_CRAWL and record.get("id"):
-            pass_through_records.append(record)
-            continue
-
-        # Collect URLs
-        url = record.get("url")
-        if url:
-            try:
-                url_list.append(validate_url(str(url)))
-            except ValueError as err:
-                rprint(f"[red]Invalid URL: {err}[/red]", file=sys.stderr)
-                return 1
-
-        # Handle 'urls' field (newline-separated)
-        urls_field = record.get("urls")
-        if urls_field:
-            for line in urls_field.split("\n"):
-                line = line.strip()
-                if line and not line.startswith("#"):
-                    try:
-                        url_list.append(validate_url(line))
-                    except ValueError as err:
-                        rprint(f"[red]Invalid URL: {err}[/red]", file=sys.stderr)
-                        return 1
-
-    # Output pass-through records first
-    if not is_tty:
-        for record in pass_through_records:
-            write_record(record)
-
-    if not url_list:
-        if pass_through_records:
-            # If we had pass-through records but no URLs, that's OK
-            rprint(f"[dim]Passed through {len(pass_through_records)} records, no new URLs[/dim]", file=sys.stderr)
-            return 0
-        rprint("[red]No valid URLs found[/red]", file=sys.stderr)
-        return 1
-
-    try:
-        # Build crawl record with all URLs as newline-separated string
-        crawl_record = {
-            "urls": "\n".join(url_list),
-            "max_depth": depth,
-            "tags_str": tag,
-            "status": status,
-            "label": "",
-        }
-
-        crawl = Crawl.from_json(crawl_record, overrides={"created_by_id": created_by_id})
-        if not crawl:
-            rprint("[red]Failed to create crawl[/red]", file=sys.stderr)
-            return 1
-
-        # Output JSONL record (only when piped)
-        if not is_tty:
-            write_record(crawl.to_json())
-
-        rprint(f"[green]Created crawl with {len(url_list)} URLs[/green]", file=sys.stderr)
-
-        # If TTY, show human-readable output
-        if is_tty:
-            rprint(f"  [dim]{crawl.id}[/dim]", file=sys.stderr)
-            for url in url_list[:5]:  # Show first 5 URLs
-                rprint(f"    {url[:70]}", file=sys.stderr)
-            if len(url_list) > 5:
-                rprint(f"    ... and {len(url_list) - 5} more", file=sys.stderr)
-
-        return 0
-
-    except Exception as e:
-        rprint(f"[red]Error creating crawl: {e}[/red]", file=sys.stderr)
-        return 1
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_crawls(
-    status: str | None = None,
-    urls__icontains: str | None = None,
-    max_depth: int | None = None,
-    limit: int | None = None,
-) -> int:
-    """List crawls as JSONL, or formatted rows in a terminal."""
-    from archivebox.crawls.models import Crawl
-    from archivebox.cli.cli_util import list_records, format_status
-
-    queryset = apply_filters(
-        Crawl.objects.order_by("-created_at"),
-        {"status": status, "urls__icontains": urls__icontains, "max_depth": max_depth},
-        limit=limit,
-    )
-    return list_records(
-        queryset,
-        plural="crawls",
-        render=lambda crawl: f"{format_status(crawl.status, 8)} [dim]{crawl.id}[/dim] {crawl.urls[:50].replace(chr(10), chr(32))}...",
-    )
-
-
-# =============================================================================
-# UPDATE
-# =============================================================================
-
-
-def update_crawls(status: str | None = None, max_depth: int | None = None) -> int:
-    """Update JSONL-selected crawls through their lifecycle operations."""
-    from django.utils import timezone
-    from archivebox.crawls.models import Crawl
-    from archivebox.cli.cli_util import update_records, update_record_status
-
-    def update(crawl):
-        if status:
-            try:
-                update_record_status(crawl, status)
-            except ValueError as err:
-                rprint(f"[red]{err}[/red]", file=sys.stderr)
-                return False
-        if max_depth is not None:
-            crawl.safe_update({"max_depth": max_depth, "modified_at": timezone.now()}, refresh=False)
-            crawl.max_depth = max_depth
-        elif not status:
-            crawl.safe_update({"modified_at": timezone.now()}, refresh=False)
-
-    return update_records(Crawl, update, plural="crawls", refresh=True)
-
-
-# =============================================================================
-# DELETE
-# =============================================================================
-
-
-def delete_crawls(yes: bool = False, dry_run: bool = False) -> int:
-    """Delete crawls selected by stdin JSONL; --yes confirms, --dry-run previews."""
-    from archivebox.cli.cli_util import delete_records
-    from archivebox.crawls.models import Crawl
-
-    return delete_records(
-        Crawl,
-        label="crawl",
-        plural="crawls",
-        preview=lambda obj: f"[dim]{obj.id}[/dim] {obj.urls[:50].replace(chr(10), chr(32))}...",
-        yes=yes,
-        dry_run=dry_run,
-    )
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage Crawl records."""
-    pass
-
-
-@main.command("create")
-@click.argument("urls", nargs=-1)
-@click.option("--depth", "-d", type=int, default=0, help="Max crawl depth (default: 0)")
-@click.option("--tag", "-t", default="", help="Comma-separated tags to add")
-@click.option("--status", "-s", default="queued", help="Initial status (default: queued)")
-def create_cmd(urls: tuple, depth: int, tag: str, status: str):
-    """Create a Crawl job from URLs or stdin."""
-    sys.exit(create_crawl(urls, depth=depth, tag=tag, status=status))
-
-
-@main.command("list")
-@click.option("--status", "-s", help="Filter by status (queued, started, sealed)")
-@click.option("--urls__icontains", help="Filter by URLs contains")
-@click.option("--max-depth", type=int, help="Filter by max depth")
-@click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List Crawls as JSONL."""
-    sys.exit(list_crawls(**kwargs))
-
-
-@main.command("update")
-@click.option("--status", "-s", help="Set status")
-@click.option("--max-depth", type=int, help="Set max depth")
-def update_cmd(**kwargs):
-    """Update Crawls from stdin JSONL."""
-    sys.exit(update_crawls(**kwargs))
-
-
-@main.command("delete")
-@click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
-@click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(**kwargs):
-    """Delete Crawls from stdin JSONL."""
-    sys.exit(delete_crawls(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,198 +0,0 @@
-#!/usr/bin/env python3
-
-"""Run abx-dl snapshot hooks for existing ArchiveBox snapshots."""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox extract"
-
-import sys
-from collections import defaultdict
-from contextlib import redirect_stdout
-
-import rich_click as click
-
-
-def _resolve_requests(records: list[dict], plugins: str) -> tuple[dict[str, set[str]], set[str]]:
-    """Resolve CLI records to snapshot-level execution requests.
-
-    ArchiveResult input is accepted as a convenient reference to its parent
-    Snapshot and plugin. It is never reset or converted into queued work.
-    """
-    from archivebox.misc.db import uuid_ref_query
-    from archivebox.core.models import ArchiveResult, Snapshot
-    from archivebox.misc.jsonl import TYPE_ARCHIVERESULT
-
-    explicit_plugins = {name.strip() for name in plugins.split(",") if name.strip()}
-    requested: dict[str, set[str]] = defaultdict(set)
-    missing: set[str] = set()
-
-    for record in records:
-        record_type = record.get("type")
-        record_id = str(record.get("id") or "")
-
-        if record_type == TYPE_ARCHIVERESULT and record.get("snapshot_id"):
-            requested[str(record["snapshot_id"])].update(explicit_plugins or {str(record.get("plugin") or "")})
-            requested[str(record["snapshot_id"])].discard("")
-            continue
-
-        if record_type == TYPE_ARCHIVERESULT:
-            result = ArchiveResult.objects.filter(uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
-            if result is not None:
-                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
-                continue
-
-        snapshot_id = str(record.get("snapshot_id") or record_id or "")
-        snapshot = Snapshot.objects.filter(id=snapshot_id).only("id").first() if snapshot_id else None
-        if snapshot is None and record.get("url"):
-            snapshot = Snapshot.objects.filter(url=record["url"]).order_by("-created_at").only("id").first()
-        # Bare UUID CLI arguments are parsed as Snapshot records because the
-        # input layer cannot know which model owns them. Preserve the public
-        # convenience of passing an ArchiveResult ID by trying that reference
-        # only after the Snapshot lookup misses.
-        if snapshot is None and record_id and not record.get("url"):
-            result = ArchiveResult.objects.filter(uuid_ref_query("id", record_id)).only("snapshot_id", "plugin").first()
-            if result is not None:
-                requested[str(result.snapshot_id)].update(explicit_plugins or {result.plugin})
-                continue
-        if snapshot is None:
-            missing.add(snapshot_id or str(record.get("url") or ""))
-            continue
-        requested[str(snapshot.id)].update(explicit_plugins)
-
-    return requested, missing
-
-
-def _run_snapshot_requests(requested: dict[str, set[str]], *, wait: bool, show_progress: bool) -> int:
-    from django.utils import timezone
-    from rich import print as rprint
-
-    from archivebox.core.models import Snapshot
-    from archivebox.services.runner import run_crawl
-
-    snapshots = {str(snapshot.id): snapshot for snapshot in Snapshot.objects.filter(id__in=requested).select_related("crawl")}
-    if not snapshots:
-        rprint("[red]No snapshots to process[/red]", file=sys.stderr)
-        return 1
-
-    from archivebox.config.common import get_config
-    from archivebox.plugins.discovery import get_enabled_plugins
-
-    for snapshot_id, plugin_names in requested.items():
-        snapshot = snapshots.get(snapshot_id)
-        if snapshot is not None and not plugin_names:
-            plugin_names.update(get_enabled_plugins(config=get_config(crawl=snapshot.crawl, snapshot=snapshot)))
-
-    if wait and any("search_backend_sonic" in plugin_names for plugin_names in requested.values()):
-        from archivebox.core.takeover_util import ensure_daemon_stack
-
-        with redirect_stdout(sys.stderr):
-            ensure_daemon_stack(reason="Sonic snapshot indexing")
-
-    # Explicit extraction resumes open/paused snapshots at the snapshot level.
-    # Sealed snapshots stay sealed during targeted maintenance backfills.
-    if wait:
-        for snapshot in snapshots.values():
-            if snapshot.status != Snapshot.StatusChoices.SEALED:
-                snapshot.update_and_requeue(
-                    status=Snapshot.StatusChoices.QUEUED,
-                    retry_at=timezone.now(),
-                )
-
-    if not wait:
-        for snapshot_id, plugin_names in requested.items():
-            snapshot = snapshots.get(snapshot_id)
-            if snapshot is None:
-                continue
-            if plugin_names:
-                snapshot.schedule_plugin_run(plugin_names, when=timezone.now())
-            elif snapshot.status == Snapshot.StatusChoices.SEALED:
-                snapshot.update_and_requeue(retry_at=timezone.now())
-            else:
-                snapshot.update_and_requeue(
-                    status=Snapshot.StatusChoices.QUEUED,
-                    retry_at=timezone.now(),
-                )
-        if show_progress:
-            rprint(f"[blue]Queued {len(snapshots)} snapshots for extraction[/blue]", file=sys.stderr)
-        return 0
-
-    grouped: dict[tuple[str, tuple[str, ...]], list[str]] = defaultdict(list)
-    for snapshot_id, plugin_names in requested.items():
-        snapshot = snapshots.get(snapshot_id)
-        if snapshot is not None:
-            grouped[(str(snapshot.crawl_id), tuple(sorted(plugin_names)))].append(snapshot_id)
-
-    for (crawl_id, plugin_names), snapshot_ids in grouped.items():
-        run_crawl(
-            crawl_id,
-            snapshot_ids=sorted(snapshot_ids),
-            selected_plugins=list(plugin_names) or None,
-            show_progress=show_progress,
-        )
-    return 0
-
-
-def run_plugins(
-    args: tuple,
-    records: list[dict] | None = None,
-    plugins: str = "",
-    wait: bool = True,
-    emit_results: bool = True,
-    show_progress: bool = True,
-) -> int:
-    """Execute selected plugins through the snapshot-level runner."""
-    from rich import print as rprint
-
-    from archivebox.core.models import Snapshot
-    from archivebox.misc.jsonl import read_args_or_stdin, write_record
-
-    if records is None:
-        records = list(read_args_or_stdin(args))
-    if not records:
-        rprint("[yellow]No snapshots provided. Pass snapshot IDs as arguments or via stdin.[/yellow]", file=sys.stderr)
-        return 1
-
-    requested, missing = _resolve_requests(records, plugins)
-    for value in sorted(missing):
-        rprint(f"[yellow]Snapshot or ArchiveResult not found: {value}[/yellow]", file=sys.stderr)
-    if not requested:
-        return 1
-
-    exit_code = _run_snapshot_requests(requested, wait=wait, show_progress=show_progress)
-    if exit_code or not emit_results:
-        return exit_code
-
-    is_tty = sys.stdout.isatty()
-    for snapshot in Snapshot.objects.filter(id__in=requested):
-        results = snapshot.archiveresult_set.all()
-        requested_plugins = requested[str(snapshot.id)]
-        if requested_plugins:
-            results = results.filter(plugin__in=requested_plugins)
-        for result in results:
-            if is_tty:
-                color = {"succeeded": "green", "failed": "red", "skipped": "yellow"}.get(result.status, "dim")
-                rprint(f"  [{color}]{result.status}[/{color}] {result.plugin} → {result.output_str or ''}", file=sys.stderr)
-            else:
-                write_record(result.to_json())
-    return 0
-
-
-@click.command()
-@click.option("--plugins", "--plugin", "-p", default="", help="Comma-separated list of plugins to run")
-@click.option("--wait/--no-wait", default=True, help="Wait for plugins to complete (default: wait)")
-@click.argument("args", nargs=-1)
-def main(plugins: str, wait: bool, args: tuple):
-    """Run plugins on Snapshots; ArchiveResult IDs select their parent plugin."""
-    from archivebox.misc.jsonl import read_args_or_stdin
-
-    records = list(read_args_or_stdin(args))
-    if not records:
-        from rich import print as rprint
-
-        rprint("[yellow]No Snapshot IDs or ArchiveResult IDs provided. Pass as arguments or via stdin.[/yellow]", file=sys.stderr)
-        sys.exit(1)
-    sys.exit(run_plugins(args, records=records, plugins=plugins, wait=wait))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,158 +0,0 @@
-#!/usr/bin/env python3
-__package__ = "archivebox.cli"
-__command__ = "archivebox help"
-
-import ast
-import importlib.util
-import os
-import sys
-from datetime import UTC, datetime
-from pathlib import Path
-
-import click
-from rich import print
-from rich.panel import Panel
-
-
-def _command_doc(cmd: str, import_path: str) -> str:
-    def first_doc_line(docstring: str | None) -> str:
-        lines = (docstring or "").splitlines()
-        return lines[0] if lines else ""
-
-    modname, _ = import_path.rsplit(".", 1)
-    spec = importlib.util.find_spec(modname)
-    if spec is None or spec.origin is None:
-        return ""
-
-    try:
-        tree = ast.parse(Path(spec.origin).read_text())
-    except OSError:
-        return ""
-
-    for name in (cmd, "main"):
-        for node in tree.body:
-            if isinstance(node, ast.FunctionDef) and node.name == name:
-                return first_doc_line(ast.get_docstring(node))
-
-    return first_doc_line(ast.get_docstring(tree))
-
-
-def help() -> None:
-    """Print the ArchiveBox help message and usage"""
-
-    from archivebox.cli import ArchiveBoxGroup
-    from archivebox.config import CONSTANTS
-    from archivebox.config.permissions import IN_DOCKER
-
-    version_msg = (
-        f"[dark_magenta]\\[{datetime.now(UTC):%Y-%m-%d %H:%M:%S}][/dark_magenta] "
-        f"[dark_red]ArchiveBox[/dark_red] [dark_goldenrod]v{CONSTANTS.VERSION}[/dark_goldenrod]: "
-        "[green4]archivebox [green3]help"
-    )
-    print(Panel(version_msg), file=sys.stderr)
-
-    COMMANDS_HELP_TEXT = (
-        "\n    ".join(f"[green]{cmd.ljust(20)}[/green] {_command_doc(cmd, path)}" for cmd, path in ArchiveBoxGroup.meta_commands.items())
-        + "\n\n    "
-        + "\n    ".join(f"[green]{cmd.ljust(20)}[/green] {_command_doc(cmd, path)}" for cmd, path in ArchiveBoxGroup.setup_commands.items())
-        + "\n\n    "
-        + "\n    ".join(
-            f"[green]{cmd.ljust(20)}[/green] {_command_doc(cmd, path)}" for cmd, path in ArchiveBoxGroup.archive_commands.items()
-        )
-    )
-
-    DOCKER_USAGE = (
-        """
-[dodger_blue3]Docker Usage:[/dodger_blue3]
-    [grey53]# using Docker Compose:[/grey53]
-    [blue]docker compose run[/blue] [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
-
-    [grey53]# using Docker:[/grey53]
-    [blue]docker run[/blue] -v [light_slate_blue]$PWD:/data[/light_slate_blue] [grey53]-p 8000:8000[/grey53] -it [dark_green]archivebox/archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
-"""
-        if IN_DOCKER
-        else ""
-    )
-    DOCKER_DOCS = (
-        "\n    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage]https://github.com/ArchiveBox/ArchiveBox/wiki/Docker[/link]"
-        if IN_DOCKER
-        else ""
-    )
-    DOCKER_OUTSIDE_HINT = "\n    [grey53]# outside of Docker:[/grey53]" if IN_DOCKER else ""
-    DOCKER_CMD_PREFIX = "[blue]docker ... [/blue]" if IN_DOCKER else ""
-
-    print(f"""{DOCKER_USAGE}
-[deep_sky_blue4]Usage:[/deep_sky_blue4]{DOCKER_OUTSIDE_HINT}
-    [dark_green]archivebox[/dark_green] [green]\\[command][/green] [green3][...args][/green3] [violet][--help][/violet] [grey53][--version][/grey53]
-
-[deep_sky_blue4]Commands:[/deep_sky_blue4]
-    {COMMANDS_HELP_TEXT}
-
-[deep_sky_blue4]Documentation:[/deep_sky_blue4]
-    [link=https://github.com/ArchiveBox/ArchiveBox/wiki]https://github.com/ArchiveBox/ArchiveBox/wiki[/link]{DOCKER_DOCS}
-    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#cli-usage]https://github.com/ArchiveBox/ArchiveBox/wiki/Usage[/link]
-    [link=https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration]https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration[/link]
-""")
-
-    if os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and CONSTANTS.ARCHIVE_DIR.is_dir():
-        pretty_out_dir = str(CONSTANTS.DATA_DIR).replace(str(Path("~").expanduser()), "~")
-        EXAMPLE_USAGE = f"""
-[light_slate_blue]DATA DIR[/light_slate_blue]: [yellow]{pretty_out_dir}[/yellow]
-
-[violet]Hint:[/violet] [i]Common maintenance tasks:[/i]
-    [dark_green]archivebox[/dark_green] [green]init[/green]      [grey53]# make sure database is up-to-date (safe to run multiple times)[/grey53]
-    [dark_green]archivebox[/dark_green] [green]install[/green]   [grey53]# make sure plugins are up-to-date (wget, chrome, singlefile, etc.)[/grey53]
-    [dark_green]archivebox[/dark_green] [green]status[/green]    [grey53]# get a health checkup report on your collection[/grey53]
-    [dark_green]archivebox[/dark_green] [green]update[/green]    [grey53]# retry any previously failed or interrupted archiving tasks[/grey53]
-
-[violet]Hint:[/violet] [i]More example usage:[/i]
-    [dark_green]archivebox[/dark_green] [green]add[/green] --depth=1 "https://example.com/some/page"
-    [dark_green]archivebox[/dark_green] [green]list[/green] --sort=timestamp --csv=timestamp,downloaded_at,url,title
-    [dark_green]archivebox[/dark_green] [green]schedule[/green] --every=day --depth=1 "https://example.com/some/feed.rss"
-    [dark_green]archivebox[/dark_green] [green]server[/green] [blue]0.0.0.0:8000[/blue]                [grey53]# Start the Web UI / API server[/grey53]
-"""
-        print(
-            Panel(
-                EXAMPLE_USAGE,
-                expand=False,
-                border_style="grey53",
-                title="[green3]:white_check_mark: A collection [light_slate_blue]DATA DIR[/light_slate_blue] is currently active[/green3]",
-                subtitle="Commands run inside this dir will only apply to this collection.",
-            ),
-        )
-    else:
-        DATA_SETUP_HELP = "\n"
-        if IN_DOCKER:
-            DATA_SETUP_HELP += "[violet]Hint:[/violet] When using Docker, you need to mount a volume to use as your data dir:\n"
-            DATA_SETUP_HELP += "    docker run [violet]-v /some/path/data:/data[/violet] archivebox/archivebox ...\n\n"
-        DATA_SETUP_HELP += "To load an [dark_blue]existing[/dark_blue] collection:\n"
-        DATA_SETUP_HELP += "    1. [green]cd[/green] ~/archivebox/data     [grey53]# go into existing [light_slate_blue]DATA DIR[/light_slate_blue] (can be anywhere)[/grey53]\n"
-        DATA_SETUP_HELP += f"    2. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]init[/green]          [grey53]# migrate to latest version (safe to run multiple times)[/grey53]\n"
-        DATA_SETUP_HELP += f"    3. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]install[/green]       [grey53]# auto-update all plugins (wget, chrome, singlefile, etc.)[/grey53]\n"
-        DATA_SETUP_HELP += f"    4. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]help[/green]          [grey53]# ...get help with next steps... [/grey53]\n\n"
-        DATA_SETUP_HELP += "To start a [sea_green1]new[/sea_green1] collection:\n"
-        DATA_SETUP_HELP += "    1. [green]mkdir[/green] ~/archivebox/data  [grey53]# create a new, empty [light_slate_blue]DATA DIR[/light_slate_blue] (can be anywhere)[/grey53]\n"
-        DATA_SETUP_HELP += "    2. [green]cd[/green] ~/archivebox/data     [grey53]# cd into the new directory[/grey53]\n"
-        DATA_SETUP_HELP += f"    3. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]init[/green]          [grey53]# initialize ArchiveBox in the new data dir[/grey53]\n"
-        DATA_SETUP_HELP += f"    4. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]install[/green]       [grey53]# auto-install all plugins (wget, chrome, singlefile, etc.)[/grey53]\n"
-        DATA_SETUP_HELP += f"    5. {DOCKER_CMD_PREFIX}[dark_green]archivebox[/dark_green] [green]help[/green]          [grey53]# ... get help with next steps... [/grey53]\n"
-        print(
-            Panel(
-                DATA_SETUP_HELP,
-                expand=False,
-                border_style="grey53",
-                title="[red]:cross_mark: No collection is currently active[/red]",
-                subtitle="All archivebox [green]commands[/green] should be run from inside a collection [light_slate_blue]DATA DIR[/light_slate_blue]",
-            ),
-        )
-
-
-@click.command()
-@click.option("--help", "-h", is_flag=True, help="Show help")
-def main(**kwargs):
-    """Print the ArchiveBox help message and usage"""
-    return help()
-
-
-if __name__ == "__main__":
-    main()
@@ -1,205 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import os
-import sys
-from pathlib import Path
-
-from rich import print
-import rich_click as click
-
-from archivebox.misc.util import docstring, enforce_types
-
-
-def _display_data_path(path: Path, data_dir: Path) -> str:
-    path = Path(path).resolve()
-    data_dir = Path(data_dir).resolve()
-    try:
-        return f"./{path.relative_to(data_dir)}"
-    except ValueError:
-        return str(path)
-
-
-@enforce_types
-def init(force: bool = False, quick: bool = False, install: bool = False) -> None:
-    """Initialize a new ArchiveBox collection in the current directory"""
-
-    from archivebox.misc.checks import check_not_inside_source_dir
-
-    check_not_inside_source_dir()
-
-    from archivebox.config import CONSTANTS, VERSION
-    from archivebox.config.common import get_config
-    from archivebox.config.collection import write_config_file
-    from archivebox.misc.db import apply_migrations, database_exists, ensure_database_ready
-    from archivebox.misc.checks import check_migrations
-
-    config = get_config()
-
-    is_empty = not len(set(os.listdir(CONSTANTS.DATA_DIR)) - CONSTANTS.ALLOWED_IN_DATA_DIR)
-    existing_index = database_exists()
-    if is_empty and not existing_index:
-        print(f"[turquoise4][+] Initializing a new ArchiveBox v{VERSION} collection...[/turquoise4]")
-        print("[green]----------------------------------------------------------------------[/green]")
-    elif existing_index:
-        # TODO: properly detect and print the existing version in current index as well
-        print(f"[green][*] Verifying and updating existing ArchiveBox collection to v{VERSION}...[/green]")
-        print("[green]----------------------------------------------------------------------[/green]")
-    else:
-        if force:
-            print("[red][!] This folder appears to already have files in it, but no index.sqlite3 is present.[/red]")
-            print("[red]    Because --force was passed, ArchiveBox will initialize anyway (which may overwrite existing files).[/red]")
-        else:
-            print(
-                "[red][X] This folder appears to already have files in it, but no index.sqlite3 present.[/red]\n\n"
-                "    You must run init in a completely empty directory, or an existing data folder.\n\n"
-                "    [violet]Hint:[/violet] To import an existing data folder make sure to cd into the folder first, \n"
-                "    then run and run 'archivebox init' to pick up where you left off.\n\n"
-                "    (Always make sure your data folder is backed up first before updating ArchiveBox)",
-            )
-            raise SystemExit(2)
-
-    if existing_index:
-        print("\n[green][*] Verifying archive folder structure...[/green]")
-    else:
-        print("\n[green][+] Building archive folder structure...[/green]")
-
-    archive_path = _display_data_path(CONSTANTS.ARCHIVE_DIR, CONSTANTS.DATA_DIR)
-    sources_path = _display_data_path(CONSTANTS.SOURCES_DIR, CONSTANTS.DATA_DIR)
-    logs_path = _display_data_path(CONSTANTS.LOGS_DIR, CONSTANTS.DATA_DIR)
-    print(f"    + {archive_path}, {sources_path}, {logs_path}...")
-    Path(CONSTANTS.SOURCES_DIR).mkdir(exist_ok=True)
-    CONSTANTS.ARCHIVE_DIR.mkdir(parents=True, exist_ok=True)
-    CONSTANTS.USERS_DIR.mkdir(parents=True, exist_ok=True)
-    Path(CONSTANTS.LOGS_DIR).mkdir(exist_ok=True)
-    for path in (Path(CONSTANTS.SOURCES_DIR), CONSTANTS.ARCHIVE_DIR, CONSTANTS.USERS_DIR, Path(CONSTANTS.LOGS_DIR)):
-        path.chmod(int(config.OUTPUT_PERMISSIONS, base=8) | 0o111)
-
-    print(f"    + {_display_data_path(CONSTANTS.CONFIG_FILE, CONSTANTS.DATA_DIR)}...")
-
-    # create the .archivebox_id file with a unique ID for this collection
-    from archivebox.config.paths import _get_collection_id
-
-    _get_collection_id(CONSTANTS.DATA_DIR, force_create=True)
-
-    # create the ArchiveBox.conf file
-    write_config_file({"SECRET_KEY": config.SECRET_KEY})
-
-    if existing_index:
-        print("\n[green][*] Verifying main SQL index and running any migrations needed...[/green]")
-    else:
-        print("\n[green][+] Building main SQL index and running initial migrations...[/green]")
-
-    # For postgres, make sure the server is reachable and create the database
-    # if it doesn't exist yet (sqlite creates its file automatically).
-    ensure_database_ready()
-
-    from archivebox.config.django import setup_django
-
-    setup_django()
-    previous_wants_init = os.environ.get("ARCHIVEBOX_WANTS_INIT")
-    os.environ["ARCHIVEBOX_WANTS_INIT"] = "1"
-    try:
-        check_migrations(blocking=True, auto_apply=False)
-
-        for migration_line in apply_migrations(CONSTANTS.DATA_DIR):
-            sys.stdout.write(f"    {migration_line}\n")
-    finally:
-        if previous_wants_init is None:
-            os.environ.pop("ARCHIVEBOX_WANTS_INIT", None)
-        else:
-            os.environ["ARCHIVEBOX_WANTS_INIT"] = previous_wants_init
-
-    from archivebox.misc.db import database_display_location, is_postgres
-
-    assert database_exists()
-    print()
-    if is_postgres():
-        print(f"    √ {database_display_location()}")
-    else:
-        print(f"    √ {_display_data_path(CONSTANTS.DATABASE_FILE, CONSTANTS.DATA_DIR)}")
-
-    print()
-    print("[dodger_blue3][*] Checking links from indexes and archive folders (safe to Ctrl+C)...[/dodger_blue3]")
-
-    from archivebox.core.models import Snapshot
-
-    snapshot_count = 0
-
-    if existing_index:
-        snapshot_count = Snapshot.objects.count()
-        print(f"    √ Loaded {snapshot_count} links from existing main index.")
-
-    print("    > Skipping orphan snapshot import during init.")
-    print()
-    print("    [violet]Hint:[/violet] To import orphaned snapshot directories and reconcile filesystem state, run:")
-    print("        archivebox update")
-
-    print("\n[green]----------------------------------------------------------------------[/green]")
-
-    from django.contrib.auth.models import User
-
-    config = get_config()
-    if (config.ADMIN_USERNAME and config.ADMIN_PASSWORD) and not User.objects.filter(
-        username=config.ADMIN_USERNAME,
-    ).exists():
-        print("[green][+] Found ADMIN_USERNAME and ADMIN_PASSWORD configuration options, creating new admin user.[/green]")
-        User.objects.create_superuser(username=config.ADMIN_USERNAME, password=config.ADMIN_PASSWORD)
-
-    if existing_index:
-        print("[green][√] Done. Verified and updated the existing ArchiveBox collection.[/green]")
-    else:
-        print(f"[green][√] Done. A new ArchiveBox collection was initialized ({snapshot_count} links).[/green]")
-
-    CONSTANTS.PERSONAS_DIR.mkdir(parents=True, exist_ok=True)
-    CONSTANTS.DEFAULT_TMP_DIR.mkdir(parents=True, exist_ok=True)
-
-    from archivebox.config.paths import create_and_chown_dir, get_or_create_working_tmp_dir, get_or_create_working_lib_dir
-
-    config = get_config()
-    create_and_chown_dir(config.TMP_DIR)
-    create_and_chown_dir(config.ABXPKG_LIB_DIR)
-    create_and_chown_dir(config.ABXPKG_LIB_DIR / "bin")
-
-    working_tmp_dir = get_or_create_working_tmp_dir(autofix=True, quiet=True)
-    if working_tmp_dir:
-        working_tmp_dir.mkdir(parents=True, exist_ok=True)
-
-    working_lib_dir = get_or_create_working_lib_dir(autofix=True, quiet=True)
-    if working_lib_dir:
-        working_lib_dir.mkdir(parents=True, exist_ok=True)
-
-    if install:
-        from archivebox.cli.archivebox_install import install as install_method
-
-        install_method()
-
-    if Snapshot.objects.count() < 25:  # hide the hints for experienced users
-        from archivebox.core.routes_util import build_admin_url
-
-        admin_url = build_admin_url("/admin/", config=config)
-        print()
-        print("    [violet]Hint:[/violet] To finish web setup, run:")
-        print(
-            f"        archivebox server  # then visit [deep_sky_blue4][link={admin_url}]{admin_url}[/link][/deep_sky_blue4]",
-        )
-        print()
-        print("    To add new links, you can run:")
-        print("        archivebox add < ~/some/path/to/list_of_links.txt")
-        print()
-        print("    For more usage and examples, run:")
-        print("        archivebox help")
-
-
-@click.command()
-@click.option("--force", "-f", is_flag=True, help="Ignore unrecognized files in current directory and initialize anyway")
-@click.option("--quick", "-q", is_flag=True, help="Run any updates or migrations without rechecking all snapshot dirs")
-@click.option("--install", "--setup", "-s", is_flag=True, help="Automatically install dependencies and extras used for archiving")
-@docstring(init.__doc__)
-def main(**kwargs) -> None:
-    init(**kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,229 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import os
-import shutil
-import re
-from pathlib import Path
-
-import rich_click as click
-from rich import print
-
-from archivebox.misc.util import docstring, enforce_types
-
-
-def _resolve_install_targets(
-    requested_names: tuple[str, ...],
-) -> tuple[list[str], list[str]]:
-    """Resolve plugin names and declared binary aliases, leaving unknown tokens raw."""
-    from archivebox.plugins.discovery import get_plugin_catalog
-
-    catalog = get_plugin_catalog()
-    plugin_names_by_lower = {plugin_name.lower(): plugin_name for plugin_name in catalog}
-    plugin_names_by_binary_alias: dict[str, set[str]] = {}
-    for plugin_name, plugin in catalog.items():
-        for required_binary in plugin.config.required_binaries:
-            aliases = {required_binary.name.lower()}
-            template_match = re.fullmatch(r"\{([A-Za-z_][A-Za-z0-9_]*)\}", required_binary.name)
-            if template_match:
-                config_key = template_match.group(1)
-                aliases.add(config_key.lower())
-                aliases.add(config_key.removesuffix("_BINARY").lower())
-                default = plugin.config.properties.get(config_key, {}).get("default")
-                if default:
-                    aliases.add(str(default).lower())
-            for alias in aliases:
-                plugin_names_by_binary_alias.setdefault(alias, set()).add(plugin_name)
-
-    requested_plugins: list[str] = []
-    raw_binary_names: list[str] = []
-    for name in requested_names:
-        normalized_name = name.lower()
-        if normalized_name in plugin_names_by_lower:
-            requested_plugins.append(plugin_names_by_lower[normalized_name])
-        elif normalized_name in plugin_names_by_binary_alias:
-            requested_plugins.extend(sorted(plugin_names_by_binary_alias[normalized_name]))
-        else:
-            raw_binary_names.append(name)
-
-    selected_plugins = catalog.select(requested_plugins) if requested_plugins else catalog.select([])
-    selected_plugin_names = {name.lower() for name in selected_plugins}
-    raw_binary_names = [name for name in raw_binary_names if name.lower() not in selected_plugin_names]
-    return sorted(selected_plugins), sorted(set(raw_binary_names))
-
-
-def _install_raw_binary_names(binary_names: list[str], binproviders: str) -> None:
-    """Install user-requested standalone binaries through the Binary lifecycle."""
-    from django.utils import timezone
-
-    from archivebox.machine.models import Binary, Machine, _canonical_binary_name
-    from archivebox.services.runner import run_due_binary
-
-    machine = Machine.current()
-    for requested_name in binary_names:
-        binary_name = _canonical_binary_name(requested_name)
-        if not binary_name:
-            raise click.UsageError(f"Invalid binary name: {requested_name}")
-        binary = Binary.objects.filter(machine=machine, name=binary_name).order_by("-modified_at").first()
-        if binary is None:
-            binary = Binary.objects.create(
-                machine=machine,
-                name=binary_name,
-                binproviders=binproviders,
-                overrides={},
-                status=Binary.StatusChoices.QUEUED,
-            )
-        elif not binary.is_valid or binary.binproviders != binproviders:
-            binary.binproviders = binproviders
-            binary.overrides = {}
-            binary.status = Binary.StatusChoices.QUEUED
-            binary.retry_at = timezone.now()
-            binary.save(update_fields=["binproviders", "overrides", "status", "retry_at", "modified_at"])
-        run_due_binary(binary, lock_seconds=60)
-
-
-def ensure_data_dir_lib_symlink(data_dir: Path, abxpkg_lib_dir: Path) -> Path | None:
-    """Expose the shared abxpkg lib dir at DATA_DIR/lib after install."""
-    link_path = data_dir / "lib"
-    target_config_path = abxpkg_lib_dir.expanduser().absolute()
-    target_path = abxpkg_lib_dir.expanduser().resolve()
-    from archivebox.config.paths import get_machine_type
-
-    scoped_data_lib_path = (link_path / get_machine_type()).absolute()
-
-    if target_config_path == scoped_data_lib_path:
-        if link_path.is_symlink() or link_path.is_file():
-            link_path.unlink()
-        target_config_path.mkdir(parents=True, exist_ok=True)
-        return None
-
-    if link_path.resolve(strict=False) == target_path:
-        return None
-
-    target_path.mkdir(parents=True, exist_ok=True)
-
-    if link_path.is_symlink():
-        if link_path.resolve(strict=False) == target_path:
-            return link_path
-        link_path.unlink()
-    elif link_path.exists():
-        shutil.rmtree(link_path)
-
-    relative_target = os.path.relpath(target_path, start=link_path.parent)
-    link_path.symlink_to(relative_target, target_is_directory=True)
-    return link_path
-
-
-@enforce_types
-def install(binaries: tuple[str, ...] = (), binproviders: str = "*", dry_run: bool = False) -> None:
-    """Detect and install ArchiveBox dependencies by running the abx-dl install flow
-
-    Examples:
-        archivebox install                              # Install all dependencies
-        archivebox install wget curl                    # Install only wget and curl
-        archivebox install --binproviders=pip yt-dlp    # Install yt-dlp using only pip
-        archivebox install --binproviders=brew,apt      # Install all deps using only brew or apt
-    """
-
-    from archivebox.config.permissions import IS_ROOT, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP
-    from archivebox.config import CONSTANTS
-    from archivebox.misc.logging import stderr
-    from archivebox.cli.archivebox_init import init
-
-    archive_dir = CONSTANTS.ARCHIVE_DIR
-
-    if dry_run:
-        print("[dim]Dry run - would detect ArchiveBox dependencies and run the abx-dl install flow[/dim]")
-        return
-
-    if not (os.access(archive_dir, os.R_OK) and archive_dir.is_dir()):
-        init()  # must init full index because we need a db to store Binary entries in
-
-    # Show what we're installing
-    if binaries:
-        print(f"\n[green][+] Installing specific binaries: {', '.join(binaries)}[/green]")
-    else:
-        print("\n[green][+] Detecting and installing all ArchiveBox dependencies...[/green]")
-
-    if binproviders != "*":
-        print(f"[green][+] Using providers: {binproviders}[/green]")
-
-    if IS_ROOT:
-        EUID = os.geteuid()
-        print()
-        print(f"[yellow]:warning:  Running as UID=[blue]{EUID}[/blue].[/yellow]")
-        print(f"    DATA_DIR will be owned by [blue]{ARCHIVEBOX_USER}:{ARCHIVEBOX_GROUP}[/blue].")
-        print()
-
-    # Set up Django
-    from archivebox.config.django import setup_django
-
-    setup_django()
-
-    plugin_names, raw_binary_names = _resolve_install_targets(binaries) if binaries else ([], [])
-    install_plugin_names = list(plugin_names)
-    if binproviders != "*":
-        install_plugin_names.extend(provider.strip() for provider in binproviders.split(",") if provider.strip())
-
-    if install_plugin_names or not binaries:
-        if not binaries:
-            from archivebox.workers.supervisord_util import require_dependency_install_memory
-
-            require_dependency_install_memory()
-        print("[+] Running plugin installer via abx-dl bus...")
-        print()
-
-        from archivebox.services.runner import run_install
-
-        run_install(plugin_names=install_plugin_names or None)
-
-    if raw_binary_names:
-        print(f"[+] Running direct binary installer via ArchiveBox binary lifecycle: {', '.join(raw_binary_names)}")
-        print()
-        _install_raw_binary_names(raw_binary_names, binproviders)
-
-    from archivebox.config.common import get_config
-
-    config = get_config(include_machine=False)
-    ensure_data_dir_lib_symlink(CONSTANTS.DATA_DIR, config.ABXPKG_LIB_DIR)
-
-    print()
-
-    # Check for superuser
-    from django.contrib.auth import get_user_model
-
-    User = get_user_model()
-
-    if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
-        from archivebox.core.routes_util import build_admin_url
-
-        stderr("\n[+] Open the Admin UI to create the first admin and finish web setup:", color="green")
-        stderr(f"    {build_admin_url('/admin/', config=get_config(resolve_plugins=False))}")
-
-    print()
-
-    # Show version to display full status including installed binaries
-    # Django is already loaded, so just import and call the function directly
-    from archivebox.cli.archivebox_version import version as show_version
-
-    show_version(quiet=False, binaries=binaries)
-
-
-@click.command()
-@click.argument("binaries", nargs=-1, type=str, required=False)
-@click.option(
-    "--binproviders",
-    "-p",
-    default="*",
-    help="Comma-separated list of providers to use (pip,npm,brew,apt,env,custom) or * for all",
-    show_default=True,
-)
-@click.option("--dry-run", "-d", is_flag=True, help="Show what would happen without actually running", default=False)
-@docstring(install.__doc__)
-def main(**kwargs) -> None:
-    install(**kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,22 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox list"
-
-import sys
-
-import rich_click as click
-
-from archivebox.cli.archivebox_snapshot import list_snapshots, snapshot_filter_options, snapshot_output_options
-
-
-@click.command()
-@snapshot_output_options
-@snapshot_filter_options(default_filter_type="substring")
-def main(**kwargs) -> None:
-    """List Snapshots."""
-    sys.exit(list_snapshots(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,75 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox machine <action> [--filters]
-
-Manage Machine records (system-managed, mostly read-only).
-
-Machine records track the host machines where ArchiveBox runs.
-They are created automatically by the system and are primarily for debugging.
-
-Actions:
-    list    - List Machines as JSONL (with optional filters)
-
-Examples:
-    # List all machines
-    archivebox machine list
-
-    # List machines by hostname
-    archivebox machine list --hostname__icontains=myserver
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox machine"
-
-import sys
-
-import rich_click as click
-
-from archivebox.cli.cli_util import apply_filters
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_machines(hostname__icontains: str | None = None, os_platform: str | None = None, limit: int | None = None) -> int:
-    """List machines as JSONL, or formatted rows in a terminal."""
-    from archivebox.machine.models import Machine
-    from archivebox.cli.cli_util import list_records
-
-    queryset = apply_filters(
-        Machine.objects.order_by("-created_at"),
-        {"hostname__icontains": hostname__icontains, "os_platform": os_platform},
-        limit=limit,
-    )
-    return list_records(
-        queryset,
-        plural="machines",
-        render=lambda machine: f"[cyan]{machine.hostname:30}[/cyan] [dim]{machine.os_platform:10}[/dim] {machine.id}",
-    )
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage Machine records (read-only, system-managed)."""
-    pass
-
-
-@main.command("list")
-@click.option("--hostname__icontains", help="Filter by hostname contains")
-@click.option("--os-platform", help="Filter by OS platform")
-@click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List Machines as JSONL."""
-    sys.exit(list_machines(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,35 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import rich_click as click
-from archivebox.misc.util import docstring, enforce_types
-
-
-@enforce_types
-def manage(args: list[str] | None = None) -> None:
-    """Run an ArchiveBox Django management command"""
-
-    from archivebox.config.common import get_config
-    from archivebox.misc.logging import stderr
-
-    config = get_config()
-    if (args and "createsuperuser" in args and "--noinput" not in args) and (config.IN_DOCKER and not config.IS_TTY):
-        stderr("[!] Warning: you need to pass -it to use interactive commands in docker", color="lightyellow")
-        stderr("    docker run -it archivebox manage {}".format(" ".join(args or ["..."])), color="lightyellow")
-        stderr("")
-
-    from django.core.management import execute_from_command_line
-
-    execute_from_command_line(["manage.py", *(args or ["help"])])
-
-
-@click.command(add_help_option=False, context_settings=dict(ignore_unknown_options=True))
-@click.argument("args", nargs=-1)
-@docstring(manage.__doc__)
-def main(args: list[str] | None = None) -> None:
-    manage(args=args)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,49 +0,0 @@
-#!/usr/bin/env python3
-"""
-archivebox mcp
-
-Start the Model Context Protocol (MCP) server in stdio mode.
-Exposes all ArchiveBox CLI commands as MCP tools for AI agents.
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox mcp"
-
-import rich_click as click
-
-from archivebox.misc.util import docstring, enforce_types
-
-
-@enforce_types
-def mcp():
-    """
-    Start the MCP server in stdio mode for AI agent control.
-
-    The MCP (Model Context Protocol) server exposes all ArchiveBox CLI commands
-    as tools that AI agents can discover and execute. It communicates via JSON-RPC
-    2.0 over stdin/stdout.
-
-    Example usage with an MCP client:
-        archivebox mcp < requests.jsonl > responses.jsonl
-
-    Or interactively:
-        archivebox mcp
-        {"jsonrpc":"2.0","id":1,"method":"initialize","params":{}}
-        {"jsonrpc":"2.0","id":2,"method":"tools/list","params":{}}
-    """
-
-    from archivebox.mcp.server import run_mcp_server
-
-    # Run the stdio server (blocks until stdin closes)
-    run_mcp_server()
-
-
-@click.command()
-@docstring(mcp.__doc__)
-def main(**kwargs):
-    """Start the MCP server in stdio mode"""
-    mcp()
-
-
-if __name__ == "__main__":
-    main()
@@ -1,49 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox oneshot"
-
-import subprocess
-import sys
-from pathlib import Path
-
-import rich_click as click
-
-from archivebox.config import CONSTANTS
-from archivebox.config.common import get_config
-
-
-@click.command(add_help_option=False, context_settings=dict(ignore_unknown_options=True))
-@click.argument("args", nargs=-1)
-def main(args: tuple[str, ...] = ()) -> None:
-    """Download URLs using abx-dl"""
-    from archivebox.misc.checks import check_not_inside_source_dir
-
-    check_not_inside_source_dir()
-
-    cwd = Path.cwd()
-    if any((path / CONSTANTS.SQL_INDEX_FILENAME).exists() for path in (cwd, *cwd.parents)):
-        raise click.ClickException(
-            "Refusing to run `archivebox oneshot` inside an ArchiveBox DATA_DIR. Use `archivebox add` here, or run oneshot from another directory.",
-        )
-    abxpkg_binary = Path(sys.executable).with_name("abxpkg")
-    if not abxpkg_binary.is_file():
-        raise click.ClickException(f"abxpkg executable is missing from the ArchiveBox environment: {abxpkg_binary}")
-    abxpkg_lib_dir = get_config(include_machine=False).ABXPKG_LIB_DIR
-    raise SystemExit(
-        subprocess.run(
-            [
-                str(abxpkg_binary),
-                f"--lib={abxpkg_lib_dir}",
-                "--binproviders=env",
-                "--install",
-                "run",
-                "abx-dl",
-                *args,
-            ],
-        ).returncode,
-    )
-
-
-if __name__ == "__main__":
-    main()
@@ -1,491 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox persona <action> [args...] [--filters]
-
-Manage Persona records (browser profiles for archiving).
-
-Actions:
-    create  - Create Personas
-    list    - List Personas as JSONL (with optional filters)
-    update  - Update Personas from stdin JSONL
-    delete  - Delete Personas from stdin JSONL
-
-Examples:
-    # Create a new persona
-    archivebox persona create work
-    archivebox persona create --import=chrome personal
-    archivebox persona create --import=edge work
-
-    # List all personas
-    archivebox persona list
-
-    # Delete a persona
-    archivebox persona list --name=old | archivebox persona delete --yes
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox persona"
-
-import sys
-import shutil
-from pathlib import Path
-from dataclasses import replace
-from collections.abc import Iterable
-
-import rich_click as click
-from rich import print as rprint
-
-from archivebox.cli.cli_util import apply_filters, list_records
-from archivebox.personas import importers as persona_importers
-
-
-def ensure_path_within_personas_dir(persona_path: Path) -> bool:
-    """
-    Verify that a persona path is within PERSONAS_DIR.
-
-    This is a safety check to prevent path traversal attacks where
-    a malicious persona name could cause operations on paths outside
-    the expected PERSONAS_DIR.
-
-    Returns:
-        True if path is safe, False otherwise
-    """
-    from archivebox.config.constants import CONSTANTS
-
-    try:
-        # Resolve both paths to absolute paths
-        personas_dir = CONSTANTS.PERSONAS_DIR.resolve()
-        resolved_path = persona_path.resolve()
-
-        # Check if resolved_path is a child of personas_dir
-        return resolved_path.is_relative_to(personas_dir)
-    except (ValueError, RuntimeError):
-        return False
-
-
-# =============================================================================
-# CREATE
-# =============================================================================
-
-
-def create_personas(
-    names: Iterable[str],
-    import_from: str | None = None,
-    profile: str | None = None,
-    source: str | None = None,
-    browser_binary: str | None = None,
-) -> int:
-    """
-    Create Personas from names.
-
-    If --import is specified, copy the browser profile to the persona directory
-    and extract cookies.
-
-    Exit codes:
-        0: Success
-        1: Failure
-    """
-    from archivebox.misc.jsonl import write_record
-    from archivebox.personas.models import Persona
-
-    is_tty = sys.stdout.isatty()
-    name_list = list(names) if names else []
-
-    if not name_list:
-        rprint("[yellow]No persona names provided. Pass names as arguments.[/yellow]", file=sys.stderr)
-        return 1
-
-    import_source = None
-    if source and not import_from:
-        rprint("[red]--source requires --import (the source browser name).[/red]", file=sys.stderr)
-        return 1
-    if import_from:
-        try:
-            if source:
-                import_source = persona_importers.resolve_custom_import_source(source, profile_dir=profile)
-                if import_source.browser != "persona":
-                    import_source = replace(import_source, browser=import_from.lower(), browser_binary=browser_binary)
-            elif import_from.startswith(("http://", "https://", "ws://", "wss://")) or Path(import_from).expanduser().is_absolute():
-                import_source = persona_importers.resolve_custom_import_source(import_from)
-            else:
-                import_source = persona_importers.resolve_browser_import_source(import_from, profile_dir=profile)
-            if browser_binary:
-                import_source = replace(import_source, browser_binary=browser_binary)
-        except ValueError as err:
-            rprint(f"[red]{err}[/red]", file=sys.stderr)
-            return 1
-
-    created_count = 0
-    for name in name_list:
-        name = name.strip()
-        if not name:
-            continue
-
-        # Validate persona name to prevent path traversal
-        is_valid, error_msg = persona_importers.validate_persona_name(name)
-        if not is_valid:
-            rprint(f'[red]Invalid persona name "{name}": {error_msg}[/red]', file=sys.stderr)
-            continue
-
-        persona, created = Persona.objects.get_or_create(name=name)
-
-        if created:
-            persona.ensure_dirs()
-            created_count += 1
-
-        else:
-            rprint(f"[dim]Persona already exists: {name}[/dim]", file=sys.stderr)
-
-        cookies_file = Path(persona.path) / "cookies.txt"
-
-        # Import browser profile if requested
-        if import_source is not None:
-            try:
-                rprint(f"[dim]Importing {import_source.display_label}; copying settings and exporting cookies...[/dim]", file=sys.stderr)
-                import_result = persona_importers.import_persona_from_source(
-                    persona,
-                    import_source,
-                    copy_profile=True,
-                    import_cookies=True,
-                    capture_storage=False,
-                )
-            except (Exception, KeyboardInterrupt) as e:
-                if created:
-                    shutil.rmtree(persona.path, ignore_errors=True)
-                    persona.delete()
-                rprint(f"[red]Failed to import browser profile: {e}[/red]", file=sys.stderr)
-                return 1
-
-            if import_result.profile_copied:
-                rprint("[green]Copied browser profile to persona[/green]", file=sys.stderr)
-            if import_result.cookies_imported:
-                rprint(f"[green]Extracted {import_result.cookie_count} cookies to {cookies_file}[/green]", file=sys.stderr)
-            elif not import_result.profile_copied:
-                rprint("[yellow]Could not import cookies automatically.[/yellow]", file=sys.stderr)
-
-            for warning in import_result.warnings:
-                rprint(f"[yellow]{warning}[/yellow]", file=sys.stderr)
-
-        if created:
-            rprint(f"[green]Created persona: {name}[/green]", file=sys.stderr)
-        if not is_tty:
-            write_record(persona.to_json())
-
-    rprint(f"[green]Created {created_count} new persona(s)[/green]", file=sys.stderr)
-    return 0
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_personas(
-    name: str | None = None,
-    name__icontains: str | None = None,
-    limit: int | None = None,
-) -> int:
-    """
-    List Personas as JSONL with optional filters.
-
-    Exit codes:
-        0: Success (even if no results)
-    """
-    from archivebox.personas.models import Persona
-
-    queryset = apply_filters(
-        Persona.objects.order_by("name"),
-        {"name": name, "name__icontains": name__icontains},
-        limit=limit,
-    )
-
-    def render(persona):
-        cookies = "[green]✓[/green]" if persona.COOKIES_FILE else "[dim]✗[/dim]"
-        chrome = "[green]✓[/green]" if Path(persona.CHROME_USER_DATA_DIR).exists() else "[dim]✗[/dim]"
-        return f"[cyan]{persona.name:20}[/cyan] cookies:{cookies} chrome:{chrome} [dim]{persona.path}[/dim]"
-
-    return list_records(queryset, plural="persona(s)", render=render)
-
-
-# =============================================================================
-# UPDATE
-# =============================================================================
-
-
-def update_personas(name: str | None = None) -> int:
-    """
-    Update Personas from stdin JSONL.
-
-    Reads Persona records from stdin and applies updates.
-    Uses PATCH semantics - only specified fields are updated.
-
-    Exit codes:
-        0: Success
-        1: No input or error
-    """
-    from archivebox.misc.jsonl import read_stdin, write_record
-    from archivebox.personas.models import Persona
-
-    is_tty = sys.stdout.isatty()
-
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    updated_count = 0
-    for record in records:
-        persona_id = record.get("id")
-        old_name = record.get("name")
-
-        if not persona_id and not old_name:
-            continue
-
-        try:
-            if persona_id:
-                persona = Persona.objects.get(id=persona_id)
-            else:
-                persona = Persona.objects.get(name=old_name)
-
-            # Apply updates from CLI flags
-            if name:
-                # Validate new name to prevent path traversal
-                is_valid, error_msg = persona_importers.validate_persona_name(name)
-                if not is_valid:
-                    rprint(f'[red]Invalid new persona name "{name}": {error_msg}[/red]', file=sys.stderr)
-                    continue
-
-                # Rename the persona directory too
-                old_path = persona.path
-                persona.name = name
-                new_path = persona.path
-
-                if old_path.exists() and old_path != new_path:
-                    shutil.move(str(old_path), str(new_path))
-
-                persona.save()
-
-            updated_count += 1
-
-            if not is_tty:
-                write_record(persona.to_json())
-
-        except Persona.DoesNotExist:
-            rprint(f"[yellow]Persona not found: {persona_id or old_name}[/yellow]", file=sys.stderr)
-            continue
-
-    rprint(f"[green]Updated {updated_count} persona(s)[/green]", file=sys.stderr)
-    return 0
-
-
-# =============================================================================
-# DELETE
-# =============================================================================
-
-
-def delete_personas(yes: bool = False, dry_run: bool = False) -> int:
-    """
-    Delete Personas from stdin JSONL.
-
-    Requires --yes flag to confirm deletion.
-
-    Exit codes:
-        0: Success
-        1: No input or missing --yes flag
-    """
-    from archivebox.misc.jsonl import read_stdin
-    from archivebox.personas.models import Persona
-
-    records = list(read_stdin())
-    if not records:
-        rprint("[yellow]No records provided via stdin[/yellow]", file=sys.stderr)
-        return 1
-
-    # Collect persona IDs or names
-    persona_ids = []
-    persona_names = []
-    for r in records:
-        if r.get("id"):
-            persona_ids.append(r["id"])
-        elif r.get("name"):
-            persona_names.append(r["name"])
-
-    if not persona_ids and not persona_names:
-        rprint("[yellow]No valid persona IDs or names in input[/yellow]", file=sys.stderr)
-        return 1
-
-    from django.db.models import Q
-
-    query = Q()
-    if persona_ids:
-        query |= Q(id__in=persona_ids)
-    if persona_names:
-        query |= Q(name__in=persona_names)
-
-    personas = Persona.objects.filter(query)
-    count = personas.count()
-
-    if count == 0:
-        rprint("[yellow]No matching personas found[/yellow]", file=sys.stderr)
-        return 0
-
-    if dry_run:
-        rprint(f"[yellow]Would delete {count} persona(s) (dry run)[/yellow]", file=sys.stderr)
-        for persona in personas:
-            rprint(f"  {persona.name} ({persona.path})", file=sys.stderr)
-        return 0
-
-    if not yes:
-        rprint("[red]Use --yes to confirm deletion[/red]", file=sys.stderr)
-        return 1
-
-    # Delete persona directories and database records
-    deleted_count = 0
-    for persona in personas:
-        persona_path = persona.path
-
-        # Safety check: ensure path is within PERSONAS_DIR before deletion
-        if not ensure_path_within_personas_dir(persona_path):
-            rprint(f'[red]Security error: persona path "{persona_path}" is outside PERSONAS_DIR. Skipping deletion.[/red]', file=sys.stderr)
-            continue
-
-        if persona_path.exists():
-            shutil.rmtree(persona_path)
-        persona.delete()
-        deleted_count += 1
-
-    rprint(f"[green]Deleted {deleted_count} persona(s)[/green]", file=sys.stderr)
-    return 0
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage Persona records (browser profiles)."""
-    pass
-
-
-@main.command("create")
-@click.argument("names", nargs=-1)
-@click.option("--import", "import_from", help="Import from chrome, chromium, brave, edge, a persona directory, or a live CDP URL")
-@click.option("--source", help="Source browser user-data directory or exact profile path (including Docker mounts)")
-@click.option(
-    "--browser-binary",
-    type=click.Path(exists=True, dir_okay=False),
-    help="Source Chromium browser executable; required for other Chromium-based browsers",
-)
-@click.option("--profile", help="Profile directory name under the user data dir (e.g. Default, Profile 1)")
-def create_cmd(names: tuple, import_from: str | None, profile: str | None, source: str | None, browser_binary: str | None):
-    """Create Personas, optionally importing from a browser profile."""
-    sys.exit(create_personas(names, import_from=import_from, profile=profile, source=source, browser_binary=browser_binary))
-
-
-@main.command("list")
-@click.option("--name", help="Filter by exact name")
-@click.option("--name__icontains", help="Filter by name contains")
-@click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List Personas as JSONL."""
-    sys.exit(list_personas(**kwargs))
-
-
-@main.command("open")
-@click.argument("name", default="Default")
-def open_cmd(name: str):
-    """Open a persona's browser, including its imported logins."""
-    import json
-    import os
-    import subprocess
-
-    import psutil
-
-    from abx_plugins import get_plugins_dir
-    from archivebox.config.common import get_config
-    from archivebox.personas.models import Persona
-
-    valid, reason = persona_importers.validate_persona_name(name)
-    if not valid:
-        raise click.ClickException(reason)
-    persona = Persona.get_or_create_named(name)
-    persona.ensure_dirs()
-    config = get_config(persona=persona)
-    if os.environ.get("DISPLAY"):
-        from archivebox.config import CONSTANTS
-
-        for process in psutil.process_iter():
-            try:
-                args = process.cmdline()
-                profile = next((arg.split("=", 1)[1] for arg in args if arg.startswith("--user-data-dir=")), None)
-                if (
-                    not profile
-                    or any(arg.startswith(("--type=", "--headless")) for arg in args)
-                    or not Path(profile).resolve().is_relative_to(CONSTANTS.DATA_DIR.resolve())
-                    or process.environ().get("DISPLAY") != os.environ["DISPLAY"]
-                ):
-                    continue
-                rprint("An ArchiveBox browser is already open on this display; waiting for it to close.")
-                process.wait()
-            except (psutil.NoSuchProcess, psutil.AccessDenied):
-                continue
-    plugins_dir = Path(get_plugins_dir()).resolve()
-    env = os.environ.copy()
-    dependencies = subprocess.run(
-        [
-            str(Path(sys.executable).with_name("abxpkg")),
-            "env",
-            "--install",
-            "--json",
-            f"--lib={config.ABXPKG_LIB_DIR}",
-            f"--deps-from={plugins_dir / 'chrome' / 'config.json'}:required_binaries",
-        ],
-        capture_output=True,
-        text=True,
-        env=env,
-    )
-    if dependencies.returncode:
-        raise click.ClickException(dependencies.stderr.strip())
-    env.update({key: str(value) for key, value in json.loads(dependencies.stdout).items()})
-    env["ARCHIVEBOX_ABX_PLUGINS_DIR"] = str(plugins_dir)
-    env["ACTIVE_PERSONA"] = persona.name
-    env["PERSONAS_DIR"] = str(persona.path.parent)
-    env["NODE_MODULES_DIR"] = str(config.ABXPKG_LIB_DIR / "pnpm" / "packages" / "chrome" / "node_modules")
-    payload = config.model_dump(mode="json")
-    payload.update(
-        ACTIVE_PERSONA=persona.name,
-        CHROME_HEADLESS=False,
-        CHROME_USER_DATA_DIR=str(persona.CHROME_USER_DATA_DIR),
-        AUTH_STORAGE_FILE=str(persona.AUTH_STORAGE_FILE or ""),
-        COOKIES_FILE=str(persona.COOKIES_FILE or ""),
-    )
-    script = Path(__file__).parent.parent / "personas" / "open_browser.js"
-    result = subprocess.run(
-        [env.get("NODE_BINARY") or "node", str(script)],
-        input=json.dumps(payload),
-        text=True,
-        env=env,
-    )
-    sys.exit(result.returncode)
-
-
-@main.command("update")
-@click.option("--name", "-n", help="Set new name")
-def update_cmd(**kwargs):
-    """Update Personas from stdin JSONL."""
-    sys.exit(update_personas(**kwargs))
-
-
-@main.command("delete")
-@click.option("--yes", "-y", is_flag=True, help="Confirm deletion")
-@click.option("--dry-run", is_flag=True, help="Show what would be deleted")
-def delete_cmd(**kwargs):
-    """Delete Personas from stdin JSONL."""
-    sys.exit(delete_personas(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,233 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-
-import rich_click as click
-
-from archivebox.misc.util import docstring, enforce_types
-
-
-EVENT_FLOW_DIAGRAM = """
-┌─────────────────────────────────────────────────────────────────────────────┐
-│                           ArchiveBox / abx-dl Flow                         │
-├─────────────────────────────────────────────────────────────────────────────┤
-│                                                                             │
-│  InstallEvent                                                               │
-│    └─ config.json > required_binaries                                       │
-│         └─ BinaryRequestEvent                                               │
-│              └─ abxpkg BinaryService builtin providers                      │
-│                   └─ BinaryEvent                                            │
-│                        └─ ArchiveBox DB history projection                  │
-│                                                                             │
-│  CrawlEvent                                                                 │
-│    └─ CrawlSetupEvent                                                       │
-│         └─ on_CrawlSetup__*                                                 │
-│                                                                             │
-│  CrawlStartEvent                                                            │
-│    └─ SnapshotEvent                                                         │
-│         └─ on_Snapshot__*                                                   │
-│              └─ ArchiveResult / Snapshot / Tag                              │
-│                                                                             │
-│  SnapshotCleanupEvent  -> internal cleanup, no direct hook family           │
-│  CrawlCleanupEvent     -> internal cleanup, no direct hook family           │
-│                                                                             │
-│  ArchiveBox projects bus events into the DB; it no longer drives plugin     │
-│  execution through the old queued model executor.                           │
-└─────────────────────────────────────────────────────────────────────────────┘
-"""
-
-
-@enforce_types
-def pluginmap(
-    show_disabled: bool = False,
-    event: str | None = None,
-    quiet: bool = False,
-) -> dict:
-    """
-    Show the current abx-dl event phases and their associated plugin hooks.
-
-    This command reflects the new bus-driven runtime, not the legacy ArchiveBox
-    event runtime. Event names are normalized to hook prefixes by
-    stripping a trailing `Event`, then ArchiveBox checks whether any matching
-    `on_{EventFamily}__*` scripts actually exist.
-    """
-    from rich.console import Console
-    from rich.table import Table
-    from rich.panel import Panel
-    from rich import box
-
-    from archivebox.plugins.hooks import (
-        discover_hooks,
-        is_background_hook,
-        normalize_hook_event_name,
-    )
-    from archivebox.plugins.discovery import (
-        BUILTIN_PLUGINS_DIR,
-        USER_PLUGINS_DIR,
-    )
-
-    console = Console()
-    prnt = console.print
-
-    event_phases = {
-        "InstallEvent": {
-            "description": "Pre-run dependency phase. Enabled plugins emit BinaryRequest events from config.json required_binaries.",
-            "emits": ["BinaryRequestEvent", "BinaryEvent", "ProcessEvent"],
-        },
-        "BinaryRequestEvent": {
-            "description": "Binary resolution phase. abxpkg BinaryService resolves or installs requested binaries using built-in providers.",
-            "emits": ["BinaryEvent", "ProcessEvent"],
-            "direct_hooks": False,
-        },
-        "BinaryEvent": {
-            "description": "Resolved binary metadata event. ArchiveBoxBinaryService projects it into the ArchiveBox DB binary cache.",
-            "emits": [],
-            "direct_hooks": False,
-        },
-        "CrawlEvent": {
-            "description": "Root crawl lifecycle event emitted by the runner.",
-            "emits": ["CrawlSetupEvent", "CrawlStartEvent", "CrawlCleanupEvent", "CrawlCompletedEvent"],
-        },
-        "CrawlSetupEvent": {
-            "description": "Crawl-scoped setup phase. on_CrawlSetup hooks launch/configure shared daemons and runtime state.",
-            "emits": ["ProcessEvent"],
-        },
-        "SnapshotEvent": {
-            "description": "Per-snapshot extraction phase. on_Snapshot hooks emit ArchiveResult, Snapshot, and Tag records.",
-            "emits": ["ArchiveResultEvent", "SnapshotEvent", "TagEvent", "ProcessEvent"],
-        },
-        "SnapshotCleanupEvent": {
-            "description": "Internal snapshot cleanup phase.",
-            "emits": ["ProcessKillEvent"],
-        },
-        "CrawlCleanupEvent": {
-            "description": "Internal crawl cleanup phase.",
-            "emits": ["ProcessKillEvent"],
-        },
-    }
-
-    if event:
-        requested = str(event).strip()
-        if requested in event_phases:
-            event_phases = {requested: event_phases[requested]}
-        else:
-            normalized_requested = normalize_hook_event_name(requested)
-            matched_name = next((name for name in event_phases if normalize_hook_event_name(name) == normalized_requested), None)
-            if matched_name is None:
-                prnt(f'[red]Error: Unknown event "{requested}". Available: {", ".join(event_phases.keys())}[/red]')
-                return {}
-            event_phases = {matched_name: event_phases[matched_name]}
-
-    result = {
-        "events": {},
-        "plugins_dir": str(BUILTIN_PLUGINS_DIR),
-        "user_plugins_dir": str(USER_PLUGINS_DIR),
-    }
-
-    if not quiet:
-        prnt()
-        prnt("[bold cyan]ArchiveBox Plugin Map[/bold cyan]")
-        prnt(f"[dim]Built-in plugins: {BUILTIN_PLUGINS_DIR}[/dim]")
-        prnt(f"[dim]User plugins: {USER_PLUGINS_DIR}[/dim]")
-        prnt()
-        prnt(
-            Panel(
-                EVENT_FLOW_DIAGRAM,
-                title="[bold green]Event Flow[/bold green]",
-                border_style="green",
-                expand=False,
-            ),
-        )
-        prnt()
-
-    for event_name, info in event_phases.items():
-        hook_event = normalize_hook_event_name(event_name)
-        has_direct_hooks = info.get("direct_hooks", True)
-        hooks = discover_hooks(event_name, filter_disabled=not show_disabled) if has_direct_hooks else []
-
-        hook_infos = []
-        for hook_path in hooks:
-            plugin_name = hook_path.parent.name
-            hook_infos.append(
-                {
-                    "path": str(hook_path),
-                    "name": hook_path.name,
-                    "plugin": plugin_name,
-                    "is_background": is_background_hook(hook_path.name),
-                    "extension": hook_path.suffix,
-                },
-            )
-
-        result["events"][event_name] = {
-            "description": info["description"],
-            "hook_event": hook_event,
-            "emits": info["emits"],
-            "hooks": hook_infos,
-            "hook_count": len(hook_infos),
-        }
-
-        if quiet:
-            continue
-
-        title_suffix = f" -> on_{hook_event}__*" if hook_infos else ""
-        table = Table(
-            title=f"[bold yellow]{event_name}[/bold yellow]{title_suffix} ({len(hooks)} hooks)",
-            box=box.ROUNDED,
-            show_header=True,
-            header_style="bold magenta",
-        )
-        table.add_column("Plugin", style="cyan", width=20)
-        table.add_column("Hook Name", style="green")
-        table.add_column("BG", justify="center", width=4)
-        table.add_column("Type", justify="center", width=5)
-
-        if hook_infos:
-            for hook in sorted(hook_infos, key=lambda h: h["name"]):
-                bg_marker = "[yellow]bg[/yellow]" if hook["is_background"] else ""
-                table.add_row(
-                    hook["plugin"],
-                    hook["name"],
-                    bg_marker,
-                    hook["extension"].lstrip("."),
-                )
-        else:
-            table.add_row("[dim]-[/dim]", "[dim]No direct hooks[/dim]", "", "")
-
-        prnt(table)
-        prnt(f"[dim]{info['description']}[/dim]")
-        if info["emits"]:
-            prnt(f"[dim]Emits: {', '.join(info['emits'])}[/dim]")
-        if not hook_infos and has_direct_hooks:
-            prnt(f"[dim]No direct on_{hook_event}__* scripts are currently defined for this event family.[/dim]")
-        elif not has_direct_hooks:
-            prnt("[dim]No direct plugin hook family. This event is handled by services.[/dim]")
-        prnt()
-
-    if not quiet:
-        total_hooks = sum(event_info["hook_count"] for event_info in result["events"].values())
-        prnt(f"[bold]Total hooks discovered: {total_hooks}[/bold]")
-        prnt()
-        prnt("[dim]Hook naming convention: on_{EventFamily}__{XX}_{description}[.bg].{ext}[/dim]")
-        prnt("[dim]Event names are normalized with a simple `Event` suffix strip before hook discovery.[/dim]")
-        prnt("[dim]If no `on_{EventFamily}__*` scripts exist, the event is shown as having no direct hooks.[/dim]")
-        prnt()
-
-    return result
-
-
-@click.command()
-@click.option("--show-disabled", "-a", is_flag=True, help="Show hooks from disabled plugins too")
-@click.option("--event", "-e", type=str, default=None, help="Filter to specific event (e.g. InstallEvent, SnapshotEvent)")
-@click.option("--quiet", "-q", is_flag=True, help="Output JSON only, no tables")
-@docstring(pluginmap.__doc__)
-def main(**kwargs):
-    import json
-
-    result = pluginmap(**kwargs)
-    if kwargs.get("quiet"):
-        print(json.dumps(result, indent=2))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,77 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox process <action> [--filters]
-
-Manage Process records (system-managed, mostly read-only).
-
-Process records track executions of binaries during extraction.
-They are created automatically by the system and are primarily for debugging.
-
-Actions:
-    list    - List Processes as JSONL (with optional filters)
-
-Examples:
-    # List all processes
-    archivebox process list
-
-    # List processes by binary
-    archivebox process list --binary-name=chrome
-
-    # List recent processes
-    archivebox process list --limit=10
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox process"
-
-import sys
-
-import rich_click as click
-
-from archivebox.cli.cli_util import apply_filters
-
-
-# =============================================================================
-# LIST
-# =============================================================================
-
-
-def list_processes(binary_name: str | None = None, machine_id: str | None = None, limit: int | None = None) -> int:
-    """List processes as JSONL, or exit codes and binary names in a terminal."""
-    from archivebox.machine.models import Process
-    from archivebox.cli.cli_util import list_records
-
-    def render(process):
-        name = process.binary.name if process.binary else "unknown"
-        exit_code = process.exit_code if process.exit_code is not None else "?"
-        color = "green" if process.exit_code == 0 else "red" if process.exit_code else "yellow"
-        return f"[{color}]exit={exit_code:3}[/{color}] [cyan]{name:15}[/cyan] [dim]{process.id}[/dim]"
-
-    queryset = Process.objects.select_related("binary", "machine").order_by("-started_at", "-created_at")
-    filters = {"binary__name": binary_name or None, "machine_id": machine_id or None}
-    return list_records(apply_filters(queryset, filters, limit=limit), plural="processes", render=render)
-
-
-# =============================================================================
-# CLI Commands
-# =============================================================================
-
-
-@click.group()
-def main():
-    """Manage Process records (read-only, system-managed)."""
-    pass
-
-
-@main.command("list")
-@click.option("--binary-name", "-b", help="Filter by binary name")
-@click.option("--machine-id", "-m", help="Filter by machine ID")
-@click.option("--limit", "-n", type=int, help="Limit number of results")
-def list_cmd(**kwargs):
-    """List Processes as JSONL."""
-    sys.exit(list_processes(**kwargs))
-
-
-if __name__ == "__main__":
-    main()
@@ -1,137 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox remove"
-
-import time
-from contextlib import nullcontext
-from pathlib import Path
-from collections.abc import Iterable
-
-import rich_click as click
-from rich.console import Console
-
-from django.db.models import QuerySet
-
-from archivebox.config import CONSTANTS
-from archivebox.config.common import get_config
-from archivebox.config.django import setup_django
-from archivebox.misc.util import enforce_types, docstring
-from archivebox.misc.checks import check_data_folder
-from archivebox.misc.logging_util import (
-    log_list_started,
-    log_list_finished,
-    log_removal_started,
-    log_removal_finished,
-)
-from archivebox.cli.archivebox_snapshot import snapshot_filter_options
-
-
-@enforce_types
-def remove(
-    filter_patterns: Iterable[str] = (),
-    filter_type: str = "exact",
-    snapshots: QuerySet | None = None,
-    after: float | None = None,
-    before: float | None = None,
-    yes: bool = False,
-    delete: bool = False,
-    out_dir: Path = CONSTANTS.DATA_DIR,
-    timeout: float | None = None,
-    **kwargs,
-) -> dict[str, object]:
-    """Remove the specified URLs from the archive"""
-
-    setup_django()
-    check_data_folder()
-    timeout = float(timeout) if timeout is not None else None
-
-    from archivebox.core.models import Snapshot
-
-    filter_kwargs = {
-        **kwargs,
-        "filter_patterns": filter_patterns,
-        "filter_type": filter_type,
-        "after": after,
-        "before": before,
-    }
-    pattern_list = list(filter_patterns)
-
-    log_list_started(pattern_list or None, filter_type)
-    progress = Console().status("Finding matching snapshots...") if get_config().SHOW_PROGRESS else nullcontext()
-    with progress:
-        if snapshots is None:
-            snapshots = Snapshot.objects.order_by("-created_at").search(**filter_kwargs)
-        # Freeze the target set up-front so a concurrent daemon writing new
-        # snapshots can't extend the deletion under us, and so the cursor isn't
-        # held open across the per-row deletes below.
-        snapshot_pks = list(snapshots.values_list("pk", flat=True))
-
-    if not snapshot_pks:
-        log_removal_finished(0, 0)
-        raise SystemExit(1)
-
-    if not yes:
-        log_list_finished(snapshots)
-        log_removal_started(snapshots, yes=False)
-
-    from archivebox.search.query import flush_search_index
-
-    started_at = time.monotonic()
-    deadline = started_at + timeout if timeout is not None else None
-
-    # Search-index flush touches a separate backend (FTS / sonic), not the
-    # main index.sqlite3 writer lock, so it's safe to do once up front.
-    flush_search_index(snapshots=snapshots)
-
-    # Delete one snapshot at a time. Each ``.delete()`` is its own short
-    # Django-atomic block, so the writer lock is released between rows and
-    # an in-flight daemon transaction can interleave instead of deadlocking.
-    # Filesystem cleanup for each row is scheduled via ``transaction.on_commit``
-    # in ``base_models/models.py`` and runs AFTER its row's tx commits — so
-    # rmtree doesn't hold the lock either.
-    #
-    deleted_snapshot_pks = []
-    timed_out = False
-    timeout_error = ""
-    for index, pk in enumerate(snapshot_pks):
-        if deadline is not None and time.monotonic() >= deadline:
-            timed_out = True
-            timeout_error = f"Remove timed out after {timeout:g}s with {len(snapshot_pks) - index} snapshots remaining."
-            break
-        deleted_count, _ = Snapshot.objects.filter(pk=pk).delete()
-        if deleted_count:
-            deleted_snapshot_pks.append(pk)
-
-    all_snapshots = Snapshot.objects.all()
-    remaining_count = all_snapshots.count()
-    deleted_snapshot_id_set = set(deleted_snapshot_pks)
-    remaining_snapshot_pks = [snapshot_id for snapshot_id in snapshot_pks if snapshot_id not in deleted_snapshot_id_set]
-    log_removal_finished(remaining_count, len(deleted_snapshot_pks))
-
-    return {
-        "removed_count": len(deleted_snapshot_pks),
-        "removed_snapshot_ids": [str(snapshot_id) for snapshot_id in deleted_snapshot_pks],
-        "not_removed_count": len(remaining_snapshot_pks),
-        "not_removed_snapshot_ids": [str(snapshot_id) for snapshot_id in remaining_snapshot_pks],
-        "success": not timed_out,
-        "error": timeout_error,
-        "timeout": timeout,
-    }
-
-
-@click.command()
-@click.option("--yes", is_flag=True, help="Remove links instantly without prompting to confirm")
-@click.option("--delete", is_flag=True, help="Compatibility no-op; archived content is deleted automatically")
-@click.option("--timeout", type=float, default=None, help="Maximum seconds to spend deleting snapshots")
-@snapshot_filter_options(default_filter_type="exact")
-@docstring(remove.__doc__)
-def main(**kwargs):
-    """Remove the specified URLs from the archive"""
-    result = remove(**kwargs)
-    if not result["success"]:
-        raise SystemExit(124)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,436 +0,0 @@
-#!/usr/bin/env python3
-
-"""
-archivebox run [--daemon] [--crawl-id=...] [--snapshot-id=...] [--binary-id=...]
-
-Unified command for processing queued work on the shared abx-dl bus.
-
-Modes:
-    - With stdin JSONL: Process piped records, exit when complete
-    - Without stdin (TTY): Run the background runner in foreground until killed
-    - --crawl-id: Run the crawl runner for a specific crawl only
-    - --snapshot-id: Run a specific snapshot through its parent crawl
-    - --binary-id: Emit a BinaryRequestEvent for a specific Binary row
-
-Examples:
-    # Run the background runner in foreground
-    archivebox run
-
-    # Run as daemon (don't exit on idle)
-    archivebox run --daemon
-
-    # Process specific records (pipe any JSONL type, exits when done)
-    archivebox snapshot list --status=queued | archivebox run
-    archivebox archiveresult list --status=failed | archivebox run
-    archivebox crawl list --status=queued | archivebox run
-
-    # Mixed types work too
-    cat mixed_records.jsonl | archivebox run
-
-    # Run the crawl runner for a specific crawl
-    archivebox run --crawl-id=019b7e90-04d0-73ed-adec-aad9cfcd863e
-
-    # Run one snapshot from an existing crawl
-    archivebox run --snapshot-id=019b7e90-5a8e-712c-9877-2c70eebe80ad
-
-    # Run one queued binary install directly on the bus
-    archivebox run --binary-id=019b7e90-5a8e-712c-9877-2c70eebe80ad
-"""
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox run"
-
-import asyncio
-import os
-import signal
-import sys
-from collections import defaultdict
-
-import rich_click as click
-from rich import print as rprint
-
-
-RUNNER_DAEMON_ENV = "ARCHIVEBOX_RUNNER_DAEMON"
-
-
-def _exit_daemon_runner_on_signal(sig: signal.Signals) -> None:
-    # A supervised `archivebox run --daemon` is intentionally a disposable
-    # child. If it receives SIGINT/SIGTERM directly, exit with the conventional
-    # signal status so supervisord treats it as an unexpected worker death and
-    # restarts only the runner. The parent `archivebox server` owns supervisord
-    # shutdown and must not be pulled down by a killed daemon worker.
-    os._exit(128 + int(sig))
-
-
-def process_stdin_records() -> int:
-    """
-    Process JSONL records from stdin.
-
-    Create-or-update behavior:
-    - Records WITHOUT id: Create via Model.from_json(), then queue
-    - Records WITH id: Lookup existing, re-queue for processing
-
-    Outputs JSONL of all processed records (for chaining).
-
-    Handles Crawl and Snapshot work records. ArchiveResult records are accepted
-    as references to their parent Snapshot and plugin.
-
-    Returns exit code (0 = success, 1 = error).
-    """
-    from django.utils import timezone
-
-    from archivebox.misc.jsonl import (
-        read_stdin,
-        write_record,
-        TYPE_CRAWL,
-        TYPE_SNAPSHOT,
-        TYPE_ARCHIVERESULT,
-        TYPE_BINARYREQUEST,
-        TYPE_BINARY,
-    )
-    from archivebox.base_models.models import get_or_create_system_user_pk
-    from archivebox.core.models import Snapshot, ArchiveResult
-    from archivebox.misc.db import uuid_ref_query
-    from archivebox.crawls.models import Crawl
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-    from archivebox.machine.models import Binary
-    from archivebox.services.runner import run_binary, run_crawl
-
-    records = list(read_stdin())
-    is_tty = sys.stdout.isatty()
-
-    if not records:
-        return 0  # Nothing to process
-
-    created_by_id = get_or_create_system_user_pk()
-    queued_count = 0
-    output_records = []
-    full_crawl_ids: set[str] = set()
-    snapshot_ids_by_crawl: dict[str, set[str]] = defaultdict(set)
-    plugin_names_by_crawl: dict[str, set[str]] = defaultdict(set)
-    run_all_plugins_for_crawl: set[str] = set()
-    binary_ids: list[str] = []
-
-    for record in records:
-        record_type = record.get("type", "")
-        record_id = record.get("id")
-
-        try:
-            if record_type == TYPE_CRAWL:
-                crawl = Crawl.from_json(record, overrides={"created_by_id": created_by_id})
-
-                if crawl:
-                    crawl.update_and_requeue(
-                        status=Crawl.StatusChoices.QUEUED,
-                        retry_at=timezone.now(),
-                    )
-                    full_crawl_ids.add(str(crawl.id))
-                    run_all_plugins_for_crawl.add(str(crawl.id))
-                    output_records.append(crawl.to_json())
-                    queued_count += 1
-
-            elif record_type == TYPE_SNAPSHOT or (record.get("url") and not record_type):
-                snapshot = Snapshot.objects.filter(id=record_id).first() if record_id else None
-                if snapshot is None:
-                    snapshot = Snapshot.from_json(record, overrides={"created_by_id": created_by_id})
-
-                if snapshot:
-                    snapshot.queue_for_extraction()
-                    crawl_id = str(snapshot.crawl_id)
-                    snapshot_ids_by_crawl[crawl_id].add(str(snapshot.id))
-                    run_all_plugins_for_crawl.add(crawl_id)
-                    output_records.append(snapshot.to_json())
-                    queued_count += 1
-
-            elif record_type == TYPE_ARCHIVERESULT:
-                snapshot_id = str(record.get("snapshot_id") or "")
-                plugin_name = str(record.get("plugin") or "")
-                archiveresult = None
-                if not snapshot_id and record_id:
-                    archiveresult = ArchiveResult.objects.filter(uuid_ref_query("id", str(record_id))).select_related("snapshot").first()
-                    if archiveresult:
-                        snapshot_id = str(archiveresult.snapshot_id)
-                        plugin_name = plugin_name or archiveresult.plugin
-                snapshot = Snapshot.objects.filter(id=snapshot_id).first() if snapshot_id else None
-
-                if snapshot:
-                    snapshot.queue_for_extraction()
-                    crawl_id = str(snapshot.crawl_id)
-                    snapshot_ids_by_crawl[crawl_id].add(str(snapshot.id))
-                    if plugin_name:
-                        plugin_names_by_crawl[crawl_id].add(str(plugin_name))
-                    output_records.append(archiveresult.to_json() if archiveresult else record)
-                    queued_count += 1
-
-            elif record_type in {TYPE_BINARYREQUEST, TYPE_BINARY}:
-                binary = Binary.objects.filter(id=record_id).first() if record_id else None
-                if binary is None:
-                    binary = Binary.from_json(record)
-
-                if binary:
-                    binary.retry_at = timezone.now()
-                    if binary.status != Binary.StatusChoices.INSTALLED:
-                        binary.status = Binary.StatusChoices.QUEUED
-                    binary.save()
-                    binary_ids.append(str(binary.id))
-                    output_records.append(binary.to_json())
-                    queued_count += 1
-
-            else:
-                # Unknown type - pass through
-                output_records.append(record)
-
-        except Exception as e:
-            rprint(f"[yellow]Error processing record: {e}[/yellow]", file=sys.stderr)
-            continue
-
-    # Output all processed records (for chaining)
-    if not is_tty:
-        for rec in output_records:
-            write_record(rec)
-
-    if queued_count == 0:
-        rprint("[yellow]No records to process[/yellow]", file=sys.stderr)
-        return 0
-
-    rprint(f"[blue]Processing {queued_count} records...[/blue]", file=sys.stderr)
-
-    for binary_id in binary_ids:
-        run_binary(binary_id)
-
-    targeted_crawl_ids = full_crawl_ids | set(snapshot_ids_by_crawl)
-    if targeted_crawl_ids:
-        for crawl_id in sorted(targeted_crawl_ids):
-            try:
-                crawl = Crawl.objects.get(id=crawl_id)
-            except Crawl.DoesNotExist:
-                continue
-            if not crawl.claim_processing_lock(lock_seconds=10):
-                rprint(f"[yellow]Crawl {crawl_id} is already owned by another runner[/yellow]", file=sys.stderr)
-                return 1
-            with foreground_shutdown_signals(), foreground_parent_watchdog():
-                run_crawl(
-                    crawl_id,
-                    snapshot_ids=None if crawl_id in full_crawl_ids else sorted(snapshot_ids_by_crawl[crawl_id]),
-                    selected_plugins=None if crawl_id in run_all_plugins_for_crawl else sorted(plugin_names_by_crawl[crawl_id]),
-                )
-    return 0
-
-
-def run_runner(
-    daemon: bool = False,
-    crawl_id: str | None = None,
-    maintenance_only: bool = False,
-) -> int:
-    """
-    Run the background runner loop.
-
-    Args:
-        daemon: Run forever (don't exit when idle)
-
-    Returns exit code (0 = success, 1 = error).
-    """
-    from archivebox.config import CONSTANTS
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-    from archivebox.machine.models import Machine, Process
-    from archivebox.core.takeover_util import enter_single_runner_gate, standby_until_foreground_runner_needed
-    from archivebox.core.recovery_util import recover_orchestrator_state
-    from archivebox.services.runner import run_pending_crawls
-
-    Machine.current()
-    current = Process.current()
-    root_command = current.root
-    if daemon and root_command.process_type in (
-        Process.TypeChoices.SERVER,
-        Process.TypeChoices.ADD,
-        Process.TypeChoices.UPDATE,
-    ):
-        # Server-owned daemon runners are persistent supervisor workers, but
-        # foreground add/update commands are allowed to borrow runner/sonic
-        # leadership without taking down Daphne. Waiting here keeps the worker
-        # on the normal runner path while preventing a server restart loop from
-        # immediately stealing the single-runner gate back from the newer CLI.
-        standby_until_foreground_runner_needed(root_command, data_dir=CONSTANTS.DATA_DIR)
-    if not enter_single_runner_gate(current, data_dir=CONSTANTS.DATA_DIR):
-        current.mark_exited()
-        return 0
-
-    recover_orchestrator_state(include_chrome=crawl_id is None, crawl_id=crawl_id)
-    if crawl_id:
-        from django.utils import timezone
-        from archivebox.crawls.models import Crawl
-
-        crawl = Crawl.objects.filter(id=crawl_id, status__in=Crawl.RUNNABLE_STATES).first()
-        now = timezone.now()
-        # Winning the single-runner gate terminates and waits for every older
-        # runner. Requeue the explicitly requested crawl so an abandoned
-        # future ownership lease cannot hide it from the unified scheduler.
-        if crawl is not None:
-            crawl.update_and_requeue(retry_at=now, refresh=False)
-    # Only a foreground `archivebox add` gets the interactive "abort current
-    # hook, continue/retry, second Ctrl+C exits" flow. Server/update/run owned
-    # orchestrators should shut down immediately and cleanly on the first signal.
-    interactive_interrupts = current.root.process_type == Process.TypeChoices.ADD
-    if daemon:
-        os.environ[RUNNER_DAEMON_ENV] = "1"
-
-    try:
-        with (
-            foreground_shutdown_signals(
-                on_signal=_exit_daemon_runner_on_signal if daemon else None,
-                raise_on_first_signal=not daemon,
-            ),
-            foreground_parent_watchdog(enabled=not daemon),
-        ):
-            run_pending_crawls(
-                daemon=daemon,
-                crawl_id=crawl_id,
-                maintenance_only=maintenance_only,
-                interactive_interrupts=interactive_interrupts,
-            )
-        return 0
-    except KeyboardInterrupt:
-        return 0
-    except asyncio.CancelledError as e:
-        if daemon:
-            rprint(f"[red]Runner cancelled unexpectedly: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-            return 1
-        return 0
-    except Exception as e:
-        rprint(f"[red]Runner error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-        return 1
-    finally:
-        current.refresh_from_db()
-        if current.status != Process.StatusChoices.EXITED:
-            current.mark_exited()
-
-
-@click.command()
-@click.option("--daemon", "-d", is_flag=True, help="Run forever (don't exit on idle)")
-@click.option("--crawl-id", help="Run the crawl runner for a specific crawl only")
-@click.option("--snapshot-id", help="Run one snapshot through its crawl")
-@click.option("--binary-id", help="Run one queued binary install directly on the bus")
-@click.option("--maintenance-only", is_flag=True, help="Only process sealed Snapshot maintenance and search-index backfills")
-@click.option("--no-stdin", is_flag=True, hidden=True, help="Run the scheduler even when stdin is not a TTY")
-def main(
-    daemon: bool,
-    crawl_id: str,
-    snapshot_id: str,
-    binary_id: str,
-    maintenance_only: bool,
-    no_stdin: bool,
-):
-    """
-    Process queued work.
-
-    Modes:
-    - No args + stdin piped: Process piped JSONL records
-    - No args + TTY: Run the crawl runner for all work
-    - --crawl-id: Run the crawl runner for that crawl only
-    - --snapshot-id: Run one snapshot through its crawl only
-    - --binary-id: Run one queued binary install directly on the bus
-    """
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-
-    if daemon and not snapshot_id and not binary_id and not crawl_id:
-        try:
-            os.environ[RUNNER_DAEMON_ENV] = "1"
-            with (
-                foreground_shutdown_signals(
-                    on_signal=_exit_daemon_runner_on_signal,
-                    raise_on_first_signal=False,
-                ),
-                foreground_parent_watchdog(enabled=False),
-            ):
-                sys.exit(run_runner(daemon=True, maintenance_only=maintenance_only))
-        except KeyboardInterrupt:
-            sys.exit(0)
-
-    with foreground_shutdown_signals(), foreground_parent_watchdog(enabled=not daemon):
-        if snapshot_id:
-            sys.exit(run_snapshot_worker(snapshot_id))
-
-        if binary_id:
-            try:
-                from archivebox.services.runner import run_binary
-
-                run_binary(binary_id)
-                sys.exit(0)
-            except KeyboardInterrupt:
-                sys.exit(0)
-            except Exception as e:
-                rprint(f"[red]Runner error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-                import traceback
-
-                traceback.print_exc()
-                sys.exit(1)
-
-        if crawl_id:
-            sys.exit(
-                run_runner(
-                    daemon=False,
-                    crawl_id=crawl_id,
-                    maintenance_only=maintenance_only,
-                ),
-            )
-
-        if maintenance_only:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=True))
-
-        if not no_stdin and not sys.stdin.isatty():
-            sys.exit(process_stdin_records())
-        else:
-            sys.exit(run_runner(daemon=daemon, maintenance_only=maintenance_only))
-
-
-def run_snapshot_worker(snapshot_id: str) -> int:
-    from archivebox.config import CONSTANTS
-    from archivebox.core.takeover_util import enter_single_runner_gate
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-    from archivebox.machine.models import Process
-    from archivebox.core.models import Snapshot
-    from archivebox.services.runner import run_due_snapshot
-    from django.utils import timezone
-
-    current = Process.current()
-    if not enter_single_runner_gate(current, data_dir=CONSTANTS.DATA_DIR):
-        current.mark_exited()
-        return 0
-
-    snapshot = None
-    try:
-        with foreground_shutdown_signals(), foreground_parent_watchdog():
-            for _ in range(10):
-                snapshot = Snapshot.objects.select_related("crawl").get(id=snapshot_id)
-                if snapshot.retry_at is None:
-                    snapshot.update_and_requeue(retry_at=timezone.now())
-                elif snapshot.retry_at > timezone.now():
-                    break
-                if not run_due_snapshot(snapshot, lock_seconds=60):
-                    break
-        return 0
-    except KeyboardInterrupt:
-        try:
-            if snapshot is not None:
-                snapshot.refresh_from_db()
-            else:
-                snapshot = Snapshot.objects.filter(id=snapshot_id).first()
-            if snapshot is not None and snapshot.status != Snapshot.StatusChoices.SEALED:
-                snapshot.update_and_requeue(retry_at=timezone.now())
-        except Exception:
-            pass
-        return 0
-    except Exception as e:
-        rprint(f"[red]Runner error: {type(e).__name__}: {e}[/red]", file=sys.stderr)
-        import traceback
-
-        traceback.print_exc()
-        return 1
-    finally:
-        current.refresh_from_db()
-        if current.status != Process.StatusChoices.EXITED:
-            current.mark_exited()
-
-
-if __name__ == "__main__":
-    main()
@@ -1,194 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import rich_click as click
-from rich import print
-
-from archivebox.misc.util import enforce_types, docstring
-
-
-@enforce_types
-def schedule(
-    add: bool = False,
-    show: bool = False,
-    clear: bool = False,
-    foreground: bool = False,
-    run_all: bool = False,
-    quiet: bool = False,
-    every: str | None = None,
-    tag: str = "",
-    depth: int | str = 0,
-    import_path: str | None = None,
-    config: dict[str, object] | None = None,
-):
-    """Manage database-backed scheduled crawls processed by the crawl runner."""
-
-    from django.utils import timezone
-
-    from archivebox.base_models.models import get_or_create_system_user_pk
-    from archivebox.crawls.models import Crawl, CrawlSchedule
-    from archivebox.crawls.schedule_util import validate_schedule
-    from archivebox.services.runner import run_pending_crawls
-
-    config_overrides = dict(config or {})
-    depth = int(depth)
-    result: dict[str, object] = {
-        "created_schedule_ids": [],
-        "disabled_count": 0,
-        "run_all_enqueued": 0,
-        "run_all_maintained": 0,
-        "active_schedule_ids": [],
-    }
-
-    def _active_schedules():
-        return CrawlSchedule.objects.filter(is_enabled=True).select_related("template").order_by("created_at")
-
-    if clear:
-        disabled_count = CrawlSchedule.objects.filter(is_enabled=True).update(
-            is_enabled=False,
-            modified_at=timezone.now(),
-        )
-        result["disabled_count"] = disabled_count
-        print(f"[green]\\[√] Disabled {disabled_count} scheduled crawl(s).[/green]")
-
-    if every or add:
-        schedule_str = (every or "day").strip()
-        validate_schedule(schedule_str)
-
-        created_by_id = get_or_create_system_user_pk()
-        is_update_schedule = not import_path
-        template_urls = import_path or ""
-        template_label = (f"Scheduled import: {template_urls}" if import_path else "Scheduled ArchiveBox update")[:64]
-        template_notes = (
-            f"Created by archivebox schedule for {template_urls}"
-            if import_path
-            else "Created by archivebox schedule to run recurring ArchiveBox maintenance."
-        )
-
-        template = Crawl.objects.create(
-            urls=template_urls,
-            max_depth=0 if is_update_schedule else depth,
-            tags_str="" if is_update_schedule else tag,
-            label=template_label,
-            notes=template_notes,
-            created_by_id=created_by_id,
-            status=Crawl.StatusChoices.SEALED,
-            retry_at=None,
-            config={
-                "DEPTH": 0 if is_update_schedule else depth,
-                # Caller-supplied overrides (e.g. {"ONLY_NEW": False}) win over the
-                # template defaults. Anything left unset falls through to the
-                # standard config stack at crawl-resolution time.
-                **config_overrides,
-            },
-        )
-        crawl_schedule = CrawlSchedule.objects.create(
-            template=template,
-            schedule=schedule_str,
-            config={**template.config, "SCHEDULE_KIND": "update" if is_update_schedule else "crawl"},
-            is_enabled=True,
-            label=template_label,
-            notes=template_notes,
-            created_by_id=created_by_id,
-        )
-        result["created_schedule_ids"] = [str(crawl_schedule.id)]
-
-        schedule_type = "maintenance update" if is_update_schedule else "crawl"
-        print(f"[green]\\[√] Created scheduled {schedule_type}.[/green]")
-        print(f"    id={crawl_schedule.id}")
-        print(f"    every={crawl_schedule.schedule}")
-        print(f"    next_run={crawl_schedule.next_run_at.isoformat()}")
-        if import_path:
-            print(f"    source={import_path}")
-
-    schedules = list(_active_schedules())
-    result["active_schedule_ids"] = [str(schedule.id) for schedule in schedules]
-
-    if show:
-        if schedules:
-            print(f"[green]\\[*] Active scheduled crawls: {len(schedules)}[/green]")
-            for scheduled_crawl in schedules:
-                template = scheduled_crawl.template
-                print(
-                    f"  - id={scheduled_crawl.id} every={scheduled_crawl.schedule} "
-                    f"next_run={scheduled_crawl.next_run_at.isoformat()} "
-                    f"source={template.urls.splitlines()[0] if template.urls else ''}",
-                )
-        else:
-            print("[yellow]\\[*] No scheduled crawls are enabled.[/yellow]")
-
-    if run_all:
-        enqueued = 0
-        maintained = 0
-        now = timezone.now()
-        for scheduled_crawl in schedules:
-            if scheduled_crawl.dispatch(queued_at=now) is None:
-                maintained += 1
-            else:
-                enqueued += 1
-        result["run_all_enqueued"] = enqueued
-        result["run_all_maintained"] = maintained
-        print(f"[green]\\[*] Enqueued {enqueued} scheduled crawl(s) immediately.[/green]")
-        if maintained:
-            run_pending_crawls(maintenance_only=True)
-            print(f"[green]\\[*] Ran {maintained} scheduled maintenance update(s).[/green]")
-        if enqueued:
-            print(
-                "[yellow]\\[*] Start `archivebox server`, `archivebox run --daemon`, or `archivebox schedule --foreground` to process the queued crawls.[/yellow]",
-            )
-
-    if foreground:
-        print(
-            "[green]\\[*] Starting global crawl runner in foreground mode. It will materialize scheduled crawls and process queued work.[/green]",
-        )
-        run_pending_crawls(daemon=True)
-
-    if quiet:
-        return result
-
-    if not any((every, add, show, clear, foreground, run_all)):
-        if schedules:
-            print("[green]\\[*] Active scheduled crawls:[/green]")
-            for scheduled_crawl in schedules:
-                print(f"  - {scheduled_crawl.id} every={scheduled_crawl.schedule} next_run={scheduled_crawl.next_run_at.isoformat()}")
-        else:
-            print("[yellow]\\[*] No scheduled crawls are enabled.[/yellow]")
-
-    return result
-
-
-@click.command()
-@click.option("--quiet", "-q", is_flag=True, help="Return structured results without extra summary output")
-@click.option("--add", is_flag=True, help="Create a new scheduled crawl")
-@click.option("--every", type=str, help='Run on an alias like daily/weekly/monthly or a cron expression such as "0 */6 * * *"')
-@click.option("--tag", "-t", default="", help="Comma-separated tags to apply to scheduled crawl snapshots")
-@click.option(
-    "--depth",
-    type=click.Choice([str(i) for i in range(5)]),
-    default="0",
-    help="Recursively archive linked pages up to N hops away",
-)
-@click.option(
-    "--only-new/--no-only-new",
-    "only_new",
-    default=None,
-    help="Skip URLs that already have a snapshot (default: inherit from ONLY_NEW config). "
-    "Pass --no-only-new to force re-archive on each scheduled run.",
-)
-@click.option("--clear", is_flag=True, help="Disable all currently enabled schedules")
-@click.option("--show", is_flag=True, help="Print all currently enabled schedules")
-@click.option("--foreground", "-f", is_flag=True, help="Run the global crawl runner in the foreground (no crontab required)")
-@click.option("--run-all", is_flag=True, help="Dispatch all enabled schedules immediately")
-@click.argument("import_path", required=False)
-@docstring(schedule.__doc__)
-def main(**kwargs):
-    """Manage database-backed scheduled crawls processed by the crawl runner."""
-    only_new = kwargs.pop("only_new", None)
-    if only_new is not None:
-        kwargs["config"] = {"ONLY_NEW": bool(only_new)}
-    schedule(**kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,10 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-__command__ = "archivebox search"
-
-from archivebox.cli.archivebox_list import main
-
-
-if __name__ == "__main__":
-    main()
@@ -1,437 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-import sys
-import os
-import socket
-import subprocess
-import time
-from collections.abc import Iterable
-
-import rich_click as click
-from rich import print
-from rich.style import Style
-from rich.text import Text
-
-from archivebox.config import CONSTANTS
-from archivebox.misc.util import docstring, enforce_types
-
-
-import re as _re
-
-_IPV4_RE = _re.compile(r"^\d{1,3}(?:\.\d{1,3}){3}$")
-_IPV6_CHARS_RE = _re.compile(r"^[0-9a-fA-F:.]+$")
-_LOCAL_BIND_HOSTS = frozenset({"0.0.0.0", "::", "::0", "127.0.0.1", "::1"})
-
-
-def _is_ipv4_literal(host: str) -> bool:
-    return bool(_IPV4_RE.match(host))
-
-
-def _is_ipv6_literal(host: str) -> bool:
-    # Bracketed (e.g. ``[2001:db8::1]``) or bare form. Require at least two
-    # colons so we don't catch random strings with one ``:``.
-    stripped = host.strip("[]")
-    return stripped.count(":") >= 2 and bool(_IPV6_CHARS_RE.match(stripped))
-
-
-def _bind_host_looks_like_ip(host: str) -> bool:
-    if not host or host in _LOCAL_BIND_HOSTS:
-        return False
-    return _is_ipv4_literal(host) or _is_ipv6_literal(host)
-
-
-def _split_bind_spec(spec: str) -> tuple[str, str]:
-    """Split a ``host:port`` / ``host`` / ``port`` spec into ``(host, port)``.
-
-    The empty strings stand in for "not provided"; the caller fills in
-    defaults. Bracketed IPv6 literals like ``[::1]:8000`` are handled.
-    """
-    spec = (spec or "").strip()
-    if not spec:
-        return "", ""
-    if spec.startswith("["):
-        # Bracketed IPv6: ``[::1]`` or ``[::1]:8000``
-        end = spec.find("]")
-        if end == -1:
-            return spec, ""  # malformed; let validator reject it
-        host = spec[: end + 1]
-        rest = spec[end + 1 :]
-        if rest.startswith(":"):
-            return host, rest[1:]
-        return host, ""
-    if ":" in spec:
-        host, _, port = spec.rpartition(":")
-        return host, port
-    # Bare token: digits = port, anything else = host
-    if spec.isdigit():
-        return "", spec
-    return spec, ""
-
-
-def _parse_and_validate_bind_spec(spec: str) -> tuple[str, str]:
-    """Resolve a CLI/config bind spec to ``(host, port)`` or hard-error.
-
-    Accepts only IP literals (v4 or v6) or the special string ``localhost``
-    (normalized to ``127.0.0.1``). Bare hostnames are rejected because the
-    bind address feeds Daphne, which has to listen on a numeric address;
-    public hostnames belong in ``BASE_URL`` instead. Empty values fall back
-    to ``127.0.0.1`` / ``8000``.
-    """
-    raw_host, raw_port = _split_bind_spec(spec)
-    host = raw_host.strip()
-    port = (raw_port or "").strip() or "8000"
-
-    if host == "" or host.lower() == "localhost":
-        host = "127.0.0.1"
-    elif _is_ipv4_literal(host) or _is_ipv6_literal(host):
-        pass
-    else:
-        print(
-            f"[red][X] Invalid BIND_ADDR host {host!r}: must be an IP literal or 'localhost'.[/red]",
-        )
-        print(
-            "[red]    Hostnames like archive.example.com are not valid bind addresses — Daphne[/red]",
-        )
-        print(
-            "[red]    listens on numeric addresses only. Bind to 0.0.0.0 and set BASE_URL instead:[/red]",
-        )
-        print(
-            f"[red]      BASE_URL=https://{host} archivebox server 0.0.0.0:{port}[/red]",
-        )
-        sys.exit(1)
-
-    try:
-        port_int = int(port)
-    except ValueError:
-        print(f"[red][X] Invalid BIND_ADDR port {port!r}: must be an integer 1-65535.[/red]")
-        sys.exit(1)
-    if not (0 < port_int < 65536):
-        print(f"[red][X] Invalid BIND_ADDR port {port_int}: must be 1-65535.[/red]")
-        sys.exit(1)
-
-    return host, port
-
-
-def _print_server_startup_warnings(config, host: str, port: str) -> None:
-    """Print startup-time security / routing warnings for the server command.
-
-    Runs only from ``archivebox server`` so other entry points (manage shell,
-    plugin lookups, etc.) don't repeat this banner on every config load.
-    """
-    if config.IS_LOWER_SECURITY_MODE:
-        print(
-            f"[yellow][!] WARNING: ArchiveBox is running with SERVER_SECURITY_MODE={config.SERVER_SECURITY_MODE}[/yellow]",
-        )
-        print("[yellow]    Archived pages may share an origin with privileged app routes in this mode.[/yellow]")
-        print("[yellow]    To switch to the safer isolated setup:[/yellow]")
-        print("[yellow]      1. Set SERVER_SECURITY_MODE=safe-subdomains-fullreplay[/yellow]")
-        print("[yellow]      2. Point *.archivebox.localhost (or your chosen base domain) at this server[/yellow]")
-        print(
-            "[yellow]      3. Configure wildcard DNS/TLS or your reverse proxy so admin., web., api., and snapshot subdomains resolve[/yellow]",
-        )
-        print()
-
-    # ``config.BASE_URL`` is the merged value (env > Machine.config > file >
-    # default), which is what the running server will actually use. Earlier we
-    # gated the "BASE_URL not set" warning on ``os.environ["BASE_URL"]`` alone,
-    # which fired noisily when the user pinned BASE_URL via Machine.config /
-    # ArchiveBox.conf instead of via env.
-    base_url = (config.BASE_URL or "").strip()
-    if base_url:
-        # BASE_URL is pinned. The only thing left to surface is a port
-        # mismatch — bind port ≠ BASE_URL's explicit port usually means the
-        # operator started the server with the wrong ``archivebox server PORT``
-        # argument (or forgot to update one side after moving the listener).
-        # A reverse-proxy setup typically omits the port in BASE_URL
-        # (``https://archive.example.com``), so we only warn when BASE_URL
-        # carries an explicit port — otherwise we'd nag every proxy deployment.
-        from urllib.parse import urlparse
-
-        try:
-            base_port = urlparse(base_url).port
-        except (ValueError, TypeError):
-            base_port = None
-        if base_port is not None and str(base_port) != str(port):
-            print(
-                f"[yellow][!] BASE_URL ({base_url}) port {base_port} does not match the port the server is running on ({port}). "
-                "Make sure this is intentional![/yellow]",
-            )
-            print()
-        return
-
-    # If the user is upgrading from 0.7.3 and already had
-    # CSRF_TRUSTED_ORIGINS set, get_base_url() will silently use that as the
-    # implicit BASE_URL. Surface what we picked so the user knows where their
-    # links / redirects are going — and tell them how to make it explicit.
-    from archivebox.core.routes_util import derive_base_url_from_csrf
-
-    csrf_derived = derive_base_url_from_csrf(config)
-    if csrf_derived:
-        print(
-            f"[yellow][!] BASE_URL is not set; auto-derived [bold]{csrf_derived}[/bold] from a single CSRF_TRUSTED_ORIGINS entry.[/yellow]",
-        )
-        print(
-            "[yellow]    Links / redirects / cookies will use that origin. To silence this hint, set BASE_URL[/yellow]",
-        )
-        print(
-            f"[yellow]    explicitly: [bold]BASE_URL={csrf_derived}[/bold] (matches your existing CSRF_TRUSTED_ORIGINS).[/yellow]",
-        )
-        print()
-        return
-
-    # BASE_URL was not set explicitly. The routes_util derivation gives one of
-    # three results, with very different risk profiles — show a tailored hint
-    # so new users coming from the 0.7.x single-domain world know whether the
-    # default is fine for them or needs attention.
-    if _bind_host_looks_like_ip(host):
-        # Real IP literal: subdomain routing can't work, URLs leak the IP.
-        # This is the most urgent case.
-        print(
-            f"[yellow][!] WARNING: BASE_URL is not set and BIND_ADDR resolves to an IP literal ({host}).[/yellow]",
-        )
-        print(
-            "[yellow]    Snapshot / admin / api URLs will be generated with the IP, and subdomain[/yellow]",
-        )
-        print(
-            "[yellow]    routing cannot work against an IP address. Set BASE_URL explicitly, e.g.[/yellow]",
-        )
-        print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
-        )
-        if config.USES_SUBDOMAIN_ROUTING:
-            print(
-                "[yellow]    Or switch SERVER_SECURITY_MODE to a one-domain mode if you can't run a hostname.[/yellow]",
-            )
-        print()
-    else:
-        # Loopback / wildcard bind. The routes_util default of
-        # http://archivebox.localhost:PORT works in a browser on the same
-        # machine, but anything else (reverse proxy, k8s ingress, LAN client)
-        # needs BASE_URL set. (Real hostnames can't reach this branch — the
-        # bind validator rejects them upfront.)
-        print(
-            "[yellow][!] BASE_URL is not set. Generated URLs will fall back to http://archivebox.localhost:<port>.[/yellow]",
-        )
-        print(
-            "[yellow]    That's fine for local browsing on this machine. Set BASE_URL when running behind[/yellow]",
-        )
-        print(
-            "[yellow]    a reverse proxy / ingress / public hostname, e.g.[/yellow]",
-        )
-        print(
-            "[yellow]      BASE_URL=https://archive.example.com archivebox server 0.0.0.0:8000[/yellow]",
-        )
-        print()
-
-
-@enforce_types
-def server(
-    runserver_args: Iterable[str] | None = None,
-    reload: bool = False,
-    debug: bool = False,
-    daemonize: bool = False,
-    nothreading: bool = False,
-    createsuperuser: bool = False,
-) -> None:
-    """Run the ArchiveBox HTTP server"""
-    from archivebox.config.common import get_config
-
-    config = get_config()
-    runserver_args = list(runserver_args or (config.BIND_ADDR,))
-
-    run_in_debug = config.DEBUG or debug or reload
-    if debug or reload:
-        os.environ["DEBUG"] = "True"
-
-    from django.contrib.auth.models import User
-
-    if createsuperuser:
-        from archivebox.cli.archivebox_manage import manage
-
-        manage(args=["createsuperuser"])
-        print()
-
-    # First non-empty positional arg is the bind spec; otherwise inherit from
-    # config (which defaults to "127.0.0.1:8000"). _parse_and_validate_bind_spec
-    # hard-errors on hostnames so the rest of the server can assume a numeric
-    # bind host.
-    bind_spec = next((arg for arg in runserver_args if arg), "")
-    host, port = _parse_and_validate_bind_spec(bind_spec)
-
-    if not User.objects.filter(is_superuser=True).exclude(username="system").exists():
-        from archivebox.core.routes_util import build_admin_url
-
-        print()
-        print("[violet]Hint:[/violet] Open the Admin UI to create the first admin and finish web setup:")
-        runtime_config = config.model_copy(update={"BIND_ADDR": f"{host}:{port}"})
-        print(f"      [green]{build_admin_url('/admin/', config=runtime_config)}[/green]")
-        if not config.BASE_URL and host not in ("127.0.0.1", "localhost"):
-            print("      (When running remotely, replace localhost with this server's IP address or hostname.)")
-        print()
-
-    if daemonize and os.environ.get("ARCHIVEBOX_SERVER_DAEMON_CHILD") != "1":
-        from archivebox.workers.supervisord_util import resolve_env_binary
-
-        log_path = CONSTANTS.LOGS_DIR / "server.log"
-        log_path.parent.mkdir(parents=True, exist_ok=True)
-        daemon_env = os.environ.copy()
-        daemon_env["ARCHIVEBOX_SERVER_DAEMON_CHILD"] = "1"
-        daemon_cmd = [str(resolve_env_binary("archivebox")), "server"]
-        if debug:
-            daemon_cmd.append("--debug")
-        if reload:
-            daemon_cmd.append("--reload")
-        if nothreading:
-            daemon_cmd.append("--nothreading")
-        daemon_cmd.extend(runserver_args)
-        with log_path.open("a", encoding="utf-8") as log_file:
-            proc = subprocess.Popen(
-                daemon_cmd,
-                cwd=os.getcwd(),
-                env=daemon_env,
-                stdin=subprocess.DEVNULL,
-                stdout=log_file,
-                stderr=log_file,
-                start_new_session=True,
-            )
-        deadline = time.monotonic() + 30
-        while time.monotonic() < deadline:
-            if proc.poll() is not None:
-                print(f"[red][X] ArchiveBox daemon server exited early with code {proc.returncode}. See {log_path}[/red]")
-                sys.exit(proc.returncode or 1)
-            try:
-                with socket.create_connection((host, int(port)), timeout=0.25):
-                    break
-            except OSError:
-                time.sleep(0.1)
-        else:
-            print(f"[red][X] ArchiveBox daemon server pid={proc.pid} did not become ready. See {log_path}[/red]")
-            proc.terminate()
-            try:
-                proc.wait(timeout=10)
-            except subprocess.TimeoutExpired:
-                proc.kill()
-                proc.wait(timeout=5)
-            sys.exit(1)
-        return
-
-    os.environ["BIND_ADDR"] = f"{host}:{port}"
-    from archivebox.core.routes_util import get_base_url
-
-    base_url = get_base_url().rstrip("/")
-    admin_url = f"{base_url}/admin/"
-
-    from archivebox.workers.supervisord_util import (
-        active_supervisord_runtime_components,
-        format_runtime_components,
-        start_server_workers,
-        stop_existing_supervisord_process,
-        is_port_in_use,
-    )
-    from archivebox.machine.models import Process
-    from archivebox.core.takeover_util import (
-        command_owns_runtime_stack,
-        current_command,
-        foreground_runner_owner,
-        runtime_stack_owner,
-        standby_until_runtime_stack_needed,
-    )
-    from archivebox.core.shutdown_util import foreground_parent_watchdog, foreground_shutdown_signals
-
-    if run_in_debug:
-        print("[green][+] Starting ArchiveBox webserver in DEBUG mode...[/green]")
-    else:
-        print("[green][+] Starting ArchiveBox webserver...[/green]")
-    bind_url = f"http://{host}:{port}"
-    bind_message = Text.from_markup(
-        "    [blink][green]>[/green][/blink] Starting ArchiveBox webserver on [dim]BIND_ADDR[/dim] ",
-    )
-    bind_message.append(bind_url, style=Style(color="deep_sky_blue4", link=bind_url))
-    print(bind_message)
-    admin_message = Text.from_markup("    [green]>[/green] Log in to ArchiveBox Admin UI on [dim]BASE_URL [/dim] ")
-    admin_message.append(admin_url, style=Style(color="deep_sky_blue3", link=admin_url))
-    print(admin_message)
-    print("    > Writing ArchiveBox error log to ./logs/errors.log")
-    print()
-
-    # Reload config after we've set os.environ["BIND_ADDR"] above so the
-    # security-mode + base-url warnings see the effective values.
-    runtime_config = get_config()
-    _print_server_startup_warnings(runtime_config, host, port)
-    command = current_command(Process.TypeChoices.SERVER, data_dir=CONSTANTS.DATA_DIR, url=bind_url)
-
-    def still_owns_runtime_stack() -> bool:
-        from django.db import connections
-
-        try:
-            return command_owns_runtime_stack(command, data_dir=CONSTANTS.DATA_DIR)
-        finally:
-            connections.close_all()
-
-    shutdown_state = None
-    try:
-        with (
-            foreground_shutdown_signals() as shutdown_state,
-            foreground_parent_watchdog(enabled=os.environ.get("ARCHIVEBOX_SERVER_DAEMON_CHILD") != "1"),
-        ):
-            while True:
-                standby_result = standby_until_runtime_stack_needed(command, data_dir=CONSTANTS.DATA_DIR)
-                older_owner = runtime_stack_owner(data_dir=CONSTANTS.DATA_DIR, exclude_id=command.id) or foreground_runner_owner(
-                    data_dir=CONSTANTS.DATA_DIR,
-                    exclude_id=command.id,
-                )
-                takeover_components = active_supervisord_runtime_components()
-                if older_owner and takeover_components:
-                    print(
-                        "[yellow][*] Taking over "
-                        f"{format_runtime_components(takeover_components)} from older existing archivebox process (pid={older_owner.pid}).[/yellow]",
-                    )
-                stop_existing_supervisord_process()
-                if is_port_in_use(host, int(port)):
-                    print(f"[red][X] Error: Port {port} is already in use[/red]")
-                    print(f"    Another process outside this ArchiveBox runtime is listening on {host}:{port}")
-                    sys.exit(1)
-
-                result = start_server_workers(
-                    host=host,
-                    port=port,
-                    daemonize=False,
-                    debug=run_in_debug,
-                    reload=reload,
-                    nothreading=nothreading,
-                    keep_running=still_owns_runtime_stack,
-                    should_stop_supervisord=still_owns_runtime_stack,
-                    resumed_from_pid=standby_result.get("previous_owner_pid") if standby_result.get("resumed") else None,
-                )
-                if result == "interrupted":
-                    break
-                if not still_owns_runtime_stack():
-                    continue
-                if result == "exited":
-                    print("[yellow][*] Runtime stack exited while this parent is still leader; restarting...[/yellow]")
-                    continue
-                break
-    except KeyboardInterrupt:
-        pass
-    finally:
-        if not shutdown_state or not shutdown_state.signal_name:
-            command.mark_exited()
-    print("\n[i][green][🟩] ArchiveBox server shut down gracefully.[/green][/i]")
-
-
-@click.command()
-@click.argument("runserver_args", nargs=-1)
-@click.option("--reload", is_flag=True, help="Enable auto-reloading when code or templates change")
-@click.option("--debug", is_flag=True, help="Enable DEBUG=True mode with more verbose errors")
-@click.option("--nothreading", is_flag=True, help="Force runserver to run in single-threaded mode")
-@click.option("--daemonize", is_flag=True, help="Run the server in the background as a daemon")
-@click.option("--createsuperuser", is_flag=True, help="Run archivebox manage createsuperuser before starting the server")
-@docstring(server.__doc__)
-def main(**kwargs):
-    server(**kwargs)
-
-
-if __name__ == "__main__":
-    main()
@@ -1,29 +0,0 @@
-#!/usr/bin/env python3
-
-__package__ = "archivebox.cli"
-
-from collections.abc import Iterable
-
-import rich_click as click
-
-from archivebox.misc.util import docstring
-
-
-def shell(args: Iterable[str] = ()) -> None:
-    """Enter an interactive ArchiveBox Django shell"""
-
-    from django.core.management import call_command, get_commands
-
-    shell_command = "shell_plus" if "shell_plus" in get_commands() else "shell"
-    call_command(shell_command, *args)
-
-
-@click.command(add_help_option=False, context_settings=dict(ignore_unknown_options=True))
-@click.argument("args", nargs=-1)
-@docstring(shell.__doc__)
-def main(args: Iterable[str] = ()) -> None:
-    shell(args=args)
-
-
-if __name__ == "__main__":
-    main()
Author	SHA1	Message	Date
Nick Sweeting	2ad7a5a1ac	Preserve historical code blocks and pin the complete docs build	2026-09-17 23:59:31 -07:00
Nick Sweeting	ca1a8c908c	Restore historical documentation for the 0.0 release line	2026-09-17 23:37:27 -07:00