Preserve historical code blocks and pin the complete docs build

Restore versioned user documentation for v0.8.6rc1
fix archivebox add
2026-09-17 23:59:30 -07:00 · 2026-09-17 23:37:07 -07:00 · 2025-12-29 02:58:49 -08:00 · 2024-11-12 23:53:34 -08:00 · 2024-11-12 22:51:22 -08:00 · 2024-11-12 22:20:11 -08:00
290 changed files with 7814 additions and 3394 deletions
@@ -0,0 +1,20 @@
+version: 2
+build:
+  os: ubuntu-24.04
+  tools:
+    python: "3.13"
+  jobs:
+    build:
+      html:
+        - uv run --no-project --with-requirements .github/docs/requirements.txt sphinx-build -W --keep-going -b html -c .github/docs docs "$READTHEDOCS_OUTPUT/html"
+submodules:
+  include:
+    - docs
+python:
+  install:
+    - method: uv
+      command: pip
+      requirements: .github/docs/requirements.txt
+sphinx:
+  configuration: .github/docs/conf.py
+  fail_on_warning: true
@@ -0,0 +1,198 @@
+name: 🐞 Bug report
+description: Report a bug or error you encountered in ArchiveBox
+title: "Bug: ..."
+assignees:
+  - pirate
+type: 'Bug'
+body:
+  - type: markdown
+    attributes:
+      value: |
+        *Please note:* it is normal to see errors occasionally for some extractors on some URLs (not every extractor will work on every type of page).  
+        Please report archiving errors if you are seeing them *consistently across many URLs* or if they are *preventing you from using ArchiveBox*.
+
+  - type: textarea
+    id: description
+    attributes:
+      label: Provide a screenshot and describe the bug
+      description: |
+        Attach a screenshot and describe what the issue is, what you expected to happen, and if relevant, the *URLs you were trying to archive*.  
+      placeholder: |
+        Got a bunch of 'singlefile was unable to archive this page' errors when trying to archive URLs from this site: https://example.com/xyz ...
+        I also tried to archive the same URLs using `singlefile` directly and some of them worked but not all of them. etc. ...
+    validations:
+      required: true
+
+  - type: textarea
+    id: steps_to_reproduce
+    attributes:
+      label: Steps to reproduce
+      description: Please include the exact steps you took to trigger the issue.
+      render: markdown
+      placeholder: |
+        1. Started ArchiveBox by running: `docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:latest`
+        2. Went to the https://127.0.0.1:8000/add/ page in Google Chrome
+        3. Typed 'https://example.com/xyz' into the 'Add URL' input field
+        4. Clicked the 'Add+' button
+        5. Got a 500 error and saw the errors below in terminal
+    validations:
+      required: true
+
+  - type: textarea
+    id: logs
+    attributes:
+      label: Logs or errors
+      description: "Paste any terminal output, logs, or errors (check `data/logs/errors.log` as well)."
+      placeholder: |
+        ╭─────────────────────────────────────────────────────────────────────────────────────────────────────────╮
+        │ [2024-11-02 19:54:28] ArchiveBox v0.8.6rc0: archivebox add https://example.com#1234567                  │
+        ╰─────────────────────────────────────────────────────────────────────────────────────────────────────────╯
+        
+        [+] [2024-11-02 19:54:29] Adding 1 links to index (crawl depth=0)...
+            > Saved verbatim input to sources/1730577269-import.txt
+            > Parsed 1 URLs from input (Generic TXT)
+            ...
+      render: shell
+    validations:
+      required: false
+
+  - type: textarea
+    id: version
+    attributes:
+      label: ArchiveBox Version
+      description: |
+        **REQUIRED:** Run the `archivebox version` command inside your collection dir and paste the *full output* here (*not just the version number*).  
+        For Docker Compose run: `docker compose run archivebox version`  
+        For plain Docker run:   `docker run -v $PWD:/data archivebox/archivebox version`  
+      render: shell
+      placeholder: |
+        0.8.6
+        ArchiveBox v0.8.6rc0 COMMIT_HASH=721427a BUILD_TIME=2024-10-21 12:57:02 1729515422
+        IN_DOCKER=False IN_QEMU=False ARCH=arm64 OS=Darwin PLATFORM=macOS-15.1-arm64-arm-64bit PYTHON=Cpython (venv)
+        EUID=502:20 UID=502:20 PUID=502:20 FS_UID=502:20 FS_PERMS=644 FS_ATOMIC=True FS_REMOTE=False
+        DEBUG=False IS_TTY=True SUDO=False ID=dfa11485:aa78ad45 SEARCH_BACKEND=ripgrep LDAP=False
+        
+        Binary Dependencies:
+        √  python     3.11.9     venv_pip   ~/.venv/bin/python
+        √  django     5.1.2      venv_pip   ~/.venv/lib/python3.11/site-packages/django/__init__.py
+        √  sqlite     2.6.0      venv_pip   ~/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py
+        √  pip        24.3.1     venv_pip   ~/.venv/bin/pip
+        ...
+    validations:
+      required: true
+
+  - type: dropdown
+    id: install_method
+    validations:
+      required: true
+    attributes:
+      label: How did you install the version of ArchiveBox you are using?
+      multiple: false
+      options:
+        - pip
+        - apt
+        - brew
+        - nix
+        - Docker (or other container system like podman/LXC/Kubernetes or TrueNAS/Cloudron/YunoHost/etc.)
+        - Other
+
+  - type: dropdown
+    id: operating_system
+    validations:
+      required: true
+    attributes:
+      label: What operating system are you running on?
+      description: |
+        Please note we are *unable to provide support for Windows users* unless you are using [Docker on Windows](https://github.com/ArchiveBox/archivebox#:~:text=windows%20without%20docker).
+      multiple: false
+      options:
+        - Linux (Ubuntu/Debian/Arch/Alpine/etc.)
+        - macOS (including Docker on macOS)
+        - BSD (FreeBSD/OpenBSD/NetBSD/etc.)
+        - Windows (including WSL, WSL2, Docker Desktop on Windows)
+        - Other
+
+  - type: checkboxes
+    id: filesystem
+    attributes:
+      label: What type of drive are you using to store your ArchiveBox data?
+      description: Are you using a [remote filesystem](https://github.com/ArchiveBox/ArchiveBox/wiki/Setting-Up-Storage#supported-remote-filesystems) or FUSE mount for `data/` or `data/archive` (e.g. NFS/SMB/CIFS/etc. or FUSE/RClone/S3/B2/OneDrive/etc.)?
+      options:
+        - label: "`data/` is on a local SSD or NVMe drive"
+          required: false
+        - label: "`data/` is on a spinning hard drive or external USB drive"
+          required: false
+        - label: "`data/` is on a network mount (e.g. NFS/SMB/CIFS/etc.)"
+          required: false
+        - label: "`data/` is on a FUSE mount (e.g. SSHFS/RClone/S3/B2/OneDrive, etc.)"
+          required: false
+  
+
+  - type: textarea
+    id: docker_compose_yml
+    attributes:
+      label: Docker Compose Configuration
+      description: "If using Docker Compose, please share your full `docker-compose.yml` file. If using plain Docker, paste the `docker run ...` command you use."
+      placeholder: |
+        services:
+            archivebox:
+                image: archivebox/archivebox:latest
+                ports:
+                    - 8000:8000
+                volumes:
+                    - ./data:/data
+                environment:
+                    - ADMIN_USERNAME=admin
+                    - ADMIN_PASSWORD=****<redact any passwords>****
+                    - ALLOWED_HOSTS=*
+                    - CSRF_TRUSTED_ORIGINS=https://archivebox.example.com
+                    - PUBLIC_INDEX=True
+                    - PUBLIC_SNAPSHOTS=True
+                    - PUBLIC_ADD_VIEW=False
+                    ...
+
+            archivebox_scheduler:
+                image: archivebox/archivebox:latest
+                command: schedule --foreground --update --every=day
+                environment:
+                    ...
+      
+          ...
+      render: shell
+    validations:
+      required: false
+
+  - type: textarea
+    id: configuration
+    attributes:
+      label: ArchiveBox Configuration
+      description: "Please share your full `data/ArchiveBox.conf` file here."
+      render: shell
+      placeholder: |
+        [SERVER_CONFIG]
+        SECRET_KEY = "*********<redact any secrets/passwords>************"
+
+        WGET_RESTRICT_FILE_NAMES=windows
+        USE_SYSTEM_WGET=true
+        CHECK_SSL_VALIDITY=false
+        ...
+    validations:
+      required: false
+
+
+  - type: markdown
+    attributes:
+      value: |
+        ---
+          
+        We strive to answer issues as quickly as possible, it usually takes us *about a ~week* to respond.  
+        Make sure your `data/` is [**fully backed up**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout) before trying anything suggested here, **we are not responsible for data loss**.  
+          
+        In the meantime please consider:  
+          
+          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+          - 👨‍✈️ [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
+          - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
+          - 📚 Reading the [Troubleshooting Guide](https://github.com/ArchiveBox/ArchiveBox/wiki)
+          - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)
+        
@@ -0,0 +1,127 @@
+name: 💡 Feature or enhancement request
+description: Suggest an idea or improvement for this project
+title: "Feature Request: ..."
+assignees:
+  - pirate
+type: 'Enhancement'
+labels: 'status: idea phase'
+body:
+  - type: dropdown
+    id: suggestion_type
+    validations:
+      required: true
+    attributes:
+      label: "What type of suggestion are you making?"
+      multiple: false
+      options:
+        - New extractor / type of content to save
+        - Proposing a new feature
+        - Modification of existing behavior
+        - Web UI or UX design improvement
+
+  - type: textarea
+    id: current_problem
+    attributes:
+      label: "What is the problem that your feature request solves?"
+      description: |
+        Describe the problem that your feature request solves, feel free to include any screenshots or examples.
+      placeholder: |
+        e.g. I need to be able to archive spanish and french subtitle files from a particular movie site https://example.com/somevideos that's going down soon.
+    validations:
+      required: true
+
+  - type: textarea
+    id: proposed_solution
+    attributes:
+      label: "What is your proposed solution?"
+      description: |
+        Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes.
+      placeholder: |
+        e.g. I specifically need a new archive method to look for multilingual subtitle files related to pages.  
+        The bigger picture solution is the ability for custom user scripts to be run in a puppeteer context during archiving.
+    validations:
+      required: true
+
+  - type: textarea
+    id: workarounds_tried
+    attributes:
+      label: "What hacks or alternative solutions have you tried to solve the problem?"
+      description: |
+        A clear and concise description of any alternative solutions, workarounds, or other software you've considered using to fix the problem.
+      placeholder: |
+        e.g. I wait for archivebox to finish archiving the page, then I manually run `yt-dlp --subs <URL>` inside 
+        the `data/archive/<timestamp>/` directory to download the subtitle files and add them to the snapshot folder.
+    validations:
+      required: true
+
+  - type: textarea
+    id: version
+    attributes:
+      label: What version of ArchiveBox are you currently using?
+      description: |
+        We need to know what version of ArchiveBox and what OS you're currently using in order to contextualize your feature request.  
+        Sometimes we've already fixed the issues in newer BETA versions, sometimes features already exist but may not be available in specific environments/versions.
+          
+        Run the `archivebox version` command inside your current collection dir and paste the *full output* here (*not just the version number*).  
+        For Docker Compose run: `docker compose run archivebox version`  
+        For plain Docker run:   `docker run -v $PWD:/data archivebox/archivebox version`  
+      render: shell
+      placeholder: |
+        0.8.6
+        ArchiveBox v0.8.6rc0 COMMIT_HASH=721427a BUILD_TIME=2024-10-21 12:57:02 1729515422
+        IN_DOCKER=False IN_QEMU=False ARCH=arm64 OS=Darwin PLATFORM=macOS-15.1-arm64-arm-64bit PYTHON=Cpython (venv)
+        EUID=502:20 UID=502:20 PUID=502:20 FS_UID=502:20 FS_PERMS=644 FS_ATOMIC=True FS_REMOTE=False
+        DEBUG=False IS_TTY=True SUDO=False ID=dfa11485:aa78ad45 SEARCH_BACKEND=ripgrep LDAP=False
+        
+        Binary Dependencies:
+        √  python     3.11.9     venv_pip   ~/.venv/bin/python
+        √  django     5.1.2      venv_pip   ~/.venv/lib/python3.11/site-packages/django/__init__.py
+        √  sqlite     2.6.0      venv_pip   ~/.venv/lib/python3.11/site-packages/django/db/backends/sqlite3/base.py
+        √  pip        24.3.1     venv_pip   ~/.venv/bin/pip
+        ...
+    validations:
+      required: true
+
+  - type: checkboxes
+    id: priority
+    attributes:
+      label: "How badly do you want this new feature?"
+      options:
+        - label: "It's an urgent deal-breaker, I can't live without it"
+          required: false
+        - label: "It's important to add it in the near-mid term future"
+          required: false
+        - label: "It would be nice to have eventually"
+          required: false
+        - label: "I'm willing to [work on a PR](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) to develop this myself"
+          required: false
+        - label: "I have [donated money](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations) to go towards fixing this issue"
+          required: false
+  
+  - type: checkboxes
+    id: satisfaction_survey
+    attributes:
+      label: Mini Survey
+      description: How do you like ArchiveBox so far?
+      options:
+        - label: "I like ArchiveBox so far / would recommend it to a friend"
+          required: false
+        - label: "I've had a lot of difficulty getting ArchiveBox set up"
+          required: false
+        - label: "I would pay $10/mo for a hosted version of ArchiveBox if it had this feature"
+          required: false
+
+  - type: markdown
+    attributes:
+      value: |
+        ---
+          
+        We strive to answer issues as quickly as possible, it usually takes us *about a ~week* to respond.  
+        Make sure your `data/` is [**fully backed up**](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#disk-layout) before trying any workarounds or BETAs suggested here, **we are not responsible for data loss**.  
+          
+        In the meantime please consider:  
+          
+          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+          - 📊 [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
+          - 🔍 [Searching the Documentation](https://docs.archivebox.io/) for answers to common questions
+          - ✨ Testing out a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases) (issues are often already fixed in our latest `BETA` releases)
@@ -0,0 +1,52 @@
+name: 📑 Documentation improvement
+description: Submit an idea or correction for the Wiki documentation
+title: "Documentation: ..."
+labels: 'touches: docs'
+type: 'Enhancement'
+assignees:
+  - pirate
+body:
+  - type: markdown
+    attributes:
+      value: |
+        If you prefer, you can submit a [Pull Request](https://github.com/ArchiveBox/docs) on https://github.com/ArchiveBox/docs to edit the docs directly instead.
+
+  - type: input
+    id: page_url
+    validations:
+      required: true
+    attributes:
+      label: "What is the URL of the page you'd like to see improved?"
+      placeholder: e.g. https://github.com/ArchiveBox/docs/wiki/Install
+
+  - type: input
+    id: section_title
+    validations:
+      required: true
+    attributes:
+      label: "What is the title of the relevant section?"
+      placeholder: e.g. Option B. Automatic Setup Script
+
+  - type: textarea
+    id: suggested_edit
+    attributes:
+      label: "What is the suggested edit?"
+      placeholder: |
+        e.g. Please document how to run the automatic setup script for ArchiveBox on TempleOS.  
+        Attach images, screenshots, code snippets, etc. anything you think would help.
+    validations:
+      required: true
+
+  - type: markdown
+    attributes:
+      value: |
+        ---
+          
+        We strive to address issues as quickly as possible, it usually takes us *about a ~week* to respond.  
+          
+        In the meantime please consider:  
+          
+          - 💰 [Donating to support ArchiveBox open-source](https://github.com/ArchiveBox/ArchiveBox/wiki/Donations)
+          - 👨‍✈️ [Hiring us for corporate deployments](https://docs.monadical.com/s/archivebox-consulting-services) with professional support, custom feature development, and help with CAPTCHAs/rate-limits
+          - 🔍 [Checking out the new ReadTheDocs Documentation](https://docs.archivebox.io/)
+          - ✨ Helping us test a newer [`BETA` release](https://github.com/ArchiveBox/ArchiveBox/releases)
@@ -1,46 +0,0 @@
---
-name: 🐞 Bug report
-about: Create a report to help us improve
-title: 'Bug: ...'
-labels: 'bug'
-assignees: ''
-
---
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you.
-(the only required section is the version information)
-->
-
-#### Describe the bug
-<!--
-A description of what the bug is, 
-what you expected to happen, 
-and any relevant context about issue.
-->
-
-#### Steps to reproduce
-<!--
-For example:
-1. Ran ArchiveBox with the following config '...'
-2. Saw this output during archiving '....'
-3. UI didn't show the thing I was expecting '....'
-->
-
-#### Screenshots or log output
-
-<!--
-If applicable, post any relevant screenshots or copy/pasted terminal output from ArchiveBox.
-If you're reporting a parsing / importing error, **you must paste a copy of your redacted import file here**.
-->
-
-#### ArchiveBox version
-
-<!-- Run the `archivebox version` command locally then copy paste the result here: -->
-```logs
-replace this line with the *full*, unshortened output of running `archivebox version`
-```
-<!-- Tickets without full version info will closed until it is provided,
-we need the full output here to help you solve your issue -->
@@ -0,0 +1,8 @@
+blank_issues_enabled: false
+contact_links:
+  - name: 💬 Chat with us on Zulip
+    url: https://zulip.archivebox.io
+    about: "Join us on our Zulip forum to chat with the developers and other users (it's similar to Discord but self-hosted)."
+  - name: 💁‍♂️ Hire us to provide archiving for your organization
+    url: https://docs.monadical.com/s/archivebox-consulting-services
+    about: "We provide hosting, develoment, and support, including on-prem/cloud setup w/ SSO & storage, CAPTCHA/rate-limiting avoidance, etc."
@@ -1,17 +0,0 @@
---
-name: 📑 Documentation change
-about: Submit a suggestion for the Wiki documentation
-title: 'Documentation: Improvement request ...'
-labels: ''
-assignees: ''
-
---
-<!-- If you perfer, you can make a PR to https://github.com/ArchiveBox/docs instead of opening an issue -->
-
-## Wiki Page URL
-<!-- e.g. https://github.com/pirate/ArchiveBox/wiki/Configuration#use_color -->
-
-
-## Suggested Edit
-<!-- e.g. Please add more example usages, or please fix `xyz` typo to be `abc`. -->
-
@@ -1,50 +0,0 @@
---
-name: 💡 Feature request
-about: Suggest an idea for this project
-title: 'Feature Request: ...'
-labels: 'changes: behavior,status: idea phase'
-assignees: ''
-
---
-
-<!--
-Please fill out the following information, 
-feel free to delete sections if they're not applicable 
-or if long issue templates annoy you :)
-->
-
-## Type 
-
- - [ ] General question or discussion
- - [ ] Propose a brand new feature
- - [ ] Request modification of existing behavior or design
-
-## What is the problem that your feature request solves
-<!--
-e.g. I need to be able to archive spanish and french subtitle files  
-from a particular <example.com> movie site that's going down soon.
-->
-
-## Describe the ideal specific solution you'd want, and whether it fits into any broader scope of changes
-<!--
-e.g. I specifically need a new archive method to look for multilingual subtitle files related to pages.  
-The bigger picture solution is the ability for custom user scripts to be run in a puppeteer context during archiving.
-->
-
-## What hacks or alternative solutions have you tried to solve the problem?
-<!--
-A clear and concise description of any alternative solutions, 
-workarounds, or other software you've considered using to fix the problem.
-->
-
-## How badly do you want this new feature?
-
- - [ ] It's an urgent deal-breaker, I can't live without it
- - [ ] It's important to add it in the near-mid term future
- - [ ] It would be nice to have eventually
-
---
-
- - [ ] I'm willing to contribute [dev time](https://github.com/ArchiveBox/ArchiveBox#archivebox-development) / [money](https://github.com/sponsors/pirate) to fix this issue
- - [ ] I like ArchiveBox so far / would recommend it to a friend
- - [ ] I've had a lot of difficulty getting ArchiveBox set up
@@ -1,9 +0,0 @@
---
-name: 💬 Question, discussion, or support request
-about: Start a discussion or ask a question about ArchiveBox
-title: 'Question: ...'
-labels: ''
-assignees: ''
-
---
-
@@ -0,0 +1,33 @@
+# Building the 0.8.6rc1 documentation
+
+The `docs` submodule remains pinned to the wiki revision shipped with 0.8.6rc1.
+This configuration renders that historical user guide without installing
+ArchiveBox, importing Django, or updating the wiki checkout.
+
+Prerequisites: [uv](https://docs.astral.sh/uv/getting-started/installation/) 0.11.3 and Python 3.13, matching CI and Read the Docs. `uv` can install Python when needed.
+
+```sh
+git submodule update --init docs
+uv run --no-project --python 3.13 --with-requirements .github/docs/requirements.txt \
+  sphinx-build -W --keep-going -b html -c .github/docs docs /tmp/archivebox-docs
+```
+
+`conf.py` adapts GitHub wiki links, heading levels, and obsolete local anchors
+in memory. Search options without a prose section link to their exact
+0.8.6rc1 source definition; the removed USE_CURL option points to CURL_BINARY. The original user documentation is not rewritten.
+Generated Python autodoc scaffolding is outside this standalone user guide;
+its legacy Django-dependent builder is not loaded.
+
+Read the Docs uses `.github/.readthedocs.yaml` and the same strict HTML command.
+Keep its project configuration-file setting pointed at that path.
+
+The release label follows the immutable `v0.8.6rc1` tag; its original
+`pyproject.toml` still declares `0.8.6rc0` and is left unchanged.
+
+`requirements.txt` pins the complete documentation dependency set. To update it,
+edit `requirements.in` and run:
+
+```sh
+uv pip compile --universal --python-version 3.13 .github/docs/requirements.in \
+  --output-file .github/docs/requirements.txt
+```
@@ -0,0 +1,199 @@
+"""Build the pinned 0.8.6rc1 user guide without importing ArchiveBox or Django."""
+
+import os
+import re
+from pathlib import Path
+from urllib.parse import unquote
+
+from markdown_it import MarkdownIt
+from myst_parser.mdit_to_docutils.base import default_slugify
+
+ROOT = Path(__file__).resolve().parents[2]
+DOCS = ROOT / "docs"
+project = "ArchiveBox"
+author = "ArchiveBox contributors"
+copyright = "ArchiveBox contributors"
+html_show_sphinx = False
+# The release tag is authoritative: its pyproject still declares 0.8.6rc0.
+release = "0.8.6rc1"
+version = release
+extensions = ["myst_parser", "sphinx_rtd_theme", "sphinxcontrib.mermaid"]
+source_suffix = {".rst": "restructuredtext", ".md": "markdown"}
+root_doc = "index"
+# These are generated autodoc scaffolding, not standalone user documentation.
+exclude_patterns = [
+    "archivebox*.rst",
+    "modules.rst",
+    "apidocs/**",
+    "_Sidebar.md",
+    "_Footer.md",
+    ".git",
+]
+html_theme = "sphinx_rtd_theme"
+html_theme_options = {
+    "navigation_depth": 3,
+    "collapse_navigation": False,
+    "version_selector": True,
+    "language_selector": False,
+}
+html_context = {"current_version": os.environ.get("READTHEDOCS_VERSION", release)}
+html_logo = str(DOCS / "logo.png")
+html_static_path = [str(DOCS / "_static")]
+html_title = f"ArchiveBox {release} documentation"
+myst_heading_anchors = 6
+myst_fence_as_directive = ["mermaid"]
+
+
+def rewrite_prose(text, transform):
+    """Rewrite navigation outside fenced and indented code blocks."""
+    lines = text.splitlines(keepends=True)
+    output = []
+    start = 0
+    for token in MarkdownIt().parse(text):
+        if token.type not in {"fence", "code_block"}:
+            continue
+        first, last = token.map
+        output.append(transform("".join(lines[start:first])))
+        output.append("".join(lines[first:last]))
+        start = last
+    output.append(transform("".join(lines[start:])))
+    return "".join(output)
+
+
+def prepare_source(app, docname, source):
+    """Adapt GitHub wiki navigation in memory, preserving the pinned sources."""
+    text = source[0]
+    if docname == "Contents":
+        text = re.sub(r"API Reference\n#############.*?(?=Meta\n####)", "", text, flags=re.DOTALL)
+        text += "\n.. toctree::\n    :hidden:\n\n    Home\n    ArchiveBox-Architecture-Diagrams\n    Upgrading-or-Merging-Archives\n"
+    if docname == "index":
+        text = text.replace("archivebox info", "archivebox status")
+        text = text.replace("pip install archivebox", f"pip install archivebox=={release}")
+        text = text.replace("ArchiveBox/ArchiveBox/tree/master", f"ArchiveBox/ArchiveBox/tree/v{release}")
+        text = text.replace(
+            "`Github <https://github.com/ArchiveBox/ArchiveBox/issues>`_",
+            "`GitHub issues <https://github.com/ArchiveBox/ArchiveBox/issues>`_",
+        )
+        text = text.replace(
+            "==========\nArchiveBox\n==========",
+            f"ArchiveBox {release}\n" + "=" * (11 + len(release)),
+        )
+    if Path(app.env.doc2path(docname)).suffix == ".md":
+
+        def wiki_link(match):
+            parts = match.group(1).split("|", 1)
+            label, target = (parts[0], parts[-1])
+            page, separator, anchor = unquote(target).partition("#")
+            page = page.replace(" ", "-")
+            page = {"Setting-Up-Authentication": "Setting-up-Authentication"}.get(page, page)
+            suffix = f"#{anchor}" if separator else ""
+            return f"[{label}]({page}.md{suffix})"
+
+        text = rewrite_prose(text, lambda prose: re.sub(r"\[\[([^\]\n]+)\]\]", wiki_link, prose))
+    source[0] = text
+
+
+# GitHub wiki headings allow skipped levels; normalize their hierarchy before MyST
+# parses them, without changing fenced code blocks or the historical source files.
+def normalize_markdown(app, docname, source):
+    if Path(app.env.doc2path(docname)).suffix != ".md":
+        return
+    text = source[0]
+    if docname == "Upgrading":
+        text = text.replace("---\n---", "---")
+    if docname == "Scheduled-Archiving":
+        text = text.replace("```bash\n$ cd", "```console\n$ cd", 1)
+    if docname == "Setting-up-Authentication":
+        text = text.replace(
+            "[Wiki: Configuration (`PUBLIC_ADD_VIEW`, `PUBLIC_SNAPSHOTS`, `PUBLIC_INDEX`)]()",
+            "[Wiki: Configuration (`PUBLIC_ADD_VIEW`, `PUBLIC_SNAPSHOTS`, `PUBLIC_INDEX`)](Configuration.md#public_index--public_snapshots--public_add_view)",
+        )
+    if docname in {"Home", "README", "Donations", "Upgrading-or-Merging-Archives"}:
+        text = "# " + ("ArchiveBox overview" if docname == "README" else docname.replace("-", " ")) + "\n\n" + text
+    lines = text.splitlines()
+    stack = []
+    for token in MarkdownIt().parse(text):
+        if token.type != "heading_open":
+            continue
+        original = int(token.tag[1])
+        while stack and stack[-1] >= original:
+            stack.pop()
+        stack.append(original)
+        line = token.map[0]
+        lines[line] = re.sub(r"^#{1,6}(?=\s)", "#" * len(stack), lines[line]).replace("\ufe0f", "")
+    text = "\n".join(lines) + "\n"
+
+    def link(match):
+        target = match.group(1)
+        if "://" in target or target.startswith("mailto:"):
+            return match.group(0)
+        page, sep, anchor = target.partition("#")
+        if page and (DOCS / (page + ".md")).exists():
+            page += ".md"
+        anchor = unquote(anchor).lower()
+        if docname == "Install" and not page:
+            anchor = {
+                "option-a-docker--docker-compose-setup-️": "option-a-docker--docker-compose-setup-",
+                "upgrading-archivebox-to-a-new-version": "next-steps-upgrading-archivebox-to-a-new-version",
+                "next-steps": "next-steps-add-some-urls-to-archive-and-try-out-cli--web-ui",
+                "ubuntudebian": "ubuntudebian-based-systems",
+                "bsd": "freebsd",
+                "manual-setup": "option-c-bare-metal-setup",
+            }.get(anchor, anchor)
+        if docname == "README" and not page:
+            anchor = {
+                "input-formats": "input-formats-how-to-pass-urls-into-archivebox-for-saving",
+                "output-formats": "output-formats-what-archivebox-saves-for-each-url",
+            }.get(anchor, anchor)
+        if (docname == "Configuration" and not page) or page == "Configuration.md":
+            other = (DOCS / "Configuration.md").read_text()
+            for heading in re.findall(r"^#+\s+(.+)$", other, re.MULTILINE):
+                if anchor in [word.lower() for word in re.findall(r"`([^`]+)`", heading)]:
+                    anchor = default_slugify(heading.replace("`", ""))
+                    break
+            aliases = {"templates_dir": "custom_templates_dir"}
+            anchor = aliases.get(anchor, anchor)
+            if anchor in {
+                "search_backend_timeout",
+                "search_backend_engine",
+                "use_curl",
+            }:
+                if anchor == "use_curl":
+                    # This obsolete wiki option no longer exists in the 0.8 config.
+                    return "](Configuration.md#curl_binary)"
+                code = (ROOT / "archivebox/config/common.py").read_text().splitlines()
+                number = next(i + 1 for i, line in enumerate(code) if anchor.upper() in line)
+                return "](https://github.com/ArchiveBox/ArchiveBox/blob/v0.8.6rc1/archivebox/config/common.py#L" + str(number) + ")"
+        if docname == "README" and anchor == "saves-lots-of-useful-stuff-for-each-imported-link":
+            anchor = "output-formats-what-archivebox-saves-for-each-url"
+        if docname == "Web-Archiving-Community":
+            anchor = {
+                "blogs": "blogs-friends-of-archivebox",
+                "articles": "articles-we-like-about-internet-archiving",
+            }.get(anchor, anchor)
+        return "](" + page + ("#" + anchor if sep else "") + ")"
+
+    def rewrite_links(prose):
+        prose = re.sub(r"\]\(([^)\s]+)\)", link, prose)
+        # Raw HTML links are outside MyST's cross-reference resolver.
+        html_anchors = {
+            "background--motivation": "background-motivation",
+            "Caveats": "caveats",
+            "contents": "web-archiving-community",
+            "%EF%B8%8F-cli-usage": "cli-usage",
+            "url_whitelist": "url-allowlist",
+            "url_blacklist": "url-denylist",
+            "input-formats": "input-formats-how-to-pass-urls-into-archivebox-for-saving",
+        }
+        for old, new in html_anchors.items():
+            prose = prose.replace(f'href="#{old}"', f'href="#{new}"')
+        prose = prose.replace("#️-cli-usage", "#cli-usage")
+        return prose
+
+    text = rewrite_prose(text, rewrite_links)
+    source[0] = text
+
+
+def setup(app):
+    app.connect("source-read", prepare_source)
+    app.connect("source-read", normalize_markdown)
@@ -0,0 +1,4 @@
+Sphinx==8.2.3
+myst-parser==4.0.1
+sphinx-rtd-theme==3.1.0
+sphinxcontrib-mermaid==2.1.1
@@ -0,0 +1,81 @@
+# This file was autogenerated by uv via the following command:
+#    uv pip compile --universal --python-version 3.13 .github/docs/requirements.in --output-file .github/docs/requirements.txt
+alabaster==1.0.0
+    # via sphinx
+babel==2.18.0
+    # via sphinx
+certifi==2026.7.22
+    # via requests
+charset-normalizer==3.5.1
+    # via requests
+colorama==0.4.6 ; sys_platform == 'win32'
+    # via sphinx
+docutils==0.21.2
+    # via
+    #   myst-parser
+    #   sphinx
+    #   sphinx-rtd-theme
+idna==3.20
+    # via requests
+imagesize==2.0.1
+    # via sphinx
+jinja2==3.1.6
+    # via
+    #   myst-parser
+    #   sphinx
+    #   sphinxcontrib-mermaid
+markdown-it-py==3.0.0
+    # via
+    #   mdit-py-plugins
+    #   myst-parser
+markupsafe==3.0.3
+    # via jinja2
+mdit-py-plugins==0.6.1
+    # via myst-parser
+mdurl==0.1.2
+    # via markdown-it-py
+myst-parser==4.0.1
+    # via -r .github/docs/requirements.in
+packaging==26.3
+    # via sphinx
+pygments==2.21.0
+    # via sphinx
+pyyaml==6.0.3
+    # via
+    #   myst-parser
+    #   sphinxcontrib-mermaid
+requests==2.34.2
+    # via sphinx
+roman-numerals==4.1.0
+    # via roman-numerals-py
+roman-numerals-py==4.1.0
+    # via sphinx
+snowballstemmer==3.1.1
+    # via sphinx
+sphinx==8.2.3
+    # via
+    #   -r .github/docs/requirements.in
+    #   myst-parser
+    #   sphinx-rtd-theme
+    #   sphinxcontrib-jquery
+    #   sphinxcontrib-mermaid
+sphinx-rtd-theme==3.1.0
+    # via -r .github/docs/requirements.in
+sphinxcontrib-applehelp==2.0.0
+    # via sphinx
+sphinxcontrib-devhelp==2.0.0
+    # via sphinx
+sphinxcontrib-htmlhelp==2.1.0
+    # via sphinx
+sphinxcontrib-jquery==4.1
+    # via sphinx-rtd-theme
+sphinxcontrib-jsmath==1.0.1
+    # via sphinx
+sphinxcontrib-mermaid==2.1.1
+    # via -r .github/docs/requirements.in
+sphinxcontrib-qthelp==2.0.0
+    # via sphinx
+sphinxcontrib-serializinghtml==2.0.0
+    # via sphinx
+urllib3==2.8.0
+    # via requests
@@ -3,8 +3,9 @@ name: Build Docker image
 on:
  workflow_dispatch:
  push:
-    branches:
-      - '**'
+    branches-ignore:
+      - docs
+      - 'docs/**'
    tags:
      - 'v*'
  # pull_request:
@@ -14,6 +15,8 @@ env:

 jobs:
  buildx:
+    # Also prevent a manual dispatch on a docs repair branch from publishing.
+    if: github.ref != 'refs/heads/docs' && !startsWith(github.ref, 'refs/heads/docs/')
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
@@ -0,0 +1,34 @@
+name: Historical Sphinx documentation
+
+on:
+  pull_request:
+    branches: [docs/restore-08-sphinx]
+    paths: ['.github/docs/**', '.github/.readthedocs.yaml', '.github/workflows/docs.yml', 'docs', 'README.md']
+  push:
+    branches: [docs/restore-08-sphinx]
+    paths: ['.github/docs/**', '.github/.readthedocs.yaml', '.github/workflows/docs.yml', 'docs', 'README.md']
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+jobs:
+  build:
+    runs-on: ubuntu-24.04
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4
+      - run: git submodule update --init docs
+      - uses: astral-sh/setup-uv@d0cc045d04ccac9d8b7881df0226f9e82c39688e # v6
+        with:
+          version: '0.11.3'
+          python-version: '3.13'
+      - name: Build the pinned historical user guide
+        env:
+          READTHEDOCS: 'True'
+          READTHEDOCS_VERSION: v0.8.6rc1
+        run: uv run --no-project --with-requirements .github/docs/requirements.txt sphinx-build -W --keep-going -b html -c .github/docs docs "$RUNNER_TEMP/docs-html"
+      - name: Verify the version selector
+        run: |
+          grep -q 'class="version-switch"' "$RUNNER_TEMP/docs-html/index.html"
+          grep -q 'readthedocs-addons-data-ready' "$RUNNER_TEMP/docs-html/_static/js/versions.js"
@@ -102,7 +102,7 @@ jobs:
        # TODO: remove this exception for windows once we get tests passing on that platform
        if: ${{ !contains(matrix.os, 'windows') }}
        run: |
-          python -m pytest -s --basetemp=tests/out --ignore=archivebox/vendor --ignore=deb_dist --ignore=pip_dist --ignore=brew_dist
+          python -m pytest -s --basetemp=tests/out --ignore=archivebox/pkgs

  docker_tests:
    runs-on: ubuntu-latest
@@ -1,9 +1,3 @@
 [submodule "docs"]
    path = docs
    url = https://github.com/ArchiveBox/ArchiveBox.wiki.git
-[submodule "archivebox/vendor/pocket"]
-	path = archivebox/vendor/pocket
-	url = https://github.com/tapanpandita/pocket
-[submodule "archivebox/vendor/pydantic-pkgr"]
-	path = archivebox/vendor/pydantic-pkgr
-	url = https://github.com/ArchiveBox/pydantic-pkgr
@@ -1,26 +0,0 @@
-# Read the Docs config for https://docs.archivebox.io
-# https://docs.readthedocs.io/en/stable/config-file/v2.html
-
-version: 2
-
-submodules:
-  include: all
-  recursive: true
-
-build:
-  os: ubuntu-22.04
-  tools:
-    python: "3.12"
-    #nodejs: "20"     # not needed unless we need the full archivebox to run while building docs for some reason
-
-sphinx:
-  configuration: docs/conf.py
-
-formats:
-  - pdf
-  - epub
-
-# https://docs.readthedocs.io/en/stable/guides/reproducible-builds.html
-python:
-  install:
-    - requirements: docs/requirements.txt
@@ -1 +0,0 @@
-archivebox.io
@@ -4,7 +4,7 @@ FROM python:3.12-slim

 WORKDIR /app

-RUN pip install archivebox==0.8.5rc44
+RUN pip install archivebox==0.8.5rc50
 RUN archivebox install

 RUN useradd -ms /bin/bash archivebox && chown -R archivebox /app
@@ -13,7 +13,7 @@
 <a href="https://github.com/ArchiveBox/ArchiveBox/blob/dev/LICENSE"><img src="https://img.shields.io/badge/Open_source-MIT-green.svg?logo=git&logoColor=green"/></a> <a href="https://github.com/ArchiveBox/ArchiveBox/commits/dev"><img src="https://img.shields.io/github/last-commit/ArchiveBox/ArchiveBox.svg?logo=Sublime+Text&logoColor=green&label=Active"/></a> &nbsp; <a href="https://github.com/ArchiveBox/ArchiveBox"><img src="https://img.shields.io/github/stars/ArchiveBox/ArchiveBox.svg?logo=github&label=Stars&logoColor=blue"/></a> &nbsp; <a href="https://hub.docker.com/r/archivebox/archivebox"><img src="https://img.shields.io/docker/pulls/archivebox/archivebox.svg?label=Docker+Pulls"/></a> <a href="https://pypi.org/project/archivebox/"><img src="https://img.shields.io/pypi/dm/archivebox?label=PyPI%20Installs&color=%235f7dae"/></a> <a href="https://chromewebstore.google.com/detail/archivebox-exporter/habonpimjphpdnmcfkaockjnffodikoj"><img src="https://img.shields.io/chrome-web-store/users/habonpimjphpdnmcfkaockjnffodikoj?label=Chrome%20Store&color=%231973e8"/></a>

 <!--<pre lang="bash" align="left"><code style="white-space: pre-line; text-align: left" align="left">
-curl -fsSL 'https://get.archivebox.io' | sh    # (or see pip/brew/Docker instructions below)
+curl -fsSL 'https://get.archivebox.io' | bash    # (or see pip/brew/Docker instructions below)
 </code></pre>-->

 </div>
@@ -27,7 +27,7 @@ Without active preservation effort, everything on the internet eventually disapp
 *ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
 <br/>

-> ➡️ Get ArchiveBox with `pip install archivebox` on [Linux](#quickstart), [macOS](#quickstart), and [Windows](#quickstart) (WSL2), or via **[Docker](#quickstart)** ⭐️.  
+> ➡️ Get ArchiveBox with `pip install archivebox` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.  

 *Once installed, it can be used as a [CLI tool](#usage), [self-hosted Web App](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python library](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [one-off command](#static-archive-exporting).*

@@ -99,7 +99,7 @@ archivebox init --setup
 <br/>
 <br/>
 # Option D: Or use the optional auto setup script to install it
-curl -fsSL 'https://get.archivebox.io' | sh
+curl -fsSL 'https://get.archivebox.io' | bash
 </code></pre>
 <br/>
 <sub>Open <a href="http://localhost:8000"><code>http://localhost:8000</code></a> to see your server's Web UI ➡️</sub>
@@ -233,7 +233,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
 <ol>
 <li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (optional, highly recommended but not required).</li>
 <li>Run the automatic setup script.
-<pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | sh</code></pre>
+<pre lang="bash"><code style="white-space: pre-line">curl -fsSL 'https://get.archivebox.io' | bash</code></pre>
 <i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-b-automatic-setup-script">Install: Bare Metal</a> in the Wiki. ➡️</i>
 </li>
 </ol>
@@ -368,7 +368,7 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C

 <ul>
 <li>Arch: <a href="https://aur.archlinux.org/packages/archivebox/"><code>yay -S archivebox</code></a> (contributed by <a href="https://github.com/imlonghao"><code>@imlonghao</code></a>)</li>
-<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | sh</code></a> (uses <code>pkg</code> + <code>pip3</code> under-the-hood)</li>
+<li>FreeBSD: <a href="https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup"><code>curl -fsSL 'https://get.archivebox.io' | bash</code></a> (uses <code>pkg</code> + <code>pip3</code> under-the-hood)</li>
 <li>Nix: <a href="https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/misc/archivebox/default.nix"><code>nix-env --install archivebox</code></a> (contributed by <a href="https://github.com/siraben"><code>@siraben</code></a>)</li>
 <li>Guix: <a href="https://packages.guix.gnu.org/packages/archivebox/"><code>guix install archivebox</code></a> (contributed by <a href="https://github.com/rakino"><code>@rakino</code></a>)</li>
 <li>More: <a href="https://github.com/ArchiveBox/ArchiveBox/issues/new"><i>contribute another distribution...!</i></a></li>
@@ -1347,6 +1347,11 @@ For low hanging fruit / easy first tickets, see: <a href="https://github.com/Arc

 **Python API Documentation:** https://docs.archivebox.io/en/dev/archivebox.html#module-archivebox.main

+**Internal Architecture Diagrams:** https://github.com/ArchiveBox/ArchiveBox/wiki/ArchiveBox-Architecture-Diagrams
+
+<img width="100%" alt="architecture diagrams" src="https://github.com/user-attachments/assets/bdf782d5-2c8f-4168-845d-6794ff5cbfec">
+
+
 ### Setup the dev environment

 <details><summary><i>Click to expand...</i></summary>
@@ -3,4 +3,4 @@ ignore = D100,D101,D102,D103,D104,D105,D202,D203,D205,D400,E131,E241,E252,E266,E
 select = F,E9,W
 max-line-length = 130
 max-complexity = 10
-exclude = migrations,tests,node_modules,vendor,venv,.venv,.venv2,.docker-venv
+exclude = migrations,tests,node_modules,vendor,venv,.venv,.venv2,.docker-venv,data,data*
@@ -13,8 +13,8 @@ __package__ = 'archivebox'

 import os
 import sys
-
 from pathlib import Path
+from typing import cast

 ASCII_LOGO = """
 █████╗ ██████╗  ██████╗██╗  ██╗██╗██╗   ██╗███████╗ ██████╗  ██████╗ ██╗  ██╗
@@ -47,11 +47,54 @@ from .monkey_patches import *                    # noqa


 # print('LOADING VENDORED LIBRARIES')
-from .vendor import load_vendored_libs           # noqa
-load_vendored_libs()
+from .pkgs import load_vendored_pkgs             # noqa
+load_vendored_pkgs()
 # print('DONE LOADING VENDORED LIBRARIES')

+# Load ABX Plugin Specifications + Default Implementations
+import abx                                       # noqa
+import abx_spec_archivebox                       # noqa
+import abx_spec_config                           # noqa
+import abx_spec_pydantic_pkgr                    # noqa
+import abx_spec_django                           # noqa
+import abx_spec_searchbackend                    # noqa

+abx.pm.add_hookspecs(abx_spec_config.PLUGIN_SPEC)
+abx.pm.register(abx_spec_config.PLUGIN_SPEC())
+
+abx.pm.add_hookspecs(abx_spec_pydantic_pkgr.PLUGIN_SPEC)
+abx.pm.register(abx_spec_pydantic_pkgr.PLUGIN_SPEC())
+
+abx.pm.add_hookspecs(abx_spec_django.PLUGIN_SPEC)
+abx.pm.register(abx_spec_django.PLUGIN_SPEC())
+
+abx.pm.add_hookspecs(abx_spec_searchbackend.PLUGIN_SPEC)
+abx.pm.register(abx_spec_searchbackend.PLUGIN_SPEC())
+
+# Cast to ArchiveBoxPluginSpec to enable static type checking of pm.hook.call() methods
+abx.pm = cast(abx.ABXPluginManager[abx_spec_archivebox.ArchiveBoxPluginSpec], abx.pm)
+pm = abx.pm
+
+
+# Load all pip-installed ABX-compatible plugins
+ABX_ECOSYSTEM_PLUGINS = abx.get_pip_installed_plugins(group='abx')
+
+# Load all built-in ArchiveBox plugins
+ARCHIVEBOX_BUILTIN_PLUGINS = {
+    'config': PACKAGE_DIR / 'config',
+    'core': PACKAGE_DIR / 'core',
+    # 'search': PACKAGE_DIR / 'search',
+    # 'core': PACKAGE_DIR / 'core',
+}
+
+# Load all user-defined ArchiveBox plugins
+USER_PLUGINS = abx.find_plugins_in_dir(Path(os.getcwd()) / 'user_plugins')
+
+# Import all plugins and register them with ABX Plugin Manager
+ALL_PLUGINS = {**ABX_ECOSYSTEM_PLUGINS, **ARCHIVEBOX_BUILTIN_PLUGINS, **USER_PLUGINS}
+LOADED_PLUGINS = abx.load_plugins(ALL_PLUGINS)
+
+# Setup basic config, constants, paths, and version
 from .config.constants import CONSTANTS                         # noqa
 from .config.paths import PACKAGE_DIR, DATA_DIR, ARCHIVE_DIR    # noqa
 from .config.version import VERSION                             # noqa
@@ -144,9 +144,10 @@ class ABIDModel(models.Model):

                allowed_to_invalidate_abid = self.abid_drift_allowed if (abid_drift_allowed is None) else abid_drift_allowed
                if allowed_to_invalidate_abid:
-                    print(f'\n#### WARNING: Change allowed despite it invalidating the ABID of an existing record ({self.__class__.__name__}.abid_drift_allowed={self.abid_drift_allowed})!', self.abid)
-                    print(change_error)
-                    print('--------------------------------------------------------------------------------------------------')
+                    # print(f'\n#### WARNING: Change allowed despite it invalidating the ABID of an existing record ({self.__class__.__name__}.abid_drift_allowed={self.abid_drift_allowed})!', self.abid)
+                    # print(change_error)
+                    # print('--------------------------------------------------------------------------------------------------')
+                    pass
                else:
                    print(f'\n#### ERROR:   Change blocked because it would invalidate ABID of an existing record ({self.__class__.__name__}.abid_drift_allowed={self.abid_drift_allowed})', self.abid)
                    print(change_error)
@@ -174,7 +175,7 @@ class ABIDModel(models.Model):
            'uri': self.abid_uri_src,
            'subtype': self.abid_subtype_src,
            'rand': self.abid_rand_src,
-            'salt': 'self.abid_salt',               # defined as static class vars at build time
+            'salt': 'self.abid_salt',                 # defined as static class vars at build time
        }

    @property
@@ -1,131 +0,0 @@
-__package__ = 'abx'
-
-import importlib
-from pathlib import Path
-from typing import Dict, Callable, List
-
-from . import hookspec as base_spec
-from abx.hookspec import hookimpl, hookspec           # noqa
-from abx.manager import pm, PluginManager             # noqa
-
-
-pm.add_hookspecs(base_spec)
-
-
-###### PLUGIN DISCOVERY AND LOADING ########################################################
-
-def get_plugin_order(plugin_entrypoint: Path):
-    order = 999
-    try:
-        # if .plugin_order file exists, use it to set the load priority
-        order = int((plugin_entrypoint.parent / '.plugin_order').read_text())
-    except FileNotFoundError:
-        pass
-    return (order, plugin_entrypoint)
-
-def register_hookspecs(hookspecs: List[str]):
-    """
-    Register all the hookspecs from a list of module names.
-    """
-    for hookspec_import_path in hookspecs:
-        hookspec_module = importlib.import_module(hookspec_import_path)
-        pm.add_hookspecs(hookspec_module)
-
-
-def find_plugins_in_dir(plugins_dir: Path, prefix: str) -> Dict[str, Path]:
-    """
-    Find all the plugins in a given directory. Just looks for an __init__.py file.
-    """
-    return {
-        f"{prefix}.{plugin_entrypoint.parent.name}": plugin_entrypoint.parent
-        for plugin_entrypoint in sorted(plugins_dir.glob("*/__init__.py"), key=get_plugin_order)
-        if plugin_entrypoint.parent.name != 'abx'
-    }   # "plugins_pkg.pip": "/app/archivebox/plugins_pkg/pip"
-
-
-def get_pip_installed_plugins(group='abx'):
-    """replaces pm.load_setuptools_entrypoints("abx"), finds plugins that registered entrypoints via pip"""
-    import importlib.metadata
-
-    DETECTED_PLUGINS = {}   # module_name: module_dir_path
-    for dist in list(importlib.metadata.distributions()):
-        for entrypoint in dist.entry_points:
-            if entrypoint.group != group or pm.is_blocked(entrypoint.name):
-                continue
-            DETECTED_PLUGINS[entrypoint.name] = Path(entrypoint.load().__file__).parent
-            # pm.register(plugin, name=ep.name)
-            # pm._plugin_distinfo.append((plugin, DistFacade(dist)))
-    return DETECTED_PLUGINS
-
-
-def get_plugins_in_dirs(plugin_dirs: Dict[str, Path]):
-    """
-    Get the mapping of dir_name: {plugin_id: plugin_dir} for all plugins in the given directories.
-    """
-    DETECTED_PLUGINS = {}
-    for plugin_prefix, plugin_dir in plugin_dirs.items():
-        DETECTED_PLUGINS.update(find_plugins_in_dir(plugin_dir, prefix=plugin_prefix))
-    return DETECTED_PLUGINS
-
-
-# Load all plugins from pip packages, archivebox built-ins, and user plugins
-
-def load_plugins(plugins_dict: Dict[str, Path]):
-    """
-    Load all the plugins from a dictionary of module names and directory paths.
-    """
-    LOADED_PLUGINS = {}
-    for plugin_module, plugin_dir in plugins_dict.items():
-        # print(f'Loading plugin: {plugin_module} from {plugin_dir}')
-        plugin_module_loaded = importlib.import_module(plugin_module)
-        pm.register(plugin_module_loaded)
-        LOADED_PLUGINS[plugin_module] = plugin_module_loaded.PLUGIN
-        # print(f'    √ Loaded plugin: {plugin_module}')
-    return LOADED_PLUGINS
-
-def get_registered_plugins():
-    """
-    Get all the plugins registered with Pluggy.
-    """
-    plugins = {}
-    plugin_to_distinfo = dict(pm.list_plugin_distinfo())
-    for plugin in pm.get_plugins():
-        plugin_info = {
-            "name": plugin.__name__,
-            "hooks": [h.name for h in pm.get_hookcallers(plugin) or ()],
-        }
-        distinfo = plugin_to_distinfo.get(plugin)
-        if distinfo:
-            plugin_info["version"] = distinfo.version
-            plugin_info["name"] = (
-                getattr(distinfo, "name", None) or distinfo.project_name
-            )
-        plugins[plugin_info["name"]] = plugin_info
-    return plugins
-
-
-
-
-def get_plugin_hooks(plugin_pkg: str | None) -> Dict[str, Callable]:
-    """
-    Get all the functions marked with @hookimpl on a module.
-    """
-    if not plugin_pkg:
-        return {}
-    
-    hooks = {}
-    
-    plugin_module = importlib.import_module(plugin_pkg)
-    for attr_name in dir(plugin_module):
-        if attr_name.startswith('_'):
-            continue
-        try:
-            attr = getattr(plugin_module, attr_name)
-            if isinstance(attr, Callable):
-                hooks[attr_name] = None
-                pm.parse_hookimpl_opts(plugin_module, attr_name)
-                hooks[attr_name] = attr
-        except Exception as e:
-            print(f'Error getting hookimpls for {plugin_pkg}: {e}')
-
-    return hooks
@@ -1,30 +0,0 @@
-__package__ = 'abx.archivebox'
-
-import os
-import importlib
-
-from typing import Dict
-from pathlib import Path
-
-
-def load_archivebox_plugins(pm, plugins_dict: Dict[str, Path]):
-    """Load archivebox plugins, very similar to abx.load_plugins but it looks for a pydantic PLUGIN model + hooks in apps.py"""
-    LOADED_PLUGINS = {}
-    for plugin_module, plugin_dir in reversed(plugins_dict.items()):
-        # print(f'Loading plugin: {plugin_module} from {plugin_dir}')
-        
-        # 1. register the plugin module directly in case it contains any look hookimpls (e.g. in __init__.py)
-        try:
-            plugin_module_loaded = importlib.import_module(plugin_module)
-            pm.register(plugin_module_loaded)
-        except Exception as e:
-            print(f'Error registering plugin: {plugin_module} - {e}')
-            
-        
-        # 2. then try to import plugin_module.apps as well
-        if os.access(plugin_dir / 'apps.py', os.R_OK):
-            plugin_apps = importlib.import_module(plugin_module + '.apps')
-            pm.register(plugin_apps)                                           # register the whole .apps  in case it contains loose hookimpls (not in a class)
-            
-        # print(f'    √ Loaded plugin: {plugin_module} {len(archivebox_plugins_found) * "🧩"}')
-    return LOADED_PLUGINS
@@ -1,117 +0,0 @@
-__package__ = "abx.archivebox"
-
-import os
-from typing import Optional, cast
-from typing_extensions import Self
-
-from pydantic import validate_call
-from pydantic_pkgr import (
-    Binary,
-    BinProvider,
-    BinProviderName,
-    AptProvider,
-    BrewProvider,
-    EnvProvider,
-)
-
-from archivebox.config import CONSTANTS
-from archivebox.config.permissions import ARCHIVEBOX_USER
-
-import abx
-
-
-class BaseBinProvider(BinProvider):
-    
-    # TODO: add install/load/load_or_install methods as abx.hookimpl methods
-    
-    @property
-    def admin_url(self) -> str:
-        # e.g. /admin/environment/binproviders/NpmBinProvider/   TODO
-        return "/admin/environment/binaries/"
-
-    @abx.hookimpl
-    def get_BINPROVIDERS(self):
-        return [self]
-
-class BaseBinary(Binary):
-
-    @staticmethod
-    def symlink_to_lib(binary, bin_dir=None) -> None:
-        from archivebox.config.common import STORAGE_CONFIG
-        bin_dir = bin_dir or STORAGE_CONFIG.LIB_DIR / 'bin'
-        
-        if not (binary.abspath and os.access(binary.abspath, os.R_OK)):
-            return
-        
-        try:
-            bin_dir.mkdir(parents=True, exist_ok=True)
-            symlink = bin_dir / binary.name
-            symlink.unlink(missing_ok=True)
-            symlink.symlink_to(binary.abspath)
-            symlink.chmod(0o777)   # make sure its executable by everyone
-        except Exception as err:
-            # print(f'[red]:warning: Failed to symlink {symlink} -> {binary.abspath}[/red] {err}')
-            # not actually needed, we can just run without it
-            pass
-        
-    @validate_call
-    def load(self, fresh=False, **kwargs) -> Self:
-        from archivebox.config.common import STORAGE_CONFIG
-        if fresh:
-            binary = super().load(**kwargs)
-            self.symlink_to_lib(binary=binary, bin_dir=STORAGE_CONFIG.LIB_DIR / 'bin')
-        else:
-            # get cached binary from db
-            try:
-                from machine.models import InstalledBinary
-                installed_binary = InstalledBinary.objects.get_from_db_or_cache(self)    # type: ignore
-                binary = InstalledBinary.load_from_db(installed_binary)
-            except Exception:
-                # maybe we are not in a DATA dir so there is no db, fallback to reading from fs
-                # (e.g. when archivebox version is run outside of a DATA dir)
-                binary = super().load(**kwargs)
-        return cast(Self, binary)
-    
-    @validate_call
-    def install(self, **kwargs) -> Self:
-        from archivebox.config.common import STORAGE_CONFIG
-        binary = super().install(**kwargs)
-        self.symlink_to_lib(binary=binary, bin_dir=STORAGE_CONFIG.LIB_DIR / 'bin')
-        return binary
-    
-    @validate_call
-    def load_or_install(self, fresh=False, **kwargs) -> Self:
-        from archivebox.config.common import STORAGE_CONFIG
-        try:
-            binary = self.load(fresh=fresh)
-            if binary and binary.version:
-                self.symlink_to_lib(binary=binary, bin_dir=STORAGE_CONFIG.LIB_DIR / 'bin')
-                return binary
-        except Exception:
-            pass
-        return self.install(**kwargs)
-    
-    @property
-    def admin_url(self) -> str:
-        # e.g. /admin/environment/config/LdapConfig/
-        return f"/admin/environment/binaries/{self.name}/"
-
-    @abx.hookimpl
-    def get_BINARIES(self):
-        return [self]
-
-
-class AptBinProvider(AptProvider, BaseBinProvider):
-    name: BinProviderName = "apt"
-    
-class BrewBinProvider(BrewProvider, BaseBinProvider):
-    name: BinProviderName = "brew"
-    
-class EnvBinProvider(EnvProvider, BaseBinProvider):
-    name: BinProviderName = "env"
-    
-    euid: Optional[int] = ARCHIVEBOX_USER
-
-apt = AptBinProvider()
-brew = BrewBinProvider()
-env = EnvBinProvider()
@@ -1,219 +0,0 @@
-__package__ = 'abx.archivebox'
-
-import json
-import os
-
-from typing import Optional, List, Literal, Annotated, Dict, Any, Tuple
-from typing_extensions import Self
-from pathlib import Path
-
-from pydantic import model_validator, AfterValidator
-from pydantic_pkgr import BinName
-from django.utils.functional import cached_property
-from django.utils import timezone
-
-import abx
-
-from .base_binary import BaseBinary
-
-
-def no_empty_args(args: List[str]) -> List[str]:
-    assert all(len(arg) for arg in args)
-    return args
-
-ExtractorName = Literal['wget', 'warc', 'media', 'singlefile'] | str
-
-HandlerFuncStr = Annotated[str, AfterValidator(lambda s: s.startswith('self.'))]
-CmdArgsList = Annotated[List[str] | Tuple[str, ...], AfterValidator(no_empty_args)]
-
-
-class BaseExtractor:
-    
-    name: ExtractorName
-    binary: BinName
-
-    output_path_func: HandlerFuncStr = 'self.get_output_path'
-    should_extract_func: HandlerFuncStr = 'self.should_extract'
-    extract_func: HandlerFuncStr = 'self.extract'
-    exec_func: HandlerFuncStr = 'self.exec'
-
-    default_args: CmdArgsList = []
-    extra_args: CmdArgsList = []
-    args: Optional[CmdArgsList] = None
-
-    @model_validator(mode='after')
-    def validate_model(self) -> Self:
-        if self.args is None:
-            self.args = [*self.default_args, *self.extra_args]
-        return self
-
-
-    def get_output_path(self, snapshot) -> Path:
-        return Path(self.__class__.__name__.lower())
-
-    def should_extract(self, uri: str, config: dict | None=None) -> bool:
-        try:
-            assert self.detect_installed_binary().version
-        except Exception:
-            raise
-            # could not load binary
-            return False
-        
-        # output_dir = self.get_output_path(snapshot)
-        # if output_dir.glob('*.*'):
-        #     return False
-        return True
-
-    @abx.hookimpl
-    def extract(self, snapshot_id: str) -> Dict[str, Any]:
-        from core.models import Snapshot
-        from archivebox import CONSTANTS
-        
-        snapshot = Snapshot.objects.get(id=snapshot_id)
-        
-        if not self.should_extract(snapshot):
-            return {}
-        
-        status = 'failed'
-        start_ts = timezone.now()
-        uplink = self.detect_network_interface()
-        installed_binary = self.detect_installed_binary()
-        machine = installed_binary.machine
-        assert uplink.machine == installed_binary.machine  # it would be *very* weird if this wasn't true
-        
-        output_dir = CONSTANTS.DATA_DIR / '.tmp' / 'extractors' / self.name / str(snapshot.abid)
-        output_dir.mkdir(parents=True, exist_ok=True)
-
-        # execute the extractor binary with the given args
-        args = [snapshot.url, *self.args] if self.args is not None else [snapshot.url, *self.default_args, *self.extra_args]
-        cmd = [str(installed_binary.abspath), *args]
-        proc = self.exec(installed_binary=installed_binary, args=args, cwd=output_dir)
-
-        # collect the output
-        end_ts = timezone.now()
-        output_files = list(str(path.relative_to(output_dir)) for path in output_dir.glob('**/*.*'))
-        stdout = proc.stdout.strip()
-        stderr = proc.stderr.strip()
-        output_json = None
-        output_text = stdout
-        try:
-            output_json = json.loads(stdout.strip())
-            output_text = None
-        except json.JSONDecodeError:
-            pass
-        
-        errors = []
-        if proc.returncode == 0:
-            status = 'success'
-        else:
-            errors.append(f'{installed_binary.name} returned non-zero exit code: {proc.returncode}')   
-
-        # increment health stats counters
-        if status == 'success':
-            machine.record_health_success()
-            uplink.record_health_success()
-            installed_binary.record_health_success()
-        else:
-            machine.record_health_failure()
-            uplink.record_health_failure()
-            installed_binary.record_health_failure()
-
-        return {
-            'extractor': self.name,
-            
-            'snapshot': {
-                'id': snapshot.id,
-                'abid': snapshot.abid,
-                'url': snapshot.url,
-                'created_by_id': snapshot.created_by_id,
-            },
-            
-            'machine': {
-                'id': machine.id,
-                'abid': machine.abid,
-                'guid': machine.guid,
-                'hostname': machine.hostname,
-                'hw_in_docker': machine.hw_in_docker,
-                'hw_in_vm': machine.hw_in_vm,
-                'hw_manufacturer': machine.hw_manufacturer,
-                'hw_product': machine.hw_product,
-                'hw_uuid': machine.hw_uuid,
-                'os_arch': machine.os_arch,
-                'os_family': machine.os_family,
-                'os_platform': machine.os_platform,
-                'os_release': machine.os_release,
-                'os_kernel': machine.os_kernel,
-            },
-            
-            'uplink': { 
-                'id': uplink.id,
-                'abid': uplink.abid,
-                'mac_address': uplink.mac_address,
-                'ip_public': uplink.ip_public,
-                'ip_local': uplink.ip_local,
-                'dns_server': uplink.dns_server,
-                'hostname': uplink.hostname,
-                'iface': uplink.iface,
-                'isp': uplink.isp,
-                'city': uplink.city,
-                'region': uplink.region,
-                'country': uplink.country,
-            },
-            
-            'binary': {
-                'id': installed_binary.id,
-                'abid': installed_binary.abid,
-                'name': installed_binary.name,
-                'binprovider': installed_binary.binprovider,
-                'abspath': installed_binary.abspath,
-                'version': installed_binary.version,
-                'sha256': installed_binary.sha256,
-            },
-
-            'cmd': cmd,
-            'stdout': stdout,
-            'stderr': stderr,
-            'returncode': proc.returncode,
-            'start_ts': start_ts,
-            'end_ts': end_ts,
-            
-            'status': status,
-            'errors': errors,
-            'output_dir': str(output_dir.relative_to(CONSTANTS.DATA_DIR)),
-            'output_files': output_files,
-            'output_json': output_json or {},
-            'output_text': output_text or '',
-        }
-
-    # TODO: move this to a hookimpl
-    def exec(self, args: CmdArgsList=(), cwd: Optional[Path]=None, installed_binary=None):
-        cwd = cwd or Path(os.getcwd())
-        binary = self.load_binary(installed_binary=installed_binary)
-        
-        return binary.exec(cmd=args, cwd=cwd)
-    
-    @cached_property
-    def BINARY(self) -> BaseBinary:
-        import abx.archivebox.reads
-        for binary in abx.archivebox.reads.get_BINARIES().values():
-            if binary.name == self.binary:
-                return binary
-        raise ValueError(f'Binary {self.binary} not found')
-    
-    def detect_installed_binary(self):
-        from machine.models import InstalledBinary
-        # hydrates binary from DB/cache if record of installed version is recent enough
-        # otherwise it finds it from scratch by detecting installed version/abspath/sha256 on host
-        return InstalledBinary.objects.get_from_db_or_cache(self.BINARY)
-
-    def load_binary(self, installed_binary=None) -> BaseBinary:
-        installed_binary = installed_binary or self.detect_installed_binary()
-        return installed_binary.load_from_db()
-    
-    def detect_network_interface(self):
-        from machine.models import NetworkInterface
-        return NetworkInterface.objects.current()
-
-    @abx.hookimpl
-    def get_EXTRACTORS(self):
-        return [self]
@@ -1,25 +0,0 @@
-__package__ = 'abx.archivebox'
-
-import abx
-
-
-class BaseReplayer:
-    """Describes how to render an ArchiveResult in several contexts"""
-    
-    url_pattern: str = '*'
-
-    row_template: str = 'plugins/generic_replayer/templates/row.html'
-    embed_template: str = 'plugins/generic_replayer/templates/embed.html'
-    fullpage_template: str = 'plugins/generic_replayer/templates/fullpage.html'
-
-    # row_view: LazyImportStr = 'plugins.generic_replayer.views.row_view'
-    # embed_view: LazyImportStr = 'plugins.generic_replayer.views.embed_view'
-    # fullpage_view: LazyImportStr = 'plugins.generic_replayer.views.fullpage_view'
-    # icon_view: LazyImportStr = 'plugins.generic_replayer.views.get_icon'
-    # thumbnail_view: LazyImportStr = 'plugins.generic_replayer.views.get_icon'
-
-    @abx.hookimpl
-    def get_REPLAYERS(self):
-        return [self]
-
-    # TODO: add hookimpl methods for get_row_template, get_embed_template, get_fullpage_template, etc...
@@ -1,25 +0,0 @@
-__package__ = 'abx.archivebox'
-
-from typing import Iterable, List
-import abc
-
-
-
-class BaseSearchBackend(abc.ABC):
-    name: str
-
-    @staticmethod
-    @abc.abstractmethod
-    def index(snapshot_id: str, texts: List[str]):
-        return
-
-    @staticmethod
-    @abc.abstractmethod
-    def flush(snapshot_ids: Iterable[str]):
-        return
-
-    @staticmethod
-    @abc.abstractmethod
-    def search(text: str) -> List[str]:
-        raise NotImplementedError("search method must be implemented by subclass")
-
@@ -1,52 +0,0 @@
-__package__ = 'abx.archivebox'
-
-from typing import Dict, Any
-
-from .. import hookspec
-
-from .base_binary import BaseBinary, BaseBinProvider
-from .base_configset import BaseConfigSet
-from .base_extractor import BaseExtractor
-from .base_searchbackend import BaseSearchBackend
-
-
-@hookspec
-def get_PLUGIN() -> Dict[str, Dict[str, Any]]:
-    return {}
-
-@hookspec
-def get_CONFIG() -> Dict[str, BaseConfigSet]:
-    return {}
-
-
-
-@hookspec
-def get_EXTRACTORS() -> Dict[str, BaseExtractor]:
-    return {}
-
-@hookspec
-def get_SEARCHBACKENDS() -> Dict[str, BaseSearchBackend]:
-    return {}
-
-# @hookspec
-# def get_REPLAYERS() -> Dict[str, BaseReplayer]:
-#     return {}
-
-# @hookspec
-# def get_ADMINDATAVIEWS():
-#     return {}
-
-# @hookspec
-# def get_QUEUES():
-#     return {}
-
-
-##############################################################
-# provided by abx.pydantic_pkgr.hookspec:
-# @hookspec
-# def get_BINARIES() -> Dict[str, BaseBinary]:
-#     return {}
-
-# @hookspec
-# def get_BINPROVIDERS() -> Dict[str, BaseBinProvider]:
-#     return {}
@@ -1,160 +0,0 @@
-__package__ = 'abx.archivebox'
-
-import importlib
-from typing import Dict, Set, Any, TYPE_CHECKING
-
-from benedict import benedict
-
-import abx
-from .. import pm
-
-if TYPE_CHECKING:
-    from .base_configset import BaseConfigSet
-    from .base_binary import BaseBinary, BaseBinProvider
-    from .base_extractor import BaseExtractor
-    from .base_searchbackend import BaseSearchBackend
-    # from .base_replayer import BaseReplayer
-    # from .base_queue import BaseQueue
-    # from .base_admindataview import BaseAdminDataView
-
-# API exposed to ArchiveBox code
-
-def get_PLUGINS() -> Dict[str, Dict[str, Any]]:
-    return benedict({
-        plugin_id: plugin
-        for plugin_dict in pm.hook.get_PLUGIN()
-            for plugin_id, plugin in plugin_dict.items()
-    })
-
-def get_PLUGIN(plugin_id: str) -> Dict[str, Any]:
-    plugin_info = get_PLUGINS().get(plugin_id, {})
-    package = plugin_info.get('package', plugin_info.get('PACKAGE', None))
-    if not package:
-        return {'id': plugin_id, 'hooks': {}}
-    module = importlib.import_module(package)
-    hooks = abx.get_plugin_hooks(module.__package__)
-    assert plugin_info and (plugin_info.get('id') or plugin_info.get('ID') or hooks)
-    
-    return benedict({
-        'id': plugin_id,
-        'label': getattr(module, '__label__', plugin_id),
-        'module': module,
-        'package': module.__package__,
-        'hooks': hooks,
-        'version': getattr(module, '__version__', '999.999.999'),
-        'author': getattr(module, '__author__', 'Unknown'),
-        'homepage': getattr(module, '__homepage__', 'https://github.com/ArchiveBox/ArchiveBox'),
-        'dependencies': getattr(module, '__dependencies__', []),
-        'source_code': module.__file__,
-        **plugin_info,
-    })
-    
-
-def get_HOOKS() -> Set[str]:
-    return {
-        hook_name
-        for plugin_id in get_PLUGINS().keys()
-            for hook_name in get_PLUGIN(plugin_id).hooks
-    }
-
-def get_CONFIGS() -> Dict[str, 'BaseConfigSet']:
-    return benedict({
-        config_id: configset
-        for plugin_configs in pm.hook.get_CONFIG()
-            for config_id, configset in plugin_configs.items()
-    })
-
-
-def get_FLAT_CONFIG() -> Dict[str, Any]:
-    return benedict({
-        key: value
-        for configset in get_CONFIGS().values()
-            for key, value in configset.model_dump().items()
-    })
-
-def get_BINPROVIDERS() -> Dict[str, 'BaseBinProvider']:
-    # TODO: move these to plugins
-    from abx.archivebox.base_binary import apt, brew, env
-    builtin_binproviders = {
-        'env': env,
-        'apt': apt,
-        'brew': brew,
-    }
-    
-    return benedict({
-        binprovider_id: binprovider
-        for plugin_binproviders in [builtin_binproviders, *pm.hook.get_BINPROVIDERS()]
-            for binprovider_id, binprovider in plugin_binproviders.items()
-    })
-
-def get_BINARIES() -> Dict[str, 'BaseBinary']:
-    return benedict({
-        binary_id: binary
-        for plugin_binaries in pm.hook.get_BINARIES()
-            for binary_id, binary in plugin_binaries.items()
-    })
-
-def get_EXTRACTORS() -> Dict[str, 'BaseExtractor']:
-    return benedict({
-        extractor_id: extractor
-        for plugin_extractors in pm.hook.get_EXTRACTORS()
-            for extractor_id, extractor in plugin_extractors.items()
-    })
-
-# def get_REPLAYERS() -> Dict[str, 'BaseReplayer']:
-#     return benedict({
-#         replayer.id: replayer
-#         for plugin_replayers in pm.hook.get_REPLAYERS()
-#             for replayer in plugin_replayers
-#     })
-
-# def get_ADMINDATAVIEWS() -> Dict[str, 'BaseAdminDataView']:
-#     return benedict({
-#         admin_dataview.id: admin_dataview
-#         for plugin_admin_dataviews in pm.hook.get_ADMINDATAVIEWS()
-#             for admin_dataview in plugin_admin_dataviews
-#     })
-
-# def get_QUEUES() -> Dict[str, 'BaseQueue']:
-#     return benedict({
-#         queue.id: queue
-#         for plugin_queues in pm.hook.get_QUEUES()
-#             for queue in plugin_queues
-#     })
-
-def get_SEARCHBACKENDS() -> Dict[str, 'BaseSearchBackend']:
-    return benedict({
-        searchbackend_id: searchbackend
-        for plugin_searchbackends in pm.hook.get_SEARCHBACKENDS()
-            for searchbackend_id,searchbackend in plugin_searchbackends.items()
-    })
-
-
-
-def get_scope_config(defaults: benedict | None = None, persona=None, seed=None, crawl=None, snapshot=None, archiveresult=None, extra_config=None):
-    """Get all the relevant config for the given scope, in correct precedence order"""
-    
-    from django.conf import settings
-    default_config: benedict = defaults or settings.CONFIG
-    
-    snapshot = snapshot or (archiveresult and archiveresult.snapshot)
-    crawl = crawl or (snapshot and snapshot.crawl)
-    seed = seed or (crawl and crawl.seed)
-    persona = persona or (crawl and crawl.persona)
-    
-    persona_config = persona.config if persona else {}
-    seed_config = seed.config if seed else {}
-    crawl_config = crawl.config if crawl else {}
-    snapshot_config = snapshot.config if snapshot else {}
-    archiveresult_config = archiveresult.config if archiveresult else {}
-    extra_config = extra_config or {}
-    
-    return {
-        **default_config,               # defaults / config file / environment variables
-        **persona_config,               # lowest precedence
-        **seed_config,
-        **crawl_config,
-        **snapshot_config,
-        **archiveresult_config,
-        **extra_config,                 # highest precedence
-    }
@@ -1,133 +0,0 @@
-__package__ = 'abx.archivebox'
-
-import importlib
-from typing import Dict, Set, Any, TYPE_CHECKING
-
-from benedict import benedict
-
-from django.conf import settings
-
-import abx
-from .. import pm
-
-
-@abx.hookimpl
-def get_or_create_snapshot(crawl, url, config):
-    pass
-
-@abx.hookimpl
-def update_crawl_schedule_next_run_at(crawl_schedule, next_run_at):
-    pass
-
-@abx.hookimpl
-def create_crawl_copy(crawl_to_copy, schedule):
-    pass
-
-@abx.hookimpl
-def create_crawl(seed, depth, tags_str, persona, created_by, config, schedule):
-    pass
-
-
-
-
-def create_crawl_from_ui_action(urls, extractor, credentials, depth, tags_str, persona, created_by, crawl_config):
-    if seed_is_remote(urls, extractor, credentials):
-        # user's seed is a remote source that will provide the urls (e.g. RSS feed URL, Pocket API, etc.)
-        uri, extractor, credentials = abx.archivebox.effects.check_remote_seed_connection(urls, extractor, credentials, created_by)
-    else:
-        # user's seed is some raw text they provided to parse for urls, save it to a file then load the file as a Seed
-        uri = abx.archivebox.writes.write_raw_urls_to_local_file(urls, extractor, tags_str, created_by)  # file:///data/sources/some_import.txt
-    
-    seed = abx.archivebox.writes.get_or_create_seed(uri=remote_uri, extractor, credentials, created_by)
-    # abx.archivebox.events.on_seed_created(seed)
-        
-    crawl = abx.archivebox.writes.create_crawl(seed=seed, depth=depth, tags_str=tags_str, persona=persona, created_by=created_by, config=crawl_config, schedule=None)
-    abx.archivebox.events.on_crawl_created(crawl)
-
-
-@abx.hookimpl(specname='on_crawl_schedule_tick')
-def create_crawl_from_crawlschedule_if_due(crawl_schedule):
-    # make sure it's not too early to run this scheduled import (makes this function indepmpotent / safe to call multiple times / every second)
-    if timezone.now() < crawl_schedule.next_run_at:
-        # it's not time to run it yet, wait for the next tick
-        return
-    else:
-        # we're going to run it now, bump the next run time so that no one else runs it at the same time as us
-        abx.archivebox.writes.update_crawl_schedule_next_run_at(crawl_schedule, next_run_at=crawl_schedule.next_run_at + crawl_schedule.interval)
-    
-    crawl_to_copy = None
-    try:
-        crawl_to_copy = crawl_schedule.crawl_set.first()  # alternatively use .last() to copy most recent crawl instead of very first crawl
-    except Crawl.DoesNotExist:
-        # there is no template crawl to base the next one off of
-        # user must add at least one crawl to a schedule that serves as the template for all future repeated crawls
-        return
-    
-    new_crawl = abx.archivebox.writes.create_crawl_copy(crawl_to_copy=crawl_to_copy, schedule=crawl_schedule)
-    abx.archivebox.events.on_crawl_created(new_crawl)
-
-
-@abx.hookimpl(specname='on_crawl_post_save')
-def create_root_snapshot_from_seed(crawl):
-    # create a snapshot for the seed URI which kicks off the crawl
-    # only a single extractor will run on it, which will produce outlinks which get added back to the crawl
-    root_snapshot, created = abx.archivebox.writes.get_or_create_snapshot(crawl=crawl, url=crawl.seed.uri, config={
-        'extractors': (
-            abx.archivebox.reads.get_extractors_that_produce_outlinks()
-            if crawl.seed.extractor == 'auto' else
-            [crawl.seed.extractor]
-        ),
-        **crawl.seed.config,
-    })
-    if created:
-        abx.archivebox.events.on_snapshot_created(root_snapshot)
-        abx.archivebox.writes.update_crawl_stats(started_at=timezone.now())
-
-
-@abx.hookimpl(specname='on_snapshot_created')
-def create_archiveresults_pending_from_snapshot(snapshot, config):
-    config = get_scope_config(
-        # defaults=settings.CONFIG_FROM_DEFAULTS,
-        # configfile=settings.CONFIG_FROM_FILE,
-        # environment=settings.CONFIG_FROM_ENVIRONMENT,
-        persona=archiveresult.snapshot.crawl.persona,
-        seed=archiveresult.snapshot.crawl.seed,
-        crawl=archiveresult.snapshot.crawl,
-        snapshot=archiveresult.snapshot,
-        archiveresult=archiveresult,
-        # extra_config=extra_config,
-    )
-    
-    extractors = abx.archivebox.reads.get_extractors_for_snapshot(snapshot, config)
-    for extractor in extractors:
-        archiveresult, created = abx.archivebox.writes.get_or_create_archiveresult_pending(
-            snapshot=snapshot,
-            extractor=extractor,
-            status='pending'
-        )
-        if created:
-            abx.archivebox.events.on_archiveresult_created(archiveresult)
-
-
-
-@abx.hookimpl(specname='on_archiveresult_updated')
-def create_snapshots_pending_from_archiveresult_outlinks(archiveresult):
-    config = get_scope_config(...)
-    
-    # check if extractor has finished succesfully, if not, dont bother checking for outlinks
-    if not archiveresult.status == 'succeeded':
-        return
-    
-    # check if we have already reached the maximum recursion depth
-    hops_to_here = abx.archivebox.reads.get_outlink_parents(crawl_pk=archiveresult.snapshot.crawl_id, url=archiveresult.url, config=config)
-    if len(hops_to_here) >= archiveresult.crawl.max_depth +1:
-        return
-    
-    # parse the output to get outlink url_entries
-    discovered_urls = abx.archivebox.reads.get_archiveresult_discovered_url_entries(archiveresult, config=config)
-    
-    for url_entry in discovered_urls:
-        abx.archivebox.writes.create_outlink_record(src=archiveresult.snapshot.url, dst=url_entry.url, via=archiveresult)
-        abx.archivebox.writes.create_snapshot(crawl=archiveresult.snapshot.crawl, url_entry=url_entry)
-        
-    # abx.archivebox.events.on_crawl_updated(archiveresult.snapshot.crawl)
@@ -1 +0,0 @@
-__package__ = 'abx.django'
@@ -1,13 +0,0 @@
-__package__ = 'abx.django'
-
-from django.apps import AppConfig
-
-
-class ABXConfig(AppConfig):
-    name = 'abx'
-
-    def ready(self):
-        import abx
-        from django.conf import settings
-        
-        abx.pm.hook.ready(settings=settings)
@@ -1,125 +0,0 @@
-__package__ = 'abx.django'
-
-from ..hookspec import hookspec
-
-
-###########################################################################################
-
-@hookspec
-def get_INSTALLED_APPS():
-    """Return a list of apps to add to INSTALLED_APPS"""
-    # e.g. ['your_plugin_type.plugin_name']
-    return []
-
-# @hookspec
-# def register_INSTALLED_APPS(INSTALLED_APPS):
-#     """Mutate INSTALLED_APPS in place to add your app in a specific position"""
-#     # idx_of_contrib = INSTALLED_APPS.index('django.contrib.auth')
-#     # INSTALLED_APPS.insert(idx_of_contrib + 1, 'your_plugin_type.plugin_name')
-#     pass
-
-
-@hookspec
-def get_TEMPLATE_DIRS():
-    return []     # e.g. ['your_plugin_type/plugin_name/templates']
-
-# @hookspec
-# def register_TEMPLATE_DIRS(TEMPLATE_DIRS):
-#     """Install django settings"""
-#     # e.g. TEMPLATE_DIRS.insert(0, 'your_plugin_type/plugin_name/templates')
-#     pass
-
-
-@hookspec
-def get_STATICFILES_DIRS():
-    return []     # e.g. ['your_plugin_type/plugin_name/static']
-
-# @hookspec
-# def register_STATICFILES_DIRS(STATICFILES_DIRS):
-#     """Mutate STATICFILES_DIRS in place to add your static dirs in a specific position"""
-#     # e.g. STATICFILES_DIRS.insert(0, 'your_plugin_type/plugin_name/static')
-#     pass
-
-
-@hookspec
-def get_MIDDLEWARE():
-    return []     # e.g. ['your_plugin_type.plugin_name.middleware.YourMiddleware']
-
-# @hookspec
-# def register_MIDDLEWARE(MIDDLEWARE):
-#     """Mutate MIDDLEWARE in place to add your middleware in a specific position"""
-#     # e.g. MIDDLEWARE.insert(0, 'your_plugin_type.plugin_name.middleware.YourMiddleware')
-#     pass
-
-
-@hookspec
-def get_AUTHENTICATION_BACKENDS():
-    return []     # e.g. ['django_auth_ldap.backend.LDAPBackend']
-
-# @hookspec
-# def register_AUTHENTICATION_BACKENDS(AUTHENTICATION_BACKENDS):
-#     """Mutate AUTHENTICATION_BACKENDS in place to add your auth backends in a specific position"""
-#     # e.g. AUTHENTICATION_BACKENDS.insert(0, 'your_plugin_type.plugin_name.backend.YourBackend')
-#     pass
-
-@hookspec
-def get_DJANGO_HUEY_QUEUES(QUEUE_DATABASE_NAME):
-    return []     # e.g. [{'name': 'your_plugin_type.plugin_name', 'HUEY': {...}}]
-
-# @hookspec
-# def register_DJANGO_HUEY(DJANGO_HUEY):
-#     """Mutate DJANGO_HUEY in place to add your huey queues in a specific position"""
-#     # e.g. DJANGO_HUEY['queues']['some_queue_name']['some_setting'] = 'some_value'
-#     pass
-
-
-@hookspec
-def get_ADMIN_DATA_VIEWS_URLS():
-    return []
-
-# @hookspec
-# def register_ADMIN_DATA_VIEWS(ADMIN_DATA_VIEWS):
-#     """Mutate ADMIN_DATA_VIEWS in place to add your admin data views in a specific position"""
-#     # e.g. ADMIN_DATA_VIEWS['URLS'].insert(0, 'your_plugin_type/plugin_name/admin_data_views.py')
-#     pass
-
-
-# @hookspec
-# def register_settings(settings):
-#     """Mutate settings in place to add your settings / modify existing settings"""
-#     # settings.SOME_KEY = 'some_value'
-#     pass
-
-
-###########################################################################################
-
-@hookspec
-def get_urlpatterns():
-    return []     # e.g. [path('your_plugin_type/plugin_name/url.py', your_view)]
-
-# @hookspec
-# def register_urlpatterns(urlpatterns):
-#     """Mutate urlpatterns in place to add your urlpatterns in a specific position"""
-#     # e.g. urlpatterns.insert(0, path('your_plugin_type/plugin_name/url.py', your_view))
-#     pass
-
-###########################################################################################
-
-@hookspec
-def register_checks():
-    """Register django checks with django system checks system"""
-    pass
-
-@hookspec
-def register_admin(admin_site):
-    """Register django admin views/models with the main django admin site instance"""
-    pass
-
-
-###########################################################################################
-
-
-@hookspec
-def ready():
-    """Called when Django apps app.ready() are triggered"""
-    pass
@@ -1,101 +0,0 @@
-__package__ = 'abx.django'
-
-import itertools
-# from benedict import benedict
-
-from .. import pm
-
-
-def get_INSTALLED_APPS():
-    return itertools.chain(*reversed(pm.hook.get_INSTALLED_APPS()))
-
-# def register_INSTALLLED_APPS(INSTALLED_APPS):
-#     pm.hook.register_INSTALLED_APPS(INSTALLED_APPS=INSTALLED_APPS)
-
-
-def get_MIDDLEWARES():
-    return itertools.chain(*reversed(pm.hook.get_MIDDLEWARE()))
-
-# def register_MIDDLEWARES(MIDDLEWARE):
-#     pm.hook.register_MIDDLEWARE(MIDDLEWARE=MIDDLEWARE)
-
-
-def get_AUTHENTICATION_BACKENDS():
-    return itertools.chain(*reversed(pm.hook.get_AUTHENTICATION_BACKENDS()))
-
-# def register_AUTHENTICATION_BACKENDS(AUTHENTICATION_BACKENDS):
-#     pm.hook.register_AUTHENTICATION_BACKENDS(AUTHENTICATION_BACKENDS=AUTHENTICATION_BACKENDS)
-
-
-def get_STATICFILES_DIRS():
-    return itertools.chain(*reversed(pm.hook.get_STATICFILES_DIRS()))
-
-# def register_STATICFILES_DIRS(STATICFILES_DIRS):
-#     pm.hook.register_STATICFILES_DIRS(STATICFILES_DIRS=STATICFILES_DIRS)
-
-
-def get_TEMPLATE_DIRS():
-    return itertools.chain(*reversed(pm.hook.get_TEMPLATE_DIRS()))
-
-# def register_TEMPLATE_DIRS(TEMPLATE_DIRS):
-#     pm.hook.register_TEMPLATE_DIRS(TEMPLATE_DIRS=TEMPLATE_DIRS)
-
-def get_DJANGO_HUEY_QUEUES(QUEUE_DATABASE_NAME='queue.sqlite3'):
-    HUEY_QUEUES = {}
-    for plugin_result in pm.hook.get_DJANGO_HUEY_QUEUES(QUEUE_DATABASE_NAME=QUEUE_DATABASE_NAME):
-        HUEY_QUEUES.update(plugin_result)
-    return HUEY_QUEUES
-
-# def register_DJANGO_HUEY(DJANGO_HUEY):
-#     pm.hook.register_DJANGO_HUEY(DJANGO_HUEY=DJANGO_HUEY)
-
-def get_ADMIN_DATA_VIEWS_URLS():
-    return itertools.chain(*reversed(pm.hook.get_ADMIN_DATA_VIEWS_URLS()))
-
-# def register_ADMIN_DATA_VIEWS(ADMIN_DATA_VIEWS):
-#     pm.hook.register_ADMIN_DATA_VIEWS(ADMIN_DATA_VIEWS=ADMIN_DATA_VIEWS)
-
-
-# def register_settings(settings):
-#     # convert settings dict to an benedict so we can set values using settings.attr = xyz notation
-#     settings_as_obj = benedict(settings, keypath_separator=None)
-    
-#     # set default values for settings that are used by plugins
-#     # settings_as_obj.INSTALLED_APPS = settings_as_obj.get('INSTALLED_APPS', [])
-#     # settings_as_obj.MIDDLEWARE = settings_as_obj.get('MIDDLEWARE', [])
-#     # settings_as_obj.AUTHENTICATION_BACKENDS = settings_as_obj.get('AUTHENTICATION_BACKENDS', [])
-#     # settings_as_obj.STATICFILES_DIRS = settings_as_obj.get('STATICFILES_DIRS', [])
-#     # settings_as_obj.TEMPLATE_DIRS = settings_as_obj.get('TEMPLATE_DIRS', [])
-#     # settings_as_obj.DJANGO_HUEY = settings_as_obj.get('DJANGO_HUEY', {'queues': {}})
-#     # settings_as_obj.ADMIN_DATA_VIEWS = settings_as_obj.get('ADMIN_DATA_VIEWS', {'URLS': []})
-    
-#     # # call all the hook functions to mutate the settings values in-place
-#     # register_INSTALLLED_APPS(settings_as_obj.INSTALLED_APPS)
-#     # register_MIDDLEWARES(settings_as_obj.MIDDLEWARE)
-#     # register_AUTHENTICATION_BACKENDS(settings_as_obj.AUTHENTICATION_BACKENDS)
-#     # register_STATICFILES_DIRS(settings_as_obj.STATICFILES_DIRS)
-#     # register_TEMPLATE_DIRS(settings_as_obj.TEMPLATE_DIRS)
-#     # register_DJANGO_HUEY(settings_as_obj.DJANGO_HUEY)
-#     # register_ADMIN_DATA_VIEWS(settings_as_obj.ADMIN_DATA_VIEWS)
-    
-#     # calls Plugin.settings(settings) on each registered plugin
-#     pm.hook.register_settings(settings=settings_as_obj)
-    
-#     # then finally update the settings globals() object will all the new settings
-#     # settings.update(settings_as_obj)
-
-
-def get_urlpatterns():
-    return list(itertools.chain(*pm.hook.urlpatterns()))
-
-def register_urlpatterns(urlpatterns):
-    pm.hook.register_urlpatterns(urlpatterns=urlpatterns)
-
-
-def register_checks():
-    """register any django system checks"""
-    pm.hook.register_checks()
-
-def register_admin(admin_site):
-    """register any django admin models/views with the main django admin site instance"""
-    pm.hook.register_admin(admin_site=admin_site)
@@ -1,14 +0,0 @@
-from pathlib import Path
-
-from pluggy import HookimplMarker
-from pluggy import HookspecMarker
-
-spec = hookspec = HookspecMarker("abx")
-impl = hookimpl = HookimplMarker("abx")
-
-
-@hookspec
-@hookimpl
-def get_system_user() -> str:
-    return Path('~').expanduser().name
-
@@ -1,30 +0,0 @@
-import inspect
-
-import pluggy
-
-
-class PluginManager(pluggy.PluginManager):
-    """
-    Patch to fix pluggy's PluginManager to work with pydantic models.
-    See: https://github.com/pytest-dev/pluggy/pull/536
-    """
-    def parse_hookimpl_opts(self, plugin, name: str) -> pluggy.HookimplOpts | None:
-        # IMPORTANT: @property methods can have side effects, and are never hookimpl
-        # if attr is a property, skip it in advance
-        plugin_class = plugin if inspect.isclass(plugin) else type(plugin)
-        if isinstance(getattr(plugin_class, name, None), property):
-            return None
-
-        # pydantic model fields are like attrs and also can never be hookimpls
-        plugin_is_pydantic_obj = hasattr(plugin, "__pydantic_core_schema__")
-        if plugin_is_pydantic_obj and name in getattr(plugin, "model_fields", {}):
-            # pydantic models mess with the class and attr __signature__
-            # so inspect.isroutine(...) throws exceptions and cant be used
-            return None
-        
-        try:
-            return super().parse_hookimpl_opts(plugin, name)
-        except AttributeError:
-            return super().parse_hookimpl_opts(type(plugin), name)
-
-pm = PluginManager("abx")
@@ -1 +0,0 @@
-__package__ = 'abx.pydantic_pkgr'
@@ -1,13 +0,0 @@
-
-from ..hookspec import hookspec
-
-###########################################################################################
-
-@hookspec
-def get_BINPROVIDERS():
-    return {}
-
-@hookspec
-def get_BINARIES():
-    return {}
-
@@ -0,0 +1,571 @@
+__package__ = 'archivebox.actors'
+
+import os
+import time
+from typing import ClassVar, Generic, TypeVar, Any, Literal, Type, Iterable, cast, get_args
+from datetime import timedelta
+from multiprocessing import Process, cpu_count
+from threading import Thread, get_native_id
+
+import psutil
+from rich import print
+from statemachine import State, StateMachine, registry
+from statemachine.mixins import MachineMixin
+
+from django import db
+from django.db.models import QuerySet, sql, Q
+from django.db.models import Model as DjangoModel
+from django.utils import timezone
+from django.utils.functional import classproperty
+
+from .models import ModelWithStateMachine
+
+# from archivebox.logging_util import TimedProgress
+
+class ActorObjectAlreadyClaimed(Exception):
+    """Raised when the Actor tries to claim the next object from the queue but it's already been claimed by another Actor"""
+    pass
+
+class ActorQueueIsEmpty(Exception):
+    """Raised when the Actor tries to get the next object from the queue but it's empty"""
+    pass
+
+CPU_COUNT = cpu_count()
+DEFAULT_MAX_TICK_TIME = 60
+DEFAULT_MAX_CONCURRENT_ACTORS = min(max(2, int(CPU_COUNT * 0.6)), 8)   # 2 < 60% * num available cpu cores < 8
+
+limit = lambda n, max: min(n, max)
+
+LaunchKwargs = dict[str, Any]
+ObjectState = State | str
+ObjectStateList = Iterable[ObjectState]
+
+ModelType = TypeVar('ModelType', bound=ModelWithStateMachine)
+
+class ActorType(Generic[ModelType]):
+    """
+    Base class for all actors. Usage:
+    
+    class FaviconActor(ActorType[FaviconArchiveResult]):
+        FINAL_STATES: ClassVar[tuple[str, ...]] = ('succeeded', 'failed', 'skipped')
+        ACTIVE_STATE: ClassVar[str] = 'started'
+        
+        @classmethod
+        def qs(cls) -> QuerySet[FaviconArchiveResult]:
+            return ArchiveResult.objects.filter(extractor='favicon')   # or leave the default: FaviconArchiveResult.objects.all()
+    """
+    
+    ### Class attributes (defined on the class at compile-time when ActorType[MyModel] is defined)
+    Model: Type[ModelType]
+    StateMachineClass: Type[StateMachine]
+    
+    STATE_FIELD_NAME: ClassVar[str]
+    ACTIVE_STATE: ClassVar[ObjectState]
+    FINAL_STATES: ClassVar[ObjectStateList]
+    EVENT_NAME: ClassVar[str] = 'tick'                                    # the event name to trigger on the obj.sm: StateMachine (usually 'tick')
+    
+    CLAIM_ORDER: ClassVar[tuple[str, ...]] = ('retry_at',)                # the .order(*args) to claim the queue objects in, use ('?',) for random order
+    CLAIM_FROM_TOP_N: ClassVar[int] = CPU_COUNT * 10                      # the number of objects to consider when atomically getting the next object from the queue
+    CLAIM_ATOMIC: ClassVar[bool] = True                                   # whether to atomically fetch+claim the next object in one query, or fetch and lock it in two queries
+    
+    MAX_TICK_TIME: ClassVar[int] = DEFAULT_MAX_TICK_TIME                  # maximum duration in seconds to process a single object
+    MAX_CONCURRENT_ACTORS: ClassVar[int] = DEFAULT_MAX_CONCURRENT_ACTORS  # maximum number of concurrent actors that can be running at once
+    
+    _SPAWNED_ACTOR_PIDS: ClassVar[list[psutil.Process]] = []      # used to record all the pids of Actors spawned on the class
+    
+    ### Instance attributes (only used within an actor instance inside a spawned actor thread/process)
+    pid: int
+    idle_count: int = 0
+    launch_kwargs: LaunchKwargs = {}
+    mode: Literal['thread', 'process'] = 'process'
+    
+    def __init_subclass__(cls) -> None:
+        """
+        Executed at class definition time (i.e. during import of any file containing class MyActor(ActorType[MyModel]): ...).
+        Loads the django Model from the Generic[ModelType] TypeVar arg and populates any missing class-level config using it.
+        """
+        if getattr(cls, 'Model', None) is None:
+            cls.Model = cls._get_model_from_generic_typevar()
+        cls._populate_missing_classvars_from_model(cls.Model)
+    
+    def __init__(self, mode: Literal['thread', 'process']|None=None, **launch_kwargs: LaunchKwargs):
+        """
+        Executed right before the Actor is spawned to create a unique Actor instance for that thread/process.
+        actor_instance.runloop() is then executed from inside the newly spawned thread/process.
+        """
+        self.mode = mode or self.mode
+        self.launch_kwargs = launch_kwargs or dict(self.launch_kwargs)
+    
+
+    ### Private Helper Methods: Not desiged to be overridden by subclasses or called by anything outside of this class
+    
+    @classproperty
+    def name(cls) -> str:
+        return cls.__name__  # type: ignore
+    
+    def __str__(self) -> str:
+        return repr(self)
+    
+    def __repr__(self) -> str:
+        """-> FaviconActor[pid=1234]"""
+        label = 'pid' if self.mode == 'process' else 'tid'
+        return f'[underline]{self.name}[/underline]\\[{label}={self.pid}]'
+    
+    @staticmethod
+    def _state_to_str(state: ObjectState) -> str:
+        """Convert a statemachine.State, models.TextChoices.choices value, or Enum value to a str"""
+        return str(state.value) if isinstance(state, State) else str(state)
+    
+    @staticmethod
+    def _sql_for_select_top_n_candidates(qs: QuerySet, claim_from_top_n: int=CLAIM_FROM_TOP_N) -> tuple[str, tuple[Any, ...]]:
+        """Get the SQL for selecting the top N candidates from the queue (to claim one from)"""
+        queryset = qs.only('id')[:claim_from_top_n]
+        select_sql, select_params = compile_sql_select(queryset)
+        return select_sql, select_params
+    
+    @staticmethod
+    def _sql_for_update_claimed_obj(qs: QuerySet, update_kwargs: dict[str, Any]) -> tuple[str, tuple[Any, ...]]:
+        """Get the SQL for updating a claimed object to mark it as ACTIVE"""
+        # qs.update(status='started', retry_at=<now + MAX_TICK_TIME>)
+        update_sql, update_params = compile_sql_update(qs, update_kwargs=update_kwargs)
+        # e.g. UPDATE core_archiveresult SET status='%s', retry_at='%s' WHERE status NOT IN ('succeeded', 'failed', 'sealed', 'started') AND retry_at <= '2024-11-04 10:14:33.240903'
+        return update_sql, update_params
+    
+    @classmethod
+    def _get_model_from_generic_typevar(cls) -> Type[ModelType]:
+        """Get the django Model from the Generic[ModelType] TypeVar arg (and check that it inherits from django.db.models.Model)"""
+        # cls.__orig_bases__ is non-standard and may be removed in the future! if this breaks,
+        # we can just require the inerited class to define the Model as a classvar manually, e.g.:
+        #     class SnapshotActor(ActorType[Snapshot]):
+        #         Model: ClassVar[Type[Snapshot]] = Snapshot
+        # https://stackoverflow.com/questions/57706180/generict-base-class-how-to-get-type-of-t-from-within-instance
+        Model = get_args(cls.__orig_bases__[0])[0]   # type: ignore
+        assert issubclass(Model, DjangoModel), f'{cls.__name__}.Model must be a valid django Model'
+        return cast(Type[ModelType], Model)
+    
+    @staticmethod
+    def _get_state_machine_cls(Model: Type[ModelType]) -> Type[StateMachine]:
+        """Get the StateMachine class for the given django Model that inherits from MachineMixin"""
+        assert issubclass(Model, MachineMixin), f'{Model.__name__} must inherit from MachineMixin and define a .state_machine_name: str'
+        model_state_machine_name = getattr(Model, 'state_machine_name', None)
+        if model_state_machine_name:
+            StateMachine = registry.get_machine_cls(model_state_machine_name)
+            assert issubclass(StateMachine, StateMachine)
+            return StateMachine
+        raise NotImplementedError(f'ActorType[{Model.__name__}] must define .state_machine_name: str that points to a valid StateMachine')
+    
+    @classmethod
+    def _get_state_machine_instance(cls, obj: ModelType) -> StateMachine:
+        """Get the StateMachine instance for the given django Model instance (and check that it is a valid instance of cls.StateMachineClass)"""
+        obj_statemachine = None
+        state_machine_attr = getattr(obj, 'state_machine_attr', 'sm')
+        try:
+            obj_statemachine = getattr(obj, state_machine_attr)
+        except Exception:
+            pass
+        
+        if not isinstance(obj_statemachine, cls.StateMachineClass):
+            raise Exception(f'{cls.__name__}: Failed to find a valid StateMachine instance at {type(obj).__name__}.{state_machine_attr}')
+            
+        return obj_statemachine
+    
+    @classmethod
+    def _populate_missing_classvars_from_model(cls, Model: Type[ModelType]):
+        """Check that the class variables are set correctly based on the ModelType"""
+        
+        # check that Model is the same as the Generic[ModelType] parameter in the class definition
+        cls.Model = getattr(cls, 'Model', None) or Model
+        if cls.Model != Model:
+            raise ValueError(f'{cls.__name__}.Model must be set to the same Model as the Generic[ModelType] parameter in the class definition')
+        
+        # check that Model has a valid StateMachine with the required event defined on it
+        cls.StateMachineClass = getattr(cls, 'StateMachineClass', None) or cls._get_state_machine_cls(cls.Model)
+        assert isinstance(cls.EVENT_NAME, str), f'{cls.__name__}.EVENT_NAME must be a str, got: {type(cls.EVENT_NAME).__name__} instead'
+        assert hasattr(cls.StateMachineClass, cls.EVENT_NAME), f'StateMachine {cls.StateMachineClass.__name__} must define a {cls.EVENT_NAME} event ({cls.__name__}.EVENT_NAME = {cls.EVENT_NAME})'
+        
+        # check that Model uses .id as its primary key field
+        primary_key_field = cls.Model._meta.pk.name
+        if primary_key_field != 'id':
+            raise NotImplementedError(f'Actors currently only support models that use .id as their primary key field ({cls.__name__} uses {cls.__name__}.{primary_key_field} as primary key)')
+        
+        # check for STATE_FIELD_NAME classvar or set it from the model's state_field_name attr
+        if not getattr(cls, 'STATE_FIELD_NAME', None):
+            if hasattr(cls.Model, 'state_field_name'):
+                cls.STATE_FIELD_NAME = getattr(cls.Model, 'state_field_name')
+            else:
+                raise NotImplementedError(f'{cls.__name__} must define a STATE_FIELD_NAME: ClassVar[str] (e.g. "status") or have a .state_field_name attr on its Model')
+        assert isinstance(cls.STATE_FIELD_NAME, str), f'{cls.__name__}.STATE_FIELD_NAME must be a str, got: {type(cls.STATE_FIELD_NAME).__name__} instead'
+            
+        # check for FINAL_STATES classvar or set it from the model's final_states attr
+        if not getattr(cls, 'FINAL_STATES', None):
+            cls.FINAL_STATES = cls.StateMachineClass.final_states
+            if not cls.FINAL_STATES:
+                raise NotImplementedError(f'{cls.__name__} must define a non-empty FINAL_STATES: ClassVar[list[str]] (e.g. ["sealed"]) or have a {cls.Model.__name__}.state_machine_name pointing to a StateMachine that provides .final_states')
+        cls.FINAL_STATES = [cls._state_to_str(state) for state in cls.FINAL_STATES]
+        assert all(isinstance(state, str) for state in cls.FINAL_STATES), f'{cls.__name__}.FINAL_STATES must be a list[str], got: {type(cls.FINAL_STATES).__name__} instead'
+        
+        # check for ACTIVE_STATE classvar or set it from the model's active_state attr
+        if not getattr(cls, 'ACTIVE_STATE', None):
+            raise NotImplementedError(f'{cls.__name__} must define an ACTIVE_STATE: ClassVar[State] (e.g. SnapshotMachine.started) ({cls.Model.__name__}.{cls.STATE_FIELD_NAME} gets set to this value to mark objects as actively processing)')
+        assert isinstance(cls.ACTIVE_STATE, (State, str)), f'{cls.__name__}.ACTIVE_STATE must be a statemachine.State | str, got: {type(cls.ACTIVE_STATE).__name__} instead'
+        
+        # check the other ClassVar attributes for valid values
+        assert cls.CLAIM_ORDER and isinstance(cls.CLAIM_ORDER, tuple) and all(isinstance(order, str) for order in cls.CLAIM_ORDER), f'{cls.__name__}.CLAIM_ORDER must be a non-empty tuple[str, ...], got: {type(cls.CLAIM_ORDER).__name__} instead'
+        assert cls.CLAIM_FROM_TOP_N > 0, f'{cls.__name__}.CLAIM_FROM_TOP_N must be a positive int, got: {cls.CLAIM_FROM_TOP_N} instead'
+        assert cls.MAX_TICK_TIME >= 1, f'{cls.__name__}.MAX_TICK_TIME must be a positive int > 1, got: {cls.MAX_TICK_TIME} instead'
+        assert cls.MAX_CONCURRENT_ACTORS >= 1, f'{cls.__name__}.MAX_CONCURRENT_ACTORS must be a positive int >=1, got: {cls.MAX_CONCURRENT_ACTORS} instead'
+        assert isinstance(cls.CLAIM_ATOMIC, bool), f'{cls.__name__}.CLAIM_ATOMIC must be a bool, got: {cls.CLAIM_ATOMIC} instead'
+
+    @classmethod
+    def _fork_actor_as_thread(cls, **launch_kwargs: LaunchKwargs) -> int:
+        """Spawn a new background thread running the actor's runloop"""
+        actor = cls(mode='thread', **launch_kwargs)
+        bg_actor_thread = Thread(target=actor.runloop)
+        bg_actor_thread.start()
+        assert bg_actor_thread.native_id is not None
+        return bg_actor_thread.native_id
+    
+    @classmethod
+    def _fork_actor_as_process(cls, **launch_kwargs: LaunchKwargs) -> int:
+        """Spawn a new background process running the actor's runloop"""
+        actor = cls(mode='process', **launch_kwargs)
+        bg_actor_process = Process(target=actor.runloop)
+        bg_actor_process.start()
+        assert bg_actor_process.pid is not None
+        cls._SPAWNED_ACTOR_PIDS.append(psutil.Process(pid=bg_actor_process.pid))
+        return bg_actor_process.pid
+    
+    
+    ### Class Methods: Called by Orchestrator on ActorType class before it has been spawned
+    
+    @classmethod
+    def get_running_actors(cls) -> list[int]:
+        """returns a list of pids of all running actors of this type"""
+        # WARNING: only works for process actors, not thread actors
+        if cls.mode == 'thread':
+            raise NotImplementedError('get_running_actors() is not implemented for thread actors')
+        return [
+            proc.pid for proc in cls._SPAWNED_ACTOR_PIDS
+            if proc.is_running() and proc.status() != 'zombie'
+        ]
+        
+    @classmethod
+    def get_actors_to_spawn(cls, queue: QuerySet, running_actors: list[int]) -> list[LaunchKwargs]:
+        """Get a list of launch kwargs for the number of actors to spawn based on the queue and currently running actors"""
+        queue_length = queue.count()
+        if not queue_length:                                      # queue is empty, spawn 0 actors
+            return []
+        
+        # WARNING:
+        # spawning new actors processes is slow/expensive, avoid spawning many actors at once in a single orchestrator tick.
+        # limit to spawning 1 or 2 at a time per orchestrator tick, and let the next tick handle starting another couple.
+        # DONT DO THIS:
+        # if queue_length > 20:                      # queue is extremely long, spawn maximum actors at once!
+        #   num_to_spawn_this_tick = cls.MAX_CONCURRENT_ACTORS
+        
+        if queue_length > 10:    
+            num_to_spawn_this_tick = 2  # spawn more actors per tick if queue is long
+        else:
+            num_to_spawn_this_tick = 1  # spawn fewer actors per tick if queue is short
+        
+        num_remaining = cls.MAX_CONCURRENT_ACTORS - len(running_actors)
+        num_to_spawn_now: int = limit(num_to_spawn_this_tick, num_remaining)
+        
+        actors_launch_kwargs: list[LaunchKwargs] = num_to_spawn_now * [{**cls.launch_kwargs}]
+        return actors_launch_kwargs
+        
+    @classmethod
+    def start(cls, mode: Literal['thread', 'process']='process', **launch_kwargs: LaunchKwargs) -> int:
+        if mode == 'thread':
+            return cls._fork_actor_as_thread(**launch_kwargs)
+        elif mode == 'process':
+            return cls._fork_actor_as_process(**launch_kwargs)
+        raise ValueError(f'Invalid actor mode: {mode} must be "thread" or "process"')
+    
+    @classproperty
+    def qs(cls) -> QuerySet[ModelType]:
+        """
+        Get the unfiltered and unsorted QuerySet of all objects that this Actor might care about.
+        Override this in the subclass to define the QuerySet of objects that the Actor is going to poll for new work.
+        (don't limit, order, or filter this by retry_at or status yet, Actor.get_queue() handles that part)
+        """
+        return cls.Model.objects.all()
+    
+    @classproperty
+    def final_q(cls) -> Q:
+        """Get the filter for objects that are in a final state"""
+        return Q(**{f'{cls.STATE_FIELD_NAME}__in': [cls._state_to_str(s) for s in cls.FINAL_STATES]})
+    
+    @classproperty
+    def active_q(cls) -> Q:
+        """Get the filter for objects that are actively processing right now"""
+        return Q(**{cls.STATE_FIELD_NAME: cls._state_to_str(cls.ACTIVE_STATE)})   # e.g. Q(status='started')
+    
+    @classproperty
+    def stalled_q(cls) -> Q:
+        """Get the filter for objects that are marked active but have timed out"""
+        return cls.active_q & Q(retry_at__lte=timezone.now())                     # e.g. Q(status='started') AND Q(<retry_at is in the past>)
+    
+    @classproperty
+    def future_q(cls) -> Q:
+        """Get the filter for objects that have a retry_at in the future"""
+        return Q(retry_at__gt=timezone.now())
+    
+    @classproperty
+    def pending_q(cls) -> Q:
+        """Get the filter for objects that are ready for processing."""
+        return ~(cls.active_q) & ~(cls.final_q) & ~(cls.future_q)
+    
+    @classmethod
+    def get_queue(cls, sort: bool=True) -> QuerySet[ModelType]:
+        """
+        Get the sorted and filtered QuerySet of objects that are ready for processing.
+        e.g. qs.exclude(status__in=('sealed', 'started'), retry_at__gt=timezone.now()).order_by('retry_at')
+        """
+        unsorted_qs = cls.qs.filter(cls.pending_q)
+        return unsorted_qs.order_by(*cls.CLAIM_ORDER) if sort else unsorted_qs
+
+    ### Instance Methods: Only called from within Actor instance after it has been spawned (i.e. forked as a thread or process)
+    
+    def runloop(self):
+        """The main runloop that starts running when the actor is spawned (as subprocess or thread) and exits when the queue is empty"""
+        self.on_startup()
+        obj_to_process: ModelType | None = None
+        last_error: BaseException | None = None
+        try:
+            while True:
+                # Get the next object to process from the queue
+                try:
+                    obj_to_process = cast(ModelType, self.get_next(atomic=self.atomic))
+                except (ActorQueueIsEmpty, ActorObjectAlreadyClaimed) as err:
+                    last_error = err
+                    obj_to_process = None
+                
+                # Handle the case where there is no next object to process
+                if obj_to_process:
+                    self.idle_count = 0   # reset idle count if we got an object
+                else:
+                    if self.idle_count >= 30:
+                        break             # stop looping and exit if queue is empty and we have idled for 30sec
+                    else:
+                        # print('Actor runloop()', f'pid={self.pid}', 'queue empty, rechecking...')
+                        self.idle_count += 1
+                        time.sleep(1)
+                        continue
+                
+                # Process the object by triggering its StateMachine.tick() method
+                self.on_tick_start(obj_to_process)
+                try:
+                    self.tick(obj_to_process)
+                except Exception as err:
+                    last_error = err
+                    # print(f'[red]🏃‍♂️ {self}.tick()[/red] {obj_to_process} ERROR: [red]{type(err).__name__}: {err}[/red]')
+                    db.connections.close_all()                         # always reset the db connection after an exception to clear any pending transactions
+                    self.on_tick_exception(obj_to_process, err)
+                finally:
+                    self.on_tick_end(obj_to_process)
+
+        except BaseException as err:
+            last_error = err
+            if isinstance(err, KeyboardInterrupt):
+                print()
+            else:
+                print(f'\n[red]🏃‍♂️ {self}.runloop() FATAL:[/red] {type(err).__name__}: {err}')
+                print(f'    Last processed object: {obj_to_process}')
+                raise
+        finally:
+            self.on_shutdown(last_obj=obj_to_process, last_error=last_error)
+    
+    def get_update_kwargs_to_claim_obj(self) -> dict[str, Any]:
+        """
+        Get the field values needed to mark an pending obj_to_process as being actively processing (aka claimed)
+        by the current Actor. returned kwargs will be applied using: qs.filter(id=obj_to_process.id).update(**kwargs).
+        F() expressions are allowed in field values if you need to update a field based on its current value.
+        Can be a defined as a normal method (instead of classmethod) on subclasses if it needs to access instance vars.
+        """
+        return {
+            self.STATE_FIELD_NAME: self.ACTIVE_STATE,
+            'retry_at': timezone.now() + timedelta(seconds=self.MAX_TICK_TIME),
+        }
+    
+    def get_next(self, atomic: bool | None=None) -> ModelType | None:
+        """get the next object from the queue, atomically locking it if self.atomic=True"""
+        atomic = self.CLAIM_ATOMIC if atomic is None else atomic
+        if atomic:
+            # fetch and claim the next object from in the queue in one go atomically
+            obj = self.get_next_atomic()
+        else:
+            # two-step claim: fetch the next object and lock it in a separate query
+            obj = self.get_next_non_atomic()
+        return obj
+    
+    def get_next_non_atomic(self) -> ModelType:
+        """
+        Naiively selects the top/first object from self.get_queue().order_by(*self.CLAIM_ORDER),
+        then claims it by running .update(status='started', retry_at=<now + MAX_TICK_TIME>).
+        
+        Do not use this method if there is more than one Actor racing to get objects from the same queue,
+        it will be slow/buggy as they'll compete to lock the same object at the same time (TOCTTOU race).
+        """
+        obj = self.get_queue().first()
+        if obj is None:
+            raise ActorQueueIsEmpty(f'No next object available in {self}.get_queue()')
+        
+        locked = self.get_queue().filter(id=obj.id).update(**self.get_update_kwargs_to_claim_obj())
+        if not locked:
+            raise ActorObjectAlreadyClaimed(f'Unable to lock the next {self.Model.__name__} object from {self}.get_queue().first()')
+        return obj
+        
+    def get_next_atomic(self) -> ModelType | None:
+        """
+        Selects the top n=50 objects from the queue and atomically claims a random one from that set.
+        This approach safely minimizes contention with other Actors trying to select from the same Queue.
+
+        The atomic query is roughly equivalent to the following:  (all done in one SQL query to avoid a TOCTTOU race)
+            top_candidates are selected from:   qs.order_by(*CLAIM_ORDER).only('id')[:CLAIM_FROM_TOP_N]
+            a single candidate is chosen using: qs.filter(id__in=top_n_candidates).order_by('?').first()
+            the chosen obj is claimed using:    qs.filter(id=chosen_obj).update(status=ACTIVE_STATE, retry_at=<now + MAX_TICK_TIME>)
+        """
+        # TODO: if we switch from SQLite to PostgreSQL in the future, we should change this
+        # to use SELECT FOR UPDATE instead of a subquery + ORDER BY RANDOM() LIMIT 1
+        
+        # e.g. SELECT id FROM core_archiveresult WHERE status NOT IN (...) AND retry_at <= '...' ORDER BY retry_at ASC LIMIT 50
+        qs = self.get_queue()
+        select_top_canidates_sql, select_params = self._sql_for_select_top_n_candidates(qs=qs)
+        assert select_top_canidates_sql.startswith('SELECT ')
+        
+        # e.g. UPDATE core_archiveresult SET status='%s', retry_at='%s' WHERE status NOT IN (...) AND retry_at <= '...'
+        update_claimed_obj_sql, update_params = self._sql_for_update_claimed_obj(qs=qs, update_kwargs=self.get_update_kwargs_to_claim_obj())
+        assert update_claimed_obj_sql.startswith('UPDATE ')
+        db_table = self.Model._meta.db_table  # e.g. core_archiveresult
+        
+        # subquery gets the pool of the top candidates e.g. self.get_queue().only('id')[:CLAIM_FROM_TOP_N]
+        # main query selects a random one from that pool, and claims it using .update(status=ACTIVE_STATE, retry_at=<now + MAX_TICK_TIME>)
+        # this is all done in one atomic SQL query to avoid TOCTTOU race conditions (as much as possible)
+        atomic_select_and_update_sql = f"""
+            {update_claimed_obj_sql} AND "{db_table}"."id" = (
+                SELECT "{db_table}"."id" FROM (
+                    {select_top_canidates_sql}
+                ) candidates
+                ORDER BY RANDOM()
+                LIMIT 1
+            )
+            RETURNING *;
+        """
+        try:
+            return self.Model.objects.raw(atomic_select_and_update_sql, (*update_params, *select_params))[0]
+        except KeyError:
+            if self.get_queue().exists():
+                raise ActorObjectAlreadyClaimed(f'Unable to lock the next {self.Model.__name__} object from {self}.get_queue().first()')
+            else:
+                raise ActorQueueIsEmpty(f'No next object available in {self}.get_queue()')
+
+    def tick(self, obj_to_process: ModelType) -> None:
+        """Call the object.sm.tick() method to process the object"""
+        print(f'[blue]🏃‍♂️ {self}.tick()[/blue] {obj_to_process}')
+        
+        # get the StateMachine instance from the object
+        obj_statemachine = self._get_state_machine_instance(obj_to_process)
+        
+        # trigger the event on the StateMachine instance
+        obj_tick_method = getattr(obj_statemachine, self.EVENT_NAME)  # e.g. obj_statemachine.tick()
+        obj_tick_method()
+        
+        # save the object to persist any state changes
+        obj_to_process.save()
+        
+    def on_startup(self) -> None:
+        if self.mode == 'thread':
+            self.pid = get_native_id()  # thread id
+            print(f'[green]🏃‍♂️ {self}.on_startup() STARTUP (THREAD)[/green]')
+        else:
+            self.pid = os.getpid()      # process id
+            print(f'[green]🏃‍♂️ {self}.on_startup() STARTUP (PROCESS)[/green]')
+        # abx.pm.hook.on_actor_startup(actor=self)
+        
+    def on_shutdown(self, last_obj: ModelType | None=None, last_error: BaseException | None=None) -> None:
+        if isinstance(last_error, KeyboardInterrupt) or last_error is None:
+            last_error_str = '[green](CTRL-C)[/green]'
+        elif isinstance(last_error, ActorQueueIsEmpty):
+            last_error_str = '[green](queue empty)[/green]'
+        elif isinstance(last_error, ActorObjectAlreadyClaimed):
+            last_error_str = '[green](queue race)[/green]'
+        else:
+            last_error_str = f'[red]{type(last_error).__name__}: {last_error}[/red]'
+
+        print(f'[grey53]🏃‍♂️ {self}.on_shutdown() SHUTTING DOWN[/grey53] {last_error_str}')
+        # abx.pm.hook.on_actor_shutdown(actor=self, last_obj=last_obj, last_error=last_error)
+        
+    def on_tick_start(self, obj_to_process: ModelType) -> None:
+        print(f'🏃‍♂️ {self}.on_tick_start() {obj_to_process}')
+        # abx.pm.hook.on_actor_tick_start(actor=self, obj_to_process=obj)
+        # self.timer = TimedProgress(self.MAX_TICK_TIME, prefix='      ')
+    
+    def on_tick_end(self, obj_to_process: ModelType) -> None:
+        print(f'🏃‍♂️ {self}.on_tick_end() {obj_to_process}')
+        # abx.pm.hook.on_actor_tick_end(actor=self, obj_to_process=obj_to_process)
+        # self.timer.end()
+    
+    def on_tick_exception(self, obj_to_process: ModelType, error: Exception) -> None:
+        print(f'[red]🏃‍♂️ {self}.on_tick_exception()[/red] {obj_to_process}: [red]{type(error).__name__}: {error}[/red]')
+        # abx.pm.hook.on_actor_tick_exception(actor=self, obj_to_process=obj_to_process, error=error)
+
+
+
+def compile_sql_select(queryset: QuerySet, filter_kwargs: dict[str, Any] | None=None, order_args: tuple[str, ...]=(), limit: int | None=None) -> tuple[str, tuple[Any, ...]]:
+    """
+    Compute the SELECT query SQL for a queryset.filter(**filter_kwargs).order_by(*order_args)[:limit] call
+    Returns a tuple of (sql, params) where sql is a template string containing %s (unquoted) placeholders for the params
+    
+    WARNING:
+    final_sql = sql % params  DOES NOT WORK to assemble the final SQL string because the %s placeholders are not quoted/escaped
+    they should always passed separately to the DB driver so it can do its own quoting/escaping to avoid SQL injection and syntax errors
+    """
+    assert isinstance(queryset, QuerySet), f'compile_sql_select(...) first argument must be a QuerySet, got: {type(queryset).__name__} instead'
+    assert filter_kwargs is None or isinstance(filter_kwargs, dict), f'compile_sql_select(...) filter_kwargs argument must be a dict[str, Any], got: {type(filter_kwargs).__name__} instead'
+    assert isinstance(order_args, tuple) and all(isinstance(arg, str) for arg in order_args), f'compile_sql_select(...) order_args argument must be a tuple[str, ...] got: {type(order_args).__name__} instead'
+    assert limit is None or isinstance(limit, int), f'compile_sql_select(...) limit argument must be an int, got: {type(limit).__name__} instead'
+    
+    queryset = queryset._chain()                      # type: ignore   # copy queryset to avoid modifying the original
+    if filter_kwargs:
+        queryset = queryset.filter(**filter_kwargs)
+    if order_args:
+        queryset = queryset.order_by(*order_args)
+    if limit is not None:
+        queryset = queryset[:limit]
+    query = queryset.query
+    
+    # e.g. SELECT id FROM core_archiveresult WHERE status NOT IN (%s, %s, %s) AND retry_at <= %s ORDER BY retry_at ASC LIMIT 50
+    select_sql, select_params = query.get_compiler(queryset.db).as_sql()
+    return select_sql, select_params
+
+
+def compile_sql_update(queryset: QuerySet, update_kwargs: dict[str, Any], filter_kwargs: dict[str, Any] | None=None) -> tuple[str, tuple[Any, ...]]:
+    """
+    Compute the UPDATE query SQL for a queryset.filter(**filter_kwargs).update(**update_kwargs) call
+    Returns a tuple of (sql, params) where sql is a template string containing %s (unquoted) placeholders for the params
+    
+    Based on the django.db.models.QuerySet.update() source code, but modified to return the SQL instead of executing the update
+    https://github.com/django/django/blob/611bf6c2e2a1b4ab93273980c45150c099ab146d/django/db/models/query.py#L1217
+    
+    WARNING:
+    final_sql = sql % params  DOES NOT WORK to assemble the final SQL string because the %s placeholders are not quoted/escaped
+    they should always passed separately to the DB driver so it can do its own quoting/escaping to avoid SQL injection and syntax errors
+    """
+    assert isinstance(queryset, QuerySet), f'compile_sql_update(...) first argument must be a QuerySet, got: {type(queryset).__name__} instead'
+    assert isinstance(update_kwargs, dict), f'compile_sql_update(...) update_kwargs argument must be a dict[str, Any], got: {type(update_kwargs).__name__} instead'
+    assert filter_kwargs is None or isinstance(filter_kwargs, dict), f'compile_sql_update(...) filter_kwargs argument must be a dict[str, Any], got: {type(filter_kwargs).__name__} instead'
+    
+    queryset = queryset._chain()                      # type: ignore   # copy queryset to avoid modifying the original
+    if filter_kwargs:
+        queryset = queryset.filter(**filter_kwargs)
+    queryset.query.clear_ordering(force=True)                          # clear any ORDER BY clauses
+    queryset.query.clear_limits()                                      # clear any LIMIT clauses aka slices[:n]
+    queryset._for_write = True                        # type: ignore
+    query = queryset.query.chain(sql.UpdateQuery)     # type: ignore
+    query.add_update_values(update_kwargs)            # type: ignore
+    query.annotations = {}                                             # clear any annotations
+    
+    # e.g. UPDATE core_archiveresult SET status='%s', retry_at='%s' WHERE status NOT IN (%s, %s, %s) AND retry_at <= %s
+    update_sql, update_params = query.get_compiler(queryset.db).as_sql()
+    return update_sql, update_params
@@ -0,0 +1,3 @@
+from django.contrib import admin
+
+# Register your models here.
@@ -0,0 +1,6 @@
+from django.apps import AppConfig
+
+
+class ActorsConfig(AppConfig):
+    default_auto_field = "django.db.models.BigAutoField"
+    name = "actors"
@@ -0,0 +1,300 @@
+from typing import ClassVar, Type, Iterable
+from datetime import datetime, timedelta
+
+from statemachine.mixins import MachineMixin
+
+from django.db import models
+from django.utils import timezone
+from django.utils.functional import classproperty
+
+from statemachine import registry, StateMachine, State
+
+from django.core import checks
+
+class DefaultStatusChoices(models.TextChoices):
+    QUEUED = 'queued', 'Queued'
+    STARTED = 'started', 'Started'
+    SEALED = 'sealed', 'Sealed'
+
+
+default_status_field: models.CharField = models.CharField(choices=DefaultStatusChoices.choices, max_length=15, default=DefaultStatusChoices.QUEUED, null=False, blank=False, db_index=True)
+default_retry_at_field: models.DateTimeField = models.DateTimeField(default=timezone.now, null=False, db_index=True)
+
+ObjectState = State | str
+ObjectStateList = Iterable[ObjectState]
+
+
+class BaseModelWithStateMachine(models.Model, MachineMixin):
+    id: models.UUIDField
+    
+    StatusChoices: ClassVar[Type[models.TextChoices]]
+    
+    # status: models.CharField
+    # retry_at: models.DateTimeField
+    
+    state_machine_name: ClassVar[str]
+    state_field_name: ClassVar[str]
+    state_machine_attr: ClassVar[str] = 'sm'
+    bind_events_as_methods: ClassVar[bool] = True
+    
+    active_state: ClassVar[ObjectState]
+    retry_at_field_name: ClassVar[str]
+    
+    class Meta:
+        abstract = True
+        
+    @classmethod
+    def check(cls, sender=None, **kwargs):
+        errors = super().check(**kwargs)
+        
+        found_id_field = False
+        found_status_field = False
+        found_retry_at_field = False
+        
+        for field in cls._meta.get_fields():
+            if getattr(field, '_is_state_field', False):
+                if cls.state_field_name == field.name:
+                    found_status_field = True
+                    if getattr(field, 'choices', None) != cls.StatusChoices.choices:
+                        errors.append(checks.Error(
+                            f'{cls.__name__}.{field.name} must have choices set to {cls.__name__}.StatusChoices.choices',
+                            hint=f'{cls.__name__}.{field.name}.choices = {getattr(field, "choices", None)!r}',
+                            obj=cls,
+                            id='actors.E011',
+                        ))
+            if getattr(field, '_is_retry_at_field', False):
+                if cls.retry_at_field_name == field.name:
+                    found_retry_at_field = True
+            if field.name == 'id' and getattr(field, 'primary_key', False):
+                found_id_field = True
+                    
+        if not found_status_field:
+            errors.append(checks.Error(
+                f'{cls.__name__}.state_field_name must be defined and point to a StatusField()',
+                hint=f'{cls.__name__}.state_field_name = {cls.state_field_name!r} but {cls.__name__}.{cls.state_field_name!r} was not found or does not refer to StatusField',
+                obj=cls,
+                id='actors.E012',
+            ))
+        if not found_retry_at_field:
+            errors.append(checks.Error(
+                f'{cls.__name__}.retry_at_field_name must be defined and point to a RetryAtField()',
+                hint=f'{cls.__name__}.retry_at_field_name = {cls.retry_at_field_name!r} but {cls.__name__}.{cls.retry_at_field_name!r} was not found or does not refer to RetryAtField',
+                obj=cls,
+                id='actors.E013',
+            ))
+            
+        if not found_id_field:
+            errors.append(checks.Error(
+                f'{cls.__name__} must have an id field that is a primary key',
+                hint=f'{cls.__name__}.id = {cls.id!r}',
+                obj=cls,
+                id='actors.E014',
+            ))
+            
+        if not isinstance(cls.state_machine_name, str):
+            errors.append(checks.Error(
+                f'{cls.__name__}.state_machine_name must be a dotted-import path to a StateMachine class',
+                hint=f'{cls.__name__}.state_machine_name = {cls.state_machine_name!r}',
+                obj=cls,
+                id='actors.E015',
+            ))
+        
+        try:
+            cls.StateMachineClass
+        except Exception as err:
+            errors.append(checks.Error(
+                f'{cls.__name__}.state_machine_name must point to a valid StateMachine class, but got {type(err).__name__} {err} when trying to access {cls.__name__}.StateMachineClass',
+                hint=f'{cls.__name__}.state_machine_name = {cls.state_machine_name!r}',
+                obj=cls,
+                id='actors.E016',
+            ))
+        
+        if cls.INITIAL_STATE not in cls.StatusChoices.values:
+            errors.append(checks.Error(
+                f'{cls.__name__}.StateMachineClass.initial_state must be present within {cls.__name__}.StatusChoices',
+                hint=f'{cls.__name__}.StateMachineClass.initial_state = {cls.StateMachineClass.initial_state!r}',
+                obj=cls,
+                id='actors.E017',
+            ))
+            
+        if cls.ACTIVE_STATE not in cls.StatusChoices.values:
+            errors.append(checks.Error(
+                f'{cls.__name__}.active_state must be set to a valid State present within {cls.__name__}.StatusChoices',
+                hint=f'{cls.__name__}.active_state = {cls.active_state!r}',
+                obj=cls,
+                id='actors.E018',
+            ))
+            
+        
+        for state in cls.FINAL_STATES:
+            if state not in cls.StatusChoices.values:
+                errors.append(checks.Error(
+                    f'{cls.__name__}.StateMachineClass.final_states must all be present within {cls.__name__}.StatusChoices',
+                    hint=f'{cls.__name__}.StateMachineClass.final_states = {cls.StateMachineClass.final_states!r}',
+                    obj=cls,
+                    id='actors.E019',
+                ))
+                break
+        return errors
+    
+    @staticmethod
+    def _state_to_str(state: ObjectState) -> str:
+        """Convert a statemachine.State, models.TextChoices.choices value, or Enum value to a str"""
+        return str(state.value) if isinstance(state, State) else str(state)
+    
+    
+    @property
+    def RETRY_AT(self) -> datetime:
+        return getattr(self, self.retry_at_field_name)
+    
+    @RETRY_AT.setter
+    def RETRY_AT(self, value: datetime):
+        setattr(self, self.retry_at_field_name, value)
+        
+    @property
+    def STATE(self) -> str:
+        return getattr(self, self.state_field_name)
+    
+    @STATE.setter
+    def STATE(self, value: str):
+        setattr(self, self.state_field_name, value)
+        
+    def bump_retry_at(self, seconds: int = 10):
+        self.RETRY_AT = timezone.now() + timedelta(seconds=seconds)
+        
+    @classproperty
+    def ACTIVE_STATE(cls) -> str:
+        return cls._state_to_str(cls.active_state)
+        
+    @classproperty
+    def INITIAL_STATE(cls) -> str:
+        return cls._state_to_str(cls.StateMachineClass.initial_state)
+    
+    @classproperty
+    def FINAL_STATES(cls) -> list[str]:
+        return [cls._state_to_str(state) for state in cls.StateMachineClass.final_states]
+    
+    @classproperty
+    def FINAL_OR_ACTIVE_STATES(cls) -> list[str]:
+        return [*cls.FINAL_STATES, cls.ACTIVE_STATE]
+        
+    @classmethod
+    def extend_choices(cls, base_choices: Type[models.TextChoices]):
+        """
+        Decorator to extend the base choices with extra choices, e.g.:
+        
+        class MyModel(ModelWithStateMachine):
+        
+            @ModelWithStateMachine.extend_choices(ModelWithStateMachine.StatusChoices)
+            class StatusChoices(models.TextChoices):
+                SUCCEEDED = 'succeeded'
+                FAILED = 'failed'
+                SKIPPED = 'skipped'
+        """
+        assert issubclass(base_choices, models.TextChoices), f'@extend_choices(base_choices) must be a TextChoices class, not {base_choices.__name__}'
+        def wrapper(extra_choices: Type[models.TextChoices]) -> Type[models.TextChoices]:
+            joined = {}
+            for item in base_choices.choices:
+                joined[item[0]] = item[1]
+            for item in extra_choices.choices:
+                joined[item[0]] = item[1]
+            return models.TextChoices('StatusChoices', joined)
+        return wrapper
+        
+    @classmethod
+    def StatusField(cls, **kwargs) -> models.CharField:
+        """
+        Used on subclasses to extend/modify the status field with updated kwargs. e.g.:
+        
+        class MyModel(ModelWithStateMachine):
+            class StatusChoices(ModelWithStateMachine.StatusChoices):
+                QUEUED = 'queued', 'Queued'
+                STARTED = 'started', 'Started'
+                SEALED = 'sealed', 'Sealed'
+                BACKOFF = 'backoff', 'Backoff'
+                FAILED = 'failed', 'Failed'
+                SKIPPED = 'skipped', 'Skipped'
+        
+            status = ModelWithStateMachine.StatusField(choices=StatusChoices.choices, default=StatusChoices.QUEUED)
+        """
+        default_kwargs = default_status_field.deconstruct()[3]
+        updated_kwargs = {**default_kwargs, **kwargs}
+        field = models.CharField(**updated_kwargs)
+        field._is_state_field = True                    # type: ignore
+        return field
+
+    @classmethod
+    def RetryAtField(cls, **kwargs) -> models.DateTimeField:
+        """
+        Used on subclasses to extend/modify the retry_at field with updated kwargs. e.g.:
+        
+        class MyModel(ModelWithStateMachine):
+            retry_at = ModelWithStateMachine.RetryAtField(editable=False)
+        """
+        default_kwargs = default_retry_at_field.deconstruct()[3]
+        updated_kwargs = {**default_kwargs, **kwargs}
+        field = models.DateTimeField(**updated_kwargs)
+        field._is_retry_at_field = True                 # type: ignore
+        return field
+    
+    @classproperty
+    def StateMachineClass(cls) -> Type[StateMachine]:
+        """Get the StateMachine class for the given django Model that inherits from MachineMixin"""
+
+        model_state_machine_name = getattr(cls, 'state_machine_name', None)
+        if model_state_machine_name:
+            StateMachineCls = registry.get_machine_cls(model_state_machine_name)
+            assert issubclass(StateMachineCls, StateMachine)
+            return StateMachineCls
+        raise NotImplementedError(f'ActorType[{cls.__name__}] must define .state_machine_name: str that points to a valid StateMachine')
+    
+    # @classproperty
+    # def final_q(cls) -> Q:
+    #     """Get the filter for objects that are in a final state"""
+    #     return Q(**{f'{cls.state_field_name}__in': cls.final_states})
+    
+    # @classproperty
+    # def active_q(cls) -> Q:
+    #     """Get the filter for objects that are actively processing right now"""
+    #     return Q(**{cls.state_field_name: cls._state_to_str(cls.active_state)})   # e.g. Q(status='started')
+    
+    # @classproperty
+    # def stalled_q(cls) -> Q:
+    #     """Get the filter for objects that are marked active but have timed out"""
+    #     return cls.active_q & Q(retry_at__lte=timezone.now())                     # e.g. Q(status='started') AND Q(<retry_at is in the past>)
+    
+    # @classproperty
+    # def future_q(cls) -> Q:
+    #     """Get the filter for objects that have a retry_at in the future"""
+    #     return Q(retry_at__gt=timezone.now())
+    
+    # @classproperty
+    # def pending_q(cls) -> Q:
+    #     """Get the filter for objects that are ready for processing."""
+    #     return ~(cls.active_q) & ~(cls.final_q) & ~(cls.future_q)
+    
+    # @classmethod
+    # def get_queue(cls) -> QuerySet:
+    #     """
+    #     Get the sorted and filtered QuerySet of objects that are ready for processing.
+    #     e.g. qs.exclude(status__in=('sealed', 'started'), retry_at__gt=timezone.now()).order_by('retry_at')
+    #     """
+    #     return cls.objects.filter(cls.pending_q)
+
+
+class ModelWithStateMachine(BaseModelWithStateMachine):
+    StatusChoices: ClassVar[Type[DefaultStatusChoices]] = DefaultStatusChoices
+    
+    status: models.CharField = BaseModelWithStateMachine.StatusField()
+    retry_at: models.DateTimeField = BaseModelWithStateMachine.RetryAtField()
+    
+    state_machine_name: ClassVar[str]      # e.g. 'core.statemachines.ArchiveResultMachine'
+    state_field_name: ClassVar[str]        = 'status'
+    state_machine_attr: ClassVar[str]      = 'sm'
+    bind_events_as_methods: ClassVar[bool] = True
+    
+    active_state: ClassVar[str]            = StatusChoices.STARTED
+    retry_at_field_name: ClassVar[str]     = 'retry_at'
+    
+    class Meta:
+        abstract = True
@@ -0,0 +1,244 @@
+__package__ = 'archivebox.actors'
+
+import os
+import time
+import itertools
+from typing import Dict, Type, Literal, ClassVar
+from django.utils.functional import classproperty
+
+from multiprocessing import Process
+from threading import Thread, get_native_id
+
+
+from rich import print
+
+from django.db.models import QuerySet
+
+from django.apps import apps
+from .actor import ActorType
+
+class Orchestrator:
+    pid: int
+    idle_count: int = 0
+    actor_types: Dict[str, Type[ActorType]]
+    mode: Literal['thread', 'process'] = 'process'
+
+    def __init__(self, actor_types: Dict[str, Type[ActorType]] | None = None, mode: Literal['thread', 'process'] | None=None):
+        self.actor_types = actor_types or self.actor_types or self.autodiscover_actor_types()
+        self.mode = mode or self.mode
+
+    def __repr__(self) -> str:
+        label = 'tid' if self.mode == 'thread' else 'pid'
+        return f'[underline]{self.name}[/underline]\\[{label}={self.pid}]'
+    
+    def __str__(self) -> str:
+        return self.__repr__()
+    
+    @classproperty
+    def name(cls) -> str:
+        return cls.__name__   # type: ignore
+    
+    def fork_as_thread(self):
+        self.thread = Thread(target=self.runloop)
+        self.thread.start()
+        assert self.thread.native_id is not None
+        return self.thread.native_id
+    
+    def fork_as_process(self):
+        self.process = Process(target=self.runloop)
+        self.process.start()
+        assert self.process.pid is not None
+        return self.process.pid
+
+    def start(self) -> int:
+        if self.mode == 'thread':
+            return self.fork_as_thread()
+        elif self.mode == 'process':
+            return self.fork_as_process()
+        raise ValueError(f'Invalid orchestrator mode: {self.mode}')
+    
+    @classmethod
+    def autodiscover_actor_types(cls) -> Dict[str, Type[ActorType]]:
+        # returns a Dict of all discovered {actor_type_id: ActorType} across the codebase
+        # override this method in a subclass to customize the actor types that are used
+        # return {'Snapshot': SnapshotActorType, 'ArchiveResult_chrome': ChromeActorType, ...}
+        return {
+            # look through all models and find all classes that inherit from ActorType
+            # actor_type.__name__: actor_type
+            # for actor_type in abx.pm.hook.get_all_ACTORS_TYPES().values()
+        }
+    
+    @classmethod
+    def get_orphaned_objects(cls, all_queues) -> list:
+        # returns a list of objects that are in the queues of all actor types but not in the queues of any other actor types
+        all_queued_ids = itertools.chain(*[queue.values('id', flat=True) for queue in all_queues.values()])
+        orphaned_objects = []
+        for model in apps.get_models():
+            if hasattr(model, 'retry_at'):
+                orphaned_objects.extend(model.objects.filter(retry_at__lt=timezone.now()).exclude(id__in=all_queued_ids))
+        return orphaned_objects
+    
+    def on_startup(self):
+        if self.mode == 'thread':
+            self.pid = get_native_id()
+            print(f'[green]👨‍✈️ {self}.on_startup() STARTUP (THREAD)[/green]')
+        elif self.mode == 'process':
+            self.pid = os.getpid()
+            print(f'[green]👨‍✈️ {self}.on_startup() STARTUP (PROCESS)[/green]')
+        # abx.pm.hook.on_orchestrator_startup(self)
+    
+    def on_shutdown(self, err: BaseException | None = None):
+        print(f'[grey53]👨‍✈️ {self}.on_shutdown() SHUTTING DOWN[/grey53]', err or '[green](gracefully)[/green]')
+        # abx.pm.hook.on_orchestrator_shutdown(self)
+        
+    def on_tick_started(self, all_queues):
+        # total_pending = sum(queue.count() for queue in all_queues.values())
+        # print(f'👨‍✈️ {self}.on_tick_started()', f'total_pending={total_pending}')
+        # abx.pm.hook.on_orchestrator_tick_started(self, actor_types, all_queues)
+        pass
+    
+    def on_tick_finished(self, all_queues, all_existing_actors, all_spawned_actors):
+        if all_spawned_actors:
+            total_queue_length = sum(queue.count() for queue in all_queues.values())
+            print(f'[grey53]👨‍✈️ {self}.on_tick_finished() queue={total_queue_length} existing_actors={len(all_existing_actors)} spawned_actors={len(all_spawned_actors)}[/grey53]')
+        # abx.pm.hook.on_orchestrator_tick_finished(self, actor_types, all_queues)
+
+    def on_idle(self, all_queues):
+        # print(f'👨‍✈️ {self}.on_idle()')
+        # abx.pm.hook.on_orchestrator_idle(self)
+        # check for orphaned objects left behind
+        if self.idle_count == 60:
+            orphaned_objects = self.get_orphaned_objects(all_queues)
+            if orphaned_objects:
+                print('[red]👨‍✈️ WARNING: some objects may not be processed, no actor has claimed them after 60s:[/red]', orphaned_objects)
+
+    def runloop(self):
+        self.on_startup()
+        try:
+            while True:
+                all_queues = {
+                    actor_type: actor_type.get_queue()
+                    for actor_type in self.actor_types.values()
+                }
+                if not all_queues:
+                    raise Exception('Failed to find any actor_types to process')
+
+                self.on_tick_started(all_queues)
+
+                all_existing_actors = []
+                all_spawned_actors = []
+
+                for actor_type, queue in all_queues.items():
+                    try:
+                        existing_actors = actor_type.get_running_actors()
+                        all_existing_actors.extend(existing_actors)
+                        actors_to_spawn = actor_type.get_actors_to_spawn(queue, existing_actors)
+                        for launch_kwargs in actors_to_spawn:
+                            new_actor_pid = actor_type.start(mode='process', **launch_kwargs)
+                            all_spawned_actors.append(new_actor_pid)
+                    except Exception as err:
+                        print(f'🏃‍♂️ ERROR: {self} Failed to get {actor_type} queue & running actors', err)
+                    except BaseException:
+                        raise
+
+                if not any(queue.exists() for queue in all_queues.values()):
+                    self.on_idle(all_queues)
+                    self.idle_count += 1
+                    time.sleep(1)
+                else:
+                    self.idle_count = 0
+                    
+                self.on_tick_finished(all_queues, all_existing_actors, all_spawned_actors)
+                time.sleep(1)
+
+        except BaseException as err:
+            if isinstance(err, KeyboardInterrupt):
+                print()
+            else:
+                print(f'\n[red]🏃‍♂️ {self}.runloop() FATAL:[/red]', err.__class__.__name__, err)
+            self.on_shutdown(err=err)
+
+
+
+from archivebox.config.django import setup_django
+
+setup_django()
+
+from core.models import ArchiveResult, Snapshot
+
+from django.utils import timezone
+
+from django import db
+from django.db import connection
+
+
+from crawls.actors import CrawlActor
+from .actor_snapshot import SnapshotActor
+
+from abx_plugin_singlefile.actors import SinglefileActor
+
+
+class FaviconActor(ActorType[ArchiveResult]):
+    CLAIM_ORDER: ClassVar[str] = 'created_at DESC'
+    CLAIM_WHERE: ClassVar[str] = 'status = "queued" AND extractor = "favicon"'
+    CLAIM_SET: ClassVar[str] = 'status = "started"'
+    
+    @classproperty
+    def QUERYSET(cls) -> QuerySet:
+        return ArchiveResult.objects.filter(status='failed', extractor='favicon')
+
+    def tick(self, obj: ArchiveResult):
+        print(f'[grey53]{self}.tick({obj.abid or obj.id}, status={obj.status}) remaining:[/grey53]', self.get_queue().count())
+        updated = ArchiveResult.objects.filter(id=obj.id, status='started').update(status='success') == 1
+        if not updated:
+            raise Exception(f'Failed to update {obj.abid or obj.id}, interrupted by another actor writing to the same object')
+        obj.refresh_from_db()
+        obj.save()
+
+
+class ExtractorsOrchestrator(Orchestrator):
+    actor_types = {
+        'CrawlActor': CrawlActor,
+        'SnapshotActor': SnapshotActor,
+        'FaviconActor': FaviconActor,
+        'SinglefileActor': SinglefileActor,
+    }
+
+
+if __name__ == '__main__':    
+    orchestrator = ExtractorsOrchestrator()
+    orchestrator.start()
+    
+    snap = Snapshot.objects.last()
+    assert snap is not None
+    created = 0
+    while True:
+        time.sleep(0.05)
+        # try:
+        #     ArchiveResult.objects.bulk_create([
+        #         ArchiveResult(
+        #             id=uuid.uuid4(),
+        #             snapshot=snap,
+        #             status='failed',
+        #             extractor='favicon',
+        #             cmd=['echo', '"hello"'],
+        #             cmd_version='1.0',
+        #             pwd='.',
+        #             start_ts=timezone.now(),
+        #             end_ts=timezone.now(),
+        #             created_at=timezone.now(),
+        #             modified_at=timezone.now(),
+        #             created_by_id=1,
+        #         )
+        #         for _ in range(100)
+        #     ])
+        #     created += 100
+        #     if created % 1000 == 0:
+        #         print(f'[blue]Created {created} ArchiveResults...[/blue]')
+        #         time.sleep(25)
+        # except Exception as err:
+        #     print(err)
+        #     db.connections.close_all()
+        # except BaseException as err:
+        #     print(err)
+        #     break
@@ -0,0 +1,3 @@
+from django.test import TestCase
+
+# Create your tests here.
@@ -0,0 +1,3 @@
+from django.shortcuts import render
+
+# Create your views here.
@@ -64,7 +64,7 @@ class NinjaAPIWithIOCapture(NinjaAPI):

        # Add Auth Headers to response
        api_token = getattr(request, '_api_token', None)
-        token_expiry = api_token.expires.isoformat() if api_token else 'Never'
+        token_expiry = api_token.expires.isoformat() if api_token and api_token.expires else 'Never'

        response['X-ArchiveBox-Auth-Method'] = getattr(request, '_api_auth_method', None) or 'None'
        response['X-ArchiveBox-Auth-Expires'] = token_expiry
@@ -1,4 +1,5 @@
-__package__ = 'archivebox.config'
+__package__ = 'config'
+__order__ = 200

 from .paths import (
    PACKAGE_DIR,                                    # noqa
@@ -8,31 +9,28 @@ from .paths import (
 from .constants import CONSTANTS, CONSTANTS_CONFIG, PACKAGE_DIR, DATA_DIR, ARCHIVE_DIR      # noqa
 from .version import VERSION                        # noqa

-
-import abx
-
+# import abx

 # @abx.hookimpl
-# def get_INSTALLED_APPS():
-#     return ['config']
-
-
-@abx.hookimpl
-def get_CONFIG():
-    from .common import (
-        SHELL_CONFIG,
-        STORAGE_CONFIG,
-        GENERAL_CONFIG,
-        SERVER_CONFIG,
-        ARCHIVING_CONFIG,
-        SEARCH_BACKEND_CONFIG,
-    )
-    return {
-        'SHELL_CONFIG': SHELL_CONFIG,
-        'STORAGE_CONFIG': STORAGE_CONFIG,
-        'GENERAL_CONFIG': GENERAL_CONFIG,
-        'SERVER_CONFIG': SERVER_CONFIG,
-        'ARCHIVING_CONFIG': ARCHIVING_CONFIG,
-        'SEARCHBACKEND_CONFIG': SEARCH_BACKEND_CONFIG,
-    }
+# def get_CONFIG():
+#     from .common import (
+#         SHELL_CONFIG,
+#         STORAGE_CONFIG,
+#         GENERAL_CONFIG,
+#         SERVER_CONFIG,
+#         ARCHIVING_CONFIG,
+#         SEARCH_BACKEND_CONFIG,
+#     )
+#     return {
+#         'SHELL_CONFIG': SHELL_CONFIG,
+#         'STORAGE_CONFIG': STORAGE_CONFIG,
+#         'GENERAL_CONFIG': GENERAL_CONFIG,
+#         'SERVER_CONFIG': SERVER_CONFIG,
+#         'ARCHIVING_CONFIG': ARCHIVING_CONFIG,
+#         'SEARCHBACKEND_CONFIG': SEARCH_BACKEND_CONFIG,
+#     }

+# @abx.hookimpl
+# def ready():
+#     for config in get_CONFIG().values():
+#         config.validate()
@@ -9,16 +9,18 @@ from configparser import ConfigParser

 from benedict import benedict

+import archivebox
+
 from archivebox.config.constants import CONSTANTS

 from archivebox.misc.logging import stderr


 def get_real_name(key: str) -> str:
-    """get the current canonical name for a given deprecated config key"""
-    from django.conf import settings
+    """get the up-to-date canonical name for a given old alias or current key"""
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
    
-    for section in settings.CONFIGS.values():
+    for section in CONFIGS.values():
        try:
            return section.aliases[key]
        except KeyError:
@@ -115,17 +117,15 @@ def load_config_file() -> Optional[benedict]:


 def section_for_key(key: str) -> Any:
-    from django.conf import settings
-    for config_section in settings.CONFIGS.values():
+    for config_section in archivebox.pm.hook.get_CONFIGS().values():
        if hasattr(config_section, key):
            return config_section
-    return None
+    raise ValueError(f'No config section found for key: {key}')


 def write_config_file(config: Dict[str, str]) -> benedict:
    """load the ini-formatted config file from DATA_DIR/Archivebox.conf"""

-    import abx.archivebox.reads
    from archivebox.misc.system import atomic_write

    CONFIG_HEADER = (
@@ -175,7 +175,7 @@ def write_config_file(config: Dict[str, str]) -> benedict:
    updated_config = {}
    try:
        # validate the updated_config by attempting to re-parse it
-        updated_config = {**load_all_config(), **abx.archivebox.reads.get_FLAT_CONFIG()}
+        updated_config = {**load_all_config(), **archivebox.pm.hook.get_FLAT_CONFIG()}
    except BaseException:                                                       # lgtm [py/catch-base-exception]
        # something went horribly wrong, revert to the previous version
        with open(f'{config_path}.bak', 'r', encoding='utf-8') as old:
@@ -233,11 +233,11 @@ def load_config(defaults: Dict[str, Any],
    return benedict(extended_config)

 def load_all_config():
-    import abx.archivebox.reads
+    import abx
    
    flat_config = benedict()
    
-    for config_section in abx.archivebox.reads.get_CONFIGS().values():
+    for config_section in abx.pm.hook.get_CONFIGS().values():
        config_section.__init__()
        flat_config.update(config_section.model_dump())
        
@@ -7,13 +7,13 @@ from typing import Dict, Optional, List
 from pathlib import Path

 from rich import print
-from pydantic import Field, field_validator, computed_field
+from pydantic import Field, field_validator
 from django.utils.crypto import get_random_string

-from abx.archivebox.base_configset import BaseConfigSet
+from abx_spec_config.base_configset import BaseConfigSet

 from .constants import CONSTANTS
-from .version import get_COMMIT_HASH, get_BUILD_TIME
+from .version import get_COMMIT_HASH, get_BUILD_TIME, VERSION
 from .permissions import IN_DOCKER

 ###################### Config ##########################
@@ -31,25 +31,20 @@ class ShellConfig(BaseConfigSet):

    ANSI: Dict[str, str]                = Field(default=lambda c: CONSTANTS.DEFAULT_CLI_COLORS if c.USE_COLOR else CONSTANTS.DISABLED_CLI_COLORS)

-    VERSIONS_AVAILABLE: bool = False             # .check_for_update.get_versions_available_on_github(c)},
-    CAN_UPGRADE: bool = False                    # .check_for_update.can_upgrade(c)},
-
-    @computed_field
    @property
    def TERM_WIDTH(self) -> int:
        if not self.IS_TTY:
            return 200
        return shutil.get_terminal_size((140, 10)).columns
    
-    @computed_field
    @property
    def COMMIT_HASH(self) -> Optional[str]:
        return get_COMMIT_HASH()
    
-    @computed_field
    @property
    def BUILD_TIME(self) -> str:
        return get_BUILD_TIME()
+ 

 SHELL_CONFIG = ShellConfig()

@@ -79,7 +74,6 @@ STORAGE_CONFIG = StorageConfig()
 class GeneralConfig(BaseConfigSet):
    TAG_SEPARATOR_PATTERN: str          = Field(default=r'[,]')

-
 GENERAL_CONFIG = GeneralConfig()


@@ -90,6 +84,7 @@ class ServerConfig(BaseConfigSet):
    CSRF_TRUSTED_ORIGINS: str           = Field(default=lambda c: 'http://localhost:8000,http://127.0.0.1:8000,http://0.0.0.0:8000,http://{}'.format(c.BIND_ADDR))
    
    SNAPSHOTS_PER_PAGE: int             = Field(default=40)
+    PREVIEW_ORIGINALS: bool             = Field(default=True)
    FOOTER_INFO: str                    = Field(default='Content is hosted for personal archiving purposes only.  Contact server owner for any takedown requests.')
    # CUSTOM_TEMPLATES_DIR: Path          = Field(default=None)  # this is now a constant

@@ -99,10 +94,10 @@ class ServerConfig(BaseConfigSet):
    
    ADMIN_USERNAME: str                 = Field(default=None)
    ADMIN_PASSWORD: str                 = Field(default=None)
+    
    REVERSE_PROXY_USER_HEADER: str      = Field(default='Remote-User')
    REVERSE_PROXY_WHITELIST: str        = Field(default='')
    LOGOUT_REDIRECT_URL: str            = Field(default='/')
-    PREVIEW_ORIGINALS: bool             = Field(default=True)
    
 SERVER_CONFIG = ServerConfig()

@@ -116,7 +111,7 @@ class ArchivingConfig(BaseConfigSet):
    MEDIA_MAX_SIZE: str                   = Field(default='750m')
    RESOLUTION: str                       = Field(default='1440,2000')
    CHECK_SSL_VALIDITY: bool              = Field(default=True)
-    USER_AGENT: str                       = Field(default='Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)')
+    USER_AGENT: str                       = Field(default=f'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/131.0.0.0 Safari/537.36 ArchiveBox/{VERSION} (+https://github.com/ArchiveBox/ArchiveBox/)')
    COOKIES_FILE: Path | None             = Field(default=None)
    
    URL_DENYLIST: str                     = Field(default=r'\.(css|js|otf|ttf|woff|woff2|gstatic\.com|googleapis\.com/css)(\?.*)?$', alias='URL_BLACKLIST')
@@ -134,17 +129,15 @@ class ArchivingConfig(BaseConfigSet):
    # CHROME_HEADLESS: bool               = Field(default=True)
    # CHROME_SANDBOX: bool                = Field(default=lambda: not SHELL_CONFIG.IN_DOCKER)

-    @field_validator('TIMEOUT', mode='after')
-    def validate_timeout(cls, v):
-        if int(v) < 5:
-            print(f'[red][!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={v} seconds)[/red]', file=sys.stderr)
+    def validate(self):
+        if int(self.TIMEOUT) < 5:
+            print(f'[red][!] Warning: TIMEOUT is set too low! (currently set to TIMEOUT={self.TIMEOUT} seconds)[/red]', file=sys.stderr)
            print('    You must allow *at least* 5 seconds for indexing and archive methods to run succesfully.', file=sys.stderr)
            print('    (Setting it to somewhere between 30 and 3000 seconds is recommended)', file=sys.stderr)
            print(file=sys.stderr)
            print('    If you want to make ArchiveBox run faster, disable specific archive methods instead:', file=sys.stderr)
            print('        https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#archive-method-toggles', file=sys.stderr)
            print(file=sys.stderr)
-        return v
    
    @field_validator('CHECK_SSL_VALIDITY', mode='after')
    def validate_check_ssl_validity(cls, v):
@@ -167,15 +160,17 @@ class ArchivingConfig(BaseConfigSet):
    @property
    def SAVE_ALLOWLIST_PTNS(self) -> Dict[re.Pattern, List[str]]:
        return {
-            re.compile(k, CONSTANTS.ALLOWDENYLIST_REGEX_FLAGS): v
-            for k, v in self.SAVE_ALLOWLIST.items()
+            # regexp: methods list
+            re.compile(key, CONSTANTS.ALLOWDENYLIST_REGEX_FLAGS): val
+            for key, val in self.SAVE_ALLOWLIST.items()
        } if self.SAVE_ALLOWLIST else {}
    
    @property
    def SAVE_DENYLIST_PTNS(self) -> Dict[re.Pattern, List[str]]:
        return {
-            re.compile(k, CONSTANTS.ALLOWDENYLIST_REGEX_FLAGS): v
-            for k, v in self.SAVE_DENYLIST.items()
+            # regexp: methods list
+            re.compile(key, CONSTANTS.ALLOWDENYLIST_REGEX_FLAGS): val
+            for key, val in self.SAVE_DENYLIST.items()
        } if self.SAVE_DENYLIST else {}

 ARCHIVING_CONFIG = ArchivingConfig()
@@ -1,3 +1,15 @@
+"""
+Constants are for things that never change at runtime.
+(but they can change from run-to-run or machine-to-machine)
+
+DATA_DIR will never change at runtime, but you can run
+archivebox from inside a different DATA_DIR on the same machine.
+
+This is loaded very early in the archivebox startup flow, so nothing in this file 
+or imported from this file should import anything from archivebox.config.common, 
+django, other INSTALLED_APPS, or anything else that is not in a standard library.
+"""
+
 __package__ = 'archivebox.config'

 import re
@@ -197,10 +209,12 @@ class ConstantsDict(Mapping):

    @classmethod
    def __getitem__(cls, key: str):
+        # so it behaves like a dict[key] == dict.key or object attr
        return getattr(cls, key)
    
    @classmethod
    def __benedict__(cls):
+        # when casting to benedict, only include uppercase keys that don't start with an underscore
        return benedict({key: value for key, value in cls.__dict__.items() if key.isupper() and not key.startswith('_')})
    
    @classmethod
@@ -214,5 +228,6 @@ class ConstantsDict(Mapping):
 CONSTANTS = ConstantsDict()
 CONSTANTS_CONFIG = CONSTANTS.__benedict__()

-# add all key: values to globals() for easier importing
-globals().update(CONSTANTS)
+# add all key: values to globals() for easier importing, e.g.:
+# from archivebox.config.constants import IS_ROOT, PERSONAS_DIR, ...
+# globals().update(CONSTANTS)
@@ -60,7 +60,7 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
        return

    with Progress(transient=True, expand=True, console=STDERR) as INITIAL_STARTUP_PROGRESS:
-        INITIAL_STARTUP_PROGRESS_TASK = INITIAL_STARTUP_PROGRESS.add_task("[green]Loading modules...", total=25)
+        INITIAL_STARTUP_PROGRESS_TASK = INITIAL_STARTUP_PROGRESS.add_task("[green]Loading modules...", total=25, visible=False)
        
        from archivebox.config.permissions import IS_ROOT, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, SudoPermission
    
@@ -97,7 +97,7 @@ def setup_django(check_db=False, in_memory_db=False) -> None:
                except Exception as e:
                    bump_startup_progress_bar(advance=1000)
                    
-                    is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ('help', 'version', '--help', '--version', 'init'))
+                    is_using_meta_cmd = any(ignored_subcommand in sys.argv for ignored_subcommand in ('help', 'version', '--help', '--version'))
                    if not is_using_meta_cmd:
                        # show error message to user only if they're not running a meta command / just trying to get help
                        STDERR.print()
@@ -45,7 +45,7 @@ def detect_installed_version(PACKAGE_DIR: Path=PACKAGE_DIR):
@cache
 def get_COMMIT_HASH() -> Optional[str]:
    try:
-        git_dir = PACKAGE_DIR / '../.git'
+        git_dir = PACKAGE_DIR.parent / '.git'
        ref = (git_dir / 'HEAD').read_text().strip().split(' ')[-1]
        commit_hash = git_dir.joinpath(ref).read_text().strip()
        return commit_hash
@@ -53,7 +53,7 @@ def get_COMMIT_HASH() -> Optional[str]:
        pass

    try:
-        return list((PACKAGE_DIR / '../.git/refs/heads/').glob('*'))[0].read_text().strip()
+        return list((PACKAGE_DIR.parent / '.git/refs/heads/').glob('*'))[0].read_text().strip()
    except Exception:
        pass
    
@@ -62,8 +62,12 @@ def get_COMMIT_HASH() -> Optional[str]:
@cache
 def get_BUILD_TIME() -> str:
    if IN_DOCKER:
-        docker_build_end_time = Path('/VERSION.txt').read_text().rsplit('BUILD_END_TIME=')[-1].split('\n', 1)[0]
-        return docker_build_end_time
+        try:
+            # if we're in the archivebox official docker image, /VERSION.txt will contain the build time
+            docker_build_end_time = Path('/VERSION.txt').read_text().rsplit('BUILD_END_TIME=')[-1].split('\n', 1)[0]
+            return docker_build_end_time
+        except Exception:
+            pass

    src_last_modified_unix_timestamp = (PACKAGE_DIR / 'README.md').stat().st_mtime
    return datetime.fromtimestamp(src_last_modified_unix_timestamp).strftime('%Y-%m-%d %H:%M:%S %s')
@@ -14,8 +14,8 @@ from django.utils.html import format_html, mark_safe
 from admin_data_views.typing import TableContext, ItemContext
 from admin_data_views.utils import render_with_table_view, render_with_item_view, ItemLink

-import abx.archivebox.reads
-
+import abx
+import archivebox
 from archivebox.config import CONSTANTS
 from archivebox.misc.util import parse_date

@@ -65,7 +65,7 @@ def obj_to_yaml(obj: Any, indent: int=0) -> str:

@render_with_table_view
 def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:
-
+    FLAT_CONFIG = archivebox.pm.hook.get_FLAT_CONFIG()
    assert request.user.is_superuser, 'Must be a superuser to view configuration settings.'

    rows = {
@@ -81,12 +81,11 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:

    relevant_configs = {
        key: val
-        for key, val in settings.FLAT_CONFIG.items()
+        for key, val in FLAT_CONFIG.items()
        if '_BINARY' in key or '_VERSION' in key
    }

-    for plugin_id, plugin in abx.archivebox.reads.get_PLUGINS().items():
-        plugin = abx.archivebox.reads.get_PLUGIN(plugin_id)
+    for plugin_id, plugin in abx.get_all_plugins().items():
        if not plugin.hooks.get('get_BINARIES'):
            continue
        
@@ -131,17 +130,16 @@ def binaries_list_view(request: HttpRequest, **kwargs) -> TableContext:
@render_with_item_view
 def binary_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:

-    assert request.user.is_superuser, 'Must be a superuser to view configuration settings.'
+    assert request.user and request.user.is_superuser, 'Must be a superuser to view configuration settings.'

    binary = None
    plugin = None
-    for plugin_id in abx.archivebox.reads.get_PLUGINS().keys():
-        loaded_plugin = abx.archivebox.reads.get_PLUGIN(plugin_id)
+    for plugin_id, plugin in abx.get_all_plugins().items():
        try:
-            for loaded_binary in loaded_plugin.hooks.get_BINARIES().values():
+            for loaded_binary in plugin['hooks'].get_BINARIES().values():
                if loaded_binary.name == key:
                    binary = loaded_binary
-                    plugin = loaded_plugin
+                    plugin = plugin
                    # break  # last write wins
        except Exception as e:
            print(e)
@@ -161,7 +159,7 @@ def binary_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
                "name": binary.name,
                "description": binary.abspath,
                "fields": {
-                    'plugin': plugin.package,
+                    'plugin': plugin['package'],
                    'binprovider': binary.loaded_binprovider,
                    'abspath': binary.loaded_abspath,
                    'version': binary.loaded_version,
@@ -215,9 +213,7 @@ def plugins_list_view(request: HttpRequest, **kwargs) -> TableContext:
                return color
        return 'black'

-    for plugin_id in settings.PLUGINS.keys():
-        
-        plugin = abx.archivebox.reads.get_PLUGIN(plugin_id)
+    for plugin_id, plugin in abx.get_all_plugins().items():
        plugin.hooks.get_BINPROVIDERS = plugin.hooks.get('get_BINPROVIDERS', lambda: {})
        plugin.hooks.get_BINARIES = plugin.hooks.get('get_BINARIES', lambda: {})
        plugin.hooks.get_CONFIG = plugin.hooks.get('get_CONFIG', lambda: {})
@@ -263,7 +259,7 @@ def plugin_detail_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:

    assert plugin_id, f'Could not find a plugin matching the specified name: {key}'

-    plugin = abx.archivebox.reads.get_PLUGIN(plugin_id)
+    plugin = abx.get_plugin(plugin_id)

    return ItemContext(
        slug=key,
@@ -1,2 +1,31 @@
 __package__ = 'archivebox.core'

+import abx
+
+@abx.hookimpl
+def register_admin(admin_site):
+    """Register the core.models views (Snapshot, ArchiveResult, Tag, etc.) with the admin site"""
+    from core.admin import register_admin
+    register_admin(admin_site)
+
+
+
+@abx.hookimpl
+def get_CONFIG():
+    from archivebox.config.common import (
+        SHELL_CONFIG,
+        STORAGE_CONFIG,
+        GENERAL_CONFIG,
+        SERVER_CONFIG,
+        ARCHIVING_CONFIG,
+        SEARCH_BACKEND_CONFIG,
+    )
+    return {
+        'SHELL_CONFIG': SHELL_CONFIG,
+        'STORAGE_CONFIG': STORAGE_CONFIG,
+        'GENERAL_CONFIG': GENERAL_CONFIG,
+        'SERVER_CONFIG': SERVER_CONFIG,
+        'ARCHIVING_CONFIG': ARCHIVING_CONFIG,
+        'SEARCHBACKEND_CONFIG': SEARCH_BACKEND_CONFIG,
+    }
+
@@ -0,0 +1,41 @@
+__package__ = 'archivebox.core'
+
+from typing import ClassVar
+
+from statemachine import State
+
+from core.models import Snapshot, ArchiveResult
+from core.statemachines import SnapshotMachine, ArchiveResultMachine
+from actors.actor import ActorType
+
+
+class SnapshotActor(ActorType[Snapshot]):
+    Model = Snapshot
+    StateMachineClass = SnapshotMachine
+    
+    ACTIVE_STATE: ClassVar[State] = SnapshotMachine.started
+    FINAL_STATES: ClassVar[list[State]] = SnapshotMachine.final_states
+    STATE_FIELD_NAME: ClassVar[str] = SnapshotMachine.state_field_name
+    
+    MAX_CONCURRENT_ACTORS: ClassVar[int] = 3
+    MAX_TICK_TIME: ClassVar[int] = 10
+    CLAIM_FROM_TOP_N: ClassVar[int] = MAX_CONCURRENT_ACTORS * 10
+
+
+
+class ArchiveResultActor(ActorType[ArchiveResult]):
+    Model = ArchiveResult
+    StateMachineClass = ArchiveResultMachine
+    
+    ACTIVE_STATE: ClassVar[State] = ArchiveResultMachine.started
+    FINAL_STATES: ClassVar[list[State]] = ArchiveResultMachine.final_states
+    STATE_FIELD_NAME: ClassVar[str] = ArchiveResultMachine.state_field_name
+    
+    MAX_CONCURRENT_ACTORS: ClassVar[int] = 6
+    MAX_TICK_TIME: ClassVar[int] = 60
+    CLAIM_FROM_TOP_N: ClassVar[int] = MAX_CONCURRENT_ACTORS * 10
+
+    # @classproperty
+    # def qs(cls) -> QuerySet[ModelType]:
+    #     """Get the unfiltered and unsorted QuerySet of all objects that this Actor might care about."""
+    #     return cls.Model.objects.filter(extractor='favicon')
@@ -8,7 +8,7 @@ from django.utils.html import format_html, mark_safe
 from django.core.exceptions import ValidationError
 from django.urls import reverse, resolve
 from django.utils import timezone
-from django.forms import forms
+from django_jsonform.forms.fields import JSONFormField

 from huey_monitor.admin import TaskModel

@@ -83,7 +83,7 @@ class ArchiveResultInline(admin.TabularInline):
        formset.form.base_fields['cmd_version'].initial = '-'
        formset.form.base_fields['pwd'].initial = str(snapshot.link_dir)
        formset.form.base_fields['created_by'].initial = request.user
-        formset.form.base_fields['cmd'] = forms.JSONField(initial=['-'])
+        formset.form.base_fields['cmd'] = JSONFormField(initial=['-'])
        formset.form.base_fields['output'].initial = 'Manually recorded cmd output...'
        
        if obj is not None:
@@ -2,7 +2,7 @@ __package__ = 'archivebox.core'

 from django.contrib import admin

-import abx.django.use
+import archivebox

 class ArchiveBoxAdmin(admin.AdminSite):
    site_header = 'ArchiveBox'
@@ -37,6 +37,6 @@ def register_admin_site():
    sites.site = archivebox_admin
    
    # register all plugins admin classes
-    abx.django.use.register_admin(archivebox_admin)
+    archivebox.pm.hook.register_admin(admin_site=archivebox_admin)
    
    return archivebox_admin
@@ -13,7 +13,7 @@ from django.template import Template, RequestContext
 from django.contrib.admin.helpers import ActionForm
 from django.contrib.admin.widgets import FilteredSelectMultiple

-from archivebox.config import DATA_DIR, VERSION
+from archivebox.config import DATA_DIR
 from archivebox.config.common import SERVER_CONFIG
 from archivebox.misc.util import htmldecode, urldecode
 from archivebox.misc.paginators import AccelleratedPaginator
@@ -32,8 +32,8 @@ from core.admin_tags import TagInline
 from core.admin_archiveresults import ArchiveResultInline, result_url


-GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': [], 'CAN_UPGRADE': False}
-
+# GLOBAL_CONTEXT = {'VERSION': VERSION, 'VERSIONS_AVAILABLE': [], 'CAN_UPGRADE': False}
+GLOBAL_CONTEXT = {}


 class SnapshotActionForm(ActionForm):
@@ -2,7 +2,7 @@ __package__ = 'archivebox.core'

 from django.apps import AppConfig

-import abx
+import archivebox


 class CoreConfig(AppConfig):
@@ -10,16 +10,11 @@ class CoreConfig(AppConfig):

    def ready(self):
        """Register the archivebox.core.admin_site as the main django admin site"""
+        from django.conf import settings
+        archivebox.pm.hook.ready(settings=settings)
+        
        from core.admin_site import register_admin_site
        register_admin_site()
        
-        abx.pm.hook.ready()


-
-
-@abx.hookimpl
-def register_admin(admin_site):
-    """Register the core.models views (Snapshot, ArchiveResult, Tag, etc.) with the admin site"""
-    from core.admin import register_admin
-    register_admin(admin_site)
@@ -0,0 +1,19 @@
+# Generated by Django 5.1.2 on 2025-12-29 10:16
+
+import django.utils.timezone
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0074_alter_snapshot_downloaded_at'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='archiveresult',
+            name='retry_at',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+    ]
@@ -0,0 +1,71 @@
+# Generated by Django 5.1.2 on 2025-12-29 10:16
+
+import django.db.models.deletion
+import django.utils.timezone
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    dependencies = [
+        ('core', '0075_archiveresult_retry_at'),
+        ('crawls', '0001_initial'),
+    ]
+
+    operations = [
+        migrations.AddField(
+            model_name='snapshot',
+            name='crawl',
+            field=models.ForeignKey(blank=True, default=None, null=True, on_delete=django.db.models.deletion.CASCADE, related_name='snapshot_set', to='crawls.crawl'),
+        ),
+        migrations.AddField(
+            model_name='snapshot',
+            name='retry_at',
+            field=models.DateTimeField(db_index=True, default=django.utils.timezone.now),
+        ),
+        migrations.AddField(
+            model_name='snapshot',
+            name='status',
+            field=models.CharField(choices=[('queued', 'Queued'), ('started', 'Started'), ('sealed', 'Sealed')], db_index=True, default='queued', max_length=15),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='cmd',
+            field=models.JSONField(blank=True, default=None, null=True),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='end_ts',
+            field=models.DateTimeField(blank=True, default=None, null=True),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='extractor',
+            field=models.CharField(choices=[('htmltotext', 'htmltotext'), ('git', 'git'), ('singlefile', 'singlefile'), ('media', 'media'), ('archive_org', 'archive_org'), ('readability', 'readability'), ('mercury', 'mercury'), ('favicon', 'favicon'), ('pdf', 'pdf'), ('headers', 'headers'), ('screenshot', 'screenshot'), ('dom', 'dom'), ('title', 'title'), ('wget', 'wget')], db_index=True, max_length=32),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='output',
+            field=models.CharField(blank=True, default=None, max_length=1024, null=True),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='pwd',
+            field=models.CharField(blank=True, default=None, max_length=256, null=True),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='snapshot',
+            field=models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, to='core.snapshot'),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='start_ts',
+            field=models.DateTimeField(blank=True, default=None, null=True),
+        ),
+        migrations.AlterField(
+            model_name='archiveresult',
+            name='status',
+            field=models.CharField(choices=[('queued', 'Queued'), ('started', 'Started'), ('backoff', 'Waiting to retry'), ('succeeded', 'Succeeded'), ('failed', 'Failed'), ('skipped', 'Skipped')], db_index=True, default='queued', max_length=15),
+        ),
+    ]
@@ -8,21 +8,25 @@ import os
 import json

 from pathlib import Path
+from datetime import timedelta

 from django.db import models
 from django.utils.functional import cached_property
 from django.utils.text import slugify
+from django.utils import timezone
 from django.core.cache import cache
 from django.urls import reverse, reverse_lazy
 from django.db.models import Case, When, Value, IntegerField
 from django.contrib import admin
 from django.conf import settings

+from actors.models import ModelWithStateMachine
+
 from archivebox.config import CONSTANTS

 from abid_utils.models import ABIDModel, ABIDField, AutoDateTimeField
 from queues.tasks import bg_archive_snapshot
-# from crawls.models import Crawl
+from crawls.models import Crawl
 # from machine.models import Machine, NetworkInterface

 from archivebox.misc.system import get_dir_size
@@ -152,7 +156,7 @@ class SnapshotManager(models.Manager):
        return super().get_queryset().prefetch_related('tags', 'archiveresult_set')  # .annotate(archiveresult_count=models.Count('archiveresult')).distinct()


-class Snapshot(ABIDModel):
+class Snapshot(ABIDModel, ModelWithStateMachine):
    abid_prefix = 'snp_'
    abid_ts_src = 'self.created_at'
    abid_uri_src = 'self.url'
@@ -160,23 +164,32 @@ class Snapshot(ABIDModel):
    abid_rand_src = 'self.id'
    abid_drift_allowed = True

+    state_machine_name = 'core.statemachines.SnapshotMachine'
+    state_field_name = 'status'
+    retry_at_field_name = 'retry_at'
+    StatusChoices = ModelWithStateMachine.StatusChoices
+    active_state = StatusChoices.STARTED
+    
    id = models.UUIDField(primary_key=True, default=None, null=False, editable=False, unique=True, verbose_name='ID')
    abid = ABIDField(prefix=abid_prefix)

-    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=None, null=False, related_name='snapshot_set')
+    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=None, null=False, related_name='snapshot_set', db_index=True)
    created_at = AutoDateTimeField(default=None, null=False, db_index=True)  # loaded from self._init_timestamp
    modified_at = models.DateTimeField(auto_now=True)
+    
+    status = ModelWithStateMachine.StatusField(choices=StatusChoices, default=StatusChoices.QUEUED)
+    retry_at = ModelWithStateMachine.RetryAtField(default=timezone.now)

    # legacy ts fields
    bookmarked_at = AutoDateTimeField(default=None, null=False, editable=True, db_index=True)
    downloaded_at = models.DateTimeField(default=None, null=True, editable=False, db_index=True, blank=True)

-    # crawl = models.ForeignKey(Crawl, on_delete=models.CASCADE, default=None, null=True, blank=True, related_name='snapshot_set')
+    crawl: Crawl = models.ForeignKey(Crawl, on_delete=models.CASCADE, default=None, null=True, blank=True, related_name='snapshot_set', db_index=True)  # type: ignore

    url = models.URLField(unique=True, db_index=True)
    timestamp = models.CharField(max_length=32, unique=True, db_index=True, editable=False)
    tags = models.ManyToManyField(Tag, blank=True, through=SnapshotTag, related_name='snapshot_set', through_fields=('snapshot', 'tag'))
-    title = models.CharField(max_length=512, null=True, blank=True, db_index=True)    
+    title = models.CharField(max_length=512, null=True, blank=True, db_index=True)

    keys = ('url', 'timestamp', 'title', 'tags', 'downloaded_at')

@@ -195,12 +208,14 @@ class Snapshot(ABIDModel):
        return result

    def __repr__(self) -> str:
-        title = (self.title_stripped or '-')[:64]
-        return f'[{self.timestamp}] {self.url[:64]} ({title})'
+        url = self.url or '<no url set>'
+        created_at = self.created_at.strftime("%Y-%m-%d %H:%M") if self.created_at else '<no timestamp set>'
+        if self.id and self.url:
+            return f'[{self.ABID}] {url[:64]} @ {created_at}'
+        return f'[{self.abid_prefix}****not*saved*yet****] {url[:64]} @ {created_at}'

    def __str__(self) -> str:
-        title = (self.title_stripped or '-')[:64]
-        return f'[{self.timestamp}] {self.url[:64]} ({title})'
+        return repr(self)

    @classmethod
    def from_json(cls, info: dict):
@@ -396,6 +411,21 @@ class Snapshot(ABIDModel):
                tags_id.append(Tag.objects.get_or_create(name=tag)[0].pk)
        self.tags.clear()
        self.tags.add(*tags_id)
+        
+    def has_pending_archiveresults(self) -> bool:
+        pending_archiveresults = self.archiveresult_set.exclude(status__in=ArchiveResult.FINAL_OR_ACTIVE_STATES)
+        return pending_archiveresults.exists()
+    
+    def create_pending_archiveresults(self) -> list['ArchiveResult']:
+        archiveresults = []
+        for extractor in EXTRACTORS:
+            archiveresult, _created = ArchiveResult.objects.get_or_create(
+                snapshot=self,
+                extractor=extractor,
+                status=ArchiveResult.INITIAL_STATE,
+            )
+            archiveresults.append(archiveresult)
+        return archiveresults


    # def get_storage_dir(self, create=True, symlink=True) -> Path:
@@ -445,13 +475,27 @@ class ArchiveResultManager(models.Manager):
            ).order_by('indexing_precedence')
        return qs

-class ArchiveResult(ABIDModel):
+class ArchiveResult(ABIDModel, ModelWithStateMachine):
    abid_prefix = 'res_'
    abid_ts_src = 'self.snapshot.created_at'
    abid_uri_src = 'self.snapshot.url'
    abid_subtype_src = 'self.extractor'
    abid_rand_src = 'self.id'
    abid_drift_allowed = True
+    
+    class StatusChoices(models.TextChoices):
+        QUEUED = 'queued', 'Queued'                     # pending, initial
+        STARTED = 'started', 'Started'                  # active
+        
+        BACKOFF = 'backoff', 'Waiting to retry'         # pending
+        SUCCEEDED = 'succeeded', 'Succeeded'            # final
+        FAILED = 'failed', 'Failed'                     # final
+        SKIPPED = 'skipped', 'Skipped'                  # final
+        
+    state_machine_name = 'core.statemachines.ArchiveResultMachine'
+    retry_at_field_name = 'retry_at'
+    state_field_name = 'status'
+    active_state = StatusChoices.STARTED

    EXTRACTOR_CHOICES = (
        ('htmltotext', 'htmltotext'),
@@ -469,29 +513,27 @@ class ArchiveResult(ABIDModel):
        ('title', 'title'),
        ('wget', 'wget'),
    )
-    STATUS_CHOICES = [
-        ("succeeded", "succeeded"),
-        ("failed", "failed"),
-        ("skipped", "skipped")
-    ]
+

    id = models.UUIDField(primary_key=True, default=None, null=False, editable=False, unique=True, verbose_name='ID')
    abid = ABIDField(prefix=abid_prefix)

-    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=None, null=False, related_name='archiveresult_set')
+    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=None, null=False, related_name='archiveresult_set', db_index=True)
    created_at = AutoDateTimeField(default=None, null=False, db_index=True)
    modified_at = models.DateTimeField(auto_now=True)
+    
+    status = ModelWithStateMachine.StatusField(choices=StatusChoices.choices, default=StatusChoices.QUEUED)
+    retry_at = ModelWithStateMachine.RetryAtField(default=timezone.now)

-    snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE, to_field='id', db_column='snapshot_id')
+    snapshot: Snapshot = models.ForeignKey(Snapshot, on_delete=models.CASCADE)   # type: ignore

-    extractor = models.CharField(choices=EXTRACTOR_CHOICES, max_length=32)
-    cmd = models.JSONField()
-    pwd = models.CharField(max_length=256)
+    extractor = models.CharField(choices=EXTRACTOR_CHOICES, max_length=32, blank=False, null=False, db_index=True)
+    cmd = models.JSONField(default=None, null=True, blank=True)
+    pwd = models.CharField(max_length=256, default=None, null=True, blank=True)
    cmd_version = models.CharField(max_length=128, default=None, null=True, blank=True)
-    output = models.CharField(max_length=1024)
-    start_ts = models.DateTimeField(db_index=True)
-    end_ts = models.DateTimeField()
-    status = models.CharField(max_length=16, choices=STATUS_CHOICES)
+    output = models.CharField(max_length=1024, default=None, null=True, blank=True)
+    start_ts = models.DateTimeField(default=None, null=True, blank=True)
+    end_ts = models.DateTimeField(default=None, null=True, blank=True)

    # the network interface that was used to download this result
    # uplink = models.ForeignKey(NetworkInterface, on_delete=models.SET_NULL, null=True, blank=True, verbose_name='Network Interface Used')
@@ -502,10 +544,17 @@ class ArchiveResult(ABIDModel):
        verbose_name = 'Archive Result'
        verbose_name_plural = 'Archive Results Log'

+    def __repr__(self):
+        snapshot_id = getattr(self, 'snapshot_id', None)
+        url = self.snapshot.url if snapshot_id else '<no url set>'
+        created_at = self.snapshot.created_at.strftime("%Y-%m-%d %H:%M") if snapshot_id else '<no timestamp set>'
+        extractor = self.extractor or '<no extractor set>'
+        if self.id and snapshot_id:
+            return f'[{self.ABID}] {url[:64]} @ {created_at} -> {extractor}'
+        return f'[{self.abid_prefix}****not*saved*yet****] {url} @ {created_at} -> {extractor}'

    def __str__(self):
-        # return f'[{self.abid}] 📅 {self.start_ts.strftime("%Y-%m-%d %H:%M")} 📄 {self.extractor} {self.snapshot.url}'
-        return self.extractor
+        return repr(self)

    # TODO: finish connecting machine.models
    # @cached_property
@@ -515,6 +564,10 @@ class ArchiveResult(ABIDModel):
    @cached_property
    def snapshot_dir(self):
        return Path(self.snapshot.link_dir)
+    
+    @cached_property
+    def url(self):
+        return self.snapshot.url

    @property
    def api_url(self) -> str:
@@ -552,7 +605,12 @@ class ArchiveResult(ABIDModel):
        return link.canonical_outputs().get(f'{self.extractor}_path')

    def output_exists(self) -> bool:
-        return os.access(self.output_path(), os.R_OK)
+        return os.path.exists(self.output_path())
+        
+    def create_output_dir(self):
+        snap_dir = self.snapshot_dir
+        snap_dir.mkdir(parents=True, exist_ok=True)
+        return snap_dir / self.output_path()


    # def get_storage_dir(self, create=True, symlink=True):
@@ -9,13 +9,12 @@ from pathlib import Path
 from django.utils.crypto import get_random_string

 import abx
-import abx.archivebox
-import abx.archivebox.reads
-import abx.django.use
+import archivebox

-from archivebox.config import DATA_DIR, PACKAGE_DIR, ARCHIVE_DIR, CONSTANTS
+from archivebox.config import DATA_DIR, PACKAGE_DIR, ARCHIVE_DIR, CONSTANTS  # noqa
 from archivebox.config.common import SHELL_CONFIG, SERVER_CONFIG      # noqa

+
 IS_MIGRATING = 'makemigrations' in sys.argv[:3] or 'migrate' in sys.argv[:3]
 IS_TESTING = 'test' in sys.argv[:3] or 'PYTEST_CURRENT_TEST' in os.environ
 IS_SHELL = 'shell' in sys.argv[:3] or 'shell_plus' in sys.argv[:3]
@@ -25,45 +24,8 @@ IS_GETTING_VERSION_OR_HELP = 'version' in sys.argv or 'help' in sys.argv or '--v
 ### ArchiveBox Plugin Settings
 ################################################################################

-PLUGIN_HOOKSPECS = [
-    'abx.django.hookspec',
-    'abx.pydantic_pkgr.hookspec',
-    'abx.archivebox.hookspec',
-]
-abx.register_hookspecs(PLUGIN_HOOKSPECS)
-
-BUILTIN_PLUGIN_DIRS = {
-    'archivebox':              PACKAGE_DIR,
-    'plugins_pkg':             PACKAGE_DIR / 'plugins_pkg',
-    'plugins_auth':            PACKAGE_DIR / 'plugins_auth',
-    'plugins_search':          PACKAGE_DIR / 'plugins_search',
-    'plugins_extractor':       PACKAGE_DIR / 'plugins_extractor',
-}
-USER_PLUGIN_DIRS = {
-    # 'user_plugins':            DATA_DIR / 'user_plugins',
-}
-
-# Discover ArchiveBox plugins
-BUILTIN_PLUGINS = abx.get_plugins_in_dirs(BUILTIN_PLUGIN_DIRS)
-PIP_PLUGINS = abx.get_pip_installed_plugins(group='archivebox')
-USER_PLUGINS = abx.get_plugins_in_dirs(USER_PLUGIN_DIRS)
-ALL_PLUGINS = {**BUILTIN_PLUGINS, **PIP_PLUGINS, **USER_PLUGINS}
-
-# Load ArchiveBox plugins
-PLUGIN_MANAGER = abx.pm
-abx.archivebox.load_archivebox_plugins(PLUGIN_MANAGER, ALL_PLUGINS)
-PLUGINS = abx.archivebox.reads.get_PLUGINS()
-
-# Load ArchiveBox config from plugins
-CONFIGS = abx.archivebox.reads.get_CONFIGS()
-CONFIG = FLAT_CONFIG = abx.archivebox.reads.get_FLAT_CONFIG()
-BINPROVIDERS = abx.archivebox.reads.get_BINPROVIDERS()
-BINARIES = abx.archivebox.reads.get_BINARIES()
-EXTRACTORS = abx.archivebox.reads.get_EXTRACTORS()
-SEARCHBACKENDS = abx.archivebox.reads.get_SEARCHBACKENDS()
-# REPLAYERS = abx.archivebox.reads.get_REPLAYERS()
-# ADMINDATAVIEWS = abx.archivebox.reads.get_ADMINDATAVIEWS()
-
+ALL_PLUGINS = archivebox.ALL_PLUGINS
+LOADED_PLUGINS = archivebox.LOADED_PLUGINS

 ################################################################################
 ### Django Core Settings
@@ -102,7 +64,8 @@ INSTALLED_APPS = [
    # 'abid_utils',                # handles ABID ID creation, handling, and models
    'config',                    # ArchiveBox config settings (loaded as a plugin, don't need to add it here) 
    'machine',                   # handles collecting and storing information about the host machine, network interfaces, installed binaries, etc.
-    'queues',                    # handles starting and managing background workers and processes
+    'actors',                    # handles starting and managing background workers and processes (orchestrators and actors)
+    'queues',                    # handles starting and managing background workers and processes (supervisord)
    'seeds',                     # handles Seed model and URL source management
    'crawls',                    # handles Crawl and CrawlSchedule models and management
    'personas',                  # handles Persona and session management
@@ -110,7 +73,7 @@ INSTALLED_APPS = [
    'api',                       # Django-Ninja-based Rest API interfaces, config, APIToken model, etc.

    # ArchiveBox plugins
-    *abx.django.use.get_INSTALLED_APPS(),  # all plugin django-apps found in archivebox/plugins_* and data/user_plugins,
+    *abx.as_list(abx.pm.hook.get_INSTALLED_APPS()),  # all plugin django-apps found in archivebox/plugins_* and data/user_plugins,

    # 3rd-party apps from PyPI that need to be loaded last
    'admin_data_views',          # handles rendering some convenient automatic read-only views of data in Django admin
@@ -125,6 +88,7 @@ INSTALLED_APPS = [



+
 MIDDLEWARE = [
    'core.middleware.TimezoneMiddleware',
    'django.middleware.security.SecurityMiddleware',
@@ -135,7 +99,7 @@ MIDDLEWARE = [
    'core.middleware.ReverseProxyAuthMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'core.middleware.CacheControlMiddleware',
-    *abx.django.use.get_MIDDLEWARES(),
+    *abx.as_list(abx.pm.hook.get_MIDDLEWARES()),
 ]


@@ -148,7 +112,7 @@ MIDDLEWARE = [
 AUTHENTICATION_BACKENDS = [
    'django.contrib.auth.backends.RemoteUserBackend',
    'django.contrib.auth.backends.ModelBackend',
-    *abx.django.use.get_AUTHENTICATION_BACKENDS(),
+    *abx.as_list(abx.pm.hook.get_AUTHENTICATION_BACKENDS()),
 ]


@@ -169,7 +133,7 @@ AUTHENTICATION_BACKENDS = [

 STATIC_URL = '/static/'
 TEMPLATES_DIR_NAME = 'templates'
-CUSTOM_TEMPLATES_ENABLED = os.access(CONSTANTS.CUSTOM_TEMPLATES_DIR, os.R_OK) and CONSTANTS.CUSTOM_TEMPLATES_DIR.is_dir()
+CUSTOM_TEMPLATES_ENABLED = os.path.isdir(CONSTANTS.CUSTOM_TEMPLATES_DIR) and os.access(CONSTANTS.CUSTOM_TEMPLATES_DIR, os.R_OK)
 STATICFILES_DIRS = [
    *([str(CONSTANTS.CUSTOM_TEMPLATES_DIR / 'static')] if CUSTOM_TEMPLATES_ENABLED else []),
    # *[
@@ -177,7 +141,7 @@ STATICFILES_DIRS = [
    #     for plugin_dir in PLUGIN_DIRS.values()
    #     if (plugin_dir / 'static').is_dir()
    # ],
-    *abx.django.use.get_STATICFILES_DIRS(),
+    *abx.as_list(abx.pm.hook.get_STATICFILES_DIRS()),
    str(PACKAGE_DIR / TEMPLATES_DIR_NAME / 'static'),
 ]

@@ -188,7 +152,7 @@ TEMPLATE_DIRS = [
    #     for plugin_dir in PLUGIN_DIRS.values()
    #     if (plugin_dir / 'templates').is_dir()
    # ],
-    *abx.django.use.get_TEMPLATE_DIRS(),
+    *abx.as_list(abx.pm.hook.get_TEMPLATE_DIRS()),
    str(PACKAGE_DIR / TEMPLATES_DIR_NAME / 'core'),
    str(PACKAGE_DIR / TEMPLATES_DIR_NAME / 'admin'),
    str(PACKAGE_DIR / TEMPLATES_DIR_NAME),
@@ -228,7 +192,7 @@ SQLITE_CONNECTION_OPTIONS = {
        # https://gcollazo.com/optimal-sqlite-settings-for-django/
        # https://litestream.io/tips/#busy-timeout
        # https://docs.djangoproject.com/en/5.1/ref/databases/#setting-pragma-options
-        "timeout": 5,
+        "timeout": 10,
        "check_same_thread": False,
        "transaction_mode": "IMMEDIATE",
        "init_command": (
@@ -267,7 +231,7 @@ if not IS_GETTING_VERSION_OR_HELP:             # dont create queue.sqlite3 file
    HUEY = {
        "huey_class": "huey.SqliteHuey",
        "filename": CONSTANTS.QUEUE_DATABASE_FILENAME,
-        "name": "system_tasks",
+        "name": "commands",
        "results": True,
        "store_none": True,
        "immediate": False,
@@ -288,11 +252,11 @@ if not IS_GETTING_VERSION_OR_HELP:             # dont create queue.sqlite3 file
    # https://huey.readthedocs.io/en/latest/contrib.html#setting-things-up
    # https://github.com/gaiacoop/django-huey
    DJANGO_HUEY = {
-        "default": "system_tasks",
+        "default": "commands",
        "queues": {
            HUEY["name"]: HUEY.copy(),
            # more registered here at plugin import-time by BaseQueue.register()
-            **abx.django.use.get_DJANGO_HUEY_QUEUES(QUEUE_DATABASE_NAME=CONSTANTS.QUEUE_DATABASE_FILENAME),
+            **abx.as_dict(abx.pm.hook.get_DJANGO_HUEY_QUEUES(QUEUE_DATABASE_NAME=CONSTANTS.QUEUE_DATABASE_FILENAME)),
        },
    }

@@ -517,7 +481,7 @@ ADMIN_DATA_VIEWS = {
                "name": "log",
            },
        },
-        *abx.django.use.get_ADMIN_DATA_VIEWS_URLS(),
+        *abx.as_list(abx.pm.hook.get_ADMIN_DATA_VIEWS_URLS()),
    ],
 }

@@ -611,7 +575,4 @@ if DEBUG_REQUESTS_TRACKER:
 # JET_TOKEN = 'some-api-token-here'


-abx.django.use.register_checks()
-# abx.archivebox.reads.register_all_hooks(globals())
-
 # import ipdb; ipdb.set_trace()
@@ -163,11 +163,6 @@ SETTINGS_LOGGING = {
            "level": "DEBUG",
            "propagate": False,
        },
-        "plugins_extractor": {
-            "handlers": ["default", "logfile"],
-            "level": "DEBUG",
-            "propagate": False,
-        },
        "httpx": {
            "handlers": ["outbound_webhooks"],
            "level": "INFO",
@@ -0,0 +1,118 @@
+__package__ = 'archivebox.snapshots'
+
+from django.utils import timezone
+
+from statemachine import State, StateMachine
+
+from core.models import Snapshot, ArchiveResult
+
+# State Machine Definitions
+#################################################
+
+
+class SnapshotMachine(StateMachine, strict_states=True):
+    """State machine for managing Snapshot lifecycle."""
+    
+    model: Snapshot
+    
+    # States
+    queued = State(value=Snapshot.StatusChoices.QUEUED, initial=True)
+    started = State(value=Snapshot.StatusChoices.STARTED)
+    sealed = State(value=Snapshot.StatusChoices.SEALED, final=True)
+    
+    # Tick Event
+    tick = (
+        queued.to.itself(unless='can_start', internal=True) |
+        queued.to(started, cond='can_start') |
+        started.to.itself(unless='is_finished', internal=True) |
+        started.to(sealed, cond='is_finished')
+    )
+    
+    def __init__(self, snapshot, *args, **kwargs):
+        self.snapshot = snapshot
+        super().__init__(snapshot, *args, **kwargs)
+        
+    def can_start(self) -> bool:
+        return self.snapshot.seed and self.snapshot.seed.uri
+        
+    def is_finished(self) -> bool:
+        return not self.snapshot.has_pending_archiveresults()
+        
+    def on_started(self):
+        self.snapshot.create_pending_archiveresults()
+        self.snapshot.bump_retry_at(seconds=60)
+        self.snapshot.save()
+        
+    def on_sealed(self):
+        self.snapshot.retry_at = None
+        self.snapshot.save()
+
+class ArchiveResultMachine(StateMachine, strict_states=True):
+    """State machine for managing ArchiveResult lifecycle."""
+    
+    model: ArchiveResult
+    
+    # States
+    queued = State(value=ArchiveResult.StatusChoices.QUEUED, initial=True)
+    started = State(value=ArchiveResult.StatusChoices.STARTED)
+    backoff = State(value=ArchiveResult.StatusChoices.BACKOFF)
+    succeeded = State(value=ArchiveResult.StatusChoices.SUCCEEDED, final=True)
+    failed = State(value=ArchiveResult.StatusChoices.FAILED, final=True)
+    
+    # Tick Event
+    tick = (
+        queued.to.itself(unless='can_start', internal=True) |
+        queued.to(started, cond='can_start') |
+        started.to.itself(unless='is_finished', internal=True) |
+        started.to(succeeded, cond='is_succeeded') |
+        started.to(failed, cond='is_failed') |
+        started.to(backoff, cond='is_backoff') |
+        backoff.to.itself(unless='can_start', internal=True) |
+        backoff.to(started, cond='can_start') |
+        backoff.to(succeeded, cond='is_succeeded') |
+        backoff.to(failed, cond='is_failed')
+    )
+
+    def __init__(self, archiveresult, *args, **kwargs):
+        self.archiveresult = archiveresult
+        super().__init__(archiveresult, *args, **kwargs)
+        
+    def can_start(self) -> bool:
+        return self.archiveresult.snapshot and self.archiveresult.snapshot.STATE == Snapshot.active_state
+    
+    def is_succeeded(self) -> bool:
+        return self.archiveresult.output_exists()
+    
+    def is_failed(self) -> bool:
+        return not self.archiveresult.output_exists()
+    
+    def is_backoff(self) -> bool:
+        return self.archiveresult.STATE == ArchiveResult.StatusChoices.BACKOFF
+    
+    def is_finished(self) -> bool:
+        return self.is_failed() or self.is_succeeded()
+
+    def on_started(self):
+        self.archiveresult.start_ts = timezone.now()
+        self.archiveresult.create_output_dir()
+        self.archiveresult.bump_retry_at(seconds=60)
+        self.archiveresult.save()
+
+    def on_backoff(self):
+        self.archiveresult.bump_retry_at(seconds=60)
+        self.archiveresult.save()
+
+    def on_succeeded(self):
+        self.archiveresult.end_ts = timezone.now()
+        self.archiveresult.save()
+
+    def on_failed(self):
+        self.archiveresult.end_ts = timezone.now()
+        self.archiveresult.save()
+        
+    def after_transition(self, event: str, source: State, target: State):
+        print(f"after '{event}' from '{source.id}' to '{target.id}'")
+        # self.archiveresult.save_merkle_index()
+        # self.archiveresult.save_html_index()
+        # self.archiveresult.save_json_index()
+        return "after_transition"
@@ -12,7 +12,6 @@ from django.views import View
 from django.views.generic.list import ListView
 from django.views.generic import FormView
 from django.db.models import Q
-from django.conf import settings
 from django.contrib import messages
 from django.contrib.auth.mixins import UserPassesTestMixin
 from django.views.decorators.csrf import csrf_exempt
@@ -21,6 +20,7 @@ from django.utils.decorators import method_decorator
 from admin_data_views.typing import TableContext, ItemContext
 from admin_data_views.utils import render_with_table_view, render_with_item_view, ItemLink

+import archivebox

 from core.models import Snapshot
 from core.forms import AddLinkForm
@@ -32,9 +32,8 @@ from archivebox.config.common import SHELL_CONFIG, SERVER_CONFIG
 from archivebox.misc.util import base_url, htmlencode, ts_to_date_str
 from archivebox.misc.serve_static import serve_static_with_byterange_support

-from ..plugins_extractor.archivedotorg.config import ARCHIVEDOTORG_CONFIG
-from ..logging_util import printable_filesize
-from ..search import query_search_index
+from archivebox.logging_util import printable_filesize
+from archivebox.search import query_search_index


 class HomepageView(View):
@@ -69,7 +68,7 @@ class SnapshotView(View):
                and embed_path
                and os.access(abs_path, os.R_OK)
                and abs_path.exists()):
-                if abs_path.is_dir() and not any(abs_path.glob('*.*')):
+                if os.path.isdir(abs_path) and not any(abs_path.glob('*.*')):
                    continue

                result_info = {
@@ -103,7 +102,7 @@ class SnapshotView(View):

        # iterate through all the files in the snapshot dir and add the biggest ones to1 the result list
        snap_dir = Path(snapshot.link_dir)
-        assert os.access(snap_dir, os.R_OK) and os.access(snap_dir, os.X_OK)
+        assert os.path.isdir(snap_dir) and os.access(snap_dir, os.R_OK)
        
        for result_file in (*snap_dir.glob('*'), *snap_dir.glob('*/*')):
            extension = result_file.suffix.lstrip('.').lower()
@@ -154,7 +153,7 @@ class SnapshotView(View):
            'status_color': 'success' if link.is_archived else 'danger',
            'oldest_archive_date': ts_to_date_str(link.oldest_archive_date),
            'warc_path': warc_path,
-            'SAVE_ARCHIVE_DOT_ORG': ARCHIVEDOTORG_CONFIG.SAVE_ARCHIVE_DOT_ORG,
+            'SAVE_ARCHIVE_DOT_ORG': archivebox.pm.hook.get_FLAT_CONFIG().SAVE_ARCHIVE_DOT_ORG,
            'PREVIEW_ORIGINALS': SERVER_CONFIG.PREVIEW_ORIGINALS,
            'archiveresults': sorted(archiveresults.values(), key=lambda r: all_types.index(r['name']) if r['name'] in all_types else -r['size']),
            'best_result': best_result,
@@ -500,21 +499,25 @@ class HealthCheckView(View):


 def find_config_section(key: str) -> str:
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
+    
    if key in CONSTANTS_CONFIG:
        return 'CONSTANT'
    matching_sections = [
-        section_id for section_id, section in settings.CONFIGS.items() if key in section.model_fields
+        section_id for section_id, section in CONFIGS.items() if key in section.model_fields
    ]
    section = matching_sections[0] if matching_sections else 'DYNAMIC'
    return section

 def find_config_default(key: str) -> str:
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
+    
    if key in CONSTANTS_CONFIG:
        return str(CONSTANTS_CONFIG[key])
    
    default_val = None

-    for config in settings.CONFIGS.values():
+    for config in CONFIGS.values():
        if key in config.model_fields:
            default_val = config.model_fields[key].default
            break
@@ -530,7 +533,9 @@ def find_config_default(key: str) -> str:
    return default_val

 def find_config_type(key: str) -> str:
-    for config in settings.CONFIGS.values():
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
+    
+    for config in CONFIGS.values():
        if hasattr(config, key):
            type_hints = get_type_hints(config)
            try:
@@ -547,7 +552,8 @@ def key_is_safe(key: str) -> bool:

@render_with_table_view
 def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
-
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
+    
    assert request.user.is_superuser, 'Must be a superuser to view configuration settings.'

    rows = {
@@ -560,7 +566,7 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
        # "Aliases": [],
    }

-    for section_id, section in reversed(list(settings.CONFIGS.items())):
+    for section_id, section in reversed(list(CONFIGS.items())):
        for key, field in section.model_fields.items():
            rows['Section'].append(section_id)   # section.replace('_', ' ').title().replace(' Config', '')
            rows['Key'].append(ItemLink(key, key=key))
@@ -570,7 +576,6 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:
            # rows['Documentation'].append(mark_safe(f'Wiki: <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Configuration#{key.lower()}">{key}</a>'))
            # rows['Aliases'].append(', '.join(find_config_aliases(key)))

-   
    section = 'CONSTANT'
    for key in CONSTANTS_CONFIG.keys():
        rows['Section'].append(section)   # section.replace('_', ' ').title().replace(' Config', '')
@@ -589,7 +594,9 @@ def live_config_list_view(request: HttpRequest, **kwargs) -> TableContext:

@render_with_item_view
 def live_config_value_view(request: HttpRequest, key: str, **kwargs) -> ItemContext:
-
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
+    FLAT_CONFIG = archivebox.pm.hook.get_FLAT_CONFIG()
+    
    assert request.user.is_superuser, 'Must be a superuser to view configuration settings.'

    # aliases = USER_CONFIG.get(key, {}).get("aliases", [])
@@ -597,7 +604,7 @@ def live_config_value_view(request: HttpRequest, key: str, **kwargs) -> ItemCont

    if key in CONSTANTS_CONFIG:
        section_header = mark_safe(f'[CONSTANTS]   &nbsp; <b><code style="color: lightgray">{key}</code></b> &nbsp; <small>(read-only, hardcoded by ArchiveBox)</small>')
-    elif key in settings.FLAT_CONFIG:
+    elif key in FLAT_CONFIG:
        section_header = mark_safe(f'data / ArchiveBox.conf &nbsp; [{find_config_section(key)}]  &nbsp; <b><code style="color: lightgray">{key}</code></b>')
    else:
        section_header = mark_safe(f'[DYNAMIC CONFIG]   &nbsp; <b><code style="color: lightgray">{key}</code></b> &nbsp; <small>(read-only, calculated at runtime)</small>')
@@ -613,7 +620,7 @@ def live_config_value_view(request: HttpRequest, key: str, **kwargs) -> ItemCont
                "fields": {
                    'Key': key,
                    'Type': find_config_type(key),
-                    'Value': settings.FLAT_CONFIG.get(key, settings.CONFIGS.get(key, None)) if key_is_safe(key) else '********',
+                    'Value': FLAT_CONFIG.get(key, CONFIGS.get(key, None)) if key_is_safe(key) else '********',
                },
                "help_texts": {
                    'Key': mark_safe(f'''
@@ -635,13 +642,13 @@ def live_config_value_view(request: HttpRequest, key: str, **kwargs) -> ItemCont
                            <code>{find_config_default(key) or '↗️ See in ArchiveBox source code...'}</code>
                        </a>
                        <br/><br/>
-                        <p style="display: {"block" if key in settings.FLAT_CONFIG else "none"}">
+                        <p style="display: {"block" if key in FLAT_CONFIG else "none"}">
                            <i>To change this value, edit <code>data/ArchiveBox.conf</code> or run:</i>
                            <br/><br/>
                            <code>archivebox config --set {key}="{
                                val.strip("'")
                                if (val := find_config_default(key)) else
-                                (repr(settings.FLAT_CONFIG[key] if key_is_safe(key) else '********')).strip("'")
+                                (repr(FLAT_CONFIG[key] if key_is_safe(key) else '********')).strip("'")
                            }"</code>
                        </p>
                    '''),
@@ -0,0 +1,23 @@
+__package__ = 'archivebox.crawls'
+
+from typing import ClassVar
+
+from crawls.models import Crawl
+from crawls.statemachines import CrawlMachine
+
+from actors.actor import ActorType, State
+
+
+class CrawlActor(ActorType[Crawl]):
+    """The Actor that manages the lifecycle of all Crawl objects"""
+    
+    Model = Crawl
+    StateMachineClass = CrawlMachine
+    
+    ACTIVE_STATE: ClassVar[State] = CrawlMachine.started
+    FINAL_STATES: ClassVar[list[State]] = CrawlMachine.final_states
+    STATE_FIELD_NAME: ClassVar[str] = Crawl.state_field_name
+    
+    MAX_CONCURRENT_ACTORS: ClassVar[int] = 3
+    MAX_TICK_TIME: ClassVar[int] = 10
+    CLAIM_FROM_TOP_N: ClassVar[int] = MAX_CONCURRENT_ACTORS * 10
@@ -0,0 +1,78 @@
+# Generated by Django 5.1.2 on 2025-12-29 10:16
+
+import abid_utils.models
+import charidfield.fields
+import django.core.validators
+import django.db.models.deletion
+import django.utils.timezone
+import statemachine.mixins
+from django.conf import settings
+from django.db import migrations, models
+
+
+class Migration(migrations.Migration):
+
+    initial = True
+
+    dependencies = [
+        ('core', '0075_archiveresult_retry_at'),
+        ('seeds', '0001_initial'),
+        migrations.swappable_dependency(settings.AUTH_USER_MODEL),
+    ]
+
+    operations = [
+        migrations.CreateModel(
+            name='CrawlSchedule',
+            fields=[
+                ('id', models.BigAutoField(auto_created=True, primary_key=True, serialize=False, verbose_name='ID')),
+                ('num_uses_failed', models.PositiveIntegerField(default=0)),
+                ('num_uses_succeeded', models.PositiveIntegerField(default=0)),
+                ('schedule', models.CharField(max_length=64)),
+                ('is_enabled', models.BooleanField(default=True)),
+                ('created_at', abid_utils.models.AutoDateTimeField(db_index=True, default=None)),
+                ('modified_at', models.DateTimeField(auto_now=True)),
+                ('created_by', models.ForeignKey(default=None, on_delete=django.db.models.deletion.CASCADE, to=settings.AUTH_USER_MODEL)),
+            ],
+            options={
+                'abstract': False,
+            },
+        ),
+        migrations.CreateModel(
+            name='Crawl',
+            fields=[
+                ('num_uses_failed', models.PositiveIntegerField(default=0)),
+                ('num_uses_succeeded', models.PositiveIntegerField(default=0)),
+                ('id', models.UUIDField(default=None, editable=False, primary_key=True, serialize=False, unique=True, verbose_name='ID')),
+                ('abid', charidfield.fields.CharIDField(blank=True, db_index=True, default=None, help_text='ABID-format identifier for this entity (e.g. snp_01BJQMF54D093DXEAWZ6JYRPAQ)', max_length=30, null=True, prefix='crl_', unique=True)),
+                ('created_at', abid_utils.models.AutoDateTimeField(db_index=True, default=None)),
+                ('modified_at', models.DateTimeField(auto_now=True)),
+                ('status', models.CharField(choices=[('queued', 'Queued'), ('started', 'Started'), ('sealed', 'Sealed')], db_index=True, default='queued', max_length=15)),
+                ('retry_at', models.DateTimeField(db_index=True, default=django.utils.timezone.now)),
+                ('max_depth', models.PositiveSmallIntegerField(default=0, validators=[django.core.validators.MinValueValidator(0), django.core.validators.MaxValueValidator(4)])),
+                ('tags_str', models.CharField(blank=True, default='', max_length=1024)),
+                ('persona', models.CharField(blank=True, default='auto', max_length=32)),
+                ('config', models.JSONField(default=dict)),
+                ('created_by', models.ForeignKey(default=None, on_delete=django.db.models.deletion.CASCADE, related_name='crawl_set', to=settings.AUTH_USER_MODEL)),
+                ('seed', models.ForeignKey(on_delete=django.db.models.deletion.PROTECT, related_name='crawl_set', to='seeds.seed')),
+                ('schedule', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, to='crawls.crawlschedule')),
+            ],
+            options={
+                'verbose_name': 'Crawl',
+                'verbose_name_plural': 'Crawls',
+            },
+            bases=(models.Model, statemachine.mixins.MachineMixin),
+        ),
+        migrations.CreateModel(
+            name='Outlink',
+            fields=[
+                ('id', models.UUIDField(default=None, editable=False, primary_key=True, serialize=False, unique=True, verbose_name='ID')),
+                ('src', models.URLField()),
+                ('dst', models.URLField()),
+                ('crawl', models.ForeignKey(on_delete=django.db.models.deletion.CASCADE, related_name='outlink_set', to='crawls.crawl')),
+                ('via', models.ForeignKey(blank=True, null=True, on_delete=django.db.models.deletion.SET_NULL, related_name='outlink_set', to='core.archiveresult')),
+            ],
+            options={
+                'unique_together': {('src', 'dst', 'via')},
+            },
+        ),
+    ]
@@ -1,13 +1,20 @@
 __package__ = 'archivebox.crawls'

+from typing import TYPE_CHECKING
 from django_stubs_ext.db.models import TypedModelMeta

+from datetime import timedelta
+
 from django.db import models
-from django.db.models import Q
 from django.core.validators import MaxValueValidator, MinValueValidator 
 from django.conf import settings
-from django.utils import timezone
 from django.urls import reverse_lazy
+from django.utils import timezone
+
+from actors.models import ModelWithStateMachine
+
+if TYPE_CHECKING:
+    from core.models import Snapshot

 from seeds.models import Seed

@@ -41,8 +48,9 @@ class CrawlSchedule(ABIDModel, ModelWithHealthStats):
        """The base crawl that each new scheduled job should copy as a template"""
        return self.crawl_set.first()

+    

-class Crawl(ABIDModel, ModelWithHealthStats):
+class Crawl(ABIDModel, ModelWithHealthStats, ModelWithStateMachine):
    """
    A single session of URLs to archive starting from a given Seed and expanding outwards. An "archiving session" so to speak.

@@ -55,16 +63,25 @@ class Crawl(ABIDModel, ModelWithHealthStats):
    abid_prefix = 'crl_'
    abid_ts_src = 'self.created_at'
    abid_uri_src = 'self.seed.uri'
-    abid_subtype_src = 'self.persona_id'
+    abid_subtype_src = 'self.persona'
    abid_rand_src = 'self.id'
    abid_drift_allowed = True
-
+    
+    state_machine_name = 'crawls.statemachines.CrawlMachine'
+    retry_at_field_name = 'retry_at'
+    state_field_name = 'status'
+    StatusChoices = ModelWithStateMachine.StatusChoices
+    active_state = StatusChoices.STARTED
+    
    id = models.UUIDField(primary_key=True, default=None, null=False, editable=False, unique=True, verbose_name='ID')
    abid = ABIDField(prefix=abid_prefix)

    created_by = models.ForeignKey(settings.AUTH_USER_MODEL, on_delete=models.CASCADE, default=None, null=False, related_name='crawl_set')
    created_at = AutoDateTimeField(default=None, null=False, db_index=True)
    modified_at = models.DateTimeField(auto_now=True)
+    
+    status = ModelWithStateMachine.StatusField(choices=StatusChoices, default=StatusChoices.QUEUED)
+    retry_at = ModelWithStateMachine.RetryAtField(default=timezone.now)

    seed = models.ForeignKey(Seed, on_delete=models.PROTECT, related_name='crawl_set', null=False, blank=False)
    max_depth = models.PositiveSmallIntegerField(default=0, validators=[MinValueValidator(0), MaxValueValidator(4)])
@@ -79,7 +96,7 @@ class Crawl(ABIDModel, ModelWithHealthStats):
    # schedule = models.JSONField()
    # config = models.JSONField()
    
-    # snapshot_set: models.Manager['Snapshot']
+    snapshot_set: models.Manager['Snapshot']
    

    class Meta(TypedModelMeta):
@@ -102,6 +119,23 @@ class Crawl(ABIDModel, ModelWithHealthStats):
    @property
    def api_docs_url(self) -> str:
        return '/api/v1/docs#/Core%20Models/api_v1_core_get_crawl'
+    
+    def has_pending_archiveresults(self) -> bool:
+        from core.models import ArchiveResult
+        
+        snapshot_ids = self.snapshot_set.values_list('id', flat=True)
+        pending_archiveresults = ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids).exclude(status__in=ArchiveResult.FINAL_OR_ACTIVE_STATES)
+        return pending_archiveresults.exists()
+    
+    def create_root_snapshot(self) -> 'Snapshot':
+        from core.models import Snapshot
+        
+        root_snapshot, _ = Snapshot.objects.get_or_create(
+            crawl=self,
+            url=self.seed.uri,
+            status=Snapshot.INITIAL_STATE,
+        )
+        return root_snapshot


 class Outlink(models.Model):
@@ -0,0 +1,48 @@
+__package__ = 'archivebox.crawls'
+
+from statemachine import State, StateMachine
+
+from crawls.models import Crawl
+
+# State Machine Definitions
+#################################################
+
+
+class CrawlMachine(StateMachine, strict_states=True):
+    """State machine for managing Crawl lifecycle."""
+    
+    model: Crawl
+    
+    # States
+    queued = State(value=Crawl.StatusChoices.QUEUED, initial=True)
+    started = State(value=Crawl.StatusChoices.STARTED)
+    sealed = State(value=Crawl.StatusChoices.SEALED, final=True)
+    
+    # Tick Event
+    tick = (
+        queued.to.itself(unless='can_start', internal=True) |
+        queued.to(started, cond='can_start') |
+        started.to.itself(unless='is_finished', internal=True) |
+        started.to(sealed, cond='is_finished')
+    )
+    
+    def __init__(self, crawl, *args, **kwargs):
+        self.crawl = crawl
+        super().__init__(crawl, *args, **kwargs)
+        
+    def can_start(self) -> bool:
+        return self.crawl.seed and self.crawl.seed.uri
+        
+    def is_finished(self) -> bool:
+        return not self.crawl.has_pending_archiveresults()
+
+
+        
+    def on_started(self):
+        self.crawl.create_root_snapshot()
+        self.crawl.bump_retry_at(seconds=10)
+        self.crawl.save()
+        
+    def on_sealed(self):
+        self.crawl.retry_at = None
+        self.crawl.save()
@@ -27,43 +27,29 @@ from ..logging_util import (
    log_archive_method_finished,
 )

-from .title import should_save_title, save_title
-from .favicon import should_save_favicon, save_favicon
-from .wget import should_save_wget, save_wget
-from .singlefile import should_save_singlefile, save_singlefile
-from .readability import should_save_readability, save_readability
-from .mercury import should_save_mercury, save_mercury
-from .htmltotext import should_save_htmltotext, save_htmltotext
-from .pdf import should_save_pdf, save_pdf
-from .screenshot import should_save_screenshot, save_screenshot
-from .dom import should_save_dom, save_dom
-from .git import should_save_git, save_git
-from .media import should_save_media, save_media
-from .archive_org import should_save_archive_dot_org, save_archive_dot_org
-from .headers import should_save_headers, save_headers
-

 ShouldSaveFunction = Callable[[Link, Optional[Path], Optional[bool]], bool]
 SaveFunction = Callable[[Link, Optional[Path], int], ArchiveResult]
 ArchiveMethodEntry = tuple[str, ShouldSaveFunction, SaveFunction]

 def get_default_archive_methods() -> List[ArchiveMethodEntry]:
+    # TODO: move to abx.pm.hook.get_EXTRACTORS()
    return [
-        ('favicon', should_save_favicon, save_favicon),
-        ('headers', should_save_headers, save_headers),
-        ('singlefile', should_save_singlefile, save_singlefile),
-        ('pdf', should_save_pdf, save_pdf),
-        ('screenshot', should_save_screenshot, save_screenshot),
-        ('dom', should_save_dom, save_dom),
-        ('wget', should_save_wget, save_wget),
-        # keep title, readability, and htmltotext below wget and singlefile, as they depend on them
-        ('title', should_save_title, save_title),
-        ('readability', should_save_readability, save_readability),
-        ('mercury', should_save_mercury, save_mercury),
-        ('htmltotext', should_save_htmltotext, save_htmltotext),
-        ('git', should_save_git, save_git),
-        ('media', should_save_media, save_media),
-        ('archive_org', should_save_archive_dot_org, save_archive_dot_org),
+        # ('favicon', should_save_favicon, save_favicon),
+        # ('headers', should_save_headers, save_headers),
+        # ('singlefile', should_save_singlefile, save_singlefile),
+        # ('pdf', should_save_pdf, save_pdf),
+        # ('screenshot', should_save_screenshot, save_screenshot),
+        # ('dom', should_save_dom, save_dom),
+        # ('wget', should_save_wget, save_wget),
+        # # keep title, readability, and htmltotext below wget and singlefile, as they depend on them
+        # ('title', should_save_title, save_title),
+        # ('readability', should_save_readability, save_readability),
+        # ('mercury', should_save_mercury, save_mercury),
+        # ('htmltotext', should_save_htmltotext, save_htmltotext),
+        # ('git', should_save_git, save_git),
+        # ('media', should_save_media, save_media),
+        # ('archive_org', should_save_archive_dot_org, save_archive_dot_org),
    ]

 ARCHIVE_METHODS_INDEXING_PRECEDENCE = [
@@ -8,6 +8,8 @@ from typing import List, Optional, Iterator, Mapping
 from django.utils.html import format_html, mark_safe   # type: ignore
 from django.core.cache import cache

+import abx
+
 from archivebox.misc.system import atomic_write
 from archivebox.misc.util import (
    enforce_types,
@@ -19,7 +21,6 @@ from archivebox.misc.util import (
 from archivebox.config import CONSTANTS, DATA_DIR, VERSION
 from archivebox.config.common import SERVER_CONFIG
 from archivebox.config.version import get_COMMIT_HASH
-from archivebox.plugins_extractor.archivedotorg.config import ARCHIVEDOTORG_CONFIG

 from .schema import Link
 from ..logging_util import printable_filesize
@@ -80,7 +81,9 @@ def write_html_link_details(link: Link, out_dir: Optional[str]=None) -> None:
@enforce_types
 def link_details_template(link: Link) -> str:

-    from ..extractors.wget import wget_output_path
+    from abx_plugin_wget.wget import wget_output_path
+    
+    SAVE_ARCHIVE_DOT_ORG = abx.pm.hook.get_FLAT_CONFIG().SAVE_ARCHIVE_DOT_ORG

    link_info = link._asdict(extended=True)

@@ -102,7 +105,7 @@ def link_details_template(link: Link) -> str:
        'status': 'archived' if link.is_archived else 'not yet archived',
        'status_color': 'success' if link.is_archived else 'danger',
        'oldest_archive_date': ts_to_date_str(link.oldest_archive_date),
-        'SAVE_ARCHIVE_DOT_ORG': ARCHIVEDOTORG_CONFIG.SAVE_ARCHIVE_DOT_ORG,
+        'SAVE_ARCHIVE_DOT_ORG': SAVE_ARCHIVE_DOT_ORG,
        'PREVIEW_ORIGINALS': SERVER_CONFIG.PREVIEW_ORIGINALS,
    })

@@ -8,6 +8,8 @@ from pathlib import Path
 from datetime import datetime, timezone
 from typing import List, Optional, Iterator, Any, Union

+import abx
+
 from archivebox.config import VERSION, DATA_DIR, CONSTANTS
 from archivebox.config.common import SERVER_CONFIG, SHELL_CONFIG

@@ -19,8 +21,6 @@ from archivebox.misc.util import enforce_types

@enforce_types
 def generate_json_index_from_links(links: List[Link], with_headers: bool):
-    from django.conf import settings
-    
    MAIN_INDEX_HEADER = {
        'info': 'This is an index of site data archived by ArchiveBox: The self-hosted web archive.',
        'schema': 'archivebox.index.json',
@@ -33,11 +33,10 @@ def generate_json_index_from_links(links: List[Link], with_headers: bool):
            'docs': 'https://github.com/ArchiveBox/ArchiveBox/wiki',
            'source': 'https://github.com/ArchiveBox/ArchiveBox',
            'issues': 'https://github.com/ArchiveBox/ArchiveBox/issues',
-            'dependencies': settings.BINARIES.to_dict(),
+            'dependencies': dict(abx.pm.hook.get_BINARIES()),
        },
    }
    
-    
    if with_headers:
        output = {
            **MAIN_INDEX_HEADER,
@@ -17,9 +17,9 @@ from dataclasses import dataclass, asdict, field, fields

 from django.utils.functional import cached_property

-from archivebox.config import ARCHIVE_DIR, CONSTANTS
+import abx

-from plugins_extractor.favicon.config import FAVICON_CONFIG
+from archivebox.config import ARCHIVE_DIR, CONSTANTS

 from archivebox.misc.system import get_dir_size
 from archivebox.misc.util import ts_to_date_str, parse_date
@@ -426,7 +426,9 @@ class Link:
    def canonical_outputs(self) -> Dict[str, Optional[str]]:
        """predict the expected output paths that should be present after archiving"""

-        from ..extractors.wget import wget_output_path
+        from abx_plugin_wget.wget import wget_output_path
+        from abx_plugin_favicon.config import FAVICON_CONFIG
+        
        # TODO: banish this awful duplication from the codebase and import these
        # from their respective extractor files
        canonical = {
@@ -8,9 +8,10 @@ from django.db import models
 from django.utils import timezone
 from django.utils.functional import cached_property

-import abx.archivebox.reads
+import abx
+import archivebox

-from abx.archivebox.base_binary import BaseBinary, BaseBinProvider
+from pydantic_pkgr import Binary, BinProvider
 from archivebox.abid_utils.models import ABIDModel, ABIDField, AutoDateTimeField, ModelWithHealthStats

 from .detect import get_host_guid, get_os_info, get_vm_info, get_host_network, get_host_stats
@@ -180,7 +181,7 @@ class NetworkInterface(ABIDModel, ModelWithHealthStats):


 class InstalledBinaryManager(models.Manager):
-    def get_from_db_or_cache(self, binary: BaseBinary) -> 'InstalledBinary':
+    def get_from_db_or_cache(self, binary: Binary) -> 'InstalledBinary':
        """Get or create an InstalledBinary record for a Binary on the local machine"""
        
        global _CURRENT_BINARIES
@@ -216,7 +217,7 @@ class InstalledBinaryManager(models.Manager):
            # if binary was not yet loaded from filesystem, do it now
            # this is expensive, we have to find it's abspath, version, and sha256, but it's necessary
            # to make sure we have a good, up-to-date record of it in the DB & in-memroy cache
-            binary = binary.load(fresh=True)
+            binary = archivebox.pm.hook.binary_load(binary=binary, fresh=True)

        assert binary.loaded_binprovider and binary.loaded_abspath and binary.loaded_version and binary.loaded_sha256, f'Failed to load binary {binary.name} abspath, version, and sha256'
        
@@ -291,8 +292,8 @@ class InstalledBinary(ABIDModel, ModelWithHealthStats):
        if not hasattr(self, 'machine'):
            self.machine = Machine.objects.current()
        if not self.binprovider:
-            all_known_binproviders = list(abx.archivebox.reads.get_BINPROVIDERS().values())
-            binary = BaseBinary(name=self.name, binproviders=all_known_binproviders).load(fresh=True)
+            all_known_binproviders = list(abx.as_dict(archivebox.pm.hook.get_BINPROVIDERS()).values())
+            binary = archivebox.pm.hook.binary_load(binary=Binary(name=self.name, binproviders=all_known_binproviders), fresh=True)
            self.binprovider = binary.loaded_binprovider.name if binary.loaded_binprovider else None
        if not self.abspath:
            self.abspath = self.BINPROVIDER.get_abspath(self.name)
@@ -304,16 +305,16 @@ class InstalledBinary(ABIDModel, ModelWithHealthStats):
        super().clean(*args, **kwargs)

    @cached_property
-    def BINARY(self) -> BaseBinary:
-        for binary in abx.archivebox.reads.get_BINARIES().values():
+    def BINARY(self) -> Binary:
+        for binary in abx.as_dict(archivebox.pm.hook.get_BINARIES()).values():
            if binary.name == self.name:
                return binary
        raise Exception(f'Orphaned InstalledBinary {self.name} {self.binprovider} was found in DB, could not find any plugin that defines it')
        # TODO: we could technically reconstruct it from scratch, but why would we ever want to do that?

    @cached_property
-    def BINPROVIDER(self) -> BaseBinProvider:
-        for binprovider in abx.archivebox.reads.get_BINPROVIDERS().values():
+    def BINPROVIDER(self) -> BinProvider:
+        for binprovider in abx.as_dict(archivebox.pm.hook.get_BINPROVIDERS()).values():
            if binprovider.name == self.binprovider:
                return binprovider
        raise Exception(f'Orphaned InstalledBinary(name={self.name}) was found in DB, could not find any plugin that defines BinProvider(name={self.binprovider})')
@@ -321,7 +322,7 @@ class InstalledBinary(ABIDModel, ModelWithHealthStats):
    # maybe not a good idea to provide this? Binary in DB is a record of the binary's config
    # whereas a loaded binary is a not-yet saved instance that may not have the same config
    # why would we want to load a binary record from the db when it could be freshly loaded?
-    def load_from_db(self) -> BaseBinary:
+    def load_from_db(self) -> Binary:
        # TODO: implement defaults arg in pydantic_pkgr
        # return self.BINARY.load(defaults={
        #     'binprovider': self.BINPROVIDER,
@@ -330,7 +331,7 @@ class InstalledBinary(ABIDModel, ModelWithHealthStats):
        #     'sha256': self.sha256,
        # })
        
-        return BaseBinary.model_validate({
+        return Binary.model_validate({
            **self.BINARY.model_dump(),
            'abspath': self.abspath and Path(self.abspath),
            'version': self.version,
@@ -340,5 +341,5 @@ class InstalledBinary(ABIDModel, ModelWithHealthStats):
            'overrides': self.BINARY.overrides,
        })

-    def load_fresh(self) -> BaseBinary:
-        return self.BINARY.load(fresh=True)
+    def load_fresh(self) -> Binary:
+        return archivebox.pm.hook.binary_load(binary=self.BINARY, fresh=True)
@@ -14,6 +14,10 @@ from crontab import CronTab, CronSlices
 from django.db.models import QuerySet
 from django.utils import timezone

+from pydantic_pkgr import Binary
+
+import abx
+import archivebox
 from archivebox.misc.checks import check_data_folder
 from archivebox.misc.util import enforce_types                         # type: ignore
 from archivebox.misc.system import get_dir_size, dedupe_cron_jobs, CRON_COMMENT
@@ -22,7 +26,7 @@ from archivebox.misc.logging import stderr, hint
 from archivebox.config import CONSTANTS, VERSION, DATA_DIR, ARCHIVE_DIR
 from archivebox.config.common import SHELL_CONFIG, SEARCH_BACKEND_CONFIG, STORAGE_CONFIG, SERVER_CONFIG, ARCHIVING_CONFIG
 from archivebox.config.permissions import SudoPermission, IN_DOCKER
-from archivebox.config.configfile import (
+from archivebox.config.collection import (
    write_config_file,
    load_all_config,
    get_real_name,
@@ -195,15 +199,13 @@ def version(quiet: bool=False,
    console = Console()
    prnt = console.print
    
-    from django.conf import settings
-    
-    from abx.archivebox.base_binary import BaseBinary, apt, brew, env
+    from abx_plugin_default_binproviders import apt, brew, env
    
    from archivebox.config.version import get_COMMIT_HASH, get_BUILD_TIME
    from archivebox.config.permissions import ARCHIVEBOX_USER, ARCHIVEBOX_GROUP, RUNNING_AS_UID, RUNNING_AS_GID
    from archivebox.config.paths import get_data_locations, get_code_locations
    
-    from plugins_auth.ldap.config import LDAP_CONFIG
+    LDAP_ENABLED = archivebox.pm.hook.get_SCOPE_CONFIG().LDAP_ENABLED


    # 0.7.1
@@ -242,7 +244,7 @@ def version(quiet: bool=False,
        f'SUDO={CONSTANTS.IS_ROOT}',
        f'ID={CONSTANTS.MACHINE_ID}:{CONSTANTS.COLLECTION_ID}',
        f'SEARCH_BACKEND={SEARCH_BACKEND_CONFIG.SEARCH_BACKEND_ENGINE}',
-        f'LDAP={LDAP_CONFIG.LDAP_ENABLED}',
+        f'LDAP={LDAP_ENABLED}',
        #f'DB=django.db.backends.sqlite3 (({CONFIG["SQLITE_JOURNAL_MODE"]})',  # add this if we have more useful info to show eventually
    )
    prnt()
@@ -264,7 +266,8 @@ def version(quiet: bool=False,

    prnt('[pale_green1][i] Binary Dependencies:[/pale_green1]')
    failures = []
-    for name, binary in list(settings.BINARIES.items()):
+    BINARIES = abx.as_dict(archivebox.pm.hook.get_BINARIES())
+    for name, binary in list(BINARIES.items()):
        if binary.name == 'archivebox':
            continue
        
@@ -295,14 +298,15 @@ def version(quiet: bool=False,
            
    prnt()
    prnt('[gold3][i] Package Managers:[/gold3]')
-    for name, binprovider in list(settings.BINPROVIDERS.items()):
+    BINPROVIDERS = abx.as_dict(archivebox.pm.hook.get_BINPROVIDERS())
+    for name, binprovider in list(BINPROVIDERS.items()):
        err = None
        
        if binproviders and binprovider.name not in binproviders:
            continue
        
        # TODO: implement a BinProvider.BINARY() method that gets the loaded binary for a binprovider's INSTALLER_BIN
-        loaded_bin = binprovider.INSTALLER_BINARY or BaseBinary(name=binprovider.INSTALLER_BIN, binproviders=[env, apt, brew])
+        loaded_bin = binprovider.INSTALLER_BINARY or Binary(name=binprovider.INSTALLER_BIN, binproviders=[env, apt, brew])
        
        abspath = None
        if loaded_bin.abspath:
@@ -1050,9 +1054,7 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
    #    - recommend user re-run with sudo if any deps need to be installed as root

    from rich import print
-    from django.conf import settings
    
-    from archivebox import CONSTANTS
    from archivebox.config.permissions import IS_ROOT, ARCHIVEBOX_USER, ARCHIVEBOX_GROUP
    from archivebox.config.paths import get_or_create_working_lib_dir

@@ -1075,11 +1077,11 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
    
    package_manager_names = ', '.join(
        f'[yellow]{binprovider.name}[/yellow]'
-        for binprovider in reversed(list(settings.BINPROVIDERS.values()))
+        for binprovider in reversed(list(abx.as_dict(abx.pm.hook.get_BINPROVIDERS()).values()))
        if not binproviders or (binproviders and binprovider.name in binproviders)
    )
    print(f'[+] Setting up package managers {package_manager_names}...')
-    for binprovider in reversed(list(settings.BINPROVIDERS.values())):
+    for binprovider in reversed(list(abx.as_dict(abx.pm.hook.get_BINPROVIDERS()).values())):
        if binproviders and binprovider.name not in binproviders:
            continue
        try:
@@ -1092,7 +1094,7 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
    
    print()
    
-    for binary in reversed(list(settings.BINARIES.values())):
+    for binary in reversed(list(abx.as_dict(abx.pm.hook.get_BINARIES()).values())):
        if binary.name in ('archivebox', 'django', 'sqlite', 'python'):
            # obviously must already be installed if we are running
            continue
@@ -1122,7 +1124,8 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
                                result = binary.install(binproviders=[binprovider_name], dry_run=dry_run).model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
                                sys.stderr.write("\033[00m\n")     # reset
                            else:
-                                result = binary.load_or_install(binproviders=[binprovider_name], fresh=True, dry_run=dry_run, quiet=False).model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
+                                loaded_binary = archivebox.pm.hook.binary_load_or_install(binary=binary, binproviders=[binprovider_name], fresh=True, dry_run=dry_run, quiet=False)
+                                result = loaded_binary.model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
                            if result and result['loaded_version']:
                                break
                        except Exception as e:
@@ -1133,9 +1136,9 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
                        binary.install(dry_run=dry_run).model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
                        sys.stderr.write("\033[00m\n")  # reset
                    else:
-                        binary.load_or_install(fresh=True, dry_run=dry_run).model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
-            if IS_ROOT:
-                LIB_DIR = 
+                        loaded_binary = archivebox.pm.hook.binary_load_or_install(binary=binary, fresh=True, dry_run=dry_run)
+                        result = loaded_binary.model_dump(exclude={'overrides', 'bin_dir', 'hook_type'})
+            if IS_ROOT and LIB_DIR:
                with SudoPermission(uid=0):
                    if ARCHIVEBOX_USER == 0:
                        os.system(f'chmod -R 777 "{LIB_DIR.resolve()}"')
@@ -1158,7 +1161,7 @@ def install(out_dir: Path=DATA_DIR, binproviders: Optional[List[str]]=None, bina
    
    print('\n[green][√] Set up ArchiveBox and its dependencies successfully.[/green]\n', file=sys.stderr)
    
-    from plugins_pkg.pip.binaries import ARCHIVEBOX_BINARY
+    from abx_plugin_pip.binaries import ARCHIVEBOX_BINARY
    
    extra_args = []
    if binproviders:
@@ -1184,8 +1187,6 @@ def config(config_options_str: Optional[str]=None,
           out_dir: Path=DATA_DIR) -> None:
    """Get and set your ArchiveBox project configuration values"""

-    import abx.archivebox.reads
-
    from rich import print

    check_data_folder()
@@ -1199,7 +1200,8 @@ def config(config_options_str: Optional[str]=None,
    elif config_options_str:
        config_options = config_options_str.split('\n')

-    from django.conf import settings
+    FLAT_CONFIG = archivebox.pm.hook.get_FLAT_CONFIG()
+    CONFIGS = archivebox.pm.hook.get_CONFIGS()
    
    config_options = config_options or []

@@ -1209,8 +1211,8 @@ def config(config_options_str: Optional[str]=None,
    if search:
        if config_options:
            config_options = [get_real_name(key) for key in config_options]
-            matching_config = {key: settings.FLAT_CONFIG[key] for key in config_options if key in settings.FLAT_CONFIG}
-            for config_section in settings.CONFIGS.values():
+            matching_config = {key: FLAT_CONFIG[key] for key in config_options if key in FLAT_CONFIG}
+            for config_section in CONFIGS.values():
                aliases = config_section.aliases
                
                for search_key in config_options:
@@ -1229,15 +1231,15 @@ def config(config_options_str: Optional[str]=None,
    elif get or no_args:
        if config_options:
            config_options = [get_real_name(key) for key in config_options]
-            matching_config = {key: settings.FLAT_CONFIG[key] for key in config_options if key in settings.FLAT_CONFIG}
-            failed_config = [key for key in config_options if key not in settings.FLAT_CONFIG]
+            matching_config = {key: FLAT_CONFIG[key] for key in config_options if key in FLAT_CONFIG}
+            failed_config = [key for key in config_options if key not in FLAT_CONFIG]
            if failed_config:
                stderr()
                stderr('[X] These options failed to get', color='red')
                stderr('    {}'.format('\n    '.join(config_options)))
                raise SystemExit(1)
        else:
-            matching_config = settings.FLAT_CONFIG
+            matching_config = FLAT_CONFIG
        
        print(printable_config(matching_config))
        raise SystemExit(not matching_config)
@@ -1258,20 +1260,20 @@ def config(config_options_str: Optional[str]=None,
            if key != raw_key:
                stderr(f'[i] Note: The config option {raw_key} has been renamed to {key}, please use the new name going forwards.', color='lightyellow')

-            if key in settings.FLAT_CONFIG:
+            if key in FLAT_CONFIG:
                new_config[key] = val.strip()
            else:
                failed_options.append(line)

        if new_config:
-            before = settings.FLAT_CONFIG
+            before = FLAT_CONFIG
            matching_config = write_config_file(new_config)
-            after = {**load_all_config(), **abx.archivebox.reads.get_FLAT_CONFIG()}
+            after = {**load_all_config(), **archivebox.pm.hook.get_FLAT_CONFIG()}
            print(printable_config(matching_config))

            side_effect_changes = {}
            for key, val in after.items():
-                if key in settings.FLAT_CONFIG and (str(before[key]) != str(after[key])) and (key not in matching_config):
+                if key in FLAT_CONFIG and (str(before[key]) != str(after[key])) and (key not in matching_config):
                    side_effect_changes[key] = after[key]
                    # import ipdb; ipdb.set_trace()

@@ -1313,7 +1315,7 @@ def schedule(add: bool=False,
    """Set ArchiveBox to regularly import URLs at specific times using cron"""
    
    check_data_folder()
-    from archivebox.plugins_pkg.pip.binaries import ARCHIVEBOX_BINARY
+    from abx_plugin_pip.binaries import ARCHIVEBOX_BINARY
    from archivebox.config.permissions import USER

    Path(CONSTANTS.LOGS_DIR).mkdir(exist_ok=True)
@@ -201,6 +201,7 @@ def check_tmp_dir(tmp_dir=None, throw=False, quiet=False, must_exist=True):


 def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_exist=True):
+    import archivebox
    from archivebox.config.permissions import ARCHIVEBOX_USER, ARCHIVEBOX_GROUP
    from archivebox.misc.logging import STDERR
    from archivebox.config.paths import dir_is_writable, get_or_create_working_lib_dir
@@ -209,6 +210,8 @@ def check_lib_dir(lib_dir: Path | None = None, throw=False, quiet=False, must_ex
    
    lib_dir = lib_dir or STORAGE_CONFIG.LIB_DIR
    
+    assert lib_dir == archivebox.pm.hook.get_LIB_DIR(), "lib_dir is not the same as the one in the flat config"
+    
    if not must_exist and not os.path.isdir(lib_dir):
        return True
    
@@ -23,7 +23,7 @@ from archivebox import CONSTANTS           # noqa
 from ..main import *                       # noqa
 from ..cli import CLI_SUBCOMMANDS

-CONFIG = settings.FLAT_CONFIG
+CONFIG = archivebox.pm.hook.get_FLAT_CONFIG()
 CLI_COMMAND_NAMES = ", ".join(CLI_SUBCOMMANDS.keys())

 if __name__ == '__main__':
@@ -55,6 +55,5 @@ if __name__ == '__main__':
    prnt('    add[blink][deep_sky_blue4]?[/deep_sky_blue4][/blink]                                                                        [grey53]# add ? after anything to get help[/]')
    prnt('    add("https://example.com/some/new/url")                                     [grey53]# call CLI methods from the shell[/]')
    prnt('    snap = Snapshot.objects.filter(url__contains="https://example.com").last()  [grey53]# query for individual snapshots[/]')
-    prnt('    archivebox.plugins_extractor.wget.apps.WGET_EXTRACTOR.extract(snap.id)      [grey53]# call an extractor directly[/]')
    prnt('    snap.archiveresult_set.all()                                                [grey53]# see extractor results[/]')
    prnt('    bool(re.compile(CONFIG.URL_DENYLIST).search("https://example.com/abc.exe")) [grey53]# test out a config change[/]')
@@ -5,7 +5,7 @@ import requests
 import json as pyjson
 import http.cookiejar

-from typing import List, Optional, Any
+from typing import List, Optional, Any, Callable
 from pathlib import Path
 from inspect import signature
 from functools import wraps
@@ -19,14 +19,13 @@ from requests.exceptions import RequestException, ReadTimeout
 from base32_crockford import encode as base32_encode                            # type: ignore
 from w3lib.encoding import html_body_declared_encoding, http_content_type_encoding
 try:
-    import chardet
+    import chardet    # type:ignore
    detect_encoding = lambda rawdata: chardet.detect(rawdata)["encoding"]
 except ImportError:
    detect_encoding = lambda rawdata: "utf-8"


-from archivebox.config import CONSTANTS
-from archivebox.config.common import ARCHIVING_CONFIG
+from archivebox.config.constants import CONSTANTS

 from .logging import COLOR_DICT

@@ -126,6 +125,7 @@ def is_static_file(url: str):
 def enforce_types(func):
    """
    Enforce function arg and kwarg types at runtime using its python3 type hints
+    Simpler version of pydantic @validate_call decorator
    """
    # TODO: check return type as well

@@ -186,11 +186,11 @@ def str_between(string: str, start: str, end: str=None) -> str:


@enforce_types
-def parse_date(date: Any) -> Optional[datetime]:
+def parse_date(date: Any) -> datetime:
    """Parse unix timestamps, iso format, and human-readable strings"""
    
    if date is None:
-        return None
+        return None    # type: ignore

    if isinstance(date, datetime):
        if date.tzinfo is None:
@@ -212,6 +212,8 @@ def parse_date(date: Any) -> Optional[datetime]:
 def download_url(url: str, timeout: int=None) -> str:
    """Download the contents of a remote url and return the text"""

+    from archivebox.config.common import ARCHIVING_CONFIG
+
    timeout = timeout or ARCHIVING_CONFIG.TIMEOUT
    session = requests.Session()

@@ -241,8 +243,12 @@ def download_url(url: str, timeout: int=None) -> str:
        return url.rsplit('/', 1)[-1]

@enforce_types
-def get_headers(url: str, timeout: int=None) -> str:
+def get_headers(url: str, timeout: int | None=None) -> str:
    """Download the contents of a remote url and return the headers"""
+    # TODO: get rid of this and use an abx pluggy hook instead
+    
+    from archivebox.config.common import ARCHIVING_CONFIG
+    
    timeout = timeout or ARCHIVING_CONFIG.TIMEOUT

    try:
@@ -283,6 +289,7 @@ def get_headers(url: str, timeout: int=None) -> str:
 def ansi_to_html(text: str) -> str:
    """
    Based on: https://stackoverflow.com/questions/19212665/python-converting-ansi-color-codes-to-html
+    Simple way to render colored CLI stdout/stderr in HTML properly, Textual/rich is probably better though.
    """

    TEMPLATE = '<span style="color: rgb{}"><br>'
@@ -306,13 +313,13 @@ def ansi_to_html(text: str) -> str:
@enforce_types
 def dedupe(options: List[str]) -> List[str]:
    """
-    Deduplicates the given options. Options that come later clobber earlier
-    conflicting options.
+    Deduplicates the given CLI args by key=value. Options that come later override earlier.
    """
    deduped = {}

    for option in options:
-        deduped[option.split('=')[0]] = option
+        key = option.split('=')[0]
+        deduped[key] = option

    return list(deduped.values())

@@ -344,6 +351,9 @@ class ExtendedEncoder(pyjson.JSONEncoder):
        
        elif cls_name in ('dict_items', 'dict_keys', 'dict_values'):
            return tuple(obj)
+        
+        elif isinstance(obj, Callable):
+            return str(obj)

        return pyjson.JSONEncoder.default(self, obj)

@@ -1,14 +1,11 @@
 __package__ = 'archivebox.parsers'

 import json
-
 from typing import IO, Iterable

-from ..index.schema import Link
-from archivebox.misc.util import (
-    enforce_types,
-)
+from archivebox.misc.util import enforce_types

+from ..index.schema import Link
 from .generic_json import jsonObjectToLink

 def parse_line(line: str):
@@ -6,8 +6,7 @@ import re
 from typing import IO, Iterable, Optional
 from configparser import ConfigParser

-from pocket import Pocket
-
+import archivebox
 from archivebox.config import CONSTANTS
 from archivebox.misc.util import enforce_types
 from archivebox.misc.system import atomic_write
@@ -22,7 +21,7 @@ API_DB_PATH = CONSTANTS.SOURCES_DIR / 'pocket_api.db'
 _BROKEN_PROTOCOL_RE = re.compile('^(http[s]?)(:/(?!/))')


-def get_pocket_articles(api: Pocket, since=None, page=0):
+def get_pocket_articles(api, since=None, page=0):
    body, headers = api.get(
        state='archive',
        sort='oldest',
@@ -94,7 +93,9 @@ def should_parse_as_pocket_api(text: str) -> bool:
 def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]:
    """Parse bookmarks from the Pocket API"""

-    from archivebox.plugins_extractor.pocket.config import POCKET_CONFIG
+    from pocket import Pocket
+
+    FLAT_CONFIG = archivebox.pm.hook.get_FLAT_CONFIG()

    input_buffer.seek(0)
    pattern = re.compile(r"^pocket:\/\/(\w+)")
@@ -102,7 +103,7 @@ def parse_pocket_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]:
        if should_parse_as_pocket_api(line):
            
            username = pattern.search(line).group(1)
-            api = Pocket(POCKET_CONFIG.POCKET_CONSUMER_KEY, POCKET_CONFIG.POCKET_ACCESS_TOKENS[username])
+            api = Pocket(FLAT_CONFIG.POCKET_CONSUMER_KEY, FLAT_CONFIG.POCKET_ACCESS_TOKENS[username])
            api.last_since = None
    
            for article in get_pocket_articles(api, since=read_since(username)):
@@ -8,9 +8,10 @@ from datetime import datetime
 from typing import IO, Iterable, Optional
 from configparser import ConfigParser

+import abx
+
 from archivebox.misc.util import enforce_types
 from archivebox.misc.system import atomic_write
-from archivebox.plugins_extractor.readwise.config import READWISE_CONFIG

 from ..index.schema import Link

@@ -62,26 +63,30 @@ def link_from_article(article: dict, sources: list):


 def write_cursor(username: str, since: str):
-    if not READWISE_CONFIG.READWISE_DB_PATH.exists():
-        atomic_write(READWISE_CONFIG.READWISE_DB_PATH, "")
+    READWISE_DB_PATH = abx.pm.hook.get_CONFIG().READWISE_DB_PATH
+    
+    if not READWISE_DB_PATH.exists():
+        atomic_write(READWISE_DB_PATH, "")

    since_file = ConfigParser()
    since_file.optionxform = str
-    since_file.read(READWISE_CONFIG.READWISE_DB_PATH)
+    since_file.read(READWISE_DB_PATH)

    since_file[username] = {"since": since}

-    with open(READWISE_CONFIG.READWISE_DB_PATH, "w+") as new:
+    with open(READWISE_DB_PATH, "w+") as new:
        since_file.write(new)


 def read_cursor(username: str) -> Optional[str]:
-    if not READWISE_CONFIG.READWISE_DB_PATH.exists():
-        atomic_write(READWISE_CONFIG.READWISE_DB_PATH, "")
+    READWISE_DB_PATH = abx.pm.hook.get_CONFIG().READWISE_DB_PATH
+    
+    if not READWISE_DB_PATH.exists():
+        atomic_write(READWISE_DB_PATH, "")

    config_file = ConfigParser()
    config_file.optionxform = str
-    config_file.read(READWISE_CONFIG.READWISE_DB_PATH)
+    config_file.read(READWISE_DB_PATH)

    return config_file.get(username, "since", fallback=None)

@@ -97,12 +102,14 @@ def should_parse_as_readwise_reader_api(text: str) -> bool:
 def parse_readwise_reader_api_export(input_buffer: IO[str], **_kwargs) -> Iterable[Link]:
    """Parse bookmarks from the Readwise Reader API"""

+    READWISE_READER_TOKENS = abx.pm.hook.get_CONFIG().READWISE_READER_TOKENS
+
    input_buffer.seek(0)
    pattern = re.compile(r"^readwise-reader:\/\/(\w+)")
    for line in input_buffer:
        if should_parse_as_readwise_reader_api(line):
            username = pattern.search(line).group(1)
-            api = ReadwiseReaderAPI(READWISE_CONFIG.READWISE_READER_TOKENS[username], cursor=read_cursor(username))
+            api = ReadwiseReaderAPI(READWISE_READER_TOKENS[username], cursor=read_cursor(username))

            for article in get_readwise_reader_articles(api):
                yield link_from_article(article, sources=[line])
@@ -0,0 +1,40 @@
+import sys
+import importlib
+from pathlib import Path
+
+PKGS_DIR = Path(__file__).parent
+
+VENDORED_PKGS = [
+    'abx',
+    # 'pydantic-pkgr',
+    # ... everything else in archivebox/pkgs/* comes after ...
+]
+
+# VENDORED_PKGS += [ ... ./pkgs/* ... ]
+for subdir in reversed(sorted(PKGS_DIR.iterdir())):
+    if subdir.is_dir() and subdir.name not in VENDORED_PKGS and not subdir.name.startswith('_'):
+        VENDORED_PKGS.append(subdir.name)
+
+
+def load_vendored_pkgs():
+    """Add archivebox/pkgs to sys.path and import all vendored python packages present within"""
+    if str(PKGS_DIR) not in sys.path:
+        sys.path.append(str(PKGS_DIR))
+    
+    for pkg_name in VENDORED_PKGS:
+        pkg_dir = PKGS_DIR / pkg_name
+        assert pkg_dir.is_dir(), f'Required vendored pkg {pkg_name} could not be found in {pkg_dir}'
+
+        try:
+            lib = importlib.import_module(pkg_name)
+            # print(f"Successfully imported lib from environment {pkg_name}")
+        except ImportError:
+            sys.path.append(str(pkg_dir))  # perhaps the pkg is in a subdirectory of the directory
+            try:
+                lib = importlib.import_module(pkg_name)
+                # print(f"Successfully imported lib from vendored fallback {pkg_name}: {inspect.getfile(lib)}")
+            except ImportError as e:
+                print(f"Failed to import required pkg from sys.path or archivebox/pkgs dir {pkg_name}: {e}", file=sys.stderr)
+                sys.exit(1)
+        
+
@@ -0,0 +1,21 @@
+__label__ = 'Archive.org'
+__homepage__ = 'https://archive.org'
+
+import abx
+
+@abx.hookimpl
+def get_CONFIG():
+    from .config import ARCHIVEDOTORG_CONFIG
+    
+    return {
+        'ARCHIVEDOTORG_CONFIG': ARCHIVEDOTORG_CONFIG
+    }
+
+
+# @abx.hookimpl
+# def get_EXTRACTORS():
+#     from .extractors import ARCHIVEDOTORG_EXTRACTOR
+#
+#     return {
+#         'archivedotorg': ARCHIVEDOTORG_EXTRACTOR,
+#     }
@@ -1,18 +1,19 @@
-__package__ = 'archivebox.extractors'
+__package__ = 'abx_plugin_archivedotorg'


 from pathlib import Path
 from typing import Optional, List, Dict, Tuple
 from collections import defaultdict

-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from archivebox.logging_util import TimedProgress
+from archivebox.index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
 from archivebox.misc.system import run, chmod_file
 from archivebox.misc.util import enforce_types, is_static_file, dedupe
-from archivebox.plugins_extractor.archivedotorg.config import ARCHIVEDOTORG_CONFIG
-from archivebox.plugins_extractor.curl.config import CURL_CONFIG
-from archivebox.plugins_extractor.curl.binaries import CURL_BINARY

-from ..logging_util import TimedProgress
+from abx_plugin_curl.config import CURL_CONFIG
+from abx_plugin_curl.binaries import CURL_BINARY
+
+from .config import ARCHIVEDOTORG_CONFIG


 def get_output_path():
@@ -1,7 +1,4 @@
-__package__ = 'plugins_extractor.archivedotorg'
-
-
-from abx.archivebox.base_configset import BaseConfigSet
+from abx_spec_config.base_configset import BaseConfigSet


 class ArchivedotorgConfig(BaseConfigSet):
@@ -0,0 +1,18 @@
+[project]
+name = "abx-plugin-archivedotorg"
+version = "2024.10.28"
+description = "Add your description here"
+readme = "README.md"
+requires-python = ">=3.10"
+dependencies = [
+    "abx>=0.1.0",
+    "abx-spec-config>=0.1.0",
+    "abx-plugin-curl>=2024.10.24",
+]
+
+[build-system]
+requires = ["hatchling"]
+build-backend = "hatchling.build"
+
+[project.entry-points.abx]
+abx_plugin_archivedotorg = "abx_plugin_archivedotorg"
@@ -0,0 +1,34 @@
+__label__ = 'Chrome'
+__author__ = 'ArchiveBox'
+
+import abx
+
+@abx.hookimpl
+def get_CONFIG():
+    from .config import CHROME_CONFIG
+    
+    return {
+        'CHROME_CONFIG': CHROME_CONFIG
+    }
+
+@abx.hookimpl
+def get_BINARIES():
+    from .binaries import CHROME_BINARY
+    
+    return {
+        'chrome': CHROME_BINARY,
+    }
+
+@abx.hookimpl
+def ready():
+    from .config import CHROME_CONFIG
+    CHROME_CONFIG.validate()
+
+
+# @abx.hookimpl
+# def get_EXTRACTORS():
+#     return {
+#         'pdf': PDF_EXTRACTOR,
+#         'screenshot': SCREENSHOT_EXTRACTOR,
+#         'dom': DOM_EXTRACTOR,
+#     }
@@ -1,5 +1,3 @@
-__package__ = 'plugins_extractor.chrome'
-
 import os
 import platform
 from pathlib import Path
@@ -7,22 +5,22 @@ from typing import List, Optional

 from pydantic import InstanceOf
 from pydantic_pkgr import (
+    Binary,
    BinProvider,
    BinName,
    BinaryOverrides,
    bin_abspath,
 )

-from abx.archivebox.base_binary import BaseBinary, env, apt, brew
+import abx

-# Depends on Other Plugins:
-from archivebox.config import CONSTANTS
-from archivebox.config.common import SHELL_CONFIG
-from plugins_pkg.puppeteer.binproviders import PUPPETEER_BINPROVIDER
-from plugins_pkg.playwright.binproviders import PLAYWRIGHT_BINPROVIDER
+from abx_plugin_default_binproviders import apt, brew, env
+from abx_plugin_puppeteer.binproviders import PUPPETEER_BINPROVIDER
+from abx_plugin_playwright.binproviders import PLAYWRIGHT_BINPROVIDER


 from .config import CHROME_CONFIG
+
 CHROMIUM_BINARY_NAMES_LINUX = [
    "chromium",
    "chromium-browser",
@@ -49,12 +47,13 @@ CHROME_BINARY_NAMES_MACOS = [
 ]
 CHROME_BINARY_NAMES = CHROME_BINARY_NAMES_LINUX + CHROME_BINARY_NAMES_MACOS

-APT_DEPENDENCIES = [
-    'apt-transport-https', 'at-spi2-common', 'chromium-browser',
+CHROME_APT_DEPENDENCIES = [
+    'apt-transport-https', 'at-spi2-common',
    'fontconfig', 'fonts-freefont-ttf', 'fonts-ipafont-gothic', 'fonts-kacst', 'fonts-khmeros', 'fonts-liberation', 'fonts-noto', 'fonts-noto-color-emoji', 'fonts-symbola', 'fonts-thai-tlwg', 'fonts-tlwg-loma-otf', 'fonts-unifont', 'fonts-wqy-zenhei',
    'libasound2', 'libatk-bridge2.0-0', 'libatk1.0-0', 'libatspi2.0-0', 'libavahi-client3', 'libavahi-common-data', 'libavahi-common3', 'libcairo2', 'libcups2',
    'libdbus-1-3', 'libdrm2', 'libfontenc1', 'libgbm1', 'libglib2.0-0', 'libice6', 'libnspr4', 'libnss3', 'libsm6', 'libunwind8', 'libx11-6', 'libxaw7', 'libxcb1',
    'libxcomposite1', 'libxdamage1', 'libxext6', 'libxfixes3', 'libxfont2', 'libxkbcommon0', 'libxkbfile1', 'libxmu6', 'libxpm4', 'libxrandr2', 'libxt6', 'x11-utils', 'x11-xkb-utils', 'xfonts-encodings',
+    'chromium-browser',
 ]


@@ -81,7 +80,7 @@ def create_macos_app_symlink(target: Path, shortcut: Path):
 ###################### Config ##########################


-class ChromeBinary(BaseBinary):
+class ChromeBinary(Binary):
    name: BinName = CHROME_CONFIG.CHROME_BINARY
    binproviders_supported: List[InstanceOf[BinProvider]] = [PUPPETEER_BINPROVIDER, env, PLAYWRIGHT_BINPROVIDER, apt, brew]
    
@@ -96,19 +95,18 @@ class ChromeBinary(BaseBinary):
            'packages': ['chromium'],                   # playwright install chromium
        },
        apt.name: {
-            'packages': APT_DEPENDENCIES,
+            'packages': CHROME_APT_DEPENDENCIES,
        },
        brew.name: {
-            'packages': ['--cask', 'chromium'],
+            'packages': ['--cask', 'chromium'] if platform.system().lower() == 'darwin' else [],
        },
    }

    @staticmethod
    def symlink_to_lib(binary, bin_dir=None) -> None:
-        from archivebox.config.common import STORAGE_CONFIG
-        bin_dir = bin_dir or STORAGE_CONFIG.LIB_DIR / 'bin'
+        bin_dir = bin_dir or abx.pm.hook.get_BIN_DIR()
        
-        if not (binary.abspath and os.access(binary.abspath, os.F_OK)):
+        if not (binary.abspath and os.path.isfile(binary.abspath)):
            return
        
        bin_dir.mkdir(parents=True, exist_ok=True)
@@ -122,7 +120,7 @@ class ChromeBinary(BaseBinary):
                # otherwise on linux we can symlink directly to binary executable
                symlink.unlink(missing_ok=True)
                symlink.symlink_to(binary.abspath)
-        except Exception as err:
+        except Exception:
            # print(f'[red]:warning: Failed to symlink {symlink} -> {binary.abspath}[/red] {err}')
            # not actually needed, we can just run without it
            pass
@@ -133,14 +131,17 @@ class ChromeBinary(BaseBinary):
        Cleans up any state or runtime files that chrome leaves behind when killed by
        a timeout or other error
        """
-        lock_file = Path("~/.config/chromium/SingletonLock").expanduser()
-
-        if SHELL_CONFIG.IN_DOCKER and os.access(lock_file, os.F_OK):
-            lock_file.unlink()
+        try:
+            linux_lock_file = Path("~/.config/chromium/SingletonLock").expanduser()
+            linux_lock_file.unlink(missing_ok=True)
+        except Exception:
+            pass
        
        if CHROME_CONFIG.CHROME_USER_DATA_DIR:
-            if os.access(CHROME_CONFIG.CHROME_USER_DATA_DIR / 'SingletonLock', os.F_OK):
-                lock_file.unlink()
+            try:
+                (CHROME_CONFIG.CHROME_USER_DATA_DIR / 'SingletonLock').unlink(missing_ok=True)
+            except Exception:
+                pass



@@ -1,5 +1,3 @@
-__package__ = 'plugins_extractor.chrome'
-
 import os
 from pathlib import Path
 from typing import List, Optional
@@ -7,8 +5,8 @@ from typing import List, Optional
 from pydantic import Field
 from pydantic_pkgr import bin_abspath

-from abx.archivebox.base_configset import BaseConfigSet
-from abx.archivebox.base_binary import env
+from abx_spec_config.base_configset import BaseConfigSet
+from abx_plugin_default_binproviders import env

 from archivebox.config import CONSTANTS
 from archivebox.config.common import ARCHIVING_CONFIG, SHELL_CONFIG
@@ -81,15 +79,16 @@ class ChromeConfig(BaseConfigSet):
    # Chrome Binary
    CHROME_BINARY: str                      = Field(default='chrome')
    CHROME_DEFAULT_ARGS: List[str]          = Field(default=[
-        '--virtual-time-budget=15000',
-        '--disable-features=DarkMode',
-        "--run-all-compositor-stages-before-draw",
-        "--hide-scrollbars",
-        "--autoplay-policy=no-user-gesture-required",
-        "--no-first-run",
-        "--use-fake-ui-for-media-stream",
-        "--use-fake-device-for-media-stream",
-        "--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'",
+        "--no-first-run",                                              # dont show any first run ui / setup prompts
+        '--virtual-time-budget=15000',                                 # accellerate any animations on the page by 15s into the future
+        '--disable-features=DarkMode',                                 # disable dark mode for archiving
+        "--run-all-compositor-stages-before-draw",                     # dont draw partially rendered content, wait until everything is ready
+        "--hide-scrollbars",                                           # hide scrollbars to prevent layout shift / scrollbar visible in screenshots
+        "--autoplay-policy=no-user-gesture-required",                  # allow media autoplay without user gesture (e.g. on mobile)
+        "--use-fake-ui-for-media-stream",                              # provide fake camera if site tries to request camera access
+        "--use-fake-device-for-media-stream",                          # provide fake camera if site tries to request camera access
+        "--simulate-outdated-no-au='Tue, 31 Dec 2099 23:59:59 GMT'",   # ignore chrome updates
+        "--force-gpu-mem-available-mb=4096",                           # allows for longer full page screenshots https://github.com/puppeteer/puppeteer/issues/5530
    ])
    CHROME_EXTRA_ARGS: List[str]           = Field(default=[])
    
@@ -189,9 +188,14 @@ class ChromeConfig(BaseConfigSet):
        #   cmd_args += ('--timeout={}'.format(options.CHROME_TIMEOUT * 1000),)
    
        if options.CHROME_USER_DATA_DIR:
+            # remove SingletonLock file
+            lockfile = options.CHROME_USER_DATA_DIR / options.CHROME_PROFILE_NAME / 'SingletonLock'
+            lockfile.unlink(missing_ok=True)
+            
            cmd_args.append('--user-data-dir={}'.format(options.CHROME_USER_DATA_DIR))
            cmd_args.append('--profile-directory={}'.format(options.CHROME_PROFILE_NAME or 'Default'))
        
+            # if CHROME_USER_DATA_DIR is set but folder is empty, create a new profile inside it
            if not os.path.isfile(options.CHROME_USER_DATA_DIR / options.CHROME_PROFILE_NAME / 'Preferences'):
                STDERR.print(f'[green]        + creating new Chrome profile in: {pretty_path(options.CHROME_USER_DATA_DIR / options.CHROME_PROFILE_NAME)}[/green]')
                cmd_args.remove('--no-first-run')
@@ -1,18 +1,18 @@
-__package__ = 'archivebox.extractors'
+__package__ = 'abx_plugin_chrome'

 from pathlib import Path
 from typing import Optional

-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from archivebox.index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
 from archivebox.misc.system import run, chmod_file, atomic_write
 from archivebox.misc.util import (
    enforce_types,
    is_static_file,
 )
-from ..logging_util import TimedProgress
+from archivebox.logging_util import TimedProgress

-from plugins_extractor.chrome.config import CHROME_CONFIG
-from plugins_extractor.chrome.binaries import CHROME_BINARY
+from .config import CHROME_CONFIG
+from .binaries import CHROME_BINARY


 def get_output_path():
@@ -1,4 +1,4 @@
-__package__ = 'archivebox.extractors'
+__package__ = 'abx_plugin_chrome'

 from pathlib import Path
 from typing import Optional
@@ -8,11 +8,11 @@ from archivebox.misc.util import (
    enforce_types,
    is_static_file,
 )
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
-from ..logging_util import TimedProgress
+from archivebox.index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from archivebox.logging_util import TimedProgress

-from plugins_extractor.chrome.config import CHROME_CONFIG
-from plugins_extractor.chrome.binaries import CHROME_BINARY
+from .config import CHROME_CONFIG
+from .binaries import CHROME_BINARY


 def get_output_path():
@@ -1,15 +1,15 @@
-__package__ = 'archivebox.extractors'
+__package__ = 'abx_plugin_chrome'

 from pathlib import Path
 from typing import Optional

 from archivebox.misc.system import run, chmod_file
 from archivebox.misc.util import enforce_types, is_static_file
-from ..index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
-from ..logging_util import TimedProgress
+from archivebox.index.schema import Link, ArchiveResult, ArchiveOutput, ArchiveError
+from archivebox.logging_util import TimedProgress

-from plugins_extractor.chrome.config import CHROME_CONFIG
-from plugins_extractor.chrome.binaries import CHROME_BINARY
+from .config import CHROME_CONFIG
+from .binaries import CHROME_BINARY


 def get_output_path():
Author	SHA1	Message	Date
Nick Sweeting	0fef07cd4d	Preserve historical code blocks and pin the complete docs build	2026-09-17 23:59:30 -07:00
Nick Sweeting	9b2e7cd33a	Restore versioned user documentation for v0.8.6rc1	2026-09-17 23:37:07 -07:00
Nick Sweeting	122bd0cc2e	fix archivebox add	2025-12-29 02:58:49 -08:00
Nick Sweeting	ec100bfe29	fix docs build for vendored pkgs	2024-11-12 23:53:34 -08:00
Nick Sweeting	f0a7198861	bump docs changes	2024-11-12 22:51:22 -08:00
Nick Sweeting	57852fd89e	fix sphinx docs build	2024-11-12 22:20:11 -08:00
Nick Sweeting	840f831572	move readthedocs config into subdir	2024-11-12 22:10:00 -08:00
Nick Sweeting	c6710a8df3	Delete CNAME	2024-11-13 01:02:04 -05:00
Nick Sweeting	5ce25d7518	Delete click_test.py	2024-11-13 00:57:50 -05:00
Nick Sweeting	ad7eec26b2	bump docs changes	2024-11-12 21:57:03 -08:00
Nick Sweeting	a0f9d3fdbc	Update README.md	2024-11-12 06:55:39 -08:00
Nick Sweeting	a9a3b153b1	more StateMachine, Actor, and Orchestrator improvements	2024-11-04 07:08:39 -08:00
Nick Sweeting	5d3c2a8a99	Update docker_entrypoint.sh	2024-11-03 17:02:36 -08:00
Nick Sweeting	99ed97836f	Prevent accidentally mounting home folder as DATA_DIR	2024-11-03 16:40:23 -08:00
Nick Sweeting	cad1be950b	Require bash for setup.sh script instead of sh	2024-11-03 16:20:50 -08:00
Nick Sweeting	fd89de5b91	Update setup.sh	2024-11-03 16:15:09 -08:00
Nick Sweeting	1148cadd7a	Update __init__.py	2024-11-03 16:12:29 -05:00
Nick Sweeting	5872375a50	Update Dockerfile.simple	2024-11-03 16:04:04 -05:00
Nick Sweeting	b7b3addbab	v0.8.6-rc: Moving plugins to independent python packages with finite state machine interfaces (#1576 )	2024-11-03 15:57:12 -05:00
Nick Sweeting	b6ab4e298e	merge dev	2024-11-03 12:56:44 -08:00
Nick Sweeting	758c0c6774	add user providable PLAYWRIGHT cache dir	2024-11-03 12:54:04 -08:00
Nick Sweeting	cc49ecb1b5	playwright: support PLAYWRIGHT_BROWSERS_PATH environment variable (#1583 )	2024-11-03 15:47:47 -05:00
Nick Sweeting	50a85ec97b	Update archivebox/plugins_pkg/playwright/binproviders.py	2024-11-03 15:47:00 -05:00
Andrew Dunham	49c520914c	playwright: support PLAYWRIGHT_BROWSERS_PATH environment variable This environment variable is used by Playwright to configure where to install browsers. If specified, the given directory is used instead of the OS-specific cache folder. For compatibility with existing Playwright installations, and better control over where these binaries are installed, check the same environment variable in PlaywrightBinProvider. Signed-off-by: Andrew Dunham <andrew@du.nham.ca>	2024-11-03 13:59:20 -05:00
Nick Sweeting	48f8416762	add new core and crawsl statemachine manager	2024-11-03 00:41:11 -07:00
Nick Sweeting	41efd010f0	add wip crawl actor spec	2024-11-02 19:54:37 -07:00
Nick Sweeting	2337f874ad	better actor atomic claim	2024-11-02 19:54:25 -07:00
Nick Sweeting	9b24fe7390	merge dev	2024-11-02 17:34:33 -07:00
Nick Sweeting	dbe5c0bc07	more orchestrator and actor improvements	2024-11-02 17:25:51 -07:00
Nick Sweeting	27f26fdde8	Update config.yml	2024-11-02 20:07:29 -04:00
Nick Sweeting	f5cf80542e	Update 3-documentation_change.yml	2024-11-02 20:05:46 -04:00
Nick Sweeting	abad13f5da	Rename documentation_change.yml to 3-documentation_change.yml	2024-11-02 20:04:38 -04:00
Nick Sweeting	7862d58ffe	Rename feature_request.yml to 2-feature_request.yml	2024-11-02 20:04:24 -04:00
Nick Sweeting	85747f9a6e	Rename bug_report.yml to 1-bug_report.yml	2024-11-02 20:04:11 -04:00
Nick Sweeting	ce6ae34655	Update config.yml	2024-11-02 20:03:19 -04:00
Nick Sweeting	12a95b5549	Create config.yml	2024-11-02 20:01:32 -04:00
Nick Sweeting	a675949bb4	Update documentation_change.yml	2024-11-02 19:52:54 -04:00
Nick Sweeting	eeac839360	Update documentation_change.yml	2024-11-02 19:52:35 -04:00
Nick Sweeting	ce6aa20ab2	Update documentation_change.yml	2024-11-02 19:50:34 -04:00
Nick Sweeting	c017491d87	Update documentation_change.yml	2024-11-02 19:49:09 -04:00
Nick Sweeting	2e0dc1f3f2	Update documentation_change.yml	2024-11-02 19:48:51 -04:00
Nick Sweeting	e68806b099	Update and rename documentation_change.md to documentation_change.yml	2024-11-02 19:44:10 -04:00
Nick Sweeting	61f150140d	Update feature_request.yml	2024-11-02 19:39:03 -04:00
Nick Sweeting	29486379e8	Update feature_request.yml	2024-11-02 19:32:09 -04:00
Nick Sweeting	80dd3c6cce	Update and rename feature_request.md to feature_request.yml	2024-11-02 19:30:44 -04:00
Nick Sweeting	983119d6ae	Delete .github/ISSUE_TEMPLATE/question_or_discussion.md	2024-11-02 19:12:44 -04:00
Nick Sweeting	2bff4f43dc	Update bug_report.yml	2024-11-02 16:43:31 -04:00
Nick Sweeting	65bb71e288	Update bug_report.yml	2024-11-02 16:39:33 -04:00
Nick Sweeting	b47b4537d1	Update bug_report.yml	2024-11-02 16:28:25 -04:00
Nick Sweeting	a0bbe55459	Update bug_report.yml	2024-11-02 16:27:31 -04:00
Nick Sweeting	8e0e9f2d29	Update bug_report.yml	2024-11-02 16:26:30 -04:00
Nick Sweeting	ea6156f03b	Update bug_report.yml	2024-11-02 16:20:31 -04:00
Nick Sweeting	8ce010afb7	Update bug_report.yml	2024-11-02 15:49:17 -04:00
Nick Sweeting	6adca829aa	Update bug_report.yml	2024-11-02 15:47:40 -04:00
Nick Sweeting	ecfdab1d01	Update and rename bug_report.md to bug_report.yml	2024-11-02 15:46:05 -04:00
Nick Sweeting	721427a484	hide progress bar on startup	2024-10-31 07:11:15 -07:00
Nick Sweeting	17faa5a507	improvements to new actor and orchestrators	2024-10-31 07:11:15 -07:00
Nick Sweeting	9c2eac4e47	add new actors and orchestrators	2024-10-31 07:11:14 -07:00
Nick Sweeting	5ea035c14d	Update README.md	2024-10-29 20:24:46 -07:00
Nick Sweeting	f56cdd2da5	add chrome flag to fix long screenshots getting cut off	2024-10-29 14:51:31 -07:00
Nick Sweeting	5efeb9d347	add get_SCOPE_CONFIG	2024-10-29 00:33:14 -07:00
Nick Sweeting	eb721bd514	tweak parser imports	2024-10-28 20:16:46 -07:00
Nick Sweeting	30cd48c30d	update lockfiles	2024-10-28 20:10:07 -07:00
Nick Sweeting	dee4eb7992	rename vendor dir to pkgs	2024-10-28 20:05:20 -07:00
Nick Sweeting	7d75867650	bump rc version since there have been tons of changes	2024-10-28 20:01:11 -07:00
Nick Sweeting	001056f292	remove vendored copy of pydantic-pkgr	2024-10-28 20:00:21 -07:00
Nick Sweeting	6530d1f4bf	remove vendored copy of pocket and add [debug] group of pkgs for running with DEBUG=True	2024-10-28 19:56:49 -07:00
Nick Sweeting	70926f1d9f	replace os.access with os.path.isdir	2024-10-28 19:06:38 -07:00
Nick Sweeting	a5d99b87b9	add more plugins	2024-10-28 18:47:58 -07:00
Nick Sweeting	d93aa46949	fix django.forms.JSONField does not exist 500 error	2024-10-28 18:47:45 -07:00
Nick Sweeting	d47d429e9d	add placeholder pyproj	2024-10-28 16:12:43 -07:00
Nick Sweeting	b3c1cb716e	move abx plugins inside vendor dir	2024-10-28 04:07:35 -07:00
Nick Sweeting	7ff2c7f4d5	Fix API crash (#1569 )	2024-10-27 12:15:17 -04:00
Ben Muthalaly	4213d7dc27	Fix API crash	2024-10-26 01:53:49 -05:00
Nick Sweeting	5d9a32c364	wip	2024-10-25 01:06:12 -07:00
Nick Sweeting	4b6f08b0fe	swap more direct settings.CONFIG access to abx getters	2024-10-24 15:42:19 -07:00
Nick Sweeting	b61f6ff8d8	rename system_tasks queue to commands queue	2024-10-24 15:41:43 -07:00
Nick Sweeting	60f0458c77	rename configfile to collection	2024-10-24 15:40:24 -07:00
Nick Sweeting	5295320b0b	add django-autotyping to debug pip group	2024-10-24 15:32:37 -07:00
Nick Sweeting	6c2f1d232a	move DEBUG=True packages into pip-available pkgs	2024-10-24 15:30:51 -07:00
Nick Sweeting	c83abd798c	bump version v0.8.5rc52 -> v0.8.5rc53	2024-10-24 15:26:40 -07:00
Nick Sweeting	6770394b95	use pep440_version when bumping version	2024-10-24 15:26:33 -07:00
Nick Sweeting	922fd42b69	bump version 0.8.5rc51 -> 0.8.5rc52	2024-10-24 15:13:26 -07:00
Nick Sweeting	9e4e5d5113	add bumpver and packages dir to pyproject.toml	2024-10-24 15:13:06 -07:00
Nick Sweeting	b6e60e63bc	bump pydantic pkgr	2024-10-24 15:12:44 -07:00
Nick Sweeting	3b1bf27b75	add more info about tzdata req	2024-10-24 14:52:36 -07:00
Nick Sweeting	27d5eb23be	add up-to-date tzdata to pip requirements	2024-10-24 14:50:08 -07:00
Nick Sweeting	9e40dd69a4	more config improvements, move away from settings GLOBALS to getters	2024-10-24 14:50:07 -07:00
Nick Sweeting	b9492f7736	silence ABID drift warning	2024-10-24 14:50:07 -07:00
Nick Sweeting	63bf902f35	chown TMP_DIR and LIB_DIR recursively inside docker_entrypoint to handle custom PUID	2024-10-23 23:28:11 -07:00
Nick Sweeting	082930b2ef	add wip states	2024-10-22 02:25:54 -07:00
Nick Sweeting	e2081e3d67	chown lib dir tmp dir in docker entrypoint	2024-10-22 02:25:37 -07:00
Nick Sweeting	72f777059f	minor plugin improvements	2024-10-22 02:25:20 -07:00
Nick Sweeting	c8790a943a	bump version	2024-10-21 04:00:42 -07:00
Nick Sweeting	1dff8bae40	dont try to install brew casks on linux	2024-10-21 03:44:45 -07:00
Nick Sweeting	cd3b937469	fix syntax error	2024-10-21 03:21:48 -07:00