Compare commits

...
106 Commits
Author SHA1 Message Date
github-actions[bot] f80e6cea2b release: v0.9.35rc103 2026-07-15 18:29:42 +00:00
Nick Sweeting 2a0f2405b0 Update SECURITY.md 2026-07-15 14:29:17 -04:00
github-actions[bot] 2d7d22776b release: v0.9.35rc102 2026-06-24 23:06:21 +00:00
Nick Sweeting a5eefee17d Retry interrupted hook projections 2026-06-24 16:05:34 -07:00
github-actions[bot] 6e82c55371 release: v0.9.35rc101 2026-06-24 20:47:50 +00:00
Nick Sweeting cbeeda9210 Keep plugin dependency selection server-side 2026-06-24 13:47:11 -07:00
github-actions[bot] c85149db13 release: v0.9.35rc100 2026-06-24 20:20:04 +00:00
Nick Sweeting cfb0f7c5d9 Keep search backend selection independent of plugin filters 2026-06-24 13:19:25 -07:00
github-actions[bot] 99282e739e release: v0.9.35rc99 2026-06-24 19:36:45 +00:00
Nick Sweeting d9b7052d5a Use abx-dl plugin dependency pruning 2026-06-24 12:36:00 -07:00
github-actions[bot] 533101033f release: v0.9.35rc98 2026-06-24 19:13:21 +00:00
Nick Sweeting ca2254d55b Centralize plugin selection dependency resolution 2026-06-24 12:12:52 -07:00
github-actions[bot] 598c5a9d19 release: v0.9.35rc97 2026-06-24 18:41:05 +00:00
Nick Sweeting 0a9a32b6e8 Respect crawl plugin selection over env defaults 2026-06-24 11:40:30 -07:00
github-actions[bot] 74b24fa8b1 release: v0.9.35rc96 2026-06-24 18:22:07 +00:00
Nick Sweeting cfbb2ba6cf Update install docs for dev uv packages 2026-06-24 11:21:36 -07:00
github-actions[bot] 928da17a29 release: v0.9.35rc95 2026-06-24 18:16:12 +00:00
Nick Sweeting bf1d7ebe15 Limit disabled extractor plugin narrowing to version tests 2026-06-24 11:15:43 -07:00
github-actions[bot] 23d215591e release: v0.9.35rc94 2026-06-24 17:58:32 +00:00
Nick Sweeting f61c8ad5ea Keep disabled extractor CLI tests from resolving plugins 2026-06-24 10:58:08 -07:00
github-actions[bot] b30f097c17 release: v0.9.35rc93 2026-06-24 17:42:26 +00:00
Nick Sweeting 28bce9ebaf Honor explicit plugin enable aliases over plugin selection 2026-06-24 10:41:59 -07:00
github-actions[bot] bd451e2fbc release: v0.9.35rc92 2026-06-24 17:14:50 +00:00
Nick Sweeting 8284321510 Resolve plugin config aliases in ArchiveBox config 2026-06-24 10:14:28 -07:00
github-actions[bot] b0a7f2d1b8 release: v0.9.35rc91 2026-06-24 16:51:57 +00:00
Nick Sweeting 3fa3794bf3 Skip disabled plugin binary checks in version 2026-06-24 09:51:28 -07:00
github-actions[bot] 1d70e5a4bc release: v0.9.35rc90 2026-06-24 16:29:07 +00:00
Nick Sweeting 11d962406d Update setup script for dev uv installs 2026-06-24 09:28:32 -07:00
github-actions[bot] 785933ee51 release: v0.9.35rc89 2026-06-24 16:08:24 +00:00
Nick Sweeting c91fea7773 Fix unreadable persona profile handling 2026-06-24 09:07:49 -07:00
github-actions[bot] 369d41f26b release: v0.9.35rc88 2026-06-22 11:13:52 +00:00
Nick Sweeting 83188f70e6 Fix stuck snapshots with disabled queued plugins 2026-06-22 04:13:22 -07:00
github-actions[bot] e9cb936c81 release: v0.9.35rc87 2026-06-22 10:48:01 +00:00
Nick Sweeting 0e75c70832 Use locked versions for related repo CI clones 2026-06-22 03:47:33 -07:00
github-actions[bot] 81d91adb49 release: v0.9.35rc86 2026-06-22 10:34:58 +00:00
Nick Sweeting 63e1eaca8c Bump abxpkg runtime dependency 2026-06-22 03:34:32 -07:00
github-actions[bot] 1d328397ab release: v0.9.35rc85 2026-06-22 10:14:42 +00:00
Nick Sweeting 682b2e25b1 Bump opencode plugin session route 2026-06-22 03:13:30 -07:00
github-actions[bot] 28d5052abb release: v0.9.35rc84 2026-06-22 09:51:04 +00:00
Nick Sweeting b0a9f388cf Bump abxpkg and abx-plugins runtime deps 2026-06-22 02:49:36 -07:00
github-actions[bot] ea0315b102 release: v0.9.35rc83 2026-06-22 07:53:57 +00:00
Nick Sweeting f0834ea1b5 Bump abxpkg runtime env fix 2026-06-22 00:53:17 -07:00
github-actions[bot] 481e4be1a1 release: v0.9.35rc82 2026-06-22 07:04:32 +00:00
Nick Sweeting 276d98b5c5 Use shared opencode plugin release 2026-06-22 00:01:24 -07:00
github-actions[bot] 8aba8e98de release: v0.9.35rc81 2026-06-22 06:27:33 +00:00
Nick Sweeting 1dd2a538b1 Bump abx-plugins floor for pnpm isolation 2026-06-21 23:26:47 -07:00
github-actions[bot] 64e7dc92e8 release: v0.9.35rc80 2026-06-22 06:12:58 +00:00
Nick Sweeting 3d4978f89d Align internal package floors 2026-06-21 23:06:53 -07:00
github-actions[bot] 1efcbbf631 release: v0.9.35rc79 2026-06-22 06:05:11 +00:00
Nick Sweeting 7516559c54 Bump abx-plugins floor for OpenCode default 2026-06-21 23:04:46 -07:00
github-actions[bot] 9976993ba6 release: v0.9.35rc78 2026-06-22 05:59:10 +00:00
Nick Sweeting 6fa3b5c9bb Test disabled OpenCode navigation explicitly 2026-06-21 22:58:47 -07:00
github-actions[bot] 395beea104 release: v0.9.35rc77 2026-06-22 05:04:25 +00:00
Nick Sweeting 7712a9803f Hide agent links when OpenCode is disabled 2026-06-21 22:03:44 -07:00
github-actions[bot] 60da688218 release: v0.9.35rc76 2026-06-22 04:33:47 +00:00
Nick Sweeting 095c0e5112 fix admin snapshot changelist preview summaries 2026-06-21 21:33:14 -07:00
github-actions[bot] f40c5629a5 release: v0.9.35rc75 2026-06-21 18:48:15 +00:00
Nick Sweeting 4982b041c6 Scope recursive crawl plugin matrix 2026-06-21 11:47:51 -07:00
github-actions[bot] 5679a2cfe7 release: v0.9.35rc74 2026-06-21 18:21:41 +00:00
Nick Sweeting 711b94161a Limit legacy config version probe 2026-06-21 11:21:17 -07:00
github-actions[bot] 7b7d57c202 release: v0.9.35rc73 2026-06-21 18:07:44 +00:00
Nick Sweeting 676c561372 Stabilize recursive crawl CI assertions 2026-06-21 11:07:21 -07:00
github-actions[bot] 3d2f926a82 release: v0.9.35rc72 2026-06-21 16:41:44 +00:00
Nick Sweeting 4e21be2b1a Read add-view runtime tags from archive DB 2026-06-21 09:41:16 -07:00
github-actions[bot] 6ac4efc864 release: v0.9.35rc71 2026-06-21 16:21:39 +00:00
Nick Sweeting 67959c9013 Pin chrome plugin CI fixes 2026-06-21 09:21:13 -07:00
github-actions[bot] 50cb4f8a8d release: v0.9.35rc70 2026-06-21 15:53:43 +00:00
Nick Sweeting 6b362fbb1a Pin corrected abx-plugins CI fixes 2026-06-21 08:53:19 -07:00
github-actions[bot] 79873adc3c release: v0.9.35rc69 2026-06-21 15:29:42 +00:00
Nick Sweeting 830422c6c6 Pin abx-plugins remote disconnect fix 2026-06-21 08:29:00 -07:00
github-actions[bot] 67a8418809 release: v0.9.35rc68 2026-06-21 15:26:36 +00:00
Nick Sweeting 75d5c53d36 Wait for all CLI import snapshots before restart 2026-06-21 08:26:10 -07:00
github-actions[bot] 8bd69a0f0e release: v0.9.35rc67 2026-06-21 15:03:28 +00:00
Nick Sweeting a350f58e44 Mark hook subprocesses as abxpkg scripts 2026-06-21 08:03:00 -07:00
github-actions[bot] c35c808da8 release: v0.9.35rc66 2026-06-21 14:51:10 +00:00
Nick Sweeting 82724e99dc Tighten public preview screenshot assertions 2026-06-21 07:50:45 -07:00
github-actions[bot] ce8a4790ae release: v0.9.35rc65 2026-06-21 14:31:03 +00:00
Nick Sweeting 614a5bd095 Prefer explicit hook library overrides 2026-06-21 07:30:32 -07:00
github-actions[bot] 215f7f5925 release: v0.9.35rc64 2026-06-21 14:10:05 +00:00
Nick Sweeting 74399c5456 Use maintenance runner for retention cleanup test 2026-06-21 07:09:35 -07:00
github-actions[bot] 284a9fde3d release: v0.9.35rc63 2026-06-21 13:50:09 +00:00
Nick Sweeting fee4bf13cb Handle sealed crawl pause race in API test 2026-06-21 06:49:38 -07:00
github-actions[bot] be9c77320c release: v0.9.35rc62 2026-06-21 13:37:31 +00:00
Nick Sweeting 1f0b2795bb Treat Chrome zombie cleanup as nonfatal 2026-06-21 06:37:00 -07:00
github-actions[bot] 6382722d2f release: v0.9.35rc61 2026-06-21 13:26:23 +00:00
Nick Sweeting 4be16fc00e Pin nonfatal Chrome cleanup plugin 2026-06-21 06:25:50 -07:00
github-actions[bot] dcccafc00e release: v0.9.35rc60 2026-06-21 13:09:33 +00:00
Nick Sweeting 3fa4fd5d18 Fix runtime hook config regressions 2026-06-21 06:09:04 -07:00
github-actions[bot] ef6f616b37 release: v0.9.35rc59 2026-06-21 12:42:53 +00:00
Nick Sweeting 0e0759a680 Skip binary probes outside data dirs 2026-06-21 05:42:26 -07:00
github-actions[bot] 947c340e74 release: v0.9.35rc58 2026-06-21 12:33:06 +00:00
Nick Sweeting 6f6eb92a90 Pin abx-plugins test fixes 2026-06-21 05:32:42 -07:00
github-actions[bot] 0064e20805 release: v0.9.35rc57 2026-06-21 12:24:51 +00:00
Nick Sweeting 59a956bf9f Show dynamic DB binaries in version output 2026-06-21 05:24:15 -07:00
github-actions[bot] afd2fac50b release: v0.9.35rc56 2026-06-21 12:01:57 +00:00
Nick Sweeting edb6a263e9 Disable package age gate in parallel CI installs 2026-06-21 05:01:33 -07:00
github-actions[bot] 1c00719f68 release: v0.9.35rc55 2026-06-21 11:55:23 +00:00
Nick Sweeting e30a4f2a1c Use fixed abx-plugins daemon test tag in CI 2026-06-21 04:54:45 -07:00
github-actions[bot] ba198e000b release: v0.9.35rc54 2026-06-21 11:38:12 +00:00
Nick Sweeting 24ed99a65d Use fixed abx-plugins cookie tag in CI 2026-06-21 04:37:46 -07:00
github-actions[bot] 2cc0035a40 release: v0.9.35rc53 2026-06-21 11:23:24 +00:00
Nick Sweeting b41f12ee38 Use fixed abx-plugins screenshot tag in CI 2026-06-21 04:22:48 -07:00
github-actions[bot] 714caded9c release: v0.9.35rc52 2026-06-21 11:15:03 +00:00
Nick Sweeting 200da8a167 Use fixed abx-plugins ublock tag in CI 2026-06-21 04:14:37 -07:00
github-actions[bot] 9b02796049 release: v0.9.35rc51 2026-06-21 10:33:57 +00:00
Nick Sweeting 062e4e12fd Use fixed abxpkg tag in CI 2026-06-21 03:33:20 -07:00
50 changed files with 1410 additions and 912 deletions

No files matched your search

+2 -1
View File
@@ -68,4 +68,5 @@ Non-logged in users should be tightly limited to prevent attacks like SQL/shell
[`SERVER_SECURITY_MODE`](https://github.com/archiveBox/archiveBox/wiki/configuration#server_security_mode) + CSRF/XSS defenses were only added in the new 0.9.0 `dev` work and are planned to land in `main` in mid-2026.
Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs. The only URLs that are blocked are URLs hosted by the archivebox server itself.
Note archivebox is designed to be able to archive any URL the server can reach, including internal intranet URLs and localhost. Users are expected to configure `URL_ALLOWLIST`/`URL_DENYLIST` accordingly if they need to limit what the server should archive.
It does a basic check to try and avoid recursively archiving snapshots hosted by the archivebox server itself, but it's not a strict security check, it's just to prevent wasted CPU/disk from recursively archiving snapshots in a browser that has the ArchiveBox Browser Extension turned on.
-7
View File
@@ -4,11 +4,6 @@ set -Eeuo pipefail
repo_name="$1"
target_dir="${2:-$repo_name}"
case "$repo_name" in
abx-plugins)
version="1.11.232"
;;
*)
version="$(
python3 - "$repo_name" <<'PY'
import re
@@ -27,8 +22,6 @@ if not match:
print(match.group(1))
PY
)"
;;
esac
echo "Cloning ArchiveBox/${repo_name}@v${version} into ${target_dir}"
git clone --depth=1 --branch "v${version}" "https://github.com/ArchiveBox/${repo_name}.git" "$target_dir"
+1 -1
View File
@@ -16,6 +16,7 @@ env:
PYTHONLEGACYWINDOWSSTDIO: utf-8
USE_COLOR: False
UV_NO_SOURCES: "1"
ABXPKG_MIN_RELEASE_AGE: "0"
jobs:
discover-tests:
@@ -374,7 +375,6 @@ jobs:
- name: Run plugin tests - ${{ matrix.plugin.name }}
env:
ABXPKG_NO_CACHE: "true"
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
TWOCAPTCHA_API_KEY: ${{ secrets.TWOCAPTCHA_API_KEY }}
API_KEY_2CAPTCHA: ${{ secrets.TWOCAPTCHA_API_KEY }}
+38 -39
View File
@@ -29,7 +29,7 @@ Without active preservation effort, everything on the internet eventually disapp
*ArchiveBox is an open source tool that lets organizations & individuals archive both public & private web content while retaining control over their data. It can be used to save copies of bookmarks, preserve evidence for legal cases, backup photos from FB/Insta/Flickr or media from YT/Soundcloud/etc., save research papers, and more...*
<br/>
> ➡️ Get ArchiveBox with `pip install archivebox>=0.9.5rc1` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
> ➡️ Get ArchiveBox with `uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` on [Linux](#quickstart)/[macOS](#quickstart), or via **[Docker](#quickstart)** ⭐️ on any OS.
*Once installed, you can interact with it through the: [Browser Extension](https://github.com/ArchiveBox/archivebox-browser-extension), [CLI](#usage), [self-hosted web interface](https://github.com/ArchiveBox/ArchiveBox/wiki/Publishing-Your-Archive), [Python API](https://github.com/ArchiveBox/ArchiveBox/wiki/Usage#python-shell-usage), or [filesystem](#static-archive-exporting).*
@@ -52,7 +52,7 @@ It also detects any content featured *inside* pages & extracts it out into a fol
- 💾 **Github**/**Gitlab**/etc. links ➡️ `clone of GIT source code`, `README`, `images`, ...
- ✨ *and more, see [Output Formats](#output-formats) below...*
You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `pip`-installed CLI, Python API, and SQLite3 APIs.
You can run ArchiveBox as a Docker web app to manage these snapshots, or continue accessing the same collection using the `uv`-installed CLI, Python API, and SQLite3 APIs.
All the ways of using it are equivalent, and provide matching features like adding tags, scheduling regular crawls, viewing logs, and more...
<br/>
@@ -68,7 +68,7 @@ The goal is to sleep soundly knowing the part of the internet you care about wil
<br/>
**📦&nbsp; Install ArchiveBox using your preferred method: `docker` / `pip` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
**📦&nbsp; Install ArchiveBox using your preferred method: `docker` / `uv` / `brew` / `apt` / etc. ([see full Quickstart below](#quickstart)).**
<details>
@@ -85,14 +85,14 @@ docker compose run archivebox init
<br/>
# Option B: Or use it as a plain Docker container:
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -it -v $PWD:/data archivebox/archivebox init
# docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
# docker run -it -v $PWD:/data archivebox/archivebox help
# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox
docker run -it -v $PWD:/data archivebox/archivebox:dev init
# docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
# docker run -it -v $PWD:/data archivebox/archivebox:dev help
# docker run -it -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
<br/>
<br/>
# Option C: Or install it with your preferred pkg manager (see Quickstart below for apt, brew, and more)
pip install 'archivebox>=0.9.5rc1'
# Option C: Or install it with uv (see Quickstart below for apt, brew, and more)
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
archivebox install
@@ -214,14 +214,14 @@ See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the C
<li>Install <a href="https://docs.docker.com/get-docker/">Docker</a> on your system (if not already installed).</li>
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -v $PWD:/data -it archivebox/archivebox init
docker run -v $PWD:/data -it archivebox/archivebox:dev init
</code></pre>
</li>
<li>Optional: Start the server then login to the Web UI <a href="http://archivebox.localhost:8000">http://archivebox.localhost:8000</a> ⇢ Admin.
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox
<pre lang="bash"><code style="white-space: pre-line">docker run -v $PWD:/data -p 8000:8000 archivebox/archivebox:dev
# completely optional, CLI can always be used without running a server
# docker run -v $PWD:/data -it [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox help
docker run -v $PWD:/data -it archivebox/archivebox:dev help
</code></pre>
<i>For more info, see <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Install#option-a-docker--docker-compose-setup-%EF%B8%8F">Install: Docker Compose</a> in the Wiki. ➡️</i>
</li>
@@ -256,13 +256,13 @@ See <a href="https://docs.sweeting.me/s/against-curl-sh">"Against curl | sh as a
<details>
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="Pip" height="28px" align="top"/> <code>pip</code></b> (macOS/Linux/BSD)</summary>
<summary><b><img src="https://user-images.githubusercontent.com/511499/117447613-ba4c5d80-af0b-11eb-8f89-1d98e31b6a79.png" alt="uv" height="28px" align="top"/> <code>uv</code></b> (macOS/Linux/BSD)</summary>
<br/>
<ol>
<li>Install <a href="https://realpython.com/installing-python/">Python >= v3.13</a> and <a href="https://nodejs.org/en/download/package-manager/">Node >= v22</a> on your system (if not already installed).</li>
<li>Install the ArchiveBox package using <code>pip3</code> (or <a href="https://docs.astral.sh/uv/guides/tools/#running-tools"><code>uvx</code></a>).
<pre lang="bash"><code style="white-space: pre-line">pip3 install --upgrade archivebox
<li>Install <a href="https://docs.astral.sh/uv/getting-started/installation/">uv</a> on your system (if not already installed).</li>
<li>Install the ArchiveBox package using <code>uv</code>.
<pre lang="bash"><code style="white-space: pre-line">uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
archivebox version
# install any missing extras shown using apt/brew/pkg/etc. see Wiki for instructions
# python@3.13 node curl wget git ripgrep ...
@@ -305,7 +305,7 @@ archivebox version # make sure the package is installed
<li>Create a new empty directory and initialize your collection (can be anywhere).
<pre lang="bash"><code style="white-space: pre-line">mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
sudo archivebox install
archivebox install
archivebox add 'https://example.com'
</code></pre>
<br/>
@@ -319,7 +319,7 @@ archivebox help
</li>
</ol>
See <a href="#%EF%B8%8F-cli-usage">below</a> for more usage examples using the CLI, Web UI, or filesystem/SQL/Python to manage your archive.<br/>
<sub>See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for more details about this distribution.</sub>
<sub>The apt package is a thin dev-channel wrapper around the normal Python install flow. See the <a href="https://github.com/ArchiveBox/debian-archivebox"><code>debian-archivebox</code></a> repo for details, and run <code>sudo archivebox install</code> only if you want it to install missing system packages via apt.</sub>
<br/><br/>
</details>
@@ -503,8 +503,8 @@ archivebox help
# equivalent: docker compose run archivebox [subcommand] [--help]
docker compose run archivebox help
# equivalent: docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
docker run -it -v $PWD:/data archivebox/archivebox help
# equivalent: docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
docker run -it -v $PWD:/data archivebox/archivebox:dev help
# optional: import your browser cookies into a persona for logged-in archiving
archivebox persona create --import=chrome personal
@@ -525,12 +525,12 @@ archivebox persona create --import=chrome personal
<summary><img src="https://user-images.githubusercontent.com/511499/117456282-08665e80-af16-11eb-91a1-8102eff54091.png" alt="curl sh automatic setup script" height="22px" align="top"/> <b>CLI Usage Examples: non-Docker</b></summary>
<br/>
<pre lang="bash"><code style="white-space: pre-line">
# make sure you have pip-installed ArchiveBox and it's available in your $PATH first
# make sure you have uv-installed ArchiveBox and it's available in your $PATH first
<br/>
# archivebox [subcommand] [--help]
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
sudo archivebox install
archivebox install
archivebox add 'https://example.com'
archivebox version # get archivebox version info + check dependencies
archivebox help # get list of archivebox subcommands that can be run
@@ -565,13 +565,13 @@ docker compose run archivebox add 'https://example.com'
<pre lang="bash"><code style="white-space: pre-line">
# make sure you create and cd into in a new empty directory first
<br/>
# docker run -it -v $PWD:/data archivebox/archivebox [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox init
docker run -v $PWD:/data -it archivebox/archivebox install
docker run -v $PWD:/data -it archivebox/archivebox version
docker run -v $PWD:/data -it archivebox/archivebox help
docker run -v $PWD:/data -it archivebox/archivebox add 'https://example.com'
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
# docker run -it -v $PWD:/data archivebox/archivebox:dev [subcommand] [--help]
docker run -v $PWD:/data -it archivebox/archivebox:dev init
docker run -v $PWD:/data -it archivebox/archivebox:dev install
docker run -v $PWD:/data -it archivebox/archivebox:dev version
docker run -v $PWD:/data -it archivebox/archivebox:dev help
docker run -v $PWD:/data -it archivebox/archivebox:dev add 'https://example.com'
# to start webserver: docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
</code></pre>
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#usage-1">Usage: Docker CLI</a> wiki. ➡️</i>
</details>
@@ -603,8 +603,8 @@ nano docker-compose.yml # setup initial ADMIN_USERNAME &
docker compose up # start the server
<br/>
# Or with a Docker container:
docker run -v $PWD:/data -it archivebox/archivebox archivebox manage createsuperuser
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox
docker run -v $PWD:/data -it archivebox/archivebox:dev archivebox manage createsuperuser
docker run -v $PWD:/data -it -p 8000:8000 archivebox/archivebox:dev
</code></pre>
<sup>Open <a href="http://web.archivebox.localhost:8000"><code>http://web.archivebox.localhost:8000</code></a> for the public UI and <a href="http://admin.archivebox.localhost:8000"><code>http://admin.archivebox.localhost:8000</code></a> for the admin UI ➡️</sup><br/>
@@ -630,15 +630,15 @@ docker compose run archivebox config --set ...
> [!TIP]
> Whether in Docker or not, ArchiveBox commands work the same way, and can be used to access the same data on-disk.
> For example, you could run the Web UI in Docker Compose, and run one-off commands with `pip`-installed ArchiveBox.
> For example, you could run the Web UI in Docker Compose, and run one-off commands with `uv`-installed ArchiveBox.
<details>
<summary><i>Expand to show comparison...</i></summary><br/>
<pre lang="bash"><code style="white-space: pre-line">
archivebox add --depth=1 'https://example.com' # add a URL with pip-installed archivebox on the host
archivebox add --depth=1 'https://example.com' # add a URL with uv-installed archivebox on the host
docker compose run archivebox add --depth=1 'https://example.com' # or w/ Docker Compose
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com' # or w/ Docker, all equivalent
docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com' # or w/ Docker, all equivalent
</code></pre>
<i>For more info, see our <a href="https://github.com/ArchiveBox/ArchiveBox/wiki/Docker">Docker</a> wiki. ➡️</i>
@@ -731,7 +731,7 @@ echo 'http://example.com' | archivebox add
echo 'any text with <a href="https://example.com">urls</a> in it' | archivebox add
# if using Docker, add -i when piping stdin:
# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox add
# echo 'https://example.com' | docker run -v $PWD:/data -i archivebox/archivebox:dev add
# if using Docker Compose, add -T when piping stdin / stdout:
# echo 'https://example.com' | docker compose run -T archivebox add
```
@@ -882,10 +882,8 @@ You don't need to install every dependency to use ArchiveBox. ArchiveBox will au
If not using Docker, make sure to keep the dependencies up-to-date yourself and check that ArchiveBox isn't reporting any incompatibility with the versions you install.
<pre lang="bash"><code style="white-space: pre-line">#install python3 and archivebox with your system package manager
# apt/brew/pip/etc install ... (see Quickstart instructions above)
<pre lang="bash"><code style="white-space: pre-line"># install uv + archivebox first (see Quickstart instructions above)
<br/>
which -a archivebox # see where you have installed archivebox
archivebox install # auto install all the extractors and extras
archivebox --version # see info and check validity of installed dependencies
</code></pre>
@@ -1451,7 +1449,8 @@ archivebox server 0.0.0.0:8000
# inside the container will reload and pick up your changes
./bin/build_docker.sh dev
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init --install
docker run -it -v $PWD/data:/data archivebox/archivebox:dev init
docker run -it -v $PWD/data:/data archivebox/archivebox:dev install
# Run the development server w/ autoreloading (but no bg workers)
docker run -it -v $PWD/data:/data -v $PWD/archivebox:/app/archivebox -p 8000:8000 archivebox/archivebox:dev server --debug --reload 0.0.0.0:8000
+2
View File
@@ -54,6 +54,7 @@ class AddCommandSchema(Schema):
snapshot_max_size: int = 0
parser: str = "auto"
plugins: str = ""
persona: str = "Default"
only_new: bool | None = None
update: bool = False
overwrite: bool = False
@@ -145,6 +146,7 @@ def cli_add(request: HttpRequest, args: AddCommandSchema):
snapshot_max_size=args.snapshot_max_size,
index_only=args.index_only,
plugins=args.plugins,
persona=args.persona,
parser=args.parser,
bg=True, # Always run in background for API calls
created_by_id=request.user.pk,
+2 -1
View File
@@ -4,6 +4,7 @@ import math
import json
import mimetypes
import re
from html import unescape
from collections import defaultdict
from pathlib import Path, PurePosixPath
from uuid import UUID
@@ -832,7 +833,7 @@ def _snapshots_rss_response(
crawl_user = snapshot.crawl.created_by if snapshot.crawl_id else None
description = f"Original URL: {snapshot.url}\nArchived snapshot: {archived_url}"
feed.add_item(
title=snapshot.title or snapshot.url,
title=unescape(snapshot.title or snapshot.url),
link=archived_url or web_base_url,
description=description,
unique_id=str(snapshot.id),
+73 -6
View File
@@ -228,6 +228,11 @@ def version(
prnt()
in_data_dir = os.access(CONSTANTS.ARCHIVE_DIR, os.R_OK) and os.access(CONSTANTS.CONFIG_FILE, os.R_OK)
if isinstance(binaries, str):
requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
else:
requested_names = {name for name in (binaries or ()) if name}
if not in_data_dir:
PANEL_TEXT = "\n".join(
(
@@ -250,6 +255,22 @@ def version(
)
prnt()
if not in_data_dir and not requested_names:
prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
prnt("", "[grey53]Dependency checks require an initialized collection DATA_DIR.[/grey53]")
prnt()
prnt("[deep_sky_blue3][i] Code locations:[/deep_sky_blue3]")
try:
for name, path in get_code_locations().items():
if isinstance(name, str) and isinstance(path, dict):
prnt(printable_folder_status(name, path), overflow="ignore", crop=False)
except Exception as e:
prnt(f" [red]Error getting code locations: {e}[/red]")
prnt()
prnt("[red][i] Data locations:[/red] (not in a data directory)")
prnt()
return []
prnt("[pale_green1][i] Binary Dependencies:[/pale_green1]")
failures = []
seen_failures: set[str] = set()
@@ -260,11 +281,6 @@ def version(
from abx_dl.dependencies import load_binary
from abx_dl.models import discover_plugins, filter_plugins
if isinstance(binaries, str):
requested_names = {name.strip() for name in binaries.split(",") if name.strip()}
else:
requested_names = {name for name in (binaries or ()) if name}
plugins = discover_plugins(runtime="archivebox")
enabled_plugins = filter_plugins(plugins, get_enabled_plugins(config=config), include_providers=True)
enabled_plugin_names = set(enabled_plugins)
@@ -339,13 +355,19 @@ def version(
):
continue
any_rows = True
installed = db_binaries.get(logical_name) if db_available else None
if _binary_record_matches_runtime(installed, config.ABXPKG_LIB_DIR):
abspath = installed.abspath
version_str = (installed.version or "unknown")[:15]
provider = (installed.binprovider or "env")[:8]
valid = True
elif not plugin_enabled and not requested_names:
# `archivebox version` is expected to verify the active runtime, not
# cold-load every optional plugin provider. Migration and status
# checks often run with PLUGINS narrowed to a tiny set; resolving
# disabled plugin binaries there can spend most of the command on
# providers the current collection will never execute.
continue
else:
loaded = load_binary(actual_record)
abspath = str(loaded.loaded_abspath or "")
@@ -353,6 +375,7 @@ def version(
provider = (loaded.loaded_binprovider.name if loaded.loaded_binprovider else "env")[:8]
valid = loaded.is_valid
any_rows = True
if valid:
display_path = (
_format_binary_abspath(
@@ -398,6 +421,50 @@ def version(
"style": "" if plugin_enabled else "dim",
},
)
if db_available:
for binary_name, installed in db_binaries.items():
if requested_names and binary_name not in requested_names:
continue
abspath = installed.abspath
version_str = (installed.version or "unknown")[:15]
provider = (installed.binprovider or "env")[:8]
row_key = _binary_row_dedupe_key(
display_name=binary_name,
valid=True,
version=version_str,
provider=provider,
abspath=abspath,
)
if row_key in seen_rows:
continue
seen_rows.add(row_key)
display_path = (
_format_binary_abspath(
abspath,
pwd=Path.cwd(),
lib_dir=config.ABXPKG_LIB_DIR,
personas_dir=CONSTANTS.PERSONAS_DIR,
home=Path.home(),
)
if compact_paths
else abspath
)
emit_row(
{
"plugin": "(database)",
"state": "detected",
"status": "[green]√[/green]",
"binary": binary_name,
"version": version_str,
"provider": provider,
"path": _render_binary_abspath(display_path) if compact_paths else display_path,
"style": "dim",
},
)
any_rows = True
any_available = True
finally:
if live_cm is not None:
live_cm.stop()
+79 -29
View File
@@ -507,6 +507,32 @@ def _plugin_user_config(config: Mapping[str, object]) -> dict[str, str]:
return {key: _plugin_user_config_value(value) for key, value in config.items()}
def _plugin_input_config(config: Mapping[str, object]) -> dict[str, object]:
"""Return raw config inputs understood by core or plugin schemas."""
input_names = _archivebox_config_input_names()
return {str(key): value for key, value in config.items() if str(key) in input_names}
def _explicit_plugin_enabled_keys(config: Mapping[str, object]) -> set[str]:
"""Return canonical ``*_ENABLED`` keys explicitly provided by the user.
``PLUGINS`` is a selector, not an override for concrete plugin enable flags.
If a user passes ``PLUGINS=wget`` and ``SAVE_WGET=False`` together, the
alias-resolved ``WGET_ENABLED=False`` must survive the model validator that
expands ``PLUGINS`` into concrete booleans.
"""
provided = {str(key) for key in config}
explicit: set[str] = set()
for key, prop in _plugin_config_properties(PLUGIN_CONFIG_SCHEMAS).items():
if not key.endswith("_ENABLED") or not isinstance(prop, Mapping):
continue
aliases = prop.get("x-aliases") if isinstance(prop, Mapping) else None
input_names = {key, *(str(alias) for alias in aliases if str(alias).strip())} if isinstance(aliases, list) else {key}
if provided & input_names:
explicit.add(key)
return explicit
def _discover_plugin_config_schemas() -> PluginSchemaDocuments:
from archivebox.plugins.discovery import discover_plugin_configs
@@ -664,7 +690,7 @@ class ArchiveBoxBaseConfig(
def _scoped_config(self, *, include_execution: bool) -> dict[str, Any]:
keys = type(self)._crawl_runtime_keys() if include_execution else type(self)._crawl_frozen_keys()
payload = self.model_dump(mode="json")
return {key: payload[key] for key in keys if payload.get(key) is not None}
return {key: payload[key] for key in keys if payload.get(key, None) is not None}
def for_crawl(self) -> dict[str, Any]:
"""Config scoped to crawl execution, without runtime object overlays."""
@@ -752,15 +778,35 @@ class ArchiveBoxBaseConfig(
@model_validator(mode="after")
def derive_plugin_enabled_config(self):
self._derive_plugin_enabled_config(respect_current_enabled=False)
return self
def _derive_plugin_enabled_config(self, *, respect_current_enabled: bool) -> None:
plugin_names = _normalize_plugins_config_value(self.PLUGINS)
selected_plugins = _plugins_with_required_plugins(plugin_names) if plugin_names else set()
enabled_config_keys = _plugin_enabled_config_keys()
disabled_plugins = []
if respect_current_enabled:
disabled_plugins = [plugin_name for plugin_name, enabled_key in enabled_config_keys.items() if not getattr(self, enabled_key)]
selected_plugin_roots = plugin_names
if selected_plugin_roots:
from abx_dl.models import discover_plugins, filter_plugins
selected_plugins = set(
filter_plugins(
discover_plugins(runtime="archivebox"),
sorted(selected_plugin_roots),
include_providers=True,
disabled_names=disabled_plugins,
),
)
else:
selected_plugins = set()
search_backend = self.SEARCH_BACKEND_ENGINE.strip().lower()
if search_backend:
selected_plugins.add(f"search_backend_{search_backend}")
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
for plugin_name, enabled_key in enabled_config_keys.items():
if plugin_names or plugin_name in selected_plugins:
setattr(self, enabled_key, plugin_name in selected_plugins)
return self
def _build_archivebox_config_model(plugin_schemas: PluginSchemaDocuments) -> type[ArchiveBoxBaseConfig]:
@@ -817,23 +863,6 @@ def _plugin_enabled_config_keys() -> dict[str, str]:
return enabled_keys
def _plugins_with_required_plugins(plugin_names: set[str]) -> set[str]:
selected = set(plugin_names)
pending = list(selected)
while pending:
plugin_name = pending.pop()
schema = PLUGIN_CONFIG_SCHEMAS.get(plugin_name, {})
required_plugins = schema.get("required_plugins") if isinstance(schema, dict) else None
if not isinstance(required_plugins, list):
continue
for required_plugin in required_plugins:
required_plugin_name = str(required_plugin).strip().lower()
if required_plugin_name and required_plugin_name not in selected:
selected.add(required_plugin_name)
pending.append(required_plugin_name)
return selected
def get_live_config_url(key: str) -> str:
return f"{LIVE_CONFIG_BASE_URL}{quote(key)}/"
@@ -1007,9 +1036,11 @@ def get_config(
config_data: ConfigPayload = dict(defaults or {})
config_data["PERSONAS_DIR"] = str(CONSTANTS.PERSONAS_DIR)
base_config_payload: ConfigPayload = {}
base_config_model = ArchiveBoxConfig()
if crawl_config_base:
config_data.update(
normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
)
config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
elif base_config is not None:
@@ -1020,7 +1051,7 @@ def get_config(
config_data.update(normalize_runtime_config(base_config_payload, exclude_runtime_derived=True, json_safe=False))
else:
config_data.update(
normalize_runtime_config(ArchiveBoxConfig().model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
normalize_runtime_config(base_config_model.model_dump(mode="json"), exclude_runtime_derived=True, json_safe=False),
)
legacy_permissions = permissions_from_legacy_public_flags({**BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE), **os.environ})
if legacy_permissions:
@@ -1072,22 +1103,29 @@ def get_config(
archivebox_scope_overrides = {key: value for key, value in scope_overrides.items() if key in _archivebox_config_input_names()}
config_data.update(archivebox_scope_overrides)
explicit_plugin_enabled_keys: set[str] = set()
if resolve_plugins:
plugin_schemas = {plugin_name: schema for plugin_name, schema in PLUGIN_CONFIG_SCHEMAS.items() if isinstance(schema, dict)}
plugin_global_config = {key: str(value) if isinstance(value, Path) else value for key, value in config_data.items()}
crawl_selected_plugins = crawl_config_base and bool(_normalize_plugins_config_value(dict(crawl.config or {}).get("PLUGINS")))
# A frozen crawl-level PLUGINS selector is the exact extractor set for
# this crawl. Env/file *_ENABLED values are ambient defaults and must
# not be re-applied after validation, or a later process env can change
# which hooks an already-queued crawl runs.
if not crawl_selected_plugins:
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(os.environ))
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(scope_overrides))
plugin_user_config = _plugin_user_config(
{
**normalize_runtime_config(config_data, only_crawl_execution=True, json_safe=False),
**_plugin_input_config(os.environ),
**scope_overrides,
},
)
if not crawl_config_base:
file_config = BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE)
explicit_plugin_enabled_keys.update(_explicit_plugin_enabled_keys(file_config))
plugin_user_config = {
**normalize_runtime_config(
BaseConfigSet.load_from_file(CONSTANTS.CONFIG_FILE),
exclude_runtime_derived=True,
json_safe=False,
),
**_plugin_user_config(_plugin_input_config(file_config)),
**plugin_user_config,
}
plugin_sections = resolve_plugin_configs(
@@ -1112,6 +1150,13 @@ def get_config(
if crawl_config_base:
config_data.update(normalize_runtime_config(dict(crawl.config or {}), exclude_crawl_execution=True, json_safe=False))
config_data.update(archivebox_scope_overrides)
if crawl_selected_plugins:
# Crawl PLUGINS is a frozen selector. Drop ambient enable flags from
# env/file/defaults before model validation so the selected plugin
# set is derived from the crawl, not from the worker process that
# happens to resume it.
for enabled_key in _plugin_enabled_config_keys().values():
config_data.pop(enabled_key, None)
# Decode JSON-encoded complex values (dict/list fields) that came from
# string-only sources before validation. ``IniConfigSettingsSource`` does
@@ -1137,6 +1182,11 @@ def get_config(
)
config = ArchiveBoxConfig.model_validate(config_data)
for key in explicit_plugin_enabled_keys:
if key in config_data:
setattr(config, key, config_data[key])
if config.PLUGINS:
config._derive_plugin_enabled_config(respect_current_enabled=True)
if redact_sensitive:
for key in type(config).model_fields:
value = config[key]
+5 -3
View File
@@ -260,13 +260,15 @@ class SnapshotChangeList(SearchResultsChangeList):
rows = (
ArchiveResult.objects.filter(snapshot_id__in=snapshot_ids, status=ArchiveResult.StatusChoices.SUCCEEDED, output_size__gt=0)
.order_by("snapshot_id", "plugin")
.values_list("snapshot_id", "plugin", "status", "output_size")
.values_list("snapshot_id", "plugin", "status", "output_size", "output_files")
)
for snapshot_id, plugin, status, output_size in rows.iterator(chunk_size=1000):
for snapshot_id, plugin, status, output_size, output_files in rows.iterator(chunk_size=1000):
if plugin in seen_plugins[snapshot_id]:
continue
seen_plugins[snapshot_id].add(plugin)
results_by_snapshot[snapshot_id].append(SimpleNamespace(plugin=plugin, status=status, output_size=output_size))
results_by_snapshot[snapshot_id].append(
SimpleNamespace(plugin=plugin, status=status, output_size=output_size, output_files=output_files),
)
for obj in self.result_list:
obj.__dict__["_admin_archiveresults"] = results_by_snapshot[obj.pk]
-14
View File
@@ -93,7 +93,6 @@ from archivebox.core.routes_util import (
from archivebox.core.forms import AddLinkForm
from archivebox.plugins.forms import get_plugin_config_binary_urls
from archivebox.crawls.models import Crawl
from archivebox.plugins.discovery import discover_plugin_configs
from archivebox.plugins.views import get_config_definition_link
from archivebox.progressmonitor.views import live_progress_view, progress_endpoint
@@ -1424,7 +1423,6 @@ class AddView(UserPassesTestMixin, FormView):
request_config = get_request_config(self.request, resolve_plugins=True)
required_search_plugin = f"search_backend_{request_config.SEARCH_BACKEND_ENGINE}".strip()
can_override_crawl_config = self._can_override_crawl_config()
plugin_configs = discover_plugin_configs() if can_override_crawl_config else {}
public_persona_config_keys = {
"CRAWL_MAX_CONCURRENT_SNAPSHOTS",
"DELETE_AFTER",
@@ -1453,17 +1451,6 @@ class AddView(UserPassesTestMixin, FormView):
"binary_urls": binary_urls,
}
recent_personas = list(persona_queryset.order_by("-created_at", "name")[:5])
plugin_dependency_map = {}
if can_override_crawl_config:
plugin_dependency_map = {
plugin_name: [
str(required_plugin).strip()
for required_plugin in (schema.get("required_plugins") or [])
if str(required_plugin).strip()
]
for plugin_name, schema in plugin_configs.items()
if isinstance(schema.get("required_plugins"), list) and schema.get("required_plugins")
}
return {
**context,
"title": "Create Crawl",
@@ -1473,7 +1460,6 @@ class AddView(UserPassesTestMixin, FormView):
"VERSION": VERSION,
"FOOTER_INFO": request_config.FOOTER_INFO,
"required_search_plugin": required_search_plugin,
"plugin_dependency_map_json": json.dumps(plugin_dependency_map, sort_keys=True),
"persona_config_map_json": json.dumps(persona_config_map, sort_keys=True, default=str),
"recent_personas": recent_personas,
"can_override_crawl_config": can_override_crawl_config,
+61 -9
View File
@@ -68,6 +68,55 @@ PERSONA_PROFILE_DIR_CANDIDATES = (
)
def _path_exists(path: Path) -> bool:
try:
return path.exists()
except OSError:
return False
def _path_is_dir(path: Path) -> bool:
try:
return path.is_dir()
except OSError:
return False
def _iter_children(path: Path) -> list[Path]:
try:
return list(path.iterdir())
except OSError:
return []
def _is_copyable_profile_entry(path: Path) -> bool:
try:
if path.is_symlink():
return True
if path.is_dir():
return os.access(path, os.R_OK | os.X_OK)
return os.access(path, os.R_OK)
except OSError:
return False
def profile_copy_ignore(src: str, names: list[str]) -> set[str]:
ignored = set(shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS)(src, names))
src_path = Path(src)
for name in names:
if name in ignored:
continue
if not _is_copyable_profile_entry(src_path / name):
# Chrome can leave generated profile components unreadable when a
# previous run used a different runtime uid, a mounted profile came
# from the host, or Chrome tightened permissions on cache/model
# directories. Those entries are not required to launch a cloned
# profile, and letting copytree abort here prevents the crawl from
# even creating its first Snapshot.
ignored.add(name)
return ignored
@dataclass(frozen=True)
class PersonaImportSource:
kind: str
@@ -394,13 +443,16 @@ def discover_persona_template_profiles(personas_dir: Path | None = None) -> list
continue
seen_roots.add(resolved_root)
if not resolved_root.exists() or not resolved_root.is_dir():
if not _path_is_dir(resolved_root):
continue
for persona_dir in sorted((path for path in resolved_root.iterdir() if path.is_dir()), key=lambda path: path.name.lower()):
for persona_dir in sorted(
(path for path in _iter_children(resolved_root) if _path_is_dir(path)),
key=lambda path: path.name.lower(),
):
for candidate_dir_name in PERSONA_PROFILE_DIR_CANDIDATES:
user_data_dir = persona_dir / candidate_dir_name
if not user_data_dir.exists() or not user_data_dir.is_dir():
if not _path_is_dir(user_data_dir):
continue
for profile_dir in _list_profile_names(user_data_dir):
@@ -594,7 +646,7 @@ def copy_browser_user_data_dir(source_dir: Path, destination_dir: Path) -> None:
source_dir,
destination_dir,
symlinks=True,
ignore=shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS),
ignore=profile_copy_ignore,
)
@@ -707,12 +759,12 @@ def export_browser_state(
def _list_profile_names(user_data_dir: Path) -> list[str]:
if not user_data_dir.exists() or not user_data_dir.is_dir():
if not _path_is_dir(user_data_dir):
return []
profiles: list[str] = []
for child in sorted(user_data_dir.iterdir(), key=lambda path: path.name.lower()):
if not child.is_dir():
for child in sorted(_iter_children(user_data_dir), key=lambda path: path.name.lower()):
if not _path_is_dir(child):
continue
if child.name == "System Profile":
continue
@@ -726,7 +778,7 @@ def _list_profile_names(user_data_dir: Path) -> list[str]:
def _looks_like_profile_dir(path: Path) -> bool:
if not path.exists() or not path.is_dir():
if not _path_is_dir(path):
return False
marker_paths = (
@@ -738,7 +790,7 @@ def _looks_like_profile_dir(path: Path) -> bool:
path / "Session Storage",
)
if any(marker.exists() for marker in marker_paths):
if any(_path_exists(marker) for marker in marker_paths):
return True
return any(path.name == prefix or path.name.startswith(prefix) for prefix in BROWSER_PROFILE_DIR_NAMES)
+24 -6
View File
@@ -11,6 +11,7 @@ Each persona has its own:
__package__ = "archivebox.personas"
import shutil
import os
from contextlib import contextmanager
from pathlib import Path
from typing import TYPE_CHECKING, Any
@@ -179,7 +180,16 @@ class Persona(ModelWithConfig):
if not profile_dir.exists():
return False
for path in profile_dir.rglob("*"):
def profile_paths():
# Chrome profiles are user-controlled filesystem state. If a cache
# or generated component is unreadable, cleanup should skip it and
# continue pruning the rest instead of blocking the crawl runner.
for dirpath, dirnames, filenames in os.walk(profile_dir, onerror=lambda _err: None):
current_dir = Path(dirpath)
for name in [*dirnames, *filenames]:
yield current_dir / name
for path in profile_paths():
if path.name in VOLATILE_PROFILE_FILE_NAMES:
try:
path.unlink()
@@ -188,13 +198,21 @@ class Persona(ModelWithConfig):
pass
for dirname in VOLATILE_PROFILE_DIR_NAMES:
for path in profile_dir.rglob(dirname):
if not path.is_dir():
for path in profile_paths():
if path.name != dirname:
continue
try:
is_dir = path.is_dir()
except OSError:
is_dir = False
if not is_dir:
continue
shutil.rmtree(path, ignore_errors=True)
cleaned = True
for path in profile_dir.rglob("*.log"):
for path in profile_paths():
if not path.match("*.log"):
continue
try:
path.unlink()
cleaned = True
@@ -252,7 +270,7 @@ class Persona(ModelWithConfig):
return self.runtime_root_for_snapshot(snapshot) / "chrome_downloads"
def copy_chrome_profile(self, source_dir: Path, destination_dir: Path) -> None:
from archivebox.personas.importers import VOLATILE_PROFILE_COPY_PATTERNS
from archivebox.personas.importers import profile_copy_ignore
destination_dir.parent.mkdir(parents=True, exist_ok=True)
shutil.rmtree(destination_dir, ignore_errors=True)
@@ -260,7 +278,7 @@ class Persona(ModelWithConfig):
source_dir,
destination_dir,
symlinks=True,
ignore=shutil.ignore_patterns(*VOLATILE_PROFILE_COPY_PATTERNS),
ignore=profile_copy_ignore,
)
def prepare_runtime_for_crawl(self, crawl, chrome_binary: str = "") -> dict[str, str]:
+21 -79
View File
@@ -64,6 +64,15 @@ def get_plugins() -> list[str]:
return sorted(set(plugins))
def get_plugin_models():
from abx_dl.models import discover_plugins
plugins = {}
for base_dir in (BUILTIN_PLUGINS_DIR, USER_PLUGINS_DIR):
plugins.update(discover_plugins(plugins_dir=base_dir, runtime="archivebox"))
return plugins
def get_plugin_name(plugin: str) -> str:
"""
Get the base plugin name without numeric prefix.
@@ -90,36 +99,18 @@ def get_enabled_plugins(config: ConfigLookup | None = None, **config_kwargs: Any
config = get_config(**config_kwargs)
def normalize_enabled_plugins(value: Any) -> list[str]:
if value is None:
return []
if isinstance(value, str):
raw = value.strip()
if not raw:
return []
if raw.startswith("["):
try:
parsed = json.loads(raw)
except json.JSONDecodeError:
parsed = None
if isinstance(parsed, list):
return [str(plugin).strip() for plugin in parsed if str(plugin).strip()]
return [plugin.strip() for plugin in raw.split(",") if plugin.strip()]
if isinstance(value, (list, tuple, set)):
return [str(plugin).strip() for plugin in value if str(plugin).strip()]
return [str(value).strip()] if str(value).strip() else []
plugins_override = config.get("PLUGINS")
if plugins_override:
return normalize_enabled_plugins(plugins_override)
enabled = []
disabled = []
for plugin in get_plugins():
plugin_config = get_plugin_special_config(plugin, config)
if plugin_config["enabled"]:
enabled.append(plugin)
else:
disabled.append(plugin)
return enabled
from abx_dl.models import filter_plugins
return list(filter_plugins(get_plugin_models(), enabled, include_providers=True, disabled_names=disabled))
def discover_plugins_that_provide_interface(
@@ -245,61 +236,12 @@ def get_plugin_special_config(plugin_name: str, config: ConfigLookup, _visited:
"""
plugin_upper = plugin_name.upper()
plugins_whitelist = config.get("PLUGINS", "")
if plugins_whitelist:
plugin_configs = discover_plugin_configs()
plugin_names = {p.strip().lower() for p in plugins_whitelist.split(",") if p.strip()}
pending = list(plugin_names)
while pending:
current = pending.pop()
schema = plugin_configs.get(current, {})
required_plugins = schema.get("required_plugins", [])
if not isinstance(required_plugins, list):
continue
for required_plugin in required_plugins:
required_plugin_name = str(required_plugin).strip().lower()
if not required_plugin_name or required_plugin_name in plugin_names:
continue
plugin_names.add(required_plugin_name)
pending.append(required_plugin_name)
if plugin_name.lower() not in plugin_names:
enabled = False
else:
enabled_key = f"{plugin_upper}_ENABLED"
enabled = config.get(enabled_key)
if enabled is None:
enabled = True
elif isinstance(enabled, str):
enabled = enabled.lower() not in ("false", "0", "no", "")
else:
enabled_key = f"{plugin_upper}_ENABLED"
enabled = config.get(enabled_key)
if enabled is None:
enabled = True
elif isinstance(enabled, str):
enabled = enabled.lower() not in ("false", "0", "no", "")
plugin_configs = discover_plugin_configs()
plugin_name_lower = plugin_name.lower()
if enabled:
visited = _visited or set()
if plugin_name_lower not in visited:
next_visited = visited | {plugin_name_lower}
schema = plugin_configs.get(plugin_name_lower, {})
required_plugins = schema.get("required_plugins", [])
if isinstance(required_plugins, list):
for required_plugin in required_plugins:
required_plugin_name = str(required_plugin).strip()
if not required_plugin_name:
continue
required_config = get_plugin_special_config(required_plugin_name, config, _visited=next_visited)
if not required_config["enabled"]:
enabled = False
break
enabled_key = f"{plugin_upper}_ENABLED"
enabled = config.get(enabled_key)
if enabled is None:
enabled = True
elif isinstance(enabled, str):
enabled = enabled.lower() not in ("false", "0", "no", "")
timeout_key = f"{plugin_upper}_TIMEOUT"
timeout = config.get(timeout_key) or config.get("TIMEOUT", 300)
+22 -6
View File
@@ -55,6 +55,7 @@ from archivebox.plugins.discovery import (
BUILTIN_PLUGINS_DIR,
USER_PLUGINS_DIR,
ConfigLookup,
get_enabled_plugins,
get_plugin_special_config,
)
@@ -215,6 +216,7 @@ def discover_hooks(
config = get_config(**config_kwargs)
enabled_plugins = set(get_enabled_plugins(config=config))
enabled_hooks = []
for hook in hooks:
@@ -229,9 +231,7 @@ def discover_hooks(
enabled_hooks.append(hook)
continue
# Check if plugin is enabled
plugin_config = get_plugin_special_config(plugin_name, config)
if plugin_config["enabled"]:
if plugin_name in enabled_plugins:
enabled_hooks.append(hook)
hooks = enabled_hooks
@@ -279,15 +279,27 @@ def run_hook(
records = process.get_records() # Get parsed JSONL output
"""
from archivebox.machine.models import Process, Machine, NetworkInterface
from archivebox.config.common import ArchiveBoxConfig, _archivebox_config_input_names, get_config, normalize_runtime_config
from archivebox.config.common import (
ArchiveBoxConfig,
_archivebox_config_input_names,
get_config,
normalize_runtime_config,
_plugin_enabled_config_keys,
)
config_scope = {key.removeprefix("config_"): kwargs.pop(key) for key in list(kwargs) if key.startswith("config_")}
config_overrides = _config_to_overrides(config)
explicit_override_keys = set(config_overrides)
resolved_config = get_config(overrides=config_overrides, **config_scope)
hook_config = normalize_runtime_config(
resolved_config.for_crawl_runtime(),
resolved_config.for_crawl_runtime(runtime_overrides=config_overrides),
json_safe=False,
)
hook_config.update(normalize_runtime_config(config_overrides, json_safe=False))
plugin_enabled_keys = set(_plugin_enabled_config_keys().values())
if plugin_enabled_keys.intersection(hook_config):
for enabled_key in plugin_enabled_keys:
hook_config.setdefault(enabled_key, False)
# Auto-detect timeout from plugin config if not explicitly provided
if timeout is None:
@@ -368,8 +380,10 @@ def run_hook(
archivebox_config_input_names = _archivebox_config_input_names()
for key in archivebox_config_input_names:
env.pop(key, None)
env.pop("PLUGINS", None)
env["DATA_DIR"] = str(CONSTANTS.DATA_DIR)
env["LIBRARY_VERSION"] = VERSION
env.setdefault("ABXPKG_FAST_SCRIPT", "1")
env.setdefault("MACHINE_ID", os.environ.get("MACHINE_ID", CONSTANTS.MACHINE_ID))
snap_dir = hook_config.get("SNAP_DIR") or _model_output_dir_from_child_path(output_dir, CONSTANTS.SNAPSHOTS_DIR_NAME)
crawl_dir = hook_config.get("CRAWL_DIR") or _model_output_dir_from_child_path(output_dir, CONSTANTS.CRAWLS_DIR_NAME)
@@ -386,7 +400,9 @@ def run_hook(
# Set Node.js module resolution paths.
# NODE_PATH may be a path list, but NODE_MODULES_DIR is a single canonical directory.
node_modules_dir = hook_config.get("NODE_MODULES_DIR")
if not node_modules_dir and lib_dir:
if lib_dir and "ABXPKG_LIB_DIR" in explicit_override_keys and "NODE_MODULES_DIR" not in explicit_override_keys:
node_modules_dir = Path(lib_dir) / "pnpm" / "packages" / "chrome" / "node_modules"
elif not node_modules_dir and lib_dir:
node_modules_dir = Path(lib_dir) / "pnpm" / "packages" / "chrome" / "node_modules"
node_path_parts = [part for part in str(hook_config.get("NODE_PATH") or "").split(os.pathsep) if part]
@@ -4,6 +4,7 @@ import asyncio
import inspect
import json
import os
import signal
import sys
import time
from collections import defaultdict
@@ -240,6 +241,14 @@ def _should_update_snapshot_title(current_title: str, next_title: str, *, snapsh
def _status_for_process_without_archive_result(event: ProcessCompletedEvent) -> str:
if event.exit_code == PROCESS_EXIT_SKIPPED:
return "skipped"
if event.exit_code in {128 + signal.SIGHUP, 128 + signal.SIGINT, 128 + signal.SIGTERM}:
# This fallback only runs when a snapshot hook exited before emitting a
# structured ArchiveResult. A polite shutdown signal means the runner
# was interrupted during ownership transfer, not that the extractor
# produced a durable negative result. Keep the hook queued so the next
# runner can retry the exact work item instead of sealing in a transient
# process-lifecycle failure.
return "queued"
if event.exit_code != 0:
return "failed"
return "noresults"
+27 -3
View File
@@ -31,6 +31,27 @@ def current_network_interface_with_machine():
return NetworkInterface.objects.select_related("machine").get(id=current_iface.id)
def normalize_process_env(env: dict) -> dict:
normalized = dict(env or {})
raw_plugins = normalized.pop("PLUGINS", "")
selected_plugins = {name.strip().lower() for name in str(raw_plugins).split(",") if name.strip()}
from archivebox.config.common import ArchiveBoxConfig, _archivebox_config_input_names, is_sensitive_config_key
allowed_config_keys = ArchiveBoxConfig._crawl_runtime_keys()
config_input_names = _archivebox_config_input_names()
for key in list(normalized):
if is_sensitive_config_key(key) or (key in config_input_names and key not in allowed_config_keys):
normalized.pop(key, None)
if selected_plugins:
from abx_dl.models import discover_plugins, filter_plugins
from archivebox.config.common import _plugin_enabled_config_keys
selected_plugins = set(filter_plugins(discover_plugins(runtime="archivebox"), sorted(selected_plugins), include_providers=True))
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
normalized.setdefault(enabled_key, "True" if plugin_name in selected_plugins else "False")
return normalized
class ProcessService(BaseService):
LISTENS_TO: ClassVar[list[type[BaseEvent]]] = [
ProcessStartedEvent,
@@ -74,6 +95,7 @@ class ProcessService(BaseService):
started_at=started_at,
)
process = await process_query.order_by("-modified_at").afirst()
process_env = normalize_process_env(event.env)
if process is None:
process = await Process.objects.acreate(
machine=iface.machine,
@@ -82,7 +104,7 @@ class ProcessService(BaseService):
worker_type=worker_type,
pwd=event.output_dir,
cmd=[event.hook_path, *event.hook_args],
env=event.env,
env=process_env,
timeout=event.timeout,
pid=event.pid or None,
url=event.url or None,
@@ -97,7 +119,7 @@ class ProcessService(BaseService):
process.pwd = event.output_dir
process.cmd = [event.hook_path, *event.hook_args]
process.env = event.env
process.env = process_env
process.timeout = event.timeout
process.pid = event.pid or None
process.url = event.url or process.url
@@ -172,6 +194,7 @@ class ProcessService(BaseService):
started_at=started_at,
)
process = await process_query.order_by("-modified_at").afirst()
process_env = normalize_process_env(event.env)
if process is None:
await Process.objects.acreate(
machine=iface.machine,
@@ -180,7 +203,7 @@ class ProcessService(BaseService):
worker_type=worker_type,
pwd=event.output_dir,
cmd=[event.hook_path, *event.hook_args],
env=event.env,
env=process_env,
timeout=event.timeout,
pid=event.pid or None,
url=event.url or None,
@@ -197,6 +220,7 @@ class ProcessService(BaseService):
"machine_id": iface.machine_id,
"iface_id": iface.id,
"pwd": event.output_dir,
"env": process_env,
"pid": event.pid or process.pid,
"url": event.url or process.url,
"process_type": process_type or process.process_type,
+66 -3
View File
@@ -55,7 +55,11 @@ from abxbus import BaseEvent
from abxbus.event_bus import EventBus, get_current_event, in_handler_context
from abxbus.event_handler import EventHandlerAbortedError, EventHandlerCancelledError
from archivebox.config.common import ArchiveBoxBaseConfig, normalize_runtime_config
from archivebox.config.common import (
ArchiveBoxBaseConfig,
normalize_runtime_config,
_plugin_enabled_config_keys,
)
from archivebox.misc.db import run_db_analyze_batch
from archivebox.core.shutdown_util import foreground_shutdown_signals, raise_if_shutdown_requested
from archivebox.search.sonic_daemon import register_sonic_daemon_event_handler
@@ -121,6 +125,10 @@ def _count_selected_hooks(plugins: dict[str, Plugin], selected_plugins: list[str
return sum(1 for plugin in selected.values() for hook in plugin.hooks if "CrawlSetup" in hook.name or "Snapshot" in hook.name)
def _is_nonfatal_setup_hook(plugin_name: str, hook_name: str) -> bool:
return plugin_name == "chrome" and hook_name.endswith("_chrome_kill_zombies")
def _discover_archivebox_plugins() -> dict[str, Plugin]:
return discover_plugins(runtime="archivebox")
@@ -800,6 +808,11 @@ class CrawlRunner:
},
)
normalized_config = normalize_runtime_config(config)
configured_plugins = [name.strip().lower() for name in str(normalized_config.get("PLUGINS") or "").split(",") if name.strip()]
if configured_plugins:
selected_plugin_names = set(filter_plugins(self.plugins, configured_plugins, include_providers=True))
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
normalized_config.setdefault(enabled_key, plugin_name in selected_plugin_names)
return {
"id": str(snapshot.id),
"url": snapshot.url,
@@ -1081,6 +1094,8 @@ class CrawlRunner:
await completed_process.wait(timeout=crawl_setup_phase_timeout)
await completed_process.event_results_list()
if completed_process.status == "failed":
if _is_nonfatal_setup_hook(plugin.name, hook.name):
continue
raise RuntimeError(f"Crawl setup hook {plugin.name}:{hook.name} failed")
async def run_snapshot(self, snapshot_id: str, crawl_start_event: CrawlStartEvent | None = None) -> None:
@@ -1130,7 +1145,14 @@ class CrawlRunner:
)(snapshot["id"])
if queued_plugins:
if snapshot_selected_plugins:
original_queued_plugins = queued_plugins
queued_plugins = queued_plugins_selected_by_config(queued_plugins)
disabled_queued_plugins = sorted(set(original_queued_plugins) - set(queued_plugins))
if disabled_queued_plugins:
await sync_to_async(skip_disabled_queued_plugins, thread_sensitive=True)(
snapshot["id"],
disabled_queued_plugins,
)
selected_hooks_by_plugin = {
plugin: hooks for plugin, hooks in (selected_hooks_by_plugin or {}).items() if plugin in queued_plugins
}
@@ -1142,7 +1164,14 @@ class CrawlRunner:
)(snapshot["id"])
if queued_plugins:
if snapshot_selected_plugins:
original_queued_plugins = queued_plugins
queued_plugins = queued_plugins_selected_by_config(queued_plugins)
disabled_queued_plugins = sorted(set(original_queued_plugins) - set(queued_plugins))
if disabled_queued_plugins:
await sync_to_async(skip_disabled_queued_plugins, thread_sensitive=True)(
snapshot["id"],
disabled_queued_plugins,
)
selected_hooks_by_plugin = {
plugin: hooks for plugin, hooks in (selected_hooks_by_plugin or {}).items() if plugin in queued_plugins
}
@@ -1173,7 +1202,13 @@ class CrawlRunner:
if snapshot_selected_plugins and remaining_queued_plugins:
remaining_queued_plugins = queued_plugins_selected_by_config(remaining_queued_plugins)
if not remaining_queued_plugins:
await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
if snapshot["status"] in ("queued", "started"):
await sync_to_async(finalize_completed_snapshot, thread_sensitive=True)(
snapshot_id,
output_dir=output_dir,
)
else:
await sync_to_async(run_snapshot_maintenance, thread_sensitive=True)(snapshot_id, output_dir=output_dir)
return
snapshot_selected_plugins = remaining_queued_plugins
plugins = filter_plugins(self.plugins, snapshot_selected_plugins, include_providers=True)
@@ -1437,6 +1472,11 @@ def queued_plugins_for_snapshot(snapshot_id: str) -> list[str] | None:
def config_overrides_for_queued_plugins(selected_plugins: list[str], **overrides: Any) -> dict[str, Any]:
config_overrides = dict(overrides)
config_overrides["PLUGINS"] = ",".join(selected_plugins)
selected_plugin_names = set(
filter_plugins(_discover_archivebox_plugins(), [plugin_name.lower() for plugin_name in selected_plugins], include_providers=True),
)
for plugin_name, enabled_key in _plugin_enabled_config_keys().items():
config_overrides[enabled_key] = plugin_name in selected_plugin_names
for plugin_name in selected_plugins:
if plugin_name.startswith("search_backend_"):
config_overrides[f"{plugin_name.upper()}_ENABLED"] = True
@@ -1479,6 +1519,28 @@ def fail_unavailable_queued_hooks(
)
def skip_disabled_queued_plugins(snapshot_id: str, plugin_names: list[str]) -> None:
from archivebox.core.models import ArchiveResult
if not plugin_names:
return
now = timezone.now()
# Queued ArchiveResult rows are durable scheduler state and can outlive a
# config change or deploy. If a plugin is no longer selected for this
# Snapshot/Crawl, leaving its old row queued keeps the Snapshot STARTED
# forever even though there is no runnable work left.
ArchiveResult.objects.filter(
snapshot_id=snapshot_id,
plugin__in=plugin_names,
status=ArchiveResult.StatusChoices.QUEUED,
).update(
status=ArchiveResult.StatusChoices.SKIPPED,
start_ts=now,
end_ts=now,
output_str="Queued plugin is disabled by this Snapshot/Crawl config",
)
def include_background_prerequisite_hooks(
selected_hooks_by_plugin: dict[str, set[str] | None],
plugins: dict[str, Plugin],
@@ -1511,12 +1573,13 @@ def include_background_prerequisite_hooks(
def snapshot_hooks_for_pending_archiveresults(snapshot) -> list[tuple[str, str]]:
from archivebox.config.common import get_config
from archivebox.core.models import Snapshot
from archivebox.plugins.discovery import get_enabled_plugins
config = get_config(crawl=snapshot.crawl, snapshot=snapshot)
snapshot_plugin_names = [name.strip() for name in str((snapshot.config or {}).get("PLUGINS") or "").split(",") if name.strip()]
crawl_plugin_names = [name.strip() for name in str((snapshot.crawl.config or {}).get("PLUGINS") or "").split(",") if name.strip()]
config_plugin_names = [name.strip() for name in str(config.PLUGINS or "").split(",") if name.strip()]
plugin_names = snapshot_plugin_names or crawl_plugin_names or config_plugin_names
plugin_names = snapshot_plugin_names or crawl_plugin_names or config_plugin_names or get_enabled_plugins(config=config)
plugins = (
filter_plugins(_discover_archivebox_plugins(), plugin_names, include_providers=True)
if plugin_names
+3 -67
View File
@@ -48,7 +48,9 @@
<div class="crawl-explanation">
<p class="crawl-tip" style="user-select: none;">
<span style="float: right">
<a href="/admin/agent">💬 Crawl with AI</a> &nbsp; | &nbsp;
{% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
<a href="/admin/agent">💬 Crawl with AI</a> &nbsp; | &nbsp;
{% endif %}
<a href="https://github.com/ArchiveBox/archivebox-browser-extension"><img src="{% static 'chrome_extension_icon.png' %}" alt="Chrome Extension" style="height: 33px; margin-top: -5px;">&nbsp; Get the extension</a>
</span>
💡 <strong>Tip:</strong> Instantly save a single URL by visiting:
@@ -372,7 +374,6 @@
if (sameDomainToggleSlot && shortcutToggles.length) {
shortcutToggles.forEach((toggle) => sameDomainToggleSlot.appendChild(toggle));
}
const pluginDependencyMap = JSON.parse('{{ plugin_dependency_map_json|default:"{}"|escapejs }}');
const personaConfigMap = JSON.parse('{{ persona_config_map_json|default:"{}"|escapejs }}');
const personaSelect = document.querySelector('select[name="persona"]');
@@ -404,22 +405,6 @@
return document.querySelector(`.plugin-section-toggle[value="${requiredSearchPlugin}"]`);
}
function getPluginCheckbox(pluginName) {
if (!pluginName) return null;
return document.querySelector(`.plugin-section-toggle[value="${pluginName}"]`);
}
function getRequiredPlugins(pluginName) {
const requiredPlugins = pluginDependencyMap[pluginName];
return Array.isArray(requiredPlugins) ? requiredPlugins : [];
}
function getDependentPlugins(pluginName) {
return Object.entries(pluginDependencyMap)
.filter(([, requiredPlugins]) => Array.isArray(requiredPlugins) && requiredPlugins.includes(pluginName))
.map(([dependentPlugin]) => dependentPlugin);
}
function getConfigEditorRows() {
return document.getElementById('id_config_rows');
}
@@ -555,47 +540,6 @@
updateURLPreview();
}
function ensureRequiredPluginsChecked(pluginName, visited = new Set()) {
if (!pluginName || visited.has(pluginName)) {
return;
}
visited.add(pluginName);
getRequiredPlugins(pluginName).forEach(requiredPlugin => {
const requiredCheckbox = getPluginCheckbox(requiredPlugin);
if (!requiredCheckbox) {
return;
}
requiredCheckbox.checked = true;
dispatchChange(requiredCheckbox);
ensureRequiredPluginsChecked(requiredPlugin, visited);
});
}
function uncheckDependentPlugins(pluginName, visited = new Set()) {
if (!pluginName || visited.has(pluginName)) {
return;
}
visited.add(pluginName);
getDependentPlugins(pluginName).forEach(dependentPlugin => {
const dependentCheckbox = getPluginCheckbox(dependentPlugin);
if (!dependentCheckbox) {
return;
}
dependentCheckbox.checked = false;
dispatchChange(dependentCheckbox);
uncheckDependentPlugins(dependentPlugin, visited);
});
}
function normalizePluginSelections() {
const checkedPlugins = Array.from(document.querySelectorAll('.plugin-section-toggle:checked'))
.map(cb => cb.value)
.filter(Boolean);
checkedPlugins.forEach(pluginName => ensureRequiredPluginsChecked(pluginName));
}
function applyRequiredSearchPlugin(preferredValue = undefined) {
const requiredCheckbox = getRequiredSearchCheckbox();
if (!requiredCheckbox) return;
@@ -1245,7 +1189,6 @@
});
}
normalizePluginSelections();
applyRequiredSearchPlugin(requiredSearchPreference);
syncEnabledPluginsConfig();
updateChromeToggleButton();
@@ -1279,7 +1222,6 @@
dispatchChange(cb);
});
normalizePluginSelections();
applyRequiredSearchPlugin(requiredSearchPreference);
syncEnabledPluginsConfig();
updateChromeToggleButton();
@@ -1289,11 +1231,6 @@
document.querySelectorAll('.plugin-section-toggle').forEach(checkbox => {
checkbox.addEventListener('change', function() {
if (this.checked) {
ensureRequiredPluginsChecked(this.value);
} else {
uncheckDependentPlugins(this.value);
}
syncEnabledPluginsConfig();
updateChromeToggleButton();
saveFormState();
@@ -1348,7 +1285,6 @@
if (personaSelect) {
applyPersonaConfig(personaSelect.value);
}
normalizePluginSelections();
applyRequiredSearchPlugin();
syncEnabledPluginsConfig();
updateChromeToggleButton();
+1 -1
View File
@@ -6,7 +6,7 @@
<a href="{% url 'Home' %}">Snapshots</a> |
<a href="/admin/core/archiveresult/?o=-1">Log</a> |
<a href="/admin/core/tag/">Tags</a> &nbsp; &nbsp;
{% if user.is_authenticated and user.is_superuser %}
{% if user.is_authenticated and user.is_superuser and request.archivebox_config.OPENCODE_ENABLED %}
<a href="/admin/agent">💬 AI</a> |
{% endif %}
<a href="{% url 'Docs' %}" target="_blank" rel="noopener noreferrer">Docs</a> |
+3 -1
View File
@@ -228,6 +228,8 @@ def run_archivebox_cmd(
"SAVE_HTMLTOTEXT": "False",
},
)
if args and args[0] == "version":
run_env["PLUGINS"] = "__archivebox_test_no_plugins__"
if env:
run_env.update(env)
_set_test_source_pythonpath(run_env)
@@ -920,7 +922,7 @@ def _wait_for_archivebox_workers(cwd: Path, env: dict[str, str], names: tuple[st
state: dict[str, Any] = {}
while time.time() < deadline:
state = _archivebox_worker_state(cwd, env)
if all(state.get(name, {}).get("statename") == "RUNNING" for name in names):
if all(isinstance(state.get(name), dict) and state[name].get("statename") == "RUNNING" for name in names):
return state
time.sleep(1)
return state
@@ -475,11 +475,14 @@ def test_update_index_only_runs_paused_search_rows_and_resume_later_runs_crawl(t
if snapshot_finished_before_pause:
indexed_state = get_crawl_runtime_state(tmp_path, crawl_id)
assert indexed_state["crawl_status"] == "paused"
assert indexed_state["crawl_retry_at"] == indexed_state["retry_at_max"]
assert indexed_state["snapshots"][0]["status"] == "sealed"
wget_results = [result for result in indexed_state["results"] if result["plugin"] == "wget"]
assert any(result["status"] == "succeeded" and result["output_size"] > 0 for result in wget_results)
if indexed_state["crawl_status"] == "paused":
assert indexed_state["crawl_retry_at"] == indexed_state["retry_at_max"]
else:
assert indexed_state["crawl_status"] == "sealed"
assert all(result["status"] not in {"queued", "started", "paused"} for result in indexed_state["results"])
captured_text = wait_for_snapshot_capture(tmp_path, recursive_test_site["root_url"], timeout=60)
assert "Root" in captured_text
assert "About" in captured_text
+7 -3
View File
@@ -162,13 +162,17 @@ def wait_for_import_processing(cwd: Path, expected_urls: set[str], *, timeout: f
import time
deadline = time.time() + timeout
counts = {url: 0 for url in expected_urls}
while time.time() < deadline:
with use_archivebox_db(cwd):
snapshot_started = Snapshot.objects.filter(url__in=expected_urls).exists()
if snapshot_started:
rows = list(Snapshot.objects.filter(url__in=expected_urls).values_list("url", flat=True))
counts = {url: 0 for url in expected_urls}
for url in rows:
counts[url] += 1
if all(count >= 1 for count in counts.values()):
return
time.sleep(1)
raise AssertionError("timed out waiting for import crawl processing to start")
raise AssertionError(f"timed out waiting for import crawl processing to start, got counts={counts}")
def wait_for_expected_import_snapshots(cwd: Path, expected_urls: set[str], *, timeout: float = 180.0) -> None:
+1 -1
View File
@@ -228,7 +228,7 @@ SEARCH_BACKEND_PASSWORD = "SecretPassword"
)
result = run_archivebox_cmd(
["version"],
["version", "--binaries=curl"],
)
assert result.returncode == 0, result.stderr
+103
View File
@@ -991,6 +991,42 @@ class TestRecoverOrchestratorState:
assert hook_names
assert all(not hook_name.endswith((".py", ".js", ".sh")) for hook_name in hook_names)
def test_snapshot_hooks_for_pending_archiveresults_respects_disabled_plugins_when_plugins_empty(self):
from django.utils import timezone
from archivebox.base_models.models import get_or_create_system_user_pk
from archivebox.crawls.models import Crawl
from archivebox.core.models import Snapshot
from archivebox.services.runner import snapshot_hooks_for_pending_archiveresults
crawl = Crawl.objects.create(
urls="https://example.com",
created_by_id=get_or_create_system_user_pk(),
config={
"CLAUDECHROME_ENABLED": False,
"CLAUDECODEEXTRACT_ENABLED": False,
"CLAUDECODECLEANUP_ENABLED": False,
"SEARCH_BACKEND_SQLITE_ENABLED": False,
},
status=Crawl.StatusChoices.STARTED,
retry_at=timezone.now(),
)
snapshot = Snapshot.objects.create(
url="https://example.com",
crawl=crawl,
status=Snapshot.StatusChoices.QUEUED,
retry_at=timezone.now(),
)
hooks = snapshot_hooks_for_pending_archiveresults(snapshot)
queued_plugins = {plugin for plugin, _hook_name in hooks}
assert "title" in queued_plugins
assert "claudechrome" not in queued_plugins
assert "claudecodeextract" not in queued_plugins
assert "claudecodecleanup" not in queued_plugins
assert "search_backend_sqlite" not in queued_plugins
def test_run_due_snapshot_pauses_child_when_parent_is_paused(self):
from django.utils import timezone
@@ -1467,6 +1503,73 @@ class TestRecoverOrchestratorState:
assert result.status == ArchiveResult.StatusChoices.FAILED
assert snapshot.retry_at is None
@pytest.mark.django_db(transaction=True)
def test_run_due_snapshot_skips_disabled_queued_plugins_and_seals_started_snapshot(self):
from django.utils import timezone
from archivebox.base_models.models import get_or_create_system_user_pk
from archivebox.crawls.models import Crawl
from archivebox.core.models import ArchiveResult, Snapshot
from archivebox.services.runner import run_due_snapshot
crawl = Crawl.objects.create(
urls="https://example.com",
created_by_id=get_or_create_system_user_pk(),
config={
"CLAUDECHROME_ENABLED": False,
"CLAUDECODEEXTRACT_ENABLED": False,
"CLAUDECODECLEANUP_ENABLED": False,
"SEARCH_BACKEND_SQLITE_ENABLED": False,
},
status=Crawl.StatusChoices.STARTED,
retry_at=timezone.now(),
)
snapshot = Snapshot.objects.create(
url="https://example.com",
crawl=crawl,
status=Snapshot.StatusChoices.STARTED,
retry_at=timezone.now(),
downloaded_at=timezone.now(),
)
ArchiveResult.objects.create(
snapshot=snapshot,
plugin="title",
hook_name="on_Snapshot__01_title",
status=ArchiveResult.StatusChoices.SUCCEEDED,
output_str="Example Domain",
)
stale_results = [
ArchiveResult.objects.create(
snapshot=snapshot,
plugin=plugin,
hook_name=hook_name,
status=ArchiveResult.StatusChoices.QUEUED,
)
for plugin, hook_name in (
("claudechrome", "on_Snapshot__47_claudechrome"),
("claudecodeextract", "on_Snapshot__58_claudecodeextract"),
("claudecodecleanup", "on_Snapshot__92_claudecodecleanup"),
("search_backend_sqlite", "on_Snapshot__90_index_sqlite"),
)
]
stale_result_ids = [result.id for result in stale_results]
stale_plugins = [result.plugin for result in stale_results]
assert run_due_snapshot(snapshot, lock_seconds=60) is True
snapshot.refresh_from_db()
assert snapshot.status == Snapshot.StatusChoices.SEALED
assert snapshot.retry_at is None
assert not snapshot.archiveresult_set.filter(
plugin__in=stale_plugins,
status=ArchiveResult.StatusChoices.QUEUED,
).exists()
for result in ArchiveResult.objects.filter(id__in=stale_result_ids):
assert result.status == ArchiveResult.StatusChoices.SKIPPED
assert result.start_ts is not None
assert result.end_ts is not None
assert "disabled by this Snapshot/Crawl config" in result.output_str
@pytest.mark.django_db(transaction=True)
@pytest.mark.timeout(300)
@pytest.mark.parametrize("chrome_isolation", ["crawl", "snapshot"])
+78 -1
View File
@@ -10,7 +10,7 @@ import tempfile
from pathlib import Path
from archivebox.config.paths import tmp_dir_socket_path_is_short_enough
from archivebox.cli.archivebox_version import _binary_row_dedupe_key
from archivebox.tests.conftest import run_archivebox_cmd
from archivebox.tests.conftest import cli_env, run_archivebox_cmd
def _make_deep_collection_dir(tmp_path: Path) -> Path:
@@ -126,6 +126,83 @@ def test_version_shows_binaries_after_init(tmp_path, initialized_archive):
assert "Binary" in output or "Dependencies" in output
def test_version_skips_disabled_plugin_binary_resolution(tmp_path):
"""Disabled plugins should not trigger live binary detection during version."""
data_dir = tmp_path / "no-plugins"
data_dir.mkdir()
env = cli_env(PLUGINS="__archivebox_test_no_plugins__", SEARCH_BACKEND_ENGINE="")
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env)
assert init_result.returncode == 0, init_result.stderr
version_result = run_archivebox_cmd(["version"], cwd=data_dir, env=env)
output = version_result.stdout + version_result.stderr
assert version_result.returncode == 0, output
assert "No required binaries declared for discovered plugins" in output
assert "not installed" not in output
def test_version_honors_legacy_save_aliases_when_disabling_extractors(tmp_path):
"""Legacy SAVE_* aliases should disable their canonical plugin flags."""
data_dir = tmp_path / "legacy-save-disabled"
data_dir.mkdir()
env = {"PLUGINS": "wget,title", "SEARCH_BACKEND_ENGINE": ""}
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env, default_cli_env=True, disable_extractors=True, timeout=120)
assert init_result.returncode == 0, init_result.stderr or init_result.stdout
config_result = run_archivebox_cmd(
["config", "--get", "WGET_ENABLED", "TITLE_ENABLED", "CHROME_ENABLED"],
cwd=data_dir,
env=env,
default_cli_env=True,
disable_extractors=True,
)
config_output = config_result.stdout + config_result.stderr
assert config_result.returncode == 0, config_output
assert "WGET_ENABLED = false" in config_output
assert "TITLE_ENABLED = false" in config_output
assert "CHROME_ENABLED = false" in config_output
version_result = run_archivebox_cmd(
["version"],
cwd=data_dir,
env=env,
default_cli_env=True,
disable_extractors=True,
timeout=60,
)
output = version_result.stdout + version_result.stderr
assert version_result.returncode == 0, output
assert "not installed" not in output
def test_plugins_selection_includes_required_plugins_via_config_cli(tmp_path):
"""PLUGINS selection should include transitive plugin dependencies exactly once."""
data_dir = tmp_path / "plugin-dependencies"
data_dir.mkdir()
env = {"PLUGINS": "wget,screenshot"}
init_result = run_archivebox_cmd(["init"], cwd=data_dir, env=env, default_cli_env=True, timeout=120)
assert init_result.returncode == 0, init_result.stderr or init_result.stdout
config_result = run_archivebox_cmd(
["config", "--get", "WGET_ENABLED", "SCREENSHOT_ENABLED", "CHROME_ENABLED", "YTDLP_ENABLED"],
cwd=data_dir,
env=env,
default_cli_env=True,
)
config_output = config_result.stdout + config_result.stderr
assert config_result.returncode == 0, config_output
assert "WGET_ENABLED = true" in config_output
assert "SCREENSHOT_ENABLED = true" in config_output
assert "CHROME_ENABLED = true" in config_output
assert "YTDLP_ENABLED = false" in config_output
def test_version_shows_data_locations(tmp_path, initialized_archive):
"""Test that version shows data directory locations."""
result = run_archivebox_cmd(["version"])
+1 -1
View File
@@ -154,7 +154,7 @@ print(json.dumps({{
retained = json.loads(stdout.strip().splitlines()[-1])
assert retained["crawl_id"] == created["crawl_id"]
_cmd_result = run_archivebox_cmd(["run", "--crawl-id", retained["crawl_id"]], cwd=tmp_path, timeout=120, env=run_env)
_cmd_result = run_archivebox_cmd(["run", "--maintenance-only"], cwd=tmp_path, timeout=120, env=run_env)
stdout, stderr, returncode = _cmd_result.stdout, _cmd_result.stderr, _cmd_result.returncode
assert returncode == 0, f"archivebox run failed:\nSTDOUT:\n{stdout}\nSTDERR:\n{stderr}"
+20 -16
View File
@@ -5,8 +5,8 @@ from pathlib import Path
from types import SimpleNamespace
from urllib.parse import quote
import psutil
import pytest
import requests
from archivebox.tests.conftest import ADMIN_TEST_HOST, run_archivebox_cmd
@@ -54,7 +54,6 @@ def opencode_archive_config(initialized_archive):
{
"ABXPKG_INSTALL_TIMEOUT": "900",
"ABXPKG_MIN_RELEASE_AGE": "0",
"ABX_RUNTIME": "archivebox",
"ARCHIVEBOX_ALLOW_NO_UNIX_SOCKETS": "true",
"OPENCODE_ENABLED": "True",
"OPENCODE_HOST": "127.0.0.1",
@@ -175,10 +174,11 @@ def test_opencode_agent_superuser_gets_admin_wrapper(admin_client, live_opencode
from abx_plugins.plugins.opencode import views
response = admin_client.get("/admin/agent", HTTP_HOST=ADMIN_TEST_HOST)
recent_session_id = views._recent_session_id(live_opencode.settings)
assert response.status_code == 200
assert f'<iframe src="{views._project_route(live_opencode.config.data_dir)}'.encode() in response.content
assert b'/session"' in response.content
assert recent_session_id
assert f'<iframe src="{views._project_route(live_opencode.config.data_dir, recent_session_id)}"'.encode() in response.content
assert b'id="header"' in response.content
assert b'id="progress-monitor"' in response.content
@@ -225,20 +225,24 @@ def test_opencode_proxy_sse_response_is_unbuffered(admin_client, live_opencode):
assert response.headers["Cache-Control"] == "no-store"
def test_opencode_starts_with_data_dir_cwd_and_isolated_state(live_opencode):
process = psutil.Process(live_opencode.process.pid)
env = process.environ()
def test_opencode_starts_with_isolated_state(live_opencode):
workdir = str(live_opencode.config.data_dir.resolve())
state_dir = live_opencode.config.state_dir
assert Path(process.cwd()).resolve() == live_opencode.config.data_dir.resolve()
assert env["BROWSER"] == "false"
assert env["HOME"] == str(live_opencode.config.state_dir / "home")
assert env["XDG_CONFIG_HOME"] == str(live_opencode.config.state_dir / "config")
assert env["XDG_DATA_HOME"] == str(live_opencode.config.state_dir / "data")
assert env["XDG_STATE_HOME"] == str(live_opencode.config.state_dir / "state")
assert env["XDG_CACHE_HOME"] == str(live_opencode.config.state_dir / "cache")
assert env["OPENCODE_DISABLE_PROJECT_CONFIG"] == "true"
assert env["GIT_CEILING_DIRECTORIES"] == workdir
project = requests.get(
f"{live_opencode.settings['origin']}/project/current",
params={"directory": workdir},
timeout=live_opencode.settings["timeout"],
)
project.raise_for_status()
assert Path(live_opencode.settings["workdir"]).resolve() == Path(workdir)
assert Path(str(project.json()["worktree"])).resolve() == Path(workdir)
assert live_opencode.process.poll() is None
assert (state_dir / "data" / "opencode" / "opencode.db").is_file()
assert (state_dir / "SKILL.md").is_file()
assert (state_dir / "config" / "opencode" / "skills" / "archivebox" / "SKILL.md").resolve() == state_dir / "SKILL.md"
assert not (live_opencode.config.data_dir / ".git" / "not-a-git").exists()
def test_opencode_state_dir_is_separate_from_workdir(tmp_path):
+7 -5
View File
@@ -484,9 +484,13 @@ def test_add_archivewebpage_installs_required_chrome_dependency(initialized_arch
assert "archivewebpage" in binaries
assert binaries["archivewebpage"]["status"] == Binary.StatusChoices.INSTALLED
assert binaries["archivewebpage"]["binprovider"] == "chromewebstore"
archivewebpage_manifest = Path(binaries["archivewebpage"]["abspath"])
archivewebpage_metadata = Path(binaries["archivewebpage"]["abspath"])
assert archivewebpage_metadata.exists()
assert archivewebpage_metadata.name == "archivewebpage.extension.json"
archivewebpage_extension = json.loads(archivewebpage_metadata.read_text(encoding="utf-8"))
archivewebpage_manifest = Path(archivewebpage_extension["unpacked_path"]) / "manifest.json"
assert archivewebpage_manifest.exists()
assert archivewebpage_manifest.name == "manifest.json"
assert json.loads(archivewebpage_manifest.read_text(encoding="utf-8"))["version"] == binaries["archivewebpage"]["version"]
plugins_seen = {plugin for plugin, _hook_name, _status, _output_str, _output_files in archive_results}
assert {"chrome", "archivewebpage"}.issubset(plugins_seen)
@@ -521,9 +525,7 @@ def test_recursive_crawl_depth_two_all_plugins_runs_snapshots_in_parallel(initia
from abx_dl.models import discover_plugins
root_url = "https://example.com/"
plugin_selection = ",".join(
sorted(plugin for plugin in discover_plugins().keys() if not plugin.startswith("claude")),
)
plugin_selection = ",".join(sorted(plugin for plugin in discover_plugins().keys() if not plugin.startswith("claude")))
env = os.environ.copy()
env.pop("CHROME_BINARY", None)
env.update(
+2 -2
View File
@@ -590,7 +590,7 @@ class TestPublicIndexSearch:
assert response.status_code == 200
content = response.content.decode()
assert "/screenshot.png" in content
assert "screenshot/screenshot.png" not in content
assert "/screenshot/screenshot.png" not in content
@override_settings(PUBLIC_INDEX=True)
def test_public_index_preview_respects_plugin_relative_screenshot_output(self, client, public_snapshot):
@@ -634,7 +634,7 @@ class TestPublicIndexSearch:
first = content.index("chrome_extension_screenshot/screenshot-1.png")
second = content.index("chrome_extension_screenshot/screenshot.png")
assert first < second
assert "screenshot/screenshot.png" not in content
assert "/screenshot/screenshot.png" not in content
assert "chrome_extension_screenshot/screenshot-2.png" not in content
@override_settings(PUBLIC_INDEX=True)
+13
View File
@@ -99,6 +99,19 @@ def test_add_view_admin_renders_plugin_config_grid(client, admin_user, monkeypat
assert b"plugin_config__wget__WGET_ENABLED" not in response.content
def test_add_view_hides_agent_link_when_opencode_is_disabled(client, admin_user):
from archivebox.machine.models import Machine
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
client.force_login(admin_user)
response = client.get(reverse("add"), HTTP_HOST=ADMIN_HOST)
assert response.status_code == 200
assert b"/admin/agent" not in response.content
assert b"Crawl with AI" not in response.content
def test_add_view_staff_user_cannot_override_raw_or_plugin_config(client):
staff_user = User.objects.create_user(
username="addviewstaff",
+97 -2
View File
@@ -344,6 +344,97 @@ def _wait_for_worker_state(cwd, worker_name: str, statename: str, timeout: int =
raise AssertionError(f"Timed out waiting for {worker_name}={statename}, last state={state}")
@pytest.mark.timeout(240)
def test_public_add_view_depth_one_crawl_skips_unreadable_persona_profile_entries(tmp_path, recursive_test_site):
init_archive(tmp_path)
chrome_profile = tmp_path / "personas" / "Default" / "chrome_profile"
default_profile = chrome_profile / "Default"
default_profile.mkdir(parents=True, exist_ok=True)
(default_profile / "Preferences").write_text('{"profile":{"name":"Default"}}', encoding="utf-8")
unreadable_dir = chrome_profile / "ActorSafetyLists"
unreadable_dir.mkdir(parents=True, exist_ok=True)
(unreadable_dir / "Preferences").write_text("{}", encoding="utf-8")
unreadable_dir.chmod(0)
port = get_free_port()
env = cli_env(
port=port,
PLUGINS="wget,parse_html_urls",
SAVE_WGET="True",
PUBLIC_INDEX="True",
PUBLIC_ADD_VIEW="True",
URL_ALLOWLIST=r"127\.0\.0\.1[:/].*",
)
try:
start_archivebox_server(tmp_path, env=env, port=port)
add_page = wait_for_http(port, host=f"web.archivebox.localhost:{port}", path="/add/")
assert add_page.status_code == 200
assert 'name="depth"' in add_page.text
response = requests.post(
f"http://127.0.0.1:{port}/add/",
headers={"Host": f"web.archivebox.localhost:{port}", "Referer": f"http://web.archivebox.localhost:{port}/add/"},
data={
"url": recursive_test_site["root_url"],
"depth": "1",
"max_urls": "10",
"crawl_max_size": "0",
"snapshot_max_size": "0",
"tag": "public-depth-one-unreadable-profile",
"url_filters_allowlist": r"127\.0\.0\.1[:/].*",
"url_filters_denylist": "",
"schedule": "",
"notes": "public depth one with unreadable chrome profile entries",
"persona": "Default",
"permissions": "public",
"start_paused": "",
"config": "{}",
},
timeout=10,
allow_redirects=False,
)
assert response.status_code in (302, 303), response.text
deadline = time.time() + 180
while time.time() < deadline:
with use_archivebox_db(tmp_path):
crawl = Crawl.objects.order_by("-created_at").first()
depth_counts = get_depth_counts(tmp_path)
child_urls = set(Snapshot.objects.filter(depth=1).values_list("url", flat=True))
failed_results = list(
ArchiveResult.objects.filter(status=ArchiveResult.StatusChoices.FAILED).values_list("plugin", "output_str"),
)
if depth_counts.get(0, 0) >= 1 and set(recursive_test_site["child_urls"]).issubset(child_urls):
break
assert not failed_results
time.sleep(2)
else:
raise AssertionError(f"timed out waiting for depth=1 crawl, got depth counts {get_depth_counts(tmp_path)}")
with use_archivebox_db(tmp_path):
crawl = Crawl.objects.order_by("-created_at").first()
snapshot_rows = list(Snapshot.objects.order_by("depth", "url").values_list("url", "depth", "status"))
result_rows = list(ArchiveResult.objects.order_by("plugin", "status").values_list("plugin", "status", "output_size"))
finally:
unreadable_dir.chmod(0o700)
stop_server(tmp_path)
assert crawl is not None
assert crawl.max_depth == 1
assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
assert (recursive_test_site["root_url"], 0, Snapshot.StatusChoices.SEALED) in snapshot_rows
assert set(recursive_test_site["child_urls"]).issubset({url for url, depth, _status in snapshot_rows if depth == 1})
assert all(
status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
for _url, _depth, status in snapshot_rows
)
assert not [row for row in result_rows if row[1] == ArchiveResult.StatusChoices.FAILED]
assert any(plugin == "wget" and status == ArchiveResult.StatusChoices.SUCCEEDED and size > 0 for plugin, status, size in result_rows)
assert any(plugin == "parse_html_urls" and status == ArchiveResult.StatusChoices.SUCCEEDED for plugin, status, _size in result_rows)
@pytest.mark.timeout(420)
def test_public_add_view_import_text_formats_preserve_metadata_and_resume_without_duplicates(tmp_path):
"""Public /add/ textarea should import rich text, survive runner restart, and preserve one row per URL."""
@@ -516,9 +607,10 @@ def test_public_add_view_rejects_file_path_and_shell_injection_payloads(tmp_path
with use_archivebox_db(tmp_path):
snapshot = Snapshot.objects.get(url=safe_url)
crawl = Crawl.objects.get()
tag_names = set(snapshot.tags.values_list("name", flat=True))
assert crawl.status in {Crawl.StatusChoices.STARTED, Crawl.StatusChoices.SEALED}
assert snapshot.status in {Snapshot.StatusChoices.QUEUED, Snapshot.StatusChoices.STARTED, Snapshot.StatusChoices.SEALED}
assert "public-ui-security" in set(snapshot.tags.values_list("name", flat=True))
assert "public-ui-security" in tag_names
@pytest.mark.timeout(180)
@@ -580,7 +672,10 @@ def test_add_view_post_creates_schedule_over_server(tmp_path, recursive_test_sit
with use_archivebox_db(tmp_path):
schedule = CrawlSchedule.objects.select_related("template").order_by("-created_at").first()
row = (schedule.schedule, schedule.template.urls, schedule.template.tags_str) if schedule else None
row = None
if schedule:
template_url = json.loads(schedule.template.urls)["url"]
row = (schedule.schedule, template_url, schedule.template.tags_str)
assert row == ("daily", recursive_test_site["root_url"], "web-ui")
finally:
+13
View File
@@ -239,6 +239,19 @@ def test_snapshot_admin_zip_links():
assert html.escape(zip_url, quote=True) in str(admin.admin_actions(snapshot))
def test_admin_navigation_hides_agent_link_when_opencode_is_disabled(client, admin_user):
from archivebox.machine.models import Machine
Machine.from_json({"config": {"OPENCODE_ENABLED": False}})
client.force_login(admin_user)
response = client.get(reverse("admin:index"), HTTP_HOST="admin.archivebox.localhost:8000")
assert response.status_code == 200
assert b"/admin/agent" not in response.content
assert b">\xf0\x9f\x92\xac AI<" not in response.content
def test_archiveresult_admin_zip_links():
from archivebox.core.admin_archiveresults import ArchiveResultAdmin
from archivebox.core.models import ArchiveResult
+22
View File
@@ -94,6 +94,28 @@ def test_discover_persona_template_profiles_finds_data_dir_personas():
assert matching[0].user_data_dir.parent.parent == CONSTANTS.PERSONAS_DIR.resolve()
def test_persona_admin_add_view_skips_unreadable_chrome_profile_entries(admin_client):
from archivebox.config.constants import CONSTANTS
chrome_profile = CONSTANTS.PERSONAS_DIR / "UnreadableTemplatePersona" / "chrome_profile"
default_profile = chrome_profile / "Default"
default_profile.mkdir(parents=True, exist_ok=True)
(default_profile / "Preferences").write_text('{"profile":{"name":"Default"}}')
unreadable_dir = chrome_profile / "ActorSafetyLists"
unreadable_dir.mkdir(parents=True, exist_ok=True)
(unreadable_dir / "Preferences").write_text("{}")
unreadable_dir.chmod(0)
try:
response = admin_client.get(reverse("admin:personas_persona_add"), HTTP_HOST=ADMIN_TEST_HOST)
finally:
unreadable_dir.chmod(0o700)
assert response.status_code == 200
assert b"UnreadableTemplatePersona" in response.content
assert b"Persona Template" in response.content
def test_persona_admin_add_view_renders_import_ui(admin_client):
source = _make_persona_template_source("RenderImportPersona")
+21 -4
View File
@@ -32,6 +32,25 @@ def test_snapshot_changelist_uses_stable_ordering_without_unordered_paginator_wa
assert b"Searching matching snapshots..." in response.content
def test_snapshot_changelist_preview_uses_prefetched_output_files(admin_client, snapshot):
from archivebox.core.models import ArchiveResult
ArchiveResult.objects.create(
snapshot=snapshot,
plugin="screenshot",
hook_name="on_Snapshot__40_screenshot.js",
status=ArchiveResult.StatusChoices.SUCCEEDED,
output_size=128,
output_files={"screenshot.png": {"size": 128, "root_relative": True}},
)
response = admin_client.get(reverse("admin:core_snapshot_changelist"), HTTP_HOST=ADMIN_TEST_HOST)
assert response.status_code == 200
assert b'class="snapshot-preview' in response.content
assert b"screenshot.png" in response.content
def test_snapshot_admin_tag_editor_escapes_tag_json_script_breakout(admin_client, snapshot):
from archivebox.core.models import Tag
@@ -188,10 +207,8 @@ def test_snapshot_admin_preview_uses_extension_screenshot_when_standard_screensh
preview = admin._get_preview_data(snapshot)
assert preview is not None
assert "screenshot/screenshot.png" in preview["img_url"]
first = preview["fallback_list"].index("chrome_extension_screenshot/screenshot-1.png")
second = preview["fallback_list"].index("chrome_extension_screenshot/screenshot.png")
assert first < second
assert "chrome_extension_screenshot/screenshot-1.png" in preview["img_url"]
assert "chrome_extension_screenshot/screenshot.png" in preview["fallback_list"]
assert "chrome_extension_screenshot/screenshot-2.png" not in preview["fallback_list"]
+3
View File
@@ -60,6 +60,9 @@ cat > .archivebox-deploy.override.yml <<EOF
services:
$DEPLOY_SERVICE:
image: $DEPLOY_IMAGE
user: ""
entrypoint: null
command: null
EOF
COMPOSE=(docker compose -f "$COMPOSE_FILE" -f .archivebox-deploy.override.yml)
+44 -49
View File
@@ -49,14 +49,22 @@ docker_pull_archivebox() {
fi
}
docker_run_archivebox_init() {
docker_run_archivebox() {
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
docker run --platform "$ARCHIVEBOX_PLATFORM" "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" init --install
docker run --platform "$ARCHIVEBOX_PLATFORM" "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" "$@"
else
docker run "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" init --install
docker run "$DOCKER_RUN_TTY_ARG" -v "$PWD":/data --rm "$ARCHIVEBOX_IMAGE" "$@"
fi
}
docker_run_archivebox_init() {
docker_run_archivebox init
}
docker_run_archivebox_install() {
docker_run_archivebox install
}
docker_run_archivebox_server() {
if [ -n "$ARCHIVEBOX_PLATFORM" ]; then
docker run --platform "$ARCHIVEBOX_PLATFORM" -v "$PWD":/data -d -p 8000:8000 --name=archivebox "$ARCHIVEBOX_IMAGE"
@@ -128,11 +136,7 @@ install_archivebox_with_uv() {
echo
echo "[+] Installing ArchiveBox python tool using uv from $ARCHIVEBOX_PACKAGE..."
if uv --no-config tool add --help > /dev/null 2>&1; then
uv --no-config tool add --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
else
uv --no-config tool install --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
fi
uv --no-config tool install --python "$ARCHIVEBOX_PYTHON" --upgrade "$ARCHIVEBOX_PACKAGE"
uv_tool_bin_dir="$(uv --no-config tool dir --bin)"
export PATH="$uv_tool_bin_dir:$PATH"
@@ -160,7 +164,8 @@ if (command -v docker > /dev/null && docker compose version > /dev/null && docke
fi
curl -fsSL "$ARCHIVEBOX_COMPOSE_URL" > docker-compose.yml
export ARCHIVEBOX_IMAGE ARCHIVEBOX_PLATFORM
docker_compose_run_archivebox init --install
docker_compose_run_archivebox init
docker_compose_run_archivebox install
echo
echo "[+] Starting ArchiveBox server using: docker compose up -d..."
docker compose up -d
@@ -187,8 +192,9 @@ elif (command -v docker > /dev/null && docker_pull_archivebox); then
fi
cd ./data
docker_run_archivebox_init
docker_run_archivebox_install
echo
echo "[+] Starting ArchiveBox server using: docker run -d archivebox/archivebox..."
echo "[+] Starting ArchiveBox server using: docker run -d $ARCHIVEBOX_IMAGE..."
docker_run_archivebox_server
wait_for_archivebox
open_archivebox
@@ -213,21 +219,20 @@ echo " ⚠️ If you want to use Docker, press [Ctrl-C] to cancel now. ⚠️
echo " Get Docker: https://docs.docker.com/get-docker/"
echo " After you've installed Docker, run this script again."
echo
echo "Otherwise, install will continue with apt/brew/pkg + uv in 12s... (press [Ctrl+C] to cancel)"
echo "Otherwise, install will continue with uv in 12s... (press [Ctrl+C] to cancel)"
echo
sleep 12 || exit 1
echo "Proceeding with system package manager..."
echo "Proceeding with uv..."
echo
echo "[i] ArchiveBox Setup Script 📦"
echo
echo " This is a helper script which installs ArchiveBox and bootstraps its Python/Node runtimes."
echo " This is a helper script which installs ArchiveBox and bootstraps its runtime dependencies."
echo " You may be prompted for a sudo password in order to install the following:"
echo
echo " - archivebox"
echo " - python3, uv, nodejs, npm (languages used by ArchiveBox and plugin installers)"
echo " - curl, wget (used to bootstrap package installation)"
echo " - extractor/plugin dependencies (installed/discovered by archivebox init --install)"
echo " - uv / curl / ca-certificates (as needed to bootstrap ArchiveBox)"
echo " - archivebox (installed by uv with Python $ARCHIVEBOX_PYTHON)"
echo " - extractor/plugin dependencies (installed/discovered by archivebox install)"
echo
echo " If you'd rather install these manually as-needed, you can find detailed documentation here:"
echo " https://github.com/ArchiveBox/ArchiveBox/wiki/Install"
@@ -235,43 +240,32 @@ echo
echo "Continuing in 12s... (press [Ctrl+C] to cancel)"
echo
sleep 12 || exit 1
echo "Proceeding to install dependencies..."
echo "Proceeding to install ArchiveBox..."
echo
# On Linux:
if which apt-get > /dev/null; then
echo "[+] Installing ArchiveBox system dependencies using apt..."
sudo apt-get update -qq
sudo apt-get install -y python3 python3-venv wget curl nodejs npm
install_archivebox_with_uv
# On Mac:
elif which brew > /dev/null; then
echo "[+] Installing ArchiveBox using Homebrew..."
brew tap archivebox/archivebox
brew update
brew install archivebox
elif which pkg > /dev/null; then
echo "[+] Installing ArchiveBox system dependencies using pkg and uv..."
sudo pkg install -y python3 py39-sqlite3 npm wget curl
install_archivebox_with_uv
else
echo "[!] Warning: Could not find aptitude/homebrew/pkg! May not be able to install all dependencies automatically."
echo
echo " If you're on macOS, make sure you have homebrew installed: https://brew.sh/"
echo " If you're on Linux, only Ubuntu/Debian/BSD systems are officially supported with this script."
echo " If you're on Windows, this script is not officially supported (Docker is recommended instead)."
echo
echo "See the README.md for Manual Setup & Troubleshooting instructions if you you're unable to run ArchiveBox after this script completes."
if ! command -v uv > /dev/null 2>&1 && ! command -v curl > /dev/null 2>&1 && ! command -v wget > /dev/null 2>&1; then
if command -v apt-get > /dev/null 2>&1; then
echo "[+] Installing curl and CA certificates to bootstrap uv using apt..."
sudo apt-get update -qq
sudo apt-get install -y curl ca-certificates
elif command -v brew > /dev/null 2>&1; then
echo "[+] Installing uv using Homebrew..."
brew install uv
elif command -v pkg > /dev/null 2>&1; then
echo "[+] Installing curl and CA certificates to bootstrap uv using pkg..."
sudo pkg install -y curl ca_root_nss
else
echo "[!] Warning: Could not find uv, curl, wget, apt, brew, or pkg."
echo
echo "See the README.md for Manual Setup & Troubleshooting instructions if you you're unable to run ArchiveBox after this script completes."
fi
fi
echo
if ! which archivebox > /dev/null 2>&1; then
ensure_uv
export PATH="$(uv --no-config tool dir --bin):$PATH"
fi
install_archivebox_with_uv
if ! which archivebox > /dev/null 2>&1; then
if ! command -v archivebox > /dev/null 2>&1; then
echo "[X] archivebox command was not found in PATH after installing!"
echo " Check to see if a previous step failed."
exit 1
@@ -285,7 +279,8 @@ if [ -f "./index.sqlite3" ]; then
mv -i ~/archivebox/* ~/archivebox/data/
fi
cd ./data
: | archivebox init --install # pipe in empty command to make sure stdin is closed
: | archivebox init # pipe in empty command to make sure stdin is closed
archivebox install
# init shows version output at the end too
echo
echo "[+] Starting ArchiveBox server using: nohup archivebox server &..."
@@ -299,7 +294,7 @@ echo " archivebox server --quick-init 0.0.0.0:8000 # start server proc
echo " archivebox manage createsuperuser # add an admin user+pass"
echo " ps aux | grep archivebox # see server process pid"
echo " pkill -f archivebox # stop the server"
echo " uv tool install --python $ARCHIVEBOX_PYTHON --upgrade '$ARCHIVEBOX_PACKAGE'; archivebox init # update versions"
echo " uv tool install --python $ARCHIVEBOX_PYTHON --upgrade '$ARCHIVEBOX_PACKAGE'; archivebox init; archivebox install # update versions"
echo " archivebox add 'https://example.com'" # archive a new URL
echo " archivebox list # see URLs archived"
echo " archivebox help # see more help & examples"
+3
View File
@@ -27,6 +27,9 @@ cat > "$OVERRIDE_FILE" <<EOF
services:
$DEPLOY_SERVICE:
image: $DEPLOY_IMAGE
user: ""
entrypoint: null
command: null
EOF
COMPOSE=(docker compose -f "$COMPOSE_FILE" -f "$OVERRIDE_FILE")
+2 -2
View File
@@ -478,7 +478,7 @@ URL users are redirected to after logging out. The default `/` keeps users on Ar
### LDAP Settings
*Options for LDAP / Active Directory authentication via [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap). Requires `pip install archivebox[ldap]` (which also pulls in the system `libldap` / `libsasl` headers).*
*Options for LDAP / Active Directory authentication via [django-auth-ldap](https://github.com/django-auth-ldap/django-auth-ldap). Requires `uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'` (which also pulls in the system `libldap` / `libsasl` headers).*
---
#### `LDAP_ENABLED`
@@ -487,7 +487,7 @@ URL users are redirected to after logging out. The default `/` keeps users on Ar
Master switch for LDAP authentication. When `True`, ArchiveBox loads the `django-auth-ldap` backend and validates that `LDAP_SERVER_URI`, `LDAP_BIND_DN`, `LDAP_BIND_PASSWORD`, and `LDAP_USER_BASE` are all set — startup fails fast otherwise.
```bash
pip install archivebox[ldap]
uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
```
Then set these configuration values:
+19 -20
View File
@@ -24,17 +24,15 @@ Running ArchiveBox with Docker allows you to manage it in a container without ex
**Official Docker Hub image: [`hub.docker.com/r/archivebox/archivebox`](https://hub.docker.com/r/archivebox/archivebox)**
```bash
docker pull archivebox/archivebox:latest
docker pull archivebox/archivebox:dev
```
- [`Dockerfile`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile)
- [`docker-compose.yml`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml)
- [`archivebox-kubernetes.yml`](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml)
Published [Docker tags](https://hub.docker.com/r/archivebox/archivebox/tags):
- `:latest`, `:stable` (latest stable release, the default)
- `:x.x` and `:x.x.x` for specific versions (e.g. `:0.7` or `:0.7.2`)
- `:dev` for unstable alpha builds (breaks often, only for developers and willing beta testers)
- `:x.xrcN` and `:x.x.xrcN` for specific RC versions
- `:sha-xxxxxxx` for builds of specific git commits (to test or pin specific PRs or commits)
<br/>
@@ -63,7 +61,7 @@ mkdir -p ~/archivebox/data && cd ~/archivebox
# download the compose file into the directory
curl -fsSL 'https://docker-compose.archivebox.io' > docker-compose.yml
# (shortcut for getting https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/docker-compose.yml)
# (shortcut for getting https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/docker-compose.yml)
# initialize your collection and create an admin user for the Web UI (or set ADMIN_USERNAME/ADMIN_PASSWORD env vars)
docker compose run archivebox init
@@ -201,10 +199,11 @@ Set `BASE_URL=https://archive.example.com` in the `.env` file next to `docker-co
Fetch and run the ArchiveBox Docker image to create your initial archive.
```bash
docker pull archivebox/archivebox
docker pull archivebox/archivebox:dev
mkdkir -p ~/archivebox/data && cd ~/archivebox/data
docker run -it -v $PWD:/data archivebox/archivebox init --setup
mkdir -p ~/archivebox/data && cd ~/archivebox/data
docker run -it -v $PWD:/data archivebox/archivebox:dev init
docker run -it -v $PWD:/data archivebox/archivebox:dev install
```
*(You can create a collection in any directory you want, `~/archivebox/data` is just used as an example here)*
@@ -221,31 +220,31 @@ See the wiki page on [Upgrading or Merging Archives: Upgrading with plain Docker
### Usage
The Docker CLI `docker run ... archivebox/archivebox [subcommand]` works just like the non-Docker `archivebox [subcommand]` CLI.
The Docker CLI `docker run ... archivebox/archivebox:dev [subcommand]` works just like the non-Docker `archivebox [subcommand]` CLI.
First, make sure you're `cd`'ed into your collection data folder (e.g. `~/archivebox/data`).
```bash
docker run -it -v $PWD:/data archivebox/archivebox help
docker run -it -v $PWD:/data archivebox/archivebox:dev help
```
To add a single URL, pass it as an arg or pipe it in via stdin:
```bash
docker run -it -v $PWD:/data archivebox/archivebox add 'https://example.com'
docker run -it -v $PWD:/data archivebox/archivebox:dev add 'https://example.com'
# OR
echo 'https://example.com' | docker run -i -v $PWD:/data archivebox/archivebox add
echo 'https://example.com' | docker run -i -v $PWD:/data archivebox/archivebox:dev add
```
To archive multiple URLs at once, pass text containing URLs in via stdin:
```bash
docker run -i -v $PWD:/data archivebox/archivebox add < urls.txt
docker run -i -v $PWD:/data archivebox/archivebox:dev add < urls.txt
# OR
curl 'https://example.com/some/rss/feed.xml' | docker run -i -v $PWD:/data archivebox/archivebox add
curl 'https://example.com/some/rss/feed.xml' | docker run -i -v $PWD:/data archivebox/archivebox:dev add
```
You can also use the `--depth=1` flag to tell ArchiveBox to recursively archive the URLs within a provided source.
```bash
docker run -it -v $PWD:/data archivebox/archivebox add --depth=1 'https://example.com/some/rss/feed.xml'
docker run -it -v $PWD:/data archivebox/archivebox:dev add --depth=1 'https://example.com/some/rss/feed.xml'
```
<br/>
@@ -256,12 +255,12 @@ The `docker run` `-v /path/on/host:/path/inside/container` flag specifies where
For example to use a folder on an external USB drive (instead of the current directory `$PWD` or `~/archivebox/data`):
```bash
docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox ...
docker run -it -v /media/USB-DRIVE/archivebox/data:/data archivebox/archivebox:dev ...
```
Then to view your data, you can look in the folder on the host `/media/USB-DRIVE/archivebox/data`, or use the Web UI:
```bash
docker run -it -v /media/USB_DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox
docker run -it -v /media/USB_DRIVE/archivebox/data:/data -p 8000:8000 archivebox/archivebox:dev
# then open https://127.0.0.1:8000
```
@@ -273,7 +272,7 @@ The easiest way is to use `archivebox config --set KEY=value` or edit `./Archive
For example, this sets `TIMEOUT=120` as a persistent setting for the collection:
```bash
docker run -it -v $PWD:/data archivebox/archivebox config --set TIMEOUT=120
docker run -it -v $PWD:/data archivebox/archivebox:dev config --set TIMEOUT=120
# OR
echo 'TIMEOUT=120' >> ./ArchiveBox.conf
```
@@ -282,8 +281,8 @@ ArchiveBox in Docker also accepts config as environment variables, see more on t
For example, this disables the screenshot extractor for a single run (without persisting for other runs):
```bash
docker run -it -v $PWD:/data -e SCREENSHOT_ENABLED=False archivebox/archivebox add 'https://example.com'
docker run -it -v $PWD:/data -e SCREENSHOT_ENABLED=False archivebox/archivebox:dev add 'https://example.com'
# OR
echo 'SCREENSHOT_ENABLED=False' >> ./.env
docker run ... --env-file=./.env archivebox/archivebox ...
docker run ... --env-file=./.env archivebox/archivebox:dev ...
```
+25 -25
View File
@@ -1,6 +1,6 @@
# Install
ArchiveBox is primarily distributed as a Python package via `pip`, but it also depends on some system packages that can be installed manually or automatically with Docker. It usually takes less than ~10min to get ArchiveBox set up and running.
ArchiveBox is primarily distributed as a Python package installed with `uv`, but it also depends on some system packages that can be installed manually or automatically with Docker. It usually takes less than ~10min to get ArchiveBox set up and running.
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/601d587d-b59f-47b9-938e-8a7fa7790176" width="20%" align="right"/>
@@ -59,12 +59,11 @@ It's also recommended to use a filesystem with compression and/or [deduplication
2. Then follow the [Quickstart](https://github.com/ArchiveBox/ArchiveBox#quickstart) guide and read the [[Docker]] wiki page for next steps. ➡️
> You can also run Dockerized ArchiveBox using [UNRAID/TrueNAS/Proxmox/etc.](https://github.com/ArchiveBox/ArchiveBox#-other-options) or [Kubernetes](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml).
> You can also run Dockerized ArchiveBox using [UNRAID/TrueNAS/Proxmox/etc.](https://github.com/ArchiveBox/ArchiveBox#-other-options) or Kubernetes.
**More info:**
- [`Dockerfile`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/Dockerfile)
- [`docker-compose.yml`](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml)
- [`archivebox-kubernetes.yml`](https://github.com/ArchiveBox/docker-archivebox/blob/master/archivebox.yml)
- [ArchiveBox Docker Quickstart](https://github.com/ArchiveBox/ArchiveBox#quickstart) + [Usage](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker) + [Configuration](https://github.com/ArchiveBox/ArchiveBox/wiki/Docker#configuration) + [Upgrading](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives) documentation
<br/>
@@ -82,9 +81,9 @@ If you're on Linux with `apt` or FreeBSD with `pkg` there is an optional auto-se
```bash
curl -fsSL 'https://get.archivebox.io' | bash
# shortcut to run https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/bin/setup.sh
# shortcut to run https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/bin/setup.sh
```
The script explains what it installs beforehand, and will prompt for user confirmation before making any changes to your system. The script uses Docker if already installed, but you can decline and it will attempt to auto-install everything using `apt`/`brew`/`pkg` + `pip` instead.
The script explains what it installs beforehand, and will prompt for user confirmation before making any changes to your system. The script uses Docker if already installed, but you can decline and it will install ArchiveBox using `uv` instead.
<img src="https://imgur.zervice.io/VMTzm0G.png" width="99%"/>
@@ -100,7 +99,7 @@ After running the setup script, continue with the [Quickstart](https://github.co
## Option C. Bare Metal Setup
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [auto-install script](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `pip` & your system package manager of choice (e.g. `apt`, `brew`, `pkg`, `nix`, etc.).
If you'd rather not use [Docker](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup) or our [auto-install script](https://github.com/ArchiveBox/ArchiveBox#%EF%B8%8F-easy-setup), you can follow these manual setup instructions to install ArchiveBox and its dependencies using `uv` & your system package manager of choice (e.g. `apt`, `brew`, `pkg`, `nix`, etc.).
See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) documentation to see the full list of dependencies and how they're used. Not all the dependencies are required for all modes. If you disable some archive methods you can skip installing those dependencies — for example, if you set [`MEDIA_ENABLED=False`](https://archivebox.github.io/abx-plugins/#media) you don't need to install `yt-dlp`, and if you set [`PDF_ENABLED=False`](https://archivebox.github.io/abx-plugins/#pdf), [`SCREENSHOT_ENABLED=False`](https://archivebox.github.io/abx-plugins/#screenshot), and [`DOM_ENABLED=False`](https://archivebox.github.io/abx-plugins/#dom) you don't need `chromium`.
@@ -125,9 +124,9 @@ See our [Dependencies](https://github.com/ArchiveBox/ArchiveBox#dependencies) do
Make sure you have [Homebrew](https://brew.sh/) installed first.
```bash
# Install ArchiveBox's dependencies manually (instead of using the all-in-one brew package)
brew install python3 node git wget curl ffmpeg yt-dlp ripgrep sonic
pip install archivebox
# Install ArchiveBox's dependencies manually (instead of using the all-in-one brew tap)
brew install uv node git wget curl ffmpeg yt-dlp ripgrep sonic
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
archivebox install
# Optional: get FFMPEG with the AAC addon
@@ -151,13 +150,15 @@ sudo apt install archivebox
mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
sudo archivebox install
archivebox install
archivebox add 'https://example.com'
```
The apt package installs the ArchiveBox Python application. Runtime extractor
The apt package is a thin dev-channel wrapper around the normal Python install
flow. Runtime extractor
dependencies such as Chromium, yt-dlp, SingleFile, and other plugin-managed
tools are installed by `sudo archivebox install`.
tools are installed by `archivebox install`; use `sudo archivebox install` only
if you want it to install missing system packages via apt.
<img src="https://cdn0.iconfinder.com/data/icons/flat-round-system/512/freebsd-512.png" width="30px" align="right"/>
@@ -175,7 +176,7 @@ sudo pkg install chromium
#### OpenBSD
```bash
sudo pkg_add python3 node wget git curl yt-dlp ffmpeg ripgrep chromium
sudo pkg_add python313 node wget git curl yt-dlp ffmpeg ripgrep chromium
```
#### Arch Linux / Nix / Guix / etc. Other OSs
@@ -187,13 +188,13 @@ See the [Quickstart](https://github.com/ArchiveBox/ArchiveBox#-package-manager-s
<img src="https://github.com/ArchiveBox/ArchiveBox/assets/511499/65315723-adae-42e4-b8c6-e44b79165ae5" width="55px" align="right"/>
### 2. Install the Python dependencies using `pip`
### 2. Install ArchiveBox using `uv`
If you are not using the apt package above, install ArchiveBox with `pip`.
If you are not using the apt package above, install ArchiveBox with `uv`.
```bash
# get the latest version of archivebox from PyPI
pip install --upgrade --ignore-installed archivebox[ldap,sonic]
# get the dev version of ArchiveBox
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
# if you see errors about ldap, install the C++ build tools + ldap headers and retry (only needed on some OSs + if you want ldap)
# apt install build-essensial python3-ldap
@@ -212,7 +213,7 @@ mkdir -p ~/archivebox/data && cd ~/archivebox/data
archivebox init
# auto-install runtime dependencies such as Chromium, yt-dlp, SingleFile, etc.
sudo archivebox install
archivebox install
# archive a first URL
archivebox add 'https://example.com'
@@ -226,13 +227,12 @@ archivebox help
### Troubleshooting
Make sure the `pip`-installed version of `archivebox` is available in your `$PATH`.
Make sure the `uv`-installed version of `archivebox` is available in your `$PATH`.
```bash
pip show archivebox # show info about the pip-installed version of archivebox
uv tool list # show info about uv-installed tools
echo $PATH # show the directories your system is searching for binaries
which -a archivebox # show all installed archivebox binaries available
which archivebox # show which archivebox binary is being called
type -a archivebox # show all installed archivebox binaries available
cd ~/archivebox/data
archivebox version # ⭐️ show lots of useful info about installed dependencies and more
@@ -280,13 +280,13 @@ See our [[Usage]] Wiki documentation page for more examples.
Make sure all apt/brew/pkg/etc. dependencies from above are installed & up-to-date first.
```bash
# get the latest archivebox version from PyPI
pip install --upgrade --ignore-installed archivebox
# get the dev version of ArchiveBox
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
# run init inside any data directories to migrate the index to the latest version
cd ~/archivebox/data
archivebox init # update collection index & apply any migrations
sudo archivebox install # update runtime dependencies to latest versions
archivebox install # update runtime dependencies to latest versions
```
Check our more detailed [Upgrading](https://github.com/ArchiveBox/ArchiveBox/wiki/Upgrading-or-Merging-Archives) documentation and [Release Notes](https://github.com/ArchiveBox/ArchiveBox/releases) if you run into any problems. ➡️
+2 -2
View File
@@ -82,9 +82,9 @@ LOGOUT_REDIRECT_URL=https://auth.yourcompany.example.com/after/logout
> Can be used with an SSO provider like [Authentik](https://github.com/goauthentik/authentik), [Authelia](https://github.com/authelia/authelia), [Okta / Auth0](https://www.okta.com/), [Keycloak](https://www.keycloak.org/), and others.
First, `pip`-install the `ldap` add-on to use this feature (not needed for Docker Archivebox).
First, install the `ldap` add-on to use this feature (not needed for Docker Archivebox).
```bash
pip install archivebox[ldap]
uv tool install --python 3.13 --upgrade 'archivebox[ldap] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
```
Then set these configuration values to finish configuring LDAP:
+1 -1
View File
@@ -144,7 +144,7 @@ Internally it functions as an index store, storing only the original IDs of the
*ArchiveBox has supported Sonic for years, and it is the most thoroughly tested and recommended backend for ArchiveBox users that need to scale beyond `ripgrep`.*
Using [sonic with ArchiveBox in Docker Compose](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml) is the easiest way to get started, though you can also use it without Docker by [installing it manually](https://github.com/valeriansaliou/sonic#installation) and then running `pip install archivebox[sonic]`.
Using [sonic with ArchiveBox in Docker Compose](https://github.com/ArchiveBox/ArchiveBox/blob/dev/docker-compose.yml) is the easiest way to get started, though you can also use it without Docker by [installing it manually](https://github.com/valeriansaliou/sonic#installation) and then running `uv tool install --python 3.13 --upgrade 'archivebox[sonic] @ git+https://github.com/ArchiveBox/ArchiveBox.git@dev'`.
```bash
# edit docker-compose.yml to uncomment the lines that enable sonic
+7 -8
View File
@@ -21,11 +21,10 @@ Then make sure `archivebox` is installed available in your `$PATH`.
```bash
apt show archivebox # show info about the apt-installed version of archivebox
brew info archivebox # show info about the brew-installed version of archivebox
pip show archivebox # show info about the pip-installed version of archivebox
uv tool list # show info about uv-installed tools
echo $PATH # show the directories your system is searching for binaries
which -a archivebox # show all installed archivebox binaries available
which archivebox # show which archivebox binary is being called
type -a archivebox # show all installed archivebox binaries available
```
**⭐️ Show the full archivebox version info + info about all installed dependencies:**
```bash
@@ -34,7 +33,7 @@ archivebox version # shows lots of useful info about installed dependencie
(ensure the version shown is the most recent available from [Releases](https://github.com/ArchiveBox/ArchiveBox/releases))
### macOS
ArchiveBox can be installed with Homebrew or `pip`/`pipx` on macOS:
ArchiveBox can be installed with Homebrew or `uv` on macOS:
```bash
brew tap archivebox/archivebox
brew install archivebox
@@ -49,7 +48,7 @@ More info: https://github.com/ArchiveBox/homebrew-archivebox
### Python
Make sure you have at least Python 3.9 installed on your system.
Make sure you have at least Python 3.13 installed on your system.
```bash
python3 --version
@@ -119,9 +118,9 @@ node --version # make sure you have node >=19 installed
npm --version # make sure you have npm installed
cd ~/archivebox/data # go into your data directory
sudo archivebox install # auto-installs runtime dependencies
archivebox install # auto-installs runtime dependencies
# equivalent to:
# curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/stable/archivebox/package.json' > package.json
# curl -fsSL 'https://raw.githubusercontent.com/ArchiveBox/ArchiveBox/dev/archivebox/package.json' > package.json
# npm install
# install npm dependencies should then be present in ~/archivebox/data/node_modules/.bin
@@ -201,7 +200,7 @@ Database and filesystem issues are uncommon but do come up from time to time (es
*ℹ️ Generally, these commands can help you resolve most issues:*
```bash
archivebox init # upgrade the archivebox collection
archivebox init --setup # upgrade the archivebox collection and all dependencies
archivebox install # upgrade the archivebox runtime dependencies
archivebox update --index-only # force an upgrade of some of the archivebox index/collection files
archivebox server --debug # run the server with more verbose debug log output
archivebox shell # access the Python API / Django management shell
+5 -5
View File
@@ -4,9 +4,9 @@
# cd /path/to/your/archivebox/data
cd ~/archivebox/data
pip install --upgrade --ignore-installed archivebox
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
# or
docker pull archivebox/archivebox:latest
docker pull archivebox/archivebox:dev
# upgrade the collection to a new version
archivebox init
@@ -27,7 +27,7 @@ archivebox init
---
*Note: It's recommended to only upgrade one major version at a time. e.g. if you're on `v0.4.14`, upgrade to `v0.5.6` next, then `v0.6.3`, and finally `v0.7.1` (as 3 separate steps).
You can specify exact versions with pip like so: `pip install archivebox==0.6.3` or with docker `docker pull archivebox/archivebox:0.6.3`. Upgrading directly across multiple major versions may work in some cases, but is not recommended for maximum data safety.*
You can specify exact versions with uv like so: `uv tool install --python 3.13 --upgrade archivebox==0.6.3` or with docker `docker pull archivebox/archivebox:0.6.3`. Upgrading directly across multiple major versions may work in some cases, but is not recommended for maximum data safety.*
---
@@ -99,7 +99,7 @@ cd ~/archivebox/data # or wherever your data folder is
killall archivebox # stop the currently running archivebox version
# upgrade ArchiveBox using the package manager you originally used to install it
pip install --upgrade --ignore-installed archivebox
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
# or
sudo apt update
sudo apt install --only-upgrade archivebox
@@ -107,7 +107,7 @@ sudo apt install --only-upgrade archivebox
curl -sSL 'https://get.archivebox.io' | sh
archivebox init # run init to upgrade the collection to the latest version
sudo archivebox install # refresh runtime dependencies if needed
archivebox install # refresh runtime dependencies if needed
archivebox update --index-only # optionally force an update of the snapshot index files (normally done lazily, see issue #962 for more info)
+1 -1
View File
@@ -26,7 +26,7 @@
All three of these ways of running ArchiveBox are equivalent and interchangeable:
- `archivebox [subcommand] [...args]`
*Using the PyPI package via `pip install archivebox`*
*Using the Python package via `uv tool install archivebox`*
- `docker run ... archivebox/archivebox [subcommand] [...args]`
*Using the official Docker image*
- `docker-compose run archivebox [subcommand] [...args]`
+1 -1
View File
@@ -496,7 +496,7 @@ You can find more organizations and initiatives on these other lists:
- [Python PyPI](https://pypi.org/project/archivebox/)
- [Docker Hub](https://hub.docker.com/r/archivebox/archivebox)
- [ArchLinux AUR](https://aur.archlinux.org/packages/archivebox)
- [Ubuntu Launchpad PPA](https://launchpad.net/~archivebox/+archive/ubuntu/archivebox)
- [Ubuntu/Debian apt wrapper](https://github.com/ArchiveBox/debian-archivebox)
---
+1 -1
View File
@@ -23,7 +23,7 @@ ArchiveBox
.. code-block:: bash
mkdir my-archive; cd my-archive/
pip install archivebox
uv tool install --python 3.13 --upgrade 'git+https://github.com/ArchiveBox/ArchiveBox.git@dev'
archivebox init
archivebox add https://example.com
+1 -1
View File
@@ -1,6 +1,6 @@
{
"name": "archivebox",
"version": "0.9.35rc50",
"version": "0.9.35rc103",
"repository": "github:ArchiveBox/ArchiveBox",
"license": "MIT",
"dependencies": {
+4 -4
View File
@@ -1,6 +1,6 @@
[project]
name = "archivebox"
version = "0.9.35rc50"
version = "0.9.35rc103"
requires-python = ">=3.13"
description = "Self-hosted internet archiving solution."
authors = [{name = "Nick Sweeting", email = "pyproject.toml@archivebox.io"}]
@@ -79,9 +79,9 @@ dependencies = [
### Extractor dependencies (optional binaries detected at runtime via shutil.which)
### Binary/Package Management
"abxbus==2.5.10", # EventBus API
"abxpkg>=1.11.219", # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
"abx-plugins>=1.11.223", # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
"abx-dl>=1.11.223", # shared ArchiveBox downloader package with blocking install preflight
"abxpkg>=1.11.239", # for: detecting, versioning, and installing binaries via apt/brew/pip/npm
"abx-plugins>=1.11.248", # shared ArchiveBox plugin package with Chrome/Puppeteer dependency wiring
"abx-dl>=1.11.235", # shared ArchiveBox downloader package with blocking install preflight
### UUID7 backport for Python <3.14
"uuid7>=0.1.0; python_version < '3.14'", # provides the uuid_extensions module on Python 3.13
]
Generated
+462 -479
View File
File diff suppressed because it is too large. Load diff